Bots and browser fingerprinting

Pages: [1]
Author Topic: Bots and browser fingerprinting  (Read 707 times)
risilloch
Gaining experience
*

Karma: 13
Posts: 87


View Profile

risilloch

Bots and browser fingerprinting
« on: May 24, 2020, 10:08:56 PM »

Hi everyone,
   i'd just thought i would relate a discovery i have recently made regarding my bot and how the bookmakers have retaliated.
I have a completely unique bot that i wrote myself have never shared and use it on an very narrow set of sport\market combinations.

Historically, bookmakers have used IP screening to stop undesirable actors from accessing their site. I suspect they would somehow
isolate and profile the access patterns of each IP address that visited. If an IP 'hit' against certain criteria then it would be blocked.

With the proliferation of VPN services this became straightforward to defeat. Simply vary your IP and the bookies can't properly profile your access patterns.

For a while my bot has not been able to access a particular bookmaker. With the advent of the pandemic i now find myself with the time to investigate the problem. It made no sense to me - i could access the site via a genuine browser session but my bot could not. So i knew it wasn't my IP they were screening. I put the problem to one side and started on updating my bot to visit some new bookies i had never used before.

To my surprise my bot could not access these either. In a browser i could. So it was definitely not my IP that was the problem. Somehow these new bookies recognised my bot as being a bot. But how could they? This was the first time i had ever visited them. They had never seen me before.

After a bit of research i came across something called browser fingerprinting and realised that this is how i had been caught.

This is used a lot by marketing firms to track your browsing activity. They cannot identify you personally but they can see your browsing habits - whoever you are. They do this by fingerprinting your browser. Your browser is uniquely identified but you remain anonymous. A browser visiting a website can reveal a lot of information to that website to the extent that you can be uniquely, if anonymously, identified.

This is what must have happened in my case. I could access the first bookmaker successfully for a while but whilst doing so it in turn was building a profile of my behaviour. At some point a decision was made that i was undesirable, took a note of my browser fingerprint and used it to block further access to their site.

But it also must have shared that fingerprint with the new bookmaker sites to successfully block me from accessing them on my first visit.
At least that is what i first thought. On reflection i thought that bookmakers are not in the business of gathering AND SHARING this sort of information.

It is more likely that they hire the services of external companies to determine browser fingerprints and block those they consider undesirable. This company then sticks that fingerprint into a central database which they then use when other bookmakers use their services.

This will be how one bookmaker was able to flag my bot as undesirable, the external company inserts the fingerprint into their central database which then benefits their other bookmaker customers when i visit them.

As it turns out defeating this fingerprinting was piss simple in my case and i can now visit all bookmakers concerned successfully after embedding certain decoy behaviours.

Arbusers recently referred to a raid on bots earlier in the year which i didn't understand but now think could be related to the uptake of bookmakers using outsourced fingerprinting technology.

Bot makers that use browser techniques (there are likely other techniques to extract price data but i am not familiar with them) should now build in the ability to disguise fingerprints via configuration. Otherwise their customers may find that the bot they buy off the shelf may not have the shelf life they had hoped for.
Logged
VidaBlue
Has experience
**

Karma: 42
Posts: 180


View Profile

VidaBlue

Re: Bots and browser fingerprinting
« Reply #1 on: May 24, 2020, 11:16:50 PM »

Thank you risilloch for sharing

Your post is very informative and I think you are quite right about the conclusion that you reach, just by doing observance and reasoning.

So far I have only anticipated the problem by programatically altering the user agent in some bots. I don't know if that has not had any importance at all, since my bots have never been blocked, except from b365 blocking IP very temporarily now and then.

Luckily for home developers, these actions are easily countered as you mention. Are you willing to elaborate on what you mean by "embedding certain decoy behaviours"?

Just speculation and reflection from this point:
The bookmakers must be trying hard to counter betting bots. The intent of this struggle is obviously to automatically filter out the bots, without blocking entire campuses and corporations, which must not be an easy thing to do. Probably few bookmakers are large enough to administer such technology in-house and they would purchase the service from a common vendor. The efforts are very welcome and they are hopefully somewhat successful in this, such that the ecosystem of gamblers, bookmakers and sharp bettors remains in a healthy balance.
Logged
arbusers
Administrator
Totally Pro
*****

Karma: 409
Posts: 4124



View Profile WWW

arbusers

Re: Bots and browser fingerprinting
« Reply #2 on: May 25, 2020, 11:33:13 AM »

Thank you risilloch for this valuable information.

We highlighted several times the dangers of Big Data and the way these are used by bookmakers. Also, it is my belief that most bookmakers can't afford to buy this data, or outsource these services. However, it is a matter of time until it becomes available to all, as Big Data companies are jumping out in an unprecedented rhythm.
In regards to bot tackling, I m 100% sure that well-known bookmakers are using clustering to detect bot action, but your report is adding a new technique that I am not aware of. I am sure these bot developers who still have their bots stopped would love to read your previous post.
Logged
risilloch
Gaining experience
*

Karma: 13
Posts: 87


View Profile

risilloch

Re: Bots and browser fingerprinting
« Reply #3 on: May 26, 2020, 04:03:19 PM »


So far I have only anticipated the problem by programatically altering the user agent in some bots. I don't know if that has not had any importance at all, since my bots have never been blocked, except from b365 blocking IP very temporarily now and then.


Changing the user agent did not help me access the sites so i don't think the fingerprint technology uses it.

You can visit https://www.deviceinfo.me/  to see the set of data that a website can determine about a visiting browser.
Its quite a large set.

But varying the user agent may hinder their ability to profile your access as that of a bot. This is speculation on my part but i don't
see how you can lose out if you vary it just on the off chance.


Are you willing to elaborate on what you mean by "embedding certain decoy behaviours"?


A bot is very single minded. It hits the site, grabs the data and leaves. As fast as it can.
Very different behaviour to a mug punter. So my bot tries to emulate a mug punter.
It might peruse some random decoy pages during its access session, visit the casino, view a promotion, accidentally load the betslip etc.


Thank you risilloch for this valuable information.


I'm just pleased i can give something back to the community
Logged
VidaBlue
Has experience
**

Karma: 42
Posts: 180


View Profile

VidaBlue

Re: Bots and browser fingerprinting
« Reply #4 on: May 26, 2020, 11:19:57 PM »

Surprisingly, my browser combination of languages is my most unique feature. User agent comes in second.

I suppose that unless we have some firm insider knowledge of how this technology works, our counter-measures are quite much based on reasoning, trial and error.

Emulating a mug punter behaviour seems like a sound approach and I have also kept that in mind ever since my most profitable automation scheme was stopped some years back. It was at a national book with apparently none of the usual limiting policies. I got greedy and left it turned on overnight, and the account immediately got banned the next morning after a wild night with tennis. No intermediate limitation, direct ban. Few mug punters would sit and bet 4 in the morning, so in retrospect I suppose that really stood out and the book's action was quite straightforward. I had it running for months - first night betting led to ban. After this, I have never had automation turned on overnight.
« Last Edit: May 26, 2020, 11:22:05 PM by VidaBlue » Logged
Pages: [1]
Print