OK, this must be a clue that my bot blocker has graduated to the head of the class as I've snared 2 coporations bypassing security measures within 24 hours pretending to be browsers.
Remember what I said about bot blocking being an onion that you keep peeling layer by layer?
The next one in our list of sneaky snoopers is Cyveillance, which apparently has been around for a while but went silently unnoticed until I cranked up the level of bot profiling on my site just a bit to see if I was missing anyone and BINGO! got 2 big fish in a day looking at the next layer of the onion.
According to what I've been reading at linuXgod's site, these boys spy for the RIAA, government and god knows who else or for what purposes. He's been trying to get them to stop crawling his site via a small back and forth of emails and they don't seem to be interested in complying.
My favorite quote is where they justify ignoring internet standards like robots.txt and mask the user agent string as a browser ""Mozilla/4.0 (compatible; MSIE 6.1; Windows XP)".
Because many sites use redirection pages to route robots to special "indexing" pages, we identify our web crawler as an IE browser to ensure it receives the same content as the majority of web surfers on the internet and to allow our programmers to concentrate on a single interpretation of thehtml standard.Well hell, doesn't that logic just make it fucking OK to ignore whether I want your robot on my server in the first place?
So you're justified in bypassing my security to stop browsers just to concentrate on a single html standard?
Well guess what, NO, YOU'RE NOT JUSTIFIED!
Here you go people, the range of IPs so block them as we're not being given any other means to detect this crawler:
whois 63.148.99.239Wish I had the bot blocker commercialized now to go mainstream and nail this nonsense.
Cyveillance QWEST-63-148-99-224 (NET-63-148-99-224-1)
63.148.99.224 - 63.148.99.255
and...
CYVEILLANCE UU-65-213-208-128-D4 (NET-65-213-208-128-1)
65.213.208.128 - 65.213.208.159
2 comments:
I don't know what possessed me to start with that Uncle Bill shit the other night, I'm sure it was the scotch, oh lord.
Glad you liked the site ;)
netcraft1="194.72.0.0/16"
netcraft2="212.95.224.0/19"
whoissc="216.145.0.0/19"
inktomi1="74.6.0.0/16"
inktomi2="72.30.0.0/16"
yahoochina1="219.154.0.0/15"
yahoochina2="222.222.0.0/15"
google="66.249.64.0/19"
askjevees="65.192.0.0/11"
hostinginestonia="213.219.64.0/18"
mostingmsdn="71.13.115.117"
microsoft="207.46.0.0/16"
pingdomateveryone="66.98.128.0/17"
tabranak="217.174.203.0/24"
verisign="208.17.184.0/24"
internetarchive="208.70.24.0/21"
websense="66.194.6.0/24"
cyveillance1="65.222.176.96/27"
cyveillance2="63.148.99.224/27"
alexa="209.237.236.0/24"
Post a Comment