Friday, May 12, 2006

Circle Jerk Scraping

Not sure what's going on with some of the scrapers exactly but it seems my "CHALLENGE" page, which has about 5 links on the page, is sending them in circles asking for the same pages over and over.

Lately I'm seeing a lot of lost scraper bots doing something like this:

GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
GET /privacy.html
GET /about.html
GET /
GET /legal.html
... and so on and so forth...

Sometimes they'll ask for hundreds of pages like this before breaking out of that loop.

Quite amusing that I seem to have put a kink in their gears.

No comments: