Scraping Down, Ad Revenue Up!

Somewhere in the past I rambled about my revenues getting stomped when greedy crawlers and thieving high speed scrapers hammered the crap out of my servers locking out legitimate visitors for minutes or hours during what could be considered a DOS attack based on the speed of requesting 40K pages.

Well here's a big shock, now that I've effectively stopped their asses cold my ad revenues have returned to levels they were about 3 months ago before the crawling became a near epidemic.

Makes you wonder if some of my listings were suffering in the Big 3 search engines because of this as my review of several log files showed that sometimes all 3 search engines were attempting to crawl at the same time some of the big time scrapers were tying up my server. That's when I suspected the Big 3 SEs timed out on those pages and lowered the listings in the SERPs like it did when I recently caused a problem and they've taken this long to get back where they were.

You know it was getting real bad when one night I get a wake-up call at 2am by my sister-in-law, who has sites on my server, calling to complain it went down 20 minutes ago while she was working on her web site and slightly afraid she did it. Pings to the server and various services confirmed it was up and running but unable to respond to data requests as it was just tied up, completely overloaded, and wouldn't even give me a prompt in SSH so I could kill the tasks and block the intruder. However, on that night I let it ride as I hate forced reboots that can crash a database (no sleep then) and sure enough it came back to normal an hour later as I expected.

Anyway, just thought someone that's been reading this saga might find this interesting as I certainly don't think it's a coincidence the ad revenue is bouncing back at the same time the crawlers have been beaten to a pulp.