i've been noticing a steady increase in my traffic. Unfortunately, like a great many other of my efforts, i seem to be more popular with robots then i am with human beings. Not that i don't mind robots (Heck, some of my best friends are robots. Ok, no they're not, but the toaster and i are on good speaking terms), but they're persistant and pesky little buggers.
Twice, so far, my site has been scavanged by a robot out of Japan. Unfortunately, it's far more agressive than the google bot is and follows every single link, be it for content or for css, javascript, graphic, form entries, you name it. This does horrible things to my bandwidth, as in several thousand hits in an hour on this page as it scavanges everything. i'd show you the chart, but i can't connect to my home box right now.
(Mind you, i've still got 24 GB of bandwidth available, even after a particularly bad month, but it's just the principle of the thing…)
i don't want to create a robots.txt file, partly because i want to be crawled. i want my stuff to appear in whatever indexes they want to publish. i'd just prefer if they didn't rape and pillage me along the way.
So i've added some smarts to my page to detect the little buggers and make my pages more bot friendly. i'll be tweaking it more over the weekend, so my apologies in advance if you get some strange results.
It could have said get your jackass files here, and pointed to your blog . . .