# # Hi Mr. Robot. What are you looking for? # User-agent: aipbot Disallow: / # I'm not sure whether this is post-modern irony, or not... # I guess that makes it post-modern after all. # Whatever; msnbot fetches this file way too often. User-agent: msnbot Disallow: /robots.txt # This rule looks for unscrupulous robots, going where they're told # not to. To the credit of the crawlers, I haven't found any yet. User-agent: * Disallow: /wheel/future.cgi # Nokia does some obnoxious digging-about. Not sure what they're # looking for, but they sure don't take a hint. # "Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) )" User-agent: * Disallow: /m/ Disallow: /mobi/ Disallow: /mobile/ Disallow: /wap/ Disallow: /imode/ Disallow: /xhtml/ ### eof ###