User-agent: * Disallow: /interesting.cgi ## I don't want to help these companies below. Got some of these from ## here: http://www.perldesignpatterns.com/robots.txt User-agent: copyright sheriff Disallow: / User-agent: TurnitinBot Disallow: / User-agent: NaverBot Disallow: / # Same as Naver User-agent: Yeti Disallow: / User-agent: FAST-WebCrawler Disallow: / # For a number of reasons User-agent: msnbot Disallow: / User-agent: Mercator Disallow: / User-agent: NPBot Disallow: / User-agent: ZyBorg Disallow: / User-agent: Slurp Disallow: / User-agent: RoboCrawl Disallow: / User-agent: OmniExplorer Disallow: / User-agent: Exabot Disallow: / User-agent: Baiduspider Disallow: / # This one ignores robots.txt anyway, but meh User-agent: Purebot Disallow: /