# Directions for robots. See this URL: # http://info.webcrawler.com/mak/projects/robots/norobots.html # for a description of the file format. # Some areas aren't worth crawling User-agent: * Disallow: /cgi-bin/ Disallow: /search/ Disallow: /user/ Disallow: /feedback Disallow: /restricted/