#robots instructions of folders not to crawl User-agent: * Disallow: /board/ Disallow: /CD/ Disallow: /cgi-bin/ Disallow: /eco-house/ Disallow: /*.png$ Disallow: /*.psd$ Disallow: /*.gif$ Disallow: /*.jpg$ Disallow: /documents/ Disallow: /ftp/ Disallow: /htsdata/ Disallow: /images/ Disallow: /Library/ Disallow: /logs/ Disallow: /media/ Disallow: /pricing/ Disallow: /print/ Disallow: /stats/ Disallow: /style/ Disallow: /thumbnails/ Disallow: /water-treatment/rb/ Disallow: /water-treatment/ap/ Disallow: /water-treatment/Nuferscience-Plocher-review.pdf Disallow: /water-treatment/water-treatments-thank.html Disallow: /water-treatment/water-treatments-back.html Disallow: /work/ User-agent: LinkWalker Disallow: /