User-agent: denova-htdig # directed to robot running on DeNova Disallow: /cgi-bin/ Disallow: /cgi-local/ Disallow: /secure/ Disallow: /htdig/ Disallow: /Hello/ Disallow: /TestHello/ Disallow: */installers Disallow: /site_media/ Disallow: /stats/ User-agent: * # directed to all spiders Disallow: /Hello/ Disallow: /TestHello/ Disallow: */installers Disallow: /site_media/ Disallow: /stats/ Disallow: *.txt Disallow: *.gif Disallow: *.jpg Disallow: *.png Disallow: *.jar Disallow: *.exe Disallow: *.zip Disallow: test*.*