Robots exclusion text

• To: www-cl@ai.mit.edu
• Subject: Robots exclusion text
• From: Chris Tomasello <ctomasello@hammer.net>
• Date: Mon, 15 Jan 1996 17:28:45 -0500
• Organization: Weidenhammer Systems Corporation
• Sender: owner-www-cl@ai.mit.edu

I have a question regarding your robots exclusion text.  I searched 
for robots exclusion text using Alta Vista and got a direct link to 
your robots exclusion text which seems to dissallow all robots.  I 
just wanted to find out if Alta Vista has indexed any of your other 
pages on your site even though you seem to be trying to keep all 
robots out.  I have contacted Alta Vista about there policies 
regarding this for a completely different reason and was told that 
their robot, which is called scooter, obeys robots exclusion text.  

Anyway, just curious about this

Chris

• Prev: Re: Redirection in CL-HTTP
• Next: Re: Redirection in CL-HTTP
• Index(es):
  • Main
  • Thread