I was looking through my error logs and I saw numerous errors with people trying to find this file on my server:

httpdocs/robots.txt

This rings a bell but I can't remember the significance. What's this file usually have in it?

EDIT>> OK, now I remember -- a file to be used to block leeching programs?

So do all those programs look for this file? Or is there some other reason someone is looking for this file?

search engines use it do determine where, when and how often to look at your files. You can tell search engines not to look in certain directories...or to come back every 14 days etc....etc...etc...

At least that is my understanding...

You are correct cite... Robots.txt is the file that tells a search engines crawler where to look and how often to come back.

If you want to know how to use them, look here:
http://www.searchtools.com/robots/robots-txt.html

It's useful if you have a password protected directory with tons of links to it, it prevents lots of forbidden errors when search engines are going through your site.

it's also useful if you have content management scripts loaded, so that the search engines are less likely to index individual items instead of your .com url.