On Tue, Dec 04, 2001 at 05:04:09PM +0200, Johann Botha wrote: > > I am also interested in this. I experience frequent visits from a web > > robot on our ftp-site (with a web front). Going to the home page of > > this robot at www.inktomi.com there was a remark that the robot respects > > robot.txt-files. > > > > How do you set up those files? > > http://www.robotstxt.org/wc/robots.html
Thanks Johann! -- Johann Spies Telefoon: 021-808 4036 Informasietegnologie, Universiteit van Stellenbosch "Go ye therefore, and teach all nations, baptizing them in the name of the Father, and of the Son, and of the Holy Ghost; Teaching them to observe all things whatsoever I have commanded you; and, lo, I am with you alway, even unto the end of the world. Amen." Matthew 28:19,20