On Fri, Feb 11, 2005 at 01:02:00PM -0500, Kevin Murphy wrote: > Hi, > > Where is Jeremy Wadsack's robot-list.html now? > > This link is now broken: > > http://wadsack-allen.com/products/robot-list.html > > (on page http://www.analog.cx/helpers/#conffiles). > > I attempted to google for it but haven't found anything. > > Thanks, > Kevin Murphy >
Or ... to regenerate it at your convenience: [EMAIL PROTECTED] tmp]$ wget http://www.robotstxt.org/wc/active/all.txt [EMAIL PROTECTED] tmp]$ grep "robot-name:" all.txt | awk -F: '{print $2}' | sed 's/^ *//g' | sort | awk '{print "ROBOTINCLUDE \"" $1 "*\""}' -- Ken Schweigert, Network Administrator Byte Productions, LLC http://www.byte-productions.com +------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.meer.net/mailman/listinfo/analog-help | | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------