> Hello, > > I posted a similar problem, and had to find the results for myself. The > problem wasn't really the robots.txt file - it was part of HtDig that > needed patching. We are using HtDig 3.2.0b4. If that's your version, I > can send you the patch I used. > > Jane Dudley, Webmaster
> Douglas Kline wrote: > > Running the htdig database-creating command fails for me. Amongst the > > diagnostic output lines I got by using a few "v"'s were: > > > > Parsing robots.txt file using myname = htdig > > > > Robots.txt line: User-agent: * > > > > Found 'user-agent' line: * > > > > Pattern: > > pushed > > Rejected: forbidden by server robots.txt! > > > > > > The site which includes the pages I wish to index has a robots.txt file wit h > > the line > > > > User-agent: * > > > > This should permit indexing by htdig. So what's going wrong? TIA. Thanks for your response. I are working on version 3.2.0b5. Would the patch you have work on that version too? Where did you find the patch? Douglas ======== Douglas Kline [EMAIL PROTECTED] ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
