Re: Disallows in robots.txt
Bert Van Kets: >It even overrides the other disalows. No override... Robots Exclusion Standard on User-agent: "(If the value is '*', the record describes the default access policy for any robot that) has not [yet] matched any of the other records." (http://info.webcrawler.com/mak/projects/robots/norobots.html) Cheers, Tuomas
Re: Disallows in robots.txt
Jonathan Knoll: >User-agent: aa >Disallow: /cgi-bin >Disallow: /stuff >Disallow: /x.html [...] >User-agent: * >Disallow: / >Wouldn't this just disallow everyone from everything? No, the file is perfectly OK... The "*" has a special meaning in the Standard: "every other User-agent (not mentioned yet)". So here aa to ddd have limited access, and all >other< bots are banned. Cheers, Tuomas
Commerce Robot Needed
Hello everyone, Our company is seeking an organization that can create, or has an existing bot designed to crawl Affiliate Merchant sites and update a local Oracle database. The functionality we are seeking is similar to popular comparison sites such as MySimon.com, Dealtime.com and Bottomdollar.com. If you have this expertise please contact me via email or telephone. Thank you. Richard Barnett [EMAIL PROTECTED] Tel: (949) 495-9205 Fax: (949) 495-9215
Re: Disallows in robots.txt
> So: > > I was looking at a robots.txt file and it had a series of disallow > instructions for various user agents, and then at the bottom was a full > disallow: [...] > Wouldn't this just disallow everyone from everything? No, it would disallow everyone but a ... d (with the specified restrictions). >From the spec: " The robot must obey the first record in /robots.txt that contains a User- Agent line whose value contains the name token of the robot as a substring. The name comparisons are case-insensitive. If no such record exists, it should obey the first record with a User-agent line with a "*" value, if present. If no record satisfied either condition, or no records are present at all, access is unlimited." Regards, Martin
Re: Commerce Robot Needed
Hi there! I am sure my team and me could do this. But we are a German-Company. If this doesn't matter, please reply to this and the work can start! Yours, Krystian Stolorz / DatenSturm - Original Message - From: Richard Barnett <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, March 24, 2000 3:17 PM Subject: Commerce Robot Needed > Hello everyone, > > Our company is seeking an organization that can create, or has an > existing bot designed to crawl Affiliate Merchant sites and update a > local Oracle database. The functionality we are seeking is similar to > popular comparison sites such as MySimon.com, Dealtime.com and > Bottomdollar.com. > > If you have this expertise please contact me via email or telephone. Thank you. > > Richard Barnett > [EMAIL PROTECTED] > Tel: (949) 495-9205 > Fax: (949) 495-9215
