RE: [Robots] robot in python?

2003-11-26 Thread Sean M. Burke
of Sound as a sensory perception, clearly not in the interest of any individual or body or civilization, if it were possible in the first place. You talk funny! This pleases me. -- Sean M. Burkehttp://search.cpan.org/~sburke/ ___ Robots mailing list

[Robots] Re: matching and UserAgent: in robots.txt

2002-03-15 Thread Sean M. Burke
there's no version to strip off. So I'm going to send Gisle Aas a patch so that the first word, minus any version suffix, is what's used for matching. It's just a matter of adding a line saying: $name = $1 if $name =~ m/(\S+)/; # get first word in the 'agent' method. -- Sean M. Burke

[Robots] matching and UserAgent: in robots.txt

2002-03-14 Thread Sean M. Burke
, I'm tempted to ask... what /is/ the proper behavior? Maybe shave the current user-agent's name at the first slash or space (getting just Banjo), and then seeing if /that/ is a substring of a given robots.txt User-Agent: line? -- Sean M. Burke[EMAIL PROTECTED]http://www.spinn.net/~sburke

[Robots] Re: matching and UserAgent: in robots.txt

2002-03-14 Thread Sean M. Burke
on this before bringing it up with the others, tho. -- Sean M. Burke[EMAIL PROTECTED]http://www.spinn.net/~sburke/ -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send help in the body of a message to [EMAIL PROTECTED].