> -----Message d'origine----- > De: David A. Desrosiers [SMTP:[EMAIL PROTECTED]] > Date: vendredi 27 septembre 2002 13:13 > Ā: Plucker Development List > Objet: Re: Stayondomain for testing > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > > It's pretty modular and innocuous, and is based on the current tip. > > ..and here's my attempt, using perl. It works for everything I've > thrown at it so far. I'll shim this into my spider and get some samples: > > # ------------------------------------------------------------------ > use strict; > my %fqdn; > > my @list = ('http://perlnewbies.com/history/searchit.html', > 'http://excite.com/search.gw?c=web&lk=excite_home_us&s', > 'http://search.msn.co.uk/spbasic.htm?MT=Presidents', > 'http://search.msn.com/results.asp?RS=CHECKED&Armada', > 'http://search.yahoo.com/bin/search?p=%22Perl', > 'http://top.cswap.com:80/cat/?biblestudies', > 'http://203.47.133.209/pipermail/plucker-dev/', > 'http://sers:[EMAIL PROTECTED]/abstract/spe/medgen/index.htm'); > > foreach (@list) { > $_ =~ s!(^.*?//)|([:/].*$)!!g; > $fqdn{$_}++; > } > > foreach my $dom (sort keys(%fqdn)) { > print "$dom = $fqdn{$dom}\n" > } > # ------------------------------------------------------------------
It doesn't work when the URL contains a login/pass (HTTP basic auth), as in : http://login:[EMAIL PROTECTED]/abstract/spe/medgen/index.htm ...where the ':' have a second meaning... The regex that handle this along with the rest is : $_ =~ s!(^.*?@)|(^.*?//)|([:/].*$)!!g; (I don't know how to better write "the longer of the strings terminated by either @ or //") Isn't there somewhere a good reference where one can find the regex to parse URL, as there is one for parsing emails addresses. NH _______________________________________________ plucker-dev mailing list [EMAIL PROTECTED] http://lists.rubberchicken.org/mailman/listinfo/plucker-dev