hi , I need help as to how i can fetch a wikipedia article i tried changing my user agent but it did not work . Although as far as my knowledge of robots.txt goes , looking at en.wikipedia.org/robots.txt it does not seem it should block a useragent (*, which is what i would normally use) from accesing a simple article like say " http://en.wikipedia.org/wiki/Sachin_Tendulkar"; but still robotparser returns false status=rp.can_fetch("*", "http://en.wikipedia.org/wiki/Sachin_Tendulkar";) where rp is a robot parser object . why is that?
-- A-M-I-T S|S
_______________________________________________ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
