On May 11, 2:32 pm, [EMAIL PROTECTED] wrote: > Folks: > > I am screen scraping a large volume of data from Yahoo Finance each > evening, and parsing with Beautiful Soup. > > I was wondering if anyone could give me some pointers on how to make > it less obvious to Yahoo that this is what I am doing, as I fear that > they probably monitor for this type of activity, and will soon ban my > IP. > > -DE
Depends on what you're doing exactly. I've done something like this and it only hits the page once: URL = 'http://quote.yahoo.com/d/quotes.csv?s=%s&f=sl1c1p2' TICKS = ('AMZN', 'AMD', 'EBAY', 'GOOG', 'MSFT', 'YHOO') u = urlopen(URL % ','.join(TICKS)) for data in u: tick, price, chg, per = data.split(',') # do something with data If you're grabbing all the data in one fell swoop (which is what you should aim for), then it's harder for Yahoo! to know what you're doing exactly. And I can't see why they'd care as that is all a browser does anyway. It's when you hit the site a bunch of times in a short period of time that sets off the alarms. Mike -- http://mail.python.org/mailman/listinfo/python-list
