"Serdar Tumgoren" <zstumgo...@gmail.com> wrote
Is there a way to break out of a function if it exceeds a certain time
limit for execution?
Yes, you can put the call in a thread then in the main thread
start a timer. If the timer expuires before the thread returns
kill the thread.thread
There's a Website I'm scraping on a regular basis, and for some reason
that I can't divine the site has radically varying response times.
Could be a lot of people have written scrapers and they all
schedule them to run at reulat intervals... If the clocks all suddenly
align you will get a massive demand on the server (its effectively
like a denial of service attack on the server!). Or it could even be
real users if it happens when some significant change occurs - like
stock/share price web sites often get big peaks at market
open and close times.
try it again a few moments later, it will sometimes work just fine
("sometimes" being the key word). I'm not certain, but I'm guessing
this inconsistent behavior has something to do with the site that I'm
scraping.
It could be anywhere in your network connection, including your
ISP. If its a local operator there may be peak periods when people
go online based on local patterns of usage, especially if its ot a big
company - they might only have a couple of 2M pipes say to the core
network.
HTH,
--
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/
_______________________________________________
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
