Re: urllib2 rate limiting

2008-01-12 Thread Dimitrios Apostolou
On Fri, 11 Jan 2008, Nick Craig-Wood wrote: > Here is an implementation based on that idea. I've used urllib rather > than urllib2 as that is what I'm familiar with. Thanks! Really nice implementation. However I'm stuck with urllib2 because of its extra functionality so I'll try to implement som

Re: urllib2 rate limiting

2008-01-11 Thread Nick Craig-Wood
Dimitrios Apostolou <[EMAIL PROTECTED]> wrote: > I want to limit the download speed when using urllib2. In particular, > having several parallel downloads, I want to make sure that their total > speed doesn't exceed a maximum value. > > I can't find a simple way to achieve this. After resear

Re: urllib2 rate limiting

2008-01-10 Thread Dimitrios Apostolou
On Thursday 10 January 2008 22:42:44 Rob Wolfe wrote: > Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > > On Thu, 10 Jan 2008, Rob Wolfe wrote: > >> Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > >>> P.S. And something simpler: How can I disallow urllib2 to follow > >>> redirections to foreign

Re: urllib2 rate limiting

2008-01-10 Thread Rob Wolfe
Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > On Thu, 10 Jan 2008, Rob Wolfe wrote: > >> Dimitrios Apostolou <[EMAIL PROTECTED]> writes: >> >>> P.S. And something simpler: How can I disallow urllib2 to follow >>> redirections to foreign hosts? >> >> You need to subclass `urllib2.HTTPRedirectHa

Re: urllib2 rate limiting

2008-01-10 Thread Dimitrios Apostolou
On Thu, 10 Jan 2008, Rob Wolfe wrote: > Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > >> P.S. And something simpler: How can I disallow urllib2 to follow >> redirections to foreign hosts? > > You need to subclass `urllib2.HTTPRedirectHandler`, override > `http_error_301` and `http_error_302` m

Re: urllib2 rate limiting

2008-01-10 Thread Rob Wolfe
Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > P.S. And something simpler: How can I disallow urllib2 to follow > redirections to foreign hosts? You need to subclass `urllib2.HTTPRedirectHandler`, override `http_error_301` and `http_error_302` methods and throw `urllib2.HTTPError` exception.

urllib2 rate limiting

2008-01-10 Thread Dimitrios Apostolou
Hello list, I want to limit the download speed when using urllib2. In particular, having several parallel downloads, I want to make sure that their total speed doesn't exceed a maximum value. I can't find a simple way to achieve this. After researching a can try some things but I'm stuck on th