Re: urllib2 slow for multiple requests

2009-05-15 Thread Tomas Svarovsky
On May 14, 6:33 pm, Richard Brodie r.bro...@rl.ac.uk wrote: Tomas Svarovsky svarovsky.to...@gmail.com wrote in message news:747b0d4f-f9fd-4fa6-bb6d-0a4365f32...@b1g2000vbc.googlegroups.com... This is a good point, but then it would manifest regardless of the language used AFAIK. And this

Re: urllib2 slow for multiple requests

2009-05-14 Thread Tomas Svarovsky
On May 13, 4:55 pm, cgoldberg cgoldb...@gmail.com wrote: Bascally it just grabs a page xy times and tells me how long it took. you aren't doing a read(), so technically you are just connecting to the web server and sending the request but never reading the content back from the socket.  So

Re: urllib2 slow for multiple requests

2009-05-14 Thread Tomas Svarovsky
One more thing, since I am stuck with 2.4 (and if this is really 2.4 issue), is there some substitution for urllib2? On May 14, 11:00 am, Tomas Svarovsky svarovsky.to...@gmail.com wrote: On May 13, 4:55 pm, cgoldberg cgoldb...@gmail.com wrote: Bascally it just grabs a page xy times and

Re: urllib2 slow for multiple requests

2009-05-14 Thread Richard Brodie
cgoldberg cgoldb...@gmail.com wrote in message news:9ae58862-1cb2-4981-ae6a-0428c7684...@z5g2000vba.googlegroups.com... you aren't doing a read(), so technically you are just connecting to the web server and sending the request but never reading the content back from the socket. But that

Re: urllib2 slow for multiple requests

2009-05-14 Thread Tomas Svarovsky
On May 14, 11:57 am, Richard Brodie r.bro...@rl.ac.uk wrote: cgoldberg cgoldb...@gmail.com wrote in message news:9ae58862-1cb2-4981-ae6a-0428c7684...@z5g2000vba.googlegroups.com... you aren't doing a read(), so technically you are just connecting to the web server and sending the request

Re: urllib2 slow for multiple requests

2009-05-14 Thread cgoldberg
The problem is, that CentOS is running on the server and there is only 2.4 available. On wich version did you ran these tests? I tested with Windows XP and Python 2.5.4. I don't have a 2.4 setup I can easily test with. you can try httplib rather than urllib2. httplib is slightly lower level

Re: urllib2 slow for multiple requests

2009-05-14 Thread cgoldberg
It might be, if the local server doesn't scale well enough to handle 100 concurrent requests. true.. I didn't think of that. I was assuming the client machine wasn't resource constrained. That would definitely lead to inaccurate timings if that was the case. --

Re: urllib2 slow for multiple requests

2009-05-14 Thread Richard Brodie
Tomas Svarovsky svarovsky.to...@gmail.com wrote in message news:747b0d4f-f9fd-4fa6-bb6d-0a4365f32...@b1g2000vbc.googlegroups.com... This is a good point, but then it would manifest regardless of the language used AFAIK. And this is not the case, ruby and php implementations are working quite

urllib2 slow for multiple requests

2009-05-13 Thread Tomas Svarovsky
Hello everybody, really new to python, so bear with me. I am trying to do some very basic scraping tool. Bascally it just grabs a page xy times and tells me how long it took. When I do this once, it is blazingly fast, but when I increase the number of repetitions, it is slowing down considerably

Re: urllib2 slow for multiple requests

2009-05-13 Thread cgoldberg
Bascally it just grabs a page xy times and tells me how long it took. you aren't doing a read(), so technically you are just connecting to the web server and sending the request but never reading the content back from the socket. So your timing wouldn't be accurate. try this instead: response