Re: httplib incredibly slow :-(

2009-09-07 Thread Chris Withers
Dieter Maurer wrote: Chris Withers writes on Thu, 13 Aug 2009 08:20:37 +0100: ... I've already established that the file downloads in seconds with [something else], so I'd like to understand why python isn't doing the same and fix the problem... A profile might help to understand what the ti

Re: httplib incredibly slow :-(

2009-08-19 Thread Aahz
In article , Chris Withers wrote: >Aahz wrote: >> >> What do you need to know for a decent example? > >Simple download of a file from a url with some auth headers added would >do me. Well, I've hacked up some sample code from my company's codebase: # !!! UNTESTED !!! c = pycurl.Curl() c.setopt

Re: httplib incredibly slow :-(

2009-08-17 Thread Chris Withers
i3dmaster wrote: Just wanted to check if you can try turning on the debug mode for httplib and see if you can read a bit more debug info on where the calls get hung. In your example, it would be conn.set_debuglevel(1) I had a look through the code this debug level controls and I don't see any

Re: httplib incredibly slow :-(

2009-08-14 Thread Dieter Maurer
Chris Withers writes on Thu, 13 Aug 2009 08:20:37 +0100: > ... > I've already established that the file downloads in seconds with > [something else], so I'd like to understand why python isn't doing the > same and fix the problem... A profile might help to understand what the time is used for.

Re: httplib incredibly slow :-(

2009-08-14 Thread Chris Withers
Aahz wrote: Sorry, I mostly have been working on our Mac port, so I'm not sure what's needed to make this work on Windows. Did you try downloading the PyCurl binary? Maybe it statically links libcurl on Windows. Shame it's not available as a bdist_egg, that's what I'm really after... What d

Re: httplib incredibly slow :-(

2009-08-13 Thread Aahz
In article , Chris Withers wrote: >Aahz wrote: >> In article , >> Chris Withers wrote: >>> >>> Does anyone know of an alternative library for creating http requests >>> and getting their responses that's faster but hopefully has a similar >>> interface? >> >> PyCurl > >This seems to be a wra

Re: httplib incredibly slow :-(

2009-08-13 Thread David Robinow
On Thu, Aug 13, 2009 at 3:20 AM, Chris Withers wrote: > David Robinow wrote: >> >> On Wed, Aug 12, 2009 at 12:37 PM, Chris Withers >> wrote: >>> >>> David Stanek wrote: Also on the same box where you run this script can you test with curl or wget? >>> >>> It's a Windows box, so no :-

Re: httplib incredibly slow :-(

2009-08-13 Thread Chris Withers
David Robinow wrote: On Wed, Aug 12, 2009 at 12:37 PM, Chris Withers wrote: David Stanek wrote: Also on the same box where you run this script can you test with curl or wget? It's a Windows box, so no :-( Why not? http://users.ugent.be/~bpuype/wget/ http://curl.haxx.se/download.html Fair

Re: httplib incredibly slow :-(

2009-08-12 Thread David Robinow
On Wed, Aug 12, 2009 at 12:37 PM, Chris Withers wrote: > David Stanek wrote: >> Also on the same box where you run this script >> can you test with curl or wget? > It's a Windows box, so no :-( Why not? http://users.ugent.be/~bpuype/wget/ http://curl.haxx.se/download.html -- http://mail.python.o

Re: httplib incredibly slow :-(

2009-08-12 Thread i3dmaster
On Aug 12, 9:37 am, Chris Withers wrote: > David Stanek wrote: > > I tried to reproduce this, but I could not. Could you paste in the > > output of your script? > > Not sure how that'll help, but sure: > > 2009-08-11 21:27:59.153000 > request: 0:00:00.109000 > response: 0:00:00.109000 > read: 0:24

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
David Stanek wrote: I tried to reproduce this, but I could not. Could you paste in the output of your script? Not sure how that'll help, but sure: 2009-08-11 21:27:59.153000 request: 0:00:00.109000 response: 0:00:00.109000 read: 0:24:31.266000 > Also on the same box where you run this script

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
shaileshkumar wrote: We use PyCURL on Windows. http://pycurl.sourceforge.net/ provides pre- built versions for Windows and it works out of the box. Does it include libcurl? Are these builds available for Python 2.6? Chris -- Simplistix - Content Management, Batch Processing & Python Consultin

Re: httplib incredibly slow :-(

2009-08-12 Thread Shailesh Kumar
Yes it includes libcurl. I didn't have to install it separately. I still continue to use Python 2.4. So cannot say about Python 2.6. - Shailesh On Wed, Aug 12, 2009 at 10:23 PM, Chris Withers wrote: > shaileshkumar wrote: > >> We use PyCURL on Windows. http://pycurl.sourceforge.net/ provides pre

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
Max Erickson wrote: There is an httplib2 (but I don't know anything further about it...): http://code.google.com/p/httplib2/ I had a look, it uses httplib, so will likely suffer from the same problems... Calling wget or curl using a subprocess is probably as easy as it is ugly, I use the w

Re: httplib incredibly slow :-(

2009-08-12 Thread David Stanek
On Tue, Aug 11, 2009 at 4:25 PM, Chris Withers wrote: > Hi All, > > I'm using the following script to download a 150Mb file: > > from base64 import encodestring > from httplib import HTTPConnection > from datetime import datetime > > conn = HTTPSConnection('localhost') > headers = {} > auth = 'Basi

Re: httplib incredibly slow :-(

2009-08-12 Thread shaileshkumar
We use PyCURL on Windows. http://pycurl.sourceforge.net/ provides pre- built versions for Windows and it works out of the box. - Shailesh On Aug 12, 7:14 pm, Max Erickson wrote: > Chris Withers wrote: > > > I'm still reeling from what seems to be such a huge problem with > > httplib that seem

Re: httplib incredibly slow :-(

2009-08-12 Thread Max Erickson
Chris Withers wrote: > > I'm still reeling from what seems to be such a huge problem with > httplib that seem to be largely ignored :-( > > Chris > There is an httplib2 (but I don't know anything further about it...): http://code.google.com/p/httplib2/ Calling wget or curl using a subprocess

Re: httplib incredibly slow :-(

2009-08-12 Thread Chris Withers
Answering myself... Chris Withers wrote: In article , Chris Withers wrote: Does anyone know of an alternative library for creating http requests and getting their responses that's faster but hopefully has a similar interface? PyCurl This seems to be a wrapper around libcurl. Does it wor

Re: httplib incredibly slow :-(

2009-08-11 Thread Chris Withers
Aahz wrote: In article , Chris Withers wrote: Does anyone know of an alternative library for creating http requests and getting their responses that's faster but hopefully has a similar interface? PyCurl This seems to be a wrapper around libcurl. Does it work on Windows? If so, where can

Re: httplib incredibly slow :-(

2009-08-11 Thread Aahz
In article , Chris Withers wrote: > >Does anyone know of an alternative library for creating http requests >and getting their responses that's faster but hopefully has a similar >interface? PyCurl -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "...string ite

httplib incredibly slow :-(

2009-08-11 Thread Chris Withers
Hi All, I'm using the following script to download a 150Mb file: from base64 import encodestring from httplib import HTTPConnection from datetime import datetime conn = HTTPSConnection('localhost') headers = {} auth = 'Basic '+encodestring('username:password').strip() headers['Authorization']=a