Re: Can anyone suggest a good HTTP/1.1 web client?
On Tue, 16 Dec 2008 12:18:17 -0800 (PST), Kottiyath n.kottiy...@gmail.com wrote: Hi all, I have to connect to a secure website every second to get the data and then post to it. I have been investigating on many web clients in python, but nothing fits the bill properly. The ones I tried implementing are: 1. httplib based - I created myself. (I cannot use urllib2 since I have to transfer files, and urllib2 doesnt have multipart content-type support) 2. Twisted web client. I also looked at mechanize etc too. The problems I face are - 1. I liked twisted a lot, but when I implemented it, I found that client support is there only for twisted.web and not twisted.web2. Since I connect to the same website every time, I would like to have persistent connections and since twisted.web is HTTP/1.0, persistent connection support is not yet there. Without persistent connections, I would have to have TCP connection handshake everytime and it is taking too much time. I'm working on a new HTTP 1.1 client for Twisted. You can follow the progress at http://twistedmatrix.com/trac/ticket/886 or check out the development branch svn://svn.twistedmatrix.com/svn/Twisted/branches/expressive-http-client-886 and give it a try. Persistent connections are not yet implemented, but there's not a whole lot left to do to support them. That means pipelining isn't supported yet, but once persistent connections are supported pipelining will be simple. As with any Twisted protocol, it's just as easy to run over SSL as over TCP, so HTTPS is essentially supported for free. There is not yet support for proxies, but this is planned as well and should be only a little more work to implement. [snip] Is there any good web client which I can use straight up? Or would I have to implement the whole thing myself? It looks like a big beast and I was wondering whether python provides it straight up. If you'd like to help out with the new Twisted HTTP client, that would be wonderful. Even if you can just try it out and report any problems you run into, that would be immensely helpful. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Can anyone suggest a good HTTP/1.1 web client?
If you'd like to help out with the new Twisted HTTP client, that would be wonderful. Even if you can just try it out and report any problems you run into, that would be immensely helpful. Jean-Paul I would love to help out with the client. But, I am extremely tied up at the current moment. I would be able to provide any sort of meaningful contribution only after ~a month. I can surely try this out and upload any bugs in the ticket you mentioned. Also, is there any planned date for persistent connections? If not, that will be the first thing I will be working on after my assignment is over - since it increasingly looks like it is essential to my system (actually, any REST applications). Thank you very much, Jean-Paul. Regards, K -- http://mail.python.org/mailman/listinfo/python-list
Re: Can anyone suggest a good HTTP/1.1 web client?
On Tue, 16 Dec 2008 18:47:41 -0800 (PST), Kottiyath n.kottiy...@gmail.com wrote: If you'd like to help out with the new Twisted HTTP client, that would be wonderful. Even if you can just try it out and report any problems you run into, that would be immensely helpful. Jean-Paul I would love to help out with the client. But, I am extremely tied up at the current moment. I would be able to provide any sort of meaningful contribution only after ~a month. I can surely try this out and upload any bugs in the ticket you mentioned. Also, is there any planned date for persistent connections? If not, that will be the first thing I will be working on after my assignment is over - since it increasingly looks like it is essential to my system (actually, any REST applications). Hard to predict when it will be done. Sometime in January is a possibility, but that's a guess, not a promise. :) Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Can anyone suggest a good HTTP/1.1 web client?
On Dec 17, 1:18 am, Kottiyath n.kottiy...@gmail.com wrote: Hi all, I have to connect to a secure website every second to get the data and then post to it. I have been investigating on many web clients in python, but nothing fits the bill properly. The ones I tried implementing are: 1. httplib based - I created myself. (I cannot use urllib2 since I have to transfer files, and urllib2 doesnt have multipart content-type support) 2. Twisted web client. I also looked at mechanize etc too. The problems I face are - 1. I liked twisted a lot, but when I implemented it, I found that client support is there only for twisted.web and not twisted.web2. Since I connect to the same website every time, I would like to have persistent connections and since twisted.web is HTTP/1.0, persistent connection support is not yet there. Without persistent connections, I would have to have TCP connection handshake everytime and it is taking too much time. 2. Since I connect to the website every second, I have to have many connections running at the same time. I am worried that creating threads for each connection is going to be a big problem (esp if the server response is slow), since the processor will get swamped - especially since there are many other activities going on in the machine. 3. I would also like to pipe line the requests - esp if the response is slow. Other requirements: 1. HTTPS Support 2. Connection through proxy. You could take a look at pycurl - python bindings for libcurl. http://pycurl.sourceforge.net/ srp Is there any good web client which I can use straight up? Or would I have to implement the whole thing myself? It looks like a big beast and I was wondering whether python provides it straight up. Regards K -- http://mail.python.org/mailman/listinfo/python-list
