Re: POE useragent and https
The reason I'm trying to get https support with PoCoCl::Useragent is because I used to use LWP::Parallel, and that turned out to be too slow/inaccurate in the way it downloads multiple items in parallel. I'm looking to mimic a web browser as much as possible, including the multi-threaded downloads it does of embedded components. Secure site slow download times was actually the main reason I couldn't use LWP::Parallel anymore. Unless anyone has any other ideas, POE looks to be the best option for this type of functionality. I may need to switch to using PoCoCl::HTTP/S, but that will require a lot of rewriting. I appreciate all the input so far, it does look to be major limitation that has existed for some time, and really does need to be added some time in the future. Lenny - Original Message - From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, November 08, 2002 12:16 PM Subject: Re: POE useragent and https > Has anyone gotten POE::Component::Client::UserAgent to work with > SSL/HTTPS pages? I've disabled nonblocking, and Net::SSL is being > used, but I still get a failure after a few bytes are read. Is there > anything else I can do, or am I out of luck? I was trying to make it work with HTTPS, and after a few rounds gave up on it. If turning non-blocking connects off does not help, fork a process and run LWP or LWP::PUA in it. Or try PoCoCl::HTTP. There may be a different way to create or handle sockets that could improve things a bit, but otherwise I don't think it's an LWP or LWP::PUA or PoCoCl::UA problem. The underlying SSL libraries just don't like non-blocking sockets at all. You can try filing bugs against them. But the other thing is, the way LWP and LWP::PUA are designed it's pretty much impossible to make them truly POE-friendly. For example, an FTP request will block for the entire duration of the connection establishment phase. So a future version of PoCoCl::UA should do away with the LWP dependency entirely. -- Kirill --- Email sent using AnyEmail (http://netbula.com/anyemail/)
Re: POE useragent and https
On Fri, 2002-11-08 at 15:16, [EMAIL PROTECTED] wrote: > I was trying to make it work with HTTPS, and after a few rounds > gave up on it. If turning non-blocking connects off does not help, > fork a process and run LWP or LWP::PUA in it. Or try PoCoCl::HTTP. > > There may be a different way to create or handle sockets that > could improve things a bit, but otherwise I don't think it's an LWP > or LWP::PUA or PoCoCl::UA problem. The underlying SSL libraries > just don't like non-blocking sockets at all. You can try filing bugs > against them. Interesting, in fact, I have done that. I have a component that runs in POE::Wheel::Run that calls LWP, and use Cache::Cache to pass back the response. I consider it a kludge though. The curious part is, if the problem is the underlying SSL library. Then how does PoCoCL::HTTP work with it at all? You see the conflict in this logic. After all, the same SSL libray is used. > But the other thing is, the way LWP and LWP::PUA are designed it's > pretty much impossible to make them truly POE-friendly. For > example, an FTP request will block for the entire duration of the > connection establishment phase. So a future version of PoCoCl::UA > should do away with the LWP dependency entirely. To do away w/ LWP totally may not be so easy. One reason is that it has become somewhat of a standard. People expect the same API's for the request, response, and UserAgent. I wonder whether it's possible to change the low level code so a fix similar to what PoCoCl::HTTP uses can be adapted. Pete
Re: POE useragent and https
> Has anyone gotten POE::Component::Client::UserAgent to work with > SSL/HTTPS pages? I've disabled nonblocking, and Net::SSL is being > used, but I still get a failure after a few bytes are read. Is there > anything else I can do, or am I out of luck? I was trying to make it work with HTTPS, and after a few rounds gave up on it. If turning non-blocking connects off does not help, fork a process and run LWP or LWP::PUA in it. Or try PoCoCl::HTTP. There may be a different way to create or handle sockets that could improve things a bit, but otherwise I don't think it's an LWP or LWP::PUA or PoCoCl::UA problem. The underlying SSL libraries just don't like non-blocking sockets at all. You can try filing bugs against them. But the other thing is, the way LWP and LWP::PUA are designed it's pretty much impossible to make them truly POE-friendly. For example, an FTP request will block for the entire duration of the connection establishment phase. So a future version of PoCoCl::UA should do away with the LWP dependency entirely. -- Kirill --- Email sent using AnyEmail (http://netbula.com/anyemail/)
Re: POE useragent and https
On Thu, Nov 07, 2002 at 10:20:26PM -0500, Peter Chen wrote: > On Thu, 2002-11-07 at 21:47, Rocco Caputo wrote: > > On Thu, Nov 07, 2002 at 11:05:28AM -0800, Lenny Rachitsky wrote: > > > > > Has anyone gotten POE::Component::Client::UserAgent to work with > > > SSL/HTTPS pages? I've disabled nonblocking, and Net::SSL is being > > > used, but I still get a failure after a few bytes are read. Is there > > > anything else I can do, or am I out of luck? > > > > > > Any help at all would be very much appreciated. > > > > We discussed this in IRC, and I spent a long look at the bowels of LWP > > to no avail. Maybe someone else can figure it out, but I'm totally at > > a loss. > > > > POE::Component::Client::HTTP works, however. > > Perhaps we can build on each other's experience. Several of us have > looked into this problem. > > Is POE::Component::Client::UserAgent still based on LWP::ParallelUA? I > recall a few months ago, we got in touch with the authors, and one of > them was working on a fix. Yes, it still uses ParallelUA. I tried it with the last recommended version and the latest one. I also hardcoded LWP::Parallel::Protocol::https to use one and then the other SSL library. Both failed. Lenny suggested Making the underlying SSL socket block, and that seems to help somewhat. It still doesn't fix the problem. Lenny also noted that LWP::ParallelUA works on its own when making the same request (https://www.wellsfargo.com/). -- Rocco Caputo - [EMAIL PROTECTED] - http://poe.perl.org/
Re: POE useragent and https
On Thu, 2002-11-07 at 21:47, Rocco Caputo wrote: > On Thu, Nov 07, 2002 at 11:05:28AM -0800, Lenny Rachitsky wrote: > > > Has anyone gotten POE::Component::Client::UserAgent to work with > > SSL/HTTPS pages? I've disabled nonblocking, and Net::SSL is being > > used, but I still get a failure after a few bytes are read. Is there > > anything else I can do, or am I out of luck? > > > > Any help at all would be very much appreciated. > > We discussed this in IRC, and I spent a long look at the bowels of LWP > to no avail. Maybe someone else can figure it out, but I'm totally at > a loss. > > POE::Component::Client::HTTP works, however. Perhaps we can build on each other's experience. Several of us have looked into this problem. Is POE::Component::Client::UserAgent still based on LWP::ParallelUA? I recall a few months ago, we got in touch with the authors, and one of them was working on a fix. Pete
Re: POE useragent and https
On Thu, Nov 07, 2002 at 11:05:28AM -0800, Lenny Rachitsky wrote: > Has anyone gotten POE::Component::Client::UserAgent to work with > SSL/HTTPS pages? I've disabled nonblocking, and Net::SSL is being > used, but I still get a failure after a few bytes are read. Is there > anything else I can do, or am I out of luck? > > Any help at all would be very much appreciated. We discussed this in IRC, and I spent a long look at the bowels of LWP to no avail. Maybe someone else can figure it out, but I'm totally at a loss. POE::Component::Client::HTTP works, however. -- Rocco Caputo - [EMAIL PROTECTED] - http://poe.perl.org/
POE useragent and https
Has anyone gotten POE::Component::Client::UserAgent to work with SSL/HTTPS pages? I've disabled nonblocking, and Net::SSL is being used, but I still get a failure after a few bytes are read. Is there anything else I can do, or am I out of luck? Any help at all would be very much appreciated. Thank you in advance, Lenny
