Re: some detailed migration questions...
Hi Oleg, thanks for the answers! I realized there is some very good documentation available which I just did not find, because it is linked only from the HttpCommons Core and not from the Client... Michael 2009/7/9 Oleg Kalnichevski ol...@apache.org Ken Krugler wrote: A few answers in-line below. Disclaimer: I've ported some HttpClient 3.1 code to 4.0, but I haven't made heavy use of the full 4.0 functionality. I am new to HTTPComponents Client but have been using HttpClient 3.1 very extensively... So I am facing some migration work. 1. How do I measure the time it took to get a response in a multithreaded scenario with a limited size of max total connections? In 3.1 I had to hack this into a custom ProtocolSocketFactory. When the socket is created I am rembering the time in a thread local scope. Is that still the way to go? Michael, I would recommend using custom protocol interceptors to collect performance related metrics. Use a request interceptor to store the start time in the HTTP context, and a response interceptor to store the finish time. For details see: http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e290 2. I want to set a different user agent per request In 3.1 I could set HostParams when executing the method. How does it work now? I believe you'd want to implement a RequestUserAgent class that sets up the user agent that you want. Seems like you might be able to do the same thing via using an HttpContext, but Oleg would know best. One could also use the 'http.useragent' parameter: http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e337 3. Setting a proxy per request httpClient.execute(proxyHost, httpget, responseHandler); Correct? Use 'http.route.default-proxy' parameter. http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e465 http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e540 4. Disabling redirects params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, false); Correct? Yes. 5. Enabling my own retry handler (which I just need because of verbose logging) ((DefaultHttpClient) this.httpClient).setHttpRequestRetryHandler(XXX); Correct? Yes. 6. When using the ThreadSafeClientConnManager how can I monitor the number of threads used or threads waiting? Well, you can call getConnectionsInPool() to get the number of in-use connections, though I think that would include connections that are no longer active but haven't been released back. Threads waiting would (I assume) mean the number of HTTP requests you've made where no free connection is available, and thus the request is hung in ClientConnectionRequest.getConnection() until it gets a connection to use, or it times out. There must be some way to find this out, but it's not obvious from the APIs. I guess the ThreadSafeClientConnManager is the replacement of the MultiThreadedHttpConnectionManager? Yes, I believe so. Or at least that's what I've changed my 3.1 code to use with 4.0. -- Ken - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
RE: Http trace for cookies
Hello Oleg, Then, how is it possible that I always receive -1 in maxAge attribute? Am I forgetting something, or doing something wrong? If I try to set the MAX_AGE_ATTR and the EXPIRES_ATTR, I get -1 again. How is the correct way to propagate the cookie expiry date? Thanks in advance, Joan. Date expires = new Date(System.currentTimeMillis() + 6); BasicClientCookie bcc = new BasicClientCookie(testName, testValue); bcc.setVersion(1); bcc.setDomain(wstest.rhodasol.es); bcc.setPath(/wsserhs/rhodasol); bcc.setSecure(false); bcc.setExpiryDate(expires); bcc.setAttribute(ClientCookie.VERSION_ATTR, 1); bcc.setAttribute(ClientCookie.DOMAIN_ATTR, wstest.rhodasol.es); bcc.setAttribute(ClientCookie.PATH_ATTR, /wsserhs/rhodasol); bcc.setAttribute(ClientCookie.MAX_AGE_ATTR, 60); bcc.setAttribute(ClientCookie.EXPIRES_ATTR, DateUtils.formatDate(expires)); -Mensaje original- De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: miércoles, 08 de julio de 2009 23:54 Para: HttpClient User Discussion Asunto: Re: Http trace for cookies Joan Balagueró Valls wrote: Hello Oleg, Im trying to send you the trace, but your mail server does not accept it. I paste here a couple of lines of this trace: This is the interesting part of the trace. The expiry date is correct in the line 7 (60 seconds from now). But it seems this expiry date does not appear in Cookie header. Joan. CookieSpec selected: best-match Cookie [version: 1][name: testName][value: testValue][domain: wstest.rhodasol.es][path: /wsserhs/rhodasol][expiry: Wed Jul 08 16:22:39 CEST 2009] match [wstest.rhodasol.es:81/wsserhs/rhodasol] Sending request: POST /wsserhs/rhodasol HTTP/1.1 POST /wsserhs/rhodasol HTTP/1.1 Accept-Encoding: gzip Content-Length: 444 Content-Type: text/xml Host: wstest.rhodasol.es:81 Connection: Keep-Alive Cookie: $Version=1; testName=testValue; $Path=/wsserhs/rhodasol; $Domain=wstest.rhodasol.es Cookie2: $Version=1 Receiving response: HTTP/1.1 200 OK HTTP/1.1 200 OK I am sorry, Joan, but I see nothing wrong with the cookie. The cookie looks perfectly valid to me. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Using HttpClient with my JSP session
Morning All I'm hoping someone can help me out here. I have a JSP running on JBoss, that all works fine. I need from within that application to use HttpClient to request a JSP page within the same application, but so I get the resulting HTML in a string that I can then use as the body for an HTML Email. My problem is that I cannot figure out, or find on Google, a clear and working example on how I make the HttpClient request use the same session as my JSP is using. I know it isn't because the session filter checks for certain things on the session, and redirects to the login page if they're missing. Thanks Tom -- View this message in context: http://www.nabble.com/Using-HttpClient-with-my-JSP-session-tp24406615p24406615.html Sent from the HttpClient-User mailing list archive at Nabble.com.
Re: mix logs when capturing httpclient.wire traces in multi-threaded application
On Wed, Jul 08, 2009 at 03:46:55PM -0700, mikkin wrote: Oleg, Thanks for the reply. Actually we would typically like each thread to have its wire trace recorded into its own file. I do not think this is possible, but you may want to consult the log4j folks for a more authoritive answer. Oleg Let me explain with eg: Thread 10 will capture wire trace of class A and store the wire trace in File A Thread 11 will capture wire trace of class B and store the wire trace in File B. Both Thread 10 and Thread 11 will be running simultaneously ofcourse. We looked into the source code a while back and realized that Wire.java has a static HEADER_WIRE and CONTENT_WIRE fields and a private constructor. So could it be that when we do Logger.getLogger(httpclient.wire) in our constructor both Thread 10 and Thread 11 in the above example are returned with the same instance of the Wire log and thus causing the logs to get mixed up? Currently we run 30 such threads and the output files are a disaster :) Also, just as an FYI, we are using httpclient 3.x. Mikkin -- View this message in context: http://www.nabble.com/mix-logs-when-capturing-httpclient.wire-traces-in-multi-threaded-application-tp24397094p24400772.html Sent from the HttpClient-User mailing list archive at Nabble.com. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: HttpClient 4 and self-signed certificate
On Wed, Jul 08, 2009 at 04:19:51PM -0700, bo wrote: Hi Oleg, HttpClient 3 had good examples on how to customize SocketFactory to allow for self-signed certificate. I tried to update these for 4.0 but run into Unconnected Sockets not implemented error. All I'm trying to do is to use HttpClient to run very basic POST authentication on the site that has self-signed certificate. Any suggestions or code examples on how to do it in v. 4 would be greatly appreciated Thanks, Bo Bo, Please have a look at the HttpClient tutorial: http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e497 Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: some detailed migration questions...
On Wed, Jul 08, 2009 at 04:53:36PM -0700, Ken Krugler wrote: Since Michael wanted to change the user agent on a per-request basis, what would be the appropriate way to do this using http.useragent? In my code, I use this (via HttpParams) to configure the ThreadSafeClientConnManager that is then used when creating my DefaultHttpClient. This seemed like a static one-time setup. Is there a way to use HttpParams to change values on a per-request basis? Virtually any parameter in HttpClient 4.0 can be overridden on the HTTP request level. http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e316 Hope this helps Oleg [snip] Thanks, -- Ken -- Ken Krugler +1 530-210-6378 - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
performance tuning possibilities?
I completed my migration to the latest version of the HttpClient successfully. It is just a bit slower now than I would have ecpected... Maybe somebody has some tuning experiences to share? * Reliability is not that important for me * It should do around 20 accesses per minute using slow proxies, connection timeout is set to 6 seconds for them. Currently it does those 20 accesses in 100 seconds :-( * I have 3 threads executing http gets concurrently. * I am going with ThreadSafeClientConnManager and 30 connections at max. Is that enough? Or too much? * I use a retry hanlder with up to 6 retries. Not sure if that is eating up much time? * TCPNoDelay is false. Would turning that on improve performance to some extent? How much? * Stale Checking is enabled. Maybe I dont need this since I use the IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every 5 seconds? * Are there more tuning opportunities available? Thx for helping! Michael
Re: performance tuning possibilities?
On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote: I completed my migration to the latest version of the HttpClient successfully. It is just a bit slower now than I would have ecpected... Maybe somebody has some tuning experiences to share? Michael, HttpClient should be _at least_ as fast as HttpClient 3.x. If performance is an overriding factor for you, you should consider using HttpCore directly and get 30 to 40% performance boost. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore * Reliability is not that important for me Make sure to turn off stale connection check and expect-continue handshake. * It should do around 20 accesses per minute using slow proxies, connection timeout is set to 6 seconds for them. Currently it does those 20 accesses in 100 seconds :-( To find out where time is being spent, turn on context logging. Make sure to disable the wire log, though, otherwise you will be measuring performance of console output. http://hc.apache.org/httpcomponents-client/logging.html * I have 3 threads executing http gets concurrently. * I am going with ThreadSafeClientConnManager and 30 connections at max. Is that enough? Or too much? You also have to increase the default limit of 2 concurrent connections per host. It should be at least the same as the number of worker thread to ensure minimal resource contension. * I use a retry hanlder with up to 6 retries. Not sure if that is eating up much time? * TCPNoDelay is false. Would turning that on improve performance to some extent? How much? Wha! Nagle's algorithm just kills HTTP performance. Make sure the TCPNoDelay is set to true. * Stale Checking is enabled. Maybe I dont need this since I use the IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every 5 seconds? The stale connection check is more evil than the Church of Scientology. Just do not use it. Hope this helps Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
EXPECT-CONTINUE HANDSHAKE
Hello Oleg, Can I set the expect-continue handshake at httpclient level? HttpParams objHttpParams = new BasicHttpParams(); HttpProtocolParams.setVersion(objHttpParams, HttpVersion.HTTP_1_1); ClientConnectionManager cm = new ThreadSafeClientConnManager(this.objHttpParams, HttpUtils.createDefaultSchemeRegistry()); this.objHttp = new DefaultHttpClient(cm, this.objHttpParams); -- this.objHttpParams.setParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, Boolean.FALSE); Or should I set at request level? Thanks, Joan. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: performance tuning possibilities?
OK, Oleg's suggestions improved the whole thing around 30%, great stuff! Where would the performance boost with using the Core directly come from? 2009/7/9 Oleg Kalnichevski ol...@apache.org On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote: I completed my migration to the latest version of the HttpClient successfully. It is just a bit slower now than I would have ecpected... Maybe somebody has some tuning experiences to share? Michael, HttpClient should be _at least_ as fast as HttpClient 3.x. If performance is an overriding factor for you, you should consider using HttpCore directly and get 30 to 40% performance boost. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore * Reliability is not that important for me Make sure to turn off stale connection check and expect-continue handshake. * It should do around 20 accesses per minute using slow proxies, connection timeout is set to 6 seconds for them. Currently it does those 20 accesses in 100 seconds :-( To find out where time is being spent, turn on context logging. Make sure to disable the wire log, though, otherwise you will be measuring performance of console output. http://hc.apache.org/httpcomponents-client/logging.html * I have 3 threads executing http gets concurrently. * I am going with ThreadSafeClientConnManager and 30 connections at max. Is that enough? Or too much? You also have to increase the default limit of 2 concurrent connections per host. It should be at least the same as the number of worker thread to ensure minimal resource contension. * I use a retry hanlder with up to 6 retries. Not sure if that is eating up much time? * TCPNoDelay is false. Would turning that on improve performance to some extent? How much? Wha! Nagle's algorithm just kills HTTP performance. Make sure the TCPNoDelay is set to true. * Stale Checking is enabled. Maybe I dont need this since I use the IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every 5 seconds? The stale connection check is more evil than the Church of Scientology. Just do not use it. Hope this helps Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
