Re: some detailed migration questions...
Hi Oleg, thanks for the answers! I realized there is some very good documentation available which I just did not find, because it is linked only from the HttpCommons Core and not from the Client... Michael 2009/7/9 Oleg Kalnichevski > Ken Krugler wrote: > >> A few answers in-line below. >> >> Disclaimer: I've ported some HttpClient 3.1 code to 4.0, but I haven't >> made heavy use of the full 4.0 functionality. >> >> I am new to HTTPComponents Client but have been using HttpClient 3.1 very >>> extensively... So I am facing some migration work. >>> >>> 1. How do I measure the time it took to get a response in a multithreaded >>> scenario with a limited size of max total connections? >>> >>> In 3.1 I had to hack this into a custom ProtocolSocketFactory. When the >>> socket is created I am rembering the time in a thread local scope. >>> >>> >> > >> Is that still the way to go? >>> >>> > Michael, > > I would recommend using custom protocol interceptors to collect performance > related metrics. Use a request interceptor to store the start time in the > HTTP context, and a response interceptor to store the finish time. For > details see: > > > http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e290 > > >>> 2. I want to set a different user agent per request >>> >>> In 3.1 I could set HostParams when executing the method. How does it work >>> now? >>> >> >> I believe you'd want to implement a RequestUserAgent class that sets up >> the user agent that you want. >> >> Seems like you might be able to do the same thing via using an >> HttpContext, but Oleg would know best. >> >> > One could also use the 'http.useragent' parameter: > > > http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e337 > > 3. Setting a proxy per request >>> >>> httpClient.execute(proxyHost, httpget, responseHandler); >>> >>> Correct? >>> >>> > Use 'http.route.default-proxy' parameter. > > http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e465 > http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e540 > > 4. Disabling redirects >>> >>> params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, false); >>> >>> Correct? >>> >>> > Yes. > > 5. Enabling my own retry handler (which I just need because of verbose >>> logging) >>> >>> ((DefaultHttpClient) this.httpClient).setHttpRequestRetryHandler(XXX); >>> >>> Correct? >>> >>> > Yes. > > 6. When using the ThreadSafeClientConnManager how can I monitor the number >>> of threads used or threads waiting? >>> >> >> Well, you can call getConnectionsInPool() to get the number of in-use >> connections, though I think that would include connections that are no >> longer active but haven't been released back. >> >> "Threads waiting" would (I assume) mean the number of HTTP requests you've >> made where no free connection is available, and thus the request is hung in >> ClientConnectionRequest.getConnection() until it gets a connection to use, >> or it times out. >> >> There must be some way to find this out, but it's not obvious from the >> APIs. >> >> I guess the ThreadSafeClientConnManager is the replacement of the >>> MultiThreadedHttpConnectionManager? >>> >> >> Yes, I believe so. Or at least that's what I've changed my 3.1 code to use >> with 4.0. >> >> -- Ken >> > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > >
RE: Http trace for cookies
Hello Oleg,
Then, how is it possible that I always receive -1 in "maxAge" attribute? Am
I forgetting something, or doing something wrong?
If I try to set the MAX_AGE_ATTR and the EXPIRES_ATTR, I get -1 again.
How is the correct way to propagate the cookie expiry date?
Thanks in advance,
Joan.
Date expires = new Date(System.currentTimeMillis() + 6);
BasicClientCookie bcc = new BasicClientCookie("testName", "testValue");
bcc.setVersion(1);
bcc.setDomain("wstest.rhodasol.es");
bcc.setPath("/wsserhs/rhodasol");
bcc.setSecure(false);
bcc.setExpiryDate(expires);
bcc.setAttribute(ClientCookie.VERSION_ATTR, "1");
bcc.setAttribute(ClientCookie.DOMAIN_ATTR, "wstest.rhodasol.es");
bcc.setAttribute(ClientCookie.PATH_ATTR, "/wsserhs/rhodasol");
bcc.setAttribute(ClientCookie.MAX_AGE_ATTR, "60");
bcc.setAttribute(ClientCookie.EXPIRES_ATTR, DateUtils.formatDate(expires));
-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org]
Enviado el: miércoles, 08 de julio de 2009 23:54
Para: HttpClient User Discussion
Asunto: Re: Http trace for cookies
Joan Balagueró Valls wrote:
> Hello Oleg,
>
>
>
> Im trying to send you the trace, but your mail server does not accept it.
>
>
>
> I paste here a couple of lines of this trace:
>
>
>
> This is the interesting part of the trace. The expiry date is correct in
the
> line 7 (60 seconds from now). But it seems this expiry date does not
appear
> in Cookie header.
>
>
>
> Joan.
>
>
>
>
>
> CookieSpec selected: best-match
>
> Cookie [version: 1][name: testName][value: testValue][domain:
> wstest.rhodasol.es][path: /wsserhs/rhodasol][expiry: Wed Jul 08 16:22:39
> CEST 2009] match [wstest.rhodasol.es:81/wsserhs/rhodasol]
>
>
>
> Sending request: POST /wsserhs/rhodasol HTTP/1.1
>
> POST /wsserhs/rhodasol HTTP/1.1
>
> Accept-Encoding: gzip
>
> Content-Length: 444
>
> Content-Type: text/xml
>
> Host: wstest.rhodasol.es:81
>
> Connection: Keep-Alive
>
> Cookie: $Version=1; testName="testValue"; $Path="/wsserhs/rhodasol";
> $Domain="wstest.rhodasol.es"
>
> Cookie2: $Version=1
>
> Receiving response: HTTP/1.1 200 OK
>
> HTTP/1.1 200 OK
>
>
I am sorry, Joan, but I see nothing wrong with the cookie. The cookie
looks perfectly valid to me.
Oleg
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Using HttpClient with my JSP session
Morning All I'm hoping someone can help me out here. I have a JSP running on JBoss, that all works fine. I need from within that application to use HttpClient to request a JSP page within the same application, but so I get the resulting HTML in a string that I can then use as the body for an HTML Email. My problem is that I cannot figure out, or find on Google, a clear and working example on how I make the HttpClient request use the same session as my JSP is using. I know it isn't because the session filter checks for certain things on the session, and redirects to the login page if they're missing. Thanks Tom -- View this message in context: http://www.nabble.com/Using-HttpClient-with-my-JSP-session-tp24406615p24406615.html Sent from the HttpClient-User mailing list archive at Nabble.com.
Re: mix logs when capturing httpclient.wire traces in multi-threaded application
On Wed, Jul 08, 2009 at 03:46:55PM -0700, mikkin wrote:
>
> Oleg,
>
> Thanks for the reply.
>
> Actually we would typically like each thread to have its wire trace recorded
> into its own file.
I do not think this is possible, but you may want to consult the log4j folks
for a more authoritive answer.
Oleg
Let me explain with eg:
>
> Thread 10 will capture wire trace of class A and store the wire trace in
> File A
> Thread 11 will capture wire trace of class B and store the wire trace in
> File B.
>
> Both Thread 10 and Thread 11 will be running simultaneously ofcourse. We
> looked into the source code a while back and realized that Wire.java has a
> static HEADER_WIRE and CONTENT_WIRE fields and a private constructor. So
> could it be that when we do Logger.getLogger("httpclient.wire") in our
> constructor both Thread 10 and Thread 11 in the above example are returned
> with the same instance of the Wire log and thus causing the logs to get
> mixed up?
>
> Currently we run 30 such threads and the output files are a disaster :)
>
> Also, just as an FYI, we are using httpclient 3.x.
>
> Mikkin
> --
> View this message in context:
> http://www.nabble.com/mix-logs-when-capturing-httpclient.wire-traces-in-multi-threaded-application-tp24397094p24400772.html
> Sent from the HttpClient-User mailing list archive at Nabble.com.
>
>
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: HttpClient 4 and self-signed certificate
On Wed, Jul 08, 2009 at 04:19:51PM -0700, bo wrote: > Hi Oleg, > > HttpClient 3 had good examples on how to customize SocketFactory to allow > for self-signed certificate. I tried to update these for 4.0 but run into > "Unconnected Sockets not implemented" error. All I'm trying to do is to use > HttpClient to run very basic POST authentication on the site that has > self-signed certificate. Any suggestions or code examples on how to do it in > v. 4 would be greatly appreciated > > Thanks, > > Bo Bo, Please have a look at the HttpClient tutorial: http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e497 Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: some detailed migration questions...
On Wed, Jul 08, 2009 at 04:53:36PM -0700, Ken Krugler wrote: > > Since Michael wanted to change the user agent on a per-request basis, > what would be the appropriate way to do this using http.useragent? > > In my code, I use this (via HttpParams) to configure the > ThreadSafeClientConnManager that is then used when creating my > DefaultHttpClient. > > This seemed like a static one-time setup. Is there a way to use > HttpParams to change values on a per-request basis? > Virtually any parameter in HttpClient 4.0 can be overridden on the HTTP request level. http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e316 Hope this helps Oleg > [snip] > > Thanks, > > -- Ken > -- > Ken Krugler > +1 530-210-6378 > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Http trace for cookies
On Thu, Jul 09, 2009 at 10:37:28AM +0200, Joan Balaguer? Valls wrote:
> Hello Oleg,
>
> Then, how is it possible that I always receive -1 in "maxAge" attribute? Am
> I forgetting something, or doing something wrong?
>
Joan
Why is this a problem? That basically means that the cookie max age is
undefined, so it should be kept for the duration of the HTTP session only.
> If I try to set the MAX_AGE_ATTR and the EXPIRES_ATTR, I get -1 again.
> How is the correct way to propagate the cookie expiry date?
>
Max-age attributes can be set on the Set-Cookie headers only. HTTP user agents
are not supposed to send the max-age value back to the origin server.
http://www.faqs.org/rfcs/rfc2965.html
Hope this helps
Oleg
> Thanks in advance,
>
> Joan.
>
>
> Date expires = new Date(System.currentTimeMillis() + 6);
>
> BasicClientCookie bcc = new BasicClientCookie("testName", "testValue");
> bcc.setVersion(1);
> bcc.setDomain("wstest.rhodasol.es");
> bcc.setPath("/wsserhs/rhodasol");
> bcc.setSecure(false);
> bcc.setExpiryDate(expires);
>
> bcc.setAttribute(ClientCookie.VERSION_ATTR, "1");
> bcc.setAttribute(ClientCookie.DOMAIN_ATTR, "wstest.rhodasol.es");
> bcc.setAttribute(ClientCookie.PATH_ATTR, "/wsserhs/rhodasol");
> bcc.setAttribute(ClientCookie.MAX_AGE_ATTR, "60");
> bcc.setAttribute(ClientCookie.EXPIRES_ATTR, DateUtils.formatDate(expires));
>
>
>
> -Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org]
> Enviado el: mi?rcoles, 08 de julio de 2009 23:54
> Para: HttpClient User Discussion
> Asunto: Re: Http trace for cookies
>
> Joan Balaguer? Valls wrote:
> > Hello Oleg,
> >
> >
> >
> > I?m trying to send you the trace, but your mail server does not accept it.
> >
> >
> >
> > I paste here a couple of lines of this trace:
> >
> >
> >
> > This is the interesting part of the trace. The expiry date is correct in
> the
> > line 7 (60 seconds from now). But it seems this expiry date does not
> appear
> > in Cookie header.
> >
> >
> >
> > Joan.
> >
> >
> >
> >
> >
> > CookieSpec selected: best-match
> >
> > Cookie [version: 1][name: testName][value: testValue][domain:
> > wstest.rhodasol.es][path: /wsserhs/rhodasol][expiry: Wed Jul 08 16:22:39
> > CEST 2009] match [wstest.rhodasol.es:81/wsserhs/rhodasol]
> >
> >
> >
> > Sending request: POST /wsserhs/rhodasol HTTP/1.1
> >
> > POST /wsserhs/rhodasol HTTP/1.1
> >
> > Accept-Encoding: gzip
> >
> > Content-Length: 444
> >
> > Content-Type: text/xml
> >
> > Host: wstest.rhodasol.es:81
> >
> > Connection: Keep-Alive
> >
> > Cookie: $Version=1; testName="testValue"; $Path="/wsserhs/rhodasol";
> > $Domain="wstest.rhodasol.es"
> >
> > Cookie2: $Version=1
> >
> > Receiving response: HTTP/1.1 200 OK
> >
> > HTTP/1.1 200 OK
> >
> >
>
> I am sorry, Joan, but I see nothing wrong with the cookie. The cookie
> looks perfectly valid to me.
>
> Oleg
>
>
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
>
>
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
performance tuning possibilities?
I completed my migration to the latest version of the HttpClient successfully. It is just a bit slower now than I would have ecpected... Maybe somebody has some tuning experiences to share? * Reliability is not that important for me * It should do around 20 accesses per minute using slow proxies, connection timeout is set to 6 seconds for them. Currently it does those 20 accesses in 100 seconds :-( * I have 3 threads executing http gets concurrently. * I am going with ThreadSafeClientConnManager and 30 connections at max. Is that enough? Or too much? * I use a retry hanlder with up to 6 retries. Not sure if that is eating up much time? * TCPNoDelay is false. Would turning that on improve performance to some extent? How much? * Stale Checking is enabled. Maybe I dont need this since I use the IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every 5 seconds? * Are there more tuning opportunities available? Thx for helping! Michael
Re: Using HttpClient with my JSP session
On Thu, Jul 09, 2009 at 02:02:02AM -0700, TomSimmons wrote: > > Morning All > > I'm hoping someone can help me out here. > > I have a JSP running on JBoss, that all works fine. I need from within that > application to use HttpClient to request a JSP page within the same > application, but so I get the resulting HTML in a string that I can then use > as the body for an HTML Email. > > My problem is that I cannot figure out, or find on Google, a clear and > working example on how I make the HttpClient request use the same session as > my JSP is using. > > I know it isn't because the session filter checks for certain things on the > session, and redirects to the login page if they're missing. > > > Thanks > > Tom Hi Tom, You need to manually add the session cookie to the cookie store bound to the execution context and make sure it gets picked up when requests are generated by setting appropriate domain and path attribute values http://hc.apache.org/httpcomponents-client/tutorial/html/ch03.html#d4e672 Oleg > -- > View this message in context: > http://www.nabble.com/Using-HttpClient-with-my-JSP-session-tp24406615p24406615.html > Sent from the HttpClient-User mailing list archive at Nabble.com. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: performance tuning possibilities?
On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote: > I completed my migration to the latest version of the HttpClient > successfully. It is just a bit slower now than I would have ecpected... > > Maybe somebody has some tuning experiences to share? > Michael, HttpClient should be _at least_ as fast as HttpClient 3.x. If performance is an overriding factor for you, you should consider using HttpCore directly and get 30 to 40% performance boost. http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore > * Reliability is not that important for me Make sure to turn off stale connection check and expect-continue handshake. > * It should do around 20 accesses per minute using slow proxies, connection > timeout is set to 6 seconds for them. Currently it does those 20 accesses in > 100 seconds :-( > To find out where time is being spent, turn on context logging. Make sure to disable the wire log, though, otherwise you will be measuring performance of console output. http://hc.apache.org/httpcomponents-client/logging.html > > * I have 3 threads executing http gets concurrently. > * I am going with ThreadSafeClientConnManager and 30 connections at max. Is > that enough? Or too much? You also have to increase the default limit of 2 concurrent connections per host. It should be at least the same as the number of worker thread to ensure minimal resource contension. > * I use a retry hanlder with up to 6 retries. Not sure if that is eating up > much time? > * TCPNoDelay is false. Would turning that on improve performance to some > extent? How much? Wha! Nagle's algorithm just kills HTTP performance. Make sure the TCPNoDelay is set to true. > * Stale Checking is enabled. Maybe I dont need this since I use the > IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every 5 > seconds? > The stale connection check is more evil than the Church of Scientology. Just do not use it. Hope this helps Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
EXPECT-CONTINUE HANDSHAKE
Hello Oleg, Can I set the expect-continue handshake at httpclient level? HttpParams objHttpParams = new BasicHttpParams(); HttpProtocolParams.setVersion(objHttpParams, HttpVersion.HTTP_1_1); ClientConnectionManager cm = new ThreadSafeClientConnManager(this.objHttpParams, HttpUtils.createDefaultSchemeRegistry()); this.objHttp = new DefaultHttpClient(cm, this.objHttpParams); --> this.objHttpParams.setParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, Boolean.FALSE); Or should I set at request level? Thanks, Joan. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: performance tuning possibilities?
OK, Oleg's suggestions improved the whole thing around 30%, great stuff! Where would the performance boost with using the Core directly come from? 2009/7/9 Oleg Kalnichevski > On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote: > > I completed my migration to the latest version of the HttpClient > > successfully. It is just a bit slower now than I would have ecpected... > > > > Maybe somebody has some tuning experiences to share? > > > > Michael, > > HttpClient should be _at least_ as fast as HttpClient 3.x. If performance > is an > overriding factor for you, you should consider using HttpCore directly and > get > 30 to 40% performance boost. > > http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore > > > > > * Reliability is not that important for me > > Make sure to turn off stale connection check and expect-continue handshake. > > > * It should do around 20 accesses per minute using slow proxies, > connection > > timeout is set to 6 seconds for them. Currently it does those 20 accesses > in > > 100 seconds :-( > > > > To find out where time is being spent, turn on context logging. Make sure > to > disable the wire log, though, otherwise you will be measuring performance > of > console output. > > http://hc.apache.org/httpcomponents-client/logging.html > > > > > > * I have 3 threads executing http gets concurrently. > > * I am going with ThreadSafeClientConnManager and 30 connections at max. > Is > > that enough? Or too much? > > You also have to increase the default limit of 2 concurrent connections per > host. It should be at least the same as the number of worker thread to > ensure > minimal resource contension. > > > > * I use a retry hanlder with up to 6 retries. Not sure if that is eating > up > > much time? > > * TCPNoDelay is false. Would turning that on improve performance to some > > extent? How much? > > Wha! Nagle's algorithm just kills HTTP performance. Make sure the > TCPNoDelay is set to true. > > > * Stale Checking is enabled. Maybe I dont need this since I use the > > IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every > 5 > > seconds? > > > > The stale connection check is more evil than the Church of Scientology. > Just do > not use it. > > Hope this helps > > Oleg > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > >
