Re: Incredibly slow gzip downloads
On Sat, 2014-02-15 at 17:06 +0100, Sebastiano Vigna wrote: > On 15 Feb 2014, at 12:47 PM, Oleg Kalnichevski wrote: > > > The problem mostly likely has been introduced by HTTPCLIENT-1432 [1]. I > > reviewed the patch once more and could not find anything obviously wrong > > with it. Try reverting the changes introduced by HTTPCLIENT-1432 and see > > Subclassing InputStream without overriding the multi-byte read() method is a > recipe for disaster... the inherited method does a byte-by-byte read. You can > see what's happening in this hprof trace: > > java.util.zip.Inflater.inflateBytes(Inflater.java:Unknown line) > java.util.zip.Inflater.inflate(Inflater.java:259) > java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152) > java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116) > java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122) > > org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:56) > java.io.InputStream.read(InputStream.java:179) > > it.unimi.di.law.warc.util.InspectableCachedHttpEntity.copyContent(InspectableCachedHttpEntity.java:67) > > copyContent() would love to read(byte[],int,int) in a buffer, but since > LazyDecompressingInputStream doesn't override it it invokes instead the > read-byte-by-byte inherited method in InputStream, which in turn now calls > for each byte the one-byte read() method from LazyDecompressingInputStream, > which invokes the one-byte read method from InflaterInputStream, which does a > multi-byte, length-one read from GZIPInputStream, which unleashes a similar > call on InflaterInputStream, which unfortunately makes a similar read using > the native inflateBytes() method. > > The result is a 10-50x decrease in speed, but I think that a trivial override > of read(byte[],int,int) in LazyDecompressingInputStream will solve the > problem. > Sebastiano Could you please raise a JIRA for this defect and tag it as regression? Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Incredibly slow gzip downloads
On 15 Feb 2014, at 12:47 PM, Oleg Kalnichevski wrote: > The problem mostly likely has been introduced by HTTPCLIENT-1432 [1]. I > reviewed the patch once more and could not find anything obviously wrong > with it. Try reverting the changes introduced by HTTPCLIENT-1432 and see Subclassing InputStream without overriding the multi-byte read() method is a recipe for disaster... the inherited method does a byte-by-byte read. You can see what's happening in this hprof trace: java.util.zip.Inflater.inflateBytes(Inflater.java:Unknown line) java.util.zip.Inflater.inflate(Inflater.java:259) java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152) java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116) java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122) org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:56) java.io.InputStream.read(InputStream.java:179) it.unimi.di.law.warc.util.InspectableCachedHttpEntity.copyContent(InspectableCachedHttpEntity.java:67) copyContent() would love to read(byte[],int,int) in a buffer, but since LazyDecompressingInputStream doesn't override it it invokes instead the read-byte-by-byte inherited method in InputStream, which in turn now calls for each byte the one-byte read() method from LazyDecompressingInputStream, which invokes the one-byte read method from InflaterInputStream, which does a multi-byte, length-one read from GZIPInputStream, which unleashes a similar call on InflaterInputStream, which unfortunately makes a similar read using the native inflateBytes() method. The result is a 10-50x decrease in speed, but I think that a trivial override of read(byte[],int,int) in LazyDecompressingInputStream will solve the problem. Ciao, seba - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
On Sat, 2014-02-15 at 16:01 +0100, Philippe Mouawad wrote:
> By default we disable retry but it can be controlled by a parameter.
>
I have say you may want to have a retry handler enabled by default if
the stale check is off. At the very least you may want to retry
idempotent requests or some specific type of exceptions (like 'server
failed to respond').
Oleg
> Thanks for the info on ConnectionReuseStrategy, it will be a way to handle
> these kind of issues.
>
> Regards
>
>
> On Sat, Feb 15, 2014 at 3:12 PM, Oleg Kalnichevski
>
> > wrote:
>
> > On Sat, 2014-02-15 at 14:46 +0100, Philippe Mouawad wrote:
> > > Hello Oleg,
> > > The problem is that is seems in this particular case, removing this check
> > > it leads to 50% error.
> > >
> >
> > What kind of HttpRequestRetryHandler implementation does JMeter 2.11
> > use? Usually with the stale check disabled one should want to retry at
> > least some requests automatically.
> >
> > > I could be explained by a max number of requests per connection setting,
> > as
> > > per their documentation:
> > > - http://aws.amazon.com/articles/1904
> > >
> > > "Also, don't overuse a connection. Amazon S3 will accept up to 100
> > requests
> > > before it closes a connection (resulting in 'connection reset'). Rather
> > > than having this happen, use a connection for 80-90 requests before
> > closing
> > > and re-opening a new connection."
> > >
> > > Is there by the way a configuration parameter in HttpClient to limit the
> > > number of requests per connection ?
> > >
> >
> > There is no parameter but one can use a custom ConnectionReuseStrategy
> > to that effect
> >
> > ---
> > ConnectionReuseStrategy reuseStrategy = new
> > DefaultConnectionReuseStrategy() {
> >
> > @Override
> > public boolean keepAlive(
> > final HttpResponse response, final HttpContext context) {
> > HttpConnection conn = (HttpConnection)
> > context.getAttribute(HttpClientContext.HTTP_CONNECTION);
> > long count = conn.getMetrics().getRequestCount();
> > if (count >= 100) {
> > return false;
> > }
> > return super.keepAlive(response, context);
> > }
> >
> > };
> > ---
> >
> > Hope this helps
> >
> > Oleg
> >
> > > Thanks
> > > Regards
> > > Philippe
> > >
> > >
> > >
> > > On Sat, Feb 15, 2014 at 12:49 PM, Oleg Kalnichevski
> > >
> > >wrote:
> > >
> > > > On Sat, 2014-02-15 at 11:23 +0100, Philippe Mouawad wrote:
> > > > > Hello ,
> > > > > yes that's it.
> > > > >
> > > > > regards
> > > > >
> > > >
> > > > Then, I am not sure I understand the problem. The stale connection
> > check
> > > > is about trading off some performance for fewer i/o errors or visa
> > > > versa.
> > > >
> > > > Oleg
> > > >
> > > > > On Saturday, February 15, 2014, Oleg Kalnichevski <
> > > > o...@ok2consulting.com
> > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote:
> > > > > > > Hello Oleg,
> > > > > > > We set this configuration in JMeter 2.11, we got recently this
> > bug
> > > > report
> > > > > > > related to this change:
> > > > > > > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119
> > > > > > >
> > > > > > > This issue seems to be faced by another person:
> > > > > > > https://twitter.com/cfwhisperer/status/428278488349417472
> > > > > > >
> > > > > > > Regards
> > > > > > > Philippe
> > > > > > >
> > > > > >
> > > > > > Philippe
> > > > > >
> > > > > > I am not sure I remember the context. Is it about turning off stale
> > > > > > connection checking?
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski <
> > ol...@apache.org
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote:
> > > > > > > > > Hello Oleg,
> > > > > > > > > Thanks for answer.
> > > > > > > > > Wouldn't it be interesting to add some threshold or delay
> > between
> > > > > > each
> > > > > > > > > check instead of doing check on each request ?
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Philippe
> > > > > > > > >
> > > > > > > >
> > > > > > > > Philippe
> > > > > > > >
> > > > > > > > I am not sure it is worth the trouble. In most circumstances
> > the
> > > > stale
> > > > > > > > connection check should be turned off anyway.
> > > > > > > >
> > > > > > > > I think it just needs to be better documented.
> > > > > > > >
> > > > > > > > Oleg
> > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski <
> > > > ol...@apache.org
> >
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote:
> > > > > > > > > > > Hello,
> > > > > > > > > > > Profiling JMeter, I noticed an important number of
> > > > > > > > SocketTimeoutException
> > > > > > > > > > > being triggered without any impact on response status.
> > > > > > > > > >
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
Also remember that JMeter has a "Use keepalive" option on the HTTP Sampler GUI.
If unchecked, JMeter will send Connection: close (or equivalent) as
part of the request.
[This does not work reliably for Java HTTP because the connections may
be pooled]
On 15 February 2014 15:01, Philippe Mouawad wrote:
> By default we disable retry but it can be controlled by a parameter.
>
> Thanks for the info on ConnectionReuseStrategy, it will be a way to handle
> these kind of issues.
>
> Regards
>
>
> On Sat, Feb 15, 2014 at 3:12 PM, Oleg Kalnichevski
>
>> wrote:
>
>> On Sat, 2014-02-15 at 14:46 +0100, Philippe Mouawad wrote:
>> > Hello Oleg,
>> > The problem is that is seems in this particular case, removing this check
>> > it leads to 50% error.
>> >
>>
>> What kind of HttpRequestRetryHandler implementation does JMeter 2.11
>> use? Usually with the stale check disabled one should want to retry at
>> least some requests automatically.
>>
>> > I could be explained by a max number of requests per connection setting,
>> as
>> > per their documentation:
>> > - http://aws.amazon.com/articles/1904
>> >
>> > "Also, don't overuse a connection. Amazon S3 will accept up to 100
>> requests
>> > before it closes a connection (resulting in 'connection reset'). Rather
>> > than having this happen, use a connection for 80-90 requests before
>> closing
>> > and re-opening a new connection."
>> >
>> > Is there by the way a configuration parameter in HttpClient to limit the
>> > number of requests per connection ?
>> >
>>
>> There is no parameter but one can use a custom ConnectionReuseStrategy
>> to that effect
>>
>> ---
>> ConnectionReuseStrategy reuseStrategy = new
>> DefaultConnectionReuseStrategy() {
>>
>> @Override
>> public boolean keepAlive(
>> final HttpResponse response, final HttpContext context) {
>> HttpConnection conn = (HttpConnection)
>> context.getAttribute(HttpClientContext.HTTP_CONNECTION);
>> long count = conn.getMetrics().getRequestCount();
>> if (count >= 100) {
>> return false;
>> }
>> return super.keepAlive(response, context);
>> }
>>
>> };
>> ---
>>
>> Hope this helps
>>
>> Oleg
>>
>> > Thanks
>> > Regards
>> > Philippe
>> >
>> >
>> >
>> > On Sat, Feb 15, 2014 at 12:49 PM, Oleg Kalnichevski
>> >
>> >wrote:
>> >
>> > > On Sat, 2014-02-15 at 11:23 +0100, Philippe Mouawad wrote:
>> > > > Hello ,
>> > > > yes that's it.
>> > > >
>> > > > regards
>> > > >
>> > >
>> > > Then, I am not sure I understand the problem. The stale connection
>> check
>> > > is about trading off some performance for fewer i/o errors or visa
>> > > versa.
>> > >
>> > > Oleg
>> > >
>> > > > On Saturday, February 15, 2014, Oleg Kalnichevski <
>> > > o...@ok2consulting.com
>> >
>> > > > wrote:
>> > > >
>> > > > >
>> > > > > On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote:
>> > > > > > Hello Oleg,
>> > > > > > We set this configuration in JMeter 2.11, we got recently this
>> bug
>> > > report
>> > > > > > related to this change:
>> > > > > > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119
>> > > > > >
>> > > > > > This issue seems to be faced by another person:
>> > > > > > https://twitter.com/cfwhisperer/status/428278488349417472
>> > > > > >
>> > > > > > Regards
>> > > > > > Philippe
>> > > > > >
>> > > > >
>> > > > > Philippe
>> > > > >
>> > > > > I am not sure I remember the context. Is it about turning off stale
>> > > > > connection checking?
>> > > > >
>> > > > > Oleg
>> > > > >
>> > > > > >
>> > > > > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski <
>> ol...@apache.org
>> > > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote:
>> > > > > > > > Hello Oleg,
>> > > > > > > > Thanks for answer.
>> > > > > > > > Wouldn't it be interesting to add some threshold or delay
>> between
>> > > > > each
>> > > > > > > > check instead of doing check on each request ?
>> > > > > > > >
>> > > > > > > > Regards
>> > > > > > > > Philippe
>> > > > > > > >
>> > > > > > >
>> > > > > > > Philippe
>> > > > > > >
>> > > > > > > I am not sure it is worth the trouble. In most circumstances
>> the
>> > > stale
>> > > > > > > connection check should be turned off anyway.
>> > > > > > >
>> > > > > > > I think it just needs to be better documented.
>> > > > > > >
>> > > > > > > Oleg
>> > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski <
>> > > ol...@apache.org
>>
>> > > > > >
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote:
>> > > > > > > > > > Hello,
>> > > > > > > > > > Profiling JMeter, I noticed an important number of
>> > > > > > > SocketTimeoutException
>> > > > > > > > > > being triggered without any impact on response status.
>> > > > > > > > > >
>> > > > > > > > > > I investigated it a bit deeper and find out it affected
>> only
>> > > > > > > HttpClient
>> > > > > > > > > > implem
HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
By default we disable retry but it can be controlled by a parameter.
Thanks for the info on ConnectionReuseStrategy, it will be a way to handle
these kind of issues.
Regards
On Sat, Feb 15, 2014 at 3:12 PM, Oleg Kalnichevski
> wrote:
> On Sat, 2014-02-15 at 14:46 +0100, Philippe Mouawad wrote:
> > Hello Oleg,
> > The problem is that is seems in this particular case, removing this check
> > it leads to 50% error.
> >
>
> What kind of HttpRequestRetryHandler implementation does JMeter 2.11
> use? Usually with the stale check disabled one should want to retry at
> least some requests automatically.
>
> > I could be explained by a max number of requests per connection setting,
> as
> > per their documentation:
> > - http://aws.amazon.com/articles/1904
> >
> > "Also, don't overuse a connection. Amazon S3 will accept up to 100
> requests
> > before it closes a connection (resulting in 'connection reset'). Rather
> > than having this happen, use a connection for 80-90 requests before
> closing
> > and re-opening a new connection."
> >
> > Is there by the way a configuration parameter in HttpClient to limit the
> > number of requests per connection ?
> >
>
> There is no parameter but one can use a custom ConnectionReuseStrategy
> to that effect
>
> ---
> ConnectionReuseStrategy reuseStrategy = new
> DefaultConnectionReuseStrategy() {
>
> @Override
> public boolean keepAlive(
> final HttpResponse response, final HttpContext context) {
> HttpConnection conn = (HttpConnection)
> context.getAttribute(HttpClientContext.HTTP_CONNECTION);
> long count = conn.getMetrics().getRequestCount();
> if (count >= 100) {
> return false;
> }
> return super.keepAlive(response, context);
> }
>
> };
> ---
>
> Hope this helps
>
> Oleg
>
> > Thanks
> > Regards
> > Philippe
> >
> >
> >
> > On Sat, Feb 15, 2014 at 12:49 PM, Oleg Kalnichevski
> >
> >wrote:
> >
> > > On Sat, 2014-02-15 at 11:23 +0100, Philippe Mouawad wrote:
> > > > Hello ,
> > > > yes that's it.
> > > >
> > > > regards
> > > >
> > >
> > > Then, I am not sure I understand the problem. The stale connection
> check
> > > is about trading off some performance for fewer i/o errors or visa
> > > versa.
> > >
> > > Oleg
> > >
> > > > On Saturday, February 15, 2014, Oleg Kalnichevski <
> > > o...@ok2consulting.com
> >
> > > > wrote:
> > > >
> > > > >
> > > > > On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote:
> > > > > > Hello Oleg,
> > > > > > We set this configuration in JMeter 2.11, we got recently this
> bug
> > > report
> > > > > > related to this change:
> > > > > > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119
> > > > > >
> > > > > > This issue seems to be faced by another person:
> > > > > > https://twitter.com/cfwhisperer/status/428278488349417472
> > > > > >
> > > > > > Regards
> > > > > > Philippe
> > > > > >
> > > > >
> > > > > Philippe
> > > > >
> > > > > I am not sure I remember the context. Is it about turning off stale
> > > > > connection checking?
> > > > >
> > > > > Oleg
> > > > >
> > > > > >
> > > > > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski <
> ol...@apache.org
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote:
> > > > > > > > Hello Oleg,
> > > > > > > > Thanks for answer.
> > > > > > > > Wouldn't it be interesting to add some threshold or delay
> between
> > > > > each
> > > > > > > > check instead of doing check on each request ?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Philippe
> > > > > > > >
> > > > > > >
> > > > > > > Philippe
> > > > > > >
> > > > > > > I am not sure it is worth the trouble. In most circumstances
> the
> > > stale
> > > > > > > connection check should be turned off anyway.
> > > > > > >
> > > > > > > I think it just needs to be better documented.
> > > > > > >
> > > > > > > Oleg
> > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski <
> > > ol...@apache.org
>
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote:
> > > > > > > > > > Hello,
> > > > > > > > > > Profiling JMeter, I noticed an important number of
> > > > > > > SocketTimeoutException
> > > > > > > > > > being triggered without any impact on response status.
> > > > > > > > > >
> > > > > > > > > > I investigated it a bit deeper and find out it affected
> only
> > > > > > > HttpClient
> > > > > > > > > > implementations.
> > > > > > > > > > Looking a bit deeper, it is due to connection stale check
> > > which
> > > > > is
> > > > > > > > > enabled
> > > > > > > > > > by default.
> > > > > > > > > > This check sets a timeout to 1ms , see :
> > > > > > > > > > -
> org.apache.http.impl.io.SocketInputBuffer#isDataAvailable
> > > > > > > > > > -
> org.apache.http.impl.AbstractHttpClientConnection#isStale
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Is this the only a
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
On Sat, 2014-02-15 at 14:46 +0100, Philippe Mouawad wrote:
> Hello Oleg,
> The problem is that is seems in this particular case, removing this check
> it leads to 50% error.
>
What kind of HttpRequestRetryHandler implementation does JMeter 2.11
use? Usually with the stale check disabled one should want to retry at
least some requests automatically.
> I could be explained by a max number of requests per connection setting, as
> per their documentation:
> - http://aws.amazon.com/articles/1904
>
> "Also, don't overuse a connection. Amazon S3 will accept up to 100 requests
> before it closes a connection (resulting in 'connection reset'). Rather
> than having this happen, use a connection for 80-90 requests before closing
> and re-opening a new connection."
>
> Is there by the way a configuration parameter in HttpClient to limit the
> number of requests per connection ?
>
There is no parameter but one can use a custom ConnectionReuseStrategy
to that effect
---
ConnectionReuseStrategy reuseStrategy = new
DefaultConnectionReuseStrategy() {
@Override
public boolean keepAlive(
final HttpResponse response, final HttpContext context) {
HttpConnection conn = (HttpConnection)
context.getAttribute(HttpClientContext.HTTP_CONNECTION);
long count = conn.getMetrics().getRequestCount();
if (count >= 100) {
return false;
}
return super.keepAlive(response, context);
}
};
---
Hope this helps
Oleg
> Thanks
> Regards
> Philippe
>
>
>
> On Sat, Feb 15, 2014 at 12:49 PM, Oleg Kalnichevski wrote:
>
> > On Sat, 2014-02-15 at 11:23 +0100, Philippe Mouawad wrote:
> > > Hello ,
> > > yes that's it.
> > >
> > > regards
> > >
> >
> > Then, I am not sure I understand the problem. The stale connection check
> > is about trading off some performance for fewer i/o errors or visa
> > versa.
> >
> > Oleg
> >
> > > On Saturday, February 15, 2014, Oleg Kalnichevski <
> > o...@ok2consulting.com>
> > > wrote:
> > >
> > > >
> > > > On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote:
> > > > > Hello Oleg,
> > > > > We set this configuration in JMeter 2.11, we got recently this bug
> > report
> > > > > related to this change:
> > > > > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119
> > > > >
> > > > > This issue seems to be faced by another person:
> > > > > https://twitter.com/cfwhisperer/status/428278488349417472
> > > > >
> > > > > Regards
> > > > > Philippe
> > > > >
> > > >
> > > > Philippe
> > > >
> > > > I am not sure I remember the context. Is it about turning off stale
> > > > connection checking?
> > > >
> > > > Oleg
> > > >
> > > > >
> > > > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski > >
> > > > wrote:
> > > > >
> > > > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote:
> > > > > > > Hello Oleg,
> > > > > > > Thanks for answer.
> > > > > > > Wouldn't it be interesting to add some threshold or delay between
> > > > each
> > > > > > > check instead of doing check on each request ?
> > > > > > >
> > > > > > > Regards
> > > > > > > Philippe
> > > > > > >
> > > > > >
> > > > > > Philippe
> > > > > >
> > > > > > I am not sure it is worth the trouble. In most circumstances the
> > stale
> > > > > > connection check should be turned off anyway.
> > > > > >
> > > > > > I think it just needs to be better documented.
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > > >
> > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski <
> > ol...@apache.org
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote:
> > > > > > > > > Hello,
> > > > > > > > > Profiling JMeter, I noticed an important number of
> > > > > > SocketTimeoutException
> > > > > > > > > being triggered without any impact on response status.
> > > > > > > > >
> > > > > > > > > I investigated it a bit deeper and find out it affected only
> > > > > > HttpClient
> > > > > > > > > implementations.
> > > > > > > > > Looking a bit deeper, it is due to connection stale check
> > which
> > > > is
> > > > > > > > enabled
> > > > > > > > > by default.
> > > > > > > > > This check sets a timeout to 1ms , see :
> > > > > > > > > - org.apache.http.impl.io.SocketInputBuffer#isDataAvailable
> > > > > > > > > - org.apache.http.impl.AbstractHttpClientConnection#isStale
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Is this the only and best way to check for stale connection ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Philippe
> > > > > > > >
> > > > > > > > Philippe
> > > > > > > >
> > > > > > > > I personally do not know of a different (better) way of
> > finding out
> > > > > > > > whether or not a blocking connection is still valid.
> > > > > > > >
> > > > > > > > Oleg
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > -
> > > > > > > > To unsubscribe, e-mail:
> > httpclient-users-unsubsc
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
Hello Oleg, The problem is that is seems in this particular case, removing this check it leads to 50% error. I could be explained by a max number of requests per connection setting, as per their documentation: - http://aws.amazon.com/articles/1904 "Also, don't overuse a connection. Amazon S3 will accept up to 100 requests before it closes a connection (resulting in 'connection reset'). Rather than having this happen, use a connection for 80-90 requests before closing and re-opening a new connection." Is there by the way a configuration parameter in HttpClient to limit the number of requests per connection ? Thanks Regards Philippe On Sat, Feb 15, 2014 at 12:49 PM, Oleg Kalnichevski wrote: > On Sat, 2014-02-15 at 11:23 +0100, Philippe Mouawad wrote: > > Hello , > > yes that's it. > > > > regards > > > > Then, I am not sure I understand the problem. The stale connection check > is about trading off some performance for fewer i/o errors or visa > versa. > > Oleg > > > On Saturday, February 15, 2014, Oleg Kalnichevski < > o...@ok2consulting.com> > > wrote: > > > > > > > > On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote: > > > > Hello Oleg, > > > > We set this configuration in JMeter 2.11, we got recently this bug > report > > > > related to this change: > > > > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119 > > > > > > > > This issue seems to be faced by another person: > > > > https://twitter.com/cfwhisperer/status/428278488349417472 > > > > > > > > Regards > > > > Philippe > > > > > > > > > > Philippe > > > > > > I am not sure I remember the context. Is it about turning off stale > > > connection checking? > > > > > > Oleg > > > > > > > > > > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski > > > > wrote: > > > > > > > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote: > > > > > > Hello Oleg, > > > > > > Thanks for answer. > > > > > > Wouldn't it be interesting to add some threshold or delay between > > > each > > > > > > check instead of doing check on each request ? > > > > > > > > > > > > Regards > > > > > > Philippe > > > > > > > > > > > > > > > > Philippe > > > > > > > > > > I am not sure it is worth the trouble. In most circumstances the > stale > > > > > connection check should be turned off anyway. > > > > > > > > > > I think it just needs to be better documented. > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski < > ol...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote: > > > > > > > > Hello, > > > > > > > > Profiling JMeter, I noticed an important number of > > > > > SocketTimeoutException > > > > > > > > being triggered without any impact on response status. > > > > > > > > > > > > > > > > I investigated it a bit deeper and find out it affected only > > > > > HttpClient > > > > > > > > implementations. > > > > > > > > Looking a bit deeper, it is due to connection stale check > which > > > is > > > > > > > enabled > > > > > > > > by default. > > > > > > > > This check sets a timeout to 1ms , see : > > > > > > > > - org.apache.http.impl.io.SocketInputBuffer#isDataAvailable > > > > > > > > - org.apache.http.impl.AbstractHttpClientConnection#isStale > > > > > > > > > > > > > > > > > > > > > > > > Is this the only and best way to check for stale connection ? > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > Philippe > > > > > > > > > > > > > > Philippe > > > > > > > > > > > > > > I personally do not know of a different (better) way of > finding out > > > > > > > whether or not a blocking connection is still valid. > > > > > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > To unsubscribe, e-mail: > httpclient-users-unsubscr...@hc.apache.org > > > > > > > For additional commands, e-mail: > > > httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > > > > > For additional commands, e-mail: > httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > > > > > > > > > -- Cordialement. Philippe Mouawad.
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
On Sat, 2014-02-15 at 11:23 +0100, Philippe Mouawad wrote: > Hello , > yes that's it. > > regards > Then, I am not sure I understand the problem. The stale connection check is about trading off some performance for fewer i/o errors or visa versa. Oleg > On Saturday, February 15, 2014, Oleg Kalnichevski > wrote: > > > > > On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote: > > > Hello Oleg, > > > We set this configuration in JMeter 2.11, we got recently this bug report > > > related to this change: > > > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119 > > > > > > This issue seems to be faced by another person: > > > https://twitter.com/cfwhisperer/status/428278488349417472 > > > > > > Regards > > > Philippe > > > > > > > Philippe > > > > I am not sure I remember the context. Is it about turning off stale > > connection checking? > > > > Oleg > > > > > > > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski > > > > > > wrote: > > > > > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote: > > > > > Hello Oleg, > > > > > Thanks for answer. > > > > > Wouldn't it be interesting to add some threshold or delay between > > each > > > > > check instead of doing check on each request ? > > > > > > > > > > Regards > > > > > Philippe > > > > > > > > > > > > > Philippe > > > > > > > > I am not sure it is worth the trouble. In most circumstances the stale > > > > connection check should be turned off anyway. > > > > > > > > I think it just needs to be better documented. > > > > > > > > Oleg > > > > > > > > > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski > > > > > > > > > > > > wrote: > > > > > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote: > > > > > > > Hello, > > > > > > > Profiling JMeter, I noticed an important number of > > > > SocketTimeoutException > > > > > > > being triggered without any impact on response status. > > > > > > > > > > > > > > I investigated it a bit deeper and find out it affected only > > > > HttpClient > > > > > > > implementations. > > > > > > > Looking a bit deeper, it is due to connection stale check which > > is > > > > > > enabled > > > > > > > by default. > > > > > > > This check sets a timeout to 1ms , see : > > > > > > > - org.apache.http.impl.io.SocketInputBuffer#isDataAvailable > > > > > > > - org.apache.http.impl.AbstractHttpClientConnection#isStale > > > > > > > > > > > > > > > > > > > > > Is this the only and best way to check for stale connection ? > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > Philippe > > > > > > > > > > > > Philippe > > > > > > > > > > > > I personally do not know of a different (better) way of finding out > > > > > > whether or not a blocking connection is still valid. > > > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > > > > > - > > > > > > To unsubscribe, e-mail: > > > > > > httpclient-users-unsubscr...@hc.apache.org > > > > > > For additional commands, e-mail: > > httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > To unsubscribe, e-mail: > > > > httpclient-users-unsubscr...@hc.apache.org > > > > For additional commands, e-mail: > > > > httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > - > > To unsubscribe, e-mail: > > httpclient-users-unsubscr...@hc.apache.org > > For additional commands, e-mail: > > httpclient-users-h...@hc.apache.org > > > > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: Incredibly slow gzip downloads
On Sat, 2014-02-15 at 11:22 +0100, Sebastiano Vigna wrote: > Something happened from HttpClient 4.3.1 to 4.3.2. > > All of a sudden, we are seeing an immense amount of CPU spent into inflating > compressed HTTP responses. We were experimenting >5000 pages/s two months > ago, but less than 1200 pages/s now (same hardware, etc.), for no apparent > reason. The CPU is 100% consumed by Inflater. > > The absolutely weird thing is that in profiling we are spending 15% doing > inflation and 1.5% doing deflation... and the same amount of data is going > through the two phases. Downgrading to 4.3.1 immediately brought back the old > performance, and we spend more time in deflation than in inflation (as it > should be). > > Maybe some change in the buffer size? > > Ciao, > Here's the diff between 4.3.1 and 4.3.2 https://github.com/apache/httpclient/compare/4.3.1...4.3.2 The problem mostly likely has been introduced by HTTPCLIENT-1432 [1]. I reviewed the patch once more and could not find anything obviously wrong with it. Try reverting the changes introduced by HTTPCLIENT-1432 and see if that resolves the problem. Oleg [1] https://issues.apache.org/jira/browse/HTTPCLIENT-1432 - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
Hello , yes that's it. regards On Saturday, February 15, 2014, Oleg Kalnichevski wrote: > > On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote: > > Hello Oleg, > > We set this configuration in JMeter 2.11, we got recently this bug report > > related to this change: > > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119 > > > > This issue seems to be faced by another person: > > https://twitter.com/cfwhisperer/status/428278488349417472 > > > > Regards > > Philippe > > > > Philippe > > I am not sure I remember the context. Is it about turning off stale > connection checking? > > Oleg > > > > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski > > > > wrote: > > > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote: > > > > Hello Oleg, > > > > Thanks for answer. > > > > Wouldn't it be interesting to add some threshold or delay between > each > > > > check instead of doing check on each request ? > > > > > > > > Regards > > > > Philippe > > > > > > > > > > Philippe > > > > > > I am not sure it is worth the trouble. In most circumstances the stale > > > connection check should be turned off anyway. > > > > > > I think it just needs to be better documented. > > > > > > Oleg > > > > > > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski > > > > > > > > > wrote: > > > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote: > > > > > > Hello, > > > > > > Profiling JMeter, I noticed an important number of > > > SocketTimeoutException > > > > > > being triggered without any impact on response status. > > > > > > > > > > > > I investigated it a bit deeper and find out it affected only > > > HttpClient > > > > > > implementations. > > > > > > Looking a bit deeper, it is due to connection stale check which > is > > > > > enabled > > > > > > by default. > > > > > > This check sets a timeout to 1ms , see : > > > > > > - org.apache.http.impl.io.SocketInputBuffer#isDataAvailable > > > > > > - org.apache.http.impl.AbstractHttpClientConnection#isStale > > > > > > > > > > > > > > > > > > Is this the only and best way to check for stale connection ? > > > > > > > > > > > > > > > > > > Regards > > > > > > Philippe > > > > > > > > > > Philippe > > > > > > > > > > I personally do not know of a different (better) way of finding out > > > > > whether or not a blocking connection is still valid. > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > - > > > > > To unsubscribe, e-mail: > > > > > httpclient-users-unsubscr...@hc.apache.org > > > > > For additional commands, e-mail: > httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > To unsubscribe, e-mail: > > > httpclient-users-unsubscr...@hc.apache.org > > > For additional commands, e-mail: > > > httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > - > To unsubscribe, e-mail: > httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: > httpclient-users-h...@hc.apache.org > > -- Cordialement. Philippe Mouawad.
Incredibly slow gzip downloads
Something happened from HttpClient 4.3.1 to 4.3.2. All of a sudden, we are seeing an immense amount of CPU spent into inflating compressed HTTP responses. We were experimenting >5000 pages/s two months ago, but less than 1200 pages/s now (same hardware, etc.), for no apparent reason. The CPU is 100% consumed by Inflater. The absolutely weird thing is that in profiling we are spending 15% doing inflation and 1.5% doing deflation... and the same amount of data is going through the two phases. Downgrading to 4.3.1 immediately brought back the old performance, and we spend more time in deflation than in inflation (as it should be). Maybe some change in the buffer size? Ciao, seba - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote: > Hello Oleg, > We set this configuration in JMeter 2.11, we got recently this bug report > related to this change: > https://issues.apache.org/bugzilla/show_bug.cgi?id=56119 > > This issue seems to be faced by another person: > https://twitter.com/cfwhisperer/status/428278488349417472 > > Regards > Philippe > Philippe I am not sure I remember the context. Is it about turning off stale connection checking? Oleg > > On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski wrote: > > > On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote: > > > Hello Oleg, > > > Thanks for answer. > > > Wouldn't it be interesting to add some threshold or delay between each > > > check instead of doing check on each request ? > > > > > > Regards > > > Philippe > > > > > > > Philippe > > > > I am not sure it is worth the trouble. In most circumstances the stale > > connection check should be turned off anyway. > > > > I think it just needs to be better documented. > > > > Oleg > > > > > > > > On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski > > wrote: > > > > > > > On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad wrote: > > > > > Hello, > > > > > Profiling JMeter, I noticed an important number of > > SocketTimeoutException > > > > > being triggered without any impact on response status. > > > > > > > > > > I investigated it a bit deeper and find out it affected only > > HttpClient > > > > > implementations. > > > > > Looking a bit deeper, it is due to connection stale check which is > > > > enabled > > > > > by default. > > > > > This check sets a timeout to 1ms , see : > > > > > - org.apache.http.impl.io.SocketInputBuffer#isDataAvailable > > > > > - org.apache.http.impl.AbstractHttpClientConnection#isStale > > > > > > > > > > > > > > > Is this the only and best way to check for stale connection ? > > > > > > > > > > > > > > > Regards > > > > > Philippe > > > > > > > > Philippe > > > > > > > > I personally do not know of a different (better) way of finding out > > > > whether or not a blocking connection is still valid. > > > > > > > > Oleg > > > > > > > > > > > > - > > > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > - > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
