RE: Cluster instability due to java.net.SocketTimeoutException: Read timed out
Hello,
I'm getting exactly the same issue (running NiFI 1.1.1 too)
I have tried to increase read timeout to 10 secs (jetty threads = 200) but I'm
still getting the same errors.
@Alessio did you solve the issue on your side ? I am interested in the
solution if you found any.
Regards,
Léonard
-Original Message-
From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
Sent: lundi 13 février 2017 10:04
To: users@nifi.apache.org
Subject: Re: Cluster instability due to java.net.SocketTimeoutException: Read
timed out
Hello,
The timeout happens when a node replicate a request to nodes to get a merged
cluster wide response, so increasing nifi.cluster.node.read.timeout might help
(in conf/nifi.properties).
Request replication shouldn't timeout ideally, increasing
nifi.web.jetty.threads might help to increase each node's capability to handle
more requests..
nifi.cluster.node.read.timeout=5 sec
nifi.web.jetty.threads=200
Would you share nifi thread dump result (./bin/nifi.sh dump, and written in
logs/nifi-bootstrap.log) while you are sending request from NiFi UI and waiting
its result arrives?
It might be helpful for us to find what causes the timeout or slowness.
Thanks,
Koji
On Mon, Feb 13, 2017 at 5:52 PM, Alessio Palma
<alessio.pa...@docomodigital.com> wrote:
> Hello again,
> this is the stack trace which is raised on the connection timeout.
>
> 2017-02-13 08:02:01,796 WARN [Replicate Request Thread-3]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET /nifi-api/flow/status to mid1-a128-3.buongiorn
> o.com:8080 due to {}
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
> at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
> URLConnectionClientHandler.java:155)
> ~[jersey-client-1.19.jar:1.19]
> at com.sun.jersey.api.client.Client.handle(Client.java:652)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIP
> ContentEncodingFilter.java:123)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
> ~[jersey-client-1.19.jar:1.19]
> at
> com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509
> )
> ~[jersey-client-1.19.jar:1.19]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolReques
> tReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolReques
> tReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
> ~[nifi-framework-cluster-1.1.1.jar:1.1.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511
> )
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> ava:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused
> by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> ~[na:1.8.0_101]
> at
> java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:170)
> ~[na:1.8.0_101]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> ~[na:1.8.0_101]
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.j
> ava:1375)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> ~[na:1.8.0_101]
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:5
> 59)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(
> AbstractDelegateHttpsURLConnection.java:185)
> ~[na:1.8.0_101]
> at
> sun.net.www.protocol.http.HttpURLConnection.getInp
Re: Cluster instability due to java.net.SocketTimeoutException: Read timed out
jersey-client-1.19.jar:1.19] > at > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) > ~[jersey-client-1.19.jar:1.19] > ... 12 common frames omitted > > perhaps it is useful to understand the problem. > > > > > From: Alessio Palma <alessio.pa...@docomodigital.com> > Sent: Monday, February 13, 2017 9:31:15 AM > To: NIFI User mailing > Subject: Cluster instability due to java.net.SocketTimeoutException: Read > timed out > > Hello all, > sometime the cluster is unstable due to "read time out". > Is here any parameter which I can fine tuning in order to reduce this > problem ? > Most timeouts (66%) are coming from > > com.sun.jersey.api.client.ClientHandlerException > > few (33%) from > > o.apache.nifi.controller.FlowController > > > Thanks in adavance > AP >
Re: Cluster instability due to java.net.SocketTimeoutException: Read timed out
Hello again,
this is the stack trace which is raised on the connection timeout.
2017-02-13 08:02:01,796 WARN [Replicate Request Thread-3]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
/nifi-api/flow/status to mid1-a128-3.buongiorn
o.com:8080 due to {}
com.sun.jersey.api.client.ClientHandlerException:
java.net.SocketTimeoutException: Read timed out
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:509)
~[jersey-client-1.19.jar:1.19]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:578)
~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:770)
~[nifi-framework-cluster-1.1.1.jar:1.1.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_101]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[na:1.8.0_101]
at java.net.SocketInputStream.read(SocketInputStream.java:170)
~[na:1.8.0_101]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[na:1.8.0_101]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
~[na:1.8.0_101]
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
~[na:1.8.0_101]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
~[na:1.8.0_101]
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
~[na:1.8.0_101]
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
~[na:1.8.0_101]
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
~[na:1.8.0_101]
at
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
~[na:1.8.0_101]
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
~[na:1.8.0_101]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
~[na:1.8.0_101]
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
~[na:1.8.0_101]
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
~[na:1.8.0_101]
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
~[na:1.8.0_101]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
~[jersey-client-1.19.jar:1.19]
at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
~[jersey-client-1.19.jar:1.19]
... 12 common frames omitted
perhaps it is useful to understand the problem.
From: Alessio Palma <alessio.pa...@docomodigital.com>
Sent: Monday, February 13, 2017 9:31:15 AM
To: NIFI User mailing
Subject: Cluster instability due to java.net.SocketTimeoutException: Read timed
out
Hello all,
sometime the cluster is unstable due to "read time out".
Is here any parameter which I can fine tuning in order to reduce this problem ?
Most timeouts (66%) are coming from
com.sun.jersey.api.client.ClientHandlerException
few (33%) from
o.apache.nifi.controller.FlowController
Thanks in adavance
AP
Cluster instability due to java.net.SocketTimeoutException: Read timed out
Hello all, sometime the cluster is unstable due to "read time out". Is here any parameter which I can fine tuning in order to reduce this problem ? Most timeouts (66%) are coming from com.sun.jersey.api.client.ClientHandlerException few (33%) from o.apache.nifi.controller.FlowController Thanks in adavance AP
