[ https://issues.apache.org/jira/browse/NIFI-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583067#comment-16583067 ]
Joseph Witt commented on NIFI-5522: ----------------------------------- [~Diego Queiroz] lots of good data/info in here so thanks for helping zero in on this. Can you please do a thread dump when it is in this state and share the entire output of it. You can get a dump by running 'bin/nifi.sh dump' and I believe you can also give it a file you want the dump written to with 'bin/nifi.sh dump dump.log' for example. If you dont specify a filename it will end up in the bootstrap.log. Either way, that should be very helpful to figure out what the thread is doing when it hits this condition. A thread dump once the 503 happens and a thread dump when you try to stop it... Thanks > HandleHttpRequest enters in fault state and does not recover > ------------------------------------------------------------ > > Key: NIFI-5522 > URL: https://issues.apache.org/jira/browse/NIFI-5522 > Project: Apache NiFi > Issue Type: Bug > Affects Versions: 1.7.0, 1.7.1 > Reporter: Diego Queiroz > Priority: Critical > Labels: security > Attachments: HandleHttpRequest_Error_Template.xml, > image-2018-08-15-21-10-27-926.png, image-2018-08-15-21-10-33-515.png, > image-2018-08-15-21-11-57-818.png, image-2018-08-15-21-15-35-364.png, > image-2018-08-15-21-19-34-431.png, image-2018-08-15-21-20-31-819.png, > test_http_req_resp.xml > > > HandleHttpRequest randomly enters in a fault state and does not recover until > I restart the node. I feel the problem is triggered when some exception > occurs (ex.: broken request, connection issues, etc), but I am usually able > to reproduce this behavior stressing the node with tons of simultaneous > requests: > {{# example script to stress server}} > {{for i in `seq 1 10000`; do}} > {{ wget ‐T10 ‐t10 ‐qO‐ 'http://127.0.0.1:64080/'>/dev/null &}} > {{done}} > When this happens, HandleHttpRequest start to return "HTTP ERROR 503 - > Service Unavailable" and does not recover from this state: > !image-2018-08-15-21-10-33-515.png! > If I try to stop the HandleHttpRequest processor, the running threads does > not terminate: > !image-2018-08-15-21-11-57-818.png! > If I force them to terminate, the listen port continue being bound by NiFi: > !image-2018-08-15-21-15-35-364.png! > If I try to connect again, I got a HTTP ERROR 500: > !image-2018-08-15-21-19-34-431.png! > > If I try to start the HandleHttpRequest processor again, it doesn't start > with the message: > * {{ERROR [Timer-Driven Process Thread-11] > o.a.n.p.standard.HandleHttpRequest > HandleHttpRequest[id=9bae326b-5ac3-3e9f-2dac-c0399d8f2ddb] > {color:#FF0000}*Failed to process session due to > org.apache.nifi.processor.exception.ProcessException: Failed to initialize > the server: org.apache.nifi.processor.exception.ProcessException: Failed to > initialize the server*{color}}}{\{ > org.apache.nifi.processor.exception.ProcessException: Failed to initialize > the server}}\{{ {{ at > org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:501)}}}}\{{ > {{ at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)}}}}\{{ > {{ at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)}}}}\{{ > {{ at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)}}}}\{{ > {{ at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)}}}}\{{ > {{ at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}}}\{{ > {{ at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)}}}}\{{ {{ at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)}}}}\{{ > {{ at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)}}}}\{{ > {{ at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}}}\{{ > {{ at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}}}\{{ > {{ at java.lang.Thread.run(Thread.java:748)}}}}{{ {color:#FF0000}*Caused by: > java.net.BindException: Address already in use*{color}}}\{{ {{ at > sun.nio.ch.Net.bind0(Native Method)}}}}\{{ {{ at > sun.nio.ch.Net.bind(Net.java:433)}}}}\{{ {{ at > sun.nio.ch.Net.bind(Net.java:425)}}}}\{{ {{ at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)}}}}\{{ > {{ at > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)}}}}\{{ {{ at > org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:298)}}}}\{{ > {{ at > org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)}}}}\{{ > {{ at > org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)}}}}\{{ > {{ at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)}}}}\{{ > {{ at org.eclipse.jetty.server.Server.doStart(Server.java:431)}}}}\{{ {{ at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)}}}}\{{ > {{ at > org.apache.nifi.processors.standard.HandleHttpRequest.initializeServer(HandleHttpRequest.java:430)}}}}\{{ > {{ at > org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:489)}} > \{ Unknown macro: { ... 11 common frames omitted}}}{{}}} > !image-2018-08-15-21-20-31-819.png! > > The only way to workaround this when it happens is chaging the port it > listens to or restarting NiFi service. I flagged this as a security issue > because it allows someone to cause a DoS to the service. > I found several similar issues, but most of them are related with old > versions, I am can confirm this affects versions 1.7.0 and 1.7.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)