Hey Vijay, I’ve seen a few people lately running into issues with InvokeHTTP. The common thread for all of them is that they are hitting servers that are using HTTP 2. Reading threads from OkHttp (the underlying HTTP library that we use), I see that a lot of people are running into issues with it. In at least several of the cases, it ends up being the HTTP Server or a proxy/load balancer in the middle that is not properly supporting the HTTP 2 protocol. Unclear if there are HTTP 2.0 related bugs in OkHttp itself or not. In any case, users often comment that changing the protocol to HTTP 1.1 resolved the issue. I filed a Jira [1] to allow that to be exposed in the InvokeHTTP processor. Hopefully that will help.
Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-8181 On Jan 28, 2021, at 4:46 PM, Joe Witt <joe.w...@gmail.com<mailto:joe.w...@gmail.com>> wrote: The likely relevant thread is here "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait() [0x000070000fed4000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.5/Native Method) - waiting on <no object reference available> at java.lang.Object.wait(java.base@11.0.5/Object.java:328) at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577) at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143) - waiting to re-lock in wait() <0x00000007e007d818> (a okhttp3.internal.http2.Http2Stream) at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) at okhttp3.RealCall.execute(RealCall.java:69) at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5/Executors.java:515) at java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5/FutureTask.java:305) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5/ScheduledThreadPoolExecutor.java:305) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.5/Thread.java:834) This implies that it is either genuinely waiting for IO that the remote server is failing to send or has gotten itself into a state it could never break out of. Can you please use this exact same flow on the latest NiFi release 1.12.1 to see if the issue remains? Thanks On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <vchh...@apple.com<mailto:vchh...@apple.com>> wrote: Hi Joe, Thanks for looking into this issue. We are on NiFi 1.10.0 Please see the attached 1. thread dump file 2. nifi-app.log 3. nifi-bootstrap.log 4. nifi.properties The InvokeHTTP configuration is as shown below. On Jan 25, 2021, at 7:24 AM, Joe Witt <joe.w...@gmail.com<mailto:joe.w...@gmail.com>> wrote: Hello If you suspect an actual stuck/hung thread then the best course of action is to generate a thread dump. This can be done in a couple ways but one of the easiest is to run 'bin/nifi.sh dump'. We would need to see the nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should include the specific nifi version being used specifics of the processor configuration. Thanks On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <jairohenaoro...@gmail.com<mailto:jairohenaoro...@gmail.com>> wrote: Hello, In my case something similar happens. I invoke a REST-JSON service that sometimes takes more than 10 minutes to respond (I have set the timeout to 15 minutes) and after the second or third call the service responds (I could check it on the server side) but the processor remains waiting for the response. I try to stop it and after "terminate" it but the thread seems to stay active, so I delete it, (disconnect it and remove it from the flow) and add a new InvokeHttp and it works again. Given the difficulty of finding steps to reproduce it, I have not reported it. I hope someone can help us. On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vchh...@apple.com<mailto:vchh...@apple.com>> wrote: Hi All, I am seeing a case where InvokeHTTP processor hangs after several successful calls to an endpoint. (Processor is paginating over millions of rows and pulling a few thousand rows at a time) <Screen Shot 2021-01-24 at 10.37.14 PM.png> If I have more than one thread scheduled, all threads hang. <Screen Shot 2021-01-24 at 10.37.55 PM.png> When this happens InvokeHTTP does not release the current flowfile at all. My only recourse is to try to stop the processor. <Screen Shot 2021-01-17 at 9.36.08 PM.png> This "stop" does not complete, at which point I have to terminate the processor. <Screen Shot 2021-01-17 at 9.36.58 PM.png> After termination of this processor, starting of the processor does not process any files. So I have to remove the processor and all its connections, delete the processor and replace it with another InvokeHTTP with same properties and connections. I have tried to set the "Connection: close" header but it didn't help. Has anyone seen this behaviors, how did you resolve it? Thanks -- Jairo Henao @jairohenaorojas
