Joe, behavior is the same with the 1.12.1 version.
Mark, I will make the change to the OkHttp client and try. I do know that this flow was working for a long time and there was possibly an upgrade done on the source endpoint. Thank you for opening the Jira. Regards, Vijay > On Jan 28, 2021, at 4:37 PM, Mark Payne <marka...@hotmail.com> wrote: > > Hey Vijay, > > I’ve seen a few people lately running into issues with InvokeHTTP. The common > thread for all of them is that they are hitting servers that are using HTTP > 2. Reading threads from OkHttp (the underlying HTTP library that we use), I > see that a lot of people are running into issues with it. In at least several > of the cases, it ends up being the HTTP Server or a proxy/load balancer in > the middle that is not properly supporting the HTTP 2 protocol. Unclear if > there are HTTP 2.0 related bugs in OkHttp itself or not. In any case, users > often comment that changing the protocol to HTTP 1.1 resolved the issue. I > filed a Jira [1] to allow that to be exposed in the InvokeHTTP processor. > Hopefully that will help. > > Thanks > -Mark > > [1] https://issues.apache.org/jira/browse/NIFI-8181 > <https://issues.apache.org/jira/browse/NIFI-8181> > >> On Jan 28, 2021, at 4:46 PM, Joe Witt <joe.w...@gmail.com >> <mailto:joe.w...@gmail.com>> wrote: >> >> The likely relevant thread is here >> >> "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms >> elapsed=4403.88s tid=0x00007fe44f16b000 nid=0xe103 in Object.wait() >> [0x000070000fed4000] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(java.base@11.0.5/Native Method) >> - waiting on <no object reference available> >> at java.lang.Object.wait(java.base@11.0.5/Object.java:328) >> at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577) >> at >> okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143) >> - waiting to re-lock in wait() <0x00000007e007d818> (a >> okhttp3.internal.http2.Http2Stream) >> at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120) >> at >> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> at >> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >> at >> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >> at >> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> at >> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) >> at >> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) >> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) >> at okhttp3.RealCall.execute(RealCall.java:69) >> at >> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793) >> at >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> at >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) >> at >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) >> at >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) >> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5/Executors.java:515) >> at >> java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5/FutureTask.java:305) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5/ScheduledThreadPoolExecutor.java:305) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1128) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628) >> at java.lang.Thread.run(java.base@11.0.5/Thread.java:834) >> >> This implies that it is either genuinely waiting for IO that the remote >> server is failing to send or has gotten itself into a state it could never >> break out of. Can you please use this exact same flow on the latest NiFi >> release 1.12.1 to see if the issue remains? >> >> Thanks >> >> On Thu, Jan 28, 2021 at 2:43 PM Vijay Chhipa <vchh...@apple.com >> <mailto:vchh...@apple.com>> wrote: >> Hi Joe, >> >> Thanks for looking into this issue. >> We are on NiFi 1.10.0 >> >> Please see the attached >> 1. thread dump file >> 2. nifi-app.log >> 3. nifi-bootstrap.log >> 4. nifi.properties >> >> The InvokeHTTP configuration is as shown below. >> >> >> >> >> >>> On Jan 25, 2021, at 7:24 AM, Joe Witt <joe.w...@gmail.com >>> <mailto:joe.w...@gmail.com>> wrote: >>> >>> Hello >>> >>> If you suspect an actual stuck/hung thread then the best course of action >>> is to generate a thread dump. This can be done in a couple ways but one of >>> the easiest is to run 'bin/nifi.sh dump'. We would need to see the >>> nifi-app.log(s) and nifi-bootstrap.log(s). In addition the report should >>> include the specific nifi version being used specifics of the processor >>> configuration. >>> >>> Thanks >>> >>> On Mon, Jan 25, 2021 at 6:12 AM Jairo Henao <jairohenaoro...@gmail.com >>> <mailto:jairohenaoro...@gmail.com>> wrote: >>> Hello, >>> >>> In my case something similar happens. I invoke a REST-JSON service that >>> sometimes takes more than 10 minutes to respond (I have set the timeout to >>> 15 minutes) and after the second or third call the service responds (I >>> could check it on the server side) but the processor remains waiting for >>> the response. I try to stop it and after "terminate" it but the thread >>> seems to stay active, so I delete it, (disconnect it and remove it from the >>> flow) and add a new InvokeHttp and it works again. >>> >>> Given the difficulty of finding steps to reproduce it, I have not reported >>> it. I hope someone can help us. >>> >>> On Mon, Jan 25, 2021 at 12:00 AM Vijay Chhipa <vchh...@apple.com >>> <mailto:vchh...@apple.com>> wrote: >>> Hi All, >>> >>> I am seeing a case where InvokeHTTP processor hangs after several >>> successful calls to an endpoint. (Processor is paginating over millions of >>> rows and pulling a few thousand rows at a time) >>> >>> >>> <Screen Shot 2021-01-24 at 10.37.14 PM.png> >>> >>> If I have more than one thread scheduled, all threads hang. >>> <Screen Shot 2021-01-24 at 10.37.55 PM.png> >>> >>> >>> When this happens InvokeHTTP does not release the current flowfile at all. >>> My only recourse is to try to stop the processor. >>> <Screen Shot 2021-01-17 at 9.36.08 PM.png> >>> >>> This "stop" does not complete, at which point I have to terminate the >>> processor. >>> <Screen Shot 2021-01-17 at 9.36.58 PM.png> >>> >>> After termination of this processor, starting of the processor does not >>> process any files. So I have to remove the processor and all its >>> connections, delete the processor and replace it with another InvokeHTTP >>> with same properties and connections. >>> >>> I have tried to set the "Connection: close" header but it didn't help. >>> >>> Has anyone seen this behaviors, how did you resolve it? >>> >>> Thanks >>> >>> >>> >>> -- >>> Jairo Henao >>> @jairohenaorojas >>> >> >
smime.p7s
Description: S/MIME cryptographic signature
