[
https://issues.apache.org/jira/browse/NIFI-7536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136088#comment-17136088
]
Mark Payne commented on NIFI-7536:
----------------------------------
The following appear to be key details when troubleshooting.
* When creating the Response that describes a Processor, NiFi waits up to 1
millisecond for validation on a Processor to complete. However, the act of
waiting up to 1 millisecond can be fairly expensive and could actually result
in several milliseconds. If it takes 10 milliseconds per processor, for 1,000
processors that equates to 10 seconds already. In reality it's probably a
little less than that, but it's a very significant amount of time for a web
request. We can eliminate this all together and just avoid waiting for
validation to complete at all.
* For each processor that we return in the response, we must also include the
Processor's status. Obtaining the status currently is very inefficient because
it captures a snapshot status of the entire flow, and then only uses the status
for that processor. Again, not a big deal for 1 processor but for 1,000
processors (or many thousands) this because extremely expensive.
* When the Cluster Coordinator receives the response that contains this list
of Processors, it must merge together the responses from all nodes. This JSON
that it receives/parses/merges is huge. The code only really needs a couple of
fields, but the entire Processor data model is used because it already existed.
This parsing and merging takes a few seconds on my laptop with a flow that has
a couple thousand processors. We need to introduce a new REST endpoint that
returns only the information we care about.
When you add up all of these different inefficiencies, it's easy to see that
this works well for a small flow but can take a really long time for a large
flow.
> Timeouts occurring updating Parameter Context
> ---------------------------------------------
>
> Key: NIFI-7536
> URL: https://issues.apache.org/jira/browse/NIFI-7536
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> I am encountering the following exception when trying to update a Parameter
> Context. The Parameter Context is new and has no Parameters. Adding a new
> parameter fails when attempting to check if all affected processors are
> stopped.
>
>
> {quote}2020-06-08 16:09:35,329 WARN [Replicate Request Thread-25]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/process-groups/root/processors to <node> due to
> java.net.SocketTimeoutException: timeout
> 2020-06-08 16:09:35,332 WARN [Replicate Request Thread-25]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> java.net.SocketTimeoutException: timeout
> at okio.Okio$4.newTimeoutException(Okio.java:232)
> at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
> at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355)
> at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227)
> at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
> at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
> at
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
> at okhttp3.RealCall.execute(RealCall.java:77)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:146)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:140)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketException: Socket closed
> at java.net.SocketInputStream.read(SocketInputStream.java:203)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
> at okio.Okio$2.read(Okio.java:140)
> at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
> ... 28 common frames omitted
> 2020-06-08 16:09:35,335 ERROR [Parameter Context Update Thread-1]
> o.a.n.web.api.ParameterContextResource Failed to update Parameter Context
> {quote}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
