David Handermann created NIFI-9448: -------------------------------------- Summary: Potential IllegalStateException when S2S HTTP Client Shutdown Key: NIFI-9448 URL: https://issues.apache.org/jira/browse/NIFI-9448 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.15.0 Reporter: David Handermann Assignee: David Handermann
NiFi Site-to-Site communication over HTTP relies on {{SiteToSiteRestApiClient}} to handle requests and responses using the Apache HttpComponents library. The HttpComponents library maintains a connection pool for processing HTTP transactions. In the course of sending or receiving files, NiFi starts a background thread to make periodic requests to the remote NiFi system in order to extend the current transaction. The {{SiteToSiteRestApiClient}} stops the background thread after completing a transaction. In some cases, the request to extend the transaction can occur after the HttpComponents connection pool is shutdown, resulting in an {{{}IllegalStateException{}}}. The background thread treats all exceptions as failure conditions and then attempts to close the S2S HTTP client itself. This causes subsequent retry requests using the same {{SiteToSiteRestApiClient}} to fail with the same {{IllegalStateException}} indicating that the connection pool is shutdown. The behavior of the extend transaction command should be changed to avoid closing the S2S HTTP client when encountering an {{IllegalStateException}}. This approach will support the potential for subsequent retries to work or fail based on existing timeout configuration settings. The following log messages provide a stack trace of the {{IllegalStateException}} in the extend transaction command and subsequent exception in the Remote Process Group Port connection. {noformat} WARN org.apache.nifi.remote.util.SiteToSiteRestApiClient: Failed to extend transaction ttl java.lang.IllegalStateException: Connection pool shut down at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.extendTransaction(SiteToSiteRestApiClient.java:1028) at org.apache.nifi.remote.util.SiteToSiteRestApiClient.lambda$startExtendingTtl$0(SiteToSiteRestApiClient.java:990) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-11-10 08:17:10,352 INFO org.apache.http.impl.execchain.RetryExec: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request: The target server failed to respond INFO org.apache.http.impl.execchain.RetryExec: Retrying request ERROR org.apache.nifi.remote.StandardRemoteGroupPort: RemoteGroupPort[name=REMOTE_PORT,targets=URL] failed to communicate with remote NiFi instance due to java.lang.IllegalStateException: Connection pool shut down {noformat} -- This message was sent by Atlassian Jira (v8.20.1#820001)