ankitsultana opened a new pull request, #10487: URL: https://github.com/apache/pinot/pull/10487
At present we don't set any configs when we create the HttpClient in `FileUploadDownloadClient`. By default, Apache Commons HttpClient uses maxTotalConns=20 and maxConnsPerRoute=2 [ref](https://www.javadoc.io/static/org.apache.httpcomponents/httpclient/4.5.3/org/apache/http/impl/conn/PoolingClientConnectionManager.html#:~:text=PoolingConnectionManager%20maintains%20a%20maximum%20limit,more%2020%20connections%20in%20total.). This can become a bottleneck when we need to replace a server, since in that case the new server may have to download 1000s of segments. You can find a sample trace from a thread-dump we took in our clusters below. This patch makes this configurable. Since apache http-client is used in a bunch of places, each corresponding use-case can add their own config prefix. The suffix of the config is enforced by `HttpClientConfig.newBuilder(PinotConfiguration, String)` in an attempt to ensure standarization across the use-cases. ``` "HelixTaskExecutor-message_handle_thread" #226 daemon prio=5 os_prio=0 cpu=122724.33ms elapsed=4094.44s tid=0x00007f41cc007800 nid=0x140 waiting on condition [0x00007f3f969ea000] -- java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00007f4595c10118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:2081) at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:393) at org.apache.http.pool.AbstractConnPool.access$300(AbstractConnPool.java:70) at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:253) - locked <0x00007f4f3edda4a8> (a org.apache.http.pool.AbstractConnPool$2) at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:198) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:306) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.pinot.common.utils.http.HttpClient.downloadFile(HttpClient.java:334) at org.apache.pinot.common.utils.FileUploadDownloadClient.downloadFile(FileUploadDownloadClient.java:1005) at org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher.lambda$fetchSegmentToLocal$0(HttpSegmentFetcher.java:66) at org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher$Lambda$438/0x00007f3ebd9ed908.call(Unknown Source) at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) at org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher.fetchSegmentToLocal(HttpSegmentFetcher.java:54) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:152) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:138) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:191) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:168) at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadFromPeersWithoutStreaming(BaseTableDataManager.java:512) at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:484) at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:450) at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:442) at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:404) at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:355) at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:162) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
