Finally, it proved to be an issue with glibc, similar with https://issues.apache.org/jira/browse/HADOOP-7154. After setting MALLOC_ARENA_MAX=4, native memory consumption got controlled.
Best regards, Duke If not now, when? If not me, who? On Sun, Mar 17, 2019 at 10:02 PM Duke DAI <[email protected]> wrote: > Hi Gurus, > > I'm struggling with native memory leak issue on my application. It works > like a crawler, the only significant thirdparty is httpclient 4.5.6. > pmap shows there are a lot of "anon" allocations but no hint where they > come from. "anon" will become more and more along with the application > running, slowly and gradually. It may be coincident with threads number > spike. > > It's sure it's not a heap issue because gc.log only showed young GCs with > G1GC, quite good performance with less than 100ms. There is NO mixed GC > because young GC can collect most of garbages in heap, 98% of heap. I even > suspected that G1GC young GC will never trigger the garbage collecting > process in native memory(by DirectByteBuffer). Some articles say only full > GC can trigger native memory garbage collecting, but they are talking about > traditional GC algorithms, not G1GC. I'm not sure about the suspect. > > I tried below JVM options to limit native memory consumption, but there is > no help. And jstat indicated there is no problem with Metaspace. > -XX:MaxMetaspaceSize=512m -Djdk.nio.maxCachedBufferSize=5000000 > > Before I try -XX:MaxDirectMemorySize(will try next), I inspected all code > to make sure there is no leak from code level. When I read the code, it > seems it does not handle IOException thrown by ResponseHandler. If > IOException(SocketTimeOutException when reading the > response.getEntity().getContent()) is hit in ResponseHandler, the framework > only handles ClientProtocolException but not IOException, so the > InputStream may get leaked there. Response.close will call > socket.close(socket input/output stream), but I wonder if the intermediate > DirectByteBuffer will be closed or not. > > Can some one shed a light here? > > @Override > public <T> T execute(final HttpHost target, final HttpRequest request, > final ResponseHandler<? extends T> responseHandler, final HttpContext > context) > throws IOException, ClientProtocolException { > Args.notNull(responseHandler, "Response handler"); > > final CloseableHttpResponse response = execute(target, request, context); > try { > final T result = responseHandler.handleResponse(response); > final HttpEntity entity = response.getEntity(); > EntityUtils.consume(entity); > return result; > } catch (final ClientProtocolException t) { > // Try to salvage the underlying connection in case of a protocol > exception > final HttpEntity entity = response.getEntity(); > try { > EntityUtils.consume(entity); > } catch (final Exception t2) { > // Log this exception. The original exception is more > // important and will be thrown to the caller. > this.log.warn("Error consuming content after an exception.", t2); > } > throw t; > } finally { > response.close(); > } > } > > > > > Best regards, > Duke > If not now, when? If not me, who? >
