[ https://issues.apache.org/jira/browse/HADOOP-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181077#comment-17181077 ]
Sean Chow commented on HADOOP-17209: ------------------------------------ Hi [~sodonnell] , the patch has 3 {{ReleaseIntArrayElements}}.:P Yeah, I I've replace with the recompiled native library in five datanode from my production. All of them works fine when getting and putting files, and the memory used seems very promising. > Erasure Coding: Native library memory leak > ------------------------------------------ > > Key: HADOOP-17209 > URL: https://issues.apache.org/jira/browse/HADOOP-17209 > Project: Hadoop Common > Issue Type: Bug > Components: native > Affects Versions: 3.3.0, 3.2.1, 3.1.3 > Reporter: Sean Chow > Assignee: Sean Chow > Priority: Major > Attachments: HADOOP-17209.001.patch, > datanode.202137.detail_diff.5.txt, image-2020-08-15-18-26-44-744.png, > image-2020-08-20-12-35-39-906.png > > > We use both {{apache-hadoop-3.1.3}} and {{CDH-6.1.1-1.cdh6.1.1.p0.875250}} > HDFS in production, and both of them have the memory increasing over {{-Xmx}} > value. > !image-2020-08-15-18-26-44-744.png! > > We use EC strategy to to save storage costs. > This's the jvm options: > {code:java} > -Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT > -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true > -Xms8589934592 -Xmx8589934592 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled > -XX:+HeapDumpOnOutOfMemoryError ...{code} > The max jvm heapsize is 8GB, but we can see the datanode RSS memory is 48g. > All the other datanodes in this hdfs cluster has the same issue. > {code:java} > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 226044 hdfs 20 0 50.6g 48g 4780 S 90.5 77.0 14728:27 > /usr/java/jdk1.8.0_162/bin/java -Dproc_datanode{code} > > This too much memory used leads to my machine unresponsive(if enable swap), > or oom-killer happens. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org