[ https://issues.apache.org/jira/browse/HADOOP-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181071#comment-17181071 ]
Stephen O'Donnell edited comment on HADOOP-17209 at 8/20/20, 9:43 AM: ---------------------------------------------------------------------- >From the tutorial posted here: [http://www.iitk.ac.in/esc101/05Aug/tutorial/native1.1/implementing/array.html] It does indeed seem that you must call ReleaseIntArrayElements each time you call GetIntArrayElements, so the change makes sense to me. However, I have never used JNI, so my knowledge in this area is very small. Grepping the code for GetIntArrayElements, I see there are 3 occurrences of this currently: {code:java} $ pwd /Users/sodonnell/source/upstream_hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode $ grep GetIntArrayElements *.c jni_common.c: tmpInputOffsets = (int*)(*env)->GetIntArrayElements(env, jni_common.c: tmpOutputOffsets = (int*)(*env)->GetIntArrayElements(env, jni_rs_decoder.c: int* tmpErasedIndexes = (int*)(*env)->GetIntArrayElements(env, {code} -This patch address 2 of them - in jni_common.c in the function getOutputs, do we need a call to ReleaseIntArrayElements there too?- This patch seems to address all 3. Earlier I missed the 3rd change in the patch file. [~seanlook] Have you been running with this patch in production for some time, and all EC operations are working fine with it? was (Author: sodonnell): >From the tutorial posted here: http://www.iitk.ac.in/esc101/05Aug/tutorial/native1.1/implementing/array.html It does indeed seem that you must call ReleaseIntArrayElements each time you call GetIntArrayElements, so the change makes sense to me. However, I have never used JNI, so my knowledge in this area is very small. Grepping the code for GetIntArrayElements, I see there are 3 occurrences of this currently: {code} $ pwd /Users/sodonnell/source/upstream_hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode $ grep GetIntArrayElements *.c jni_common.c: tmpInputOffsets = (int*)(*env)->GetIntArrayElements(env, jni_common.c: tmpOutputOffsets = (int*)(*env)->GetIntArrayElements(env, jni_rs_decoder.c: int* tmpErasedIndexes = (int*)(*env)->GetIntArrayElements(env, {code} This patch address 2 of them - in jni_common.c in the function getOutputs, do we need a call to ReleaseIntArrayElements there too? {code} void getOutputs(JNIEnv *env, jobjectArray outputs, jintArray outputOffsets, unsigned char** destOutputs, int num) { int numOutputs = (*env)->GetArrayLength(env, outputs); int i, *tmpOutputOffsets; jobject byteBuffer; if (numOutputs != num) { THROW(env, "java/lang/InternalError", "Invalid outputs"); } tmpOutputOffsets = (int*)(*env)->GetIntArrayElements(env, outputOffsets, NULL); for (i = 0; i < numOutputs; i++) { byteBuffer = (*env)->GetObjectArrayElement(env, outputs, i); destOutputs[i] = (unsigned char *)((*env)->GetDirectBufferAddress(env, byteBuffer)); destOutputs[i] += tmpOutputOffsets[i]; } } {code} [~seanlook] Have you been running with this patch in production for some time, and all EC operations are working fine with it? > Erasure Coding: Native library memory leak > ------------------------------------------ > > Key: HADOOP-17209 > URL: https://issues.apache.org/jira/browse/HADOOP-17209 > Project: Hadoop Common > Issue Type: Bug > Components: native > Affects Versions: 3.3.0, 3.2.1, 3.1.3 > Reporter: Sean Chow > Assignee: Sean Chow > Priority: Major > Attachments: HADOOP-17209.001.patch, > datanode.202137.detail_diff.5.txt, image-2020-08-15-18-26-44-744.png, > image-2020-08-20-12-35-39-906.png > > > We use both {{apache-hadoop-3.1.3}} and {{CDH-6.1.1-1.cdh6.1.1.p0.875250}} > HDFS in production, and both of them have the memory increasing over {{-Xmx}} > value. > !image-2020-08-15-18-26-44-744.png! > > We use EC strategy to to save storage costs. > This's the jvm options: > {code:java} > -Dproc_datanode -Dhdfs.audit.logger=INFO,RFAAUDIT > -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true > -Xms8589934592 -Xmx8589934592 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled > -XX:+HeapDumpOnOutOfMemoryError ...{code} > The max jvm heapsize is 8GB, but we can see the datanode RSS memory is 48g. > All the other datanodes in this hdfs cluster has the same issue. > {code:java} > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 226044 hdfs 20 0 50.6g 48g 4780 S 90.5 77.0 14728:27 > /usr/java/jdk1.8.0_162/bin/java -Dproc_datanode{code} > > This too much memory used leads to my machine unresponsive(if enable swap), > or oom-killer happens. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org