[ https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955983#comment-16955983 ]
Tsz-wo Sze commented on HDDS-2331: ---------------------------------- Chunk size and array size may not be necessarily the same. Chunk size (e.g. 16MB) is a larger value than array size (e.g. 1MB). A chunk could use a list of arrays as the buffer. > Client OOME due to buffer retention > ----------------------------------- > > Key: HDDS-2331 > URL: https://issues.apache.org/jira/browse/HDDS-2331 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client > Affects Versions: 0.5.0 > Reporter: Attila Doroszlai > Assignee: Shashikant Banerjee > Priority: Critical > Attachments: profiler.png > > > Freon random key generator exhausts default heap after just few hundred 1MB > keys. Heap dump on OOME reveals 150+ instances of > {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}. > Steps to reproduce: > # Start Ozone cluster with 1 datanode > # Start Freon (5K keys of size 1MB) > Result: OOME after a few hundred keys > {noformat} > $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone > $ docker-compose up -d > $ docker-compose exec scm bash > $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError' > $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1 > --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120 > --bufferSize 65536 > ... > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java_pid289.hprof ... > Heap dump file created [1456141975 bytes in 7.760 secs] > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org