[ https://issues.apache.org/jira/browse/TEZ-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajesh Balamohan resolved TEZ-1944. ----------------------------------- Resolution: Duplicate Thanks [~ozawa]. Checked this usecase after TEZ-1911 and it works fine. Marking this as duplicate of TEZ-1911. > OOM when using tez.runtime.shuffle.memory-to-memory.enable=true > --------------------------------------------------------------- > > Key: TEZ-1944 > URL: https://issues.apache.org/jira/browse/TEZ-1944 > Project: Apache Tez > Issue Type: Bug > Reporter: Rajesh Balamohan > > ETL workload at 10 TB scale > {code} > $HIVE_HOME/bin/hive -hiveconf tez.runtime.io.sort.factor=200 --hiveconf > hive.tez.auto.reducer.parallelism=true --hiveconf > tez.am.heartbeat.interval-ms.max=20 --hiveconf tez.runtime.io.sort.mb=1200 > --hiveconf tez.runtime.sort.threads=2 --hiveconf hive.tez.container.size=4096 > --hiveconf tez.runtime.shuffle.memory-to-memory.enable=true --hiveconf > tez.runtime.shuffle.memory-to-memory.segments=4 > create table testData as select > ss_sold_date_sk,ss_sold_time_sk,ss_item_sk,ss_customer_sk,ss_quantity,ss_sold_date > from store_sales distribute by ss_sold_date; > {code} > Tasks starts throwing OOM with the following exception. > {code} > aused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.<init>(MapOutput.java:81) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.createMemoryMapOutput(MapOutput.java:133) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.unconditionalReserve(MergeManager.java:397) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.access$300(MergeManager.java:74) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$IntermediateMemoryToMemoryMerger.merge(MergeManager.java:530) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89) > , errorMessage=Shuffle Runner > Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: > error in shuffle in MemToMemMerger [Map_1] > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:349) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:328) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > Created this to track issues related to memory to memory merging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)