Trying to run a query on Tez with the following configurations
*set hive.tez.container.size=5120* *set mapreduce.map.child.java.opts=-Xmx5120M* *set hive.tez.java.opts=-Xmx4096M* *set hive.auto.convert.join.noconditionaltask.size=805306000* *set tez.am.resource.memory.mb=5120* *set tez.am.java.opts=-Xmx4096M* The above config settings were set after running https://github.com/hortonworks/hdp-configuration-utils/blob/master/2.1/hdp-configuration-utils.py to get the right memory configs Tried with both set tez.runtime.io.sort.mb=512 set mapreduce.task.io.sort.mb=512 and set tez.runtime.io.sort.mb=2048 set mapreduce.task.io.sort.mb=2048 The query I am trying run is *select sum(tab1.m1),sum(tab1.m2)* * from tab1 join tab2 dm on tab1.col1=tab2.col1* * where tab1.dt = '2014-06-01' * * and tab2.col2 = '..'* * and tab2.col3 IN ('..')* * group by TAB1.col1* *where TAB1.col1 has high cardinality(around 700- 800 million)* And its going OOM during shuffle phase. errorMessage=Fetch failed Container released by application, AttemptID:attempt_1407396011310_1577_1_01_000000_4 Info:Error: exceptionThrown=java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46) at org.apache.tez.runtime.library.shuffle.common.MemoryFetchedInput.<init>(MemoryFetchedInput.java:38) at org.apache.tez.runtime.library.shuffle.common.impl.SimpleFetchedInputAllocator.allocate(SimpleFetchedInputAllocator.java:137) at org.apache.tez.runtime.library.shuffle.common.Fetcher.fetchInputs(Fetcher.java:252) at org.apache.tez.runtime.library.shuffle.common.Fetcher.call(Fetcher.java:184) at org.apache.tez.runtime.library.shuffle.common.Fetcher.call(Fetcher.java:59) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Please advice if the configurations look ok? Do I need to change anything? Thanks Suma
