[ https://issues.apache.org/jira/browse/TEZ-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth resolved TEZ-3779. --------------------------------- Resolution: Not A Bug Please send a mail to the hive-user list for issues like this. > Tez query failed with OutOfMemoryError: Java heap space > ------------------------------------------------------- > > Key: TEZ-3779 > URL: https://issues.apache.org/jira/browse/TEZ-3779 > Project: Apache Tez > Issue Type: Bug > Affects Versions: 0.8.5 > Reporter: Xin Yang > > Tez query failed with OutOfMemoryError > Query: > {code:java} > select a11.ISSR_CTRY_CD CTRY_CD, > a14.DMSTC_INTL_IND DMSTC_INTL_IND, > a11.ISSR_USR_BUS_ID bus_id, > ' ' CustCol_73, > a11.CPD_MNTH_ID CPD_MONTH_ID, > a11.prod_afs_cd_vcis prod_acct_fund_srce_cd_vcis, > sum((Case when a13.card_prsnt_cd in (1) then a11.auth_tran_us_amt > else NULL end)) AUTHTRANAMTUSD, > sum((Case when a13.card_prsnt_cd in (1) then a11.CS_TRAN_CNT else > NULL end)) AUTHTRANCNT, > (Case when max((Case when a13.card_prsnt_cd in (1) then 1 else 0 > end)) = 1 then count(distinct (Case when a13.card_prsnt_cd in (1) then > a11.pymt_crd_acct_num_norm else NULL end)) else NULL end) WJXBFS1, > max((Case when a13.card_prsnt_cd in (1) then 1 else 0 end)) > GODWFLAG1_1, > sum((Case when a13.card_prsnt_cd in (0) then a11.auth_tran_us_amt > else NULL end)) AUTHTRANAMTUSD1, > sum((Case when a13.card_prsnt_cd in (0) then a11.CS_TRAN_CNT else > NULL end)) AUTHTRANCNT1, > (Case when max((Case when a13.card_prsnt_cd in (0) then 1 else 0 > end)) = 1 then count(distinct (Case when a13.card_prsnt_cd in (0) then > a11.pymt_crd_acct_num_norm else NULL end)) else NULL end) WJXBFS2, > max((Case when a13.card_prsnt_cd in (0) then 1 else 0 end)) > GODWFLAG4_1 > from opebi_bi.tcaef_auth_dtl_h a11 > join OPCODE.TEDC_ECI_MOTO a12 > on (a11.ECI_MOTO_CD = a12.ECI_MOTO_CD) > join OPCODE.TEDC_CARD_PRSNT_EBI a13 > on (a11.POS_ENTRY_MODE_CD = a13.POS_ENTRY_MODE_CD and > a11.POS_ENV_CD = a13.POS_ENV_CD and > a12.eci_moto_grp_cd = a13.eci_moto_grp_cd) > join OPCODE.TEDC_ACCT_MRCH_JRSDCTN_CD a14 > on (a11.VCIS_ACCT_MRCH_JRSDCTN_CD = a14.ACCT_MRCH_JRSDCTN_CD) > join OPCODE.TEDC_GLBL_PROD_ID a15 > on (a11.ALP_ACCT_PROD_ID = a15.PROD_ID_CD) > join OPCODE.TEDC_AUTH_RESP_CD a16 > on (a11.resp_cd = a16.AUTH_RESP_CD) > where (a11.MRCH_CATG_CD not in (6010, 6011) > and a11.CPD_MNTH_ID BETWEEN 201602 and 201602 > and a11.PROC_TRAN_CD in ('00') > and a11.ISSR_CTRY_CD in (76) > and a11.reqst_msg_typ_cd in ('0100', '0200') > and a16.AUTH_RESP_RLUP_CD in (0, 1, 4, 5) > and a11.resp_cd not in ('13', '--') > and a11.reqst_msg_typ_cd in ('0100', '0200', '0000') > and a11.stip_advc_cd in ('1', '2', '3', '4', '5', '6') > and a11.ACQR_BIN_NUM not in (746922) > and a15.PROD_BRND_CD in ('VISA') > and a15.PROD_ID_PLTFRM_CD in ('BZ', ' ', 'CN', 'GV', 'CO') > and a11.acqr_pcr_num not in ('8088', '9088') > and (a13.card_prsnt_cd in (1) > or a13.card_prsnt_cd in (0))) > group by a11.ISSR_CTRY_CD, > a14.DMSTC_INTL_IND, > a11.ISSR_USR_BUS_ID, > a11.CPD_MNTH_ID, > a11.prod_afs_cd_vcis; > {code} > Stacktrace: > {code:java} > Status: Failed > Vertex failed, vertexName=Map 3, vertexId=vertex_1495595408051_21107_2_03, > diagnostics=[Task failed, taskId=task_1495595408051_21107_2_03_000000, > diagnostics=[TaskAttempt 0 failed, info=[Error: exceptio > nThrown=java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46) > at > org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput.<init>(MemoryFetchedInput.java:38) > at > org.apache.tez.runtime.library.common.shuffle.impl.SimpleFetchedInputAllocator.allocate(SimpleFetchedInputAllocator.java:141) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:717) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:489) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:398) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:195) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:70) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > , errorMessage=Fetch failed:java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46) > at > org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput.<init>(MemoryFetchedInput.java:38) > at > org.apache.tez.runtime.library.common.shuffle.impl.SimpleFetchedInputAllocator.allocate(SimpleFetchedInputAllocator.java:141) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.fetchInputs(Fetcher.java:717) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:489) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:398) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:195) > at > org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:70) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > ], TaskAttempt 1 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator > initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > ], TaskAttempt 1 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator > initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Map operator initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java > heap space > at > org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:388) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:378) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481) > at > org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438) > at > org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:214) > ... 15 more > Caused by: java.util.concurrent.ExecutionException: > java.lang.OutOfMemoryError: Java heap space > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:386) > ... 20 more > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.hive.serde2.WriteBuffers.nextBufferToWrite(WriteBuffers.java:241) > at > org.apache.hadoop.hive.serde2.WriteBuffers.write(WriteBuffers.java:217) > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$LazyBinaryKvWriter.writeKey(MapJoinBytesTableContainer.java:235) > at > org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:445) > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:365) > at > org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:191) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:288) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:173) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:169) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:92) > ... 4 more > ], TaskAttempt 2 failed, info=[Error: Failure while running > task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator > initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Map operator initialization failed > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147) > ... 14 more > ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 > killedTasks:0, Vertex vertex_1495595408051_21107_2_03 [Map 3] killed/failed > due to:null]Vertex killed, vertexName=Reducer 7, vertexId=ve > rtex_1495595408051_21107_2_06, diagnostics=[Vertex received Kill while in > RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, > failedTasks:0 killedTasks:2, Vertex vertex_1495595408051_211 > 07_2_06 [Reducer 7] killed/failed due to:null]Vertex killed, vertexName=Map > 6, vertexId=vertex_1495595408051_21107_2_05, diagnostics=[Vertex received > Kill while in RUNNING state., Vertex did not succeed > due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex > vertex_1495595408051_21107_2_05 [Map 6] killed/failed due to:null]Vertex > killed, vertexName=Map 5, vertexId=vertex_1495595408051_21107_2 > _04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did > not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex > vertex_1495595408051_21107_2_04 [Map 5] killed/fai > led due to:null]Vertex killed, vertexName=Map 1, > vertexId=vertex_1495595408051_21107_2_02, diagnostics=[Vertex received Kill > while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, > failedTasks:0 killedTasks:41, Vertex vertex_1495595408051_21107_2_02 [Map 1] > killed/failed due to:null]DAG did not succeed due to VERTEX_FAILURE. > failedVertices:1 killedVertices:4 > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)