Hi. We're using Hive 3.0.1 and we're currently experiencing this issue:
*Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1581309524541_0094_14_00, diagnostics=[Vertex vertex_1581309524541_0094_14_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGeneratorat org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4122)at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:207)at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2932)at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2879)at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2861)at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1957)at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:206)at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2317)at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2303)at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)... 25 moreCaused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19294)at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19258)at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19360)at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19355)at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.parseFrom(DAGProtos.java:19552)at org.apache.tez.common.TezUtils.createConfFromByteString(TezUtils.java:116)at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:130)... 30 more]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1581309524541_0094_14_01, diagnostics=[Vertex received Kill in NEW state., Vertex vertex_1581309524541_0094_14_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1* Our tables are in ORC format and we probably have 1k Hive table in that cluster. I see these issues below but I don't think any of them are related to what we're experiencing. https://issues.apache.org/jira/browse/HIVE-11592 https://issues.apache.org/jira/browse/HIVE-11268 I see that the *sizeLimit* for CodedInputStream is set to *PROTOBUF_MESSAGE_MAX_LIMIT* = *1073741824*. This issue just popped out a few days ago. Is there any workaround for this? And would guys know how we can lower the protobuf message size per request? BTW, we've tried deleting the contents of TAB_COL_STATS but it didn't help. Thanks, Bernard
