abstractdog commented on PR #414: URL: https://github.com/apache/tez/pull/414#issuecomment-2893479169
> > Do we need to specify: mapreduce.framework.name as yarn as well? > > for me earlier it never use to work unless, I specify `export HADOOP_USER_CLASSPATH_FIRST=true` > > does it work for you without that, even BigTop had to add that https://github.com/apache/bigtop/pull/1246/files#diff-f68b85f9302907e466b58d438376afb074df98fdbe571d30c188cd1767ff11eeR18 > > yeah, I can see this workaround happening everywhere, but here, it has just worked OOTB, maybe a certain state of defining ENV vars like HADOOP_CLASSPATH? I don't know what about: > > 1. I'm playing with it if I can reproduce their problems > 2. can you try the script on your side if it works? if the script works without the additional export for you too, we might want to publish it as is, proving that no further classpath hack are needed > > let me check mapreduce.framework.name as well, for me, the script simply ran a Tez DAG, so I haven't configured anything more...but this is really interesting, I'll discover wow, that's indeed needed, otherwise I get exotic exception like ``` java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.getSerializedSize(DAGProtos.java:21080) at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:75) at org.apache.tez.common.TezUtils.writeConfInPB(TezUtils.java:162) at org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:82) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.createMRInputPayload(MRInputHelpers.java:717) at org.apache.tez.mapreduce.input.MRInput$MRInputHelpersInternal.createMRInputPayload(MRInput.java:712) at org.apache.tez.mapreduce.input.MRInput$MRInputConfigBuilder.createGeneratorDataSource(MRInput.java:336) at org.apache.tez.mapreduce.input.MRInput$MRInputConfigBuilder.build(MRInput.java:266) at org.apache.tez.examples.OrderedWordCount.createDAG(OrderedWordCount.java:130) at org.apache.tez.examples.OrderedWordCount.runJob(OrderedWordCount.java:200) at org.apache.tez.examples.TezExampleBase._execute(TezExampleBase.java:245) at org.apache.tez.examples.TezExampleBase.run(TezExampleBase.java:126) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82) at org.apache.tez.examples.OrderedWordCount.main(OrderedWordCount.java:208) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.tez.examples.ExampleDriver.main(ExampleDriver.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:328) at org.apache.hadoop.util.RunJar.main(RunJar.java:241) ``` adding this export right before the Tez DAG submission -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
