[ https://issues.apache.org/jira/browse/TEZ-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15556137#comment-15556137 ]
TezQA commented on TEZ-3330: ---------------------------- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12832178/TEZ-3330.01.patch against master revision dceb365. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 3.0.1) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2025//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/2025//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2025//console This message is automatically generated. > Error on avro M/R job with Tez: missing configuration property > -------------------------------------------------------------- > > Key: TEZ-3330 > URL: https://issues.apache.org/jira/browse/TEZ-3330 > Project: Apache Tez > Issue Type: Bug > Reporter: Manuel Godbert > Assignee: Siddharth Seth > Labels: newbie > Attachments: TEZ-3330.01.patch, TEZ-3330.temp.2.patch, > TEZ-3330.temp.patch > > > I tried running the simple avro M/R job MapredColorCount, that I found in the > examples of avro release 1.7.7. > It failed with the following trace: > {code} > errorMessage=Shuffle Runner > Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: > Error while doing final merge > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:378) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:337) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.NullPointerException > at java.io.StringReader.<init>(StringReader.java:50) > at org.apache.avro.Schema$Parser.parse(Schema.java:917) > at org.apache.avro.Schema.parse(Schema.java:966) > at org.apache.avro.mapred.AvroJob.getMapOutputSchema(AvroJob.java:78) > at > org.apache.avro.mapred.AvroKeyComparator.setConf(AvroKeyComparator.java:39) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) > at > org.apache.tez.runtime.library.common.ConfigUtils.getIntermediateInputKeyComparator(ConfigUtils.java:133) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.finalMerge(MergeManager.java:915) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.close(MergeManager.java:540) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:376) > ... 6 more > {code} > Digging a bit I saw that during shuffle Tez can't access some of the > configuration properties of the job. In our example it is the > avro.output.schema that is missing. > With some more complicated code I could get one step further and a similar > issue happened when the valuesIterator for the reducer was being built: > {code} > java.lang.NullPointerException > at java.io.StringReader.<init>(StringReader.java:50) > at org.apache.avro.Schema$Parser.parse(Schema.java:917) > at org.apache.avro.Schema.parse(Schema.java:966) > at org.apache.avro.mapred.AvroJob.getMapOutputSchema(AvroJob.java:78) > at > org.apache.avro.mapred.AvroSerialization.getDeserializer(AvroSerialization.java:53) > at > org.apache.hadoop.io.serializer.SerializationFactory.getDeserializer(SerializationFactory.java:90) > at > org.apache.tez.runtime.library.common.ValuesIterator.<init>(ValuesIterator.java:80) > at > org.apache.tez.runtime.library.input.OrderedGroupedKVInput.createValuesIterator(OrderedGroupedKVInput.java:287) > {code} > I am using HDP2.4, Tez 0.7.0, avro 1.7.4 -- This message was sent by Atlassian JIRA (v6.3.4#6332)