[ https://issues.apache.org/jira/browse/PIG-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387849#comment-14387849 ]
liyunzhang_intel commented on PIG-4269: --------------------------------------- Hi [~mohitsabharwal]: The reason why testAccumWithSort and testAccumAfterNestedOp fail in spark while not in MR is: POSort in not generated in the plan of MR while in the plan of spark. Currently, we can not remove POSort in spark plan because we need poSort to generate SortConverter to implement the sort feature in spark. The detail error stackflow of "Caught error from UDF: org.apache.pig.test.utils.AccumulatorBagCount exec() should not be called. " is org.apache.pig.backend.hadoop.executionengine.util.AccumulatorOptimizerUtil#addAccumulatorSpark org.apache.pig.backend.hadoop.executionengine.util.AccumulatorOptimizerUtil#check if org.apache.pig.backend.hadoop.executionengine.util.AccumulatorOptimizerUtil#check meets POSort, it returns false and makes AccumulatorOptimizerUtil.java#foundUDF is false and po_foreach#setAccumulative()(see https://github.com/kellyzly/pig/blob/spark/src/org/apache/pig/backend/hadoop/executionengine/util/AccumulatorOptimizerUtil.java#L129) will not be executed and po_foreach#isAccumulative() is false. If po_foreach#isAccumulative() false,org.apache.pig.EvalFunc#exec will be executed(see https://github.com/kellyzly/pig/blob/spark/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java#L361). In org.apache.pig.test.utils.AccumulatorBagCount#exec(), exception "exec() should not be called." is thrown out. > Enable unit test "TestAccumulator" for spark > -------------------------------------------- > > Key: PIG-4269 > URL: https://issues.apache.org/jira/browse/PIG-4269 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: liyunzhang_intel > Assignee: liyunzhang_intel > Fix For: spark-branch > > Attachments: PIG-4269.patch, PIG-4269_1.patch, PIG-4269_Jekins.png, > TEST-org.apache.pig.test.TestAccumulator.txt > > > error log is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)