[ https://issues.apache.org/jira/browse/PIG-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786019#comment-17786019 ]
Koji Noguchi commented on PIG-5444: ----------------------------------- This is how it looks when testFRJoinOut8 is run by itself when MultiQueryOptimizerTez happen to work on A before B. Note POValueOutputTez on B is there to prevent the overlapping edges. {code:java} Tez vertex scope-48 # Plan on vertex B: Split - scope-61 | | | Local Rearrange[tuple]{int}(false) - scope-27 -> scope-44 | | | | | Project[int][0] - scope-23 | | | POValueOutputTez - scope-49 -> [scope-54] | |---B: New For Each(false,false)[bag] - scope-18 | | | Cast[int] - scope-13 | | | |---Project[bytearray][0] - scope-12 | | | Cast[int] - scope-16 | | | |---Project[bytearray][1] - scope-15 | |---B: Load(hdfs://localhost:38814/user/gtrain/testFrJoinInput2.txt:org.apache.pig.builtin.PigStorage) - scope-11 Tez vertex scope-54 # Plan on vertex Local Rearrange[tuple]{int}(false) - scope-39 -> scope-44 | | | Project[int][1] - scope-35 | |---POValueInputTez - scope-55 <- scope-48 Tez vertex scope-44 # Plan on vertex A: Split - scope-60 | | | E: Store(hdfs://localhost:38814/tmp/temp-1966813510/tmp-652837441:org.apache.pig.impl.io.InterStorage) - scope-62 -> scope-43 | | | |---D: FRJoin[tuple] - scope-36 <- scope-54 | | | | | Project[int][1] - scope-34 | | | | | Project[int][1] - scope-35 | | | E: Store(hdfs://localhost:38814/tmp/temp-1966813510/tmp-652837441:org.apache.pig.impl.io.InterStorage) - scope-63 -> scope-43 | | | |---C: FRJoin[tuple] - scope-24 <- scope-48 | | | | | Project[int][0] - scope-22 | | | | | Project[int][0] - scope-23 | |---A: New For Each(false,false)[bag] - scope-7 | | | Cast[int] - scope-2 | | | |---Project[bytearray][0] - scope-1 | | | Cast[int] - scope-5 | | | |---Project[bytearray][1] - scope-4 | |---A: Load(hdfs://localhost:38814/user/gtrain/testFrJoinInput.txt:org.apache.pig.builtin.PigStorage) - scope-0 {code} > TestFRJoin.testFRJoinOut7 and testFRJoinOut8 failing with Edge already > defined error on Tez > ------------------------------------------------------------------------------------------- > > Key: PIG-5444 > URL: https://issues.apache.org/jira/browse/PIG-5444 > Project: Pig > Issue Type: Bug > Components: tez > Reporter: Koji Noguchi > Assignee: Koji Noguchi > Priority: Major > > With Tez, when testing individual tests (TestFRJoin.testFRJoinOut7 and > testFRJoinOut8) separately, they pass the tests. But when entire TestFRJoin > is run, these two tests on Tez are failing with > {noformat} > Unable to open iterator for alias E > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias E > at org.apache.pig.PigServer.openIterator(PigServer.java:1024) > at org.apache.pig.test.TestFRJoin.testFRJoinOut7(TestFRJoin.java:409) > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias E > at org.apache.pig.PigServer.storeEx(PigServer.java:1127) > at org.apache.pig.PigServer.store(PigServer.java:1086) > at org.apache.pig.PigServer.openIterator(PigServer.java:999) > Caused by: > org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR > 2017: Internal error creating job configuration. > at > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:153) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.compile(TezJobCompiler.java:81) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:200) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1479) > at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464) > at org.apache.pig.PigServer.storeEx(PigServer.java:1123) > Caused by: java.lang.IllegalArgumentException: Edge [scope-632 : > org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] -> > [scope-628 : > org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ({ > BROADCAST : org.apache.tez.runtime.library.input.UnorderedKVInput >> > PERSISTED >> org.apache.tez.runtime.library.output.UnorderedKVOutput >> > NullEdgeManager }) already defined! > at org.apache.tez.dag.api.DAG.addEdge(DAG.java:296) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:410) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265) > at > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56) > at > org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.buildDAG(TezJobCompiler.java:69) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:120) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)