[ https://issues.apache.org/jira/browse/PIG-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mohit Sabharwal updated PIG-4190: --------------------------------- Summary: Implement replicated join in Spark engine (was: Make repl join work with Spark) > Implement replicated join in Spark engine > ----------------------------------------- > > Key: PIG-4190 > URL: https://issues.apache.org/jira/browse/PIG-4190 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: Praveen Rachabattuni > Assignee: Mohit Sabharwal > Fix For: spark-branch > > > Related e2e tests: Union_7, Union_8, Union_13 > Sample script: > a = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age, gpa); > b = load '/user/pig/tests/data/singlefile/studentcolon10k' using > PigStorage(':') as (name, age, gpa); > c = union a, b; > d = load '/user/pig/tests/data/singlefile/votertab10k' as (name, age, > registration, contributions); > e = join c by name, d by name using 'replicated'; > store e into '/user/pig/out/praveenr-1411380943-nightly.conf/Union_7.out'; > Pig Stack Trace > --------------- > ERROR 0: java.lang.IllegalArgumentException: Spork unsupported > PhysicalOperator: (Name: e: FRJoin[tuple] - scope-6 Operator Key: scope-6) > org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.IllegalArgumentException: Spork unsupported PhysicalOperator: > (Name: e: FRJoin[tuple] - scope-6 Operator Key: scope-6) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:285) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1378) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363) > at org.apache.pig.PigServer.execute(PigServer.java:1352) > at org.apache.pig.PigServer.executeBatch(PigServer.java:403) > at org.apache.pig.PigServer.executeBatch(PigServer.java:386) > at > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:233) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:204) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:611) > at org.apache.pig.Main.main(Main.java:164) > Caused by: java.lang.IllegalArgumentException: Spork unsupported > PhysicalOperator: (Name: e: FRJoin[tuple] - scope-6 Operator Key: scope-6) > at > org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:239) > at > org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:232) > at > org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:140) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:279) > ... 11 more > ================================================================================ -- This message was sent by Atlassian JIRA (v6.3.4#6332)