[ https://issues.apache.org/jira/browse/PIG-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204584#comment-14204584 ]
Praveen Rachabattuni commented on PIG-4231: ------------------------------------------- [~Carlos Balduz] I have uploaded patch(!PIG-4231-2.patch) combining three individual files, also, I have taken the privilege to cache line from CounterConverter.java. I think cache should be a choice to be left to the user, unless we find it optimizing the whole pipeline. {code} jRdd.cache() {code} Tested in spark local mode and committed to spark branch. Thanks Carlos. > Make rank work with Spark > ------------------------- > > Key: PIG-4231 > URL: https://issues.apache.org/jira/browse/PIG-4231 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: Carlos Balduz > Assignee: Carlos Balduz > Labels: spork > Attachments: CounterConverter.java, PIG-4231-2.patch, PIG-4231.patch, > RankConverter.java > > > Rank does not work with Spark since PORank and POCounter have not been > implemented yet. > Pig Stack Trace > --------------- > ERROR 0: java.lang.IllegalArgumentException: Spork unsupported > PhysicalOperator: (Name: DATA: POCounter[tuple] - scope-146 Operator Key: > scope-146) > org.apache.pig.backend.executionengine.ExecException: ERROR 0: > java.lang.IllegalArgumentException: Spork unsupported PhysicalOperator: > (Name: DATA: POCounter[tuple] - scope-146 Operator Key: scope-146) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:285) > at org.apache.pig.PigServer.launchPlan(PigServer.java:1378) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363) > at org.apache.pig.PigServer.execute(PigServer.java:1352) > at org.apache.pig.PigServer.executeBatch(PigServer.java:403) > at org.apache.pig.PigServer.executeBatch(PigServer.java:386) > at > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:233) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:204) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > at org.apache.pig.Main.run(Main.java:482) > at org.apache.pig.Main.main(Main.java:164) -- This message was sent by Atlassian JIRA (v6.3.4#6332)