[ https://issues.apache.org/jira/browse/PIG-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048202#comment-15048202 ]
liyunzhang_intel commented on PIG-4621: --------------------------------------- [~zulfiali], [~mohitsabharwal] and [~xuefuz]: The patch modified the code in JobControlCompiler.java. I don't know whether this change can be accepted by the community. Let's use an example to show why script with illustrate fails in spark mode: {code} visits = LOAD './visits' AS (user:chararray, ulr:chararray, timestamp:chararray); recent_visits = FILTER visits BY timestamp >= '20081201'; dump recent_visits; illustrate recent_visits; {code} The exception is ClassCastException: can not cast SparkScriptState to MRScriptState and the error stack is: {code} org.apache.pig.tools.pigstats.mapreduce.MRScriptState.get(MRScriptState.java:67) org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:512) org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:327) org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:110) org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:259) org.apache.pig.pen.ExampleGenerator.readBaseData(ExampleGenerator.java:223) org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:155) org.apache.pig.PigServer.getExamples(PigServer.java:1305) org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:812) org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:818) org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:385) org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) org.apache.pig.Main.run(Main.java:624) org.apache.pig.Main.main(Main.java:170) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.apache.hadoop.util.RunJar.main(RunJar.java:212) {code} The reason why this exception is thrown out is because illustrate will call LocalMapReduceSimulator.launchPig and MRScriptState#get is called. ScriptState.get() returns SparkScriptState and we cast SparkScriptState to MRScriptState. {code} public static MRScriptState get() { return (MRScriptState) ScriptState.get(); } {code} In mr implementation, it shows that it calls LocalMapReduceSimulator#launchPig to execute a small set of data to show the execution steps. If we really need implement it in spark mode. I think we need to implement a class like LocalMapReduceSimulator. After investigating the code, I found currently illustrate is *only* implemented in mr mode. > Enable Illustrate in spark > -------------------------- > > Key: PIG-4621 > URL: https://issues.apache.org/jira/browse/PIG-4621 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: liyunzhang_intel > Assignee: Syed Zulfiqar Ali > Fix For: spark-branch > > > Current we don't support illustrate in spark mode. > How illustrate works > see:http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE -- This message was sent by Atlassian JIRA (v6.3.4#6332)