Adam, do you happen to know what's the status of the spark branch, time line for it to be ready for merge with trunk?
On Wed, Dec 21, 2016 at 10:37 AM, Adam Szita <sz...@cloudera.com> wrote: > Hi Canan, > > If you mean the trunk branch as https://github.com/apache/pig/tree/trunk/ > then no, Spark as execution engine is not available there yet, you'll have > to check out the actual 'spark' branch: > https://github.com/apache/pig/tree/spark/ > Also, what log entries can you see before the exception? > > On trunk the full log should be something like: > > *./pig -x spark_local* > Listening for transport dt_socket at address: 5005 > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ > 16/12/21 09:24:29 ERROR pig.Main: ERROR 2040: Unknown exec type: > spark_local > 16/12/21 09:24:29 WARN pig.Main: There is no log file to write to. > 16/12/21 09:24:29 ERROR pig.Main: org.apache.pig.PigException: ERROR 2040: > Unknown exec type: spark_local > ... > > > For the same command on spark branch you should see: > > *./pig -x spark_local* > Listening for transport dt_socket at address: 5005 > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : SPARK > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : SPARK_LOCAL > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Picked SPARK_LOCAL as the > ExecType > 2016-12-21 09:33:14,073 [main] INFO org.apache.pig.Main - Apache Pig > version 0.16.0-SNAPSHOT (rUnversioned directory) compiled Dec 21 2016, > 09:32:12 > ... > > Regards, > Adam > > On 21 December 2016 at 08:20, canan chen <ccn...@gmail.com> wrote: > > > Thanks Divya, but this blog doesn't help me. I am using pig trunk which > > already has spark. > > > > And I use this command to build pig > > > > ant clean -Dhadoopversion=23 jar > > > > > > > > > > On Wed, Dec 21, 2016 at 2:54 PM Divya Gehlot <divya.htco...@gmail.com> > > wrote: > > > > > > > > http://blog.cloudera.com/blog/2014/09/pig-is-flying-apache- > > pig-on-apache-spark/ > > > > > > Hope this helps > > > > > > > > > Thanks, > > > Divya > > > > > > On 21 December 2016 at 11:13, canan chen <ccn...@gmail.com> wrote: > > > > > > > I try to run pig on spark. But hit the following error. Could anyone > > help > > > > me on that ? And BTW where can I find document about how to use pig > on > > > > spark. > > > > > > > > > > > > 16/12/21 11:11:44 ERROR pig.Main: org.apache.pig.PigException: ERROR > > > 2040: > > > > Unknown exec type: SPARK > > > > at org.apache.pig.ExecTypeProvider.selectExecType( > > > > ExecTypeProvider.java:50) > > > > at org.apache.pig.impl.PigContext.<init>(PigContext.java:251) > > > > at org.apache.pig.Main.run(Main.java:390) > > > > at org.apache.pig.Main.main(Main.java:176) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > > > > > > > > >
