Hi: For the question in the mail: do you happen to know what's the status of the spark branch, time line for it to be ready for merge with trunk?
Currently, the community is reviewing the code of spark branch. It has been several months since last review(2016/5/22) from the review board. I don’t know the end of the review process. @Rohini, can we drive it faster? Appreciate to get some feedback from you. Best Regards Kelly Zhang/Zhang,Liyun From: Adam Szita [mailto:sz...@cloudera.com] Sent: Wednesday, December 21, 2016 8:51 PM To: user@pig.apache.org Cc: Zhang, Liyun <liyun.zh...@intel.com>; Rohini Palaniswamy <rohini.adi...@gmail.com> Subject: Re: How to run pig on spark ? Hi Ronald, There is a jira/review case opened for the merge itself: PIG-4854<https://issues.apache.org/jira/browse/PIG-4854>, https://reviews.apache.org/r/45667/ As for exact timelines Liyun / Rohini might be able to help you out. Adam On 21 December 2016 at 10:37, Ronald Green <green.ron...@gmail.com<mailto:green.ron...@gmail.com>> wrote: Adam, do you happen to know what's the status of the spark branch, time line for it to be ready for merge with trunk? On Wed, Dec 21, 2016 at 10:37 AM, Adam Szita <sz...@cloudera.com<mailto:sz...@cloudera.com>> wrote: > Hi Canan, > > If you mean the trunk branch as https://github.com/apache/pig/tree/trunk/ > then no, Spark as execution engine is not available there yet, you'll have > to check out the actual 'spark' branch: > https://github.com/apache/pig/tree/spark/ > Also, what log entries can you see before the exception? > > On trunk the full log should be something like: > > *./pig -x spark_local* > Listening for transport dt_socket at address: 5005 > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ > 16/12/21 09:24:29 ERROR pig.Main: ERROR 2040: Unknown exec type: > spark_local > 16/12/21 09:24:29 WARN pig.Main: There is no log file to write to. > 16/12/21 09:24:29 ERROR pig.Main: org.apache.pig.PigException: ERROR 2040: > Unknown exec type: spark_local > ... > > > For the same command on spark branch you should see: > > *./pig -x spark_local* > Listening for transport dt_socket at address: 5005 > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : SPARK > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : SPARK_LOCAL > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Picked SPARK_LOCAL as the > ExecType > 2016-12-21 09:33:14,073 [main] INFO org.apache.pig.Main - Apache Pig > version 0.16.0-SNAPSHOT (rUnversioned directory) compiled Dec 21 2016, > 09:32:12 > ... > > Regards, > Adam > > On 21 December 2016 at 08:20, canan chen > <ccn...@gmail.com<mailto:ccn...@gmail.com>> wrote: > > > Thanks Divya, but this blog doesn't help me. I am using pig trunk which > > already has spark. > > > > And I use this command to build pig > > > > ant clean -Dhadoopversion=23 jar > > > > > > > > > > On Wed, Dec 21, 2016 at 2:54 PM Divya Gehlot > > <divya.htco...@gmail.com<mailto:divya.htco...@gmail.com>> > > wrote: > > > > > > > > http://blog.cloudera.com/blog/2014/09/pig-is-flying-apache- > > pig-on-apache-spark/ > > > > > > Hope this helps > > > > > > > > > Thanks, > > > Divya > > > > > > On 21 December 2016 at 11:13, canan chen > > > <ccn...@gmail.com<mailto:ccn...@gmail.com>> wrote: > > > > > > > I try to run pig on spark. But hit the following error. Could anyone > > help > > > > me on that ? And BTW where can I find document about how to use pig > on > > > > spark. > > > > > > > > > > > > 16/12/21 11:11:44 ERROR pig.Main: org.apache.pig.PigException: ERROR > > > 2040: > > > > Unknown exec type: SPARK > > > > at org.apache.pig.ExecTypeProvider.selectExecType( > > > > ExecTypeProvider.java:50) > > > > at org.apache.pig.impl.PigContext.<init>(PigContext.java:251) > > > > at org.apache.pig.Main.run(Main.java:390) > > > > at org.apache.pig.Main.main(Main.java:176) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java: > > > > 62) > > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > > > > > > > > >
