Happy new year everybody! I'm sure I'm not the only one looking forward to Pig on Spark. :) Is there a clear timeline for it?
On Thu, Dec 22, 2016 at 7:20 AM, Zhang, Liyun <liyun.zh...@intel.com> wrote: > Hi: > For the question in the mail: > do you happen to know what's the status of the spark branch, time line for > it to be ready for merge with trunk? > > Currently, the community is reviewing the code of spark branch. It has > been several months since last review(2016/5/22) from the review board. > I don’t know the end of the review process. @Rohini, can we drive it > faster? Appreciate to get some feedback from you. > > > Best Regards > Kelly Zhang/Zhang,Liyun > > > > From: Adam Szita [mailto:sz...@cloudera.com] > Sent: Wednesday, December 21, 2016 8:51 PM > To: user@pig.apache.org > Cc: Zhang, Liyun <liyun.zh...@intel.com>; Rohini Palaniswamy < > rohini.adi...@gmail.com> > Subject: Re: How to run pig on spark ? > > Hi Ronald, > > There is a jira/review case opened for the merge itself: PIG-4854< > https://issues.apache.org/jira/browse/PIG-4854>, > https://reviews.apache.org/r/45667/ > As for exact timelines Liyun / Rohini might be able to help you out. > > Adam > > On 21 December 2016 at 10:37, Ronald Green <green.ron...@gmail.com<mailto: > green.ron...@gmail.com>> wrote: > Adam, > > do you happen to know what's the status of the spark branch, time line for > it to be ready for merge with trunk? > > On Wed, Dec 21, 2016 at 10:37 AM, Adam Szita <sz...@cloudera.com<mailto: > sz...@cloudera.com>> wrote: > > > Hi Canan, > > > > If you mean the trunk branch as https://github.com/apache/pig/ > tree/trunk/ > > then no, Spark as execution engine is not available there yet, you'll > have > > to check out the actual 'spark' branch: > > https://github.com/apache/pig/tree/spark/ > > Also, what log entries can you see before the exception? > > > > On trunk the full log should be something like: > > > > *./pig -x spark_local* > > Listening for transport dt_socket at address: 5005 > > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE > > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL > > 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ > > 16/12/21 09:24:29 ERROR pig.Main: ERROR 2040: Unknown exec type: > > spark_local > > 16/12/21 09:24:29 WARN pig.Main: There is no log file to write to. > > 16/12/21 09:24:29 ERROR pig.Main: org.apache.pig.PigException: ERROR > 2040: > > Unknown exec type: spark_local > > ... > > > > > > For the same command on spark branch you should see: > > > > *./pig -x spark_local* > > Listening for transport dt_socket at address: 5005 > > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE > > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : SPARK > > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : > SPARK_LOCAL > > 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Picked SPARK_LOCAL as the > > ExecType > > 2016-12-21 09:33:14,073 [main] INFO org.apache.pig.Main - Apache Pig > > version 0.16.0-SNAPSHOT (rUnversioned directory) compiled Dec 21 2016, > > 09:32:12 > > ... > > > > Regards, > > Adam > > > > On 21 December 2016 at 08:20, canan chen <ccn...@gmail.com<mailto:ccnfd > u...@gmail.com>> wrote: > > > > > Thanks Divya, but this blog doesn't help me. I am using pig trunk which > > > already has spark. > > > > > > And I use this command to build pig > > > > > > ant clean -Dhadoopversion=23 jar > > > > > > > > > > > > > > > On Wed, Dec 21, 2016 at 2:54 PM Divya Gehlot <divya.htco...@gmail.com< > mailto:divya.htco...@gmail.com>> > > > wrote: > > > > > > > > > > > http://blog.cloudera.com/blog/2014/09/pig-is-flying-apache- > > > pig-on-apache-spark/ > > > > > > > > Hope this helps > > > > > > > > > > > > Thanks, > > > > Divya > > > > > > > > On 21 December 2016 at 11:13, canan chen <ccn...@gmail.com<mailto: > ccn...@gmail.com>> wrote: > > > > > > > > > I try to run pig on spark. But hit the following error. Could > anyone > > > help > > > > > me on that ? And BTW where can I find document about how to use pig > > on > > > > > spark. > > > > > > > > > > > > > > > 16/12/21 11:11:44 ERROR pig.Main: org.apache.pig.PigException: > ERROR > > > > 2040: > > > > > Unknown exec type: SPARK > > > > > at org.apache.pig.ExecTypeProvider.selectExecType( > > > > > ExecTypeProvider.java:50) > > > > > at org.apache.pig.impl.PigContext.<init>(PigContext.java:251) > > > > > at org.apache.pig.Main.run(Main.java:390) > > > > > at org.apache.pig.Main.main(Main.java:176) > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > at > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java: > > > > > 62) > > > > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > > DelegatingMethodAccessorImpl.java:43) > > > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > > > > > > > > > > > > > > >
