Re: PIG to Spark

I may be wrong here, but when I see github and apache pig, it says that there are 8 contributors, and when I see github and look at apache spark it says there are more than 1000 contributors. And if the above is true I ask myself, why not shift to SPARK by learning it? I also started with map

Re: PIG to Spark

Pig support spark engine now, so you can leverage spark execution with pig script. I am afraid there's no solution to convert pig script to spark api code Pralabh Kumar 于2018年1月8日周一 下午11:25写道： > Hi > > Is there a convenient way /open source project to convert PIG

Re: Re： Pig 0.13, Spark, Spork

-- 原始邮件 -- *发件人:* Mayur Rustagi;mayur.rust...@gmail.com; *发送时间:* 2014年7月7日(星期一) 晚上11:55 *收件人:* user@spark.apache.orguser@spark.apache.org; *主题:* Re: Pig 0.13, Spark, Spork That version is old :). We are not forking pig but cleanly separating out pig execution

Re: Pig 0.13, Spark, Spork

Hi, We have fixed many major issues around Spork deploying it with some customers. Would be happy to provide a working version to you to try out. We are looking for more folks to try it out submit bugs. Regards Mayur Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com

Re: Pig 0.13, Spark, Spork

That version is old :). We are not forking pig but cleanly separating out pig execution engine. Let me know if you are willing to give it a go. Also would love to know what features of pig you are using ? Regards Mayur Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com

Re： Pig 0.13, Spark, Spork

...@gmail.com; 发送时间: 2014年7月7日(星期一) 晚上11:55 收件人: user@spark.apache.orguser@spark.apache.org; 主题: Re: Pig 0.13, Spark, Spork That version is old :). We are not forking pig but cleanly separating out pig execution engine. Let me know if you are willing to give it a go. Also would love to know

Re: Pig on Spark

posted. Can you please keep me informed about your progress as well. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you make any progress on this. My team

Re: Pig on Spark

I've only had a quick look at Pig, but it seems that a declarative layer on top of Spark couldn't be anything other than a big win, as it allows developers to declare *what* they want, permitting the compiler to determine how best poke at the RDD API to implement it. In my brief time with Spark,

Re: Pig on Spark

It depends, personally I have the opposite opinion. IMO expressing pipelines in a functional language feels natural, you just have to get used with the language (scala). Testing spark jobs is easy where testing a Pig script is much harder and not natural. If you want a more high level language

Re: Pig on Spark

I've only had a quick look at Pig, but it seems that a declarative layer on top of Spark couldn't be anything other than a big win, as it allows developers to declare *what* they want, permitting the compiler to determine how best poke at the RDD API to implement it. The devil is in the

Re: Pig on Spark

Hi, We got spork working on spark 0.9.0 Repository available at: https://github.com/sigmoidanalytics/pig/tree/spork-hadoopasm-fix Please suggest your feedback. - Lalit Yadav la...@sigmoidanalytics.com -- View this message in context:

Re: Pig on Spark

will keep you posted. Can you please keep me informed about your progress as well. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you make any progress

Re: Pig on Spark

. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you make any progress on this. My team is also trying it out would love to know some detail so progress. Mayur Rustagi Ph: +1

Re: Pig on Spark

set up next month. I will keep you posted. Can you please keep me informed about your progress as well. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you

Re: Pig on Spark

as well. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you make any progress on this. My team is also trying it out would love to know some detail so

Re: Pig on Spark

set up next month. I will keep you posted. Can you please keep me informed about your progress as well. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you

Re: Pig on Spark

. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you make any progress on this. My team is also trying it out would love to know some detail so progress. Mayur Rustagi Ph: +1 (760) 203

Re: Pig on Spark

. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you make any progress on this. My team is also trying it out would love to know some detail so progress. Mayur Rustagi Ph: +1

Re: Pig on Spark

. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark To: user@spark.apache.org Hi Sameer, Did you make any progress on this. My team is also trying it out would love to know some detail so progress. Mayur Rustagi

Re: Pig on Spark

Hi, I have been following Aniket's spork github repository. https://github.com/aniket486/pig I have done all the changes mentioned in recently modified pig-spark file. I am using: hadoop 2.0.5 alpha spark-0.8.1-incubating mesos 0.16.0 ##PIG variables export

RE: Pig on Spark

Hi Mayur,We are planning to upgrade our distribution MR1 MR2 (YARN) and the goal is to get SPROK set up next month. I will keep you posted. Can you please keep me informed about your progress as well. From: mayur.rust...@gmail.com Date: Mon, 10 Mar 2014 11:47:56 -0700 Subject: Re: Pig on Spark

Re: PIG to SPARK

Thanks Mayur. I don't have clear idea on how pipe works wanted to understand more on it. But when do we use pipe() and how it works ?. Can you please share some sample code if you have ( even pseudo-code is fine ) ? It will really help. Regards, Suman Bharadwaj S On Thu, Mar 6, 2014 at 3:46 AM,

Re: Pig on Spark

I had asked a similar question on the dev mailing list a while back (Jan 22nd).  See the archives:  http://mail-archives.apache.org/mod_mbox/spark-dev/201401.mbox/browser - look for spork. Basically Matei said: Yup, that was it, though I believe people at Twitter picked it up again recently.

Re: Pig on Spark

There is some work to make this work on yarn at https://github.com/aniket486/pig. (So, compile pig with ant -Dhadoopversion=23) You can look at https://github.com/aniket486/pig/blob/spork/pig-spark to find out what sort of env variables you need (sorry, I haven't been able to clean this up-

RE: Pig on Spark

Hi Aniket,Many thanks! I will check this out. Date: Thu, 6 Mar 2014 13:46:50 -0800 Subject: Re: Pig on Spark From: aniket...@gmail.com To: user@spark.apache.org; tgraves...@yahoo.com There is some work to make this work on yarn at https://github.com/aniket486/pig. (So, compile pig with ant

Re: PIG to SPARK

The real question is why do you want to run pig script using Spark Are you planning to user spark as underlying processing engine for Spark? thats not simple Are you planning to feed Pig data to spark for further processing, then you can write it to HDFS trigger your spark script. rdd.pipe is