Re: PIG to Spark

[Gourav Sengupta]

I may be wrong here, but when I see github and apache pig, it says that
there are 8 contributors, and when I see github and look at apache spark it
says there are more than 1000 contributors. And if the above is true I ask
myself, why not shift to SPARK by learning it?

I also started with map reduce JAVA nightmarish coding, and then HIVE and
then Pig and then realised the best time spent is the time used for
actually solving data problems than programming problems. I know
consultants who have ended up convincing their clients that its better to
write JAVA programs than use SPARK SQL and then have spent close to 2.5
years not being able to deliver anything that works where the actual
project was just a single SPARK SQL.

Personally I prefer to learn and adapt and transfer existing code to a
platform that gives me the maximum business benefit with least headaches.

But once again it is just a matter of opinion.


Regards,
Gourav Sengupta

On Mon, Jan 8, 2018 at 3:25 PM, Pralabh Kumar 
wrote:

> Hi
>
> Is there a convenient way /open source project to convert PIG scripts to
> Spark.
>
>
> Regards
> Pralabh Kumar
>

Re: PIG to Spark

[Jeff Zhang]

Pig support spark engine now, so you can leverage spark execution with pig
script.

I am afraid there's no solution to convert pig script to spark api code





Pralabh Kumar 于2018年1月8日周一 下午11:25写道：

> Hi
>
> Is there a convenient way /open source project to convert PIG scripts to
> Spark.
>
>
> Regards
> Pralabh Kumar
>

Re: PIG to SPARK

[suman bharadwaj]

Thanks Mayur. I don't have clear idea on how pipe works wanted to
understand more on it. But when do we use pipe() and how it works ?. Can
you please share some sample code if you have ( even pseudo-code is fine )
? It will really help.

Regards,
Suman Bharadwaj S


On Thu, Mar 6, 2014 at 3:46 AM, Mayur Rustagi wrote:

> The real question is why do you want to run pig script using Spark
> Are you planning to user spark as underlying processing engine for Spark?
> thats not simple
> Are you planning to feed Pig data to spark for further processing, then
> you can write it to HDFS & trigger your spark script.
>
> rdd.pipe is basically similar to Hadoop streaming, allowing you to run a
> script on each partition of the RDD & get output as another RDD.
> Regards
> Mayur
>
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi 
>
>
>
> On Wed, Mar 5, 2014 at 10:29 AM, suman bharadwaj wrote:
>
>> Hi,
>>
>> How can i call pig script using SPARK. Can I use rdd.pipe() here ?
>>
>> And can anyone share sample implementation of rdd.pipe () and if you can
>> explain how rdd.pipe() works, it would really really help.
>>
>> Regards,
>> SB
>>
>
>

Re: PIG to SPARK

[Mayur Rustagi]

The real question is why do you want to run pig script using Spark
Are you planning to user spark as underlying processing engine for Spark?
thats not simple
Are you planning to feed Pig data to spark for further processing, then you
can write it to HDFS & trigger your spark script.

rdd.pipe is basically similar to Hadoop streaming, allowing you to run a
script on each partition of the RDD & get output as another RDD.
Regards
Mayur


Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi 



On Wed, Mar 5, 2014 at 10:29 AM, suman bharadwaj wrote:

> Hi,
>
> How can i call pig script using SPARK. Can I use rdd.pipe() here ?
>
> And can anyone share sample implementation of rdd.pipe () and if you can
> explain how rdd.pipe() works, it would really really help.
>
> Regards,
> SB
>