it is just a matter of opinion.
Regards,
Gourav Sengupta
On Mon, Jan 8, 2018 at 3:25 PM, Pralabh Kumar <pralabhku...@gmail.com>
wrote:
> Hi
>
> Is there a convenient way /open source project to convert PIG scripts to
> Spark.
>
>
> Regards
> Pralabh Kumar
>
Pig support spark engine now, so you can leverage spark execution with pig
script.
I am afraid there's no solution to convert pig script to spark api code
Pralabh Kumar <pralabhku...@gmail.com>于2018年1月8日周一 下午11:25写道：
> Hi
>
> Is there a convenient way /open source project
Hi
Is there a convenient way /open source project to convert PIG scripts to
Spark.
Regards
Pralabh Kumar
>
> Hi,
As a beginner ,I have below queries on Spork(Pig on Spark).
I have cloned git clone https://github.com/apache/pig -b spark .
1.On which version of Pig and Spark , Spork is being built ?
2. I followed the steps mentioned in https://issues.apache.org/ji
ra/browse/PIG-4059 a
com> wrote:
> >>> Details at logfile: /home/pig/pig_1448425672112.log
>
> You need to check the log file for details
>
>
>
>
> On Wed, Nov 25, 2015 at 1:57 PM, Divya Gehlot <divya.htco...@gmail.com>
> wrote:
>
>> Hi,
>>
>>
>> As a b
Analytics), Aniket
Mokashi (Google), Greg Owen (DataBricks), Amit Kumar Behera (Sigmoid
Analytics), Mahesh Kalakoti (Sigmoid Analytics)
Not to mention Spark Pig communities.
Regards
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi https://twitter.com
(Sigmoid Analytics)
Not to mention Spark Pig communities.
Regards
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi https://twitter.com/mayur_rustagi
Banga
(Sigmoid Analytics), Anish Haldiya (Sigmoid Analytics), Aniket Mokashi
(Google), Greg Owen (DataBricks), Amit Kumar Behera (Sigmoid Analytics), Mahesh
Kalakoti (Sigmoid Analytics)
Not to mention Spark Pig communities.
Regards
Mayur Rustagi
Ph: +1 (760) 203 3257
http
-- 原始邮件 --
*发件人:* Mayur Rustagi;mayur.rust...@gmail.com;
*发送时间:* 2014年7月7日(星期一) 晚上11:55
*收件人:* user@spark.apache.orguser@spark.apache.org;
*主题:* Re: Pig 0.13, Spark, Spork
That version is old :).
We are not forking pig but cleanly separating out pig execution
Hi,
I was wondering what was the state of the Pig+Spark initiative now that the
execution engine of Pig is pluggable? Granted, it was done in order to use
Tez but could it be used by Spark? I know about a 'theoretical' project
called Spork but I don't know any stable and maintained version
@mayur_rustagi https://twitter.com/mayur_rustagi
On Mon, Jul 7, 2014 at 8:21 PM, Bertrand Dechoux decho...@gmail.com wrote:
Hi,
I was wondering what was the state of the Pig+Spark initiative now that
the execution engine of Pig is pluggable? Granted, it was done in order to
use Tez but could
it yet but I am interested in the state of the
initiative.
What's your point of view (personal and/or professional) about the Pig
0.13 release?
Is the pluggable execution engine flexible enough in order to avoid having
Spork as a fork of Pig? Pig + Spark + Fork = Spork :D
As a (for now) external
...@gmail.com;
发送时间: 2014年7月7日(星期一) 晚上11:55
收件人: user@spark.apache.orguser@spark.apache.org;
主题: Re: Pig 0.13, Spark, Spork
That version is old :). We are not forking pig but cleanly separating out pig
execution engine. Let me know if you are willing to give it a go.
Also would love to know
on data flow
2. Some have codebases in Pig already are just looking to do it
faster. I am yet to benchmark that on Pig on spark.
I agree that pig on spark cannot solve a lot problems but it can solve
some without forcing the end customer to do anything even close to coding,
I believe thr
I've only had a quick look at Pig, but it seems that a declarative
layer on top of Spark couldn't be anything other than a big win, as it
allows developers to declare *what* they want, permitting the compiler
to determine how best poke at the RDD API to implement it.
In my brief time with Spark,
It depends, personally I have the opposite opinion.
IMO expressing pipelines in a functional language feels natural, you just
have to get used with the language (scala).
Testing spark jobs is easy where testing a Pig script is much harder and
not natural.
If you want a more high level language
I've only had a quick look at Pig, but it seems that a declarative
layer on top of Spark couldn't be anything other than a big win, as it
allows developers to declare *what* they want, permitting the compiler
to determine how best poke at the RDD API to implement it.
The devil is in the
/Pig-on-Spark-tp2367p4668.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
semantics away from the user lets him
focus on data flow
2. Some have codebases in Pig already are just looking to do it faster. I
am yet to benchmark that on Pig on spark.
I agree that pig on spark cannot solve a lot problems but it can solve some
without forcing the end customer to do anything even
.
I am yet to benchmark that on Pig on spark.
I agree that pig on spark cannot solve a lot problems but it can solve
some without forcing the end customer to do anything even close to coding,
I believe thr is quite some value in making Spark accessible to larger
group of audience.
End
semantics away from the user lets him
focus on data flow
2. Some have codebases in Pig already are just looking to do it faster.
I am yet to benchmark that on Pig on spark.
I agree that pig on spark cannot solve a lot problems but it can solve
some without forcing the end customer to do anything
are just looking to do it
faster. I am yet to benchmark that on Pig on spark.
I agree that pig on spark cannot solve a lot problems but it can solve
some without forcing the end customer to do anything even close to coding,
I believe thr is quite some value in making Spark accessible to larger
semantics away from the user
lets him focus on data flow
2. Some have codebases in Pig already are just looking to do it
faster. I am yet to benchmark that on Pig on spark.
I agree that pig on spark cannot solve a lot problems but it can solve
some without forcing the end customer to do anything
on Pig.
Finally got it working. Couple of high level bugs right now:
- Getting it working on Spark 0.9.0
- Getting UDF working
- Getting generate functionality working
- Exhaustive test suite on Spark on Pig
are you maintaining a Jira somewhere?
I am currently trying to deploy
for all the work on Pig.
Finally got it working. Couple of high level bugs right now:
- Getting it working on Spark 0.9.0
- Getting UDF working
- Getting generate functionality working
- Exhaustive test suite on Spark on Pig
are you maintaining a Jira somewhere?
I am currently
Hi Ankit,
Thanx for all the work on Pig.
Finally got it working. Couple of high level bugs right now:
- Getting it working on Spark 0.9.0
- Getting UDF working
- Getting generate functionality working
- Exhaustive test suite on Spark on Pig
are you maintaining a Jira somewhere?
I am
Hi,
I have been following Aniket's spork github repository.
https://github.com/aniket486/pig
I have done all the changes mentioned in recently modified pig-spark file.
I am using:
hadoop 2.0.5 alpha
spark-0.8.1-incubating
mesos 0.16.0
##PIG variables
export *HADOOP_CONF_DIR*=$HADOOP_INSTALL/etc
Hi Mayur,We are planning to upgrade our distribution MR1 MR2 (YARN) and the
goal is to get SPROK set up next month. I will keep you posted. Can you please
keep me informed about your progress as well.
From: mayur.rust...@gmail.com
Date: Mon, 10 Mar 2014 11:47:56 -0700
Subject: Re: Pig on Spark
, Mayur Rustagi mayur.rust...@gmail.comwrote:
The real question is why do you want to run pig script using Spark
Are you planning to user spark as underlying processing engine for Spark?
thats not simple
Are you planning to feed Pig data to spark for further processing, then
you can write
Hi everyone,
We are using to Pig to build our data pipeline. I came across Spork -- Pig on
Spark at: https://github.com/dvryaboy/pig and not sure if it is still active.
Can someone please let me know the status of Spork or any other effort that
will let us run Pig on Spark? We can
to Pig to build our data pipeline. I came across Spork -- Pig on
Spark at: https://github.com/dvryaboy/pig and not sure if it is still active.
Can someone please let me know the status of Spork or any other effort that
will let us run Pig on Spark? We can significantly benefit by using Spark
There is some work to make this work on yarn at
https://github.com/aniket486/pig. (So, compile pig with ant
-Dhadoopversion=23)
You can look at https://github.com/aniket486/pig/blob/spork/pig-spark to
find out what sort of env variables you need (sorry, I haven't been able to
clean this up
Hi Aniket,Many thanks! I will check this out.
Date: Thu, 6 Mar 2014 13:46:50 -0800
Subject: Re: Pig on Spark
From: aniket...@gmail.com
To: user@spark.apache.org; tgraves...@yahoo.com
There is some work to make this work on yarn at
https://github.com/aniket486/pig. (So, compile pig with ant
The real question is why do you want to run pig script using Spark
Are you planning to user spark as underlying processing engine for Spark?
thats not simple
Are you planning to feed Pig data to spark for further processing, then you
can write it to HDFS trigger your spark script.
rdd.pipe
34 matches
Mail list logo
