Hi guys, previously I checked out the old "spork" and updated it to Hadoop 2.0, Scala 2.10.3 and Spark 0.9.1, see github project of mine https://github.com/pelick/flare-spork
It it also highly experimental, and just directly mapping pig physical operations to spark RDD transformations/actions. It works for simple requests. :) I am also interested on the progress of spork, is it undergoing in Twitter in an un open-source way? ------------------ Thanks Zhang Baofeng Blog | Github | Weibo | LinkedIn ------------------ 原始邮件 ------------------ 发件人: "Mayur Rustagi";<mayur.rust...@gmail.com>; 发送时间: 2014年7月7日(星期一) 晚上11:55 收件人: "user@spark.apache.org"<user@spark.apache.org>; 主题: Re: Pig 0.13, Spark, Spork That version is old :). We are not forking pig but cleanly separating out pig execution engine. Let me know if you are willing to give it a go. Also would love to know what features of pig you are using ? Regards Mayur Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Mon, Jul 7, 2014 at 8:46 PM, Bertrand Dechoux <decho...@gmail.com> wrote: I saw a wiki page from your company but with an old version of Spark.http://docs.sigmoidanalytics.com/index.php/Setting_up_spork_with_spark_0.8.1 I have no reason to use it yet but I am interested in the state of the initiative. What's your point of view (personal and/or professional) about the Pig 0.13 release? Is the pluggable execution engine flexible enough in order to avoid having Spork as a fork of Pig? Pig + Spark + Fork = Spork :D As a (for now) external observer, I am glad to see competition in that space. It can only be good for the community in the end. Bertrand Dechoux On Mon, Jul 7, 2014 at 5:00 PM, Mayur Rustagi <mayur.rust...@gmail.com> wrote: Hi,We have fixed many major issues around Spork & deploying it with some customers. Would be happy to provide a working version to you to try out. We are looking for more folks to try it out & submit bugs. Regards Mayur Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Mon, Jul 7, 2014 at 8:21 PM, Bertrand Dechoux <decho...@gmail.com> wrote: Hi, I was wondering what was the state of the Pig+Spark initiative now that the execution engine of Pig is pluggable? Granted, it was done in order to use Tez but could it be used by Spark? I know about a 'theoretical' project called Spork but I don't know any stable and maintained version of it. Regards Bertrand Dechoux
