Hi, please correct me if I'm wrong, but I thought Spark runs on top of Mesos. Does it not require Mesos to run? - Wing Yew
On Wed, Sep 26, 2012 at 10:42 AM, MTG dev <[email protected]> wrote: > Thanks Bruno! > > I have created BIGTOP-715 > > Cheers, > MTG dev > > On Wed, Sep 26, 2012 at 01:19AM, Bruno Mahé wrote: >> zOn 09/25/2012 10:46 AM, MTG dev wrote: >> >Hi there. >> > >> >Apparently, I am not in a position to say what role Spark can play in the >> >Bigtop for I am not speaking for neither of those projects. >> > >> >However, I can tell that Spark provides a number of the advantages compare >> >to >> >a traditional MapReduce model: stateful computational model with a need to >> >write everything back to file system after step, in-memory calculations, >> >higher level of primitives expressed in a functional language, etc. These >> >advantages combined with low-latency planner result in a very significant >> >performance improvement. I'd suggest to go over spark-project.org for more >> >information. >> > >> >I am not an expert on Drill, but I'd say that Spark give immediate benefits >> >over the former because it is already here and can be used by anyone ;) >> > >> >As for integration with Bigtop: Spark doesn't require any special >> >integration >> >with the rest of the stack - it might use HDFS as the underlying storage, >> >but >> >that's about it. >> > >> >Looks like there's an ongoing development to allow Spark to use Hive's >> >SerDes, >> >but I am not completely sure about its status. >> > >> >On Mon, Sep 24, 2012 at 09:59PM, Roman Shaposhnik wrote: >> >>On Mon, Sep 24, 2012 at 8:52 PM, Anatoli Fomenko <[email protected]> >> >>wrote: >> >>>Hi Alef, >> >>> >> >>>Great news! >> >>> >> >>>Spark developers are interested in developing Spark packages and >> >>>contributing them to open source. Since you already have them, >> >>>what would you think about contributing the source to BigTop? >> > >> >We don't have any plans of holding the sources of the packages back, but we >> >are working on rpm packaging right now. Once the work is over, we should be >> >able to contribute it back to the community. Shall there be a JIRA ticket >> >for >> >that or something? >> > >> >With regards, >> > Alef >> > MTG dev team >> > >> >> >> Great news! >> >> >> And yes, there should be a ticket. It will be helpful to organize >> any work around it. >> >> Thanks, >> Bruno >> >>
