On Wed, Sep 26, 2012 at 04:06PM, Wing Yew Poon wrote: > Hi, > please correct me if I'm wrong, but I thought Spark runs on top of > Mesos. Does it not require Mesos to run?
Yes, you are right. That's why Mesos library is a part of Spark's dependencies. Now, there are 'on top of Mesos' and 'on Mesos cluster', which are quite different apparently ;) Cos > - Wing Yew > > On Wed, Sep 26, 2012 at 10:42 AM, MTG dev <[email protected]> wrote: > > Thanks Bruno! > > > > I have created BIGTOP-715 > > > > Cheers, > > MTG dev > > > > On Wed, Sep 26, 2012 at 01:19AM, Bruno MahИ wrote: > >> zOn 09/25/2012 10:46 AM, MTG dev wrote: > >> >Hi there. > >> > > >> >Apparently, I am not in a position to say what role Spark can play in the > >> >Bigtop for I am not speaking for neither of those projects. > >> > > >> >However, I can tell that Spark provides a number of the advantages > >> >compare to > >> >a traditional MapReduce model: stateful computational model with a need to > >> >write everything back to file system after step, in-memory calculations, > >> >higher level of primitives expressed in a functional language, etc. These > >> >advantages combined with low-latency planner result in a very significant > >> >performance improvement. I'd suggest to go over spark-project.org for more > >> >information. > >> > > >> >I am not an expert on Drill, but I'd say that Spark give immediate > >> >benefits > >> >over the former because it is already here and can be used by anyone ;) > >> > > >> >As for integration with Bigtop: Spark doesn't require any special > >> >integration > >> >with the rest of the stack - it might use HDFS as the underlying storage, > >> >but > >> >that's about it. > >> > > >> >Looks like there's an ongoing development to allow Spark to use Hive's > >> >SerDes, > >> >but I am not completely sure about its status. > >> > > >> >On Mon, Sep 24, 2012 at 09:59PM, Roman Shaposhnik wrote: > >> >>On Mon, Sep 24, 2012 at 8:52 PM, Anatoli Fomenko <[email protected]> > >> >>wrote: > >> >>>Hi Alef, > >> >>> > >> >>>Great news! > >> >>> > >> >>>Spark developers are interested in developing Spark packages and > >> >>>contributing them to open source. Since you already have them, > >> >>>what would you think about contributing the source to BigTop? > >> > > >> >We don't have any plans of holding the sources of the packages back, but > >> >we > >> >are working on rpm packaging right now. Once the work is over, we should > >> >be > >> >able to contribute it back to the community. Shall there be a JIRA ticket > >> >for > >> >that or something? > >> > > >> >With regards, > >> > Alef > >> > MTG dev team > >> > > >> > >> > >> Great news! > >> > >> > >> And yes, there should be a ticket. It will be helpful to organize > >> any work around it. > >> > >> Thanks, > >> Bruno > >> > >>
