zOn 09/25/2012 10:46 AM, MTG dev wrote:
Hi there.
Apparently, I am not in a position to say what role Spark can play in the
Bigtop for I am not speaking for neither of those projects.
However, I can tell that Spark provides a number of the advantages compare to
a traditional MapReduce model: stateful computational model with a need to
write everything back to file system after step, in-memory calculations,
higher level of primitives expressed in a functional language, etc. These
advantages combined with low-latency planner result in a very significant
performance improvement. I'd suggest to go over spark-project.org for more
information.
I am not an expert on Drill, but I'd say that Spark give immediate benefits
over the former because it is already here and can be used by anyone ;)
As for integration with Bigtop: Spark doesn't require any special integration
with the rest of the stack - it might use HDFS as the underlying storage, but
that's about it.
Looks like there's an ongoing development to allow Spark to use Hive's SerDes,
but I am not completely sure about its status.
On Mon, Sep 24, 2012 at 09:59PM, Roman Shaposhnik wrote:
On Mon, Sep 24, 2012 at 8:52 PM, Anatoli Fomenko <[email protected]> wrote:
Hi Alef,
Great news!
Spark developers are interested in developing Spark packages and
contributing them to open source. Since you already have them,
what would you think about contributing the source to BigTop?
We don't have any plans of holding the sources of the packages back, but we
are working on rpm packaging right now. Once the work is over, we should be
able to contribute it back to the community. Shall there be a JIRA ticket for
that or something?
With regards,
Alef
MTG dev team
Great news!
And yes, there should be a ticket. It will be helpful to organize any
work around it.
Thanks,
Bruno
