On Sat, Apr 12, 2014 at 11:58 AM, Andrew Purtell <apurt...@apache.org> wrote: >> > The reason I ask is I'm wondering how Slider differentiates from projects >> > like Apache Twill or Apache Bigtop that are already existing vehicles for >> > achieving the aims discussed in the Slider proposal. >> >> >> Twill: handles all the AM logic for running new code packaged as a JAR with >> an executor method >> Bigtop: stack testing > > > As a Bigtop committer, I disagree with this narrow interpretation of the > scope of the project, but this is my personal opinion and I am not PMC...
A strong +1 here! Bigtop attacks a problem of packaging and deploying Hadoop stacks from a classical UNIX packaging background. We are also slowly moving into container/VM/OSv packaging territory which could be an extremely exciting way of side-stepping the general installer issues (something that Ambari struggles mightily with). Something like Slider tries to leapfrog and side-step UNIX packaging and deployment altogether. This is an interesting take on the problem, but ultimately the jury is still out on what the level of adoption for "everything is now YARN" will be. At the end of the day, we will need both for a really long time. > For example, we package Hadoop core and ecosystem services both for > deployment, have Puppet based deployment automation (which can be used more > generally than merely for setting up test clusters), and I have been > considering filing JIRAs to tie in cgroups at the whole stack level here. > What is missing of course is a hierarchical model for resource management, > and tools within the components for differentiated service levels, but that > is another discussion. On that note, I find YARN's attitude towards cgroups be, how shall I put it, optimistic. If you look carefully you can see that the Linux community has completely given up an pretending that one can use naked cgroup trees for reliable resource partitioning: http://www.freedesktop.org/wiki/Software/systemd/PaxControlGroups/ It is now clear to me that the path Linux distros are endorsing is via the brokers such as systemd: http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/ As such I'd see Bigtop providing quite a bit of value out of the box via tight integration with systemd. YARN, at the moment, is in a trickier situation. > HBase and accumulo do have their own ZK binding mechanism, so don't really >> need their own registry. But to work with their data you do need the >> relevant client apps. I would like to have some standard for at least >> publishing the core binding information in a way that could be parsed by >> any client app (CLI, web UI, other in-cluster apps) > > > +1 to such a standard. I've been known to advocate Apache Helix as a standard API to build exactly that type of distributed application architecture. In fact, if only I had more spare time on my hand, I'd totally prototype Helix-based YARN APIs. Thanks, Roman. --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org