I'd like to comment on what has been said about Twill. >> Twill: handles all the AM logic for running new code packaged as a JAR with >> an executor method
The goal of Twill is much broader than to support writing new code. Its goal is to ease the development and deployment of any distributed service in a cluster. Its impact is most significantly for the development of new applications but here are some other aspects: - Twill can run existing Java applications, even if they were not developed against Twill APIs and do not make use of Twill's advanced features, After all the existing main() method of Java programs bears a lot of resemblance to a runnable. For example, we have successfully run Presto on Twill, and the same should be possible for other distributed services written in Java (and non-Java language support is on the horizon for Twill as well). - Twill does provide very useful features that most distributed apps need (such as service discovery, lifecycle management, elastic scaling, etc.). Even if the first step may be to port an existing applications without modification, it is very much possible that these applications - over time - make use of Twill features. - Twill is not limited to YARN, in fact its APIs are completely YARN-independent, and in addition to a YARN-based implementation, there can well be others - for example, it is not far--fetched to think of a MesosTwillRunner. In this sense, Twill is more generic than Slider, which is optimized specifically for a few existing distributed services. At this time, Slider is probably more optimized for running - say - HBase or Accumulo than Twill. Yet most likely Twill will eventually have all the required features, and wouldn't it be better if things like ZK-bindings were done in a widely used and common way? Twill has the potential to lead us there. In the meantime, Slider can fill the gap. Cheers -Andreas. On Sat, Apr 12, 2014 at 3:38 PM, Roman Shaposhnik <r...@apache.org> wrote: > On Sat, Apr 12, 2014 at 11:58 AM, Andrew Purtell <apurt...@apache.org> > wrote: > >> > The reason I ask is I'm wondering how Slider differentiates from > projects > >> > like Apache Twill or Apache Bigtop that are already existing vehicles > for > >> > achieving the aims discussed in the Slider proposal. > >> > >> > >> Twill: handles all the AM logic for running new code packaged as a JAR > with > >> an executor method > >> Bigtop: stack testing > > > > > > As a Bigtop committer, I disagree with this narrow interpretation of the > > scope of the project, but this is my personal opinion and I am not PMC... > > A strong +1 here! Bigtop attacks a problem of packaging and deploying > Hadoop stacks from a classical UNIX packaging background. We are > also slowly moving into container/VM/OSv packaging territory which > could be an extremely exciting way of side-stepping the general installer > issues (something that Ambari struggles mightily with). > > Something like Slider tries to leapfrog and side-step UNIX packaging and > deployment altogether. This is an interesting take on the problem, but > ultimately the jury is still out on what the level of adoption for > "everything > is now YARN" will be. > > At the end of the day, we will need both for a really long time. > > > For example, we package Hadoop core and ecosystem services both for > > deployment, have Puppet based deployment automation (which can be used > more > > generally than merely for setting up test clusters), and I have been > > considering filing JIRAs to tie in cgroups at the whole stack level here. > > What is missing of course is a hierarchical model for resource > management, > > and tools within the components for differentiated service levels, but > that > > is another discussion. > > On that note, I find YARN's attitude towards cgroups be, how shall I put > it, > optimistic. If you look carefully you can see that the Linux community > has completely given up an pretending that one can use naked cgroup > trees for reliable resource partitioning: > http://www.freedesktop.org/wiki/Software/systemd/PaxControlGroups/ > > It is now clear to me that the path Linux distros are endorsing is via > the brokers such as systemd: > > http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/ > > As such I'd see Bigtop providing quite a bit of value out of the box > via tight integration with systemd. YARN, at the moment, is in a trickier > situation. > > > HBase and accumulo do have their own ZK binding mechanism, so don't > really > >> need their own registry. But to work with their data you do need the > >> relevant client apps. I would like to have some standard for at least > >> publishing the core binding information in a way that could be parsed by > >> any client app (CLI, web UI, other in-cluster apps) > > > > > > +1 to such a standard. > > I've been known to advocate Apache Helix as a standard API to build > exactly that type of distributed application architecture. In fact, if only > I had more spare time on my hand, I'd totally prototype Helix-based > YARN APIs. > > Thanks, > Roman. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
