On Sat, Apr 12, 2014 at 11:58 AM, Andrew Purtell <apurt...@apache.org> wrote:
>> > The reason I ask is I'm wondering how Slider differentiates from projects
>> > like Apache Twill or Apache Bigtop that are already existing vehicles for
>> > achieving the aims discussed in the Slider proposal.
>>
>>
>> Twill: handles all the AM logic for running new code packaged as a JAR with
>> an executor method
>> Bigtop: stack testing
>
>
> As a Bigtop committer, I disagree with this narrow interpretation of the
> scope of the project, but this is my personal opinion and I am not PMC...
A strong +1 here! Bigtop attacks a problem of packaging and deploying
Hadoop stacks from a classical UNIX packaging background. We are
also slowly moving into container/VM/OSv packaging territory which
could be an extremely exciting way of side-stepping the general installer
issues (something that Ambari struggles mightily with).
Something like Slider tries to leapfrog and side-step UNIX packaging and
deployment altogether. This is an interesting take on the problem, but
ultimately the jury is still out on what the level of adoption for "everything
is now YARN" will be.
At the end of the day, we will need both for a really long time.
> For example, we package Hadoop core and ecosystem services both for
> deployment, have Puppet based deployment automation (which can be used more
> generally than merely for setting up test clusters), and I have been
> considering filing JIRAs to tie in cgroups at the whole stack level here.
> What is missing of course is a hierarchical model for resource management,
> and tools within the components for differentiated service levels, but that
> is another discussion.
On that note, I find YARN's attitude towards cgroups be, how shall I put it,
optimistic. If you look carefully you can see that the Linux community
has completely given up an pretending that one can use naked cgroup
trees for reliable resource partitioning:
http://www.freedesktop.org/wiki/Software/systemd/PaxControlGroups/
It is now clear to me that the path Linux distros are endorsing is via
the brokers such as systemd:
http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
As such I'd see Bigtop providing quite a bit of value out of the box
via tight integration with systemd. YARN, at the moment, is in a trickier
situation.
> HBase and accumulo do have their own ZK binding mechanism, so don't really
>> need their own registry. But to work with their data you do need the
>> relevant client apps. I would like to have some standard for at least
>> publishing the core binding information in a way that could be parsed by
>> any client app (CLI, web UI, other in-cluster apps)
>
>
> +1 to such a standard.
I've been known to advocate Apache Helix as a standard API to build
exactly that type of distributed application architecture. In fact, if only
I had more spare time on my hand, I'd totally prototype Helix-based
YARN APIs.
Thanks,
Roman.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org
