Well put, Gour. Completely agree with the Apple AppStore analogy.
Gour Saha wrote:
Very good points Josh and Lei. I totally agree with you on the importance
of the app-packages.
Given that YARN-4692 is all about long-running apps/services and primarily
focused around Docker, apps are at its core. I guess the Apple App Store
and its devices can be only as great as its apps. Having said that, I am
pretty sure sample apps have to be provided by the platform. The samples
and guidelines will then help the app-owner community to build more and
more. YARN-4692 design doc already talks about the concept of image stores
in section (3.3.3) Packaging.
Naturally, the existing Slider apps need a home, and I am sure will be of
prime importance when we take the discussion to the Hadoop community, once
we have a consensus here.
-Gour
On 3/16/16, 9:45 PM, "Lei Guo"<lei...@huawei.com> wrote:
Some quick thoughts regarding the app-packages topic, I was the owner of
a cluster management product development a few years ago, the product can
help to build a HPC cluster within 15-30 minutes from bare metal. The
major challenge for me was to define the boundary between product feature
and best practice. Each customer is different, from driver required for
HW to the IP address planning, there were so many miscellaneous
requirements on product enhancements. If we put everything in product, it
may triple our delivery effort and each small feature may can only help a
couple of customers. We had to try our best to generalize the
requirements as product feature and put more best practice guide or
sample in community/support channel.
Back to the topic on app-packages, I see it's in similar position as
cookbook/recipe in Chef. It's good enough for some basic customer cases,
but must be customized/enhanced to support advanced customer cases. So
basic app-packages could be as a sample in Yarn package, but advanced
case or customization should only be published in certain community way
as it does need the customer's effort to make it fit certain environment.
Just my two cents,
Lei
-----Original Message-----
From: Josh Elser [mailto:els...@apache.org]
Sent: Wednesday, March 16, 2016 10:58 PM
To: dev@slider.incubator.apache.org
Subject: Re: [DISCUSS] Next Steps for Slider& First-Class Services in
YARN (YARN-4692)
Thanks for the proposal, Gour! Interesting thought.
I think it makes sense. As YARN is maturing, long-lived services
becoming a primitive is a natural progression. Slider is likely at the
forefront of building such a primitive on YARN (from a lot of great
planning/design from Steve).
I think this would definitely be an interesting conversation to be had
with YARN (if the other podling members are of the same mindset). I
think how this plays out would require a bit of planning/coordination
from the Hadoop PMC side.
Now, there is the other half of Slider: the app-packages. My gut
reaction is that YARN would have no interest in owning/maintaining
these. This is a bit concerning to me because Slider on its own really
isn't that exciting. It's the app-packages that make it so enticing --
build a zip, install it to your cluster, and suddenly users can start
dynamically creating clusters (HBase, Accumulo, Storm, etc). I would be
strongly opposed to any plan to merge Slider into YARN/Hadoop without a
clear path forward on where the app-packages would live. This is
extremely important to me.
I'd love to see where this conversation can go.
- Josh
Gour Saha wrote:
Slider community,
The YARN team is discussing in
YARN-4692<https://issues.apache.org/jira/browse/YARN-4692> on how to
add "first class services" directly to YARN. Some of the names in the
discussion document should be familiar: that's because Slider is
essentially the original long-lived application in YARN.
With YARN-4692<https://issues.apache.org/jira/browse/YARN-4692>, it is
apparent that the Apache Hadoop YARN community is working towards
providing direct support for long-lived services. I think we need to
look at that proposal and think "where and how does Slider relate to
this".
Apache Slider (incubating) has been in the business of creating and
managing long-running services in YARN for a couple of years. Today it
is being used in production YARN clusters across several companies (big
and small). Several production-grade applications (data and non-data)
are available as sample packages. A good number of them have been
contributed by interested parties like Lucidworks contributing a Solr
Slider Application Package and DataTorrent contributing a Kafka Slider
Application Package.
Slider has been pretty good at taking existing applications and turning
them into long-lived services in YARN. YARN offers the core scheduling,
execution and failure reporting functions; slider takes that and adds:
advanced container placement (history; anti-affine, escalation
policies), configuration, dynamic binding, monitoring, failure handling,
and an API for clients. It's also driven a lot of the
YARN-896<https://issues.apache.org/jira/browse/YARN-896> "long-lived
services" development: long-lived failure resilience, the YARN registry,
container-preservation over YARN restarts. Big chunks of that code
actually came from the Slider team. This was always a goal of the work
even in its Hoya predecessor: show that YARN can be used to host
applications like HBase, and identify where it can be be improved.
What does it mean for Slider if YARN starts doing this directly?
Slider provides a lot of the basic functionalities for long-running
services proposed in YARN-4692. It is a universal YARN app-master and
lets application-owners focus on their application functionalities,
while it handles the internals of orchestrating services on YARN.
Which means: we have an opportunity here to contribute the core of
slider into YARN itself, and, with it in YARN, use it as the basis for
the full TODO-list of YARN-4692.
The YARN team gets the stable codebase that's evolved over the past few
years: something to deploy applications in a YARN cluster. What does
Slider get? We'd get to be the foundation for long lived YARN services
with the new work on top.
Would this work? What's wrong with the idea? How do we do it if we want
to go with it?
I would like to call upon the community to weigh in their thoughts and
opinions on this topic.
-Gour
