Thanks Jon, Billie. Billie, I totally agree to your point. The agent code and probably any other python pieces have to be either rewritten or thrown away. An important point in YARN-4692 is that application owners should be able to bring their applications in industry standard package formats like Docker say, and YARN should be able to run them as is without any modifications. This tells me that the architecture might have to be completely agent-less. This will make migration of Slider code-base (primarily the client and AM code) to YARN a little bit simpler. Of course, we still have to think about how to continue to support existing tarball packages in this agent-less world.
>From what I understand from this discussion thread over the last 2+ weeks is that, there is general consensus in the Slider community to pursue this and initiate further discussions with the YARN community. If I don¹t hear from anyone else till end of today, I will consider this discussion thread to be closed. -Gour On 3/31/16, 8:48 AM, "Billie Rinaldi" <billie.rina...@gmail.com> wrote: >Beyond the app packages, I think we should have a clear plan for what is >going to happen with the agent code. It seems doubtful that YARN would be >willing to take on a bunch of python code, but the agent provider is >currently the only provider implementation we have. On the bright side, we >are not doing as much with the agent as we could be, and it might not be >too onerous to move its functionality elsewhere. > >In any case, I think Gour's proposal is good, and it will be worthwhile to >work out these details as we proceed. > >On Tue, Mar 15, 2016 at 7:44 PM, Gour Saha <gs...@hortonworks.com> wrote: > >> Slider community, >> >> >> The YARN team is discussing in YARN-4692< >> https://issues.apache.org/jira/browse/YARN-4692> on how to add "first >> class services" directly to YARN. Some of the names in the discussion >> document should be familiar: that's because Slider is essentially the >> original long-lived application in YARN. >> >> >> With YARN-4692<https://issues.apache.org/jira/browse/YARN-4692>, it is >> apparent that the Apache Hadoop YARN community is working towards >>providing >> direct support for long-lived services. I think we need to look at that >> proposal and think "where and how does Slider relate to this". >> >> >> Apache Slider (incubating) has been in the business of creating and >> managing long-running services in YARN for a couple of years. Today it >>is >> being used in production YARN clusters across several companies (big and >> small). Several production-grade applications (data and non-data) are >> available as sample packages. A good number of them have been >>contributed >> by interested parties like Lucidworks contributing a Solr Slider >> Application Package and DataTorrent contributing a Kafka Slider >>Application >> Package. >> >> >> Slider has been pretty good at taking existing applications and turning >> them into long-lived services in YARN. YARN offers the core scheduling, >> execution and failure reporting functions; slider takes that and adds: >> advanced container placement (history; anti-affine, escalation >>policies), >> configuration, dynamic binding, monitoring, failure handling, and an API >> for clients. It's also driven a lot of the YARN-896< >> https://issues.apache.org/jira/browse/YARN-896> "long-lived services" >> development: long-lived failure resilience, the YARN registry, >> container-preservation over YARN restarts. Big chunks of that code >>actually >> came from the Slider team. This was always a goal of the work even in >>its >> Hoya predecessor: show that YARN can be used to host applications like >> HBase, and identify where it can be be improved. >> >> >> What does it mean for Slider if YARN starts doing this directly? >> >> >> Slider provides a lot of the basic functionalities for long-running >> services proposed in YARN-4692. It is a universal YARN app-master and >>lets >> application-owners focus on their application functionalities, while it >> handles the internals of orchestrating services on YARN. >> >> >> Which means: we have an opportunity here to contribute the core of >>slider >> into YARN itself, and, with it in YARN, use it as the basis for the full >> TODO-list of YARN-4692. >> >> >> The YARN team gets the stable codebase that's evolved over the past few >> years: something to deploy applications in a YARN cluster. What does >>Slider >> get? We'd get to be the foundation for long lived YARN services with the >> new work on top. >> >> >> Would this work? What's wrong with the idea? How do we do it if we want >>to >> go with it? >> >> >> I would like to call upon the community to weigh in their thoughts and >> opinions on this topic. >> >> -Gour >> >>
