Is suspect we haven't written down the architecture thoroughly enough.

There's some description of it in two recent presentations

1. Slider http://www.slideshare.net/steve_l/slider-a
2. YARN services http://www.slideshare.net/steve_l/yarn-services


Slider is an YARN service; everything in that presentation matters. Page 22
also shows what state we keep in the AM -more specifically, what is
persisted to HDFS, what is rebuilt from information from the running
containers when the AM restarts -and what is lost on an AM restart.


   - The Slider application master takes the configuration JSON files in
   HDFS, and works to keep the actual count of every component matching that
   requested in resources.json
   - It asks YARN for new containers when needed, releases them when
   surplus. It persists the placement to a rolehistory file (more
   specificlaly, it saves a new file when things change). When new containers
   are needed (including on application start), if a component is flagged as
   wanting or needing placement where it was before, the old locations are
   used in the YARN requests
   - It supports plugin "providers" which handle the actual work of
   configuring and running the nodes. Historically we had some Java-based ones
   for each application; now we only have the AgentProvider. We've left the
   split in just in case we ever want to change providers, something that may
   be one route to better docker support (see the relevant JIRA there)
   - The agent provider runs a little python agent in every requested
   container (one container per component instance), which reports to the
   agent provider over an SSL connection.
   - The Agent Provider tells the agent what to do, e.g. install a binary,
   start it.
   - The AM keeps running, keeping the number of actual containers equal to
   the desired one (if there is enough space in the cluster)
   - Incoming events (yarn container messages, IPC calls) are stuck into an
   event queue and asynchronously processed
   - There's also a scheduled event queue for delayed operations, these are
   promoted to the main event queue when their scheduled time is reached.
   - If the AM fails, the components keep running;
   - The AM is restarted, reloads its persistent state and brings up new
   web site/services on the new host and ports. It receives the list of
   containers from YARN, determines the component type in each of these
   containers, and rebuilds its state.
   - the agents in the containers look for the restarted AM's new agent web
   service via the YARN registry.
   - The Agent Provider gets those notifications and rebuilds its state.
   All containers running the actual target application keeps running. All
   containers not running the application are released and replacements
   requested. This is because we don't know why the app isn't running or what
   state it was through the install/launch process. It's easier to start from
   the beginning.
   - If the running program finishes, the agent finishes, which triggers a
   YARN notification. The AM notes the event, and if unexpected increments the
   failure count for that component type (RoleStatus class).
   - A new container is then requested and the agent install/run begins
   again.
   - The component failure count will trigger the slider app failure if it
   reaches a threshold, this is how slider knows to give up if an application
   just isn't working. There's a scheduled count resetter to to stop a few
   failures over a week triggering application failure; we care more about the
   number of failures per hour.
   - There's a very minimal Web UI and IPC channel for some operations,
   only two of which change the AM state: flex and stop. Flex hands the AM an
   updated resources.json, which the AM then persists and reacts to by
   changing the desired state of the cluster.
   - The CLI class, SliderClient can be used as an API -Ambari does this so
   that it can deploy an application via Slider and manage it. (There's a bit
   more to the Ambari integration; ask if you are curious)

Ongoing work includes

   - A REST API to replace the IPC channel; see SLIDER-151.
   - Rolling upgrades of an application so you don't need to stop HBase to
   update it. (SLIDER-290)
   - Docker support (SLIDER-532)
   - better component placement (better anti-affinity for applications like
   Kafka)
   - helping get more packages up, which will drive what improvements we
   need in slider.

Some of these changes are going to need YARN enhancements (REST,
placement), so are going to only be usable in production on future Hadoop
versions (Hadoop 2.8+). We're going to try to put off a Hadoop 2.8+ only
switch as long as possible.


That's pretty much it.



On 11 December 2014 at 02:05, liujs3 <[email protected]> wrote:

> Hi, my company use slider to run some application like spark and
> hbase,they let me show the slider's architecture,and run it on our product
> line . could you give some architecture's study or report?
> Now every day I study the source ..it always looks like not clear and
> personal.
>
>
>
>
>
>
> Best Regards!
> ocean floyd
> 亚信科技(中 国)有限公司
> www.asiainfo.com
> BDX
> Mp:18513780630
> E-mail:[email protected]
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Reply via email to