Modified: incubator/reef/site/glossary.html URL: http://svn.apache.org/viewvc/incubator/reef/site/glossary.html?rev=1700380&r1=1700379&r2=1700380&view=diff ============================================================================== --- incubator/reef/site/glossary.html (original) +++ incubator/reef/site/glossary.html Tue Sep 1 02:08:00 2015 @@ -1,552 +1,552 @@ -<!DOCTYPE html> -<!-- - | Generated by Apache Maven Doxia at 2015-08-20 - | Rendered using Apache Maven Fluido Skin 1.4 ---> -<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> - <head> - <meta charset="UTF-8" /> - <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150820" /> - <meta http-equiv="Content-Language" content="en" /> - <title>Apache REEF - Glossary</title> - <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> - <link rel="stylesheet" href="./css/site.css" /> - <link rel="stylesheet" href="./css/print.css" media="print" /> - - - <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> - - - -<script src="js/release.js" type="text/javascript"></script> - - </head> - <body class="topBarEnabled"> - - - - - - <div id="topbar" class="navbar navbar-fixed-top "> - <div class="navbar-inner"> - <div class="container-fluid"> - <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </a> - - <ul class="nav"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache REEF <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="index.html" title="Overview">Overview</a> -</li> - - <li> <a href="faq.html" title="FAQ">FAQ</a> -</li> - - <li> <a href="license.html" title="License">License</a> -</li> - - <li> <a href="downloads.html" title="Downloads">Downloads</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="introduction.html" title="Introduction to REEF">Introduction to REEF</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; title="REEF Tutorial">REEF Tutorial</a> -</li> - - <li> <a href="talks.html" title="Talks">Talks</a> -</li> - - <li> <a href="glossary.html" title="Glossary">Glossary</a> -</li> - - <li> <a href="tang.html" title="Tang">Tang</a> -</li> - - <li> <a href="wake.html" title="Wake">Wake</a> -</li> - - <li> <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API">0.10.0-incubating API</a> -</li> - - <li> <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API">0.11.0-incubating API</a> -</li> - - <li> <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API">0.12.0-incubating API</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribution <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; title="Contributing">Contributing</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; title="Committer Guide">Committer Guide</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; title="Coding Guidelines">Coding Guidelines</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="team.html" title="Team">Team</a> -</li> - - <li> <a href="mailing-list.html" title="Mailing List">Mailing List</a> -</li> - - <li> <a href="https://issues.apache.org/jira/browse/REEF"; title="Issue Tracker">Issue Tracker</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; title="Powered By">Powered By</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/foundation/"; title="Apache Software Foundation">Apache Software Foundation</a> -</li> - - <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> -</li> - - <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> -</li> - - <li> <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; title="Apache License">Apache License</a> -</li> - - <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsorship">Sponsorship</a> -</li> - - <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> -</li> - </ul> - </li> - </ul> - - - - <ul class="nav pull-right"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li> <a href="https://github.com/apache/incubator-reef"; title="Apache REEF GitHub">Apache REEF GitHub</a> -</li> - <li> <a href="http://www.apache.org"; title="Apache">Apache</a> -</li> - <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> -</li> - </ul> - </li> - </ul> - - </div> - - </div> - </div> - </div> - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="./" id="bannerLeft" title="Apache REEF"> - <img src="ApacheREEF_logo_no_margin_small.png" alt="Apache REEF"/> - </a> - </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="ApacheIncubator.png" /> - </a> - </div> - <div class="clear"><hr/></div> - </div> - - <div id="breadcrumbs"> - <ul class="breadcrumb"> - - - <li class=""> - <a href="index.html" title="Apache REEF"> - Apache REEF</a> - <span class="divider">|</span> - </li> - <li class="active ">Glossary</li> - - - - - </ul> - </div> - - - <div class="row-fluid"> - <div id="leftColumn" class="span2"> - <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">Apache REEF</li> - - <li> - - <a href="index.html" title="Overview"> - <span class="none"></span> - Overview</a> - </li> - - <li> - - <a href="faq.html" title="FAQ"> - <span class="none"></span> - FAQ</a> - </li> - - <li> - - <a href="license.html" title="License"> - <span class="none"></span> - License</a> - </li> - - <li> - - <a href="downloads.html" title="Downloads"> - <span class="none"></span> - Downloads</a> - </li> - <li class="nav-header">Documentation</li> - - <li> - - <a href="introduction.html" title="Introduction to REEF"> - <span class="none"></span> - Introduction to REEF</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; class="externalLink" title="REEF Tutorial"> - <span class="none"></span> - REEF Tutorial</a> - </li> - - <li> - - <a href="talks.html" title="Talks"> - <span class="none"></span> - Talks</a> - </li> - - <li class="active"> - - <a href="#"><span class="none"></span>Glossary</a> - </li> - - <li> - - <a href="tang.html" title="Tang"> - <span class="none"></span> - Tang</a> - </li> - - <li> - - <a href="wake.html" title="Wake"> - <span class="none"></span> - Wake</a> - </li> - - <li> - - <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API"> - <span class="none"></span> - 0.10.0-incubating API</a> - </li> - - <li> - - <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API"> - <span class="none"></span> - 0.11.0-incubating API</a> - </li> - - <li> - - <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API"> - <span class="none"></span> - 0.12.0-incubating API</a> - </li> - <li class="nav-header">Contribution</li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; class="externalLink" title="Contributing"> - <span class="none"></span> - Contributing</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; class="externalLink" title="Committer Guide"> - <span class="none"></span> - Committer Guide</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; class="externalLink" title="Coding Guidelines"> - <span class="none"></span> - Coding Guidelines</a> - </li> - <li class="nav-header">Community</li> - - <li> - - <a href="team.html" title="Team"> - <span class="none"></span> - Team</a> - </li> - - <li> - - <a href="mailing-list.html" title="Mailing List"> - <span class="none"></span> - Mailing List</a> - </li> - - <li> - - <a href="https://issues.apache.org/jira/browse/REEF"; class="externalLink" title="Issue Tracker"> - <span class="none"></span> - Issue Tracker</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; class="externalLink" title="Powered By"> - <span class="none"></span> - Powered By</a> - </li> - <li class="nav-header">ASF</li> - - <li> - - <a href="http://www.apache.org/foundation/"; class="externalLink" title="Apache Software Foundation"> - <span class="none"></span> - Apache Software Foundation</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> - <span class="none"></span> - How Apache Works</a> - </li> - - <li> - - <a href="http://incubator.apache.org/"; class="externalLink" title="Apache Incubator"> - <span class="none"></span> - Apache Incubator</a> - </li> - - <li> - - <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; class="externalLink" title="Apache License"> - <span class="none"></span> - Apache License</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsorship"> - <span class="none"></span> - Sponsorship</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> - <span class="none"></span> - Thanks</a> - </li> - </ul> - - - - <hr /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> - </a> - </div> - </div> - </div> - - - <div id="bodyColumn" class="span10" > - - <!-- Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. --><h1>Glossary</h1> - -<ul> - -<li><a href="#context">Context</a></li> - -<li><a href="#driver">Driver</a></li> - -<li><a href="#evaluator">Evaluator</a></li> - -<li><a href="#task">Task</a></li> -</ul> -<div class="section"> -<div class="section"> -<h3><a name="context"></a>Context<a name="Context"></a></h3> -<p>Contexts are a way to structure the state and Configuration of an Evaluator. A Context exists on one and only one individual Evaluator. Each Evaluator has at least one Context, which we refer to as the <i>root</i> Context. This root context is special, as it is <i>required</i> on all Evaluators and because closing it is synonymous to releasing the Evaluator. In many simple REEF programs, this is the only Context used and it is therefore convenient to think of it as synonymous to “the Evaluator”: The Driver can submit Tasks to it, is notifified when they are done and can close the root context when it wants to dispose of the Evaluator. </p> -<p>Contexts are formed by calls to <tt>submitContext()</tt> to the event types that allow this (<tt>AllocatedEvaluator</tt> and <tt>ActiveContext</tt>) Contexts are the main way for an Evaluator to be exposed and accessed. For instance, Tasks are submitted to an <tt>ActiveContext</tt> which represents the top Context on the Evaluator.</p> -<p>Beyond this, a Driver can submit a Context to the root, or in fact any, Context, as long as the resulting structure is that of a stack: The root Context forms the bottom of the stack, the top-most Context is called <i>active</i>, hence the <tt>ActiveContext</tt> event. The two can be one and the same, and often are: The root Context is the subject of the first <tt>ActiveContext</tt> event on an Evaluator.</p> -<p>Nomenclature: When Context B is submitted to an already existing Context A, we say that Context A is the parent Context of Context B. Also, Context B is the child of Context A.</p> -<p>It is only the <tt>ActiveContext</tt> that allows the submission of Tasks or child Contexts. Hence, one can think of the whole Evaluator structure as that of a stack: the root Context at the bottom, layers of Contexts in the middle and either the current <tt>ActiveContext</tt> or the current Task at the top.</p> -<div class="section"> -<h4>Objects and Configuration: What’s in a Context?<a name="Objects_and_Configuration:_Whats_in_a_Context"></a></h4> -<p>It is convenient to think of a Context as a <tt>Configuration</tt> that gets merged with the <tt>Configuration</tt> supplied for Tasks and child Contexts. While not entirely true (see below), this view allows us to show just <i>why</i> Contexts are a convenient construct.</p> -<p>It is often the case that subsequent tasks that get executed on an Evaluator want access to the same Configuration variables and / or the same objects. Consider a simple <tt>LinkedList</tt> bound to a named parameter. If that linked list is part of the subsequent Task <tt>Configurations</tt> submited, each Task will get its very <i>own</i> <tt>LinkedList</tt>. If the named parameter is bound in the Context <tt>Configuration</tt>, all Tasks subsequently submitted to the Context will get the very <i>same</i> <tt>LinkedList</tt> instance.</p></div> -<div class="section"> -<h4>Contexts are (Tang) Injectors<a name="Contexts_are_Tang_Injectors"></a></h4> -<p>This mechanism is implemented by using Tang’s <tt>Injector</tt>s. On the Evaluator, a Task is launched by first <i>forking</i> the Context’s <tt>Injector</tt> with the Task<tt>Configuration</tt> and then requesting an instance of the <tt>Task</tt> interface from that forked <tt>Injector</tt>. By this mechanism and the fact that objects are singletons with respect to an <tt>Injector</tt> in Tang, object sharing can be implemented. All objects already instantiated on the Context <tt>Injector</tt> will also be referenced by the Task<tt>Injector</tt>. Hence, the <tt>LinkedList</tt> in the example above would be shared amongst subsequent Task <tt>Injectors</tt> in the construction of the <tt>Task</tt> instance.</p></div></div> -<div class="section"> -<h3><a name="driver"></a>Driver<a name="Driver"></a></h3> -<p>REEF imposes a centralized control flow design on applications: All events are routed to the master node, called the Driver. REEF also prescribes event-driven programming for the Driver. In that sense, the application provided Driver is a collection of event handlers for the various events exposed in <tt>DriverConfiguration</tt>. While most of these deal with occurrences during the computation (Evaluator allocation, Task launch, …), several stand out as life-cycle events of the Driver, and therefore the application:</p> -<div class="section"> -<h4>ON_START<a name="ON_START"></a></h4> -<p>This event is triggered by REEF when the Driver is ready to start executing. At this point communication with the Resource Manager has been established, all event handlers have been instantiated and the event graph in the Driver was deemed to be complete enough to start. In a typical application, this is when the Driver requests the first set of Evaluators.</p></div> -<div class="section"> -<h4>ON_STOP<a name="ON_STOP"></a></h4> -<p>This event is fired right before the Driver shuts down. REEF determines Driver shutdown by proof that no future events can happen:</p> - -<ul> - -<li>The Clock has no outstanding alarms.</li> - -<li>The resource manager has no outstanding requests.</li> - -<li>The application has no Evaluators allocated anymore.</li> -</ul> -<p>Hence, the <tt>ON_STOP</tt> event can be used to prevent Driver shutdown, e.g. in applications that need to wait for external triggers (e.g. a UI) to proceed. To do so, one would schedule an alarm in the <tt>ON_STOP</tt> handler.</p></div></div> -<div class="section"> -<h3><a name="evaluator"></a>Evaluator<a name="Evaluator"></a></h3> -<div class="section"> -<h4>Evaluators and Tasks<a name="Evaluators_and_Tasks"></a></h4> -<p>The Evaluator is the runtime environment for Tasks. On one Evaluator, there is either no or one Task executing at any given point in time. Different or multiple executions of the same Tasks can be executed in sequence on an Evaluator. The Evaluator and Task lifecycle are decoupled: Whenever a Task finishes, the Driver receives the CompletedTask event, which contains a reference to the Evaluator the Task executed on. It is then up to the Driver to decide whether to return the Evaluator to the resource manager or to make other use of it, e.g. by submitting another task.</p></div> -<div class="section"> -<h4>Evaluators and Contexts<a name="Evaluators_and_Contexts"></a></h4> -<p>Contexts are REEF’s form of state management inside of the Evaluator. See the <a href="#context">Context</a> section for more information.</p></div> -<div class="section"> -<h4>Evaluators and the Resource Manager<a name="Evaluators_and_the_Resource_Manager"></a></h4> -<p>On typical resource managers, an Evaluator is a process executing inside a container. Depending on the resource manager, that process may or may not be guarded by a resource or security isolation layer.</p> -<p>This also means that the Evaluator, not the Task, is the unit of resource consumption: while an Evaluator is occupying a Container, that Container is “used” from the perspective of the Resource Manager.That is true even if the Evaluator is idle from the perspective of the Driver, i.e. when no Task is running on it.</p></div></div> -<div class="section"> -<h3><a name="task"></a>Task<a name="Task"></a></h3> -<div class="section"> -<h4>Definition<a name="Definition"></a></h4> -<p>A Task in REEF is a unit of work to be executed on an Evaluator. In its simplest form, a Task is merely an object implementing the Task interface which prescribes a single method:</p> - -<div class="source"><pre class="prettyprint">public byte[] call(byte[] input); -</pre></div> -<p>From REEF’s perspective, a Task is therefore a single threaded method call. It starts when entering the call method. It is a <tt>RunningTask</tt> while it hasn’t returned from it and is a <tt>CompletedTask</tt> when it has. Should there be an Exception thrown by <tt>call()</tt>, we call it a <tt>FailedTask</tt>.</p> -<p>Task identity is established by a user-defined string set in <tt>TaskConfiguration.IDENTIFIER</tt>. All subsequent task-related events in the Driver will carry that ID. Note that REEF doesn’t take any particular precautions to ensure unique Task identifiers. It is up to the application to do so. While technically feasible to assign the same identifier to multiple Tasks, this isn’t advised as it makes error handling, debugging and logging unnecessarily hard.</p></div> -<div class="section"> -<h4>Inputs and outputs of a Task<a name="Inputs_and_outputs_of_a_Task"></a></h4> -<p>The return value of the <tt>call</tt> method will be made available to the Driver as part of the <tt>CompletedTask</tt> event. Note that it isn’t advised to return large values in this fashion, but merely small control flow or status information. Sending large data on this channel creates the risk of overloading the Driver at scale. The networking APIs provided by REEF IO are much better suited for data transmissions than this channel.</p> -<p>The parameter given to the call method is also to be used in a similar fashion: The Driver passes its value as part of the Task submission. It is meant e.g. to convey a restart point for the task. Note that the same functionality can now be better provided by Tang and a constructor parameter.</p></div> -<div class="section"> -<h4>Communicating between a Task and a Driver<a name="Communicating_between_a_Task_and_a_Driver"></a></h4> -<p>REEF provides some facilities to communicate between a Driver and a Task. These mostly stem from allowing application code to “free-ride” on REEF’s control flow channels such as the heartbeat between the Evaluator and the Task.</p> -<div class="section"> -<h5>Sending a message from the Driver to a Task<a name="Sending_a_message_from_the_Driver_to_a_Task"></a></h5> -<p>REEF maintains a heartbeat between any Evaluator and the Driver. There are two ways by which a heartbeat can be triggered.</p> - -<ul> - -<li> -<p>Upon some schedule (which may also vary at runtime due to load conditions on the Driver), each Evaluator will report its current status to the Driver. This is used by the Driver to maintain health status and load statistics of the Evaluators.</p></li> - -<li> -<p>Whenever the status of the Evaluator changes (e.g. when a Task completes), a Heartbeat is triggered immediately.</p></li> -</ul> -<p>Whenever the Evaluator performs a heartbeat, it will ask the Task whether it has any message to share with the Driver by inquiring the class registered in <tt>TaskConfiguration.ON_SEND_MESSAGE</tt>. It is wise for that message to be small, as we otherwise run the risk of overwhelming the Driver with heartbeat traffic at scale.</p></div></div> -<div class="section"> -<h4>Multithreaded Tasks<a name="Multithreaded_Tasks"></a></h4> -<p>Just because REEF views a Task as a method call doesn’t restrict the Task to be single threaded. A Task is free to spawn threads in the course of its execution. However, a Task that does so needs to take care of a few considerations:</p> - -<ul> - -<li> -<p>All Threads spawned need to exit before the <tt>Task.call()</tt> method returns. Otherwise, you run the risk of resource leakage.</p></li> - -<li> -<p>Exceptions on spawned Threads need to be caught and re-thrown by the <tt>Thread.call()</tt> method. Before that, all spawned threads need to be shut down, just like during a normal exit of <tt>Task.call()</tt>. If an exception from an another thread isn’t caught, REEF’s JVM level exception handler will catch it and declare a FailedEvaluator. This is inefficient, but not technically wrong: The Driver will then have to allocate another Evaluator and try again.</p></li> -</ul></div></div></div> - </div> - </div> - </div> - - <hr/> - - <footer> - <div class="container-fluid"> - <div class="row-fluid"> - -<div class="container-fluid"> - <div class="row-fluid"> - <a href="http://www.apache.org";>Apache Software Foundation</a> - </div> -</div> - -<div class="row span12">Apache REEF, REEF, Apache, the Apache feather logo, and the Apache REEF logo are trademarks - of The Apache Software Foundation. All other marks mentioned may be trademarks or registered - trademarks of their respective owners.</div> - </div> - - - </div> - </footer> - </body> -</html> \ No newline at end of file +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-09-01 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20150901" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache REEF - Glossary</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="js/release.js" type="text/javascript"></script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache REEF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="Overview">Overview</a> +</li> + + <li> <a href="faq.html" title="FAQ">FAQ</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + + <li> <a href="downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="introduction.html" title="Introduction to REEF">Introduction to REEF</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; title="REEF Tutorial">REEF Tutorial</a> +</li> + + <li> <a href="talks.html" title="Talks">Talks</a> +</li> + + <li> <a href="glossary.html" title="Glossary">Glossary</a> +</li> + + <li> <a href="tang.html" title="Tang">Tang</a> +</li> + + <li> <a href="wake.html" title="Wake">Wake</a> +</li> + + <li> <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API">0.10.0-incubating API</a> +</li> + + <li> <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API">0.11.0-incubating API</a> +</li> + + <li> <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API">0.12.0-incubating API</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribution <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; title="Contributing">Contributing</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; title="Committer Guide">Committer Guide</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; title="Coding Guidelines">Coding Guidelines</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="team.html" title="Team">Team</a> +</li> + + <li> <a href="mailing-list.html" title="Mailing List">Mailing List</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/REEF"; title="Issue Tracker">Issue Tracker</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; title="Powered By">Powered By</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> +</li> + + <li> <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; title="Apache License">Apache License</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsorship">Sponsorship</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + + + <ul class="nav pull-right"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li> <a href="https://github.com/apache/incubator-reef"; title="Apache REEF GitHub">Apache REEF GitHub</a> +</li> + <li> <a href="http://www.apache.org"; title="Apache">Apache</a> +</li> + <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> +</li> + </ul> + </li> + </ul> + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="./" id="bannerLeft" title="Apache REEF"> + <img src="ApacheREEF_logo_no_margin_small.png" alt="Apache REEF"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="ApacheIncubator.png" /> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="index.html" title="Apache REEF"> + Apache REEF</a> + <span class="divider">|</span> + </li> + <li class="active ">Glossary</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache REEF</li> + + <li> + + <a href="index.html" title="Overview"> + <span class="none"></span> + Overview</a> + </li> + + <li> + + <a href="faq.html" title="FAQ"> + <span class="none"></span> + FAQ</a> + </li> + + <li> + + <a href="license.html" title="License"> + <span class="none"></span> + License</a> + </li> + + <li> + + <a href="downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentation</li> + + <li> + + <a href="introduction.html" title="Introduction to REEF"> + <span class="none"></span> + Introduction to REEF</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; class="externalLink" title="REEF Tutorial"> + <span class="none"></span> + REEF Tutorial</a> + </li> + + <li> + + <a href="talks.html" title="Talks"> + <span class="none"></span> + Talks</a> + </li> + + <li class="active"> + + <a href="#"><span class="none"></span>Glossary</a> + </li> + + <li> + + <a href="tang.html" title="Tang"> + <span class="none"></span> + Tang</a> + </li> + + <li> + + <a href="wake.html" title="Wake"> + <span class="none"></span> + Wake</a> + </li> + + <li> + + <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API"> + <span class="none"></span> + 0.10.0-incubating API</a> + </li> + + <li> + + <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API"> + <span class="none"></span> + 0.11.0-incubating API</a> + </li> + + <li> + + <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API"> + <span class="none"></span> + 0.12.0-incubating API</a> + </li> + <li class="nav-header">Contribution</li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; class="externalLink" title="Contributing"> + <span class="none"></span> + Contributing</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; class="externalLink" title="Committer Guide"> + <span class="none"></span> + Committer Guide</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; class="externalLink" title="Coding Guidelines"> + <span class="none"></span> + Coding Guidelines</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="team.html" title="Team"> + <span class="none"></span> + Team</a> + </li> + + <li> + + <a href="mailing-list.html" title="Mailing List"> + <span class="none"></span> + Mailing List</a> + </li> + + <li> + + <a href="https://issues.apache.org/jira/browse/REEF"; class="externalLink" title="Issue Tracker"> + <span class="none"></span> + Issue Tracker</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; class="externalLink" title="Powered By"> + <span class="none"></span> + Powered By</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <span class="none"></span> + How Apache Works</a> + </li> + + <li> + + <a href="http://incubator.apache.org/"; class="externalLink" title="Apache Incubator"> + <span class="none"></span> + Apache Incubator</a> + </li> + + <li> + + <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; class="externalLink" title="Apache License"> + <span class="none"></span> + Apache License</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsorship"> + <span class="none"></span> + Sponsorship</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <span class="none"></span> + Thanks</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <!-- Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. --><h1>Glossary</h1> + +<ul> + +<li><a href="#context">Context</a></li> + +<li><a href="#driver">Driver</a></li> + +<li><a href="#evaluator">Evaluator</a></li> + +<li><a href="#task">Task</a></li> +</ul> +<div class="section"> +<div class="section"> +<h3><a name="context"></a>Context<a name="Context"></a></h3> +<p>Contexts are a way to structure the state and Configuration of an Evaluator. A Context exists on one and only one individual Evaluator. Each Evaluator has at least one Context, which we refer to as the <i>root</i> Context. This root context is special, as it is <i>required</i> on all Evaluators and because closing it is synonymous to releasing the Evaluator. In many simple REEF programs, this is the only Context used and it is therefore convenient to think of it as synonymous to “the Evaluator”: The Driver can submit Tasks to it, is notifified when they are done and can close the root context when it wants to dispose of the Evaluator. </p> +<p>Contexts are formed by calls to <tt>submitContext()</tt> to the event types that allow this (<tt>AllocatedEvaluator</tt> and <tt>ActiveContext</tt>) Contexts are the main way for an Evaluator to be exposed and accessed. For instance, Tasks are submitted to an <tt>ActiveContext</tt> which represents the top Context on the Evaluator.</p> +<p>Beyond this, a Driver can submit a Context to the root, or in fact any, Context, as long as the resulting structure is that of a stack: The root Context forms the bottom of the stack, the top-most Context is called <i>active</i>, hence the <tt>ActiveContext</tt> event. The two can be one and the same, and often are: The root Context is the subject of the first <tt>ActiveContext</tt> event on an Evaluator.</p> +<p>Nomenclature: When Context B is submitted to an already existing Context A, we say that Context A is the parent Context of Context B. Also, Context B is the child of Context A.</p> +<p>It is only the <tt>ActiveContext</tt> that allows the submission of Tasks or child Contexts. Hence, one can think of the whole Evaluator structure as that of a stack: the root Context at the bottom, layers of Contexts in the middle and either the current <tt>ActiveContext</tt> or the current Task at the top.</p> +<div class="section"> +<h4>Objects and Configuration: What’s in a Context?<a name="Objects_and_Configuration:_Whats_in_a_Context"></a></h4> +<p>It is convenient to think of a Context as a <tt>Configuration</tt> that gets merged with the <tt>Configuration</tt> supplied for Tasks and child Contexts. While not entirely true (see below), this view allows us to show just <i>why</i> Contexts are a convenient construct.</p> +<p>It is often the case that subsequent tasks that get executed on an Evaluator want access to the same Configuration variables and / or the same objects. Consider a simple <tt>LinkedList</tt> bound to a named parameter. If that linked list is part of the subsequent Task <tt>Configurations</tt> submited, each Task will get its very <i>own</i> <tt>LinkedList</tt>. If the named parameter is bound in the Context <tt>Configuration</tt>, all Tasks subsequently submitted to the Context will get the very <i>same</i> <tt>LinkedList</tt> instance.</p></div> +<div class="section"> +<h4>Contexts are (Tang) Injectors<a name="Contexts_are_Tang_Injectors"></a></h4> +<p>This mechanism is implemented by using Tang’s <tt>Injector</tt>s. On the Evaluator, a Task is launched by first <i>forking</i> the Context’s <tt>Injector</tt> with the Task<tt>Configuration</tt> and then requesting an instance of the <tt>Task</tt> interface from that forked <tt>Injector</tt>. By this mechanism and the fact that objects are singletons with respect to an <tt>Injector</tt> in Tang, object sharing can be implemented. All objects already instantiated on the Context <tt>Injector</tt> will also be referenced by the Task<tt>Injector</tt>. Hence, the <tt>LinkedList</tt> in the example above would be shared amongst subsequent Task <tt>Injectors</tt> in the construction of the <tt>Task</tt> instance.</p></div></div> +<div class="section"> +<h3><a name="driver"></a>Driver<a name="Driver"></a></h3> +<p>REEF imposes a centralized control flow design on applications: All events are routed to the master node, called the Driver. REEF also prescribes event-driven programming for the Driver. In that sense, the application provided Driver is a collection of event handlers for the various events exposed in <tt>DriverConfiguration</tt>. While most of these deal with occurrences during the computation (Evaluator allocation, Task launch, …), several stand out as life-cycle events of the Driver, and therefore the application:</p> +<div class="section"> +<h4>ON_START<a name="ON_START"></a></h4> +<p>This event is triggered by REEF when the Driver is ready to start executing. At this point communication with the Resource Manager has been established, all event handlers have been instantiated and the event graph in the Driver was deemed to be complete enough to start. In a typical application, this is when the Driver requests the first set of Evaluators.</p></div> +<div class="section"> +<h4>ON_STOP<a name="ON_STOP"></a></h4> +<p>This event is fired right before the Driver shuts down. REEF determines Driver shutdown by proof that no future events can happen:</p> + +<ul> + +<li>The Clock has no outstanding alarms.</li> + +<li>The resource manager has no outstanding requests.</li> + +<li>The application has no Evaluators allocated anymore.</li> +</ul> +<p>Hence, the <tt>ON_STOP</tt> event can be used to prevent Driver shutdown, e.g. in applications that need to wait for external triggers (e.g. a UI) to proceed. To do so, one would schedule an alarm in the <tt>ON_STOP</tt> handler.</p></div></div> +<div class="section"> +<h3><a name="evaluator"></a>Evaluator<a name="Evaluator"></a></h3> +<div class="section"> +<h4>Evaluators and Tasks<a name="Evaluators_and_Tasks"></a></h4> +<p>The Evaluator is the runtime environment for Tasks. On one Evaluator, there is either no or one Task executing at any given point in time. Different or multiple executions of the same Tasks can be executed in sequence on an Evaluator. The Evaluator and Task lifecycle are decoupled: Whenever a Task finishes, the Driver receives the CompletedTask event, which contains a reference to the Evaluator the Task executed on. It is then up to the Driver to decide whether to return the Evaluator to the resource manager or to make other use of it, e.g. by submitting another task.</p></div> +<div class="section"> +<h4>Evaluators and Contexts<a name="Evaluators_and_Contexts"></a></h4> +<p>Contexts are REEF’s form of state management inside of the Evaluator. See the <a href="#context">Context</a> section for more information.</p></div> +<div class="section"> +<h4>Evaluators and the Resource Manager<a name="Evaluators_and_the_Resource_Manager"></a></h4> +<p>On typical resource managers, an Evaluator is a process executing inside a container. Depending on the resource manager, that process may or may not be guarded by a resource or security isolation layer.</p> +<p>This also means that the Evaluator, not the Task, is the unit of resource consumption: while an Evaluator is occupying a Container, that Container is “used” from the perspective of the Resource Manager.That is true even if the Evaluator is idle from the perspective of the Driver, i.e. when no Task is running on it.</p></div></div> +<div class="section"> +<h3><a name="task"></a>Task<a name="Task"></a></h3> +<div class="section"> +<h4>Definition<a name="Definition"></a></h4> +<p>A Task in REEF is a unit of work to be executed on an Evaluator. In its simplest form, a Task is merely an object implementing the Task interface which prescribes a single method:</p> + +<div class="source"><pre class="prettyprint">public byte[] call(byte[] input); +</pre></div> +<p>From REEF’s perspective, a Task is therefore a single threaded method call. It starts when entering the call method. It is a <tt>RunningTask</tt> while it hasn’t returned from it and is a <tt>CompletedTask</tt> when it has. Should there be an Exception thrown by <tt>call()</tt>, we call it a <tt>FailedTask</tt>.</p> +<p>Task identity is established by a user-defined string set in <tt>TaskConfiguration.IDENTIFIER</tt>. All subsequent task-related events in the Driver will carry that ID. Note that REEF doesn’t take any particular precautions to ensure unique Task identifiers. It is up to the application to do so. While technically feasible to assign the same identifier to multiple Tasks, this isn’t advised as it makes error handling, debugging and logging unnecessarily hard.</p></div> +<div class="section"> +<h4>Inputs and outputs of a Task<a name="Inputs_and_outputs_of_a_Task"></a></h4> +<p>The return value of the <tt>call</tt> method will be made available to the Driver as part of the <tt>CompletedTask</tt> event. Note that it isn’t advised to return large values in this fashion, but merely small control flow or status information. Sending large data on this channel creates the risk of overloading the Driver at scale. The networking APIs provided by REEF IO are much better suited for data transmissions than this channel.</p> +<p>The parameter given to the call method is also to be used in a similar fashion: The Driver passes its value as part of the Task submission. It is meant e.g. to convey a restart point for the task. Note that the same functionality can now be better provided by Tang and a constructor parameter.</p></div> +<div class="section"> +<h4>Communicating between a Task and a Driver<a name="Communicating_between_a_Task_and_a_Driver"></a></h4> +<p>REEF provides some facilities to communicate between a Driver and a Task. These mostly stem from allowing application code to “free-ride” on REEF’s control flow channels such as the heartbeat between the Evaluator and the Task.</p> +<div class="section"> +<h5>Sending a message from the Driver to a Task<a name="Sending_a_message_from_the_Driver_to_a_Task"></a></h5> +<p>REEF maintains a heartbeat between any Evaluator and the Driver. There are two ways by which a heartbeat can be triggered.</p> + +<ul> + +<li> +<p>Upon some schedule (which may also vary at runtime due to load conditions on the Driver), each Evaluator will report its current status to the Driver. This is used by the Driver to maintain health status and load statistics of the Evaluators.</p></li> + +<li> +<p>Whenever the status of the Evaluator changes (e.g. when a Task completes), a Heartbeat is triggered immediately.</p></li> +</ul> +<p>Whenever the Evaluator performs a heartbeat, it will ask the Task whether it has any message to share with the Driver by inquiring the class registered in <tt>TaskConfiguration.ON_SEND_MESSAGE</tt>. It is wise for that message to be small, as we otherwise run the risk of overwhelming the Driver with heartbeat traffic at scale.</p></div></div> +<div class="section"> +<h4>Multithreaded Tasks<a name="Multithreaded_Tasks"></a></h4> +<p>Just because REEF views a Task as a method call doesn’t restrict the Task to be single threaded. A Task is free to spawn threads in the course of its execution. However, a Task that does so needs to take care of a few considerations:</p> + +<ul> + +<li> +<p>All Threads spawned need to exit before the <tt>Task.call()</tt> method returns. Otherwise, you run the risk of resource leakage.</p></li> + +<li> +<p>Exceptions on spawned Threads need to be caught and re-thrown by the <tt>Thread.call()</tt> method. Before that, all spawned threads need to be shut down, just like during a normal exit of <tt>Task.call()</tt>. If an exception from an another thread isn’t caught, REEF’s JVM level exception handler will catch it and declare a FailedEvaluator. This is inefficient, but not technically wrong: The Driver will then have to allocate another Evaluator and try again.</p></li> +</ul></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<div class="container-fluid"> + <div class="row-fluid"> + <a href="http://www.apache.org";>Apache Software Foundation</a> + </div> +</div> + +<div class="row span12">Apache REEF, REEF, Apache, the Apache feather logo, and the Apache REEF logo are trademarks + of The Apache Software Foundation. All other marks mentioned may be trademarks or registered + trademarks of their respective owners.</div> + </div> + + + </div> + </footer> + </body> +</html>
