Modified: incubator/reef/site/index.html URL: http://svn.apache.org/viewvc/incubator/reef/site/index.html?rev=1696854&r1=1696853&r2=1696854&view=diff ============================================================================== --- incubator/reef/site/index.html (original) +++ incubator/reef/site/index.html Thu Aug 20 20:44:29 2015 @@ -1,480 +1,480 @@ -<!DOCTYPE html> -<!-- - | Generated by Apache Maven Doxia at 2015-08-19 - | Rendered using Apache Maven Fluido Skin 1.4 ---> -<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> - <head> - <meta charset="UTF-8" /> - <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150819" /> - <meta http-equiv="Content-Language" content="en" /> - <title>Apache REEF - Apache REEF</title> - <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> - <link rel="stylesheet" href="./css/site.css" /> - <link rel="stylesheet" href="./css/print.css" media="print" /> - - - <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> - - - -<script src="js/release.js" type="text/javascript"></script> - - </head> - <body class="topBarEnabled"> - - - - - - <div id="topbar" class="navbar navbar-fixed-top "> - <div class="navbar-inner"> - <div class="container-fluid"> - <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </a> - - <ul class="nav"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache REEF <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="index.html" title="Overview">Overview</a> -</li> - - <li> <a href="faq.html" title="FAQ">FAQ</a> -</li> - - <li> <a href="license.html" title="License">License</a> -</li> - - <li> <a href="downloads.html" title="Downloads">Downloads</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="introduction.html" title="Introduction to REEF">Introduction to REEF</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; title="REEF Tutorial">REEF Tutorial</a> -</li> - - <li> <a href="talks.html" title="Talks">Talks</a> -</li> - - <li> <a href="glossary.html" title="Glossary">Glossary</a> -</li> - - <li> <a href="tang.html" title="Tang">Tang</a> -</li> - - <li> <a href="wake.html" title="Wake">Wake</a> -</li> - - <li> <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API">0.10.0-incubating API</a> -</li> - - <li> <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API">0.11.0-incubating API</a> -</li> - - <li> <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API">0.12.0-incubating API</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribution <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; title="Contributing">Contributing</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; title="Committer Guide">Committer Guide</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; title="Coding Guidelines">Coding Guidelines</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="team.html" title="Team">Team</a> -</li> - - <li> <a href="mailing-list.html" title="Mailing List">Mailing List</a> -</li> - - <li> <a href="https://issues.apache.org/jira/browse/REEF"; title="Issue Tracker">Issue Tracker</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; title="Powered By">Powered By</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/foundation/"; title="Apache Software Foundation">Apache Software Foundation</a> -</li> - - <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> -</li> - - <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> -</li> - - <li> <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; title="Apache License">Apache License</a> -</li> - - <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsorship">Sponsorship</a> -</li> - - <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> -</li> - </ul> - </li> - </ul> - - - - <ul class="nav pull-right"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li> <a href="https://github.com/apache/incubator-reef"; title="Apache REEF GitHub">Apache REEF GitHub</a> -</li> - <li> <a href="http://www.apache.org"; title="Apache">Apache</a> -</li> - <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> -</li> - </ul> - </li> - </ul> - - </div> - - </div> - </div> - </div> - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="./" id="bannerLeft" title="Apache REEF"> - <img src="ApacheREEF_logo_no_margin_small.png" alt="Apache REEF"/> - </a> - </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="ApacheIncubator.png" /> - </a> - </div> - <div class="clear"><hr/></div> - </div> - - <div id="breadcrumbs"> - <ul class="breadcrumb"> - - - <li class=""> - <a href="index.html" title="Apache REEF"> - Apache REEF</a> - <span class="divider">|</span> - </li> - <li class="active ">Apache REEF</li> - - - - - </ul> - </div> - - - <div class="row-fluid"> - <div id="leftColumn" class="span2"> - <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">Apache REEF</li> - - <li class="active"> - - <a href="#"><span class="none"></span>Overview</a> - </li> - - <li> - - <a href="faq.html" title="FAQ"> - <span class="none"></span> - FAQ</a> - </li> - - <li> - - <a href="license.html" title="License"> - <span class="none"></span> - License</a> - </li> - - <li> - - <a href="downloads.html" title="Downloads"> - <span class="none"></span> - Downloads</a> - </li> - <li class="nav-header">Documentation</li> - - <li> - - <a href="introduction.html" title="Introduction to REEF"> - <span class="none"></span> - Introduction to REEF</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; class="externalLink" title="REEF Tutorial"> - <span class="none"></span> - REEF Tutorial</a> - </li> - - <li> - - <a href="talks.html" title="Talks"> - <span class="none"></span> - Talks</a> - </li> - - <li> - - <a href="glossary.html" title="Glossary"> - <span class="none"></span> - Glossary</a> - </li> - - <li> - - <a href="tang.html" title="Tang"> - <span class="none"></span> - Tang</a> - </li> - - <li> - - <a href="wake.html" title="Wake"> - <span class="none"></span> - Wake</a> - </li> - - <li> - - <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API"> - <span class="none"></span> - 0.10.0-incubating API</a> - </li> - - <li> - - <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API"> - <span class="none"></span> - 0.11.0-incubating API</a> - </li> - - <li> - - <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API"> - <span class="none"></span> - 0.12.0-incubating API</a> - </li> - <li class="nav-header">Contribution</li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; class="externalLink" title="Contributing"> - <span class="none"></span> - Contributing</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; class="externalLink" title="Committer Guide"> - <span class="none"></span> - Committer Guide</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; class="externalLink" title="Coding Guidelines"> - <span class="none"></span> - Coding Guidelines</a> - </li> - <li class="nav-header">Community</li> - - <li> - - <a href="team.html" title="Team"> - <span class="none"></span> - Team</a> - </li> - - <li> - - <a href="mailing-list.html" title="Mailing List"> - <span class="none"></span> - Mailing List</a> - </li> - - <li> - - <a href="https://issues.apache.org/jira/browse/REEF"; class="externalLink" title="Issue Tracker"> - <span class="none"></span> - Issue Tracker</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; class="externalLink" title="Powered By"> - <span class="none"></span> - Powered By</a> - </li> - <li class="nav-header">ASF</li> - - <li> - - <a href="http://www.apache.org/foundation/"; class="externalLink" title="Apache Software Foundation"> - <span class="none"></span> - Apache Software Foundation</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> - <span class="none"></span> - How Apache Works</a> - </li> - - <li> - - <a href="http://incubator.apache.org/"; class="externalLink" title="Apache Incubator"> - <span class="none"></span> - Apache Incubator</a> - </li> - - <li> - - <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; class="externalLink" title="Apache License"> - <span class="none"></span> - Apache License</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsorship"> - <span class="none"></span> - Sponsorship</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> - <span class="none"></span> - Thanks</a> - </li> - </ul> - - - - <hr /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> - </a> - </div> - </div> - </div> - - - <div id="bodyColumn" class="span10" > - - <!-- Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. --><h1>Apache REEF</h1> -<div class="section"> -<div class="section"> -<h3>What is REEF?<a name="What_is_REEF"></a></h3> -<p><b>REEF</b>, the Retainable Evaluator Execution Framework, is our approach to simplify and unify the lower layers of big data systems on modern resource managers.</p> -<p>For managers like Apache YARN, Apache Mesos, Google Omega, and Facebook Corona, REEF provides a centralized control plane abstraction that can be used to build a decentralized data plane for supporting big data systems. Special consideration is given to graph computation and machine learning applications, both of which require data <i>retention</i> on allocated resources to execute multiple passes over the data.</p> -<p>More broadly, applications that run on YARN will have the need for a variety of data-processing tasks e.g., data shuffle, group communication, aggregation, checkpointing, and many more. Rather than reimplement these for each application, REEF aims to provide them in a library form, so that they can be reused by higher-level applications and tuned for a specific domain problem e.g., Machine Learning.</p> -<p>In that sense, our long-term vision is that REEF will mature into a Big Data Application Server, that will host a variety of tool kits and applications, on modern resource managers.</p> - - - <img src="REEFDiagram.png" alt="" /> - </img> -</div> -<div class="section"> -<h3>How can I get started?<a name="How_can_I_get_started"></a></h3> -<p>The official home for the REEF (and Tang and Wake) source code is at the Apache Software Foundation. You can check out the current code via:</p> - -<div class="source"><pre class="prettyprint">$ git clone git://git.apache.org/incubator-reef.git -</pre></div> -<p>or directly access its GitHub page <a class="externalLink" href="https://github.com/apache/incubator-reef";>here</a>.</p> -<p>Detailed information about REEF and using it can be found in the <a href="faq.html">FAQ</a> and the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/REEF/Tutorials";>Tutorial</a>.</p> -<p>If you wish to contribute, start at the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/REEF/Contributing";>Contributing</a> tutorial or the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide";>Committer Guide</a>!</p></div> -<div class="section"> -<h3>Further questions?<a name="Further_questions"></a></h3> -<p>Please visit our <a href="faq.html">Frequently Asked Questions</a> page or use our <a href="mailing-list.html">Mailing List</a> to send us a question!</p></div> -<div class="section"> -<h3>Disclaimer<a name="Disclaimer"></a></h3> -<p>Apache REEF is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p></div></div> - </div> - </div> - </div> - - <hr/> - - <footer> - <div class="container-fluid"> - <div class="row-fluid"> - -<div class="container-fluid"> - <div class="row-fluid"> - <a href="http://www.apache.org";>Apache Software Foundation</a> - </div> -</div> - -<div class="row span12">Apache REEF, REEF, Apache, the Apache feather logo, and the Apache REEF logo are trademarks - of The Apache Software Foundation. All other marks mentioned may be trademarks or registered - trademarks of their respective owners.</div> - </div> - - - </div> - </footer> - </body> +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-08-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20150820" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache REEF - Apache REEF</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="js/release.js" type="text/javascript"></script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache REEF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="Overview">Overview</a> +</li> + + <li> <a href="faq.html" title="FAQ">FAQ</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + + <li> <a href="downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="introduction.html" title="Introduction to REEF">Introduction to REEF</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; title="REEF Tutorial">REEF Tutorial</a> +</li> + + <li> <a href="talks.html" title="Talks">Talks</a> +</li> + + <li> <a href="glossary.html" title="Glossary">Glossary</a> +</li> + + <li> <a href="tang.html" title="Tang">Tang</a> +</li> + + <li> <a href="wake.html" title="Wake">Wake</a> +</li> + + <li> <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API">0.10.0-incubating API</a> +</li> + + <li> <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API">0.11.0-incubating API</a> +</li> + + <li> <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API">0.12.0-incubating API</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribution <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; title="Contributing">Contributing</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; title="Committer Guide">Committer Guide</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; title="Coding Guidelines">Coding Guidelines</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="team.html" title="Team">Team</a> +</li> + + <li> <a href="mailing-list.html" title="Mailing List">Mailing List</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/REEF"; title="Issue Tracker">Issue Tracker</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; title="Powered By">Powered By</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> +</li> + + <li> <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; title="Apache License">Apache License</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsorship">Sponsorship</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + + + <ul class="nav pull-right"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li> <a href="https://github.com/apache/incubator-reef"; title="Apache REEF GitHub">Apache REEF GitHub</a> +</li> + <li> <a href="http://www.apache.org"; title="Apache">Apache</a> +</li> + <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> +</li> + </ul> + </li> + </ul> + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="./" id="bannerLeft" title="Apache REEF"> + <img src="ApacheREEF_logo_no_margin_small.png" alt="Apache REEF"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="ApacheIncubator.png" /> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="index.html" title="Apache REEF"> + Apache REEF</a> + <span class="divider">|</span> + </li> + <li class="active ">Apache REEF</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache REEF</li> + + <li class="active"> + + <a href="#"><span class="none"></span>Overview</a> + </li> + + <li> + + <a href="faq.html" title="FAQ"> + <span class="none"></span> + FAQ</a> + </li> + + <li> + + <a href="license.html" title="License"> + <span class="none"></span> + License</a> + </li> + + <li> + + <a href="downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentation</li> + + <li> + + <a href="introduction.html" title="Introduction to REEF"> + <span class="none"></span> + Introduction to REEF</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; class="externalLink" title="REEF Tutorial"> + <span class="none"></span> + REEF Tutorial</a> + </li> + + <li> + + <a href="talks.html" title="Talks"> + <span class="none"></span> + Talks</a> + </li> + + <li> + + <a href="glossary.html" title="Glossary"> + <span class="none"></span> + Glossary</a> + </li> + + <li> + + <a href="tang.html" title="Tang"> + <span class="none"></span> + Tang</a> + </li> + + <li> + + <a href="wake.html" title="Wake"> + <span class="none"></span> + Wake</a> + </li> + + <li> + + <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API"> + <span class="none"></span> + 0.10.0-incubating API</a> + </li> + + <li> + + <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API"> + <span class="none"></span> + 0.11.0-incubating API</a> + </li> + + <li> + + <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API"> + <span class="none"></span> + 0.12.0-incubating API</a> + </li> + <li class="nav-header">Contribution</li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; class="externalLink" title="Contributing"> + <span class="none"></span> + Contributing</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; class="externalLink" title="Committer Guide"> + <span class="none"></span> + Committer Guide</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; class="externalLink" title="Coding Guidelines"> + <span class="none"></span> + Coding Guidelines</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="team.html" title="Team"> + <span class="none"></span> + Team</a> + </li> + + <li> + + <a href="mailing-list.html" title="Mailing List"> + <span class="none"></span> + Mailing List</a> + </li> + + <li> + + <a href="https://issues.apache.org/jira/browse/REEF"; class="externalLink" title="Issue Tracker"> + <span class="none"></span> + Issue Tracker</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; class="externalLink" title="Powered By"> + <span class="none"></span> + Powered By</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <span class="none"></span> + How Apache Works</a> + </li> + + <li> + + <a href="http://incubator.apache.org/"; class="externalLink" title="Apache Incubator"> + <span class="none"></span> + Apache Incubator</a> + </li> + + <li> + + <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; class="externalLink" title="Apache License"> + <span class="none"></span> + Apache License</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsorship"> + <span class="none"></span> + Sponsorship</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <span class="none"></span> + Thanks</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <!-- Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. --><h1>Apache REEF</h1> +<div class="section"> +<div class="section"> +<h3>What is REEF?<a name="What_is_REEF"></a></h3> +<p><b>REEF</b>, the Retainable Evaluator Execution Framework, is our approach to simplify and unify the lower layers of big data systems on modern resource managers.</p> +<p>For managers like Apache YARN, Apache Mesos, Google Omega, and Facebook Corona, REEF provides a centralized control plane abstraction that can be used to build a decentralized data plane for supporting big data systems. Special consideration is given to graph computation and machine learning applications, both of which require data <i>retention</i> on allocated resources to execute multiple passes over the data.</p> +<p>More broadly, applications that run on YARN will have the need for a variety of data-processing tasks e.g., data shuffle, group communication, aggregation, checkpointing, and many more. Rather than reimplement these for each application, REEF aims to provide them in a library form, so that they can be reused by higher-level applications and tuned for a specific domain problem e.g., Machine Learning.</p> +<p>In that sense, our long-term vision is that REEF will mature into a Big Data Application Server, that will host a variety of tool kits and applications, on modern resource managers.</p> + + + <img src="REEFDiagram.png" alt="" /> + </img> +</div> +<div class="section"> +<h3>How can I get started?<a name="How_can_I_get_started"></a></h3> +<p>The official home for the REEF (and Tang and Wake) source code is at the Apache Software Foundation. You can check out the current code via:</p> + +<div class="source"><pre class="prettyprint">$ git clone git://git.apache.org/incubator-reef.git +</pre></div> +<p>or directly access its GitHub page <a class="externalLink" href="https://github.com/apache/incubator-reef";>here</a>.</p> +<p>Detailed information about REEF and using it can be found in the <a href="faq.html">FAQ</a> and the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/REEF/Tutorials";>Tutorial</a>.</p> +<p>If you wish to contribute, start at the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/REEF/Contributing";>Contributing</a> tutorial or the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide";>Committer Guide</a>!</p></div> +<div class="section"> +<h3>Further questions?<a name="Further_questions"></a></h3> +<p>Please visit our <a href="faq.html">Frequently Asked Questions</a> page or use our <a href="mailing-list.html">Mailing List</a> to send us a question!</p></div> +<div class="section"> +<h3>Disclaimer<a name="Disclaimer"></a></h3> +<p>Apache REEF is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.</p></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<div class="container-fluid"> + <div class="row-fluid"> + <a href="http://www.apache.org";>Apache Software Foundation</a> + </div> +</div> + +<div class="row span12">Apache REEF, REEF, Apache, the Apache feather logo, and the Apache REEF logo are trademarks + of The Apache Software Foundation. All other marks mentioned may be trademarks or registered + trademarks of their respective owners.</div> + </div> + + + </div> + </footer> + </body> </html> \ No newline at end of file
Modified: incubator/reef/site/introduction.html URL: http://svn.apache.org/viewvc/incubator/reef/site/introduction.html?rev=1696854&r1=1696853&r2=1696854&view=diff ============================================================================== --- incubator/reef/site/introduction.html (original) +++ incubator/reef/site/introduction.html Thu Aug 20 20:44:29 2015 @@ -1,487 +1,487 @@ -<!DOCTYPE html> -<!-- - | Generated by Apache Maven Doxia at 2015-08-19 - | Rendered using Apache Maven Fluido Skin 1.4 ---> -<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> - <head> - <meta charset="UTF-8" /> - <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20150819" /> - <meta http-equiv="Content-Language" content="en" /> - <title>Apache REEF - Introduction to REEF</title> - <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> - <link rel="stylesheet" href="./css/site.css" /> - <link rel="stylesheet" href="./css/print.css" media="print" /> - - - <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> - - - -<script src="js/release.js" type="text/javascript"></script> - - </head> - <body class="topBarEnabled"> - - - - - - <div id="topbar" class="navbar navbar-fixed-top "> - <div class="navbar-inner"> - <div class="container-fluid"> - <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </a> - - <ul class="nav"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache REEF <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="index.html" title="Overview">Overview</a> -</li> - - <li> <a href="faq.html" title="FAQ">FAQ</a> -</li> - - <li> <a href="license.html" title="License">License</a> -</li> - - <li> <a href="downloads.html" title="Downloads">Downloads</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="introduction.html" title="Introduction to REEF">Introduction to REEF</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; title="REEF Tutorial">REEF Tutorial</a> -</li> - - <li> <a href="talks.html" title="Talks">Talks</a> -</li> - - <li> <a href="glossary.html" title="Glossary">Glossary</a> -</li> - - <li> <a href="tang.html" title="Tang">Tang</a> -</li> - - <li> <a href="wake.html" title="Wake">Wake</a> -</li> - - <li> <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API">0.10.0-incubating API</a> -</li> - - <li> <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API">0.11.0-incubating API</a> -</li> - - <li> <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API">0.12.0-incubating API</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribution <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; title="Contributing">Contributing</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; title="Committer Guide">Committer Guide</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; title="Coding Guidelines">Coding Guidelines</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="team.html" title="Team">Team</a> -</li> - - <li> <a href="mailing-list.html" title="Mailing List">Mailing List</a> -</li> - - <li> <a href="https://issues.apache.org/jira/browse/REEF"; title="Issue Tracker">Issue Tracker</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; title="Powered By">Powered By</a> -</li> - </ul> - </li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> - <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/foundation/"; title="Apache Software Foundation">Apache Software Foundation</a> -</li> - - <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> -</li> - - <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> -</li> - - <li> <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; title="Apache License">Apache License</a> -</li> - - <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsorship">Sponsorship</a> -</li> - - <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> -</li> - </ul> - </li> - </ul> - - - - <ul class="nav pull-right"> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li> <a href="https://github.com/apache/incubator-reef"; title="Apache REEF GitHub">Apache REEF GitHub</a> -</li> - <li> <a href="http://www.apache.org"; title="Apache">Apache</a> -</li> - <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> -</li> - </ul> - </li> - </ul> - - </div> - - </div> - </div> - </div> - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="./" id="bannerLeft" title="Apache REEF"> - <img src="ApacheREEF_logo_no_margin_small.png" alt="Apache REEF"/> - </a> - </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="ApacheIncubator.png" /> - </a> - </div> - <div class="clear"><hr/></div> - </div> - - <div id="breadcrumbs"> - <ul class="breadcrumb"> - - - <li class=""> - <a href="index.html" title="Apache REEF"> - Apache REEF</a> - <span class="divider">|</span> - </li> - <li class="active ">Introduction to REEF</li> - - - - - </ul> - </div> - - - <div class="row-fluid"> - <div id="leftColumn" class="span2"> - <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">Apache REEF</li> - - <li> - - <a href="index.html" title="Overview"> - <span class="none"></span> - Overview</a> - </li> - - <li> - - <a href="faq.html" title="FAQ"> - <span class="none"></span> - FAQ</a> - </li> - - <li> - - <a href="license.html" title="License"> - <span class="none"></span> - License</a> - </li> - - <li> - - <a href="downloads.html" title="Downloads"> - <span class="none"></span> - Downloads</a> - </li> - <li class="nav-header">Documentation</li> - - <li class="active"> - - <a href="#"><span class="none"></span>Introduction to REEF</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; class="externalLink" title="REEF Tutorial"> - <span class="none"></span> - REEF Tutorial</a> - </li> - - <li> - - <a href="talks.html" title="Talks"> - <span class="none"></span> - Talks</a> - </li> - - <li> - - <a href="glossary.html" title="Glossary"> - <span class="none"></span> - Glossary</a> - </li> - - <li> - - <a href="tang.html" title="Tang"> - <span class="none"></span> - Tang</a> - </li> - - <li> - - <a href="wake.html" title="Wake"> - <span class="none"></span> - Wake</a> - </li> - - <li> - - <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API"> - <span class="none"></span> - 0.10.0-incubating API</a> - </li> - - <li> - - <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API"> - <span class="none"></span> - 0.11.0-incubating API</a> - </li> - - <li> - - <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API"> - <span class="none"></span> - 0.12.0-incubating API</a> - </li> - <li class="nav-header">Contribution</li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; class="externalLink" title="Contributing"> - <span class="none"></span> - Contributing</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; class="externalLink" title="Committer Guide"> - <span class="none"></span> - Committer Guide</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; class="externalLink" title="Coding Guidelines"> - <span class="none"></span> - Coding Guidelines</a> - </li> - <li class="nav-header">Community</li> - - <li> - - <a href="team.html" title="Team"> - <span class="none"></span> - Team</a> - </li> - - <li> - - <a href="mailing-list.html" title="Mailing List"> - <span class="none"></span> - Mailing List</a> - </li> - - <li> - - <a href="https://issues.apache.org/jira/browse/REEF"; class="externalLink" title="Issue Tracker"> - <span class="none"></span> - Issue Tracker</a> - </li> - - <li> - - <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; class="externalLink" title="Powered By"> - <span class="none"></span> - Powered By</a> - </li> - <li class="nav-header">ASF</li> - - <li> - - <a href="http://www.apache.org/foundation/"; class="externalLink" title="Apache Software Foundation"> - <span class="none"></span> - Apache Software Foundation</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> - <span class="none"></span> - How Apache Works</a> - </li> - - <li> - - <a href="http://incubator.apache.org/"; class="externalLink" title="Apache Incubator"> - <span class="none"></span> - Apache Incubator</a> - </li> - - <li> - - <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; class="externalLink" title="Apache License"> - <span class="none"></span> - Apache License</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsorship"> - <span class="none"></span> - Sponsorship</a> - </li> - - <li> - - <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> - <span class="none"></span> - Thanks</a> - </li> - </ul> - - - - <hr /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> - </a> - </div> - </div> - </div> - - - <div id="bodyColumn" class="span10" > - - <!-- Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. --><h1>Introduction to REEF</h1> -<p>It is instructive to first remind us of how MapReduce works: Hadoop Map-Reduce schedules compute tasks on containers for executing map and reduce functions on record data. The basic structure of a Map-Reduce job is as follows. For each input block, schedule a map task that passes each internal record to a user-defined map function and materializes the output in key-sorted order. Further, assign a user-defined number of reduce tasks to non-overlapping portions of the key-space from the map output, then shuffle it across the network to where the respective reduce task is scheduled. For each reduce task, perform a global key-based sort on the shuffled data, group it by key and call the reduce function on each record group; storing the output in a durable fashion (i.e., HDFS).</p> -<p>From the perspective of the scheduler, a number of issues arise that must be appropriately handled in order to scale-out to massive datasets. First, each map task should be scheduled close to where the input block resides; ideally on the same machine or rack. Second, failures can occur at the task level at any step; requiring backup tasks to be scheduled or the job being aborted. Third, performance bottlenecks can cause an imbalance in the task-level progress. The scheduler must react to these stragglers by scheduling clones and incorporating the logical task that crosses the finish line first.</p> -<p>Anyone of these issues can limit the scale-out degree of a Map-Reduce job. In what follows, we perscribe a scheduler framework that provides task life-cycle management mechanisms. Using this framework, we developed a complete version of the Map-Reduce runtime that addresses the above issues. Our framework is designed around three components.</p> - -<ol style="list-style-type: decimal"> - -<li><tt>Driver</tt>: captures the client code embodying the scheduler</li> - -<li><tt>Evaluator</tt>: provides a runtime environment on a container.</li> - -<li><tt>Task</tt>: encapsulates the task-level client code to be executed in an Evaluator.</li> -</ol> -<p>Below, we describe the client facing interfaces to these components. The core REEF control flow design is based on the reactive extensions (Rx), which enforce asynchronous message-passing method signatures. In Rx terms, interfaces are based on an observer pattern, which expose methods that accept messages from a (possibly remote) asynchronous caller.</p> -<p><br /></br></p> - - <img src="reef-architecture.png" alt="" /></img> - -<p><i>A running REEF job with two <tt>RunningEvaluators</tt> and one <tt>RunningTask</tt></i></p> -<p><br /></br></p> -<p>The above figure presents the REEF components in terms of a running application, which is written in terms of a <tt>Driver</tt> and task-specific <tt>Task</tt> modules. The application code is packaged and submitted to a REEF client API, which in turn submits a REEF-AM configuration to YARN. The REEF-AM contains a runtime for launching the <tt>Driver</tt> and client libraries for requesting Evaluators and launching <tt>Activities</tt>. When a request for <tt>Evaluators</tt> is made, the REEF-AM negotiates containers with the YARN-RM and launches an <tt>Evaluator</tt> runtime on the YARN-NM that hosts the allocated container. In turn, the <tt>Driver</tt> is given an <tt>Evaluator</tt> object reference, which it uses to submit an <tt>Task</tt>. The <tt>Driver</tt> is also given a <tt>Task</tt> object reference, which it may use to send messages to the <tt>Task</tt> running in the <tt>Evaluator</tt>. The REEF layer implements these communication channels and encodes the computationa l life-cycle as state transitions, which are surfaced to the <tt>Driver</tt> in the form of Rx messages.</p> -<div class="section"> -<div class="section"> -<h3>Computational Life-Cycle<a name="Computational_Life-Cycle"></a></h3> -<p><br /></br></p> - - <img src="states-horizontal.png" alt="" /></img> - -<p><i>States of <tt>Evaluator</tt>, <tt>Contexts</tt>, and <tt>Activities</tt></i></p> -<p><br /></br></p> -<p>The Figure above describes the state transitions for (a) <tt>Evaluator</tt> and <tt>Context</tt> and (b) <tt>Task</tt> components. Each state transition is associated with an object reference that is surfaced to the <tt>Driver</tt> in an Rx-style interface. For instance, when the YARN-RM notifies the REEF-AM of an allocated container, the <tt>Driver</tt> is given an <tt>AllocatedEvaluator</tt> object; containing methods for adding configurations (i.e., for data services, see below) and file resources, and submit methods that bootstraps the <tt>Evaluator</tt> runtime on the YARN-NM. When an <tt>Evaluator</tt> bootstrap successfully completes, the <tt>Driver</tt> is given an <tt>ActiveContext</tt> object, which can be used to launch <tt>Activities</tt> or to initiate a close, which triggers a shutdown event at the <tt>Evaluator</tt> runtime and a subsequent container release at the YARN-RM. If at any point a failure occurs, the <tt>Driver</tt> is passed a <tt>FailedEvaluator</tt> o bject; containing an exception trace when possible.</p> -<p>Recall that the <tt>Driver</tt> launches a <tt>Task</tt> on a submit method call from the <tt>ActiveContext</tt> reference. This state transition is denoted in the Figure above by the edge labeled submit; spanning the two state machines. The REEF-AM passes a <tt>RunningActivity</tt> object to the <tt>Driver</tt> after receiving confirmation of a successful <tt>Task</tt> start or resume. The <tt>Driver</tt> may use the <tt>RunningActivity</tt> reference to close or suspend the execution; triggering a <tt>CompletedActivity</tt> or <tt>SuspendedActivity</tt> object reference to the <tt>Driver</tt>. The <tt>SuspendedActivity</tt> object contains a memento used to resume the execution on some (possibly alternative) <tt>ActiveContext</tt>. Exceptions during the <tt>Task</tt> execution are surfaced to the <tt>Driver</tt> in the form of a <tt>FailedActivity</tt>, which contains the actual exception object.</p></div> -<div class="section"> -<h3>Task Component<a name="Task_Component"></a></h3> -<p>A <tt>Task</tt> encapsulates the task work of a job. The client interface contains a single synchronous call method that takes an optional memento argument and returns a byte array, which will be packaged with the <tt>CompletedActivity</tt> object surfaced to the <tt>Driver</tt>. An exception may be thrown at any point during the call method; returning control back to the <tt>Evaluator</tt>, which packages the exception and sends it to the <tt>Driver</tt> where it is surfaced as a <tt>FailedActivity</tt>. The <tt>Evaluator</tt> periodically performs a heartbeat with the REEF-AM to convey its status information. A <tt>Task</tt> can optionally implement a method interface that, when called, returns a (bounded) byte array, which the <tt>Evaluator</tt> includes in its heartbeat to the REEF-AM and surfaced to the <tt>Driver</tt>.</p></div></div> - </div> - </div> - </div> - - <hr/> - - <footer> - <div class="container-fluid"> - <div class="row-fluid"> - -<div class="container-fluid"> - <div class="row-fluid"> - <a href="http://www.apache.org";>Apache Software Foundation</a> - </div> -</div> - -<div class="row span12">Apache REEF, REEF, Apache, the Apache feather logo, and the Apache REEF logo are trademarks - of The Apache Software Foundation. All other marks mentioned may be trademarks or registered - trademarks of their respective owners.</div> - </div> - - - </div> - </footer> - </body> +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-08-20 + | Rendered using Apache Maven Fluido Skin 1.4 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20150820" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache REEF - Introduction to REEF</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.4.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.4.min.js"></script> + + + +<script src="js/release.js" type="text/javascript"></script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container-fluid"> + <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </a> + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache REEF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="Overview">Overview</a> +</li> + + <li> <a href="faq.html" title="FAQ">FAQ</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + + <li> <a href="downloads.html" title="Downloads">Downloads</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="introduction.html" title="Introduction to REEF">Introduction to REEF</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; title="REEF Tutorial">REEF Tutorial</a> +</li> + + <li> <a href="talks.html" title="Talks">Talks</a> +</li> + + <li> <a href="glossary.html" title="Glossary">Glossary</a> +</li> + + <li> <a href="tang.html" title="Tang">Tang</a> +</li> + + <li> <a href="wake.html" title="Wake">Wake</a> +</li> + + <li> <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API">0.10.0-incubating API</a> +</li> + + <li> <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API">0.11.0-incubating API</a> +</li> + + <li> <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API">0.12.0-incubating API</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribution <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; title="Contributing">Contributing</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; title="Committer Guide">Committer Guide</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; title="Coding Guidelines">Coding Guidelines</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="team.html" title="Team">Team</a> +</li> + + <li> <a href="mailing-list.html" title="Mailing List">Mailing List</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/REEF"; title="Issue Tracker">Issue Tracker</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; title="Powered By">Powered By</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/"; title="Apache Software Foundation">Apache Software Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> +</li> + + <li> <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; title="Apache License">Apache License</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsorship">Sponsorship</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + + + <ul class="nav pull-right"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">External Links <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li> <a href="https://github.com/apache/incubator-reef"; title="Apache REEF GitHub">Apache REEF GitHub</a> +</li> + <li> <a href="http://www.apache.org"; title="Apache">Apache</a> +</li> + <li> <a href="http://incubator.apache.org/"; title="Apache Incubator">Apache Incubator</a> +</li> + </ul> + </li> + </ul> + + </div> + + </div> + </div> + </div> + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="./" id="bannerLeft" title="Apache REEF"> + <img src="ApacheREEF_logo_no_margin_small.png" alt="Apache REEF"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> + <img src="ApacheIncubator.png" /> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="index.html" title="Apache REEF"> + Apache REEF</a> + <span class="divider">|</span> + </li> + <li class="active ">Introduction to REEF</li> + + + + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">Apache REEF</li> + + <li> + + <a href="index.html" title="Overview"> + <span class="none"></span> + Overview</a> + </li> + + <li> + + <a href="faq.html" title="FAQ"> + <span class="none"></span> + FAQ</a> + </li> + + <li> + + <a href="license.html" title="License"> + <span class="none"></span> + License</a> + </li> + + <li> + + <a href="downloads.html" title="Downloads"> + <span class="none"></span> + Downloads</a> + </li> + <li class="nav-header">Documentation</li> + + <li class="active"> + + <a href="#"><span class="none"></span>Introduction to REEF</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"; class="externalLink" title="REEF Tutorial"> + <span class="none"></span> + REEF Tutorial</a> + </li> + + <li> + + <a href="talks.html" title="Talks"> + <span class="none"></span> + Talks</a> + </li> + + <li> + + <a href="glossary.html" title="Glossary"> + <span class="none"></span> + Glossary</a> + </li> + + <li> + + <a href="tang.html" title="Tang"> + <span class="none"></span> + Tang</a> + </li> + + <li> + + <a href="wake.html" title="Wake"> + <span class="none"></span> + Wake</a> + </li> + + <li> + + <a href="apidocs/0.10.0-incubating/index.html" title="0.10.0-incubating API"> + <span class="none"></span> + 0.10.0-incubating API</a> + </li> + + <li> + + <a href="apidocs/0.11.0-incubating/index.html" title="0.11.0-incubating API"> + <span class="none"></span> + 0.11.0-incubating API</a> + </li> + + <li> + + <a href="apidocs/0.12.0-incubating/index.html" title="0.12.0-incubating API"> + <span class="none"></span> + 0.12.0-incubating API</a> + </li> + <li class="nav-header">Contribution</li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Contributing"; class="externalLink" title="Contributing"> + <span class="none"></span> + Contributing</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"; class="externalLink" title="Committer Guide"> + <span class="none"></span> + Committer Guide</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"; class="externalLink" title="Coding Guidelines"> + <span class="none"></span> + Coding Guidelines</a> + </li> + <li class="nav-header">Community</li> + + <li> + + <a href="team.html" title="Team"> + <span class="none"></span> + Team</a> + </li> + + <li> + + <a href="mailing-list.html" title="Mailing List"> + <span class="none"></span> + Mailing List</a> + </li> + + <li> + + <a href="https://issues.apache.org/jira/browse/REEF"; class="externalLink" title="Issue Tracker"> + <span class="none"></span> + Issue Tracker</a> + </li> + + <li> + + <a href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"; class="externalLink" title="Powered By"> + <span class="none"></span> + Powered By</a> + </li> + <li class="nav-header">ASF</li> + + <li> + + <a href="http://www.apache.org/foundation/"; class="externalLink" title="Apache Software Foundation"> + <span class="none"></span> + Apache Software Foundation</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/how-it-works.html"; class="externalLink" title="How Apache Works"> + <span class="none"></span> + How Apache Works</a> + </li> + + <li> + + <a href="http://incubator.apache.org/"; class="externalLink" title="Apache Incubator"> + <span class="none"></span> + Apache Incubator</a> + </li> + + <li> + + <a href="http://www.apache.org/licenses/LICENSE-2.0.html"; class="externalLink" title="Apache License"> + <span class="none"></span> + Apache License</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/sponsorship.html"; class="externalLink" title="Sponsorship"> + <span class="none"></span> + Sponsorship</a> + </li> + + <li> + + <a href="http://www.apache.org/foundation/thanks.html"; class="externalLink" title="Thanks"> + <span class="none"></span> + Thanks</a> + </li> + </ul> + + + + <hr /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span10" > + + <!-- Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. --><h1>Introduction to REEF</h1> +<p>It is instructive to first remind us of how MapReduce works: Hadoop Map-Reduce schedules compute tasks on containers for executing map and reduce functions on record data. The basic structure of a Map-Reduce job is as follows. For each input block, schedule a map task that passes each internal record to a user-defined map function and materializes the output in key-sorted order. Further, assign a user-defined number of reduce tasks to non-overlapping portions of the key-space from the map output, then shuffle it across the network to where the respective reduce task is scheduled. For each reduce task, perform a global key-based sort on the shuffled data, group it by key and call the reduce function on each record group; storing the output in a durable fashion (i.e., HDFS).</p> +<p>From the perspective of the scheduler, a number of issues arise that must be appropriately handled in order to scale-out to massive datasets. First, each map task should be scheduled close to where the input block resides; ideally on the same machine or rack. Second, failures can occur at the task level at any step; requiring backup tasks to be scheduled or the job being aborted. Third, performance bottlenecks can cause an imbalance in the task-level progress. The scheduler must react to these stragglers by scheduling clones and incorporating the logical task that crosses the finish line first.</p> +<p>Anyone of these issues can limit the scale-out degree of a Map-Reduce job. In what follows, we perscribe a scheduler framework that provides task life-cycle management mechanisms. Using this framework, we developed a complete version of the Map-Reduce runtime that addresses the above issues. Our framework is designed around three components.</p> + +<ol style="list-style-type: decimal"> + +<li><tt>Driver</tt>: captures the client code embodying the scheduler</li> + +<li><tt>Evaluator</tt>: provides a runtime environment on a container.</li> + +<li><tt>Task</tt>: encapsulates the task-level client code to be executed in an Evaluator.</li> +</ol> +<p>Below, we describe the client facing interfaces to these components. The core REEF control flow design is based on the reactive extensions (Rx), which enforce asynchronous message-passing method signatures. In Rx terms, interfaces are based on an observer pattern, which expose methods that accept messages from a (possibly remote) asynchronous caller.</p> +<p><br /></br></p> + + <img src="reef-architecture.png" alt="" /></img> + +<p><i>A running REEF job with two <tt>RunningEvaluators</tt> and one <tt>RunningTask</tt></i></p> +<p><br /></br></p> +<p>The above figure presents the REEF components in terms of a running application, which is written in terms of a <tt>Driver</tt> and task-specific <tt>Task</tt> modules. The application code is packaged and submitted to a REEF client API, which in turn submits a REEF-AM configuration to YARN. The REEF-AM contains a runtime for launching the <tt>Driver</tt> and client libraries for requesting <tt>Evaluators</tt> and launching <tt>Tasks</tt>. When a request for <tt>Evaluators</tt> is made, the REEF-AM negotiates containers with the YARN-RM and launches an <tt>Evaluator</tt> runtime on the YARN-NM that hosts the allocated container. In turn, the <tt>Driver</tt> is given an <tt>Evaluator</tt> object reference, which it uses to submit an <tt>Task</tt>. The <tt>Driver</tt> is also given a <tt>Task</tt> object reference, which it may use to send messages to the <tt>Task</tt> running in the <tt>Evaluator</tt>. The REEF layer implements these communication channels and encodes the computat ional life-cycle as state transitions, which are surfaced to the <tt>Driver</tt> in the form of Rx messages.</p> +<div class="section"> +<div class="section"> +<h3>Computational Life-Cycle<a name="Computational_Life-Cycle"></a></h3> +<p><br /></br></p> + + <img src="states-horizontal.png" alt="" /></img> + +<p><i>States of <tt>Evaluator</tt>, <tt>Contexts</tt>, and <tt>Tasks</tt></i></p> +<p><br /></br></p> +<p>The Figure above describes the state transitions for (a) <tt>Evaluator</tt> and <tt>Context</tt> and (b) <tt>Task</tt> components. Each state transition is associated with an object reference that is surfaced to the <tt>Driver</tt> in an Rx-style interface. For instance, when the YARN-RM notifies the REEF-AM of an allocated container, the <tt>Driver</tt> is given an <tt>AllocatedEvaluator</tt> object; containing methods for adding configurations (i.e., for data services, see below) and file resources, and submit methods that bootstraps the <tt>Evaluator</tt> runtime on the YARN-NM. When an <tt>Evaluator</tt> bootstrap successfully completes, the <tt>Driver</tt> is given an <tt>ActiveContext</tt> object, which can be used to launch <tt>Tasks</tt> or to initiate a close, which triggers a shutdown event at the <tt>Evaluator</tt> runtime and a subsequent container release at the YARN-RM. If at any point a failure occurs, the <tt>Driver</tt> is passed a <tt>FailedEvaluator</tt> object ; containing an exception trace when possible.</p> +<p>Recall that the <tt>Driver</tt> launches a <tt>Task</tt> on a submit method call from the <tt>ActiveContext</tt> reference. This state transition is denoted in the Figure above by the edge labeled submit; spanning the two state machines. The REEF-AM passes a <tt>RunningTask</tt> object to the <tt>Driver</tt> after receiving confirmation of a successful <tt>Task</tt> start or resume. The <tt>Driver</tt> may use the <tt>RunningTask</tt> reference to close or suspend the execution; triggering a <tt>CompletedTask</tt> or <tt>SuspendedTask</tt> object reference to the <tt>Driver</tt>. The <tt>SuspendedTask</tt> object contains a memento used to resume the execution on some (possibly alternative) <tt>ActiveContext</tt>. Exceptions during the <tt>Task</tt> execution are surfaced to the <tt>Driver</tt> in the form of a <tt>FailedTask</tt>, which contains the actual exception object.</p></div> +<div class="section"> +<h3>Task Component<a name="Task_Component"></a></h3> +<p>A <tt>Task</tt> encapsulates the task work of a job. The client interface contains a single synchronous call method that takes an optional memento argument and returns a byte array, which will be packaged with the <tt>CompletedTask</tt> object surfaced to the <tt>Driver</tt>. An exception may be thrown at any point during the call method; returning control back to the <tt>Evaluator</tt>, which packages the exception and sends it to the <tt>Driver</tt> where it is surfaced as a <tt>FailedTask</tt>. The <tt>Evaluator</tt> periodically performs a heartbeat with the REEF-AM to convey its status information. A <tt>Task</tt> can optionally implement a method interface that, when called, returns a (bounded) byte array, which the <tt>Evaluator</tt> includes in its heartbeat to the REEF-AM and surfaced to the <tt>Driver</tt>.</p></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row-fluid"> + +<div class="container-fluid"> + <div class="row-fluid"> + <a href="http://www.apache.org";>Apache Software Foundation</a> + </div> +</div> + +<div class="row span12">Apache REEF, REEF, Apache, the Apache feather logo, and the Apache REEF logo are trademarks + of The Apache Software Foundation. All other marks mentioned may be trademarks or registered + trademarks of their respective owners.</div> + </div> + + + </div> + </footer> + </body> </html> \ No newline at end of file
