http://git-wip-us.apache.org/repos/asf/storm-site/blob/6e122a12/content/releases/1.2.1/Joins.html ---------------------------------------------------------------------- diff --git a/content/releases/1.2.1/Joins.html b/content/releases/1.2.1/Joins.html deleted file mode 100644 index 98dcafa..0000000 --- a/content/releases/1.2.1/Joins.html +++ /dev/null @@ -1,350 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Joining Streams in Storm Core</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 1.2.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.1/index.html">1.2.1</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/06/04/storm122-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Joining Streams in Storm Core</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<div class="documentation-content"><p>Storm core supports joining multiple data streams into one with the help of <code>JoinBolt</code>. -<code>JoinBolt</code> is a Windowed bolt, i.e. it waits for the configured window duration to match up the -tuples among the streams being joined. This helps align the streams within a Window boundary.</p> - -<p>Each of <code>JoinBolt</code>'s incoming data streams must be Fields Grouped on a single field. A stream -should only be joined with the other streams using the field on which it has been FieldsGrouped.<br> -Knowing this will help understand the join syntax described below. </p> - -<h2 id="performing-joins">Performing Joins</h2> - -<p>Consider the following SQL join involving 4 tables:</p> -<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">select</span> <span class="n">userId</span><span class="p">,</span> <span class="n">key4</span><span class="p">,</span> <span class="n">key2</span><span class="p">,</span> <span class="n">key3</span> -<span class="k">from</span> <span class="n">table1</span> -<span class="k">inner</span> <span class="k">join</span> <span class="n">table2</span> <span class="k">on</span> <span class="n">table2</span><span class="p">.</span><span class="n">userId</span> <span class="o">=</span> <span class="n">table1</span><span class="p">.</span><span class="n">key1</span> -<span class="k">inner</span> <span class="k">join</span> <span class="n">table3</span> <span class="k">on</span> <span class="n">table3</span><span class="p">.</span><span class="n">key3</span> <span class="o">=</span> <span class="n">table2</span><span class="p">.</span><span class="n">userId</span> -<span class="k">left</span> <span class="k">join</span> <span class="n">table4</span> <span class="k">on</span> <span class="n">table4</span><span class="p">.</span><span class="n">key4</span> <span class="o">=</span> <span class="n">table3</span><span class="p">.</span><span class="n">key3</span> -</code></pre></div> -<p>Similar joins could be expressed on tuples generated by 4 spouts using <code>JoinBolt</code>:</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">JoinBolt</span> <span class="n">jbolt</span> <span class="o">=</span> <span class="k">new</span> <span class="n">JoinBolt</span><span class="o">(</span><span class="s">"spout1"</span><span class="o">,</span> <span class="s">"key1"</span><span class="o">)</span> <span class="c1">// from spout1 </span> - <span class="o">.</span><span class="na">join</span> <span class="o">(</span><span class="s">"spout2"</span><span class="o">,</span> <span class="s">"userId"</span><span class="o">,</span> <span class="s">"spout1"</span><span class="o">)</span> <span class="c1">// inner join spout2 on spout2.userId = spout1.key1</span> - <span class="o">.</span><span class="na">join</span> <span class="o">(</span><span class="s">"spout3"</span><span class="o">,</span> <span class="s">"key3"</span><span class="o">,</span> <span class="s">"spout2"</span><span class="o">)</span> <span class="c1">// inner join spout3 on spout3.key3 = spout2.userId </span> - <span class="o">.</span><span class="na">leftJoin</span> <span class="o">(</span><span class="s">"spout4"</span><span class="o">,</span> <span class="s">"key4"</span><span class="o">,</span> <span class="s">"spout3"</span><span class="o">)</span> <span class="c1">// left join spout4 on spout4.key4 = spout3.key3</span> - <span class="o">.</span><span class="na">select</span> <span class="o">(</span><span class="s">"userId, key4, key2, spout3:key3"</span><span class="o">)</span> <span class="c1">// chose output fields</span> - <span class="o">.</span><span class="na">withTumblingWindow</span><span class="o">(</span> <span class="k">new</span> <span class="n">Duration</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">MINUTES</span><span class="o">)</span> <span class="o">)</span> <span class="o">;</span> - -<span class="n">topoBuilder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="s">"joiner"</span><span class="o">,</span> <span class="n">jbolt</span><span class="o">,</span> <span class="mi">1</span><span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"spout1"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"key1"</span><span class="o">)</span> <span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"spout2"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"userId"</span><span class="o">)</span> <span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"spout3"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"key3"</span><span class="o">)</span> <span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"spout4"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"key4"</span><span class="o">)</span> <span class="o">);</span> -</code></pre></div> -<p>The bolt constructor takes two arguments. The 1st argument introduces the data from <code>spout1</code> -to be the first stream and specifies that it will always use field <code>key1</code> when joining this with the others streams. -The name of the component specified must refer to the spout or bolt that is directly connected to the Join bolt. -Here data received from <code>spout1</code> must be fields grouped on <code>key1</code>. Similarly, each of the <code>leftJoin()</code> and <code>join()</code> method -calls introduce a new stream along with the field to use for the join. As seen in above example, the same FieldsGrouping -requirement applies to these streams as well. The 3rd argument to the join methods refers to another stream with which -to join.</p> - -<p>The <code>select()</code> method is used to specify the output fields. The argument to <code>select</code> is a comma separated list of fields. -Individual field names can be prefixed with a stream name to disambiguate between the same field name occurring in -multiple streams as follows: <code>.select("spout3:key3, spout4:key3")</code>. Nested tuple types are supported if the -nesting has been done using <code>Map</code>s. For example <code>outer.inner.innermost</code> refers to a field that is nested three levels -deep where <code>outer</code> and <code>inner</code> are of type <code>Map</code>. </p> - -<p>Stream name prefix is not allowed for the fields in any of the join() arguments, but nested fields are supported. </p> - -<p>The call to <code>withTumblingWindow()</code> above, configures the join window to be a 10 minute tumbling window. Since <code>JoinBolt</code> -is a Windowed Bolt, we can also use the <code>withWindow</code> method to configure it as a sliding window (see tips section below). </p> - -<h2 id="stream-names-and-join-order">Stream names and Join order</h2> - -<ul> -<li>Stream names must be introduced (in constructor or as 1st arg to various join methods) before being referred -to (in the 3rd argument of the join methods). Forward referencing of stream names, as shown below, is not allowed:</li> -</ul> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="k">new</span> <span class="n">JoinBolt</span><span class="o">(</span> <span class="s">"spout1"</span><span class="o">,</span> <span class="s">"key1"</span><span class="o">)</span> - <span class="o">.</span><span class="na">join</span> <span class="o">(</span> <span class="s">"spout2"</span><span class="o">,</span> <span class="s">"userId"</span><span class="o">,</span> <span class="s">"spout3"</span><span class="o">)</span> <span class="c1">//not allowed. spout3 not yet introduced</span> - <span class="o">.</span><span class="na">join</span> <span class="o">(</span> <span class="s">"spout3"</span><span class="o">,</span> <span class="s">"key3"</span><span class="o">,</span> <span class="s">"spout1"</span><span class="o">)</span> -</code></pre></div> -<ul> -<li>Internally, the joins will be performed in the order expressed by the user.</li> -</ul> - -<h2 id="joining-based-on-stream-names">Joining based on Stream names</h2> - -<p>For simplicity, Storm topologies often use the <code>default</code> stream. Topologies can also use named streams -instead of <code>default</code> streams. To support such topologies, <code>JoinBolt</code> can be configured to use stream -names, instead of source component (spout/bolt) names, via the constructor's first argument:</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="k">new</span> <span class="n">JoinBolt</span><span class="o">(</span><span class="n">JoinBolt</span><span class="o">.</span><span class="na">Selector</span><span class="o">.</span><span class="na">STREAM</span><span class="o">,</span> <span class="s">"stream1"</span><span class="o">,</span> <span class="s">"key1"</span><span class="o">)</span> - <span class="o">.</span><span class="na">join</span><span class="o">(</span><span class="s">"stream2"</span><span class="o">,</span> <span class="s">"key2"</span><span class="o">)</span> - <span class="o">...</span> -</code></pre></div> -<p>The first argument <code>JoinBolt.Selector.STREAM</code> informs the bolt that <code>stream1/2/3/4</code> refer to named streams -(as opposed to names of upstream spouts/bolts).</p> - -<p>The below example joins two named streams from four spouts:</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="k">new</span> <span class="n">JoinBolt</span><span class="o">(</span><span class="n">JoinBolt</span><span class="o">.</span><span class="na">Selector</span><span class="o">.</span><span class="na">STREAM</span><span class="o">,</span> <span class="s">"stream1"</span><span class="o">,</span> <span class="s">"key1"</span><span class="o">)</span> - <span class="o">.</span><span class="na">join</span> <span class="o">(</span><span class="s">"stream2"</span><span class="o">,</span> <span class="s">"userId"</span><span class="o">,</span> <span class="s">"stream1"</span> <span class="o">)</span> - <span class="o">.</span><span class="na">select</span> <span class="o">(</span><span class="s">"userId, key1, key2"</span><span class="o">)</span> - <span class="o">.</span><span class="na">withTumblingWindow</span><span class="o">(</span> <span class="k">new</span> <span class="n">Duration</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">MINUTES</span><span class="o">)</span> <span class="o">)</span> <span class="o">;</span> - -<span class="n">topoBuilder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="s">"joiner"</span><span class="o">,</span> <span class="n">jbolt</span><span class="o">,</span> <span class="mi">1</span><span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"bolt1"</span><span class="o">,</span> <span class="s">"stream1"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"key1"</span><span class="o">)</span> <span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"bolt2"</span><span class="o">,</span> <span class="s">"stream1"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"key1"</span><span class="o">)</span> <span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"bolt3"</span><span class="o">,</span> <span class="s">"stream2"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"userId"</span><span class="o">)</span> <span class="o">)</span> - <span class="o">.</span><span class="na">fieldsGrouping</span><span class="o">(</span><span class="s">"bolt4"</span><span class="o">,</span> <span class="s">"stream1"</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"key1"</span><span class="o">)</span> <span class="o">);</span> -</code></pre></div> -<p>In the above example, it is possible that <code>bolt1</code>, for example, is emitting other streams also. But the join bolt -is only subscribing to <code>stream1</code> & <code>stream2</code> from the different bolts. <code>stream1</code> from <code>bolt1</code>, <code>bolt2</code> and <code>bolt4</code> -is treated as a single stream and joined against <code>stream2</code> from <code>bolt3</code>.</p> - -<h2 id="limitations">Limitations:</h2> - -<ol> -<li><p>Currently only INNER and LEFT joins are supported. </p></li> -<li><p>Unlike SQL, which allows joining the same table on different keys to different tables, here the same one field must be used -on a stream. Fields Grouping ensures the right tuples are routed to the right instances of a Join Bolt. Consequently the -FieldsGrouping field must be same as the join field, for correct results. To perform joins on multiple fields, the fields -can be combined into one field and then sent to the Join bolt. </p></li> -</ol> - -<h2 id="tips">Tips:</h2> - -<ol> -<li><p>Joins can be CPU and memory intensive. The larger the data accumulated in the current window (proportional to window -length), the longer it takes to do the join. Having a short sliding interval (few seconds for example) triggers frequent -joins. Consequently performance can suffer if using large window lengths or small sliding intervals or both.</p></li> -<li><p>Duplication of joined records across windows is possible when using Sliding Windows. This is because the tuples continue to exist -across multiple windows when using Sliding Windows.</p></li> -<li><p>If message timeouts are enabled, ensure the timeout setting (topology.message.timeout.secs) is large enough to comfortably -accommodate the window size, plus the additional processing by other spouts and bolts.</p></li> -<li><p>Joining a window of two streams with M and N elements each, <em>in the worst case</em>, can produce MxN elements with every output tuple -anchored to one tuple from each input stream. This can mean a lot of output tuples from JoinBolt and even more ACKs for downstream bolts -to emit. This can place a substantial pressure on the messaging system and dramatically slowdown the topology if not careful. -To manage the load on the messaging subsystem, it is advisable to:</p> - -<ul> -<li>Increase the worker's heap (topology.worker.max.heap.size.mb).</li> -<li><strong>If</strong> ACKing is not necessary for your topology, disable ACKers (topology.acker.executors=0).</li> -<li>Disable event logger (topology.eventlogger.executors=0).</li> -<li>Turn of topology debugging (topology.debug=false).</li> -<li>Set topology.max.spout.pending to a value large enough to accommodate an estimated full window worth of tuples plus some more for headroom. -This helps mitigate the possibility of spouts emitting excessive tuples when messaging subsystem is experiencing excessive load. This situation -can occur when its value is set to null.</li> -<li>Lastly, keep the window size to the minimum value necessary for solving the problem at hand.</li> -</ul></li> -</ol> -</div> - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> -
http://git-wip-us.apache.org/repos/asf/storm-site/blob/6e122a12/content/releases/1.2.1/Kestrel-and-Storm.html ---------------------------------------------------------------------- diff --git a/content/releases/1.2.1/Kestrel-and-Storm.html b/content/releases/1.2.1/Kestrel-and-Storm.html deleted file mode 100644 index df98144..0000000 --- a/content/releases/1.2.1/Kestrel-and-Storm.html +++ /dev/null @@ -1,412 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Storm and Kestrel</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 1.2.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.1/index.html">1.2.1</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/06/04/storm122-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Storm and Kestrel</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<div class="documentation-content"><p>This page explains how to use Storm to consume items from a Kestrel cluster.</p> - -<h2 id="preliminaries">Preliminaries</h2> - -<h3 id="storm">Storm</h3> - -<p>This tutorial uses examples from the <a href="https://github.com/nathanmarz/storm-kestrel";>storm-kestrel</a> project and the <a href="http://github.com/apache/storm/blob/1.2.1/examples/storm-starter";>storm-starter</a> project. It's recommended that you clone those projects and follow along with the examples. Read <a href="Setting-up-development-environment.html">Setting up development environment</a> and <a href="Creating-a-new-Storm-project.html">Creating a new Storm project</a> to get your machine set up.</p> - -<h3 id="kestrel">Kestrel</h3> - -<p>It assumes you are able to run locally a Kestrel server as described <a href="https://github.com/nathanmarz/storm-kestrel";>here</a>.</p> - -<h2 id="kestrel-server-and-queue">Kestrel Server and Queue</h2> - -<p>A single kestrel server has a set of queues. A Kestrel queue is a very simple message queue that runs on the JVM and uses the memcache protocol (with some extensions) to talk to clients. For details, look at the implementation of the <a href="https://github.com/nathanmarz/storm-kestrel/blob/master/src/jvm/org/apache/storm/spout/KestrelThriftClient.java";>KestrelThriftClient</a> class provided in <a href="https://github.com/nathanmarz/storm-kestrel";>storm-kestrel</a> project.</p> - -<p>Each queue is strictly ordered following the FIFO (first in, first out) principle. To keep up with performance items are cached in system memory; though, only the first 128MB is kept in memory. When stopping the server, the queue state is stored in a journal file.</p> - -<p>Further, details can be found <a href="https://github.com/nathanmarz/kestrel/blob/master/docs/guide.md";>here</a>.</p> - -<p>Kestrel is: -* fast -* small -* durable -* reliable</p> - -<p>For instance, Twitter uses Kestrel as the backbone of its messaging infrastructure as described <a href="http://bhavin.directi.com/notes-on-kestrel-the-open-source-twitter-queue/";>here</a>.</p> - -<h2 id="add-items-to-kestrel">Add items to Kestrel</h2> - -<p>At first, we need to have a program that can add items to a Kestrel queue. The following method takes benefit of the KestrelClient implementation in <a href="https://github.com/nathanmarz/storm-kestrel";>storm-kestrel</a>. It adds sentences into a Kestrel queue randomly chosen out of an array that holds five possible sentences.</p> -<div class="highlight"><pre><code class="language-" data-lang=""> private static void queueSentenceItems(KestrelClient kestrelClient, String queueName) - throws ParseError, IOException { - - String[] sentences = new String[] { - "the cow jumped over the moon", - "an apple a day keeps the doctor away", - "four score and seven years ago", - "snow white and the seven dwarfs", - "i am at two with nature"}; - - Random _rand = new Random(); - - for(int i=1; i<=10; i++){ - - String sentence = sentences[_rand.nextInt(sentences.length)]; - - String val = "ID " + i + " " + sentence; - - boolean queueSucess = kestrelClient.queue(queueName, val); - - System.out.println("queueSucess=" +queueSucess+ " [" + val +"]"); - } - } -</code></pre></div> -<h2 id="remove-items-from-kestrel">Remove items from Kestrel</h2> - -<p>This method dequeues items from a queue without removing them. -``` - private static void dequeueItems(KestrelClient kestrelClient, String queueName) throws IOException, ParseError - { - for(int i=1; i<=12; i++){</p> -<div class="highlight"><pre><code class="language-" data-lang=""> Item item = kestrelClient.dequeue(queueName); - - if(item==null){ - System.out.println("The queue (" + queueName + ") contains no items."); - } - else - { - byte[] data = item._data; - - String receivedVal = new String(data); - - System.out.println("receivedItem=" + receivedVal); - } - } -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang=""> -This method dequeues items from a queue and then removes them. -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">private static void dequeueAndRemoveItems(KestrelClient kestrelClient, String queueName) -throws IOException, ParseError - { - for(int i=1; i<=12; i++){ - - Item item = kestrelClient.dequeue(queueName); - - - if(item==null){ - System.out.println("The queue (" + queueName + ") contains no items."); - } - else - { - int itemID = item._id; - - - byte[] data = item._data; - - String receivedVal = new String(data); - - kestrelClient.ack(queueName, itemID); - - System.out.println("receivedItem=" + receivedVal); - } - } -} -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang=""> -## Add Items continuously to Kestrel - -This is our final program to run in order to add continuously sentence items to a queue called **sentence_queue** of a locally running Kestrel server. - -In order to stop it type a closing bracket char ']' in console and hit 'Enter'. - -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">import java.io.IOException; -import java.io.InputStream; -import java.util.Random; - -import org.apache.storm.spout.KestrelClient; -import org.apache.storm.spout.KestrelClient.Item; -import org.apache.storm.spout.KestrelClient.ParseError; - -public class AddSentenceItemsToKestrel { - - /** - * @param args - */ - public static void main(String[] args) { - - InputStream is = System.in; - - char closing_bracket = ']'; - - int val = closing_bracket; - - boolean aux = true; - - try { - - KestrelClient kestrelClient = null; - String queueName = "sentence_queue"; - - while(aux){ - - kestrelClient = new KestrelClient("localhost",22133); - - queueSentenceItems(kestrelClient, queueName); - - kestrelClient.close(); - - Thread.sleep(1000); - - if(is.available()>0){ - if(val==is.read()) - aux=false; - } - } - } catch (IOException e) { - // TODO Auto-generated catch block - e.printStackTrace(); - } - catch (ParseError e) { - // TODO Auto-generated catch block - e.printStackTrace(); - } catch (InterruptedException e) { - // TODO Auto-generated catch block - e.printStackTrace(); - } - - System.out.println("end"); - - } -} -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">## Using KestrelSpout - -This topology reads sentences off of a Kestrel queue using KestrelSpout, splits the sentences into its constituent words (Bolt: SplitSentence), and then emits for each word the number of times it has seen that word before (Bolt: WordCount). How data is processed is described in detail in [Guaranteeing message processing](Guaranteeing-message-processing.html). - -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">TopologyBuilder builder = new TopologyBuilder(); -builder.setSpout("sentences", new KestrelSpout("localhost",22133,"sentence_queue",new StringScheme())); -builder.setBolt("split", new SplitSentence(), 10) - .shuffleGrouping("sentences"); -builder.setBolt("count", new WordCount(), 20) - .fieldsGrouping("split", new Fields("word")); -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang=""> -## Execution - -At first, start your local kestrel server in production or development mode. - -Than, wait about 5 seconds in order to avoid a ConnectException. - -Now execute the program to add items to the queue and launch the Storm topology. The order in which you launch the programs is of no importance. - -If you run the topology with TOPOLOGY_DEBUG you should see tuples being emitted in the topology. -</code></pre></div></div> - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> - http://git-wip-us.apache.org/repos/asf/storm-site/blob/6e122a12/content/releases/1.2.1/Lifecycle-of-a-topology.html ---------------------------------------------------------------------- diff --git a/content/releases/1.2.1/Lifecycle-of-a-topology.html b/content/releases/1.2.1/Lifecycle-of-a-topology.html deleted file mode 100644 index e069039..0000000 --- a/content/releases/1.2.1/Lifecycle-of-a-topology.html +++ /dev/null @@ -1,339 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Lifecycle of a Storm Topology</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 1.2.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.1/index.html">1.2.1</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/06/04/storm122-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Lifecycle of a Storm Topology</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<div class="documentation-content"><p>(<strong>NOTE</strong>: this page is based on the 0.7.1 code; many things have changed since then, including a split between tasks and executors, and a reorganization of the code under <code>storm-core/src</code> rather than <code>src/</code>.)</p> - -<p>This page explains in detail the lifecycle of a topology from running the "storm jar" command to uploading the topology to Nimbus to the supervisors starting/stopping workers to workers and tasks setting themselves up. It also explains how Nimbus monitors topologies and how topologies are shutdown when they are killed.</p> - -<p>First a couple of important notes about topologies:</p> - -<ol> -<li>The actual topology that runs is different than the topology the user specifies. The actual topology has implicit streams and an implicit "acker" bolt added to manage the acking framework (used to guarantee data processing). The implicit topology is created via the <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/common.clj#L188";>system-topology!</a> function.</li> -<li><code>system-topology!</code> is used in two places: - -<ul> -<li>when Nimbus is creating tasks for the topology <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L316";>code</a></li> -<li>in the worker so it knows where it needs to route messages to <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/worker.clj#L90";>code</a></li> -</ul></li> -</ol> - -<h2 id="starting-a-topology">Starting a topology</h2> - -<ul> -<li>"storm jar" command executes your class with the specified arguments. The only special thing that "storm jar" does is set the "storm.jar" environment variable for use by <code>StormSubmitter</code> later. <a href="https://github.com/apache/storm/blob/0.7.1/bin/storm#L101";>code</a></li> -<li><p>When your code uses <code>StormSubmitter.submitTopology</code>, <code>StormSubmitter</code> takes the following actions:</p> - -<ul> -<li>First, <code>StormSubmitter</code> uploads the jar if it hasn't been uploaded before. <a href="https://github.com/apache/storm/blob/0.7.1/src/jvm/org/apache/storm/StormSubmitter.java#L83";>code</a></li> -<li>Jar uploading is done via Nimbus's Thrift interface <a href="https://github.com/apache/storm/blob/0.7.1/src/storm.thrift#L200";>code</a></li> -<li><code>beginFileUpload</code> returns a path in Nimbus's inbox</li> -<li>15 kilobytes are uploaded at a time through <code>uploadChunk</code></li> -<li><code>finishFileUpload</code> is called when it's finished uploading</li> -<li>Here is Nimbus's implementation of those Thrift methods: <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L694";>code</a></li> -<li>Second, <code>StormSubmitter</code> calls <code>submitTopology</code> on the Nimbus thrift interface <a href="https://github.com/apache/storm/blob/0.7.1/src/jvm/org/apache/storm/StormSubmitter.java#L60";>code</a></li> -<li>The topology config is serialized using JSON (JSON is used so that writing DSL's in any language is as easy as possible)</li> -<li>Notice that the Thrift <code>submitTopology</code> call takes in the Nimbus inbox path where the jar was uploaded</li> -</ul></li> -<li><p>Nimbus receives the topology submission. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L639";>code</a></p></li> -<li><p>Nimbus normalizes the topology configuration. The main purpose of normalization is to ensure that every single task will have the same serialization registrations, which is critical for getting serialization working correctly. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L557";>code</a></p></li> -<li><p>Nimbus sets up the static state for the topology <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L661";>code</a></p> - -<ul> -<li>Jars and configs are kept on local filesystem because they're too big for Zookeeper. The jar and configs are copied into the path {nimbus local dir}/stormdist/{topology id}</li> -<li><code>setup-storm-static</code> writes task -> component mapping into ZK</li> -<li><code>setup-heartbeats</code> creates a ZK "directory" in which tasks can heartbeat</li> -</ul></li> -<li><p>Nimbus calls <code>mk-assignment</code> to assign tasks to machines <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L458";>code</a></p> - -<ul> -<li>Assignment record definition is here: <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/common.clj#L25";>code</a></li> -<li>Assignment contains: - -<ul> -<li><code>master-code-dir</code>: used by supervisors to download the correct jars/configs for the topology from Nimbus</li> -<li><code>task->node+port</code>: Map from a task id to the worker that task should be running on. (A worker is identified by a node/port pair)</li> -<li><code>node->host</code>: A map from node id to hostname. This is used so workers know which machines to connect to to communicate with other workers. Node ids are used to identify supervisors so that multiple supervisors can be run on one machine. One place this is done is with Mesos integration.</li> -<li><code>task->start-time-secs</code>: Contains a map from task id to the timestamp at which Nimbus launched that task. This is used by Nimbus when monitoring topologies, as tasks are given a longer timeout to heartbeat when they're first launched (the launch timeout is configured by "nimbus.task.launch.secs" config)</li> -</ul></li> -</ul></li> -<li><p>Once topologies are assigned, they're initially in a deactivated mode. <code>start-storm</code> writes data into Zookeeper so that the cluster knows the topology is active and can start emitting tuples from spouts. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L504";>code</a></p></li> -<li><p>TODO cluster state diagram (show all nodes and what's kept everywhere)</p></li> -<li><p>Supervisor runs two functions in the background:</p> - -<ul> -<li><code>synchronize-supervisor</code>: This is called whenever assignments in Zookeeper change and also every 10 seconds. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/supervisor.clj#L241";>code</a> - -<ul> -<li>Downloads code from Nimbus for topologies assigned to this machine for which it doesn't have the code yet. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/supervisor.clj#L258";>code</a></li> -<li>Writes into local filesystem what this node is supposed to be running. It writes a map from port -> LocalAssignment. LocalAssignment contains a topology id as well as the list of task ids for that worker. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/supervisor.clj#L13";>code</a></li> -</ul></li> -<li><code>sync-processes</code>: Reads from the LFS what <code>synchronize-supervisor</code> wrote and compares that to what's actually running on the machine. It then starts/stops worker processes as necessary to synchronize. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/supervisor.clj#L177";>code</a></li> -</ul></li> -<li><p>Worker processes start up through the <code>mk-worker</code> function <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/worker.clj#L67";>code</a></p> - -<ul> -<li>Worker connects to other workers and starts a thread to monitor for changes. So if a worker gets reassigned, the worker will automatically reconnect to the other worker's new location. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/worker.clj#L123";>code</a></li> -<li>Monitors whether a topology is active or not and stores that state in the <code>storm-active-atom</code> variable. This variable is used by tasks to determine whether or not to call <code>nextTuple</code> on the spouts. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/worker.clj#L155";>code</a></li> -<li>The worker launches the actual tasks as threads within it <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/worker.clj#L178";>code</a></li> -</ul></li> -<li><p>Tasks are set up through the <code>mk-task</code> function <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/task.clj#L160";>code</a></p> - -<ul> -<li>Tasks set up routing function which takes in a stream and an output tuple and returns a list of task ids to send the tuple to <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/task.clj#L207";>code</a> (there's also a 3-arity version used for direct streams)</li> -<li>Tasks set up the spout-specific or bolt-specific code with <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/task.clj#L241";>code</a></li> -</ul></li> -</ul> - -<h2 id="topology-monitoring">Topology Monitoring</h2> - -<ul> -<li>Nimbus monitors the topology during its lifetime - -<ul> -<li>Schedules recurring task on the timer thread to check the topologies <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L623";>code</a></li> -<li>Nimbus's behavior is represented as a finite state machine <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L98";>code</a></li> -<li>The "monitor" event is called on a topology every "nimbus.monitor.freq.secs", which calls <code>reassign-topology</code> through <code>reassign-transition</code> <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L497";>code</a></li> -<li><code>reassign-topology</code> calls <code>mk-assignments</code>, the same function used to assign the topology the first time. <code>mk-assignments</code> is also capable of incrementally updating a topology - -<ul> -<li><code>mk-assignments</code> checks heartbeats and reassigns workers as necessary</li> -<li>Any reassignments change the state in ZK, which will trigger supervisors to synchronize and start/stop workers</li> -</ul></li> -</ul></li> -</ul> - -<h2 id="killing-a-topology">Killing a topology</h2> - -<ul> -<li>"storm kill" command runs this code which just calls the Nimbus Thrift interface to kill the topology: <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/command/kill_topology.clj";>code</a></li> -<li>Nimbus receives the kill command <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L671";>code</a></li> -<li>Nimbus applies the "kill" transition to the topology <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L676";>code</a></li> -<li>The kill transition function changes the status of the topology to "killed" and schedules the "remove" event to run "wait time seconds" in the future. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L63";>code</a> - -<ul> -<li>The wait time defaults to the topology message timeout but can be overridden with the -w flag in the "storm kill" command</li> -<li>This causes the topology to be deactivated for the wait time before its actually shut down. This gives the topology a chance to finish processing what it's currently processing before shutting down the workers</li> -<li>Changing the status during the kill transition ensures that the kill protocol is fault-tolerant to Nimbus crashing. On startup, if the status of the topology is "killed", Nimbus schedules the remove event to run "wait time seconds" in the future <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L111";>code</a></li> -</ul></li> -<li>Removing a topology cleans out the assignment and static information from ZK <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L116";>code</a></li> -<li>A separate cleanup thread runs the <code>do-cleanup</code> function which will clean up the heartbeat dir and the jars/configs stored locally. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/org/apache/storm/daemon/nimbus.clj#L577";>code</a></li> -</ul> -</div> - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> - http://git-wip-us.apache.org/repos/asf/storm-site/blob/6e122a12/content/releases/1.2.1/Local-mode.html ---------------------------------------------------------------------- diff --git a/content/releases/1.2.1/Local-mode.html b/content/releases/1.2.1/Local-mode.html deleted file mode 100644 index 8297148..0000000 --- a/content/releases/1.2.1/Local-mode.html +++ /dev/null @@ -1,242 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Local Mode</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 1.2.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.1/index.html">1.2.1</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/06/04/storm122-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Local Mode</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<div class="documentation-content"><p>Local mode simulates a Storm cluster in process and is useful for developing and testing topologies. Running topologies in local mode is similar to running topologies <a href="Running-topologies-on-a-production-cluster.html">on a cluster</a>. </p> - -<p>To create an in-process cluster, simply use the <code>LocalCluster</code> class. For example:</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kn">import</span> <span class="nn">org.apache.storm.LocalCluster</span><span class="o">;</span> - -<span class="n">LocalCluster</span> <span class="n">cluster</span> <span class="o">=</span> <span class="k">new</span> <span class="n">LocalCluster</span><span class="o">();</span> -</code></pre></div> -<p>You can then submit topologies using the <code>submitTopology</code> method on the <code>LocalCluster</code> object. Just like the corresponding method on <a href="javadocs/org/apache/storm/StormSubmitter.html">StormSubmitter</a>, <code>submitTopology</code> takes a name, a topology configuration, and the topology object. You can then kill a topology using the <code>killTopology</code> method which takes the topology name as an argument.</p> - -<p>To shutdown a local cluster, simple call:</p> -<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">cluster</span><span class="o">.</span><span class="na">shutdown</span><span class="o">();</span> -</code></pre></div> -<h3 id="common-configurations-for-local-mode">Common configurations for local mode</h3> - -<p>You can see a full list of configurations <a href="javadocs/org/apache/storm/Config.html">here</a>.</p> - -<ol> -<li><strong>Config.TOPOLOGY_MAX_TASK_PARALLELISM</strong>: This config puts a ceiling on the number of threads spawned for a single component. Oftentimes production topologies have a lot of parallelism (hundreds of threads) which places unreasonable load when trying to test the topology in local mode. This config lets you easy control that parallelism.</li> -<li><strong>Config.TOPOLOGY_DEBUG</strong>: When this is set to true, Storm will log a message every time a tuple is emitted from any spout or bolt. This is extremely useful for debugging.</li> -</ol> -</div> - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> -
