This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/storm-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a1438f2ee3 publishing 2026-07-25T17:05:10+00:00
a1438f2ee3 is described below
commit a1438f2ee364db25e93cc13d528c747c5f37687a
Author: GitHub Actions Bot <>
AuthorDate: Sat Jul 25 17:05:10 2026 +0000
publishing 2026-07-25T17:05:10+00:00
---
feed.xml | 4 +-
...rm-Scheduler.html => Binary-distributions.html} | 78 +++++++++++++++++-----
...heduler.html => JitterAwareStreamGrouping.html} | 63 ++++++++++++-----
...Storm-Scheduler.html => Local-dev-cluster.html} | 65 +++++++++++++-----
releases/3.0.0/Storm-Scheduler.html | 50 ++++++++++++++
releases/3.0.0/index.html | 4 ++
.../Binary-distributions.html} | 78 +++++++++++++++++-----
.../JitterAwareStreamGrouping.html} | 63 ++++++++++++-----
.../Local-dev-cluster.html} | 65 +++++++++++++-----
releases/current/Storm-Scheduler.html | 50 ++++++++++++++
releases/current/index.html | 4 ++
11 files changed, 414 insertions(+), 110 deletions(-)
diff --git a/feed.xml b/feed.xml
index f9a9ec8be1..4eafb80452 100644
--- a/feed.xml
+++ b/feed.xml
@@ -5,8 +5,8 @@
<description>Apache Storm is a free and open source distributed realtime
computation system. Storm makes it easy to reliably process unbounded streams
of data, doing for realtime processing what Hadoop did for batch
processing.</description>
<link>https://storm.apache.org/</link>
<atom:link href="https://storm.apache.org/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Sat, 25 Jul 2026 14:57:50 +0000</pubDate>
- <lastBuildDate>Sat, 25 Jul 2026 14:57:50 +0000</lastBuildDate>
+ <pubDate>Sat, 25 Jul 2026 17:04:52 +0000</pubDate>
+ <lastBuildDate>Sat, 25 Jul 2026 17:04:52 +0000</lastBuildDate>
<generator>Jekyll v3.10.0</generator>
<item>
diff --git a/releases/3.0.0/Storm-Scheduler.html
b/releases/3.0.0/Binary-distributions.html
similarity index 69%
copy from releases/3.0.0/Storm-Scheduler.html
copy to releases/3.0.0/Binary-distributions.html
index 6c38c13975..2f20e67ea1 100644
--- a/releases/3.0.0/Storm-Scheduler.html
+++ b/releases/3.0.0/Binary-distributions.html
@@ -13,19 +13,19 @@
- <title>Scheduler</title>
+ <title>Binary Distributions</title>
<meta name="description" content="Apache Storm is a free and open source
distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
- <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Binary-distributions.html";>
- <meta property="og:title" content="Scheduler">
+ <meta property="og:title" content="Binary Distributions">
<meta property="og:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta property="og:type" content="website">
- <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Binary-distributions.html";>
<meta property="og:image"
content="https://storm.apache.org/images/logo.png";>
<meta property="og:site_name" content="Apache Storm">
<meta name="twitter:card" content="summary">
- <meta name="twitter:title" content="Scheduler">
+ <meta name="twitter:title" content="Binary Distributions">
<meta name="twitter:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta name="twitter:image"
content="https://storm.apache.org/images/logo.png";>
@@ -179,7 +179,7 @@
<div class="container-fluid">
- <h1 class="page-title">Scheduler</h1>
+ <h1 class="page-title">Binary Distributions</h1>
<div class="row">
<div class="col-md-12">
@@ -187,25 +187,67 @@
<p class="post-meta"></p>
-<div class="documentation-content"><p>Storm now has 4 kinds of built-in
schedulers: <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/DefaultScheduler.java";>DefaultScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/IsolationScheduler.java";>IsolationScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/sch
[...]
+<div class="documentation-content"><p>Storm 3.0.0 ships two binary
distributions. Both contain identical core functionality; they differ only in
which optional plugins are bundled.</p>
-<h2 id="pluggable-scheduler">Pluggable scheduler</h2>
+<table><thead>
+<tr>
+<th>Distribution</th>
+<th>Filename</th>
+<th>Approx. size</th>
+</tr>
+</thead><tbody>
+<tr>
+<td><strong>Full</strong></td>
+<td><code>apache-storm-<version>.tar.gz</code></td>
+<td>~395 MB of jars</td>
+</tr>
+<tr>
+<td><strong>Lite</strong></td>
+<td><code>apache-storm-<version>-lite.tar.gz</code></td>
+<td>~207 MB of jars (~47% smaller)</td>
+</tr>
+</tbody></table>
-<p>You can implement your own scheduler to replace the default scheduler to
assign executors to workers. You configure the class to use the
"storm.scheduler" config in your storm.yaml, and your scheduler must
implement the <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-client/src/jvm/org/apache/storm/scheduler/IScheduler.java";>IScheduler</a>
interface.</p>
+<h2 id="what-the-lite-distribution-omits">What the lite distribution omits</h2>
-<h2 id="isolation-scheduler">Isolation Scheduler</h2>
+<h3 id="storm-autocreds-79-mb">storm-autocreds (−79 MB)</h3>
-<p>The isolation scheduler makes it easy and safe to share a cluster among
many topologies. The isolation scheduler lets you specify which topologies
should be "isolated", meaning that they run on a dedicated set of
machines within the cluster where no other topologies will be running. These
isolated topologies are given priority on the cluster, so resources will be
allocated to isolated topologies if there's competition with non-isolated
topologies, and resources will be t [...]
+<p><code>storm-autocreds</code> provides Nimbus and Supervisor with the
ability to populate and renew HDFS and HBase delegation tokens on
<strong>Kerberos-secured clusters</strong>. It pulls in the full Hadoop and
HBase client trees. The plugin is off by default and is only needed on secure
Hadoop deployments.</p>
-<p>You can configure the isolation scheduler in the Nimbus configuration by
setting "storm.scheduler" to
"org.apache.storm.scheduler.IsolationScheduler". Then, use the
"isolation.scheduler.machines" config to specify how many machines
each topology should get. This configuration is a map from topology name to the
number of isolated machines allocated to this topology. For example:</p>
-<div class="highlight"><pre><code class="language-"
data-lang="">isolation.scheduler.machines:
- "my-topology": 8
- "tiny-topology": 1
- "some-other-topology": 3
+<p>In the lite distribution, only the
<code>external/storm-autocreds/README</code> ships. To install the plugin on
demand, run:</p>
+<div class="highlight"><pre><code class="language-"
data-lang="">bin/storm-autocreds-fetch
</code></pre></div>
-<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
+<p>This uses Maven to resolve <code>org.apache.storm:storm-autocreds</code>
and its runtime dependencies from Maven Central and copies them into
<code>extlib-daemon/</code>. Restart Nimbus and Supervisor afterwards.</p>
-<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+<p>Pass <code>--version</code> and <code>--dest</code> to override the
detected Storm version or target directory. Pass <code>--</code> to forward
arguments to Maven (e.g. to use an internal mirror or an offline local
repository):</p>
+<div class="highlight"><pre><code class="language-"
data-lang="">bin/storm-autocreds-fetch -- -s /path/to/settings.xml
+bin/storm-autocreds-fetch -- -Dmaven.repo.local=/path/to/offline-repo -o
+</code></pre></div>
+<h3 id="storm-kafka-monitor-38-mb">storm-kafka-monitor (−38 MB)</h3>
+
+<p><code>storm-kafka-monitor</code> displays Kafka spout consumer lag in the
Storm UI and powers the <code>bin/storm-kafka-monitor</code> command. It pulls
in the Kafka client library and is only needed when running Kafka spouts and
wanting lag metrics. When it is absent, the UI degrades gracefully: the lag
column shows an actionable message rather than failing, and the
<code>bin/storm-kafka-monitor</code> wrapper prints a hint to run the fetch
command.</p>
+
+<p>In the lite distribution, only the
<code>external/storm-kafka-monitor/README</code> ships. To install it on
demand, run:</p>
+<div class="highlight"><pre><code class="language-"
data-lang="">bin/storm-kafka-monitor-fetch
+</code></pre></div>
+<p>This resolves <code>org.apache.storm:storm-kafka-monitor</code> and its
runtime dependencies into <code>lib-tools/storm-kafka-monitor/</code>. No
daemon restart is required; the UI picks it up on the next request.</p>
+
+<p>The same <code>--version</code>, <code>--dest</code>, and <code>--</code>
passthrough options are available as for <code>storm-autocreds-fetch</code>.</p>
+
+<h2 id="lib-common-shared-jar-de-duplication">lib-common: shared jar
de-duplication</h2>
+
+<p>Both distributions include a structural change that is transparent to
users: jars that were previously duplicated across the daemon classpath
(<code>lib/</code>) and the worker classpath (<code>lib-worker/</code>) are now
kept in a single <code>lib-common/</code> directory. <code>bin/storm.py</code>
adds <code>lib-common</code> to both classpaths, so the runtime behaviour is
unchanged. Only byte-identical jars (same filename and SHA-256) were
de-duplicated; no version was silently mer [...]
+
+<h2 id="which-distribution-should-i-use">Which distribution should I use?</h2>
+
+<p>Use the <strong>lite distribution</strong> unless you specifically need one
of the unbundled plugins:</p>
+
+<ul>
+<li>If you run topologies on a <strong>Kerberos-secured Hadoop or HBase
cluster</strong>, run <code>bin/storm-autocreds-fetch</code> after installing
from the lite distribution, or use the full distribution.</li>
+<li>If you want <strong>Kafka spout lag</strong> in the Storm UI or the
<code>storm-kafka-monitor</code> command, run
<code>bin/storm-kafka-monitor-fetch</code> after installing from the lite
distribution, or use the full distribution.</li>
+</ul>
+
+<p>For all other use cases — including running Kafka spout topologies without
the lag UI feature — the lite distribution is sufficient and carries no
functional difference from the full distribution.</p>
</div>
diff --git a/releases/3.0.0/Storm-Scheduler.html
b/releases/3.0.0/JitterAwareStreamGrouping.html
similarity index 69%
copy from releases/3.0.0/Storm-Scheduler.html
copy to releases/3.0.0/JitterAwareStreamGrouping.html
index 6c38c13975..ae57076d92 100644
--- a/releases/3.0.0/Storm-Scheduler.html
+++ b/releases/3.0.0/JitterAwareStreamGrouping.html
@@ -13,19 +13,19 @@
- <title>Scheduler</title>
+ <title>JitterAwareStreamGrouping</title>
<meta name="description" content="Apache Storm is a free and open source
distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
- <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/JitterAwareStreamGrouping.html";>
- <meta property="og:title" content="Scheduler">
+ <meta property="og:title" content="JitterAwareStreamGrouping">
<meta property="og:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta property="og:type" content="website">
- <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/JitterAwareStreamGrouping.html";>
<meta property="og:image"
content="https://storm.apache.org/images/logo.png";>
<meta property="og:site_name" content="Apache Storm">
<meta name="twitter:card" content="summary">
- <meta name="twitter:title" content="Scheduler">
+ <meta name="twitter:title" content="JitterAwareStreamGrouping">
<meta name="twitter:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta name="twitter:image"
content="https://storm.apache.org/images/logo.png";>
@@ -179,7 +179,7 @@
<div class="container-fluid">
- <h1 class="page-title">Scheduler</h1>
+ <h1 class="page-title">JitterAwareStreamGrouping</h1>
<div class="row">
<div class="col-md-12">
@@ -187,25 +187,52 @@
<p class="post-meta"></p>
-<div class="documentation-content"><p>Storm now has 4 kinds of built-in
schedulers: <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/DefaultScheduler.java";>DefaultScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/IsolationScheduler.java";>IsolationScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/sch
[...]
+<div class="documentation-content"><p><code>JitterAwareStreamGrouping</code>
is a stream grouping that selects downstream tasks based on observed per-task
execution jitter. It steers traffic away from tasks experiencing backpressure
or execution variance, producing a more even distribution of work over time.</p>
-<h2 id="pluggable-scheduler">Pluggable scheduler</h2>
+<h2 id="background-the-jitter-metric">Background: the jitter metric</h2>
-<p>You can implement your own scheduler to replace the default scheduler to
assign executors to workers. You configure the class to use the
"storm.scheduler" config in your storm.yaml, and your scheduler must
implement the <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-client/src/jvm/org/apache/storm/scheduler/IScheduler.java";>IScheduler</a>
interface.</p>
+<p>Storm 3.0.0 introduces a per-task jitter metric based on the EWMA algorithm
in <a href="https://www.rfc-editor.org/rfc/rfc1889#appendix-A";>RFC 1889
Appendix A</a>. The metric tracks inter-arrival time variance in each
task's processing loop. A lightweight control loop propagates these
measurements to upstream components at regular intervals without touching the
data path.</p>
-<h2 id="isolation-scheduler">Isolation Scheduler</h2>
+<p>A task with a steady processing rate accumulates near-zero jitter. A task
that stalls, GC-pauses, or falls behind its input queue accumulates a rising
jitter estimate, which the grouping uses as a signal to route tuples
elsewhere.</p>
-<p>The isolation scheduler makes it easy and safe to share a cluster among
many topologies. The isolation scheduler lets you specify which topologies
should be "isolated", meaning that they run on a dedicated set of
machines within the cluster where no other topologies will be running. These
isolated topologies are given priority on the cluster, so resources will be
allocated to isolated topologies if there's competition with non-isolated
topologies, and resources will be t [...]
+<h2 id="usage">Usage</h2>
-<p>You can configure the isolation scheduler in the Nimbus configuration by
setting "storm.scheduler" to
"org.apache.storm.scheduler.IsolationScheduler". Then, use the
"isolation.scheduler.machines" config to specify how many machines
each topology should get. This configuration is a map from topology name to the
number of isolated machines allocated to this topology. For example:</p>
-<div class="highlight"><pre><code class="language-"
data-lang="">isolation.scheduler.machines:
- "my-topology": 8
- "tiny-topology": 1
- "some-other-topology": 3
+<p>Use <code>customGrouping</code> with a
<code>JitterAwareStreamGrouping</code> instance in place of
<code>shuffleGrouping</code>:</p>
+<div class="highlight"><pre><code class="language-java" data-lang="java"><span
class="kn">import</span> <span
class="nn">org.apache.storm.grouping.JitterAwareStreamGrouping</span><span
class="o">;</span>
+
+<span class="nc">TopologyBuilder</span> <span class="n">builder</span> <span
class="o">=</span> <span class="k">new</span> <span
class="nc">TopologyBuilder</span><span class="o">();</span>
+<span class="n">builder</span><span class="o">.</span><span
class="na">setSpout</span><span class="o">(</span><span
class="s">"spout"</span><span class="o">,</span> <span class="k">new</span>
<span class="nc">MySpout</span><span class="o">(),</span> <span
class="mi">1</span><span class="o">);</span>
+<span class="n">builder</span><span class="o">.</span><span
class="na">setBolt</span><span class="o">(</span><span
class="s">"bolt"</span><span class="o">,</span> <span class="k">new</span>
<span class="nc">MyBolt</span><span class="o">(),</span> <span
class="mi">4</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">customGrouping</span><span
class="o">(</span><span class="s">"spout"</span><span class="o">,</span> <span
class="k">new</span> <span class="nc">JitterAwareStreamGrouping</span><span
class="o">());</span>
+</code></pre></div>
+<p><code>JitterAwareStreamGrouping</code> is serializable and safe across
workers. Before any jitter data arrives (e.g., at topology startup) it falls
back to random task selection.</p>
+
+<h2 id="benchmarking">Benchmarking</h2>
+
+<p>The <code>storm-perf</code> module includes a dedicated benchmark
topology:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">storm jar
storm-perf/target/storm-perf-*.jar \
+ org.apache.storm.perf.JitterAwareGroupingTopology
</code></pre></div>
-<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
+<p>Run it alongside the equivalent shuffle-grouping topology to measure the
effect in your cluster. The <a href="Local-dev-cluster.html">local development
cluster</a> provides Grafana dashboards with per-task jitter visibility.</p>
+
+<h2 id="when-to-use-it">When to use it</h2>
+
+<ul>
+<li>Bolts with variable execution time (external I/O, cache misses,
non-uniform key distributions) where a subset of tasks may consistently
lag.</li>
+<li>Topologies where a single hot task creates cascading backpressure
upstream.</li>
+<li>As a drop-in replacement for <code>shuffleGrouping</code> on any stream
where tasks are interchangeable and execution uniformity matters.</li>
+</ul>
+
+<h2 id="when-not-to-use-it">When not to use it</h2>
+
+<ul>
+<li><strong>Data-locality requirements</strong>: use
<code>fieldsGrouping</code> where tuples must reach a specific task.</li>
+<li><strong>Low parallelism</strong> (2 tasks): the grouping has fewer
candidates to choose between and the benefit diminishes.</li>
+<li><strong>Order-sensitive streams</strong>: like any non-deterministic
grouping, this does not preserve tuple order across tasks.</li>
+<li><strong>Intra-worker traffic</strong>: for streams where both components
run in the same worker, <code>localOrShuffleGrouping</code> avoids
serialization entirely and is preferable.</li>
+</ul>
-<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+<p>The jitter control loop runs on all topologies regardless of whether
<code>JitterAwareStreamGrouping</code> is in use. The grouping simply reads
measurements the loop already produces; enabling it carries no additional
overhead on the data path.</p>
</div>
diff --git a/releases/3.0.0/Storm-Scheduler.html
b/releases/3.0.0/Local-dev-cluster.html
similarity index 76%
copy from releases/3.0.0/Storm-Scheduler.html
copy to releases/3.0.0/Local-dev-cluster.html
index 6c38c13975..3284623e43 100644
--- a/releases/3.0.0/Storm-Scheduler.html
+++ b/releases/3.0.0/Local-dev-cluster.html
@@ -13,19 +13,19 @@
- <title>Scheduler</title>
+ <title>Local Development Cluster</title>
<meta name="description" content="Apache Storm is a free and open source
distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
- <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Local-dev-cluster.html";>
- <meta property="og:title" content="Scheduler">
+ <meta property="og:title" content="Local Development Cluster">
<meta property="og:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta property="og:type" content="website">
- <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Local-dev-cluster.html";>
<meta property="og:image"
content="https://storm.apache.org/images/logo.png";>
<meta property="og:site_name" content="Apache Storm">
<meta name="twitter:card" content="summary">
- <meta name="twitter:title" content="Scheduler">
+ <meta name="twitter:title" content="Local Development Cluster">
<meta name="twitter:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta name="twitter:image"
content="https://storm.apache.org/images/logo.png";>
@@ -179,7 +179,7 @@
<div class="container-fluid">
- <h1 class="page-title">Scheduler</h1>
+ <h1 class="page-title">Local Development Cluster</h1>
<div class="row">
<div class="col-md-12">
@@ -187,25 +187,54 @@
<p class="post-meta"></p>
-<div class="documentation-content"><p>Storm now has 4 kinds of built-in
schedulers: <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/DefaultScheduler.java";>DefaultScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/IsolationScheduler.java";>IsolationScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/sch
[...]
+<div class="documentation-content"><p>Storm ships a Docker Compose
configuration under <code>docker/</code> that provisions a fully distributed
cluster on a single machine. Unlike <a href="Local-mode.html">local mode</a>,
this setup forces inter-worker traffic across real sockets, making it suitable
for benchmarking serialization, compression, and network-sensitive behaviours
that only appear with genuine inter-process communication.</p>
-<h2 id="pluggable-scheduler">Pluggable scheduler</h2>
+<h2 id="whats-included">What's included</h2>
-<p>You can implement your own scheduler to replace the default scheduler to
assign executors to workers. You configure the class to use the
"storm.scheduler" config in your storm.yaml, and your scheduler must
implement the <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-client/src/jvm/org/apache/storm/scheduler/IScheduler.java";>IScheduler</a>
interface.</p>
+<ul>
+<li><strong>Nimbus</strong>, <strong>ZooKeeper</strong>, and two
<strong>Supervisors</strong> on an isolated Docker network</li>
+<li><strong>Prometheus</strong> scraping Storm's Metrics V2 endpoint</li>
+<li><strong>Grafana</strong> with a pre-built per-task dashboard covering
throughput, latency, and jitter</li>
+<li><strong><code>netsim.sh</code></strong> — a utility that injects
controlled latency and jitter between Supervisors using <code>tc
netem</code></li>
+</ul>
-<h2 id="isolation-scheduler">Isolation Scheduler</h2>
+<h2 id="starting-the-cluster">Starting the cluster</h2>
+<div class="highlight"><pre><code class="language-" data-lang="">cd docker/
+docker compose up -d
+</code></pre></div>
+<p>Once healthy, Grafana is available at <code>http://localhost:3000</code>
(default credentials: <code>admin</code> / <code>admin</code>). The Storm UI is
at <code>http://localhost:8080</code>.</p>
+
+<h2 id="submitting-a-topology">Submitting a topology</h2>
+
+<p>The <code>storm-perf</code> module contains topologies designed for this
environment:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">storm jar
storm-perf/target/storm-perf-*.jar \
+ org.apache.storm.perf.FileReadWordCountTopo \
+ -c nimbus.seeds='["nimbus"]' \
+ -c storm.zookeeper.servers='["zookeeper"]'
+</code></pre></div>
+<p>Any topology JAR works. Point <code>nimbus.seeds</code> and
<code>storm.zookeeper.servers</code> at the Docker hostnames as shown above.</p>
-<p>The isolation scheduler makes it easy and safe to share a cluster among
many topologies. The isolation scheduler lets you specify which topologies
should be "isolated", meaning that they run on a dedicated set of
machines within the cluster where no other topologies will be running. These
isolated topologies are given priority on the cluster, so resources will be
allocated to isolated topologies if there's competition with non-isolated
topologies, and resources will be t [...]
+<h2 id="network-simulation">Network simulation</h2>
-<p>You can configure the isolation scheduler in the Nimbus configuration by
setting "storm.scheduler" to
"org.apache.storm.scheduler.IsolationScheduler". Then, use the
"isolation.scheduler.machines" config to specify how many machines
each topology should get. This configuration is a map from topology name to the
number of isolated machines allocated to this topology. For example:</p>
-<div class="highlight"><pre><code class="language-"
data-lang="">isolation.scheduler.machines:
- "my-topology": 8
- "tiny-topology": 1
- "some-other-topology": 3
+<p><code>netsim.sh</code> wraps <code>tc netem</code> to shape traffic between
the two Supervisor containers:</p>
+<div class="highlight"><pre><code class="language-" data-lang=""># Add 3 ms
latency and 1 ms jitter (typical intra-datacenter profile)
+./netsim.sh apply 3ms 1ms
+
+# Remove all shaping
+./netsim.sh reset
+
+# Show current tc settings on both supervisors
+./netsim.sh status
+
+# Round-trip ping between supervisors
+./netsim.sh ping
</code></pre></div>
-<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
+<p>Changes take effect immediately and are visible in the Grafana latency
panels in real time.</p>
-<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+<h2 id="teardown">Teardown</h2>
+<div class="highlight"><pre><code class="language-" data-lang="">docker
compose down
+</code></pre></div>
+<p>Persistent volumes are not configured; all topology data and metrics are
lost on teardown. This cluster is intended for development and benchmarking,
not production use.</p>
</div>
diff --git a/releases/3.0.0/Storm-Scheduler.html
b/releases/3.0.0/Storm-Scheduler.html
index 6c38c13975..ec31681d45 100644
--- a/releases/3.0.0/Storm-Scheduler.html
+++ b/releases/3.0.0/Storm-Scheduler.html
@@ -206,6 +206,56 @@
<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+
+<p>When choosing which hosts to assign an isolated topology to, the scheduler
sorts eligible hosts by three criteria in order:</p>
+
+<ol>
+<li><strong>Assignable slots (descending)</strong> — prefer hosts with greater
total slot capacity.</li>
+<li><strong>Free slots (descending)</strong> — among hosts of equal capacity,
prefer the one with more currently free slots, minimising the number of worker
evictions needed.</li>
+<li><strong>Hostname (ascending)</strong> — alphabetical tiebreaker for
deterministic assignments.</li>
+</ol>
+
+<p>The secondary and tertiary sorts were added in Storm 3.0.0. Previously only
the primary sort applied, which could cause unnecessary evictions when two
hosts had the same total capacity but different numbers of occupied slots.</p>
+
+<h2 id="evenscheduler">EvenScheduler</h2>
+
+<p><code>EvenScheduler</code> (used directly or via
<code>DefaultScheduler</code>) distributes workers as evenly as possible across
available supervisors. It is the default when no custom scheduler is configured
and <code>IsolationScheduler</code> is not in use.</p>
+
+<h3 id="idle-supervisor-rebalance">Idle supervisor rebalance</h3>
+
+<p><em>Available since Storm 3.0.0. Disabled by default.</em></p>
+
+<p>When a supervisor returns from maintenance, its slots are normally left
idle: each topology's desired worker count is already satisfied on the
surviving supervisors, so <code>needsScheduling</code> reports nothing to do.
Workers only migrate back if an operator manually rebalances or restarts every
affected topology.</p>
+
+<p>Setting <code>nimbus.even.rebalance.idle.supervisor.enabled: true</code>
adds an opt-in pass that relocates workers onto the returned supervisor
automatically in a single scheduling round, distributed across topologies in
round-robin order.</p>
+
+<table><thead>
+<tr>
+<th>Key</th>
+<th>Default</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td><code>nimbus.even.rebalance.idle.supervisor.enabled</code></td>
+<td><code>false</code></td>
+<td>Master switch. When false the pass never runs.</td>
+</tr>
+<tr>
+<td><code>nimbus.even.rebalance.max.free.per.topology</code></td>
+<td><code>0</code></td>
+<td>Per-topology cap on workers relocated per round. <code>0</code> means no
additional cap beyond the even-distribution budget.</td>
+</tr>
+<tr>
+<td><code>nimbus.even.rebalance.idle.supervisor.min.stable.rounds</code></td>
+<td><code>3</code></td>
+<td>Number of consecutive monitoring rounds a supervisor must be seen as
stable before it is eligible as a relocation target. Set to <code>0</code> to
disable the flap guard.</td>
+</tr>
+</tbody></table>
+
+<p>A relocation is a worker JVM restart, involving brief tuple replay, JIT
re-warmup, and possible windowed or stateful bolt state restore. Keep this off
for topologies with stateful or windowed bolts that have a non-trivial restore
cost, latency-sensitive topologies sensitive to JIT re-warmup, and clusters
where supervisors are prone to flapping.</p>
+
+<p><strong>Scope:</strong> <code>EvenScheduler</code> and
<code>DefaultScheduler</code> only. <code>ResourceAwareScheduler</code> is
unaffected.</p>
</div>
diff --git a/releases/3.0.0/index.html b/releases/3.0.0/index.html
index 87c7ac10dd..83c45d5d85 100644
--- a/releases/3.0.0/index.html
+++ b/releases/3.0.0/index.html
@@ -250,6 +250,9 @@ We're also notifying it via annotating classes with
marker interface <code>@
<li><a href="ClusterMetrics.html">Daemon Metrics/Monitoring</a></li>
<li><a href="windows-users-guide.html">Windows users guide</a></li>
<li><a href="Classpath-handling.html">Classpath handling</a></li>
+<li><a href="Binary-distributions.html">Binary Distributions (full vs.
lite)</a></li>
+<li><a href="Cluster-State-Serialization.html">Cluster State
Serialization</a></li>
+<li><a href="Local-dev-cluster.html">Local Development Cluster</a></li>
</ul>
<h3 id="intermediate">Intermediate</h3>
@@ -266,6 +269,7 @@ We're also notifying it via annotating classes with
marker interface <code>@
<li><a href="State-checkpointing.html">State Checkpointing</a></li>
<li><a href="Windowing.html">Windowing</a></li>
<li><a href="Joins.html">Joining Streams</a></li>
+<li><a href="JitterAwareStreamGrouping.html">JitterAwareStreamGrouping</a></li>
<li><a href="distcache-blobstore.html">Blobstore(Distcache)</a></li>
</ul>
diff --git a/releases/3.0.0/Storm-Scheduler.html
b/releases/current/Binary-distributions.html
similarity index 69%
copy from releases/3.0.0/Storm-Scheduler.html
copy to releases/current/Binary-distributions.html
index 6c38c13975..f77f69a40b 100644
--- a/releases/3.0.0/Storm-Scheduler.html
+++ b/releases/current/Binary-distributions.html
@@ -13,19 +13,19 @@
- <title>Scheduler</title>
+ <title>Binary Distributions</title>
<meta name="description" content="Apache Storm is a free and open source
distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
- <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <link rel="canonical"
href="https://storm.apache.org/releases/current/Binary-distributions.html";>
- <meta property="og:title" content="Scheduler">
+ <meta property="og:title" content="Binary Distributions">
<meta property="og:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta property="og:type" content="website">
- <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <meta property="og:url"
content="https://storm.apache.org/releases/current/Binary-distributions.html";>
<meta property="og:image"
content="https://storm.apache.org/images/logo.png";>
<meta property="og:site_name" content="Apache Storm">
<meta name="twitter:card" content="summary">
- <meta name="twitter:title" content="Scheduler">
+ <meta name="twitter:title" content="Binary Distributions">
<meta name="twitter:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta name="twitter:image"
content="https://storm.apache.org/images/logo.png";>
@@ -179,7 +179,7 @@
<div class="container-fluid">
- <h1 class="page-title">Scheduler</h1>
+ <h1 class="page-title">Binary Distributions</h1>
<div class="row">
<div class="col-md-12">
@@ -187,25 +187,67 @@
<p class="post-meta"></p>
-<div class="documentation-content"><p>Storm now has 4 kinds of built-in
schedulers: <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/DefaultScheduler.java";>DefaultScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/IsolationScheduler.java";>IsolationScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/sch
[...]
+<div class="documentation-content"><p>Storm 3.0.0 ships two binary
distributions. Both contain identical core functionality; they differ only in
which optional plugins are bundled.</p>
-<h2 id="pluggable-scheduler">Pluggable scheduler</h2>
+<table><thead>
+<tr>
+<th>Distribution</th>
+<th>Filename</th>
+<th>Approx. size</th>
+</tr>
+</thead><tbody>
+<tr>
+<td><strong>Full</strong></td>
+<td><code>apache-storm-<version>.tar.gz</code></td>
+<td>~395 MB of jars</td>
+</tr>
+<tr>
+<td><strong>Lite</strong></td>
+<td><code>apache-storm-<version>-lite.tar.gz</code></td>
+<td>~207 MB of jars (~47% smaller)</td>
+</tr>
+</tbody></table>
-<p>You can implement your own scheduler to replace the default scheduler to
assign executors to workers. You configure the class to use the
"storm.scheduler" config in your storm.yaml, and your scheduler must
implement the <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-client/src/jvm/org/apache/storm/scheduler/IScheduler.java";>IScheduler</a>
interface.</p>
+<h2 id="what-the-lite-distribution-omits">What the lite distribution omits</h2>
-<h2 id="isolation-scheduler">Isolation Scheduler</h2>
+<h3 id="storm-autocreds-79-mb">storm-autocreds (−79 MB)</h3>
-<p>The isolation scheduler makes it easy and safe to share a cluster among
many topologies. The isolation scheduler lets you specify which topologies
should be "isolated", meaning that they run on a dedicated set of
machines within the cluster where no other topologies will be running. These
isolated topologies are given priority on the cluster, so resources will be
allocated to isolated topologies if there's competition with non-isolated
topologies, and resources will be t [...]
+<p><code>storm-autocreds</code> provides Nimbus and Supervisor with the
ability to populate and renew HDFS and HBase delegation tokens on
<strong>Kerberos-secured clusters</strong>. It pulls in the full Hadoop and
HBase client trees. The plugin is off by default and is only needed on secure
Hadoop deployments.</p>
-<p>You can configure the isolation scheduler in the Nimbus configuration by
setting "storm.scheduler" to
"org.apache.storm.scheduler.IsolationScheduler". Then, use the
"isolation.scheduler.machines" config to specify how many machines
each topology should get. This configuration is a map from topology name to the
number of isolated machines allocated to this topology. For example:</p>
-<div class="highlight"><pre><code class="language-"
data-lang="">isolation.scheduler.machines:
- "my-topology": 8
- "tiny-topology": 1
- "some-other-topology": 3
+<p>In the lite distribution, only the
<code>external/storm-autocreds/README</code> ships. To install the plugin on
demand, run:</p>
+<div class="highlight"><pre><code class="language-"
data-lang="">bin/storm-autocreds-fetch
</code></pre></div>
-<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
+<p>This uses Maven to resolve <code>org.apache.storm:storm-autocreds</code>
and its runtime dependencies from Maven Central and copies them into
<code>extlib-daemon/</code>. Restart Nimbus and Supervisor afterwards.</p>
-<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+<p>Pass <code>--version</code> and <code>--dest</code> to override the
detected Storm version or target directory. Pass <code>--</code> to forward
arguments to Maven (e.g. to use an internal mirror or an offline local
repository):</p>
+<div class="highlight"><pre><code class="language-"
data-lang="">bin/storm-autocreds-fetch -- -s /path/to/settings.xml
+bin/storm-autocreds-fetch -- -Dmaven.repo.local=/path/to/offline-repo -o
+</code></pre></div>
+<h3 id="storm-kafka-monitor-38-mb">storm-kafka-monitor (−38 MB)</h3>
+
+<p><code>storm-kafka-monitor</code> displays Kafka spout consumer lag in the
Storm UI and powers the <code>bin/storm-kafka-monitor</code> command. It pulls
in the Kafka client library and is only needed when running Kafka spouts and
wanting lag metrics. When it is absent, the UI degrades gracefully: the lag
column shows an actionable message rather than failing, and the
<code>bin/storm-kafka-monitor</code> wrapper prints a hint to run the fetch
command.</p>
+
+<p>In the lite distribution, only the
<code>external/storm-kafka-monitor/README</code> ships. To install it on
demand, run:</p>
+<div class="highlight"><pre><code class="language-"
data-lang="">bin/storm-kafka-monitor-fetch
+</code></pre></div>
+<p>This resolves <code>org.apache.storm:storm-kafka-monitor</code> and its
runtime dependencies into <code>lib-tools/storm-kafka-monitor/</code>. No
daemon restart is required; the UI picks it up on the next request.</p>
+
+<p>The same <code>--version</code>, <code>--dest</code>, and <code>--</code>
passthrough options are available as for <code>storm-autocreds-fetch</code>.</p>
+
+<h2 id="lib-common-shared-jar-de-duplication">lib-common: shared jar
de-duplication</h2>
+
+<p>Both distributions include a structural change that is transparent to
users: jars that were previously duplicated across the daemon classpath
(<code>lib/</code>) and the worker classpath (<code>lib-worker/</code>) are now
kept in a single <code>lib-common/</code> directory. <code>bin/storm.py</code>
adds <code>lib-common</code> to both classpaths, so the runtime behaviour is
unchanged. Only byte-identical jars (same filename and SHA-256) were
de-duplicated; no version was silently mer [...]
+
+<h2 id="which-distribution-should-i-use">Which distribution should I use?</h2>
+
+<p>Use the <strong>lite distribution</strong> unless you specifically need one
of the unbundled plugins:</p>
+
+<ul>
+<li>If you run topologies on a <strong>Kerberos-secured Hadoop or HBase
cluster</strong>, run <code>bin/storm-autocreds-fetch</code> after installing
from the lite distribution, or use the full distribution.</li>
+<li>If you want <strong>Kafka spout lag</strong> in the Storm UI or the
<code>storm-kafka-monitor</code> command, run
<code>bin/storm-kafka-monitor-fetch</code> after installing from the lite
distribution, or use the full distribution.</li>
+</ul>
+
+<p>For all other use cases — including running Kafka spout topologies without
the lag UI feature — the lite distribution is sufficient and carries no
functional difference from the full distribution.</p>
</div>
diff --git a/releases/3.0.0/Storm-Scheduler.html
b/releases/current/JitterAwareStreamGrouping.html
similarity index 69%
copy from releases/3.0.0/Storm-Scheduler.html
copy to releases/current/JitterAwareStreamGrouping.html
index 6c38c13975..7e3318b8ee 100644
--- a/releases/3.0.0/Storm-Scheduler.html
+++ b/releases/current/JitterAwareStreamGrouping.html
@@ -13,19 +13,19 @@
- <title>Scheduler</title>
+ <title>JitterAwareStreamGrouping</title>
<meta name="description" content="Apache Storm is a free and open source
distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
- <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <link rel="canonical"
href="https://storm.apache.org/releases/current/JitterAwareStreamGrouping.html";>
- <meta property="og:title" content="Scheduler">
+ <meta property="og:title" content="JitterAwareStreamGrouping">
<meta property="og:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta property="og:type" content="website">
- <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <meta property="og:url"
content="https://storm.apache.org/releases/current/JitterAwareStreamGrouping.html";>
<meta property="og:image"
content="https://storm.apache.org/images/logo.png";>
<meta property="og:site_name" content="Apache Storm">
<meta name="twitter:card" content="summary">
- <meta name="twitter:title" content="Scheduler">
+ <meta name="twitter:title" content="JitterAwareStreamGrouping">
<meta name="twitter:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta name="twitter:image"
content="https://storm.apache.org/images/logo.png";>
@@ -179,7 +179,7 @@
<div class="container-fluid">
- <h1 class="page-title">Scheduler</h1>
+ <h1 class="page-title">JitterAwareStreamGrouping</h1>
<div class="row">
<div class="col-md-12">
@@ -187,25 +187,52 @@
<p class="post-meta"></p>
-<div class="documentation-content"><p>Storm now has 4 kinds of built-in
schedulers: <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/DefaultScheduler.java";>DefaultScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/IsolationScheduler.java";>IsolationScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/sch
[...]
+<div class="documentation-content"><p><code>JitterAwareStreamGrouping</code>
is a stream grouping that selects downstream tasks based on observed per-task
execution jitter. It steers traffic away from tasks experiencing backpressure
or execution variance, producing a more even distribution of work over time.</p>
-<h2 id="pluggable-scheduler">Pluggable scheduler</h2>
+<h2 id="background-the-jitter-metric">Background: the jitter metric</h2>
-<p>You can implement your own scheduler to replace the default scheduler to
assign executors to workers. You configure the class to use the
"storm.scheduler" config in your storm.yaml, and your scheduler must
implement the <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-client/src/jvm/org/apache/storm/scheduler/IScheduler.java";>IScheduler</a>
interface.</p>
+<p>Storm 3.0.0 introduces a per-task jitter metric based on the EWMA algorithm
in <a href="https://www.rfc-editor.org/rfc/rfc1889#appendix-A";>RFC 1889
Appendix A</a>. The metric tracks inter-arrival time variance in each
task's processing loop. A lightweight control loop propagates these
measurements to upstream components at regular intervals without touching the
data path.</p>
-<h2 id="isolation-scheduler">Isolation Scheduler</h2>
+<p>A task with a steady processing rate accumulates near-zero jitter. A task
that stalls, GC-pauses, or falls behind its input queue accumulates a rising
jitter estimate, which the grouping uses as a signal to route tuples
elsewhere.</p>
-<p>The isolation scheduler makes it easy and safe to share a cluster among
many topologies. The isolation scheduler lets you specify which topologies
should be "isolated", meaning that they run on a dedicated set of
machines within the cluster where no other topologies will be running. These
isolated topologies are given priority on the cluster, so resources will be
allocated to isolated topologies if there's competition with non-isolated
topologies, and resources will be t [...]
+<h2 id="usage">Usage</h2>
-<p>You can configure the isolation scheduler in the Nimbus configuration by
setting "storm.scheduler" to
"org.apache.storm.scheduler.IsolationScheduler". Then, use the
"isolation.scheduler.machines" config to specify how many machines
each topology should get. This configuration is a map from topology name to the
number of isolated machines allocated to this topology. For example:</p>
-<div class="highlight"><pre><code class="language-"
data-lang="">isolation.scheduler.machines:
- "my-topology": 8
- "tiny-topology": 1
- "some-other-topology": 3
+<p>Use <code>customGrouping</code> with a
<code>JitterAwareStreamGrouping</code> instance in place of
<code>shuffleGrouping</code>:</p>
+<div class="highlight"><pre><code class="language-java" data-lang="java"><span
class="kn">import</span> <span
class="nn">org.apache.storm.grouping.JitterAwareStreamGrouping</span><span
class="o">;</span>
+
+<span class="nc">TopologyBuilder</span> <span class="n">builder</span> <span
class="o">=</span> <span class="k">new</span> <span
class="nc">TopologyBuilder</span><span class="o">();</span>
+<span class="n">builder</span><span class="o">.</span><span
class="na">setSpout</span><span class="o">(</span><span
class="s">"spout"</span><span class="o">,</span> <span class="k">new</span>
<span class="nc">MySpout</span><span class="o">(),</span> <span
class="mi">1</span><span class="o">);</span>
+<span class="n">builder</span><span class="o">.</span><span
class="na">setBolt</span><span class="o">(</span><span
class="s">"bolt"</span><span class="o">,</span> <span class="k">new</span>
<span class="nc">MyBolt</span><span class="o">(),</span> <span
class="mi">4</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">customGrouping</span><span
class="o">(</span><span class="s">"spout"</span><span class="o">,</span> <span
class="k">new</span> <span class="nc">JitterAwareStreamGrouping</span><span
class="o">());</span>
+</code></pre></div>
+<p><code>JitterAwareStreamGrouping</code> is serializable and safe across
workers. Before any jitter data arrives (e.g., at topology startup) it falls
back to random task selection.</p>
+
+<h2 id="benchmarking">Benchmarking</h2>
+
+<p>The <code>storm-perf</code> module includes a dedicated benchmark
topology:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">storm jar
storm-perf/target/storm-perf-*.jar \
+ org.apache.storm.perf.JitterAwareGroupingTopology
</code></pre></div>
-<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
+<p>Run it alongside the equivalent shuffle-grouping topology to measure the
effect in your cluster. The <a href="Local-dev-cluster.html">local development
cluster</a> provides Grafana dashboards with per-task jitter visibility.</p>
+
+<h2 id="when-to-use-it">When to use it</h2>
+
+<ul>
+<li>Bolts with variable execution time (external I/O, cache misses,
non-uniform key distributions) where a subset of tasks may consistently
lag.</li>
+<li>Topologies where a single hot task creates cascading backpressure
upstream.</li>
+<li>As a drop-in replacement for <code>shuffleGrouping</code> on any stream
where tasks are interchangeable and execution uniformity matters.</li>
+</ul>
+
+<h2 id="when-not-to-use-it">When not to use it</h2>
+
+<ul>
+<li><strong>Data-locality requirements</strong>: use
<code>fieldsGrouping</code> where tuples must reach a specific task.</li>
+<li><strong>Low parallelism</strong> (2 tasks): the grouping has fewer
candidates to choose between and the benefit diminishes.</li>
+<li><strong>Order-sensitive streams</strong>: like any non-deterministic
grouping, this does not preserve tuple order across tasks.</li>
+<li><strong>Intra-worker traffic</strong>: for streams where both components
run in the same worker, <code>localOrShuffleGrouping</code> avoids
serialization entirely and is preferable.</li>
+</ul>
-<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+<p>The jitter control loop runs on all topologies regardless of whether
<code>JitterAwareStreamGrouping</code> is in use. The grouping simply reads
measurements the loop already produces; enabling it carries no additional
overhead on the data path.</p>
</div>
diff --git a/releases/3.0.0/Storm-Scheduler.html
b/releases/current/Local-dev-cluster.html
similarity index 76%
copy from releases/3.0.0/Storm-Scheduler.html
copy to releases/current/Local-dev-cluster.html
index 6c38c13975..6a4f2ba06d 100644
--- a/releases/3.0.0/Storm-Scheduler.html
+++ b/releases/current/Local-dev-cluster.html
@@ -13,19 +13,19 @@
- <title>Scheduler</title>
+ <title>Local Development Cluster</title>
<meta name="description" content="Apache Storm is a free and open source
distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
- <link rel="canonical"
href="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <link rel="canonical"
href="https://storm.apache.org/releases/current/Local-dev-cluster.html";>
- <meta property="og:title" content="Scheduler">
+ <meta property="og:title" content="Local Development Cluster">
<meta property="og:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta property="og:type" content="website">
- <meta property="og:url"
content="https://storm.apache.org/releases/3.0.0/Storm-Scheduler.html";>
+ <meta property="og:url"
content="https://storm.apache.org/releases/current/Local-dev-cluster.html";>
<meta property="og:image"
content="https://storm.apache.org/images/logo.png";>
<meta property="og:site_name" content="Apache Storm">
<meta name="twitter:card" content="summary">
- <meta name="twitter:title" content="Scheduler">
+ <meta name="twitter:title" content="Local Development Cluster">
<meta name="twitter:description" content="Apache Storm is a free and open
source distributed realtime computation system. Storm makes it easy to reliably
process unbounded streams of data, doing for realtime processing what Hadoop
did for batch processing.">
<meta name="twitter:image"
content="https://storm.apache.org/images/logo.png";>
@@ -179,7 +179,7 @@
<div class="container-fluid">
- <h1 class="page-title">Scheduler</h1>
+ <h1 class="page-title">Local Development Cluster</h1>
<div class="row">
<div class="col-md-12">
@@ -187,25 +187,54 @@
<p class="post-meta"></p>
-<div class="documentation-content"><p>Storm now has 4 kinds of built-in
schedulers: <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/DefaultScheduler.java";>DefaultScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/scheduler/IsolationScheduler.java";>IsolationScheduler</a>,
<a
href="https://github.com/apache/storm/blob/v3.0.0/storm-server/src/main/java/org/apache/storm/sch
[...]
+<div class="documentation-content"><p>Storm ships a Docker Compose
configuration under <code>docker/</code> that provisions a fully distributed
cluster on a single machine. Unlike <a href="Local-mode.html">local mode</a>,
this setup forces inter-worker traffic across real sockets, making it suitable
for benchmarking serialization, compression, and network-sensitive behaviours
that only appear with genuine inter-process communication.</p>
-<h2 id="pluggable-scheduler">Pluggable scheduler</h2>
+<h2 id="whats-included">What's included</h2>
-<p>You can implement your own scheduler to replace the default scheduler to
assign executors to workers. You configure the class to use the
"storm.scheduler" config in your storm.yaml, and your scheduler must
implement the <a
href="https://github.com/apache/storm/blob/v3.0.0/storm-client/src/jvm/org/apache/storm/scheduler/IScheduler.java";>IScheduler</a>
interface.</p>
+<ul>
+<li><strong>Nimbus</strong>, <strong>ZooKeeper</strong>, and two
<strong>Supervisors</strong> on an isolated Docker network</li>
+<li><strong>Prometheus</strong> scraping Storm's Metrics V2 endpoint</li>
+<li><strong>Grafana</strong> with a pre-built per-task dashboard covering
throughput, latency, and jitter</li>
+<li><strong><code>netsim.sh</code></strong> — a utility that injects
controlled latency and jitter between Supervisors using <code>tc
netem</code></li>
+</ul>
-<h2 id="isolation-scheduler">Isolation Scheduler</h2>
+<h2 id="starting-the-cluster">Starting the cluster</h2>
+<div class="highlight"><pre><code class="language-" data-lang="">cd docker/
+docker compose up -d
+</code></pre></div>
+<p>Once healthy, Grafana is available at <code>http://localhost:3000</code>
(default credentials: <code>admin</code> / <code>admin</code>). The Storm UI is
at <code>http://localhost:8080</code>.</p>
+
+<h2 id="submitting-a-topology">Submitting a topology</h2>
+
+<p>The <code>storm-perf</code> module contains topologies designed for this
environment:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">storm jar
storm-perf/target/storm-perf-*.jar \
+ org.apache.storm.perf.FileReadWordCountTopo \
+ -c nimbus.seeds='["nimbus"]' \
+ -c storm.zookeeper.servers='["zookeeper"]'
+</code></pre></div>
+<p>Any topology JAR works. Point <code>nimbus.seeds</code> and
<code>storm.zookeeper.servers</code> at the Docker hostnames as shown above.</p>
-<p>The isolation scheduler makes it easy and safe to share a cluster among
many topologies. The isolation scheduler lets you specify which topologies
should be "isolated", meaning that they run on a dedicated set of
machines within the cluster where no other topologies will be running. These
isolated topologies are given priority on the cluster, so resources will be
allocated to isolated topologies if there's competition with non-isolated
topologies, and resources will be t [...]
+<h2 id="network-simulation">Network simulation</h2>
-<p>You can configure the isolation scheduler in the Nimbus configuration by
setting "storm.scheduler" to
"org.apache.storm.scheduler.IsolationScheduler". Then, use the
"isolation.scheduler.machines" config to specify how many machines
each topology should get. This configuration is a map from topology name to the
number of isolated machines allocated to this topology. For example:</p>
-<div class="highlight"><pre><code class="language-"
data-lang="">isolation.scheduler.machines:
- "my-topology": 8
- "tiny-topology": 1
- "some-other-topology": 3
+<p><code>netsim.sh</code> wraps <code>tc netem</code> to shape traffic between
the two Supervisor containers:</p>
+<div class="highlight"><pre><code class="language-" data-lang=""># Add 3 ms
latency and 1 ms jitter (typical intra-datacenter profile)
+./netsim.sh apply 3ms 1ms
+
+# Remove all shaping
+./netsim.sh reset
+
+# Show current tc settings on both supervisors
+./netsim.sh status
+
+# Round-trip ping between supervisors
+./netsim.sh ping
</code></pre></div>
-<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
+<p>Changes take effect immediately and are visible in the Grafana latency
panels in real time.</p>
-<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+<h2 id="teardown">Teardown</h2>
+<div class="highlight"><pre><code class="language-" data-lang="">docker
compose down
+</code></pre></div>
+<p>Persistent volumes are not configured; all topology data and metrics are
lost on teardown. This cluster is intended for development and benchmarking,
not production use.</p>
</div>
diff --git a/releases/current/Storm-Scheduler.html
b/releases/current/Storm-Scheduler.html
index d9f0b1220d..3db1b1250c 100644
--- a/releases/current/Storm-Scheduler.html
+++ b/releases/current/Storm-Scheduler.html
@@ -206,6 +206,56 @@
<p>Any topologies submitted to the cluster not listed there will not be
isolated. Note that there is no way for a user of Storm to affect their
isolation settings – this is only allowed by the administrator of the cluster
(this is very much intentional).</p>
<p>The isolation scheduler solves the multi-tenancy problem – avoiding
resource contention between topologies – by providing full isolation between
topologies. The intention is that "productionized" topologies should
be listed in the isolation config, and test or in-development topologies should
not. The remaining machines on the cluster serve the dual role of failover for
isolated topologies and for running the non-isolated topologies.</p>
+
+<p>When choosing which hosts to assign an isolated topology to, the scheduler
sorts eligible hosts by three criteria in order:</p>
+
+<ol>
+<li><strong>Assignable slots (descending)</strong> — prefer hosts with greater
total slot capacity.</li>
+<li><strong>Free slots (descending)</strong> — among hosts of equal capacity,
prefer the one with more currently free slots, minimising the number of worker
evictions needed.</li>
+<li><strong>Hostname (ascending)</strong> — alphabetical tiebreaker for
deterministic assignments.</li>
+</ol>
+
+<p>The secondary and tertiary sorts were added in Storm 3.0.0. Previously only
the primary sort applied, which could cause unnecessary evictions when two
hosts had the same total capacity but different numbers of occupied slots.</p>
+
+<h2 id="evenscheduler">EvenScheduler</h2>
+
+<p><code>EvenScheduler</code> (used directly or via
<code>DefaultScheduler</code>) distributes workers as evenly as possible across
available supervisors. It is the default when no custom scheduler is configured
and <code>IsolationScheduler</code> is not in use.</p>
+
+<h3 id="idle-supervisor-rebalance">Idle supervisor rebalance</h3>
+
+<p><em>Available since Storm 3.0.0. Disabled by default.</em></p>
+
+<p>When a supervisor returns from maintenance, its slots are normally left
idle: each topology's desired worker count is already satisfied on the
surviving supervisors, so <code>needsScheduling</code> reports nothing to do.
Workers only migrate back if an operator manually rebalances or restarts every
affected topology.</p>
+
+<p>Setting <code>nimbus.even.rebalance.idle.supervisor.enabled: true</code>
adds an opt-in pass that relocates workers onto the returned supervisor
automatically in a single scheduling round, distributed across topologies in
round-robin order.</p>
+
+<table><thead>
+<tr>
+<th>Key</th>
+<th>Default</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td><code>nimbus.even.rebalance.idle.supervisor.enabled</code></td>
+<td><code>false</code></td>
+<td>Master switch. When false the pass never runs.</td>
+</tr>
+<tr>
+<td><code>nimbus.even.rebalance.max.free.per.topology</code></td>
+<td><code>0</code></td>
+<td>Per-topology cap on workers relocated per round. <code>0</code> means no
additional cap beyond the even-distribution budget.</td>
+</tr>
+<tr>
+<td><code>nimbus.even.rebalance.idle.supervisor.min.stable.rounds</code></td>
+<td><code>3</code></td>
+<td>Number of consecutive monitoring rounds a supervisor must be seen as
stable before it is eligible as a relocation target. Set to <code>0</code> to
disable the flap guard.</td>
+</tr>
+</tbody></table>
+
+<p>A relocation is a worker JVM restart, involving brief tuple replay, JIT
re-warmup, and possible windowed or stateful bolt state restore. Keep this off
for topologies with stateful or windowed bolts that have a non-trivial restore
cost, latency-sensitive topologies sensitive to JIT re-warmup, and clusters
where supervisors are prone to flapping.</p>
+
+<p><strong>Scope:</strong> <code>EvenScheduler</code> and
<code>DefaultScheduler</code> only. <code>ResourceAwareScheduler</code> is
unaffected.</p>
</div>
diff --git a/releases/current/index.html b/releases/current/index.html
index 5d48567eb0..69bd447f52 100644
--- a/releases/current/index.html
+++ b/releases/current/index.html
@@ -250,6 +250,9 @@ We're also notifying it via annotating classes with
marker interface <code>@
<li><a href="ClusterMetrics.html">Daemon Metrics/Monitoring</a></li>
<li><a href="windows-users-guide.html">Windows users guide</a></li>
<li><a href="Classpath-handling.html">Classpath handling</a></li>
+<li><a href="Binary-distributions.html">Binary Distributions (full vs.
lite)</a></li>
+<li><a href="Cluster-State-Serialization.html">Cluster State
Serialization</a></li>
+<li><a href="Local-dev-cluster.html">Local Development Cluster</a></li>
</ul>
<h3 id="intermediate">Intermediate</h3>
@@ -266,6 +269,7 @@ We're also notifying it via annotating classes with
marker interface <code>@
<li><a href="State-checkpointing.html">State Checkpointing</a></li>
<li><a href="Windowing.html">Windowing</a></li>
<li><a href="Joins.html">Joining Streams</a></li>
+<li><a href="JitterAwareStreamGrouping.html">JitterAwareStreamGrouping</a></li>
<li><a href="distcache-blobstore.html">Blobstore(Distcache)</a></li>
</ul>
![https://storm.apache.org/images/logo.png"](https://storm.apache.org/images/logo.png"){kind=link}