This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
The following commit(s) were added to refs/heads/master by this push:
new 8df49081e docs: add missing 3.0.0 feature documentation (#8949)
8df49081e is described below
commit 8df49081e9bd03c7fcea94c7d1898603fee1f006
Author: reiabreu <[email protected]>
AuthorDate: Sat Jul 25 16:54:36 2026 +0100
docs: add missing 3.0.0 feature documentation (#8949)
---
docs/Binary-distributions.md | 60 ++++++++++++++++++++++++++++++++++++
docs/JitterAwareStreamGrouping.md | 54 +++++++++++++++++++++++++++++++++
docs/Local-dev-cluster.md | 64 +++++++++++++++++++++++++++++++++++++++
docs/Storm-Scheduler.md | 30 ++++++++++++++++++
docs/index.md | 4 +++
5 files changed, 212 insertions(+)
diff --git a/docs/Binary-distributions.md b/docs/Binary-distributions.md
new file mode 100644
index 000000000..51c7ab73c
--- /dev/null
+++ b/docs/Binary-distributions.md
@@ -0,0 +1,60 @@
+---
+title: Binary Distributions
+layout: documentation
+documentation: true
+---
+
+Storm 3.0.0 ships two binary distributions. Both contain identical core
functionality; they differ only in which optional plugins are bundled.
+
+| Distribution | Filename | Approx. size |
+|---|---|---|
+| **Full** | `apache-storm-<version>.tar.gz` | ~395 MB of jars |
+| **Lite** | `apache-storm-<version>-lite.tar.gz` | ~207 MB of jars (~47%
smaller) |
+
+## What the lite distribution omits
+
+### storm-autocreds (−79 MB)
+
+`storm-autocreds` provides Nimbus and Supervisor with the ability to populate
and renew HDFS and HBase delegation tokens on **Kerberos-secured clusters**. It
pulls in the full Hadoop and HBase client trees. The plugin is off by default
and is only needed on secure Hadoop deployments.
+
+In the lite distribution, only the `external/storm-autocreds/README` ships. To
install the plugin on demand, run:
+
+```
+bin/storm-autocreds-fetch
+```
+
+This uses Maven to resolve `org.apache.storm:storm-autocreds` and its runtime
dependencies from Maven Central and copies them into `extlib-daemon/`. Restart
Nimbus and Supervisor afterwards.
+
+Pass `--version` and `--dest` to override the detected Storm version or target
directory. Pass `--` to forward arguments to Maven (e.g. to use an internal
mirror or an offline local repository):
+
+```
+bin/storm-autocreds-fetch -- -s /path/to/settings.xml
+bin/storm-autocreds-fetch -- -Dmaven.repo.local=/path/to/offline-repo -o
+```
+
+### storm-kafka-monitor (−38 MB)
+
+`storm-kafka-monitor` displays Kafka spout consumer lag in the Storm UI and
powers the `bin/storm-kafka-monitor` command. It pulls in the Kafka client
library and is only needed when running Kafka spouts and wanting lag metrics.
When it is absent, the UI degrades gracefully: the lag column shows an
actionable message rather than failing, and the `bin/storm-kafka-monitor`
wrapper prints a hint to run the fetch command.
+
+In the lite distribution, only the `external/storm-kafka-monitor/README`
ships. To install it on demand, run:
+
+```
+bin/storm-kafka-monitor-fetch
+```
+
+This resolves `org.apache.storm:storm-kafka-monitor` and its runtime
dependencies into `lib-tools/storm-kafka-monitor/`. No daemon restart is
required; the UI picks it up on the next request.
+
+The same `--version`, `--dest`, and `--` passthrough options are available as
for `storm-autocreds-fetch`.
+
+## lib-common: shared jar de-duplication
+
+Both distributions include a structural change that is transparent to users:
jars that were previously duplicated across the daemon classpath (`lib/`) and
the worker classpath (`lib-worker/`) are now kept in a single `lib-common/`
directory. `bin/storm.py` adds `lib-common` to both classpaths, so the runtime
behaviour is unchanged. Only byte-identical jars (same filename and SHA-256)
were de-duplicated; no version was silently merged. This accounts for
approximately 71 MB of the reduction.
+
+## Which distribution should I use?
+
+Use the **lite distribution** unless you specifically need one of the
unbundled plugins:
+
+- If you run topologies on a **Kerberos-secured Hadoop or HBase cluster**, run
`bin/storm-autocreds-fetch` after installing from the lite distribution, or use
the full distribution.
+- If you want **Kafka spout lag** in the Storm UI or the `storm-kafka-monitor`
command, run `bin/storm-kafka-monitor-fetch` after installing from the lite
distribution, or use the full distribution.
+
+For all other use cases — including running Kafka spout topologies without the
lag UI feature — the lite distribution is sufficient and carries no functional
difference from the full distribution.
diff --git a/docs/JitterAwareStreamGrouping.md
b/docs/JitterAwareStreamGrouping.md
new file mode 100644
index 000000000..2932c6aff
--- /dev/null
+++ b/docs/JitterAwareStreamGrouping.md
@@ -0,0 +1,54 @@
+---
+title: JitterAwareStreamGrouping
+layout: documentation
+documentation: true
+---
+
+`JitterAwareStreamGrouping` is a stream grouping that selects downstream tasks
based on observed per-task execution jitter. It steers traffic away from tasks
experiencing backpressure or execution variance, producing a more even
distribution of work over time.
+
+## Background: the jitter metric
+
+Storm 3.0.0 introduces a per-task jitter metric based on the EWMA algorithm in
[RFC 1889 Appendix A](https://www.rfc-editor.org/rfc/rfc1889#appendix-A). The
metric tracks inter-arrival time variance in each task's processing loop. A
lightweight control loop propagates these measurements to upstream components
at regular intervals without touching the data path.
+
+A task with a steady processing rate accumulates near-zero jitter. A task that
stalls, GC-pauses, or falls behind its input queue accumulates a rising jitter
estimate, which the grouping uses as a signal to route tuples elsewhere.
+
+## Usage
+
+Use `customGrouping` with a `JitterAwareStreamGrouping` instance in place of
`shuffleGrouping`:
+
+```java
+import org.apache.storm.grouping.JitterAwareStreamGrouping;
+
+TopologyBuilder builder = new TopologyBuilder();
+builder.setSpout("spout", new MySpout(), 1);
+builder.setBolt("bolt", new MyBolt(), 4)
+ .customGrouping("spout", new JitterAwareStreamGrouping());
+```
+
+`JitterAwareStreamGrouping` is serializable and safe across workers. Before
any jitter data arrives (e.g., at topology startup) it falls back to random
task selection.
+
+## Benchmarking
+
+The `storm-perf` module includes a dedicated benchmark topology:
+
+```
+storm jar storm-perf/target/storm-perf-*.jar \
+ org.apache.storm.perf.JitterAwareGroupingTopology
+```
+
+Run it alongside the equivalent shuffle-grouping topology to measure the
effect in your cluster. The [local development cluster](Local-dev-cluster.html)
provides Grafana dashboards with per-task jitter visibility.
+
+## When to use it
+
+- Bolts with variable execution time (external I/O, cache misses, non-uniform
key distributions) where a subset of tasks may consistently lag.
+- Topologies where a single hot task creates cascading backpressure upstream.
+- As a drop-in replacement for `shuffleGrouping` on any stream where tasks are
interchangeable and execution uniformity matters.
+
+## When not to use it
+
+- **Data-locality requirements**: use `fieldsGrouping` where tuples must reach
a specific task.
+- **Low parallelism** (2 tasks): the grouping has fewer candidates to choose
between and the benefit diminishes.
+- **Order-sensitive streams**: like any non-deterministic grouping, this does
not preserve tuple order across tasks.
+- **Intra-worker traffic**: for streams where both components run in the same
worker, `localOrShuffleGrouping` avoids serialization entirely and is
preferable.
+
+The jitter control loop runs on all topologies regardless of whether
`JitterAwareStreamGrouping` is in use. The grouping simply reads measurements
the loop already produces; enabling it carries no additional overhead on the
data path.
diff --git a/docs/Local-dev-cluster.md b/docs/Local-dev-cluster.md
new file mode 100644
index 000000000..9b4947402
--- /dev/null
+++ b/docs/Local-dev-cluster.md
@@ -0,0 +1,64 @@
+---
+title: Local Development Cluster
+layout: documentation
+documentation: true
+---
+
+Storm ships a Docker Compose configuration under `docker/` that provisions a
fully distributed cluster on a single machine. Unlike [local
mode](Local-mode.html), this setup forces inter-worker traffic across real
sockets, making it suitable for benchmarking serialization, compression, and
network-sensitive behaviours that only appear with genuine inter-process
communication.
+
+## What's included
+
+- **Nimbus**, **ZooKeeper**, and two **Supervisors** on an isolated Docker
network
+- **Prometheus** scraping Storm's Metrics V2 endpoint
+- **Grafana** with a pre-built per-task dashboard covering throughput,
latency, and jitter
+- **`netsim.sh`** — a utility that injects controlled latency and jitter
between Supervisors using `tc netem`
+
+## Starting the cluster
+
+```
+cd docker/
+docker compose up -d
+```
+
+Once healthy, Grafana is available at `http://localhost:3000` (default
credentials: `admin` / `admin`). The Storm UI is at `http://localhost:8080`.
+
+## Submitting a topology
+
+The `storm-perf` module contains topologies designed for this environment:
+
+```
+storm jar storm-perf/target/storm-perf-*.jar \
+ org.apache.storm.perf.FileReadWordCountTopo \
+ -c nimbus.seeds='["nimbus"]' \
+ -c storm.zookeeper.servers='["zookeeper"]'
+```
+
+Any topology JAR works. Point `nimbus.seeds` and `storm.zookeeper.servers` at
the Docker hostnames as shown above.
+
+## Network simulation
+
+`netsim.sh` wraps `tc netem` to shape traffic between the two Supervisor
containers:
+
+```
+# Add 3 ms latency and 1 ms jitter (typical intra-datacenter profile)
+./netsim.sh apply 3ms 1ms
+
+# Remove all shaping
+./netsim.sh reset
+
+# Show current tc settings on both supervisors
+./netsim.sh status
+
+# Round-trip ping between supervisors
+./netsim.sh ping
+```
+
+Changes take effect immediately and are visible in the Grafana latency panels
in real time.
+
+## Teardown
+
+```
+docker compose down
+```
+
+Persistent volumes are not configured; all topology data and metrics are lost
on teardown. This cluster is intended for development and benchmarking, not
production use.
diff --git a/docs/Storm-Scheduler.md b/docs/Storm-Scheduler.md
index 4b9807ea2..23c7bff17 100644
--- a/docs/Storm-Scheduler.md
+++ b/docs/Storm-Scheduler.md
@@ -25,3 +25,33 @@ Any topologies submitted to the cluster not listed there
will not be isolated. N
The isolation scheduler solves the multi-tenancy problem – avoiding resource
contention between topologies – by providing full isolation between topologies.
The intention is that "productionized" topologies should be listed in the
isolation config, and test or in-development topologies should not. The
remaining machines on the cluster serve the dual role of failover for isolated
topologies and for running the non-isolated topologies.
+When choosing which hosts to assign an isolated topology to, the scheduler
sorts eligible hosts by three criteria in order:
+
+1. **Assignable slots (descending)** — prefer hosts with greater total slot
capacity.
+2. **Free slots (descending)** — among hosts of equal capacity, prefer the one
with more currently free slots, minimising the number of worker evictions
needed.
+3. **Hostname (ascending)** — alphabetical tiebreaker for deterministic
assignments.
+
+The secondary and tertiary sorts were added in Storm 3.0.0. Previously only
the primary sort applied, which could cause unnecessary evictions when two
hosts had the same total capacity but different numbers of occupied slots.
+
+## EvenScheduler
+
+`EvenScheduler` (used directly or via `DefaultScheduler`) distributes workers
as evenly as possible across available supervisors. It is the default when no
custom scheduler is configured and `IsolationScheduler` is not in use.
+
+### Idle supervisor rebalance
+
+*Available since Storm 3.0.0. Disabled by default.*
+
+When a supervisor returns from maintenance, its slots are normally left idle:
each topology's desired worker count is already satisfied on the surviving
supervisors, so `needsScheduling` reports nothing to do. Workers only migrate
back if an operator manually rebalances or restarts every affected topology.
+
+Setting `nimbus.even.rebalance.idle.supervisor.enabled: true` adds an opt-in
pass that relocates workers onto the returned supervisor automatically in a
single scheduling round, distributed across topologies in round-robin order.
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `nimbus.even.rebalance.idle.supervisor.enabled` | `false` | Master switch.
When false the pass never runs. |
+| `nimbus.even.rebalance.max.free.per.topology` | `0` | Per-topology cap on
workers relocated per round. `0` means no additional cap beyond the
even-distribution budget. |
+| `nimbus.even.rebalance.idle.supervisor.min.stable.rounds` | `3` | Number of
consecutive monitoring rounds a supervisor must be seen as stable before it is
eligible as a relocation target. Set to `0` to disable the flap guard. |
+
+A relocation is a worker JVM restart, involving brief tuple replay, JIT
re-warmup, and possible windowed or stateful bolt state restore. Keep this off
for topologies with stateful or windowed bolts that have a non-trivial restore
cost, latency-sensitive topologies sensitive to JIT re-warmup, and clusters
where supervisors are prone to flapping.
+
+**Scope:** `EvenScheduler` and `DefaultScheduler` only.
`ResourceAwareScheduler` is unaffected.
+
diff --git a/docs/index.md b/docs/index.md
index 8a7a31550..561d4f3e6 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -57,6 +57,9 @@ We're also notifying it via annotating classes with marker
interface `@Interface
* [Daemon Metrics/Monitoring](ClusterMetrics.html)
* [Windows users guide](windows-users-guide.html)
* [Classpath handling](Classpath-handling.html)
+* [Binary Distributions (full vs. lite)](Binary-distributions.html)
+* [Cluster State Serialization](Cluster-State-Serialization.html)
+* [Local Development Cluster](Local-dev-cluster.html)
### Intermediate
@@ -71,6 +74,7 @@ We're also notifying it via annotating classes with marker
interface `@Interface
* [State Checkpointing](State-checkpointing.html)
* [Windowing](Windowing.html)
* [Joining Streams](Joins.html)
+* [JitterAwareStreamGrouping](JitterAwareStreamGrouping.html)
* [Blobstore(Distcache)](distcache-blobstore.html)
### Debugging
