This is an automated email from the ASF dual-hosted git repository. reiabreu pushed a commit to branch docs/3.0.0-new-features in repository https://gitbox.apache.org/repos/asf/storm.git
commit efeecedc804a865821b3ce145bf7dd484cde5310 Author: Rui Abreu <[email protected]> AuthorDate: Sat Jul 25 15:21:32 2026 +0100 docs: add missing 3.0.0 feature documentation - Binary-distributions.md: full vs. lite distribution, fetch scripts for storm-autocreds and storm-kafka-monitor, lib-common deduplication - JitterAwareStreamGrouping.md: RFC 1889 EWMA jitter metric, usage, when to use vs. shuffleGrouping - Local-dev-cluster.md: Docker Compose cluster, Prometheus/Grafana, netsim.sh - Storm-Scheduler.md: EvenScheduler idle supervisor rebalance section - index.md: link all new pages; link existing Cluster-State-Serialization.md which was present but unreachable from the index Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --- docs/Binary-distributions.md | 60 ++++++++++++++++++++++++++++++++++++ docs/JitterAwareStreamGrouping.md | 54 +++++++++++++++++++++++++++++++++ docs/Local-dev-cluster.md | 64 +++++++++++++++++++++++++++++++++++++++ docs/Storm-Scheduler.md | 22 ++++++++++++++ docs/index.md | 4 +++ 5 files changed, 204 insertions(+) diff --git a/docs/Binary-distributions.md b/docs/Binary-distributions.md new file mode 100644 index 000000000..51c7ab73c --- /dev/null +++ b/docs/Binary-distributions.md @@ -0,0 +1,60 @@ +--- +title: Binary Distributions +layout: documentation +documentation: true +--- + +Storm 3.0.0 ships two binary distributions. Both contain identical core functionality; they differ only in which optional plugins are bundled. + +| Distribution | Filename | Approx. size | +|---|---|---| +| **Full** | `apache-storm-<version>.tar.gz` | ~395 MB of jars | +| **Lite** | `apache-storm-<version>-lite.tar.gz` | ~207 MB of jars (~47% smaller) | + +## What the lite distribution omits + +### storm-autocreds (−79 MB) + +`storm-autocreds` provides Nimbus and Supervisor with the ability to populate and renew HDFS and HBase delegation tokens on **Kerberos-secured clusters**. It pulls in the full Hadoop and HBase client trees. The plugin is off by default and is only needed on secure Hadoop deployments. + +In the lite distribution, only the `external/storm-autocreds/README` ships. To install the plugin on demand, run: + +``` +bin/storm-autocreds-fetch +``` + +This uses Maven to resolve `org.apache.storm:storm-autocreds` and its runtime dependencies from Maven Central and copies them into `extlib-daemon/`. Restart Nimbus and Supervisor afterwards. + +Pass `--version` and `--dest` to override the detected Storm version or target directory. Pass `--` to forward arguments to Maven (e.g. to use an internal mirror or an offline local repository): + +``` +bin/storm-autocreds-fetch -- -s /path/to/settings.xml +bin/storm-autocreds-fetch -- -Dmaven.repo.local=/path/to/offline-repo -o +``` + +### storm-kafka-monitor (−38 MB) + +`storm-kafka-monitor` displays Kafka spout consumer lag in the Storm UI and powers the `bin/storm-kafka-monitor` command. It pulls in the Kafka client library and is only needed when running Kafka spouts and wanting lag metrics. When it is absent, the UI degrades gracefully: the lag column shows an actionable message rather than failing, and the `bin/storm-kafka-monitor` wrapper prints a hint to run the fetch command. + +In the lite distribution, only the `external/storm-kafka-monitor/README` ships. To install it on demand, run: + +``` +bin/storm-kafka-monitor-fetch +``` + +This resolves `org.apache.storm:storm-kafka-monitor` and its runtime dependencies into `lib-tools/storm-kafka-monitor/`. No daemon restart is required; the UI picks it up on the next request. + +The same `--version`, `--dest`, and `--` passthrough options are available as for `storm-autocreds-fetch`. + +## lib-common: shared jar de-duplication + +Both distributions include a structural change that is transparent to users: jars that were previously duplicated across the daemon classpath (`lib/`) and the worker classpath (`lib-worker/`) are now kept in a single `lib-common/` directory. `bin/storm.py` adds `lib-common` to both classpaths, so the runtime behaviour is unchanged. Only byte-identical jars (same filename and SHA-256) were de-duplicated; no version was silently merged. This accounts for approximately 71 MB of the reduction. + +## Which distribution should I use? + +Use the **lite distribution** unless you specifically need one of the unbundled plugins: + +- If you run topologies on a **Kerberos-secured Hadoop or HBase cluster**, run `bin/storm-autocreds-fetch` after installing from the lite distribution, or use the full distribution. +- If you want **Kafka spout lag** in the Storm UI or the `storm-kafka-monitor` command, run `bin/storm-kafka-monitor-fetch` after installing from the lite distribution, or use the full distribution. + +For all other use cases — including running Kafka spout topologies without the lag UI feature — the lite distribution is sufficient and carries no functional difference from the full distribution. diff --git a/docs/JitterAwareStreamGrouping.md b/docs/JitterAwareStreamGrouping.md new file mode 100644 index 000000000..2932c6aff --- /dev/null +++ b/docs/JitterAwareStreamGrouping.md @@ -0,0 +1,54 @@ +--- +title: JitterAwareStreamGrouping +layout: documentation +documentation: true +--- + +`JitterAwareStreamGrouping` is a stream grouping that selects downstream tasks based on observed per-task execution jitter. It steers traffic away from tasks experiencing backpressure or execution variance, producing a more even distribution of work over time. + +## Background: the jitter metric + +Storm 3.0.0 introduces a per-task jitter metric based on the EWMA algorithm in [RFC 1889 Appendix A](https://www.rfc-editor.org/rfc/rfc1889#appendix-A). The metric tracks inter-arrival time variance in each task's processing loop. A lightweight control loop propagates these measurements to upstream components at regular intervals without touching the data path. + +A task with a steady processing rate accumulates near-zero jitter. A task that stalls, GC-pauses, or falls behind its input queue accumulates a rising jitter estimate, which the grouping uses as a signal to route tuples elsewhere. + +## Usage + +Use `customGrouping` with a `JitterAwareStreamGrouping` instance in place of `shuffleGrouping`: + +```java +import org.apache.storm.grouping.JitterAwareStreamGrouping; + +TopologyBuilder builder = new TopologyBuilder(); +builder.setSpout("spout", new MySpout(), 1); +builder.setBolt("bolt", new MyBolt(), 4) + .customGrouping("spout", new JitterAwareStreamGrouping()); +``` + +`JitterAwareStreamGrouping` is serializable and safe across workers. Before any jitter data arrives (e.g., at topology startup) it falls back to random task selection. + +## Benchmarking + +The `storm-perf` module includes a dedicated benchmark topology: + +``` +storm jar storm-perf/target/storm-perf-*.jar \ + org.apache.storm.perf.JitterAwareGroupingTopology +``` + +Run it alongside the equivalent shuffle-grouping topology to measure the effect in your cluster. The [local development cluster](Local-dev-cluster.html) provides Grafana dashboards with per-task jitter visibility. + +## When to use it + +- Bolts with variable execution time (external I/O, cache misses, non-uniform key distributions) where a subset of tasks may consistently lag. +- Topologies where a single hot task creates cascading backpressure upstream. +- As a drop-in replacement for `shuffleGrouping` on any stream where tasks are interchangeable and execution uniformity matters. + +## When not to use it + +- **Data-locality requirements**: use `fieldsGrouping` where tuples must reach a specific task. +- **Low parallelism** (2 tasks): the grouping has fewer candidates to choose between and the benefit diminishes. +- **Order-sensitive streams**: like any non-deterministic grouping, this does not preserve tuple order across tasks. +- **Intra-worker traffic**: for streams where both components run in the same worker, `localOrShuffleGrouping` avoids serialization entirely and is preferable. + +The jitter control loop runs on all topologies regardless of whether `JitterAwareStreamGrouping` is in use. The grouping simply reads measurements the loop already produces; enabling it carries no additional overhead on the data path. diff --git a/docs/Local-dev-cluster.md b/docs/Local-dev-cluster.md new file mode 100644 index 000000000..9b4947402 --- /dev/null +++ b/docs/Local-dev-cluster.md @@ -0,0 +1,64 @@ +--- +title: Local Development Cluster +layout: documentation +documentation: true +--- + +Storm ships a Docker Compose configuration under `docker/` that provisions a fully distributed cluster on a single machine. Unlike [local mode](Local-mode.html), this setup forces inter-worker traffic across real sockets, making it suitable for benchmarking serialization, compression, and network-sensitive behaviours that only appear with genuine inter-process communication. + +## What's included + +- **Nimbus**, **ZooKeeper**, and two **Supervisors** on an isolated Docker network +- **Prometheus** scraping Storm's Metrics V2 endpoint +- **Grafana** with a pre-built per-task dashboard covering throughput, latency, and jitter +- **`netsim.sh`** — a utility that injects controlled latency and jitter between Supervisors using `tc netem` + +## Starting the cluster + +``` +cd docker/ +docker compose up -d +``` + +Once healthy, Grafana is available at `http://localhost:3000` (default credentials: `admin` / `admin`). The Storm UI is at `http://localhost:8080`. + +## Submitting a topology + +The `storm-perf` module contains topologies designed for this environment: + +``` +storm jar storm-perf/target/storm-perf-*.jar \ + org.apache.storm.perf.FileReadWordCountTopo \ + -c nimbus.seeds='["nimbus"]' \ + -c storm.zookeeper.servers='["zookeeper"]' +``` + +Any topology JAR works. Point `nimbus.seeds` and `storm.zookeeper.servers` at the Docker hostnames as shown above. + +## Network simulation + +`netsim.sh` wraps `tc netem` to shape traffic between the two Supervisor containers: + +``` +# Add 3 ms latency and 1 ms jitter (typical intra-datacenter profile) +./netsim.sh apply 3ms 1ms + +# Remove all shaping +./netsim.sh reset + +# Show current tc settings on both supervisors +./netsim.sh status + +# Round-trip ping between supervisors +./netsim.sh ping +``` + +Changes take effect immediately and are visible in the Grafana latency panels in real time. + +## Teardown + +``` +docker compose down +``` + +Persistent volumes are not configured; all topology data and metrics are lost on teardown. This cluster is intended for development and benchmarking, not production use. diff --git a/docs/Storm-Scheduler.md b/docs/Storm-Scheduler.md index 4b9807ea2..bfb87e42e 100644 --- a/docs/Storm-Scheduler.md +++ b/docs/Storm-Scheduler.md @@ -25,3 +25,25 @@ Any topologies submitted to the cluster not listed there will not be isolated. N The isolation scheduler solves the multi-tenancy problem – avoiding resource contention between topologies – by providing full isolation between topologies. The intention is that "productionized" topologies should be listed in the isolation config, and test or in-development topologies should not. The remaining machines on the cluster serve the dual role of failover for isolated topologies and for running the non-isolated topologies. +## EvenScheduler + +`EvenScheduler` (used directly or via `DefaultScheduler`) distributes workers as evenly as possible across available supervisors. It is the default when no custom scheduler is configured and `IsolationScheduler` is not in use. + +### Idle supervisor rebalance + +*Available since Storm 3.0.0. Disabled by default.* + +When a supervisor returns from maintenance, its slots are normally left idle: each topology's desired worker count is already satisfied on the surviving supervisors, so `needsScheduling` reports nothing to do. Workers only migrate back if an operator manually rebalances or restarts every affected topology. + +Setting `nimbus.even.rebalance.idle.supervisor.enabled: true` adds an opt-in pass that relocates workers onto the returned supervisor automatically in a single scheduling round, distributed across topologies in round-robin order. + +| Key | Default | Description | +|-----|---------|-------------| +| `nimbus.even.rebalance.idle.supervisor.enabled` | `false` | Master switch. When false the pass never runs. | +| `nimbus.even.rebalance.max.free.per.topology` | `0` | Per-topology cap on workers relocated per round. `0` means no additional cap beyond the even-distribution budget. | +| `nimbus.even.rebalance.idle.supervisor.min.stable.rounds` | `3` | Number of consecutive monitoring rounds a supervisor must be seen as stable before it is eligible as a relocation target. Set to `0` to disable the flap guard. | + +A relocation is a worker JVM restart, involving brief tuple replay, JIT re-warmup, and possible windowed or stateful bolt state restore. Keep this off for topologies with stateful or windowed bolts that have a non-trivial restore cost, latency-sensitive topologies sensitive to JIT re-warmup, and clusters where supervisors are prone to flapping. + +**Scope:** `EvenScheduler` and `DefaultScheduler` only. `ResourceAwareScheduler` is unaffected. + diff --git a/docs/index.md b/docs/index.md index 8a7a31550..561d4f3e6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -57,6 +57,9 @@ We're also notifying it via annotating classes with marker interface `@Interface * [Daemon Metrics/Monitoring](ClusterMetrics.html) * [Windows users guide](windows-users-guide.html) * [Classpath handling](Classpath-handling.html) +* [Binary Distributions (full vs. lite)](Binary-distributions.html) +* [Cluster State Serialization](Cluster-State-Serialization.html) +* [Local Development Cluster](Local-dev-cluster.html) ### Intermediate @@ -71,6 +74,7 @@ We're also notifying it via annotating classes with marker interface `@Interface * [State Checkpointing](State-checkpointing.html) * [Windowing](Windowing.html) * [Joining Streams](Joins.html) +* [JitterAwareStreamGrouping](JitterAwareStreamGrouping.html) * [Blobstore(Distcache)](distcache-blobstore.html) ### Debugging
