This is an automated email from the ASF dual-hosted git repository. rzo1 pushed a commit to branch bump-java-21-and-doc-updates in repository https://gitbox.apache.org/repos/asf/storm.git
commit 738aa8e9d52b92199905beb7fccd79a2556abe4b Author: Richard Zowalla <[email protected]> AuthorDate: Sun May 10 19:02:08 2026 +0200 Bump Java to 21, update docs for Storm 3.x and javadoc.io migration - Raise compiler level to Java 21 (javaVersion in root pom.xml) - Update GitHub Actions: build on JDK 21, test matrix [21, 25] - Update dev Dockerfile to openjdk-21-jdk - Replace "Storm 2.x" with "Storm 3.x" in docs - Rewrite 91 relative (javadocs/...) links across 22 docs to absolute https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/... URLs, matching the storm-site migration to javadoc.io - Fix Docker-support.md Jekyll/Liquid rendering: wrap {{.State.Pid}} Go-template syntax in {% raw %} so Jekyll does not try to parse it - Refresh docs/README.md guidance to recommend javadoc.io URLs --- .github/workflows/maven.yaml | 8 ++--- dev-tools/docker/Dockerfile | 2 +- docs/Command-line-client.md | 2 +- docs/Common-patterns.md | 6 ++-- docs/Concepts.md | 42 +++++++++++----------- docs/Configuration.md | 10 +++--- docs/Distributed-RPC.md | 2 +- docs/Docker-support.md | 2 +- docs/Eventlogging.md | 4 +-- docs/Guaranteeing-message-processing.md | 8 ++--- docs/Hooks.md | 10 +++--- docs/Local-mode.md | 4 +-- docs/Metrics.md | 6 ++-- docs/README.md | 4 +-- docs/Running-topologies-on-a-production-cluster.md | 6 ++-- docs/SECURITY.md | 2 +- docs/Serialization-(prior-to-0.6.0).md | 4 +-- docs/Serialization.md | 2 +- docs/Setting-up-a-Storm-cluster.md | 4 +-- docs/Structure-of-the-codebase.md | 8 ++--- docs/Transactional-topologies.md | 18 +++++----- docs/Trident-API-Overview.md | 2 +- docs/Tutorial.md | 4 +-- ...standing-the-parallelism-of-a-Storm-topology.md | 16 ++++----- docs/index.md | 2 +- docs/storm-kafka-client.md | 22 ++++++------ pom.xml | 2 +- 27 files changed, 101 insertions(+), 101 deletions(-) diff --git a/.github/workflows/maven.yaml b/.github/workflows/maven.yaml index 7f168c9e0..98666dfc8 100644 --- a/.github/workflows/maven.yaml +++ b/.github/workflows/maven.yaml @@ -22,7 +22,7 @@ on: branches: [ "master" ] jobs: - # Compile once on the minimum supported JDK and share artifacts with test jobs + # Compile once on the minimum supported JDK (21) and share artifacts with test jobs build: runs-on: ubuntu-latest timeout-minutes: 30 @@ -38,11 +38,11 @@ jobs: uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 with: python-version: '3.10' - - name: Set up JDK 17 + - name: Set up JDK 21 uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0 with: distribution: temurin - java-version: 17 + java-version: 21 - name: Ensure a clean state without storm artifacts run: rm -rf ~/.m2/repository/org/apache/storm - name: Build project (compile + install, skip tests) @@ -66,7 +66,7 @@ jobs: timeout-minutes: 45 strategy: matrix: - java: [ 17, 21, 25 ] + java: [ 21, 25 ] module: [ Client, Server, Core, External, Integration-Test ] experimental: [false] fail-fast: false diff --git a/dev-tools/docker/Dockerfile b/dev-tools/docker/Dockerfile index 0a61ac33c..3691e04b4 100644 --- a/dev-tools/docker/Dockerfile +++ b/dev-tools/docker/Dockerfile @@ -27,7 +27,7 @@ RUN apt-get update && \ libssl-dev libtool pkg-config \ ruby ruby-dev \ python3.10 pip \ - openjdk-17-jdk \ + openjdk-21-jdk \ build-essential && \ rm -rf /var/lib/apt/lists/* diff --git a/docs/Command-line-client.md b/docs/Command-line-client.md index 82c1b30c3..e7faa1002 100644 --- a/docs/Command-line-client.md +++ b/docs/Command-line-client.md @@ -45,7 +45,7 @@ These commands are: Syntax: `storm jar topology-jar-path class ...` -Runs the main method of `class` with the specified arguments. The storm jars and configs in `~/.storm` are put on the classpath. The process is configured so that [StormSubmitter](javadocs/org/apache/storm/StormSubmitter.html) will upload the jar at `topology-jar-path` when the topology is submitted. +Runs the main method of `class` with the specified arguments. The storm jars and configs in `~/.storm` are put on the classpath. The process is configured so that [StormSubmitter](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/StormSubmitter.html) will upload the jar at `topology-jar-path` when the topology is submitted. When you want to ship other jars which is not included to application jar, you can pass them to `--jars` option with comma-separated string. For example, --jars "your-local-jar.jar,your-local-jar2.jar" will load your-local-jar.jar and your-local-jar2.jar. diff --git a/docs/Common-patterns.md b/docs/Common-patterns.md index 92c8c0075..d870703d3 100644 --- a/docs/Common-patterns.md +++ b/docs/Common-patterns.md @@ -23,7 +23,7 @@ If you want reliability in your data processing, the right way to do this is to If the bolt emits tuples, then you may want to use multi-anchoring to ensure reliability. It all depends on the specific application. See [Guaranteeing message processing](Guaranteeing-message-processing.html) for more details on how reliability works. ### BasicBolt -Many bolts follow a similar pattern of reading an input tuple, emitting zero or more tuples based on that input tuple, and then acking that input tuple immediately at the end of the execute method. Bolts that match this pattern are things like functions and filters. This is such a common pattern that Storm exposes an interface called [IBasicBolt](javadocs/org/apache/storm/topology/IBasicBolt.html) that automates this pattern for you. See [Guaranteeing message processing](Guaranteeing-mes [...] +Many bolts follow a similar pattern of reading an input tuple, emitting zero or more tuples based on that input tuple, and then acking that input tuple immediately at the end of the execute method. Bolts that match this pattern are things like functions and filters. This is such a common pattern that Storm exposes an interface called [IBasicBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IBasicBolt.html) that automates this pattern for you. See [...] ### In-memory caching + fields grouping combo @@ -71,11 +71,11 @@ The topology needs an extra layer of processing to aggregate the partial counts ### TimeCacheMap for efficiently keeping a cache of things that have been recently updated -You sometimes want to keep a cache in memory of items that have been recently "active" and have items that have been inactive for some time automatically expire. [TimeCacheMap](javadocs/org/apache/storm/utils/TimeCacheMap.html) is an efficient data structure for doing this and provides hooks so you can insert callbacks whenever an item is expired. +You sometimes want to keep a cache in memory of items that have been recently "active" and have items that have been inactive for some time automatically expire. [TimeCacheMap](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/utils/TimeCacheMap.html) is an efficient data structure for doing this and provides hooks so you can insert callbacks whenever an item is expired. ### CoordinatedBolt and KeyedFairBolt for Distributed RPC -When building distributed RPC applications on top of Storm, there are two common patterns that are usually needed. These are encapsulated by [CoordinatedBolt](javadocs/org/apache/storm/task/CoordinatedBolt.html) and [KeyedFairBolt](javadocs/org/apache/storm/task/KeyedFairBolt.html) which are part of the "standard library" that ships with the Storm codebase. +When building distributed RPC applications on top of Storm, there are two common patterns that are usually needed. These are encapsulated by [CoordinatedBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/CoordinatedBolt.html) and [KeyedFairBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/KeyedFairBolt.html) which are part of the "standard library" that ships with the Storm codebase. `CoordinatedBolt` wraps the bolt containing your logic and figures out when your bolt has received all the tuples for any given request. It makes heavy use of direct streams to do this. diff --git a/docs/Concepts.md b/docs/Concepts.md index fd8fa88c1..edbe07a18 100644 --- a/docs/Concepts.md +++ b/docs/Concepts.md @@ -21,7 +21,7 @@ The logic for a realtime application is packaged into a Storm topology. A Storm **Resources:** -* [TopologyBuilder](javadocs/org/apache/storm/topology/TopologyBuilder.html): use this class to construct topologies in Java +* [TopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html): use this class to construct topologies in Java * [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html) * [Local mode](Local-mode.html): Read this to learn how to develop and test topologies in local mode. @@ -29,28 +29,28 @@ The logic for a realtime application is packaged into a Storm topology. A Storm The stream is the core abstraction in Storm. A stream is an unbounded sequence of tuples that is processed and created in parallel in a distributed fashion. Streams are defined with a schema that names the fields in the stream's tuples. By default, tuples can contain integers, longs, shorts, bytes, strings, doubles, floats, booleans, and byte arrays. You can also define your own serializers so that custom types can be used natively within tuples. -Every stream is given an id when declared. Since single-stream spouts and bolts are so common, [OutputFieldsDeclarer](javadocs/org/apache/storm/topology/OutputFieldsDeclarer.html) has convenience methods for declaring a single stream without specifying an id. In this case, the stream is given the default id of "default". +Every stream is given an id when declared. Since single-stream spouts and bolts are so common, [OutputFieldsDeclarer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/OutputFieldsDeclarer.html) has convenience methods for declaring a single stream without specifying an id. In this case, the stream is given the default id of "default". **Resources:** -* [Tuple](javadocs/org/apache/storm/tuple/Tuple.html): streams are composed of tuples -* [OutputFieldsDeclarer](javadocs/org/apache/storm/topology/OutputFieldsDeclarer.html): used to declare streams and their schemas +* [Tuple](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/tuple/Tuple.html): streams are composed of tuples +* [OutputFieldsDeclarer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/OutputFieldsDeclarer.html): used to declare streams and their schemas * [Serialization](Serialization.html): Information about Storm's dynamic typing of tuples and declaring custom serializations ### Spouts A spout is a source of streams in a topology. Generally spouts will read tuples from an external source and emit them into the topology (e.g. a Kestrel queue or the Twitter API). Spouts can either be __reliable__ or __unreliable__. A reliable spout is capable of replaying a tuple if it failed to be processed by Storm, whereas an unreliable spout forgets about the tuple as soon as it is emitted. -Spouts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](javadocs/org/apache/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [SpoutOutputCollector](javadocs/org/apache/storm/spout/SpoutOutputCollector.html). +Spouts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [SpoutOutputCollector](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/spout/SpoutOutputCollector.html). The main method on spouts is `nextTuple`. `nextTuple` either emits a new tuple into the topology or simply returns if there are no new tuples to emit. It is imperative that `nextTuple` does not block for any spout implementation, because Storm calls all the spout methods on the same thread. -The other main methods on spouts are `ack` and `fail`. These are called when Storm detects that a tuple emitted from the spout either successfully completed through the topology or failed to be completed. `ack` and `fail` are only called for reliable spouts. See [the Javadoc](javadocs/org/apache/storm/spout/ISpout.html) for more information. +The other main methods on spouts are `ack` and `fail`. These are called when Storm detects that a tuple emitted from the spout either successfully completed through the topology or failed to be completed. `ack` and `fail` are only called for reliable spouts. See [the Javadoc](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/spout/ISpout.html) for more information. **Resources:** -* [IRichSpout](javadocs/org/apache/storm/topology/IRichSpout.html): this is the interface that spouts must implement. +* [IRichSpout](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IRichSpout.html): this is the interface that spouts must implement. * [Guaranteeing message processing](Guaranteeing-message-processing.html) ### Bolts @@ -59,26 +59,26 @@ All processing in topologies is done in bolts. Bolts can do anything from filter Bolts can do simple stream transformations. Doing complex stream transformations often requires multiple steps and thus multiple bolts. For example, transforming a stream of tweets into a stream of trending images requires at least two steps: a bolt to do a rolling count of retweets for each image, and one or more bolts to stream out the top X images (you can do this particular stream transformation in a more scalable way with three bolts than with two). -Bolts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](javadocs/org/apache/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [OutputCollector](javadocs/org/apache/storm/task/OutputCollector.html). +Bolts can emit more than one stream. To do so, declare multiple streams using the `declareStream` method of [OutputFieldsDeclarer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/OutputFieldsDeclarer.html) and specify the stream to emit to when using the `emit` method on [OutputCollector](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/OutputCollector.html). -When you declare a bolt's input streams, you always subscribe to specific streams of another component. If you want to subscribe to all the streams of another component, you have to subscribe to each one individually. [InputDeclarer](javadocs/org/apache/storm/topology/InputDeclarer.html) has syntactic sugar for subscribing to streams declared on the default stream id. Saying `declarer.shuffleGrouping("1")` subscribes to the default stream on component "1" and is equivalent to `declarer.s [...] +When you declare a bolt's input streams, you always subscribe to specific streams of another component. If you want to subscribe to all the streams of another component, you have to subscribe to each one individually. [InputDeclarer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/InputDeclarer.html) has syntactic sugar for subscribing to streams declared on the default stream id. Saying `declarer.shuffleGrouping("1")` subscribes to the default stream [...] -The main method in bolts is the `execute` method which takes in as input a new tuple. Bolts emit new tuples using the [OutputCollector](javadocs/org/apache/storm/task/OutputCollector.html) object. Bolts must call the `ack` method on the `OutputCollector` for every tuple they process so that Storm knows when tuples are completed (and can eventually determine that its safe to ack the original spout tuples). For the common case of processing an input tuple, emitting 0 or more tuples based o [...] +The main method in bolts is the `execute` method which takes in as input a new tuple. Bolts emit new tuples using the [OutputCollector](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/OutputCollector.html) object. Bolts must call the `ack` method on the `OutputCollector` for every tuple they process so that Storm knows when tuples are completed (and can eventually determine that its safe to ack the original spout tuples). For the common case of processing [...] -Its perfectly fine to launch new threads in bolts that do processing asynchronously. [OutputCollector](javadocs/org/apache/storm/task/OutputCollector.html) is thread-safe and can be called at any time. +Its perfectly fine to launch new threads in bolts that do processing asynchronously. [OutputCollector](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/OutputCollector.html) is thread-safe and can be called at any time. **Resources:** -* [IRichBolt](javadocs/org/apache/storm/topology/IRichBolt.html): this is general interface for bolts. -* [IBasicBolt](javadocs/org/apache/storm/topology/IBasicBolt.html): this is a convenience interface for defining bolts that do filtering or simple functions. -* [OutputCollector](javadocs/org/apache/storm/task/OutputCollector.html): bolts emit tuples to their output streams using an instance of this class +* [IRichBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IRichBolt.html): this is general interface for bolts. +* [IBasicBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IBasicBolt.html): this is a convenience interface for defining bolts that do filtering or simple functions. +* [OutputCollector](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/OutputCollector.html): bolts emit tuples to their output streams using an instance of this class * [Guaranteeing message processing](Guaranteeing-message-processing.html) ### Stream groupings Part of defining a topology is specifying for each bolt which streams it should receive as input. A stream grouping defines how that stream should be partitioned among the bolt's tasks. -There are eight built-in stream groupings in Storm, and you can implement a custom stream grouping by implementing the [CustomStreamGrouping](javadocs/org/apache/storm/grouping/CustomStreamGrouping.html) interface: +There are eight built-in stream groupings in Storm, and you can implement a custom stream grouping by implementing the [CustomStreamGrouping](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/grouping/CustomStreamGrouping.html) interface: 1. **Shuffle grouping**: Tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples. 2. **Fields grouping**: The stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id"'s may go to different tasks. @@ -86,25 +86,25 @@ There are eight built-in stream groupings in Storm, and you can implement a cust 4. **All grouping**: The stream is replicated across all the bolt's tasks. Use this grouping with care. 5. **Global grouping**: The entire stream goes to a single one of the bolt's tasks. Specifically, it goes to the task with the lowest id. 6. **None grouping**: This grouping specifies that you don't care how the stream is grouped. Currently, none groupings are equivalent to shuffle groupings. Eventually though, Storm will push down bolts with none groupings to execute in the same thread as the bolt or spout they subscribe from (when possible). -7. **Direct grouping**: This is a special kind of grouping. A stream grouped this way means that the __producer__ of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the [emitDirect](javadocs/org/apache/storm/task/OutputCollector.html#emitDirect-int-java.util.Collection-java.util.List-) methods. A bolt can get the [...] +7. **Direct grouping**: This is a special kind of grouping. A stream grouped this way means that the __producer__ of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the [emitDirect](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/OutputCollector.html#emitDirect-int-java.util.Collec [...] 8. **Local or shuffle grouping**: If the target bolt has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping. **Resources:** -* [TopologyBuilder](javadocs/org/apache/storm/topology/TopologyBuilder.html): use this class to define topologies -* [InputDeclarer](javadocs/org/apache/storm/topology/InputDeclarer.html): this object is returned whenever `setBolt` is called on `TopologyBuilder` and is used for declaring a bolt's input streams and how those streams should be grouped +* [TopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html): use this class to define topologies +* [InputDeclarer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/InputDeclarer.html): this object is returned whenever `setBolt` is called on `TopologyBuilder` and is used for declaring a bolt's input streams and how those streams should be grouped ### Reliability Storm guarantees that every spout tuple will be fully processed by the topology. It does this by tracking the tree of tuples triggered by every spout tuple and determining when that tree of tuples has been successfully completed. Every topology has a "message timeout" associated with it. If Storm fails to detect that a spout tuple has been completed within that timeout, then it fails the tuple and replays it later. -To take advantage of Storm's reliability capabilities, you must tell Storm when new edges in a tuple tree are being created and tell Storm whenever you've finished processing an individual tuple. These are done using the [OutputCollector](javadocs/org/apache/storm/task/OutputCollector.html) object that bolts use to emit tuples. Anchoring is done in the `emit` method, and you declare that you're finished with a tuple using the `ack` method. +To take advantage of Storm's reliability capabilities, you must tell Storm when new edges in a tuple tree are being created and tell Storm whenever you've finished processing an individual tuple. These are done using the [OutputCollector](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/OutputCollector.html) object that bolts use to emit tuples. Anchoring is done in the `emit` method, and you declare that you're finished with a tuple using the `ack` method. This is all explained in much more detail in [Guaranteeing message processing](Guaranteeing-message-processing.html). ### Tasks -Each spout or bolt executes as many tasks across the cluster. Each task corresponds to one thread of execution, and stream groupings define how to send tuples from one set of tasks to another set of tasks. You set the parallelism for each spout or bolt in the `setSpout` and `setBolt` methods of [TopologyBuilder](javadocs/org/apache/storm/topology/TopologyBuilder.html). +Each spout or bolt executes as many tasks across the cluster. Each task corresponds to one thread of execution, and stream groupings define how to send tuples from one set of tasks to another set of tasks. You set the parallelism for each spout or bolt in the `setSpout` and `setBolt` methods of [TopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html). ### Workers @@ -112,7 +112,7 @@ Topologies execute across one or more worker processes. Each worker process is a **Resources:** -* [Config.TOPOLOGY_WORKERS](javadocs/org/apache/storm/Config.html#TOPOLOGY_WORKERS): this config sets the number of workers to allocate for executing the topology +* [Config.TOPOLOGY_WORKERS](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_WORKERS): this config sets the number of workers to allocate for executing the topology ### Performance Tuning diff --git a/docs/Configuration.md b/docs/Configuration.md index 5b980d07c..e3ffad70a 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -5,7 +5,7 @@ documentation: true --- Storm has a variety of configurations for tweaking the behavior of nimbus, supervisors, and running topologies. Some configurations are system configurations and cannot be modified on topology by topology basis, whereas other configurations can be modified per topology. -Every configuration has a default value defined in [defaults.yaml]({{page.git-blob-base}}/conf/defaults.yaml) in the Storm codebase. You can override these configurations by defining a storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can define a topology-specific configuration that you submit along with your topology when using [StormSubmitter](javadocs/org/apache/storm/StormSubmitter.html). However, the topology-specific configuration can only override configs pr [...] +Every configuration has a default value defined in [defaults.yaml]({{page.git-blob-base}}/conf/defaults.yaml) in the Storm codebase. You can override these configurations by defining a storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can define a topology-specific configuration that you submit along with your topology when using [StormSubmitter](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/StormSubmitter.html). However, the topology-s [...] Storm 0.7.0 and onwards lets you override configuration on a per-bolt/per-spout basis. The only configurations that can be overridden this way are: @@ -23,9 +23,9 @@ The preference order for configuration values is defaults.yaml < storm.yaml < to # Bolts, Spouts, and Plugins In almost all cases configuration for a bolt or a spout should be done through setters on the bolt or spout implementation and not the topology conf. In some rare cases, it may make sense to -expose topology wide configurations that are not currently a part of [Config](javadocs/org/apache/storm/Config.html) or [DaemonConfig](javadocs/org/apache/storm/DaemonConfig.html) such as +expose topology wide configurations that are not currently a part of [Config](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html) or [DaemonConfig](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/DaemonConfig.html) such as when writing a custom scheduler or a plugin to some part of storm. In those -cases you can create your own class like Config but implements [Validated](javadocs/org/apache/storm/validation/Validated.html). Any `public static final String` field declared in this +cases you can create your own class like Config but implements [Validated](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/validation/Validated.html). Any `public static final String` field declared in this class will be treated as a config and annotations from the `org.apache.storm.validation.ConfigValidationAnnotations` class can be used to enforce what is stored in that config. To let the validator know about this class you need to treat the class like a service that will be loaded through a ServiceLoader for the Validated class and include a `META-INF/services/org.apache.storm.validation.Validated` file in your jar that holds @@ -33,8 +33,8 @@ the name of your Config class. **Resources:** -* [Config](javadocs/org/apache/storm/Config.html): a listing of client configurations as well as a helper class for creating topology specific configurations -* [DaemonConfig](javadocs/org/apache/storm/DaemonConfig.html): a listing of Storm Daemon configurations. +* [Config](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html): a listing of client configurations as well as a helper class for creating topology specific configurations +* [DaemonConfig](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/DaemonConfig.html): a listing of Storm Daemon configurations. * [defaults.yaml]({{page.git-blob-base}}/conf/defaults.yaml): the default values for all configurations * [Setting up a Storm cluster](Setting-up-a-Storm-cluster.html): explains how to create and configure a Storm cluster * [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html): lists useful configurations when running topologies on a cluster diff --git a/docs/Distributed-RPC.md b/docs/Distributed-RPC.md index 9931e692c..437ffa877 100644 --- a/docs/Distributed-RPC.md +++ b/docs/Distributed-RPC.md @@ -36,7 +36,7 @@ A client sends the DRPC server the name of the function to execute and the argum ### LinearDRPCTopologyBuilder -Storm comes with a topology builder called [LinearDRPCTopologyBuilder](javadocs/org/apache/storm/drpc/LinearDRPCTopologyBuilder.html) that automates almost all the steps involved for doing DRPC. These include: +Storm comes with a topology builder called [LinearDRPCTopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/drpc/LinearDRPCTopologyBuilder.html) that automates almost all the steps involved for doing DRPC. These include: 1. Setting up the spout 2. Returning the results to the DRPC server diff --git a/docs/Docker-support.md b/docs/Docker-support.md index 8a2695e4a..1d7c754a7 100644 --- a/docs/Docker-support.md +++ b/docs/Docker-support.md @@ -122,7 +122,7 @@ and you don't need to set them in the worker-launcher.cfg unless you need to cha You can profile your worker processes by clicking on the profiling buttons (jstack, heap, etc) on storm UI. If you have sudo permission, you can also run `sudo nsenter --target <container-pid> --pid --mount --setuid <uid> --setgid <gid>` to enter the container. Then you can run `jstack`, `jmap` etc inside the container. `<container-pid>` is the pid of the container process on the host. -`<container-pid>` can be obtained by running `sudo docker inspect --format '{{.State.Pid}}' <container-id>` command. +`<container-pid>` can be obtained by running `sudo docker inspect --format '{% raw %}{{.State.Pid}}{% endraw %}' <container-id>` command. `<uid>` and `<gid>` are the user id and group id of the container owner, respectively. ## Seccomp security profiles diff --git a/docs/Eventlogging.md b/docs/Eventlogging.md index 6fbf1ed0d..f8c9d3f39 100644 --- a/docs/Eventlogging.md +++ b/docs/Eventlogging.md @@ -107,7 +107,7 @@ To register event logger to your topology, add to your topology's configuration conf.registerEventLogger(org.apache.storm.metric.FileBasedEventLogger.class); ``` -You can refer [Config#registerEventLogger](javadocs/org/apache/storm/Config.html#registerEventLogger-java.lang.Class-) and overloaded methods from javadoc. +You can refer [Config#registerEventLogger](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#registerEventLogger-java.lang.Class-) and overloaded methods from javadoc. Otherwise edit the storm.yaml config file: @@ -119,7 +119,7 @@ topology.event.logger.register: endpoint: "event-logger.mycompany.org" ``` -When you implement your own event logger, `arguments` is passed to Map<String, Object> when [IEventLogger#prepare](javadocs/org/apache/storm/metric/IEventLogger.html#prepare-java.util.Map-java.lang.Map-org.apache.storm.task.TopologyContext-) is called. +When you implement your own event logger, `arguments` is passed to Map<String, Object> when [IEventLogger#prepare](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/metric/IEventLogger.html#prepare-java.util.Map-java.lang.Map-org.apache.storm.task.TopologyContext-) is called. Please keep in mind that EventLoggerBolt is just a kind of Bolt, so whole throughput of the topology will go down when registered event loggers cannot keep up handling incoming events, so you may want to take care of the Bolt like normal Bolt. One of idea to avoid this is making your implementation of IEventLogger as `non-blocking` fashion. diff --git a/docs/Guaranteeing-message-processing.md b/docs/Guaranteeing-message-processing.md index 4c2314cd9..fc49f3faf 100644 --- a/docs/Guaranteeing-message-processing.md +++ b/docs/Guaranteeing-message-processing.md @@ -26,11 +26,11 @@ This topology reads sentences off of a Kestrel queue, splits the sentences into  -Storm considers a tuple coming off a spout "fully processed" when the tuple tree has been exhausted and every message in the tree has been processed. A tuple is considered failed when its tree of messages fails to be fully processed within a specified timeout. This timeout can be configured on a topology-specific basis using the [Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS](javadocs/org/apache/storm/Config.html#TOPOLOGY_MESSAGE_TIMEOUT_SECS) configuration and defaults to 30 seconds. +Storm considers a tuple coming off a spout "fully processed" when the tuple tree has been exhausted and every message in the tree has been processed. A tuple is considered failed when its tree of messages fails to be fully processed within a specified timeout. This timeout can be configured on a topology-specific basis using the [Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_MESSAGE_TIMEOUT_SECS) con [...] ### What happens if a message is fully processed or fails to be fully processed? -To understand this question, let's take a look at the lifecycle of a tuple coming off of a spout. For reference, here is the interface that spouts implement (see the [Javadoc](javadocs/org/apache/storm/spout/ISpout.html) for more information): +To understand this question, let's take a look at the lifecycle of a tuple coming off of a spout. For reference, here is the interface that spouts implement (see the [Javadoc](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/spout/ISpout.html) for more information): ```java public interface ISpout extends Serializable { @@ -132,11 +132,11 @@ In contrast, bolts that do aggregations or joins may delay acking a tuple until ### How do I make my applications work correctly given that tuples can be replayed? -As always in software design, the answer is "it depends." If you really want exactly once semantics use the [Trident](Trident-tutorial.html) API. In some cases, like with a lot of analytics, dropping data is OK so disabling the fault tolerance by setting the number of acker bolts to 0 [Config.TOPOLOGY_ACKERS](javadocs/org/apache/storm/Config.html#TOPOLOGY_ACKERS). But in some cases you want to be sure that everything was processed at least once and nothing was dropped. This is especial [...] +As always in software design, the answer is "it depends." If you really want exactly once semantics use the [Trident](Trident-tutorial.html) API. In some cases, like with a lot of analytics, dropping data is OK so disabling the fault tolerance by setting the number of acker bolts to 0 [Config.TOPOLOGY_ACKERS](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_ACKERS). But in some cases you want to be sure that everything was processed at lea [...] ### How does Storm implement reliability in an efficient way? -A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker tasks for a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](javadocs/org/apache/storm/Config.html#TOPOLOGY_ACKERS). Storm defaults TOPOLOGY_ACKERS to one task per worker. +A Storm topology has a set of special "acker" tasks that track the DAG of tuples for every spout tuple. When an acker sees that a DAG is complete, it sends a message to the spout task that created the spout tuple to ack the message. You can set the number of acker tasks for a topology in the topology configuration using [Config.TOPOLOGY_ACKERS](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_ACKERS). Storm defaults TOPOLOGY_ACKERS to one t [...] The best way to understand Storm's reliability implementation is to look at the lifecycle of tuples and tuple DAGs. When a tuple is created in a topology, whether in a spout or a bolt, it is given a random 64 bit id. These ids are used by ackers to track the tuple DAG for every spout tuple. diff --git a/docs/Hooks.md b/docs/Hooks.md index c88a5632c..2b1fd3baa 100644 --- a/docs/Hooks.md +++ b/docs/Hooks.md @@ -4,14 +4,14 @@ layout: documentation documentation: true --- ## Task hooks -Storm provides hooks with which you can insert custom code to run on any number of events within Storm. You create a hook by extending the [BaseTaskHook](javadocs/org/apache/storm/hooks/BaseTaskHook.html) class and overriding the appropriate method for the event you want to catch. There are two ways to register your hook: +Storm provides hooks with which you can insert custom code to run on any number of events within Storm. You create a hook by extending the [BaseTaskHook](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/hooks/BaseTaskHook.html) class and overriding the appropriate method for the event you want to catch. There are two ways to register your hook: -1. In the open method of your spout or prepare method of your bolt using the [TopologyContext](javadocs/org/apache/storm/task/TopologyContext.html#addTaskHook) method. -2. Through the Storm configuration using the ["topology.auto.task.hooks"](javadocs/org/apache/storm/Config.html#TOPOLOGY_AUTO_TASK_HOOKS) config. These hooks are automatically registered in every spout or bolt, and are useful for doing things like integrating with a custom monitoring system. +1. In the open method of your spout or prepare method of your bolt using the [TopologyContext](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/TopologyContext.html#addTaskHook) method. +2. Through the Storm configuration using the ["topology.auto.task.hooks"](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_AUTO_TASK_HOOKS) config. These hooks are automatically registered in every spout or bolt, and are useful for doing things like integrating with a custom monitoring system. ## Worker hooks -Storm also provides worker-level hooks that are called during worker startup, before any bolts or spouts are prepared/opened. You can create such a hook by extending [BaseWorkerHook](javadocs/org/apache/storm/hooks/BaseWorkerHook) (an implementation of [IWorkerHook](javadocs/org/apache/storm/hooks/IWorkerHook.html)) and overriding the methods you want to implement. You can register your hook via `TopologyBuilder.addWorkerHook`. -The `IWorkerHook#start(Map, WorkerUserContext)` lifecycle method exposes [WorkerUserContext](javadocs/org/apache/storm/hooks/IWorkerHook.html) which provides a way to set application-level common resources via `setResource(String, Object)` method. This resource can then be retrieved by tasks, both spouts (via `open(Map, TopologyContext, SpoutOutputCollector`) and bolts (via `prepare(Map, TopologyContext, OutputCollector`), by calling `TopologyContext#getResource(String)`. +Storm also provides worker-level hooks that are called during worker startup, before any bolts or spouts are prepared/opened. You can create such a hook by extending [BaseWorkerHook](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/hooks/BaseWorkerHook) (an implementation of [IWorkerHook](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/hooks/IWorkerHook.html)) and overriding the methods you want to implement. You can register your ho [...] +The `IWorkerHook#start(Map, WorkerUserContext)` lifecycle method exposes [WorkerUserContext](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/hooks/IWorkerHook.html) which provides a way to set application-level common resources via `setResource(String, Object)` method. This resource can then be retrieved by tasks, both spouts (via `open(Map, TopologyContext, SpoutOutputCollector`) and bolts (via `prepare(Map, TopologyContext, OutputCollector`), by calling `Top [...] ## Shared State amongst components and hooks Storm provides ways to share resources across different components via the following ways: diff --git a/docs/Local-mode.md b/docs/Local-mode.md index 90890e360..b462fc4fd 100644 --- a/docs/Local-mode.md +++ b/docs/Local-mode.md @@ -27,7 +27,7 @@ try (LocalCluster cluster = new LocalCluster()) { } ``` -You can then submit topologies using the `submitTopology` method on the `LocalCluster` object. Just like the corresponding method on [StormSubmitter](javadocs/org/apache/storm/StormSubmitter.html), `submitTopology` takes a name, a topology configuration, and the topology object. You can then kill a topology using the `killTopology` method which takes the topology name as an argument. +You can then submit topologies using the `submitTopology` method on the `LocalCluster` object. Just like the corresponding method on [StormSubmitter](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/StormSubmitter.html), `submitTopology` takes a name, a topology configuration, and the topology object. You can then kill a topology using the `killTopology` method which takes the topology name as an argument. The `LocalCluster` is an `AutoCloseable` and will shut down when close is called. @@ -68,7 +68,7 @@ Or you could also modify the IDE to run "org.apache.storm.LocalCluster" instead ### Common configurations for local mode -You can see a full list of configurations [here](javadocs/org/apache/storm/Config.html). +You can see a full list of configurations [here](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html). 1. **Config.TOPOLOGY_MAX_TASK_PARALLELISM**: This config puts a ceiling on the number of threads spawned for a single component. Oftentimes production topologies have a lot of parallelism (hundreds of threads) which places unreasonable load when trying to test the topology in local mode. This config lets you easy control that parallelism. 2. **Config.TOPOLOGY_DEBUG**: When this is set to true, Storm will log a message every time a tuple is emitted from any spout or bolt. This is extremely useful for debugging.A diff --git a/docs/Metrics.md b/docs/Metrics.md index 4a2db2728..580d66e9b 100644 --- a/docs/Metrics.md +++ b/docs/Metrics.md @@ -34,7 +34,7 @@ To register metrics consumer to your topology, add to your topology's configurat conf.registerMetricsConsumer(org.apache.storm.metric.LoggingMetricsConsumer.class, 1); ``` -You can refer [Config#registerMetricsConsumer](javadocs/org/apache/storm/Config.html#registerMetricsConsumer-java.lang.Class-) and overloaded methods from javadoc. +You can refer [Config#registerMetricsConsumer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#registerMetricsConsumer-java.lang.Class-) and overloaded methods from javadoc. Otherwise edit the storm.yaml config file: @@ -56,7 +56,7 @@ Storm provides some built-in metrics consumers for you to try out to see which m Also, Storm exposes the interface [`IMetricsConsumer`]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java) for implementing Metrics Consumer so you can create custom metrics consumers and attach to their topologies, or use other great implementation of Metrics Consumers provided by Storm community. Some of examples are [versign/storm-graphite](https://github.com/verisign/storm-graphite), and [storm-metrics-statsd](https://github.com/endgameinc/s [...] -When you implement your own metrics consumer, `argument` is passed to Object when [IMetricsConsumer#prepare](javadocs/org/apache/storm/metric/api/IMetricsConsumer.html#prepare-java.util.Map-java.lang.Object-org.apache.storm.task.TopologyContext-org.apache.storm.task.IErrorReporter-) is called, so you need to infer the Java type of configured value on yaml, and do explicit type casting. +When you implement your own metrics consumer, `argument` is passed to Object when [IMetricsConsumer#prepare](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/metric/api/IMetricsConsumer.html#prepare-java.util.Map-java.lang.Object-org.apache.storm.task.TopologyContext-org.apache.storm.task.IErrorReporter-) is called, so you need to infer the Java type of configured value on yaml, and do explicit type casting. Please keep in mind that MetricsConsumerBolt is just a kind of Bolt, so whole throughput of the topology will go down when registered metrics consumers cannot keep up handling incoming metrics, so you may want to take care of those Bolts like normal Bolts. One of idea to avoid this is making your implementation of Metrics Consumer as `non-blocking` fashion. @@ -84,7 +84,7 @@ public void prepare(Map conf, TopologyContext context, OutputCollector collector } ``` -The meaning of first and second parameters are straightforward, metric name and instance of IMetric. Third parameter of [TopologyContext#registerMetric](javadocs/org/apache/storm/task/TopologyContext.html#registerMetric-java.lang.String-T-int-) is the period (seconds) to publish and reset the metric. +The meaning of first and second parameters are straightforward, metric name and instance of IMetric. Third parameter of [TopologyContext#registerMetric](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/task/TopologyContext.html#registerMetric-java.lang.String-T-int-) is the period (seconds) to publish and reset the metric. Last, let's increment the value when Bolt.execute() is executed. diff --git a/docs/README.md b/docs/README.md index e92193a7e..a06f56ddb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -106,8 +106,8 @@ If you wanted to create a maven string to tell people what dependency to use you </dependency> ``` -If you want to refer to a javadoc for the current release use a relative path. It will be in the javadocs subdirectory. +If you want to refer to a javadoc for the current release, link directly to javadoc.io (which serves Javadoc on demand from Maven Central artifacts): ``` -[TopologyBuilder](javadocs/org/apache/storm/topology/TopologyBuilder.html) +[TopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html) ``` diff --git a/docs/Running-topologies-on-a-production-cluster.md b/docs/Running-topologies-on-a-production-cluster.md index b92f3b25f..032ad1421 100644 --- a/docs/Running-topologies-on-a-production-cluster.md +++ b/docs/Running-topologies-on-a-production-cluster.md @@ -5,9 +5,9 @@ documentation: true --- Running topologies on a production cluster is similar to running in [Local mode](Local-mode.html). Here are the steps: -1) Define the topology (Use [TopologyBuilder](javadocs/org/apache/storm/topology/TopologyBuilder.html) if defining using Java) +1) Define the topology (Use [TopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html) if defining using Java) -2) Use [StormSubmitter](javadocs/org/apache/storm/StormSubmitter.html) to submit the topology to the cluster. `StormSubmitter` takes as input the name of the topology, a configuration for the topology, and the topology itself. For example: +2) Use [StormSubmitter](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/StormSubmitter.html) to submit the topology to the cluster. `StormSubmitter` takes as input the name of the topology, a configuration for the topology, and the topology itself. For example: ```java Config conf = new Config(); @@ -47,7 +47,7 @@ You can find out how to configure your `storm` client to talk to a Storm cluster ### Common configurations -There are a variety of configurations you can set per topology. A list of all the configurations you can set can be found [here](javadocs/org/apache/storm/Config.html). The ones prefixed with "TOPOLOGY" can be overridden on a topology-specific basis (the other ones are cluster configurations and cannot be overridden). Here are some common ones that are set for a topology: +There are a variety of configurations you can set per topology. A list of all the configurations you can set can be found [here](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html). The ones prefixed with "TOPOLOGY" can be overridden on a topology-specific basis (the other ones are cluster configurations and cannot be overridden). Here are some common ones that are set for a topology: 1. **Config.TOPOLOGY_WORKERS**: This sets the number of worker processes to use to execute the topology. For example, if you set this to 25, there will be 25 Java processes across the cluster executing all the tasks. If you had a combined 150 parallelism across all components in the topology, each worker process will have 6 tasks running within it as threads. 2. **Config.TOPOLOGY_ACKER_EXECUTORS**: This sets the number of executors that will track tuple trees and detect when a spout tuple has been fully processed. Ackers are an integral part of Storm's reliability model and you can read more about them on [Guaranteeing message processing](Guaranteeing-message-processing.html). By not setting this variable or setting it as null, Storm will set the number of acker executors to be equal to the number of workers configured for this topology. If t [...] diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 41cf254f7..03c71993c 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -697,7 +697,7 @@ nimbus.groups: ### DRPC -Storm provides the Access Control List for the DRPC Authorizer.Users can see [org.apache.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer](javadocs/org/apache/storm/security/auth/authorizer/DRPCSimpleACLAuthorizer.html) for more details. +Storm provides the Access Control List for the DRPC Authorizer.Users can see [org.apache.storm.security.auth.authorizer.DRPCSimpleACLAuthorizer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/security/auth/authorizer/DRPCSimpleACLAuthorizer.html) for more details. There are several DRPC ACL related configurations. diff --git a/docs/Serialization-(prior-to-0.6.0).md b/docs/Serialization-(prior-to-0.6.0).md index 0337b8cf7..8db4cd20d 100644 --- a/docs/Serialization-(prior-to-0.6.0).md +++ b/docs/Serialization-(prior-to-0.6.0).md @@ -19,7 +19,7 @@ Let's dive into Storm's API for defining custom serializations. There are two st #### Creating a serializer -Custom serializers implement the [ISerialization](javadocs/backtype/storm/serialization/ISerialization.html) interface. Implementations specify how to serialize and deserialize types into a binary format. +Custom serializers implement the [ISerialization](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/backtype/storm/serialization/ISerialization.html) interface. Implementations specify how to serialize and deserialize types into a binary format. The interface looks like this: @@ -45,6 +45,6 @@ Once you create a serializer, you need to tell Storm it exists. This is done thr Serializer registrations are done through the Config.TOPOLOGY_SERIALIZATIONS config and is simply a list of serialization class names. -Storm provides helpers for registering serializers in a topology config. The [Config](javadocs/backtype/storm/Config.html) class has a method called `addSerialization` that takes in a serializer class to add to the config. +Storm provides helpers for registering serializers in a topology config. The [Config](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/backtype/storm/Config.html) class has a method called `addSerialization` that takes in a serializer class to add to the config. There's an advanced config called Config.TOPOLOGY_SKIP_MISSING_SERIALIZATIONS. If you set this to true, Storm will ignore any serializations that are registered but do not have their code available on the classpath. Otherwise, Storm will throw errors when it can't find a serialization. This is useful if you run many topologies on a cluster that each have different serializations, but you want to declare all the serializations across all topologies in the `storm.yaml` files. diff --git a/docs/Serialization.md b/docs/Serialization.md index 56aadc689..4d2749105 100644 --- a/docs/Serialization.md +++ b/docs/Serialization.md @@ -41,7 +41,7 @@ topology.kryo.register: `com.mycompany.CustomType1` and `com.mycompany.CustomType3` will use the `FieldsSerializer`, whereas `com.mycompany.CustomType2` will use `com.mycompany.serializer.CustomType2Serializer` for serialization. -Storm provides helpers for registering serializers in a topology config. The [Config](javadocs/org/apache/storm/Config.html) class has a method called `registerSerialization` that takes in a registration to add to the config. +Storm provides helpers for registering serializers in a topology config. The [Config](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html) class has a method called `registerSerialization` that takes in a registration to add to the config. There's an advanced config called `Config.TOPOLOGY_SKIP_MISSING_KRYO_REGISTRATIONS`. If you set this to true, Storm will ignore any serializations that are registered but do not have their code available on the classpath. Otherwise, Storm will throw errors when it can't find a serialization. This is useful if you run many topologies on a cluster that each have different serializations, but you want to declare all the serializations across all topologies in the `storm.yaml` files. diff --git a/docs/Setting-up-a-Storm-cluster.md b/docs/Setting-up-a-Storm-cluster.md index 558defa5e..afb358047 100644 --- a/docs/Setting-up-a-Storm-cluster.md +++ b/docs/Setting-up-a-Storm-cluster.md @@ -29,7 +29,7 @@ A few notes about Zookeeper deployment: Next you need to install Storm's dependencies on Nimbus and the worker machines. These are: -1. Java 11+ (Apache Storm 2.x is tested through GitHub actions against Java 11, Java 17 and Java 21) +1. Java 21+ (Apache Storm 3.x is tested through GitHub actions against Java 21 and Java 25) 2. Python 3.x These are the versions of the dependencies that have been tested with Storm. Storm may or may not work with different versions of Java and/or Python. @@ -92,7 +92,7 @@ drpc.servers: ["111.222.333.44"] ### Monitoring Health of Supervisors -Storm provides a mechanism by which administrators can configure the supervisor to run administrator supplied scripts periodically to determine if a node is healthy or not. Administrators can have the supervisor determine if the node is in a healthy state by performing any checks of their choice in scripts located in storm.health.check.dir. If a script detects the node to be in an unhealthy state, it must return a non-zero exit code. In pre-Storm 2.x releases, a bug considered a script e [...] +Storm provides a mechanism by which administrators can configure the supervisor to run administrator supplied scripts periodically to determine if a node is healthy or not. Administrators can have the supervisor determine if the node is in a healthy state by performing any checks of their choice in scripts located in storm.health.check.dir. If a script detects the node to be in an unhealthy state, it must return a non-zero exit code. In pre-Storm 3.x releases, a bug considered a script e [...] If the supervisor is running with supervision "/bin/storm node-health-check" can be called to determine if the supervisor should be launched or if the node is unhealthy. diff --git a/docs/Structure-of-the-codebase.md b/docs/Structure-of-the-codebase.md index 0aaf01b0c..077943722 100644 --- a/docs/Structure-of-the-codebase.md +++ b/docs/Structure-of-the-codebase.md @@ -38,16 +38,16 @@ Note that the structure spouts also have a `ComponentCommon` field, and so spout The interfaces for Storm are generally specified as Java interfaces. The main interfaces are: -1. [IRichBolt](javadocs/org/apache/storm/topology/IRichBolt.html) -2. [IRichSpout](javadocs/org/apache/storm/topology/IRichSpout.html) -3. [TopologyBuilder](javadocs/org/apache/storm/topology/TopologyBuilder.html) +1. [IRichBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IRichBolt.html) +2. [IRichSpout](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IRichSpout.html) +3. [TopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html) The strategy for the majority of the interfaces is to: 1. Specify the interface using a Java interface 2. Provide a base class that provides default implementations when appropriate -You can see this strategy at work with the [BaseRichSpout](javadocs/org/apache/storm/topology/base/BaseRichSpout.html) class. +You can see this strategy at work with the [BaseRichSpout](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/base/BaseRichSpout.html) class. Spouts and bolts are serialized into the Thrift definition of the topology as described above. diff --git a/docs/Transactional-topologies.md b/docs/Transactional-topologies.md index 612ac32f1..e05360124 100644 --- a/docs/Transactional-topologies.md +++ b/docs/Transactional-topologies.md @@ -81,7 +81,7 @@ Finally, another thing to note is that transactional topologies require a source ## The basics through example -You build transactional topologies by using [TransactionalTopologyBuilder](javadocs/org/apache/storm/transactional/TransactionalTopologyBuilder.html). Here's the transactional topology definition for a topology that computes the global count of tuples from the input stream. This code comes from [TransactionalGlobalCount]({{page.git-blob-base}}/examples/storm-starter/src/jvm/org/apache/storm/starter/TransactionalGlobalCount.java) in storm-starter. +You build transactional topologies by using [TransactionalTopologyBuilder](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/transactional/TransactionalTopologyBuilder.html). Here's the transactional topology definition for a topology that computes the global count of tuples from the input stream. This code comes from [TransactionalGlobalCount]({{page.git-blob-base}}/examples/storm-starter/src/jvm/org/apache/storm/starter/TransactionalGlobalCount.java) in storm- [...] ```java MemoryTransactionalSpout spout = new MemoryTransactionalSpout(DATA, new Fields("word"), PARTITION_TAKE_PER_BATCH); @@ -132,7 +132,7 @@ public static class BatchCount extends BaseBatchBolt { A new instance of this object is created for every batch that's being processed. The actual bolt this runs within is called [BatchBoltExecutor](https://github.com/apache/storm/blob/0.7.0/src/jvm/org/apache/storm/coordination/BatchBoltExecutor.java) and manages the creation and cleanup for these objects. -The `prepare` method parameterizes this batch bolt with the Storm config, the topology context, an output collector, and the id for this batch of tuples. In the case of transactional topologies, the id will be a [TransactionAttempt](javadocs/org/apache/storm/transactional/TransactionAttempt.html) object. The batch bolt abstraction can be used in Distributed RPC as well which uses a different type of id for the batches. `BatchBolt` can actually be parameterized with the type of the id, so [...] +The `prepare` method parameterizes this batch bolt with the Storm config, the topology context, an output collector, and the id for this batch of tuples. In the case of transactional topologies, the id will be a [TransactionAttempt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/transactional/TransactionAttempt.html) object. The batch bolt abstraction can be used in Distributed RPC as well which uses a different type of id for the batches. `BatchBolt` can act [...] ```java public abstract class BaseTransactionalBolt extends BaseBatchBolt<TransactionAttempt> { @@ -211,9 +211,9 @@ This section outlines the different pieces of the transactional topology API. There are three kinds of bolts possible in a transactional topology: -1. [BasicBolt](javadocs/org/apache/storm/topology/base/BaseBasicBolt.html): This bolt doesn't deal with batches of tuples and just emits tuples based on a single tuple of input. -2. [BatchBolt](javadocs/org/apache/storm/topology/base/BaseBatchBolt.html): This bolt processes batches of tuples. `execute` is called for each tuple, and `finishBatch` is called when the batch is complete. -3. BatchBolt's that are marked as committers: The only difference between this bolt and a regular batch bolt is when `finishBatch` is called. A committer bolt has `finishedBatch` called during the commit phase. The commit phase is guaranteed to occur only after all prior batches have successfully committed, and it will be retried until all bolts in the topology succeed the commit for the batch. There are two ways to make a `BatchBolt` a committer, by having the `BatchBolt` implement the [...] +1. [BasicBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/base/BaseBasicBolt.html): This bolt doesn't deal with batches of tuples and just emits tuples based on a single tuple of input. +2. [BatchBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/base/BaseBatchBolt.html): This bolt processes batches of tuples. `execute` is called for each tuple, and `finishBatch` is called when the batch is complete. +3. BatchBolt's that are marked as committers: The only difference between this bolt and a regular batch bolt is when `finishBatch` is called. A committer bolt has `finishedBatch` called during the commit phase. The commit phase is guaranteed to occur only after all prior batches have successfully committed, and it will be retried until all bolts in the topology succeed the commit for the batch. There are two ways to make a `BatchBolt` a committer, by having the `BatchBolt` implement the [...] #### Processing phase vs. commit phase in bolts @@ -237,7 +237,7 @@ Notice that you don't have to do any acking or anchoring when working with trans #### Failing a transaction -When using regular bolts, you can call the `fail` method on `OutputCollector` to fail the tuple trees of which that tuple is a member. Since transactional topologies hide the acking framework from you, they provide a different mechanism to fail a batch (and cause the batch to be replayed). Just throw a [FailedException](javadocs/org/apache/storm/topology/FailedException.html). Unlike regular exceptions, this will only cause that particular batch to replay and will not crash the process. +When using regular bolts, you can call the `fail` method on `OutputCollector` to fail the tuple trees of which that tuple is a member. Since transactional topologies hide the acking framework from you, they provide a different mechanism to fail a batch (and cause the batch to be replayed). Just throw a [FailedException](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/FailedException.html). Unlike regular exceptions, this will only cause that particula [...] ### Transactional spout @@ -251,11 +251,11 @@ The coordinator on the left is a regular Storm spout that emits a tuple whenever The need to be idempotent with respect to the tuples it emits requires a `TransactionalSpout` to store a small amount of state. The state is stored in Zookeeper. -The details of implementing a `TransactionalSpout` are in [the Javadoc](javadocs/org/apache/storm/transactional/ITransactionalSpout.html). +The details of implementing a `TransactionalSpout` are in [the Javadoc](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/transactional/ITransactionalSpout.html). #### Partitioned Transactional Spout -A common kind of transactional spout is one that reads the batches from a set of partitions across many queue brokers. For example, this is how [KafkaTridentSpoutTransactional]({{page.git-tree-base}}/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutTransactional.java) works. An `IPartitionedTransactionalSpout` automates the bookkeeping work of managing the state for each partition to ensure idempotent replayability. See [the Javadoc](javadoc [...] +A common kind of transactional spout is one that reads the batches from a set of partitions across many queue brokers. For example, this is how [KafkaTridentSpoutTransactional]({{page.git-tree-base}}/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutTransactional.java) works. An `IPartitionedTransactionalSpout` automates the bookkeeping work of managing the state for each partition to ensure idempotent replayability. See [the Javadoc](https:/ [...] ### Configuration @@ -325,7 +325,7 @@ In this scenario, tuples 41-50 are skipped. By failing all subsequent transactio By failing all subsequent transactions on failure, no tuples are skipped. This also shows that a requirement of transactional spouts is that they always emit where the last transaction left off. -A non-idempotent transactional spout is more concisely referred to as an "OpaqueTransactionalSpout" (opaque is the opposite of idempotent). [IOpaquePartitionedTransactionalSpout](javadocs/org/apache/storm/transactional/partitioned/IOpaquePartitionedTransactionalSpout.html) is an interface for implementing opaque partitioned transactional spouts, of which [KafkaTridentSpoutOpaque]({{page.git-tree-base}}/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/Trident/KafkaTr [...] +A non-idempotent transactional spout is more concisely referred to as an "OpaqueTransactionalSpout" (opaque is the opposite of idempotent). [IOpaquePartitionedTransactionalSpout](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/transactional/partitioned/IOpaquePartitionedTransactionalSpout.html) is an interface for implementing opaque partitioned transactional spouts, of which [KafkaTridentSpoutOpaque]({{page.git-tree-base}}/external/storm-kafka-client/src/main [...] ## Implementation diff --git a/docs/Trident-API-Overview.md b/docs/Trident-API-Overview.md index b9f074c22..0a560711d 100644 --- a/docs/Trident-API-Overview.md +++ b/docs/Trident-API-Overview.md @@ -409,7 +409,7 @@ Example of using `HBaseWindowStoreFactory` for windowing can be seen below. ``` -Detailed description of all the above APIs in this section can be found [here](javadocs/org/apache/storm/trident/Stream.html) +Detailed description of all the above APIs in this section can be found [here](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/trident/Stream.html) #### Example applications Example applications of these APIs are located at [TridentHBaseWindowingStoreTopology]({{page.git-blob-base}}/examples/storm-starter/src/jvm/org/apache/storm/starter/trident/TridentHBaseWindowingStoreTopology.java) diff --git a/docs/Tutorial.md b/docs/Tutorial.md index c4ebffaec..8a02fd899 100644 --- a/docs/Tutorial.md +++ b/docs/Tutorial.md @@ -103,11 +103,11 @@ This topology contains a spout and two bolts. The spout emits words, and each bo This code defines the nodes using the `setSpout` and `setBolt` methods. These methods take as input a user-specified id, an object containing the processing logic, and the amount of parallelism you want for the node. In this example, the spout is given id "words" and the bolts are given ids "exclaim1" and "exclaim2". -The object containing the processing logic implements the [IRichSpout](javadocs/org/apache/storm/topology/IRichSpout.html) interface for spouts and the [IRichBolt](javadocs/org/apache/storm/topology/IRichBolt.html) interface for bolts. +The object containing the processing logic implements the [IRichSpout](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IRichSpout.html) interface for spouts and the [IRichBolt](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/IRichBolt.html) interface for bolts. The last parameter, how much parallelism you want for the node, is optional. It indicates how many threads should execute that component across the cluster. If you omit it, Storm will only allocate one thread for that node. -`setBolt` returns an [InputDeclarer](javadocs/org/apache/storm/topology/InputDeclarer.html) object that is used to define the inputs to the Bolt. Here, component "exclaim1" declares that it wants to read all the tuples emitted by component "words" using a shuffle grouping, and component "exclaim2" declares that it wants to read all the tuples emitted by component "exclaim1" using a shuffle grouping. "shuffle grouping" means that tuples should be randomly distributed from the input tasks [...] +`setBolt` returns an [InputDeclarer](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/InputDeclarer.html) object that is used to define the inputs to the Bolt. Here, component "exclaim1" declares that it wants to read all the tuples emitted by component "words" using a shuffle grouping, and component "exclaim2" declares that it wants to read all the tuples emitted by component "exclaim1" using a shuffle grouping. "shuffle grouping" means that tuples sh [...] If you wanted component "exclaim2" to read all the tuples emitted by both component "words" and component "exclaim1", you would write component "exclaim2"'s definition like this: diff --git a/docs/Understanding-the-parallelism-of-a-Storm-topology.md b/docs/Understanding-the-parallelism-of-a-Storm-topology.md index c48102de2..3dbbd59ed 100644 --- a/docs/Understanding-the-parallelism-of-a-Storm-topology.md +++ b/docs/Understanding-the-parallelism-of-a-Storm-topology.md @@ -30,25 +30,25 @@ The following sections give an overview of the various configuration options and ### Number of worker processes * Description: How many worker processes to create _for the topology_ across machines in the cluster. -* Configuration option: [TOPOLOGY_WORKERS](javadocs/org/apache/storm/Config.html#TOPOLOGY_WORKERS) +* Configuration option: [TOPOLOGY_WORKERS](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_WORKERS) * How to set in your code (examples): - * [Config#setNumWorkers](javadocs/org/apache/storm/Config.html) + * [Config#setNumWorkers](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html) ### Number of executors (threads) * Description: How many executors to spawn _per component_. * Configuration option: None (pass ``parallelism_hint`` parameter to ``setSpout`` or ``setBolt``) * How to set in your code (examples): - * [TopologyBuilder#setSpout()](javadocs/org/apache/storm/topology/TopologyBuilder.html) - * [TopologyBuilder#setBolt()](javadocs/org/apache/storm/topology/TopologyBuilder.html) + * [TopologyBuilder#setSpout()](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html) + * [TopologyBuilder#setBolt()](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/TopologyBuilder.html) * Note that as of Storm 0.8 the ``parallelism_hint`` parameter now specifies the initial number of executors (not tasks!) for that bolt. ### Number of tasks * Description: How many tasks to create _per component_. -* Configuration option: [TOPOLOGY_TASKS](javadocs/org/apache/storm/Config.html#TOPOLOGY_TASKS) +* Configuration option: [TOPOLOGY_TASKS](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_TASKS) * How to set in your code (examples): - * [ComponentConfigurationDeclarer#setNumTasks()](javadocs/org/apache/storm/topology/ComponentConfigurationDeclarer.html) + * [ComponentConfigurationDeclarer#setNumTasks()](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/topology/ComponentConfigurationDeclarer.html) Here is an example code snippet to show these settings in practice: @@ -91,7 +91,7 @@ StormSubmitter.submitTopology( And of course Storm comes with additional configuration settings to control the parallelism of a topology, including: -* [TOPOLOGY_MAX_TASK_PARALLELISM](javadocs/org/apache/storm/Config.html#TOPOLOGY_MAX_TASK_PARALLELISM): This setting puts a ceiling on the number of executors that can be spawned for a single component. It is typically used during testing to limit the number of threads spawned when running a topology in local mode. You can set this option via e.g. [Config#setMaxTaskParallelism()](javadocs/org/apache/storm/Config.html#setMaxTaskParallelism(int)). +* [TOPOLOGY_MAX_TASK_PARALLELISM](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/Config.html#TOPOLOGY_MAX_TASK_PARALLELISM): This setting puts a ceiling on the number of executors that can be spawned for a single component. It is typically used during testing to limit the number of threads spawned when running a topology in local mode. You can set this option via e.g. [Config#setMaxTaskParallelism()](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/ [...] ## How to change the parallelism of a running topology @@ -119,5 +119,5 @@ $ storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10 * [Running topologies on a production cluster](Running-topologies-on-a-production-cluster.html) * [Local mode](Local-mode.html) * [Tutorial](Tutorial.html) -* [Storm API documentation](javadocs/), most notably the class ``Config`` +* [Storm API documentation](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/), most notably the class ``Config`` diff --git a/docs/index.md b/docs/index.md index 6c3dbc216..8a7a31550 100644 --- a/docs/index.md +++ b/docs/index.md @@ -5,7 +5,7 @@ documentation: true --- ### Basics of Storm -* [Javadoc](javadocs/index.html) +* [Javadoc](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/index.html) * [Tutorial](Tutorial.html) * [Concepts](Concepts.html) * [Scheduler](Storm-Scheduler.html) diff --git a/docs/storm-kafka-client.md b/docs/storm-kafka-client.md index 73c88c72d..8504ed89f 100644 --- a/docs/storm-kafka-client.md +++ b/docs/storm-kafka-client.md @@ -152,9 +152,9 @@ of Java generics. The deserializers can be specified via the consumer propertie There are a few key configs to pay attention to. `setFirstPollOffsetStrategy` allows you to set where to start consuming data from. This is used both in case of failure recovery and starting the spout -for the first time. The allowed values are listed in the [FirstPollOffsetStrategy javadocs](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.FirstPollOffsetStrategy.html). +for the first time. The allowed values are listed in the [FirstPollOffsetStrategy javadocs](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.FirstPollOffsetStrategy.html). -`setProcessingGuarantee` lets you configure what processing guarantees the spout will provide. This affects how soon consumed offsets can be committed, and the frequency of commits. See the [ProcessingGuarantee javadoc](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.ProcessingGuarantee.html) for details. +`setProcessingGuarantee` lets you configure what processing guarantees the spout will provide. This affects how soon consumed offsets can be committed, and the frequency of commits. See the [ProcessingGuarantee javadoc](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.ProcessingGuarantee.html) for details. `setRecordTranslator` allows you to modify how the spout converts a Kafka Consumer Record into a Tuple, and which stream that tuple will be published into. By default the "topic", "partition", "offset", "key", and "value" will be emitted to the "default" stream. If you want to output entries to different @@ -279,7 +279,7 @@ When selecting a kafka client version, you should ensure - # Kafka Spout Performance Tuning -The Kafka spout provides two internal parameters to control its performance. The parameters can be set using the [setOffsetCommitPeriodMs](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setOffsetCommitPeriodMs-long-) and [setMaxUncommittedOffsets](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setMaxUncommittedOffsets-int-) methods. +The Kafka spout provides two internal parameters to control its performance. The parameters can be set using the [setOffsetCommitPeriodMs](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setOffsetCommitPeriodMs-long-) and [setMaxUncommittedOffsets](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setMaxUncommittedOffsets-int-) methods. * "offset.commit.period.ms" controls how often the spout commits to Kafka * "max.uncommitted.offsets" controls how many offsets can be pending commit before another poll can take place @@ -289,7 +289,7 @@ The [Kafka consumer config] (http://kafka.apache.org/documentation.html#consumer * “fetch.min.bytes” * “fetch.max.wait.ms” -* [Kafka Consumer](http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html) instance poll timeout, which is specified for each Kafka spout using the [setPollTimeoutMs](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setPollTimeoutMs-long-) method. +* [Kafka Consumer](http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html) instance poll timeout, which is specified for each Kafka spout using the [setPollTimeoutMs](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setPollTimeoutMs-long-) method. <br/> Depending on the structure of your Kafka cluster, distribution of the data, and availability of data to poll, these parameters will have to be configured appropriately. Please refer to the Kafka documentation on Kafka parameter tuning. @@ -329,15 +329,15 @@ and Kafka 0.10.1.0 [ConsumerConfig](https://kafka.apache.org/0101/javadoc/index. | SpoutConfig | KafkaSpoutConfig/ConsumerConfig | KafkaSpoutConfig Usage | | ------------- | ------------------------------- | ---------------------- | -| **Setting:** `startOffsetTime`<br><br> **Default:** `EarliestTime`<br>________________________________________________ <br> **Setting:** `forceFromStart` <br><br> **Default:** `false` <br><br> `startOffsetTime` & `forceFromStart` together determine the starting offset. `forceFromStart` determines whether the Zookeeper offset is ignored. `startOffsetTime` sets the timestamp that determines the beginning offset, in case there is no offset in Zookeeper, or the Zookeeper offset is ignored [...] -| **Setting:** `scheme`<br><br> The interface that specifies how a `ByteBuffer` from a Kafka topic is transformed into Storm tuple <br>**Default:** `RawMultiScheme` | **Setting:** [`Deserializers`](https://kafka.apache.org/11/javadoc/org/apache/kafka/common/serialization/Deserializer.html)| [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, <deserializer-class>)`](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setProp-java.lang.Strin [...] -| **Setting:** `fetchSizeBytes`<br><br> Message fetch size -- the number of bytes to attempt to fetch in one request to a Kafka server <br> **Default:** `1MB` | **Setting:** [`max.partition.fetch.bytes`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) | [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, <int-value>)`](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setProp-java.lang.String-java.lang.Object-)| -| **Setting:** `bufferSizeBytes`<br><br> Buffer size (in bytes) for network requests. The buffer size which consumer has for pulling data from producer <br> **Default:** `1MB`| **Setting:** [`receive.buffer.bytes`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) | [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.RECEIVE_BUFFER_CONFIG, <int-value>)`](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setProp-java.lang.String-java.lang.Object-)| +| **Setting:** `startOffsetTime`<br><br> **Default:** `EarliestTime`<br>________________________________________________ <br> **Setting:** `forceFromStart` <br><br> **Default:** `false` <br><br> `startOffsetTime` & `forceFromStart` together determine the starting offset. `forceFromStart` determines whether the Zookeeper offset is ignored. `startOffsetTime` sets the timestamp that determines the beginning offset, in case there is no offset in Zookeeper, or the Zookeeper offset is ignored [...] +| **Setting:** `scheme`<br><br> The interface that specifies how a `ByteBuffer` from a Kafka topic is transformed into Storm tuple <br>**Default:** `RawMultiScheme` | **Setting:** [`Deserializers`](https://kafka.apache.org/11/javadoc/org/apache/kafka/common/serialization/Deserializer.html)| [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, <deserializer-class>)`](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/Kaf [...] +| **Setting:** `fetchSizeBytes`<br><br> Message fetch size -- the number of bytes to attempt to fetch in one request to a Kafka server <br> **Default:** `1MB` | **Setting:** [`max.partition.fetch.bytes`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) | [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, <int-value>)`](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#s [...] +| **Setting:** `bufferSizeBytes`<br><br> Buffer size (in bytes) for network requests. The buffer size which consumer has for pulling data from producer <br> **Default:** `1MB`| **Setting:** [`receive.buffer.bytes`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) | [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.RECEIVE_BUFFER_CONFIG, <int-value>)`](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#s [...] | **Setting:** `socketTimeoutMs`<br><br> **Default:** `10000` | **N/A** || -| **Setting:** `useStartOffsetTimeIfOffsetOutOfRange`<br><br> **Default:** `true` | **Setting:** [`auto.offset.reset`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) <br><br> **Default:** Note that the default value for `auto.offset.reset` is `earliest` if you have [`ProcessingGuarantee`](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.ProcessingGuarantee.html) set to `AT_LEAST_ONCE`, but the default value is `latest` otherwise.| [`<KafkaSpoutConfig-Builder>.set [...] -| **Setting:** `fetchMaxWait`<br><br> Maximum time in ms to wait for the response <br> **Default:** `10000` | **Setting:** [`fetch.max.wait.ms`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) | [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, <value>)`](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setProp-java.lang.String-java.lang.Object-)| +| **Setting:** `useStartOffsetTimeIfOffsetOutOfRange`<br><br> **Default:** `true` | **Setting:** [`auto.offset.reset`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) <br><br> **Default:** Note that the default value for `auto.offset.reset` is `earliest` if you have [`ProcessingGuarantee`](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.ProcessingGuarantee.html) set to `AT_LEAST_ONCE`, but the default value is `la [...] +| **Setting:** `fetchMaxWait`<br><br> Maximum time in ms to wait for the response <br> **Default:** `10000` | **Setting:** [`fetch.max.wait.ms`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs) | [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, <value>)`](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setProp-java.lang.String-java.lang.Object-)| | **Setting:** `maxOffsetBehind`<br><br> Specifies how long a spout attempts to retry the processing of a failed tuple. One of the scenarios is when a failing tuple's offset is more than `maxOffsetBehind` behind the acked offset, the spout stops retrying the tuple.<br>**Default:** `LONG.MAX_VALUE`| **N/A** || -| **Setting:** `clientId`| **Setting:** [`client.id`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs)| [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.CLIENT_ID_CONFIG, <String>)`](javadocs/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setProp-java.lang.String-java.lang.Object-)| +| **Setting:** `clientId`| **Setting:** [`client.id`](http://kafka.apache.org/10/documentation.html#newconsumerconfigs)| [`<KafkaSpoutConfig-Builder>.setProp(ConsumerConfig.CLIENT_ID_CONFIG, <String>)`](https://javadoc.io/doc/org.apache.storm/storm-client/3.0.0/org/apache/storm/kafka/spout/KafkaSpoutConfig.Builder.html#setProp-java.lang.String-java.lang.Object-)| If you are using this table to upgrade your topology to use `storm-kafka-client` instead of `storm-kafka`, then you will also need to migrate the consumer offsets from ZooKeeper to Kafka broker. Use [`storm-kafka-migration`](https://github.com/apache/storm/tree/master/external/storm-kafka-migration) tool to migrate the Kafka consumer offsets. diff --git a/pom.xml b/pom.xml index 4babe149e..824b4ab3c 100644 --- a/pom.xml +++ b/pom.xml @@ -68,7 +68,7 @@ </issueManagement> <properties> - <javaVersion>17</javaVersion> + <javaVersion>21</javaVersion> <maven.javadoc.failOnError>false</maven.javadoc.failOnError> <maven.javadoc.failOnWarnings>false</maven.javadoc.failOnWarnings>
