This is an automated email from the ASF dual-hosted git repository.
lucasbru pushed a commit to branch markdown
in repository https://gitbox.apache.org/repos/asf/kafka-site.git
The following commit(s) were added to refs/heads/markdown by this push:
new 94f3446554 Update 4.2 Kafka Streams documentation (#818)
94f3446554 is described below
commit 94f3446554c55b293268b3e29b14c947c60f9c8a
Author: Lucas Brutschy <[email protected]>
AuthorDate: Wed Feb 25 12:57:24 2026 +0100
Update 4.2 Kafka Streams documentation (#818)
* Add Kafka Streams rebalance protocol and kafka-streams-group-sh docs for
4.2
Add new documentation pages for the Kafka Streams rebalance protocol and
the kafka-streams-group.sh tool. These were added to the Kafka 4.2 branch
but were missing from the website.
* Fix broken documentation links for streams rebalance protocol
Use direct paths instead of going through the /documentation/ JavaScript
redirect, and use absolute versioned paths instead of relative links.
This fixes broken links reported in PR review.
---
.../developer-guide/streams-rebalance-protocol.md | 26 +++
content/en/42/getting-started/upgrade.md | 9 +-
.../42/streams/developer-guide/app-reset-tool.md | 2 +-
.../developer-guide/kafka-streams-group-sh.md | 183 +++++++++++++++
.../developer-guide/streams-rebalance-protocol.md | 255 +++++++++++++++++++++
content/en/42/streams/upgrade-guide.md | 19 +-
6 files changed, 487 insertions(+), 7 deletions(-)
diff --git
a/content/en/42/documentation/streams/developer-guide/streams-rebalance-protocol.md
b/content/en/42/documentation/streams/developer-guide/streams-rebalance-protocol.md
new file mode 100644
index 0000000000..2e7bfb5ce4
--- /dev/null
+++
b/content/en/42/documentation/streams/developer-guide/streams-rebalance-protocol.md
@@ -0,0 +1,26 @@
+---
+title: "Documentation Redirect"
+robots: "noindex"
+_build:
+ list: false
+---
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+
+{{< doc-redirect >}}
diff --git a/content/en/42/getting-started/upgrade.md
b/content/en/42/getting-started/upgrade.md
index a02fe9f0da..6b40e8cb70 100644
--- a/content/en/42/getting-started/upgrade.md
+++ b/content/en/42/getting-started/upgrade.md
@@ -34,6 +34,7 @@ type: docs
* The `--max-partition-memory-bytes` option in `kafka-console-producer` is
deprecated and will be removed in Kafka 5.0. Please use `--batch-size` instead.
* Queues for Kafka ([KIP-932](https://cwiki.apache.org/confluence/x/4hA0Dw))
is production-ready in Apache Kafka 4.2. This feature introduces a new kind of
group called share groups, as an alternative to consumer groups. Consumers in a
share group cooperatively consume records from topics, without assigning each
partition to just one consumer. Share groups also introduce per-record
acknowledgement and counting of delivery attempts. Use share groups in cases
where records are processed [...]
+ * The Streams Rebalance Protocol
([KIP-1071](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol))
is now production-ready for its core feature set. This broker-driven
rebalancing system designed specifically for Kafka Streams applications
provides faster, more stable rebalances and better observability. For more
information about the supported feature set, usage, and migration, please refer
to the [Streams developer guide](/{version}/streams/develo [...]
* The `org.apache.kafka.common.header.internals.RecordHeader` class has been
updated to be read thread-safe. See
[KIP-1205](https://cwiki.apache.org/confluence/x/nYmhFg) for details. In other
words, each individual `Header` object within a `ConsumerRecord`'s `headers`
can now be safely read from multiple threads concurrently.
* The `org.apache.kafka.disallowed.login.modules` config was deprecated.
Please use the `org.apache.kafka.allowed.login.modules` instead.
* The `remote.log.manager.thread.pool.size` config was deprecated. Please
use the `remote.log.manager.follower.thread.pool.size` instead.
@@ -84,8 +85,7 @@ For further details, please refer to
[KIP-1120](https://cwiki.apache.org/conflue
* Deprecated `org.apache.kafka.streams.KafkaStreams$CloseOptions` and its
related methods, such as
`KafkaStreams#close(org.apache.kafka.streams.KafkaStreams$CloseOptions)`. As a
replacement, please use `org.apache.kafka.streams.CloseOptions` and
`KafkaStreams#close(org.apache.kafka.streams.CloseOptions)`. For further
details, please refer to
[KIP-1153](https://cwiki.apache.org/confluence/x/QAq9F).
* A new implementation of `ConnectorClientConfigOverridePolicy`,
`AllowlistConnectorClientConfigOverridePolicy`, has been added. This enables
specifying the configurations that connectors can override via
`connector.client.config.override.allowlist`. From Kafka 5.0.0, this will be
the default
[connector.client.config.override.policy](documentation/#connectconfigs_connector.client.config.override.policy)
policy. The `PrincipalConnectorClientConfigOverridePolicy` policy is now
deprecated [...]
* It is now possible to specify the start time for a Kafka Streams
punctuation, instead of relying on the non-deterministic time when you register
it. For further details, please refer to
[KIP-1146](https://cwiki.apache.org/confluence/x/9QqWF).
- * Added an optional `--node-id` flag to the `FeatureCommand` command. It
specifies the node to describe. If not provided, an arbitrary node is used.
-
+ * Added an optional `--node-id` flag to the `FeatureCommand` command. It
specifies the node to describe. If not provided, an arbitrary node is used.
## Upgrading to 4.1.0
@@ -110,7 +110,10 @@ For further details, please refer to
[KIP-1120](https://cwiki.apache.org/conflue
* **Admin**
* The `listConsumerGroups()` and
`listConsumerGroups(ListConsumerGroupsOptions)` methods in `Admin` are
deprecated, and will be removed in the next major version. Use
`Admin.listGroups(ListGroupsOptions.forConsumerGroups())` instead.
* **Kafka Streams**
- * The `window.size.ms` and `window.inner.serde.class` in `StreamsConfig`
are deprecated. Use the corresponding string constants defined in
`TimeWindowedSerializer`, `TimeWindowedDeserializer`,
`SessionWindowedSerializer` and `SessionWindowedDeserializer` instead.
+ * Early Access for the Streams rebalance protocol. Following KIP-848,
[KIP-1071](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol)
adds broker side task assignment for Kafka Streams applications.
+ This feature is in Early Access, disabled by default, and not for
production use-cases.
+ We encourage users to try out the new "streams" group feature in lower
environments using "throw away" broker cluster,
+ and provide early feedback.
diff --git a/content/en/42/streams/developer-guide/app-reset-tool.md
b/content/en/42/streams/developer-guide/app-reset-tool.md
index c2d3b26d53..38c4eee681 100644
--- a/content/en/42/streams/developer-guide/app-reset-tool.md
+++ b/content/en/42/streams/developer-guide/app-reset-tool.md
@@ -56,7 +56,7 @@ Prerequisites
# Step 1: Run the application reset tool
-If you are using **streams rebalance protocol** (available since AK 4.2), use
the [Streams groups CLI](kafka-streams-group-sh.html#reset-offsets).
+If you are using **streams rebalance protocol** (available since AK 4.2), use
the [Streams groups
CLI](/{version}/streams/developer-guide/kafka-streams-group-sh#reset-offsets).
If you are using **classic rebalance protocol** , run the classic application
reset tool as described below.
diff --git a/content/en/42/streams/developer-guide/kafka-streams-group-sh.md
b/content/en/42/streams/developer-guide/kafka-streams-group-sh.md
new file mode 100644
index 0000000000..61c217f35d
--- /dev/null
+++ b/content/en/42/streams/developer-guide/kafka-streams-group-sh.md
@@ -0,0 +1,183 @@
+---
+title: Kafka Streams Groups Tool
+type: docs
+description:
+weight: 15
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+---
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Use `kafka-streams-groups.sh` to manage **Streams groups** for the Streams
Rebalance Protocol (KIP‑1071): list and describe groups, inspect members and
offsets/lag, reset or delete offsets for input topics, and delete groups
(optionally including internal topics).
+
+
+# Overview
+
+A **Streams group** is a broker‑coordinated group type for Kafka Streams that
uses Streams‑specific RPCs and metadata, distinct from classic consumer groups.
The CLI surfaces Streams‑specific states, assignments, and input‑topic offsets
to simplify visibility and administration.
+
+**Use with care:** Mutating operations (offset resets/deletes, group deletion)
affect how applications will reprocess data when restarted. Always preview with
\--dry-run before executing and ensure application instances are
stopped/inactive and the group is empty before executing the command.
+
+# What the Streams Groups tool does
+
+ * **List Streams groups** across a cluster and display or filter by group
state (Empty, Not Ready, Assigning, Reconciling, Stable, Dead).
+ * **Describe a Streams group** and show:
+ * Group state, group epoch, target assignment epoch (with `--state`,
`--verbose` for additional details).
+ * Per‑member info such as epochs, current vs target assignments, and
whether a member still uses the classic protocol (with `--members` and
`--verbose`).
+ * Input‑topic offsets and lag (with `--offsets`), to understand how far
behind processing is.
+ * **Reset input‑topic offsets** for a Streams group to control reprocessing
boundaries using precise specifiers (earliest, latest, to‑offset, to‑datetime,
by‑duration, shift‑by, from‑file). Requires `--dry-run` or `--execute` and
inactive instances.
+ * **Delete offsets** for input topics to force re‑consumption on next start.
+ * **Delete a Streams group** to clean up broker‑side Streams metadata
(offsets, topology, assignments). Optionally delete all, or a subset of,
**internal topics** at the same time using `--internal-topics`.
+
+
+
+# Usage
+
+The script is located in `bin/kafka-streams-groups.sh` and connects to your
cluster via `--bootstrap-server`. For secured clusters, pass AdminClient
properties using `--command-config`.
+
+
+ $ kafka-streams-groups.sh --bootstrap-server <host:port> [COMMAND]
[OPTIONS]
+
+**Note:** `kafka-streams-groups.sh` complements the Streams Admin API for
Streams groups. The CLI exposes list/describe/delete operations and offset
management similar in spirit to consumer-group tools, but tailored to Streams
groups defined in KIP‑1071.
+
+# Commands
+
+## List Streams groups
+
+Discovering groups
+
+
+ # List all Streams groups
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 --list
+
+
+## Describe Streams groups
+
+Inspecting group's state, members, and lag
+
+
+ # Describe a group: state + epochs
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --describe --group my-streams-app --state --verbose
+
+ # Describe a group: members (assignments vs target, classic/streams)
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --describe --group my-streams-app --members --verbose
+
+ # Describe a group: input-topic offsets and lag
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --describe --group my-streams-app --offsets
+
+
+## Reset input-topic offsets (preview, then apply) {#reset-offsets}
+
+Ensure all application instances are stopped/inactive. Always preview changes
with `--dry-run` before using `--execute`.
+
+
+ # Preview resetting all input topics to a specific timestamp
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --group my-streams-app \
+ --reset-offsets --all-input-topics --to-datetime 2025-01-31T23:57:00.000
\
+ --dry-run
+
+ # Apply the reset
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --group my-streams-app \
+ --reset-offsets --all-input-topics --to-datetime 2025-01-31T23:57:00.000
\
+ --execute
+
+
+## Delete offsets to force re-consumption
+
+Delete offsets for all or specific input topics to have the group re-read data
on restart.
+
+
+ # Delete offsets for all input topics (execute)
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --group my-streams-app \
+ --delete-offsets --all-input-topics --execute
+
+ # Delete offsets for specific topics
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --group my-streams-app \
+ --delete-offsets --topic input-a --topic input-b --execute
+
+
+## Delete a Streams group (cleanup)
+
+Delete broker-side Streams metadata for a group and optionally remove a subset
of internal topics.
+
+
+ # Delete Streams group metadata
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --delete --group my-streams-app
+
+ # Delete a subset of internal topics alongside the group (use with care)
+ kafka-streams-groups.sh --bootstrap-server localhost:9092 \
+ --delete --group my-streams-app \
+ --internal-topics my-app-repartition-0,my-app-changelog
+
+
+# All options and flags
+
+## Core actions
+
+ * `--list`: List Streams groups. Use `--state` to display/filter by state.
+ * `--describe`: Describe a group selected by `--group`. Combine with:
+ * `--state` (group state and epochs), `--members` (members and
assignments), `--offsets` (input and repartition topics offsets/lag).
+ * `--verbose` for additional details (e.g., leader epochs where
applicable).
+ * `--reset-offsets`: Reset input-topic offsets (one group at a time;
instances should be inactive). Choose exactly one specifier:
+ * `--to-earliest`, `--to-latest`, `--to-current`, `--to-offset <n>`
+ * `--by-duration <PnDTnHnMnS>`, `--to-datetime <YYYY-MM-DDTHH:mm:SS.sss>`
+ * `--shift-by <n>` (±), `--from-file` (CSV)
+Scope:
+ * `--all-input-topics` or one/more `--topic <name>`; some builds also
support `--all-topics` (all input topics per broker topology metadata).
+Safety:
+ * Requires `--dry-run` or `--execute`.
+ * `--delete-offsets`: Delete offsets for `--all-input-topics`, specific
`--topic` names, or `--from-file`.
+ * `--delete`: Delete Streams group metadata; optionally pass
`--internal-topics <list>` to delete a subset of internal topics.
+
+
+
+## Common flags
+
+ * `--group <id>`: Target Streams group (application.id).
+ * `--all-groups`: Operate on all groups (allowed with `--delete`).
+ * `--bootstrap-server <host:port>`: Broker(s) to connect to (required).
+ * `--command-config <file>`: Properties for AdminClient (security, timeouts,
etc.).
+ * `--timeout <ms>`: Wait time for group stabilization in some operations
(default: 5000ms).
+ * `--dry-run`, `--execute`: Preview vs apply for mutating operations.
+ * `--help`, `--version`, `--verbose`: Usage, version, verbosity.
+
+
+
+# Best practices and safety
+
+ * Preview changes with `--dry-run` to verify topic scope and impact before
`--execute`.
+ * Use `--internal-topics` carefully: deleting internal topics removes state
backing topics; only do this when you intend to rebuild state from input topics.
+
+
+
+This page documents `kafka-streams-groups.sh` capabilities for Streams groups
as defined by KIP‑1071 and implemented in Apache Kafka.
+
+ * [Documentation](/documentation)
+ * [Kafka Streams](/documentation/streams)
+ * [Developer Guide](/documentation/streams/developer-guide/)
+
+
diff --git
a/content/en/42/streams/developer-guide/streams-rebalance-protocol.md
b/content/en/42/streams/developer-guide/streams-rebalance-protocol.md
new file mode 100644
index 0000000000..0b56d41134
--- /dev/null
+++ b/content/en/42/streams/developer-guide/streams-rebalance-protocol.md
@@ -0,0 +1,255 @@
+---
+title: Streams Rebalance Protocol
+description:
+weight: 14
+tags: ['kafka', 'docs']
+aliases:
+keywords:
+type: docs
+---
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+The Streams Rebalance Protocol is a broker-driven rebalancing system designed
specifically for Kafka Streams applications. Following the pattern of KIP-848,
which moved rebalance coordination of plain consumers from clients to brokers,
KIP-1071 extends this model to Kafka Streams workloads.
+
+# Overview
+
+Instead of clients computing new assignments on the client during rebalance
events involving all members of the group, assignments are computed
continuously on the broker. Instead of using a consumer group, the streams
application registers as a **streams group** with the broker, which manages and
exposes all metadata required for coordination of the streams application
instances.
+
+This approach brings Kafka Streams coordination in line with the modern
broker-driven rebalance model introduced for consumers in KIP-848, providing a
dedicated group type with streams-specific semantics and metadata management.
+
+# What's Supported in This Version
+
+The following features are available in the current release:
+
+* **Core Streams Group Rebalance Protocol**: The `group.protocol=streams`
configuration enables the dedicated streams rebalance protocol. This separates
streams groups from consumer groups and provides a streams-specific group
membership lifecycle and metadata management on the broker.
+
+* **Sticky Task Assignor**: A basic task assignment strategy that minimizes
task movement during rebalances is included.
+
+* **Interactive Query Support**: IQ operations are compatible with the new
streams protocol.
+
+* **New Admin RPC**: The StreamsGroupDescribe RPC provides streams-specific
metadata separate from consumer group information, with corresponding access
via the [`Admin`](/{version}/javadoc/org/apache/kafka/clients/admin/Admin.html)
interface.
+
+* **CLI Integration**: You can list, describe, and delete streams groups via
the
[bin/kafka-streams-groups.sh](/{version}/streams/developer-guide/kafka-streams-group-sh/)
script.
+
+* **Offline Migration**: After shutting down all members and waiting for their
`session.timeout.ms` to expire (or forcing an explicit group leave), a classic
group can be converted to a streams group and a streams group can be converted
to a classic group. The only broker-side group data that will be preserved are
the committed offsets. Internal topics (changelog and repartition topics) will
continue to exist as regular Kafka topics.
+
+# What's Not Supported in This Version
+
+The following features are not yet available and should be avoided when using
the new protocol:
+
+* **Static Membership**: Setting a client `instance.id` will be rejected.
+
+* **Topology Updates**: If a topology is changed significantly (e.g., by
adding new source topics or changing the number of subtopologies), a new
streams group must be created.
+
+* **High Availability Assignor**: Only the sticky assignor is supported. This
implies that "warmup tasks" and rack aware assignment are not supported yet.
+
+* **Regular Expressions**: Pattern-based topic subscription is not supported.
+
+* **Online Migration**: Group migration while the application is running is
not available between the classic and new streams protocol.
+
+* **Custom Client Supplier**: Using a custom `KafkaClientSupplier` will only
allow so provide restore/global consumer, producer, and admin client. It's not
possible to provide the "main" consumer when "streams" groups are enabled.
+
+# Why Use the Streams Rebalance Protocol?
+
+The Streams Rebalance Protocol offers several key advantages over the classic
client-driven protocol:
+
+* **Broker-Driven Coordination**: Centralizes task assignment logic on brokers
instead of the client. This provides consistent, authoritative task assignment
decisions from a single coordination point, and reduces the potential for
split-brain scenarios.
+
+* **Faster, More Stable Rebalances**: Reduces rebalance duration and impact by
removing the global synchronization point. This minimizes application downtime
during membership changes or failures.
+
+* **Better Observability**: Provides dedicated metrics and admin interfaces
that separate streams from consumer groups, leading to clearer troubleshooting
with broker-side observability. See the [streams groups
metrics](/{version}/operations/monitoring#group-coordinator-monitoring)
documentation for details.
+
+# Enabling the Protocol
+
+The Streams Rebalance Protocol is enabled by default on new clusters starting
with Apache Kafka 4.2. Both brokers and clients must be running Apache Kafka
4.2 or later to use this protocol.
+
+## Broker Configuration
+
+The protocol is enabled by default on new Apache Kafka 4.2 clusters. To enable
the feature on existing clusters (after upgrading to 4.2) or to explicitly
control it:
+
+Enable the feature:
+```
+bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature
streams.version=1
+```
+
+Disable the feature:
+```
+bin/kafka-features.sh --bootstrap-server localhost:9092 downgrade --feature
streams.version=0
+```
+
+## Client Configuration
+
+In your Kafka Streams application configuration, set:
+```
+group.protocol=streams
+```
+
+# Configuration
+
+## Broker Configuration
+
+The following broker configurations control the behavior of streams groups.
For complete details, see the [broker
configuration](/{version}/configuration/broker-configs) documentation.
+
+*
[`group.coordinator.rebalance.protocols`](/{version}/configuration/broker-configs#brokerconfigs_group.coordinator.rebalance.protocols):
The list of enabled rebalance protocols. `"streams"` is included in the list
of protocols to enable streams groups.
+*
[`group.streams.session.timeout.ms`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.session.timeout.ms):
The default timeout for all streams group (if not specifically overwritten for
a specific streams group) to detect client failures when using the streams
group protocol.
+*
[`group.streams.min.session.timeout.ms`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.min.session.timeout.ms):
The minimum session timeout.
+*
[`group.streams.max.session.timeout.ms`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.max.session.timeout.ms):
The maximum session timeout.
+*
[`group.streams.heartbeat.interval.ms`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.heartbeat.interval.ms):
The default heartbeat interval given to the members.
+*
[`group.streams.min.heartbeat.interval.ms`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.min.heartbeat.interval.ms):
The minimum heartbeat interval.
+*
[`group.streams.max.heartbeat.interval.ms`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.max.heartbeat.interval.ms):
The maximum heartbeat interval.
+*
[`group.streams.max.size`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.max.size):
The maximum number of streams clients that a single streams group can
accommodate.
+*
[`group.streams.num.standby.replicas`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.num.standby.replicas):
The default number of standby replicas for each task.
+*
[`group.streams.max.standby.replicas`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.max.standby.replicas):
Maximum for dynamic configurations of the standby replica configuration.
+*
[`group.streams.initial.rebalance.delay.ms`](/{version}/configuration/broker-configs#brokerconfigs_group.streams.initial.rebalance.delay.ms):
The first rebalance of a new (ie, previously empty) group is delayed by this
amount to allow more members to join the group.
+
+## Group Configuration
+
+Configurations for the resource type `GROUP` are available in
`DescribeConfigs` and `IncrementalAlterConfigs` to override the default broker
configurations dynamically for specific groups. These can be set using the
[`Admin`](/{version}/javadoc/org/apache/kafka/clients/admin/Admin.html) Java
interface or the `bin/kafka-configs.sh` utility.
+
+For complete details, see the [group
configuration](/{version}/configuration/group-configs) documentation.
+
+The following group-level configurations are available for streams groups:
+
+*
[`streams.session.timeout.ms`](/{version}/configuration/group-configs#groupconfigs_streams.session.timeout.ms):
The timeout to detect client failures when using the streams group protocol.
+*
[`streams.heartbeat.interval.ms`](/{version}/configuration/group-configs#groupconfigs_streams.heartbeat.interval.ms):
The heartbeat interval given to the members.
+*
[`streams.num.standby.replicas`](/{version}/configuration/group-configs#groupconfigs_streams.num.standby.replicas):
The number of standby replicas for each task.
+*
[`streams.initial.rebalance.delay.ms`](/{version}/configuration/group-configs#groupconfigs_streams.initial.rebalance.delay.ms):
The first rebalance of a group is delayed by this amount to allow more members
to join the group.
+
+### Example: Setting Group-Level Configuration
+```
+bin/kafka-configs.sh --bootstrap-server localhost:9092 \
+ --alter --entity-type groups --entity-name wordcount \
+ --add-config streams.num.standby.replicas=1
+```
+
+**Note:** In the streams rebalance protocol, `session.timeout.ms`,
`heartbeat.interval.ms` and `num.standby.replicas` are group-level
configurations, which are ignored when they are set on the client side. Use the
`bin/kafka-configs.sh` tool to set these configurations as shown above.
+
+## Streams Configuration
+
+For complete details on all Kafka Streams configurations, see the [streams
configuration](/{version}/configuration/kafka-streams-configs) documentation.
+
+The following client configuration enables the streams rebalance protocol:
+
+*
[`group.protocol`](/{version}/configuration/kafka-streams-configs#streamsconfigs_group.protocol):
A flag which indicates if the streams rebalance protocol should be used. Set
to `streams` to enable (default is `classic`).
+
+### Ignored Configurations
+
+The following configurations are ignored when the streams rebalance protocol
is enabled:
+*
[`acceptable.recovery.lag`](/{version}/configuration/kafka-streams-configs#streamsconfigs_acceptable.recovery.lag)
+*
[`max.warmup.replicas`](/{version}/configuration/kafka-streams-configs#streamsconfigs_max.warmup.replicas)
+*
[`num.standby.replicas`](/{version}/configuration/kafka-streams-configs#streamsconfigs_num.standby.replicas)
(use group-level configuration instead)
+*
[`probing.rebalance.interval.ms`](/{version}/configuration/kafka-streams-configs#streamsconfigs_probing.rebalance.interval.ms)
+*
[`rack.aware.assignment.tags`](/{version}/configuration/kafka-streams-configs#streamsconfigs_rack.aware.assignment.tags)
+*
[`rack.aware.assignment.strategy`](/{version}/configuration/kafka-streams-configs#streamsconfigs_rack.aware.assignment.strategy)
+*
[`rack.aware.assignment.traffic_cost`](/{version}/configuration/kafka-streams-configs#streamsconfigs_rack.aware.assignment.traffic_cost)
+*
[`rack.aware.assignment.non_overlap_cost`](/{version}/configuration/kafka-streams-configs#streamsconfigs_rack.aware.assignment.non_overlap_cost)
+*
[`task.assignor.class`](/{version}/configuration/kafka-streams-configs#streamsconfigs_task.assignor.class)
+*
[`session.timeout.ms`](/{version}/configuration/kafka-streams-configs#streamsconfigs_session.timeout.ms)
(use group-level configuration instead)
+*
[`heartbeat.interval.ms`](/{version}/configuration/kafka-streams-configs#streamsconfigs_heartbeat.interval.ms)
(use group-level configuration instead)
+
+# Administration
+
+## Admin API
+
+Use the "streams groups" methods of the
[`Admin`](/{version}/javadoc/org/apache/kafka/clients/admin/Admin.html)
interface to manage streams groups programmatically. These APIs are mostly
backed by the same implementations as the consumer group API.
+
+The main differences from consumer group APIs are:
+
+* The `describeStreamsGroups` uses the DescribeStreamsGroup RPC and contains
different information than consumer groups.
+* A streams group has an extra state - `NOT_READY` - and no legacy states from
the classic protocol.
+* `removeMembersFromConsumerGroup` will not have a corresponding API in this
version, as it uses the LeaveGroup RPC for classic consumer groups, which is
not available for KIP-848-style groups.
+
+## kafka-streams-groups.sh
+
+A new tool called `bin/kafka-streams-groups.sh` is added for working with
streams groups. It replaces `bin/kafka-streams-application-reset.sh` for
streams groups and can be used to list, describe, and delete streams groups.
See the [kafka-streams-groups.sh
documentation](/{version}/streams/developer-guide/kafka-streams-group-sh/) for
detailed usage information.
+
+# Architecture and How It Works
+
+## Streams Groups
+
+The protocol introduces the concept of a **streams group** in parallel to a
consumer group. Streams clients use a dedicated heartbeat RPC,
`StreamsGroupHeartbeat`, to join a group, leave a group, and update the group
coordinator about its currently owned tasks and its client-specific metadata.
+
+The group coordinator manages a streams group similarly to a consumer group,
continuously updating the group member metadata via heartbeat responses and
running assignment logic when changes are detected. A new group type called
`streams` is introduced to the group coordinator, with new record key and value
types for group metadata, topology metadata, and group member metadata. These
records are persisted in the `__consumer_offsets` topic.
+
+A group can either be a streams group, a share group, or a consumer group,
defined by the first heartbeat request using the corresponding GroupId.
+
+## Topology Configuration and Validation
+
+To assign tasks among streams clients, the group coordinator uses topology
metadata that is initialized when a member joins the group and persisted in the
consumer offsets topic.
+
+Whenever a member joins the streams group, the first heartbeat request
contains metadata of the topology. The metadata describes the topology as a set
of subtopologies, each identified by a unique string identifier and containing
metadata relevant for creation of internal topics and assignment.
+
+### Topology Validation and NOT_READY State
+
+During the handling of the streams group heartbeat, the group coordinator may
detect that source/sink or internal topics required by the topology do not
exist or differ in their configuration from what is required for the topology
to execute successfully. This triggers a "topology configuration" process, in
which the group coordinator performs the following steps:
+
+* Check that all configured source topics exist.
+* Check that "copartition groups" are satisfied - that is, all source topics
that are supposed to be copartitioned are indeed copartitioned.
+* Derive the required number of partitions for all internal topics from the
source topic configuration.
+* Check that all internal topics exist with the right configuration.
+
+If any source topics or internal topics are missing, the group enters a state
`NOT_READY`. In `NOT_READY`, all heartbeats will be handled as usual (so they
typically should not fail), but in the heartbeat response, the status will
indicate which kind of problem exists. All members will get an empty assignment
when the group is in `NOT_READY` state.
+
+## Centralized Assignment Configuration
+
+Core assignment options are configured centrally on the broker, without
relying on each client's configuration. This allows tuning a streams group
without redeploying the streams application. The core assignment option
introduced on the broker-side is `num.standby.replicas`. This can be configured
both globally on the broker and dynamically for specific streams groups through
the `IncrementalAlterConfigs` and `DescribeConfigs` RPCs.
+
+The last used assignment configuration is stored in the group metadata on the
broker. This way, if an assignment configuration is dynamically changed,
reassignment can be triggered immediately.
+
+# Monitoring and Metrics
+
+The existing group metrics are extended to differentiate between streams
groups and consumer groups and account for streams group states. For complete
details, see the [streams groups
metrics](/{version}/operations/monitoring#group-coordinator-monitoring)
documentation.
+
+## Group Count by Protocol
+
+Number of groups based on type of protocol, where the list of protocols is
extended by the `protocol=streams` variation:
+```
+kafka.server:type=group-coordinator-metrics,name=group-count,protocol={consumer|classic|streams}
+```
+
+## Streams Group Count by State
+
+Number of streams groups based on state:
+```
+kafka.server:type=group-coordinator-metrics,name=streams-group-count,state={empty|not_ready|assigning|reconciling|stable|dead}
+```
+
+## Streams Group Rebalances
+
+Streams group rebalances sensor:
+```
+kafka.server:type=group-coordinator-metrics,name=streams-group-rebalance-rate
+
+kafka.server:type=group-coordinator-metrics,name=streams-group-rebalance-count
+```
+
+# Migration from Classic Protocol
+
+Currently, only offline migration is supported. To migrate a Kafka Streams
application from the classic protocol to the streams rebalance protocol:
+
+1. Shut down all application instances.
+2. Wait for the `session.timeout.ms` to expire so the group becomes empty (or
force an explicit group leave).
+3. Update the application configuration to set `group.protocol=streams`.
+4. Restart the application instances.
+
+The only broker-side group data that will be preserved are the committed
offsets. All other group metadata will be recreated when the application starts
with the new protocol. Internal topics (changelog and repartition topics) will
continue to exist as regular Kafka topics.
+
+Similarly, you can convert a streams group back to a classic group by
following the same process but setting `group.protocol=classic`.
+
+**Warning:** Online migration (migrating while the application is running) is
not available in this version. Plan for a maintenance window when migrating
between protocols.
diff --git a/content/en/42/streams/upgrade-guide.md
b/content/en/42/streams/upgrade-guide.md
index 1dea56e561..bec8f25d34 100644
--- a/content/en/42/streams/upgrade-guide.md
+++ b/content/en/42/streams/upgrade-guide.md
@@ -53,7 +53,9 @@ Starting in version 4.0.0, Kafka Streams will only be
compatible when running ag
Downgrading from 3.5.x or newer version to 3.4.x or older version needs
special attention: Since 3.5.0 release, Kafka Streams uses a new serialization
format for repartition topics. This means that older versions of Kafka Streams
would not be able to recognize the bytes written by newer versions, and hence
it is harder to downgrade Kafka Streams with version 3.5.0 or newer to older
versions in-flight. For more details, please refer to
[KIP-904](https://cwiki.apache.org/confluence/x/P5VbD [...]
-Downgrading from 3.0.x or newer version to 2.8.x or older version needs
special attention: Since 3.0.0 release, Kafka Streams uses a newer RocksDB
version whose on-disk format changed. This means that old versioned RocksDB
would not be able to recognize the bytes written by that newer versioned
RocksDB, and hence it is harder to downgrade Kafka Streams with version 3.0.0
or newer to older versions in-flight. Users need to wipe out the local RocksDB
state stores written by the new version [...]
+Downgrading from 3.0.x or newer version to 2.8.x or older version needs
special attention: Since 3.0.0 release, Kafka Streams uses a newer RocksDB
version whose on-disk format changed. This means that old versioned RocksDB
would not be able to recognize the bytes written by that newer versioned
RocksDB, and hence it is harder to downgrade Kafka Streams with version 3.0.0
or newer to older versions in-flight. Users need to first delete the local
RocksDB state stores written by the new ver [...]
+
+Downgrading from 4.0.x or newer version to older than 4.0.0 needs special
attention: Since 4.0.0 release, Kafka Streams upgraded RocksDB from version
7.9.2 to 9.7.3. This upgrade introduces a RocksDB file format version change
from version 5 to version 6 (introduced in RocksDB 8.6). While the newer
RocksDB version (9.7.3) can read state stores written in the old format
(version 5), the older RocksDB versions cannot read state stores written in the
new format (version 6). This means that [...]
Kafka Streams does not support running multiple instances of the same
application as different processes on the same physical state directory.
Starting in 2.8.0 (as well as 2.7.1 and 2.6.2), this restriction will be
enforced. If you wish to run more than one instance of Kafka Streams, you must
configure them with different values for `state.dir`.
@@ -63,6 +65,15 @@ Since 2.6.0 release, Kafka Streams depends on a RocksDB
version that requires Ma
## Streams API changes in 4.2.0
+### General Availability for a core feature set of the Streams Rebalance
Protocol (KIP-1071)
+
+The Streams Rebalance Protocol is a broker-driven rebalancing system designed
specifically for Kafka Streams applications.
+This release marks the General Availability for the core functionality
detailed in
[KIP-1071](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol).
+For more information about the feature set, design, usage and migration,
+please refer to the [developer
guide](/{version}/streams/developer-guide/streams-rebalance-protocol).
+
+### Other changes
+
Kafka Streams now supports Dead Letter Queue (DLQ). A new config
`errors.deadletterqueue.topic.name` allows to specify the name of the DLQ
topic. When set and `DefaultProductionExceptionHandler` is used, records that
cause exceptions will be forwarded to the DLQ topic. If a custom exception
handler is used, it is up to the custom handler to build DLQ records to send,
hence, depending on the implementation, the `errors.deadletterqueue.topic.name`
configuration may be ignored. `org.apache. [...]
We introduce a new `org.apache.kafka.streams.CloseOptions` class which
replaces the existing `org.apache.kafka.streams.KafkaStreams$CloseOptions`. The
latter is deprecated and will be removed in the next major release.
`CloseOptions` class allows to specify close timeout and group membership
operation - whether the consumer needs to leave the group or remain in the
group. More details can be found in
[KIP-1153](https://cwiki.apache.org/confluence/x/QAq9F).
@@ -510,7 +521,8 @@ Kafka Streams API (rows)
</td>
<td>
-4.1.x
+4.1.x and
+4.2.x
</td> </tr>
<tr>
<td>
@@ -543,7 +555,8 @@ compatible
3.8.x and
3.9.x and
4.0.x and
-4.1.x
+4.1.x and
+4.2.x
</td>
<td>
