This is an automated email from the ASF dual-hosted git repository.
lucasbru pushed a commit to branch 4.1
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/4.1 by this push:
new 32c8cfa87fd KAFKA-19379: Basic upgrade guide for KIP-1071 EA (#20029)
32c8cfa87fd is described below
commit 32c8cfa87fd50715072db3f14490c6ee04628794
Author: Lucas Brutschy <[email protected]>
AuthorDate: Mon Jun 30 09:28:22 2025 +0200
KAFKA-19379: Basic upgrade guide for KIP-1071 EA (#20029)
Basic documentation describing: - That it's in EA now
- What it does
- What features are not yet supported
- How to enable it / disable it
- Any changes in the interfaces
- kafka-streams-groups.sh
- StreamsGroupDescribe
- How to provide feedback
Reviewers: Andrew Schofield <[email protected]>, Matthias J. Sax
<[email protected]>
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Matthias J. Sax <[email protected]>
---
docs/streams/developer-guide/config-streams.html | 15 ++++
docs/streams/upgrade-guide.html | 101 +++++++++++++++++++++++
2 files changed, 116 insertions(+)
diff --git a/docs/streams/developer-guide/config-streams.html
b/docs/streams/developer-guide/config-streams.html
index c1f23e994e1..6a555ae3ccf 100644
--- a/docs/streams/developer-guide/config-streams.html
+++ b/docs/streams/developer-guide/config-streams.html
@@ -80,6 +80,7 @@ settings.put(... , ...);</code></pre>
<li><a class="reference internal"
href="#deserialization-exception-handler"
id="id7">deserialization.exception.handler</a></li>
<li><a class="reference internal" href="#enable-metrics-push"
id="id43">enable.metrics.push</a></li>
<li><a class="reference internal"
href="#ensure-explicit-internal-resource-naming"
id="id46">ensure.explicit.internal.resource.naming</a></li>
+ <li><a class="reference internal" href="#group-protocol"
id="id47">group.protocol</a></li>
<li><a class="reference internal"
href="#log-summary-interval-ms" id="id40">log.summary.interval.ms</a></li>
<li><a class="reference internal" href="#max-task-idle-ms"
id="id28">max.task.idle.ms</a></li>
<li><a class="reference internal" href="#max-warmup-replicas"
id="id29">max.warmup.replicas</a></li>
@@ -775,6 +776,20 @@
streamsConfiguration.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG,
</div>
</blockquote>
</div>
+ <div class="section" id="group-protocol">
+ <h4><a class="toc-backref" href="#id47">group.protocol</a><a
class="headerlink" href="#group-protocol" title="Permalink to this
headline"></a></h4>
+ <blockquote>
+ <div>
+ <p>
+ The group protocol used by the Kafka Streams client used for
coordination.
+ It determines how the client will communicate with the Kafka
brokers and other clients in the same group.
+ The default value is <code class="docutils literal"><span
class="pre">"classic"</span></code>, which is the classic consumer group
protocol.
+ Can be set to <code class="docutils literal"><span
class="pre">"streams"</span></code> (requires broker-side enablement) to enable
the new Kafka Streams group protocol.
+ Note that the "streams" rebalance protocol is an Early Access
feature and should not be used in production.
+ </p>
+ </div>
+ </blockquote>
+ </div>
<div class="section" id="rack-aware-assignment-non-overlap-cost">
<h4><a class="toc-backref"
href="#id37">rack.aware.assignment.non_overlap_cost</a><a class="headerlink"
href="#rack-aware-assignment-non-overlap-cost" title="Permalink to this
headline"></a></h4>
<blockquote>
diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index a2031907563..8816f3b933a 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -141,6 +141,107 @@
<h3><a id="streams_api_changes_410"
href="#streams_api_changes_410">Streams API changes in 4.1.0</a></h3>
+ <h4>Early Access of the Streams Rebalance Protocol</h4>
+
+ <p>
+ The Streams Rebalance Protocol is a broker-driven rebalancing system
designed specifically for Kafka
+ Streams applications. Following the pattern of KIP-848, which moved
rebalance coordination of plain consumers
+ from clients to brokers, KIP-1071 extends this model to Kafka Streams
workloads. Instead of clients
+ computing new assignments on the client during rebalance events
involving all members of the group, assignments are
+ computed continuously on the broker. Instead of using a consumer
group, the streams application registers as a
+ streams group with the broker, which manages and exposes all metadata
required for coordination of the
+ streams application instances.
+ </p>
+
+ <p>
+ This Early Access release covers a subset of the functionality
detailed in
+ <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol";>KIP-1071</a>.
+ Do not use the new protocol in production. The API is subject to
change in future
+ releases.
+ </p>
+
+ <p><strong>What's Included in Early Access</strong></p>
+
+ <ul>
+ <li><strong>Core Streams Group Rebalance Protocol:</strong> The
<code>group.protocol=streams</code> configuration
+ enables the dedicated streams rebalance protocol. This separates
streams groups from consumer groups and
+ provides a streams-specific group membership lifecycle and
metadata management on the broker.</li>
+ <li><strong>Sticky Task Assignor:</strong> A basic task assignment
strategy that minimizes task movement
+ during rebalances is included.</li>
+ <li><strong>Interactive Query Support:</strong> IQ operations are
compatible with the new streams protocol.</li>
+ <li><strong>New Admin RPC:</strong> The
<code>StreamsGroupDescribe</code> RPC provides streams-specific metadata
+ separate from consumer group information, with corresponding
access via the <code>Admin</code> client.</li>
+ <li><strong>CLI Integration:</strong> You can list, describe, and
delete streams groups via the <code>kafka-streams-groups.sh</code> script.</li>
+ </ul>
+
+ <p><strong>What's Not Included in Early Access</strong></p>
+
+ <ul>
+ <li><strong>Static Membership:</strong> Setting a client `instance.id`
will be rejected.</li>
+ <li><strong>Topology Updates:</strong> If a topology is changed
significantly (e.g., by adding new source topics
+ or changing the number of sub-topologies), a new streams group
must be created.</li>
+ <li><strong>High Availability Assignor:</strong> Only the sticky
assignor is supported.</li>
+ <li><strong>Regular Expressions:</strong> Pattern-based topic
subscription is not supported.</li>
+ <li><strong>Reset Operations:</strong> CLI offset reset operations are
not supported.</li>
+ <li><strong>Protocol Migration:</strong> Group migration is not
available between the classic and new streams protocols.</li>
+ </ul>
+
+ <p><strong>Why Use the Streams Rebalance Protocol?</strong></p>
+
+ <ul>
+ <li>
+ <strong>Broker-Driven Coordination:</strong>
+ Centralizes task assignment logic on brokers instead of the
client. This provides consistent,
+ authoritative task assignment decisions from a single coordination
point and reduces the potential for
+ split-brain scenarios.
+ </li>
+ <li>
+ <strong>Faster, More Stable Rebalances:</strong>
+ Reduces rebalance duration and impact by removing the global
synchronization point. This minimizes
+ application downtime during membership changes or failures.
+ </li>
+ <li>
+ <strong>Better Observability:</strong>
+ Provides dedicated metrics and admin interfaces that separate
streams from consumer groups, leading to
+ clearer troubleshooting with broker-side observability.
+ </li>
+ </ul>
+
+ <p>
+ Enabling the protocol requires the brokers and clients are running
Apache Kafka 4.1. It should be enabled
+ only on new clusters for testing purposes.
+ Set <code>unstable.feature.versions.enable=true</code> for controllers
and brokers, and
+ set <code>unstable.api.versions.enable=true</code> on the brokers as
well. In your Kafka Streams application
+ configuration, set <code>group.protocol=streams</code>.
+ </p>
+
+ <p>
+ Migration between the classic consumer group protocol and the Streams
Rebalance Protocol is not supported in
+ either direction. An application using this protocol must use a new
<code>application.id</code> that has not
+ been used by any application on the classic protocol. Furthermore,
this ID must not be in use as a
+ <code>group.id</code> by any consumer ("classic" or "consumer") nor
share-group application.
+ It is also possible to delete a previous consumer group using
<code>kafka-consumer-groups.sh</code> before
+ starting the application with the new protocol, which will however
also delete all offsets for that group.
+ </p>
+
+ <p>
+ To operate the new streams groups, explore the options of
<code>kafka-streams-groups.sh</code> to list,
+ describe, and delete streams groups. In the new protocol,
<code>session.timeout.ms</code>,
+ <code>heartbeat.interval.ms</code> and
<code>num.standby.replicas</code> are group-level configurations,
+ which are ignored when they are set on the client side. Use the
<code>kafka-configs.sh</code> tool to set
+ these configurations, for example:
+ <code>kafka-configs.sh --bootstrap-server localhost:9092 --alter
--entity-type groups
+ --entity-name wordcount --add-config
streams.num.standby.replicas=1</code>.
+ </p>
+
+ <p>
+ Please provide feedback on this feature via the
+ <a href="https://kafka.apache.org/contact";>Kafka mailing lists</a> or
by filing
+ <a href="https://kafka.apache.org/contributing";>JIRA issues</a>.
+ </p>
+
+ <h4>Other changes</h4>
+
<p>
The introduction of <a
href="https://cwiki.apache.org/confluence/x/4Y_MEw";>KIP-1111</a>
enables you to enforce explicit naming for all internal resources of
the topology, including internal topics (e.g., changelog and repartition
topics) and their associated state stores.
