This is an automated email from the ASF dual-hosted git repository. lucasbru pushed a commit to branch kip1071_docs in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 7e2a6c335760a415907e7f3dc83022343eb129f6 Author: Lucas Brutschy <[email protected]> AuthorDate: Tue Jun 24 14:29:49 2025 +0200 inital docs --- docs/streams/developer-guide/config-streams.html | 15 +++++ docs/streams/upgrade-guide.html | 85 ++++++++++++++++++++++++ 2 files changed, 100 insertions(+) diff --git a/docs/streams/developer-guide/config-streams.html b/docs/streams/developer-guide/config-streams.html index c1f23e994e1..e90c0d0302a 100644 --- a/docs/streams/developer-guide/config-streams.html +++ b/docs/streams/developer-guide/config-streams.html @@ -80,6 +80,7 @@ settings.put(... , ...);</code></pre> <li><a class="reference internal" href="#deserialization-exception-handler" id="id7">deserialization.exception.handler</a></li> <li><a class="reference internal" href="#enable-metrics-push" id="id43">enable.metrics.push</a></li> <li><a class="reference internal" href="#ensure-explicit-internal-resource-naming" id="id46">ensure.explicit.internal.resource.naming</a></li> + <li><a class="reference internal" href="#group-protocol" id="id47">group.protocol</a></li> <li><a class="reference internal" href="#log-summary-interval-ms" id="id40">log.summary.interval.ms</a></li> <li><a class="reference internal" href="#max-task-idle-ms" id="id28">max.task.idle.ms</a></li> <li><a class="reference internal" href="#max-warmup-replicas" id="id29">max.warmup.replicas</a></li> @@ -775,6 +776,20 @@ streamsConfiguration.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, </div> </blockquote> </div> + <div class="section" id="group-protocol"> + <h4><a class="toc-backref" href="#id47">group.protocol</a><a class="headerlink" href="#group-protocol" title="Permalink to this headline"></a></h4> + <blockquote> + <div> + <p> + The group protocol used by the Kafka Streams client used for coordination. + This is used to determine how the client will communicate with the Kafka brokers and other clients in the same group. + The default value is <code class="docutils literal"><span class="pre">"classic"</span></code>, which is the classic consumer group protocol. + Set to <code class="docutils literal"><span class="pre">"streams"</span></code> to enable the new Kafka Streams group protocol. + Note that the streams rebalance protocol is an Early Access feature and should not be used in production. + </p> + </div> + </blockquote> + </div> <div class="section" id="rack-aware-assignment-non-overlap-cost"> <h4><a class="toc-backref" href="#id37">rack.aware.assignment.non_overlap_cost</a><a class="headerlink" href="#rack-aware-assignment-non-overlap-cost" title="Permalink to this headline"></a></h4> <blockquote> diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html index a2031907563..7157e6757db 100644 --- a/docs/streams/upgrade-guide.html +++ b/docs/streams/upgrade-guide.html @@ -141,6 +141,91 @@ <h3><a id="streams_api_changes_410" href="#streams_api_changes_410">Streams API changes in 4.1.0</a></h3> + <h4>Early Access of the Streams Rebalance Protocol</h4> + + <p> + The Streams Rebalance Protocol is a broker-driven rebalancing system designed specifically for Kafka + Streams applications. Following the pattern of KIP-848, which moved rebalance coordination of plain consumers + from clients to brokers, KIP-1071 extends this model to Kafka Streams workloads. Instead of clients + computing new assignments on the client during rebalance events involving all members of the group, assignments are + computed continuously on the broker. Instead of using a consumer group, the streams application registers as a + streams groups with the broker, which manages and exposes all metadata required for coordination of the + streams application instances. + </p> + + <p> + This Early Access release covers a subset of the functionality detailed in + <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol";>KIP-1071</a>. + Do not use the new protocol in production.. The API is subject to change in future + releases. + </p> + + <p><strong>What's Included in Early Access</strong></p> + + <ul> + <li><strong>Core Streams Group Rebalance Protocol:</strong> The <code>group.protocol=streams</code> configuration + enables the dedicated streams rebalance protocol. This separates streams groups from consumer groups and + provides a streams-specific group membership lifecycle and metadata management on the broker.</li> + <li><strong>Sticky Task Assignor:</strong> A basic task assignment strategy that minimizes task movement + during rebalances is included.</li> + <li><strong>Interactive Query Support:</strong> IQ operations are compatible with the new streams protocol.</li> + <li><strong>New Admin RPC:</strong> The <code>StreamsGroupDescribe</code> RPC provides streams-specific metadata + separate from consumer group information, with corresponding access via the <code>Admin</code> client.</li> + <li><strong>CLI Integration:</strong> You can list, describe, and delete streams groups via the <code>kafka-streams-groups.sh</code> script.</li> + </ul> + + <p><strong>What's Not Included in Early Access</strong></p> + + <ul> + <li><strong>Static Membership:</strong> Setting a client `instance.id` will be rejected.</li> + <li><strong>Topology Updates:</strong> If a topology is changed significantly (e.g., by adding new source topics + or changing the number of sub-topologies), a new streams group must be created.</li> + <li><strong>High Availability Assignor:</strong> Only the sticky assignor is supported.</li> + <li><strong>Regular Expressions:</strong> Pattern-based topic subscription is not supported.</li> + <li><strong>Reset Operations:</strong> CLI offset reset operations are not supported.</li> + <li><strong>Protocol Migration:</strong> No automated migration is available between the classic and new streams protocols.</li> + </ul> + + <p><strong>Why Use the Streams Rebalance Protocol?</strong></p> + + <ul> + <li> + <strong>Broker-Driven Coordination:</strong> + Centralizes task assignment logic on brokers instead of the client. This provides consistent, + authoritative task assignment decisions from a single coordination point and reduces the potential for + split-brain scenarios. + </li> + <li> + <strong>Faster, More Stable Rebalances:</strong> + Reduces rebalance duration and impact by removing the global synchronization point. This minimizes + application downtime during membership changes or failures. + </li> + <li> + <strong>Better Observability:</strong> + Provides dedicated metrics and admin interfaces that separate streams from consumer groups, leading to + clearer troubleshooting with broker-side observability. + </li> + </ul> + + <p> + To enable the protocol, set <code>unstable.feature.versions.enable=true</code> for controllers and brokers, and + set <code>unstable.api.versions.enable=true</code> on the brokers as well. In your Kafka Streams application + configuration, set <code>group.protocol=streams</code>. + </p> + + <p> + Migration between the classic consumer group protocol and the Streams Rebalance Protocol is not supported in + either direction. An application using this protocol must use a new <code>application.id</code> that has not + been used by any application on the classic protocol. Furthermore, this ID must not be in use as a + <code>group.id</code> by any consumer application. + </p> + + <p> + Please provide feedback on this feature via the Kafka mailing lists or by filing JIRA issues. + </p> + + <h4>Other changes</h4> + <p> The introduction of <a href="https://cwiki.apache.org/confluence/x/4Y_MEw";>KIP-1111</a> enables you to enforce explicit naming for all internal resources of the topology, including internal topics (e.g., changelog and repartition topics) and their associated state stores.
