Hi Kevin, Thanks for the quick response!
AH1: If the goal of the KIP is to remove the need to run `kafka-storage format` before running kafka on only a subset of the nodes in a cluster, I think it would be worth making that very clear (maybe at the beginning of Proposed Changes in a summary that details what the new format/startup procedure looks like). AH2: > Mainly because going forward, > meta.properties V2 will be written on newly formatted nodes + > non-formatting nodes. Additionally, starting kafka on any node requires > cluster.id to be present in meta.properties. Otherwise, the node crashes. The above seems confusing to me - are we saying that cluster.id will be required in meta.properties to start any kind of node? In your response to AH6 you seem to say the opposite? > Right now, starting kafka on any node requires cluster.id to be > present in meta.properties. Otherwise, the node crashes. As part of this > KIP, we no longer want that to be the case for every node. Best, Alyssa On Wed, Feb 25, 2026 at 2:23 PM Kevin Wu <[email protected]> wrote: > Hi Alyssa, > > Thanks for the replies and questions. > > RE AH1: The goal of the KIP (at least for now) is to remove the need to run > `kafka-storage format` before running kafka on brokers and observer > controllers. Sure, I can change the wording. > > RE AH2: I don't think you would need to? Mainly because going forward, > meta.properties V2 will be written on newly formatted nodes + > non-formatting nodes. Additionally, starting kafka on any node requires > cluster.id to be present in meta.properties. Otherwise, the node crashes. > > RE AH5: I specifically mentioned the `.checkpoint` file created from > formatting as part of this write path because the active controller writes > its contents to the log when it becomes leader. If there is no > bootstrap/0-0.checkpoint on the leader, it will default to the latest MV, > and after this KIP, that would mean the active controller would write > ClusterIdRecord alongside the MV during the bootstrap metadata write. > > RE AH6: Right now, starting kafka on any node requires cluster.id to be > present in meta.properties. Otherwise, the node crashes. As part of this > KIP, we no longer want that to be the case for every node. There are two > approaches for this, one that I think is more suitable for a minor release, > and one I think is better for 5.0 due to compatibility cases (although this > is definitely still up for discussion and I'd like to know everyone's > thoughts). > The former is what I described in the KIP, which is that any kafka node > that can become the active controller must have meta.properties with a > cluster.id to start kafka successfully. This removes the formatting > requirement for broker-only nodes and controllers who are not part of the > KRaft voter set. In this case, we do not need to cover for the case where > the leader does not have a meta.properties, because this node would have > crashed before becoming leader. > The latter is to remove the requirement for all nodes to format (keep in > mind that dynamic quorums currently must have at least one node format to > elect a leader in a proper configuration), but formatting would still be > optional to allow for control over feature levels etc.. In this case, kafka > would need to write its own cluster id upon electing an active controller. > However, we would then need to support dynamic quorum bootstrapping during > startup too. I don't think this is a good idea at the moment, as it would > mean introducing a static config to determine who are the "bootstrap > controllers," and the formatting logic for dynamic quorum is then > duplicated at startup. What happens if both the static config exist and the > node was formatted? You can imagine this same problem for every other > feature too. Basically, I think it is really confusing to users and bad > design if kafka can work without formatting any nodes, but also you can > format if you want. If we want to remove the requirement of any node having > to format, we should just remove the `kafka-storage format` and probably > the whole storage tool altogether, but that should be in a major release. > If we go with this approach now, another concern I have is about older MV > clusters whose nodes have software versions that support this feature. How > would a new software version broker persist the cluster id if the operator > did not format the node? I think it is easier to communicate how the latter > approach works in a major release. > > Best, > Kevin Wu > > On Wed, Feb 25, 2026 at 9:57 PM Alyssa Huang via dev <[email protected] > > > wrote: > > > Hey Kevin, thanks for the KIP! > > > > AH1: > > > > > The reason for this KIP is to remove the requirement of brokers needed > to > > > run the storage tool before starting Kafka. > > > > I'd like to clarify whether you're saying the goal of the KIP is to > remove > > the need of running just the storage format command or the storage tool > at > > all. Also, I misunderstood > > > > > Persisting this data does not need to be done before starting kafka. > > > > as saying we don't need to persist this data at all and that actually > > messed up my understanding of the rest of your KIP on my first read - can > > we change the wording to "Persisting this data does not have to be done > > during storage formatting and can be done later during startup". > > > > AH2: In the MV upgrade case, do we see any value in updating the > > meta.properties file to v2? > > > > AH3: > > > > > Since this feature is associated with a new metadata record and > > > MetadataVersion, broker bootstrapping with cluster ID is required on > all > > > MVs < X where X is the first MV that supports this feature. Because > some > > > MetadataVersion is resolved during each node's formatting, we can > > determine > > > at format time if a ClusterIdRecord is needed as part of a controller's > > > 0-0/bootstrap.checkpoint. > > > > Can you go into some detail on how the MV is resolved during a node's > > formatting? This might be interesting for readers to know in the > > provisioning and re-formatting/adding a single node case. > > > > AH4: Under Proposed Changes the first header is: > > > > > meta.properties can be written during kafka broker/controller startup > if > > > it doesn't exist already (from formatting) > > > > Nit, but should this be 'meta.properties' *will* be written during kafka > > broker/controller startup if it doesn't exist already? > > > > AH5: Could the following be expanded a bit? "write-path" might be too > > generic since all nodes will write the MV they are formatted with to the > > bootstrap metadata (but only on leader election will the elected node > > attempt to write the bootstrap metadata MV to the metadata log). I found > > "formatting a node (specifically controllers who can become leader)" > > especially confusing, since I'm assuming you're saying the > > write-to-metadata-log path for MV. > > > > > This is enforceable along the write-path for MV, which occurs at two > > > points: formatting a node (specifically controllers who can become > > leader) > > > and upgrading the MV using kafka-features upgrade > > > > > > AH6: > > > > > However, kafka should still be able to handle the case where a leader > is > > > elected who does not have clusterId in meta.properties, which can occur > > if > > > a majority of voters do not have a clusterId > > > > The wording makes it sound like this is the exceptional case, but before > I > > read JR6 and your respond I had assumed this would be the normal case > for a > > cluster provisioned with KIP-1262 support. Can we make this more clear in > > the KIP? Why are we making cluster id optional in meta.properties v2 if > we > > expect to still write it? > > > > We can still enforce that bootstrap controllers must have formatted (and > > > therefore persisted a cluster id to meta.properties) prior to starting > > > kafka. > > > > I'm confused if this is saying we will continue to enforce persisting > > cluster id to meta.properties (given that you've also said kafka will > > handle the case where a leader is elected w/o clusterid in its > > meta.properties) > > > > AH7: > > > > > Nodes discover persist cluster id to meta.properties from metadata > > > publishing pipeline > > > > > Seems like there's an extra word in this header > > > > Thanks in advance for your responses :) > > Alyssa > > > > On Wed, Feb 18, 2026 at 8:49 AM Kevin Wu <[email protected]> wrote: > > > > > Hi Jun, > > > > > > Thanks for the replies and questions. > > > > > > RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One > > > thing I'm not sure about yet is whether or not the record field should > be > > > of UUID or String type. This is because kafka's quickstart docs refer > to > > > setting `--cluster-id` to a UUID in the storage tool. However, many > > places > > > in kafka broker/controller code (e.g. the raft client, broker lifecycle > > > manager, and even the formatter itself) only require this type to be a > > > String. Since not all Strings are valid UUIDs, making this record field > > of > > > type UUID might be too restrictive and complicate upgrading the MV for > > > existing clusters, since they might have a non-UUID cluster id string, > > but > > > need to write this record when upgrading to an MV that supports this > > > feature. Let me know what you think. > > > > > > RE JR2: Any controller node formatted with `--standalone, > > > --initial-controllers` or who is part of the static voter set defined > by > > > `controller.quorum.voters` can write the ClusterIdRecord by including > the > > > `--cluster-id` argument to `kafka-storage format`. However, if the MV > of > > > the cluster supports it, there is exactly one writer of this record to > > the > > > cluster metadata partition. The writer is the first active controller, > > who > > > writes this record alongside other bootstrap metadata records (e.g. > > > metadata version) during controller activation. At this point, we > already > > > depend on MV existing, since the active controller writes these > bootstrap > > > metadata records as a transaction if the MV supports it. I think > writing > > > the cluster id record would follow a similar pattern. > > > > > > RE JR3: When a node formats, it will write the meta.properties file. > > During > > > formatting, a node must resolve the MV it wants to format with, which > is > > > explained more in RE JR5. I need to think about this more, but I think > we > > > should keep `--cluster-id` as a required flag for invoking the format > > > command. If a broker/observer controller does not format, > meta.properties > > > is written without cluster id immediately after startup (i.e. where we > > read > > > it from disk now in KafkaRaftServer). > > > > > > RE JR4: Yeah, will do. In this context, when I say observers I'm > > referring > > > to any controllers who are not part of the KRaft voter set when they > > start > > > kafka, or any brokers. I will make this explicit in the KIP. From the > > > perspective of this feature and KRaft leader election, controller nodes > > who > > > format with `--no-initial-controllers`, controller nodes who are not > part > > > of `controller.quorum.voters`, and brokers, all do not "need" to > format, > > > since they cannot become the active controller. This means they can > > resolve > > > metadata like the cluster id after discovering the leader. We have a > > > similar pattern with how controller nodes who format with > > > `--no-initial-controllers` discover the kraft version of the cluster. > > > > > > RE JR5: If a node formats, it must resolve a metadata version with > which > > to > > > format. This comes from the `--release-version/--feature` flag and > > defaults > > > to the latest production MV. Therefore, when a node formats with a > > metadata > > > version that supports this feature, it will write the ClusterIdRecord > to > > > its `0-0/bootstrap.checkpoint`. If the node formats with a metadata > > version > > > that does not support this feature, it does not write ClusterIdRecord > to > > > its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is > assumed > > > that this node is part of a cluster whose MV supports this. Otherwise, > > this > > > is a misconfiguration and the node will fail to register with the > leader > > > since there is no way for it to persist cluster id to its > meta.properties > > > without formatting. > > > > > > Although I did not specify this yet on the KIP explicitly, after some > > > offline discussion I think it makes sense to enforce the following > > > invariant as part of the feature design: if the persisted metadata > > version > > > supports this feature, the ClusterId record must also be persisted. > This > > is > > > enforceable on the write-path for MV, which occurs at two points-- > during > > > formatting and during feature upgrades. There is a similar pattern with > > > kraft.version, as it gets written to disk at the same two points. > > > > > > RE JR6: The main motivation for writing cluster id to meta.properties > as > > > well is because it can act as a projection of the cluster metadata > > > partition which essentially only exposes the cluster id to readers. For > > > example, the raft layer needs to be aware of the cluster id for its own > > RPC > > > handling/validation, but raft cannot read metadata records. There are > > many > > > readers of this cluster id value during the startup of the cluster. > > > Therefore, avoiding a read of the metadata partition to discover the > > value > > > of this metadata will prevent more complications of the startup code. > > > > > > Best, > > > Kevin Wu > > > > > > > > > On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <[email protected]> > > > wrote: > > > > > > > Hi, Kevin, > > > > > > > > Thanks for the KIP. A few comments. > > > > > > > > JR1. ClusterIdRecord : Could you define the record format? > > > > > > > > JR2. "a new MetadataVersion that supports encoding/decoding this > > record. > > > > This means that during formatting, the bootstrap ClusterIdRecord is > > only > > > > written if the cluster is formatted with a MV that supports this > > > feature." > > > > Could you describe who writes the ClusterIdRecord? Is it the leader > > > > controller? Also, when is the record written? Do we guarantee that MV > > is > > > > available at that time? > > > > > > > > JR3. "meta.properties can be written during kafka broker/controller > > > startup > > > > if it doesn't exist already (from formatting)" > > > > Could you describe when meta.properties is written? Is MV available > at > > > that > > > > time? > > > > > > > > JR4. "Introduce a metadata record for cluster id + observers persist > > > > cluster id to meta.properties from metadata publishing pipeline" > > > > Could you clarify what observers are? Are they observer controllers > or > > > are > > > > they brokers (which are referred to as observers to the controller)? > > > > > > > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record > > > during > > > > formatting" > > > > This sounds like adding a ClusterIdRecord is optional. If so, could > you > > > > describe when a record will be added and when a record will not be > > added? > > > > > > > > JR6. "However, kafka should still be able to handle the case where a > > > leader > > > > is elected without a cluster id in meta.properties , since KRaft does > > not > > > > need cluster.id in order to elect a leader. > > > > In this case, the active controller will write a cluster id > > > > record during the bootstrap metadata write." > > > > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the > > first > > > > time, these nodes need to persist this to meta.properties". Why do we > > > need > > > > to introduce a separate place to write the cluster id to > > > meta.properties. > > > > > > > > Jun > > > > > > > > > > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <[email protected]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Manually bumping this thread after finalizing a design. > > > > > KIP link: > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories > > > > > > > > > > Best, > > > > > Kevin Wu > > > > > > > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <[email protected]> > > > wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > I would like to start a discussion on KIP-1262, which proposes > > > removing > > > > > > the formatting requirement for brokers and observer controllers. > > > > > Currently, > > > > > > I am considering two high-level designs, and would appreciate > > > community > > > > > > feedback on both approaches to decide on a final design. > > > > > > > > > > > > KIP link: > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories > > > > > > > > > > > > Best, > > > > > > Kevin Wu > > > > > > > > > > > > > > > > > > > > >
