Hi David, Thanks for the explanations. Few comments below.
7001. Sounds good. 7002. Sounds good. The --force-downgrade-all option can be used for the basic CLI while the --force-downgrade option can be used for the advanced CLI. 7003. I like your suggestion on separate sub-commands, I agree it's more convenient to use. 7004/7005. Your explanation sounds good to me. Regarding the min finalized version level, this becomes useful for feature version deprecation as explained here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP584:Versioningschemeforfeatures-Featureversiondeprecation . This is not implemented yet, and the work item is tracked in KAFKA-10622. Cheers, Kowshik On Fri, Oct 15, 2021 at 11:38 AM David Arthur <mum...@gmail.com> wrote: > > > > How does the active controller know what is a valid `metadata.version` > > to persist? Could the active controller learn this from the > > ApiVersions response from all of the inactive controllers? > > > The active controller should probably validate whatever value is read from > meta.properties against its own range of supported versions (statically > defined in code). If the operator sets a version unsupported by the active > controller, that sounds like a configuration error and we should shutdown. > I'm not sure what other validation we could do here without introducing > ordering dependencies (e.g., must have quorum before initializing the > version) > > For example, let's say that we have a cluster that only has remote > > controllers, what are the valid metadata.version in that case? > > > I believe it would be the intersection of supported versions across all > brokers and controllers. This does raise a concern with upgrading the > metadata.version in general. Currently, the active controller only > validates the target version based on the brokers' support versions. We > will need to include controllers supported versions here as well (using > ApiVersions, probably). > > On Fri, Oct 15, 2021 at 1:44 PM José Armando García Sancio > <jsan...@confluent.io.invalid> wrote: > > > On Fri, Oct 15, 2021 at 7:24 AM David Arthur <mum...@gmail.com> wrote: > > > Hmm. So I think you are proposing the following flow: > > > > 1. Cluster metadata partition replicas establish a quorum using > > > > ApiVersions and the KRaft protocol. > > > > 2. Inactive controllers send a registration RPC to the active > > controller. > > > > 3. The active controller persists this information to the metadata > log. > > > > > > > > > What happens if the inactive controllers send a metadata.version range > > > > that is not compatible with the metadata.version set for the cluster? > > > > > > > > > As we discussed offline, we don't need the explicit registration step. > > Once > > > a controller has joined the quorum, it will learn about the finalized > > > "metadata.version" level once it reads that record. > > > > How does the active controller know what is a valid `metadata.version` > > to persist? Could the active controller learn this from the > > ApiVersions response from all of the inactive controllers? For > > example, let's say that we have a cluster that only has remote > > controllers, what are the valid metadata.version in that case? > > > > > If it encounters a > > > version it can't support it should probably shutdown since it might not > > be > > > able to process any more records. > > > > I think that makes sense. If a controller cannot replay the metadata > > log, it might as well not be part of the quorum. If the cluster > > continues in this state it won't guarantee availability based on the > > replication factor. > > > > Thanks > > -- > > -Jose > > > > > -- > David Arthur >
