Tom, thanks for taking a look!
1. Yes you're right, I'll fix that
2. Components refers to broker objects like ReplicaManager,
ClientQuotaManager, etc. Controller components (so far) don't need to
reload any state based on metadata.version changes.
3. Yes, if a controller reads a
Hi David,
Thanks for the KIP. Sorry I'm so late in looking at this. I have a few
questions.
1. In the Proposed Changes Overview it says "The controller validates that
the cluster can be safely downgraded to this version (override with
--force)" I think that be "--unsafe"?
2. Also in this
Hi, David,
Thanks for the reply. No more questions from me.
Jun
On Tue, Nov 30, 2021 at 1:52 PM David Arthur
wrote:
> Thanks Jun,
>
> 30: I clarified the description of the "upgrade" command to:
>
> The FEATURE and VERSION arguments can be repeated to indicate an upgrade of
> > multiple
Thanks Jun,
30: I clarified the description of the "upgrade" command to:
The FEATURE and VERSION arguments can be repeated to indicate an upgrade of
> multiple features in one request. Alternatively, the RELEASE flag can be
> given to upgrade to the default versions of the specified release.
Hi, David,
Thanks for the reply. Just one more minor comment.
30. ./kafka-features.sh upgrade: It seems that the release param is
optional. Could you describe the semantic when release is not specified?
Jun
On Mon, Nov 29, 2021 at 5:06 PM David Arthur
wrote:
> Jun, I updated the KIP with the
Jun, I updated the KIP with the "disable" CLI.
For 16, I think you're asking how we can safely introduce the
initial version of other feature flags in the future. This will probably
depend on the nature of the feature that the new flag is gating. We can
probably take a similar approach and say
Hi, David,
Thanks for the reply. The new CLI sounds reasonable to me.
16.
For case C, choosing the latest version sounds good to me.
For case B, for metadata.version, we pick version 1 since it just happens
that for metadata.version version 1 is backward compatible. How do we make
this more
Jun,
The KIP has some changes to the CLI for KIP-584. With Jason's suggestion
incorporated, these three commands would look like:
1) kafka-features.sh upgrade --release latest
upgrades all known features to their defaults in the latest release
2) kafka-features.sh downgrade --release 3.x
Hi, Jason, David,
Just to clarify on the interaction with the end user, the design in KIP-584
allows one to do the following.
(1) kafka-features.sh --upgrade-all --bootstrap-server
kafka-broker0.prn1:9071 to upgrade all features to the latest version known
by the tool. The user doesn't need to
Colin, thanks for the detailed response. I understand what you're saying
and I agree with your rationale.
It seems like we could just initialize their cluster.metadata to 1 when the
> software is upgraded to 3.2.
>
Concretely, this means the controller would see that there is no
Jason,
1/2. You've got it right. The intention is that metadata.version will gate
both RPCs (like IBP does) as well as metadata records. So, when a broker
sees that metadata.version changed, it may start advertising new RPCs and
it will need to refresh the ApiVersions it has for other brokers. I
Hi Colin/David,
> Like David said, basically it boils down to creating a feature flag for
the new proposed __consumer_offsets version, or using a new
IBP/metadata.version for it. Both approaches have pros and cons. Using an
IBP/metadata.version bump reduces the size of the testing matrix. But
A few additional questions:
1. Currently the IBP tells us what version of individual inter-broker RPCs
will be used. I think the plan in this KIP is to use ApiVersions request
instead to find the highest compatible version (just like clients). Do I
have that right?
2. The following wasn't very
On Wed, Nov 17, 2021, at 11:28, Jason Gustafson wrote:
> Hi David,
>
> Forgive me if this ground has been covered already. Today, we have a few
> other things that we have latched onto the IBP, such as upgrades to the
> format of records in __consumer_offsets. I've been assuming that
>
On Wed, Nov 17, 2021, at 12:28, David Arthur wrote:
> Colin,
>
> I wasn't intending to suggest that IBP and metadata.version co-exist long
> term in KRaft. I was thinking we would have one final IBP that would signal
> KRaft clusters to start looking at metadata.version instead. This
> was working
Jason, thanks for the review! No, I don't think that has been covered
explicitly and no metadata.version will not gate things like our offsets
format. The intention is that we will introduce new KIP-584 feature flags
for those formats once the need arises. For ZK clusters, we'll probably
keep
Colin,
I wasn't intending to suggest that IBP and metadata.version co-exist long
term in KRaft. I was thinking we would have one final IBP that would signal
KRaft clusters to start looking at metadata.version instead. This
was working under the assumption that we should be able to upgrade 3.0
Hi David,
Forgive me if this ground has been covered already. Today, we have a few
other things that we have latched onto the IBP, such as upgrades to the
format of records in __consumer_offsets. I've been assuming that
metadata.version is not covering this. Is that right or is there some other
Hi, Colin,
Thanks for the reply.
For case b, I am not sure that I understand your suggestion. Does "each
subsequent level for metadata.version corresponds to an IBP version" mean
that we need to keep IBP forever? Could you describe the upgrade process in
this case?
Jun
On Tue, Nov 16, 2021 at
On Tue, Nov 16, 2021, at 15:13, Jun Rao wrote:
> Hi, David, Colin,
>
> Thanks for the reply.
>
> 16. Discussed with David offline a bit. We have 3 cases.
> a. We upgrade from an old version where the metadata.version has already
> been finalized. In this case it makes sense to stay with that
Hi, David, Colin,
Thanks for the reply.
16. Discussed with David offline a bit. We have 3 cases.
a. We upgrade from an old version where the metadata.version has already
been finalized. In this case it makes sense to stay with that feature
version after the upgrade.
b. We upgrade from an old
Yeah I agree that checking a majority of voters support the
metadata.version is sufficient. What I was originally considering is
whether (in the future) we could consider encoding the metadata.version
value in the vote request as well, so that the elected leader is supposed
to have a version
On Tue, Nov 16, 2021, at 13:36, Guozhang Wang wrote:
> Hi Colin,
>
> If we allow downgrades which would be appended in metadata.version, then
> the length of the __cluster_medata log may not be safe to indicate higher
> versions, since older version records could still be appended later than a
>
Hi Colin,
If we allow downgrades which would be appended in metadata.version, then
the length of the __cluster_medata log may not be safe to indicate higher
versions, since older version records could still be appended later than a
new version record right?
On Tue, Nov 16, 2021 at 1:16 PM Colin
Thanks, I think having the leader election to consider the metadata.version
would be a good idea moving forward, but we do not need to include in this
KIP.
On Tue, Nov 16, 2021 at 6:37 AM David Arthur
wrote:
> Guozhang,
>
> 1. By requiring a majority of all controller nodes to support the
On Tue, Nov 16, 2021, at 06:36, David Arthur wrote:
> An interesting case here is how to handle a version update if a majority of
> the quorum supports it, but the leader doesn't. For example, if C1 was the
> leader and an upgrade to version 4 was requested. Maybe this would trigger
> C1 to resign
Hi David,
On Mon, Nov 15, 2021, at 08:13, David Arthur wrote:
> This (probably) means the quorum peers must continue to rely on ApiVersions
> to negotiate compatible RPC versions and these versions cannot depend on
> metadata.version.
I agree. Can we explicitly spell out in KIP that quorum peers
On Fri, Nov 5, 2021, at 08:10, David Arthur wrote:
> Colin and Jun, thanks for the additional comments!
>
> Colin:
>
>> We've been talking about having an automated RPC compatibility checker
>
> Do we have a way to mark fields in schemas as deprecated? It can stay in
> the RPC, it just complicates
On Fri, Nov 5, 2021, at 15:18, Jun Rao wrote:
> Hi, David,
>
> Thanks for the reply.
>
> 16. My first concern is that the KIP picks up meta.version inconsistently
> during the deployment. If a new cluster is started, we pick up the highest
> version. If we upgrade, we leave the feature version
Guozhang,
1. By requiring a majority of all controller nodes to support the version
selected by the leader, we increase the likelihood that the next leader
will also support it. We can't guarantee that all nodes definitely support
the selected metadata.version because there could always be an
Thanks David,
1. Got it. One thing I'm still not very clear is why it's sufficient to
select a metadata.version which is supported by majority of the quorum, but
not the whole quorum (i.e. choosing the lowest version among all the quorum
members)? Since the leader election today does not take
Guozhang, thanks for the review!
1, As we've defined it so far, the initial metadata.version is set by an
operator via the "kafka-storage.sh" tool. It would be possible for
different values to be selected, but only the quorum leader would commit a
FeatureLevelRecord with the version they read
Jun, thanks for the comments!
16, When a new cluster is deployed, we don't select the highest available
metadata.version, but rather the quorum leader picks a bootstrap version
defined in meta.properties. As mentioned earlier, we should add validation
here to ensure a majority of the followers
Hello David,
Thanks for the very nice writeup! It helped me a lot to refresh my memory
on KIP-630/590/584 :)
I just had two clarification questions after reading through the KIP:
1. For the initialization procedure, do we guarantee that all the quorum
nodes (inactive candidates and leaders,
Hi, David,
Thanks for the reply.
16. My first concern is that the KIP picks up meta.version inconsistently
during the deployment. If a new cluster is started, we pick up the highest
version. If we upgrade, we leave the feature version unchanged.
Intuitively, it seems that independent of how a
Colin and Jun, thanks for the additional comments!
Colin:
> We've been talking about having an automated RPC compatibility checker
Do we have a way to mark fields in schemas as deprecated? It can stay in
the RPC, it just complicates the logic a bit.
> It would be nice if the active controller
Hi, David, Jose and Colin,
Thanks for the reply. A few more comments.
12. It seems that we haven't updated the AdminClient accordingly?
14. "Metadata snapshot is generated and sent to the other inactive
controllers and to brokers". I thought we wanted each broker to generate
its own snapshot
On Tue, Oct 12, 2021, at 10:34, Jun Rao wrote:
> Hi, David,
>
> One more comment.
>
> 16. The main reason why KIP-584 requires finalizing a feature manually is
> that in the ZK world, the controller doesn't know all brokers in a cluster.
> A broker temporarily down is not registered in ZK. in the
Hi David,
Thanks again for the KIP.
David Arthur wrote:
> 101. Change AllowDowngrade bool to DowngradeType int8 in
> UpgradeFeatureRequest RPC. I'm wondering if we can kind of "cheat" on this
> incompatible change since it's not currently in use and totally remove the
> old field and leave
Hey folks, I just updated the KIP with details on proposed changes to the
kafka-features.sh tool. It includes four proposed sub-commands which will
provide the Basic and Advanced functions detailed in KIP-584. Please have a
look, thanks!
Hi David,
Thanks for the explanations. Few comments below.
7001. Sounds good.
7002. Sounds good. The --force-downgrade-all option can be used for the
basic CLI while the --force-downgrade option can be used for the advanced
CLI.
7003. I like your suggestion on separate sub-commands, I agree
>
> How does the active controller know what is a valid `metadata.version`
> to persist? Could the active controller learn this from the
> ApiVersions response from all of the inactive controllers?
The active controller should probably validate whatever value is read from
meta.properties against
On Fri, Oct 15, 2021 at 7:24 AM David Arthur wrote:
> Hmm. So I think you are proposing the following flow:
> > 1. Cluster metadata partition replicas establish a quorum using
> > ApiVersions and the KRaft protocol.
> > 2. Inactive controllers send a registration RPC to the active controller.
> >
Hey everyone, I've updated the KIP with a few items we've discussed so far:
101. Change AllowDowngrade bool to DowngradeType int8 in
UpgradeFeatureRequest RPC. I'm wondering if we can kind of "cheat" on this
incompatible change since it's not currently in use and totally remove the
old field and
Jose,
Are you saying that for metadata.version X different software versions
> could generate different snapshots? If so, I would consider this an
> implementation bug, no? The format and content of a snapshot is a
> public API that needs to be supported across software versions.
I agree this
Kowshik, thanks for the review!
7001. An enum sounds like a good idea here. Especially since setting
Upgrade=false and Force=true doesn't really make sense. An enum would avoid
confusing/invalid combinations of flags
7002. How about adding --force-downgrade as an alternative to the
--downgrade
On Tue, Oct 12, 2021 at 10:57 AM Colin McCabe wrote:
> > 11. For downgrades, it would be useful to describe how to determine the
> > downgrade process (generating new snapshot, propagating the snapshot, etc)
> > has completed. We could block the UpdateFeature request until the process
> > is
On Thu, Oct 7, 2021 at 5:20 PM Jun Rao wrote:
> 7. Jose, what control records were you referring?
>
Hey Jun, in KRaft we have 3 control records.
- LeaderChangeMessage - this is persistent in the replica log when a
new leader gets elected and the epoch increases. We never included
this record in
On Tue, Oct 5, 2021 at 8:53 AM David Arthur wrote:
> 2. Generate snapshot on downgrade
> > > Metadata snapshot is generated and sent to the other inactive
> > controllers and to brokers (this snapshot may be lossy!)
> > Why do we need to send this downgraded snapshot to the brokers? The
> >
Hi David,
Thanks for the great KIP! It's good to see KIP-584 is starting to get used.
Few comments below.
7001. In the UpdateFeaturesRequest definition, there is a newly introduced
ForceDowngrade parameter. There is also an existing AllowDowngrade
parameter. My understanding is that the
Jun and Colin, thanks very much for the comments. See below
10. Colin, I agree that ApiVersionsRequest|Response is the most
straightforward approach here
11. This brings up an interesting point. Once a UpdateFeature request is
finished processing, a subsequent "describe features"
On Wed, Sep 29, 2021, at 10:34, David Arthur wrote:
> Hey all,
>
> I'd like to start a discussion around upgrades for KRaft. This design does
> not cover any of the ZooKeeper to KRaft migration (which will be covered in
> a future KIP).
>
> The proposal includes the addition of a
On Thu, Oct 7, 2021, at 17:19, Jun Rao wrote:
> Hi, David,
>
> Thanks for the KIP. A few comments below.
>
> 10. It would be useful to describe how the controller node determines the
> RPC version used to communicate to other controller nodes. There seems to
> be a bootstrap problem. A controller
Hi, David,
One more comment.
16. The main reason why KIP-584 requires finalizing a feature manually is
that in the ZK world, the controller doesn't know all brokers in a cluster.
A broker temporarily down is not registered in ZK. in the KRaft world, the
controller keeps track of all brokers,
Hi, David,
Thanks for the KIP. A few comments below.
10. It would be useful to describe how the controller node determines the
RPC version used to communicate to other controller nodes. There seems to
be a bootstrap problem. A controller node can't read the log and
therefore the feature level
Jose, thanks for the thorough review and comments!
I am out of the office until next week, so I probably won't be able to
update the KIP until then. Here are some replies to your questions:
1. Generate snapshot on upgrade
> > Metadata snapshot is generated and sent to the other nodes
> Why does
One more comment.
7.Downgrade records
I think we should explicitly mention that the downgrade process will
downgrade both metadata records and controller records. In KIP-630 we
introduced two control records for snapshots.
Thank you David for the detailed KIP.
1. Generate snapshot on upgrade
> Metadata snapshot is generated and sent to the other nodes
Why does the Active Controller need to generate a new snapshot and
force a snapshot fetch from the replicas (inactive controller and
brokers) on an upgrade? Isn't
Hey all,
I'd like to start a discussion around upgrades for KRaft. This design does
not cover any of the ZooKeeper to KRaft migration (which will be covered in
a future KIP).
The proposal includes the addition of a "metadata.version" feature flag and
deprecates the IBP. A procedure for
59 matches
Mail list logo
