[DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hey folks, I am starting a discussion thread for managing unstable metadata versions in Apache Kafka. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka This KIP is actually already implemented in 3.7 with PR https://github.com/apache/kafka/pull/14860. I have created this KIP to explain the motivation and how managing Metadata Versions is expected to work. Comments are greatly appreciated as this process can always be improved. -- --Proven
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi Proven, Thanks for the KIP. I think there is a need for this capability, for those of us who deploy from trunk (or branches dervied from trunk). With regard to "unstable.metadata.versions.enable": is this going to be a documented configuration, or an internal one? I am guessing we want it to be documented, so that users can use it. If we do, we should probably also very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. That includes logging an ERROR message on startup, etc. It would be good to document if a release can go out that contains "future MVs" that are unstable. Like can we make a 3.8 release that contains IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV? Personally I think the answer should be "yes," but with the usual caveats. When the actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably won't work, and you won't be able to upgrade. (It was unstable, we told you not to use it.) best, Colin On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote: > Hey folks, > > I am starting a discussion thread for managing unstable metadata > versions > in Apache Kafka. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > This KIP is actually already implemented in 3.7 with PR > https://github.com/apache/kafka/pull/14860. > I have created this KIP to explain the motivation and how managing Metadata > Versions is expected to work. > Comments are greatly appreciated as this process can always be improved. > > -- > --Proven
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hey all, I was wondering how often we plan to update LATEST_PRODUCTION metadata version. Is this something we should do as soon as the feature is complete or something we do when we are releasing kafka. When is the time we abandon a MV so that other features can be unblocked? I am just considering a feature that may end up missing a release. It seems like maybe that MV would block future metadata versions until we decide the feature won't make the cut. From that point, all "ready" features should be able to be released. I'm also wondering if the KIP should include some information about how a metadata should be abandoned. Maybe there is a specific message to write in the file? So folks who were maybe waiting on that version know they can release their feature? I am also assuming that we don't shift all the waiting metadata versions when we abandon a version, but it would be good to clarify and include in the KIP. Thanks, Justine On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe wrote: > Hi Proven, > > Thanks for the KIP. I think there is a need for this capability, for those > of us who deploy from trunk (or branches dervied from trunk). > > With regard to "unstable.metadata.versions.enable": is this going to be a > documented configuration, or an internal one? I am guessing we want it to > be documented, so that users can use it. If we do, we should probably also > very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. That > includes logging an ERROR message on startup, etc. > > It would be good to document if a release can go out that contains "future > MVs" that are unstable. Like can we make a 3.8 release that contains > IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV? Personally I > think the answer should be "yes," but with the usual caveats. When the > actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably > won't work, and you won't be able to upgrade. (It was unstable, we told you > not to use it.) > > best, > Colin > > > On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote: > > Hey folks, > > > > I am starting a discussion thread for managing unstable metadata > > versions > > in Apache Kafka. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > > > This KIP is actually already implemented in 3.7 with PR > > https://github.com/apache/kafka/pull/14860. > > I have created this KIP to explain the motivation and how managing > Metadata > > Versions is expected to work. > > Comments are greatly appreciated as this process can always be improved. > > > > -- > > --Proven >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: > Hey all, > > I was wondering how often we plan to update LATEST_PRODUCTION metadata > version. Is this something we should do as soon as the feature is complete > or something we do when we are releasing kafka. When is the time we abandon > a MV so that other features can be unblocked? Hi Justine, Thanks for reviewing. The idea is that you should bump LATEST_PRODUCTION when you want to take a feature to production. That could mean deploying it internally somewhere to production, or doing an Apache release that lets everyone deploy the thing to production. Not in production? No need to care about this. Make any changes you like. As a corollary, we should keep the LATEST_PRODUCTION version as low as it can be. If you haven't tested the feature, don't freeze it in stone yet. > > I am just considering a feature that may end up missing a release. It seems > like maybe that MV would block future metadata versions until we decide the > feature won't make the cut. From that point, all "ready" features should be > able to be released. The intention is the opposite. A feature in an unstable metadata version doesn't block anything. You can always move a feature from one unstable metadata version to another if the feature starts taking too long to finish. > I'm also wondering if the KIP should include some information about how a > metadata should be abandoned. Maybe there is a specific message to write in > the file? So folks who were maybe waiting on that version know they can > release their feature? > > I am also assuming that we don't shift all the waiting metadata versions > when we abandon a version, but it would be good to clarify and include in > the KIP. I'm not sure what you mean by abandoning a version. We never abandon a version once it's stable. Unstable versions can change. I wouldn't describe this as "abandonment", just the MV changing prior to release. In a similar way, the contents of the 3.7 branch will change up until 3.7.0 is released. Once it gets released, it's never unreleased. We just move on to 3.7.1. Same thing here. best, Colin > > Thanks, > > Justine > > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe wrote: > >> Hi Proven, >> >> Thanks for the KIP. I think there is a need for this capability, for those >> of us who deploy from trunk (or branches dervied from trunk). >> >> With regard to "unstable.metadata.versions.enable": is this going to be a >> documented configuration, or an internal one? I am guessing we want it to >> be documented, so that users can use it. If we do, we should probably also >> very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. That >> includes logging an ERROR message on startup, etc. >> >> It would be good to document if a release can go out that contains "future >> MVs" that are unstable. Like can we make a 3.8 release that contains >> IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV? Personally I >> think the answer should be "yes," but with the usual caveats. When the >> actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably >> won't work, and you won't be able to upgrade. (It was unstable, we told you >> not to use it.) >> >> best, >> Colin >> >> >> On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote: >> > Hey folks, >> > >> > I am starting a discussion thread for managing unstable metadata >> > versions >> > in Apache Kafka. >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka >> > >> > This KIP is actually already implemented in 3.7 with PR >> > https://github.com/apache/kafka/pull/14860. >> > I have created this KIP to explain the motivation and how managing >> Metadata >> > Versions is expected to work. >> > Comments are greatly appreciated as this process can always be improved. >> > >> > -- >> > --Proven >>
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hey Colin, I had some offline discussions with Proven previously and it seems like he said something different so I'm glad I brought it up here. Let's clarify if we are ok with reordering unstable metadata versions :) Justine On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe wrote: > On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: > > Hey all, > > > > I was wondering how often we plan to update LATEST_PRODUCTION metadata > > version. Is this something we should do as soon as the feature is > complete > > or something we do when we are releasing kafka. When is the time we > abandon > > a MV so that other features can be unblocked? > > Hi Justine, > > Thanks for reviewing. > > The idea is that you should bump LATEST_PRODUCTION when you want to take a > feature to production. That could mean deploying it internally somewhere to > production, or doing an Apache release that lets everyone deploy the thing > to production. > > Not in production? No need to care about this. Make any changes you like. > > As a corollary, we should keep the LATEST_PRODUCTION version as low as it > can be. If you haven't tested the feature, don't freeze it in stone yet. > > > > > I am just considering a feature that may end up missing a release. It > seems > > like maybe that MV would block future metadata versions until we decide > the > > feature won't make the cut. From that point, all "ready" features should > be > > able to be released. > > The intention is the opposite. A feature in an unstable metadata version > doesn't block anything. You can always move a feature from one unstable > metadata version to another if the feature starts taking too long to finish. > > > I'm also wondering if the KIP should include some information about how a > > metadata should be abandoned. Maybe there is a specific message to write > in > > the file? So folks who were maybe waiting on that version know they can > > release their feature? > > > > I am also assuming that we don't shift all the waiting metadata versions > > when we abandon a version, but it would be good to clarify and include in > > the KIP. > > I'm not sure what you mean by abandoning a version. We never abandon a > version once it's stable. > > Unstable versions can change. I wouldn't describe this as "abandonment", > just the MV changing prior to release. > > In a similar way, the contents of the 3.7 branch will change up until > 3.7.0 is released. Once it gets released, it's never unreleased. We just > move on to 3.7.1. Same thing here. > > best, > Colin > > > > > Thanks, > > > > Justine > > > > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe wrote: > > > >> Hi Proven, > >> > >> Thanks for the KIP. I think there is a need for this capability, for > those > >> of us who deploy from trunk (or branches dervied from trunk). > >> > >> With regard to "unstable.metadata.versions.enable": is this going to be > a > >> documented configuration, or an internal one? I am guessing we want it > to > >> be documented, so that users can use it. If we do, we should probably > also > >> very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. > That > >> includes logging an ERROR message on startup, etc. > >> > >> It would be good to document if a release can go out that contains > "future > >> MVs" that are unstable. Like can we make a 3.8 release that contains > >> IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV? > Personally I > >> think the answer should be "yes," but with the usual caveats. When the > >> actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably > >> won't work, and you won't be able to upgrade. (It was unstable, we told > you > >> not to use it.) > >> > >> best, > >> Colin > >> > >> > >> On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote: > >> > Hey folks, > >> > > >> > I am starting a discussion thread for managing unstable metadata > >> > versions > >> > in Apache Kafka. > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > >> > > >> > This KIP is actually already implemented in 3.7 with PR > >> > https://github.com/apache/kafka/pull/14860. > >> > I have created this KIP to explain the motivation and how managing > >> Metadata > >> > Versions is expected to work. > >> > Comments are greatly appreciated as this process can always be > improved. > >> > > >> > -- > >> > --Proven > >> >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi Justine, Yes, this is an important point to clarify. Proven can comment more, but my understanding is that we can do anything to unstable metadata versions. Reorder them, delete them, change them in any other way. There are no stability guarantees. If the current text is unclear let's add more examples of what we can do (which is anything) :) best, Colin On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote: > Hey Colin, > > I had some offline discussions with Proven previously and it seems like he > said something different so I'm glad I brought it up here. > > Let's clarify if we are ok with reordering unstable metadata versions :) > > Justine > > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe wrote: > >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: >> > Hey all, >> > >> > I was wondering how often we plan to update LATEST_PRODUCTION metadata >> > version. Is this something we should do as soon as the feature is >> complete >> > or something we do when we are releasing kafka. When is the time we >> abandon >> > a MV so that other features can be unblocked? >> >> Hi Justine, >> >> Thanks for reviewing. >> >> The idea is that you should bump LATEST_PRODUCTION when you want to take a >> feature to production. That could mean deploying it internally somewhere to >> production, or doing an Apache release that lets everyone deploy the thing >> to production. >> >> Not in production? No need to care about this. Make any changes you like. >> >> As a corollary, we should keep the LATEST_PRODUCTION version as low as it >> can be. If you haven't tested the feature, don't freeze it in stone yet. >> >> > >> > I am just considering a feature that may end up missing a release. It >> seems >> > like maybe that MV would block future metadata versions until we decide >> the >> > feature won't make the cut. From that point, all "ready" features should >> be >> > able to be released. >> >> The intention is the opposite. A feature in an unstable metadata version >> doesn't block anything. You can always move a feature from one unstable >> metadata version to another if the feature starts taking too long to finish. >> >> > I'm also wondering if the KIP should include some information about how a >> > metadata should be abandoned. Maybe there is a specific message to write >> in >> > the file? So folks who were maybe waiting on that version know they can >> > release their feature? >> > >> > I am also assuming that we don't shift all the waiting metadata versions >> > when we abandon a version, but it would be good to clarify and include in >> > the KIP. >> >> I'm not sure what you mean by abandoning a version. We never abandon a >> version once it's stable. >> >> Unstable versions can change. I wouldn't describe this as "abandonment", >> just the MV changing prior to release. >> >> In a similar way, the contents of the 3.7 branch will change up until >> 3.7.0 is released. Once it gets released, it's never unreleased. We just >> move on to 3.7.1. Same thing here. >> >> best, >> Colin >> >> > >> > Thanks, >> > >> > Justine >> > >> > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe wrote: >> > >> >> Hi Proven, >> >> >> >> Thanks for the KIP. I think there is a need for this capability, for >> those >> >> of us who deploy from trunk (or branches dervied from trunk). >> >> >> >> With regard to "unstable.metadata.versions.enable": is this going to be >> a >> >> documented configuration, or an internal one? I am guessing we want it >> to >> >> be documented, so that users can use it. If we do, we should probably >> also >> >> very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER. >> That >> >> includes logging an ERROR message on startup, etc. >> >> >> >> It would be good to document if a release can go out that contains >> "future >> >> MVs" that are unstable. Like can we make a 3.8 release that contains >> >> IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV? >> Personally I >> >> think the answer should be "yes," but with the usual caveats. When the >> >> actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably >> >> won't work, and you won't be able to upgrade. (It was unstable, we told >> you >> >> not to use it.) >> >> >> >> best, >> >> Colin >> >> >> >> >> >> On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote: >> >> > Hey folks, >> >> > >> >> > I am starting a discussion thread for managing unstable metadata >> >> > versions >> >> > in Apache Kafka. >> >> > >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka >> >> > >> >> > This KIP is actually already implemented in 3.7 with PR >> >> > https://github.com/apache/kafka/pull/14860. >> >> > I have created this KIP to explain the motivation and how managing >> >> Metadata >> >> > Versions is expected to work. >> >> > Comments are greatly appreciated as this process can always be >> improved. >> >> > >> >> > -- >> >> > --Proven >> >> >>
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi folks, Thank you for the questions. Let me clarify about reorder first. The reorder of unstable metadata versions should be infrequent. The time you reorder is when a feature that requires a higher metadata version to enable becomes "production ready" and the features with unstable metadata versions less than the new stable one are moved to metadata versions greater than the new stable feature. When we reorder, we are always allocating a new MV and we are never reusing an existing MV even if it was also unstable. This way a developer upgrading their environment with a specific unstable MV might see existing functionality stop working but they won't see new MV dependent functionality magically appear. The feature set for a given unstable MV version can only decrease with reordering. How do we define "production ready" and when should we bump LATEST_PRODUCTION? I would like to define it to be the point where the feature is code complete with tests and the KIP for it is approved. However even with this definition if the feature later develops a major issue it could still block future features until the issue is fixed which is what we are trying to avoid here. We could be much more formal about this and let the release manager for a release define what is stable for a given release and then do the bump just after the branch is created on the branch. When an RC candidate is accepted, the bump would be backported. I would like to hear other ideas here. --Proven On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe wrote: > Hi Justine, > > Yes, this is an important point to clarify. Proven can comment more, but > my understanding is that we can do anything to unstable metadata versions. > Reorder them, delete them, change them in any other way. There are no > stability guarantees. If the current text is unclear let's add more > examples of what we can do (which is anything) :) > > best, > Colin > > > On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote: > > Hey Colin, > > > > I had some offline discussions with Proven previously and it seems like > he > > said something different so I'm glad I brought it up here. > > > > Let's clarify if we are ok with reordering unstable metadata versions :) > > > > Justine > > > > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe wrote: > > > >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: > >> > Hey all, > >> > > >> > I was wondering how often we plan to update LATEST_PRODUCTION metadata > >> > version. Is this something we should do as soon as the feature is > >> complete > >> > or something we do when we are releasing kafka. When is the time we > >> abandon > >> > a MV so that other features can be unblocked? > >> > >> Hi Justine, > >> > >> Thanks for reviewing. > >> > >> The idea is that you should bump LATEST_PRODUCTION when you want to > take a > >> feature to production. That could mean deploying it internally > somewhere to > >> production, or doing an Apache release that lets everyone deploy the > thing > >> to production. > >> > >> Not in production? No need to care about this. Make any changes you > like. > >> > >> As a corollary, we should keep the LATEST_PRODUCTION version as low as > it > >> can be. If you haven't tested the feature, don't freeze it in stone yet. > >> > >> > > >> > I am just considering a feature that may end up missing a release. It > >> seems > >> > like maybe that MV would block future metadata versions until we > decide > >> the > >> > feature won't make the cut. From that point, all "ready" features > should > >> be > >> > able to be released. > >> > >> The intention is the opposite. A feature in an unstable metadata version > >> doesn't block anything. You can always move a feature from one unstable > >> metadata version to another if the feature starts taking too long to > finish. > >> > >> > I'm also wondering if the KIP should include some information about > how a > >> > metadata should be abandoned. Maybe there is a specific message to > write > >> in > >> > the file? So folks who were maybe waiting on that version know they > can > >> > release their feature? > >> > > >> > I am also assuming that we don't shift all the waiting metadata > versions > >> > when we abandon a version, but it would be good to clarify and > include in > >> > the KIP. > >> > >> I'm not sure what you mean by abandoning a version. We never abandon a > >> version once it's stable. > >> > >> Unstable versions can change. I wouldn't describe this as "abandonment", > >> just the MV changing prior to release. > >> > >> In a similar way, the contents of the 3.7 branch will change up until > >> 3.7.0 is released. Once it gets released, it's never unreleased. We just > >> move on to 3.7.1. Same thing here. > >> > >> best, > >> Colin > >> > >> > > >> > Thanks, > >> > > >> > Justine > >> > > >> > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe > wrote: > >> > > >> >> Hi Proven, > >> >> > >> >> Thanks for the KIP. I think there is a need for this capability, for > >> those
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote: > Hi folks, > > Thank you for the questions. > > Let me clarify about reorder first. The reorder of unstable metadata > versions should be infrequent. Why does it need to be infrequent? We should be able to reorder unstable metadata versions as often as we like. There are no guarantees about unstable MVs. > The time you reorder is when a feature that > requires a higher metadata version to enable becomes "production ready" and > the features with unstable metadata versions less than the new stable one > are moved to metadata versions greater than the new stable feature. When we > reorder, we are always allocating a new MV and we are never reusing an > existing MV even if it was also unstable. This way a developer upgrading > their environment with a specific unstable MV might see existing > functionality stop working but they won't see new MV dependent > functionality magically appear. The feature set for a given unstable MV > version can only decrease with reordering. If you use an unstable MV, you probably won't be able to upgrade your software. Because whenever something changes, you'll probably get serialization exceptions being thrown inside the controller. Fatal ones. Given that this is true, there's no reason to have special rules about what we can and can't do with unstable MVs. We can do anything. > > How do we define "production ready" and when should we bump > LATEST_PRODUCTION? I would like to define it to be the point where the > feature is code complete with tests and the KIP for it is approved. However > even with this definition if the feature later develops a major issue it > could still block future features until the issue is fixed which is what we > are trying to avoid here. We could be much more formal about this and let > the release manager for a release define what is stable for a given release > and then do the bump just after the branch is created on the branch. When > an RC candidate is accepted, the bump would be backported. I would like to > hear other ideas here. > Yeah, it's an interesting question. Overall, I think developers should define when a feature is production ready. The question to ask is, "are you ready to take this feature to production in your workplace?" I think most developers do have a sense of this. Obviously bugs and mistakes can happen, but I think this standard would avoid most of the issues that we're trying to avoid by having unstable MVs in the first place. ELR is a good example. Nobody would have said that it was production ready in 3.7 ... hence it belonged (and still belongs) in an unstable MV, until that changes (hopefully soon :) ) best, Colin > --Proven > > On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe wrote: > >> Hi Justine, >> >> Yes, this is an important point to clarify. Proven can comment more, but >> my understanding is that we can do anything to unstable metadata versions. >> Reorder them, delete them, change them in any other way. There are no >> stability guarantees. If the current text is unclear let's add more >> examples of what we can do (which is anything) :) >> >> best, >> Colin >> >> >> On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote: >> > Hey Colin, >> > >> > I had some offline discussions with Proven previously and it seems like >> he >> > said something different so I'm glad I brought it up here. >> > >> > Let's clarify if we are ok with reordering unstable metadata versions :) >> > >> > Justine >> > >> > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe wrote: >> > >> >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: >> >> > Hey all, >> >> > >> >> > I was wondering how often we plan to update LATEST_PRODUCTION metadata >> >> > version. Is this something we should do as soon as the feature is >> >> complete >> >> > or something we do when we are releasing kafka. When is the time we >> >> abandon >> >> > a MV so that other features can be unblocked? >> >> >> >> Hi Justine, >> >> >> >> Thanks for reviewing. >> >> >> >> The idea is that you should bump LATEST_PRODUCTION when you want to >> take a >> >> feature to production. That could mean deploying it internally >> somewhere to >> >> production, or doing an Apache release that lets everyone deploy the >> thing >> >> to production. >> >> >> >> Not in production? No need to care about this. Make any changes you >> like. >> >> >> >> As a corollary, we should keep the LATEST_PRODUCTION version as low as >> it >> >> can be. If you haven't tested the feature, don't freeze it in stone yet. >> >> >> >> > >> >> > I am just considering a feature that may end up missing a release. It >> >> seems >> >> > like maybe that MV would block future metadata versions until we >> decide >> >> the >> >> > feature won't make the cut. From that point, all "ready" features >> should >> >> be >> >> > able to be released. >> >> >> >> The intention is the opposite. A feature in an unstable metadata version >> >> doesn't block a
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi folks, > If you use an unstable MV, you probably won't be able to upgrade your > software. Because whenever something changes, you'll probably get > serialization exceptions being thrown inside the controller. Fatal ones. Thanks for this clarification. I think this concrete risk should be highlighted in the KIP and in the "unstable.metadata.versions.enable" documentation. In the test plan, should we also have one system test checking that "features with a stable MV will never have that MV changed"? On Wed, Jan 10, 2024 at 8:16 AM Colin McCabe wrote: > > On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote: > > Hi folks, > > > > Thank you for the questions. > > > > Let me clarify about reorder first. The reorder of unstable metadata > > versions should be infrequent. > > Why does it need to be infrequent? We should be able to reorder unstable > metadata versions as often as we like. There are no guarantees about unstable > MVs. > > > The time you reorder is when a feature that > > requires a higher metadata version to enable becomes "production ready" and > > the features with unstable metadata versions less than the new stable one > > are moved to metadata versions greater than the new stable feature. When we > > reorder, we are always allocating a new MV and we are never reusing an > > existing MV even if it was also unstable. This way a developer upgrading > > their environment with a specific unstable MV might see existing > > functionality stop working but they won't see new MV dependent > > functionality magically appear. The feature set for a given unstable MV > > version can only decrease with reordering. > > If you use an unstable MV, you probably won't be able to upgrade your > software. Because whenever something changes, you'll probably get > serialization exceptions being thrown inside the controller. Fatal ones. > > Given that this is true, there's no reason to have special rules about what > we can and can't do with unstable MVs. We can do anything. > > > > > How do we define "production ready" and when should we bump > > LATEST_PRODUCTION? I would like to define it to be the point where the > > feature is code complete with tests and the KIP for it is approved. However > > even with this definition if the feature later develops a major issue it > > could still block future features until the issue is fixed which is what we > > are trying to avoid here. We could be much more formal about this and let > > the release manager for a release define what is stable for a given release > > and then do the bump just after the branch is created on the branch. When > > an RC candidate is accepted, the bump would be backported. I would like to > > hear other ideas here. > > > > Yeah, it's an interesting question. Overall, I think developers should define > when a feature is production ready. > > The question to ask is, "are you ready to take this feature to production in > your workplace?" I think most developers do have a sense of this. Obviously > bugs and mistakes can happen, but I think this standard would avoid most of > the issues that we're trying to avoid by having unstable MVs in the first > place. > > ELR is a good example. Nobody would have said that it was production ready in > 3.7 ... hence it belonged (and still belongs) in an unstable MV, until that > changes (hopefully soon :) ) > > best, > Colin > > > --Proven > > > > On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe wrote: > > > >> Hi Justine, > >> > >> Yes, this is an important point to clarify. Proven can comment more, but > >> my understanding is that we can do anything to unstable metadata versions. > >> Reorder them, delete them, change them in any other way. There are no > >> stability guarantees. If the current text is unclear let's add more > >> examples of what we can do (which is anything) :) > >> > >> best, > >> Colin > >> > >> > >> On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote: > >> > Hey Colin, > >> > > >> > I had some offline discussions with Proven previously and it seems like > >> he > >> > said something different so I'm glad I brought it up here. > >> > > >> > Let's clarify if we are ok with reordering unstable metadata versions :) > >> > > >> > Justine > >> > > >> > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe wrote: > >> > > >> >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: > >> >> > Hey all, > >> >> > > >> >> > I was wondering how often we plan to update LATEST_PRODUCTION metadata > >> >> > version. Is this something we should do as soon as the feature is > >> >> complete > >> >> > or something we do when we are releasing kafka. When is the time we > >> >> abandon > >> >> > a MV so that other features can be unblocked? > >> >> > >> >> Hi Justine, > >> >> > >> >> Thanks for reviewing. > >> >> > >> >> The idea is that you should bump LATEST_PRODUCTION when you want to > >> take a > >> >> feature to production. That could mean deploying it internally > >> somewhere to > >> >> producti
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hmm it seems like Colin and Proven are disagreeing with whether we can swap unstable metadata versions. > When we reorder, we are always allocating a new MV and we are never reusing an existing MV even if it was also unstable. > Given that this is true, there's no reason to have special rules about what we can and can't do with unstable MVs. We can do anything I don't have a strong preference either way, but I think we should agree on one approach. The benefit of reordering and reusing is that we can release features that are ready earlier and we have more flexibility. With the approach where we always create a new MV, I am concerned with having many "empty" MVs. This would encourage waiting until the release before we decide an incomplete feature is not ready and moving its MV into the future. (The abandoning comment I made earlier -- that is consistent with Proven's approach) I think the only potential issue with reordering is that it could be a bit confusing and *potentially *prone to errors. Note I say potentially because I think it depends on folks' understanding with this new unstable metadata version concept. I echo Federico's comments about making sure the risks are highlighted. Thanks, Justine On Wed, Jan 10, 2024 at 1:16 AM Federico Valeri wrote: > Hi folks, > > > If you use an unstable MV, you probably won't be able to upgrade your > software. Because whenever something changes, you'll probably get > serialization exceptions being thrown inside the controller. Fatal ones. > > Thanks for this clarification. I think this concrete risk should be > highlighted in the KIP and in the "unstable.metadata.versions.enable" > documentation. > > In the test plan, should we also have one system test checking that > "features with a stable MV will never have that MV changed"? > > On Wed, Jan 10, 2024 at 8:16 AM Colin McCabe wrote: > > > > On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote: > > > Hi folks, > > > > > > Thank you for the questions. > > > > > > Let me clarify about reorder first. The reorder of unstable metadata > > > versions should be infrequent. > > > > Why does it need to be infrequent? We should be able to reorder unstable > metadata versions as often as we like. There are no guarantees about > unstable MVs. > > > > > The time you reorder is when a feature that > > > requires a higher metadata version to enable becomes "production > ready" and > > > the features with unstable metadata versions less than the new stable > one > > > are moved to metadata versions greater than the new stable feature. > When we > > > reorder, we are always allocating a new MV and we are never reusing an > > > existing MV even if it was also unstable. This way a developer > upgrading > > > their environment with a specific unstable MV might see existing > > > functionality stop working but they won't see new MV dependent > > > functionality magically appear. The feature set for a given unstable MV > > > version can only decrease with reordering. > > > > If you use an unstable MV, you probably won't be able to upgrade your > software. Because whenever something changes, you'll probably get > serialization exceptions being thrown inside the controller. Fatal ones. > > > > Given that this is true, there's no reason to have special rules about > what we can and can't do with unstable MVs. We can do anything. > > > > > > > > How do we define "production ready" and when should we bump > > > LATEST_PRODUCTION? I would like to define it to be the point where the > > > feature is code complete with tests and the KIP for it is approved. > However > > > even with this definition if the feature later develops a major issue > it > > > could still block future features until the issue is fixed which is > what we > > > are trying to avoid here. We could be much more formal about this and > let > > > the release manager for a release define what is stable for a given > release > > > and then do the bump just after the branch is created on the branch. > When > > > an RC candidate is accepted, the bump would be backported. I would > like to > > > hear other ideas here. > > > > > > > Yeah, it's an interesting question. Overall, I think developers should > define when a feature is production ready. > > > > The question to ask is, "are you ready to take this feature to > production in your workplace?" I think most developers do have a sense of > this. Obviously bugs and mistakes can happen, but I think this standard > would avoid most of the issues that we're trying to avoid by having > unstable MVs in the first place. > > > > ELR is a good example. Nobody would have said that it was production > ready in 3.7 ... hence it belonged (and still belongs) in an unstable MV, > until that changes (hopefully soon :) ) > > > > best, > > Colin > > > > > --Proven > > > > > > On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe > wrote: > > > > > >> Hi Justine, > > >> > > >> Yes, this is an important point to clarify. Proven can comment mo
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
On Wed, Jan 10, 2024, at 09:16, Justine Olshan wrote: > Hmm it seems like Colin and Proven are disagreeing with whether we can swap > unstable metadata versions. > >> When we reorder, we are always allocating a new MV and we are never > reusing an existing MV even if it was also unstable. > >> Given that this is true, there's no reason to have special rules about > what we can and can't do with unstable MVs. We can do anything > > I don't have a strong preference either way, but I think we should agree on > one approach. > The benefit of reordering and reusing is that we can release features that > are ready earlier and we have more flexibility. With the approach where we > always create a new MV, I am concerned with having many "empty" MVs. This > would encourage waiting until the release before we decide an incomplete > feature is not ready and moving its MV into the future. (The > abandoning comment I made earlier -- that is consistent with Proven's > approach) > > I think the only potential issue with reordering is that it could be a bit > confusing and *potentially *prone to errors. Note I say potentially because > I think it depends on folks' understanding with this new unstable metadata > version concept. I echo Federico's comments about making sure the risks are > highlighted. > I agree that the risks should be highlighted. That's why I mentioned that ERROR messages should be logged, and so forth. When we say you can't use this in production, we really really mean it. You will not be able to upgrade. Unstable metadata versions are for devlopers only. best, Colin > > Thanks, > > Justine > > On Wed, Jan 10, 2024 at 1:16 AM Federico Valeri > wrote: > >> Hi folks, >> >> > If you use an unstable MV, you probably won't be able to upgrade your >> software. Because whenever something changes, you'll probably get >> serialization exceptions being thrown inside the controller. Fatal ones. >> >> Thanks for this clarification. I think this concrete risk should be >> highlighted in the KIP and in the "unstable.metadata.versions.enable" >> documentation. >> >> In the test plan, should we also have one system test checking that >> "features with a stable MV will never have that MV changed"? >> >> On Wed, Jan 10, 2024 at 8:16 AM Colin McCabe wrote: >> > >> > On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote: >> > > Hi folks, >> > > >> > > Thank you for the questions. >> > > >> > > Let me clarify about reorder first. The reorder of unstable metadata >> > > versions should be infrequent. >> > >> > Why does it need to be infrequent? We should be able to reorder unstable >> metadata versions as often as we like. There are no guarantees about >> unstable MVs. >> > >> > > The time you reorder is when a feature that >> > > requires a higher metadata version to enable becomes "production >> ready" and >> > > the features with unstable metadata versions less than the new stable >> one >> > > are moved to metadata versions greater than the new stable feature. >> When we >> > > reorder, we are always allocating a new MV and we are never reusing an >> > > existing MV even if it was also unstable. This way a developer >> upgrading >> > > their environment with a specific unstable MV might see existing >> > > functionality stop working but they won't see new MV dependent >> > > functionality magically appear. The feature set for a given unstable MV >> > > version can only decrease with reordering. >> > >> > If you use an unstable MV, you probably won't be able to upgrade your >> software. Because whenever something changes, you'll probably get >> serialization exceptions being thrown inside the controller. Fatal ones. >> > >> > Given that this is true, there's no reason to have special rules about >> what we can and can't do with unstable MVs. We can do anything. >> > >> > > >> > > How do we define "production ready" and when should we bump >> > > LATEST_PRODUCTION? I would like to define it to be the point where the >> > > feature is code complete with tests and the KIP for it is approved. >> However >> > > even with this definition if the feature later develops a major issue >> it >> > > could still block future features until the issue is fixed which is >> what we >> > > are trying to avoid here. We could be much more formal about this and >> let >> > > the release manager for a release define what is stable for a given >> release >> > > and then do the bump just after the branch is created on the branch. >> When >> > > an RC candidate is accepted, the bump would be backported. I would >> like to >> > > hear other ideas here. >> > > >> > >> > Yeah, it's an interesting question. Overall, I think developers should >> define when a feature is production ready. >> > >> > The question to ask is, "are you ready to take this feature to >> production in your workplace?" I think most developers do have a sense of >> this. Obviously bugs and mistakes can happen, but I think this standard >> would avoid most of the issues
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi Federico, Thank you for the suggestions. I've added them to the KIP. --Proven On Wed, Jan 10, 2024 at 4:16 AM Federico Valeri wrote: > Hi folks, > > > If you use an unstable MV, you probably won't be able to upgrade your > software. Because whenever something changes, you'll probably get > serialization exceptions being thrown inside the controller. Fatal ones. > > Thanks for this clarification. I think this concrete risk should be > highlighted in the KIP and in the "unstable.metadata.versions.enable" > documentation. > > In the test plan, should we also have one system test checking that > "features with a stable MV will never have that MV changed"? > > On Wed, Jan 10, 2024 at 8:16 AM Colin McCabe wrote: > > > > On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote: > > > Hi folks, > > > > > > Thank you for the questions. > > > > > > Let me clarify about reorder first. The reorder of unstable metadata > > > versions should be infrequent. > > > > Why does it need to be infrequent? We should be able to reorder unstable > metadata versions as often as we like. There are no guarantees about > unstable MVs. > > > > > The time you reorder is when a feature that > > > requires a higher metadata version to enable becomes "production > ready" and > > > the features with unstable metadata versions less than the new stable > one > > > are moved to metadata versions greater than the new stable feature. > When we > > > reorder, we are always allocating a new MV and we are never reusing an > > > existing MV even if it was also unstable. This way a developer > upgrading > > > their environment with a specific unstable MV might see existing > > > functionality stop working but they won't see new MV dependent > > > functionality magically appear. The feature set for a given unstable MV > > > version can only decrease with reordering. > > > > If you use an unstable MV, you probably won't be able to upgrade your > software. Because whenever something changes, you'll probably get > serialization exceptions being thrown inside the controller. Fatal ones. > > > > Given that this is true, there's no reason to have special rules about > what we can and can't do with unstable MVs. We can do anything. > > > > > > > > How do we define "production ready" and when should we bump > > > LATEST_PRODUCTION? I would like to define it to be the point where the > > > feature is code complete with tests and the KIP for it is approved. > However > > > even with this definition if the feature later develops a major issue > it > > > could still block future features until the issue is fixed which is > what we > > > are trying to avoid here. We could be much more formal about this and > let > > > the release manager for a release define what is stable for a given > release > > > and then do the bump just after the branch is created on the branch. > When > > > an RC candidate is accepted, the bump would be backported. I would > like to > > > hear other ideas here. > > > > > > > Yeah, it's an interesting question. Overall, I think developers should > define when a feature is production ready. > > > > The question to ask is, "are you ready to take this feature to > production in your workplace?" I think most developers do have a sense of > this. Obviously bugs and mistakes can happen, but I think this standard > would avoid most of the issues that we're trying to avoid by having > unstable MVs in the first place. > > > > ELR is a good example. Nobody would have said that it was production > ready in 3.7 ... hence it belonged (and still belongs) in an unstable MV, > until that changes (hopefully soon :) ) > > > > best, > > Colin > > > > > --Proven > > > > > > On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe > wrote: > > > > > >> Hi Justine, > > >> > > >> Yes, this is an important point to clarify. Proven can comment more, > but > > >> my understanding is that we can do anything to unstable metadata > versions. > > >> Reorder them, delete them, change them in any other way. There are no > > >> stability guarantees. If the current text is unclear let's add more > > >> examples of what we can do (which is anything) :) > > >> > > >> best, > > >> Colin > > >> > > >> > > >> On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote: > > >> > Hey Colin, > > >> > > > >> > I had some offline discussions with Proven previously and it seems > like > > >> he > > >> > said something different so I'm glad I brought it up here. > > >> > > > >> > Let's clarify if we are ok with reordering unstable metadata > versions :) > > >> > > > >> > Justine > > >> > > > >> > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe > wrote: > > >> > > > >> >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: > > >> >> > Hey all, > > >> >> > > > >> >> > I was wondering how often we plan to update LATEST_PRODUCTION > metadata > > >> >> > version. Is this something we should do as soon as the feature is > > >> >> complete > > >> >> > or something we do when we are releasing kafka. When is
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
We have two approaches here for how we update unstable metadata versions. 1. The update will only increase MVs of unstable features to a value greater than the new stable feature. The idea is that a specific unstable MV may support some set of features and in the future that set is always a strict subset of the current set. The issue is that moving a feature to make way for a stable feature with a higher MV will leave holes. 2. We are free to reorder the MV for any unstable feature. This removes the hole issue, but does make the unstable MVs more muddled. There isn't the same binary state for a MV where a feature is available or there is a hole. We also have two ends of the spectrum as to when we update the stable MV. 1. We update at release points which reduces the amount of churn of the unstable MVs and makes a stronger correlation between accepted features and stable MVs for a release but means less testing on trunk as a stable MV. 2. We update when the developers of a feature think it is done. This leads to features being available for more testing in trunk but forces the next release to include it as stable. I'd like more feedback from others on these two dimensions. --Proven On Wed, Jan 10, 2024 at 12:16 PM Justine Olshan wrote: > Hmm it seems like Colin and Proven are disagreeing with whether we can swap > unstable metadata versions. > > > When we reorder, we are always allocating a new MV and we are never > reusing an existing MV even if it was also unstable. > > > Given that this is true, there's no reason to have special rules about > what we can and can't do with unstable MVs. We can do anything > > I don't have a strong preference either way, but I think we should agree on > one approach. > The benefit of reordering and reusing is that we can release features that > are ready earlier and we have more flexibility. With the approach where we > always create a new MV, I am concerned with having many "empty" MVs. This > would encourage waiting until the release before we decide an incomplete > feature is not ready and moving its MV into the future. (The > abandoning comment I made earlier -- that is consistent with Proven's > approach) > > I think the only potential issue with reordering is that it could be a bit > confusing and *potentially *prone to errors. Note I say potentially because > I think it depends on folks' understanding with this new unstable metadata > version concept. I echo Federico's comments about making sure the risks are > highlighted. > > Thanks, > > Justine > > On Wed, Jan 10, 2024 at 1:16 AM Federico Valeri > wrote: > > > Hi folks, > > > > > If you use an unstable MV, you probably won't be able to upgrade your > > software. Because whenever something changes, you'll probably get > > serialization exceptions being thrown inside the controller. Fatal ones. > > > > Thanks for this clarification. I think this concrete risk should be > > highlighted in the KIP and in the "unstable.metadata.versions.enable" > > documentation. > > > > In the test plan, should we also have one system test checking that > > "features with a stable MV will never have that MV changed"? > > > > On Wed, Jan 10, 2024 at 8:16 AM Colin McCabe wrote: > > > > > > On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote: > > > > Hi folks, > > > > > > > > Thank you for the questions. > > > > > > > > Let me clarify about reorder first. The reorder of unstable metadata > > > > versions should be infrequent. > > > > > > Why does it need to be infrequent? We should be able to reorder > unstable > > metadata versions as often as we like. There are no guarantees about > > unstable MVs. > > > > > > > The time you reorder is when a feature that > > > > requires a higher metadata version to enable becomes "production > > ready" and > > > > the features with unstable metadata versions less than the new stable > > one > > > > are moved to metadata versions greater than the new stable feature. > > When we > > > > reorder, we are always allocating a new MV and we are never reusing > an > > > > existing MV even if it was also unstable. This way a developer > > upgrading > > > > their environment with a specific unstable MV might see existing > > > > functionality stop working but they won't see new MV dependent > > > > functionality magically appear. The feature set for a given unstable > MV > > > > version can only decrease with reordering. > > > > > > If you use an unstable MV, you probably won't be able to upgrade your > > software. Because whenever something changes, you'll probably get > > serialization exceptions being thrown inside the controller. Fatal ones. > > > > > > Given that this is true, there's no reason to have special rules about > > what we can and can't do with unstable MVs. We can do anything. > > > > > > > > > > > How do we define "production ready" and when should we bump > > > > LATEST_PRODUCTION? I would like to define it to be the point where
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi Proven, I'd say that we should do 2 & 2. The idea is that for small features that can be done and stabilized within a short period of time (with one or very few commits) that's exactly what happens -- people interested in testing in-progress feature could take unstable code from a patch (or private branch / fork) with the expectation that that private code could create a state that will not be compatible with anything (or may be completely broken for that matter -- in the end of the day it's a functionality that may not be fully tested or even fully implemented); and once the feature is stable it goes to trunk it is fully committed there, if the bugs are found they'd get fixed "forward". The 2 & 2 option pretty much extends this to large features -- if a feature is above stable MV, then going above it is like getting some in-progress code for early testing with the expectation that something may not fully work or leave system in upgradable state; promoting a feature into a state MV would come with the expectation that the feature gets fully committed and any bugs will be fixed "forward". -Artem On Thu, Jan 11, 2024 at 10:16 AM Proven Provenzano wrote: > We have two approaches here for how we update unstable metadata versions. > >1. The update will only increase MVs of unstable features to a value >greater than the new stable feature. The idea is that a specific > unstable >MV may support some set of features and in the future that set is > always a >strict subset of the current set. The issue is that moving a feature to >make way for a stable feature with a higher MV will leave holes. >2. We are free to reorder the MV for any unstable feature. This removes >the hole issue, but does make the unstable MVs more muddled. There isn't >the same binary state for a MV where a feature is available or there is > a >hole. > > > We also have two ends of the spectrum as to when we update the stable MV. > >1. We update at release points which reduces the amount of churn of the >unstable MVs and makes a stronger correlation between accepted features > and >stable MVs for a release but means less testing on trunk as a stable MV. >2. We update when the developers of a feature think it is done. This >leads to features being available for more testing in trunk but forces > the >next release to include it as stable. > > > I'd like more feedback from others on these two dimensions. > --Proven > > > > On Wed, Jan 10, 2024 at 12:16 PM Justine Olshan > wrote: > > > Hmm it seems like Colin and Proven are disagreeing with whether we can > swap > > unstable metadata versions. > > > > > When we reorder, we are always allocating a new MV and we are never > > reusing an existing MV even if it was also unstable. > > > > > Given that this is true, there's no reason to have special rules about > > what we can and can't do with unstable MVs. We can do anything > > > > I don't have a strong preference either way, but I think we should agree > on > > one approach. > > The benefit of reordering and reusing is that we can release features > that > > are ready earlier and we have more flexibility. With the approach where > we > > always create a new MV, I am concerned with having many "empty" MVs. This > > would encourage waiting until the release before we decide an incomplete > > feature is not ready and moving its MV into the future. (The > > abandoning comment I made earlier -- that is consistent with Proven's > > approach) > > > > I think the only potential issue with reordering is that it could be a > bit > > confusing and *potentially *prone to errors. Note I say potentially > because > > I think it depends on folks' understanding with this new unstable > metadata > > version concept. I echo Federico's comments about making sure the risks > are > > highlighted. > > > > Thanks, > > > > Justine > > > > On Wed, Jan 10, 2024 at 1:16 AM Federico Valeri > > wrote: > > > > > Hi folks, > > > > > > > If you use an unstable MV, you probably won't be able to upgrade your > > > software. Because whenever something changes, you'll probably get > > > serialization exceptions being thrown inside the controller. Fatal > ones. > > > > > > Thanks for this clarification. I think this concrete risk should be > > > highlighted in the KIP and in the "unstable.metadata.versions.enable" > > > documentation. > > > > > > In the test plan, should we also have one system test checking that > > > "features with a stable MV will never have that MV changed"? > > > > > > On Wed, Jan 10, 2024 at 8:16 AM Colin McCabe > wrote: > > > > > > > > On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote: > > > > > Hi folks, > > > > > > > > > > Thank you for the questions. > > > > > > > > > > Let me clarify about reorder first. The reorder of unstable > metadata > > > > > versions should be infrequent. > > > > > > > > Why does it need to be infrequent? We should be able to reorder > > unstable > > > metadat
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
On Thu, Jan 11, 2024 at 10:43 PM Artem Livshits wrote: > > Hi Proven, > > I'd say that we should do 2 & 2. The idea is that for small features that > can be done and stabilized within a short period of time (with one or very > few commits) that's exactly what happens -- people interested in testing > in-progress feature could take unstable code from a patch (or private > branch / fork) with the expectation that that private code could create a > state that will not be compatible with anything (or may be completely > broken for that matter -- in the end of the day it's a functionality that > may not be fully tested or even fully implemented); and once the feature is > stable it goes to trunk it is fully committed there, if the bugs are found > they'd get fixed "forward". I agree with this reasoning. > The 2 & 2 option pretty much extends this to > large features -- if a feature is above stable MV, then going above it is > like getting some in-progress code for early testing with the expectation > that something may not fully work or leave system in upgradable state; Usually I expect that an early access feature may not fully work, but not that it could affect upgrades. I think this is less obvious, that's why I asked to document clearly. > promoting a feature into a state MV would come with the expectation that > the feature gets fully committed and any bugs will be fixed "forward". > > -Artem > > On Thu, Jan 11, 2024 at 10:16 AM Proven Provenzano > wrote: > > > We have two approaches here for how we update unstable metadata versions. > > > >1. The update will only increase MVs of unstable features to a value > >greater than the new stable feature. The idea is that a specific > > unstable > >MV may support some set of features and in the future that set is > > always a > >strict subset of the current set. The issue is that moving a feature to > >make way for a stable feature with a higher MV will leave holes. > >2. We are free to reorder the MV for any unstable feature. This removes > >the hole issue, but does make the unstable MVs more muddled. There isn't > >the same binary state for a MV where a feature is available or there is > > a > >hole. > > > > > > We also have two ends of the spectrum as to when we update the stable MV. > > > >1. We update at release points which reduces the amount of churn of the > >unstable MVs and makes a stronger correlation between accepted features > > and > >stable MVs for a release but means less testing on trunk as a stable MV. > >2. We update when the developers of a feature think it is done. This > >leads to features being available for more testing in trunk but forces > > the > >next release to include it as stable. > > > > > > I'd like more feedback from others on these two dimensions. > > --Proven > > > > > > > > On Wed, Jan 10, 2024 at 12:16 PM Justine Olshan > > wrote: > > > > > Hmm it seems like Colin and Proven are disagreeing with whether we can > > swap > > > unstable metadata versions. > > > > > > > When we reorder, we are always allocating a new MV and we are never > > > reusing an existing MV even if it was also unstable. > > > > > > > Given that this is true, there's no reason to have special rules about > > > what we can and can't do with unstable MVs. We can do anything > > > > > > I don't have a strong preference either way, but I think we should agree > > on > > > one approach. > > > The benefit of reordering and reusing is that we can release features > > that > > > are ready earlier and we have more flexibility. With the approach where > > we > > > always create a new MV, I am concerned with having many "empty" MVs. This > > > would encourage waiting until the release before we decide an incomplete > > > feature is not ready and moving its MV into the future. (The > > > abandoning comment I made earlier -- that is consistent with Proven's > > > approach) > > > > > > I think the only potential issue with reordering is that it could be a > > bit > > > confusing and *potentially *prone to errors. Note I say potentially > > because > > > I think it depends on folks' understanding with this new unstable > > metadata > > > version concept. I echo Federico's comments about making sure the risks > > are > > > highlighted. > > > > > > Thanks, > > > > > > Justine > > > > > > On Wed, Jan 10, 2024 at 1:16 AM Federico Valeri > > > wrote: > > > > > > > Hi folks, > > > > > > > > > If you use an unstable MV, you probably won't be able to upgrade your > > > > software. Because whenever something changes, you'll probably get > > > > serialization exceptions being thrown inside the controller. Fatal > > ones. > > > > > > > > Thanks for this clarification. I think this concrete risk should be > > > > highlighted in the KIP and in the "unstable.metadata.versions.enable" > > > > documentation. > > > > > > > > In the test plan, should we also have one system test checking that > > > > "features with a sta
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
I also agree with 2 & 2 with reasoning along the same lines as Artem. Thanks, Andrew > On 12 Jan 2024, at 09:15, Federico Valeri wrote: > > On Thu, Jan 11, 2024 at 10:43 PM Artem Livshits > wrote: >> >> Hi Proven, >> >> I'd say that we should do 2 & 2. The idea is that for small features that >> can be done and stabilized within a short period of time (with one or very >> few commits) that's exactly what happens -- people interested in testing >> in-progress feature could take unstable code from a patch (or private >> branch / fork) with the expectation that that private code could create a >> state that will not be compatible with anything (or may be completely >> broken for that matter -- in the end of the day it's a functionality that >> may not be fully tested or even fully implemented); and once the feature is >> stable it goes to trunk it is fully committed there, if the bugs are found >> they'd get fixed "forward". > > I agree with this reasoning. > >> The 2 & 2 option pretty much extends this to >> large features -- if a feature is above stable MV, then going above it is >> like getting some in-progress code for early testing with the expectation >> that something may not fully work or leave system in upgradable state; > > Usually I expect that an early access feature may not fully work, but > not that it could affect upgrades. I think this is less obvious, > that's why I asked to document clearly. > >> promoting a feature into a state MV would come with the expectation that >> the feature gets fully committed and any bugs will be fixed "forward". >> >> -Artem >> >> On Thu, Jan 11, 2024 at 10:16 AM Proven Provenzano >> wrote: >> >>> We have two approaches here for how we update unstable metadata versions. >>> >>> 1. The update will only increase MVs of unstable features to a value >>> greater than the new stable feature. The idea is that a specific >>> unstable >>> MV may support some set of features and in the future that set is >>> always a >>> strict subset of the current set. The issue is that moving a feature to >>> make way for a stable feature with a higher MV will leave holes. >>> 2. We are free to reorder the MV for any unstable feature. This removes >>> the hole issue, but does make the unstable MVs more muddled. There isn't >>> the same binary state for a MV where a feature is available or there is >>> a >>> hole. >>> >>> >>> We also have two ends of the spectrum as to when we update the stable MV. >>> >>> 1. We update at release points which reduces the amount of churn of the >>> unstable MVs and makes a stronger correlation between accepted features >>> and >>> stable MVs for a release but means less testing on trunk as a stable MV. >>> 2. We update when the developers of a feature think it is done. This >>> leads to features being available for more testing in trunk but forces >>> the >>> next release to include it as stable. >>> >>> >>> I'd like more feedback from others on these two dimensions. >>> --Proven >>> >>> >>> >>> On Wed, Jan 10, 2024 at 12:16 PM Justine Olshan >>> wrote: >>> Hmm it seems like Colin and Proven are disagreeing with whether we can >>> swap unstable metadata versions. > When we reorder, we are always allocating a new MV and we are never reusing an existing MV even if it was also unstable. > Given that this is true, there's no reason to have special rules about what we can and can't do with unstable MVs. We can do anything I don't have a strong preference either way, but I think we should agree >>> on one approach. The benefit of reordering and reusing is that we can release features >>> that are ready earlier and we have more flexibility. With the approach where >>> we always create a new MV, I am concerned with having many "empty" MVs. This would encourage waiting until the release before we decide an incomplete feature is not ready and moving its MV into the future. (The abandoning comment I made earlier -- that is consistent with Proven's approach) I think the only potential issue with reordering is that it could be a >>> bit confusing and *potentially *prone to errors. Note I say potentially >>> because I think it depends on folks' understanding with this new unstable >>> metadata version concept. I echo Federico's comments about making sure the risks >>> are highlighted. Thanks, Justine On Wed, Jan 10, 2024 at 1:16 AM Federico Valeri wrote: > Hi folks, > >> If you use an unstable MV, you probably won't be able to upgrade your > software. Because whenever something changes, you'll probably get > serialization exceptions being thrown inside the controller. Fatal >>> ones. > > Thanks for this clarification. I think this concrete risk should be > highlighted in the KIP and in the "unstable.metadata.version
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi, Thanks for the KIP. Reading the discussion, I think a lot of the confusion is due to the "unstable" naming. People are then trying to figure out when we think something is stable in the "this is battle-tested" sense. But this flag should not be about that. We can have an MV for a feature that is not yet production-ready (and we did that when KRaft itself was not production ready). I think this flag is about metadata versions that are basically unsupported - if you use it, you get to keep the pieces. They exist solely to make the lives of Apache Kafka developers easier. I would even suggest that the config we introduce be of the internal variety, ie it won't show in the generated documentation and there won't be any compatibility guarantee. Thoughts? Ismael On Fri, Jan 5, 2024 at 7:33 AM Proven Provenzano wrote: > Hey folks, > > I am starting a discussion thread for managing unstable metadata versions > in Apache Kafka. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > This KIP is actually already implemented in 3.7 with PR > https://github.com/apache/kafka/pull/14860. > I have created this KIP to explain the motivation and how managing Metadata > Versions is expected to work. > Comments are greatly appreciated as this process can always be improved. > > -- > --Proven >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi Ismael, I think the concern I have about a MV for a feature that is not production ready is that it blocks any development/features with higher MV versions that could be production ready. I do see your point though. Previously MV/IBP was about pure broker compatibility and not about the confidence in the feature it is gating. I do wonder though if it could be useful to have that sort of gating. I also wonder if an internal config could be useful so that the average user doesn't have to worry about the complexities of unstable metadata versions (and their risks). I am ok with options 2 and 2 as well by the way. Justine On Fri, Jan 12, 2024 at 7:36 AM Ismael Juma wrote: > Hi, > > Thanks for the KIP. > > Reading the discussion, I think a lot of the confusion is due to the > "unstable" naming. People are then trying to figure out when we think > something is stable in the "this is battle-tested" sense. But this flag > should not be about that. We can have an MV for a feature that is not yet > production-ready (and we did that when KRaft itself was not production > ready). I think this flag is about metadata versions that are basically > unsupported - if you use it, you get to keep the pieces. They exist solely > to make the lives of Apache Kafka developers easier. I would even suggest > that the config we introduce be of the internal variety, ie it won't show > in the generated documentation and there won't be any compatibility > guarantee. > > Thoughts? > > Ismael > > On Fri, Jan 5, 2024 at 7:33 AM Proven Provenzano > wrote: > > > Hey folks, > > > > I am starting a discussion thread for managing unstable metadata versions > > in Apache Kafka. > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > > > This KIP is actually already implemented in 3.7 with PR > > https://github.com/apache/kafka/pull/14860. > > I have created this KIP to explain the motivation and how managing > Metadata > > Versions is expected to work. > > Comments are greatly appreciated as this process can always be improved. > > > > -- > > --Proven > > >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Justine, For features that are not production-ready, they should have an additional configuration (not the metadata version) that enables/disables it. The MV specific features we ship are something we have to support and make sure we don't break going forward. Ismael On Fri, Jan 12, 2024 at 9:26 AM Justine Olshan wrote: > Hi Ismael, > > I think the concern I have about a MV for a feature that is not production > ready is that it blocks any development/features with higher MV versions > that could be production ready. > > I do see your point though. Previously MV/IBP was about pure broker > compatibility and not about the confidence in the feature it is gating. I > do wonder though if it could be useful to have that sort of gating. > I also wonder if an internal config could be useful so that the average > user doesn't have to worry about the complexities of unstable metadata > versions (and their risks). > > I am ok with options 2 and 2 as well by the way. > > Justine > > On Fri, Jan 12, 2024 at 7:36 AM Ismael Juma wrote: > > > Hi, > > > > Thanks for the KIP. > > > > Reading the discussion, I think a lot of the confusion is due to the > > "unstable" naming. People are then trying to figure out when we think > > something is stable in the "this is battle-tested" sense. But this flag > > should not be about that. We can have an MV for a feature that is not yet > > production-ready (and we did that when KRaft itself was not production > > ready). I think this flag is about metadata versions that are basically > > unsupported - if you use it, you get to keep the pieces. They exist > solely > > to make the lives of Apache Kafka developers easier. I would even suggest > > that the config we introduce be of the internal variety, ie it won't show > > in the generated documentation and there won't be any compatibility > > guarantee. > > > > Thoughts? > > > > Ismael > > > > On Fri, Jan 5, 2024 at 7:33 AM Proven Provenzano > > wrote: > > > > > Hey folks, > > > > > > I am starting a discussion thread for managing unstable metadata > versions > > > in Apache Kafka. > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > > > > > This KIP is actually already implemented in 3.7 with PR > > > https://github.com/apache/kafka/pull/14860. > > > I have created this KIP to explain the motivation and how managing > > Metadata > > > Versions is expected to work. > > > Comments are greatly appreciated as this process can always be > improved. > > > > > > -- > > > --Proven > > > > > >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi Ismael, I debated including something about feature flags in my last comment, but maybe I should have. What you said makes sense. Justine On Fri, Jan 12, 2024 at 9:31 AM Ismael Juma wrote: > Justine, > > For features that are not production-ready, they should have an additional > configuration (not the metadata version) that enables/disables it. The MV > specific features we ship are something we have to support and make sure we > don't break going forward. > > Ismael > > On Fri, Jan 12, 2024 at 9:26 AM Justine Olshan > > wrote: > > > Hi Ismael, > > > > I think the concern I have about a MV for a feature that is not > production > > ready is that it blocks any development/features with higher MV versions > > that could be production ready. > > > > I do see your point though. Previously MV/IBP was about pure broker > > compatibility and not about the confidence in the feature it is gating. I > > do wonder though if it could be useful to have that sort of gating. > > I also wonder if an internal config could be useful so that the average > > user doesn't have to worry about the complexities of unstable metadata > > versions (and their risks). > > > > I am ok with options 2 and 2 as well by the way. > > > > Justine > > > > On Fri, Jan 12, 2024 at 7:36 AM Ismael Juma wrote: > > > > > Hi, > > > > > > Thanks for the KIP. > > > > > > Reading the discussion, I think a lot of the confusion is due to the > > > "unstable" naming. People are then trying to figure out when we think > > > something is stable in the "this is battle-tested" sense. But this flag > > > should not be about that. We can have an MV for a feature that is not > yet > > > production-ready (and we did that when KRaft itself was not production > > > ready). I think this flag is about metadata versions that are basically > > > unsupported - if you use it, you get to keep the pieces. They exist > > solely > > > to make the lives of Apache Kafka developers easier. I would even > suggest > > > that the config we introduce be of the internal variety, ie it won't > show > > > in the generated documentation and there won't be any compatibility > > > guarantee. > > > > > > Thoughts? > > > > > > Ismael > > > > > > On Fri, Jan 5, 2024 at 7:33 AM Proven Provenzano > > > wrote: > > > > > > > Hey folks, > > > > > > > > I am starting a discussion thread for managing unstable metadata > > versions > > > > in Apache Kafka. > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > > > > > > > This KIP is actually already implemented in 3.7 with PR > > > > https://github.com/apache/kafka/pull/14860. > > > > I have created this KIP to explain the motivation and how managing > > > Metadata > > > > Versions is expected to work. > > > > Comments are greatly appreciated as this process can always be > > improved. > > > > > > > > -- > > > > --Proven > > > > > > > > > >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
I think using feature flags (whether we support a framework and tooling for feature flags or just an ad-hoc XyzEnabled flag) can be an alternative to this KIP. I think the value of this KIP is that it's trying to propose a systemic approach for gating functionality that may take multiple releases to develop. A problem with ad-hoc feature flags is that it's useful during feature development, so that people who are working on this feature (or are interested in beta-testing the feature) can get early access (without any guarantees on compatibility or even correctness); but then the feature flags often stick forever after the feature development is done (and as time moves one, the new code is written in such a way that it's not possible to turn the feature off any more cleanly). I'd also clarify how I think about "stable". Ismael made a comment " something is stable in the "this is battle-tested" sense.". I don't think it has to be "battle-tested", it just has to meet the bar of being in the trunk. Again, thinking of a small single-commit feature -- to commit to trunk, the feature doesn't have to be "battle-tested", but it should be complete (and not just a bunch of TODOs), with tests written and some level of dev-testing done, so that once the release is cut, we can find and fix bugs and make it release-quality (as opposed to reverting the whole thing). We can apply the same "stability" bar for features to be in the stable MV -- fully complete, tests written and some level of dev testing done. -Artem On Fri, Jan 12, 2024 at 10:12 AM Justine Olshan wrote: > Hi Ismael, > > I debated including something about feature flags in my last comment, but > maybe I should have. > What you said makes sense. > > Justine > > On Fri, Jan 12, 2024 at 9:31 AM Ismael Juma wrote: > > > Justine, > > > > For features that are not production-ready, they should have an > additional > > configuration (not the metadata version) that enables/disables it. The MV > > specific features we ship are something we have to support and make sure > we > > don't break going forward. > > > > Ismael > > > > On Fri, Jan 12, 2024 at 9:26 AM Justine Olshan > > > > wrote: > > > > > Hi Ismael, > > > > > > I think the concern I have about a MV for a feature that is not > > production > > > ready is that it blocks any development/features with higher MV > versions > > > that could be production ready. > > > > > > I do see your point though. Previously MV/IBP was about pure broker > > > compatibility and not about the confidence in the feature it is > gating. I > > > do wonder though if it could be useful to have that sort of gating. > > > I also wonder if an internal config could be useful so that the average > > > user doesn't have to worry about the complexities of unstable metadata > > > versions (and their risks). > > > > > > I am ok with options 2 and 2 as well by the way. > > > > > > Justine > > > > > > On Fri, Jan 12, 2024 at 7:36 AM Ismael Juma wrote: > > > > > > > Hi, > > > > > > > > Thanks for the KIP. > > > > > > > > Reading the discussion, I think a lot of the confusion is due to the > > > > "unstable" naming. People are then trying to figure out when we think > > > > something is stable in the "this is battle-tested" sense. But this > flag > > > > should not be about that. We can have an MV for a feature that is not > > yet > > > > production-ready (and we did that when KRaft itself was not > production > > > > ready). I think this flag is about metadata versions that are > basically > > > > unsupported - if you use it, you get to keep the pieces. They exist > > > solely > > > > to make the lives of Apache Kafka developers easier. I would even > > suggest > > > > that the config we introduce be of the internal variety, ie it won't > > show > > > > in the generated documentation and there won't be any compatibility > > > > guarantee. > > > > > > > > Thoughts? > > > > > > > > Ismael > > > > > > > > On Fri, Jan 5, 2024 at 7:33 AM Proven Provenzano > > > > wrote: > > > > > > > > > Hey folks, > > > > > > > > > > I am starting a discussion thread for managing unstable metadata > > > versions > > > > > in Apache Kafka. > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka > > > > > > > > > > This KIP is actually already implemented in 3.7 with PR > > > > > https://github.com/apache/kafka/pull/14860. > > > > > I have created this KIP to explain the motivation and how managing > > > > Metadata > > > > > Versions is expected to work. > > > > > Comments are greatly appreciated as this process can always be > > > improved. > > > > > > > > > > -- > > > > > --Proven > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > I think using feature flags (whether we support a framework and tooling for > feature flags or just an ad-hoc XyzEnabled flag) can be an alternative to > this KIP. I think the value of this KIP is that it's trying to propose a > systemic approach for gating functionality that may take multiple releases > to develop. A problem with ad-hoc feature flags is that it's useful during > feature development, so that people who are working on this feature (or are > interested in beta-testing the feature) can get early access (without any > guarantees on compatibility or even correctness); but then the feature > flags often stick forever after the feature development is done (and as > time moves one, the new code is written in such a way that it's not > possible to turn the feature off any more cleanly). > Hi Artem, I think feature flags are somewhat orthogonal to the stable / unstable discussion. Even if every new feature was a feature flag, you probably still wouldn't want to stabilize the features immediately, to avoid maintaining a lot of alpha stuff forever. (I also think that feature flags should be used sparingly, if at all, because of the way that they exponentially increase the test matrix. But that's a tangent, I think, given the first point...) > > I'd also clarify how I think about "stable". Ismael made a comment " > something is stable in the "this is battle-tested" sense.". I don't think > it has to be "battle-tested", it just has to meet the bar of being in the > trunk. Again, thinking of a small single-commit feature -- to commit to > trunk, the feature doesn't have to be "battle-tested", but it should be > complete (and not just a bunch of TODOs), with tests written and some level > of dev-testing done, so that once the release is cut, we can find and fix > bugs and make it release-quality (as opposed to reverting the whole > thing). We can apply the same "stability" bar for features to be in the > stable MV -- fully complete, tests written and some level of dev testing > done. > I'm struggling a bit with your phrasing. Are you suggesting that unstable MVs would not be able to be in trunk? I think we do want them to be able to go into trunk. If they have to go into a branch, then there is actually no need for any of this. Doing big features in long-lived branches is one genuine alternative to unstable MVs, I think. But it's not an alternative that works well with our current tooling for continuous integration, deployment, building, etc. I think it would also impact developer productivity somewhat negatively. best, Colin > > -Artem > > On Fri, Jan 12, 2024 at 10:12 AM Justine Olshan > wrote: > >> Hi Ismael, >> >> I debated including something about feature flags in my last comment, but >> maybe I should have. >> What you said makes sense. >> >> Justine >> >> On Fri, Jan 12, 2024 at 9:31 AM Ismael Juma wrote: >> >> > Justine, >> > >> > For features that are not production-ready, they should have an >> additional >> > configuration (not the metadata version) that enables/disables it. The MV >> > specific features we ship are something we have to support and make sure >> we >> > don't break going forward. >> > >> > Ismael >> > >> > On Fri, Jan 12, 2024 at 9:26 AM Justine Olshan >> > >> > wrote: >> > >> > > Hi Ismael, >> > > >> > > I think the concern I have about a MV for a feature that is not >> > production >> > > ready is that it blocks any development/features with higher MV >> versions >> > > that could be production ready. >> > > >> > > I do see your point though. Previously MV/IBP was about pure broker >> > > compatibility and not about the confidence in the feature it is >> gating. I >> > > do wonder though if it could be useful to have that sort of gating. >> > > I also wonder if an internal config could be useful so that the average >> > > user doesn't have to worry about the complexities of unstable metadata >> > > versions (and their risks). >> > > >> > > I am ok with options 2 and 2 as well by the way. >> > > >> > > Justine >> > > >> > > On Fri, Jan 12, 2024 at 7:36 AM Ismael Juma wrote: >> > > >> > > > Hi, >> > > > >> > > > Thanks for the KIP. >> > > > >> > > > Reading the discussion, I think a lot of the confusion is due to the >> > > > "unstable" naming. People are then trying to figure out when we think >> > > > something is stable in the "this is battle-tested" sense. But this >> flag >> > > > should not be about that. We can have an MV for a feature that is not >> > yet >> > > > production-ready (and we did that when KRaft itself was not >> production >> > > > ready). I think this flag is about metadata versions that are >> basically >> > > > unsupported - if you use it, you get to keep the pieces. They exist >> > > solely >> > > > to make the lives of Apache Kafka developers easier. I would even >> > suggest >> > > > that the config we introduce be of the internal variety, ie it won't >> > show >> >
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi Colin, > I think feature flags are somewhat orthogonal to the stable / unstable discussion I think feature flags can be used as an alternative to achieve similar results as stable / unstable functionality. As well as long-lived feature branches. In my experience, I've seen feature flags to be more successful than feature branches for changes of existing functionality. I also think that stable / unstable MV would work better than feature branches. I just wanted to mention it for completeness, not sure if we should start a full fledged discussion on these topics. > I'm struggling a bit with your phrasing. Are you suggesting that unstable MVs would not be able to be in trunk? Unstable MV should be in trunk. The wording is related to when we can promote "unstable" to "stable". -Artem On Mon, Jan 15, 2024 at 10:03 PM Colin McCabe wrote: > On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > > I think using feature flags (whether we support a framework and tooling > for > > feature flags or just an ad-hoc XyzEnabled flag) can be an alternative to > > this KIP. I think the value of this KIP is that it's trying to propose a > > systemic approach for gating functionality that may take multiple > releases > > to develop. A problem with ad-hoc feature flags is that it's useful > during > > feature development, so that people who are working on this feature (or > are > > interested in beta-testing the feature) can get early access (without any > > guarantees on compatibility or even correctness); but then the feature > > flags often stick forever after the feature development is done (and as > > time moves one, the new code is written in such a way that it's not > > possible to turn the feature off any more cleanly). > > > > Hi Artem, > > I think feature flags are somewhat orthogonal to the stable / unstable > discussion. Even if every new feature was a feature flag, you probably > still wouldn't want to stabilize the features immediately, to avoid > maintaining a lot of alpha stuff forever. > > (I also think that feature flags should be used sparingly, if at all, > because of the way that they exponentially increase the test matrix. But > that's a tangent, I think, given the first point...) > > > > > I'd also clarify how I think about "stable". Ismael made a comment " > > something is stable in the "this is battle-tested" sense.". I don't > think > > it has to be "battle-tested", it just has to meet the bar of being in the > > trunk. Again, thinking of a small single-commit feature -- to commit to > > trunk, the feature doesn't have to be "battle-tested", but it should be > > complete (and not just a bunch of TODOs), with tests written and some > level > > of dev-testing done, so that once the release is cut, we can find and fix > > bugs and make it release-quality (as opposed to reverting the whole > > thing). We can apply the same "stability" bar for features to be in the > > stable MV -- fully complete, tests written and some level of dev testing > > done. > > > > I'm struggling a bit with your phrasing. Are you suggesting that unstable > MVs would not be able to be in trunk? I think we do want them to be able to > go into trunk. If they have to go into a branch, then there is actually no > need for any of this. > > Doing big features in long-lived branches is one genuine alternative to > unstable MVs, I think. But it's not an alternative that works well with our > current tooling for continuous integration, deployment, building, etc. I > think it would also impact developer productivity somewhat negatively. > > best, > Colin > > > > > > -Artem > > > > On Fri, Jan 12, 2024 at 10:12 AM Justine Olshan > > wrote: > > > >> Hi Ismael, > >> > >> I debated including something about feature flags in my last comment, > but > >> maybe I should have. > >> What you said makes sense. > >> > >> Justine > >> > >> On Fri, Jan 12, 2024 at 9:31 AM Ismael Juma wrote: > >> > >> > Justine, > >> > > >> > For features that are not production-ready, they should have an > >> additional > >> > configuration (not the metadata version) that enables/disables it. > The MV > >> > specific features we ship are something we have to support and make > sure > >> we > >> > don't break going forward. > >> > > >> > Ismael > >> > > >> > On Fri, Jan 12, 2024 at 9:26 AM Justine Olshan > >> > > >> > wrote: > >> > > >> > > Hi Ismael, > >> > > > >> > > I think the concern I have about a MV for a feature that is not > >> > production > >> > > ready is that it blocks any development/features with higher MV > >> versions > >> > > that could be production ready. > >> > > > >> > > I do see your point though. Previously MV/IBP was about pure broker > >> > > compatibility and not about the confidence in the feature it is > >> gating. I > >> > > do wonder though if it could be useful to have that sort of gating. > >> > > I also wonder if an internal config could be useful so that the > average > >> > > user doesn't have to worry about t
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi, Proven, Thanks for the KIP. I am not sure about the reordering approach proposed in the KIP. Let's say in a release we have features X and Y, depending on MV IV1 and IV2, respectively. At the release time, feature Y is ready, but X is not. I guess the proposal is to move IV1 to a new MV IV3? The issue is that IV2 could have made changes on top of IV1. For example, IV2 could have evolved the schema of the same inter broker request as IV1. In that case, what does IV3 represent? We can't simply take the changes associated with IV1 since it could have conflicts with IV2. Even when there are no conflicts, I am not sure if the approach supports the trunk deployment model. Let's say we move two unstable MV IV3 and IV4 to IV7 and IV8. If someone deployed code corresponding to IV4 and later upgraded the code to IV7, he/she (1) wouldn't be able to set IV4 in the upgrade code since it no longer exists and (2) would be surprised that the request protocol that worked under IV4 doesn't work now. Thanks, Jun On Fri, Jan 19, 2024 at 2:13 PM Artem Livshits wrote: > Hi Colin, > > > I think feature flags are somewhat orthogonal to the stable / unstable > discussion > > I think feature flags can be used as an alternative to achieve similar > results as stable / unstable functionality. As well as long-lived feature > branches. In my experience, I've seen feature flags to be more successful > than feature branches for changes of existing functionality. I also think > that stable / unstable MV would work better than feature branches. I just > wanted to mention it for completeness, not sure if we should start a full > fledged discussion on these topics. > > > I'm struggling a bit with your phrasing. Are you suggesting that unstable > MVs would not be able to be in trunk? > > Unstable MV should be in trunk. The wording is related to when we can > promote "unstable" to "stable". > > -Artem > > > On Mon, Jan 15, 2024 at 10:03 PM Colin McCabe wrote: > > > On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > > > I think using feature flags (whether we support a framework and tooling > > for > > > feature flags or just an ad-hoc XyzEnabled flag) can be an alternative > to > > > this KIP. I think the value of this KIP is that it's trying to > propose a > > > systemic approach for gating functionality that may take multiple > > releases > > > to develop. A problem with ad-hoc feature flags is that it's useful > > during > > > feature development, so that people who are working on this feature (or > > are > > > interested in beta-testing the feature) can get early access (without > any > > > guarantees on compatibility or even correctness); but then the feature > > > flags often stick forever after the feature development is done (and as > > > time moves one, the new code is written in such a way that it's not > > > possible to turn the feature off any more cleanly). > > > > > > > Hi Artem, > > > > I think feature flags are somewhat orthogonal to the stable / unstable > > discussion. Even if every new feature was a feature flag, you probably > > still wouldn't want to stabilize the features immediately, to avoid > > maintaining a lot of alpha stuff forever. > > > > (I also think that feature flags should be used sparingly, if at all, > > because of the way that they exponentially increase the test matrix. But > > that's a tangent, I think, given the first point...) > > > > > > > > I'd also clarify how I think about "stable". Ismael made a comment " > > > something is stable in the "this is battle-tested" sense.". I don't > > think > > > it has to be "battle-tested", it just has to meet the bar of being in > the > > > trunk. Again, thinking of a small single-commit feature -- to commit > to > > > trunk, the feature doesn't have to be "battle-tested", but it should be > > > complete (and not just a bunch of TODOs), with tests written and some > > level > > > of dev-testing done, so that once the release is cut, we can find and > fix > > > bugs and make it release-quality (as opposed to reverting the whole > > > thing). We can apply the same "stability" bar for features to be in > the > > > stable MV -- fully complete, tests written and some level of dev > testing > > > done. > > > > > > > I'm struggling a bit with your phrasing. Are you suggesting that unstable > > MVs would not be able to be in trunk? I think we do want them to be able > to > > go into trunk. If they have to go into a branch, then there is actually > no > > need for any of this. > > > > Doing big features in long-lived branches is one genuine alternative to > > unstable MVs, I think. But it's not an alternative that works well with > our > > current tooling for continuous integration, deployment, building, etc. I > > think it would also impact developer productivity somewhat negatively. > > > > best, > > Colin > > > > > > > > > > -Artem > > > > > > On Fri, Jan 12, 2024 at 10:12 AM Justine Olshan > > > wrote: > > > > > >> Hi Ismael, > > >> > > >>
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
On Tue, Jan 23, 2024, at 11:21, Jun Rao wrote: > Hi, Proven, > > Thanks for the KIP. > > I am not sure about the reordering approach proposed in the KIP. Let's say > in a release we have features X and Y, depending on MV IV1 and IV2, > respectively. At the release time, feature Y is ready, but X is not. I > guess the proposal is to move IV1 to a new MV IV3? Hi Jun, In your example, if X is not ready, it should be moved into an unstable MV. Then if Y is ready, it can be moved into a stable MV and we can advance the last stable MV. > The issue is that IV2 > could have made changes on top of IV1. For example, IV2 could have evolved > the schema of the same inter broker request as IV1. In that case, what does > IV3 represent? We can't simply take the changes associated with IV1 since > it could have conflicts with IV2. If Y changed the same file as X, we can just edit the file so that Y's change come first. Nobody using a stable MV should be affected, since they will be generating records on an older (stable) version. > Even when there are no conflicts, I am not sure if the approach supports > the trunk deployment model. Let's say we move two unstable MV IV3 and IV4 > to IV7 and IV8. If someone deployed code corresponding to IV4 and later > upgraded the code to IV7, he/she (1) wouldn't be able to set IV4 in the > upgrade code since it no longer exists and (2) would be surprised that the > request protocol that worked under IV4 doesn't work now. > Upgrades are not supported if you are running an unstable MV. It's intended only for testing. Colin > > Thanks, > > Jun > > On Fri, Jan 19, 2024 at 2:13 PM Artem Livshits > wrote: > >> Hi Colin, >> >> > I think feature flags are somewhat orthogonal to the stable / unstable >> discussion >> >> I think feature flags can be used as an alternative to achieve similar >> results as stable / unstable functionality. As well as long-lived feature >> branches. In my experience, I've seen feature flags to be more successful >> than feature branches for changes of existing functionality. I also think >> that stable / unstable MV would work better than feature branches. I just >> wanted to mention it for completeness, not sure if we should start a full >> fledged discussion on these topics. >> >> > I'm struggling a bit with your phrasing. Are you suggesting that unstable >> MVs would not be able to be in trunk? >> >> Unstable MV should be in trunk. The wording is related to when we can >> promote "unstable" to "stable". >> >> -Artem >> >> >> On Mon, Jan 15, 2024 at 10:03 PM Colin McCabe wrote: >> >> > On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: >> > > I think using feature flags (whether we support a framework and tooling >> > for >> > > feature flags or just an ad-hoc XyzEnabled flag) can be an alternative >> to >> > > this KIP. I think the value of this KIP is that it's trying to >> propose a >> > > systemic approach for gating functionality that may take multiple >> > releases >> > > to develop. A problem with ad-hoc feature flags is that it's useful >> > during >> > > feature development, so that people who are working on this feature (or >> > are >> > > interested in beta-testing the feature) can get early access (without >> any >> > > guarantees on compatibility or even correctness); but then the feature >> > > flags often stick forever after the feature development is done (and as >> > > time moves one, the new code is written in such a way that it's not >> > > possible to turn the feature off any more cleanly). >> > > >> > >> > Hi Artem, >> > >> > I think feature flags are somewhat orthogonal to the stable / unstable >> > discussion. Even if every new feature was a feature flag, you probably >> > still wouldn't want to stabilize the features immediately, to avoid >> > maintaining a lot of alpha stuff forever. >> > >> > (I also think that feature flags should be used sparingly, if at all, >> > because of the way that they exponentially increase the test matrix. But >> > that's a tangent, I think, given the first point...) >> > >> > > >> > > I'd also clarify how I think about "stable". Ismael made a comment " >> > > something is stable in the "this is battle-tested" sense.". I don't >> > think >> > > it has to be "battle-tested", it just has to meet the bar of being in >> the >> > > trunk. Again, thinking of a small single-commit feature -- to commit >> to >> > > trunk, the feature doesn't have to be "battle-tested", but it should be >> > > complete (and not just a bunch of TODOs), with tests written and some >> > level >> > > of dev-testing done, so that once the release is cut, we can find and >> fix >> > > bugs and make it release-quality (as opposed to reverting the whole >> > > thing). We can apply the same "stability" bar for features to be in >> the >> > > stable MV -- fully complete, tests written and some level of dev >> testing >> > > done. >> > > >> > >> > I'm struggling a bit with your phrasing. Are you suggesting that unstabl
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Hi, Colin, Thanks for the reply. "If Y changed the same file as X, we can just edit the file so that Y's change come first." It seems the effort is more than just editing the file. One has to change all the logic around the old IV. Also, what happens to the client? A client may have implemented a version for a request corresponding to an unstable feature. When the unstable feature is re-ordered, do all clients (including those non-java ones) need to change the PRC implementation for the existing underlying requests? Jun On Mon, Jan 29, 2024 at 9:57 AM Colin McCabe wrote: > On Tue, Jan 23, 2024, at 11:21, Jun Rao wrote: > > Hi, Proven, > > > > Thanks for the KIP. > > > > I am not sure about the reordering approach proposed in the KIP. Let's > say > > in a release we have features X and Y, depending on MV IV1 and IV2, > > respectively. At the release time, feature Y is ready, but X is not. I > > guess the proposal is to move IV1 to a new MV IV3? > > Hi Jun, > > In your example, if X is not ready, it should be moved into an unstable > MV. Then if Y is ready, it can be moved into a stable MV and we can advance > the last stable MV. > > > The issue is that IV2 > > could have made changes on top of IV1. For example, IV2 could have > evolved > > the schema of the same inter broker request as IV1. In that case, what > does > > IV3 represent? We can't simply take the changes associated with IV1 since > > it could have conflicts with IV2. > > If Y changed the same file as X, we can just edit the file so that Y's > change come first. > > Nobody using a stable MV should be affected, since they will be generating > records on an older (stable) version. > > > Even when there are no conflicts, I am not sure if the approach supports > > the trunk deployment model. Let's say we move two unstable MV IV3 and IV4 > > to IV7 and IV8. If someone deployed code corresponding to IV4 and later > > upgraded the code to IV7, he/she (1) wouldn't be able to set IV4 in the > > upgrade code since it no longer exists and (2) would be surprised that > the > > request protocol that worked under IV4 doesn't work now. > > > > Upgrades are not supported if you are running an unstable MV. It's > intended only for testing. > > Colin > > > > > Thanks, > > > > Jun > > > > On Fri, Jan 19, 2024 at 2:13 PM Artem Livshits > > wrote: > > > >> Hi Colin, > >> > >> > I think feature flags are somewhat orthogonal to the stable / > unstable > >> discussion > >> > >> I think feature flags can be used as an alternative to achieve similar > >> results as stable / unstable functionality. As well as long-lived > feature > >> branches. In my experience, I've seen feature flags to be more > successful > >> than feature branches for changes of existing functionality. I also > think > >> that stable / unstable MV would work better than feature branches. I > just > >> wanted to mention it for completeness, not sure if we should start a > full > >> fledged discussion on these topics. > >> > >> > I'm struggling a bit with your phrasing. Are you suggesting that > unstable > >> MVs would not be able to be in trunk? > >> > >> Unstable MV should be in trunk. The wording is related to when we can > >> promote "unstable" to "stable". > >> > >> -Artem > >> > >> > >> On Mon, Jan 15, 2024 at 10:03 PM Colin McCabe > wrote: > >> > >> > On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > >> > > I think using feature flags (whether we support a framework and > tooling > >> > for > >> > > feature flags or just an ad-hoc XyzEnabled flag) can be an > alternative > >> to > >> > > this KIP. I think the value of this KIP is that it's trying to > >> propose a > >> > > systemic approach for gating functionality that may take multiple > >> > releases > >> > > to develop. A problem with ad-hoc feature flags is that it's useful > >> > during > >> > > feature development, so that people who are working on this feature > (or > >> > are > >> > > interested in beta-testing the feature) can get early access > (without > >> any > >> > > guarantees on compatibility or even correctness); but then the > feature > >> > > flags often stick forever after the feature development is done > (and as > >> > > time moves one, the new code is written in such a way that it's not > >> > > possible to turn the feature off any more cleanly). > >> > > > >> > > >> > Hi Artem, > >> > > >> > I think feature flags are somewhat orthogonal to the stable / unstable > >> > discussion. Even if every new feature was a feature flag, you probably > >> > still wouldn't want to stabilize the features immediately, to avoid > >> > maintaining a lot of alpha stuff forever. > >> > > >> > (I also think that feature flags should be used sparingly, if at all, > >> > because of the way that they exponentially increase the test matrix. > But > >> > that's a tangent, I think, given the first point...) > >> > > >> > > > >> > > I'd also clarify how I think about "stable". Ismael made a comment > " > >> > > something is s
Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka
Looks like this KIP has gone quiet for a bit, but here with an update (maybe this will revitalize the conversation too :) ) For KIP-1022 we want to introduce more features and will have a config ` unstable.version.enable` I know that this KIP proposed a config `unstable.metadata.version.enable` so just want to figure out how these will interact. I would propose just replacing the metadata specific one, but I know this config is actually already in code and being used, so this makes it a bit more tricky. An alternative approach is to have both configs. The metadata config would just control metadata, and the general one all features. What do folks think, Justine On Mon, Jan 29, 2024 at 1:42 PM Jun Rao wrote: > Hi, Colin, > > Thanks for the reply. > > "If Y changed the same file as X, we can just edit the file so that Y's > change come first." > > It seems the effort is more than just editing the file. One has to change > all the logic around the old IV. Also, what happens to the client? A client > may have implemented a version for a request corresponding to an unstable > feature. When the unstable feature is re-ordered, do all clients (including > those non-java ones) need to change the PRC implementation for the existing > underlying requests? > > Jun > > On Mon, Jan 29, 2024 at 9:57 AM Colin McCabe wrote: > > > On Tue, Jan 23, 2024, at 11:21, Jun Rao wrote: > > > Hi, Proven, > > > > > > Thanks for the KIP. > > > > > > I am not sure about the reordering approach proposed in the KIP. Let's > > say > > > in a release we have features X and Y, depending on MV IV1 and IV2, > > > respectively. At the release time, feature Y is ready, but X is not. I > > > guess the proposal is to move IV1 to a new MV IV3? > > > > Hi Jun, > > > > In your example, if X is not ready, it should be moved into an unstable > > MV. Then if Y is ready, it can be moved into a stable MV and we can > advance > > the last stable MV. > > > > > The issue is that IV2 > > > could have made changes on top of IV1. For example, IV2 could have > > evolved > > > the schema of the same inter broker request as IV1. In that case, what > > does > > > IV3 represent? We can't simply take the changes associated with IV1 > since > > > it could have conflicts with IV2. > > > > If Y changed the same file as X, we can just edit the file so that Y's > > change come first. > > > > Nobody using a stable MV should be affected, since they will be > generating > > records on an older (stable) version. > > > > > Even when there are no conflicts, I am not sure if the approach > supports > > > the trunk deployment model. Let's say we move two unstable MV IV3 and > IV4 > > > to IV7 and IV8. If someone deployed code corresponding to IV4 and later > > > upgraded the code to IV7, he/she (1) wouldn't be able to set IV4 in the > > > upgrade code since it no longer exists and (2) would be surprised that > > the > > > request protocol that worked under IV4 doesn't work now. > > > > > > > Upgrades are not supported if you are running an unstable MV. It's > > intended only for testing. > > > > Colin > > > > > > > > Thanks, > > > > > > Jun > > > > > > On Fri, Jan 19, 2024 at 2:13 PM Artem Livshits > > > wrote: > > > > > >> Hi Colin, > > >> > > >> > I think feature flags are somewhat orthogonal to the stable / > > unstable > > >> discussion > > >> > > >> I think feature flags can be used as an alternative to achieve similar > > >> results as stable / unstable functionality. As well as long-lived > > feature > > >> branches. In my experience, I've seen feature flags to be more > > successful > > >> than feature branches for changes of existing functionality. I also > > think > > >> that stable / unstable MV would work better than feature branches. I > > just > > >> wanted to mention it for completeness, not sure if we should start a > > full > > >> fledged discussion on these topics. > > >> > > >> > I'm struggling a bit with your phrasing. Are you suggesting that > > unstable > > >> MVs would not be able to be in trunk? > > >> > > >> Unstable MV should be in trunk. The wording is related to when we can > > >> promote "unstable" to "stable". > > >> > > >> -Artem > > >> > > >> > > >> On Mon, Jan 15, 2024 at 10:03 PM Colin McCabe > > wrote: > > >> > > >> > On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > > >> > > I think using feature flags (whether we support a framework and > > tooling > > >> > for > > >> > > feature flags or just an ad-hoc XyzEnabled flag) can be an > > alternative > > >> to > > >> > > this KIP. I think the value of this KIP is that it's trying to > > >> propose a > > >> > > systemic approach for gating functionality that may take multiple > > >> > releases > > >> > > to develop. A problem with ad-hoc feature flags is that it's > useful > > >> > during > > >> > > feature development, so that people who are working on this > feature > > (or > > >> > are > > >> > > interested in beta-testing the feature) can get early access > > (without > >
