[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323879#comment-17323879 ] Haoran Xuan commented on KAFKA-12574: - Thanks, [~guozhang] for the clarification! I'm curious about the responsibility of KAFKA-9689, do we still need it for client-side automatic discovery of server-side eos feature version? If the answer is yes, I will go back to re-work the plan and implementation on KAFKA-9689, thanks! > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322417#comment-17322417 ] Guozhang Wang commented on KAFKA-12574: --- Hi [~feyman] I'm aware of KAFKA-9689 as well, and also discussed with [~boyang] about that. I also feel that, doing how swapping online is quite hard, and also would bring in much complexity with the code as well, and hence suggested we moving towards the direction to enforce the config change. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322296#comment-17322296 ] Haoran Xuan commented on KAFKA-12574: - Hi, previously I was working on KAFKA-9689, in which [~bchen225242] and I were thinking of leveraging the feature versioning system to automatically detect the `exactly_once_semantics` and also make the `StreamThread` able to hot swap from `eos` to `eos-beta`. During which I found that the hot swapping the `exactly_once_semantics` is kind of difficult and I haven't thought it thorough, this discussion here made me realize that maybe I'm not heading in the correct direction. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315156#comment-17315156 ] Ismael Juma commented on KAFKA-12574: - For 4.0, we can make eos-v2 the default and make it a non-issue for most people. Then this config will only be there for people who want the weaker semantics. :) > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315153#comment-17315153 ] Guozhang Wang commented on KAFKA-12574: --- I'm fine with option 2) as well, to stay with `eos-v2` eventually. Honestly I think we should try to avoid such upgrade path ever in the future -- i.e. a rolling bounce with config change -- but I know that one should never say never :) So although we may not ever have a `eos-v3`, this is not a bad state to end with `eos-v2`. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[
https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315144#comment-17315144
]
Ismael Juma commented on KAFKA-12574:
-
{quote}Regarding the proposal, I thin users would find it very odd if we moved
on to eos-v2 and then suddenly deprecated it and went back to just "eos" –
makes it seem like there was a problem with eos-v2. I would be fine with just
staying on eos-v2 though. For one thing it leaves the door open to further
developments in eos that need to be gated by a config, eg eos-v3, if we ever
have need for that again.
{quote}
+1
> Deprecate eos-alpha
> ---
>
> Key: KAFKA-12574
> URL: https://issues.apache.org/jira/browse/KAFKA-12574
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Blocker
> Labels: needs-kip
> Fix For: 3.0.0
>
>
> In KIP-447 we introduced a new thread-producer which is capable of
> exactly-once semantics across multiple tasks. The new mode of EOS, called
> eos-beta, is intended to eventually be the preferred processing mode for EOS
> as it improves the performance and scaling of partitions/tasks. The only
> downside is that it requires brokers to be on version 2.5+ in order to
> understand the latest APIs that are necessary for this thread-producer.
> We should consider deprecating the eos-alpha config, ie
> StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate
> to the new-and-improved processing mode, and upgrade their brokers if
> necessary.
> Eventually we would like to be able to remove the eos-alpha code paths from
> Streams as this will help to simplify the logic and reduce the processing
> mode branching. But since this will break client-broker compatibility, and
> 2.5 is still a relatively recent version, we probably can't actually remove
> eos-alpha in the near future
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315142#comment-17315142 ] A. Sophie Blee-Goldman commented on KAFKA-12574: > This is well documented in > https://kafka.apache.org/27/documentation/streams/upgrade-guide Ah, you're right. I must have been on an older version of the docs -- I find Google often takes me to 2.4 docs for some reason. We need better SEO :/ (I filed a ticket for this already a while back) Also not sure how I forgot about EosBetaUpgradeIntegrationTest after spending so much time trying to help fix it -- sorry for doubting our test coverage and docs. Regarding the proposal, I thin users would find it very odd if we moved on to eos-v2 and then suddenly deprecated it and went back to just "eos" -- makes it seem like there was a problem with eos-v2. I would be fine with just staying on eos-v2 though. For one thing it leaves the door open to further developments in eos that need to be gated by a config, eg eos-v3, if we ever have need for that again. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[
https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313484#comment-17313484
]
Matthias J. Sax commented on KAFKA-12574:
-
I agree that "hot swapping" might be too aggressive.
About the naming question: it might be good to move off the "-beta" name sooner
than later. Thus I would propose the following:
* Deprecate `eos` and `eos-beta` in 3.0 and add `eos-v2` (as new name for
"beta) – this should ensure that people understand that the new EOS is better
* remove `eos` (config and impl) and `eos-beta` config (we know only have
eos-v2 config)
* Add `eos` back an 4.x and deprecate `eos-v2`
* Remove `eos-v2` in 5.0
{quote}Regarding the eos-alpha to eos-beta upgrade path, I agree we should well
document the upgrade path
{quote}
This is well documented in
[https://kafka.apache.org/27/documentation/streams/upgrade-guide]
{noformat}
Starting in Kafka Streams 2.6.x, a new processing mode "exactly_once_beta"
(configurable via parameter processing.guarantee) is available. To use this new
feature, your brokers must be on version 2.5.x or newer. A switch from
"exactly_once" to "exactly_once_beta" (or the other way around) is only
possible if the application is on version 2.6.x. If you want to upgrade your
application from an older version and enable this feature, you first need to
upgrade your application to version 2.6.x, staying on "exactly_once", and then
do second round of rolling bounces to switch to "exactly_once_beta". For a
downgrade, do the reverse: first switch the config from "exactly_once_beta" to
"exactly_once" to disable the feature in your 2.6.x application. Afterward, you
can downgrade your application to a pre-2.6.x version.
{noformat}
{quote}Besides, when checking on the upgrade test paths, I think our system
tests do not have a case to cover the eos -> eos-beta upgrade path
{quote}
Correct. Because we test this with `EosBetaUpgradeIntegrationTest` – we don't
need a system test because the upgrade from alpha to beta happens in the same
version.
> Deprecate eos-alpha
> ---
>
> Key: KAFKA-12574
> URL: https://issues.apache.org/jira/browse/KAFKA-12574
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Blocker
> Labels: needs-kip
> Fix For: 3.0.0
>
>
> In KIP-447 we introduced a new thread-producer which is capable of
> exactly-once semantics across multiple tasks. The new mode of EOS, called
> eos-beta, is intended to eventually be the preferred processing mode for EOS
> as it improves the performance and scaling of partitions/tasks. The only
> downside is that it requires brokers to be on version 2.5+ in order to
> understand the latest APIs that are necessary for this thread-producer.
> We should consider deprecating the eos-alpha config, ie
> StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate
> to the new-and-improved processing mode, and upgrade their brokers if
> necessary.
> Eventually we would like to be able to remove the eos-alpha code paths from
> Streams as this will help to simplify the logic and reduce the processing
> mode branching. But since this will break client-broker compatibility, and
> 2.5 is still a relatively recent version, we probably can't actually remove
> eos-alpha in the near future
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312782#comment-17312782 ] Guozhang Wang commented on KAFKA-12574: --- Okay, if we are concerned about 4) to be too aggressive, I'm also happy with 3) as well. On the other hand I'm a bit reserved about adding the suffix as a workaround, since the config name is `processing.guarantee` and putting the value as `exactly_once_semantics` seems a bit misaligned. Regarding the upgrade path, I agree we should well document the upgrade path (we should at least copy-paste what's in https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics#KIP447:Producerscalabilityforexactlyoncesemantics-Compatibility,Deprecation,andMigrationPlan). Besides, when checking on the upgrade test paths, I think our system tests do not have a case to cover the eos -> eos-beta upgrade path yet, but I only looked at https://issues.apache.org/jira/browse/KAFKA-9832 so maybe there're other tests that I forgot. [~mjsax] [~boyang] could you chime in and let us know if it is covered in system upgrade test or not? > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312563#comment-17312563 ] A. Sophie Blee-Goldman commented on KAFKA-12574: I think maybe we weren't yet confident in the stability when it first rolled out so we decided not to deprecate eos-alpha (and called it "-beta"), but we are now: hence why I created this ticket to deprecate it. But Ismael has pulled me off the fence, I agree now that swapping out this config from beneath users is going to cause way more problems than it will solve. Also, how sure are we that you can safely and smoothly upgrade from eos-alpha to eos-beta with a rolling bounce? I assume this is possible since we don't document otherwise that I've seen, but do we have tests in place for this? > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[
https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312389#comment-17312389
]
Ismael Juma commented on KAFKA-12574:
-
To clarify my last comment, I think it would have been less surprising if the
config had been deprecated for a few releases. But it doesn't look like we did
that:
{quote}"The processing guarantee that should be used. " +
"Possible values are " + AT_LEAST_ONCE + " (default), " +
"" + EXACTLY_ONCE + " (requires brokers version 0.11.0 or
higher), " +
"and " + EXACTLY_ONCE_BETA + " (requires brokers version 2.5 or
higher). " +
"Note that exactly-once processing requires a cluster of at least three
brokers by default what is the " +
"recommended setting for production; for development you can change this, by
adjusting broker setting " +
"transaction.state.log.replication.factor and
transaction.state.log.min.isr."
{quote}
> Deprecate eos-alpha
> ---
>
> Key: KAFKA-12574
> URL: https://issues.apache.org/jira/browse/KAFKA-12574
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: A. Sophie Blee-Goldman
>Priority: Blocker
> Labels: needs-kip
> Fix For: 3.0.0
>
>
> In KIP-447 we introduced a new thread-producer which is capable of
> exactly-once semantics across multiple tasks. The new mode of EOS, called
> eos-beta, is intended to eventually be the preferred processing mode for EOS
> as it improves the performance and scaling of partitions/tasks. The only
> downside is that it requires brokers to be on version 2.5+ in order to
> understand the latest APIs that are necessary for this thread-producer.
> We should consider deprecating the eos-alpha config, ie
> StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate
> to the new-and-improved processing mode, and upgrade their brokers if
> necessary.
> Eventually we would like to be able to remove the eos-alpha code paths from
> Streams as this will help to simplify the logic and reduce the processing
> mode branching. But since this will break client-broker compatibility, and
> 2.5 is still a relatively recent version, we probably can't actually remove
> eos-alpha in the near future
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312069#comment-17312069 ] Ismael Juma commented on KAFKA-12574: - Sophie, we have seen these kinds of issues with kafka cluster upgrades even though the upgrade procedure has been the same for years (requires setting IBP, etc.). And for every case we see, there are 100 that we don't. This is the reason why we want to remove the IBP step from Kafka upgrades. Anything unusual will cause issues for sure. That is not to say that you can never break things, but 1 year is typically considered pretty new for infra software. It seems pretty odd and unexpected to change the meaning of a config in this way. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312027#comment-17312027 ] A. Sophie Blee-Goldman commented on KAFKA-12574: I would expect users who don't read the upgrade guide to at least be somewhat careful when performing a rolling upgrade, especially when it's a breaking change release, and at least check that the bounced node had successfully restarted before they go about bouncing the rest of them. And in that case, the first bounced node would fail immediately and clearly. So only if they (a) skipped the upgrade guide, and (b) did not monitor a single node during the rolling upgrade, would they experience an outage. FWIW I'm a little on the fence about (4) as well, but a user would have to be pretty nonchalant to experience a full blown outage. A slightly better take on 3 might be to make the new config StreamsConfig#EXACTLY_ONCE_SEMANTICS -- basically take advantage of the fact that the current config is just EXACTLY_ONCE/EXACTLY_ONCE_BETA, and pop the _SEMANTICS suffix on there so we can avoid coming up with a new name. I guess we could also deprecate AT_LEAST_ONCE and replace it with AT_LEAST_ONCE_SEMANTICS for full consistency > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312002#comment-17312002 ] Ismael Juma commented on KAFKA-12574: - `4` seems a bit aggressive for Apache Kafka 3.0. It's a one line change _if_ you know about it. I am pretty sure a lot of people will miss it and have a bad experience (an outage basically). AK 2.5 was only released 1 year ago. It would make sense to do the change you're proposing in Apache Kafka 4.0 though. In 4.0, all of the `eos` config options would result in the same thing. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311995#comment-17311995 ] Guozhang Wang commented on KAFKA-12574: --- Yeah I think in 3.0 we would not remove the old eos implementations, it's just about how we want to deprecate the old configs in 3.0 to encourage people to move to the new eos protocol. So far we have several options: 1. Just deprecate `eos`, and encourage people to go with `eos-beta`. Eventually we would only have `eos-beta` 2. Similar to 1): we deprecate `eos` and `eos-beta`, and introduce a `eos-v2` which is the same as `eos-beta`, and encourage people to go with `eos-v2`. Eventually we would only have `eos-v2`. It is slightly better than 1) since people usually do not like to stick with "beta" but rather "v2". 3. Similar to 2), except that we would rename `eos-v2` to `eos` after we remove the old `eos`. It has cleaner name for first-time users but may confuse existing users. 4. Very different to 1/2/3: we deprecate `eos-beta`, and introduce a `eos-alpha`, and tell users if they upgrade to 3.0 and do not want to use the new `eos`, switch to `eos-alpha`. We would later (3.1?) deprecate `eos-alpha` and eventually only have `eos`. It has cleaner name but require a one-liner code change for people who want to stay as-is when upgrading. Personally I'm a bit leaning towards 4), since besides it would eventually end up with just `eos`, with `eos-alpha` I personally feel it is probably even more "persuasive" to let people move away from it than naming the new one as shiny "v2" to let people move to it; its downsides, on the other side, the one-liner change is indeed concerning, but I feel it is not a huge burden. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311939#comment-17311939 ] Matthias J. Sax commented on KAFKA-12574: - We discussed "auto-fall back" but there are issues and thus we did not do it. However, I think that [~ijuma] concerns is that is might be too early to _remove_ eos-v1 (as user might not be able to upgrade to the newest KafkaStreams if they use eos but are stuck with older brokers) and I agree – however, we are talking about _deprecation_, not removal, and thus we can move forward IMHO. We would remove `eos-v1` earliest in 4.0 but we would have a new discussion to see if we think it's ok to remove or still be too early. But we should not discuss this on the ticket but on the KIP I guess? > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311787#comment-17311787 ] A. Sophie Blee-Goldman commented on KAFKA-12574: > I think this is slightly better than new-eos. I think it's much better than "new-eos" -- that wasn't a serious proposal :) Regarding 2, you mean dynamically switch back to eos-alpha if we detect that the brokers aren't on a high enough version? My impression was that we hadn't implemented this because of concerns over either the risk or the LOE of dynamic adaption to broker version. Maybe [~mjsax] or [~bchen225242] can shed some light on this -- it seems to me like we could just issue a test API call on startup before actually processing anything, and if the brokers don't support the right version then we initialize with the task-producer and fall back to eos-alpha. > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311750#comment-17311750 ] Ismael Juma commented on KAFKA-12574: - A few comments: # I think eos-v2 would be a good way to encourage people to use the right config. I think this is slightly better than new-eos. # Requiring 2.5+ seems a bit premature. Can we not pick eos-v2 if the broker supports it? We have the feature versioning stuff for this reason, right? > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311717#comment-17311717 ] A. Sophie Blee-Goldman commented on KAFKA-12574: That SGTM -- if users blindly upgrade without reading the guide then it should fail fast, and we can expand the UnsupportedVersionException to point users with older brokers towards the new eos-alpha config. I guess we would also deprecate eos-beta in 3.0 then, and refer users towards the plain "eos" > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311697#comment-17311697 ] Guozhang Wang commented on KAFKA-12574: --- Yeah.. I also felt this procedure requires users to "jump back and forth" in their configs to finally upgrade to the new eos too, hence was clarifying with you. At the same time, I really cannot find a better name than `EXACTLY_ONCE` itself. Another, more intrusive procedure, would be: directly declare in 3.0 that, 1) `eos` would require broker 2.5+, and 2) if you do not want to upgrade, make a one-line code change to use `eos-alpha` (this would be newly added). This is, admittedly requiring many users to make code change for eos, but still this is just a one-liner change and they can get exactly the same behavior if they want to stay as-is. We can then consider deprecating `eos-alpha` in 3.1 and removing in the future. WDYT? > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311685#comment-17311685 ] A. Sophie Blee-Goldman commented on KAFKA-12574: Yes, something like that. It's kind of a hassle so if we want to limit the changes users will have to make then we could try to think up a new name for eos-beta which doesn't overlap with the current "eos" (which is really eos-alpha). That way we can deprecate eos(eos-alpha) AND eos-beta at the same time, and point users to always use the "new-eos" after that. Then we can just remove both those after the deprecation period and users will only need to switch their configs one time. Not sure what an appropriate name would be besides "eos"/EXACTLY_ONCE though...thoughts? > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311648#comment-17311648 ] Guozhang Wang commented on KAFKA-12574: --- I agree we should try to encourage users to switch to the new config so that we can eventually remove the eos-alpha code base, since many of the planned future development on the task management, scheduling etc would be much more cumbersome with the mixed alpha/beta models. Regarding the procedure, note that today we have the config names as `eos` (for alpha) and `eos-beta`, and if the end goal is to have `eos-beta` to replace `eos`, then what we would have is, probably: * deprecate `eos`, encourage people to go with `eos-beta`. * remove `eos`, and then we can remove the alpha code path. * deprecate `eos-beta` and then add `eos` back. * remove `eos-beta`. Is that what you have in mind [~ableegoldman]? > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12574) Deprecate eos-alpha
[ https://issues.apache.org/jira/browse/KAFKA-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310993#comment-17310993 ] A. Sophie Blee-Goldman commented on KAFKA-12574: Given the current name, "eos-beta", it's likely that many users have chosen to stick with eos-alpha as the "beta" may cause them to assume this mode is not production ready. If/when we're confident that it is indeed production ready, we should also consider deprecating the StreamsConfig.EXACTLY_ONCE_BETA config name and replacing "eos-beta" with a more confidence-inspiring name > Deprecate eos-alpha > --- > > Key: KAFKA-12574 > URL: https://issues.apache.org/jira/browse/KAFKA-12574 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: A. Sophie Blee-Goldman >Priority: Major > Fix For: 3.0.0 > > > In KIP-447 we introduced a new thread-producer which is capable of > exactly-once semantics across multiple tasks. The new mode of EOS, called > eos-beta, is intended to eventually be the preferred processing mode for EOS > as it improves the performance and scaling of partitions/tasks. The only > downside is that it requires brokers to be on version 2.5+ in order to > understand the latest APIs that are necessary for this thread-producer. > We should consider deprecating the eos-alpha config, ie > StreamsConfig.EXACTLY_ONCE, to encourage new. & existing EOS users to migrate > to the new-and-improved processing mode, and upgrade their brokers if > necessary. > Eventually we would like to be able to remove the eos-alpha code paths from > Streams as this will help to simplify the logic and reduce the processing > mode branching. But since this will break client-broker compatibility, and > 2.5 is still a relatively recent version, we probably can't actually remove > eos-alpha in the near future -- This message was sent by Atlassian Jira (v8.3.4#803005)
