[VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Luke Chen]

Hi all,

Some context about this thread:
In KIP-679, we set the default producer config with
`enable.idempotence=true` and `acks=all` in v3.0.0. Here's the discussion
and vote thread link:
https://lists.apache.org/thread/hfwwpyl3kyj381v45dfnyrw9ddrkyc29
https://lists.apache.org/thread/g6gr9m2pxp5wvnbk2p4jpv5vvgh22c92

However, we found there's a bug that causes the default producer is not an
idempotent producer even though the config `enable.idempotence=true` is
correctly set. Please check KAFKA-13598
 for more detail. During
the implementation, we found in kafka-console-producer.sh, we have an
option: `request-required-acks` that can configure the acks  setting in
Producer. It was originally default to 1. But after this KIP, we set the
default enable.idempotence  to true, so we have to also set the default acks
config here to -1 for this change.

I've updated the KIP to add this change to address this change.

Thank you.
Luke

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Cheng Tan]

Thanks you all for discussing and voting on this KIP. Now it has been approved 
with 4 binding votes.

Best, - Cheng

> On Dec 8, 2020, at 12:52 AM, David Jacot  wrote:
> 
> +1 (binding)
> 
> Thanks for the KIP, Cheng!
> 
> On Tue, Dec 8, 2020 at 12:23 AM Ismael Juma  wrote:
> 
>> Thanks, +1 (binding).
>> 
>> On Mon, Dec 7, 2020 at 1:40 PM Cheng Tan  wrote:
>> 
>>> Hi Ismael,
>>> 
>>> Yes. Add deprecation warning for `IDEMPOTENT_WRITE` in 3.0 makes sense.
>>> I’ve updated the KIP’s “DEMPOTENT_WRITE Deprecation” section to reflect
>>> your suggestion. Please let me know if you have more suggestions. Thanks.
>>> 
>>> 
>>> Best, - Cheng Tan
>>> 
>>> 
 On Dec 7, 2020, at 6:42 AM, Ismael Juma  wrote:
 
 Thanks for the KIP Cheng. One suggestion: should we add the
>> `kafka-acls`
 deprecation warnings for `IDEMPOTENT_WRITE` in 3.0? That would give
>> time
 for authorizer implementations to be updated.
 
 Ismael
 
 On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan >> c...@confluent.io>> wrote:
 
> Hi all,
> 
> I’m proposing a new KIP for enabling the strongest delivery guarantee
>> by
> default. Today Kafka support EOS and N-1 concurrent failure tolerance
>>> but
> the default settings haven’t bring them out of the box. The proposal
> changes include the producer defaults change to `ack=all` and
> `enable.idempotence=true`. Also, the ACL operation type
>>> IDEMPOTENCE_WRITE
> will be deprecated. If a producer has WRITE permission to any topic,
>> it
> will be able to request a producer id and perform idempotent produce.
> 
> KIP here:
> 
> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> <
> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
>>> <
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
 
>> 
> 
> Please vote in this mail thread.
> 
> Thanks
> 
> - Cheng Tan
>>> 
>>> 
>> 

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[David Jacot]

+1 (binding)

Thanks for the KIP, Cheng!

On Tue, Dec 8, 2020 at 12:23 AM Ismael Juma  wrote:

> Thanks, +1 (binding).
>
> On Mon, Dec 7, 2020 at 1:40 PM Cheng Tan  wrote:
>
> > Hi Ismael,
> >
> > Yes. Add deprecation warning for `IDEMPOTENT_WRITE` in 3.0 makes sense.
> > I’ve updated the KIP’s “DEMPOTENT_WRITE Deprecation” section to reflect
> > your suggestion. Please let me know if you have more suggestions. Thanks.
> >
> >
> > Best, - Cheng Tan
> >
> >
> > > On Dec 7, 2020, at 6:42 AM, Ismael Juma  wrote:
> > >
> > > Thanks for the KIP Cheng. One suggestion: should we add the
> `kafka-acls`
> > > deprecation warnings for `IDEMPOTENT_WRITE` in 3.0? That would give
> time
> > > for authorizer implementations to be updated.
> > >
> > > Ismael
> > >
> > > On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan  > c...@confluent.io>> wrote:
> > >
> > >> Hi all,
> > >>
> > >> I’m proposing a new KIP for enabling the strongest delivery guarantee
> by
> > >> default. Today Kafka support EOS and N-1 concurrent failure tolerance
> > but
> > >> the default settings haven’t bring them out of the box. The proposal
> > >> changes include the producer defaults change to `ack=all` and
> > >> `enable.idempotence=true`. Also, the ACL operation type
> > IDEMPOTENCE_WRITE
> > >> will be deprecated. If a producer has WRITE permission to any topic,
> it
> > >> will be able to request a producer id and perform idempotent produce.
> > >>
> > >> KIP here:
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> > >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> > >
> > >>>
> > >>
> > >> Please vote in this mail thread.
> > >>
> > >> Thanks
> > >>
> > >> - Cheng Tan
> >
> >
>

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Ismael Juma]

Thanks, +1 (binding).

On Mon, Dec 7, 2020 at 1:40 PM Cheng Tan  wrote:

> Hi Ismael,
>
> Yes. Add deprecation warning for `IDEMPOTENT_WRITE` in 3.0 makes sense.
> I’ve updated the KIP’s “DEMPOTENT_WRITE Deprecation” section to reflect
> your suggestion. Please let me know if you have more suggestions. Thanks.
>
>
> Best, - Cheng Tan
>
>
> > On Dec 7, 2020, at 6:42 AM, Ismael Juma  wrote:
> >
> > Thanks for the KIP Cheng. One suggestion: should we add the `kafka-acls`
> > deprecation warnings for `IDEMPOTENT_WRITE` in 3.0? That would give time
> > for authorizer implementations to be updated.
> >
> > Ismael
> >
> > On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan  c...@confluent.io>> wrote:
> >
> >> Hi all,
> >>
> >> I’m proposing a new KIP for enabling the strongest delivery guarantee by
> >> default. Today Kafka support EOS and N-1 concurrent failure tolerance
> but
> >> the default settings haven’t bring them out of the box. The proposal
> >> changes include the producer defaults change to `ack=all` and
> >> `enable.idempotence=true`. Also, the ACL operation type
> IDEMPOTENCE_WRITE
> >> will be deprecated. If a producer has WRITE permission to any topic, it
> >> will be able to request a producer id and perform idempotent produce.
> >>
> >> KIP here:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> >> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> >
> >>>
> >>
> >> Please vote in this mail thread.
> >>
> >> Thanks
> >>
> >> - Cheng Tan
>
>

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Cheng Tan]

Hi Ismael,

Yes. Add deprecation warning for `IDEMPOTENT_WRITE` in 3.0 makes sense. I’ve 
updated the KIP’s “DEMPOTENT_WRITE Deprecation” section to reflect your 
suggestion. Please let me know if you have more suggestions. Thanks.


Best, - Cheng Tan


> On Dec 7, 2020, at 6:42 AM, Ismael Juma  wrote:
> 
> Thanks for the KIP Cheng. One suggestion: should we add the `kafka-acls`
> deprecation warnings for `IDEMPOTENT_WRITE` in 3.0? That would give time
> for authorizer implementations to be updated.
> 
> Ismael
> 
> On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan  > wrote:
> 
>> Hi all,
>> 
>> I’m proposing a new KIP for enabling the strongest delivery guarantee by
>> default. Today Kafka support EOS and N-1 concurrent failure tolerance but
>> the default settings haven’t bring them out of the box. The proposal
>> changes include the producer defaults change to `ack=all` and
>> `enable.idempotence=true`. Also, the ACL operation type IDEMPOTENCE_WRITE
>> will be deprecated. If a producer has WRITE permission to any topic, it
>> will be able to request a producer id and perform idempotent produce.
>> 
>> KIP here:
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
>>  
>> 
>>> 
>> 
>> Please vote in this mail thread.
>> 
>> Thanks
>> 
>> - Cheng Tan

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Ismael Juma]

Thanks for the KIP Cheng. One suggestion: should we add the `kafka-acls`
deprecation warnings for `IDEMPOTENT_WRITE` in 3.0? That would give time
for authorizer implementations to be updated.

Ismael

On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan  wrote:

> Hi all,
>
> I’m proposing a new KIP for enabling the strongest delivery guarantee by
> default. Today Kafka support EOS and N-1 concurrent failure tolerance but
> the default settings haven’t bring them out of the box. The proposal
> changes include the producer defaults change to `ack=all` and
> `enable.idempotence=true`. Also, the ACL operation type IDEMPOTENCE_WRITE
> will be deprecated. If a producer has WRITE permission to any topic, it
> will be able to request a producer id and perform idempotent produce.
>
> KIP here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> >
>
> Please vote in this mail thread.
>
> Thanks
>
> - Cheng Tan

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Rajini Sivaram]

+1 (binding)

Thanks for the KIP, Cheng!

Regards,

Rajini


On Mon, Dec 7, 2020 at 6:14 AM Gwen Shapira  wrote:

> +1 (binding). Awesome suggestion, Cheng.
>
> On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan  wrote:
> >
> > Hi all,
> >
> > I’m proposing a new KIP for enabling the strongest delivery guarantee by
> default. Today Kafka support EOS and N-1 concurrent failure tolerance but
> the default settings haven’t bring them out of the box. The proposal
> changes include the producer defaults change to `ack=all` and
> `enable.idempotence=true`. Also, the ACL operation type IDEMPOTENCE_WRITE
> will be deprecated. If a producer has WRITE permission to any topic, it
> will be able to request a producer id and perform idempotent produce.
> >
> > KIP here:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679:+Producer+will+enable+the+strongest+delivery+guarantee+by+default
> >
> >
> > Please vote in this mail thread.
> >
> > Thanks
> >
> > - Cheng Tan
>
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Gwen Shapira]

+1 (binding). Awesome suggestion, Cheng.

On Fri, Dec 4, 2020 at 11:00 AM Cheng Tan  wrote:
>
> Hi all,
>
> I’m proposing a new KIP for enabling the strongest delivery guarantee by 
> default. Today Kafka support EOS and N-1 concurrent failure tolerance but the 
> default settings haven’t bring them out of the box. The proposal changes 
> include the producer defaults change to `ack=all` and 
> `enable.idempotence=true`. Also, the ACL operation type IDEMPOTENCE_WRITE 
> will be deprecated. If a producer has WRITE permission to any topic, it will 
> be able to request a producer id and perform idempotent produce.
>
> KIP here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
>  
> 
>
> Please vote in this mail thread.
>
> Thanks
>
> - Cheng Tan



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[VOTE] KIP-679: Producer will enable the strongest delivery guarantee by default

[Cheng Tan]

Hi all,

I’m proposing a new KIP for enabling the strongest delivery guarantee by 
default. Today Kafka support EOS and N-1 concurrent failure tolerance but the 
default settings haven’t bring them out of the box. The proposal changes 
include the producer defaults change to `ack=all` and 
`enable.idempotence=true`. Also, the ACL operation type IDEMPOTENCE_WRITE will 
be deprecated. If a producer has WRITE permission to any topic, it will be able 
to request a producer id and perform idempotent produce.

KIP here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default
 


Please vote in this mail thread.

Thanks

- Cheng Tan