Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #21

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

[Sophie Blee-Goldman]

Hey all,

This KIP has been accepted with

four +1 (binding) votes from John, Guozhang, Matthias, and myself
four +1 (non-binding) votes from Leah, Walker, Lotz, and Israel

Thanks everyone. Israel will take it from here

On Thu, Apr 8, 2021 at 2:58 PM Sophie Blee-Goldman 
wrote:

> > I would also like to volunteer to implement it if that is ok.
>
> That would be awesome -- I've been pretty overbooked lately and was
> literally just about
> to ask for volunteers to take over this KIP. Perfect timing :)
>
> The KIP still has about 4 hours to go on the voting but it looks like
> it'll pass, so feel free to
> start working on a PR. Thanks for volunteering!
>
> On Thu, Apr 8, 2021 at 2:55 PM Israel Ekpo  wrote:
>
>> I have reviewed the KIP. The motivation makes sense and the recommended
>> API
>> changes make sense as well.
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
>>
>> So +1
>>
>> I would also like to volunteer to implement it if that is ok.
>>
>>
>>
>> On Thu, Apr 8, 2021 at 5:42 PM Matthias J. Sax  wrote:
>>
>> > +1 (binding)
>> >
>> > On 4/6/21 10:15 AM, Lotz Utfpr wrote:
>> > > Makes sense to me! +1
>> > >
>> > > Apologies for being brief. This email was sent from my mobile phone.
>> > >
>> > >> On 6 Apr 2021, at 18:45, Walker Carlson
>> 
>> > wrote:
>> > >>
>> > >> This makes sense to me +1!
>> > >>
>> > >> Walker
>> > >>
>> > >>> On Tue, Apr 6, 2021 at 11:08 AM Guozhang Wang 
>> > wrote:
>> > >>>
>> > >>> +1. Thanks!
>> > >>>
>> > >>> On Tue, Apr 6, 2021 at 7:00 AM Leah Thomas
>> > 
>> > >>> wrote:
>> > >>>
>> >  Thanks for picking this up, Sophie. +1 from me, non-binding.
>> > 
>> >  Leah
>> > 
>> > > On Mon, Apr 5, 2021 at 9:42 PM John Roesler 
>> > wrote:
>> > 
>> > > Thanks, Sophie,
>> > >
>> > > I’m +1 (binding)
>> > >
>> > > -John
>> > >
>> > > On Mon, Apr 5, 2021, at 21:34, Sophie Blee-Goldman wrote:
>> > >> Hey all,
>> > >>
>> > >> I'd like to start the voting on KIP-633, to drop the awkward 24
>> hour
>> > > grace
>> > >> period and improve the API to raise visibility on an important
>> > >>> concept
>> >  in
>> > >> Kafka Streams: grace period nad out-of-order data handling.
>> > >>
>> > >> Here's the KIP:
>> > >>
>> > >
>> > 
>> > >>>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
>> > >> <
>> > >
>> > 
>> > >>>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24hr+default+grace+period
>> > >>
>> > >>
>> > >> Cheers,
>> > >> Sophie
>> > >>
>> > >
>> > 
>> > >>>
>> > >>>
>> > >>> --
>> > >>> -- Guozhang
>> > >>>
>> >
>>
>

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Sophie Blee-Goldman]

Alright, here's the detailed proposal for KAFKA-12477. This assumes we will
change the default assignor to ["cooperative-sticky", "range"] in KIP-726.
It also acknowledges that users may attempt any kind of upgrade without
reading the docs, and so we need to put in safeguards against data
corruption rather than assume everyone will follow the safe upgrade path.

With this proposal,
1) New applications on 3.0 will enable cooperative rebalancing by default
2) Existing applications which don’t set an assignor can safely upgrade to
3.0 using a single rolling bounce with no extra steps, and will
automatically transition to cooperative rebalancing
3) Existing applications which do set an assignor that uses EAGER can
likewise upgrade their applications to COOPERATIVE with a single rolling
bounce
4) Once on 3.0, applications can safely go back and forth between EAGER and
COOPERATIVE
5) Applications can safely downgrade away from 3.0

The high-level idea for dynamic protocol upgrades is that the group will
leverage the assignor selected by the group coordinator to determine when
it’s safe to upgrade to COOPERATIVE, and trigger a fail-safe to protect the
group in case of rare events or user misconfiguration. The group
coordinator selects the most preferred assignor that’s supported by all
members of the group, so we know that all members will support COOPERATIVE
once we receive the “cooperative-sticky” assignor after a rebalance. At
this point, each member can upgrade their own protocol to COOPERATIVE.
However, there may be situations in which an EAGER member may join the
group even after upgrading to COOPERATIVE. For example, during a rolling
upgrade if the last remaining member on the old bytecode misses a
rebalance, the other members will be allowed to upgrade to COOPERATIVE. If
the old member rejoins and is chosen to be the group leader before it’s
upgraded to 3.0, it won’t be aware that the other members of the group have
not yet revoked their partitions when computing the assignment.

Short Circuit:
The risk of mixing the cooperative and eager rebalancing protocols is that
a partition may be assigned to one member while it has yet to be revoked
from its previous owner. The danger is that the new owner may begin
processing and committing offsets for this partition while the previous
owner is also committing offsets in its #onPartitionsRevoked callback,
which is invoked at the end of the rebalance in the cooperative protocol.
This can result in these consumers overwriting each other’s offsets and
getting a corrupted view of the partition. Note that it’s not possible to
commit during a rebalance, so we can protect against offset corruption by
blocking further commits after we detect that the group leader may not
understand COOPERATIVE, but before we invoke #onPartitionsRevoked. This is
the “short-circuit” — if we detect that the group is in an unsafe state, we
invoke #onPartitionsLost instead of #onPartitionsRevoked and explicitly
prevent offsets from being committed on those revoked partitions.

Consumer procedure:
Upon startup, the consumer will initially select the highest
commonly-supported protocol across its configured assignors. With
["cooperative-sticky", "range”], the initial protocol will be EAGER when
the member first joins the group. Following a rebalance, each member will
check the selected assignor. If the chosen assignor supports COOPERATIVE,
the member can upgrade their used protocol to COOPERATIVE and no further
action is required. If the member is already on COOPERATIVE but the
selected assignor does NOT support it, then we need to trigger the
short-circuit. In this case we will invoke #onPartitionsLost instead of
#onPartitionsRevoked, and set a flag to block any attempts at committing
those partitions which have been revoked. If a commit is attempted, as may
be the case if the user does not implement #onPartitionsLost (see
KAFKA-12638), we will throw a CommitFailedException which will be bubbled
up through poll() after completing the rebalance. The member will then
downgrade its protocol to EAGER for the next rebalance.

Let me know what you think,
Sophie

On Fri, Apr 2, 2021 at 7:08 PM Luke Chen  wrote:

> Hi Sophie,
> Making the default to "cooperative-sticky, range" is a smart idea, to
> ensure we can at least fall back to rangeAssignor if consumers are not
> following our recommended upgrade path. I updated the KIP accordingly.
>
> Hi Chris,
> No problem, I updated the KIP to include the change in Connect.
>
> Thank you very much.
>
> Luke
>
> On Thu, Apr 1, 2021 at 3:24 AM Chris Egerton 
> wrote:
>
> > Hi all,
> >
> > @Sophie - I like the sound of the dual-protocol default. The smooth
> upgrade
> > path it permits sounds fantastic!
> >
> > @Luke - Do you think we can also include Connect in this KIP? Right now
> we
> > don't set any custom partition assignment strategies for the consumer
> > groups we bring up for sink tasks, and if we continue to just use the
> > default, the assignment s

Re: [VOTE] KIP-725: Streamlining configurations for TimeWindowedDeserializer.

[Sagar]

Thanks for the very precise time John :D

I think that time has been passed so I am going to close the voting.

Thanks!
Sagar.

On Fri, Apr 9, 2021 at 12:28 AM John Roesler  wrote:

> Hi Sagar,
>
> It does have three binding votes, but it looks like it's a
> little shy of the mandatory 72 hours you have to leave the
> vote open:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process
>
> Assuming no one shows up with a veto, you can close the vote
> after 2021/04/09 02:12:47 UTC.
>
> That said, I've never seen a veto, so I think you're safe to
> start preparing the PR for reviews.
>
> Thanks again,
> John
>
>
> On Fri, 2021-04-09 at 00:22 +0530, Sagar wrote:
> > Thanks Sophie/ John/ Leah and Guozhang.
> >
> > Can I assume that this kip has received sufficient votes to be marked as
> > accepted ?
> >
> > Sagar.
> >
> > On Tue, 6 Apr 2021 at 9:38 PM, Guozhang Wang  wrote:
> >
> > > +1. Thanks!
> > >
> > > On Tue, Apr 6, 2021 at 7:01 AM Leah Thomas
> 
> > > wrote:
> > >
> > > > Hi Sagar, +1 non-binding. Thanks again for doing this.
> > > >
> > > > Leah
> > > >
> > > > On Mon, Apr 5, 2021 at 9:40 PM John Roesler 
> wrote:
> > > >
> > > > > Thanks, Sagar!
> > > > >
> > > > > I’m +1 (binding)
> > > > >
> > > > > -John
> > > > >
> > > > > On Mon, Apr 5, 2021, at 21:35, Sophie Blee-Goldman wrote:
> > > > > > Thanks for the KIP! +1 (binding) from me
> > > > > >
> > > > > > Cheers,
> > > > > > Sophie
> > > > > >
> > > > > > On Mon, Apr 5, 2021 at 7:13 PM Sagar 
> > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I would like to start voting on the following KIP:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177047930
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Sagar.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
>
>
>

[jira] [Created] (KAFKA-12638) Remove default implementation of ConsumerRebalanceListener#onPartitionsLost

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12638:
--

 Summary: Remove default implementation of 
ConsumerRebalanceListener#onPartitionsLost
 Key: KAFKA-12638
 URL: https://issues.apache.org/jira/browse/KAFKA-12638
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: A. Sophie Blee-Goldman


When we added the #onPartitionsLost callback to the ConsumerRebalanceListener 
in KIP-429, we gave it a default implementation that just invoked the existing 
#onPartitionsRevoked method for backwards compatibility. This is somewhat 
inconvenient, since we generally want to invoke #onPartitionsLost in order to 
skip the committing of offsets on revoked partitions, which is exactly what 
#onPartitionsRevoked does.

I don't think we can just remove it in 3.0 since we haven't indicated that we 
"deprecated" the default implementation or logged a warning that we intend to 
remove the default in a future release (as we did for the 
RocksDBConfigSetter#close method in Streams, for example). We should try to add 
such a warning now, so we can remove it in a future release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12637) Remove deprecated PartitionAssignor interface

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12637:
--

 Summary: Remove deprecated PartitionAssignor interface
 Key: KAFKA-12637
 URL: https://issues.apache.org/jira/browse/KAFKA-12637
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: A. Sophie Blee-Goldman
 Fix For: 3.0.0


In KIP-429, we deprecated the existing PartitionAssignor interface in order to 
move it out of the internals package and better align the name with other 
pluggable Consumer interfaces. We added an adapter to convert from existing 
o.a.k.clients.consumer.internals.PartitionAssignor to the new 
o.a.k.clients.consumer.ConsumerPartitionAssignor and support the deprecated 
interface. This was deprecated in 2.4, so we should be ok to remove it and the 
adaptor in 3.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #20

[Apache Jenkins Server]

See 

Re: [VOTE] 2.6.2 RC1

[Guozhang Wang]

Looked over the javadocs and web docs again. Download the jars and check
the license files are all updated now (thanks to John again!). +1

On Thu, Apr 8, 2021 at 1:48 PM Sophie Blee-Goldman
 wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 2.6.2.
>
> Apache Kafka 2.6.2 is a bugfix release and fixes 35 issues since the 2.6.1
> release. Please see the release notes for more information.
>
> Release notes for the 2.6.2 release:
> https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, April 13th, 9am PST
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/javadoc/
>
> * Tag to be voted upon (off 2.6 branch) is the 2.6.2 tag:
> https://github.com/apache/kafka/releases/tag/2.6.2-rc1
>
> * Documentation:
> https://kafka.apache.org/26/documentation.html
>
> * Protocol:
> https://kafka.apache.org/26/protocol.html
>
> * Successful Jenkins builds for the 2.6 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka-2.6-jdk8/114/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4458/
>
> /**
>
> Thanks,
> Sophie
>


-- 
-- Guozhang

[jira] [Created] (KAFKA-12636) Ensure retention still enforced for compacted topics if cleaning is not enabled

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-12636:
---

 Summary: Ensure retention still enforced for compacted topics if 
cleaning is not enabled
 Key: KAFKA-12636
 URL: https://issues.apache.org/jira/browse/KAFKA-12636
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


We rely on a periodic task in LogManager to delete old segments of 
non-compacted topics which either have breached retention time or which have 
been explicitly deleted by a call to DeleteRecords. For compacted topics, we 
rely on the cleaning task itself to do the same since a compacted topic may 
also be configured with "delete" retention.

If log cleaning is not enabled, we still need to enforce retention semantics 
for compacted topics, but the current logic in LogManager excludes them from 
consideration:

{code}
// clean current logs.
val deletableLogs = {
  if (cleaner != null) {
// prevent cleaner from working on same partitions when changing 
cleanup policy
cleaner.pauseCleaningForNonCompactedPartitions()
  } else {
currentLogs.filter {
  case (_, log) => !log.config.compact
}
  }
}
{code}

It seems to me that we should remove the filtering when log cleaning is not 
enabled. The logic in `deleteOldSegments` will ensure that only the appropriate 
retention checks are made based on the topic configuration.

Of course it's kind of weird for a user to have a compacted topic when the 
cleaner is not enabled in the first place.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : Kafka » kafka-2.7-jdk8 #144

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

[Sophie Blee-Goldman]

> I would also like to volunteer to implement it if that is ok.

That would be awesome -- I've been pretty overbooked lately and was
literally just about
to ask for volunteers to take over this KIP. Perfect timing :)

The KIP still has about 4 hours to go on the voting but it looks like it'll
pass, so feel free to
start working on a PR. Thanks for volunteering!

On Thu, Apr 8, 2021 at 2:55 PM Israel Ekpo  wrote:

> I have reviewed the KIP. The motivation makes sense and the recommended API
> changes make sense as well.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
>
> So +1
>
> I would also like to volunteer to implement it if that is ok.
>
>
>
> On Thu, Apr 8, 2021 at 5:42 PM Matthias J. Sax  wrote:
>
> > +1 (binding)
> >
> > On 4/6/21 10:15 AM, Lotz Utfpr wrote:
> > > Makes sense to me! +1
> > >
> > > Apologies for being brief. This email was sent from my mobile phone.
> > >
> > >> On 6 Apr 2021, at 18:45, Walker Carlson  >
> > wrote:
> > >>
> > >> This makes sense to me +1!
> > >>
> > >> Walker
> > >>
> > >>> On Tue, Apr 6, 2021 at 11:08 AM Guozhang Wang 
> > wrote:
> > >>>
> > >>> +1. Thanks!
> > >>>
> > >>> On Tue, Apr 6, 2021 at 7:00 AM Leah Thomas
> > 
> > >>> wrote:
> > >>>
> >  Thanks for picking this up, Sophie. +1 from me, non-binding.
> > 
> >  Leah
> > 
> > > On Mon, Apr 5, 2021 at 9:42 PM John Roesler 
> > wrote:
> > 
> > > Thanks, Sophie,
> > >
> > > I’m +1 (binding)
> > >
> > > -John
> > >
> > > On Mon, Apr 5, 2021, at 21:34, Sophie Blee-Goldman wrote:
> > >> Hey all,
> > >>
> > >> I'd like to start the voting on KIP-633, to drop the awkward 24
> hour
> > > grace
> > >> period and improve the API to raise visibility on an important
> > >>> concept
> >  in
> > >> Kafka Streams: grace period nad out-of-order data handling.
> > >>
> > >> Here's the KIP:
> > >>
> > >
> > 
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
> > >> <
> > >
> > 
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24hr+default+grace+period
> > >>
> > >>
> > >> Cheers,
> > >> Sophie
> > >>
> > >
> > 
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> >
>

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

[Israel Ekpo]

I have reviewed the KIP. The motivation makes sense and the recommended API
changes make sense as well.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams

So +1

I would also like to volunteer to implement it if that is ok.



On Thu, Apr 8, 2021 at 5:42 PM Matthias J. Sax  wrote:

> +1 (binding)
>
> On 4/6/21 10:15 AM, Lotz Utfpr wrote:
> > Makes sense to me! +1
> >
> > Apologies for being brief. This email was sent from my mobile phone.
> >
> >> On 6 Apr 2021, at 18:45, Walker Carlson 
> wrote:
> >>
> >> This makes sense to me +1!
> >>
> >> Walker
> >>
> >>> On Tue, Apr 6, 2021 at 11:08 AM Guozhang Wang 
> wrote:
> >>>
> >>> +1. Thanks!
> >>>
> >>> On Tue, Apr 6, 2021 at 7:00 AM Leah Thomas
> 
> >>> wrote:
> >>>
>  Thanks for picking this up, Sophie. +1 from me, non-binding.
> 
>  Leah
> 
> > On Mon, Apr 5, 2021 at 9:42 PM John Roesler 
> wrote:
> 
> > Thanks, Sophie,
> >
> > I’m +1 (binding)
> >
> > -John
> >
> > On Mon, Apr 5, 2021, at 21:34, Sophie Blee-Goldman wrote:
> >> Hey all,
> >>
> >> I'd like to start the voting on KIP-633, to drop the awkward 24 hour
> > grace
> >> period and improve the API to raise visibility on an important
> >>> concept
>  in
> >> Kafka Streams: grace period nad out-of-order data handling.
> >>
> >> Here's the KIP:
> >>
> >
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
> >> <
> >
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24hr+default+grace+period
> >>
> >>
> >> Cheers,
> >> Sophie
> >>
> >
> 
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
>

Re: [DISCUSS] KIP-633: Drop 24 hour default of grace period in Streams

[Matthias J. Sax]

Thanks!

On 4/8/21 2:06 PM, Sophie Blee-Goldman wrote:
> 1) Since the new APIs (eg ofSizeWithNoGrace and ofSizeAndGrace) are the
> only static constructors
> after this change, there seems to be no reason to keep the .grace around --
> you've already specified
> it with your choice of the static constructor.
> 
> 2) Ack, updated the KIP
> 
> 3) Ack, fixed
> 
> On Tue, Apr 6, 2021 at 7:03 PM Matthias J. Sax  wrote:
> 
>> Thanks for the KIP Sophie. It make total sense to get rid of default
>> grace period of 24h.
>>
>>
>> Some questions/comments:
>>
>> (1) Is there any particular reason why we want to remove
>> `grace(Duration)` method?
>>
>>
>> (2) About `SlidingWindows#withTimeDifferenceAndGrace` -- personally I
>> think it's worth to clean it up right now -- given that sliding windows
>> are rather new the "splash radius" should be small.
>>
>>
>>
>>
>> (3) Some nits on wording:
>>
>>> This config determines how long after a window closes any new data will
>> still be processed
>>
>> Should be "after a window ends" -- a window is closed after grace period
>> passed.
>>
>>
>>> one which indicates to use no grace period and not handle out-of-order
>> data
>>
>> Seems strictly not correct -- if there is a window from 0 to 100 and you
>> get record with ts 99,98,97,...,0 all but the first of those records are
>> out-of-order but they are still processed even with a grace period of zero.
>>
>> Maybe better: "one which indicate to use no grace period and close the
>> window immediately when the window ends."
>>
>>
>>> and make a conscious decision to skip the grace period and drop
>> out-of-order records,
>>
>> Maybe better: "and make a conscious decision to skip the grace period
>> and close a window immediately"
>>
>>
>>
>> -Matthias
>>
>>
>>
>>
>> On 3/31/21 5:02 PM, Guozhang Wang wrote:
>>> Hello Sophie,
>>>
>>> I agree that the old 24-hour grace period should be updated, and I also
>>> think now it is a better idea to make the grace period "mandatory" from
>> the
>>> API names since it is a very important concept and hence worth
>> emphasizing
>>> to users up front.
>>>
>>> Guozhang
>>>
>>> On Wed, Mar 31, 2021 at 1:58 PM John Roesler 
>> wrote:
>>>
 Thanks for bringing this up, Sophie!

 This has indeed been a pain point for a lot of people.

 It's a really thorny issue with no obvious "right" solution.
 I think your proposal is a good one.

 Thanks,
 -John

 On Wed, 2021-03-31 at 13:28 -0700, Sophie Blee-Goldman
 wrote:
> Hey all,
>
> It's finally time to reconsider the default grace period in Kafka
 Streams,
> and hopefully save a lot of suppression users from the pain of figuring
 out
> why their results don't show up until 24 hours later. Please check out
 the
> proposal and let me know what you think.
>
> KIP:
>

>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
> <

>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24hr+default+grace+period
>
>
> JIRA: https://issues.apache.org/jira/browse/KAFKA-8613
>
> Cheers,
> Sophie



>>>
>>
> 

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

[Matthias J. Sax]

+1 (binding)

On 4/6/21 10:15 AM, Lotz Utfpr wrote:
> Makes sense to me! +1
> 
> Apologies for being brief. This email was sent from my mobile phone.
> 
>> On 6 Apr 2021, at 18:45, Walker Carlson  
>> wrote:
>>
>> This makes sense to me +1!
>>
>> Walker
>>
>>> On Tue, Apr 6, 2021 at 11:08 AM Guozhang Wang  wrote:
>>>
>>> +1. Thanks!
>>>
>>> On Tue, Apr 6, 2021 at 7:00 AM Leah Thomas 
>>> wrote:
>>>
 Thanks for picking this up, Sophie. +1 from me, non-binding.

 Leah

> On Mon, Apr 5, 2021 at 9:42 PM John Roesler  wrote:

> Thanks, Sophie,
>
> I’m +1 (binding)
>
> -John
>
> On Mon, Apr 5, 2021, at 21:34, Sophie Blee-Goldman wrote:
>> Hey all,
>>
>> I'd like to start the voting on KIP-633, to drop the awkward 24 hour
> grace
>> period and improve the API to raise visibility on an important
>>> concept
 in
>> Kafka Streams: grace period nad out-of-order data handling.
>>
>> Here's the KIP:
>>
>

>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
>> <
>

>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24hr+default+grace+period
>>
>>
>> Cheers,
>> Sophie
>>
>

>>>
>>>
>>> --
>>> -- Guozhang
>>>

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

[Ron Dagostino]

Hi David.  I'm wondering if it might be a good idea to have the broker
send information about the last block it successfully received when it
requests a new block.  As the RPC stands right now it can't be
idempotent -- it just tells the controller "provide me a new block,
please".  One case where it might be useful for the RPC to be
idempotent is if the broker never receives the response from the
controller such that it asks again.  That would result in the burning
of the block that the controller provided but that the broker never
received.  Now, granted, the ID space is 64 bits, so we would have to
make ~2^54 requests to burn the entire space, and that isn't going to
happen.  So whether this is actually needed is questionable.  And it
might not be worth it to write the controller side code to make it act
idempotently even if we added the request field to make it possible.
But I figured this is worth mentioning even if we explicitly decide to
reject it.

Ron

On Thu, Apr 8, 2021 at 3:16 PM Ron Dagostino  wrote:
>
> Oh, I see.  Yes, my mistake -- I read it wrong.  You are right that
> all we need in the metadata log is the latest value allocated.
>
> Ron
>
> On Thu, Apr 8, 2021 at 11:21 AM David Arthur  wrote:
> >
> > Ron -- I considered making the RPC response and record use the same (or
> > very similar) fields, but the use case is slightly different. A broker
> > handling the RPC needs to know the bounds of the block since it has no idea
> > what the block size is. Also, the brokers will normally see non-contiguous
> > blocks.
> >
> > For the metadata log, we can just keep track of the latest producer Id that
> > was allocated. It's kind of like a high watermark for producer IDs. This
> > actually saves us from needing an extra field in the record (the KIP has
> > just ProducerIdEnd => int64 in the record).
> >
> > Does that make sense?
> >
> > On Wed, Apr 7, 2021 at 8:44 AM Ron Dagostino  wrote:
> >
> > > Thanks for the KIP, David.
> > >
> > > With the RPC returning a start and length, should the record in the
> > > metadata log do the same thing for consistency and to save the byte
> > > per record?
> > >
> > > Ron
> > >
> > >
> > > On Tue, Apr 6, 2021 at 11:06 PM Ismael Juma  wrote:
> > > >
> > > > Great, thanks. Instead of calling it "bridge release", can we say 3.0?
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Apr 6, 2021 at 7:48 PM David Arthur  wrote:
> > > >
> > > > > Thanks for the feedback, Ismael. Renaming the RPC and using start+len
> > > > > instead of start+end sounds fine.
> > > > >
> > > > > And yes, the controller will allocate the IDs in ZK mode for the 
> > > > > bridge
> > > > > release.
> > > > >
> > > > > I'll update the KIP to reflect these points.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Tue, Apr 6, 2021 at 7:30 PM Ismael Juma  wrote:
> > > > >
> > > > > > Sorry, one more question: the allocation of ids will be done by the
> > > > > > controller even in ZK mode, right?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 4:26 PM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > One additional comment: if you return the number of ids instead of
> > > the
> > > > > > end
> > > > > > > range, you can use an int32.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 4:25 PM Ismael Juma 
> > > wrote:
> > > > > > >
> > > > > > >> Thanks for the KIP, David. Any reason not to rename
> > > > > > >> AllocateProducerIdBlockRequest to AllocateProducerIdsRequest?
> > > > > > >>
> > > > > > >> Ismael
> > > > > > >>
> > > > > > >> On Tue, Apr 6, 2021 at 3:51 PM David Arthur 
> > > wrote:
> > > > > > >>
> > > > > > >>> Hello everyone,
> > > > > > >>>
> > > > > > >>> I'd like to start the discussion for KIP-730
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode
> > > > > > >>>
> > > > > > >>> This KIP proposes a new RPC for generating blocks of IDs for
> > > > > > >>> transactional
> > > > > > >>> and idempotent producers.
> > > > > > >>>
> > > > > > >>> Cheers,
> > > > > > >>> David Arthur
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > David Arthur
> > > > >
> > >
> >
> >
> > --
> > David Arthur

Re: [DISCUSS] KIP-633: Drop 24 hour default of grace period in Streams

[Sophie Blee-Goldman]

1) Since the new APIs (eg ofSizeWithNoGrace and ofSizeAndGrace) are the
only static constructors
after this change, there seems to be no reason to keep the .grace around --
you've already specified
it with your choice of the static constructor.

2) Ack, updated the KIP

3) Ack, fixed

On Tue, Apr 6, 2021 at 7:03 PM Matthias J. Sax  wrote:

> Thanks for the KIP Sophie. It make total sense to get rid of default
> grace period of 24h.
>
>
> Some questions/comments:
>
> (1) Is there any particular reason why we want to remove
> `grace(Duration)` method?
>
>
> (2) About `SlidingWindows#withTimeDifferenceAndGrace` -- personally I
> think it's worth to clean it up right now -- given that sliding windows
> are rather new the "splash radius" should be small.
>
>
>
>
> (3) Some nits on wording:
>
> > This config determines how long after a window closes any new data will
> still be processed
>
> Should be "after a window ends" -- a window is closed after grace period
> passed.
>
>
> > one which indicates to use no grace period and not handle out-of-order
> data
>
> Seems strictly not correct -- if there is a window from 0 to 100 and you
> get record with ts 99,98,97,...,0 all but the first of those records are
> out-of-order but they are still processed even with a grace period of zero.
>
> Maybe better: "one which indicate to use no grace period and close the
> window immediately when the window ends."
>
>
> > and make a conscious decision to skip the grace period and drop
> out-of-order records,
>
> Maybe better: "and make a conscious decision to skip the grace period
> and close a window immediately"
>
>
>
> -Matthias
>
>
>
>
> On 3/31/21 5:02 PM, Guozhang Wang wrote:
> > Hello Sophie,
> >
> > I agree that the old 24-hour grace period should be updated, and I also
> > think now it is a better idea to make the grace period "mandatory" from
> the
> > API names since it is a very important concept and hence worth
> emphasizing
> > to users up front.
> >
> > Guozhang
> >
> > On Wed, Mar 31, 2021 at 1:58 PM John Roesler 
> wrote:
> >
> >> Thanks for bringing this up, Sophie!
> >>
> >> This has indeed been a pain point for a lot of people.
> >>
> >> It's a really thorny issue with no obvious "right" solution.
> >> I think your proposal is a good one.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Wed, 2021-03-31 at 13:28 -0700, Sophie Blee-Goldman
> >> wrote:
> >>> Hey all,
> >>>
> >>> It's finally time to reconsider the default grace period in Kafka
> >> Streams,
> >>> and hopefully save a lot of suppression users from the pain of figuring
> >> out
> >>> why their results don't show up until 24 hours later. Please check out
> >> the
> >>> proposal and let me know what you think.
> >>>
> >>> KIP:
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
> >>> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24hr+default+grace+period
> >>>
> >>>
> >>> JIRA: https://issues.apache.org/jira/browse/KAFKA-8613
> >>>
> >>> Cheers,
> >>> Sophie
> >>
> >>
> >>
> >
>

Re: [VOTE] 2.6.2 RC0

[Sophie Blee-Goldman]

Closing this vote, please refer to the [VOTE] 2.6.2 RC1 thread

On Wed, Mar 31, 2021 at 3:41 PM Justin Mclean  wrote:

> Hi,
>
> > Can you clarify a few things?
>
> Sorry I'm not subscribed to this list and only just saw this.
>
> > - On the first point, the only thing I see is zstd, which we do not, in
> > fact, ship the library itself, just the jni bindings. Was there anything
> > else you saw?
>
> In the source release you are not (I assume) going to have jersey, BTW the
> LICENSE and NOTICE files disagree about what license it's under. There
> probably no need to mention it in NOTICE.
>
> Ignoring the stream code I think there is some 3rd party code in the
> source release that would normally be mentioned in LICENSE. [1][2][3][4]
>
> Thanks,
> Justin
>
> 1.
> ./clients/src/main/java/org/apache/kafka/common/utils/PureJavaCrc32C.java
> 2.
> ./streams/src/main/java/org/apache/kafka/streams/state/internals/Murmur3.java
> 3 ./tests/kafkatest/utils/util.py (and a couple of other files)
> 4 ./gradlew
>
>
>

[VOTE] 2.6.2 RC1

[Sophie Blee-Goldman]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 2.6.2.

Apache Kafka 2.6.2 is a bugfix release and fixes 35 issues since the 2.6.1
release. Please see the release notes for more information.

Release notes for the 2.6.2 release:
https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, April 13th, 9am PST

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/javadoc/

* Tag to be voted upon (off 2.6 branch) is the 2.6.2 tag:
https://github.com/apache/kafka/releases/tag/2.6.2-rc1

* Documentation:
https://kafka.apache.org/26/documentation.html

* Protocol:
https://kafka.apache.org/26/protocol.html

* Successful Jenkins builds for the 2.6 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka-2.6-jdk8/114/
System tests:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4458/

/**

Thanks,
Sophie

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #5

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12635) Mirrormaker 2 offset sync is incorrect if the target partition is empty

[Frank Yi (Jira)]

Frank Yi created KAFKA-12635:


 Summary: Mirrormaker 2 offset sync is incorrect if the target 
partition is empty
 Key: KAFKA-12635
 URL: https://issues.apache.org/jira/browse/KAFKA-12635
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.7.0
Reporter: Frank Yi


This bug occurs when using Mirrormaker with "sync.group.offsets.enabled = true".

If a source partition is empty, but the source consumer group's offset for that 
partition is non-zero, then Mirrormaker sets the target consumer group's offset 
for that partition to the literal, not translated, offset of the source 
consumer group. This state can be reached if the source consumer group consumed 
some records that were now deleted (like by a retention policy), or if 
Mirrormaker replication is set to start at "latest". This bug causes the target 
consumer group's lag for that partition to be negative and breaks offset sync 
for that partition until lag is positive.

The correct behavior when the source partition is empty would be to set the 
target offset to the translated offset, not literal offset, which in this case 
would always be 0. 

Original email thread on this issue: 
https://lists.apache.org/thread.html/r7c54ee5f57227367b911d4abffa72781772d8dd3b72d75eb65ee19f7%40%3Cusers.kafka.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

[Ron Dagostino]

Oh, I see.  Yes, my mistake -- I read it wrong.  You are right that
all we need in the metadata log is the latest value allocated.

Ron

On Thu, Apr 8, 2021 at 11:21 AM David Arthur  wrote:
>
> Ron -- I considered making the RPC response and record use the same (or
> very similar) fields, but the use case is slightly different. A broker
> handling the RPC needs to know the bounds of the block since it has no idea
> what the block size is. Also, the brokers will normally see non-contiguous
> blocks.
>
> For the metadata log, we can just keep track of the latest producer Id that
> was allocated. It's kind of like a high watermark for producer IDs. This
> actually saves us from needing an extra field in the record (the KIP has
> just ProducerIdEnd => int64 in the record).
>
> Does that make sense?
>
> On Wed, Apr 7, 2021 at 8:44 AM Ron Dagostino  wrote:
>
> > Thanks for the KIP, David.
> >
> > With the RPC returning a start and length, should the record in the
> > metadata log do the same thing for consistency and to save the byte
> > per record?
> >
> > Ron
> >
> >
> > On Tue, Apr 6, 2021 at 11:06 PM Ismael Juma  wrote:
> > >
> > > Great, thanks. Instead of calling it "bridge release", can we say 3.0?
> > >
> > > Ismael
> > >
> > > On Tue, Apr 6, 2021 at 7:48 PM David Arthur  wrote:
> > >
> > > > Thanks for the feedback, Ismael. Renaming the RPC and using start+len
> > > > instead of start+end sounds fine.
> > > >
> > > > And yes, the controller will allocate the IDs in ZK mode for the bridge
> > > > release.
> > > >
> > > > I'll update the KIP to reflect these points.
> > > >
> > > > Thanks!
> > > >
> > > > On Tue, Apr 6, 2021 at 7:30 PM Ismael Juma  wrote:
> > > >
> > > > > Sorry, one more question: the allocation of ids will be done by the
> > > > > controller even in ZK mode, right?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Apr 6, 2021 at 4:26 PM Ismael Juma 
> > wrote:
> > > > >
> > > > > > One additional comment: if you return the number of ids instead of
> > the
> > > > > end
> > > > > > range, you can use an int32.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 4:25 PM Ismael Juma 
> > wrote:
> > > > > >
> > > > > >> Thanks for the KIP, David. Any reason not to rename
> > > > > >> AllocateProducerIdBlockRequest to AllocateProducerIdsRequest?
> > > > > >>
> > > > > >> Ismael
> > > > > >>
> > > > > >> On Tue, Apr 6, 2021 at 3:51 PM David Arthur 
> > wrote:
> > > > > >>
> > > > > >>> Hello everyone,
> > > > > >>>
> > > > > >>> I'd like to start the discussion for KIP-730
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode
> > > > > >>>
> > > > > >>> This KIP proposes a new RPC for generating blocks of IDs for
> > > > > >>> transactional
> > > > > >>> and idempotent producers.
> > > > > >>>
> > > > > >>> Cheers,
> > > > > >>> David Arthur
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > David Arthur
> > > >
> >
>
>
> --
> David Arthur

Re: [VOTE] KIP-725: Streamlining configurations for TimeWindowedDeserializer.

[John Roesler]

Hi Sagar,

It does have three binding votes, but it looks like it's a
little shy of the mandatory 72 hours you have to leave the
vote open:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process

Assuming no one shows up with a veto, you can close the vote
after 2021/04/09 02:12:47 UTC.

That said, I've never seen a veto, so I think you're safe to
start preparing the PR for reviews.

Thanks again,
John


On Fri, 2021-04-09 at 00:22 +0530, Sagar wrote:
> Thanks Sophie/ John/ Leah and Guozhang.
> 
> Can I assume that this kip has received sufficient votes to be marked as
> accepted ?
> 
> Sagar.
> 
> On Tue, 6 Apr 2021 at 9:38 PM, Guozhang Wang  wrote:
> 
> > +1. Thanks!
> > 
> > On Tue, Apr 6, 2021 at 7:01 AM Leah Thomas 
> > wrote:
> > 
> > > Hi Sagar, +1 non-binding. Thanks again for doing this.
> > > 
> > > Leah
> > > 
> > > On Mon, Apr 5, 2021 at 9:40 PM John Roesler  wrote:
> > > 
> > > > Thanks, Sagar!
> > > > 
> > > > I’m +1 (binding)
> > > > 
> > > > -John
> > > > 
> > > > On Mon, Apr 5, 2021, at 21:35, Sophie Blee-Goldman wrote:
> > > > > Thanks for the KIP! +1 (binding) from me
> > > > > 
> > > > > Cheers,
> > > > > Sophie
> > > > > 
> > > > > On Mon, Apr 5, 2021 at 7:13 PM Sagar 
> > > wrote:
> > > > > 
> > > > > > Hi All,
> > > > > > 
> > > > > > I would like to start voting on the following KIP:
> > > > > > 
> > > > > > 
> > > > 
> > > 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177047930
> > > > > > 
> > > > > > Thanks!
> > > > > > Sagar.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> > 
> > --
> > -- Guozhang
> > 

Re: [VOTE] KIP-725: Streamlining configurations for TimeWindowedDeserializer.

[Sagar]

Thanks Sophie/ John/ Leah and Guozhang.

Can I assume that this kip has received sufficient votes to be marked as
accepted ?

Sagar.

On Tue, 6 Apr 2021 at 9:38 PM, Guozhang Wang  wrote:

> +1. Thanks!
>
> On Tue, Apr 6, 2021 at 7:01 AM Leah Thomas 
> wrote:
>
> > Hi Sagar, +1 non-binding. Thanks again for doing this.
> >
> > Leah
> >
> > On Mon, Apr 5, 2021 at 9:40 PM John Roesler  wrote:
> >
> > > Thanks, Sagar!
> > >
> > > I’m +1 (binding)
> > >
> > > -John
> > >
> > > On Mon, Apr 5, 2021, at 21:35, Sophie Blee-Goldman wrote:
> > > > Thanks for the KIP! +1 (binding) from me
> > > >
> > > > Cheers,
> > > > Sophie
> > > >
> > > > On Mon, Apr 5, 2021 at 7:13 PM Sagar 
> > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I would like to start voting on the following KIP:
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177047930
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>

[jira] [Resolved] (KAFKA-12619) Ensure LeaderChange message is committed before initializing high watermark

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-12619.
-
Fix Version/s: 2.8.1
   3.0.0
   Resolution: Fixed

> Ensure LeaderChange message is committed before initializing high watermark
> ---
>
> Key: KAFKA-12619
> URL: https://issues.apache.org/jira/browse/KAFKA-12619
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.0.0, 2.8.1
>
>
> KIP-595 describes an extra condition on commitment here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595:ARaftProtocolfortheMetadataQuorum-Fetch.
>  In order to ensure that a newly elected leader's committed entries cannot 
> get lost, it must commit one record from its own epoch. This guarantees that 
> its latest entry is larger (in terms of epoch/offset) than any previously 
> written record which ensures that any future leader must also include it. 
> This is the purpose of the LeaderChange record which is written to the log as 
> soon as the leader gets elected.
> We have this check implemented here: 
> https://github.com/apache/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/LeaderState.java#L122.
>  However, the check needs to be a strict inequality since the epoch start 
> offset does not reflect the LeaderChange record itself. In other words, the 
> check is off by one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] 2.7.1 RC2

[Mickael Maison]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 2.7.1.

Since 2.7.1 RC1, the following JIRAs have been fixed: KAFKA-12593,
KAFKA-12474, KAFKA-12602.

Release notes for the 2.7.1 release:
https://home.apache.org/~mimaison/kafka-2.7.1-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Friday, April 16, 5pm BST

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~mimaison/kafka-2.7.1-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~mimaison/kafka-2.7.1-rc2/javadoc/

* Tag to be voted upon (off 2.7 branch) is the 2.7.1 tag:
https://github.com/apache/kafka/releases/tag/2.7.1-rc2

* Documentation:
https://kafka.apache.org/27/documentation.html

* Protocol:
https://kafka.apache.org/27/protocol.html

* Successful Jenkins builds for the 2.7 branch:
The build is still running, I'll update the thread once it's complete

/**

Thanks,
Mickael

[jira] [Resolved] (KAFKA-12457) Implications of KIP-516 for quorum controller

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-12457.
-
Resolution: Fixed

> Implications of KIP-516 for quorum controller
> -
>
> Key: KAFKA-12457
> URL: https://issues.apache.org/jira/browse/KAFKA-12457
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Justine Olshan
>Priority: Major
>  Labels: kip-500
>
> KIP-516 introduces topic IDs to Kafka. We are in the process of updating many 
> of the protocols to support them. In most cases, we are dropping the topic 
> name entirely from new API versions. I think there are two open questions for 
> KIP-500 in regard to this:
> 1. Can we assume topic ID existence in KIP-500? 
> I think the answer here is yes, and the existing code already assumes it. The 
> nice thing is that KIP-516 brings with it the logic to create topic IDs for 
> existing topics. We can rely on this ability in the bridge release to ensure 
> that all topics have topic IDs. And we can add it to pre-upgrade validations.
> 2. What topic ID should be used for `@metadata`? 
> There are basically two options for this: either use a sentinel topic ID or 
> let the controller generate a new one and write it into a `TopicRecord` when 
> the cluster first initializes. If we assume long term that we won't be able 
> to use topic names in the inter-broker protocol, then a sentinel might really 
> be a necessity since brokers would need to know the topic ID before they can 
> send fetches. In other words, if we generate a unique ID, then we probably 
> still need some sentinel so that followers can fetch the initial 
> `TopicRecord` which contains the ID.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » kafka-2.7-jdk8 #143

[Apache Jenkins Server]

See 


Changes:

[Mickael Maison] KAFKA-12602: Fix LICENSE file (#10474)


--
[...truncated 6.91 MB...]
org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@23d1c307, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@23d1c307, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@1e39c3ca, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@1e39c3ca, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > te

[jira] [Created] (KAFKA-12634) Should checkpoint after restore finished

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-12634:
---

 Summary: Should checkpoint after restore finished
 Key: KAFKA-12634
 URL: https://issues.apache.org/jira/browse/KAFKA-12634
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


For state stores, Kafka Streams maintains local checkpoint files to track the 
offsets of the state store changelog topics. The checkpoint is updated on 
commit or when a task is closed cleanly.

However, after a successful restore, the checkpoint is not written. Thus, if an 
instance crashes after restore but before committing, even if the state is on 
local disk the checkpoint file is missing (indicating that there is no state) 
and thus state would be restored from scratch.

While for most cases, the time between restore end and next commit is small, 
there are cases when this time could be large, for example if there is no new 
input data to be processed (if there is no input data, the commit would be 
skipped).

Thus, we should write the checkpoint file after a successful restore to close 
this gap (or course, only for at-least-once processing).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12633) Remove deprecated "TopologyTestDriver#pipeInput / readOutput"

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-12633:
-

 Summary: Remove deprecated "TopologyTestDriver#pipeInput / 
readOutput"
 Key: KAFKA-12633
 URL: https://issues.apache.org/jira/browse/KAFKA-12633
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang
Assignee: Guozhang Wang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12630) Remove deprecated KafkaClientSupplier#getAdminClient

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-12630.
---
Fix Version/s: 3.0.0
   Resolution: Fixed

> Remove deprecated KafkaClientSupplier#getAdminClient
> 
>
> Key: KAFKA-12630
> URL: https://issues.apache.org/jira/browse/KAFKA-12630
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 3.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

[David Arthur]

Ron -- I considered making the RPC response and record use the same (or
very similar) fields, but the use case is slightly different. A broker
handling the RPC needs to know the bounds of the block since it has no idea
what the block size is. Also, the brokers will normally see non-contiguous
blocks.

For the metadata log, we can just keep track of the latest producer Id that
was allocated. It's kind of like a high watermark for producer IDs. This
actually saves us from needing an extra field in the record (the KIP has
just ProducerIdEnd => int64 in the record).

Does that make sense?

On Wed, Apr 7, 2021 at 8:44 AM Ron Dagostino  wrote:

> Thanks for the KIP, David.
>
> With the RPC returning a start and length, should the record in the
> metadata log do the same thing for consistency and to save the byte
> per record?
>
> Ron
>
>
> On Tue, Apr 6, 2021 at 11:06 PM Ismael Juma  wrote:
> >
> > Great, thanks. Instead of calling it "bridge release", can we say 3.0?
> >
> > Ismael
> >
> > On Tue, Apr 6, 2021 at 7:48 PM David Arthur  wrote:
> >
> > > Thanks for the feedback, Ismael. Renaming the RPC and using start+len
> > > instead of start+end sounds fine.
> > >
> > > And yes, the controller will allocate the IDs in ZK mode for the bridge
> > > release.
> > >
> > > I'll update the KIP to reflect these points.
> > >
> > > Thanks!
> > >
> > > On Tue, Apr 6, 2021 at 7:30 PM Ismael Juma  wrote:
> > >
> > > > Sorry, one more question: the allocation of ids will be done by the
> > > > controller even in ZK mode, right?
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Apr 6, 2021 at 4:26 PM Ismael Juma 
> wrote:
> > > >
> > > > > One additional comment: if you return the number of ids instead of
> the
> > > > end
> > > > > range, you can use an int32.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Apr 6, 2021 at 4:25 PM Ismael Juma 
> wrote:
> > > > >
> > > > >> Thanks for the KIP, David. Any reason not to rename
> > > > >> AllocateProducerIdBlockRequest to AllocateProducerIdsRequest?
> > > > >>
> > > > >> Ismael
> > > > >>
> > > > >> On Tue, Apr 6, 2021 at 3:51 PM David Arthur 
> wrote:
> > > > >>
> > > > >>> Hello everyone,
> > > > >>>
> > > > >>> I'd like to start the discussion for KIP-730
> > > > >>>
> > > > >>>
> > > > >>>
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode
> > > > >>>
> > > > >>> This KIP proposes a new RPC for generating blocks of IDs for
> > > > >>> transactional
> > > > >>> and idempotent producers.
> > > > >>>
> > > > >>> Cheers,
> > > > >>> David Arthur
> > > > >>>
> > > > >>
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
>


-- 
David Arthur

Re: I have a question about apache kafka.

[John Roesler]

Hello Yun Han Nam,

The users@ list is more typical for this kind of topic, but this list is fine, 
too.

What’s the question?

Thanks,
John

On Thu, Apr 8, 2021, at 06:25, 남윤한[Yun Han Nam] wrote:
> Hi.
> 
> I want to ask you a question about Apache Kafka, is this the right one?
> 
> It is a technical question about error log that comes too often.​
>

I have a question about apache kafka.

[남윤한 [Yun Han Nam]]

Hi.

I want to ask you a question about Apache Kafka, is this the right one?

It is a technical question about error log that comes too often.​

Re: [VOTE] 2.7.1 RC1

[Mickael Maison]

Hi,

Closing this vote as we've had to address
https://issues.apache.org/jira/browse/KAFKA-12602.
I'll get a new RC out shortly

Thanks


On Fri, Mar 26, 2021 at 11:48 AM Mickael Maison  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 2.7.1.
>
> Since 2.7.1 RC0, the following JIRAs have been fixed: KAFKA-12508
>
> Release notes for the 2.7.1 release:
> https://home.apache.org/~mimaison/kafka-2.7.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, April 10, 5pm PST
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mimaison/kafka-2.7.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mimaison/kafka-2.7.1-rc1/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.1 tag:
> https://github.com/apache/kafka/releases/tag/2.7.1-rc1
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> The build is still running, I'll update the thread once it's complete
>
> /**
>
> Thanks,
> Mickael

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #18

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-12623) Fix LICENSE in 2.7

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-12623.

Resolution: Fixed

> Fix LICENSE in 2.7
> --
>
> Key: KAFKA-12623
> URL: https://issues.apache.org/jira/browse/KAFKA-12623
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: John Roesler
>Assignee: Mickael Maison
>Priority: Blocker
> Fix For: 2.7.1
>
>
> Just splitting this out as a sub-task.
> I've fixed the parent ticket on trunk and 2.8.
> You'll need to cherry-pick the fix from 2.8 (see 
> [https://github.com/apache/kafka/pull/10474)]
> Then, you can follow the manual verification steps I detailed here: 
> https://issues.apache.org/jira/browse/KAFKA-12622



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #17

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12632) Exception may missed when source task failing

[Jira]

戈震 created KAFKA-12632:
--

 Summary: Exception may missed when source task failing
 Key: KAFKA-12632
 URL: https://issues.apache.org/jira/browse/KAFKA-12632
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.6.1
Reporter: 戈震


When task get a exception, it will jump out of while block and call method 
commitOffsets() in finally block in method 
org.apache.kafka.connect.runtime.WorkerSourceTask.execute(). 

But if there is a exception thrown in commitOffsets(), the origin exception 
instance will be missed. 
{code:java}
//代码占位符
package org.apache.kafka.connect.runtime;

class WorkerSourceTask extends WorkerTask {
@Override
public void execute() {
try {
// throw some exception.
} catch (InterruptedException e) {
} finally {
// if some exception thrown in this method, the origin exception 
will be missed
commitOffsets();
}
}
}
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck

[Tom Bentley]

Congratulations Bill!

On Thu, Apr 8, 2021 at 2:36 AM Luke Chen  wrote:

> Congratulations Bill!
>
> Luke
>
> On Thu, Apr 8, 2021 at 9:17 AM Matthias J. Sax  wrote:
>
> > Hi,
> >
> > It's my pleasure to announce that Bill Bejeck in now a member of the
> > Kafka PMC.
> >
> > Bill has been a Kafka committer since Feb 2019. He has remained
> > active in the community since becoming a committer.
> >
> >
> >
> > Congratulations Bill!
> >
> >  -Matthias, on behalf of Apache Kafka PMC
> >
>