[jira] [Created] (KAFKA-14842) MirrorCheckpointTask can reduce the rpc calls of "listConsumerGroupOffsets(group)" of irrelevant groups at each poll

[hudeqi (Jira)]

hudeqi created KAFKA-14842:
--

 Summary: MirrorCheckpointTask can reduce the rpc calls of 
"listConsumerGroupOffsets(group)" of irrelevant groups at each poll
 Key: KAFKA-14842
 URL: https://issues.apache.org/jira/browse/KAFKA-14842
 Project: Kafka
  Issue Type: Sub-task
  Components: KafkaConnect, mirrormaker
Affects Versions: 3.3.2
Reporter: hudeqi
Assignee: hudeqi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

[Dan S]

I have created the jira, the PR will follow within the next few days.
https://issues.apache.org/jira/browse/KAFKA-14841
Thanks again

On Thu, Mar 23, 2023 at 7:33 PM Philip Nee  wrote:

> Awesome!
>
> On Thu, Mar 23, 2023 at 12:31 PM Dan S  wrote:
>
> > I'll pick it up, thanks!
> >
> > On Thu, Mar 23, 2023, 19:27 Philip Nee  wrote:
> >
> > > Hey Dan,
> > >
> > > Your analysis looks right. I do see TODO item there to implement the
> > > rebalance callback. Would you like to create a jira issue and work on
> > that?
> > >
> > > Thanks,
> > > P
> > >
> > > On Thu, Mar 23, 2023 at 12:11 PM Dan S  wrote:
> > >
> > > > Hi Philip,
> > > >
> > > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our
> case
> > > > we're calling the variant of subscribe that takes a custom
> > > > ConsumerRebalanceListener(which among other things logs when it's
> > > called),
> > > > and we're then calling rebalance (to simulate a rebalance) and
> removing
> > > all
> > > > partitions from the consumer, polling a few times, and then adding
> them
> > > > back. We're noticing our custom listener is never called, which was
> > > > unexpected, but based on the code analysis in the original email
> seems
> > to
> > > > be the current implementation. The question is whether this is indeed
> > > > desired behavior.
> > > >
> > > > On Thu, Mar 23, 2023, 18:17 Philip Nee  wrote:
> > > >
> > > > > Hey Dan,
> > > > >
> > > > > Thanks for looking into this. Are you talking about MockConsumer?
> If
> > > you
> > > > > invoke subscribe(Collection topics), it actually registers
> a
> > > Noop
> > > > > callback. Perhaps this is what you are seeing?
> > > > >
> > > > > P
> > > > >
> > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S 
> > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > It seems to me based on reading the code, that the consumer
> > rebalance
> > > > > > listener that is passed into the mock consumer when subscribing
> to
> > a
> > > > > topic
> > > > > > is not actually called when a rebalance is simulated. My
> > > understanding
> > > > is
> > > > > > that the consumer rebalance listener is called from the consumer
> > > > > > coordinator, which is called by kafka consumer. The mock consumer
> > > > doesn't
> > > > > > seem to use the consumer coordinator or use any other mechanism
> to
> > > call
> > > > > the
> > > > > > consumer rebalance listener. Is my understanding correct? Would
> it
> > > make
> > > > > > sense to trigger the consumer rebalance listener when rebalance
> is
> > > > > called?
> > > > > >
> > > > > > I would be willing to try to make the patch if the behavior is
> > indeed
> > > > > > currently incorrect/incomplete.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Daniel
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-14841) Call ConsumerRebalanceListener when MockConsumer rebalanced

[Daniel Scanteianu (Jira)]

Daniel Scanteianu created KAFKA-14841:
-

 Summary: Call ConsumerRebalanceListener when MockConsumer 
rebalanced
 Key: KAFKA-14841
 URL: https://issues.apache.org/jira/browse/KAFKA-14841
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Daniel Scanteianu
Assignee: Daniel Scanteianu


Currently, passing a custom ConsumerRebalanceListener to mock client is 
supported, but if a rebalance actually happens, the ConsumerRebalanceListener 
is not notified of assigned or revoked TopicPartitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1703

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14840) Handle KRaft snapshots in dual-write mode

[David Arthur (Jira)]

David Arthur created KAFKA-14840:


 Summary: Handle KRaft snapshots in dual-write mode
 Key: KAFKA-14840
 URL: https://issues.apache.org/jira/browse/KAFKA-14840
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft
Affects Versions: 3.4.0
Reporter: David Arthur
Assignee: David Arthur
 Fix For: 3.5.0, 3.4.1


While the KRaft controller is making writes back to ZK during the migration, we 
need to handle the case when a snapshot is loaded. This can happen for a number 
of reasons in KRaft.

The difficulty here is we will need to compare the loaded snapshot with the 
entire state in ZK. Most likely, this will be a very expensive operation.

Without this, dual-write mode cannot safely tolerate a snapshot being loaded, 
so marking this as a 3.5 blocker.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

[Philip Nee]

Awesome!

On Thu, Mar 23, 2023 at 12:31 PM Dan S  wrote:

> I'll pick it up, thanks!
>
> On Thu, Mar 23, 2023, 19:27 Philip Nee  wrote:
>
> > Hey Dan,
> >
> > Your analysis looks right. I do see TODO item there to implement the
> > rebalance callback. Would you like to create a jira issue and work on
> that?
> >
> > Thanks,
> > P
> >
> > On Thu, Mar 23, 2023 at 12:11 PM Dan S  wrote:
> >
> > > Hi Philip,
> > >
> > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
> > > we're calling the variant of subscribe that takes a custom
> > > ConsumerRebalanceListener(which among other things logs when it's
> > called),
> > > and we're then calling rebalance (to simulate a rebalance) and removing
> > all
> > > partitions from the consumer, polling a few times, and then adding them
> > > back. We're noticing our custom listener is never called, which was
> > > unexpected, but based on the code analysis in the original email seems
> to
> > > be the current implementation. The question is whether this is indeed
> > > desired behavior.
> > >
> > > On Thu, Mar 23, 2023, 18:17 Philip Nee  wrote:
> > >
> > > > Hey Dan,
> > > >
> > > > Thanks for looking into this. Are you talking about MockConsumer?  If
> > you
> > > > invoke subscribe(Collection topics), it actually registers a
> > Noop
> > > > callback. Perhaps this is what you are seeing?
> > > >
> > > > P
> > > >
> > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S 
> wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > It seems to me based on reading the code, that the consumer
> rebalance
> > > > > listener that is passed into the mock consumer when subscribing to
> a
> > > > topic
> > > > > is not actually called when a rebalance is simulated. My
> > understanding
> > > is
> > > > > that the consumer rebalance listener is called from the consumer
> > > > > coordinator, which is called by kafka consumer. The mock consumer
> > > doesn't
> > > > > seem to use the consumer coordinator or use any other mechanism to
> > call
> > > > the
> > > > > consumer rebalance listener. Is my understanding correct? Would it
> > make
> > > > > sense to trigger the consumer rebalance listener when rebalance is
> > > > called?
> > > > >
> > > > > I would be willing to try to make the patch if the behavior is
> indeed
> > > > > currently incorrect/incomplete.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Daniel
> > > > >
> > > >
> > >
> >
>

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

[Dan S]

I'll pick it up, thanks!

On Thu, Mar 23, 2023, 19:27 Philip Nee  wrote:

> Hey Dan,
>
> Your analysis looks right. I do see TODO item there to implement the
> rebalance callback. Would you like to create a jira issue and work on that?
>
> Thanks,
> P
>
> On Thu, Mar 23, 2023 at 12:11 PM Dan S  wrote:
>
> > Hi Philip,
> >
> > Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
> > we're calling the variant of subscribe that takes a custom
> > ConsumerRebalanceListener(which among other things logs when it's
> called),
> > and we're then calling rebalance (to simulate a rebalance) and removing
> all
> > partitions from the consumer, polling a few times, and then adding them
> > back. We're noticing our custom listener is never called, which was
> > unexpected, but based on the code analysis in the original email seems to
> > be the current implementation. The question is whether this is indeed
> > desired behavior.
> >
> > On Thu, Mar 23, 2023, 18:17 Philip Nee  wrote:
> >
> > > Hey Dan,
> > >
> > > Thanks for looking into this. Are you talking about MockConsumer?  If
> you
> > > invoke subscribe(Collection topics), it actually registers a
> Noop
> > > callback. Perhaps this is what you are seeing?
> > >
> > > P
> > >
> > > On Thu, Mar 23, 2023 at 11:11 AM Dan S  wrote:
> > >
> > > > Hello all,
> > > >
> > > > It seems to me based on reading the code, that the consumer rebalance
> > > > listener that is passed into the mock consumer when subscribing to a
> > > topic
> > > > is not actually called when a rebalance is simulated. My
> understanding
> > is
> > > > that the consumer rebalance listener is called from the consumer
> > > > coordinator, which is called by kafka consumer. The mock consumer
> > doesn't
> > > > seem to use the consumer coordinator or use any other mechanism to
> call
> > > the
> > > > consumer rebalance listener. Is my understanding correct? Would it
> make
> > > > sense to trigger the consumer rebalance listener when rebalance is
> > > called?
> > > >
> > > > I would be willing to try to make the patch if the behavior is indeed
> > > > currently incorrect/incomplete.
> > > >
> > > > Thanks,
> > > >
> > > > Daniel
> > > >
> > >
> >
>

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

[Philip Nee]

Hey Dan,

Your analysis looks right. I do see TODO item there to implement the
rebalance callback. Would you like to create a jira issue and work on that?

Thanks,
P

On Thu, Mar 23, 2023 at 12:11 PM Dan S  wrote:

> Hi Philip,
>
> Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
> we're calling the variant of subscribe that takes a custom
> ConsumerRebalanceListener(which among other things logs when it's called),
> and we're then calling rebalance (to simulate a rebalance) and removing all
> partitions from the consumer, polling a few times, and then adding them
> back. We're noticing our custom listener is never called, which was
> unexpected, but based on the code analysis in the original email seems to
> be the current implementation. The question is whether this is indeed
> desired behavior.
>
> On Thu, Mar 23, 2023, 18:17 Philip Nee  wrote:
>
> > Hey Dan,
> >
> > Thanks for looking into this. Are you talking about MockConsumer?  If you
> > invoke subscribe(Collection topics), it actually registers a Noop
> > callback. Perhaps this is what you are seeing?
> >
> > P
> >
> > On Thu, Mar 23, 2023 at 11:11 AM Dan S  wrote:
> >
> > > Hello all,
> > >
> > > It seems to me based on reading the code, that the consumer rebalance
> > > listener that is passed into the mock consumer when subscribing to a
> > topic
> > > is not actually called when a rebalance is simulated. My understanding
> is
> > > that the consumer rebalance listener is called from the consumer
> > > coordinator, which is called by kafka consumer. The mock consumer
> doesn't
> > > seem to use the consumer coordinator or use any other mechanism to call
> > the
> > > consumer rebalance listener. Is my understanding correct? Would it make
> > > sense to trigger the consumer rebalance listener when rebalance is
> > called?
> > >
> > > I would be willing to try to make the patch if the behavior is indeed
> > > currently incorrect/incomplete.
> > >
> > > Thanks,
> > >
> > > Daniel
> > >
> >
>

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

[Dan S]

Hi Philip,

Thanks for the quick reply. Yes, it's the MockConsumer, but in our case
we're calling the variant of subscribe that takes a custom
ConsumerRebalanceListener(which among other things logs when it's called),
and we're then calling rebalance (to simulate a rebalance) and removing all
partitions from the consumer, polling a few times, and then adding them
back. We're noticing our custom listener is never called, which was
unexpected, but based on the code analysis in the original email seems to
be the current implementation. The question is whether this is indeed
desired behavior.

On Thu, Mar 23, 2023, 18:17 Philip Nee  wrote:

> Hey Dan,
>
> Thanks for looking into this. Are you talking about MockConsumer?  If you
> invoke subscribe(Collection topics), it actually registers a Noop
> callback. Perhaps this is what you are seeing?
>
> P
>
> On Thu, Mar 23, 2023 at 11:11 AM Dan S  wrote:
>
> > Hello all,
> >
> > It seems to me based on reading the code, that the consumer rebalance
> > listener that is passed into the mock consumer when subscribing to a
> topic
> > is not actually called when a rebalance is simulated. My understanding is
> > that the consumer rebalance listener is called from the consumer
> > coordinator, which is called by kafka consumer. The mock consumer doesn't
> > seem to use the consumer coordinator or use any other mechanism to call
> the
> > consumer rebalance listener. Is my understanding correct? Would it make
> > sense to trigger the consumer rebalance listener when rebalance is
> called?
> >
> > I would be willing to try to make the patch if the behavior is indeed
> > currently incorrect/incomplete.
> >
> > Thanks,
> >
> > Daniel
> >
>

Re: Should the mock consumer call the consumer rebalance listener on rebalance?

[Philip Nee]

Hey Dan,

Thanks for looking into this. Are you talking about MockConsumer?  If you
invoke subscribe(Collection topics), it actually registers a Noop
callback. Perhaps this is what you are seeing?

P

On Thu, Mar 23, 2023 at 11:11 AM Dan S  wrote:

> Hello all,
>
> It seems to me based on reading the code, that the consumer rebalance
> listener that is passed into the mock consumer when subscribing to a topic
> is not actually called when a rebalance is simulated. My understanding is
> that the consumer rebalance listener is called from the consumer
> coordinator, which is called by kafka consumer. The mock consumer doesn't
> seem to use the consumer coordinator or use any other mechanism to call the
> consumer rebalance listener. Is my understanding correct? Would it make
> sense to trigger the consumer rebalance listener when rebalance is called?
>
> I would be willing to try to make the patch if the behavior is indeed
> currently incorrect/incomplete.
>
> Thanks,
>
> Daniel
>

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Yi,

> Does it mean with a flexible version, the future
version of these value types will stay at the version where the flexibility
is first introduced? Will there be any need to bump the version again in
the future?

Yes, there will be no need to bump the version since we will only be adding
tagged fields but in the chance the version is bumped, we will deserialize
to the highest known (flexible) version which will ignore unknown tagged
fields.

> To enforce the version not bumping, is it possible to have a unit test to
gate?

Do you have some tests in mind? I don't think we can tell whether a version
was bumped or not.

Best,
Jeff

On Thu, Mar 23, 2023 at 12:07 PM Yi Ding  wrote:

> Hi Jeff,
>
> Thanks for the update. Does it mean with a flexible version, the future
> version of these value types will stay at the version where the flexibility
> is first introduced? Will there be any need to bump the version again in
> the future?
> To enforce the version not bumping, is it possible to have a unit test to
> gate?
>
>
> On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim 
> wrote:
>
> > Hi all,
> >
> > After discussing with my colleagues, I have repurposed the KIP for a
> > general downgrade solution for both transaction and group coordinators.
> The
> > KIP no longer discusses the downgrade path but instead lays out the
> > foundation for future downgrade solutions.
> >
> > Link:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
> >
> > Thanks,
> > Jeff
> >
> > On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim  wrote:
> >
> > > Hi David and Justine,
> > >
> > > Thank you both for the detailed feedback.
> > >
> > > David,
> > >
> > > 1. That makes sense. I revised the "Reading new fields" section with
> how
> > > we can downgrade to the highest known version and that this was
> confirmed
> > > via unit testing. I also attempted to dive deeper into using tagged
> > fields
> > > and the rejected alternative. Please let me know what you think.
> > >
> > > 2. Under "Restrictions and Guidelines" I updated the paragraph to
> clearly
> > > state that we want to use tagged fields across all record types
> > introduced
> > > in KIP-848 including OffsetCommitValue.
> > >
> > > 3. Would it be possible to bump the OffsetCommitValue record version to
> > > make it flexible along with the changes to parse with the highest known
> > > version? I'm not sure I understand why we cannot make both changes
> > together.
> > >
> > > 4. I completely missed this. Added some notes at the end of
> "Restrictions
> > > and Guidelines". Unfortunately I can't think of a solution at the
> moment.
> > > Will get back to you.
> > >
> > > 5. I have a section under "Identifying New Record Types" that discusses
> > > this:
> > >  > We can automate the cleanup by writing tombstones when the
> coordinator
> > > reads unrecognizable records. This may add duplicate work if tombstones
> > > were already added but not yet pruned by the log cleaner.
> > > This is a sure way to delete any unknown record types even if the
> > operator
> > > does not follow the steps.
> > >
> > > 6. Thanks, I have expanded on the section on transactional offset
> > commits.
> > > As for log compaction, my understanding was that we can control the
> > process
> > > by forcing compaction. Is my understanding incorrect?
> > >
> > > 7. We throw an exception if ConfigResource.type is unknown which is
> true
> > > in our case because KIP-848 introduces a new GROUP resource type. We
> will
> > > need to add some sort of safeguard if the dynamic configs are not
> deleted
> > > before the server downgrade. I'll give you an update once I update the
> > > section.
> > >
> > > Justine,
> > >
> > > On the transactional offset commits, I have updated the KIP to reflect
> > > your points after our offline discussion (much appreciated). It seems
> > that
> > > the work required to abort from the server side is fairly big and will
> > > require additional investigation if we are to go down this path.
> > >
> > > As for downgrade limitations, I missed that part so thanks for the
> catch
> > > (along with David). Unfortunately, the proposed design won't allow
> > > downgrades even after the new record types are deleted because the
> > existing
> > > OffsetCommitValue record is not flexible and will not be able to parse
> > > the topicId tagged field. I'll think more about this and get back to
> you.
> > >
> > > Best,
> > > Jeff
> > >
> > > On Mon, Mar 20, 2023 at 2:16 PM Justine Olshan
> > >  wrote:
> > >
> > >> Hey Jeff and David,
> > >>
> > >> Thanks for the KIP! I was also looking into this a bit since I may
> want
> > to
> > >> change the record format for KIP-890 as well (finally implementing the
> > >> record change from KIP-360 to better support the epoch bump. This will
> > >> potentially be helpful for me to implement that work.
> > >>
> > >> I discussed some aspects with David offline. Namely, the alternative
> > >> solution 

Should the mock consumer call the consumer rebalance listener on rebalance?

[Dan S]

Hello all,

It seems to me based on reading the code, that the consumer rebalance
listener that is passed into the mock consumer when subscribing to a topic
is not actually called when a rebalance is simulated. My understanding is
that the consumer rebalance listener is called from the consumer
coordinator, which is called by kafka consumer. The mock consumer doesn't
seem to use the consumer coordinator or use any other mechanism to call the
consumer rebalance listener. Is my understanding correct? Would it make
sense to trigger the consumer rebalance listener when rebalance is called?

I would be willing to try to make the patch if the behavior is indeed
currently incorrect/incomplete.

Thanks,

Daniel

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1702

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 544945 lines...]
[2023-03-23T17:49:21.581Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() 
PASSED
[2023-03-23T17:49:21.581Z] 
[2023-03-23T17:49:21.581Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() STARTED
[2023-03-23T17:49:21.581Z] 
[2023-03-23T17:49:21.581Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZooKeeperSessionStateMetric() PASSED
[2023-03-23T17:49:21.581Z] 
[2023-03-23T17:49:21.581Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testExceptionInBeforeInitializingSession() 
STARTED
[2023-03-23T17:49:22.506Z] 
[2023-03-23T17:49:22.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testExceptionInBeforeInitializingSession() 
PASSED
[2023-03-23T17:49:22.506Z] 
[2023-03-23T17:49:22.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetChildrenExistingZNode() STARTED
[2023-03-23T17:49:22.506Z] 
[2023-03-23T17:49:22.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetChildrenExistingZNode() PASSED
[2023-03-23T17:49:22.506Z] 
[2023-03-23T17:49:22.506Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testConnection() STARTED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testConnection() PASSED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZNodeChangeHandlerForCreation() STARTED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testZNodeChangeHandlerForCreation() PASSED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetAclExistingZNode() STARTED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testGetAclExistingZNode() PASSED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testSessionExpiryDuringClose() STARTED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testSessionExpiryDuringClose() PASSED
[2023-03-23T17:49:23.432Z] 
[2023-03-23T17:49:23.432Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 176 > ZooKeeperClientTest > testReinitializeAfterAuthFailure() STARTED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] > Task :streams:integrationTest
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 PASSED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores() STARTED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores() PASSED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() PASSED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores() STARTED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores() PASSED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 177 > StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault() STARTED
[2023-03-23T17:49:24.357Z] 
[2023-03-23T17:49:24.357Z] Gradle Test Run :streams:integrationTest > 

[jira] [Created] (KAFKA-14839) Exclude protected variable from JavaDocs

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-14839:
---

 Summary: Exclude protected variable from JavaDocs
 Key: KAFKA-14839
 URL: https://issues.apache.org/jira/browse/KAFKA-14839
 Project: Kafka
  Issue Type: Bug
  Components: documentation, streams
Reporter: Matthias J. Sax


Cf 
[https://kafka.apache.org/31/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#enableSpuriousResultFix]

The variable `enableSpuriousResultFix` is protected, and it's not public API, 
and thus should not show up in the JavaDocs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Yi Ding]

Hi Jeff,

Thanks for the update. Does it mean with a flexible version, the future
version of these value types will stay at the version where the flexibility
is first introduced? Will there be any need to bump the version again in
the future?
To enforce the version not bumping, is it possible to have a unit test to
gate?


On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim 
wrote:

> Hi all,
>
> After discussing with my colleagues, I have repurposed the KIP for a
> general downgrade solution for both transaction and group coordinators. The
> KIP no longer discusses the downgrade path but instead lays out the
> foundation for future downgrade solutions.
>
> Link:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
>
> Thanks,
> Jeff
>
> On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim  wrote:
>
> > Hi David and Justine,
> >
> > Thank you both for the detailed feedback.
> >
> > David,
> >
> > 1. That makes sense. I revised the "Reading new fields" section with how
> > we can downgrade to the highest known version and that this was confirmed
> > via unit testing. I also attempted to dive deeper into using tagged
> fields
> > and the rejected alternative. Please let me know what you think.
> >
> > 2. Under "Restrictions and Guidelines" I updated the paragraph to clearly
> > state that we want to use tagged fields across all record types
> introduced
> > in KIP-848 including OffsetCommitValue.
> >
> > 3. Would it be possible to bump the OffsetCommitValue record version to
> > make it flexible along with the changes to parse with the highest known
> > version? I'm not sure I understand why we cannot make both changes
> together.
> >
> > 4. I completely missed this. Added some notes at the end of "Restrictions
> > and Guidelines". Unfortunately I can't think of a solution at the moment.
> > Will get back to you.
> >
> > 5. I have a section under "Identifying New Record Types" that discusses
> > this:
> >  > We can automate the cleanup by writing tombstones when the coordinator
> > reads unrecognizable records. This may add duplicate work if tombstones
> > were already added but not yet pruned by the log cleaner.
> > This is a sure way to delete any unknown record types even if the
> operator
> > does not follow the steps.
> >
> > 6. Thanks, I have expanded on the section on transactional offset
> commits.
> > As for log compaction, my understanding was that we can control the
> process
> > by forcing compaction. Is my understanding incorrect?
> >
> > 7. We throw an exception if ConfigResource.type is unknown which is true
> > in our case because KIP-848 introduces a new GROUP resource type. We will
> > need to add some sort of safeguard if the dynamic configs are not deleted
> > before the server downgrade. I'll give you an update once I update the
> > section.
> >
> > Justine,
> >
> > On the transactional offset commits, I have updated the KIP to reflect
> > your points after our offline discussion (much appreciated). It seems
> that
> > the work required to abort from the server side is fairly big and will
> > require additional investigation if we are to go down this path.
> >
> > As for downgrade limitations, I missed that part so thanks for the catch
> > (along with David). Unfortunately, the proposed design won't allow
> > downgrades even after the new record types are deleted because the
> existing
> > OffsetCommitValue record is not flexible and will not be able to parse
> > the topicId tagged field. I'll think more about this and get back to you.
> >
> > Best,
> > Jeff
> >
> > On Mon, Mar 20, 2023 at 2:16 PM Justine Olshan
> >  wrote:
> >
> >> Hey Jeff and David,
> >>
> >> Thanks for the KIP! I was also looking into this a bit since I may want
> to
> >> change the record format for KIP-890 as well (finally implementing the
> >> record change from KIP-360 to better support the epoch bump. This will
> >> potentially be helpful for me to implement that work.
> >>
> >> I discussed some aspects with David offline. Namely, the alternative
> >> solution to rewrite the records in the old format upon downgrade. One
> plus
> >> to this approach is that it should trigger compaction for all the
> existing
> >> (new format) records.
> >> The main issue we ran into was how to handle ongoing transactions. (Ie,
> >> could we abort and rewrite with the old format) I think that's one of
> the
> >> main downsides. I think it could potentially be possible to do this, but
> >> we'd definitely need a server-side mechanism to abort and rewrite the
> >> records.
> >>
> >> I think the trouble I was running into with the current solution is that
> >> we
> >> can only downgrade to a version slightly before the new group
> coordinator.
> >> If someone was running 3.2 and they upgrade to 3.6, but find a bug in
> 3.4
> >> (or any version between 3.5 and the one they upgraded from), then they
> can
> >> only downgrade to 3.5. This puts us in a difficult spot. I guess in this
> >> scenario, 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Chia-Ping Tsai]

> If you have a KIP that is accepted, make sure it is listed in
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> and that it's status is accurate.

Thanks for the reminder. Have added KIP-641 to the list.

Thanks,
Chia-Ping

> Mickael Maison  於 2023年3月23日 下午11:51 寫道：
> 
> Hi all,
> 
> KIP Freeze was yesterday. The next milestone is feature freeze on April 12.
> If you have a KIP that is accepted, make sure it is listed in
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> and that it's status is accurate.
> 
> Thanks,
> Mickael
> 
> On Fri, Mar 17, 2023 at 6:22 PM Christo Lolov  wrote:
>> 
>> Hello!
>> 
>> What would you suggest as the best way to get more eyes on KIP-902 as I 
>> would like it to be included it in 3.5.0?
>> 
>> Best,
>> Christo
>> 
>>> On 16 Mar 2023, at 10:33, Mickael Maison  wrote:
>>> 
>>> Hi,
>>> 
>>> This is a reminder that KIP freeze is less than a week away (22 Mar).
>>> For a KIP to be considered for this release, it must be voted and
>>> accepted by that date.
>>> 
>>> Feature freeze will be 3 weeks after this, so if you want KIPs or
>>> other significant changes in the release, please get them ready soon.
>>> 
>>> Thanks,
>>> Mickael
>>> 
 On Tue, Feb 14, 2023 at 10:44 PM Ismael Juma  wrote:
 
 Thanks!
 
 Ismael
 
 On Tue, Feb 14, 2023 at 1:07 PM Mickael Maison 
 wrote:
 
> Hi Ismael,
> 
> Good call. I shifted all dates by 2 weeks and moved them to Wednesdays.
> 
> Thanks,
> Mickael
> 
> On Tue, Feb 14, 2023 at 6:01 PM Ismael Juma  wrote:
>> 
>> Thanks Mickael. A couple of notes:
>> 
>> 1. We typically choose a Wednesday for the various freeze dates - there
> are
>> often 1-2 day slips and it's better if that doesn't require people
>> working through the weekend.
>> 2. Looks like we're over a month later compared to the equivalent release
>> last year (
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.0). I
>> understand that some of it is due to 3.4.0 slipping, but I wonder if we
>> could perhaps aim for the KIP freeze to be one or two weeks earlier.
>> 
>> Ismael
>> 
>> On Tue, Feb 14, 2023 at 8:00 AM Mickael Maison > 
>> wrote:
>> 
>>> Hi,
>>> 
>>> I've created a release plan for 3.5.0 in the wiki:
>>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
>>> 
>>> Current dates are:
>>> 1) KIP Freeze: 07 Apr 2023
>>> 2) Feature Freeze: 27 Apr 2023
>>> 3) Code Freeze: 11 May 2023
>>> 
>>> Please take a look at the plan. Let me know if there are other KIPs
>>> targeting 3.5.0.
>>> Also if you are the author of one of the KIPs that's missing a status
>>> (or the status is incorrect) please update it and let me know.
>>> 
>>> Thanks,
>>> Mickael
>>> 
>>> 
>>> On Thu, Feb 9, 2023 at 9:23 AM Bruno Cadonna 
> wrote:
 
 Thanks, Mickael!
 
 Best,
 Bruno
 
 On 09.02.23 03:15, Luke Chen wrote:
> Hi Mickael,
> Thanks for volunteering!
> 
> Luke
> 
> On Thu, Feb 9, 2023 at 6:23 AM Chris Egerton <
> fearthecel...@gmail.com>
> wrote:
> 
>> Thanks for volunteering, Mickael!
>> 
>> On Wed, Feb 8, 2023 at 1:12 PM José Armando García Sancio
>>  wrote:
>> 
>>> Thanks for volunteering Mickael.
>>> 
>>> --
>>> -José
>>> 
>> 
> 
>>> 
> 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi Christo,

Thanks for opening KIP-902. Since the timeline to Kafka 4.0 is still
unclear, I think this KIP makes sense overall. Kafka should not depend
on end of life dependencies.
However the timing is unfortunate and considering the risks of
upgrading ZooKeeper I think it's a bit too late to make the change in
3.5.0. I'm open to reconsider if there's a strong consensus in the
community that this would be really valuable and if you can
demonstrate the risk is low.

Thanks,
Mickael

On Thu, Mar 23, 2023 at 4:51 PM Mickael Maison  wrote:
>
> Hi all,
>
> KIP Freeze was yesterday. The next milestone is feature freeze on April 12.
> If you have a KIP that is accepted, make sure it is listed in
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> and that it's status is accurate.
>
> Thanks,
> Mickael
>
> On Fri, Mar 17, 2023 at 6:22 PM Christo Lolov  wrote:
> >
> > Hello!
> >
> > What would you suggest as the best way to get more eyes on KIP-902 as I 
> > would like it to be included it in 3.5.0?
> >
> > Best,
> > Christo
> >
> > > On 16 Mar 2023, at 10:33, Mickael Maison  wrote:
> > >
> > > Hi,
> > >
> > > This is a reminder that KIP freeze is less than a week away (22 Mar).
> > > For a KIP to be considered for this release, it must be voted and
> > > accepted by that date.
> > >
> > > Feature freeze will be 3 weeks after this, so if you want KIPs or
> > > other significant changes in the release, please get them ready soon.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >> On Tue, Feb 14, 2023 at 10:44 PM Ismael Juma  wrote:
> > >>
> > >> Thanks!
> > >>
> > >> Ismael
> > >>
> > >> On Tue, Feb 14, 2023 at 1:07 PM Mickael Maison 
> > >> wrote:
> > >>
> > >>> Hi Ismael,
> > >>>
> > >>> Good call. I shifted all dates by 2 weeks and moved them to Wednesdays.
> > >>>
> > >>> Thanks,
> > >>> Mickael
> > >>>
> > >>> On Tue, Feb 14, 2023 at 6:01 PM Ismael Juma  wrote:
> > 
> >  Thanks Mickael. A couple of notes:
> > 
> >  1. We typically choose a Wednesday for the various freeze dates - there
> > >>> are
> >  often 1-2 day slips and it's better if that doesn't require people
> >  working through the weekend.
> >  2. Looks like we're over a month later compared to the equivalent 
> >  release
> >  last year (
> >  https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.0). 
> >  I
> >  understand that some of it is due to 3.4.0 slipping, but I wonder if we
> >  could perhaps aim for the KIP freeze to be one or two weeks earlier.
> > 
> >  Ismael
> > 
> >  On Tue, Feb 14, 2023 at 8:00 AM Mickael Maison 
> >   > 
> >  wrote:
> > 
> > > Hi,
> > >
> > > I've created a release plan for 3.5.0 in the wiki:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > >
> > > Current dates are:
> > > 1) KIP Freeze: 07 Apr 2023
> > > 2) Feature Freeze: 27 Apr 2023
> > > 3) Code Freeze: 11 May 2023
> > >
> > > Please take a look at the plan. Let me know if there are other KIPs
> > > targeting 3.5.0.
> > > Also if you are the author of one of the KIPs that's missing a status
> > > (or the status is incorrect) please update it and let me know.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > On Thu, Feb 9, 2023 at 9:23 AM Bruno Cadonna 
> > >>> wrote:
> > >>
> > >> Thanks, Mickael!
> > >>
> > >> Best,
> > >> Bruno
> > >>
> > >> On 09.02.23 03:15, Luke Chen wrote:
> > >>> Hi Mickael,
> > >>> Thanks for volunteering!
> > >>>
> > >>> Luke
> > >>>
> > >>> On Thu, Feb 9, 2023 at 6:23 AM Chris Egerton <
> > >>> fearthecel...@gmail.com>
> > >>> wrote:
> > >>>
> >  Thanks for volunteering, Mickael!
> > 
> >  On Wed, Feb 8, 2023 at 1:12 PM José Armando García Sancio
> >   wrote:
> > 
> > > Thanks for volunteering Mickael.
> > >
> > > --
> > > -José
> > >
> > 
> > >>>
> > >
> > >>>

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi all,

KIP Freeze was yesterday. The next milestone is feature freeze on April 12.
If you have a KIP that is accepted, make sure it is listed in
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
and that it's status is accurate.

Thanks,
Mickael

On Fri, Mar 17, 2023 at 6:22 PM Christo Lolov  wrote:
>
> Hello!
>
> What would you suggest as the best way to get more eyes on KIP-902 as I would 
> like it to be included it in 3.5.0?
>
> Best,
> Christo
>
> > On 16 Mar 2023, at 10:33, Mickael Maison  wrote:
> >
> > Hi,
> >
> > This is a reminder that KIP freeze is less than a week away (22 Mar).
> > For a KIP to be considered for this release, it must be voted and
> > accepted by that date.
> >
> > Feature freeze will be 3 weeks after this, so if you want KIPs or
> > other significant changes in the release, please get them ready soon.
> >
> > Thanks,
> > Mickael
> >
> >> On Tue, Feb 14, 2023 at 10:44 PM Ismael Juma  wrote:
> >>
> >> Thanks!
> >>
> >> Ismael
> >>
> >> On Tue, Feb 14, 2023 at 1:07 PM Mickael Maison 
> >> wrote:
> >>
> >>> Hi Ismael,
> >>>
> >>> Good call. I shifted all dates by 2 weeks and moved them to Wednesdays.
> >>>
> >>> Thanks,
> >>> Mickael
> >>>
> >>> On Tue, Feb 14, 2023 at 6:01 PM Ismael Juma  wrote:
> 
>  Thanks Mickael. A couple of notes:
> 
>  1. We typically choose a Wednesday for the various freeze dates - there
> >>> are
>  often 1-2 day slips and it's better if that doesn't require people
>  working through the weekend.
>  2. Looks like we're over a month later compared to the equivalent release
>  last year (
>  https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.0). I
>  understand that some of it is due to 3.4.0 slipping, but I wonder if we
>  could perhaps aim for the KIP freeze to be one or two weeks earlier.
> 
>  Ismael
> 
>  On Tue, Feb 14, 2023 at 8:00 AM Mickael Maison  
>  wrote:
> 
> > Hi,
> >
> > I've created a release plan for 3.5.0 in the wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> >
> > Current dates are:
> > 1) KIP Freeze: 07 Apr 2023
> > 2) Feature Freeze: 27 Apr 2023
> > 3) Code Freeze: 11 May 2023
> >
> > Please take a look at the plan. Let me know if there are other KIPs
> > targeting 3.5.0.
> > Also if you are the author of one of the KIPs that's missing a status
> > (or the status is incorrect) please update it and let me know.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Thu, Feb 9, 2023 at 9:23 AM Bruno Cadonna 
> >>> wrote:
> >>
> >> Thanks, Mickael!
> >>
> >> Best,
> >> Bruno
> >>
> >> On 09.02.23 03:15, Luke Chen wrote:
> >>> Hi Mickael,
> >>> Thanks for volunteering!
> >>>
> >>> Luke
> >>>
> >>> On Thu, Feb 9, 2023 at 6:23 AM Chris Egerton <
> >>> fearthecel...@gmail.com>
> >>> wrote:
> >>>
>  Thanks for volunteering, Mickael!
> 
>  On Wed, Feb 8, 2023 at 1:12 PM José Armando García Sancio
>   wrote:
> 
> > Thanks for volunteering Mickael.
> >
> > --
> > -José
> >
> 
> >>>
> >
> >>>

[jira] [Resolved] (KAFKA-14814) Skip restart of connectors when redundant resume request is made

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14814.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Skip restart of connectors when redundant resume request is made
> 
>
> Key: KAFKA-14814
> URL: https://issues.apache.org/jira/browse/KAFKA-14814
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chaitanya Mukka
>Priority: Minor
> Fix For: 3.5.0
>
>
> Consecutive requests to the {{PUT /connectors//resume}} endpoint will 
> cause the Connector to be restarted. This is a little wasteful and conflicts 
> with the idempotent nature of that endpoint. We can tweak the 
> {{MemoryConfigBackingStore}} and {{KafkaConfigBackingStore}} classes to not 
> invoke the {{onConnectorTargetStateChange}} method of their 
> {{ConfigUpdateListener}} instance if they pick up a new target state that 
> matches the current target state of the connector.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14838) MM2 Connector/Task clients should specify client ID based on ID and role

[Daniel Urban (Jira)]

Daniel Urban created KAFKA-14838:


 Summary: MM2 Connector/Task clients should specify client ID based 
on ID and role
 Key: KAFKA-14838
 URL: https://issues.apache.org/jira/browse/KAFKA-14838
 Project: Kafka
  Issue Type: Improvement
Reporter: Daniel Urban
Assignee: Daniel Urban


MM2 code creates a lot of Kafka clients internally. These clients generate a 
lot of logs, but since the client.id is not properly specified, connecting the 
dots between a specific Connector/Task and its internal client is close to 
impossible.

The client.id of such clients should specify the Connector name/Task ID, and 
should also specify the role of the client. E.g. MirrorSourceConnector uses 
multiple admin clients, and their client.id should reflect the difference 
between them. This will help log analysis significantly, especially in MM2 mode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14837) The MirrorCheckPointConnector of MM2 will rebalance frequently, when the source cluster group is many more and changes frequently (but the list of configured synchronous

[hudeqi (Jira)]

hudeqi created KAFKA-14837:
--

 Summary: The MirrorCheckPointConnector of MM2 will rebalance 
frequently, when the source cluster group is many more and changes frequently 
(but the list of configured synchronous group does not change)
 Key: KAFKA-14837
 URL: https://issues.apache.org/jira/browse/KAFKA-14837
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect, mirrormaker
Affects Versions: 3.3.2
Reporter: hudeqi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14836) UtilsTest testToLogDateTimeFormat unit test failing in some cases

[Tamas Barnabas Egyed (Jira)]

Tamas Barnabas Egyed created KAFKA-14836:


 Summary: UtilsTest testToLogDateTimeFormat unit test failing in 
some cases
 Key: KAFKA-14836
 URL: https://issues.apache.org/jira/browse/KAFKA-14836
 Project: Kafka
  Issue Type: Bug
Reporter: Tamas Barnabas Egyed
Assignee: Tamas Barnabas Egyed


 
{code:java}
org.apache.kafka.common.utils.UtilsTest.testToLogDateTimeFormat {code}
test is failing in some cases. It uses hard coded datetime ({*}2020-11-09 
12:34:05{*}), while the assertation expected part contains the offset of the 
current time ({*}Instant.now(){*}). This can lead to problems if there is a 
summer and winter time for the specific location. We should use the same 
datetime in the offset query instead of the current time.
{noformat}
org.opentest4j.AssertionFailedError: expected: <2020-11-09 12:34:05,123 -07:00> 
but was: <2020-11-09 12:34:05,123 -08:00>{noformat}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)