[jira] [Created] (KAFKA-16400) Update notification settings for cve detection scheduled workflow

[Vedarth Sharma (Jira)]

Vedarth Sharma created KAFKA-16400:
--

 Summary: Update notification settings for cve detection scheduled 
workflow
 Key: KAFKA-16400
 URL: https://issues.apache.org/jira/browse/KAFKA-16400
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vedarth Sharma
Assignee: Vedarth Sharma


Currently failure email is only sent to the last github user who edited the 
pipeline. We need to figure out a way to send the workflow failure email 
notification to dev mailing list



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-942: Add Power(ppc64le) support

[Vaibhav Nazare]

Hi Team,

We need to add an additional job for ppc64le at 
https://ci-builds.apache.org/job/Kafka/job/kafka/ using a new Jenkins file 
created specific for ppc64le
PR: https://github.com/apache/kafka/pull/13817

We have updated the Jenkinfile.ppc64le with required changes to run builds 
daily on trunk.

But we need help to enable job at 
https://ci-builds.apache.org/job/Kafka/job/kafka/ specific to ppc64le.

Thanks,
Vaibhav

Re: [VOTE] 3.6.2 RC1

[Chia-Ping Tsai]

hi Manikumar

> Pls let me know after merging the PR. I will generate RC2 later today.

Sure. We will complete it ASAP


> Manikumar  於 2024年3月22日 上午9:26 寫道：
> 
> Hi,
> 
> Thanks. Since we have merged KAFKA-16342
> , it's probably better
> to take the PR for KAFKA-16341
>  for consistency.
> I am canceling the RC1 in favour of including KAFKA-16341
> .
> 
> Chia-Ping,
> 
> Pls let me know after merging the PR. I will generate RC2 later today.
> 
> 
> Thank you,
> 
> 
> 
> 
> 
> 
> On Thu, Mar 21, 2024 at 9:10 PM Chia-Ping Tsai  wrote:
> 
>>> Is this a regression from the 3.5.0 release?
>> 
>> I believe the bug is existent for a while, but it is not a true issue until
>> we allowed users to fetch offset of max timestamp.
>> 
>>> Can we update the "Affects Version/s" field on JIRA?
>> 
>> done. I attach the tags for active branches - 3.6.1 and 3.7.0
>> 
>> Manikumar  於 2024年3月21日 週四 下午11:12寫道：
>> 
>>> Hi Chia-Ping,
>>> 
>>> Thanks for letting me know.
>>> 
>>> Is this a regression from the 3.5.0 release?  Can we update the "Affects
>>> Version/s" field on JIRA?
>>> 
>>> Thanks,
>>> 
>>> 
>>> On Thu, Mar 21, 2024 at 5:06 PM Chia-Ping Tsai 
>> wrote:
>>> 
 hi Manikumar,
 
 There is a bug fix which needs to be backport to 3.6 (
 https://issues.apache.org/jira/browse/KAFKA-16341)
 
 It fixes the incorrect offset of max timestamp in non-compress path.
>> The
 other paths are already fixed by
 https://issues.apache.org/jira/browse/KAFKA-16342.
 
 Personally, we should backport both fixes for all paths, and we can
 complete the backport today.
 
 Sorry for bring this news to RC1.
 
 Best,
 Chia-Ping
 
 
 Manikumar  於 2024年3月21日 週四 下午6:11寫道：
 
> Hello Kafka users, developers and client-developers,
> 
> This is the first candidate for release of Apache Kafka 3.6.2.
> 
> This is a bugfix release with several fixes, including dependency
> version bumps for CVEs.
> 
> Release notes for the 3.6.2 release:
> 
>> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html
> 
> *** Please download, test and vote by Tuesday, March 26th
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
> 
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/
> 
> 
> * Maven artifacts to be voted upon:
> 
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> 
> * Javadoc:
> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/
> 
> * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> https://github.com/apache/kafka/releases/tag/3.6.2-rc1
> 
> * Documentation:
> https://kafka.apache.org/36/documentation.html
> 
> * Protocol:
> https://kafka.apache.org/36/protocol.html
> 
> * Successful Jenkins builds for the 3.6 branch:
> Unit/integration tests:
> 
> 
 
>>> 
>> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
> (with few flaky failures)
> System tests: I will update system test results
> 
> Thanks,
> Manikumar
> 
 
>>> 
>> 

Re: [VOTE] 3.6.2 RC1

[Manikumar]

Hi,

Thanks. Since we have merged KAFKA-16342
, it's probably better
to take the PR for KAFKA-16341
 for consistency.
I am canceling the RC1 in favour of including KAFKA-16341
.

Chia-Ping,

Pls let me know after merging the PR. I will generate RC2 later today.


Thank you,






On Thu, Mar 21, 2024 at 9:10 PM Chia-Ping Tsai  wrote:

> > Is this a regression from the 3.5.0 release?
>
> I believe the bug is existent for a while, but it is not a true issue until
> we allowed users to fetch offset of max timestamp.
>
> > Can we update the "Affects Version/s" field on JIRA?
>
> done. I attach the tags for active branches - 3.6.1 and 3.7.0
>
> Manikumar  於 2024年3月21日 週四 下午11:12寫道：
>
> > Hi Chia-Ping,
> >
> > Thanks for letting me know.
> >
> > Is this a regression from the 3.5.0 release?  Can we update the "Affects
> > Version/s" field on JIRA?
> >
> > Thanks,
> >
> >
> > On Thu, Mar 21, 2024 at 5:06 PM Chia-Ping Tsai 
> wrote:
> >
> > > hi Manikumar,
> > >
> > > There is a bug fix which needs to be backport to 3.6 (
> > > https://issues.apache.org/jira/browse/KAFKA-16341)
> > >
> > > It fixes the incorrect offset of max timestamp in non-compress path.
> The
> > > other paths are already fixed by
> > > https://issues.apache.org/jira/browse/KAFKA-16342.
> > >
> > > Personally, we should backport both fixes for all paths, and we can
> > > complete the backport today.
> > >
> > > Sorry for bring this news to RC1.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > > Manikumar  於 2024年3月21日 週四 下午6:11寫道：
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for release of Apache Kafka 3.6.2.
> > > >
> > > > This is a bugfix release with several fixes, including dependency
> > > > version bumps for CVEs.
> > > >
> > > > Release notes for the 3.6.2 release:
> > > >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Tuesday, March 26th
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/
> > > >
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.6.2-rc1
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/36/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/36/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.6 branch:
> > > > Unit/integration tests:
> > > >
> > > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
> > > > (with few flaky failures)
> > > > System tests: I will update system test results
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
>

Re:[DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Hi,

What is the difference between this KIP and KIP-975: Docker Image for Apache 
Kafka? 

From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:  
dev@kafka.apache.org
Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

Hi everyone,

I would like to start the discussion on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
age+for+Apache+Kafka
 .

This KIP aims to introduce JVM based Docker Official Image (DOI) for Apache
Kafka.

Regards,
Krish.

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #161

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 412488 lines...]
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testMigrateTopicConfigs() PASSED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testNonIncreasingKRaftEpoch() STARTED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testNonIncreasingKRaftEpoch() PASSED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testMigrateEmptyZk() STARTED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testMigrateEmptyZk() PASSED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testTopicAndBrokerConfigsMigrationWithSnapshots() 
STARTED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testTopicAndBrokerConfigsMigrationWithSnapshots() 
PASSED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testClaimAndReleaseExistingController() STARTED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testClaimAndReleaseExistingController() PASSED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testClaimAbsentController() STARTED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testClaimAbsentController() PASSED
[2024-03-21T20:39:07.833Z] 
[2024-03-21T20:39:07.833Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testIdempotentCreateTopics() STARTED
[2024-03-21T20:39:09.202Z] 
[2024-03-21T20:39:09.202Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testIdempotentCreateTopics() PASSED
[2024-03-21T20:39:09.202Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testCreateNewTopic() STARTED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testCreateNewTopic() PASSED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testUpdateExistingTopicWithNewAndChangedPartitions() 
STARTED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZkMigrationClientTest > testUpdateExistingTopicWithNewAndChangedPartitions() 
PASSED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() STARTED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testZNodeChangeHandlerForDataChange() PASSED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testZooKeeperSessionStateMetric() STARTED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testZooKeeperSessionStateMetric() PASSED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testExceptionInBeforeInitializingSession() STARTED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testExceptionInBeforeInitializingSession() PASSED
[2024-03-21T20:39:09.203Z] 
[2024-03-21T20:39:09.203Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testGetChildrenExistingZNode() STARTED
[2024-03-21T20:39:10.503Z] 
[2024-03-21T20:39:10.503Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testGetChildrenExistingZNode() PASSED
[2024-03-21T20:39:10.503Z] 
[2024-03-21T20:39:10.503Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testConnection() STARTED
[2024-03-21T20:39:10.503Z] 
[2024-03-21T20:39:10.503Z] Gradle Test Run :core:test > Gradle Test Executor 91 
> ZooKeeperClientTest > testConnection() PASSED
[2024-03-21T20:39:10.503Z] 
[2024-03-21T20:39:10.503Z] Gradle Test Run :core:test > Gradle Test 

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi Jun,

On Thu, Mar 14, 2024 at 3:38 PM Jun Rao  wrote:
> 52. Admin client: Could you provide a bit more details on the changes?

I updated the KIP to include the API changes to the Admin client.

Thanks,
-- 
-José

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

[Justine Olshan]

Hey Ahmed,

Echoing Andrew's comments to clean up the public interfaces, it was unclear
to me if this new flag applies to all the options or just the "latest"
option.
Clarifying that and showing a few examples could help.

Thanks,
Justine

On Thu, Mar 21, 2024 at 9:43 AM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi,
> Glad to hear you’re better.
>
> Re-reading the KIP, I think the changes to the public interfaces are
> probably only the addition
> of the new `--isolation` flag on the `kafka-get-offsets.sh` tool. The
> `KafkaAdminClient.listOffsets`
> parameters already have what you need I think (OffsetSpec.LATEST and
> ListOffsetOptions.isolationLevel).
>
> If you tidy up the `Public Interfaces` section with the revised syntax for
> the tool, I think the
> committers will be able to see more easily what the changes are and it
> would then be ready to vote
> in my opinion.
>
> Thanks,
> Andrew
>
> > On 20 Mar 2024, at 12:56, Ahmed Sobeh 
> wrote:
> >
> > Hi Andrew,
> >
> > Apologies for disappearing, I had to undergo knee surgery, all good now!
> >
> > I adjusted the KIP as you suggested, do you think it's ready for the
> voting
> > stage?
> >
> > On Wed, Mar 6, 2024 at 4:02 PM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi Ahmed,
> >> Looks good to me with one remaining comment.
> >>
> >> You’ve used:
> >>
> >> bin/kafka-get-offsets.sh  … --time -1 --isolation -committed
> >> bin/kafka-get-offsets.sh  … --time latest --isolation -committed
> >> bin/kafka-get-offsets.sh  … --time -1 --isolation -uncommitted
> >> bin/kafka-get-offsets.sh  … --time latest --isolation -uncommitted
> >>
> >> I find the flags starting with a single - to be a bit unusual and
> doesn’t
> >> match the --time options such as “latest”. I suggest:
> >>
> >> bin/kafka-get-offsets.sh  … --time -1 --isolation committed
> >> bin/kafka-get-offsets.sh  … --time latest --isolation committed
> >> bin/kafka-get-offsets.sh  … --time -1 --isolation uncommitted
> >> bin/kafka-get-offsets.sh  … --time latest --isolation uncommitted
> >>
> >> Or even:
> >>
> >> bin/kafka-get-offsets.sh  … --time -1 --isolation read-committed
> >> bin/kafka-get-offsets.sh  … --time latest --isolation read-committed
> >> bin/kafka-get-offsets.sh  … --time -1 --isolation read-uncommitted
> >> bin/kafka-get-offsets.sh  … --time latest --isolation read-uncommitted
> >>
> >> Apart from that nit, looks good to me.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>
> >>> On 5 Mar 2024, at 16:35, Ahmed Sobeh 
> >> wrote:
> >>>
> >>> I adjusted the KIP according to what we agreed on, let me know if you
> >> have
> >>> any comments!
> >>>
> >>> Best,
> >>> Ahmed
> >>>
> >>> On Thu, Feb 29, 2024 at 1:44 AM Luke Chen  wrote:
> >>>
>  Hi Ahmed,
> 
>  Thanks for the KIP!
> 
>  Comments:
>  1. If we all agree with the suggestion from Andrew, you could update
> the
>  KIP.
> 
>  Otherwise, LGTM!
> 
> 
>  Thanks.
>  Luke
> 
>  On Thu, Feb 29, 2024 at 1:32 AM Andrew Schofield <
>  andrew_schofield_j...@outlook.com> wrote:
> 
> > Hi Ahmed,
> > Could do. Personally, I find the existing “--time -1” totally horrid
> > anyway, which was why
> > I suggested an alternative. I think your suggestion of a flag for
> > isolation level is much
> > better than -6.
> >
> > Maybe I should put in a KIP which adds:
> > --latest (as a synonym for --time -1)
> > --earliest (as a synonym for --time -2)
> > --max-timestamp (as a synonym for --time -3)
> >
> > That’s really what I would prefer. If the user has a timestamp, use
> > `--time`. If they want a
> > specific special offset, use a separate flag.
> >
> > Thanks,
> > Andrew
> >
> >> On 28 Feb 2024, at 09:22, Ahmed Sobeh  >
> > wrote:
> >>
> >> Hi Andrew,
> >>
> >> Thanks for the hint! That sounds reasonable, do you think adding a
> >> conditional argument, something like "--time -1 --isolation
> >> -committed"
> > and
> >> "--time -1 --isolation -uncommitted" would make sense to keep the
> >> consistency of getting the offset by time? or do you think having a
> > special
> >> argument for this case is better?
> >>
> >> On Tue, Feb 27, 2024 at 2:19 PM Andrew Schofield <
> >> andrew_schofield_j...@outlook.com> wrote:
> >>
> >>> Hi Ahmed,
> >>> Thanks for the KIP.  It looks like a useful addition.
> >>>
> >>> The use of negative timestamps, and in particular letting the user
> >> use
> >>> `--time -1` or the equivalent `--time latest`
> >>> is a bit peculiar in this tool already. The negative timestamps
> come
> > from
> >>> org.apache.kafka.common.requests.ListOffsetsRequest,
> >>> but you’re not actually adding another value to that. As a result,
> I
> >>> really wouldn’t recommend using -6 for the new
> >>> flag. LSO is really a variant of -1 with 

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

[Andrew Schofield]

Hi,
Glad to hear you’re better.

Re-reading the KIP, I think the changes to the public interfaces are probably 
only the addition
of the new `--isolation` flag on the `kafka-get-offsets.sh` tool. The 
`KafkaAdminClient.listOffsets`
parameters already have what you need I think (OffsetSpec.LATEST and 
ListOffsetOptions.isolationLevel).

If you tidy up the `Public Interfaces` section with the revised syntax for the 
tool, I think the
committers will be able to see more easily what the changes are and it would 
then be ready to vote
in my opinion.

Thanks,
Andrew

> On 20 Mar 2024, at 12:56, Ahmed Sobeh  wrote:
>
> Hi Andrew,
>
> Apologies for disappearing, I had to undergo knee surgery, all good now!
>
> I adjusted the KIP as you suggested, do you think it's ready for the voting
> stage?
>
> On Wed, Mar 6, 2024 at 4:02 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
>> Hi Ahmed,
>> Looks good to me with one remaining comment.
>>
>> You’ve used:
>>
>> bin/kafka-get-offsets.sh  … --time -1 --isolation -committed
>> bin/kafka-get-offsets.sh  … --time latest --isolation -committed
>> bin/kafka-get-offsets.sh  … --time -1 --isolation -uncommitted
>> bin/kafka-get-offsets.sh  … --time latest --isolation -uncommitted
>>
>> I find the flags starting with a single - to be a bit unusual and doesn’t
>> match the --time options such as “latest”. I suggest:
>>
>> bin/kafka-get-offsets.sh  … --time -1 --isolation committed
>> bin/kafka-get-offsets.sh  … --time latest --isolation committed
>> bin/kafka-get-offsets.sh  … --time -1 --isolation uncommitted
>> bin/kafka-get-offsets.sh  … --time latest --isolation uncommitted
>>
>> Or even:
>>
>> bin/kafka-get-offsets.sh  … --time -1 --isolation read-committed
>> bin/kafka-get-offsets.sh  … --time latest --isolation read-committed
>> bin/kafka-get-offsets.sh  … --time -1 --isolation read-uncommitted
>> bin/kafka-get-offsets.sh  … --time latest --isolation read-uncommitted
>>
>> Apart from that nit, looks good to me.
>>
>> Thanks,
>> Andrew
>>
>>
>>> On 5 Mar 2024, at 16:35, Ahmed Sobeh 
>> wrote:
>>>
>>> I adjusted the KIP according to what we agreed on, let me know if you
>> have
>>> any comments!
>>>
>>> Best,
>>> Ahmed
>>>
>>> On Thu, Feb 29, 2024 at 1:44 AM Luke Chen  wrote:
>>>
 Hi Ahmed,

 Thanks for the KIP!

 Comments:
 1. If we all agree with the suggestion from Andrew, you could update the
 KIP.

 Otherwise, LGTM!


 Thanks.
 Luke

 On Thu, Feb 29, 2024 at 1:32 AM Andrew Schofield <
 andrew_schofield_j...@outlook.com> wrote:

> Hi Ahmed,
> Could do. Personally, I find the existing “--time -1” totally horrid
> anyway, which was why
> I suggested an alternative. I think your suggestion of a flag for
> isolation level is much
> better than -6.
>
> Maybe I should put in a KIP which adds:
> --latest (as a synonym for --time -1)
> --earliest (as a synonym for --time -2)
> --max-timestamp (as a synonym for --time -3)
>
> That’s really what I would prefer. If the user has a timestamp, use
> `--time`. If they want a
> specific special offset, use a separate flag.
>
> Thanks,
> Andrew
>
>> On 28 Feb 2024, at 09:22, Ahmed Sobeh 
> wrote:
>>
>> Hi Andrew,
>>
>> Thanks for the hint! That sounds reasonable, do you think adding a
>> conditional argument, something like "--time -1 --isolation
>> -committed"
> and
>> "--time -1 --isolation -uncommitted" would make sense to keep the
>> consistency of getting the offset by time? or do you think having a
> special
>> argument for this case is better?
>>
>> On Tue, Feb 27, 2024 at 2:19 PM Andrew Schofield <
>> andrew_schofield_j...@outlook.com> wrote:
>>
>>> Hi Ahmed,
>>> Thanks for the KIP.  It looks like a useful addition.
>>>
>>> The use of negative timestamps, and in particular letting the user
>> use
>>> `--time -1` or the equivalent `--time latest`
>>> is a bit peculiar in this tool already. The negative timestamps come
> from
>>> org.apache.kafka.common.requests.ListOffsetsRequest,
>>> but you’re not actually adding another value to that. As a result, I
>>> really wouldn’t recommend using -6 for the new
>>> flag. LSO is really a variant of -1 with read_committed isolation
 level.
>>>
>>> I think that perhaps it would be better to add `--last-stable` as an
>>> alternative to `--time`. Then you’ll get the LSO with
>>> cleaner syntax.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
 On 27 Feb 2024, at 10:12, Ahmed Sobeh >>
>>> wrote:

 Hi all,
 I would like to start a discussion on KIP-1021, which would enable
>>> getting
 LSO in kafka-get-offsets.sh:

>>>
>

>> 

Re: [DISCUSS] KIP-932: Queues for Kafka

[Justine Olshan]

Thanks Andrew,

That answers some of the questions I have.

With respect to the limits -- how will this be implemented? One issue we
saw with producers is "short-lived" producers that send one message and
disconnect.
Due to how expiration works for producer state, if we have a simple limit
for producer IDs, all new producers are blocked until the old ones expire.
Will we block new group members as well if we reach our limit?

In the consumer case, we have a heartbeat which can be used for expiration
behavior and avoid the headache we see on the producer side, but I can
imagine a case where misuse of the groups themselves could occur -- ie
creating a short lived share group that I believe will take some time to
expire. Do we have considerations for this case?

I also plan to re-read the read-committed section and may have further
questions there.

You also mentioned in the KIP how there are a few inter-broker hops to the
share coordinator, etc for a given read operation of a partition. Are we
concerned about performance here? My work in transactions and trying to
optimize performance made me realize how expensive these inter-broker hops
can be.

Justine

On Thu, Mar 21, 2024 at 7:37 AM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Justine,
> Thanks for your comment. Sorry for the delay responding.
>
> It was not my intent to leave a query unanswered. I have modified the KIP
> as a result
> of the discussion and I think perhaps I didn’t neatly close off the email
> thread.
>
> In summary:
>
> * The share-partition leader does not maintain an explicit cache of
> records that it has
> fetched. When it fetches records, it does “shallow” iteration to look at
> the batch
> headers only so that it understands at least the base/last offset of the
> records within.
> It is left to the consumers to do the “deep” iteration of the records
> batches they fetch.
>
> * It may sometimes be necessary to re-fetch records for redelivery. This
> is essentially
> analogous to two consumer groups independently fetching the same records
> today.
> We will be relying on the efficiency of the page cache.
>
> * The zero-copy optimisation is not possible for records fetched for
> consumers in
> share groups. The KIP does not affect the use of the zero-copy
> optimisation for any
> scenarios in which it currently applies (this was not true in one earlier
> version of the KIP).
>
> * I share concern about memory usage, partly because of the producer state
> management
> area. To keep a lid on memory use, the number of share groups, the number
> of members
> of each share group, and the number of in-flight messages per partition in
> a share group
> are all limited. The aim is to get the in-memory state to be nice and
> compact, probably
> at the expense of throughput. Over time, I’m sure we’ll optimise and get a
> bit more
> headroom. Limits like these cannot easily be applied retrospectively, so
> the limits are
> there right at the start.
>
> * I have reworked the section on read-committed isolation level, and the
> complexity
> and memory usage of the approach is significantly improved.
>
> I hope this answers your question.
>
> Thanks,
> Andrew
>
>
> > On 18 Mar 2024, at 20:47, Justine Olshan 
> wrote:
> >
> > Hey Andrew,
> >
> > I noticed you started the voting thread, but there seems to be a few
> > questions that were not answered. One was Jun's about memory usage
> >> How much additional heap memory will the server use? Do we need to cache
> > records in heap? If so, is the cache bounded?
> >
> > Your response was
> >> This area needs more work. Using a share group surely gets the broker to
> > do
> > more manipulation of the data that it fetches than a regular consumer. I
> > want to minimise
> > this and need to research before providing a comprehensive answer. I
> > suspect zero-copy
> > is lost and that we do not cache records in heap. I will confirm later
> on.
> >
> > I am also concerned about memory usage from my producer state management
> > work, so I want to make sure we have thought about it here -- not just in
> > the case Jun mentioned but generally.
> >
> > Likewise, we have seen issues with large consumer groups and too many
> > producer IDs. Are there any concerns with an analogous situation with too
> > many share group members or share groups? Are they any ways we try to
> > handle this or mitigate risks with respect to memory usage and client
> > connections (wrt to rebalances for example)
> >
> > Thanks,
> >
> > Justine
> >
> > On Fri, Mar 8, 2024 at 12:51 AM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi Manikumar,
> >> Thanks for your queries.
> >>
> >> 1) Delivery count is added to the ConsumerRecord class so that a
> consumer
> >> can
> >> tell how often a record has been processed. I imagine that some
> >> applications might
> >> want to take different actions based on whether a record has previously
> >> failed. This
> >> enables richer error handling 

Re: [VOTE] 3.6.2 RC1

[Chia-Ping Tsai]

> Is this a regression from the 3.5.0 release?

I believe the bug is existent for a while, but it is not a true issue until
we allowed users to fetch offset of max timestamp.

> Can we update the "Affects Version/s" field on JIRA?

done. I attach the tags for active branches - 3.6.1 and 3.7.0

Manikumar  於 2024年3月21日 週四 下午11:12寫道：

> Hi Chia-Ping,
>
> Thanks for letting me know.
>
> Is this a regression from the 3.5.0 release?  Can we update the "Affects
> Version/s" field on JIRA?
>
> Thanks,
>
>
> On Thu, Mar 21, 2024 at 5:06 PM Chia-Ping Tsai  wrote:
>
> > hi Manikumar,
> >
> > There is a bug fix which needs to be backport to 3.6 (
> > https://issues.apache.org/jira/browse/KAFKA-16341)
> >
> > It fixes the incorrect offset of max timestamp in non-compress path. The
> > other paths are already fixed by
> > https://issues.apache.org/jira/browse/KAFKA-16342.
> >
> > Personally, we should backport both fixes for all paths, and we can
> > complete the backport today.
> >
> > Sorry for bring this news to RC1.
> >
> > Best,
> > Chia-Ping
> >
> >
> > Manikumar  於 2024年3月21日 週四 下午6:11寫道：
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the first candidate for release of Apache Kafka 3.6.2.
> > >
> > > This is a bugfix release with several fixes, including dependency
> > > version bumps for CVEs.
> > >
> > > Release notes for the 3.6.2 release:
> > > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Tuesday, March 26th
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/
> > >
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> > > https://github.com/apache/kafka/releases/tag/3.6.2-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/36/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/36/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.6 branch:
> > > Unit/integration tests:
> > >
> > >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
> > > (with few flaky failures)
> > > System tests: I will update system test results
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>

Re: [DISCUSS] KIP-1008: ParKa - the Marriage of Parquet and Kafka

[Andrew Schofield]

Hi Xinli,
Thanks for the KIP. I see that the discussion thread has died down, which is 
often a tricky situation with a KIP.

I’ve been thinking about this KIP for a while and it was really good to be able 
to attend the Kafka Summit London
session to get a proper understanding of it. I think it’s a really interesting 
idea, but….

I’m afraid I think this is not a good approach to solving the problem. If I 
understand correctly, the fundamental
aim is to insert Parquet data from Kafka clients efficiently into a data lake. 
Part of the attraction is that Parquet
compression is extremely efficient for the truly massive batches (up to 100,000 
records) that you described at
Kafka Summit. While this is an entirely sensible aim, I think there are better 
ways to do the same thing with
Kafka.

The KIP seems asymmetrical to me. The “rows” of your Parquet data are 
individual Kafka records, which are then
batched up and encoded into Parquet with an appropriate schema. But on the 
consumption side, you don’t
really want to treat them as individual records at all. Instead, the entire 
batch is intended to be received and copied into the
data lake as a unit. You also are relying on KIP-712, which is not an approved 
KIP.

I started wondering whether simply introducing Parquet as a new compression 
format for Kafka would be a neat
and generally useful way to proceed. Would it really be generally applicable in 
the way that say, zstd is?
Would it work with all of the parts of Kafka such as the log cleaner? I think 
the answer is that it would not. It’s an
effective encoding only for massive batches, and the requirement that the 
compressor needs to know the schema
means that components in the “middle” that might need to apply compression 
would need to know this information.

I think the best approach would be for your Kafka records to be sent to Kafka 
already in Parquet format. So, you
would accumulate the rows of data into large batches, encode and compress into 
Parquet using the correct schema,
and then send to Kafka in a batch containing one large record. Then the 
existing producer and consumer could be used
without change. I know that this means you end up with very large records, but 
you’re actually ending up with a very similar
situation by creating huge batches of 100,000 smaller records which still need 
to be accommodated by Kafka.

Just my 2 cents. I hope I’ve managed to revitalise the discussion thread and 
get some more opinions too.

Thanks,
Andrew

> On 21 Nov 2023, at 17:20, Xinli shang  wrote:
>
> Hi, all
>
> Can I ask for a discussion on the KIP just created KIP-1008: ParKa - the
> Marriage of Parquet and Kafka
> 
> ?
>
> --
> Xinli Shang

Re: [VOTE] 3.6.2 RC1

[Manikumar]

Hi Chia-Ping,

Thanks for letting me know.

Is this a regression from the 3.5.0 release?  Can we update the "Affects
Version/s" field on JIRA?

Thanks,


On Thu, Mar 21, 2024 at 5:06 PM Chia-Ping Tsai  wrote:

> hi Manikumar,
>
> There is a bug fix which needs to be backport to 3.6 (
> https://issues.apache.org/jira/browse/KAFKA-16341)
>
> It fixes the incorrect offset of max timestamp in non-compress path. The
> other paths are already fixed by
> https://issues.apache.org/jira/browse/KAFKA-16342.
>
> Personally, we should backport both fixes for all paths, and we can
> complete the backport today.
>
> Sorry for bring this news to RC1.
>
> Best,
> Chia-Ping
>
>
> Manikumar  於 2024年3月21日 週四 下午6:11寫道：
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.6.2.
> >
> > This is a bugfix release with several fixes, including dependency
> > version bumps for CVEs.
> >
> > Release notes for the 3.6.2 release:
> > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, March 26th
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/
> >
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.2-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > Unit/integration tests:
> >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
> > (with few flaky failures)
> > System tests: I will update system test results
> >
> > Thanks,
> > Manikumar
> >
>

[jira] [Resolved] (KAFKA-16206) KRaftMigrationZkWriter tries to delete deleted topic configs twice

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-16206.

Fix Version/s: 3.8.0
   Resolution: Fixed

> KRaftMigrationZkWriter tries to delete deleted topic configs twice
> --
>
> Key: KAFKA-16206
> URL: https://issues.apache.org/jira/browse/KAFKA-16206
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft, migration
>Reporter: David Arthur
>Assignee: Alyssa Huang
>Priority: Minor
> Fix For: 3.8.0
>
>
> When deleting a topic, we see spurious ERROR logs from 
> kafka.zk.migration.ZkConfigMigrationClient:
>  
> {code:java}
> Did not delete ConfigResource(type=TOPIC, name='xxx') since the node did not 
> exist. {code}
> This seems to happen because ZkTopicMigrationClient#deleteTopic is deleting 
> the topic, partitions, and config ZNodes in one shot. Subsequent calls from 
> KRaftMigrationZkWriter to delete the config encounter a NO_NODE since the 
> ZNode is already gone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-932: Queues for Kafka

[Andrew Schofield]

Hi Justine,
Thanks for your comment. Sorry for the delay responding.

It was not my intent to leave a query unanswered. I have modified the KIP as a 
result
of the discussion and I think perhaps I didn’t neatly close off the email 
thread.

In summary:

* The share-partition leader does not maintain an explicit cache of records 
that it has
fetched. When it fetches records, it does “shallow” iteration to look at the 
batch
headers only so that it understands at least the base/last offset of the 
records within.
It is left to the consumers to do the “deep” iteration of the records batches 
they fetch.

* It may sometimes be necessary to re-fetch records for redelivery. This is 
essentially
analogous to two consumer groups independently fetching the same records today.
We will be relying on the efficiency of the page cache.

* The zero-copy optimisation is not possible for records fetched for consumers 
in
share groups. The KIP does not affect the use of the zero-copy optimisation for 
any
scenarios in which it currently applies (this was not true in one earlier 
version of the KIP).

* I share concern about memory usage, partly because of the producer state 
management
area. To keep a lid on memory use, the number of share groups, the number of 
members
of each share group, and the number of in-flight messages per partition in a 
share group
are all limited. The aim is to get the in-memory state to be nice and compact, 
probably
at the expense of throughput. Over time, I’m sure we’ll optimise and get a bit 
more
headroom. Limits like these cannot easily be applied retrospectively, so the 
limits are
there right at the start.

* I have reworked the section on read-committed isolation level, and the 
complexity
and memory usage of the approach is significantly improved.

I hope this answers your question.

Thanks,
Andrew


> On 18 Mar 2024, at 20:47, Justine Olshan  wrote:
> 
> Hey Andrew,
> 
> I noticed you started the voting thread, but there seems to be a few
> questions that were not answered. One was Jun's about memory usage
>> How much additional heap memory will the server use? Do we need to cache
> records in heap? If so, is the cache bounded?
> 
> Your response was
>> This area needs more work. Using a share group surely gets the broker to
> do
> more manipulation of the data that it fetches than a regular consumer. I
> want to minimise
> this and need to research before providing a comprehensive answer. I
> suspect zero-copy
> is lost and that we do not cache records in heap. I will confirm later on.
> 
> I am also concerned about memory usage from my producer state management
> work, so I want to make sure we have thought about it here -- not just in
> the case Jun mentioned but generally.
> 
> Likewise, we have seen issues with large consumer groups and too many
> producer IDs. Are there any concerns with an analogous situation with too
> many share group members or share groups? Are they any ways we try to
> handle this or mitigate risks with respect to memory usage and client
> connections (wrt to rebalances for example)
> 
> Thanks,
> 
> Justine
> 
> On Fri, Mar 8, 2024 at 12:51 AM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
> 
>> Hi Manikumar,
>> Thanks for your queries.
>> 
>> 1) Delivery count is added to the ConsumerRecord class so that a consumer
>> can
>> tell how often a record has been processed. I imagine that some
>> applications might
>> want to take different actions based on whether a record has previously
>> failed. This
>> enables richer error handling for bad records. In the future, I plan
>> another KIP to
>> enhance error handling.
>> 
>> 2) It is only possible to delete a share group which is empty. As a
>> result, all
>> well-behaved consumers will have closed their share sessions. After a
>> short while,
>> the share-partition leaders will discard the share-partition information
>> from memory.
>> 
>> In the presence of badly behaved consumers, a consumer would have to
>> pretend to
>> be a member of a share group. There are several cases:
>> 
>> a) If the share-partition leader still has in-memory state for the deleted
>> share-group, it will
>> continue to fetch records but it will be fenced by the share coordinator
>> when it attempts to
>> write its persistent state.
>> 
>> b) If the share-partition leader does not have in-memory state, it will
>> attempt to read it
>> from the share coordinator and this will fail.
>> 
>> 3) I will add metrics for the share coordinator today. This was an
>> omission. Thanks for catching it.
>> 
>> Thanks,
>> Andrew
>> 
>>> On 6 Mar 2024, at 17:53, Manikumar  wrote:
>>> 
>>> Hi Andrew,
>>> 
>>> Thanks for the updated KIP. Few queries below:
>>> 
>>> 1. What is the use-case of deliveryCount in ShareFetchResponse?
>>> 2. During delete share groups, Do we need to clean any in-memory state
>> from
>>> share-partition leaders?
>>> 3. Any metrics for the share-coordinator?
>>> 
>>> Thanks
>>> Manikumar
>>> 
>>> On Wed, Feb 

Re: [VOTE] 3.6.2 RC1

[Chia-Ping Tsai]

hi Manikumar,

There is a bug fix which needs to be backport to 3.6 (
https://issues.apache.org/jira/browse/KAFKA-16341)

It fixes the incorrect offset of max timestamp in non-compress path. The
other paths are already fixed by
https://issues.apache.org/jira/browse/KAFKA-16342.

Personally, we should backport both fixes for all paths, and we can
complete the backport today.

Sorry for bring this news to RC1.

Best,
Chia-Ping


Manikumar  於 2024年3月21日 週四 下午6:11寫道：

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 3.6.2.
>
> This is a bugfix release with several fixes, including dependency
> version bumps for CVEs.
>
> Release notes for the 3.6.2 release:
> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, March 26th
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/
>
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/
>
> * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> https://github.com/apache/kafka/releases/tag/3.6.2-rc1
>
> * Documentation:
> https://kafka.apache.org/36/documentation.html
>
> * Protocol:
> https://kafka.apache.org/36/protocol.html
>
> * Successful Jenkins builds for the 3.6 branch:
> Unit/integration tests:
>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
> (with few flaky failures)
> System tests: I will update system test results
>
> Thanks,
> Manikumar
>

[DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Krish Vora]

Hi everyone,

I would like to start the discussion on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka
 .

This KIP aims to introduce JVM based Docker Official Image (DOI) for Apache
Kafka.

Regards,
Krish.

[jira] [Resolved] (KAFKA-16318) Add javadoc to KafkaMetric

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16318.
---
Fix Version/s: 3.8.0
   Resolution: Fixed

> Add javadoc to KafkaMetric
> --
>
> Key: KAFKA-16318
> URL: https://issues.apache.org/jira/browse/KAFKA-16318
> Project: Kafka
>  Issue Type: Bug
>  Components: docs
>Reporter: Mickael Maison
>Assignee: Johnny Hsu
>Priority: Major
> Fix For: 3.8.0
>
>
> KafkaMetric is part of the public API but it's missing javadoc describing the 
> class and several of its methods.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] 3.6.2 RC1

[Manikumar]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.6.2.

This is a bugfix release with several fixes, including dependency
version bumps for CVEs.

Release notes for the 3.6.2 release:
https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, March 26th

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~manikumar/kafka-3.6.2-rc1/


* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
https://github.com/apache/kafka/releases/tag/3.6.2-rc1

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
Unit/integration tests:
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.6/159/tests
(with few flaky failures)
System tests: I will update system test results

Thanks,
Manikumar

Re: [DISCUSS] Different retention semantics for active segment rotation

[Jorge Esteban Quilcate Otoya]

Sure! good to know that is tracked.

Thanks, Luke!

On Thu, 21 Mar 2024 at 07:52, Luke Chen  wrote:

> Hi Jorge,
>
> You should check the JIRA:
> https://issues.apache.org/jira/browse/KAFKA-16385
> where we had some discussion.
> Welcome to provide your thoughts there.
>
> Thanks.
> Luke
>
> On Thu, Mar 21, 2024 at 3:33 PM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Hi dev community,
> >
> > I'd like to share some findings on how rotation of active segment differ
> > depending on whether topic retention is time- or size-based.
> >
> > I was (wrongly) under the assumption that active segments are only
> rotated
> > when segment configs (segment.bytes (1GiB) or segment.ms (7d)) or global
> > log configs (log.roll.ms) force it  -- regardless of the retention
> > configuration.
> > This seems to be different depending on how retention is defined:
> >
> > - If a topic has a retention based on time[1], the condition to rotate
> the
> > active segment is based on the latest timestamp. If the difference with
> > current time is largest than retention time, then segment (including
> > active) should be deleted. Active segment is rotated, and in next round
> is
> > deleted.
> >
> > - If a topic has retention based on size[2] though, the condition not
> only
> > depends on the size of the segment itself but first on the total log
> size,
> > forcing to always have at least a single (active) segment: first
> difference
> > between total log size and retention is calculated, let's say a single
> > segment of 5MB and retention is 1MB; then total difference is 4MB, then
> the
> > condition to delete validates if the difference of the current segment
> and
> > the total difference is higher than zero, then delete. As the segment
> size
> > will always be higher than the total difference when there is a single
> > segment, then there will always be at least 1 segment. In this case the
> > only case where active segment is rotated it is when a new message
> arrives.
> >
> > Added steps to reproduce[3].
> >
> > Maybe I missing something obvious, but this seems inconsistent to me.
> > Either both retention configs should rotate active segments, or none of
> > them should and active segment should be only governed by segment
> bytes|ms
> > configs or log.roll config.
> >
> > I believe it's a useful feature to "force" active segment rotation
> without
> > changing segment of global log rotation given that features like
> Compaction
> > and Tiered Storage can benefit from this; but would like to clarify this
> > behavior and make it consistent for both retention options, and/or call
> it
> > out explicitly in the documentation.
> >
> > Looking forward to your feedback!
> >
> > Jorge.
> >
> > [1]:
> >
> >
> https://github.com/apache/kafka/blob/55a6d30ccbe971f4d2e99aeb3b1a773ffe5792a2/core/src/main/scala/kafka/log/UnifiedLog.scala#L1566
> > [2]:
> >
> >
> https://github.com/apache/kafka/blob/55a6d30ccbe971f4d2e99aeb3b1a773ffe5792a2/core/src/main/scala/kafka/log/UnifiedLog.scala#L1575-L1583
> >
> > [3]: https://gist.github.com/jeqo/d32cf07493ee61f3da58ac5e77b192b2
> >
>

[jira] [Created] (KAFKA-16399) Add JBOD support in tiered storage

[Luke Chen (Jira)]

Luke Chen created KAFKA-16399:
-

 Summary: Add JBOD support in tiered storage
 Key: KAFKA-16399
 URL: https://issues.apache.org/jira/browse/KAFKA-16399
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen
Assignee: Luke Chen


Add JBOD support in tiered storage



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Different retention semantics for active segment rotation

[Luke Chen]

Hi Jorge,

You should check the JIRA: https://issues.apache.org/jira/browse/KAFKA-16385
where we had some discussion.
Welcome to provide your thoughts there.

Thanks.
Luke

On Thu, Mar 21, 2024 at 3:33 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi dev community,
>
> I'd like to share some findings on how rotation of active segment differ
> depending on whether topic retention is time- or size-based.
>
> I was (wrongly) under the assumption that active segments are only rotated
> when segment configs (segment.bytes (1GiB) or segment.ms (7d)) or global
> log configs (log.roll.ms) force it  -- regardless of the retention
> configuration.
> This seems to be different depending on how retention is defined:
>
> - If a topic has a retention based on time[1], the condition to rotate the
> active segment is based on the latest timestamp. If the difference with
> current time is largest than retention time, then segment (including
> active) should be deleted. Active segment is rotated, and in next round is
> deleted.
>
> - If a topic has retention based on size[2] though, the condition not only
> depends on the size of the segment itself but first on the total log size,
> forcing to always have at least a single (active) segment: first difference
> between total log size and retention is calculated, let's say a single
> segment of 5MB and retention is 1MB; then total difference is 4MB, then the
> condition to delete validates if the difference of the current segment and
> the total difference is higher than zero, then delete. As the segment size
> will always be higher than the total difference when there is a single
> segment, then there will always be at least 1 segment. In this case the
> only case where active segment is rotated it is when a new message arrives.
>
> Added steps to reproduce[3].
>
> Maybe I missing something obvious, but this seems inconsistent to me.
> Either both retention configs should rotate active segments, or none of
> them should and active segment should be only governed by segment bytes|ms
> configs or log.roll config.
>
> I believe it's a useful feature to "force" active segment rotation without
> changing segment of global log rotation given that features like Compaction
> and Tiered Storage can benefit from this; but would like to clarify this
> behavior and make it consistent for both retention options, and/or call it
> out explicitly in the documentation.
>
> Looking forward to your feedback!
>
> Jorge.
>
> [1]:
>
> https://github.com/apache/kafka/blob/55a6d30ccbe971f4d2e99aeb3b1a773ffe5792a2/core/src/main/scala/kafka/log/UnifiedLog.scala#L1566
> [2]:
>
> https://github.com/apache/kafka/blob/55a6d30ccbe971f4d2e99aeb3b1a773ffe5792a2/core/src/main/scala/kafka/log/UnifiedLog.scala#L1575-L1583
>
> [3]: https://gist.github.com/jeqo/d32cf07493ee61f3da58ac5e77b192b2
>

[DISCUSS] Different retention semantics for active segment rotation

[Jorge Esteban Quilcate Otoya]

Hi dev community,

I'd like to share some findings on how rotation of active segment differ
depending on whether topic retention is time- or size-based.

I was (wrongly) under the assumption that active segments are only rotated
when segment configs (segment.bytes (1GiB) or segment.ms (7d)) or global
log configs (log.roll.ms) force it  -- regardless of the retention
configuration.
This seems to be different depending on how retention is defined:

- If a topic has a retention based on time[1], the condition to rotate the
active segment is based on the latest timestamp. If the difference with
current time is largest than retention time, then segment (including
active) should be deleted. Active segment is rotated, and in next round is
deleted.

- If a topic has retention based on size[2] though, the condition not only
depends on the size of the segment itself but first on the total log size,
forcing to always have at least a single (active) segment: first difference
between total log size and retention is calculated, let's say a single
segment of 5MB and retention is 1MB; then total difference is 4MB, then the
condition to delete validates if the difference of the current segment and
the total difference is higher than zero, then delete. As the segment size
will always be higher than the total difference when there is a single
segment, then there will always be at least 1 segment. In this case the
only case where active segment is rotated it is when a new message arrives.

Added steps to reproduce[3].

Maybe I missing something obvious, but this seems inconsistent to me.
Either both retention configs should rotate active segments, or none of
them should and active segment should be only governed by segment bytes|ms
configs or log.roll config.

I believe it's a useful feature to "force" active segment rotation without
changing segment of global log rotation given that features like Compaction
and Tiered Storage can benefit from this; but would like to clarify this
behavior and make it consistent for both retention options, and/or call it
out explicitly in the documentation.

Looking forward to your feedback!

Jorge.

[1]:
https://github.com/apache/kafka/blob/55a6d30ccbe971f4d2e99aeb3b1a773ffe5792a2/core/src/main/scala/kafka/log/UnifiedLog.scala#L1566
[2]:
https://github.com/apache/kafka/blob/55a6d30ccbe971f4d2e99aeb3b1a773ffe5792a2/core/src/main/scala/kafka/log/UnifiedLog.scala#L1575-L1583

[3]: https://gist.github.com/jeqo/d32cf07493ee61f3da58ac5e77b192b2

[jira] [Created] (KAFKA-16398) mirror-maker2 running into OOM while filtering high number of messages

[Srivignesh (Jira)]

Srivignesh created KAFKA-16398:
--

 Summary: mirror-maker2 running into OOM while filtering high 
number of messages
 Key: KAFKA-16398
 URL: https://issues.apache.org/jira/browse/KAFKA-16398
 Project: Kafka
  Issue Type: Bug
  Components: connect, mirrormaker
Affects Versions: 3.6.1
Reporter: Srivignesh


Based on custom predicate, our application is filtering messages during 
mirroring. 

When the HasHeader:test method of the predicate returns false (when it has to 
drop messages from mirroring), it encounters below exceptions. 

However when it returns true (the messages are forwarded for mirroring), it 
works fine without OOM. 

Note: This issue doesn't occur in version 2.8.0.

Exception stacktraces:
{code:java}
line java.lang.OutOfMemoryError: Java heap space
line     at org.apache.kafka.common.utils.Utils.toArray(Utils.java:289)
line     at org.apache.kafka.common.utils.Utils.toArray(Utils.java:252)
line     at org.apache.kafka.common.utils.Utils.toNullableArray(Utils.java:270)
line     at 
org.apache.kafka.common.serialization.Deserializer.deserialize(Deserializer.java:73)
line     at 
org.apache.kafka.clients.consumer.internals.CompletedFetch.parseRecord(CompletedFetch.java:300)
line     at 
org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:263)
line     at 
org.apache.kafka.clients.consumer.internals.AbstractFetch.fetchRecords(AbstractFetch.java:340)
line     at 
org.apache.kafka.clients.consumer.internals.AbstractFetch.collectFetch(AbstractFetch.java:306)
line     at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1262)
line     at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1186)
line     at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
line     at 
org.apache.kafka.connect.mirror.MirrorSourceTask.poll(MirrorSourceTask.java:153)
line     at 
org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:469)
line     at 
org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:357)
line     at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)
line     at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
line     at 
org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:77)
line     at 
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)
line     at 
org.apache.kafka.connect.runtime.isolation.Plugins$$Lambda$841/0x7f55cc4c3d78.run(Unknown
 Source)
line     at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
line     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
line     at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
line     at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
line     at java.base/java.lang.Thread.run(Thread.java:840) {code}
{code:java}
line java.lang.OutOfMemoryError: Java heap space line     at 
java.base/java.nio.HeapByteBuffer.(HeapByteBuffer.java:64)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1025: Optionally URL-encode clientID and clientSecret in authorization header

[Doğuşcan Namal]

Hi Nelson, thanks for the KIP.

>From the RFC:
```
The authorization server MUST require the use of TLS as described in
   Section 1.6 when sending requests using password authentication.
```

I believe we already have an enforcement for OAuth to be enabled only in
SSLChannel but would be good to double check. Sending secrets over
plaintext is a security bad practice :)

+1 (non-binding) from me.

On Tue, 19 Mar 2024 at 16:00, Nelson B.  wrote:

> Hi all,
>
> I would like to start a vote on KIP-1025
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1025%3A+Optionally+URL-encode+clientID+and+clientSecret+in+authorization+header
> >,
> which would optionally URL-encode clientID and clientSecret in the
> authorization header.
>
> I feel like all possible issues have been addressed in the discussion
> thread.
>
> Thanks,
>