Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Ismael Juma]

Hi Krishna,

Looks like you sent the response to me and not the mailing list,
please include the mailing list in the replies. Comments below.

On Mon, Sep 25, 2023 at 11:45 AM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Ismael,
> Thanks for the questions.
>
>1. We intend to support only the latest Java supported by Apache
>Kafka(As per this documentation
> Apache Kafka currently
>supports Java 8, Java 11, and Java 17) which currently is Java 17. If
>Apache Kafka supports Java 21 in the future, we will align with it.
>
> We are already building and testing with Java 21 (
https://github.com/apache/kafka/pull/14451 updates `README.md` to indicate
that). By 3.7.0 (the next release), we'll have Java 21 as one of the
officially supported versions. I think we should start with that version
for both docker image KIPs.

>
>1. For users seeking a Docker image with an alternative Java version,
>they will have the flexibility to build their own Docker image utilising
>the Dockerfiles we provide. In our documentation, we will provide clear
>guidance on the designated base images for various Java versions.
>
> This sounds good to me. We should include these details as part of the KIP
and also the documentation for the docker images. More specifically, we
should state that we will update the Java major version as part of minor
Apache Kafka releases. The implication is that users who include broker
plugins alongside the broker should use custom images to ensure their
custom code is not broken by Java upgrades.

>
>1. Apache Kafka only requires JRE, not JDK, for operation. Utilizing a
>base image with only JRE, rather than JDK, is a logical choice as it
>significantly reduces the size of the docker image.
>Upon further investigation, I discovered the eclipse-temurin
>
> 
>docker image, which is notably smaller than Redhat’s ubi8 docker image (263
>MB vs 375 MB). Additionally, the fact that Apache Flink relies on
>eclipse-temurin base images
>
> 
>further increases our confidence in their dependability(Will make this
>change in the KIP).
>
> Yes, eclipse-temurin looks like a good choice to me. Nice size reduction!

>
>1. I'll conduct comparisons between our docker image and existing
>ones, and incorporate the findings into the KIP. I'll keep you posted on
>the same.
>
> Excellent, thanks!

Ismael


> On Wed, Sep 20, 2023 at 11:26 PM Ismael Juma  wrote:
>
>> Hi Krishna,
>>
>> Thanks for the KIP. A few quick questions:
>>
>> 1. Since this will only be available for Kafka 3.7 in the best case, I
>> suggest we go with Java 21 instead of Java 17. Also, we should be clear
>> about Java version expectations. Are we allowed to change the Java version
>> whenever we want? Or can it only be done in major versions? I prefer the
>> former option by default, but a subset of users may have their own plugins
>> running within the broker. Such users may need to have control over the
>> Java version they use - how would they achieve tha
>> 2. We state that ubi-minimal would be the base image - it would be good to
>> provide some details on why this was the choice.
>> 3. How does this docker image compare to the existing popular images?
>> Perhaps we can compare with the 3 most popular images for Apache Kafka
>> these days.
>>
>> Ismael
>>
>> On Fri, Sep 8, 2023 at 12:57 AM Krishna Agarwal <
>> krishna0608agar...@gmail.com> wrote:
>>
>> > Hi,
>> > Apache Kafka does not have an official docker image currently.
>> > I want to submit a KIP to publish a docker image for Apache Kafka.
>> >
>> > KIP-975: Docker Image for Apache Kafka
>> > <
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
>> > >
>> >
>> > Regards,
>> > Krishna
>> >
>>
>

[GitHub] [kafka-site] tombentley commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

tombentley commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-173498

   @mimaison are you happy with the latest changes which don't touch the 
`.htaccess`? While I think the duplication is not _ideal_, this change would 
make it easier docs contributors. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] tombentley commented on a diff in pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

tombentley commented on code in PR #410:
URL: https://github.com/apache/kafka-site/pull/410#discussion_r1336588112


##
start-preview.sh:
##
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+set -euxo pipefail
+
+docker build -t kafka-site-preview .
+
+docker run -it --rm --name mypreview -p 8080:80 -v 
"$PWD":/usr/local/apache2/htdocs/ kafka-site-preview

Review Comment:
   I needed to add :z to the volume mount for this to work with podman (as 
opposed to docker) on Fedora)
   
   ```suggestion
   docker run -it --rm --name mypreview -p 8080:80 -v 
"$PWD":/usr/local/apache2/htdocs/:z kafka-site-preview
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15505) MM2 consumer group subscription different on target cluster

[Srinivas Boga (Jira)]

Srinivas Boga created KAFKA-15505:
-

 Summary: MM2 consumer group subscription different on target 
cluster
 Key: KAFKA-15505
 URL: https://issues.apache.org/jira/browse/KAFKA-15505
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Reporter: Srinivas Boga


I am running mirrormaker2 in distributed mode with 3 nodes and 10 tasks each , 
kafka version: 3.5.1

On the source cluster , i see the consumer group has subscriptions of topic-A 
and topic-B, but on the target cluster i am seeing only the subscription of 
topic-A

restarting did not help, Also the the consumed offsets of the topic-A are 
different on the target cluster, i see the lag of zero for all partitions on 
source cluster, but the lag keeps building up on the target cluster

 

Any help on this would be appreciated

 

Thanks,

-srini

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Jose,

Sounds good to me.
Let's have further discussion in JIRA/PR, and target to v3.6.1/v3.7.0.

Thanks.
Luke

On Tue, Sep 26, 2023 at 1:35 AM José Armando García Sancio
 wrote:

> On Sat, Sep 23, 2023 at 3:08 AM Luke Chen  wrote:
> >
> > Hi Satish,
> >
> > I found the current KRaft implementation will have "split brain" issue
> when
> > network partition happens, which will cause inconsistent metadata
> returned
> > from the controller.
> > Filed KAFKA-15489 
> for
> > this issue, and PR  is ready
> > for review.
> >
> > Even though this is not a regression issue (this has already existed
> since
> > the 1st release of KRaft feature), I think this is an important issue
> since
> > KRaft is announced production ready.
> > Not sure what other people's thoughts are.
>
> Thanks for the report and PR Luke. This looks related to this issue:
> https://issues.apache.org/jira/browse/KAFKA-13621
>
> Do you agree? We can move our conversation to those issues but I also
> agree that I don't think this issue should be a release blocker.
>
> Thanks!
> -José
>

[GitHub] [kafka-site] jolshan merged pull request #553: MINOR: Update Justine to add PMC member

[via GitHub]

jolshan merged PR #553:
URL: https://github.com/apache/kafka-site/pull/553


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] jolshan opened a new pull request, #553: MINOR: Update Justine to add PMC member

[via GitHub]

jolshan opened a new pull request, #553:
URL: https://github.com/apache/kafka-site/pull/553

   Updating the site to reflect the change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Calvin Liu]

Hi Jun
Thanks for the comments.

40. If we change to None, it is not guaranteed for no data loss. For users
who are not able to validate the data with external resources, manual
intervention does not give a better result but a loss of availability. So
practically speaking, the Balance mode would be a better default value.

41. No, it represents how we want to do the unclean leader election. If it
is false, the unclean leader election will be the old random way.
Otherwise, the unclean recovery will be used.

42. Good catch. Updated.

43. Only the first 20 topics will be served. Others will be returned with
InvalidRequestError

44. The order matters. The desired leader entries match with the topic
partition list by the index.

45. Thanks! Updated.

46. Good advice! Updated.

47.1, updated the comment. Basically it will elect the replica in the
desiredLeader field to be the leader

47.2 We can let the admin client do the conversion. Using the desiredLeader
field in the json format seems easier for users.

48. Once the MV version is downgraded, all the ELR related fields will be
removed on the next partition change. The controller will also ignore the
ELR fields. Updated the KIP.

49. Yes, it would be deprecated/removed.


On Mon, Sep 25, 2023 at 3:49 PM Jun Rao  wrote:

> Hi, Calvin,
>
> Thanks for the updated KIP. Made another pass. A few more comments below.
>
> 40. unclean.leader.election.enable.false ->
> unclean.recovery.strategy.Balanced: The Balanced mode could still lead to
> data loss. So, I am wondering if unclean.leader.election.enable.false
> should map to None?
>
> 41. unclean.recovery.manager.enabled: I am not sure why we introduce this
> additional config. Is it the same as unclean.recovery.strategy=None?
>
> 42. DescribeTopicResponse.TopicAuthorizedOperations: Should this be at the
> topic level?
>
> 43. "Limit: 20 topics max per request": Could we describe what happens if
> the request includes more than 20 topics?
>
> 44. ElectLeadersRequest.DesiredLeaders: Could we describe whether the
> ordering matters?
>
> 45. GetReplicaLogInfo.TopicPartitions: "about": "The topic partitions to
> elect leaders.": The description in "about" is incorrect.
>
> 46. GetReplicaLogInfoResponse: Should we nest partitions under topicId to
> be consistent with other types of responses?
>
> 47. kafka-leader-election.sh:
> 47.1 Could we explain DESIGNATION?
> 47.2 desiredLeader: Should it be a list to match the field in
> ElectLeadersRequest?
>
> 48. We could add a section on downgrade?
>
> 49. LastKnownLeader: This seems only needed in the first phase of
> delivering ELR. Will it be removed when the complete KIP is delivered?
>
> Thanks,
>
> Jun
>
> On Tue, Sep 19, 2023 at 1:30 PM Colin McCabe  wrote:
>
> > Hi Calvin,
> >
> > Thanks for the explanations. I like the idea of using none, balanced,
> > aggressive. We also had an offline discussion about why it is good to
> use a
> > new config key (basically, so that we can deprecate the old one which had
> > only false/true values in 4.0) With these changes, I am +1.
> >
> > best,
> > Colin
> >
> > On Mon, Sep 18, 2023, at 15:54, Calvin Liu wrote:
> > > Hi Colin,
> > > Also, can we deprecate unclean.leader.election.enable in 4.0? Before
> > that,
> > > we can have both the config unclean.recovery.strategy and
> > > unclean.leader.election.enable
> > > and using the unclean.recovery.Enabled to determine which config to use
> > > during the unclean leader election.
> > >
> > > On Mon, Sep 18, 2023 at 3:51 PM Calvin Liu  wrote:
> > >
> > >> Hi Colin,
> > >> For the unclean.recovery.strategy config name, how about we use the
> > >> following
> > >> None. It basically means no unclean recovery will be performed.
> > >> Aggressive. It means availability goes first. Whenever the partition
> > can't
> > >> elect a durable replica, the controller will try the unclean recovery.
> > >> Balanced. It is the balance point of the availability
> first(Aggressive)
> > >> and least availability(None). The controller performs unclean recovery
> > when
> > >> both ISR and ELR are empty.
> > >>
> > >>
> > >> On Fri, Sep 15, 2023 at 11:42 AM Calvin Liu 
> wrote:
> > >>
> > >>> Hi Colin,
> > >>>
> > >>> > So, the proposal is that if someone sets
> > "unclean.leader.election.enable
> > >>> = true"...
> > >>>
> > >>>
> > >>> The idea is to use one of the unclean.leader.election.enable and
> > >>> unclean.recovery.strategy based on the unclean.recovery.Enabled. A
> > possible
> > >>> version can be
> > >>>
> > >>> If unclean.recovery.Enabled:
> > >>>
> > >>> {
> > >>>
> > >>> Check unclean.recovery.strategy. If set, use it. Otherwise, check
> > >>> unclean.leader.election.enable and translate it to
> > >>> unclean.recovery.strategy.
> > >>>
> > >>> } else {
> > >>>
> > >>> Use unclean.leader.election.enable
> > >>>
> > >>> }
> > >>>
> > >>>
> > >>> —
> > >>>
> > >>> >The configuration key should be "unclean.recovery.manager.enabled",
> > >>> right?
> > >>>
> > >>>
> > >>> I think we 

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Colin McCabe]

On Mon, Sep 25, 2023, at 15:48, Jun Rao wrote:
> Hi, Calvin,
>
> Thanks for the updated KIP. Made another pass. A few more comments below.
>
> 40. unclean.leader.election.enable.false ->
> unclean.recovery.strategy.Balanced: The Balanced mode could still lead to
> data loss. So, I am wondering if unclean.leader.election.enable.false
> should map to None?
>

Hi Jun,

Thanks for the re-review.

The issue with mapping unclean.leader.election.enable=false to 
unclean.recovery.strategy=None is that it will be a large behavior change for 
existing users. For example, in the scenario where 3 or more brokers lose power 
at the same time, if unclean.recovery.strategy=None, the cluster will never 
recover without manual intervention. I don't think this really is viable for us 
since users who upgrade will see it as just "broken." In contrast, 
unclean.recovery.strategy=Balanced implement the current behavior (basically).

We all agree that it's awkward to keep around unclean.leader.election.enable, 
which is why this KIP deprecates it. We should be able to remove it in 4.0 and 
just have unclean.recovery.strategy, which will make things clearer.

Agreed on your other comments -- thanks again for reviewing.

best,
Colin


> 41. unclean.recovery.manager.enabled: I am not sure why we introduce this
> additional config. Is it the same as unclean.recovery.strategy=None?
>
> 42. DescribeTopicResponse.TopicAuthorizedOperations: Should this be at the
> topic level?
>
> 43. "Limit: 20 topics max per request": Could we describe what happens if
> the request includes more than 20 topics?
>
> 44. ElectLeadersRequest.DesiredLeaders: Could we describe whether the
> ordering matters?
>
> 45. GetReplicaLogInfo.TopicPartitions: "about": "The topic partitions to
> elect leaders.": The description in "about" is incorrect.
>
> 46. GetReplicaLogInfoResponse: Should we nest partitions under topicId to
> be consistent with other types of responses?
>
> 47. kafka-leader-election.sh:
> 47.1 Could we explain DESIGNATION?
> 47.2 desiredLeader: Should it be a list to match the field in
> ElectLeadersRequest?
>
> 48. We could add a section on downgrade?
>
> 49. LastKnownLeader: This seems only needed in the first phase of
> delivering ELR. Will it be removed when the complete KIP is delivered?
>
> Thanks,
>
> Jun
>
> On Tue, Sep 19, 2023 at 1:30 PM Colin McCabe  wrote:
>
>> Hi Calvin,
>>
>> Thanks for the explanations. I like the idea of using none, balanced,
>> aggressive. We also had an offline discussion about why it is good to use a
>> new config key (basically, so that we can deprecate the old one which had
>> only false/true values in 4.0) With these changes, I am +1.
>>
>> best,
>> Colin
>>
>> On Mon, Sep 18, 2023, at 15:54, Calvin Liu wrote:
>> > Hi Colin,
>> > Also, can we deprecate unclean.leader.election.enable in 4.0? Before
>> that,
>> > we can have both the config unclean.recovery.strategy and
>> > unclean.leader.election.enable
>> > and using the unclean.recovery.Enabled to determine which config to use
>> > during the unclean leader election.
>> >
>> > On Mon, Sep 18, 2023 at 3:51 PM Calvin Liu  wrote:
>> >
>> >> Hi Colin,
>> >> For the unclean.recovery.strategy config name, how about we use the
>> >> following
>> >> None. It basically means no unclean recovery will be performed.
>> >> Aggressive. It means availability goes first. Whenever the partition
>> can't
>> >> elect a durable replica, the controller will try the unclean recovery.
>> >> Balanced. It is the balance point of the availability first(Aggressive)
>> >> and least availability(None). The controller performs unclean recovery
>> when
>> >> both ISR and ELR are empty.
>> >>
>> >>
>> >> On Fri, Sep 15, 2023 at 11:42 AM Calvin Liu  wrote:
>> >>
>> >>> Hi Colin,
>> >>>
>> >>> > So, the proposal is that if someone sets
>> "unclean.leader.election.enable
>> >>> = true"...
>> >>>
>> >>>
>> >>> The idea is to use one of the unclean.leader.election.enable and
>> >>> unclean.recovery.strategy based on the unclean.recovery.Enabled. A
>> possible
>> >>> version can be
>> >>>
>> >>> If unclean.recovery.Enabled:
>> >>>
>> >>> {
>> >>>
>> >>> Check unclean.recovery.strategy. If set, use it. Otherwise, check
>> >>> unclean.leader.election.enable and translate it to
>> >>> unclean.recovery.strategy.
>> >>>
>> >>> } else {
>> >>>
>> >>> Use unclean.leader.election.enable
>> >>>
>> >>> }
>> >>>
>> >>>
>> >>> —
>> >>>
>> >>> >The configuration key should be "unclean.recovery.manager.enabled",
>> >>> right?
>> >>>
>> >>>
>> >>> I think we have two ways of choosing a leader uncleanly, unclean leader
>> >>> election and unclean recovery(log inspection) and we try to switch
>> between
>> >>> them.
>> >>>
>> >>> Do you mean we want to develop two ways of performing the unclean
>> >>> recovery and one of them is using “unclean recovery manager”? I guess
>> we
>> >>> haven’t discussed the second way.
>> >>>
>> >>>
>> >>> —---
>> >>>
>> >>> >How do these 4 levels of overrides 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #68

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Jun Rao]

Hi, Calvin,

Thanks for the updated KIP. Made another pass. A few more comments below.

40. unclean.leader.election.enable.false ->
unclean.recovery.strategy.Balanced: The Balanced mode could still lead to
data loss. So, I am wondering if unclean.leader.election.enable.false
should map to None?

41. unclean.recovery.manager.enabled: I am not sure why we introduce this
additional config. Is it the same as unclean.recovery.strategy=None?

42. DescribeTopicResponse.TopicAuthorizedOperations: Should this be at the
topic level?

43. "Limit: 20 topics max per request": Could we describe what happens if
the request includes more than 20 topics?

44. ElectLeadersRequest.DesiredLeaders: Could we describe whether the
ordering matters?

45. GetReplicaLogInfo.TopicPartitions: "about": "The topic partitions to
elect leaders.": The description in "about" is incorrect.

46. GetReplicaLogInfoResponse: Should we nest partitions under topicId to
be consistent with other types of responses?

47. kafka-leader-election.sh:
47.1 Could we explain DESIGNATION?
47.2 desiredLeader: Should it be a list to match the field in
ElectLeadersRequest?

48. We could add a section on downgrade?

49. LastKnownLeader: This seems only needed in the first phase of
delivering ELR. Will it be removed when the complete KIP is delivered?

Thanks,

Jun

On Tue, Sep 19, 2023 at 1:30 PM Colin McCabe  wrote:

> Hi Calvin,
>
> Thanks for the explanations. I like the idea of using none, balanced,
> aggressive. We also had an offline discussion about why it is good to use a
> new config key (basically, so that we can deprecate the old one which had
> only false/true values in 4.0) With these changes, I am +1.
>
> best,
> Colin
>
> On Mon, Sep 18, 2023, at 15:54, Calvin Liu wrote:
> > Hi Colin,
> > Also, can we deprecate unclean.leader.election.enable in 4.0? Before
> that,
> > we can have both the config unclean.recovery.strategy and
> > unclean.leader.election.enable
> > and using the unclean.recovery.Enabled to determine which config to use
> > during the unclean leader election.
> >
> > On Mon, Sep 18, 2023 at 3:51 PM Calvin Liu  wrote:
> >
> >> Hi Colin,
> >> For the unclean.recovery.strategy config name, how about we use the
> >> following
> >> None. It basically means no unclean recovery will be performed.
> >> Aggressive. It means availability goes first. Whenever the partition
> can't
> >> elect a durable replica, the controller will try the unclean recovery.
> >> Balanced. It is the balance point of the availability first(Aggressive)
> >> and least availability(None). The controller performs unclean recovery
> when
> >> both ISR and ELR are empty.
> >>
> >>
> >> On Fri, Sep 15, 2023 at 11:42 AM Calvin Liu  wrote:
> >>
> >>> Hi Colin,
> >>>
> >>> > So, the proposal is that if someone sets
> "unclean.leader.election.enable
> >>> = true"...
> >>>
> >>>
> >>> The idea is to use one of the unclean.leader.election.enable and
> >>> unclean.recovery.strategy based on the unclean.recovery.Enabled. A
> possible
> >>> version can be
> >>>
> >>> If unclean.recovery.Enabled:
> >>>
> >>> {
> >>>
> >>> Check unclean.recovery.strategy. If set, use it. Otherwise, check
> >>> unclean.leader.election.enable and translate it to
> >>> unclean.recovery.strategy.
> >>>
> >>> } else {
> >>>
> >>> Use unclean.leader.election.enable
> >>>
> >>> }
> >>>
> >>>
> >>> —
> >>>
> >>> >The configuration key should be "unclean.recovery.manager.enabled",
> >>> right?
> >>>
> >>>
> >>> I think we have two ways of choosing a leader uncleanly, unclean leader
> >>> election and unclean recovery(log inspection) and we try to switch
> between
> >>> them.
> >>>
> >>> Do you mean we want to develop two ways of performing the unclean
> >>> recovery and one of them is using “unclean recovery manager”? I guess
> we
> >>> haven’t discussed the second way.
> >>>
> >>>
> >>> —---
> >>>
> >>> >How do these 4 levels of overrides interact with your new
> >>> configurations?
> >>>
> >>>
> >>> I do notice in the Kraft controller code, the method to check whether
> >>> perform unclean leader election is hard coded to false since
> >>> 2021(uncleanLeaderElectionEnabledForTopic). Isn’t it a good chance to
> >>> completely deprecate the unclean.leader.election.enable? We don’t even
> have
> >>> to worry about the config conversion.
> >>>
> >>> On the other hand, whatever the override is, as long as the controller
> >>> can have the final effective unclean.leader.election.enable, the topic
> >>> level config unclean.recovery.strategy, the cluster level config
> >>> unclean.recovery.Enabled, the controller can calculate the correct
> methods
> >>> to use right?
> >>>
> >>>
> >>> On Fri, Sep 15, 2023 at 10:02 AM Colin McCabe 
> wrote:
> >>>
>  On Thu, Sep 14, 2023, at 22:23, Calvin Liu wrote:
>  > Hi Colin
>  > 1. I think using the new config name is more clear.
>  >a. The unclean leader election is actually removed if unclean
>  > recovery is in use.
>  >

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2230

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 313958 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > StateQueryResultTest 
> Zero query results shouldn't error STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > StateQueryResultTest 
> Zero query results shouldn't error PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > StateQueryResultTest 
> Valid query results still works STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > StateQueryResultTest 
> Valid query results still works PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@18b74c9b, 
org.apache.kafka.test.MockInternalProcessorContext@3038ceee STARTED

Gradle Test Run :streams:test > Gradle Test Executor 82 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@18b74c9b, 
org.apache.kafka.test.MockInternalProcessorContext@3038ceee PASSED

Gradle Test Run :streams:test > Gradle Test Executor 82 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@4554569e, 

[jira] [Created] (KAFKA-15504) Upgrade snappy java to version 1.1.10.4

[Said BOUDJELDA (Jira)]

Said BOUDJELDA created KAFKA-15504:
--

 Summary: Upgrade snappy java to version 1.1.10.4
 Key: KAFKA-15504
 URL: https://issues.apache.org/jira/browse/KAFKA-15504
 Project: Kafka
  Issue Type: Improvement
Reporter: Said BOUDJELDA
Assignee: Said BOUDJELDA


The version 1.1.10.4 contains a fix of 
[CVE-2023-43642|https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv]
 as mentioned on the release notes of the library 
[https://github.com/xerial/snappy-java/releases/tag/v1.1.10.4]  Fixed 
SnappyInputStream so as not to allocate too large memory when decompressing 
data with an extremely large chunk size by

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #67

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Ron,

I think we can generalize the deconfigured directory scenario
in your last question to address this situation too.

When handling a broker registration request, the controller
can check if OfflineLogDirs=false and any UUIDs are missing
in OnlineLogDirs, compared with the previous registration,
and assign replicas in the missing dirs to Uuid.OfflineDir.

--
Igor

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2229

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 314834 lines...]
Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToSuspendShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToSuspendShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldOnlyKeepLastUpdateAction() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldOnlyKeepLastUpdateAction() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToRecycle() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToRecycle() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToUpdateInputPartitionsShouldRemoveTaskFromPendingUpdateActions()
 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldVerifyIfPendingTaskToRecycleExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToUpdateInputPartitions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToCloseDirtyShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToSuspend() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldVerifyIfPendingTaskToInitExist() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToCloseCleanShouldRemoveTaskFromPendingUpdateActions() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldDrainPendingTasksToCreate() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldDrainPendingTasksToCreate() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
onlyRemovePendingTaskToRecycleShouldRemoveTaskFromPendingUpdateActions() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseClean() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldAddAndRemovePendingTaskToCloseDirty() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldKeepAddedTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > TasksTest > 
shouldKeepAddedTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> Zero query results shouldn't error STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> Zero query results shouldn't error PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> Valid query results still works STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> Valid query results still works PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Krishna Agarwal]

Hi Viktor,
Thanks for the questions.

   1. While the docker image outlined in KIP-975 is designed for production
   environments, it is equally suitable for development and testing purposes.
   We will furnish the docker image, allowing users the flexibility to employ
   it according to their specific needs.
   2. The configs will be injected into the docker container through
   environment variables. These environment variables will have a prefix
   allowing for efficient parsing to extract the relevant properties.(Will add
   this implementation in the KIP as well once we converge on this.)
   3. Regarding this question, I'll conduct a test on my end after gaining
   a better understanding, and then provide you with a response.

Regards,
Krishna


On Tue, Sep 19, 2023 at 3:42 PM Viktor Somogyi-Vass
 wrote:

> Hi Ismael,
>
> I'm not trying to advocate against the docker image, I just pointed out
> that the current scoping of the KIP may be a bit too generic and thought
> that KIP-974 and KIP-975 were aiming for mostly the same thing and can be
> discussed under one umbrella. Apologies if this was rooted in a
> misunderstanding.
>
> Kirshna,
>
> I think we need to refine the KIP a bit more. I think there are some
> interfaces that we need to include in the KIP as Kafka has plugins in
> certain cases where users are expected to provide implementation and I
> think it's worth discussing this in the KIP as they're kind of interfaces
> for users. Here are my questions in order:
> 1. In what environments do you want the image to be used? As I understand
> it would replace the current testing image and serve as a basis for
> development, but would it aim at production use cases too (docker-compose,
> Kubernetes, etc.)?
> 2. How do you plan to forward configs to the broker? Do we expect a
> populated server.properties file placed in a certain location or should the
> docker image create this file based on some input (like env vars)?
> 3. Certain parts can be pluggable, like metric reporters or remote log
> implementations that were just introduced by KIP-405. These manifest in jar
> files that must be put on the classpath of Kafka while certain classnames
> have to be configured. How do you plan to implement this, how do we
> allow users to configure such things?
>
> Thanks,
> Viktor
>
>
>
>
> On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
>  wrote:
>
> > Hello,
> >
> > I think this would be a wonderful improvement to the ecosystem. While
> > Viktor is correct that most Docker pipelines eventually lead to a
> > kubernetes deployment, that should not stop us from creating an
> > Official Docker Image. Creating a Docker image would allow us to ensure a
> > level of quality and support for people who want to deploy Kafka as a
> > container on baremetal machines, it could allow us to create
> > a sandbox/developer environment for new contributors and developers to
> test
> > and have a single agreed upon environment that kafka works in for future
> > KIPs and would most likely spawn more contributions from people wanting
> to
> > optimize kafka for k8s.
> >
> >
> > I am 100% for this and will gladly help if approved.
> >
> > Kenneth
> >
> > On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:
> >
> > > Hi Viktor,
> > >
> > > I disagree. Docker is a very popular deployment tool and it's not only
> > used
> > > with Kubernetes.
> > >
> > > Ismael
> > >
> > > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > I think you should merge this KIP and KIP-974
> >  as there are
> > overlaps as
> > > > Federico pointed out on KIP-974
> > . I think you
> > should keep that one as it
> > > > has well defined goals (improve tests) while I feel this one is too
> > > > generic. Docker is usually just a tool for either testing or
> > Kubernetes,
> > > so
> > > > they have very well defined use-cases. In the case of Flink for
> > instance
> > > > the image is used for its kubernetes operator. The use case would
> > > determine
> > > > a lot of things and I think a generic image would likely not fit the
> > > needs
> > > > of all use-cases.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> > > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> > > > krishna0608agar...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Apache Kafka does not have an official docker image currently.
> > > > > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > > >
> > > > > KIP-975  >:
> > Docker Image for Apache Kafka
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > >
> > > > >
> > > > > Regards,
> > > > > Krishna
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-15503) CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15503:
--

 Summary: CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 
10.0.16, 11.0.16, 12.0.1
 Key: KAFKA-15503
 URL: https://issues.apache.org/jira/browse/KAFKA-15503
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.7.0, 2.6.1, 3.4.1, 3.6.0, 3.5.1
Reporter: Rafael Rios Saavedra
Assignee: Divij Vaidya
 Fix For: 2.8.0, 2.7.1, 2.6.2, 3.0.0


CVE-2023-40167 and CVE-2023-36479 vulnerabilities affects Jetty version 
{*}9.4.51{*}. For more information see 
[https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40167] 
[https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-364749] 

Upgrading to Jetty version *9.4.52, 10.0.16, 11.0.16, 12.0.1* should address 
this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[José Armando García Sancio]

On Sat, Sep 23, 2023 at 3:08 AM Luke Chen  wrote:
>
> Hi Satish,
>
> I found the current KRaft implementation will have "split brain" issue when
> network partition happens, which will cause inconsistent metadata returned
> from the controller.
> Filed KAFKA-15489  for
> this issue, and PR  is ready
> for review.
>
> Even though this is not a regression issue (this has already existed since
> the 1st release of KRaft feature), I think this is an important issue since
> KRaft is announced production ready.
> Not sure what other people's thoughts are.

Thanks for the report and PR Luke. This looks related to this issue:
https://issues.apache.org/jira/browse/KAFKA-13621

Do you agree? We can move our conversation to those issues but I also
agree that I don't think this issue should be a release blocker.

Thanks!
-José

[jira] [Resolved] (KAFKA-15117) SslTransportLayerTest.testValidEndpointIdentificationCN fails with Java 20 & 21

[Rajini Sivaram (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-15117.

  Reviewer: Rajini Sivaram
Resolution: Fixed

> SslTransportLayerTest.testValidEndpointIdentificationCN fails with Java 20 & 
> 21
> ---
>
> Key: KAFKA-15117
> URL: https://issues.apache.org/jira/browse/KAFKA-15117
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Purshotam Chauhan
>Priority: Major
> Fix For: 3.7.0
>
>
> All variations fail as seen below. These tests have been disabled when run 
> with Java 20 & 21 for now.
> {code:java}
> Gradle Test Run :clients:test > Gradle Test Executor 12 > 
> SslTransportLayerTest > testValidEndpointIdentificationCN(Args) > [1] 
> tlsProtocol=TLSv1.2, useInlinePem=false FAILED
>     org.opentest4j.AssertionFailedError: Channel 0 was not ready after 30 
> seconds ==> expected:  but was: 
>         at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>         at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>         at 
> app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
>         at 
> app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
>         at 
> app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
>         at 
> app//org.apache.kafka.common.network.NetworkTestUtils.waitForChannelReady(NetworkTestUtils.java:107)
>         at 
> app//org.apache.kafka.common.network.NetworkTestUtils.checkClientConnection(NetworkTestUtils.java:70)
>         at 
> app//org.apache.kafka.common.network.SslTransportLayerTest.verifySslConfigs(SslTransportLayerTest.java:1296)
>         at 
> app//org.apache.kafka.common.network.SslTransportLayerTest.testValidEndpointIdentificationCN(SslTransportLayerTest.java:202)
> org.apache.kafka.common.network.SslTransportLayerTest.testValidEndpointIdentificationCN(Args)[2]
>  failed, log available in 
> /home/ijuma/src/kafka/clients/build/reports/testOutput/org.apache.kafka.common.network.SslTransportLayerTest.testValidEndpointIdentificationCN(Args)[2].test.stdout
> Gradle Test Run :clients:test > Gradle Test Executor 12 > 
> SslTransportLayerTest > testValidEndpointIdentificationCN(Args) > [2] 
> tlsProtocol=TLSv1.2, useInlinePem=true FAILED
>     org.opentest4j.AssertionFailedError: Channel 0 was not ready after 30 
> seconds ==> expected:  but was: 
>         at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>         at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>         at 
> app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
>         at 
> app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
>         at 
> app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
>         at 
> app//org.apache.kafka.common.network.NetworkTestUtils.waitForChannelReady(NetworkTestUtils.java:107)
>         at 
> app//org.apache.kafka.common.network.NetworkTestUtils.checkClientConnection(NetworkTestUtils.java:70)
>         at 
> app//org.apache.kafka.common.network.SslTransportLayerTest.verifySslConfigs(SslTransportLayerTest.java:1296)
>         at 
> app//org.apache.kafka.common.network.SslTransportLayerTest.testValidEndpointIdentificationCN(SslTransportLayerTest.java:202)
> org.apache.kafka.common.network.SslTransportLayerTest.testValidEndpointIdentificationCN(Args)[3]
>  failed, log available in 
> /home/ijuma/src/kafka/clients/build/reports/testOutput/org.apache.kafka.common.network.SslTransportLayerTest.testValidEndpointIdentificationCN(Args)[3].test.stdout
> Gradle Test Run :clients:test > Gradle Test Executor 12 > 
> SslTransportLayerTest > testValidEndpointIdentificationCN(Args) > [3] 
> tlsProtocol=TLSv1.3, useInlinePem=false FAILED
>     org.opentest4j.AssertionFailedError: Channel 0 was not ready after 30 
> seconds ==> expected:  but was: 
>         at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>         at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>         at 
> app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
>         at 
> app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
>         at 
> app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
>         at 
> app//org.apache.kafka.common.network.NetworkTestUtils.waitForChannelReady(NetworkTestUtils.java:107)
>         at 
> app//org.apache.kafka.common.network.NetworkTestUtils.checkClientConnection(NetworkTestUtils.java:70)
>  

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

[Erik van Oosten]

Hi Viktor,

Good questions!

1. Auto-commits would only work with approach 1 in the KIP. Any async 
solution is incompatible with auto-commits. Do you think the text will 
improve when this is mentioned?


2. That is entirely correct. If you use async commits you can await 
completion by doing a single sync commit with an empty offsets Map (this 
will work as of Kafka 3.6.0).


Is there anything I can do to make the text clearer?

Kind regards,
    Erik.


Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass:

Hi Erik,

I'm still trying to wrap my head around the KIP, however I have a few
questions that weren't clear to me regarding offset commits:
1. Would auto-commits interfere with the behavior defined in your KIP or
would it work the same as manual commits?
2. As I see you don't separate offset commits by whether they're sync or
async. For sync commits timing isn't really a problem but how would you
change work in case of async offset commits? There can be a few caveats
there as you may not know whether a commit is finished or not until your
callback is called.

Thanks,
Viktor

On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten
 wrote:


Hi all,

I would like to start the discussion on KIP-983: Full speed async
processing during rebalance [1].

The idea is that we can prevent the drop in throughput during a
cooperative rebalance.

I am curious to your ideas and comments.

Kind regards,
  Erik.

[1]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance


--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com

[GitHub] [kafka-site] mimaison merged pull request #552: Add Kafka Connect book

[via GitHub]

mimaison merged PR #552:
URL: https://github.com/apache/kafka-site/pull/552


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15502) Handle large keystores in SslEngineValidator

[Manikumar (Jira)]

Manikumar created KAFKA-15502:
-

 Summary: Handle large keystores in SslEngineValidator
 Key: KAFKA-15502
 URL: https://issues.apache.org/jira/browse/KAFKA-15502
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Manikumar
Assignee: Manikumar


We have observed an issue where inter broker SSL listener is not coming up for 
large keystores (size >16K)

1. Currently validator code doesn't work well with large stores. Right now, 
WRAP returns if there is already data in the buffer. But if we need more data 
to be wrapped for UNWRAP to succeed, we end up looping forever.

2. Observed large TLSv3 post handshake messages are not getting read and 
causing UNWRAP loop forever. This is observed with JDK17+
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Luke for reporting the High CVE issue. Ismael also confirmed
that High CVEs are generally considered as release blockers.

Agree with all that we should consider this CVE as a release blocker.
We will include other CVE also with the next RC.

~Satish.


On Mon, 25 Sept 2023 at 03:04, Luke Chen  wrote:
>
> Hi Divij,
>
> About the system tests, it's me helping Satish working on it since our team
> has internal jenkins pipeline for it.
> Here's the result:
> https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC?usp=sharing
>
> I'm mainly focusing on the failed tests.
> For kraft_upgrade_test, I've fixed in this PR
> . After the fix, the rerun
> results look good.
> For quota_test, it's a known issue in our environment. After rerun, all
> passed.
>
> Thanks.
> Luke
>
>
>
> On Mon, Sep 25, 2023 at 4:09 PM Divij Vaidya 
> wrote:
>
> > Correction: posted the wrong JIRA in my previous email. Instead of
> > https://issues.apache.org/jira/browse/KAFKA-15001, please consider
> > this https://issues.apache.org/jira/browse/KAFKA-15487
> >
> > --
> > Divij Vaidya
> >
> > On Mon, Sep 25, 2023 at 10:04 AM Divij Vaidya 
> > wrote:
> > >
> > > Hi Satish
> > >
> > > 1. I agree with Luke. It's a "high" severity vulnerability and we
> > > should create another RC with the upgraded Snappy version. If we
> > > create another RC, we should also fix a different CVE resported in
> > > https://issues.apache.org/jira/browse/KAFKA-15001
> > >
> > > 2. I was hoping you could post the results of system tests before I
> > > vote on this. I am particularly interested in looking at
> > > producer/consumer performance results since we have quite a few
> > > changes in this release. What is the plan on the system tests?
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Mon, Sep 25, 2023 at 9:10 AM Luke Chen  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > Snappy-java published a new vulnerability
> > > > <
> > https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv
> > >
> > > > that will cause OOM error in the server.
> > > > Kafka is also impacted by this vulnerability since it's like
> > CVE-2023-34455
> > > > .
> > > > We'd better bump the snappy-java version to bypass this vulnerability.
> > > > PR  is created to run the
> > CI
> > > > build.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > > On Mon, Sep 25, 2023 at 2:38 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks to everyone who voted for this release.
> > > > >
> > > > > We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
> > > > > deadline. Please try RC1 and send your vote to this email thread.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > >
> > > > > On Sun, 24 Sept 2023 at 13:23, Justine Olshan
> > > > >  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I've done the following:
> > > > > > - Verified signature
> > > > > > - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> > > > > > - Run unit + integration tests
> > > > > > - Ran a shorter Trogdor transactional-produce-bench on a single
> > broker
> > > > > > cluster (KRaft and ZK) to verify transactional workloads worked
> > > > > reasonably
> > > > > >
> > > > > > Minor thing (we can discuss elsewhere and is non-blocking for the
> > > > > release)
> > > > > > but if ZK has been deprecated since 3.5 we should move up the
> > Kraft setup
> > > > > > in the quickstart guide  here
> > > > > > .
> > > > > >
> > > > > > +1 (binding) from me.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri <
> > fedeval...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Satish, I did the following to verify the release:
> > > > > > >
> > > > > > > - Verified signature and checksum
> > > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > > - Ran all unit and integration tests
> > > > > > > - Spot checked release notes and documentation
> > > > > > > - Ran a custom client using staging artifacts on a 3-nodes
> > cluster
> > > > > > > - Tested tiered storage with one of the available RSM
> > implementations
> > > > > > >
> > > > > > > +1 (non binding)
> > > > > > >
> > > > > > > Thanks
> > > > > > > Fede
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen 
> > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > I verified with:
> > > > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > > > 2. Making sure the checksum are correct
> > > > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > > >
> > > > > > > > I filed KAFKA-15491 <
> > > > > https://issues.apache.org/jira/browse/KAFKA-15491
> > > > > > > >for
> > > 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2228

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

[Viktor Somogyi-Vass]

Hi Erik,

I'm still trying to wrap my head around the KIP, however I have a few
questions that weren't clear to me regarding offset commits:
1. Would auto-commits interfere with the behavior defined in your KIP or
would it work the same as manual commits?
2. As I see you don't separate offset commits by whether they're sync or
async. For sync commits timing isn't really a problem but how would you
change work in case of async offset commits? There can be a few caveats
there as you may not know whether a commit is finished or not until your
callback is called.

Thanks,
Viktor

On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten
 wrote:

> Hi all,
>
> I would like to start the discussion on KIP-983: Full speed async
> processing during rebalance [1].
>
> The idea is that we can prevent the drop in throughput during a
> cooperative rebalance.
>
> I am curious to your ideas and comments.
>
> Kind regards,
>  Erik.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance
>
>
> --
> Erik van Oosten
> e.vanoos...@grons.nl
> https://day-to-day-stuff.blogspot.com
>
>

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

[Rajini Sivaram]

Congratulations, Justine!

Regards,

Rajini

On Mon, Sep 25, 2023 at 9:40 AM Lucas Brutschy
 wrote:

> Congrats, Justine!
>
> On Mon, Sep 25, 2023 at 9:20 AM Bruno Cadonna  wrote:
> >
> > Congrats, Justine! Well deserved!
> >
> > Best,
> > Bruno
> >
> > On 9/25/23 5:28 AM, ziming deng wrote:
> > > Congratulations Justine!
> > >
> > >
> > >> On Sep 25, 2023, at 00:01, Viktor Somogyi-Vass <
> viktor.somo...@cloudera.com.INVALID> wrote:
> > >>
> > >> Congrats Justine!
> > >>
> > >> On Sun, Sep 24, 2023, 17:45 Kirk True  wrote:
> > >>
> > >>> Congratulations Justine! Thanks for all your great work!
> > >>>
> >  On Sep 24, 2023, at 8:37 AM, John Roesler 
> wrote:
> > 
> >  Congratulations, Justine!
> >  -John
> > 
> >  On Sun, Sep 24, 2023, at 05:05, Mickael Maison wrote:
> > > Congratulations Justine!
> > >
> > > On Sun, Sep 24, 2023 at 5:04 AM Sophie Blee-Goldman
> > >  wrote:
> > >>
> > >> Congrats Justine!
> > >>
> > >> On Sat, Sep 23, 2023, 4:36 PM Tom Bentley 
> wrote:
> > >>
> > >>> Congratulations!
> > >>>
> > >>> On Sun, 24 Sept 2023 at 12:32, Satish Duggana <
> > >>> satish.dugg...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congratulations Justine!!
> > 
> >  On Sat, 23 Sept 2023 at 15:46, Bill Bejeck 
> > >>> wrote:
> > >
> > > Congrats Justine!
> > >
> > > -Bill
> > >
> > > On Sat, Sep 23, 2023 at 6:23 PM Greg Harris
> > >>>  > >
> > > wrote:
> > >
> > >> Congratulations Justine!
> > >>
> > >> On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said
> > >>  wrote:
> > >>>
> > >>> Congrats Justin !
> > >>>
> > >>> On Sat 23 Sep 2023 at 14:44, Randall Hauch  >
> > >>> wrote:
> > >>>
> >  Congratulations, Justine!
> > 
> >  On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash <
> >  kamal.chandraprak...@gmail.com> wrote:
> > 
> > > Congrats Justine!
> > >
> > > On Sat, Sep 23, 2023, 13:28 Divij Vaidya <
> >  divijvaidy...@gmail.com>
> >  wrote:
> > >
> > >> Congratulations Justine!
> > >>
> > >> On Sat 23. Sep 2023 at 07:06, Chris Egerton <
> > >> fearthecel...@gmail.com>
> > >> wrote:
> > >>
> > >>> Congrats Justine!
> > >>> On Fri, Sep 22, 2023, 20:47 Guozhang Wang <
> >  guozhang.wang...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congratulations!
> > 
> >  On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai <
> > >> tzuli...@apache.org
> > 
> >  wrote:
> > >
> > > Congratulations Justine!
> > >
> > > On Fri, Sep 22, 2023, 19:25 Philip Nee <
> >  philip...@gmail.com>
> > > wrote:
> > >
> > >> Congrats Justine!
> > >>
> > >> On Fri, Sep 22, 2023 at 7:07 PM Luke Chen <
> > >> show...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi, Everyone,
> > >>>
> > >>> Justine Olshan has been a Kafka committer since
> > >>> Dec.
> > >> 2022.
> >  She
> > >> has
> >  been
> > >>> very active and instrumental to the community since
> > >> becoming
> >  a
> >  committer.
> > >>> It's my pleasure to announce that Justine is now a
> > >> member of
> > >> Kafka
> >  PMC.
> > >>>
> > >>> Congratulations Justine!
> > >>>
> > >>> Luke
> > >>> on behalf of Apache Kafka PMC
> > >>>
> > >>
> > 
> > >>>
> > >>
> > >
> > 
> > >>
> > 
> > 
> > >>>
> > >>>
> > >>>
> > >
>

[jira] [Resolved] (KAFKA-15485) Support building with Java 21 (LTS release)

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-15485.
-
Resolution: Fixed

> Support building with Java 21 (LTS release)
> ---
>
> Key: KAFKA-15485
> URL: https://issues.apache.org/jira/browse/KAFKA-15485
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Divij Vaidya
>Assignee: Ismael Juma
>Priority: Major
> Fix For: 3.7.0
>
>
> JDK-21 is the latest LTS release which reached GA on 19th Sept 2023. This 
> ticket aims to upgrade JDK used by Kafka to JDK-21 (currently it's JDK20).
> Thanks to proactive work done by [~ijuma] earlier [1][2][3], I do not 
> anticipate major hiccups while upgrading to JDK-21.
> As part of this JIRA we want to:
> 1. Upgrade Kafka to JDK 21
> 2. Replace the CI build for JDK 20 with JDK 21 (similar to [3] below)
> 3. Update the README (see[4]) to mention Kafka's support for JDK-21
> [1] [https://github.com/apache/kafka/pull/13840]
> [2] [https://github.com/apache/kafka/pull/13582]
> [3] [https://github.com/apache/kafka/pull/12948] 
> [4] [https://github.com/apache/kafka/pull/14061] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15501) Kafka to KRaft combined mode migration v3.5

[Ritvik Gupta (Jira)]

Ritvik Gupta created KAFKA-15501:


 Summary: Kafka to KRaft combined mode migration v3.5
 Key: KAFKA-15501
 URL: https://issues.apache.org/jira/browse/KAFKA-15501
 Project: Kafka
  Issue Type: Improvement
  Components: controller, kraft
Reporter: Ritvik Gupta


This is regarding the 
[KIP-866|https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration]
 KRaft migration steps for {*}Kafka v3.5.1{*}.

We want to migrate our Kafka ZK mode cluster to KRaft mode and are following 
the steps mentioned 
[here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration#KIP866ZooKeepertoKRaftMigration-MigrationOverview]
 for separate KRaft brokers and controllers migration.

As mentioned in the KIP-866, currently the [combined mode 
migration|https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration#KIP866ZooKeepertoKRaftMigration-CombinedModeMigrationSupport]
 is not supported and is only suitable for dev environments, but we want to 
utilize *combined mode* for our production environments and continue with our 
current set of broker machines ( as otherwise we would need to provision 
additional controller node machines ).

By which version can we expect the *KRaft combined mode* migration be available 
?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] mimaison opened a new pull request, #552: Add Kafka Connect book

[via GitHub]

mimaison opened a new pull request, #552:
URL: https://github.com/apache/kafka-site/pull/552

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15500) Code bug in SslPrincipalMapper.java

[Svyatoslav (Jira)]

Svyatoslav created KAFKA-15500:
--

 Summary: Code bug in SslPrincipalMapper.java
 Key: KAFKA-15500
 URL: https://issues.apache.org/jira/browse/KAFKA-15500
 Project: Kafka
  Issue Type: Bug
  Components: clients, security
Affects Versions: 3.5.1
Reporter: Svyatoslav


Code bug in:

if (toLowerCase && result != null) {
                result = result.toLowerCase(Locale.ENGLISH);
            } else if (toUpperCase{color:#FF} & {color}result != null) {
                result = result.toUpperCase(Locale.ENGLISH);
            }



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] lucasbru merged pull request #551: MINOR: Add Lucas Brutschy to committers

[via GitHub]

lucasbru merged PR #551:
URL: https://github.com/apache/kafka-site/pull/551


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15499) Fix the flaky DeleteSegmentsDueToLogStartOffsetBreachTest

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-15499:


 Summary: Fix the flaky DeleteSegmentsDueToLogStartOffsetBreachTest
 Key: KAFKA-15499
 URL: https://issues.apache.org/jira/browse/KAFKA-15499
 Project: Kafka
  Issue Type: Test
Reporter: Kamal Chandraprakash
Assignee: Kamal Chandraprakash


Flaky test failure is reported in 
[https://github.com/apache/kafka/pull/14330#issuecomment-1717195473]



{code:java}
java.lang.AssertionError: [BrokerId=0] The base offset of the first log segment 
of topicA-0 in the log directory is 7 which is smaller than the expected offset 
8. The directory of topicA-0 is made of the following files: 
leader-epoch-checkpoint
0009.timeindex
remote_log_snapshot
0009.index
0007.timeindex
0007.index
0007.snapshot
0005.snapshot
0009.log
partition.metadata
0009.snapshot
0007.log
at 
org.apache.kafka.tiered.storage.utils.BrokerLocalStorage.waitForOffset(BrokerLocalStorage.java:118)
at 
org.apache.kafka.tiered.storage.utils.BrokerLocalStorage.waitForEarliestLocalOffset(BrokerLocalStorage.java:77)
at 
org.apache.kafka.tiered.storage.actions.ProduceAction.doExecute(ProduceAction.java:121)
at 
org.apache.kafka.tiered.storage.TieredStorageTestAction.execute(TieredStorageTestAction.java:25)
at 
org.apache.kafka.tiered.storage.TieredStorageTestHarness.executeTieredStorageTest(TieredStorageTestHarness.java:177){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #66

[Apache Jenkins Server]

See 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Divij,

About the system tests, it's me helping Satish working on it since our team
has internal jenkins pipeline for it.
Here's the result:
https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC?usp=sharing

I'm mainly focusing on the failed tests.
For kraft_upgrade_test, I've fixed in this PR
. After the fix, the rerun
results look good.
For quota_test, it's a known issue in our environment. After rerun, all
passed.

Thanks.
Luke



On Mon, Sep 25, 2023 at 4:09 PM Divij Vaidya 
wrote:

> Correction: posted the wrong JIRA in my previous email. Instead of
> https://issues.apache.org/jira/browse/KAFKA-15001, please consider
> this https://issues.apache.org/jira/browse/KAFKA-15487
>
> --
> Divij Vaidya
>
> On Mon, Sep 25, 2023 at 10:04 AM Divij Vaidya 
> wrote:
> >
> > Hi Satish
> >
> > 1. I agree with Luke. It's a "high" severity vulnerability and we
> > should create another RC with the upgraded Snappy version. If we
> > create another RC, we should also fix a different CVE resported in
> > https://issues.apache.org/jira/browse/KAFKA-15001
> >
> > 2. I was hoping you could post the results of system tests before I
> > vote on this. I am particularly interested in looking at
> > producer/consumer performance results since we have quite a few
> > changes in this release. What is the plan on the system tests?
> >
> > --
> > Divij Vaidya
> >
> > On Mon, Sep 25, 2023 at 9:10 AM Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > Snappy-java published a new vulnerability
> > > <
> https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv
> >
> > > that will cause OOM error in the server.
> > > Kafka is also impacted by this vulnerability since it's like
> CVE-2023-34455
> > > .
> > > We'd better bump the snappy-java version to bypass this vulnerability.
> > > PR  is created to run the
> CI
> > > build.
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > > On Mon, Sep 25, 2023 at 2:38 PM Satish Duggana <
> satish.dugg...@gmail.com>
> > > wrote:
> > >
> > > > Thanks to everyone who voted for this release.
> > > >
> > > > We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
> > > > deadline. Please try RC1 and send your vote to this email thread.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > >
> > > > On Sun, 24 Sept 2023 at 13:23, Justine Olshan
> > > >  wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > I've done the following:
> > > > > - Verified signature
> > > > > - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> > > > > - Run unit + integration tests
> > > > > - Ran a shorter Trogdor transactional-produce-bench on a single
> broker
> > > > > cluster (KRaft and ZK) to verify transactional workloads worked
> > > > reasonably
> > > > >
> > > > > Minor thing (we can discuss elsewhere and is non-blocking for the
> > > > release)
> > > > > but if ZK has been deprecated since 3.5 we should move up the
> Kraft setup
> > > > > in the quickstart guide  here
> > > > > .
> > > > >
> > > > > +1 (binding) from me.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri <
> fedeval...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Satish, I did the following to verify the release:
> > > > > >
> > > > > > - Verified signature and checksum
> > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > - Ran all unit and integration tests
> > > > > > - Spot checked release notes and documentation
> > > > > > - Ran a custom client using staging artifacts on a 3-nodes
> cluster
> > > > > > - Tested tiered storage with one of the available RSM
> implementations
> > > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > Thanks
> > > > > > Fede
> > > > > >
> > > > > >
> > > > > > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen 
> wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > I verified with:
> > > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > > 2. Making sure the checksum are correct
> > > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > >
> > > > > > > I filed KAFKA-15491 <
> > > > https://issues.apache.org/jira/browse/KAFKA-15491
> > > > > > >for
> > > > > > > log output improvement while testing stream application.
> > > > > > > It won't be blocker in v3.6.0.
> > > > > > >
> > > > > > > For KAFKA-15489 <
> https://issues.apache.org/jira/browse/KAFKA-15489>,
> > > > I'm
> > > > > > > fine if we decide to fix it in v3.6.1/v3.7.0.
> > > > > > >
> > > > > > > +1 (binding) from me.
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  >
> > > > wrote:
> > > > > > >
> > > > > > > > Given that this is not a regression and there have been no
> reports
> > > 

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Gantigmaa Selenge]

Hi Christo,

Thank you for writing the KIP.

I recently raised a PR to add metrics for tracking remote segment deletions
(https://github.com/apache/kafka/pull/14375) but realised those metrics
were not mentioned in the original KIP-405 or KIP-930. Do you think these
would make sense to be added to this KIP and get included in the discussion?

Regards,
Gantigmaa

On Wed, Aug 9, 2023 at 1:53 PM Christo Lolov  wrote:

> Heya Kamal,
>
> Thank you for going through the KIP and for the question!
>
> I have been thinking about this and as an operator I might find it the most
> useful to know all three of them actually.
>
> I would find knowing the size in bytes useful to determine how much disk I
> might need to add temporarily to compensate for the slowdown.
> I would find knowing the number of records useful, because using the
> MessagesInPerSec metric I would be able to determine how old the records
> which are facing problems are.
> I would find knowing the number of segments useful because I would be able
> to correlate this with whether I need to change
> *remote.log.manager.task.interval.ms
>  *to a lower or higher value.
>
> What are your thoughts on the above? Would you find some of them more
> useful than others?
>
> Best,
> Christo
>
> On Tue, 8 Aug 2023 at 16:43, Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Christo,
> >
> > Thanks for the KIP!
> >
> > The proposed tiered storage metrics are useful. The unit mentioned in the
> > KIP is the number of records.
> > Each topic can have varying amounts of records in a segment depending on
> > the record size.
> >
> > Do you think having the tier-lag by number of segments (or) size of
> > segments in bytes will be useful
> > to the operator?
> >
> > Thanks,
> > Kamal
> >
> > On Tue, Aug 8, 2023 at 8:56 PM Christo Lolov 
> > wrote:
> >
> > > Hello all!
> > >
> > > I would like to start a discussion for KIP-963: Upload and delete lag
> > > metrics in Tiered Storage (
> https://cwiki.apache.org/confluence/x/sZGzDw
> > ).
> > >
> > > The purpose of this KIP is to introduce a couple of metrics to track
> lag
> > > with respect to remote storage from the point of view of Kafka.
> > >
> > > Thanks in advance for leaving a review!
> > >
> > > Best,
> > > Christo
> > >
> >
>

[GitHub] [kafka-site] lucasbru opened a new pull request, #551: MINOR: Add Lucas Brutschy to committers

[via GitHub]

lucasbru opened a new pull request, #551:
URL: https://github.com/apache/kafka-site/pull/551

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

[Lucas Brutschy]

Congrats, Justine!

On Mon, Sep 25, 2023 at 9:20 AM Bruno Cadonna  wrote:
>
> Congrats, Justine! Well deserved!
>
> Best,
> Bruno
>
> On 9/25/23 5:28 AM, ziming deng wrote:
> > Congratulations Justine!
> >
> >
> >> On Sep 25, 2023, at 00:01, Viktor Somogyi-Vass 
> >>  wrote:
> >>
> >> Congrats Justine!
> >>
> >> On Sun, Sep 24, 2023, 17:45 Kirk True  wrote:
> >>
> >>> Congratulations Justine! Thanks for all your great work!
> >>>
>  On Sep 24, 2023, at 8:37 AM, John Roesler  wrote:
> 
>  Congratulations, Justine!
>  -John
> 
>  On Sun, Sep 24, 2023, at 05:05, Mickael Maison wrote:
> > Congratulations Justine!
> >
> > On Sun, Sep 24, 2023 at 5:04 AM Sophie Blee-Goldman
> >  wrote:
> >>
> >> Congrats Justine!
> >>
> >> On Sat, Sep 23, 2023, 4:36 PM Tom Bentley  wrote:
> >>
> >>> Congratulations!
> >>>
> >>> On Sun, 24 Sept 2023 at 12:32, Satish Duggana <
> >>> satish.dugg...@gmail.com>
> >>> wrote:
> >>>
>  Congratulations Justine!!
> 
>  On Sat, 23 Sept 2023 at 15:46, Bill Bejeck 
> >>> wrote:
> >
> > Congrats Justine!
> >
> > -Bill
> >
> > On Sat, Sep 23, 2023 at 6:23 PM Greg Harris
> >>>  >
> > wrote:
> >
> >> Congratulations Justine!
> >>
> >> On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said
> >>  wrote:
> >>>
> >>> Congrats Justin !
> >>>
> >>> On Sat 23 Sep 2023 at 14:44, Randall Hauch 
> >>> wrote:
> >>>
>  Congratulations, Justine!
> 
>  On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash <
>  kamal.chandraprak...@gmail.com> wrote:
> 
> > Congrats Justine!
> >
> > On Sat, Sep 23, 2023, 13:28 Divij Vaidya <
>  divijvaidy...@gmail.com>
>  wrote:
> >
> >> Congratulations Justine!
> >>
> >> On Sat 23. Sep 2023 at 07:06, Chris Egerton <
> >> fearthecel...@gmail.com>
> >> wrote:
> >>
> >>> Congrats Justine!
> >>> On Fri, Sep 22, 2023, 20:47 Guozhang Wang <
>  guozhang.wang...@gmail.com>
> >>> wrote:
> >>>
>  Congratulations!
> 
>  On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai <
> >> tzuli...@apache.org
> 
>  wrote:
> >
> > Congratulations Justine!
> >
> > On Fri, Sep 22, 2023, 19:25 Philip Nee <
>  philip...@gmail.com>
> > wrote:
> >
> >> Congrats Justine!
> >>
> >> On Fri, Sep 22, 2023 at 7:07 PM Luke Chen <
> >> show...@gmail.com>
> >> wrote:
> >>
> >>> Hi, Everyone,
> >>>
> >>> Justine Olshan has been a Kafka committer since
> >>> Dec.
> >> 2022.
>  She
> >> has
>  been
> >>> very active and instrumental to the community since
> >> becoming
>  a
>  committer.
> >>> It's my pleasure to announce that Justine is now a
> >> member of
> >> Kafka
>  PMC.
> >>>
> >>> Congratulations Justine!
> >>>
> >>> Luke
> >>> on behalf of Apache Kafka PMC
> >>>
> >>
> 
> >>>
> >>
> >
> 
> >>
> 
> 
> >>>
> >>>
> >>>
> >

[jira] [Created] (KAFKA-15498) [CVE fix] Upgrade Snappy-Java to 1.1.10.4

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15498:


 Summary: [CVE fix] Upgrade Snappy-Java to 1.1.10.4
 Key: KAFKA-15498
 URL: https://issues.apache.org/jira/browse/KAFKA-15498
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.5.1, 3.4.1
Reporter: Luke Chen
Assignee: Luke Chen
 Fix For: 3.6.0


Snappy-java published a new vulnerability
<[https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv]>
that will cause OOM error in the server.
Kafka is also impacted by this vulnerability since it's like CVE-2023-34455
<[https://nvd.nist.gov/vuln/detail/CVE-2023-34455]>.
We'd better bump the snappy-java version to bypass this vulnerability.
PR <[https://github.com/apache/kafka/pull/14434]> is created to run the CI
build.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Divij Vaidya]

Correction: posted the wrong JIRA in my previous email. Instead of
https://issues.apache.org/jira/browse/KAFKA-15001, please consider
this https://issues.apache.org/jira/browse/KAFKA-15487

--
Divij Vaidya

On Mon, Sep 25, 2023 at 10:04 AM Divij Vaidya  wrote:
>
> Hi Satish
>
> 1. I agree with Luke. It's a "high" severity vulnerability and we
> should create another RC with the upgraded Snappy version. If we
> create another RC, we should also fix a different CVE resported in
> https://issues.apache.org/jira/browse/KAFKA-15001
>
> 2. I was hoping you could post the results of system tests before I
> vote on this. I am particularly interested in looking at
> producer/consumer performance results since we have quite a few
> changes in this release. What is the plan on the system tests?
>
> --
> Divij Vaidya
>
> On Mon, Sep 25, 2023 at 9:10 AM Luke Chen  wrote:
> >
> > Hi Satish,
> >
> > Snappy-java published a new vulnerability
> > 
> > that will cause OOM error in the server.
> > Kafka is also impacted by this vulnerability since it's like CVE-2023-34455
> > .
> > We'd better bump the snappy-java version to bypass this vulnerability.
> > PR  is created to run the CI
> > build.
> >
> > Thanks.
> > Luke
> >
> >
> > On Mon, Sep 25, 2023 at 2:38 PM Satish Duggana 
> > wrote:
> >
> > > Thanks to everyone who voted for this release.
> > >
> > > We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
> > > deadline. Please try RC1 and send your vote to this email thread.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Sun, 24 Sept 2023 at 13:23, Justine Olshan
> > >  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > I've done the following:
> > > > - Verified signature
> > > > - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> > > > - Run unit + integration tests
> > > > - Ran a shorter Trogdor transactional-produce-bench on a single broker
> > > > cluster (KRaft and ZK) to verify transactional workloads worked
> > > reasonably
> > > >
> > > > Minor thing (we can discuss elsewhere and is non-blocking for the
> > > release)
> > > > but if ZK has been deprecated since 3.5 we should move up the Kraft 
> > > > setup
> > > > in the quickstart guide  here
> > > > .
> > > >
> > > > +1 (binding) from me.
> > > >
> > > > Justine
> > > >
> > > > On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Hi Satish, I did the following to verify the release:
> > > > >
> > > > > - Verified signature and checksum
> > > > > - Built from source with Java 17 and Scala 2.13
> > > > > - Ran all unit and integration tests
> > > > > - Spot checked release notes and documentation
> > > > > - Ran a custom client using staging artifacts on a 3-nodes cluster
> > > > > - Tested tiered storage with one of the available RSM implementations
> > > > >
> > > > > +1 (non binding)
> > > > >
> > > > > Thanks
> > > > > Fede
> > > > >
> > > > >
> > > > > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I verified with:
> > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > 2. Making sure the checksum are correct
> > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > >
> > > > > > I filed KAFKA-15491 <
> > > https://issues.apache.org/jira/browse/KAFKA-15491
> > > > > >for
> > > > > > log output improvement while testing stream application.
> > > > > > It won't be blocker in v3.6.0.
> > > > > >
> > > > > > For KAFKA-15489 ,
> > > I'm
> > > > > > fine if we decide to fix it in v3.6.1/v3.7.0.
> > > > > >
> > > > > > +1 (binding) from me.
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > > > On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > Given that this is not a regression and there have been no reports
> > > for
> > > > > over
> > > > > > > a year, I think it's ok for this to land in 3.6.1.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Luke for reporting KRaft issue[1].
> > > > > > > >
> > > > > > > > I am not sure whether it is a release blocker for 3.6.0. Need
> > > input
> > > > > > > > from other KRaft experts also to finalize the decision. Even if
> > > we
> > > > > > > > adopt a fix, do not we need to bake it for some time before it 
> > > > > > > > is
> > > > > > > > pushed to production to avoid any regressions as this change is
> > > in
> > > > > the
> > > > > > > > critical paths?
> > > > > > > >
> > > > > > > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > > > > > > >
> > > > > > > > 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Divij Vaidya]

Hi Satish

1. I agree with Luke. It's a "high" severity vulnerability and we
should create another RC with the upgraded Snappy version. If we
create another RC, we should also fix a different CVE resported in
https://issues.apache.org/jira/browse/KAFKA-15001

2. I was hoping you could post the results of system tests before I
vote on this. I am particularly interested in looking at
producer/consumer performance results since we have quite a few
changes in this release. What is the plan on the system tests?

--
Divij Vaidya

On Mon, Sep 25, 2023 at 9:10 AM Luke Chen  wrote:
>
> Hi Satish,
>
> Snappy-java published a new vulnerability
> 
> that will cause OOM error in the server.
> Kafka is also impacted by this vulnerability since it's like CVE-2023-34455
> .
> We'd better bump the snappy-java version to bypass this vulnerability.
> PR  is created to run the CI
> build.
>
> Thanks.
> Luke
>
>
> On Mon, Sep 25, 2023 at 2:38 PM Satish Duggana 
> wrote:
>
> > Thanks to everyone who voted for this release.
> >
> > We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
> > deadline. Please try RC1 and send your vote to this email thread.
> >
> > Thanks,
> > Satish.
> >
> >
> > On Sun, 24 Sept 2023 at 13:23, Justine Olshan
> >  wrote:
> > >
> > > Hi Satish,
> > >
> > > I've done the following:
> > > - Verified signature
> > > - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> > > - Run unit + integration tests
> > > - Ran a shorter Trogdor transactional-produce-bench on a single broker
> > > cluster (KRaft and ZK) to verify transactional workloads worked
> > reasonably
> > >
> > > Minor thing (we can discuss elsewhere and is non-blocking for the
> > release)
> > > but if ZK has been deprecated since 3.5 we should move up the Kraft setup
> > > in the quickstart guide  here
> > > .
> > >
> > > +1 (binding) from me.
> > >
> > > Justine
> > >
> > > On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri 
> > > wrote:
> > >
> > > > Hi Satish, I did the following to verify the release:
> > > >
> > > > - Verified signature and checksum
> > > > - Built from source with Java 17 and Scala 2.13
> > > > - Ran all unit and integration tests
> > > > - Spot checked release notes and documentation
> > > > - Ran a custom client using staging artifacts on a 3-nodes cluster
> > > > - Tested tiered storage with one of the available RSM implementations
> > > >
> > > > +1 (non binding)
> > > >
> > > > Thanks
> > > > Fede
> > > >
> > > >
> > > > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > I verified with:
> > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > 2. Making sure the checksum are correct
> > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > >
> > > > > I filed KAFKA-15491 <
> > https://issues.apache.org/jira/browse/KAFKA-15491
> > > > >for
> > > > > log output improvement while testing stream application.
> > > > > It won't be blocker in v3.6.0.
> > > > >
> > > > > For KAFKA-15489 ,
> > I'm
> > > > > fine if we decide to fix it in v3.6.1/v3.7.0.
> > > > >
> > > > > +1 (binding) from me.
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Given that this is not a regression and there have been no reports
> > for
> > > > over
> > > > > > a year, I think it's ok for this to land in 3.6.1.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana <
> > > > satish.dugg...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Luke for reporting KRaft issue[1].
> > > > > > >
> > > > > > > I am not sure whether it is a release blocker for 3.6.0. Need
> > input
> > > > > > > from other KRaft experts also to finalize the decision. Even if
> > we
> > > > > > > adopt a fix, do not we need to bake it for some time before it is
> > > > > > > pushed to production to avoid any regressions as this change is
> > in
> > > > the
> > > > > > > critical paths?
> > > > > > >
> > > > > > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Satish.
> > > > > > >
> > > > > > > On Sat, 23 Sept 2023 at 03:08, Luke Chen 
> > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > I found the current KRaft implementation will have "split
> > brain"
> > > > issue
> > > > > > > when
> > > > > > > > network partition happens, which will cause inconsistent
> > metadata
> > > > > > > returned
> > > > > > > > from the controller.
> > > > > > > > Filed KAFKA-15489 <
> > > > https://issues.apache.org/jira/browse/KAFKA-15489>
> > > > > > > for
> > > > > > > > 

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

[Bruno Cadonna]

Congrats, Justine! Well deserved!

Best,
Bruno

On 9/25/23 5:28 AM, ziming deng wrote:

Congratulations Justine!



On Sep 25, 2023, at 00:01, Viktor Somogyi-Vass 
 wrote:

Congrats Justine!

On Sun, Sep 24, 2023, 17:45 Kirk True  wrote:


Congratulations Justine! Thanks for all your great work!


On Sep 24, 2023, at 8:37 AM, John Roesler  wrote:

Congratulations, Justine!
-John

On Sun, Sep 24, 2023, at 05:05, Mickael Maison wrote:

Congratulations Justine!

On Sun, Sep 24, 2023 at 5:04 AM Sophie Blee-Goldman
 wrote:


Congrats Justine!

On Sat, Sep 23, 2023, 4:36 PM Tom Bentley  wrote:


Congratulations!

On Sun, 24 Sept 2023 at 12:32, Satish Duggana <

satish.dugg...@gmail.com>

wrote:


Congratulations Justine!!

On Sat, 23 Sept 2023 at 15:46, Bill Bejeck 

wrote:


Congrats Justine!

-Bill

On Sat, Sep 23, 2023 at 6:23 PM Greg Harris



wrote:


Congratulations Justine!

On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said
 wrote:


Congrats Justin !

On Sat 23 Sep 2023 at 14:44, Randall Hauch 

wrote:



Congratulations, Justine!

On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:


Congrats Justine!

On Sat, Sep 23, 2023, 13:28 Divij Vaidya <

divijvaidy...@gmail.com>

wrote:



Congratulations Justine!

On Sat 23. Sep 2023 at 07:06, Chris Egerton <

fearthecel...@gmail.com>

wrote:


Congrats Justine!
On Fri, Sep 22, 2023, 20:47 Guozhang Wang <

guozhang.wang...@gmail.com>

wrote:


Congratulations!

On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai <

tzuli...@apache.org


wrote:


Congratulations Justine!

On Fri, Sep 22, 2023, 19:25 Philip Nee <

philip...@gmail.com>

wrote:



Congrats Justine!

On Fri, Sep 22, 2023 at 7:07 PM Luke Chen <

show...@gmail.com>

wrote:



Hi, Everyone,

Justine Olshan has been a Kafka committer since

Dec.

2022.

She

has

been

very active and instrumental to the community since

becoming

a

committer.

It's my pleasure to announce that Justine is now a

member of

Kafka

PMC.


Congratulations Justine!

Luke
on behalf of Apache Kafka PMC

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Satish,

Snappy-java published a new vulnerability

that will cause OOM error in the server.
Kafka is also impacted by this vulnerability since it's like CVE-2023-34455
.
We'd better bump the snappy-java version to bypass this vulnerability.
PR  is created to run the CI
build.

Thanks.
Luke


On Mon, Sep 25, 2023 at 2:38 PM Satish Duggana 
wrote:

> Thanks to everyone who voted for this release.
>
> We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
> deadline. Please try RC1 and send your vote to this email thread.
>
> Thanks,
> Satish.
>
>
> On Sun, 24 Sept 2023 at 13:23, Justine Olshan
>  wrote:
> >
> > Hi Satish,
> >
> > I've done the following:
> > - Verified signature
> > - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> > - Run unit + integration tests
> > - Ran a shorter Trogdor transactional-produce-bench on a single broker
> > cluster (KRaft and ZK) to verify transactional workloads worked
> reasonably
> >
> > Minor thing (we can discuss elsewhere and is non-blocking for the
> release)
> > but if ZK has been deprecated since 3.5 we should move up the Kraft setup
> > in the quickstart guide  here
> > .
> >
> > +1 (binding) from me.
> >
> > Justine
> >
> > On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri 
> > wrote:
> >
> > > Hi Satish, I did the following to verify the release:
> > >
> > > - Verified signature and checksum
> > > - Built from source with Java 17 and Scala 2.13
> > > - Ran all unit and integration tests
> > > - Spot checked release notes and documentation
> > > - Ran a custom client using staging artifacts on a 3-nodes cluster
> > > - Tested tiered storage with one of the available RSM implementations
> > >
> > > +1 (non binding)
> > >
> > > Thanks
> > > Fede
> > >
> > >
> > > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > I verified with:
> > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > 2. Making sure the checksum are correct
> > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > >
> > > > I filed KAFKA-15491 <
> https://issues.apache.org/jira/browse/KAFKA-15491
> > > >for
> > > > log output improvement while testing stream application.
> > > > It won't be blocker in v3.6.0.
> > > >
> > > > For KAFKA-15489 ,
> I'm
> > > > fine if we decide to fix it in v3.6.1/v3.7.0.
> > > >
> > > > +1 (binding) from me.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma 
> wrote:
> > > >
> > > > > Given that this is not a regression and there have been no reports
> for
> > > over
> > > > > a year, I think it's ok for this to land in 3.6.1.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Luke for reporting KRaft issue[1].
> > > > > >
> > > > > > I am not sure whether it is a release blocker for 3.6.0. Need
> input
> > > > > > from other KRaft experts also to finalize the decision. Even if
> we
> > > > > > adopt a fix, do not we need to bake it for some time before it is
> > > > > > pushed to production to avoid any regressions as this change is
> in
> > > the
> > > > > > critical paths?
> > > > > >
> > > > > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > > > > >
> > > > > > Thanks,
> > > > > > Satish.
> > > > > >
> > > > > > On Sat, 23 Sept 2023 at 03:08, Luke Chen 
> wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > I found the current KRaft implementation will have "split
> brain"
> > > issue
> > > > > > when
> > > > > > > network partition happens, which will cause inconsistent
> metadata
> > > > > > returned
> > > > > > > from the controller.
> > > > > > > Filed KAFKA-15489 <
> > > https://issues.apache.org/jira/browse/KAFKA-15489>
> > > > > > for
> > > > > > > this issue, and PR 
> is
> > > > > ready
> > > > > > > for review.
> > > > > > >
> > > > > > > Even though this is not a regression issue (this has already
> > > existed
> > > > > > since
> > > > > > > the 1st release of KRaft feature), I think this is an important
> > > issue
> > > > > > since
> > > > > > > KRaft is announced production ready.
> > > > > > > Not sure what other people's thoughts are.
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Thu, Sep 21, 2023 at 6:33 PM Josep Prat
> > >  > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > I ran the following validation steps:
> > > > > > > > - Built from source with Java 11 and Scala 2.13
> > > > > > > > - Verified Signatures and hashes of the artifacts 

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks to everyone who voted for this release.

We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
deadline. Please try RC1 and send your vote to this email thread.

Thanks,
Satish.


On Sun, 24 Sept 2023 at 13:23, Justine Olshan
 wrote:
>
> Hi Satish,
>
> I've done the following:
> - Verified signature
> - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> - Run unit + integration tests
> - Ran a shorter Trogdor transactional-produce-bench on a single broker
> cluster (KRaft and ZK) to verify transactional workloads worked reasonably
>
> Minor thing (we can discuss elsewhere and is non-blocking for the release)
> but if ZK has been deprecated since 3.5 we should move up the Kraft setup
> in the quickstart guide  here
> .
>
> +1 (binding) from me.
>
> Justine
>
> On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri 
> wrote:
>
> > Hi Satish, I did the following to verify the release:
> >
> > - Verified signature and checksum
> > - Built from source with Java 17 and Scala 2.13
> > - Ran all unit and integration tests
> > - Spot checked release notes and documentation
> > - Ran a custom client using staging artifacts on a 3-nodes cluster
> > - Tested tiered storage with one of the available RSM implementations
> >
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> >
> > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I verified with:
> > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > 2. Making sure the checksum are correct
> > > 3. Browsing release notes, documents, javadocs, protocols.
> > >
> > > I filed KAFKA-15491  > >for
> > > log output improvement while testing stream application.
> > > It won't be blocker in v3.6.0.
> > >
> > > For KAFKA-15489 , I'm
> > > fine if we decide to fix it in v3.6.1/v3.7.0.
> > >
> > > +1 (binding) from me.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  wrote:
> > >
> > > > Given that this is not a regression and there have been no reports for
> > over
> > > > a year, I think it's ok for this to land in 3.6.1.
> > > >
> > > > Ismael
> > > >
> > > > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Luke for reporting KRaft issue[1].
> > > > >
> > > > > I am not sure whether it is a release blocker for 3.6.0. Need input
> > > > > from other KRaft experts also to finalize the decision. Even if we
> > > > > adopt a fix, do not we need to bake it for some time before it is
> > > > > pushed to production to avoid any regressions as this change is in
> > the
> > > > > critical paths?
> > > > >
> > > > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I found the current KRaft implementation will have "split brain"
> > issue
> > > > > when
> > > > > > network partition happens, which will cause inconsistent metadata
> > > > > returned
> > > > > > from the controller.
> > > > > > Filed KAFKA-15489 <
> > https://issues.apache.org/jira/browse/KAFKA-15489>
> > > > > for
> > > > > > this issue, and PR  is
> > > > ready
> > > > > > for review.
> > > > > >
> > > > > > Even though this is not a regression issue (this has already
> > existed
> > > > > since
> > > > > > the 1st release of KRaft feature), I think this is an important
> > issue
> > > > > since
> > > > > > KRaft is announced production ready.
> > > > > > Not sure what other people's thoughts are.
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > > > On Thu, Sep 21, 2023 at 6:33 PM Josep Prat
> >  > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > I ran the following validation steps:
> > > > > > > - Built from source with Java 11 and Scala 2.13
> > > > > > > - Verified Signatures and hashes of the artifacts generated
> > > > > > > - Navigated through Javadoc including links to JDK classes
> > > > > > > - Run the unit tests
> > > > > > > - Run integration tests
> > > > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > > > >
> > > > > > >
> > > > > > > I +1 this release (non-binding)
> > > > > > >
> > > > > > > Thanks for your efforts!
> > > > > > >
> > > > > > > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Greg for verifying the release including the earlier
> > > > > > > > blocker(KAFKA-15473) verification.
> > > > > > > >
> > > > > > > > ~Satish.
> > > > > > > >
> > > > > > > > On Wed, 20 Sept 2023 at 22:30, Greg Harris
> > > > >  > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >