Re: CI on Kafka seems to be failing even on trunk branch

[deng ziming]

You can help to solve this bug first if you are free, 
https://issues.apache.org/jira/browse/KAFKA-13907 
, since this bug can be 
reproduced locally, it won't be hard to investigated.

> On May 17, 2022, at 12:34 AM, Lim Qingwei  wrote:
> 
> Hi, I am a newbie.
> 
> I noticed Jenkins builds are failing, is this expected?
> My PR has some failures, but I don't think they are relevant, how should I
> proceed?
> 
> Can I still request for reviews?
> 
> Kind regards

[jira] [Created] (KAFKA-13907) Fix hanging ServerShutdownTest.testCleanShutdownWithKRaftControllerUnavailable

[dengziming (Jira)]

dengziming created KAFKA-13907:
--

 Summary: Fix hanging 
ServerShutdownTest.testCleanShutdownWithKRaftControllerUnavailable
 Key: KAFKA-13907
 URL: https://issues.apache.org/jira/browse/KAFKA-13907
 Project: Kafka
  Issue Type: Bug
Reporter: dengziming


ServerShutdownTest.testCleanShutdownWithKRaftControllerUnavailable will hang up 
waiting for controlled shutdown, there may be some bug related to it.

since this bug can be reproduced locally, it won't be hard to investigated.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: CI on Kafka seems to be failing even on trunk branch

[Luke Chen]

Thanks for Divij's answer.
One thing I'd like to add is that it'd be better if you can help
investigate and fix those flaky tests. :)

Thank you.
Luke

On Tue, May 17, 2022 at 4:14 AM Divij Vaidya 
wrote:

> Hi Lim
>
> We have quite a few tests which are flaky and fail stochastically from time
> to time.
>
> I would recommend that you check the tests which are failing for your PR.
> Search for JIRA for the test failures. If you spot a flaky test JIRA for
> the failure associated with your PR and the stack trace mentioned in the
> JIRA is same as yours, then add this information to your PR. This activity
> will help the reviewers understand whether the test failure are due to your
> changes or not.
>
> Alternatively, another approach (which I don’t recommend) is keep adding
> commits to the PR to trigger the full test run and hope one of them is
> successful.
>
> Please feel free to ask questions if you need help with finding the flaky
> test.
>
> On Mon, May 16, 2022 at 6:35 PM Lim Qingwei  wrote:
>
> > Hi, I am a newbie.
> >
> > I noticed Jenkins builds are failing, is this expected?
> > My PR has some failures, but I don't think they are relevant, how should
> I
> > proceed?
> >
> > Can I still request for reviews?
> >
> > Kind regards
> >
> --
> Divij Vaidya
>

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[Niket Goel]

Thanks for the call out David. We will populate these fields for the
Observers as well. I will clarify this in the KIP.

On Mon, May 16, 2022 at 1:50 PM David Arthur  wrote:

> Niket, thanks for the KIP!
>
> Sorry for the late feedback on this, but I just had a quick question. The
> KIP indicates the two new fields will be set for voters only, however this
> ReplicaState struct is also used by the Observers in
> DescribeQuorumResponse. Will we simply fill in -1 for these values, or do
> we intend to report the last fetch and caught-up time of the observers as
> well?
>
> Thanks!
> David
>
>
> On Mon, May 16, 2022 at 1:46 PM Niket Goel 
> wrote:
>
> > Hi all,
> >
> > Thank you for the feedback on this. I have started a voting thread for
> > this KIP here:
> > https://lists.apache.org/thread/bkb7gsbxpljh5qh014ztffq7bldjrb2x
> >
> > Thanks
> > Niket Goel
> >
> >
> > From: Niket Goel 
> > Date: Thursday, May 12, 2022 at 5:25 PM
> > To: dev@kafka.apache.org 
> > Subject: Re: [DISCUSS] KIP-836: Addition of Information in
> > DescribeQuorumResponse about Voter Lag
> > Appreciate the careful review Jose.!
> >
> > Ack on 1 and 2. Will fix.
> >
> > For number 3 (and I am using [1] as a reference for this discussion), I
> > think the correct language to use would be:
> >
> > "Whenever a new fetch request
> > comes in the replica's last caught up time is updated to the time of
> > this fetch request if it requests an offset greater than or equal to the
> > leader's
> > current end offset"
> > Does that sound right now?
> >
> > Although I think I will go ahead and rewrite the explanation in a way
> that
> > is more understandable. Thanks for pointing this out.
> >
> > Thanks
> >
> > [1]
> >
> https://github.com/apache/kafka/blob/fa59be4e770627cd34cef85986b58ad7f606928d/core/src/main/scala/kafka/cluster/Replica.scala#L97
> >
> >
> >
> > On Thu, May 12, 2022 at 3:20 PM José Armando García Sancio
> >  wrote:
> > Thanks for the Kafka improvement Niket.
> >
> > 1. For the fields `LastFetchTime` and `LastCaughtUpTime`, Kafka tends
> > to use the suffix "Timestamp" when the value is an absolute wall clock
> > value.
> >
> > 2. The method `result()` for the type `DescribeQuorumResult` returns
> > the type `DescribeQuorumResponseData`. The types generated from the
> > RPC JSON schema are internal to Kafka and not exposed to clients. For
> > the admin client we should use a different type that is explicitly
> > public. See `org.apache.kafka.client.admin.DescribeTopicsResult` for
> > an example.
> >
> > 3. The proposed section has his sentence "Whenever a new fetch request
> > comes in the replica's last caught up time is updated to the time of
> > the fetch request if it requests an offset greater than the leader's
> > current end offset." Did you mean "previous fetch time" instead of
> > "last caught up time"? What do you mean by "requests an offset greater
> > than the leader's current end offset.?" Excluding diverging logs the
> > follower fetch offset should never be greater than the leader LEO.
> >
> > Thanks,
> > -José
> >
> >
> > --
> > - Niket
> >
>


-- 
- Niket

[jira] [Resolved] (KAFKA-13899) Inconsistent error codes returned from AlterConfig APIs

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13899.
-
Fix Version/s: 3.2.1
   Resolution: Fixed

> Inconsistent error codes returned from AlterConfig APIs
> ---
>
> Key: KAFKA-13899
> URL: https://issues.apache.org/jira/browse/KAFKA-13899
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.2.1
>
>
> In the AlterConfigs/IncrementalAlterConfigs zk handler, we return 
> INVALID_REQUEST and INVALID_CONFIG inconsistently. The problem is in 
> `LogConfig.validate`. We may either return `ConfigException` or 
> `InvalidConfigException`. When the first of these is thrown, we catch it and 
> convert to INVALID_REQUEST. It seems more consistent to convert to 
> INVALID_CONFIG.
> Note that the kraft implementation returns INVALID_CONFIG consistently.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[Jun Rao]

Hi, Jose,

Thanks for the KIP. Just a couple of minor comments.

20. For the metric type and name, we use the camel names in some cases and
dashed lower names in some other cases. Should we make them consistent?

21. Could you document the meaning of load-processing-time?

Thanks,

Jun

On Mon, May 16, 2022 at 9:01 AM José Armando García Sancio
 wrote:

> Hi all,
>
> Thanks for your feedback. I started a voting thread here:
> https://lists.apache.org/thread/x1cy5otpf7mj9ytghnktr5hog27hdf7k
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #935

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 627753 lines...]
[2022-05-16T18:20:37.719Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceThreads PASSED
[2022-05-16T18:20:37.719Z] 
[2022-05-16T18:20:37.719Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalStateException STARTED
[2022-05-16T18:20:42.081Z] 
[2022-05-16T18:20:42.081Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalStateException PASSED
[2022-05-16T18:20:42.081Z] 
[2022-05-16T18:20:42.081Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler STARTED
[2022-05-16T18:20:48.538Z] 
[2022-05-16T18:20:48.538Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownThreadUsingOldHandler PASSED
[2022-05-16T18:20:48.538Z] 
[2022-05-16T18:20:48.538Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread STARTED
[2022-05-16T18:20:51.849Z] 
[2022-05-16T18:20:51.849Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-3362c993-5f9f-4a68-bcdf-b625f3fe6b37-GlobalStreamThread"
[2022-05-16T18:20:51.849Z] 
[2022-05-16T18:20:51.849Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread PASSED
[2022-05-16T18:20:51.849Z] 
[2022-05-16T18:20:51.849Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalArgumentException STARTED
[2022-05-16T18:20:54.940Z] 
[2022-05-16T18:20:54.940Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClientWhenIllegalArgumentException PASSED
[2022-05-16T18:20:54.940Z] 
[2022-05-16T18:20:54.940Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceSingleThread STARTED
[2022-05-16T18:20:59.322Z] 
[2022-05-16T18:20:59.322Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-ba03cb32-f95c-442d-9e7f-8c9797663a87-StreamThread-1"
[2022-05-16T18:20:59.322Z] 
[2022-05-16T18:20:59.322Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceSingleThread PASSED
[2022-05-16T18:20:59.322Z] 
[2022-05-16T18:20:59.322Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownMultipleThreadApplication STARTED
[2022-05-16T18:21:24.371Z] 
[2022-05-16T18:21:24.371Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownMultipleThreadApplication PASSED
[2022-05-16T18:21:24.371Z] 
[2022-05-16T18:21:24.371Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClient STARTED
[2022-05-16T18:21:25.524Z] 
[2022-05-16T18:21:25.524Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClient PASSED
[2022-05-16T18:21:25.524Z] 
[2022-05-16T18:21:25.524Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownSingleThreadApplication STARTED
[2022-05-16T18:21:36.748Z] 
[2022-05-16T18:21:36.748Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownSingleThreadApplication PASSED
[2022-05-16T18:21:36.748Z] 
[2022-05-16T18:21:36.748Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldEmitSameRecordAfterFailover STARTED
[2022-05-16T18:21:43.254Z] 
[2022-05-16T18:21:43.254Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldEmitSameRecordAfterFailover PASSED
[2022-05-16T18:21:44.356Z] 
[2022-05-16T18:21:44.356Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[at_least_once] STARTED
[2022-05-16T18:22:37.603Z] 
[2022-05-16T18:22:37.603Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[at_least_once] PASSED
[2022-05-16T18:22:37.603Z] 
[2022-05-16T18:22:37.603Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once] STARTED
[2022-05-16T18:23:48.254Z] 
[2022-05-16T18:23:48.254Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once] PASSED

Re: [VOTE] KIP-835: Monitor KRaft Controller Quorum Health

[Guozhang Wang]

Hello José,

Could you elaborate a bit on what does "load-processing-time-us" measure? I
looked through the discussion thread and the KIP / JIRA but cannot find its
definitions.


Guozhang

On Mon, May 16, 2022 at 1:30 PM David Arthur  wrote:

> Thanks José, the KIP looks good to me!
>
> +1 binding
>
> -David
>
> On Mon, May 16, 2022 at 11:59 AM José Armando García Sancio
>  wrote:
>
> > Hi all,
> >
> > I would like to start a vote for KIP-835:
> > https://cwiki.apache.org/confluence/x/0xShD
> >
> > Thanks you,
> > -José
> >
>
>
> --
> David Arthur
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[David Arthur]

Niket, thanks for the KIP!

Sorry for the late feedback on this, but I just had a quick question. The
KIP indicates the two new fields will be set for voters only, however this
ReplicaState struct is also used by the Observers in
DescribeQuorumResponse. Will we simply fill in -1 for these values, or do
we intend to report the last fetch and caught-up time of the observers as
well?

Thanks!
David


On Mon, May 16, 2022 at 1:46 PM Niket Goel 
wrote:

> Hi all,
>
> Thank you for the feedback on this. I have started a voting thread for
> this KIP here:
> https://lists.apache.org/thread/bkb7gsbxpljh5qh014ztffq7bldjrb2x
>
> Thanks
> Niket Goel
>
>
> From: Niket Goel 
> Date: Thursday, May 12, 2022 at 5:25 PM
> To: dev@kafka.apache.org 
> Subject: Re: [DISCUSS] KIP-836: Addition of Information in
> DescribeQuorumResponse about Voter Lag
> Appreciate the careful review Jose.!
>
> Ack on 1 and 2. Will fix.
>
> For number 3 (and I am using [1] as a reference for this discussion), I
> think the correct language to use would be:
>
> "Whenever a new fetch request
> comes in the replica's last caught up time is updated to the time of
> this fetch request if it requests an offset greater than or equal to the
> leader's
> current end offset"
> Does that sound right now?
>
> Although I think I will go ahead and rewrite the explanation in a way that
> is more understandable. Thanks for pointing this out.
>
> Thanks
>
> [1]
> https://github.com/apache/kafka/blob/fa59be4e770627cd34cef85986b58ad7f606928d/core/src/main/scala/kafka/cluster/Replica.scala#L97
>
>
>
> On Thu, May 12, 2022 at 3:20 PM José Armando García Sancio
>  wrote:
> Thanks for the Kafka improvement Niket.
>
> 1. For the fields `LastFetchTime` and `LastCaughtUpTime`, Kafka tends
> to use the suffix "Timestamp" when the value is an absolute wall clock
> value.
>
> 2. The method `result()` for the type `DescribeQuorumResult` returns
> the type `DescribeQuorumResponseData`. The types generated from the
> RPC JSON schema are internal to Kafka and not exposed to clients. For
> the admin client we should use a different type that is explicitly
> public. See `org.apache.kafka.client.admin.DescribeTopicsResult` for
> an example.
>
> 3. The proposed section has his sentence "Whenever a new fetch request
> comes in the replica's last caught up time is updated to the time of
> the fetch request if it requests an offset greater than the leader's
> current end offset." Did you mean "previous fetch time" instead of
> "last caught up time"? What do you mean by "requests an offset greater
> than the leader's current end offset.?" Excluding diverging logs the
> follower fetch offset should never be greater than the leader LEO.
>
> Thanks,
> -José
>
>
> --
> - Niket
>

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

[Colin McCabe]

On Mon, May 16, 2022, at 10:24, Omnia Ibrahim wrote:
> Hi Colin,
>
> Thanks for your reply.
>
> This KIP doesn’t aim to solve any security concerns, but rather a conflict
> of responsibilities within any Kafka ecosystem that includes MM2 and any
> resource management solution. I’m not sure that was clear, so I’m concerned
> about the motivation for your suggestion to close this KIP.
>

Hi Omnia,

Thanks for the clarification. I agree it's reasonable for people to want to use 
their own implementations of Admin. And we could have a config for this, so 
that it becomes pluggable (possibly in other places than MirrorMaker, although 
we don't have to do that in this KIP).

We could even try to make this easier on developers. For example, we could 
provide a public ForwardingAdmin class that forwards all requests to the 
regular KafkaAdminClient. Then, people could make their custom class inherit 
from ForwardingAdmin and override just the specific methods that they wanted to 
override. So they don't have to implement all the methods, but just the ones 
that are different for them.

I just wanted to make sure we weren't creating a second Admin client interface 
-- I think that would really be hard for us to support long-term.

>
> It is generally accepted that resource management should be centralized,
> especially on the scale of mirroring N number of clusters. The point of
> this KIP is that any sort of topic management / federate solution /
> up-front capacity planning system will be at odds with MM2 if MM2 keeps
> using the Admin client directly.
>

Thanks for the explanation. That makes sense.

>
> I understand your concern that the interface proposed in the first approach
> may become too similar to the existing Admin interface. I’ll update the
> proposal by moving Ryanne’s previous suggestion to re-use the Admin
> interface and add configuration to accept a custom implementation.
>

+1.

>
> If you still feel this KIP should be closed but can understand its
> motivation I can close this one and create a new one.
>

I think it's reasonable to keep this one open and make the changes you talked 
about above.

regards,
Colin

>
> Thanks,
> Omnia
>
> On Fri, May 13, 2022 at 6:10 PM Colin McCabe  wrote:
>
>> On Wed, May 11, 2022, at 15:07, Omnia Ibrahim wrote:
>> > Hi Colin,
>> > I don't mind the idea of MM2 users implementing the AdminClient
>> interface.
>> > However, there're two disadvantages to this.
>> >
>> >1. Having around 70 methods definitions to have "NotImplemented" is
>> one
>> >downside, and keep up with these if the AdminClient interface changes.
>> >2. It makes it hard to list what admin functionality MM2 uses as MM2
>> >interactions with AdminClient in the codebase are in many places.
>> >
>> > I guess it's OK for MM2 users who want to build their admin client to
>> carry
>> > this burden, as I explained in my previous response to the discussion
>> > thread. And we can do some cleanup to the codebase to have all Admin
>> > interactions in MM2 in a utils class or something like that to make it
>> > easier to navigate what MM2 needs from the Admin interface.
>> >
>>
>> Hi Omnia,
>>
>> Anyone who wants to extend Kafka with proprietary tooling does need to
>> keep up with the Kafka API. We have done everything we can to make this
>> easier. We rigorously define what the API is through the KIP process, and
>> make it possible to extend by making it an interface rather than concrete
>> class. We also have a pretty lengthy deprecation process for these APIs.
>>
>> >
>> > Maybe I'm misunderstanding the use-case you're describing here. But it
>> >> seems to me that if you create a proxy that has the ability to do any
>> admin
>> >> operation, and give MM2 access to that proxy, the security model is the
>> >> same as just giving MM2 admin access. (Or it may be worse if the
>> sysadmin
>> >> doesn't know what this proxy is doing, and doesn't lock it down...)
>> >>
>> >
>> > MM2 runs with the assumption that it has
>> >
>> >- "CREATE" ACLs for topics on the source clusters to create
>> `heartbeat`
>> >topics.
>> >- "CREATE"  and "ALTER" ACLs to create topics, add partitions, update
>> >topics' config and topics' ACLs (in future, will also include group
>> ACLS as
>> >Mikael mentioned before in the thread) on the destination clusters.
>> >
>> > Most organisations have some resource management or federated solutions
>> > (some would even have a budget system as part of these systems) to manage
>> > Kafka resources, and these systems are usually the only application
>> allowed
>> > to initializing a client with "CREATE" and "ALTER" ACLs. They don't grant
>> > these ACLs to any other teams/groups/applications to create such a client
>> > outside these systems, so assuming MM2 can bypass these systems and use
>> the
>> > AdminClient directly to create/update resources isn't valid. This is the
>> > primary concern here.
>> >
>> > The KIP is trying to give MM2 more 

Re: [VOTE] KIP-835: Monitor KRaft Controller Quorum Health

[David Arthur]

Thanks José, the KIP looks good to me!

+1 binding

-David

On Mon, May 16, 2022 at 11:59 AM José Armando García Sancio
 wrote:

> Hi all,
>
> I would like to start a vote for KIP-835:
> https://cwiki.apache.org/confluence/x/0xShD
>
> Thanks you,
> -José
>


-- 
David Arthur

Re: CI on Kafka seems to be failing even on trunk branch

[Divij Vaidya]

Hi Lim

We have quite a few tests which are flaky and fail stochastically from time
to time.

I would recommend that you check the tests which are failing for your PR.
Search for JIRA for the test failures. If you spot a flaky test JIRA for
the failure associated with your PR and the stack trace mentioned in the
JIRA is same as yours, then add this information to your PR. This activity
will help the reviewers understand whether the test failure are due to your
changes or not.

Alternatively, another approach (which I don’t recommend) is keep adding
commits to the PR to trigger the full test run and hope one of them is
successful.

Please feel free to ask questions if you need help with finding the flaky
test.

On Mon, May 16, 2022 at 6:35 PM Lim Qingwei  wrote:

> Hi, I am a newbie.
>
> I noticed Jenkins builds are failing, is this expected?
> My PR has some failures, but I don't think they are relevant, how should I
> proceed?
>
> Can I still request for reviews?
>
> Kind regards
>
-- 
Divij Vaidya

Re: [DISCUSS] KIP-838 Simulate batching and compression

[Sergio Daniel Troiano]

Hi Divij,

First of all thanks for your time and dedication.

About point one:
You are right, the idea is to have "in real time" visibility of the way the
clients are using the service as that is translated into a lot of money
saving.
I agree with the further vision although I think we are still far away from
it :)

About the resource usage my idea is to be zero invasive, so taking a few MB
samples once every few hours will be more than enough to understand the
produced pattern, so in this case the CPU usage is only a cost for the
producer and consumer.
Worth to mention that the additional 3% extra usage while producing is
negligible compared to the gain of batching and compression but maybe that
discussion is not related to this KIP, that is a decision between the
cluster admin and the clients.

About the "auto tuning" that is a great idea, again I think it is very
ambitious for the scope of this KIP but if the core of this is properly
done then this can be used in the future.


About point two:
Below is detailed the benefits of bathing and compression :
- Reduction of network bandwidth while data is produced.
- Reduction of disk usage to store the data, less IO for read and write the
segments (supposing the message format has not to be converted)
- Reduction of network traffic while data is replicated.
- Reduction of network traffic while the data is consumer.

The script I propose will output the percentage of network traffic
reduction + the disk space saved.
- Batching will be recommended based on the parameters
$batching-window-time (ms) and $min-records-for-batching the idea is to
check the CreationTime of each batch, lets suppose we use:

batching-window-time = 300
min-records-for-batching = 30

* This means we want to check if at least we can batch together 30 records
in 300 ms, this could be in 2 batches or in 30 (one record per batch)
* If the batching is achievable then we jump the next check to simulate the
compression even if the compression is already applied as batching more
data will improve the compression ratio.
* Finally the payload ( a few MB are brought to memory in order to get its
current size, then it is
compressed and the difference is calculated.


As a side note I think if the classes are properly created this can be
reused in the future for a more "automagic" way of usage. Again I really
like the idea of allowing the cluster to configure the producers (maybe the
producer could have a parameter to allow this)

I did not enter into details about the code as I would like to know if the
idea worth it, I use this "solution" in the company I work and it saved us
a lot of money, for now we have just get the output from the dump-logs.sh
script in order to see the CreateTime and the compression type, this is
a first step but we can't yet simulate the compression.
So for now we reach our clients saying "there is a potential benefit of
cost reduction if you apply these changes in the producer"


I hope this help, please feel free to add more feedback

Best regards.
Sergio Troiano










On Mon, 16 May 2022 at 10:35, Divij Vaidya  wrote:

> Thank you for the KIP Sergio.
>
> High level thoughts:
> 1\ I understand that the idea here is to provide better visibility to the
> admins about potential improvements using compression and modifying batch
> size. I would take it a step further and say that we should be providing
> this visibility in a programmatic push based manner and make this system
> generic enough so that adding new "optimization rules" in the future is
> seamless. Perhaps, have a "diagnostic" mode in the cluster, which can be
> dynamically enabled. In such a mode, the cluster would run a set of
> "optimization" rules (at the cost of additional CPU cycles). One of such
> rules would be the compression rule you mentioned in your KIP. At the end
> of the diagnostic run, the generated report would contain a set of
> recommendations. To begin with, we can introduce this "diagnostic" as a
> one-time run by admin and later, enhance it further to be triggered
> periodically in the cluster automatically (with results being published via
> existing metric libraries). Even further down the line, this could lead to
> "auto-tuning" producer libraries based on recommendations from the server.
>
> KIP implementation specific comments/questions:
> 2\ Can you please add the algorithm that would be used to determine whether
> compression is recommended or not? I am assuming that the algorithm would
> take into account the factors impacting compression optimization such as
> CPU utilization, network bandwidth, decompression cost by the consumers
> etc.
> 3\ Can you please add the algorithm that would be used to determine whether
> batching is recommended?
>
>
> Divij Vaidya
>
>
>
> On Mon, May 16, 2022 at 8:42 AM Sergio Daniel Troiano
>  wrote:
>
> > Hey guys!
> >
> > I would like to start an early discussion on this:
> >
> >
> 

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[Niket Goel]

Hi all,

Thank you for the feedback on this. I have started a voting thread for this KIP 
here:
https://lists.apache.org/thread/bkb7gsbxpljh5qh014ztffq7bldjrb2x

Thanks
Niket Goel


From: Niket Goel 
Date: Thursday, May 12, 2022 at 5:25 PM
To: dev@kafka.apache.org 
Subject: Re: [DISCUSS] KIP-836: Addition of Information in 
DescribeQuorumResponse about Voter Lag
Appreciate the careful review Jose.!

Ack on 1 and 2. Will fix.

For number 3 (and I am using [1] as a reference for this discussion), I think 
the correct language to use would be:

"Whenever a new fetch request
comes in the replica's last caught up time is updated to the time of
this fetch request if it requests an offset greater than or equal to the 
leader's
current end offset"
Does that sound right now?

Although I think I will go ahead and rewrite the explanation in a way that is 
more understandable. Thanks for pointing this out.

Thanks

[1] 
https://github.com/apache/kafka/blob/fa59be4e770627cd34cef85986b58ad7f606928d/core/src/main/scala/kafka/cluster/Replica.scala#L97



On Thu, May 12, 2022 at 3:20 PM José Armando García Sancio 
 wrote:
Thanks for the Kafka improvement Niket.

1. For the fields `LastFetchTime` and `LastCaughtUpTime`, Kafka tends
to use the suffix "Timestamp" when the value is an absolute wall clock
value.

2. The method `result()` for the type `DescribeQuorumResult` returns
the type `DescribeQuorumResponseData`. The types generated from the
RPC JSON schema are internal to Kafka and not exposed to clients. For
the admin client we should use a different type that is explicitly
public. See `org.apache.kafka.client.admin.DescribeTopicsResult` for
an example.

3. The proposed section has his sentence "Whenever a new fetch request
comes in the replica's last caught up time is updated to the time of
the fetch request if it requests an offset greater than the leader's
current end offset." Did you mean "previous fetch time" instead of
"last caught up time"? What do you mean by "requests an offset greater
than the leader's current end offset.?" Excluding diverging logs the
follower fetch offset should never be greater than the leader LEO.

Thanks,
-José


--
- Niket

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

[Omnia Ibrahim]

Hi Colin,

Thanks for your reply.

This KIP doesn’t aim to solve any security concerns, but rather a conflict
of responsibilities within any Kafka ecosystem that includes MM2 and any
resource management solution. I’m not sure that was clear, so I’m concerned
about the motivation for your suggestion to close this KIP.

It is generally accepted that resource management should be centralized,
especially on the scale of mirroring N number of clusters. The point of
this KIP is that any sort of topic management / federate solution /
up-front capacity planning system will be at odds with MM2 if MM2 keeps
using the Admin client directly.

I understand your concern that the interface proposed in the first approach
may become too similar to the existing Admin interface. I’ll update the
proposal by moving Ryanne’s previous suggestion to re-use the Admin
interface and add configuration to accept a custom implementation.

If you still feel this KIP should be closed but can understand its
motivation I can close this one and create a new one.

Thanks,
Omnia

On Fri, May 13, 2022 at 6:10 PM Colin McCabe  wrote:

> On Wed, May 11, 2022, at 15:07, Omnia Ibrahim wrote:
> > Hi Colin,
> > I don't mind the idea of MM2 users implementing the AdminClient
> interface.
> > However, there're two disadvantages to this.
> >
> >1. Having around 70 methods definitions to have "NotImplemented" is
> one
> >downside, and keep up with these if the AdminClient interface changes.
> >2. It makes it hard to list what admin functionality MM2 uses as MM2
> >interactions with AdminClient in the codebase are in many places.
> >
> > I guess it's OK for MM2 users who want to build their admin client to
> carry
> > this burden, as I explained in my previous response to the discussion
> > thread. And we can do some cleanup to the codebase to have all Admin
> > interactions in MM2 in a utils class or something like that to make it
> > easier to navigate what MM2 needs from the Admin interface.
> >
>
> Hi Omnia,
>
> Anyone who wants to extend Kafka with proprietary tooling does need to
> keep up with the Kafka API. We have done everything we can to make this
> easier. We rigorously define what the API is through the KIP process, and
> make it possible to extend by making it an interface rather than concrete
> class. We also have a pretty lengthy deprecation process for these APIs.
>
> >
> > Maybe I'm misunderstanding the use-case you're describing here. But it
> >> seems to me that if you create a proxy that has the ability to do any
> admin
> >> operation, and give MM2 access to that proxy, the security model is the
> >> same as just giving MM2 admin access. (Or it may be worse if the
> sysadmin
> >> doesn't know what this proxy is doing, and doesn't lock it down...)
> >>
> >
> > MM2 runs with the assumption that it has
> >
> >- "CREATE" ACLs for topics on the source clusters to create
> `heartbeat`
> >topics.
> >- "CREATE"  and "ALTER" ACLs to create topics, add partitions, update
> >topics' config and topics' ACLs (in future, will also include group
> ACLS as
> >Mikael mentioned before in the thread) on the destination clusters.
> >
> > Most organisations have some resource management or federated solutions
> > (some would even have a budget system as part of these systems) to manage
> > Kafka resources, and these systems are usually the only application
> allowed
> > to initializing a client with "CREATE" and "ALTER" ACLs. They don't grant
> > these ACLs to any other teams/groups/applications to create such a client
> > outside these systems, so assuming MM2 can bypass these systems and use
> the
> > AdminClient directly to create/update resources isn't valid. This is the
> > primary concern here.
> >
> > The KIP is trying to give MM2 more flexibility to allow organisations to
> > integrate MM2 with their resource management system as they see fit
> without
> > forcing them to disable most MM2 features.
> >
> > Hope this make sense and clear it up.
> >
>
> The point I was trying to make is that there is no additional security
> here. If you have some agent that has all the permissions, and MM2 can talk
> to that agent and tell it what to do, then that is equivalent to just
> giving MM2 all the permissions. So while there may be other reasons to use
> this kind of agent-based architecture, added security isn't one.
>
> In any case, I think we should close this KIP since we already have an
> Admin API. There isn't a need to create a public API for admin operations.
>
> best,
> Colin
>
>
> >
> > On Wed, May 11, 2022 at 9:09 PM Colin McCabe  wrote:
> >
> >> Hi Omnia Ibrahim,
> >>
> >> I'm sorry, but I am -1 on adding competing Admin interfaces. This would
> >> create confusion and a heavier maintenance burden for the project.
> >>
> >> Since the org.apache.kafka.clients.admin.Admin interface is a Java
> >> interface, any third-party software that wants to insert its own
> >> implementation of the interface can do so 

[VOTE] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[Niket Goel]

Hi all,

I would like to start a vote for KIP-836:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Addition+of+Information+in+DescribeQuorumResponse+about+Voter+Lag

Thanks
Niket Goel

CI on Kafka seems to be failing even on trunk branch

[Lim Qingwei]

Hi, I am a newbie.

I noticed Jenkins builds are failing, is this expected?
My PR has some failures, but I don't think they are relevant, how should I
proceed?

Can I still request for reviews?

Kind regards

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[José Armando García Sancio]

Hi all,

Thanks for your feedback. I started a voting thread here:
https://lists.apache.org/thread/x1cy5otpf7mj9ytghnktr5hog27hdf7k

[VOTE] KIP-835: Monitor KRaft Controller Quorum Health

[José Armando García Sancio]

Hi all,

I would like to start a vote for KIP-835:
https://cwiki.apache.org/confluence/x/0xShD

Thanks you,
-José

[jira] [Resolved] (KAFKA-12635) Mirrormaker 2 offset sync is incorrect if the target partition is empty

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-12635.

Fix Version/s: 3.3.0
 Assignee: Mickael Maison  (was: Ning Zhang)
   Resolution: Fixed

> Mirrormaker 2 offset sync is incorrect if the target partition is empty
> ---
>
> Key: KAFKA-12635
> URL: https://issues.apache.org/jira/browse/KAFKA-12635
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.7.0
>Reporter: Frank Yi
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 3.3.0
>
>
> This bug occurs when using Mirrormaker with "sync.group.offsets.enabled = 
> true".
> If a source partition is empty, but the source consumer group's offset for 
> that partition is non-zero, then Mirrormaker sets the target consumer group's 
> offset for that partition to the literal, not translated, offset of the 
> source consumer group. This state can be reached if the source consumer group 
> consumed some records that were now deleted (like by a retention policy), or 
> if Mirrormaker replication is set to start at "latest". This bug causes the 
> target consumer group's lag for that partition to be negative and breaks 
> offset sync for that partition until lag is positive.
> The correct behavior when the source partition is empty would be to set the 
> target offset to the translated offset, not literal offset, which in this 
> case would always be 0. 
> Original email thread on this issue: 
> https://lists.apache.org/thread.html/r7c54ee5f57227367b911d4abffa72781772d8dd3b72d75eb65ee19f7%40%3Cusers.kafka.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #934

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

[Jim Hughes]

Hi all,


With 5 binding votes (John, Bruno, Sophie, Matthias, Bill) and 4
non-binding votes (Guozhang, Luke, Leah, Walker), the vote for KIP-834
passes!


Thanks all for the great discussion.

I have a PR up here: https://github.com/apache/kafka/pull/12161


Thanks in advance for feedback on the PR!


Cheers,


JIm

On Fri, May 13, 2022 at 12:04 PM Walker Carlson
 wrote:

> +1 from me (non-binding)
>
> Walker
>
> On Wed, May 11, 2022 at 12:36 PM Leah Thomas  >
> wrote:
>
> > Thanks Jim, great discussion. +1 from me (non-binding)
> >
> > Cheers,
> > Leah
> >
> > On Wed, May 11, 2022 at 10:14 AM Bill Bejeck  wrote:
> >
> > > Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > -Bill
> > >
> > > On Wed, May 11, 2022 at 9:36 AM Luke Chen  wrote:
> > >
> > > > Hi Jim,
> > > >
> > > > I'm +1. (please add some note in KIP about the stream resetting tool
> > > can't
> > > > be used in paused state)
> > > > Thanks for the KIP!
> > > >
> > > > Luke
> > > >
> > > > On Wed, May 11, 2022 at 9:09 AM Guozhang Wang 
> > > wrote:
> > > >
> > > > > Thanks Jim. +1 from me.
> > > > >
> > > > > On Tue, May 10, 2022 at 4:51 PM Matthias J. Sax 
> > > > wrote:
> > > > >
> > > > > > I had one minor question on the discuss thread. It's mainly about
> > > > > > clarifying and document the user contract. I am fine either way.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 5/10/22 12:32 PM, Sophie Blee-Goldman wrote:
> > > > > > > Thanks for the KIP! +1 (binding)
> > > > > > >
> > > > > > > On Tue, May 10, 2022, 12:24 PM Bruno Cadonna <
> cado...@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Thanks Jim,
> > > > > > >>
> > > > > > >> +1 (binding)
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Bruno
> > > > > > >>
> > > > > > >> On 10.05.22 21:19, John Roesler wrote:
> > > > > > >>> Thanks Jim,
> > > > > > >>>
> > > > > > >>> I’m +1 (binding)
> > > > > > >>>
> > > > > > >>> -John
> > > > > > >>>
> > > > > > >>> On Tue, May 10, 2022, at 14:05, Jim Hughes wrote:
> > > > > >  Hi all,
> > > > > > 
> > > > > >  I'm asking for a vote on KIP-834:
> > > > > > 
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> > > > > > 
> > > > > >  Thanks in advance!
> > > > > > 
> > > > > >  Jim
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-13906) Invalid replica state transition

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-13906:
---

 Summary: Invalid replica state transition
 Key: KAFKA-13906
 URL: https://issues.apache.org/jira/browse/KAFKA-13906
 Project: Kafka
  Issue Type: Bug
  Components: controller, core, replication
Affects Versions: 3.1.1, 3.0.1, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.0.2, 3.1.2, 
3.2.1
Reporter: Igor Soarez


The controller runs into an IllegalStateException when reacting to changes in 
broker membership status if there are topics that are pending deletion.

 

How to reproduce:
 # Setup cluster with 3 brokers
 # Create a topic with a partition being led by each broker and produce some 
data
 # Kill one of the brokers that is not the controller, and keep that broker down
 # Delete the topic
 # Restart the other broker that is not the controller

 

Logs and stacktrace:

{{[2022-05-16 11:53:25,482] ERROR [Controller id=1 epoch=1] Controller 1 epoch 
1 initiated state change of replica 3 for partition test-topic-2 from 
ReplicaDeletionSuccessful to ReplicaDeletionIneligible failed 
(state.change.logger)}}
{{java.lang.IllegalStateException: Replica 
[Topic=test-topic,Partition=2,Replica=3] should be in the 
OfflineReplica,ReplicaDeletionStarted states before moving to 
ReplicaDeletionIneligible state. Instead it is in ReplicaDeletionSuccessful 
state}}
{{        at 
kafka.controller.ZkReplicaStateMachine.logInvalidTransition(ReplicaStateMachine.scala:442)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2(ReplicaStateMachine.scala:164)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2$adapted(ReplicaStateMachine.scala:164)}}
{{        at scala.collection.immutable.List.foreach(List.scala:333)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.doHandleStateChanges(ReplicaStateMachine.scala:164)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2(ReplicaStateMachine.scala:112)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2$adapted(ReplicaStateMachine.scala:111)}}
{{        at 
kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)}}
{{        at 
scala.collection.immutable.HashMap.foreachEntry(HashMap.scala:1092)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:111)}}
{{        at 
kafka.controller.TopicDeletionManager.failReplicaDeletion(TopicDeletionManager.scala:157)}}
{{        at 
kafka.controller.KafkaController.onReplicasBecomeOffline(KafkaController.scala:638)}}
{{        at 
kafka.controller.KafkaController.onBrokerFailure(KafkaController.scala:599)}}
{{        at 
kafka.controller.KafkaController.processBrokerChange(KafkaController.scala:1623)}}
{{        at 
kafka.controller.KafkaController.process(KafkaController.scala:2534)}}
{{        at 
kafka.controller.QueuedEvent.process(ControllerEventManager.scala:52)}}
{{        at 
kafka.controller.ControllerEventManager$ControllerEventThread.process$1(ControllerEventManager.scala:130)}}
{{--}}
{{[2022-05-16 11:53:40,726] ERROR [Controller id=1 epoch=1] Controller 1 epoch 
1 initiated state change of replica 3 for partition test-topic-2 from 
ReplicaDeletionSuccessful to OnlineReplica failed (state.change.logger)}}
{{java.lang.IllegalStateException: Replica 
[Topic=test-topic,Partition=2,Replica=3] should be in the 
NewReplica,OnlineReplica,OfflineReplica,ReplicaDeletionIneligible states before 
moving to OnlineReplica state. Instead it is in ReplicaDeletionSuccessful 
state}}
{{        at 
kafka.controller.ZkReplicaStateMachine.logInvalidTransition(ReplicaStateMachine.scala:442)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2(ReplicaStateMachine.scala:164)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2$adapted(ReplicaStateMachine.scala:164)}}
{{        at scala.collection.immutable.List.foreach(List.scala:333)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.doHandleStateChanges(ReplicaStateMachine.scala:164)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2(ReplicaStateMachine.scala:112)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2$adapted(ReplicaStateMachine.scala:111)}}
{{        at 
kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)}}
{{        at 
scala.collection.immutable.HashMap.foreachEntry(HashMap.scala:1092)}}
{{        at 
kafka.controller.ZkReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:111)}}
{{        at 
kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:543)}}
{{        at 
kafka.controller.KafkaController.processBrokerChange(KafkaController.scala:1607)}}
{{        at 
kafka.controller.KafkaController.process(KafkaController.scala:2534)}}

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #933

[Apache Jenkins Server]

See 

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

[Bruno Cadonna]

Hi Divij,

The documentation that is shipped with Apache Kafka 3.2.0 cannot be 
changed anymore. To change that we would need to create a new release 
candidate which would be disproportionate for such a document change.


The PR you linked will be shipped in the docs of the next release. If 
someone wants to get this documentation change into the current Apache 
Kafka website before the next release, this person could open a PR 
against the kafka-site repo.


BTW, the doc line that was changed was not wrong. It was just a bit longer.

Best,
Bruno



On 16.05.22 11:16, Divij Vaidya wrote:

Perhaps, we want to merge in https://github.com/apache/kafka/pull/12164
before this release?

Divij Vaidya



On Mon, May 16, 2022 at 11:30 AM Bruno Cadonna  wrote:


Mickael,

Thanks a lot!

Totally, missed that!

Best,
Bruno

On 16.05.22 10:13, Mickael Maison wrote:

Hi Bruno,

The blog post looks good.
I just spotted a typo in the description of KIP-769, the endpoint is
"/connector-plugins" instead of "/connect-plugins" (3 times).

Thanks,
Mickael

On Sat, May 14, 2022 at 5:45 PM Federico Valeri 

wrote:


Hi Bruno,

+1 (non binding)

- Checked documentation and blog post
- Ran some client tests using the staged artifacts
- Verified the signing key and sha512 sum
- Built from source
- Ran unit and integration tests

Thanks
Fede

On Thu, May 12, 2022 at 12:03 PM Bruno Cadonna 

wrote:


Hi all,

Please review the blog post for the Apache Kafka 3.2.0 release:



https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache8


I will accept comments until Monday, May 16th EOD PT.

Best,
Bruno


On 06.05.22 14:52, 'David Jacot' via kafka-clients wrote:

Thanks for running the release, Bruno.

I performed the following validations:
* Verified all checksums and signatures.
* Built from source and ran unit tests.
* Ran the first quickstart steps for both ZK and KRaft.
* Spotchecked the doc and the Javadocs.

+1 (binding)

Best,
David

On Thu, May 5, 2022 at 10:36 AM Jakub Scholz  wrote:


+1 (non-binding).

I used the Scala 2.13 binaries and the staged Maven artifacts and ran
various tests with them. Thanks for doing the release.

Jakub

On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna 

wrote:



Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 3.2.0.

* log4j 1.x is replaced with reload4j (KAFKA-9366)
* StandardAuthorizer for KRaft (KIP-801)
* Send a hint to the partition leader to recover the partition

(KIP-704)

* Top-level error code field in DescribeLogDirsResponse (KIP-784)
* kafka-console-producer writes headers and null values (KIP-798 and
KIP-810)
* JoinGroupRequest and LeaveGroupRequest have a reason attached

(KIP-800)

* Static membership protocol lets the leader skip assignment

(KIP-814)

* Rack-aware standby task assignment in Kafka Streams (KIP-708)
* Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
* Connect APIs list all connector plugins and retrieve their
configuration (KIP-769)
* TimestampConverter SMT supports different unix time precisions

(KIP-808)

* Connect source tasks handle producer exceptions (KIP-779)


Release notes for the 3.2.0 release:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, May 10th, 9am PDT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/

* Maven artifacts to be voted upon:


https://repository.apache.org/content/groups/staging/org/apache/kafka/


* Javadoc:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/

* Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
https://github.com/apache/kafka/releases/tag/3.2.0-rc1

* Documentation:
https://kafka.apache.org/32/documentation.html

* Protocol:
https://kafka.apache.org/32/protocol.html

* Successful Jenkins builds for the 3.2 branch:
Unit/integration tests: I'll share a link once the builds complete
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/

/**

Thanks,
Bruno

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #932

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-831: Add metric for log recovery progress

[deng ziming]

Hello Luke, thanks for this KIP,

+1 (non-binding)

--
Best,
Ziming

> On May 16, 2022, at 3:59 PM, Divij Vaidya  wrote:
> 
> +1 (non-binding)
> 
> 
> Divij Vaidya
> 
> 
> 
> On Mon, May 16, 2022 at 10:12 AM Luke Chen  wrote:
> 
>> Hi all,
>> 
>> I'd like to start a vote on KIP to expose metrics for log recovery
>> progress. These metrics would let the admins have a way to monitor the log
>> recovery progress.
>> 
>> Details can be found here:
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress
>> 
>> Any feedback is appreciated.
>> 
>> Thank you.
>> Luke
>> 

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

[Divij Vaidya]

Perhaps, we want to merge in https://github.com/apache/kafka/pull/12164
before this release?

Divij Vaidya



On Mon, May 16, 2022 at 11:30 AM Bruno Cadonna  wrote:

> Mickael,
>
> Thanks a lot!
>
> Totally, missed that!
>
> Best,
> Bruno
>
> On 16.05.22 10:13, Mickael Maison wrote:
> > Hi Bruno,
> >
> > The blog post looks good.
> > I just spotted a typo in the description of KIP-769, the endpoint is
> > "/connector-plugins" instead of "/connect-plugins" (3 times).
> >
> > Thanks,
> > Mickael
> >
> > On Sat, May 14, 2022 at 5:45 PM Federico Valeri 
> wrote:
> >>
> >> Hi Bruno,
> >>
> >> +1 (non binding)
> >>
> >> - Checked documentation and blog post
> >> - Ran some client tests using the staged artifacts
> >> - Verified the signing key and sha512 sum
> >> - Built from source
> >> - Ran unit and integration tests
> >>
> >> Thanks
> >> Fede
> >>
> >> On Thu, May 12, 2022 at 12:03 PM Bruno Cadonna 
> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Please review the blog post for the Apache Kafka 3.2.0 release:
> >>>
> >>>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache8
> >>>
> >>> I will accept comments until Monday, May 16th EOD PT.
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>>
> >>> On 06.05.22 14:52, 'David Jacot' via kafka-clients wrote:
>  Thanks for running the release, Bruno.
> 
>  I performed the following validations:
>  * Verified all checksums and signatures.
>  * Built from source and ran unit tests.
>  * Ran the first quickstart steps for both ZK and KRaft.
>  * Spotchecked the doc and the Javadocs.
> 
>  +1 (binding)
> 
>  Best,
>  David
> 
>  On Thu, May 5, 2022 at 10:36 AM Jakub Scholz  wrote:
> >
> > +1 (non-binding).
> >
> > I used the Scala 2.13 binaries and the staged Maven artifacts and ran
> > various tests with them. Thanks for doing the release.
> >
> > Jakub
> >
> > On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna 
> wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the second candidate for release of Apache Kafka 3.2.0.
> >>
> >> * log4j 1.x is replaced with reload4j (KAFKA-9366)
> >> * StandardAuthorizer for KRaft (KIP-801)
> >> * Send a hint to the partition leader to recover the partition
> (KIP-704)
> >> * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> >> * kafka-console-producer writes headers and null values (KIP-798 and
> >> KIP-810)
> >> * JoinGroupRequest and LeaveGroupRequest have a reason attached
> (KIP-800)
> >> * Static membership protocol lets the leader skip assignment
> (KIP-814)
> >> * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> >> * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> >> * Connect APIs list all connector plugins and retrieve their
> >> configuration (KIP-769)
> >> * TimestampConverter SMT supports different unix time precisions
> (KIP-808)
> >> * Connect source tasks handle producer exceptions (KIP-779)
> >>
> >>
> >> Release notes for the 3.2.0 release:
> >> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Tuesday, May 10th, 9am PDT
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> https://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
> >>
> >> * Maven artifacts to be voted upon:
> >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc:
> >> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
> >>
> >> * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> >> https://github.com/apache/kafka/releases/tag/3.2.0-rc1
> >>
> >> * Documentation:
> >> https://kafka.apache.org/32/documentation.html
> >>
> >> * Protocol:
> >> https://kafka.apache.org/32/protocol.html
> >>
> >> * Successful Jenkins builds for the 3.2 branch:
> >> Unit/integration tests: I'll share a link once the builds complete
> >> System tests:
> >> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
> >>
> >> /**
> >>
> >> Thanks,
> >> Bruno
> >>
> 
>

Re: [ANNOUNCE] Apache Kafka 3.1.1

[Luke Chen]

Thanks Tom!

On Mon, May 16, 2022 at 4:31 PM Bruno Cadonna  wrote:

> Thanks Tom!
>
> On 16.05.22 10:25, David Jacot wrote:
> > Thanks for driving this release, Tom!
> >
> > On Fri, May 13, 2022 at 12:50 PM Tom Bentley 
> wrote:
> >>
> >> The Apache Kafka community is pleased to announce the release for
> >> Apache Kafka 3.1.1
> >>
> >> Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> >> since 3.1.0.
> >>
> >> All of the changes in this release can be found in the release notes:
> >> https://www.apache.org/dist/kafka/3.1.1/RELEASE_NOTES.html
> >>
> >>
> >> You can download the source and binary release (Scala 2.13 and 2.12)
> from:
> >> https://kafka.apache.org/downloads#3.1.1
> >>
> >>
> ---
> >>
> >>
> >> Apache Kafka is a distributed streaming platform with four core APIs:
> >>
> >>
> >> ** The Producer API allows an application to publish a stream of
> records to
> >> one or more Kafka topics.
> >>
> >> ** The Consumer API allows an application to subscribe to one or more
> >> topics and process the stream of records produced to them.
> >>
> >> ** The Streams API allows an application to act as a stream processor,
> >> consuming an input stream from one or more topics and producing an
> >> output stream to one or more output topics, effectively transforming the
> >> input streams to output streams.
> >>
> >> ** The Connector API allows building and running reusable producers or
> >> consumers that connect Kafka topics to existing applications or data
> >> systems. For example, a connector to a relational database might
> >> capture every change to a table.
> >>
> >>
> >> With these APIs, Kafka can be used for two broad classes of application:
> >>
> >> ** Building real-time streaming data pipelines that reliably get data
> >> between systems or applications.
> >>
> >> ** Building real-time streaming applications that transform or react
> >> to the streams of data.
> >>
> >>
> >> Apache Kafka is in use at large and small companies worldwide, including
> >> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> >> Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >>
> >> A big thank you for the following 28 contributors to this release!
> >>
> >> Bounkong Khamphousone, Chris Egerton, David Jacot, dengziming, Ed B,
> >> Edwin, Idan Kamara, Ismael Juma, Jason Gustafson, Jules Ivanic, Justin
> >> Lee, Justine Olshan, Konstantine Karantasis, Kvicii, Luke Chen, Marc
> >> Löhe, Matthias J. Sax, Mike Lothian, Philip Nee, prince-mahajan,
> >> Randall Hauch, Stanislav Vodetskyi, sunshujie1990, Tom Bentley,
> >> Vincent Jiang, Xiaobing Fang, Xiaoyue Xue, Yang Yu
> >>
> >> We welcome your help and feedback. For more information on how to
> >> report problems, and to get involved, visit the project website at
> >> https://kafka.apache.org/
> >>
> >> Thank you!
> >>
> >>
> >> Regards,
> >> Tom
>

[jira] [Created] (KAFKA-13905) Fix failing ServerShutdownTest.testCleanShutdownAfterFailedStartupDueToCorruptLogs

[Luke Chen (Jira)]

Luke Chen created KAFKA-13905:
-

 Summary: Fix failing 
ServerShutdownTest.testCleanShutdownAfterFailedStartupDueToCorruptLogs
 Key: KAFKA-13905
 URL: https://issues.apache.org/jira/browse/KAFKA-13905
 Project: Kafka
  Issue Type: Test
Reporter: Luke Chen
Assignee: dengziming


h3. Error Message

org.opentest4j.AssertionFailedError: Unexpected exception 
java.lang.RuntimeException: Received a fatal error while waiting for the broker 
to catch up with the current cluster metadata. ==> expected:  but was: 

h3. Stacktrace

org.opentest4j.AssertionFailedError: Unexpected exception 
java.lang.RuntimeException: Received a fatal error while waiting for the broker 
to catch up with the current cluster metadata. ==> expected:  but was: 
 at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at 
org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40) at 
org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210) at 
kafka.server.ServerShutdownTest.verifyCleanShutdownAfterFailedStartup(ServerShutdownTest.scala:198)
 at 
kafka.server.ServerShutdownTest.testCleanShutdownAfterFailedStartupDueToCorruptLogs(ServerShutdownTest.scala:168)



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Divij Vaidya]

Thanks for addressing my comments Mickael. No more comments/suggestions
from my side. LGTM.

Divij Vaidya



On Tue, May 10, 2022 at 6:10 PM Mickael Maison 
wrote:

> Hi Colin,
>
> Thanks for the suggestion.
>
> I guess there are pros and cons with both methods. In my mind I'm
> expecting these values to always be there in the long run (once people
> have upgraded to brokers that support this feature). So I thought
> having a primitive directly may be nicer to use in the future. On the
> other hand OptionalLong avoids exposing magic values to users and
> makes error cases (brokers that support the feature but hit an error)
> a bit nicer.
>
> We seem to already be using Optional fields in a few *Result classes
> so I'll follow your suggestion. As a bonus that also resolves the
> naming questions from Divij!
>
> I've updated the KIP accordingly.
>
> Thanks,
> Mickael
>
> On Wed, May 4, 2022 at 2:52 PM Igor Soarez  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for writing this KIP. This would be a very useful improvement!
> >
> > --
> > Igor
> >
> > On Thu, Apr 7, 2022, at 10:16 AM, Mickael Maison wrote:
> > > Hi,
> > >
> > > I wrote a small KIP to expose the total and usable space of logdirs
> > > via the DescribeLogDirs API:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > >
> > > Please take a look and let me know if you have any feedback.
> > >
> > > Thanks,
> > > Mickael
>

Re: [DISCUSS] KIP-838 Simulate batching and compression

[Divij Vaidya]

Thank you for the KIP Sergio.

High level thoughts:
1\ I understand that the idea here is to provide better visibility to the
admins about potential improvements using compression and modifying batch
size. I would take it a step further and say that we should be providing
this visibility in a programmatic push based manner and make this system
generic enough so that adding new "optimization rules" in the future is
seamless. Perhaps, have a "diagnostic" mode in the cluster, which can be
dynamically enabled. In such a mode, the cluster would run a set of
"optimization" rules (at the cost of additional CPU cycles). One of such
rules would be the compression rule you mentioned in your KIP. At the end
of the diagnostic run, the generated report would contain a set of
recommendations. To begin with, we can introduce this "diagnostic" as a
one-time run by admin and later, enhance it further to be triggered
periodically in the cluster automatically (with results being published via
existing metric libraries). Even further down the line, this could lead to
"auto-tuning" producer libraries based on recommendations from the server.

KIP implementation specific comments/questions:
2\ Can you please add the algorithm that would be used to determine whether
compression is recommended or not? I am assuming that the algorithm would
take into account the factors impacting compression optimization such as
CPU utilization, network bandwidth, decompression cost by the consumers etc.
3\ Can you please add the algorithm that would be used to determine whether
batching is recommended?


Divij Vaidya



On Mon, May 16, 2022 at 8:42 AM Sergio Daniel Troiano
 wrote:

> Hey guys!
>
> I would like to start an early discussion on this:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-838+Simulate+batching+and+compression
>
>
> Thanks!
>

Re: [ANNOUNCE] Apache Kafka 3.1.1

[Bruno Cadonna]

Thanks Tom!

On 16.05.22 10:25, David Jacot wrote:

Thanks for driving this release, Tom!

On Fri, May 13, 2022 at 12:50 PM Tom Bentley  wrote:


The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.1.1

Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
since 3.1.0.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.1.1/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.13 and 2.12) from:
https://kafka.apache.org/downloads#3.1.1

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 28 contributors to this release!

Bounkong Khamphousone, Chris Egerton, David Jacot, dengziming, Ed B,
Edwin, Idan Kamara, Ismael Juma, Jason Gustafson, Jules Ivanic, Justin
Lee, Justine Olshan, Konstantine Karantasis, Kvicii, Luke Chen, Marc
Löhe, Matthias J. Sax, Mike Lothian, Philip Nee, prince-mahajan,
Randall Hauch, Stanislav Vodetskyi, sunshujie1990, Tom Bentley,
Vincent Jiang, Xiaobing Fang, Xiaoyue Xue, Yang Yu

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,
Tom

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

[Bruno Cadonna]

Mickael,

Thanks a lot!

Totally, missed that!

Best,
Bruno

On 16.05.22 10:13, Mickael Maison wrote:

Hi Bruno,

The blog post looks good.
I just spotted a typo in the description of KIP-769, the endpoint is
"/connector-plugins" instead of "/connect-plugins" (3 times).

Thanks,
Mickael

On Sat, May 14, 2022 at 5:45 PM Federico Valeri  wrote:


Hi Bruno,

+1 (non binding)

- Checked documentation and blog post
- Ran some client tests using the staged artifacts
- Verified the signing key and sha512 sum
- Built from source
- Ran unit and integration tests

Thanks
Fede

On Thu, May 12, 2022 at 12:03 PM Bruno Cadonna  wrote:


Hi all,

Please review the blog post for the Apache Kafka 3.2.0 release:

https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache8

I will accept comments until Monday, May 16th EOD PT.

Best,
Bruno


On 06.05.22 14:52, 'David Jacot' via kafka-clients wrote:

Thanks for running the release, Bruno.

I performed the following validations:
* Verified all checksums and signatures.
* Built from source and ran unit tests.
* Ran the first quickstart steps for both ZK and KRaft.
* Spotchecked the doc and the Javadocs.

+1 (binding)

Best,
David

On Thu, May 5, 2022 at 10:36 AM Jakub Scholz  wrote:


+1 (non-binding).

I used the Scala 2.13 binaries and the staged Maven artifacts and ran
various tests with them. Thanks for doing the release.

Jakub

On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna  wrote:


Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 3.2.0.

* log4j 1.x is replaced with reload4j (KAFKA-9366)
* StandardAuthorizer for KRaft (KIP-801)
* Send a hint to the partition leader to recover the partition (KIP-704)
* Top-level error code field in DescribeLogDirsResponse (KIP-784)
* kafka-console-producer writes headers and null values (KIP-798 and
KIP-810)
* JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
* Static membership protocol lets the leader skip assignment (KIP-814)
* Rack-aware standby task assignment in Kafka Streams (KIP-708)
* Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
* Connect APIs list all connector plugins and retrieve their
configuration (KIP-769)
* TimestampConverter SMT supports different unix time precisions (KIP-808)
* Connect source tasks handle producer exceptions (KIP-779)


Release notes for the 3.2.0 release:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, May 10th, 9am PDT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/

* Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
https://github.com/apache/kafka/releases/tag/3.2.0-rc1

* Documentation:
https://kafka.apache.org/32/documentation.html

* Protocol:
https://kafka.apache.org/32/protocol.html

* Successful Jenkins builds for the 3.2 branch:
Unit/integration tests: I'll share a link once the builds complete
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/

/**

Thanks,
Bruno

Re: [ANNOUNCE] Apache Kafka 3.1.1

[David Jacot]

Thanks for driving this release, Tom!

On Fri, May 13, 2022 at 12:50 PM Tom Bentley  wrote:
>
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.1.1
>
> Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> since 3.1.0.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.1.1/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.13 and 2.12) from:
> https://kafka.apache.org/downloads#3.1.1
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 28 contributors to this release!
>
> Bounkong Khamphousone, Chris Egerton, David Jacot, dengziming, Ed B,
> Edwin, Idan Kamara, Ismael Juma, Jason Gustafson, Jules Ivanic, Justin
> Lee, Justine Olshan, Konstantine Karantasis, Kvicii, Luke Chen, Marc
> Löhe, Matthias J. Sax, Mike Lothian, Philip Nee, prince-mahajan,
> Randall Hauch, Stanislav Vodetskyi, sunshujie1990, Tom Bentley,
> Vincent Jiang, Xiaobing Fang, Xiaoyue Xue, Yang Yu
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
> Tom

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

[Mickael Maison]

Hi Bruno,

The blog post looks good.
I just spotted a typo in the description of KIP-769, the endpoint is
"/connector-plugins" instead of "/connect-plugins" (3 times).

Thanks,
Mickael

On Sat, May 14, 2022 at 5:45 PM Federico Valeri  wrote:
>
> Hi Bruno,
>
> +1 (non binding)
>
> - Checked documentation and blog post
> - Ran some client tests using the staged artifacts
> - Verified the signing key and sha512 sum
> - Built from source
> - Ran unit and integration tests
>
> Thanks
> Fede
>
> On Thu, May 12, 2022 at 12:03 PM Bruno Cadonna  wrote:
> >
> > Hi all,
> >
> > Please review the blog post for the Apache Kafka 3.2.0 release:
> >
> > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache8
> >
> > I will accept comments until Monday, May 16th EOD PT.
> >
> > Best,
> > Bruno
> >
> >
> > On 06.05.22 14:52, 'David Jacot' via kafka-clients wrote:
> > > Thanks for running the release, Bruno.
> > >
> > > I performed the following validations:
> > > * Verified all checksums and signatures.
> > > * Built from source and ran unit tests.
> > > * Ran the first quickstart steps for both ZK and KRaft.
> > > * Spotchecked the doc and the Javadocs.
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > David
> > >
> > > On Thu, May 5, 2022 at 10:36 AM Jakub Scholz  wrote:
> > >>
> > >> +1 (non-binding).
> > >>
> > >> I used the Scala 2.13 binaries and the staged Maven artifacts and ran
> > >> various tests with them. Thanks for doing the release.
> > >>
> > >> Jakub
> > >>
> > >> On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna  wrote:
> > >>
> > >>> Hello Kafka users, developers and client-developers,
> > >>>
> > >>> This is the second candidate for release of Apache Kafka 3.2.0.
> > >>>
> > >>> * log4j 1.x is replaced with reload4j (KAFKA-9366)
> > >>> * StandardAuthorizer for KRaft (KIP-801)
> > >>> * Send a hint to the partition leader to recover the partition (KIP-704)
> > >>> * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> > >>> * kafka-console-producer writes headers and null values (KIP-798 and
> > >>> KIP-810)
> > >>> * JoinGroupRequest and LeaveGroupRequest have a reason attached 
> > >>> (KIP-800)
> > >>> * Static membership protocol lets the leader skip assignment (KIP-814)
> > >>> * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> > >>> * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> > >>> * Connect APIs list all connector plugins and retrieve their
> > >>> configuration (KIP-769)
> > >>> * TimestampConverter SMT supports different unix time precisions 
> > >>> (KIP-808)
> > >>> * Connect source tasks handle producer exceptions (KIP-779)
> > >>>
> > >>>
> > >>> Release notes for the 3.2.0 release:
> > >>> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
> > >>>
> > >>> *** Please download, test and vote by Tuesday, May 10th, 9am PDT
> > >>>
> > >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >>> https://kafka.apache.org/KEYS
> > >>>
> > >>> * Release artifacts to be voted upon (source and binary):
> > >>> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
> > >>>
> > >>> * Maven artifacts to be voted upon:
> > >>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>>
> > >>> * Javadoc:
> > >>> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
> > >>>
> > >>> * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> > >>> https://github.com/apache/kafka/releases/tag/3.2.0-rc1
> > >>>
> > >>> * Documentation:
> > >>> https://kafka.apache.org/32/documentation.html
> > >>>
> > >>> * Protocol:
> > >>> https://kafka.apache.org/32/protocol.html
> > >>>
> > >>> * Successful Jenkins builds for the 3.2 branch:
> > >>> Unit/integration tests: I'll share a link once the builds complete
> > >>> System tests:
> > >>> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
> > >>>
> > >>> /**
> > >>>
> > >>> Thanks,
> > >>> Bruno
> > >>>
> > >

Re: [VOTE] KIP-831: Add metric for log recovery progress

[Divij Vaidya]

+1 (non-binding)


Divij Vaidya



On Mon, May 16, 2022 at 10:12 AM Luke Chen  wrote:

> Hi all,
>
> I'd like to start a vote on KIP to expose metrics for log recovery
> progress. These metrics would let the admins have a way to monitor the log
> recovery progress.
>
> Details can be found here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress
>
> Any feedback is appreciated.
>
> Thank you.
> Luke
>

[jira] [Resolved] (KAFKA-12703) Allow unencrypted private keys when using PEM files

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-12703.
-
Fix Version/s: 3.3.0
   Resolution: Fixed

> Allow unencrypted private keys when using PEM files
> ---
>
> Key: KAFKA-12703
> URL: https://issues.apache.org/jira/browse/KAFKA-12703
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 2.8.0
>Reporter: Brian Bascoy
>Priority: Major
> Fix For: 3.3.0
>
>
> Unencrypted PEM files seem to be internally [supported in the 
> codebase|https://github.com/apache/kafka/blob/a46beb9d29781e0709baf596601122f770a5fa31/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L509]
>  but setting an ssl.key.password is currently enforced by createKeystore (on 
> DefaultSslEngineFactory). I was unable to find a reason for this, so I wonder 
> if this limitation could simply be removed:
>  
> [https://github.com/pera/kafka/commit/8df2feab5fc6955cf8c89a7d132f05d8f562e16b]
>  
> Thanks



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[VOTE] KIP-831: Add metric for log recovery progress

[Luke Chen]

Hi all,

I'd like to start a vote on KIP to expose metrics for log recovery
progress. These metrics would let the admins have a way to monitor the log
recovery progress.

Details can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress

Any feedback is appreciated.

Thank you.
Luke