[jira] [Resolved] (KAFKA-17008) Update zookeeper to 3.8.4 or 3.9.2 to address CVE-2024-23944

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-17008.

Resolution: Duplicate

> Update zookeeper to 3.8.4 or 3.9.2 to address CVE-2024-23944
> 
>
> Key: KAFKA-17008
> URL: https://issues.apache.org/jira/browse/KAFKA-17008
> Project: Kafka
>  Issue Type: Bug
>Reporter: Arushi Helms
>Priority: Major
>
> Update zookeeper to 3.8.4 or 3.9.2 to address CVE-2024-23944.
> I could not find an existing ticket for this, if there one then please mark 
> this as duplicate. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17015) ContextualRecord#hashCode()、ProcessorRecordContext#hashCode() Should not be deprecated and throw an exception

[dujian0068 (Jira)]

dujian0068 created KAFKA-17015:
--

 Summary: 
ContextualRecord#hashCode()、ProcessorRecordContext#hashCode() Should not be 
deprecated and throw an exception
 Key: KAFKA-17015
 URL: https://issues.apache.org/jira/browse/KAFKA-17015
 Project: Kafka
  Issue Type: Improvement
Reporter: dujian0068


when review PR#16416。 I find function 
`ContextualRecord#hashCode()、ProcessorRecordContext#hashCode() ` be deprecated 
because they have a mutable attribute, which will cause the hashCode to change。 

I don't think that hashCode should be discarded just because it is mutable. 
HashCode is a very important property of an object. It just shouldn't be used 
for hash addressing, like ArayList

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3036

[Apache Jenkins Server]

See 

[VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

[Federico Valeri]

Hi all, I'd like to kick off a vote on KIP-1057.

Design doc:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool

Discussion thread:
https://lists.apache.org/thread/kxx1h4qwshgcjh4d5xzqltkx5mx9qopm

Thanks,
Fede

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Alieh Saeedi]

Hi all,


It is very exciting to see all the experts here raising very good points.

As we go further, we see more and more options to improve our solution,
which makes concluding and updating the KIP impossible.


The main suggestions so far are:

1. `flush` with `flushOptions` as input parameter

2. `commitTx` with `commitOptions` as input parameter

3. `send` must throw the exception


My concern about the 3rd suggestion:

1. Does the change cause any issue with backward compatibility?

2. The `send (bad record)` already transits the transaction to the error
state. No user, including Streams is able to transit the transaction back
from the error state. Do you mean we remove the
`maybeTransitionToErrorState(e)` from here

as well?

Cheers,
Alieh


On Fri, Jun 21, 2024 at 8:45 AM Andrew Schofield 
wrote:

> Hi Artem,
> I think you make a good point which is worth further consideration. If
> any of the existing methods is really ripe for a change here, it’s the
> send() that actually caused the problem. If that can be fixed so there are
> no situations in which a lurking error breaks a transaction, that might be
> the best.
>
> Thanks,
> Andrew
>
> > On 21 Jun 2024, at 01:51, Artem Livshits 
> wrote:
> >
> >> I thought we still wait for requests (and their errors) to come in and
> > could handle fatal errors appropriately.
> >
> > We do wait for requests, but my understanding is that when
> > commitTransaction("ignore send errors") we want to ignore errors.  So if
> we
> > do
> >
> > 1. send
> > 2. commitTransaction("ignore send errors")
> >
> > the commit will succeed.  You can look at the example in
> > https://issues.apache.org/jira/browse/KAFKA-9279 and just substitute
> > commitTransaction with commitTransaction("ignore send errors") and we get
> > the buggy behavior back :-).  Actually, this would potentially be even
> > worse than the original buggy behavior because the bug was that we
> ignored
> > errors that happened in the "send()" method itself, not necessarily the
> > ones that we got from the broker.
> >
> > Actually, looking at https://github.com/apache/kafka/pull/11508/files,
> > wouldn't a better solution be to just throw the error from the "send"
> > method itself, rather than trying to set it to be thrown during commit?
> > This way the example in https://issues.apache.org/jira/browse/KAFKA-9279
> > would be fixed, and at the same time it would give an opportunity for KS
> to
> > catch the error and ignore it if needed.  Not sure if we need a KIP for
> > that, just do a better fix of the old bug.
> >
> > -Artem
> >
> > On Thu, Jun 20, 2024 at 4:58 PM Justine Olshan
> 
> > wrote:
> >
> >> I'm a bit late to the party, but the discussion here looks reasonable.
> >> Moving the logic to a transactional method makes sense to me and makes
> me
> >> feel a bit better about keeping the complexity in the methods relevant
> to
> >> the issue.
> >>
> >>> One minor concern is that if we set "ignore send
> >> errors" (or whatever we decide to name it) option without explicit
> flush,
> >> it'll actually lead to broken behavior as the application won't be able
> to
> >> stop a commit from proceeding even on fatal errors.
> >>
> >> Is this with respect to the case a request is still inflight when we
> call
> >> commitTransaction? I thought we still wait for requests (and their
> errors)
> >> to come in and could handle fatal errors appropriately.
> >>
> >> Justine
> >>
> >> On Thu, Jun 20, 2024 at 4:32 PM Artem Livshits
> >>  wrote:
> >>
> >>> Hi Matthias (and other folks who suggested ideas),
> >>>
>  maybe `commitTransaction(CommitOptions)` or similar could be a good
> >> way
> >>> forward?
> >>>
> >>> I like this approach.  One minor concern is that if we set "ignore send
> >>> errors" (or whatever we decide to name it) option without explicit
> flush,
> >>> it'll actually lead to broken behavior as the application won't be able
> >> to
> >>> stop a commit from proceeding even on fatal errors.  But I guess we'll
> >> just
> >>> have to clearly document it.
> >>>
> >>> In some way we are basically adding a flag to optionally restore the
> >>> https://issues.apache.org/jira/browse/KAFKA-9279 bug, which is the
> >>> motivation for all these changes, anyway :-).
> >>>
> >>> -Artem
> >>>
> >>>
> >>> On Thu, Jun 20, 2024 at 2:18 PM Matthias J. Sax 
> >> wrote:
> >>>
>  Seems the option to use a config does not get a lot of support.
> 
>  So we need to go with some form or "overload / new method". I think
>  Chris' point about not coupling it to `flush()` but rather
>  `commitTransaction()` is actually a very good one; for non-tx case,
> the
>  different flush variants would not make sense.
> 
>  I also like Lianet's idea to pass in some "options" object, so maybe
>  `commitTransaction(CommitOptions)` or simila

[jira] [Created] (KAFKA-17016) Align the behavior of GaugeWrapper and MeterWrapper

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17016:
--

 Summary: Align the behavior of GaugeWrapper and MeterWrapper
 Key: KAFKA-17016
 URL: https://issues.apache.org/jira/browse/KAFKA-17016
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


MeterWrapper [0] can auto-recreate the removed metrics, but GaugeWrapper [1] 
can't. We should align the behavior in order to avoid potential bugs.


[0] 
https://github.com/apache/kafka/blob/9b5b434e2a6b2d5290ea403fc02859b1c523d8aa/core/src/main/scala/kafka/server/KafkaRequestHandler.scala#L261
[1] 
https://github.com/apache/kafka/blob/9b5b434e2a6b2d5290ea403fc02859b1c523d8aa/core/src/main/scala/kafka/server/KafkaRequestHandler.scala#L286



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1052: Enable warmup in producer performance test

[Luke Chen]

Hi Matt,

Thanks for the KIP!
I agree having the warm-up records could help correctly analyze the
performance.

Some questions:
1. It looks like we will add 2 more options to producer perf tool:
 - --warmup-records
 - --combined-summary

Is this correct?
In the "public interface" section, only 1 is mentioned. Could you update it?
Also, in the KIP, you use the word: "An option such as "--warmup-records"
should be added...", it sounds like it is not decided, yet.
I suggest you update to say, we will add "--warmup-records" option for
" to make it clear.

2. What will be the output containing both warm-up and steady-state results?
Could you give an example directly?

For better understanding, I'd suggest you refer to KIP-1057

to add some examples using `kafka-producer-perf-test.sh` with the new
option, to show what it will output.

Thank you.
Luke

On Fri, Jun 21, 2024 at 10:39 AM Welch, Matt  wrote:

> Hi Divij,
>
> Thanks for your response.  You raise some very important points.
> I've updated the KIP to clarify the changes discussed here.
>
> 1. I agree that warmup stats should be printed separately.  I see two
> cases here, both of which would have two summary lines printed at the end
> of the producer perf test.  In the first case, warmup-separate, the warmup
> stats are printed first as warmup-only, followed by a second print of the
> steady state performance. In the second case, warmup-combined, the first
> print would look identical to the summary line that's currently used and
> would reflect the "whole test", with a second summary print of the
> steady-state performance.  This second case would allow for users to
> compare what the test would have looked like without a warmup to results of
> the test with a warmup. Although I've been looking at the second case
> lately, I can see merits of both cases and would be happy to support the
> warmup-separate case if that's the preference of the community.  Regarding
> the JMX metrics accumulated by Kafka, we need to decide if we should reset
> the JMX metrics between the warmup and steady state. While I like the idea
> of having separate JMX buckets for warmup and steady state, these
> statistics are usually heavily windowed, so should naturally settle toward
> steady-state values after a warmup.
>
> 2. The total number of records sent by the producer and defined by
> '--num-records' MUST be strictly greater than the '--warmup-records' or an
> error will be thrown. Negative warmup records should similarly throw an
> error.  Specifying warmup-records of "0" should have behavior identical to
> the current implementation.
>
> 3.  You're correct that choosing the warmup duration can have a
> significant impact on the test output if care is not taken.  I've updated
> the proposed change to describe a simplistic process to choose how many
> warmup records to use.  Without understanding all the factors that go into
> a warmup, a user could run a test and watch the time series output of the
> producer test to determine when steady state has been reached and warmup
> has completed.  The number of records at which the producer hits steady
> state could then be used in subsequent tests. In practice, we find that 1
> minute is a good warmup for most cases, since aside from networking and
> storage initialization, even the JVM should be warmed up by then and using
> compiled code rather than interpreted byte code. This is more a heuristic,
> however, and measured latency and throughput of the system should be used
> to determine steady state.
>
> 4.  The current design has the user specifying the warmup records like
> they would specify the number of records for the test. While this is
> related to the throughput, it seemed a better option to have the user
> specify the number of records in the warmup, rather than some kind of
> duration which would be more complex to track. I completely agree with your
> concern of warmup affecting steady state, however, especially in short
> tests. With a warmup "removing" some of the high latency from steady state
> results, it could be tempting for users to run very short tests since they
> no longer need to wait long to achieve a repeatable steady-state result. I
> would consider this a case of insufficient warmup since Kafka could still
> be processing the warmup records as you mention. Best practice for warmup
> duration would be to hit steady state during the warmup and only then
> consider it a successful warmup. Our preferred process is to monitor
> producer latency until it hits steady state in a first test, then double
> that duration for the warmup in subsequent testing. One minute is usually
> sufficient. A problem does occur when using unlimited throughput since the
> user does not yet know how fast the producers will send so can't estimate
> warmup records. If the iterative testing described above is not 

Re: [VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

[Luke Chen]

Hi Fede,

Thanks for the KIP!
+1 from me.

Luke

On Fri, Jun 21, 2024 at 6:44 PM Federico Valeri 
wrote:

> Hi all, I'd like to kick off a vote on KIP-1057.
>
> Design doc:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool
>
> Discussion thread:
> https://lists.apache.org/thread/kxx1h4qwshgcjh4d5xzqltkx5mx9qopm
>
> Thanks,
> Fede
>

[jira] [Resolved] (KAFKA-17007) Fix SourceAndTarget#equal

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17007.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Fix SourceAndTarget#equal
> -
>
> Key: KAFKA-17007
> URL: https://issues.apache.org/jira/browse/KAFKA-17007
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
> Fix For: 3.9.0
>
>
> In reviewing https://github.com/apache/kafka/pull/16404 I noticed that 
> SourceAndTarget is a part of public class. Hence, we should fix the `equal` 
> that it does not check the class type [0].
> [0] 
> https://github.com/apache/kafka/blob/trunk/connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/SourceAndTarget.java#L49



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17012) Enable testMeasureCommitSyncDuration, testMeasureCommittedDurationOnFailure, testInvalidGroupMetadata, testMeasureCommittedDuration, testOffsetsForTimesTimeout, testBeg

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17012.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Enable testMeasureCommitSyncDuration, testMeasureCommittedDurationOnFailure, 
> testInvalidGroupMetadata, testMeasureCommittedDuration, 
> testOffsetsForTimesTimeout, testBeginningOffsetsTimeout and 
> testEndOffsetsTimeout for AsyncConsumer
> 
>
> Key: KAFKA-17012
> URL: https://issues.apache.org/jira/browse/KAFKA-17012
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: xuanzhang gong
>Priority: Minor
> Fix For: 3.9.0
>
>
> just test my fingers - it seems "testMeasureCommitSyncDuration, 
> testMeasureCommittedDurationOnFailure, testInvalidGroupMetadata, 
> testMeasureCommittedDuration, testOffsetsForTimesTimeout, 
> testBeginningOffsetsTimeout, testEndOffsetsTimeout" can work with 
> AsyncConsumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

[Kamal Chandraprakash]

Hi Federico,

Thanks for the KIP! +1 from me.

On Fri, Jun 21, 2024 at 5:47 PM Luke Chen  wrote:

> Hi Fede,
>
> Thanks for the KIP!
> +1 from me.
>
> Luke
>
> On Fri, Jun 21, 2024 at 6:44 PM Federico Valeri 
> wrote:
>
> > Hi all, I'd like to kick off a vote on KIP-1057.
> >
> > Design doc:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool
> >
> > Discussion thread:
> > https://lists.apache.org/thread/kxx1h4qwshgcjh4d5xzqltkx5mx9qopm
> >
> > Thanks,
> > Fede
> >
>

Re: [PR] Howly added to powered-by page [kafka-site]

[via GitHub]

mjsax merged PR #498:
URL: https://github.com/apache/kafka-site/pull/498


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Added Deep.BI - Predictive Analytics & AI Platform [kafka-site]

[via GitHub]

mjsax merged PR #500:
URL: https://github.com/apache/kafka-site/pull/500


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Create edenlab.svg [kafka-site]

[via GitHub]

mjsax commented on PR #502:
URL: https://github.com/apache/kafka-site/pull/502#issuecomment-2177278235

   @Anykry -- not sure if I understand this PR. The logo won't show up anywhere 
without a powered-by entry the html?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Create edenlab.svg [kafka-site]

[via GitHub]

mjsax commented on PR #502:
URL: https://github.com/apache/kafka-site/pull/502#issuecomment-2177279909

   Cf https://github.com/apache/kafka-site/pull/503#discussion_r1645236142
   
   Maybe we need to close this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Update powered-by.html [kafka-site]

[via GitHub]

mjsax commented on PR #503:
URL: https://github.com/apache/kafka-site/pull/503#issuecomment-2177280319

   Logo is added in a different PR: 
https://github.com/apache/kafka-site/pull/502
   
   But not sure if we need to close both.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] adding spitha on powered-by [kafka-site]

[via GitHub]

mjsax commented on PR #515:
URL: https://github.com/apache/kafka-site/pull/515#issuecomment-2177283797

   @VictorParkM -- happy to merge this, but it seems your web-page does not 
follow ASF guidelines with regard to respecting the Apache Kafka trademark.
   
   - https://www.apache.org/foundation/marks/ a
   - https://www.apache.org/foundation/marks/faq/#attribution
   
   Could you address this issue on your side?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Update powered-by.html [kafka-site]

[via GitHub]

mjsax commented on code in PR #503:
URL: https://github.com/apache/kafka-site/pull/503#discussion_r1645236142


##
powered-by.html:
##
@@ -763,6 +763,11 @@
 "logo": "nussknacker.svg",
 "logoBgColor": "#ff",
 "description": "Nussknacker is a low-code tool that allows IT teams to 
hand over decision algorithms to non-technical users. Apache Kafka is 
Nussknacker's primary input and output interface in streaming use cases - 
Nussknacker reads events from Kafka, applies decision algorithms and outputs 
actions to Kafka."
+}, {
+   "link": "https://www.edenlab.io";,

Review Comment:
   Seems this does not exist any longer...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Netstratum info updated [kafka-site]

[via GitHub]

mjsax commented on PR #534:
URL: https://github.com/apache/kafka-site/pull/534#issuecomment-2177285732

   @netstratum-labs -- can you rebase this PR to resolve merge conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Adding OREXES to powered-by section [kafka-site]

[via GitHub]

mjsax commented on PR #591:
URL: https://github.com/apache/kafka-site/pull/591#issuecomment-2177287181

   @siifuu -- Can you rebase this PR to resolve merge conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add Skillsoft to "Powered By" [kafka-site]

[via GitHub]

mjsax merged PR #601:
URL: https://github.com/apache/kafka-site/pull/601


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3037

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1052: Enable warmup in producer performance test

[Federico Valeri]

Hi Matt, I thanks for the KIP, this is a really useful feature.

In public interfaces, you say that the output won't change by default,
so I guess this means that --combined-summary will be false by
default, otherwise we would break the producer_performance system
test. Is that correct? I think a couple of command line snippets would
help here.

I think it would be great to also add a warmup phase to the consumer
perf tool, but this probably deserves it's own KIP as we don't have
latency stats there.


On Fri, Jun 21, 2024 at 2:16 PM Luke Chen  wrote:
>
> Hi Matt,
>
> Thanks for the KIP!
> I agree having the warm-up records could help correctly analyze the
> performance.
>
> Some questions:
> 1. It looks like we will add 2 more options to producer perf tool:
>  - --warmup-records
>  - --combined-summary
>
> Is this correct?
> In the "public interface" section, only 1 is mentioned. Could you update it?
> Also, in the KIP, you use the word: "An option such as "--warmup-records"
> should be added...", it sounds like it is not decided, yet.
> I suggest you update to say, we will add "--warmup-records" option for
> " to make it clear.
>
> 2. What will be the output containing both warm-up and steady-state results?
> Could you give an example directly?
>
> For better understanding, I'd suggest you refer to KIP-1057
> 
> to add some examples using `kafka-producer-perf-test.sh` with the new
> option, to show what it will output.
>
> Thank you.
> Luke
>
> On Fri, Jun 21, 2024 at 10:39 AM Welch, Matt  wrote:
>
> > Hi Divij,
> >
> > Thanks for your response.  You raise some very important points.
> > I've updated the KIP to clarify the changes discussed here.
> >
> > 1. I agree that warmup stats should be printed separately.  I see two
> > cases here, both of which would have two summary lines printed at the end
> > of the producer perf test.  In the first case, warmup-separate, the warmup
> > stats are printed first as warmup-only, followed by a second print of the
> > steady state performance. In the second case, warmup-combined, the first
> > print would look identical to the summary line that's currently used and
> > would reflect the "whole test", with a second summary print of the
> > steady-state performance.  This second case would allow for users to
> > compare what the test would have looked like without a warmup to results of
> > the test with a warmup. Although I've been looking at the second case
> > lately, I can see merits of both cases and would be happy to support the
> > warmup-separate case if that's the preference of the community.  Regarding
> > the JMX metrics accumulated by Kafka, we need to decide if we should reset
> > the JMX metrics between the warmup and steady state. While I like the idea
> > of having separate JMX buckets for warmup and steady state, these
> > statistics are usually heavily windowed, so should naturally settle toward
> > steady-state values after a warmup.
> >
> > 2. The total number of records sent by the producer and defined by
> > '--num-records' MUST be strictly greater than the '--warmup-records' or an
> > error will be thrown. Negative warmup records should similarly throw an
> > error.  Specifying warmup-records of "0" should have behavior identical to
> > the current implementation.
> >
> > 3.  You're correct that choosing the warmup duration can have a
> > significant impact on the test output if care is not taken.  I've updated
> > the proposed change to describe a simplistic process to choose how many
> > warmup records to use.  Without understanding all the factors that go into
> > a warmup, a user could run a test and watch the time series output of the
> > producer test to determine when steady state has been reached and warmup
> > has completed.  The number of records at which the producer hits steady
> > state could then be used in subsequent tests. In practice, we find that 1
> > minute is a good warmup for most cases, since aside from networking and
> > storage initialization, even the JVM should be warmed up by then and using
> > compiled code rather than interpreted byte code. This is more a heuristic,
> > however, and measured latency and throughput of the system should be used
> > to determine steady state.
> >
> > 4.  The current design has the user specifying the warmup records like
> > they would specify the number of records for the test. While this is
> > related to the throughput, it seemed a better option to have the user
> > specify the number of records in the warmup, rather than some kind of
> > duration which would be more complex to track. I completely agree with your
> > concern of warmup affecting steady state, however, especially in short
> > tests. With a warmup "removing" some of the high latency from steady state
> > results, it could be tempting for users to run very short tests since they
> > no longer need to wait

[jira] [Created] (KAFKA-17017) AsyncConsumer#unsubscribe does not clean the assigned partitions

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17017:
--

 Summary: AsyncConsumer#unsubscribe does not clean the assigned 
partitions
 Key: KAFKA-17017
 URL: https://issues.apache.org/jira/browse/KAFKA-17017
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


According to docs [0] `Consumer#unsubscribe` should clean both subscribed and 
assigned partitions. However, there are two issues about `AsyncConsumer`

1)  if we don't set group id,  `AsyncConsumer#unsubscribe`[1] will be no-op
2)  if we set group id, `AsyncConsumer` is always in `UNSUBSCRIBED` state and 
so `MembershipManagerImpl#leaveGroup` will be no-op

[0] 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L759
[1] 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1479
[2] 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java#L666



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17018) Metadata version 3.9 should return Fetch version 17

[Jira]

José Armando García Sancio created KAFKA-17018:
--

 Summary: Metadata version 3.9 should return Fetch version 17
 Key: KAFKA-17018
 URL: https://issues.apache.org/jira/browse/KAFKA-17018
 Project: Kafka
  Issue Type: Sub-task
Reporter: José Armando García Sancio
 Fix For: 3.9.0


MetadataVersion.fetchRequestVersion() should return 17 for version 3.9



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.7.1 RC2

[Jakub Scholz]

+1 (non-binding). I used the staged binaries (based on Scala 2.13) and
Maven artifacts to run my tests. All seems to work fine.

Thanks & Regards
Jakub

On Wed, Jun 19, 2024 at 10:55 AM Igor Soarez  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.7.1.
>
> This is a bugfix release with several fixes.
>
> Release notes for the 3.7.1 release:
> https://home.apache.org/~soarez/kafka-3.7.1-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday June 28, 11am UTC.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~soarez/kafka-3.7.1-rc2/
>
> * Docker release artifact to be voted upon:
> apache/kafka:3.7.1-rc2
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~soarez/kafka-3.7.1-rc2/javadoc/
>
> * Tag to be voted upon (off 3.7 branch) is the 3.7.1 tag:
> https://github.com/apache/kafka/releases/tag/3.7.1-rc2
>
> * Documentation:
> https://kafka.apache.org/37/documentation.html
>
> * Protocol:
> https://kafka.apache.org/37/protocol.html
>
> * Successful Jenkins builds for the 3.7 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/184/
> The latest test run includes some flaky tests, all of which were confirmed
> to pass locally.
>
> System tests:
> I don't have access to the Jenkins instance used for system tests in
> https://jenkins.confluent.io/job/system-test-kafka/job/3.7
> Luke has kindly shared results in the previous RC (thank you Luke!),
> and all issues have been addressed.
> If anyone with access is able to confirm the latest test results, please
> reply with details.
>
> * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> Docker Build Test Pipeline:
> https://github.com/apache/kafka/actions/runs/9572915509
>
> /**
>
> Thanks,
>
> --
> Igor Soarez
>

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Matthias J. Sax]

Not sure if we can change send and make it throw, given that send() is 
async? That is why users can register a `Callback` to begin with, right?


And Alieh's point about backward compatibility is also a fair concern.



Actually, this would potentially be even
worse than the original buggy behavior because the bug was that we ignored
errors that happened in the "send()" method itself, not necessarily the
ones that we got from the broker.


My understanding was that `commitTx(swallowError)` would only swallow 
`send()` errors, not errors about the actually commit. I agree that it 
would be very bad to swallow errors about the actual tx commit...


It's a fair question if this might be too subtle; to make it explicit, 
we could use `CommitOpions#ignorePendingSendErors()` [working name] to 
make it clear.



If we think it's too subtle to change commit to swallow send() errors, 
maybe going with `flush(FlushOptions)` would be clearer (and we can use 
`FlushOption#swallowSendErrorsForTransactions()` [working name] to be 
explicitly that the `FlushOption` for now has only an effect for TX).



Thoughts?


-Matthias



On 6/21/24 4:10 AM, Alieh Saeedi wrote:

Hi all,


It is very exciting to see all the experts here raising very good points.

As we go further, we see more and more options to improve our solution,
which makes concluding and updating the KIP impossible.


The main suggestions so far are:

1. `flush` with `flushOptions` as input parameter

2. `commitTx` with `commitOptions` as input parameter

3. `send` must throw the exception


My concern about the 3rd suggestion:

1. Does the change cause any issue with backward compatibility?

2. The `send (bad record)` already transits the transaction to the error
state. No user, including Streams is able to transit the transaction back
from the error state. Do you mean we remove the
`maybeTransitionToErrorState(e)` from here

as well?

Cheers,
Alieh


On Fri, Jun 21, 2024 at 8:45 AM Andrew Schofield 
wrote:


Hi Artem,
I think you make a good point which is worth further consideration. If
any of the existing methods is really ripe for a change here, it’s the
send() that actually caused the problem. If that can be fixed so there are
no situations in which a lurking error breaks a transaction, that might be
the best.

Thanks,
Andrew


On 21 Jun 2024, at 01:51, Artem Livshits 

wrote:



I thought we still wait for requests (and their errors) to come in and

could handle fatal errors appropriately.

We do wait for requests, but my understanding is that when
commitTransaction("ignore send errors") we want to ignore errors.  So if

we

do

1. send
2. commitTransaction("ignore send errors")

the commit will succeed.  You can look at the example in
https://issues.apache.org/jira/browse/KAFKA-9279 and just substitute
commitTransaction with commitTransaction("ignore send errors") and we get
the buggy behavior back :-).  Actually, this would potentially be even
worse than the original buggy behavior because the bug was that we

ignored

errors that happened in the "send()" method itself, not necessarily the
ones that we got from the broker.

Actually, looking at https://github.com/apache/kafka/pull/11508/files,
wouldn't a better solution be to just throw the error from the "send"
method itself, rather than trying to set it to be thrown during commit?
This way the example in https://issues.apache.org/jira/browse/KAFKA-9279
would be fixed, and at the same time it would give an opportunity for KS

to

catch the error and ignore it if needed.  Not sure if we need a KIP for
that, just do a better fix of the old bug.

-Artem

On Thu, Jun 20, 2024 at 4:58 PM Justine Olshan



wrote:


I'm a bit late to the party, but the discussion here looks reasonable.
Moving the logic to a transactional method makes sense to me and makes

me

feel a bit better about keeping the complexity in the methods relevant

to

the issue.


One minor concern is that if we set "ignore send

errors" (or whatever we decide to name it) option without explicit

flush,

it'll actually lead to broken behavior as the application won't be able

to

stop a commit from proceeding even on fatal errors.

Is this with respect to the case a request is still inflight when we

call

commitTransaction? I thought we still wait for requests (and their

errors)

to come in and could handle fatal errors appropriately.

Justine

On Thu, Jun 20, 2024 at 4:32 PM Artem Livshits
 wrote:


Hi Matthias (and other folks who suggested ideas),


maybe `commitTransaction(CommitOptions)` or similar could be a good

way

forward?

I like this approach.  One minor concern is that if we set "ignore send
errors" (or whatever we decide to name it) option without explicit

flush,

it'll actually lead to broken behavior as the application won't be able

to

stop a commit from procee

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Chris Egerton]

Hi Artem,

I think it'd make sense to throw directly from send whenever possible,
instead of returning an already-completed future. I didn't do that in my
bug fix to try to be conservative about breaking changes but this seems to
have caused its own set of headaches. It would be a little less flexible
though, since (as you note) it would still be impossible to commit
transactions after errors have been reported from brokers.

I'll leave it up to the Kafka Streams folks to decide if that flexibility
is required. If it is, then users could explicitly call flush() before
committing (and ignoring errors for) or aborting a transaction, if they
want to implement fine-grained error handling logic such as allowing errors
for a subset of topics to be ignored.

Hi Matthias,

Most of the time you're right and we can't throw from send(); however, in
this case (client-side record-too-large exception), the error is actually
noticed by the producer before send() returns, so it should be possible to
throw directly.

Cheers,

Chris

On Fri, Jun 21, 2024, 14:25 Matthias J. Sax  wrote:

> Not sure if we can change send and make it throw, given that send() is
> async? That is why users can register a `Callback` to begin with, right?
>
> And Alieh's point about backward compatibility is also a fair concern.
>
>
> > Actually, this would potentially be even
> > worse than the original buggy behavior because the bug was that we
> ignored
> > errors that happened in the "send()" method itself, not necessarily the
> > ones that we got from the broker.
>
> My understanding was that `commitTx(swallowError)` would only swallow
> `send()` errors, not errors about the actually commit. I agree that it
> would be very bad to swallow errors about the actual tx commit...
>
> It's a fair question if this might be too subtle; to make it explicit,
> we could use `CommitOpions#ignorePendingSendErors()` [working name] to
> make it clear.
>
>
> If we think it's too subtle to change commit to swallow send() errors,
> maybe going with `flush(FlushOptions)` would be clearer (and we can use
> `FlushOption#swallowSendErrorsForTransactions()` [working name] to be
> explicitly that the `FlushOption` for now has only an effect for TX).
>
>
> Thoughts?
>
>
> -Matthias
>
>
>
> On 6/21/24 4:10 AM, Alieh Saeedi wrote:
> > Hi all,
> >
> >
> > It is very exciting to see all the experts here raising very good points.
> >
> > As we go further, we see more and more options to improve our solution,
> > which makes concluding and updating the KIP impossible.
> >
> >
> > The main suggestions so far are:
> >
> > 1. `flush` with `flushOptions` as input parameter
> >
> > 2. `commitTx` with `commitOptions` as input parameter
> >
> > 3. `send` must throw the exception
> >
> >
> > My concern about the 3rd suggestion:
> >
> > 1. Does the change cause any issue with backward compatibility?
> >
> > 2. The `send (bad record)` already transits the transaction to the error
> > state. No user, including Streams is able to transit the transaction back
> > from the error state. Do you mean we remove the
> > `maybeTransitionToErrorState(e)` from here
> > <
> https://github.com/apache/kafka/blob/9b5b434e2a6b2d5290ea403fc02859b1c523d8aa/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L1112
> >
> > as well?
> >
> > Cheers,
> > Alieh
> >
> >
> > On Fri, Jun 21, 2024 at 8:45 AM Andrew Schofield <
> andrew_schofi...@live.com>
> > wrote:
> >
> >> Hi Artem,
> >> I think you make a good point which is worth further consideration. If
> >> any of the existing methods is really ripe for a change here, it’s the
> >> send() that actually caused the problem. If that can be fixed so there
> are
> >> no situations in which a lurking error breaks a transaction, that might
> be
> >> the best.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>> On 21 Jun 2024, at 01:51, Artem Livshits  .INVALID>
> >> wrote:
> >>>
>  I thought we still wait for requests (and their errors) to come in and
> >>> could handle fatal errors appropriately.
> >>>
> >>> We do wait for requests, but my understanding is that when
> >>> commitTransaction("ignore send errors") we want to ignore errors.  So
> if
> >> we
> >>> do
> >>>
> >>> 1. send
> >>> 2. commitTransaction("ignore send errors")
> >>>
> >>> the commit will succeed.  You can look at the example in
> >>> https://issues.apache.org/jira/browse/KAFKA-9279 and just substitute
> >>> commitTransaction with commitTransaction("ignore send errors") and we
> get
> >>> the buggy behavior back :-).  Actually, this would potentially be even
> >>> worse than the original buggy behavior because the bug was that we
> >> ignored
> >>> errors that happened in the "send()" method itself, not necessarily the
> >>> ones that we got from the broker.
> >>>
> >>> Actually, looking at https://github.com/apache/kafka/pull/11508/files,
> >>> wouldn't a better solution be to just throw the error from the "send"
> >>> method itself, rather than trying to set it to be thrown d

[jira] [Created] (KAFKA-17019) Producer TimeoutException should include root cause

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-17019:
---

 Summary: Producer TimeoutException should include root cause
 Key: KAFKA-17019
 URL: https://issues.apache.org/jira/browse/KAFKA-17019
 Project: Kafka
  Issue Type: Improvement
  Components: producer 
Reporter: Matthias J. Sax


With KAFKA-16965 we added a "root cause" to some `TimeoutException` throws by 
the producer. However, it's only a partial solution to address a specific issue.

We should consider to add the "root cause" for _all_ `TimeoutException` cases 
and unify/cleanup the code to get an holistic solution to the problem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Matthias J. Sax]

If we want to limit it to `RecordTooLargeException` throwing from 
`send()` directly make sense. Thanks for calling it out.


It's still a question of backward compatibility? `send()` does throw 
exceptions already, including generic `KafkaException`. Not sure if this 
helps with backward compatibility? Could we just add a new exception 
type (which is a child of `KafkaException`)?


The Producer JavaDocs are not totally explicit about it IMHO.

I think we could expect that some generic error handling path gets 
executed. For the TX-case, I would assume that a TX would be aborted if 
`send()` throws or that the producer would be `closed()`. Overall this 
might be safe?



It would be a little less flexible

though, since (as you note) it would still be impossible to commit
transactions after errors have been reported from brokers.


KS would still need a way to clear the error state of the producer. We 
could catch a `RecordTooLargeException` from `send()`, call the handler 
and let it decide what to do next. But if it does return `CONTINUE` to 
swallow the error and drop the poison pill record on the floor, we would 
want to move forward and commit the transaction.


But the question is: if we cannot add a record to the tx, does the 
producer need to go into error state? In the end, we did throw and 
inform the app that the record was _not_ added, and it's up to the app 
to decide what to do next?


If we report the error only via the `Callback` it's a different story, 
because the contract for this case is clearly specified on the JavaDocs:



When used as part of a transaction, it is not necessary to define a callback or 
check the result of the future
in order to detect errors from send. If any of the send calls 
failed with an irrecoverable error,
the final {@link #commitTransaction()} call will fail and throw the exception 
from the last failed send. When
this happens, your application should call {@link #abortTransaction()} to reset 
the state and continue to send
data.




-Matthias


On 6/21/24 11:42 AM, Chris Egerton wrote:

Hi Artem,

I think it'd make sense to throw directly from send whenever possible,
instead of returning an already-completed future. I didn't do that in my
bug fix to try to be conservative about breaking changes but this seems to
have caused its own set of headaches. It would be a little less flexible
though, since (as you note) it would still be impossible to commit
transactions after errors have been reported from brokers.

I'll leave it up to the Kafka Streams folks to decide if that flexibility
is required. If it is, then users could explicitly call flush() before
committing (and ignoring errors for) or aborting a transaction, if they
want to implement fine-grained error handling logic such as allowing errors
for a subset of topics to be ignored.

Hi Matthias,

Most of the time you're right and we can't throw from send(); however, in
this case (client-side record-too-large exception), the error is actually
noticed by the producer before send() returns, so it should be possible to
throw directly.

Cheers,

Chris

On Fri, Jun 21, 2024, 14:25 Matthias J. Sax  wrote:


Not sure if we can change send and make it throw, given that send() is
async? That is why users can register a `Callback` to begin with, right?

And Alieh's point about backward compatibility is also a fair concern.



Actually, this would potentially be even
worse than the original buggy behavior because the bug was that we

ignored

errors that happened in the "send()" method itself, not necessarily the
ones that we got from the broker.


My understanding was that `commitTx(swallowError)` would only swallow
`send()` errors, not errors about the actually commit. I agree that it
would be very bad to swallow errors about the actual tx commit...

It's a fair question if this might be too subtle; to make it explicit,
we could use `CommitOpions#ignorePendingSendErors()` [working name] to
make it clear.


If we think it's too subtle to change commit to swallow send() errors,
maybe going with `flush(FlushOptions)` would be clearer (and we can use
`FlushOption#swallowSendErrorsForTransactions()` [working name] to be
explicitly that the `FlushOption` for now has only an effect for TX).


Thoughts?


-Matthias



On 6/21/24 4:10 AM, Alieh Saeedi wrote:

Hi all,


It is very exciting to see all the experts here raising very good points.

As we go further, we see more and more options to improve our solution,
which makes concluding and updating the KIP impossible.


The main suggestions so far are:

1. `flush` with `flushOptions` as input parameter

2. `commitTx` with `commitOptions` as input parameter

3. `send` must throw the exception


My concern about the 3rd suggestion:

1. Does the change cause any issue with backward compatibility?

2. The `send (bad record)` already transits the transaction to the error
state. No user, including Streams is able to transit the transaction back
from the error state. Do you mean we 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #57

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-1022 Formatting and Updating Features

[Colin McCabe]

Hi all,

It seems that there was a bug in older versions of Kafka which caused 
deserialization problems when a supported feature range included 0. For 
example, the range for group.version of [0, 1] would be a problem in this 
situation.

This obviously makes supportedVersions kind of useless. Any feature that 
doesn't exist today is effectively at v0 today (v0 is equivalent to "off"). But 
if we can't declare that the server supports [0, 1] or similar, we can't 
declare that it supports the feature being off. Therefore, no rolling upgrades 
are possible.

We noticed this bug during the 3.8 release when we noticed problems in upgrade 
tests. As an addendum to KIP-1022, we're adding the following solution:

- There will be a new v4 for ApiVersionsRequest

- Clients that sent v4 will promise to correctly handle ranges that start with 
0, such as [0, 1]

- The server will simply leave out the features whose minimum supported value 
is 0 for clients that send v3 

- ApiVersionsRequest v4 will be supported in AK 3.9 and above. AK 3.8 will ship 
with ApiVersionsRequest v3 just as today.

thanks,
Colin


On Mon, Apr 15, 2024, at 11:01, Justine Olshan wrote:
> Hey folks,
>
> Thanks everyone! I will go ahead and call it.
> The KIP passes with the following +1 votes:
>
> - Andrew Schofield (non-binding)
> - David Jacot (binding)
> - José Armando García Sancio (binding)
> - Jun Rao (binding)
>
> Thanks again,
> Justine
>
> On Fri, Apr 12, 2024 at 11:16 AM Jun Rao  wrote:
>
>> Hi, Justine,
>>
>> Thanks for the KIP. +1
>>
>> Jun
>>
>> On Wed, Apr 10, 2024 at 9:13 AM José Armando García Sancio
>>  wrote:
>>
>> > Hi Justine,
>> >
>> > +1 (binding)
>> >
>> > Thanks for the improvement.
>> > --
>> > -José
>> >
>>

Re: [VOTE] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Thanks Colin,

This makes sense to me. Namely in the case where we perhaps don't want to
support version 0 anymore, we need the range to be able to not include 0.
(In other words, we can't assume 0 is supported)
It is unfortunate that this change is a bit tricky, but I think it's the
best option.

Can you clarify
> The server will simply leave out the features whose minimum supported
value is 0 for clients that send v3

For 3.8, I planned to set the 0s in the response to 1. Is it better to
suppress the zero version features in the response so we are consistent
between trunk and 3.8?

Thanks,
Justine

On Fri, Jun 21, 2024 at 4:34 PM Colin McCabe  wrote:

> Hi all,
>
> It seems that there was a bug in older versions of Kafka which caused
> deserialization problems when a supported feature range included 0. For
> example, the range for group.version of [0, 1] would be a problem in this
> situation.
>
> This obviously makes supportedVersions kind of useless. Any feature that
> doesn't exist today is effectively at v0 today (v0 is equivalent to "off").
> But if we can't declare that the server supports [0, 1] or similar, we
> can't declare that it supports the feature being off. Therefore, no rolling
> upgrades are possible.
>
> We noticed this bug during the 3.8 release when we noticed problems in
> upgrade tests. As an addendum to KIP-1022, we're adding the following
> solution:
>
> - There will be a new v4 for ApiVersionsRequest
>
> - Clients that sent v4 will promise to correctly handle ranges that start
> with 0, such as [0, 1]
>
> - The server will simply leave out the features whose minimum supported
> value is 0 for clients that send v3
>
> - ApiVersionsRequest v4 will be supported in AK 3.9 and above. AK 3.8 will
> ship with ApiVersionsRequest v3 just as today.
>
> thanks,
> Colin
>
>
> On Mon, Apr 15, 2024, at 11:01, Justine Olshan wrote:
> > Hey folks,
> >
> > Thanks everyone! I will go ahead and call it.
> > The KIP passes with the following +1 votes:
> >
> > - Andrew Schofield (non-binding)
> > - David Jacot (binding)
> > - José Armando García Sancio (binding)
> > - Jun Rao (binding)
> >
> > Thanks again,
> > Justine
> >
> > On Fri, Apr 12, 2024 at 11:16 AM Jun Rao 
> wrote:
> >
> >> Hi, Justine,
> >>
> >> Thanks for the KIP. +1
> >>
> >> Jun
> >>
> >> On Wed, Apr 10, 2024 at 9:13 AM José Armando García Sancio
> >>  wrote:
> >>
> >> > Hi Justine,
> >> >
> >> > +1 (binding)
> >> >
> >> > Thanks for the improvement.
> >> > --
> >> > -José
> >> >
> >>
>

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Kirk True]

Hi all,

The JavaDoc for Producer.flush() states:

Applications don't need to call this method for transactional producers, since 
the commitTransaction() will flush all buffered records before performing the 
commit. This ensures that all the send() calls made since the previous 
beginTransaction() are completed before the commit.

Because the documentation states that invoking flush() isn’t necessary, if we 
added a flag to flush(), wouldn’t we need to add that flag to 
commitTransaction() too? Otherwise users would have to know to explicitly 
change their code to invoke flush(). Also, flush() is used outside of the 
transactional Producer use case, adding a transaction-specific flag to that 
method is a bit confusing. 

As Chris mentioned, it makes intuitive sense to introduce behavior changes as 
close as possible to where it’s relevant, i.e. commitTransaction(). A 
“CommitOptions” class would follow the pattern used in the Admin client.

Thanks,
Kirk

> On Jun 21, 2024, at 12:28 PM, Matthias J. Sax  wrote:
> 
> If we want to limit it to `RecordTooLargeException` throwing from `send()` 
> directly make sense. Thanks for calling it out.
> 
> It's still a question of backward compatibility? `send()` does throw 
> exceptions already, including generic `KafkaException`. Not sure if this 
> helps with backward compatibility? Could we just add a new exception type 
> (which is a child of `KafkaException`)?
> 
> The Producer JavaDocs are not totally explicit about it IMHO.
> 
> I think we could expect that some generic error handling path gets executed. 
> For the TX-case, I would assume that a TX would be aborted if `send()` throws 
> or that the producer would be `closed()`. Overall this might be safe?
> 
>> It would be a little less flexible
>>> though, since (as you note) it would still be impossible to commit
>>> transactions after errors have been reported from brokers.
> 
> KS would still need a way to clear the error state of the producer. We could 
> catch a `RecordTooLargeException` from `send()`, call the handler and let it 
> decide what to do next. But if it does return `CONTINUE` to swallow the error 
> and drop the poison pill record on the floor, we would want to move forward 
> and commit the transaction.
> 
> But the question is: if we cannot add a record to the tx, does the producer 
> need to go into error state? In the end, we did throw and inform the app that 
> the record was _not_ added, and it's up to the app to decide what to do next?
> 
> If we report the error only via the `Callback` it's a different story, 
> because the contract for this case is clearly specified on the JavaDocs:
> 
>> When used as part of a transaction, it is not necessary to define a callback 
>> or check the result of the future
>> in order to detect errors from send. If any of the send calls 
>> failed with an irrecoverable error,
>> the final {@link #commitTransaction()} call will fail and throw the 
>> exception from the last failed send. When
>> this happens, your application should call {@link #abortTransaction()} to 
>> reset the state and continue to send
>> data.
> 
> 
> 
> -Matthias
> 
> 
> On 6/21/24 11:42 AM, Chris Egerton wrote:
>> Hi Artem,
>> I think it'd make sense to throw directly from send whenever possible,
>> instead of returning an already-completed future. I didn't do that in my
>> bug fix to try to be conservative about breaking changes but this seems to
>> have caused its own set of headaches. It would be a little less flexible
>> though, since (as you note) it would still be impossible to commit
>> transactions after errors have been reported from brokers.
>> I'll leave it up to the Kafka Streams folks to decide if that flexibility
>> is required. If it is, then users could explicitly call flush() before
>> committing (and ignoring errors for) or aborting a transaction, if they
>> want to implement fine-grained error handling logic such as allowing errors
>> for a subset of topics to be ignored.
>> Hi Matthias,
>> Most of the time you're right and we can't throw from send(); however, in
>> this case (client-side record-too-large exception), the error is actually
>> noticed by the producer before send() returns, so it should be possible to
>> throw directly.
>> Cheers,
>> Chris
>> On Fri, Jun 21, 2024, 14:25 Matthias J. Sax  wrote:
>>> Not sure if we can change send and make it throw, given that send() is
>>> async? That is why users can register a `Callback` to begin with, right?
>>> 
>>> And Alieh's point about backward compatibility is also a fair concern.
>>> 
>>> 
 Actually, this would potentially be even
 worse than the original buggy behavior because the bug was that we
>>> ignored
 errors that happened in the "send()" method itself, not necessarily the
 ones that we got from the broker.
>>> 
>>> My understanding was that `commitTx(swallowError)` would only swallow
>>> `send()` errors, not errors about the actually commit. I agree that it
>>> would be very bad to 

Re: [VOTE] KIP-1022 Formatting and Updating Features

[Colin McCabe]

I think it's better to suppress the response in v3. The issue with modifying it 
is that there may be scenarios where [1, 1] is the actual supported range, and 
we'd want to know that. But leaving out the feature should be OK for older 
clients (it will be the case with clients old enough to send a v0, v1, or v2 
ApiVersionsRequest anyway)

best,
Colin

On Fri, Jun 21, 2024, at 16:46, Justine Olshan wrote:
> Thanks Colin,
>
> This makes sense to me. Namely in the case where we perhaps don't want to
> support version 0 anymore, we need the range to be able to not include 0.
> (In other words, we can't assume 0 is supported)
> It is unfortunate that this change is a bit tricky, but I think it's the
> best option.
>
> Can you clarify
>> The server will simply leave out the features whose minimum supported
> value is 0 for clients that send v3
>
> For 3.8, I planned to set the 0s in the response to 1. Is it better to
> suppress the zero version features in the response so we are consistent
> between trunk and 3.8?
>
> Thanks,
> Justine
>
> On Fri, Jun 21, 2024 at 4:34 PM Colin McCabe  wrote:
>
>> Hi all,
>>
>> It seems that there was a bug in older versions of Kafka which caused
>> deserialization problems when a supported feature range included 0. For
>> example, the range for group.version of [0, 1] would be a problem in this
>> situation.
>>
>> This obviously makes supportedVersions kind of useless. Any feature that
>> doesn't exist today is effectively at v0 today (v0 is equivalent to "off").
>> But if we can't declare that the server supports [0, 1] or similar, we
>> can't declare that it supports the feature being off. Therefore, no rolling
>> upgrades are possible.
>>
>> We noticed this bug during the 3.8 release when we noticed problems in
>> upgrade tests. As an addendum to KIP-1022, we're adding the following
>> solution:
>>
>> - There will be a new v4 for ApiVersionsRequest
>>
>> - Clients that sent v4 will promise to correctly handle ranges that start
>> with 0, such as [0, 1]
>>
>> - The server will simply leave out the features whose minimum supported
>> value is 0 for clients that send v3
>>
>> - ApiVersionsRequest v4 will be supported in AK 3.9 and above. AK 3.8 will
>> ship with ApiVersionsRequest v3 just as today.
>>
>> thanks,
>> Colin
>>
>>
>> On Mon, Apr 15, 2024, at 11:01, Justine Olshan wrote:
>> > Hey folks,
>> >
>> > Thanks everyone! I will go ahead and call it.
>> > The KIP passes with the following +1 votes:
>> >
>> > - Andrew Schofield (non-binding)
>> > - David Jacot (binding)
>> > - José Armando García Sancio (binding)
>> > - Jun Rao (binding)
>> >
>> > Thanks again,
>> > Justine
>> >
>> > On Fri, Apr 12, 2024 at 11:16 AM Jun Rao 
>> wrote:
>> >
>> >> Hi, Justine,
>> >>
>> >> Thanks for the KIP. +1
>> >>
>> >> Jun
>> >>
>> >> On Wed, Apr 10, 2024 at 9:13 AM José Armando García Sancio
>> >>  wrote:
>> >>
>> >> > Hi Justine,
>> >> >
>> >> > +1 (binding)
>> >> >
>> >> > Thanks for the improvement.
>> >> > --
>> >> > -José
>> >> >
>> >>
>>

Re: [VOTE] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Ok makes sense. I will update my PR.

On Fri, Jun 21, 2024 at 5:09 PM Colin McCabe  wrote:

> I think it's better to suppress the response in v3. The issue with
> modifying it is that there may be scenarios where [1, 1] is the actual
> supported range, and we'd want to know that. But leaving out the feature
> should be OK for older clients (it will be the case with clients old enough
> to send a v0, v1, or v2 ApiVersionsRequest anyway)
>
> best,
> Colin
>
> On Fri, Jun 21, 2024, at 16:46, Justine Olshan wrote:
> > Thanks Colin,
> >
> > This makes sense to me. Namely in the case where we perhaps don't want to
> > support version 0 anymore, we need the range to be able to not include 0.
> > (In other words, we can't assume 0 is supported)
> > It is unfortunate that this change is a bit tricky, but I think it's the
> > best option.
> >
> > Can you clarify
> >> The server will simply leave out the features whose minimum supported
> > value is 0 for clients that send v3
> >
> > For 3.8, I planned to set the 0s in the response to 1. Is it better to
> > suppress the zero version features in the response so we are consistent
> > between trunk and 3.8?
> >
> > Thanks,
> > Justine
> >
> > On Fri, Jun 21, 2024 at 4:34 PM Colin McCabe  wrote:
> >
> >> Hi all,
> >>
> >> It seems that there was a bug in older versions of Kafka which caused
> >> deserialization problems when a supported feature range included 0. For
> >> example, the range for group.version of [0, 1] would be a problem in
> this
> >> situation.
> >>
> >> This obviously makes supportedVersions kind of useless. Any feature that
> >> doesn't exist today is effectively at v0 today (v0 is equivalent to
> "off").
> >> But if we can't declare that the server supports [0, 1] or similar, we
> >> can't declare that it supports the feature being off. Therefore, no
> rolling
> >> upgrades are possible.
> >>
> >> We noticed this bug during the 3.8 release when we noticed problems in
> >> upgrade tests. As an addendum to KIP-1022, we're adding the following
> >> solution:
> >>
> >> - There will be a new v4 for ApiVersionsRequest
> >>
> >> - Clients that sent v4 will promise to correctly handle ranges that
> start
> >> with 0, such as [0, 1]
> >>
> >> - The server will simply leave out the features whose minimum supported
> >> value is 0 for clients that send v3
> >>
> >> - ApiVersionsRequest v4 will be supported in AK 3.9 and above. AK 3.8
> will
> >> ship with ApiVersionsRequest v3 just as today.
> >>
> >> thanks,
> >> Colin
> >>
> >>
> >> On Mon, Apr 15, 2024, at 11:01, Justine Olshan wrote:
> >> > Hey folks,
> >> >
> >> > Thanks everyone! I will go ahead and call it.
> >> > The KIP passes with the following +1 votes:
> >> >
> >> > - Andrew Schofield (non-binding)
> >> > - David Jacot (binding)
> >> > - José Armando García Sancio (binding)
> >> > - Jun Rao (binding)
> >> >
> >> > Thanks again,
> >> > Justine
> >> >
> >> > On Fri, Apr 12, 2024 at 11:16 AM Jun Rao 
> >> wrote:
> >> >
> >> >> Hi, Justine,
> >> >>
> >> >> Thanks for the KIP. +1
> >> >>
> >> >> Jun
> >> >>
> >> >> On Wed, Apr 10, 2024 at 9:13 AM José Armando García Sancio
> >> >>  wrote:
> >> >>
> >> >> > Hi Justine,
> >> >> >
> >> >> > +1 (binding)
> >> >> >
> >> >> > Thanks for the improvement.
> >> >> > --
> >> >> > -José
> >> >> >
> >> >>
> >>
>

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Kirk True]

Hi Matthias,

> On Jun 21, 2024, at 12:28 PM, Matthias J. Sax  wrote:
> 
> If we want to limit it to `RecordTooLargeException` throwing from `send()` 
> directly make sense. Thanks for calling it out.
> 
> It's still a question of backward compatibility? `send()` does throw 
> exceptions already, including generic `KafkaException`. Not sure if this 
> helps with backward compatibility? Could we just add a new exception type 
> (which is a child of `KafkaException`)?
> 
> The Producer JavaDocs are not totally explicit about it IMHO.
> 
> I think we could expect that some generic error handling path gets executed. 
> For the TX-case, I would assume that a TX would be aborted if `send()` throws 
> or that the producer would be `closed()`. Overall this might be safe?
> 
>> It would be a little less flexible
>>> though, since (as you note) it would still be impossible to commit
>>> transactions after errors have been reported from brokers.
> 
> KS would still need a way to clear the error state of the producer. We could 
> catch a `RecordTooLargeException` from `send()`, call the handler and let it 
> decide what to do next. But if it does return `CONTINUE` to swallow the error 
> and drop the poison pill record on the floor, we would want to move forward 
> and commit the transaction.
> 
> But the question is: if we cannot add a record to the tx, does the producer 
> need to go into error state? In the end, we did throw and inform the app that 
> the record was _not_ added, and it's up to the app to decide what to do next?

That’s an excellent question…

Imagine the user’s application is writing information to a database instead of 
Kafka. If there’s a table with a CHAR(1) column and this SQL statement was 
attempted, what should happen?

INSERT INTO foo VALUES (’not sure’);

Yes, that DML would fail, sure, but would the user expect that the connection 
used by database library would get stuck in some kind of error state? A user 
would be able catch the error and either continue or abort, based on their 
business rules.

So I agree with what I believe you’re implying: we shouldn’t poison the 
Producer/TransactionManager on certain types of application-level errors in 
send().

Kirk

> If we report the error only via the `Callback` it's a different story, 
> because the contract for this case is clearly specified on the JavaDocs:
> 
>> When used as part of a transaction, it is not necessary to define a callback 
>> or check the result of the future
>> in order to detect errors from send. If any of the send calls 
>> failed with an irrecoverable error,
>> the final {@link #commitTransaction()} call will fail and throw the 
>> exception from the last failed send. When
>> this happens, your application should call {@link #abortTransaction()} to 
>> reset the state and continue to send
>> data.
> 
> 
> 
> -Matthias
> 
> 
> On 6/21/24 11:42 AM, Chris Egerton wrote:
>> Hi Artem,
>> I think it'd make sense to throw directly from send whenever possible,
>> instead of returning an already-completed future. I didn't do that in my
>> bug fix to try to be conservative about breaking changes but this seems to
>> have caused its own set of headaches. It would be a little less flexible
>> though, since (as you note) it would still be impossible to commit
>> transactions after errors have been reported from brokers.
>> I'll leave it up to the Kafka Streams folks to decide if that flexibility
>> is required. If it is, then users could explicitly call flush() before
>> committing (and ignoring errors for) or aborting a transaction, if they
>> want to implement fine-grained error handling logic such as allowing errors
>> for a subset of topics to be ignored.
>> Hi Matthias,
>> Most of the time you're right and we can't throw from send(); however, in
>> this case (client-side record-too-large exception), the error is actually
>> noticed by the producer before send() returns, so it should be possible to
>> throw directly.
>> Cheers,
>> Chris
>> On Fri, Jun 21, 2024, 14:25 Matthias J. Sax  wrote:
>>> Not sure if we can change send and make it throw, given that send() is
>>> async? That is why users can register a `Callback` to begin with, right?
>>> 
>>> And Alieh's point about backward compatibility is also a fair concern.
>>> 
>>> 
 Actually, this would potentially be even
 worse than the original buggy behavior because the bug was that we
>>> ignored
 errors that happened in the "send()" method itself, not necessarily the
 ones that we got from the broker.
>>> 
>>> My understanding was that `commitTx(swallowError)` would only swallow
>>> `send()` errors, not errors about the actually commit. I agree that it
>>> would be very bad to swallow errors about the actual tx commit...
>>> 
>>> It's a fair question if this might be too subtle; to make it explicit,
>>> we could use `CommitOpions#ignorePendingSendErors()` [working name] to
>>> make it clear.
>>> 
>>> 
>>> If we think it's too subtle to change commit to sw

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Matthias J. Sax]

Hey Kirk,

can you elaborate on a few points?


Otherwise users would have to know to explicitly change their code to invoke 
flush().


Why? If we would add an option to `flush(FlushOption)`, the existing 
`flush()` w/o any option will still be there, right? If we would really 
deprecate existing `flush()`, it would just mean that we would pass 
"default FlushOption" into an implicit flush (and yes, we would need to 
define what this would be).


I think there is no clear winner (as pointed out in my last reply), and 
both `flush(FlushOption)` and `commitTx(CommitOption)` has advantages 
and drawbacks. Guess we need to just agree on which tradeoff we want to 
move forward with?



Not sure if your database example is a 1:1 fit? I think, the better 
comparison would be:


BEGIN TX;
INSERT INTO foo VALUES (’a’);
INSERT INTO foo VALUES (’b’);
INSERT INTO foo VALUES (’c’);
INSERT INTO foo VALUES (’not sure’);

For this case, the full TX would roll back, right? I still think that 
allowing users to just skip over the last error, and continue the TX 
would be ok. In the end, we provide a programmatic API, and not a 
declarative one as SQL. Of course, default behavior would still be to 
put the producer into error state, and the user would need to call 
`abortTransaction()` to move forward.



-Matthias

On 6/21/24 5:26 PM, Kirk True wrote:

Hi Matthias,


On Jun 21, 2024, at 12:28 PM, Matthias J. Sax  wrote:

If we want to limit it to `RecordTooLargeException` throwing from `send()` 
directly make sense. Thanks for calling it out.

It's still a question of backward compatibility? `send()` does throw exceptions 
already, including generic `KafkaException`. Not sure if this helps with 
backward compatibility? Could we just add a new exception type (which is a 
child of `KafkaException`)?

The Producer JavaDocs are not totally explicit about it IMHO.

I think we could expect that some generic error handling path gets executed. 
For the TX-case, I would assume that a TX would be aborted if `send()` throws 
or that the producer would be `closed()`. Overall this might be safe?


It would be a little less flexible

though, since (as you note) it would still be impossible to commit
transactions after errors have been reported from brokers.


KS would still need a way to clear the error state of the producer. We could 
catch a `RecordTooLargeException` from `send()`, call the handler and let it 
decide what to do next. But if it does return `CONTINUE` to swallow the error 
and drop the poison pill record on the floor, we would want to move forward and 
commit the transaction.

But the question is: if we cannot add a record to the tx, does the producer 
need to go into error state? In the end, we did throw and inform the app that 
the record was _not_ added, and it's up to the app to decide what to do next?


That’s an excellent question…

Imagine the user’s application is writing information to a database instead of 
Kafka. If there’s a table with a CHAR(1) column and this SQL statement was 
attempted, what should happen?

 INSERT INTO foo VALUES (’not sure’);

Yes, that DML would fail, sure, but would the user expect that the connection 
used by database library would get stuck in some kind of error state? A user 
would be able catch the error and either continue or abort, based on their 
business rules.

So I agree with what I believe you’re implying: we shouldn’t poison the 
Producer/TransactionManager on certain types of application-level errors in 
send().

Kirk


If we report the error only via the `Callback` it's a different story, because 
the contract for this case is clearly specified on the JavaDocs:


When used as part of a transaction, it is not necessary to define a callback or 
check the result of the future
in order to detect errors from send. If any of the send calls 
failed with an irrecoverable error,
the final {@link #commitTransaction()} call will fail and throw the exception 
from the last failed send. When
this happens, your application should call {@link #abortTransaction()} to reset 
the state and continue to send
data.




-Matthias


On 6/21/24 11:42 AM, Chris Egerton wrote:

Hi Artem,
I think it'd make sense to throw directly from send whenever possible,
instead of returning an already-completed future. I didn't do that in my
bug fix to try to be conservative about breaking changes but this seems to
have caused its own set of headaches. It would be a little less flexible
though, since (as you note) it would still be impossible to commit
transactions after errors have been reported from brokers.
I'll leave it up to the Kafka Streams folks to decide if that flexibility
is required. If it is, then users could explicitly call flush() before
committing (and ignoring errors for) or aborting a transaction, if they
want to implement fine-grained error handling logic such as allowing errors
for a subset of topics to be ignored.
Hi Matthias,
Most of the time you're right and we can't throw from se

Re: [VOTE] KIP-1056 - Deprecate `default.` prefix for exception handler in StreamsConfig

[Matthias J. Sax]

+1 (binding) from my side.


I understand the concerns raised, but would personally move forward with 
this KIP as-is. If we cannot get three votes, it would get naturally 
discarded.



-Matthias

On 6/19/24 11:33 AM, Muralidhar Basani wrote:

Hi all,

I would like to call a vote on KIP-1056 - Deprecate `default.` exception
handler configs in StreamsConfig.

KIP -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1056%3A+Remove+%60default.%60+prefix+for+exception+handler+StreamsConfig

Discussion thread -
https://lists.apache.org/thread/khmznzjd5cd7wbdyt2bk776nx19gycxc

Thanks,
Murali

[jira] [Created] (KAFKA-17020) After enabling tiered storage, occasional residual logs are left in the replica

[Jianbin Chen (Jira)]

Jianbin Chen created KAFKA-17020:


 Summary: After enabling tiered storage, occasional residual logs 
are left in the replica
 Key: KAFKA-17020
 URL: https://issues.apache.org/jira/browse/KAFKA-17020
 Project: Kafka
  Issue Type: Wish
Affects Versions: 3.7.0
Reporter: Jianbin Chen


After enabling tiered storage, occasional residual logs are left in the replica.
Based on the observed phenomenon, the index values of the rolled-out logs 
generated by the replica and the leader are not the same. As a result, the logs 
uploaded to S3 at the same time do not include the corresponding log files on 
the replica side, making it impossible to delete the local logs.
[!https://private-user-images.githubusercontent.com/19943636/341939158-d0b87a7d-aca1-4700-b3e1-fceff0530c79.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwMzY1OTIsIm5iZiI6MTcxOTAzNjI5MiwicGF0aCI6Ii8xOTk0MzYzNi8zNDE5MzkxNTgtZDBiODdhN2QtYWNhMS00NzAwLWIzZTEtZmNlZmYwNTMwYzc5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjIyVDA2MDQ1MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY3ZDQ2OGIxMmE3OGI2Njc2YzdkNzkwMzlhNmM5MzAxNjY0MWZiMzA2ZjgwNzgzM2JlYTMxMzM4Njk1NGI5MDYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Sdsvwn0dUi_p1dG0W_AvQY6Iqeimy_UZ8VldKUS1Q0E!|https://private-user-images.githubusercontent.com/19943636/341939158-d0b87a7d-aca1-4700-b3e1-fceff0530c79.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwMzY1OTIsIm5iZiI6MTcxOTAzNjI5MiwicGF0aCI6Ii8xOTk0MzYzNi8zNDE5MzkxNTgtZDBiODdhN2QtYWNhMS00NzAwLWIzZTEtZmNlZmYwNTMwYzc5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjIyVDA2MDQ1MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY3ZDQ2OGIxMmE3OGI2Njc2YzdkNzkwMzlhNmM5MzAxNjY0MWZiMzA2ZjgwNzgzM2JlYTMxMzM4Njk1NGI5MDYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Sdsvwn0dUi_p1dG0W_AvQY6Iqeimy_UZ8VldKUS1Q0E]
leader config：
num.partitions=3
default.replication.factor=2
delete.topic.enable=true
auto.create.topics.enable=false
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=1
offsets.retention.minutes=4320
log.roll.ms=8640
log.local.retention.ms=60
log.segment.bytes=536870912
num.replica.fetchers=1
log.retention.ms=1581120
remote.log.manager.thread.pool.size=4
remote.log.reader.threads=4
remote.log.metadata.topic.replication.factor=3
remote.log.storage.system.enable=true
remote.log.metadata.topic.retention.ms=18000
rsm.config.fetch.chunk.cache.class=io.aiven.kafka.tieredstorage.fetch.cache.DiskChunkCache
rsm.config.fetch.chunk.cache.path=/data01/kafka-tiered-storage-cache
# Pick some cache size, 16 GiB here:
rsm.config.fetch.chunk.cache.size=34359738368
rsm.config.fetch.chunk.cache.retention.ms=120
# # # Prefetching size, 16 MiB here:
rsm.config.fetch.chunk.cache.prefetch.max.size=33554432
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
rsm.config.storage.s3.bucket.name=
rsm.config.storage.s3.region=us-west-1
rsm.config.storage.aws.secret.access.key=
rsm.config.storage.aws.access.key.id=
rsm.config.chunk.size=8388608
remote.log.storage.manager.class.path=/home/admin/core-0.0.1-SNAPSHOT/*:/home/admin/s3-0.0.1-SNAPSHOT/*
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.listener.name=PLAINTEXT
rsm.config.upload.rate.limit.bytes.per.second=31457280
replica config：
num.partitions=3
default.replication.factor=2
delete.topic.enable=true
auto.create.topics.enable=false
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=1
offsets.retention.minutes=4320
log.roll.ms=8640
log.local.retention.ms=60
log.segment.bytes=536870912
num.replica.fetchers=1
log.retention.ms=1581120
remote.log.manager.thread.pool.size=4
remote.log.reader.threads=4
remote.log.metadata.topic.replication.factor=3
remote.log.storage.system.enable=true
#remote.log.metadata.topic.retention.ms=18000
rsm.config.fetch.chunk.cache.class=io.aiven.kafka.tieredstorage.fetch.cache.DiskChunkCache
rsm.config.fetch.chunk.cache.path=/data01/kafka-tiered-storage-cache
# Pick some cache size, 16 GiB here:
rsm.config.fetch.chunk.cache.size=34359738368