[jira] [Created] (KAFKA-15245) Improve Tiered Storage Metrics

[Abhijeet Kumar (Jira)]

Abhijeet Kumar created KAFKA-15245:
--

 Summary: Improve Tiered Storage Metrics
 Key: KAFKA-15245
 URL: https://issues.apache.org/jira/browse/KAFKA-15245
 Project: Kafka
  Issue Type: Sub-task
Reporter: Abhijeet Kumar
Assignee: Abhijeet Kumar


Rename existing tiered storage metrics to remove ambiguity and add metrics for 
the RemoteIndexCache.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15005) Status of KafkaConnect task not correct

[Sagar Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sagar Rao resolved KAFKA-15005.
---
Resolution: Duplicate

> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 2.5.1, 3.0.0, 3.3.2
>Reporter: Yu Wang
>Priority: Major
>
> Our MM2 is running version 2.5.1.
> After a rebalance of our MM2 source tasks, we found there were several tasks 
> always in *UNASSIGNED* status, even the real tasks already started. 
> So we dump the payload of the status topic of Kafka Connect, and found the 
> last two status change is status *RUNNING* followed by status 
> {*}UNASSIGNED{*}.
> {code:java}
> LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 
> headerKeys: [] key: task-7 payload: 
> {"state":"RUNNING","trace":null,"worker_id":"worker-1","generation":437643}
> LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 
> headerKeys: [] key: task-7 payload: 
> {"state":"UNASSIGNED","trace":null,"worker_id":"worker-2","generation":437643}
>  {code}
> But usually, the RUNNING status should be appended after the UNASSIGNED, 
> because the worker coordinator will revoked the tasks before start new tasks.
> Then we checked the log of our MM2 worker. And found that, during that time, 
> there was a task that revoked on worker-2 and started on worker-1.
>  
> Worker-1
> {code:java}
> [2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, 
> groupId=__group] Starting task task-7 
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> [2023-05-15 09:24:45,951] INFO Creating task task-7 
> (org.apache.kafka.connect.runtime.Worker) {code}
> Worker-2
> {code:java}
> [2023-05-15 09:24:40,922] INFO Stopping task task-7 
> (org.apache.kafka.connect.runtime.Worker) {code}
>  
> So I think the incorrect status was caused by the revoked task finished later 
> than the new started task, which made the UNASSIGNED status append to that 
> status topic after the RUNNING status. 
>  
> After reading the code of DistributeHerder, I found that the task revoking is 
> running in a thread pool, the revoke operation just return after submit all 
> the callables. So I think even in the same worker, there is not a guarantee 
> that the revoke operation will always finish before the new tasks start.
> {code:java}
> for (final ConnectorTaskId taskId : tasks) {
> callables.add(getTaskStoppingCallable(taskId));
> }
> // The actual timeout for graceful task/connector stop is applied in worker's
> // stopAndAwaitTask/stopAndAwaitConnector methods.
> startAndStop(callables);
> log.info("Finished stopping tasks in preparation for rebalance"); {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Colin for the update on the mentioned KIPs.

~Satish.

On Mon, 24 Jul 2023 at 23:09, Colin McCabe  wrote:
>
> Hi Satish,
>
> I removed "KIP-866 ZooKeeper to KRaft Migration" from the list of pending 
> KIPs, since that one was shipped in 3.4. I added "KIP-868 Metadata 
> Transactions", since we are planning on implementing this in 3.6. (The KIP 
> was approved a while ago, but not yet shipped.)
>
> I also added "KIP-938: Add more metrics for measuring KRaft performance," 
> which is a new KIP we are implemeting in 3.6 (The JIRA is open now for 
> review.) Same for KIP-919 which is being voted on now.
>
> best,
> Colin
>
> On Mon, Jul 24, 2023, at 03:59, Satish Duggana wrote:
> > A gentle reminder on the KIP freeze date: 26th Jul. Please try to
> > close discussion/vote threads asap.
> >
> > Thanks,
> > Satish.
> >
> > On Sun, 23 Jul 2023 at 11:10, Satish Duggana  
> > wrote:
> >>
> >> Thanks Colov/Divij for adding the KIP-952. I do not think it is a
> >> blocker for 3.6.0. We can discuss the KIP in the respective thread.
> >>
> >> ~Satish.
> >>
> >> On Sun, 23 Jul 2023 at 07:21, Satish Duggana  
> >> wrote:
> >> >
> >> > Thanks ShunKang for the update. I added both the KIPs to the wiki.
> >> > Please feel free to update the wiki with the latest.
> >> >
> >> > ~Satish.
> >> >
> >> > On Sat, 22 Jul 2023 at 22:50, ShunKang Lin  
> >> > wrote:
> >> > >
> >> > > Hi Satish,
> >> > >
> >> > > Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory 
> >> > > copy" [1]
> >> > > and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory
> >> > > copying" [2] to the release plan?
> >> > > Thanks!
> >> > >
> >> > > [1]
> >> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> >> > >
> >> > > [2]
> >> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> >> > > I would appreciate a few more reviews on the pull request (
> >> > > https://github.com/apache/kafka/pull/12685) for KIP-872.
> >> > >
> >> > > Best,
> >> > > ShunKang
> >> > >
> >> > > Divij Vaidya  于2023年7月22日周六 20:06写道：
> >> > >
> >> > > > Hi Satish
> >> > > >
> >> > > > I have added the following accepted KIPs to the release plan. Please 
> >> > > > let me
> >> > > > know if something requires a change.
> >> > > >
> >> > > > Accepted KIPs -
> >> > > >
> >> > > > 1.
> >> > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
> >> > > >
> >> > > > 2.
> >> > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
> >> > > >
> >> > > >
> >> > > > Pending discussion KIP which I believe is important to be merged 
> >> > > > into 3.6 -
> >> > > >
> >> > > > 3.
> >> > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Divij Vaidya
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
> >> > > > 
> >> > > > wrote:
> >> > > >
> >> > > > > Thanks Hao for the update on KIP-925.
> >> > > > >
> >> > > > > On Thu, 20 Jul 2023 at 23:05, Hao Li  
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > Hi Satish,
> >> > > > > >
> >> > > > > > KIP-925 was accepted and currently under implementation. I just 
> >> > > > > > added
> >> > > > it
> >> > > > > to
> >> > > > > > the release plan.
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Hao
> >> > > > > >
> >> > > > > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
> >> > > > > > 
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hello!
> >> > > > > > >
> >> > > > > > > A couple of days ago I opened a new KIP for discussion - 
> >> > > > > > > KIP-952
> >> > > > [1]. I
> >> > > > > > > believe it might be a blocker for the release of 3.6.0, but I 
> >> > > > > > > wanted
> >> > > > to
> >> > > > > > > bring it up here for a decision on its urgency with the 
> >> > > > > > > current set
> >> > > > of
> >> > > > > > > people who are looking at Tiered Storage (Satish, Luke, Ivan, 
> >> > > > > > > Divij)
> >> > > > > given
> >> > > > > > > that the date for KIP freeze is fast approaching.
> >> > > > > > > What are your thoughts on the matter?
> >> > > > > > >
> >> > > > > > > [1]
> >> > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Christo
> >> > > > > > >
> >> > > > > > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana <
> >> > > > satish.dugg...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Yash,

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2033

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 393817 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 2h 52m 25s
231 actionable tasks: 124 executed, 107 up-to-date

Publishing build scan...
https://ge.apache.org/s/sxf72jqcza6hq


See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2023-07-24-23-08-35.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] junit
Recording test results

Gradle Test Run :streams:integrationTest > Gradle Test Executor 185 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 185 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED
[Checks API] No suitable checks publisher found.
[Pipeline] echo
Skipping Kafka Streams archetype test for Java 17
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQuerySpecificActivePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldFailWithIllegalArgumentExceptionWhenIQPartitionerReturnsMultiplePartitions()
 PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQueryAllStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > 
StoreQueryIntegrationTest > 

[jira] [Created] (KAFKA-15244) Connect PluginType.from(Class) result is incorrect when subclassing multiple plugin interfaces

[Greg Harris (Jira)]

Greg Harris created KAFKA-15244:
---

 Summary: Connect PluginType.from(Class) result is incorrect when 
subclassing multiple plugin interfaces
 Key: KAFKA-15244
 URL: https://issues.apache.org/jira/browse/KAFKA-15244
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Greg Harris
Assignee: Greg Harris
 Fix For: 3.6.0


The PluginType enum encodes the multiple different types of plugins that the 
plugin discovery infrastructure supports. The `PluginType.from(Class)` method 
is used to evaluate an arbitrary plugin subclass to determine what type it is.

This is flawed, because multiple plugin types are interfaces, and a single 
class may implement multiple plugin interfaces. This has the effect that when a 
plugin implements multiple classes, it appears multiple times, but each time it 
is listed under the same arbitrary type. For example, a PluginDesc 
object may contain the type PluginType.HEADER_CONVERTER, which is incorrect.

This method should be removed to force callers to consider that one plugin may 
implement multiple types.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2032

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14920) Address timeouts and out of order sequences

[Justine Olshan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justine Olshan resolved KAFKA-14920.

Resolution: Fixed

> Address timeouts and out of order sequences
> ---
>
> Key: KAFKA-14920
> URL: https://issues.apache.org/jira/browse/KAFKA-14920
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 3.6.0
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Blocker
>
> KAFKA-14844 showed the destructive nature of a timeout on the first produce 
> request for a topic partition (ie one that has no state in psm)
> Since we currently don't validate the first sequence (we will in part 2 of 
> kip-890), any transient error on the first produce can lead to out of order 
> sequences that never recover.
> Originally, KAFKA-14561 relied on the producer's retry mechanism for these 
> transient issues, but until that is fixed, we may need to retry from in the 
> AddPartitionsManager instead. We addressed the concurrent transactions, but 
> there are other errors like coordinator loading that we could run into and 
> see increased out of order issues.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

[Sophie Blee-Goldman]

Awesome summary (seriously) -- would you kindly offer your organizational
skills to every ongoing KIP from henceforth? We need you :P

A few answers/comments:

A2: I think there is a 3rd sub-option here, which is to leave
versioned-ness out of this KIP entirely, return only the non-versioned
stores for now, and then switch over to the versioned stores (only) when
the time comes to flip the switch on making them the default across the
DSL. This has the advantage of retaining the current behavior/semantics and
provides a clear way to transition smoothly in the future, since it seems
we will want to cut to all versioned state stores rather than offer users a
choice between versioned or non-versioned stores going forward (similar to
how we only offer timestamped stores presently, and have completely
replaced non-timestamped stores in the DSL.) . In both the timestamped and
versioned cases, the old stores are/will still be available or accessible
to users via the bare StoreSuppliers, should they somehow desire or require
the old store type. Ultimately, I think either this or option (1) would be
preferable, though I think it should be up to Matthias or anyone else
involved in the versioned stores feature to decide which approach makes
sense in the context of that feature's future plans.

A3: sounds reasonable to me

A5: Also sounds fine to me, though I'll let others chime in with/if they
have an alternative suggestion/preference. I guess the other contender
would be something like DSLStoreImpl or something like that?



On Mon, Jul 24, 2023 at 9:36 AM Almog Gavra  wrote:

> Lots of thoughts! Happy to see the thriving discussion on this post - lots
> going on so I'm trying to enumerate them to keep things organized (prefix
> "A" for "Almog" so we can use numbers in responses for other things ;P).
>
> A1. Question around closing implementation gaps (e.g. no rocks based
> suppression store)
> A2. Specifically how to handle Versioned stores
> A3. Configuration (new config/reuse old one + new one and ordering of
> config resolution)
> A4. Drawing a line between what is implementation detail (not exposed in
> API) and what is customizable (exposed in API)
> A5. Naming of StoreTypeSpec
> A6. Param classes in StoreBuilders
>
> --
>
> Here are summaries for where it seems each of these stands (trying not to
> add any additional opinion yet):
>
> A1. Sophie/Guozhang/Me (if I count hah!) seem to agree that it is worth
> pushing this KIP through independently of the implementation gaps as it
> doesn't seem to move the intermediate state further from the end state.
> Matthias originally had some concerns.
>
> A2. There's questions around whether versioned stores should be their own
> configurable option or whether they are an implementation detail that the
> StoreSpec should decide. It seems like the discussion is converging here,
> this should be an implementation detail.
>
> A3. Matthias/Guozhang prefer adding CUSTOM and then having an additional
> config to determine the implementation. Sophie prefers deprecating the old
> config. Guozhang additional suggested flipping the resolution order such
> that the old config is only respected in a DefaultStoreSpec implementation.
>
> A4. This KIP (or rather, the discussion on the KIP) blurs the lines between
> top level store types (KV, windowed, session) and the underlying
> implementation of them (timestamped, versioned, kv-list). It seems everyone
> is in alignment to ensure that we keep these two things separate and that
> the line is clearly delineated in the text of the KIP.
>
> A5. Guozhang and Sophie agree that the current name StoreTypeSpec is
> misleading, as it's really an implementation spec, not a type
> specification.
>
> A6. Agreement that this is an improvement, Sophie believes this can be done
> in a follow up but we should ensure our naming is good here so there's no
> conflicts down the line.
>
> -
>
> Ok, phew! Hopefully that covers it all! Now for my thoughts, hopefully
> wrapping up some of these discussions:
>
> A1.  @Matthias - are you still hesitant here? What would you need to be
> convinced here?
>
> A2. Since we are all in agreement that versioned stores should be an
> implementation detail, we have two options:
>
> (1) we can extend the KVParams to include a parameter that indicates
> whether or not the store should be versioned
> (2) we can introduce a configuration for whether or not to use a versioned
> store, and each implementation can choose to read/ignore that config
>
> Any preferences? (1) would align more closely with what we are doing today
> (they are a top-level concept in the Stores API).
>
> A3. I like Guozhang's suggestion of making the "primary" configuration to
> be the new one, and then having a "DefaultStoreTypeSpec" (using the old
> naming) which respects the old configuration. That seems to solve nearly
> all the concerns (e.g. it'd be easy to see where the enum is used 

[GitHub] [kafka-site] mjsax commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

[via GitHub]

mjsax commented on PR #528:
URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1648602694

   Follow up PR to get the latest comments back ported to `kafka.git`: 
https://github.com/apache/kafka/pull/14085


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mjsax merged pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

[via GitHub]

mjsax merged PR #528:
URL: https://github.com/apache/kafka-site/pull/528


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

[Sophie Blee-Goldman]

Thanks Shay! You and Matthias have convinced me, I'm happy with the current
proposal. I think once you make the minor
updates to the KIP document this will be ready for voting again.

Cheers,
Sophie

On Mon, Jul 24, 2023 at 8:26 AM Shay Lin  wrote:

> Hi Sophie and Matthias, thanks for your comments and replies.
>
> 1. Scope of change: KStreams only or KStreams/KTable
> I took some time to digest your points, looking through how KStreams
> triggers repartitions today. I noticed that `repartitionRequired`is a flag
> in KStreamImpl etc and not in KTableImpl etc. When I look further, in the
> case of KTable, instead of passing in a boolean flag, a repartition node `
> TableRepartitionMapNode` is directly created. I went back and referenced
> the two issue tickets KAFKA-10844 and KAFKA-4835, both requests were
> focused on KStreams, i.e. not to change the partition why the input streams
> are already correctly keyed. Is it possible that in the case of KTable,
> users always intend to repartition (change key) when they call on
> aggregate? -- (this was written before I saw Matthias's comment)
>
> Overall, based on the tickets, I see the benefit of doing a contained
> change focusing on KStreams, i.e. repartitionRequired, which would solve
> the pain points nicely. If we ran into similar complaints/optimization
> requests for KTable down the line, we can address them on top of this(let
> me know if we have these requests already, I might just be negligent).
>
> 2. API: markAsPartitioned() vs config
> If we go with the KStreams only scope, markAsPartition() is more
> adequate, i.e. maps nicely to repartitionRequired. There is a list of
> NamedOperations that may or may not trigger repartition based on its
> context(KStreams or KTable) which would make the implementation more
> confusing.
>
> 3. KIP documentation: Thanks for providing the links to previous KIPs. I
> will be adding the three use cases and javadoc. I will also document the
> risks when it relates to IQ and Join.
>
> Best,
> Shay
>
> On Fri, Jul 21, 2023 at 5:55 PM Matthias J. Sax  wrote:
>
> > I agree that it could easily be misused. There is a few Jira tickets for
> > cases when people want to "cancel" a repartition step. I would hope
> > those tickets are linked to the KIP (if not, we should do this, and
> > maybe even c those cases as motivation into the KIP itself)?
> >
> > It's always a tricky question to what extend we want to guide users, and
> > to what extend we need to give levers for advances case (and how to
> > design those levers...) It's for sure a good idea to call out "use with
> > case" in the JavaDocs for the new method.
> >
> >
> > -Matthias
> >
> > On 7/21/23 3:34 PM, Sophie Blee-Goldman wrote:
> > > I guess I felt a bit uneasy about how this could be used/abused while
> > > reading the KIP, but if we truly believe this is an advanced feature,
> I'm
> > > fine with the way things currently are. It doesn't feel like the best
> > API,
> > > but it does seem to be the best *possible* API given the way things
> are.
> > >
> > > W.r.t the KTable notes, that all makes sense to me. I just wanted to
> lay
> > > out all the potential cases to make sure we had our bases covered.
> > >
> > > I still think an example or two would help, but the only thing I will
> > > actually wait on before feeling comfortable enough to vote on this
> would
> > be
> > > a clear method signature (and maybe sample javadocs) in the "Public
> > > Interfaces" section.
> > >
> > > Thanks again for the KIP Shay! Hope I haven't dragged it out too much
> > >
> > > On Fri, Jul 21, 2023 at 3:19 PM Matthias J. Sax 
> > wrote:
> > >
> > >> Some thought about the API question.
> > >>
> > >>
> >  A. kstream.groupBy(...).aggregate(...)
> > >>
> > >> This can be re-writtten as
> > >>
> > >> kstream.selectKey(...)
> > >>  .markAsRepartitioned()
> > >>  .groupByKey()
> > >>  .aggregate()
> > >>
> > >> Given that `markAsRepartitoned` is an advanced feature, I think it
> would
> > >> be ok?
> > >>
> > >>
> >  B. ktable.groupBy(...).aggregate(...)
> > >>
> > >> For KTable aggregation, not sure how useful it would be? In the end,
> an
> > >> table aggregation does only make sense if we pick something from the
> > >> value, ie, we indeed change the key?
> > >>
> > >>
> >  C. kstream.selectKey(...).join(ktable)
> > >>
> > >> We can just insert a `markAsRepartitioned()` after `selectKey` to
> avoid
> > >> repartitioning of the left input KStream.
> > >>
> > >>
> > >>> KStream.selectKey(...).toTable().join(...)
> > >>
> > >> Not sure if I understand what you try to say with this example? In the
> > >> end, `selectKey(...).toTable()` would repartiton. If I know that one
> can
> > >> upsert directly, one inserts a `markAsRepartitioned()` in between.
> > >>
> > >>
> > >> In general, the use case seems to be that the key is not in the right
> > >> "format", or there is no key, but data was partitioned by a
> > >> value-attribute upstream and we just 

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Crispin Bernier]

I updated the wiki to include both results along with their average.

Thank you,
Crispin

On Mon, Jul 24, 2023 at 10:58 AM Ismael Juma  wrote:

> Hi Crispin,
>
> One additional question, the wiki says "The results are averaged over 2
> runs.". Can you please provide some measure of variance in the
> distribution, i.e. were both results similar to each other for both cases?
>
> Ismael
>
> On Fri, Jul 21, 2023 at 11:31 AM Ismael Juma  wrote:
>
> > Thanks for the update Crispin - very helpful to have actual performance
> > data. 2-5% for the default configuration is a bit on the low side for
> this
> > kind of proposal.
> >
> > Ismael
> >
> > On Thu, Jul 20, 2023 at 11:33 PM Crispin Bernier
> >  wrote:
> >
> >> Benchmark numbers have been posted on the KIP, please review.
> >>
> >> On 2023/07/20 13:03:00 Mayank Shekhar Narula wrote:
> >> > Jun
> >> >
> >> > Thanks for the feedback.
> >> >
> >> > Numbers to follow.
> >> >
> >> > If we don't plan to
> >> > > bump up the FetchResponse version, we could just remove the
> reference
> >> to
> >> > > version 16.
> >> >
> >> > Fixed.
> >> >
> >> > On Thu, Jul 20, 2023 at 1:28 AM Jun Rao 
> >> wrote:
> >> >
> >> > > Hi, Mayank,
> >> > >
> >> > > Thanks for the KIP. I agree with others that it would be useful to
> >> see the
> >> > > performance results. Otherwise, just a minor comment. If we don't
> >> plan to
> >> > > bump up the FetchResponse version, we could just remove the
> reference
> >> to
> >> > > version 16.
> >> > >
> >> > > Jun
> >> > >
> >> > > On Wed, Jul 19, 2023 at 2:31 PM Mayank Shekhar Narula <
> >> > > mayanks.nar...@gmail.com> wrote:
> >> > >
> >> > > > Luke
> >> > > >
> >> > > > Thanks for the interest in the KIP.
> >> > > >
> >> > > > But what if the consumer was fetching from the follower?
> >> > > >
> >> > > > We already include `PreferredReadReplica` in the fetch response.
> >> > > > > Should we put the node info of PreferredReadReplica under this
> >> case,
> >> > > > > instead of the leader's info?
> >> > > > >
> >> > > >
> >> > > > PreferredReadReplica is the decided on the leader. Looking at the
> >> Java
> >> > > > client code, AbstractFetch::selectReadReplica, first fetch request
> >> goes
> >> > > to
> >> > > > Leader of the partition -> Sends back PreferredReadReplica -> Next
> >> fetch
> >> > > > uses PreferredReadReplica. So as long as leader is available,
> >> > > > PreferredReadReplica would be found in subsequent fetches.
> >> > > >
> >> > > > Also, under this case, should we include the leader's info in the
> >> > > response?
> >> > > >
> >> > > >
> >> > > > In this case, I think the follower would fail the fetch if it
> knows
> >> a
> >> > > > different leader. If the follower knows a newer leader, it would
> >> return
> >> > > new
> >> > > > leader information in the response, for the client to act on.
> >> > > >
> >> > > >
> >> > > > Will we include the leader/node info in the response when having
> >> > > > > `UNKNOWN_LEADER_EPOCH` error?
> >> > > >
> >> > > >
> >> > > > My understanding is UNKNOWN_LEADER_EPOCH when a request from a
> >> client
> >> > > has a
> >> > > > newer epoch than the broker. So the client is already up to date
> on
> >> new
> >> > > > leader information, it's the broker that has the catching up to
> do.
> >> I
> >> > > think
> >> > > > there might be some optimisations to make sure the broker
> refreshes
> >> its
> >> > > > metadata quickly, so it can quickly recover to handle requests
> that
> >> > > > previously returned UNKNOWN_LEADER_EPOCH. But this work is outside
> >> the
> >> > > > scope of this KIP, as for now this KIP focusses on client-side
> >> > > > optimisations.
> >> > > >
> >> > > > Mayank
> >> > > >
> >> > > > On Tue, Jul 18, 2023 at 8:51 AM Luke Chen 
> wrote:
> >> > > >
> >> > > > > Hi Mayank,
> >> > > > >
> >> > > > > Thanks for the KIP!
> >> > > > >
> >> > > > > Some questions:
> >> > > > > 1. I can see most of the cases we only care about consumer fetch
> >> from
> >> > > the
> >> > > > > leader.
> >> > > > > But what if the consumer was fetching from the follower?
> >> > > > > We already include `PreferredReadReplica` in the fetch response.
> >> > > > > Should we put the node info of PreferredReadReplica under this
> >> case,
> >> > > > > instead of the leader's info?
> >> > > > > Also, under this case, should we include the leader's info in
> the
> >> > > > response?
> >> > > > >
> >> > > > > 2. Will we include the leader/node info in the response when
> >> having
> >> > > > > `UNKNOWN_LEADER_EPOCH` error?
> >> > > > > I think it's fine we ignore the `UNKNOWN_LEADER_EPOCH` error
> >> since when
> >> > > > > this happens, the node might have some error which should
> refresh
> >> the
> >> > > > > metadata. On the other hand, it might also be good if we can
> heal
> >> the
> >> > > > node
> >> > > > > soon to do produce/consume works.
> >> > > > >
> >> > > > >
> >> > > > > Thank you.
> >> > > > > Luke
> >> > > > >
> >> > > > > On Tue, Jul 18, 2023 at 2:00 AM Philip Nee 
> >> > > 

[DISCUSS] KIP-955: Add stream-table join on foreign key

[Igor Fomenko]

Hello developers of the Kafka Streams,

I would like to start discussion on KIP-955: Add stream-table join on
foreign key

This KIP proposes the new API to join KStrem with KTable based on foreign
key relation.
Ths KIP was inspired by one of my former projects to integrate RDBMS
databases with data consumers using Change Data Capture and Kafka.
If we had the capability in Kafka Stream to join KStream with KTable on
foreign key this would simplify our implementation significantly.

Looking forward to your feedback and discussion.

Regards,

Igor

[VOTE] KIP-951: Leader discovery optimisations for the client

[Mayank Shekhar Narula]

Hi all

Opening the voting process for this KIP, given the deadline for KIP
acceptance for 3.6 is impending. Open to keep the discussion going if the
community needs more time to converge.

KIP -> https://cwiki.apache.org/confluence/x/F4uzDw

Thanks for participating in the discussion so far!

-- 
Regards,
Mayank Shekhar Narula

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Mayank Shekhar Narula]

David,

We never backport new features to old releases. This new feature will be
> only available from 3.6 (or 3.7) onwards for both client and server.


Good to know. I think that makes the argument for bumping the version even
stronger.

On Mon, Jul 24, 2023 at 5:01 PM David Jacot 
wrote:

> Hi Mayank,
>
> We never backport new features to old releases. This new feature will be
> only available from 3.6 (or 3.7) onwards for both client and server.
>
> Best,
> David
>
> On Mon, Jul 24, 2023 at 5:20 PM Mayank Shekhar Narula <
> mayanks.nar...@gmail.com> wrote:
>
> > Thanks Jose/David/Ismael for your inputs.
> >
> > Not bumping the version, would require both broker & client to backport
> > changes. Especially for FetchResponse, as backporting would have to be
> done
> > all the way back to 3.1, so this effort isn't trivial, and was originally
> > underestimated.
> >
> > Considering backporting effort, and given it does make it easier to
> reason
> > about the client-server interaction, I will bump the version. For
> Produce &
> > Fetch, new bumped versions are going to be 10 & 16 respectively.
> >
> > Do note that FetchResponse has currentLeader introduced at version 12,
> but
> > only Java client at version 16 will support it. But that's the best we
> can
> > do.
> >
> >
> >
> >
> >
> >
> > On Mon, Jul 24, 2023 at 3:32 PM David Jacot  >
> > wrote:
> >
> > > Hi Mayank,
> > >
> > > 01. Hum... I understand your reasoning. I think that this is mainly
> > > beneficial for clients lagging behind in terms of supported versions.
> > > However, it is the opposite for the java client which is up to date.
> > > Personally, I would rather prefer to bump both versions and to add the
> > > tagged fields. This would allow us to better reason about what the
> client
> > > is supposed to do when we see the version on the server side.
> Otherwise,
> > we
> > > will never know if the client uses this or not.
> > >
> > > 03. Yeah, ignorable is not needed if the field has the same version.
> > >
> > > Thanks,
> > > David
> > >
> > > On Mon, Jul 24, 2023 at 4:09 PM José Armando García Sancio
> > >  wrote:
> > >
> > > > Hey Mayank,
> > > >
> > > > It is probably binary compatible to have the NodeEndponts fielld at
> > > > taggedVersion 12+ but I think it is misleading as a code reviewer.
> The
> > > > Java Kafka client at version 12 will never be able to handle those
> > > > fields. Or are you planning to backport these improvements to those
> > > > clients and brokers? If not, can we set the version and taggedVersion
> > > > to 15+?
> > > >
> > > > Thanks,
> > > > --
> > > > -José
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Mayank Shekhar Narula
> >
>


-- 
Regards,
Mayank Shekhar Narula

Re: Apache Kafka 3.6.0 release

[Colin McCabe]

Hi Satish,

I removed "KIP-866 ZooKeeper to KRaft Migration" from the list of pending KIPs, 
since that one was shipped in 3.4. I added "KIP-868 Metadata Transactions", 
since we are planning on implementing this in 3.6. (The KIP was approved a 
while ago, but not yet shipped.)

I also added "KIP-938: Add more metrics for measuring KRaft performance," which 
is a new KIP we are implemeting in 3.6 (The JIRA is open now for review.) Same 
for KIP-919 which is being voted on now.

best,
Colin

On Mon, Jul 24, 2023, at 03:59, Satish Duggana wrote:
> A gentle reminder on the KIP freeze date: 26th Jul. Please try to
> close discussion/vote threads asap.
>
> Thanks,
> Satish.
>
> On Sun, 23 Jul 2023 at 11:10, Satish Duggana  wrote:
>>
>> Thanks Colov/Divij for adding the KIP-952. I do not think it is a
>> blocker for 3.6.0. We can discuss the KIP in the respective thread.
>>
>> ~Satish.
>>
>> On Sun, 23 Jul 2023 at 07:21, Satish Duggana  
>> wrote:
>> >
>> > Thanks ShunKang for the update. I added both the KIPs to the wiki.
>> > Please feel free to update the wiki with the latest.
>> >
>> > ~Satish.
>> >
>> > On Sat, 22 Jul 2023 at 22:50, ShunKang Lin  
>> > wrote:
>> > >
>> > > Hi Satish,
>> > >
>> > > Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory copy" 
>> > > [1]
>> > > and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory
>> > > copying" [2] to the release plan?
>> > > Thanks!
>> > >
>> > > [1]
>> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
>> > >
>> > > [2]
>> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
>> > > I would appreciate a few more reviews on the pull request (
>> > > https://github.com/apache/kafka/pull/12685) for KIP-872.
>> > >
>> > > Best,
>> > > ShunKang
>> > >
>> > > Divij Vaidya  于2023年7月22日周六 20:06写道：
>> > >
>> > > > Hi Satish
>> > > >
>> > > > I have added the following accepted KIPs to the release plan. Please 
>> > > > let me
>> > > > know if something requires a change.
>> > > >
>> > > > Accepted KIPs -
>> > > >
>> > > > 1.
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
>> > > >
>> > > > 2.
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
>> > > >
>> > > >
>> > > > Pending discussion KIP which I believe is important to be merged into 
>> > > > 3.6 -
>> > > >
>> > > > 3.
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
>> > > >
>> > > >
>> > > > --
>> > > > Divij Vaidya
>> > > >
>> > > >
>> > > >
>> > > > On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
>> > > > 
>> > > > wrote:
>> > > >
>> > > > > Thanks Hao for the update on KIP-925.
>> > > > >
>> > > > > On Thu, 20 Jul 2023 at 23:05, Hao Li  
>> > > > > wrote:
>> > > > > >
>> > > > > > Hi Satish,
>> > > > > >
>> > > > > > KIP-925 was accepted and currently under implementation. I just 
>> > > > > > added
>> > > > it
>> > > > > to
>> > > > > > the release plan.
>> > > > > >
>> > > > > >
>> > > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Hao
>> > > > > >
>> > > > > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
>> > > > > > 
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hello!
>> > > > > > >
>> > > > > > > A couple of days ago I opened a new KIP for discussion - KIP-952
>> > > > [1]. I
>> > > > > > > believe it might be a blocker for the release of 3.6.0, but I 
>> > > > > > > wanted
>> > > > to
>> > > > > > > bring it up here for a decision on its urgency with the current 
>> > > > > > > set
>> > > > of
>> > > > > > > people who are looking at Tiered Storage (Satish, Luke, Ivan, 
>> > > > > > > Divij)
>> > > > > given
>> > > > > > > that the date for KIP freeze is fast approaching.
>> > > > > > > What are your thoughts on the matter?
>> > > > > > >
>> > > > > > > [1]
>> > > > > > >
>> > > > > > >
>> > > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Christo
>> > > > > > >
>> > > > > > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana <
>> > > > satish.dugg...@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Yash,
>> > > > > > > > Thanks for the update. Added KIP-793 to the release plan. 
>> > > > > > > > Please
>> > > > feel
>> > > > > > > > free to update the release wiki with any other updates on the 
>> > > > > > > > KIP.
>> > > > > > > >
>> > > > > > > > ~Satish.
>> > > > > > > >
>> > > > > > > > On Fri, 7 Jul 2023 at 10:52, Yash Mayya 
>> > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Hi Satish,
>> > > > > > > > >
>> > > > 

[VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

[Colin McCabe]

Hi all,

I'd like to start the vote for KIP-919: Allow AdminClient to Talk Directly with 
the KRaft Controller Quorum and add Controller Registration.

The KIP is here: https://cwiki.apache.org/confluence/x/Owo0Dw

Thanks to everyone who reviewed the proposal.

best,
Colin

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2031

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 294842 lines...]
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-7: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-13: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-14: SMOKE-TEST-CLIENT-CLOSED
streams-12: SMOKE-TEST-CLIENT-CLOSED

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 2h 26m 19s
231 actionable tasks: 124 executed, 107 up-to-date

Publishing build scan...
https://ge.apache.org/s/y43etypcthke4


See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk_2/build/reports/profile/profile-2023-07-24-14-15-41.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[Pipeline] echo
Skipping Kafka Streams archetype test for Java 20
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 4m 47s
89 actionable tasks: 41 executed, 48 up-to-date

Publishing build scan...
https://ge.apache.org/s/76p7ci7bt3a6m

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.6.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.6.0-SNAPSHOT/streams-quickstart-3.6.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.6.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- resources:2.7:testResources (default-testResources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- archetype:2.2:jar (default-jar) @ streams-quickstart-java ---
[INFO] Building 

Bug wile create users

[Sergio Daniel Troiano]

Hi everyone!

We think we found a bug, we dug deep and found the solution.
Please take a look at the bug reported, I am happy to deliver the patch
https://issues.apache.org/jira/browse/KAFKA-15243


Thanks!

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

[Almog Gavra]

Lots of thoughts! Happy to see the thriving discussion on this post - lots
going on so I'm trying to enumerate them to keep things organized (prefix
"A" for "Almog" so we can use numbers in responses for other things ;P).

A1. Question around closing implementation gaps (e.g. no rocks based
suppression store)
A2. Specifically how to handle Versioned stores
A3. Configuration (new config/reuse old one + new one and ordering of
config resolution)
A4. Drawing a line between what is implementation detail (not exposed in
API) and what is customizable (exposed in API)
A5. Naming of StoreTypeSpec
A6. Param classes in StoreBuilders

--

Here are summaries for where it seems each of these stands (trying not to
add any additional opinion yet):

A1. Sophie/Guozhang/Me (if I count hah!) seem to agree that it is worth
pushing this KIP through independently of the implementation gaps as it
doesn't seem to move the intermediate state further from the end state.
Matthias originally had some concerns.

A2. There's questions around whether versioned stores should be their own
configurable option or whether they are an implementation detail that the
StoreSpec should decide. It seems like the discussion is converging here,
this should be an implementation detail.

A3. Matthias/Guozhang prefer adding CUSTOM and then having an additional
config to determine the implementation. Sophie prefers deprecating the old
config. Guozhang additional suggested flipping the resolution order such
that the old config is only respected in a DefaultStoreSpec implementation.

A4. This KIP (or rather, the discussion on the KIP) blurs the lines between
top level store types (KV, windowed, session) and the underlying
implementation of them (timestamped, versioned, kv-list). It seems everyone
is in alignment to ensure that we keep these two things separate and that
the line is clearly delineated in the text of the KIP.

A5. Guozhang and Sophie agree that the current name StoreTypeSpec is
misleading, as it's really an implementation spec, not a type specification.

A6. Agreement that this is an improvement, Sophie believes this can be done
in a follow up but we should ensure our naming is good here so there's no
conflicts down the line.

-

Ok, phew! Hopefully that covers it all! Now for my thoughts, hopefully
wrapping up some of these discussions:

A1.  @Matthias - are you still hesitant here? What would you need to be
convinced here?

A2. Since we are all in agreement that versioned stores should be an
implementation detail, we have two options:

(1) we can extend the KVParams to include a parameter that indicates
whether or not the store should be versioned
(2) we can introduce a configuration for whether or not to use a versioned
store, and each implementation can choose to read/ignore that config

Any preferences? (1) would align more closely with what we are doing today
(they are a top-level concept in the Stores API).

A3. I like Guozhang's suggestion of making the "primary" configuration to
be the new one, and then having a "DefaultStoreTypeSpec" (using the old
naming) which respects the old configuration. That seems to solve nearly
all the concerns (e.g. it'd be easy to see where the enum is used because
it would only be used within that one class instead of littered throughout
the code base).

@Sophie, unless you have objections here I will update the KIP to do that.

A4. I will make these changes to the KIP to make it clear.

A5. I will rename it `DslStorePlugin` - any objections to this name?

A6. Let's punt this ;) I agree with everyone that this would be a welcome
improvement and that this KIP is aligned with moving in that direction.
Given how much discussion there was on this KIP, which is minor relative to
making the changes to StoreBuilder API, I'd rather not tie the two together.

Cheers & Thanks everyone for the thoughts!
- Almog

On Sun, Jul 23, 2023 at 5:15 PM Sophie Blee-Goldman 
wrote:

> Guozhang:
>
> On your 2nd point:
>
> > "impl types" (in hindsight it may not be a good name) for rocksdb /
> memory / custom, and we used "store types" for kv / windowed / sessioned
> etc,
> First off, thanks so much for this clarification -- using "store type" here
> was definitely making me uncomfortable as this usually refers to KV vs
> window, etc -- but I just couldn't for the life of me think of the right
> term for rocks vs IM. We should 100% change to something like StoreImplSpec
> for this kind of interface.
>
> > As for list-value store (for stream-stream Join)
> Again, glad you mentioned this -- I forgot how the extra stream-stream join
> store is not a "regular" KV Store but rather this special list-value store.
> If we proceed with something like the current approach, perhaps that should
> be a boolean (or enum) parameter in the KVConfig, similar to the
> EmitStrategy? After all, the high-level goal of this KIP is to be able to
> fully customize all DSL state stores, and 

[jira] [Created] (KAFKA-15243) User creation mismatch

[Sergio Troiano (Jira)]

Sergio Troiano created KAFKA-15243:
--

 Summary: User creation mismatch
 Key: KAFKA-15243
 URL: https://issues.apache.org/jira/browse/KAFKA-15243
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.3.2
Reporter: Sergio Troiano
Assignee: Sergio Troiano


We found the Kafka users were not created properly, so let's suppose we create 
the user [myu...@myuser.com|mailto:myu...@myuser.com]

 

COMMAND:
{code:java}
/etc/new_kafka/bin/kafka-configs.sh  --bootstrap-server localhost:9092 --alter 
--add-config 
'SCRAM-SHA-256=[iterations=4096,password=blabla],SCRAM-SHA-256=[password=blabla]'
 --entity-type users --entity-name myu...@myuser.com{code}
RESPONSE:
{code:java}
Completed updating config for user myu...@myuser.com{code}
When listing the users I see the user was created as an encoded string

COMMAND
{code:java}
kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type 
users|grep myuser {code}
RESPONSE
{code:java}
SCRAM credential configs for user-principal 'myuser%40myuser.com' are 
SCRAM-SHA-256=iterations=8192, SCRAM-SHA-512=iterations=4096 {code}
 

So basically the user is being "sanitized" and giving a false OK to the user 
requester. The user requested does not exist as it should, it creates the 
encoded one instead.

 

I dug deep in the code until I found this is happening in the 
ZkAdminManager.scala in this line 

 
{code:java}
adminZkClient.changeConfigs(ConfigType.User, Sanitizer.sanitize(user), 
configsByPotentiallyValidUser(user)) {code}
So removing the Sanitizer fix the problem, but I have a couple of doubts

I checked we Sanitize because of some JMX metrics, but in this case I don't 
know if this is really needed, supossing this is needed I think we should 
forbid to create users with characters that will be encoded.

Even worse after creating an user in general we create ACLs and they are 
created properly without encoding the characters, this creates a mismatch 
between the user and the ACLs.

 

 

So I can work on fixing this, but I think we need to decide :

 

A) We forbid to create users with characters that will be encoded, so we fail 
in the user creation step.

 

B) We allow the user creation with special characters and remove the 
Sanitizer.sanitize(user) from the 2 places where it shows up in the file 
ZkAdminManager.scala

 

 

And of course if we go for B we need to create the tests.

Please let me know what you think and i can work on it



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Erik van Oosten]

Hello Xiangyuan LI,

I am not familiar with coroutines, nor with Kotlin. You will have to 
work with the documentation: 
https://kotlinlang.org/docs/coroutines-overview.html


However, I am familiar with Zio and Cats-effects (both Scala libraries). 
In both Zio and Cats-effects one creates effects (aka workflows) which 
are descriptions of a computation. For example, when executing the Scala 
code `val effect = ZIO.attempt(println("Hello world!"))` one creates 
only a description; it does not print anything yet. The language to 
describe these effects is rich enough to describe entire applications 
including things like concurrency. In fact, the language is so rich, 
that it is the most convenient way that I know to safely write highly 
concurrent and async applications. For many developer teams the 
performance penalty (which is real but not big) is worth it.


To execute a Zio or Cats effect one gives it to the runtime. The runtime 
then schedules the work on one of the threads in its thread-pool. Zio, 
nor Cats-effects supports running an effect on the thread that manages 
the thread-pool.


I hope this clear enough.

Kind regards,
    Erik.


Op 24-07-2023 om 05:21 schreef Xiangyuan LI:

Hi Erik:
 I read KIP-944 and email list roughly, it seems most Java developer not
familiar with the conception of "coroutine" so cannot imagine why code of
one function without Thread.start() may run in separate threads and even
developer couldn't control it. Maybe you need a more elaborate description
to demonstrate how coroutine code run.

Erik van Oosten  于2023年7月23日周日 17:47写道：


Hi David,

  > Could you elaborate a bit more on why the callbacks must be ran in
another thread vs in the invoker thread?

I have been thinking on how to explain this for 2 months now and it is
not easy. It has something to do with that you cannot control what a
thread is doing if you have to also run on that thread. But I just
realized that /for me/ it really comes down to this:

  We want to use Zio in the callback. Zio does not support it.

There are more reasons as can be read in KAFKA-7143. But I do not know
anything about Kotlin so I cannot elaborate on that.

Kind regards,
   Erik.

Op 22-07-2023 om 21:39 schreef David Jacot:

Hi Erik,

Thanks for the KIP. I would like to better understand the motivation of
this KIP. I am not familiar with async runtimes so please excuse me if I
ask stupid questions.

Could you elaborate a bit more on why the callbacks must be ran in

another

thread vs in the invoker thread? This is not clear to me. In the example
that you use with the ConsumerRebalanceListener, I would have thought

that

calling commitSync (without changing thread) would have achieved the

same.

The invoker has to wait anyway on the offset commit completion so using
another thread does not bring any benefit here.  I suppose that I am
missing something here…

Regarding Chris’ proposal, this feels like a hack to me. The issue with

it

is that we cannot guarantee it in the long term if it is not part of

*the*

Consumer interface.

I second what Chris said. We are all trying to understand the motivation

in

order to find a good solution for Kafka. I apologize if this creates
frustration. This is definitely not our goal.

Best,
David

PS: I just saw that you opened a new KIP based on Chris’ idea. This is

not

necessary. You can just update the current KIP based on the discussion.

Le sam. 22 juil. 2023 à 18:34, Erik van Oosten



a écrit :


Colin, Matthias, Chris,

I have expanded the use case description in KIP-944. I hope it is more
clear what we're trying to achieve.

https://cwiki.apache.org/confluence/x/chw0Dw

Kind regards,
   Erik.


Op 22-07-2023 om 17:23 schreef Erik van Oosten:

Hello Chris,

Thanks for elaborating Matthias' words. Apparently the use case
description is too terse. Indeed, that is not FUD and that is
something I can work with.


It's also worth mentioning that what's proposed in the KIP is only

blocked by the private access modifier on the KafkaConsumer::acquire
and KafkaConsumer::release methods. If we upgraded the visibility of
these methods from private to protected, it would be possible for
subclasses to implement the proposal in KIP-944, without any KIPs or
other changes to the official Java clients library.

That is absolutely brilliant! Since I am pretty sure I am using the
consumer correctly, I could replace acquire and release with an empty
method body and be done.

/Is making acquire and release protected something that other people
can live with?/
If yes, I will create a new PR with just that change.

Kind regards,
  Erik.


Op 22-07-2023 om 16:39 schreef Chris Egerton:

Hi Erik,

I don't think Matthias is bringing FUD to the discussion. Many of the
people who maintain Kafka are familiar with Kafka client internals
and the
Java programming language, but not necessarily other JVM languages or
asynchronous runtimes. I think it's reasonable to ask for a code
snippet or
two 

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[David Jacot]

Hi Mayank,

We never backport new features to old releases. This new feature will be
only available from 3.6 (or 3.7) onwards for both client and server.

Best,
David

On Mon, Jul 24, 2023 at 5:20 PM Mayank Shekhar Narula <
mayanks.nar...@gmail.com> wrote:

> Thanks Jose/David/Ismael for your inputs.
>
> Not bumping the version, would require both broker & client to backport
> changes. Especially for FetchResponse, as backporting would have to be done
> all the way back to 3.1, so this effort isn't trivial, and was originally
> underestimated.
>
> Considering backporting effort, and given it does make it easier to reason
> about the client-server interaction, I will bump the version. For Produce &
> Fetch, new bumped versions are going to be 10 & 16 respectively.
>
> Do note that FetchResponse has currentLeader introduced at version 12, but
> only Java client at version 16 will support it. But that's the best we can
> do.
>
>
>
>
>
>
> On Mon, Jul 24, 2023 at 3:32 PM David Jacot 
> wrote:
>
> > Hi Mayank,
> >
> > 01. Hum... I understand your reasoning. I think that this is mainly
> > beneficial for clients lagging behind in terms of supported versions.
> > However, it is the opposite for the java client which is up to date.
> > Personally, I would rather prefer to bump both versions and to add the
> > tagged fields. This would allow us to better reason about what the client
> > is supposed to do when we see the version on the server side. Otherwise,
> we
> > will never know if the client uses this or not.
> >
> > 03. Yeah, ignorable is not needed if the field has the same version.
> >
> > Thanks,
> > David
> >
> > On Mon, Jul 24, 2023 at 4:09 PM José Armando García Sancio
> >  wrote:
> >
> > > Hey Mayank,
> > >
> > > It is probably binary compatible to have the NodeEndponts fielld at
> > > taggedVersion 12+ but I think it is misleading as a code reviewer. The
> > > Java Kafka client at version 12 will never be able to handle those
> > > fields. Or are you planning to backport these improvements to those
> > > clients and brokers? If not, can we set the version and taggedVersion
> > > to 15+?
> > >
> > > Thanks,
> > > --
> > > -José
> > >
> >
>
>
> --
> Regards,
> Mayank Shekhar Narula
>

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[José Armando García Sancio]

Hi Mayank,

On Mon, Jul 24, 2023 at 8:21 AM Mayank Shekhar Narula
 wrote:
>
> Thanks Jose/David/Ismael for your inputs.
>
> Not bumping the version, would require both broker & client to backport
> changes. Especially for FetchResponse, as backporting would have to be done
> all the way back to 3.1, so this effort isn't trivial, and was originally
> underestimated.
>
> Considering backporting effort, and given it does make it easier to reason
> about the client-server interaction, I will bump the version. For Produce &
> Fetch, new bumped versions are going to be 10 & 16 respectively.
>
> Do note that FetchResponse has currentLeader introduced at version 12, but
> only Java client at version 16 will support it. But that's the best we can
> do.
>

Makes sense to me. Thanks for the changes!

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

[Shay Lin]

Hi Sophie and Matthias, thanks for your comments and replies.

1. Scope of change: KStreams only or KStreams/KTable
I took some time to digest your points, looking through how KStreams
triggers repartitions today. I noticed that `repartitionRequired`is a flag
in KStreamImpl etc and not in KTableImpl etc. When I look further, in the
case of KTable, instead of passing in a boolean flag, a repartition node `
TableRepartitionMapNode` is directly created. I went back and referenced
the two issue tickets KAFKA-10844 and KAFKA-4835, both requests were
focused on KStreams, i.e. not to change the partition why the input streams
are already correctly keyed. Is it possible that in the case of KTable,
users always intend to repartition (change key) when they call on
aggregate? -- (this was written before I saw Matthias's comment)

Overall, based on the tickets, I see the benefit of doing a contained
change focusing on KStreams, i.e. repartitionRequired, which would solve
the pain points nicely. If we ran into similar complaints/optimization
requests for KTable down the line, we can address them on top of this(let
me know if we have these requests already, I might just be negligent).

2. API: markAsPartitioned() vs config
If we go with the KStreams only scope, markAsPartition() is more
adequate, i.e. maps nicely to repartitionRequired. There is a list of
NamedOperations that may or may not trigger repartition based on its
context(KStreams or KTable) which would make the implementation more
confusing.

3. KIP documentation: Thanks for providing the links to previous KIPs. I
will be adding the three use cases and javadoc. I will also document the
risks when it relates to IQ and Join.

Best,
Shay

On Fri, Jul 21, 2023 at 5:55 PM Matthias J. Sax  wrote:

> I agree that it could easily be misused. There is a few Jira tickets for
> cases when people want to "cancel" a repartition step. I would hope
> those tickets are linked to the KIP (if not, we should do this, and
> maybe even c those cases as motivation into the KIP itself)?
>
> It's always a tricky question to what extend we want to guide users, and
> to what extend we need to give levers for advances case (and how to
> design those levers...) It's for sure a good idea to call out "use with
> case" in the JavaDocs for the new method.
>
>
> -Matthias
>
> On 7/21/23 3:34 PM, Sophie Blee-Goldman wrote:
> > I guess I felt a bit uneasy about how this could be used/abused while
> > reading the KIP, but if we truly believe this is an advanced feature, I'm
> > fine with the way things currently are. It doesn't feel like the best
> API,
> > but it does seem to be the best *possible* API given the way things are.
> >
> > W.r.t the KTable notes, that all makes sense to me. I just wanted to lay
> > out all the potential cases to make sure we had our bases covered.
> >
> > I still think an example or two would help, but the only thing I will
> > actually wait on before feeling comfortable enough to vote on this would
> be
> > a clear method signature (and maybe sample javadocs) in the "Public
> > Interfaces" section.
> >
> > Thanks again for the KIP Shay! Hope I haven't dragged it out too much
> >
> > On Fri, Jul 21, 2023 at 3:19 PM Matthias J. Sax 
> wrote:
> >
> >> Some thought about the API question.
> >>
> >>
>  A. kstream.groupBy(...).aggregate(...)
> >>
> >> This can be re-writtten as
> >>
> >> kstream.selectKey(...)
> >>  .markAsRepartitioned()
> >>  .groupByKey()
> >>  .aggregate()
> >>
> >> Given that `markAsRepartitoned` is an advanced feature, I think it would
> >> be ok?
> >>
> >>
>  B. ktable.groupBy(...).aggregate(...)
> >>
> >> For KTable aggregation, not sure how useful it would be? In the end, an
> >> table aggregation does only make sense if we pick something from the
> >> value, ie, we indeed change the key?
> >>
> >>
>  C. kstream.selectKey(...).join(ktable)
> >>
> >> We can just insert a `markAsRepartitioned()` after `selectKey` to avoid
> >> repartitioning of the left input KStream.
> >>
> >>
> >>> KStream.selectKey(...).toTable().join(...)
> >>
> >> Not sure if I understand what you try to say with this example? In the
> >> end, `selectKey(...).toTable()` would repartiton. If I know that one can
> >> upsert directly, one inserts a `markAsRepartitioned()` in between.
> >>
> >>
> >> In general, the use case seems to be that the key is not in the right
> >> "format", or there is no key, but data was partitioned by a
> >> value-attribute upstream and we just want to extract this
> >> value-attribute into the key. Both seems to be KStream cases?
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 7/15/23 1:43 PM, Sophie Blee-Goldman wrote:
> >>> Hey Shay, while I don't have any specific concerns about the new public
> >> API
> >>> in this KIP, I'd like to better understand how this feature will work
> >>> before I vote. We should document the behavior of this new operator
> >> clearly
> >>> in the KIP as well -- you don't 

Re: [ANNOUNCE] Apache Kafka 3.5.1

[Federico Valeri]

Thanks Divij :)

On Mon, Jul 24, 2023 at 2:28 PM Luke Chen  wrote:
>
> Thanks for running the release!
>
> Luke
>
> On Mon, Jul 24, 2023 at 6:35 PM Manyanda Chitimbo <
> manyanda.chiti...@gmail.com> wrote:
>
> > Thanks for running the release Divij.
> >
> > On Mon 24 Jul 2023 at 12:15, Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Thanks for running the release, Divij!
> > >
> > > On Mon, Jul 24, 2023 at 3:17 PM Divij Vaidya  wrote:
> > >
> > > > The Apache Kafka community is pleased to announce the release for
> > Apache
> > > > Kafka 3.5.1
> > > >
> > > > This release is a security patch release. It upgrades the dependency,
> > > > snappy-java, to a version which is not vulnerable to CVE-2023-34455.
> > You
> > > > can find more information about the CVE at Kafka CVE list
> > > > .
> > > >
> > > > Additionally, this releases fixes a regression introduced in 3.3.0,
> > which
> > > > caused security.protocol configuration values to be restricted to upper
> > > > case only. With this release, security.protocol values are
> > > > case insensitive. See KAFKA-15053
> > > >  for details.
> > > >
> > > > All of the changes in this release can be found in the release notes:
> > > > https://www.apache.org/dist/kafka/3.5.1/RELEASE_NOTES.html
> > > >
> > > > You can download the source and binary release (Scala 2.11 and Scala
> > > 2.12)
> > > > from:
> > > > https://kafka.apache.org/downloads#3.5.1
> > > >
> > > >
> > > >
> > >
> > ---
> > > >
> > > > Apache Kafka is a distributed streaming platform with four core APIs:
> > > >
> > > >
> > > > ** The Producer API allows an application to publish a stream of
> > records
> > > to
> > > > one or more Kafka topics.
> > > >
> > > > ** The Consumer API allows an application to subscribe to one or more
> > > > topics and process the stream of records produced to them.
> > > >
> > > > ** The Streams API allows an application to act as a stream processor,
> > > > consuming an input stream from one or more topics and producing an
> > > > output stream to one or more output topics, effectively transforming
> > the
> > > > input streams to output streams.
> > > >
> > > > ** The Connector API allows building and running reusable producers or
> > > > consumers that connect Kafka topics to existing applications or data
> > > > systems. For example, a connector to a relational database might
> > > > capture every change to a table.
> > > >
> > > > With these APIs, Kafka can be used for two broad classes of
> > application:
> > > >
> > > > ** Building real-time streaming data pipelines that reliably get data
> > > > between systems or applications.
> > > >
> > > > ** Building real-time streaming applications that transform or react
> > > > to the streams of data.
> > > >
> > > > Apache Kafka is in use at large and small companies worldwide,
> > including
> > > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> > Rabobank,
> > > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > > >
> > > > A big thank you for the following 22 contributors to this release!
> > > (Please
> > > > report an unintended omission)
> > > >
> > > > Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai,
> > > > Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij
> > > > Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael
> > Maison,
> > > > minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya
> > > >
> > > > We welcome your help and feedback. For more information on how to
> > > > report problems, and to get involved, visit the project website at
> > > > https://kafka.apache.org/
> > > >
> > > > Thank you!
> > > >
> > > > Regards,
> > > > Divij Vaidya
> > > > Release Manager for Apache Kafka 3.5.1
> > > >
> > >
> > --
> > Manyanda Chitimbo.
> >

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Mayank Shekhar Narula]

Thanks Jose/David/Ismael for your inputs.

Not bumping the version, would require both broker & client to backport
changes. Especially for FetchResponse, as backporting would have to be done
all the way back to 3.1, so this effort isn't trivial, and was originally
underestimated.

Considering backporting effort, and given it does make it easier to reason
about the client-server interaction, I will bump the version. For Produce &
Fetch, new bumped versions are going to be 10 & 16 respectively.

Do note that FetchResponse has currentLeader introduced at version 12, but
only Java client at version 16 will support it. But that's the best we can
do.






On Mon, Jul 24, 2023 at 3:32 PM David Jacot 
wrote:

> Hi Mayank,
>
> 01. Hum... I understand your reasoning. I think that this is mainly
> beneficial for clients lagging behind in terms of supported versions.
> However, it is the opposite for the java client which is up to date.
> Personally, I would rather prefer to bump both versions and to add the
> tagged fields. This would allow us to better reason about what the client
> is supposed to do when we see the version on the server side. Otherwise, we
> will never know if the client uses this or not.
>
> 03. Yeah, ignorable is not needed if the field has the same version.
>
> Thanks,
> David
>
> On Mon, Jul 24, 2023 at 4:09 PM José Armando García Sancio
>  wrote:
>
> > Hey Mayank,
> >
> > It is probably binary compatible to have the NodeEndponts fielld at
> > taggedVersion 12+ but I think it is misleading as a code reviewer. The
> > Java Kafka client at version 12 will never be able to handle those
> > fields. Or are you planning to backport these improvements to those
> > clients and brokers? If not, can we set the version and taggedVersion
> > to 15+?
> >
> > Thanks,
> > --
> > -José
> >
>


-- 
Regards,
Mayank Shekhar Narula

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Ismael Juma]

Hi Crispin,

One additional question, the wiki says "The results are averaged over 2
runs.". Can you please provide some measure of variance in the
distribution, i.e. were both results similar to each other for both cases?

Ismael

On Fri, Jul 21, 2023 at 11:31 AM Ismael Juma  wrote:

> Thanks for the update Crispin - very helpful to have actual performance
> data. 2-5% for the default configuration is a bit on the low side for this
> kind of proposal.
>
> Ismael
>
> On Thu, Jul 20, 2023 at 11:33 PM Crispin Bernier
>  wrote:
>
>> Benchmark numbers have been posted on the KIP, please review.
>>
>> On 2023/07/20 13:03:00 Mayank Shekhar Narula wrote:
>> > Jun
>> >
>> > Thanks for the feedback.
>> >
>> > Numbers to follow.
>> >
>> > If we don't plan to
>> > > bump up the FetchResponse version, we could just remove the reference
>> to
>> > > version 16.
>> >
>> > Fixed.
>> >
>> > On Thu, Jul 20, 2023 at 1:28 AM Jun Rao 
>> wrote:
>> >
>> > > Hi, Mayank,
>> > >
>> > > Thanks for the KIP. I agree with others that it would be useful to
>> see the
>> > > performance results. Otherwise, just a minor comment. If we don't
>> plan to
>> > > bump up the FetchResponse version, we could just remove the reference
>> to
>> > > version 16.
>> > >
>> > > Jun
>> > >
>> > > On Wed, Jul 19, 2023 at 2:31 PM Mayank Shekhar Narula <
>> > > mayanks.nar...@gmail.com> wrote:
>> > >
>> > > > Luke
>> > > >
>> > > > Thanks for the interest in the KIP.
>> > > >
>> > > > But what if the consumer was fetching from the follower?
>> > > >
>> > > > We already include `PreferredReadReplica` in the fetch response.
>> > > > > Should we put the node info of PreferredReadReplica under this
>> case,
>> > > > > instead of the leader's info?
>> > > > >
>> > > >
>> > > > PreferredReadReplica is the decided on the leader. Looking at the
>> Java
>> > > > client code, AbstractFetch::selectReadReplica, first fetch request
>> goes
>> > > to
>> > > > Leader of the partition -> Sends back PreferredReadReplica -> Next
>> fetch
>> > > > uses PreferredReadReplica. So as long as leader is available,
>> > > > PreferredReadReplica would be found in subsequent fetches.
>> > > >
>> > > > Also, under this case, should we include the leader's info in the
>> > > response?
>> > > >
>> > > >
>> > > > In this case, I think the follower would fail the fetch if it knows
>> a
>> > > > different leader. If the follower knows a newer leader, it would
>> return
>> > > new
>> > > > leader information in the response, for the client to act on.
>> > > >
>> > > >
>> > > > Will we include the leader/node info in the response when having
>> > > > > `UNKNOWN_LEADER_EPOCH` error?
>> > > >
>> > > >
>> > > > My understanding is UNKNOWN_LEADER_EPOCH when a request from a
>> client
>> > > has a
>> > > > newer epoch than the broker. So the client is already up to date on
>> new
>> > > > leader information, it's the broker that has the catching up to do.
>> I
>> > > think
>> > > > there might be some optimisations to make sure the broker refreshes
>> its
>> > > > metadata quickly, so it can quickly recover to handle requests that
>> > > > previously returned UNKNOWN_LEADER_EPOCH. But this work is outside
>> the
>> > > > scope of this KIP, as for now this KIP focusses on client-side
>> > > > optimisations.
>> > > >
>> > > > Mayank
>> > > >
>> > > > On Tue, Jul 18, 2023 at 8:51 AM Luke Chen  wrote:
>> > > >
>> > > > > Hi Mayank,
>> > > > >
>> > > > > Thanks for the KIP!
>> > > > >
>> > > > > Some questions:
>> > > > > 1. I can see most of the cases we only care about consumer fetch
>> from
>> > > the
>> > > > > leader.
>> > > > > But what if the consumer was fetching from the follower?
>> > > > > We already include `PreferredReadReplica` in the fetch response.
>> > > > > Should we put the node info of PreferredReadReplica under this
>> case,
>> > > > > instead of the leader's info?
>> > > > > Also, under this case, should we include the leader's info in the
>> > > > response?
>> > > > >
>> > > > > 2. Will we include the leader/node info in the response when
>> having
>> > > > > `UNKNOWN_LEADER_EPOCH` error?
>> > > > > I think it's fine we ignore the `UNKNOWN_LEADER_EPOCH` error
>> since when
>> > > > > this happens, the node might have some error which should refresh
>> the
>> > > > > metadata. On the other hand, it might also be good if we can heal
>> the
>> > > > node
>> > > > > soon to do produce/consume works.
>> > > > >
>> > > > >
>> > > > > Thank you.
>> > > > > Luke
>> > > > >
>> > > > > On Tue, Jul 18, 2023 at 2:00 AM Philip Nee 
>> > > wrote:
>> > > > >
>> > > > > > Hey Mayank:
>> > > > > >
>> > > > > > For #1: I think fetch and produce behave a bit differently on
>> > > metadata.
>> > > > > > Maybe it is worth highlighting the changes for each client in
>> detail.
>> > > > In
>> > > > > > producer did you mean by the metadata timeout before sending out
>> > > > produce
>> > > > > > requests? For consumer: I think for fetches it requires user to
>> retry
>> > > > if
>> > > > > > the 

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Ismael Juma]

On Mon, Jul 24, 2023 at 3:32 PM David Jacot 
wrote:

> 01. Hum... I understand your reasoning. I think that this is mainly
> beneficial for clients lagging behind in terms of supported versions.
> However, it is the opposite for the java client which is up to date.
> Personally, I would rather prefer to bump both versions and to add the
> tagged fields. This would allow us to better reason about what the client
> is supposed to do when we see the version on the server side. Otherwise, we
> will never know if the client uses this or not.
>

+1

Ismael

Re: [DISCUSS] KIP-714: Client metrics and observability

[Andrew Schofield]

Hi Milind,
Thanks for your questions about the KIP.

1) I did some archaeology and looked at historical versions of the KIP. I think 
this is
just a mistake. 5 minutes is the default metric push interval. 30 minutes is a 
mystery
to me. I’ve updated the KIP.

2) I think there are two situations in which INVALID_RECORD might occur.
a) The client might perhaps be using a content-type that the broker does not 
support.
The KIP mentions content-type as a future extension, but there’s only one 
supported
to start with. Until we have multiple content-types, this seems out of scope. I 
think a
future KIP would add another error code for this.
b) The client might perhaps have a bug which means the metrics payload is 
malformed.
Logging a warning and attempting the next metrics push on the push interval 
seems
appropriate.

UNKNOWN_SUBSCRIPTION_ID would indeed be handled by making an immediate
GetTelemetrySubscriptionsRequest.

UNSUPPORTED_COMPRESSION_TYPE seems like either a client bug or perhaps
a situation in which a broker sends a compression type in a 
GetTelemetrySubscriptionsResponse
which is subsequently not supported when its used with a PushTelemetryRequest.
We do want the client to have the opportunity to get an up-to-date list of 
supported
compression types. I think an immediate GetTelemetrySubscriptionsRequest is 
appropriate.

3) If a client attempts a subsequent handshake with a Null ClientInstanceId, the
receiving broker may not already know the client's existing ClientInstanceId. 
If the
receiving broker knows the existing ClientInstanceId, it simply responds the 
existing
value back to the client. If it does not know the existing ClientInstanceId, it 
will create
a new client instance ID and respond with that.

I will update the KIP with these clarifications.

Thanks,
Andrew

> On 17 Jul 2023, at 14:21, Milind Luthra  wrote:
>
> Hi Andrew, thanks for this KIP.
>
> I had a few questions regarding the "Error handling" section.
>
> 1. It mentions that "The 5 and 30 minute retries are to eventually trigger
> a retry and avoid having to restart clients if the cluster metrics
> configuration is disabled temporarily, e.g., by operator error, rolling
> upgrades, etc."
> But this 30 min interval isn't mentioned anywhere else. What is it
> referring to?
>
> 2. For the actual errors:
> INVALID_RECORD : The action required is to "Log a warning to the
> application and schedule the next GetTelemetrySubscriptionsRequest to 5
> minutes". Why is this 5 minutes, and not something like PushIntervalMs? And
> also, why are we scheduling a GetTelemetrySubscriptionsRequest in this
> case, if the serialization is broken?
> UNKNOWN_SUBSCRIPTION_ID , UNSUPPORTED_COMPRESSION_TYPE : just to confirm,
> the GetTelemetrySubscriptionsRequest needs to be scheduled immediately
> after the PushTelemetry response, correct?
>
> 3. For "Subsequent GetTelemetrySubscriptionsRequests must include the
> ClientInstanceId returned in the first response, regardless of broker":
> Will a broker error be returned in case some implementation of this KIP
> violates this accidentally and sends a request with ClientInstanceId = Null
> even when it's been obtained already? Or will a new ClientInstanceId be
> returned without an error?
>
> Thanks!
>
> On Tue, Jun 13, 2023 at 8:38 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
>> Hi,
>> I would like to start a new discussion thread on KIP-714: Client metrics
>> and observability.
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability
>>
>> I have edited the proposal significantly to reduce the scope. The overall
>> mechanism for client metric subscriptions is unchanged, but the
>> KIP is now based on the existing client metrics, rather than introducing
>> new metrics. The purpose remains helping cluster operators
>> investigate performance problems experienced by clients without requiring
>> changes to the client application code or configuration.
>>
>> Thanks,
>> Andrew

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[David Jacot]

Hi Mayank,

01. Hum... I understand your reasoning. I think that this is mainly
beneficial for clients lagging behind in terms of supported versions.
However, it is the opposite for the java client which is up to date.
Personally, I would rather prefer to bump both versions and to add the
tagged fields. This would allow us to better reason about what the client
is supposed to do when we see the version on the server side. Otherwise, we
will never know if the client uses this or not.

03. Yeah, ignorable is not needed if the field has the same version.

Thanks,
David

On Mon, Jul 24, 2023 at 4:09 PM José Armando García Sancio
 wrote:

> Hey Mayank,
>
> It is probably binary compatible to have the NodeEndponts fielld at
> taggedVersion 12+ but I think it is misleading as a code reviewer. The
> Java Kafka client at version 12 will never be able to handle those
> fields. Or are you planning to backport these improvements to those
> clients and brokers? If not, can we set the version and taggedVersion
> to 15+?
>
> Thanks,
> --
> -José
>

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Omnia Ibrahim]

Hi Chris, I updated the KIP to address your feedback. Thanks for the vote.

On Mon, Jul 24, 2023 at 1:30 PM Chris Egerton 
wrote:

> Hi Omnia,
>
> I think there's a few clarifications that should still be made on the KIP,
> but assuming these are agreeable, I'm +1 (binding)
>
> - In the description for the
> replication.policy.internal.topic.separator.enabled property (in the
> "Public Interfaces" section), we should specify that it affects only the
> checkpoints and offset syncs topic
> - We can remove the code snippet from the "Proposed Changes" section (right
> now it's a little buggy; there's two different implementations for the same
> "internalSuffix" method, and there are references to an "internalSeparator"
> method but no implementation for it); since we don't usually require
> specific code changes in KIPs, I think as long as we can describe the
> changes we're proposing in the "Public Interfaces" section, that should be
> enough for this KIP
>
> Cheers,
>
> Chris
>
> On Mon, Jul 24, 2023 at 2:04 AM Federico Valeri 
> wrote:
>
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> >
> > On Sun, Jul 23, 2023 at 6:30 PM Omnia Ibrahim 
> > wrote:
> > >
> > > Hi everyone,
> > > I would like to open a vote for KIP-949. The proposal is here
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> > > .
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> > >
> > >
> > > Thanks
> > > Omnia
> >
>

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[José Armando García Sancio]

Hey Mayank,

It is probably binary compatible to have the NodeEndponts fielld at
taggedVersion 12+ but I think it is misleading as a code reviewer. The
Java Kafka client at version 12 will never be able to handle those
fields. Or are you planning to backport these improvements to those
clients and brokers? If not, can we set the version and taggedVersion
to 15+?

Thanks,
-- 
-José

[jira] [Created] (KAFKA-15242) FixedKeyProcessor testing is unusable

[Zlstibor Veljkovic (Jira)]

Zlstibor Veljkovic created KAFKA-15242:
--

 Summary: FixedKeyProcessor testing is unusable
 Key: KAFKA-15242
 URL: https://issues.apache.org/jira/browse/KAFKA-15242
 Project: Kafka
  Issue Type: Bug
Reporter: Zlstibor Veljkovic


Using mock processor context to get the forwarded message doesn't work.
Also there is not a well documented way for testing FixedKeyProcessors.

Please see the repo at [https://github.com/zveljkovic/kafka-repro]
but most important piece is test file with runtime and compile time errors:
[https://github.com/zveljkovic/kafka-repro/blob/master/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-942: Add Power(ppc64le) support

[Divij Vaidya]

Hi Vaibhav

1. Could you please add the details that we discussed here in this
thread to the KIP (e.g. why is it safe to add this infra now and which
other projects are using it). This would help to have KIP as a source
of record if someone wants to read about this decision in future.

2. Given that we merge ~5-6 commits a day to trunk, I would be
inclined towards running this infra on every merge to trunk (instead
of a daily run as I suggested earlier). It won't be run for every PR
though.

These are the last two thoughts from my side. Otherwise, it looks good.

--
Divij Vaidya


On Mon, Jul 10, 2023 at 7:09 PM Vaibhav Nazare
 wrote:
>
> Hi Colin and Mickael,
>
> Yes I agree we don't have to run on every build , may be as mentioned we can 
> run the job once/twice a day at specific time.
> According the test plan the tests which I mentioned are not specific to 
> ppc64le , I was running tests on the trunk branch which were common across 
> platforms on that specific commit and I have removed the table and mentioned 
> the test plan accordingly when to run the jobs.
>
>
>
> -Original Message-
> From: Mickael Maison 
> Sent: Tuesday, July 4, 2023 3:12 PM
> To: dev@kafka.apache.org
> Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support
>
> Hi,
>
> Thanks for the KIP!
> In the Test Plan section you mentioned a few unit and integration tests are 
> failing. Are these flaky tests or did you find issues related to ppc64le? If 
> they are just flaky tests, I think we can remove that table and instead 
> describe how often we intend to run the tests in the CI.
>
> Thanks,
> Mickael
>
> On Tue, Jul 4, 2023 at 12:04 AM Colin McCabe  wrote:
> >
> > I agree with Divij. A nightly Apache Kafka build for PowerPC would be 
> > welcome. But it shouldn't run on every build, since the extra time and 
> > complexity would not be worth it.
> >
> > By the way, are there any features or plugins we don't intend to support on 
> > PPC? If there are, this KIP would be a good place to spell them out.
> >
> > Naively, I would think all of our Java and Scala code should work on
> > PPC without changes. However, there may be library dependencies that
> > don't exist on PPC. (We have to remember that the last desktop PowerPC
> > chip that an average user could buy shipped in 2005)
> >
> > best,
> > Colin
> >
> >
> > On Mon, Jun 19, 2023, at 23:12, Vaibhav Nazare wrote:
> > > Thank you for response Divij.
> > >
> > > 1. We are going to use ASF infra provided nodes for better
> > > availability and stability as there are 3 power9 nodes managed
> > > officially by ASF infra team themselves.
> > > Ref:
> > > INVALID URI REMOVED
> > > rg_jira_browse_INFRA-2D24663=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I
> > > 3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11
> > > WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=EeWMMaaUqDLlqUjJmw3_ouI9lt
> > > 4B_EUGyH8OZ_VTIRE=
> > > INVALID URI REMOVED.
> > > apache.org_view_Shared-2520-2D-2520ppc64le-2520nodes_=DwIFaQ=jf_
> > > iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=
> > > wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=O
> > > R-kg7uZdehAVxU7TdCUruDpGb0yOAEa0wW2dC3P3yU=
> > > previously used power node details for apache/kafka CI:
> > > RAM- 16GB
> > > VCPUs- 8 VCPU
> > > Disk- 160GB
> > > for shared VMs we need to check with ASF infra team to provide
> > > details
> > >
> > > 2. We can run nightly builds once or twice in a day on specific
> > > period of time instead of every commit 3. apache/camel
> > > INVALID URI REMOVED
> > > rg_job_Camel_job_el_=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=J35xYPMn5tCJRn91aARKgyiwas_cu4RGeSsh50tV4GI=
> > >   has already enabled CI for power platform they are using same H/W 
> > > resources as
> > > RAM- 16GB
> > > VCPUs- 8 VCPU
> > > Disk- 160GB
> > >
> > > -Original Message-
> > > From: Divij Vaidya 
> > > Sent: Monday, June 19, 2023 10:20 PM
> > > To: dev@kafka.apache.org
> > > Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le)
> > > support
> > >
> > > Thank you for the KIP Vaibhav.
> > >
> > > 1. Builds for power architecture were intentionally disabled in the
> > > past since the infrastructure was flaky [1]. Could you please add to
> > > the KIP on what has changed since then?
> > > 2. What do you think about an alternative solution where we run a
> > > nightly build for this platform instead of running the CI with every
> > > PR/commit?
> > > 3. To bolster the case for this KIP, could you please add
> > > information from other Apache projects who are already running CI
> > > for this platform? Is their CI stable on Apache Infra hosts?
> > >
> > >
> > > [1]
> > > INVALID URI REMOVED
> > > he_kafka_pull_12380=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHA
> > > urpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH7
> > > 

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Chris Egerton]

Hi Omnia,

I think there's a few clarifications that should still be made on the KIP,
but assuming these are agreeable, I'm +1 (binding)

- In the description for the
replication.policy.internal.topic.separator.enabled property (in the
"Public Interfaces" section), we should specify that it affects only the
checkpoints and offset syncs topic
- We can remove the code snippet from the "Proposed Changes" section (right
now it's a little buggy; there's two different implementations for the same
"internalSuffix" method, and there are references to an "internalSeparator"
method but no implementation for it); since we don't usually require
specific code changes in KIPs, I think as long as we can describe the
changes we're proposing in the "Public Interfaces" section, that should be
enough for this KIP

Cheers,

Chris

On Mon, Jul 24, 2023 at 2:04 AM Federico Valeri 
wrote:

> +1 (non binding)
>
> Thanks
> Fede
>
>
> On Sun, Jul 23, 2023 at 6:30 PM Omnia Ibrahim 
> wrote:
> >
> > Hi everyone,
> > I would like to open a vote for KIP-949. The proposal is here
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> > .
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> >
> >
> > Thanks
> > Omnia
>

Re: [ANNOUNCE] Apache Kafka 3.5.1

[Luke Chen]

Thanks for running the release!

Luke

On Mon, Jul 24, 2023 at 6:35 PM Manyanda Chitimbo <
manyanda.chiti...@gmail.com> wrote:

> Thanks for running the release Divij.
>
> On Mon 24 Jul 2023 at 12:15, Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Thanks for running the release, Divij!
> >
> > On Mon, Jul 24, 2023 at 3:17 PM Divij Vaidya  wrote:
> >
> > > The Apache Kafka community is pleased to announce the release for
> Apache
> > > Kafka 3.5.1
> > >
> > > This release is a security patch release. It upgrades the dependency,
> > > snappy-java, to a version which is not vulnerable to CVE-2023-34455.
> You
> > > can find more information about the CVE at Kafka CVE list
> > > .
> > >
> > > Additionally, this releases fixes a regression introduced in 3.3.0,
> which
> > > caused security.protocol configuration values to be restricted to upper
> > > case only. With this release, security.protocol values are
> > > case insensitive. See KAFKA-15053
> > >  for details.
> > >
> > > All of the changes in this release can be found in the release notes:
> > > https://www.apache.org/dist/kafka/3.5.1/RELEASE_NOTES.html
> > >
> > > You can download the source and binary release (Scala 2.11 and Scala
> > 2.12)
> > > from:
> > > https://kafka.apache.org/downloads#3.5.1
> > >
> > >
> > >
> >
> ---
> > >
> > > Apache Kafka is a distributed streaming platform with four core APIs:
> > >
> > >
> > > ** The Producer API allows an application to publish a stream of
> records
> > to
> > > one or more Kafka topics.
> > >
> > > ** The Consumer API allows an application to subscribe to one or more
> > > topics and process the stream of records produced to them.
> > >
> > > ** The Streams API allows an application to act as a stream processor,
> > > consuming an input stream from one or more topics and producing an
> > > output stream to one or more output topics, effectively transforming
> the
> > > input streams to output streams.
> > >
> > > ** The Connector API allows building and running reusable producers or
> > > consumers that connect Kafka topics to existing applications or data
> > > systems. For example, a connector to a relational database might
> > > capture every change to a table.
> > >
> > > With these APIs, Kafka can be used for two broad classes of
> application:
> > >
> > > ** Building real-time streaming data pipelines that reliably get data
> > > between systems or applications.
> > >
> > > ** Building real-time streaming applications that transform or react
> > > to the streams of data.
> > >
> > > Apache Kafka is in use at large and small companies worldwide,
> including
> > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> Rabobank,
> > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > >
> > > A big thank you for the following 22 contributors to this release!
> > (Please
> > > report an unintended omission)
> > >
> > > Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai,
> > > Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij
> > > Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael
> Maison,
> > > minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya
> > >
> > > We welcome your help and feedback. For more information on how to
> > > report problems, and to get involved, visit the project website at
> > > https://kafka.apache.org/
> > >
> > > Thank you!
> > >
> > > Regards,
> > > Divij Vaidya
> > > Release Manager for Apache Kafka 3.5.1
> > >
> >
> --
> Manyanda Chitimbo.
>

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

A gentle reminder on the KIP freeze date: 26th Jul. Please try to
close discussion/vote threads asap.

Thanks,
Satish.

On Sun, 23 Jul 2023 at 11:10, Satish Duggana  wrote:
>
> Thanks Colov/Divij for adding the KIP-952. I do not think it is a
> blocker for 3.6.0. We can discuss the KIP in the respective thread.
>
> ~Satish.
>
> On Sun, 23 Jul 2023 at 07:21, Satish Duggana  wrote:
> >
> > Thanks ShunKang for the update. I added both the KIPs to the wiki.
> > Please feel free to update the wiki with the latest.
> >
> > ~Satish.
> >
> > On Sat, 22 Jul 2023 at 22:50, ShunKang Lin  
> > wrote:
> > >
> > > Hi Satish,
> > >
> > > Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory copy" 
> > > [1]
> > > and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory
> > > copying" [2] to the release plan?
> > > Thanks!
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> > >
> > > [2]
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> > > I would appreciate a few more reviews on the pull request (
> > > https://github.com/apache/kafka/pull/12685) for KIP-872.
> > >
> > > Best,
> > > ShunKang
> > >
> > > Divij Vaidya  于2023年7月22日周六 20:06写道：
> > >
> > > > Hi Satish
> > > >
> > > > I have added the following accepted KIPs to the release plan. Please 
> > > > let me
> > > > know if something requires a change.
> > > >
> > > > Accepted KIPs -
> > > >
> > > > 1.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
> > > >
> > > > 2.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
> > > >
> > > >
> > > > Pending discussion KIP which I believe is important to be merged into 
> > > > 3.6 -
> > > >
> > > > 3.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > >
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
> > > > 
> > > > wrote:
> > > >
> > > > > Thanks Hao for the update on KIP-925.
> > > > >
> > > > > On Thu, 20 Jul 2023 at 23:05, Hao Li  
> > > > > wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > KIP-925 was accepted and currently under implementation. I just 
> > > > > > added
> > > > it
> > > > > to
> > > > > > the release plan.
> > > > > >
> > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > > > > >
> > > > > > Thanks,
> > > > > > Hao
> > > > > >
> > > > > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hello!
> > > > > > >
> > > > > > > A couple of days ago I opened a new KIP for discussion - KIP-952
> > > > [1]. I
> > > > > > > believe it might be a blocker for the release of 3.6.0, but I 
> > > > > > > wanted
> > > > to
> > > > > > > bring it up here for a decision on its urgency with the current 
> > > > > > > set
> > > > of
> > > > > > > people who are looking at Tiered Storage (Satish, Luke, Ivan, 
> > > > > > > Divij)
> > > > > given
> > > > > > > that the date for KIP freeze is fast approaching.
> > > > > > > What are your thoughts on the matter?
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > > > > >
> > > > > > > Best,
> > > > > > > Christo
> > > > > > >
> > > > > > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana <
> > > > satish.dugg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yash,
> > > > > > > > Thanks for the update. Added KIP-793 to the release plan. Please
> > > > feel
> > > > > > > > free to update the release wiki with any other updates on the 
> > > > > > > > KIP.
> > > > > > > >
> > > > > > > > ~Satish.
> > > > > > > >
> > > > > > > > On Fri, 7 Jul 2023 at 10:52, Yash Mayya 
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Satish,
> > > > > > > > >
> > > > > > > > > KIP-793 [1] just passed voting and we should be able to wrap 
> > > > > > > > > up
> > > > the
> > > > > > > > > implementation in time for the 3.6.0 feature freeze. Could we 
> > > > > > > > > add
> > > > > it to
> > > > > > > > the
> > > > > > > > > release plan?
> > > > > > > > >
> > > > > > > > > [1] -
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Yash
> > > > > > > > >
> > > > > > > > > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana <
> > > > > > > satish.dugg...@gmail.com
> > > > > > > > >
> > > > > 

Re: [ANNOUNCE] Apache Kafka 3.5.1

[Manyanda Chitimbo]

Thanks for running the release Divij.

On Mon 24 Jul 2023 at 12:15, Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Thanks for running the release, Divij!
>
> On Mon, Jul 24, 2023 at 3:17 PM Divij Vaidya  wrote:
>
> > The Apache Kafka community is pleased to announce the release for Apache
> > Kafka 3.5.1
> >
> > This release is a security patch release. It upgrades the dependency,
> > snappy-java, to a version which is not vulnerable to CVE-2023-34455. You
> > can find more information about the CVE at Kafka CVE list
> > .
> >
> > Additionally, this releases fixes a regression introduced in 3.3.0, which
> > caused security.protocol configuration values to be restricted to upper
> > case only. With this release, security.protocol values are
> > case insensitive. See KAFKA-15053
> >  for details.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.5.1/RELEASE_NOTES.html
> >
> > You can download the source and binary release (Scala 2.11 and Scala
> 2.12)
> > from:
> > https://kafka.apache.org/downloads#3.5.1
> >
> >
> >
> ---
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records
> to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 22 contributors to this release!
> (Please
> > report an unintended omission)
> >
> > Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai,
> > Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij
> > Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael Maison,
> > minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > https://kafka.apache.org/
> >
> > Thank you!
> >
> > Regards,
> > Divij Vaidya
> > Release Manager for Apache Kafka 3.5.1
> >
>
-- 
Manyanda Chitimbo.

Re: [ANNOUNCE] Apache Kafka 3.5.1

[Kamal Chandraprakash]

Thanks for running the release, Divij!

On Mon, Jul 24, 2023 at 3:17 PM Divij Vaidya  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 3.5.1
>
> This release is a security patch release. It upgrades the dependency,
> snappy-java, to a version which is not vulnerable to CVE-2023-34455. You
> can find more information about the CVE at Kafka CVE list
> .
>
> Additionally, this releases fixes a regression introduced in 3.3.0, which
> caused security.protocol configuration values to be restricted to upper
> case only. With this release, security.protocol values are
> case insensitive. See KAFKA-15053
>  for details.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.5.1/RELEASE_NOTES.html
>
> You can download the source and binary release (Scala 2.11 and Scala 2.12)
> from:
> https://kafka.apache.org/downloads#3.5.1
>
>
> ---
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 22 contributors to this release! (Please
> report an unintended omission)
>
> Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai,
> Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij
> Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael Maison,
> minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Divij Vaidya
> Release Manager for Apache Kafka 3.5.1
>

[jira] [Created] (KAFKA-15241) Compute tiered offset by keeping the respective epochs in scope.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15241:
--

 Summary: Compute tiered offset by keeping the respective epochs in 
scope.
 Key: KAFKA-15241
 URL: https://issues.apache.org/jira/browse/KAFKA-15241
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.6.0
Reporter: Satish Duggana
Assignee: Kamal Chandraprakash


This is a followup on the discussion 
[thread|https://github.com/apache/kafka/pull/14004#discussion_r1268911909]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache Kafka 3.5.1

[Divij Vaidya]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 3.5.1

This release is a security patch release. It upgrades the dependency,
snappy-java, to a version which is not vulnerable to CVE-2023-34455. You
can find more information about the CVE at Kafka CVE list
.

Additionally, this releases fixes a regression introduced in 3.3.0, which
caused security.protocol configuration values to be restricted to upper
case only. With this release, security.protocol values are
case insensitive. See KAFKA-15053
 for details.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.5.1/RELEASE_NOTES.html

You can download the source and binary release (Scala 2.11 and Scala 2.12)
from:
https://kafka.apache.org/downloads#3.5.1

---

Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.

With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.

Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 22 contributors to this release! (Please
report an unintended omission)

Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai,
Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij
Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael Maison,
minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Divij Vaidya
Release Manager for Apache Kafka 3.5.1

[jira] [Created] (KAFKA-15240) BrokerToControllerChannelManager cache activeController error cause DefaultAlterPartitionManager send AlterPartition request failed

[shilin Lu (Jira)]

shilin Lu created KAFKA-15240:
-

 Summary: BrokerToControllerChannelManager cache activeController 
error cause DefaultAlterPartitionManager send AlterPartition request failed
 Key: KAFKA-15240
 URL: https://issues.apache.org/jira/browse/KAFKA-15240
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.5.0, 2.8.2, 2.8.1, 2.8.0
 Environment: 2.8.1 kafka version
Reporter: shilin Lu
Assignee: shilin Lu
 Attachments: image-2023-07-24-16-35-56-589.png

After KIP-497，partition leader do not use zk to propagateIsrChanges，it will 
send AlterPartitionRequest to controller to propagateIsrChanges.Then broker 
will cache active controller node info through controllerNodeProvider interface.

2023.07.12，in kafka product environment，we find so much `Broker had a stale 
broker epoch` when send partitionAlterRequest to controller.And in this kafka 
cluster has so much replica not in isr assignment with replica fetch is 
correct.So it only propagateIsrChanges failed.

!https://iwiki.woa.com/tencent/api/attachments/s3/url?attachmentid=3165506!

But there has something strange，if broker send partitionAlterRequest failed 
controller will print some log like this.But in active controller node not find 
this log info

!image-2023-07-24-16-35-56-589.png!

Then i just suspect this broker connect to an error active controller.Through 
network packet capture, find this broker connect to an unfamiliar broker 
port(9092) send request.Refer to this kafka cluster operation history，find this 
unfamiliar broker is an old broker node in this cluster and this node is a 
controller node in new kafka cluster.

Current BrokerToControllerChannelManager update active controller only happened 
when disconnect or responseCode is NOT_CONTROLLER. So when no request send and 
error broker node is another kafka cluster controller node，this case will 
repetite.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-930: Tiered Storage Metrics

[Abhijeet Kumar]

Hi All,

I created KIP-930 for adding RemoteIndexCache stats and also to rename some
tiered storage metrics added as part of KIP-405

to remove ambiguity.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics

Feedback and suggestions are welcome.

Regards,
Abhijeet.

[GitHub] [kafka-site] divijvaidya merged pull request #533: MINOR: Website changes for 3.5.1 release

[via GitHub]

divijvaidya merged PR #533:
URL: https://github.com/apache/kafka-site/pull/533


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15239) producerPerformance system test for old client failed after v3.5.0

[Luke Chen (Jira)]

Luke Chen created KAFKA-15239:
-

 Summary: producerPerformance system test for old client failed 
after v3.5.0
 Key: KAFKA-15239
 URL: https://issues.apache.org/jira/browse/KAFKA-15239
 Project: Kafka
  Issue Type: Test
  Components: system tests
Affects Versions: 3.6.0
Reporter: Luke Chen


In running producer performance tool in system test for old client, we will try 
to run with the dev-branch's jar file, to make sure it is backward compatible, 
as described 
[here|https://github.com/apache/kafka/blob/trunk/tests/kafkatest/services/performance/producer_performance.py#L86-L88],.
 But since we're moving tools  

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15238) Connect workers can be disabled by DLQ related stuck admin client calls

[Yash Mayya (Jira)]

Yash Mayya created KAFKA-15238:
--

 Summary: Connect workers can be disabled by DLQ related stuck 
admin client calls
 Key: KAFKA-15238
 URL: https://issues.apache.org/jira/browse/KAFKA-15238
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Yash Mayya
Assignee: Yash Mayya


When Kafka Connect is run in distributed mode - if a sink connector's task is 
restarted (via a worker's REST API), the following sequence of steps will occur 
(on the DistributedHerder's thread):

 
 # The existing sink task will be stopped 
([ref|https://github.com/apache/kafka/blob/4981fa939d588645401619bfc3e321dc523d10e7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1367])
 # A new sink task will be started 
([ref|https://github.com/apache/kafka/blob/4981fa939d588645401619bfc3e321dc523d10e7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1867C40-L1867C40])
 # As a part of the above step, a new {{WorkerSinkTask}} will be instantiated 
([ref|https://github.com/apache/kafka/blob/4981fa939d588645401619bfc3e321dc523d10e7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L656-L663])
 # The DLQ reporter (see 
[KIP-298|https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect])
 for the sink task is also instantiated and configured as a part of this 
([ref|https://github.com/apache/kafka/blob/4981fa939d588645401619bfc3e321dc523d10e7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L1800])
 # The DLQ reporter setup involves two synchronous admin client calls to list 
topics and create the DLQ topic if it isn't already created 
([ref|https://github.com/apache/kafka/blob/4981fa939d588645401619bfc3e321dc523d10e7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/errors/DeadLetterQueueReporter.java#L84-L87])

 

All of these are occurring synchronously on the herder's tick thread - in this 
portion 
[here|https://github.com/apache/kafka/blob/4981fa939d588645401619bfc3e321dc523d10e7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L457-L469]
 where external requests are run. If the admin client call in the DLQ reporter 
setup step blocks for some time (due to auth failures and retries or network 
issues or whatever other reason), this can cause the Connect worker to become 
non-functional (REST API requests will timeout) and even fall out of the 
Connect cluster and become a zombie (since the tick thread also drives group 
membership functions).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15237) Implement `offsets.commit.timeout.ms` setting

[David Jacot (Jira)]

David Jacot created KAFKA-15237:
---

 Summary: Implement `offsets.commit.timeout.ms` setting
 Key: KAFKA-15237
 URL: https://issues.apache.org/jira/browse/KAFKA-15237
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Federico Valeri]

+1 (non binding)

Thanks
Fede


On Sun, Jul 23, 2023 at 6:30 PM Omnia Ibrahim  wrote:
>
> Hi everyone,
> I would like to open a vote for KIP-949. The proposal is here
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> .
> 
>
> Thanks
> Omnia