Re: Streams: clarification needed, checkpoint vs. position files

[Sophie Blee-Goldman]

Ah, good thing John showed up to clear things up -- sorry for the
misinformation
in my first reply (fake news!)

Generally speaking only public APIs *need* to be included in the KIP
document,
anything internal is an implementation detail as you said. In some cases it
may still be appropriate to touch on the actual implementation to fully
describe
the KIP but yu don't need to worry about that unless the KIP is making
drastic
changes to the internal architecture or the implementation has some effect
on the final semantics that user see, etc

On Mon, Nov 14, 2022 at 9:15 AM Nick Telford  wrote:

> Thank you both, the key point I was missing was that position files track
> the offsets of the *source* topics, whereas the checkpoint file tracks the
> offset(s) of the changelog topics.
>
> Do you know if I need to include interface/API changes to internal
> classes/interfaces (those under the
> org.apache.kafka.streams.processor.internals package) in a KIP, or are they
> considered implementation details?
>
> Cheers,
> Nick
>
> On Sat, 12 Nov 2022 at 03:59, John Roesler  wrote:
>
> > Hi all,
> >
> > Just to clarify: there actually is a position file. It was a small detail
> > of the IQv2 implementation to add it, otherwise a persistent store's
> > position would be lost after a restart.
> >
> > Otherwise, Sophie is right on the money. The checkpoint refers to an
> > offset in the changelog, while the position refers to offsets in the
> task's
> > input topics topics. So they are similar in function and structure, but
> > they refer to two different things.
> >
> > I agree that, given this, it doesn't seem like consolidating them (for
> > example, into one file) would be worth it. It would make the code more
> > complicated without deduping any information.
> >
> > I hope this helps, and look forward to what you're cooking up, Nick!
> > -John
> >
> > On 2022/11/12 00:50:27 Sophie Blee-Goldman wrote:
> > > Hey Nick,
> > >
> > > I haven't been following the new IQv2 work very closely so take this
> > with a
> > > grain of salt,
> > > but as far as I'm aware there's no such thing as "position files" --
> the
> > > Position is just an
> > > in-memory object and is related to a user's query against the state
> > store,
> > > whereas a
> > > checkpoint file reflects the current state of the store ie how much of
> > the
> > > changelog it
> > > contains.
> > >
> > > In other words while these might look like they do similar things, the
> > > actual usage and
> > > implementation of Positions vs checkpoint files is pretty much
> unrelated.
> > > So I don't think
> > > it would sense for Streams to try and consolidate these or replace one
> > with
> > > another.
> > >
> > > Hope this answers your question, and I'll ping John to make sure I'm
> not
> > > misleading
> > > you regarding the usage/intention of Positions
> > >
> > > Sophie
> > >
> > > On Fri, Nov 11, 2022 at 6:48 AM Nick Telford 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'm trying to understand how StateStores work internally for some
> > changes
> > > > that I plan to propose, and I'd like some clarification around
> > checkpoint
> > > > files and position files.
> > > >
> > > > It appears as though position files are relatively new, and were
> > created as
> > > > part of the IQv2 initiative, as a means to track the position of the
> > local
> > > > state store so that reads could be bound by particular positions?
> > > >
> > > > Checkpoint files look much older, and are managed by the Task itself
> > > > (actually, ProcessorStateManager). It looks like this is used
> > exclusively
> > > > for determining a) whether to restore a store, and b) which offsets
> to
> > > > restore from?
> > > >
> > > > If I've understood the above correctly, is there any scope to
> > potentially
> > > > replace checkpoint files with StateStore#position()?
> > > >
> > > > Regards,
> > > >
> > > > Nick
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Luke Chen]

Hi Sophie,

Here it is:
KIP-881:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
I've helped create an entry in under discussion table in KIP page.

Luke

On Wed, Nov 16, 2022 at 11:18 AM Sophie Blee-Goldman
 wrote:

> Rajini can you send me the link to this KIP? I don't see it on the KIP page
> yet
>
> Actually this is a good reminder to everyone: *please make sure you have
> added*
> *and/or updated all KIPs on the *KIP status page
> <
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >.
> After KIP freeze I will go over
> this page to compile the list of planned KIP content for 3.4
>
> Thanks all,
> Sophie
>
> On Tue, Nov 15, 2022 at 7:10 PM Sophie Blee-Goldman 
> wrote:
>
> > Thanks Rajini, we can include KIP-881 in the 3.4 release assuming it will
> > pass.
> > I'll add it to the release page
> >
> > On Tue, Nov 15, 2022 at 3:51 AM Rajini Sivaram 
> > wrote:
> >
> >> Hi Sophie,
> >>
> >> I was out of office and hence couldn't get voting started for KIP-881 in
> >> time. I will start the vote for the KIP today. If there are sufficient
> >> votes by tomorrow (16th Nov), can we include this KIP in 3.4, even
> though
> >> voting will only complete on the 17th? It is a small KIP, so we can
> merge
> >> by feature freeze.
> >>
> >> Thank you,
> >>
> >> Rajini
> >>
> >>
> >> On Thu, Nov 10, 2022 at 4:02 PM Sophie Blee-Goldman
> >>  wrote:
> >>
> >> > Hello again,
> >> >
> >> > This is a reminder that the KIP freeze deadline is approaching, all
> KIPs
> >> > must be voted
> >> > and accepted by *next Wednesday* *(the 16th)*
> >> >
> >> > Keep in mind that to allow for the full voting period, this means you
> >> must
> >> > kick off the
> >> > vote for your KIP no later than* next Monday* (*the 14th*).
> >> >
> >> > The feature freeze deadline will be 2 weeks after this, so make sure
> to
> >> get
> >> > your KIPs in!
> >> >
> >> > Best,
> >> > Sophie
> >> >
> >> > On Tue, Oct 18, 2022 at 2:01 PM Sophie Blee-Goldman <
> >> sop...@confluent.io>
> >> > wrote:
> >> >
> >> > > Hey all,
> >> > >
> >> > > I've created the release page for 3.4.0 with the current plan, which
> >> you
> >> > > can find here:
> >> > >
> >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0
> >> > >
> >> > > The freeze deadlines for this release are as follows:
> >> > >
> >> > > 1. KIP Freeze November 16th, 2022
> >> > > 2. Feature Freeze November 30th, 2022
> >> > > 3. Code Freeze December 7th, 2022
> >> > >
> >> > > Please take a look at the list of planned KIPs for 3.4.0 and let me
> >> know
> >> > > if you have any
> >> > > others that you are targeting in the upcoming release.
> >> > >
> >> > > Cheers,
> >> > > Sophie
> >> > >
> >> > > On Mon, Oct 10, 2022 at 9:22 AM Matthew Benedict de Detrich
> >> > >  wrote:
> >> > >
> >> > >> Thanks for volunteering!
> >> > >>
> >> > >> On Mon, 10 Oct 2022, 10:56 Bruno Cadonna, 
> >> wrote:
> >> > >>
> >> > >> > +1
> >> > >> >
> >> > >> > Thanks Sophie!
> >> > >> >
> >> > >> > Best,
> >> > >> > Bruno
> >> > >> >
> >> > >> > On 06.10.22 00:01, Sophie Blee-Goldman wrote:
> >> > >> > > Hey all,
> >> > >> > >
> >> > >> > > I'd like to volunteer as release manager for the next feature
> >> > release,
> >> > >> > > which will be Apache
> >> > >> > > Kafka 3.4.0. If that sounds good to everyone I'll update this
> >> thread
> >> > >> with
> >> > >> > > the release plan in the coming week.
> >> > >> > >
> >> > >> > > Cheers,
> >> > >> > > A. Sophie Blee-Goldman
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Sophie Blee-Goldman]

Rajini can you send me the link to this KIP? I don't see it on the KIP page
yet

Actually this is a good reminder to everyone: *please make sure you have
added*
*and/or updated all KIPs on the *KIP status page
.
After KIP freeze I will go over
this page to compile the list of planned KIP content for 3.4

Thanks all,
Sophie

On Tue, Nov 15, 2022 at 7:10 PM Sophie Blee-Goldman 
wrote:

> Thanks Rajini, we can include KIP-881 in the 3.4 release assuming it will
> pass.
> I'll add it to the release page
>
> On Tue, Nov 15, 2022 at 3:51 AM Rajini Sivaram 
> wrote:
>
>> Hi Sophie,
>>
>> I was out of office and hence couldn't get voting started for KIP-881 in
>> time. I will start the vote for the KIP today. If there are sufficient
>> votes by tomorrow (16th Nov), can we include this KIP in 3.4, even though
>> voting will only complete on the 17th? It is a small KIP, so we can merge
>> by feature freeze.
>>
>> Thank you,
>>
>> Rajini
>>
>>
>> On Thu, Nov 10, 2022 at 4:02 PM Sophie Blee-Goldman
>>  wrote:
>>
>> > Hello again,
>> >
>> > This is a reminder that the KIP freeze deadline is approaching, all KIPs
>> > must be voted
>> > and accepted by *next Wednesday* *(the 16th)*
>> >
>> > Keep in mind that to allow for the full voting period, this means you
>> must
>> > kick off the
>> > vote for your KIP no later than* next Monday* (*the 14th*).
>> >
>> > The feature freeze deadline will be 2 weeks after this, so make sure to
>> get
>> > your KIPs in!
>> >
>> > Best,
>> > Sophie
>> >
>> > On Tue, Oct 18, 2022 at 2:01 PM Sophie Blee-Goldman <
>> sop...@confluent.io>
>> > wrote:
>> >
>> > > Hey all,
>> > >
>> > > I've created the release page for 3.4.0 with the current plan, which
>> you
>> > > can find here:
>> > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0
>> > >
>> > > The freeze deadlines for this release are as follows:
>> > >
>> > > 1. KIP Freeze November 16th, 2022
>> > > 2. Feature Freeze November 30th, 2022
>> > > 3. Code Freeze December 7th, 2022
>> > >
>> > > Please take a look at the list of planned KIPs for 3.4.0 and let me
>> know
>> > > if you have any
>> > > others that you are targeting in the upcoming release.
>> > >
>> > > Cheers,
>> > > Sophie
>> > >
>> > > On Mon, Oct 10, 2022 at 9:22 AM Matthew Benedict de Detrich
>> > >  wrote:
>> > >
>> > >> Thanks for volunteering!
>> > >>
>> > >> On Mon, 10 Oct 2022, 10:56 Bruno Cadonna, 
>> wrote:
>> > >>
>> > >> > +1
>> > >> >
>> > >> > Thanks Sophie!
>> > >> >
>> > >> > Best,
>> > >> > Bruno
>> > >> >
>> > >> > On 06.10.22 00:01, Sophie Blee-Goldman wrote:
>> > >> > > Hey all,
>> > >> > >
>> > >> > > I'd like to volunteer as release manager for the next feature
>> > release,
>> > >> > > which will be Apache
>> > >> > > Kafka 3.4.0. If that sounds good to everyone I'll update this
>> thread
>> > >> with
>> > >> > > the release plan in the coming week.
>> > >> > >
>> > >> > > Cheers,
>> > >> > > A. Sophie Blee-Goldman
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Sophie Blee-Goldman]

Thanks Rajini, we can include KIP-881 in the 3.4 release assuming it will
pass.
I'll add it to the release page

On Tue, Nov 15, 2022 at 3:51 AM Rajini Sivaram 
wrote:

> Hi Sophie,
>
> I was out of office and hence couldn't get voting started for KIP-881 in
> time. I will start the vote for the KIP today. If there are sufficient
> votes by tomorrow (16th Nov), can we include this KIP in 3.4, even though
> voting will only complete on the 17th? It is a small KIP, so we can merge
> by feature freeze.
>
> Thank you,
>
> Rajini
>
>
> On Thu, Nov 10, 2022 at 4:02 PM Sophie Blee-Goldman
>  wrote:
>
> > Hello again,
> >
> > This is a reminder that the KIP freeze deadline is approaching, all KIPs
> > must be voted
> > and accepted by *next Wednesday* *(the 16th)*
> >
> > Keep in mind that to allow for the full voting period, this means you
> must
> > kick off the
> > vote for your KIP no later than* next Monday* (*the 14th*).
> >
> > The feature freeze deadline will be 2 weeks after this, so make sure to
> get
> > your KIPs in!
> >
> > Best,
> > Sophie
> >
> > On Tue, Oct 18, 2022 at 2:01 PM Sophie Blee-Goldman  >
> > wrote:
> >
> > > Hey all,
> > >
> > > I've created the release page for 3.4.0 with the current plan, which
> you
> > > can find here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0
> > >
> > > The freeze deadlines for this release are as follows:
> > >
> > > 1. KIP Freeze November 16th, 2022
> > > 2. Feature Freeze November 30th, 2022
> > > 3. Code Freeze December 7th, 2022
> > >
> > > Please take a look at the list of planned KIPs for 3.4.0 and let me
> know
> > > if you have any
> > > others that you are targeting in the upcoming release.
> > >
> > > Cheers,
> > > Sophie
> > >
> > > On Mon, Oct 10, 2022 at 9:22 AM Matthew Benedict de Detrich
> > >  wrote:
> > >
> > >> Thanks for volunteering!
> > >>
> > >> On Mon, 10 Oct 2022, 10:56 Bruno Cadonna,  wrote:
> > >>
> > >> > +1
> > >> >
> > >> > Thanks Sophie!
> > >> >
> > >> > Best,
> > >> > Bruno
> > >> >
> > >> > On 06.10.22 00:01, Sophie Blee-Goldman wrote:
> > >> > > Hey all,
> > >> > >
> > >> > > I'd like to volunteer as release manager for the next feature
> > release,
> > >> > > which will be Apache
> > >> > > Kafka 3.4.0. If that sounds good to everyone I'll update this
> thread
> > >> with
> > >> > > the release plan in the coming week.
> > >> > >
> > >> > > Cheers,
> > >> > > A. Sophie Blee-Goldman
> > >> > >
> > >> >
> > >>
> > >
> >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1355

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1354

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 428683 lines...]
[2022-11-16T00:22:13.614Z] [INFO] 

[2022-11-16T00:22:13.614Z] [INFO] Total time:  1.729 s
[2022-11-16T00:22:13.614Z] [INFO] Finished at: 2022-11-16T00:22:14Z
[2022-11-16T00:22:13.614Z] [INFO] 

[Pipeline] dir
[2022-11-16T00:22:14.153Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2@2/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2022-11-16T00:22:16.388Z] + mvn compile
[2022-11-16T00:22:17.356Z] [INFO] Scanning for projects...
[2022-11-16T00:22:17.356Z] [INFO] 
[2022-11-16T00:22:17.357Z] [INFO] -< 
streams.examples:streams.examples >--
[2022-11-16T00:22:17.357Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2022-11-16T00:22:17.357Z] [INFO] [ jar 
]-
[2022-11-16T00:22:18.325Z] [INFO] 
[2022-11-16T00:22:18.325Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2022-11-16T00:22:18.325Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2022-11-16T00:22:18.325Z] [INFO] Copying 1 resource
[2022-11-16T00:22:18.325Z] [INFO] 
[2022-11-16T00:22:18.325Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2022-11-16T00:22:18.325Z] [INFO] Changes detected - recompiling the module!
[2022-11-16T00:22:18.325Z] [INFO] Compiling 3 source files to 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2@2/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2022-11-16T00:22:19.294Z] [INFO] 

[2022-11-16T00:22:19.294Z] [INFO] BUILD SUCCESS
[2022-11-16T00:22:19.294Z] [INFO] 

[2022-11-16T00:22:19.294Z] [INFO] Total time:  2.109 s
[2022-11-16T00:22:19.294Z] [INFO] Finished at: 2022-11-16T00:22:20Z
[2022-11-16T00:22:19.294Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-11-16T00:22:20.767Z] > Task :core:classes
[2022-11-16T00:22:20.767Z] > Task :core:compileTestJava NO-SOURCE
[2022-11-16T00:22:51.481Z] > Task :core:compileTestScala
[2022-11-16T00:23:33.302Z] > Task :core:testClasses
[2022-11-16T00:23:49.857Z] > Task :streams:compileTestJava
[2022-11-16T00:23:49.857Z] > Task :streams:testClasses
[2022-11-16T00:23:49.857Z] > Task :streams:testJar
[2022-11-16T00:23:49.857Z] > Task :streams:testSrcJar
[2022-11-16T00:23:49.857Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-11-16T00:23:49.857Z] > Task :streams:publishToMavenLocal
[2022-11-16T00:23:49.857Z] 
[2022-11-16T00:23:49.857Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-11-16T00:23:49.857Z] 
[2022-11-16T00:23:49.857Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-11-16T00:23:49.857Z] 
[2022-11-16T00:23:49.857Z] See 
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
[2022-11-16T00:23:49.857Z] 
[2022-11-16T00:23:49.857Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2022-11-16T00:23:49.857Z] Please consult deprecation warnings for more details.
[2022-11-16T00:23:49.857Z] 
[2022-11-16T00:23:49.857Z] BUILD SUCCESSFUL in 3m 21s
[2022-11-16T00:23:49.857Z] 81 actionable tasks: 35 executed, 46 up-to-date
[Pipeline] sh
[2022-11-16T00:23:52.644Z] + grep ^version= gradle.properties
[2022-11-16T00:23:52.644Z] + cut -d= -f 2
[Pipeline] dir
[2022-11-16T00:23:53.366Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2022-11-16T00:23:55.607Z] + mvn clean install -Dgpg.skip
[2022-11-16T00:23:56.577Z] [INFO] Scanning for projects...
[2022-11-16T00:23:56.577Z] [INFO] 

[2022-11-16T00:23:56.577Z] [INFO] Reactor Build Order:
[2022-11-16T00:23:56.577Z] [INFO] 
[2022-11-16T00:23:56.577Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2022-11-16T00:23:56.577Z] [INFO] streams-quickstart-java   
 [maven-arche

Re: [DISCUSS] KIP-884: Add config to configure KafkaClientSupplier in Kafka Streams

[Hao Li]

Thanks for the questions Matthias!

1. I think we can check the config in the constructor which doesn't take
the client supplier as a parameter. This one:

public KafkaStreams(final Topology topology,
final Properties props) {
...
}

If users provide a client supplier in another constructor, the config won't
be checked and the provided one will be used, which is code would override
the config.

2. I'm fine with `default.client.supplier` and make it the
`DefaultKafkaClientSupplier`

Thanks,
Hao


On Tue, Nov 15, 2022 at 4:11 PM Matthias J. Sax  wrote:

> Thanks for the KIP Hao.
>
> What is the behavior if users set the config and also pass in a client
> supplier into the constructor?
>
> Following other config/API patterns we use, it seems the best thing
> would be if the code would overwrite the config?
>
> If we do this, should we change the config name to
> `default.client.supplier` and not make it `null`, but set the default
> supplier we use currently?
>
> This way, the config and code would behave the same as other configs
> like `default.timestamp.extractor` and similar.
>
>
> -Matthias
>
>
>
> On 11/15/22 3:35 PM, Hao Li wrote:
> > Hi all,
> >
> > I have submitted KIP-884 to add config to configure KafkaClientSupplier
> and
> > would like to start a discussion:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-884%3A+Add+config+to+configure+KafkaClientSupplier+in+Kafka+Streams
> >
> >
>


-- 
Thanks,
Hao

Re: [DISCUSS] KIP-884: Add config to configure KafkaClientSupplier in Kafka Streams

[Matthias J. Sax]

Thanks for the KIP Hao.

What is the behavior if users set the config and also pass in a client 
supplier into the constructor?


Following other config/API patterns we use, it seems the best thing 
would be if the code would overwrite the config?


If we do this, should we change the config name to 
`default.client.supplier` and not make it `null`, but set the default 
supplier we use currently?


This way, the config and code would behave the same as other configs 
like `default.timestamp.extractor` and similar.



-Matthias



On 11/15/22 3:35 PM, Hao Li wrote:

Hi all,

I have submitted KIP-884 to add config to configure KafkaClientSupplier and
would like to start a discussion:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-884%3A+Add+config+to+configure+KafkaClientSupplier+in+Kafka+Streams

[DISCUSS] KIP-884: Add config to configure KafkaClientSupplier in Kafka Streams

[Hao Li]

Hi all,

I have submitted KIP-884 to add config to configure KafkaClientSupplier and
would like to start a discussion:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-884%3A+Add+config+to+configure+KafkaClientSupplier+in+Kafka+Streams


-- 
Thanks,
Hao

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Jun Rao]

Hi, Rajini,

Thanks for the updated KIP. Just another minor comment. It would be useful
to list all RPC requests whose version needs to be bumped because of the
changes in ConsumerProtocolSubscription.

Jun

On Tue, Nov 15, 2022 at 3:45 AM Rajini Sivaram 
wrote:

> Hi David,
>
> Sorry, I was out of office and hence the delay in responding. Thanks for
> reviewing the KIP and answering Viktor's question (thanks for the review,
> Viktor).
>
> Responses below:
> 01)  I was in two minds about adding new assignors, because as you said,
> user experience is better if assignors used racks when available. But I was
> a bit concerned about changing the algorithm in existing applications which
> were already configuring `client.rack`. It felt less risky to add new
> assignor implementations instead. But we can retain existing logic if a)
> rack information is not available and b) racks have all partitions. So the
> only case where logic will be different is when rack information is
> available because consumers chose to use `client.rack` to benefit from
> improved locality, but racks only have a subset of partitions. It seems
> reasonable to make existing assignors rack-aware in this case to improve
> locality. I have updated the KIP. Will wait and see if there are any
> objections to this change.
>
> 02) Updated 1), so existing assignor classes will be used.
>
> 03) Updated the KIP to use version 3, thanks.
>
> If there are no concerns or further comments, I will start voting later
> today.
>
> Thank you,
>
> Rajini
>
>
> On Fri, Nov 4, 2022 at 9:58 AM David Jacot 
> wrote:
>
> > Hi Viktor,
> >
> > I can actually answer your question. KIP-848 already includes rack
> > awareness in the protocol. It is actually the other way around, this
> > KIP takes the idea from KIP-848 to implement it in the current
> > protocol in order to realize the benefits sooner. The new protocol
> > will take a while to be implemented.
> >
> > Best,
> > David
> >
> > On Fri, Nov 4, 2022 at 10:55 AM David Jacot  wrote:
> > >
> > > Hi Rajini,
> > >
> > > Thanks for the KIP. I have a few questions/comments:
> > >
> > > 01. If I understood correctly, the plan is to add new assignors which
> > > are rack aware. Is this right? I wonder if it is a judicious choice
> > > here. The main drawback is that clients must be configured correctly
> > > in order to get the benefits. From a user experience perspective, it
> > > would be much better if we would only require our users to set
> > > client.rack. However, I understand the argument of keeping the
> > > existing assignors as-is in order to limit the risk but it also means
> > > that we will have to maintain multiple assignors with a somewhat
> > > similar core logic (within a rack). What do you think?
> > >
> > > 02. If we proceed with new rack-aware assignors, we should mention
> > > their fully qualified names in the KIP as they will become part of our
> > > public interfaces.
> > >
> > > 03. KIP-792 has already introduced version 2 of the subscription. The
> > > KIP is accepted but the PR is not merged yet. This KIP should use
> > > version 3.
> > >
> > > Best,
> > > David
> > >
> > > On Thu, Nov 3, 2022 at 5:58 PM Viktor Somogyi-Vass
> > >  wrote:
> > > >
> > > > Hi Rajini,
> > > >
> > > > If I understand correctly, the client.rack config would stay
> supported
> > > > after KIP-848 but does it expand the scope of that KIP too with this
> > > > config? I mean that currently you propose
> ConsumerProtocolSubscription
> > to
> > > > be used but this protocol won't be available and we need to transfer
> > the
> > > > config to the coordinator via other means. Should this be added to
> > that KIP?
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > > On Wed, Nov 2, 2022 at 9:50 PM Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thank you for the review. Yes, we should add rack id to
> > Subscription, had
> > > > > missed that part. Updated the KIP, thank you for pointing that out!
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Wed, Nov 2, 2022 at 7:06 PM Jun Rao 
> > wrote:
> > > > >
> > > > > > Hi, Rajini,
> > > > > >
> > > > > > Thanks for the KIP. Just one comment.
> > > > > >
> > > > > > Should we add rackId to GroupSubscription.Subscription for each
> > member?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Nov 2, 2022 at 4:57 AM Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have submitted KIP-881 to implement rack-aware partition
> > assignment
> > > > > for
> > > > > > > consumers:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
> > > > > > > .
> > > > > > > It adds rack id to the consumer group protocol to propagate
> rack
> > > > > > > information so t

[jira] [Resolved] (KAFKA-14360) Documentation: Streams Security page has broken links

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14360.
-
Resolution: Fixed

> Documentation: Streams Security page has broken links
> -
>
> Key: KAFKA-14360
> URL: https://issues.apache.org/jira/browse/KAFKA-14360
> Project: Kafka
>  Issue Type: Bug
>Reporter: Adam
>Assignee: Shay Lin
>Priority: Major
> Fix For: 3.4.0
>
>
> A number of links on the 'Streams Security' page are 404-ing
> https://kafka.apache.org/documentation/streams/developer-guide/security.html
> * Kafka’s security features 
> https://kafka.apache.org/documentation/documentation.html#security
> * Java Producer and Consumer API 
> https://kafka.apache.org/documentation/clients/index.html#kafka-clients



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12495) Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-12495.
---
Resolution: Fixed

> Unbalanced connectors/tasks distribution will happen in Connect's incremental 
> cooperative assignor
> --
>
> Key: KAFKA-12495
> URL: https://issues.apache.org/jira/browse/KAFKA-12495
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Luke Chen
>Assignee: Sagar Rao
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: image-2021-03-18-15-04-57-854.png, 
> image-2021-03-18-15-05-52-557.png, image-2021-03-18-15-07-27-103.png
>
>
> In Kafka Connect, we implement incremental cooperative rebalance algorithm 
> based on KIP-415 
> ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect].
>  However, we have a bad assumption in the algorithm implementation, which is: 
> after revoking rebalance completed, the member(worker) count will be the same 
> as the previous round of reblance.
>  
> Let's take a look at the example in the KIP-415:
> !image-2021-03-18-15-07-27-103.png|width=441,height=556!
> It works well for most cases. But what if W4 added after 1st rebalance 
> completed and before 2nd rebalance started? Let's see what will happened? 
> Let's see this example: (we'll use 10 tasks here):
>  
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, 
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4, 
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, 
> BT4, BT5]
> W1 becomes leader
> W1 computes and sends assignments:
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [AT4, AT5, BC0, BT1, 
> BT2, BT4, BT4, BT5])
> W2(delay: 0, assigned: [], revoked: [])
> W3(delay: 0, assigned: [], revoked: [])
> W1 stops revoked resources
> W1 rejoins with assignment: [AC0, AT1, AT2, AT3]
> Rebalance is triggered
> W2 joins with assignment: []
> W3 joins with assignment: []
> // one more member joined
> W4 joins with assignment: []
> W1 becomes leader
> W1 computes and sends assignments:
> // We assigned all the previous revoked Connectors/Tasks to the new member, 
> but we didn't revoke any more C/T in this round, which cause unbalanced 
> distribution
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [])
> W2(delay: 0, assigned: [AT4, AT5, BC0], revoked: [])
> W2(delay: 0, assigned: [BT1, BT2, BT4], revoked: [])
> W2(delay: 0, assigned: [BT4, BT5], revoked: [])
> {code}
> Because we didn't allow to do consecutive revoke in two consecutive 
> rebalances (under the same leader), we will have this uneven distribution 
> under this situation. We should allow consecutive rebalance to have another 
> round of revocation to revoke the C/T to the other members in this case.
> expected:
> {code:java}
> Initial group and assignment: W1([AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, 
> BT2, BT4, BT4, BT5])
> Config topic contains: AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, BT4, 
> BT5
> W1 is current leader
> W2 joins with assignment: []
> Rebalance is triggered
> W3 joins while rebalance is still active with assignment: []
> W1 joins with assignment: [AC0, AT1, AT2, AT3, AT4, AT5, BC0, BT1, BT2, BT4, 
> BT4, BT5]
> W1 becomes leader
> W1 computes and sends assignments:
> W1(delay: 0, assigned: [AC0, AT1, AT2, AT3], revoked: [AT4, AT5, BC0, BT1, 
> BT2, BT4, BT4, BT5])
> W2(delay: 0, assigned: [], revoked: [])
> W3(delay: 0, assigned: [], revoked: [])
> W1 stops revoked resources
> W1 rejoins with assignment: [AC0, AT1, AT2, AT3]
> Rebalance is triggered
> W2 joins with assignment: []
> W3 joins with assignment: []
> // one more member joined
> W4 joins with assignment: []
> W1 becomes leader
> W1 computes and sends assignments:
> // We assigned all the previous revoked Connectors/Tasks to the new member, 
> **and also revoke some C/T** 
> W1(delay: 0, assigned: [AC0, AT1, AT2], revoked: [AT3])
> W2(delay: 0, assigned: [AT4, AT5, BC0], revoked: [])
> W3(delay: 0, assigned: [BT1, BT2, BT4], revoked: [])
> W4(delay: 0, assigned: [BT4, BT5], revoked: [])
> // another round of rebalance to assign the new revoked C/T to the other 
> members
> W1 rejoins with assignment: [AC0, AT1, AT2] 
> Rebalance is triggered 
> W2 joins with assignment: [AT4, AT5, BC0] 
> W3 joins with assignment: [BT1, BT2, BT4]
> W4 joins with assignment: [BT4, BT5]
> 

[jira] [Resolved] (KAFKA-13764) Potential improvements for Connect incremental rebalancing logic

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13764.
---
Resolution: Fixed

> Potential improvements for Connect incremental rebalancing logic
> 
>
> Key: KAFKA-13764
> URL: https://issues.apache.org/jira/browse/KAFKA-13764
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Minor
>
> There are a few small changes that we might make to the incremental 
> rebalancing logic for Kafka Connect to improve distribution of connectors and 
> tasks across a cluster and address potential bugs:
>  # During assignment, assign new connectors and tasks across the cluster 
> before calculating revocations that may be necessary in order to balance the 
> cluster. This way, we can potentially skip a round of revocation by using 
> newly-created connectors and tasks to balance out the cluster.
>  # Perform connector and task revocation in more cases, such as when one or 
> more connectors are reconfigured to use fewer tasks, which can possibly lead 
> to an imbalanced cluster.
>  # Fix [this 
> line|https://github.com/apache/kafka/blob/06ca4850c5b2b12e972f48e03fe4f9c1032f9a3e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java#L248]
>  to use the same aggregation logic that's used 
> [here|https://github.com/apache/kafka/blob/06ca4850c5b2b12e972f48e03fe4f9c1032f9a3e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java#L273-L281]
>  in order to avoid overwriting map values when they should be combined 
> instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Maulin Vasavada]

+1 (non-binding).

Makes sense, Rajini. This would be a great addition.

On Tue, Nov 15, 2022 at 10:55 AM Rajini Sivaram 
wrote:

> Hi all,
>
> I would like to start vote on KIP-881 to support rack-aware partition
> assignment for Kafka consumers:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
> .
>
> Thank you!
>
> Regards,
>
> Rajini
>

[jira] [Resolved] (KAFKA-14309) Kafka Streams upgrade tests do not cover for FK-joins

[Lucas Brutschy (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Brutschy resolved KAFKA-14309.

Resolution: Fixed

> Kafka Streams upgrade tests do not cover for FK-joins
> -
>
> Key: KAFKA-14309
> URL: https://issues.apache.org/jira/browse/KAFKA-14309
> Project: Kafka
>  Issue Type: Bug
>Reporter: Lucas Brutschy
>Assignee: Lucas Brutschy
>Priority: Major
>
> The current streams upgrade system test for FK joins inserts the production 
> of foreign key data and an actual foreign key join in every version of 
> SmokeTestDriver except for the latest. The effect was that FK join upgrades 
> are not tested at all, since no FK join code is executed after the bounce in 
> the system test.
> We should enable the FK-join code in the system test.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Rajini Sivaram]

Hi all,

I would like to start vote on KIP-881 to support rack-aware partition
assignment for Kafka consumers:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
.

Thank you!

Regards,

Rajini

[jira] [Created] (KAFKA-14393) Change default size based retention for the cluster metadata partition

[Jira]

José Armando García Sancio created KAFKA-14393:
--

 Summary: Change default size based retention for the cluster 
metadata partition
 Key: KAFKA-14393
 URL: https://issues.apache.org/jira/browse/KAFKA-14393
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft
Reporter: José Armando García Sancio
Assignee: José Armando García Sancio
 Fix For: 3.4.0


Implement this part of the KIP:

The default value for the {{metadata.max.retention.bytes}}  will be changed 
from {{-1}} (disabled)  to {{100 * 1024 * 1024}}  (100MB).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-883: Add delete callback method to Connector API

[Sagar]

Hey Hector,

Thanks for the KIP. I have a minor suggestion in terms of naming. Since
this is a callback method, would it make sense to call it onDelete()?

Also, the failure scenarios discussed by Greg would need handling. Among
other things, I like the idea of having a timeout for graceful shutdown or
else try a force shutdown. What do you think about that approach?

Thanks!
Sagar.

On Sat, Nov 12, 2022 at 1:53 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> Thanks Greg for taking your time to review not just the KIP but also the
> PR.
>
> 1. You made very valid points regarding the behavior of the destroy()
> callback for connectors that don't follow the happy path. After thinking
> about it, I decided to tweak the implementation a bit and have the
> destroy() method be called during the worker shutdown: this means it will
> share the same guarantees the connector#stop() method has. An alternative
> implementation can be to have an overloaded connector#stop(boolean deleted)
> method that signals a connector that it is being stopped due to deletion,
> but I think that having a separate destroy() method provides clearer
> semantics.
>
> I'll make sure to ammend the KIP with these details.
>
> 3. Without going too deep on the types of operations that can be performed
> by a connector when it's being deleted, I can imagine the
> org.apache.kafka.connect.source.SourceConnector base class having a default
> implementation that deletes the connector's offsets automatically
> (controlled by a property); this is in the context of KIP-875 (first-class
> offsets support in Kafka Connect). Similar behaviors can be introduced for
> the SinkConnector, however I'm not sure if this KIP is the right place to
> discuss all the possibilities, or if we shoold keeping it more
> narrow-focused on  providing a callback mechanism for when connectors are
> deleted, and what the expectations are around this newly introduced method.
> What do you think?
>
>
> From: dev@kafka.apache.org At: 11/09/22 16:55:04 UTC-5:00To:
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-883: Add delete callback method to Connector API
>
> Hi Hector,
>
> Thanks for the KIP!
>
> This is certainly missing functionality from the native Connect framework,
> and we should try to make it possible to inform connectors about this part
> of their lifecycle.
> However, as with most functionality that was left out of the initial
> implementation of the framework, the details are more challenging to work
> out.
>
> 1. What happens when the destroy call throws an error, how does the
> framework respond?
>
> This is unspecified in the KIP, and it appears that your proposed changes
> could cause the herder to fail.
> From the perspective of operators & connector developers, what is a
> reasonable expectation to have for failure of a destroy?
> I could see operators wanting both a graceful-delete to make use of this
> new feature, and a force-delete for when the graceful-delete fails.
> A connector developer could choose to swallow all errors encountered, or
> fail-fast to indicate to the operator that there is an issue with the
> graceful-delete flow.
> If the alternative is crashing the herder, connector developers may choose
> to hide serious errors, which is undesirable.
>
> 2. What happens when the destroy() call takes a long time to complete, or
> is interrupted?
>
> It appears that your implementation serially destroy()s each appropriate
> connector, and may prevent the herder thread from making progress while the
> operation is ongoing.
> We have previously had to patch Connect to perform all connector and task
> operations on a background thread, because some connector method
> implementations can stall indefinitely.
> Connect also has the notion of "cancelling" a connector/task if a graceful
> shutdown timeout operation takes too long. Perhaps some of that design or
> machinery may be useful to protect this method call as well.
>
> More specific to the destroy() call itself, what happens when a connector
> completes part of a destroy operation and then cannot complete the
> remainder, either due to timing out or a worker crashing?
> What is the contract with the connector developer about this method? Is the
> destroy() only started exactly once during the lifetime of the connector,
> or may it be retried?
>
> 3. What should be considered a reasonable custom implementation of the
> destroy() call? What resources should it clean up by default?
>
> I think we can broadly categorize the state a connector mutates among the
> following
> * Framework-managed state (e.g. source offsets, consumer offsets)
> * Implementation detail state (e.g. debezium db history topic, audit
> tables, temporary accounts)
> * Third party system data (e.g. the actual data being written by a sink
> connector)
> * Third party system metadata (e.g. tables in a database, delivery
> receipts, permissions)
>
> I think it's apparent that the framework-mana

[jira] [Created] (KAFKA-14392) KRaft should comment controller.socket.timeout.ms <= broker.session.timeout.ms

[Ron Dagostino (Jira)]

Ron Dagostino created KAFKA-14392:
-

 Summary: KRaft should comment controller.socket.timeout.ms <= 
broker.session.timeout.ms
 Key: KAFKA-14392
 URL: https://issues.apache.org/jira/browse/KAFKA-14392
 Project: Kafka
  Issue Type: Improvement
Reporter: Ron Dagostino
Assignee: Ron Dagostino


KRaft brokers maintain their liveness in the cluster by sending 
BROKER_HEARTBEAT requests to the active controller; the active controller 
fences a broker if it doesn't receive a heartbeat request from that broker 
within the period defined by `broker.session.timeout.ms`.  The broker should 
use a request timeout for its BROKER_HEARTBEAT requests that is not larger than 
the session timeout being used by the controller; doing so creates the 
possibility that upon controller failover the broker might not cancel an 
existing heartbeat request in time and then subsequently heartbeat to the new 
controller to maintain an uninterrupted session in the cluster.  In other 
words, a failure of the active controller could result in under-replicated (or 
under-min ISR) partitions simply due to a delay in brokers heartbeating to the 
new controller.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-7206) Enable batching in FindCoordinator

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-7206.
---
Fix Version/s: 3.0.0
   Resolution: Duplicate

Duplicate of https://issues.apache.org/jira/browse/KAFKA-12663

 

> Enable batching in FindCoordinator
> --
>
> Key: KAFKA-7206
> URL: https://issues.apache.org/jira/browse/KAFKA-7206
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Reporter: Yishun Guan
>Assignee: Yishun Guan
>Priority: Critical
>  Labels: needs-discussion, needs-kip, newbie++
> Fix For: 3.0.0
>
>
> To quote [~guozhang] :
> "The proposal is that, we extend FindCoordinatorRequest to have multiple 
> consumer ids: today each FindCoordinatorRequest only contains a single 
> consumer id, so in our scenario we need to send N request for N consumer 
> groups still. If we can request for coordinators in a single request, then 
> the workflow could be simplified to:
>  # send a single FindCoordinatorRequest to a broker asking for coordinators 
> of all consumer groups.
>  1.a) note that the response may still succeed in finding some coordinators 
> while error on others, and we need to handle them on that granularity (see 
> below).
>  # and then for the collected coordinator, group them by coordinator id and 
> send one request per coordinator destination.
> Note that this change would require the version to be bumped up, to 
> FIND_COORDINATOR_REQUEST_V3 for such protocol changes, also the RESPONSE 
> version should be bumped up in order to include multiple coordinators."
> A KIP is needed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request the account for Kafka Jira

[Tom Bentley]

Hi Simon,

Usernames have to be all lower case ascii letters and digits, so I used
simonzhang. You should have a password reset email where you can complete
the account creation.

Thanks for your interest in Apache Kafka.

On Tue, 15 Nov 2022 at 15:25, zsy112371  wrote:

> Hi team,
> I would like to request an account for the kafka jira.
>
> username: Simon Zhang
> display name: Simon Zhang
> email address: zsy112...@163.com
>
> Thank you
>
>

Request the account for Kafka Jira

[zsy112371]

Hi team,
I would like to request an account for the kafka jira.

username: Simon Zhang
display name: Simon Zhang
email address: zsy112...@163.com

Thank you

[jira] [Resolved] (KAFKA-14389) CVE-2022-34917 | Fixed in 3.3.1

[masood (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

masood resolved KAFKA-14389.

Resolution: Fixed

> CVE-2022-34917 | Fixed in 3.3.1
> ---
>
> Key: KAFKA-14389
> URL: https://issues.apache.org/jira/browse/KAFKA-14389
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.3.1
>Reporter: masood
>Priority: Major
>
> the following link has not been updated with v3.3.1. 
> [https://kafka.apache.org/cve-list#CVE-2022-34917]
> Its says the fixed is available until 3.2.3, so does it mean this CVE is 
> re-introduced in the v3.3.1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14391) Add ConsumerGroupHeartbeat API

[David Jacot (Jira)]

David Jacot created KAFKA-14391:
---

 Summary: Add ConsumerGroupHeartbeat API
 Key: KAFKA-14391
 URL: https://issues.apache.org/jira/browse/KAFKA-14391
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot


Add ConsumerGroupHeartbeat API and gate it by a hidden feature flag.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1353

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14324) [CVE-2018-25032] introduced by rocksdbjni:6.29.4.1

[Christo Lolov (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christo Lolov resolved KAFKA-14324.
---
Resolution: Fixed

> [CVE-2018-25032] introduced by rocksdbjni:6.29.4.1
> --
>
> Key: KAFKA-14324
> URL: https://issues.apache.org/jira/browse/KAFKA-14324
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.1.2, 3.2.3, 3.3.1
>Reporter: VZhang
>Assignee: Christo Lolov
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: 6.29.4.1_to_7.1.2_compat_report.html, 
> 6.29.4.1_to_7.7.3_compat_report.html
>
>
> Hi, Team
> There is an old CVE introduced by rocksdbjni-6.29.4.1, which has already been 
> fixed by 
> [https://github.com/facebook/rocksdb/commit/5dbdb197f19644d3f53f75781a3ef56e4387134b]
> [https://nvd.nist.gov/vuln/detail/cve-2018-25032]
> *Current Description:* 
> zlib before 1.2.12 allows memory corruption when deflating (i.e., when 
> compressing) if the input has many distant matches.
> CVE-2018-25032 - CVSS Score:{*}7.5{*} (v3.0) (zlib-1.2.11)
> Please help to upgrade the rocksdb.
> Thanks



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Rajini Sivaram]

Hi Sophie,

I was out of office and hence couldn't get voting started for KIP-881 in
time. I will start the vote for the KIP today. If there are sufficient
votes by tomorrow (16th Nov), can we include this KIP in 3.4, even though
voting will only complete on the 17th? It is a small KIP, so we can merge
by feature freeze.

Thank you,

Rajini


On Thu, Nov 10, 2022 at 4:02 PM Sophie Blee-Goldman
 wrote:

> Hello again,
>
> This is a reminder that the KIP freeze deadline is approaching, all KIPs
> must be voted
> and accepted by *next Wednesday* *(the 16th)*
>
> Keep in mind that to allow for the full voting period, this means you must
> kick off the
> vote for your KIP no later than* next Monday* (*the 14th*).
>
> The feature freeze deadline will be 2 weeks after this, so make sure to get
> your KIPs in!
>
> Best,
> Sophie
>
> On Tue, Oct 18, 2022 at 2:01 PM Sophie Blee-Goldman 
> wrote:
>
> > Hey all,
> >
> > I've created the release page for 3.4.0 with the current plan, which you
> > can find here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0
> >
> > The freeze deadlines for this release are as follows:
> >
> > 1. KIP Freeze November 16th, 2022
> > 2. Feature Freeze November 30th, 2022
> > 3. Code Freeze December 7th, 2022
> >
> > Please take a look at the list of planned KIPs for 3.4.0 and let me know
> > if you have any
> > others that you are targeting in the upcoming release.
> >
> > Cheers,
> > Sophie
> >
> > On Mon, Oct 10, 2022 at 9:22 AM Matthew Benedict de Detrich
> >  wrote:
> >
> >> Thanks for volunteering!
> >>
> >> On Mon, 10 Oct 2022, 10:56 Bruno Cadonna,  wrote:
> >>
> >> > +1
> >> >
> >> > Thanks Sophie!
> >> >
> >> > Best,
> >> > Bruno
> >> >
> >> > On 06.10.22 00:01, Sophie Blee-Goldman wrote:
> >> > > Hey all,
> >> > >
> >> > > I'd like to volunteer as release manager for the next feature
> release,
> >> > > which will be Apache
> >> > > Kafka 3.4.0. If that sounds good to everyone I'll update this thread
> >> with
> >> > > the release plan in the coming week.
> >> > >
> >> > > Cheers,
> >> > > A. Sophie Blee-Goldman
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Rajini Sivaram]

Hi David,

Sorry, I was out of office and hence the delay in responding. Thanks for
reviewing the KIP and answering Viktor's question (thanks for the review,
Viktor).

Responses below:
01)  I was in two minds about adding new assignors, because as you said,
user experience is better if assignors used racks when available. But I was
a bit concerned about changing the algorithm in existing applications which
were already configuring `client.rack`. It felt less risky to add new
assignor implementations instead. But we can retain existing logic if a)
rack information is not available and b) racks have all partitions. So the
only case where logic will be different is when rack information is
available because consumers chose to use `client.rack` to benefit from
improved locality, but racks only have a subset of partitions. It seems
reasonable to make existing assignors rack-aware in this case to improve
locality. I have updated the KIP. Will wait and see if there are any
objections to this change.

02) Updated 1), so existing assignor classes will be used.

03) Updated the KIP to use version 3, thanks.

If there are no concerns or further comments, I will start voting later
today.

Thank you,

Rajini


On Fri, Nov 4, 2022 at 9:58 AM David Jacot 
wrote:

> Hi Viktor,
>
> I can actually answer your question. KIP-848 already includes rack
> awareness in the protocol. It is actually the other way around, this
> KIP takes the idea from KIP-848 to implement it in the current
> protocol in order to realize the benefits sooner. The new protocol
> will take a while to be implemented.
>
> Best,
> David
>
> On Fri, Nov 4, 2022 at 10:55 AM David Jacot  wrote:
> >
> > Hi Rajini,
> >
> > Thanks for the KIP. I have a few questions/comments:
> >
> > 01. If I understood correctly, the plan is to add new assignors which
> > are rack aware. Is this right? I wonder if it is a judicious choice
> > here. The main drawback is that clients must be configured correctly
> > in order to get the benefits. From a user experience perspective, it
> > would be much better if we would only require our users to set
> > client.rack. However, I understand the argument of keeping the
> > existing assignors as-is in order to limit the risk but it also means
> > that we will have to maintain multiple assignors with a somewhat
> > similar core logic (within a rack). What do you think?
> >
> > 02. If we proceed with new rack-aware assignors, we should mention
> > their fully qualified names in the KIP as they will become part of our
> > public interfaces.
> >
> > 03. KIP-792 has already introduced version 2 of the subscription. The
> > KIP is accepted but the PR is not merged yet. This KIP should use
> > version 3.
> >
> > Best,
> > David
> >
> > On Thu, Nov 3, 2022 at 5:58 PM Viktor Somogyi-Vass
> >  wrote:
> > >
> > > Hi Rajini,
> > >
> > > If I understand correctly, the client.rack config would stay supported
> > > after KIP-848 but does it expand the scope of that KIP too with this
> > > config? I mean that currently you propose ConsumerProtocolSubscription
> to
> > > be used but this protocol won't be available and we need to transfer
> the
> > > config to the coordinator via other means. Should this be added to
> that KIP?
> > >
> > > Thanks,
> > > Viktor
> > >
> > > On Wed, Nov 2, 2022 at 9:50 PM Rajini Sivaram  >
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thank you for the review. Yes, we should add rack id to
> Subscription, had
> > > > missed that part. Updated the KIP, thank you for pointing that out!
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Wed, Nov 2, 2022 at 7:06 PM Jun Rao 
> wrote:
> > > >
> > > > > Hi, Rajini,
> > > > >
> > > > > Thanks for the KIP. Just one comment.
> > > > >
> > > > > Should we add rackId to GroupSubscription.Subscription for each
> member?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Nov 2, 2022 at 4:57 AM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have submitted KIP-881 to implement rack-aware partition
> assignment
> > > > for
> > > > > > consumers:
> > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
> > > > > > .
> > > > > > It adds rack id to the consumer group protocol to propagate rack
> > > > > > information so that rack-aware assignors can be added to benefit
> from
> > > > > > locality.
> > > > > >
> > > > > > Feedback and suggestions are welcome!
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > >
>

[jira] [Created] (KAFKA-14390) Kafka and Zookeeper with FIPS SASL has error

[lkgen (Jira)]

lkgen created KAFKA-14390:
-

 Summary: Kafka and Zookeeper with FIPS SASL has error
 Key: KAFKA-14390
 URL: https://issues.apache.org/jira/browse/KAFKA-14390
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 3.3.1, 2.7.1
Reporter: lkgen


When setting Zookeeper to work with SASL plain and Kafka to connect to 
zookeeper with SASL and Java has FIPS enabled on a FIPS enabled Redhat machine
Kafka cannot connect with zookeeper

Used Apache Kafka 3.3.1 for both zookeeper and client
Operating system Redhat 8
Java 11 installed using yum install java-11-openjdk
set fips mode using command as root
fips-mode-setup --enable
rebooted machine using
systemctl reboot
verified fips mode with
fips-mode-setup --check

set JAVA_HOME to /usr/lib/jvm/jre-11

in config/zookeeper.properties, added properties
admin.enableServer = false
authProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme = sasl

in config/server.properties, added properties
advertised.listeners = SASL_PLAINTEXT://localhost:9092
listeners = SASL_PLAINTEXT://:9092
security.inter.broker.protocol = SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol = PLAIN
sasl.enabled.mechanisms = PLAIN
super.users = User:admin

$HOME/zookeeper_jaas.conf value:
Server {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="0x572f372b5da34874a3fdf4d8002f"
    user_admin="0x572f372b5da34874a3fdf4d8002f";
};

$HOME/server_jaas.conf value:
KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="0x572f372b5da34874a3fdf4d8002f"
    user_admin="0x572f372b5da34874a3fdf4d8002f";
};
Client {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="0x572f372b5da34874a3fdf4d8002f";
};

for zookeeper, in kafka directory ran
setenv JAVA_HOME /usr/lib/jvm/jre-11
setenv KAFKA_OPTS -Djava.security.auth.login.config=$HOME/zookeeper_jaas.conf
./bin/zookeeper-server-start.sh ./config/zookeeper.properties

for kafka, in kafka directory ran
setenv JAVA_HOME /usr/lib/jvm/jre-11
setenv KAFKA_OPTS -Djava.security.auth.login.config=$HOME/server_jaas.conf
./bin/kafka-server-start.sh ./config/server.properties

Kafka exits with SASL error
[2022-11-15 13:21:04,344] ERROR SASL authentication with Zookeeper Quorum 
member failed. (org.apache.zookeeper.ClientCnxn)
javax.security.sasl.SaslException: saslClient failed to initialize properly: 
it's null.
        at 
org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:399)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1220)
[2022-11-15 13:21:04,348] INFO EventThread shut down for session: 
0x1000575ab11 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 13:21:04,348] INFO [ZooKeeperClient Kafka server] Closing. 
(kafka.zookeeper.ZooKeeperClient)
[2022-11-15 13:21:04,351] INFO [ZooKeeperClient Kafka server] Closed. 
(kafka.zookeeper.ZooKeeperClient)
[2022-11-15 13:21:04,355] ERROR Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before 
or while waiting for connection
        at 
kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:260)
        at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:108)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1980)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:503)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:203)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2022-11-15 13:21:04,356] INFO shutting down (kafka.server.KafkaServer)
[2022-11-15 13:21:04,363] INFO App info kafka.server for 0 unregistered 
(org.apache.kafka.common.utils.AppInfoParser)
[2022-11-15 13:21:04,363] INFO shut down completed (kafka.server.KafkaServer)
[2022-11-15 13:21:04,363] ERROR Exiting Kafka due to fatal exception during 
startup. (kafka.Kafka$)
kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before 
or while waiting for connection
        at 
kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:260)
        at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:108)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1980)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:503)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:203)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2022-11-15 13:21:04,368] INFO shutting down (kafka.server.KafkaServer)

When removing FIPS by changing the java conf/security/java.security to have
security.useSystemPropertiesFile=false

The problem does not happen but this property 

[jira] [Resolved] (KAFKA-13401) Introduce a new Interface to manage Kafka resources in MM2

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13401.

Resolution: Fixed

> Introduce a new Interface to manage Kafka resources in MM2
> --
>
> Key: KAFKA-13401
> URL: https://issues.apache.org/jira/browse/KAFKA-13401
> Project: Kafka
>  Issue Type: Improvement
>  Components: mirrormaker
>Reporter: Omnia Ibrahim
>Assignee: Omnia Ibrahim
>Priority: Major
> Fix For: 3.4.0
>
>
> MM2 uses AdminClients to create and modify topics; while this simplifies the 
> implementation of MM2 it has the following problems 
>  * It assumes that whoever runs MM2 must have admin access to both source and 
> destination clusters. This assumption is not necessarily valid all the time.
>  * It creates conflict in use-cases where centralised systems or tools manage 
> Kafka resources. 
> It would be easier if customers could provide how they want to manage Kafka 
> topics and configs through admin client or using their centralised system or 
> tools. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)