[VOTE] KIP-808: Add support for unix epoch precision in TimestampConverter SMT

2022-01-18 Thread Julien Chanaud
Hi everyone,

I'd like to start a vote for KIP-808: Add support for unix epoch precision
in TimestampConverter SMT

https://cwiki.apache.org/confluence/x/GJuqCw

Thanks for your help,

Julien


[jira] [Created] (KAFKA-13598) producer config didn't throw exception when retries and acks config changed

2022-01-18 Thread Luke Chen (Jira)
Luke Chen created KAFKA-13598:
-

 Summary: producer config didn't throw exception when retries and 
acks config changed
 Key: KAFKA-13598
 URL: https://issues.apache.org/jira/browse/KAFKA-13598
 Project: Kafka
  Issue Type: Bug
  Components: clients, config
Affects Versions: 3.0.0, 3.1.0
Reporter: Luke Chen
Assignee: Luke Chen






--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [VOTE] 3.1.0 RC1

2022-01-18 Thread Jakub Scholz
+1 (non-binding). I used the staged Scala 2.13 binaries and the staging
Maven repository to run my tests. All seems to work fine, no issues found.

Thanks
Jakub

On Wed, Jan 12, 2022 at 1:59 PM David Jacot  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.1.0.
>
> * Apache Kafka supports Java 17
> * The FetchRequest supports Topic IDs (KIP-516)
> * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
> * Add broker count metrics (KIP-748)
> * Differentiate consistently metric latency measured in millis and
> nanos (KIP-773)
> * The eager rebalance protocol is deprecated (KAFKA-13439)
> * Add TaskId field to StreamsException (KIP-783)
> * Custom partitioners in foreign-key joins (KIP-775)
> * Fetch/findSessions queries with open endpoints for
> SessionStore/WindowStore (KIP-766)
> * Range queries with open endpoints (KIP-763)
> * Add total blocked time metric to Streams (KIP-761)
> * Add additional configuration to control MirrorMaker2 internal topics
> naming convention (KIP-690)
>
> Release notes for the 3.1.0 release:
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, January 17, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag:
> https://github.com/apache/kafka/releases/tag/3.1.0-rc1
>
> * Documentation:
> https://kafka.apache.org/31/documentation.html
>
> * Protocol:
> https://kafka.apache.org/31/protocol.html
>
> * Successful Jenkins builds for the 3.1 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/
>
> /**
>
> Thanks,
> David
>


Re: [VOTE] KIP-811: Add config min.repartition.purge.interval.ms to Kafka Streams

2022-01-18 Thread Nick Telford
Hi everyone,

With 3 binding +1 votes and no -1 votes, this vote passes. KIP-811 has been
Accepted.

Regards,

Nick

On Sun, 16 Jan 2022 at 15:50, Sagar  wrote:

> Hey Nick,
>
> Thanks for the KIP. I am maybe late to the game here, but maybe it might be
> beneficial to mention a couple of example scenarios on how the values of
> commit.interval.ms and this new config could affect the overall behavior .
> Like what happens if the 2 values are very similar v/s if one of them is
> very high as compared to the other. Let me know what you think.
>
> Other than that, Im +1 (non-binding).
>
> Thanks!
> Sagar.
>
> On Sun, Jan 16, 2022 at 6:17 PM Luke Chen  wrote:
>
> > Hi Nick,
> >
> > Thanks for the KIP!
> > +1 (non-binding)
> >
> > Luke
> >
> > On Sat, Jan 15, 2022 at 4:55 AM Matthias J. Sax 
> wrote:
> >
> > > +1 (binding)
> > >
> > > On 1/14/22 06:32, John Roesler wrote:
> > > > Thanks for the KIP, Nick!
> > > >
> > > > +1 (binding)
> > > >
> > > > -John
> > > >
> > > > On Fri, Jan 14, 2022, at 07:40, Bruno Cadonna wrote:
> > > >> Hi Nick,
> > > >>
> > > >> Since the title of the KIP slightly changed after the vote was
> opened
> > > >> also the link to the KIP changed as a result. This is should be a
> > > >> working link:
> > > >>
> > > >> https://cwiki.apache.org/confluence/x/JY-kCw
> > > >>
> > > >> Anyways, Thanks for the KIP!
> > > >>
> > > >> I am +1 (binding)
> > > >>
> > > >> Best,
> > > >> Bruno
> > > >>
> > > >>
> > > >>
> > > >> On 12.01.22 16:34, Nick Telford wrote:
> > > >>> Hi everyone,
> > > >>>
> > > >>> I'd like to call a vote to adopt KIP-811: Add config
> > > >>> min.repartition.purge.interval.ms to Kafka Streams
> > > >>> <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-811%3A+Add+config+min.repartition.purge.interval.ms+to+Kafka+Streams
> > > >
> > > >>> .
> > > >>>
> > > >>> Regards
> > > >>>
> > > >>> Nick Telford
> > > >>>
> > >
> >
>


Re: Memory leak on kafka-clients 3.0.0

2022-01-18 Thread Luke Chen
Hi Willian

Thanks for reporting this issue.
We can discuss this issue in KAFKA-13597.
As I mentioned in KAFKA-13597, I suspected that this could be:
1. bad config without correct validator
2. idempotence related components (ex: transactionManger) causes the memory
leak.

We need your help to narrow down the issue for troubleshooting.

Thank you.
Luke

On Tue, Jan 18, 2022 at 12:16 AM Willian Dallastella 
wrote:

> Hello,
>
> I wonder if you are aware of some memory leak on kafka-clients 3.0.0.
> I'm having this issue reported here:
> https://github.com/spring-projects/spring-kafka/issues/2056
>
> It is a Spring Boot 2.5.7 application that started failing when updated to
> Spring boot 2.6.1.
> This updated kafka-clients 2.7.1 to 3.0.0 and started causing the memory
> leak.
>
> Downgrading just the lib kafka-clients to 2.7.1 solves the issue.
> Any idea?
>
> Best regards,
> Willian Dallastella
>


Re: [VOTE] 3.1.0 RC1

2022-01-18 Thread Israel Ekpo
Performed the following validations using the tools available here:

https://github.com/izzyacademy/apache-kafka-release-party

   - Verified signatures, keys and hashes for release artifacts
   - Deployed Multi-Node Cluster in Legacy Mode (with Zookeeper)
   - Deployed Multi-Node Cluster in KRaft Mode (without Zookeeper)
   - Can confirm that KAFKA-13456 works as expected after switching to 3.1.0
   - Briefly Walked Through 3.1 Documentation, Javadocs and Protocol Pages

+1 (non-binding) for the release candidate

Thanks for running this release

Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/


On Wed, Jan 12, 2022 at 7:59 AM David Jacot  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.1.0.
>
> * Apache Kafka supports Java 17
> * The FetchRequest supports Topic IDs (KIP-516)
> * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
> * Add broker count metrics (KIP-748)
> * Differentiate consistently metric latency measured in millis and
> nanos (KIP-773)
> * The eager rebalance protocol is deprecated (KAFKA-13439)
> * Add TaskId field to StreamsException (KIP-783)
> * Custom partitioners in foreign-key joins (KIP-775)
> * Fetch/findSessions queries with open endpoints for
> SessionStore/WindowStore (KIP-766)
> * Range queries with open endpoints (KIP-763)
> * Add total blocked time metric to Streams (KIP-761)
> * Add additional configuration to control MirrorMaker2 internal topics
> naming convention (KIP-690)
>
> Release notes for the 3.1.0 release:
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, January 17, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag:
> https://github.com/apache/kafka/releases/tag/3.1.0-rc1
>
> * Documentation:
> https://kafka.apache.org/31/documentation.html
>
> * Protocol:
> https://kafka.apache.org/31/protocol.html
>
> * Successful Jenkins builds for the 3.1 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/
>
> /**
>
> Thanks,
> David
>


Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

2022-01-18 Thread Colin McCabe
Hi José,

Thanks for the KIP.

The KIP talks a bit about "recovery," which is a new term (as far as I know). 
If I understand correctly, this is a state that the partition enters into after 
an unclean leader election. I would suggest using a different term for this, 
since we already use the term "log recovery" to express the cleanup we do after 
an unclean broker restart. How about something like "leader cleansing"? 
Whatever term we pick, we should probably use it in all the metadata rather 
than is_unclean, etc.

I think we also need to spell out a few other things about "cleansing" state. 
For example, if the controller needs to change the leader of the partition 
again, before cleansing completes, does that mean the partition is still 
unclean? I would guess yes. We should spell this out in the KIP.

With regard to AlterIsr, it seems like we might end up in a scenario where the 
broker is on an old version that doesn't support the cleansing state, and the 
controller is on a new version that does. In that case, it seems like we could 
get stuck in the cleansing state for an arbitrary amount of time. We will not 
exit it until the broker sends an AlterIsr, but the broker may not send one of 
those ever (for example, if we go down to one uncleanly elected replica, and 
stay there).

The most obvious way to resolve this would be to use the IBP / metadata.version 
to gate using this feature. So it sounds like we should add a few lines to the 
KIP indicating that we're doing that.

I think the compatibility logic needs needs more thought. One example is, if we 
have some brokers in the cluster that support KIP-704 and some that don't, do 
we auto-clear the cleansing state when transitioning leadership uncleanly to an 
old broker? I think you probably have to. So we should discuss this in the KIP.

best,
Colin


On Mon, Jan 17, 2022, at 19:27, José Armando García Sancio wrote:
> Jose wrote:
>> I'll update the KIP with this information. The leader will return
>> "NOT_LEADER_OR_FOLLOWER" for any partition that is still recovering
>> for Fetch, Produce, OffsetsForLeaderEpoch and DeleteRecords requests.
>> This error type is retriable by the clients.
>
> I forgot to include ListOffset. Updated the KIP to include this information.
>
> Thanks
> -Jose


[jira] [Created] (KAFKA-13599) Upgrade RocksDB to 6.27.3

2022-01-18 Thread Jonathan Albrecht (Jira)
Jonathan Albrecht created KAFKA-13599:
-

 Summary: Upgrade RocksDB to 6.27.3
 Key: KAFKA-13599
 URL: https://issues.apache.org/jira/browse/KAFKA-13599
 Project: Kafka
  Issue Type: Task
  Components: streams
Reporter: Jonathan Albrecht
 Attachments: compat_report.html

RocksDB v6.27.3 has been released and it is the first release to support s390x. 
RocksDB is currently the only dependency in gradle/dependencies.gradle without 
s390x support.

RocksDB v6.27.3 has added some new options that require an update to 
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
 but no other changes are needed to upgrade.

A compatibility report is attached for the current version 6.22.1.1 -> 6.27.3



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [DISCUSS] KIP-719: Add Log4J2 Appender

2022-01-18 Thread Colin McCabe
On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> Hi Dongjin,
>
> We're also looking into this matter as our stack was also affected by all
> the log4j hell and users increasingly pushing us to upgrade to log4j2 or
> logback because of the existing vulnerabilities in log4j1.
> Regarding the points raised by Haruki:
>

I've heard the same thing from other people -- that there is now more interest 
in moving from log4j1.x to log4j2.x, after all the recent vulnerabilities in 
the latter. I found this a bit strange. Kafka avoided all the log4shell 
vulnerabilities exactly because we DIDN'T move to log4j 2.x. (Yes, I am aware 
that there is a longstanding vulnerability in that one log sink in log4j 1.x, 
but you can just not use that one.)

I haven't thought about this very hard. Maybe it's still a good idea to move to 
log4j2. But it's odd that nobody is commenting about how in this case, not 
updating actually prevented a major security incident for Kafka.

best,
Colin

>
> a) In my opinion the best would be to make the dynamic logger support
> (Log4jController and LoggingResource) pluggable for log4j2 and logback (so
> an interface could be used to define the dynamic logging control methods
> and a config to specify the implementation). That way we're not bound to
> either logback or log4j and seems like a low-effort thing to do.
> Additionally this could be used in Connect too in LoggingResource.
>
> b) I think testing dependencies aren't that important from the user
> perspective, it's fine to either use log4j2 or logback, whichever is
> easier. Kafka is either used from the distribution (tgz) or pulled in
> through maven, but test dependencies shouldn't be exposed to the world.
>
> c) I would support deprecating the appender in favor of the log4j2 Kafka
> appender. VerifiableLog4jAppender is intended as a testing tool anyway, so
> I think it's less important to change this to logback.
>
> Future vulnerabilities will always be found in either logback or log4j2 or
> any other logging framework, so I think the safest approach is to allow
> users to choose their implementation, while in tests I think we're free to
> use whatever we want as that shouldn't be constrained by vulnerabilities.
>
> Viktor
>
> On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada  wrote:
>
>> Thanks for the clarification.
>>
>> About 2, I wan't aware of those concerns.
>> Let me check them first.
>>
>>
>> Thanks,
>>
>> 2021年12月23日(木) 13:37 Dongjin Lee :
>>
>> > Hi Haruki,
>> >
>> >
>> > Thanks for organizing the issue.
>> >
>> >
>> > If the community prefers logback, I will gladly change the dependency and
>> > update the PR. However, it has the following issues:
>> >
>> >
>> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719
>> are
>> > not released yet. So, using log4j2 (whose recent update pace is so high)
>> > will not affect the users.
>> >
>> >
>> > 2. To switch to logback, the following features should be reworked:
>> >
>> >
>> >   a. Dynamic logger level configuration (core, connect)
>> >
>> >   b. Logging tests (streams)
>> >
>> >   c. Kafka Appender (tools)
>> >
>> >
>> > a and b are the most challenging ones since there is little documentation
>> > on how to do this, so it requires analyzing the implementation itself.
>> > (what I actually did with log4j2) About c, logback does not provide a
>> Kafka
>> > Appender so we have to provide an equivalent.
>> >
>> >
>> > It is why I prefer to use log4j2. How do you think?
>> >
>> >
>> > Thanks,
>> >
>> > Dongjin
>> >
>> >
>> > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada 
>> wrote:
>> >
>> > > Hi, Dongjin,
>> > >
>> > > Sorry for interrupting the discussion.
>> > > And thank you for your hard work about KIP-653, KIP-719.
>> > >
>> > > I understand that KIP-653 is already accepted so log4j2 is the choice
>> of
>> > > the Kafka community though, I'm now feeling that logback is a better
>> > choice
>> > > here.
>> > >
>> > > Reasons:
>> > >
>> > > - even after "log4shell", several vulnerabilities found on log4j2 so
>> new
>> > > versions are released and users have to update in high-pace
>> > > * actually, a CVE was also reported for logback (CVE-2021-42550)
>> but
>> > it
>> > > requires edit-permission of the config file for an attacker so it's
>> much
>> > > less threatening
>> > > - log4j1.x and logback are made by same developer (ceki), so
>> > substantially
>> > > the successor of log4j1 is logback rather than log4j2
>> > > - in Hadoop project, seems similar suggestion was made from a PMC
>> > > * https://issues.apache.org/jira/browse/HADOOP-12956
>> > >
>> > >
>> > > What do you think about adopting logback instead?
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > 2021年12月21日(火) 18:02 Dongjin Lee :
>> > >
>> > > > Hi Mickael,
>> > > >
>> > > > > In the meantime, you may want to bump the VOTE thread too.
>> > > >
>> > > > Sure, I just reset the voting thread with a brief context.
>> > > >
>> > > > Thanks,
>> > > > Dongjin
>> > > >
>> > > > On Tue, D

Please add jonathan.albrecht to the contributer's group in jira

2022-01-18 Thread Jonathan Albrecht

Hello Team,

I'd like to contribute to kafka. Could I be added to the contributer's
group in Jira? My user name in Apache Jira is jonathan.albrecht.

Happy to provide any other info if needed.

Thanks,

Jonathan Albrecht
jonathan.albre...@ibm.com


Re: Please add jonathan.albrecht to the contributer's group in jira

2022-01-18 Thread Bruno Cadonna

Hi Jonathan,

I added you to the contributors group. You are now able to assign 
tickets to yourself.


Thank you for your interest in Apache Kafka!

Best,
Bruno

On 18.01.22 18:02, Jonathan Albrecht wrote:


Hello Team,

I'd like to contribute to kafka. Could I be added to the contributer's
group in Jira? My user name in Apache Jira is jonathan.albrecht.

Happy to provide any other info if needed.

Thanks,

Jonathan Albrecht
jonathan.albre...@ibm.com



Requesting permission to the apache kafka project

2022-01-18 Thread Philip Nee
Hello,

Would like to gain permission to the AK for contribution.

wikiID: pnee
jiraID: pnee

Thanks,
P


[jira] [Resolved] (KAFKA-13597) Memory leak with kafka-clients 3.0.0

2022-01-18 Thread Willian Dallastella (Jira)


 [ 
https://issues.apache.org/jira/browse/KAFKA-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Willian Dallastella resolved KAFKA-13597.
-
Resolution: Not A Problem

> Memory leak with kafka-clients 3.0.0
> 
>
> Key: KAFKA-13597
> URL: https://issues.apache.org/jira/browse/KAFKA-13597
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 3.0.0
>Reporter: Willian Dallastella
>Priority: Major
> Attachments: image-2022-01-17-12-59-43-411.png, 
> image-2022-01-17-13-01-43-418.png
>
>
> I'm having this issue reported here: 
> [https://github.com/spring-projects/spring-kafka/issues/2056]
>  
> It is a Spring Boot 2.5.7 application that started failing (out of memory) 
> when updated to Spring Boot 2.6.1.
> This updated *kafka-clients {color:#00875a}2.7.1{color}* to 
> *{color:#ff}3.0.0{color}* and started causing the memory leak.
>  
> The service is using a standard spring *katkaTemplate* to send messages.
> {code:java}
> kafkaTemplate.send(
> topic,
> id,
> data
> ).addCallback(kafkaLoggerCallback) {code}
> This producer is quite heavy (~2.5k messages/s).
> My kafka config:
> {code:java}
> spring:
>     kafka:
>       bootstrap-servers:
>         - my_kafka_host:9092
>       properties:
>         client.id: my_client_id
>         ssl.endpoint.identification.algorithm: https
>         request.timeout.ms: 2
>         retry.backoff.ms: 500
>         sasl.jaas.config: 
> org.apache.kafka.common.security.plain.PlainLoginModule required 
> username="my_user" password="my_password";
>         security.protocol: SASL_SSL
>         sasl.mechanism: PLAIN
>         interceptor.classes: 
> io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
>         confluent.monitoring.interceptor.sasl.mechanism: PLAIN
>         confluent.monitoring.interceptor.security.protocol: SASL_SSL
>         confluent.monitoring.interceptor.sasl.jaas.config: 
> org.apache.kafka.common.security.plain.PlainLoginModule required 
> username="my_user" password="my_password";
>         schema.registry.basic.auth.user.info: sr_ccloud_key:sr_ccloud_key
>         schema.registry.url: https://my_schema_registry
>         ack: 1
>       producer:
>         key-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
>         value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer 
> {code}
>  
> *Downgrading* just the lib *kafka-clients* to *{color:#00875a}2.7.1{color}* 
> or *{color:#00875a}2.8.1{color}* solves the issue.
> h2. Memory leak sample
>  * The green line around 16:40 is the service using kafka-clients 2.7.1
>  * The pink line around 16:55 is without downgrading, that means 
> kafka-clients 3.0.0
> !image-2022-01-17-12-59-43-411.png|width=1017,height=303!
>  
> h2. Profiling sample
>  
> !image-2022-01-17-13-01-43-418.png!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #617

2022-01-18 Thread Apache Jenkins Server
See 




[jira] [Created] (KAFKA-13600) Rebalances while streams is in degraded state can stores to be assigned and restore from scratch

2022-01-18 Thread Tim Patterson (Jira)
Tim Patterson created KAFKA-13600:
-

 Summary: Rebalances while streams is in degraded state can stores 
to be assigned and restore from scratch
 Key: KAFKA-13600
 URL: https://issues.apache.org/jira/browse/KAFKA-13600
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.0.0, 2.8.1
Reporter: Tim Patterson


Consider this scenario:
 # A node is lost from the cluster.
 # A rebalance is kicked off with a new "target assignment"'s(ie the rebalance 
is attempting to move a lot of tasks - see 
https://issues.apache.org/jira/browse/KAFKA-10121).
 # The kafka cluster is now a bit more sluggish from the increased load.
 # A Rolling Deploy happens triggering rebalances, during the rebalance 
processing continues but offsets can't be committed(Or nodes are restarted but 
fail to commit offsets)
 # The most caught up nodes now aren't within `acceptableRecoveryLag` and so 
the task is started in it's "target assignment" location, restoring all state 
from scratch and delaying further processing instead of using the "almost 
caught up" node.

We've hit this a few times and having lots of state (~25TB worth) and being 
heavy users of IQ this is not ideal for us.

While we can increase `acceptableRecoveryLag` to larger values to try get 
around this that causes other issues (ie a warmup becoming active when its 
still quite far behind)



The solution seems to be to balance "balanced assignment" with "most caught up 
nodes".

We've got a fork where we do just this and it's made a huge difference to the 
reliability of our cluster.

Our change is to simply use the most caught up node if the "target node" is 
more than `acceptableRecoveryLag` behind.
This gives up some of the load balancing type behaviour of the existing code 
but in practise doesn't seem to matter too much.

I guess maybe an algorithm that identified candidate nodes as those being 
within `acceptableRecoveryLag` of the most caught up node might allow the best 
of both worlds.

 

Our fork is

[https://github.com/apache/kafka/compare/trunk...tim-patterson:fix_balance_uncaughtup?expand=1]
(We also moved the capacity constraint code to happen after all the stateful 
assignment to prioritise standby tasks over warmup tasks)

Ideally we don't want to maintain a fork of kafka streams going forward so are 
hoping to get a bit of discussion / agreement on the best way to handle this.
More than happy to contribute code/test different algo's in production system 
or anything else to help with this issue



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: Need access to Apache Kafka JIRA - username - manojp111

2022-01-18 Thread Manoj Pardeshi
Gentle Reminder.
Thanks & Regards,
Manoj

On Sun, Jan 16, 2022 at 7:58 PM Manoj Pardeshi 
wrote:

> Hi Apache Kafka moderators,
> Please give access to JIRA so that I can contribute to the project.
> Thanks,
> Manoj
>