[jira] [Created] (KAFKA-17719) Connect may fail to start tasks when reading from a compacted config topic
Chaitanya Mukka created KAFKA-17719:
---
Summary: Connect may fail to start tasks when reading from a
compacted config topic
Key: KAFKA-17719
URL: https://issues.apache.org/jira/browse/KAFKA-17719
Project: Kafka
Issue Type: Bug
Components: connect
Reporter: Chaitanya Mukka
The fix for KAFKA-16838 (via [https://github.com/apache/kafka/pull/16122)]
alters the logic for materializing a view of the config topic to ignore task
configs when there is no configuration for that connector present earlier in
the config topic. However, the logic fails to consider topics that might get
compacted over time.
In particular, when we have a connector {{C1}} running fine, the records in the
config topic for the connector will look something like {{{}C1, T1, T2,
Task-commit-record{}}}.
If the connector gets a config update that doesn't produce any new task configs
(note that this is a valid case when there are no task changes) we only produce
a Connector config record. The config topic now looks like {{{}C1, T1, T2,
Task-commit-record, C1{}}}. However, if the topic gets compacted we will end up
with {{{}T1, T2, Task-commit-record, C1{}}}. This can be a common scenario in
large and old connect clusters.
Based on the changes for KAFKA-16838, when the connect worker reads this config
state it ignores the task configs for this while the connector is still active
and we might have active assignments for the same. The symptom of this issue is
an NPE which shows up when trying to start the tasks:
{noformat}
java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because
"inputMap" is null
at org.apache.kafka.common.utils.Utils.castToStringObjectMap
at org.apache.kafka.common.config.AbstractConfig.
at org.apache.kafka.common.config.AbstractConfig.
at org.apache.kafka.connect.runtime.TaskConfig.
at org.apache.kafka.connect.runtime.Worker.startTask)
at org.apache.kafka.connect.runtime.Worker.startSourceTask
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$41
at java.base/java.util.concurrent.FutureTask.run
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run
at java.base/java.lang.Thread.run(Thread.java:1583){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [ANNOUNCE] New committer: Yash Mayya
Congrats, Yash!! Well deserved. Chaitanya Mukka On 21 Sep 2023 at 8:58 PM +0530, Bruno Cadonna , wrote: > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > Yash Mayya. > > Yash's major contributions are around Connect. > > Yash authored the following KIPs: > > KIP-793: Allow sink connectors to be used with topic-mutating SMTs > KIP-882: Kafka Connect REST API configuration validation timeout > improvements > KIP-970: Deprecate and remove Connect's redundant task configurations > endpoint > KIP-980: Allow creating connectors in a stopped state > > Overall, Yash is known for insightful and friendly input to discussions > and his high quality contributions. > > Congratulations, Yash! > > Thanks, > > Bruno (on behalf of the Apache Kafka PMC)
Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton
Congrats Chris! On Fri, 10 Mar, 2023, 12:26 am Manikumar, wrote: > Congrats Chris!. Well deserved. > > On Fri, Mar 10, 2023 at 12:23 AM Josep Prat > wrote: > > > > Congrats Chris! > > > > ——— > > Josep Prat > > > > Aiven Deutschland GmbH > > > > Alexanderufer 3-7, 10117 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > m: +491715557497 > > > > w: aiven.io > > > > e: josep.p...@aiven.io > > > > On Thu, Mar 9, 2023, 19:22 Mickael Maison > wrote: > > > > > Congratulations Chris! > > > > > > On Thu, Mar 9, 2023 at 7:17 PM Bill Bejeck wrote: > > > > > > > > Congratulations Chris! > > > > > > > > On Thu, Mar 9, 2023 at 1:12 PM Jun Rao > wrote: > > > > > > > > > Hi, Everyone, > > > > > > > > > > Chris Egerton has been a Kafka committer since July 2022. He has > been > > > very > > > > > instrumental to the community since becoming a committer. It's my > > > pleasure > > > > > to announce that Chris is now a member of Kafka PMC. > > > > > > > > > > Congratulations Chris! > > > > > > > > > > Jun > > > > > on behalf of Apache Kafka PMC > > > > > > > > >
Re: [VOTE] KIP-882: Kafka Connect REST API timeout improvements
Thanks for the KIP Yash! And also fixing the `double connector config validations` on POST. +1 (non-binding) on the KIP Regards, Chaitanya Mukka On 1 Mar 2023, 6:33 PM +0530, Yash Mayya , wrote: > Hi all, > > I'd like to call for a vote on the (hopefully) straightforward KIP-882 > which adds support for configuring request timeouts on Kafka Connect REST > APIs via query parameters along with a couple of related small improvements. > > KIP - > https://cwiki.apache.org/confluence/display/KAFKA/KIP-882%3A+Kafka+Connect+REST+API+timeout+improvements > > Discussion thread - > https://lists.apache.org/thread/cygy115qmwpc3nj5omnj0crws2dw8nor > > Thanks, > Yash
Re: Request permissions to contribute to Kafka
Thanks Chris! On 21 Feb 2023, 8:18 PM +0530, Chris Egerton , wrote: > Hi Chaitanya, > > You should be good to go. Happy KIPping! > > Cheers, > > Chris > > On Tue, Feb 21, 2023 at 9:33 AM Chaitanya Mukka > wrote: > > > Hi folks! > > > > I am Chaitanya Mukka. I would like to start contributing to Kafka. Can I > > get permission to AK Jira and permissions to Confluence to create KIPs? > > > > Jira and Wiki ID: cmukka > > > > > > Thanks and Regards, > > Chaitanya Mukka > >
Request permissions to contribute to Kafka
Hi folks! I am Chaitanya Mukka. I would like to start contributing to Kafka. Can I get permission to AK Jira and permissions to Confluence to create KIPs? Jira and Wiki ID: cmukka Thanks and Regards, Chaitanya Mukka
