yashmayya commented on code in PR #12490: URL: https://github.com/apache/kafka/pull/12490#discussion_r940928631
########## connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java: ########## @@ -853,6 +853,9 @@ private void processConnectorConfigRecord(String connectorName, SchemaAndValue v connectorConfigs.remove(connectorName); connectorTaskCounts.remove(connectorName); taskConfigs.keySet().removeIf(taskId -> taskId.connector().equals(connectorName)); + deferredTaskUpdates.remove(connectorName); + connectorTaskCountRecords.remove(connectorName); Review Comment: Tbh, it doesn't really seem like it's worth the mess of null handling everywhere. I'm gonna back out this change and make this a single line PR 😆 Btw, unrelated to this PR - but I noticed that the task config generations aren't persisted to the config topic, they're simply maintained in the in-memory map. So, based on worker restarts, different workers (herders to be precise) could have different views on the task config generations for the same connector because the config topic is compacted and older task commit records could be lost (the task config generation is incremented by 1 each time a task commit record is encountered). From what I can tell, task config generations are used in two places primarily - i) We compare the task config generation from prior to a EOS task startup with the task config generation after the task has initialized its transactional producer (and fail it if there's a mismatch indicating a new set of tasks have been brought up) ii) At the end of a zombie fencing, if the task config generation is greater than the task config generation at the beginning of the zombie fencing, a `409` is returned by the fencing endpoint because a new set of tasks were brought up in the meanwhile I'm struggling to think of any cases where this would cause an issue (i.e. different herders having different values for task config generations of the same connector) but I was hoping that you could verify my understanding here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org