Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Justine, I added a section under Proposed Changes -> Timestamp Validation Logic to capture how the INVALID_TIMESTAMP is handled in this KIP. Please let me know if there are any additional areas you would like me to address. Thanks, Mehari

Re: [DISCUSS] Adding non-committers as Github collaborators

Hi Luke, Sorry for the late reply. Can you also add me to the whitelist? I believe I'm supposed to be there as well. Matthias and John can vouch for me :) My github ID is "lihaosky". Thanks, Hao On Fri, Jun 2, 2023 at 4:43 PM Greg Harris wrote: > Luke, > > I see that the PR has been merged, bu

Re: [DISCUSS] Adding non-committers as Github collaborators

Luke, I see that the PR has been merged, but I don't believe it re-sent the invitation. Thanks Greg On Wed, May 31, 2023 at 6:46 PM Luke Chen wrote: > > Hi Greg and Andrew, > > Sorry, I don't know how to re-sent the invitation. > It looks like it is auto sent after the .asf.yaml updated. > Sin

[jira] [Created] (KAFKA-15054) Add configs and logic to decide if rack aware assignment should be enabled

Hao Li created KAFKA-15054: -- Summary: Add configs and logic to decide if rack aware assignment should be enabled Key: KAFKA-15054 URL: https://issues.apache.org/jira/browse/KAFKA-15054 Project: Kafka

[jira] [Created] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0

Bo Gao created KAFKA-15053: -- Summary: Regression for security.protocol validation starting from 3.3.0 Key: KAFKA-15053 URL: https://issues.apache.org/jira/browse/KAFKA-15053 Project: Kafka Issue Ty

[jira] [Created] (KAFKA-15052) Fix flaky test QuorumControllerTest.testBalancePartitionLeaders()

Dimitar Dimitrov created KAFKA-15052: Summary: Fix flaky test QuorumControllerTest.testBalancePartitionLeaders() Key: KAFKA-15052 URL: https://issues.apache.org/jira/browse/KAFKA-15052 Project: Ka

Requesting permissions to contribute/create KIPs for apache Kafka

Wiki ID: arunav.sanyal91 JIRA ID: arunav.sanyal91 -- Arunav Sanyal Software Engineer Amazon

Re: [DISCUSS] Regarding Old PRs

I think this is a great idea. If we don’t want the auto-close functionality, we can set it to -1 I realize this isn’t a vote, but I’m +1 on this -David On Fri, Jun 2, 2023 at 15:34 Colin McCabe wrote: > That should read "30 days without activity" > > (I am assuming we have the ability to deter

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

Hi Daniel, Are there any cases of Object::toString being used by internal classes for logging context that can't be covered by MDC contexts? For example, anything logged by WorkerSinkTask (or any WorkerTask subclass) already has the MDC set for the task [1]. Since the Object::toString-prefixed sty

Re: [DISCUSS] KIP-938: Add more metrics for measuring KRaft performance

Thanks for the clarification and explanation, Colin. Looks good to me. Ron On Fri, Jun 2, 2023 at 2:52 PM Colin McCabe wrote: > > Hi Ron, > > Thanks for the review. > > On Fri, Jun 2, 2023, at 11:26, Ron Dagostino wrote: > > Thanks for the KIP, Colin. The KIP cals one metric > > "NewActiveCont

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

Hi Jorge, This is a good KIP which addresses a real gap we have today. I would suggest renaming the "configs" parameter to "proposedConfigs," in both the new and old RequestMetadata constructors, to make things clearer. This would be a binary and API-compatible change in Java. We should also cl

Re: [DISCUSS] Regarding Old PRs

That should read "30 days without activity" (I am assuming we have the ability to determine when a PR was last updated on GH) best, Colin On Fri, Jun 2, 2023, at 12:32, Colin McCabe wrote: > Hi all, > > Looking at GitHub, I have a bunch of Kafka PRs of my own that I've > allowed to become stal

Re: [DISCUSS] Regarding Old PRs

Hi all, Looking at GitHub, I have a bunch of Kafka PRs of my own that I've allowed to become stale, and I guess are pushing up these numbers! Overall I support the goal of putting a time limit on PRs, just so that we can focus our review bandwidth. It may be best to start with a simple approac

Re: [DISCUSS] KIP-934: Add DeleteTopicPolicy

Hi Jorge, On the plus side, the change is small and pretty easy to support. On the minus side, the stated use-case (preventing deletion of __consumer_offsets) seems easy to solve via ACLs. The CreateTopics case is different... it's not as easy to solve via ACLs because people wanted to enforcc

Re: [DISCUSS] KIP-928: Making Kafka resilient to log directories becoming full

Hi Christo, We're not adding new stuff to ZK at this point (it's deprecated), so it would be good to drop that from the design. With regard to the "saturated" state: I'm skeptical that compaction could really move the needle much in terms of freeing up space -- in most workloads I've seen, it

Re: [DISCUSS] KIP-938: Add more metrics for measuring KRaft performance

Hi Ron, Thanks for the review. On Fri, Jun 2, 2023, at 11:26, Ron Dagostino wrote: > Thanks for the KIP, Colin. The KIP cals one metric > "NewActiveControllersCount" but we don't append "Count" to the other > metric names (e.g. it is "TimedOutBrokerHeartbeats" instead of > "TimedOutBrokerHeartbe

Re: [DISCUSS] KIP-938: Add more metrics for measuring KRaft performance

Thanks for the KIP, Colin. The KIP cals one metric "NewActiveControllersCount" but we don't append "Count" to the other metric names (e.g. it is "TimedOutBrokerHeartbeats" instead of "TimedOutBrokerHeartbeatsCount"). Should we be consistent (either use the suffix everywhere or don't use it anywhe

[jira] [Created] (KAFKA-15051) docs: add missing connector plugin endpoint to documentation

Jorge Esteban Quilcate Otoya created KAFKA-15051: Summary: docs: add missing connector plugin endpoint to documentation Key: KAFKA-15051 URL: https://issues.apache.org/jira/browse/KAFKA-15051

Re: [DISCUSS] KIP-928: Making Kafka resilient to log directories becoming full

Hi Christo, Thank you for the KIP. Kafka is very sensitive to filesystem errors, and at the first IO error the whole log directory is permanently considered offline. It seems your proposal aims to increase the robustness of Kafka, and that's a positive improvement. I have some questions: 11. "In

[jira] [Resolved] (KAFKA-15012) JsonConverter fails when there are leading Zeros in a field

[ https://issues.apache.org/jira/browse/KAFKA-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15012. --- Fix Version/s: 3.6.0 Resolution: Fixed > JsonConverter fails when there are leading Z

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

Hello Bruno, I think this covers your questions. Let me know what you think 2. We can use a changelog topic. I think we can treat it like any other store and recover in the usual manner. Also implementation is on disk 3. The description is in the public interfaces description. I will copy it int

[jira] [Resolved] (KAFKA-14863) Plugins which do not have a valid no-args constructor are visible in the REST API

[ https://issues.apache.org/jira/browse/KAFKA-14863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14863. --- Fix Version/s: 3.6.0 Resolution: Fixed > Plugins which do not have a valid no-args co

[jira] [Resolved] (KAFKA-15019) Improve handling of broker heartbeat timeouts

[ https://issues.apache.org/jira/browse/KAFKA-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-15019. Fix Version/s: 3.5.0 Resolution: Fixed > Improve handling of broker heartbeat timeo

[jira] [Resolved] (KAFKA-14996) The KRaft controller should properly handle overly large user operations

[ https://issues.apache.org/jira/browse/KAFKA-14996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-14996. Fix Version/s: 3.5.0 Assignee: Colin McCabe (was: Edoardo Comar) Resolutio

[jira] [Resolved] (KAFKA-15010) KRaft Controller doesn't reconcile with Zookeeper metadata upon becoming new controller while in dual write mode.

[ https://issues.apache.org/jira/browse/KAFKA-15010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-15010. Resolution: Fixed > KRaft Controller doesn't reconcile with Zookeeper metadata upon becomi

Re: [VOTE] 3.5.0 RC0

Mickael, all of our migration fixes are in. Thanks! David On Thu, Jun 1, 2023 at 6:10 PM Colin McCabe wrote: > Hi Mickael, > > Can you start the new RC tomorrow? There's one last PR we'd like to get in. > > If we can't get it in by tomorrow then let's go ahead anyway. > > Thanks very much, > Co

Re: [DISCUSS] KIP-921 OpenJDK CRaC support

Hi Divij, I think that your understanding is correct. The rest of replies inline... On 30. 05. 23 15:31, Divij Vaidya wrote: Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Re: [DISCUSS] KIP-921 OpenJDK CRaC support

Hi Colin, yes, with the draft status this is kind of chicken-egg problem - we'd like to collect feedback and round the edges in implementation before it lands in JDK and for that a support in common frameworks/libraries is necessary. But I understand the reluctance to a non-finalized feature.

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

Hi Walker, Thanks for the updates! 2. It is still not clear to me how a failure is handled. I do not understand what you mean by "recover from an OffsetCheckpoint". My understanding is that the buffer needs to be replicated into its own Kafka topic. The input topic is not enough. The offset

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #142

See Changes: -- [...truncated 349020 lines...] [2023-06-02T11:32:18.630Z] [2023-06-02T11:32:18.630Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 168 > Kafka

Re: [VOTE] 3.4.1 RC3

Hi Tom, it failed for me a couple of times, I rebooted and things suddenly worked. So maybe there was a dangling process holding a port from previous test failures. On Fri, Jun 2, 2023 at 10:52 AM Tom Bentley wrote: > Hi Luke, > > Thanks for running the release. > > I've checked signatures, eyeb

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Andrew, Thank you for the clarification. One follow-up to read_committed mode: Taking the change in message ordering guarantees into account, does this mean that in queues, share-group consumers will be able to consume committed records AFTER the LSO? Thanks, Daniel Andrew Schofield ezt írta (

[jira] [Created] (KAFKA-15050) Prompts in the quickstarts

Tom Bentley created KAFKA-15050: --- Summary: Prompts in the quickstarts Key: KAFKA-15050 URL: https://issues.apache.org/jira/browse/KAFKA-15050 Project: Kafka Issue Type: Improvement Co

Re: [VOTE] 3.4.1 RC3

Hi Luke, Thanks for running the release. I've checked signatures, eyeballed the Javadocs, built from source and run the unit and integration tests. DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate fails for me repeatedly. I opened https://issues.apache.org/jira/browse/KAFKA-15049 for

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Daniel, Thanks for your questions. 1) Yes, read_committed fetch will still be possible. 2) You weren’t wrong that this is a broad question :) Broadly speaking, I can see two ways of managing the in-flight records: the share-partition leader does it, or the share-group coordinator does it. I w

[jira] [Created] (KAFKA-15049) Flaky test DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate

Tom Bentley created KAFKA-15049: --- Summary: Flaky test DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate Key: KAFKA-15049 URL: https://issues.apache.org/jira/browse/KAFKA-15049 Project: Kafka

Re: [DISCUSS] Regarding Old PRs

Hi all, I want to say that in my experience, I always felt better as a contributor when a person told me something than when a bot did. That being said, I'm not against bots, and I think they might be a great solution once we have a manageable number of open PRs. Another great question that addin