[jira] [Resolved] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore

[ https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16310. Resolution: Fixed > ListOffsets doesn't report the offset with maxTimestamp anymore >

[jira] [Resolved] (KAFKA-16487) Support to define server properties by ClusterTestDefaults

[ https://issues.apache.org/jira/browse/KAFKA-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16487. Fix Version/s: 3.8.0 Resolution: Fixed > Support to define server properties by

[jira] [Created] (KAFKA-16502) Fix flaky EOSUncleanShutdownIntegrationTest#shouldWorkWithUncleanShutdownWipeOutStateStore

Chia-Ping Tsai created KAFKA-16502: -- Summary: Fix flaky EOSUncleanShutdownIntegrationTest#shouldWorkWithUncleanShutdownWipeOutStateStore Key: KAFKA-16502 URL: https://issues.apache.org/jira/browse/KAFKA-16502

[jira] [Created] (KAFKA-16501) Fix flaky DelegationTokenEndToEndAuthorizationWithOwnerTest#testCreateUserWithDelegationToken

Chia-Ping Tsai created KAFKA-16501: -- Summary: Fix flaky DelegationTokenEndToEndAuthorizationWithOwnerTest#testCreateUserWithDelegationToken Key: KAFKA-16501 URL: https://issues.apache.org/jira/browse/KAFKA-16501

[jira] [Created] (KAFKA-16500) Fix flaky DynamicBrokerReconfigurationTest#testTrustStoreAlter

Chia-Ping Tsai created KAFKA-16500: -- Summary: Fix flaky DynamicBrokerReconfigurationTest#testTrustStoreAlter Key: KAFKA-16500 URL: https://issues.apache.org/jira/browse/KAFKA-16500 Project: Kafka

[jira] [Created] (KAFKA-16499) Fix flaky ClientMetricsManagerTest#testCacheEviction

Chia-Ping Tsai created KAFKA-16499: -- Summary: Fix flaky ClientMetricsManagerTest#testCacheEviction Key: KAFKA-16499 URL: https://issues.apache.org/jira/browse/KAFKA-16499 Project: Kafka

[jira] [Created] (KAFKA-16498) fix flaky OffsetsApiIntegrationTest34s#testGetSinkConnectorOffsets

Chia-Ping Tsai created KAFKA-16498: -- Summary: fix flaky OffsetsApiIntegrationTest34s#testGetSinkConnectorOffsets Key: KAFKA-16498 URL: https://issues.apache.org/jira/browse/KAFKA-16498 Project:

[jira] [Created] (KAFKA-16497) Fix flaky SaslAuthenticatorFailureNoDelayTest16s#testInvalidPasswordSaslScram

Chia-Ping Tsai created KAFKA-16497: -- Summary: Fix flaky SaslAuthenticatorFailureNoDelayTest16s#testInvalidPasswordSaslScram Key: KAFKA-16497 URL: https://issues.apache.org/jira/browse/KAFKA-16497

[jira] [Resolved] (KAFKA-15160) Message bytes duplication in Kafka headers when compression is enabled

[ https://issues.apache.org/jira/browse/KAFKA-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phuc Hong Tran resolved KAFKA-15160. Resolution: Won't Fix > Message bytes duplication in Kafka headers when compression is

[jira] [Resolved] (KAFKA-15558) Determine if Timer should be used elsewhere in PrototypeAsyncConsumer.updateFetchPositions()

[ https://issues.apache.org/jira/browse/KAFKA-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phuc Hong Tran resolved KAFKA-15558. Resolution: Fixed > Determine if Timer should be used elsewhere in >

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

I took a quick look at the code -- looks like the previous behavior was not to set a top level error if one particular feature had an issue. We can do that. I think it could make some sense to specify errors on features that were not valid and use the top level error to indicate that the request

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi, Andrew, Thanks for the reply. A few more comments. 41. 41.1 How does the partition leader obtain the group epoch to set WriteShareGroupStateRequest.StateEpoch? 41.2 What's the benefit of having the group coordinator initialize the state and the partition leader set the SPSO? It seems simpler

Re: [jira] [Created] (KAFKA-16496) Make the default of receive.buffer.bytes -1 instead of 5

unsubscribe > On Apr 9, 2024, at 5:56 PM, Hechao Li (Jira) wrote: > > Hechao Li created KAFKA-16496: > - > > Summary: Make the default of receive.buffer.bytes -1 instead of 5 > Key: KAFKA-16496 > URL:

[jira] [Created] (KAFKA-16496) Make the default of receive.buffer.bytes -1 instead of 5

Hechao Li created KAFKA-16496: - Summary: Make the default of receive.buffer.bytes -1 instead of 5 Key: KAFKA-16496 URL: https://issues.apache.org/jira/browse/KAFKA-16496 Project: Kafka Issue

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hi Ivan, I think you have to go one way or the other with the cluster ID, so I think removing that from this KIP might be the best. I think there’s another KIP waiting to be written for ensuring consistency of clusters, but I think that wouldn’t conflict at all with this one. Thanks, Andrew >

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hi Ivan, I think you have to go one way or the other with the cluster ID, so I think removing that from this KIP might be the best. I think there’s another KIP waiting to be written for ensuring consistency of clusters, but I think that wouldn’t conflict at all with this one. Thanks, Andrew >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2799

See

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

José, INVALID_UPDATE_VERSION was added as part of KIP-497. The KIP seems to be lacking some details on the error. https://cwiki.apache.org/confluence/display/KAFKA/KIP-497%3A+Add+inter-broker+API+to+alter+ISR https://github.com/apache/kafka/commit/57de67db22eb373f92ec5dd449d317ed2bc8b8d1 The

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi Justine, Thanks for the KIP. I see that the KIP doesn't make any updates to the UpdateFeatures RPC. I was trying to understand how errors will be communicated to the client. Are you planning to use the INVALID_UPDATE_VERSION error and overwrite the ErrorMessage field for all of the

Re: [DISCUSS] KIP-1024: Make the restore behavior of GlobalKTables with custom processors configureable

Hey all, (1) no I hadn't considered just naming the methods differently. I actually really like this idea and am for it. Except we need 3 different methods now. One for no processor, one for a processor that should restore and one that reprocesses. How about `addCustomGlobalStore` and

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi Bruno and Bill, To complete the Damien's purposes about the point 3. Processing errors are caught and handled by the ProcessingErrorHandler, at the precise moment when records are processed by processor nodes. The handling will be performed in the "process" method of the ProcessorNode, such

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hi Andrew and all, I looked deeper into the code [1] and it seems the Metadata class is OK with cluster ID changing. So I'm thinking that the rebootstrapping shouldn't introduce a new failure mode here. And I should remove the mention of this cluster ID checks from the KIP. Best, Ivan [1]

Re: [EXT] Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi Bruno, Bill, First of all, thanks a lot for all your useful comments. > 1. and 2. > I am wondering whether we should expose the processor node ID -- which > basically is the processor node name -- in the ProcessingContext > interface. I think the processor node ID fits well in the >

[jira] [Created] (KAFKA-16495) Fix flaky TransactionsWithTieredStoreTest#testCommitTransactionTimeout

Chia-Ping Tsai created KAFKA-16495: -- Summary: Fix flaky TransactionsWithTieredStoreTest#testCommitTransactionTimeout Key: KAFKA-16495 URL: https://issues.apache.org/jira/browse/KAFKA-16495 Project:

[jira] [Created] (KAFKA-16494) Fix flaky PlaintextConsumerFetchTest#testLowMaxFetchSizeForRequestAndPartition

Chia-Ping Tsai created KAFKA-16494: -- Summary: Fix flaky PlaintextConsumerFetchTest#testLowMaxFetchSizeForRequestAndPartition Key: KAFKA-16494 URL: https://issues.apache.org/jira/browse/KAFKA-16494

[jira] [Created] (KAFKA-16493) Avoid unneeded subscription regex check if metadata version unchanged

Lianet Magrans created KAFKA-16493: -- Summary: Avoid unneeded subscription regex check if metadata version unchanged Key: KAFKA-16493 URL: https://issues.apache.org/jira/browse/KAFKA-16493 Project:

[jira] [Resolved] (KAFKA-13907) Fix hanging ServerShutdownTest.testCleanShutdownWithKRaftControllerUnavailable

[ https://issues.apache.org/jira/browse/KAFKA-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-13907. Fix Version/s: 3.8.0 Resolution: Fixed > Fix hanging

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

Looks like this KIP has gone quiet for a bit, but here with an update (maybe this will revitalize the conversation too :) ) For KIP-1022 we want to introduce more features and will have a config ` unstable.version.enable` I know that this KIP proposed a config `unstable.metadata.version.enable`

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi Jun, Makes sense to me. It seems like KIP-1014 has been inactive recently. I can update my KIP and mention this change on that discussion thread. Justine On Tue, Apr 9, 2024 at 9:01 AM Jun Rao wrote: > Hi, Justine, > > A single config makes sense to me too. We just need to reach consensus

[jira] [Resolved] (KAFKA-16485) Fix broker metrics to follow kebab/hyphen case

[ https://issues.apache.org/jira/browse/KAFKA-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-16485. - Fix Version/s: 3.8.0 Resolution: Fixed Merged the PR to trunk. > Fix broker metrics to follow

Re: [EXT] Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi Damien, Sebastien and Loic, Thanks for the KIP, this is a much-needed addition. I like the approach of getting the plumbing in for handling processor errors, allowing users to implement more complex solutions as needed. Overall how where the KIP Is now LGTM, modulo outstanding comments. I

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi, Justine, A single config makes sense to me too. We just need to reach consensus with KIP-1014. Thanks, Jun On Mon, Apr 8, 2024 at 5:06 PM Justine Olshan wrote: > Hey Jun, > > That's a good question. I think maybe for simplicity, we can have a single > config? > If that makes sense, I

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

Hi Nick, Thanks for breaking out the KIP from KIP-892! Here a couple of comments/questions: 1. In Kafka Streams, we have a design guideline which says to not use the "get"-prefix for getters on the public API. Could you please change getCommittedOffsets() to committedOffsets()? 2. It is

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

+1 let's not introduce a new api and mark it immediately as deprecated :) On your second comment Luke, one thing we need to clarify is when do we consider remote storage to be DISABLED for a topic? Particularly, what is the state when the remote storage is being deleted in case of

Re: [DISCUSS] Minimum constraint for segment.ms

Hi, I support changing constraints for mentioned settings. I've noticed that first producing big messages, and then setting `segment.bytes` to low value causes unwanted consequences. I did not notice that it would delete all the records, but I did not set it to one, but instead the case is

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

I should also note that the probability of false positives does not fall below shape.P because as it approaches shape.P a new layer is created and filters are added to that. So no layer in the LayeredBloomFilter exceeds shape.P thus the entire filter does not exceed shape.P. Claude On Tue, Apr

[DISCUSS] KIP-936 Throttle number of active PIDs

The overall design for KIP-936 seems sound to me. I would make the following changes: Replace the "TimedBloomFilter" with a "LayeredBloomFilter" from commons-collections v4.5 Define the producer.id.quota.window.size.seconds to be the length of time that a Bloom filter of PIDs will exist. Define

[jira] [Created] (KAFKA-16492) Flaky test: testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted – org.apache.kafka.connect.integration.OffsetsApiIntegrationTest

Apoorv Mittal created KAFKA-16492: - Summary: Flaky test: testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted – org.apache.kafka.connect.integration.OffsetsApiIntegrationTest Key: KAFKA-16492 URL:

[jira] [Created] (KAFKA-16491) Flaky test: randomClusterPerturbationsShouldConverge[rackAwareStrategy=balance_subtopology] – org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConverg

Apoorv Mittal created KAFKA-16491: - Summary: Flaky test: randomClusterPerturbationsShouldConverge[rackAwareStrategy=balance_subtopology] – org.apache.kafka.streams.processor.internals.assignment.TaskAssignorConvergenceTest Key:

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Hi Christo, > I would then opt for moving information from DisableRemoteTopic within the StopReplicas API which will then disappear in KRaft world as it is already scheduled for deprecation. What do you think? Sounds good to me. Thanks. Luke On Tue, Apr 9, 2024 at 6:46 PM Christo Lolov wrote:

[jira] [Created] (KAFKA-16490) Upgrade gradle from 8.6 to 8.7

Chia-Ping Tsai created KAFKA-16490: -- Summary: Upgrade gradle from 8.6 to 8.7 Key: KAFKA-16490 URL: https://issues.apache.org/jira/browse/KAFKA-16490 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-16488) fix flaky MirrorConnectorsIntegrationExactlyOnceTest#testReplication

Chia-Ping Tsai created KAFKA-16488: -- Summary: fix flaky MirrorConnectorsIntegrationExactlyOnceTest#testReplication Key: KAFKA-16488 URL: https://issues.apache.org/jira/browse/KAFKA-16488 Project:

[jira] [Created] (KAFKA-16489) Fix flaky ZkMigrationIntegrationTest#testDualWrite

Chia-Ping Tsai created KAFKA-16489: -- Summary: Fix flaky ZkMigrationIntegrationTest#testDualWrite Key: KAFKA-16489 URL: https://issues.apache.org/jira/browse/KAFKA-16489 Project: Kafka Issue

Re: [EXT] Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi again, I have additional questions/comments. 6. What record is exactly passed to the handler? Is it the input record to the task? Is it the input record to the processor node? Is it the input record to the processor? 7. Could you please add the packages of the Java

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Heya Luke! I thought a bit more about it and I reached the same conclusion as you for 2 as a follow-up from 1. In other words, in KRaft world I don't think the controller needs to wait for acknowledgements for the brokers. All we care about is that the leader (who is responsible for

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Hi Christo, Doguscan, Can you please address the comments in this mail thread and update the KIP accordingly? Thanks, Satish. On Wed, 3 Apr 2024 at 12:30, Luke Chen wrote: > > Hi Christo, > > 1. I agree with Doguscan that in KRaft mode, the controller won't send RPCs > to the brokers (except in

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

+1 to Jun for adding the consumer fetching from a follower scenario also to the existing section that talked about the drawback when a node built with last-tiered-offset has become a leader. As Abhijeet mentioned, we plan to have a follow-up KIP that will address these by having a deprioritzation

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Abhijeet, Thanks for the KIP to improve the tiered storage feature! Questions: 1. We could also get the "pending-upload-offset" and epoch via remote log metadata, instead of adding a new API to fetch from the leader. Could you explain why you choose the later approach, instead of the former?

Re: [EXT] Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi Loïc, Damien, and Sébastien, Thanks for the KIP! I find it really great that you contribute back to Kafka Streams concepts you developed for kstreamplify so that everybody can take advantage from your improvements. I have a couple of questions/comments: 1. and 2. I am wondering whether

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi, To complete the Sébastien's answer about the point 3, here is an example of how users could simply type the record key or value, based on a custom process exception handler: Properties streamProps = new Properties();

[jira] [Created] (KAFKA-16487) Support to define server properties by ClusterTestDefaults

Chia-Ping Tsai created KAFKA-16487: -- Summary: Support to define server properties by ClusterTestDefaults Key: KAFKA-16487 URL: https://issues.apache.org/jira/browse/KAFKA-16487 Project: Kafka

[jira] [Resolved] (KAFKA-16456) Can't stop kafka debug logs

[ https://issues.apache.org/jira/browse/KAFKA-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-16456. -- Resolution: Not A Problem You can also dynamically change the broker loggers

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hi Ivan, Thanks for the KIP. I can see situations in which this would be helpful. I have one question. The KIP says the client checks the cluster ID when it re-bootstraps and that it will fail if the cluster ID doesn’t match the previously known one. How does it fail? Which exception does it

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Christo, Please find my comments inline. On Fri, Apr 5, 2024 at 12:36 PM Christo Lolov wrote: > Hello Abhijeet and Jun, > > I have been mulling this KIP over a bit more in recent days! > > re: Jun > > I wasn't aware we apply 2.1 and 2.2 for reserving new timestamps - in > retrospect it

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi Jun, Thank you for taking the time to review the KIP. Please find my comments inline. On Fri, Apr 5, 2024 at 12:09 AM Jun Rao wrote: > Hi, Abhijeet, > > Thanks for the KIP. Left a few comments. > > 1. "A drawback of using the last-tiered-offset is that this new follower > would possess only