[jira] [Created] (KAFKA-16666) Remove unused class `GroupMetadataMessageFormatter` and `OffsetsMessageFormatter`

Chia-Ping Tsai created KAFKA-1: -- Summary: Remove unused class `GroupMetadataMessageFormatter` and `OffsetsMessageFormatter` Key: KAFKA-1 URL: https://issues.apache.org/jira/browse/KAFKA-1

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Luke, Thanks for the review! DelayedFetch and DelayedRemoteFetch are orthogonal calls . Only one of them will be active in a given FETCH request. The fetch request might take more than 500

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Jun, Thanks for your reply. 161. ShareGroupListing and ShareGroupDescription are using the same pattern as ConsumerGroupListing and ConsumerGroupDescription. I have gone for consistency which I think is probably best here. It’s what I would expect if I had previously used the admin API for

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2869

See

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

That's good questions... I could think of a few approaches, but I admit it might all be a little bit tricky to code up... However if we don't solve this problem, I think this KIP does not really solve the core issue we are facing? In the end, if we rely on the `.checkpoint` file to compute a

[jira] [Resolved] (KAFKA-16474) AsyncKafkaConsumer might send out heartbeat request without waiting for its response

[ https://issues.apache.org/jira/browse/KAFKA-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-16474. Resolution: Fixed Ran the test several times, the client log also looks fine. >

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi, Andrew, Thanks for the reply. A few more comments. 161. ShareGroupListing.state() returns an optional, but ShareGroupDescription.state() does not. Should we make them consistent? Also, it seems a bit weird to return optional with an UNKNOWN state. 162. Should GroupListing include

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Jun, Thanks for your reply. 147. Yes, I see what you mean. The rebalance latency will indeed be very short by comparison. I have removed the rebalance latency metrics from the client and retained the rebalance count and rate. 150. Yes, I think so. I have tweaked the text so that the simple

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

Hi Matthias, 200: I like the idea in general. However, it is not clear to me how the behavior should be with multiple stream threads in the same Kafka Streams client. What stream thread opens which store? How can a stream thread pass an open store to another stream thread that got the

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi, Andrew, Thanks for the reply. 147. There seems to be some difference between consumer groups and share groups. In the consumer groups, if a client receives a heartbeat response to revoke some partitions, it may have to commit offsets before revoking partitions or it may have to call the

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

Please also update the KIP. To get a wiki account created, please request it via a commet on this ticket: https://issues.apache.org/jira/browse/INFRA-25451 After you have the account, please share your wiki id, and we can give you write permission on the wiki. -Matthias On 5/3/24 6:30

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

117f: Good point by Bruno. We should check for this, and could have an additional `INVALID_STANDBY_TASK` error code? -Matthias On 5/3/24 5:52 AM, Guozhang Wang wrote: Hi Sophie, Re: As for the return type of the TaskAssignmentUtils, I think that makes sense. LGTM. On Fri, May 3, 2024 at

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

101: Yes, but what I am saying is, that we don't need to flush the .position file to disk periodically, but only maintain it in main memory, and only write it to disk on close() to preserve it across restarts. This way, it would never be ahead, but might only lag? But with my better

Re: [DISCUSS] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

Hi David, I can't think about any valid use case where changing the default value would be useful, and your motivation is convincing. Thanks. On Fri, May 3, 2024 at 6:01 AM Andrew Schofield wrote: > > Hi David, > I think this KIP is a very good idea. It would be good to get rid of this > cruft.

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

+1 (binding) On 5/3/24 8:52 AM, Federico Valeri wrote: Hi Fred, this is a useful addition. +1 non binding Thanks On Fri, May 3, 2024 at 4:11 PM Andrew Schofield wrote: Hi Fred, Thanks for the KIP. It’s turned out nice and elegant I think. Definitely a worthwhile improvement. +1

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

What about (106) to unify both exiting callback methods of `ProductionExceptionHandler` into a single one, instead of adding two new ones? Damien's last reply about it was: I will think about unifying, I do agree it would be cleaner. There was not follow up on this question, and the KIP

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

Hi Fred, this is a useful addition. +1 non binding Thanks On Fri, May 3, 2024 at 4:11 PM Andrew Schofield wrote: > > Hi Fred, > Thanks for the KIP. It’s turned out nice and elegant I think. Definitely a > worthwhile improvement. > > +1 (non-binding) > > Thanks, > Andrew > > > On 30 Apr 2024,

[jira] [Created] (KAFKA-16665) Fail to get partition's position from within onPartitionsAssigned callback in new consumer

Lianet Magrans created KAFKA-16665: -- Summary: Fail to get partition's position from within onPartitionsAssigned callback in new consumer Key: KAFKA-16665 URL: https://issues.apache.org/jira/browse/KAFKA-16665

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi all, A summary of the KIP and the discussions: The KIP introduces a handler interface for Producer in order to handle two exceptions: RecordTooLargeException and UnknownTopicOrPartitionException. The handler handles the exceptions per-record. - Do we need this handler? [Motivation and

[jira] [Created] (KAFKA-16664) Re-add EventAccumulator.take(timeout)

Jeff Kim created KAFKA-16664: Summary: Re-add EventAccumulator.take(timeout) Key: KAFKA-16664 URL: https://issues.apache.org/jira/browse/KAFKA-16664 Project: Kafka Issue Type: Sub-task

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Hey folks, I shared this with Omnia offline: One concern I have is with the length of time we keep "seen" producer IDs. It seems like the default is 1 hour. If a producer produces every 2 hours or so, and we are hitting the limit, it seems like we will throttle it even though we've seen it before

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

Hi Fred, Thanks for the KIP. It’s turned out nice and elegant I think. Definitely a worthwhile improvement. +1 (non-binding) Thanks, Andrew > On 30 Apr 2024, at 14:02, Frédérik Rouleau > wrote: > > Hi all, > > As there is no more activity for a while on the discuss thread, I think we > can

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Kafka developers! I just wanted to remind you all of the upcoming relevant dates for Kafka 3.8.0: - KIP freeze is on May 15th (this is in a little less than 2 weeks) - Feature freeze is on May 29th (this is in a little more than 25 days). If there is a KIP you really want to have in the 3.8

[jira] [Created] (KAFKA-16663) CoordinatorRuntime write timer tasks should be cancelled once HWM advances

Jeff Kim created KAFKA-16663: Summary: CoordinatorRuntime write timer tasks should be cancelled once HWM advances Key: KAFKA-16663 URL: https://issues.apache.org/jira/browse/KAFKA-16663 Project: Kafka

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

Hi Matthias, Sorry this fell out of my radar for a bit. Revisiting the topic, I think you’re right and we accept the duplicated nesting as an appropriate solution to not affect the larger public API. I can update my PR with the change. Regards, Shashwat Pandey On Wed, May 1, 2024 at 11:00 PM

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

Hi Sophie, I have updated the KIP and the PR. Regards,

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Hi Sophie, Re: As for the return type of the TaskAssignmentUtils, I think that makes sense. LGTM. On Fri, May 3, 2024 at 2:26 AM Bruno Cadonna wrote: > > Hi Sophie, > > 117f: > I think, removing the STATEFUL and STATELESS types is not enough to > avoid the error Guozhang mentioned. The

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

Hi, Murali. First, could you add the KIP-1042 to the index ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) as well so that everyone can find it easily? I took a look at the KIP, then I have 2 questions: 1. Though the new MATCH resource pattern type may reduce

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

As I wrote in [1], the ACL evaluation algorithm needs to be specified with respect to the specificity of the pattern so that we know exactly which if *-accounts-* takes precedence over nl-accounts-* or visa versa. I think that we should spell out that precedence is evaluated as follows: 1.

[jira] [Created] (KAFKA-16662) UnwritableMetadataException: Metadata has been lost

Tobias Bohn created KAFKA-16662: --- Summary: UnwritableMetadataException: Metadata has been lost Key: KAFKA-16662 URL: https://issues.apache.org/jira/browse/KAFKA-16662 Project: Kafka Issue

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

Took me awhile to find it but the link to the KIP is https://cwiki.apache.org/confluence/display/KAFKA/KIP-1042%3A+Support+for+wildcard+when+creating+new+acls On Fri, May 3, 2024 at 10:13 AM Murali Basani wrote: > Hello, > > I'd like to propose a suggestion to our resource patterns in Kafka

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi Bruno, Good catch, KIP has been updated accordingly. Regards, Loïc

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

Hi Damien, Sébastien, and Loïc, Thanks for the KIP! +1 (binding) Best, Bruno On 4/26/24 4:00 PM, Damien Gasparina wrote: Hi all, We would like to start a vote for KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing The KIP is available on

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi, the KIP looks great! public static final String PROCESS_EXCEPTION_HANDLER_CLASS_CONFIG = "process.exception.handler". needs to be changed to public static final String PROCESSING_EXCEPTION_HANDLER_CLASS_CONFIG = "processing.exception.handler". The name of the constant has been

[DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Just as a reminder, here the link for the vote: https://lists.apache.org/thread/o1kvv8zjzsp72ohcjpckdy544ko9tjjb regards Sébastien De : Sebastien Viale Envoyé : vendredi 3 mai 2024 10:35 À : dev@kafka.apache.org Objet : [DISCUSS] KIP-1033: Add Kafka Streams

[DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi, So, we all agree to revert to the regular Headers interface in ErrorHandlerContext. We will update the KIP accordingly. @Sophie => Yes, this is the last remaining question, and it has been open for voting since last week. Thanks Sébastien De : Andrew

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

Hi Matthias, 101: Let's assume a RocksDB store, but I think the following might be true also for other store implementations. With this KIP, if Kafka Streams commits the offsets, the committed offsets will be stored in an in-memory data structure (i.e. the memtable) and stay there until

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi Kamal, Thanks for the KIP! Sorry for the late review. Overall LGTM! Just 1 question: If one fetch request contains 2 partitions: [p1, p2] fetch.max.wait.ms: 500, remote.fetch.max.wait.ms: 1000 And now, p1 fetch offset is the log end offset and has no new data coming, and p2 fetch offset is

[DISCUSS] KIP-1042 support for wildcard when creating new acls

Hello, I'd like to propose a suggestion to our resource patterns in Kafka ACLs. Currently, when adding new ACLs in Kafka, we have two types of resource patterns for topics: - LITERAL - PREFIXED However, when it comes to listing or removing ACLs, we have a couple more options: - MATCH

[jira] [Created] (KAFKA-16661) add a lower `log.initial.task.delay.ms` value to integration test framework

Luke Chen created KAFKA-16661: - Summary: add a lower `log.initial.task.delay.ms` value to integration test framework Key: KAFKA-16661 URL: https://issues.apache.org/jira/browse/KAFKA-16661 Project: Kafka

[jira] [Created] (KAFKA-16660) reduce the check interval to speedup DelegationTokenRequestsTest

Chia-Ping Tsai created KAFKA-16660: -- Summary: reduce the check interval to speedup DelegationTokenRequestsTest Key: KAFKA-16660 URL: https://issues.apache.org/jira/browse/KAFKA-16660 Project: Kafka

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Hi Sophie, 117f: I think, removing the STATEFUL and STATELESS types is not enough to avoid the error Guozhang mentioned. The StreamsPartitionAssignor passes the information whether a task is stateless or stateful into the task assignor. However, the task assignor can return a standby task for

Re: Suggestion about support for wildcard when creating new acls

I think that if this is introduced (and perhaps even if it is not) we need a clear ACL evaluation process. I know we have both allow and deny, and that deny takes precedence over allow. But let's consider two scenarios 1. Unintended access. Let's assume we start with the 6 topics Murali used

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Also, I think using `stopReplicas` request is a good idea because it won't cause any problems while migrating to KRaft mode. The stopReplicas request is one of the request that KRaft controller will send to ZK brokers during migration. Thanks. Luke On Fri, May 3, 2024 at 11:48 AM Luke Chen

[jira] [Resolved] (KAFKA-16572) allow defining number of disks per broker in ClusterTest

[ https://issues.apache.org/jira/browse/KAFKA-16572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16572. Fix Version/s: 3.8.0 Resolution: Fixed > allow defining number of disks per broker