Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

ow `.position` information is added. > > >>>>>>> The KIP only says: "position offsets will be stored in RocksDB, > > >>>>>>> in the same column family as the changelog offsets". Do you > > >>>>>&g

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

nges as suggested. > > > > > >> > > > > > >> 7. Combined with Matthias's point about RocksDB, I'm convinced > > that this > > > > > >> is the wrong KIP for these. I'll introduce the additional Rocks > > metrics > > > >

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

kInfo approach. However, task metadata interfaces > start to proliferate a bit too much in our code base. We have > TaskMetadata, TaskInfo, and finally Task that provide similar methods. I > think we should try to consolidate those interfaces. Does not need to > happen in this KIP, but we should

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

, or a new suggestion of their own. As always, let us know if you have any questions or concerns or feedback of any kind. Thanks! On Mon, May 6, 2024 at 1:33 PM Sophie Blee-Goldman wrote: > Thanks guys. Updated the error codes in both the code and the explanation > under "Public Changes"

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

+1 (binding) Thanks! On Tue, May 14, 2024 at 6:58 PM Matthias J. Sax wrote: > +1 (binding) > > On 5/14/24 9:19 AM, Lucas Brutschy wrote: > > Hi Nick! > > > > Thanks for the KIP. > > > > +1 (binding) > > > > On Tue, May 14, 2024 at 5:16 PM Nick Telford > wrote: > >> > >> Hi everyone, > >> > >>

[jira] [Resolved] (KAFKA-15170) CooperativeStickyAssignor cannot adjust assignment correctly

[ https://issues.apache.org/jira/browse/KAFKA-15170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-15170. Resolution: Fixed > CooperativeStickyAssignor cannot adjust assignm

[jira] [Created] (KAFKA-16758) Extend Consumer#close with option to leave the group or not

A. Sophie Blee-Goldman created KAFKA-16758: -- Summary: Extend Consumer#close with option to leave the group or not Key: KAFKA-16758 URL: https://issues.apache.org/jira/browse/KAFKA-16758

Re: [KAFKA-16361] Rack aware sticky assignor minQuota violations

Hey, Took a look at the ticket and left a comment. Seems like narrowing down which versions this affects is a good first step, after that I'm happy to help you understand the code and put together a fix if you're willing. I'd recommend starting out by going through the "constrainedAssign" method

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

Thanks for the KIP guys! +1 (binding) On Mon, May 13, 2024 at 6:02 AM Bill Bejeck wrote: > Thanks for the KIP, this will be a great addition! > > +1(binding) > > -Bill > > On Fri, May 3, 2024 at 4:48 AM Bruno Cadonna wrote: > > > Hi Damien, Sébastien, and Loïc, > > > > Thanks for the KIP! > >

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

, 2024 at 10:10 PM Sophie Blee-Goldman wrote: > Sounds like we're more or less in agreement here. I think the KIP just > needs one small update still, which is the SerializationExceptionOrigin > enum. > > As discussed there are a few options for this and we're all happy to defer >

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

comfortable voting for this KIP. > > Thanks, > Bill > > On Thu, May 9, 2024 at 4:28 PM Sophie Blee-Goldman > wrote: > > > The type safety issue is definitely not solved by having two separate > > callbacks. I just think it gets a bit worse by mashing them into

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

> > -Matthias > > On 5/8/24 3:12 PM, Sophie Blee-Goldman wrote: > > Well I definitely don't feel super strongly about it, and more > importantly, > > I'm not a user. So I will happily defer to the preference of anyone who > > will actually be using this feature. But

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

r experience across the board. > > Would be great to hear from others about this point. It's not that I > strongly object to having two methods, and I would not block this KIP on > this question. > > > > -Matthias > > > On 5/7/24 3:40 PM, Sophie Blee-Goldman wrote:

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

First of all, I agree that it makes sense to maintain the two separate callbacks for the ProductionExceptionHandler, since one of them is specifically for serialization exceptions while the other is used for everything/anything else. I also think we can take advantage of this fact to simplify

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

+1 (binding) thanks for the KIP! On Fri, May 3, 2024 at 9:13 AM Matthias J. Sax wrote: > +1 (binding) > > On 5/3/24 8:52 AM, Federico Valeri wrote: > > Hi Fred, this is a useful addition. > > > > +1 non binding > > > > Thanks > > > > On Fri, May 3, 2024 at 4:11 PM Andrew Schofield > > wrote:

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2:43 AM, Matthias J. Sax wrote: > >>> Thanks Sophie. My bad. You are of course right about `TaskAssignment` > >>> and the StreamsPartitionAssignor's responsibitliy to map tasks of a > >>> instance to consumers. When I wrote my reply, I forgot about this > detail. > &g

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

nt will make it seem like each of these utility methods return a "full" and final assignment that can just be returned as-is from the TaskAssignor's #assign method. Whereas they are each just a single step in the full assignment process, and not the final product. Does that make sense? On

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

gt; > > Does this make sense? If we assume that the task-assignment is > > deterministic, we would end up with an infinite retry loop? Also, > > assuming that an client leave the group, we cannot assign some task any > > longer... I would rather throw a StreamsExcep

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

I tend to agree that we should just return a pure Headers instead of introducing a new class/interface to protect overwriting them. I think a pretty good case has been made already so I won't add onto it, just wanted to voice my support. Is that the only remaining question on this KIP? Might be

Re: KAFKA-16221

Hey Matthias -- I'm quite sure you should *not* do what we're doing in https://github.com/apache/kafka/pull/15315. That's definitely a bad hack, and IIUC the only reason we accepted it was because the choice was between implementing a hacky temporary fix and blocking the entire release while we

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

nity>[image: YouTube] > <https://youtube.com/confluent> > > [image: Try Confluent Cloud for Free] > < > https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic > > > > > On Tue, Apr 30, 2024 at 10:54 PM Sophie Blee-Goldman < &

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

care to do something different based on whether the error happened during key or value deserialization. Thoughts? On Tue, Apr 30, 2024 at 1:45 PM Sophie Blee-Goldman wrote: > Hey Fred, I think this is looking good but I just want to follow up on > what Kirk asked earlier about having bot

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

Hey Fred, I think this is looking good but I just want to follow up on what Kirk asked earlier about having both the ByteBuffer and byte[] forms. Can't users just use the ByteBuffer versions and convert them into a byte[] themselves? In some cases it maybe makes sense to offer some additional APIs

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

ic Changes" as well. Please let me know if there are any concerns, or if you have suggestions for how else we can handle an invalid assignment On Tue, Apr 30, 2024 at 11:39 AM Sophie Blee-Goldman wrote: > Thanks guys! I agree with what Lucas said about 117c, we can always loosen > a rest

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

should be allowed. > >> 117c) This is the tricky case. However, I'm leaning towards not > >> allowing this, unless we have a concrete use case. This will block us > >> from potentially using a single consumer for active and standby tasks > >> in th

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Yeah I think that sums it up well. Either you computed a *possible* assignment, or you returned something that makes it literally impossible for the StreamsPartitionAssignor to decipher/translate into an actual group assignment, in which case it should just fail That's more or less it for the

Re: [VOTE] KIP-924: customizable task assignment for Streams

+1 (binding) thanks for driving this KIP! On Tue, Apr 16, 2024 at 1:46 PM Rohan Desai wrote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-924%3A+customizable+task+assignment+for+Streams > > As this KIP has been open for a while, and gone through a couple rounds of >

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

o what we do > currently: we return the old assignment, and request an immediate follow > up rebalance? For this case, the user code does not need to know > anything about it, and does not need to do anything special, and it > would become a provided built-in feature what seems desirabl

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

or them to implement. Updated the KIP to make this more clear by breaking up the classes into "User APIs" and "Read-only APIs" (with a short description of each category). 119. Good point, added a public constructor for AssignedTask 120. Also good point, gave the #onAssignm

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

ProcessID processId(); > >> > >> Instant followupRebalanceDeadline(); > >> > >> Set assignment(); > >> > >> enum AssignedTaskType { > >> STATELESS, > >> STATEFUL, > >> STANDBY >

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

Thanks for all the context everyone. I think in light of all the information that's been shared, I would agree with David that neither the Record nor ConsumerRecord feels appropriate and that we should just create a new class/interface that holds the info we want to expose. We definitely don't

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

s in AssignedTask that > return a boolean you could have one method -- say type() -- that returns > the enum. > > (4) > Does the return type of assignment need to be a map from task ID to > AssignedTask? Wouldn't it be enough to be a collection of AssignedTasks > with AssignedTask containing

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

to consider. Sorry for any confusion I caused On Tue, Apr 16, 2024 at 11:15 AM Sophie Blee-Goldman wrote: > Ah, thanks for the additional context. I should have looked at the code > before I opened my mouth (so to speak) > > In that case, I fully agree that using Record instead of Co

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

Ah, thanks for the additional context. I should have looked at the code before I opened my mouth (so to speak) In that case, I fully agree that using Record instead of ConsumerRecord makes sense. It does indeed seem like by definition, if there is a DeserializationException then there is no

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

r Kirk, can you elaborate on the specific cases we're worried about here? We should really highlight those in the getter javadocs to explain why it's an Optional. What fields would/could be missing? On Tue, Apr 16, 2024 at 10:11 AM Sophie Blee-Goldman wrote: > I think some missing context here (

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

I think some missing context here (which can maybe be added in the Motivation section as background) is that the deserialization is actually done within Streams, not within the Consumer. Since the consumers in Kafka Streams might be subscribed to multiple topics with different data types, it has

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

that mean for the StateStore > interface? Obviously we can't require that the constructor take the TaskId. > Is it enough to add the parameter to the StoreSupplier? > > Would doing this be in-scope for this KIP, or are we over-complicating it? > > Nick > > On Fri, 12 Apr 2024 at 21

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

Congrats Greg! Happy to have you On Sun, Apr 14, 2024 at 9:26 AM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Congrats, Greg!! > > On Sun 14. Apr 2024 at 15.05, Josep Prat > wrote: > > > Congrats Greg!!! > > > > > > Best, > > > > Josep Prat > > Open Source Engineering

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

Somewhat minor point overall, but it actually drives me crazy that you can't get access to the taskId of a StateStore until #init is called. This has caused me a huge headache personally (since the same is true for processors and I was trying to do something that's probably too hacky to actually

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

I think the bigger question here is: why is checkstyle complaining about this import? Does anyone know? On Thu, Apr 11, 2024 at 11:12 AM Frédérik Rouleau wrote: > Hi everyone, > > I have made some changes to take in account comments. I have replaced the > ConsumerRecord by Record. As it was not

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

ul in this case. > > >> >> > > >> >> Thanks, > > >> >> Bill > > >> >> > > >> >> On Tue, Apr 9, 2024 at 6:54 AM Bruno Cadonna > > >> mailto:cado...@apache.org>> wrote: > > >> >> > > >&g

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Thanks for the KIP, this will make a lot of people very happy. Wanted to chime in on a few points that have been raised so far and add some of my own (numbering with an S to distinguish my points from the previous ones) S1. > 1.a I really meant ProducerRecord, that's the class used to forward

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi Damien, First off thanks for the KIP, this is definitely a much needed feature. On the whole it seems pretty straightforward and I am in favor of the proposal. Just a few questions and suggestions here and there: 1. One of the #handle method's parameters is "ProcessorNode node", but

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

thanks Sophie, I've made the updates, would appreciate one more look > before > > submission > > > > On Wed, Mar 20, 2024 at 8:36 AM Sophie Blee-Goldman < > sop...@responsive.dev> > > wrote: > > > >> A few minor notes on the KIP but otherwise I t

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

t; have monolithic apps like ours, and most instead de-couple unrelated topics > via different apps, which reduces the impact of incompatible upstream > schema changes. > > Thanks for your reviews and feedback, I've learned a lot, as always; this > time, mostly about how, when authoring a KIP,

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Cool, looks good to me! Seems like there is no further feedback, so maybe we can start to call for a vote? However, since as noted we are setting aside time to discuss this during the sync next Thursday, we can also wait until after that meeting to officially kick off the vote. On Fri, Apr 5,

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

Before I go into my own feedback, which I know not everyone has the time to read, I just want to announce that we plan to go over this KIP during the next Streams project sync on April 11. If you are interested in joining the discussion and need an invitation, just reach out to me in a separate

Re: [VOTE] KIP-1020 Move `window.size.ms` and `windowed.inner.serde.class` from `StreamsConfig` to TimeWindowedDe/Serializer class

+1 (binding) Thanks Lucia! On Tue, Apr 2, 2024 at 12:23 AM Matthias J. Sax wrote: > +1 (binding) > > > -Matthias > > On 4/1/24 7:44 PM, Lucia Cerchie wrote: > > Hello everyone, > > > > I'd like to call a vote on KIP-1020 > > < >

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

gt; https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804 > > On Sun, Mar 17, 2024 at 7:09 PM Sophie Blee-Goldman > > wrote: > > > Sounds good! > > > > @Lucia when you have a moment can you update the KIP with > > the new proposal, includ

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

il. > > I agree that the new config should take preference in case both are > specified. This should be pointed out in the KIP, as it's an important > contract the user needs to understand. > > > -Matthias > > On 3/14/24 6:18 PM, Sophie Blee-Goldman wrote: >

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

not place the "inner.window.class.serde" > >> in the TimeWindowedSerde class either. To me, it actually makes > >> the most sense to define it in both the TimeWindowedSerializer > >> and the TimeWindowedDeserializer. > > Not sure either. What you propose is f

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

stom stores could just register a custom metric > about it to begin with? > > > > -Matthias > > On 10/25/23 4:41 PM, Sophie Blee-Goldman wrote: > >> > >> If we used "iterator-duration-max", for > >> example, would it not be confusing that it

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

honest questions about some technical > details. I think it would be good to consider there trade-offs and > explain in the KIP why we want to do what. That's all. > > > > -Matthias > > On 3/12/24 11:24 PM, Sophie Blee-Goldman wrote: > >> > >> I see way

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

> >> to the KIP. I'll also add putting it in the `TimeWindowedSerde` class > with > >> some add'tl guidance on the docs addition. > >> > >> Also, I double-checked setting window.size.ms on the client and it > >> doesn't throw an error at all, in res

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

nfluence over how they're assigned), a user cannot use > > TaskId alone to determine which Processors would be affected. > > > > What do you think would be the best way to provide this information to > > exception handlers? I was originally thinking that users could > instant

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

Thanks for responding Matthias -- you got there first, but I was going to say exactly the same thing as in your most reply. In other words, I see the `windowed.inner.serde.class` as being in the same boat as the ` window.size.ms` config, so whatever we do with one we should do for the other. I do

[jira] [Resolved] (KAFKA-16277) CooperativeStickyAssignor does not spread topics evenly among consumer group

[ https://issues.apache.org/jira/browse/KAFKA-16277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-16277. Resolution: Fixed > CooperativeStickyAssignor does not spread top

Re: Kafka trunk test & build stability

se > > > >> > to set a global test timeout as 10 mins using the junit5 feature > > > >> > < > > > >> > > > > >> > > > > > > https://junit.org/junit5/docs/current/user-guide/#writing-tests-declarative-timeouts-default-time

Re: Kafka trunk test & build stability

On a related note, has anyone else had trouble getting even a single run with no build failures lately? I've had multiple pure-docs PRs blocked for days or even weeks because of miscellaneous infra, test, and timeout failures. I know we just had a discussion about whether it's acceptable to ever

Re: [DISCUSS] Should we continue to merge without a green build? No!

Thanks for that link Ismael -- I'd never seen it before. That's super useful In fact, it helped me realize another area where we could really improve things. I took a look at the worst offenders on that chart and was debugging the 2nd most flaky Streams test when I expanded the timeframe and saw

Re: [DISCUSS] Should we continue to merge without a green build? No!

ot driving away our community of contributors. This isn't meant to discourage anyone, actually the opposite: if we want to start gating PRs on passing builds, we need to start tackling flaky tests now! On Tue, Nov 21, 2023 at 1:19 PM Sophie Blee-Goldman wrote: > In the interest of moving t

Re: [DISCUSS] Should we continue to merge without a green build? No!

...any thoughts on how to implement this policy? And when to actually start enforcing it? On Tue, Nov 21, 2023 at 1:18 PM Sophie Blee-Goldman wrote: > So...where did we land on this? > > To take a few steps back and make sure we're all talking about the same > thing, I want to me

Re: [DISCUSS] Should we continue to merge without a green build? No!

So...where did we land on this? To take a few steps back and make sure we're all talking about the same thing, I want to mention that I myself was responsible for merging a PR that broke the build a few weeks ago. There was a warning that only appeared in some of the versions, but when checking

Re: [VOTE] KIP-892: Transactional StateStores

+1 (binding) Thanks a lot for this KIP! On Mon, Nov 13, 2023 at 8:39 AM Lucas Brutschy wrote: > Hi Nick, > > really happy with the final KIP. Thanks a lot for the hard work! > > +1 (binding) > > Lucas > > On Mon, Nov 13, 2023 at 4:20 PM Colt McNealy wrote: > > > > +1 (non-binding). > > > >

Re: [VOTE] KIP-998: Give ProducerConfig(props, doLog) constructor protected access

irable (and don't generate JavaDocs for protected > > methods...). Maybe it's less of an issue for clients, but given my > > experience with Kafka Streams, and it large API, I prefer to guide users > > by avoiding "leaky" abstractions. > > > > -Matthias >

Re: [VOTE] KIP-998: Give ProducerConfig(props, doLog) constructor protected access

On 11/3/23 6:06 AM, Ismael Juma wrote: > > > Hi Sophie, > > > > > > I was trying to understand the goal of the change and it's not totally > > > clear to me. If the goal is to allow third party applications to > > customize > > > the logging be

Re: Apache Kafka 3.7.0 Release

d 4 days later) > > > - Feature Freeze - *December 6 *(moved 2 days earlier) > > > - Code Freeze - *December 20* > > > > > > If anyone has any thoughts against this proposal - please let me know! > It > > > would be good to settle on this early. These will be the da

[VOTE] KIP-998: Give ProducerConfig(props, doLog) constructor protected access

Hey all, This is a trivial one-liner change that it was determined should go through a KIP during the PR review process (see this thread for context). Since the change itself was already reviewed and approved I'm skipping the

[jira] [Created] (KAFKA-15782) Establish concrete project conventions to define public APIs that require a KIP

A. Sophie Blee-Goldman created KAFKA-15782: -- Summary: Establish concrete project conventions to define public APIs that require a KIP Key: KAFKA-15782 URL: https://issues.apache.org/jira/browse/KAFKA

[jira] [Created] (KAFKA-15781) Change ProducerConfig(props, doLog) constructor to protected

A. Sophie Blee-Goldman created KAFKA-15781: -- Summary: Change ProducerConfig(props, doLog) constructor to protected Key: KAFKA-15781 URL: https://issues.apache.org/jira/browse/KAFKA-15781

[jira] [Resolved] (KAFKA-15116) Kafka Streams processing blocked during rebalance

[ https://issues.apache.org/jira/browse/KAFKA-15116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-15116. Resolution: Not A Problem > Kafka Streams processing blocked dur

[jira] [Resolved] (KAFKA-12550) Introduce RESTORING state to the KafkaStreams FSM

[ https://issues.apache.org/jira/browse/KAFKA-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-12550. Resolution: Won't Fix Closing this out since it's usefulness is preempted

[jira] [Resolved] (KAFKA-15463) StreamsException: Accessing from an unknown node

[ https://issues.apache.org/jira/browse/KAFKA-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-15463. Resolution: Not A Problem > StreamsException: Accessing from an unkn

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

> not pause. > > 4c. > Regardless, I like this idea, but I have very little knowledge about making > changes to the rebalance/network protocol. It looks like this could be > added via StreamsPartitionAssignor#subscriptionUserData? I might need some > help designing this aspect

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

nclude improved Block Cache metrics, as my own memory leak investigations > have trended in that direction. Do you think the following metrics should > be included in this KIP, or should I create a new KIP? > >- block-cache-index-usage (number of bytes occupied by index blocks) >-

Re: [VOTE] KIP-988 Streams StandbyUpdateListener

Happy to see this -- that's a +1 (binding) from me On Mon, Oct 23, 2023 at 6:33 AM Bill Bejeck wrote: > This is a great addition > > +1(binding) > > -Bill > > On Fri, Oct 20, 2023 at 2:29 PM Almog Gavra wrote: > > > +1 (non-binding) - great improvement, thanks Colt & Eduwer! > > > > On Tue,

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

e Otter Council > adjourns regarding this matter? > > Cheers, > Colt McNealy > > *Founder, LittleHorse.dev* > > > On Mon, Oct 23, 2023 at 10:44 PM Sophie Blee-Goldman < > sop...@responsive.dev> > wrote: > > > Just want to checkpoint the current state of this KIP

Re: Apache Kafka 3.7.0 Release

dative 4-week period. > > > Note that historically, we have set all the deadlines on a Wednesday and > when in doubt erred on the side of an earlier deadline ... We can, and > often have, allowed things to come in late between the Wednesday freeze > deadline and the following Fr

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

Hey Nick, A few high-level thoughts: 1. We definitely don't want to piggyback on the SUSPENDED task state, as this is currently more like an intermediate state that a task passes through as it's being closed/migrated elsewhere, it doesn't really mean that a task is "suspended" and there's no

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

I actually think we could implement Lucas' suggestion pretty easily and without too much additional effort. We have full control over the iterator that is returned by the various range queries, so it would be easy to register a gauge metric for how long it has been since the iterator was created.

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

;> One thing on our list of TODO's regarding reliability is to > >>>>>>>> determine > >>>>>>>>>> how > >>>>>>>>>>>> to configure `session.timeout.ms`. In our Kubernetes > >>>>>>>>

Re: Apache Kafka 3.7.0 Release

for unexpected blockers/delays and time away during the winter holidays Thoughts? On Mon, Oct 23, 2023 at 3:14 PM Sophie Blee-Goldman wrote: > Can you add the 3.7 plan to the release schedule page? > > (this --> > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+p

Re: Apache Kafka 3.7.0 Release

Can you add the 3.7 plan to the release schedule page? (this --> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan) Thanks! On Sun, Oct 15, 2023 at 2:27 AM Stanislav Kozlovski wrote: > Hey Chris, > > Thanks for the catch! It was indeed copied and I wasn't sure what to make

Re: [DISCUSS] 3.5.2 Release

Thanks for volunteering Luke! On Thu, Oct 12, 2023 at 2:55 AM Levani Kokhreidze wrote: > Hi Divij, > > Thanks for the explanation, makes sense. > > Hi Luke, thanks you! It would be awesome to see 3.5.2 out. > > Best, > Levani > > > On 12. Oct 2023, at 12:39, Luke Chen wrote: > > > > Hi Levani

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hey Nick! First of all thanks for taking up this awesome feature, I'm sure every single Kafka Streams user and dev would agree that it is sorely needed. I've just been catching up on the KIP and surrounding discussion, so please forgive me for any misunderstandings or misinterpretations of the

Re: [DISCUSS] KIP-714: Client metrics and observability

Regarding the naming, I personally think `clientInstanceId` makes sense for the plain clients -- especially if we might later introduce the notion of an `applicationInstanceId`. I'm not a huge fan of `clientsInstanceIds` for the Kafka Streams API, though, can we use `clientInstanceIds` instead?

[jira] [Resolved] (KAFKA-15571) StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn't implement onRestoreSuspended

[ https://issues.apache.org/jira/browse/KAFKA-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman resolved KAFKA-15571. Resolution: Fixed > StateRestoreListener#onRestoreSuspended is ne

Re: [VOTE] KIP-714: Client metrics and observability

This looks great! +1 (binding) Sophie On Wed, Oct 11, 2023 at 1:46 PM Matthias J. Sax wrote: > +1 (binding) > > On 9/13/23 5:48 PM, Jason Gustafson wrote: > > Hey Andrew, > > > > +1 on the KIP. For many users of Kafka, it may not be fully understood > how > > much of a challenge client

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

Hey Colt! Thanks for the KIP -- this will be a great addition to Streams, I can't believe we've gone so long without this. Overall the proposal makes sense, but I had a handful of fairly minor questions and suggestions/requests 1. Seems like the last sentence in the 2nd paragraph of the

Re: Re: Re: [DISCUSS] KIP-972: Add the metric of the current running version of kafka

Just to chime in here since I recently went through a similar thing, I support adding the version as a tag instead of introducing an entirely new metric for this. In fact I just implemented exactly this in a project that uses Kafka, for these reasons: 1. Adding the version as a tag means that all

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

Congrats Justine! On Sat, Sep 23, 2023, 4:36 PM Tom Bentley wrote: > Congratulations! > > On Sun, 24 Sept 2023 at 12:32, Satish Duggana > wrote: > > > Congratulations Justine!! > > > > On Sat, 23 Sept 2023 at 15:46, Bill Bejeck wrote: > > > > > > Congrats Justine! > > > > > > -Bill > > > > >

Re: [ANNOUNCE] New committer: Lucas Brutschy

Congrats Lucas!

Re: Apache Kafka 3.6.0 release

> On Sat, 2 Sept 2023 at 08:44, Sophie Blee-Goldman > wrote: > > > > Hey Satish, someone reported a minor bug in the Streams application > > shutdown which was a recent regression, though not strictly a new one: > was > > introduced in 3.4 I believe. > >

Re: Apache Kafka 3.6.0 release

Hey Satish, someone reported a minor bug in the Streams application shutdown which was a recent regression, though not strictly a new one: was introduced in 3.4 I believe. The fix seems to be super lightweight and low-risk so I was hoping to slip it into 3.6 if that's ok with you? They plan to

Re: [VOTE] KIP-759: Unneeded repartition canceling

+1 (binding) thanks Shay! On Wed, Aug 2, 2023 at 1:19 AM Bruno Cadonna wrote: > Hi, > > +1 (binding) > > Thanks for the KIP! > > Best, > Bruno > > On 8/2/23 1:19 AM, Bill Bejeck wrote: > > I caught up on the discussion thread and the KIP LGTM. > > > > +1(binding) > > > > On Tue, Aug 1, 2023 at

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

> In general, I would propose to limit the scope of a config. In this case > > the config should provide suppliers for state stores for the DSL. > > > > BTW, maybe it is a good idea to let DslStorePlugin extend Configurable. > > > > Best, > > Bruno > >

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

Jul 25, 2023 at 9:20 AM Almog Gavra > > wrote: > > > > > >> Glad you like my KIP-secretary skills ;) > > >> > > >> A2. I'm definitely happy to take your suggestion here and not do > > anything > > >> special w.r.t. Versioned stor

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

and that this KIP is aligned with moving in that direction. > Given how much discussion there was on this KIP, which is minor relative to > making the changes to StoreBuilder API, I'd rather not tie the two > together. > > Cheers & Thanks everyone for the thoughts! > - Almog >

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

design those levers...) It's for sure a good idea to call out "use with > > case" in the JavaDocs for the new method. > > > > > > -Matthias > > > > On 7/21/23 3:34 PM, Sophie Blee-Goldman wrote: > > > I guess I felt a bit uneasy about how

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

here I could not, more power to you! :P On Sun, Jul 23, 2023 at 4:48 PM Sophie Blee-Goldman wrote: > Matthias: > > I'm not sure I agree with (or maybe don't follow) this take: >> >> we need all kind of `StoreTypeSpec` implementations, >> and it might also imply that w

  1   2   3   4   5   6   7   8   >