Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

2024-08-20 Thread Guozhang Wang
ault implementation to be something like > hash(topology.describe()) -- although it probably makes sense to > implement it more efficiently. > > If we agree to use a sensitive topology ID, we'd need to make sure > that rolling bounces still work without completely pausing p

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

2024-08-19 Thread Guozhang Wang
but it would also mean we cannot remain available even during > minor modifications to the topology - something that is possible > today. > > w.r.t. a) - we'd still need to have an initialization step at some > point to tell the broker about the required internal topics etc. >

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

2024-08-19 Thread Guozhang Wang
least > > as long as KIP-1035 hasn't landed. However, as you correctly pointed > > out, the assignor will have to accept offsets for overlapping sets of > > dormant tasks. I updated the KIP to make this explicit. If the > > corresponding offset information for one task conflicts be

Re: KIP-1077: Deprecate `ForeachProcessor` and move to internal package

2024-08-16 Thread Guozhang Wang
Thanks, and good catch! LGTM. On Sun, Aug 4, 2024 at 5:13 AM 黃竣陽 wrote: > > I think this proposal is clear, so should move this class to internal package. > > > TengYao Chi 於 2024年8月4日 下午5:48 寫道: > > > > Cool, I have read the code at trunk, this class is only used in KStream. > > I think this pr

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

2024-08-16 Thread Guozhang Wang
Hello Lucas, Thanks for the great KIP. I've read it through and it looks good to me. As we've discussed, much of my thoughts would be outside the scope of this very well scoped and defined KIP, so I will omit them for now. The only one I had related to this KIP is about topology updating. I under

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2024-05-03 Thread Guozhang Wang
f it received a valid > >>> assignment from the TaskAssignor with only one copy of a task > >>> assigned to a > >>> single KAfkaStreams client, and then somehow ended up assigning that > >>> task > >>> to multiple consumers

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2024-05-01 Thread Guozhang Wang
Jumping back to the party here :) 107: I agree with the rationale behind this, and `numProcessingThreads` looks good to me as it covers both the current and future scenarios. 117: I agree with Lucas and Bruno, and would add: * 117e: unknown taskID: fail * 117f: inconsistent task types (e.g. a

Re: [ANNOUNCE] New committer: Igor Soarez

2024-05-01 Thread Guozhang Wang
Congratulations! On Tue, Apr 30, 2024 at 2:49 AM ziming deng wrote: > > Congrats Igor! > > > On Apr 26, 2024, at 13:53, Christo Lolov wrote: > > > > Congratulations Igor :) ! > > > > On Thu, 25 Apr 2024 at 17:07, Igor Soarez wrote: > > > >> Thanks everyone, I'm very honoured to join! > >> > >>

Re: [ANNOUNCE] Apache Kafka 3.7.0

2024-02-29 Thread Guozhang Wang
; > Abhijeet Kumar, Akhilesh Chaganti, Alieh, Alieh Saeedi, Almog Gavra, > > > Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew > > > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal, > > > Arnout Engelen, Arpit Goyal, Artem Livshits,

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-12-24 Thread Guozhang Wang
t; > their external stores outside of the context of Kafka Streams (with the > > understanding that there are fewer semantic guarantees). > > > > On Sat, Dec 2, 2023 at 9:38 AM Guozhang Wang > > wrote: > > > >> Hey Almog, > >> > >> Sorry fo

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-12-02 Thread Guozhang Wang
M-AGGREGATE-03 > Processor: Aggregate-Project (stores: []) > --> KTABLE-TOSTREAM-06 > <-- Aggregate-Aggregate-ToOutputSchema > Processor: KTABLE-TOSTREAM-000006 (stores: []) > --> KSTREAM-SINK-07 > <-- Aggregate-Pro

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-11-18 Thread Guozhang Wang
> >>> > > >>> On Fri, Jul 28, 2023 at 4:55 PM Almog Gavra > > >> wrote: > > >>> > > >>>> OK! I think I got everything, but I'll give the KIP another read with > > >>>> fresh eyes later. Ju

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

2023-11-18 Thread Guozhang Wang
Thanks Alieh, I read through the wiki page and the DISCUSS thread, all LGTM except a minor thing in javadoc: "The query returns the records with a global ascending order of keys. The records with the same key are ordered based on their insertion timestamp in ascending order. Both the global and p

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2023-11-18 Thread Guozhang Wang
Hi Rohan, I took another look at the updated wiki page and do not have any major questions. Regarding returning a plugin object v.s. configuring a plugin object, I do not have a strong opinion except that the latter seems more consistent with existing patterns. Just curious, any other motivations

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2023-11-09 Thread Guozhang Wang
Hello Rohan, Thanks for the KIP! Overall it looks very nice. Just some quick thoughts : 1. All the API logic is granular at the Task level, except the previousOwnerForPartition func. I’m not clear what’s the motivation behind it, does our controller also want to change how the partitions->tasks m

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

2023-10-29 Thread Guozhang Wang
ture-proofing. > >>>>> > >>>>> However, it currently includes the caveat that READ_UNCOMMITTED is > >> not > >>>>> available under EOS. I think this is the coupling you are alluding > >> to? > >>>>> > >>>>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

2023-10-28 Thread Guozhang Wang
Congratulations Satish! On Sat, Oct 28, 2023 at 12:59 AM Luke Chen wrote: > > Congrats Satish! > > Luke > > On Sat, Oct 28, 2023 at 11:16 AM ziming deng > wrote: > > > Congratulations Satish! > > > > > On Oct 27, 2023, at 23:03, Jun Rao wrote: > > > > > > Hi, Everyone, > > > > > > Satish Duggan

Re: [VOTE] KIP-988 Streams StandbyUpdateListener

2023-10-17 Thread Guozhang Wang
+1 from me. On Mon, Oct 16, 2023 at 1:56 AM Lucas Brutschy wrote: > > Hi, > > thanks again for the KIP! > > +1 (binding) > > Cheers, > Lucas > > > > On Sun, Oct 15, 2023 at 9:13 AM Colt McNealy wrote: > > > > Hello there, > > > > I'd like to call a vote on KIP-988 (co-authored by my friend and c

Re: [VOTE] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

2023-10-17 Thread Guozhang Wang
Seems my previous msg was sent to the wrong recipient, just resending.. On Fri, Oct 13, 2023 at 7:06 PM Guozhang Wang wrote: > > Thanks Hanyu. I made a pass on the KIP and read through the DISCUSS > thread. Do not have any comments. +1 > > On Fri, Oct 13, 2023 at 9:29 AM Hany

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

2023-10-14 Thread Guozhang Wang
ementation detail. Under > > normal circumstances (no rebalances or task migrations), the changelog > > reader will be in STANDBY_UPDATING, which means it will be updating standby > > tasks as long as there are new records in the changelog topic. That's why I > > pref

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

2023-10-13 Thread Guozhang Wang
Hello Nick, First of all, thanks a lot for the great effort you've put in driving this KIP! I really like it coming through finally, as many people in the community have raised this. At the same time I honestly feel a bit ashamed for not putting enough of my time supporting it and pushing it throu

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

2023-10-13 Thread Guozhang Wang
Hello Colt, Thanks for writing the KIP! I have read through the updated KIP and overall it looks great. I only have minor naming comments (well, aren't naming the least boring stuff to discuss and that takes the most of the time for KIPs :P): 1. I tend to agree with Sophie regarding whether or no

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

2023-10-13 Thread Guozhang Wang
Thanks Alieh for the KIP, as well as a nice summary of all the discussions! Just my 2c regarding your open questions: 1. I just checked KIP-889 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores) and we used "VersionedRecord get(K key, long asOfTimestamp);", so I

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

2023-09-22 Thread Guozhang Wang
Congratulations! On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai wrote: > > Congratulations Justine! > > On Fri, Sep 22, 2023, 19:25 Philip Nee wrote: > > > Congrats Justine! > > > > On Fri, Sep 22, 2023 at 7:07 PM Luke Chen wrote: > > > > > Hi, Everyone, > > > > > > Justine Olshan has been

Re: [ANNOUNCE] New committer: Lucas Brutschy

2023-09-21 Thread Guozhang Wang
Congratulations, Lucas! On Thu, Sep 21, 2023 at 8:45 AM Bruno Cadonna wrote: > > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > Lucas Brutschy. > > Lucas' major contributions are around Kafka Streams. > > Lucas' significantly contributed to the state updater >

Re: [ANNOUNCE] New committer: Yash Mayya

2023-09-21 Thread Guozhang Wang
Congrats, Yash! On Thu, Sep 21, 2023 at 8:28 AM Bruno Cadonna wrote: > > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > Yash Mayya. > > Yash's major contributions are around Connect. > > Yash authored the following KIPs: > > KIP-793: Allow sink connectors to be

Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

2023-08-06 Thread Guozhang Wang
; > thanks for pointing out ValueJoinerWithKey. In the end, it's just a > documentation change, ie, point out that the passed in key could be > `null` and similar? > > -Matthias > > > On 8/2/23 3:20 PM, Guozhang Wang wrote: > > Thanks Florin for the writeup, > &

Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

2023-08-02 Thread Guozhang Wang
Thanks Florin for the writeup, One quick thing I'd like to bring up is that in KIP-149 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner) we introduced ValueJoinerWithKey which is aimed to enhance ValueJoiner. It

Re: [VOTE] KIP-954: expand default DSL store configuration to custom types

2023-07-29 Thread Guozhang Wang
Thanks Almog! I made a pass over the updated wiki and have no more questions. +1 Guozhang On Wed, Jul 26, 2023 at 8:14 AM Almog Gavra wrote: > > Hello Everyone, > > Opening the voting for KIP-954. The discussion is converging, but please > feel free to chime in on the last few conversation point

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-07-27 Thread Guozhang Wang
ither RocksDB or InMemory, based on what > is returned for the #persistent >API by the StreamJoined's WindowStoreSupplier > > Does that sound right? We can clarify this further in the KIP if need be > > On Thu, Jul 27, 2023 at 10:48 AM Guozhang Wang > wrote:

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-07-27 Thread Guozhang Wang
define one enum for each > >> Config > >>>>>>> class > >>>>>>>> so there is no accidentally requesting a LIST_VALUE subtype on a > >>>>>>>> WindowStore). Thoughts? > >>>>>>>> > >&

Re: plz help me code review

2023-07-27 Thread Guozhang Wang
Hello Xiangyuan, Thanks for your contributions and thanks for raising it to community's attention. I will take a look at it sooner than later. On Thu, Jul 27, 2023 at 2:41 AM Xiangyuan LI wrote: > > Hi kafka team: > I raise a pr https://github.com/apache/kafka/pull/13965 to fix > https://issue

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

2023-07-23 Thread Guozhang Wang
Thanks everyone for the great discussions so far! I first saw the JIRA and left some quick thoughts without being aware of the already-written KIP (kudos to Almog, very great one) and the DISCUSS thread here. And I happily find some of my initial thoughts align with the KIP already :) Would like t

Re: Request permission to contribute

2023-07-23 Thread Guozhang Wang
Hello Taras, I saw your ID in the contributors list already, could you check if you can create JIRAs now? Guozhang On Fri, Jul 21, 2023 at 4:00 AM Taras Ledkov wrote: > > Hi, Kafka Team. > > I'm following this wiki to request permission to contribute to Apache Kafka > https://cwiki.apache.org/c

Re: [ANNOUNCE] New committer: Greg Harris

2023-07-10 Thread Guozhang Wang
Congratulations! On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch wrote: > > Congratulations, Greg. > > On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison > wrote: > > > Congratulations Greg! > > > > On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck > > wrote: > > > > > > Congrats Greg! > > > > > > -Bill >

Re: [ANNOUNCE] New PMC chair: Mickael Maison

2023-04-24 Thread Guozhang Wang
Congratulations Mickael! Guozhang On Sat, Apr 22, 2023 at 9:30 PM Kowshik Prakasam wrote: > > Thanks a lot Jun for your hard work and contributions over the years. > Congrats Mickael on your new role, well deserved! Wishing both of you, and > the rest of the community the very best! > > > Cheer

Re: Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

2023-04-11 Thread Guozhang Wang
e rest of KIP-914 in the 3.5 release. > >>> > >>> Apologies for not noticing these additional semantics implications > >>> earlier, > >>> Victoria > >>> > >>> -- Forwarded message - > >>> From: Victori

Re: Failing tests in Kafka stream is under investigation now

2023-04-06 Thread Guozhang Wang
Hi Luke, Thanks for investigating it, I think my PR broke some tests somehow --- I'm still looking through it why the local runs did not capture them.. my apologies. Guozhang On Thu, Apr 6, 2023 at 4:57 AM Chia-Ping Tsai wrote: > > hi Luke > > Thanks for your time and effort. It would be nice t

[jira] [Created] (KAFKA-14847) Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS

2023-03-24 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-14847: - Summary: Separate the callers of commitAllTasks v.s. commitTasks for EOS(-v2) and ALOS Key: KAFKA-14847 URL: https://issues.apache.org/jira/browse/KAFKA-14847

Re: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-20 Thread Guozhang Wang
+1, thank you Victoria! On Sat, Mar 18, 2023 at 8:27 AM Victoria Xia wrote: > > Hi all, > > I'd like to start a vote on KIP-914 for updating the Kafka Streams join > processors to use proper timestamp-based semantics in applications with > versioned stores: > https://cwiki.apache.org/confluence/d

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-17 Thread Guozhang Wang
t; history for the provided timestamp, or (the actual case in question) > > because we query older than available history. If you agree, can we add > > this to the KIP? > > > > For left-table-table joins, there seems to be no special impact, but it > > should be cal

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-15 Thread Guozhang Wang
akes sense anyway), there will be no > compatibility concerns with introducing the config in a future release. > It's out of scope for this particular KIP as a result, but can hopefully be > introduced as part of the next release after 3.5. > > Best, > Victoria > > On W

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

2023-03-15 Thread Guozhang Wang
Thanks Victoria for the great writeup, with a thorough analysis and trade-offs. I do not have any major questions about the proposal. Just one thing that might be out of scope: if users want to enable the versioned table feature across the topology, should we allow them to do it via a single confi

Re: Regarding Jira Account Creation

2023-03-14 Thread Guozhang Wang
Hi Rohit, You can request the account creation in this url: https://selfserve.apache.org/jira-account.html On Tue, Mar 14, 2023 at 7:19 AM Rohit - wrote: > > Hi, I am Rohit, working as a Software Engineer at Confluent. Recently I > started working on https://issues.apache.org/jira/browse/KAFKA-1

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-10 Thread Guozhang Wang
Congrats David! On Fri, Mar 10, 2023 at 7:29 AM Randall Hauch wrote: > > Congratulations, David! > > Randall > > On Fri, Mar 10, 2023 at 5:17 AM Viktor Somogyi-Vass > wrote: > > > Congrats David! > > > > On Fri, Mar 10, 2023 at 9:12 AM Tom Bentley wrote: > > > > > Congratulations! > > > > > > O

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-10 Thread Guozhang Wang
Congrats Chris! On Fri, Mar 10, 2023 at 8:41 AM Jeremy Custenborder wrote: > > Congrats buddy! > > On Fri, Mar 10, 2023 at 9:28 AM Randall Hauch wrote: > > > > Congratulations, Chris! > > > > Randall > > > > On Fri, Mar 10, 2023 at 9:07 AM David Arthur > > wrote: > > > > > Congrats, Chris! > >

[jira] [Resolved] (KAFKA-12639) AbstractCoordinator ignores backoff timeout when joining the consumer group

2023-02-28 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-12639. --- Fix Version/s: 3.5.0 Resolution: Fixed > AbstractCoordinator ignores backoff time

Re: [VOTE] KIP-907: Add Boolean Serde to public interface

2023-02-27 Thread Guozhang Wang
+1, thanks! On Fri, Feb 24, 2023 at 5:43 AM SpacRocket wrote: > Hi Everyone, > > I'd like to call for a vote on KIP-907, which proposes new public classes > to the package org.apache.kafka.common.serialization: > - BooleanSerde > - BooleanSerializer > - BooleanDeserializer > > KIP: > lists.apac

Re: [VOTE] KIP-904: Kafka Streams - Guarantee subtractor is called before adder if key has not changed

2023-02-27 Thread Guozhang Wang
+1. On Sun, Feb 26, 2023 at 4:27 PM Fq Public wrote: > > Hi everyone, > > I'd like to start the vote on KIP-904: Kafka Streams - Guarantee subtractor > is called before adder if key has not changed. > The KIP is available here: https://cwiki.apache.org/confluence/x/P5VbDg > The easiest way to vie

Re: [VOTE] KIP-890: Transactions Server Side Defense

2023-02-23 Thread Guozhang Wang
>> > >> On Fri, Feb 3, 2023 at 1:36 AM Matthias J. Sax wrote: > >> > >> > Thanks for the KIP! > >> > > >> > +1 (binding) > >> > > >> > > >> > On 2/2/23 4:18 PM, Artem Livshits wrote: > >>

[jira] [Resolved] (KAFKA-14253) StreamsPartitionAssignor should print the member count in assignment logs

2023-02-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-14253. --- Fix Version/s: 3.5.0 Resolution: Fixed > StreamsPartitionAssignor should print

Re: [ANNOUNCE] New committer: Lucas Bradstreet

2023-02-16 Thread Guozhang Wang
Congratulations, Lucas! On Thu, Feb 16, 2023 at 3:18 PM Kowshik Prakasam wrote: > > Congratulations, Lucas! > > > Cheers, > Kowshik > > On Thu, Feb 16, 2023, 2:07 PM Justine Olshan > wrote: > > > Congratulations Lucas! > > > > Thanks for your mentorship on some of my KIPs as well :) > > > > On T

Re: Requesting permissions to contribute to Apache Kafka.

2023-02-15 Thread Guozhang Wang
Hello Zhangheng, I've granted your permission on the Apache Kafka wiki space. You should be able to create new KIP pages. Guozhang On Wed, Feb 15, 2023 at 6:02 AM hzh0425 wrote: > > Hello, > > My name is zhangheng huang. I want to start contributing to the Apache Kafka > project and want the

Re: Re: [DISCUSS] KIP-904 Kafka Streams - Guarantee subtractor is called before adder if key has not changed

2023-02-14 Thread Guozhang Wang
uot;inconsistent" states. In the worst case, users in this situation will > > need to update any strict tests that check specifically for the presence of > > transient "inconsistent" states. > > > > What do you think? > > > > Thanks, > > F

Re: [DISCUSS] KIP-904 Kafka Streams - Guarantee subtractor is called before adder if key has not changed

2023-02-07 Thread Guozhang Wang
Hello Farooq, Thanks for the very detailed proposal! I think this is a great idea. Just a few thoughts: 1. I regret that we over-optimized the Changed serde format for footprint while making it less extensible. It seems to me that a two rolling bounce migration is unavoidable.. Regarding the form

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2023-01-26 Thread Guozhang Wang
> +1 (binding) > > -Matthias > > On 1/24/23 1:17 PM, Guozhang Wang wrote: > > Hi Matthias: > > > > re "paused" -> "suspended": I got your point now, thanks. Just to > > clarify the two functions are a bit different: "pause

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2023-01-24 Thread Guozhang Wang
quot;: that's a good point. I like it to function in the same way as the "records-rate" metrics. Will update the wiki. re: making "restore-remaining-records-total" at INFO level: sounds good to me too. I will also update the metric name a bit to be more specific. On Thu,

Re: [VOTE] KIP-890: Transactions Server Side Defense

2023-01-23 Thread Guozhang Wang
Thanks Justine, I have no further comments on the KIP. +1. On Tue, Jan 17, 2023 at 10:34 AM Jason Gustafson wrote: > > +1. Thanks Justine! > > -Jason > > On Tue, Jan 10, 2023 at 3:46 PM Colt McNealy wrote: > > > (non-binding) +1. Thank you for the KIP, Justine! I've read it; it makes > > sense t

Re: [DISCUSS] KIP-890 Server Side Defense

2023-01-20 Thread Guozhang Wang
previously returned to > producers. As for not being fatal, I think part of the theory was that in > many cases, the producer would be disconnected. (See point 1) and this > would just be an error to return from the server. I did plan to think about > other cases, so let me know if you t

Re: [DISCUSS] KIP-890 Server Side Defense

2023-01-20 Thread Guozhang Wang
Hello Justine, Thanks for the great write-up! I made a quick pass through it and here are some thoughts (I have not been able to read through this thread so pardon me if they have overlapped or subsumed by previous comments): First are some meta ones: 1. I think we need to also improve the clien

[jira] [Created] (KAFKA-14641) Cleanup CommitNeeded after EOS-V1 is removed

2023-01-19 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-14641: - Summary: Cleanup CommitNeeded after EOS-V1 is removed Key: KAFKA-14641 URL: https://issues.apache.org/jira/browse/KAFKA-14641 Project: Kafka Issue Type

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2023-01-19 Thread Guozhang Wang
t; > >> Guozhang, > >> > >> Thanks for the KIP! > >> > >> +1 (binding) > >> > >> Best, > >> Bruno > >> > >> On 25.10.22 22:07, Walker Carlson wrote: > >>> +1 non binding > >>> > >>

Re: [ANNOUNCE] New committer: Josep Prat

2023-01-17 Thread Guozhang Wang
Congratulations, Josep! On Tue, Jan 3, 2023 at 11:23 AM Josep Prat wrote: > Thanks all again! :) > > On Tue, Jan 3, 2023 at 6:19 PM Bill Bejeck > wrote: > > > Congratulations, Josep! > > > > -Bill > > > > On Tue, Dec 20, 2022 at 9:03 PM Luke Chen wrote: > > > > > Congratulations, Josep! > > >

Re: [ANNOUNCE] New committer: Edoardo Comar

2023-01-17 Thread Guozhang Wang
Congratulations, Edoardo! On Tue, Jan 10, 2023 at 9:11 AM Bruno Cadonna wrote: > Congrats! > > Best, > Bruno > > On 10.01.23 11:00, Edoardo Comar wrote: > > Many thanks everyone ! > > > > On Mon, 9 Jan 2023 at 19:40, Rajini Sivaram > wrote: > > > >> Congratulations, Edo! > >> > >> Regards, > >>

Re: [ANNOUNCE] New committer: Satish Duggana

2023-01-17 Thread Guozhang Wang
Congratulations, Satish! On Tue, Jan 10, 2023 at 11:31 AM Rajini Sivaram wrote: > Congratulations, Satish! > > Regards, > > Rajini > > On Tue, Jan 10, 2023 at 5:12 PM Bruno Cadonna wrote: > > > Congrats! > > > > Best, > > Bruno > > > > On 24.12.22 12:44, Manikumar wrote: > > > Congrats, Satish!

Re: [ANNOUNCE] New committer: Justine Olshan

2023-01-17 Thread Guozhang Wang
Congratulations, Justine (I'm also late)! On Wed, Jan 11, 2023 at 12:17 AM Bruno Cadonna wrote: > Hi Justine, > > Re-reading my message I realized that my message might be > misinterpreted. I meant that I am late with congratulating you due to > the holidays, NOT that it took you long becoming a

Re: [ANNOUNCE] New committer: Walker Carlson

2023-01-17 Thread Guozhang Wang
Congrats, Walker! On Tue, Jan 17, 2023 at 2:20 PM Chris Egerton wrote: > Congrats, Walker! > > On Tue, Jan 17, 2023, 17:07 Bill Bejeck wrote: > > > Congratulations, Walker! > > > > -Bill > > > > On Tue, Jan 17, 2023 at 4:57 PM Matthias J. Sax > wrote: > > > > > Dear community, > > > > > > I am

[ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread Guozhang Wang
leasure to announce that Bruno has agreed to join the Kafka PMC. Congratulations, Bruno! -- Guozhang Wang, on behalf of Apache Kafka PMC

[VOTE] KIP-869: Improve Streams State Restoration Visibility

2022-10-12 Thread Guozhang Wang
Hello all, I'd like to start a vote for the following KIP, aiming to improve Kafka Stream's restoration visibility via new metrics and callback methods: https://cwiki.apache.org/confluence/display/KAFKA/KIP-869%3A+Improve+Streams+State+Restoration+Visibility Thanks! -- Guozhang

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-10-12 Thread Guozhang Wang
I've updated the KIP doc and would start calling for a vote for this KIP now. On Tue, Oct 11, 2022 at 4:26 AM Nick Telford wrote: > Hi Guozhang, > > What you propose sounds good to me. Having the more detailed Task-level > metrics at DEBUG makes sense. > > Regards, > > Nick > -- -- Guozhang

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-10-10 Thread Guozhang Wang
t is to monitor the "lag" of a state updater > > thread that might potentially contain multiple state stores. > > Furthermore, do we need to issue a remote call to record this metric or > > do we get this information from each poll? > > > > 3. > > Coul

Re: [ANNOUNCE] New committer: Deng Ziming

2022-10-10 Thread Guozhang Wang
Congrats Ziming! Thanks a lot for your contributions. Guozhang On Mon, Oct 10, 2022 at 9:31 AM Jason Gustafson wrote: > Hi All > > The PMC for Apache Kafka has invited Deng Ziming to become a committer, > and we are excited to announce that he has accepted! > > Ziming has been contributing to K

Re: [DISCUSS] Apache Kafka 3.4.0 release

2022-10-06 Thread Guozhang Wang
4.0 is gonna be the next big milestone :) Thanks Sophie! On Wed, Oct 5, 2022 at 6:08 PM David Jacot wrote: > +1. Thanks, Sophie! > > Le mer. 5 oct. 2022 à 19:57, Luke Chen a écrit : > > > Hi Sophie, > > > > Thanks for volunteering! > > > > Luke > > > > On Thu, Oct 6, 2022 at 6:17 AM José Armand

Re: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-10-04 Thread Guozhang Wang
Hello David, I've made my final pass on the doc and I think it looks good now. +1. Guozhang On Wed, Sep 14, 2022 at 1:37 PM Guozhang Wang wrote: > Thanks David, > > There are a few minor comments pending in the discussion thread, and one > is about whethe

Re: Request for contributor permissions

2022-10-02 Thread Guozhang Wang
Hello Ashmeet, Thanks for your interest in contributing. I've added you with contributor permissions. Guozhang On Sat, Oct 1, 2022 at 4:20 AM Ashmeet Lamba wrote: > Hello, > > I'd like to request for contributor permissions. JIRA ID: ashmeetlamba > > Thank you. > Ashmeet > -- -- Guozhang

Re: I would like to contribute

2022-09-29 Thread Guozhang Wang
de them in the proposal. Guozhang On Thu, Sep 29, 2022 at 1:21 PM Collins, Alex wrote: > Awesome. Yes. alex_collins > ____ > From: Guozhang Wang > Sent: Thursday, September 29, 2022 12:55:27 PM > To: dev@kafka.apache.org > Subject: Re: I would like

Re: I would like to contribute

2022-09-29 Thread Guozhang Wang
Hello Alex, Thank you for your interests!Do you already have an apache id? If yes I can help letting you be able to create corresponding wiki pages. On Thu, Sep 29, 2022 at 11:56 AM Collins, Alex wrote: > I have three things to contribute: > > > * Bug fix for console producer/consumers. I’v

Re: [DISCUSS] KIP-871: Fix ByteBufferSerializer#serialize(String, ByteBuffer) compatible problem

2022-09-26 Thread Guozhang Wang
Hello ShunKang, Thanks for filing the report. I looked at the source code and I agree it's a bug. The reason we did not get it in the past is probably just because we were inefficiently keep byte-copying and hence for almost all cases, the offset is set to 0 when serialization happens. Note that

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-26 Thread Guozhang Wang
> to make things harder. I am not so familiar with Streams though so my > intuition could be wrong here. > > David > > On Mon, Sep 26, 2022 at 7:26 PM Guozhang Wang wrote: > > > > Regarding the version, what I was thinking is that in the HB request, for &

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-26 Thread Guozhang Wang
tions)) but this seems error prone because it > is not clear whether a partition is usable or not. Or do you think > that we should not provide the partitions but only the metadata? > > Best, > David > > On Fri, Sep 23, 2022 at 9:40 PM Guozhang Wang wrote: > > > > Hell

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-23 Thread Guozhang Wang
nd in fact the creation of such tasks as a result of the metadata bookkeeping does not need to wait until all the partitions that are yet-assigned have been completely assigned to the member. Such information may not always be updatable in an incremental manner as the partitions-revoked / parti

Re: [VOTE] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-22 Thread Guozhang Wang
+1, thanks ShunKang. Though its proposed motivation is on consumer fetcher's deserialization, I think adding an overloaded method with ByteBuffer would help with other serde places on the client side as well. Guozhang On Thu, Sep 22, 2022 at 9:41 AM ShunKang Lin wrote: > Hi everyone, > > I'd

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-22 Thread Guozhang Wang
Could you start a separate VOTE email thread calling for votes? On Thu, Sep 22, 2022 at 9:19 AM ShunKang Lin wrote: > Hi Guozhang, > > Thanks for your help! By the way, what should I do next? > > Best, > ShunKang > > Guozhang Wang 于2022年9月22日周四 23:21写道: > > >

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-22 Thread Guozhang Wang
Thanks ShunKang, I made a few nit edits on the Motivation section as well. LGTM for me now. On Thu, Sep 22, 2022 at 7:33 AM ShunKang Lin wrote: > Hi Guozhang, > > I've updated the "Motivation" section of the KIP, please take a look. > > Thanks. > ShunKang >

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-20 Thread Guozhang Wang
Hello Bruno/Nick, I've updated the KIP wiki to reflect the incorporated comments from you, please feel free to take another look and let me know what you think. Guozhang On Tue, Sep 20, 2022 at 9:37 AM Guozhang Wang wrote: > Hi Nick, > > Thanks for the reviews, and I think

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-20 Thread Guozhang Wang
ions. Guozhang On Tue, Sep 20, 2022 at 10:24 AM Guozhang Wang wrote: > 1. Ack, thanks. > 2. Sounds good, thanks for clarifying. > > On Tue, Sep 20, 2022 at 9:50 AM ShunKang Lin > wrote: > >> Hi Guozhang, >> >> Thanks for your comments! >> >> 1. We c

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-20 Thread Guozhang Wang
d `default ByteBuffer serializeToByteBuffer(String > topic, Headers headers, T data)` in Serializer to reduce memory copy in > `KafkaProducer#doSend(ProducerRecord, Callback)`, but this change is a bit > big, I prefer to submit another one KIP to do the job. > > Thanks. > ShunKang

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-20 Thread Guozhang Wang
to derive useful information like the > estimated time remaining for restoration (by dividing the remaining total > by the restoration rate). > > Regards, > > Nick > > On Mon, 19 Sept 2022 at 19:57, Guozhang Wang wrote: > > > Hello Bruno, > > > > Tha

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-19 Thread Guozhang Wang
A separate question regarding the proposed API as well: what do you think to also augment the serializers with ByteBuffer return type in order to be symmetric with deserializers? On Mon, Sep 19, 2022 at 3:32 PM Guozhang Wang wrote: > Hello ShunKang, > > Thanks for filing the prop

Re: [DISCUSS] KIP-863: Reduce Fetcher#parseRecord() memory copy

2022-09-19 Thread Guozhang Wang
Hello ShunKang, Thanks for filing the proposal, and sorry for the late reply! I looked over your KIP proposal and the PR, in general I think I agree that adding an overloaded function with `ByteBuffer` param is beneficial, but I have a meta question regarding it's impact on Kafka consumer: my und

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-19 Thread Guozhang Wang
t. In other words, restoration is finite. Standby tasks rather > update continuously their states. They can be up-to-date or lagging. I > see that you could say "restored" instead of "up-to-date" and "not > restored" instead of "lagging", but IMO

Re: KafkaConsumer refactor proposal

2022-09-16 Thread Guozhang Wang
gt; I'll think again about these naming. > 2. Great suggestions. > 3. I think I'll start with defining a few basic events, like assign, > commits, and fetch. Then we can modify that as we progress with the > project. > > Thanks, > P > > > On Thu, Sep 15, 2022

[DISCUSS] KIP-869: Improve Streams State Restoration Visibility

2022-09-16 Thread Guozhang Wang
Hello everyone, I'd like to start a discussion for the following KIP, aiming to improve Kafka Stream's restoration visibility via new metrics and callback methods: https://cwiki.apache.org/confluence/display/KAFKA/KIP-869%3A+Improve+Streams+State+Restoration+Visibility Thanks! -- Guozhang

[jira] [Created] (KAFKA-14239) Merge StateRestorationIntegrationTest into RestoreIntegrationTest

2022-09-16 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-14239: - Summary: Merge StateRestorationIntegrationTest into RestoreIntegrationTest Key: KAFKA-14239 URL: https://issues.apache.org/jira/browse/KAFKA-14239 Project: Kafka

Re: Request for contributor permissions

2022-09-16 Thread Guozhang Wang
Hello Akshay, Thanks for your interest! You should have the permissions now. Guozhang On Fri, Sep 16, 2022 at 7:40 AM Akshay Bhadange wrote: > Hello sir/madam , > > I am really interested in this project and want to contribute.so please > provide me the contributor permission.my jira id is > (

Re: KafkaConsumer refactor proposal

2022-09-15 Thread Guozhang Wang
Hello Philip, Thanks for writing down the 1-pager. Just to clarify, the reason we wrote this as a 1-pager instead of a KIP is that so far all the implementations should be internal, and hence not impacting anything on the public APIs. If, e.g. we found along with the implementation that some publi

Re: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-14 Thread Guozhang Wang
Thanks David, There are a few minor comments pending in the discussion thread, and one is about whether we should merge PreparePartitionAssignment with HB. But I think the KIP itself is in pretty good shape now. Thanks! Guozhang On Fri, Sep 9, 2022 at 1:32 AM David Jacot wrote: > Hi all, > >

Re: [VOTE] KIP-862: Self-join optimization for stream-stream joins

2022-09-13 Thread Guozhang Wang
liki Papavasileiou wrote: > > Hey Guozhang, > > > > Great suggestion, I made the change. > > > > Best, > > Vicky > > > > On Fri, Sep 9, 2022 at 10:43 PM Guozhang Wang > wrote: > > > >> Thanks Vicky, that reads much clearer now. > >>

Re: [VOTE] KIP-862: Self-join optimization for stream-stream joins

2022-09-09 Thread Guozhang Wang
the new value > strings for each optimization rule. Let me rephrase that in the KIP to make > it clearer. > Is it better now? > > Best, > Vicky > > On Thu, Sep 8, 2022 at 9:07 PM Guozhang Wang wrote: > > > Thanks Vicky, > > > > I read through the KIP agai

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-09-09 Thread Guozhang Wang
? > > Best, > David > > On Fri, Sep 9, 2022 at 7:10 AM Guozhang Wang wrote: > > > > Hello David, > > > > One of Jun's comments make me thinking: > > > > ``` > > In this case, a new assignment is triggered by the client side > > assig

  1   2   3   4   5   6   7   8   9   10   >