[jira] [Created] (KAFKA-10485) Use a separate error code for replication related errors

2020-09-15 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10485: - Summary: Use a separate error code for replication related errors Key: KAFKA-10485 URL: https://issues.apache.org/jira/browse/KAFKA-10485 Project: Kafka

Re: [VOTE] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

2020-09-09 Thread Guozhang Wang
+1. Thanks for the KIP Gokul ! Guozhang On Tue, Sep 8, 2020 at 6:52 PM Matthias J. Sax wrote: > +1 (binding) > > On 9/8/20 2:49 PM, Jason Gustafson wrote: > > +1 Thanks for the KIP! > > > > On Thu, Sep 3, 2020 at 12:25 PM Gokul Srinivas > wrote: > > > >> Hi, > >> > >> I would like to call a vo

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

2020-09-09 Thread Guozhang Wang
Hello Bruno, Finally got some time to review your KIP and the discussion thread now.. a few comments below: 1) I'm with Matthias about the newly added numberOfAliveStreamThreads v.s. existing localThreadsMetadata: to me it seems we can always achieve the first based on the second. It seems not wo

Re: [VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

2020-09-08 Thread Guozhang Wang
+1. Thanks! Guozhang On Tue, Sep 8, 2020 at 3:04 PM Ron Dagostino wrote: > +1 (non-binding) -- Thanks, Jason! > > Ron > > On Tue, Sep 8, 2020 at 2:04 PM Jason Gustafson wrote: > > > Hi All, > > > > I'd like to start a vote on KIP-664: > > > > > https://cwiki.apache.org/confluence/display/KAFKA

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

2020-09-08 Thread Guozhang Wang
ts about having variably sized windows, which > this > > config doesn't handle. I like the name change and updated the wiki to > > reflect that, and to clarify that the default value will still be > > Long.MAX_VALUE. > > > > Thanks for your feedback! > > Lea

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

2020-09-08 Thread Guozhang Wang
config doesn't handle. I like the name change and updated the wiki to > reflect that, and to clarify that the default value will still be > Long.MAX_VALUE. > > Thanks for your feedback! > Leah > > On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang wrote: > > > Hel

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

2020-09-08 Thread Guozhang Wang
Hello Leah, Thanks for initiating this. I just have one minor clarification question here: the config "window.size.ms" seems to be used as the default window size when reading from a topic that represents windowed records right? I.e. if there are multiple topics that represent windowed records but

Re: [VOTE] KIP-614: Add Prefix Scan support for State Stores

2020-09-02 Thread Guozhang Wang
Thanks for the KIP Sagar. I'm +1 (binding) too. Guozhang On Tue, Sep 1, 2020 at 1:24 PM Bill Bejeck wrote: > Thanks for the KIP! This is a great addition to the streams API. > > +1 (binding) > > -Bill > > On Tue, Sep 1, 2020 at 12:33 PM Sagar wrote: > > > Hi All, > > > > Bumping the thread ag

Kafka Meetup hosted Online, Wednesday 5:00pm, Sep 2nd, 2020

2020-09-01 Thread Guozhang Wang
Hello folks, The Bay Area Kafka meetup will continue to be hosted online this month, tomorrow (Sep 2nd) at 5:00pm. This time we will have guest speakers from Twitter to talk about their journey to adopt Apache Kafka. *RSVP*: https://www.meetup.com/KafkaBayArea/events/272643868/ *Date* 5:00pm, W

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

2020-08-31 Thread Guozhang Wang
n the schema. > Fixed now! > > -Jason > > On Mon, Aug 31, 2020 at 3:04 PM Guozhang Wang wrote: > > > On Mon, Aug 31, 2020 at 12:28 PM Jason Gustafson > > wrote: > > > > > Hey Guozhang, > > > > > > Thanks for the detailed comments. Re

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

2020-08-31 Thread Guozhang Wang
l be a huge operational improvement > > and overall it looks great to me. > > > > I'm not sure how much value the MaxActiveTransactionDuration metric adds, > > given that we have the --find-hanging option in the tool. As you mention, > > instances of these transactio

Re: Request to subscribe to kafka mailing list

2020-08-31 Thread Guozhang Wang
Hi SaiTejia, You can add yourself to the mailing list, it's self service: https://kafka.apache.org/contact Guozhang On Sat, Aug 29, 2020 at 8:35 AM SaiTeja Ramisetty wrote: > Regards, > SaiTeja - Data Engineer > -- -- Guozhang

Re: [DISCUSS] KIP-664: Provide tooling to detect and abort hanging transactions

2020-08-27 Thread Guozhang Wang
Hi Jason, Thanks for the written KIP. I think this is going to be a very useful tool for operational improvements since with eos in its current stage, we cannot confidently assert that we are bug-free, and even in the future when we are confident this is still going to be leveraged by older versio

Re: [VOTE] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

2020-08-25 Thread Guozhang Wang
+1. Thanks Bruno! Guozhang On Tue, Aug 25, 2020 at 4:00 AM Bruno Cadonna wrote: > Hi, > > I would like to start the vote for > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-662%3A+Throw+Exception+when+Source+Topics+of+a+Streams+App+are+Deleted > > Best, > Bruno > -- -- Guozhang

Re: [DISCUSS] KIP-662: Throw Exception when Source Topics of a Streams App are Deleted

2020-08-24 Thread Guozhang Wang
Hello Bruno, Thanks for the KIP, it sounds good to me as well. Just a minor comment: we would include which package the new "MissingSourceTopicException" class belongs to. Guozhang On Fri, Aug 21, 2020 at 11:53 AM John Roesler wrote: > Thanks for the KIP, Bruno! > > Your proposal sounds goo

Re: KIP idea: Separate publish request from the subscribe request

2020-08-24 Thread Guozhang Wang
Hello Ming, Thanks for bringing this to the community. Just to clarify your proposal, are you suggesting to use a separate port for fetch requests from all other requests including produce, but also e.g. metadata, commit/fetch-offsets, and other inter-broker requests? If yes that would mean the co

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

2020-08-18 Thread Guozhang Wang
I'm leaning towards design B primarily because it reminds me of the Firefox logo which I like a lot. But I also share Adam's concern that it should better not obscure the Kafka logo --- so if we can tweak a bit to fix it my vote goes to B, otherwise A :) Guozhang On Tue, Aug 18, 2020 at 9:48 AM

[jira] [Resolved] (KAFKA-10391) Streams should overwrite checkpoint excluding corrupted partitions

2020-08-12 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10391. --- Fix Version/s: 2.7.0 Resolution: Fixed > Streams should overwrite checkpo

[jira] [Created] (KAFKA-10391) Streams should overwrite checkpoint excluding corrupted partitions

2020-08-11 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10391: - Summary: Streams should overwrite checkpoint excluding corrupted partitions Key: KAFKA-10391 URL: https://issues.apache.org/jira/browse/KAFKA-10391 Project: Kafka

[jira] [Resolved] (KAFKA-9450) Decouple inner state flushing from committing

2020-08-11 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9450. -- Fix Version/s: 2.7.0 Resolution: Fixed > Decouple inner state flushing from committ

Re: [ANNOUNCE] New Kafka PMC Member: John Roesler

2020-08-10 Thread Guozhang Wang
Congratulations! On Mon, Aug 10, 2020 at 1:11 PM Jun Rao wrote: > Hi, Everyone, > > John Roesler has been a Kafka committer since Nov. 5, 2019. He has remained > active in the community since becoming a committer. It's my pleasure to > announce that John is now a member of Kafka PMC. > > Congrat

[jira] [Resolved] (KAFKA-9659) Kafka Streams / Consumer configured for static membership fails on "fatal exception: group.instance.id gets fenced"

2020-08-10 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9659. -- Fix Version/s: 2.6.0 Resolution: Fixed > Kafka Streams / Consumer configured for sta

Re: [ANNOUNCE] Apache Kafka 2.6.0

2020-08-06 Thread Guozhang Wang
; > Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, > avalsa, > > Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, > Brian > > Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, > Chris > > Egerton, Colin Patrick

[jira] [Created] (KAFKA-10362) When resuming Streams active task with EOS, the checkpoint file should be deleted

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10362: - Summary: When resuming Streams active task with EOS, the checkpoint file should be deleted Key: KAFKA-10362 URL: https://issues.apache.org/jira/browse/KAFKA-10362

[jira] [Created] (KAFKA-10357) Handle accidental deletion of repartition-topics as exceptional failure

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10357: - Summary: Handle accidental deletion of repartition-topics as exceptional failure Key: KAFKA-10357 URL: https://issues.apache.org/jira/browse/KAFKA-10357 Project

[jira] [Created] (KAFKA-10356) Handle accidental deletion of sink-topics as exceptional failure

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10356: - Summary: Handle accidental deletion of sink-topics as exceptional failure Key: KAFKA-10356 URL: https://issues.apache.org/jira/browse/KAFKA-10356 Project: Kafka

[jira] [Created] (KAFKA-10355) Handle accidental deletion of source-topics as exceptional failure

2020-08-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10355: - Summary: Handle accidental deletion of source-topics as exceptional failure Key: KAFKA-10355 URL: https://issues.apache.org/jira/browse/KAFKA-10355 Project: Kafka

Re: New Website Layout

2020-08-03 Thread Guozhang Wang
Hello Ben, I took a look at the corresponding PR and that looks great to me. Just one minor comment otherwise I'm all in for the new UX! * The "see full list" at the homepage just has a drop-down list, maybe better to re-direct to "powered by" if we click it. Guozhang On Fri, Jun 26, 2020 at 5:

Re: [ANNOUNCE] New Kafka PMC Member: Mickael Maison

2020-07-31 Thread Guozhang Wang
Congratulations! On Fri, Jul 31, 2020 at 9:24 AM Rajini Sivaram wrote: > Congratulations, Mickael! > > Regards, > > Rajini > > > On Fri, Jul 31, 2020 at 5:08 PM Boyang Chen > wrote: > > > Congrats Mickael! > > > > On Fri, Jul 31, 2020 at 8:47 AM Bill Bejeck wrote: > > > > > Congrats Mickael! >

Re: [VOTE] KIP-450: Sliding Window Aggregations in the DSL

2020-07-28 Thread Guozhang Wang
+1 (binding) On Tue, Jul 28, 2020 at 4:44 PM Matthias J. Sax wrote: > +1 (binding) > > On 7/28/20 4:35 PM, Sophie Blee-Goldman wrote: > > Thanks for the KIP! It's been an enlightening discussion > > > > +1 (non-binding) > > > > Sophie > > > > On Tue, Jul 28, 2020 at 8:03 AM Leah Thomas > wrote:

Re: [VOTE] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

2020-07-27 Thread Guozhang Wang
+1. Thanks Jorge for bringing in this KIP! Guozhang On Mon, Jul 27, 2020 at 10:07 AM Leah Thomas wrote: > Hi Jorge, > > Looks great. +1 (non-binding) > > Best, > Leah > > On Thu, Jul 16, 2020 at 6:39 PM Sophie Blee-Goldman > wrote: > > > Hey Jorge, > > > > Thanks for the reminder -- +1 (non-bi

Re: [VOTE] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-27 Thread Guozhang Wang
> > >> -Matthias > > > >> > > > >> On 5/15/20 11:48 AM, John Roesler wrote: > > > >>> Thanks, Bruno! > > > >>> > > > >>> I’m +1 (binding) > > > >>> > > > >>> -John

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-27 Thread Guozhang Wang
se block > cache. Compaction uses the OS to read in the data to compact. But it > also uses fadvise to tell the kernel to not cache the data in the OS > buffer cache. > > Hope that clears up the conflicts! ;-) > > Best, > Bruno > > On 24.07.20 19:37, Guozhang Wang w

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-24 Thread Guozhang Wang
on is that the memtables are actually not hard > > bounded at all, while the block cache is soft bounded by default but can > > be configured to be hard bounded. And obviously the OS cache is not > > exactly bounded but it shouldn't cause you to run out of usable memory

Re: Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-23 Thread Guozhang Wang
. Is that right? On Wed, Jul 22, 2020 at 4:28 AM Bruno Cadonna wrote: > Hi Guozhang, > > Thank you for your feedback! > > I answered inline. > > Best, > Bruno > > > On 21.07.20 00:39, Guozhang Wang wrote: > > Hello Bruno, > > > > Thanks for the

Re: [DISCUSS] KIP-450: Sliding Windows

2020-07-20 Thread Guozhang Wang
e? I.e., systems are generally easier to use when they follow a few > > simple, composable rules than when they have a lot of different, specific > > rules. > > > > > > == > > > > New point: (4): > > It would be nice to include some examples of use

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

2020-07-20 Thread Guozhang Wang
k off discussion again. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB > > Best, > Bruno > > On 15.05.20 17:11, Bill Bejeck wrote: > > Thanks for the KIP, Bruno. Having sensible, easy to acce

[jira] [Created] (KAFKA-10294) Consider whether some of ProcessorStateException should be auto-handled by Streams

2020-07-19 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10294: - Summary: Consider whether some of ProcessorStateException should be auto-handled by Streams Key: KAFKA-10294 URL: https://issues.apache.org/jira/browse/KAFKA-10294

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-19 Thread Guozhang Wang
tract across collections of metrics. > > My thoughts are when raft is used for other topic partitions, then we would include the partition as a tag in the metric in addition to the `type` and `metric-name`. Does that work? > > Thanks again for the KIP, > > Tom > > > > O

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-19 Thread Guozhang Wang
; wrote: > > > Hi All, > > > > Just a quick update on the proposal. We have decided to move quorum > > reassignment to a separate KIP: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-642%3A+Dynamic+quorum+reassignment > > . > > Th

[jira] [Created] (KAFKA-10283) Consolidate client-level and consumer-level assignment within ClientState

2020-07-16 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10283: - Summary: Consolidate client-level and consumer-level assignment within ClientState Key: KAFKA-10283 URL: https://issues.apache.org/jira/browse/KAFKA-10283 Project

Re: [DISCUSS] KIP-450: Sliding Windows

2020-07-14 Thread Guozhang Wang
Hello Leah, Thanks for the nice written KIP. A few thoughts: 1) I echo the other reviewer's comments regarding the typing: why extending TimeWindow instead of just extending Window? 2) I also feel that emitting policy for this type of windowing aggregation may be different from the existing ones

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

2020-07-10 Thread Guozhang Wang
Hello Colin, Thanks for the nice written KIP. A few meta comments: 1) We need to talk a bit about broker failure detection: is that piggy backed with fencing? i.e. should the controller immediately migrate leader partitions from the fenced brokers? On one side, when a broker is fenced it cannot t

[jira] [Created] (KAFKA-10263) Do not create standbys for those revoking active tasks if it is not stateful

2020-07-09 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10263: - Summary: Do not create standbys for those revoking active tasks if it is not stateful Key: KAFKA-10263 URL: https://issues.apache.org/jira/browse/KAFKA-10263

[jira] [Resolved] (KAFKA-10134) High CPU issue during rebalance in Kafka consumer after upgrading to 2.5

2020-07-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10134. --- Resolution: Fixed > High CPU issue during rebalance in Kafka consumer after upgrading

Re: Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-04 Thread Guozhang Wang
Hello, Thanks for reaching out to the community for this. I think (maybe you've also suggested) it is rather an observation on producer client than on streams client. Generally speaking we want to know if we can fail fast if the metadata cannot be found in producer.send() call. And here are my tho

Re: [VOTE] KIP-623: Add "internal-topics" option to streams application reset tool

2020-06-29 Thread Guozhang Wang
Thanks Joel, the KIP lgtm. A minor suggestion is to explain where users can get the list of internal topics of a given application, and maybe also add it as part of the helper scripts, for example via topology description. Overall, I'm +1 as well (binding). Guozhang On Sat, Jun 27, 2020 at 4:3

Re: [VOTE] KIP-418: A method-chaining way to branch KStream

2020-06-29 Thread Guozhang Wang
+1 (binding). Thanks Ivan! Guozhang On Mon, Jun 29, 2020 at 3:55 AM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > This will be a great addition. Thanks Ivan! > > +1 (non-binding) > > On Fri, Jun 26, 2020 at 7:07 PM John Roesler wrote: > > > Thanks, Ivan! > > > > I’m +1 (bin

Re: [VOTE] KIP-629: Use racially neutral terms in our codebase

2020-06-25 Thread Guozhang Wang
+1 (binding). On Thu, Jun 25, 2020 at 4:32 PM John Roesler wrote: > Thanks, Xavier! > > I’m +1 (binding) > > -John > > On Thu, Jun 25, 2020, at 17:43, Xavier Léauté wrote: > > Hi Everyone, > > > > I would like to initiate the voting process for KIP-629. > > > https://cwiki.apache.org/confluence

[jira] [Created] (KAFKA-10199) Separate state restoration into separate threads

2020-06-25 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10199: - Summary: Separate state restoration into separate threads Key: KAFKA-10199 URL: https://issues.apache.org/jira/browse/KAFKA-10199 Project: Kafka Issue

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-06-24 Thread Guozhang Wang
tarting a new one. Guozhang On Fri, Jun 12, 2020 at 9:26 AM Boyang Chen wrote: > Thanks for the suggestions Guozhang. > > On Thu, Jun 11, 2020 at 2:51 PM Guozhang Wang wrote: > > > Hello Boyang, > > > > Thanks for the updated information. A few questions here: &g

[ANNOUNCE] New committer: Xi Hu

2020-06-24 Thread Guozhang Wang
The PMC for Apache Kafka has invited Xi Hu as a committer and we are pleased to announce that he has accepted! Xi Hu has been actively contributing to Kafka since 2016, and is well recognized especially for his non-code contributions: he maintains a tech blog post evangelizing Kafka in the Chinese

Re: [DISCUSS] KIP-629: Use racially neutral terms in our codebase

2020-06-22 Thread Guozhang Wang
Xavier, thanks for the KIP! The proposed names make sense to me. Guozhang On Mon, Jun 22, 2020 at 4:24 PM Xavier Léauté wrote: > Please check the list for updated config / argument names. > > I also added a proposal to replace the term "blackout" with "backoff", > which is used internally in th

[ANNOUNCE] New committer: Boyang Chen

2020-06-22 Thread Guozhang Wang
The PMC for Apache Kafka has invited Boyang Chen as a committer and we are pleased to announce that he has accepted! Boyang has been active in the Kafka community more than two years ago. Since then he has presented his experience operating with Kafka Streams at Pinterest as well as several featur

[jira] [Resolved] (KAFKA-10167) Streams EOS-Beta should not try to get end-offsets as read-committed

2020-06-18 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10167. --- Resolution: Fixed > Streams EOS-Beta should not try to get end-offsets as read-commit

[jira] [Created] (KAFKA-10184) Flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores

2020-06-18 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10184: - Summary: Flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores Key: KAFKA-10184 URL: https://issues.apache.org/jira/browse/KAFKA

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-06-17 Thread Guozhang Wang
Thanks for the KIP Boyang, +1 from me. Guozhang On Wed, Jun 17, 2020 at 1:40 PM Colin McCabe wrote: > Thanks, Boyang! +1 (binding) > > best, > Colin > > On Mon, Jun 15, 2020, at 12:59, Boyang Chen wrote: > > Thanks for more feedback Colin! I have addressed them in the KIP. > > > > Boyang > >

[jira] [Created] (KAFKA-10167) Streams EOS-Beta should not try to get end-offsets as read-committed

2020-06-15 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10167: - Summary: Streams EOS-Beta should not try to get end-offsets as read-committed Key: KAFKA-10167 URL: https://issues.apache.org/jira/browse/KAFKA-10167 Project

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-06-11 Thread Guozhang Wang
ao wrote: > > > >> Hi, Guozhang and Jason, > >> > >> Thanks for the reply. A couple of more replies. > >> > >> 102. Still not sure about this. How is the tombstone issue addressed in > >> the > >> non-voter and the observer. They

[jira] [Resolved] (KAFKA-9182) Flaky Test org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosEnabled

2020-06-10 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9182. -- Fix Version/s: 2.6.0 Resolution: Fixed Resolved as part of https://issues.apache.org

[jira] [Resolved] (KAFKA-9991) Flaky Test KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled

2020-06-10 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9991. -- Fix Version/s: 2.6.0 Resolution: Fixed > Flaky T

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

2020-06-09 Thread Guozhang Wang
P wiki page > itself, I would like to proceed with the VOTE. > > > -Matthias > > On 5/20/20 12:30 PM, Guozhang Wang wrote: > > Thanks Matthias, > > > > I agree with you on all the bullet points above. Regarding the > admin-client > > outer-loop retries

[jira] [Created] (KAFKA-10122) Consumer should allow heartbeat during rebalance as well

2020-06-08 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-10122: - Summary: Consumer should allow heartbeat during rebalance as well Key: KAFKA-10122 URL: https://issues.apache.org/jira/browse/KAFKA-10122 Project: Kafka

[jira] [Resolved] (KAFKA-10005) Decouple RestoreListener from RestoreCallback and not enable bulk loading for RocksDB

2020-06-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10005. --- Fix Version/s: 2.6.0 Assignee: Guozhang Wang Resolution: Fixed > Decou

[jira] [Resolved] (KAFKA-8803) Stream will not start due to TimeoutException: Timeout expired after 60000milliseconds while awaiting InitProducerId

2020-06-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8803. -- Resolution: Fixed > Stream will not start due to TimeoutException: Timeout expired af

Re: Granting permission for Create KIP

2020-06-08 Thread Guozhang Wang
Done, added Rhett.Wang On Mon, Jun 8, 2020 at 7:40 AM wang120445...@sina.com < wang120445...@sina.com> wrote: > > Please grant permission for Create KIP to wiki ID: wang120445...@sina.com > Rhett.wang > > > wang120445...@sina.com > -- -- Guozhang

Re: Please I Want to Subscribe

2020-06-04 Thread Guozhang Wang
Hello Ricardo, Here you go for the guidelines, it is self-service: https://kafka.apache.org/contact Guozhang On Wed, Jun 3, 2020 at 1:09 PM Ricardo Ferreira wrote: > Please I Want to Subscribe > > -- -- Guozhang

Re: [VOTE] KIP-589: Add API to update Replica state in Controller

2020-05-28 Thread Guozhang Wang
David, thanks for the KIP. I'm +1 on it as well. One note is that in post-ZK world, we would need a different way to get broker epoch since it is updated as ZKversion today. I believe we would have this discussion in a different KIP though. Guozhang On Wed, May 27, 2020 at 8:26 PM Colin McCabe

[jira] [Resolved] (KAFKA-9561) Update task input partitions when topic metadata changes

2020-05-27 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9561. -- Resolution: Fixed > Update task input partitions when topic metadata chan

[jira] [Resolved] (KAFKA-9898) Flaky Test StoreQueryIntegrationTest.shouldQueryAllStalePartitionStores

2020-05-26 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9898. -- Fix Version/s: 2.6.0 Resolution: Fixed > Flaky T

[jira] [Resolved] (KAFKA-10022) console-producer supports the setting of client.id

2020-05-24 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10022. --- Fix Version/s: 2.6.0 Resolution: Fixed > console-producer supports the setting

[jira] [Resolved] (KAFKA-1056) Evenly Distribute Intervals in OffsetIndex

2020-05-20 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-1056. -- Resolution: Fixed > Evenly Distribute Intervals in OffsetIn

Re: [DISCUSS] KIP-609: Use Pre-registration and Blocking Calls for Better Transaction Efficiency

2020-05-20 Thread Guozhang Wang
ogic for AddPartitionToTxn call so that we could > maintain a future queue and wait for all the delta change completions, > instead of blocking on the first sent out one. Does that make sense? > > On Wed, May 20, 2020 at 2:28 PM Guozhang Wang wrote: > > > Hello Matthias, > >

Re: [DISCUSS] KIP-609: Use Pre-registration and Blocking Calls for Better Transaction Efficiency

2020-05-20 Thread Guozhang Wang
wrote: > > > > > > > > > Thanks Guozhang for the context. The producer batch is either > > bounded by > > > > > the size or the linger time. For the default 10ms linger and 100ms > > > > > transaction commit time, the producer will be capped by > >

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

2020-05-20 Thread Guozhang Wang
throwing any timeout exception: if all tasks are RUNNING > and just polling for new data, but no progress is made because of a > network issue, `poll()` would just return no data but not through. > > > > @Bruno > > > Wouldn't it be better to specify > > task.timeo

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

2020-05-19 Thread Guozhang Wang
r` know if it's a > wrapped serde or not? Furthermore, how do wrapped serdes expose their > inner serdes? > > I am also not sure what the purpose of TopologyDescriber is? Would it > mabye be better to add new interface `Serdes` can implement instead? > > > -Matthias >

Re: [VOTE] KIP-613: Add end-to-end latency metrics to Streams

2020-05-19 Thread Guozhang Wang
+1 on the updated proposal. Thanks Sophie. On Tue, May 19, 2020 at 11:51 AM John Roesler wrote: > Thanks for the KIP, Sophie. > > I'm +1 (binding) on the current proposal. > > Thanks, > -John > > On Fri, May 15, 2020, at 19:45, Guozhang Wang wrote: > > Hi Sop

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

2020-05-18 Thread Guozhang Wang
I will try to implement this in my existing PR and after I've confirmed it works, I will update the final wiki for voting. Guozhang On Mon, May 18, 2020 at 9:13 PM Guozhang Wang wrote: > Hello Andy, > > Thanks a lot for your comments! I do not mind at all :) > > I think

Re: [DISCUSS] KIP-598: Augment TopologyDescription with store and source / sink serde information

2020-05-18 Thread Guozhang Wang
signs labels to Serde names. > > For example, for key and value serdes the label could be "key" and > > "value". Or something similar. > > > > Best, > > Bruno > > > > > > > > On Mon, May 4, 2020 at 2:25 AM Guozhang Wang wro

Re: [VOTE] KIP-608: Expose Kafka Metrics in Authorizer

2020-05-18 Thread Guozhang Wang
Zhiguo, thanks for the KIP. +1 from me. Guozhang On Fri, May 15, 2020 at 2:21 PM Zhiguo Huang wrote: > Thanks to everyone for their input. I've incorporated the changes, and I > think this is ready for voting. > > To summarize, the KIP simply proposes to add a feature which expose > instance o

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

2020-05-18 Thread Guozhang Wang
Hi Matthias, Sorry for flooding the thread, but with this KIP I feel the design scope of https://issues.apache.org/jira/browse/KAFKA-6520 can be simplified a lot and may it the design can be just piggy-backed as part of this KIP, wdyt? Guozhang On Mon, May 18, 2020 at 9:47 AM Guozhang Wang

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

2020-05-18 Thread Guozhang Wang
emark: Wouldn't it be better to specify > task.timeout.ms to -1 if no retry should be done? IMO it would make > the config more intuitive because 0 would not have two possible > meanings (i.e. try once and never try) anymore. > > Best, > Bruno > > On Sat, May 16, 2020 at

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

2020-05-18 Thread Guozhang Wang
e caching (or putting into the state store if caching is > > disabled), right? > > > > Best, > > Bruno > > > > On Sat, May 16, 2020 at 6:57 PM Guozhang Wang > wrote: > > > > > > We do have a big JIRA for making sensor recording to be sufficient. But

Re: [DISCUSS] KIP-609: Use Pre-registration and Blocking Calls for Better Transaction Efficiency

2020-05-17 Thread Guozhang Wang
here record processing and send() happen concurrently. > > Boyang > > On Sat, May 16, 2020 at 10:19 AM Guozhang Wang wrote: > > > Hello Boyang, > > > > Thanks for the proposed KIP, overall it makes sense to me. > > > > One non-public API related point

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

2020-05-16 Thread Guozhang Wang
Hello Matthias, Thanks for the updated KIP, overall I'm +1 on this proposal. Some minor comments (I know gmail mixed that again for me so I'm leaving it as a combo for both DISCUSS and VOTE :) 1) There are some inconsistent statements in the proposal regarding what to deprecated: at the beginning

Re: [DISCUSS] KIP-609: Use Pre-registration and Blocking Calls for Better Transaction Efficiency

2020-05-16 Thread Guozhang Wang
Hello Boyang, Thanks for the proposed KIP, overall it makes sense to me. One non-public API related point that I'd like to make though, is that in KafkaProducer.send call we can potentially defer sending AddPartitionsToTxn request until the sender is about to send the first batch -- this is what

[jira] [Resolved] (KAFKA-9617) Replica Fetcher can mark partition as failed when max.message.bytes is changed

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9617. -- Fix Version/s: 2.6.0 Resolution: Fixed > Replica Fetcher can mark partition as fai

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

2020-05-16 Thread Guozhang Wang
ng! > > -John > > On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote: > > Cool, that's good points -- we should not let our current metrics > proposal > > to depend on future roadmaps, since they can still be removed later if > ever > > becoming not valuable

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

2020-05-16 Thread Guozhang Wang
slightly off but only on the order of ms and > not on the order > of hours, and it's still more useful than no metric at all > > > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang wrote: > > > Hi Sophie, > > > > I think your motivation makes sense, in fact

[jira] [Resolved] (KAFKA-10001) Store's own restore listener should be triggered in store changelog reader

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10001. --- Fix Version/s: 2.6.0 Resolution: Fixed > Store's own restore listener s

[jira] [Resolved] (KAFKA-8918) Flaky Test org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskCancellation

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8918. -- Resolution: Fixed > Flaky T

[jira] [Resolved] (KAFKA-8197) Flaky Test kafka.server.DynamicBrokerConfigTest > testPasswordConfigEncoderSecretChange

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8197. -- Resolution: Fixed > Flaky Test kafka.server.DynamicBrokerConfigT

[jira] [Resolved] (KAFKA-7042) Fall back to the old behavior when the broker is too old to recognize LIST_OFFSET versions

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-7042. -- Resolution: Not A Problem > Fall back to the old behavior when the broker is too old

[jira] [Resolved] (KAFKA-5719) Create a quickstart archetype project for Kafka Streams

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5719. -- Resolution: Fixed > Create a quickstart archetype project for Kafka Stre

[jira] [Resolved] (KAFKA-1985) Document on possible error codes for each response type

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-1985. -- Resolution: Fixed > Document on possible error codes for each response t

[jira] [Resolved] (KAFKA-1617) Move Metadata Cache to TopicManager and handling of Offset Request to LogManager

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-1617. -- Resolution: Won't Fix > Move Metadata Cache to TopicManager and handling of Offset Re

[jira] [Resolved] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3103. -- Resolution: Fixed > Transient Failure

[jira] [Resolved] (KAFKA-3168) Failure in kafka.integration.PrimitiveApiTest.testPipelinedProduceRequests

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3168. -- Resolution: Fixed > Failure in kafka.integration.PrimitiveApiTest.testPipelinedProduceReque

[jira] [Resolved] (KAFKA-3738) Add load system tests for Streams

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3738. -- Resolution: Fixed We have soak tests and perf benchmarks today. > Add load system tests

[jira] [Resolved] (KAFKA-1602) Use single error code for offset commit response

2020-05-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-1602. -- Resolution: Not A Problem > Use single error code for offset commit respo

<    1   2   3   4   5   6   7   8   9   10   >