Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

2019-08-13 Thread Guozhang Wang
but non-stream user has to code the callback by hand, do you think > > the > > > > > convenience we sacrifice here worth the simplification benefit? > > > > > > > > > > > > > > > > 3. Can you clarify the behavi

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

2019-08-14 Thread Guozhang Wang
listener is to consolidate the entry point of consumer internal > > states. Compared with letting consumer generate a deep-copy of metadata > > every time we call #sendOffsetsToTransactions, using a callback seems > > reducing unnecessary updates towards the metadata. WDY

Re: KIP Permission Request

2017-06-10 Thread Guozhang Wang
Added. Thanks. Guozhang On Sat, Jun 10, 2017 at 1:28 AM, Grant Neale wrote: > Good afternoon, > > > I am writing to request permission to create Kafka Improvement Proposal > (KIP) pages on the Apache Kafka confluence. > > > My wiki username is grantne...@hotmail.com. > > > The KIP relates to a

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

2017-06-10 Thread Guozhang Wang
> Thanks for updating the KIP! > > > > -Matthias > > > > On 6/1/17 6:18 PM, Bill Bejeck wrote: > > > +1 > > > > > > Thanks, > > > Bill > > > > > > On Thu, Jun 1, 2017 at 7:45 PM, Guozhang Wang > > wrote: > > &

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

2017-06-11 Thread Guozhang Wang
Eno, Thanks for bringing this proposal up and sorry for getting late on this. Here are my two cents: 1. First some meta comments regarding "fail fast" v.s. "making progress". I agree that in general we should better "enforce user to do the right thing" in system design, but we also need to keep in

Re: Changing JIRA notifications to remove Single Email Address (dev@kafka.apache.org)

2017-06-12 Thread Guozhang Wang
Bill > > > > On Fri, Jun 9, 2017 at 4:56 PM, Gwen Shapira wrote: > > > >> +1, thank you > >> > >> On Fri, Jun 9, 2017 at 1:37 PM Jeff Widman wrote: > >> > >>> +1 > >>> > >>> Thanks for driving this > &

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

2017-06-14 Thread Guozhang Wang
setters on it and also need to > > have > > >> the end offset available such that people can use it derive progress. > > >> Slightly different, maybe the StateRestoreContext interface could be: > > >> > > >> long beginOffset() > > &g

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-06-14 Thread Guozhang Wang
er, we can check if we get the > > `withKey` variant and use a corresponding runtime class for execution, > > so we get only a single time check. Thus, for the `withKey` variant, the > > will be a `transfrom(V value)` method, but we will never call it. > > > > Maybe we could

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2017-06-14 Thread Guozhang Wang
gt; > >>> > > >>> 2) where the serde should be defined (if necessary). Looking at our > > >>> existing APIs in KGroupedStreams, we always offer two aggregate() > > >>> methods. The first one takes the name of the store and associated > >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

2017-06-14 Thread Guozhang Wang
ny more overloads. > > > > For those reason, I advocate for a simple interface with a single method > > that passes in a RecordContext object. > > > > > > -Matthias > > > > > > On 6/6/17 5:15 PM, Guozhang Wang wrote: > > > Thanks for the co

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2017-06-14 Thread Guozhang Wang
other streams we will just drop it on the floor. If that is actually not the case, that we call initializer on any one of the co-grouped streams' incoming records, then I'm open to set the initializer at the `aggregate` call as well. Guozhang On Wed, Jun 14, 2017 at 2:23 PM, Guozhang Wang

[NOTICE] New mailing list for JIRA notifications

2017-06-15 Thread Guozhang Wang
Hello All, We have just added a new mailing list in which all JIRA notifications will be sent to. At the same time, will only be notified on "ISSUE {CREATED, RESOLVED, REOPENED}" moving forward. I will update the web docs (https://kafka.apache.org/contact) shortly as well. We hope this will hel

Re: [VOTE] KIP-160: Augment KStream.print() to allow users pass in extra parameters in the printed string

2017-06-15 Thread Guozhang Wang
James, As I commented on the discussion thread, have you considered doing the same for writeAsText as well? Guozhang On Thu, Jun 15, 2017 at 4:33 PM, Matthias J. Sax wrote: > +1 > > Thanks for the KIP! > > > -Matthias > > On 6/14/17 6:59 PM, James Chain wrote: > > I had start a voting process

Re: [VOTE] KIP-160: Augment KStream.print() to allow users pass in extra parameters in the printed string

2017-06-16 Thread Guozhang Wang
Yes. Please update the wiki page of KIP-160 as well, and then recall the voting process. For those who already voted, please re-review the wiki page and re-cast your vote. Guozhang On Thu, Jun 15, 2017 at 7:08 PM, James Chain wrote: > @Guozhang > > OK, I will add it on writeAsText as well. > S

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

2017-06-21 Thread Guozhang Wang
etStateRestoreListener" on KafkaStreams as this allows users > >> to > >> >> define an anonymous instance that has access to local scope for > >> reporting > >> >> purposes. This is a similar pattern we use for > >> >> KafkaStreams.setStateListene

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

2017-06-21 Thread Guozhang Wang
More specifically, if we can replace the first parameter from the String store name to the store instance itself, would that be sufficient to cover ` StateRestoreNotification`? On Wed, Jun 21, 2017 at 7:13 PM, Guozhang Wang wrote: > Bill, > > I'm wondering w

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

2017-06-21 Thread Guozhang Wang
KIP-161:+streams+deserialization+ > exception+handlers> > > Any other feedback appreciated, otherwise I’ll start the vote soon. > > Thanks > Eno > > > On Jun 12, 2017, at 6:28 AM, Guozhang Wang wrote: > > > > Eno, Thanks for bringing this proposal up and sorr

Re: [VOTE] KIP-161: streams deserialization exception handlers

2017-06-23 Thread Guozhang Wang
+1. On Fri, Jun 23, 2017 at 6:42 AM, Bill Bejeck wrote: > Thanks for the KIP! > > +1 > > -Bill > > On Fri, Jun 23, 2017 at 7:15 AM, Damian Guy wrote: > > > Thanks for the KIP Eno. > > +1 (binding) > > > > On Fri, 23 Jun 2017 at 11:00 Eno Thereska > wrote: > > > > > Starting voting thread for:

Re: KIP-160 - Augment KStream.print() to allow users pass in extra parameters in the printed string

2017-06-23 Thread Guozhang Wang
James, Does it make sense to include `writeAsText` also in your KIP? Other than that the proposal lgtm. Re-casting my +1 here. Guozhang On Mon, Jun 12, 2017 at 6:59 PM, James Chain wrote: > Agree with Guozhang, +1 > > James Chien > -- -- Guozhang

Re: [VOTE] KIP-160: Augment KStream.print() to allow users pass in extra parameters in the printed string

2017-06-23 Thread Guozhang Wang
+1 On Fri, Jun 23, 2017 at 3:16 AM, Eno Thereska wrote: > +1 thanks! > > Eno > > On 23 Jun 2017, at 05:29, James Chain wrote: > > > > Hi all, > > > > I apply original idea on KStream#writeAsText() and also update my pull > > request. > > Please re-review and re-cast the vote. > > > > James Chie

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

2017-06-26 Thread Guozhang Wang
s > to > > the state store for resource management tasks. > > > > However I like this approach better and I have some ideas I can do in the > > implementation, so I'll update the KIP accordingly. > > > > Thanks, > > Bill > > > > On Wed,

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

2017-06-26 Thread Guozhang Wang
quot;Sriram Subramanian" wrote: > > > +1 > > > > On Fri, Jun 9, 2017 at 2:24 PM, Jay Kreps wrote: > > > > > +1 > > > > > > -Jay > > > > > > On Thu, Jun 8, 2017 at 11:16 AM, Guozhang Wang > > wrote: > > >

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

2017-06-27 Thread Guozhang Wang
FKA-5494>. Guozhang On Fri, Feb 24, 2017 at 7:44 PM, Guozhang Wang wrote: > Correction: Non-Binding +0 count is 2. > > On Fri, Feb 24, 2017 at 7:44 PM, Guozhang Wang wrote: > >> Hello all, >> >> Thanks a billion for your votes and comments. We have collected

0.11.0.0 RC2 Vote

2017-06-27 Thread Guozhang Wang
%40mail.gmail.com%3E --- On Mon, Jun 26, 2017 at 9:36 PM, Guozhang Wang wrote: > +1 > > Verified 0110 web docs and java docs; verified quick start with 2.11 / > 2.12 scala versions. > > One minor observation: on the web docs we show the cmd for 2.11 scala > version

Re: [DISCUSS] Streams DSL/StateStore Refactoring

2017-06-28 Thread Guozhang Wang
I played the current proposal a bit with https://github.com/dguy/kafka/ tree/dsl-experiment, and here are my observations: 1. Personally I prefer "stream.group(mapper) / stream.groupByKey()" than "stream.group().withKeyMapper(mapper) / stream.group()" Since 1) withKeyMapper is not enfo

Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

2017-06-28 Thread Guozhang Wang
gt; > Aaron Coburn, Adrian McCague, Aegeaner, Akash Sethi, Akhilesh Naidu, Alex > > Loddengaard, Allen Xiang, amethystic, Amit Daga, Andrew Olson, Andrey > > Dyachkov, anukin, Apurva Mehta, Armin Braun, Balint Molnar, Ben Stopford, > > Bernard Leach, Bharat Viswanadham, Bill Bejeck

Re: [DISCUSS] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

2017-06-30 Thread Guozhang Wang
Thanks for the proposal Damian. The PR looks promising to me. One minor comment for the wiki page is that you can add some example on how to call the proposed interface (e.g. the ones you added in unit test in the PR). Another question is that this variable will not be included in the web docs in

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-05 Thread Guozhang Wang
Congrats, Ismael !! Guozhang On Wed, Jul 5, 2017 at 2:06 PM, Rajini Sivaram wrote: > Congratulations, Ismael! > > On Wed, Jul 5, 2017 at 9:55 PM, Jun Rao wrote: > > > Hi, Everyone, > > > > Ismael Juma has been active in the Kafka community since he became > > a Kafka committer about a year ag

Re: Transactional Consistency beyond KIP-98 (transactional consumer)

2017-07-05 Thread Guozhang Wang
Hello Werner, 1) Regarding the long transaction, we did discuss about this in the design doc ( https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=id.z9b0a6k2a3zm), and as stated the main reason we chose to go with the current design is to preserve offset

Re: [DISCUSS] KIP-167: Add interface for the state store restoration process

2017-07-05 Thread Guozhang Wang
Bill, Thanks for the updated KIP. From your WIP PR I think one piece of information that still need to be included in the KIP is that the library will check the type of the restoreCallback in runtime, and if it is extending StateRestoreListener it will also execute its functions at the runtime. SO

Re: [DISCUSS] Streams DSL/StateStore Refactoring

2017-07-07 Thread Guozhang Wang
;>>> it. The reason I like it is because it makes it clear to the user > > that > > >> a > > >>>>> call to KGroupedStream#count will return a KTable not some > > intermediate > > >>>>> class that I need to undetstand. > > &

Re: [DISCUSS] Streams DSL/StateStore Refactoring

2017-07-07 Thread Guozhang Wang
I messed the indentation on github code repos; this would be easier to read: https://codeshare.io/GLWW8K Guozhang On Fri, Jul 7, 2017 at 1:30 AM, Guozhang Wang wrote: > Hi Damian / Kyle, > > I think I agree with you guys about the pros / cons of using the builder > pattern v.s

Re: [DISCUSS] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

2017-07-10 Thread Guozhang Wang
Thanks Damian. LGTM. Guozhang On Mon, Jul 10, 2017 at 1:35 PM, Damian Guy wrote: > Thanks Guozhang, i added a couple of example usages to the KIP > > On Fri, 30 Jun 2017 at 17:06 Guozhang Wang wrote: > > > Thanks for the proposal Damian. > > > > The PR looks prom

[ANNOUNCE] New Kafka PMC member Jason Gustafson

2017-07-11 Thread Guozhang Wang
Hi Everyone, Jason Gustafson has been very active in contributing to the Kafka community since he became a Kafka committer last September and has done lots of significant work including the most recent exactly-once project. In addition, Jason has initiated or participated in the design discussion

Re: Apply for access for Kafka KIP creation permission

2017-07-13 Thread Guozhang Wang
Hi Xi, Have granted you the permissions. Guozhang On Wed, Jul 12, 2017 at 7:58 PM, Hu Xi wrote: > Apply for access for KIP creation permission. > > > WikiID: huxi_2b > > Email: huxi...@hotmail.com > -- -- Guozhang

Re: Comments on JIRAs

2017-07-14 Thread Guozhang Wang
Hello Tom, All committers should have subscribed to the newly created jira@kafka mailing list, so they still get all the notifications. If you want to ping someone who's not committer and not yet watching the ticket, then you do need to ping her id in your comment (once she has replied to your co

Re: [VOTE] KIP-167: Add interface for the state store restoration process

2017-07-15 Thread Guozhang Wang
+1. Thanks Bill ! On Sat, Jul 15, 2017 at 11:01 AM, Sriram Subramanian wrote: > +1 > > On Sat, Jul 15, 2017 at 9:45 AM, Damian Guy wrote: > > > +1 > > > > On Thu, 13 Jul 2017 at 07:13 Eno Thereska > wrote: > > > > > +1 (non-binding). > > > > > > Thanks Bill. > > > > > > Eno > > > > On 12 Jul 2

Re: [VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

2017-07-16 Thread Guozhang Wang
+1. Thanks! On Sat, Jul 8, 2017 at 1:35 AM, Damian Guy wrote: > +1 > On Fri, 7 Jul 2017 at 16:08, Eno Thereska wrote: > > > +1 (non-binding) Thanks. > > > > Eno > > > On 6 Jul 2017, at 21:49, Gwen Shapira wrote: > > > > > > +1 > > > > > > On Wed, Jul 5, 2017 at 9:25 AM Matthias J. Sax > > > w

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

2017-07-17 Thread Guozhang Wang
+1 On Mon, Jul 17, 2017 at 5:42 AM, Eno Thereska wrote: > +1 (non-binding) > > Thanks > Eno > > On Jul 17, 2017, at 12:43 PM, Damian Guy wrote: > > > > Hi, > > > > I'd like to kick off the vote for KIP-173: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 173%3A+Add+prefix+to+Stream

Re: Dealing with noisy timestamps in kafka streams + 10.1

2017-07-17 Thread Guozhang Wang
Hello Greg, Sorry for the long delay in responses, but very glad to see you have gone this far to resolve the problem. All of your solutions make sense to me. I think for the first problem your raised, it is a general problem not only for Streams' sending changelog records but for any clients tha

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

2017-07-17 Thread Guozhang Wang
gt; to > > new nodes joining? For example, if streams detects a rebalance, before > > processors are initialized, that only add new nodes, if the configuration > > has not been overridden, write to the log? > > > > > > > > On Thu, Jun 8, 2017 at 2:56 PM, Gu

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

2017-07-18 Thread Guozhang Wang
n thousands of rebalances occur back to back and will > likely take quite a bit bandwidth. I am a little worried about the > performance impact in that case. Although request quota might help to > throttle the rebalance, that seems not the most ideal solution. > > Thanks, > >

[DISCUSS] 2017 October release planning and release version

2017-07-18 Thread Guozhang Wang
Hi all, With 0.11.0.0 out of the way, I would like to volunteer to be the release manager for our next time-based feature release. See https://cwiki.apache.org/ confluence/display/KAFKA/Time+Based+Release+Plan if you missed previous communication on time-based releases or need a reminder. I put t

Re: [DISCUSS] 2017 October release planning and release version

2017-07-18 Thread Guozhang Wang
and our versioning > policy. > > A couple of minor questions inline. > > On Tue, Jul 18, 2017 at 3:36 PM, Guozhang Wang wrote: > > > major.minor.bugfix[.release-candidate] > > > > I think you mean major.minor.bugfix-rc (i.e. we typically use a dash > instead o

Re: [DISCUSS] 2017 October release planning and release version

2017-07-18 Thread Guozhang Wang
gt; > I am +1 on naming the next release 1.0.0. As you said, Kafka is mature > > enough and this will make it easier for others to understand our > versioning > > policy. > > > > A couple of minor questions inline. > > > > On Tue, Jul 18, 2017 at 3:36 PM, Gu

Re: [DISCUSS] 2017 October release planning and release version

2017-07-19 Thread Guozhang Wang
Hi Stevo, Just trying to add to what Ismael has already replied you: > Practice/"features" like protocol version being a parameter, and defaulting > to latest so auto updated with dependency update which introduces new > protocol/behavior should not be used in public client APIs. To switch > bet

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

2017-07-19 Thread Guozhang Wang
Per 1. I suggested exposing the constant since we are doing so for consumer and producer configs prefix as well (CONSUMER_PREFIX, etc). Guozhang On Wed, Jul 19, 2017 at 6:01 AM, Ismael Juma wrote: > Thanks for the KIP, Damian. +1 (binding). A couple of minor comments: > > 1. Do we need to expos

Re: [DISCUSS] 2017 October release planning and release version

2017-07-20 Thread Guozhang Wang
ould drop the Unstable annotations completely. > > > > Cheers, > > Damian > > > > On Wed, 19 Jul 2017 at 21:36 Guozhang Wang wrote: > > > > > Hi Stevo, > > > > > > Just trying to add to what Ismael has already replied you: > > > &

Re: [DISCUSS] 2017 October release planning and release version

2017-07-20 Thread Guozhang Wang
.11> . Guozhang On Thu, Jul 20, 2017 at 4:51 AM, Damian Guy wrote: > +1 on 1.0! > Are we also going to move to java 8? > I also think we should drop the Unstable annotations completely. > > Cheers, > Damian > > On Wed, 19 Jul 2017 at 21:36 Guozhang Wang wrote: > > &g

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-20 Thread Guozhang Wang
Colin, Thanks for the writeup! This KIP lgtm. Guozhang On Thu, Jul 20, 2017 at 10:52 AM, Colin McCabe wrote: > Hi all, > > I posted "KIP-180: Add a broker metric specifying the number of consumer > group rebalances in progress" for discussion: > > https://cwiki.apache.org/confluence/display

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-21 Thread Guozhang Wang
My two cents: "Dead" and "Empty" states are transient: groups usually only leaves in this state for a short while and then being deleted or transited to other states. Since we have the existing "*NumGroups*" metric, `*NumGroups - **NumGroupsRebalancing - **NumGroupsAwaitingSync`* should cover the

Re: [DISCUSS] 2017 October release planning and release version

2017-07-21 Thread Guozhang Wang
Thanks Ismael. I agree with you on all these points, and for some of these points like 3) we never have a written-down policy though in practice we tend to follow some patterns. To me deciding what's the version number of the next major release does not necessarily mean we need now to change any o

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-21 Thread Guozhang Wang
tes if we're going to cover any of > them specifically. > > -Jason > > > > On Fri, Jul 21, 2017 at 9:45 AM, Guozhang Wang wrote: > > > My two cents: > > > > "Dead" and "Empty" states are transient: groups usually only leaves in >

Re: [VOTE] KIP-167 (Addendum): Add interface for the state store restoration process

2017-07-21 Thread Guozhang Wang
+1 On Thu, Jul 20, 2017 at 11:00 PM, Matthias J. Sax wrote: > +1 > > On 7/20/17 4:22 AM, Bill Bejeck wrote: > > Hi, > > > > After working on the PR for this KIP I discovered that we need to add and > > additional parameter (TopicPartition) to the StateRestoreListener > interface > > methods. > >

Re: [DISCUSS] 2017 October release planning and release version

2017-07-21 Thread Guozhang Wang
e some expectations that people > have for projects that have reached 1.0.0 and we should try to allocate > time for the important ones. > > Ismael > > On 21 Jul 2017 8:07 pm, "Guozhang Wang" wrote: > > > Thanks Ismael. I agree with you on all these points, and f

Re: Kafka doc : official repo or web site repo ?

2017-07-21 Thread Guozhang Wang
Ewen, Paolo: There is already a PR from Derrick who did the kafka-site PR for kafka repo as well: https://github.com/apache/kafka/pull/3540 We usually only first push to kafka, and then piggy-back to kafka-site upon releases; for some changes that we'd like to push in the middle of a release cycl

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-24 Thread Guozhang Wang
g this state name publicly in these metrics, > perhaps it makes sense to do this rename now. Thoughts? > > best, > Colin > > > On Fri, Jul 21, 2017, at 13:52, Colin McCabe wrote: > > That's a good point. I revised the KIP to add metrics for all the group > > stat

Re: [DISCUSS] 2017 October release planning and release version

2017-07-24 Thread Guozhang Wang
gt; > Other projects (like Flink) do a fantastic job with this regard and we > > should learn from them. > > > > -Matthias > > > > On 7/21/17 9:50 PM, Guozhang Wang wrote: > > > That's fair enough too. > > > > > > Guozhang > > &g

Re: [DISCUSS] 2017 October release planning and release version

2017-07-24 Thread Guozhang Wang
el to to sync up with you (probably over JIRA) since he mentioned he will create a ticket for capturing all the aspects. Guozhang On Mon, Jul 24, 2017 at 12:28 PM, Guozhang Wang wrote: > Thanks for everyone who has shared their thoughts and feedbacks to this > discussion. Here is my conclu

1.0.0 First Update

2017-07-24 Thread Guozhang Wang
Hello all, I'm still working on the tentative release dates for the next 1.0.0 major release, but just want to give you a heads up now: 1. I have updated the release versions in KIP wiki and the release plan. Details of the coming major release can be found in this page: https://cwiki.apache.org

Re: [DISCUSS] 2017 October release planning and release version

2017-07-25 Thread Guozhang Wang
Makes sense, have updated the wiki page to use dashes for rc numbers. Guozhang On Tue, Jul 25, 2017 at 3:40 AM, Ismael Juma wrote: > On Tue, Jul 18, 2017 at 4:04 PM, Guozhang Wang wrote: > > > I was actually thinking about using dot as well for the rc as well moving > > for

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-26 Thread Guozhang Wang
`StartingRebalance` or something that makes > it clear that it is part of the rebalance instead of a step before the > actual rebalance. `AwaitingSync` could also be `CompletingRebalance`, but > not sure if that's better. > > Ismael > > On Mon, Jul 24, 2017 at 7:02 PM, Guozhang

Re: [VOTE] KIP-164 Add unavailablePartitionCount and per-partition Unavailable metrics

2017-07-26 Thread Guozhang Wang
Hello, I would like to call out someone (committer) to voluntarily shepherd this KIP and drive it to be merged for 1.0.0. Please feel free to add your name on KIP-164 on the release wiki page: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913 Guozhang On Wed, Jul 26, 20

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-28 Thread Guozhang Wang
in > > > On Wed, Jul 26, 2017, at 11:25, Guozhang Wang wrote: > > To me `PreparingRebalance` sounds better than `StartingRebalance` since > > only by the end of that stage we have formed a new group. More > > specifically, this this the workflow from the coordinator&

Re: Contribution to Apache Kafka

2017-07-31 Thread Guozhang Wang
Hi Chanchal, I have added you to the contributor list and assigned you to KAFKA-5676. Guozhang On Mon, Jul 31, 2017 at 4:19 AM, Chanchal Singh wrote: > Dear Team, > > I am Chanchal Singh currently working as an Senior Software Engineer-Big > Data with AdPushup Inc. I want to start contributi

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-08-03 Thread Guozhang Wang
be named in those terms? For example: RebalanceJoin and > RebalanceAssignment. What do you think? > > -Jason > > On Fri, Jul 28, 2017 at 11:18 AM, Guozhang Wang > wrote: > > > I feel we can change `AwaitSync` to `completeRebalance` while keeping the > > other as

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-08-04 Thread Guozhang Wang
who have to dip into the code, figure > out a problem, and ultimately explain it. Since we already have the > PreparingRebalance state, maybe we could just rename the AwaitingSync state > to CompletingRebalance? > > -Jason > > On Thu, Aug 3, 2017 at 6:09 PM, Guozhang Wang

Re: [VOTE] KIP-177: Consumer perf tool should count rebalance time

2017-08-04 Thread Guozhang Wang
+1 On Fri, Aug 4, 2017 at 11:02 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding) > > Ismael > > On Fri, Aug 4, 2017 at 3:13 AM, Hu Xi wrote: > > > Hi all, > > > > > > I'd like to kick off the vote for KIP-177: https://cwiki.apache.org/ > > confluence/display/ARIES/KIP-177%3A+Consumer+p

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

2017-08-07 Thread Guozhang Wang
Thanks for the KIP proposal, I thought one suggestion before this discussion is to deprecate the " log.cleaner.enable" and always turn on compaction for those topics that have compact policies? Guozhang On Sat, Aug 5, 2017 at 9:36 AM, Pranav Maniar wrote: > Hi All, > > Following a discussion

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-08-08 Thread Guozhang Wang
Damian, Thanks for the proposal, I had a few comments on the APIs: 1. Printed#withFile seems not needed, as users should always spec if it is to sysOut or to File at the beginning. In addition as a second thought, I think serdes are not useful for prints anyways since we assume `toString` is prov

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-08-09 Thread Guozhang Wang
with(final Serde valueSerde) as two function signatures. Guozhang On Wed, Aug 9, 2017 at 6:20 AM, Damian Guy wrote: > On Tue, 8 Aug 2017 at 20:11 Guozhang Wang wrote: > > > Damian, > > > > Thanks for the proposal, I had a few comments on the APIs: > > > >

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-08-09 Thread Guozhang Wang
Text or not? Guozhang On Wed, Aug 9, 2017 at 11:38 AM, Guozhang Wang wrote: > >> The key idea is that by using the same function name string for static > >> constructor and member functions, users do not need to remember what > are > >> the differences but can call t

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

2017-08-10 Thread Guozhang Wang
+1 from me On Wed, Aug 9, 2017 at 9:40 AM, Jason Gustafson wrote: > +1 on the bump to 7 days. Wanted to mention one minor point. The > OffsetCommit RPC still provides the ability to set the retention time from > the client, but we do not use it in the consumer. Should we consider adding > a cons

Re: [VOTE] KIP-180: Adding a new metric for brokers specifying the number of consumer group rebalances in progress

2017-08-13 Thread Guozhang Wang
+1. Thanks. On Sat, Aug 12, 2017 at 11:18 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding). Seems like the KIP title has to be > updated still. > > Ismael > > On Fri, Aug 11, 2017 at 11:00 PM, Colin McCabe wrote: > > > Hi all, > > > > I think it's a good time to vote on KIP-180. It ad

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-13 Thread Guozhang Wang
Hi all, Here are my two cents: 1. I'm inclined to agree with Ewen that "acks" config is semi-orthogonal to the exactly-once semantics. For example, if some topics only have replication factor 1, then `acks = all` does not improve on any guarantees. More generally speaking, setting idempotency + r

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-14 Thread Guozhang Wang
Just want to clarify that regarding 1), I'm fine with changing it to `all` but just wanted to argue it is not necessarily correlate with the exactly-once semantics, but rather on persistence v.s. availability trade-offs, so I'd like to discuss them separately. Regarding 2), one minor concern I had

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-15 Thread Guozhang Wang
g 15, 2017 at 1:15 PM, Jay Kreps wrote: > Hey Guozhang, > > I think the argument is that with acks=1 the message could be lost and > hence you aren't guaranteeing exactly once delivery. > > -Jay > > On Mon, Aug 14, 2017 at 1:36 PM, Guozhang Wang wrote: > > >

Re: [DISCUSS] 0.11.0.1 bug fix release

2017-08-16 Thread Guozhang Wang
Thanks Damian for driving the release! On Wed, Aug 16, 2017 at 5:40 AM, Ismael Juma wrote: > Thanks for driving the release Damian. Sounds good to me. > > Ismael > > On Wed, Aug 16, 2017 at 1:29 PM, Damian Guy wrote: > > > Hi, > > > > It seems like it must be time for 0.11.0.1 bug fix release!

Re: KAFKA-5726

2017-08-17 Thread Guozhang Wang
Seems you are already added to the contributor list. Feel free to assign to yourself. Guozhang On Thu, Aug 17, 2017 at 12:46 AM, Attila Kreiner wrote: > Hi All, > > I am new here, so I picked an easy issue and created a PR: > https://github.com/apache/kafka/pull/3669 > https://issues.apache.or

1.0.0 Release Update

2017-08-18 Thread Guozhang Wang
Hello folks, Just want to give you a quick heads up on 1.0.0. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913 * KIPs: we have a total of 19 KIPs accepted so far, and 10 of them have already merged to trunk. For those KIP contributors, just a kind reminder: please updat

Re: Confluence access rights

2017-08-22 Thread Guozhang Wang
Attila, I have added you to the wiki page list. Cheers. Guozhang On Tue, Aug 22, 2017 at 4:29 AM, Attila Kreiner wrote: > Hey All, > > I need to create a KIP for this: > https://issues.apache.org/jira/browse/KAFKA-5726 > https://github.com/apache/kafka/pull/3669 > > However, I can't edit the

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-08-23 Thread Guozhang Wang
gt;>> Yep > > >>> > > >>>> > > >>>> > > >>>> -Matthias > > >>>> > > >>>> > > >>>> > > >>>> On 8/22/17 6:55 AM, Damian Guy wrote: > > >>>>&

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

2017-08-23 Thread Guozhang Wang
Congrats Jiangjie! Guozhang On Wed, Aug 23, 2017 at 10:20 PM, Joel Koshy wrote: > Hi everyone, > > Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has > contributed significantly to several major patches, reviews and discussions > since. I am glad to announce that Becket i

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-08-24 Thread Guozhang Wang
+1. Thanks Damian! On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck wrote: > Thanks for the KIP! > > +1 > > Thanks, > Bill > > On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy wrote: > > > Hi, > > > > I'd like to kick off the voting thread for KIP-182: > > https://cwiki.apache.org/confluence/display/KA

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-08-24 Thread Guozhang Wang
t; I've updated the kip to reflect Bill's comment and also to make > > StreamBuilder methods have topic as the first param, i.e., > > StreamBuilder#stream no longer accepts varargs. > > > > On Thu, 24 Aug 2017 at 09:12 Damian Guy wrote: > > > >> On Thu, 24 Aug

Re: [DISCUSS] KIP-171: Extend Consumer Group Reset Offset for Stream Application

2017-08-29 Thread Guozhang Wang
Hi Jorge, Thanks for the KIP. It would be a great to add feature to the reset tools. I made a pass over it and it looks good to me overall. I have a few comments: 1. For all the scenarios, do we allow users to specify more than one parameters? If not could you make that clear in the wiki, e.g. we

Re: [DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-08-29 Thread Guozhang Wang
Sounds good. On Tue, Aug 29, 2017 at 6:12 PM, Becket Qin wrote: > Sounds good to me as well. > > On Tue, Aug 29, 2017 at 2:43 AM, Ismael Juma wrote: > > > Sounds good to me too. Since this is a non controversial change, I > suggest > > starting the vote in 1-2 days if no-one else comments. > >

Re: [VOTE] KIP-138: Change punctuate semantics

2017-08-30 Thread Guozhang Wang
at, May 13, 2017 at 8:13 AM, Michal Borowiecki < michal.borowie...@openbet.com> wrote: > Thank you all! > > This KIP passed the vote with 3 binding and 5 non-binding +1s: > > +1 (binding) from Guozhang Wang, Ismael Juma and Ewen Cheslack-Postava > > +1 (non-binding) f

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-08-31 Thread Guozhang Wang
+1 On Thu, Aug 31, 2017 at 7:57 AM, Bill Bejeck wrote: > +1 > > Thanks, > Bill > > On Thu, Aug 31, 2017 at 10:31 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > +1 (non-binding) > > > > Thanks. > > --Vahid > > > > > > > > > > From: Molnár Bálint > > To: dev@kafka.apache.

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-09-05 Thread Guozhang Wang
t;>>> boolean loggingEnabled(); > >>>> > >>>> These are needed when we are building the topology and determining > >>>> changelog topic names and configs. > >>>> > >>>> > >>>> I've also upda

Re: [VOTE] KIP-138: Change punctuate semantics

2017-09-05 Thread Guozhang Wang
t; >> +1 >> >> On 8/30/17 12:00 PM, Bill Bejeck wrote: >> >>> +1 >>> >>> On Wed, Aug 30, 2017 at 1:06 PM, Damian Guy >>> wrote: >>> >>> +1 >>>> >>>> On Wed, 30 Aug 2017 at 17:49 Guozhang Wang

Re: Permissions to create a KIP?

2017-09-05 Thread Guozhang Wang
Hi Matthias, I have granted you the permission. Cheers. Guozhang On Tue, Sep 5, 2017 at 2:21 PM, Matthias Margush < matthias.marg...@fundingcircle.com> wrote: > Hi, > > Can I be set up with access to author a KIP ( > https://cwiki.apache.org/confluence/display/KAFKA/ > Kafka+Improvement+Propo

Re: [VOTE] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-09-06 Thread Guozhang Wang
Sorry for the delay, just made a pass over the wiki page. +1 On Mon, Aug 14, 2017 at 12:02 AM, Manikumar wrote: > +1 (non-binding) > > On Fri, Aug 11, 2017 at 8:09 PM, Mickael Maison > wrote: > > > +1 non-binding, thanks Vahid > > > > On Wed, Aug 9, 2017 at 9:31 PM, Jason Gustafson > > wrote:

1.0.0 KIPs Update

2017-09-06 Thread Guozhang Wang
Hello folks, This is a heads up on 1.0.0 progress: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913 We have one week left towards the KIP deadline, which is Sept. 13th. There are still a lot of KIPs that under discussion / voting process. For the KIP proposer, please kee

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-09-06 Thread Guozhang Wang
tthias J. Sax : > >> > > >> > > +1 > >> > > > >> > > On 8/31/17 8:49 AM, Jason Gustafson wrote: > >> > > > +1 > >> > > > > >> > > > On Thu, Aug 31, 2017 at 8:36 AM, Mickael Maison < > >&g

Re: 1.0.0 KIPs Update

2017-09-06 Thread Guozhang Wang
s ready for review. > > Thanks. > --Vahid > > > > From: Guozhang Wang > To: "dev@kafka.apache.org" > Date: 09/06/2017 01:45 PM > Subject:1.0.0 KIPs Update > > > > Hello folks, > > This is a heads up on 1.0.

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-09-06 Thread Guozhang Wang
Thanks for confirming. On Wed, Sep 6, 2017 at 2:07 PM, Ismael Juma wrote: > The PR for this was merged an hour or two ago. > > Ismael > > On 6 Sep 2017 9:51 pm, "Guozhang Wang" wrote: > > > Thanks Attila, > > > > Please note that the feature freez

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

2017-09-06 Thread Guozhang Wang
+1. Thanks. On Wed, Sep 6, 2017 at 7:57 AM, Ismael Juma wrote: > Thanks for the KIP. +1 (binding). Please make it clear in the KIP that > removal will happen in 2.0.0. > > Ismael > > On Tue, Aug 8, 2017 at 11:53 AM, Paolo Patierno > wrote: > > > Hi devs, > > > > > > I didn't see any more commen

Re: 1.0.0 KIPs Update

2017-09-07 Thread Guozhang Wang
Hi Tom, It seems KIP-183 is still in the discussion phase, and voting has not been started? Guozhang On Thu, Sep 7, 2017 at 1:13 AM, Tom Bentley wrote: > Would it be possible to add KIP-183 to the list too, please? > > Thanks, > > Tom > > On 6 September 2017 at 22:04,

Re: 1.0.0 KIPs Update

2017-09-07 Thread Guozhang Wang
Actually my bad, there is already a voting thread and you asked people to recast a vote on a small change. On Thu, Sep 7, 2017 at 2:27 PM, Guozhang Wang wrote: > Hi Tom, > > It seems KIP-183 is still in the discussion phase, and voting has not been > started? > > > Guozhan

<    3   4   5   6   7   8   9   10   11   12   >