Re: Speeding up integration tests

[Damian Guy]

Just to throw my 2c in here. It would also be good if we could run the
tests that failed "last time" first. That way at least if they are broken
then we'd fail fast rather than waiting for ever. Not much sucks more than
waiting for 2 hours to find out a test failed when it fails regularly and
you could have known within minutes...

On Wed, 27 Mar 2019 at 15:17, Viktor Somogyi-Vass 
wrote:

> Hi All,
>
> I've created a PR for what we have internally for retrying flaky tests. Any
> reviews and ideas are welcome: https://github.com/apache/kafka/pull/6506
> It's basically collects the failed classes and reruns them at the end. If
> they successful it overwrites the test report.
>
> Thanks,
> Viktor
>
> On Mon, Mar 11, 2019 at 1:05 PM Stanislav Kozlovski <
> stanis...@confluent.io>
> wrote:
>
> > I agree with Ron.
> > I think improving the framework with a configurable number of retries on
> > some tests will yield the highest ROI in terms of passing builds.
> >
> > On Fri, Mar 8, 2019 at 10:48 PM Ron Dagostino  wrote:
> >
> > > It's a classic problem: you can't string N things together serially and
> > > expect high reliability.  5,000 tests in a row isn't going to give you
> a
> > > bunch of 9's.  It feels to me that the test frameworks themselves
> should
> > > support a more robust model -- like a way to tag a test as "retry me up
> > to
> > > N times before you really consider me a failure" or something like
> that.
> > >
> > > Ron
> > >
> > > On Fri, Mar 8, 2019 at 11:40 AM Stanislav Kozlovski <
> > > stanis...@confluent.io>
> > > wrote:
> > >
> > > > > We internally have an improvement for a half a year now which
> reruns
> > > the
> > > > flaky test classes at the end of the test gradle task, lets you know
> > that
> > > > they were rerun and probably flaky. It fails the build only if the
> > second
> > > > run of the test class was also unsuccessful. I think it works pretty
> > > good,
> > > > we mostly have green builds. If there is interest, I can try to
> > > contribute
> > > > that.
> > > >
> > > > That does sound very intriguing. Does it rerun the test classes that
> > > failed
> > > > or some known, marked classes? If it is the former, I can see a lot
> of
> > > > value in having that automated in our PR builds. I wonder what others
> > > think
> > > > of this
> > > >
> > > > On Thu, Feb 28, 2019 at 6:04 PM Viktor Somogyi-Vass <
> > > > viktorsomo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey All,
> > > > >
> > > > > Thanks for the loads of ideas.
> > > > >
> > > > > @Stanislav, @Sonke
> > > > > I probably left it out from my email but I really imagined this as
> a
> > > > > case-by-case basis change. If we think that it wouldn't cause
> > problems,
> > > > > then it might be applied. That way we'd limit the blast radius
> > > somewhat.
> > > > > The 1 hour gain is really just the most optimistic scenario, I'm
> > almost
> > > > > sure that not every test could be transformed to use a common
> > cluster.
> > > > > We internally have an improvement for a half a year now which
> reruns
> > > the
> > > > > flaky test classes at the end of the test gradle task, lets you
> know
> > > that
> > > > > they were rerun and probably flaky. It fails the build only if the
> > > second
> > > > > run of the test class was also unsuccessful. I think it works
> pretty
> > > > good,
> > > > > we mostly have green builds. If there is interest, I can try to
> > > > contribute
> > > > > that.
> > > > >
> > > > > >I am also extremely annoyed at times by the amount of coffee I
> have
> > to
> > > > > drink before tests finish
> > > > > Just please don't get a heart attack :)
> > > > >
> > > > > @Ron, @Colin
> > > > > You bring up a very good point that it is easier and frees up more
> > > > > resources if we just run change specific tests and it's good to
> know
> > > > that a
> > > > > similar solution (meaning using a shared resource for testing) have
> > > > failed
> > > > > elsewhere. I second Ron on the test categorization though, although
> > as
> > > a
> > > > > first attempt I think using a flaky retry + running only the
> > necessary
> > > > > tests would help in both time saving and effectiveness. Also it
> would
> > > be
> > > > > easier to achieve.
> > > > >
> > > > > @Ismael
> > > > > Yea, it'd be interesting to profile the startup/shutdown, I've
> never
> > > done
> > > > > that. Perhaps I'll set some time apart for that :). It's definitely
> > > true
> > > > > though that if we see a significant delay there we wouldn't just
> > > improve
> > > > > the efficiency of the tests but also customer experience.
> > > > >
> > > > > Best,
> > > > > Viktor
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 28, 2019 at 8:12 AM Ismael Juma 
> > wrote:
> > > > >
> > > > > > It's an idea that has come up before and worth exploring
> > eventually.
> > > > > > However, I'd first try to optimize the server startup/shutdown
> > > process.
> > > > > If
> > > > > > we measure where the time is going, maybe some opport

Re: [VOTE] KIP-443: Return to default segment.ms and segment.index.bytes in Streams repartition topics

[Damian Guy]

+1

On Fri, 29 Mar 2019 at 01:59, John Roesler  wrote:

> +1 (nonbinding) from me.
>
> On Thu, Mar 28, 2019 at 7:08 PM Guozhang Wang  wrote:
>
> > Hello folks,
> >
> > I'd like to directly start a voting thread on this simple KIP to change
> the
> > default override values for repartition topics:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-443%3A+Return+to+default+segment.ms+and+segment.index.bytes+in+Streams+repartition+topics
> >
> > The related PR can be found here as well:
> > https://github.com/apache/kafka/pull/6511
> >
> > If you have any thoughts or feedbacks, they are more than welcomed as
> well.
> >
> >
> > -- Guozhang
> >
>

Re: [VOTE] KIP-422: Use the default value of max.poll.interval in Streams

[Damian Guy]

+1

On Wed, 27 Mar 2019 at 21:38, John Roesler  wrote:

> Ah, good point, Guozhang. I'll remove that mention from the KIP.
>
> On Wed, Mar 27, 2019 at 3:30 PM Bill Bejeck  wrote:
>
> > +1 for me,
> >
> > Thanks,
> > Bill
> >
> > On Wed, Mar 27, 2019 at 4:13 PM Guozhang Wang 
> wrote:
> >
> > > +1 from me.
> > >
> > > Though note that we cannot make such changes in older versions since
> even
> > > if we release new versions out of those branches they are considered
> > > bug-fix only and hence should not have any interface impacting changes.
> > >
> > >
> > > Guozhang
> > >
> > > On Wed, Mar 27, 2019 at 12:55 PM John Roesler 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Since the KIP is so small, I'm going to optimistically start the vote
> > for
> > > > KIP-422 to remove our "max int" default max.poll.interval.ms in
> > Streams
> > > > and
> > > > fall back to the Consumer default of five minutes.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-442%3A+Return+to+default+max+poll+interval+in+Streams
> > > >
> > > > Permalink: https://cwiki.apache.org/confluence/x/1COGBg
> > > >
> > > > See also: https://issues.apache.org/jira/browse/KAFKA-6399
> > > >
> > > > Please let me know if you have any objections and wish to return to
> the
> > > > discussion phase!
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [ANNOUNCE] New Committer: Bill Bejeck

[Damian Guy]

Congratulations Bill!

On Wed, 13 Feb 2019 at 16:51, Satish Duggana 
wrote:

> Congratulations Bill!
>
> On Thu, Feb 14, 2019 at 6:41 AM Marcelo Barbosa
>  wrote:
> >
> > Wow! Congrats Bill!
> > Cheers,
> > Barbosa
> > Em quarta-feira, 13 de fevereiro de 2019 23:03:54 BRST, Guozhang
> Wang  escreveu:
> >
> >  Hello all,
> >
> > The PMC of Apache Kafka is happy to announce that we've added Bill Bejeck
> > as our newest project committer.
> >
> > Bill has been active in the Kafka community since 2015. He has made
> > significant contributions to the Kafka Streams project with more than 100
> > PRs and 4 authored KIPs, including the streams topology optimization
> > framework. Bill's also very keen on tightening Kafka's unit test / system
> > tests coverage, which is a great value to our project codebase.
> >
> > In addition, Bill has been very active in evangelizing Kafka for stream
> > processing in the community. He has given several Kafka meetup talks in
> the
> > past year, including a presentation at Kafka Summit SF. He's also
> authored
> > a book about Kafka Streams (
> > https://www.manning.com/books/kafka-streams-in-action), as well as
> various
> > of posts in public venues like DZone as well as his personal blog (
> > http://codingjunkie.net/).
> >
> > We really appreciate the contributions and are looking forward to see
> more
> > from him. Congratulations, Bill !
> >
> >
> > Guozhang, on behalf of the Apache Kafka PMC
> >
>

Re: [VOTE] KIP-420: Add Single Value Fetch in Session Stores

[Damian Guy]

+1

On Wed, 23 Jan 2019 at 23:23, Guozhang Wang  wrote:

> Hello Matthias,
>
> Cool. I'd add it to the wiki page as well.
>
>
> Guozhang
>
> On Sat, Jan 19, 2019 at 10:59 AM Matthias J. Sax 
> wrote:
>
> > Thanks for the KIP Guozhang!
> >
> > Would it make sense to add a default implementation for the new method?
> > I am not sure, and I actually think it would not make sense:
> >
> >  - Kafka Streams provided stores will implement the method anyway
> >  - Kafka Streams relies on a proper implementation for custom stores
> > (because the new method in used during flush()).
> >
> > Thus, it seems that not adding a default implementation and hitting a
> > compilation error is better than hitting a runtime error later.
> >
> > However, I think it's worth to mention this on the KIP (ie, why not to
> > add a default implementation).
> >
> >
> > -Matthias
> >
> > On 1/18/19 10:33 PM, Guozhang Wang wrote:
> > > Hi Boyang,
> > >
> > > Thanks for the feedback!
> > >
> > > Although its direct result is a bug fix, it still changes the public
> > apis.
> > > And we cannot enlarge the scope of a vote / adopted KIP that has been
> > taken
> > > place in a previous release, so I think it is worthwhile with a new
> one.
> > >
> > >
> > > Guozhang
> > >
> > > On Fri, Jan 18, 2019 at 10:14 PM Boyang Chen 
> > wrote:
> > >
> > >> Hey Guozhang,
> > >>
> > >> this is nice catch! One question I have is that this seems more like a
> > bug
> > >> fix than a new feature proposal, maybe we could just update KIP-261
> > >> interface and resolve the JIRA to track the change?
> > >>
> > >> Boyang
> > >>
> > >> 
> > >> From: Guozhang Wang 
> > >> Sent: Saturday, January 19, 2019 1:07 PM
> > >> To: dev
> > >> Subject: [VOTE] KIP-420: Add Single Value Fetch in Session Stores
> > >>
> > >> Hello folks,
> > >>
> > >> I'd like to calling for a last-minute vote on the following KIP:
> > >>
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-420%3A+Add+Single+Value+Fetch+in+Session+Stores
> > >>
> > >> The idea comes from debugging a long lurking bug, but as an
> > afterthought I
> > >> think it should be included long time ago when we did KIP-261 [1].
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-261%3A+Add+Single+Value+Fetch+in+Window+Stores
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> >
> >
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-258: Allow to Store Record Timestamps in RocksDB

[Damian Guy]

+1

On Wed, 16 Jan 2019 at 05:09, Patrik Kleindl  wrote:

> +1 (non-binding)
> Thanks too
> Best regards
> Patrik
>
> > Am 16.01.2019 um 03:30 schrieb Bill Bejeck :
> >
> > Thanks for the KIP Matthias.
> >
> > +1
> >
> > -Bill
> >
> > On Tue, Jan 15, 2019 at 7:33 PM Matthias J. Sax 
> > wrote:
> >
> >> Hi,
> >>
> >> I would like to start the vote for KIP-258:
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-258%3A+Allow+to+Store+Record+Timestamps+in+RocksDB
> >>
> >> The KIP adds new stores that allow to store record timestamps next to
> >> key and value. Additionally, we will allow to upgrade exiting stores to
> >> the new stores; this will allow us to use the new stores in the DSL with
> >> a smooth upgrade path.
> >>
> >>
> >> -Matthias
> >>
> >>
>

Re: [VOTE] KIP-379: Multiple Consumer Group Management

[Damian Guy]

+1 (binding) thanks

On Tue, 15 Jan 2019 at 09:43, Alex D  wrote:

> Hello guys,
>
> 2 votes from Jason, Gwen
> Any binding votes?
>
>
> Hello M Manna,
> If migrating to JSON makes sense, you can post a new KIP for this purpose.
>
> Thanks,
> Alex Dunayevsky
>
> On Tue, 15 Jan 2019, 04:24 Jason Gustafson 
> > +1 Thanks for the KIP!
> >
> > -Jason
> >
> > On Mon, Jan 14, 2019 at 5:15 PM Gwen Shapira  wrote:
> >
> > > I was also wondering about that. I think its for compatibility with
> > > the existing output.
> > >
> > > We can have a separate KIP to add JSON output.
> > >
> > > On Mon, Jan 14, 2019 at 7:55 AM M. Manna  wrote:
> > > >
> > > > Hi,
> > > >
> > > > Thanks for this KIP. Could you kindly clarify why CSV format is
> useful,
> > > but
> > > > not anything else?
> > > >
> > > > CSV format is ancient and the only reason it keep existing is various
> > > > application is because legacy applications aren't moving away from
> > using
> > > > them. Would you would consider JSON or YAML?
> > > >
> > > > Also, if you think about the kafka-reassign-partitions - it's also
> > using
> > > > JSON, not CSV. That is my only point. However, if majority feels that
> > > it's
> > > > not
> > > > an issue I believe it's a team decision after all :).
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > On Mon, 14 Jan 2019 at 15:06, Gwen Shapira 
> wrote:
> > > >
> > > > > +1. Thanks, that will be very helpful.
> > > > >
> > > > > On Mon, Jan 14, 2019, 4:43 AM Alex D  > wrote:
> > > > >
> > > > > > Hello guys,
> > > > > >
> > > > > > We need your VOTES for the KIP-379: Multiple Consumer Group
> > > Management.
> > > > > >
> > > > > > KIP-379:
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%3A+Multiple+Consumer+Group+Management
> > > > > >
> > > > > > PR: https://github.com/apache/kafka/pull/5726/
> > > > > >
> > > > > > Let's start the voting session.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Alex Dunayevsky
> > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
>

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

[Damian Guy]

+1 binding

On Thu, 10 Jan 2019 at 16:57, Bill Bejeck  wrote:

> +1 from me.  Great job on the KIP.
>
> -Bill
>
> On Thu, Jan 10, 2019 at 11:35 AM John Roesler  wrote:
>
> > It's a +1 (nonbinding) from me as well.
> >
> > Thanks for sticking with this, Adam!
> > -John
> >
> > On Wed, Jan 9, 2019 at 6:22 PM Guozhang Wang  wrote:
> >
> > > Hello Adam,
> > >
> > > I'm +1 on the current proposal, thanks!
> > >
> > >
> > > Guozhang
> > >
> > > On Mon, Jan 7, 2019 at 6:13 AM Adam Bellemare <
> adam.bellem...@gmail.com>
> > > wrote:
> > >
> > > > Hi All
> > > >
> > > > I would like to call a new vote on KIP-213. The design has changed
> > > > substantially. Perhaps more importantly, the KIP and associated
> > > > documentation has been greatly simplified. I know this KIP has been
> on
> > > the
> > > > mailing list for a long time, but the help from John Roesler and
> > Guozhang
> > > > Wang have helped put it into a much better state. I would appreciate
> > any
> > > > feedback or votes.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable
> > > >
> > > >
> > > >
> > > > Thank you
> > > >
> > > > Adam Bellemare
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [VOTE] KIP-414: Expose Embedded ClientIds in Kafka Streams

[Damian Guy]

+1

On Fri, 11 Jan 2019 at 05:09, John Roesler  wrote:

> Hi Guozhang,
>
> It sounds reasonable to me. I'm +1 (nonbinding).
>
> -John
>
> On Tue, Jan 8, 2019 at 8:51 PM Guozhang Wang  wrote:
>
> > Hello folks,
> >
> > I'd like to start a voting process for the following KIP:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-414%3A+Expose+Embedded+ClientIds+in+Kafka+Streams
> >
> > It is a pretty straight-forward and small augment to Stream's public
> > ThreadMetadata interface, as an outcome of the discussion on KIP-345.
> Hence
> > I think we can skip the DISCUSS thread and move on to voting directly. If
> > people have any questions about the context of KIP-414 or KIP-345, please
> > feel free to read these two wiki pages and let me know.
> >
> >
> > -- Guozhang
> >
>

Re: [VOTE] [REMINDER] KIP-383 Pluggable interface for SSL Factory

[Damian Guy]

must be my gmail playing up. This appears to be the DISCUSS thread to me...

On Thu, 20 Dec 2018 at 18:42, Harsha  wrote:

> Damian,
>This is the VOTE thread. There is a DISCUSS thread which
> concluded in it.
>
> -Harsha
>
>
> On Wed, Dec 19, 2018, at 5:04 AM, Pellerin, Clement wrote:
> > I did that and nobody came.
> > https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:kip-383
> > I don't understand why this feature is not more popular.
> > It's the solution to one Jira and a work-around for a handful more Jiras.
> >
> > -Original Message-
> > From: Damian Guy [mailto:damian@gmail.com]
> > Sent: Wednesday, December 19, 2018 7:38 AM
> > To: dev
> > Subject: Re: [VOTE] [REMINDER] KIP-383 Pluggable interface for SSL
> Factory
> >
> > Hi Clement,
> >
> > You should start a separate thread for the vote, i.e., one with a subject
> > of [VOTE] KIP-383 ...
> >
> > Looks like you haven't done this?
>

Re: [VOTE] [REMINDER] KIP-383 Pluggable interface for SSL Factory

[Damian Guy]

Hi Clement,

You should start a separate thread for the vote, i.e., one with a subject
of [VOTE] KIP-383 ...

Looks like you haven't done this?

On Tue, 18 Dec 2018 at 17:33, Harsha  wrote:

> Yes. +1 binding.
>
> Thanks,
> Harsha
>
> On Mon, Dec 17, 2018, at 5:21 AM, Pellerin, Clement wrote:
> > I'm new here. Is this vote binding or not?
> >
> > -Original Message-
> > From: Harsha [mailto:ka...@harsha.io]
> > Sent: Saturday, December 15, 2018 1:59 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] [REMINDER] KIP-383 Pluggable interface for SSL
> Factory
> >
> > Overall LGTM. +1.
> >
> > Thanks,
> > Harsha
>

Re: Vote for KIP-393 (Fix time windowed serde to deserialize changelog topic)

[Damian Guy]

+1 (binding)

On Wed, 12 Dec 2018 at 13:27, Adam Bellemare 
wrote:

> +1 (non-binding) from me. Looks like a pretty clear-cut case.
>
>
>
> On Tue, Dec 11, 2018 at 1:11 AM Shawn Nguyen 
> wrote:
>
> > Thanks for the feedback Guozhang! I updated the KIP.
> >
> > In the meantime, could I ask for additional binding votes/approval on
> this
> > KIP proposal?
> >
> > Shawn
> >
> > On Thu, Dec 6, 2018 at 1:22 PM Liquan Pei  wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Wed, Dec 5, 2018 at 4:51 PM Guozhang Wang 
> wrote:
> > >
> > >> Hello Shawn,
> > >>
> > >> Thanks for the writeup. I've made a pass over it and here are some
> minor
> > >> comments:
> > >>
> > >> 1) As we discussed in the PR:
> https://github.com/apache/kafka/pull/5307
> > ,
> > >> the public APIs that we will add is
> > >>
> > >> In WindowedSerdes:
> > >> ```
> > >> static public  Serde>
> > timeWindowedChangelogSerdeFrom(final
> > >> Class type, final long windowSize)
> > >> ```
> > >>
> > >> In TimeWindowedSerde
> > >> ```
> > >> TimeWindowedSerde forChangelog(final boolean);
> > >> ```
> > >>
> > >> Other classes such as WindowedKeySchema are internal classes for
> > >> implementation details and hence do not need to be listed in the wiki
> as
> > >> public APIs.
> > >>
> > >>
> > >> 2) The wiki doc may reads a bit confusing for audience who are not
> > >> familiar
> > >> with the PR, since we mentioned the "forChangelog()" function and the
> > >> "isChangelog" parameter without clear definitions, but only explained
> > what
> > >> it is later in the docs as java code examples. I think rephrasing the
> > >> early
> > >> paragraphs to explain a bit more why we will add a new internal field
> > >> along
> > >> with a setter, its semantics (its default value and how
> deserialization
> > >> will be different depending on that) would be better.
> > >>
> > >> Otherwise, I'm +1 on the KIP, thanks!
> > >>
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Wed, Dec 5, 2018 at 8:18 AM Shawn Nguyen 
> > >> wrote:
> > >>
> > >> > Hey all,
> > >> >
> > >> > I wanted to start a vote on approval of KIP-393
> > >> > <
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-393%3A+Time+windowed+serde+to+properly+deserialize+changelog+input+topic
> > >> > >
> > >> > to
> > >> > fix the current time windowed serde for properly deserializing
> > changelog
> > >> > input topics. Let me know what you guys think.
> > >> >
> > >> > Thanks,
> > >> > Shawn
> > >> >
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> > > --
> > > Liquan Pei
> > > Software Engineer, Confluent Inc
> > >
> >
>

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Damian Guy]

Thanks +1 binding

On Sun, 7 Oct 2018 at 03:16, Chia-Ping Tsai  wrote:

> (just bump)
>
> We have following votes:
>
> non-binding
> 1. John Roesler
> 2. vito jeng
> 3. Richard Yu
> 4.  Bill Bejeck
> 5. Satish Duggana
>
> binding
> 1. Matthias J. Sax
> 2. Ismael Juma
>
> it needs one more ticket :)
>
> Cheers,
> Chia-Ping
>
> On 2018/07/05 14:45:01, Chia-Ping Tsai  wrote:
> > hi all,
> >
> > I would like to start voting on "KIP-331 Add default implementation to
> close() and configure() for Serializer, Deserializer and Serde"
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> >
> > Cheers,
> > Chia-Ping
> >
>

Re: [DISCUSS] KIP-378: Enable Dependency Injection for Kafka Streams handlers

[Damian Guy]

Hi Wladimir,

Of the two approaches in the KIP - i feel the second approach is cleaner.
However, am i correct in assuming that you want to have the
`ConfiguredStreamsFactory` as a ctor arg in `StreamsConfig` so that Spring
can inject this for you?

Otherwise you could just put the ApplicationContext as a property in the
config and then use that via the configure method of the appropriate
handler to get your actual handler.

Thanks,
Damian

On Tue, 9 Oct 2018 at 01:55, Guozhang Wang  wrote:

> John, thanks for the explanation, now it makes much more sense to me.
>
> As for the concrete approach, to me it seems the first option requires less
> changes than the second (ConfiguredStreamsFactory based) approach, whereas
> the second one requires an additional interface that is overlapping with
> the AbstractConfig.
>
> I'm aware that in KafkaProducer / KafkaConsumer we do not have public
> constructors for taking a ProducerConfig or ConsumerConfig directly, and
> anyone using Spring can share how you've worked around it by far? If it is
> very awkward I'm not against just adding the XXXConfigs to the constructors
> directly.
>
> Guozhang
>
> On Fri, Oct 5, 2018 at 1:48 PM, John Roesler  wrote:
>
> > Hi Wladimir,
> >
> > Thanks for the KIP!
> >
> > As I mentioned in the PR discussion, I personally prefer not to recommend
> > overriding StreamsConfig for this purpose.
> >
> > It seems like a person wishing to create a DI shim would have to acquire
> > quite a deep understanding of the class and its usage to figure out what
> > exactly to override to accomplish their goals without breaking
> everything.
> > I'm honestly impressed with the method you came up with to create your
> > Spring/Streams shim.
> >
> > I think we can make to path for the next person smoother by going with
> > something more akin to the ConfiguredStreamsFactory. This is a
> constrained
> > interface that tells you exactly what you have to implement to create
> such
> > a shim.
> >
> > A few thoughts:
> > 1. it seems like we can keep all the deprecated constructors still
> > deprecated
> >
> > 2. we could add just one additional constructor to each of KafkaStreams
> and
> > TopologyTestDriver to still take a Properties, but also your new
> > ConfiguredStreamsFactory
> >
> > 3. I don't know if I'm sold on the name ConfiguredStreamsFactory, since
> it
> > does not produce configured streams. Instead, it produces configured
> > instances... How about ConfiguredInstanceFactory?
> >
> > 4. if I understand the usage correctly, it's actually a pretty small
> number
> > of classes that we actually make via getConfiguredInstance. Offhand, I
> can
> > think of the key/value Serdes, the deserialization exception handler, and
> > the production exception handler.
> > Perhaps, instead of maintaining a generic "class instantiator", we could
> > explore a factory interface that just has methods for creating exactly
> the
> > kinds of things we need to create. In fact, we already have something
> like
> > this: org.apache.kafka.streams.KafkaClientSupplier . Do you think we
> could
> > just add some more methods to that interface (and maybe rename it)
> instead?
> >
> > Thanks,
> > -John
> >
> > On Fri, Oct 5, 2018 at 3:31 PM John Roesler  wrote:
> >
> > > Hi Guozhang,
> > >
> > > I'm going to drop in a little extra context from the preliminary PR
> > > discussion (https://github.com/apache/kafka/pull/5344).
> > >
> > > The issue isn't that it's impossible to use Streams within a Spring
> app,
> > > just that the interplay between our style of construction/configuration
> > and
> > > Spring's is somewhat awkward compared to the normal experience with
> > > dependency injection.
> > >
> > > I'm guessing users of dependency injection would not like the approach
> > you
> > > offered. I believe it's commonly considered an antipattern when using
> DI
> > > frameworks to pass the injector directly into the class being
> > constructed.
> > > Wladimir has also offered an alternative usage within the current
> > framework
> > > of injecting pre-constructed dependencies into the Properties, and then
> > > retrieving and casting them inside the configured class.
> > >
> > > It seems like this KIP is more about offering a more elegant interface
> to
> > > DI users.
> > >
> > > One of the points that Wladimir raised on his PR discussion was the
> > desire
> > > to configure the classes in a typesafe way in the constructor (thus
> > > allowing the use of immutable classes).
> > >
> > > With this KIP, it would be possible for a DI user to:
> > > 1. register a Streams-Spring or Streams-Guice (etc) "plugin" (via
> either
> > > of the mechanisms he proposed)
> > > 2. simply make the Serdes, exception handlers, etc, available on the
> > class
> > > path with the DI annotations
> > > 3. start the app
> > >
> > > There's no need to mess with passing dependencies (or the injector)
> > > through the properties.
> > >
> > > Sorry for "injecting" myself into your discussion, but it 

Re: [VOTE] KIP-302 - Enable Kafka clients to use all DNS resolved IP addresses

[Damian Guy]

Thanks  +1 (binding)

On Sun, 23 Sep 2018 at 23:43 Edoardo Comar  wrote:

> bumping the thread as the KIP needs 2 more binding votes ... pretty please
> ...
> --
>
> Edoardo Comar
>
> IBM Event Streams
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
>
> From:   "Skrzypek, Jonathan" 
> To: "dev@kafka.apache.org" 
> Date:   20/09/2018 17:08
> Subject:RE: [VOTE] KIP-302 - Enable Kafka clients to use all DNS
> resolved IP addresses
>
>
>
> Ok thanks.
>
> +1 (non-binding)
>
> The only thing I'm not too sure about is the naming around configuration
> entries for this, both for KIP-235 and KIP-302.
>
> KIP-235 expands DNS A records for bootstrap :
> resolve.canonical.bootstrap.servers.only
> KIP-302 expands DNS A records for advertised.listeners : use.all.dns.ips
>
> I'm a bit concerned that those don't easily explain what this does.
> Documentation helps obviously, but would we have suggestions for better
> naming ?
> I'm fine if we go for those but worth thinking about I think.
>
> Also, we probably want a third option to have both ? That's why we
> initially put in ".only" for KIP-235's parameter.
>
> Jonathan Skrzypek
>
>
> -Original Message-
> From: Edoardo Comar [mailto:eco...@uk.ibm.com]
> Sent: 20 September 2018 09:55
> To: dev@kafka.apache.org
> Subject: RE: [VOTE] KIP-302 - Enable Kafka clients to use all DNS resolved
> IP addresses
>
> Hi Jonathan
> we'll update the PR for KIP-302 soon. We do not need KIP-235 actually,
> they only share the name of the configuration entry.
>
> thanks
> Edo
>
> PS - we need votes :-)
>
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From:   "Skrzypek, Jonathan" 
> To: "dev@kafka.apache.org" 
> Date:   19/09/2018 16:12
> Subject:***UNCHECKED*** RE: [VOTE] KIP-302 - Enable Kafka clients
> to use all  DNS resolved IP addresses
>
>
>
> I'm assuming this needs KIP-235 to be merged.
> Unfortunately I've tripped over some merge issues with git and struggled
> to fix.
> Hopefully this is fixed but any help appreciated :
> https://github.com/apache/kafka/pull/4485
>
>
>
> Jonathan Skrzypek
>
>
>
> -Original Message-
> From: Eno Thereska [mailto:eno.there...@gmail.com]
> Sent: 19 September 2018 11:01
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-302 - Enable Kafka clients to use all DNS resolved
> IP addresses
>
> +1 (non-binding).
>
> Thanks
> Eno
>
> On Wed, Sep 19, 2018 at 10:09 AM, Rajini Sivaram 
> wrote:
>
> > Hi Edo,
> >
> > Thanks for the KIP!
> >
> > +1 (binding)
> >
> > On Tue, Sep 18, 2018 at 3:51 PM, Edoardo Comar 
> wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start the vote on KIP-302:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
>
> > > 302+-+Enable+Kafka+clients+to+use+all+DNS+resolved+IP+addresses
> > >
> > > We'd love to get this in 2.1.0
> > > Kip freeze is just a few days away ... please cast your votes  :-):-)
> > >
> > > Thanks!!
> > > Edo
> > >
> > > --
> > >
> > > Edoardo Comar
> > >
> > > IBM Message Hub
> > >
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> > 3AU
> > >
> >
>
> 
>
> Your Personal Data: We may collect and process information about you that
> may be subject to data protection laws. For more information about how we
> use and disclose your personal data, how we protect your information, our
> legal basis to use your information, your rights and who you can contact,
> please refer to:
> http://www.gs.com/privacy-notices
> <
> http://www.gs.com/privacy-notices
>
> >
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
> 
>
> Your Personal Data: We may collect and process information about you that
> may be subject to data protection laws. For more information about how we
> use and disclose your personal data, how we protect your information, our
> legal basis to use your information, your rights and who you can contact,
> please refer to: www.gs.com/privacy-notices<
> http://www.gs.com/privacy-notices
> >
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Re: [VOTE] KIP-372: Naming Repartition Topics for Joins and Grouping

[Damian Guy]

+1 (binding)

On Tue, 18 Sep 2018 at 16:33 Bill Bejeck  wrote:

> All,
>
> In starting work on the PR for KIP-372, the Grouped interface needed some
> method renaming to be more consistent with the other configuration classes
> (Joined, Produced, etc.).  As such I've updated the Grouped code section of
> the KIP.
>
> As these changes address a comment from Matthias on the initial draft of
> the KIP and don't change any of the existing behavior already outlined,  I
> don't think a re-vote is required.
>
> Thanks,
> Bill
>
> On Tue, Sep 18, 2018 at 10:09 AM John Roesler  wrote:
>
> > +1 (non-binding)
> >
> > Thanks!
> >
> > On Mon, Sep 17, 2018 at 7:29 PM Dongjin Lee  wrote:
> >
> > > Great improvements. +1. (Non-binding)
> > >
> > > On Tue, Sep 18, 2018 at 5:14 AM Matthias J. Sax  >
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > -Matthias
> > > >
> > > > On 9/17/18 1:12 PM, Guozhang Wang wrote:
> > > > > +1 from me, thanks Bill !
> > > > >
> > > > > On Mon, Sep 17, 2018 at 12:43 PM, Bill Bejeck 
> > > wrote:
> > > > >
> > > > >> All,
> > > > >>
> > > > >> I'd like to start the voting process for KIP-372.  Here's the link
> > to
> > > > the
> > > > >> updated proposal
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> 372%3A+Naming+Repartition+Topics+for+Joins+and+Grouping
> > > > >>
> > > > >> I'll start with my own +1.
> > > > >>
> > > > >> Thanks,
> > > > >> Bill
> > > > >>
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github:  github.com/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > slideshare:
> > > www.slideshare.net/dongjinleekr
> > > *
> > >
> >
>

Re: [DISCUSS] KIP-358: Migrate Streams API to Duration instead of long ms times

[Damian Guy]

Thanks +1 (binding)

On Sun, 16 Sep 2018 at 19:37 Nikolay Izhikov  wrote:

> Dear commiters,
>
> I got two binding +1 in [VOTE] thread for this KIP [1].
> I still need one more.
>
> Please, take a look at KIP.
>
> [1]
> https://lists.apache.org/thread.html/e976352e7e42d459091ee66ac790b6a0de7064eac0c57760d50c983b@%3Cdev.kafka.apache.org%3E
>
> В Чт, 13/09/2018 в 19:33 +0300, Nikolay Izhikov пишет:
> > Fixed.
> >
> > Thanks, for help!
> >
> > Please, take a look and vote.
> >
> > В Чт, 13/09/2018 в 08:40 -0700, Matthias J. Sax пишет:
> > > No need to start a new voting thread :)
> > >
> > > For the KIP update, I think it should be:
> > >
> > > > ReadOnlyWindowStore {
> > > > //Deprecated methods.
> > > > WindowStoreIterator fetch(K key, long timeFrom, long timeTo);
> > > > KeyValueIterator, V> fetch(K from, K to, long
> timeFrom, long timeTo);
> > > > KeyValueIterator, V> fetchAll(long timeFrom, long
> timeTo);
> > > >
> > > > //New methods.
> > > > WindowStoreIterator fetch(K key, Instant from, Duration
> duration) throws IllegalArgumentException;
> > > > KeyValueIterator, V> fetch(K from, K to, Instant
> from, Duration duration) throws IllegalArgumentException;
> > > > KeyValueIterator, V> fetchAll(Instant from, Duration
> duration) throws IllegalArgumentException;
> > > > }
> > > >
> > > >
> > > > WindowStore {
> > > > //New methods.
> > > > WindowStoreIterator fetch(K key, long timeFrom, long timeTo);
> > > > KeyValueIterator, V> fetch(K from, K to, long
> timeFrom, long timeTo);
> > > > KeyValueIterator, V> fetchAll(long timeFrom, long
> timeTo);
> > > > }
> > >
> > > Ie, long-versions are replaced with Instant/Duration in
> > > `ReadOnlyWindowStore`, and `long` method are added in `WindowStore` --
> > > this way, we effectively "move" the long-versions from
> > > `ReadOnlyWindowStore` to `WindowStore`.
> > >
> > > -Matthias
> > >
> > > On 9/13/18 8:08 AM, Nikolay Izhikov wrote:
> > > > Hello, Matthias.
> > > >
> > > > > I like the KIP as-is. Feel free to start a VOTE thread.
> > > >
> > > >
> > > > I'm already started one [1].
> > > > Can you vote in it or I should create a new one?
> > > >
> > > >
> > > > I've updated KIP.
> > > > This has been changed:
> > > >
> > > > ReadOnlyWindowStore {
> > > > //Deprecated methods.
> > > > WindowStoreIterator fetch(K key, long timeFrom, long timeTo);
> > > > KeyValueIterator, V> fetch(K from, K to, long
> timeFrom, long timeTo);
> > > > KeyValueIterator, V> fetchAll(long timeFrom, long
> timeTo);
> > > > }
> > > >
> > > > WindowStore {
> > > >   //New methods.
> > > > WindowStoreIterator fetch(K key, Instant from, Duration
> duration) throws IllegalArgumentException;
> > > > KeyValueIterator, V> fetch(K from, K to, Instant
> from, Duration duration) throws IllegalArgumentException;
> > > > KeyValueIterator, V> fetchAll(Instant from, Duration
> duration) throws IllegalArgumentException;
> > > > }
> > > >
> > > > [1]
> https://lists.apache.org/thread.html/e976352e7e42d459091ee66ac790b6a0de7064eac0c57760d50c983b@%3Cdev.kafka.apache.org%3E
> > > >
> > > > В Ср, 12/09/2018 в 15:46 -0700, Matthias J. Sax пишет:
> > > > > Great!
> > > > >
> > > > > I did not double check the ReadOnlySessionStore interface before,
> and
> > > > > just assumed it would take a timestamp, too. My bad.
> > > > >
> > > > > Please update the KIP for ReadOnlyWindowStore and WindowStore.
> > > > >
> > > > > I like the KIP as-is. Feel free to start a VOTE thread. Even if
> there
> > > > > might be minor follow up comments, we can vote in parallel.
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 9/12/18 1:06 PM, John Roesler wrote:
> > > > > > Hi Nikolay,
> > > > > >
> > > > > > Yes, the changes we discussed for ReadOnlyXxxStore and XxxStore
> should be
> > > > > > in this KIP.
> > > > > >
> > > > > > And you're right, it seems like ReadOnlySessionStore is not
> necessary to
> > > > > > touch, since it doesn't reference any `long` timestamps.
> > > > > >
> > > > > > Thanks,
> > > > > > -John
> > > > > >
> > > > > > On Wed, Sep 12, 2018 at 4:36 AM Nikolay Izhikov <
> nizhi...@apache.org> wrote:
> > > > > >
> > > > > > > Hello, Matthias.
> > > > > > >
> > > > > > > > His proposal is, to deprecate existing methods on
> `ReadOnlyWindowStore`>
> > > > > > >
> > > > > > > and `ReadOnlySessionStore` and add them to `WindowStore` and>
> `SessionStore`
> > > > > > > > Does this make sense?
> > > > > > >
> > > > > > > You both are experienced Kafka developers, so yes, it does
> make a sense to
> > > > > > > me :).
> > > > > > > Do we want to make this change in KIP-358 or it required
> another KIP?
> > > > > > >
> > > > > > > > Btw: the KIP misses `ReadOnlySessionStore` atm.
> > > > > > >
> > > > > > > Sorry, but I don't understand you.
> > > > > > > As far as I can see, there is only 2 methods in
> `ReadOnlySessionStore`.
> > > > > > > Which method should be migrated to Duration?
> > > > > > >
> > > > > > >
> > > > >

Re: [VOTE] KIP-366 - Make FunctionConversations private

[Damian Guy]

+1 (binding)

Thanks

On Fri, 14 Sep 2018 at 10:00 Joan Goyeau  wrote:

> Ok so we have only one binding vote up to now. I guess we need at least 3
> binding votes.
>
> Thanks
>
> On Fri, 14 Sep 2018 at 09:59 Joan Goyeau  wrote:
>
> > Matt, This is now all updated.
> >
> > On Fri, 7 Sep 2018 at 03:34 Matthias J. Sax 
> wrote:
> >
> >> Can you please update the KIP accordingly?
> >>
> >> It still says "make private" instead of "deprecating"
> >>
> >> -Matthias
> >>
> >> On 9/6/18 10:07 AM, Attila Sasvári wrote:
> >> > +1 (non-binding)
> >> >
> >> > On Thu, Sep 6, 2018 at 6:38 PM Guozhang Wang 
> >> wrote:
> >> >
> >> >> +1 for deprecating and copying the class over to internals.
> >> >>
> >> >> On Thu, Sep 6, 2018 at 6:56 AM, Bill Bejeck 
> wrote:
> >> >>
> >> >>> +1
> >> >>>
> >> >>> -Bill
> >> >>>
> >> >>> On Thu, Sep 6, 2018 at 4:29 AM Joan Goyeau  wrote:
> >> >>>
> >>  Sournds good, I'll make the deprecation and copy the class over.
> >> 
> >>  Thanks
> >> 
> >>  On Wed, 5 Sep 2018 at 22:48 John Roesler 
> wrote:
> >> 
> >> > I'm a +1 (non-binding) because we doubt the class is in use.
> >> >
> >> > If you decide to copy it to a private version and deprecate the
> >> >>> original
> >> > instead, as Matthias suggested, I would still be a +1.
> >> >
> >> > Thanks,
> >> > -John
> >> >
> >> > On Sat, Sep 1, 2018 at 6:47 AM Joan Goyeau 
> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> As pointed out in this comment
> >> >> https://github.com/apache/kafka/pull/5539#discussion_r212380648
> >> >>> "This
> >> >> class
> >> >> was already defaulted to public visibility, and we can't retract
> it
> >>  now,
> >> >> without a KIP.", the object FunctionConversions is only of
> internal
> >> >>> use
> >> > and
> >> >> therefore should be private to the lib only so that we can do
> >> >> changes
> >> >> without going through KIP like this one.
> >> >>
> >> >> KIP:
> >> >>
> >> >>
> >> >
> >>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-366%3A+Make+
> >> >>> FunctionConversions+private
> >> >>
> >> >> Please make your votes.
> >> >> Thanks
> >> >>
> >> >
> >> 
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> -- Guozhang
> >> >>
> >> >
> >>
> >>
>

Re: [VOTE] KIP-365: Materialized, Serialized, Joined, Consumed and Produced with implicit Serde

[Damian Guy]

+1

On Sun, 2 Sep 2018 at 15:20 Matthias J. Sax  wrote:

> +1 (binding)
>
> On 9/1/18 2:40 PM, Guozhang Wang wrote:
> > +1 (binding).
> >
> > On Mon, Aug 27, 2018 at 5:20 PM, Dongjin Lee  wrote:
> >
> >> +1 (non-binding)
> >>
> >> On Tue, Aug 28, 2018 at 8:53 AM Bill Bejeck  wrote:
> >>
> >>> +1
> >>>
> >>> -Bill
> >>>
> >>> On Mon, Aug 27, 2018 at 3:24 PM Ted Yu  wrote:
> >>>
>  +1
> 
>  On Mon, Aug 27, 2018 at 12:18 PM John Roesler 
> >> wrote:
> 
> > +1 (non-binding)
> >
> > On Sat, Aug 25, 2018 at 1:16 PM Joan Goyeau  wrote:
> >
> >> Hi,
> >>
> >> We want to make sure that we always have a serde for all
> >>> Materialized,
> >> Serialized, Joined, Consumed and Produced.
> >> For that we can make use of the implicit parameters in Scala.
> >>
> >> KIP:
> >>
> >>
> >
> 
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>
> 365%3A+Materialized%2C+Serialized%2C+Joined%2C+Consumed+and+Produced+with+
> >> implicit+Serde
> >>
> >> Github PR: https://github.com/apache/kafka/pull/5551
> >>
> >> Please make your votes.
> >> Thanks
> >>
> >
> 
> >>>
> >>> --
> >>> *Dongjin Lee*
> >>>
> >>> *A hitchhiker in the mathematical world.*
> >>>
> >>> *github:  github.com/dongjinleekr
> >>> linkedin: kr.linkedin.com/in/
> >> dongjinleekr
> >>> slideshare:
> www.slideshare.net/
> >> dongjinleekr
> >>> *
> >>>
> >>
> >
> >
> >
>
>

Re: [VOTE] KIP-353: Allow Users to Configure Multi-Streams Timestamp Synchronization Behavior

[Damian Guy]

+1

On Tue, 14 Aug 2018 at 19:35 Ted Yu  wrote:

> +1
>  Original message From: Bill Bejeck 
> Date: 8/14/18  11:09 AM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> Re: [VOTE] KIP-353: Allow Users to Configure Multi-Streams Timestamp
> Synchronization Behavior
> +1
>
> Thanks,
> Bill
>
> On Thu, Aug 9, 2018 at 4:20 PM John Roesler  wrote:
>
> > +1 non-binding
> >
> > On Thu, Aug 9, 2018 at 3:14 PM Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > On 8/9/18 11:57 AM, Guozhang Wang wrote:
> > > > Hello all,
> > > >
> > > > I would like to start the voting processing on the following KIP, to
> > > allow
> > > > users control when a task can be processed based on its buffered
> > records,
> > > > and how the stream time of a task be advanced.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 353%3A+Improve+Kafka+Streams+Timestamp+Synchronization
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > -- Guozhang
> > > >
> > >
> > >
> >
>

Re: [VOTE] KIP-356: Add withCachingDisabled() to StoreBuilder

[Damian Guy]

+1

On Tue, 14 Aug 2018 at 22:58 Matthias J. Sax  wrote:

> +1 (binding)
>
> On 8/14/18 11:16 AM, Eno Thereska wrote:
> > +1 (non binding)
> >
> > Thanks
> > Eno
> >
> > On Tue, Aug 14, 2018 at 10:53 AM, Bill Bejeck  wrote:
> >
> >> Thanks for the KIP.
> >>
> >> +1
> >>
> >> -Bill
> >>
> >> On Tue, Aug 14, 2018 at 1:42 PM Guozhang Wang 
> wrote:
> >>
> >>> Hello folks,
> >>>
> >>> I'd like to start a voting thread on the following KIP:
> >>>
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-356%3A+Add+
> >> withCachingDisabled%28%29+to+StoreBuilder
> >>>
> >>> It is a pretty straightforward one, adding a missing API to
> StoreBuilder
> >>> which should actually be added at the very beginning but somehow was
> >> lost.
> >>> Hence I skipped the DISCUSS process of it. But if you have any
> feedbacks
> >>> please feel free to share as well.
> >>>
> >>>
> >>>
> >>> -- Guozhang
> >>>
> >>
> >
>
>

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

[Damian Guy]

Thanks John! +1

On Mon, 30 Jul 2018 at 23:58 Guozhang Wang  wrote:

> Yes, the addendum lgtm as well. Thanks!
>
> On Mon, Jul 30, 2018 at 3:34 PM, John Roesler  wrote:
>
> > Another thing that came up after I started working on an implementation
> is
> > that in addition to deprecating "retention" from the Windows interface,
> we
> > also need to deprecate "segmentInterval", for the same reasons. I simply
> > overlooked it previously. I've updated the KIP accordingly.
> >
> > Hopefully, this doesn't change anyone's vote.
> >
> > Thanks,
> > -John
> >
> > On Mon, Jul 30, 2018 at 5:31 PM John Roesler  wrote:
> >
> > > Thanks Guozhang,
> > >
> > > Thanks for that catch. to clarify, currently, events are "late" only
> when
> > > they are older than the retention period. Currently, we detect this in
> > the
> > > processor and record it as a "skipped-record". We then do not attempt
> to
> > > store the event in the window store. If a user provided a
> pre-configured
> > > window store with a retention period smaller than the one they specify
> > via
> > > Windows#until, the segmented store will drop the update with no metric
> > and
> > > record a debug-level log.
> > >
> > > With KIP-328, with the introduction of "grace period" and moving
> > retention
> > > fully into the state store, we need to have metrics for both "late
> > events"
> > > (new records older than the grace period) and "expired window events"
> > (new
> > > records for windows that are no longer retained in the state store). I
> > > already proposed metrics for the late events, and I've just updated the
> > KIP
> > > with metrics for the expired window events. I also updated the KIP to
> > make
> > > it clear that neither late nor expired events will count as
> > > "skipped-records" any more.
> > >
> > > -John
> > >
> > > On Mon, Jul 30, 2018 at 4:22 PM Guozhang Wang 
> > wrote:
> > >
> > >> Hi John,
> > >>
> > >> Thanks for the updated KIP, +1 from me, and one minor suggestion:
> > >>
> > >> Following your suggestion of the differentiation of `skipped-records`
> > v.s.
> > >> `late-event-drop`, we should probably consider moving the scenarios
> > where
> > >> records got ignored due the window not being available any more in
> > >> windowed
> > >> aggregation operators from the `skipped-records` metrics recording to
> > the
> > >> `late-event-drop` metrics recording.
> > >>
> > >>
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Mon, Jul 30, 2018 at 1:36 PM, Bill Bejeck 
> wrote:
> > >>
> > >> > Thanks for the KIP!
> > >> >
> > >> > +1
> > >> >
> > >> > -Bill
> > >> >
> > >> > On Mon, Jul 30, 2018 at 3:42 PM Ted Yu  wrote:
> > >> >
> > >> > > +1
> > >> > >
> > >> > > On Mon, Jul 30, 2018 at 11:46 AM John Roesler 
> > >> wrote:
> > >> > >
> > >> > > > Hello devs,
> > >> > > >
> > >> > > > The discussion of KIP-328 has gone some time with no new
> comments,
> > >> so I
> > >> > > am
> > >> > > > calling for a vote!
> > >> > > >
> > >> > > > Here's the KIP: https://cwiki.apache.org/confluence/x/sQU0BQ
> > >> > > >
> > >> > > > The basic idea is to provide:
> > >> > > > * more usable control over update rate (vs the current state
> store
> > >> > > caches)
> > >> > > > * the final-result-for-windowed-computations feature which
> > several
> > >> > people
> > >> > > > have requested
> > >> > > >
> > >> > > > Thanks,
> > >> > > > -John
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Damian Guy]

You have 3 binding votes, so i'll defer to the others.

On Thu, 2 Aug 2018 at 04:41 Nishanth Pradeep  wrote:

> The only issue I see with this is that Sink#topic would also need to be
> Optional as was pointed out already. Since Sink#topic is a preexisting
> method, changing its return type would break backwards compatibility.
>
> On the other hand, it might be worth it to rip that bandaid now.
>
> Best,
> Nishanth Pradeep
>
> On Wed, Aug 1, 2018 at 11:56 AM Guozhang Wang  wrote:
>
>> For source node, only one of `Set topicsSet` and `TopicPattern
>> topicPattern()` will be specified by the user. Similarly for sink node,
>> only one of `String` and `TopicNameExtractor` will be specified by the
>> user. Although I've not seen Nishanth's updated PR, I think when it is not
>> specified today we will return null in that case.
>>
>> If we want to improve on this situation with Optional, we'd need to do it
>> on all of these functions. Also note that for `Source#toString()` and
>> `Sink#toString()` we should only include the specified field in the
>> resulted representation.
>>
>>
>> Guozhang
>>
>> On Wed, Aug 1, 2018 at 5:08 AM, Damian Guy  wrote:
>>
>> > Ewen - no as I don't believe they are never null. Whereas the
>> > topicNameExtractor method returns null if it is the default extractor or
>> > the extractor. So i think this would be better to be optional as it is
>> > optionally returning a TopicNameExtractor
>> >
>> > On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava 
>> > wrote:
>> >
>> > > Generally +1 (binding)
>> > >
>> > > It would be helpful to just provide the full, updated interfaces in
>> the
>> > > KIP and mark things as new with comments if needed. I had to go back
>> and
>> > > read the discussion thread to make sure I was understanding the intent
>> > > correctly.
>> > >
>> > > Damian -- if we make that Optional, shouldn't the methods on Source
>> also
>> > > be Optional types?
>> > >
>> > > -Ewen
>> > >
>> > > On Mon, Jul 30, 2018 at 11:13 PM Damian Guy 
>> > wrote:
>> > >
>> > >> Hi Nishanth,
>> > >>
>> > >> I have one nit on the KIP. I think the topicNameExtractor method
>> should
>> > >> return Optional rather than null.
>> > >> Sorry I'm late here.
>> > >>
>> > >> Thanks,
>> > >> Damian
>> > >>
>> > >> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep > >
>> > >> wrote:
>> > >>
>> > >> > We need one more binding vote.
>> > >> >
>> > >> > Binding Votes:
>> > >> >
>> > >> >- Matthias J. Sax
>> > >> >- Guozhang Wong
>> > >> >
>> > >> > Community Votes:
>> > >> >
>> > >> >- Bill Bejeck
>> > >> >- Ted Yu
>> > >> >
>> > >> > Best,
>> > >> > Nishanth Pradeep
>> > >> >
>> > >> > On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck 
>> > wrote:
>> > >> >
>> > >> > > Thanks for the KIP!
>> > >> > >
>> > >> > > +1
>> > >> > >
>> > >> > > -Bill
>> > >> > >
>> > >> > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang <
>> wangg...@gmail.com>
>> > >> > wrote:
>> > >> > >
>> > >> > > > +1
>> > >> > > >
>> > >> > > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
>> > >> > matth...@confluent.io
>> > >> > > >
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > +1 (binding)
>> > >> > > > >
>> > >> > > > > -Matthias
>> > >> > > > >
>> > >> > > > > On 7/25/18 7:47 PM, Ted Yu wrote:
>> > >> > > > > > +1
>> > >> > > > > >
>> > >> > > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
>> > >> > > > nishanth...@gmail.com>
>> > >> > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > >> Hello,
>> > >> > > > > >>
>> > >> > > > > >> I'm calling a vote for KIP-321:
>> > >> > > > > >>
>> > >> > > > > >>
>> > >> > > > > >>
>> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 321%3A+Update+
>> > >> > > > > TopologyDescription+to+better+represent+Source+and+Sink+Nodes
>> > >> > > > > >>
>> > >> > > > > >> Best,
>> > >> > > > > >> Nishanth Pradeep
>> > >> > > > > >>
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > --
>> > >> > > > -- Guozhang
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Damian Guy]

Ewen - no as I don't believe they are never null. Whereas the
topicNameExtractor method returns null if it is the default extractor or
the extractor. So i think this would be better to be optional as it is
optionally returning a TopicNameExtractor

On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava 
wrote:

> Generally +1 (binding)
>
> It would be helpful to just provide the full, updated interfaces in the
> KIP and mark things as new with comments if needed. I had to go back and
> read the discussion thread to make sure I was understanding the intent
> correctly.
>
> Damian -- if we make that Optional, shouldn't the methods on Source also
> be Optional types?
>
> -Ewen
>
> On Mon, Jul 30, 2018 at 11:13 PM Damian Guy  wrote:
>
>> Hi Nishanth,
>>
>> I have one nit on the KIP. I think the topicNameExtractor method should
>> return Optional rather than null.
>> Sorry I'm late here.
>>
>> Thanks,
>> Damian
>>
>> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep 
>> wrote:
>>
>> > We need one more binding vote.
>> >
>> > Binding Votes:
>> >
>> >- Matthias J. Sax
>> >- Guozhang Wong
>> >
>> > Community Votes:
>> >
>> >- Bill Bejeck
>> >- Ted Yu
>> >
>> > Best,
>> > Nishanth Pradeep
>> >
>> > On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck  wrote:
>> >
>> > > Thanks for the KIP!
>> > >
>> > > +1
>> > >
>> > > -Bill
>> > >
>> > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang 
>> > wrote:
>> > >
>> > > > +1
>> > > >
>> > > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
>> > matth...@confluent.io
>> > > >
>> > > > wrote:
>> > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > -Matthias
>> > > > >
>> > > > > On 7/25/18 7:47 PM, Ted Yu wrote:
>> > > > > > +1
>> > > > > >
>> > > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
>> > > > nishanth...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Hello,
>> > > > > >>
>> > > > > >> I'm calling a vote for KIP-321:
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-321%3A+Update+
>> > > > > TopologyDescription+to+better+represent+Source+and+Sink+Nodes
>> > > > > >>
>> > > > > >> Best,
>> > > > > >> Nishanth Pradeep
>> > > > > >>
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > -- Guozhang
>> > > >
>> > >
>> >
>>
>

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Damian Guy]

Hi Nishanth,

I have one nit on the KIP. I think the topicNameExtractor method should
return Optional rather than null.
Sorry I'm late here.

Thanks,
Damian

On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep  wrote:

> We need one more binding vote.
>
> Binding Votes:
>
>- Matthias J. Sax
>- Guozhang Wong
>
> Community Votes:
>
>- Bill Bejeck
>- Ted Yu
>
> Best,
> Nishanth Pradeep
>
> On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck  wrote:
>
> > Thanks for the KIP!
> >
> > +1
> >
> > -Bill
> >
> > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang 
> wrote:
> >
> > > +1
> > >
> > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > -Matthias
> > > >
> > > > On 7/25/18 7:47 PM, Ted Yu wrote:
> > > > > +1
> > > > >
> > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
> > > nishanth...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >> I'm calling a vote for KIP-321:
> > > > >>
> > > > >>
> > > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-321%3A+Update+
> > > > TopologyDescription+to+better+represent+Source+and+Sink+Nodes
> > > > >>
> > > > >> Best,
> > > > >> Nishanth Pradeep
> > > > >>
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

[Damian Guy]

; Apache Kafka is a distributed streaming platform with four core APIs:
>
>
>
> ** The Producer API allows an application to publish a stream records to
>
> one or more Kafka topics.
>
>
>
> ** The Consumer API allows an application to subscribe to one or more
>
> topics and process the stream of records produced to them.
>
>
>
> ** The Streams API allows an application to act as a stream processor,
>
> consuming an input stream from one or more topics and producing an
>
> output stream to one or more output topics, effectively transforming the
>
> input streams to output streams.
>
>
>
> ** The Connector API allows building and running reusable producers or
>
> consumers that connect Kafka topics to existing applications or data
>
> systems. For example, a connector to a relational database might
>
> capture every change to a table.
>
>
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
>
> ** Building real-time streaming data pipelines that reliably get data
>
> between systems or applications.
>
>
>
> ** Building real-time streaming applications that transform or react
>
> to the streams of data.
>
>
>
>
>
>
>
> Apache Kafka is in use at large and small companies worldwide, including
>
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
>
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
>
>
>
>
>
>
> A big thank you for the following 131 contributors to this release!
>
>
>
> Adem Efe Gencer, Alex D, Alex Dunayevsky, Allen Wang, Andras Beni,
>
> Andy Bryant, Andy Coates, Anna Povzner, Arjun Satish, asutosh936,
>
> Attila Sasvari, bartdevylder, Benedict Jin, Bill Bejeck, Blake Miller,
>
> Boyang Chen, cburroughs, Chia-Ping Tsai, Chris Egerton, Colin P. Mccabe,
>
> Colin Patrick McCabe, ConcurrencyPractitioner, Damian Guy, dan norwood,
>
> Daniel Shuy, Daniel Wojda, Dark, David Glasser, Debasish Ghosh, Detharon,
>
> Dhruvil Shah, Dmitry Minkovsky, Dong Lin, Edoardo Comar, emmanuel Harel,
>
> Eugene Sevastyanov, Ewen Cheslack-Postava, Fedor Bobin, fedosov-alexander,
>
> Filipe Agapito, Florian Hussonnois, fredfp, Gilles Degols, gitlw, Gitomain,
>
> Guangxian, Gunju Ko, Gunnar Morling, Guozhang Wang, hmcl, huxi, huxihx,
>
> Igor Kostiakov, Ismael Juma, Jacek Laskowski, Jagadesh Adireddi,
>
> Jarek Rudzinski, Jason Gustafson, Jeff Klukas, Jeremy Custenborder,
>
> Jiangjie (Becket) Qin, Jiangjie Qin, JieFang.He, Jimin Hsieh, Joan Goyeau,
>
> Joel Hamill, John Roesler, Jon Lee, Jorge Quilcate Otoya, Jun Rao,
>
> Kamal C, khairy, Koen De Groote, Konstantine Karantasis, Lee Dongjin,
>
> Liju John, Liquan Pei, lisa2lisa, Lucas Wang, Magesh Nandakumar,
>
> Magnus Edenhill, Magnus Reftel, Manikumar Reddy, Manikumar Reddy O,
>
> manjuapu, Mats Julian Olsen, Matthias J. Sax, Max Zheng, maytals,
>
> Michael Arndt, Michael G. Noll, Mickael Maison, nafshartous, Nick Travers,
>
> nixsticks, Paolo Patierno, parafiend, Patrik Erdes, Radai Rosenblatt,
>
> Rajini Sivaram, Randall Hauch, ro7m, Robert Yokota, Roman Khlebnov,
>
> Ron Dagostino, Sandor Murakozi, Sasaki Toru, Sean Glover,
>
> Sebastian Bauersfeld, Siva Santhalingam, Stanislav Kozlovski, Stephane
> Maarek,
>
> Stuart Perks, Surabhi Dixit, Sönke Liebau, taekyung, tedyu, Thomas Leplus,
>
> UVN, Vahid Hashemian, Valentino Proietti, Viktor Somogyi, Vitaly Pushkar,
>
> Wladimir Schmidt, wushujames, Xavier Léauté, xin, yaphet,
>
> Yaswanth Kumar, ying-zheng, Yu
>
>
>
>
>
>
>
> We welcome your help and feedback. For more information on how to
>
> report problems, and to get involved, visit the project website at
>
> https://kafka.apache.org/
>
>
>
>
>
> Thank you!
>
>
>
>
>
> Regards,
>
>
>
> Rajini
>

Re: [VOTE] KIP-313: Add KStream.flatTransform and KStream.flatTransformValues

[Damian Guy]

+1

On Mon, 25 Jun 2018 at 02:16 Matthias J. Sax  wrote:

> +1 (binding)
>
>
> -Matthias
>
> On 6/22/18 10:25 AM, Bill Bejeck wrote:
> > Thanks for the KIP, +1.
> >
> > -Bill
> >
> > On Fri, Jun 22, 2018 at 1:08 PM Ted Yu  wrote:
> >
> >> +1
> >>
> >> On Fri, Jun 22, 2018 at 2:50 AM, Bruno Cadonna 
> wrote:
> >>
> >>> Hi list,
> >>>
> >>> I would like to voting on this KIP.
> >>>
> >>> I created a first PR[1] that adds flatTransform. Once I get some
> >>> feedback, I will start work on flatTransformValues.
> >>>
> >>> Best regards,
> >>> Bruno
> >>>
> >>> [1] https://github.com/apache/kafka/pull/5273
> >>>
> >>
> >
>
>

Re: [VOTE] KIP-330: Add retentionPeriod in SessionBytesStoreSupplier

[Damian Guy]

+1

On Thu, 28 Jun 2018 at 02:16 Ted Yu  wrote:

> +1
>
> On Wed, Jun 27, 2018 at 4:40 PM, Bill Bejeck  wrote:
>
> > +1
> >
> > -Bill
> >
> > On Wed, Jun 27, 2018 at 7:39 PM Guozhang Wang 
> wrote:
> >
> > > Hello folks,
> > >
> > > I'd like to start a voting thread on KIP-330. I've intentionally
> skipped
> > > the discuss phase since it is a pretty straight-forward public API
> change
> > > and should actually be added since day one. The bug fix of KAFKA-7071
> > > helped us to discover this overlook.
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

[Damian Guy]

+1

On Tue, 26 Jun 2018 at 17:22 Bill Bejeck  wrote:

> +1
>
> On Mon, Jun 25, 2018 at 11:07 PM Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> > On 6/25/18 3:00 PM, Guozhang Wang wrote:
> > > +1
> > >
> > > On Mon, Jun 25, 2018 at 2:58 PM, Ted Yu  wrote:
> > >
> > >> +1
> > >>
> > >> On Mon, Jun 25, 2018 at 2:56 PM, John Roesler 
> > wrote:
> > >>
> > >>> Hello All,
> > >>>
> > >>> Thanks for the discussion on KIP-319. I'd now like to start the
> voting.
> > >>>
> > >>> As a reminder, KIP-319 proposes a fix to an issue I identified in
> > >>> KAFKA-7080. Specifically, the issue is that we're creating
> > >>> CachingWindowStore with the *number of segments* instead of the
> > *segment
> > >>> size*.
> > >>>
> > >>> Here's the jira: https://issues.apache.org/jira/browse/KAFKA-7080
> > >>> Here's the KIP: https://cwiki.apache.org/confluence/x/mQU0BQ
> > >>>
> > >>> Additionally, here's a draft PR for clarity:
> > >>> https://github.com/apache/kafka/pull/5257
> > >>>
> > >>> Thanks,
> > >>> -John
> > >>>
> > >>
> > >
> > >
> > >
> >
> >
>

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

[Damian Guy]

+1 (binding)

Thanks

On Wed, 27 Jun 2018 at 18:50 Bill Bejeck  wrote:

> +1
>
> -Bill
>
> On Wed, Jun 27, 2018 at 12:47 PM Manikumar 
> wrote:
>
> > +1 (non-binding)
> >
> > Thanks.
> >
> > On Wed, Jun 27, 2018 at 10:15 PM Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > On 6/26/18 2:33 PM, Guozhang Wang wrote:
> > > > +1. Thanks.
> > > >
> > > > On Tue, Jun 26, 2018 at 2:31 PM, Yishun Guan 
> > wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> I am starting a vote on this KIP:
> > > >>
> > > >> https://cwiki.apache.org/confluence/x/lQg0BQ
> > > >>
> > > >> Thanks,
> > > >> Yishun
> > > >>
> > > >
> > > >
> > > >
> > >
> > >
> >
>

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

[Damian Guy]

Thanks Bill! +1

On Mon, 25 Jun 2018 at 18:57 Ted Yu  wrote:

> +1
>
> On Mon, Jun 25, 2018 at 9:45 AM, Guozhang Wang  wrote:
>
> > +1.
> >
> > On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > On 6/25/18 6:11 AM, Bill Bejeck wrote:
> > > > All,
> > > > I'd like to start a vote for this KIP now.
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] KIP-313: Add KStream.flatTransform and KStream.flatTransformValues

[Damian Guy]

Thanks for the KIP. LGTM

On Mon, 11 Jun 2018 at 00:44 Matthias J. Sax  wrote:

> Thanks for the KIP.
>
> I don't have any comments at this pint. Overall I am +1 on the KIP.
>
>
> -Matthias
>
> On 6/8/18 2:56 PM, Bruno Cadonna wrote:
> > Hi list,
> >
> > I created KIP-313 [1] for JIRA issue KAFKA-4217 [2] and I would like to
> > put the KIP up for discussion.
> >
> > Best regards,
> > Bruno
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-313%3A+Add+KStream.flatTransform+and+KStream.flatTransformValues
> >
> > [2] https://issues.apache.org/jira/browse/KAFKA-4217
> >
>
>

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

[Damian Guy]

Hi Florian,

Thanks for the KIP. What about KTable and other DSL interfaces? Will they
not want to be able to do the same thing?
It would be good to see a complete set of the public API changes.

Cheers,
Damian

On Wed, 30 May 2018 at 19:45 Guozhang Wang  wrote:

> Hello Florian,
>
> Thanks for the KIP. I have some meta feedbacks on the proposal:
>
> 1. You mentioned that this `Processed` object will be added to a new
> overloaded variant of all the stateless operators, what about the stateful
> operators? Would like to hear your opinions if you have thought about that:
> note for stateful operators they will usually be mapped to multiple
> processor node names, so we probably need to come up with some ways to
> define all their names.
>
> 2. I share the same concern with Bill as for adding lots of new overload
> functions into the stateless operators, as we have just spent quite some
> effort in trimming them since 1.0.0 release. If the goal is to just provide
> some "hints" on the generated processor node names, not strictly enforcing
> the exact names that to be generated, then how about we just add a new
> function to `KStream` and `KTable` classes like: "as(Processed)", with the
> semantics as "the latest operators that generate this KStream / KTable will
> be named accordingly to this hint".
>
> The only caveat, is that for all operators like `KStream#to` and
> `KStream#print` that returns void, this alternative would not work. But for
> the current operators:
>
> a. KStream#print,
> b. KStream#foreach,
> c. KStream#to,
> d. KStream#process
>
> I personally felt that except `KStream#process` users would not usually
> bother to override their names, and for `KStream#process` we could add an
> overload variant with the additional Processed object.
>
>
> 3. In your example, the processor names are still added with a suffix like
> "
> -00", is this intentional? If yes, why (I thought with user
> specified processor name hints we will not add suffix to distinguish
> different nodes of the same type any more)?
>
>
> Guozhang
>
>
> On Tue, May 29, 2018 at 6:47 AM, Bill Bejeck  wrote:
>
> > Hi Florian,
> >
> > Thanks for the KIP.  I think being able to add more context to the
> > processor names would be useful.
> >
> > I like the idea of adding a "withProcessorName" to Produced, Consumed and
> > Joined.
> >
> > But instead of adding the "Processed" parameter to a large percentage of
> > the methods, which would result in overloaded methods (which we removed
> > quite a bit with KIP-182) what do you think of adding a method
> > to the AbstractStream class "withName(String processorName)"? BTW I"m not
> > married to the method name, it's the best I can do off the top of my
> head.
> >
> > For the methods that return void, we'd have to add a parameter, but that
> > would at least cut down on the number of overloaded methods in the API.
> >
> > Just my 2 cents.
> >
> > Thanks,
> > Bill
> >
> > On Sun, May 27, 2018 at 4:13 PM, Florian Hussonnois <
> fhussonn...@gmail.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I would like to start a new discussion on following KIP :
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL
> > >
> > > This is still a draft.
> > >
> > > Looking forward for your feedback.
> > > --
> > > Florian HUSSONNOIS
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Damian Guy]

+1

On Tue, 22 May 2018 at 00:12, Gwen Shapira  wrote:

> Recasting my +1
>
> On Mon, May 21, 2018 at 3:38 PM, Guozhang Wang  wrote:
>
> > Thanks Matthias, will do.
> >
> > On Mon, May 21, 2018 at 3:35 PM, Matthias J. Sax 
> > wrote:
> >
> > > Thanks. I am fine with changing the `StreamPartitioner` interface
> > directly.
> > >
> > > Can you add the idea bout `DynamicStreamPartitioner` to "rejected
> > > alternative" section. Thx.
> > >
> > >
> > > recasting +1 (binding)
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 5/21/18 3:04 PM, Guozhang Wang wrote:
> > > > Hello Matthias,
> > > >
> > > > I've tried it out on the PR, the implementation should be fine but
> one
> > > > concern I had is that, as you may also realize users of
> > > > DynamicStreamPartitioner needs to implement two interface functions,
> > with
> > > > and without the topic name if it is extending from StreamPartitioner;
> > we
> > > > could also let it to not extend from StreamPartition so it has one
> > > function
> > > > only but then we'd need Produced to have two functions allowing
> > > > StreamPartitioner and DynamicStreamPartitioner. Thinking about the
> pros
> > > and
> > > > cons I'm think it may be better to just change the interface of
> > > > StreamPartitioner itself, since even without dynamic routing,
> allowing
> > > the
> > > > topic name could let users to give one partitioner implementation
> that
> > > > branch on the topic names other than having one partitioner per
> topic.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Mon, May 21, 2018 at 11:56 AM, Matthias J. Sax <
> > matth...@confluent.io
> > > >
> > > > wrote:
> > > >
> > > >> I think that the risk of the change is moderate as I expect most
> > people
> > > >> to use the DefaultStreamPartitioner.
> > > >>
> > > >> However, there would still be possibility to define a new interface
> > > >> instead of changing the old:
> > > >>
> > > >>> public interface DynamicStreamPartitioner {
> > > >>> Integer partition(String topic, K key, V value, int
> > numPartitions);
> > > >>> }
> > > >>
> > > >> The newly added methods `Topology#addSink` and `KStream#to` would
> take
> > > >> this new interface instead of the old.
> > > >>
> > > >> Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to
> > make
> > > >> runtime code work though...
> > > >>
> > > >> WDYT?
> > > >>
> > > >> -Matthias
> > > >>
> > > >> On 5/21/18 11:47 AM, Guozhang Wang wrote:
> > > >>> Hello everyone,
> > > >>>
> > > >>> While implementing the PR for this KIP I realized there is once
> place
> > > >> which
> > > >>> we should consider modifying on public APIs as well:
> > > >>> StreamPartitioner#partition, to add the topic name string. Note it
> > will
> > > >> be
> > > >>> a incompatible change that requires users who have customized
> > > >>> StreamPartitioner implementations.
> > > >>>
> > > >>> I've updated the wiki page of KIP-303, please recast your vote on
> > this
> > > >>> thread. Thanks!!!
> > > >>>
> > > >>>
> > > >>> Guozhang
> > > >>>
> > > >>>
> > > >>> On Thu, May 17, 2018 at 3:15 PM, John Roesler 
> > > wrote:
> > > >>>
> > >  +1 non-binding
> > > 
> > >  On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax <
> > > matth...@confluent.io
> > > >>>
> > >  wrote:
> > > 
> > > > +1 (binding)
> > > >
> > > >
> > > > On 5/17/18 12:18 PM, Ted Yu wrote:
> > > >> +1
> > > >>  Original message From: Gwen Shapira <
> > >  g...@confluent.io>
> > > > Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev <
> dev@kafka.apache.org
> > >
> > > > Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
> > >  Streams'
> > > > Topology Sink
> > > >> Yay, its about time :)
> > > >>
> > > >> +1
> > > >>
> > > >> On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang <
> > wangg...@gmail.com
> > > >
> > > > wrote:
> > > >>
> > > >>> Hello folks,
> > > >>>
> > > >>> I'd like to start a voting thread on adding dynamic routing
> > > > functionality
> > > >>> in Streams sink node. Please find a KIP here:
> > > >>>
> > > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >>> 303%3A+Add+Dynamic+Routing+in+Streams+Sink
> > > >>>
> > > >>>
> > > >>> And the PR itself ready for review as well under KAFKA-4936:
> > > >>>
> > > >>> https://github.com/apache/kafka/pull/5018
> > > >>>
> > > >>>
> > > >>>
> > > >>> Thanks!
> > > >>> -- Guozhang
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > > 
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

[Damian Guy]

+1 (binding)

Thanks

On Mon, 21 May 2018 at 04:59 Ron Dagostino  wrote:

> Hi Committers.  One more binding affirmative vote is required if KIP 255
> is to have a chance of being included in the 2.0.0 release.  Please vote
> today.
>
> Ron
>
> > On May 18, 2018, at 9:27 PM, Ron Dagostino  wrote:
> >
> > Hi committers.  KIP 255 still needs 1 more binding vote.  Currently
> there are two binding + 1 votes, from Rajini and Jun, and three non-binding
> +1 votes, from Mickael, Manikumar, and myself.
> >
> > Please vote by the Monday deadline.
> >
> > Ron
> >
> >> On Thu, May 17, 2018 at 10:59 AM, Ron Dagostino 
> wrote:
> >> Hi Jun.  I've updated the KIP to add a new section titled "Summary for
> Production Use" that includes this information along with a consolidated
> set of references to the applicable specifications.  Thanks for the
> questions.
> >>
> >> *We still need another binding vote* (currently there are two binding +
> 1 votes, from Rajini and Jun, and three non-binding +1 votes, from Mickael,
> Manikumar, and myself).
> >>
> >> Please vote before the May 22nd KIP Freeze deadline so this KIP can be
> included in the 2.0.0 release.
> >>
> >> A pull request is available and includes additional commits reflecting
> initial review comments: https://github.com/apache/kafka/pull/4994
> >>
> >> Ron
> >>
> >>> On Wed, May 16, 2018 at 8:09 PM, Jun Rao  wrote:
> >>> Hi, Ron,
> >>>
> >>> Thanks. I understand now. It may be useful to add a reference to JWT
> in the
> >>> KIP.
> >>>
> >>> Jun
> >>>
> >>> On Tue, May 15, 2018 at 6:51 PM, Ron Dagostino 
> wrote:
> >>>
> >>> > Hi Jun.  I think you are getting at the fact that OAuth 2 is a
> flexible
> >>> > framework that allows different installations to do things
> differently.  It
> >>> > is true that the principal name in Kafka could come from any claim
> in the
> >>> > token.  Most of the time it would come from the 'sub' claim, but it
> could
> >>> > certainly come from another claim, or it could be only indirectly
> based on
> >>> > a claim value (maybe certain text would be trimmed or
> prefixed/suffixed).
> >>> > The point, which I think you are getting at, is that because the
> framework
> >>> > is flexible we need to accommodate that flexibility.  The callback
> handler
> >>> > class defined by the listener.name.sasl_ssl.oauthbearer.sasl.server.
> >>> > callback.handler.class configuration value gives us the required
> >>> > flexibility.  As an example, I have an implementation that leverages
> a
> >>> > popular open source JOSE library to parse the compact serialization,
> >>> > retrieve the public key if it has not yet been retrieved, verify the
> >>> > digital signature, and map the 'sub' claim to the OAuthBearerToken's
> >>> > principal name (which becomes the SASL authorization ID, which
> becomes the
> >>> > Kafka principal name).  I could just as easily have mapped a
> different
> >>> > claim to the OAuthBearerToken's principal name, manipulated the
> 'sub' claim
> >>> > value in some way, etc.  I write the callback handler code, so I
> complete
> >>> > flexibility to do whatever my OAuth 2 installation requires me to do.
> >>> >
> >>> > Ron
> >>> >
> >>> > On Tue, May 15, 2018 at 1:39 PM, Jun Rao  wrote:
> >>> >
> >>> > > Hi, Ron,
> >>> > >
> >>> > > Thanks for the reply. I understood your answers to #2 and #3.
> >>> > >
> >>> > > For #1, will the server map all clients' principal name to the
> value
> >>> > > associated with "sub" claim? How do we support mapping different
> clients
> >>> > to
> >>> > > different principal names?
> >>> > >
> >>> > > Jun
> >>> > >
> >>> > > On Mon, May 14, 2018 at 7:02 PM, Ron Dagostino 
> >>> > wrote:
> >>> > >
> >>> > > > Hi Jun.  Thanks for the +1 vote.
> >>> > > >
> >>> > > > Regarding the first question about token claims, yes, you have it
> >>> > correct
> >>> > > > about translating the OAuth token to a principle name via a JAAS
> module
> >>> > > > option in the default unsecured case.  Specifically, the OAuth
> SASL
> >>> > > Server
> >>> > > > implementation is responsible for setting the authorization ID,
> and it
> >>> > > gets
> >>> > > > the authorization ID from the OAuthBearerToken's principalName()
> >>> > method.
> >>> > > > The listener.name.sasl_ssl.oauthbearer.sasl.server.
> >>> > > callback.handler.class
> >>> > > > is responsible for handling an instance of
> OAuthBearerValidatorCallback
> >>> > > to
> >>> > > > accept a token compact serialization from the client and return
> an
> >>> > > instance
> >>> > > > of OAuthBearerToken (assuming the compact serialization
> validates), and
> >>> > > in
> >>> > > > the default unsecured case the builtin unsecured validator
> callback
> >>> > > handler
> >>> > > > defines the OAuthBearerToken.principalName() method to return the
> >>> > 'sub'
> >>> > > > claim value by default (with the actual claim it uses being
> >>> > configurable
> >>> > > > via the unsecuredValidatorPrincipalClaimName JAAS module
> option).  So
> >>> > > that
> >>> > > > is h

Re: [VOTE] KIP-290: Support for wildcard suffixed ACLs

[Damian Guy]

+1 (binding)

On Sat, 19 May 2018 at 03:51 Piyush Vijay  wrote:

> Hi everyone,
>
> I would like to start a vote for
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+wildcard+suffixed+ACLs
> .
>
> The KIP proposes a way to support wildcard-suffixed resource names in Kafka
> ACLs.
>
> The main challenge was to support it in a backward compatible way because
> resources like consumer groups don't have defined naming convention and can
> have '*' in their names.
>
> Please take a look.
>
> Thanks
>
> Piyush Vijay
>

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Damian Guy]

+1 (binding)

Thanks

On Mon, 21 May 2018 at 11:26 Edoardo Comar  wrote:

> Hi, KIP freeze is tomorrow. Anyone else would like to cast their vote ?
>
> voting so far :
> binding: Guozhang, Rajini
> non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt
>
> On 18 May 2018 at 09:38, Edoardo Comar  wrote:
> > bumping again as this KIP is still missing one binding vote ...
> >
> > voting so far :
> > binding: Guozhang, Rajini
> > non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt
> >
> > Edo
> >
> > On 17 May 2018 at 03:15, Matt Farmer  wrote:
> >> +1 (non-binding)
> >>
> >> On Tue, May 15, 2018 at 4:26 AM, Edoardo Comar 
> wrote:
> >>
> >>> Hi,
> >>> bumping the thread as the current vote count for this KIP is
> >>> 2 binding +1
> >>> 5 non-binding +1
> >>>
> >>> thanks, Edo
> >>>
> >>> On 8 May 2018 at 16:14, Edoardo Comar  wrote:
> >>> > Hi,
> >>> > bumping the thread as the current vote count for this KIP is
> >>> > 2 binding +1
> >>> > 5 non-binding +1
> >>> >
> >>> > so still missing a binding vote please
> >>> >
> >>> > thanks,
> >>> > Edo
> >>> >
> >>> >
> >>> > On 30 April 2018 at 12:49, Manikumar 
> wrote:
> >>> >>
> >>> >> +1 (non-binding)
> >>> >>
> >>> >> Thanks
> >>> >>
> >>> >> On Thu, Apr 26, 2018 at 9:59 PM, Colin McCabe 
> >>> wrote:
> >>> >>
> >>> >> > +1 (non-binding)
> >>> >> >
> >>> >> > best,
> >>> >> > Colin
> >>> >> >
> >>> >> >
> >>> >> > On Wed, Apr 25, 2018, at 02:45, Edoardo Comar wrote:
> >>> >> > > Hi,
> >>> >> > >
> >>> >> > > The discuss thread on KIP-277 (
> >>> >> > > https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html
> )
> >>> >> > > seems to have been fruitful and concerns have been addressed,
> please
> >>> >> > allow
> >>> >> > > me start a vote on it:
> >>> >> > >
> >>> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> >> > 277+-+Fine+Grained+ACL+for+CreateTopics+API
> >>> >> > >
> >>> >> > > I will update the small PR to the latest KIP semantics if the
> vote
> >>> >> > passes
> >>> >> > > (as I hope :-).
> >>> >> > >
> >>> >> > > cheers
> >>> >> > > Edo
> >>> >> > > --
> >>> >> > >
> >>> >> > > Edoardo Comar
> >>> >> > >
> >>> >> > > IBM Message Hub
> >>> >> > >
> >>> >> > > IBM UK Ltd, Hursley Park, SO21 2JN
> >>> >> > > Unless stated otherwise above:
> >>> >> > > IBM United Kingdom Limited - Registered in England and Wales
> with
> >>> >> > > number
> >>> >> > > 741598.
> >>> >> > > Registered office: PO Box 41, North Harbour, Portsmouth,
> Hampshire
> >>> PO6
> >>> >> > 3AU
> >>> >> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > "When the people fear their government, there is tyranny; when the
> >>> > government fears the people, there is liberty." [Thomas Jefferson]
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> "When the people fear their government, there is tyranny; when the
> >>> government fears the people, there is liberty." [Thomas Jefferson]
> >>>
> >
> >
> >
> > --
> > "When the people fear their government, there is tyranny; when the
> > government fears the people, there is liberty." [Thomas Jefferson]
>
>
>
> --
> "When the people fear their government, there is tyranny; when the
> government fears the people, there is liberty." [Thomas Jefferson]
>

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Damian Guy]

Thanks Guozhang. +1

Note: the link to the KIP is broken

On Thu, 17 May 2018 at 11:38 Guozhang Wang  wrote:

> Hello folks,
>
> I'd like to start a voting thread on adding dynamic routing functionality
> in Streams sink node. Please find a KIP here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 303%3A+Add+Dynamic+Routing+in+Streams+Sink
>
>
> And the PR itself ready for review as well under KAFKA-4936:
>
> https://github.com/apache/kafka/pull/5018
>
>
>
> Thanks!
> -- Guozhang
>

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

[Damian Guy]

Hi Guozhang, yes i think it would make sense to add this now as having the
additional record context would be valuable. Though i'm happy either way.

On Wed, 16 May 2018 at 23:07 Guozhang Wang  wrote:

> Hi Damian,
>
> My current plan is to add the "RichKeyValueMapper" when getting in KIP-159
> to include the record context in this dynamic routing feature. So just to
> clarify: are you more concerning that if we are going to do that anyways in
> the future, we should not add overloaded functions with "KeyValueMapper" as
> of now since they will be subsumed soon?
>
>
> Guozhang
>
>
> On Wed, May 16, 2018 at 2:59 PM, Damian Guy  wrote:
>
>> Overall i'm a +1 on this, but i'm not a big fan of using the
>> KeyValueMapper
>> to choose the topic. It is a bit counter-intuitve to me. I'd prefer to add
>> a class specifically for it and possibly pass in the RecordContext
>>
>> On Wed, 16 May 2018 at 13:22 Guozhang Wang  wrote:
>>
>> > Hello folks,
>> >
>> > Please let me know if you have further feedbacks; if there is no more
>> > feedbacks I'm going to start the voting thread soon.
>> >
>> >
>> > Guozhang
>> >
>> >
>> > On Wed, May 16, 2018 at 8:31 AM, Guozhang Wang 
>> wrote:
>> >
>> > > I have thought about exposing record context as well, and in the end I
>> > > decided to piggy-back it with KIP-159. And if we want to indeed reuse
>> the
>> > > class it will be:
>> > >
>> > > ```
>> > > public interface RichKeyValueMapper {
>> > > VR apply(final K key, final V value, final RecordContext
>> > > recordContext);
>> > > }
>> > > ```
>> > >
>> > >
>> > >
>> > > Guozhang
>> > >
>> > >
>> > > On Tue, May 15, 2018 at 10:04 PM, Matthias J. Sax <
>> matth...@confluent.io
>> > >
>> > > wrote:
>> > >
>> > >> Just my 2 cents:
>> > >>
>> > >> I am fine with `KeyValueMapper` (+1 for code reusage) -- the JavaDocs
>> > >> will explain what the `KeyValueMapper` is supposed to do, ie, extract
>> > >> and return the sink topic name from the key-value pair.
>> > >>
>> > >> A side remark though: do we think that accessing key/value is
>> > >> sufficient? Or should we provide access to the full metadata? We
>> could
>> > >> also do this with KIP-159 of course -- but this would come earliest
>> in
>> > >> 2.1. As an alternative we could add a `TopicNameExtractor` to expose
>> the
>> > >> whole record context. The advantage would be, that we don't need to
>> > >> change it via KIP-159 later again. WDYT?
>> > >>
>> > >> -Matthias
>> > >>
>> > >> On 5/15/18 5:57 PM, Bill Bejeck wrote:
>> > >> > Thanks for the KIP Guozhang, it's a +1 for me.
>> > >> >
>> > >> > As for re-using the KeyValueMapper for choosing the topic, I am on
>> the
>> > >> > fence, a more explicitly named class would be more clear, but I'm
>> not
>> > >> sure
>> > >> > it's worth a new class that will primarily perform the same
>> actions as
>> > >> the
>> > >> > KeyValueMapper.
>> > >> >
>> > >> > Thanks,
>> > >> > Bill
>> > >> >
>> > >> > On Tue, May 15, 2018 at 5:52 PM, Guozhang Wang > >
>> > >> wrote:
>> > >> >
>> > >> >> Hello John:
>> > >> >>
>> > >> >> * As for the type superclass, it is mainly for allowing super
>> class
>> > >> serdes.
>> > >> >> More details can be found here:
>> > >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > >> >> 100+-+Relax+Type+constraints+in+Kafka+Streams+API
>> > >> >>
>> > >> >> * I may have slight preference on reusing existing classes but I
>> > think
>> > >> most
>> > >> >> of my rationales are quite subjective. Personally I do not find
>> `self
>> > >> >> documenting` worth a new class but I can be convinced if people
>> have
>> > >> other
>> > >> >> motivations doing it :)

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

[Damian Guy]

Overall i'm a +1 on this, but i'm not a big fan of using the KeyValueMapper
to choose the topic. It is a bit counter-intuitve to me. I'd prefer to add
a class specifically for it and possibly pass in the RecordContext

On Wed, 16 May 2018 at 13:22 Guozhang Wang  wrote:

> Hello folks,
>
> Please let me know if you have further feedbacks; if there is no more
> feedbacks I'm going to start the voting thread soon.
>
>
> Guozhang
>
>
> On Wed, May 16, 2018 at 8:31 AM, Guozhang Wang  wrote:
>
> > I have thought about exposing record context as well, and in the end I
> > decided to piggy-back it with KIP-159. And if we want to indeed reuse the
> > class it will be:
> >
> > ```
> > public interface RichKeyValueMapper {
> > VR apply(final K key, final V value, final RecordContext
> > recordContext);
> > }
> > ```
> >
> >
> >
> > Guozhang
> >
> >
> > On Tue, May 15, 2018 at 10:04 PM, Matthias J. Sax  >
> > wrote:
> >
> >> Just my 2 cents:
> >>
> >> I am fine with `KeyValueMapper` (+1 for code reusage) -- the JavaDocs
> >> will explain what the `KeyValueMapper` is supposed to do, ie, extract
> >> and return the sink topic name from the key-value pair.
> >>
> >> A side remark though: do we think that accessing key/value is
> >> sufficient? Or should we provide access to the full metadata? We could
> >> also do this with KIP-159 of course -- but this would come earliest in
> >> 2.1. As an alternative we could add a `TopicNameExtractor` to expose the
> >> whole record context. The advantage would be, that we don't need to
> >> change it via KIP-159 later again. WDYT?
> >>
> >> -Matthias
> >>
> >> On 5/15/18 5:57 PM, Bill Bejeck wrote:
> >> > Thanks for the KIP Guozhang, it's a +1 for me.
> >> >
> >> > As for re-using the KeyValueMapper for choosing the topic, I am on the
> >> > fence, a more explicitly named class would be more clear, but I'm not
> >> sure
> >> > it's worth a new class that will primarily perform the same actions as
> >> the
> >> > KeyValueMapper.
> >> >
> >> > Thanks,
> >> > Bill
> >> >
> >> > On Tue, May 15, 2018 at 5:52 PM, Guozhang Wang 
> >> wrote:
> >> >
> >> >> Hello John:
> >> >>
> >> >> * As for the type superclass, it is mainly for allowing super class
> >> serdes.
> >> >> More details can be found here:
> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> >> 100+-+Relax+Type+constraints+in+Kafka+Streams+API
> >> >>
> >> >> * I may have slight preference on reusing existing classes but I
> think
> >> most
> >> >> of my rationales are quite subjective. Personally I do not find `self
> >> >> documenting` worth a new class but I can be convinced if people have
> >> other
> >> >> motivations doing it :)
> >> >>
> >> >>
> >> >> Guozhang
> >> >>
> >> >>
> >> >> On Tue, May 15, 2018 at 11:19 AM, John Roesler 
> >> wrote:
> >> >>
> >> >>> Thanks for the KIP, Guozhang.
> >> >>>
> >> >>> It looks good overall to me; I just have one question:
> >> >>> * Why do we bound the generics of KVMapper in KStream to be
> >> >> superclass-of-K
> >> >>> and superclass-of-V instead of exactly K and V, as in Topology? I
> >> might
> >> >> be
> >> >>> thinking about it wrong, but that seems backwards to me. If
> anything,
> >> >>> bounding to be a subclass-of-K or subclass-of-V would seem right to
> >> me.
> >> >>>
> >> >>> One extra thought: I agree that KVMapper
> >> is an
> >> >>> applicable type for extracting the topic name, but I wonder what the
> >> >> value
> >> >>> of reusing the KVMapper for this purpose is. Would defining a new
> >> class,
> >> >>> say TopicNameExtractor, provide the same functionality while
> >> being a
> >> >>> bit more self-documenting?
> >> >>>
> >> >>> Thanks,
> >> >>> -John
> >> >>>
> >> >>> On Tue, May 15, 2018 at 12:32 AM, Guozhang Wang  >
> >> >>> wrote:
> >> >>>
> >>  Hello folks,
> >> 
> >>  I'd like to start a discussion on adding dynamic routing
> >> functionality
> >> >> in
> >>  Streams sink node. I.e. users do not need to specify the topic name
> >> at
> >>  compilation time but can dynamically determine which topic to send
> to
> >> >>> based
> >>  on each record's key value pairs. Please find a KIP here:
> >> 
> >>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>  303%3A+Add+Dynamic+Routing+in+Streams+Sink
> >> 
> >>  Any feedbacks are highly appreciated.
> >> 
> >>  Thanks!
> >> 
> >>  -- Guozhang
> >> 
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> -- Guozhang
> >> >>
> >> >
> >>
> >>
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Damian Guy]

+1
On Tue, 15 May 2018 at 15:04, Ted Yu  wrote:

> +1
>  Original message From: Guozhang Wang 
> Date: 5/15/18  2:34 PM  (GMT-08:00) To: dev@kafka.apache.org Subject: Re:
> [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology
> Optimization
> +1 (binding).
>
> On Tue, May 15, 2018 at 2:16 PM, Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> >
> > On 5/15/18 1:45 PM, Bill Bejeck wrote:
> > > Hi all,
> > >
> > > I'd like to start a vote on KIP-295: Add Streams Configuration Allowing
> > for
> > > Optional Topology Optimization.
> > >
> > > KIP wiki page:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 295%3A+Add+Streams+Configuration+Allowing+for+
> > Optional+Topology+Optimization
> > >
> > > Discussion thread:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg87593.html
> > >
> > > Thanks,
> > > Bill
> > >
> >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-244: Add Record Header support to Kafka Streams

[Damian Guy]

Thanks. +1 (binding)

On Tue, 15 May 2018 at 01:04 Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> @Guozhang added. Thanks!
>
> El mar., 15 may. 2018 a las 5:50, Matthias J. Sax ( >)
> escribió:
>
> > +1 (binding)
> >
> > Thanks a lot for the KIP!
> >
> > -Matthias
> >
> > On 5/14/18 10:17 AM, Guozhang Wang wrote:
> > > +1 from me
> > >
> > > One more comment on the wiki: while reviewing the PR I realized that
> in `
> > > MockProcessorContext.java
> > > <
> >
> https://github.com/apache/kafka/pull/4955/files#diff-d5440e7338f775230019a86e6bcacccb
> > >`
> > > we are also adding one additional API plus modifying the existing
> > > `setRecordMetadata` API. Since this class is part of the public
> > test-utils
> > > package we should claim it in the wiki as well.
> > >
> > >
> > > Guozhang
> > >
> > > On Mon, May 14, 2018 at 8:43 AM, Ted Yu  wrote:
> > >
> > >> +1
> > >>
> > >> On Mon, May 14, 2018 at 8:31 AM, Jorge Esteban Quilcate Otoya <
> > >> quilcate.jo...@gmail.com> wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> I would like to start a vote on KIP-244: Add Record Header support to
> > >> Kafka
> > >>> Streams
> > >>>
> > >>> KIP wiki page:
> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>> 244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> > >>>
> > >>> The discussion thread is here:
> > >>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201805.
> > >>> mbox/%3CCAC3UcJvrgcBfe6%3DiW6%2BuTWsLB%2B4CsHgRmDx9TvCzJQrWvfg7_w%
> > >>> 40mail.gmail.com%3E
> > >>>
> > >>> Cheers,
> > >>> Jorge.
> > >>>
> > >>
> > >
> > >
> > >
> >
> >
>

Re: [VOTE] KIP-292: Add KTable#transformValues() method in Kafka Streams DSL

[Damian Guy]

+1

On Fri, 11 May 2018 at 14:30 Matthias J. Sax  wrote:

> +1 (binding)
>
>
> On 5/11/18 2:17 PM, Ted Yu wrote:
> > +1
> >
> > On Fri, May 11, 2018 at 2:16 PM, Guozhang Wang 
> wrote:
> >
> >> +1 (binding)
> >>
> >> Also slightly changed the title beyond changing [DISCUSS] to [VOTE]
> since
> >> Gmail will collapse these two threads into one, so people may not
> realize
> >> there is a voting started already.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Fri, May 11, 2018 at 2:07 PM, Andy Coates  wrote:
> >>
> >>> Hi all,
> >>>
> >>> I would like to start the vote on KIP-292: Add transformValues() method
> >> to
> >>> KTable
> >>>
> >>> The link to this KIP is here:
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 292%3A+Add+transformValues%28%29+method+to+KTable
> >>>
> >>> The discussion thread is here:
> >>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201805.mbox/%
> >>> 3CCAO1RfpHyhEEwkwad5yxd0rLgPEK-PJ4y0%2B0y%3DwidHP0uOtToJQ%
> >>> 40mail.gmail.com%3E
> >>>
> >>> Thanks,
> >>>
> >>> Andy
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

Re: [VOTE] KIP-206: Add support for UUID serialization and deserialization

[Damian Guy]

+1 (binding)

On Fri, 4 May 2018 at 08:55 Rajini Sivaram  wrote:

> Hi Brandon,
>
> +1 (binding)
>
> Thanks for the KIP!
>
> Regards,
>
> Rajini
>
> On Fri, May 4, 2018 at 2:14 PM, Brandon Kirchner <
> brandon.kirch...@gmail.com
> > wrote:
>
> > I'd like to resurrect this one last time, any chance we can get some more
> > binding votes and move this forward?
> >
> > Brandon K.
> >
> > On Fri, Feb 2, 2018 at 11:35 AM, Colin McCabe 
> wrote:
> >
> > > Hi Brandon,
> > >
> > > I think people are generally busy working on the upcoming release now.
> > > Sorry for the inconvenience.
> > >
> > > best,
> > >
> > >
> > > On Fri, Feb 2, 2018, at 07:33, Brandon Kirchner wrote:
> > > > I'd really like this to get 2 more binding votes. If that doesn't
> > happen,
> > > > how / can this still move forward? Not sure what the procedure is...
> > > >
> > > > Brandon K.
> > > >
> > > > On Tue, Jan 30, 2018 at 9:56 AM, Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 (non binding)
> > > > > Thanks for the KIP
> > > > >
> > > > > On Tue, Jan 30, 2018 at 9:49 AM, Manikumar <
> > manikumar.re...@gmail.com>
> > > > > wrote:
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Tue, Jan 30, 2018 at 11:50 AM, Ewen Cheslack-Postava <
> > > > > e...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> On Fri, Jan 26, 2018 at 9:16 AM, Colin McCabe <
> cmcc...@apache.org
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > +1 (non-binding)
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Jan 26, 2018, at 08:29, Ted Yu wrote:
> > > > > >> > > +1
> > > > > >> > >
> > > > > >> > > On Fri, Jan 26, 2018 at 7:00 AM, Brandon Kirchner <
> > > > > >> > > brandon.kirch...@gmail.com> wrote:
> > > > > >> > >
> > > > > >> > > > Hi all,
> > > > > >> > > >
> > > > > >> > > > I would like to (re)start the voting process for KIP-206:
> > > > > >> > > >
> > > > > >> > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> > > > 206%3A+Add+support+for+UUID+serialization+and+
> > deserialization
> > > > > >> > > >  > > > > >> > > > 206%3A+Add+support+for+UUID+serialization+and+
> > > deserialization>*
> > > > > >> > > >
> > > > > >> > > > The KIP adds a UUID serializer and deserializer. Possible
> > > > > >> > implementation
> > > > > >> > > > can be seen here --
> > > > > >> > > >
> > > > > >> > > > https://github.com/apache/kafka/pull/4438
> > > > > >> > > >
> > > > > >> > > > Original discussion and voting thread can be seen here --
> > > > > >> > > > http://search-hadoop.com/m/Kafka/uyzND1dlgePJY7l9?subj=+
> > > > > >> > > > DISCUSS+KIP+206+Add+support+for+UUID+serialization+and+
> > > > > >> deserialization
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Thanks!
> > > > > >> > > > Brandon K.
> > > > > >> > > >
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
>

Re: [DISCUSS] KIP-292: Add transformValues() method to KTable

[Damian Guy]

I'm a +1 for Guozhang's suggestion

On Fri, 11 May 2018 at 10:20 Andy Coates  wrote:

> Makes sense to me.  What do others think?
>
> On 11 May 2018 at 10:13, Guozhang Wang  wrote:
>
> > Hi folks,
> >
> > While looking into the overloaded functions, I'm wondering if we can save
> > the transformers without key, i.e. only add two overloaded functions:
> >
> >  KTable transformValues(final
> ValueTransformerWithKeySupplier > super K, ? super V, ? extends VR> valueTransformerSupplier,
> >final String... stateStoreNames);
> >
> >  KTable transformValues(final
> ValueTransformerWithKeySupplier > super K, ? super V, ? extends VR> valueTransformerSupplier,
> >final Materialized > KeyValueStore> materialized,
> >final String... stateStoreNames);
> >
> > The reason is that, in KIP-149 we've added the overloaded functions
> > `withKey`, which should be covering the case without key already because
> if
> > users do not really need the key, they can just take it as a dummy
> > parameter. We did not deprecate the old ones since some of them have just
> > been added one version back. But if we agree that by the end of the day
> we
> > would only maintain the overloaded value functions "with key" only, then
> we
> > should not add the ones without keys any more in new KIPs.
> >
> > WDYT?
> >
> >
> > Guozhang
> >
> >
> > On Fri, May 11, 2018 at 9:42 AM, Andy Coates  wrote:
> >
> > > Sorry for my lack of response - I've been out of action with a bad back
> > for
> > > a few days!
> > >
> > > I originally had the `Materialized` overloads added to the API. I'll
> > update
> > > the KIP / PR with these again. In terms of semantics, as Matthias
> > suggests,
> > > these should be consistent with filter() and mapValues(), etc.
> > >
> > > On 8 May 2018 at 17:59, Guozhang Wang  wrote:
> > >
> > > > To follow on Matthias and Damian's comments here:
> > > >
> > > > If we are going to add the overload functions as
> > > >
> > > > ```
> > > >  KTable transformValues(final ValueTransformerSupplier > super
> > > > V,
> > > > ? extends VR> valueTransformerSupplier,
> > > >final String... stateStoreNames,
> > > >final Materialized > > > VR, KeyValueStore materialized);
> > > >
> > > >  KTable transformValues(final
> ValueTransformerWithKeySupplie
> > > r > > > super K, ? super V, ? extends VR> valueTransformerSupplier,
> > > >final String... stateStoreNames,
> > > >  final Materialized > > > VR, KeyValueStore materialized);
> > > > ```
> > > >
> > > > Then are we going to still only allow the valueTransofmer.init() /
> > > > process() to be able to access N stores, with N stores specified with
> > the
> > > > stateStoreNames, but not the one specified in materialized.name()?
> > > > Personally I think it should be the case as the materialized store
> > should
> > > > be managed by the Streams library itself, but we should probably help
> > > users
> > > > to understand if they have some stores used for the same purpose
> > (storing
> > > > the value that are going to be sent to the downstream changelog
> stream
> > of
> > > > KTable), they should save that store and not creating by themselves
> as
> > it
> > > > will be auto created by the Streams library.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, May 8, 2018 at 7:45 AM, Damian Guy 
> > wrote:
> > > >
> > > > > Initially i thought materializing a store would be overkill, but
> > from a
> > > > > consistency point of view it makes sense to add an overload that
> > takes
> > > a
> > > > > `Materialized` and only create the store if that is supplied.
> > > > >
> > > > > On Sun, 6 May 2018 at 17:52 Matthias J. Sax  >
> > > > wrote:
> > > > >
> > > > > > Andy,
> > > > > >
> > > > > > 

Re: [DISCUSS] KIP-292: Add transformValues() method to KTable

[Damian Guy]

Initially i thought materializing a store would be overkill, but from a
consistency point of view it makes sense to add an overload that takes a
`Materialized` and only create the store if that is supplied.

On Sun, 6 May 2018 at 17:52 Matthias J. Sax  wrote:

> Andy,
>
> thanks for the KIP. I don't have any further comments.
>
> My 2cents about Guozhang's questions: as I like consistent behavior, I
> think transfromValues() should behave the same way as filter() and
> mapValues().
>
>
> -Matthias
>
> On 5/2/18 2:24 PM, Guozhang Wang wrote:
> > Hello Andy,
> >
> > Thanks for the KIP. The motivation and the general proposal looks good to
> > me. I think in KTable it is indeed valuable to add the functions that
> does
> > not change key, such as mapValues, transformValues, and filter.
> >
> > There are a few meta comments I have about the semantics of the newly
> added
> > functions:
> >
> > 1) For the resulted KTable, how should its "queryableStoreName()" be
> > returning?
> >
> > 2) More specifically, how do we decide if the resulted KTable is to be
> > materialized or not? E.g. if there is no store names provided then it is
> > likely that the resulted KTable is not materialized, or at least not
> > logically materialized and not be queryable. What if there is at least
> one
> > state store provided? Will any of them be provided as the materialized
> > store, or should we still add a Materialized parameter for this purpose?
> >
> > 3) For its internal implementations, how should the key/value serde,
> > sendOldValues flag etc be inherited from its parent processor node?
> >
> >
> > Guozhang
> >
> >
> > On Wed, May 2, 2018 at 12:43 PM, Andy Coates  wrote:
> >
> >> Hi everyone,
> >>
> >> I would like to start a discussion for KIP 292. I would appreciate it if
> >> you could review and provide feedback.
> >>
> >> KIP: KIP-292: Add transformValues() method to KTable
> >>  >> 292%3A+Add+transformValues%28%29+method+to+KTable>
> >> Jira: KAFKA-6849 
> >>
> >>PR: #4959 
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Andy
> >>
> >
> >
> >
>
>

Re: [VOTE] KIP-276 Add StreamsConfig prefix for different consumers

[Damian Guy]

Thanks +1

On Fri, 20 Apr 2018 at 23:41 Bill Bejeck  wrote:

> +1
>
> Thanks,
> Bill
>
> On Thu, Apr 19, 2018 at 9:50 AM, Boyang Chen  wrote:
>
> > Thanks guys！
> >
> > Sent from my iPhone
> >
> > > On Apr 19, 2018, at 8:28 PM, Matthias J. Sax 
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > -Matthias
> > >
> > >> On 4/19/18 9:08 AM, Ted Yu wrote:
> > >> +1
> > >>  Original message From: Guozhang Wang <
> > wangg...@gmail.com> Date: 4/18/18  10:18 PM  (GMT-08:00) To:
> > dev@kafka.apache.org Subject: Re: [VOTE] KIP-276 Add StreamsConfig
> prefix
> > for different consumers
> > >> Thanks Boyang, +1 from me.
> > >>
> > >>> On Wed, Apr 18, 2018 at 4:43 PM, Boyang Chen 
> > wrote:
> > >>>
> > >>> Hey guys,
> > >>>
> > >>>
> > >>> sorry I forgot to include a summary of this KIP.  Basically the idea
> > is to
> > >>> separate the stream config for different internal consumers of Kafka
> > >>> Streams by supplying prefixes . We currently have
> > >>>
> > >>> (1) "main.consumer." for the main consumer
> > >>>
> > >>> (2) "restore.consumer." for the restore consumer
> > >>>
> > >>> (3) "global.consumer." for the global consumer
> > >>>
> > >>>
> > >>> Best,
> > >>>
> > >>> Boyang
> > >>>
> > >>> 
> > >>> From: Boyang Chen 
> > >>> Sent: Tuesday, April 17, 2018 12:18 PM
> > >>> To: dev@kafka.apache.org
> > >>> Subject: [VOTE] KIP-276 Add StreamsConfig prefix for different
> > consumers
> > >>>
> > >>>
> > >>> Hey friends.
> > >>>
> > >>>
> > >>> I would like to start a vote on KIP 276: add StreamsConfig prefix for
> > >>> different consumers.
> > >>>
> > >>> KIP: here > >>> 276+Add+StreamsConfig+prefix+for+different+consumers>
> > >>>
> > >>> Pull request: here
> > >>>
> > >>> Jira: here
> > >>>
> > >>>
> > >>> Let me know if you have questions.
> > >>>
> > >>>
> > >>> Thank you!
> > >>>
> > >>>
> > >>>  > >>> 276+Add+StreamsConfig+prefix+for+different+consumers>
> > >>>
> > >>>
> > >>
> > >>
> > >
> >
>

Re: [VOTE] KIP-279: Fix log divergence between leader and follower after fast leader fail over

[Damian Guy]

Thanks Anna +1
On Sun, 15 Apr 2018 at 15:40, Guozhang Wang  wrote:

> Anna, thanks for the KIP! +1 from me.
>
> Just one minor comment: `Broker A will respond with offset 11`, seems it
> should be `21`.
>
> On Sun, Apr 15, 2018 at 3:48 AM, Dong Lin  wrote:
>
> > Thanks for the KIP! LGTM. +1
> >
> > On Sat, Apr 14, 2018 at 5:31 PM, Ted Yu  wrote:
> >
> > > +1
> > >
> > > On Sat, Apr 14, 2018 at 3:54 PM, Anna Povzner 
> wrote:
> > >
> > > > Hi All,
> > > >
> > > >
> > > > I would like to start the vote on KIP-279: Fix log divergence between
> > > > leader and follower after fast leader fail over.
> > > >
> > > >
> > > > For reference, here's the KIP wiki:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 279%3A+Fix+log+divergence+between+leader+and+follower+
> > > > after+fast+leader+fail+over
> > > >
> > > >
> > > >
> > > > and discussion thread:
> > > > https://www.mail-archive.com/dev@kafka.apache.org/msg86753.html
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Anna
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-270 A Scala wrapper library for Kafka Streams

[Damian Guy]

Thanks for the KIP Debasish - +1 binding

On Wed, 11 Apr 2018 at 12:16 Ted Yu  wrote:

> +1
>  Original message From: Debasish Ghosh <
> debasish.gh...@lightbend.com> Date: 4/11/18  3:09 AM  (GMT-08:00) To:
> dev@kafka.apache.org Subject: [VOTE] KIP-270 A Scala wrapper library for
> Kafka Streams
> Hello everyone -
>
> This is in continuation to the discussion regarding
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> ,
> which is a KIP for implementing a Scala wrapper library for Kafka Streams.
>
> We have had a PR (https://github.com/apache/kafka/pull/4756) for quite
> some
> time now and we have had lots of discussions and suggested improvements.
> Thanks to all who participated in the discussion and came up with all the
> suggestions for improvements.
>
> The purpose of this thread is to get an agreement on the implementation and
> have it included as part of Kafka.
>
> Looking forward ..
>
> regards.
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>

Re: [VOTE] KIP-284 Set default retention ms for Streams repartition topics to Long.MAX_VALUE

[Damian Guy]

+1 binding

On Mon, 9 Apr 2018 at 21:20 Guozhang Wang  wrote:

> +1 (binding)
>
> On Mon, Apr 9, 2018 at 11:21 AM, Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> > -Matthias
> >
> > On 4/9/18 8:57 AM, Bill Bejeck wrote:
> > > Thanks for the KIP.
> > >
> > > +1
> > >
> > > -Bill
> > >
> > > On Mon, Apr 9, 2018 at 7:57 AM, zhenya Sun  wrote:
> > >
> > >> +1
> > >>
> > >>
> > >> from my iphone！
> > >> On 04/09/2018 17:27, khaireddine Rezgui wrote:
> > >> Hi guys,
> > >>
> > >> I created this thread to get your agreement about the new default
> value
> > of
> > >> the retention_ms in kafka stream repartion topics.
> > >>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 284%3A+Set+default+retention+ms+for+Streams+repartition+
> > >> topics+to+Long.MAX_VALUE
> > >>
> > >> Thanks,
> > >>
> > >
> >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-274: Kafka Streams Skipped Records Metrics

[Damian Guy]

+1

On Thu, 5 Apr 2018 at 01:56 Matthias J. Sax  wrote:

> +1 (binding) on the latest proposal.
>
>
> -Matthias
>
>
> On 4/2/18 12:10 PM, Richard Yu wrote:
> > +1
> >
> > On Mon, Apr 2, 2018 at 8:42 AM, Guozhang Wang 
> wrote:
> >
> >> +1 (binding).
> >>
> >> On Mon, Apr 2, 2018 at 7:22 AM, Ted Yu  wrote:
> >>
> >>> +1
> >>>
> >>> On Mon, Apr 2, 2018 at 7:11 AM, Bill Bejeck  wrote:
> >>>
>  Thanks for the KIP.
> 
>  +1
> 
>  -Bill
> 
>  On Mon, Apr 2, 2018 at 10:09 AM, John Roesler 
> >> wrote:
> 
> > Dear Kafka community,
> >
> > I am proposing KIP-274 to improve visibility when Streams skips
> >> invalid
> > records.
> >
> > The proposal is to simplify the metrics and add warning logs.
> >
> > Please find details in the wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 274%3A+Kafka+Streams+Skipped+Records+Metrics
> >
> > Thanks,
> >
> > -John
> >
> 
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

Re: Subject: [VOTE] 1.1.0 RC3

[Damian Guy]

Hi,

Closing of this vote thread as there will be another RC.

Thanks,
Damian

On Mon, 19 Mar 2018 at 23:47 Ismael Juma  wrote:

> Vahid,
>
> The Java 9 Connect issue is similar to the one being fixed for Trogdor in
> the following PR:
>
> https://github.com/apache/kafka/pull/4725
>
> We need to do something similar for Connect.
>
> Ismael
>
> On Fri, Mar 16, 2018 at 3:10 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Damian,
> >
> > Thanks for running the release.
> >
> > I tried building from source and running the quick start on Linux &
> > Windows with both Java 8 & 9.
> > Here's the result:
> >
> > +-+-+-+
> > | |  Linux  | Windows |
> > + +-+-+
> > | | J8 | J9 | J8 | J9 |
> > +-+++++
> > |  Build  |  + |  + |  + |  + |
> > +-+++++
> > |  Single broker  |  + |  + |  + |  + |
> > | produce/consume |||||
> > +-+++++
> > | Connect |  + |  ? |  - |  - |
> > +-+++++
> > | Streams |  + |  + |  + |  + |
> > +-+++++
> >
> > ?: Connect quickstart on Linux with Java 9 runs but the connect tool
> > throws a bunch of exceptions (https://www.codepile.net/pile/yVg8XJB8)
> > -: Connect quickstart on Windows fails (Java 8:
> > https://www.codepile.net/pile/xJGra6BP, Java 9:
> > https://www.codepile.net/pile/oREYeORK)
> >
> > Given that Windows is not an officially supported platform, and the
> > exceptions with Linux/Java 9 are not breaking the functionality, my vote
> > is a +1 (non-binding).
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From:   Damian Guy 
> > To: dev@kafka.apache.org, us...@kafka.apache.org,
> > kafka-clie...@googlegroups.com
> > Date:   03/15/2018 07:55 AM
> > Subject:Subject: [VOTE] 1.1.0 RC3
> >
> >
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the fourth candidate for release of Apache Kafka 1.1.0.
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > apache.org_confluence_pages_viewpage.action-3FpageId-
> > 3D75957546&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> >
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2s
> > gX46mjZPws01U&s=cKgJtQXXRauZ3HSAoSbsC9SLVTAkO-pbLdPrOCBuJzE&e=
> >
> >
> > A few highlights:
> >
> > * Significant Controller improvements (much faster and session expiration
> > edge cases fixed)
> > * Data balancing across log directories (JBOD)
> > * More efficient replication when the number of partitions is large
> > * Dynamic Broker Configs
> > * Delegation tokens (KIP-48)
> > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> >
> >
> > Release notes for the 1.1.0 release:
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> > apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_RELEASE-5FNOTES.
> > html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> > kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=
> > 26FgbzRhKImhxyEkB4KzDPG-l8W_Y99xU6LykOAgpFI&e=
> >
> >
> > *** Please download, test and vote by Monday, March 19, 9am PDT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
> > apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> >
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2s
> > gX46mjZPws01U&s=xlnrfgxVFMRCKk8pTOhujyC-Um4ogtsxK6Xwks6mc3U&e=
> >
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> > apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_&d=DwIBaQ&c=jf_
> > iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=
> > Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=
> > ulHUeYnWIp28Gsn4VV1NK3FrGV4Jn5rUpuU6tvgekME&e=
> >
> >
> > * Maven artifacts to be voted upon:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__
> > repository.apache.org_content_groups_staging_&d=DwIBaQ&c=jf_
> > iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE

Re: [VOTE] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

[Damian Guy]

+1

On Wed, 21 Mar 2018 at 01:44 abel-sun  wrote:

>
>Thanks you of your offer ,agree with you!
>
> On 2018/03/21 00:56:11, Richard Yu  wrote:
> > Hi Matthias,
> > Thanks for setting up the upgrade path.
> >
> > +1 (non-binding)
> >
> > On Tue, Mar 20, 2018 at 3:42 PM, Matthias J. Sax 
> > wrote:
> >
> > > Hi,
> > >
> > > I would like to start the vote for KIP-268:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 268%3A+Simplify+Kafka+Streams+Rebalance+Metadata+Upgrade
> > >
> > > PR https://github.com/apache/kafka/pull/4636 contains the fixes to
> > > upgrade from metadata version 1 to 2. Some tests are still missing but
> > > I'll add them asap.
> > >
> > > For "version probing" including new metadata version 3 I plan to do a
> > > follow-up PR after PR-4636 is merged.
> > >
> > >
> > > -Matthias
> > >
> > >
> >
>

Re: Subject: [VOTE] 1.1.0 RC3

[Damian Guy]

We have a green system test build -
https://jenkins.confluent.io/job/system-test-kafka/job/1.1/43/
<https://jenkins.confluent.io/job/system-test-kafka/job/1.1/>

Thanks,
Damian

On Fri, 16 Mar 2018 at 22:10 Vahid S Hashemian 
wrote:

> Hi Damian,
>
> Thanks for running the release.
>
> I tried building from source and running the quick start on Linux &
> Windows with both Java 8 & 9.
> Here's the result:
>
> +-+-+-+
> | |  Linux  | Windows |
> + +-+-+
> | | J8 | J9 | J8 | J9 |
> +-+++++
> |  Build  |  + |  + |  + |  + |
> +-+++++
> |  Single broker  |  + |  + |  + |  + |
> | produce/consume |||||
> +-+++++
> | Connect |  + |  ? |  - |  - |
> +-+++++
> | Streams |  + |  + |  + |  + |
> +-+++++
>
> ?: Connect quickstart on Linux with Java 9 runs but the connect tool
> throws a bunch of exceptions (https://www.codepile.net/pile/yVg8XJB8)
> -: Connect quickstart on Windows fails (Java 8:
> https://www.codepile.net/pile/xJGra6BP, Java 9:
> https://www.codepile.net/pile/oREYeORK)
>
> Given that Windows is not an officially supported platform, and the
> exceptions with Linux/Java 9 are not breaking the functionality, my vote
> is a +1 (non-binding).
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Damian Guy 
> To: dev@kafka.apache.org, us...@kafka.apache.org,
> kafka-clie...@googlegroups.com
> Date:   03/15/2018 07:55 AM
> Subject:Subject: [VOTE] 1.1.0 RC3
>
>
>
> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.0.
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D75957546&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=cKgJtQXXRauZ3HSAoSbsC9SLVTAkO-pbLdPrOCBuJzE&e=
>
>
> A few highlights:
>
> * Significant Controller improvements (much faster and session expiration
> edge cases fixed)
> * Data balancing across log directories (JBOD)
> * More efficient replication when the number of partitions is large
> * Dynamic Broker Configs
> * Delegation tokens (KIP-48)
> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>
>
> Release notes for the 1.1.0 release:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_RELEASE-5FNOTES.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=26FgbzRhKImhxyEkB4KzDPG-l8W_Y99xU6LykOAgpFI&e=
>
>
> *** Please download, test and vote by Monday, March 19, 9am PDT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=xlnrfgxVFMRCKk8pTOhujyC-Um4ogtsxK6Xwks6mc3U&e=
>
>
> * Release artifacts to be voted upon (source and binary):
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=ulHUeYnWIp28Gsn4VV1NK3FrGV4Jn5rUpuU6tvgekME&e=
>
>
> * Maven artifacts to be voted upon:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_groups_staging_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=G9o4hXVXF0bjL_a3Wocod9GUEfy9WBBgoGa2u6yFKQw&e=
>
>
> * Javadoc:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_javadoc_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=2auaI4IIJhEORGYm1Kdpxt5TDHh0PzSvtK77lC3SJVY&e=
>
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_tree_1.1.0-2Drc3&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=h7G8XPD8vAWl_gqySi2Iocag5NnP32IT_PyirPC3Lss&

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

[Damian Guy]

Hi Debasish,

Thanks for the KIP - will be a great addition to streams. I've only had a
quick scan, but seeing as the Scala classes are going to be in their own
package could we drop the S at the end of the class names?

Thanks,
Damian


On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh 
wrote:

> Hi -
>
> A new KIP, KIP-270 is up for discussion:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
>
> The relevant JIRA issue: https://issues.apache.org/jira/browse/KAFKA-6670
>
> The library as proposed in the KIP has been implemented at
> https://github.com/lightbend/kafka-streams-scala and the current release
> is
> 0.2.0 (
> https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
> We at Lightbend has been using it since quite some time now.
>
> regards.
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>

Subject: [VOTE] 1.1.0 RC3

[Damian Guy]

Hello Kafka users, developers and client-developers,

This is the fourth candidate for release of Apache Kafka 1.1.0.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546

A few highlights:

* Significant Controller improvements (much faster and session expiration
edge cases fixed)
* Data balancing across log directories (JBOD)
* More efficient replication when the number of partitions is large
* Dynamic Broker Configs
* Delegation tokens (KIP-48)
* Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)


Release notes for the 1.1.0 release:
http://home.apache.org/~damianguy/kafka-1.1.0-rc3/RELEASE_NOTES.html

*** Please download, test and vote by Monday, March 19, 9am PDT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~damianguy/kafka-1.1.0-rc3/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~damianguy/kafka-1.1.0-rc3/javadoc/

* Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
https://github.com/apache/kafka/tree/1.1.0-rc3


* Documentation:
http://kafka.apache.org/11/documentation.html


* Protocol:
http://kafka.apache.org/11/protocol.html



* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/82/

Note: I'll update with passing system tests link once successful. If not
then we can do another RC

/**

Thanks,
Damian

Re: [VOTE] 1.1.0 RC2

[Damian Guy]

Hi All,

I'm going to close this vote thread as there has been another blocker
merged to 1.1. I'll create another RC later today.

Thanks,
Damian

On Thu, 15 Mar 2018 at 03:31 Satish Duggana 
wrote:

> +1 (non-binding)
>
> - Ran testAll/releaseTarGzAll on 1.1.0-RC2 tag
> - Ran through quickstart of core/streams
> - Ran a cluster from binaries built form sources
>   - with multiple operations.
>
> Thanks,
> Satish.
>
>
> On Thu, Mar 15, 2018 at 3:25 AM, Harsha  wrote:
>
> > +1
> > Ran tests
> > Ran a 3 node cluster to test basic operations.
> >
> > Thanks,
> > Harsha
> >
> > On Wed, Mar 14, 2018, at 9:04 AM, Ted Yu wrote:
> > > +1
> > >
> > > Ran test suite - passed (apart from testMetricsLeak which is flaky).
> > >
> > > On Wed, Mar 14, 2018 at 3:30 AM, Damian Guy 
> > wrote:
> > >
> > > > Thanks for pointing out Satish. Links updated:
> > > >
> > > > 
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the third candidate for release of Apache Kafka 1.1.0.
> > > >
> > > > This is minor version release of Apache Kakfa. It Includes 29 new
> KIPs.
> > > > Please see the release plan for more details:
> > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=75957546
> > > >
> > > > A few highlights:
> > > >
> > > > * Significant Controller improvements (much faster and session
> > expiration
> > > > edge cases fixed)
> > > > * Data balancing across log directories (JBOD)
> > > > * More efficient replication when the number of partitions is large
> > > > * Dynamic Broker Configs
> > > > * Delegation tokens (KIP-48)
> > > > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > > >
> > > > Release notes for the 1.1.0 release:
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Friday, March 16, 1pm PDT>
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc2/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/
> > > >
> > > > * Javadoc:
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/
> > > >
> > > > * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> > > > https://github.com/apache/kafka/tree/1.1.0-rc2
> > > >
> > > >
> > > > * Documentation:
> > > > http://kafka.apache.org/11/documentation.html
> > > > <http://kafka.apache.org/1/documentation.html>
> > > >
> > > > * Protocol:
> > > > http://kafka.apache.org/11/protocol.html
> > > > <http://kafka.apache.org/1/protocol.html>
> > > >
> > > > * Successful Jenkins builds for the 1.1 branch:
> > > > Unit/integration tests: https://builds.apache.org/job/
> > kafka-1.1-jdk7/78
> > > > System tests: https://jenkins.confluent.io/
> > job/system-test-kafka/job/1.1/
> > > > 38/
> > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=75957546
> > > >
> > > > On Wed, 14 Mar 2018 at 04:41 Satish Duggana <
> satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Damian,
> > > > > Thanks for starting vote thread for 1.1.0 release.
> > > > >
> > > > > There may be a typo on the tag to be voted upon for this release
> > > > candidate.
> > > > > I guess it should be
> https://github.com/apache/kafka/tree/1.1.0-rc2
> > > > > instead
> > > > > of https://github.com/apache/kafka/tree/1.1.0-rc.
> > > > >
> > > > > On Wed, Mar 14, 2018 at 8:27 AM, Satish Duggana <
> > > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Damian,
> > > > > > Given release plan 

Re: [VOTE] 1.1.0 RC2

[Damian Guy]

Thanks for pointing out Satish. Links updated:



Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 1.1.0.

This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
Please see the release plan for more details:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546

A few highlights:

* Significant Controller improvements (much faster and session expiration
edge cases fixed)
* Data balancing across log directories (JBOD)
* More efficient replication when the number of partitions is large
* Dynamic Broker Configs
* Delegation tokens (KIP-48)
* Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)

Release notes for the 1.1.0 release:
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Friday, March 16, 1pm PDT>

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/

* Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
https://github.com/apache/kafka/tree/1.1.0-rc2


* Documentation:
http://kafka.apache.org/11/documentation.html
<http://kafka.apache.org/1/documentation.html>

* Protocol:
http://kafka.apache.org/11/protocol.html
<http://kafka.apache.org/1/protocol.html>

* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.1/38/

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546

On Wed, 14 Mar 2018 at 04:41 Satish Duggana 
wrote:

> Hi Damian,
> Thanks for starting vote thread for 1.1.0 release.
>
> There may be a typo on the tag to be voted upon for this release candidate.
> I guess it should be https://github.com/apache/kafka/tree/1.1.0-rc2
> instead
> of https://github.com/apache/kafka/tree/1.1.0-rc.
>
> On Wed, Mar 14, 2018 at 8:27 AM, Satish Duggana 
> wrote:
>
> > Hi Damian,
> > Given release plan link in earlier mail is about 1.0 release. You may
> want
> > to replace that with 1.1.0 release plan link[1].
> >
> > 1 - https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=75957546
> >
> > Thanks,
> > Satish.
> >
> > On Wed, Mar 14, 2018 at 12:47 AM, Damian Guy 
> wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the third candidate for release of Apache Kafka 1.1.0.
> >>
> >> This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
> >> Please see the release plan for more details:
> >>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913
> >>
> >> A few highlights:
> >>
> >> * Significant Controller improvements (much faster and session
> expiration
> >> edge cases fixed)
> >> * Data balancing across log directories (JBOD)
> >> * More efficient replication when the number of partitions is large
> >> * Dynamic Broker Configs
> >> * Delegation tokens (KIP-48)
> >> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> >>
> >> Release notes for the 1.1.0 release:
> >> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by Friday, March 16, 1pm PDT>
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> http://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/
> >>
> >> * Javadoc:
> >> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/
> >>
> >> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> >> https://github.com/apache/kafka/tree/1.1.0-rc
> >> <https://github.com/apache/kafka/tree/1.1.0-rc1>2
> >>
> >>
> >> * Documentation:
> >> http://kafka.apache.org/11/documentation.html
> >> <http://kafka.apache.org/1/documentation.html>
> >>
> >> * Protocol:
> >> http://kafka.apache.org/11/protocol.html
> >> <http://kafka.apache.org/1/protocol.html>
> >>
> >> * Successful Jenkins builds for the 1.1 branch:
> >> Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
> >> System tests: https://jenkins.confluent.io/j
> >> ob/system-test-kafka/job/1.1/38/
> >>
> >> /**
> >>
> >> Thanks,
> >> Damian
> >>
> >>
> >> *
> >>
> >
> >
>

[VOTE] 1.1.0 RC2

[Damian Guy]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 1.1.0.

This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
Please see the release plan for more details:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913

A few highlights:

* Significant Controller improvements (much faster and session expiration
edge cases fixed)
* Data balancing across log directories (JBOD)
* More efficient replication when the number of partitions is large
* Dynamic Broker Configs
* Delegation tokens (KIP-48)
* Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)

Release notes for the 1.1.0 release:
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Friday, March 16, 1pm PDT>

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/

* Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
https://github.com/apache/kafka/tree/1.1.0-rc
2


* Documentation:
http://kafka.apache.org/11/documentation.html


* Protocol:
http://kafka.apache.org/11/protocol.html


* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.1/38/

/**

Thanks,
Damian


*

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Damian Guy]

+1
On Fri, 9 Mar 2018 at 21:30, Matthias J. Sax  wrote:

> Guozhang, renaming the class is an internal change and I think it's not
> required to cover it in the KIP.
>
>
> +1 (binding)
>
>
> -Matthias
>
> On 3/9/18 1:26 PM, Guozhang Wang wrote:
> > Hi John,
> >
> > You mentioned you are going to do the renaming for the existing class
> > to InternalMockProcessorContext
> > but that seems not reflected on the KIP wiki, could you add that in the
> > proposed changes section?
> >
> > Other than that, I'm +1 on the KIP.
> >
> > Thanks!
> >
> > Guozhang
> >
> > On Fri, Mar 9, 2018 at 12:39 PM, Ted Yu  wrote:
> >
> >> +1
> >>  Original message From: Bill Bejeck 
> >> Date: 3/9/18  12:29 PM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> Re:
> >> [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test
> Utils
> >> Thanks for the KIP.
> >>
> >> +1
> >>
> >> -Bill
> >>
> >> On Fri, Mar 9, 2018 at 3:16 PM, John Roesler  wrote:
> >>
> >>> Hello all,
> >>>
> >>> I'd like to start voting on KIP-267, to introduce a
> MockProcessorContent
> >>> enabling Processor, Transformer, and ValueTransformer authors to
> >> unit-test
> >>> their components.
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+
> >>> Test+Utils?src=jira
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>
> >
> >
> >
>
>

Re: Want to contribute

[Damian Guy]

Hi Sanket,

Done.
Thanks,
Damian

On Fri, 9 Mar 2018 at 14:56 Sanket Band  wrote:

> Hi Damian, Matthias
>
> Could you add me to the KAFKA project on jira, my jira id is sband
>
> Thanks
> Sanket
>
> On Fri, 9 Mar 2018, 20:23 Damian Guy,  wrote:
>
> > Hi Eduardo,
> >
> > There are usually JIRAs that have the newbie label. This would be a good
> > place to start.
> >
> > Cheers,
> > Damian
> >
> > On Fri, 9 Mar 2018 at 13:57 Eduardo Mello  wrote:
> >
> > > Hello,
> > > I want to begin to code for open sources projects in my free time and I
> > > want to help the Apache Kafka!
> > > I already have read the How to Contribute and Code Guidelines.
> > > Do u guys have a label for "easy" Jira's issue or something like this?
> > >
> > > Att,
> > > Eduardo Mello
> > >
> >
>

Re: Want to contribute

[Damian Guy]

Hi Eduardo,

There are usually JIRAs that have the newbie label. This would be a good
place to start.

Cheers,
Damian

On Fri, 9 Mar 2018 at 13:57 Eduardo Mello  wrote:

> Hello,
> I want to begin to code for open sources projects in my free time and I
> want to help the Apache Kafka!
> I already have read the How to Contribute and Code Guidelines.
> Do u guys have a label for "easy" Jira's issue or something like this?
>
> Att,
> Eduardo Mello
>

Re: [VOTE] 1.1.0 RC1

[Damian Guy]

Hi Jeff,

Thanks, we will look into this.

Regards,
Damian

On Thu, 8 Mar 2018 at 18:27 Jeff Chao  wrote:

> Hello,
>
> We at Heroku have run 1.1.0 RC1 through our normal performance and
> regression test suite and have found performance to be comparable to 1.0.0.
>
> That said, we're however -1 (non-binding) since this release includes
> Zookeeper 3.4.11 <https://issues.apache.org/jira/browse/KAFKA-6390> which
> is affected by the critical regression ZOOKEEPER-2960
> <https://issues.apache.org/jira/browse/ZOOKEEPER-2960>. As 3.4.12 isn't
> released yet, it might be better to have 3.4.10 included instead.
>
> Jeff
> Heroku
>
>
> On Tue, Mar 6, 2018 at 1:19 PM, Ted Yu  wrote:
>
> > +1
> >
> > Checked signature
> > Ran test suite - apart from flaky testMetricsLeak, other tests passed.
> >
> > On Tue, Mar 6, 2018 at 2:45 AM, Damian Guy  wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 1.1.0.
> > >
> > > This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
> > > Please see the release plan for more details:
> > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=71764913
> > >
> > > A few highlights:
> > >
> > > * Significant Controller improvements (much faster and session
> expiration
> > > edge cases fixed)
> > > * Data balancing across log directories (JBOD)
> > > * More efficient replication when the number of partitions is large
> > > * Dynamic Broker Configs
> > > * Delegation tokens (KIP-48)
> > > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > >
> > > Release notes for the 1.1.0 release:
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Friday, March 9th, 5pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> > > https://github.com/apache/kafka/tree/1.1.0-rc1
> > >
> > >
> > > * Documentation:
> > > http://kafka.apache.org/11/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/11/protocol.html
> > >
> > > * Successful Jenkins builds for the 1.1 branch:
> > > Unit/integration tests:
> https://builds.apache.org/job/kafka-1.1-jdk7/68
> > > System tests: https://jenkins.confluent.io/
> > job/system-test-kafka/job/1.1/
> > > 30/
> > >
> > > /**
> > >
> > > Thanks,
> > > Damian Guy
> > >
> >
>

Re: [VOTE] KIP-265: Make Windowed Serde to public APIs

[Damian Guy]

Thanks Guozhang +1

On Tue, 6 Mar 2018 at 00:26 Hu Xi  wrote:

> +1 (non-binding)
>
> 
> 发件人: Matthias J. Sax 
> 发送时间: 2018年3月6日 8:19
> 收件人: dev@kafka.apache.org
> 主题: Re: [VOTE] KIP-265: Make Windowed Serde to public APIs
>
> +1 (binding)
>
> Thanks for the KIP Guozhang!
>
> -Matthias
>
> On 3/5/18 2:41 PM, Bill Bejeck wrote:
> > Thanks for the KIP Guozhang.
> >
> > +1
> >
> > -Bill
> >
> > On Mon, Mar 5, 2018 at 5:11 PM, Ted Yu  wrote:
> >
> >> +1
> >>
> >> On Mon, Mar 5, 2018 at 1:46 PM, Guozhang Wang 
> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I'd like to start voting on KIP-265, on making windowed serde to public
> >>> APIs of Kafka Streams. It involves a couple of new configs, plus a few
> >> new
> >>> public classes for windowed serializer and deserializer, and also
> adding
> >>> the corresponding console consumer options in order to fetch from a
> topic
> >>> written by a windowed store.
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 265%3A+Make+Windowed+Serde+to+public+APIs
> >>>
> >>>
> >>>
> >>> Thanks,
> >>> -- Guozhang
> >>>
> >>
> >
>
>

[VOTE] 1.1.0 RC1

[Damian Guy]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 1.1.0.

This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
Please see the release plan for more details:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913

A few highlights:

* Significant Controller improvements (much faster and session expiration
edge cases fixed)
* Data balancing across log directories (JBOD)
* More efficient replication when the number of partitions is large
* Dynamic Broker Configs
* Delegation tokens (KIP-48)
* Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)

Release notes for the 1.1.0 release:
http://home.apache.org/~damianguy/kafka-1.1.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Friday, March 9th, 5pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~damianguy/kafka-1.1.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~damianguy/kafka-1.1.0-rc1/javadoc/

* Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
https://github.com/apache/kafka/tree/1.1.0-rc1


* Documentation:
http://kafka.apache.org/11/documentation.html

* Protocol:
http://kafka.apache.org/11/protocol.html

* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/68
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.1/30/

/**

Thanks,
Damian Guy

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC0

[Damian Guy]

Thanks Jun

On Fri, 2 Mar 2018 at 02:25 Jun Rao  wrote:

> KAFKA-6111 is now merged to 1.1 branch.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 1, 2018 at 2:50 PM, Jun Rao  wrote:
>
>> Hi, Damian,
>>
>> It would also be useful to include KAFKA-6111, which prevents 
>> deleteLogDirEventNotifications
>> path to be deleted correctly from Zookeeper. The patch should be committed
>> later today.
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Mar 1, 2018 at 1:47 PM, Damian Guy  wrote:
>>
>>> Thanks Jason. Assuming the system tests pass i'll cut RC1 tomorrow.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Thu, 1 Mar 2018 at 19:10 Jason Gustafson  wrote:
>>>
>>>> The fix has been merged to 1.1.
>>>>
>>>> Thanks,
>>>> Jason
>>>>
>>>> On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy 
>>>> wrote:
>>>>
>>>> > Hi Jason,
>>>> >
>>>> > Ok - thanks. Let me know how you get on.
>>>> >
>>>> > Cheers,
>>>> > Damian
>>>> >
>>>> > On Wed, 28 Feb 2018 at 19:23 Jason Gustafson 
>>>> wrote:
>>>> >
>>>> > > Hey Damian,
>>>> > >
>>>> > > I think we should consider
>>>> > > https://issues.apache.org/jira/browse/KAFKA-6593
>>>> > > for the release. I have a patch available, but still working on
>>>> > validating
>>>> > > both the bug and the fix.
>>>> > >
>>>> > > -Jason
>>>> > >
>>>> > > On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax <
>>>> matth...@confluent.io>
>>>> > > wrote:
>>>> > >
>>>> > > > No. Both will be released.
>>>> > > >
>>>> > > > -Matthias
>>>> > > >
>>>> > > > On 2/28/18 6:32 AM, Marina Popova wrote:
>>>> > > > > Sorry, maybe a stupid question, but:
>>>> > > > >  I see that Kafka 1.0.1 RC2 is still not released, but now
>>>> 1.1.0 RC0
>>>> > is
>>>> > > > coming up...
>>>> > > > > Does it mean 1.0.1 will be abandoned and we should be looking
>>>> forward
>>>> > > to
>>>> > > > 1.1.0 instead?
>>>> > > > >
>>>> > > > > thanks!
>>>> > > > >
>>>> > > > > ​Sent with ProtonMail Secure Email.​
>>>> > > > >
>>>> > > > > ‐‐‐ Original Message ‐‐‐
>>>> > > > >
>>>> > > > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
>>>> > > > vahidhashem...@us.ibm.com> wrote:
>>>> > > > >
>>>> > > > >> +1 (non-binding)
>>>> > > > >>
>>>> > > > >> Built the source and ran quickstart (including streams)
>>>> successfully
>>>> > > on
>>>> > > > >>
>>>> > > > >> Ubuntu (with both Java 8 and Java 9).
>>>> > > > >>
>>>> > > > >> I understand the Windows platform is not officially supported,
>>>> but I
>>>> > > ran
>>>> > > > >>
>>>> > > > >> the same on Windows 10, and except for Step 7 (Connect)
>>>> everything
>>>> > > else
>>>> > > > >>
>>>> > > > >> worked fine.
>>>> > > > >>
>>>> > > > >> There are a number of warning and errors (including
>>>> > > > >>
>>>> > > > >> java.lang.ClassNotFoundException). Here's the final error
>>>> message:
>>>> > > > >>
>>>> > > > >>> bin\\windows\\connect-standalone.bat
>>>> config\\connect-standalone.
>>>> > > > properties
>>>> > > > >>
>>>> > > > >> config\\connect-file-source.properties
>>>> config\\connect-file-sink.
>>>> > > > properties
>>>> > > > >>
>>>> > > > >> ...
>>>> > > > >>
>

Re: [DISCUSS] KIP-265: Make Windowed Serde to public APIs

[Damian Guy]

Thanks Guozhang, LGTM

On Fri, 2 Mar 2018 at 03:00 Hu Xi  wrote:

> Guozhang,
>
>
> Thanks for this KIP. Please help confirm questions below:
>
>   1.   Do we also need to retrofit `SimpleConsumerShell` to have it
> support these newly-added serdes?
>   2.   Does this KIP cover the changes for ConsoleConsumer as well?
>
>
> 
> 发件人: Guozhang Wang 
> 发送时间: 2018年3月2日 8:20
> 收件人: dev@kafka.apache.org
> 主题: [DISCUSS] KIP-265: Make Windowed Serde to public APIs
>
> Hello all,
>
> I'd like to have a discussion on making windowed serde to public APIs of
> Kafka Streams. It involves a couple of new configs, plus a few new public
> classes for windowed serializer and deserializer, and also adding the
> corresponding console consumer options in order to fetch from a topic
> written by a windowed store.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-265%3A+Make+Windowed+Serde+to+public+APIs
>
> I'd love to hear from your opinions on the proposed APIs, and if you have
> already encountered this and have to implement your own serdes, does the
> current public API fit your needs.
>
>
> Thanks,
>
> -- Guozhang
>

Re: [VOTE] 1.1.0 RC0

[Damian Guy]

Thanks Jason. Assuming the system tests pass i'll cut RC1 tomorrow.

Thanks,
Damian

On Thu, 1 Mar 2018 at 19:10 Jason Gustafson  wrote:

> The fix has been merged to 1.1.
>
> Thanks,
> Jason
>
> On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy  wrote:
>
> > Hi Jason,
> >
> > Ok - thanks. Let me know how you get on.
> >
> > Cheers,
> > Damian
> >
> > On Wed, 28 Feb 2018 at 19:23 Jason Gustafson  wrote:
> >
> > > Hey Damian,
> > >
> > > I think we should consider
> > > https://issues.apache.org/jira/browse/KAFKA-6593
> > > for the release. I have a patch available, but still working on
> > validating
> > > both the bug and the fix.
> > >
> > > -Jason
> > >
> > > On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax <
> matth...@confluent.io>
> > > wrote:
> > >
> > > > No. Both will be released.
> > > >
> > > > -Matthias
> > > >
> > > > On 2/28/18 6:32 AM, Marina Popova wrote:
> > > > > Sorry, maybe a stupid question, but:
> > > > >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0
> RC0
> > is
> > > > coming up...
> > > > > Does it mean 1.0.1 will be abandoned and we should be looking
> forward
> > > to
> > > > 1.1.0 instead?
> > > > >
> > > > > thanks!
> > > > >
> > > > > ​Sent with ProtonMail Secure Email.​
> > > > >
> > > > > ‐‐‐ Original Message ‐‐‐
> > > > >
> > > > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com> wrote:
> > > > >
> > > > >> +1 (non-binding)
> > > > >>
> > > > >> Built the source and ran quickstart (including streams)
> successfully
> > > on
> > > > >>
> > > > >> Ubuntu (with both Java 8 and Java 9).
> > > > >>
> > > > >> I understand the Windows platform is not officially supported,
> but I
> > > ran
> > > > >>
> > > > >> the same on Windows 10, and except for Step 7 (Connect) everything
> > > else
> > > > >>
> > > > >> worked fine.
> > > > >>
> > > > >> There are a number of warning and errors (including
> > > > >>
> > > > >> java.lang.ClassNotFoundException). Here's the final error message:
> > > > >>
> > > > >>> bin\\windows\\connect-standalone.bat config\\connect-standalone.
> > > > properties
> > > > >>
> > > > >> config\\connect-file-source.properties config\\connect-file-sink.
> > > > properties
> > > > >>
> > > > >> ...
> > > > >>
> > > > >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
> > > > >>
> > > > >> (org.apache.kafka.connect.cli.ConnectStandalone)
> > > > >>
> > > > >> java.lang.NoClassDefFoundError:
> > > > >>
> > > > >> org/apache/kafka/connect/transforms/util/RegexValidator
> > > > >>
> > > > >> at
> > > > >>
> > > > >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
> > > > clinit>(SinkConnectorConfig.java:46)
> > > > >>
> > > > >> at
> > > > >>
> > > > >>
> > > > >> org.apache.kafka.connect.runtime.AbstractHerder.
> > > > validateConnectorConfig(AbstractHerder.java:263)
> > > > >>
> > > > >> at
> > > > >>
> > > > >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
> > > > putConnectorConfig(StandaloneHerder.java:164)
> > > > >>
> > > > >> at
> > > > >>
> > > > >> org.apache.kafka.connect.cli.ConnectStandalone.main(
> > > > ConnectStandalone.java:107)
> > > > >>
> > > > >> Caused by: java.lang.ClassNotFoundException:
> > > > >>
> > > > >> org.apache.kafka.connect.transforms.util.RegexValidator
> > > > >>
> > > > >> at
> > > > >>
> > > > >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
&

Re: [VOTE] 1.1.0 RC0

[Damian Guy]

Hi Jason,

Ok - thanks. Let me know how you get on.

Cheers,
Damian

On Wed, 28 Feb 2018 at 19:23 Jason Gustafson  wrote:

> Hey Damian,
>
> I think we should consider
> https://issues.apache.org/jira/browse/KAFKA-6593
> for the release. I have a patch available, but still working on validating
> both the bug and the fix.
>
> -Jason
>
> On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax 
> wrote:
>
> > No. Both will be released.
> >
> > -Matthias
> >
> > On 2/28/18 6:32 AM, Marina Popova wrote:
> > > Sorry, maybe a stupid question, but:
> > >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0 RC0 is
> > coming up...
> > > Does it mean 1.0.1 will be abandoned and we should be looking forward
> to
> > 1.1.0 instead?
> > >
> > > thanks!
> > >
> > > ​Sent with ProtonMail Secure Email.​
> > >
> > > ‐‐‐ Original Message ‐‐‐
> > >
> > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Built the source and ran quickstart (including streams) successfully
> on
> > >>
> > >> Ubuntu (with both Java 8 and Java 9).
> > >>
> > >> I understand the Windows platform is not officially supported, but I
> ran
> > >>
> > >> the same on Windows 10, and except for Step 7 (Connect) everything
> else
> > >>
> > >> worked fine.
> > >>
> > >> There are a number of warning and errors (including
> > >>
> > >> java.lang.ClassNotFoundException). Here's the final error message:
> > >>
> > >>> bin\\windows\\connect-standalone.bat config\\connect-standalone.
> > properties
> > >>
> > >> config\\connect-file-source.properties config\\connect-file-sink.
> > properties
> > >>
> > >> ...
> > >>
> > >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
> > >>
> > >> (org.apache.kafka.connect.cli.ConnectStandalone)
> > >>
> > >> java.lang.NoClassDefFoundError:
> > >>
> > >> org/apache/kafka/connect/transforms/util/RegexValidator
> > >>
> > >> at
> > >>
> > >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
> > clinit>(SinkConnectorConfig.java:46)
> > >>
> > >> at
> > >>
> > >>
> > >> org.apache.kafka.connect.runtime.AbstractHerder.
> > validateConnectorConfig(AbstractHerder.java:263)
> > >>
> > >> at
> > >>
> > >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
> > putConnectorConfig(StandaloneHerder.java:164)
> > >>
> > >> at
> > >>
> > >> org.apache.kafka.connect.cli.ConnectStandalone.main(
> > ConnectStandalone.java:107)
> > >>
> > >> Caused by: java.lang.ClassNotFoundException:
> > >>
> > >> org.apache.kafka.connect.transforms.util.RegexValidator
> > >>
> > >> at
> > >>
> > >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
> > BuiltinClassLoader.java:582)
> > >>
> > >> at
> > >>
> > >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.
> > loadClass(ClassLoaders.java:185)
> > >>
> > >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
> > >>
> > >> ... 4 more
> > >>
> > >> Thanks for running the release.
> > >>
> > >> --Vahid
> > >>
> > >> From: Damian Guy damian@gmail.com
> > >>
> > >> To: dev@kafka.apache.org, us...@kafka.apache.org,
> > >>
> > >> kafka-clie...@googlegroups.com
> > >>
> > >> Date: 02/24/2018 08:16 AM
> > >>
> > >> Subject: \[VOTE\] 1.1.0 RC0
> > >>
> > >> Hello Kafka users, developers and client-developers,
> > >>
> > >> This is the first candidate for release of Apache Kafka 1.1.0.
> > >>
> > >> This is minor version release of Apache Kakfa. It Includes 29 new
> KIPs.
> > >>
> > >> Please see the release plan for more details:
> > >>
> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > apache.org_confluence_pages_viewpage.a

Re: [VOTE] KIP-261: Add Single Value Fetch in Window Stores

[Damian Guy]

Thanks Guozhang!
+1

On Sat, 24 Feb 2018 at 19:12 Ted Yu  wrote:

> +1
>
> On Sat, Feb 24, 2018 at 11:10 AM, Guozhang Wang 
> wrote:
>
> > Hi all,
> >
> > I want to start voting on KIP-261 to add a new API for window stores in
> > order to optimize our current windowed aggregation implementations inside
> > Streams DSL:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 261%3A+Add+Single+Value+Fetch+in+Window+Stores
> >
> >
> > Please cast your vote before next Wednesday, EOD.
> >
> >
> > Thanks,
> > -- Guozhang
> >
>

[VOTE] 1.1.0 RC0

[Damian Guy]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 1.1.0.

This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
Please see the release plan for more details:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913

A few highlights:

* Significant Controller improvements (much faster and session expiration
edge cases fixed)
* Data balancing across log directories (JBOD)
* More efficient replication when the number of partitions is large
* Dynamic Broker Configs
* Delegation tokens (KIP-48)
* Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)

Release notes for the 1.1.0 release:
http://home.apache.org/~damianguy/kafka-1.1.0-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Wednesday, February 28th, 5pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~damianguy/kafka-1.1.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~damianguy/kafka-1.1.0-rc0/javadoc/

* Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
https://github.com/apache/kafka/tree/1.1.0-rc0


* Documentation:
http://kafka.apache.org/11/documentation.html

* Protocol:
http://kafka.apache.org/11/protocol.html

* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/63/
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.1/21/


Finally, just want to say thanks to the people that have helped me get the
System Tests green this week. In particular:

Konstantine Karantasis
Randall Hauch
Colin Mcabe


/**

Thanks,
Damian

Re: [DISCUSS] KIP-261: Add Single Value Fetch in Window Stores

[Damian Guy]

Thanks Guozhang. +1

On Thu, 22 Feb 2018 at 23:44 Guozhang Wang  wrote:

> Thanks Ted, have updated the wiki page.
>
> On Thu, Feb 22, 2018 at 1:48 PM, Ted Yu  wrote:
>
> > +1
> >
> > There were some typos:
> > CachingWindowedStore -> CachingWindowStore
> > RocksDBWindowedStore -> RocksDBWindowStore
> > KStreamWindowedAggregate -> KStreamWindowAggregate
> > KStreamWindowedReduce -> KStreamWindowReduce
> >
> > Cheers
> >
> > On Thu, Feb 22, 2018 at 1:34 PM, Guozhang Wang 
> wrote:
> >
> > > Hi all,
> > >
> > > I have submitted KIP-261 to add a new API for window stores in order to
> > > optimize our current windowed aggregation implementations inside
> Streams
> > > DSL
> > > :
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 261%3A+Add+Single+Value+Fetch+in+Window+Stores
> > >
> > > This change would require people who have customized window store
> > > implementations to make code changes as part of their upgrade path.
> But I
> > > think it is worth while given that the fraction of customized window
> > store
> > > should be very small.
> > >
> > >
> > > Feedback and suggestions are welcome.
> > >
> > > Thanks,
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] 1.0.1 RC2

[Damian Guy]

+1

Built src and ran tests
Ran streams quickstart

On Thu, 22 Feb 2018 at 21:32 Ted Yu  wrote:

> +1
>
> MetricsTest#testMetricsLeak failed but it is flaky test
>
> On Wed, Feb 21, 2018 at 4:06 PM, Ewen Cheslack-Postava 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for release of Apache Kafka 1.0.1.
> >
> > This is a bugfix release for the 1.0 branch that was first released with
> > 1.0.0 about 3 months ago. We've fixed 49 issues since that release. Most
> of
> > these are non-critical, but in aggregate these fixes will have
> significant
> > impact. A few of the more significant fixes include:
> >
> > * KAFKA-6277: Make loadClass thread-safe for class loaders of Connect
> > plugins
> > * KAFKA-6185: Selector memory leak with high likelihood of OOM in case of
> > down conversion
> > * KAFKA-6269: KTable state restore fails after rebalance
> > * KAFKA-6190: GlobalKTable never finishes restoring when consuming
> > transactional messages
> > * KAFKA-6529: Stop file descriptor leak when client disconnects with
> staged
> > receives
> > * KAFKA-6238: Issues with protocol version when applying a rolling
> upgrade
> > to 1.0.0
> >
> > Release notes for the 1.0.1 release:
> > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Saturday Feb 24, 9pm PT ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/
> >
> > * Javadoc:
> > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/javadoc/
> >
> > * Tag to be voted upon (off 1.0 branch) is the 1.0.1 tag:
> > https://github.com/apache/kafka/tree/1.0.1-rc2
> >
> > * Documentation:
> > http://kafka.apache.org/10/documentation.html
> >
> > * Protocol:
> > http://kafka.apache.org/10/protocol.html
> >
> > /**
> >
> > Thanks,
> > Ewen Cheslack-Postava
> >
>

[jira] [Resolved] (KAFKA-6577) Connect standalone SASL file source and sink test fails without explanation

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-6577.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

Issue resolved by pull request 4610
[https://github.com/apache/kafka/pull/4610]

> Connect standalone SASL file source and sink test fails without explanation
> ---
>
> Key: KAFKA-6577
> URL: https://issues.apache.org/jira/browse/KAFKA-6577
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect, system tests
>Affects Versions: 1.1.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Blocker
> Fix For: 1.2.0, 1.1.0
>
>
> The 
> {{tests/kafkatest/tests/connect/connect_test.py::ConnectStandaloneFileTest.test_file_source_and_sink}}
>  test is failing with the SASL configuration without a sufficient 
> explanation. During the test, the Connect worker fails to start, but the 
> Connect log contains no useful information.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: 1.1 release progress

[Damian Guy]

Hi,

A quick update on the 1.1 release. I'm in the process of generating RC0,
but I'm waiting to get a passing system test build before pushing. Stay
tuned!

Thanks,
Damian

On Fri, 16 Feb 2018 at 09:49 Damian Guy  wrote:

> Hi Becket,
>
> Thanks for letting me know.
>
> Damian
>
> On Fri, 16 Feb 2018 at 01:48 Ismael Juma  wrote:
>
>> Sounds good. Thanks Becket.
>>
>> Ismael
>>
>> On 15 Feb 2018 5:30 pm, "Becket Qin"  wrote:
>>
>> > Hi Ismael,
>> >
>> > Yes, I am working on the fix. Will submit patch today.
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> > On Thu, Feb 15, 2018 at 2:53 PM, Ismael Juma  wrote:
>> >
>> > > Hi Becket,
>> > >
>> > > Thanks for filing that. Are you working on a fix?
>> > >
>> > > Ismael
>> > >
>> > > On Thu, Feb 15, 2018 at 2:51 PM, Becket Qin 
>> > wrote:
>> > >
>> > > > Hi Damian,
>> > > >
>> > > > I just created another ticket KAFKA-6568, which I believe should
>> also
>> > be
>> > > a
>> > > > blocker unless people disagree.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jiangjie (Becket) Qin
>> > > >
>> > > > On Wed, Feb 14, 2018 at 8:52 AM, Damian Guy 
>> > > wrote:
>> > > >
>> > > > > Hi All,
>> > > > >
>> > > > > The first 1.1 RC is due to be cut, however we currently have 2
>> > blockers
>> > > > > outstanding:
>> > > > > https://issues.apache.org/jira/browse/KAFKA-6517 (which i suspect
>> > will
>> > > > be
>> > > > > merged shortly)
>> > > > > and
>> > > > > https://issues.apache.org/jira/browse/KAFKA-6549
>> > > > >
>> > > > > Once we have finished these issues I can create the first RC.
>> > Hopefully
>> > > > > before the end of the week.
>> > > > >
>> > > > > Regards,
>> > > > > Damian
>> > > > >
>> > > > >
>> > > > > On Mon, 5 Feb 2018 at 18:04 Damian Guy 
>> wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > We are just over a week away from code freeze. We currently
>> have 44
>> > > > > issues
>> > > > > > in progress and 3 open blockers:
>> > > > > > https://issues.apache.org/jira/projects/KAFKA/versions/12339769
>> > > > > >
>> > > > > > If you have something that is in progress that is not going to
>> make
>> > > it
>> > > > > > into 1.1 can you please move it to a future release.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Damian
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: 1.1 release progress

[Damian Guy]

Hi Becket,

Thanks for letting me know.

Damian

On Fri, 16 Feb 2018 at 01:48 Ismael Juma  wrote:

> Sounds good. Thanks Becket.
>
> Ismael
>
> On 15 Feb 2018 5:30 pm, "Becket Qin"  wrote:
>
> > Hi Ismael,
> >
> > Yes, I am working on the fix. Will submit patch today.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Feb 15, 2018 at 2:53 PM, Ismael Juma  wrote:
> >
> > > Hi Becket,
> > >
> > > Thanks for filing that. Are you working on a fix?
> > >
> > > Ismael
> > >
> > > On Thu, Feb 15, 2018 at 2:51 PM, Becket Qin 
> > wrote:
> > >
> > > > Hi Damian,
> > > >
> > > > I just created another ticket KAFKA-6568, which I believe should also
> > be
> > > a
> > > > blocker unless people disagree.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Wed, Feb 14, 2018 at 8:52 AM, Damian Guy 
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > The first 1.1 RC is due to be cut, however we currently have 2
> > blockers
> > > > > outstanding:
> > > > > https://issues.apache.org/jira/browse/KAFKA-6517 (which i suspect
> > will
> > > > be
> > > > > merged shortly)
> > > > > and
> > > > > https://issues.apache.org/jira/browse/KAFKA-6549
> > > > >
> > > > > Once we have finished these issues I can create the first RC.
> > Hopefully
> > > > > before the end of the week.
> > > > >
> > > > > Regards,
> > > > > Damian
> > > > >
> > > > >
> > > > > On Mon, 5 Feb 2018 at 18:04 Damian Guy 
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We are just over a week away from code freeze. We currently have
> 44
> > > > > issues
> > > > > > in progress and 3 open blockers:
> > > > > > https://issues.apache.org/jira/projects/KAFKA/versions/12339769
> > > > > >
> > > > > > If you have something that is in progress that is not going to
> make
> > > it
> > > > > > into 1.1 can you please move it to a future release.
> > > > > >
> > > > > > Thanks,
> > > > > > Damian
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: 1.1 release progress

[Damian Guy]

Hi All,

The first 1.1 RC is due to be cut, however we currently have 2 blockers
outstanding:
https://issues.apache.org/jira/browse/KAFKA-6517 (which i suspect will be
merged shortly)
and
https://issues.apache.org/jira/browse/KAFKA-6549

Once we have finished these issues I can create the first RC. Hopefully
before the end of the week.

Regards,
Damian


On Mon, 5 Feb 2018 at 18:04 Damian Guy  wrote:

> Hi all,
>
> We are just over a week away from code freeze. We currently have 44 issues
> in progress and 3 open blockers:
> https://issues.apache.org/jira/projects/KAFKA/versions/12339769
>
> If you have something that is in progress that is not going to make it
> into 1.1 can you please move it to a future release.
>
> Thanks,
> Damian
>

Re: [VOTE] KIP-251: Allow timestamp manipulation in Processor API

[Damian Guy]

Thanks Matthias, +1

On Fri, 9 Feb 2018 at 02:42 Ted Yu  wrote:

> +1
>  Original message From: "Matthias J. Sax" <
> matth...@confluent.io> Date: 2/8/18  6:05 PM  (GMT-08:00) To:
> dev@kafka.apache.org Subject: [VOTE] KIP-251: Allow timestamp
> manipulation in Processor API
> Hi,
>
> I want to start the vote for KIP-251:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-251%3A+Allow+timestamp+manipulation+in+Processor+API
>
>
> -Matthias
>
>

1.1 release progress

[Damian Guy]

Hi all,

We are just over a week away from code freeze. We currently have 44 issues
in progress and 3 open blockers:
https://issues.apache.org/jira/projects/KAFKA/versions/12339769

If you have something that is in progress that is not going to make it into
1.1 can you please move it to a future release.

Thanks,
Damian

New release branch 1.1

[Damian Guy]

Hello Kafka developers and friends,

We now have a release branch for the 1.1 release (with 1.1.0 as the version).
Trunk has been bumped to 1.2.0-SNAPSHOT.

I'll be going over the JIRAs to move every non-blocker from this release to
the next release.

>From this point, most changes should go to trunk.
*Blockers (existing and new that we discover while testing the release)
will be double-committed. *Please discuss with your reviewer whether your
PR should go to trunk or to trunk+release so they can merge accordingly.

*Please help us test the release! *

Thanks!

Damian

[jira] [Resolved] (KAFKA-6378) NullPointerException on KStream-GlobalKTable leftJoin when KeyValueMapper returns null

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-6378.
---
   Resolution: Fixed
Fix Version/s: 1.1.0

Issue resolved by pull request 4424
[https://github.com/apache/kafka/pull/4424]

> NullPointerException on KStream-GlobalKTable leftJoin when KeyValueMapper 
> returns null
> --
>
> Key: KAFKA-6378
> URL: https://issues.apache.org/jira/browse/KAFKA-6378
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Andy Bryant
>Priority: Major
> Fix For: 1.1.0
>
>
> On a Stream->GlobalKTable leftJoin if the KeyValueMapper returns null, the 
> stream fails with a NullPointerException (see stacktrace below). On Kafka 
> 0.11.0.0 the stream processes this successfully, calling the ValueJoiner with 
> the table value set to null.
> The use-case for this is joining a stream to a table containing reference 
> data where the stream foreign key may be null. There is no straight-forward 
> workaround in this case with Kafka 1.0.0 without having to resort to either 
> generating a key that will never match or branching the stream for records 
> that don't have the foreign key.
> Exception in thread "workshop-simple-example-client-StreamThread-1" 
> java.lang.NullPointerException
>   at java.base/java.util.Objects.requireNonNull(Objects.java:221)
>   at 
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.get(CachingKeyValueStore.java:136)
>   at 
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.get(CachingKeyValueStore.java:35)
>   at 
> org.apache.kafka.streams.state.internals.InnerMeteredKeyValueStore.get(InnerMeteredKeyValueStore.java:184)
>   at 
> org.apache.kafka.streams.state.internals.MeteredKeyValueBytesStore.get(MeteredKeyValueBytesStore.java:116)
>   at 
> org.apache.kafka.streams.kstream.internals.KTableSourceValueGetterSupplier$KTableSourceValueGetter.get(KTableSourceValueGetterSupplier.java:49)
>   at 
> org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:56)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
>   at 
> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
>   at 
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
>   at 
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403)
>   at 
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

1.1 release progress

[Damian Guy]

Hi,

Just a quick updated on the 1.1 release. Feature freeze is on the 30th of
January, so please ensure any major features have been merged and that any
minor features have PRs by this time.

We currently have 57 issues in progress and 117 todo:
https://issues.apache.org/jira/projects/KAFKA/versions/12339769
If people can start moving stuff out of 1.1 that would be great, otherwise
anything that is todo will be moved to the next release.

Thanks,
Damian

Re: 1.1 KIPs

[Damian Guy]

Thanks Vahid, i've updated the release plan

On Wed, 24 Jan 2018 at 13:31 Vahid S Hashemian 
wrote:

> Hi Damian,
>
> Could you please add KIP-229 to the list? It was approved earlier this
> week
> https://www.mail-archive.com/dev@kafka.apache.org/msg84851.html
>
> Thanks for running the release.
> --Vahid
>
>
>
>
> From:   Damian Guy 
> To: dev@kafka.apache.org
> Date:   01/24/2018 01:20 PM
> Subject:Re: 1.1 KIPs
>
>
>
> Hi,
>
> The KIP deadline has passed and i've updated the release plan:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D75957546&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=TVUESwz92IFTGWgOw60U-bc5Fih9bzGtCRahKUZlSTc&s=ZsBKyGkffkjDH0yHsVpxNP-_qhSR681YSL48Bi5cTBs&e=
>
> If there is anything i've missed please let me know.
>
> The feature freeze deadline is January 30th. At this point i'll cut the
> branch for 1.1. So please make sure any major features have been committed
> by then.
>
> Thanks,
> Damian
>
> On Tue, 23 Jan 2018 at 15:29 Damian Guy  wrote:
>
> > Hi,
> >
> > Today is the KIP deadline. Tomorrow I'll update the release page with
> any
> > KIPS that have recently been voted on and accepted according to the
> process
> > that can be found here:
> >
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_Kafka-2BImprovement-2BProposals-23KafkaImprovementProposals-2DProcess&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=TVUESwz92IFTGWgOw60U-bc5Fih9bzGtCRahKUZlSTc&s=JEIfWpaEOFJ8NmyLHUs7NLlcgPgsNEdM1Iy9Y5Tr3Vs&e=
>
> >
> > Thanks,
> > Damian
> >
> > On Thu, 18 Jan 2018 at 12:47 Damian Guy  wrote:
> >
> >> Gentle reminder that KIP deadline is just 5 days away. If there is
> >> anything that wants to be in 1.1 and hasn't been voted on yet, now is
> the
> >> time!
> >>
> >> On Thu, 18 Jan 2018 at 08:49 Damian Guy  wrote:
> >>
> >>> Hi Xavier,
> >>> I'll add it to the plan.
> >>>
> >>> Thanks,
> >>> Damian
> >>>
> >>> On Tue, 16 Jan 2018 at 19:04 Xavier Léauté 
> wrote:
> >>>
> >>>> Hi Damian, I believe the list should also include KAFKA-5886 (KIP-91)
> >>>> which
> >>>> was voted for 1.0 but wasn't ready to be merged in time.
> >>>>
> >>>> On Tue, Jan 16, 2018 at 5:13 AM Damian Guy 
> >>>> wrote:
> >>>>
> >>>> > Hi,
> >>>> >
> >>>> > This is a reminder that we have one week left until the KIP
> deadline
> >>>> of Jan
> >>>> > 23. There are still some KIPs that are under discussion and/or
> being
> >>>> voted
> >>>> > on. Please keep in mind that the voting needs to be complete before
> >>>> the
> >>>> > deadline for the KIP to be added to the release.
> >>>> >
> >>>> >
> >>>>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D75957546&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=TVUESwz92IFTGWgOw60U-bc5Fih9bzGtCRahKUZlSTc&s=ZsBKyGkffkjDH0yHsVpxNP-_qhSR681YSL48Bi5cTBs&e=
>
> >>>> >
> >>>> > Thanks,
> >>>> > Damian
> >>>> >
> >>>>
> >>>
>
>
>
>
>

Re: 1.1 KIPs

[Damian Guy]

Hi,

The KIP deadline has passed and i've updated the release plan:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
If there is anything i've missed please let me know.

The feature freeze deadline is January 30th. At this point i'll cut the
branch for 1.1. So please make sure any major features have been committed
by then.

Thanks,
Damian

On Tue, 23 Jan 2018 at 15:29 Damian Guy  wrote:

> Hi,
>
> Today is the KIP deadline. Tomorrow I'll update the release page with any
> KIPS that have recently been voted on and accepted according to the process
> that can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process
>
> Thanks,
> Damian
>
> On Thu, 18 Jan 2018 at 12:47 Damian Guy  wrote:
>
>> Gentle reminder that KIP deadline is just 5 days away. If there is
>> anything that wants to be in 1.1 and hasn't been voted on yet, now is the
>> time!
>>
>> On Thu, 18 Jan 2018 at 08:49 Damian Guy  wrote:
>>
>>> Hi Xavier,
>>> I'll add it to the plan.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Tue, 16 Jan 2018 at 19:04 Xavier Léauté  wrote:
>>>
>>>> Hi Damian, I believe the list should also include KAFKA-5886 (KIP-91)
>>>> which
>>>> was voted for 1.0 but wasn't ready to be merged in time.
>>>>
>>>> On Tue, Jan 16, 2018 at 5:13 AM Damian Guy 
>>>> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > This is a reminder that we have one week left until the KIP deadline
>>>> of Jan
>>>> > 23. There are still some KIPs that are under discussion and/or being
>>>> voted
>>>> > on. Please keep in mind that the voting needs to be complete before
>>>> the
>>>> > deadline for the KIP to be added to the release.
>>>> >
>>>> >
>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
>>>> >
>>>> > Thanks,
>>>> > Damian
>>>> >
>>>>
>>>

Re: 1.1 KIPs

[Damian Guy]

Hi,

Today is the KIP deadline. Tomorrow I'll update the release page with any
KIPS that have recently been voted on and accepted according to the process
that can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process

Thanks,
Damian

On Thu, 18 Jan 2018 at 12:47 Damian Guy  wrote:

> Gentle reminder that KIP deadline is just 5 days away. If there is
> anything that wants to be in 1.1 and hasn't been voted on yet, now is the
> time!
>
> On Thu, 18 Jan 2018 at 08:49 Damian Guy  wrote:
>
>> Hi Xavier,
>> I'll add it to the plan.
>>
>> Thanks,
>> Damian
>>
>> On Tue, 16 Jan 2018 at 19:04 Xavier Léauté  wrote:
>>
>>> Hi Damian, I believe the list should also include KAFKA-5886 (KIP-91)
>>> which
>>> was voted for 1.0 but wasn't ready to be merged in time.
>>>
>>> On Tue, Jan 16, 2018 at 5:13 AM Damian Guy  wrote:
>>>
>>> > Hi,
>>> >
>>> > This is a reminder that we have one week left until the KIP deadline
>>> of Jan
>>> > 23. There are still some KIPs that are under discussion and/or being
>>> voted
>>> > on. Please keep in mind that the voting needs to be complete before the
>>> > deadline for the KIP to be added to the release.
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
>>> >
>>> > Thanks,
>>> > Damian
>>> >
>>>
>>

Re: [VOTE] KIP-208: Add SSL support to Kafka Connect REST interface

[Damian Guy]

Thanks Jakub!

+1 (binding)

On Thu, 18 Jan 2018 at 23:49 Jakub Scholz  wrote:

> Hi all,
>
> We still need at least 2 more binding +1s. I think that the PR (
> https://github.com/apache/kafka/pull/4429) is shaping good. If we get the
> votes, we should be able to make the 1.1.0 release.
>
> Thanks & Regards
> Jakub
>
> On Fri, Jan 5, 2018 at 4:30 AM, Ewen Cheslack-Postava 
> wrote:
>
> > Jakub,
> >
> > I left a few comments in the discuss thread, but I'll also reply here
> just
> > to bump the VOTE thread's visibility. I would like to resolve the few
> > comments I left, but I am effectively +1 on this, the comments I left
> were
> > mainly details.
> >
> > Committers that could help with the necessary votes would probably be
> Gwen
> > and Jason (but others more than welcome to help out too :)
> >
> > -Ewen
> >
> > On Mon, Nov 6, 2017 at 1:52 AM, Jakub Scholz  wrote:
> >
> > > Hi all,
> > >
> > > Just a reminder that htis is still up for vote. I think this is
> important
> > > featrue which would deserve your votes.
> > >
> > > Regards
> > > Jakub
> > >
> > > On Mon, Oct 30, 2017 at 9:24 PM, Jakub Scholz  wrote:
> > >
> > > > Hi,
> > > >
> > > > It seems there are no more comments for this KIP, so I would like to
> > > start
> > > > the voting .
> > > >
> > > > For more details about the KIP-208 go to
> > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
> > > >  > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface>*
> > > >
> > > > Thanks & Regards
> > > > Jakub
> > > >
> > >
> >
>

Re: 1.1 KIPs

[Damian Guy]

Gentle reminder that KIP deadline is just 5 days away. If there is anything
that wants to be in 1.1 and hasn't been voted on yet, now is the time!

On Thu, 18 Jan 2018 at 08:49 Damian Guy  wrote:

> Hi Xavier,
> I'll add it to the plan.
>
> Thanks,
> Damian
>
> On Tue, 16 Jan 2018 at 19:04 Xavier Léauté  wrote:
>
>> Hi Damian, I believe the list should also include KAFKA-5886 (KIP-91)
>> which
>> was voted for 1.0 but wasn't ready to be merged in time.
>>
>> On Tue, Jan 16, 2018 at 5:13 AM Damian Guy  wrote:
>>
>> > Hi,
>> >
>> > This is a reminder that we have one week left until the KIP deadline of
>> Jan
>> > 23. There are still some KIPs that are under discussion and/or being
>> voted
>> > on. Please keep in mind that the voting needs to be complete before the
>> > deadline for the KIP to be added to the release.
>> >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
>> >
>> > Thanks,
>> > Damian
>> >
>>
>

Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

[Damian Guy]

+1

On Thu, 18 Jan 2018 at 15:14 Bill Bejeck  wrote:

> Thanks for the KIP.
>
> +1
>
> -Bill
>
> On Wed, Jan 17, 2018 at 9:09 PM, Matthias J. Sax 
> wrote:
>
> > Hi,
> >
> > I would like to start the vote for KIP-247:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 247%3A+Add+public+test+utils+for+Kafka+Streams
> >
> >
> > -Matthias
> >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

[Damian Guy]

Congratulations Rajini!

On Thu, 18 Jan 2018 at 00:57 Hu Xi  wrote:

> Congratulations, Rajini Sivaram.  Very well deserved!
>
>
> 
> 发件人: Konstantine Karantasis 
> 发送时间: 2018年1月18日 6:23
> 收件人: dev@kafka.apache.org
> 抄送: us...@kafka.apache.org
> 主题: Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram
>
> Congrats Rajini!
>
> -Konstantine
>
> On Wed, Jan 17, 2018 at 2:18 PM, Becket Qin  wrote:
>
> > Congratulations, Rajini!
> >
> > On Wed, Jan 17, 2018 at 1:52 PM, Ismael Juma  wrote:
> >
> > > Congratulations Rajini!
> > >
> > > On 17 Jan 2018 10:49 am, "Gwen Shapira"  wrote:
> > >
> > > Dear Kafka Developers, Users and Fans,
> > >
> > > Rajini Sivaram became a committer in April 2017.  Since then, she
> > remained
> > > active in the community and contributed major patches, reviews and KIP
> > > discussions. I am glad to announce that Rajini is now a member of the
> > > Apache Kafka PMC.
> > >
> > > Congratulations, Rajini and looking forward to your future
> contributions.
> > >
> > > Gwen, on behalf of Apache Kafka PMC
> > >
> >
>

Re: 1.1 KIPs

[Damian Guy]

Hi Xavier,
I'll add it to the plan.

Thanks,
Damian

On Tue, 16 Jan 2018 at 19:04 Xavier Léauté  wrote:

> Hi Damian, I believe the list should also include KAFKA-5886 (KIP-91) which
> was voted for 1.0 but wasn't ready to be merged in time.
>
> On Tue, Jan 16, 2018 at 5:13 AM Damian Guy  wrote:
>
> > Hi,
> >
> > This is a reminder that we have one week left until the KIP deadline of
> Jan
> > 23. There are still some KIPs that are under discussion and/or being
> voted
> > on. Please keep in mind that the voting needs to be complete before the
> > deadline for the KIP to be added to the release.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
> >
> > Thanks,
> > Damian
> >
>

Re: Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor

[Damian Guy]

Thanks for the KIP!  +1

On Sat, 13 Jan 2018 at 12:30 Boyang Chen  wrote:

> Hey Matt and Guozhang,
>
>
> I have already updated the pull request:
> https://github.com/apache/kafka/pull/4354
>
> and the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-245%3A+Use+Properties+instead+of+StreamsConfig+in+KafkaStreams+constructor
>
>
> to reflect the change proposed by Guozhang(adding a 4th constructor)
>
> Let me know your thoughts!
>
>
> Best,
>
> Boyang
>
>
> 
> From: Boyang Chen 
> Sent: Saturday, January 13, 2018 9:37 AM
> To: Matthias J. Sax
> Subject: Re: Vote for KIP-245: Use Properties instead of StreamsConfig in
> KafkaStreams constructor
>
>
> Sounds good, will do. However I don't receive the +1 emails, interesting...
>
>
> 
> From: Matthias J. Sax 
> Sent: Saturday, January 13, 2018 9:32 AM
> To: Boyang Chen
> Subject: Re: Vote for KIP-245: Use Properties instead of StreamsConfig in
> KafkaStreams constructor
>
> Guozhang left a comment about having a 4th overload. I agree that we
> should add this 4th overload.
>
> Please update the KIP accordingly and follow up on the mailing list
> thread. Than we can vote it through.
>
> Thx.
>
> -Matthias
>
> On 1/12/18 4:48 PM, Boyang Chen wrote:
> > Hey Matt,
> >
> >
> > I haven't received any approval/veto on this KIP. Everything is ready
> > but only needs one approval. Any step I should take?
> >
> > Thanks for the help!
> >
> > Boyang
> >
> >
> >
> > 
> > *From:* Matthias J. Sax 
> > *Sent:* Saturday, January 13, 2018 3:47 AM
> > *To:* dev@kafka.apache.org
> > *Subject:* Re: Vote for KIP-245: Use Properties instead of StreamsConfig
> > in KafkaStreams constructor
> >
> > Boyang,
> >
> > what is the status of this KIP? The release plan for 1.1 was just
> > announced and we like to get this KIP into the release.
> >
> > Thx.
> >
> >
> > -Matthias
> >
> > On 1/2/18 11:18 AM, Guozhang Wang wrote:
> >> Boyang,
> >>
> >> Thanks for the proposed change, the wiki page lgtm. One minor comment
> >> otherwise I'm +1:
> >>
> >> For the new API, we now also have a constructor that accepts both a
> >> clientSupplier and a Time, so we should consider having four overloads
> in
> >> total:
> >>
> >>
> >> // New API (using Properties)
> >> public KafkaStreams(final Topology, final Properties props)
> >> public KafkaStreams(final Topology, final Properties props, final Time
> time)
> >> public KafkaStreams(final Topology, final Properties props, final
> >> KafkaClientSupplier
> >> clientSupplier)
> >> public KafkaStreams(final Topology, final Properties props, final
> >> KafkaClientSupplier
> >> clientSupplier, final Time time)
> >>
> >> Guozhang
> >>
> >> On Tue, Dec 26, 2017 at 7:26 PM, Satish Duggana <
> satish.dugg...@gmail.com>
> >> wrote:
> >>
> >>> Thanks for the KIP, +1 from me.
> >>>
> >>> On Wed, Dec 27, 2017 at 7:42 AM, Bill Bejeck 
> wrote:
> >>>
>  Thanks for the KIP.  +1 for me.
> 
>  On Tue, Dec 26, 2017 at 6:22 PM Ted Yu  wrote:
> 
> > +1 from me as well.
> >
> > On Tue, Dec 26, 2017 at 10:41 AM, Matthias J. Sax <
> >>> matth...@confluent.io
> >
> > wrote:
> >
> >> Thanks for the KIP Boyang!
> >>
> >> I don't have any further comments.
> >>
> >> +1 from me.
> >>
> >> @Ted: This is a rather simple KIP, thus, skipping the DISCUSS thread
> >> seems ok to me.
> >>
> >>
> >>
> >> -Matthias
> >>
> >>
> >> @Boyang: it's recommended to use this format for the subject
> >>
> >> "[VOTE] KIP-245: ..."
> >>
> >> Same for DISCUSS threads. People are used to those headlines and
> they
> >> pay more attention than. For this KIP, just leave it as it though.
> >>> For
> >> future reference only
> >> .
> >>
> >>
> >> On 12/26/17 4:55 AM, Ted Yu wrote:
> >>> Normally a DISCUSS thread precedes VOTE thread so that people have
> > ample
> >> time examining the proposal.
> >>>  Original message From: Boyang Chen <
> > bche...@outlook.com>
> >> Date: 12/26/17  1:22 AM  (GMT-07:00) To: dev@kafka.apache.org
> >>> Subject:
> >> Vote for KIP-245: Use Properties instead of StreamsConfig in
>  KafkaStreams
> >> constructor
> >>> Hi there,
> >>>
> >>> I'm Boyang who is a newbie contributor to Kafka. I would like to
>  start
> > a
> >> vote for the KIP-245:
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>
> > 245%3A+Use+Properties+instead+of+StreamsConfig+in+
>  KafkaStreams+constructor
> >>>
> >>>
> >>> This is linked with JIRA: https://issues.apache.org/
> > issues.apache.org 
> issues.apache.org
> issues.apache.org
> Apache currently hosts two different issue tracking systems, Bugzilla and
> Jira. To find out how to rep

1.1 KIPs

[Damian Guy]

Hi,

This is a reminder that we have one week left until the KIP deadline of Jan
23. There are still some KIPs that are under discussion and/or being voted
on. Please keep in mind that the voting needs to be complete before the
deadline for the KIP to be added to the release.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546

Thanks,
Damian

Re: [ANNOUNCE] New committer: Matthias J. Sax

[Damian Guy]

Congrats Matthias! Very well deserved
On Sat, 13 Jan 2018 at 00:57, Bill Bejeck  wrote:

> Congratulations Matthias!
>
> -Bill
>
> On Fri, Jan 12, 2018 at 7:43 PM, Damian Guy  wrote:
>
> > Can't think of anyone me deserving! Congratulations Matthias!
> > On Sat, 13 Jan 2018 at 00:17, Ismael Juma  wrote:
> >
> > > Congratulations Matthias!
> > >
> > > On 12 Jan 2018 10:59 pm, "Guozhang Wang"  wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our
> > > > newest Kafka committer.
> > > >
> > > > Matthias has made tremendous contributions to Kafka Streams API since
> > > early
> > > > 2016. His footprint has been all over the places in Streams: in the
> > past
> > > > two years he has been the main driver on improving the join semantics
> > > > inside Streams DSL, summarizing all their shortcomings and bridging
> the
> > > > gaps; he has also been largely working on the exactly-once semantics
> of
> > > > Streams by leveraging on the transaction messaging feature in 0.11.0.
> > In
> > > > addition, Matthias have been very active in community activity that
> > goes
> > > > beyond mailing list: he's getting the close to 1000 up votes and 100
> > > > helpful flags on SO for answering almost all questions about Kafka
> > > Streams.
> > > >
> > > > Thank you for your contribution and welcome to Apache Kafka,
> Matthias!
> > > >
> > > >
> > > >
> > > > Guozhang, on behalf of the Apache Kafka PMC
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Matthias J. Sax

[Damian Guy]

Can't think of anyone me deserving! Congratulations Matthias!
On Sat, 13 Jan 2018 at 00:17, Ismael Juma  wrote:

> Congratulations Matthias!
>
> On 12 Jan 2018 10:59 pm, "Guozhang Wang"  wrote:
>
> > Hello everyone,
> >
> > The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our
> > newest Kafka committer.
> >
> > Matthias has made tremendous contributions to Kafka Streams API since
> early
> > 2016. His footprint has been all over the places in Streams: in the past
> > two years he has been the main driver on improving the join semantics
> > inside Streams DSL, summarizing all their shortcomings and bridging the
> > gaps; he has also been largely working on the exactly-once semantics of
> > Streams by leveraging on the transaction messaging feature in 0.11.0. In
> > addition, Matthias have been very active in community activity that goes
> > beyond mailing list: he's getting the close to 1000 up votes and 100
> > helpful flags on SO for answering almost all questions about Kafka
> Streams.
> >
> > Thank you for your contribution and welcome to Apache Kafka, Matthias!
> >
> >
> >
> > Guozhang, on behalf of the Apache Kafka PMC
> >
>

[DISCUSS] February Release Plan

[Damian Guy]

Hi all,

I would like to volunteer to be the release manager
for our next time-based feature release (v1.1.0). See
https://cwiki.apache.org/
confluence/display/KAFKA/Time+Based+Release+Plan if you missed
previous communication on time-based releases or need a reminder.

I put together a draft release plan
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546 with
February 2018 as the release month
(as previously agreed) and a list of KIPs that have already been voted:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546

Some important (and fast approaching) dates:
KIP Freeze: January 23, 2018
Feature Freeze: January 30, 2018

Thanks,
Damian

[jira] [Resolved] (KAFKA-6412) Improve synchronization in CachingKeyValueStore methods

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-6412.
---
   Resolution: Fixed
Fix Version/s: 1.1.0

Issue resolved by pull request 4372
[https://github.com/apache/kafka/pull/4372]

> Improve synchronization in CachingKeyValueStore methods
> ---
>
> Key: KAFKA-6412
> URL: https://issues.apache.org/jira/browse/KAFKA-6412
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Ted Yu
> Fix For: 1.1.0
>
> Attachments: 6412-jmh.v1.txt, k-6412.v1.txt
>
>
> Currently CachingKeyValueStore methods are synchronized at method level.
> It seems we can use read lock for getter and write lock for put / delete 
> methods.
> For getInternal(), if the underlying thread is streamThread, the 
> getInternal() may trigger eviction. This can be handled by obtaining write 
> lock at the beginning of the method for streamThread.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-233: Simplify StreamsBuilder#addGlobalStore

[Damian Guy]

+1

On Wed, 3 Jan 2018 at 03:49 Matthias J. Sax  wrote:

> It must be two different names, as we add two processor to the topology:
> a source processor that only read the data from a topic, and the actual
> processor that maintains the global table.
>
>
> -Matthias
>
> On 1/2/18 7:14 PM, Ewen Cheslack-Postava wrote:
> > +1 binding, seems like a nice simplification.
> >
> > Regarding the source and processor name, do they actually need to be
> unique
> > or could they use the same value? Since these use incrementing integers,
> it
> > could be nice for debuggability/understanding to have them use the same
> > name if possible instead of generating 2 separate names.
> >
> > -Ewen
> >
> > On Tue, Jan 2, 2018 at 9:12 AM, Guozhang Wang 
> wrote:
> >
> >> On a side note, could you update the "compatibility and upgrade"
> section,
> >> that when users start to make code changes to leverage the new API, what
> >> kind of upgrade executions they'd need to do? I feel they need to rename
> >> topics / etc.
> >>
> >> On Tue, Jan 2, 2018 at 9:10 AM, Guozhang Wang 
> wrote:
> >>
> >>> +1, thanks!
> >>>
> >>> On Wed, Dec 27, 2017 at 6:01 PM, Ted Yu  wrote:
> >>>
>  +1
> 
>  On Wed, Dec 27, 2017 at 12:15 PM, Bill Bejeck 
> >> wrote:
> 
> > +1
> >
> > On Wed, Dec 27, 2017 at 3:07 PM, Matthias J. Sax <
> >> matth...@confluent.io
> >
> > wrote:
> >
> >> +1
> >>
> >> On 12/26/17 9:00 PM, Panuwat Anawatmongkhon wrote:
> >>> Hi all,
> >>> I would like to start the vote thread.
> >>> This is link for the kip.
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-233%
>  3A+Simplify+
> >> StreamsBuilder%23addGlobalStore
> >>>
> >>> Cheers
> >>>
> >>
> >>
> >
> 
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

[Damian Guy]

+1

On Wed, 20 Dec 2017 at 21:09 Ted Yu  wrote:

> Ping for more (binding) votes.
>
> The pull request is ready.
>
> On Fri, Dec 15, 2017 at 12:57 PM, Guozhang Wang 
> wrote:
>
> > +1 (binding), thanks!
> >
> > On Fri, Dec 15, 2017 at 11:56 AM, Ted Yu  wrote:
> >
> > > Hi,
> > > Here is the discussion thread:
> > >
> > > http://search-hadoop.com/m/Kafka/uyzND12QnH514pPO9?subj=
> > > Re+DISCUSS+KIP+239+Add+queryableStoreName+to+GlobalKTable
> > >
> > > Please vote on this KIP.
> > >
> > > Thanks
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

[Damian Guy]

+1

On Tue, 19 Dec 2017 at 19:13 Bill Bejeck  wrote:

> +1
>
> On Tue, Dec 19, 2017 at 1:21 PM, Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 12/19/17 9:23 AM, Guozhang Wang wrote:
> > > Hello,
> > >
> > > I'm calling for a re-vote on the additional API changes in this KIP.
> > While
> > > working on the implementation I found it's better to add the following
> > > functions for programmability:
> > >
> > > * Overloaded KafkaStreams constructor to allow overriding the `Time`
> > object.
> > > * Overloaded AbstractConfig#originalsWithPrefix() to allow specifying
> > > whether or not to strip the prefix in the key of the returned map.
> > >
> > > The corresponding wiki page (
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier)
> > > has been updated.
> > >
> > > Gwen / Damian, could you take a quick look at the updated wiki page and
> > > re-cast your vote?
> > >
> > >
> > > Guozhang
> > >
> > >
> > >
> > >
> > > On Mon, Nov 20, 2017 at 11:12 AM, Guozhang Wang 
> > wrote:
> > >
> > >> +1 from myself as well.
> > >>
> > >> I'm closing this KIP as accepted with 3 binding votes (Gwen, Damian,
> me)
> > >> and 3 non-binding votes (Colin, Ted, Matthias).
> > >>
> > >>
> > >> Guozhang
> > >>
> > >> On Mon, Nov 20, 2017 at 9:56 AM, Damian Guy 
> > wrote:
> > >>
> > >>> +1
> > >>>
> > >>> On Mon, 20 Nov 2017 at 17:52 Gwen Shapira  wrote:
> > >>>
> > >>>> +1
> > >>>>
> > >>>> Make sense. We have a supplier for every other client type :)
> > >>>>
> > >>>> On Fri, Nov 17, 2017 at 1:33 PM Matthias J. Sax <
> > matth...@confluent.io>
> > >>>> wrote:
> > >>>>
> > >>>>> +1
> > >>>>>
> > >>>>> On 11/17/17 9:35 AM, Ted Yu wrote:
> > >>>>>> +1
> > >>>>>>
> > >>>>>> On Fri, Nov 17, 2017 at 9:34 AM, Bill Bejeck 
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> +1
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Bill
> > >>>>>>>
> > >>>>>>> On Fri, Nov 17, 2017 at 12:13 PM, Colin McCabe <
> cmcc...@apache.org
> > >>>>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>>> +1 (non-binding)
> > >>>>>>>>
> > >>>>>>>> Colin
> > >>>>>>>>
> > >>>>>>>> On Tue, Nov 14, 2017, at 10:02, Guozhang Wang wrote:
> > >>>>>>>>> Hello folks,
> > >>>>>>>>>
> > >>>>>>>>> I have filed a new KIP on adding AdminClient into Streams for
> > >>>> internal
> > >>>>>>>>> topic management.
> > >>>>>>>>>
> > >>>>>>>>> Please review and cast your vote on this thread.
> > >>>>>>>>>
> > >>>>>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>>>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
> > >>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>>>> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier>*
> > >>>>>>>>>
> > >>>>>>>>> The discussion is in another thread so if you have detailed
> > >>>> questions
> > >>>>>>>>> please chime in there.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> -- Guozhang
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> > >
> >
> >
>

[jira] [Created] (KAFKA-6360) RocksDB segments not removed when store is closed causes re-initialization to fail

[Damian Guy (JIRA)]

Damian Guy created KAFKA-6360:
-

 Summary: RocksDB segments not removed when store is closed causes 
re-initialization to fail
 Key: KAFKA-6360
 URL: https://issues.apache.org/jira/browse/KAFKA-6360
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.1.0
Reporter: Damian Guy
Assignee: Damian Guy
Priority: Blocker
 Fix For: 1.1.0


When a store is re-initialized it is first closed, before it is opened again. 
When this happens the segments in the {{Segments}} class are closed, but they 
are not removed from the list of segments. So when the store is re-initialized 
the old closed segments are used. This results in:
{code}
[2017-12-13 09:29:32,037] ERROR [streams-saak-test-client-StreamThread-3] task 
[1_3] Failed to flush state store KSTREAM-AGGREGATE-STATE-STORE-24:  
(org.apache.kafka.streams.processor.internals.ProcessorStateManager)
org.apache.kafka.streams.errors.InvalidStateStoreException: Store 
KSTREAM-AGGREGATE-STATE-STORE-24.151308000 is currently closed
at 
org.apache.kafka.streams.state.internals.RocksDBStore.validateStoreOpen(RocksDBStore.java:241)
at 
org.apache.kafka.streams.state.internals.RocksDBStore.put(RocksDBStore.java:289)
at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:102)
at 
org.apache.kafka.streams.state.internals.RocksDBSessionStore.put(RocksDBSessionStore.java:122)
at 
org.apache.kafka.streams.state.internals.ChangeLoggingSessionBytesStore.put(ChangeLoggingSessionBytesStore.java:78)
at 
org.apache.kafka.streams.state.internals.ChangeLoggingSessionBytesStore.put(ChangeLoggingSessionBytesStore.java:33)
at 
org.apache.kafka.streams.state.internals.CachingSessionStore.putAndMaybeForward(CachingSessionStore.java:179)
at 
org.apache.kafka.streams.state.internals.CachingSessionStore.access$000(CachingSessionStore.java:38)
at 
org.apache.kafka.streams.state.internals.CachingSessionStore$1.apply(CachingSessionStore.java:88)
at 
org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:142)
at 
org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:100)
at 
org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:127)
at 
org.apache.kafka.streams.state.internals.CachingSessionStore.flush(CachingSessionStore.java:196)
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)