[DISCUSS] KIPs

[Jay Kreps]

The idea of KIPs came up in a previous discussion but there was no real
crisp definition of what they were. Here is an attempt at defining a
process:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

The trick here is to have something light-weight enough that it isn't a
hassle for small changes, but enough so that changes get the eyeballs of
the committers and heavy users.

Thoughts?

-Jay

Re: [DISCUSS] KIPs

[Gwen Shapira]

Sorry for late response, Magnus. See my comments inline:

On Fri, Jan 23, 2015 at 7:31 AM, Magnus Edenhill  wrote:
> Wouldn't it make sense to move away from these rich binary broker
> descriptors ({ host, port, proto })
> (which require protocol churning on change), and simply use URIs instead?

We use different representations in different places, so I'm not sure
which one you mean...

Our clients will use host:port and --security.protocol in the
configuration to specify the protocol (this is to make it easier to
ensure one protocol per client)
ZK registration is in JSON
Internal objects are binary, since parsing the URI over and over will be a PITA
Wire protocol doesn't include a security protocol (since we don't
negotiate it), so its still host+port, as it always was.

>
> E.g.:
>   kafka://[:port]/ <-- cleantext proto on standard port 9092
>   kafkas://[:port] <-- SSL enveloped proto on standard port 9093
>   kafkas://@[:port]/  <-- SSL enveloped, with user
> authentication ..
>   kafkafuturetech://.../#opts <-- six months from now.
>
> Trailing #fragment_ids could be used to hint the client on protocol
> versions, supported authentications, etc.
>
> This also makes error reporting more meaningful on the client, e.g compare:
>   "Unsupported protocol 19 on broker foo:1234"
>  to
>   "Unsupported protocol kafkafturetech on broker foo:1234"

I agree that the second error is more readable, but I'm not sure why
you think its currently unfeasible on clients?

>
> A positive side effect would be a more generalized topic addressing in
> clients:
>kafkacat kafka:///mytopic/3?offset=end  <-- tail partition 3
> of mytopic

Clients can pretty much do what they want, right? As long as they call
Kafka with the right configuration, its up to them to decide how to
accept arguments.

> Just an idea,
> Magnus
>
>
> 2015-01-23 5:43 GMT+01:00 Jun Rao :
>
>> Reviewed the latest patch in KAFKA-1809 :).
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira 
>> wrote:
>>
>> > Thanks for validating our ideas. Updated the KIP with the workflow.
>> >
>> > Now if you can nudge Jun to review the latest patch... ;)
>> >
>> >
>> > On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps  wrote:
>> > > Oh yeah I think that is better, I hadn't thought of that approach! Any
>> > way
>> > > you could describe the usage in the KIP, just for completeness?
>> > >
>> > > -Jay
>> > >
>> > > On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira 
>> > > wrote:
>> > >
>> > >> I think what you described was the original design, so no wonder you
>> > >> are confused :)
>> > >>
>> > >> Following suggestions from Jun, I changed it a bit. The current model
>> > is:
>> > >>
>> > >> - Clients (producers and consumers) need to know about the broker
>> > >> ports in advance. They don't need to know about all brokers, but they
>> > >> need to know at least one host:port pair that speaks the protocol they
>> > >> want to use. The change is that all host:port pairs in broker.list
>> > >> must be of the same protocol and match the security.protocol
>> > >> configuration parameter.
>> > >>
>> > >> - Client uses security.protocol configuration parameter to open a
>> > >> connection to one of the brokers and sends the good old
>> > >> MetadataRequest. The broker knows which port it got the connection on,
>> > >> therefore it knows which security protocol is expected (it needs to
>> > >> use the same protocol to accept the connection and respond), and
>> > >> therefore it can send a response that contains only the host:port
>> > >> pairs that are relevant to that protocol.
>> > >>
>> > >> - From the client side the MetadataResponse did not change - it
>> > >> contains a list of brokerId,host,port that the client can connect to.
>> > >> The fact that all those broker endpoints were chosen out of a larger
>> > >> collection to match the right protocol is irrelevant for the client.
>> > >>
>> > >> I really like the new design since it preserves a lot of the same
>> > >> configurations and APIs.
>> > >>
>> > >> Thoughts?
>> > >>
>> > >> Gwen
>> > >>
>> > >> On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps 
>> wrote:
>> > >> > I think I am still confused. In addition to the
>> UpdateMetadataRequest
>> > >> don't
>> > >> > we have to change the MetadataResponse so that it's possible for
>> > clients
>> > >> to
>> > >> > discover the new ports? Or is that a second phase? I was imagining
>> it
>> > >> > worked by basically allowing the brokers to advertise multiple
>> ports,
>> > one
>> > >> > per security type, and then in the client you configure a protocol
>> > which
>> > >> > will implicitly choose the port from the options returned in
>> metadata
>> > to
>> > >> > you...
>> > >> >
>> > >> > Likewise in the ConsumerMetadataResponse we are currently giving
>> back
>> > >> full
>> > >> > broker information. I think we would have two options here: either
>> > change
>> > >> > the broker information included in that response to match the
>> > >> > metada

Re: [DISCUSS] KIPs

[Joel Koshy]

One amendment I would like to bring up for consideration wrt the KIP
process (before we formally include it in our by-laws) is to not
restrict the votes to be a lazy majority of the PMC, and to instead
make it a lazy majority of committers.

Our current requirement for code changes per our by-laws are +1 from a
committer (who is not the contributor) followed by lazy approval. I
think a lazy majority vote for more significant code changes (i.e., a
KIP) should be sufficient.

Any objection to this?

On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> Great! Sounds like everyone is on the same page
> 
>- I created a template page to make things easier. If you do Tools->Copy
>on this page you can just fill in the italic portions with your details.
>- I retrofitted KIP-1 to match this formatting
>- I added the metadata section people asked for (a link to the
>discussion, the JIRA, and the current status). Let's make sure we remember
>to update the current status as things are figured out.
>- Let's keep the discussion on the mailing list rather than on the wiki
>pages. It makes sense to do one or the other so all the comments are in one
>place and I think prior experience is that the wiki comments are the worse
>way.
> 
> I think it would be great do KIPs for some of the in-flight items folks
> mentioned.
> 
> -Jay
> 
> On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira  wrote:
> 
> > +1
> >
> > Will be happy to provide a KIP for the multiple-listeners patch.
> >
> > Gwen
> >
> > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein  wrote:
> > > +1 to everything we have been saying and where this (has settled to)/(is
> > > settling to).
> > >
> > > I am sure other folks have some more feedback and think we should try to
> > > keep this discussion going if need be. I am also a firm believer of form
> > > following function so kicking the tires some to flesh out the details of
> > > this and have some organic growth with the process will be healthy and
> > > beneficial to the community.
> > >
> > > For my part, what I will do is open a few KIP based on some of the work I
> > > have been involved with for 0.8.3. Off the top of my head this would
> > > include 1) changes to re-assignment of partitions 2) kafka cli 3) global
> > > configs 4) security white list black list by ip 5) SSL 6) We probably
> > will
> > > have lots of Security related KIPs and should treat them all individually
> > > when the time is appropriate 7) Kafka on Mesos.
> > >
> > > If someone else wants to jump in to start getting some of the security
> > KIP
> > > that we are going to have in 0.8.3 I think that would be great (e.g.
> > > Multiple Listeners for Kafka Brokers). There are also a few other
> > tickets I
> > > can think of that are important to have in the code in 0.8.3 that should
> > > have KIP also that I haven't really been involved in. I will take a few
> > > minutes and go through JIRA (one I can think of like auto assign id that
> > is
> > > already committed I think) and ask for a KIP if appropriate or if I feel
> > > that I can write it up (both from a time and understanding perspective)
> > do
> > > so.
> > >
> > > long story short, I encourage folks to start moving ahead with the KIP
> > for
> > > 0.8.3 as how we operate. any objections?
> > >
> > > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
> > wrote:
> > >
> > >> +1 on the idea, and we could mutually link the KIP wiki page with the
> > the
> > >> created JIRA ticket (i.e. include the JIRA number on the page and the
> > KIP
> > >> url on the ticket description).
> > >>
> > >> Regarding the KIP process, probably we do not need two phase
> > communication
> > >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
> > clear
> > >> while people discuss about that.
> > >>
> > >> About who should trigger the process, I think the only involved people
> > >> would be 1) when the patch is submitted / or even the ticket is created,
> > >> the assignee could choose to start the KIP process if she thought it is
> > >> necessary; 2) the reviewer of the patch can also suggest starting KIP
> > >> discussions.
> > >>
> > >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
> > >> wrote:
> > >>
> > >> > +1 to Ewen's suggestions: Deprecation, status and version.
> > >> >
> > >> > Perhaps add the JIRA where the KIP was implemented to the metadata.
> > >> > This will help tie things together.
> > >> >
> > >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
> > >> >  wrote:
> > >> > > I think adding a section about deprecation would be helpful. A good
> > >> > > fraction of the time I would expect the goal of a KIP is to fix or
> > >> > replace
> > >> > > older functionality that needs continued support for compatibility,
> > but
> > >> > > should eventually be phased out. This helps Kafka devs understand
> > how
> > >> > long
> > >> > > they'll end up supporting multiple versions of features and helps
> > users
> > >> > > understand

Re: [DISCUSS] KIPs

[Jay Kreps]

None on my part.

-Jay

On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy  wrote:

> One amendment I would like to bring up for consideration wrt the KIP
> process (before we formally include it in our by-laws) is to not
> restrict the votes to be a lazy majority of the PMC, and to instead
> make it a lazy majority of committers.
>
> Our current requirement for code changes per our by-laws are +1 from a
> committer (who is not the contributor) followed by lazy approval. I
> think a lazy majority vote for more significant code changes (i.e., a
> KIP) should be sufficient.
>
> Any objection to this?
>
> On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > Great! Sounds like everyone is on the same page
> >
> >- I created a template page to make things easier. If you do
> Tools->Copy
> >on this page you can just fill in the italic portions with your
> details.
> >- I retrofitted KIP-1 to match this formatting
> >- I added the metadata section people asked for (a link to the
> >discussion, the JIRA, and the current status). Let's make sure we
> remember
> >to update the current status as things are figured out.
> >- Let's keep the discussion on the mailing list rather than on the
> wiki
> >pages. It makes sense to do one or the other so all the comments are
> in one
> >place and I think prior experience is that the wiki comments are the
> worse
> >way.
> >
> > I think it would be great do KIPs for some of the in-flight items folks
> > mentioned.
> >
> > -Jay
> >
> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
> wrote:
> >
> > > +1
> > >
> > > Will be happy to provide a KIP for the multiple-listeners patch.
> > >
> > > Gwen
> > >
> > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
> wrote:
> > > > +1 to everything we have been saying and where this (has settled
> to)/(is
> > > > settling to).
> > > >
> > > > I am sure other folks have some more feedback and think we should
> try to
> > > > keep this discussion going if need be. I am also a firm believer of
> form
> > > > following function so kicking the tires some to flesh out the
> details of
> > > > this and have some organic growth with the process will be healthy
> and
> > > > beneficial to the community.
> > > >
> > > > For my part, what I will do is open a few KIP based on some of the
> work I
> > > > have been involved with for 0.8.3. Off the top of my head this would
> > > > include 1) changes to re-assignment of partitions 2) kafka cli 3)
> global
> > > > configs 4) security white list black list by ip 5) SSL 6) We probably
> > > will
> > > > have lots of Security related KIPs and should treat them all
> individually
> > > > when the time is appropriate 7) Kafka on Mesos.
> > > >
> > > > If someone else wants to jump in to start getting some of the
> security
> > > KIP
> > > > that we are going to have in 0.8.3 I think that would be great (e.g.
> > > > Multiple Listeners for Kafka Brokers). There are also a few other
> > > tickets I
> > > > can think of that are important to have in the code in 0.8.3 that
> should
> > > > have KIP also that I haven't really been involved in. I will take a
> few
> > > > minutes and go through JIRA (one I can think of like auto assign id
> that
> > > is
> > > > already committed I think) and ask for a KIP if appropriate or if I
> feel
> > > > that I can write it up (both from a time and understanding
> perspective)
> > > do
> > > > so.
> > > >
> > > > long story short, I encourage folks to start moving ahead with the
> KIP
> > > for
> > > > 0.8.3 as how we operate. any objections?
> > > >
> > > > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > >> +1 on the idea, and we could mutually link the KIP wiki page with
> the
> > > the
> > > >> created JIRA ticket (i.e. include the JIRA number on the page and
> the
> > > KIP
> > > >> url on the ticket description).
> > > >>
> > > >> Regarding the KIP process, probably we do not need two phase
> > > communication
> > > >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
> > > clear
> > > >> while people discuss about that.
> > > >>
> > > >> About who should trigger the process, I think the only involved
> people
> > > >> would be 1) when the patch is submitted / or even the ticket is
> created,
> > > >> the assignee could choose to start the KIP process if she thought
> it is
> > > >> necessary; 2) the reviewer of the patch can also suggest starting
> KIP
> > > >> discussions.
> > > >>
> > > >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira <
> gshap...@cloudera.com>
> > > >> wrote:
> > > >>
> > > >> > +1 to Ewen's suggestions: Deprecation, status and version.
> > > >> >
> > > >> > Perhaps add the JIRA where the KIP was implemented to the
> metadata.
> > > >> > This will help tie things together.
> > > >> >
> > > >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
> > > >> >  wrote:
> > > >> > > I think adding a section about deprecation would be helpful. A
> good
> > > >> > > fract

Re: [DISCUSS] KIPs

[Neha Narkhede]

Sounds good.

On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps  wrote:

> None on my part.
>
> -Jay
>
> On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy  wrote:
>
> > One amendment I would like to bring up for consideration wrt the KIP
> > process (before we formally include it in our by-laws) is to not
> > restrict the votes to be a lazy majority of the PMC, and to instead
> > make it a lazy majority of committers.
> >
> > Our current requirement for code changes per our by-laws are +1 from a
> > committer (who is not the contributor) followed by lazy approval. I
> > think a lazy majority vote for more significant code changes (i.e., a
> > KIP) should be sufficient.
> >
> > Any objection to this?
> >
> > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > Great! Sounds like everyone is on the same page
> > >
> > >- I created a template page to make things easier. If you do
> > Tools->Copy
> > >on this page you can just fill in the italic portions with your
> > details.
> > >- I retrofitted KIP-1 to match this formatting
> > >- I added the metadata section people asked for (a link to the
> > >discussion, the JIRA, and the current status). Let's make sure we
> > remember
> > >to update the current status as things are figured out.
> > >- Let's keep the discussion on the mailing list rather than on the
> > wiki
> > >pages. It makes sense to do one or the other so all the comments are
> > in one
> > >place and I think prior experience is that the wiki comments are the
> > worse
> > >way.
> > >
> > > I think it would be great do KIPs for some of the in-flight items folks
> > > mentioned.
> > >
> > > -Jay
> > >
> > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
> > wrote:
> > >
> > > > +1
> > > >
> > > > Will be happy to provide a KIP for the multiple-listeners patch.
> > > >
> > > > Gwen
> > > >
> > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
> > wrote:
> > > > > +1 to everything we have been saying and where this (has settled
> > to)/(is
> > > > > settling to).
> > > > >
> > > > > I am sure other folks have some more feedback and think we should
> > try to
> > > > > keep this discussion going if need be. I am also a firm believer of
> > form
> > > > > following function so kicking the tires some to flesh out the
> > details of
> > > > > this and have some organic growth with the process will be healthy
> > and
> > > > > beneficial to the community.
> > > > >
> > > > > For my part, what I will do is open a few KIP based on some of the
> > work I
> > > > > have been involved with for 0.8.3. Off the top of my head this
> would
> > > > > include 1) changes to re-assignment of partitions 2) kafka cli 3)
> > global
> > > > > configs 4) security white list black list by ip 5) SSL 6) We
> probably
> > > > will
> > > > > have lots of Security related KIPs and should treat them all
> > individually
> > > > > when the time is appropriate 7) Kafka on Mesos.
> > > > >
> > > > > If someone else wants to jump in to start getting some of the
> > security
> > > > KIP
> > > > > that we are going to have in 0.8.3 I think that would be great
> (e.g.
> > > > > Multiple Listeners for Kafka Brokers). There are also a few other
> > > > tickets I
> > > > > can think of that are important to have in the code in 0.8.3 that
> > should
> > > > > have KIP also that I haven't really been involved in. I will take a
> > few
> > > > > minutes and go through JIRA (one I can think of like auto assign id
> > that
> > > > is
> > > > > already committed I think) and ask for a KIP if appropriate or if I
> > feel
> > > > > that I can write it up (both from a time and understanding
> > perspective)
> > > > do
> > > > > so.
> > > > >
> > > > > long story short, I encourage folks to start moving ahead with the
> > KIP
> > > > for
> > > > > 0.8.3 as how we operate. any objections?
> > > > >
> > > > > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > >> +1 on the idea, and we could mutually link the KIP wiki page with
> > the
> > > > the
> > > > >> created JIRA ticket (i.e. include the JIRA number on the page and
> > the
> > > > KIP
> > > > >> url on the ticket description).
> > > > >>
> > > > >> Regarding the KIP process, probably we do not need two phase
> > > > communication
> > > > >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should
> be
> > > > clear
> > > > >> while people discuss about that.
> > > > >>
> > > > >> About who should trigger the process, I think the only involved
> > people
> > > > >> would be 1) when the patch is submitted / or even the ticket is
> > created,
> > > > >> the assignee could choose to start the KIP process if she thought
> > it is
> > > > >> necessary; 2) the reviewer of the patch can also suggest starting
> > KIP
> > > > >> discussions.
> > > > >>
> > > > >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira <
> > gshap...@cloudera.com>
> > > > >> wrote:
> > > > >>
> > > > >> > +1 to Ewen's suggestions: Deprecation, status and v

Re: [DISCUSS] KIPs

[Joe Stein]

+1

On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede  wrote:

> Sounds good.
>
> On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps  wrote:
>
> > None on my part.
> >
> > -Jay
> >
> > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy  wrote:
> >
> > > One amendment I would like to bring up for consideration wrt the KIP
> > > process (before we formally include it in our by-laws) is to not
> > > restrict the votes to be a lazy majority of the PMC, and to instead
> > > make it a lazy majority of committers.
> > >
> > > Our current requirement for code changes per our by-laws are +1 from a
> > > committer (who is not the contributor) followed by lazy approval. I
> > > think a lazy majority vote for more significant code changes (i.e., a
> > > KIP) should be sufficient.
> > >
> > > Any objection to this?
> > >
> > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > Great! Sounds like everyone is on the same page
> > > >
> > > >- I created a template page to make things easier. If you do
> > > Tools->Copy
> > > >on this page you can just fill in the italic portions with your
> > > details.
> > > >- I retrofitted KIP-1 to match this formatting
> > > >- I added the metadata section people asked for (a link to the
> > > >discussion, the JIRA, and the current status). Let's make sure we
> > > remember
> > > >to update the current status as things are figured out.
> > > >- Let's keep the discussion on the mailing list rather than on the
> > > wiki
> > > >pages. It makes sense to do one or the other so all the comments
> are
> > > in one
> > > >place and I think prior experience is that the wiki comments are
> the
> > > worse
> > > >way.
> > > >
> > > > I think it would be great do KIPs for some of the in-flight items
> folks
> > > > mentioned.
> > > >
> > > > -Jay
> > > >
> > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira  >
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Will be happy to provide a KIP for the multiple-listeners patch.
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
> > > wrote:
> > > > > > +1 to everything we have been saying and where this (has settled
> > > to)/(is
> > > > > > settling to).
> > > > > >
> > > > > > I am sure other folks have some more feedback and think we should
> > > try to
> > > > > > keep this discussion going if need be. I am also a firm believer
> of
> > > form
> > > > > > following function so kicking the tires some to flesh out the
> > > details of
> > > > > > this and have some organic growth with the process will be
> healthy
> > > and
> > > > > > beneficial to the community.
> > > > > >
> > > > > > For my part, what I will do is open a few KIP based on some of
> the
> > > work I
> > > > > > have been involved with for 0.8.3. Off the top of my head this
> > would
> > > > > > include 1) changes to re-assignment of partitions 2) kafka cli 3)
> > > global
> > > > > > configs 4) security white list black list by ip 5) SSL 6) We
> > probably
> > > > > will
> > > > > > have lots of Security related KIPs and should treat them all
> > > individually
> > > > > > when the time is appropriate 7) Kafka on Mesos.
> > > > > >
> > > > > > If someone else wants to jump in to start getting some of the
> > > security
> > > > > KIP
> > > > > > that we are going to have in 0.8.3 I think that would be great
> > (e.g.
> > > > > > Multiple Listeners for Kafka Brokers). There are also a few other
> > > > > tickets I
> > > > > > can think of that are important to have in the code in 0.8.3 that
> > > should
> > > > > > have KIP also that I haven't really been involved in. I will
> take a
> > > few
> > > > > > minutes and go through JIRA (one I can think of like auto assign
> id
> > > that
> > > > > is
> > > > > > already committed I think) and ask for a KIP if appropriate or
> if I
> > > feel
> > > > > > that I can write it up (both from a time and understanding
> > > perspective)
> > > > > do
> > > > > > so.
> > > > > >
> > > > > > long story short, I encourage folks to start moving ahead with
> the
> > > KIP
> > > > > for
> > > > > > 0.8.3 as how we operate. any objections?
> > > > > >
> > > > > > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> +1 on the idea, and we could mutually link the KIP wiki page
> with
> > > the
> > > > > the
> > > > > >> created JIRA ticket (i.e. include the JIRA number on the page
> and
> > > the
> > > > > KIP
> > > > > >> url on the ticket description).
> > > > > >>
> > > > > >> Regarding the KIP process, probably we do not need two phase
> > > > > communication
> > > > > >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should
> > be
> > > > > clear
> > > > > >> while people discuss about that.
> > > > > >>
> > > > > >> About who should trigger the process, I think the only involved
> > > people
> > > > > >> would be 1) when the patch is submitted / or even the ticket is
> > > created,
> > > > > >>

Re: [DISCUSS] KIPs

[Joel Koshy]

Sorry about this - I actually meant to suggest lazy consensus (which
is a stronger requirement): "3 binding +1 votes and no binding
vetoes."

I have updated the wiki to lazy consensus - but can change it back if
there is a reasonable objection.

On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> +1
> 
> On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede  wrote:
> 
> > Sounds good.
> >
> > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps  wrote:
> >
> > > None on my part.
> > >
> > > -Jay
> > >
> > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy  wrote:
> > >
> > > > One amendment I would like to bring up for consideration wrt the KIP
> > > > process (before we formally include it in our by-laws) is to not
> > > > restrict the votes to be a lazy majority of the PMC, and to instead
> > > > make it a lazy majority of committers.
> > > >
> > > > Our current requirement for code changes per our by-laws are +1 from a
> > > > committer (who is not the contributor) followed by lazy approval. I
> > > > think a lazy majority vote for more significant code changes (i.e., a
> > > > KIP) should be sufficient.
> > > >
> > > > Any objection to this?
> > > >
> > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > > Great! Sounds like everyone is on the same page
> > > > >
> > > > >- I created a template page to make things easier. If you do
> > > > Tools->Copy
> > > > >on this page you can just fill in the italic portions with your
> > > > details.
> > > > >- I retrofitted KIP-1 to match this formatting
> > > > >- I added the metadata section people asked for (a link to the
> > > > >discussion, the JIRA, and the current status). Let's make sure we
> > > > remember
> > > > >to update the current status as things are figured out.
> > > > >- Let's keep the discussion on the mailing list rather than on the
> > > > wiki
> > > > >pages. It makes sense to do one or the other so all the comments
> > are
> > > > in one
> > > > >place and I think prior experience is that the wiki comments are
> > the
> > > > worse
> > > > >way.
> > > > >
> > > > > I think it would be great do KIPs for some of the in-flight items
> > folks
> > > > > mentioned.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira  > >
> > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Will be happy to provide a KIP for the multiple-listeners patch.
> > > > > >
> > > > > > Gwen
> > > > > >
> > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
> > > > wrote:
> > > > > > > +1 to everything we have been saying and where this (has settled
> > > > to)/(is
> > > > > > > settling to).
> > > > > > >
> > > > > > > I am sure other folks have some more feedback and think we should
> > > > try to
> > > > > > > keep this discussion going if need be. I am also a firm believer
> > of
> > > > form
> > > > > > > following function so kicking the tires some to flesh out the
> > > > details of
> > > > > > > this and have some organic growth with the process will be
> > healthy
> > > > and
> > > > > > > beneficial to the community.
> > > > > > >
> > > > > > > For my part, what I will do is open a few KIP based on some of
> > the
> > > > work I
> > > > > > > have been involved with for 0.8.3. Off the top of my head this
> > > would
> > > > > > > include 1) changes to re-assignment of partitions 2) kafka cli 3)
> > > > global
> > > > > > > configs 4) security white list black list by ip 5) SSL 6) We
> > > probably
> > > > > > will
> > > > > > > have lots of Security related KIPs and should treat them all
> > > > individually
> > > > > > > when the time is appropriate 7) Kafka on Mesos.
> > > > > > >
> > > > > > > If someone else wants to jump in to start getting some of the
> > > > security
> > > > > > KIP
> > > > > > > that we are going to have in 0.8.3 I think that would be great
> > > (e.g.
> > > > > > > Multiple Listeners for Kafka Brokers). There are also a few other
> > > > > > tickets I
> > > > > > > can think of that are important to have in the code in 0.8.3 that
> > > > should
> > > > > > > have KIP also that I haven't really been involved in. I will
> > take a
> > > > few
> > > > > > > minutes and go through JIRA (one I can think of like auto assign
> > id
> > > > that
> > > > > > is
> > > > > > > already committed I think) and ask for a KIP if appropriate or
> > if I
> > > > feel
> > > > > > > that I can write it up (both from a time and understanding
> > > > perspective)
> > > > > > do
> > > > > > > so.
> > > > > > >
> > > > > > > long story short, I encourage folks to start moving ahead with
> > the
> > > > KIP
> > > > > > for
> > > > > > > 0.8.3 as how we operate. any objections?
> > > > > > >
> > > > > > > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang <
> > wangg...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> +1 on the idea, and we could mutually link the KIP wiki page
> > with
> > > > the
> > > > > > the
> > > > > > >> created JIRA ticket (i.e.

Re: [DISCUSS] KIPs

[Gwen Shapira]

Isn't requiring 3 binding votes a bit overly strict here? We are talking
about patches which in can be fixed, reverted, etc. Not releases, which
have legal implications.

Why not go with usual definition: "lazy" = "No strong objections for few
days"?
This means contributors will not be blocked on issues where no one cares to
comment (and we had few of those).

Gwen



On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  wrote:

> Sorry about this - I actually meant to suggest lazy consensus (which
> is a stronger requirement): "3 binding +1 votes and no binding
> vetoes."
>
> I have updated the wiki to lazy consensus - but can change it back if
> there is a reasonable objection.
>
> On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> > +1
> >
> > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede  wrote:
> >
> > > Sounds good.
> > >
> > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps  wrote:
> > >
> > > > None on my part.
> > > >
> > > > -Jay
> > > >
> > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy 
> wrote:
> > > >
> > > > > One amendment I would like to bring up for consideration wrt the
> KIP
> > > > > process (before we formally include it in our by-laws) is to not
> > > > > restrict the votes to be a lazy majority of the PMC, and to instead
> > > > > make it a lazy majority of committers.
> > > > >
> > > > > Our current requirement for code changes per our by-laws are +1
> from a
> > > > > committer (who is not the contributor) followed by lazy approval. I
> > > > > think a lazy majority vote for more significant code changes
> (i.e., a
> > > > > KIP) should be sufficient.
> > > > >
> > > > > Any objection to this?
> > > > >
> > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > > > Great! Sounds like everyone is on the same page
> > > > > >
> > > > > >- I created a template page to make things easier. If you do
> > > > > Tools->Copy
> > > > > >on this page you can just fill in the italic portions with
> your
> > > > > details.
> > > > > >- I retrofitted KIP-1 to match this formatting
> > > > > >- I added the metadata section people asked for (a link to the
> > > > > >discussion, the JIRA, and the current status). Let's make
> sure we
> > > > > remember
> > > > > >to update the current status as things are figured out.
> > > > > >- Let's keep the discussion on the mailing list rather than
> on the
> > > > > wiki
> > > > > >pages. It makes sense to do one or the other so all the
> comments
> > > are
> > > > > in one
> > > > > >place and I think prior experience is that the wiki comments
> are
> > > the
> > > > > worse
> > > > > >way.
> > > > > >
> > > > > > I think it would be great do KIPs for some of the in-flight items
> > > folks
> > > > > > mentioned.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
> gshap...@cloudera.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Will be happy to provide a KIP for the multiple-listeners
> patch.
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <
> joe.st...@stealth.ly>
> > > > > wrote:
> > > > > > > > +1 to everything we have been saying and where this (has
> settled
> > > > > to)/(is
> > > > > > > > settling to).
> > > > > > > >
> > > > > > > > I am sure other folks have some more feedback and think we
> should
> > > > > try to
> > > > > > > > keep this discussion going if need be. I am also a firm
> believer
> > > of
> > > > > form
> > > > > > > > following function so kicking the tires some to flesh out the
> > > > > details of
> > > > > > > > this and have some organic growth with the process will be
> > > healthy
> > > > > and
> > > > > > > > beneficial to the community.
> > > > > > > >
> > > > > > > > For my part, what I will do is open a few KIP based on some
> of
> > > the
> > > > > work I
> > > > > > > > have been involved with for 0.8.3. Off the top of my head
> this
> > > > would
> > > > > > > > include 1) changes to re-assignment of partitions 2) kafka
> cli 3)
> > > > > global
> > > > > > > > configs 4) security white list black list by ip 5) SSL 6) We
> > > > probably
> > > > > > > will
> > > > > > > > have lots of Security related KIPs and should treat them all
> > > > > individually
> > > > > > > > when the time is appropriate 7) Kafka on Mesos.
> > > > > > > >
> > > > > > > > If someone else wants to jump in to start getting some of the
> > > > > security
> > > > > > > KIP
> > > > > > > > that we are going to have in 0.8.3 I think that would be
> great
> > > > (e.g.
> > > > > > > > Multiple Listeners for Kafka Brokers). There are also a few
> other
> > > > > > > tickets I
> > > > > > > > can think of that are important to have in the code in 0.8.3
> that
> > > > > should
> > > > > > > > have KIP also that I haven't really been involved in. I will
> > > take a
> > > > > few
> > > > > > > > minutes and go through JIRA (one I can think of like auto
>

Re: [DISCUSS] KIPs

[Jay Kreps]

Hey Joel,

The problem with lazy consensus is that some people are too lazy. :-) I
think the whole point of this is that you need to actively ensure people
have read and understand the change before you proceed. I basically think
all of us should be reading these proposals and giving feedback.

-Jay

On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  wrote:

> Sorry about this - I actually meant to suggest lazy consensus (which
> is a stronger requirement): "3 binding +1 votes and no binding
> vetoes."
>
> I have updated the wiki to lazy consensus - but can change it back if
> there is a reasonable objection.
>
> On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> > +1
> >
> > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede  wrote:
> >
> > > Sounds good.
> > >
> > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps  wrote:
> > >
> > > > None on my part.
> > > >
> > > > -Jay
> > > >
> > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy 
> wrote:
> > > >
> > > > > One amendment I would like to bring up for consideration wrt the
> KIP
> > > > > process (before we formally include it in our by-laws) is to not
> > > > > restrict the votes to be a lazy majority of the PMC, and to instead
> > > > > make it a lazy majority of committers.
> > > > >
> > > > > Our current requirement for code changes per our by-laws are +1
> from a
> > > > > committer (who is not the contributor) followed by lazy approval. I
> > > > > think a lazy majority vote for more significant code changes
> (i.e., a
> > > > > KIP) should be sufficient.
> > > > >
> > > > > Any objection to this?
> > > > >
> > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > > > Great! Sounds like everyone is on the same page
> > > > > >
> > > > > >- I created a template page to make things easier. If you do
> > > > > Tools->Copy
> > > > > >on this page you can just fill in the italic portions with
> your
> > > > > details.
> > > > > >- I retrofitted KIP-1 to match this formatting
> > > > > >- I added the metadata section people asked for (a link to the
> > > > > >discussion, the JIRA, and the current status). Let's make
> sure we
> > > > > remember
> > > > > >to update the current status as things are figured out.
> > > > > >- Let's keep the discussion on the mailing list rather than
> on the
> > > > > wiki
> > > > > >pages. It makes sense to do one or the other so all the
> comments
> > > are
> > > > > in one
> > > > > >place and I think prior experience is that the wiki comments
> are
> > > the
> > > > > worse
> > > > > >way.
> > > > > >
> > > > > > I think it would be great do KIPs for some of the in-flight items
> > > folks
> > > > > > mentioned.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
> gshap...@cloudera.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Will be happy to provide a KIP for the multiple-listeners
> patch.
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <
> joe.st...@stealth.ly>
> > > > > wrote:
> > > > > > > > +1 to everything we have been saying and where this (has
> settled
> > > > > to)/(is
> > > > > > > > settling to).
> > > > > > > >
> > > > > > > > I am sure other folks have some more feedback and think we
> should
> > > > > try to
> > > > > > > > keep this discussion going if need be. I am also a firm
> believer
> > > of
> > > > > form
> > > > > > > > following function so kicking the tires some to flesh out the
> > > > > details of
> > > > > > > > this and have some organic growth with the process will be
> > > healthy
> > > > > and
> > > > > > > > beneficial to the community.
> > > > > > > >
> > > > > > > > For my part, what I will do is open a few KIP based on some
> of
> > > the
> > > > > work I
> > > > > > > > have been involved with for 0.8.3. Off the top of my head
> this
> > > > would
> > > > > > > > include 1) changes to re-assignment of partitions 2) kafka
> cli 3)
> > > > > global
> > > > > > > > configs 4) security white list black list by ip 5) SSL 6) We
> > > > probably
> > > > > > > will
> > > > > > > > have lots of Security related KIPs and should treat them all
> > > > > individually
> > > > > > > > when the time is appropriate 7) Kafka on Mesos.
> > > > > > > >
> > > > > > > > If someone else wants to jump in to start getting some of the
> > > > > security
> > > > > > > KIP
> > > > > > > > that we are going to have in 0.8.3 I think that would be
> great
> > > > (e.g.
> > > > > > > > Multiple Listeners for Kafka Brokers). There are also a few
> other
> > > > > > > tickets I
> > > > > > > > can think of that are important to have in the code in 0.8.3
> that
> > > > > should
> > > > > > > > have KIP also that I haven't really been involved in. I will
> > > take a
> > > > > few
> > > > > > > > minutes and go through JIRA (one I can think of like auto
> assign
> > > id
> > > > > that
> > > > > > > is
> > > > > > > > alre

Re: [DISCUSS] KIPs

[Joel Koshy]

The original requirement was lazy majority of PMC which definitely
seems overly restrictive.

> Why not go with usual definition: "lazy" = "No strong objections for few
> days"?
> This means contributors will not be blocked on issues where no one cares to
> comment (and we had few of those).

I think one benefit of not doing the "just lazy" approach would be to
ensure that the proposal has been vetted - otherwise a contributor may
go ahead and spend a ton of time proceeding with an implementation
plan only to receive a -1 on a code change which automatically moves
to a lazy majority anyway. If a change is non-controversial and not
particularly extensive, then a KIP is not required in the first place
so this is less of an issue (i.e., if the code review moves to a lazy
majority).

For a large piece of work (that encourage a KIP), the more restrictive
voting requirements probably make sense to avoid the undesirable
scenario of a lot of effort spent on a feature only to have it
subsequently voted down. Does this sound reasonable?  I changed the
wiki back to lazy majority - which makes it less restrictive than it
was, but I think we should all agree whether to make it even less
restrictive or not.

Yes there are KIPs that are currently blocked on feedback/votes, but I
don't think it is an issue of not caring to comment vs having so many
KIPs and other code reviews in flight that people are just swamped.

Joel

On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
> Isn't requiring 3 binding votes a bit overly strict here? We are talking
> about patches which in can be fixed, reverted, etc. Not releases, which
> have legal implications.
> 
> Why not go with usual definition: "lazy" = "No strong objections for few
> days"?
> This means contributors will not be blocked on issues where no one cares to
> comment (and we had few of those).
> 
> Gwen
> 
> 
> 
> On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  wrote:
> 
> > Sorry about this - I actually meant to suggest lazy consensus (which
> > is a stronger requirement): "3 binding +1 votes and no binding
> > vetoes."
> >
> > I have updated the wiki to lazy consensus - but can change it back if
> > there is a reasonable objection.
> >
> > On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> > > +1
> > >
> > > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede  wrote:
> > >
> > > > Sounds good.
> > > >
> > > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps  wrote:
> > > >
> > > > > None on my part.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy 
> > wrote:
> > > > >
> > > > > > One amendment I would like to bring up for consideration wrt the
> > KIP
> > > > > > process (before we formally include it in our by-laws) is to not
> > > > > > restrict the votes to be a lazy majority of the PMC, and to instead
> > > > > > make it a lazy majority of committers.
> > > > > >
> > > > > > Our current requirement for code changes per our by-laws are +1
> > from a
> > > > > > committer (who is not the contributor) followed by lazy approval. I
> > > > > > think a lazy majority vote for more significant code changes
> > (i.e., a
> > > > > > KIP) should be sufficient.
> > > > > >
> > > > > > Any objection to this?
> > > > > >
> > > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > > > > Great! Sounds like everyone is on the same page
> > > > > > >
> > > > > > >- I created a template page to make things easier. If you do
> > > > > > Tools->Copy
> > > > > > >on this page you can just fill in the italic portions with
> > your
> > > > > > details.
> > > > > > >- I retrofitted KIP-1 to match this formatting
> > > > > > >- I added the metadata section people asked for (a link to the
> > > > > > >discussion, the JIRA, and the current status). Let's make
> > sure we
> > > > > > remember
> > > > > > >to update the current status as things are figured out.
> > > > > > >- Let's keep the discussion on the mailing list rather than
> > on the
> > > > > > wiki
> > > > > > >pages. It makes sense to do one or the other so all the
> > comments
> > > > are
> > > > > > in one
> > > > > > >place and I think prior experience is that the wiki comments
> > are
> > > > the
> > > > > > worse
> > > > > > >way.
> > > > > > >
> > > > > > > I think it would be great do KIPs for some of the in-flight items
> > > > folks
> > > > > > > mentioned.
> > > > > > >
> > > > > > > -Jay
> > > > > > >
> > > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
> > gshap...@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Will be happy to provide a KIP for the multiple-listeners
> > patch.
> > > > > > > >
> > > > > > > > Gwen
> > > > > > > >
> > > > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <
> > joe.st...@stealth.ly>
> > > > > > wrote:
> > > > > > > > > +1 to everything we have been saying and where this (has
> > settled
> > > > > > to)/(is
> > > > > > 

Re: [DISCUSS] KIPs

[Joel Koshy]

Yes - I realized that afterward. It is back to plain lazy majority.

On Thu, Feb 05, 2015 at 04:33:48PM -0800, Jay Kreps wrote:
> Hey Joel,
> 
> The problem with lazy consensus is that some people are too lazy. :-) I
> think the whole point of this is that you need to actively ensure people
> have read and understand the change before you proceed. I basically think
> all of us should be reading these proposals and giving feedback.
> 
> -Jay
> 
> On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  wrote:
> 
> > Sorry about this - I actually meant to suggest lazy consensus (which
> > is a stronger requirement): "3 binding +1 votes and no binding
> > vetoes."
> >
> > I have updated the wiki to lazy consensus - but can change it back if
> > there is a reasonable objection.
> >
> > On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> > > +1
> > >
> > > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede  wrote:
> > >
> > > > Sounds good.
> > > >
> > > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps  wrote:
> > > >
> > > > > None on my part.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy 
> > wrote:
> > > > >
> > > > > > One amendment I would like to bring up for consideration wrt the
> > KIP
> > > > > > process (before we formally include it in our by-laws) is to not
> > > > > > restrict the votes to be a lazy majority of the PMC, and to instead
> > > > > > make it a lazy majority of committers.
> > > > > >
> > > > > > Our current requirement for code changes per our by-laws are +1
> > from a
> > > > > > committer (who is not the contributor) followed by lazy approval. I
> > > > > > think a lazy majority vote for more significant code changes
> > (i.e., a
> > > > > > KIP) should be sufficient.
> > > > > >
> > > > > > Any objection to this?
> > > > > >
> > > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > > > > Great! Sounds like everyone is on the same page
> > > > > > >
> > > > > > >- I created a template page to make things easier. If you do
> > > > > > Tools->Copy
> > > > > > >on this page you can just fill in the italic portions with
> > your
> > > > > > details.
> > > > > > >- I retrofitted KIP-1 to match this formatting
> > > > > > >- I added the metadata section people asked for (a link to the
> > > > > > >discussion, the JIRA, and the current status). Let's make
> > sure we
> > > > > > remember
> > > > > > >to update the current status as things are figured out.
> > > > > > >- Let's keep the discussion on the mailing list rather than
> > on the
> > > > > > wiki
> > > > > > >pages. It makes sense to do one or the other so all the
> > comments
> > > > are
> > > > > > in one
> > > > > > >place and I think prior experience is that the wiki comments
> > are
> > > > the
> > > > > > worse
> > > > > > >way.
> > > > > > >
> > > > > > > I think it would be great do KIPs for some of the in-flight items
> > > > folks
> > > > > > > mentioned.
> > > > > > >
> > > > > > > -Jay
> > > > > > >
> > > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
> > gshap...@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Will be happy to provide a KIP for the multiple-listeners
> > patch.
> > > > > > > >
> > > > > > > > Gwen
> > > > > > > >
> > > > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <
> > joe.st...@stealth.ly>
> > > > > > wrote:
> > > > > > > > > +1 to everything we have been saying and where this (has
> > settled
> > > > > > to)/(is
> > > > > > > > > settling to).
> > > > > > > > >
> > > > > > > > > I am sure other folks have some more feedback and think we
> > should
> > > > > > try to
> > > > > > > > > keep this discussion going if need be. I am also a firm
> > believer
> > > > of
> > > > > > form
> > > > > > > > > following function so kicking the tires some to flesh out the
> > > > > > details of
> > > > > > > > > this and have some organic growth with the process will be
> > > > healthy
> > > > > > and
> > > > > > > > > beneficial to the community.
> > > > > > > > >
> > > > > > > > > For my part, what I will do is open a few KIP based on some
> > of
> > > > the
> > > > > > work I
> > > > > > > > > have been involved with for 0.8.3. Off the top of my head
> > this
> > > > > would
> > > > > > > > > include 1) changes to re-assignment of partitions 2) kafka
> > cli 3)
> > > > > > global
> > > > > > > > > configs 4) security white list black list by ip 5) SSL 6) We
> > > > > probably
> > > > > > > > will
> > > > > > > > > have lots of Security related KIPs and should treat them all
> > > > > > individually
> > > > > > > > > when the time is appropriate 7) Kafka on Mesos.
> > > > > > > > >
> > > > > > > > > If someone else wants to jump in to start getting some of the
> > > > > > security
> > > > > > > > KIP
> > > > > > > > > that we are going to have in 0.8.3 I think that would be
> > great
> > > > > (e.g.
> > > > > > > > > Multiple Listeners for Kafka Brok

Re: [DISCUSS] KIPs

[Gwen Shapira]

>
>
> Yes there are KIPs that are currently blocked on feedback/votes, but I
> don't think it is an issue of not caring to comment vs having so many
> KIPs and other code reviews in flight that people are just swamped.
>
>
This is exactly my concern.
Even now important patches and features have very long development and
review cycles due to Kafka's small and very busy committer community. I
feel that this change takes things in the wrong direction


> Joel
>
> On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
> > Isn't requiring 3 binding votes a bit overly strict here? We are talking
> > about patches which in can be fixed, reverted, etc. Not releases, which
> > have legal implications.
> >
> > Why not go with usual definition: "lazy" = "No strong objections for few
> > days"?
> > This means contributors will not be blocked on issues where no one cares
> to
> > comment (and we had few of those).
> >
> > Gwen
> >
> >
> >
> > On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  wrote:
> >
> > > Sorry about this - I actually meant to suggest lazy consensus (which
> > > is a stronger requirement): "3 binding +1 votes and no binding
> > > vetoes."
> > >
> > > I have updated the wiki to lazy consensus - but can change it back if
> > > there is a reasonable objection.
> > >
> > > On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> > > > +1
> > > >
> > > > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede 
> wrote:
> > > >
> > > > > Sounds good.
> > > > >
> > > > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps 
> wrote:
> > > > >
> > > > > > None on my part.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy  >
> > > wrote:
> > > > > >
> > > > > > > One amendment I would like to bring up for consideration wrt
> the
> > > KIP
> > > > > > > process (before we formally include it in our by-laws) is to
> not
> > > > > > > restrict the votes to be a lazy majority of the PMC, and to
> instead
> > > > > > > make it a lazy majority of committers.
> > > > > > >
> > > > > > > Our current requirement for code changes per our by-laws are +1
> > > from a
> > > > > > > committer (who is not the contributor) followed by lazy
> approval. I
> > > > > > > think a lazy majority vote for more significant code changes
> > > (i.e., a
> > > > > > > KIP) should be sufficient.
> > > > > > >
> > > > > > > Any objection to this?
> > > > > > >
> > > > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > > > > > Great! Sounds like everyone is on the same page
> > > > > > > >
> > > > > > > >- I created a template page to make things easier. If you
> do
> > > > > > > Tools->Copy
> > > > > > > >on this page you can just fill in the italic portions with
> > > your
> > > > > > > details.
> > > > > > > >- I retrofitted KIP-1 to match this formatting
> > > > > > > >- I added the metadata section people asked for (a link
> to the
> > > > > > > >discussion, the JIRA, and the current status). Let's make
> > > sure we
> > > > > > > remember
> > > > > > > >to update the current status as things are figured out.
> > > > > > > >- Let's keep the discussion on the mailing list rather
> than
> > > on the
> > > > > > > wiki
> > > > > > > >pages. It makes sense to do one or the other so all the
> > > comments
> > > > > are
> > > > > > > in one
> > > > > > > >place and I think prior experience is that the wiki
> comments
> > > are
> > > > > the
> > > > > > > worse
> > > > > > > >way.
> > > > > > > >
> > > > > > > > I think it would be great do KIPs for some of the in-flight
> items
> > > > > folks
> > > > > > > > mentioned.
> > > > > > > >
> > > > > > > > -Jay
> > > > > > > >
> > > > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
> > > gshap...@cloudera.com
> > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > Will be happy to provide a KIP for the multiple-listeners
> > > patch.
> > > > > > > > >
> > > > > > > > > Gwen
> > > > > > > > >
> > > > > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <
> > > joe.st...@stealth.ly>
> > > > > > > wrote:
> > > > > > > > > > +1 to everything we have been saying and where this (has
> > > settled
> > > > > > > to)/(is
> > > > > > > > > > settling to).
> > > > > > > > > >
> > > > > > > > > > I am sure other folks have some more feedback and think
> we
> > > should
> > > > > > > try to
> > > > > > > > > > keep this discussion going if need be. I am also a firm
> > > believer
> > > > > of
> > > > > > > form
> > > > > > > > > > following function so kicking the tires some to flesh
> out the
> > > > > > > details of
> > > > > > > > > > this and have some organic growth with the process will
> be
> > > > > healthy
> > > > > > > and
> > > > > > > > > > beneficial to the community.
> > > > > > > > > >
> > > > > > > > > > For my part, what I will do is open a few KIP based on
> some
> > > of
> > > > > the
> > > > > > > work I
> > > > > > > > > > have been involv

Re: [DISCUSS] KIPs

[Joel Koshy]

> This is exactly my concern.
> Even now important patches and features have very long development and
> review cycles due to Kafka's small and very busy committer community. I
> feel that this change takes things in the wrong direction

I agree that we should improve on this.  I think the only concern in
making the vote less restrictive is what I mentioned earlier - i.e., a
contributor starting on a major feature only to have to rework or
abandon a design later during code reviews which is even more
frustrating.

We could also discuss having some SLA in place on turn-around time for
a KIP (say, a week?). Given that KIPs are (sort of by definition)
major features and/or backward incompatible changes it may even be
reasonable to have at most only two or three KIPs under active
discussion at any given time. That impacts feature velocity but I
don't see a way out of that.

BTW (personally) I'm fine with making it less restrictive - since we
have been implicitly doing that so far in the absence of KIPs.  I
think KIPs are useful by themselves (i.e., voting aside).

Thanks,

Joel

On Thu, Feb 05, 2015 at 04:57:03PM -0800, Gwen Shapira wrote:
> >
> >
> > Yes there are KIPs that are currently blocked on feedback/votes, but I
> > don't think it is an issue of not caring to comment vs having so many
> > KIPs and other code reviews in flight that people are just swamped.
> >
> >
> This is exactly my concern.
> Even now important patches and features have very long development and
> review cycles due to Kafka's small and very busy committer community. I
> feel that this change takes things in the wrong direction
> 
> 
> > Joel
> >
> > On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
> > > Isn't requiring 3 binding votes a bit overly strict here? We are talking
> > > about patches which in can be fixed, reverted, etc. Not releases, which
> > > have legal implications.
> > >
> > > Why not go with usual definition: "lazy" = "No strong objections for few
> > > days"?
> > > This means contributors will not be blocked on issues where no one cares
> > to
> > > comment (and we had few of those).
> > >
> > > Gwen
> > >
> > >
> > >
> > > On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  wrote:
> > >
> > > > Sorry about this - I actually meant to suggest lazy consensus (which
> > > > is a stronger requirement): "3 binding +1 votes and no binding
> > > > vetoes."
> > > >
> > > > I have updated the wiki to lazy consensus - but can change it back if
> > > > there is a reasonable objection.
> > > >
> > > > On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> > > > > +1
> > > > >
> > > > > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede 
> > wrote:
> > > > >
> > > > > > Sounds good.
> > > > > >
> > > > > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps 
> > wrote:
> > > > > >
> > > > > > > None on my part.
> > > > > > >
> > > > > > > -Jay
> > > > > > >
> > > > > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy  > >
> > > > wrote:
> > > > > > >
> > > > > > > > One amendment I would like to bring up for consideration wrt
> > the
> > > > KIP
> > > > > > > > process (before we formally include it in our by-laws) is to
> > not
> > > > > > > > restrict the votes to be a lazy majority of the PMC, and to
> > instead
> > > > > > > > make it a lazy majority of committers.
> > > > > > > >
> > > > > > > > Our current requirement for code changes per our by-laws are +1
> > > > from a
> > > > > > > > committer (who is not the contributor) followed by lazy
> > approval. I
> > > > > > > > think a lazy majority vote for more significant code changes
> > > > (i.e., a
> > > > > > > > KIP) should be sufficient.
> > > > > > > >
> > > > > > > > Any objection to this?
> > > > > > > >
> > > > > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > > > > > > > > Great! Sounds like everyone is on the same page
> > > > > > > > >
> > > > > > > > >- I created a template page to make things easier. If you
> > do
> > > > > > > > Tools->Copy
> > > > > > > > >on this page you can just fill in the italic portions with
> > > > your
> > > > > > > > details.
> > > > > > > > >- I retrofitted KIP-1 to match this formatting
> > > > > > > > >- I added the metadata section people asked for (a link
> > to the
> > > > > > > > >discussion, the JIRA, and the current status). Let's make
> > > > sure we
> > > > > > > > remember
> > > > > > > > >to update the current status as things are figured out.
> > > > > > > > >- Let's keep the discussion on the mailing list rather
> > than
> > > > on the
> > > > > > > > wiki
> > > > > > > > >pages. It makes sense to do one or the other so all the
> > > > comments
> > > > > > are
> > > > > > > > in one
> > > > > > > > >place and I think prior experience is that the wiki
> > comments
> > > > are
> > > > > > the
> > > > > > > > worse
> > > > > > > > >way.
> > > > > > > > >
> > > > > > > > > I think it would be great do KIPs for some of the in-flight
> > items
> > > > > > folks
> > > 

Re: [DISCUSS] KIPs

[Jiangjie Qin]

I¹m having an impression that KIP is mostly for new features but not for
bug fixes. But I agree with Joel that it might make sense to have some big
patches, even if they are bug fixes, to follow the KIP like process which
is more strict.

Jiangjie (Becket) Qin

On 2/5/15, 4:57 PM, "Gwen Shapira"  wrote:

>>
>>
>> Yes there are KIPs that are currently blocked on feedback/votes, but I
>> don't think it is an issue of not caring to comment vs having so many
>> KIPs and other code reviews in flight that people are just swamped.
>>
>>
>This is exactly my concern.
>Even now important patches and features have very long development and
>review cycles due to Kafka's small and very busy committer community. I
>feel that this change takes things in the wrong direction
>
>
>> Joel
>>
>> On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
>> > Isn't requiring 3 binding votes a bit overly strict here? We are
>>talking
>> > about patches which in can be fixed, reverted, etc. Not releases,
>>which
>> > have legal implications.
>> >
>> > Why not go with usual definition: "lazy" = "No strong objections for
>>few
>> > days"?
>> > This means contributors will not be blocked on issues where no one
>>cares
>> to
>> > comment (and we had few of those).
>> >
>> > Gwen
>> >
>> >
>> >
>> > On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy 
>>wrote:
>> >
>> > > Sorry about this - I actually meant to suggest lazy consensus (which
>> > > is a stronger requirement): "3 binding +1 votes and no binding
>> > > vetoes."
>> > >
>> > > I have updated the wiki to lazy consensus - but can change it back
>>if
>> > > there is a reasonable objection.
>> > >
>> > > On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
>> > > > +1
>> > > >
>> > > > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede 
>> wrote:
>> > > >
>> > > > > Sounds good.
>> > > > >
>> > > > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps 
>> wrote:
>> > > > >
>> > > > > > None on my part.
>> > > > > >
>> > > > > > -Jay
>> > > > > >
>> > > > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy
>>> >
>> > > wrote:
>> > > > > >
>> > > > > > > One amendment I would like to bring up for consideration wrt
>> the
>> > > KIP
>> > > > > > > process (before we formally include it in our by-laws) is to
>> not
>> > > > > > > restrict the votes to be a lazy majority of the PMC, and to
>> instead
>> > > > > > > make it a lazy majority of committers.
>> > > > > > >
>> > > > > > > Our current requirement for code changes per our by-laws
>>are +1
>> > > from a
>> > > > > > > committer (who is not the contributor) followed by lazy
>> approval. I
>> > > > > > > think a lazy majority vote for more significant code changes
>> > > (i.e., a
>> > > > > > > KIP) should be sufficient.
>> > > > > > >
>> > > > > > > Any objection to this?
>> > > > > > >
>> > > > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
>> > > > > > > > Great! Sounds like everyone is on the same page
>> > > > > > > >
>> > > > > > > >- I created a template page to make things easier. If
>>you
>> do
>> > > > > > > Tools->Copy
>> > > > > > > >on this page you can just fill in the italic portions
>>with
>> > > your
>> > > > > > > details.
>> > > > > > > >- I retrofitted KIP-1 to match this formatting
>> > > > > > > >- I added the metadata section people asked for (a link
>> to the
>> > > > > > > >discussion, the JIRA, and the current status). Let's
>>make
>> > > sure we
>> > > > > > > remember
>> > > > > > > >to update the current status as things are figured out.
>> > > > > > > >- Let's keep the discussion on the mailing list rather
>> than
>> > > on the
>> > > > > > > wiki
>> > > > > > > >pages. It makes sense to do one or the other so all the
>> > > comments
>> > > > > are
>> > > > > > > in one
>> > > > > > > >place and I think prior experience is that the wiki
>> comments
>> > > are
>> > > > > the
>> > > > > > > worse
>> > > > > > > >way.
>> > > > > > > >
>> > > > > > > > I think it would be great do KIPs for some of the
>>in-flight
>> items
>> > > > > folks
>> > > > > > > > mentioned.
>> > > > > > > >
>> > > > > > > > -Jay
>> > > > > > > >
>> > > > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
>> > > gshap...@cloudera.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > +1
>> > > > > > > > >
>> > > > > > > > > Will be happy to provide a KIP for the
>>multiple-listeners
>> > > patch.
>> > > > > > > > >
>> > > > > > > > > Gwen
>> > > > > > > > >
>> > > > > > > > > On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <
>> > > joe.st...@stealth.ly>
>> > > > > > > wrote:
>> > > > > > > > > > +1 to everything we have been saying and where this
>>(has
>> > > settled
>> > > > > > > to)/(is
>> > > > > > > > > > settling to).
>> > > > > > > > > >
>> > > > > > > > > > I am sure other folks have some more feedback and
>>think
>> we
>> > > should
>> > > > > > > try to
>> > > > > > > > > > keep this discussion going if need be. I am also a
>>firm
>> > > believer
>> > > > >

Re: [DISCUSS] KIPs

[Harsha]

Joel,
   Having only 2 or 3 KIPS under active discussion is concerning.
   This will slow down development process as well.
Having a turn-around time for a KIP is a good idea but what will happen
if it didn't received required votes within that time frame.
Its probably more helpful for contributors if its "lazy" as in "no
strong objections" .
Just to make sure this is only for KIPs not for regular bug fixes right?
Thanks,
Harsha



   

On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
> I¹m having an impression that KIP is mostly for new features but not for
> bug fixes. But I agree with Joel that it might make sense to have some
> big
> patches, even if they are bug fixes, to follow the KIP like process which
> is more strict.
> 
> Jiangjie (Becket) Qin
> 
> On 2/5/15, 4:57 PM, "Gwen Shapira"  wrote:
> 
> >>
> >>
> >> Yes there are KIPs that are currently blocked on feedback/votes, but I
> >> don't think it is an issue of not caring to comment vs having so many
> >> KIPs and other code reviews in flight that people are just swamped.
> >>
> >>
> >This is exactly my concern.
> >Even now important patches and features have very long development and
> >review cycles due to Kafka's small and very busy committer community. I
> >feel that this change takes things in the wrong direction
> >
> >
> >> Joel
> >>
> >> On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
> >> > Isn't requiring 3 binding votes a bit overly strict here? We are
> >>talking
> >> > about patches which in can be fixed, reverted, etc. Not releases,
> >>which
> >> > have legal implications.
> >> >
> >> > Why not go with usual definition: "lazy" = "No strong objections for
> >>few
> >> > days"?
> >> > This means contributors will not be blocked on issues where no one
> >>cares
> >> to
> >> > comment (and we had few of those).
> >> >
> >> > Gwen
> >> >
> >> >
> >> >
> >> > On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy 
> >>wrote:
> >> >
> >> > > Sorry about this - I actually meant to suggest lazy consensus (which
> >> > > is a stronger requirement): "3 binding +1 votes and no binding
> >> > > vetoes."
> >> > >
> >> > > I have updated the wiki to lazy consensus - but can change it back
> >>if
> >> > > there is a reasonable objection.
> >> > >
> >> > > On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> >> > > > +1
> >> > > >
> >> > > > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede 
> >> wrote:
> >> > > >
> >> > > > > Sounds good.
> >> > > > >
> >> > > > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps 
> >> wrote:
> >> > > > >
> >> > > > > > None on my part.
> >> > > > > >
> >> > > > > > -Jay
> >> > > > > >
> >> > > > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy
> >> >> >
> >> > > wrote:
> >> > > > > >
> >> > > > > > > One amendment I would like to bring up for consideration wrt
> >> the
> >> > > KIP
> >> > > > > > > process (before we formally include it in our by-laws) is to
> >> not
> >> > > > > > > restrict the votes to be a lazy majority of the PMC, and to
> >> instead
> >> > > > > > > make it a lazy majority of committers.
> >> > > > > > >
> >> > > > > > > Our current requirement for code changes per our by-laws
> >>are +1
> >> > > from a
> >> > > > > > > committer (who is not the contributor) followed by lazy
> >> approval. I
> >> > > > > > > think a lazy majority vote for more significant code changes
> >> > > (i.e., a
> >> > > > > > > KIP) should be sufficient.
> >> > > > > > >
> >> > > > > > > Any objection to this?
> >> > > > > > >
> >> > > > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> >> > > > > > > > Great! Sounds like everyone is on the same page
> >> > > > > > > >
> >> > > > > > > >- I created a template page to make things easier. If
> >>you
> >> do
> >> > > > > > > Tools->Copy
> >> > > > > > > >on this page you can just fill in the italic portions
> >>with
> >> > > your
> >> > > > > > > details.
> >> > > > > > > >- I retrofitted KIP-1 to match this formatting
> >> > > > > > > >- I added the metadata section people asked for (a link
> >> to the
> >> > > > > > > >discussion, the JIRA, and the current status). Let's
> >>make
> >> > > sure we
> >> > > > > > > remember
> >> > > > > > > >to update the current status as things are figured out.
> >> > > > > > > >- Let's keep the discussion on the mailing list rather
> >> than
> >> > > on the
> >> > > > > > > wiki
> >> > > > > > > >pages. It makes sense to do one or the other so all the
> >> > > comments
> >> > > > > are
> >> > > > > > > in one
> >> > > > > > > >place and I think prior experience is that the wiki
> >> comments
> >> > > are
> >> > > > > the
> >> > > > > > > worse
> >> > > > > > > >way.
> >> > > > > > > >
> >> > > > > > > > I think it would be great do KIPs for some of the
> >>in-flight
> >> items
> >> > > > > folks
> >> > > > > > > > mentioned.
> >> > > > > > > >
> >> > > > > > > > -Jay
> >> > > > > > > >
> >> > > > > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
> >> > > 

Re: [DISCUSS] KIPs

[Joel Koshy]

I'm just thinking aloud - I don't know what a good number would be, and it
is just one possibility to streamline how KIPs are processed. It largely
depends on how complex the proposals are. What would be concerning is if
there are 10 different threads all dealing with large KIPs and no one has
the time to give due diligence to each one and all those threads grind to a
halt due to confusion, incomplete context and misunderstandings.

On Thursday, February 5, 2015, Harsha  wrote:

> Joel,
>Having only 2 or 3 KIPS under active discussion is concerning.
>This will slow down development process as well.
> Having a turn-around time for a KIP is a good idea but what will happen
> if it didn't received required votes within that time frame.
> Its probably more helpful for contributors if its "lazy" as in "no
> strong objections" .
> Just to make sure this is only for KIPs not for regular bug fixes right?
> Thanks,
> Harsha
>
>
>
>
>
> On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
> > I¹m having an impression that KIP is mostly for new features but not for
> > bug fixes. But I agree with Joel that it might make sense to have some
> > big
> > patches, even if they are bug fixes, to follow the KIP like process which
> > is more strict.
> >
> > Jiangjie (Becket) Qin
> >
> > On 2/5/15, 4:57 PM, "Gwen Shapira" >
> wrote:
> >
> > >>
> > >>
> > >> Yes there are KIPs that are currently blocked on feedback/votes, but I
> > >> don't think it is an issue of not caring to comment vs having so many
> > >> KIPs and other code reviews in flight that people are just swamped.
> > >>
> > >>
> > >This is exactly my concern.
> > >Even now important patches and features have very long development and
> > >review cycles due to Kafka's small and very busy committer community. I
> > >feel that this change takes things in the wrong direction
> > >
> > >
> > >> Joel
> > >>
> > >> On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
> > >> > Isn't requiring 3 binding votes a bit overly strict here? We are
> > >>talking
> > >> > about patches which in can be fixed, reverted, etc. Not releases,
> > >>which
> > >> > have legal implications.
> > >> >
> > >> > Why not go with usual definition: "lazy" = "No strong objections for
> > >>few
> > >> > days"?
> > >> > This means contributors will not be blocked on issues where no one
> > >>cares
> > >> to
> > >> > comment (and we had few of those).
> > >> >
> > >> > Gwen
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  >
> > >>wrote:
> > >> >
> > >> > > Sorry about this - I actually meant to suggest lazy consensus
> (which
> > >> > > is a stronger requirement): "3 binding +1 votes and no binding
> > >> > > vetoes."
> > >> > >
> > >> > > I have updated the wiki to lazy consensus - but can change it back
> > >>if
> > >> > > there is a reasonable objection.
> > >> > >
> > >> > > On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
> > >> > > > +1
> > >> > > >
> > >> > > > On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede <
> n...@confluent.io >
> > >> wrote:
> > >> > > >
> > >> > > > > Sounds good.
> > >> > > > >
> > >> > > > > On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps <
> jay.kr...@gmail.com >
> > >> wrote:
> > >> > > > >
> > >> > > > > > None on my part.
> > >> > > > > >
> > >> > > > > > -Jay
> > >> > > > > >
> > >> > > > > > On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy
> > >>
> > >> >
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > > > One amendment I would like to bring up for consideration
> wrt
> > >> the
> > >> > > KIP
> > >> > > > > > > process (before we formally include it in our by-laws) is
> to
> > >> not
> > >> > > > > > > restrict the votes to be a lazy majority of the PMC, and
> to
> > >> instead
> > >> > > > > > > make it a lazy majority of committers.
> > >> > > > > > >
> > >> > > > > > > Our current requirement for code changes per our by-laws
> > >>are +1
> > >> > > from a
> > >> > > > > > > committer (who is not the contributor) followed by lazy
> > >> approval. I
> > >> > > > > > > think a lazy majority vote for more significant code
> changes
> > >> > > (i.e., a
> > >> > > > > > > KIP) should be sufficient.
> > >> > > > > > >
> > >> > > > > > > Any objection to this?
> > >> > > > > > >
> > >> > > > > > > On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
> > >> > > > > > > > Great! Sounds like everyone is on the same page
> > >> > > > > > > >
> > >> > > > > > > >- I created a template page to make things easier. If
> > >>you
> > >> do
> > >> > > > > > > Tools->Copy
> > >> > > > > > > >on this page you can just fill in the italic portions
> > >>with
> > >> > > your
> > >> > > > > > > details.
> > >> > > > > > > >- I retrofitted KIP-1 to match this formatting
> > >> > > > > > > >- I added the metadata section people asked for (a
> link
> > >> to the
> > >> > > > > > > >discussion, the JIRA, and the current status). Let's
> > >>make
> > >> > > sure we
> > >> > > > > > > remember
> > >> > > > >

Re: [DISCUSS] KIPs

[Joel Koshy]

Just wanted to add a few more comments on this: KIPs were suggested as
a process to help reach early consensus on a major change or not so
major (but tricky or backward incompatible) change in order to reduce
the likelihood of multiple iterations and complete rewrites during
code reviews (which is time-intensive for both the contributor and
reviewers); as well as to reduce the likelihood of surprises (say, if
a patch inadvertently changes a public API).  So KIPs are intended to
speed up development since a clear path is charted out and there is
prior consensus on whether a feature and its design/implementation
make sense or not.

Obviously this breaks down if KIPs are not being actively discussed -
again I think we can do much better here. I think we ended up with a
backlog because as soon as the KIP wiki was started, a number of
pre-existing jiras and discussions were moved there - all within a few
days. Now that there are quite a few outstanding KIPs I think we just
need to methodically work through those - preferably a couple at a
time. I looked through the list and I think we should be able to
resolve all of them relatively quickly if everyone is on board with
this.

> > Its probably more helpful for contributors if its "lazy" as in "no
> > strong objections" .

Gwen also suggested this and this also sounds ok to me as I wrote
earlier - what do others think? This is important especially if
majority in the community think if this less restrictive policy would
spur and not hinder development - I'm not sure that it does. I
completely agree that KIPs fail to a large degree as far as the
original motivation goes if they require a lazy majority but the
DISCUSS threads are stalled. IOW regardless of that discussion, I
think we should rejuvenate some of those threads especially now that
0.8.2 is out of the way.

Thanks,

Joel

On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
> I'm just thinking aloud - I don't know what a good number would be, and it
> is just one possibility to streamline how KIPs are processed. It largely
> depends on how complex the proposals are. What would be concerning is if
> there are 10 different threads all dealing with large KIPs and no one has
> the time to give due diligence to each one and all those threads grind to a
> halt due to confusion, incomplete context and misunderstandings.
> 
> On Thursday, February 5, 2015, Harsha  wrote:
> 
> > Joel,
> >Having only 2 or 3 KIPS under active discussion is concerning.
> >This will slow down development process as well.
> > Having a turn-around time for a KIP is a good idea but what will happen
> > if it didn't received required votes within that time frame.
> > Its probably more helpful for contributors if its "lazy" as in "no
> > strong objections" .
> > Just to make sure this is only for KIPs not for regular bug fixes right?
> > Thanks,
> > Harsha
> >
> >
> >
> >
> >
> > On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
> > > I¹m having an impression that KIP is mostly for new features but not for
> > > bug fixes. But I agree with Joel that it might make sense to have some
> > > big
> > > patches, even if they are bug fixes, to follow the KIP like process which
> > > is more strict.
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On 2/5/15, 4:57 PM, "Gwen Shapira" >
> > wrote:
> > >
> > > >>
> > > >>
> > > >> Yes there are KIPs that are currently blocked on feedback/votes, but I
> > > >> don't think it is an issue of not caring to comment vs having so many
> > > >> KIPs and other code reviews in flight that people are just swamped.
> > > >>
> > > >>
> > > >This is exactly my concern.
> > > >Even now important patches and features have very long development and
> > > >review cycles due to Kafka's small and very busy committer community. I
> > > >feel that this change takes things in the wrong direction
> > > >
> > > >
> > > >> Joel
> > > >>
> > > >> On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
> > > >> > Isn't requiring 3 binding votes a bit overly strict here? We are
> > > >>talking
> > > >> > about patches which in can be fixed, reverted, etc. Not releases,
> > > >>which
> > > >> > have legal implications.
> > > >> >
> > > >> > Why not go with usual definition: "lazy" = "No strong objections for
> > > >>few
> > > >> > days"?
> > > >> > This means contributors will not be blocked on issues where no one
> > > >>cares
> > > >> to
> > > >> > comment (and we had few of those).
> > > >> >
> > > >> > Gwen
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy  > >
> > > >>wrote:
> > > >> >
> > > >> > > Sorry about this - I actually meant to suggest lazy consensus
> > (which
> > > >> > > is a stronger requirement): "3 binding +1 votes and no binding
> > > >> > > vetoes."
> > > >> > >
> > > >> > > I have updated the wiki to lazy consensus - but can change it back
> > > >>if
> > > >> > > there is a reasonable objection.
> > > >> > >
> > > >> > > On Thu, Feb 05, 2015 

Re: [DISCUSS] KIPs

[Guozhang Wang]

Here are my cents:

We could use "lazy" as the default policy for KIP, i.e. :

1. Once a KIP is created at least one committer should engage and provide
feedbacks within a few days;
2. If the committer(s) think it may be a bigger issue / change and requires
broader discussion, she(they) may call out to upgrade to "majority" or even
"consensus" and urge other committers to get involved.

Hence it is committers' responsibility to guarantee step 1).

Guozhang

On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy  wrote:

> Just wanted to add a few more comments on this: KIPs were suggested as
> a process to help reach early consensus on a major change or not so
> major (but tricky or backward incompatible) change in order to reduce
> the likelihood of multiple iterations and complete rewrites during
> code reviews (which is time-intensive for both the contributor and
> reviewers); as well as to reduce the likelihood of surprises (say, if
> a patch inadvertently changes a public API).  So KIPs are intended to
> speed up development since a clear path is charted out and there is
> prior consensus on whether a feature and its design/implementation
> make sense or not.
>
> Obviously this breaks down if KIPs are not being actively discussed -
> again I think we can do much better here. I think we ended up with a
> backlog because as soon as the KIP wiki was started, a number of
> pre-existing jiras and discussions were moved there - all within a few
> days. Now that there are quite a few outstanding KIPs I think we just
> need to methodically work through those - preferably a couple at a
> time. I looked through the list and I think we should be able to
> resolve all of them relatively quickly if everyone is on board with
> this.
>
> > > Its probably more helpful for contributors if its "lazy" as in "no
> > > strong objections" .
>
> Gwen also suggested this and this also sounds ok to me as I wrote
> earlier - what do others think? This is important especially if
> majority in the community think if this less restrictive policy would
> spur and not hinder development - I'm not sure that it does. I
> completely agree that KIPs fail to a large degree as far as the
> original motivation goes if they require a lazy majority but the
> DISCUSS threads are stalled. IOW regardless of that discussion, I
> think we should rejuvenate some of those threads especially now that
> 0.8.2 is out of the way.
>
> Thanks,
>
> Joel
>
> On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
> > I'm just thinking aloud - I don't know what a good number would be, and
> it
> > is just one possibility to streamline how KIPs are processed. It largely
> > depends on how complex the proposals are. What would be concerning is if
> > there are 10 different threads all dealing with large KIPs and no one has
> > the time to give due diligence to each one and all those threads grind
> to a
> > halt due to confusion, incomplete context and misunderstandings.
> >
> > On Thursday, February 5, 2015, Harsha  wrote:
> >
> > > Joel,
> > >Having only 2 or 3 KIPS under active discussion is concerning.
> > >This will slow down development process as well.
> > > Having a turn-around time for a KIP is a good idea but what will happen
> > > if it didn't received required votes within that time frame.
> > > Its probably more helpful for contributors if its "lazy" as in "no
> > > strong objections" .
> > > Just to make sure this is only for KIPs not for regular bug fixes
> right?
> > > Thanks,
> > > Harsha
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
> > > > Iąm having an impression that KIP is mostly for new features but not
> for
> > > > bug fixes. But I agree with Joel that it might make sense to have
> some
> > > > big
> > > > patches, even if they are bug fixes, to follow the KIP like process
> which
> > > > is more strict.
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On 2/5/15, 4:57 PM, "Gwen Shapira"  >
> > > wrote:
> > > >
> > > > >>
> > > > >>
> > > > >> Yes there are KIPs that are currently blocked on feedback/votes,
> but I
> > > > >> don't think it is an issue of not caring to comment vs having so
> many
> > > > >> KIPs and other code reviews in flight that people are just
> swamped.
> > > > >>
> > > > >>
> > > > >This is exactly my concern.
> > > > >Even now important patches and features have very long development
> and
> > > > >review cycles due to Kafka's small and very busy committer
> community. I
> > > > >feel that this change takes things in the wrong direction
> > > > >
> > > > >
> > > > >> Joel
> > > > >>
> > > > >> On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
> > > > >> > Isn't requiring 3 binding votes a bit overly strict here? We are
> > > > >>talking
> > > > >> > about patches which in can be fixed, reverted, etc. Not
> releases,
> > > > >>which
> > > > >> > have legal implications.
> > > > >> >
> > > > >> > Why not go with usual definition: "lazy" = "No 

Re: [DISCUSS] KIPs

[Joe Stein]

Guozhang's idea is interesting. It does promote collaboration.

Not all KIPs are created equally and anyone (including the person reviewing
it) would be able to call out and request a stronger vote for it to go in.

I think that having 3 x +1 is good (sometimes) because it reduces the risk
of things getting lost in the shuffle and missing changes, which does, will
and always happen.

We also need to discuss 0.8.3.0, 0.9.0.0 and such
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan it
should all be KIP we think should be in where, right?

- Joe Stein

On Fri, Feb 6, 2015 at 2:59 AM, Guozhang Wang  wrote:

> Here are my cents:
>
> We could use "lazy" as the default policy for KIP, i.e. :
>
> 1. Once a KIP is created at least one committer should engage and provide
> feedbacks within a few days;
> 2. If the committer(s) think it may be a bigger issue / change and requires
> broader discussion, she(they) may call out to upgrade to "majority" or even
> "consensus" and urge other committers to get involved.
>
> Hence it is committers' responsibility to guarantee step 1).
>
> Guozhang
>
> On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy  wrote:
>
> > Just wanted to add a few more comments on this: KIPs were suggested as
> > a process to help reach early consensus on a major change or not so
> > major (but tricky or backward incompatible) change in order to reduce
> > the likelihood of multiple iterations and complete rewrites during
> > code reviews (which is time-intensive for both the contributor and
> > reviewers); as well as to reduce the likelihood of surprises (say, if
> > a patch inadvertently changes a public API).  So KIPs are intended to
> > speed up development since a clear path is charted out and there is
> > prior consensus on whether a feature and its design/implementation
> > make sense or not.
> >
> > Obviously this breaks down if KIPs are not being actively discussed -
> > again I think we can do much better here. I think we ended up with a
> > backlog because as soon as the KIP wiki was started, a number of
> > pre-existing jiras and discussions were moved there - all within a few
> > days. Now that there are quite a few outstanding KIPs I think we just
> > need to methodically work through those - preferably a couple at a
> > time. I looked through the list and I think we should be able to
> > resolve all of them relatively quickly if everyone is on board with
> > this.
> >
> > > > Its probably more helpful for contributors if its "lazy" as in "no
> > > > strong objections" .
> >
> > Gwen also suggested this and this also sounds ok to me as I wrote
> > earlier - what do others think? This is important especially if
> > majority in the community think if this less restrictive policy would
> > spur and not hinder development - I'm not sure that it does. I
> > completely agree that KIPs fail to a large degree as far as the
> > original motivation goes if they require a lazy majority but the
> > DISCUSS threads are stalled. IOW regardless of that discussion, I
> > think we should rejuvenate some of those threads especially now that
> > 0.8.2 is out of the way.
> >
> > Thanks,
> >
> > Joel
> >
> > On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
> > > I'm just thinking aloud - I don't know what a good number would be, and
> > it
> > > is just one possibility to streamline how KIPs are processed. It
> largely
> > > depends on how complex the proposals are. What would be concerning is
> if
> > > there are 10 different threads all dealing with large KIPs and no one
> has
> > > the time to give due diligence to each one and all those threads grind
> > to a
> > > halt due to confusion, incomplete context and misunderstandings.
> > >
> > > On Thursday, February 5, 2015, Harsha  wrote:
> > >
> > > > Joel,
> > > >Having only 2 or 3 KIPS under active discussion is concerning.
> > > >This will slow down development process as well.
> > > > Having a turn-around time for a KIP is a good idea but what will
> happen
> > > > if it didn't received required votes within that time frame.
> > > > Its probably more helpful for contributors if its "lazy" as in "no
> > > > strong objections" .
> > > > Just to make sure this is only for KIPs not for regular bug fixes
> > right?
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
> > > > > Iąm having an impression that KIP is mostly for new features but
> not
> > for
> > > > > bug fixes. But I agree with Joel that it might make sense to have
> > some
> > > > > big
> > > > > patches, even if they are bug fixes, to follow the KIP like process
> > which
> > > > > is more strict.
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On 2/5/15, 4:57 PM, "Gwen Shapira"  > >
> > > > wrote:
> > > > >
> > > > > >>
> > > > > >>
> > > > > >> Yes there are KIPs that are currently blocked on feedback/votes,
> > but I
> > > > > >> don't think it is an issu

Re: [DISCUSS] KIPs

[Jay Kreps]

I think we are focused on making committing new changes easier, but what we
have seen is actually that isn't the bulk of the work (especially with this
kind of "public interface" change where it generally has a big user
impact). I think we actually really need the core committers and any other
interested parties to stop and fully read each KIP and think about it. If
we don't have time to do that we usually just end up spending a lot more
time after the fact trying to rework things latter when it is a lot harder.
So I really think we should have every active committer read, comment, and
vote on each KIP. I think this may require a little bit of work to
co-ordinate/bug people but will end up being worth it because each person
on the project will have a holistic picture of what is going on.

-Jay

On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy  wrote:

> Just wanted to add a few more comments on this: KIPs were suggested as
> a process to help reach early consensus on a major change or not so
> major (but tricky or backward incompatible) change in order to reduce
> the likelihood of multiple iterations and complete rewrites during
> code reviews (which is time-intensive for both the contributor and
> reviewers); as well as to reduce the likelihood of surprises (say, if
> a patch inadvertently changes a public API).  So KIPs are intended to
> speed up development since a clear path is charted out and there is
> prior consensus on whether a feature and its design/implementation
> make sense or not.
>
> Obviously this breaks down if KIPs are not being actively discussed -
> again I think we can do much better here. I think we ended up with a
> backlog because as soon as the KIP wiki was started, a number of
> pre-existing jiras and discussions were moved there - all within a few
> days. Now that there are quite a few outstanding KIPs I think we just
> need to methodically work through those - preferably a couple at a
> time. I looked through the list and I think we should be able to
> resolve all of them relatively quickly if everyone is on board with
> this.
>
> > > Its probably more helpful for contributors if its "lazy" as in "no
> > > strong objections" .
>
> Gwen also suggested this and this also sounds ok to me as I wrote
> earlier - what do others think? This is important especially if
> majority in the community think if this less restrictive policy would
> spur and not hinder development - I'm not sure that it does. I
> completely agree that KIPs fail to a large degree as far as the
> original motivation goes if they require a lazy majority but the
> DISCUSS threads are stalled. IOW regardless of that discussion, I
> think we should rejuvenate some of those threads especially now that
> 0.8.2 is out of the way.
>
> Thanks,
>
> Joel
>
> On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
> > I'm just thinking aloud - I don't know what a good number would be, and
> it
> > is just one possibility to streamline how KIPs are processed. It largely
> > depends on how complex the proposals are. What would be concerning is if
> > there are 10 different threads all dealing with large KIPs and no one has
> > the time to give due diligence to each one and all those threads grind
> to a
> > halt due to confusion, incomplete context and misunderstandings.
> >
> > On Thursday, February 5, 2015, Harsha  wrote:
> >
> > > Joel,
> > >Having only 2 or 3 KIPS under active discussion is concerning.
> > >This will slow down development process as well.
> > > Having a turn-around time for a KIP is a good idea but what will happen
> > > if it didn't received required votes within that time frame.
> > > Its probably more helpful for contributors if its "lazy" as in "no
> > > strong objections" .
> > > Just to make sure this is only for KIPs not for regular bug fixes
> right?
> > > Thanks,
> > > Harsha
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
> > > > Iąm having an impression that KIP is mostly for new features but not
> for
> > > > bug fixes. But I agree with Joel that it might make sense to have
> some
> > > > big
> > > > patches, even if they are bug fixes, to follow the KIP like process
> which
> > > > is more strict.
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On 2/5/15, 4:57 PM, "Gwen Shapira"  >
> > > wrote:
> > > >
> > > > >>
> > > > >>
> > > > >> Yes there are KIPs that are currently blocked on feedback/votes,
> but I
> > > > >> don't think it is an issue of not caring to comment vs having so
> many
> > > > >> KIPs and other code reviews in flight that people are just
> swamped.
> > > > >>
> > > > >>
> > > > >This is exactly my concern.
> > > > >Even now important patches and features have very long development
> and
> > > > >review cycles due to Kafka's small and very busy committer
> community. I
> > > > >feel that this change takes things in the wrong direction
> > > > >
> > > > >
> > > > >> Joel
> > > > >>
> > > > >> On Thu, Feb 05, 2015 a

Re: [DISCUSS] KIPs

[Jay Kreps]

A problem I am having is actually understanding which KIPs are intended to
be complete proposals and which are works in progress. Joe you seem to have
a bunch of these. Can you move them elsewhere until they are really fully
done and ready for review and discussion?

-Jay

On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps  wrote:

> I think we are focused on making committing new changes easier, but what
> we have seen is actually that isn't the bulk of the work (especially with
> this kind of "public interface" change where it generally has a big user
> impact). I think we actually really need the core committers and any other
> interested parties to stop and fully read each KIP and think about it. If
> we don't have time to do that we usually just end up spending a lot more
> time after the fact trying to rework things latter when it is a lot harder.
> So I really think we should have every active committer read, comment, and
> vote on each KIP. I think this may require a little bit of work to
> co-ordinate/bug people but will end up being worth it because each person
> on the project will have a holistic picture of what is going on.
>
> -Jay
>
> On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy  wrote:
>
>> Just wanted to add a few more comments on this: KIPs were suggested as
>> a process to help reach early consensus on a major change or not so
>> major (but tricky or backward incompatible) change in order to reduce
>> the likelihood of multiple iterations and complete rewrites during
>> code reviews (which is time-intensive for both the contributor and
>> reviewers); as well as to reduce the likelihood of surprises (say, if
>> a patch inadvertently changes a public API).  So KIPs are intended to
>> speed up development since a clear path is charted out and there is
>> prior consensus on whether a feature and its design/implementation
>> make sense or not.
>>
>> Obviously this breaks down if KIPs are not being actively discussed -
>> again I think we can do much better here. I think we ended up with a
>> backlog because as soon as the KIP wiki was started, a number of
>> pre-existing jiras and discussions were moved there - all within a few
>> days. Now that there are quite a few outstanding KIPs I think we just
>> need to methodically work through those - preferably a couple at a
>> time. I looked through the list and I think we should be able to
>> resolve all of them relatively quickly if everyone is on board with
>> this.
>>
>> > > Its probably more helpful for contributors if its "lazy" as in "no
>> > > strong objections" .
>>
>> Gwen also suggested this and this also sounds ok to me as I wrote
>> earlier - what do others think? This is important especially if
>> majority in the community think if this less restrictive policy would
>> spur and not hinder development - I'm not sure that it does. I
>> completely agree that KIPs fail to a large degree as far as the
>> original motivation goes if they require a lazy majority but the
>> DISCUSS threads are stalled. IOW regardless of that discussion, I
>> think we should rejuvenate some of those threads especially now that
>> 0.8.2 is out of the way.
>>
>> Thanks,
>>
>> Joel
>>
>> On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
>> > I'm just thinking aloud - I don't know what a good number would be, and
>> it
>> > is just one possibility to streamline how KIPs are processed. It largely
>> > depends on how complex the proposals are. What would be concerning is if
>> > there are 10 different threads all dealing with large KIPs and no one
>> has
>> > the time to give due diligence to each one and all those threads grind
>> to a
>> > halt due to confusion, incomplete context and misunderstandings.
>> >
>> > On Thursday, February 5, 2015, Harsha  wrote:
>> >
>> > > Joel,
>> > >Having only 2 or 3 KIPS under active discussion is concerning.
>> > >This will slow down development process as well.
>> > > Having a turn-around time for a KIP is a good idea but what will
>> happen
>> > > if it didn't received required votes within that time frame.
>> > > Its probably more helpful for contributors if its "lazy" as in "no
>> > > strong objections" .
>> > > Just to make sure this is only for KIPs not for regular bug fixes
>> right?
>> > > Thanks,
>> > > Harsha
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
>> > > > Iąm having an impression that KIP is mostly for new features but
>> not for
>> > > > bug fixes. But I agree with Joel that it might make sense to have
>> some
>> > > > big
>> > > > patches, even if they are bug fixes, to follow the KIP like process
>> which
>> > > > is more strict.
>> > > >
>> > > > Jiangjie (Becket) Qin
>> > > >
>> > > > On 2/5/15, 4:57 PM, "Gwen Shapira" > >
>> > > wrote:
>> > > >
>> > > > >>
>> > > > >>
>> > > > >> Yes there are KIPs that are currently blocked on feedback/votes,
>> but I
>> > > > >> don't think it is an issue of not caring to comment vs having so
>> many
>> 

Re: [DISCUSS] KIPs

[Joel Koshy]

I'm looking through a couple of the KIP threads today and had the same
issue; and thought it would be useful to do a status round-up of KIPs.
We could incorporate status in the title itself (so we can quickly see
it in the child-page list) but I just added a table to the top-level
wiki. Hopefully that captures the current state accurately so I know
which threads to follow-up on.

Thanks,

Joel

On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
> A problem I am having is actually understanding which KIPs are intended to
> be complete proposals and which are works in progress. Joe you seem to have
> a bunch of these. Can you move them elsewhere until they are really fully
> done and ready for review and discussion?
> 
> -Jay
> 
> On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps  wrote:
> 
> > I think we are focused on making committing new changes easier, but what
> > we have seen is actually that isn't the bulk of the work (especially with
> > this kind of "public interface" change where it generally has a big user
> > impact). I think we actually really need the core committers and any other
> > interested parties to stop and fully read each KIP and think about it. If
> > we don't have time to do that we usually just end up spending a lot more
> > time after the fact trying to rework things latter when it is a lot harder.
> > So I really think we should have every active committer read, comment, and
> > vote on each KIP. I think this may require a little bit of work to
> > co-ordinate/bug people but will end up being worth it because each person
> > on the project will have a holistic picture of what is going on.
> >
> > -Jay
> >
> > On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy  wrote:
> >
> >> Just wanted to add a few more comments on this: KIPs were suggested as
> >> a process to help reach early consensus on a major change or not so
> >> major (but tricky or backward incompatible) change in order to reduce
> >> the likelihood of multiple iterations and complete rewrites during
> >> code reviews (which is time-intensive for both the contributor and
> >> reviewers); as well as to reduce the likelihood of surprises (say, if
> >> a patch inadvertently changes a public API).  So KIPs are intended to
> >> speed up development since a clear path is charted out and there is
> >> prior consensus on whether a feature and its design/implementation
> >> make sense or not.
> >>
> >> Obviously this breaks down if KIPs are not being actively discussed -
> >> again I think we can do much better here. I think we ended up with a
> >> backlog because as soon as the KIP wiki was started, a number of
> >> pre-existing jiras and discussions were moved there - all within a few
> >> days. Now that there are quite a few outstanding KIPs I think we just
> >> need to methodically work through those - preferably a couple at a
> >> time. I looked through the list and I think we should be able to
> >> resolve all of them relatively quickly if everyone is on board with
> >> this.
> >>
> >> > > Its probably more helpful for contributors if its "lazy" as in "no
> >> > > strong objections" .
> >>
> >> Gwen also suggested this and this also sounds ok to me as I wrote
> >> earlier - what do others think? This is important especially if
> >> majority in the community think if this less restrictive policy would
> >> spur and not hinder development - I'm not sure that it does. I
> >> completely agree that KIPs fail to a large degree as far as the
> >> original motivation goes if they require a lazy majority but the
> >> DISCUSS threads are stalled. IOW regardless of that discussion, I
> >> think we should rejuvenate some of those threads especially now that
> >> 0.8.2 is out of the way.
> >>
> >> Thanks,
> >>
> >> Joel
> >>
> >> On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
> >> > I'm just thinking aloud - I don't know what a good number would be, and
> >> it
> >> > is just one possibility to streamline how KIPs are processed. It largely
> >> > depends on how complex the proposals are. What would be concerning is if
> >> > there are 10 different threads all dealing with large KIPs and no one
> >> has
> >> > the time to give due diligence to each one and all those threads grind
> >> to a
> >> > halt due to confusion, incomplete context and misunderstandings.
> >> >
> >> > On Thursday, February 5, 2015, Harsha  wrote:
> >> >
> >> > > Joel,
> >> > >Having only 2 or 3 KIPS under active discussion is concerning.
> >> > >This will slow down development process as well.
> >> > > Having a turn-around time for a KIP is a good idea but what will
> >> happen
> >> > > if it didn't received required votes within that time frame.
> >> > > Its probably more helpful for contributors if its "lazy" as in "no
> >> > > strong objections" .
> >> > > Just to make sure this is only for KIPs not for regular bug fixes
> >> right?
> >> > > Thanks,
> >> > > Harsha
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Feb 5, 2015

Re: [DISCUSS] KIPs

[Joe Stein]

I like the round-up, looks good, thanks Joel.

I should be able to get KIP-5 and KIP-6 to have more detail in the coming
days.

~ Joestein

On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy  wrote:

> I'm looking through a couple of the KIP threads today and had the same
> issue; and thought it would be useful to do a status round-up of KIPs.
> We could incorporate status in the title itself (so we can quickly see
> it in the child-page list) but I just added a table to the top-level
> wiki. Hopefully that captures the current state accurately so I know
> which threads to follow-up on.
>
> Thanks,
>
> Joel
>
> On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
> > A problem I am having is actually understanding which KIPs are intended
> to
> > be complete proposals and which are works in progress. Joe you seem to
> have
> > a bunch of these. Can you move them elsewhere until they are really fully
> > done and ready for review and discussion?
> >
> > -Jay
> >
> > On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps  wrote:
> >
> > > I think we are focused on making committing new changes easier, but
> what
> > > we have seen is actually that isn't the bulk of the work (especially
> with
> > > this kind of "public interface" change where it generally has a big
> user
> > > impact). I think we actually really need the core committers and any
> other
> > > interested parties to stop and fully read each KIP and think about it.
> If
> > > we don't have time to do that we usually just end up spending a lot
> more
> > > time after the fact trying to rework things latter when it is a lot
> harder.
> > > So I really think we should have every active committer read, comment,
> and
> > > vote on each KIP. I think this may require a little bit of work to
> > > co-ordinate/bug people but will end up being worth it because each
> person
> > > on the project will have a holistic picture of what is going on.
> > >
> > > -Jay
> > >
> > > On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy 
> wrote:
> > >
> > >> Just wanted to add a few more comments on this: KIPs were suggested as
> > >> a process to help reach early consensus on a major change or not so
> > >> major (but tricky or backward incompatible) change in order to reduce
> > >> the likelihood of multiple iterations and complete rewrites during
> > >> code reviews (which is time-intensive for both the contributor and
> > >> reviewers); as well as to reduce the likelihood of surprises (say, if
> > >> a patch inadvertently changes a public API).  So KIPs are intended to
> > >> speed up development since a clear path is charted out and there is
> > >> prior consensus on whether a feature and its design/implementation
> > >> make sense or not.
> > >>
> > >> Obviously this breaks down if KIPs are not being actively discussed -
> > >> again I think we can do much better here. I think we ended up with a
> > >> backlog because as soon as the KIP wiki was started, a number of
> > >> pre-existing jiras and discussions were moved there - all within a few
> > >> days. Now that there are quite a few outstanding KIPs I think we just
> > >> need to methodically work through those - preferably a couple at a
> > >> time. I looked through the list and I think we should be able to
> > >> resolve all of them relatively quickly if everyone is on board with
> > >> this.
> > >>
> > >> > > Its probably more helpful for contributors if its "lazy" as in "no
> > >> > > strong objections" .
> > >>
> > >> Gwen also suggested this and this also sounds ok to me as I wrote
> > >> earlier - what do others think? This is important especially if
> > >> majority in the community think if this less restrictive policy would
> > >> spur and not hinder development - I'm not sure that it does. I
> > >> completely agree that KIPs fail to a large degree as far as the
> > >> original motivation goes if they require a lazy majority but the
> > >> DISCUSS threads are stalled. IOW regardless of that discussion, I
> > >> think we should rejuvenate some of those threads especially now that
> > >> 0.8.2 is out of the way.
> > >>
> > >> Thanks,
> > >>
> > >> Joel
> > >>
> > >> On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
> > >> > I'm just thinking aloud - I don't know what a good number would be,
> and
> > >> it
> > >> > is just one possibility to streamline how KIPs are processed. It
> largely
> > >> > depends on how complex the proposals are. What would be concerning
> is if
> > >> > there are 10 different threads all dealing with large KIPs and no
> one
> > >> has
> > >> > the time to give due diligence to each one and all those threads
> grind
> > >> to a
> > >> > halt due to confusion, incomplete context and misunderstandings.
> > >> >
> > >> > On Thursday, February 5, 2015, Harsha  wrote:
> > >> >
> > >> > > Joel,
> > >> > >Having only 2 or 3 KIPS under active discussion is
> concerning.
> > >> > >This will slow down development process as well.
> > >> > > Having a turn-around time for a KIP is a

Re: [DISCUSS] KIPs

[Jay Kreps]

Yeah no pressure. I think you added a holding area for incomplete KIPs,
right? I think that is a good idea. We definitely need a place to stash
these while they are getting built out...

-Jay

On Mon, Feb 9, 2015 at 6:10 PM, Joe Stein  wrote:

> I like the round-up, looks good, thanks Joel.
>
> I should be able to get KIP-5 and KIP-6 to have more detail in the coming
> days.
>
> ~ Joestein
>
> On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy  wrote:
>
> > I'm looking through a couple of the KIP threads today and had the same
> > issue; and thought it would be useful to do a status round-up of KIPs.
> > We could incorporate status in the title itself (so we can quickly see
> > it in the child-page list) but I just added a table to the top-level
> > wiki. Hopefully that captures the current state accurately so I know
> > which threads to follow-up on.
> >
> > Thanks,
> >
> > Joel
> >
> > On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
> > > A problem I am having is actually understanding which KIPs are intended
> > to
> > > be complete proposals and which are works in progress. Joe you seem to
> > have
> > > a bunch of these. Can you move them elsewhere until they are really
> fully
> > > done and ready for review and discussion?
> > >
> > > -Jay
> > >
> > > On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps 
> wrote:
> > >
> > > > I think we are focused on making committing new changes easier, but
> > what
> > > > we have seen is actually that isn't the bulk of the work (especially
> > with
> > > > this kind of "public interface" change where it generally has a big
> > user
> > > > impact). I think we actually really need the core committers and any
> > other
> > > > interested parties to stop and fully read each KIP and think about
> it.
> > If
> > > > we don't have time to do that we usually just end up spending a lot
> > more
> > > > time after the fact trying to rework things latter when it is a lot
> > harder.
> > > > So I really think we should have every active committer read,
> comment,
> > and
> > > > vote on each KIP. I think this may require a little bit of work to
> > > > co-ordinate/bug people but will end up being worth it because each
> > person
> > > > on the project will have a holistic picture of what is going on.
> > > >
> > > > -Jay
> > > >
> > > > On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy 
> > wrote:
> > > >
> > > >> Just wanted to add a few more comments on this: KIPs were suggested
> as
> > > >> a process to help reach early consensus on a major change or not so
> > > >> major (but tricky or backward incompatible) change in order to
> reduce
> > > >> the likelihood of multiple iterations and complete rewrites during
> > > >> code reviews (which is time-intensive for both the contributor and
> > > >> reviewers); as well as to reduce the likelihood of surprises (say,
> if
> > > >> a patch inadvertently changes a public API).  So KIPs are intended
> to
> > > >> speed up development since a clear path is charted out and there is
> > > >> prior consensus on whether a feature and its design/implementation
> > > >> make sense or not.
> > > >>
> > > >> Obviously this breaks down if KIPs are not being actively discussed
> -
> > > >> again I think we can do much better here. I think we ended up with a
> > > >> backlog because as soon as the KIP wiki was started, a number of
> > > >> pre-existing jiras and discussions were moved there - all within a
> few
> > > >> days. Now that there are quite a few outstanding KIPs I think we
> just
> > > >> need to methodically work through those - preferably a couple at a
> > > >> time. I looked through the list and I think we should be able to
> > > >> resolve all of them relatively quickly if everyone is on board with
> > > >> this.
> > > >>
> > > >> > > Its probably more helpful for contributors if its "lazy" as in
> "no
> > > >> > > strong objections" .
> > > >>
> > > >> Gwen also suggested this and this also sounds ok to me as I wrote
> > > >> earlier - what do others think? This is important especially if
> > > >> majority in the community think if this less restrictive policy
> would
> > > >> spur and not hinder development - I'm not sure that it does. I
> > > >> completely agree that KIPs fail to a large degree as far as the
> > > >> original motivation goes if they require a lazy majority but the
> > > >> DISCUSS threads are stalled. IOW regardless of that discussion, I
> > > >> think we should rejuvenate some of those threads especially now that
> > > >> 0.8.2 is out of the way.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Joel
> > > >>
> > > >> On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
> > > >> > I'm just thinking aloud - I don't know what a good number would
> be,
> > and
> > > >> it
> > > >> > is just one possibility to streamline how KIPs are processed. It
> > largely
> > > >> > depends on how complex the proposals are. What would be concerning
> > is if
> > > >> > there are 10 different threads all dealing with large KIPs and n

Re: [DISCUSS] KIPs

[Joe Stein]

I did https://cwiki.apache.org/confluence/display/KAFKA/drafts

~ Joestein

On Mon, Feb 9, 2015 at 11:01 PM, Jay Kreps  wrote:

> Yeah no pressure. I think you added a holding area for incomplete KIPs,
> right? I think that is a good idea. We definitely need a place to stash
> these while they are getting built out...
>
> -Jay
>
> On Mon, Feb 9, 2015 at 6:10 PM, Joe Stein  wrote:
>
> > I like the round-up, looks good, thanks Joel.
> >
> > I should be able to get KIP-5 and KIP-6 to have more detail in the coming
> > days.
> >
> > ~ Joestein
> >
> > On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy  wrote:
> >
> > > I'm looking through a couple of the KIP threads today and had the same
> > > issue; and thought it would be useful to do a status round-up of KIPs.
> > > We could incorporate status in the title itself (so we can quickly see
> > > it in the child-page list) but I just added a table to the top-level
> > > wiki. Hopefully that captures the current state accurately so I know
> > > which threads to follow-up on.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
> > > > A problem I am having is actually understanding which KIPs are
> intended
> > > to
> > > > be complete proposals and which are works in progress. Joe you seem
> to
> > > have
> > > > a bunch of these. Can you move them elsewhere until they are really
> > fully
> > > > done and ready for review and discussion?
> > > >
> > > > -Jay
> > > >
> > > > On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps 
> > wrote:
> > > >
> > > > > I think we are focused on making committing new changes easier, but
> > > what
> > > > > we have seen is actually that isn't the bulk of the work
> (especially
> > > with
> > > > > this kind of "public interface" change where it generally has a big
> > > user
> > > > > impact). I think we actually really need the core committers and
> any
> > > other
> > > > > interested parties to stop and fully read each KIP and think about
> > it.
> > > If
> > > > > we don't have time to do that we usually just end up spending a lot
> > > more
> > > > > time after the fact trying to rework things latter when it is a lot
> > > harder.
> > > > > So I really think we should have every active committer read,
> > comment,
> > > and
> > > > > vote on each KIP. I think this may require a little bit of work to
> > > > > co-ordinate/bug people but will end up being worth it because each
> > > person
> > > > > on the project will have a holistic picture of what is going on.
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy 
> > > wrote:
> > > > >
> > > > >> Just wanted to add a few more comments on this: KIPs were
> suggested
> > as
> > > > >> a process to help reach early consensus on a major change or not
> so
> > > > >> major (but tricky or backward incompatible) change in order to
> > reduce
> > > > >> the likelihood of multiple iterations and complete rewrites during
> > > > >> code reviews (which is time-intensive for both the contributor and
> > > > >> reviewers); as well as to reduce the likelihood of surprises (say,
> > if
> > > > >> a patch inadvertently changes a public API).  So KIPs are intended
> > to
> > > > >> speed up development since a clear path is charted out and there
> is
> > > > >> prior consensus on whether a feature and its design/implementation
> > > > >> make sense or not.
> > > > >>
> > > > >> Obviously this breaks down if KIPs are not being actively
> discussed
> > -
> > > > >> again I think we can do much better here. I think we ended up
> with a
> > > > >> backlog because as soon as the KIP wiki was started, a number of
> > > > >> pre-existing jiras and discussions were moved there - all within a
> > few
> > > > >> days. Now that there are quite a few outstanding KIPs I think we
> > just
> > > > >> need to methodically work through those - preferably a couple at a
> > > > >> time. I looked through the list and I think we should be able to
> > > > >> resolve all of them relatively quickly if everyone is on board
> with
> > > > >> this.
> > > > >>
> > > > >> > > Its probably more helpful for contributors if its "lazy" as in
> > "no
> > > > >> > > strong objections" .
> > > > >>
> > > > >> Gwen also suggested this and this also sounds ok to me as I wrote
> > > > >> earlier - what do others think? This is important especially if
> > > > >> majority in the community think if this less restrictive policy
> > would
> > > > >> spur and not hinder development - I'm not sure that it does. I
> > > > >> completely agree that KIPs fail to a large degree as far as the
> > > > >> original motivation goes if they require a lazy majority but the
> > > > >> DISCUSS threads are stalled. IOW regardless of that discussion, I
> > > > >> think we should rejuvenate some of those threads especially now
> that
> > > > >> 0.8.2 is out of the way.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Joel
> > > > >>
> > > > >> On Thu, Feb 05, 2015 at 08:56

Re: [DISCUSS] KIPs

[Neha Narkhede]

Thanks Joel. The status table is useful.

On Mon, Feb 9, 2015 at 8:25 PM, Joe Stein  wrote:

> I did https://cwiki.apache.org/confluence/display/KAFKA/drafts
>
> ~ Joestein
>
> On Mon, Feb 9, 2015 at 11:01 PM, Jay Kreps  wrote:
>
> > Yeah no pressure. I think you added a holding area for incomplete KIPs,
> > right? I think that is a good idea. We definitely need a place to stash
> > these while they are getting built out...
> >
> > -Jay
> >
> > On Mon, Feb 9, 2015 at 6:10 PM, Joe Stein  wrote:
> >
> > > I like the round-up, looks good, thanks Joel.
> > >
> > > I should be able to get KIP-5 and KIP-6 to have more detail in the
> coming
> > > days.
> > >
> > > ~ Joestein
> > >
> > > On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy 
> wrote:
> > >
> > > > I'm looking through a couple of the KIP threads today and had the
> same
> > > > issue; and thought it would be useful to do a status round-up of
> KIPs.
> > > > We could incorporate status in the title itself (so we can quickly
> see
> > > > it in the child-page list) but I just added a table to the top-level
> > > > wiki. Hopefully that captures the current state accurately so I know
> > > > which threads to follow-up on.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > > On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
> > > > > A problem I am having is actually understanding which KIPs are
> > intended
> > > > to
> > > > > be complete proposals and which are works in progress. Joe you seem
> > to
> > > > have
> > > > > a bunch of these. Can you move them elsewhere until they are really
> > > fully
> > > > > done and ready for review and discussion?
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps 
> > > wrote:
> > > > >
> > > > > > I think we are focused on making committing new changes easier,
> but
> > > > what
> > > > > > we have seen is actually that isn't the bulk of the work
> > (especially
> > > > with
> > > > > > this kind of "public interface" change where it generally has a
> big
> > > > user
> > > > > > impact). I think we actually really need the core committers and
> > any
> > > > other
> > > > > > interested parties to stop and fully read each KIP and think
> about
> > > it.
> > > > If
> > > > > > we don't have time to do that we usually just end up spending a
> lot
> > > > more
> > > > > > time after the fact trying to rework things latter when it is a
> lot
> > > > harder.
> > > > > > So I really think we should have every active committer read,
> > > comment,
> > > > and
> > > > > > vote on each KIP. I think this may require a little bit of work
> to
> > > > > > co-ordinate/bug people but will end up being worth it because
> each
> > > > person
> > > > > > on the project will have a holistic picture of what is going on.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy  >
> > > > wrote:
> > > > > >
> > > > > >> Just wanted to add a few more comments on this: KIPs were
> > suggested
> > > as
> > > > > >> a process to help reach early consensus on a major change or not
> > so
> > > > > >> major (but tricky or backward incompatible) change in order to
> > > reduce
> > > > > >> the likelihood of multiple iterations and complete rewrites
> during
> > > > > >> code reviews (which is time-intensive for both the contributor
> and
> > > > > >> reviewers); as well as to reduce the likelihood of surprises
> (say,
> > > if
> > > > > >> a patch inadvertently changes a public API).  So KIPs are
> intended
> > > to
> > > > > >> speed up development since a clear path is charted out and there
> > is
> > > > > >> prior consensus on whether a feature and its
> design/implementation
> > > > > >> make sense or not.
> > > > > >>
> > > > > >> Obviously this breaks down if KIPs are not being actively
> > discussed
> > > -
> > > > > >> again I think we can do much better here. I think we ended up
> > with a
> > > > > >> backlog because as soon as the KIP wiki was started, a number of
> > > > > >> pre-existing jiras and discussions were moved there - all
> within a
> > > few
> > > > > >> days. Now that there are quite a few outstanding KIPs I think we
> > > just
> > > > > >> need to methodically work through those - preferably a couple
> at a
> > > > > >> time. I looked through the list and I think we should be able to
> > > > > >> resolve all of them relatively quickly if everyone is on board
> > with
> > > > > >> this.
> > > > > >>
> > > > > >> > > Its probably more helpful for contributors if its "lazy" as
> in
> > > "no
> > > > > >> > > strong objections" .
> > > > > >>
> > > > > >> Gwen also suggested this and this also sounds ok to me as I
> wrote
> > > > > >> earlier - what do others think? This is important especially if
> > > > > >> majority in the community think if this less restrictive policy
> > > would
> > > > > >> spur and not hinder development - I'm not sure that it does. I
> > > > > >> completely agree that KIPs fail to a large degree as far as t

Re: [DISCUSS] KIPs

[Joe Stein]

Thanks Jay for kicking this off! I think the confluence page you wrote up
is a great start.


The KIP makes sense to me (at a minimum) if there is going to be any
"breaking change". This way Kafka can continue to grow and blossom and we
have a process in place if we are going to release a thorn ... and when we
do it is *CLEAR* about what and why that is. We can easily document which
KIPs where involved with this release (which I think should get committed
afterwards somewhere so no chance of edit after release). This approach I
had been thinking about also allows changes to occur as they do now as long
as they are backwards compatible.  Hopefully we never need a KIP but when
we do the PMC can vote on it and folks can read the release notes with
*CLEAR* understanding what is going to break their existing setup... at
least that is how I have been thinking about it.


Let me know what you think about this base minimum approach... I hadn't
really thought of the KIP for *ANY* "major change" and I have to think more
about that. I have some other comments for minor items in the confluence
page I will make once I think more about how I feel having a KIP for more
than what I was thinking about already.


I do think we should have "major changes" go through confluence, mailing
list discuss and JIRA but kind of feel we have been doing that already ...
if there are cases where that isn't the case we should highlight and learn
from them and formalize that more if need be.


/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/

On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps  wrote:

> The idea of KIPs came up in a previous discussion but there was no real
> crisp definition of what they were. Here is an attempt at defining a
> process:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> The trick here is to have something light-weight enough that it isn't a
> hassle for small changes, but enough so that changes get the eyeballs of
> the committers and heavy users.
>
> Thoughts?
>
> -Jay
>

Re: [DISCUSS] KIPs

[Jay Kreps]

Hey Joe,

Yeah I guess the question is what is the definition of major? I agree we
definitely don't want to generate a bunch of paperwork. We have enough
problems just getting all the contributions reviewed and checked in in a
timely fashion...

So obviously bug fixes would not apply here.

I think it is also pretty clear that big features should get reviewed and
discussed. To pick on myself, for example, the log compaction work was done
without enough public discussion about how it worked and why (imho). I
hope/claim that enough rigour in thinking about use-cases and operations
and so on was done that it turned out well, but the discussion was just
between a few people with no real public output. This kind of feature is
clearly a big change and something we should discuss.

If we limit ourselves to just the public contracts the KIP introduces the
discussion would just be on the new configs and monitoring without really a
discussion of the design and how it works which is obviously closely
related.

I don't think this should be more work because in practice we are making
wiki pages for any big thing anyway. So this would just be a consistent way
of numbering and structuring these pages. This would also give a clear call
to action: "hey kafka people, come read my wiki and think this through".

I actually thinking the voting aspect is less important. I think it is
generally clear when there is agreement on something and not. So from my
point of view we could actually just eliminate that part if that is too
formal, it just seemed like a good way to formally adopt something.

To address some of your comments from the wiki:

1. This doesn't inhibit someone coming along and putting up a patch. It is
just that when they do if it is a big thing introducing new functionality
we would ask for a little discussion on the basic feature/contracts prior
to code review.

2. We definitely definitely don't want people generating a lot of these
things every time they have an idea that they aren't going to implement. So
this is only applicable to things you absolutely will check in code for. We
also don't want to be making proposals before things are thought through,
which often requires writing the code. So I think the right time for a KIP
is when you are far enough along that you know the issues and tradeoffs but
not so far along that you are going to be totally opposed to any change.
Sometimes that is prior to writing any code and sometimes not until you are
practically done.

The key problem I see this fixing is that there is enough new development
happening that it is pretty hard for everyone to review every line of every
iteration of every patch. But all of us should be fully aware of new
features, the ramifications, the new public interfaces, etc. If we aren't
aware of that we can't really build a holistic system that is beautiful and
consistent across. So the idea is that if you fully review the KIPs you can
be sure that even if you don't know every new line of code, you know the
major changes coming in.

-Jay



On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein  wrote:

> Thanks Jay for kicking this off! I think the confluence page you wrote up
> is a great start.
>
>
> The KIP makes sense to me (at a minimum) if there is going to be any
> "breaking change". This way Kafka can continue to grow and blossom and we
> have a process in place if we are going to release a thorn ... and when we
> do it is *CLEAR* about what and why that is. We can easily document which
> KIPs where involved with this release (which I think should get committed
> afterwards somewhere so no chance of edit after release). This approach I
> had been thinking about also allows changes to occur as they do now as long
> as they are backwards compatible.  Hopefully we never need a KIP but when
> we do the PMC can vote on it and folks can read the release notes with
> *CLEAR* understanding what is going to break their existing setup... at
> least that is how I have been thinking about it.
>
>
> Let me know what you think about this base minimum approach... I hadn't
> really thought of the KIP for *ANY* "major change" and I have to think more
> about that. I have some other comments for minor items in the confluence
> page I will make once I think more about how I feel having a KIP for more
> than what I was thinking about already.
>
>
> I do think we should have "major changes" go through confluence, mailing
> list discuss and JIRA but kind of feel we have been doing that already ...
> if there are cases where that isn't the case we should highlight and learn
> from them and formalize that more if need be.
>
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
> On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps  wrote:
>
> > The idea of KIPs 

Re: [DISCUSS] KIPs

[Joel Koshy]

I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
doing this in one form or the other. However, IMO it is fairly ad hoc
- i.e., a combination of DISCUSS threads, jira comments, RB and code
comments, wikis and html documentation. In the past I have had to dig
into a bunch of these to try and figure out why something was
implemented a certain way. I think KIPs can help a lot here first by
providing guidelines on what to think about (compatibility, new APIs,
etc.) when working through a major feature; and second by becoming a
crisp source of truth documentation for new releases.  E.g., for
feature X: see relevant KIPs: a, b, c, etc.

On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
> Hey Joe,
> 
> Yeah I guess the question is what is the definition of major? I agree we
> definitely don't want to generate a bunch of paperwork. We have enough
> problems just getting all the contributions reviewed and checked in in a
> timely fashion...
> 
> So obviously bug fixes would not apply here.
> 
> I think it is also pretty clear that big features should get reviewed and
> discussed. To pick on myself, for example, the log compaction work was done
> without enough public discussion about how it worked and why (imho). I
> hope/claim that enough rigour in thinking about use-cases and operations
> and so on was done that it turned out well, but the discussion was just
> between a few people with no real public output. This kind of feature is
> clearly a big change and something we should discuss.
> 
> If we limit ourselves to just the public contracts the KIP introduces the
> discussion would just be on the new configs and monitoring without really a
> discussion of the design and how it works which is obviously closely
> related.
> 
> I don't think this should be more work because in practice we are making
> wiki pages for any big thing anyway. So this would just be a consistent way
> of numbering and structuring these pages. This would also give a clear call
> to action: "hey kafka people, come read my wiki and think this through".
> 
> I actually thinking the voting aspect is less important. I think it is
> generally clear when there is agreement on something and not. So from my
> point of view we could actually just eliminate that part if that is too
> formal, it just seemed like a good way to formally adopt something.
> 
> To address some of your comments from the wiki:
> 
> 1. This doesn't inhibit someone coming along and putting up a patch. It is
> just that when they do if it is a big thing introducing new functionality
> we would ask for a little discussion on the basic feature/contracts prior
> to code review.
> 
> 2. We definitely definitely don't want people generating a lot of these
> things every time they have an idea that they aren't going to implement. So
> this is only applicable to things you absolutely will check in code for. We
> also don't want to be making proposals before things are thought through,
> which often requires writing the code. So I think the right time for a KIP
> is when you are far enough along that you know the issues and tradeoffs but
> not so far along that you are going to be totally opposed to any change.
> Sometimes that is prior to writing any code and sometimes not until you are
> practically done.
> 
> The key problem I see this fixing is that there is enough new development
> happening that it is pretty hard for everyone to review every line of every
> iteration of every patch. But all of us should be fully aware of new
> features, the ramifications, the new public interfaces, etc. If we aren't
> aware of that we can't really build a holistic system that is beautiful and
> consistent across. So the idea is that if you fully review the KIPs you can
> be sure that even if you don't know every new line of code, you know the
> major changes coming in.
> 
> -Jay
> 
> 
> 
> On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein  wrote:
> 
> > Thanks Jay for kicking this off! I think the confluence page you wrote up
> > is a great start.
> >
> >
> > The KIP makes sense to me (at a minimum) if there is going to be any
> > "breaking change". This way Kafka can continue to grow and blossom and we
> > have a process in place if we are going to release a thorn ... and when we
> > do it is *CLEAR* about what and why that is. We can easily document which
> > KIPs where involved with this release (which I think should get committed
> > afterwards somewhere so no chance of edit after release). This approach I
> > had been thinking about also allows changes to occur as they do now as long
> > as they are backwards compatible.  Hopefully we never need a KIP but when
> > we do the PMC can vote on it and folks can read the release notes with
> > *CLEAR* understanding what is going to break their existing setup... at
> > least that is how I have been thinking about it.
> >
> >
> > Let me know what you think about this base minimum approach... I hadn't
> > really thought of th

Re: [DISCUSS] KIPs

[Ewen Cheslack-Postava]

I think adding a section about deprecation would be helpful. A good
fraction of the time I would expect the goal of a KIP is to fix or replace
older functionality that needs continued support for compatibility, but
should eventually be phased out. This helps Kafka devs understand how long
they'll end up supporting multiple versions of features and helps users
understand when they're going to have to make updates to their applications.

Less important but useful -- having a bit of standard metadata like PEPs
do. Two I think are important are status (if someone lands on the KIP page,
can they tell whether this KIP was ever completed?) and/or the version the
KIP was first released in.



On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy  wrote:

> I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
> doing this in one form or the other. However, IMO it is fairly ad hoc
> - i.e., a combination of DISCUSS threads, jira comments, RB and code
> comments, wikis and html documentation. In the past I have had to dig
> into a bunch of these to try and figure out why something was
> implemented a certain way. I think KIPs can help a lot here first by
> providing guidelines on what to think about (compatibility, new APIs,
> etc.) when working through a major feature; and second by becoming a
> crisp source of truth documentation for new releases.  E.g., for
> feature X: see relevant KIPs: a, b, c, etc.
>
> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
> > Hey Joe,
> >
> > Yeah I guess the question is what is the definition of major? I agree we
> > definitely don't want to generate a bunch of paperwork. We have enough
> > problems just getting all the contributions reviewed and checked in in a
> > timely fashion...
> >
> > So obviously bug fixes would not apply here.
> >
> > I think it is also pretty clear that big features should get reviewed and
> > discussed. To pick on myself, for example, the log compaction work was
> done
> > without enough public discussion about how it worked and why (imho). I
> > hope/claim that enough rigour in thinking about use-cases and operations
> > and so on was done that it turned out well, but the discussion was just
> > between a few people with no real public output. This kind of feature is
> > clearly a big change and something we should discuss.
> >
> > If we limit ourselves to just the public contracts the KIP introduces the
> > discussion would just be on the new configs and monitoring without
> really a
> > discussion of the design and how it works which is obviously closely
> > related.
> >
> > I don't think this should be more work because in practice we are making
> > wiki pages for any big thing anyway. So this would just be a consistent
> way
> > of numbering and structuring these pages. This would also give a clear
> call
> > to action: "hey kafka people, come read my wiki and think this through".
> >
> > I actually thinking the voting aspect is less important. I think it is
> > generally clear when there is agreement on something and not. So from my
> > point of view we could actually just eliminate that part if that is too
> > formal, it just seemed like a good way to formally adopt something.
> >
> > To address some of your comments from the wiki:
> >
> > 1. This doesn't inhibit someone coming along and putting up a patch. It
> is
> > just that when they do if it is a big thing introducing new functionality
> > we would ask for a little discussion on the basic feature/contracts prior
> > to code review.
> >
> > 2. We definitely definitely don't want people generating a lot of these
> > things every time they have an idea that they aren't going to implement.
> So
> > this is only applicable to things you absolutely will check in code for.
> We
> > also don't want to be making proposals before things are thought through,
> > which often requires writing the code. So I think the right time for a
> KIP
> > is when you are far enough along that you know the issues and tradeoffs
> but
> > not so far along that you are going to be totally opposed to any change.
> > Sometimes that is prior to writing any code and sometimes not until you
> are
> > practically done.
> >
> > The key problem I see this fixing is that there is enough new development
> > happening that it is pretty hard for everyone to review every line of
> every
> > iteration of every patch. But all of us should be fully aware of new
> > features, the ramifications, the new public interfaces, etc. If we aren't
> > aware of that we can't really build a holistic system that is beautiful
> and
> > consistent across. So the idea is that if you fully review the KIPs you
> can
> > be sure that even if you don't know every new line of code, you know the
> > major changes coming in.
> >
> > -Jay
> >
> >
> >
> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein 
> wrote:
> >
> > > Thanks Jay for kicking this off! I think the confluence page you wrote
> up
> > > is a great start.
> > >
> > >
> > > The KIP

Re: [DISCUSS] KIPs

[Gwen Shapira]

+1 to Ewen's suggestions: Deprecation, status and version.

Perhaps add the JIRA where the KIP was implemented to the metadata.
This will help tie things together.

On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
 wrote:
> I think adding a section about deprecation would be helpful. A good
> fraction of the time I would expect the goal of a KIP is to fix or replace
> older functionality that needs continued support for compatibility, but
> should eventually be phased out. This helps Kafka devs understand how long
> they'll end up supporting multiple versions of features and helps users
> understand when they're going to have to make updates to their applications.
>
> Less important but useful -- having a bit of standard metadata like PEPs
> do. Two I think are important are status (if someone lands on the KIP page,
> can they tell whether this KIP was ever completed?) and/or the version the
> KIP was first released in.
>
>
>
> On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy  wrote:
>
>> I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
>> doing this in one form or the other. However, IMO it is fairly ad hoc
>> - i.e., a combination of DISCUSS threads, jira comments, RB and code
>> comments, wikis and html documentation. In the past I have had to dig
>> into a bunch of these to try and figure out why something was
>> implemented a certain way. I think KIPs can help a lot here first by
>> providing guidelines on what to think about (compatibility, new APIs,
>> etc.) when working through a major feature; and second by becoming a
>> crisp source of truth documentation for new releases.  E.g., for
>> feature X: see relevant KIPs: a, b, c, etc.
>>
>> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
>> > Hey Joe,
>> >
>> > Yeah I guess the question is what is the definition of major? I agree we
>> > definitely don't want to generate a bunch of paperwork. We have enough
>> > problems just getting all the contributions reviewed and checked in in a
>> > timely fashion...
>> >
>> > So obviously bug fixes would not apply here.
>> >
>> > I think it is also pretty clear that big features should get reviewed and
>> > discussed. To pick on myself, for example, the log compaction work was
>> done
>> > without enough public discussion about how it worked and why (imho). I
>> > hope/claim that enough rigour in thinking about use-cases and operations
>> > and so on was done that it turned out well, but the discussion was just
>> > between a few people with no real public output. This kind of feature is
>> > clearly a big change and something we should discuss.
>> >
>> > If we limit ourselves to just the public contracts the KIP introduces the
>> > discussion would just be on the new configs and monitoring without
>> really a
>> > discussion of the design and how it works which is obviously closely
>> > related.
>> >
>> > I don't think this should be more work because in practice we are making
>> > wiki pages for any big thing anyway. So this would just be a consistent
>> way
>> > of numbering and structuring these pages. This would also give a clear
>> call
>> > to action: "hey kafka people, come read my wiki and think this through".
>> >
>> > I actually thinking the voting aspect is less important. I think it is
>> > generally clear when there is agreement on something and not. So from my
>> > point of view we could actually just eliminate that part if that is too
>> > formal, it just seemed like a good way to formally adopt something.
>> >
>> > To address some of your comments from the wiki:
>> >
>> > 1. This doesn't inhibit someone coming along and putting up a patch. It
>> is
>> > just that when they do if it is a big thing introducing new functionality
>> > we would ask for a little discussion on the basic feature/contracts prior
>> > to code review.
>> >
>> > 2. We definitely definitely don't want people generating a lot of these
>> > things every time they have an idea that they aren't going to implement.
>> So
>> > this is only applicable to things you absolutely will check in code for.
>> We
>> > also don't want to be making proposals before things are thought through,
>> > which often requires writing the code. So I think the right time for a
>> KIP
>> > is when you are far enough along that you know the issues and tradeoffs
>> but
>> > not so far along that you are going to be totally opposed to any change.
>> > Sometimes that is prior to writing any code and sometimes not until you
>> are
>> > practically done.
>> >
>> > The key problem I see this fixing is that there is enough new development
>> > happening that it is pretty hard for everyone to review every line of
>> every
>> > iteration of every patch. But all of us should be fully aware of new
>> > features, the ramifications, the new public interfaces, etc. If we aren't
>> > aware of that we can't really build a holistic system that is beautiful
>> and
>> > consistent across. So the idea is that if you fully review the KIPs 

Re: [DISCUSS] KIPs

[Guozhang Wang]

+1 on the idea, and we could mutually link the KIP wiki page with the the
created JIRA ticket (i.e. include the JIRA number on the page and the KIP
url on the ticket description).

Regarding the KIP process, probably we do not need two phase communication
of a [DISCUSS] followed by [VOTE], as Jay said the voting should be clear
while people discuss about that.

About who should trigger the process, I think the only involved people
would be 1) when the patch is submitted / or even the ticket is created,
the assignee could choose to start the KIP process if she thought it is
necessary; 2) the reviewer of the patch can also suggest starting KIP
discussions.

On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
wrote:

> +1 to Ewen's suggestions: Deprecation, status and version.
>
> Perhaps add the JIRA where the KIP was implemented to the metadata.
> This will help tie things together.
>
> On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
>  wrote:
> > I think adding a section about deprecation would be helpful. A good
> > fraction of the time I would expect the goal of a KIP is to fix or
> replace
> > older functionality that needs continued support for compatibility, but
> > should eventually be phased out. This helps Kafka devs understand how
> long
> > they'll end up supporting multiple versions of features and helps users
> > understand when they're going to have to make updates to their
> applications.
> >
> > Less important but useful -- having a bit of standard metadata like PEPs
> > do. Two I think are important are status (if someone lands on the KIP
> page,
> > can they tell whether this KIP was ever completed?) and/or the version
> the
> > KIP was first released in.
> >
> >
> >
> > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy  wrote:
> >
> >> I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
> >> doing this in one form or the other. However, IMO it is fairly ad hoc
> >> - i.e., a combination of DISCUSS threads, jira comments, RB and code
> >> comments, wikis and html documentation. In the past I have had to dig
> >> into a bunch of these to try and figure out why something was
> >> implemented a certain way. I think KIPs can help a lot here first by
> >> providing guidelines on what to think about (compatibility, new APIs,
> >> etc.) when working through a major feature; and second by becoming a
> >> crisp source of truth documentation for new releases.  E.g., for
> >> feature X: see relevant KIPs: a, b, c, etc.
> >>
> >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
> >> > Hey Joe,
> >> >
> >> > Yeah I guess the question is what is the definition of major? I agree
> we
> >> > definitely don't want to generate a bunch of paperwork. We have enough
> >> > problems just getting all the contributions reviewed and checked in
> in a
> >> > timely fashion...
> >> >
> >> > So obviously bug fixes would not apply here.
> >> >
> >> > I think it is also pretty clear that big features should get reviewed
> and
> >> > discussed. To pick on myself, for example, the log compaction work was
> >> done
> >> > without enough public discussion about how it worked and why (imho). I
> >> > hope/claim that enough rigour in thinking about use-cases and
> operations
> >> > and so on was done that it turned out well, but the discussion was
> just
> >> > between a few people with no real public output. This kind of feature
> is
> >> > clearly a big change and something we should discuss.
> >> >
> >> > If we limit ourselves to just the public contracts the KIP introduces
> the
> >> > discussion would just be on the new configs and monitoring without
> >> really a
> >> > discussion of the design and how it works which is obviously closely
> >> > related.
> >> >
> >> > I don't think this should be more work because in practice we are
> making
> >> > wiki pages for any big thing anyway. So this would just be a
> consistent
> >> way
> >> > of numbering and structuring these pages. This would also give a clear
> >> call
> >> > to action: "hey kafka people, come read my wiki and think this
> through".
> >> >
> >> > I actually thinking the voting aspect is less important. I think it is
> >> > generally clear when there is agreement on something and not. So from
> my
> >> > point of view we could actually just eliminate that part if that is
> too
> >> > formal, it just seemed like a good way to formally adopt something.
> >> >
> >> > To address some of your comments from the wiki:
> >> >
> >> > 1. This doesn't inhibit someone coming along and putting up a patch.
> It
> >> is
> >> > just that when they do if it is a big thing introducing new
> functionality
> >> > we would ask for a little discussion on the basic feature/contracts
> prior
> >> > to code review.
> >> >
> >> > 2. We definitely definitely don't want people generating a lot of
> these
> >> > things every time they have an idea that they aren't going to
> implement.
> >> So
> >> > this is only applicable to things you absolutely will check 

Re: [DISCUSS] KIPs

[Joe Stein]

+1 to everything we have been saying and where this (has settled to)/(is
settling to).

I am sure other folks have some more feedback and think we should try to
keep this discussion going if need be. I am also a firm believer of form
following function so kicking the tires some to flesh out the details of
this and have some organic growth with the process will be healthy and
beneficial to the community.

For my part, what I will do is open a few KIP based on some of the work I
have been involved with for 0.8.3. Off the top of my head this would
include 1) changes to re-assignment of partitions 2) kafka cli 3) global
configs 4) security white list black list by ip 5) SSL 6) We probably will
have lots of Security related KIPs and should treat them all individually
when the time is appropriate 7) Kafka on Mesos.

If someone else wants to jump in to start getting some of the security KIP
that we are going to have in 0.8.3 I think that would be great (e.g.
Multiple Listeners for Kafka Brokers). There are also a few other tickets I
can think of that are important to have in the code in 0.8.3 that should
have KIP also that I haven't really been involved in. I will take a few
minutes and go through JIRA (one I can think of like auto assign id that is
already committed I think) and ask for a KIP if appropriate or if I feel
that I can write it up (both from a time and understanding perspective) do
so.

long story short, I encourage folks to start moving ahead with the KIP for
0.8.3 as how we operate. any objections?

On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang  wrote:

> +1 on the idea, and we could mutually link the KIP wiki page with the the
> created JIRA ticket (i.e. include the JIRA number on the page and the KIP
> url on the ticket description).
>
> Regarding the KIP process, probably we do not need two phase communication
> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be clear
> while people discuss about that.
>
> About who should trigger the process, I think the only involved people
> would be 1) when the patch is submitted / or even the ticket is created,
> the assignee could choose to start the KIP process if she thought it is
> necessary; 2) the reviewer of the patch can also suggest starting KIP
> discussions.
>
> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
> wrote:
>
> > +1 to Ewen's suggestions: Deprecation, status and version.
> >
> > Perhaps add the JIRA where the KIP was implemented to the metadata.
> > This will help tie things together.
> >
> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
> >  wrote:
> > > I think adding a section about deprecation would be helpful. A good
> > > fraction of the time I would expect the goal of a KIP is to fix or
> > replace
> > > older functionality that needs continued support for compatibility, but
> > > should eventually be phased out. This helps Kafka devs understand how
> > long
> > > they'll end up supporting multiple versions of features and helps users
> > > understand when they're going to have to make updates to their
> > applications.
> > >
> > > Less important but useful -- having a bit of standard metadata like
> PEPs
> > > do. Two I think are important are status (if someone lands on the KIP
> > page,
> > > can they tell whether this KIP was ever completed?) and/or the version
> > the
> > > KIP was first released in.
> > >
> > >
> > >
> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy 
> wrote:
> > >
> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
> > >> doing this in one form or the other. However, IMO it is fairly ad hoc
> > >> - i.e., a combination of DISCUSS threads, jira comments, RB and code
> > >> comments, wikis and html documentation. In the past I have had to dig
> > >> into a bunch of these to try and figure out why something was
> > >> implemented a certain way. I think KIPs can help a lot here first by
> > >> providing guidelines on what to think about (compatibility, new APIs,
> > >> etc.) when working through a major feature; and second by becoming a
> > >> crisp source of truth documentation for new releases.  E.g., for
> > >> feature X: see relevant KIPs: a, b, c, etc.
> > >>
> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
> > >> > Hey Joe,
> > >> >
> > >> > Yeah I guess the question is what is the definition of major? I
> agree
> > we
> > >> > definitely don't want to generate a bunch of paperwork. We have
> enough
> > >> > problems just getting all the contributions reviewed and checked in
> > in a
> > >> > timely fashion...
> > >> >
> > >> > So obviously bug fixes would not apply here.
> > >> >
> > >> > I think it is also pretty clear that big features should get
> reviewed
> > and
> > >> > discussed. To pick on myself, for example, the log compaction work
> was
> > >> done
> > >> > without enough public discussion about how it worked and why
> (imho). I
> > >> > hope/claim that enough rigour in thinking about use-cases and
> > operations
> > >> > 

Re: [DISCUSS] KIPs

[Gwen Shapira]

+1

Will be happy to provide a KIP for the multiple-listeners patch.

Gwen

On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein  wrote:
> +1 to everything we have been saying and where this (has settled to)/(is
> settling to).
>
> I am sure other folks have some more feedback and think we should try to
> keep this discussion going if need be. I am also a firm believer of form
> following function so kicking the tires some to flesh out the details of
> this and have some organic growth with the process will be healthy and
> beneficial to the community.
>
> For my part, what I will do is open a few KIP based on some of the work I
> have been involved with for 0.8.3. Off the top of my head this would
> include 1) changes to re-assignment of partitions 2) kafka cli 3) global
> configs 4) security white list black list by ip 5) SSL 6) We probably will
> have lots of Security related KIPs and should treat them all individually
> when the time is appropriate 7) Kafka on Mesos.
>
> If someone else wants to jump in to start getting some of the security KIP
> that we are going to have in 0.8.3 I think that would be great (e.g.
> Multiple Listeners for Kafka Brokers). There are also a few other tickets I
> can think of that are important to have in the code in 0.8.3 that should
> have KIP also that I haven't really been involved in. I will take a few
> minutes and go through JIRA (one I can think of like auto assign id that is
> already committed I think) and ask for a KIP if appropriate or if I feel
> that I can write it up (both from a time and understanding perspective) do
> so.
>
> long story short, I encourage folks to start moving ahead with the KIP for
> 0.8.3 as how we operate. any objections?
>
> On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang  wrote:
>
>> +1 on the idea, and we could mutually link the KIP wiki page with the the
>> created JIRA ticket (i.e. include the JIRA number on the page and the KIP
>> url on the ticket description).
>>
>> Regarding the KIP process, probably we do not need two phase communication
>> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be clear
>> while people discuss about that.
>>
>> About who should trigger the process, I think the only involved people
>> would be 1) when the patch is submitted / or even the ticket is created,
>> the assignee could choose to start the KIP process if she thought it is
>> necessary; 2) the reviewer of the patch can also suggest starting KIP
>> discussions.
>>
>> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
>> wrote:
>>
>> > +1 to Ewen's suggestions: Deprecation, status and version.
>> >
>> > Perhaps add the JIRA where the KIP was implemented to the metadata.
>> > This will help tie things together.
>> >
>> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
>> >  wrote:
>> > > I think adding a section about deprecation would be helpful. A good
>> > > fraction of the time I would expect the goal of a KIP is to fix or
>> > replace
>> > > older functionality that needs continued support for compatibility, but
>> > > should eventually be phased out. This helps Kafka devs understand how
>> > long
>> > > they'll end up supporting multiple versions of features and helps users
>> > > understand when they're going to have to make updates to their
>> > applications.
>> > >
>> > > Less important but useful -- having a bit of standard metadata like
>> PEPs
>> > > do. Two I think are important are status (if someone lands on the KIP
>> > page,
>> > > can they tell whether this KIP was ever completed?) and/or the version
>> > the
>> > > KIP was first released in.
>> > >
>> > >
>> > >
>> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy 
>> wrote:
>> > >
>> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
>> > >> doing this in one form or the other. However, IMO it is fairly ad hoc
>> > >> - i.e., a combination of DISCUSS threads, jira comments, RB and code
>> > >> comments, wikis and html documentation. In the past I have had to dig
>> > >> into a bunch of these to try and figure out why something was
>> > >> implemented a certain way. I think KIPs can help a lot here first by
>> > >> providing guidelines on what to think about (compatibility, new APIs,
>> > >> etc.) when working through a major feature; and second by becoming a
>> > >> crisp source of truth documentation for new releases.  E.g., for
>> > >> feature X: see relevant KIPs: a, b, c, etc.
>> > >>
>> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
>> > >> > Hey Joe,
>> > >> >
>> > >> > Yeah I guess the question is what is the definition of major? I
>> agree
>> > we
>> > >> > definitely don't want to generate a bunch of paperwork. We have
>> enough
>> > >> > problems just getting all the contributions reviewed and checked in
>> > in a
>> > >> > timely fashion...
>> > >> >
>> > >> > So obviously bug fixes would not apply here.
>> > >> >
>> > >> > I think it is also pretty clear that big features should get
>> reviewed
>> > and
>> > >> > dis

Re: [DISCUSS] KIPs

[Jay Kreps]

Great! Sounds like everyone is on the same page

   - I created a template page to make things easier. If you do Tools->Copy
   on this page you can just fill in the italic portions with your details.
   - I retrofitted KIP-1 to match this formatting
   - I added the metadata section people asked for (a link to the
   discussion, the JIRA, and the current status). Let's make sure we remember
   to update the current status as things are figured out.
   - Let's keep the discussion on the mailing list rather than on the wiki
   pages. It makes sense to do one or the other so all the comments are in one
   place and I think prior experience is that the wiki comments are the worse
   way.

I think it would be great do KIPs for some of the in-flight items folks
mentioned.

-Jay

On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira  wrote:

> +1
>
> Will be happy to provide a KIP for the multiple-listeners patch.
>
> Gwen
>
> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein  wrote:
> > +1 to everything we have been saying and where this (has settled to)/(is
> > settling to).
> >
> > I am sure other folks have some more feedback and think we should try to
> > keep this discussion going if need be. I am also a firm believer of form
> > following function so kicking the tires some to flesh out the details of
> > this and have some organic growth with the process will be healthy and
> > beneficial to the community.
> >
> > For my part, what I will do is open a few KIP based on some of the work I
> > have been involved with for 0.8.3. Off the top of my head this would
> > include 1) changes to re-assignment of partitions 2) kafka cli 3) global
> > configs 4) security white list black list by ip 5) SSL 6) We probably
> will
> > have lots of Security related KIPs and should treat them all individually
> > when the time is appropriate 7) Kafka on Mesos.
> >
> > If someone else wants to jump in to start getting some of the security
> KIP
> > that we are going to have in 0.8.3 I think that would be great (e.g.
> > Multiple Listeners for Kafka Brokers). There are also a few other
> tickets I
> > can think of that are important to have in the code in 0.8.3 that should
> > have KIP also that I haven't really been involved in. I will take a few
> > minutes and go through JIRA (one I can think of like auto assign id that
> is
> > already committed I think) and ask for a KIP if appropriate or if I feel
> > that I can write it up (both from a time and understanding perspective)
> do
> > so.
> >
> > long story short, I encourage folks to start moving ahead with the KIP
> for
> > 0.8.3 as how we operate. any objections?
> >
> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
> wrote:
> >
> >> +1 on the idea, and we could mutually link the KIP wiki page with the
> the
> >> created JIRA ticket (i.e. include the JIRA number on the page and the
> KIP
> >> url on the ticket description).
> >>
> >> Regarding the KIP process, probably we do not need two phase
> communication
> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
> clear
> >> while people discuss about that.
> >>
> >> About who should trigger the process, I think the only involved people
> >> would be 1) when the patch is submitted / or even the ticket is created,
> >> the assignee could choose to start the KIP process if she thought it is
> >> necessary; 2) the reviewer of the patch can also suggest starting KIP
> >> discussions.
> >>
> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
> >> wrote:
> >>
> >> > +1 to Ewen's suggestions: Deprecation, status and version.
> >> >
> >> > Perhaps add the JIRA where the KIP was implemented to the metadata.
> >> > This will help tie things together.
> >> >
> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
> >> >  wrote:
> >> > > I think adding a section about deprecation would be helpful. A good
> >> > > fraction of the time I would expect the goal of a KIP is to fix or
> >> > replace
> >> > > older functionality that needs continued support for compatibility,
> but
> >> > > should eventually be phased out. This helps Kafka devs understand
> how
> >> > long
> >> > > they'll end up supporting multiple versions of features and helps
> users
> >> > > understand when they're going to have to make updates to their
> >> > applications.
> >> > >
> >> > > Less important but useful -- having a bit of standard metadata like
> >> PEPs
> >> > > do. Two I think are important are status (if someone lands on the
> KIP
> >> > page,
> >> > > can they tell whether this KIP was ever completed?) and/or the
> version
> >> > the
> >> > > KIP was first released in.
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy 
> >> wrote:
> >> > >
> >> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we are
> already
> >> > >> doing this in one form or the other. However, IMO it is fairly ad
> hoc
> >> > >> - i.e., a combination of DISCUSS threads, jira comments, RB and
> code
> >> > >> comments, wikis and

Re: [DISCUSS] KIPs

[Gwen Shapira]

I created a KIP for the multi-port broker change.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs

I'm not re-opening the discussion, since it was agreed on over a month
ago and implementation is close to complete (I hope!). Lets consider
this voted and accepted?

Gwen

On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps  wrote:
> Great! Sounds like everyone is on the same page
>
>- I created a template page to make things easier. If you do Tools->Copy
>on this page you can just fill in the italic portions with your details.
>- I retrofitted KIP-1 to match this formatting
>- I added the metadata section people asked for (a link to the
>discussion, the JIRA, and the current status). Let's make sure we remember
>to update the current status as things are figured out.
>- Let's keep the discussion on the mailing list rather than on the wiki
>pages. It makes sense to do one or the other so all the comments are in one
>place and I think prior experience is that the wiki comments are the worse
>way.
>
> I think it would be great do KIPs for some of the in-flight items folks
> mentioned.
>
> -Jay
>
> On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira  wrote:
>
>> +1
>>
>> Will be happy to provide a KIP for the multiple-listeners patch.
>>
>> Gwen
>>
>> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein  wrote:
>> > +1 to everything we have been saying and where this (has settled to)/(is
>> > settling to).
>> >
>> > I am sure other folks have some more feedback and think we should try to
>> > keep this discussion going if need be. I am also a firm believer of form
>> > following function so kicking the tires some to flesh out the details of
>> > this and have some organic growth with the process will be healthy and
>> > beneficial to the community.
>> >
>> > For my part, what I will do is open a few KIP based on some of the work I
>> > have been involved with for 0.8.3. Off the top of my head this would
>> > include 1) changes to re-assignment of partitions 2) kafka cli 3) global
>> > configs 4) security white list black list by ip 5) SSL 6) We probably
>> will
>> > have lots of Security related KIPs and should treat them all individually
>> > when the time is appropriate 7) Kafka on Mesos.
>> >
>> > If someone else wants to jump in to start getting some of the security
>> KIP
>> > that we are going to have in 0.8.3 I think that would be great (e.g.
>> > Multiple Listeners for Kafka Brokers). There are also a few other
>> tickets I
>> > can think of that are important to have in the code in 0.8.3 that should
>> > have KIP also that I haven't really been involved in. I will take a few
>> > minutes and go through JIRA (one I can think of like auto assign id that
>> is
>> > already committed I think) and ask for a KIP if appropriate or if I feel
>> > that I can write it up (both from a time and understanding perspective)
>> do
>> > so.
>> >
>> > long story short, I encourage folks to start moving ahead with the KIP
>> for
>> > 0.8.3 as how we operate. any objections?
>> >
>> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
>> wrote:
>> >
>> >> +1 on the idea, and we could mutually link the KIP wiki page with the
>> the
>> >> created JIRA ticket (i.e. include the JIRA number on the page and the
>> KIP
>> >> url on the ticket description).
>> >>
>> >> Regarding the KIP process, probably we do not need two phase
>> communication
>> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
>> clear
>> >> while people discuss about that.
>> >>
>> >> About who should trigger the process, I think the only involved people
>> >> would be 1) when the patch is submitted / or even the ticket is created,
>> >> the assignee could choose to start the KIP process if she thought it is
>> >> necessary; 2) the reviewer of the patch can also suggest starting KIP
>> >> discussions.
>> >>
>> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
>> >> wrote:
>> >>
>> >> > +1 to Ewen's suggestions: Deprecation, status and version.
>> >> >
>> >> > Perhaps add the JIRA where the KIP was implemented to the metadata.
>> >> > This will help tie things together.
>> >> >
>> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
>> >> >  wrote:
>> >> > > I think adding a section about deprecation would be helpful. A good
>> >> > > fraction of the time I would expect the goal of a KIP is to fix or
>> >> > replace
>> >> > > older functionality that needs continued support for compatibility,
>> but
>> >> > > should eventually be phased out. This helps Kafka devs understand
>> how
>> >> > long
>> >> > > they'll end up supporting multiple versions of features and helps
>> users
>> >> > > understand when they're going to have to make updates to their
>> >> > applications.
>> >> > >
>> >> > > Less important but useful -- having a bit of standard metadata like
>> >> PEPs
>> >> > > do. Two I think are important are status (if someone lands on the
>> KIP

Re: [DISCUSS] KIPs

[Jay Kreps]

Hey Gwen,

Could we get the actual changes in that KIP? I.e. changes to metadata
request, changes to UpdateMetadataRequest, new configs and what will their
valid values be, etc. This kind of says that those things will change but
doesn't say what they will change to...

-Jay

On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira  wrote:

> I created a KIP for the multi-port broker change.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
>
> I'm not re-opening the discussion, since it was agreed on over a month
> ago and implementation is close to complete (I hope!). Lets consider
> this voted and accepted?
>
> Gwen
>
> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps  wrote:
> > Great! Sounds like everyone is on the same page
> >
> >- I created a template page to make things easier. If you do
> Tools->Copy
> >on this page you can just fill in the italic portions with your
> details.
> >- I retrofitted KIP-1 to match this formatting
> >- I added the metadata section people asked for (a link to the
> >discussion, the JIRA, and the current status). Let's make sure we
> remember
> >to update the current status as things are figured out.
> >- Let's keep the discussion on the mailing list rather than on the
> wiki
> >pages. It makes sense to do one or the other so all the comments are
> in one
> >place and I think prior experience is that the wiki comments are the
> worse
> >way.
> >
> > I think it would be great do KIPs for some of the in-flight items folks
> > mentioned.
> >
> > -Jay
> >
> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
> wrote:
> >
> >> +1
> >>
> >> Will be happy to provide a KIP for the multiple-listeners patch.
> >>
> >> Gwen
> >>
> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
> wrote:
> >> > +1 to everything we have been saying and where this (has settled
> to)/(is
> >> > settling to).
> >> >
> >> > I am sure other folks have some more feedback and think we should try
> to
> >> > keep this discussion going if need be. I am also a firm believer of
> form
> >> > following function so kicking the tires some to flesh out the details
> of
> >> > this and have some organic growth with the process will be healthy and
> >> > beneficial to the community.
> >> >
> >> > For my part, what I will do is open a few KIP based on some of the
> work I
> >> > have been involved with for 0.8.3. Off the top of my head this would
> >> > include 1) changes to re-assignment of partitions 2) kafka cli 3)
> global
> >> > configs 4) security white list black list by ip 5) SSL 6) We probably
> >> will
> >> > have lots of Security related KIPs and should treat them all
> individually
> >> > when the time is appropriate 7) Kafka on Mesos.
> >> >
> >> > If someone else wants to jump in to start getting some of the security
> >> KIP
> >> > that we are going to have in 0.8.3 I think that would be great (e.g.
> >> > Multiple Listeners for Kafka Brokers). There are also a few other
> >> tickets I
> >> > can think of that are important to have in the code in 0.8.3 that
> should
> >> > have KIP also that I haven't really been involved in. I will take a
> few
> >> > minutes and go through JIRA (one I can think of like auto assign id
> that
> >> is
> >> > already committed I think) and ask for a KIP if appropriate or if I
> feel
> >> > that I can write it up (both from a time and understanding
> perspective)
> >> do
> >> > so.
> >> >
> >> > long story short, I encourage folks to start moving ahead with the KIP
> >> for
> >> > 0.8.3 as how we operate. any objections?
> >> >
> >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
> >> wrote:
> >> >
> >> >> +1 on the idea, and we could mutually link the KIP wiki page with the
> >> the
> >> >> created JIRA ticket (i.e. include the JIRA number on the page and the
> >> KIP
> >> >> url on the ticket description).
> >> >>
> >> >> Regarding the KIP process, probably we do not need two phase
> >> communication
> >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
> >> clear
> >> >> while people discuss about that.
> >> >>
> >> >> About who should trigger the process, I think the only involved
> people
> >> >> would be 1) when the patch is submitted / or even the ticket is
> created,
> >> >> the assignee could choose to start the KIP process if she thought it
> is
> >> >> necessary; 2) the reviewer of the patch can also suggest starting KIP
> >> >> discussions.
> >> >>
> >> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira <
> gshap...@cloudera.com>
> >> >> wrote:
> >> >>
> >> >> > +1 to Ewen's suggestions: Deprecation, status and version.
> >> >> >
> >> >> > Perhaps add the JIRA where the KIP was implemented to the metadata.
> >> >> > This will help tie things together.
> >> >> >
> >> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
> >> >> >  wrote:
> >> >> > > I think adding a section about deprecation would be helpful. A
> good
> >> >> > > fraction o

Re: [DISCUSS] KIPs

[Gwen Shapira]

Good point :)

I added the specifics of the new  UpdateMetadataRequest, which is the
only protocol bump in this change.

Highlighted the broker and producer/consumer configuration changes,
added some example values and added the new zookeeper json.

Hope this makes things clearer.

On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps  wrote:
> Hey Gwen,
>
> Could we get the actual changes in that KIP? I.e. changes to metadata
> request, changes to UpdateMetadataRequest, new configs and what will their
> valid values be, etc. This kind of says that those things will change but
> doesn't say what they will change to...
>
> -Jay
>
> On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira  wrote:
>
>> I created a KIP for the multi-port broker change.
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
>>
>> I'm not re-opening the discussion, since it was agreed on over a month
>> ago and implementation is close to complete (I hope!). Lets consider
>> this voted and accepted?
>>
>> Gwen
>>
>> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps  wrote:
>> > Great! Sounds like everyone is on the same page
>> >
>> >- I created a template page to make things easier. If you do
>> Tools->Copy
>> >on this page you can just fill in the italic portions with your
>> details.
>> >- I retrofitted KIP-1 to match this formatting
>> >- I added the metadata section people asked for (a link to the
>> >discussion, the JIRA, and the current status). Let's make sure we
>> remember
>> >to update the current status as things are figured out.
>> >- Let's keep the discussion on the mailing list rather than on the
>> wiki
>> >pages. It makes sense to do one or the other so all the comments are
>> in one
>> >place and I think prior experience is that the wiki comments are the
>> worse
>> >way.
>> >
>> > I think it would be great do KIPs for some of the in-flight items folks
>> > mentioned.
>> >
>> > -Jay
>> >
>> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
>> wrote:
>> >
>> >> +1
>> >>
>> >> Will be happy to provide a KIP for the multiple-listeners patch.
>> >>
>> >> Gwen
>> >>
>> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
>> wrote:
>> >> > +1 to everything we have been saying and where this (has settled
>> to)/(is
>> >> > settling to).
>> >> >
>> >> > I am sure other folks have some more feedback and think we should try
>> to
>> >> > keep this discussion going if need be. I am also a firm believer of
>> form
>> >> > following function so kicking the tires some to flesh out the details
>> of
>> >> > this and have some organic growth with the process will be healthy and
>> >> > beneficial to the community.
>> >> >
>> >> > For my part, what I will do is open a few KIP based on some of the
>> work I
>> >> > have been involved with for 0.8.3. Off the top of my head this would
>> >> > include 1) changes to re-assignment of partitions 2) kafka cli 3)
>> global
>> >> > configs 4) security white list black list by ip 5) SSL 6) We probably
>> >> will
>> >> > have lots of Security related KIPs and should treat them all
>> individually
>> >> > when the time is appropriate 7) Kafka on Mesos.
>> >> >
>> >> > If someone else wants to jump in to start getting some of the security
>> >> KIP
>> >> > that we are going to have in 0.8.3 I think that would be great (e.g.
>> >> > Multiple Listeners for Kafka Brokers). There are also a few other
>> >> tickets I
>> >> > can think of that are important to have in the code in 0.8.3 that
>> should
>> >> > have KIP also that I haven't really been involved in. I will take a
>> few
>> >> > minutes and go through JIRA (one I can think of like auto assign id
>> that
>> >> is
>> >> > already committed I think) and ask for a KIP if appropriate or if I
>> feel
>> >> > that I can write it up (both from a time and understanding
>> perspective)
>> >> do
>> >> > so.
>> >> >
>> >> > long story short, I encourage folks to start moving ahead with the KIP
>> >> for
>> >> > 0.8.3 as how we operate. any objections?
>> >> >
>> >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
>> >> wrote:
>> >> >
>> >> >> +1 on the idea, and we could mutually link the KIP wiki page with the
>> >> the
>> >> >> created JIRA ticket (i.e. include the JIRA number on the page and the
>> >> KIP
>> >> >> url on the ticket description).
>> >> >>
>> >> >> Regarding the KIP process, probably we do not need two phase
>> >> communication
>> >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
>> >> clear
>> >> >> while people discuss about that.
>> >> >>
>> >> >> About who should trigger the process, I think the only involved
>> people
>> >> >> would be 1) when the patch is submitted / or even the ticket is
>> created,
>> >> >> the assignee could choose to start the KIP process if she thought it
>> is
>> >> >> necessary; 2) the reviewer of the patch can also suggest starting KIP
>> >> >> discussions.
>> >> >>
>> >> >> On Fri, Jan 16, 2015 at 10:

Re: [DISCUSS] KIPs

[Jay Kreps]

I think I am still confused. In addition to the UpdateMetadataRequest don't
we have to change the MetadataResponse so that it's possible for clients to
discover the new ports? Or is that a second phase? I was imagining it
worked by basically allowing the brokers to advertise multiple ports, one
per security type, and then in the client you configure a protocol which
will implicitly choose the port from the options returned in metadata to
you...

Likewise in the ConsumerMetadataResponse we are currently giving back full
broker information. I think we would have two options here: either change
the broker information included in that response to match the
metadataresponse or else remove the broker information entirely and just
return the node id (since in order to use that request you would already
have to have the cluster metadata). The second option may be cleaner since
it means we won't have to continue evolving those two in lockstep...

-Jay

On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira  wrote:

> Good point :)
>
> I added the specifics of the new  UpdateMetadataRequest, which is the
> only protocol bump in this change.
>
> Highlighted the broker and producer/consumer configuration changes,
> added some example values and added the new zookeeper json.
>
> Hope this makes things clearer.
>
> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps  wrote:
> > Hey Gwen,
> >
> > Could we get the actual changes in that KIP? I.e. changes to metadata
> > request, changes to UpdateMetadataRequest, new configs and what will
> their
> > valid values be, etc. This kind of says that those things will change but
> > doesn't say what they will change to...
> >
> > -Jay
> >
> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira 
> wrote:
> >
> >> I created a KIP for the multi-port broker change.
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
> >>
> >> I'm not re-opening the discussion, since it was agreed on over a month
> >> ago and implementation is close to complete (I hope!). Lets consider
> >> this voted and accepted?
> >>
> >> Gwen
> >>
> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps 
> wrote:
> >> > Great! Sounds like everyone is on the same page
> >> >
> >> >- I created a template page to make things easier. If you do
> >> Tools->Copy
> >> >on this page you can just fill in the italic portions with your
> >> details.
> >> >- I retrofitted KIP-1 to match this formatting
> >> >- I added the metadata section people asked for (a link to the
> >> >discussion, the JIRA, and the current status). Let's make sure we
> >> remember
> >> >to update the current status as things are figured out.
> >> >- Let's keep the discussion on the mailing list rather than on the
> >> wiki
> >> >pages. It makes sense to do one or the other so all the comments
> are
> >> in one
> >> >place and I think prior experience is that the wiki comments are
> the
> >> worse
> >> >way.
> >> >
> >> > I think it would be great do KIPs for some of the in-flight items
> folks
> >> > mentioned.
> >> >
> >> > -Jay
> >> >
> >> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
> >> wrote:
> >> >
> >> >> +1
> >> >>
> >> >> Will be happy to provide a KIP for the multiple-listeners patch.
> >> >>
> >> >> Gwen
> >> >>
> >> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
> >> wrote:
> >> >> > +1 to everything we have been saying and where this (has settled
> >> to)/(is
> >> >> > settling to).
> >> >> >
> >> >> > I am sure other folks have some more feedback and think we should
> try
> >> to
> >> >> > keep this discussion going if need be. I am also a firm believer of
> >> form
> >> >> > following function so kicking the tires some to flesh out the
> details
> >> of
> >> >> > this and have some organic growth with the process will be healthy
> and
> >> >> > beneficial to the community.
> >> >> >
> >> >> > For my part, what I will do is open a few KIP based on some of the
> >> work I
> >> >> > have been involved with for 0.8.3. Off the top of my head this
> would
> >> >> > include 1) changes to re-assignment of partitions 2) kafka cli 3)
> >> global
> >> >> > configs 4) security white list black list by ip 5) SSL 6) We
> probably
> >> >> will
> >> >> > have lots of Security related KIPs and should treat them all
> >> individually
> >> >> > when the time is appropriate 7) Kafka on Mesos.
> >> >> >
> >> >> > If someone else wants to jump in to start getting some of the
> security
> >> >> KIP
> >> >> > that we are going to have in 0.8.3 I think that would be great
> (e.g.
> >> >> > Multiple Listeners for Kafka Brokers). There are also a few other
> >> >> tickets I
> >> >> > can think of that are important to have in the code in 0.8.3 that
> >> should
> >> >> > have KIP also that I haven't really been involved in. I will take a
> >> few
> >> >> > minutes and go through JIRA (one I can think of like auto assign id
> >> that
> >> >> is
> >> >> > already committed I 

Re: [DISCUSS] KIPs

[Gwen Shapira]

I think what you described was the original design, so no wonder you
are confused :)

Following suggestions from Jun, I changed it a bit. The current model is:

- Clients (producers and consumers) need to know about the broker
ports in advance. They don't need to know about all brokers, but they
need to know at least one host:port pair that speaks the protocol they
want to use. The change is that all host:port pairs in broker.list
must be of the same protocol and match the security.protocol
configuration parameter.

- Client uses security.protocol configuration parameter to open a
connection to one of the brokers and sends the good old
MetadataRequest. The broker knows which port it got the connection on,
therefore it knows which security protocol is expected (it needs to
use the same protocol to accept the connection and respond), and
therefore it can send a response that contains only the host:port
pairs that are relevant to that protocol.

- From the client side the MetadataResponse did not change - it
contains a list of brokerId,host,port that the client can connect to.
The fact that all those broker endpoints were chosen out of a larger
collection to match the right protocol is irrelevant for the client.

I really like the new design since it preserves a lot of the same
configurations and APIs.

Thoughts?

Gwen

On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps  wrote:
> I think I am still confused. In addition to the UpdateMetadataRequest don't
> we have to change the MetadataResponse so that it's possible for clients to
> discover the new ports? Or is that a second phase? I was imagining it
> worked by basically allowing the brokers to advertise multiple ports, one
> per security type, and then in the client you configure a protocol which
> will implicitly choose the port from the options returned in metadata to
> you...
>
> Likewise in the ConsumerMetadataResponse we are currently giving back full
> broker information. I think we would have two options here: either change
> the broker information included in that response to match the
> metadataresponse or else remove the broker information entirely and just
> return the node id (since in order to use that request you would already
> have to have the cluster metadata). The second option may be cleaner since
> it means we won't have to continue evolving those two in lockstep...
>
> -Jay
>
> On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira  wrote:
>
>> Good point :)
>>
>> I added the specifics of the new  UpdateMetadataRequest, which is the
>> only protocol bump in this change.
>>
>> Highlighted the broker and producer/consumer configuration changes,
>> added some example values and added the new zookeeper json.
>>
>> Hope this makes things clearer.
>>
>> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps  wrote:
>> > Hey Gwen,
>> >
>> > Could we get the actual changes in that KIP? I.e. changes to metadata
>> > request, changes to UpdateMetadataRequest, new configs and what will
>> their
>> > valid values be, etc. This kind of says that those things will change but
>> > doesn't say what they will change to...
>> >
>> > -Jay
>> >
>> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira 
>> wrote:
>> >
>> >> I created a KIP for the multi-port broker change.
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
>> >>
>> >> I'm not re-opening the discussion, since it was agreed on over a month
>> >> ago and implementation is close to complete (I hope!). Lets consider
>> >> this voted and accepted?
>> >>
>> >> Gwen
>> >>
>> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps 
>> wrote:
>> >> > Great! Sounds like everyone is on the same page
>> >> >
>> >> >- I created a template page to make things easier. If you do
>> >> Tools->Copy
>> >> >on this page you can just fill in the italic portions with your
>> >> details.
>> >> >- I retrofitted KIP-1 to match this formatting
>> >> >- I added the metadata section people asked for (a link to the
>> >> >discussion, the JIRA, and the current status). Let's make sure we
>> >> remember
>> >> >to update the current status as things are figured out.
>> >> >- Let's keep the discussion on the mailing list rather than on the
>> >> wiki
>> >> >pages. It makes sense to do one or the other so all the comments
>> are
>> >> in one
>> >> >place and I think prior experience is that the wiki comments are
>> the
>> >> worse
>> >> >way.
>> >> >
>> >> > I think it would be great do KIPs for some of the in-flight items
>> folks
>> >> > mentioned.
>> >> >
>> >> > -Jay
>> >> >
>> >> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
>> >> wrote:
>> >> >
>> >> >> +1
>> >> >>
>> >> >> Will be happy to provide a KIP for the multiple-listeners patch.
>> >> >>
>> >> >> Gwen
>> >> >>
>> >> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
>> >> wrote:
>> >> >> > +1 to everything we have been saying and where this (has settled
>> >> to)/(is
>> >> >> > sett

Re: [DISCUSS] KIPs

[Jay Kreps]

Oh yeah I think that is better, I hadn't thought of that approach! Any way
you could describe the usage in the KIP, just for completeness?

-Jay

On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira 
wrote:

> I think what you described was the original design, so no wonder you
> are confused :)
>
> Following suggestions from Jun, I changed it a bit. The current model is:
>
> - Clients (producers and consumers) need to know about the broker
> ports in advance. They don't need to know about all brokers, but they
> need to know at least one host:port pair that speaks the protocol they
> want to use. The change is that all host:port pairs in broker.list
> must be of the same protocol and match the security.protocol
> configuration parameter.
>
> - Client uses security.protocol configuration parameter to open a
> connection to one of the brokers and sends the good old
> MetadataRequest. The broker knows which port it got the connection on,
> therefore it knows which security protocol is expected (it needs to
> use the same protocol to accept the connection and respond), and
> therefore it can send a response that contains only the host:port
> pairs that are relevant to that protocol.
>
> - From the client side the MetadataResponse did not change - it
> contains a list of brokerId,host,port that the client can connect to.
> The fact that all those broker endpoints were chosen out of a larger
> collection to match the right protocol is irrelevant for the client.
>
> I really like the new design since it preserves a lot of the same
> configurations and APIs.
>
> Thoughts?
>
> Gwen
>
> On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps  wrote:
> > I think I am still confused. In addition to the UpdateMetadataRequest
> don't
> > we have to change the MetadataResponse so that it's possible for clients
> to
> > discover the new ports? Or is that a second phase? I was imagining it
> > worked by basically allowing the brokers to advertise multiple ports, one
> > per security type, and then in the client you configure a protocol which
> > will implicitly choose the port from the options returned in metadata to
> > you...
> >
> > Likewise in the ConsumerMetadataResponse we are currently giving back
> full
> > broker information. I think we would have two options here: either change
> > the broker information included in that response to match the
> > metadataresponse or else remove the broker information entirely and just
> > return the node id (since in order to use that request you would already
> > have to have the cluster metadata). The second option may be cleaner
> since
> > it means we won't have to continue evolving those two in lockstep...
> >
> > -Jay
> >
> > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira 
> wrote:
> >
> >> Good point :)
> >>
> >> I added the specifics of the new  UpdateMetadataRequest, which is the
> >> only protocol bump in this change.
> >>
> >> Highlighted the broker and producer/consumer configuration changes,
> >> added some example values and added the new zookeeper json.
> >>
> >> Hope this makes things clearer.
> >>
> >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps  wrote:
> >> > Hey Gwen,
> >> >
> >> > Could we get the actual changes in that KIP? I.e. changes to metadata
> >> > request, changes to UpdateMetadataRequest, new configs and what will
> >> their
> >> > valid values be, etc. This kind of says that those things will change
> but
> >> > doesn't say what they will change to...
> >> >
> >> > -Jay
> >> >
> >> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira 
> >> wrote:
> >> >
> >> >> I created a KIP for the multi-port broker change.
> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
> >> >>
> >> >> I'm not re-opening the discussion, since it was agreed on over a
> month
> >> >> ago and implementation is close to complete (I hope!). Lets consider
> >> >> this voted and accepted?
> >> >>
> >> >> Gwen
> >> >>
> >> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps 
> >> wrote:
> >> >> > Great! Sounds like everyone is on the same page
> >> >> >
> >> >> >- I created a template page to make things easier. If you do
> >> >> Tools->Copy
> >> >> >on this page you can just fill in the italic portions with your
> >> >> details.
> >> >> >- I retrofitted KIP-1 to match this formatting
> >> >> >- I added the metadata section people asked for (a link to the
> >> >> >discussion, the JIRA, and the current status). Let's make sure
> we
> >> >> remember
> >> >> >to update the current status as things are figured out.
> >> >> >- Let's keep the discussion on the mailing list rather than on
> the
> >> >> wiki
> >> >> >pages. It makes sense to do one or the other so all the comments
> >> are
> >> >> in one
> >> >> >place and I think prior experience is that the wiki comments are
> >> the
> >> >> worse
> >> >> >way.
> >> >> >
> >> >> > I think it would be great do KIPs for some of the in-flight

Re: [DISCUSS] KIPs

[Gwen Shapira]

Thanks for validating our ideas. Updated the KIP with the workflow.

Now if you can nudge Jun to review the latest patch... ;)


On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps  wrote:
> Oh yeah I think that is better, I hadn't thought of that approach! Any way
> you could describe the usage in the KIP, just for completeness?
>
> -Jay
>
> On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira 
> wrote:
>
>> I think what you described was the original design, so no wonder you
>> are confused :)
>>
>> Following suggestions from Jun, I changed it a bit. The current model is:
>>
>> - Clients (producers and consumers) need to know about the broker
>> ports in advance. They don't need to know about all brokers, but they
>> need to know at least one host:port pair that speaks the protocol they
>> want to use. The change is that all host:port pairs in broker.list
>> must be of the same protocol and match the security.protocol
>> configuration parameter.
>>
>> - Client uses security.protocol configuration parameter to open a
>> connection to one of the brokers and sends the good old
>> MetadataRequest. The broker knows which port it got the connection on,
>> therefore it knows which security protocol is expected (it needs to
>> use the same protocol to accept the connection and respond), and
>> therefore it can send a response that contains only the host:port
>> pairs that are relevant to that protocol.
>>
>> - From the client side the MetadataResponse did not change - it
>> contains a list of brokerId,host,port that the client can connect to.
>> The fact that all those broker endpoints were chosen out of a larger
>> collection to match the right protocol is irrelevant for the client.
>>
>> I really like the new design since it preserves a lot of the same
>> configurations and APIs.
>>
>> Thoughts?
>>
>> Gwen
>>
>> On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps  wrote:
>> > I think I am still confused. In addition to the UpdateMetadataRequest
>> don't
>> > we have to change the MetadataResponse so that it's possible for clients
>> to
>> > discover the new ports? Or is that a second phase? I was imagining it
>> > worked by basically allowing the brokers to advertise multiple ports, one
>> > per security type, and then in the client you configure a protocol which
>> > will implicitly choose the port from the options returned in metadata to
>> > you...
>> >
>> > Likewise in the ConsumerMetadataResponse we are currently giving back
>> full
>> > broker information. I think we would have two options here: either change
>> > the broker information included in that response to match the
>> > metadataresponse or else remove the broker information entirely and just
>> > return the node id (since in order to use that request you would already
>> > have to have the cluster metadata). The second option may be cleaner
>> since
>> > it means we won't have to continue evolving those two in lockstep...
>> >
>> > -Jay
>> >
>> > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira 
>> wrote:
>> >
>> >> Good point :)
>> >>
>> >> I added the specifics of the new  UpdateMetadataRequest, which is the
>> >> only protocol bump in this change.
>> >>
>> >> Highlighted the broker and producer/consumer configuration changes,
>> >> added some example values and added the new zookeeper json.
>> >>
>> >> Hope this makes things clearer.
>> >>
>> >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps  wrote:
>> >> > Hey Gwen,
>> >> >
>> >> > Could we get the actual changes in that KIP? I.e. changes to metadata
>> >> > request, changes to UpdateMetadataRequest, new configs and what will
>> >> their
>> >> > valid values be, etc. This kind of says that those things will change
>> but
>> >> > doesn't say what they will change to...
>> >> >
>> >> > -Jay
>> >> >
>> >> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira 
>> >> wrote:
>> >> >
>> >> >> I created a KIP for the multi-port broker change.
>> >> >>
>> >> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
>> >> >>
>> >> >> I'm not re-opening the discussion, since it was agreed on over a
>> month
>> >> >> ago and implementation is close to complete (I hope!). Lets consider
>> >> >> this voted and accepted?
>> >> >>
>> >> >> Gwen
>> >> >>
>> >> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps 
>> >> wrote:
>> >> >> > Great! Sounds like everyone is on the same page
>> >> >> >
>> >> >> >- I created a template page to make things easier. If you do
>> >> >> Tools->Copy
>> >> >> >on this page you can just fill in the italic portions with your
>> >> >> details.
>> >> >> >- I retrofitted KIP-1 to match this formatting
>> >> >> >- I added the metadata section people asked for (a link to the
>> >> >> >discussion, the JIRA, and the current status). Let's make sure
>> we
>> >> >> remember
>> >> >> >to update the current status as things are figured out.
>> >> >> >- Let's keep the discussion on the mailing list rather than on
>> the
>> >> >

Re: [DISCUSS] KIPs

[Jun Rao]

Reviewed the latest patch in KAFKA-1809 :).

Thanks,

Jun

On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira 
wrote:

> Thanks for validating our ideas. Updated the KIP with the workflow.
>
> Now if you can nudge Jun to review the latest patch... ;)
>
>
> On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps  wrote:
> > Oh yeah I think that is better, I hadn't thought of that approach! Any
> way
> > you could describe the usage in the KIP, just for completeness?
> >
> > -Jay
> >
> > On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira 
> > wrote:
> >
> >> I think what you described was the original design, so no wonder you
> >> are confused :)
> >>
> >> Following suggestions from Jun, I changed it a bit. The current model
> is:
> >>
> >> - Clients (producers and consumers) need to know about the broker
> >> ports in advance. They don't need to know about all brokers, but they
> >> need to know at least one host:port pair that speaks the protocol they
> >> want to use. The change is that all host:port pairs in broker.list
> >> must be of the same protocol and match the security.protocol
> >> configuration parameter.
> >>
> >> - Client uses security.protocol configuration parameter to open a
> >> connection to one of the brokers and sends the good old
> >> MetadataRequest. The broker knows which port it got the connection on,
> >> therefore it knows which security protocol is expected (it needs to
> >> use the same protocol to accept the connection and respond), and
> >> therefore it can send a response that contains only the host:port
> >> pairs that are relevant to that protocol.
> >>
> >> - From the client side the MetadataResponse did not change - it
> >> contains a list of brokerId,host,port that the client can connect to.
> >> The fact that all those broker endpoints were chosen out of a larger
> >> collection to match the right protocol is irrelevant for the client.
> >>
> >> I really like the new design since it preserves a lot of the same
> >> configurations and APIs.
> >>
> >> Thoughts?
> >>
> >> Gwen
> >>
> >> On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps  wrote:
> >> > I think I am still confused. In addition to the UpdateMetadataRequest
> >> don't
> >> > we have to change the MetadataResponse so that it's possible for
> clients
> >> to
> >> > discover the new ports? Or is that a second phase? I was imagining it
> >> > worked by basically allowing the brokers to advertise multiple ports,
> one
> >> > per security type, and then in the client you configure a protocol
> which
> >> > will implicitly choose the port from the options returned in metadata
> to
> >> > you...
> >> >
> >> > Likewise in the ConsumerMetadataResponse we are currently giving back
> >> full
> >> > broker information. I think we would have two options here: either
> change
> >> > the broker information included in that response to match the
> >> > metadataresponse or else remove the broker information entirely and
> just
> >> > return the node id (since in order to use that request you would
> already
> >> > have to have the cluster metadata). The second option may be cleaner
> >> since
> >> > it means we won't have to continue evolving those two in lockstep...
> >> >
> >> > -Jay
> >> >
> >> > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira 
> >> wrote:
> >> >
> >> >> Good point :)
> >> >>
> >> >> I added the specifics of the new  UpdateMetadataRequest, which is the
> >> >> only protocol bump in this change.
> >> >>
> >> >> Highlighted the broker and producer/consumer configuration changes,
> >> >> added some example values and added the new zookeeper json.
> >> >>
> >> >> Hope this makes things clearer.
> >> >>
> >> >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps 
> wrote:
> >> >> > Hey Gwen,
> >> >> >
> >> >> > Could we get the actual changes in that KIP? I.e. changes to
> metadata
> >> >> > request, changes to UpdateMetadataRequest, new configs and what
> will
> >> >> their
> >> >> > valid values be, etc. This kind of says that those things will
> change
> >> but
> >> >> > doesn't say what they will change to...
> >> >> >
> >> >> > -Jay
> >> >> >
> >> >> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira <
> gshap...@cloudera.com>
> >> >> wrote:
> >> >> >
> >> >> >> I created a KIP for the multi-port broker change.
> >> >> >>
> >> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
> >> >> >>
> >> >> >> I'm not re-opening the discussion, since it was agreed on over a
> >> month
> >> >> >> ago and implementation is close to complete (I hope!). Lets
> consider
> >> >> >> this voted and accepted?
> >> >> >>
> >> >> >> Gwen
> >> >> >>
> >> >> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps 
> >> >> wrote:
> >> >> >> > Great! Sounds like everyone is on the same page
> >> >> >> >
> >> >> >> >- I created a template page to make things easier. If you do
> >> >> >> Tools->Copy
> >> >> >> >on this page you can just fill in the italic portions with
> your
> >> >> >> det

Re: [DISCUSS] KIPs

[Magnus Edenhill]

Wouldn't it make sense to move away from these rich binary broker
descriptors ({ host, port, proto })
(which require protocol churning on change), and simply use URIs instead?

E.g.:
  kafka://[:port]/ <-- cleantext proto on standard port 9092
  kafkas://[:port] <-- SSL enveloped proto on standard port 9093
  kafkas://@[:port]/  <-- SSL enveloped, with user
authentication ..
  kafkafuturetech://.../#opts <-- six months from now.

Trailing #fragment_ids could be used to hint the client on protocol
versions, supported authentications, etc.

This also makes error reporting more meaningful on the client, e.g compare:
  "Unsupported protocol 19 on broker foo:1234"
 to
  "Unsupported protocol kafkafturetech on broker foo:1234"


A positive side effect would be a more generalized topic addressing in
clients:
   kafkacat kafka:///mytopic/3?offset=end  <-- tail partition 3
of mytopic

Just an idea,
Magnus


2015-01-23 5:43 GMT+01:00 Jun Rao :

> Reviewed the latest patch in KAFKA-1809 :).
>
> Thanks,
>
> Jun
>
> On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira 
> wrote:
>
> > Thanks for validating our ideas. Updated the KIP with the workflow.
> >
> > Now if you can nudge Jun to review the latest patch... ;)
> >
> >
> > On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps  wrote:
> > > Oh yeah I think that is better, I hadn't thought of that approach! Any
> > way
> > > you could describe the usage in the KIP, just for completeness?
> > >
> > > -Jay
> > >
> > > On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira 
> > > wrote:
> > >
> > >> I think what you described was the original design, so no wonder you
> > >> are confused :)
> > >>
> > >> Following suggestions from Jun, I changed it a bit. The current model
> > is:
> > >>
> > >> - Clients (producers and consumers) need to know about the broker
> > >> ports in advance. They don't need to know about all brokers, but they
> > >> need to know at least one host:port pair that speaks the protocol they
> > >> want to use. The change is that all host:port pairs in broker.list
> > >> must be of the same protocol and match the security.protocol
> > >> configuration parameter.
> > >>
> > >> - Client uses security.protocol configuration parameter to open a
> > >> connection to one of the brokers and sends the good old
> > >> MetadataRequest. The broker knows which port it got the connection on,
> > >> therefore it knows which security protocol is expected (it needs to
> > >> use the same protocol to accept the connection and respond), and
> > >> therefore it can send a response that contains only the host:port
> > >> pairs that are relevant to that protocol.
> > >>
> > >> - From the client side the MetadataResponse did not change - it
> > >> contains a list of brokerId,host,port that the client can connect to.
> > >> The fact that all those broker endpoints were chosen out of a larger
> > >> collection to match the right protocol is irrelevant for the client.
> > >>
> > >> I really like the new design since it preserves a lot of the same
> > >> configurations and APIs.
> > >>
> > >> Thoughts?
> > >>
> > >> Gwen
> > >>
> > >> On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps 
> wrote:
> > >> > I think I am still confused. In addition to the
> UpdateMetadataRequest
> > >> don't
> > >> > we have to change the MetadataResponse so that it's possible for
> > clients
> > >> to
> > >> > discover the new ports? Or is that a second phase? I was imagining
> it
> > >> > worked by basically allowing the brokers to advertise multiple
> ports,
> > one
> > >> > per security type, and then in the client you configure a protocol
> > which
> > >> > will implicitly choose the port from the options returned in
> metadata
> > to
> > >> > you...
> > >> >
> > >> > Likewise in the ConsumerMetadataResponse we are currently giving
> back
> > >> full
> > >> > broker information. I think we would have two options here: either
> > change
> > >> > the broker information included in that response to match the
> > >> > metadataresponse or else remove the broker information entirely and
> > just
> > >> > return the node id (since in order to use that request you would
> > already
> > >> > have to have the cluster metadata). The second option may be cleaner
> > >> since
> > >> > it means we won't have to continue evolving those two in lockstep...
> > >> >
> > >> > -Jay
> > >> >
> > >> > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira <
> gshap...@cloudera.com>
> > >> wrote:
> > >> >
> > >> >> Good point :)
> > >> >>
> > >> >> I added the specifics of the new  UpdateMetadataRequest, which is
> the
> > >> >> only protocol bump in this change.
> > >> >>
> > >> >> Highlighted the broker and producer/consumer configuration changes,
> > >> >> added some example values and added the new zookeeper json.
> > >> >>
> > >> >> Hope this makes things clearer.
> > >> >>
> > >> >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps 
> > wrote:
> > >> >> > Hey Gwen,
> > >> >> >
> > >> >> > Could we get the actual changes in that KIP? I.e. changes