Re: [DISCUSS] KIPs

[Neha Narkhede]

Thanks Joel. The status table is useful.

On Mon, Feb 9, 2015 at 8:25 PM, Joe Stein joe.st...@stealth.ly wrote:

 I did https://cwiki.apache.org/confluence/display/KAFKA/drafts

 ~ Joestein

 On Mon, Feb 9, 2015 at 11:01 PM, Jay Kreps jay.kr...@gmail.com wrote:

  Yeah no pressure. I think you added a holding area for incomplete KIPs,
  right? I think that is a good idea. We definitely need a place to stash
  these while they are getting built out...
 
  -Jay
 
  On Mon, Feb 9, 2015 at 6:10 PM, Joe Stein joe.st...@stealth.ly wrote:
 
   I like the round-up, looks good, thanks Joel.
  
   I should be able to get KIP-5 and KIP-6 to have more detail in the
 coming
   days.
  
   ~ Joestein
  
   On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy jjkosh...@gmail.com
 wrote:
  
I'm looking through a couple of the KIP threads today and had the
 same
issue; and thought it would be useful to do a status round-up of
 KIPs.
We could incorporate status in the title itself (so we can quickly
 see
it in the child-page list) but I just added a table to the top-level
wiki. Hopefully that captures the current state accurately so I know
which threads to follow-up on.
   
Thanks,
   
Joel
   
On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
 A problem I am having is actually understanding which KIPs are
  intended
to
 be complete proposals and which are works in progress. Joe you seem
  to
have
 a bunch of these. Can you move them elsewhere until they are really
   fully
 done and ready for review and discussion?

 -Jay

 On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps jay.kr...@gmail.com
   wrote:

  I think we are focused on making committing new changes easier,
 but
what
  we have seen is actually that isn't the bulk of the work
  (especially
with
  this kind of public interface change where it generally has a
 big
user
  impact). I think we actually really need the core committers and
  any
other
  interested parties to stop and fully read each KIP and think
 about
   it.
If
  we don't have time to do that we usually just end up spending a
 lot
more
  time after the fact trying to rework things latter when it is a
 lot
harder.
  So I really think we should have every active committer read,
   comment,
and
  vote on each KIP. I think this may require a little bit of work
 to
  co-ordinate/bug people but will end up being worth it because
 each
person
  on the project will have a holistic picture of what is going on.
 
  -Jay
 
  On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com
 
wrote:
 
  Just wanted to add a few more comments on this: KIPs were
  suggested
   as
  a process to help reach early consensus on a major change or not
  so
  major (but tricky or backward incompatible) change in order to
   reduce
  the likelihood of multiple iterations and complete rewrites
 during
  code reviews (which is time-intensive for both the contributor
 and
  reviewers); as well as to reduce the likelihood of surprises
 (say,
   if
  a patch inadvertently changes a public API).  So KIPs are
 intended
   to
  speed up development since a clear path is charted out and there
  is
  prior consensus on whether a feature and its
 design/implementation
  make sense or not.
 
  Obviously this breaks down if KIPs are not being actively
  discussed
   -
  again I think we can do much better here. I think we ended up
  with a
  backlog because as soon as the KIP wiki was started, a number of
  pre-existing jiras and discussions were moved there - all
 within a
   few
  days. Now that there are quite a few outstanding KIPs I think we
   just
  need to methodically work through those - preferably a couple
 at a
  time. I looked through the list and I think we should be able to
  resolve all of them relatively quickly if everyone is on board
  with
  this.
 
Its probably more helpful for contributors if its lazy as
 in
   no
strong objections .
 
  Gwen also suggested this and this also sounds ok to me as I
 wrote
  earlier - what do others think? This is important especially if
  majority in the community think if this less restrictive policy
   would
  spur and not hinder development - I'm not sure that it does. I
  completely agree that KIPs fail to a large degree as far as the
  original motivation goes if they require a lazy majority but the
  DISCUSS threads are stalled. IOW regardless of that discussion,
 I
  think we should rejuvenate some of those threads especially now
  that
  0.8.2 is out of the way.
 
  Thanks,
 
  Joel
 
  On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
   I'm just thinking aloud - I don't know what a good number
 would
   be,
and
  it
   

Re: [DISCUSS] KIPs

[Joe Stein]

I like the round-up, looks good, thanks Joel.

I should be able to get KIP-5 and KIP-6 to have more detail in the coming
days.

~ Joestein

On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy jjkosh...@gmail.com wrote:

 I'm looking through a couple of the KIP threads today and had the same
 issue; and thought it would be useful to do a status round-up of KIPs.
 We could incorporate status in the title itself (so we can quickly see
 it in the child-page list) but I just added a table to the top-level
 wiki. Hopefully that captures the current state accurately so I know
 which threads to follow-up on.

 Thanks,

 Joel

 On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
  A problem I am having is actually understanding which KIPs are intended
 to
  be complete proposals and which are works in progress. Joe you seem to
 have
  a bunch of these. Can you move them elsewhere until they are really fully
  done and ready for review and discussion?
 
  -Jay
 
  On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps jay.kr...@gmail.com wrote:
 
   I think we are focused on making committing new changes easier, but
 what
   we have seen is actually that isn't the bulk of the work (especially
 with
   this kind of public interface change where it generally has a big
 user
   impact). I think we actually really need the core committers and any
 other
   interested parties to stop and fully read each KIP and think about it.
 If
   we don't have time to do that we usually just end up spending a lot
 more
   time after the fact trying to rework things latter when it is a lot
 harder.
   So I really think we should have every active committer read, comment,
 and
   vote on each KIP. I think this may require a little bit of work to
   co-ordinate/bug people but will end up being worth it because each
 person
   on the project will have a holistic picture of what is going on.
  
   -Jay
  
   On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com
 wrote:
  
   Just wanted to add a few more comments on this: KIPs were suggested as
   a process to help reach early consensus on a major change or not so
   major (but tricky or backward incompatible) change in order to reduce
   the likelihood of multiple iterations and complete rewrites during
   code reviews (which is time-intensive for both the contributor and
   reviewers); as well as to reduce the likelihood of surprises (say, if
   a patch inadvertently changes a public API).  So KIPs are intended to
   speed up development since a clear path is charted out and there is
   prior consensus on whether a feature and its design/implementation
   make sense or not.
  
   Obviously this breaks down if KIPs are not being actively discussed -
   again I think we can do much better here. I think we ended up with a
   backlog because as soon as the KIP wiki was started, a number of
   pre-existing jiras and discussions were moved there - all within a few
   days. Now that there are quite a few outstanding KIPs I think we just
   need to methodically work through those - preferably a couple at a
   time. I looked through the list and I think we should be able to
   resolve all of them relatively quickly if everyone is on board with
   this.
  
 Its probably more helpful for contributors if its lazy as in no
 strong objections .
  
   Gwen also suggested this and this also sounds ok to me as I wrote
   earlier - what do others think? This is important especially if
   majority in the community think if this less restrictive policy would
   spur and not hinder development - I'm not sure that it does. I
   completely agree that KIPs fail to a large degree as far as the
   original motivation goes if they require a lazy majority but the
   DISCUSS threads are stalled. IOW regardless of that discussion, I
   think we should rejuvenate some of those threads especially now that
   0.8.2 is out of the way.
  
   Thanks,
  
   Joel
  
   On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
I'm just thinking aloud - I don't know what a good number would be,
 and
   it
is just one possibility to streamline how KIPs are processed. It
 largely
depends on how complex the proposals are. What would be concerning
 is if
there are 10 different threads all dealing with large KIPs and no
 one
   has
the time to give due diligence to each one and all those threads
 grind
   to a
halt due to confusion, incomplete context and misunderstandings.
   
On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:
   
 Joel,
Having only 2 or 3 KIPS under active discussion is
 concerning.
This will slow down development process as well.
 Having a turn-around time for a KIP is a good idea but what will
   happen
 if it didn't received required votes within that time frame.
 Its probably more helpful for contributors if its lazy as in no
 strong objections .
 Just to make sure this is only for KIPs not for regular bug fixes
   

Re: [DISCUSS] KIPs

[Jay Kreps]

Yeah no pressure. I think you added a holding area for incomplete KIPs,
right? I think that is a good idea. We definitely need a place to stash
these while they are getting built out...

-Jay

On Mon, Feb 9, 2015 at 6:10 PM, Joe Stein joe.st...@stealth.ly wrote:

 I like the round-up, looks good, thanks Joel.

 I should be able to get KIP-5 and KIP-6 to have more detail in the coming
 days.

 ~ Joestein

 On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy jjkosh...@gmail.com wrote:

  I'm looking through a couple of the KIP threads today and had the same
  issue; and thought it would be useful to do a status round-up of KIPs.
  We could incorporate status in the title itself (so we can quickly see
  it in the child-page list) but I just added a table to the top-level
  wiki. Hopefully that captures the current state accurately so I know
  which threads to follow-up on.
 
  Thanks,
 
  Joel
 
  On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
   A problem I am having is actually understanding which KIPs are intended
  to
   be complete proposals and which are works in progress. Joe you seem to
  have
   a bunch of these. Can you move them elsewhere until they are really
 fully
   done and ready for review and discussion?
  
   -Jay
  
   On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps jay.kr...@gmail.com
 wrote:
  
I think we are focused on making committing new changes easier, but
  what
we have seen is actually that isn't the bulk of the work (especially
  with
this kind of public interface change where it generally has a big
  user
impact). I think we actually really need the core committers and any
  other
interested parties to stop and fully read each KIP and think about
 it.
  If
we don't have time to do that we usually just end up spending a lot
  more
time after the fact trying to rework things latter when it is a lot
  harder.
So I really think we should have every active committer read,
 comment,
  and
vote on each KIP. I think this may require a little bit of work to
co-ordinate/bug people but will end up being worth it because each
  person
on the project will have a holistic picture of what is going on.
   
-Jay
   
On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com
  wrote:
   
Just wanted to add a few more comments on this: KIPs were suggested
 as
a process to help reach early consensus on a major change or not so
major (but tricky or backward incompatible) change in order to
 reduce
the likelihood of multiple iterations and complete rewrites during
code reviews (which is time-intensive for both the contributor and
reviewers); as well as to reduce the likelihood of surprises (say,
 if
a patch inadvertently changes a public API).  So KIPs are intended
 to
speed up development since a clear path is charted out and there is
prior consensus on whether a feature and its design/implementation
make sense or not.
   
Obviously this breaks down if KIPs are not being actively discussed
 -
again I think we can do much better here. I think we ended up with a
backlog because as soon as the KIP wiki was started, a number of
pre-existing jiras and discussions were moved there - all within a
 few
days. Now that there are quite a few outstanding KIPs I think we
 just
need to methodically work through those - preferably a couple at a
time. I looked through the list and I think we should be able to
resolve all of them relatively quickly if everyone is on board with
this.
   
  Its probably more helpful for contributors if its lazy as in
 no
  strong objections .
   
Gwen also suggested this and this also sounds ok to me as I wrote
earlier - what do others think? This is important especially if
majority in the community think if this less restrictive policy
 would
spur and not hinder development - I'm not sure that it does. I
completely agree that KIPs fail to a large degree as far as the
original motivation goes if they require a lazy majority but the
DISCUSS threads are stalled. IOW regardless of that discussion, I
think we should rejuvenate some of those threads especially now that
0.8.2 is out of the way.
   
Thanks,
   
Joel
   
On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
 I'm just thinking aloud - I don't know what a good number would
 be,
  and
it
 is just one possibility to streamline how KIPs are processed. It
  largely
 depends on how complex the proposals are. What would be concerning
  is if
 there are 10 different threads all dealing with large KIPs and no
  one
has
 the time to give due diligence to each one and all those threads
  grind
to a
 halt due to confusion, incomplete context and misunderstandings.

 On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:

  Joel,
 Having only 2 or 3 KIPS under active discussion is

Re: [DISCUSS] KIPs

[Joel Koshy]

I'm looking through a couple of the KIP threads today and had the same
issue; and thought it would be useful to do a status round-up of KIPs.
We could incorporate status in the title itself (so we can quickly see
it in the child-page list) but I just added a table to the top-level
wiki. Hopefully that captures the current state accurately so I know
which threads to follow-up on.

Thanks,

Joel

On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
 A problem I am having is actually understanding which KIPs are intended to
 be complete proposals and which are works in progress. Joe you seem to have
 a bunch of these. Can you move them elsewhere until they are really fully
 done and ready for review and discussion?
 
 -Jay
 
 On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps jay.kr...@gmail.com wrote:
 
  I think we are focused on making committing new changes easier, but what
  we have seen is actually that isn't the bulk of the work (especially with
  this kind of public interface change where it generally has a big user
  impact). I think we actually really need the core committers and any other
  interested parties to stop and fully read each KIP and think about it. If
  we don't have time to do that we usually just end up spending a lot more
  time after the fact trying to rework things latter when it is a lot harder.
  So I really think we should have every active committer read, comment, and
  vote on each KIP. I think this may require a little bit of work to
  co-ordinate/bug people but will end up being worth it because each person
  on the project will have a holistic picture of what is going on.
 
  -Jay
 
  On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com wrote:
 
  Just wanted to add a few more comments on this: KIPs were suggested as
  a process to help reach early consensus on a major change or not so
  major (but tricky or backward incompatible) change in order to reduce
  the likelihood of multiple iterations and complete rewrites during
  code reviews (which is time-intensive for both the contributor and
  reviewers); as well as to reduce the likelihood of surprises (say, if
  a patch inadvertently changes a public API).  So KIPs are intended to
  speed up development since a clear path is charted out and there is
  prior consensus on whether a feature and its design/implementation
  make sense or not.
 
  Obviously this breaks down if KIPs are not being actively discussed -
  again I think we can do much better here. I think we ended up with a
  backlog because as soon as the KIP wiki was started, a number of
  pre-existing jiras and discussions were moved there - all within a few
  days. Now that there are quite a few outstanding KIPs I think we just
  need to methodically work through those - preferably a couple at a
  time. I looked through the list and I think we should be able to
  resolve all of them relatively quickly if everyone is on board with
  this.
 
Its probably more helpful for contributors if its lazy as in no
strong objections .
 
  Gwen also suggested this and this also sounds ok to me as I wrote
  earlier - what do others think? This is important especially if
  majority in the community think if this less restrictive policy would
  spur and not hinder development - I'm not sure that it does. I
  completely agree that KIPs fail to a large degree as far as the
  original motivation goes if they require a lazy majority but the
  DISCUSS threads are stalled. IOW regardless of that discussion, I
  think we should rejuvenate some of those threads especially now that
  0.8.2 is out of the way.
 
  Thanks,
 
  Joel
 
  On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
   I'm just thinking aloud - I don't know what a good number would be, and
  it
   is just one possibility to streamline how KIPs are processed. It largely
   depends on how complex the proposals are. What would be concerning is if
   there are 10 different threads all dealing with large KIPs and no one
  has
   the time to give due diligence to each one and all those threads grind
  to a
   halt due to confusion, incomplete context and misunderstandings.
  
   On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:
  
Joel,
   Having only 2 or 3 KIPS under active discussion is concerning.
   This will slow down development process as well.
Having a turn-around time for a KIP is a good idea but what will
  happen
if it didn't received required votes within that time frame.
Its probably more helpful for contributors if its lazy as in no
strong objections .
Just to make sure this is only for KIPs not for regular bug fixes
  right?
Thanks,
Harsha
   
   
   
   
   
On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
 Iąm having an impression that KIP is mostly for new features but
  not for
 bug fixes. But I agree with Joel that it might make sense to have
  some
 big
 patches, even if they are bug fixes, to 

Re: [DISCUSS] KIPs

[Joe Stein]

I did https://cwiki.apache.org/confluence/display/KAFKA/drafts

~ Joestein

On Mon, Feb 9, 2015 at 11:01 PM, Jay Kreps jay.kr...@gmail.com wrote:

 Yeah no pressure. I think you added a holding area for incomplete KIPs,
 right? I think that is a good idea. We definitely need a place to stash
 these while they are getting built out...

 -Jay

 On Mon, Feb 9, 2015 at 6:10 PM, Joe Stein joe.st...@stealth.ly wrote:

  I like the round-up, looks good, thanks Joel.
 
  I should be able to get KIP-5 and KIP-6 to have more detail in the coming
  days.
 
  ~ Joestein
 
  On Mon, Feb 9, 2015 at 9:01 PM, Joel Koshy jjkosh...@gmail.com wrote:
 
   I'm looking through a couple of the KIP threads today and had the same
   issue; and thought it would be useful to do a status round-up of KIPs.
   We could incorporate status in the title itself (so we can quickly see
   it in the child-page list) but I just added a table to the top-level
   wiki. Hopefully that captures the current state accurately so I know
   which threads to follow-up on.
  
   Thanks,
  
   Joel
  
   On Fri, Feb 06, 2015 at 12:47:46PM -0800, Jay Kreps wrote:
A problem I am having is actually understanding which KIPs are
 intended
   to
be complete proposals and which are works in progress. Joe you seem
 to
   have
a bunch of these. Can you move them elsewhere until they are really
  fully
done and ready for review and discussion?
   
-Jay
   
On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps jay.kr...@gmail.com
  wrote:
   
 I think we are focused on making committing new changes easier, but
   what
 we have seen is actually that isn't the bulk of the work
 (especially
   with
 this kind of public interface change where it generally has a big
   user
 impact). I think we actually really need the core committers and
 any
   other
 interested parties to stop and fully read each KIP and think about
  it.
   If
 we don't have time to do that we usually just end up spending a lot
   more
 time after the fact trying to rework things latter when it is a lot
   harder.
 So I really think we should have every active committer read,
  comment,
   and
 vote on each KIP. I think this may require a little bit of work to
 co-ordinate/bug people but will end up being worth it because each
   person
 on the project will have a holistic picture of what is going on.

 -Jay

 On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com
   wrote:

 Just wanted to add a few more comments on this: KIPs were
 suggested
  as
 a process to help reach early consensus on a major change or not
 so
 major (but tricky or backward incompatible) change in order to
  reduce
 the likelihood of multiple iterations and complete rewrites during
 code reviews (which is time-intensive for both the contributor and
 reviewers); as well as to reduce the likelihood of surprises (say,
  if
 a patch inadvertently changes a public API).  So KIPs are intended
  to
 speed up development since a clear path is charted out and there
 is
 prior consensus on whether a feature and its design/implementation
 make sense or not.

 Obviously this breaks down if KIPs are not being actively
 discussed
  -
 again I think we can do much better here. I think we ended up
 with a
 backlog because as soon as the KIP wiki was started, a number of
 pre-existing jiras and discussions were moved there - all within a
  few
 days. Now that there are quite a few outstanding KIPs I think we
  just
 need to methodically work through those - preferably a couple at a
 time. I looked through the list and I think we should be able to
 resolve all of them relatively quickly if everyone is on board
 with
 this.

   Its probably more helpful for contributors if its lazy as in
  no
   strong objections .

 Gwen also suggested this and this also sounds ok to me as I wrote
 earlier - what do others think? This is important especially if
 majority in the community think if this less restrictive policy
  would
 spur and not hinder development - I'm not sure that it does. I
 completely agree that KIPs fail to a large degree as far as the
 original motivation goes if they require a lazy majority but the
 DISCUSS threads are stalled. IOW regardless of that discussion, I
 think we should rejuvenate some of those threads especially now
 that
 0.8.2 is out of the way.

 Thanks,

 Joel

 On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
  I'm just thinking aloud - I don't know what a good number would
  be,
   and
 it
  is just one possibility to streamline how KIPs are processed. It
   largely
  depends on how complex the proposals are. What would be
 concerning
   is if
  there are 10 different threads all dealing with large KIPs and
 no
   one
 has
  the time to give 

Re: [DISCUSS] KIPs

[Jay Kreps]

A problem I am having is actually understanding which KIPs are intended to
be complete proposals and which are works in progress. Joe you seem to have
a bunch of these. Can you move them elsewhere until they are really fully
done and ready for review and discussion?

-Jay

On Fri, Feb 6, 2015 at 12:09 PM, Jay Kreps jay.kr...@gmail.com wrote:

 I think we are focused on making committing new changes easier, but what
 we have seen is actually that isn't the bulk of the work (especially with
 this kind of public interface change where it generally has a big user
 impact). I think we actually really need the core committers and any other
 interested parties to stop and fully read each KIP and think about it. If
 we don't have time to do that we usually just end up spending a lot more
 time after the fact trying to rework things latter when it is a lot harder.
 So I really think we should have every active committer read, comment, and
 vote on each KIP. I think this may require a little bit of work to
 co-ordinate/bug people but will end up being worth it because each person
 on the project will have a holistic picture of what is going on.

 -Jay

 On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com wrote:

 Just wanted to add a few more comments on this: KIPs were suggested as
 a process to help reach early consensus on a major change or not so
 major (but tricky or backward incompatible) change in order to reduce
 the likelihood of multiple iterations and complete rewrites during
 code reviews (which is time-intensive for both the contributor and
 reviewers); as well as to reduce the likelihood of surprises (say, if
 a patch inadvertently changes a public API).  So KIPs are intended to
 speed up development since a clear path is charted out and there is
 prior consensus on whether a feature and its design/implementation
 make sense or not.

 Obviously this breaks down if KIPs are not being actively discussed -
 again I think we can do much better here. I think we ended up with a
 backlog because as soon as the KIP wiki was started, a number of
 pre-existing jiras and discussions were moved there - all within a few
 days. Now that there are quite a few outstanding KIPs I think we just
 need to methodically work through those - preferably a couple at a
 time. I looked through the list and I think we should be able to
 resolve all of them relatively quickly if everyone is on board with
 this.

   Its probably more helpful for contributors if its lazy as in no
   strong objections .

 Gwen also suggested this and this also sounds ok to me as I wrote
 earlier - what do others think? This is important especially if
 majority in the community think if this less restrictive policy would
 spur and not hinder development - I'm not sure that it does. I
 completely agree that KIPs fail to a large degree as far as the
 original motivation goes if they require a lazy majority but the
 DISCUSS threads are stalled. IOW regardless of that discussion, I
 think we should rejuvenate some of those threads especially now that
 0.8.2 is out of the way.

 Thanks,

 Joel

 On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
  I'm just thinking aloud - I don't know what a good number would be, and
 it
  is just one possibility to streamline how KIPs are processed. It largely
  depends on how complex the proposals are. What would be concerning is if
  there are 10 different threads all dealing with large KIPs and no one
 has
  the time to give due diligence to each one and all those threads grind
 to a
  halt due to confusion, incomplete context and misunderstandings.
 
  On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:
 
   Joel,
  Having only 2 or 3 KIPS under active discussion is concerning.
  This will slow down development process as well.
   Having a turn-around time for a KIP is a good idea but what will
 happen
   if it didn't received required votes within that time frame.
   Its probably more helpful for contributors if its lazy as in no
   strong objections .
   Just to make sure this is only for KIPs not for regular bug fixes
 right?
   Thanks,
   Harsha
  
  
  
  
  
   On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
Iąm having an impression that KIP is mostly for new features but
 not for
bug fixes. But I agree with Joel that it might make sense to have
 some
big
patches, even if they are bug fixes, to follow the KIP like process
 which
is more strict.
   
Jiangjie (Becket) Qin
   
On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com
 javascript:;
   wrote:
   


 Yes there are KIPs that are currently blocked on feedback/votes,
 but I
 don't think it is an issue of not caring to comment vs having so
 many
 KIPs and other code reviews in flight that people are just
 swamped.


This is exactly my concern.
Even now important patches and features have very long development
 and
review cycles due to 

Re: [DISCUSS] KIPs

[Guozhang Wang]

Here are my cents:

We could use lazy as the default policy for KIP, i.e. :

1. Once a KIP is created at least one committer should engage and provide
feedbacks within a few days;
2. If the committer(s) think it may be a bigger issue / change and requires
broader discussion, she(they) may call out to upgrade to majority or even
consensus and urge other committers to get involved.

Hence it is committers' responsibility to guarantee step 1).

Guozhang

On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com wrote:

 Just wanted to add a few more comments on this: KIPs were suggested as
 a process to help reach early consensus on a major change or not so
 major (but tricky or backward incompatible) change in order to reduce
 the likelihood of multiple iterations and complete rewrites during
 code reviews (which is time-intensive for both the contributor and
 reviewers); as well as to reduce the likelihood of surprises (say, if
 a patch inadvertently changes a public API).  So KIPs are intended to
 speed up development since a clear path is charted out and there is
 prior consensus on whether a feature and its design/implementation
 make sense or not.

 Obviously this breaks down if KIPs are not being actively discussed -
 again I think we can do much better here. I think we ended up with a
 backlog because as soon as the KIP wiki was started, a number of
 pre-existing jiras and discussions were moved there - all within a few
 days. Now that there are quite a few outstanding KIPs I think we just
 need to methodically work through those - preferably a couple at a
 time. I looked through the list and I think we should be able to
 resolve all of them relatively quickly if everyone is on board with
 this.

   Its probably more helpful for contributors if its lazy as in no
   strong objections .

 Gwen also suggested this and this also sounds ok to me as I wrote
 earlier - what do others think? This is important especially if
 majority in the community think if this less restrictive policy would
 spur and not hinder development - I'm not sure that it does. I
 completely agree that KIPs fail to a large degree as far as the
 original motivation goes if they require a lazy majority but the
 DISCUSS threads are stalled. IOW regardless of that discussion, I
 think we should rejuvenate some of those threads especially now that
 0.8.2 is out of the way.

 Thanks,

 Joel

 On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
  I'm just thinking aloud - I don't know what a good number would be, and
 it
  is just one possibility to streamline how KIPs are processed. It largely
  depends on how complex the proposals are. What would be concerning is if
  there are 10 different threads all dealing with large KIPs and no one has
  the time to give due diligence to each one and all those threads grind
 to a
  halt due to confusion, incomplete context and misunderstandings.
 
  On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:
 
   Joel,
  Having only 2 or 3 KIPS under active discussion is concerning.
  This will slow down development process as well.
   Having a turn-around time for a KIP is a good idea but what will happen
   if it didn't received required votes within that time frame.
   Its probably more helpful for contributors if its lazy as in no
   strong objections .
   Just to make sure this is only for KIPs not for regular bug fixes
 right?
   Thanks,
   Harsha
  
  
  
  
  
   On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
Iąm having an impression that KIP is mostly for new features but not
 for
bug fixes. But I agree with Joel that it might make sense to have
 some
big
patches, even if they are bug fixes, to follow the KIP like process
 which
is more strict.
   
Jiangjie (Becket) Qin
   
On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com
 javascript:;
   wrote:
   


 Yes there are KIPs that are currently blocked on feedback/votes,
 but I
 don't think it is an issue of not caring to comment vs having so
 many
 KIPs and other code reviews in flight that people are just
 swamped.


This is exactly my concern.
Even now important patches and features have very long development
 and
review cycles due to Kafka's small and very busy committer
 community. I
feel that this change takes things in the wrong direction


 Joel

 On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
  Isn't requiring 3 binding votes a bit overly strict here? We are
talking
  about patches which in can be fixed, reverted, etc. Not
 releases,
which
  have legal implications.
 
  Why not go with usual definition: lazy = No strong
 objections for
few
  days?
  This means contributors will not be blocked on issues where no
 one
cares
 to
  comment (and we had few of those).
 
  Gwen
 
 
 
  On Thu, Feb 5, 2015 at 4:14 PM, 

Re: [DISCUSS] KIPs

[Joe Stein]

Guozhang's idea is interesting. It does promote collaboration.

Not all KIPs are created equally and anyone (including the person reviewing
it) would be able to call out and request a stronger vote for it to go in.

I think that having 3 x +1 is good (sometimes) because it reduces the risk
of things getting lost in the shuffle and missing changes, which does, will
and always happen.

We also need to discuss 0.8.3.0, 0.9.0.0 and such
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan it
should all be KIP we think should be in where, right?

- Joe Stein

On Fri, Feb 6, 2015 at 2:59 AM, Guozhang Wang wangg...@gmail.com wrote:

 Here are my cents:

 We could use lazy as the default policy for KIP, i.e. :

 1. Once a KIP is created at least one committer should engage and provide
 feedbacks within a few days;
 2. If the committer(s) think it may be a bigger issue / change and requires
 broader discussion, she(they) may call out to upgrade to majority or even
 consensus and urge other committers to get involved.

 Hence it is committers' responsibility to guarantee step 1).

 Guozhang

 On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com wrote:

  Just wanted to add a few more comments on this: KIPs were suggested as
  a process to help reach early consensus on a major change or not so
  major (but tricky or backward incompatible) change in order to reduce
  the likelihood of multiple iterations and complete rewrites during
  code reviews (which is time-intensive for both the contributor and
  reviewers); as well as to reduce the likelihood of surprises (say, if
  a patch inadvertently changes a public API).  So KIPs are intended to
  speed up development since a clear path is charted out and there is
  prior consensus on whether a feature and its design/implementation
  make sense or not.
 
  Obviously this breaks down if KIPs are not being actively discussed -
  again I think we can do much better here. I think we ended up with a
  backlog because as soon as the KIP wiki was started, a number of
  pre-existing jiras and discussions were moved there - all within a few
  days. Now that there are quite a few outstanding KIPs I think we just
  need to methodically work through those - preferably a couple at a
  time. I looked through the list and I think we should be able to
  resolve all of them relatively quickly if everyone is on board with
  this.
 
Its probably more helpful for contributors if its lazy as in no
strong objections .
 
  Gwen also suggested this and this also sounds ok to me as I wrote
  earlier - what do others think? This is important especially if
  majority in the community think if this less restrictive policy would
  spur and not hinder development - I'm not sure that it does. I
  completely agree that KIPs fail to a large degree as far as the
  original motivation goes if they require a lazy majority but the
  DISCUSS threads are stalled. IOW regardless of that discussion, I
  think we should rejuvenate some of those threads especially now that
  0.8.2 is out of the way.
 
  Thanks,
 
  Joel
 
  On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
   I'm just thinking aloud - I don't know what a good number would be, and
  it
   is just one possibility to streamline how KIPs are processed. It
 largely
   depends on how complex the proposals are. What would be concerning is
 if
   there are 10 different threads all dealing with large KIPs and no one
 has
   the time to give due diligence to each one and all those threads grind
  to a
   halt due to confusion, incomplete context and misunderstandings.
  
   On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:
  
Joel,
   Having only 2 or 3 KIPS under active discussion is concerning.
   This will slow down development process as well.
Having a turn-around time for a KIP is a good idea but what will
 happen
if it didn't received required votes within that time frame.
Its probably more helpful for contributors if its lazy as in no
strong objections .
Just to make sure this is only for KIPs not for regular bug fixes
  right?
Thanks,
Harsha
   
   
   
   
   
On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
 Iąm having an impression that KIP is mostly for new features but
 not
  for
 bug fixes. But I agree with Joel that it might make sense to have
  some
 big
 patches, even if they are bug fixes, to follow the KIP like process
  which
 is more strict.

 Jiangjie (Becket) Qin

 On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com
  javascript:;
wrote:

 
 
  Yes there are KIPs that are currently blocked on feedback/votes,
  but I
  don't think it is an issue of not caring to comment vs having so
  many
  KIPs and other code reviews in flight that people are just
  swamped.
 
 
 This is exactly my concern.
 Even now important patches and features have 

Re: [DISCUSS] KIPs

[Jay Kreps]

I think we are focused on making committing new changes easier, but what we
have seen is actually that isn't the bulk of the work (especially with this
kind of public interface change where it generally has a big user
impact). I think we actually really need the core committers and any other
interested parties to stop and fully read each KIP and think about it. If
we don't have time to do that we usually just end up spending a lot more
time after the fact trying to rework things latter when it is a lot harder.
So I really think we should have every active committer read, comment, and
vote on each KIP. I think this may require a little bit of work to
co-ordinate/bug people but will end up being worth it because each person
on the project will have a holistic picture of what is going on.

-Jay

On Thu, Feb 5, 2015 at 11:24 PM, Joel Koshy jjkosh...@gmail.com wrote:

 Just wanted to add a few more comments on this: KIPs were suggested as
 a process to help reach early consensus on a major change or not so
 major (but tricky or backward incompatible) change in order to reduce
 the likelihood of multiple iterations and complete rewrites during
 code reviews (which is time-intensive for both the contributor and
 reviewers); as well as to reduce the likelihood of surprises (say, if
 a patch inadvertently changes a public API).  So KIPs are intended to
 speed up development since a clear path is charted out and there is
 prior consensus on whether a feature and its design/implementation
 make sense or not.

 Obviously this breaks down if KIPs are not being actively discussed -
 again I think we can do much better here. I think we ended up with a
 backlog because as soon as the KIP wiki was started, a number of
 pre-existing jiras and discussions were moved there - all within a few
 days. Now that there are quite a few outstanding KIPs I think we just
 need to methodically work through those - preferably a couple at a
 time. I looked through the list and I think we should be able to
 resolve all of them relatively quickly if everyone is on board with
 this.

   Its probably more helpful for contributors if its lazy as in no
   strong objections .

 Gwen also suggested this and this also sounds ok to me as I wrote
 earlier - what do others think? This is important especially if
 majority in the community think if this less restrictive policy would
 spur and not hinder development - I'm not sure that it does. I
 completely agree that KIPs fail to a large degree as far as the
 original motivation goes if they require a lazy majority but the
 DISCUSS threads are stalled. IOW regardless of that discussion, I
 think we should rejuvenate some of those threads especially now that
 0.8.2 is out of the way.

 Thanks,

 Joel

 On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
  I'm just thinking aloud - I don't know what a good number would be, and
 it
  is just one possibility to streamline how KIPs are processed. It largely
  depends on how complex the proposals are. What would be concerning is if
  there are 10 different threads all dealing with large KIPs and no one has
  the time to give due diligence to each one and all those threads grind
 to a
  halt due to confusion, incomplete context and misunderstandings.
 
  On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:
 
   Joel,
  Having only 2 or 3 KIPS under active discussion is concerning.
  This will slow down development process as well.
   Having a turn-around time for a KIP is a good idea but what will happen
   if it didn't received required votes within that time frame.
   Its probably more helpful for contributors if its lazy as in no
   strong objections .
   Just to make sure this is only for KIPs not for regular bug fixes
 right?
   Thanks,
   Harsha
  
  
  
  
  
   On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
Iąm having an impression that KIP is mostly for new features but not
 for
bug fixes. But I agree with Joel that it might make sense to have
 some
big
patches, even if they are bug fixes, to follow the KIP like process
 which
is more strict.
   
Jiangjie (Becket) Qin
   
On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com
 javascript:;
   wrote:
   


 Yes there are KIPs that are currently blocked on feedback/votes,
 but I
 don't think it is an issue of not caring to comment vs having so
 many
 KIPs and other code reviews in flight that people are just
 swamped.


This is exactly my concern.
Even now important patches and features have very long development
 and
review cycles due to Kafka's small and very busy committer
 community. I
feel that this change takes things in the wrong direction


 Joel

 On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
  Isn't requiring 3 binding votes a bit overly strict here? We are
talking
  about patches which in can be fixed, reverted, etc. Not
 releases,
  

Re: [DISCUSS] KIPs

[Harsha]

Joel,
   Having only 2 or 3 KIPS under active discussion is concerning.
   This will slow down development process as well.
Having a turn-around time for a KIP is a good idea but what will happen
if it didn't received required votes within that time frame.
Its probably more helpful for contributors if its lazy as in no
strong objections .
Just to make sure this is only for KIPs not for regular bug fixes right?
Thanks,
Harsha



   

On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
 I¹m having an impression that KIP is mostly for new features but not for
 bug fixes. But I agree with Joel that it might make sense to have some
 big
 patches, even if they are bug fixes, to follow the KIP like process which
 is more strict.
 
 Jiangjie (Becket) Qin
 
 On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com wrote:
 
 
 
  Yes there are KIPs that are currently blocked on feedback/votes, but I
  don't think it is an issue of not caring to comment vs having so many
  KIPs and other code reviews in flight that people are just swamped.
 
 
 This is exactly my concern.
 Even now important patches and features have very long development and
 review cycles due to Kafka's small and very busy committer community. I
 feel that this change takes things in the wrong direction
 
 
  Joel
 
  On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
   Isn't requiring 3 binding votes a bit overly strict here? We are
 talking
   about patches which in can be fixed, reverted, etc. Not releases,
 which
   have legal implications.
  
   Why not go with usual definition: lazy = No strong objections for
 few
   days?
   This means contributors will not be blocked on issues where no one
 cares
  to
   comment (and we had few of those).
  
   Gwen
  
  
  
   On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com
 wrote:
  
Sorry about this - I actually meant to suggest lazy consensus (which
is a stronger requirement): 3 binding +1 votes and no binding
vetoes.
   
I have updated the wiki to lazy consensus - but can change it back
 if
there is a reasonable objection.
   
On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
 +1

 On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io
  wrote:

  Sounds good.
 
  On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com
  wrote:
 
   None on my part.
  
   -Jay
  
   On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy
 jjkosh...@gmail.com
  
wrote:
  
One amendment I would like to bring up for consideration wrt
  the
KIP
process (before we formally include it in our by-laws) is to
  not
restrict the votes to be a lazy majority of the PMC, and to
  instead
make it a lazy majority of committers.
   
Our current requirement for code changes per our by-laws
 are +1
from a
committer (who is not the contributor) followed by lazy
  approval. I
think a lazy majority vote for more significant code changes
(i.e., a
KIP) should be sufficient.
   
Any objection to this?
   
On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
 Great! Sounds like everyone is on the same page

- I created a template page to make things easier. If
 you
  do
Tools-Copy
on this page you can just fill in the italic portions
 with
your
details.
- I retrofitted KIP-1 to match this formatting
- I added the metadata section people asked for (a link
  to the
discussion, the JIRA, and the current status). Let's
 make
sure we
remember
to update the current status as things are figured out.
- Let's keep the discussion on the mailing list rather
  than
on the
wiki
pages. It makes sense to do one or the other so all the
comments
  are
in one
place and I think prior experience is that the wiki
  comments
are
  the
worse
way.

 I think it would be great do KIPs for some of the
 in-flight
  items
  folks
 mentioned.

 -Jay

 On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
gshap...@cloudera.com
  
wrote:

  +1
 
  Will be happy to provide a KIP for the
 multiple-listeners
patch.
 
  Gwen
 
  On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
joe.st...@stealth.ly
wrote:
   +1 to everything we have been saying and where this
 (has
settled
to)/(is
   settling to).
  
   I am sure other folks have some more feedback and
 think
  we
should
try to
   keep this discussion going if need be. I am also a
 firm
believer
  of
form
   following function so kicking the tires some 

Re: [DISCUSS] KIPs

[Joel Koshy]

I'm just thinking aloud - I don't know what a good number would be, and it
is just one possibility to streamline how KIPs are processed. It largely
depends on how complex the proposals are. What would be concerning is if
there are 10 different threads all dealing with large KIPs and no one has
the time to give due diligence to each one and all those threads grind to a
halt due to confusion, incomplete context and misunderstandings.

On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:

 Joel,
Having only 2 or 3 KIPS under active discussion is concerning.
This will slow down development process as well.
 Having a turn-around time for a KIP is a good idea but what will happen
 if it didn't received required votes within that time frame.
 Its probably more helpful for contributors if its lazy as in no
 strong objections .
 Just to make sure this is only for KIPs not for regular bug fixes right?
 Thanks,
 Harsha





 On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
  I¹m having an impression that KIP is mostly for new features but not for
  bug fixes. But I agree with Joel that it might make sense to have some
  big
  patches, even if they are bug fixes, to follow the KIP like process which
  is more strict.
 
  Jiangjie (Becket) Qin
 
  On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com javascript:;
 wrote:
 
  
  
   Yes there are KIPs that are currently blocked on feedback/votes, but I
   don't think it is an issue of not caring to comment vs having so many
   KIPs and other code reviews in flight that people are just swamped.
  
  
  This is exactly my concern.
  Even now important patches and features have very long development and
  review cycles due to Kafka's small and very busy committer community. I
  feel that this change takes things in the wrong direction
  
  
   Joel
  
   On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
Isn't requiring 3 binding votes a bit overly strict here? We are
  talking
about patches which in can be fixed, reverted, etc. Not releases,
  which
have legal implications.
   
Why not go with usual definition: lazy = No strong objections for
  few
days?
This means contributors will not be blocked on issues where no one
  cares
   to
comment (and we had few of those).
   
Gwen
   
   
   
On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com
 javascript:;
  wrote:
   
 Sorry about this - I actually meant to suggest lazy consensus
 (which
 is a stronger requirement): 3 binding +1 votes and no binding
 vetoes.

 I have updated the wiki to lazy consensus - but can change it back
  if
 there is a reasonable objection.

 On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
  +1
 
  On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede 
 n...@confluent.io javascript:;
   wrote:
 
   Sounds good.
  
   On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps 
 jay.kr...@gmail.com javascript:;
   wrote:
  
None on my part.
   
-Jay
   
On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy
  jjkosh...@gmail.com javascript:;
   
 wrote:
   
 One amendment I would like to bring up for consideration
 wrt
   the
 KIP
 process (before we formally include it in our by-laws) is
 to
   not
 restrict the votes to be a lazy majority of the PMC, and
 to
   instead
 make it a lazy majority of committers.

 Our current requirement for code changes per our by-laws
  are +1
 from a
 committer (who is not the contributor) followed by lazy
   approval. I
 think a lazy majority vote for more significant code
 changes
 (i.e., a
 KIP) should be sufficient.

 Any objection to this?

 On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
  Great! Sounds like everyone is on the same page
 
 - I created a template page to make things easier. If
  you
   do
 Tools-Copy
 on this page you can just fill in the italic portions
  with
 your
 details.
 - I retrofitted KIP-1 to match this formatting
 - I added the metadata section people asked for (a
 link
   to the
 discussion, the JIRA, and the current status). Let's
  make
 sure we
 remember
 to update the current status as things are figured
 out.
 - Let's keep the discussion on the mailing list
 rather
   than
 on the
 wiki
 pages. It makes sense to do one or the other so all
 the
 comments
   are
 in one
 place and I think prior experience is that the wiki
   comments
 are
   the
 worse
 way.
 
  I think it would be great do KIPs for some of the
  in-flight
   items
   folks
  mentioned.
 
  -Jay
 
  On Sat, 

Re: [DISCUSS] KIPs

[Joel Koshy]

Just wanted to add a few more comments on this: KIPs were suggested as
a process to help reach early consensus on a major change or not so
major (but tricky or backward incompatible) change in order to reduce
the likelihood of multiple iterations and complete rewrites during
code reviews (which is time-intensive for both the contributor and
reviewers); as well as to reduce the likelihood of surprises (say, if
a patch inadvertently changes a public API).  So KIPs are intended to
speed up development since a clear path is charted out and there is
prior consensus on whether a feature and its design/implementation
make sense or not.

Obviously this breaks down if KIPs are not being actively discussed -
again I think we can do much better here. I think we ended up with a
backlog because as soon as the KIP wiki was started, a number of
pre-existing jiras and discussions were moved there - all within a few
days. Now that there are quite a few outstanding KIPs I think we just
need to methodically work through those - preferably a couple at a
time. I looked through the list and I think we should be able to
resolve all of them relatively quickly if everyone is on board with
this.

  Its probably more helpful for contributors if its lazy as in no
  strong objections .

Gwen also suggested this and this also sounds ok to me as I wrote
earlier - what do others think? This is important especially if
majority in the community think if this less restrictive policy would
spur and not hinder development - I'm not sure that it does. I
completely agree that KIPs fail to a large degree as far as the
original motivation goes if they require a lazy majority but the
DISCUSS threads are stalled. IOW regardless of that discussion, I
think we should rejuvenate some of those threads especially now that
0.8.2 is out of the way.

Thanks,

Joel

On Thu, Feb 05, 2015 at 08:56:13PM -0800, Joel Koshy wrote:
 I'm just thinking aloud - I don't know what a good number would be, and it
 is just one possibility to streamline how KIPs are processed. It largely
 depends on how complex the proposals are. What would be concerning is if
 there are 10 different threads all dealing with large KIPs and no one has
 the time to give due diligence to each one and all those threads grind to a
 halt due to confusion, incomplete context and misunderstandings.
 
 On Thursday, February 5, 2015, Harsha ka...@harsha.io wrote:
 
  Joel,
 Having only 2 or 3 KIPS under active discussion is concerning.
 This will slow down development process as well.
  Having a turn-around time for a KIP is a good idea but what will happen
  if it didn't received required votes within that time frame.
  Its probably more helpful for contributors if its lazy as in no
  strong objections .
  Just to make sure this is only for KIPs not for regular bug fixes right?
  Thanks,
  Harsha
 
 
 
 
 
  On Thu, Feb 5, 2015, at 05:59 PM, Jiangjie Qin wrote:
   I¹m having an impression that KIP is mostly for new features but not for
   bug fixes. But I agree with Joel that it might make sense to have some
   big
   patches, even if they are bug fixes, to follow the KIP like process which
   is more strict.
  
   Jiangjie (Becket) Qin
  
   On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com javascript:;
  wrote:
  
   
   
Yes there are KIPs that are currently blocked on feedback/votes, but I
don't think it is an issue of not caring to comment vs having so many
KIPs and other code reviews in flight that people are just swamped.
   
   
   This is exactly my concern.
   Even now important patches and features have very long development and
   review cycles due to Kafka's small and very busy committer community. I
   feel that this change takes things in the wrong direction
   
   
Joel
   
On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
 Isn't requiring 3 binding votes a bit overly strict here? We are
   talking
 about patches which in can be fixed, reverted, etc. Not releases,
   which
 have legal implications.

 Why not go with usual definition: lazy = No strong objections for
   few
 days?
 This means contributors will not be blocked on issues where no one
   cares
to
 comment (and we had few of those).

 Gwen



 On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com
  javascript:;
   wrote:

  Sorry about this - I actually meant to suggest lazy consensus
  (which
  is a stronger requirement): 3 binding +1 votes and no binding
  vetoes.
 
  I have updated the wiki to lazy consensus - but can change it back
   if
  there is a reasonable objection.
 
  On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
   +1
  
   On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede 
  n...@confluent.io javascript:;
wrote:
  
Sounds good.
   
On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps 
  jay.kr...@gmail.com 

Re: [DISCUSS] KIPs

[Jay Kreps]

None on my part.

-Jay

On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com wrote:

 One amendment I would like to bring up for consideration wrt the KIP
 process (before we formally include it in our by-laws) is to not
 restrict the votes to be a lazy majority of the PMC, and to instead
 make it a lazy majority of committers.

 Our current requirement for code changes per our by-laws are +1 from a
 committer (who is not the contributor) followed by lazy approval. I
 think a lazy majority vote for more significant code changes (i.e., a
 KIP) should be sufficient.

 Any objection to this?

 On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
  Great! Sounds like everyone is on the same page
 
 - I created a template page to make things easier. If you do
 Tools-Copy
 on this page you can just fill in the italic portions with your
 details.
 - I retrofitted KIP-1 to match this formatting
 - I added the metadata section people asked for (a link to the
 discussion, the JIRA, and the current status). Let's make sure we
 remember
 to update the current status as things are figured out.
 - Let's keep the discussion on the mailing list rather than on the
 wiki
 pages. It makes sense to do one or the other so all the comments are
 in one
 place and I think prior experience is that the wiki comments are the
 worse
 way.
 
  I think it would be great do KIPs for some of the in-flight items folks
  mentioned.
 
  -Jay
 
  On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com
 wrote:
 
   +1
  
   Will be happy to provide a KIP for the multiple-listeners patch.
  
   Gwen
  
   On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly
 wrote:
+1 to everything we have been saying and where this (has settled
 to)/(is
settling to).
   
I am sure other folks have some more feedback and think we should
 try to
keep this discussion going if need be. I am also a firm believer of
 form
following function so kicking the tires some to flesh out the
 details of
this and have some organic growth with the process will be healthy
 and
beneficial to the community.
   
For my part, what I will do is open a few KIP based on some of the
 work I
have been involved with for 0.8.3. Off the top of my head this would
include 1) changes to re-assignment of partitions 2) kafka cli 3)
 global
configs 4) security white list black list by ip 5) SSL 6) We probably
   will
have lots of Security related KIPs and should treat them all
 individually
when the time is appropriate 7) Kafka on Mesos.
   
If someone else wants to jump in to start getting some of the
 security
   KIP
that we are going to have in 0.8.3 I think that would be great (e.g.
Multiple Listeners for Kafka Brokers). There are also a few other
   tickets I
can think of that are important to have in the code in 0.8.3 that
 should
have KIP also that I haven't really been involved in. I will take a
 few
minutes and go through JIRA (one I can think of like auto assign id
 that
   is
already committed I think) and ask for a KIP if appropriate or if I
 feel
that I can write it up (both from a time and understanding
 perspective)
   do
so.
   
long story short, I encourage folks to start moving ahead with the
 KIP
   for
0.8.3 as how we operate. any objections?
   
On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com
   wrote:
   
+1 on the idea, and we could mutually link the KIP wiki page with
 the
   the
created JIRA ticket (i.e. include the JIRA number on the page and
 the
   KIP
url on the ticket description).
   
Regarding the KIP process, probably we do not need two phase
   communication
of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
   clear
while people discuss about that.
   
About who should trigger the process, I think the only involved
 people
would be 1) when the patch is submitted / or even the ticket is
 created,
the assignee could choose to start the KIP process if she thought
 it is
necessary; 2) the reviewer of the patch can also suggest starting
 KIP
discussions.
   
On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
 gshap...@cloudera.com
wrote:
   
 +1 to Ewen's suggestions: Deprecation, status and version.

 Perhaps add the JIRA where the KIP was implemented to the
 metadata.
 This will help tie things together.

 On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
 e...@confluent.io wrote:
  I think adding a section about deprecation would be helpful. A
 good
  fraction of the time I would expect the goal of a KIP is to fix
 or
 replace
  older functionality that needs continued support for
 compatibility,
   but
  should eventually be phased out. This helps Kafka devs
 understand
   how
 long
  they'll end up supporting multiple versions of features 

Re: [DISCUSS] KIPs

[Gwen Shapira]

Isn't requiring 3 binding votes a bit overly strict here? We are talking
about patches which in can be fixed, reverted, etc. Not releases, which
have legal implications.

Why not go with usual definition: lazy = No strong objections for few
days?
This means contributors will not be blocked on issues where no one cares to
comment (and we had few of those).

Gwen



On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com wrote:

 Sorry about this - I actually meant to suggest lazy consensus (which
 is a stronger requirement): 3 binding +1 votes and no binding
 vetoes.

 I have updated the wiki to lazy consensus - but can change it back if
 there is a reasonable objection.

 On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
  +1
 
  On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io wrote:
 
   Sounds good.
  
   On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com wrote:
  
None on my part.
   
-Jay
   
On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com
 wrote:
   
 One amendment I would like to bring up for consideration wrt the
 KIP
 process (before we formally include it in our by-laws) is to not
 restrict the votes to be a lazy majority of the PMC, and to instead
 make it a lazy majority of committers.

 Our current requirement for code changes per our by-laws are +1
 from a
 committer (who is not the contributor) followed by lazy approval. I
 think a lazy majority vote for more significant code changes
 (i.e., a
 KIP) should be sufficient.

 Any objection to this?

 On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
  Great! Sounds like everyone is on the same page
 
 - I created a template page to make things easier. If you do
 Tools-Copy
 on this page you can just fill in the italic portions with
 your
 details.
 - I retrofitted KIP-1 to match this formatting
 - I added the metadata section people asked for (a link to the
 discussion, the JIRA, and the current status). Let's make
 sure we
 remember
 to update the current status as things are figured out.
 - Let's keep the discussion on the mailing list rather than
 on the
 wiki
 pages. It makes sense to do one or the other so all the
 comments
   are
 in one
 place and I think prior experience is that the wiki comments
 are
   the
 worse
 way.
 
  I think it would be great do KIPs for some of the in-flight items
   folks
  mentioned.
 
  -Jay
 
  On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
 gshap...@cloudera.com
   
 wrote:
 
   +1
  
   Will be happy to provide a KIP for the multiple-listeners
 patch.
  
   Gwen
  
   On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
 joe.st...@stealth.ly
 wrote:
+1 to everything we have been saying and where this (has
 settled
 to)/(is
settling to).
   
I am sure other folks have some more feedback and think we
 should
 try to
keep this discussion going if need be. I am also a firm
 believer
   of
 form
following function so kicking the tires some to flesh out the
 details of
this and have some organic growth with the process will be
   healthy
 and
beneficial to the community.
   
For my part, what I will do is open a few KIP based on some
 of
   the
 work I
have been involved with for 0.8.3. Off the top of my head
 this
would
include 1) changes to re-assignment of partitions 2) kafka
 cli 3)
 global
configs 4) security white list black list by ip 5) SSL 6) We
probably
   will
have lots of Security related KIPs and should treat them all
 individually
when the time is appropriate 7) Kafka on Mesos.
   
If someone else wants to jump in to start getting some of the
 security
   KIP
that we are going to have in 0.8.3 I think that would be
 great
(e.g.
Multiple Listeners for Kafka Brokers). There are also a few
 other
   tickets I
can think of that are important to have in the code in 0.8.3
 that
 should
have KIP also that I haven't really been involved in. I will
   take a
 few
minutes and go through JIRA (one I can think of like auto
 assign
   id
 that
   is
already committed I think) and ask for a KIP if appropriate
 or
   if I
 feel
that I can write it up (both from a time and understanding
 perspective)
   do
so.
   
long story short, I encourage folks to start moving ahead
 with
   the
 KIP
   for
0.8.3 as how we operate. any objections?
   
On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
   wangg...@gmail.com

   wrote:
   
+1 on the idea, and we could mutually link the KIP wiki page
   

Re: [DISCUSS] KIPs

[Joe Stein]

+1

On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io wrote:

 Sounds good.

 On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com wrote:

  None on my part.
 
  -Jay
 
  On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com wrote:
 
   One amendment I would like to bring up for consideration wrt the KIP
   process (before we formally include it in our by-laws) is to not
   restrict the votes to be a lazy majority of the PMC, and to instead
   make it a lazy majority of committers.
  
   Our current requirement for code changes per our by-laws are +1 from a
   committer (who is not the contributor) followed by lazy approval. I
   think a lazy majority vote for more significant code changes (i.e., a
   KIP) should be sufficient.
  
   Any objection to this?
  
   On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
Great! Sounds like everyone is on the same page
   
   - I created a template page to make things easier. If you do
   Tools-Copy
   on this page you can just fill in the italic portions with your
   details.
   - I retrofitted KIP-1 to match this formatting
   - I added the metadata section people asked for (a link to the
   discussion, the JIRA, and the current status). Let's make sure we
   remember
   to update the current status as things are figured out.
   - Let's keep the discussion on the mailing list rather than on the
   wiki
   pages. It makes sense to do one or the other so all the comments
 are
   in one
   place and I think prior experience is that the wiki comments are
 the
   worse
   way.
   
I think it would be great do KIPs for some of the in-flight items
 folks
mentioned.
   
-Jay
   
On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com
 
   wrote:
   
 +1

 Will be happy to provide a KIP for the multiple-listeners patch.

 Gwen

 On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly
   wrote:
  +1 to everything we have been saying and where this (has settled
   to)/(is
  settling to).
 
  I am sure other folks have some more feedback and think we should
   try to
  keep this discussion going if need be. I am also a firm believer
 of
   form
  following function so kicking the tires some to flesh out the
   details of
  this and have some organic growth with the process will be
 healthy
   and
  beneficial to the community.
 
  For my part, what I will do is open a few KIP based on some of
 the
   work I
  have been involved with for 0.8.3. Off the top of my head this
  would
  include 1) changes to re-assignment of partitions 2) kafka cli 3)
   global
  configs 4) security white list black list by ip 5) SSL 6) We
  probably
 will
  have lots of Security related KIPs and should treat them all
   individually
  when the time is appropriate 7) Kafka on Mesos.
 
  If someone else wants to jump in to start getting some of the
   security
 KIP
  that we are going to have in 0.8.3 I think that would be great
  (e.g.
  Multiple Listeners for Kafka Brokers). There are also a few other
 tickets I
  can think of that are important to have in the code in 0.8.3 that
   should
  have KIP also that I haven't really been involved in. I will
 take a
   few
  minutes and go through JIRA (one I can think of like auto assign
 id
   that
 is
  already committed I think) and ask for a KIP if appropriate or
 if I
   feel
  that I can write it up (both from a time and understanding
   perspective)
 do
  so.
 
  long story short, I encourage folks to start moving ahead with
 the
   KIP
 for
  0.8.3 as how we operate. any objections?
 
  On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
 wangg...@gmail.com
  
 wrote:
 
  +1 on the idea, and we could mutually link the KIP wiki page
 with
   the
 the
  created JIRA ticket (i.e. include the JIRA number on the page
 and
   the
 KIP
  url on the ticket description).
 
  Regarding the KIP process, probably we do not need two phase
 communication
  of a [DISCUSS] followed by [VOTE], as Jay said the voting should
  be
 clear
  while people discuss about that.
 
  About who should trigger the process, I think the only involved
   people
  would be 1) when the patch is submitted / or even the ticket is
   created,
  the assignee could choose to start the KIP process if she
 thought
   it is
  necessary; 2) the reviewer of the patch can also suggest
 starting
   KIP
  discussions.
 
  On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
   gshap...@cloudera.com
  wrote:
 
   +1 to Ewen's suggestions: Deprecation, status and version.
  
   Perhaps add the JIRA where the KIP was implemented to the
   metadata.
   This will help tie things together.
  
   On Fri, 

Re: [DISCUSS] KIPs

[Joel Koshy]

Sorry about this - I actually meant to suggest lazy consensus (which
is a stronger requirement): 3 binding +1 votes and no binding
vetoes.

I have updated the wiki to lazy consensus - but can change it back if
there is a reasonable objection.

On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
 +1
 
 On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io wrote:
 
  Sounds good.
 
  On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com wrote:
 
   None on my part.
  
   -Jay
  
   On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com wrote:
  
One amendment I would like to bring up for consideration wrt the KIP
process (before we formally include it in our by-laws) is to not
restrict the votes to be a lazy majority of the PMC, and to instead
make it a lazy majority of committers.
   
Our current requirement for code changes per our by-laws are +1 from a
committer (who is not the contributor) followed by lazy approval. I
think a lazy majority vote for more significant code changes (i.e., a
KIP) should be sufficient.
   
Any objection to this?
   
On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
 Great! Sounds like everyone is on the same page

- I created a template page to make things easier. If you do
Tools-Copy
on this page you can just fill in the italic portions with your
details.
- I retrofitted KIP-1 to match this formatting
- I added the metadata section people asked for (a link to the
discussion, the JIRA, and the current status). Let's make sure we
remember
to update the current status as things are figured out.
- Let's keep the discussion on the mailing list rather than on the
wiki
pages. It makes sense to do one or the other so all the comments
  are
in one
place and I think prior experience is that the wiki comments are
  the
worse
way.

 I think it would be great do KIPs for some of the in-flight items
  folks
 mentioned.

 -Jay

 On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com
  
wrote:

  +1
 
  Will be happy to provide a KIP for the multiple-listeners patch.
 
  Gwen
 
  On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly
wrote:
   +1 to everything we have been saying and where this (has settled
to)/(is
   settling to).
  
   I am sure other folks have some more feedback and think we should
try to
   keep this discussion going if need be. I am also a firm believer
  of
form
   following function so kicking the tires some to flesh out the
details of
   this and have some organic growth with the process will be
  healthy
and
   beneficial to the community.
  
   For my part, what I will do is open a few KIP based on some of
  the
work I
   have been involved with for 0.8.3. Off the top of my head this
   would
   include 1) changes to re-assignment of partitions 2) kafka cli 3)
global
   configs 4) security white list black list by ip 5) SSL 6) We
   probably
  will
   have lots of Security related KIPs and should treat them all
individually
   when the time is appropriate 7) Kafka on Mesos.
  
   If someone else wants to jump in to start getting some of the
security
  KIP
   that we are going to have in 0.8.3 I think that would be great
   (e.g.
   Multiple Listeners for Kafka Brokers). There are also a few other
  tickets I
   can think of that are important to have in the code in 0.8.3 that
should
   have KIP also that I haven't really been involved in. I will
  take a
few
   minutes and go through JIRA (one I can think of like auto assign
  id
that
  is
   already committed I think) and ask for a KIP if appropriate or
  if I
feel
   that I can write it up (both from a time and understanding
perspective)
  do
   so.
  
   long story short, I encourage folks to start moving ahead with
  the
KIP
  for
   0.8.3 as how we operate. any objections?
  
   On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
  wangg...@gmail.com
   
  wrote:
  
   +1 on the idea, and we could mutually link the KIP wiki page
  with
the
  the
   created JIRA ticket (i.e. include the JIRA number on the page
  and
the
  KIP
   url on the ticket description).
  
   Regarding the KIP process, probably we do not need two phase
  communication
   of a [DISCUSS] followed by [VOTE], as Jay said the voting should
   be
  clear
   while people discuss about that.
  
   About who should trigger the process, I think the only involved
people
   would be 1) when the patch is submitted / or even the ticket is
created,
   the assignee could choose to start the 

Re: [DISCUSS] KIPs

[Neha Narkhede]

Sounds good.

On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com wrote:

 None on my part.

 -Jay

 On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com wrote:

  One amendment I would like to bring up for consideration wrt the KIP
  process (before we formally include it in our by-laws) is to not
  restrict the votes to be a lazy majority of the PMC, and to instead
  make it a lazy majority of committers.
 
  Our current requirement for code changes per our by-laws are +1 from a
  committer (who is not the contributor) followed by lazy approval. I
  think a lazy majority vote for more significant code changes (i.e., a
  KIP) should be sufficient.
 
  Any objection to this?
 
  On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
   Great! Sounds like everyone is on the same page
  
  - I created a template page to make things easier. If you do
  Tools-Copy
  on this page you can just fill in the italic portions with your
  details.
  - I retrofitted KIP-1 to match this formatting
  - I added the metadata section people asked for (a link to the
  discussion, the JIRA, and the current status). Let's make sure we
  remember
  to update the current status as things are figured out.
  - Let's keep the discussion on the mailing list rather than on the
  wiki
  pages. It makes sense to do one or the other so all the comments are
  in one
  place and I think prior experience is that the wiki comments are the
  worse
  way.
  
   I think it would be great do KIPs for some of the in-flight items folks
   mentioned.
  
   -Jay
  
   On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com
  wrote:
  
+1
   
Will be happy to provide a KIP for the multiple-listeners patch.
   
Gwen
   
On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly
  wrote:
 +1 to everything we have been saying and where this (has settled
  to)/(is
 settling to).

 I am sure other folks have some more feedback and think we should
  try to
 keep this discussion going if need be. I am also a firm believer of
  form
 following function so kicking the tires some to flesh out the
  details of
 this and have some organic growth with the process will be healthy
  and
 beneficial to the community.

 For my part, what I will do is open a few KIP based on some of the
  work I
 have been involved with for 0.8.3. Off the top of my head this
 would
 include 1) changes to re-assignment of partitions 2) kafka cli 3)
  global
 configs 4) security white list black list by ip 5) SSL 6) We
 probably
will
 have lots of Security related KIPs and should treat them all
  individually
 when the time is appropriate 7) Kafka on Mesos.

 If someone else wants to jump in to start getting some of the
  security
KIP
 that we are going to have in 0.8.3 I think that would be great
 (e.g.
 Multiple Listeners for Kafka Brokers). There are also a few other
tickets I
 can think of that are important to have in the code in 0.8.3 that
  should
 have KIP also that I haven't really been involved in. I will take a
  few
 minutes and go through JIRA (one I can think of like auto assign id
  that
is
 already committed I think) and ask for a KIP if appropriate or if I
  feel
 that I can write it up (both from a time and understanding
  perspective)
do
 so.

 long story short, I encourage folks to start moving ahead with the
  KIP
for
 0.8.3 as how we operate. any objections?

 On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com
 
wrote:

 +1 on the idea, and we could mutually link the KIP wiki page with
  the
the
 created JIRA ticket (i.e. include the JIRA number on the page and
  the
KIP
 url on the ticket description).

 Regarding the KIP process, probably we do not need two phase
communication
 of a [DISCUSS] followed by [VOTE], as Jay said the voting should
 be
clear
 while people discuss about that.

 About who should trigger the process, I think the only involved
  people
 would be 1) when the patch is submitted / or even the ticket is
  created,
 the assignee could choose to start the KIP process if she thought
  it is
 necessary; 2) the reviewer of the patch can also suggest starting
  KIP
 discussions.

 On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
  gshap...@cloudera.com
 wrote:

  +1 to Ewen's suggestions: Deprecation, status and version.
 
  Perhaps add the JIRA where the KIP was implemented to the
  metadata.
  This will help tie things together.
 
  On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
  e...@confluent.io wrote:
   I think adding a section about deprecation would be helpful. A
  good
   fraction of the time I would expect the goal of a KIP is to
 fix
  or
  replace

Re: [DISCUSS] KIPs

[Jay Kreps]

Hey Joel,

The problem with lazy consensus is that some people are too lazy. :-) I
think the whole point of this is that you need to actively ensure people
have read and understand the change before you proceed. I basically think
all of us should be reading these proposals and giving feedback.

-Jay

On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com wrote:

 Sorry about this - I actually meant to suggest lazy consensus (which
 is a stronger requirement): 3 binding +1 votes and no binding
 vetoes.

 I have updated the wiki to lazy consensus - but can change it back if
 there is a reasonable objection.

 On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
  +1
 
  On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io wrote:
 
   Sounds good.
  
   On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com wrote:
  
None on my part.
   
-Jay
   
On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com
 wrote:
   
 One amendment I would like to bring up for consideration wrt the
 KIP
 process (before we formally include it in our by-laws) is to not
 restrict the votes to be a lazy majority of the PMC, and to instead
 make it a lazy majority of committers.

 Our current requirement for code changes per our by-laws are +1
 from a
 committer (who is not the contributor) followed by lazy approval. I
 think a lazy majority vote for more significant code changes
 (i.e., a
 KIP) should be sufficient.

 Any objection to this?

 On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
  Great! Sounds like everyone is on the same page
 
 - I created a template page to make things easier. If you do
 Tools-Copy
 on this page you can just fill in the italic portions with
 your
 details.
 - I retrofitted KIP-1 to match this formatting
 - I added the metadata section people asked for (a link to the
 discussion, the JIRA, and the current status). Let's make
 sure we
 remember
 to update the current status as things are figured out.
 - Let's keep the discussion on the mailing list rather than
 on the
 wiki
 pages. It makes sense to do one or the other so all the
 comments
   are
 in one
 place and I think prior experience is that the wiki comments
 are
   the
 worse
 way.
 
  I think it would be great do KIPs for some of the in-flight items
   folks
  mentioned.
 
  -Jay
 
  On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
 gshap...@cloudera.com
   
 wrote:
 
   +1
  
   Will be happy to provide a KIP for the multiple-listeners
 patch.
  
   Gwen
  
   On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
 joe.st...@stealth.ly
 wrote:
+1 to everything we have been saying and where this (has
 settled
 to)/(is
settling to).
   
I am sure other folks have some more feedback and think we
 should
 try to
keep this discussion going if need be. I am also a firm
 believer
   of
 form
following function so kicking the tires some to flesh out the
 details of
this and have some organic growth with the process will be
   healthy
 and
beneficial to the community.
   
For my part, what I will do is open a few KIP based on some
 of
   the
 work I
have been involved with for 0.8.3. Off the top of my head
 this
would
include 1) changes to re-assignment of partitions 2) kafka
 cli 3)
 global
configs 4) security white list black list by ip 5) SSL 6) We
probably
   will
have lots of Security related KIPs and should treat them all
 individually
when the time is appropriate 7) Kafka on Mesos.
   
If someone else wants to jump in to start getting some of the
 security
   KIP
that we are going to have in 0.8.3 I think that would be
 great
(e.g.
Multiple Listeners for Kafka Brokers). There are also a few
 other
   tickets I
can think of that are important to have in the code in 0.8.3
 that
 should
have KIP also that I haven't really been involved in. I will
   take a
 few
minutes and go through JIRA (one I can think of like auto
 assign
   id
 that
   is
already committed I think) and ask for a KIP if appropriate
 or
   if I
 feel
that I can write it up (both from a time and understanding
 perspective)
   do
so.
   
long story short, I encourage folks to start moving ahead
 with
   the
 KIP
   for
0.8.3 as how we operate. any objections?
   
On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang 
   wangg...@gmail.com

   wrote:
   
+1 on the idea, and we could mutually link the KIP wiki page
   with
 the
   the
created JIRA ticket (i.e. 

Re: [DISCUSS] KIPs

[Joel Koshy]

Yes - I realized that afterward. It is back to plain lazy majority.

On Thu, Feb 05, 2015 at 04:33:48PM -0800, Jay Kreps wrote:
 Hey Joel,
 
 The problem with lazy consensus is that some people are too lazy. :-) I
 think the whole point of this is that you need to actively ensure people
 have read and understand the change before you proceed. I basically think
 all of us should be reading these proposals and giving feedback.
 
 -Jay
 
 On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com wrote:
 
  Sorry about this - I actually meant to suggest lazy consensus (which
  is a stronger requirement): 3 binding +1 votes and no binding
  vetoes.
 
  I have updated the wiki to lazy consensus - but can change it back if
  there is a reasonable objection.
 
  On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
   +1
  
   On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io wrote:
  
Sounds good.
   
On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com wrote:
   
 None on my part.

 -Jay

 On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com
  wrote:

  One amendment I would like to bring up for consideration wrt the
  KIP
  process (before we formally include it in our by-laws) is to not
  restrict the votes to be a lazy majority of the PMC, and to instead
  make it a lazy majority of committers.
 
  Our current requirement for code changes per our by-laws are +1
  from a
  committer (who is not the contributor) followed by lazy approval. I
  think a lazy majority vote for more significant code changes
  (i.e., a
  KIP) should be sufficient.
 
  Any objection to this?
 
  On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
   Great! Sounds like everyone is on the same page
  
  - I created a template page to make things easier. If you do
  Tools-Copy
  on this page you can just fill in the italic portions with
  your
  details.
  - I retrofitted KIP-1 to match this formatting
  - I added the metadata section people asked for (a link to the
  discussion, the JIRA, and the current status). Let's make
  sure we
  remember
  to update the current status as things are figured out.
  - Let's keep the discussion on the mailing list rather than
  on the
  wiki
  pages. It makes sense to do one or the other so all the
  comments
are
  in one
  place and I think prior experience is that the wiki comments
  are
the
  worse
  way.
  
   I think it would be great do KIPs for some of the in-flight items
folks
   mentioned.
  
   -Jay
  
   On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
  gshap...@cloudera.com

  wrote:
  
+1
   
Will be happy to provide a KIP for the multiple-listeners
  patch.
   
Gwen
   
On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
  joe.st...@stealth.ly
  wrote:
 +1 to everything we have been saying and where this (has
  settled
  to)/(is
 settling to).

 I am sure other folks have some more feedback and think we
  should
  try to
 keep this discussion going if need be. I am also a firm
  believer
of
  form
 following function so kicking the tires some to flesh out the
  details of
 this and have some organic growth with the process will be
healthy
  and
 beneficial to the community.

 For my part, what I will do is open a few KIP based on some
  of
the
  work I
 have been involved with for 0.8.3. Off the top of my head
  this
 would
 include 1) changes to re-assignment of partitions 2) kafka
  cli 3)
  global
 configs 4) security white list black list by ip 5) SSL 6) We
 probably
will
 have lots of Security related KIPs and should treat them all
  individually
 when the time is appropriate 7) Kafka on Mesos.

 If someone else wants to jump in to start getting some of the
  security
KIP
 that we are going to have in 0.8.3 I think that would be
  great
 (e.g.
 Multiple Listeners for Kafka Brokers). There are also a few
  other
tickets I
 can think of that are important to have in the code in 0.8.3
  that
  should
 have KIP also that I haven't really been involved in. I will
take a
  few
 minutes and go through JIRA (one I can think of like auto
  assign
id
  that
is
 already committed I think) and ask for a KIP if appropriate
  or
if I
  feel
 that I can write it up (both from a time and understanding
  perspective)
do
 so.

 long story short, I encourage folks to start moving ahead
  with
the
  KIP
for
 

Re: [DISCUSS] KIPs

[Joel Koshy]

The original requirement was lazy majority of PMC which definitely
seems overly restrictive.

 Why not go with usual definition: lazy = No strong objections for few
 days?
 This means contributors will not be blocked on issues where no one cares to
 comment (and we had few of those).

I think one benefit of not doing the just lazy approach would be to
ensure that the proposal has been vetted - otherwise a contributor may
go ahead and spend a ton of time proceeding with an implementation
plan only to receive a -1 on a code change which automatically moves
to a lazy majority anyway. If a change is non-controversial and not
particularly extensive, then a KIP is not required in the first place
so this is less of an issue (i.e., if the code review moves to a lazy
majority).

For a large piece of work (that encourage a KIP), the more restrictive
voting requirements probably make sense to avoid the undesirable
scenario of a lot of effort spent on a feature only to have it
subsequently voted down. Does this sound reasonable?  I changed the
wiki back to lazy majority - which makes it less restrictive than it
was, but I think we should all agree whether to make it even less
restrictive or not.

Yes there are KIPs that are currently blocked on feedback/votes, but I
don't think it is an issue of not caring to comment vs having so many
KIPs and other code reviews in flight that people are just swamped.

Joel

On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
 Isn't requiring 3 binding votes a bit overly strict here? We are talking
 about patches which in can be fixed, reverted, etc. Not releases, which
 have legal implications.
 
 Why not go with usual definition: lazy = No strong objections for few
 days?
 This means contributors will not be blocked on issues where no one cares to
 comment (and we had few of those).
 
 Gwen
 
 
 
 On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com wrote:
 
  Sorry about this - I actually meant to suggest lazy consensus (which
  is a stronger requirement): 3 binding +1 votes and no binding
  vetoes.
 
  I have updated the wiki to lazy consensus - but can change it back if
  there is a reasonable objection.
 
  On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
   +1
  
   On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io wrote:
  
Sounds good.
   
On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com wrote:
   
 None on my part.

 -Jay

 On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com
  wrote:

  One amendment I would like to bring up for consideration wrt the
  KIP
  process (before we formally include it in our by-laws) is to not
  restrict the votes to be a lazy majority of the PMC, and to instead
  make it a lazy majority of committers.
 
  Our current requirement for code changes per our by-laws are +1
  from a
  committer (who is not the contributor) followed by lazy approval. I
  think a lazy majority vote for more significant code changes
  (i.e., a
  KIP) should be sufficient.
 
  Any objection to this?
 
  On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
   Great! Sounds like everyone is on the same page
  
  - I created a template page to make things easier. If you do
  Tools-Copy
  on this page you can just fill in the italic portions with
  your
  details.
  - I retrofitted KIP-1 to match this formatting
  - I added the metadata section people asked for (a link to the
  discussion, the JIRA, and the current status). Let's make
  sure we
  remember
  to update the current status as things are figured out.
  - Let's keep the discussion on the mailing list rather than
  on the
  wiki
  pages. It makes sense to do one or the other so all the
  comments
are
  in one
  place and I think prior experience is that the wiki comments
  are
the
  worse
  way.
  
   I think it would be great do KIPs for some of the in-flight items
folks
   mentioned.
  
   -Jay
  
   On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
  gshap...@cloudera.com

  wrote:
  
+1
   
Will be happy to provide a KIP for the multiple-listeners
  patch.
   
Gwen
   
On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
  joe.st...@stealth.ly
  wrote:
 +1 to everything we have been saying and where this (has
  settled
  to)/(is
 settling to).

 I am sure other folks have some more feedback and think we
  should
  try to
 keep this discussion going if need be. I am also a firm
  believer
of
  form
 following function so kicking the tires some to flesh out the
  details of
 this and have some organic growth with the process will be
healthy
  and
 beneficial to the 

Re: [DISCUSS] KIPs

[Joel Koshy]

 This is exactly my concern.
 Even now important patches and features have very long development and
 review cycles due to Kafka's small and very busy committer community. I
 feel that this change takes things in the wrong direction

I agree that we should improve on this.  I think the only concern in
making the vote less restrictive is what I mentioned earlier - i.e., a
contributor starting on a major feature only to have to rework or
abandon a design later during code reviews which is even more
frustrating.

We could also discuss having some SLA in place on turn-around time for
a KIP (say, a week?). Given that KIPs are (sort of by definition)
major features and/or backward incompatible changes it may even be
reasonable to have at most only two or three KIPs under active
discussion at any given time. That impacts feature velocity but I
don't see a way out of that.

BTW (personally) I'm fine with making it less restrictive - since we
have been implicitly doing that so far in the absence of KIPs.  I
think KIPs are useful by themselves (i.e., voting aside).

Thanks,

Joel

On Thu, Feb 05, 2015 at 04:57:03PM -0800, Gwen Shapira wrote:
 
 
  Yes there are KIPs that are currently blocked on feedback/votes, but I
  don't think it is an issue of not caring to comment vs having so many
  KIPs and other code reviews in flight that people are just swamped.
 
 
 This is exactly my concern.
 Even now important patches and features have very long development and
 review cycles due to Kafka's small and very busy committer community. I
 feel that this change takes things in the wrong direction
 
 
  Joel
 
  On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
   Isn't requiring 3 binding votes a bit overly strict here? We are talking
   about patches which in can be fixed, reverted, etc. Not releases, which
   have legal implications.
  
   Why not go with usual definition: lazy = No strong objections for few
   days?
   This means contributors will not be blocked on issues where no one cares
  to
   comment (and we had few of those).
  
   Gwen
  
  
  
   On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com wrote:
  
Sorry about this - I actually meant to suggest lazy consensus (which
is a stronger requirement): 3 binding +1 votes and no binding
vetoes.
   
I have updated the wiki to lazy consensus - but can change it back if
there is a reasonable objection.
   
On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
 +1

 On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io
  wrote:

  Sounds good.
 
  On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com
  wrote:
 
   None on my part.
  
   -Jay
  
   On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com
  
wrote:
  
One amendment I would like to bring up for consideration wrt
  the
KIP
process (before we formally include it in our by-laws) is to
  not
restrict the votes to be a lazy majority of the PMC, and to
  instead
make it a lazy majority of committers.
   
Our current requirement for code changes per our by-laws are +1
from a
committer (who is not the contributor) followed by lazy
  approval. I
think a lazy majority vote for more significant code changes
(i.e., a
KIP) should be sufficient.
   
Any objection to this?
   
On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
 Great! Sounds like everyone is on the same page

- I created a template page to make things easier. If you
  do
Tools-Copy
on this page you can just fill in the italic portions with
your
details.
- I retrofitted KIP-1 to match this formatting
- I added the metadata section people asked for (a link
  to the
discussion, the JIRA, and the current status). Let's make
sure we
remember
to update the current status as things are figured out.
- Let's keep the discussion on the mailing list rather
  than
on the
wiki
pages. It makes sense to do one or the other so all the
comments
  are
in one
place and I think prior experience is that the wiki
  comments
are
  the
worse
way.

 I think it would be great do KIPs for some of the in-flight
  items
  folks
 mentioned.

 -Jay

 On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
gshap...@cloudera.com
  
wrote:

  +1
 
  Will be happy to provide a KIP for the multiple-listeners
patch.
 
  Gwen
 
  On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
joe.st...@stealth.ly
wrote:
   +1 to everything we have been saying and where this (has
settled

Re: [DISCUSS] KIPs

[Jiangjie Qin]

I¹m having an impression that KIP is mostly for new features but not for
bug fixes. But I agree with Joel that it might make sense to have some big
patches, even if they are bug fixes, to follow the KIP like process which
is more strict.

Jiangjie (Becket) Qin

On 2/5/15, 4:57 PM, Gwen Shapira gshap...@cloudera.com wrote:



 Yes there are KIPs that are currently blocked on feedback/votes, but I
 don't think it is an issue of not caring to comment vs having so many
 KIPs and other code reviews in flight that people are just swamped.


This is exactly my concern.
Even now important patches and features have very long development and
review cycles due to Kafka's small and very busy committer community. I
feel that this change takes things in the wrong direction


 Joel

 On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
  Isn't requiring 3 binding votes a bit overly strict here? We are
talking
  about patches which in can be fixed, reverted, etc. Not releases,
which
  have legal implications.
 
  Why not go with usual definition: lazy = No strong objections for
few
  days?
  This means contributors will not be blocked on issues where no one
cares
 to
  comment (and we had few of those).
 
  Gwen
 
 
 
  On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com
wrote:
 
   Sorry about this - I actually meant to suggest lazy consensus (which
   is a stronger requirement): 3 binding +1 votes and no binding
   vetoes.
  
   I have updated the wiki to lazy consensus - but can change it back
if
   there is a reasonable objection.
  
   On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
+1
   
On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io
 wrote:
   
 Sounds good.

 On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com
 wrote:

  None on my part.
 
  -Jay
 
  On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy
jjkosh...@gmail.com
 
   wrote:
 
   One amendment I would like to bring up for consideration wrt
 the
   KIP
   process (before we formally include it in our by-laws) is to
 not
   restrict the votes to be a lazy majority of the PMC, and to
 instead
   make it a lazy majority of committers.
  
   Our current requirement for code changes per our by-laws
are +1
   from a
   committer (who is not the contributor) followed by lazy
 approval. I
   think a lazy majority vote for more significant code changes
   (i.e., a
   KIP) should be sufficient.
  
   Any objection to this?
  
   On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
Great! Sounds like everyone is on the same page
   
   - I created a template page to make things easier. If
you
 do
   Tools-Copy
   on this page you can just fill in the italic portions
with
   your
   details.
   - I retrofitted KIP-1 to match this formatting
   - I added the metadata section people asked for (a link
 to the
   discussion, the JIRA, and the current status). Let's
make
   sure we
   remember
   to update the current status as things are figured out.
   - Let's keep the discussion on the mailing list rather
 than
   on the
   wiki
   pages. It makes sense to do one or the other so all the
   comments
 are
   in one
   place and I think prior experience is that the wiki
 comments
   are
 the
   worse
   way.
   
I think it would be great do KIPs for some of the
in-flight
 items
 folks
mentioned.
   
-Jay
   
On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
   gshap...@cloudera.com
 
   wrote:
   
 +1

 Will be happy to provide a KIP for the
multiple-listeners
   patch.

 Gwen

 On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
   joe.st...@stealth.ly
   wrote:
  +1 to everything we have been saying and where this
(has
   settled
   to)/(is
  settling to).
 
  I am sure other folks have some more feedback and
think
 we
   should
   try to
  keep this discussion going if need be. I am also a
firm
   believer
 of
   form
  following function so kicking the tires some to flesh
 out the
   details of
  this and have some organic growth with the process
will
 be
 healthy
   and
  beneficial to the community.
 
  For my part, what I will do is open a few KIP based on
 some
   of
 the
   work I
  have been involved with for 0.8.3. Off the top of my
head
   this
  would
  include 1) changes to re-assignment of partitions 2)
 kafka
   cli 3)
   global
  configs 4) security white list black list by ip 5) SSL
 6) We
  probably
 will
  have lots of Security related KIPs and should treat
them
 all
   individually
  when the 

Re: [DISCUSS] KIPs

[Gwen Shapira]

 Yes there are KIPs that are currently blocked on feedback/votes, but I
 don't think it is an issue of not caring to comment vs having so many
 KIPs and other code reviews in flight that people are just swamped.


This is exactly my concern.
Even now important patches and features have very long development and
review cycles due to Kafka's small and very busy committer community. I
feel that this change takes things in the wrong direction


 Joel

 On Thu, Feb 05, 2015 at 04:19:54PM -0800, Gwen Shapira wrote:
  Isn't requiring 3 binding votes a bit overly strict here? We are talking
  about patches which in can be fixed, reverted, etc. Not releases, which
  have legal implications.
 
  Why not go with usual definition: lazy = No strong objections for few
  days?
  This means contributors will not be blocked on issues where no one cares
 to
  comment (and we had few of those).
 
  Gwen
 
 
 
  On Thu, Feb 5, 2015 at 4:14 PM, Joel Koshy jjkosh...@gmail.com wrote:
 
   Sorry about this - I actually meant to suggest lazy consensus (which
   is a stronger requirement): 3 binding +1 votes and no binding
   vetoes.
  
   I have updated the wiki to lazy consensus - but can change it back if
   there is a reasonable objection.
  
   On Thu, Feb 05, 2015 at 06:17:44PM -0500, Joe Stein wrote:
+1
   
On Thu, Feb 5, 2015 at 6:16 PM, Neha Narkhede n...@confluent.io
 wrote:
   
 Sounds good.

 On Thu, Feb 5, 2015 at 2:35 PM, Jay Kreps jay.kr...@gmail.com
 wrote:

  None on my part.
 
  -Jay
 
  On Thu, Feb 5, 2015 at 11:50 AM, Joel Koshy jjkosh...@gmail.com
 
   wrote:
 
   One amendment I would like to bring up for consideration wrt
 the
   KIP
   process (before we formally include it in our by-laws) is to
 not
   restrict the votes to be a lazy majority of the PMC, and to
 instead
   make it a lazy majority of committers.
  
   Our current requirement for code changes per our by-laws are +1
   from a
   committer (who is not the contributor) followed by lazy
 approval. I
   think a lazy majority vote for more significant code changes
   (i.e., a
   KIP) should be sufficient.
  
   Any objection to this?
  
   On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
Great! Sounds like everyone is on the same page
   
   - I created a template page to make things easier. If you
 do
   Tools-Copy
   on this page you can just fill in the italic portions with
   your
   details.
   - I retrofitted KIP-1 to match this formatting
   - I added the metadata section people asked for (a link
 to the
   discussion, the JIRA, and the current status). Let's make
   sure we
   remember
   to update the current status as things are figured out.
   - Let's keep the discussion on the mailing list rather
 than
   on the
   wiki
   pages. It makes sense to do one or the other so all the
   comments
 are
   in one
   place and I think prior experience is that the wiki
 comments
   are
 the
   worse
   way.
   
I think it would be great do KIPs for some of the in-flight
 items
 folks
mentioned.
   
-Jay
   
On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
   gshap...@cloudera.com
 
   wrote:
   
 +1

 Will be happy to provide a KIP for the multiple-listeners
   patch.

 Gwen

 On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein 
   joe.st...@stealth.ly
   wrote:
  +1 to everything we have been saying and where this (has
   settled
   to)/(is
  settling to).
 
  I am sure other folks have some more feedback and think
 we
   should
   try to
  keep this discussion going if need be. I am also a firm
   believer
 of
   form
  following function so kicking the tires some to flesh
 out the
   details of
  this and have some organic growth with the process will
 be
 healthy
   and
  beneficial to the community.
 
  For my part, what I will do is open a few KIP based on
 some
   of
 the
   work I
  have been involved with for 0.8.3. Off the top of my head
   this
  would
  include 1) changes to re-assignment of partitions 2)
 kafka
   cli 3)
   global
  configs 4) security white list black list by ip 5) SSL
 6) We
  probably
 will
  have lots of Security related KIPs and should treat them
 all
   individually
  when the time is appropriate 7) Kafka on Mesos.
 
  If someone else wants to jump in to start getting some
 of the
   security
 KIP
  that we are going to have in 0.8.3 I think that would be
   great
  (e.g.
  Multiple Listeners for Kafka Brokers). There are also a
 few
   other
   

Re: [DISCUSS] KIPs

[Joel Koshy]

One amendment I would like to bring up for consideration wrt the KIP
process (before we formally include it in our by-laws) is to not
restrict the votes to be a lazy majority of the PMC, and to instead
make it a lazy majority of committers.

Our current requirement for code changes per our by-laws are +1 from a
committer (who is not the contributor) followed by lazy approval. I
think a lazy majority vote for more significant code changes (i.e., a
KIP) should be sufficient.

Any objection to this?

On Sun, Jan 18, 2015 at 10:31:08AM -0800, Jay Kreps wrote:
 Great! Sounds like everyone is on the same page
 
- I created a template page to make things easier. If you do Tools-Copy
on this page you can just fill in the italic portions with your details.
- I retrofitted KIP-1 to match this formatting
- I added the metadata section people asked for (a link to the
discussion, the JIRA, and the current status). Let's make sure we remember
to update the current status as things are figured out.
- Let's keep the discussion on the mailing list rather than on the wiki
pages. It makes sense to do one or the other so all the comments are in one
place and I think prior experience is that the wiki comments are the worse
way.
 
 I think it would be great do KIPs for some of the in-flight items folks
 mentioned.
 
 -Jay
 
 On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com wrote:
 
  +1
 
  Will be happy to provide a KIP for the multiple-listeners patch.
 
  Gwen
 
  On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly wrote:
   +1 to everything we have been saying and where this (has settled to)/(is
   settling to).
  
   I am sure other folks have some more feedback and think we should try to
   keep this discussion going if need be. I am also a firm believer of form
   following function so kicking the tires some to flesh out the details of
   this and have some organic growth with the process will be healthy and
   beneficial to the community.
  
   For my part, what I will do is open a few KIP based on some of the work I
   have been involved with for 0.8.3. Off the top of my head this would
   include 1) changes to re-assignment of partitions 2) kafka cli 3) global
   configs 4) security white list black list by ip 5) SSL 6) We probably
  will
   have lots of Security related KIPs and should treat them all individually
   when the time is appropriate 7) Kafka on Mesos.
  
   If someone else wants to jump in to start getting some of the security
  KIP
   that we are going to have in 0.8.3 I think that would be great (e.g.
   Multiple Listeners for Kafka Brokers). There are also a few other
  tickets I
   can think of that are important to have in the code in 0.8.3 that should
   have KIP also that I haven't really been involved in. I will take a few
   minutes and go through JIRA (one I can think of like auto assign id that
  is
   already committed I think) and ask for a KIP if appropriate or if I feel
   that I can write it up (both from a time and understanding perspective)
  do
   so.
  
   long story short, I encourage folks to start moving ahead with the KIP
  for
   0.8.3 as how we operate. any objections?
  
   On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com
  wrote:
  
   +1 on the idea, and we could mutually link the KIP wiki page with the
  the
   created JIRA ticket (i.e. include the JIRA number on the page and the
  KIP
   url on the ticket description).
  
   Regarding the KIP process, probably we do not need two phase
  communication
   of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
  clear
   while people discuss about that.
  
   About who should trigger the process, I think the only involved people
   would be 1) when the patch is submitted / or even the ticket is created,
   the assignee could choose to start the KIP process if she thought it is
   necessary; 2) the reviewer of the patch can also suggest starting KIP
   discussions.
  
   On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira gshap...@cloudera.com
   wrote:
  
+1 to Ewen's suggestions: Deprecation, status and version.
   
Perhaps add the JIRA where the KIP was implemented to the metadata.
This will help tie things together.
   
On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
e...@confluent.io wrote:
 I think adding a section about deprecation would be helpful. A good
 fraction of the time I would expect the goal of a KIP is to fix or
replace
 older functionality that needs continued support for compatibility,
  but
 should eventually be phased out. This helps Kafka devs understand
  how
long
 they'll end up supporting multiple versions of features and helps
  users
 understand when they're going to have to make updates to their
applications.

 Less important but useful -- having a bit of standard metadata like
   PEPs
 do. Two I think are important are status (if 

Re: [DISCUSS] KIPs

[Gwen Shapira]

Sorry for late response, Magnus. See my comments inline:

On Fri, Jan 23, 2015 at 7:31 AM, Magnus Edenhill mag...@edenhill.se wrote:
 Wouldn't it make sense to move away from these rich binary broker
 descriptors ({ host, port, proto })
 (which require protocol churning on change), and simply use URIs instead?

We use different representations in different places, so I'm not sure
which one you mean...

Our clients will use host:port and --security.protocol in the
configuration to specify the protocol (this is to make it easier to
ensure one protocol per client)
ZK registration is in JSON
Internal objects are binary, since parsing the URI over and over will be a PITA
Wire protocol doesn't include a security protocol (since we don't
negotiate it), so its still host+port, as it always was.


 E.g.:
   kafka://host[:port]/ -- cleantext proto on standard port 9092
   kafkas://host[:port] -- SSL enveloped proto on standard port 9093
   kafkas://user:pass@host[:port]/  -- SSL enveloped, with user
 authentication ..
   kafkafuturetech://.../#opts -- six months from now.

 Trailing #fragment_ids could be used to hint the client on protocol
 versions, supported authentications, etc.

 This also makes error reporting more meaningful on the client, e.g compare:
   Unsupported protocol 19 on broker foo:1234
  to
   Unsupported protocol kafkafturetech on broker foo:1234

I agree that the second error is more readable, but I'm not sure why
you think its currently unfeasible on clients?


 A positive side effect would be a more generalized topic addressing in
 clients:
kafkacat kafka://bootstrap/mytopic/3?offset=end  -- tail partition 3
 of mytopic

Clients can pretty much do what they want, right? As long as they call
Kafka with the right configuration, its up to them to decide how to
accept arguments.

 Just an idea,
 Magnus


 2015-01-23 5:43 GMT+01:00 Jun Rao j...@confluent.io:

 Reviewed the latest patch in KAFKA-1809 :).

 Thanks,

 Jun

 On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira gshap...@cloudera.com
 wrote:

  Thanks for validating our ideas. Updated the KIP with the workflow.
 
  Now if you can nudge Jun to review the latest patch... ;)
 
 
  On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps j...@confluent.io wrote:
   Oh yeah I think that is better, I hadn't thought of that approach! Any
  way
   you could describe the usage in the KIP, just for completeness?
  
   -Jay
  
   On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira gshap...@cloudera.com
   wrote:
  
   I think what you described was the original design, so no wonder you
   are confused :)
  
   Following suggestions from Jun, I changed it a bit. The current model
  is:
  
   - Clients (producers and consumers) need to know about the broker
   ports in advance. They don't need to know about all brokers, but they
   need to know at least one host:port pair that speaks the protocol they
   want to use. The change is that all host:port pairs in broker.list
   must be of the same protocol and match the security.protocol
   configuration parameter.
  
   - Client uses security.protocol configuration parameter to open a
   connection to one of the brokers and sends the good old
   MetadataRequest. The broker knows which port it got the connection on,
   therefore it knows which security protocol is expected (it needs to
   use the same protocol to accept the connection and respond), and
   therefore it can send a response that contains only the host:port
   pairs that are relevant to that protocol.
  
   - From the client side the MetadataResponse did not change - it
   contains a list of brokerId,host,port that the client can connect to.
   The fact that all those broker endpoints were chosen out of a larger
   collection to match the right protocol is irrelevant for the client.
  
   I really like the new design since it preserves a lot of the same
   configurations and APIs.
  
   Thoughts?
  
   Gwen
  
   On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps jay.kr...@gmail.com
 wrote:
I think I am still confused. In addition to the
 UpdateMetadataRequest
   don't
we have to change the MetadataResponse so that it's possible for
  clients
   to
discover the new ports? Or is that a second phase? I was imagining
 it
worked by basically allowing the brokers to advertise multiple
 ports,
  one
per security type, and then in the client you configure a protocol
  which
will implicitly choose the port from the options returned in
 metadata
  to
you...
   
Likewise in the ConsumerMetadataResponse we are currently giving
 back
   full
broker information. I think we would have two options here: either
  change
the broker information included in that response to match the
metadataresponse or else remove the broker information entirely and
  just
return the node id (since in order to use that request you would
  already
have to have the cluster metadata). The second option may be cleaner
   since
it means we 

Re: [DISCUSS] KIPs

[Magnus Edenhill]

Wouldn't it make sense to move away from these rich binary broker
descriptors ({ host, port, proto })
(which require protocol churning on change), and simply use URIs instead?

E.g.:
  kafka://host[:port]/ -- cleantext proto on standard port 9092
  kafkas://host[:port] -- SSL enveloped proto on standard port 9093
  kafkas://user:pass@host[:port]/  -- SSL enveloped, with user
authentication ..
  kafkafuturetech://.../#opts -- six months from now.

Trailing #fragment_ids could be used to hint the client on protocol
versions, supported authentications, etc.

This also makes error reporting more meaningful on the client, e.g compare:
  Unsupported protocol 19 on broker foo:1234
 to
  Unsupported protocol kafkafturetech on broker foo:1234


A positive side effect would be a more generalized topic addressing in
clients:
   kafkacat kafka://bootstrap/mytopic/3?offset=end  -- tail partition 3
of mytopic

Just an idea,
Magnus


2015-01-23 5:43 GMT+01:00 Jun Rao j...@confluent.io:

 Reviewed the latest patch in KAFKA-1809 :).

 Thanks,

 Jun

 On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira gshap...@cloudera.com
 wrote:

  Thanks for validating our ideas. Updated the KIP with the workflow.
 
  Now if you can nudge Jun to review the latest patch... ;)
 
 
  On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps j...@confluent.io wrote:
   Oh yeah I think that is better, I hadn't thought of that approach! Any
  way
   you could describe the usage in the KIP, just for completeness?
  
   -Jay
  
   On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira gshap...@cloudera.com
   wrote:
  
   I think what you described was the original design, so no wonder you
   are confused :)
  
   Following suggestions from Jun, I changed it a bit. The current model
  is:
  
   - Clients (producers and consumers) need to know about the broker
   ports in advance. They don't need to know about all brokers, but they
   need to know at least one host:port pair that speaks the protocol they
   want to use. The change is that all host:port pairs in broker.list
   must be of the same protocol and match the security.protocol
   configuration parameter.
  
   - Client uses security.protocol configuration parameter to open a
   connection to one of the brokers and sends the good old
   MetadataRequest. The broker knows which port it got the connection on,
   therefore it knows which security protocol is expected (it needs to
   use the same protocol to accept the connection and respond), and
   therefore it can send a response that contains only the host:port
   pairs that are relevant to that protocol.
  
   - From the client side the MetadataResponse did not change - it
   contains a list of brokerId,host,port that the client can connect to.
   The fact that all those broker endpoints were chosen out of a larger
   collection to match the right protocol is irrelevant for the client.
  
   I really like the new design since it preserves a lot of the same
   configurations and APIs.
  
   Thoughts?
  
   Gwen
  
   On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps jay.kr...@gmail.com
 wrote:
I think I am still confused. In addition to the
 UpdateMetadataRequest
   don't
we have to change the MetadataResponse so that it's possible for
  clients
   to
discover the new ports? Or is that a second phase? I was imagining
 it
worked by basically allowing the brokers to advertise multiple
 ports,
  one
per security type, and then in the client you configure a protocol
  which
will implicitly choose the port from the options returned in
 metadata
  to
you...
   
Likewise in the ConsumerMetadataResponse we are currently giving
 back
   full
broker information. I think we would have two options here: either
  change
the broker information included in that response to match the
metadataresponse or else remove the broker information entirely and
  just
return the node id (since in order to use that request you would
  already
have to have the cluster metadata). The second option may be cleaner
   since
it means we won't have to continue evolving those two in lockstep...
   
-Jay
   
On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira 
 gshap...@cloudera.com
   wrote:
   
Good point :)
   
I added the specifics of the new  UpdateMetadataRequest, which is
 the
only protocol bump in this change.
   
Highlighted the broker and producer/consumer configuration changes,
added some example values and added the new zookeeper json.
   
Hope this makes things clearer.
   
On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps jay.kr...@gmail.com
  wrote:
 Hey Gwen,

 Could we get the actual changes in that KIP? I.e. changes to
  metadata
 request, changes to UpdateMetadataRequest, new configs and what
  will
their
 valid values be, etc. This kind of says that those things will
  change
   but
 doesn't say what they will change to...

 -Jay

 On Mon, Jan 19, 2015 at 

Re: [DISCUSS] KIPs

[Jun Rao]

Reviewed the latest patch in KAFKA-1809 :).

Thanks,

Jun

On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira gshap...@cloudera.com
wrote:

 Thanks for validating our ideas. Updated the KIP with the workflow.

 Now if you can nudge Jun to review the latest patch... ;)


 On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps j...@confluent.io wrote:
  Oh yeah I think that is better, I hadn't thought of that approach! Any
 way
  you could describe the usage in the KIP, just for completeness?
 
  -Jay
 
  On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira gshap...@cloudera.com
  wrote:
 
  I think what you described was the original design, so no wonder you
  are confused :)
 
  Following suggestions from Jun, I changed it a bit. The current model
 is:
 
  - Clients (producers and consumers) need to know about the broker
  ports in advance. They don't need to know about all brokers, but they
  need to know at least one host:port pair that speaks the protocol they
  want to use. The change is that all host:port pairs in broker.list
  must be of the same protocol and match the security.protocol
  configuration parameter.
 
  - Client uses security.protocol configuration parameter to open a
  connection to one of the brokers and sends the good old
  MetadataRequest. The broker knows which port it got the connection on,
  therefore it knows which security protocol is expected (it needs to
  use the same protocol to accept the connection and respond), and
  therefore it can send a response that contains only the host:port
  pairs that are relevant to that protocol.
 
  - From the client side the MetadataResponse did not change - it
  contains a list of brokerId,host,port that the client can connect to.
  The fact that all those broker endpoints were chosen out of a larger
  collection to match the right protocol is irrelevant for the client.
 
  I really like the new design since it preserves a lot of the same
  configurations and APIs.
 
  Thoughts?
 
  Gwen
 
  On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps jay.kr...@gmail.com wrote:
   I think I am still confused. In addition to the UpdateMetadataRequest
  don't
   we have to change the MetadataResponse so that it's possible for
 clients
  to
   discover the new ports? Or is that a second phase? I was imagining it
   worked by basically allowing the brokers to advertise multiple ports,
 one
   per security type, and then in the client you configure a protocol
 which
   will implicitly choose the port from the options returned in metadata
 to
   you...
  
   Likewise in the ConsumerMetadataResponse we are currently giving back
  full
   broker information. I think we would have two options here: either
 change
   the broker information included in that response to match the
   metadataresponse or else remove the broker information entirely and
 just
   return the node id (since in order to use that request you would
 already
   have to have the cluster metadata). The second option may be cleaner
  since
   it means we won't have to continue evolving those two in lockstep...
  
   -Jay
  
   On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira gshap...@cloudera.com
  wrote:
  
   Good point :)
  
   I added the specifics of the new  UpdateMetadataRequest, which is the
   only protocol bump in this change.
  
   Highlighted the broker and producer/consumer configuration changes,
   added some example values and added the new zookeeper json.
  
   Hope this makes things clearer.
  
   On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps jay.kr...@gmail.com
 wrote:
Hey Gwen,
   
Could we get the actual changes in that KIP? I.e. changes to
 metadata
request, changes to UpdateMetadataRequest, new configs and what
 will
   their
valid values be, etc. This kind of says that those things will
 change
  but
doesn't say what they will change to...
   
-Jay
   
On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira 
 gshap...@cloudera.com
   wrote:
   
I created a KIP for the multi-port broker change.
   
   
  
 
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
   
I'm not re-opening the discussion, since it was agreed on over a
  month
ago and implementation is close to complete (I hope!). Lets
 consider
this voted and accepted?
   
Gwen
   
On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps jay.kr...@gmail.com
   wrote:
 Great! Sounds like everyone is on the same page

- I created a template page to make things easier. If you do
Tools-Copy
on this page you can just fill in the italic portions with
 your
details.
- I retrofitted KIP-1 to match this formatting
- I added the metadata section people asked for (a link to
 the
discussion, the JIRA, and the current status). Let's make
 sure
  we
remember
to update the current status as things are figured out.
- Let's keep the discussion on the mailing list rather than
 on
  the

Re: [DISCUSS] KIPs

[Jay Kreps]

Oh yeah I think that is better, I hadn't thought of that approach! Any way
you could describe the usage in the KIP, just for completeness?

-Jay

On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira gshap...@cloudera.com
wrote:

 I think what you described was the original design, so no wonder you
 are confused :)

 Following suggestions from Jun, I changed it a bit. The current model is:

 - Clients (producers and consumers) need to know about the broker
 ports in advance. They don't need to know about all brokers, but they
 need to know at least one host:port pair that speaks the protocol they
 want to use. The change is that all host:port pairs in broker.list
 must be of the same protocol and match the security.protocol
 configuration parameter.

 - Client uses security.protocol configuration parameter to open a
 connection to one of the brokers and sends the good old
 MetadataRequest. The broker knows which port it got the connection on,
 therefore it knows which security protocol is expected (it needs to
 use the same protocol to accept the connection and respond), and
 therefore it can send a response that contains only the host:port
 pairs that are relevant to that protocol.

 - From the client side the MetadataResponse did not change - it
 contains a list of brokerId,host,port that the client can connect to.
 The fact that all those broker endpoints were chosen out of a larger
 collection to match the right protocol is irrelevant for the client.

 I really like the new design since it preserves a lot of the same
 configurations and APIs.

 Thoughts?

 Gwen

 On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps jay.kr...@gmail.com wrote:
  I think I am still confused. In addition to the UpdateMetadataRequest
 don't
  we have to change the MetadataResponse so that it's possible for clients
 to
  discover the new ports? Or is that a second phase? I was imagining it
  worked by basically allowing the brokers to advertise multiple ports, one
  per security type, and then in the client you configure a protocol which
  will implicitly choose the port from the options returned in metadata to
  you...
 
  Likewise in the ConsumerMetadataResponse we are currently giving back
 full
  broker information. I think we would have two options here: either change
  the broker information included in that response to match the
  metadataresponse or else remove the broker information entirely and just
  return the node id (since in order to use that request you would already
  have to have the cluster metadata). The second option may be cleaner
 since
  it means we won't have to continue evolving those two in lockstep...
 
  -Jay
 
  On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira gshap...@cloudera.com
 wrote:
 
  Good point :)
 
  I added the specifics of the new  UpdateMetadataRequest, which is the
  only protocol bump in this change.
 
  Highlighted the broker and producer/consumer configuration changes,
  added some example values and added the new zookeeper json.
 
  Hope this makes things clearer.
 
  On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps jay.kr...@gmail.com wrote:
   Hey Gwen,
  
   Could we get the actual changes in that KIP? I.e. changes to metadata
   request, changes to UpdateMetadataRequest, new configs and what will
  their
   valid values be, etc. This kind of says that those things will change
 but
   doesn't say what they will change to...
  
   -Jay
  
   On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira gshap...@cloudera.com
  wrote:
  
   I created a KIP for the multi-port broker change.
  
  
 
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
  
   I'm not re-opening the discussion, since it was agreed on over a
 month
   ago and implementation is close to complete (I hope!). Lets consider
   this voted and accepted?
  
   Gwen
  
   On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps jay.kr...@gmail.com
  wrote:
Great! Sounds like everyone is on the same page
   
   - I created a template page to make things easier. If you do
   Tools-Copy
   on this page you can just fill in the italic portions with your
   details.
   - I retrofitted KIP-1 to match this formatting
   - I added the metadata section people asked for (a link to the
   discussion, the JIRA, and the current status). Let's make sure
 we
   remember
   to update the current status as things are figured out.
   - Let's keep the discussion on the mailing list rather than on
 the
   wiki
   pages. It makes sense to do one or the other so all the comments
  are
   in one
   place and I think prior experience is that the wiki comments are
  the
   worse
   way.
   
I think it would be great do KIPs for some of the in-flight items
  folks
mentioned.
   
-Jay
   
On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira 
 gshap...@cloudera.com
   wrote:
   
+1
   
Will be happy to provide a KIP for the multiple-listeners patch.
   
Gwen

Re: [DISCUSS] KIPs

[Gwen Shapira]

I think what you described was the original design, so no wonder you
are confused :)

Following suggestions from Jun, I changed it a bit. The current model is:

- Clients (producers and consumers) need to know about the broker
ports in advance. They don't need to know about all brokers, but they
need to know at least one host:port pair that speaks the protocol they
want to use. The change is that all host:port pairs in broker.list
must be of the same protocol and match the security.protocol
configuration parameter.

- Client uses security.protocol configuration parameter to open a
connection to one of the brokers and sends the good old
MetadataRequest. The broker knows which port it got the connection on,
therefore it knows which security protocol is expected (it needs to
use the same protocol to accept the connection and respond), and
therefore it can send a response that contains only the host:port
pairs that are relevant to that protocol.

- From the client side the MetadataResponse did not change - it
contains a list of brokerId,host,port that the client can connect to.
The fact that all those broker endpoints were chosen out of a larger
collection to match the right protocol is irrelevant for the client.

I really like the new design since it preserves a lot of the same
configurations and APIs.

Thoughts?

Gwen

On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps jay.kr...@gmail.com wrote:
 I think I am still confused. In addition to the UpdateMetadataRequest don't
 we have to change the MetadataResponse so that it's possible for clients to
 discover the new ports? Or is that a second phase? I was imagining it
 worked by basically allowing the brokers to advertise multiple ports, one
 per security type, and then in the client you configure a protocol which
 will implicitly choose the port from the options returned in metadata to
 you...

 Likewise in the ConsumerMetadataResponse we are currently giving back full
 broker information. I think we would have two options here: either change
 the broker information included in that response to match the
 metadataresponse or else remove the broker information entirely and just
 return the node id (since in order to use that request you would already
 have to have the cluster metadata). The second option may be cleaner since
 it means we won't have to continue evolving those two in lockstep...

 -Jay

 On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira gshap...@cloudera.com wrote:

 Good point :)

 I added the specifics of the new  UpdateMetadataRequest, which is the
 only protocol bump in this change.

 Highlighted the broker and producer/consumer configuration changes,
 added some example values and added the new zookeeper json.

 Hope this makes things clearer.

 On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps jay.kr...@gmail.com wrote:
  Hey Gwen,
 
  Could we get the actual changes in that KIP? I.e. changes to metadata
  request, changes to UpdateMetadataRequest, new configs and what will
 their
  valid values be, etc. This kind of says that those things will change but
  doesn't say what they will change to...
 
  -Jay
 
  On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira gshap...@cloudera.com
 wrote:
 
  I created a KIP for the multi-port broker change.
 
 
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
 
  I'm not re-opening the discussion, since it was agreed on over a month
  ago and implementation is close to complete (I hope!). Lets consider
  this voted and accepted?
 
  Gwen
 
  On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps jay.kr...@gmail.com
 wrote:
   Great! Sounds like everyone is on the same page
  
  - I created a template page to make things easier. If you do
  Tools-Copy
  on this page you can just fill in the italic portions with your
  details.
  - I retrofitted KIP-1 to match this formatting
  - I added the metadata section people asked for (a link to the
  discussion, the JIRA, and the current status). Let's make sure we
  remember
  to update the current status as things are figured out.
  - Let's keep the discussion on the mailing list rather than on the
  wiki
  pages. It makes sense to do one or the other so all the comments
 are
  in one
  place and I think prior experience is that the wiki comments are
 the
  worse
  way.
  
   I think it would be great do KIPs for some of the in-flight items
 folks
   mentioned.
  
   -Jay
  
   On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com
  wrote:
  
   +1
  
   Will be happy to provide a KIP for the multiple-listeners patch.
  
   Gwen
  
   On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly
  wrote:
+1 to everything we have been saying and where this (has settled
  to)/(is
settling to).
   
I am sure other folks have some more feedback and think we should
 try
  to
keep this discussion going if need be. I am also a firm believer of
  form

Re: [DISCUSS] KIPs

[Gwen Shapira]

Thanks for validating our ideas. Updated the KIP with the workflow.

Now if you can nudge Jun to review the latest patch... ;)


On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps j...@confluent.io wrote:
 Oh yeah I think that is better, I hadn't thought of that approach! Any way
 you could describe the usage in the KIP, just for completeness?

 -Jay

 On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira gshap...@cloudera.com
 wrote:

 I think what you described was the original design, so no wonder you
 are confused :)

 Following suggestions from Jun, I changed it a bit. The current model is:

 - Clients (producers and consumers) need to know about the broker
 ports in advance. They don't need to know about all brokers, but they
 need to know at least one host:port pair that speaks the protocol they
 want to use. The change is that all host:port pairs in broker.list
 must be of the same protocol and match the security.protocol
 configuration parameter.

 - Client uses security.protocol configuration parameter to open a
 connection to one of the brokers and sends the good old
 MetadataRequest. The broker knows which port it got the connection on,
 therefore it knows which security protocol is expected (it needs to
 use the same protocol to accept the connection and respond), and
 therefore it can send a response that contains only the host:port
 pairs that are relevant to that protocol.

 - From the client side the MetadataResponse did not change - it
 contains a list of brokerId,host,port that the client can connect to.
 The fact that all those broker endpoints were chosen out of a larger
 collection to match the right protocol is irrelevant for the client.

 I really like the new design since it preserves a lot of the same
 configurations and APIs.

 Thoughts?

 Gwen

 On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps jay.kr...@gmail.com wrote:
  I think I am still confused. In addition to the UpdateMetadataRequest
 don't
  we have to change the MetadataResponse so that it's possible for clients
 to
  discover the new ports? Or is that a second phase? I was imagining it
  worked by basically allowing the brokers to advertise multiple ports, one
  per security type, and then in the client you configure a protocol which
  will implicitly choose the port from the options returned in metadata to
  you...
 
  Likewise in the ConsumerMetadataResponse we are currently giving back
 full
  broker information. I think we would have two options here: either change
  the broker information included in that response to match the
  metadataresponse or else remove the broker information entirely and just
  return the node id (since in order to use that request you would already
  have to have the cluster metadata). The second option may be cleaner
 since
  it means we won't have to continue evolving those two in lockstep...
 
  -Jay
 
  On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira gshap...@cloudera.com
 wrote:
 
  Good point :)
 
  I added the specifics of the new  UpdateMetadataRequest, which is the
  only protocol bump in this change.
 
  Highlighted the broker and producer/consumer configuration changes,
  added some example values and added the new zookeeper json.
 
  Hope this makes things clearer.
 
  On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps jay.kr...@gmail.com wrote:
   Hey Gwen,
  
   Could we get the actual changes in that KIP? I.e. changes to metadata
   request, changes to UpdateMetadataRequest, new configs and what will
  their
   valid values be, etc. This kind of says that those things will change
 but
   doesn't say what they will change to...
  
   -Jay
  
   On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira gshap...@cloudera.com
  wrote:
  
   I created a KIP for the multi-port broker change.
  
  
 
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
  
   I'm not re-opening the discussion, since it was agreed on over a
 month
   ago and implementation is close to complete (I hope!). Lets consider
   this voted and accepted?
  
   Gwen
  
   On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps jay.kr...@gmail.com
  wrote:
Great! Sounds like everyone is on the same page
   
   - I created a template page to make things easier. If you do
   Tools-Copy
   on this page you can just fill in the italic portions with your
   details.
   - I retrofitted KIP-1 to match this formatting
   - I added the metadata section people asked for (a link to the
   discussion, the JIRA, and the current status). Let's make sure
 we
   remember
   to update the current status as things are figured out.
   - Let's keep the discussion on the mailing list rather than on
 the
   wiki
   pages. It makes sense to do one or the other so all the comments
  are
   in one
   place and I think prior experience is that the wiki comments are
  the
   worse
   way.
   
I think it would be great do KIPs for some of the in-flight items
  folks

Re: [DISCUSS] KIPs

[Jay Kreps]

Hey Gwen,

Could we get the actual changes in that KIP? I.e. changes to metadata
request, changes to UpdateMetadataRequest, new configs and what will their
valid values be, etc. This kind of says that those things will change but
doesn't say what they will change to...

-Jay

On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira gshap...@cloudera.com wrote:

 I created a KIP for the multi-port broker change.

 https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs

 I'm not re-opening the discussion, since it was agreed on over a month
 ago and implementation is close to complete (I hope!). Lets consider
 this voted and accepted?

 Gwen

 On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps jay.kr...@gmail.com wrote:
  Great! Sounds like everyone is on the same page
 
 - I created a template page to make things easier. If you do
 Tools-Copy
 on this page you can just fill in the italic portions with your
 details.
 - I retrofitted KIP-1 to match this formatting
 - I added the metadata section people asked for (a link to the
 discussion, the JIRA, and the current status). Let's make sure we
 remember
 to update the current status as things are figured out.
 - Let's keep the discussion on the mailing list rather than on the
 wiki
 pages. It makes sense to do one or the other so all the comments are
 in one
 place and I think prior experience is that the wiki comments are the
 worse
 way.
 
  I think it would be great do KIPs for some of the in-flight items folks
  mentioned.
 
  -Jay
 
  On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com
 wrote:
 
  +1
 
  Will be happy to provide a KIP for the multiple-listeners patch.
 
  Gwen
 
  On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly
 wrote:
   +1 to everything we have been saying and where this (has settled
 to)/(is
   settling to).
  
   I am sure other folks have some more feedback and think we should try
 to
   keep this discussion going if need be. I am also a firm believer of
 form
   following function so kicking the tires some to flesh out the details
 of
   this and have some organic growth with the process will be healthy and
   beneficial to the community.
  
   For my part, what I will do is open a few KIP based on some of the
 work I
   have been involved with for 0.8.3. Off the top of my head this would
   include 1) changes to re-assignment of partitions 2) kafka cli 3)
 global
   configs 4) security white list black list by ip 5) SSL 6) We probably
  will
   have lots of Security related KIPs and should treat them all
 individually
   when the time is appropriate 7) Kafka on Mesos.
  
   If someone else wants to jump in to start getting some of the security
  KIP
   that we are going to have in 0.8.3 I think that would be great (e.g.
   Multiple Listeners for Kafka Brokers). There are also a few other
  tickets I
   can think of that are important to have in the code in 0.8.3 that
 should
   have KIP also that I haven't really been involved in. I will take a
 few
   minutes and go through JIRA (one I can think of like auto assign id
 that
  is
   already committed I think) and ask for a KIP if appropriate or if I
 feel
   that I can write it up (both from a time and understanding
 perspective)
  do
   so.
  
   long story short, I encourage folks to start moving ahead with the KIP
  for
   0.8.3 as how we operate. any objections?
  
   On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com
  wrote:
  
   +1 on the idea, and we could mutually link the KIP wiki page with the
  the
   created JIRA ticket (i.e. include the JIRA number on the page and the
  KIP
   url on the ticket description).
  
   Regarding the KIP process, probably we do not need two phase
  communication
   of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
  clear
   while people discuss about that.
  
   About who should trigger the process, I think the only involved
 people
   would be 1) when the patch is submitted / or even the ticket is
 created,
   the assignee could choose to start the KIP process if she thought it
 is
   necessary; 2) the reviewer of the patch can also suggest starting KIP
   discussions.
  
   On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
 gshap...@cloudera.com
   wrote:
  
+1 to Ewen's suggestions: Deprecation, status and version.
   
Perhaps add the JIRA where the KIP was implemented to the metadata.
This will help tie things together.
   
On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
e...@confluent.io wrote:
 I think adding a section about deprecation would be helpful. A
 good
 fraction of the time I would expect the goal of a KIP is to fix
 or
replace
 older functionality that needs continued support for
 compatibility,
  but
 should eventually be phased out. This helps Kafka devs understand
  how
long
 they'll end up supporting multiple 

Re: [DISCUSS] KIPs

[Gwen Shapira]

Good point :)

I added the specifics of the new  UpdateMetadataRequest, which is the
only protocol bump in this change.

Highlighted the broker and producer/consumer configuration changes,
added some example values and added the new zookeeper json.

Hope this makes things clearer.

On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps jay.kr...@gmail.com wrote:
 Hey Gwen,

 Could we get the actual changes in that KIP? I.e. changes to metadata
 request, changes to UpdateMetadataRequest, new configs and what will their
 valid values be, etc. This kind of says that those things will change but
 doesn't say what they will change to...

 -Jay

 On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira gshap...@cloudera.com wrote:

 I created a KIP for the multi-port broker change.

 https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs

 I'm not re-opening the discussion, since it was agreed on over a month
 ago and implementation is close to complete (I hope!). Lets consider
 this voted and accepted?

 Gwen

 On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps jay.kr...@gmail.com wrote:
  Great! Sounds like everyone is on the same page
 
 - I created a template page to make things easier. If you do
 Tools-Copy
 on this page you can just fill in the italic portions with your
 details.
 - I retrofitted KIP-1 to match this formatting
 - I added the metadata section people asked for (a link to the
 discussion, the JIRA, and the current status). Let's make sure we
 remember
 to update the current status as things are figured out.
 - Let's keep the discussion on the mailing list rather than on the
 wiki
 pages. It makes sense to do one or the other so all the comments are
 in one
 place and I think prior experience is that the wiki comments are the
 worse
 way.
 
  I think it would be great do KIPs for some of the in-flight items folks
  mentioned.
 
  -Jay
 
  On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com
 wrote:
 
  +1
 
  Will be happy to provide a KIP for the multiple-listeners patch.
 
  Gwen
 
  On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly
 wrote:
   +1 to everything we have been saying and where this (has settled
 to)/(is
   settling to).
  
   I am sure other folks have some more feedback and think we should try
 to
   keep this discussion going if need be. I am also a firm believer of
 form
   following function so kicking the tires some to flesh out the details
 of
   this and have some organic growth with the process will be healthy and
   beneficial to the community.
  
   For my part, what I will do is open a few KIP based on some of the
 work I
   have been involved with for 0.8.3. Off the top of my head this would
   include 1) changes to re-assignment of partitions 2) kafka cli 3)
 global
   configs 4) security white list black list by ip 5) SSL 6) We probably
  will
   have lots of Security related KIPs and should treat them all
 individually
   when the time is appropriate 7) Kafka on Mesos.
  
   If someone else wants to jump in to start getting some of the security
  KIP
   that we are going to have in 0.8.3 I think that would be great (e.g.
   Multiple Listeners for Kafka Brokers). There are also a few other
  tickets I
   can think of that are important to have in the code in 0.8.3 that
 should
   have KIP also that I haven't really been involved in. I will take a
 few
   minutes and go through JIRA (one I can think of like auto assign id
 that
  is
   already committed I think) and ask for a KIP if appropriate or if I
 feel
   that I can write it up (both from a time and understanding
 perspective)
  do
   so.
  
   long story short, I encourage folks to start moving ahead with the KIP
  for
   0.8.3 as how we operate. any objections?
  
   On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com
  wrote:
  
   +1 on the idea, and we could mutually link the KIP wiki page with the
  the
   created JIRA ticket (i.e. include the JIRA number on the page and the
  KIP
   url on the ticket description).
  
   Regarding the KIP process, probably we do not need two phase
  communication
   of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
  clear
   while people discuss about that.
  
   About who should trigger the process, I think the only involved
 people
   would be 1) when the patch is submitted / or even the ticket is
 created,
   the assignee could choose to start the KIP process if she thought it
 is
   necessary; 2) the reviewer of the patch can also suggest starting KIP
   discussions.
  
   On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira 
 gshap...@cloudera.com
   wrote:
  
+1 to Ewen's suggestions: Deprecation, status and version.
   
Perhaps add the JIRA where the KIP was implemented to the metadata.
This will help tie things together.
   
On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
e...@confluent.io wrote:
 I 

Re: [DISCUSS] KIPs

[Gwen Shapira]

I created a KIP for the multi-port broker change.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs

I'm not re-opening the discussion, since it was agreed on over a month
ago and implementation is close to complete (I hope!). Lets consider
this voted and accepted?

Gwen

On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps jay.kr...@gmail.com wrote:
 Great! Sounds like everyone is on the same page

- I created a template page to make things easier. If you do Tools-Copy
on this page you can just fill in the italic portions with your details.
- I retrofitted KIP-1 to match this formatting
- I added the metadata section people asked for (a link to the
discussion, the JIRA, and the current status). Let's make sure we remember
to update the current status as things are figured out.
- Let's keep the discussion on the mailing list rather than on the wiki
pages. It makes sense to do one or the other so all the comments are in one
place and I think prior experience is that the wiki comments are the worse
way.

 I think it would be great do KIPs for some of the in-flight items folks
 mentioned.

 -Jay

 On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com wrote:

 +1

 Will be happy to provide a KIP for the multiple-listeners patch.

 Gwen

 On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly wrote:
  +1 to everything we have been saying and where this (has settled to)/(is
  settling to).
 
  I am sure other folks have some more feedback and think we should try to
  keep this discussion going if need be. I am also a firm believer of form
  following function so kicking the tires some to flesh out the details of
  this and have some organic growth with the process will be healthy and
  beneficial to the community.
 
  For my part, what I will do is open a few KIP based on some of the work I
  have been involved with for 0.8.3. Off the top of my head this would
  include 1) changes to re-assignment of partitions 2) kafka cli 3) global
  configs 4) security white list black list by ip 5) SSL 6) We probably
 will
  have lots of Security related KIPs and should treat them all individually
  when the time is appropriate 7) Kafka on Mesos.
 
  If someone else wants to jump in to start getting some of the security
 KIP
  that we are going to have in 0.8.3 I think that would be great (e.g.
  Multiple Listeners for Kafka Brokers). There are also a few other
 tickets I
  can think of that are important to have in the code in 0.8.3 that should
  have KIP also that I haven't really been involved in. I will take a few
  minutes and go through JIRA (one I can think of like auto assign id that
 is
  already committed I think) and ask for a KIP if appropriate or if I feel
  that I can write it up (both from a time and understanding perspective)
 do
  so.
 
  long story short, I encourage folks to start moving ahead with the KIP
 for
  0.8.3 as how we operate. any objections?
 
  On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com
 wrote:
 
  +1 on the idea, and we could mutually link the KIP wiki page with the
 the
  created JIRA ticket (i.e. include the JIRA number on the page and the
 KIP
  url on the ticket description).
 
  Regarding the KIP process, probably we do not need two phase
 communication
  of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
 clear
  while people discuss about that.
 
  About who should trigger the process, I think the only involved people
  would be 1) when the patch is submitted / or even the ticket is created,
  the assignee could choose to start the KIP process if she thought it is
  necessary; 2) the reviewer of the patch can also suggest starting KIP
  discussions.
 
  On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira gshap...@cloudera.com
  wrote:
 
   +1 to Ewen's suggestions: Deprecation, status and version.
  
   Perhaps add the JIRA where the KIP was implemented to the metadata.
   This will help tie things together.
  
   On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
   e...@confluent.io wrote:
I think adding a section about deprecation would be helpful. A good
fraction of the time I would expect the goal of a KIP is to fix or
   replace
older functionality that needs continued support for compatibility,
 but
should eventually be phased out. This helps Kafka devs understand
 how
   long
they'll end up supporting multiple versions of features and helps
 users
understand when they're going to have to make updates to their
   applications.
   
Less important but useful -- having a bit of standard metadata like
  PEPs
do. Two I think are important are status (if someone lands on the
 KIP
   page,
can they tell whether this KIP was ever completed?) and/or the
 version
   the
KIP was first released in.
   
   
   
On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy jjkosh...@gmail.com
  wrote:
   
 

Re: [DISCUSS] KIPs

[Jay Kreps]

Great! Sounds like everyone is on the same page

   - I created a template page to make things easier. If you do Tools-Copy
   on this page you can just fill in the italic portions with your details.
   - I retrofitted KIP-1 to match this formatting
   - I added the metadata section people asked for (a link to the
   discussion, the JIRA, and the current status). Let's make sure we remember
   to update the current status as things are figured out.
   - Let's keep the discussion on the mailing list rather than on the wiki
   pages. It makes sense to do one or the other so all the comments are in one
   place and I think prior experience is that the wiki comments are the worse
   way.

I think it would be great do KIPs for some of the in-flight items folks
mentioned.

-Jay

On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira gshap...@cloudera.com wrote:

 +1

 Will be happy to provide a KIP for the multiple-listeners patch.

 Gwen

 On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein joe.st...@stealth.ly wrote:
  +1 to everything we have been saying and where this (has settled to)/(is
  settling to).
 
  I am sure other folks have some more feedback and think we should try to
  keep this discussion going if need be. I am also a firm believer of form
  following function so kicking the tires some to flesh out the details of
  this and have some organic growth with the process will be healthy and
  beneficial to the community.
 
  For my part, what I will do is open a few KIP based on some of the work I
  have been involved with for 0.8.3. Off the top of my head this would
  include 1) changes to re-assignment of partitions 2) kafka cli 3) global
  configs 4) security white list black list by ip 5) SSL 6) We probably
 will
  have lots of Security related KIPs and should treat them all individually
  when the time is appropriate 7) Kafka on Mesos.
 
  If someone else wants to jump in to start getting some of the security
 KIP
  that we are going to have in 0.8.3 I think that would be great (e.g.
  Multiple Listeners for Kafka Brokers). There are also a few other
 tickets I
  can think of that are important to have in the code in 0.8.3 that should
  have KIP also that I haven't really been involved in. I will take a few
  minutes and go through JIRA (one I can think of like auto assign id that
 is
  already committed I think) and ask for a KIP if appropriate or if I feel
  that I can write it up (both from a time and understanding perspective)
 do
  so.
 
  long story short, I encourage folks to start moving ahead with the KIP
 for
  0.8.3 as how we operate. any objections?
 
  On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com
 wrote:
 
  +1 on the idea, and we could mutually link the KIP wiki page with the
 the
  created JIRA ticket (i.e. include the JIRA number on the page and the
 KIP
  url on the ticket description).
 
  Regarding the KIP process, probably we do not need two phase
 communication
  of a [DISCUSS] followed by [VOTE], as Jay said the voting should be
 clear
  while people discuss about that.
 
  About who should trigger the process, I think the only involved people
  would be 1) when the patch is submitted / or even the ticket is created,
  the assignee could choose to start the KIP process if she thought it is
  necessary; 2) the reviewer of the patch can also suggest starting KIP
  discussions.
 
  On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira gshap...@cloudera.com
  wrote:
 
   +1 to Ewen's suggestions: Deprecation, status and version.
  
   Perhaps add the JIRA where the KIP was implemented to the metadata.
   This will help tie things together.
  
   On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
   e...@confluent.io wrote:
I think adding a section about deprecation would be helpful. A good
fraction of the time I would expect the goal of a KIP is to fix or
   replace
older functionality that needs continued support for compatibility,
 but
should eventually be phased out. This helps Kafka devs understand
 how
   long
they'll end up supporting multiple versions of features and helps
 users
understand when they're going to have to make updates to their
   applications.
   
Less important but useful -- having a bit of standard metadata like
  PEPs
do. Two I think are important are status (if someone lands on the
 KIP
   page,
can they tell whether this KIP was ever completed?) and/or the
 version
   the
KIP was first released in.
   
   
   
On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy jjkosh...@gmail.com
  wrote:
   
I'm definitely +1 on the KIP concept. As Joe mentioned, we are
 already
doing this in one form or the other. However, IMO it is fairly ad
 hoc
- i.e., a combination of DISCUSS threads, jira comments, RB and
 code
comments, wikis and html documentation. In the past I have had to
 dig
into a bunch of these to try and figure out why something was
implemented a certain way. I think KIPs can help a lot here 

Re: [DISCUSS] KIPs

[Joe Stein]

+1 to everything we have been saying and where this (has settled to)/(is
settling to).

I am sure other folks have some more feedback and think we should try to
keep this discussion going if need be. I am also a firm believer of form
following function so kicking the tires some to flesh out the details of
this and have some organic growth with the process will be healthy and
beneficial to the community.

For my part, what I will do is open a few KIP based on some of the work I
have been involved with for 0.8.3. Off the top of my head this would
include 1) changes to re-assignment of partitions 2) kafka cli 3) global
configs 4) security white list black list by ip 5) SSL 6) We probably will
have lots of Security related KIPs and should treat them all individually
when the time is appropriate 7) Kafka on Mesos.

If someone else wants to jump in to start getting some of the security KIP
that we are going to have in 0.8.3 I think that would be great (e.g.
Multiple Listeners for Kafka Brokers). There are also a few other tickets I
can think of that are important to have in the code in 0.8.3 that should
have KIP also that I haven't really been involved in. I will take a few
minutes and go through JIRA (one I can think of like auto assign id that is
already committed I think) and ask for a KIP if appropriate or if I feel
that I can write it up (both from a time and understanding perspective) do
so.

long story short, I encourage folks to start moving ahead with the KIP for
0.8.3 as how we operate. any objections?

On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang wangg...@gmail.com wrote:

 +1 on the idea, and we could mutually link the KIP wiki page with the the
 created JIRA ticket (i.e. include the JIRA number on the page and the KIP
 url on the ticket description).

 Regarding the KIP process, probably we do not need two phase communication
 of a [DISCUSS] followed by [VOTE], as Jay said the voting should be clear
 while people discuss about that.

 About who should trigger the process, I think the only involved people
 would be 1) when the patch is submitted / or even the ticket is created,
 the assignee could choose to start the KIP process if she thought it is
 necessary; 2) the reviewer of the patch can also suggest starting KIP
 discussions.

 On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira gshap...@cloudera.com
 wrote:

  +1 to Ewen's suggestions: Deprecation, status and version.
 
  Perhaps add the JIRA where the KIP was implemented to the metadata.
  This will help tie things together.
 
  On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
  e...@confluent.io wrote:
   I think adding a section about deprecation would be helpful. A good
   fraction of the time I would expect the goal of a KIP is to fix or
  replace
   older functionality that needs continued support for compatibility, but
   should eventually be phased out. This helps Kafka devs understand how
  long
   they'll end up supporting multiple versions of features and helps users
   understand when they're going to have to make updates to their
  applications.
  
   Less important but useful -- having a bit of standard metadata like
 PEPs
   do. Two I think are important are status (if someone lands on the KIP
  page,
   can they tell whether this KIP was ever completed?) and/or the version
  the
   KIP was first released in.
  
  
  
   On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy jjkosh...@gmail.com
 wrote:
  
   I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
   doing this in one form or the other. However, IMO it is fairly ad hoc
   - i.e., a combination of DISCUSS threads, jira comments, RB and code
   comments, wikis and html documentation. In the past I have had to dig
   into a bunch of these to try and figure out why something was
   implemented a certain way. I think KIPs can help a lot here first by
   providing guidelines on what to think about (compatibility, new APIs,
   etc.) when working through a major feature; and second by becoming a
   crisp source of truth documentation for new releases.  E.g., for
   feature X: see relevant KIPs: a, b, c, etc.
  
   On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
Hey Joe,
   
Yeah I guess the question is what is the definition of major? I
 agree
  we
definitely don't want to generate a bunch of paperwork. We have
 enough
problems just getting all the contributions reviewed and checked in
  in a
timely fashion...
   
So obviously bug fixes would not apply here.
   
I think it is also pretty clear that big features should get
 reviewed
  and
discussed. To pick on myself, for example, the log compaction work
 was
   done
without enough public discussion about how it worked and why
 (imho). I
hope/claim that enough rigour in thinking about use-cases and
  operations
and so on was done that it turned out well, but the discussion was
  just
between a few people with no real public output. This kind of
 feature
  

Re: [DISCUSS] KIPs

[Joel Koshy]

I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
doing this in one form or the other. However, IMO it is fairly ad hoc
- i.e., a combination of DISCUSS threads, jira comments, RB and code
comments, wikis and html documentation. In the past I have had to dig
into a bunch of these to try and figure out why something was
implemented a certain way. I think KIPs can help a lot here first by
providing guidelines on what to think about (compatibility, new APIs,
etc.) when working through a major feature; and second by becoming a
crisp source of truth documentation for new releases.  E.g., for
feature X: see relevant KIPs: a, b, c, etc.

On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
 Hey Joe,
 
 Yeah I guess the question is what is the definition of major? I agree we
 definitely don't want to generate a bunch of paperwork. We have enough
 problems just getting all the contributions reviewed and checked in in a
 timely fashion...
 
 So obviously bug fixes would not apply here.
 
 I think it is also pretty clear that big features should get reviewed and
 discussed. To pick on myself, for example, the log compaction work was done
 without enough public discussion about how it worked and why (imho). I
 hope/claim that enough rigour in thinking about use-cases and operations
 and so on was done that it turned out well, but the discussion was just
 between a few people with no real public output. This kind of feature is
 clearly a big change and something we should discuss.
 
 If we limit ourselves to just the public contracts the KIP introduces the
 discussion would just be on the new configs and monitoring without really a
 discussion of the design and how it works which is obviously closely
 related.
 
 I don't think this should be more work because in practice we are making
 wiki pages for any big thing anyway. So this would just be a consistent way
 of numbering and structuring these pages. This would also give a clear call
 to action: hey kafka people, come read my wiki and think this through.
 
 I actually thinking the voting aspect is less important. I think it is
 generally clear when there is agreement on something and not. So from my
 point of view we could actually just eliminate that part if that is too
 formal, it just seemed like a good way to formally adopt something.
 
 To address some of your comments from the wiki:
 
 1. This doesn't inhibit someone coming along and putting up a patch. It is
 just that when they do if it is a big thing introducing new functionality
 we would ask for a little discussion on the basic feature/contracts prior
 to code review.
 
 2. We definitely definitely don't want people generating a lot of these
 things every time they have an idea that they aren't going to implement. So
 this is only applicable to things you absolutely will check in code for. We
 also don't want to be making proposals before things are thought through,
 which often requires writing the code. So I think the right time for a KIP
 is when you are far enough along that you know the issues and tradeoffs but
 not so far along that you are going to be totally opposed to any change.
 Sometimes that is prior to writing any code and sometimes not until you are
 practically done.
 
 The key problem I see this fixing is that there is enough new development
 happening that it is pretty hard for everyone to review every line of every
 iteration of every patch. But all of us should be fully aware of new
 features, the ramifications, the new public interfaces, etc. If we aren't
 aware of that we can't really build a holistic system that is beautiful and
 consistent across. So the idea is that if you fully review the KIPs you can
 be sure that even if you don't know every new line of code, you know the
 major changes coming in.
 
 -Jay
 
 
 
 On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein joe.st...@stealth.ly wrote:
 
  Thanks Jay for kicking this off! I think the confluence page you wrote up
  is a great start.
 
 
  The KIP makes sense to me (at a minimum) if there is going to be any
  breaking change. This way Kafka can continue to grow and blossom and we
  have a process in place if we are going to release a thorn ... and when we
  do it is *CLEAR* about what and why that is. We can easily document which
  KIPs where involved with this release (which I think should get committed
  afterwards somewhere so no chance of edit after release). This approach I
  had been thinking about also allows changes to occur as they do now as long
  as they are backwards compatible.  Hopefully we never need a KIP but when
  we do the PMC can vote on it and folks can read the release notes with
  *CLEAR* understanding what is going to break their existing setup... at
  least that is how I have been thinking about it.
 
 
  Let me know what you think about this base minimum approach... I hadn't
  really thought of the KIP for *ANY* major change and I have to think more
  about that. I have some other 

Re: [DISCUSS] KIPs

[Ewen Cheslack-Postava]

I think adding a section about deprecation would be helpful. A good
fraction of the time I would expect the goal of a KIP is to fix or replace
older functionality that needs continued support for compatibility, but
should eventually be phased out. This helps Kafka devs understand how long
they'll end up supporting multiple versions of features and helps users
understand when they're going to have to make updates to their applications.

Less important but useful -- having a bit of standard metadata like PEPs
do. Two I think are important are status (if someone lands on the KIP page,
can they tell whether this KIP was ever completed?) and/or the version the
KIP was first released in.



On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy jjkosh...@gmail.com wrote:

 I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
 doing this in one form or the other. However, IMO it is fairly ad hoc
 - i.e., a combination of DISCUSS threads, jira comments, RB and code
 comments, wikis and html documentation. In the past I have had to dig
 into a bunch of these to try and figure out why something was
 implemented a certain way. I think KIPs can help a lot here first by
 providing guidelines on what to think about (compatibility, new APIs,
 etc.) when working through a major feature; and second by becoming a
 crisp source of truth documentation for new releases.  E.g., for
 feature X: see relevant KIPs: a, b, c, etc.

 On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
  Hey Joe,
 
  Yeah I guess the question is what is the definition of major? I agree we
  definitely don't want to generate a bunch of paperwork. We have enough
  problems just getting all the contributions reviewed and checked in in a
  timely fashion...
 
  So obviously bug fixes would not apply here.
 
  I think it is also pretty clear that big features should get reviewed and
  discussed. To pick on myself, for example, the log compaction work was
 done
  without enough public discussion about how it worked and why (imho). I
  hope/claim that enough rigour in thinking about use-cases and operations
  and so on was done that it turned out well, but the discussion was just
  between a few people with no real public output. This kind of feature is
  clearly a big change and something we should discuss.
 
  If we limit ourselves to just the public contracts the KIP introduces the
  discussion would just be on the new configs and monitoring without
 really a
  discussion of the design and how it works which is obviously closely
  related.
 
  I don't think this should be more work because in practice we are making
  wiki pages for any big thing anyway. So this would just be a consistent
 way
  of numbering and structuring these pages. This would also give a clear
 call
  to action: hey kafka people, come read my wiki and think this through.
 
  I actually thinking the voting aspect is less important. I think it is
  generally clear when there is agreement on something and not. So from my
  point of view we could actually just eliminate that part if that is too
  formal, it just seemed like a good way to formally adopt something.
 
  To address some of your comments from the wiki:
 
  1. This doesn't inhibit someone coming along and putting up a patch. It
 is
  just that when they do if it is a big thing introducing new functionality
  we would ask for a little discussion on the basic feature/contracts prior
  to code review.
 
  2. We definitely definitely don't want people generating a lot of these
  things every time they have an idea that they aren't going to implement.
 So
  this is only applicable to things you absolutely will check in code for.
 We
  also don't want to be making proposals before things are thought through,
  which often requires writing the code. So I think the right time for a
 KIP
  is when you are far enough along that you know the issues and tradeoffs
 but
  not so far along that you are going to be totally opposed to any change.
  Sometimes that is prior to writing any code and sometimes not until you
 are
  practically done.
 
  The key problem I see this fixing is that there is enough new development
  happening that it is pretty hard for everyone to review every line of
 every
  iteration of every patch. But all of us should be fully aware of new
  features, the ramifications, the new public interfaces, etc. If we aren't
  aware of that we can't really build a holistic system that is beautiful
 and
  consistent across. So the idea is that if you fully review the KIPs you
 can
  be sure that even if you don't know every new line of code, you know the
  major changes coming in.
 
  -Jay
 
 
 
  On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein joe.st...@stealth.ly
 wrote:
 
   Thanks Jay for kicking this off! I think the confluence page you wrote
 up
   is a great start.
  
  
   The KIP makes sense to me (at a minimum) if there is going to be any
   breaking change. This way Kafka can continue to grow and blossom 

Re: [DISCUSS] KIPs

[Gwen Shapira]

+1 to Ewen's suggestions: Deprecation, status and version.

Perhaps add the JIRA where the KIP was implemented to the metadata.
This will help tie things together.

On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
e...@confluent.io wrote:
 I think adding a section about deprecation would be helpful. A good
 fraction of the time I would expect the goal of a KIP is to fix or replace
 older functionality that needs continued support for compatibility, but
 should eventually be phased out. This helps Kafka devs understand how long
 they'll end up supporting multiple versions of features and helps users
 understand when they're going to have to make updates to their applications.

 Less important but useful -- having a bit of standard metadata like PEPs
 do. Two I think are important are status (if someone lands on the KIP page,
 can they tell whether this KIP was ever completed?) and/or the version the
 KIP was first released in.



 On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy jjkosh...@gmail.com wrote:

 I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
 doing this in one form or the other. However, IMO it is fairly ad hoc
 - i.e., a combination of DISCUSS threads, jira comments, RB and code
 comments, wikis and html documentation. In the past I have had to dig
 into a bunch of these to try and figure out why something was
 implemented a certain way. I think KIPs can help a lot here first by
 providing guidelines on what to think about (compatibility, new APIs,
 etc.) when working through a major feature; and second by becoming a
 crisp source of truth documentation for new releases.  E.g., for
 feature X: see relevant KIPs: a, b, c, etc.

 On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
  Hey Joe,
 
  Yeah I guess the question is what is the definition of major? I agree we
  definitely don't want to generate a bunch of paperwork. We have enough
  problems just getting all the contributions reviewed and checked in in a
  timely fashion...
 
  So obviously bug fixes would not apply here.
 
  I think it is also pretty clear that big features should get reviewed and
  discussed. To pick on myself, for example, the log compaction work was
 done
  without enough public discussion about how it worked and why (imho). I
  hope/claim that enough rigour in thinking about use-cases and operations
  and so on was done that it turned out well, but the discussion was just
  between a few people with no real public output. This kind of feature is
  clearly a big change and something we should discuss.
 
  If we limit ourselves to just the public contracts the KIP introduces the
  discussion would just be on the new configs and monitoring without
 really a
  discussion of the design and how it works which is obviously closely
  related.
 
  I don't think this should be more work because in practice we are making
  wiki pages for any big thing anyway. So this would just be a consistent
 way
  of numbering and structuring these pages. This would also give a clear
 call
  to action: hey kafka people, come read my wiki and think this through.
 
  I actually thinking the voting aspect is less important. I think it is
  generally clear when there is agreement on something and not. So from my
  point of view we could actually just eliminate that part if that is too
  formal, it just seemed like a good way to formally adopt something.
 
  To address some of your comments from the wiki:
 
  1. This doesn't inhibit someone coming along and putting up a patch. It
 is
  just that when they do if it is a big thing introducing new functionality
  we would ask for a little discussion on the basic feature/contracts prior
  to code review.
 
  2. We definitely definitely don't want people generating a lot of these
  things every time they have an idea that they aren't going to implement.
 So
  this is only applicable to things you absolutely will check in code for.
 We
  also don't want to be making proposals before things are thought through,
  which often requires writing the code. So I think the right time for a
 KIP
  is when you are far enough along that you know the issues and tradeoffs
 but
  not so far along that you are going to be totally opposed to any change.
  Sometimes that is prior to writing any code and sometimes not until you
 are
  practically done.
 
  The key problem I see this fixing is that there is enough new development
  happening that it is pretty hard for everyone to review every line of
 every
  iteration of every patch. But all of us should be fully aware of new
  features, the ramifications, the new public interfaces, etc. If we aren't
  aware of that we can't really build a holistic system that is beautiful
 and
  consistent across. So the idea is that if you fully review the KIPs you
 can
  be sure that even if you don't know every new line of code, you know the
  major changes coming in.
 
  -Jay
 
 
 
  On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein joe.st...@stealth.ly
 

Re: [DISCUSS] KIPs

[Guozhang Wang]

+1 on the idea, and we could mutually link the KIP wiki page with the the
created JIRA ticket (i.e. include the JIRA number on the page and the KIP
url on the ticket description).

Regarding the KIP process, probably we do not need two phase communication
of a [DISCUSS] followed by [VOTE], as Jay said the voting should be clear
while people discuss about that.

About who should trigger the process, I think the only involved people
would be 1) when the patch is submitted / or even the ticket is created,
the assignee could choose to start the KIP process if she thought it is
necessary; 2) the reviewer of the patch can also suggest starting KIP
discussions.

On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira gshap...@cloudera.com
wrote:

 +1 to Ewen's suggestions: Deprecation, status and version.

 Perhaps add the JIRA where the KIP was implemented to the metadata.
 This will help tie things together.

 On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
 e...@confluent.io wrote:
  I think adding a section about deprecation would be helpful. A good
  fraction of the time I would expect the goal of a KIP is to fix or
 replace
  older functionality that needs continued support for compatibility, but
  should eventually be phased out. This helps Kafka devs understand how
 long
  they'll end up supporting multiple versions of features and helps users
  understand when they're going to have to make updates to their
 applications.
 
  Less important but useful -- having a bit of standard metadata like PEPs
  do. Two I think are important are status (if someone lands on the KIP
 page,
  can they tell whether this KIP was ever completed?) and/or the version
 the
  KIP was first released in.
 
 
 
  On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy jjkosh...@gmail.com wrote:
 
  I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
  doing this in one form or the other. However, IMO it is fairly ad hoc
  - i.e., a combination of DISCUSS threads, jira comments, RB and code
  comments, wikis and html documentation. In the past I have had to dig
  into a bunch of these to try and figure out why something was
  implemented a certain way. I think KIPs can help a lot here first by
  providing guidelines on what to think about (compatibility, new APIs,
  etc.) when working through a major feature; and second by becoming a
  crisp source of truth documentation for new releases.  E.g., for
  feature X: see relevant KIPs: a, b, c, etc.
 
  On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
   Hey Joe,
  
   Yeah I guess the question is what is the definition of major? I agree
 we
   definitely don't want to generate a bunch of paperwork. We have enough
   problems just getting all the contributions reviewed and checked in
 in a
   timely fashion...
  
   So obviously bug fixes would not apply here.
  
   I think it is also pretty clear that big features should get reviewed
 and
   discussed. To pick on myself, for example, the log compaction work was
  done
   without enough public discussion about how it worked and why (imho). I
   hope/claim that enough rigour in thinking about use-cases and
 operations
   and so on was done that it turned out well, but the discussion was
 just
   between a few people with no real public output. This kind of feature
 is
   clearly a big change and something we should discuss.
  
   If we limit ourselves to just the public contracts the KIP introduces
 the
   discussion would just be on the new configs and monitoring without
  really a
   discussion of the design and how it works which is obviously closely
   related.
  
   I don't think this should be more work because in practice we are
 making
   wiki pages for any big thing anyway. So this would just be a
 consistent
  way
   of numbering and structuring these pages. This would also give a clear
  call
   to action: hey kafka people, come read my wiki and think this
 through.
  
   I actually thinking the voting aspect is less important. I think it is
   generally clear when there is agreement on something and not. So from
 my
   point of view we could actually just eliminate that part if that is
 too
   formal, it just seemed like a good way to formally adopt something.
  
   To address some of your comments from the wiki:
  
   1. This doesn't inhibit someone coming along and putting up a patch.
 It
  is
   just that when they do if it is a big thing introducing new
 functionality
   we would ask for a little discussion on the basic feature/contracts
 prior
   to code review.
  
   2. We definitely definitely don't want people generating a lot of
 these
   things every time they have an idea that they aren't going to
 implement.
  So
   this is only applicable to things you absolutely will check in code
 for.
  We
   also don't want to be making proposals before things are thought
 through,
   which often requires writing the code. So I think the right time for a
  KIP
   is when you are far enough along that you know 

Re: [DISCUSS] KIPs

[Joe Stein]

Thanks Jay for kicking this off! I think the confluence page you wrote up
is a great start.


The KIP makes sense to me (at a minimum) if there is going to be any
breaking change. This way Kafka can continue to grow and blossom and we
have a process in place if we are going to release a thorn ... and when we
do it is *CLEAR* about what and why that is. We can easily document which
KIPs where involved with this release (which I think should get committed
afterwards somewhere so no chance of edit after release). This approach I
had been thinking about also allows changes to occur as they do now as long
as they are backwards compatible.  Hopefully we never need a KIP but when
we do the PMC can vote on it and folks can read the release notes with
*CLEAR* understanding what is going to break their existing setup... at
least that is how I have been thinking about it.


Let me know what you think about this base minimum approach... I hadn't
really thought of the KIP for *ANY* major change and I have to think more
about that. I have some other comments for minor items in the confluence
page I will make once I think more about how I feel having a KIP for more
than what I was thinking about already.


I do think we should have major changes go through confluence, mailing
list discuss and JIRA but kind of feel we have been doing that already ...
if there are cases where that isn't the case we should highlight and learn
from them and formalize that more if need be.


/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps jay.kr...@gmail.com wrote:

 The idea of KIPs came up in a previous discussion but there was no real
 crisp definition of what they were. Here is an attempt at defining a
 process:

 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

 The trick here is to have something light-weight enough that it isn't a
 hassle for small changes, but enough so that changes get the eyeballs of
 the committers and heavy users.

 Thoughts?

 -Jay

Re: [DISCUSS] KIPs

[Jay Kreps]

Hey Joe,

Yeah I guess the question is what is the definition of major? I agree we
definitely don't want to generate a bunch of paperwork. We have enough
problems just getting all the contributions reviewed and checked in in a
timely fashion...

So obviously bug fixes would not apply here.

I think it is also pretty clear that big features should get reviewed and
discussed. To pick on myself, for example, the log compaction work was done
without enough public discussion about how it worked and why (imho). I
hope/claim that enough rigour in thinking about use-cases and operations
and so on was done that it turned out well, but the discussion was just
between a few people with no real public output. This kind of feature is
clearly a big change and something we should discuss.

If we limit ourselves to just the public contracts the KIP introduces the
discussion would just be on the new configs and monitoring without really a
discussion of the design and how it works which is obviously closely
related.

I don't think this should be more work because in practice we are making
wiki pages for any big thing anyway. So this would just be a consistent way
of numbering and structuring these pages. This would also give a clear call
to action: hey kafka people, come read my wiki and think this through.

I actually thinking the voting aspect is less important. I think it is
generally clear when there is agreement on something and not. So from my
point of view we could actually just eliminate that part if that is too
formal, it just seemed like a good way to formally adopt something.

To address some of your comments from the wiki:

1. This doesn't inhibit someone coming along and putting up a patch. It is
just that when they do if it is a big thing introducing new functionality
we would ask for a little discussion on the basic feature/contracts prior
to code review.

2. We definitely definitely don't want people generating a lot of these
things every time they have an idea that they aren't going to implement. So
this is only applicable to things you absolutely will check in code for. We
also don't want to be making proposals before things are thought through,
which often requires writing the code. So I think the right time for a KIP
is when you are far enough along that you know the issues and tradeoffs but
not so far along that you are going to be totally opposed to any change.
Sometimes that is prior to writing any code and sometimes not until you are
practically done.

The key problem I see this fixing is that there is enough new development
happening that it is pretty hard for everyone to review every line of every
iteration of every patch. But all of us should be fully aware of new
features, the ramifications, the new public interfaces, etc. If we aren't
aware of that we can't really build a holistic system that is beautiful and
consistent across. So the idea is that if you fully review the KIPs you can
be sure that even if you don't know every new line of code, you know the
major changes coming in.

-Jay



On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein joe.st...@stealth.ly wrote:

 Thanks Jay for kicking this off! I think the confluence page you wrote up
 is a great start.


 The KIP makes sense to me (at a minimum) if there is going to be any
 breaking change. This way Kafka can continue to grow and blossom and we
 have a process in place if we are going to release a thorn ... and when we
 do it is *CLEAR* about what and why that is. We can easily document which
 KIPs where involved with this release (which I think should get committed
 afterwards somewhere so no chance of edit after release). This approach I
 had been thinking about also allows changes to occur as they do now as long
 as they are backwards compatible.  Hopefully we never need a KIP but when
 we do the PMC can vote on it and folks can read the release notes with
 *CLEAR* understanding what is going to break their existing setup... at
 least that is how I have been thinking about it.


 Let me know what you think about this base minimum approach... I hadn't
 really thought of the KIP for *ANY* major change and I have to think more
 about that. I have some other comments for minor items in the confluence
 page I will make once I think more about how I feel having a KIP for more
 than what I was thinking about already.


 I do think we should have major changes go through confluence, mailing
 list discuss and JIRA but kind of feel we have been doing that already ...
 if there are cases where that isn't the case we should highlight and learn
 from them and formalize that more if need be.


 /***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /

 On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps jay.kr...@gmail.com wrote:

  The idea of KIPs came up in a