I'm definitely +1 on the KIP concept. As Joe mentioned, we are already
doing this in one form or the other. However, IMO it is fairly ad hoc
- i.e., a combination of DISCUSS threads, jira comments, RB and code
comments, wikis and html documentation. In the past I have had to dig
into a bunch of these to try and figure out why something was
implemented a certain way. I think KIPs can help a lot here first by
providing guidelines on what to think about (compatibility, new APIs,
etc.) when working through a major feature; and second by becoming a
crisp source of truth documentation for new releases.  E.g., for
feature X: see relevant KIPs: a, b, c, etc.

On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote:
> Hey Joe,
> 
> Yeah I guess the question is what is the definition of major? I agree we
> definitely don't want to generate a bunch of paperwork. We have enough
> problems just getting all the contributions reviewed and checked in in a
> timely fashion...
> 
> So obviously bug fixes would not apply here.
> 
> I think it is also pretty clear that big features should get reviewed and
> discussed. To pick on myself, for example, the log compaction work was done
> without enough public discussion about how it worked and why (imho). I
> hope/claim that enough rigour in thinking about use-cases and operations
> and so on was done that it turned out well, but the discussion was just
> between a few people with no real public output. This kind of feature is
> clearly a big change and something we should discuss.
> 
> If we limit ourselves to just the public contracts the KIP introduces the
> discussion would just be on the new configs and monitoring without really a
> discussion of the design and how it works which is obviously closely
> related.
> 
> I don't think this should be more work because in practice we are making
> wiki pages for any big thing anyway. So this would just be a consistent way
> of numbering and structuring these pages. This would also give a clear call
> to action: "hey kafka people, come read my wiki and think this through".
> 
> I actually thinking the voting aspect is less important. I think it is
> generally clear when there is agreement on something and not. So from my
> point of view we could actually just eliminate that part if that is too
> formal, it just seemed like a good way to formally adopt something.
> 
> To address some of your comments from the wiki:
> 
> 1. This doesn't inhibit someone coming along and putting up a patch. It is
> just that when they do if it is a big thing introducing new functionality
> we would ask for a little discussion on the basic feature/contracts prior
> to code review.
> 
> 2. We definitely definitely don't want people generating a lot of these
> things every time they have an idea that they aren't going to implement. So
> this is only applicable to things you absolutely will check in code for. We
> also don't want to be making proposals before things are thought through,
> which often requires writing the code. So I think the right time for a KIP
> is when you are far enough along that you know the issues and tradeoffs but
> not so far along that you are going to be totally opposed to any change.
> Sometimes that is prior to writing any code and sometimes not until you are
> practically done.
> 
> The key problem I see this fixing is that there is enough new development
> happening that it is pretty hard for everyone to review every line of every
> iteration of every patch. But all of us should be fully aware of new
> features, the ramifications, the new public interfaces, etc. If we aren't
> aware of that we can't really build a holistic system that is beautiful and
> consistent across. So the idea is that if you fully review the KIPs you can
> be sure that even if you don't know every new line of code, you know the
> major changes coming in.
> 
> -Jay
> 
> 
> 
> On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein <joe.st...@stealth.ly> wrote:
> 
> > Thanks Jay for kicking this off! I think the confluence page you wrote up
> > is a great start.
> >
> >
> > The KIP makes sense to me (at a minimum) if there is going to be any
> > "breaking change". This way Kafka can continue to grow and blossom and we
> > have a process in place if we are going to release a thorn ... and when we
> > do it is *CLEAR* about what and why that is. We can easily document which
> > KIPs where involved with this release (which I think should get committed
> > afterwards somewhere so no chance of edit after release). This approach I
> > had been thinking about also allows changes to occur as they do now as long
> > as they are backwards compatible.  Hopefully we never need a KIP but when
> > we do the PMC can vote on it and folks can read the release notes with
> > *CLEAR* understanding what is going to break their existing setup... at
> > least that is how I have been thinking about it.
> >
> >
> > Let me know what you think about this base minimum approach... I hadn't
> > really thought of the KIP for *ANY* "major change" and I have to think more
> > about that. I have some other comments for minor items in the confluence
> > page I will make once I think more about how I feel having a KIP for more
> > than what I was thinking about already.
> >
> >
> > I do think we should have "major changes" go through confluence, mailing
> > list discuss and JIRA but kind of feel we have been doing that already ...
> > if there are cases where that isn't the case we should highlight and learn
> > from them and formalize that more if need be.
> >
> >
> > /*******************************************
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps <jay.kr...@gmail.com> wrote:
> >
> > > The idea of KIPs came up in a previous discussion but there was no real
> > > crisp definition of what they were. Here is an attempt at defining a
> > > process:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> > > The trick here is to have something light-weight enough that it isn't a
> > > hassle for small changes, but enough so that changes get the eyeballs of
> > > the committers and heavy users.
> > >
> > > Thoughts?
> > >
> > > -Jay
> > >
> >

Reply via email to