Hey Joe, Yeah I guess the question is what is the definition of major? I agree we definitely don't want to generate a bunch of paperwork. We have enough problems just getting all the contributions reviewed and checked in in a timely fashion...
So obviously bug fixes would not apply here. I think it is also pretty clear that big features should get reviewed and discussed. To pick on myself, for example, the log compaction work was done without enough public discussion about how it worked and why (imho). I hope/claim that enough rigour in thinking about use-cases and operations and so on was done that it turned out well, but the discussion was just between a few people with no real public output. This kind of feature is clearly a big change and something we should discuss. If we limit ourselves to just the public contracts the KIP introduces the discussion would just be on the new configs and monitoring without really a discussion of the design and how it works which is obviously closely related. I don't think this should be more work because in practice we are making wiki pages for any big thing anyway. So this would just be a consistent way of numbering and structuring these pages. This would also give a clear call to action: "hey kafka people, come read my wiki and think this through". I actually thinking the voting aspect is less important. I think it is generally clear when there is agreement on something and not. So from my point of view we could actually just eliminate that part if that is too formal, it just seemed like a good way to formally adopt something. To address some of your comments from the wiki: 1. This doesn't inhibit someone coming along and putting up a patch. It is just that when they do if it is a big thing introducing new functionality we would ask for a little discussion on the basic feature/contracts prior to code review. 2. We definitely definitely don't want people generating a lot of these things every time they have an idea that they aren't going to implement. So this is only applicable to things you absolutely will check in code for. We also don't want to be making proposals before things are thought through, which often requires writing the code. So I think the right time for a KIP is when you are far enough along that you know the issues and tradeoffs but not so far along that you are going to be totally opposed to any change. Sometimes that is prior to writing any code and sometimes not until you are practically done. The key problem I see this fixing is that there is enough new development happening that it is pretty hard for everyone to review every line of every iteration of every patch. But all of us should be fully aware of new features, the ramifications, the new public interfaces, etc. If we aren't aware of that we can't really build a holistic system that is beautiful and consistent across. So the idea is that if you fully review the KIPs you can be sure that even if you don't know every new line of code, you know the major changes coming in. -Jay On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein <joe.st...@stealth.ly> wrote: > Thanks Jay for kicking this off! I think the confluence page you wrote up > is a great start. > > > The KIP makes sense to me (at a minimum) if there is going to be any > "breaking change". This way Kafka can continue to grow and blossom and we > have a process in place if we are going to release a thorn ... and when we > do it is *CLEAR* about what and why that is. We can easily document which > KIPs where involved with this release (which I think should get committed > afterwards somewhere so no chance of edit after release). This approach I > had been thinking about also allows changes to occur as they do now as long > as they are backwards compatible. Hopefully we never need a KIP but when > we do the PMC can vote on it and folks can read the release notes with > *CLEAR* understanding what is going to break their existing setup... at > least that is how I have been thinking about it. > > > Let me know what you think about this base minimum approach... I hadn't > really thought of the KIP for *ANY* "major change" and I have to think more > about that. I have some other comments for minor items in the confluence > page I will make once I think more about how I feel having a KIP for more > than what I was thinking about already. > > > I do think we should have "major changes" go through confluence, mailing > list discuss and JIRA but kind of feel we have been doing that already ... > if there are cases where that isn't the case we should highlight and learn > from them and formalize that more if need be. > > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > The idea of KIPs came up in a previous discussion but there was no real > > crisp definition of what they were. Here is an attempt at defining a > > process: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > The trick here is to have something light-weight enough that it isn't a > > hassle for small changes, but enough so that changes get the eyeballs of > > the committers and heavy users. > > > > Thoughts? > > > > -Jay > > >
