Harsha thanks for opening the discussion on this KIP. While I understand he founding members' stand that the Kafka project can not expand its surface to a large number of clients, I strongly agree with your well explained points below and support your KIP.
A REST API is not just on the same level as any client, it's a basic building block of open web technologies. It's the API that most of our first time user want to try out (or that would be users ask for and expect to be there). A REST API for Kafka and a robust server implementation under the open governance of the Apache community would be most welcome. +1 Edo -------------------------------------------------- Edoardo Comar IBM MessageHub IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU Harsha Chintalapani <ka...@harsha.io> wrote on 02/10/2016 21:23:15: > From: Harsha Chintalapani <ka...@harsha.io> > To: dev@kafka.apache.org > Date: 02/10/2016 21:23 > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > Neha & Jay, > We did look at the open source alternatives. Our concern > is what's the patch acceptance and adding features/ bug-fixes to the > existing project under a Github (although it's licensed under Apache 2.0). > It would be great if that project made available under Apache and driven by > the community. Adding to the above, not all Kafka users are interested in > using the Java client API, they would like to have simple REST API where > they can code against using any language. I do believe this adds value to > Apache Kafka in itself. > > "For 1, I don't think there is value in giving in to the NIH syndrome and > reinventing the wheel. What I'm looking for is a detailed comparison of the > gaps and why those can't be improved in the REST proxy that already exists > and is actively maintained." > > We are not looking at this as NIH. What we are worried about is a project > that's not maintained in a community. So the process of accepting patches > and priorities is not clear, and it's not developed in Apache community. > Not only that, existing REST API project doesn't support new client API and > hence there is no security support either. > We don't know the timeline when that's made available. We would like to add > admin functionality into the REST API. So the Roadmap of that project is > not driven by Apache. > > > "This doesn't materially have an impact on expanding the usability of > Kafka. In my experience, REST proxy + Java clients only cover ~50% of all > Kafka users, and maybe 10% of those are the ones who will use the REST > proxy. The remaining 50% are non-java client users (C, python, go, node > etc)." > > REST API is most often asked feature in my interactions with Kafka users. > In an organization, There will be independent teams who will write their > Kafka clients using different language libraries available today, and > there is no way to standardize this. Instead of supporting several > different client libraries users will be interested in using a REST API > server. The need for a REST API will only increase as more and more users > start using Kafka. > > "More surface area means more work to keep things consistent. Failure > to do that has, in fact, hurt the user experience." > Having myriad Kafka client GitHub projects that support different languages > hurts the user experience and pushes burden to maintain these libraries. > REST API is a simple code base that uses existing java client libraries to > make life easier for the users. > > Thanks, > Harsha > > On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <n...@confluent.io> wrote: > > > Manikumar, > > > > Thanks for sharing the proposal. I think there are 2 parts to this > > discussion - > > > > 1. Should we rewrite a REST proxy given that there is a feature-complete, > > open-source and actively maintained REST proxy in the community? > > 2. Does adding a REST proxy to Apache Kafka make us more agile and maintain > > the high-quality experience that Kafka users have today? > > > > For 1, I don't think there is value in giving in to the NIH syndrome and > > reinventing the wheel. What I'm looking for is a detailed comparison of the > > gaps and why those can't be improved in the REST proxy that already exists > > and is actively maintained. For example, we depend on zkClient and have > > found as well as fixed several bugs by working closely with the people who > > maintain zkClient. This should be possible for REST proxy as well, right? > > > > For 2, I'd like us to review our history of expanding the surface area to > > add more clients in the past. Here is a summary - > > > > - This doesn't materially have an impact on expanding the usability of > > Kafka. In my experience, REST proxy + Java clients only cover ~50% of > > all > > Kafka users, and maybe 10% of those are the ones who will use the REST > > proxy. The remaining 50% are non-java client users (C, python, go, node > > etc). > > - People are a lot more excited about promising to contribute while > > adding the surface area but not on an ongoing basis down the line. > > - More surface area means more work to keep things consistent. Failure > > to do that has, in fact, hurt the user experience. > > - More surface area hurts agility. We want to do a few things really > > well as well as be agile to be able to build on our core competency. > > > > > > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <manikumar.re...@gmail.com> > > wrote: > > > > > Hi Jay, > > > > > > Thanks for your reply. > > > > > > I agree that we can not add all the clients/tools available in ecosystem > > > page to Kafka repo itself. But we feel REST Interface is different from > > > other clients/tools. Since any language that can work with HTTP can > > > easily integrate with this interface, Having an "official" REST > > > interface helps user community. This also helps us to integrate well > > > with external management and provisioning tools. Apache Kafka release > > > with Java clients + REST interface is sufficient for most of the user > > > deployments/requirements. This helps users to deal with less number > > > of distributions/builds. > > > > > > Thanks, > > > Manikumar > > > > > > > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <j...@confluent.io> wrote: > > > > > > > Hey guys, > > > > > > > > There's already a REST interface maintained as a separate project--it's > > > > open source and apache licensed and actively maintained ( > > > > https://github.com/confluentinc/kafka-rest). What is wrong with that? > > > You > > > > mentioned that there was some compatibility concern, but compatibility > > > has > > > > to do with the consumer protocol guarantees not the repo the code is > > in, > > > > right? Not sure that concern makes sense. > > > > > > > > We could argue for adding pretty much anything and everything in the > > > > ecosystem page in Kafka itself but I'm not sure that would make the > > > project > > > > more agile. > > > > > > > > -Jay > > > > > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <manikumar.re...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi Kafka Devs, > > > > > > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > > > > > > > There are already open-source alternatives are available. But we > > would > > > > > like to add REST server that > > > > > many users ask for under Apache Kafka repo. Many data Infra tools > > comes > > > > up > > > > > with Rest Interface. > > > > > It is useful to have inbuilt Rest API support for Produce, Consume > > > > messages > > > > > and admin interface for > > > > > integrating with external management and provisioning tools.This will > > > > also > > > > > allow the maintenance of > > > > > REST server and adding new features makes it easy because apache > > > > community. > > > > > > > > > > The KIP wiki is the following: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > 80%3A+Kafka+Rest+Server > > > > > > > > > > Your comments and feedback are welcome. > > > > > > > > > > Thanks, > > > > > Manikumar > > > > > > > > > > > > > > -- > > Thanks, > > Neha > > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
