I've been a little reluctant to wade into this discussion since I've spent a lot of my own time on Confluent's kafka-rest project myself. Seems obvious I would hope that time is not wasted and the project succeeds, right? I think it's the same for a lot of others who have submitted patches to it. Admittedly, progress has not been as rapid as we would like. It would be great if we could get more contributions, especially from the companies that seem interested in offering a service like it. Nevertheless, I'd like to see an attempt to resolve any problems with that project in its respective community first before the Kafka project takes over.
Thanks, Jason On Thu, Oct 20, 2016 at 2:26 PM, Harsha Ch <harsha...@gmail.com> wrote: > Jay, > REST API is something every user is in need of. If the argument is to > clone and write your API, this will do a disservice to the users as they > now have to choose one vs. others instead of keeping one API that is > supported in Kafka community. > > "Pre-emptively re-creating another > REST layer when it seems like we all quite agree on what needs to be done > and we have an existing code base for HTTP/Kafka access that is heavily > used in production seems quite silly." > > Exactly our point. Why can't we develop this in Apache Kafka > community? Instead of us open sourcing another GitHub project and creating > a divide in users and another version of API. Let's build this in Kafka > Community and use the governance model that is proven to provide vendor > free user driven consensus features. The argument that is adding this REST > server to Kafka will affect the agility of the project doesn't mak sense. > > It looks like your argument is either we develop all these small tools or > none at all. We as a community need to look at supporting critical > tools/API. Instead of dividing this project into individual external > communities. We should build this as part of Kafka which best serves the > needs of users. > The Streams and Connect projects that were pushed into Kafka could > have been left in their own Github projects based on your arguments. What > about the REST API is so different that such that it should stay out of the > Kafka project? From my experience, more users are asking for the REST API. > > Thanks, > Harsha > > > > > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote: > > > I think the questions around governance make sense, I think we should > > really clarify that to make the process more clear so it can be fully > > inclusive. > > > > The idea that we should not collaborate on what is there now, though, > > because in the future we might disagree about direction does not really > > make sense to me. If in the future we disagree, that is the beauty of > open > > source, you can always fork off a copy of the code and start an > independent > > project either in Apache or elsewhere. Pre-emptively re-creating another > > REST layer when it seems like we all quite agree on what needs to be done > > and we have an existing code base for HTTP/kafka access that is heavily > > used in production seems quite silly. > > > > Let me give some background on how I at least think about these things. > > I've participated in open source projects out of LinkedIn via github as > > well as via the ASF. I don't think there is a "right" answer to how to do > > these but rather some tradeoffs. We thought about this quite a lot in the > > context of Kafka based on the experience with the Hadoop ecosystem as > well > > as from other open source communities. > > > > There is a rich ecosystem around Kafka. Many of the projects are quite > > small--single clients or tools that do a single thing well--and almost > none > > of them are top level apache projects. I don't think trying to force each > > of these to turn into independent Apache projects is necessarily the best > > thing for the ecosystem. > > > > My observation of how this can go wrong is really what I think has > happened > > to the Hadoop ecosystem. There you see quite a zoo of projects which all > > drift apart and don't quite work together well. Coordinating even simple > > changes and standardization across these is exceptionally difficult. The > > result is a bit of a mess for users--the pieces just don't really come > > together very well. This makes sense for independent infrastructure > systems > > (Kudu vs HDFS) but I'm not at all convinced that doing this for every > > little tool or helper library has lead to a desirable state. I think the > > mode of operating where the Hadoop vendors spawn off a few new Apache > > projects for each new product initiative, especially since often that > > project is only valued by that vendor (and the other vendor has a > different > > competing Apache project) doesn't necessarily do a better job at > producing > > high quality communities or high quality software. > > > > These tools/connects/clients/proxies and other integration pieces can > take > > many forms, but my take of what makes one of these things good is that it > > remains simple, does its one thing well, and cleaves as closely as > possible > > to the conventions for Kafka itself--i.e. doesn't invent new ways of > > monitoring, configuring, etc. For the tools we've contributed we've tried > > really hard to make them consistent with Kafka as well as with each other > > in how testing, configuration, monitoring, etc works. > > > > I think what Apache does superbly well is create a community for > managing a > > large infrastructure layer like Kafka in a vendor independent way. What I > > think is less successful is attempting to form full and independent > apache > > communities around very simple single purpose tools, especially if you > hope > > for these to come together into a cohesive toolset across multiple such > > tools. Much of what Apache does--create a collective decision making > > process for resolving disagreement, help to trademark and protect the > marks > > of the project, etc just isn't that relevant for simple single-purpose > > tools. > > > > So my take is there are a couple of options: > > > > 1. We can try to put all the small tools into the Apache Project. I > > think this is not the right approach as there is simply too many of > > them, > > many in different languages, serving different protocols, integrating > > with > > particular systems, and a single community can't effectively maintain > > them > > all. Doing this would significantly slow the progress of the Kafka > > project. > > As a protocol for messaging, I don't really see a case for including > > REST > > but not MQTT or AMQP which are technically much better suited to > > messaging > > and both are widely used for that. > > 2. We can treat ecosystem projects that aren't top level Apache > projects > > as invalid and try to recreate them all as Apache projects. Honestly, > > though, if you go to the Kafka ecosystem page virtually none of the > most > > popular add-ons to Kafka are Apache projects. The most successful > > things in > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of > other > > clients, as well as the existing REST layer have succeeded at > developing > > communities that actively contribute and use these pieces and I don't > > know > > that that is a bad thing unless that community proves to be > uninclusive, > > unresponsive, or goes in a bad technical direction--and those are > > failure > > modes that all open source efforts face. > > 3. We can do what I think makes the most sense and try to work with > the > > projects that exist in the ecosystem and if the project doesn't have a > > responsive community or wants to go in a different direction fork or > > recreate that work. > > > > Of course any person can choose whatever of these options they want. But > > from my point of view, option (3) has been the path of the community so > far > > and I think it has been quite successful. > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io> > > wrote: > > > > > Neha, > > > "But I haven't seen any community emails or patches being submitted by > > you > > > guys, so I'm wondering why you are concerned about whether the > community > > is > > > open to accepting patches or not." > > > > > > I think you are talking about contributing patches to this repository > > > right? https://github.com/confluentinc/kafka-rest . All I am saying > > > the guidelines/governance model is not clear on the project and I guess > > its > > > driven by opening a github issue request. Its the repository owned by > > > confluent and as much I appreciate that the features we mentioned are > in > > > the roadmap and welcoming us to contribute to the project. It doesn't > > > gurantee what we want to add in the furture will be in your roadmap. > > > > > > Hence the reason having it part of Kafka community will help a lot as > > other > > > users can participate in the discussions. We are happy to drive any > > > feature additions through KIPs which gives everyone a chance to > > participate > > > and add to the discussions. > > > > > > Thanks, > > > Harsha > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com> > > > wrote: > > > > > > > +1 > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > Also i agree about the contribution comments made earlier in the > > thread, > > > i > > > > personally am less likely to spend any of mine, or give project time > > > within > > > > my internal projects to developers contributing to another commercial > > > > companies project even so technically open source, as then there is > > that > > > > commercial companies interest will always prevail and essentially can > > > > always have a final vote where disagreement. Im sure they never > intend > > > to, > > > > but there is that true reality. This is why we have community open > > source > > > > projects. > > > > > > > > I can find many different implementations now of a rest endpoint on > > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages > in > > > > their implementation. By making / providing one this would bring > > together > > > > these solutions, unifying those developers and also bringing the best > > of > > > > all. > > > > > > > > I understand the concern on the community burden adding/supporting > more > > > > surface area for every client. But something like REST is universal > and > > > > worthy to be owned by the community. > > > > > > > > Mike > > > > > > > > > > > > ________________________________________ > > > > From: Andrew Schofield <andrew_schofield_j...@outlook.com> > > > > Sent: Saturday, October 8, 2016 1:19 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > There's a massive difference between the governance of Kafka and the > > > > governance of the REST proxy. > > > > > > > > In Kafka, there is a broad community of people contributing their > > > opinions > > > > about future enhancements in the form of KIPs. There's some really > deep > > > > consideration that goes into some of the trickier KIPs. There are > > people > > > > outside Confluent deeply knowledgeable in Kafka and building the > > > > reputations to become committers. I get the impression that the > roadmap > > > of > > > > Kafka is not really community-owned (what's the big feature for Kafka > > > 0.11, > > > > for example), but the conveyor belt of smaller features in the form > of > > > KIPs > > > > works nicely. It's a good example of open-source working well. > > > > > > > > The equivalent for the REST proxy is basically issues on GitHub. The > > > > roadmap is less clear. There's not really a community properly > engaged > > in > > > > the way that there is with Kafka. So, you could say that it's clear > > that > > > > fewer people are interested, but I think the whole governance thing > > is a > > > > big barrier to engagement. And it's looking like it's getting out of > > > date. > > > > > > > > In technical terms, I can think of two big improvements to the REST > > > proxy. > > > > First, it needs to use the new consumer API so that it's possible to > > > secure > > > > connections between the REST proxy and Kafka. I don't care too much > > which > > > > method calls it uses actually uses to consume messages, but I do > care > > > that > > > > I cannot secure connections because of network security rules. > Second, > > > > there's an affinity between a consumer and the instance of the REST > > proxy > > > > to which it first connected. Kafka itself avoids this kind of > affinity > > > for > > > > good reason, and in the name of availability the REST proxy should > too. > > > > These are natural KIPs. > > > > > > > > I think it would be good to have the code for the REST proxy > > contributed > > > > to Apache Kafka so that it would be able to be developed in the same > > way. > > > > > > > > Andrew Schofield > > > > > > > > From: Suresh Srinivas <sur...@hortonworks.com> > > > > Sent: 07 October 2016 22:41:52 > > > > To: dev@kafka.apache.org > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > ASF already gives us a clear framework and governance model for > > community > > > > development. This is already understood by the people contributing to > > > > Apache Kafka project, and they are the same people who want to > > contribute > > > > to the REST server capability as well. Everyone is in agreement on > the > > > > need for collaborating on this effort. So why not contribute the code > > to > > > > Apache Kafka. This will help avoid duplication of effort and forks > that > > > > may crop up, hugely benefitting the user community. This will also > > avoid > > > > having to define a process similar to ASF on a GitHub project and > > instead > > > > there is a single community with clear understanding community > process > > as > > > > defined in ASF. > > > > > > > > As others have said, this is an important capability for Apache > Kafka. > > It > > > > is worth maintaining this as a part of the project. > > > > > > > > Regards, > > > > Suresh > > > > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <ofir.ma...@equalum.io> wrote: > > > > > > > > >I personally think it would be quite wasteful to re-implement the > REST > > > > >gateway just because that an actively-maintained piece of > > > Apache-licensed > > > > >software is not governed directly by the Apache Kafka community... > > While > > > > >kafka-rest repo is owned by Confluent, the contributors including > the > > > main > > > > >one are also part of the Apache Kafka community, so there is a > chance > > > to > > > > >work this out. > > > > > > > > > >However, there are two valid concerns here that could be addressed, > > > around > > > > >community and accessibility: > > > > >>> What we are worried about is a project > > > > >>> that's not maintained in a community. So the process of accepting > > > > >>>patches > > > > >>> and priorities is not clear, and it's not developed in Apache > > > > >>>community. > > > > >>> Not only that, existing REST API project doesn't support new > client > > > API > > > > >and > > > > >>> hence there is no security support either. > > > > > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest community > can > > > > >clarify that - what is their contribution policy, dev style, roadmap > > > etc. > > > > >If they want, they can make an effort to encourage participation > from > > > > >people outside Confluent (easily accept contributions, invite > external > > > > >commiters or have open dev process similar to Apache Kafka etc), as > > > there > > > > >is definitely seems to be some interest on the list. That might > clear > > > the > > > > >community concern and help kafka-rest project (but that is a > > calculation > > > > >Confluent will have to make). > > > > > > > > > >The other, independent, concern is that REST is something that is > > > expected > > > > >to be available out of the box with Kafka. I personally don't feel > > > > >strongly > > > > >about it (better use proper, efficient APIs from day one), though it > > is > > > > >definitely way smaller than adding a stream processing engine to the > > > > >project :) > > > > >Again,the kafka-rest "community" could take steps to make it even > > easier > > > > >to > > > > >install, configure and run kafka-rest for new users on vanilla > Apache > > > > >Kafka > > > > >(outside the Confluent platform), if they wish that (or welcome > > > > >contributions to that end), but that is up to them. > > > > >Finally, if after the above steps were taken there would still a > > strong > > > > >desire to include a great rest gateway with Apache Kafka, I assume > the > > > > >community could hypothetically fork the existing kafka-rest into an > > > Apache > > > > >Kafka subproject and maintain it "within Apache" instead of > > implementing > > > > >it > > > > >from scratch (though I'm not a lawyer etc) - but I cannot imagine it > > > > >happen > > > > >without Confluent blessing, and I think that is likely much less > > optimal > > > > >(pulling in other Confluent / Apache licensed dependencies) than > > having > > > a > > > > >separate external community around kafka-rest. > > > > > > > > > > > > > > >Just my two cents, > > > > > > > > > > > > > > >Ofir Manor > > > > > > > > > >Co-Founder & CTO | Equalum > > > > > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> <+972%2054-780-1286> | > > Email: > > > > ofir.ma...@equalum.io > > > > > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani < > ka...@harsha.io > > > > > > > >wrote: > > > > > > > > > >> Neha & Jay, > > > > >> We did look at the open source alternatives. Our > > > > >>concern > > > > >> is what's the patch acceptance and adding features/ bug-fixes to > the > > > > >> existing project under a Github (although it's licensed under > Apache > > > > >>2.0). > > > > >> It would be great if that project made available under Apache and > > > > >>driven by > > > > >> the community. Adding to the above, not all Kafka users are > > > interested > > > > >>in > > > > >> using the Java client API, they would like to have simple REST API > > > where > > > > >> they can code against using any language. I do believe this adds > > value > > > > >>to > > > > >> Apache Kafka in itself. > > > > >> > > > > >> "For 1, I don't think there is value in giving in to the NIH > > syndrome > > > > >>and > > > > >> reinventing the wheel. What I'm looking for is a detailed > comparison > > > of > > > > >>the > > > > >> gaps and why those can't be improved in the REST proxy that > already > > > > >>exists > > > > >> and is actively maintained." > > > > >> > > > > >> We are not looking at this as NIH. What we are worried about is a > > > > >>project > > > > >> that's not maintained in a community. So the process of accepting > > > > >>patches > > > > >> and priorities is not clear, and it's not developed in Apache > > > community. > > > > >> Not only that, existing REST API project doesn't support new > client > > > API > > > > >>and > > > > >> hence there is no security support either. > > > > >> We don't know the timeline when that's made available. We would > like > > > to > > > > >>add > > > > >> admin functionality into the REST API. So the Roadmap of that > > project > > > is > > > > >> not driven by Apache. > > > > >> > > > > >> > > > > >> "This doesn't materially have an impact on expanding the usability > > of > > > > >> Kafka. In my experience, REST proxy + Java clients only cover > > ~50% > > > of > > > > >> all > > > > >> Kafka users, and maybe 10% of those are the ones who will use > the > > > > >>REST > > > > >> proxy. The remaining 50% are non-java client users (C, python, > > go, > > > > >>node > > > > >> etc)." > > > > >> > > > > >> REST API is most often asked feature in my interactions with Kafka > > > > >>users. > > > > >> In an organization, There will be independent teams who will write > > > their > > > > >> Kafka clients using different language libraries available today, > > and > > > > >> there is no way to standardize this. Instead of supporting several > > > > >> different client libraries users will be interested in using a > REST > > > API > > > > >> server. The need for a REST API will only increase as more and > more > > > > >>users > > > > >> start using Kafka. > > > > >> > > > > >> "More surface area means more work to keep things consistent. > > Failure > > > > >> to do that has, in fact, hurt the user experience." > > > > >> Having myriad Kafka client GitHub projects that support different > > > > >>languages > > > > >> hurts the user experience and pushes burden to maintain these > > > libraries. > > > > >> REST API is a simple code base that uses existing java client > > > libraries > > > > >>to > > > > >> make life easier for the users. > > > > >> > > > > >> Thanks, > > > > >> Harsha > > > > >> > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <n...@confluent.io> > > > > wrote: > > > > >> > > > > >> > Manikumar, > > > > >> > > > > > >> > Thanks for sharing the proposal. I think there are 2 parts to > this > > > > >> > discussion - > > > > >> > > > > > >> > 1. Should we rewrite a REST proxy given that there is a > > > > >>feature-complete, > > > > >> > open-source and actively maintained REST proxy in the community? > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more agile > and > > > > >> maintain > > > > >> > the high-quality experience that Kafka users have today? > > > > >> > > > > > >> > For 1, I don't think there is value in giving in to the NIH > > syndrome > > > > >>and > > > > >> > reinventing the wheel. What I'm looking for is a detailed > > comparison > > > > >>of > > > > >> the > > > > >> > gaps and why those can't be improved in the REST proxy that > > already > > > > >> exists > > > > >> > and is actively maintained. For example, we depend on zkClient > and > > > > >>have > > > > >> > found as well as fixed several bugs by working closely with the > > > people > > > > >> who > > > > >> > maintain zkClient. This should be possible for REST proxy as > well, > > > > >>right? > > > > >> > > > > > >> > For 2, I'd like us to review our history of expanding the > surface > > > > >>area to > > > > >> > add more clients in the past. Here is a summary - > > > > >> > > > > > >> > - This doesn't materially have an impact on expanding the > > > > >>usability of > > > > >> > Kafka. In my experience, REST proxy + Java clients only cover > > > ~50% > > > > >>of > > > > >> > all > > > > >> > Kafka users, and maybe 10% of those are the ones who will use > > the > > > > >>REST > > > > >> > proxy. The remaining 50% are non-java client users (C, > python, > > > go, > > > > >> node > > > > >> > etc). > > > > >> > - People are a lot more excited about promising to contribute > > > while > > > > >> > adding the surface area but not on an ongoing basis down the > > > line. > > > > >> > - More surface area means more work to keep things > consistent. > > > > >>Failure > > > > >> > to do that has, in fact, hurt the user experience. > > > > >> > - More surface area hurts agility. We want to do a few things > > > > >>really > > > > >> > well as well as be agile to be able to build on our core > > > > >>competency. > > > > >> > > > > > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar < > > manikumar.re...@gmail.com > > > > > > > > >> > wrote: > > > > >> > > > > > >> > > Hi Jay, > > > > >> > > > > > > >> > > Thanks for your reply. > > > > >> > > > > > > >> > > I agree that we can not add all the clients/tools available in > > > > >> ecosystem > > > > >> > > page to Kafka repo itself. But we feel REST Interface is > > different > > > > >>from > > > > >> > > other clients/tools. Since any language that can work with > HTTP > > > can > > > > >> > > easily integrate with this interface, Having an "official" > REST > > > > >> > > interface helps user community. This also helps us to > integrate > > > well > > > > >> > > with external management and provisioning tools. Apache Kafka > > > > >>release > > > > >> > > with Java clients + REST interface is sufficient for most of > the > > > > >>user > > > > >> > > deployments/requirements. This helps users to deal with less > > > number > > > > >> > > of distributions/builds. > > > > >> > > > > > > >> > > Thanks, > > > > >> > > Manikumar > > > > >> > > > > > > >> > > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <j...@confluent.io> > > > wrote: > > > > >> > > > > > > >> > > > Hey guys, > > > > >> > > > > > > > >> > > > There's already a REST interface maintained as a separate > > > > >> project--it's > > > > >> > > > open source and apache licensed and actively maintained ( > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is wrong > > with > > > > > > > > > > > > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka > > > > github.com > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. > > It > > > > makes it easy to produce and consume messages, view the state of the > > > > cluster, and ... > > > > > > > > >> that? > > > > >> > > You > > > > >> > > > mentioned that there was some compatibility concern, but > > > > >> compatibility > > > > >> > > has > > > > >> > > > to do with the consumer protocol guarantees not the repo the > > > code > > > > >>is > > > > >> > in, > > > > >> > > > right? Not sure that concern makes sense. > > > > >> > > > > > > > >> > > > We could argue for adding pretty much anything and > everything > > in > > > > >>the > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that would > > make > > > > >>the > > > > >> > > project > > > > >> > > > more agile. > > > > >> > > > > > > > >> > > > -Jay > > > > >> > > > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar < > > > > >> manikumar.re...@gmail.com > > > > >> > > > > > > >> > > > wrote: > > > > >> > > > > > > > >> > > > > Hi Kafka Devs, > > > > >> > > > > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka > > Repository. > > > > >> > > > > > > > > >> > > > > There are already open-source alternatives are available. > > But > > > > >>we > > > > >> > would > > > > >> > > > > like to add REST server that > > > > >> > > > > many users ask for under Apache Kafka repo. Many data > Infra > > > > >>tools > > > > >> > comes > > > > >> > > > up > > > > >> > > > > with Rest Interface. > > > > >> > > > > It is useful to have inbuilt Rest API support for Produce, > > > > >>Consume > > > > >> > > > messages > > > > >> > > > > and admin interface for > > > > >> > > > > integrating with external management and provisioning > > > tools.This > > > > >> will > > > > >> > > > also > > > > >> > > > > allow the maintenance of > > > > >> > > > > REST server and adding new features makes it easy because > > > apache > > > > >> > > > community. > > > > >> > > > > > > > > >> > > > > The KIP wiki is the following: > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > >> > > > > 80%3A+Kafka+Rest+Server > > > > >> > > > > > > > > >> > > > > Your comments and feedback are welcome. > > > > >> > > > > > > > > >> > > > > Thanks, > > > > >> > > > > Manikumar > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > -- > > > > >> > Thanks, > > > > >> > Neha > > > > >> > > > > > >> > > > > The information contained in this email is strictly confidential and > > for > > > > the use of the addressee only, unless otherwise indicated. If you are > > not > > > > the intended recipient, please do not read, copy, use or disclose to > > > others > > > > this message or any attachment. Please also notify the sender by > > replying > > > > to this email or by telephone (+44(020 7896 0011) and then delete the > > > email > > > > and any copies of it. Opinions, conclusion (etc) that do not relate > to > > > the > > > > official business of this company shall be understood as neither > given > > > nor > > > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > > > registered in England and Wales, company number 04008957) and IG > Index > > > > Limited (a company registered in England and Wales, company number > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > Hill, > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and > > IG > > > > Index Limited (register number 114059) are authorised and regulated > by > > > the > > > > Financial Conduct Authority. > > > > > > > > > >
