[DISCUSS] KIP-80: Kafka REST Server
Hi Kafka Devs, I created KIP-80 to add Kafka REST Server to Kafka Repository. There are already open-source alternatives are available. But we would like to add REST server that many users ask for under Apache Kafka repo. Many data Infra tools comes up with Rest Interface. It is useful to have inbuilt Rest API support for Produce, Consume messages and admin interface for integrating with external management and provisioning tools.This will also allow the maintenance of REST server and adding new features makes it easy because apache community. The KIP wiki is the following: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 80%3A+Kafka+Rest+Server Your comments and feedback are welcome. Thanks, Manikumar
Re: [DISCUSS] KIP-80: Kafka REST Server
Thanks Mani for the KIP. I'll go over it and add my thoughts on this thread. On Wed, Sep 28, 2016 at 12:04 AM Manikumar wrote: > Hi Kafka Devs, > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > There are already open-source alternatives are available. But we would > like to add REST server that > many users ask for under Apache Kafka repo. Many data Infra tools comes up > with Rest Interface. > It is useful to have inbuilt Rest API support for Produce, Consume messages > and admin interface for > integrating with external management and provisioning tools.This will also > allow the maintenance of > REST server and adding new features makes it easy because apache community. > > The KIP wiki is the following: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 80%3A+Kafka+Rest+Server > > Your comments and feedback are welcome. > > Thanks, > Manikumar >
Re: [DISCUSS] KIP-80: Kafka REST Server
Hey guys, There's already a REST interface maintained as a separate project--it's open source and apache licensed and actively maintained ( https://github.com/confluentinc/kafka-rest). What is wrong with that? You mentioned that there was some compatibility concern, but compatibility has to do with the consumer protocol guarantees not the repo the code is in, right? Not sure that concern makes sense. We could argue for adding pretty much anything and everything in the ecosystem page in Kafka itself but I'm not sure that would make the project more agile. -Jay On Wed, Sep 28, 2016 at 12:04 AM, Manikumar wrote: > Hi Kafka Devs, > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > There are already open-source alternatives are available. But we would > like to add REST server that > many users ask for under Apache Kafka repo. Many data Infra tools comes up > with Rest Interface. > It is useful to have inbuilt Rest API support for Produce, Consume messages > and admin interface for > integrating with external management and provisioning tools.This will also > allow the maintenance of > REST server and adding new features makes it easy because apache community. > > The KIP wiki is the following: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 80%3A+Kafka+Rest+Server > > Your comments and feedback are welcome. > > Thanks, > Manikumar >
Re: [DISCUSS] KIP-80: Kafka REST Server
Hi Jay, Thanks for your reply. I agree that we can not add all the clients/tools available in ecosystem page to Kafka repo itself. But we feel REST Interface is different from other clients/tools. Since any language that can work with HTTP can easily integrate with this interface, Having an "official" REST interface helps user community. This also helps us to integrate well with external management and provisioning tools. Apache Kafka release with Java clients + REST interface is sufficient for most of the user deployments/requirements. This helps users to deal with less number of distributions/builds. Thanks, Manikumar On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps wrote: > Hey guys, > > There's already a REST interface maintained as a separate project--it's > open source and apache licensed and actively maintained ( > https://github.com/confluentinc/kafka-rest). What is wrong with that? You > mentioned that there was some compatibility concern, but compatibility has > to do with the consumer protocol guarantees not the repo the code is in, > right? Not sure that concern makes sense. > > We could argue for adding pretty much anything and everything in the > ecosystem page in Kafka itself but I'm not sure that would make the project > more agile. > > -Jay > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar > wrote: > > > Hi Kafka Devs, > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > There are already open-source alternatives are available. But we would > > like to add REST server that > > many users ask for under Apache Kafka repo. Many data Infra tools comes > up > > with Rest Interface. > > It is useful to have inbuilt Rest API support for Produce, Consume > messages > > and admin interface for > > integrating with external management and provisioning tools.This will > also > > allow the maintenance of > > REST server and adding new features makes it easy because apache > community. > > > > The KIP wiki is the following: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 80%3A+Kafka+Rest+Server > > > > Your comments and feedback are welcome. > > > > Thanks, > > Manikumar > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
Manikumar, Thanks for sharing the proposal. I think there are 2 parts to this discussion - 1. Should we rewrite a REST proxy given that there is a feature-complete, open-source and actively maintained REST proxy in the community? 2. Does adding a REST proxy to Apache Kafka make us more agile and maintain the high-quality experience that Kafka users have today? For 1, I don't think there is value in giving in to the NIH syndrome and reinventing the wheel. What I'm looking for is a detailed comparison of the gaps and why those can't be improved in the REST proxy that already exists and is actively maintained. For example, we depend on zkClient and have found as well as fixed several bugs by working closely with the people who maintain zkClient. This should be possible for REST proxy as well, right? For 2, I'd like us to review our history of expanding the surface area to add more clients in the past. Here is a summary - - This doesn't materially have an impact on expanding the usability of Kafka. In my experience, REST proxy + Java clients only cover ~50% of all Kafka users, and maybe 10% of those are the ones who will use the REST proxy. The remaining 50% are non-java client users (C, python, go, node etc). - People are a lot more excited about promising to contribute while adding the surface area but not on an ongoing basis down the line. - More surface area means more work to keep things consistent. Failure to do that has, in fact, hurt the user experience. - More surface area hurts agility. We want to do a few things really well as well as be agile to be able to build on our core competency. On Sat, Oct 1, 2016 at 5:38 AM Manikumar wrote: > Hi Jay, > > Thanks for your reply. > > I agree that we can not add all the clients/tools available in ecosystem > page to Kafka repo itself. But we feel REST Interface is different from > other clients/tools. Since any language that can work with HTTP can > easily integrate with this interface, Having an "official" REST > interface helps user community. This also helps us to integrate well > with external management and provisioning tools. Apache Kafka release > with Java clients + REST interface is sufficient for most of the user > deployments/requirements. This helps users to deal with less number > of distributions/builds. > > Thanks, > Manikumar > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps wrote: > > > Hey guys, > > > > There's already a REST interface maintained as a separate project--it's > > open source and apache licensed and actively maintained ( > > https://github.com/confluentinc/kafka-rest). What is wrong with that? > You > > mentioned that there was some compatibility concern, but compatibility > has > > to do with the consumer protocol guarantees not the repo the code is in, > > right? Not sure that concern makes sense. > > > > We could argue for adding pretty much anything and everything in the > > ecosystem page in Kafka itself but I'm not sure that would make the > project > > more agile. > > > > -Jay > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar > > wrote: > > > > > Hi Kafka Devs, > > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > > > There are already open-source alternatives are available. But we would > > > like to add REST server that > > > many users ask for under Apache Kafka repo. Many data Infra tools comes > > up > > > with Rest Interface. > > > It is useful to have inbuilt Rest API support for Produce, Consume > > messages > > > and admin interface for > > > integrating with external management and provisioning tools.This will > > also > > > allow the maintenance of > > > REST server and adding new features makes it easy because apache > > community. > > > > > > The KIP wiki is the following: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 80%3A+Kafka+Rest+Server > > > > > > Your comments and feedback are welcome. > > > > > > Thanks, > > > Manikumar > > > > > > -- Thanks, Neha
Re: [DISCUSS] KIP-80: Kafka REST Server
Neha & Jay, We did look at the open source alternatives. Our concern is what's the patch acceptance and adding features/ bug-fixes to the existing project under a Github (although it's licensed under Apache 2.0). It would be great if that project made available under Apache and driven by the community. Adding to the above, not all Kafka users are interested in using the Java client API, they would like to have simple REST API where they can code against using any language. I do believe this adds value to Apache Kafka in itself. "For 1, I don't think there is value in giving in to the NIH syndrome and reinventing the wheel. What I'm looking for is a detailed comparison of the gaps and why those can't be improved in the REST proxy that already exists and is actively maintained." We are not looking at this as NIH. What we are worried about is a project that's not maintained in a community. So the process of accepting patches and priorities is not clear, and it's not developed in Apache community. Not only that, existing REST API project doesn't support new client API and hence there is no security support either. We don't know the timeline when that's made available. We would like to add admin functionality into the REST API. So the Roadmap of that project is not driven by Apache. "This doesn't materially have an impact on expanding the usability of Kafka. In my experience, REST proxy + Java clients only cover ~50% of all Kafka users, and maybe 10% of those are the ones who will use the REST proxy. The remaining 50% are non-java client users (C, python, go, node etc)." REST API is most often asked feature in my interactions with Kafka users. In an organization, There will be independent teams who will write their Kafka clients using different language libraries available today, and there is no way to standardize this. Instead of supporting several different client libraries users will be interested in using a REST API server. The need for a REST API will only increase as more and more users start using Kafka. "More surface area means more work to keep things consistent. Failure to do that has, in fact, hurt the user experience." Having myriad Kafka client GitHub projects that support different languages hurts the user experience and pushes burden to maintain these libraries. REST API is a simple code base that uses existing java client libraries to make life easier for the users. Thanks, Harsha On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede wrote: > Manikumar, > > Thanks for sharing the proposal. I think there are 2 parts to this > discussion - > > 1. Should we rewrite a REST proxy given that there is a feature-complete, > open-source and actively maintained REST proxy in the community? > 2. Does adding a REST proxy to Apache Kafka make us more agile and maintain > the high-quality experience that Kafka users have today? > > For 1, I don't think there is value in giving in to the NIH syndrome and > reinventing the wheel. What I'm looking for is a detailed comparison of the > gaps and why those can't be improved in the REST proxy that already exists > and is actively maintained. For example, we depend on zkClient and have > found as well as fixed several bugs by working closely with the people who > maintain zkClient. This should be possible for REST proxy as well, right? > > For 2, I'd like us to review our history of expanding the surface area to > add more clients in the past. Here is a summary - > >- This doesn't materially have an impact on expanding the usability of >Kafka. In my experience, REST proxy + Java clients only cover ~50% of > all >Kafka users, and maybe 10% of those are the ones who will use the REST >proxy. The remaining 50% are non-java client users (C, python, go, node >etc). >- People are a lot more excited about promising to contribute while >adding the surface area but not on an ongoing basis down the line. >- More surface area means more work to keep things consistent. Failure >to do that has, in fact, hurt the user experience. >- More surface area hurts agility. We want to do a few things really >well as well as be agile to be able to build on our core competency. > > > On Sat, Oct 1, 2016 at 5:38 AM Manikumar > wrote: > > > Hi Jay, > > > > Thanks for your reply. > > > > I agree that we can not add all the clients/tools available in ecosystem > > page to Kafka repo itself. But we feel REST Interface is different from > > other clients/tools. Since any language that can work with HTTP can > > easily integrate with this interface, Having an "official" REST > > interface helps user community. This also helps us to integrate well > > with external management and provisioning tools. Apache Kafka release > > with Java clients + REST interface is sufficient for most of the user > > deployments/requirements. This helps users to deal with less number > > of distributions/builds. > > > > Thanks, > > Manik
Re: [DISCUSS] KIP-80: Kafka REST Server
+ 1 to rest client (don't mind if it's the current confluent version or something else) We are a multi language company and the quality of the other clients that are not Java are really hit and miss. A rest endpoint a user could just pump messages into or subscribe to would be amazing. On Sun, Oct 2, 2016 at 9:23 PM, Harsha Chintalapani wrote: > Neha & Jay, > We did look at the open source alternatives. Our concern > is what's the patch acceptance and adding features/ bug-fixes to the > existing project under a Github (although it's licensed under Apache 2.0). > It would be great if that project made available under Apache and driven by > the community. Adding to the above, not all Kafka users are interested in > using the Java client API, they would like to have simple REST API where > they can code against using any language. I do believe this adds value to > Apache Kafka in itself. > > "For 1, I don't think there is value in giving in to the NIH syndrome and > reinventing the wheel. What I'm looking for is a detailed comparison of the > gaps and why those can't be improved in the REST proxy that already exists > and is actively maintained." > > We are not looking at this as NIH. What we are worried about is a project > that's not maintained in a community. So the process of accepting patches > and priorities is not clear, and it's not developed in Apache community. > Not only that, existing REST API project doesn't support new client API and > hence there is no security support either. > We don't know the timeline when that's made available. We would like to add > admin functionality into the REST API. So the Roadmap of that project is > not driven by Apache. > > > "This doesn't materially have an impact on expanding the usability of >Kafka. In my experience, REST proxy + Java clients only cover ~50% of > all >Kafka users, and maybe 10% of those are the ones who will use the REST >proxy. The remaining 50% are non-java client users (C, python, go, node >etc)." > > REST API is most often asked feature in my interactions with Kafka users. > In an organization, There will be independent teams who will write their > Kafka clients using different language libraries available today, and > there is no way to standardize this. Instead of supporting several > different client libraries users will be interested in using a REST API > server. The need for a REST API will only increase as more and more users > start using Kafka. > > "More surface area means more work to keep things consistent. Failure >to do that has, in fact, hurt the user experience." > Having myriad Kafka client GitHub projects that support different languages > hurts the user experience and pushes burden to maintain these libraries. > REST API is a simple code base that uses existing java client libraries to > make life easier for the users. > > Thanks, > Harsha > > On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede wrote: > > > Manikumar, > > > > Thanks for sharing the proposal. I think there are 2 parts to this > > discussion - > > > > 1. Should we rewrite a REST proxy given that there is a feature-complete, > > open-source and actively maintained REST proxy in the community? > > 2. Does adding a REST proxy to Apache Kafka make us more agile and > maintain > > the high-quality experience that Kafka users have today? > > > > For 1, I don't think there is value in giving in to the NIH syndrome and > > reinventing the wheel. What I'm looking for is a detailed comparison of > the > > gaps and why those can't be improved in the REST proxy that already > exists > > and is actively maintained. For example, we depend on zkClient and have > > found as well as fixed several bugs by working closely with the people > who > > maintain zkClient. This should be possible for REST proxy as well, right? > > > > For 2, I'd like us to review our history of expanding the surface area to > > add more clients in the past. Here is a summary - > > > >- This doesn't materially have an impact on expanding the usability of > >Kafka. In my experience, REST proxy + Java clients only cover ~50% of > > all > >Kafka users, and maybe 10% of those are the ones who will use the REST > >proxy. The remaining 50% are non-java client users (C, python, go, > node > >etc). > >- People are a lot more excited about promising to contribute while > >adding the surface area but not on an ongoing basis down the line. > >- More surface area means more work to keep things consistent. Failure > >to do that has, in fact, hurt the user experience. > >- More surface area hurts agility. We want to do a few things really > >well as well as be agile to be able to build on our core competency. > > > > > > On Sat, Oct 1, 2016 at 5:38 AM Manikumar > > wrote: > > > > > Hi Jay, > > > > > > Thanks for your reply. > > > > > > I agree that we can not add all the clients/tools available in > ecosystem > > > page to Kafka repo
Re: [DISCUSS] KIP-80: Kafka REST Server
Harsha thanks for opening the discussion on this KIP. While I understand he founding members' stand that the Kafka project can not expand its surface to a large number of clients, I strongly agree with your well explained points below and support your KIP. A REST API is not just on the same level as any client, it's a basic building block of open web technologies. It's the API that most of our first time user want to try out (or that would be users ask for and expect to be there). A REST API for Kafka and a robust server implementation under the open governance of the Apache community would be most welcome. +1 Edo -- Edoardo Comar IBM MessageHub IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU Harsha Chintalapani wrote on 02/10/2016 21:23:15: > From: Harsha Chintalapani > To: dev@kafka.apache.org > Date: 02/10/2016 21:23 > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > Neha & Jay, > We did look at the open source alternatives. Our concern > is what's the patch acceptance and adding features/ bug-fixes to the > existing project under a Github (although it's licensed under Apache 2.0). > It would be great if that project made available under Apache and driven by > the community. Adding to the above, not all Kafka users are interested in > using the Java client API, they would like to have simple REST API where > they can code against using any language. I do believe this adds value to > Apache Kafka in itself. > > "For 1, I don't think there is value in giving in to the NIH syndrome and > reinventing the wheel. What I'm looking for is a detailed comparison of the > gaps and why those can't be improved in the REST proxy that already exists > and is actively maintained." > > We are not looking at this as NIH. What we are worried about is a project > that's not maintained in a community. So the process of accepting patches > and priorities is not clear, and it's not developed in Apache community. > Not only that, existing REST API project doesn't support new client API and > hence there is no security support either. > We don't know the timeline when that's made available. We would like to add > admin functionality into the REST API. So the Roadmap of that project is > not driven by Apache. > > > "This doesn't materially have an impact on expanding the usability of >Kafka. In my experience, REST proxy + Java clients only cover ~50% of all >Kafka users, and maybe 10% of those are the ones who will use the REST >proxy. The remaining 50% are non-java client users (C, python, go, node >etc)." > > REST API is most often asked feature in my interactions with Kafka users. > In an organization, There will be independent teams who will write their > Kafka clients using different language libraries available today, and > there is no way to standardize this. Instead of supporting several > different client libraries users will be interested in using a REST API > server. The need for a REST API will only increase as more and more users > start using Kafka. > > "More surface area means more work to keep things consistent. Failure >to do that has, in fact, hurt the user experience." > Having myriad Kafka client GitHub projects that support different languages > hurts the user experience and pushes burden to maintain these libraries. > REST API is a simple code base that uses existing java client libraries to > make life easier for the users. > > Thanks, > Harsha > > On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede wrote: > > > Manikumar, > > > > Thanks for sharing the proposal. I think there are 2 parts to this > > discussion - > > > > 1. Should we rewrite a REST proxy given that there is a feature-complete, > > open-source and actively maintained REST proxy in the community? > > 2. Does adding a REST proxy to Apache Kafka make us more agile and maintain > > the high-quality experience that Kafka users have today? > > > > For 1, I don't think there is value in giving in to the NIH syndrome and > > reinventing the wheel. What I'm looking for is a detailed comparison of the > > gaps and why those can't be improved in the REST proxy that already exists > > and is actively maintained. For example, we depend on zkClient and have > > found as well as fixed several bugs by working closely with the people who > > maintain zkClient. This should be possible for REST proxy as well, right? > > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
I also think it would be good to have a REST client that can interact with the new management API's coming down the pipe. On Tue, Oct 4, 2016 at 10:35 PM, Edoardo Comar wrote: > Harsha > thanks for opening the discussion on this KIP. > > While I understand he founding members' stand that the Kafka project can > not expand its surface to a large number of clients, > I strongly agree with your well explained points below and support your > KIP. > > A REST API is not just on the same level as any client, it's a basic > building block of open web technologies. > It's the API that most of our first time user want to try out (or that > would be users ask for and expect to be there). > > A REST API for Kafka and a robust server implementation > under the open governance of the Apache community would be most welcome. > > +1 > > Edo > -- > Edoardo Comar > IBM MessageHub > > IBM United Kingdom Limited Registered in England and Wales with number > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 > 3AU > > Harsha Chintalapani wrote on 02/10/2016 21:23:15: > > > From: Harsha Chintalapani > > To: dev@kafka.apache.org > > Date: 02/10/2016 21:23 > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > Neha & Jay, > > We did look at the open source alternatives. Our > concern > > is what's the patch acceptance and adding features/ bug-fixes to the > > existing project under a Github (although it's licensed under Apache > 2.0). > > It would be great if that project made available under Apache and driven > by > > the community. Adding to the above, not all Kafka users are interested > in > > using the Java client API, they would like to have simple REST API where > > they can code against using any language. I do believe this adds value > to > > Apache Kafka in itself. > > > > "For 1, I don't think there is value in giving in to the NIH syndrome > and > > reinventing the wheel. What I'm looking for is a detailed comparison of > the > > gaps and why those can't be improved in the REST proxy that already > exists > > and is actively maintained." > > > > We are not looking at this as NIH. What we are worried about is a > project > > that's not maintained in a community. So the process of accepting > patches > > and priorities is not clear, and it's not developed in Apache community. > > Not only that, existing REST API project doesn't support new client API > and > > hence there is no security support either. > > We don't know the timeline when that's made available. We would like to > add > > admin functionality into the REST API. So the Roadmap of that project is > > not driven by Apache. > > > > > > "This doesn't materially have an impact on expanding the usability of > >Kafka. In my experience, REST proxy + Java clients only cover ~50% of > all > >Kafka users, and maybe 10% of those are the ones who will use the > REST > >proxy. The remaining 50% are non-java client users (C, python, go, > node > >etc)." > > > > REST API is most often asked feature in my interactions with Kafka > users. > > In an organization, There will be independent teams who will write their > > Kafka clients using different language libraries available today, and > > there is no way to standardize this. Instead of supporting several > > different client libraries users will be interested in using a REST API > > server. The need for a REST API will only increase as more and more > users > > start using Kafka. > > > > "More surface area means more work to keep things consistent. Failure > >to do that has, in fact, hurt the user experience." > > Having myriad Kafka client GitHub projects that support different > languages > > hurts the user experience and pushes burden to maintain these libraries. > > REST API is a simple code base that uses existing java client libraries > to > > make life easier for the users. > > > > Thanks, > > Harsha > > > > On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede wrote: > > > > > Manikumar, > > > > > > Thanks for sharing the proposal. I think there are 2 parts to this > > > discussion - > > > > > > 1. Should we rewrite a REST proxy given that there is a > feature-complete, > > > open-source and actively maintained REST proxy in the community? > > > 2.
Re: [DISCUSS] KIP-80: Kafka REST Server
I personally think it would be quite wasteful to re-implement the REST gateway just because that an actively-maintained piece of Apache-licensed software is not governed directly by the Apache Kafka community... While kafka-rest repo is owned by Confluent, the contributors including the main one are also part of the Apache Kafka community, so there is a chance to work this out. However, there are two valid concerns here that could be addressed, around community and accessibility: >> What we are worried about is a project >> that's not maintained in a community. So the process of accepting patches >> and priorities is not clear, and it's not developed in Apache community. >> Not only that, existing REST API project doesn't support new client API and >> hence there is no security support either. This might be easy to fix. Maybe Confluent / kafka-rest community can clarify that - what is their contribution policy, dev style, roadmap etc. If they want, they can make an effort to encourage participation from people outside Confluent (easily accept contributions, invite external commiters or have open dev process similar to Apache Kafka etc), as there is definitely seems to be some interest on the list. That might clear the community concern and help kafka-rest project (but that is a calculation Confluent will have to make). The other, independent, concern is that REST is something that is expected to be available out of the box with Kafka. I personally don't feel strongly about it (better use proper, efficient APIs from day one), though it is definitely way smaller than adding a stream processing engine to the project :) Again,the kafka-rest "community" could take steps to make it even easier to install, configure and run kafka-rest for new users on vanilla Apache Kafka (outside the Confluent platform), if they wish that (or welcome contributions to that end), but that is up to them. Finally, if after the above steps were taken there would still a strong desire to include a great rest gateway with Apache Kafka, I assume the community could hypothetically fork the existing kafka-rest into an Apache Kafka subproject and maintain it "within Apache" instead of implementing it from scratch (though I'm not a lawyer etc) - but I cannot imagine it happen without Confluent blessing, and I think that is likely much less optimal (pulling in other Confluent / Apache licensed dependencies) than having a separate external community around kafka-rest. Just my two cents, Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani wrote: > Neha & Jay, > We did look at the open source alternatives. Our concern > is what's the patch acceptance and adding features/ bug-fixes to the > existing project under a Github (although it's licensed under Apache 2.0). > It would be great if that project made available under Apache and driven by > the community. Adding to the above, not all Kafka users are interested in > using the Java client API, they would like to have simple REST API where > they can code against using any language. I do believe this adds value to > Apache Kafka in itself. > > "For 1, I don't think there is value in giving in to the NIH syndrome and > reinventing the wheel. What I'm looking for is a detailed comparison of the > gaps and why those can't be improved in the REST proxy that already exists > and is actively maintained." > > We are not looking at this as NIH. What we are worried about is a project > that's not maintained in a community. So the process of accepting patches > and priorities is not clear, and it's not developed in Apache community. > Not only that, existing REST API project doesn't support new client API and > hence there is no security support either. > We don't know the timeline when that's made available. We would like to add > admin functionality into the REST API. So the Roadmap of that project is > not driven by Apache. > > > "This doesn't materially have an impact on expanding the usability of >Kafka. In my experience, REST proxy + Java clients only cover ~50% of > all >Kafka users, and maybe 10% of those are the ones who will use the REST >proxy. The remaining 50% are non-java client users (C, python, go, node >etc)." > > REST API is most often asked feature in my interactions with Kafka users. > In an organization, There will be independent teams who will write their > Kafka clients using different language libraries available today, and > there is no way to standardize this. Instead of supporting several > different client libraries users will be interested in using a REST API > server. The need for a REST API will only increase as more and more users > start using Kafka. > > "More surface area means more work to keep things consistent. Failure >to do that has, in fact, hurt the user experience." > Having myriad Kafka client GitHub projects that suppo
Re: [DISCUSS] KIP-80: Kafka REST Server
Hi Manikumar, I agree totally agree that REST is important. What I don't understand is why we'd duplicate the existing REST interface inside the Kafka project. That seems to needlessly fragment things. -Jay On Sat, Oct 1, 2016 at 5:38 AM, Manikumar wrote: > Hi Jay, > > Thanks for your reply. > > I agree that we can not add all the clients/tools available in ecosystem > page to Kafka repo itself. But we feel REST Interface is different from > other clients/tools. Since any language that can work with HTTP can > easily integrate with this interface, Having an "official" REST > interface helps user community. This also helps us to integrate well > with external management and provisioning tools. Apache Kafka release > with Java clients + REST interface is sufficient for most of the user > deployments/requirements. This helps users to deal with less number > of distributions/builds. > > Thanks, > Manikumar > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps wrote: > > > Hey guys, > > > > There's already a REST interface maintained as a separate project--it's > > open source and apache licensed and actively maintained ( > > https://github.com/confluentinc/kafka-rest). What is wrong with that? > You > > mentioned that there was some compatibility concern, but compatibility > has > > to do with the consumer protocol guarantees not the repo the code is in, > > right? Not sure that concern makes sense. > > > > We could argue for adding pretty much anything and everything in the > > ecosystem page in Kafka itself but I'm not sure that would make the > project > > more agile. > > > > -Jay > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar > > wrote: > > > > > Hi Kafka Devs, > > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > > > There are already open-source alternatives are available. But we would > > > like to add REST server that > > > many users ask for under Apache Kafka repo. Many data Infra tools comes > > up > > > with Rest Interface. > > > It is useful to have inbuilt Rest API support for Produce, Consume > > messages > > > and admin interface for > > > integrating with external management and provisioning tools.This will > > also > > > allow the maintenance of > > > REST server and adding new features makes it easy because apache > > community. > > > > > > The KIP wiki is the following: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 80%3A+Kafka+Rest+Server > > > > > > Your comments and feedback are welcome. > > > > > > Thanks, > > > Manikumar > > > > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
Ofir, " personally think it would be quite wasteful to re-implement the REST gateway just because that an actively-maintained piece of Apache-licensed software is not governed directly by the Apache Kafka community... While kafka-rest repo is owned by Confluent, the contributors including the main one are also part of the Apache Kafka community, so there is a chance to work this out." It is important for the code to be under Apache guidelines for contributions so that it is driven by the community. As part of Kafka community, we would like to improve the project and add new features. If kafka-rest repo can be donated to Apache Kafka project we will be more than happy to welcome that and add our features on top of it. Thanks, harsha On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps wrote: Hi Manikumar, I agree totally agree that REST is important. What I don't understand is why we'd duplicate the existing REST interface inside the Kafka project. That seems to needlessly fragment things. -Jay On Sat, Oct 1, 2016 at 5:38 AM, Manikumar wrote: > Hi Jay, > > Thanks for your reply. > > I agree that we can not add all the clients/tools available in ecosystem > page to Kafka repo itself. But we feel REST Interface is different from > other clients/tools. Since any language that can work with HTTP can > easily integrate with this interface, Having an "official" REST > interface helps user community. This also helps us to integrate well > with external management and provisioning tools. Apache Kafka release > with Java clients + REST interface is sufficient for most of the user > deployments/requirements. This helps users to deal with less number > of distributions/builds. > > Thanks, > Manikumar > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps wrote: > > > Hey guys, > > > > There's already a REST interface maintained as a separate project--it's > > open source and apache licensed and actively maintained ( > > https://github.com/confluentinc/kafka-rest). What is wrong with that? > You > > mentioned that there was some compatibility concern, but compatibility > has > > to do with the consumer protocol guarantees not the repo the code is in, > > right? Not sure that concern makes sense. > > > > We could argue for adding pretty much anything and everything in the > > ecosystem page in Kafka itself but I'm not sure that would make the > project > > more agile. > > > > -Jay > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar > > wrote: > > > > > Hi Kafka Devs, > > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > > > There are already open-source alternatives are available. But we would > > > like to add REST server that > > > many users ask for under Apache Kafka repo. Many data Infra tools comes > > up > > > with Rest Interface. > > > It is useful to have inbuilt Rest API support for Produce, Consume > > messages > > > and admin interface for > > > integrating with external management and provisioning tools.This will > > also > > > allow the maintenance of > > > REST server and adding new features makes it easy because apache > > community. > > > > > > The KIP wiki is the following: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 80%3A+Kafka+Rest+Server > > > > > > Your comments and feedback are welcome. > > > > > > Thanks, > > > Manikumar > > > > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
Ofir, … " personally think it would be quite wasteful to re-implement the REST gateway just because that an actively-maintained piece of Apache-licensed software is not governed directly by the Apache Kafka community... While kafka-rest repo is owned by Confluent, the contributors including the main one are also part of the Apache Kafka community, so there is a chance to work this out." It is important for the code to be under Apache guidelines for contributions so that it is driven by the community. As part of Kafka community, we would like to improve the project and add new features. If kafka-rest repo can be donated to Apache Kafka project we will be more than happy to welcome that and add our features on top of it. Thanks, harsha On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps wrote: > Hi Manikumar, > > I agree totally agree that REST is important. What I don't understand is > why we'd duplicate the existing REST interface inside the Kafka project. > That seems to needlessly fragment things. > > -Jay > > On Sat, Oct 1, 2016 at 5:38 AM, Manikumar > wrote: > > > Hi Jay, > > > > Thanks for your reply. > > > > I agree that we can not add all the clients/tools available in ecosystem > > page to Kafka repo itself. But we feel REST Interface is different from > > other clients/tools. Since any language that can work with HTTP can > > easily integrate with this interface, Having an "official" REST > > interface helps user community. This also helps us to integrate well > > with external management and provisioning tools. Apache Kafka release > > with Java clients + REST interface is sufficient for most of the user > > deployments/requirements. This helps users to deal with less number > > of distributions/builds. > > > > Thanks, > > Manikumar > > > > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps wrote: > > > > > Hey guys, > > > > > > There's already a REST interface maintained as a separate project--it's > > > open source and apache licensed and actively maintained ( > > > https://github.com/confluentinc/kafka-rest). What is wrong with that? > > You > > > mentioned that there was some compatibility concern, but compatibility > > has > > > to do with the consumer protocol guarantees not the repo the code is > in, > > > right? Not sure that concern makes sense. > > > > > > We could argue for adding pretty much anything and everything in the > > > ecosystem page in Kafka itself but I'm not sure that would make the > > project > > > more agile. > > > > > > -Jay > > > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar > > > > wrote: > > > > > > > Hi Kafka Devs, > > > > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > > > > > There are already open-source alternatives are available. But we > would > > > > like to add REST server that > > > > many users ask for under Apache Kafka repo. Many data Infra tools > comes > > > up > > > > with Rest Interface. > > > > It is useful to have inbuilt Rest API support for Produce, Consume > > > messages > > > > and admin interface for > > > > integrating with external management and provisioning tools.This will > > > also > > > > allow the maintenance of > > > > REST server and adding new features makes it easy because apache > > > community. > > > > > > > > The KIP wiki is the following: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > 80%3A+Kafka+Rest+Server > > > > > > > > Your comments and feedback are welcome. > > > > > > > > Thanks, > > > > Manikumar > > > > > > > > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
Harsha/Mani, I completely agree that adding admin API support and security are important features for the Kafka REST proxy. Luckily the roadmap items that you mentioned as being important for a Kafka REST proxy server are exactly the ones the community working on this REST proxy want to add to it :-) But I haven't seen any community emails or patches being submitted by you guys, so I'm wondering why you are concerned about whether the community is open to accepting patches or not. Does that make sense? On Fri, Oct 7, 2016 at 12:03 PM Harsha Chintalapani wrote: > Ofir, > … > " personally think it would be quite wasteful to re-implement the REST > gateway just because that an actively-maintained piece of Apache-licensed > software is not governed directly by the Apache Kafka community... While > kafka-rest repo is owned by Confluent, the contributors including the main > one are also part of the Apache Kafka community, so there is a chance to > work this out." > It is important for the code to be under Apache guidelines for > contributions so that it is driven by the community. > As part of Kafka community, we would like to improve the project and add > new features. If kafka-rest repo can be donated to Apache Kafka project > we will be more than happy to welcome that and add our features on top of > it. > > Thanks, > harsha > > On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps wrote: > > > Hi Manikumar, > > > > I agree totally agree that REST is important. What I don't understand is > > why we'd duplicate the existing REST interface inside the Kafka project. > > That seems to needlessly fragment things. > > > > -Jay > > > > On Sat, Oct 1, 2016 at 5:38 AM, Manikumar > > wrote: > > > > > Hi Jay, > > > > > > Thanks for your reply. > > > > > > I agree that we can not add all the clients/tools available in > ecosystem > > > page to Kafka repo itself. But we feel REST Interface is different from > > > other clients/tools. Since any language that can work with HTTP can > > > easily integrate with this interface, Having an "official" REST > > > interface helps user community. This also helps us to integrate well > > > with external management and provisioning tools. Apache Kafka release > > > with Java clients + REST interface is sufficient for most of the user > > > deployments/requirements. This helps users to deal with less number > > > of distributions/builds. > > > > > > Thanks, > > > Manikumar > > > > > > > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps wrote: > > > > > > > Hey guys, > > > > > > > > There's already a REST interface maintained as a separate > project--it's > > > > open source and apache licensed and actively maintained ( > > > > https://github.com/confluentinc/kafka-rest). What is wrong with > that? > > > You > > > > mentioned that there was some compatibility concern, but > compatibility > > > has > > > > to do with the consumer protocol guarantees not the repo the code is > > in, > > > > right? Not sure that concern makes sense. > > > > > > > > We could argue for adding pretty much anything and everything in the > > > > ecosystem page in Kafka itself but I'm not sure that would make the > > > project > > > > more agile. > > > > > > > > -Jay > > > > > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar < > manikumar.re...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi Kafka Devs, > > > > > > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > > > > > > > There are already open-source alternatives are available. But we > > would > > > > > like to add REST server that > > > > > many users ask for under Apache Kafka repo. Many data Infra tools > > comes > > > > up > > > > > with Rest Interface. > > > > > It is useful to have inbuilt Rest API support for Produce, Consume > > > > messages > > > > > and admin interface for > > > > > integrating with external management and provisioning tools.This > will > > > > also > > > > > allow the maintenance of > > > > > REST server and adding new features makes it easy because apache > > > > community. > > > > > > > > > > The KIP wiki is the following: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > 80%3A+Kafka+Rest+Server > > > > > > > > > > Your comments and feedback are welcome. > > > > > > > > > > Thanks, > > > > > Manikumar > > > > > > > > > > > > > > > -- Thanks, Neha
Re: [DISCUSS] KIP-80: Kafka REST Server
ASF already gives us a clear framework and governance model for community development. This is already understood by the people contributing to Apache Kafka project, and they are the same people who want to contribute to the REST server capability as well. Everyone is in agreement on the need for collaborating on this effort. So why not contribute the code to Apache Kafka. This will help avoid duplication of effort and forks that may crop up, hugely benefitting the user community. This will also avoid having to define a process similar to ASF on a GitHub project and instead there is a single community with clear understanding community process as defined in ASF. As others have said, this is an important capability for Apache Kafka. It is worth maintaining this as a part of the project. Regards, Suresh On 10/6/16, 8:32 AM, "Ofir Manor" wrote: >I personally think it would be quite wasteful to re-implement the REST >gateway just because that an actively-maintained piece of Apache-licensed >software is not governed directly by the Apache Kafka community... While >kafka-rest repo is owned by Confluent, the contributors including the main >one are also part of the Apache Kafka community, so there is a chance to >work this out. > >However, there are two valid concerns here that could be addressed, around >community and accessibility: >>> What we are worried about is a project >>> that's not maintained in a community. So the process of accepting >>>patches >>> and priorities is not clear, and it's not developed in Apache >>>community. >>> Not only that, existing REST API project doesn't support new client API >and >>> hence there is no security support either. > >This might be easy to fix. Maybe Confluent / kafka-rest community can >clarify that - what is their contribution policy, dev style, roadmap etc. >If they want, they can make an effort to encourage participation from >people outside Confluent (easily accept contributions, invite external >commiters or have open dev process similar to Apache Kafka etc), as there >is definitely seems to be some interest on the list. That might clear the >community concern and help kafka-rest project (but that is a calculation >Confluent will have to make). > >The other, independent, concern is that REST is something that is expected >to be available out of the box with Kafka. I personally don't feel >strongly >about it (better use proper, efficient APIs from day one), though it is >definitely way smaller than adding a stream processing engine to the >project :) >Again,the kafka-rest "community" could take steps to make it even easier >to >install, configure and run kafka-rest for new users on vanilla Apache >Kafka >(outside the Confluent platform), if they wish that (or welcome >contributions to that end), but that is up to them. >Finally, if after the above steps were taken there would still a strong >desire to include a great rest gateway with Apache Kafka, I assume the >community could hypothetically fork the existing kafka-rest into an Apache >Kafka subproject and maintain it "within Apache" instead of implementing >it >from scratch (though I'm not a lawyer etc) - but I cannot imagine it >happen >without Confluent blessing, and I think that is likely much less optimal >(pulling in other Confluent / Apache licensed dependencies) than having a >separate external community around kafka-rest. > > >Just my two cents, > > >Ofir Manor > >Co-Founder & CTO | Equalum > >Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani >wrote: > >> Neha & Jay, >> We did look at the open source alternatives. Our >>concern >> is what's the patch acceptance and adding features/ bug-fixes to the >> existing project under a Github (although it's licensed under Apache >>2.0). >> It would be great if that project made available under Apache and >>driven by >> the community. Adding to the above, not all Kafka users are interested >>in >> using the Java client API, they would like to have simple REST API where >> they can code against using any language. I do believe this adds value >>to >> Apache Kafka in itself. >> >> "For 1, I don't think there is value in giving in to the NIH syndrome >>and >> reinventing the wheel. What I'm looking for is a detailed comparison of >>the >> gaps and why those can't be improved in the REST proxy that already >>exists >> and is actively maintained." >> >> We are not looking at this as NIH. What we are worried about is a >>project >> that's not maintained in a community. So the process of accepting >>patches >> and priorities is not clear, and it's not developed in Apache community. >> Not only that, existing REST API project doesn't support new client API >>and >> hence there is no security support either. >> We don't know the timeline when that's made available. We would like to >>add >> admin functionality into the REST API. So the Roadmap of that project is >> not driven by Apache. >
Re: [DISCUSS] KIP-80: Kafka REST Server
There's a massive difference between the governance of Kafka and the governance of the REST proxy. In Kafka, there is a broad community of people contributing their opinions about future enhancements in the form of KIPs. There's some really deep consideration that goes into some of the trickier KIPs. There are people outside Confluent deeply knowledgeable in Kafka and building the reputations to become committers. I get the impression that the roadmap of Kafka is not really community-owned (what's the big feature for Kafka 0.11, for example), but the conveyor belt of smaller features in the form of KIPs works nicely. It's a good example of open-source working well. The equivalent for the REST proxy is basically issues on GitHub. The roadmap is less clear. There's not really a community properly engaged in the way that there is with Kafka. So, you could say that it's clear that fewer people are interested, but I think the whole governance thing is a big barrier to engagement. And it's looking like it's getting out of date. In technical terms, I can think of two big improvements to the REST proxy. First, it needs to use the new consumer API so that it's possible to secure connections between the REST proxy and Kafka. I don't care too much which method calls it uses actually uses to consume messages, but I do care that I cannot secure connections because of network security rules. Second, there's an affinity between a consumer and the instance of the REST proxy to which it first connected. Kafka itself avoids this kind of affinity for good reason, and in the name of availability the REST proxy should too. These are natural KIPs. I think it would be good to have the code for the REST proxy contributed to Apache Kafka so that it would be able to be developed in the same way. Andrew Schofield From: Suresh Srinivas Sent: 07 October 2016 22:41:52 To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-80: Kafka REST Server ASF already gives us a clear framework and governance model for community development. This is already understood by the people contributing to Apache Kafka project, and they are the same people who want to contribute to the REST server capability as well. Everyone is in agreement on the need for collaborating on this effort. So why not contribute the code to Apache Kafka. This will help avoid duplication of effort and forks that may crop up, hugely benefitting the user community. This will also avoid having to define a process similar to ASF on a GitHub project and instead there is a single community with clear understanding community process as defined in ASF. As others have said, this is an important capability for Apache Kafka. It is worth maintaining this as a part of the project. Regards, Suresh On 10/6/16, 8:32 AM, "Ofir Manor" wrote: >I personally think it would be quite wasteful to re-implement the REST >gateway just because that an actively-maintained piece of Apache-licensed >software is not governed directly by the Apache Kafka community... While >kafka-rest repo is owned by Confluent, the contributors including the main >one are also part of the Apache Kafka community, so there is a chance to >work this out. > >However, there are two valid concerns here that could be addressed, around >community and accessibility: >>> What we are worried about is a project >>> that's not maintained in a community. So the process of accepting >>>patches >>> and priorities is not clear, and it's not developed in Apache >>>community. >>> Not only that, existing REST API project doesn't support new client API >and >>> hence there is no security support either. > >This might be easy to fix. Maybe Confluent / kafka-rest community can >clarify that - what is their contribution policy, dev style, roadmap etc. >If they want, they can make an effort to encourage participation from >people outside Confluent (easily accept contributions, invite external >commiters or have open dev process similar to Apache Kafka etc), as there >is definitely seems to be some interest on the list. That might clear the >community concern and help kafka-rest project (but that is a calculation >Confluent will have to make). > >The other, independent, concern is that REST is something that is expected >to be available out of the box with Kafka. I personally don't feel >strongly >about it (better use proper, efficient APIs from day one), though it is >definitely way smaller than adding a stream processing engine to the >project :) >Again,the kafka-rest "community" could take steps to make it even easier >to >install, configure and run kafka-rest for new users on vanilla Apache >Kafka >(outside the Confluent platform), if they wish that (or welc
Re: [DISCUSS] KIP-80: Kafka REST Server
+1 I agree on the governance comments whole heartedly. Also i agree about the contribution comments made earlier in the thread, i personally am less likely to spend any of mine, or give project time within my internal projects to developers contributing to another commercial companies project even so technically open source, as then there is that commercial companies interest will always prevail and essentially can always have a final vote where disagreement. Im sure they never intend to, but there is that true reality. This is why we have community open source projects. I can find many different implementations now of a rest endpoint on GitHub, BitBucket etc. Each one has their benefits and disadvantages in their implementation. By making / providing one this would bring together these solutions, unifying those developers and also bringing the best of all. I understand the concern on the community burden adding/supporting more surface area for every client. But something like REST is universal and worthy to be owned by the community. Mike From: Andrew Schofield Sent: Saturday, October 8, 2016 1:19 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-80: Kafka REST Server There's a massive difference between the governance of Kafka and the governance of the REST proxy. In Kafka, there is a broad community of people contributing their opinions about future enhancements in the form of KIPs. There's some really deep consideration that goes into some of the trickier KIPs. There are people outside Confluent deeply knowledgeable in Kafka and building the reputations to become committers. I get the impression that the roadmap of Kafka is not really community-owned (what's the big feature for Kafka 0.11, for example), but the conveyor belt of smaller features in the form of KIPs works nicely. It's a good example of open-source working well. The equivalent for the REST proxy is basically issues on GitHub. The roadmap is less clear. There's not really a community properly engaged in the way that there is with Kafka. So, you could say that it's clear that fewer people are interested, but I think the whole governance thing is a big barrier to engagement. And it's looking like it's getting out of date. In technical terms, I can think of two big improvements to the REST proxy. First, it needs to use the new consumer API so that it's possible to secure connections between the REST proxy and Kafka. I don't care too much which method calls it uses actually uses to consume messages, but I do care that I cannot secure connections because of network security rules. Second, there's an affinity between a consumer and the instance of the REST proxy to which it first connected. Kafka itself avoids this kind of affinity for good reason, and in the name of availability the REST proxy should too. These are natural KIPs. I think it would be good to have the code for the REST proxy contributed to Apache Kafka so that it would be able to be developed in the same way. Andrew Schofield From: Suresh Srinivas Sent: 07 October 2016 22:41:52 To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-80: Kafka REST Server ASF already gives us a clear framework and governance model for community development. This is already understood by the people contributing to Apache Kafka project, and they are the same people who want to contribute to the REST server capability as well. Everyone is in agreement on the need for collaborating on this effort. So why not contribute the code to Apache Kafka. This will help avoid duplication of effort and forks that may crop up, hugely benefitting the user community. This will also avoid having to define a process similar to ASF on a GitHub project and instead there is a single community with clear understanding community process as defined in ASF. As others have said, this is an important capability for Apache Kafka. It is worth maintaining this as a part of the project. Regards, Suresh On 10/6/16, 8:32 AM, "Ofir Manor" wrote: >I personally think it would be quite wasteful to re-implement the REST >gateway just because that an actively-maintained piece of Apache-licensed >software is not governed directly by the Apache Kafka community... While >kafka-rest repo is owned by Confluent, the contributors including the main >one are also part of the Apache Kafka community, so there is a chance to >work this out. > >However, there are two valid concerns here that could be addressed, around >community and accessibility: >>> What we are worried about is a project >>> that's not maintained in a community. So the process of accepting >>>patches >>> and priorities is not clear, and it's not developed in Apache >>>community. >>> Not only that, existing REST API project
Re: [DISCUSS] KIP-80: Kafka REST Server
Neha, "But I haven't seen any community emails or patches being submitted by you guys, so I'm wondering why you are concerned about whether the community is open to accepting patches or not." I think you are talking about contributing patches to this repository right? https://github.com/confluentinc/kafka-rest . All I am saying the guidelines/governance model is not clear on the project and I guess its driven by opening a github issue request. Its the repository owned by confluent and as much I appreciate that the features we mentioned are in the roadmap and welcoming us to contribute to the project. It doesn't gurantee what we want to add in the furture will be in your roadmap. Hence the reason having it part of Kafka community will help a lot as other users can participate in the discussions. We are happy to drive any feature additions through KIPs which gives everyone a chance to participate and add to the discussions. Thanks, Harsha On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce wrote: > +1 > > I agree on the governance comments whole heartedly. > > Also i agree about the contribution comments made earlier in the thread, i > personally am less likely to spend any of mine, or give project time within > my internal projects to developers contributing to another commercial > companies project even so technically open source, as then there is that > commercial companies interest will always prevail and essentially can > always have a final vote where disagreement. Im sure they never intend to, > but there is that true reality. This is why we have community open source > projects. > > I can find many different implementations now of a rest endpoint on > GitHub, BitBucket etc. Each one has their benefits and disadvantages in > their implementation. By making / providing one this would bring together > these solutions, unifying those developers and also bringing the best of > all. > > I understand the concern on the community burden adding/supporting more > surface area for every client. But something like REST is universal and > worthy to be owned by the community. > > Mike > > > > From: Andrew Schofield > Sent: Saturday, October 8, 2016 1:19 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > There's a massive difference between the governance of Kafka and the > governance of the REST proxy. > > In Kafka, there is a broad community of people contributing their opinions > about future enhancements in the form of KIPs. There's some really deep > consideration that goes into some of the trickier KIPs. There are people > outside Confluent deeply knowledgeable in Kafka and building the > reputations to become committers. I get the impression that the roadmap of > Kafka is not really community-owned (what's the big feature for Kafka 0.11, > for example), but the conveyor belt of smaller features in the form of KIPs > works nicely. It's a good example of open-source working well. > > The equivalent for the REST proxy is basically issues on GitHub. The > roadmap is less clear. There's not really a community properly engaged in > the way that there is with Kafka. So, you could say that it's clear that > fewer people are interested, but I think the whole governance thing is a > big barrier to engagement. And it's looking like it's getting out of date. > > In technical terms, I can think of two big improvements to the REST proxy. > First, it needs to use the new consumer API so that it's possible to secure > connections between the REST proxy and Kafka. I don't care too much which > method calls it uses actually uses to consume messages, but I do care that > I cannot secure connections because of network security rules. Second, > there's an affinity between a consumer and the instance of the REST proxy > to which it first connected. Kafka itself avoids this kind of affinity for > good reason, and in the name of availability the REST proxy should too. > These are natural KIPs. > > I think it would be good to have the code for the REST proxy contributed > to Apache Kafka so that it would be able to be developed in the same way. > > Andrew Schofield > > From: Suresh Srinivas > Sent: 07 October 2016 22:41:52 > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > ASF already gives us a clear framework and governance model for community > development. This is already understood by the people contributing to > Apache Kafka project, and they are the same people who want to contribute > to the REST server capability as well. Everyone is in agreement on the > need for collaborating on this effort. So why n
Re: [DISCUSS] KIP-80: Kafka REST Server
osystem and if the project doesn't have a responsive community or wants to go in a different direction fork or recreate that work. Of course any person can choose whatever of these options they want. But from my point of view, option (3) has been the path of the community so far and I think it has been quite successful. -Jay On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani wrote: > Neha, > "But I haven't seen any community emails or patches being submitted by you > guys, so I'm wondering why you are concerned about whether the community is > open to accepting patches or not." > > I think you are talking about contributing patches to this repository > right? https://github.com/confluentinc/kafka-rest . All I am saying > the guidelines/governance model is not clear on the project and I guess its > driven by opening a github issue request. Its the repository owned by > confluent and as much I appreciate that the features we mentioned are in > the roadmap and welcoming us to contribute to the project. It doesn't > gurantee what we want to add in the furture will be in your roadmap. > > Hence the reason having it part of Kafka community will help a lot as other > users can participate in the discussions. We are happy to drive any > feature additions through KIPs which gives everyone a chance to participate > and add to the discussions. > > Thanks, > Harsha > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce > wrote: > > > +1 > > > > I agree on the governance comments whole heartedly. > > > > Also i agree about the contribution comments made earlier in the thread, > i > > personally am less likely to spend any of mine, or give project time > within > > my internal projects to developers contributing to another commercial > > companies project even so technically open source, as then there is that > > commercial companies interest will always prevail and essentially can > > always have a final vote where disagreement. Im sure they never intend > to, > > but there is that true reality. This is why we have community open source > > projects. > > > > I can find many different implementations now of a rest endpoint on > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in > > their implementation. By making / providing one this would bring together > > these solutions, unifying those developers and also bringing the best of > > all. > > > > I understand the concern on the community burden adding/supporting more > > surface area for every client. But something like REST is universal and > > worthy to be owned by the community. > > > > Mike > > > > > > > > From: Andrew Schofield > > Sent: Saturday, October 8, 2016 1:19 AM > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > There's a massive difference between the governance of Kafka and the > > governance of the REST proxy. > > > > In Kafka, there is a broad community of people contributing their > opinions > > about future enhancements in the form of KIPs. There's some really deep > > consideration that goes into some of the trickier KIPs. There are people > > outside Confluent deeply knowledgeable in Kafka and building the > > reputations to become committers. I get the impression that the roadmap > of > > Kafka is not really community-owned (what's the big feature for Kafka > 0.11, > > for example), but the conveyor belt of smaller features in the form of > KIPs > > works nicely. It's a good example of open-source working well. > > > > The equivalent for the REST proxy is basically issues on GitHub. The > > roadmap is less clear. There's not really a community properly engaged in > > the way that there is with Kafka. So, you could say that it's clear that > > fewer people are interested, but I think the whole governance thing is a > > big barrier to engagement. And it's looking like it's getting out of > date. > > > > In technical terms, I can think of two big improvements to the REST > proxy. > > First, it needs to use the new consumer API so that it's possible to > secure > > connections between the REST proxy and Kafka. I don't care too much which > > method calls it uses actually uses to consume messages, but I do care > that > > I cannot secure connections because of network security rules. Second, > > there's an affinity between a consumer and the instance of the REST proxy > > to which it first connected.
Re: [DISCUSS] KIP-80: Kafka REST Server
re additions through KIPs which gives everyone a chance to > participate > > and add to the discussions. > > > > Thanks, > > Harsha > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce > > wrote: > > > > > +1 > > > > > > I agree on the governance comments whole heartedly. > > > > > > Also i agree about the contribution comments made earlier in the > thread, > > i > > > personally am less likely to spend any of mine, or give project time > > within > > > my internal projects to developers contributing to another commercial > > > companies project even so technically open source, as then there is > that > > > commercial companies interest will always prevail and essentially can > > > always have a final vote where disagreement. Im sure they never intend > > to, > > > but there is that true reality. This is why we have community open > source > > > projects. > > > > > > I can find many different implementations now of a rest endpoint on > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in > > > their implementation. By making / providing one this would bring > together > > > these solutions, unifying those developers and also bringing the best > of > > > all. > > > > > > I understand the concern on the community burden adding/supporting more > > > surface area for every client. But something like REST is universal and > > > worthy to be owned by the community. > > > > > > Mike > > > > > > > > > > > > From: Andrew Schofield > > > Sent: Saturday, October 8, 2016 1:19 AM > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > There's a massive difference between the governance of Kafka and the > > > governance of the REST proxy. > > > > > > In Kafka, there is a broad community of people contributing their > > opinions > > > about future enhancements in the form of KIPs. There's some really deep > > > consideration that goes into some of the trickier KIPs. There are > people > > > outside Confluent deeply knowledgeable in Kafka and building the > > > reputations to become committers. I get the impression that the roadmap > > of > > > Kafka is not really community-owned (what's the big feature for Kafka > > 0.11, > > > for example), but the conveyor belt of smaller features in the form of > > KIPs > > > works nicely. It's a good example of open-source working well. > > > > > > The equivalent for the REST proxy is basically issues on GitHub. The > > > roadmap is less clear. There's not really a community properly engaged > in > > > the way that there is with Kafka. So, you could say that it's clear > that > > > fewer people are interested, but I think the whole governance thing > is a > > > big barrier to engagement. And it's looking like it's getting out of > > date. > > > > > > In technical terms, I can think of two big improvements to the REST > > proxy. > > > First, it needs to use the new consumer API so that it's possible to > > secure > > > connections between the REST proxy and Kafka. I don't care too much > which > > > method calls it uses actually uses to consume messages, but I do care > > that > > > I cannot secure connections because of network security rules. Second, > > > there's an affinity between a consumer and the instance of the REST > proxy > > > to which it first connected. Kafka itself avoids this kind of affinity > > for > > > good reason, and in the name of availability the REST proxy should too. > > > These are natural KIPs. > > > > > > I think it would be good to have the code for the REST proxy > contributed > > > to Apache Kafka so that it would be able to be developed in the same > way. > > > > > > Andrew Schofield > > > > > > From: Suresh Srinivas > > > Sent: 07 October 2016 22:41:52 > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > ASF already gives us a clear framework and governance model for > community > > > development. This is already understood by the people contributing to > > > Apache Kafka project, and they are the same people who want to > contribute > >
Re: [DISCUSS] KIP-80: Kafka REST Server
rom my point of view, option (3) has been the path of the community so > far > > and I think it has been quite successful. > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani > > wrote: > > > > > Neha, > > > "But I haven't seen any community emails or patches being submitted by > > you > > > guys, so I'm wondering why you are concerned about whether the > community > > is > > > open to accepting patches or not." > > > > > > I think you are talking about contributing patches to this repository > > > right? https://github.com/confluentinc/kafka-rest . All I am saying > > > the guidelines/governance model is not clear on the project and I guess > > its > > > driven by opening a github issue request. Its the repository owned by > > > confluent and as much I appreciate that the features we mentioned are > in > > > the roadmap and welcoming us to contribute to the project. It doesn't > > > gurantee what we want to add in the furture will be in your roadmap. > > > > > > Hence the reason having it part of Kafka community will help a lot as > > other > > > users can participate in the discussions. We are happy to drive any > > > feature additions through KIPs which gives everyone a chance to > > participate > > > and add to the discussions. > > > > > > Thanks, > > > Harsha > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce > > > wrote: > > > > > > > +1 > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > Also i agree about the contribution comments made earlier in the > > thread, > > > i > > > > personally am less likely to spend any of mine, or give project time > > > within > > > > my internal projects to developers contributing to another commercial > > > > companies project even so technically open source, as then there is > > that > > > > commercial companies interest will always prevail and essentially can > > > > always have a final vote where disagreement. Im sure they never > intend > > > to, > > > > but there is that true reality. This is why we have community open > > source > > > > projects. > > > > > > > > I can find many different implementations now of a rest endpoint on > > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages > in > > > > their implementation. By making / providing one this would bring > > together > > > > these solutions, unifying those developers and also bringing the best > > of > > > > all. > > > > > > > > I understand the concern on the community burden adding/supporting > more > > > > surface area for every client. But something like REST is universal > and > > > > worthy to be owned by the community. > > > > > > > > Mike > > > > > > > > > > > > > > > > From: Andrew Schofield > > > > Sent: Saturday, October 8, 2016 1:19 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > There's a massive difference between the governance of Kafka and the > > > > governance of the REST proxy. > > > > > > > > In Kafka, there is a broad community of people contributing their > > > opinions > > > > about future enhancements in the form of KIPs. There's some really > deep > > > > consideration that goes into some of the trickier KIPs. There are > > people > > > > outside Confluent deeply knowledgeable in Kafka and building the > > > > reputations to become committers. I get the impression that the > roadmap > > > of > > > > Kafka is not really community-owned (what's the big feature for Kafka > > > 0.11, > > > > for example), but the conveyor belt of smaller features in the form > of > > > KIPs > > > > works nicely. It's a good example of open-source working well. > > > > > > > > The equivalent for the REST proxy is basically issues on GitHub. The > > > > roadmap is less clear. There's not really a community properly >
Re: [DISCUSS] KIP-80: Kafka REST Server
MQTT or AMQP which are technically much better suited to > > messaging > >and both are widely used for that. > >2. We can treat ecosystem projects that aren't top level Apache > projects > >as invalid and try to recreate them all as Apache projects. Honestly, > >though, if you go to the Kafka ecosystem page virtually none of the > most > >popular add-ons to Kafka are Apache projects. The most successful > > things in > >the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of > other > >clients, as well as the existing REST layer have succeeded at > developing > >communities that actively contribute and use these pieces and I don't > > know > >that that is a bad thing unless that community proves to be > uninclusive, > >unresponsive, or goes in a bad technical direction--and those are > > failure > >modes that all open source efforts face. > >3. We can do what I think makes the most sense and try to work with > the > >projects that exist in the ecosystem and if the project doesn't have a > >responsive community or wants to go in a different direction fork or > >recreate that work. > > > > Of course any person can choose whatever of these options they want. But > > from my point of view, option (3) has been the path of the community so > far > > and I think it has been quite successful. > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani > > wrote: > > > > > Neha, > > > "But I haven't seen any community emails or patches being submitted by > > you > > > guys, so I'm wondering why you are concerned about whether the > community > > is > > > open to accepting patches or not." > > > > > > I think you are talking about contributing patches to this repository > > > right? https://github.com/confluentinc/kafka-rest . All I am saying > > > the guidelines/governance model is not clear on the project and I guess > > its > > > driven by opening a github issue request. Its the repository owned by > > > confluent and as much I appreciate that the features we mentioned are > in > > > the roadmap and welcoming us to contribute to the project. It doesn't > > > gurantee what we want to add in the furture will be in your roadmap. > > > > > > Hence the reason having it part of Kafka community will help a lot as > > other > > > users can participate in the discussions. We are happy to drive any > > > feature additions through KIPs which gives everyone a chance to > > participate > > > and add to the discussions. > > > > > > Thanks, > > > Harsha > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce > > > wrote: > > > > > > > +1 > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > Also i agree about the contribution comments made earlier in the > > thread, > > > i > > > > personally am less likely to spend any of mine, or give project time > > > within > > > > my internal projects to developers contributing to another commercial > > > > companies project even so technically open source, as then there is > > that > > > > commercial companies interest will always prevail and essentially can > > > > always have a final vote where disagreement. Im sure they never > intend > > > to, > > > > but there is that true reality. This is why we have community open > > source > > > > projects. > > > > > > > > I can find many different implementations now of a rest endpoint on > > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages > in > > > > their implementation. By making / providing one this would bring > > together > > > > these solutions, unifying those developers and also bringing the best > > of > > > > all. > > > > > > > > I understand the concern on the community burden adding/supporting > more > > > > surface area for every client. But something like REST is universal > and > > > > worthy to be owned by the community. > > > > > > > > Mike > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
gt; is > > > > open to accepting patches or not." > > > > > > > > I think you are talking about contributing patches to this repository > > > > right? https://github.com/confluentinc/kafka-rest . All I am saying > > > > the guidelines/governance model is not clear on the project and I > guess > > > its > > > > driven by opening a github issue request. Its the repository owned > by > > > > confluent and as much I appreciate that the features we mentioned are > > in > > > > the roadmap and welcoming us to contribute to the project. It doesn't > > > > gurantee what we want to add in the furture will be in your roadmap. > > > > > > > > Hence the reason having it part of Kafka community will help a lot as > > > other > > > > users can participate in the discussions. We are happy to drive any > > > > feature additions through KIPs which gives everyone a chance to > > > participate > > > > and add to the discussions. > > > > > > > > Thanks, > > > > Harsha > > > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce < > michael.pea...@ig.com> > > > > wrote: > > > > > > > > > +1 > > > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > > > Also i agree about the contribution comments made earlier in the > > > thread, > > > > i > > > > > personally am less likely to spend any of mine, or give project > time > > > > within > > > > > my internal projects to developers contributing to another > commercial > > > > > companies project even so technically open source, as then there is > > > that > > > > > commercial companies interest will always prevail and essentially > can > > > > > always have a final vote where disagreement. Im sure they never > > intend > > > > to, > > > > > but there is that true reality. This is why we have community open > > > source > > > > > projects. > > > > > > > > > > I can find many different implementations now of a rest endpoint on > > > > > GitHub, BitBucket etc. Each one has their benefits and > disadvantages > > in > > > > > their implementation. By making / providing one this would bring > > > together > > > > > these solutions, unifying those developers and also bringing the > best > > > of > > > > > all. > > > > > > > > > > I understand the concern on the community burden adding/supporting > > more > > > > > surface area for every client. But something like REST is universal > > and > > > > > worthy to be owned by the community. > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > From: Andrew Schofield > > > > > Sent: Saturday, October 8, 2016 1:19 AM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > > > There's a massive difference between the governance of Kafka and > the > > > > > governance of the REST proxy. > > > > > > > > > > In Kafka, there is a broad community of people contributing their > > > > opinions > > > > > about future enhancements in the form of KIPs. There's some really > > deep > > > > > consideration that goes into some of the trickier KIPs. There are > > > people > > > > > outside Confluent deeply knowledgeable in Kafka and building the > > > > > reputations to become committers. I get the impression that the > > roadmap > > > > of > > > > > Kafka is not really community-owned (what's the big feature for > Kafka > > > > 0.11, > > > > > for example), but the conveyor belt of smaller features in the form > > of > > > > KIPs > > > > > works nicely. It's a good example of open-source working well. > > > > > > > > > > The equivalent for the REST proxy is basically issues on GitHub. > The > > > > > roadmap is less clear. There's not really a community properly > > engaged > > > in > > > > &
Re: [DISCUSS] KIP-80: Kafka REST Server
e > communities around very simple single purpose tools, especially if you hope > for these to come together into a cohesive toolset across multiple such > tools. Much of what Apache does--create a collective decision making > process for resolving disagreement, help to trademark and protect the marks > of the project, etc just isn't that relevant for simple single-purpose > tools. > > So my take is there are a couple of options: > >1. We can try to put all the small tools into the Apache Project. I >think this is not the right approach as there is simply too many of > them, >many in different languages, serving different protocols, integrating > with >particular systems, and a single community can't effectively maintain > them >all. Doing this would significantly slow the progress of the Kafka > project. >As a protocol for messaging, I don't really see a case for including > REST >but not MQTT or AMQP which are technically much better suited to > messaging >and both are widely used for that. >2. We can treat ecosystem projects that aren't top level Apache projects >as invalid and try to recreate them all as Apache projects. Honestly, >though, if you go to the Kafka ecosystem page virtually none of the most >popular add-ons to Kafka are Apache projects. The most successful > things in >the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of other >clients, as well as the existing REST layer have succeeded at developing >communities that actively contribute and use these pieces and I don't > know >that that is a bad thing unless that community proves to be uninclusive, >unresponsive, or goes in a bad technical direction--and those are > failure >modes that all open source efforts face. >3. We can do what I think makes the most sense and try to work with the >projects that exist in the ecosystem and if the project doesn't have a >responsive community or wants to go in a different direction fork or >recreate that work. > > Of course any person can choose whatever of these options they want. But > from my point of view, option (3) has been the path of the community so far > and I think it has been quite successful. > > -Jay > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani > wrote: > > > Neha, > > "But I haven't seen any community emails or patches being submitted by > you > > guys, so I'm wondering why you are concerned about whether the community > is > > open to accepting patches or not." > > > > I think you are talking about contributing patches to this repository > > right? https://github.com/confluentinc/kafka-rest . All I am saying > > the guidelines/governance model is not clear on the project and I guess > its > > driven by opening a github issue request. Its the repository owned by > > confluent and as much I appreciate that the features we mentioned are in > > the roadmap and welcoming us to contribute to the project. It doesn't > > gurantee what we want to add in the furture will be in your roadmap. > > > > Hence the reason having it part of Kafka community will help a lot as > other > > users can participate in the discussions. We are happy to drive any > > feature additions through KIPs which gives everyone a chance to > participate > > and add to the discussions. > > > > Thanks, > > Harsha > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce > > wrote: > > > > > +1 > > > > > > I agree on the governance comments whole heartedly. > > > > > > Also i agree about the contribution comments made earlier in the > thread, > > i > > > personally am less likely to spend any of mine, or give project time > > within > > > my internal projects to developers contributing to another commercial > > > companies project even so technically open source, as then there is > that > > > commercial companies interest will always prevail and essentially can > > > always have a final vote where disagreement. Im sure they never intend > > to, > > > but there is that true reality. This is why we have community open > source > > > projects. > > > > > > I can find many different implementations now of a rest endpoint on > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in > > > their implementation. By making / providing one this would bring > together > > > these solutions, unifying those developers and
Re: [DISCUSS] KIP-80: Kafka REST Server
rantee what we want to add in the furture will be in your roadmap. > > > > > > Hence the reason having it part of Kafka community will help a lot as > > other > > > users can participate in the discussions. We are happy to drive any > > > feature additions through KIPs which gives everyone a chance to > > participate > > > and add to the discussions. > > > > > > Thanks, > > > Harsha > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce > > > wrote: > > > > > > > +1 > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > Also i agree about the contribution comments made earlier in the > > thread, > > > i > > > > personally am less likely to spend any of mine, or give project time > > > within > > > > my internal projects to developers contributing to another commercial > > > > companies project even so technically open source, as then there is > > that > > > > commercial companies interest will always prevail and essentially can > > > > always have a final vote where disagreement. Im sure they never > intend > > > to, > > > > but there is that true reality. This is why we have community open > > source > > > > projects. > > > > > > > > I can find many different implementations now of a rest endpoint on > > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages > in > > > > their implementation. By making / providing one this would bring > > together > > > > these solutions, unifying those developers and also bringing the best > > of > > > > all. > > > > > > > > I understand the concern on the community burden adding/supporting > more > > > > surface area for every client. But something like REST is universal > and > > > > worthy to be owned by the community. > > > > > > > > Mike > > > > > > > > > > > > > > > > From: Andrew Schofield > > > > Sent: Saturday, October 8, 2016 1:19 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > There's a massive difference between the governance of Kafka and the > > > > governance of the REST proxy. > > > > > > > > In Kafka, there is a broad community of people contributing their > > > opinions > > > > about future enhancements in the form of KIPs. There's some really > deep > > > > consideration that goes into some of the trickier KIPs. There are > > people > > > > outside Confluent deeply knowledgeable in Kafka and building the > > > > reputations to become committers. I get the impression that the > roadmap > > > of > > > > Kafka is not really community-owned (what's the big feature for Kafka > > > 0.11, > > > > for example), but the conveyor belt of smaller features in the form > of > > > KIPs > > > > works nicely. It's a good example of open-source working well. > > > > > > > > The equivalent for the REST proxy is basically issues on GitHub. The > > > > roadmap is less clear. There's not really a community properly > engaged > > in > > > > the way that there is with Kafka. So, you could say that it's clear > > that > > > > fewer people are interested, but I think the whole governance thing > > is a > > > > big barrier to engagement. And it's looking like it's getting out of > > > date. > > > > > > > > In technical terms, I can think of two big improvements to the REST > > > proxy. > > > > First, it needs to use the new consumer API so that it's possible to > > > secure > > > > connections between the REST proxy and Kafka. I don't care too much > > which > > > > method calls it uses actually uses to consume messages, but I do > care > > > that > > > > I cannot secure connections because of network security rules. > Second, > > > > there's an affinity between a consumer and the instance of the REST > > proxy > > > > to which it first connected. Kafka itself avoids this kind of > affinity > > > for > > > > good reason, and in the name of availability the RES
Re: [DISCUSS] KIP-80: Kafka REST Server
So from my reading essentially the first question needs to answered/and voted on is: Is Apache Kafka Community only about the Core or does the apache community also support some subprojects (and just we need some better way to manage this) If vote for Core only wins, then the following should be removed: Kafka Connect Kafka Stream If vote for Core only loses (aka we will support subprojects) then: We should look to add Kafka Rest And we should look to see how we can manage better govern and manage submodules. A good example which id propose here is how some other communities in Apache do this. Each Module has a Module Management Committee(MMC), this is like almost the PMC but at a per module basis. This MMC should essentially hold the binding votes for that module. The MMC should be made up of a single representative from each organisation (so no single organisation can fully veto the community it has to a genuine consenus) The MMC requires at least 3 members (so there cant be a tied vote on 2) For a new Module to be added a MMC committee should be sought A new Module is only capable of being added if the above requirements can be met (e.g. 3 people wishing to step up, from 3 organisations) so that only actively support modules would be added The PMC reviews each module every 6months or Year. If MMC is inactive, a vote/call to find replacements if raised, if none are forthcoming dropping the MMC to less than 3 then the module moves to "the attic" (very much like apache attic but a little more aggressively) This way the PMC does not need to micro manage every module We only add modules where some amount of active support and maintenance and use is provided by the community We have an automatic way to retire old or inactive projects. Thoughts? Mike From: Harsha Ch Sent: Thursday, October 20, 2016 10:26 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-80: Kafka REST Server Jay, REST API is something every user is in need of. If the argument is to clone and write your API, this will do a disservice to the users as they now have to choose one vs. others instead of keeping one API that is supported in Kafka community. "Pre-emptively re-creating another REST layer when it seems like we all quite agree on what needs to be done and we have an existing code base for HTTP/Kafka access that is heavily used in production seems quite silly." Exactly our point. Why can't we develop this in Apache Kafka community? Instead of us open sourcing another GitHub project and creating a divide in users and another version of API. Let's build this in Kafka Community and use the governance model that is proven to provide vendor free user driven consensus features. The argument that is adding this REST server to Kafka will affect the agility of the project doesn't mak sense. It looks like your argument is either we develop all these small tools or none at all. We as a community need to look at supporting critical tools/API. Instead of dividing this project into individual external communities. We should build this as part of Kafka which best serves the needs of users. The Streams and Connect projects that were pushed into Kafka could have been left in their own Github projects based on your arguments. What about the REST API is so different that such that it should stay out of the Kafka project? From my experience, more users are asking for the REST API. Thanks, Harsha On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps wrote: > I think the questions around governance make sense, I think we should > really clarify that to make the process more clear so it can be fully > inclusive. > > The idea that we should not collaborate on what is there now, though, > because in the future we might disagree about direction does not really > make sense to me. If in the future we disagree, that is the beauty of open > source, you can always fork off a copy of the code and start an independent > project either in Apache or elsewhere. Pre-emptively re-creating another > REST layer when it seems like we all quite agree on what needs to be done > and we have an existing code base for HTTP/kafka access that is heavily > used in production seems quite silly. > > Let me give some background on how I at least think about these things. > I've participated in open source projects out of LinkedIn via github as > well as via the ASF. I don't think there is a "right" answer to how to do > these but rather some tradeoffs. We thought about this quite a lot in the > context of Kafka based on the experience with the Hadoop ecosystem as well > as from other open source communities. > > There is a rich ecosystem around Kafka. Many of the projects are quite > small--single clients or tools that do a single thing well--and almost none > of t
Re: [DISCUSS] KIP-80: Kafka REST Server
Harsha Ch wrote on 20/10/2016 22:26:53: > The Streams and Connect projects that were pushed into Kafka could > have been left in their own Github projects based on your arguments. What > about the REST API is so different that such that it should stay out of the > Kafka project? From my experience, more users are asking for the REST API. Thanks Harsha for keeping pushing this issue. Your experience about users matches ours ! cheers Edo -- Edoardo Comar IBM MessageHub eco...@uk.ibm.com IBM UK Ltd, Hursley Park, SO21 2JN IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU Harsha Ch wrote on 20/10/2016 22:26:53: > From: Harsha Ch > To: dev@kafka.apache.org > Date: 20/10/2016 22:32 > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > Jay, > REST API is something every user is in need of. If the argument is to > clone and write your API, this will do a disservice to the users as they > now have to choose one vs. others instead of keeping one API that is > supported in Kafka community. > > "Pre-emptively re-creating another > REST layer when it seems like we all quite agree on what needs to be done > and we have an existing code base for HTTP/Kafka access that is heavily > used in production seems quite silly." > >Exactly our point. Why can't we develop this in Apache Kafka > community? Instead of us open sourcing another GitHub project and creating > a divide in users and another version of API. Let's build this in Kafka > Community and use the governance model that is proven to provide vendor > free user driven consensus features. The argument that is adding this REST > server to Kafka will affect the agility of the project doesn't mak sense. > > It looks like your argument is either we develop all these small tools or > none at all. We as a community need to look at supporting critical > tools/API. Instead of dividing this project into individual external > communities. We should build this as part of Kafka which best serves the > needs of users. > The Streams and Connect projects that were pushed into Kafka could > have been left in their own Github projects based on your arguments. What > about the REST API is so different that such that it should stay out of the > Kafka project? From my experience, more users are asking for the REST API. > > Thanks, > Harsha > > > > > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps wrote: > > > I think the questions around governance make sense, I think we should > > really clarify that to make the process more clear so it can be fully > > inclusive. > > > > The idea that we should not collaborate on what is there now, though, > > because in the future we might disagree about direction does not really > > make sense to me. If in the future we disagree, that is the beauty of open > > source, you can always fork off a copy of the code and start an independent > > project either in Apache or elsewhere. Pre-emptively re-creating another > > REST layer when it seems like we all quite agree on what needs to be done > > and we have an existing code base for HTTP/kafka access that is heavily > > used in production seems quite silly. > > > > Let me give some background on how I at least think about these things. > > I've participated in open source projects out of LinkedIn via github as > > well as via the ASF. I don't think there is a "right" answer to how to do > > these but rather some tradeoffs. We thought about this quite a lot in the > > context of Kafka based on the experience with the Hadoop ecosystem as well > > as from other open source communities. > > > > There is a rich ecosystem around Kafka. Many of the projects are quite > > small--single clients or tools that do a single thing well--and almost none > > of them are top level apache projects. I don't think trying to force each > > of these to turn into independent Apache projects is necessarily the best > > thing for the ecosystem. > > > > My observation of how this can go wrong is really what I think has happened > > to the Hadoop ecosystem. There you see quite a zoo of projects which all > > drift apart and don't quite work together well. Coordinating even simple > > changes and standardization across these is exceptionally difficult. The > > result is a bit of a mess for users--the pieces just don't really come > > together very well. This makes sense for independent infrastructure systems > > (
Re: [DISCUSS] KIP-80: Kafka REST Server
Hi Michael, Can you please share which Apache projects have a MMC? I couldn't find anything after a quick google. Ismael On Fri, Oct 21, 2016 at 7:28 AM, Michael Pearce wrote: > So from my reading essentially the first question needs to answered/and > voted on is: > > Is Apache Kafka Community only about the Core or does the apache community > also support some subprojects (and just we need some better way to manage > this) > > If vote for Core only wins, then the following should be removed: > Kafka Connect > Kafka Stream > > If vote for Core only loses (aka we will support subprojects) then: > We should look to add Kafka Rest > > And we should look to see how we can manage better govern and manage > submodules. > > A good example which id propose here is how some other communities in > Apache do this. > > Each Module has a Module Management Committee(MMC), this is like almost > the PMC but at a per module basis. > > This MMC should essentially hold the binding votes for that module. > The MMC should be made up of a single representative from each > organisation (so no single organisation can fully veto the community it > has to a genuine consenus) > The MMC requires at least 3 members (so there cant be a tied vote on 2) > For a new Module to be added a MMC committee should be sought > A new Module is only capable of being added if the above requirements can > be met (e.g. 3 people wishing to step up, from 3 organisations) so that > only actively support modules would be added > > The PMC reviews each module every 6months or Year. If MMC is inactive, a > vote/call to find replacements if raised, if none are forthcoming dropping > the MMC to less than 3 then the module moves to "the attic" (very much like > apache attic but a little more aggressively) > > This way the PMC does not need to micro manage every module > We only add modules where some amount of active support and maintenance > and use is provided by the community > We have an automatic way to retire old or inactive projects. > > Thoughts? > Mike > > > ____________ > From: Harsha Ch > Sent: Thursday, October 20, 2016 10:26 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > Jay, > REST API is something every user is in need of. If the argument is to > clone and write your API, this will do a disservice to the users as they > now have to choose one vs. others instead of keeping one API that is > supported in Kafka community. > > "Pre-emptively re-creating another > REST layer when it seems like we all quite agree on what needs to be done > and we have an existing code base for HTTP/Kafka access that is heavily > used in production seems quite silly." > >Exactly our point. Why can't we develop this in Apache Kafka > community? Instead of us open sourcing another GitHub project and creating > a divide in users and another version of API. Let's build this in Kafka > Community and use the governance model that is proven to provide vendor > free user driven consensus features. The argument that is adding this REST > server to Kafka will affect the agility of the project doesn't mak sense. > > It looks like your argument is either we develop all these small tools or > none at all. We as a community need to look at supporting critical > tools/API. Instead of dividing this project into individual external > communities. We should build this as part of Kafka which best serves the > needs of users. > The Streams and Connect projects that were pushed into Kafka could > have been left in their own Github projects based on your arguments. What > about the REST API is so different that such that it should stay out of the > Kafka project? From my experience, more users are asking for the REST API. > > Thanks, > Harsha > > > > > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps wrote: > > > I think the questions around governance make sense, I think we should > > really clarify that to make the process more clear so it can be fully > > inclusive. > > > > The idea that we should not collaborate on what is there now, though, > > because in the future we might disagree about direction does not really > > make sense to me. If in the future we disagree, that is the beauty of > open > > source, you can always fork off a copy of the code and start an > independent > > project either in Apache or elsewhere. Pre-emptively re-creating another > > REST layer when it seems like we all quite agree on what needs to be done > > and we have an existing code base for HTTP/kafka access that is heavily > > used i
Re: [DISCUSS] KIP-80: Kafka REST Server
Here is Ignite’s - essentially idea is having different owners/maintainers per module/area that keep a check on that area. https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute On 10/21/16, 3:44 PM, "isma...@gmail.com on behalf of Ismael Juma" wrote: Hi Michael, Can you please share which Apache projects have a MMC? I couldn't find anything after a quick google. Ismael On Fri, Oct 21, 2016 at 7:28 AM, Michael Pearce wrote: > So from my reading essentially the first question needs to answered/and > voted on is: > > Is Apache Kafka Community only about the Core or does the apache community > also support some subprojects (and just we need some better way to manage > this) > > If vote for Core only wins, then the following should be removed: > Kafka Connect > Kafka Stream > > If vote for Core only loses (aka we will support subprojects) then: > We should look to add Kafka Rest > > And we should look to see how we can manage better govern and manage > submodules. > > A good example which id propose here is how some other communities in > Apache do this. > > Each Module has a Module Management Committee(MMC), this is like almost > the PMC but at a per module basis. > > This MMC should essentially hold the binding votes for that module. > The MMC should be made up of a single representative from each > organisation (so no single organisation can fully veto the community it > has to a genuine consenus) > The MMC requires at least 3 members (so there cant be a tied vote on 2) > For a new Module to be added a MMC committee should be sought > A new Module is only capable of being added if the above requirements can > be met (e.g. 3 people wishing to step up, from 3 organisations) so that > only actively support modules would be added > > The PMC reviews each module every 6months or Year. If MMC is inactive, a > vote/call to find replacements if raised, if none are forthcoming dropping > the MMC to less than 3 then the module moves to "the attic" (very much like > apache attic but a little more aggressively) > > This way the PMC does not need to micro manage every module > We only add modules where some amount of active support and maintenance > and use is provided by the community > We have an automatic way to retire old or inactive projects. > > Thoughts? > Mike > > > ________________ > From: Harsha Ch > Sent: Thursday, October 20, 2016 10:26 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > Jay, > REST API is something every user is in need of. If the argument is to > clone and write your API, this will do a disservice to the users as they > now have to choose one vs. others instead of keeping one API that is > supported in Kafka community. > > "Pre-emptively re-creating another > REST layer when it seems like we all quite agree on what needs to be done > and we have an existing code base for HTTP/Kafka access that is heavily > used in production seems quite silly." > >Exactly our point. Why can't we develop this in Apache Kafka > community? Instead of us open sourcing another GitHub project and creating > a divide in users and another version of API. Let's build this in Kafka > Community and use the governance model that is proven to provide vendor > free user driven consensus features. The argument that is adding this REST > server to Kafka will affect the agility of the project doesn't mak sense. > > It looks like your argument is either we develop all these small tools or > none at all. We as a community need to look at supporting critical > tools/API. Instead of dividing this project into individual external > communities. We should build this as part of Kafka which best serves the > needs of users. > The Streams and Connect projects that were pushed into Kafka could > have been left in their own Github projects based on your arguments. What > about the REST API is so different that such that it should stay out of the > Kafka project? From my experience, more users are asking for the REST API. > > Thanks, > Harsha > > > > > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps wrote: > > > I think the questions around governance make sense, I think we should > > really clarify that to make the
Re: [DISCUSS] KIP-80: Kafka REST Server
valid and try to recreate them all as Apache projects. > > Honestly, > > > >though, if you go to the Kafka ecosystem page virtually none of > the > > > most > > > >popular add-ons to Kafka are Apache projects. The most successful > > > > things in > > > >the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of > > > other > > > >clients, as well as the existing REST layer have succeeded at > > > developing > > > >communities that actively contribute and use these pieces and I > > don't > > > > know > > > >that that is a bad thing unless that community proves to be > > > uninclusive, > > > >unresponsive, or goes in a bad technical direction--and those are > > > > failure > > > >modes that all open source efforts face. > > > >3. We can do what I think makes the most sense and try to work > with > > > the > > > >projects that exist in the ecosystem and if the project doesn't > have > > a > > > >responsive community or wants to go in a different direction fork > or > > > >recreate that work. > > > > > > > > Of course any person can choose whatever of these options they want. > > But > > > > from my point of view, option (3) has been the path of the community > so > > > far > > > > and I think it has been quite successful. > > > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani < > ka...@harsha.io > > > > > > > wrote: > > > > > > > > > Neha, > > > > > "But I haven't seen any community emails or patches being submitted > > by > > > > you > > > > > guys, so I'm wondering why you are concerned about whether the > > > community > > > > is > > > > > open to accepting patches or not." > > > > > > > > > > I think you are talking about contributing patches to this > repository > > > > > right? https://github.com/confluentinc/kafka-rest . All I am > saying > > > > > the guidelines/governance model is not clear on the project and I > > guess > > > > its > > > > > driven by opening a github issue request. Its the repository owned > > by > > > > > confluent and as much I appreciate that the features we mentioned > are > > > in > > > > > the roadmap and welcoming us to contribute to the project. It > doesn't > > > > > gurantee what we want to add in the furture will be in your > roadmap. > > > > > > > > > > Hence the reason having it part of Kafka community will help a lot > as > > > > other > > > > > users can participate in the discussions. We are happy to drive > any > > > > > feature additions through KIPs which gives everyone a chance to > > > > participate > > > > > and add to the discussions. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce < > > michael.pea...@ig.com> > > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > > > > > Also i agree about the contribution comments made earlier in the > > > > thread, > > > > > i > > > > > > personally am less likely to spend any of mine, or give project > > time > > > > > within > > > > > > my internal projects to developers contributing to another > > commercial > > > > > > companies project even so technically open source, as then there > is > > > > that > > > > > > commercial companies interest will always prevail and essentially > > can > > > > > > always have a final vote where disagreement. Im sure they never > > > intend > > > > > to, > > > > > > but there is that true reality. This is why we have community > open > > > &g
Re: [DISCUSS] KIP-80: Kafka REST Server
At the high level, I think ideally it makes sense to add a component to Apache Kafka if (1) it's widely needed and (2) it needs tight integration with Kafka core. For Kafka Stream, we do expect stream processing will be used widely in the future. Implementation wise, Kafka Stream only supports getting data from Kafka and leverages quite a few of the core functionalities in Kafka core. For example, it uses customized rebalance callback in the consumer and uses the compacted topic heavily. So, having Kafka Stream in the same repo makes it easier for testing when those core functionalities evolve over time. Kafka Connect is in the same situation. For rest proxy, whether it's widely used or not is going to be a bit subjective. However, there are likely places that can live without a rest proxy. The rest proxy is just a proxy for the regular clients and doesn't need to be tightly integrated with Kafka core. So, the case for including rest proxy in Apache Kafka is probably not as strong as Kafka Stream and Kafka Connect. Thanks, Jun On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce wrote: > So from my reading essentially the first question needs to answered/and > voted on is: > > Is Apache Kafka Community only about the Core or does the apache community > also support some subprojects (and just we need some better way to manage > this) > > If vote for Core only wins, then the following should be removed: > Kafka Connect > Kafka Stream > > If vote for Core only loses (aka we will support subprojects) then: > We should look to add Kafka Rest > > And we should look to see how we can manage better govern and manage > submodules. > > A good example which id propose here is how some other communities in > Apache do this. > > Each Module has a Module Management Committee(MMC), this is like almost > the PMC but at a per module basis. > > This MMC should essentially hold the binding votes for that module. > The MMC should be made up of a single representative from each > organisation (so no single organisation can fully veto the community it > has to a genuine consenus) > The MMC requires at least 3 members (so there cant be a tied vote on 2) > For a new Module to be added a MMC committee should be sought > A new Module is only capable of being added if the above requirements can > be met (e.g. 3 people wishing to step up, from 3 organisations) so that > only actively support modules would be added > > The PMC reviews each module every 6months or Year. If MMC is inactive, a > vote/call to find replacements if raised, if none are forthcoming dropping > the MMC to less than 3 then the module moves to "the attic" (very much like > apache attic but a little more aggressively) > > This way the PMC does not need to micro manage every module > We only add modules where some amount of active support and maintenance > and use is provided by the community > We have an automatic way to retire old or inactive projects. > > Thoughts? > Mike > > > ____ > From: Harsha Ch > Sent: Thursday, October 20, 2016 10:26 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > Jay, > REST API is something every user is in need of. If the argument is to > clone and write your API, this will do a disservice to the users as they > now have to choose one vs. others instead of keeping one API that is > supported in Kafka community. > > "Pre-emptively re-creating another > REST layer when it seems like we all quite agree on what needs to be done > and we have an existing code base for HTTP/Kafka access that is heavily > used in production seems quite silly." > >Exactly our point. Why can't we develop this in Apache Kafka > community? Instead of us open sourcing another GitHub project and creating > a divide in users and another version of API. Let's build this in Kafka > Community and use the governance model that is proven to provide vendor > free user driven consensus features. The argument that is adding this REST > server to Kafka will affect the agility of the project doesn't mak sense. > > It looks like your argument is either we develop all these small tools or > none at all. We as a community need to look at supporting critical > tools/API. Instead of dividing this project into individual external > communities. We should build this as part of Kafka which best serves the > needs of users. > The Streams and Connect projects that were pushed into Kafka could > have been left in their own Github projects based on your arguments. What > about the REST API is so different that such that it should stay out of the > Kafka project? From my experience, more users are asking
Re: [DISCUSS] KIP-80: Kafka REST Server
Are you saying Kafka REST is subjective but Kafka Streams and Kafka Connect are not subjective? > "there are likely places that can live without a rest proxy" There are also places that can live without Kafka Streams and Kafka Connect. Nacho On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao wrote: > At the high level, I think ideally it makes sense to add a component to > Apache Kafka if (1) it's widely needed and (2) it needs tight integration > with Kafka core. For Kafka Stream, we do expect stream processing will be > used widely in the future. Implementation wise, Kafka Stream only supports > getting data from Kafka and leverages quite a few of the core > functionalities in Kafka core. For example, it uses customized rebalance > callback in the consumer and uses the compacted topic heavily. So, having > Kafka Stream in the same repo makes it easier for testing when those core > functionalities evolve over time. Kafka Connect is in the same situation. > > For rest proxy, whether it's widely used or not is going to be a bit > subjective. However, there are likely places that can live without a rest > proxy. The rest proxy is just a proxy for the regular clients and doesn't > need to be tightly integrated with Kafka core. So, the case for including > rest proxy in Apache Kafka is probably not as strong as Kafka Stream and > Kafka Connect. > > Thanks, > > Jun > > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce > wrote: > > > So from my reading essentially the first question needs to answered/and > > voted on is: > > > > Is Apache Kafka Community only about the Core or does the apache > community > > also support some subprojects (and just we need some better way to manage > > this) > > > > If vote for Core only wins, then the following should be removed: > > Kafka Connect > > Kafka Stream > > > > If vote for Core only loses (aka we will support subprojects) then: > > We should look to add Kafka Rest > > > > And we should look to see how we can manage better govern and manage > > submodules. > > > > A good example which id propose here is how some other communities in > > Apache do this. > > > > Each Module has a Module Management Committee(MMC), this is like almost > > the PMC but at a per module basis. > > > > This MMC should essentially hold the binding votes for that module. > > The MMC should be made up of a single representative from each > > organisation (so no single organisation can fully veto the community it > > has to a genuine consenus) > > The MMC requires at least 3 members (so there cant be a tied vote on 2) > > For a new Module to be added a MMC committee should be sought > > A new Module is only capable of being added if the above requirements can > > be met (e.g. 3 people wishing to step up, from 3 organisations) so that > > only actively support modules would be added > > > > The PMC reviews each module every 6months or Year. If MMC is inactive, a > > vote/call to find replacements if raised, if none are forthcoming > dropping > > the MMC to less than 3 then the module moves to "the attic" (very much > like > > apache attic but a little more aggressively) > > > > This way the PMC does not need to micro manage every module > > We only add modules where some amount of active support and maintenance > > and use is provided by the community > > We have an automatic way to retire old or inactive projects. > > > > Thoughts? > > Mike > > > > > > > > From: Harsha Ch > > Sent: Thursday, October 20, 2016 10:26 PM > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > Jay, > > REST API is something every user is in need of. If the argument is > to > > clone and write your API, this will do a disservice to the users as they > > now have to choose one vs. others instead of keeping one API that is > > supported in Kafka community. > > > > "Pre-emptively re-creating another > > REST layer when it seems like we all quite agree on what needs to be done > > and we have an existing code base for HTTP/Kafka access that is heavily > > used in production seems quite silly." > > > >Exactly our point. Why can't we develop this in Apache Kafka > > community? Instead of us open sourcing another GitHub project and > creating > > a divide in users and another version of API. Let's build this in Kafka > > Community and use the governance model that is proven to provide vendor &
Re: [DISCUSS] KIP-80: Kafka REST Server
>unresponsive, or goes in a bad technical direction--and those are > > > > failure > > > >modes that all open source efforts face. > > > >3. We can do what I think makes the most sense and try to work > with > > > the > > > >projects that exist in the ecosystem and if the project doesn't > > have a > > > >responsive community or wants to go in a different direction fork > or > > > >recreate that work. > > > > > > > > Of course any person can choose whatever of these options they want. > > But > > > > from my point of view, option (3) has been the path of the community > so > > > far > > > > and I think it has been quite successful. > > > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani < > ka...@harsha.io > > > > > > > wrote: > > > > > > > > > Neha, > > > > > "But I haven't seen any community emails or patches being submitted > > by > > > > you > > > > > guys, so I'm wondering why you are concerned about whether the > > > community > > > > is > > > > > open to accepting patches or not." > > > > > > > > > > I think you are talking about contributing patches to this > repository > > > > > right? https://github.com/confluentinc/kafka-rest . All I am > saying > > > > > the guidelines/governance model is not clear on the project and I > > guess > > > > its > > > > > driven by opening a github issue request. Its the repository owned > > by > > > > > confluent and as much I appreciate that the features we mentioned > are > > > in > > > > > the roadmap and welcoming us to contribute to the project. It > doesn't > > > > > gurantee what we want to add in the furture will be in your > roadmap. > > > > > > > > > > Hence the reason having it part of Kafka community will help a lot > as > > > > other > > > > > users can participate in the discussions. We are happy to drive > any > > > > > feature additions through KIPs which gives everyone a chance to > > > > participate > > > > > and add to the discussions. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce < > > michael.pea...@ig.com> > > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > > > > > Also i agree about the contribution comments made earlier in the > > > > thread, > > > > > i > > > > > > personally am less likely to spend any of mine, or give project > > time > > > > > within > > > > > > my internal projects to developers contributing to another > > commercial > > > > > > companies project even so technically open source, as then there > is > > > > that > > > > > > commercial companies interest will always prevail and essentially > > can > > > > > > always have a final vote where disagreement. Im sure they never > > > intend > > > > > to, > > > > > > but there is that true reality. This is why we have community > open > > > > source > > > > > > projects. > > > > > > > > > > > > I can find many different implementations now of a rest endpoint > on > > > > > > GitHub, BitBucket etc. Each one has their benefits and > > disadvantages > > > in > > > > > > their implementation. By making / providing one this would bring > > > > together > > > > > > these solutions, unifying those developers and also bringing the > > best > > > > of > > > > > > all. > > > > > > > > > > > > I understand the concern on the community burden > adding/supporting > > > more > > > > > > surface a
Re: [DISCUSS] KIP-80: Kafka REST Server
h Module has a Module Management Committee(MMC), this is like almost > > > the PMC but at a per module basis. > > > > > > This MMC should essentially hold the binding votes for that module. > > > The MMC should be made up of a single representative from each > > > organisation (so no single organisation can fully veto the community > it > > > has to a genuine consenus) > > > The MMC requires at least 3 members (so there cant be a tied vote on 2) > > > For a new Module to be added a MMC committee should be sought > > > A new Module is only capable of being added if the above requirements > can > > > be met (e.g. 3 people wishing to step up, from 3 organisations) so that > > > only actively support modules would be added > > > > > > The PMC reviews each module every 6months or Year. If MMC is inactive, > a > > > vote/call to find replacements if raised, if none are forthcoming > > dropping > > > the MMC to less than 3 then the module moves to "the attic" (very much > > like > > > apache attic but a little more aggressively) > > > > > > This way the PMC does not need to micro manage every module > > > We only add modules where some amount of active support and maintenance > > > and use is provided by the community > > > We have an automatic way to retire old or inactive projects. > > > > > > Thoughts? > > > Mike > > > > > > > > > > > > From: Harsha Ch > > > Sent: Thursday, October 20, 2016 10:26 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > Jay, > > > REST API is something every user is in need of. If the argument > is > > to > > > clone and write your API, this will do a disservice to the users as > they > > > now have to choose one vs. others instead of keeping one API that is > > > supported in Kafka community. > > > > > > "Pre-emptively re-creating another > > > REST layer when it seems like we all quite agree on what needs to be > done > > > and we have an existing code base for HTTP/Kafka access that is heavily > > > used in production seems quite silly." > > > > > >Exactly our point. Why can't we develop this in Apache Kafka > > > community? Instead of us open sourcing another GitHub project and > > creating > > > a divide in users and another version of API. Let's build this in Kafka > > > Community and use the governance model that is proven to provide vendor > > > free user driven consensus features. The argument that is adding this > > REST > > > server to Kafka will affect the agility of the project doesn't mak > sense. > > > > > > It looks like your argument is either we develop all these small tools > or > > > none at all. We as a community need to look at supporting critical > > > tools/API. Instead of dividing this project into individual external > > > communities. We should build this as part of Kafka which best serves > the > > > needs of users. > > > The Streams and Connect projects that were pushed into Kafka > > could > > > have been left in their own Github projects based on your arguments. > What > > > about the REST API is so different that such that it should stay out of > > the > > > Kafka project? From my experience, more users are asking for the REST > > API. > > > > > > Thanks, > > > Harsha > > > > > > > > > > > > > > > > > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps wrote: > > > > > > > I think the questions around governance make sense, I think we should > > > > really clarify that to make the process more clear so it can be fully > > > > inclusive. > > > > > > > > The idea that we should not collaborate on what is there now, though, > > > > because in the future we might disagree about direction does not > really > > > > make sense to me. If in the future we disagree, that is the beauty of > > > open > > > > source, you can always fork off a copy of the code and start an > > > independent > > > > project either in Apache or elsewhere. Pre-emptively re-creating > > another > > > > REST layer when it seems like we all quite agree on what needs to be > > done > > > > and we have
Re: [DISCUSS] KIP-80: Kafka REST Server
gt; > > > > > So from my reading essentially the first question needs to > answered/and > > > > voted on is: > > > > > > > > Is Apache Kafka Community only about the Core or does the apache > > > community > > > > also support some subprojects (and just we need some better way to > > manage > > > > this) > > > > > > > > If vote for Core only wins, then the following should be removed: > > > > Kafka Connect > > > > Kafka Stream > > > > > > > > If vote for Core only loses (aka we will support subprojects) then: > > > > We should look to add Kafka Rest > > > > > > > > And we should look to see how we can manage better govern and manage > > > > submodules. > > > > > > > > A good example which id propose here is how some other communities in > > > > Apache do this. > > > > > > > > Each Module has a Module Management Committee(MMC), this is like > almost > > > > the PMC but at a per module basis. > > > > > > > > This MMC should essentially hold the binding votes for that module. > > > > The MMC should be made up of a single representative from each > > > > organisation (so no single organisation can fully veto the community > > it > > > > has to a genuine consenus) > > > > The MMC requires at least 3 members (so there cant be a tied vote on > 2) > > > > For a new Module to be added a MMC committee should be sought > > > > A new Module is only capable of being added if the above requirements > > can > > > > be met (e.g. 3 people wishing to step up, from 3 organisations) so > that > > > > only actively support modules would be added > > > > > > > > The PMC reviews each module every 6months or Year. If MMC is > inactive, > > a > > > > vote/call to find replacements if raised, if none are forthcoming > > > dropping > > > > the MMC to less than 3 then the module moves to "the attic" (very > much > > > like > > > > apache attic but a little more aggressively) > > > > > > > > This way the PMC does not need to micro manage every module > > > > We only add modules where some amount of active support and > maintenance > > > > and use is provided by the community > > > > We have an automatic way to retire old or inactive projects. > > > > > > > > Thoughts? > > > > Mike > > > > > > > > > > > > > > > > From: Harsha Ch > > > > Sent: Thursday, October 20, 2016 10:26 PM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > Jay, > > > > REST API is something every user is in need of. If the argument > > is > > > to > > > > clone and write your API, this will do a disservice to the users as > > they > > > > now have to choose one vs. others instead of keeping one API that is > > > > supported in Kafka community. > > > > > > > > "Pre-emptively re-creating another > > > > REST layer when it seems like we all quite agree on what needs to be > > done > > > > and we have an existing code base for HTTP/Kafka access that is > heavily > > > > used in production seems quite silly." > > > > > > > >Exactly our point. Why can't we develop this in Apache Kafka > > > > community? Instead of us open sourcing another GitHub project and > > > creating > > > > a divide in users and another version of API. Let's build this in > Kafka > > > > Community and use the governance model that is proven to provide > vendor > > > > free user driven consensus features. The argument that is adding this > > > REST > > > > server to Kafka will affect the agility of the project doesn't mak > > sense. > > > > > > > > It looks like your argument is either we develop all these small > tools > > or > > > > none at all. We as a community need to look at supporting critical > > > > tools/API. Instead of dividing this project into individual external > > > > communities. We should build this as part of Kafka which best serves > > the > > > > needs of users. > > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
For Kafka Stream, we do expect stream processing will >be >> > used widely in the future. Implementation wise, Kafka Stream only >> supports >> > getting data from Kafka and leverages quite a few of the core >> > functionalities in Kafka core. For example, it uses customized >>rebalance >> > callback in the consumer and uses the compacted topic heavily. So, >having >> > Kafka Stream in the same repo makes it easier for testing when those >core >> > functionalities evolve over time. Kafka Connect is in the same >situation. >> > >> > For rest proxy, whether it's widely used or not is going to be a bit >> > subjective. However, there are likely places that can live without a >rest >> > proxy. The rest proxy is just a proxy for the regular clients and >doesn't >> > need to be tightly integrated with Kafka core. So, the case for >including >> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream >>and >> > Kafka Connect. >> > >> > Thanks, >> > >> > Jun >> > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce >> >> > wrote: >> > >> > > So from my reading essentially the first question needs to >answered/and >> > > voted on is: >> > > >> > > Is Apache Kafka Community only about the Core or does the apache >> > community >> > > also support some subprojects (and just we need some better way to >> manage >> > > this) >> > > >> > > If vote for Core only wins, then the following should be removed: >> > > Kafka Connect >> > > Kafka Stream >> > > >> > > If vote for Core only loses (aka we will support subprojects) then: >> > > We should look to add Kafka Rest >> > > >> > > And we should look to see how we can manage better govern and manage >> > > submodules. >> > > >> > > A good example which id propose here is how some other communities >>in >> > > Apache do this. >> > > >> > > Each Module has a Module Management Committee(MMC), this is like >almost >> > > the PMC but at a per module basis. >> > > >> > > This MMC should essentially hold the binding votes for that module. >> > > The MMC should be made up of a single representative from each >> > > organisation (so no single organisation can fully veto the >>community >> it >> > > has to a genuine consenus) >> > > The MMC requires at least 3 members (so there cant be a tied vote on >2) >> > > For a new Module to be added a MMC committee should be sought >> > > A new Module is only capable of being added if the above >>requirements >> can >> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so >that >> > > only actively support modules would be added >> > > >> > > The PMC reviews each module every 6months or Year. If MMC is >>inactive, >> a >> > > vote/call to find replacements if raised, if none are forthcoming >> > dropping >> > > the MMC to less than 3 then the module moves to "the attic" (very >>much >> > like >> > > apache attic but a little more aggressively) >> > > >> > > This way the PMC does not need to micro manage every module >> > > We only add modules where some amount of active support and >maintenance >> > > and use is provided by the community >> > > We have an automatic way to retire old or inactive projects. >> > > >> > > Thoughts? >> > > Mike >> > > >> > > >> > > >> > > From: Harsha Ch >> > > Sent: Thursday, October 20, 2016 10:26 PM >> > > To: dev@kafka.apache.org >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server >> > > >> > > Jay, >> > > REST API is something every user is in need of. If the >>argument >> is >> > to >> > > clone and write your API, this will do a disservice to the users as >> they >> > > now have to choose one vs. others instead of keeping one API that is >> > > supported in Kafka community. >> > > >> > > "Pre-emptively re-creating another >> > > REST layer when it seems like we all quite agree on what needs to be >> done >> > > and
Re: [DISCUSS] KIP-80: Kafka REST Server
> > >> > > Is Apache Kafka Community only about the Core or does the apache > >> > community > >> > > also support some subprojects (and just we need some better way to > >> manage > >> > > this) > >> > > > >> > > If vote for Core only wins, then the following should be removed: > >> > > Kafka Connect > >> > > Kafka Stream > >> > > > >> > > If vote for Core only loses (aka we will support subprojects) then: > >> > > We should look to add Kafka Rest > >> > > > >> > > And we should look to see how we can manage better govern and manage > >> > > submodules. > >> > > > >> > > A good example which id propose here is how some other communities > >>in > >> > > Apache do this. > >> > > > >> > > Each Module has a Module Management Committee(MMC), this is like > >almost > >> > > the PMC but at a per module basis. > >> > > > >> > > This MMC should essentially hold the binding votes for that module. > >> > > The MMC should be made up of a single representative from each > >> > > organisation (so no single organisation can fully veto the > >>community > >> it > >> > > has to a genuine consenus) > >> > > The MMC requires at least 3 members (so there cant be a tied vote on > >2) > >> > > For a new Module to be added a MMC committee should be sought > >> > > A new Module is only capable of being added if the above > >>requirements > >> can > >> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so > >that > >> > > only actively support modules would be added > >> > > > >> > > The PMC reviews each module every 6months or Year. If MMC is > >>inactive, > >> a > >> > > vote/call to find replacements if raised, if none are forthcoming > >> > dropping > >> > > the MMC to less than 3 then the module moves to "the attic" (very > >>much > >> > like > >> > > apache attic but a little more aggressively) > >> > > > >> > > This way the PMC does not need to micro manage every module > >> > > We only add modules where some amount of active support and > >maintenance > >> > > and use is provided by the community > >> > > We have an automatic way to retire old or inactive projects. > >> > > > >> > > Thoughts? > >> > > Mike > >> > > > >> > > > >> > > > >> > > From: Harsha Ch > >> > > Sent: Thursday, October 20, 2016 10:26 PM > >> > > To: dev@kafka.apache.org > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > >> > > > >> > > Jay, > >> > > REST API is something every user is in need of. If the > >>argument > >> is > >> > to > >> > > clone and write your API, this will do a disservice to the users as > >> they > >> > > now have to choose one vs. others instead of keeping one API that is > >> > > supported in Kafka community. > >> > > > >> > > "Pre-emptively re-creating another > >> > > REST layer when it seems like we all quite agree on what needs to be > >> done > >> > > and we have an existing code base for HTTP/Kafka access that is > >heavily > >> > > used in production seems quite silly." > >> > > > >> > >Exactly our point. Why can't we develop this in Apache Kafka > >> > > community? Instead of us open sourcing another GitHub project and > >> > creating > >> > > a divide in users and another version of API. Let's build this in > >Kafka > >> > > Community and use the governance model that is proven to provide > >vendor > >> > > free user driven consensus features. The argument that is adding > >>this > >> > REST > >> > > server to Kafka will affect the agility of the project doesn't mak > >> sense. > >> > > > >> > > It looks like your argument is either we develop all these small > >>tools > >> or > >> > > none at all. We
Re: [DISCUSS] KIP-80: Kafka REST Server
a few of the core > > >> > functionalities in Kafka core. For example, it uses customized > > >>rebalance > > >> > callback in the consumer and uses the compacted topic heavily. So, > > >having > > >> > Kafka Stream in the same repo makes it easier for testing when those > > >core > > >> > functionalities evolve over time. Kafka Connect is in the same > > >situation. > > >> > > > >> > For rest proxy, whether it's widely used or not is going to be a bit > > >> > subjective. However, there are likely places that can live without a > > >rest > > >> > proxy. The rest proxy is just a proxy for the regular clients and > > >doesn't > > >> > need to be tightly integrated with Kafka core. So, the case for > > >including > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream > > >>and > > >> > Kafka Connect. > > >> > > > >> > Thanks, > > >> > > > >> > Jun > > >> > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce > > >> > > >> > wrote: > > >> > > > >> > > So from my reading essentially the first question needs to > > >answered/and > > >> > > voted on is: > > >> > > > > >> > > Is Apache Kafka Community only about the Core or does the apache > > >> > community > > >> > > also support some subprojects (and just we need some better way to > > >> manage > > >> > > this) > > >> > > > > >> > > If vote for Core only wins, then the following should be removed: > > >> > > Kafka Connect > > >> > > Kafka Stream > > >> > > > > >> > > If vote for Core only loses (aka we will support subprojects) > then: > > >> > > We should look to add Kafka Rest > > >> > > > > >> > > And we should look to see how we can manage better govern and > manage > > >> > > submodules. > > >> > > > > >> > > A good example which id propose here is how some other communities > > >>in > > >> > > Apache do this. > > >> > > > > >> > > Each Module has a Module Management Committee(MMC), this is like > > >almost > > >> > > the PMC but at a per module basis. > > >> > > > > >> > > This MMC should essentially hold the binding votes for that > module. > > >> > > The MMC should be made up of a single representative from each > > >> > > organisation (so no single organisation can fully veto the > > >>community > > >> it > > >> > > has to a genuine consenus) > > >> > > The MMC requires at least 3 members (so there cant be a tied vote > on > > >2) > > >> > > For a new Module to be added a MMC committee should be sought > > >> > > A new Module is only capable of being added if the above > > >>requirements > > >> can > > >> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so > > >that > > >> > > only actively support modules would be added > > >> > > > > >> > > The PMC reviews each module every 6months or Year. If MMC is > > >>inactive, > > >> a > > >> > > vote/call to find replacements if raised, if none are forthcoming > > >> > dropping > > >> > > the MMC to less than 3 then the module moves to "the attic" (very > > >>much > > >> > like > > >> > > apache attic but a little more aggressively) > > >> > > > > >> > > This way the PMC does not need to micro manage every module > > >> > > We only add modules where some amount of active support and > > >maintenance > > >> > > and use is provided by the community > > >> > > We have an automatic way to retire old or inactive projects. > > >> > > > > >> > > Thoughts? > > >> > > Mike > > >> > > > > >> > > > > >> > > > > >> > > From: Harsha Ch > > >> > > Sent: Thursday, October 2
Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > > > > > > > > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis > > > > > > > >wrote: > > > > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and Kafka > > > >Connect > > > >> are not subjective? > > > >> > > > >> > "there are likely places that can live without a rest proxy" > > > >> > > > >> There are also places that can live without Kafka Streams and Kafka > > > >> Connect. > > > >> > > > >> Nacho > > > >> > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao wrote: > > > >> > > > >> > At the high level, I think ideally it makes sense to add a > component > > > >>to > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight > > > >integration > > > >> > with Kafka core. For Kafka Stream, we do expect stream processing > > will > > > >be > > > >> > used widely in the future. Implementation wise, Kafka Stream only > > > >> supports > > > >> > getting data from Kafka and leverages quite a few of the core > > > >> > functionalities in Kafka core. For example, it uses customized > > > >>rebalance > > > >> > callback in the consumer and uses the compacted topic heavily. So, > > > >having > > > >> > Kafka Stream in the same repo makes it easier for testing when > those > > > >core > > > >> > functionalities evolve over time. Kafka Connect is in the same > > > >situation. > > > >> > > > > >> > For rest proxy, whether it's widely used or not is going to be a > bit > > > >> > subjective. However, there are likely places that can live > without a > > > >rest > > > >> > proxy. The rest proxy is just a proxy for the regular clients and > > > >doesn't > > > >> > need to be tightly integrated with Kafka core. So, the case for > > > >including > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka > Stream > > > >>and > > > >> > Kafka Connect. > > > >> > > > > >> > Thanks, > > > >> > > > > >> > Jun > > > >> > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce > > > >> > > > >> > wrote: > > > >> > > > > >> > > So from my reading essentially the first question needs to > > > >answered/and > > > >> > > voted on is: > > > >> > > > > > >> > > Is Apache Kafka Community only about the Core or does the apache > > > >> > community > > > >> > > also support some subprojects (and just we need some better way > to > > > >> manage > > > >> > > this) > > > >> > > > > > >> > > If vote for Core only wins, then the following should be > removed: > > > >> > > Kafka Connect > > > >> > > Kafka Stream > > > >> > > > > > >> > > If vote for Core only loses (aka we will support subprojects) > > then: > > > >> > > We should look to add Kafka Rest > > > >> > > > > > >> > > And we should look to see how we can manage better govern and > > manage > > > >> > > submodules. > > > >> > > > > > >> > > A good example which id propose here is how some other > communities > > > >>in > > > >> > > Apache do this. > > > >> > > > > > >> > > Each Module has a Module Management Committee(MMC), this is like > > > >almost > > > >> > > the PMC but at a per module basis. > > > >> > > > > > >> > > This MMC should essentially hold the binding votes for that > > module. > > > >> > > The MMC should be made up of a single representative from each > > > >> > > organisation (so no single organisation can fully veto the > > > >>community > > > >> it > > > >> > > has to a genuine consenus) > > > >> >
Re: [DISCUSS] KIP-80: Kafka REST Server
gt; > > > >Can the streaming platform exist without Connect? - No. Data > > integration > > > > >is > > > > >fundamental to building an end to end platform > > > > > > > > > >Can the streaming platform exist without stream processing? - No. > > > > >Processing stream data again is a core part of streaming platform. > > > > > > > > > >Can the streaming platform exist without clients? - We at least need > > one > > > > >client library to complete the platform. Our Java clients help us to > > > > >complete the platform story. The rest of the clients are built and > > > > >maintained outside the project. > > > > > > > > > >Can the platform exist without the rest proxy? - Yes. The proxy does > > not > > > > >complete the platform vision in anyway. It is just a good to have > tool > > > > >that > > > > >might be required by quite a few users and there is an active > project > > > that > > > > >works on this - https://github.com/confluentinc/kafka-rest > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis > > > > > > > > > >wrote: > > > > > > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and > Kafka > > > > >Connect > > > > >> are not subjective? > > > > >> > > > > >> > "there are likely places that can live without a rest proxy" > > > > >> > > > > >> There are also places that can live without Kafka Streams and > Kafka > > > > >> Connect. > > > > >> > > > > >> Nacho > > > > >> > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao > wrote: > > > > >> > > > > >> > At the high level, I think ideally it makes sense to add a > > component > > > > >>to > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight > > > > >integration > > > > >> > with Kafka core. For Kafka Stream, we do expect stream > processing > > > will > > > > >be > > > > >> > used widely in the future. Implementation wise, Kafka Stream > only > > > > >> supports > > > > >> > getting data from Kafka and leverages quite a few of the core > > > > >> > functionalities in Kafka core. For example, it uses customized > > > > >>rebalance > > > > >> > callback in the consumer and uses the compacted topic heavily. > So, > > > > >having > > > > >> > Kafka Stream in the same repo makes it easier for testing when > > those > > > > >core > > > > >> > functionalities evolve over time. Kafka Connect is in the same > > > > >situation. > > > > >> > > > > > >> > For rest proxy, whether it's widely used or not is going to be a > > bit > > > > >> > subjective. However, there are likely places that can live > > without a > > > > >rest > > > > >> > proxy. The rest proxy is just a proxy for the regular clients > and > > > > >doesn't > > > > >> > need to be tightly integrated with Kafka core. So, the case for > > > > >including > > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka > > Stream > > > > >>and > > > > >> > Kafka Connect. > > > > >> > > > > > >> > Thanks, > > > > >> > > > > > >> > Jun > > > > >> > > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce > > > > >> > > > > >> > wrote: > > > > >> > > > > > >> > > So from my reading essentially the first question needs to > > > > >answered/and > > > > >> > > voted on is: > > > > >> > > > > > > >> > > Is Apache Kafka Community only about the Core or does the > apache > > > > >> > community > > > > >> > > also support some subprojects (and just we need some better
Re: [DISCUSS] KIP-80: Kafka REST Server
t; > >Can the streaming platform exist without clients? - We at least > need > > > one > > > > > >client library to complete the platform. Our Java clients help us > to > > > > > >complete the platform story. The rest of the clients are built and > > > > > >maintained outside the project. > > > > > > > > > > > >Can the platform exist without the rest proxy? - Yes. The proxy > does > > > not > > > > > >complete the platform vision in anyway. It is just a good to have > > tool > > > > > >that > > > > > >might be required by quite a few users and there is an active > > project > > > > that > > > > > >works on this - https://github.com/confluentinc/kafka-rest > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis > > > > > > > > > > > >wrote: > > > > > > > > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and > > Kafka > > > > > >Connect > > > > > >> are not subjective? > > > > > >> > > > > > >> > "there are likely places that can live without a rest proxy" > > > > > >> > > > > > >> There are also places that can live without Kafka Streams and > > Kafka > > > > > >> Connect. > > > > > >> > > > > > >> Nacho > > > > > >> > > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao > > wrote: > > > > > >> > > > > > >> > At the high level, I think ideally it makes sense to add a > > > component > > > > > >>to > > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight > > > > > >integration > > > > > >> > with Kafka core. For Kafka Stream, we do expect stream > > processing > > > > will > > > > > >be > > > > > >> > used widely in the future. Implementation wise, Kafka Stream > > only > > > > > >> supports > > > > > >> > getting data from Kafka and leverages quite a few of the core > > > > > >> > functionalities in Kafka core. For example, it uses customized > > > > > >>rebalance > > > > > >> > callback in the consumer and uses the compacted topic heavily. > > So, > > > > > >having > > > > > >> > Kafka Stream in the same repo makes it easier for testing when > > > those > > > > > >core > > > > > >> > functionalities evolve over time. Kafka Connect is in the same > > > > > >situation. > > > > > >> > > > > > > >> > For rest proxy, whether it's widely used or not is going to > be a > > > bit > > > > > >> > subjective. However, there are likely places that can live > > > without a > > > > > >rest > > > > > >> > proxy. The rest proxy is just a proxy for the regular clients > > and > > > > > >doesn't > > > > > >> > need to be tightly integrated with Kafka core. So, the case > for > > > > > >including > > > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka > > > Stream > > > > > >>and > > > > > >> > Kafka Connect. > > > > > >> > > > > > > >> > Thanks, > > > > > >> > > > > > > >> > Jun > > > > > >> > > > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce > > > > > >> > > > > > >> > wrote: > > > > > >> > > > > > > >> > > So from my reading essentially the first question needs to > > > > > >answered/and > > > > > >> > > voted on is: > > > > > >> > > > > > > > >> > > Is Apache Kafka Community only about the Core or does the > > apache > > > > > >> > community > > > > > >> > > al
Re: [DISCUSS] KIP-80: Kafka REST Server
exist without clients? - We at least > > need > > > > one > > > > > > >client library to complete the platform. Our Java clients help > us > > to > > > > > > >complete the platform story. The rest of the clients are built > and > > > > > > >maintained outside the project. > > > > > > > > > > > > > >Can the platform exist without the rest proxy? - Yes. The proxy > > does > > > > not > > > > > > >complete the platform vision in anyway. It is just a good to > have > > > tool > > > > > > >that > > > > > > >might be required by quite a few users and there is an active > > > project > > > > > that > > > > > > >works on this - https://github.com/confluentinc/kafka-rest > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis > > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and > > > Kafka > > > > > > >Connect > > > > > > >> are not subjective? > > > > > > >> > > > > > > >> > "there are likely places that can live without a rest proxy" > > > > > > >> > > > > > > >> There are also places that can live without Kafka Streams and > > > Kafka > > > > > > >> Connect. > > > > > > >> > > > > > > >> Nacho > > > > > > >> > > > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao > > > wrote: > > > > > > >> > > > > > > >> > At the high level, I think ideally it makes sense to add a > > > > component > > > > > > >>to > > > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs > tight > > > > > > >integration > > > > > > >> > with Kafka core. For Kafka Stream, we do expect stream > > > processing > > > > > will > > > > > > >be > > > > > > >> > used widely in the future. Implementation wise, Kafka Stream > > > only > > > > > > >> supports > > > > > > >> > getting data from Kafka and leverages quite a few of the > core > > > > > > >> > functionalities in Kafka core. For example, it uses > customized > > > > > > >>rebalance > > > > > > >> > callback in the consumer and uses the compacted topic > heavily. > > > So, > > > > > > >having > > > > > > >> > Kafka Stream in the same repo makes it easier for testing > when > > > > those > > > > > > >core > > > > > > >> > functionalities evolve over time. Kafka Connect is in the > same > > > > > > >situation. > > > > > > >> > > > > > > > >> > For rest proxy, whether it's widely used or not is going to > > be a > > > > bit > > > > > > >> > subjective. However, there are likely places that can live > > > > without a > > > > > > >rest > > > > > > >> > proxy. The rest proxy is just a proxy for the regular > clients > > > and > > > > > > >doesn't > > > > > > >> > need to be tightly integrated with Kafka core. So, the case > > for > > > > > > >including > > > > > > >> > rest proxy in Apache Kafka is probably not as strong as > Kafka > > > > Stream > > > > > > >>and > > > > > > >> > Kafka Connect. > > > > > > >> > > > > > > > >> > Thanks, > > > > > > >> > > > > > > > >> > Jun > > > > > > >> > > > > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce > > > > > > >> > > > > > > >> > wrote: > > > > > >
Re: [DISCUSS] KIP-80: Kafka REST Server
sing frameworks out > > > there, > > > > > and > > > > > > > >they all have integration into Kafka. > > > > > > > >It doesn't need to be developed within Kafka. > > > > > > > > > > > > > > > > > > > > > > > >"Can the platform exist without the rest proxy? - Yes. The > proxy > > > > does > > > > > > not > > > > > > > >complete the platform vision in anyway. It is just a good to > > have > > > > tool > > > > > > > >that > > > > > > > >might be required by quite a few users and there is an active > > > > project > > > > > > that > > > > > > > >works on this - https://github.com/confluentinc/kafka-rest"; > > > > > > > > > > > > > > > >The rest proxy is as important as any API. The vision that > shown > > > > here > > > > > > > >http://kafka.apache.org/intro > > > > > > > >require users to write the producers and consumers to get > their > > > data > > > > > > into > > > > > > > >and out of Kafka, without which having Streams or Connect > won't > > > help > > > > > > > >anyone. > > > > > > > >The rest proxy makes easier for users get their data into > Kafka. > > > > > > > >Adding the rest proxy to the project doesn't invalidate the > > > current > > > > > > > >vision, > > > > > > > >it only strengthens it. > > > > > > > > > > > > > > > >Thanks, > > > > > > > >Harsha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian < > > > > r...@confluent.io> > > > > > > > >wrote: > > > > > > > > > > > > > > > >FWIW, Apache Kafka has evolved a lot from where it started. It > > did > > > > > start > > > > > > > >as > > > > > > > >a messaging system. Over time we realized that that the vision > > for > > > > > Kafka > > > > > > > >is > > > > > > > >to build a streaming platform and not just a messaging system. > > You > > > > can > > > > > > > >take > > > > > > > >a look at the site for more description about what comprises > the > > > > > > streaming > > > > > > > >platform http://kafka.apache.org/ and > > > http://kafka.apache.org/intro > > > > . > > > > > > > > > > > > > > > >Can the streaming platform exist without Connect? - No. Data > > > > > integration > > > > > > > >is > > > > > > > >fundamental to building an end to end platform > > > > > > > > > > > > > > > >Can the streaming platform exist without stream processing? - > > No. > > > > > > > >Processing stream data again is a core part of streaming > > platform. > > > > > > > > > > > > > > > >Can the streaming platform exist without clients? - We at > least > > > need > > > > > one > > > > > > > >client library to complete the platform. Our Java clients help > > us > > > to > > > > > > > >complete the platform story. The rest of the clients are built > > and > > > > > > > >maintained outside the project. > > > > > > > > > > > > > > > >Can the platform exist without the rest proxy? - Yes. The > proxy > > > does > > > > > not > > > > > > > >complete the platform vision in anyway. It is just a good to > > have > > > > tool > > > > > > > >that > > > > > > > >might be required by quite a few users and there is an active > > > > project > > > > > > that > > > > > > > >works on this - https://github.com/confluentinc/kafka-rest > > > &