Hi, To be honest, I'm not sure I understand who is the "we" in the "We're going to post ...". If there are any discussions or plans to replace Zookeeper, then I think they should be done here on the developer list instead of teasing people for several months with some secretive projects.
Thanks & Regards Jakub On Mon, May 14, 2018 at 9:57 AM Molnár Bálint <molnarcsi...@gmail.com> wrote: > Hi Colin, > > Do you have a rough estimate on that? > > Thanks, > Balint > > Colin McCabe <cmcc...@apache.org> ezt írta (időpont: 2018. máj. 9., Sze, > 19:53): > > > Hi Molnar, > > > > The points Ismael brought up earlier (and that were brought up on KIP-30) > > are still relevant here. As Ismael said, the goal is to get rid of > > external dependencies here. We're going to post more about this soon > > (sorry for the delay) > > > > thanks, > > Colin > > > > > > On Wed, May 9, 2018, at 07:29, Molnár Bálint wrote: > > > Hi, > > > I just rebased the Etcd implementation proposal on trunk. Pinging to > see > > if > > > anyone has feedback on my questions from my previous email. > > > > > > Molnár Bálint <molnarcsi...@gmail.com> ezt írta (időpont: 2018. ápr. > 4., > > > Sze, 10:08): > > > > > > > Hi, > > > > Thanks again for the feedback. > > > > > > > > Is there already ongoing work for having an own consensus > > implementation > > > > within Kafka? > > > > If that work haven't started yet, we think there is value in having > an > > > > interim solution, that allows the use of another consensus system > > besides > > > > Zookeeper. > > > > > > > > We ask the community to take a look at the Etcd implementation > proposal > > > > we created and provide feedback on that. > > > > This helps to asses rather this approach is viable at all. > > > > > > > > We are open to collaborate on integrating our proposed Etcd > > implementation > > > > into any integration test system, to certify that all use cases works > > as > > > > expected. > > > > > > > > Balint > > > > > > > > 2018-03-30 22:21 GMT+02:00 Gwen Shapira <g...@confluent.io>: > > > > > > > >> Hi, > > > >> > > > >> I had an offline discussion with Ismael and wanted to summarize the > > > >> comments and questions he raised so we are all on the same page. > > > >> > > > >> The core issue is that this change adds a new public API. Since we > > already > > > >> know that the goal for the next 1-2 years is to get rid of ZK > > completely. > > > >> Do we want to go to the effort of adding (and discussing and > > reviewing) a > > > >> new public API, knowing that it will be completely removed in a > year? > > And > > > >> since building and testing a plugin also involves effort, will > anyone > > do > > > >> it > > > >> for something that is going to be temporary by design? > > > >> > > > >> Ismael, correct me if this isn't a fair representation of your > > concerns. > > > >> > > > >> Gwen > > > >> > > > >> > > > >> > > > >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io> > > wrote: > > > >> > > > >> > Few other concerns that were raised in the previous discussion > were > > > >> around > > > >> > the challenges both to maintainers and users in making this API > > > >> pluggable > > > >> > and how does making the interface pluggable aligns with future > > goals for > > > >> > the project. At the time this was difficult to discuss because > there > > > >> wasn't > > > >> > a concrete proposal. I want to discuss these points in the context > > of > > > >> this > > > >> > specific proposal: > > > >> > > > > >> > 1. Problem: Pluggable APIs mean larger surface testing area and > > multiple > > > >> > implementations to cover. > > > >> > In this case: At the time, the Kafka project didn't have much > > > >> > experience with pluggable APIs and components, so the concerns > were > > very > > > >> > valid. Right now Kafka has multiple pluggable components - > > Connectors, > > > >> > converters, transformations, authentication protocols, > authorization > > > >> > database, coordination protocol, serializers, etc. I think that > as a > > > >> > community we gotten better at testing the interface, testing the > > very > > > >> few > > > >> > implementations that are included in Apache Kafka itself and > > allowing > > > >> the > > > >> > community to innovate and validate outside of the Kafka project. I > > don't > > > >> > recall major issues either from lack of testing or from usability > > > >> > perspective. > > > >> > > > > >> > 2. Problem: Users don't want to choose a consensus implementation, > > they > > > >> > just don't want ZK. > > > >> > In this case: I agree that users don't actually want to spend > > time > > > >> > choosing consensus implementation and a simpler deployment model > > would > > > >> > serve them better. IMO, if Apache Kafka ships with our well-tested > > ZK > > > >> > implementation, 99% of the users will choose to use that (a vast > > > >> majority > > > >> > uses our less-than-amazing authorization plugin), and the few that > > > >> really > > > >> > need something else for whatever reason, will be able to get what > > they > > > >> > need. As Jake said, we need to face the fact that development > > > >> trajectory of > > > >> > ZK isn't amazing at the moment, that it is lacking features our > > users > > > >> need > > > >> > (SSL) and it will be good to allow the user community to explore > > > >> > alternatives. > > > >> > > > > >> > 3. Problem: Why got to the effort of refactoring if we know we > want > > to > > > >> get > > > >> > rid of ZK. > > > >> > In this case: This change isn't huge, it doesn't rewrite large > > > >> > portions of Kafka and it does not make the future direction any > more > > > >> > difficult. If in 2 weeks or 2 month or 2 years we'll have a > ZK-less > > > >> > solution, applying it on Kafka with this KIP isn't any more > > challenging > > > >> > than applying it to Kafka without this KIP. It is a step in an > > > >> orthogonal > > > >> > direction, but not opposite direction. I think that letting the > > perfect > > > >> > become the enemy of the good is a repeated failure mode in this > > > >> community. > > > >> > Can we discuss whether this proposal is good even if there is a > more > > > >> > complex proposal that may be better? As long as they don't > conflict? > > > >> > > > > >> > Gwen > > > >> > > > > >> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint < > > molnarcsi...@gmail.com> > > > >> > wrote: > > > >> > > > > >> >> Thanks, for the feedback. > > > >> >> > > > >> >> Developing an internal consensus service inside Kafka would > > require a > > > >> team > > > >> >> dedicated to this task. > > > >> >> We second what Flavio said in > > > >> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f > > > >> >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@% > > 3Cdev.kafka.apache.org > > > >> %3E > > > >> >> that > > > >> >> getting an implementation which really works and is maintainable > is > > > >> >> difficult. > > > >> >> We think that Kafka being able to use another widely used > consensus > > > >> system > > > >> >> beside Zookeeper its a safer and workable solution. > > > >> >> It will be easier for users to use a consensus system with Kafka > > that > > > >> they > > > >> >> are already familiar with. > > > >> >> > > > >> >> > > > >> >> The implementation found here: > > > >> >> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/ > > > >> >> kafka-on-etcd/core/src/main/scala/kafka/etcd > > > >> >> is a first version of enabling Etcd in Kafka. > > > >> >> This implementation hooked in Etcd with a slight change in the > > existing > > > >> >> interfaces. While this implementation works its far from being > > > >> complete. > > > >> >> Ideally existing interfaces should be reworked to abstract out > the > > used > > > >> >> consensus system. > > > >> >> We opened this KIP to start a discussion and the community to > have > > a > > > >> look > > > >> >> at the initial implementation and receive feedback if this > > initiative > > > >> is > > > >> >> viable. > > > >> >> > > > >> >> Balint > > > >> >> > > > >> >> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>: > > > >> >> > > > >> >> > I can understand the concerns about the plugability of > > > >> Zookeeper/Etcd. > > > >> >> It > > > >> >> > would not be good for Kafka community if it splits into several > > > >> groups > > > >> >> > using different implementations. > > > >> >> > > > > >> >> > On the other hand, Zookeeper development seems to be a bit > > stalled. > > > >> So > > > >> >> > maybe there should be at least a discussion whether it makes > > sense to > > > >> >> > replace Zookeeper with something like Etcd or not. > > > >> >> > > > > >> >> > JAkub > > > >> >> > > > > >> >> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint < > > > >> molnarcsi...@gmail.com> > > > >> >> > wrote: > > > >> >> > > > > >> >> > > Hi all, > > > >> >> > > > > > >> >> > > I have created KIP-273: Kafka to support using ETCD beside > > > >> Zookeeper > > > >> >> > > > > > >> >> > > Here is the link to the KIP: > > > >> >> > > > > > >> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > >> >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper > > > >> >> > > > > > >> >> > > Looking forward to the discussion. > > > >> >> > > > > > >> >> > > Thanks, > > > >> >> > > Balint > > > >> >> > > > > > >> >> > > > > >> >> > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > *Gwen Shapira* > > > >> > Product Manager | Confluent > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap > > > >> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > > >> > <http://www.confluent.io/blog> > > > >> > > > > >> > > > > >> > > > >> > > > >> -- > > > >> *Gwen Shapira* > > > >> Product Manager | Confluent > > > >> 650.450.2760 | @gwenshap > > > >> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > > >> <http://www.confluent.io/blog> > > > >> > > > > > > > > > > >
