On 03/15/2017 11:37 AM, Davanum Srinivas wrote: > Monty, Team, > > Sorry for the top post: > > Support for etcd/tooz in devstack (with file driver as default) - > https://review.openstack.org/#/c/445432/ > > As of right now both zookeeper driver and etcd driver is working fine: > https://review.openstack.org/#/c/445630/ > https://review.openstack.org/#/c/445629/ > > The problem we have from before is that we do not have any CI jobs > that used zookeeper. > > I am leaning towards just throwing the etcd as default and if folks > are interested in zookeeper then they can add specific CI jobs with > DLM_BACKEND variable set.
That doesn't bother me - zk as the default choice was because at the time zk worked and etcd did not. That said - etcd3 is a newer/better thing - so maybe instead of driving etcd home as a default before we add etcd3 support, we just change tooz to support etcd3, add the devstack jobs to use that, and start from a position that doesn't involve dealing with any legacy? > On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor <mord...@inaugust.com> wrote: >> On 03/15/2017 03:13 AM, Jay Pipes wrote: >>> On 03/14/2017 05:01 PM, Clint Byrum wrote: >>>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400: >>>>> On 03/14/2017 02:50 PM, Julien Danjou wrote: >>>>>> On Tue, Mar 14 2017, Jay Pipes wrote: >>>>>> >>>>>>> Not tooz, because I'm not interested in a DLM nor leader election >>>>>>> library >>>>>>> (that's what the underlying etcd3 cluster handles for me), only a >>>>>>> fast service >>>>>>> liveness/healthcheck system, but it shows usage of etcd3 and Google >>>>>>> Protocol >>>>>>> Buffers implementing a simple API for liveness checking and host >>>>>>> maintenance >>>>>>> reporting. >>>>>> >>>>>> Cool cool. So that's the same feature that we implemented in tooz 3 >>>>>> years ago. It's called "group membership". You create a group, make >>>>>> nodes join it, and you know who's dead/alive and get notified when >>>>>> their >>>>>> status change. >>>>> >>>>> The point of os-lively is not to provide a thin API over ZooKeeper's >>>>> group membership interface. The point of os-lively is to remove the need >>>>> to have a database (RDBMS) record of a service in Nova. >>>> >>>> That's also the point of tooz's group membership API: >>>> >>>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping >>> >>> Did you take a look at the code I wrote in os-lively? What part of the >>> tooz group membership API do you think I would have used? >>> >>> Again, this was a weekend project that I was moving fast on. I looked at >>> tooz and didn't see how I could use it for my purposes, which was to >>> store a versioned object in a consistent key/value store with support >>> for transactional semantics when storing index and data records at the >>> same time [1] >>> >>> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511 >>> >>> >>> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional >>> semantics in a consistent key/value store that I needed. >>> >>> tooz is cool, but it's not what I was looking for. It's solving a >>> different problem than I was trying to solve. >>> >>> This isn't a case of NIH, despite what Julien is trying to intimate in >>> his emails. >>> >>>>> tooz simply abstracts a group membership API across a number of drivers. >>>>> I don't need that. I need a way to maintain a service record (with >>>>> maintenance period information, region, and an evolvable data record >>>>> format) and query those service records in an RDBMS-like manner but >>>>> without the RDBMS being involved. >>>>> >>>>>>> servicegroup API with os-lively and eliminate Nova's use of an >>>>>>> RDBMS for >>>>>>> service liveness checking, which should dramatically reduce the >>>>>>> amount of both >>>>>>> DB traffic as well as conductor/MQ service update traffic. >>>>>> >>>>>> Interesting. Joshua and Vilob tried to push usage of tooz group >>>>>> membership a couple of years ago, but it got nowhere. Well, no, they >>>>>> got >>>>>> 2 specs written IIRC: >>>>>> >>>>>> >>>>>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html >>>>>> >>>>>> >>>>>> But then it died for whatever reasons on Nova side. >>>>> >>>>> It died because it didn't actually solve a problem. >>>>> >>>>> The problem is that even if we incorporate tooz, we would still need to >>>>> have a service table in the RDBMS and continue to query it over and over >>>>> again in the scheduler and API nodes. >>>> >>>> Most likely it was designed with hesitance to have a tooz requirement >>>> to be a source of truth. But it's certainly not a problem for most tooz >>>> backends to be a source of truth. Certainly not for etcd or ZK, which >>>> are both designed to be that. >>>> >>>>> I want all service information in the same place, and I don't want to >>>>> use an RDBMS for that information. etcd3 provides an ideal place to >>>>> store service record information. Google Protocol Buffers is an ideal >>>>> data format for evolvable versioned objects. os-lively presents an API >>>>> that solves the problem I want to solve in Nova. tooz didn't. >>>> >>>> Was there something inherent in tooz's design that prevented you from >>>> adding it to tooz's group API? Said API already includes liveness (watch >>>> the group that corresponds to the service you want). >>> >>> See above about transactional semantics. >>> >>> I'm actually happy to add an etcd3 group membership driver to tooz, >>> though. After the experience gained this weekend using etcd3, I'd like >>> to do that. >>> >>> Still doesn't mean that tooz would be the appropriate choice for what I >>> was trying to do with os-lively, though. >>> >>>> The only thing missing is being able to get groups and group members >>>> by secondary indexes. etcd3's built in indexes by field are pretty nice >>> >>> Not sure what you're talking about. etcd3 doesn't have any indexing by >>> field. I built the os-lively library primarily as a well-defined set of >>> index overlays (by uuid, by host, by service type, and by region) over >>> etcd3's key/value store. >>> >>>> for that, but ZK can likely also do it too by maintaining the index in >>>> the driver. >>> >>> Maybe, I'm not sure, I didn't spend much time this weekend looking at >>> ZooKeeper. >> >> a) awesome. when the rest of this dips momentarily into words that might >> sound negative, please hear it all wrapped in an "awesome" and know that >> my personal desire is to see the thing you're working on be successful >> without undue burden... >> >> b) In Tokyo, we had the big discussion about DLMs (where at least my >> intent going in to the room was to get us to pick one and only one). >> There were three camps in the room who were all vocal: >> >> 1) YES! Let's just pick one, I don't care which one >> 2) I hate Java I don't want to run Zookeeper, so we can't pick that >> 3) I hate go/don't trust coreos I don't want to run etcd so we can't >> pick that >> >> Because of 2 and 3 the group represented by 1 lost and we ended up with: >> "crap, we have to use an abstraction library" >> >> I'd argue that unless something has changed significantly, having Nova >> grow a direct depend on etcd when the DLM discussion brought us to "the >> operators in the room have expressed a need for a pluggable choice >> between at least zk and etcd" should be pretty much a non-starter. >> >> Now, being that I was personally in group 1, I'd be THRILLED if we >> could, as a community, decide to pick one and skip having an abstraction >> library. I still don't care which one - and you know I love gRPC/protobuf. >> >> But I do think that given the anti-etcd sentiment that was expressed was >> equally as vehement as the anti-zk sentiment, that we need to circle >> back and make a legit call on this topic. >> >> If we can pick one, I think having special-purpose libraries like >> os-lively for specific purposes would be neat. >> >> If we still can't pick one, then I think adding the liveness check you >> implemented for os-lively as a new feature in tooz and also implementing >> the same thing in the zk driver would be necessary. (of course, that'll >> probably depend on getting etcd3 support added to tooz and making sure >> there is a good functional test for etcd3. >> >> c) awesome >> >>>> I understand abstractions can seem pretty cumbersome when you're moving >>>> fast. It's not something I want to see stand in your way. But it would >>>> be nice to see where there's deficiency in tooz so we can be there for >>>> the next project that needs it and maybe eventually factor out direct >>>> etcd3 usage so users who have maybe chosen ZK as their tooz backend can >>>> also benefit from your work. >>> >>> It's not a deficiency in tooz. It's a different problem domain. Look at >>> the os-lively API and show me how you think I could have used tooz to >>> implement that API. >> >> I think I already said this above - but what I was reading/hearing is >> "why not add add the feature you need to tooz" ... not "tooz does that >> already" - however, as you said, you were doing quick weekend POC work, >> so it's possible that adding this to tooz is a next step. >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
