Tom - I understand the desire to create bindings outside the core. The point I was trying to make earlier around version semantics and testing was to 'Hedge' the risk. It basically creates a contract between core & framework+bindings writers.
No one ever intends to break compatibility, but it happens all the time and usually in some very subtle ways at first. A great example of this is a patch I recently submitted to Mesos where the cgroup code was writing an extra <<endln out. Earlier versions of the kernel had no issue with this, but recent modifications would cause the cgroup code to fail. Very subtle, and boom-goes-the-dynamite. Below was an email I sent a while back, that outlines a possible hedge/contract. Please let me know what you think. -------------------------- > > Greetings! > > I've conversed with folks about the idea of having a more formalized release > and branching strategy, such that others who are downstream can rely on > certain version semantics when planning upgrades, etc. This becomes doubly > important as we start to trend towards a 1.0 release, and folks will depend > heavily on it for their core infrastructure, and APIs (Frameworks, and EC). > > Therefore, I wanted to propose a more formalized branching and release > strategy, and see what others think. I slightly modified this pattern from > the Condor & Kernel projects, which have well established processes. > > ------------------------------ > Basic Idea: > > 1.) Create 2 Main Branches (Stable/Devel-Master based) > 2.) Devel releases are cadence/time based and lightly tested. > 3.) Stable series only accepts bug fixes. Merge path for all bug fixes > deemed worthy, are through the stable series up to master. > 4.) @ some point devel goes through a *hardning phase* and becomes the new > stable. > > ------------------------------ > Version Semantics: > > Major.Minor.Revision-PatchBuild > > Major: > - Compatibility breakage (usually protocol or api shift), or enough minors > to justify change. If there is a major version change it should be taken with care and notify downstream usually through the mailing lists. > > Minor: > - Devel (Odd) - 1.1.x > - Stable (Even) - 1.0.x > > Revision: > - Devel - Cadence # Some set of feature enhancements > - Stable - Bug and security fixes only (Higher bar of entry) > > PatchBuild: > - Upstream - Whoops our bad, we found a bug or two > - Downstream - Back-port build variant. > > ------------------------------ > Series/Branches: > > Development Series - (Odd Minor #'s): 1.1.x > The development series branches/tags are cadence based, and come off of > master. All new features are added to master. All bug fixes should be > merged through the stable series into the master. It should be ok to > introduce destabilizing features from time to time, provided its agreed upon > by a Sheppard. > > Stable Series - (Even Minor #'s): 1.0.x > Stable series should *only contain* bug fixes. This way, downstream folks > have a common understanding that behavior should be maintained. Should > downstream folks wish to back-port features, they can do that at their own > risk. Every release of the stable series has some measure of quality > then > a +1. E.g. running some clusters for a period of time (X), > In this model, stable series should be "stable" for writers against the API(s). > Transition from Devel-> Stable: > After some point, the development series needs to go through a hardening > phase. This could include static analysis + running on some production > cluster for a period of time. Folks typically plan the transition around a > conference series in order to announce the cool new features. + You could test the bindings during this phase ^ but for stable series they should just work. > ------------------------------ Cheers, Tim ----- Original Message ----- > From: "Tom Arnfeld" <t...@duedil.com> > To: user@mesos.apache.org > Sent: Tuesday, July 15, 2014 2:50:47 AM > Subject: Re: Mesos language bindings in the wild > Hey Tim, > I can see your point, and am finding it hard to think of any compelling > arguments against the issue of fragmentation, but I do have a few thoughts.p > That said, I would strongly suggest taking ease-of-use and language specific > code structures into consideration. A huge monolithic build system might not > be a good thing either, if I'm not mistaken that's why twitter built Pants. > Spark is actually a great example here, it's going to be a huge pain to > publish PySpark to PYPI because of the way they structure the code, unless > you force users to use a bleeding edge version of setuptools to be able to > install the software. In the case of PySpark (and other libraries that > require compiled dependencies, see Hadoofus on github which I collaborated > on this exact issue). It's a nightmare. Projects that work well with python > setuptools are projects that are just python, from my experience. > That said, it's only a nightmare when you *have* precompiled dependencies > that need to be part of the build process. This is no longer the case with > the new mesos bindings, so why make it so hard? > Take Go as another example (this is similar to installing pip dependencies > from github too) - a user can simply plug in the path to a repository and > away they go. It's easy, and will rapidly speed up adoption IMO. This isn't > something that can easily be done if it's not in it's own repo, and the > Mesos repository is pretty huge now. > My opinion is largely from a users perspective. However, I would ask the > question - how often does the framework API change in such a way that it > breaks compatibility? Will there be a need to orchestrate releases among 20 > language bindings to get a release of the core out, how often? Would it be > easier for developers to implement a change and also make that change across > all languages at the same time, is that even really going to happen? > It's also worth considering release cycles, with all bindings being built > into the core, it requires them all the be release together (or it's a git > tag pain). Given that lots of the bindings are going to be (and already are) > community driven, and only a few people are in charge of the Mesos release > cycle (taking at least a few weeks for a release to come out) the pace for > each binding has to be the same, and there's no autonomy. > My personal feeling is that develop user experience isn't thought about > enough is these sorts of situations, and not having a good experience either > to use or work on the code is a pain and can slow down adoption. > Would be interested to hear what you all think, or if you completely disagree > :-) > Tom. > On Tuesday, 15 July 2014, Tim St Clair < tstcl...@redhat.com > wrote: > > So... your response basically capitulates to the fragmentation argument: > > > "Yes we will have binding strewn about of questionable quality that may, or > > may not, work with core." > > > The point that I'm trying to make is, fragmentation *is not* a good thing. > > > -------------------------------------- > > > Case in point - The Hadoop Ecosystem (fragmentation) > > > In order for anyone to make a salient stack of any measure, vendors have to > > knit together components into a stack which can then be consumed by the > > masses. > > > -------------------------------------- > > > Counterpoint - Spark (curating) libraries > > > Spark bundles 1st order interface libraries as part of a curated core. You > > are guaranteed that the core will inter-operate, and PySpark is given 1st > > class standing. > > > -------------------------------------- > > > This is a bad idea, unless there is a plan to hedge the risk. > > > -Tim > > > ----- Original Message ----- > > > > From: "yifan" < myan...@msn.com > > > > > To: user@mesos.apache.org > > > > Sent: Monday, July 14, 2014 7:10:34 PM > > > > Subject: Re: Mesos language bindings in the wild > > > > > > > > Hi Tim, > > > > > > > > I found that in zookeeper, they also separate the bindings from the core. > > > > > > > > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings > > > > > > > > So, IMHO, I think it should be the maintainer's responsibility to keep > > > > the binding in healthy state, with clear documentation of which version > > > > of the mesos core they supports. > > > > > > > > Yifan > > > > > > > > > > > > > > > > On 07/14/2014 11:30 AM, Tim St Clair wrote: > > > > > So I fear the fragmentation that can occur if we provide native > > > > bindings > > > > > outside of the core, unless there is some mechanism for testing, & a > > > > well > > > > > established versioning scheme. > > > > > > > > > > IMHO, priority inversion on 'versioning' should come before bindings to > > > > > ensure we adhere to policy. > > > > > > > > > > Thoughts? > > > > > > > > > > -Tim > > > > > > > > > > > > > > > ----- Original Message ----- > > > > >> From: "Tom Arnfeld" < t...@duedil.com > > > > > >> To: d...@mesos.apache.org > > > > >> Cc: user@mesos.apache.org > > > > >> Sent: Friday, July 11, 2014 10:22:59 AM > > > > >> Subject: Re: Mesos language bindings in the wild > > > > >> > > > > >> Very exciting. I'd vote +1 for splitting them out. Especially if you > > > > >> look at the common way of using Go imports, just stick the project on > > > > >> GitHub and import it directly using " github.com/mesos/mesos-go " or > > > > >> similar. > > > > >> > > > > >> I guess one argument is that you have more fragmentation of the code > > > > >> (e.g every library has it's own copy of the protos) but I'm not sure > > > > >> that's a bad thing. > > > > >> > > > > >> Just my two cents. Looking forward to this! > > > > >> > > > > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg < tho...@saunter.org > > > > >>> wrote: > > > > >>> > > > > >>> I've started preparing the python bindings to hopefully take this > > > > >>> route ( https://reviews.apache.org/r/23224/ would love some reviews! > > > > >>> ). In fact, there is already a native python implementation of both > > > > >>> libprocess and the framework apis! ( > > > >>> https://github.com/wickman/pesos/ > > > > >>> , https://github.com/wickman/compactor ). > > > > >>> > > > > >>> What are the benefits of bindings being part of the project source > > > > >>> itself instead of having blessed implementations like mesos-python > > > > >>> where the source and versioning becomes separate? I've been running > > > > >>> into difficulties making automake and python's build tools play > > > >>> nicely > > > > >>> together. It seems like there'd be more flexibility in general by > > > > >>> splitting them out. > > > > >>> > > > > >>> > > > > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen < > > > >>>> nik...@mesosphere.io > > > >>>> > > > > > >>>> wrote: > > > > >>>> I just wanted to clarify - native, meaning _no_ dependency to > > > >>>> libmesos > > > > >>>> and > > > > >>>> native to its language (only Go, only Python and so on) i.e. use the > > > > >>>> low-level API. > > > > >>>> > > > > >>>> Sorry for the confusion, > > > > >>>> Niklas > > > > >>>> > > > > >>>> > > > > >>>>> On 10 July 2014 15:55, Dominic Hamon < dha...@twopensource.com > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>> In my dream world, we wouldn't need any native bindings. I can > > > >>>>> imagine > > > > >>>>> having example frameworks or starter frameworks that use the > > > >>>>> low-level > > > > >>>>> API > > > > >>>>> (the wire protocol with protocol buffers for message passing), but > > > > >>>>> nothing > > > > >>>>> like we have that needs C or JNI, etc. > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen < > > > >>>>> nik...@mesosphere.io > > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>>> Hi all, > > > > >>>>>> > > > > >>>>>> I wanted to start a discussion around the language bindings in the > > > > >>>>>> wild > > > > >>>>>> (Go, Haskell, native Python, Go, Java and so on) and possibly get > > > >>>>>> to > > > >>>>>> a > > > > >>>>>> strategy where we start bringing those into Mesos proper. As most > > > > >>>>>> things > > > > >>>>>> points towards, it will probably make sense to focus on the native > > > > >>>>>> "bindings" leveraging the low-level API. To name one candidate to > > > > >>>>>> start > > > > >>>>>> with, we are especially interested in getting Go native support in > > > > >>>>>> Mesos > > > > >>>>>> proper (and in a solid state). So Vladimir, we'd be super thrilled > > > >>>>>> to > > > > >>>>> start > > > > >>>>>> collaborating with you on your current work. > > > > >>>>>> > > > > >>>>>> We are interested to hear what thoughts you all might have on > > > >>>>>> this. > > > > >>>>>> > > > > >>>>>> Thanks, > > > > >>>>>> Niklas > > > > > > > > > > > > -- > > > > Gu Yifan > > > > > > > > > > > -- > > > Cheers, > > > Timothy St. Clair > > > Red Hat Inc. > -- Cheers, Timothy St. Clair Red Hat Inc.
