Hey Connor, It would be great to hear back about this and get the ball rolling on a shared registry. Let me know what you think!
Thanks, Dave On Wed, Dec 3, 2014 at 1:11 PM, Dave Lester <daveles...@gmail.com> wrote: > > Hi Connor, > > Thanks for sharing the work you've been doing with Universe. Sounds > complimentary and pretty exciting! I thought I'd take a stab at making some > clarifications. > > To rephrase your proposal by using a comparison, you're proposing the > "homebrew model" of package management here in terms of how the registry is > maintained, correct? A registry of metadata that's hosted via a single > GitHub repository, that community members submit pull requests to when they > want to have a release included in the registry. > > In this case of Homebrew, they manage "formulas"; in our case it would be > package.json files (which is how NodeJS handles package info). > > *Architecture* > The only way Universe diverges from my original thinking is that you're > advocating decentralization of package metadata, whereas my original > thought was to have it much more centralized. There's no reason it couldn't > be brought together through a single interface (command-line, or searchable > web interface) so I don't think it's incompatible technically, although I > do think it creates some potential social/governance overhead (more below). > > *Metadata* > The package.json file is minimal and similar to what you see in most > package management systems. Where you're taking it a step further is a JSON > manifest for how to actually run the program, which is outside of the scope > of what I was hoping to accomplish but it looks awesome. > > *Managing The Registry* > One part of your proposal that's unclear to me is when information is > pushed to the registry repository, and how often (for each release, or just > when the package is first added). Additionally, what information is pushed > here -- just a pointer to the git repo? It's my current understanding that > the package.json and framework.json files are both within the actual > package's repo, not the registry. I would love to hear how you're thinking > about this. > > If it's just a pointer to another repo, it would make it possible for any > package already vetted to become part of the registry to do their own > updates without requiring someone to accept a PR. I think this is ideal. If > it's the latter, it brings up some interesting questions related to who > would manage this and under what guidelines they would do this vetting; > we'd essentially get a PR for every release within the community at that > point. > > *The Registry "Home"* > I feel strongly that this should be a community-driven effort, and I see > your proposal along with my own as very compatible. Would love to work > together, and if this works as I hope it will be a huge driver of activity > in the ecosystem. > > Dave > > On Tue, Dec 2, 2014 at 1:43 AM, Jing Dong <j...@qubitdigital.com> wrote: > >> I like the idea of having a single UI for installing frameworks. One Web >> UI endpoint for managing setting and view state of multiple installed >> frameworks. >> >> >> On 2 Dec 2014, at 09:26, Billy Bones <gael.ther...@gmail.com> wrote: >> >> Wooo mesosphere is making such a good work pushing forward Mesos >> technologies! >> >> I deeply think that Mesos should implement registry as a core feature. >> >> Let me explain that, Installing manually frameworks / modules / other >> third party adds-on is quite cool when you're beginner and want to learn >> how the things works, but once you're in production, you've to be fast and >> react quickly. >> >> Let said that I'm a system engineer that have customers requests like >> "Hum... Could you had this " insert Framework/module/other name here" on >> our cluster region?", I'll have to add it as soon and quick as possible on >> the targeted environment, I'll be OK to do it twice, but I wont make my job >> a request nightmare and I'll probably search for a way to provide to my >> customers a restrictive access (WebUI) to the cluster manager and allow >> them to deploy those new add-on by themself. >> >> With this study case in mind, I'll be happy to have a registry which >> allow a filtering mechanism to hide or show frameworks/modules/etc by >> status (Official Apache Content / Public Content / Private Content). >> >> Here is my thoughts about the registry articulation: >> >> - Mesos Registry is an integrated part of Mesos master services. >> - Mesos Registry is an endpoint available to WebUI and CLI. >> - Mesos Registry is nothing but a metadata registry. >> - Mesos Registry save its configs and metadata on a key/value store >> (zookeeper?). >> - Mesos Registry is empty at the first launch. >> - Mesos Registry as three views: Official Apache Content | Publicly >> Maintained Content | Private Maintained Content >> - Mesos Registry views content is: >> - Official Apache Content == Link to existing Apache add-ons hosted >> on Github/Git repository + metadata (Like those proposed by Dave and >> Connor). >> - Publicly Maintained Content == Links to existing repositories >> (Github / Git / other) + metadata (Like those proposed by Dave and >> Connor). >> - Private Content == Links to existing private (GIT/GITHUB/Other >> repositories) + metadata (Like those proposed by Dave and Connor), this >> repository is kinda special as it is hosted and created by the cluster >> operators and could be a mixed content of locally maintained repository >> (GIT repos on a HDFS or TAR on HDFS) and public content repository >> cloned >> from the public content URL/Metadata. >> >> I don't know if I'm really clear, so if I'm not, let me know it, I'll do >> some sketches :D >> >> >> 2014-12-02 1:53 GMT+01:00 Connor Doyle <con...@mesosphere.io>: >> >>> Hi Dave, >>> >>> This is a timely topic, since we have been prototyping and mocking up >>> something similar at Mesosphere. We created a new public GitHub repository >>> for it about three weeks ago called "universe" ( >>> http://github.com/mesosphere/universe). >>> >>> Although we have added some informal specs, it's very malleable at this >>> point. We're very much interested in making our "universe" compatible >>> with, or the same as, the registry you're proposing. Without delving into >>> implementation details, some of the goals we have in mind are outlined >>> below. >>> >>> Data Source: >>> >>> The package repository should be easily consumable by third-party >>> command-line and other programs. There should be a condensed “index” >>> representation of the package repository available. >>> >>> Packages within the repository should be versioned. >>> >>> The package repository format itself should be versioned. >>> >>> Decentralization and Composability: >>> >>> The package metadata should be hosted in a public place (we like GitHub) >>> so that additional packages can be added by the community by simply >>> submitting pull requests. We have added some rudimentary commit hooks and >>> automated validation to protect the repo against breaking changes. >>> >>> It’s important that no single entity “owns the keys” to the universe, >>> and that the spec and implementation remain public. It should be easy and >>> free for organizations to maintain a private package repository. >>> >>> A corollary is that it should be easy for consumers to pull from a >>> hierarchy of upstream repositories. One setup we have in mind is that an >>> organization might have staging and production repositories running >>> internally. Packages are pushed to staging where integration testing can >>> run before “deployment” to production. If a package isn’t in the local >>> repository it might be looked up and installed from upstream. >>> >>> <hierarchy.png> >>> >>> >>> Repositories should be able to be proxied and cached in this way. >>> Organizations should be able to isolate their datacenter but also >>> selectively add external packages for experimentation. The system should be >>> sufficiently portable and extensible to accomodate these and similar use >>> cases. >>> >>> Meta-Framework Descriptors: >>> >>> Our conception of the package repository is a bit more expansive than >>> just Mesos frameworks; it includes descriptions of how to install any piece >>> of server software on a Mesos cluster. Frameworks and non-frameworks alike >>> may be installed using some other meta-framework that’s responsible for >>> starting all other cluster services. Likely candidates for this role are >>> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually >>> Kubernetes. In any case, the repository spec should not be prescriptive >>> with respect to this choice. >>> >>> The package repository metadata should make it easy for Mesos framework >>> authors (and authors of non-Mesos-aware programs) to describe how to >>> install their software on a Mesos cluster. To this end, our prototype >>> package spec allows for Meta-framework descriptor files for each package in >>> the repository. For example for a given package we might see a >>> `marathon.json` file as well as a `my-app.aurora` file. >>> >>> An obvious concern is how to specify site-specific arguments upon >>> installation. Here packages should describe data that must be marshalled >>> from the environment (e.g. by prompting a user) and combined with the raw >>> meta-framework descriptor to launch the app. These configuration >>> parameters should be agnostic of the supported meta-frameworks. More >>> concretely, in our prototype we describe configuration data in terms of a >>> JSON-Schema. >>> >>> CLI Integration: >>> >>> Part of our proposed package format is an optional descriptor for how to >>> fetch and install the command-line tools for interacting with the >>> application. For now, we only have one implementation of this, which is to >>> fetch a python egg from PyPI. >>> >>> Governance: >>> >>> All in all, we think that making this effort more community driven is a >>> healthy way to proceed. Any input is very welcome. For example, if others >>> think that what we have is a good starting point we could transfer >>> ownership of the repository to the mesos organization on GitHub. >>> >>> Cheers, >>> -- >>> Connor Doyle >>> http://mesosphere.com >>> >>> >>> >>> >>> >>> On Nov 30, 2014, at 17:32, Dave Lester <daveles...@gmail.com> wrote: >>> >>> As the number of Mesos frameworks grows (and now, a module system), I >>> think it's time to create a community-maintained registry with the goal of >>> making frameworks and modules easier to discover, contribute to, and >>> install. >>> >>> There's already a JIRA ticket tracking this (MESOS-1759) and I've >>> chatted with several folks (thanks in particular Victor Vieux, Tom Arnfeld, >>> Vinod Kone, Timothy St Clair, and Joe Stein). I'd like to advance the >>> conversation by offering a proposal on the public mailing list. >>> >>> I imagine two initiatives to achieve this: >>> >>> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with >>> how Jenkins maintains their plugins on GitHub [1], but they allow >>> individual plugins to have their own repo within their GH org. Plugins are >>> repos with a specific naming convention (in their case, they've appended >>> "-plugin" to the repo name but this isn't the same approach we'd need to >>> take). Having a shared GH org creates greater visibility to what people are >>> doing, and encourages community participation and ownership. >>> >>> In the case of Mesos, not all frameworks will jump at the chance to have >>> community hosting which is fine. But particularly for smaller frameworks, I >>> think this is a good option given the success Jenkins has had. We >>> could potentially use the existing /mesos github org, or create a new org >>> /mesos-extensions or something of the like. It seems like the only >>> potential snag here is that we'll want to be explicit that these aren't >>> Apache-blessed plugins and instead maintained by the larger ecosystem, but >>> I believe we can achieve that by offering a notice in the GH org >>> description. >>> >>> 2) A registry that allows framework writers to publish new versions of >>> their frameworks to a central repository that could be programmatically >>> accessed and rendered online. The v1 could be incredibly simple, but I >>> think this is a foundational piece that can grow with the project in the >>> future. Since this is a little bit more-involved, I've created a separate >>> Google Doc [2] to begin drafting the scope for this: >>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing, >>> and have intentionally punted on some of the implementation details until >>> the scope is finalized and I gauge interest from users and potential >>> collaborators. >>> >>> What do you think? If folks are on board I will assign the JIRA issue to >>> myself and get to work; I'd also welcome collaborators to help get this off >>> the ground since I think it will be a boost for the community. Feedback >>> is welcome! >>> >>> Thanks, >>> Dave >>> >>> [1] https://github.com/jenkinsci/ >>> [2] >>> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing >>> >>> >>> >> >
