Wooo mesosphere is making such a good work pushing forward Mesos
technologies!
I deeply think that Mesos should implement registry as a core feature.
Let me explain that, Installing manually frameworks / modules / other third
party adds-on is quite cool when you're beginner and want to learn how the
things works, but once you're in production, you've to be fast and react
quickly.
Let said that I'm a system engineer that have customers requests like
"Hum... Could you had this " insert Framework/module/other name here" on
our cluster region?", I'll have to add it as soon and quick as possible on
the targeted environment, I'll be OK to do it twice, but I wont make my job
a request nightmare and I'll probably search for a way to provide to my
customers a restrictive access (WebUI) to the cluster manager and allow
them to deploy those new add-on by themself.
With this study case in mind, I'll be happy to have a registry which allow
a filtering mechanism to hide or show frameworks/modules/etc by status
(Official Apache Content / Public Content / Private Content).
Here is my thoughts about the registry articulation:
- Mesos Registry is an integrated part of Mesos master services.
- Mesos Registry is an endpoint available to WebUI and CLI.
- Mesos Registry is nothing but a metadata registry.
- Mesos Registry save its configs and metadata on a key/value store
(zookeeper?).
- Mesos Registry is empty at the first launch.
- Mesos Registry as three views: Official Apache Content | Publicly
Maintained Content | Private Maintained Content
- Mesos Registry views content is:
- Official Apache Content == Link to existing Apache add-ons hosted on
Github/Git repository + metadata (Like those proposed by Dave and Connor).
- Publicly Maintained Content == Links to existing repositories
(Github / Git / other) + metadata (Like those proposed by Dave
and Connor).
- Private Content == Links to existing private (GIT/GITHUB/Other
repositories) + metadata (Like those proposed by Dave and Connor), this
repository is kinda special as it is hosted and created by the cluster
operators and could be a mixed content of locally maintained repository
(GIT repos on a HDFS or TAR on HDFS) and public content repository cloned
from the public content URL/Metadata.
I don't know if I'm really clear, so if I'm not, let me know it, I'll do
some sketches :D
2014-12-02 1:53 GMT+01:00 Connor Doyle <con...@mesosphere.io>:
> Hi Dave,
>
> This is a timely topic, since we have been prototyping and mocking up
> something similar at Mesosphere. We created a new public GitHub repository
> for it about three weeks ago called "universe" (
> http://github.com/mesosphere/universe).
>
> Although we have added some informal specs, it's very malleable at this
> point. We're very much interested in making our "universe" compatible
> with, or the same as, the registry you're proposing. Without delving into
> implementation details, some of the goals we have in mind are outlined
> below.
>
> Data Source:
>
> The package repository should be easily consumable by third-party
> command-line and other programs. There should be a condensed “index”
> representation of the package repository available.
>
> Packages within the repository should be versioned.
>
> The package repository format itself should be versioned.
>
> Decentralization and Composability:
>
> The package metadata should be hosted in a public place (we like GitHub)
> so that additional packages can be added by the community by simply
> submitting pull requests. We have added some rudimentary commit hooks and
> automated validation to protect the repo against breaking changes.
>
> It’s important that no single entity “owns the keys” to the universe, and
> that the spec and implementation remain public. It should be easy and free
> for organizations to maintain a private package repository.
>
> A corollary is that it should be easy for consumers to pull from a
> hierarchy of upstream repositories. One setup we have in mind is that an
> organization might have staging and production repositories running
> internally. Packages are pushed to staging where integration testing can
> run before “deployment” to production. If a package isn’t in the local
> repository it might be looked up and installed from upstream.
>
>
>
> Repositories should be able to be proxied and cached in this way.
> Organizations should be able to isolate their datacenter but also
> selectively add external packages for experimentation. The system should be
> sufficiently portable and extensible to accomodate these and similar use
> cases.
>
> Meta-Framework Descriptors:
>
> Our conception of the package repository is a bit more expansive than just
> Mesos frameworks; it includes descriptions of how to install any piece of
> server software on a Mesos cluster. Frameworks and non-frameworks alike
> may be installed using some other meta-framework that’s responsible for
> starting all other cluster services. Likely candidates for this role are
> the long-lived frameworks: Aurora, Marathon, Singularity, and eventually
> Kubernetes. In any case, the repository spec should not be prescriptive
> with respect to this choice.
>
> The package repository metadata should make it easy for Mesos framework
> authors (and authors of non-Mesos-aware programs) to describe how to
> install their software on a Mesos cluster. To this end, our prototype
> package spec allows for Meta-framework descriptor files for each package in
> the repository. For example for a given package we might see a
> `marathon.json` file as well as a `my-app.aurora` file.
>
> An obvious concern is how to specify site-specific arguments upon
> installation. Here packages should describe data that must be marshalled
> from the environment (e.g. by prompting a user) and combined with the raw
> meta-framework descriptor to launch the app. These configuration
> parameters should be agnostic of the supported meta-frameworks. More
> concretely, in our prototype we describe configuration data in terms of a
> JSON-Schema.
>
> CLI Integration:
>
> Part of our proposed package format is an optional descriptor for how to
> fetch and install the command-line tools for interacting with the
> application. For now, we only have one implementation of this, which is to
> fetch a python egg from PyPI.
>
> Governance:
>
> All in all, we think that making this effort more community driven is a
> healthy way to proceed. Any input is very welcome. For example, if others
> think that what we have is a good starting point we could transfer
> ownership of the repository to the mesos organization on GitHub.
>
> Cheers,
> --
> Connor Doyle
> http://mesosphere.com
>
>
>
>
>
> On Nov 30, 2014, at 17:32, Dave Lester <daveles...@gmail.com> wrote:
>
> As the number of Mesos frameworks grows (and now, a module system), I
> think it's time to create a community-maintained registry with the goal of
> making frameworks and modules easier to discover, contribute to, and
> install.
>
> There's already a JIRA ticket tracking this (MESOS-1759) and I've chatted
> with several folks (thanks in particular Victor Vieux, Tom Arnfeld, Vinod
> Kone, Timothy St Clair, and Joe Stein). I'd like to advance the
> conversation by offering a proposal on the public mailing list.
>
> I imagine two initiatives to achieve this:
>
> 1) Shared hosting via a GitHub org. I'm not sure if you're familiar with
> how Jenkins maintains their plugins on GitHub [1], but they allow
> individual plugins to have their own repo within their GH org. Plugins are
> repos with a specific naming convention (in their case, they've appended
> "-plugin" to the repo name but this isn't the same approach we'd need to
> take). Having a shared GH org creates greater visibility to what people are
> doing, and encourages community participation and ownership.
>
> In the case of Mesos, not all frameworks will jump at the chance to have
> community hosting which is fine. But particularly for smaller frameworks, I
> think this is a good option given the success Jenkins has had. We
> could potentially use the existing /mesos github org, or create a new org
> /mesos-extensions or something of the like. It seems like the only
> potential snag here is that we'll want to be explicit that these aren't
> Apache-blessed plugins and instead maintained by the larger ecosystem, but
> I believe we can achieve that by offering a notice in the GH org
> description.
>
> 2) A registry that allows framework writers to publish new versions of
> their frameworks to a central repository that could be programmatically
> accessed and rendered online. The v1 could be incredibly simple, but I
> think this is a foundational piece that can grow with the project in the
> future. Since this is a little bit more-involved, I've created a separate
> Google Doc [2] to begin drafting the scope for this:
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing,
> and have intentionally punted on some of the implementation details until
> the scope is finalized and I gauge interest from users and potential
> collaborators.
>
> What do you think? If folks are on board I will assign the JIRA issue to
> myself and get to work; I'd also welcome collaborators to help get this off
> the ground since I think it will be a boost for the community. Feedback
> is welcome!
>
> Thanks,
> Dave
>
> [1] https://github.com/jenkinsci/
> [2]
> https://docs.google.com/document/d/1sOoPtEyLlST5GTU5iSccqWUsAOlJhf4N9cHoylylYgY/edit?usp=sharing
>
>
>
