Re: Semver instead of revisions for charms?

[Stuart Bishop]

On 28 February 2016 at 19:49, Marco Ceppi  wrote:
> There is no plan at this time to move to anything other than revision
> numbers for charm. The idea behind the system is there should really /never/
> be any backwards compatible breaks. If you find yourself running into a
> situation, make use of the upgrade-charm hook to update/rewrite any internal
> data structures you're using to the new version.
>
> This way charms are always forward looking and up-gradable as such. While
> I'm a huge fan of semver, adding that as a primitive for charms means there
> may be chances where a user of a charm won't be able to upgrade. That's a
> precedence we don't want to set.

I'm not sure how semantic versioning would make the situation any
different. I can introduce a backwards incompatible change just as
easily between cs:foo/99 and cs:foo/100 as I can between v3.9.4 and
v4.0.0. For what its worth, I'm switching to semantic version numbers
in my branches via tags in order to properly document changes and
tested upgrade paths. This is decoupled from the charm store
revisions, as several stable releases can be made before it gets
synced into the charm store.

I believe backwards compatibility issues are impossible to avoid
completely. Sometimes they are necessary, sometimes they are
accidental, and reverting landed revisions very dangerous as it
introduces a completely untested upgrade path for deploys made from
the reverted version (it might be weeks before a backwards
incompatible change is found). While we can all do our best,
occasionally it has to happen and all we can do is make things as
clear as possible to users.

I find versioning is a common issue, particularly regarding upgrades.
It seems more common than not that people are deploying from branches
rather than the charm store (either for bug fixes, requested features,
or lack of egress). I hope the version tags, git commit ids and maybe
a version.yaml will make these deploys much easier to deal with. I was
thinking a version.yaml convention would be good for composed charms,
as charm-build could merge them and we get tracking of which versions
of what layers got used in a particular build. And layers don't get
charm store revision ids. Once we have version numbers, we can do
proper bug tracking and milestones etc. on all these components that
can be used to build a charm. Machine readable, so the charm store
could display it and we could tell at a glance which charms need
rebuilding because they embed buggy or insecure layers.

-- 
Stuart Bishop 

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

model-migration: the what, why, and progress

[Tim Penhey]

Hi folks,

I'm writing this to communicate what we are doing, why, and how its
going. Something I hope will prompt others to do the same.

I'm also keeping this on the internal juju list because I will be
mentioning JAAS.

What and why:

Model migration is a key component of JAAS and allows us to work with
the most up to date Juju Controllers without fear. With the new world
order where a controller can host multiple models, we are now in the
situation where upgrading a controller could have dramatic and drastic
problems for other models hosted on it should things go wrong. This is
not something we even want to have as an option where we are dealing
with models that are not ours.

Models hosted in a controller are not able to upgrade to versions of
Juju beyond that of the controller they are running on. So one of two
things needs to happen:
 1) we upgrade the controller
 2) we move the model to a newer controller

Once one of these things has happened, the hosted model can upgrade to a
new version (up to what the controller is running).

Since we have already decided that (1) is not a viable option in the
JAAS world, (2) is what we are working on.

Model migration also gives us the ability to load balance should one
controller get overloaded by moving some models to a different controller.

Progress:

Team Onyx has been focusing on model migrations for some time now, and
many pieces are falling into place.

In order to migrate a model, as a user, you need to be a "controller
admin" in both the source and target controllers. There is a CLI command
to initiate the migration "juju migrate". Status of the migration will
be reported as part of "juju status".

This starts a chain of events that will result in the model being moved
and all the agents looking at the new controller.

One of the first big changes was to change the machine agent to use the
new dependency engine for managing dependencies between the various
workers. This is needed to cleanly quiesce environments.  This work
landed into master some time ago, and we are changing the individual
workers one at a time to fit into the new engine.

Some complexities and steps are not shown to avoid too much detail.

The general process of migration goes something like this:
 0) initial checks to confirm no large pending changes or error states
   - no machines being provisioned
   - no missing agents
   - no dying or dead entities
   - there is a grace period, but if it doesn't settle down, the
migration is aborted
 1) the model is quiesced and agents enter a read only state
 2) the general state is checked again to make sure all things are quiet
and stable
 3) the model is serialized into a versioned, database agnostic, wire
format and imported into the target controller
 4) binaries used by the model, tools, charms, resources etc are added
to the target controller
 5) logs are streamed across
 6) all agents are told to check in with the new controller to confirm
access and that the state looks consistent as it would expect
 7) agents switch over to new controller
 8) documents in old controller are cleaned up

At any state prior to switching the agents over, the migration may be
aborted and the model goes back into an active mode.

Right now we are working like mad to get this working.

The model import/export is at a stage where by the end of this week we
should be able to deal with "simple" environments, where simple means
not dealing with the new features like: payloads, storage, networks, or
resources.  These extra parts will be added ASAP.

Several weeks ago I was able to do a proof of concept migration of
simple machines (this was before I got services, units and relations
going) between two controllers using a separate binary to do the export
and import and much manual-hacker-fu.

We are actively working on the controller managing the migration process.

We are making good progress I do feel that things are going to be tight.

The import/export work is in the feature branch "model-migrations".
The control aspect is in the feature branch "model-migration-control".
Moving all the environment workers to a nested dependency engine is the
feature branch "MADE-state-workers" (MADE was short hand for Machine
Agent Dependency Engine). The machine-dep-engine feature branch merged
into master back in January. The MADE-workers feature branch landed
earlier this month. Individual worker changes are now targeting master.

Each of the feature branches are independent of each other for now.

A side benefit I see of the migration work is the potential to speed up
'juju status' significantly.  The model export takes the entire db into
a structured object model, and it reads entire collections at a time, so
is much faster. I'd love to test this in real world large environments.

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Semver instead of revisions for charms?

[Patrik Karisch]

Hi,

is there any plan to implement semantic versioning for charms and step away
from revisions which can't indicate major BC breaks compared to a major
release with semver?

Cheerz
Patrik
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju