Re: Juju's dependencies, juju/utils and a road forward

[roger peppe]

I agree wholeheartedly about splitting up the utils package.

We talked about this potential issue when it was initially created
(and named "trivial", for trivial functionality with no better place
and no dependencies).

The dependency graph is crucial - one of the main reasons that utils
is problematic is because it has so many external dependencies.

> I'd like to move juju to only use "stable" dependency branches. No more
> '-unstable'.

It's perhaps worth observing that "unstable" is the default for any
package that's not under gopkg.in. At current count there are 99 such
packages in juju's dependencies, all potentially unstable, so I'm not
sure it's worth picking out the
explicit gopkg.in unstable paths for blame here.

Of course, this depends on how careful package authors are to maintain
backward compatibility. Some are pretty good, some not so much.

That said, I agree that using explicitly stable dependencies is a
worthwhile goal. In particular, the ongoing move towards using
bakery.v2-unstable is probably not a great direction, because that
uses a particular commit that just happens to be the one that the
charm store currently uses. Given that the charmstore dependency is
only there for some tests, I'd suggest that the effort might be better
expended in changing those tests so that they don't need to import the
charmstore code directly (for example by writing a fake charmstore
server, or making the tests talk to an external charmstore server -
or, ideally, both). Then it becomes more feasible to make the Juju
code use bakery.v2.

What I'd really like is for us to fast forward a year so that we can
use the the upcoming vgo dependency stuff, but time travel isn't
currently an option, sadly.

One important thing is to try to commit to API stability as much as
possible. A large number (maybe most) of the issues come from places
where APIs have been changed without regard for backward
compatibility. Using semantic versions to tag releases is also a good
idea in general.



On 4 April 2018 at 01:52, Tim Penhey  wrote:
> Hi folks,
>
> Juju has a problem with its dependencies. We have been trying to upgrade
> a number of our dependencies recently have have been hitting road
> blocks. Mostly because of changes to common libraries that are
> incompatible with other libraries that we use.
>
> I'm pretty sure that the root of all these problems is the juju/utils
> package. It has become a dumping ground for any common functions that
> packages may want to share. However these common functions are changed
> arbitrarily with only thoughts around how they impact a subset of the
> downstreams of juju/utils.
>
> I think that the only sane way forward is to move all the functionality
> we care about out of juju/utils into more cohesive, smaller, packages
> that have meaningful names and you should be able to determine what they do.
>
> Where the current clashes came from were a few places.
>  1) we wanted to change our mgo dependency
>  2) a new file path utility function was wanted
>  3) the debugstatus package inside utils was updated in a backward
> incompatible way
>
> The root problem here is that the utils package tries to do too many
> things. Any package called "utils" should be a warning, and we should
> have listened to Dave Cheney when he brought this up originally.
>
> I think the way forward is to break up the utils package. I'd like to
> propose that we don't add anything new to the utils package, and only
> deletions are allowed. We create new packages that contain the
> functionality we require, and have them be much more controlled on their
> own dependencies.
>
> Thoughts?
>
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: More juju upgrade-juju failings

[Nicholas Skaggs]

Sandor, thanks for this perspective! It was really helpful to see how 
upgrades went for you in real life, and more importantly, that 2.3.x 
seems to have gone smoothly. We'll be carefully watching and monitoring 
2.3->2.4 upgrades as the release draws nearer.


Nicholas

On 04/01/2018 04:04 AM, Sandor Zeestraten wrote:

Hi Nicholas,

Thanks for the input. I wrote up a short blog post about our 
experiences going from 2.1 to 2.3. Hopefully it provides some feedback 
and can be helpful for others in the same position.


http://zeestrataca.com/posts/upgrading-juju/ 



Regards
--
Sandor Zeestraten

On Thu, Mar 22, 2018 at 4:39 PM, Nicholas Skaggs 
mailto:nicholas.ska...@canonical.com>> 
wrote:


Sandor, re:sharing experiences, if you want to frame some
scenarios you've had trouble with, please feel free to share
those. Distilling it down into a repeatable deployment -> upgrade
will help us understand and account for it. As Tim mentioned,
tools like juju-upgrader are generally candidates for
incorporation into juju itself, provided they prove valuable to
the community at large. We always welcome any PR's, but especially
so for improvements that add this functionality.

Nicholas

On 03/21/2018 08:43 PM, Tim Penhey wrote:

Hi Sandor,

Streamlining upgrades is definitely on the team's road-map. We
are aware
of the juju-upgrader plugin, and will be looking to
incorporate some of
that functionality into the core codebase.

The core team has worked on better upgrade testing as part of
our CI,
which enables more test scenarios to help discover issues
between more
versions.

Cheers,
Tim

On 22/03/18 05:32, Sandor Zeestraten wrote:

Is this shared google doc open for external contributors?

After spending a while upgrading our 2.1.x environments to
2.3.x and
hitting some bugs along the way in staging (see below),
I'd agree that
the process needs a bit of love and wouldn't mind sharing
some experiences.

Rick mentioned https://launchpad.net/juju-upgrader
 on the Juju show a
couple of episodes back, but I haven't seen it mentioned
anywhere else
yet. Are those tools supposed to deal with some of these
issues and
eventually be rolled into juju-core?

https://bugs.launchpad.net/juju/+bug/1746265

https://bugs.launchpad.net/juju/+bug/1748294

https://bugs.launchpad.net/juju/+bug/1697936


Regards
--
Sandor Zeestraten

On Wed, Feb 28, 2018 at 8:30 AM, Mark Shuttleworth
mailto:m...@ubuntu.com>
>> wrote:


     I think this UX on the upgrade process has obvious
problems. Nobody
     should have to guess at what to do, in which sequence.

     Can I suggest that we have a shared Google doc to
mock up a nice
     experience starting with the simple command 'juju
upgrade' which then
     walks people through the process, including the
distinction between
     upgrading charms, agents and controllers, as well as
the ability to do
     aerospace-grade upgrades through live migration to
newer controllers?

     Mark

     On 02/27/2018 11:26 PM, Tim Penhey wrote:
     > Hi Daniel,
     >
     > The issue here is that you are upgrading the
default model, not the
     > controller model itself.
     >
     > I think we could make the error messaging more
clear, and also
     have the
     > command also check the controller version before
showing a lot of
     > baffling output.
     >
     > What you need to do is upgrade the controller model
first, so either
     > switch to it or run:
     >
     >   juju upgrade-juju -m controller --agent-version 2.3.3
     >
     > Cheers,
     > Tim
     >
     > On 28/02/18 11:19, Daniel Bidwell wrote:
     >> I am running on juju 2.3.3-xenial-amd64 and my
controllers are
     running
     >> in VMware Vsphere with version 2.3.2.  I thought
that it would be a
  

Re: More juju upgrade-juju failings

[Sandor Zeestraten]

Glad to help. I reported http://pad.lv/1761288 regarding the ability to
abort/rollback broken upgrades in order to get out of sticky upgrade
situations.
Also hope to see a clean up of the UX and the addition of actual dry-run
capabilities in http://pad.lv/1638714

We're looking forward to upgrading from 2.3 to 2.4 with ease and confidence!

Regards
--
Sandor Zeestraten

On Wed, Apr 4, 2018 at 8:13 PM, Nicholas Skaggs <
nicholas.ska...@canonical.com> wrote:

> Sandor, thanks for this perspective! It was really helpful to see how
> upgrades went for you in real life, and more importantly, that 2.3.x seems
> to have gone smoothly. We'll be carefully watching and monitoring 2.3->2.4
> upgrades as the release draws nearer.
>
> Nicholas
>
> On 04/01/2018 04:04 AM, Sandor Zeestraten wrote:
>
>> Hi Nicholas,
>>
>> Thanks for the input. I wrote up a short blog post about our experiences
>> going from 2.1 to 2.3. Hopefully it provides some feedback and can be
>> helpful for others in the same position.
>>
>> http://zeestrataca.com/posts/upgrading-juju/ <
>> http://zeestrataca.com/posts/upgrading-juju/>
>>
>> Regards
>> --
>> Sandor Zeestraten
>>
>>
>> On Thu, Mar 22, 2018 at 4:39 PM, Nicholas Skaggs <
>> nicholas.ska...@canonical.com >
>> wrote:
>>
>> Sandor, re:sharing experiences, if you want to frame some
>> scenarios you've had trouble with, please feel free to share
>> those. Distilling it down into a repeatable deployment -> upgrade
>> will help us understand and account for it. As Tim mentioned,
>> tools like juju-upgrader are generally candidates for
>> incorporation into juju itself, provided they prove valuable to
>> the community at large. We always welcome any PR's, but especially
>> so for improvements that add this functionality.
>>
>> Nicholas
>>
>> On 03/21/2018 08:43 PM, Tim Penhey wrote:
>>
>> Hi Sandor,
>>
>> Streamlining upgrades is definitely on the team's road-map. We
>> are aware
>> of the juju-upgrader plugin, and will be looking to
>> incorporate some of
>> that functionality into the core codebase.
>>
>> The core team has worked on better upgrade testing as part of
>> our CI,
>> which enables more test scenarios to help discover issues
>> between more
>> versions.
>>
>> Cheers,
>> Tim
>>
>> On 22/03/18 05:32, Sandor Zeestraten wrote:
>>
>> Is this shared google doc open for external contributors?
>>
>> After spending a while upgrading our 2.1.x environments to
>> 2.3.x and
>> hitting some bugs along the way in staging (see below),
>> I'd agree that
>> the process needs a bit of love and wouldn't mind sharing
>> some experiences.
>>
>> Rick mentioned https://launchpad.net/juju-upgrader
>>  on the Juju show a
>> couple of episodes back, but I haven't seen it mentioned
>> anywhere else
>> yet. Are those tools supposed to deal with some of these
>> issues and
>> eventually be rolled into juju-core?
>>
>> https://bugs.launchpad.net/juju/+bug/1746265
>> 
>> https://bugs.launchpad.net/juju/+bug/1748294
>> 
>> https://bugs.launchpad.net/juju/+bug/1697936
>> 
>>
>> Regards
>> --
>> Sandor Zeestraten
>>
>> On Wed, Feb 28, 2018 at 8:30 AM, Mark Shuttleworth
>> mailto:m...@ubuntu.com>
>> >> wrote:
>>
>>
>>  I think this UX on the upgrade process has obvious
>> problems. Nobody
>>  should have to guess at what to do, in which sequence.
>>
>>  Can I suggest that we have a shared Google doc to
>> mock up a nice
>>  experience starting with the simple command 'juju
>> upgrade' which then
>>  walks people through the process, including the
>> distinction between
>>  upgrading charms, agents and controllers, as well as
>> the ability to do
>>  aerospace-grade upgrades through live migration to
>> newer controllers?
>>
>>  Mark
>>
>>  On 02/27/2018 11:26 PM, Tim Penhey wrote:
>>  > Hi Daniel,
>>  >
>>  > The issue here is that you are upgrading the
>> default model, not the
>>  > controller model itself.
>>  >
>>  > I think we could make the error messaging more
>> cle