Juju 2.0~ Local LXD provider workflows are awesome!

2016-02-03 Thread James Page 
Hi All

I've been using Juju 2.0 (built from source with an in-flight patch for LXD
2.0 right now - but that should be resolved soon) with the local LXD
provider on Ubuntu Xenial development to test some work we've been doing to
get OpenStack running on-top of LXD in a single machine.

That's now working quite well (a few rough edges), but is not the main
topic for my post.

1) Multiple models, single controller

Alongside LXD support, you can also create multiple models against a single
controller, so I've been creating models to deploy, test and review
specific pieces of work (reviewing the midonet charms right now for
example):

  juju bootstrap
  juju create-model midonet-review
  juju switch midonet-review

 and then deploy away;  Not having to re-bootstrap a controller
every-time I want to tear-down and redeploy, or push something new up for
test optimizes my workflow nicely.

2) Tweaking container profiles

For each model, Juju creates a profile in LXD (named juju-)-
and its quite possible to make additions to that profile for your specific
model requirements - here's the one we wrote for openstack-on-lxd:

name: juju-openstack-on-lxd
config:
  boot.autostart: "true"
  linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables
  security.nesting: "true"
  security.privileged: "true"
devices:
  eth0:
mtu: "9000"
name: eth0
nictype: bridged
parent: lxcbr0
type: nic
  eth1:
mtu: "9000"
name: eth1
nictype: bridged
parent: lxcbr0
type: nic
  kvm:
path: /dev/kvm
type: unix-char
  root:
path: /
type: disk
  tun:
path: /dev/net/tun
type: unix-char

This adds a-lot to the default profile, but at a high level ensures that
each container gets two network interfaces with a high mtu to avoid packet
fragmentation, can access a few devices required for virt networking and
process management - and also switches the container into 'privileged' mode
that we need for Open vSwitch support in a container right now (Tycho is
working on fixing that so we can run unprivileged).  Read more about LXD
profiles here:

  https://github.com/lxc/lxd/blob/master/specs/configuration.md

Editing is super easy - 'lxc profile edit '.

3) Pause/Resume containers

I've found a few bits that LXD provides outside of Juju quite useful as
well - specifically I've been away from regular power for the last few
days, so I've been using the 'pause' feature of containers to freeze
containers, stopping CPU consumption and making my battery last a alot
longer without destroying and re-deploying the environment (which would
consume far more battery anyway) - here's 'pause-juju':

  for container in `lxc list | grep RUNNING | grep juju | awk '{ print $2
}'`; do
  lxc pause $container
  done

and 'resume-juju':

  for container in `lxc list | grep FROZEN | grep juju | awk '{ print $2
}'`; do
  lxc start $container
  done

I'm doing this outside of Juju right now - but I think it would make a
great feature!

All container processes still consume memory, but stop consuming cpu cycles
until resumed.

Oh - and use the ZFS backend for LXD - its superfast!:


https://insights.ubuntu.com/2015/11/06/using-lxd-with-a-file-based-zfs-pool-on-ubuntu-wily/

Hope people find that all useful!

Cheers

James
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Problem deploying a bundle

2016-02-03 Thread Adam Stokes 
You'll need to update your revisions in the bundle file. We ran into this
problem as well yesterday and it has to deal with charmhelpers regex when
matching package_names -> openstack_versions.

On Tue, Feb 2, 2016 at 10:10 PM, Pshem Kowalczyk  wrote:

> Hi,
>
> I've created my own openstack bundle (attached) that I use to re-deploy
> our staging environment. I've used that bundle a number of times in the
> past.
>
> I've attempted to upgrade to 16.01 charms and try the trusty-mitaka
> openstack, but ran into issues so decided to wipe out the whole environment
> and start from scratch.
>
> To my great surprise the bundle doesn't deploy any more. Multiple charms
> fail with messages like:
>
> 2016-02-03 02:49:19 ERROR juju-log FATAL ERROR: Could not determine
> OpenStack codename for version 8.0.1 (keystone)
> 2016-02-03 02:47:49 ERROR juju-log FATAL ERROR: Could not determine
> OpenStack codename for version 11.0.1 (glance)
> 016-02-03 02:48:30 ERROR juju-log FATAL ERROR: Could not determine
> OpenStack codename for version 7.0.1 (cinder)
>
> etc.
>
> Why are they failing now? And the more important question - how do I make
> it work again? Do I have to upgrade my bundle file to use the new charms?
>
> kind regards
> Pshem
>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju 2.0~ Local LXD provider workflows are awesome!

2016-02-03 Thread Jorge O. Castro 
On Wed, Feb 3, 2016 at 1:09 PM, James Page  wrote:
> Hope people find that all useful!

Indeed, we also have fresh documentation thanks to Peter Matulis. I've
been using these all week to spin up on lxd and juju:

https://jujucharms.com/docs/devel/config-LXD

As always everyone please feel free to open a PR or file an issue.

Also a huge +1 to the LXD/ZFS/Juju combination; all of these
performance improvements across the different parts of the stack are
really coming together.

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju 2.0~ Local LXD provider workflows are awesome!

2016-02-03 Thread Mark Shuttleworth 
On 03/02/16 12:09, James Page wrote:
>   juju create-model midonet-review
>   juju switch midonet-review

Thanks for the feedback James, it's great to see these bits coming
together so nicely :)

Should we automatically switch to a new model when you create it? I
suspect the common case is create-model-then-work-in-it rather than
create-model-then-do-something-else.

Thoughts?
Mark

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju 2.0~ Local LXD provider workflows are awesome!

2016-02-03 Thread Louis Bouchard 
Hi,

Le 03/02/2016 14:23, Mark Shuttleworth a écrit :
> On 03/02/16 12:09, James Page wrote:
>>   juju create-model midonet-review
>>   juju switch midonet-review
> 
> Thanks for the feedback James, it's great to see these bits coming
> together so nicely :)
> 
> Should we automatically switch to a new model when you create it? I
> suspect the common case is create-model-then-work-in-it rather than
> create-model-then-do-something-else.
> 

Don't we all love 'git checkout -b' to land into the newly created branch ???

How about juju create-model -b  ;-)


> Thoughts?
> Mark
> 

...Louis

-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer   Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju 2.0~ Local LXD provider workflows are awesome!

2016-02-03 Thread Antonio Rosales 
On Wed, Feb 3, 2016 at 1:09 PM, James Page  wrote:
> Hi All
>
> I've been using Juju 2.0 (built from source with an in-flight patch for LXD
> 2.0 right now - but that should be resolved soon) with the local LXD
> provider on Ubuntu Xenial development to test some work we've been doing to
> get OpenStack running on-top of LXD in a single machine.
>
> That's now working quite well (a few rough edges), but is not the main topic
> for my post.

This is solid, thanks for sharing here.

-Antonio

>
> 1) Multiple models, single controller
>
> Alongside LXD support, you can also create multiple models against a single
> controller, so I've been creating models to deploy, test and review specific
> pieces of work (reviewing the midonet charms right now for example):
>
>   juju bootstrap
>   juju create-model midonet-review
>   juju switch midonet-review
>
>  and then deploy away;  Not having to re-bootstrap a controller
> every-time I want to tear-down and redeploy, or push something new up for
> test optimizes my workflow nicely.
>
> 2) Tweaking container profiles
>
> For each model, Juju creates a profile in LXD (named juju-)- and
> its quite possible to make additions to that profile for your specific model
> requirements - here's the one we wrote for openstack-on-lxd:
>
> name: juju-openstack-on-lxd
> config:
>   boot.autostart: "true"
>   linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables
>   security.nesting: "true"
>   security.privileged: "true"
> devices:
>   eth0:
> mtu: "9000"
> name: eth0
> nictype: bridged
> parent: lxcbr0
> type: nic
>   eth1:
> mtu: "9000"
> name: eth1
> nictype: bridged
> parent: lxcbr0
> type: nic
>   kvm:
> path: /dev/kvm
> type: unix-char
>   root:
> path: /
> type: disk
>   tun:
> path: /dev/net/tun
> type: unix-char
>
> This adds a-lot to the default profile, but at a high level ensures that
> each container gets two network interfaces with a high mtu to avoid packet
> fragmentation, can access a few devices required for virt networking and
> process management - and also switches the container into 'privileged' mode
> that we need for Open vSwitch support in a container right now (Tycho is
> working on fixing that so we can run unprivileged).  Read more about LXD
> profiles here:
>
>   https://github.com/lxc/lxd/blob/master/specs/configuration.md
>
> Editing is super easy - 'lxc profile edit '.
>
> 3) Pause/Resume containers
>
> I've found a few bits that LXD provides outside of Juju quite useful as well
> - specifically I've been away from regular power for the last few days, so
> I've been using the 'pause' feature of containers to freeze containers,
> stopping CPU consumption and making my battery last a alot longer without
> destroying and re-deploying the environment (which would consume far more
> battery anyway) - here's 'pause-juju':
>
>   for container in `lxc list | grep RUNNING | grep juju | awk '{ print $2
> }'`; do
>   lxc pause $container
>   done
>
> and 'resume-juju':
>
>   for container in `lxc list | grep FROZEN | grep juju | awk '{ print $2
> }'`; do
>   lxc start $container
>   done
>
> I'm doing this outside of Juju right now - but I think it would make a great
> feature!
>
> All container processes still consume memory, but stop consuming cpu cycles
> until resumed.
>
> Oh - and use the ZFS backend for LXD - its superfast!:
>
>
> https://insights.ubuntu.com/2015/11/06/using-lxd-with-a-file-based-zfs-pool-on-ubuntu-wily/
>
> Hope people find that all useful!
>
> Cheers
>
> James
>
>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>



-- 
Antonio Rosales
Ecosystem Engineering
Canonical

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Problem deploying a bundle

2016-02-03 Thread Pshem Kowalczyk 
Hi,

On Wed, 3 Feb 2016 at 22:55 James Page  wrote:

> Hi Pshem
>
> {cut}
>>
>

> Why are they failing now? And the more important question - how do I make
>> it work again? Do I have to upgrade my bundle file to use the new charms?
>>
>
> Yes - use the new charms in the charm store with Liberty rather than
> Mitaka; there was a stable charm update just prior to 16.01 to fix the
> problem you are hitting (version detection of point releases for the new
> semantic naming upstream in OpenStack).
>
> The point release for liberty where release earlier in the week, so you're
> now picking up .1 versions in stead of .0 versions.
>
> The 16.01 charm release includes these fixes so will work OK.
>
> Thank you for the explanation. Is it possible to create a bundle file that
doesn't have specific versions of charms (and instead pulls in the current
ones)?

kind regards
Pshem
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju 2.0~ Local LXD provider workflows are awesome!

2016-02-03 Thread Rick Harding 
Yes it's intended to auto switch for you. If it does not we need to correct
it.

On Wed, Feb 3, 2016, 2:49 PM James Page  wrote:

> On Wed, 3 Feb 2016 at 13:23 Mark Shuttleworth  wrote:
>
>> On 03/02/16 12:09, James Page wrote:
>> >   juju create-model midonet-review
>> >   juju switch midonet-review
>>
>> Thanks for the feedback James, it's great to see these bits coming
>> together so nicely :)
>>
>> Should we automatically switch to a new model when you create it? I
>> suspect the common case is create-model-then-work-in-it rather than
>> create-model-then-do-something-else.
>
>
> Actually we do the auto switch already - the switch in my example is not
> required (just tested with a new model).
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Juju terminology change: controllers and models

2016-02-03 Thread John Meinel 
What did we end up doing for Login? I'm thinking it would be good to keep a
v1 login around that would tell <=1.25 clients that they need a different
version of their client to talk to this server.

John
=:->
On Feb 3, 2016 9:03 PM, "Eric Snow"  wrote:

> Thanks for this in-depth break-down, Ian.  It really helps.
>
> -eric
>
> On Tue, Feb 2, 2016 at 6:33 PM, Ian Booth  wrote:
> > Hey all
> >
> > As has been mentioned previously in this list, for the Juju 2.0 release
> we have
> > been working on fundamental terminology changes. In particular, we now
> talk
> > about controllers and models instead of state servers and environments.
> >
> > To this end, a rather large change has landed in master and the upcoming
> > 2.0-alpha2 release of Juju will reflect these changes. There are several
> things
> > to be aware of. We have also taken the opportunity to remove a lot of
> code which
> > existed to support older Juju clients. Needless to say, this Juju 2.0
> release
> > will not support upgrading from 1.x - it works only as a clean install.
> >
> > Note: some of the changes will initially break the GUI and users of the
> Python
> > Juju Client - work is underway to update these products for the next
> alpha3
> > release scheduled for next week. For those wishing to continue to test
> Juju 2.0
> > without the breaking changes, the alpha1 release is still available via
> > ppa:juju/experimental. Separate communications to affected stakeholders
> has/will
> > be made as part of the 2.0-alpha2 release.
> >
> > So, the changes are roughly broken down as follows:
> >
> > - CLI command name changes
> > - facade name changes
> > - api method and parameter name changes
> > - facade method restructure
> > - internal api name changes
> > - external artifact/data changes (including on the wire changes)
> > - deprecated and older version facades are removed
> >
> > 1. CLI command name changes
> >
> > As an obvious example, create-environment becomes create-model. We also
> have
> > destroy-controller etc. This alpha2 release will also contain many of
> the other
> > CLI changes targetted for 2.0 eg juju backup create becomes juju
> create-backup.
> > Not all 2.0 CLI syntax is supported yet, but all the environment -> model
> > changes are done.
> >
> > You will also use -m  instead of -e .
> >
> > The release notes will go into more detail.
> >
> > All user facing text now refers to model instead of environment.
> >
> > 2. Facade name changes
> >
> > If you are curious, see https://goo.gl/l4JqGd for a representative
> listing of
> > all facade and method names and which ones have been changed.
> >
> > The main one is EnvironmentManager becomes ModelManager. These changes
> affect
> > external API clients like the GUI and Python Juju client.
> >
> > 3. api method and parameter name changes
> >
> > By way of example:
> > EnvironInfo() on the undertaker facade becomes ModelInfo().
> > The param struct ModifyEnvironUsers becomes ModifyModelUsers etc.
> > EnvironTag attributes become ModelTag.
> >
> > 4. Service facade method restructure
> >
> > As part of making our facades more manageable and maintainable when API
> changes
> > are required, a whole bunch of service related methods are moved off the
> Client
> > facade and onto the Service facade. This had already been started months
> ago,
> > and there were shims in place to keep existing clients working, but now
> the job
> > is finished.
> > eg Client.AddRelation() becomes Service.AddRelation() etc.
> >
> > This change will break the GUI and Python Juju client.
> >
> > 5. Internal API name changes
> >
> > Things like state.AllEnvironments() becomes state.AllModels(), we now use
> > names.ModelTag instead of names.EnvironTag, and many, many more.
> >
> > Note: the names package has not been forked into a .V2 yet (with
> EnvironTag
> > removed) as there are dependencies to sort out. Please do not use
> EnvironTag
> > anymore.
> >
> > 6. External artifact/data changes (including on the wire changes)
> >
> > There are several main examples here.
> > On the wire, we transmit model-uuid tags rather than environment-uuid
> tags.
> > In mongo, we store model-uuid doc fields rather than env-uuid.
> > In agent.conf files we store Model info rather than Environment tags.
> > In the controller blob store, we store and manage blobs for buckets
> rather than
> > environments.
> > The controller HTTP endpoints are /model/ > In backups we store model tags and details, not environment.
> >
> > With the blobstore, we've create a .V2 branch which core uses. The
> charmstore
> > will continue to use V1 for now.
> >
> > 7. Deprecated and older version facades are removed
> >
> > All facade versions have been bumped up. Older facades are removed, and
> we don't
> > support fallback to older servers. The main example for facade removal
> is uniter
> > v0 and v1 are gone. With deprecated API removal, service deployment now
> expects
> >

Re: Juju 2.0~ Local LXD provider workflows are awesome!

2016-02-03 Thread James Page 
On Wed, 3 Feb 2016 at 13:23 Mark Shuttleworth  wrote:

> On 03/02/16 12:09, James Page wrote:
> >   juju create-model midonet-review
> >   juju switch midonet-review
>
> Thanks for the feedback James, it's great to see these bits coming
> together so nicely :)
>
> Should we automatically switch to a new model when you create it? I
> suspect the common case is create-model-then-work-in-it rather than
> create-model-then-do-something-else.


Actually we do the auto switch already - the switch in my example is not
required (just tested with a new model).
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju