Re: Null provider and failing early

[David Cheney]

I think they should be ignored, but preserved.

On Tue, Aug 27, 2013 at 11:40 AM, Tim Penhey tim.pen...@canonical.com wrote:
 On 27/08/13 11:50, Andrew Wilkins wrote:
 On Tue, Aug 27, 2013 at 8:44 AM, Tim Penhey tim.pen...@canonical.com
 mailto:tim.pen...@canonical.com wrote:

 Perhaps we should have a sanity-check type callback into the provider
 with the constraints at the time we want to add a machine.  This would
 give the null provider the early fail mechanism, and could also allow
 other providers to error if people as asking for constraints that really
 don't make sense.


 I'm sort of thinking out aloud here: maybe this could used for checking
 environment-specific constraints too? Some kind of
 VerifyMachineConstraints method that will ensure your add-machine/--to
 constraints are valid for the current provider/environment. In this case
 constraints may be nil, but the null provider would just always return
 false.

 This is exactly what I'm thinking as well, but I had forgotten about the
 provider specific constraints.

 It does raise the question of what should happen if a provider specific
 constraint is passed to a provider that can't handle it.  My suggestion
 is we fail.

 Tim

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Focus for the next three weeks

[David Cheney]

Sorry to add more wood to the pile, but 1.14.0 needs to go out, basically now.

https://launchpad.net/juju-core/+milestone/1.14.0

Shows three issues (one is merged on trunk, but needs backporting)

On Tue, Sep 10, 2013 at 1:39 PM, Tim Penhey tim.pen...@canonical.com wrote:
 Hi folks,

 Lets start with the target: we have Saucy being released next month, and
 we want to get a rocking release done for that.

 Working backwards, and giving ourselves a week slack (which we will end
 up needing) before final freeze, takes us roughly to the end of September.

 I'd like to see where we are on our targets, and work out what we can
 get done in the next three weeks that makes as many people happy as
 possible.

 1. MAAS Tag support

 This seems to be a key need for a number of engagements we have, and
 also a reason why some MAAS/Juju demos are still using pyJuju.  We have
 talked about this a lot, but I haven't seen any other movement or
 progress on it.

 What is the current state of play with the provider or cloud specific
 constraints?  Does anyone see real blockers to having MAAS tag
 deployment constraints done in the next three weeks?  Who is going to be
 the lead on this?  William?

 2. Container Pain

 Right now we don't have a way to restrict the users creating containers
 that they cannot address.  I propose that we add a method to the
 environment interface that is SupportsContainers() bool.  This can then
 be used to stop people at least creating containers that they can't use.

 This I think we need to do in the next three weeks, and make sure the
 add-machine and deployment commands respect whether containers are
 supported or not.  Currently the only provider that actually supports
 containers is MAAS.

 Next I think we should have SupportedContainers() on a machine.
 Different machines can support different containers.  However I'm not
 sure we should do this in the next three weeks, so will be the focus of
 a different email.

 3. API only non-bootstrap agents

 How close are we on this one?  I know we have been making good progress,
 but what is the likelihood of this being done in the next three weeks?

 4. Manual provisioning, Manual bootstrap, Null provider

 I think we are making good progress on this, and should have this in.

 5. KVM containers

 I have some work done by Robie Basak that provides us with some command
 line tools for having images available, and starting/stopping and
 listing kvm containers.  This needs someone on it, possibly me, to take
 it further.  Based on the newness of the tools, I have about a 50%
 confidence rating on this right now.

 6. Container addressability for EC2 / Openstack

 I don't know where this is at right now.  Can Martin please fill us in.

 7. Simplestream tools

 I talked with Ian earlier and he thinks there is about another week of
 work to polish this up and add documentation.  This will really help the
 use of Juju in private clouds, or non-certified public clouds.


 8. What else do we have that is new or note worthy that we are working
 on in the next three weeks?


 Cheers,
 Tim

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

1.15.0 release this week ?

[David Cheney]

Hello,

What is the temperature of folks about doing a 1.15.0 release this
week ? My spider senses make me think it is worth more time to bed
down the tools changes.

Thoughts ?

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Juju 1.14.0 draft release notes

[David Cheney]

https://docs.google.com/a/canonical.com/document/d/1PNtuwad70FrbtdyA0Wx4-yKLGGguA7P62sa_Bfsfi7U/edit#

Waiting on the final 1.14.0 issues to be resolved before we can push
this to Jamespage for Saucy.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: 1.14 doesn't actually build

[David Cheney]

Only goose and juju-core have tags. If you add a tag to gwacl, please
update scripts/build-release-tarball/build-release-tarball.sh

On Tue, Sep 17, 2013 at 8:07 AM, Ian Booth ian.bo...@canonical.com wrote:
 Surely when we tag a release, we also tag the matching revisions of all the
 dependencies? gwacl was indeed changed so that GetAnonymousFileURL() returns a
 string and an error. Wouldn't the solution be to checkout the tagged revisions
 of the various dependencies?

 On 17/09/13 07:05, Nate Finch wrote:
 It was just brought to my attention on IRC that, due to a change to gwacl
 after 1.14 was cut, 1.14 doesn't actually build right now (if you just use
 the most recent version of gwacl).

 The error is

 provider/azure/storage.go:96: multiple-value context.GetAnonymousFileURL()
 in single-value context

 Figured I'd let the list know, since I'm not sure what the correct fix is.

 -Nate




 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Consistent test failures with saucy

[David Cheney]

Have a hunt in the code, i bet there is something that lists all the
series from P through to R, but doesn't include saucy, possibly in
fixtures or the dummy provider

On Mon, Sep 30, 2013 at 9:31 AM, Tim Penhey tim.pen...@canonical.com wrote:
 Hi folks,

 Most of the upgrade worked fine.

 I am however finding some problems with the test suite.  I have one test
 that consistently fails:

 bootstrap_test.go:78:
 s.runAllowRetriesTest(c, test)
 bootstrap_test.go:99:
 c.Check(findToolsRetryValues, gc.DeepEquals, test.expectedAllowRetry)
 ... obtained []bool = []bool{true, true}
 ... expected []bool = []bool{true}

 bootstrap_test.go:78:
 s.runAllowRetriesTest(c, test)
 bootstrap_test.go:100:
 c.Assert(err, gc.ErrorMatches, test.err)
 ... error string = tools not found
 ... regex string = no matching tools available

 OOPS: 164 passed, 1 FAILED


 Anyone else getting this?

 Tim

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Basic Juju-Core CI is up

[David Cheney]

THIS IS GREAT!

I especially like the fact I don't need 2fa or a million ssh tunnels
to view the build status. Kudos for developing in the open.

On Wed, Nov 13, 2013 at 7:13 AM, Curtis Hovey-Canonical
cur...@canonical.com wrote:
 We have established basic CI of juju-core. The jenkin's instance is a
 charm running on canonitack
 http://162.213.34.53:8080/

 --
 Curtis Hovey
 Canonical Cloud Development and Operations
 http://launchpad.net/~sinzui

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Synchronous Bootstrap

[David Cheney]

\o/

On Tue, Dec 3, 2013 at 5:31 PM, Andrew Wilkins
andrew.wilk...@canonical.com wrote:
 Ahoy,

 Just wanted to inform everyone that synchronous bootstrap has landed on
 trunk. If you've got any scripts that bootstrap an environment and expect it
 to chug away in the background, then some script changes will be necessary.

 If you hit Ctrl+C, juju will catch the SIGINT and cancel the procedure; the
 newly created instance will then be terminated.

 Bootstrap will provide feedback as it goes through the synchronous part of
 the cloud-init script. This is using code previously written for manual
 provisioning, updated to be more comprehensible/useful to users. Example
 output:

 $ juju bootstrap --upload-tools
 Launching instance
  - i-de79c1b8
 Waiting for DNS name.
  - ec2-54-224-239-35.compute-1.amazonaws.com
 Attempting to connect to ec2-54-224-239-35.compute-1.amazonaws.com:22.
 Warning: Permanently added
 'ec2-54-224-239-35.compute-1.amazonaws.com,54.224.239.35' (ECDSA) to the
 list of known hosts.
 Logging to /var/log/cloud-init-output.log on remote host
 Installing add-apt-repository
 Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu
 precise-updates/cloud-tools main
 Running apt-get update
 Running apt-get upgrade
 Installing package: git
 Installing package: mongodb-server
 Fetching tools: wget --no-verbose -O $bin/tools.tar.gz
 'https://s3.amazonaws.com/juju-cf8c93be48c13f5b4dcc60da85d84e31/tools/releases/juju-1.17.0.1-precise-amd64.tgz?redacted'
 Starting MongoDB server (juju-db)
 Bootstrapping Juju machine agent
 Starting Juju machine agent (jujud-machine-0)
 Connection to ec2-54-224-239-35.compute-1.amazonaws.com closed.

 Cheers,
 Andrew

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

juju bootstrap w/ gccgo built cli and tools

[David Cheney]

Good news, everyone!

We are ---this--- close to getting a gccgo build juju working.

lucky(~/src/launchpad.net/juju-core) % juju bootstrap -v --upload-tools

verbose is deprecated with the current meaning, use show-log

2013-12-05 05:57:50 INFO juju.environs open.go:156 environment info
already exists; using New not Prepare

2013-12-05 05:57:50 INFO juju.provider.ec2 ec2.go:176 opening
environment ap-southeast-2

2013-12-05 05:57:50 INFO juju.environs.tools build.go:162 found
existing jujud: /home/dfc/bin/jujud

2013-12-05 05:57:50 INFO juju.environs.tools build.go:172 target:
/tmp/juju-tools033710042/jujud

 compiled with gccgo

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:244 built
1.17.0.1-saucy-amd64 (5583kB)

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:70 listing available tools

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:94 found 2 tools

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:104 listing target bucket

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:117 found 0 tools
in target; 2 tools to be copied

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:155 copying
1.17.0.1-precise-amd64 from
file:///tmp/142843249/tools/releases/juju-1.17.0.1-precise-amd64.tgz

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:169 copying
tools/releases/juju-1.17.0.1-precise-amd64.tgz

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:183 downloaded
tools/releases/juju-1.17.0.1-precise-amd64.tgz (5583kB), uploading

2013-12-05 05:57:53 INFO juju.environs.sync sync.go:184 download
5583kB, uploading

2013-12-05 05:59:09 INFO juju.environs.sync sync.go:155 copying
1.17.0.1-saucy-amd64 from
file:///tmp/142843249/tools/releases/juju-1.17.0.1-saucy-amd64.tgz

2013-12-05 05:59:09 INFO juju.environs.sync sync.go:169 copying
tools/releases/juju-1.17.0.1-saucy-amd64.tgz

2013-12-05 05:59:09 INFO juju.environs.sync sync.go:183 downloaded
tools/releases/juju-1.17.0.1-saucy-amd64.tgz (5583kB), uploading

2013-12-05 05:59:09 INFO juju.environs.sync sync.go:184 download
5583kB, uploading

2013-12-05 06:00:18 INFO juju.environs.sync sync.go:122 copied 2 tools

2013-12-05 06:00:18 INFO juju.environs.sync sync.go:124 generating
tools metadata

2013-12-05 06:00:19 INFO juju.environs.tools simplestreams.go:357
Writing tools/streams/v1/index.json

2013-12-05 06:00:19 INFO juju.environs.tools simplestreams.go:357
Writing tools/streams/v1/com.ubuntu.juju:released:tools.json

2013-12-05 06:00:19 INFO juju.environs.sync sync.go:136 tools metadata written

2013-12-05 06:00:22 INFO juju.environs.bootstrap bootstrap.go:45
bootstrapping environment ap-southeast-2

2013-12-05 06:00:22 INFO juju.environs.tools tools.go:85 reading tools
with major.minor version 1.17

2013-12-05 06:00:22 INFO juju.environs.tools tools.go:93 filtering
tools by version: 1.17.0.1

2013-12-05 06:00:22 INFO juju.environs.tools tools.go:96 filtering
tools by series: precise

2013-12-05 06:00:22 INFO juju.environs.bootstrap bootstrap.go:57
picked newest version: 1.17.0.1

Launching instance

2013-12-05 06:00:29 INFO juju.provider.ec2 ec2.go:418 started instance
i-813692be

 - i-813692be

Waiting for DNS name..

 - ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com

Attempting to connect to
ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com:22

2013-12-05 06:01:45 INFO juju.cloudinit.sshinit configure.go:24
Provisioning machine agent on
ubu...@ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com

Warning: Permanently added
'ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com,54.253.221.206'
(ECDSA) to the list of known hosts.

Logging to /var/log/cloud-init-output.log on remote host

Installing add-apt-repository

Adding apt repository: deb
http://ubuntu-cloud.archive.canonical.com/ubuntu
precise-updates/cloud-tools main

Running apt-get update

Running apt-get upgrade

Installing package: git

Installing package: cpu-checker

Installing package: mongodb-server

Fetching tools: wget --no-verbose -O $bin/tools.tar.gz
'https://s3-ap-southeast-2.amazonaws.com/juju-syd-en-ee-ii/tools/releases/juju-1.17.0.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOAExpires=1701756022Signature=us5XxZbNtgjiPQ5kMVVyC1b9450%3D'

Starting MongoDB server (juju-db)

Bootstrapping Juju machine agent

Connection to ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com closed.

Stopping instance...

2013-12-05 06:05:07 ERROR juju supercommand.go:282 exit status 1

Unfortunately the agent will fail to start because

lucky(~/src/launchpad.net/juju-core) % ldd $(which jujud)

linux-vdso.so.1 =  (0x7fffbc3fe000)

libgo.so.5 = /usr/lib/libgo.so.5 (0x7f57fc52e000)

^^^ this library

libpthread.so.0 = /lib/x86_64-linux-gnu/libpthread.so.0
(0x7f57fc311000)

libm.so.6 = /lib/x86_64-linux-gnu/libm.so.6 (0x7f57fc00c000)

libgcc_s.so.1 = /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x7f57fbdf6000)

libc.so.6 = /lib/x86_64-linux-gnu/libc.so.6 

Re: juju bootstrap w/ gccgo built cli and tools

[David Cheney]

On Fri, Dec 6, 2013 at 3:39 PM, John Arbash Meinel
j...@arbash-meinel.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 2013-12-06 6:24, David Cheney wrote:
 Ok, good news first.

 gccgo compiled tools work fine.

 statically linking libgo also makes the tools as self contained as
 our gc compiled ones.

 Bad news,

 jujud is 40mb, -Os or -O2 effect, in fact the latter makes it a bit
 larger

 the binaries cannot be stripped, the debug data is required for
 operation of the program, i'm guessing gccgo's reflection
 implementation requires the debug symbols.

 Dave

 Two tidbits from me:

 1) jujud on my system is 20MB, so this is approx 2x larger. A fair
 amount, but I think with compression we saw it wasn't quite as big of
 a deal (4.6MB vs 5.5MB in tgz form for the last test, but that was a
 smaller binary).

 2) The Ubuntu security team would *really* like it if we used a system
 libgo.so so that they could supply security fixes for it (like the
 built in SSL libs) without having to have us rebuild all of our jujud
 binaries. Which would save us some of that size (I think you said
 approx 10MB was libgo.so), at a cost of having to have a libgo package
 for all supported platforms. If we name the package appropriately
 (libgo-1.1, etc) then we can probably even still migrate to a
 different version of the runtime when it becomes useful (we just
 install a different package in cloud-init).

That is a fair request, but brings with it a lot of complexity.

It isn't sufficient to have a libgo.so, but one that very closely
matches the one that the tools were built against. Considering that
libgo.so contains all the important parts of the standard library like
crypto, compression, http, json, reflection, etc. We also need to
ensure that whatever tools we are deploying, are running with an up to
date libgo.so.

In short, libgo.so and the tools need to be distributed together -- or
-- the tools need to control the version of libgo that is installed on
the system. This would be easier to accomplish if the tools were
packaged as a deb. But that come with other complications for things
like upgrades.



 John
 =:-
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.13 (Cygwin)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

 iEUEARECAAYFAlKhVPYACgkQJdeBCYSNAAMAGgCXRPrIqBlFaHOEAuLA4zRZGarV
 ZACfS1NM6K6bIZHzaRTBvxO8f/LRrKE=
 =Xa62
 -END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

saucy images on hp cloud

[David Cheney]

Hello,

I am trying to deploy a saucy machine (via the ubuntu charm) on hp cloud.

lucky(~/src/launchpad.net/juju-core) % juju deploy -v
--repository=$HOME/charms local:saucy/ubuntu
verbose is deprecated with the current meaning, use show-log
2014-02-04 01:33:46 INFO juju api.go:231 connecting to API addresses: [
15.185.109.120:17070]
2014-02-04 01:33:46 INFO juju apiclient.go:118 state/api: dialing wss://
15.185.109.120:17070/
2014-02-04 01:33:47 INFO juju apiclient.go:128 state/api: connection
established
Added charm local:saucy/ubuntu-1 to the environment.
2014-02-04 01:33:50 INFO juju.cmd supercommand.go:298 command finished
lucky(~/src/launchpad.net/juju-core) % juju status
environment: hp
machines:
  0:
agent-state: started
agent-version: 1.17.3.1
dns-name: 15.185.109.120
instance-id: 3500935
instance-state: ACTIVE
series: precise
hardware: arch=amd64 cpu-cores=1 mem=1024M root-disk=30720M
  1:
agent-state-info: '(error: no saucy images in az-1.region-a.geo-1
with arches
  [amd64])'
instance-id: pending
series: saucy
services:
  ubuntu:
charm: local:saucy/ubuntu-1
exposed: false
units:
  ubuntu/0:
agent-state: pending
machine: 1

This has not been working for some time.

Is this a known issue, or something I should be raising ?

Cheers

Dave
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: saucy images on hp cloud

[David Cheney]

Antonio,

Awesome, thank you.

Alternatively, if there is a Trusty image available, I can use that
directly, rather than booting a saucy image and dist upgrading it to trusty.

Cheers

Dave


On Wed, Feb 5, 2014 at 1:40 AM, Antonio Rosales 
antonio.rosa...@canonical.com wrote:

 As a side note there is an issue on HP for new users that only access
 to 13.5 HP cloud services not being able to deploy images that are
 indexed from the 12.12 service. I have a support ticket opened with HP
 support to bring the 12.12 images forward to 13.5.

 The issue David seeing is different and I have been able to reproduce.
  I can verify there is a 13.10 image on HP cloud, however it is not
 indexed at:

 http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:hpcloud.json

 I'll get with the team Certified Public Cloud team that indexes these
 images and see if we can get an updated stream.

 -thanks,
 Antonio Rosales

 On Tue, Feb 4, 2014 at 3:37 AM, David Cheney david.che...@canonical.com
 wrote:
  Hello,
 
  I am trying to deploy a saucy machine (via the ubuntu charm) on hp cloud.
 
  lucky(~/src/launchpad.net/juju-core) % juju deploy -v
  --repository=$HOME/charms local:saucy/ubuntu
  verbose is deprecated with the current meaning, use show-log
  2014-02-04 01:33:46 INFO juju api.go:231 connecting to API addresses:
  [15.185.109.120:17070]
  2014-02-04 01:33:46 INFO juju apiclient.go:118 state/api: dialing
  wss://15.185.109.120:17070/
  2014-02-04 01:33:47 INFO juju apiclient.go:128 state/api: connection
  established
  Added charm local:saucy/ubuntu-1 to the environment.
  2014-02-04 01:33:50 INFO juju.cmd supercommand.go:298 command finished
  lucky(~/src/launchpad.net/juju-core) % juju status
  environment: hp
  machines:
0:
  agent-state: started
  agent-version: 1.17.3.1
  dns-name: 15.185.109.120
  instance-id: 3500935
  instance-state: ACTIVE
  series: precise
  hardware: arch=amd64 cpu-cores=1 mem=1024M root-disk=30720M
1:
  agent-state-info: '(error: no saucy images in az-1.region-a.geo-1
 with
  arches
[amd64])'
  instance-id: pending
  series: saucy
  services:
ubuntu:
  charm: local:saucy/ubuntu-1
  exposed: false
  units:
ubuntu/0:
  agent-state: pending
  machine: 1
 
  This has not been working for some time.
 
  Is this a known issue, or something I should be raising ?
 
  Cheers
 
  Dave
 
  --
  Juju-dev mailing list
  Juju-dev@lists.ubuntu.com
  Modify settings or unsubscribe at:
  https://lists.ubuntu.com/mailman/listinfo/juju-dev
 



 --
 Antonio Rosales
 Juju Ecosystem
 Canonical

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: MySQL charm: hook failed: config-changed

[David Cheney]

I don't think this is related to the amount of memory provided, especially
as Daniele is using the local provider.

2014-02-03 16:07:23 INFO juju.worker.uniter context.go:255 HOOK Unable
to open /sys/kernel/security/apparmor/.replace - Permission denied
2014-02-03 16:07:23 INFO juju.worker.uniter context.go:255 HOOK
apparmor_parser: Unable to replace /usr/sbin/mysqld.  Permission
denied; attempted to load a profile while confined?


Looks like an apparmor issue to me.



On Wed, Feb 5, 2014 at 8:04 AM, Daniele Stroppa
daniele.stro...@joyent.comwrote:

 Mark,

 I tried again giving 2G and 4G to the local provider, but no luck.

 Daniele


 On Tue, Feb 4, 2014 at 8:13 PM, Mark Shuttleworth m...@ubuntu.com wrote:


 Hi Daniele, I think the question is whether you have enough memory - can
 you give it more with the local provider?

 Mark


 On 04/02/14 15:55, Daniele Stroppa wrote:

 Maarten, Curtis,

  I tried to deploy mysql on the local provider with as little as 256M of
 memory, but I still get the same error.

  Daniele


 On Tue, Feb 4, 2014 at 12:22 PM, Curtis Hovey-Canonical 
 cur...@canonical.com wrote:

 On Tue, Feb 4, 2014 at 12:29 PM, Maarten Ectors
 maarten.ect...@canonical.com wrote:
  In the past I had an issue with the MySQL charm whereby it had some
  parameter set to 4GB and I did not have enough memory. I don't
 remember the
  exact parameter but it is in the /etc/mysql/my.cnf. By reducing it to a
  smaller number that fits. If this problem still exists then let me
 know and
  we will file a bug...

  The error is ambiguous, something went wrong after mysql installed.
 This does however match what we saw when mysql didn't get enough
 memory.

 When I create deploy mysql on azure and ec2, i do
 juju bootstrap --contraints mem=2G mysql
 For hp I give it 4G
 juju bootstrap --contraints mem=4G mysql

 --
 Curtis Hovey
 Canonical Cloud Development and Operations
 http://launchpad.net/~sinzui

 --
 Juju mailing list
 j...@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju







 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: MySQL charm: hook failed: config-changed

[David Cheney]

You said earlier that you were using the local provider. Have you applied
the raring kernel update to this machine ? This is required if you want to
use LXC on Precise.


On Thu, Feb 6, 2014 at 3:44 AM, Daniele Stroppa
daniele.stro...@joyent.comwrote:

 David,

 I'm using juju 1.16.4.1 on Ubuntu 12.04. If I just deploy mysql with no
 config changes, the issue still exists. However, if I decrease the size of
 the innodb_buffer_pool_size setting (using juju set mysql
 dataset-size='512M') as stated in
 https://bugs.launchpad.net/juju-core/+bug/1236994 it works fine.

 Daniele


 On Wed, Feb 5, 2014 at 9:30 AM, David Cheney 
 david.che...@canonical.comwrote:

 https://bugs.launchpad.net/juju-core/+bug/1236994 is a dup of
 https://bugs.launchpad.net/juju-core/+bug/1247299, both of which are
 claimed to be resolved.

 Daniele, if this is still a problem with the latest development release
 of Juju please reopen one of these or open a new issue as soon as possible.


 On Wed, Feb 5, 2014 at 8:25 PM, Jorge O. Castro jo...@ubuntu.com wrote:

 On Tue, Feb 4, 2014 at 11:10 PM, David Cheney
 david.che...@canonical.com wrote:
  Looks like an apparmor issue to me.

 We've had this problem in the past with MySQL/apparmor/local provider:

 https://bugs.launchpad.net/juju-core/+bug/1236994
 https://bugs.launchpad.net/juju-core/+bug/1247299



 --
 Jorge Castro
 Canonical Ltd.
 http://juju.ubuntu.com/ - Automate your Cloud Infrastructure




-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: saucy images on hp cloud

[David Cheney]

Hi Antonio,

 Do you have an ETA or a ticket number for this issue ?

Cheers

Dave



On Wed, Feb 5, 2014 at 3:09 PM, David Cheney david.che...@canonical.comwrote:

 Hi Antonio,

  Do you have an ETA or a ticket number for this issue ?

 Cheers

 Dave


 On Wed, Feb 5, 2014 at 7:45 AM, David Cheney 
 david.che...@canonical.comwrote:

 Antonio,

 Awesome, thank you.

 Alternatively, if there is a Trusty image available, I can use that
 directly, rather than booting a saucy image and dist upgrading it to trusty.

 Cheers

 Dave


 On Wed, Feb 5, 2014 at 1:40 AM, Antonio Rosales 
 antonio.rosa...@canonical.com wrote:

 As a side note there is an issue on HP for new users that only access
 to 13.5 HP cloud services not being able to deploy images that are
 indexed from the 12.12 service. I have a support ticket opened with HP
 support to bring the 12.12 images forward to 13.5.

 The issue David seeing is different and I have been able to reproduce.
  I can verify there is a 13.10 image on HP cloud, however it is not
 indexed at:

 http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:hpcloud.json

 I'll get with the team Certified Public Cloud team that indexes these
 images and see if we can get an updated stream.

 -thanks,
 Antonio Rosales

 On Tue, Feb 4, 2014 at 3:37 AM, David Cheney david.che...@canonical.com
 wrote:
  Hello,
 
  I am trying to deploy a saucy machine (via the ubuntu charm) on hp
 cloud.
 
  lucky(~/src/launchpad.net/juju-core) % juju deploy -v
  --repository=$HOME/charms local:saucy/ubuntu
  verbose is deprecated with the current meaning, use show-log
  2014-02-04 01:33:46 INFO juju api.go:231 connecting to API addresses:
  [15.185.109.120:17070]
  2014-02-04 01:33:46 INFO juju apiclient.go:118 state/api: dialing
  wss://15.185.109.120:17070/
  2014-02-04 01:33:47 INFO juju apiclient.go:128 state/api: connection
  established
  Added charm local:saucy/ubuntu-1 to the environment.
  2014-02-04 01:33:50 INFO juju.cmd supercommand.go:298 command finished
  lucky(~/src/launchpad.net/juju-core) % juju status
  environment: hp
  machines:
0:
  agent-state: started
  agent-version: 1.17.3.1
  dns-name: 15.185.109.120
  instance-id: 3500935
  instance-state: ACTIVE
  series: precise
  hardware: arch=amd64 cpu-cores=1 mem=1024M root-disk=30720M
1:
  agent-state-info: '(error: no saucy images in
 az-1.region-a.geo-1 with
  arches
[amd64])'
  instance-id: pending
  series: saucy
  services:
ubuntu:
  charm: local:saucy/ubuntu-1
  exposed: false
  units:
ubuntu/0:
  agent-state: pending
  machine: 1
 
  This has not been working for some time.
 
  Is this a known issue, or something I should be raising ?
 
  Cheers
 
  Dave
 
  --
  Juju-dev mailing list
  Juju-dev@lists.ubuntu.com
  Modify settings or unsubscribe at:
  https://lists.ubuntu.com/mailman/listinfo/juju-dev
 



 --
 Antonio Rosales
 Juju Ecosystem
 Canonical




-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: no log tests

[David Cheney]

Hi Rog,

I tried this but it causes unrelated test failures as tests which scan for
output now no longer find anything as the global log level is raised to
ERROR.

Maybe this is a bug.

Cheers

Dave


On Thu, Mar 27, 2014 at 8:21 AM, roger peppe rogpe...@gmail.com wrote:

 Alternatively you can run the tests with -juju.log ERROR (sent from my
 phone so unable to double check the extact syntax)
 On 26 Mar 2014 19:00, Z. Nate Finch nate.fi...@canonical.com wrote:

 So, I often get failed tests that are so obscured by our log output that
 I can't even tell what's failing.  So I made a little thing to filter out
 the logs.  Yes, I'm sure you could do this with some bash stuff or grep or
 whatever, but I'll write bash when my job or my life depends on it, and not
 before.

 just do

 go get github.com/natefinch/nolog

 And then you can run nolog instead of go test.  It passes arguments
 through, so you can do

 nolog ./... -gocheck.v

 (or whatever)
 and it'll output everything except the lines that start with [LOG], which
 makes it a lot easier to see

 The filtering is really dumb, it just filters out lines that start with
 [LOG], but it seems to work fine for my needs.

 -Nate

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Go 1.3 beta can build Juju

[David Cheney]

Hello,

Go 1.3 beta will ship this week, or at some point before the
conference on the 24th.

For a long time Go 1.3 beta could not build Juju properly, the tests
would crash the runtime and other strange issues.

I've been able to work with the upstream to provide them with a
working trusty image and Juju development environment (let's hear it
for the ubuntu charm!!) so they could reproduce and debug the issue.

This isn't the first time that Juju has broken the Go trunk, in the
previous 1.2 cycle, Juju code showed up some interesting reflection
and escape analysis bugs.

Cheers

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Go 1.3 beta can build Juju

[David Cheney]

On Thu, Apr 17, 2014 at 11:22 AM, Tim Penhey tim.pen...@canonical.com wrote:
 On 17/04/14 13:20, David Cheney wrote:
 Hello,

 Go 1.3 beta will ship this week, or at some point before the
 conference on the 24th.

 For a long time Go 1.3 beta could not build Juju properly, the tests
 would crash the runtime and other strange issues.

 I've been able to work with the upstream to provide them with a
 working trusty image and Juju development environment (let's hear it
 for the ubuntu charm!!) so they could reproduce and debug the issue.

 This isn't the first time that Juju has broken the Go trunk, in the
 previous 1.2 cycle, Juju code showed up some interesting reflection
 and escape analysis bugs.

 So are you saying that the juju codebase exercises conditions that
 aren't covered in upstream golang tests?

Yes


 Are they adding tests so it doesn't happen again?

Yes


 Tim


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Go 1.3 beta can build Juju

[David Cheney]

On Thu, Apr 17, 2014 at 11:24 AM, Tim Penhey tim.pen...@canonical.com wrote:
 On 17/04/14 13:24, David Cheney wrote:
 On Thu, Apr 17, 2014 at 11:22 AM, Tim Penhey tim.pen...@canonical.com 
 wrote:
 On 17/04/14 13:20, David Cheney wrote:
 Hello,

 Go 1.3 beta will ship this week, or at some point before the
 conference on the 24th.

 For a long time Go 1.3 beta could not build Juju properly, the tests
 would crash the runtime and other strange issues.

 I've been able to work with the upstream to provide them with a
 working trusty image and Juju development environment (let's hear it
 for the ubuntu charm!!) so they could reproduce and debug the issue.

 This isn't the first time that Juju has broken the Go trunk, in the
 previous 1.2 cycle, Juju code showed up some interesting reflection
 and escape analysis bugs.

 So are you saying that the juju codebase exercises conditions that
 aren't covered in upstream golang tests?

 Yes


 Are they adding tests so it doesn't happen again?

 Yes

 This is actually kinda cool.

I'm glad that we can help make Go more reliable. Issues which have
been reported in the past on closed source codebases have required
significant hoop jumping to get to the point of a reproducable test
case.

As an open source project we can help the upstream reproduce issues by
just pointing them to our source and helping them get there
development environment going. All I did was deploy cs:trusty/ubuntu,
checkout juju and run make install-dependencies, then handed over the
keys.


 Is there a release log for 1.3 that describes the changes from 1.2?

It's being written, we're still  4 weeks out from release, so is not
complete, http://tip.golang.org/doc/go1.3

 Tim


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: juju 1.20.0 and future plans.

[David Cheney]

Excellent writeup. I agree with your conclusions -- we must reduce the
number of open issues, not reclassify them. My vote has always been to
close an issue that we cannot realistically fix in the next cycle, but
I understand this has always been unpopular.

On Fri, Apr 18, 2014 at 5:47 AM, Curtis Hovey-Canonical
cur...@canonical.com wrote:
 NOW

 I moved a lot of bugs from 1.19.1 to 1.20.0 to make it easier to see
 out immediate priorities. If we fix a bug in a future milestone I will
 retarget it to the current goal. The intent is to make is clear what
 must be fixed verses issues that may be fixed.

 Their are about 20 issues I *think* need to be resolved that will
 define 1.19.1 features and fixes. I hope to release 1.19.1 next week.

 There are about 50 bugs targeted to 1.20.0. This is 2 weeks of bugs.
 We can expect more bugs as users try the new features. So we are
 really 3 or more weeks away from 1.20.0  We can anticipate 1.19.2 and
 1.19.3, maybe 1.19.4 before .1.20.0

 FUTURE

 We fix about 25 bugs a week. Many bugs are fixed a couple days after
 we discover them. I think our milestones have a capacity of 20 issue,
 with room for another 5 added as we fix new bugs.

 We fixed about 400 bugs in juju-core and its libraries in the last 6
 months. That is more than half of all bugs ever fixed in the project.
 Well done. However, only 100 of those bugs were known when we planned
 the cycle; 3 out of 4 bugs are fallout from improvements or escalated
 issues from our past. Developers are reporting many of the fallout
 bugs a couple days after a merge into trunk. Users report the
 remaining bugs after a release.

 Our 6 month cycle capacity is 100 high and critical issues. This a
 problem for us because there are more than 250 High bugs in Lp. We
 have 15 months of issues we want to commit to fix soon. We cannot. Our
 goals change every few months; We, our stakeholders, and our users are
 misinformed about our priorities and we cannot make plans based on the
 high bugs. We need to lower the importance of more than half the high
 bugs to make it clear that some issues will only be fixed by
 opportunity or assistance.

 I am aware that we intend to have more engineers in the next cycle,
 but they will need 3 or months of become competent. We can set no more
 than 150 bugs as high as stretch goal.

 Lowering bugs to medium is not helpful. I have years of experience
 with how projects use bugs in Lp. A medium bug in a large project is
 not more likely to be fixed than a low bug. Opportunity doesn't
 discriminate by importance. Bugs classified as medium importance are
 not likely to be escalated to high in the next cycle. Each cycle with
 new priorities continues to leap ahead of the medium bugs.

 If lowering the importance of our bugs is a political problem, we can
 create a milestone call horizon that has just the bugs we expect to
 fix in the next 6 months. The unscheduled high bugs will violate the
 intent of scheduling our commitments. In reality the unscheduled bugs
 will linger and likely be demoted to Low as interest in them wanes.

 --
 Curtis Hovey
 Canonical Cloud Development and Operations
 http://launchpad.net/~sinzui

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/ssh updates

[David Cheney]

On Sun, Apr 27, 2014 at 8:19 AM, Martin Packman
martin.pack...@canonical.com wrote:
 I've poked utils/ssh a bit during gophercon.

 Unfortuately the update to use tip of go.crypto can't land right now
 as it requires a couple of standard library crypto bits that go 1.1.2
 lacks. Perhaps we can talk about bumping the requirement for building
 trunk juju-core to go 1.2 and adding that in our golang ppa.

Oh. I thought that we already had the requirement of 1.2. So plus one
on making that a dependency, 1.2.1 is the shipping version in trusty
and will be for a long time, so I think it makes sense to target that.

 The branch to not generate lots of rsa keys for the tests did land,
 which makes the testrun a little faster... or a lot faster on arm.

 Before:
 http://paste.ubuntu.com/7338971/
 ok  launchpad.net/juju-core/utils/ssh   375.048s

 After:
 http://paste.ubuntu.com/7338978/
 ok  launchpad.net/juju-core/utils/ssh   4.533s


 My chromebook is now much happier... except when it has to run lbox.

 Martin

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Preferred wrapping for long function declarations

[David Cheney]

I don't want to have this bikeshed - I vote for people to use their
best judgement and permit anything which fits through gofmt.

On Tue, May 13, 2014 at 3:37 PM, John Meinel j...@arbash-meinel.com wrote:
 In the risk of running a bikeshedding paint-fest, I'm curious what the
 best-recommended practice is for wrapping function declarations that get a
 bit wide.

 Currently I'm writing a function that looks like this:
 func (c *CustomCaller) MethodCaller(rootName string, version int, methodName
 string) (rpcreflect.MethodCaller, error) {
 }

 But the line is about 119 characters, which isn't terrible, but is a bit
 wide.

 Should we:

 Not worry about it, you don't usually need to paste into an email and have
 crummy wrapping anyway, and 80-character terminals are useless. (and who
 would want to actually look at more than one document side-by-side anyway)
 Wrap everything to a single indent:
 This can be a slightly shallow:
 func (c *CustomCaller) MethodCaller(
 rootName string,
 version int,
 methodName string) (
 rpcreflect.MethodCaller,
 error) {
 }
 or go all the way with:

 func (c *CustomCaller) MethodCaller(
 rootName string,
 version int,
 methodName string,
 ) (
 rpcreflect.MethodCaller,
 error,
 ) {
 }

 (note that gofmt forces the )( and ){ to be left aligned).
 Of the two here I probably prefer the latter, because you can at least
 visually distinguish which collection is the parameters and what grouping is
 the return values.
 args then errors:
 func (c *CustomCaller) MethodCaller(
 rootName string, version int, methodName string) (
 rpcreflect.MethodCaller, error) {
 }
 My particular unhappiness is because the opening character has to be on the
 previous line so that we don't get the implicit semicolons.
 Fit what you can in ~even lines:
 func (c *CustomCaller) MethodCaller(rootName string, version int,
 methodName string) (rpcreflect.MethodCaller, error) {
 }
 This also the example for  wrap at 80 characters because you can't break
 methodName from string, you have to end the line with punctuation.
 close to 8, break out errors
 func (c *CustomCaller) MethodCaller(rootName string, version int, methodName
 string) (
 rpcreflect.MethodCaller, error) {
 }
 Note that my email client forces the wrapping on the first line.
 You're doing it wrong, functions with 3 params and 2 return types should be
 turned into some sort of Params struct instead. (I'd agree if it was 5 but
 3 and 2 isn't really very many.)


 I think our policy is (3), and maybe that's the best of what I've
 encountered.It means that either everything is on one line, or everything is
 a single value per line, with clear delineation between args and errors.

 Thoughts?
 John
 =:-

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Making juju work better with gccgo

[David Cheney]

Thanks Tim,

I think we're in the position that even if you don't use gccgo for
your own development (understandable, it is slower), now that we have
platforms we support in Trusty that must use gccgo, armv8 and ppc64el,
for any new development, the code has to pass under both compilers.

This is a bit of a bummer, but at least if you are running trusty on
your development environment, you have all the bits you need[1][2].

Running tests, for a single package then becomes

go test  go test -compiler=gccgo

With respect to the whole Juju test suite, even after months of work
there still remain a few tests which do not pass under trusty, these
are tracked with bugs in lp, just look for the gccgo label if you find
yourself at a loose end.

Cheers

Dave

[1] Prior to trusty, you'd probably be using gccgo-4.8, which is not
of acceptable quality for use with Juju.
[2] If for some reason you don't want to upgrade to trusty, we may be
able to ask Doko for a backport of gccgo-4.9, but really, just upgrade
to trusty.

On Fri, May 16, 2014 at 12:44 PM, Tim Penhey tim.pen...@canonical.com wrote:
 I realise I left this bit hanging:

 On 16/05/14 14:41, Tim Penhey wrote:
 A key issue that Dave and I have been poking with a stick is the test
 suite for juju/errgo, as it had some failures with gccgo (all passed
 with gc).

 In the end it had to do with the runtime.Caller method for stack
 analysis, and the synthetic methods that gccgo was creating.  The
 solution was simply enough to convert the tests to use gocheck, that way
 the synthetic methods that were being created are higher in the stack
 that we ever look in the tests, and it is all good.

 Tim


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: [Proposal] Requiring Go 1.2 across the board

[David Cheney]

William, could you please raise this at the team leads meeting tonight
on my behalf. The steps look quite straight forward, they just need to
go on a card and get done.

On Fri, May 16, 2014 at 5:53 PM, William Reade
william.re...@canonical.com wrote:
 +1


 On Fri, May 16, 2014 at 9:18 AM, Andrew Wilkins
 andrew.wilk...@canonical.com wrote:

 On Fri, May 16, 2014 at 3:08 PM, David Cheney david.che...@canonical.com
 wrote:

 Hello,

 This is a proposal that we raise the minimum Go spec from Go 1.1 to Go
 1.2.

 Motivation:

 * Now that T is out, we have to support Juju on two series (Precise
 and Trusty[1]) with two compilers, gc and gccgo.
 * gccgo-4.9, which is version we using in trusty, supports the Go 1.2
 spec
 * gc-1.1.2 which is available in a ppa for precise, obviously supports Go
 1.1
 * gc-1.2.1 is the shipping compiler on Trusty, it supports Go 1.2

 So now we have three compilers, two versions of gc, and one of gccgo.

 We are in the situation where code written for gccgo under T on ppc64
 or armv8 will not pass the bot running Go 1.1.1.

 For this reason I would like to reduce this matrix.

 Recommendation:

 * Get the trusty compiler into a backport ppa for precise[1]
 * Upgrade the bot to use that compiler, raising the minimum compiler
 spec to 1.2 across the board


 SGTM. The bot will be changing soon to a Jenkins lander; seems like an
 opportune time for making the change.


 Thoughts / Discussion / Spoiled fruit ?

 Dave

 [1] I am ignoring the intermediate, non LTS series', as there are no
 charms for them, nor do CTS offer support for them. If this is
 unacceptable, anything which applies to Precise wrt. backports, also
 applies to Q, R and S.

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

What is required to switch back to using the tip of goose ?

[David Cheney]

Why is the Openstack provider pinned to an old version of lauchpad.net/goose ?

What is required to use the latest version ?

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: What is required to switch back to using the tip of goose ?

[David Cheney]

There also appears to be a small API breakage

# launchpad.net/juju-core/provider/openstack
provider/openstack/storage.go:59: assignment count mismatch: 2 = 3

On Fri, May 16, 2014 at 7:48 PM, John Meinel j...@arbash-meinel.com wrote:
 AFAIK there is one patch which landed a couple of days ago that changes an
 API that used to only return an io.ReadCloser, error that now returns
 io.ReadCloser, Headers, error, and we can safely ignore the Headers (someone
 else needed it who wanted to use goose).

 I think the only reason we aren't running an updated version is because
 nobody has cared to do so. I believe it is a 3 character fix for Get() but I
 haven't really tested the rest.

 John
 =:-



 On Fri, May 16, 2014 at 12:47 PM, David Cheney david.che...@canonical.com
 wrote:

 Why is the Openstack provider pinned to an old version of
 lauchpad.net/goose ?

 What is required to use the latest version ?

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: What is required to switch back to using the tip of goose ?

[David Cheney]

Oh yes, we are talking about the same thing. I'll propose a branch

On Fri, May 16, 2014 at 7:50 PM, David Cheney
david.che...@canonical.com wrote:
 There also appears to be a small API breakage

 # launchpad.net/juju-core/provider/openstack
 provider/openstack/storage.go:59: assignment count mismatch: 2 = 3

 On Fri, May 16, 2014 at 7:48 PM, John Meinel j...@arbash-meinel.com wrote:
 AFAIK there is one patch which landed a couple of days ago that changes an
 API that used to only return an io.ReadCloser, error that now returns
 io.ReadCloser, Headers, error, and we can safely ignore the Headers (someone
 else needed it who wanted to use goose).

 I think the only reason we aren't running an updated version is because
 nobody has cared to do so. I believe it is a 3 character fix for Get() but I
 haven't really tested the rest.

 John
 =:-



 On Fri, May 16, 2014 at 12:47 PM, David Cheney david.che...@canonical.com
 wrote:

 Why is the Openstack provider pinned to an old version of
 lauchpad.net/goose ?

 What is required to use the latest version ?

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: What is required to switch back to using the tip of goose ?

[David Cheney]

FYI. This change has landed, please do the godeps dance.

On Fri, May 16, 2014 at 7:54 PM, David Cheney
david.che...@canonical.com wrote:
 Oh yes, we are talking about the same thing. I'll propose a branch

 On Fri, May 16, 2014 at 7:50 PM, David Cheney
 david.che...@canonical.com wrote:
 There also appears to be a small API breakage

 # launchpad.net/juju-core/provider/openstack
 provider/openstack/storage.go:59: assignment count mismatch: 2 = 3

 On Fri, May 16, 2014 at 7:48 PM, John Meinel j...@arbash-meinel.com wrote:
 AFAIK there is one patch which landed a couple of days ago that changes an
 API that used to only return an io.ReadCloser, error that now returns
 io.ReadCloser, Headers, error, and we can safely ignore the Headers (someone
 else needed it who wanted to use goose).

 I think the only reason we aren't running an updated version is because
 nobody has cared to do so. I believe it is a 3 character fix for Get() but I
 haven't really tested the rest.

 John
 =:-



 On Fri, May 16, 2014 at 12:47 PM, David Cheney david.che...@canonical.com
 wrote:

 Why is the Openstack provider pinned to an old version of
 lauchpad.net/goose ?

 What is required to use the latest version ?

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: store migration

[David Cheney]

On Tue, May 20, 2014 at 11:55 PM, Francesco Banconi
francesco.banc...@canonical.com wrote:
 Hi all,

 FYI this morning the UI team discussed about possibile paths to migrate the 
 store code from lp:juju-core to Github.
 We are sharing this plan so that we can synchronize/collaborate with other 
 teams with similar needs in their todo list.

 Here are the store package dependencies as obtained with go list:

 Deps:
 $ go list -f {{range .Deps}}{{println .}}{{end}} 
 launchpad.net/juju-core/store | grep juju-core
 launchpad.net/juju-core/charm
 launchpad.net/juju-core/charm/hooks
 launchpad.net/juju-core/juju/osenv
 launchpad.net/juju-core/schema
 launchpad.net/juju-core/thirdparty/pbkdf2
 launchpad.net/juju-core/utils
 launchpad.net/juju-core/utils/set
 launchpad.net/juju-core/utils/zip

 Tests deps:
 $ for i in `go list -f {{range .Deps}}{{println .}}{{end}} 
 launchpad.net/juju-core/store`; do go list -f {{range 
 .XTestImports}}{{println .}}{{end}} $i; done | grep juju-core | sort | uniq
 launchpad.net/juju-core/charm
 launchpad.net/juju-core/charm/testing
 launchpad.net/juju-core/environs/config
 launchpad.net/juju-core/juju/osenv
 launchpad.net/juju-core/schema
 launchpad.net/juju-core/testing
 launchpad.net/juju-core/testing/filetesting
 launchpad.net/juju-core/testing/testbase
 launchpad.net/juju-core/utils
 launchpad.net/juju-core/utils/set
 launchpad.net/juju-core/utils/zip

 As you can see, there are some incremental steps we will need to follow to 
 achieve our goal, I’ll try to describe them below including notes from 
 William. I suppose we can encounter complications in this path, but hopefully 
 at the end we’ll have a good starting point for the store.
 William agreed on the goals of this migration (having a common/separate 
 module which can be reused by both juju-core and the GUI).

 Just two notes before sketching a possible plan:
 - the store must be able to be configured to use its own db or the juju-core 
 one based on the context it is used;
 - we need a way to migrate partial Bazaar commit history to git (perhaps 
 someone has already experience with this?).

 A possible plan is as follow:

 1) Migrate juju-core/thirdparty to juju/thirdparty (Github).
 Since there are no dependencies here, this seems to be a good first candidate.

nac, these are out horrors. The code we forked should either be pushed
back upstream, and/or captured by godeps. I think the whole thirdparty
thing happened before we had godeps.


 2) Migrate juju-core/utils to juju/utils (Github).
 package:
  launchpad.net/juju-core/utils
 deps:
  launchpad.net/juju-core/juju/osenv
  launchpad.net/juju-core/thirdparty/pbkdf2
 tests deps:
  launchpad.net/juju-core/juju/osenv
  launchpad.net/juju-core/testing/testbase

Sounds like the best candidate to start with.

 I did not look at juju/osenv: we might want to migrate that as well, or just 
 refactor the code to remove the dependency. Note that each time we move a 
 package we need to also move the relevant code in juju-core/testing to 
 juju/testing. The latter already exists on Github. The juju-core/testing 
 module has lots of dependencies on other packages in juju-core, so if we 
 encounter those we will need to handle that (I presume by refactoring test 
 code). In the utils case, juju-core/testing/testbase does not seem to depend 
 on anything, so we should be ok.

 3) Migrate juju-core/schema to juju/utils/schema. The schema package has no 
 juju-core dependencies.

Yes to migrating it, no to renaming it. Schema is useful as a top
level project, rather than buried in an amorphous utils repo.


 4) Migrate juju-core/charm. This has some pre-requisites. Basically IIUC 
 charm defines config and meta and those definitions are tangled with the 
 underlaying Mongo documents. William suggested to decouple that, implementing 
 separate data structures to be used to (de)serialize data. This way, changes 
 to charm database structure can be detected earlier and core developers are 
 able to react accordingly. Soon this could also involve actions data.

I don't see the value in splitting off this repo unless you need it
for the store.


 5) Move the store.

 For each step AFAICT what we need to do is similar to the following:
 1) possible preliminary work to move the testing stuff;
 2) create Github project (if it does not exist);
 3) add readme, copying, license files etc;
 4) notify developers we are locking the package;
 5) migrate the code;
 6) fix package imports if required (e.g. for sub-packages);
 7) fix package tests;
 8) land a juju-core branch which:
- removes the package;
- fixes all the imports;
- includes the new dependency info in the dependencies.tsv file;
 9) notify developers about the new external dependency.

 Thanks!

 --
 Francesco


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify 

Re: (not) using loggo in github.com/juju/errgo

[David Cheney]

SGTM. https://github.com/juju/errgo/pull/6

On Tue, May 20, 2014 at 10:06 PM, John Meinel j...@arbash-meinel.com wrote:
 I'm fine having our split out packages depend on each other where it is
 actually useful (errors clearly depends on errgo), but it does sound like
 loggo isn't a very strong dependency and can just be removed.

 John
 =:-



 On Tue, May 20, 2014 at 3:27 PM, Nate Finch nate.fi...@canonical.com
 wrote:

 Actually, I just looked at the code again, and the code that uses loggo
 will be compiled out unless you modify the code to change a constant.  Let's
 just remove the code and the package include, and if people want to modify
 the code to print out debugging info, they can do that however they want.


 On Tue, May 20, 2014 at 7:24 AM, Nate Finch nate.fi...@canonical.com
 wrote:

 I just noticed that errgo uses loggo for one single debug statement.  I
 think this is a mistake if we want errgo to be reusable by the community.
 We should strive to make our independent packages actually independent
 removing unnecessary dependencies is a big part of that.  We shouldn't force
 people to include loggo in their product if all they want is errgo
 especially given that we're barely using loggo in the package.

 Can we remove the debug code, or use the stdlib's logging or something?
 I don't think we're getting a lot of value out of using loggo in this
 package.



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: store migration

[David Cheney]

 With regard to the thirdparty package in core, I don't see that we need it any
 more. The only thing in it is pbkdf2, which is available from go.crypto which 
 we
 import from anyway for ssh authorised keys functionality. So I say we nuke it.

I'm going to fix this right now.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: (not) using loggo in github.com/juju/errgo

[David Cheney]

This has been done. github.com/juju/errgo no longer has a dependency
on github.com/juju/loggo

On Wed, May 21, 2014 at 7:39 AM, David Cheney
david.che...@canonical.com wrote:
 SGTM. https://github.com/juju/errgo/pull/6

 On Tue, May 20, 2014 at 10:06 PM, John Meinel j...@arbash-meinel.com wrote:
 I'm fine having our split out packages depend on each other where it is
 actually useful (errors clearly depends on errgo), but it does sound like
 loggo isn't a very strong dependency and can just be removed.

 John
 =:-



 On Tue, May 20, 2014 at 3:27 PM, Nate Finch nate.fi...@canonical.com
 wrote:

 Actually, I just looked at the code again, and the code that uses loggo
 will be compiled out unless you modify the code to change a constant.  Let's
 just remove the code and the package include, and if people want to modify
 the code to print out debugging info, they can do that however they want.


 On Tue, May 20, 2014 at 7:24 AM, Nate Finch nate.fi...@canonical.com
 wrote:

 I just noticed that errgo uses loggo for one single debug statement.  I
 think this is a mistake if we want errgo to be reusable by the community.
 We should strive to make our independent packages actually independent
 removing unnecessary dependencies is a big part of that.  We shouldn't 
 force
 people to include loggo in their product if all they want is errgo
 especially given that we're barely using loggo in the package.

 Can we remove the debug code, or use the stdlib's logging or something?
 I don't think we're getting a lot of value out of using loggo in this
 package.



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: moving bzr wip branch to git

[David Cheney]

Nice

On Wed, Jun 4, 2014 at 2:08 PM, Jesse Meek jesse.m...@canonical.com wrote:
 So the command that worked for me (thanks for the help Tim) was:

 ~/go/src/github.com/juju/juju$ (cd ~/go/src/launchpad.net/juju-core  bzr
 diff --color=never -r ancestor::parent) | patch  -p0 --merge

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju is on github

[David Cheney]

Make it so

On Thu, Jun 5, 2014 at 6:59 AM, Menno Smits menno.sm...@canonical.com wrote:
 On 5 June 2014 06:58, Jorge Niedbalski jorge.niedbal...@canonical.com
 wrote:


 2cents: Why not rename the CONTRIBUTING file to CONTRIBUTING.md
 so it will be rendered as a markdown file by github?


 That seems sensible. Any objections?



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: End Of Review marker

[David Cheney]

Rietveld also supports git

On Thu, Jun 12, 2014 at 8:46 PM, Ian Booth ian.bo...@canonical.com wrote:
 It's also the same when you are responding to review comments. You want to 
 mark
 them all as Done (or whatever) and have those go out in a batch to let the
 reviewer know they can come back and +1.

 Surely we're not the only people annoyed by this? I wonder what more 
 experienced
 github users do. Or maybe people know that github sucks for code reviews and 
 use
 gerrit or something else?

 On 12/06/14 20:38, Horacio Duran wrote:
 Hey, I don't know if this bugs everyone or just me but it happens very
 often that I am working while people are reviewing my code on gh. While
 people is reviewing and commenting on the code I keep getting mails and the
 diff page from the pr keeps changing. To know when its all done and I can
 finally try to answer/fix all the comments I usually wait until my phone
 stops ringing madly with mails but I think that we could do better. At the
 end of the diff page there is a comment box where you can add comments
 (where you usually add your $$merge$$ or LGTM) We could add something
 there, like Done just to let the author know we are done with the review
 and not just reading a big confusing chunk of code.
 What do you people think?




 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Splitting out state/api into its own repo

[David Cheney]

On Fri, Jun 27, 2014 at 4:21 PM, John Meinel j...@arbash-meinel.com wrote:
 Just my quick thought, I think moving it out from state/api into just a
 top level api would  be reasonable and a lot less clumsy than trying to
 pull it out into an entirely separate repository.

+1

I don't think the api package is useful outside Juju (at this time)
and splitting it into another repo just doubles the amount of work.


 I'm not sure if Gustavo realized that state/apiserver is the HTTP(-ish,
 its really just JSON rpc over a websocket) interface to state. state/api
 is all go client code (agents or user client code). And it would at least be
 really nice for all those things to have an import test that says does not
 depend on anything under state.)

 It certainly would make it clearer that nothing that is using the api should
 import something from, say, apiserver, or state directly.

 John
 =:-


 On Thu, Jun 26, 2014 at 9:23 PM, roger peppe rogpe...@gmail.com wrote:

 I have a slightly different proposal, inspired by the recent
 Go proposal for internal imports (http://golang.org/s/go14internal),
 which currently looks like it will actually be implemented.

 We move all public facing APIs into top level packages within
 the juju repo, and move everything else under
 github.com/juju/juju/internal

 Packages I would suggest should be exported:

 github.com/juju/juju/api// moved from state/api
 github.com/juju/juju/cmd/juju
 github.com/juju/juju/cmd/jujud
 ... (all the other commands, but not the cmd package itself)
 github.com/juju/
 github.com/juju/juju/cmd/internal/jujucmd// containing
 supercommand.go from current cmd
 github.com/juju/juju/juju  // but only the API-related pieces

 Some packages would want moving, because they're user-facing
 but currently inside packages that we would probably not want to
 export:

 github.com/juju/environs/configstore

 is the only one I can think of right now.

 Others would want splitting. For example, the environs package
 is a mix between user-facing and internal stuff right now.
 It would be great to take out all the user-facing config file stuff
 (that might sit
 well inside the juju package).

 Still others would benefit from being made available externally.
 I think the rpc package is probably one of those that would
 sit well inside its own repo.

 Some packages could benefit from having their own internal
 directory - the uniter is one of those, for example. The
 apiserver too has many sub-packages that should not really
 be visible to the rest of juju.

 In the end, we should end up with an API that it might actually be
 feasible to stabilise. I'd like juju to move under gopkg.in at some
 point, providing useful stability guarantees for external users
 that might want to build Go programs based on our code.

   cheers,
 rog.

 On 26 June 2014 17:41, Eric Snow eric.s...@canonical.com wrote:
  (I've put a more structured proposal below, but here's some context.)
 
  Over the last couple weeks I've been spinning up on the juju code
  base, which is large enough to dissipate any hope of understanding it
  all quickly. wink Most of what I've focused on is relative to the
  juju tools and the remote API, both for the sake of backup/restore.
 
  One thing that threw me off is that it has not been obvious that the
  code under `state/api` is the public-facing API for juju's state (as
  someone recently explained to me).  For a while I thought `state/api`
  held the state API client code, but from what I understand now it
  actually contains all the public facing code for the juju state API
  (and [consequently?] for juju as a whole?).  In some regard I would
  expect a public API to be sit in a top-level package (rather than
  nested down like it is).
 
  A couple of other things got in the way.  For one, we basically don't
  have any documentation for the API.  I expect that it will mirror the
  documentation for the juju subcommands/tools pretty closely, but
  regardless the doc doesn't exist.  Setting up godoc for the api
  package would be great and so would a new page in the juju
  documentation.  I realize there has been some discussion on this point
  of late, from which I expect a doc will take shape in the short term.
  In the meantime, we've actually been telling people to wrap calls to
  the CLI tools rather than using the API.  The documentation side of
  things is a somewhat orthogonal, though topically related, issue.
 
  For another, while there has been a pretty good effort to keep the
  `api` package relatively un-entwined from the rest of juju, there were
  a few times when I found it hard to follow what was going on.  This
  was particularly true of the underlying state RPC implementation,
  though at this point things make a lot more sense.  Having a separate
  repo would help delineate the boundary between the API and juju
  itself, which should in turn help make the API code easier to follow.
 
  So...
 

Re: Landing bot job changed

[David Cheney]

Additionally, the bot is no longer reporting back to the PR that the
build failed to merge.

On Mon, Jul 7, 2014 at 11:11 AM, David Cheney
david.che...@canonical.com wrote:
 The lander is broken

 + exit 0
 All passed, sending merge
 /tmp/hudson6791049747065303228.sh: line 44:
 /home/ubuntu/jenkins-github-lander/bin/lander-merge-result: No such
 file or directory
 /tmp/hudson6791049747065303228.sh: line 44:
 /home/ubuntu/jenkins-github-lander/bin/lander-merge-result: No such
 file or directory
 Build step 'Execute shell' marked build as failure
 Description set: davecheney 128-update-ssh-terminal

 On Sat, Jul 5, 2014 at 9:56 PM, Martin Packman
 martin.pack...@canonical.com wrote:
 I have switched the jenkins landing bot to a new job, as part of the
 work on getting it faster and more reliable. If you want to check back
 on old results, they are preserved for now under a different job name:

 http://juju-ci.vapour.ws:8080/job/github-merge-juju-old/

 I'm actually having second thoughts in writing this, maybe just
 renaming the job to keep the links in existing prs correct would have
 been better. I can still do that if anyone has strong opinions.

 Martin

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Conditional compilation primer

[David Cheney]

Hello,

Tim asked me to circulate this to aide those who have to review code
that deals with conditional compilation.

http://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool

Cheers

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Can we get rid of the hash(password) dance?

[David Cheney]

Yes, please!

On Thu, Jul 17, 2014 at 2:49 PM, John Meinel j...@arbash-meinel.com wrote:
 Michael is working on changing how we handle sessions with Mongo, and
 noticed that his first attempt started running into Auth failures.
 It turned out that this was because of the hash(password) dance. (For those
 who don't know, in certain circumstances we used to seed the password for
 the database with the hash(password) and then once we had a secure channel
 we would replace it with the real password.)

 I believe all of our production bootstrap code has gotten rid of the
 password dance, because we now just use cloud-init to bring up a machine and
 then SSH into that machine to finish provisioning. Thus all the information
 is already over the secure SSH channel, instead of the insecure cloud-init
 user data.

 From what I can tell poking around the code base, the only place that still
 uses the hash(password) is actually in the Dummy provider.

 I feel like we're at a point where we can safely remove that from the Dummy
 provider, and also remove the fallback code in our 'connect to the database'
 code. (If we leave it in, then I think after changing the password just
 reconnecting to the database is fine, because it should happen infrequently.

 Thoughts?

 John
 =:-


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Possible landing bot issues

[David Cheney]

Hello,

I think there are two problems with the current landing bot.

1. the bot doesn't appear to NACK changes that are not gofmt'd, this
means changes land which others cannot fix because their .gitconfig
prevents them.

2. the bot doesn't appear to be enforcing it's lockdown rules as the
comment it places on the PR contains the magic string that the bot
looks for when it see's another $$ instruction.

Thank you for your time.

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Possible landing bot issues

[David Cheney]

I managed to sneak one in an hour ago, is the fix live ?

On Thu, Jul 31, 2014 at 10:18 AM, Ian Booth ian.bo...@canonical.com wrote:

 I think there are two problems with the current landing bot.


 Thanks for raising the issues.


 1. the bot doesn't appear to NACK changes that are not gofmt'd, this
 means changes land which others cannot fix because their .gitconfig
 prevents them.


 Still on the todo list, sorry. Will be fixed asap.

 2. the bot doesn't appear to be enforcing it's lockdown rules as the
 comment it places on the PR contains the magic string that the bot
 looks for when it see's another $$ instruction.


 Curtis has already fixed this plus landed some other imporvements (see other
 emails).

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

apiserver/testing.FakeAuthoriser is not a good mock

[David Cheney]

Hello,

This message serves as background to a series of changes coming soon.

The goal of this message, and the changes that follow, is to describe
the problems that I see with the FakeAuthoriser and a describe in
general terms the steps I propose to address those shortcomings.

Background:

The FakeAuthoriser implements the apiserver/common.Authoriser
interface and is used heavily in the apiserver tests. The other
implementation of this interface is the apiserver itself,
specifically, apiserver.srvRoot.

tf. The goal of the FakeAuthoriser is to imitate apiserver.svrRoot.

I assert that this is currently not true and that the tests are
structured to expect the behaviour of the FakeAuthorizer rather than
the contract of apiserver/common.Authoriser as defined by
apiserver.srvRoot.

Issues:

This section describes the ways that the FakeAuthoriser diverges from
the behaviour of the srvRoot authoriser.

1. In srvRoot, all authorisation stems ultimately from the
state.Entity object referenced from the srvRoot instance. This is a
problem as our authorisation system is based on the _tag_ passed over
the api, not the state.Entity, although for practical purposes the tag
of the authenticated entity and the entity itself are interchangeable.

Proposal: remove Authoriser.GetAuthEntity() and replace all calls with
Authoriser.GetAuthTag(). I have a branch for this, it's  30 lines as
there are only 3 places in the apiserver where we do this and they
usually call the .Tag method on the entity they get back from
GetAuthEntity.

2. This extends from point 1, while the svrRoot derives everything
from svrRoot.entity, the FakeAuthoriser allows the caller to pass a
unique value for the tag and the entity of the authorisee. This leads
to impossible situations where the FakeAuthorizer returns nil for
AuthGetEntity but a non nil value for AuthGetTag. Other permutations
exist in our tests and are expected by the test logic.

Propsal: Once step 1 is fixed the difference between svrRoot and
FakeAuthoriser is the former starts from a state.Entity and derives a
tag from which other values are derived, the latter skips the initial
step and starts from a tag. This work falls out of the solution to
steps 1 and 3.

3. The helper methods, AuthMachineAgent(), AuthUnitAgent() on the
Authoriser interface are implemented differently in svrRoot and
FakeAuthoriser. In tests it is quite common to take the FakeAuthoriser
from the test suite, copy it and change some of these values leading
to impossible situations, ie, the tag or entity of the FakeAuthoriser
pointing to a Unit, but AuthMachineAgent() returning true.

Proposal: The simplest solution is to copy the implementation of these
helper methods from svrRoot to FakeAuthoriser. A more involved
solution would be to factor these methods out to be functions in the
apiserver/common package that take an Authorizer. This second step may
not pay for itself.

These steps resolves the majority of the discontinuity between the two
implementations and will resolve a set of blocking issues I've hit
converting the apiserver and state packages to speak tags natively.

Thanks for your time

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: apiserver/testing.FakeAuthoriser is not a good mock

[David Cheney]

On Thu, Jul 31, 2014 at 5:12 PM, William Reade
william.re...@canonical.com wrote:
 On Thu, Jul 31, 2014 at 7:30 AM, David Cheney david.che...@canonical.com
 wrote:

 Proposal: remove Authoriser.GetAuthEntity() and replace all calls with
 Authoriser.GetAuthTag(). I have a branch for this, it's  30 lines as
 there are only 3 places in the apiserver where we do this and they
 usually call the .Tag method on the entity they get back from
 GetAuthEntity.


 SGTM.

 2. This extends from point 1, while the svrRoot derives everything
 from svrRoot.entity, the FakeAuthoriser allows the caller to pass a
 unique value for the tag and the entity of the authorisee. This leads
 to impossible situations where the FakeAuthorizer returns nil for
 AuthGetEntity but a non nil value for AuthGetTag. Other permutations
 exist in our tests and are expected by the test logic.

 Propsal: Once step 1 is fixed the difference between svrRoot and
 FakeAuthoriser is the former starts from a state.Entity and derives a
 tag from which other values are derived, the latter skips the initial
 step and starts from a tag. This work falls out of the solution to
 steps 1 and 3.

 3. The helper methods, AuthMachineAgent(), AuthUnitAgent() on the
 Authoriser interface are implemented differently in svrRoot and
 FakeAuthoriser. In tests it is quite common to take the FakeAuthoriser
 from the test suite, copy it and change some of these values leading
 to impossible situations, ie, the tag or entity of the FakeAuthoriser
 pointing to a Unit, but AuthMachineAgent() returning true.

 Proposal: The simplest solution is to copy the implementation of these
 helper methods from svrRoot to FakeAuthoriser. A more involved
 solution would be to factor these methods out to be functions in the
 apiserver/common package that take an Authorizer. This second step may
 not pay for itself.


 I would strongly favour the latter solution. The benefit of having a fake
 authorizer whose behaviour diverges is that (at least in theory) it allows
 for comprehensive testing against arbitrary authorizer behaviour;
 constraining that behaviour so we're only testing realistic situations will
 be great, but I fear that doing so by copy-pasting code will only lead to
 more divergences in future. Let's make sure we're using the same logic
 across the board.

You got it boss.


 Cheers
 William

 These steps resolves the majority of the discontinuity between the two
 implementations and will resolve a set of blocking issues I've hit
 converting the apiserver and state packages to speak tags natively.

 Thanks for your time

 Dave

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

CI tests are installing Juju on the CI machine ...

[David Cheney]

Creating config file /etc/mercurial/hgrc.d/hgext.rc with new version
Setting up zip (3.0-4) ...
Setting up juju-core (1.20.1-0ubuntu1~12.04.1~juju1) ...
update-alternatives: using /usr/lib/juju-1.20.1/bin/juju to provide
/usr/bin/juju (juju) in auto mode.
Setting up rsyslog-gnutls (5.8.6-1ubuntu8.6) ...

^ it's leaking in from a dependency, this is risky to say the least.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Port ranges - restricting opening and closing ranges

[David Cheney]

Yes, absolutely.

On Tue, Aug 5, 2014 at 8:33 PM, Domas Monkus domas.mon...@canonical.com wrote:
 A follow-up question: should closing a port that was not opened previous to
 that result in an error?

 Domas


 On Fri, Jun 27, 2014 at 2:13 PM, Matthew Williams
 matthew.willi...@canonical.com wrote:

 +1 on an opened-ports hook tool, I've added it to the task list


 On Fri, Jun 27, 2014 at 9:41 AM, William Reade
 william.re...@canonical.com wrote:

 Agreed. Note, though, that we'll want to give charms a way to know what
 ports they have already opened: I think this is a case where
 look-before-you-leap maybe beats easier-ask-forgiveness-than-permission (and
 the consequent requirement that error messages be parsed...). An
 opened-ports hook tool should do the trick.


 On Thu, Jun 26, 2014 at 9:18 PM, Gustavo Niemeyer gust...@niemeyer.net
 wrote:

 +1 to Mark's point. Handling exact matches is much easier, and does
 not prevent a fancier feature later, if there's ever the need.

 On Thu, Jun 26, 2014 at 3:38 PM, Mark Ramm-Christensen (Canonical.com)
 mark.ramm-christen...@canonical.com wrote:
  My belief is that as long as the error messages are clear, and it is
  easy to
  close 8000-9000 and then open 8000-8499 and 8600-9000, we are fine.
  Of
  course it is nicer if we can do that automatically for you, but I
  don't
  see why we can't add that later, and I think there is a value in
  keeping a
  port-range as an atomic data-object either way.
 
  --Mark Ramm
 
 
  On Thu, Jun 26, 2014 at 2:11 PM, Domas Monkus
  domas.mon...@canonical.com
  wrote:
 
  Hi,
  me and Matthew Williams are working on support for port ranges in
  juju.
  There is one question that the networking model document does not
  answer
  explicitly and the simplicity (or complexity) of the implementation
  depends
  greatly on that.
 
  Should we only allow units to close exactly the same port ranges that
  they
  have opened? That is, if a unit opens the port range [8000-9000], can
  it
  later close ports [8500-8600], effectively splitting the previously
  opened
  port range in half?
 
  Domas
 
  --
  Juju-dev mailing list
  Juju-dev@lists.ubuntu.com
  Modify settings or unsubscribe at:
  https://lists.ubuntu.com/mailman/listinfo/juju-dev
 
 
 
  --
  Juju-dev mailing list
  Juju-dev@lists.ubuntu.com
  Modify settings or unsubscribe at:
  https://lists.ubuntu.com/mailman/listinfo/juju-dev
 



 --

 gustavo @ http://niemeyer.net

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: avoiding unnecessarily entering upgrade mode

[David Cheney]

SGTM.

On Wed, Aug 6, 2014 at 11:10 AM, Menno Smits menno.sm...@canonical.com wrote:
 Right now, a Juju machine agent is in upgrade mode from the moment it
 starts until the upgrade-steps worker is finished. During this period API
 logins are heavily restricted and most of the agent's workers don't start
 until upgrade mode stops.

 This happens even when there is no upgrade to perform. The upgrade-steps
 worker always runs at machine agent startup and upgrade mode is in force
 until it finishes.

 Upgrade mode is typically short-lived (say 10 seconds) but if something is
 wrong (e.g. mongo issues) the upgrade-steps worker may take longer or not
 finish resulting in the user seeing lots of upgrade in progress messages
 from the client and in the logs.
 This is particularly confusing when a user hasn't even requested an upgrade
 themselves.

 I would like to change the machine agent so that upgrade mode is only
 entered if the version in agent.conf is different from the running software
 version. This would mean that upgrade mode is only entered if there is an
 actual upgrade to perform.

 The version in agent.conf is only updated after a successful upgrade so it
 is the right thing to use to determine if upgrade mode should be entered.

 The current behaviour means that the (idempotent) upgrade steps for the
 current version are always run each time the machine agent starts. If the
 change I'm proposing is implemented this will no longer happen. Does this
 seem like a problem to anyone? For instance, do we rely on the upgrade steps
 for the current version being run after bootstrap?

 The ticket for this work is at: https://bugs.launchpad.net/bugs/1350111

 Cheers,
 Menno



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Label: Ready For Review

[David Cheney]

What if the review needs the author to rework ?

On Tue, Aug 12, 2014 at 7:55 AM, Nate Finch nate.fi...@canonical.com wrote:
 Merge it and it'll get closed and out of the list of open PRs.  I presume
 the submitter is paying enough attention to merge their own stuff.

 On Aug 11, 2014 5:44 PM, David Cheney david.che...@canonical.com wrote:

 How can we remove the label once the review has been done ?

 On Tue, Aug 12, 2014 at 4:33 AM, Nate Finch nate.fi...@canonical.com
 wrote:
  I made a label on github.com/juju/juju (and coincidentally
  github.com/juju/utils) called Ready For Review. The reason for the label
  is
  that it is often difficult to figure out what branches are actually
  ready to
  be reviewed and which ones are really WIP and therefore aren't waiting
  to be
  reviewed.  It's simple to filter by labels to see what's assigned to
  Ready
  For Review, so the on-call reviewers (or anyone else) can find stuff to
  review.
 
  I did this because some people had mentioned to me that they had
  branches
  that were waiting for reviews, but no one was reviewing them.  Pinging
  people who are online works, but it's hard to ping people who aren't
  online so I figured this was easier and gives everyone somewhere to
  go
  to find what PR's are languishing.
 
  I know we have the WIP: prefix for branches that aren't ready to be
  generally reviewed but that's opt-out, which means it's easy to
  forget
  to put that on your branch and have people think it's ready for review
  when
  it's not  which means people tend to err on the side of just not
  reviewing stuff.  The Ready For Review label is opt-in, so there's no
  doubt
  that the submitter thinks it's ready.
 
  It currently requires someone on this list to add the label (at least
  for
  github.com/juju/juju), which is somewhat unfortunate, but it's really
  only
  needed if you think your code won't get reviewed otherwise... and maybe
  just
  asking someone to add that label will encourage them to review your
  code.
 
  -Nate
 
  --
  Juju-dev mailing list
  Juju-dev@lists.ubuntu.com
  Modify settings or unsubscribe at:
  https://lists.ubuntu.com/mailman/listinfo/juju-dev
 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[David Cheney]

Ian asked me to post my thoughts here.

I am not in favour of applications rolling their own logs, I believe
that applications should not know anything about their log output,
they should just dump it all to stdout and another process should take
care of shuttling the data, logging it, culling it, whatever.

This is summarised here, http://12factor.net/logs

I think our current system with rsyslog is working fine and there is
no reason to remove it.

The problems with all-machines.log being to large is independent of
log rolling or any of these other arguments. We simply log too much.
There would be no request for log rolling it all-machines.log
contained only useful data.

Dave

On Sat, Aug 9, 2014 at 2:58 AM, Nate Finch nate.fi...@canonical.com wrote:
 So, rsyslog rotation works fine on Linux, but we can't do that on Windows.
 If we have to do something different for Windows, I'd rather just do one
 thing which is cross platform compatible for all our OSes, and not have to
 support a different configuration for each OS.  Doing it all in-application
 also insulates us from external dependencies... if some future or past
 Ubuntu series (or CentOS) has a different version of rsyslog, it could
 behave differently / require a different configuration, etc.



 On Fri, Aug 8, 2014 at 12:40 PM, David Britton david.brit...@canonical.com
 wrote:

 On Fri, Aug 08, 2014 at 12:03:21PM -0400, Nate Finch wrote:
 [...]
  remote syslog and to the local file log, we wouldn't need to worry about
  log rotation of the local log screwing up what gets sent to the remote

 Do the standard rsyslog log rotation mechanisms not function well?

 On Windows, what about the event log (which has remote
 viewing/aggregation capabilities built in)?

 --
 David Britton david.brit...@canonical.com



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju trunk now is compatible with Mongo 2.6

[David Cheney]

Will CI failures using mongodb-2.6 be considered critical and block the build ?

On Fri, Aug 15, 2014 at 10:07 AM, Ian Booth ian.bo...@canonical.com wrote:
 A little while back we started work as a background task to port Juju to work
 with Mongo 2.6. After a final push from Andrew to get everything finalised, 
 the
 current Juju trunk now has Mongo 2.6 support finished/working.

 Note that Mongo 2.4.x (as shipped in cloud archive and/or trusty) is still the
 officially supported version for Juju.

 The reason for this work was due to the fact that Mongo 2.6 offers various 
 fixes
 and improvements over 2.4; some of these fixes are for issues Juju encounters
 but which will not be fixed for 2.4
 eg https://jira.mongodb.org/browse/SERVER-11807
 Also as time goes on, we need to have a viable pathway to move to Mongo 2.6 
 (and
 later versions) since that's where most of the development focus will be. Plus
 Mongo 2.6 is already in proposed for Utopic, so we now have the option to
 package juju-mongodb with a consistent version of Mongo as ships with the 
 distro.

 What we will do now is set up regular test runs using Mongo 2.6 on CI so we'll
 have data available about that combination's reliability etc.


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: First customer pain point pull request - default-hook

[David Cheney]

Nope, still not receiving you clearly

On Mon, Aug 18, 2014 at 7:20 AM, Tim Penhey tim.pen...@canonical.com wrote:
 -BEGIN PGP MESSAGE-
 Charset: windows-1252
 Version: GnuPG v1

 hQIMA233D38ktbXXAQ/+OxwEOmqX/SS+lyJGh7WBYOZge3P8G9+bdu0u8v/sGioD
 xbPcpRAp4D3GaCyaMS231JTdvWjMgegw9f6AnnaeLiEdcmFi3YBEzvnDCR5+Lfpb
 KNMZFUsL9CYtsBy+tWBFkowsFwiKy66UiOQDRGW6G6V+5v/L2v4TUoDN+LE74/h2
 jbLuGCUX4od4UYyWDLfj5NPN0DlgYrt5kiv2NCC2QUmYqVrpuoFKl+dFkbt8WIti
 WXV6TjoFIbeRd83m3A4++f0jNlsUg9WWVnBIYVgomL/ndbBC8DDwWijAJfqh660P
 gUspAWx9/QxzoMXU/nUAfPoGpim0tRsd41FfrncgPGV1P15MBNcsLrR20QOhiwU5
 C/QO7M34VbdTySw5bK6xPX6SpVtIS7fNsqtJbZuhzpR1e9IaIERYPfKwZz20YGqa
 WBnzr80FUcFrq98BqcAGbQfgkHhMRBoS22mPCSrTkh8ki5lKkckc4UGrr3/JhJuD
 B2sTxfIQosbHbGBHhFn/gED/lGKeMMxA7lZdsMcJqgeou1/ggrbqc3vrqWsGs4mq
 9hKm2blZJEMGrhqpAEJ0cd5b6a76u5AKsr+F7G468XHWLrs1wMpyFiXURG9Hzp3N
 PmbzNomZmG8412QEEua8XDP3YT+3g54yakNr/vXisST44rbaZCCA0qhmj09jGf6F
 AQ4DdFn2rLtx3aMQA/9vSXW55V6QVufALE7cMXzYxhliBUnihnXRJRkjxUo54DY1
 HoeJwfHAoxQcvAlf5HXO/rfSe+CFKqZlP4kzgHXKHimXHCiukmqUIrEIiDCFQ2pm
 lvLJXccOya5F/wzPoaBzRWFmHoSZm790kvFM7Uy6Ko6V/5gRaKmCsdXOMBjmTAQA
 gjYVJqdifYIFRAdcH3IypC2lq1Ck1ndghv2r9T9QMlI6zZ1qJ6e/puwL6TKUxAqY
 FAqCj/rqm0qkirVPdGh0rPmfjV4PSnOLaTgbvGkul5wh3iJjddhZFyfaxxVTjRBY
 SFjKk5U3yAvym5L5XdaICWZDMOuZjKNj01VDeKTQKwXSwLABvWDovh0Sd9D/cf8a
 KHcW+CMHh89I1NyiW+6elJAU0HAnUtAHvlmoHzxLNsHtodQcdHX2sFU2UNYH+bj3
 6N4blppY0wFhIOQ5AzehdKef5fhAKQL+SZ+IMY5o5oZHxsMxCVg4K7jwhfjtA9UH
 Omf5mffBCF6dCTpg6nTIk/Fc92LNPxeX9ougfudFKMNCd9LHt2YMn9Q9CBgMEwxP
 nBAPcQhSCqtNg/QYNSWlBQS0M2d5oVvrG/YgzDWHsG3jcpKwS/S5R8QNP/U12U9W
 jKxvIV8F37ORzROO1NSeftWP0LrYagVPgZoFriXgEOP3cCEEILeKqNPtsGBvDFHO
 NgActMmYhMuND67rnuFjrqc410WEzIQ70wjw4BEitzLTvgw1fvViriXbqT8G8tfq
 wXTCRAj85GErd4BMG1Ve60PA7jLuf95XORftyLQQBIkaZksfcH2JXg4Dr1FaVQXV
 oiLWzuO5ezCh/Bhf4j7Zc1dhFg==
 =3J0G
 -END PGP MESSAGE-

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[David Cheney]

Hi Horatio,

I don't see a way to resolve the very disparate set of opinions you've
highlighted below. It's also not clear from your email who is
responsible for making a decision.

I suggest reframing the discussion as user stories. ie

* As a Juju user with a Windows workstation I want to use juju
debug-log (this should work today)
* As a Juju user who has deployed a mixed environment (to the best of
my knowledge there is no requirement for the state servers to run on
Windows, this seems contrary to Canonical's goal of Free Software)
containing a windows workload charm I want to view the logs from that
charm.

Dave

On Wed, Aug 27, 2014 at 5:35 AM, Horacio Duran
horacio.du...@canonical.com wrote:
 Hey, In an effort to move forward with juju's windows integration I have
 summarized what seems to be the core points of this discussion to the best
 of my ability (please excuse me if I missed or misunderstood something).
 The two core points of discussion on this thread are:
 * should we remove all-machines.log: which has been voted against, at least
 for the moment, since it is used for debug-log.
 * how do we support logging in windows: The strongest suggestions here are a
 syslog package by gabriel and logging into MongoDB by Gustavo.

 We do require some decision on the front of windows logging to have a
 complete windows support. Ideally we need senior citizens of juju dev
 community to weight into this in order to get a clear path to follow.

 Here is a summary I made to help myself while following this discussion:

 Nate original suggestion:
 * Remove all-machines.log: Claiming it takes a lot of space and it is not a
 multi platform solution

 Tim, John, Aaaron, etc:
 * all-machines.log is required for debug-log
 * makes it big and it would be nice to rotate it.

 Nate, gabriel:
 * keep all-machines.log
 * use a go-only solution (syslog package with ports from gabriel for
 windows)
 John
 * agrees.

 Nate, gabriel:
 * remove rsyslog from al OSes in favor of one solution that fits all OSes
 * Replace with go only solution.

 Dave:
 * Dont mind about the logs, make it just output and let external tools
 handle logging and rotation.
 * all-machines.log might be a bit bloated and it could contain less data
 that is more useful.
 (Here is the reference to 12factor that will later be attibuted to nate)
 Ian:
 * Agrees with dave, yet we should provide a rolling mechanism.

 Gabriel:
 * Windows does not support capturing stdout as a logging mechanism, it
 requires to explicitly log into the event log.
 * Thinks that using rsyslog to stream logs from agents to state server is
 too much overhead on external tools.
 * Proposes replacing external rsyslog with in app solution for the case of
 streaming logs.
 * Alternative solution, he does not recommend it, to create (and bundle with
 jujud.exe) a wrapper for windows only.

 Gustavo:
 * Present a possible alternative by using a MongoDB capped collection
 which will suit our use cases but does not recommend it because of the idea
 needs maturing on some details.

 Matt:
 * We should provide the option to log to stdout or syslog.

 Kapil:
 * Supports Gustavo's idea of logging in a structured form into Mongo as it
 makes sense to dump structured data with structure instead of serializing it
 to be de-serialized later.
 * We can send also messages to syslog and let OPS people collec them
 themselves.

 Gabriel (summarizing)
 * I will be looking into event log for local windows logging. This will
 probably require writing a package.
 * the syslog change will solve in the sort term, the aggregation issue from
 Windows nodes (when something better comes along, I will personally send a
 case of beer or ice-cream...or both, to whomever removes syslog as a
 dependency)
 * lumberjack works *now* for local logging on both Windows and Ubuntu. It
 simply removes 2 dependencies (for logging) with just a few lines of code...

 --
 Horacio


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: getting rid of all-machines.log

[David Cheney]

On Wed, Aug 27, 2014 at 3:04 PM, Ian Booth ian.bo...@canonical.com wrote:


 On 27/08/14 14:56, John Meinel wrote:
 My personal vote would be:
 a) Use something that can write directly to multiple syslog receivers over
 a TLS encrypted connection from inside the jujud binary (e.g. don't use
 rsyslog to read the log files and forward them on to the state servers, but
 just write directly)

 I may be misremembering, but at the time that was the preferred approach. But
 then someone said Go's inbuilt syslog APIs were broke, so the compromise was 
 to
 use rsyslog forwarding.

 Does anyone else recall why it may have been said that Go's syslog APIs are 
 broken?

The reconnect logic is broken in all the version's of the syslog api.
The general consensus is that package is a mistake and should not be
used.


 I'd actually like to continue keeping a log file on local disk, as it can
 help diagnose when the problem is that you are failing to connect to
 somewhere else.

 Big +1 IMHO. I've always been a fan of logging to a text file readable in vi,
 emacs, whatever. Lowest common denominator is often the only option for
 debugging. I have no issue with logging elsewhere also, but can we please keep
 the log file as well.


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: gccgo internal compiler errors

[David Cheney]

nah, we have a fix upstream, we just need to get that backported to
trusty then this becomes a non issue.

On Fri, Aug 29, 2014 at 7:44 PM, Matthew Williams
matthew.willi...@canonical.com wrote:
 As it's something we need to be doing for a while yet is there value in
 adding this as a task that gets run by the landing bot?

 Thanks

 Matty


 On Thu, Aug 28, 2014 at 11:48 PM, Tim Penhey tim.pen...@canonical.com
 wrote:

 Hi folks,

 I spent some time this morning looking at
 https://bugs.launchpad.net/juju-core/+bug/1362636

 A critical regression that was breaking CI on power.

 There is a bug in gccgo where we hit an internal compiler error when
 comparing an interface to a concrete type that implements the interface
 (as opposed to a pointer to the concrete type implementing the interface).

 This impacts some of the names.Tag rework that is going on.

 If you try to compare:
var tag names.Tag = names.NewMachineTag(1)

if names.NewUnitTag(1) == tag {
   // BOOM!!!
}

 This is entirely valid Go, and works fine with gc, but gccgo barfs
 horribly.

 My fix is here: https://github.com/juju/juju/pull/633

 This is just a warning.

 Remember folks that we need to support gccgo still (for at least another
 year until we have power and arm64 using gc).

 You can test locally by doing this:
   go test -compiler gccgo

 If you install the gccgo packages, which I don't remember, but hopefully
 someone will follow up with.

 Cheers,
 Tim

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

State should not depend on the API server

[David Cheney]

Hello,

This is an introductory email to explain my upcoming series of pull requests.

The theme is simple: state must not depend on the api server.

This is currently not true, state depends on api/params because some
of the api types have leaked into the state package and are being
directly stored in mongo.

Please speak up if you disagree with this proposal.

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: State should not depend on the API server

[David Cheney]

The goal I have been tasked with is breaking the interdependency
between the state package and the _client_ api types in
state/api/params.

I don't have any opinion on adding additional layers assuming that
their dependancies flow downwards, ie

labix.org/mgo - juju/state - juju/apiserver

On Mon, Sep 1, 2014 at 4:03 PM, John Meinel j...@arbash-meinel.com wrote:
 FWIW I'd favor 3 layers, though it does mean you have to do copying between
 structs that would likely otherwise be almost identical. A State layer for
 saving in the DB, an API layer for communication, and a Model layer for use
 in the Agents/Clients.

 John
 =:-



 On Mon, Sep 1, 2014 at 9:40 AM, David Cheney david.che...@canonical.com
 wrote:

 Hello,

 This is an introductory email to explain my upcoming series of pull
 requests.

 The theme is simple: state must not depend on the api server.

 This is currently not true, state depends on api/params because some
 of the api types have leaked into the state package and are being
 directly stored in mongo.

 Please speak up if you disagree with this proposal.

 Dave

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Logger prefixes for api/apiserver are changing

[David Cheney]

Wow. I'm sorry I broke that, it didn't even occur to me.

I wonder if there is a better way to handle this default case of the
name of the logger follows the package path), ie

var logger = loggo.Logger() // or something, it could be a new
method, logger.Default(), or something.

which will inspect the call tree and determine its logger name automagically.

On Tue, Sep 2, 2014 at 8:00 PM, Dimiter Naydenov
dimiter.nayde...@canonical.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi,

 As you may already know, this pull request just landed today, moving
 state/api/ - api/ and state/apiserver/ - apiserver/ :
 https://github.com/juju/juju/pull/655

 It didn't fix the logger prefixes in some places, which I'm fixing
 with this follow-up: https://github.com/juju/juju/pull/659

 So be aware anything in the moved to top-level packages api and
 apiserver will use juju.api.* and juju.apiserver.* logger
 prefixes from now on (existing code will be converted as soon as #659
 lands, but for new code please follow the same pattern).

 Cheers!
 - --
 Dimiter Naydenov dimiter.nayde...@canonical.com
 juju-core team
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1

 iQEcBAEBAgAGBQJUBZUhAAoJENzxV2TbLzHwewkH/AhtPOv1h3w+9LkyQdmH7Pbl
 28cucj422TOXIek65+tB4fGmeCglQTPJkaLcsImWxKhYU7oAP9iT52lKjHiV377t
 l7sSVprKWp1AfF307lT6xI8derNnoSJtEHTwbXZKc4LaQt60Zv/++yuc4xLu/zYJ
 z2rkypveSTfGrL7AqdoSsgmj1HbbO3uw9dlz17Mw96FuHsPZbOm28P+g3Y8FYjLR
 4EBkZ+qHoUGl0dUV4hQyezFiY5H3SXPhqEIpm2ImPpj5MfNHPd1w7PHlPzXVcrkC
 WxWSzpdPsNhOi3AWGT9lwLaLkMA4emt/24exHHVifKFnzOy/Ob4Dr/QnYaB8zfM=
 =Mkd/
 -END PGP SIGNATURE-

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Logger prefixes for api/apiserver are changing

[David Cheney]

I'll take thumper's temperature for this feature at the standup and
see what he thinks.

On Wed, Sep 3, 2014 at 12:30 AM, Eric Snow eric.s...@canonical.com wrote:
 On Tue, Sep 2, 2014 at 5:07 AM, David Cheney david.che...@canonical.com 
 wrote:
 I wonder if there is a better way to handle this default case of the
 name of the logger follows the package path), ie

 var logger = loggo.Logger() // or something, it could be a new
 method, logger.Default(), or something.

 which will inspect the call tree and determine its logger name automagically.

 +1

 -eric

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Copyright information in headers

[David Cheney]

I think this is needless busy work, I vote that as long as there is a
copyright header with _a_ year, that is sufficient.

On Thu, Sep 4, 2014 at 7:50 AM, Ian Booth ian.bo...@canonical.com wrote:
 Hi folks

 The question recently came up in reviews as to whether we should be updating 
 the
 date in the copyright statement in the file header when we make a change to 
 the
 code in that file. I sought clarification from Robie Basak, who previously had
 provided input on licensing issues and compliance for getting Juju included in
 trusty. Below is what he said.

 TL;DR;
 It doesn't really matter, we just need to agree on a policy. It is suggested
 though that we do update the date when we make a change. Agree?

 snip

 What's our policy for dates in copyright headers?

 // Copyright 2012, 2013 Canonical Ltd.
 // Licensed under the AGPLv3, see LICENCE file for details.

 From the point of view of acceptability for Ubuntu, it doesn't
 particularly matter, and I don't believe it'll cause any issue for us
 whatever you do here. I'll certainly be happy to upload whether or not
 you update the date.

 I'll try to explain my perspective on this, but I'm not entirely
 confident that there isn't something I'm missing for the broader
 picture, so note that I Am Not A Lawyer, etc.

 For the above, do we need to add 2014 if we modify the file this year?
 Or is the date just meant to be the year the file was first published?

 I think it's meant to be the sum of all the copyright claims on the
 file. So if you add some new code, you have a copyright claim on the new
 code in the newer year in which you made it.

 AIUI, the purpose of the date is that since copyright expires
 (theoretically, anyway), updating the date updates the copyright claim,
 which would give us more control in the (eventual) event that copyright
 expires.

 In practice, IMHO this is never going to matter since nobody is going to
 care about the copyright on a piece of software that is that old anyway.
 But I suppose laws could change, so the right thing to do would be to
 add a new year whenever you make a change in a new year on a per-file
 file basis. BTW, it's common to fold 2012, 2013, 2014 to just
 2012-2014.

 But I don't particularly care for upload purposes.



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: A beginner's adventure in Charm authoring

[David Cheney]

On Fri, Sep 5, 2014 at 5:04 AM, Nate Finch nate.fi...@canonical.com wrote:
 given that we currently use the path, you can't have one charm for multiple
 series anyway.

For deploying local: charms, symlinks work fine here.

 This would at least be better than what we have right now,
 and would be backwards compatible (older jujus would just require the old
 style local deploy and would ignore the extra series specification in the
 metadata).


 On Thu, Sep 4, 2014 at 2:58 PM, Curtis Hovey-Canonical
 cur...@canonical.com wrote:

 On Thu, Sep 4, 2014 at 3:26 AM, John Meinel j...@arbash-meinel.com
 wrote:
 ...
  At the very least we need to know what OS Series the charm is targeting.
  Which is currently only inferred from the path. I don't particularly
  like
  it, and I think the code that searches your whole repository and then
  picks
  the best one is bad, as it confuses people far more often than it is
  helpful.
  (If you have $REPO, and have $REPO/precise/charm and
  $REPO/precise/charm-backup but the 'revision' number in charm-backup is
  higher for whatever reason, juju deploy --repository=$REPO charm will
  actually deploy charm-backup)

 I thought we agreed in Burlington that the charm can declare the
 series in the metadata.yaml
 series: trusty

 A list of series was rejected I recall. There was a plan to change
 charm store ingestion to read the metadata.yaml. Maybe this fell apart
 because without support for a list of series/oses, you cannot have one
 charm supporting more than one series.

 --
 Curtis Hovey
 Canonical Cloud Development and Operations
 http://launchpad.net/~sinzui



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: ReviewBoard is now the official review tool for juju

[David Cheney]

Is there a setting to make reviewboard show the diff in a review by
the default ? For some reason for me it always requires me to hit the
'view diff' link.

On Tue, Sep 16, 2014 at 6:39 AM, Ian Booth ian.bo...@canonical.com wrote:


 On 16/09/14 00:50, Eric Snow wrote:
 On Mon, Sep 15, 2014 at 8:09 AM, Eric Snow eric.s...@canonical.com wrote:
 Yeah, those steps are a lot, though keep in mind that effectively it's
 only 2 steps more than before if you use the -p flag to rbt post and
 were already keeping your local master up to date.

 Just to be clear, here are the steps again, slightly reformatted:

 (0). Rebase relative to upstream master.
   - if origin is different than upstream, sync and push it

 Before I create a new branch, I ensure my local and origin (forked copy) 
 master
 branches are up to date. However, once the branch is created, thereafter I do
 not rebase because it has caused nothing but trouble, with lots of manual 
 effort
 required to fix things up wrt conflicts etc. And it should not be necessary -
 the code repository should be able to track the state of play such that when 
 you
 request a merge, it knows how to create the correct diff against the current
 official trunk which will be merged into.


 Step (0) is also pretty easy and I'll argue that people should be
 doing it anyway.


 Disagree :-)
 I never (or very, very rarely) had to do this with Launchpad and bzr and 
 things
 Just Worked. I don't do it now with github and pull requests. I'd like to 
 think
 we'd be able to avoid the burden moving forward also.

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: State should not depend on the API server

[David Cheney]

On Tue, Sep 16, 2014 at 7:59 PM, William Reade
william.re...@canonical.com wrote:
 On Thu, Sep 11, 2014 at 3:35 PM, Nate Finch nate.fi...@canonical.com wrote:
 I don't see how having different identical structs that are updated
 simultaneously in any way prevents any problems with compatibility.

 If we're updating those structs simultaneously, we're completely
 missing the point. Once we've defined a pure-data struct that might be
 persisted or sent over the wire we *must not change that struct* -- if
 we want to send or store different data, we need a new struct.

 Maybe I'm missing something from the proposal, but it seems like this just
 adds busywork without actually solving any problems that weren't caused by
 incorrect use of the code in the first place.

 Isn't that tautological? AFAICS, storing a charm.Meta (or a
 params.anything) directly in the DB *is* incorrect use of the code,
 but nobody realises that until it's too late: that is the problem, and
 that's what we're addressing.

For example, apiserver/params.StateServingInfo was being stored
directly in the database. Woe to the person that updates the api only
to discover that data was silently deleted from the state database :(


 Separation of logic is absolutely a good thing.  Separation of data is not
 nearly so useful.

 It's harder than you might think to separate the data from the logic
 that acts on it ;).

 Cheers
 William

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: ReviewBoard is now the official review tool for juju

[David Cheney]

On Tue, Sep 16, 2014 at 7:27 PM, roger peppe roger.pe...@canonical.com wrote:
 On 16 September 2014 09:22, Jonathan Aquilina jaquil...@eagleeyet.net wrote:
 If i am not mistaken if you have multiple commits in a branch git has
 something built in called git squash. This obviously eliminates the 5 step
 process into one merge and one push.

 I don't see that command. Are you thinking of the squash
 functionality of rebase -i?

 FWIW, I never run those five steps in sequence together.
 Usually I just get to a situation where I know that I have all tests
 passing and I'm up to date with master (for example I've done a merge
 some time ago, probably before fixing a bunch of tests).

 Then it's just:

 $ git reset upstream/master
 $ git commit -am 'my commit message'

Nice trick, so that resets the underlying branch to master, leaving
everything you have comitted or merged uncomitted ?


 which is usually quicker than running rebase -i, and much
 quicker if the rebase replay leads to conflicts.

   cheers,
 rog.

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Doing chained diffs w/ Reviewboard

[David Cheney]

+1 to that

On Thu, Sep 18, 2014 at 7:53 PM, Adam Collard
adam.coll...@canonical.com wrote:
 On 18 September 2014 10:49, John Meinel j...@arbash-meinel.com wrote:

 Has anyone succeeded in getting this to work?

 The steps I tried to do were:

  git co master
  git pull upstream master
  git co base-branch
  git diff master...  base.diff
  git co dependent-branch
  git diff master...  dependent.diff
  git merge-base master HEAD  remember-this-rev

 And then put the dependent.diff into the Diff: *, and then the
 base.diff into Parent Diff: and then 'remember-this-rev' into the Base
 Commit ID.

 I also tried putting git merge-base master base-branch as the Base
 Commit ID.


 This makes me think you're using the UI to do this.

 Let me repeat my Public Safety Announcement: Do NOT use ReviewBoard's UI for
 uploading diffs. Please for $deity's sake use rbt post.

 https://www.reviewboard.org/docs/rbtools/0.6/rbt/commands/post/#distributed-version-control-systems

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Doing chained diffs w/ Reviewboard

[David Cheney]

Also, be watchful for the other reviewboard footgun, paged diffs.

Reviewboard pages large reviews, so if you're used to thinking 'phew,
i've gotten to the end of the page, i'm done, check again, there
maybe a surprise waiting for you at the bottom of the page.

On Thu, Sep 18, 2014 at 9:03 PM, David Cheney
david.che...@canonical.com wrote:
 +1 to that

 On Thu, Sep 18, 2014 at 7:53 PM, Adam Collard
 adam.coll...@canonical.com wrote:
 On 18 September 2014 10:49, John Meinel j...@arbash-meinel.com wrote:

 Has anyone succeeded in getting this to work?

 The steps I tried to do were:

  git co master
  git pull upstream master
  git co base-branch
  git diff master...  base.diff
  git co dependent-branch
  git diff master...  dependent.diff
  git merge-base master HEAD  remember-this-rev

 And then put the dependent.diff into the Diff: *, and then the
 base.diff into Parent Diff: and then 'remember-this-rev' into the Base
 Commit ID.

 I also tried putting git merge-base master base-branch as the Base
 Commit ID.


 This makes me think you're using the UI to do this.

 Let me repeat my Public Safety Announcement: Do NOT use ReviewBoard's UI for
 uploading diffs. Please for $deity's sake use rbt post.

 https://www.reviewboard.org/docs/rbtools/0.6/rbt/commands/post/#distributed-version-control-systems

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Is ReviewBoard a good thing?

[David Cheney]

There were three problem reviewboard was supposed to address, review
comments are sent immediately, no side by side diffs, and no way to to
chained proposals.

I think that over the last few months we've all learnt to live with
the first issue

On the second, github now does nice side by side diffs.

On the third, it appears reviewboard leaves you solidly to your own
devices to implement chained proposals.

I'm with Jesse, I vote to stop using reviewboard, I don't think it's
paying it's way.


On Fri, Sep 19, 2014 at 10:19 AM, Jesse Meek jesse.m...@canonical.com wrote:
 We moved to GitHub in the hope of lowering the bar to contributors outside
 the team. GitHub is *the* platform and process for open source software.
 This was the logic behind the move. It was deemed to have the most mindshare
 and we sacrificed our prefered platform and process to be part of that
 mindshare.

 We are now leaving that 'main stream' process to something that suits the
 tastes of our team - ReviewBoard. This adds friction for new contributors
 (friction everyone has experienced this week). If we value our preferred
 methods of reviewing over keeping to a well known process for outside
 contributors, the best process was launchpad + rietveld. Shouldn't we simply
 return to that.

 Considering we have been successfully using GitHub for several months now,
 using reviewboard is not a necessity. Obviously, I will go with whatever the
 team decides, but I'm concerned that we have moved to reviewboard without
 considering that it undermines (as far as I can see) our primary reason for
 using GitHub.

 Jess

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Is ReviewBoard a good thing?

[David Cheney]

On Sat, Sep 20, 2014 at 1:02 AM, Eric Snow eric.s...@canonical.com wrote:
 On Thu, Sep 18, 2014 at 6:32 PM, David Cheney
 david.che...@canonical.com wrote:
 There were three problem reviewboard was supposed to address, review
 comments are sent immediately, no side by side diffs, and no way to to
 chained proposals.

 It will be worth being extra clear on ReviewBoard's pros and cons.
 I'll open a new thread.


 I think that over the last few months we've all learnt to live with
 the first issue

 I for one haven't.  On several occasions I have made a comment that I
 later realized was not valid, but am not able to simply remove since
 it already sent.


 On the second, github now does nice side by side diffs.

 Agreed.


 On the third, it appears reviewboard leaves you solidly to your own
 devices to implement chained proposals.

 rbt supports it just fine and reviewboard itself supports it (see the
 Depends on field on every review request).  What support did you
 have in mind?

Ian and Tim want whatever they had with launchpad. I never used lp
like that so I never felt the loss; I just propose one branch at a
time and have others waiting in the wings that I can propose as soon
as the predecessor lands.


 -eric

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: psa pls don't use transactions for single doc unobserved atomic updates

[David Cheney]

Hi Kapil,

I'm sure that the 'put everything in a transaction' pattern is applied
unilaterally.

Do you have some specific operations in mind ?

Dave

On Mon, Sep 22, 2014 at 7:46 AM, Kapil Thangavelu
kapil.thangav...@canonical.com wrote:
 Its sort of misses the point on why we're doing client side transactions.
 Mongodb has builtin atomic operations on an individual document. We use
 client side txns (multiple order of magnitude slower) for multi-document
 txns *and/or* things we want to observe for watches.

 -k


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Versioned packages and testing dependencies

[David Cheney]

John, Roger,

This situation right here is why I am strongly opposed to the gopkg.in
versioning model. I recommend reverting to the original launchpad
location.

Dave

On Mon, Sep 22, 2014 at 9:56 PM, roger peppe roger.pe...@canonical.com wrote:
 That's an interesting observation and I think I agree.

 The general rule is probably something like this:

 - If a type is part of the exported API of a versioned package and the
 package is changed to import that type from somewhere
 else, the package's version must be incremented.

 Given that versions usually apply to repositories, not packages,
 that does unfortunately mean that all the repositories which
 export testing functionality using gocheck need to have
 their versions incremented.

 I don't think it all has to happen at the same time, because APIs
 remain stable. It does mean that juju-core can only be updated
 after all of its dependencies have been updated though.

   cheers,
 rog.

 On 22 September 2014 12:14, John Meinel j...@arbash-meinel.com wrote:
 So Bogdan Teleaga was kind enough to put in the effort to move all of our
 source trees to start importing from gopkg.in/check.v1 rather than
 depending on labix.org/gocheck.

 However, this means that if we land those changes, code that depends on the
 testing infrastructure provided by those packages will break because a
 labix.org/gocheck.Suite is not a gopkg.in/check.v1.Suite.
 And even further, Checkers are different (so we have to update
 github.com/juju/testing/checkers/ before other code.)

 Since we're doing versioned stable APIs I'm wondering if this means things
 need a version bump. I *think* the answer is yes, because if you just did:
   go get gopkg.in/juju/charm.v3

 Before the change, you could run cd github.com/juju/juju; go test ./...
 but if we change just charm.v3 but not juju/juju then go test ./... just
 breaks.
 Thus, changing your testing infrastructure while it doesn't change your API
 it changes your *test interface* which makes it an unstable API break.

 And we have the unfortunate process that *all* of our code is tested via
 'gocheck' which means that we have to update-the-world to rev its import.

 Certainly I'd rather not do the work if we don't have to, so I'd like other
 people's input if we feel we have to.

 John
 =:-


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Is there a setting in ReviewBoard to let me see *both* the comments and the diffs

[David Cheney]

Hi

Reviewboard is driving me nuts. Either it can't do this, or I have put
it in some mode where it does this, but it can only show me the review
comments *OR* the diff, but not both.

Is there a setting that shows both diffs and the stream of review
comments _on_the_same_screen ?

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: how to update dependencies.tsv

[David Cheney]

Does your $GOPATH include more than one entry ?

The make target should probably just expect godeps to be in your $PATH.

On Wed, Oct 29, 2014 at 6:37 PM, Dimiter Naydenov 
dimiter.nayde...@canonical.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 +1, in fact there is already a godeps target in the Makefile, but I
 never managed to get it working. It's looking for $(GOPATH)/bin/godeps
 binary and I have it right there on my machine, but make still claims
  File `godeps' does not exist. (when running $ make -d godeps).

 Somebody with better make-fu knowledge should probably give a hand
 with this :)

 On 29.10.2014 05:58, John Meinel wrote:
  can we please just have make dependencies.tsv do the right thing
  so we don't have to remember which set of flags and env vars we
  need to use this time?
 
  I'm also not 100% sure that we'll have even downloaded all the
  windows dependencies if they are a strict superset given that you
  are running go get on a Linux machine to start with.
 
  John =:-
 
 
  On Tue, Oct 28, 2014 at 11:41 PM, Nate Finch
  nate.fi...@canonical.com mailto:nate.fi...@canonical.com
  wrote:
 
  We have a few windows dependencies that are not (and in some cases
  can not be) imported for the linux build.  Luckily, the windows
  dependencies are a strict super set of the linux dependencies, so
  we can still use godeps to create dependencies.tsv, just by
  setting GOOS first, thusly (from the root of github.com/juju/juju
  http://github.com/juju/juju):
 
  GOOS=windows godeps -t ./...  dependencies.tsv
 
  Please use this command line when updating dependencies.tsv so
  that is keeps a standard format which will minimize conflicts in
  the file.
 
  -Nate
 
  -- Juju-dev mailing list Juju-dev@lists.ubuntu.com
  mailto:Juju-dev@lists.ubuntu.com Modify settings or unsubscribe
  at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
 
 
 
 


 - --
 Dimiter Naydenov dimiter.nayde...@canonical.com
 juju-core team
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1

 iQEcBAEBAgAGBQJUUJlIAAoJENzxV2TbLzHwND0IAMS3c79LeWwTDnf6sEyMwTcm
 HdvBxaxEeJLNqv8Sq+mK4jWyMAtmoQ/9Azup7g9DZmb1aJXD94n25nf19NNdBkxi
 wzLBMqq3iEkKY+09js8r5S4ScRMOErBrhL7wd4+PDFTL1y5SeDAacBDCzZQKeQlY
 IRkIfOXjfSpQuALONZNUlToMAuTQzC80gf914QX7DOpKMFhsiS6cMIb2nYlAScEZ
 UFjJwRrTEgjiGoouq/KnmahhYK4i4dsdQDy9a44lTgIb+igoAi6OBzTJ8ktqac+G
 +e+sS25SHJ+T2R+iPtlJnExW3raa9vHxIJQBazB8C/QjLDqrOtzt1XSe5WxlQxs=
 =akp+
 -END PGP SIGNATURE-

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Please stop adding dependencies to the state package

[David Cheney]

This is https://bugs.launchpad.net/juju-core/+bug/1392520

I have assigned it to myself.

On Fri, Nov 14, 2014 at 10:33 AM, Eric Snow eric.s...@canonical.com wrote:
 On Thu, Nov 13, 2014 at 4:05 PM, David Cheney
 david.che...@canonical.com wrote:
 Hello,

 Please stop adding dependencies to the state package.

 The dependency tree for state must go

 mgo - state - things that depend on state

 At the moment state depends on things like backups, multiwatchers,
 presence, toolstorage.

 This is wrong.

 Eric, i'm not picking on you here, it's just that b sorts first, and
 this is the one I saw today. There are many other examples, including
 the multiwatcher which also break this model

 If you add functionality to state, then the code has to go ***INSIDE**
 state, or be written to take a *state.State.

 Please help me out by not adding more dependencies to state.

 Thanks for pointing this out, Dave.  I'll fix this for backups.

 -eric

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Too space, or not two space

[David Cheney]

To quote Butterick's
http://practicaltypography.com/one-space-between-sentences.html

 I think two spaces look bet­ter so that’s what I’m go­ing to use.

I’m telling you the rule. If you want to put per­sonal taste ahead of
the rule, I can’t stop you. But per­sonal taste doesn’t make the rule
evap­o­rate. It’s like say­ing “I don’t like how sub­junc­tive-mood
verbs sound, so I’m never go­ing to use them.”

On Wed, Nov 19, 2014 at 11:10 AM, Jesse Meek jesse.m...@canonical.com wrote:
 I'm coming across double spaces after a full stop in comments. Some consider
 this an error, others an intentional style that makes the comment cleaner to
 read.

 Yes, it's a minor point but in the name of consistency and avoiding future
 review ping pong on the issue, I'm calling for a vote:

 Should we have a single space after full stops in comments?
 http://xkcd.com/1285/

 Please reply +1 for yes to one space, -1 for no to one space. I'll tally up
 votes on Friday and update our style guide accordingly.

 My vote: +1

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Too space, or not two space

[David Cheney]

 I think you guys have more things to worry about than the spacing style for
 comments.

I'm happy to let this discussion die at this point.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: git clients vulnerability found

[David Cheney]

Fortunately all of us run an operating system with a case sensitive
file system, right ?

On Fri, Dec 19, 2014 at 8:32 AM, Horacio Duran
horacio.du...@canonical.com wrote:
 Heads up people
 https://github.com/blog/1938-vulnerability-announced-update-your-git-clients
 apparently there is a vulnerability in git client that might be exploited
 via pushing certain snippets to the repos.

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: separating out golang juju client from

[David Cheney]

There is no reason for the 130 (at last count) packages that
constitute juju-core (not counting the dozens of other packages we
bring in as dependencies) to live in the same repository.

If licensing is the lever that we use to break up this monolithic
repository, consider me +1

On Fri, Dec 19, 2014 at 11:05 PM, Kapil Thangavelu
kapil.thangav...@canonical.com wrote:


 On Fri, Dec 19, 2014 at 7:02 AM, Nate Finch nate.fi...@canonical.com
 wrote:

 While I am generally for using more permissive licenses, I'm not sure how
 useful that might be... most significant changes require modifications to
 both the client and the server, or at least to libraries used by both.


 That sort of misses the point of building apps that use juju apis. Yes the
 two packages need to be updated together for new changes same as today.


 There's not that much code under cmd/juju compared to the whole rest of
 the repo.


 Again its not about that code, its about building other applications and
 facilitating integrations.


 cheers,
 Kapil


 On Fri, Dec 19, 2014 at 6:03 AM, Kapil Thangavelu
 kapil.thangav...@canonical.com wrote:

 one of the issues with having it in tree, means client usage falls under
 the AGPL. We want to have the client used widely under a more permissive
 license. I've already had contributions to other projects n'acked due to
 license on our libraries. I'd like to see it moved to a separate repo so
 that's possible. Thoughts?

 cheers,
 Kapil



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

What are blocks ?

[David Cheney]

Hello,

Can someone direct me to the design documents for the enigmatic
'block' functionality which is showing up in the review queue.

Thanks

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Testing on windows

[David Cheney]

It is clear that we need blocking CI jobs for Windows and CentOS --
has this been added to the Nuremberg agenda ?

On Thu, Mar 19, 2015 at 10:24 PM, Gabriel Samfira
gsamf...@cloudbasesolutions.com wrote:
 On 19.03.2015 12:07, Michael Foord wrote:

 Do all tests pass (or skip) on Windows now? Do we have a CI job
 running them?
 There is a PR under review, and one other fix pending that will make all
 tests pass in juju core again. There is no job yet in the CI that blocks
 master if tests fail on Windows. We should be able to turn it on once:
 http://reviews.vapour.ws/r/1119/ merges and one other fix lands in
 github.com/juju/testing.

 Until then, it is still useful to test on Windows before pushing, to
 limit the risk of breaking anything else. Its also good practice, as
 windows does offer some differences from Linux, and we need to be aware
 of them when writing code. We also need to get accustomed to writing new
 features with multiple platforms in mind. It would really be bad to
 create a bigger gap between the platforms then already exists (playing
 catch up is never a good thing).

 A set of branches for CentOS support will soon be proposed, and soon
 after that Debian (Jessie), so testing on windows should be good
 practice for when that happens :).

 Cheers,
 Gabriel





 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Proposal: move package replicaset to its own GitHub repo

[David Cheney]

Do it!

On Wed, Mar 25, 2015 at 9:22 PM, Nate Finch nate.fi...@canonical.com wrote:
 +100, thanks for doing this, Andrew!

 On Wed, Mar 25, 2015 at 6:16 AM, roger peppe roger.pe...@canonical.com
 wrote:

 This seems like a grand plan to me.

 On 25 March 2015 at 08:12, Andrew Wilkins andrew.wilk...@canonical.com
 wrote:
  I'd like to move the replicaset package to github.com/juju/replicaset.
  For
  rationale and preparatory changes, see: http://reviews.vapour.ws/r/1260/
 
  Please debate here. Silence == tacit approval.
 
  Thanks,
  Andrew
 
  --
  Juju-dev mailing list
  Juju-dev@lists.ubuntu.com
  Modify settings or unsubscribe at:
  https://lists.ubuntu.com/mailman/listinfo/juju-dev
 

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Public Service Annoucement

[David Cheney]

Why doesn't the bot run this hook ?

On Fri, Apr 24, 2015 at 9:54 PM, Matthew Williams
matthew.willi...@canonical.com wrote:
 There's a very useful pre-push hook available for juju that does useful
 things like run gofmt, go vet, checks we can build etc.

 If you don't have it setup there's some lovely instructions in
 Contributing.md:

 https://github.com/juju/juju/blob/master/CONTRIBUTING.md#local-clone

 Thanks for listening

 Matty

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Public Service Annoucement

[David Cheney]

Sorry. I was not clear.

Why isn't the bot running the pre-commit hook as part of the CI test ?

What we have now is a warning, gleefully ignored.

What we need is an error, which rejects the PR if it doesn't pass the
pre-commit hook.

On Fri, Apr 24, 2015 at 10:28 PM, John Meinel j...@arbash-meinel.com wrote:
 The bot cannot make changes to the tree and commit it back to trunk, because
 it doesn't have direct push rights to github. Instead it has API rights to
 accept this merge. So even if it did run the changes, it wouldn't end up
 in trunk, and we could only reject the merge if it changed things. (As I
 understand it)

 John
 =:-

 On Fri, Apr 24, 2015 at 3:56 PM, David Cheney david.che...@canonical.com
 wrote:

 Why doesn't the bot run this hook ?

 On Fri, Apr 24, 2015 at 9:54 PM, Matthew Williams
 matthew.willi...@canonical.com wrote:
  There's a very useful pre-push hook available for juju that does useful
  things like run gofmt, go vet, checks we can build etc.
 
  If you don't have it setup there's some lovely instructions in
  Contributing.md:
 
  https://github.com/juju/juju/blob/master/CONTRIBUTING.md#local-clone
 
  Thanks for listening
 
  Matty
 
  --
  Juju-dev mailing list
  Juju-dev@lists.ubuntu.com
  Modify settings or unsubscribe at:
  https://lists.ubuntu.com/mailman/listinfo/juju-dev
 

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

linux/arm64 benchmark improvements

[David Cheney]

Hello,

Here are the results of the work Aram and I did last week on improving
the performance of our arm64 port of Go. This is comparing the build
from April 06 (before we started) to today (April 16th).

http://paste.ubuntu.com/10831430/

Cheers

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Please run your tests with -race

[David Cheney]

Hello,

TL;DR - juju has lots of problems with data races, please test your
code with the -race flag to ensure it doesn't get worse while we try
to fix the problem.

Longer version:

In debugging https://bugs.launchpad.net/bugs/1456398 I found that
there are multiple data races in the Juju code base. It's been long
suspected that the test's are racy, PatchValue is super easy to
introduce a race if all the workers started by a test have not exited
before the suite's tear down function runs.

However, more serious races have been discovered, such as
https://bugs.launchpad.net/bugs/1456857 which affects code all the way
back to 1.22.

Why is a data race bad ?

Ok, so you're looking at https://bugs.launchpad.net/bugs/1456857 and
you're thinking, so maybe the tls code accidentally uses the wrong
certificate for a little bit, how bad is that?

The problem is data races affect the integrity of the structures that
the garbage collector uses. In the example above replacing the
certificate means one CPU can see the new value, and another
potentially the old value. When it comes to to run the gc, depending
on which CPU walks that chain of pointers it may think that the old
certificate is still live, or the new certificate is unreachable --
and boom that memory is marked as free and the certificate corrupted.

The short version is: there are no safe data races, and Juju is not
reliable until they have all been fixed.

How to run the race detector ?

The race detector comes with Go and is available by adding the -race
flag to invocations of go test, so what was

go test github.com/juju/juju/...

becomes

go test -race github.com/juju/juju/...

The downside of this is the race detector has significant overhead, at
least 2x, so tests will be even slower.

Thanks

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Please run your tests with -race

[David Cheney]

Here is a report of the current races. http://paste.ubuntu.com/11256308/

On Thu, May 21, 2015 at 11:10 AM, Tim Penhey tim.pen...@canonical.com wrote:
 Thanks Dave for this great write up.

 In order to drive the data races to zero, and make a soon to be added CI
 test as voting, I propose that as folks are writing tests in packages,
 races in at least that file, an possibly that package should be fixed.

 Tim

 On 21/05/15 12:12, David Cheney wrote:
 Hello,

 TL;DR - juju has lots of problems with data races, please test your
 code with the -race flag to ensure it doesn't get worse while we try
 to fix the problem.

 Longer version:

 In debugging https://bugs.launchpad.net/bugs/1456398 I found that
 there are multiple data races in the Juju code base. It's been long
 suspected that the test's are racy, PatchValue is super easy to
 introduce a race if all the workers started by a test have not exited
 before the suite's tear down function runs.

 However, more serious races have been discovered, such as
 https://bugs.launchpad.net/bugs/1456857 which affects code all the way
 back to 1.22.

 Why is a data race bad ?

 Ok, so you're looking at https://bugs.launchpad.net/bugs/1456857 and
 you're thinking, so maybe the tls code accidentally uses the wrong
 certificate for a little bit, how bad is that?

 The problem is data races affect the integrity of the structures that
 the garbage collector uses. In the example above replacing the
 certificate means one CPU can see the new value, and another
 potentially the old value. When it comes to to run the gc, depending
 on which CPU walks that chain of pointers it may think that the old
 certificate is still live, or the new certificate is unreachable --
 and boom that memory is marked as free and the certificate corrupted.

 The short version is: there are no safe data races, and Juju is not
 reliable until they have all been fixed.

 How to run the race detector ?

 The race detector comes with Go and is available by adding the -race
 flag to invocations of go test, so what was

 go test github.com/juju/juju/...

 becomes

 go test -race github.com/juju/juju/...

 The downside of this is the race detector has significant overhead, at
 least 2x, so tests will be even slower.

 Thanks

 Dave



-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: I'm concerned

[David Cheney]

This should be achievable. go test sends SIGQUIT on timeout, we can
setup a SIGQUIT handler in the topmost suite (or import it as a side
effect package), do whatever cleanup is needed, then os.Exit, unhandle
the signal and try to send SIGQUIT to ourselves, or just panic.

On Wed, Jun 17, 2015 at 3:25 PM, Tim Penhey tim.pen...@canonical.com wrote:
 Hey team,

 I am getting more and more concerned about the length of time that
 master has been cursed.

 It seems that sometime recently we have introduced serious instability
 in cmd/jujud/agent, and it is often getting wedged and killed by the
 test timeout.

 I have spent some time looking, but I have not yet found a definitive
 cause.  At least some of the time the agent is failing to stop and is
 deadlocked.

 This is an intermittent failure, but intermittent enough that often at
 least one of the unit test runs fails with this problem cursing the
 entire run.

 One think I have considered to aid in the debugging is to add some code
 to the juju base suites somewhere (or in testing) that adds a goroutine
 that will dump the gocheck log just before the test gets killed due to
 timeout - perhaps a minute before. Not sure if we have access to the
 timeout or not, but we can at least make a sensible guess.

 This would give us at least some logging to work through on these
 situations where the test is getting killed due to running too long.

 If no one looks at this and fixes it overnight, I'll start poking it
 with a long stick tomorrow.

 Cheers,
 Tim

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Pointer Receivers on Error() and String() methods

[David Cheney]

Yup, I agree with Rog. This is fixing the problem in the wrong place.

On Mon, Aug 24, 2015 at 5:14 PM, roger peppe roger.pe...@canonical.com wrote:
 I'm afraid I'm not convinced. Declaring the Error method on the
 pointer receiver is idiomatic (just grep for ' Error\(' in the Go source)
 and is a useful indicator that the error value is always intended
 to be a pointer.

 There's a much simpler fix for this: let the errors package
 recover from this itself. We can just make Err.Error call fmt.Sprint
 to get the error message (a one line change)

 Then a wrapped nil error will print nil just like normal nil
 errors.


 On 20 August 2015 at 03:45, Nate Finch nate.fi...@canonical.com wrote:
 tl;dr:  Don't.  Use a value receiver.  99% of the time you can just delete
 the * on the receiver and it'll still work.  If you really must use a
 pointer, please handle the case where you're called with a nil receiver.

 I spent most of today trying to understand why my new hook command was
 producing this output:

 error: %!v(PANIC=runtime error: invalid memory address or nil pointer
 dereference)

 It took me a while to figure out that this is what fmt.Printf(error: %v\n,
 err) outputs when err's Error() method panics.  If you're using %s or %v to
 print a value (or use Println which implicitly uses %v), then fmt will look
 for a String() method or Error() method on the value to call, and use the
 output of that for the value's string output.  If that method panics, fmt
 prints the panic in the way you see above (everything after the PANIC=).

 Of course, the problem here is that there's no type being written, and since
 it was an error interface, it could almost anything.  Using %#v skips
 calling the Error/String methods and prints out the values in a go format,
 which told me this was a juju/errors.Err value, wrapping an API params Error
 value which was a nil pointer.  When we call Error() on an errors.Err, we
 call Error() on the cause explicitly, which was panicking.

 Here's a minimal reproduction http://play.golang.org/p/ncNVrza-hn (you'll
 have to copy it to a local file and go run it, since the playground won't
 run code external to the stdlib).

 So what's sort of interesting is that printing the error before it gets
 Traced works fine, but after the trace it is not fine.  The errors.Err's
 Error() function looks like it's explicitly calling the Error() method on
 the wrapped Cause error, which is probably the problem.  fmt.Printf must use
 some reflection magic to avoid doing that.

 Now, the root cause of this particular bug is actually my own mistake.  Line
 21 should check if orig is nil and then assign nil explicitly to err if it
 is.  Then errors.Trace would be able to tell that the error is nil and would
 just return nil itself, instead of thinking it's a valid error and wrapping
 it.

 However, you can sidestep this entirely by doing one of two things:  either
 just make the Error() (or String()) method use a value receiver.. in which
 case this code would produce this output:

 %!v(PANIC=value method main.MyError.Error called using nil *MyError pointer)

 (you can try it with the repro code I linked to)

 This printout is a lot more helpful and useful and obvious than the other
 nil pointer printout.

 OR

 Just handle a nil receiver:

 func (e *MyError) Error() string {
 if e == nil {
 return nil MyError
 }
 return e.Message
 }

 (note that it is dereferencing the pointer to e to access the Message field
 which causes the panic. Calling a method on a nil pointer is totally fine
 and will not panic if the code inside does not try to derefence the pointer
 to get to a field).

 Grepping through our code, I see a lot of pointer receivers on Error and
 String methods (45 and 77 respectively).  I think we should at least change
 all of these to be value methods (unless that's not possible.   That's a
 trivial change, and gives a much more useful printout when the pointer is
 nil.

 -Nate

 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev


 --
 Juju-dev mailing list
 Juju-dev@lists.ubuntu.com
 Modify settings or unsubscribe at: 
 https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: ppc64le timeouts cursing things

[David Cheney]

On Fri, Jul 17, 2015 at 5:08 AM, Dimiter Naydenov
dimiter.nayde...@canonical.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 17.07.2015 12:07, James Tunnicliffe wrote:
 /me opens can of worms
 Thanks for starting the discussion :)


 Having spent perhaps too long trying to parallelise the running of
 the unit test suite over multiple machines using various bat guano
 crazy ideas, I know too much about this but haven't got an easy
 fix. I do know the right fix is to re-write the very long tests
 that we have.

 If you want to find long running tests, go test -v ./... -check.v
 is the command to run at top level. You will get a lot of output,
 but it isn't difficult to find tests that take longer than 10
 seconds with grep and I am sure I could dig the script out that I
 wrote that examines the output and tells you all tests over a
 certain runtime.

 When you run go test ./... at the top of juju/juju it runs suites
 in parallel. If you have multiple long tests in a suite then it has
 a significant impact on the total runtime. We have no way with the
 current tools to exclude single tests without modifying the tests
 themselves;

 How about GOMAXPROCS=1 go test ./... ? Won't that force the runtime to
 run all suites sequentially?

When you glob packages, ./..., each _package_ is run in parallel, up
to the number of cores on the machine. That is, each package is
compiled in test scope, producing a $PKG.test and that is run in
parallel. Inside each package suites are run in sequence,
alphabetically.

You can adjust the parallelism of the former with the -P flag --
GOMAXPROCS is the wrong hammer for this job.


 if we did we could run all the tests that take less than a few
 seconds by maintaining a list of long tests, and run those long
 tests as a separate, parallel task. The real fix is to put some
 effort into making the long running tests more unit test and less
 full stack test. 30+ seconds is not what we want. The least worst
 idea I have is making a sub-suite for tests that take  10 seconds,
 one test per suite, so the standard tools will run them in parallel
 with everything else. Providing you have many CPUs there is a
 reasonable chance this will help. It is not remotely nice though.

 Using go tool pprof can also help figuring out why certain tests take
 a long time and/or memory. I'm planning to experiment with it and come
 up with some feedback.


 0 ✓ dooferlad@homework2
 ~/dev/go/src/github.com/juju/juju/worker/uniter $ go test -check.v

 Shorter tests deleted from this list. The longest are: PASS:
 uniter_test.go:1508: UniterSuite.TestActionEvents 39.711s PASS:
 uniter_test.go:1114: UniterSuite.TestUniterRelations 16.276s PASS:
 uniter_test.go:970: UniterSuite.TestUniterUpgradeGitConflicts
 11.354s

 These are a worth a look: PASS: uniter_test.go:2053:
 UniterSuite.TestLeadership 5.146s PASS: util_unix_test.go:103:
 UniterSuite.TestRunCommand 6.946s PASS: uniter_test.go:2104:
 UniterSuite.TestStorage 4.593s PASS: uniter_test.go:1367:
 UniterSuite.TestUniterCollectMetrics 4.102s PASS:
 uniter_test.go:774: UniterSuite.TestUniterDeployerConversion
 6.904s PASS: uniter_test.go:427:
 UniterSuite.TestUniterDyingReaction 5.772s PASS:
 uniter_test.go:393: UniterSuite.TestUniterHookSynchronisation
 4.546s PASS: uniter_test.go:1274:
 UniterSuite.TestUniterRelationErrors 4.536s PASS:
 uniter_test.go:476: UniterSuite.TestUniterSteadyStateUpgrade
 6.405s PASS: uniter_test.go:895:
 UniterSuite.TestUniterUpgradeConflicts 6.430s

 ok   github.com/juju/juju/worker/uniter 175.014s

 James

 On 17 July 2015 at 04:59, Tim Penhey tim.pen...@canonical.com
 wrote:
 Hi Curtis,

 I have been looking at some of the recent cursings from ppc64le,
 and the last two included timeouts for the worker/uniter tests.

 On my machine, amd64, i7, 16 gig ram, I get the following:

 $ time go test 2015-07-17 03:53:03 WARNING juju.worker.uniter
 upgrade123.go:26 no uniter state file found for unit
 unit-mysql-0, skipping uniter upgrade step OK: 51 passed PASS ok
 github.com/juju/juju/worker/uniter  433.256s

 real7m24.270s user3m18.647s sys 1m2.472s

 Now lets ignore the the logging output that someone should fix,
 we can see how long it takes here. Given that gccgo on power is
 slower, we are going to do two things:

 1) increase the timeouts for the uniter

 2) change the uniter tests

 WRT to point 2, most of the uniter tests are actually fully
 functional end to end tests, and should not be run every time we
 land code.

 They should be moved into the featuretest package.

 Thanks, Tim

 -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify
 settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju-dev



 - --
 Dimiter Naydenov dimiter.nayde...@canonical.com
 Juju Core Sapphire team http://juju.ubuntu.com
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2.0.22 (GNU/Linux)

 iQEcBAEBAgAGBQJVqPAiAAoJENzxV2TbLzHwIHYIAKLXI2F4V/Jp+3rFLqbOCrgx
 

version.Current changes

[David Cheney]

Hello,

This is an informational message to explain the recent changes to
version.Current.

version.Current used to be a version.Binary value consisting of the
version number, an architecture and a series value.

As of this afternoon, version.Current is now a version.Number value;
series and arch have been removed.

Why was this done ?

Over the years version.Current has been a fantastically useful set of
values to have around, and as such has been called into service for a
wide range of uses, many of which were unrelated to its original
purpose of tracking the current version of the juju binary.

To counteract this, the uses of version.Current have been teased apart
and in many places replaced with more appropriate abstractions.
version.Current now exists to track the version number of this version
of Juju -- that's it.

Where has version.Current.Number gone ?

version.Current _is_ the version.Number now.

Where has version.Current.Arch gone ?

The arch of the machine the code is running on is available from
github.com/juju/utils/arch.HostArch()

You can patch it with

s.PatchValue(, func() string { "new arch" / arch.Arm64
or other constant })

Please be mindful to only use arch.HostArch when you want the
architecture of the machine you are running on. In many cases, if you
want the architecture of the processor you are running on
runtime.GOARCH may be more appropriate.

Where has version.Current.Series gone ?

The series of the machine the code is running on is available from
github.com/juju/utils/series.HostSeries()

You can patch it with

s.PatchValue(, func() string { "new series" })

Please be mindful to only use series.HostSeries() when you want the
_series_ that matches the machine you are running on. In many cases,
if you want the operating system, runtime.GOOS may be more
appropriate.

Thanks

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Cheryl graduating

[David Cheney]

Congratulations Cheryl!

On Tue, Sep 1, 2015 at 11:46 AM, Tim Penhey  wrote:
> Hi folks,
>
> I have been very happy with the quality of Cheryl's reviews and the time
> is right for her to be considered a fully graduated reviewer.
>
> Thanks for all the reviews Cheryl.
>
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Upgrading minimum Go version?

[David Cheney]

Martin: Can you setup a unit test job that uses Go 1.5 ? I will apply
the same process I did for the -race job, to get the build voting,
then reenable the tests that were skipped.

On Fri, Nov 27, 2015 at 12:24 AM, Martin Packman
 wrote:
> On 26/11/2015, Andrew Wilkins  wrote:
>> Hi (mostly Curtis),
>>
>> Is there a plan to bump the minimum Go version? Some of our dependencies do
>> not build with Go 1.2. The LXD provider only builds with Go 1.3 (I think?),
>> and I've got a PR up that updates the azure-sdk-for-go dependency, but it's
>> blocked because the newer doesn't build with Go 1.2.
>
> There are two issues with requiring a newer go version:
>
> * The test suite needs to actually pass reliably with go 1.5
> * How we get a version of juju that doesn't build with go 1.2 into
> trusty needs resolving
>
> With putting the newest juju into older Ubuntu releases, our strategy
> so far has been to SRU back the same source as we put in the current
> release, which we have a special allowance for. We cannot SRU a newer
> go toolchain over the existing working 1.2 version, for hopefully
> obvious reasons. Someone else may have better ideas, but I think that
> leaves us basically three options:
>
> * Stop putting new jujus in older ubuntus
> * Use backports rather than -updates (this is what lxd does I believe)
> * SRU a newer go toolchain to some alternative, juju only location
>
> All of that is pretty big process changes that we can't just go ahead and do.
>
> Test failures have been gradually tackled, we're down to a small-ish
> set of issues causing reliable red results when we run unit tests on
> vivid(1.3) and wily/xenial(1.5).
>
> "TestContainerProvisionerStarted did not start"
> http://reports.vapour.ws/releases/issue/5614b0e7749a5613c5aff2e0
>
> "TestAPI2ResultError fails"
> http://reports.vapour.ws/releases/issue/5640b28f749a562a3f656b39
>
> "unexpected message panic"
> http://reports.vapour.ws/releases/issue/55f34dfd749a561a11823b0b
>
> Seem to be the ones I see most in recent test failures.
>
> Martin
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

fcntl won't work in threaded go applications, it barely works in non
threaded applications.

I'm not interested in "doesn't work on windows" arguments. Yes, we
have to support windows, but that doesn't mean we have to be dragged
down to it's lowest common denominator.

I think it's fine to develop a lock type that does the best available
for each platform using conditional compilation.

On Tue, Dec 1, 2015 at 11:47 AM, Andrew Wilkins
<andrew.wilk...@canonical.com> wrote:
> On Tue, Dec 1, 2015 at 8:03 AM David Cheney <david.che...@canonical.com>
> wrote:
>>
>> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
>> <andrew.wilk...@canonical.com> wrote:
>> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney <david.che...@canonical.com>
>> > wrote:
>> >>
>> >> Doesn't look like there is windows support, and it uses fcntl (flock)
>> >> under the hood, which is what we have now.
>> >
>> >
>> > flock isn't the problematic thing Tim is talking about. utils/fslock
>> > attempts to create a directory in a known location, and if it succeeds,
>> > it
>> > "holds the lock". Unlocking means removing the directory.
>>
>> The problem is if the process dies/exits/goes mental while holding the
>> lock we get into this existential gridlock where we're not sure if the
>> process _is_ alive, so we shouldn't remove the lock, or the process is
>> dead, so we should remove the lock.
>>
>> abstract unix domain sockets do not have this problem on windows; kill
>> the process, file is removed from the abstract namespace.
>
>
> POSIX advisory file locks (flock) do not have this problem either. See:
> man(2) fcntl, section "Advisory record locking". When the file descriptor is
> closed, the lock is released; file descriptors are closed when the process
> dies.
>
> We could build a mutex on top of a unix domain socket, but then we'll have
> something completely separate for Windows. Shared named mutex? I'm not
> convinced the overall solution would be any more robust, and I'm pretty sure
> it's going to be more complicated. Happy to be proven wrong though.
>
>> >
>> > We would have to contribute Windows support, but it's not hard, I've
>> > done it
>> > before.
>> >
>> >>
>> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
>> >> <casey.marsh...@canonical.com> wrote:
>> >> > How about github.com/camlistore/lock ?
>> >> >
>> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey
>> >> > <tim.pen...@canonical.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi folks,
>> >> >>
>> >> >> The fslock was a mistake that I added to the codebase some time
>> >> >> back.
>> >> >> It
>> >> >> provided an overly simplistic solution to a more complex problem.
>> >> >>
>> >> >> Really the filesystem shouldn't be used as a locking mechanism.
>> >> >>
>> >> >> Most of the code that exists for the fslock now is working around
>> >> >> its
>> >> >> deficiencies. Instead we should be looking for a better replacement.
>> >> >>
>> >> >> Some "features" that were added to fslock were added to work around
>> >> >> the
>> >> >> issue that the lock did not die with the process that created it, so
>> >> >> some mechanism was needed to determine whether the lock should be
>> >> >> broken
>> >> >> or not.
>> >> >>
>> >> >> What we really need is a good OS agnostic abstraction that provides
>> >> >> the
>> >> >> ability to create a "named" lock, acquire the lock, release the
>> >> >> lock,
>> >> >> and make sure that the lock dies when the process dies, so another
>> >> >> process that is waiting can acquire the lock. This way no
>> >> >> "BreakLock"
>> >> >> functionality is required, nor do we need to try and do think like
>> >> >> remember which process owns the lock.
>> >> >>
>> >> >> So...
>> >> >>
>> >> >> We have three current operating systems we need to support:
>> >> >>
>> >> >> Linux - Ubuntu and CentOS
>> >> >> MacOS - client only - bit the CLI uses a lock for th

Re: utils/fslock needs to DIAF

[David Cheney]

Doesn't look like there is windows support, and it uses fcntl (flock)
under the hood, which is what we have now.

On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
 wrote:
> How about github.com/camlistore/lock ?
>
> On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey 
> wrote:
>>
>> Hi folks,
>>
>> The fslock was a mistake that I added to the codebase some time back. It
>> provided an overly simplistic solution to a more complex problem.
>>
>> Really the filesystem shouldn't be used as a locking mechanism.
>>
>> Most of the code that exists for the fslock now is working around its
>> deficiencies. Instead we should be looking for a better replacement.
>>
>> Some "features" that were added to fslock were added to work around the
>> issue that the lock did not die with the process that created it, so
>> some mechanism was needed to determine whether the lock should be broken
>> or not.
>>
>> What we really need is a good OS agnostic abstraction that provides the
>> ability to create a "named" lock, acquire the lock, release the lock,
>> and make sure that the lock dies when the process dies, so another
>> process that is waiting can acquire the lock. This way no "BreakLock"
>> functionality is required, nor do we need to try and do think like
>> remember which process owns the lock.
>>
>> So...
>>
>> We have three current operating systems we need to support:
>>
>> Linux - Ubuntu and CentOS
>> MacOS - client only - bit the CLI uses a lock for the local cache
>> Windows
>>
>> For Linux, and possibly MacOS, flock is a possibility, but can we do
>> better? Is there something that is better suited?
>>
>> For Windows, while you can create global semaphores or mutex instances,
>> I'm not sure of entities that die with the process. Can people recommend
>> solutions?
>>
>> Cheers,
>> Tim
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Reviews for merges _from_ master to feature branches

[David Cheney]

Hello,

Why are reviewers being created for merges from master to feature
branches ? What purpose does this serve ?

Thanks

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

On Tue, Dec 1, 2015 at 10:43 AM, Tim Penhey  wrote:
> Hi folks,
>
> The fslock was a mistake that I added to the codebase some time back. It
> provided an overly simplistic solution to a more complex problem.
>
> Really the filesystem shouldn't be used as a locking mechanism.
>
> Most of the code that exists for the fslock now is working around its
> deficiencies. Instead we should be looking for a better replacement.
>
> Some "features" that were added to fslock were added to work around the
> issue that the lock did not die with the process that created it, so
> some mechanism was needed to determine whether the lock should be broken
> or not.
>
> What we really need is a good OS agnostic abstraction that provides the
> ability to create a "named" lock, acquire the lock, release the lock,
> and make sure that the lock dies when the process dies, so another
> process that is waiting can acquire the lock. This way no "BreakLock"
> functionality is required, nor do we need to try and do think like
> remember which process owns the lock.
>
> So...
>
> We have three current operating systems we need to support:
>
> Linux - Ubuntu and CentOS
> MacOS - client only - bit the CLI uses a lock for the local cache
> Windows
>
> For Linux, and possibly MacOS, flock is a possibility, but can we do
> better? Is there something that is better suited?

For linux the best alternative I know of for a per process mutex is to
use a unix domain socket in the abstract namespace. These are
automatically removed when the listening socket is closed, or the
process dies, there is no need to cleanup stale pids.

For other posix systems a pidfile, or on disk unix domain socket is
probably the best we can do.

>
> For Windows, while you can create global semaphores or mutex instances,
> I'm not sure of entities that die with the process. Can people recommend
> solutions?
>
> Cheers,
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

Please no. The better way is to use an abstract unix domain socket to
create a mutex.

On Tue, Dec 1, 2015 at 10:56 AM, Andrew Wilkins
 wrote:
> On Tue, Dec 1, 2015 at 7:43 AM Tim Penhey  wrote:
>>
>> Hi folks,
>>
>> The fslock was a mistake that I added to the codebase some time back. It
>> provided an overly simplistic solution to a more complex problem.
>>
>> Really the filesystem shouldn't be used as a locking mechanism.
>>
>> Most of the code that exists for the fslock now is working around its
>> deficiencies. Instead we should be looking for a better replacement.
>>
>> Some "features" that were added to fslock were added to work around the
>> issue that the lock did not die with the process that created it, so
>> some mechanism was needed to determine whether the lock should be broken
>> or not.
>>
>> What we really need is a good OS agnostic abstraction that provides the
>> ability to create a "named" lock, acquire the lock, release the lock,
>> and make sure that the lock dies when the process dies, so another
>> process that is waiting can acquire the lock. This way no "BreakLock"
>> functionality is required, nor do we need to try and do think like
>> remember which process owns the lock.
>>
>> So...
>>
>> We have three current operating systems we need to support:
>>
>> Linux - Ubuntu and CentOS
>> MacOS - client only - bit the CLI uses a lock for the local cache
>> Windows
>>
>> For Linux, and possibly MacOS, flock is a possibility, but can we do
>> better? Is there something that is better suited?
>>
>> For Windows, while you can create global semaphores or mutex instances,
>> I'm not sure of entities that die with the process. Can people recommend
>> solutions?
>
>
> I've been wanting to do this for a long time (I think I've whinged in your
> vicinity about it before), but I've held off because of uncertainty about
> compatibility with NFS (which is probably a rare scenario, and only for the
> client).
>
> I think it was jam that brought up the heritage of directory locking in bzr,
> where flock doesn't work reliably over NFS. I think that's old news though.
> The manpage for flock discusses the NFS/kernel limitations of flock; since
> we don't go back past precise, we should be fine.
>
> I think flock (fcntl F_SETLK/F_GETLK) is an appropriate option for Linux and
> OS X. Windows has FileLock
> (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365202(v=vs.85).aspx),
> and LockFileEx (for more control). Just as with F_SETLK, if the process
> dies, the lock is released.
>
> Cheers,
> Andrew
>
>> Cheers,
>> Tim
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Reviews for merges _from_ master to feature branches

[David Cheney]

It's not gonna get through the merge bot if that's the case. To
clarify, using the landing bot, thumbs up, pointless reviews that
nobody reviews and have poor descriptions, thumbs down.

On Tue, Dec 1, 2015 at 11:20 AM, Rick Harding
<rick.hard...@canonical.com> wrote:
> Sanity check on merge conflicts resolved?
>
>
> On Mon, Nov 30, 2015, 7:18 PM David Cheney <david.che...@canonical.com>
> wrote:
>>
>> Hello,
>>
>> Why are reviewers being created for merges from master to feature
>> branches ? What purpose does this serve ?
>>
>> Thanks
>>
>> Dave
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Upgrading minimum Go version?

[David Cheney]

As of this morning, the unit tests pass on xenial (thanks to Andrew
for getting them over the hump) using Go 1.5

On Tue, Dec 1, 2015 at 3:21 AM, Curtis Hovey-Canonical
 wrote:
> On Mon, Nov 30, 2015 at 10:36 AM, John Meinel  wrote:
>> Given how often people still use "--upload-tools" for things like private
>> clouds (and is definitely the one used for local provider), I'd really worry
>> about having a jujud on your local machine that wasn't built with the same
>> toolchain as the one you get from "juju bootstrap" in other cases. Very easy
>> to end up with hard to understand/reproduce bugs.
>
> I share your concerns John.
>
> We have seen several bugs marked invalid because the agent uploads is
> essentially unknown origin because Juju is more than willing to fake a
> version when uploading. There are also reports of Juju attempting to
> build agents from source:
> https://bugs.launchpad.net/juju-core/+bug/1399606
>
> We do want everyone using the agent in streams, which have a lot more
> testing behind them. The agent (jujud) on users of the Juju stable PPA
> is the same that was used to make streams. This is not true for users
> of Ubuntu's packages, and we know Ubuntu can change its tool chain
> between the time we create official agents and it builds its packages.
> We do testing to certify they Ubuntu clients and agents are
> functionally equivalent to those in the PPA and streams. As Ubuntu
> Wily+ prefers system Go dev packages to the embedded go packages we
> provide, Ubuntu's packages will contain more divergence than Trusty.
> We are obligated to do on-demand testing to certify a change to a
> system Go dev package.
>
> Ideally, We would separate the jujud from juju-core package that
> provides the client. Users don't get a jujud when they install a
> client. Instead, Juju can get the desired agents from streams.
>
> --
> Curtis Hovey
> Canonical Cloud Development and Operations
> http://launchpad.net/~sinzui
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
<andrew.wilk...@canonical.com> wrote:
> On Tue, Dec 1, 2015 at 7:57 AM David Cheney <david.che...@canonical.com>
> wrote:
>>
>> Doesn't look like there is windows support, and it uses fcntl (flock)
>> under the hood, which is what we have now.
>
>
> flock isn't the problematic thing Tim is talking about. utils/fslock
> attempts to create a directory in a known location, and if it succeeds, it
> "holds the lock". Unlocking means removing the directory.

The problem is if the process dies/exits/goes mental while holding the
lock we get into this existential gridlock where we're not sure if the
process _is_ alive, so we shouldn't remove the lock, or the process is
dead, so we should remove the lock.

abstract unix domain sockets do not have this problem on windows; kill
the process, file is removed from the abstract namespace.

>
> We would have to contribute Windows support, but it's not hard, I've done it
> before.
>
>>
>> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
>> <casey.marsh...@canonical.com> wrote:
>> > How about github.com/camlistore/lock ?
>> >
>> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey <tim.pen...@canonical.com>
>> > wrote:
>> >>
>> >> Hi folks,
>> >>
>> >> The fslock was a mistake that I added to the codebase some time back.
>> >> It
>> >> provided an overly simplistic solution to a more complex problem.
>> >>
>> >> Really the filesystem shouldn't be used as a locking mechanism.
>> >>
>> >> Most of the code that exists for the fslock now is working around its
>> >> deficiencies. Instead we should be looking for a better replacement.
>> >>
>> >> Some "features" that were added to fslock were added to work around the
>> >> issue that the lock did not die with the process that created it, so
>> >> some mechanism was needed to determine whether the lock should be
>> >> broken
>> >> or not.
>> >>
>> >> What we really need is a good OS agnostic abstraction that provides the
>> >> ability to create a "named" lock, acquire the lock, release the lock,
>> >> and make sure that the lock dies when the process dies, so another
>> >> process that is waiting can acquire the lock. This way no "BreakLock"
>> >> functionality is required, nor do we need to try and do think like
>> >> remember which process owns the lock.
>> >>
>> >> So...
>> >>
>> >> We have three current operating systems we need to support:
>> >>
>> >> Linux - Ubuntu and CentOS
>> >> MacOS - client only - bit the CLI uses a lock for the local cache
>> >> Windows
>> >>
>> >> For Linux, and possibly MacOS, flock is a possibility, but can we do
>> >> better? Is there something that is better suited?
>> >>
>> >> For Windows, while you can create global semaphores or mutex instances,
>> >> I'm not sure of entities that die with the process. Can people
>> >> recommend
>> >> solutions?
>> >>
>> >> Cheers,
>> >> Tim
>> >>
>> >> --
>> >> Juju-dev mailing list
>> >> Juju-dev@lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>> >
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev@lists.ubuntu.com
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

http://0pointer.de/blog/projects/locking.html

In short, opening the same file twice and asserting a lock on it will succeed.

On Tue, Dec 1, 2015 at 12:04 PM, Andrew Wilkins
<andrew.wilk...@canonical.com> wrote:
> On Tue, Dec 1, 2015 at 8:57 AM David Cheney <david.che...@canonical.com>
> wrote:
>>
>> fcntl won't work in threaded go applications, it barely works in non
>> threaded applications.
>
>
> This is news to me. I've used it plenty in the past, in multi-threaded
> programs. Please point me at relevant literature that explains where it
> doesn't work.
>
>>
>> I'm not interested in "doesn't work on windows" arguments. Yes, we
>> have to support windows, but that doesn't mean we have to be dragged
>> down to it's lowest common denominator.
>
>
> Agreed, if we're actually crippling anything.
>
>>
>> I think it's fine to develop a lock type that does the best available
>> for each platform using conditional compilation.
>
>
> Again, agreed, but only if there's something to be gained by doing this. I'm
> still not convinced there is.
>
>> On Tue, Dec 1, 2015 at 11:47 AM, Andrew Wilkins
>> <andrew.wilk...@canonical.com> wrote:
>> > On Tue, Dec 1, 2015 at 8:03 AM David Cheney <david.che...@canonical.com>
>> > wrote:
>> >>
>> >> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
>> >> <andrew.wilk...@canonical.com> wrote:
>> >> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney
>> >> > <david.che...@canonical.com>
>> >> > wrote:
>> >> >>
>> >> >> Doesn't look like there is windows support, and it uses fcntl
>> >> >> (flock)
>> >> >> under the hood, which is what we have now.
>> >> >
>> >> >
>> >> > flock isn't the problematic thing Tim is talking about. utils/fslock
>> >> > attempts to create a directory in a known location, and if it
>> >> > succeeds,
>> >> > it
>> >> > "holds the lock". Unlocking means removing the directory.
>> >>
>> >> The problem is if the process dies/exits/goes mental while holding the
>> >> lock we get into this existential gridlock where we're not sure if the
>> >> process _is_ alive, so we shouldn't remove the lock, or the process is
>> >> dead, so we should remove the lock.
>> >>
>> >> abstract unix domain sockets do not have this problem on windows; kill
>> >> the process, file is removed from the abstract namespace.
>> >
>> >
>> > POSIX advisory file locks (flock) do not have this problem either. See:
>> > man(2) fcntl, section "Advisory record locking". When the file
>> > descriptor is
>> > closed, the lock is released; file descriptors are closed when the
>> > process
>> > dies.
>> >
>> > We could build a mutex on top of a unix domain socket, but then we'll
>> > have
>> > something completely separate for Windows. Shared named mutex? I'm not
>> > convinced the overall solution would be any more robust, and I'm pretty
>> > sure
>> > it's going to be more complicated. Happy to be proven wrong though.
>> >
>> >> >
>> >> > We would have to contribute Windows support, but it's not hard, I've
>> >> > done it
>> >> > before.
>> >> >
>> >> >>
>> >> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
>> >> >> <casey.marsh...@canonical.com> wrote:
>> >> >> > How about github.com/camlistore/lock ?
>> >> >> >
>> >> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey
>> >> >> > <tim.pen...@canonical.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi folks,
>> >> >> >>
>> >> >> >> The fslock was a mistake that I added to the codebase some time
>> >> >> >> back.
>> >> >> >> It
>> >> >> >> provided an overly simplistic solution to a more complex problem.
>> >> >> >>
>> >> >> >> Really the filesystem shouldn't be used as a locking mechanism.
>> >> >> >>
>> >> >> >> Most of the code that exists for the fslock now is working around
>> >> >> >> its
>> >> >> >> def

Re: utils/fslock needs to DIAF

[David Cheney]

On Tue, Dec 1, 2015 at 8:10 PM, roger peppe <roger.pe...@canonical.com> wrote:
> So I'm with axw on this one - flock seems like it is a reasonable tool for
> the job here. FWIW a Unix domain socket also suffers from the "won't
> work across NFS" problem. Abstract unix domain sockets also
> have the problem that they won't work with long file paths (what
> were they thinking?)

This is false, abstract unix domains sockets have nothing to do with
nfs, they don't live on a filesystem. wrt to a long path, don't think
of it as a path, think of it as a per machine key.

> We have been using github.com/camlistore/lock and although it's not
> totally ideal (see https://github.com/camlistore/lock/issues/10 )
> it does the job. Note that it's non-blocking, so a blocking
> layer above it is necessary, for example see the lockFile
> function in 
> https://github.com/juju/persistent-cookiejar/blob/master/serialize.go
>
> The Windows named mutex thing does sound interesting because
> it's a blocking API, which is actually what we need. On the other
> hand, under Windows, files opened for writing are locked by default
> anyway, so it might be easier just to leverage that property.
> The camlistore lock code could use some improvement for the
> Windows platform - we could either fork it, or bradfitz would
> probably be amenable to a PR.
>
>   cheers,
> rog.
>
> On 1 December 2015 at 04:12, Nate Finch <nate.fi...@canonical.com> wrote:
>> I'm not a linux expert, but definitely a named mutex is exactly the correct
>> thing to use for Windows.  Using something else for this purpose would be
>> very surprising to a Windows dev and much more likely to be buggy.  I don't
>> see any reason we need to use the exact same implementation on all OSes if
>> there is something that does exactly the right thing without any finagling.
>> FWIW, mutexes do die with the process:
>> https://msdn.microsoft.com/en-us/library/windows/desktop/ms682411(v=vs.85).aspx
>>
>> On Mon, Nov 30, 2015 at 8:29 PM Andrew Wilkins
>> <andrew.wilk...@canonical.com> wrote:
>>>
>>> On Tue, Dec 1, 2015 at 9:07 AM David Cheney <david.che...@canonical.com>
>>> wrote:
>>>>
>>>> http://0pointer.de/blog/projects/locking.html
>>>>
>>>> In short, opening the same file twice and asserting a lock on it will
>>>> succeed.
>>>
>>>
>>> Thanks. The article is a bit exasperated. Yes, there are problems to be
>>> aware of, but it doesn't make them unusable in all cases.
>>>  - Juju agents don't get installed onto NFS file systems, so doesn't
>>> matter for the agents.
>>>  - We're in full control of the files we're locking, we're not locking
>>> some file like /etc/passwd where some other random bit of code in the
>>> process is going to open/close it and release the lock by accident.
>>>  - We don't need byte-range locking.
>>>
>>> So only the original uncertainty remains: do we care about clients running
>>> their home directory on an NFS share, where the NFS *server* is too old to
>>> support flock?
>>>
>>> Maybe a named mutex on Windows and a domain socket on *NIX is the way to
>>> go. I'm not dead set on flock; I just want something that is simple, robust
>>> and portable.
>>>
>>> Cheers,
>>> Andrew
>>>
>>>> On Tue, Dec 1, 2015 at 12:04 PM, Andrew Wilkins
>>>> <andrew.wilk...@canonical.com> wrote:
>>>> > On Tue, Dec 1, 2015 at 8:57 AM David Cheney
>>>> > <david.che...@canonical.com>
>>>> > wrote:
>>>> >>
>>>> >> fcntl won't work in threaded go applications, it barely works in non
>>>> >> threaded applications.
>>>> >
>>>> >
>>>> > This is news to me. I've used it plenty in the past, in multi-threaded
>>>> > programs. Please point me at relevant literature that explains where it
>>>> > doesn't work.
>>>> >
>>>> >>
>>>> >> I'm not interested in "doesn't work on windows" arguments. Yes, we
>>>> >> have to support windows, but that doesn't mean we have to be dragged
>>>> >> down to it's lowest common denominator.
>>>> >
>>>> >
>>>> > Agreed, if we're actually crippling anything.
>>>> >
>>>> >>
>>>> >> I think it's fine to develop a lock type that does the best available
>>>> >> for each platform using conditional co

The CI build time continue to rise alarmingly

[David Cheney]

CI build times are now an average of 36 minutes. That means in a
typical 8 hour work day, assuming doing nothing other than landing
branches, less than 16 commits can be landed.

While bugs can be worked on and reviewed in parallel, landing is a
sequential action that blocks everyone, and given the landing bot is
batting less than 0.500, this limits the practical number of changes
that can be landed in a day, a sprint, a iteration, or a development
cycle.

I cannot make it any clearer than this, the speed of CI limits the
velocity of this team.

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev