Re: Can we get an update on the individual juju CLI/API tasks?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-12-06 5:51, Tim Penhey wrote: > Here are the outstanding items from the blueprint [1] > > api-endpoints: TODO upgrade-charm: TODO sync-tools: TODO api-endpoints and sync-tools aren't on anyone's immediate work, afaict. I think api-endpoints is something that already has an API, so should be in the 'trivial to finish' column. upgrade-charm is blocked on dimiter's work on PutCharm for deploy. Which got a good design and I think we feel confident about what needs to be done, it just needs doing. Dimiter can give a more explicit statement of where its at. > > [dimitern] deploy: INPROGRESS [gz] status: INPROGRESS status is about halfway done (from what I've heard from Martin). Just got bumped a bit because of the interruptions and 1.16 focus for a few things. > [axwalk] destroy-environment: INPROGRESS > > This hasn't been updated for ages, and I know that we have moved on > to the hard ones now, but can we share the current status of each > of these? > > For the ones that are in progress, how complete are they? And what > is the ETA to get them done? Both status and deploy could be up for review by early next week (from what I can tell). There is also one more bit which is 1.16 compatibility for trunk, which needs at least 1 more patch to "juju add-machine". We had originally been doing all the work as just working against trunk, but we changed and decided to make 1.18 have fallback code against a 1.16 server (for commands that didn't have an API already). I did most of them, but AddMachine had a second code path that I didn't get to. John =:-> > > Cheers, > > Tim > > [1] > https://blueprints.launchpad.net/juju-core/+spec/t-cloud-juju-cli-api > -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKhZSgACgkQJdeBCYSNAAPiBwCgiT6h9Q3VSUUYUDHJNfnU6hvw tvMAoIdDguHspOMxXEggMXREAueqpjdB =vCzv -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
On Fri, Dec 6, 2013 at 3:39 PM, John Arbash Meinel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 2013-12-06 6:24, David Cheney wrote: >> Ok, good news first. >> >> gccgo compiled tools work fine. >> >> statically linking libgo also makes the tools as self contained as >> our gc compiled ones. >> >> Bad news, >> >> jujud is 40mb, -Os or -O2 effect, in fact the latter makes it a bit >> larger >> >> the binaries cannot be stripped, the debug data is required for >> operation of the program, i'm guessing gccgo's reflection >> implementation requires the debug symbols. >> >> Dave > > Two tidbits from me: > > 1) jujud on my system is 20MB, so this is approx 2x larger. A fair > amount, but I think with compression we saw it wasn't quite as big of > a deal (4.6MB vs 5.5MB in tgz form for the last test, but that was a > smaller binary). > > 2) The Ubuntu security team would *really* like it if we used a system > libgo.so so that they could supply security fixes for it (like the > built in SSL libs) without having to have us rebuild all of our jujud > binaries. Which would save us some of that size (I think you said > approx 10MB was libgo.so), at a cost of having to have a libgo package > for all supported platforms. If we name the package appropriately > (libgo-1.1, etc) then we can probably even still migrate to a > different version of the runtime when it becomes useful (we just > install a different package in cloud-init). That is a fair request, but brings with it a lot of complexity. It isn't sufficient to have a libgo.so, but one that very closely matches the one that the tools were built against. Considering that libgo.so contains all the important parts of the standard library like crypto, compression, http, json, reflection, etc. We also need to ensure that whatever tools we are deploying, are running with an up to date libgo.so. In short, libgo.so and the tools need to be distributed together -- or -- the tools need to control the version of libgo that is installed on the system. This would be easier to accomplish if the tools were packaged as a deb. But that come with other complications for things like upgrades. > > John > =:-> > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.13 (Cygwin) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEUEARECAAYFAlKhVPYACgkQJdeBCYSNAAMAGgCXRPrIqBlFaHOEAuLA4zRZGarV > ZACfS1NM6K6bIZHzaRTBvxO8f/LRrKE= > =Xa62 > -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-12-06 6:24, David Cheney wrote: > Ok, good news first. > > gccgo compiled tools work fine. > > statically linking libgo also makes the tools as self contained as > our gc compiled ones. > > Bad news, > > jujud is 40mb, -Os or -O2 effect, in fact the latter makes it a bit > larger > > the binaries cannot be stripped, the debug data is required for > operation of the program, i'm guessing gccgo's reflection > implementation requires the debug symbols. > > Dave Two tidbits from me: 1) jujud on my system is 20MB, so this is approx 2x larger. A fair amount, but I think with compression we saw it wasn't quite as big of a deal (4.6MB vs 5.5MB in tgz form for the last test, but that was a smaller binary). 2) The Ubuntu security team would *really* like it if we used a system libgo.so so that they could supply security fixes for it (like the built in SSL libs) without having to have us rebuild all of our jujud binaries. Which would save us some of that size (I think you said approx 10MB was libgo.so), at a cost of having to have a libgo package for all supported platforms. If we name the package appropriately (libgo-1.1, etc) then we can probably even still migrate to a different version of the runtime when it becomes useful (we just install a different package in cloud-init). John =:-> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEUEARECAAYFAlKhVPYACgkQJdeBCYSNAAMAGgCXRPrIqBlFaHOEAuLA4zRZGarV ZACfS1NM6K6bIZHzaRTBvxO8f/LRrKE= =Xa62 -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
Ok, good news first. gccgo compiled tools work fine. statically linking libgo also makes the tools as self contained as our gc compiled ones. Bad news, jujud is 40mb, -Os or -O2 effect, in fact the latter makes it a bit larger the binaries cannot be stripped, the debug data is required for operation of the program, i'm guessing gccgo's reflection implementation requires the debug symbols. Dave On Thu, Dec 5, 2013 at 5:42 PM, Andrew Wilkins wrote: > On Thu, Dec 5, 2013 at 2:37 PM, David Cheney > wrote: > >> On Thu, Dec 5, 2013 at 5:32 PM, John Meinel >> wrote: >> > It sounds like we should be bundling the libgo.so into the tar.gz that >> we >> > are building. Either statically (so we dont have to worry about system >> skew) >> > or just have it know to load it in the same dir as the executable. We >> > *could* try to use a system lib, but then we're back into dealing with >> cross >> > series version skew. (What version of libgo.so would we get on precise, >> vs >> > trusty, etc) >> >> Yup, _not_ linking it statically will be an entire dimension of hurt. So >> >> lucky(~/src/launchpad.net/juju-core) % go install -v -compiler gccgo >> -gccgoflags=-static-libgo ./... >> >> did the business, and I could bootstrap, however ... >> >> Installing add-apt-repository >> >> Adding apt repository: deb >> http://ubuntu-cloud.archive.canonical.com/ubuntu >> precise-updates/cloud-tools main >> >> Running apt-get update >> >> Running apt-get upgrade >> >> Installing package: git >> >> Installing package: cpu-checker >> >> Installing package: mongodb-server >> >> Fetching tools: wget --no-verbose -O $bin/tools.tar.gz >> ' >> https://s3-ap-southeast-2.amazonaws.com/juju-syd-en-ee-ii/tools/releases/juju-1.17.0.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Expires=1701757655&Signature=h2ujOoLJidL%2FEKTHnmOrAZ9e0A8%3D >> ' >> >> Starting MongoDB server (juju-db) >> >> Bootstrapping Juju machine agent >> >> Starting Juju machine agent (jujud-machine-0) >> >> Connection to ec2-54-253-158-49.ap-southeast-2.compute.amazonaws.comclosed. >> >> 2013-12-05 06:32:52 INFO juju supercommand.go:286 command finished >> >> lucky(~/src/launchpad.net/juju-core) % juju ssh 0 >> >> WARNING discarding API open error: >> >> ERROR environment has no access-key or secret-key > > > That's a bug with API-based secrets pushing. I have a CL pending. > In the mean time, do "juju status" first. > > Nice work, btw. > > >> >> > >> > Also I was surprised that the tar.gz is significantly bigger even >> without >> > libgo.so in it. (Lasting time I uploaded tools it was 4.6MB, your log >> shows >> > it as 5.5MB, and juju-1.10 was like 2.4MB IIRC) >> >> It's 10.4mb when libgo is linked statically. >> >> -- >> Juju-dev mailing list >> Juju-dev@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> > > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Can we get an update on the individual juju CLI/API tasks?
For destroy-environment: I first need to land https://codereview.appspot.com/28880043/ and then get back to https://codereview.appspot.com/22870045/ ETA: not sure, but not this week, and probably not next week either. I hope to have the first CL accepted soon, but the second one needs reworking. Cheers, Andrew On Fri, Dec 6, 2013 at 9:51 AM, Tim Penhey wrote: > Here are the outstanding items from the blueprint [1] > > api-endpoints: TODO > upgrade-charm: TODO > sync-tools: TODO > > [dimitern] deploy: INPROGRESS > [gz] status: INPROGRESS > [axwalk] destroy-environment: INPROGRESS > > This hasn't been updated for ages, and I know that we have moved on to > the hard ones now, but can we share the current status of each of these? > > For the ones that are in progress, how complete are they? And what is > the ETA to get them done? > > Cheers, > > Tim > > [1] https://blueprints.launchpad.net/juju-core/+spec/t-cloud-juju-cli-api > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Can we get an update on the individual juju CLI/API tasks?
Here are the outstanding items from the blueprint [1] api-endpoints: TODO upgrade-charm: TODO sync-tools: TODO [dimitern] deploy: INPROGRESS [gz] status: INPROGRESS [axwalk] destroy-environment: INPROGRESS This hasn't been updated for ages, and I know that we have moved on to the hard ones now, but can we share the current status of each of these? For the ones that are in progress, how complete are they? And what is the ETA to get them done? Cheers, Tim [1] https://blueprints.launchpad.net/juju-core/+spec/t-cloud-juju-cli-api -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: New juju-mongodb package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ... >> Installed sizes: > >> -rwxr-xr-x 1 root root 7.2M Sep 5 15:30 /usr/bin/mongo >> -rwxr-xr-x 1 root root 13M Sep 5 15:30 /usr/bin/mongodump >> -rwxr-xr-x 1 root root 13M Sep 5 15:30 /usr/bin/mongoexport >> -rwxr-xr-x 1 root root 13M Sep 5 15:30 /usr/bin/mongoimport >> -rwxr-xr-x 1 root root 13M Sep 5 15:30 /usr/bin/mongorestore > >> Expect ~50% ish compression for the package itself. > >> Lets think about download sizes/times for end-users in this >> context please :-). It costs time for every server that's >> provisioned. > > Compressed package size comes to 24MB. So the discussion ended up that we could probably get away with a minimum of: mongo (requires running server, but doesn't bundle v8 itself) mongod mongodump mongos We can use 'mongo' instead of 'mongoexport' for the specific use case (just have to run it before stopping the db). We'll definitely want easy access to mongorestore, though I don't think we need mongoimport. I get the feeling if we do have a 'juju-mongodb' which has our minimum set, we probably would still want the possibility of installing 'juju-mongodb-tools' which has all of mongoexport/mongoimport/mongooplog/mongoperf/morgorestore/mongosniff, etc. Essentially mongodb-clients but guaranteed to work with the version of mongo we have running. Is that silly? Looking here: http://archive.ubuntu.com/ubuntu/pool/universe/m/mongodb/ It looks like the major size issue is the v8 stuff given that mongodb-client on 2.2.4 is 20M, but on 2.4.6 it is 60M. Is there a reason it can't be a shared library between the ~4 client binaries? Given that mongodb-server is 9.2M, I assume most of the size is those extra client libs. If it is just about what we are actively advocating having available 'mongorestore' would also go into the above list, but it is the sort of thing that you don't do "regularly" in a healthy system. John =:-> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKg0HYACgkQJdeBCYSNAANlNgCcCrqa0qDUNZj58xSbVKwozRyW deoAnjZvyEUnucRXPI+f7PE8zgwLrmGG =Uxcn -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: New juju-mongodb package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 02/12/13 10:36, James Page wrote: > On 02/12/13 09:15, John Arbash Meinel wrote: Which of these is core to the function of juju, and which are useful for debugging? Please also that the mongodb-clients package will still be available in universe so you will always be able to get any of the client tools that way as well (I'll probably keep those packages maintained to the latest point release for the released series). >> Looking at the juju-backup script, it is intended to use both >> dump and export. The export is so that we end up with JSON format >> for our "settings" table so we can extract content from there as >> part of restore (what credentials were you using, etc). > >> Are they both really big? If so, we can probably use: mongo >> --eval "STUFF" instead of "mongoexport", but that would again >> presume that we have a 'mongo' binary available. > >> I *believe* the data in the mongo dump is just BSON encoded, so >> it would be possible to post-filter it with a bson aware tool to >> get it into human-consumable form (rather than using mongoexport >> at backup time). > >> I guess the question is, how expensive is it to include it? For >> Juju core folks the cost of including it is all externality so >> *we* aren't particularly motivated to keep our toolset minimal. >> But if there is a use case we want to support, we're willing to >> help. > > Installed sizes: > > -rwxr-xr-x 1 root root 7.2M Sep 5 15:30 /usr/bin/mongo -rwxr-xr-x > 1 root root 13M Sep 5 15:30 /usr/bin/mongodump -rwxr-xr-x 1 root > root 13M Sep 5 15:30 /usr/bin/mongoexport -rwxr-xr-x 1 root root > 13M Sep 5 15:30 /usr/bin/mongoimport -rwxr-xr-x 1 root root 13M > Sep 5 15:30 /usr/bin/mongorestore > > Expect ~50% ish compression for the package itself. > > Lets think about download sizes/times for end-users in this > context please :-). It costs time for every server that's > provisioned. Compressed package size comes to 24MB. - -- James Page Technical Lead Ubuntu Server Team james.p...@canonical.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.15 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJSoKRoAAoJEL/srsug59jDcnsQAKoY6A4V0FX+NqEXiBIyMDWZ 2xPgZ+CHFSyYIS/nMcWjwhflH3FBFV+XbDUA6KMnYoJDwWdDP6Z3TtTFnHEqWPed 3VipV98CUezNlWomG850t2ztww4V9uZQoiEiVZ7il0lj6o2dEVvMuaobHGQ9ednN 0+ewKbTYibouJ8CbgmXBgLKnhKVS7Bq6UwE0zo3EZ/MQlPk5gHgxhmQt3jZLvkvJ iQ8iGoY5Z7KLR8WioFhDlnCKLBo+DN9jJeiv7DXfm2DKD6xfKFDvSLKnA13vB6RW zucgeY+5+0+HB096DLxrmft34icv3Ygp/MBzccjDTtygNYnT1tFFKEKSyMHZHJHP S0jeUm0LkKsmuex01VTksd3GrNsxJBIenXJc25MkfcC1FNZUOebrnZJ1PEJTcFLr ml1tZY57xp3j3wnt3kobjQ3vfllZvHcKpgGRB3o9q0w1XGKq9BrGBezElGPCTNLU 7RU3Hq3INqJ+kuaNgItiYZX52hmrRM1317zGrWSA69Kc7GAQibnjfxTP4Jdf5EAO mvuld7MkwdwcQpHDO/4BKgoBCotKcg6pJguHqgO9C7ZR1WY1V1V8UsmDv1YH0irV 9uwPqs3ElT+iNg1eWYggA/iMkSL8DYdNoozpYSQ8n1IN8cl+MsDVD1BBoGcDcZoC ro6tNvXFf1wxNdIeJ/i0 =Wcc/ -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Starting on private charms
Greetings, As discussed in a meeting yesterday, I'm sending the list a quick note about our intention to start working on support for private charms. The motivation for the feature is somewhat obvious: people should be able to create charms for content that they're not comfortable making publicly available. The initial plan is rather straightforward as well. We need to: - Integrate the Ubuntu Single Sign On system with the charm store backend; - Adapt our clients (juju, the charm browser, etc) so that they're able to authenticate while talking to the charm store backend; - Find a way to harvest and fetch private charms out of Launchpad, in very similar ways to what we do today with the public content. The details of the interactions with Launchpad are still being sorted out, as we need to understand what's a cheap way to leverage what already works, what needs to be improved to enable a good workflow, and who can do it. This is just a kickoff email to raise any conversations needed (if any). Casey Marshall, Andrew Deane, Matthew Williams, and others will be pushing the actual work. The ball is now rolling.. gustavo @ http://niemeyer.net -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: SSH keys
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 13-12-04 11:15 PM, Andrew Wilkins wrote: > So, synchronous bootstrap broke CI. The reason for this is that > we're now using SSH as part of the process; I can see in the CI > logs that a non-default identity file is being used with "juju > scp". That explains why "juju ssh" (during bootstrap) is failing -- > it just tries the default keys. So the first thing is, it seems messy to require more than one set of credentials for a machine. Until this point, only the ca-cert and ca-private-key were required. authorized_keys was a helper for ssh and scp, which are not really part of the juju abstraction-- they're ways to escape the juju abstraction and mess with the machine directly. Which is why CI hasn't had this issue until now. Now, I personally would love it if you solved this by ditching the ca-cert and ca-private-key and just use SSH everywhere, because on canonistack and presumably, other private clouds, SSH is the only realistic way to access machines: https://bugs.launchpad.net/juju-core/+bug/1224057 But it's worth pointing out that SSH is a complicated beast, and wrapping everything just right is challenging. For example, bootstrap just emitted the "IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!" message on CI: http://162.213.35.54:8080/job/canonistack-upgrade-deploy/112/console So an alternate option is to replace SCP with an API operation. More code perhaps, but less fragility that way. > To workaround, CI could specify the key in ~/.ssh/config (see > https://bugs.launchpad.net/juju-core/+bug/1257371/comments/9). To > fix the problem for good, we can do a couple of things: - Add yet > more configuration to Juju to specify which key to connect with, or > (and/or?) - Auto-generate an SSH key for each new environment at > bootstrap. > > The second option is far more user-friendly IMHO; the less > configuration the better. Is there any reason why we should not do > this? If we did this, then "authorized-keys" would be changed to > implicitly include the auto-generated public key. If you're doing the second, please do the first. We want to be able to pass around environments.yaml files for environments managed by teams[1]. I don't think that's hard-- everything that's valid in a jenv seems to work in environments.yaml so far. Another option is to auto-generate the credentials, but stick them in the control-bucket. That way, they can be used automatically by anyone with access to the bucket, and no one has to worry about getting SSH keys to their teammates. > On a related note, it occurred to me that the Windows CLI won't be > able to bootstrap anymore. We're going to need to update the code > to use the plink executable from PuTTY. [1] I know it's a goal to be able to pass .jenv files around. That's great, but there are problems. It doesn't work yet, because juju still requires an entry in environments.yaml. But more importantly, it doesn't get us reproducible bootstraps, where if I tear down an environment and stand it up again, my teammates can use their existing config to access the new version of the environment. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKgmSgACgkQ0F+nu1YWqI3D0QCfQm74ipXEZpmRFNrghDsaGCCL yTAAni1xn5RA4oWg4/KP5FdmouimWWUY =p5PL -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
