Re: juju bootstrap w/ gccgo built cli and tools
On Fri, Dec 6, 2013 at 3:39 PM, John Arbash Meinel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 2013-12-06 6:24, David Cheney wrote: >> Ok, good news first. >> >> gccgo compiled tools work fine. >> >> statically linking libgo also makes the tools as self contained as >> our gc compiled ones. >> >> Bad news, >> >> jujud is 40mb, -Os or -O2 effect, in fact the latter makes it a bit >> larger >> >> the binaries cannot be stripped, the debug data is required for >> operation of the program, i'm guessing gccgo's reflection >> implementation requires the debug symbols. >> >> Dave > > Two tidbits from me: > > 1) jujud on my system is 20MB, so this is approx 2x larger. A fair > amount, but I think with compression we saw it wasn't quite as big of > a deal (4.6MB vs 5.5MB in tgz form for the last test, but that was a > smaller binary). > > 2) The Ubuntu security team would *really* like it if we used a system > libgo.so so that they could supply security fixes for it (like the > built in SSL libs) without having to have us rebuild all of our jujud > binaries. Which would save us some of that size (I think you said > approx 10MB was libgo.so), at a cost of having to have a libgo package > for all supported platforms. If we name the package appropriately > (libgo-1.1, etc) then we can probably even still migrate to a > different version of the runtime when it becomes useful (we just > install a different package in cloud-init). That is a fair request, but brings with it a lot of complexity. It isn't sufficient to have a libgo.so, but one that very closely matches the one that the tools were built against. Considering that libgo.so contains all the important parts of the standard library like crypto, compression, http, json, reflection, etc. We also need to ensure that whatever tools we are deploying, are running with an up to date libgo.so. In short, libgo.so and the tools need to be distributed together -- or -- the tools need to control the version of libgo that is installed on the system. This would be easier to accomplish if the tools were packaged as a deb. But that come with other complications for things like upgrades. > > John > =:-> > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.13 (Cygwin) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEUEARECAAYFAlKhVPYACgkQJdeBCYSNAAMAGgCXRPrIqBlFaHOEAuLA4zRZGarV > ZACfS1NM6K6bIZHzaRTBvxO8f/LRrKE= > =Xa62 > -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-12-06 6:24, David Cheney wrote: > Ok, good news first. > > gccgo compiled tools work fine. > > statically linking libgo also makes the tools as self contained as > our gc compiled ones. > > Bad news, > > jujud is 40mb, -Os or -O2 effect, in fact the latter makes it a bit > larger > > the binaries cannot be stripped, the debug data is required for > operation of the program, i'm guessing gccgo's reflection > implementation requires the debug symbols. > > Dave Two tidbits from me: 1) jujud on my system is 20MB, so this is approx 2x larger. A fair amount, but I think with compression we saw it wasn't quite as big of a deal (4.6MB vs 5.5MB in tgz form for the last test, but that was a smaller binary). 2) The Ubuntu security team would *really* like it if we used a system libgo.so so that they could supply security fixes for it (like the built in SSL libs) without having to have us rebuild all of our jujud binaries. Which would save us some of that size (I think you said approx 10MB was libgo.so), at a cost of having to have a libgo package for all supported platforms. If we name the package appropriately (libgo-1.1, etc) then we can probably even still migrate to a different version of the runtime when it becomes useful (we just install a different package in cloud-init). John =:-> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEUEARECAAYFAlKhVPYACgkQJdeBCYSNAAMAGgCXRPrIqBlFaHOEAuLA4zRZGarV ZACfS1NM6K6bIZHzaRTBvxO8f/LRrKE= =Xa62 -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
Ok, good news first. gccgo compiled tools work fine. statically linking libgo also makes the tools as self contained as our gc compiled ones. Bad news, jujud is 40mb, -Os or -O2 effect, in fact the latter makes it a bit larger the binaries cannot be stripped, the debug data is required for operation of the program, i'm guessing gccgo's reflection implementation requires the debug symbols. Dave On Thu, Dec 5, 2013 at 5:42 PM, Andrew Wilkins wrote: > On Thu, Dec 5, 2013 at 2:37 PM, David Cheney > wrote: > >> On Thu, Dec 5, 2013 at 5:32 PM, John Meinel >> wrote: >> > It sounds like we should be bundling the libgo.so into the tar.gz that >> we >> > are building. Either statically (so we dont have to worry about system >> skew) >> > or just have it know to load it in the same dir as the executable. We >> > *could* try to use a system lib, but then we're back into dealing with >> cross >> > series version skew. (What version of libgo.so would we get on precise, >> vs >> > trusty, etc) >> >> Yup, _not_ linking it statically will be an entire dimension of hurt. So >> >> lucky(~/src/launchpad.net/juju-core) % go install -v -compiler gccgo >> -gccgoflags=-static-libgo ./... >> >> did the business, and I could bootstrap, however ... >> >> Installing add-apt-repository >> >> Adding apt repository: deb >> http://ubuntu-cloud.archive.canonical.com/ubuntu >> precise-updates/cloud-tools main >> >> Running apt-get update >> >> Running apt-get upgrade >> >> Installing package: git >> >> Installing package: cpu-checker >> >> Installing package: mongodb-server >> >> Fetching tools: wget --no-verbose -O $bin/tools.tar.gz >> ' >> https://s3-ap-southeast-2.amazonaws.com/juju-syd-en-ee-ii/tools/releases/juju-1.17.0.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Expires=1701757655&Signature=h2ujOoLJidL%2FEKTHnmOrAZ9e0A8%3D >> ' >> >> Starting MongoDB server (juju-db) >> >> Bootstrapping Juju machine agent >> >> Starting Juju machine agent (jujud-machine-0) >> >> Connection to ec2-54-253-158-49.ap-southeast-2.compute.amazonaws.comclosed. >> >> 2013-12-05 06:32:52 INFO juju supercommand.go:286 command finished >> >> lucky(~/src/launchpad.net/juju-core) % juju ssh 0 >> >> WARNING discarding API open error: >> >> ERROR environment has no access-key or secret-key > > > That's a bug with API-based secrets pushing. I have a CL pending. > In the mean time, do "juju status" first. > > Nice work, btw. > > >> >> > >> > Also I was surprised that the tar.gz is significantly bigger even >> without >> > libgo.so in it. (Lasting time I uploaded tools it was 4.6MB, your log >> shows >> > it as 5.5MB, and juju-1.10 was like 2.4MB IIRC) >> >> It's 10.4mb when libgo is linked statically. >> >> -- >> Juju-dev mailing list >> Juju-dev@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> > > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
On Thu, Dec 5, 2013 at 2:37 PM, David Cheney wrote: > On Thu, Dec 5, 2013 at 5:32 PM, John Meinel > wrote: > > It sounds like we should be bundling the libgo.so into the tar.gz that we > > are building. Either statically (so we dont have to worry about system > skew) > > or just have it know to load it in the same dir as the executable. We > > *could* try to use a system lib, but then we're back into dealing with > cross > > series version skew. (What version of libgo.so would we get on precise, > vs > > trusty, etc) > > Yup, _not_ linking it statically will be an entire dimension of hurt. So > > lucky(~/src/launchpad.net/juju-core) % go install -v -compiler gccgo > -gccgoflags=-static-libgo ./... > > did the business, and I could bootstrap, however ... > > Installing add-apt-repository > > Adding apt repository: deb > http://ubuntu-cloud.archive.canonical.com/ubuntu > precise-updates/cloud-tools main > > Running apt-get update > > Running apt-get upgrade > > Installing package: git > > Installing package: cpu-checker > > Installing package: mongodb-server > > Fetching tools: wget --no-verbose -O $bin/tools.tar.gz > ' > https://s3-ap-southeast-2.amazonaws.com/juju-syd-en-ee-ii/tools/releases/juju-1.17.0.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Expires=1701757655&Signature=h2ujOoLJidL%2FEKTHnmOrAZ9e0A8%3D > ' > > Starting MongoDB server (juju-db) > > Bootstrapping Juju machine agent > > Starting Juju machine agent (jujud-machine-0) > > Connection to ec2-54-253-158-49.ap-southeast-2.compute.amazonaws.comclosed. > > 2013-12-05 06:32:52 INFO juju supercommand.go:286 command finished > > lucky(~/src/launchpad.net/juju-core) % juju ssh 0 > > WARNING discarding API open error: > > ERROR environment has no access-key or secret-key That's a bug with API-based secrets pushing. I have a CL pending. In the mean time, do "juju status" first. Nice work, btw. > > > > > Also I was surprised that the tar.gz is significantly bigger even without > > libgo.so in it. (Lasting time I uploaded tools it was 4.6MB, your log > shows > > it as 5.5MB, and juju-1.10 was like 2.4MB IIRC) > > It's 10.4mb when libgo is linked statically. > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
On Thu, Dec 5, 2013 at 5:32 PM, John Meinel wrote: > It sounds like we should be bundling the libgo.so into the tar.gz that we > are building. Either statically (so we dont have to worry about system skew) > or just have it know to load it in the same dir as the executable. We > *could* try to use a system lib, but then we're back into dealing with cross > series version skew. (What version of libgo.so would we get on precise, vs > trusty, etc) Yup, _not_ linking it statically will be an entire dimension of hurt. So lucky(~/src/launchpad.net/juju-core) % go install -v -compiler gccgo -gccgoflags=-static-libgo ./... did the business, and I could bootstrap, however ... Installing add-apt-repository Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main Running apt-get update Running apt-get upgrade Installing package: git Installing package: cpu-checker Installing package: mongodb-server Fetching tools: wget --no-verbose -O $bin/tools.tar.gz 'https://s3-ap-southeast-2.amazonaws.com/juju-syd-en-ee-ii/tools/releases/juju-1.17.0.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Expires=1701757655&Signature=h2ujOoLJidL%2FEKTHnmOrAZ9e0A8%3D' Starting MongoDB server (juju-db) Bootstrapping Juju machine agent Starting Juju machine agent (jujud-machine-0) Connection to ec2-54-253-158-49.ap-southeast-2.compute.amazonaws.com closed. 2013-12-05 06:32:52 INFO juju supercommand.go:286 command finished lucky(~/src/launchpad.net/juju-core) % juju ssh 0 WARNING discarding API open error: ERROR environment has no access-key or secret-key > > Also I was surprised that the tar.gz is significantly bigger even without > libgo.so in it. (Lasting time I uploaded tools it was 4.6MB, your log shows > it as 5.5MB, and juju-1.10 was like 2.4MB IIRC) It's 10.4mb when libgo is linked statically. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: juju bootstrap w/ gccgo built cli and tools
It sounds like we should be bundling the libgo.so into the tar.gz that we are building. Either statically (so we dont have to worry about system skew) or just have it know to load it in the same dir as the executable. We *could* try to use a system lib, but then we're back into dealing with cross series version skew. (What version of libgo.so would we get on precise, vs trusty, etc) Also I was surprised that the tar.gz is significantly bigger even without libgo.so in it. (Lasting time I uploaded tools it was 4.6MB, your log shows it as 5.5MB, and juju-1.10 was like 2.4MB IIRC) John =:-> On Dec 5, 2013 10:11 AM, "David Cheney" wrote: > Good news, everyone! > > We are --->this<--- close to getting a gccgo build juju working. > > lucky(~/src/launchpad.net/juju-core) % juju bootstrap -v --upload-tools > > verbose is deprecated with the current meaning, use show-log > > 2013-12-05 05:57:50 INFO juju.environs open.go:156 environment info > already exists; using New not Prepare > > 2013-12-05 05:57:50 INFO juju.provider.ec2 ec2.go:176 opening > environment "ap-southeast-2" > > 2013-12-05 05:57:50 INFO juju.environs.tools build.go:162 found > existing jujud: /home/dfc/bin/jujud > > 2013-12-05 05:57:50 INFO juju.environs.tools build.go:172 target: > /tmp/juju-tools033710042/jujud > > compiled with gccgo > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:244 built > 1.17.0.1-saucy-amd64 (5583kB) > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:70 listing available > tools > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:94 found 2 tools > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:104 listing target > bucket > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:117 found 0 tools > in target; 2 tools to be copied > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:155 copying > 1.17.0.1-precise-amd64 from > file:///tmp/142843249/tools/releases/juju-1.17.0.1-precise-amd64.tgz > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:169 copying > tools/releases/juju-1.17.0.1-precise-amd64.tgz > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:183 downloaded > tools/releases/juju-1.17.0.1-precise-amd64.tgz (5583kB), uploading > > 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:184 download > 5583kB, uploading > > 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:155 copying > 1.17.0.1-saucy-amd64 from > file:///tmp/142843249/tools/releases/juju-1.17.0.1-saucy-amd64.tgz > > 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:169 copying > tools/releases/juju-1.17.0.1-saucy-amd64.tgz > > 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:183 downloaded > tools/releases/juju-1.17.0.1-saucy-amd64.tgz (5583kB), uploading > > 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:184 download > 5583kB, uploading > > 2013-12-05 06:00:18 INFO juju.environs.sync sync.go:122 copied 2 tools > > 2013-12-05 06:00:18 INFO juju.environs.sync sync.go:124 generating > tools metadata > > 2013-12-05 06:00:19 INFO juju.environs.tools simplestreams.go:357 > Writing tools/streams/v1/index.json > > 2013-12-05 06:00:19 INFO juju.environs.tools simplestreams.go:357 > Writing tools/streams/v1/com.ubuntu.juju:released:tools.json > > 2013-12-05 06:00:19 INFO juju.environs.sync sync.go:136 tools metadata > written > > 2013-12-05 06:00:22 INFO juju.environs.bootstrap bootstrap.go:45 > bootstrapping environment "ap-southeast-2" > > 2013-12-05 06:00:22 INFO juju.environs.tools tools.go:85 reading tools > with major.minor version 1.17 > > 2013-12-05 06:00:22 INFO juju.environs.tools tools.go:93 filtering > tools by version: 1.17.0.1 > > 2013-12-05 06:00:22 INFO juju.environs.tools tools.go:96 filtering > tools by series: precise > > 2013-12-05 06:00:22 INFO juju.environs.bootstrap bootstrap.go:57 > picked newest version: 1.17.0.1 > > Launching instance > > 2013-12-05 06:00:29 INFO juju.provider.ec2 ec2.go:418 started instance > "i-813692be" > > - i-813692be > > Waiting for DNS name.. > > - ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com > > Attempting to connect to > ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com:22 > > 2013-12-05 06:01:45 INFO juju.cloudinit.sshinit configure.go:24 > Provisioning machine agent on > ubu...@ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com > > Warning: Permanently added > 'ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com,54.253.221.206' > (ECDSA) to the list of known hosts. > > Logging to /var/log/cloud-init-output.log on remote host > > Installing add-apt-repository > > Adding apt repository: deb > http://ubuntu-cloud.archive.canonical.com/ubuntu > precise-updates/cloud-tools main > > Running apt-get update > > Running apt-get upgrade > > Installing package: git > > Installing package: cpu-checker > > Installing package: mongodb-server > > Fetching tools: wget --no-verbose -O $bin/tools.tar.gz > ' > https://s3-ap-southeast-2.amazonaws.com/juju-syd-en-ee-ii/tools/releases/juju-1.17.0.1-precise-a
juju bootstrap w/ gccgo built cli and tools
Good news, everyone! We are --->this<--- close to getting a gccgo build juju working. lucky(~/src/launchpad.net/juju-core) % juju bootstrap -v --upload-tools verbose is deprecated with the current meaning, use show-log 2013-12-05 05:57:50 INFO juju.environs open.go:156 environment info already exists; using New not Prepare 2013-12-05 05:57:50 INFO juju.provider.ec2 ec2.go:176 opening environment "ap-southeast-2" 2013-12-05 05:57:50 INFO juju.environs.tools build.go:162 found existing jujud: /home/dfc/bin/jujud 2013-12-05 05:57:50 INFO juju.environs.tools build.go:172 target: /tmp/juju-tools033710042/jujud compiled with gccgo 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:244 built 1.17.0.1-saucy-amd64 (5583kB) 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:70 listing available tools 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:94 found 2 tools 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:104 listing target bucket 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:117 found 0 tools in target; 2 tools to be copied 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:155 copying 1.17.0.1-precise-amd64 from file:///tmp/142843249/tools/releases/juju-1.17.0.1-precise-amd64.tgz 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:169 copying tools/releases/juju-1.17.0.1-precise-amd64.tgz 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:183 downloaded tools/releases/juju-1.17.0.1-precise-amd64.tgz (5583kB), uploading 2013-12-05 05:57:53 INFO juju.environs.sync sync.go:184 download 5583kB, uploading 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:155 copying 1.17.0.1-saucy-amd64 from file:///tmp/142843249/tools/releases/juju-1.17.0.1-saucy-amd64.tgz 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:169 copying tools/releases/juju-1.17.0.1-saucy-amd64.tgz 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:183 downloaded tools/releases/juju-1.17.0.1-saucy-amd64.tgz (5583kB), uploading 2013-12-05 05:59:09 INFO juju.environs.sync sync.go:184 download 5583kB, uploading 2013-12-05 06:00:18 INFO juju.environs.sync sync.go:122 copied 2 tools 2013-12-05 06:00:18 INFO juju.environs.sync sync.go:124 generating tools metadata 2013-12-05 06:00:19 INFO juju.environs.tools simplestreams.go:357 Writing tools/streams/v1/index.json 2013-12-05 06:00:19 INFO juju.environs.tools simplestreams.go:357 Writing tools/streams/v1/com.ubuntu.juju:released:tools.json 2013-12-05 06:00:19 INFO juju.environs.sync sync.go:136 tools metadata written 2013-12-05 06:00:22 INFO juju.environs.bootstrap bootstrap.go:45 bootstrapping environment "ap-southeast-2" 2013-12-05 06:00:22 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.17 2013-12-05 06:00:22 INFO juju.environs.tools tools.go:93 filtering tools by version: 1.17.0.1 2013-12-05 06:00:22 INFO juju.environs.tools tools.go:96 filtering tools by series: precise 2013-12-05 06:00:22 INFO juju.environs.bootstrap bootstrap.go:57 picked newest version: 1.17.0.1 Launching instance 2013-12-05 06:00:29 INFO juju.provider.ec2 ec2.go:418 started instance "i-813692be" - i-813692be Waiting for DNS name.. - ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com Attempting to connect to ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com:22 2013-12-05 06:01:45 INFO juju.cloudinit.sshinit configure.go:24 Provisioning machine agent on ubu...@ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com Warning: Permanently added 'ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com,54.253.221.206' (ECDSA) to the list of known hosts. Logging to /var/log/cloud-init-output.log on remote host Installing add-apt-repository Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main Running apt-get update Running apt-get upgrade Installing package: git Installing package: cpu-checker Installing package: mongodb-server Fetching tools: wget --no-verbose -O $bin/tools.tar.gz 'https://s3-ap-southeast-2.amazonaws.com/juju-syd-en-ee-ii/tools/releases/juju-1.17.0.1-precise-amd64.tgz?AWSAccessKeyId=AKIAJ4SOKUWG25EDMAOA&Expires=1701756022&Signature=us5XxZbNtgjiPQ5kMVVyC1b9450%3D' Starting MongoDB server (juju-db) Bootstrapping Juju machine agent Connection to ec2-54-253-221-206.ap-southeast-2.compute.amazonaws.com closed. Stopping instance... 2013-12-05 06:05:07 ERROR juju supercommand.go:282 exit status 1 Unfortunately the agent will fail to start because lucky(~/src/launchpad.net/juju-core) % ldd $(which jujud) linux-vdso.so.1 => (0x7fffbc3fe000) libgo.so.5 => /usr/lib/libgo.so.5 (0x7f57fc52e000) ^^^ this library libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f57fc311000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f57fc00c000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f57fbdf6000) libc.so.6 => /lib/x86_64-linu
