Re: lxc bug blocks 1.17.1 release.
This looks a lot like a Trusty-specific issue to me, and nothing to do with sudo. When did CI move to Trusty? Trusty no longer has the /etc/lxc/auto directory, and instead the LXC container config file has an attribute that says the container should be auto-started with the machine. Jesse has an in-progress CL that addresses this by checking if the auto directory exists before attempting to create the auto-start symlink. Cheers, Andrew On Sat, Jan 25, 2014 at 1:53 PM, Curtis Hovey-Canonical < cur...@canonical.com> wrote: > I reported https://bugs.launchpad.net/juju-core/+bug/1272590 > "cannot deploy with lxc with new sudo rules" > > In summary, since the new rules break CI scripting, I have tested > 1.17.1 local deployments myself. I cannot deploy local services, nor > can I destroy them cleanly (/me reported a separate bug to track that > azure and openstack are shutting down machines instead of deleting > them) > > Our choices are: > 1. Release any way. Note that local provider is broken. >Release 1.17.2 a day or two later when we have a fix. > > 2. Fix the issue. I think we need to manually test still, but maybe > CI will work without changes. > > 3. Rollback the sudo/lxc changes. > > > -- > Curtis Hovey > Canonical Cloud Development and Operations > http://launchpad.net/~sinzui > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
Hi Andrew. On Fri, Jan 24, 2014 at 8:40 PM, Andrew Wilkins wrote: > This looks a lot like a Trusty-specific issue to me, and nothing to do with > sudo. When did CI move to Trusty? CI is on Precise. I am on trusty. CI has not passed local upgrade to 1.17.1 and deploy with 1.17.1 tests since lp:juju-core r2248. The next revision tested was 2253 (a lot of revs landed while CI was retesting intermittent failures) CI sees this on when is uses 1.17.1 juju to boostrap: juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee bootstrap.log 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:77 Making /var/lib/jenkins/juju-ci/environments 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:67 Making /var/lib/jenkins/juju-ci/environments/local.jenv owned by 1000:1000 2014-01-25 14:51:09 ERROR juju.cmd supercommand.go:294 cannot create new info for environment "local": chown /var/lib/jenkins/juju-ci/environments/local.jenv: operation not permitted I suspect the moving back in forth between juju 1.17.1 and juju 1.16.5 is the reason that juju 1.16.5 can no long bootstrap. Juju and lxc are left in a bad state after attempting to use 1.17.1: juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee bootstrap.log 2014-01-25 14:59:53 ERROR juju.state open.go:93 TLS handshake failed: x509: certificate signed by unknown authority 2014-01-25 14:59:54 ERROR juju.agent agent.go:470 failed to initialize state: no reachable servers 2014-01-25 14:59:54 ERROR juju supercommand.go:282 no reachable servers > Trusty no longer has the /etc/lxc/auto directory, and instead the LXC > container config file has an attribute that says the container should be > auto-started with the machine. Jesse has an in-progress CL that addresses > this by checking if the auto directory exists before attempting to create > the auto-start symlink. Great. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
On Sat, Jan 25, 2014 at 11:17 PM, Curtis Hovey-Canonical < cur...@canonical.com> wrote: > Hi Andrew. > > On Fri, Jan 24, 2014 at 8:40 PM, Andrew Wilkins > wrote: > > This looks a lot like a Trusty-specific issue to me, and nothing to do > with > > sudo. When did CI move to Trusty? > > CI is on Precise. I am on trusty. > Okay. > CI has not passed local upgrade to 1.17.1 and deploy with 1.17.1 tests > since lp:juju-core r2248. The next revision tested was 2253 (a lot of > revs landed while CI was retesting intermittent failures) > > CI sees this on when is uses 1.17.1 juju to boostrap: > > juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee > bootstrap.log > 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:77 Making > /var/lib/jenkins/juju-ci/environments > 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:67 Making > /var/lib/jenkins/juju-ci/environments/local.jenv owned by 1000:1000 > 2014-01-25 14:51:09 ERROR juju.cmd supercommand.go:294 cannot create > new info for environment "local": chown > /var/lib/jenkins/juju-ci/environments/local.jenv: operation not > permitted > This looks like it's still running under sudo. Is that the case? (I see it's not in the command line, but is this a script that is run under sudo?) The "Making ... owned by 1000:1000" line shouldn't be there. I suspect the moving back in forth between juju 1.17.1 and juju 1.16.5 > is the reason that juju 1.16.5 can no long bootstrap. Juju and lxc are > left in a bad state after attempting to use 1.17.1: > > juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee > bootstrap.log > 2014-01-25 14:59:53 ERROR juju.state open.go:93 TLS handshake failed: > x509: certificate signed by unknown authority > 2014-01-25 14:59:54 ERROR juju.agent agent.go:470 failed to initialize > state: no reachable servers > 2014-01-25 14:59:54 ERROR juju supercommand.go:282 no reachable servers > > > Trusty no longer has the /etc/lxc/auto directory, and instead the LXC > > container config file has an attribute that says the container should be > > auto-started with the machine. Jesse has an in-progress CL that addresses > > this by checking if the auto directory exists before attempting to create > > the auto-start symlink. > > Great. > > -- > Curtis Hovey > Canonical Cloud Development and Operations > http://launchpad.net/~sinzui > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
On 26/01/14 19:28, Andrew Wilkins wrote: > On Sat, Jan 25, 2014 at 11:17 PM, Curtis Hovey-Canonical > mailto:cur...@canonical.com>> wrote: > CI has not passed local upgrade to 1.17.1 and deploy with 1.17.1 tests > since lp:juju-core r2248. The next revision tested was 2253 (a lot of > revs landed while CI was retesting intermittent failures) > > CI sees this on when is uses 1.17.1 juju to boostrap: > > juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee > bootstrap.log > 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:77 Making > /var/lib/jenkins/juju-ci/environments > 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:67 Making > /var/lib/jenkins/juju-ci/environments/local.jenv owned by 1000:1000 > 2014-01-25 14:51:09 ERROR juju.cmd supercommand.go:294 cannot create > new info for environment "local": chown > /var/lib/jenkins/juju-ci/environments/local.jenv: operation not > permitted > > > This looks like it's still running under sudo. Is that the case? (I see > it's not in the command line, but is this a script that is run under sudo?) > The "Making ... owned by 1000:1000" line shouldn't be there. OR... the env vars are set even though the user isn't using sudo, or the sudo has been used run the tarmac script as a different user. Whatever it is, we shouldn't need this any more. I'll submit a proposal Monday morning to clean this up. Tim -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
On 26/01/14 19:28, Andrew Wilkins wrote: > On Sat, Jan 25, 2014 at 11:17 PM, Curtis Hovey-Canonical > mailto:cur...@canonical.com>> wrote: > CI sees this on when is uses 1.17.1 juju to boostrap: > > juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee > bootstrap.log > 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:77 Making > /var/lib/jenkins/juju-ci/environments > 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:67 Making > /var/lib/jenkins/juju-ci/environments/local.jenv owned by 1000:1000 > 2014-01-25 14:51:09 ERROR juju.cmd supercommand.go:294 cannot create > new info for environment "local": chown > /var/lib/jenkins/juju-ci/environments/local.jenv: operation not > permitted > > > This looks like it's still running under sudo. Is that the case? (I see > it's not in the command line, but is this a script that is run under sudo?) > The "Making ... owned by 1000:1000" line shouldn't be there. Looking at the code a little more, and poking the command line, if the script uses sudo to run something as another user, we would see something like this, as the SUDO_UID and SUDO_GID are set but we are not running as root, so if the current user would not be able to chown to a different user. As mentioned before, this really isn't needed any more with the changes that landed last week. Tim -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
On 27/01/14 08:13, Tim Penhey wrote: > On 26/01/14 19:28, Andrew Wilkins wrote: >> On Sat, Jan 25, 2014 at 11:17 PM, Curtis Hovey-Canonical >> mailto:cur...@canonical.com>> wrote: >> CI sees this on when is uses 1.17.1 juju to boostrap: >> >> juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee >> bootstrap.log >> 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:77 Making >> /var/lib/jenkins/juju-ci/environments >> 2014-01-25 14:51:09 DEBUG juju.environs.configstore disk.go:67 Making >> /var/lib/jenkins/juju-ci/environments/local.jenv owned by 1000:1000 >> 2014-01-25 14:51:09 ERROR juju.cmd supercommand.go:294 cannot create >> new info for environment "local": chown >> /var/lib/jenkins/juju-ci/environments/local.jenv: operation not >> permitted >> >> >> This looks like it's still running under sudo. Is that the case? (I see >> it's not in the command line, but is this a script that is run under sudo?) >> The "Making ... owned by 1000:1000" line shouldn't be there. > > Looking at the code a little more, and poking the command line, if the > script uses sudo to run something as another user, we would see > something like this, as the SUDO_UID and SUDO_GID are set but we are not > running as root, so if the current user would not be able to chown to a > different user. > > As mentioned before, this really isn't needed any more with the changes > that landed last week. https://codereview.appspot.com/51150044/ I tested this locally as I have two different users on my laptop. Going: sudo su thumper I was then running as thumper but SUDO_GID was set. I was able to bootstrap a local environment as thumper. Tim -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
Juju 1.16.5 is run with sudo, juju 1.17.* is not. > juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee bootstrap.log ^ All tests are run as jenkins. When I manually ran this test I was jenkins via ubuntu using "sudo su". The juju was 1.17.1 made from r2265. Sudo will likely be run for some operation before running juju bootstrap. Jenkins is in the sudoers list so that it can setup the tests. In the case of an upgrade test, juju 1.16.5 is bootstrapped using sudo, then juju 1.17.1 is used to call upgrade-juju. juju 1.16.5 is always used to teardown tests; it always calls sudo. This might be an transitionary problem where the local jenv was created with 1.16.5? I add a rule to remove local.jenv before any local test starts, but the tests did not pass. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
On Mon, Jan 27, 2014 at 12:17 PM, Aaron Bentley wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 14-01-27 08:29 AM, Curtis Hovey-Canonical wrote: > > Juju 1.16.5 is run with sudo, juju 1.17.* is not. > > > >> juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee > >> bootstrap.log > > > > ^ All tests are run as jenkins. When I manually ran this test I > > was jenkins via ubuntu using "sudo su". The juju was 1.17.1 made > > from r2265. > > Don't do that. That will trigger known bugs. You have to ssh into > the machine as jenkins. Your orange key should let you do that > directly, but you can also do "sudo su jenkins" and then "ssh > jenkins@localhost". > Isn't this the same as doing sudo su - jenkins ? If not, what's the difference? > > Aaron > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.14 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAlLmamEACgkQ0F+nu1YWqI0R5gCfe64EDYQIm6naSaGLR4chcZwy > 9TUAn2BYn0md/RWKZl2UFmEj/s7M9vYe > =6dtU > -END PGP SIGNATURE- > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Diogo M. Matsubara -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14-01-27 09:24 AM, Diogo Matsubara wrote: > > > > On Mon, Jan 27, 2014 at 12:17 PM, Aaron Bentley > mailto:aaron.bent...@canonical.com>> > wrote: > > On 14-01-27 08:29 AM, Curtis Hovey-Canonical wrote: >> Juju 1.16.5 is run with sudo, juju 1.17.* is not. > >>> juju --debug bootstrap -e local --constraints mem=2G 2>&1 | >>> tee bootstrap.log > >> ^ All tests are run as jenkins. When I manually ran this test I >> was jenkins via ubuntu using "sudo su". The juju was 1.17.1 made >> from r2265. > > Don't do that. That will trigger known bugs. You have to ssh > into the machine as jenkins. Your orange key should let you do > that directly, but you can also do "sudo su jenkins" and then "ssh > jenkins@localhost". > > >> Isn't this the same as doing sudo su - jenkins ? If not, what's >> the difference? The difference is that because you're logged in to the instance directly as jenkins, not su'ed from ubuntu, juju won't try to chown your files to be owned by ubuntu. The second ssh obfuscates the original userid from Juju, because it's a new login. https://bugs.launchpad.net/juju-core/+bug/1245647 Aaron -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLmbWUACgkQ0F+nu1YWqI3cHgCcC+coR5chHegQjdPiEZ8C5FeO OawAn3lBRKEvf2IE6bGU+r+5cFok8qtP =eCso -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14-01-27 08:29 AM, Curtis Hovey-Canonical wrote: > Juju 1.16.5 is run with sudo, juju 1.17.* is not. > >> juju --debug bootstrap -e local --constraints mem=2G 2>&1 | tee >> bootstrap.log > > ^ All tests are run as jenkins. When I manually ran this test I > was jenkins via ubuntu using "sudo su". The juju was 1.17.1 made > from r2265. Don't do that. That will trigger known bugs. You have to ssh into the machine as jenkins. Your orange key should let you do that directly, but you can also do "sudo su jenkins" and then "ssh jenkins@localhost". Aaron -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLmamEACgkQ0F+nu1YWqI0R5gCfe64EDYQIm6naSaGLR4chcZwy 9TUAn2BYn0md/RWKZl2UFmEj/s7M9vYe =6dtU -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
sudo su - jenkins would work here and avoids triggering the ref'd issue, its giving a login shell without the SUDO env vars (ie shed the original identity) that juju was looking up to try to find the original user to chown. -k On Mon, Jan 27, 2014 at 9:29 AM, Aaron Bentley wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 14-01-27 09:24 AM, Diogo Matsubara wrote: > > > > > > > > On Mon, Jan 27, 2014 at 12:17 PM, Aaron Bentley > > mailto:aaron.bent...@canonical.com>> > > wrote: > > > > On 14-01-27 08:29 AM, Curtis Hovey-Canonical wrote: > >> Juju 1.16.5 is run with sudo, juju 1.17.* is not. > > > >>> juju --debug bootstrap -e local --constraints mem=2G 2>&1 | > >>> tee bootstrap.log > > > >> ^ All tests are run as jenkins. When I manually ran this test I > >> was jenkins via ubuntu using "sudo su". The juju was 1.17.1 made > >> from r2265. > > > > Don't do that. That will trigger known bugs. You have to ssh > > into the machine as jenkins. Your orange key should let you do > > that directly, but you can also do "sudo su jenkins" and then "ssh > > jenkins@localhost". > > > > > >> Isn't this the same as doing sudo su - jenkins ? If not, what's > >> the difference? > > The difference is that because you're logged in to the instance > directly as jenkins, not su'ed from ubuntu, juju won't try to chown > your files to be owned by ubuntu. The second ssh obfuscates the > original userid from Juju, because it's a new login. > > https://bugs.launchpad.net/juju-core/+bug/1245647 > > Aaron > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.14 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAlLmbWUACgkQ0F+nu1YWqI3cHgCcC+coR5chHegQjdPiEZ8C5FeO > OawAn3lBRKEvf2IE6bGU+r+5cFok8qtP > =eCso > -END PGP SIGNATURE- > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: lxc bug blocks 1.17.1 release.
On 28/01/14 08:19, Kapil Thangavelu wrote: > > sudo su - jenkins would work here and avoids triggering the ref'd issue, > its giving a login shell without the SUDO env vars (ie shed the original > identity) that juju was looking up to try to find the original user to > chown. FWIW the fix has landed in trunk and this shouldn't be an issue any more. I guessed that this was the case, and did the same here to test. Tim -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
