[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Merge proposal linked: https://code.launchpad.net/~kajiya/+git/google-guest-agent/+merge/472244 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
This bug was fixed in the package google-guest-agent - 20240716.00-0ubuntu1~20.04.0 --- google-guest-agent (20240716.00-0ubuntu1~20.04.0) focal; urgency=medium * No-change rebuild for Focal. google-guest-agent (20240716.00-0ubuntu1) oracular; urgency=medium [ Utkarsh Gupta ] * New upstream version 20240716.00. (LP: #2073163) * Update vendored dependencies. * Set DH_GOLANG_INSTALL_EXTRA to editions_defaults.binpb. [ Chloé Smith ] * d/p/0006-add-after-cloud-config-service.patch: add patch to force ordering between cloud-init and the guest agent. (LP: #2057965) -- Chloé 'kajiya' Smith Thu, 25 Jul 2024 20:01:29 +0100 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
This bug was fixed in the package google-guest-agent - 20240716.00-0ubuntu1~22.04.0 --- google-guest-agent (20240716.00-0ubuntu1~22.04.0) jammy; urgency=medium * No-change rebuild for Jammy. google-guest-agent (20240716.00-0ubuntu1) oracular; urgency=medium [ Utkarsh Gupta ] * New upstream version 20240716.00. (LP: #2073163) * Update vendored dependencies. * Set DH_GOLANG_INSTALL_EXTRA to editions_defaults.binpb. [ Chloé Smith ] * d/p/0006-add-after-cloud-config-service.patch: add patch to force ordering between cloud-init and the guest agent. (LP: #2057965) -- Chloé 'kajiya' Smith Fri, 26 Jul 2024 14:49:34 +0100 ** Changed in: google-guest-agent (Ubuntu Jammy) Status: Fix Committed => Fix Released ** Changed in: google-guest-agent (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
This bug was fixed in the package google-guest-agent - 20240716.00-0ubuntu1~24.04.0 --- google-guest-agent (20240716.00-0ubuntu1~24.04.0) noble; urgency=medium * No-change rebuild for Noble. google-guest-agent (20240716.00-0ubuntu1) oracular; urgency=medium [ Utkarsh Gupta ] * New upstream version 20240716.00. (LP: #2073163) * Update vendored dependencies. * Set DH_GOLANG_INSTALL_EXTRA to editions_defaults.binpb. [ Chloé Smith ] * d/p/0006-add-after-cloud-config-service.patch: add patch to force ordering between cloud-init and the guest agent. (LP: #2057965) -- Chloé 'kajiya' Smith Fri, 26 Jul 2024 15:07:50 +0100 ** Changed in: google-guest-agent (Ubuntu Noble) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
VALIDATION All images used below were built with snapshots from 20240306T030400Z and 20240510T030400Z for cloud-init and ubuntu-pro-client respectively. === FOCAL FOSSA === kajiya@little-HAL:~$ cat startup.sh #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list touch /home/ubuntu/hello.txt -- kajiya@little-HAL:~$ gcloud compute instances create focal-startup- bug-2057965 --image testing-ubuntu-guest-2004-focal-v20240823 --image- project ubuntu-kajiya --metadata-from-file=startup-script=startup.sh kajiya@little-HAL:~$ ssh ubuntu@35.204.59.206 -- ubuntu@focal-startup-bug-2057965:~$ pro --version 31.2.3~20.04 ubuntu@focal-startup-bug-2057965:~$ cloud-init --version /usr/bin/cloud-init 23.4.4-0ubuntu0~20.04.1 ubuntu@focal-startup-bug-2057965:~$ ls hello.txt ubuntu@focal-startup-bug-2057965:~$ diff /tmp/startup-sources.list /etc/apt/sources.list ubuntu@focal-startup-bug-2057965:~$ === JAMMY JELLYFISH === kajiya@little-HAL:~$ cat startup.sh #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list touch /home/ubuntu/hello.txt -- kajiya@little-HAL:~$ gcloud compute instances create jammy-startup- bug-2057965 --image testing-ubuntu-guest-2204-jammy-v20240823 --image- project ubuntu-kajiya --metadata-from-file=startup-script=startup.sh kajiya@little-HAL:~$ ssh ubuntu@34.90.192.236 -- ubuntu@jammy-startup-bug-2057965:~$ pro --version 31.2.3~22.04 ubuntu@jammy-startup-bug-2057965:~$ cloud-init --version /usr/bin/cloud-init 23.4.4-0ubuntu0~22.04.1 ubuntu@jammy-startup-bug-2057965:~$ ls hello.txt ubuntu@jammy-startup-bug-2057965:~$ diff /tmp/startup-sources.list /etc/apt/sources.list ubuntu@jammy-startup-bug-2057965:~$ === NOBLE NUMBAT === kajiya@little-HAL:~$ cat startup.sh #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list touch /home/ubuntu/hello.txt -- kajiya@little-HAL:~$ gcloud compute instances create noble-startup- bug-2057965 --image testing-ubuntu-guest-2404-noble-amd64-v20240823 --image-project ubuntu-kajiya --metadata-from-file=startup- script=startup.sh kajiya@little-HAL:~$ ssh ubuntu@35.204.72.82 -- ubuntu@noble-startup-bug-2057965:~$ ls hello.txt ubuntu@noble-startup-bug-2057965:~$ pro --version 31.2.3 ubuntu@noble-startup-bug-2057965:~$ cloud-init --version /usr/bin/cloud-init 24.1-0ubuntu1 ubuntu@noble-startup-bug-2057965:~$ diff /tmp/startup-sources.list /etc/apt/sources.list ubuntu@noble-startup-bug-2057965:~$ ubuntu@noble-startup-bug-2057965:~$ systemd-analyze critical-chain The time when unit became active or started is printed after the "@" character. The time the unit took to start is printed after the "+" character. graphical.target @23.527s └─multi-user.target @23.527s └─getty.target @23.527s └─getty@tty1.service @23.527s └─system-getty.slice @23.522s └─setvtrgb.service @23.462s +57ms └─plymouth-quit-wait.service @23.435s +11ms └─systemd-user-sessions.service @23.394s +34ms └─cloud-config.service @22.305s +1.080s <<< └─basic.target @11.709s └─sockets.target @11.700s └─snapd.socket @11.628s +59ms └─sysinit.target @11.517s └─cloud-init.service @8.942s +2.206s └─systemd-networkd-wait-online.service @8.893s +31ms └─systemd-networkd.service @8.692s +175ms └─network-pre.target @8.677s └─cloud-init-local.service @4.842s +3.822s └─systemd-remount-fs.service @2.192s +117ms └─systemd-fsck-root.service @1.917s +243ms └─systemd-journald.socket @1.643s └─-.mount @1.520s └─-.slice @1.520s ** Tags removed: verification-needed verification-needed-focal verification-needed-jammy verification-needed-noble ** Tags added: verification-done verification-done-focal verification-done-jammy verification-done-noble -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Merge proposal linked: https://code.launchpad.net/~kajiya/+git/google-guest-agent/+merge/471840 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Merge proposal linked: https://code.launchpad.net/~kajiya/+git/google-guest-agent/+merge/471839 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
Hello Catherine, or anyone else affected, Accepted google-guest-agent into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google- guest-agent/20240716.00-0ubuntu1~24.04.0 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-noble. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: google-guest-agent (Ubuntu Noble) Status: Confirmed => Fix Committed ** Tags added: verification-needed verification-needed-noble ** Changed in: google-guest-agent (Ubuntu Jammy) Status: New => Fix Committed ** Tags added: verification-needed-jammy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Merge proposal linked: https://code.launchpad.net/~kajiya/+git/google-guest-agent/+merge/470200 ** Merge proposal linked: https://code.launchpad.net/~kajiya/+git/google-guest-agent/+merge/470201 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
This bug was fixed in the package google-guest-agent - 20240716.00-0ubuntu1 --- google-guest-agent (20240716.00-0ubuntu1) oracular; urgency=medium [ Utkarsh Gupta ] * New upstream version 20240716.00. (LP: #2073163) * Update vendored dependencies. * Set DH_GOLANG_INSTALL_EXTRA to editions_defaults.binpb. [ Chloé Smith ] * d/p/0006-add-after-cloud-config-service.patch: add patch to force ordering between cloud-init and the guest agent. (LP: #2057965) -- Utkarsh Gupta Thu, 18 Jul 2024 23:47:17 +0530 ** Changed in: google-guest-agent (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
Ubuntu 23.10 (Mantic Minotaur) has reached end of life, so this bug will not be fixed for that specific release. ** Changed in: google-guest-agent (Ubuntu Mantic) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
> Why the patch in https://bugs.launchpad.net/ubuntu/+source/google- guest-agent/+bug/2057965/comments/5 and Impact description order gga after cloud-final.service but the merged (temporal) cpc_packaging.extra MRs order gga after cloud-config.service? cloud-final.service runs after cloud-config.service so either will work. I proposed cloud-final.service in my patch since prior to `debian/patches/0005-make-service-directive-explicit.patch` being added, that was the ordering constraint in place. The 0005 patch removes it due to redundancy, which has since clearly been changed but I thought it was cleaner to revert to the original ordering. Either ordering (cloud- config and cloud-final) appear to solve the issue in testing. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
Why the patch in https://bugs.launchpad.net/ubuntu/+source/google-guest- agent/+bug/2057965/comments/5 and Impact description order gga after cloud-final.service but the merged (temporal) cpc_packaging.extra MRs order gga after cloud-config.service? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Description changed: [ Impact ] In certain situations (consistently with ubuntu-pro=31.2 and cloud- init=23.4.4), cloud-config.service has not completed before google- startup-scripts.service runs. This can cause startup scripts that rely on apt to fail, as cloud-init is responsible for reconfiguring sources.list to point at the GCE archives. Since pro and cloud-init are backported to all older releases, this bug will affect them too. The change that results in this race condition is the removal an ordering condition between pro and cloud-init, so adding `After=cloud- final.service` to google-startup-scripts.service should ensure that the startup scripts are correctly run regardless of the ordering (or lack thereof) between other services. [ Test Plan ] To reproduce: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh [...] $ ssh [INSTANCE IP] > diff /tmp/startup-sources.list /etc/apt/sources.list 0a1,8 > ## Note, this file is written by cloud-init on first boot of an instance > ## modifications made here will not survive a re-bundle. > ## if you wish to make changes you can: > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg > ## or do the same in user-data > ## b.) add sources in /etc/apt/sources.list.d > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl > 3,4c11,12 < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted --- > deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted > # deb-src http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted 8,9c16,17 < deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted --- [...] - - Since this bug particularly effects first boot (once sources.list is configured with the GCE mirrors on first boot it will remain correctly configured), the best way to test that fix is correctly created will be to create an image with pro pinned at 31.2, cloud-init pinned at 23.4.4, and google-guest-agent install from proposed. The test would be: + Since this bug particularly effects first boot (once sources.list is + configured with the GCE mirrors on first boot it will remain correctly + configured), the best way to test that fix is correctly created will be + to create an image with pro pinned at 31.2, cloud-init pinned at 23.4.4, + and google-guest-agent install from proposed. The test would be: 1. Create an instance with startup script as above $ gcloud compute instances create startup-test --image [IMAGE_NAME] --image-project [IMAGE PROJECT] --metadata-from-file=startup-script=startup_script.sh 2. SSH into the instance and verify pro/cloud-init/google-guest-agent versions/source > pro --version 32.1~[RELEASE] > cloud-init --version /usr/bin/cloud-init 23.4.4-0ubuntu0~[RELEASE] > apt-cache policy google-guest-agent [ensure from -proposed] 3. Verify startup script ran correctly after cloud-config.service. > diff /tmp/startup-sources.list /etc/apt/sources.list > - [ Where problems could occur ] - #TODO STILL - - * Think about what the upload changes in the software. Imagine the change is -wrong or breaks something else: how would this show up? - - * It is assumed that any SRU candidate patch is well-tested before -upload and has a low overall risk of regression, but it's important -to make the effort to think about what ''could'' happen in the -event of a regression. - - * This must '''never''' be "None" or "Low", or entirely an argument as to why -your upload is low risk. - - * This both shows the SRU team that the risks have been considered, -and provides guidance to testers in regression-testing the SRU. + Since this introduces a new ordering constraint, it will likely have + performance impacts (google-startup-scripts will run later). This seems + preferable to breaking a subset of startup scripts in some situations; + it is not uncommon to use startup scripts to install packages so it's + important for the mirrors to be correctly configured. [ Other Info ] - + Original bug report retained below. New GCP dailies are failing startup-script tests, due to configuration via cloud-init not being fully completed, apt sources for example, when startup scripts are run. The failure can be reproduced as follows: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
The attachment "0006-order-startup-scripts-after-cloud-final.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Description changed: + [ Impact ] + + In certain situations (consistently with ubuntu-pro=31.2 and cloud- + init=23.4.4), cloud-config.service has not completed before google- + startup-scripts.service runs. This can cause startup scripts that rely + on apt to fail, as cloud-init is responsible for reconfiguring + sources.list to point at the GCE archives. + + Since pro and cloud-init are backported to all older releases, this bug + will affect them too. + + The change that results in this race condition is the removal an + ordering condition between pro and cloud-init, so adding `After=cloud- + final.service` to google-startup-scripts.service should ensure that the + startup scripts are correctly run regardless of the ordering (or lack + thereof) between other services. + + [ Test Plan ] + + To reproduce: + + Using startup_script.sh: + #!/bin/bash + cp /etc/apt/sources.list /tmp/startup-sources.list + + $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh + [...] + $ ssh [INSTANCE IP] + > diff /tmp/startup-sources.list /etc/apt/sources.list + 0a1,8 + > ## Note, this file is written by cloud-init on first boot of an instance + > ## modifications made here will not survive a re-bundle. + > ## if you wish to make changes you can: + > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg + > ## or do the same in user-data + > ## b.) add sources in /etc/apt/sources.list.d + > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl + > + 3,4c11,12 + < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted + < # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted + --- + > deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted + > # deb-src http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted + 8,9c16,17 + < deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted + < # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted + --- + [...] + + + Since this bug particularly effects first boot (once sources.list is configured with the GCE mirrors on first boot it will remain correctly configured), the best way to test that fix is correctly created will be to create an image with pro pinned at 31.2, cloud-init pinned at 23.4.4, and google-guest-agent install from proposed. The test would be: + + 1. Create an instance with startup script as above + $ gcloud compute instances create startup-test --image [IMAGE_NAME] --image-project [IMAGE PROJECT] --metadata-from-file=startup-script=startup_script.sh + + 2. SSH into the instance and verify pro/cloud-init/google-guest-agent versions/source + > pro --version + 32.1~[RELEASE] + > cloud-init --version + /usr/bin/cloud-init 23.4.4-0ubuntu0~[RELEASE] + > apt-cache policy google-guest-agent + [ensure from -proposed] + + 3. Verify startup script ran correctly after cloud-config.service. + > diff /tmp/startup-sources.list /etc/apt/sources.list + > + + + [ Where problems could occur ] + + #TODO STILL + + * Think about what the upload changes in the software. Imagine the change is +wrong or breaks something else: how would this show up? + + * It is assumed that any SRU candidate patch is well-tested before +upload and has a low overall risk of regression, but it's important +to make the effort to think about what ''could'' happen in the +event of a regression. + + * This must '''never''' be "None" or "Low", or entirely an argument as to why +your upload is low risk. + + * This both shows the SRU team that the risks have been considered, +and provides guidance to testers in regression-testing the SRU. + + [ Other Info ] + + Original bug report retained below. + New GCP dailies are failing startup-script tests, due to configuration via cloud-init not being fully completed, apt sources for example, when startup scripts are run. The failure can be reproduced as follows: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh [...] $ ssh [INSTANCE IP] > diff /tmp/startup-sources.list /etc/apt/sources.list 0a1,8 > ## Note, this file is written by cloud-init on first boot of an instance > ## modifications made here will not survive a re-bundle. > ## if you wish to make changes you can: > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg > ## or do the same in user-data > ## b.) add sources in /etc/apt/sources.list.d > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl > 3,4c11,12 < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted < # deb-src http://archiv
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
I believe the attached patch should fix this issue. ** Patch added: "0006-order-startup-scripts-after-cloud-final.patch" https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+attachment/5756329/+files/0006-order-startup-scripts-after-cloud-final.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Also affects: google-guest-agent (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: google-guest-agent (Ubuntu Noble) Importance: Undecided Status: Confirmed ** Also affects: google-guest-agent (Ubuntu Mantic) Importance: Undecided Status: New ** Also affects: google-guest-agent (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: google-guest-agent (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: google-guest-agent (Ubuntu Xenial) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
We are seeing similar failures across older images (mantic and earlier) but not in noble, perhaps because noble is using cloud-init 24.1 (which runs faster due to not waiting on snap seeding). However, even in noble, we don't see cloud-config.service in the critical chain: catred@startup-test-noble:~$ systemd-analyze critical-chain The time when unit became active or started is printed after the "@" characte> The time the unit took to start is printed after the "+" character. graphical.target @22.713s └─multi-user.target @22.713s └─snapd.seeded.service @19.697s +3.013s └─snapd.service @12.837s +6.842s └─basic.target @12.232s └─sockets.target @12.213s └─snapd.socket @12.098s +94ms └─sysinit.target @11.916s └─cloud-init.service @8.520s +2.561s └─systemd-networkd-wait-online.service @8.464s +20ms └─systemd-networkd.service @8.332s +82ms └─network-pre.target @8.300s └─cloud-init-local.service @4.820s +3.460s └─systemd-remount-fs.service @1.780s +92ms └─systemd-fsck-root.service @1.391s +245ms └─systemd-journald.socket @1.056s └─-.mount @763ms └─-.slice @762ms -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
@holmanb > google-startup-scripts.service starts well after the network is online You are correct. I have updated the bug description now. > due to configuration via cloud-init not being fully completed, apt sources for example, when startup scripts are run ** Description changed: - New GCP dailies are failing startup-script tests, due to network not - being fully set up when startup scripts are run. The failure can be - reproduced as follows: + New GCP dailies are failing startup-script tests, due to configuration + via cloud-init not being fully completed, apt sources for example, when + startup scripts are run. The failure can be reproduced as follows: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list - $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh [...] $ ssh [INSTANCE IP] > diff /tmp/startup-sources.list /etc/apt/sources.list 0a1,8 > ## Note, this file is written by cloud-init on first boot of an instance > ## modifications made here will not survive a re-bundle. > ## if you wish to make changes you can: > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg > ## or do the same in user-data > ## b.) add sources in /etc/apt/sources.list.d > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl - > + > 3,4c11,12 < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted --- > deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted > # deb-src http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted 8,9c16,17 < deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted --- [...] - - On earlier images (such as ubuntu-2204-jammy-v20240307 in ubuntu-os-cloud) do not show this behaviour. The change is due to a change in ubuntu-pro 31 (see https://github.com/canonical/ubuntu-pro-client/blob/dfe1f1ed4678c50240d4e251f41d33bb4034135e/debian/changelog#L40 for details) that removes a systemd ordering on cloud-config.service. As side effect of this change was the removal of cloud-config.service (and ubuntu-advantage.service) from systemd's critical chain. + On earlier images (such as ubuntu-2204-jammy-v20240307 in ubuntu-os- + cloud) do not show this behaviour. The change is due to a change in + ubuntu-pro 31 (see https://github.com/canonical/ubuntu-pro- + client/blob/dfe1f1ed4678c50240d4e251f41d33bb4034135e/debian/changelog#L40 + for details) that removes a systemd ordering on cloud-config.service. + As side effect of this change was the removal of cloud-config.service + (and ubuntu-advantage.service) from systemd's critical chain. On v20240307 (startup scripts execute correctly): catred@startup-test-control:~$ systemd-analyze critical-chain google-startup-scripts.service The time when unit became active or started is printed after the "@" character. The time the unit took to start is printed after the "+" character. google-startup-scripts.service +18.262s └─multi-user.target @28.480s - └─ubuntu-advantage.service @28.480s - └─cloud-config.service @27.372s +1.095s - └─snapd.seeded.service @20.048s +7.312s - └─snapd.service @12.469s +7.555s - └─basic.target @11.558s - └─sockets.target @11.540s - └─snap.lxd.daemon.unix.socket @24.376s - └─sysinit.target @10.825s - └─cloud-init.service @8.432s +2.267s - └─systemd-networkd-wait-online.service @6.467s +1.935s - └─systemd-networkd.service @6.347s +112ms - └─network-pre.target @6.328s - └─cloud-init-local.service @4.309s +2.006s - └─systemd-remount-fs.service @1.829s +68ms - └─systemd-fsck-root.service @1.587s +160ms - └─systemd-journald.socket @1.292s - └─system.slice @1.068s - └─-.slice @1.068s - + └─ubuntu-advantage.service @28.480s + └─cloud-config.service @27.372s +1.095s + └─snapd.seeded.service @20.048s +7.312s + └─snapd.service @12.469s +7.555s + └─basic.target @11.558s + └─sockets.target @11.540s + └─snap.lxd.daemon.unix.socket @24.376s + └─sysinit.target @10.825s + └─cloud-init.service @8.432s +2.267s + └─systemd-networkd-wait-online.service @6.467s +1.935s + └─systemd-networkd.service @6.347s +112ms + └─network-pre.target @6.328s + └─cloud-init-loc
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
> due to network not being fully set up when startup scripts are run. google-startup-scripts.service starts well after the network is online, so I think that the actual reason is that cc_apt_configure hasn't ran yet when the startup script starts. The diff looks like some apt mirrors configured by cloud-init, possibly via GCE's vendordata? > This can be fixed by adding an explict `After=cloud-config.service` to the google-startup-scripts.service file, which enforces the correct ordering between google-startup-scripts and cloud-init. +1 agreed, that should fix it. However if ssh login needs to be possible (which I do not know) when the script runs, then you may want to order after systemd-user-sessions.service instead (see this bug[1] for context). That all said, the second critical chain looks unusual - what took ~16.6s between sysinit.target and multi-user.target? [1] https://bugs.launchpad.net/uvtool/+bug/2039441 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: google-guest-agent (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
