[Bug 2059952] Re: pro sometimes runs before cloud-config.service
I reran the Xenial verification process and found that this proposed package does not change the behavior; that is, with the proposed package, pro services are still enable on first boot. Upon further investigation, I found that the version of cloud-init in xenial is 21.1-19-gbad84ad4-0ubuntu1~16.04.4 which does not contain the logic to drive the pro auto-attach process (which is where this bug exists). The details of the testing I did are shown below: $ gcloud compute instances create pro-order-bug --image testing- ubuntu-1604-xenial-v20240417 --image-project ubuntu-catred --metadata- from-file=user-data=userdata.yaml --zone us-central1-a catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client ubuntu-pro-client: Installed: 31.2.3~16.04 Candidate: 31.2.3~16.04 Version table: *** 31.2.3~16.04 100 100 /var/lib/dpkg/status 31.2.2~16.04 500 500 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages catred@pro-order-bug:~$ pro status SERVICE ENTITLED STATUS DESCRIPTION cc-eal yes disabled Common Criteria EAL2 Provisioning Packages cis yes disabled Security compliance and audit tools esm-apps yes enabled Expanded Security Maintenance for Applications esm-infrayes enabled Expanded Security Maintenance for Infrastructure livepatchyes warning Current kernel is not supported NOTICES The current kernel (4.15.0-1160-gcp, x86_64) is not supported by livepatch. Supported kernels are listed here: https://ubuntu.com/security/livepatch/docs/kernels Either switch to a supported kernel or `pro disable livepatch` to dismiss this warning. For a list of all Ubuntu Pro services, run 'pro status --all' Enable services with: pro enable Account: ubuntu-catred Subscription: ubuntu-catred Valid until: Fri Dec 31 00:00:00 UTC Technical support level: essential catred@pro-order-bug:~$ sudo apt-cache policy cloud-init cloud-init: Installed: 21.1-19-gbad84ad4-0ubuntu1~16.04.4 Candidate: 21.1-19-gbad84ad4-0ubuntu1~16.04.4 Version table: *** 21.1-19-gbad84ad4-0ubuntu1~16.04.4 510 510 https://esm.ubuntu.com/infra/ubuntu xenial-infra-security/main amd64 Packages 100 /var/lib/dpkg/status 21.1-19-gbad84ad4-0ubuntu1~16.04.2 500 500 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 0.7.7~bzr1212-0ubuntu1 500 500 http://us-central1.gce.archive.ubuntu.com/ubuntu xenial/main amd64 Packages -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
After some further investigation it seems that this bug does not occur in AWS (since the available pro image does not include the affected packages) and testing is not possible in Azure due to their image registration policies. Please let me know if the verification can be approved with solely the GCP results. ** Tags removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-jammy verification-needed- mantic verification-needed-xenial ** Tags added: verification-done-bionic verification-done-focal verification-done-jammy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
We do not publish GCP pro images for mantic so the bug does occur there, hence there is no validation for mantic GCP. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
Xenial Validation GCP: $ gcloud compute instances create pro-order-bug --image testing- ubuntu-1604-xenial-v20240417 --image-project ubuntu-catred --metadata- from-file=user-data=userdata.yaml --zone us-central1-a catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client ubuntu-pro-client: Installed: 31.2.3~16.04 Candidate: 31.2.3~16.04 Version table: *** 31.2.3~16.04 100 100 /var/lib/dpkg/status 31.2.2~16.04 500 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING' catred@pro-order-bug:~$ pro status SERVICE AVAILABLE DESCRIPTION cc-eal yesCommon Criteria EAL2 Provisioning Packages cis yesSecurity compliance and audit tools esm-apps yesExpanded Security Maintenance for Applications esm-infrayesExpanded Security Maintenance for Infrastructure fips yesNIST-certified FIPS crypto packages fips-updates yesFIPS compliant crypto packages with stable security updates livepatchyesCurrent kernel is not supported ros yesSecurity Updates for the Robot Operating System ros-updates yesAll Updates for the Robot Operating System NOTICES Operation in progress: pro.daemon.attempt_auto_attach For a list of all Ubuntu Pro services, run 'pro status --all' This machine is not attached to an Ubuntu Pro subscription. See https://ubuntu.com/pro Supported livepatch kernels are listed here: https://ubuntu.com/security/livepatch/docs/kernels No warnings from cloud-init; all services are disabled. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
Bionic Validation GCP: $ gcloud compute instances create pro-order-bug --image testing- ubuntu-1804-bionic-v20240417 --image-project ubuntu-catred --metadata- from-file=user-data=userdata.yaml --zone us-central1-a catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client ubuntu-pro-client: Installed: 31.2.3~18.04 Candidate: 31.2.3~18.04 Version table: *** 31.2.3~18.04 100 100 /var/lib/dpkg/status 31.2.2~18.04 500 500 http://us-central1.gce.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING' catred@pro-order-bug:~$ pro status SERVICE ENTITLED STATUS DESCRIPTION cc-eal yes disabled Common Criteria EAL2 Provisioning Packages cis yes disabled Security compliance and audit tools esm-apps yes disabled Expanded Security Maintenance for Applications esm-infrayes disabled Expanded Security Maintenance for Infrastructure fips yes disabled NIST-certified FIPS crypto packages fips-updates yes disabled FIPS compliant crypto packages with stable security updates livepatchyes disabled Canonical Livepatch service For a list of all Ubuntu Pro services, run 'pro status --all' Enable services with: pro enable Account: ubuntu-catred Subscription: ubuntu-catred Valid until: Fri Dec 31 00:00:00 UTC Technical support level: essential No warnings from cloud-init; all services are disabled. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
Focal Validation GCP: $ gcloud compute instances create pro-order-bug --image testing- ubuntu-2004-focal-v20240416 --image-project ubuntu-catred --metadata- from-file=user-data=userdata.yaml --zone us-central1-a catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client ubuntu-pro-client: Installed: 31.2.3~20.04 Candidate: 31.2.3~20.04 Version table: *** 31.2.3~20.04 100 100 /var/lib/dpkg/status 31.2.2~20.04 500 500 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING' catred@pro-order-bug:~$ pro status SERVICE ENTITLED STATUS DESCRIPTION anbox-cloud yes disabled Scalable Android in the cloud esm-apps yes disabled Expanded Security Maintenance for Applications esm-infrayes disabled Expanded Security Maintenance for Infrastructure fips yes disabled NIST-certified FIPS crypto packages fips-updates yes disabled FIPS compliant crypto packages with stable security updates livepatchyes disabled Canonical Livepatch service usg yes disabled Security compliance and audit tools For a list of all Ubuntu Pro services, run 'pro status --all' Enable services with: pro enable Account: ubuntu-catred Subscription: ubuntu-catred Valid until: Fri Dec 31 00:00:00 UTC Technical support level: essential No warnings from cloud-init; all services are disabled. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
Jammy Validation GCP: $ gcloud compute instances create pro-order-bug --image testing- ubuntu-2204-jammy-v20240416 --image-project ubuntu-catred --metadata- from-file=user-data=userdata.yaml --zone us-central1-a catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client ubuntu-pro-client: Installed: 31.2.3~22.04 Candidate: 31.2.3~22.04 Version table: *** 31.2.3~22.04 100 100 /var/lib/dpkg/status 31.2.2~22.04 500 (phased 40%) 500 http://us-central1.gce.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING' catred@pro-order-bug:~$ pro status SERVICE ENTITLED STATUS DESCRIPTION anbox-cloud yes disabled Scalable Android in the cloud esm-apps yes disabled Expanded Security Maintenance for Applications esm-infrayes disabled Expanded Security Maintenance for Infrastructure fips-preview yes disabled Preview of FIPS crypto packages undergoing certification with NIST fips-updates yes disabled FIPS compliant crypto packages with stable security updates livepatchyes disabled Canonical Livepatch service usg yes disabled Security compliance and audit tools For a list of all Ubuntu Pro services, run 'pro status --all' Enable services with: pro enable Account: ubuntu-catred Subscription: ubuntu-catred Valid until: Fri Dec 31 00:00:00 UTC Technical support level: essential No warnings from cloud-init; all services are disabled. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
** Description changed: [ Impact ] Currently, the Pro client support a daemon named ubuntu-advantage.service that performs two actions: * Actively look for Pro licenses on Azure and GCP images to perform an auto-attach * Retry auto-attach on Pro images if that command fails on boot Therefore, this daemon is only being activated on generic Azure and GCP images and all Pro cloud images. This daemon was originally setup to run after the cloud-config.service. However, due to a race condition, this is no longer happening. Right now, we manually check in the daemon code to see if the cloud-config service has finished. Unfortunately, this new logic now breaks the current Pro setup through cloud-init userdata in both GCP and Azure Pro cloud images. That is because our daemon is now running before cloud-init has even started running. This means that the daemon will perform the attach and not cloud-init itself. This will be clearer, in the following example: Let's imagine this situation where a user is launching a Pro GCP image: 1) User provides the following cloud-init userdata to the cloud image before booting it: #cloud-config ubuntu_advantage: enable: [] This means that the user wants no services to be enabled, but still want to attach to the Pro license. 2) Our daemon starts running before cloud-config.service has even started 3) Our daemon see the cloud-config.service as inactive and proceeds normally 4) Our daemon identifies that the user is running on a GCP instance and there is a valid Pro license for it. 5) Due to that, our daemon auto-attach the machine completely ignoring the cloud-init directives. Therefore, to fix that issue we need to guarantee that we will only execute the daemon, if and only if, cloud-init has already started. That is because, on this situation, the cloud-config.service will already perform the attach operation following the user directives. When the daemon starts running, it will see that the image is already attached and do nothing. Finally, given this scenario, this bug is only affecting GCP/Azure Pro images, as these are the only ones that will be able to reach the flow described here. [Discussion] To address that issue, we are now also checking if the cloud-init service has already started if we detect that cloud-config service is inactive. If it isn't, the daemon will sleep for an specific amount of time before trying again. [ Test Plan ] Since this is a first boot issue, we will need to create a custom image with the package in proposed. Then, we need to guarantee that Pro configuration delivered through cloud-init is being honored when we launch the image. Additionally, it is worth noting that we cannot reproduce this issue on a VM easily. That is because, we would need "mock" the VM to pass as one of the affected clouds and also add a valid Pro license to it. - However, CPC is already aware of this issue and will help us creating - the test plan here. + Build image that pulls pro from -proposed but otherwise follows the + standard pro image build hook. Upload and register the image with the + cloud for testing. + + #Set cloud-init userdata that disables all pro services + $ cat userdata.yaml + #cloud-config + + ubuntu_advantage: + enable: [] + + #Instantiate VM (GCP) + $ gcloud compute instances create pro-order-bug-mantic --image [IMAGE_NAME] --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a + + #Instantiate VM (Azure) + [TODO] + + #On VM, validate version of pro and bugfix (services disable, no cloud-init warnings in log) + $ apt-cache policy ubuntu-pro-client + ubuntu-pro-client: + Installed: 31.2.3~[RELEASE] + Candidate: 31.2.3~[RELEASE] + + $ cat /var/log/cloud-init.log | grep 'WARNING' + + $ pro status + SERVICE AVAILABLE DESCRIPTION + anbox-cloud yesScalable Android in the cloud + esm-apps yesExpanded Security Maintenance for Applications + esm-infrayesExpanded Security Maintenance for Infrastructure + landscapeyesManagement and administration tool for Ubuntu + livepatchyesCurrent kernel is not supported + + For a list of all Ubuntu Pro services, run 'pro status --all' + + This machine is not attached to an Ubuntu Pro subscription. + See https://ubuntu.com/pro + + Supported livepatch kernels are listed here: + https://ubuntu.com/security/livepatch/docs/kernels + + + If the bug is still present, there will be a WARNING in the cloud-init log and pro status will return something similar to: + SERVICE ENTITLED STATUS DESCRIPTION + anbox-cloud yes disabled Scalable Android in the cloud + esm-apps yes enabled Expanded Security Maintenance for Applications + esm-infrayes enabled Expanded Security
[Bug 2059952] Re: pro sometimes runs before cloud-config.service
Test plan details: Build image that pulls pro from -proposed. If necessary, I can expand on the exact bartender command/changes made. Upload and register the image with a cloud (GCP will be used for testing since that was where I first observed the bug and could reliably reproduce). #Instantiate VM $ cat userdata.yaml #cloud-config ubuntu_advantage: enable: [] $ gcloud compute instances create pro-order-bug-mantic --image [IMAGE_NAME] --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a #On VM, validate version of pro and bugfix (services disable, no cloud-init warnings in log) $ apt-cache policy ubuntu-pro-client $ cat /var/log/cloud-init.log | grep 'WARNING' $ pro status -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059952 Title: pro sometimes runs before cloud-config.service To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2059952/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2060578] Re: postfix installed during release upgrade to Noble
We also see this bug running do-release-upgrade on GCE base VMs. Prior to running the upgrade and after running the upgrade, postfix is not installed: catred@jammy-to-noble-upgrade-arm64:~$ dpkg --no-pager --list postfix dpkg-query: no packages found matching postfix If I create a second SSH shell during the upgrade while paused on the postfix prompt, I see: catred@jammy-to-noble-upgrade-base-amd64:~$ dpkg --no-pager --list postfix Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==---= un postfix (no description available) The prompt I get is: Package configuration ┌┤ Postfix Configuration ├┐ │ Please select the mail server configuration type that best meets your │ │ needs. │ │ │ │ No configuration: │ │ Should be chosen to leave the current configuration unchanged.│ │ Internet site: │ │ Mail is sent and received directly using SMTP.│ │ Internet with smarthost: │ │ Mail is received directly using SMTP or by running a utility such │ │ as fetchmail. Outgoing mail is sent using a smarthost.│ │ Satellite system: │ │ All mail is sent to another machine, called a 'smarthost', for│ │ delivery. │ │ Local only:│ │ The only delivered mail is the mail for local users. There is no │ │ network. │ │ │ │ General mail configuration type:│ │ │ │No configuration │ │Internet Site│ │Internet with smarthost │ │Satellite system │ │Local only │ │ │ │ │ │ │ │ │ └─┘ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2060578 Title: postfix installed during release upgrade to Noble To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/2060578/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2060558] Re: AppArmor profile `unconfined_restrictions` missing for noble 6.8 kernel
** Changed in: livecd-rootfs (Ubuntu Noble) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2060558 Title: AppArmor profile `unconfined_restrictions` missing for noble 6.8 kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2060558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2052789] Re: AppArmor profiles missing in kernel 5.15.0-1051+ release
Patch for updating the changelog to cover the added function as well as the new apparmor directory ** Patch added: "LP2059730.patch" https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2052789/+attachment/5760382/+files/LP2059730.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2052789 Title: AppArmor profiles missing in kernel 5.15.0-1051+ release To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2052789/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2059730] Re: Focal 5.15 kernel apparmor mismatch missing snap_validate_preseed functionality in livecd-rootfs
Patch for updating the changelog to cover the added function as well as the new apparmor directory ** Patch added: "LP2059730.patch" https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2059730/+attachment/5760381/+files/LP2059730.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059730 Title: Focal 5.15 kernel apparmor mismatch missing snap_validate_preseed functionality in livecd-rootfs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2059730/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
> Why the patch in https://bugs.launchpad.net/ubuntu/+source/google- guest-agent/+bug/2057965/comments/5 and Impact description order gga after cloud-final.service but the merged (temporal) cpc_packaging.extra MRs order gga after cloud-config.service? cloud-final.service runs after cloud-config.service so either will work. I proposed cloud-final.service in my patch since prior to `debian/patches/0005-make-service-directive-explicit.patch` being added, that was the ordering constraint in place. The 0005 patch removes it due to redundancy, which has since clearly been changed but I thought it was cleaner to revert to the original ordering. Either ordering (cloud- config and cloud-final) appear to solve the issue in testing. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Description changed: [ Impact ] In certain situations (consistently with ubuntu-pro=31.2 and cloud- init=23.4.4), cloud-config.service has not completed before google- startup-scripts.service runs. This can cause startup scripts that rely on apt to fail, as cloud-init is responsible for reconfiguring sources.list to point at the GCE archives. Since pro and cloud-init are backported to all older releases, this bug will affect them too. The change that results in this race condition is the removal an ordering condition between pro and cloud-init, so adding `After=cloud- final.service` to google-startup-scripts.service should ensure that the startup scripts are correctly run regardless of the ordering (or lack thereof) between other services. [ Test Plan ] To reproduce: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh [...] $ ssh [INSTANCE IP] > diff /tmp/startup-sources.list /etc/apt/sources.list 0a1,8 > ## Note, this file is written by cloud-init on first boot of an instance > ## modifications made here will not survive a re-bundle. > ## if you wish to make changes you can: > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg > ## or do the same in user-data > ## b.) add sources in /etc/apt/sources.list.d > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl > 3,4c11,12 < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted --- > deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted > # deb-src http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted 8,9c16,17 < deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted --- [...] - - Since this bug particularly effects first boot (once sources.list is configured with the GCE mirrors on first boot it will remain correctly configured), the best way to test that fix is correctly created will be to create an image with pro pinned at 31.2, cloud-init pinned at 23.4.4, and google-guest-agent install from proposed. The test would be: + Since this bug particularly effects first boot (once sources.list is + configured with the GCE mirrors on first boot it will remain correctly + configured), the best way to test that fix is correctly created will be + to create an image with pro pinned at 31.2, cloud-init pinned at 23.4.4, + and google-guest-agent install from proposed. The test would be: 1. Create an instance with startup script as above $ gcloud compute instances create startup-test --image [IMAGE_NAME] --image-project [IMAGE PROJECT] --metadata-from-file=startup-script=startup_script.sh 2. SSH into the instance and verify pro/cloud-init/google-guest-agent versions/source > pro --version 32.1~[RELEASE] > cloud-init --version /usr/bin/cloud-init 23.4.4-0ubuntu0~[RELEASE] > apt-cache policy google-guest-agent [ensure from -proposed] 3. Verify startup script ran correctly after cloud-config.service. > diff /tmp/startup-sources.list /etc/apt/sources.list > - [ Where problems could occur ] - #TODO STILL - - * Think about what the upload changes in the software. Imagine the change is -wrong or breaks something else: how would this show up? - - * It is assumed that any SRU candidate patch is well-tested before -upload and has a low overall risk of regression, but it's important -to make the effort to think about what ''could'' happen in the -event of a regression. - - * This must '''never''' be "None" or "Low", or entirely an argument as to why -your upload is low risk. - - * This both shows the SRU team that the risks have been considered, -and provides guidance to testers in regression-testing the SRU. + Since this introduces a new ordering constraint, it will likely have + performance impacts (google-startup-scripts will run later). This seems + preferable to breaking a subset of startup scripts in some situations; + it is not uncommon to use startup scripts to install packages so it's + important for the mirrors to be correctly configured. [ Other Info ] - + Original bug report retained below. New GCP dailies are failing startup-script tests, due to configuration via cloud-init not being fully completed, apt sources for example, when startup scripts are run. The failure can be reproduced as follows: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
** Description changed: + [ Impact ] + + In certain situations (consistently with ubuntu-pro=31.2 and cloud- + init=23.4.4), cloud-config.service has not completed before google- + startup-scripts.service runs. This can cause startup scripts that rely + on apt to fail, as cloud-init is responsible for reconfiguring + sources.list to point at the GCE archives. + + Since pro and cloud-init are backported to all older releases, this bug + will affect them too. + + The change that results in this race condition is the removal an + ordering condition between pro and cloud-init, so adding `After=cloud- + final.service` to google-startup-scripts.service should ensure that the + startup scripts are correctly run regardless of the ordering (or lack + thereof) between other services. + + [ Test Plan ] + + To reproduce: + + Using startup_script.sh: + #!/bin/bash + cp /etc/apt/sources.list /tmp/startup-sources.list + + $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh + [...] + $ ssh [INSTANCE IP] + > diff /tmp/startup-sources.list /etc/apt/sources.list + 0a1,8 + > ## Note, this file is written by cloud-init on first boot of an instance + > ## modifications made here will not survive a re-bundle. + > ## if you wish to make changes you can: + > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg + > ## or do the same in user-data + > ## b.) add sources in /etc/apt/sources.list.d + > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl + > + 3,4c11,12 + < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted + < # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted + --- + > deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted + > # deb-src http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted + 8,9c16,17 + < deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted + < # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted + --- + [...] + + + Since this bug particularly effects first boot (once sources.list is configured with the GCE mirrors on first boot it will remain correctly configured), the best way to test that fix is correctly created will be to create an image with pro pinned at 31.2, cloud-init pinned at 23.4.4, and google-guest-agent install from proposed. The test would be: + + 1. Create an instance with startup script as above + $ gcloud compute instances create startup-test --image [IMAGE_NAME] --image-project [IMAGE PROJECT] --metadata-from-file=startup-script=startup_script.sh + + 2. SSH into the instance and verify pro/cloud-init/google-guest-agent versions/source + > pro --version + 32.1~[RELEASE] + > cloud-init --version + /usr/bin/cloud-init 23.4.4-0ubuntu0~[RELEASE] + > apt-cache policy google-guest-agent + [ensure from -proposed] + + 3. Verify startup script ran correctly after cloud-config.service. + > diff /tmp/startup-sources.list /etc/apt/sources.list + > + + + [ Where problems could occur ] + + #TODO STILL + + * Think about what the upload changes in the software. Imagine the change is +wrong or breaks something else: how would this show up? + + * It is assumed that any SRU candidate patch is well-tested before +upload and has a low overall risk of regression, but it's important +to make the effort to think about what ''could'' happen in the +event of a regression. + + * This must '''never''' be "None" or "Low", or entirely an argument as to why +your upload is low risk. + + * This both shows the SRU team that the risks have been considered, +and provides guidance to testers in regression-testing the SRU. + + [ Other Info ] + + Original bug report retained below. + New GCP dailies are failing startup-script tests, due to configuration via cloud-init not being fully completed, apt sources for example, when startup scripts are run. The failure can be reproduced as follows: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh [...] $ ssh [INSTANCE IP] > diff /tmp/startup-sources.list /etc/apt/sources.list 0a1,8 > ## Note, this file is written by cloud-init on first boot of an instance > ## modifications made here will not survive a re-bundle. > ## if you wish to make changes you can: > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg > ## or do the same in user-data > ## b.) add sources in /etc/apt/sources.list.d > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl > 3,4c11,12 < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted < # deb-src
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
I believe the attached patch should fix this issue. ** Patch added: "0006-order-startup-scripts-after-cloud-final.patch" https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+attachment/5756329/+files/0006-order-startup-scripts-after-cloud-final.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] Re: google-startup-scripts runs before cloud-init finished network setup
We are seeing similar failures across older images (mantic and earlier) but not in noble, perhaps because noble is using cloud-init 24.1 (which runs faster due to not waiting on snap seeding). However, even in noble, we don't see cloud-config.service in the critical chain: catred@startup-test-noble:~$ systemd-analyze critical-chain The time when unit became active or started is printed after the "@" characte> The time the unit took to start is printed after the "+" character. graphical.target @22.713s └─multi-user.target @22.713s └─snapd.seeded.service @19.697s +3.013s └─snapd.service @12.837s +6.842s └─basic.target @12.232s └─sockets.target @12.213s └─snapd.socket @12.098s +94ms └─sysinit.target @11.916s └─cloud-init.service @8.520s +2.561s └─systemd-networkd-wait-online.service @8.464s +20ms └─systemd-networkd.service @8.332s +82ms └─network-pre.target @8.300s └─cloud-init-local.service @4.820s +3.460s └─systemd-remount-fs.service @1.780s +92ms └─systemd-fsck-root.service @1.391s +245ms └─systemd-journald.socket @1.056s └─-.mount @763ms └─-.slice @762ms -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2057965] [NEW] google-startup-scripts runs before cloud-init finished network setup
Public bug reported: New GCP dailies are failing startup-script tests, due to network not being fully set up when startup scripts are run. The failure can be reproduced as follows: Using startup_script.sh: #!/bin/bash cp /etc/apt/sources.list /tmp/startup-sources.list $ gcloud compute instances create startup-test --image daily-ubuntu-2204-jammy-v20240314 --image-project ubuntu-os-cloud-devel --metadata-from-file=startup-script=startup_script.sh [...] $ ssh [INSTANCE IP] > diff /tmp/startup-sources.list /etc/apt/sources.list 0a1,8 > ## Note, this file is written by cloud-init on first boot of an instance > ## modifications made here will not survive a re-bundle. > ## if you wish to make changes you can: > ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg > ## or do the same in user-data > ## b.) add sources in /etc/apt/sources.list.d > ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl > 3,4c11,12 < deb http://archive.ubuntu.com/ubuntu/ jammy main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted --- > deb http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main restricted > # deb-src http://us-central1.gce.archive.ubuntu.com/ubuntu/ jammy main > restricted 8,9c16,17 < deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted < # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted --- [...] On earlier images (such as ubuntu-2204-jammy-v20240307 in ubuntu-os-cloud) do not show this behaviour. The change is due to a change in ubuntu-pro 31 (see https://github.com/canonical/ubuntu-pro-client/blob/dfe1f1ed4678c50240d4e251f41d33bb4034135e/debian/changelog#L40 for details) that removes a systemd ordering on cloud-config.service. As side effect of this change was the removal of cloud-config.service (and ubuntu-advantage.service) from systemd's critical chain. On v20240307 (startup scripts execute correctly): catred@startup-test-control:~$ systemd-analyze critical-chain google-startup-scripts.service The time when unit became active or started is printed after the "@" character. The time the unit took to start is printed after the "+" character. google-startup-scripts.service +18.262s └─multi-user.target @28.480s └─ubuntu-advantage.service @28.480s └─cloud-config.service @27.372s +1.095s └─snapd.seeded.service @20.048s +7.312s └─snapd.service @12.469s +7.555s └─basic.target @11.558s └─sockets.target @11.540s └─snap.lxd.daemon.unix.socket @24.376s └─sysinit.target @10.825s └─cloud-init.service @8.432s +2.267s └─systemd-networkd-wait-online.service @6.467s +1.935s └─systemd-networkd.service @6.347s +112ms └─network-pre.target @6.328s └─cloud-init-local.service @4.309s +2.006s └─systemd-remount-fs.service @1.829s +68ms └─systemd-fsck-root.service @1.587s +160ms └─systemd-journald.socket @1.292s └─system.slice @1.068s └─-.slice @1.068s On v20240314 (startup scripts fail): catred@startup-test:~$ systemd-analyze critical-chain google-startup-scripts.service The time when unit became active or started is printed after the "@" characte> The time the unit took to start is printed after the "+" character. google-startup-scripts.service +260ms └─multi-user.target @29.237s └─chrony.service @30.240s +56ms └─basic.target @13.364s └─sockets.target @13.225s └─snap.lxd.user-daemon.unix.socket @26.765s └─sysinit.target @12.550s └─cloud-init.service @7.933s +4.503s └─systemd-networkd-wait-online.service @6.741s +1.171s └─systemd-networkd.service @6.593s +124ms └─network-pre.target @6.573s └─cloud-init-local.service @4.478s +2.083s └─systemd-remount-fs.service @1.717s +64ms └─systemd-fsck-root.service @1.510s +95ms └─systemd-journald.socket @1.193s └─-.mount @974ms └─-.slice @974ms This can be fixed by adding an explict `After=cloud-config.service` to the google-startup-scripts.service file, which enforces the correct ordering between google-startup-scripts and cloud-init. ** Affects: google-guest-agent (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2057965 Title: google-startup-scripts runs before cloud-init finished network setup To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2057965/+subscriptions --
