Bug#1040979: write_files: defer not working any more
Am 14.07.2023 um 23:46 schrieb Noah Meyerhans:
Can you show that write_files works without the defer flag?
Writing files without defer works.
2023-07-15 10:57:15,044 - modules.py[DEBUG]: Running module write-files
('/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py'>)
with frequency once-per-instance
2023-07-15 10:57:15,045 - handlers.py[DEBUG]: start:
init-network/config-write-files: running config-write-files with
frequency once-per-instance
2023-07-15 10:57:15,045 - util.py[DEBUG]: Writing to
/var/lib/cloud/instances//sem/config_write_files - wb: [644] 24 bytes
2023-07-15 10:57:15,045 - helpers.py[DEBUG]: Running config-write-files
using lock ('/var/lib/cloud/instances//sem/config_write_files'>)
2023-07-15 10:57:15,045 - util.py[DEBUG]: Writing to /test - wb: [644]
117 bytes
2023-07-15 10:57:15,045 - util.py[DEBUG]: Changing the ownership of
/test to 0:0
2023-07-15 10:57:15,046 - handlers.py[DEBUG]: finish:
init-network/config-write-files: SUCCESS: config-write-files ran
successfully
Also, I'm curious to see what happens if you invoke the
write_files_deferred module by hand with a command like
$ sudo cloud-init single --name cc_write_files_deferred \
--frequency always --report
# cloud-init single --name cc_write_files_deferred \
--frequency always --report
Cloud-init v. 22.4.2 running 'single' at Sat, 15 Jul 2023 10:59:34
+. Up 146.09 seconds.
After that the files marked as defer are created.
It should generate messages in /var/log/cloud-init.log.
2023-07-15 10:59:34,786 - util.py[DEBUG]: Cloud-init v. 22.4.2 running
'single' at Sat, 15 Jul 2023 10:59:34 +. Up 146.09 seconds.
2023-07-15 10:59:34,787 - util.py[DEBUG]: Reading from
/usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json
(quiet=False)
2023-07-15 10:59:34,788 - util.py[DEBUG]: Read 133834 bytes from
/usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json
2023-07-15 10:59:34,790 - stages.py[DEBUG]: Using distro class 'cloudinit.distros.debian.Distro'>
2023-07-15 10:59:34,790 - modules.py[DEBUG]: Running module
cc_write_files_deferred ('cloudinit.config.cc_write_files_deferred' from
'/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files_deferred.py'>)
with frequency always
2023-07-15 10:59:34,790 - handlers.py[DEBUG]: start:
single/cc_write_files_deferred/config-cc_write_files_deferred: running
config-cc_write_files_deferred with frequency always
2023-07-15 10:59:34,790 - helpers.py[DEBUG]: Running
config-cc_write_files_deferred using lock (object at 0x7f6fad0c3590>)
2023-07-15 10:59:34,790 - util.py[DEBUG]: Writing to /test-defer - wb:
[644] 41 bytes
2023-07-15 10:59:34,791 - util.py[DEBUG]: Changing the ownership of
/51-79p.cnf to 0:0
2023-07-15 10:59:34,791 - handlers.py[DEBUG]: finish:
single/cc_write_files_deferred/config-cc_write_files_deferred: SUCCESS:
config-cc_write_files_deferred ran successfully
2023-07-15 10:59:34,791 - util.py[DEBUG]: Reading from /proc/uptime
(quiet=False)
2023-07-15 10:59:34,791 - util.py[DEBUG]: Read 14 bytes from /proc/uptime
2023-07-15 10:59:34,791 - util.py[DEBUG]: cloud-init mode 'single' took
0.070 seconds (0.07)
2023-07-15 10:59:34,791 - handlers.py[DEBUG]: finish:
single/cc_write_files_deferred: SUCCESS: running single module
cc_write_files_deferred
Best,
Sven
Bug#1040979: write_files: defer not working any more
Also, I'm curious to see what happens if you invoke the write_files_deferred module by hand with a command like $ sudo cloud-init single --name cc_write_files_deferred \ --frequency always --report It should generate messages in /var/log/cloud-init.log.
Bug#1040979: write_files: defer not working any more
On 7/14/23 06:49, Sven Strickroth wrote:
I'm not able to reproduce the behavior you describe. defer seems to
work as expected. For example:
admin@ip-10-0-0-87:~$ ec2-metadata --ami-id
ami-id: ami-0544719b13af6edc3
admin@ip-10-0-0-87:~$ sudo cat /var/lib/cloud/instance/user-data.txt
#cloud-config
write_files:
- content: |
testing. this file should be owned by the dynamically created user
path: /test-file
owner: admin:admin
defer: true
admin@ip-10-0-0-87:~$ ls -l /test-file
-rw-r--r-- 1 admin admin 67 Jul 13 16:59 /test-file
That example does not work for me on Hetzner.cloud.
Strange. The functionality works as expected for me using the current
bookworm AMIs on Amazon EC2, as shown in the output below. I don't have
an account with Hetzner, so I'm afraid you're going to have to collect
some more information.
Can you show that the "defer" field in your cloud-config is making
through to the input that's passed to cloud-init?
Can you show that write_files works without the defer flag?
The write_files modules are defined to run once per instance. Is it
possible that your launches are not showing up as new instances,
resulting in cloud-init skipping their execution?
noah
admin@ip-10-0-0-199:~$ sudo cat /var/lib/cloud/instance/user-data.txt
#cloud-config
locale: de_DE.UTF-8
timezone: Europe/Berlin
package_update: true # Implied with `package_upgrade: true`
package_upgrade: true
packages:
['joe','mariadb-server','aptitude','psmisc','bzip2','fail2ban','postfix','apticron','nagios-nrpe-server','monitoring-plugins','libmariadb3','borgbackup','logwatch']
write_files:
- content: |
[mysqldump]
max_allowed_packet = 60M
path: /etc/mysql/mariadb.conf.d/51-dump.cnf
defer: true
admin@ip-10-0-0-199:~$ cat /etc/mysql/mariadb.conf.d/51-dump.cnf
[mysqldump]
max_allowed_packet = 60M
admin@ip-10-0-0-199:~$ grep write-files-deferred /var/log/cloud-init.log
2023-07-14 21:28:41,034 - modules.py[DEBUG]: Running module
write-files-deferred (from
'/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files_deferred.py'>)
with frequency once-per-instance
2023-07-14 21:28:41,034 - handlers.py[DEBUG]: start:
modules-final/config-write-files-deferred: running
config-write-files-deferred with frequency once-per-instance
2023-07-14 21:28:41,036 - helpers.py[DEBUG]: Running
config-write-files-deferred using lock ('/var/lib/cloud/instances/i-096c9d6b1c3c04be6/sem/config_write_files_deferred'>)
2023-07-14 21:28:41,036 - handlers.py[DEBUG]: finish:
modules-final/config-write-files-deferred: SUCCESS:
config-write-files-deferred ran successfully
Bug#1040979: write_files: defer not working any more
Am 13.07.2023 um 19:07 schrieb Noah Meyerhans: I'm not able to reproduce the behavior you describe. defer seems to work as expected. For example: admin@ip-10-0-0-87:~$ ec2-metadata --ami-id ami-id: ami-0544719b13af6edc3 admin@ip-10-0-0-87:~$ sudo cat /var/lib/cloud/instance/user-data.txt #cloud-config write_files: - content: | testing. this file should be owned by the dynamically created user path: /test-file owner: admin:admin defer: true admin@ip-10-0-0-87:~$ ls -l /test-file -rw-r--r-- 1 admin admin 67 Jul 13 16:59 /test-file That example does not work for me on Hetzner.cloud. I note that your cloud-config attempts to set the locale: #cloud-config locale: de_DE.UTF-8 I suspect that that's failing (see #955733) and interfering with subsequent module execution. If you remove the `locale` setting, does your userdata work as expected? Running without locale does not change anything. "grep write-files-deferred cloud-init.log" returns nothing. Best, Sven
Bug#1040979: write_files: defer not working any more
Control: tags -1 + moreinfo On Thu, Jul 13, 2023 at 05:30:52PM +0200, Sven Strickroth wrote: > starting with the shipped version in Debian 12 the write_files feature with > defer option does not work any more. I'm not able to reproduce the behavior you describe. defer seems to work as expected. For example: admin@ip-10-0-0-87:~$ ec2-metadata --ami-id ami-id: ami-0544719b13af6edc3 admin@ip-10-0-0-87:~$ sudo cat /var/lib/cloud/instance/user-data.txt #cloud-config write_files: - content: | testing. this file should be owned by the dynamically created user path: /test-file owner: admin:admin defer: true admin@ip-10-0-0-87:~$ ls -l /test-file -rw-r--r-- 1 admin admin 67 Jul 13 16:59 /test-file And in /var/log/cloud-init.log we see it being handled as expected: 2023-07-13 16:59:53,893 - modules.py[DEBUG]: Running module write-files-deferred () with frequency once-per-instance 2023-07-13 16:59:53,893 - handlers.py[DEBUG]: start: modules-final/config-write-files-deferred: running config-write-files-deferred with frequency once-per-instance 2023-07-13 16:59:53,893 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/i-016aa1a2f2e89c783/sem/config_write_files_deferred - wb: [644] 24 bytes 2023-07-13 16:59:53,893 - helpers.py[DEBUG]: Running config-write-files-deferred using lock () 2023-07-13 16:59:53,893 - util.py[DEBUG]: Writing to /test-file - wb: [644] 67 bytes 2023-07-13 16:59:53,894 - util.py[DEBUG]: Changing the ownership of /test-file to 1000:1001 2023-07-13 16:59:53,894 - handlers.py[DEBUG]: finish: modules-final/config-write-files-deferred: SUCCESS: config-write-files-deferred ran successfully I note that your cloud-config attempts to set the locale: > #cloud-config > locale: de_DE.UTF-8 I suspect that that's failing (see #955733) and interfering with subsequent module execution. If you remove the `locale` setting, does your userdata work as expected? noah
Bug#1040979: write_files: defer not working any more
Package: cloud-init Version: 22.4.2-1 Severity: important X-Debbugs-Cc: em...@cs-ware.de Dear Maintainer, starting with the shipped version in Debian 12 the write_files feature with defer option does not work any more. Example: #cloud-config locale: de_DE.UTF-8 timezone: Europe/Berlin package_update: true # Implied with `package_upgrade: true` package_upgrade: true packages: ['joe','mariadb-server','aptitude','psmisc','bzip2','fail2ban','postfix','apticron','nagios-nrpe-server','monitoring-plugins','libmariadb3','borgbackup','logwatch'] write_files: - content: | [mysqldump] max_allowed_packet = 60M path: /etc/mysql/mariadb.conf.d/51-dump.cnf defer: true The file /etc/mysql/mariadb.conf.d/51-dump.cnf is not created any more (compared to Debian 11; 20.4.1-2+deb11) by cloud-init. Nothing can be seen in the logs. This significantly limits how cloud-init can be used to set up new machines. Providing an updated version as backport etc. does not help as cloud-init is particularly used to set up a machine from a plain base image. It would be important to get a fix in the next point release. -- System Information: Debian Release: 12.0 APT prefers stable-security APT policy: (500, 'stable-security'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-10-amd64 (SMP w/3 CPU threads; PREEMPT) Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) Versions of packages cloud-init depends on: ii eject 2.38.1-5+b1 ii fdisk 2.38.1-5+b1 ii gdisk 1.0.9-2.1 ii isc-dhcp-client4.4.3-P1-2 ii locales2.36-9 ii lsb-release12.0-1 ii procps 2:4.0.2-3 ii python33.11.2-1+b1 ii python3-configobj 5.0.8-1 ii python3-jinja2 3.1.2-1 ii python3-jsonpatch 1.32-2 ii python3-jsonschema 4.10.3-1 ii python3-netifaces 0.11.0-2+b1 ii python3-oauthlib 3.2.2-1 ii python3-requests 2.28.1+dfsg-1 ii python3-serial 3.5-1.1 ii python3-yaml 6.0-3+b2 ii sysvinit-utils [lsb-base] 3.06-4 ii util-linux 2.38.1-5+b1 Versions of packages cloud-init recommends: ii cloud-guest-utils 0.33-1 ii eatmydata 130-2 ii sudo 1.9.13p3-1 Versions of packages cloud-init suggests: ii btrfs-progs 6.2-1 ii e2fsprogs1.47.0-2 ii xfsprogs 6.1.0-1 -- no debconf information
