** Description changed: [ Impact ] With the current version of python3-defaults in jammy and noble, packages which utilize py3compile and py3clean in their maintscripts (which is more or less any package which includes Python modules), and which also are affected by diversions installed by dpkg-divert, will fail to install/upgrade/remove on systems with non-UTF-8 locales, which include non-ASCII characters in their localized dpkg -L messages (and which therefore produce output that is not valid UTF-8). As this has the potential to break apt upgrade, the impact is quite severe (especially if this occurs during a dist-upgrade). However, the scope is probably *fairly* limited given this only affects systems with non-UTF-8 locales (something that has not been the default for many years), and which produce non-ASCII output (e.g. French characters with diacriticals) in their localized messages for dpkg -L. [ Rectification Plan ] Firstly, python3-defaults must be patched in oracular, noble, and jammy. doko has uploaded the fix upstream and synced this to oracular. Benjamin has sponsored the noble and jammy SRUs. Next, affected packages must be patched to include a "Breaks" line on earlier versions of python3-minimal (the python3-defaults binary package containing py3clean and py3compile). Packages which install Python modules are easily selected, but those affected by diversions are (computationally) impossible to select (see comment 18 for details). Thus, determining the set of affected packages was non-trivial, but Benjamin has kindly produced such a list by empirical means (see comment 24). These will be added to the ticket as they are uploaded. + + [ Known affected packages ] + + ``` + $ for section in main restricted universe multiverse; do echo "noble $section:"; sqlite3 affected.sqlite3 "SELECT package, source FROM package_affected WHERE section = '$section' and install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) ORDER BY package;"; echo; done + noble main: + cloud-init|cloud-init + nvme-stas|nvme-stas + python3-rtslib-fb|python-rtslib-fb + targetcli-fb|targetcli-fb + ubiquity|ubiquity + ubiquity-frontend-debconf|ubiquity + ubiquity-frontend-gtk|ubiquity + ubuntu-drivers-common|ubuntu-drivers-common + ubuntu-pro-client|ubuntu-advantage-tools + walinuxagent|walinuxagent + + noble restricted: + + noble universe: + a2d|a2d + apertium-apy|apertium-apy + blueman|blueman + borgmatic|borgmatic + condor|condor + cpupower-gui|cpupower-gui + debomatic|debomatic + dell-recovery|dell-recovery + deluge-web|deluge + depthcharge-tools|depthcharge-tools + dhcpy6d|dhcpy6d + dkimpy-milter|dkimpy-milter + fenrir|fenrir + galileo|galileo + glances|glances + graphite-carbon|graphite-carbon + grokmirror|grokmirror + guerillabackup|guerillabackup + ionit|ionit + jupyterhub|jupyterhub + liquidctl|liquidctl + localslackirc|localslackirc + logdata-anomaly-miner|logdata-anomaly-miner + lomiri|lomiri + mailman3|mailman3 + mailman3-full|mailman3 + matrix-sydent|matrix-sydent + mom|mom + mopidy|mopidy + mpd-sima|mpd-sima + nordugrid-arc-arex|nordugrid-arc + notus-scanner|notus-scanner + nrpe-ng|nrpe-ng + onionprobe|onionprobe + openscap-daemon|openscap-daemon + ospd-openvas|ospd-openvas + pagekite|pagekite + pagure|pagure + pagure-ev-server|pagure + pagure-milters|pagure + pglistener|pglistener + policyd-rate-limit|policyd-rate-limit + postfix-mta-sts-resolver|postfix-mta-sts-resolver + powa-collector|powa-collector + prelude-correlator|prelude-correlator + prometheus-pgbouncer-exporter|prometheus-pgbouncer-exporter + prometheus-xmpp-alerts|prometheus-xmpp-alerts + pyroman|pyroman + python3-autopilot|autopilot + python3-btchip|btchip-python + python3-pyro5|pyro5 + qweborf|weborf + razercfg|razercfg + sgt-launcher|sgt-launcher + sms4you|sms4you + supervisor|supervisor + syncplay-server|syncplay + timekpr-next|timekpr-next + twms|twms + ubiquity-frontend-kde|ubiquity + vanguards|vanguards + vaultlocker|vaultlocker + voctomix-outcasts|voctomix-outcasts + xdiagnose|xdiagnose + + noble multiverse: + r4d|r4d + virtualbox|virtualbox + ``` + + Code and resulting database can be found on + https://github.com/bdrung/bdrung-scripts/tree/lp-2075337-find-affected [ Test Plan ] Where $series is jammy or noble as appropriate, and $package is the name of the affected package being SRU'd: $ lxc launch ubuntu:$series py3clean-test $ lxc shell py3clean-test # locale-gen fr_FR Generating locales (this might take a while)... fr_FR.ISO-8859-1... done Generation complete. # update-locale LANG=fr_FR # exit # adjust terminal to ISO-8859-1 $ lxc shell py3clean-test # apt update && apt install --reinstall $package With the current version of python3-defaults in jammy and noble this should fail at the final apt install --reinstall step. With the proposed version, it should succeed. [ What Could Go Wrong? ] The potential for regression is low since the python3-defaults changes are minimal and have been reasonably tested, and the only change to affected packages is the addition of a Breaks clause (i.e. the content of the packages are unaffected, just the meta-data). The major cause for concern is that we are not 100% certain that the Breaks addition is sufficient to fix this. The Debian policy manual is ambiguous as to whether Breaks implies that the fixed package must be unpacked *before* the prerm maintscript of the package it breaks is executed; the policy manual mentions this must happen before the "unpack" phase, but prerm scripts run *before* the unpack phase. However, in all experiments run so far, this did indeed resolve the problem. Still, as an extra hedge against failure here, we intend that the python3-defaults SRU is released ASAP, and that we then delay a week to allow it to propagate before releasing SRUs for the affected packages. [ Original Description ] We noticed this in bug 2060769 but think it is a separate issue. dh_python3 is putting a py3clean invocation into package prerm maintainer scripts. When that runs, it explodes if the system character set is for example ISO-8859-1. This appears to affect every package for which dh_python3 has inserted a call to py3clean into the package prerm. python3-minimal 3.12.3-0ubuntu1 In this reproducer, cloud-init is at 24.1.3-0ubuntu3.3. Steps to reproduce: $ lxc launch ubuntu:noble rbasak-pro-l8n $ lxc shell rbasak-pro-l8n # locale-gen fr_FR Generating locales (this might take a while)... fr_FR.ISO-8859-1... done Generation complete. # update-locale LANG=fr_FR # exit # adjust terminal to ISO-8859-1 $ lxc shell rbasak-pro-l8n # apt update && apt install --reinstall cloud-init Expected results: success Actual results: Lecture des listes de paquets... Fait Construction de l'arbre des dépendances... Fait Lecture des informations d'état... Fait 0 mis à jour, 0 nouvellement installés, 1 réinstallés, 0 à enlever et 5 non mis à jour. Il est nécessaire de prendre 598 ko dans les archives. Après cette opération, 0 o d'espace disque supplémentaires seront utilisés. Réception de :1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 cloud-init all 24.1.3-0ubuntu3.3 [598 kB] 598 ko réceptionnés en 1s (926 ko/s) Préconfiguration des paquets... (Lecture de la base de données... 34406 fichiers et répertoires déjà installés.) Préparation du dépaquetage de .../cloud-init_24.1.3-0ubuntu3.3_all.deb ... Traceback (most recent call last): File "/usr/bin/py3clean", line 210, in <module> main() File "/usr/bin/py3clean", line 196, in main pfiles = set(dpf.from_package(options.package)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/python3/debpython/files.py", line 54, in from_package stdout = str(stdout, 'utf-8') ^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte dpkg: avertissement: le sous-processus ancien paquet cloud-init script pre-removal a renvoyé un état de sortie d'erreur 1 dpkg: tentative d'exécution du script du nouveau paquet à la place... Traceback (most recent call last): File "/usr/bin/py3clean", line 210, in <module> main() File "/usr/bin/py3clean", line 196, in main pfiles = set(dpf.from_package(options.package)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/python3/debpython/files.py", line 54, in from_package stdout = str(stdout, 'utf-8') ^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte dpkg: erreur de traitement de l'archive /var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb (--unpack) : le sous-processus nouveau cloud-init paquet pre-removal script a renvoyé un état de sortie d'erreur 1 dmesg: read kernel buffer failed: Operation not permitted Traceback (most recent call last): File "/usr/bin/py3compile", line 323, in <module> main() File "/usr/bin/py3compile", line 302, in main compile(files, versions, File "/usr/bin/py3compile", line 185, in compile for fn, versions_to_compile in filter_files(files, e_patterns, versions): File "/usr/bin/py3compile", line 128, in filter_files for fpath in files: File "/usr/share/python3/debpython/files.py", line 71, in filter_public for fn in files: File "/usr/share/python3/debpython/files.py", line 54, in from_package stdout = str(stdout, 'utf-8') ^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte dpkg: erreur lors du nettoyage: le sous-processus paquet cloud-init script post-installation installé a renvoyé un état de sortie d'erreur 1 Des erreurs ont été rencontrées pendant l'exécution : /var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1)
-- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to dh-python in Ubuntu. https://bugs.launchpad.net/bugs/2075337 Title: [SRU] py3clean fails when using alternate character set Status in cloud-init package in Ubuntu: New Status in dh-python package in Ubuntu: Invalid Status in python3-defaults package in Ubuntu: Confirmed Status in ubuntu-advantage-tools package in Ubuntu: New Status in cloud-init source package in Jammy: Invalid Status in dh-python source package in Jammy: Invalid Status in python3-defaults source package in Jammy: Confirmed Status in ubuntu-advantage-tools source package in Jammy: Invalid Status in cloud-init source package in Noble: New Status in dh-python source package in Noble: Invalid Status in python3-defaults source package in Noble: Confirmed Status in ubuntu-advantage-tools source package in Noble: New Bug description: [ Impact ] With the current version of python3-defaults in jammy and noble, packages which utilize py3compile and py3clean in their maintscripts (which is more or less any package which includes Python modules), and which also are affected by diversions installed by dpkg-divert, will fail to install/upgrade/remove on systems with non-UTF-8 locales, which include non-ASCII characters in their localized dpkg -L messages (and which therefore produce output that is not valid UTF-8). As this has the potential to break apt upgrade, the impact is quite severe (especially if this occurs during a dist-upgrade). However, the scope is probably *fairly* limited given this only affects systems with non-UTF-8 locales (something that has not been the default for many years), and which produce non-ASCII output (e.g. French characters with diacriticals) in their localized messages for dpkg -L. [ Rectification Plan ] Firstly, python3-defaults must be patched in oracular, noble, and jammy. doko has uploaded the fix upstream and synced this to oracular. Benjamin has sponsored the noble and jammy SRUs. Next, affected packages must be patched to include a "Breaks" line on earlier versions of python3-minimal (the python3-defaults binary package containing py3clean and py3compile). Packages which install Python modules are easily selected, but those affected by diversions are (computationally) impossible to select (see comment 18 for details). Thus, determining the set of affected packages was non-trivial, but Benjamin has kindly produced such a list by empirical means (see comment 24). These will be added to the ticket as they are uploaded. [ Known affected packages ] ``` $ for section in main restricted universe multiverse; do echo "noble $section:"; sqlite3 affected.sqlite3 "SELECT package, source FROM package_affected WHERE section = '$section' and install_affected = 1 AND (remove_affected = 1 OR remove_affected ISNULL) ORDER BY package;"; echo; done noble main: cloud-init|cloud-init nvme-stas|nvme-stas python3-rtslib-fb|python-rtslib-fb targetcli-fb|targetcli-fb ubiquity|ubiquity ubiquity-frontend-debconf|ubiquity ubiquity-frontend-gtk|ubiquity ubuntu-drivers-common|ubuntu-drivers-common ubuntu-pro-client|ubuntu-advantage-tools walinuxagent|walinuxagent noble restricted: noble universe: a2d|a2d apertium-apy|apertium-apy blueman|blueman borgmatic|borgmatic condor|condor cpupower-gui|cpupower-gui debomatic|debomatic dell-recovery|dell-recovery deluge-web|deluge depthcharge-tools|depthcharge-tools dhcpy6d|dhcpy6d dkimpy-milter|dkimpy-milter fenrir|fenrir galileo|galileo glances|glances graphite-carbon|graphite-carbon grokmirror|grokmirror guerillabackup|guerillabackup ionit|ionit jupyterhub|jupyterhub liquidctl|liquidctl localslackirc|localslackirc logdata-anomaly-miner|logdata-anomaly-miner lomiri|lomiri mailman3|mailman3 mailman3-full|mailman3 matrix-sydent|matrix-sydent mom|mom mopidy|mopidy mpd-sima|mpd-sima nordugrid-arc-arex|nordugrid-arc notus-scanner|notus-scanner nrpe-ng|nrpe-ng onionprobe|onionprobe openscap-daemon|openscap-daemon ospd-openvas|ospd-openvas pagekite|pagekite pagure|pagure pagure-ev-server|pagure pagure-milters|pagure pglistener|pglistener policyd-rate-limit|policyd-rate-limit postfix-mta-sts-resolver|postfix-mta-sts-resolver powa-collector|powa-collector prelude-correlator|prelude-correlator prometheus-pgbouncer-exporter|prometheus-pgbouncer-exporter prometheus-xmpp-alerts|prometheus-xmpp-alerts pyroman|pyroman python3-autopilot|autopilot python3-btchip|btchip-python python3-pyro5|pyro5 qweborf|weborf razercfg|razercfg sgt-launcher|sgt-launcher sms4you|sms4you supervisor|supervisor syncplay-server|syncplay timekpr-next|timekpr-next twms|twms ubiquity-frontend-kde|ubiquity vanguards|vanguards vaultlocker|vaultlocker voctomix-outcasts|voctomix-outcasts xdiagnose|xdiagnose noble multiverse: r4d|r4d virtualbox|virtualbox ``` Code and resulting database can be found on https://github.com/bdrung/bdrung-scripts/tree/lp-2075337-find-affected [ Test Plan ] Where $series is jammy or noble as appropriate, and $package is the name of the affected package being SRU'd: $ lxc launch ubuntu:$series py3clean-test $ lxc shell py3clean-test # locale-gen fr_FR Generating locales (this might take a while)... fr_FR.ISO-8859-1... done Generation complete. # update-locale LANG=fr_FR # exit # adjust terminal to ISO-8859-1 $ lxc shell py3clean-test # apt update && apt install --reinstall $package With the current version of python3-defaults in jammy and noble this should fail at the final apt install --reinstall step. With the proposed version, it should succeed. [ What Could Go Wrong? ] The potential for regression is low since the python3-defaults changes are minimal and have been reasonably tested, and the only change to affected packages is the addition of a Breaks clause (i.e. the content of the packages are unaffected, just the meta-data). The major cause for concern is that we are not 100% certain that the Breaks addition is sufficient to fix this. The Debian policy manual is ambiguous as to whether Breaks implies that the fixed package must be unpacked *before* the prerm maintscript of the package it breaks is executed; the policy manual mentions this must happen before the "unpack" phase, but prerm scripts run *before* the unpack phase. However, in all experiments run so far, this did indeed resolve the problem. Still, as an extra hedge against failure here, we intend that the python3-defaults SRU is released ASAP, and that we then delay a week to allow it to propagate before releasing SRUs for the affected packages. [ Original Description ] We noticed this in bug 2060769 but think it is a separate issue. dh_python3 is putting a py3clean invocation into package prerm maintainer scripts. When that runs, it explodes if the system character set is for example ISO-8859-1. This appears to affect every package for which dh_python3 has inserted a call to py3clean into the package prerm. python3-minimal 3.12.3-0ubuntu1 In this reproducer, cloud-init is at 24.1.3-0ubuntu3.3. Steps to reproduce: $ lxc launch ubuntu:noble rbasak-pro-l8n $ lxc shell rbasak-pro-l8n # locale-gen fr_FR Generating locales (this might take a while)... fr_FR.ISO-8859-1... done Generation complete. # update-locale LANG=fr_FR # exit # adjust terminal to ISO-8859-1 $ lxc shell rbasak-pro-l8n # apt update && apt install --reinstall cloud-init Expected results: success Actual results: Lecture des listes de paquets... Fait Construction de l'arbre des dépendances... Fait Lecture des informations d'état... Fait 0 mis à jour, 0 nouvellement installés, 1 réinstallés, 0 à enlever et 5 non mis à jour. Il est nécessaire de prendre 598 ko dans les archives. Après cette opération, 0 o d'espace disque supplémentaires seront utilisés. Réception de :1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 cloud-init all 24.1.3-0ubuntu3.3 [598 kB] 598 ko réceptionnés en 1s (926 ko/s) Préconfiguration des paquets... (Lecture de la base de données... 34406 fichiers et répertoires déjà installés.) Préparation du dépaquetage de .../cloud-init_24.1.3-0ubuntu3.3_all.deb ... Traceback (most recent call last): File "/usr/bin/py3clean", line 210, in <module> main() File "/usr/bin/py3clean", line 196, in main pfiles = set(dpf.from_package(options.package)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/python3/debpython/files.py", line 54, in from_package stdout = str(stdout, 'utf-8') ^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte dpkg: avertissement: le sous-processus ancien paquet cloud-init script pre-removal a renvoyé un état de sortie d'erreur 1 dpkg: tentative d'exécution du script du nouveau paquet à la place... Traceback (most recent call last): File "/usr/bin/py3clean", line 210, in <module> main() File "/usr/bin/py3clean", line 196, in main pfiles = set(dpf.from_package(options.package)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/python3/debpython/files.py", line 54, in from_package stdout = str(stdout, 'utf-8') ^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte dpkg: erreur de traitement de l'archive /var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb (--unpack) : le sous-processus nouveau cloud-init paquet pre-removal script a renvoyé un état de sortie d'erreur 1 dmesg: read kernel buffer failed: Operation not permitted Traceback (most recent call last): File "/usr/bin/py3compile", line 323, in <module> main() File "/usr/bin/py3compile", line 302, in main compile(files, versions, File "/usr/bin/py3compile", line 185, in compile for fn, versions_to_compile in filter_files(files, e_patterns, versions): File "/usr/bin/py3compile", line 128, in filter_files for fpath in files: File "/usr/share/python3/debpython/files.py", line 71, in filter_public for fn in files: File "/usr/share/python3/debpython/files.py", line 54, in from_package stdout = str(stdout, 'utf-8') ^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2417: invalid continuation byte dpkg: erreur lors du nettoyage: le sous-processus paquet cloud-init script post-installation installé a renvoyé un état de sortie d'erreur 1 Des erreurs ont été rencontrées pendant l'exécution : /var/cache/apt/archives/cloud-init_24.1.3-0ubuntu3.3_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2075337/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
