[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
This bug was fixed in the package curtin - 0.1.0~bzr195-0ubuntu1~14.04.1 --- curtin (0.1.0~bzr195-0ubuntu1~14.04.1) trusty-proposed; urgency=medium * New upstream snapshot. - hardware enablement: ppc64 support (LP: #1386394) - hardware enablement: know kernel mapping for utopic (hwe-u = 3.16) (LP: #1386394) - feature: support installing disk images including windows. (LP: #1386394) - feature: support creating swap file (LP: #1386394) - feature: support reporting logs back to MAAS (LP: #1386394) - feature: enable logging of installation to /var/log/curtin/install.log (LP: #1378910) - bug fix: extract tar files with xattr support when available (LP: #1313550) - bug fix: fix broken use of os.path.join for curtin hooks (LP: #1328521) - bug fix: util.subp to decode command output as utf-8 (LP: #1370249). - bug fix: call update-grub to ensure that /boot/grub/grub.cfg is created (LP: #1373137) - bug fix: do not use '--acl' when extracting tar files (LP: #1382632) as it inadvertently writes default directory acls. - bug fix: invoke lsblk with '--output' rather than '--out' to avoid ambiguity in newer versions of lsblk (LP: #1386275) - internal: part2bd helper added in helpers/common - internal: helpers: inherit curtin_verbosity (make the helper tools verbose if curtin invoked with verbose flags) -- Scott Moser smo...@ubuntu.com Mon, 27 Oct 2014 20:58:43 -0400 ** Changed in: curtin (Ubuntu Trusty) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
I verified this for trusty deployment with maas 1.7.0+bzr3299-0ubuntu3~trusty1 and curtin version was 0.1.0~bzr195-0ubuntu1~14.04.1~ppa0. The steps to test this was to edit the root-tgz after uncompressing and extracting it, doing a chroot. I then did a sudo apt-get install deja- dup-backend-gvfs which also installed deja-dup and fonts-dejavu-core. The non-ascii characters were in description for deja-dup-backend-gvfs: Remote server support for Déjà Dup. After exiting, I then recreated the compressed tgz image from the modified files and copied the file to cached filename in /var/lib/maas/boot-resources/cache, and to root-tgz file in /var/lib/maas/boot- resources/current/ubuntu/amd64/generic/trusty/daily. Finally, I restarted tgt. I then deployed trusty through maas which worked. The 2nd test was to do a juju bootstrap which also passed. To make sure that I was dealing with the right image, I verified that the deja-up packages were installed on the deployed node. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
Hello Daniel, or anyone else affected, Accepted curtin into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr195-0ubuntu1~14.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: curtin (Ubuntu Trusty) Status: New = Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
** Branch linked: lp:ubuntu/trusty-proposed/curtin -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
** Description changed: I was trying to deploy an image based on an Ubuntu Desktop 14.04 install using MAAS 1.7.0~beta2+bzr2967-0ubuntu1~ppa1 on Ubuntu 14.04 server. I made a curtin-compatible tar.gz which includes cloud-init, put it in place of root-tgz in my maas server, and tried to deploy a node (using juju bootstrap). Juju ended up complaining and timing out, and the node never fully completed installation. On closer look, I found curtin crashes when running curthooks, specifically at some point it does dpkg --list, and the code that handles this (utils.subp) assumes output from its commands is pure ascii and so crashes when the output contains non-ascii. For Ubuntu Desktop, in particular, the output contains Déjȧ Dup, which is what causes things to barf. I SSHd into the half-installed node and ran curtin manually like so: sudo CURTIN_STACKTRACE=1 /curtin/bin/curtin --verbose --verbose --log- file /media/root-rw/curtin-install.log install --config=configs/config-000.cfg http://10.10.10.1/MAAS/static/images/ubuntu/amd64/generic/trusty/release /root-tgz It ended in this: Running command ['chroot', '/tmp/tmpQ_dk0_/target', 'dpkg-query', '--list'] with allowed return codes [0] (shell=False, capture=True) Traceback (most recent call last): File /curtin/curtin/commands/main.py, line 96, in main sys.exit(args.func(args)) File /curtin/curtin/commands/curthooks.py, line 443, in curthooks apply_debconf_selections(cfg, target) File /curtin/curtin/commands/curthooks.py, line 219, in apply_debconf_selections pkgs_installed = get_installed_packages(target) File /curtin/curtin/commands/curthooks.py, line 259, in get_installed_packages (out, _err) = util.subp(cmd, capture=True) File /curtin/curtin/util.py, line 58, in subp out = out.decode() UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) Traceback (most recent call last): File /curtin/curtin/commands/main.py, line 96, in main sys.exit(args.func(args)) File /curtin/curtin/commands/install.py, line 267, in cmd_install stage.run() File /curtin/curtin/commands/install.py, line 104, in run util.subp(cmd, shell=shell, env=self.env) File /curtin/curtin/util.py, line 68, in subp cmd=args) ProcessExecutionError: Unexpected error while running command. Command: ['curtin', 'curthooks'] Exit code: 3 Reason: - Stdout: '' Stderr: '' Unexpected error while running command. Command: ['curtin', 'curthooks'] Exit code: 3 Reason: - Stdout: '' Stderr: '' I'm attaching the full log as well. + + + SRU curtin + == + + [Impact] + + curtin can't handle non-ascii characters in dpkg --list output as + mentioned in the original bug report descripition. This bug doesn't + allow curtin to continue and terminates the installation. + + This fix is needed so that we can make subprocess calls with output that + contains non-ascii characters. + + The patch supplied by the bug reporter enables non-ascii character + subprocess output decoding for curtin. + + + [Test Case] + + To reproduce the bug, follow the same steps as the bug report + description mentiones. Namely, making sure that dpkg --list will + contain output with non-ascii characters. + + + [Regression Potential] + + utf-8 is backwards compatible with ascii, so the likely hood of + regressions is small. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
** Description changed: I was trying to deploy an image based on an Ubuntu Desktop 14.04 install using MAAS 1.7.0~beta2+bzr2967-0ubuntu1~ppa1 on Ubuntu 14.04 server. I made a curtin-compatible tar.gz which includes cloud-init, put it in place of root-tgz in my maas server, and tried to deploy a node (using juju bootstrap). Juju ended up complaining and timing out, and the node never fully completed installation. On closer look, I found curtin crashes when running curthooks, specifically at some point it does dpkg --list, and the code that handles this (utils.subp) assumes output from its commands is pure ascii and so crashes when the output contains non-ascii. For Ubuntu Desktop, in particular, the output contains Déjȧ Dup, which is what causes things to barf. I SSHd into the half-installed node and ran curtin manually like so: sudo CURTIN_STACKTRACE=1 /curtin/bin/curtin --verbose --verbose --log- file /media/root-rw/curtin-install.log install --config=configs/config-000.cfg http://10.10.10.1/MAAS/static/images/ubuntu/amd64/generic/trusty/release /root-tgz It ended in this: Running command ['chroot', '/tmp/tmpQ_dk0_/target', 'dpkg-query', '--list'] with allowed return codes [0] (shell=False, capture=True) Traceback (most recent call last): File /curtin/curtin/commands/main.py, line 96, in main sys.exit(args.func(args)) File /curtin/curtin/commands/curthooks.py, line 443, in curthooks apply_debconf_selections(cfg, target) File /curtin/curtin/commands/curthooks.py, line 219, in apply_debconf_selections pkgs_installed = get_installed_packages(target) File /curtin/curtin/commands/curthooks.py, line 259, in get_installed_packages (out, _err) = util.subp(cmd, capture=True) File /curtin/curtin/util.py, line 58, in subp out = out.decode() UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) Traceback (most recent call last): File /curtin/curtin/commands/main.py, line 96, in main sys.exit(args.func(args)) File /curtin/curtin/commands/install.py, line 267, in cmd_install stage.run() File /curtin/curtin/commands/install.py, line 104, in run util.subp(cmd, shell=shell, env=self.env) File /curtin/curtin/util.py, line 68, in subp cmd=args) ProcessExecutionError: Unexpected error while running command. Command: ['curtin', 'curthooks'] Exit code: 3 Reason: - Stdout: '' Stderr: '' Unexpected error while running command. Command: ['curtin', 'curthooks'] Exit code: 3 Reason: - Stdout: '' Stderr: '' I'm attaching the full log as well. - SRU curtin == [Impact] curtin can't handle non-ascii characters in dpkg --list output as mentioned in the original bug report descripition. This bug doesn't allow curtin to continue and terminates the installation. This fix is needed so that we can make subprocess calls with output that contains non-ascii characters. The patch supplied by the bug reporter enables non-ascii character subprocess output decoding for curtin. - [Test Case] To reproduce the bug, follow the same steps as the bug report - description mentiones. Namely, making sure that dpkg --list will - contain output with non-ascii characters. + description mentions. Namely, making sure that dpkg --list will contain + output with non-ascii characters. - - [Regression Potential] + [Regression Potential] utf-8 is backwards compatible with ascii, so the likely hood of regressions is small. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1370249] Re: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
** Also affects: curtin (Ubuntu) Importance: Undecided Status: New ** Changed in: curtin (Ubuntu) Status: New = Fix Released ** Also affects: curtin (Ubuntu Utopic) Importance: Undecided Status: Fix Released ** Also affects: curtin (Ubuntu Trusty) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1370249 Title: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output) To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1370249/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
