[Bug 1833618] Re: failing to deploy Ubuntu Disco
All good for a final review and Eoan upload (followed by the SRUs then). ** Description changed: + [Impact] + + * sg3-utils-udev package introduces 55-scsi-sg3_id.rules udev rule that + take places BEFORE rules either finding correct ID_SERIAL for the USB + block device, or calculating it from "usb_id" parameter: + + - this rule changes USB ID_SERIAL parameter for the ones given by + the Vital Product Data from the block device. + + - USB memory sticks might not provide VPD pages (0x80, 0x83) and, + because of that, the USB ID_SERIAL will be broken. + + - because the failure in sg_inq command from the 55-XXX udev rules + file, to obtain the VPDs, the udev ACTION(add) takes long time and + consumers might not have considered a "settle" need. + + * Anything depending on ID_SERIAL udev attribute from USB block devices + might not find appropriate device. + + * Any type of udev consumer not calculating the need for a big time-consuming "udevadm settle" right after a pen drive is inserted + might not be able to see all USB block devices attributes from udev. + + [Test Case] + + * Install package "sg3-utils-udev", monitor journalctl -f while + introducing a memory stick in your machine and watch sg_inq command + failing: + + systemd-udevd[1021]: Process '/usr/bin/sg_inq --export + --inhex=/sys/block/sdb/device/vpd_pg83 --raw' failed with exit code 15. + + * Compare udevadm attributes with and without "sg3-utils-udev" package + is installed: + + Without sg3-utils 55-scsi-sg3_id.rules: + + $ udevadm info --query=all --name=/dev/sdb | grep -i serial + E: ID_SERIAL=Corsair_Voyager_Mini_3.0_070851D0E490C776-0:0 + E: ID_SERIAL_SHORT=070851D0E490C776 + + With 55-scsi-sg3_id.rules: + + $ udevadm info --query=all --name=/dev/sdb | grep -i serial + E: ID_SERIAL=3 + E: ID_SERIAL_SHORT=2000acde48234567 + + [Regression Potential] + + * Minor: either I brake functionality for sg3-utils packages on USB + block devices. Since sg3-utils is used mostly for SCSI debugging, + management and investigations, its a very minor risk. + + * The change is a 1 line change in a udev rule file, very easy to test + following the test case. + + [Other Info] + + + ORIGINAL DESCRIPTION: + * MAAS version: 2.4.2 (7034-g2f5deb8b8-0ubuntu1) MAAS UI changes: ``` Node changed status - From 'Deploying' to 'Failed deployment' Node installation failure - 'curtin' curtin command install ``` when console_log shows: ``` [ 244.588201] cloud-init[3139]: File "/curtin/curtin/commands/block_meta.py", line 1515, in get_path_if_present [ 244.588389] cloud-init[3139]: return get_path_to_storage_volume(disk, config) [ 244.588588] cloud-init[3139]: File "/curtin/curtin/commands/block_meta.py", line 433, in get_path_to_storage_volume [ 244.588774] cloud-init[3139]: specified to identify disk") [ 244.588963] cloud-init[3139]: ValueError: serial, wwn or path to block dev must be specified to identify disk [ 244.589153] cloud-init[3139]: serial, wwn or path to block dev must be specified to identify disk - [ 244.589338] cloud-init[3139]: + [ 244.589338] cloud-init[3139]: [ 244.589523] cloud-init[3139]: Stderr: '' [ 244.589708] cloud-init[3139]: Unexpected error while running command. [ 244.589905] cloud-init[3139]: Command: ['curtin', 'block-meta', 'custom'] [ 244.590088] cloud-init[3139]: Exit code: 3 [ 244.590271] cloud-init[3139]: Reason: - [ 244.590459] cloud-init[3139]: Stdout: start: cmd-install/stage-partitioning/builtin/cmd-block-meta: curtin command block-meta ``` Reproducible: yes - aarch64, arm64 machine: + aarch64, arm64 machine: Manufacturer: CN8890-2000BG2601-CP-Y-G Part Number: CN88xx Cavium SOC = Cavium THUNDERX Boot Stub = BDK Version: 04.02.14, 2.38 Branch: gigabyte-2.38, Built: Fri Aug 18 01:16:58 UTC 2017 Board Model:gbt-mt30 Board name: MT30-GS1-00 System name:R120-T33-00 Board Revision: 1.x Board SKU: 04 SATA SKU version:1.1 Node: 0 (Fixed) Chip: 0xa1 Pass 2.1 SKU: CN8890-2000BG2601-CP-Y-G *MAAS get-curtin-config: https://paste.ubuntu.com/p/k54HHN55ZF/ ** Summary changed: - failing to deploy Ubuntu Disco + MAAS can't deploy Ubuntu if ID_SERIAL of any block device is broken (USB pendrive in this case). ** Changed in: sg3-utils (Debian) Importance: Unknown => Undecided ** Changed in: sg3-utils (Debian) Status: Unknown => New ** Changed in: sg3-utils (Debian) Remote watch: Debian Bug tracker #1833618 => None ** Package changed: sg3-utils (Debian) => maas ** Changed in: maas Status: New => Invalid ** Changed in: maas Status: Invalid => In Progress ** Changed in: maas Assignee: (unassigned) => Rafael David Tinoco (rafaeldtinoco) ** Changed in: curtin Importance: Un
[Bug 1833618] Re: failing to deploy Ubuntu Disco
This it the upstream pull request: https://github.com/hreinecke/sg3_utils/pull/47 And I have updated Debian pull request as well: https://salsa.debian.org/linux-blocks- team/sg3-utils/merge_requests/2/commits I'm preparing updates for Eoan, Disco and Bionic fixes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Thanks for confirming that! On the other hand, patch seems to be good, even for upstream. >From the merge request comment: """ ACTUALLY, it made little difference because the new rules: Oct 04 09:34:46 macbook systemd-udevd[30592]: Process '/usr/bin/sg_inq --export --inhex=/sys/class/scsi_device/2:0:0:0/device/vpd_pg80 --raw' failed with exit code 15. Oct 04 09:34:46 macbook systemd-udevd[30592]: Process '/usr/bin/sg_inq --export --inhex=/sys/class/scsi_device/2:0:0:0/device/vpd_pg83 --raw' failed with exit code 15. Still don't consider USB bus exception. The rule you mentioned checks for anything (else) then /dev/sd* or /dev/sr* which is not the case for pendrives, for example. Thus, the need for me to have added: DEVPATH=="*usb*", GOTO="sg3_utils_id_end" because: E: DEVPATH=/devices/pci:00/:00:14.0/usb1 it the only attribute I could differentiate USB from SCSI BEFORE the actual VPD reading. I'll suggest my change to upstream now, possibly mentioning a Forwarded item in the patches. """ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
A follow-up on comment #47 - I've proven that the reboot loop had nothing to do with your change, I just missed a required section of the preseed. Updated file attached just as an fyi. ** Attachment added: "curtin_userdata_ubuntu_arm64_generic_disco_seidel-FLAKYMEMORY" https://bugs.launchpad.net/ubuntu/+source/sg3-utils/+bug/1833618/+attachment/5294075/+files/curtin_userdata_ubuntu_arm64_generic_disco_seidel-FLAKYMEMORY -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
No no, you are definitely right, I totally missed upstream change, I did "on my own", not sure why I thought upstream didn't have any change/fix (too many bugs in my head perhaps). I'll reply you in the MR, but, we should go with whatever upstream has done, so delta is reduced. I might re-do the MRs, even the one for Debian. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1833618] Re: failing to deploy Ubuntu Disco
On Thu, Oct 3, 2019 at 4:42 AM Rafael David Tinoco wrote: > > Hey Dann, nope, the file changed is Debian related only. hmm.. I see the file in the upstream source, and Debian hasn't patched it. But, perhaps you mean that upstream is not impacted. And yeah, I can see that this may be the case. Upstream appear to have applied a change post 1.44 to only run sg_inq on known devices. I'll comment further on that via your MR. > I'm glad it > worked, I'll ask server team review and merge then! Tks for testing it!! np - thanks for working this! ** Bug watch added: Debian Bug tracker #1833618 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1833618 ** Also affects: sg3-utils (Debian) via https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1833618 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Hey Dann, nope, the file changed is Debian related only. I'm glad it worked, I'll ask server team review and merge then! Tks for testing it!! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
@rafaeldtinoco: Also, I wonder if it wouldn't make sense to submit your patch upstream? https://github.com/hreinecke/sg3_utils -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
I MAAS deployed an impacted system w/ the attached preseed to include your PPA, and curtin successfully installed. Afterward, the system entered a deploy loop, but there's no sign that this was related to your fix - rather, it may just be a boot order issue. Interrupting the loop and telling it to boot from disk caused it to successfully reach deployed state. ** Attachment added: "curtin_userdata_ubuntu_arm64_generic_disco_seidel-FLAKYMEMORY" https://bugs.launchpad.net/ubuntu/+source/sg3-utils/+bug/1833618/+attachment/5293754/+files/curtin_userdata_ubuntu_arm64_generic_disco_seidel-FLAKYMEMORY -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Disco PPA package (and MR) is good. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Note: I have to re-submit Disco MR and a new PPA package, as the compiler in Disco is likely more recent and treating lots of "new warnings" as errors (-Wimplicit-fallthrough, etc...). I'll do it tomorrow! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Summary: Merge Requests (to be reviewed): # Debian MR: https://salsa.debian.org/linux-blocks-team/sg3-utils/merge_requests/2 # Eoan https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373438 # Disco https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373439 # Bionic https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373440 PPA (to be tested: amd64, arm64 and s390x): https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1833618 I have already asked for server review on merge requests, I would appreciate, if possible that the packages provided in PPA are tested. Thank you very much! Rafael D. Tinoco ** Merge proposal linked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373438 ** Merge proposal linked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373439 ** Merge proposal linked: https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/sg3-utils/+git/sg3-utils/+merge/373440 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
I have asked upstream to consider the following merge: https://salsa.debian.org/linux-blocks-team/sg3-utils/merge_requests/2 ** Changed in: sg3-utils (Ubuntu Bionic) Status: Confirmed => In Progress ** Changed in: sg3-utils (Ubuntu Disco) Status: Confirmed => In Progress ** Changed in: sg3-utils (Ubuntu Eoan) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
$ rmadison sg3-utils-udev
sg3-utils-udev | 1.40-0ubuntu1 | xenial | all
sg3-utils-udev | 1.42-2ubuntu1 | bionic | all
sg3-utils-udev | 1.42-2ubuntu1 | disco | all
sg3-utils-udev | 1.44-1ubuntu1 | eoan | all
And:
[ Mauricio Faria de Oliveira ]
* sg3-utils-udev (new binary package): provide the udev rules for
SCSI-ID mappings and symlinks at boot time for multipath-tools:
- debian/control: add sg3-utils-udev
- debian/rules: build it with override_dh_{install,clean}
- debian/initramfs/hooks: add_udev_rules() and copy_exec() 'sg_inq'
to initramfs
- debian/sg3-utils-udev.post{inst,rm}: call update-initramfs
Marking this affecting Bionic, Disco and Eoan.
Will also provide a MR do Debian/Salsa.
** Also affects: sg3-utils (Ubuntu Bionic)
Importance: Undecided
Status: New
** Also affects: sg3-utils (Ubuntu Eoan)
Importance: High
Assignee: Rafael David Tinoco (rafaeldtinoco)
Status: Confirmed
** Also affects: sg3-utils (Ubuntu Disco)
Importance: Undecided
Status: New
** Changed in: curtin
Status: Confirmed => Invalid
** Changed in: curtin
Importance: High => Undecided
** Changed in: curtin
Assignee: Rafael David Tinoco (rafaeldtinoco) => (unassigned)
** Changed in: sg3-utils (Ubuntu Disco)
Status: New => Confirmed
** Changed in: sg3-utils (Ubuntu Bionic)
Status: New => Confirmed
** Changed in: sg3-utils (Ubuntu Disco)
Importance: Undecided => High
** Changed in: sg3-utils (Ubuntu Bionic)
Importance: Undecided => High
** Changed in: sg3-utils (Ubuntu Disco)
Assignee: (unassigned) => Rafael David Tinoco (rafaeldtinoco)
** Changed in: sg3-utils (Ubuntu Bionic)
Assignee: (unassigned) => Rafael David Tinoco (rafaeldtinoco)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1833618
Title:
failing to deploy Ubuntu Disco
To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Alright, I'll change the udev rules from sg3-utils-udev package to ignore USB devices. From the initial ACTION (add) event, I only get simple data from udev: S: disk/by-id/usb-Corsair_Voyager_Mini_3.0_070851D0E490C776-0:0 S: disk/by-path/pci-:00:14.0-usb-0:2:1.0-scsi-0:0:0:0 S: disk/by-label/Ubuntu\x2019.04\x20amd64 S: disk/by-uuid/2019-04-16-19-19-59-00 E: DEVPATH=/devices/pci:00/:00:14.0/usb2/2-2/2-2:1.0/host2/target2:0:0/2:0:0:0/block/sdc E: SUBSYSTEM=block E: DEVNAME=/dev/sdc E: DEVTYPE=disk E: MAJOR=8 E: MINOR=32 E: USEC_INITIALIZED=376735479267 Before the device is fully initialized. I will try to ignore DEVPATH containing *usb*, for example, at least for the VPD pages 0x80 and 0x83, I'll get back soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Adding: # Ignore USB block devices as they don't contain VPD 0x80 and 0x83 # and their SERIAL is calculated from usb_id in persistent-storage rules DEVPATH=="*usb*", GOTO="sg3_utils_id_end" before #SCSI EVPD page 0x80 values in 55-scsi-sg3_id.rules file fixes the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
I meant "for USB devices and that would be "enough"" in previous comment. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Well, checking udev rules, its exactly what you said. I guess we could change sg3 udev rules not to execute sg_inq --export --inhex to /sys/block/$kernel/device/vpd_pg83 for SCSI devices and that would be "enough" for this issue. --inhex=FN|-I FNread ASCII hex from file FN instead of DEVICE; if used with --raw then read binary from FN Looks like sg_inq gets the VPD 0×83 page directly from in-cache sysfs kernel file - from the initial device initialization made by kernel - instead of submitting an I/O to the device like this: $ sudo sg_vpd /dev/sdc Supported VPD pages VPD page: invalid VPD response; probably a STANDARD INQUIRY response fetching VPD page failed: Malformed response which would also give us an error. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
BTW, you got: 15 the utility is unable to open, close or use the given DEVICE or some other file. The given file name could be incorrect or there may be permission problems. Adding the '-v' option may give more information. And this is likely because USB devices don't have VPD, so sysfs is not showing "/sys/block/XXX/device/vpd_pg83": >From SCSI SPC-3 standard: """ Device Identification VPD page (page number: 0x83). Since SPC-3, support for this page has been flagged as mandatory. This page can be fetched by using the --ident option. The reference document used for interpreting VPD pages (and the INQUIRY standard response) is T10/1713-D Revision 36e (SPC-4, 24 August 2012) found at http://www.t10.org . Obsolete and reserved items in the standard INQUIRY response output are displayed in square brackets. """ BUT it only serves for SCSI devices for RAW DEVICE MAPPING technique. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
This also happens in amd64 (and likely other architectures), only when having "sg3-utils-udev" package installed. Command "udevadm settle" only returns after the initial "sg_inq" command also returns, raising the question: (1) Would call "udevmadm settle" before "udevadm info" be enough to get A USB device serial ? (not saying it would be the same serial as it would have been without sg3-utils-udev rules in place). Another question: (2) Is this device initialization taking long because of bad "sg3-utils- udev" rules being executed before the initial I/O commands can be submitted to the device ? (very likely). Stack shows: [<0>] blk_execute_rq+0x75/0xb0 [<0>] sg_io+0x176/0x460 [<0>] scsi_cmd_ioctl+0x1b9/0x3f0 [<0>] scsi_cmd_blk_ioctl+0x51/0x61 [<0>] sd_ioctl+0xcd/0x1c0 [<0>] blkdev_ioctl+0x4c1/0x9f0 [<0>] block_ioctl+0x3d/0x50 [<0>] do_vfs_ioctl+0xa9/0x640 [<0>] ksys_ioctl+0x67/0x90 [<0>] __x64_sys_ioctl+0x1a/0x20 [<0>] do_syscall_64+0x5a/0x110 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [<0>] 0x we are blocked waiting for the VPD (as it was sent through sd_ioctl command to block device layer). I'll trace kernel to check why blk_execute_rq() is taking so long when sg3-utils-udev package is installed (likely command might still be held because device it not yet initialized, due to bad udev rules, "to check"). -- Regarding your concern: YES, it IS A different serial, AND... sometimes, udevadm returns no serial at all (until "udev settle" returns, and this behavior only happens when sg3-utils-udev rules are in place and during the first sg_inq command, suggesting the device is initializing with the bad udev rules). I'll "bisect" the rules to find the guilty one(s), and, yes, this might be orthogonal to the issue you need, if you need "same serials" in place. For that, you would need a higher priority rule setting SCSI SERIAL based on anything you want (since SCSI SERIAL is not present in USB devices, not the VITAL PRODUCT PAGE, like regular SCSI devices). Likely, the correct thing to do here is to keep the USB serial as the "emulated" (by udev rules) SCSI SERIAL. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
The _real_ concern is: do you get the _same_ ID_SERIAL value with and without the sg3-utils- udev package installed? Can you compare: /dev/disk/by-id/*https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
@Rhyan, thanks for the quick pointers, helped a lot =), I think I found
out what is going on.
Linux mylab 5.0.0-30-generic #32-Ubuntu SMP Wed Sep 18 00:24:43 UTC 2019
aarch64 aarch64 aarch64 GNU/Linux
Using a memstick in an aarch64 server:
$ sudo sg_inq -d /dev/sdb
standard INQUIRY:
PQual=0 Device_type=0 RMB=1 LU_CONG=0 version=0x06 [SPC-4]
[AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2
SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 [BQue=0]
EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 [Linked=0] [TranDis=0] CmdQue=0
length=36 (0x24) Peripheral device type: disk
Vendor identification: Corsair
Product identification: Voyager Mini 3.0
Product revision level: 000B
Unit serial number: 0C7707865010
command sg_inq returns right away with proper info and, with 60
-persistent-storage.rules only:
$ sudo udevadm info --query=all --name=/dev/sdb | grep -i serial
E: ID_SERIAL=Corsair_Voyager_Mini_3.0_070851D0E490C776-0:0
E: ID_SERIAL_SHORT=070851D0E490C776
$ sudo sg_inq -d /dev/sdb
$ sudo udevadm info --query=all --name=/dev/sdb | grep -i serial
AFTER sg_inq command returns (10 seconds or more) AND with 55-scsi-
sg3.rules installed, then we get:
$ sudo udevadm info --query=all --name=/dev/sdb | grep -i serial
E: ID_SERIAL=3
E: ID_SERIAL_SHORT=2000acde48234567
Which means that its likely that curtin did not wait enough (or settled)
for the device VPD to "arrive". I haven't found out why the "USB"
("SCSI") vpd gets so long to be returned when sg-utils-udev is
installed, will investigate.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1833618
Title:
failing to deploy Ubuntu Disco
To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
@Rafael, the critical path would be to check the output from: udevadm info --query=all --name=/dev/sd[cd] | grep -i serial ID_SERIAL does not get set on Disco where we have sg3-utils package install, which adds the s3g-utils-udev package, and includes the rules file 55-scsi-sg3_id.rules. This udev rule uses sg_inq --export /path/to/device to query for device properties. I suspect you can reproduce the failure on the device by running sg_inq --export on the usb block device. Note that in 55-scsi-sg3 rules, if no ID_SERIAL is found via the previous import, then the rules construct an ID_SERIAL from various scsi id values; This is the core issue since normally 60-persistent-storage.rules would use 'ata_id', 'usb_id' and 'scsi_id' to extract the ID_SERIAL value, but since 55 rules run before 60, the sg3-utils-udev package sets ID_SERIAL, and prevents the 60-persistent-storage.rules from setting the value to match. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
@Patricia,
I'll concentrate efforts into the sg3-utils issue with disco for now.
For Eoan, based on the console output you attached to the bug, it seems
that one of the services listed as "getty@ttyAMA0.service" dependencies
did not start before timeout occurred. You can check systemd output
after the failure to know which one was the one to blame.
In a short list, the dependencies for getty on ttyAMA0 would be:
getty@ttyAMA0.service
● ├─system-getty.slice
● └─sysinit.target
● ├─blk-availability.service
● ├─dev-hugepages.mount
● ├─dev-mqueue.mount
● ├─grub-initrd-fallback.service
● ├─keyboard-setup.service
● ├─kmod-static-nodes.service
● ├─proc-sys-fs-binfmt_misc.automount
● ├─resolvconf.service
● ├─setvtrgb.service
● ├─sys-fs-fuse-connections.mount
● ├─sys-kernel-config.mount
● ├─sys-kernel-debug.mount
● ├─systemd-ask-password-console.path
● ├─systemd-binfmt.service
● ├─systemd-hwdb-update.service
● ├─systemd-journal-flush.service
● ├─systemd-journald.service
● ├─systemd-machine-id-commit.service
● ├─systemd-modules-load.service
● ├─systemd-random-seed.service
● ├─systemd-sysctl.service
● ├─systemd-sysusers.service
● ├─systemd-tmpfiles-setup-dev.service
● ├─systemd-tmpfiles-setup.service
● ├─systemd-udev-trigger.service
● ├─systemd-udevd.service
● ├─systemd-update-utmp.service
● ├─cryptsetup.target
● ├─local-fs.target
● │ ├─-.mount
● │ ├─boot-efi.mount
● │ ├─systemd-fsck-root.service
● │ └─systemd-remount-fs.service
● └─swap.target
In a recursive list, you can see it as:
https://pastebin.ubuntu.com/p/qx27jrJwFh/
So, for now, concentrating efforts in the following output:
"""
start: cmd-install/stage-partitioning/builtin/cmd-block-meta: curtin command
block-meta
get_path_to_storage_volume for volume sdb
Processing serial 1510611512340057 via udev to 1510611512340057
TIMED BLOCK_META: 0.001
finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: curtin
command block-meta
Traceback (most recent call last):
File "/curtin/curtin/commands/main.py", line 202, in main
ret = args.func(args)
File "/curtin/curtin/log.py", line 97, in wrapper
return log_time("TIMED %s: " % msg, func, *args, **kwargs)
File "/curtin/curtin/log.py", line 79, in log_time
return func(*args, **kwargs)
File "/curtin/curtin/commands/block_meta.py", line 71, in block_meta
extract_storage_ordered_dict(cfg))
File "/curtin/curtin/commands/block_meta.py", line 1518, in
get_disk_paths_from_storage_config
for (k, v) in storage_config.items()
File "/curtin/curtin/commands/block_meta.py", line 1521, in
not config.value_as_boolean(v.get('preserve'))]
File "/curtin/curtin/commands/block_meta.py", line 1515, in
get_path_if_present
return get_path_to_storage_volume(disk, config)
File "/curtin/curtin/commands/block_meta.py", line 433, in
get_path_to_storage_volume
specified to identify disk")
ValueError: serial, wwn or path to block dev must be specified to identify disk
serial, wwn or path to block dev must be specified to identify disk
builtin command failed
"""
For this bug.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1833618
Title:
failing to deploy Ubuntu Disco
To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
** Changed in: sg3-utils (Ubuntu) Status: New => Incomplete ** Changed in: sg3-utils (Ubuntu) Status: Incomplete => Confirmed ** Changed in: sg3-utils (Ubuntu) Importance: Undecided => High ** Changed in: sg3-utils (Ubuntu) Assignee: (unassigned) => Rafael David Tinoco (rafaeldtinoco) ** Changed in: curtin Status: Triaged => Confirmed ** Changed in: curtin Importance: Undecided => High ** Changed in: curtin Assignee: (unassigned) => Rafael David Tinoco (rafaeldtinoco) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
MAAS deployments of disco to Marvell "Sabre" boards (CN99XX SoC) with more than one disk also fails. Internal node: * fails on "starbuck" with 2 disks * passes on "hotdog" and "apollo" server that both only have 1 disk -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
@Ryan, do you know who owns sg-utils? Is there any further information or system access that we can provide? Many thanks, Andy. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1833618] Re: failing to deploy Ubuntu Disco
On Thu, Sep 12, 2019 at 5:00 PM Patricia Domingues < patricia.doming...@canonical.com> wrote: > Ryan, > Sorry for delay, we had a BMC issue and it required a manual intervention > to fix it. > I've tested Eoan in that machine and it also fails, it doens't show the > same curtin error as Disco shows, but Eoan is not deployed. > It get stuck and MAAS shows `Marking node failed - Node operation > 'Deploying' timed out after 40 minutes.` > Attaching the console log for it (machine nominated as seidel). > Let me know if I can provide any other info. > > ** Attachment added: "eoan_console_seidel" > > https://bugs.launchpad.net/curtin/+bug/1833618/+attachment/5288410/+files/eoan_console_seidel Thanks. I expected it would fail as well. We'll need to dig into the sg-utils udev rules. Ryan -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
Ryan, Sorry for delay, we had a BMC issue and it required a manual intervention to fix it. I've tested Eoan in that machine and it also fails, it doens't show the same curtin error as Disco shows, but Eoan is not deployed. It get stuck and MAAS shows `Marking node failed - Node operation 'Deploying' timed out after 40 minutes.` Attaching the console log for it (machine nominated as seidel). Let me know if I can provide any other info. ** Attachment added: "eoan_console_seidel" https://bugs.launchpad.net/curtin/+bug/1833618/+attachment/5288410/+files/eoan_console_seidel -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1833618] Re: failing to deploy Ubuntu Disco
On Tue, Sep 3, 2019 at 9:31 AM Andrew Cloke wrote: > @Ryan, do you know if the Server team looks after sg3-utils-udev, or > should we be chasing the Foundations team to look into this? > Not quite sure, I'll follow up. Also, Patricia, can you try an Eoan install to see if this is maybe fix in newer release? Ryan > Thanks. > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1833618 > > Title: > failing to deploy Ubuntu Disco > > To manage notifications about this bug go to: > https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
@Ryan, do you know if the Server team looks after sg3-utils-udev, or should we be chasing the Foundations team to look into this? Thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
I think it worth mentioning, as the machine above that we are not able to deploy Disco is a Gigabyte(gbt-mt30)Cavium ThunderX 88XX, I've decided to test Disco in other machine, not Gigabyte, but Cavium ThunderX 88XX(CRB-2s) and I've noticed that in one machine Disco is successfully deployed and in another it fails. The only difference between these machines are the disks: -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
- it fails in a machine where it has more than one disk - in this case an USB stick. (machine nominated as recht) curtin: curtin_recht_failed_disco https://paste.ubuntu.com/p/NVkPnxVDn5/ ** Attachment added: "syslog_recht_failed_disco" https://bugs.launchpad.net/curtin/+bug/1833618/+attachment/5285882/+files/syslog_recht_failed_disco -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
- it is deployed in a machine where it has just one disk. (machine nominated as wright) curtin: curtin_wright_success_disco https://paste.ubuntu.com/p/7CmYw57Rn6/ ** Attachment added: "syslog_wright_success_disco" https://bugs.launchpad.net/curtin/+bug/1833618/+attachment/5285881/+files/syslog_wright_success_disco -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1833618] Re: failing to deploy Ubuntu Disco
It's actually the sg3-utils-udev package which provides new udev rules to inquery about scsi device's ID_SERIAL. ** Also affects: sg3-utils (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1833618 Title: failing to deploy Ubuntu Disco To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1833618/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
