[Bug 1856871] Re: i/o error if next unused loop device is queried
** Changed in: linux (Ubuntu) Assignee: Eric Desrochers (slashd) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
** Changed in: linux (Ubuntu) Status: Incomplete => In Progress ** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Mauricio Faria de Oliveira (mfo) ** Changed in: parted (Ubuntu) Status: New => Invalid ** Changed in: linux (Ubuntu) Assignee: Mauricio Faria de Oliveira (mfo) => Eric Desrochers (slashd) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
The upstream proposal fix that mfo and I worked on has been applied: https://patchwork.kernel.org/project/linux- block/patch/20210222154123.61797-1-...@canonical.com/ - Eric -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
** Changed in: udev (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
marking invalid for systemd, as i think we all agree it's not an issue caused by systemd, but please feel free to re-mark it for systemd if i'm incorrect ** Changed in: systemd (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
I reproduced the behaviour using 5.5 upstream kernel by: 1) Mounting a loop device 2) Setup frace for all loop function for capture purposes 3) Then umount the loop device trace_pipe reveal the following: "umount-1850 [000] 471.727511: loop_release_xfer <-__loop_clr_fd" As cascardo mentioned earlier it might be in the way that loop device are detached, now that I know what function to look at, I'll investigate further. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
I reproduced the behaviour using 5.5 upstream kernel by: 1) Mounting a loop device 2) Setup frace for all loop function for capture purposes 3) Then umount the loop device trace_pipe reveal the following: "umount-1850 [000] 471.727511: loop_release_xfer <-__loop_clr_fd" As cascardo mentioned earlier it might be in the way that loop device are clear, now that I know what function to look at, I'll investigate further. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
so 2 things come to my mind right now Does the kernel (loop driver) do the right thing by not clear/reset the loop device' stats after the umount/detached operation ? and/or Does parted need to be smarter and not only based is detection on stat() to assume if its a legit device or not to be probed ? Let's circle back on Jan 2020. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
so 2 things come to my mind right now Does the kernel (loop driver) do the right thing by not clear/reset the loop device' stats after the umount/detached operation ? and/or Does parted need to smarter and not only based is detection only on stat() ? Let's circle back on Jan 2020. ** Also affects: parted (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
Ok so in fact the inconsistency is due to "parted" that will try to
probe devices iff the given block device return informations from
stat(), regardless if the block device is available or not.
Seems like the only trigger/criteria is the stat() return. Since the
loop device stat doesn't clear/reset once umounted/detacted, then parted
assume it can go ahead and probe it assuming it is a legit device.
# libparted/arch/linux.c
618 static int
619 _device_stat (PedDevice* dev, struct stat * dev_stat)
620 {
621 PED_ASSERT (dev != NULL);
622 PED_ASSERT (!dev->external_mode);
623
624 while (1) {
625 if (!stat (dev->path, dev_stat)) {
626 return 1;
627 } else {
628 if (ped_exception_throw (
629 PED_EXCEPTION_ERROR,
630 PED_EXCEPTION_RETRY_CANCEL,
631 _("Could not stat device %s - %s."),
632 dev->path,
633 strerror (errno))
634 != PED_EXCEPTION_RETRY)
635 return 0;
636 }
637 }
638 }
Example:
# parted -s $(losetup -f) unit s print
Warning: Error fsyncing/closing /dev/loop18: Input/output error
# stat $(losetup -f)
File: /dev/loop18
Size: 0 Blocks: 0 IO Block: 4096 block special file
Device: 6h/6d Inode: 509 Links: 1 Device type: 7,12
Access: (0660/brw-rw) Uid: (0/root) Gid: (6/disk)
Access: 2019-12-25 13:05:15.543108777 -0500
Modify: 2019-12-25 13:05:15.543108777 -0500
Change: 2019-12-25 13:05:15.543108777 -0500
Birth: -
# parted -s /dev/loop19 unit s print
Error: Could not stat device /dev/loop19 - No such file or directory.
# stat /dev/loop19
stat: cannot stat '/dev/loop19': No such file or directory
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1856871
Title:
i/o error if next unused loop device is queried
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
Agreed with your comment #11 will start a discussion with the linux-block maintainer. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
It then might be a problem with "/dev/loop-control" device node. Which
dynamically find or allocate a free device, but also add and remove loop
devices from the running system.
# drivers/block/loop.c
2090 static void loop_remove(struct loop_device *lo)
2091 {
2092 del_gendisk(lo->lo_disk);
2093 blk_cleanup_queue(lo->lo_queue);
2094 blk_mq_free_tag_set(>tag_set);
2095 put_disk(lo->lo_disk);
2096 kfree(lo);
2097 }
2177 case LOOP_CTL_REMOVE:
2178 ret = loop_lookup(, parm);
2179 if (ret < 0)
2180 break;
2181 if (lo->lo_state != Lo_unbound) {
2182 ret = -EBUSY;
2183 break;
2184 }
2185 if (atomic_read(>lo_refcnt) > 0) {
2186 ret = -EBUSY;
2187 break;
2188 }
2189 lo->lo_disk->private_data = NULL;
2190 idr_remove(_index_idr, lo->lo_number);
2191 loop_remove(lo);
2192 break;
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1856871
Title:
i/o error if next unused loop device is queried
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
Exactly my point about consistency. They should either return EIO in both cases or succeed in both cases. Which behavior will depend on: 1) which is the easier solution; 2) what upstream thinks is okay. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
@thadeu, To respond your question "What is the exact problem this is causing?" So far it's not causing much problem, it's pretty harmless, but while running sosreport block plugin (which most Canonical customer uses) it may lead to output "blk" error to the stderr and save syslog || kern.log. As an fyi, we are re-working the sosreport block plugin upstream to prevent to query unused disk, but still, while I agree with your EIO statement, I don't think a detached loop device should behave differently from a "never" detached loop device, if there current state is "unused" unless I missed something. IMHO they should produce consistent behaviour if 'unused' no ? - Eric ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
so the problem is inside the block layer ? More likely the loop driver not undoing something it does when a file is attached to it. But could as well be something related to the multiqueue support, so could still be something in the block layer. Still needs investigation. Anyway, as this is not exactly a difference in behavior between the next available loop device and other detached loop devices, what is the exact problem this is causing? I don't see why getting EIO for a detached loop device is the wrong behavior here. I agree there is an inconsistency, but I would accept EIO when trying to fsync a detached device. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
I did some quick testing and it seems this happens on whatever loop device that had a file attached and then detached. No matter if it's the next available or not. So, if you create a new loop device without ever attaching a file to it, it seems the block layer is not setup sufficiently so any requests will really go through it. But when it is attached, then detached, the block layer still sends requests to the loop driver, which will result in the EIO as it is detached. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
FWIW snapd mounts and unmounts a squashfs on startup to determine whether the system can mount squashfs's. It does this using mount, and cleans up with umount -l. That is, it's not via systemd in this instance in particular. I'm setting it as invalid for snapd, but if this behaviour is somehow tickling a bug and there's a workaround that could avoid it, let us know and we'd be happy to accommodate (set the bug task back to New so our triage picks it up). ** Changed in: snapd (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
# parted code (3.3-1) from focal
-
libparted/arch/linux.c:
.
=> if (fsync (fd) < 0 || close (fd) < 0)
if (ped_exception_throw (
PED_EXCEPTION_WARNING,
PED_EXCEPTION_RETRY +
PED_EXCEPTION_IGNORE,
=> _("Error fsyncing/closing %s:
%s"),
name, strerror (errno))
== PED_EXCEPTION_RETRY)
goto retry;
}
}
free (name);
# FSYNC(2)
RETURN VALUE
On success, these system calls return zero. On error, -1 is returned,
and errno is set appropriately.
-
so definitely fsync() fails returning "-1" on the next unused loop
device as oppose to any other unused loop device.
The question is why unused loop device fails on fsync() ONLY when it is
the next unused loop device after a system boot (losetup -f) ?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1856871
Title:
i/o error if next unused loop device is queried
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1856871/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
losetup -l
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
DIO LOG-SEC
/dev/loop1 0 0 1 1 /var/lib/snapd/snaps/core_7270.snap
0 512
/dev/loop2 0 0 1 1 /var/lib/snapd/snaps/core_8268.snap
0 512
/dev/loop0 0 0 1 1 /var/lib/snapd/snaps/firefox_287.snap
0 512
# losetup -f
/dev/loop3
# strace parted -s $(losetup -f) unit s print
ioctl(3, BLKSSZGET, [512]) = 0
fadvise64(3, 0, 0, POSIX_FADV_RANDOM) = 0
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 3), ...}) = 0
ioctl(3, BLKGETSIZE64, [0]) = 0
openat(AT_FDCWD, "/sys/dev/block/7:3", O_RDONLY|O_CLOEXEC) = 4
openat(4, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(4)= 0
ioctl(3, BLKALIGNOFF, [0]) = 0
ioctl(3, BLKIOMIN, [512]) = 0
ioctl(3, BLKIOOPT, [0]) = 0
ioctl(3, BLKPBSZGET, [512]) = 0
ioctl(3, BLKSSZGET, [512]) = 0
ioctl(3, BLKGETSIZE64, [0]) = 0
ioctl(3, BLKGETSIZE64, [0]) = 0
fsync(3)= -1 EIO (Input/output error)
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/libc.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/libc.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/libc.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/libc.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en.utf8/LC_MESSAGES/libc.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/libc.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
write(2, "Warning", 7Warning) = 7
write(2, ": ", 2: ) = 2
write(2, "Error fsyncing/closing /dev/loop"..., 53Error fsyncing/closing
/dev/loop3: Input/output error) = 53
write(2, "\n", 1
) = 1
close(1)= 0
close(2)= 0
exit_group(1) = ?
+++ exited with 1 +++
# strace parted -s /dev/loop5 unit s print
ioctl(3, BLKSSZGET, [512]) = 0
fadvise64(3, 0, 0, POSIX_FADV_RANDOM) = 0
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 5), ...}) = 0
ioctl(3, BLKGETSIZE64, [0]) = 0
openat(AT_FDCWD, "/sys/dev/block/7:5", O_RDONLY|O_CLOEXEC) = 4
openat(4, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(4)= 0
ioctl(3, BLKALIGNOFF, [0]) = 0
ioctl(3, BLKIOMIN, [512]) = 0
ioctl(3, BLKIOOPT, [0]) = 0
ioctl(3, BLKPBSZGET, [512]) = 0
ioctl(3, BLKSSZGET, [512]) = 0
ioctl(3, BLKGETSIZE64, [0]) = 0
ioctl(3, BLKGETSIZE64, [0]) = 0
fsync(3)= 0
close(3)= 0
close(1)= 0
close(2)= 0
exit_group(1) = ?
+++ exited with 1 +++
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1856871
Title:
i/o error if next unused loop device is queried
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1856871/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
Talking with mvo (snap team) on the subject trying to eliminate some components: [09:48:27] so snapd does not do anything "special" with block devices beside using them [09:48:39] it uses the standard system mount units to do that [09:49:01] so if you see strange things with extra block devices, I'm inclined to point my finger at systemd :/ ** Also affects: snapd (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
Interesting fact, If I modified "snapd.service" to prevent snapd to run and reboot. I don't get the problem with the first unused loop device which is /dev/loop0 since nothing is using a loop device. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
** Description changed: This is reproducible in Bionic and late. Here's an example running 'focal': $ lsb_release -cs focal $ uname -r 5.3.0-24-generic The error is: blk_update_request: I/O error, dev loop2, sector 0 + + and on more recent kernel: + + kernel: [18135.185709] blk_update_request: I/O error, dev loop18, sector + 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 + How to trigger it: $ sosreport -o block or more precisely the cmd causing the situation inside the block plugin: $ parted -s $(losetup -f) unit s print https://github.com/sosreport/sos/blob/master/sos/plugins/block.py#L52 but if I run it on the next next unused loop device, in this case /dev/loop3 (which is also unused), no errors. While I agree that sosreport shouldn't query unused loop devices, there is definitely something going on with the next unused loop device. What is differentiate loop2 and loop3 and any other unused ones ? 3 things so far I have noticed: * loop2 is the next unused loop device (losetup -f) * A reboot is needed (if some loop modification (snap install, mount loop, ...) has been made at runtime * I have also noticed that loop2 (or whatever the next unused one is) have some stat as oppose to other unused loop devices. The stat exist already right after the system boot for the next unused loop device. /sys/block/loop2/stat :: 2 0 10 0 1 0 0 0 0 0 0 2 = number of read I/Os processed 10 = number of sectors read 1 = number of write I/Os processed Explanation of each column: https://www.kernel.org/doc/html/latest/block/stat.html while /dev/loop3 doesn't /sys/block/loop3/stat :: 0 0 0 0 0 0 0 0 0 0 0 Which tells me that something during the boot process most likely acquired (on purpose or not) the next unused loop and possibly didn't released it well enough. If loop2 is generating errors, and I install a snap, the snap squashfs will take loop2, making loop3 the next unused loop device. If I query loop3 with 'parted' right after, no errors. If I reboot, and query loop3 again, then no I'll have an error. To triggers the errors it need to be after a reboot and it only impact the first unused loop device available (losetup -f). This was tested with focal/systemd whic his very close to latest upstream code. This has been test with latest v5.5 mainline kernel as well. For now, I don't think it's a kernel problem, I'm more thinking of a userspace misbehaviour dealing with loop device (or block device) at boot. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
I also filed a sosreport upstream issue, because I believe sosreport shouldn't try to query "unused" block device such as loop device. https://github.com/sosreport/sos/issues/1897 but situation remain that there is something going wrong with loop device and systemd/systemd-udevd at boot as explained above. ** Bug watch added: github.com/sosreport/sos/issues #1897 https://github.com/sosreport/sos/issues/1897 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1856871] Re: i/o error if next unused loop device is queried
Booting with "systemd.log_level=debug" revealed:
Dec 18 17:26:22 ubuntu systemd-udevd[1326]: value '[dmi/id]sys_vendor' is 'QEMU'
Dec 18 17:26:22 ubuntu systemd-udevd[1326]: created db file
'/run/udev/data/b7:2' for '/devices/virtual/block/loop2'
Dec 18 17:26:22 ubuntu systemd[1]: dev-loop2.device: Changed dead -> tentative
Dec 18 17:26:22 ubuntu systemd-udevd[1326]: created db file
'/run/udev/data/b7:2' for '/devices/virtual/block/loop2'
Dec 18 17:26:22 ubuntu systemd-udevd[1326]: passed 293 byte device to netlink
monitor 0x557a8c0f8680
Dec 18 17:26:22 ubuntu systemd[1120]: dev-loop2.device: Changed dead ->
tentative
...
Dec 18 17:26:22 ubuntu systemd[1120]: sys-devices-virtual-block-loop2.device:
Changed dead -> plugged
Dec 18 17:26:22 ubuntu systemd[1120]: dev-loop2.device: Changed tentative ->
plugged
Dec 18 17:26:22 ubuntu systemd-udevd[529]: passed 179 byte device to netlink
monitor 0x557a8c0c7bc0
Dec 18 17:26:22 ubuntu systemd[1]: sys-devices-virtual-block-loop2.device:
Changed dead -> plugged
Dec 18 17:26:22 ubuntu systemd[1]: dev-loop2.device: Changed tentative ->
plugged
...
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a path=/org/freedesktop/systemd1/unit/dev_2dloop2_2edevice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=1009
reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a path=/org/freedesktop/systemd1/unit/dev_2dloop2_2edevice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=1010
reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
...
Dec 18 17:26:22 ubuntu systemd-udevd[1328]: created db file
'/run/udev/data/b7:2' for '/devices/virtual/block/loop2'
Dec 18 17:26:22 ubuntu systemd-udevd[1328]: created db file
'/run/udev/data/b7:2' for '/devices/virtual/block/loop2'
Dec 18 17:26:22 ubuntu systemd-udevd[1328]: passed 252 byte device to netlink
monitor 0x557a8c0f8680
Dec 18 17:26:22 ubuntu systemd[1]: sys-devices-virtual-block-loop2.device:
Installed new job sys-devices-virtual-block-loop2.device/nop as 412
Dec 18 17:26:22 ubuntu systemd[1]: dev-loop2.device: Installed new job
dev-loop2.device/nop as 413
Dec 18 17:26:22 ubuntu systemd[1]: sys-devices-virtual-block-loop2.device:
Changed plugged -> dead
Dec 18 17:26:22 ubuntu systemd[1120]: sys-devices-virtual-block-loop2.device:
Installed new job sys-devices-virtual-block-loop2.device/nop as 14
Dec 18 17:26:22 ubuntu systemd[1]: dev-loop2.device: Changed plugged -> dead
Dec 18 17:26:22 ubuntu systemd[1120]: dev-loop2.device: Installed new job
dev-loop2.device/nop as 15
Dec 18 17:26:22 ubuntu systemd[1120]: sys-devices-virtual-block-loop2.device:
Changed plugged -> dead
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a path=/org/freedesktop/systemd1/unit/dev_2dloop2_2edevice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=1028
reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Dec 18 17:26:22 ubuntu systemd[1120]: dev-loop2.device: Changed plugged -> dead
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a path=/org/freedesktop/systemd1/unit/dev_2dloop2_2edevice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=1029
reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Dec 18 17:26:22 ubuntu systemd[1120]: dev-loop2.device: Job
dev-loop2.device/nop finished, result=done
Dec 18 17:26:22 ubuntu systemd[1120]: sys-devices-virtual-block-loop2.device:
Job sys-devices-virtual-block-loop2.device/nop finished, result=done
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a
path=/org/freedesktop/systemd1/unit/sys_2ddevices_2dvirtual_2dblock_2dloop2_2edevice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=1030
reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Dec 18 17:26:22 ubuntu systemd[1120]: sys-devices-virtual-block-loop2.device:
Collecting.
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a
path=/org/freedesktop/systemd1/unit/sys_2ddevices_2dvirtual_2dblock_2dloop2_2edevice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=1031
reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Dec 18 17:26:22 ubuntu systemd[1120]: dev-loop2.device: Collecting.
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a path=/org/freedesktop/systemd1
interface=org.freedesktop.systemd1.Manager member=JobNew cookie=1032
reply_cookie=0 signature=uos error-name=n/a error-message=n/a
Dec 18 17:26:22 ubuntu systemd[1]: Sent message type=signal sender=n/a
destination=n/a path=/org/freedesktop/systemd1
interface=org.freedesktop.systemd1.Manager member=JobNew cookie=1033
reply_cookie=0
[Bug 1856871] Re: i/o error if next unused loop device is queried
** Description changed: This is reproducible in Bionic and late. Here's an example running 'focal': $ lsb_release -cs focal $ uname -r 5.3.0-24-generic + The error is: + blk_update_request: I/O error, dev loop2, sector 0 + How to trigger it: - $ sosreport -o block - or more precisely the command causing the situation inside the block plugin: + or more precisely the cmd causing the situation inside the block plugin: $ parted -s /dev/$(losetup -f) unit s print https://github.com/sosreport/sos/blob/master/sos/plugins/block.py#L52 but if I run it on the next next unused loop device, in this case /dev/loop3 (which is also unused), no errors. While I agree that sosreport shouldn't query unused loop devices, there is definitely something going on with the next unused loop device. - What is the difference between loop2 and loop3 and other unused one ? + What is differentiate loop2 and loop3 and any other unused ones ? 3 things so far I have noticed: - * The loop device need to be the next unused loop device (losetup -f) + * loop2 is the next unused loop device (losetup -f) * A reboot is needed (if some loop modification (snap install, mount loop, ...) has been made at runtime * I have also noticed that loop2 (or whatever the next unused one is) have some stat as oppose to other unused loop devices. The stat exist already right after the system boot for the next unused loop device. /sys/block/loop2/stat :: 2 0 10 0 1 0 0 0 0 0 0 2 = number of read I/Os processed - 10 = number of sectors read + 10 = number of sectors read 1 = number of write I/Os processed Explanation of each column: https://www.kernel.org/doc/html/latest/block/stat.html while /dev/loop3 doesn't /sys/block/loop3/stat :: 0 0 0 0 0 0 0 0 0 0 0 - - Which tells me that something during the boot process most likely acquired (on purpose or not) the next unused loop and possibly didn't released it well. + Which tells me that something during the boot process most likely + acquired (on purpose or not) the next unused loop and possibly didn't + released it well enough. If loop2 is generating errors, and I install a snap, the snap squashfs will take loop2, making loop3 the next unused loop device. If I query loop3 with 'parted' right after, no errors. If I reboot, and query loop3 again, then no I'll have an error. To triggers the errors it need to be after a reboot and it only impact the first unused loop device available (losetup -f). - This was tested with focal/systemd whic his very close to latest upstream code. - This has been test with latest v5.5 kernel as well. For now, I don't think it's a kernel problem, I'm more thinking of a userspace misbehaviour dealing with loop device at boot. + This was tested with focal/systemd whic his very close to latest + upstream code. + + This has been test with latest v5.5 mainline kernel as well. + + For now, I don't think it's a kernel problem, I'm more thinking of a + userspace misbehaviour dealing with loop device (or block device) at + boot. ** Description changed: -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1856871 Title: i/o error if next unused loop device is queried To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1856871/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
