Re: [systemd-devel] [dm-devel] RFC: one more time: SCSI device identification
On Wed, 2021-04-28 at 10:09 +1000, Erwin van Londen wrote: > > On Tue, 2021-04-27 at 16:41 -0400, Ewan D. Milne wrote: > > On Tue, 2021-04-27 at 20:33 +, Martin Wilck wrote: > > > On Tue, 2021-04-27 at 16:14 -0400, Ewan D. Milne wrote: > > > > There's no way to do that, in principle. Because there could > > > > be > > > > other I/Os in flight. You might (somehow) avoid retrying an > > > > I/O > > > > that got a UA until you figured out if something changed, but > > > > other > > > > I/Os can already have been sent to the target, or issued before > > > > you > > > > get to look at the status. > > If something happens on a storage side where a lun gets it's > attributes changed (any, doesn't matter which one) a UA should be > sent. Also all outstanding IO's on that lun should be returning an > Abort as it can no longer warrant the validity of any IO due to these > changes. Especially when parameters are involved like reservations > (PR's) etc. If that does not happen from an array side all bets are > off as the only way to be able to get back in business is to start > from scratch. Perhaps an array might abort I/Os it has received in the Device Server whensomething changes. I have no idea if most or any arrays actually do that. But, what about I/O that has already been queued from the host to thehost bus adapter? I don't see how we can abort those I/Os properly.Most high-performance HBAs have a queue of commands and a queueof responses, there could be lots of commands queued before wemanage to notice an interesting status. And AFAIK there is no conditionalmechanism that could hold them off (and, they could be in- flight on thewire anyway). I get what you are saying about what SAM describes, I just don't see howwe can guarantee we don't send any further commands after the statuswith the UA is sent back, before we can understand what happened. -Ewan > > > > > > Right. But in practice, a WWID change will hardly happen under > > > full > > > IO > > > load. The storage side will probably have to block IO while this > > > happens, at least for a short time period. So blocking and > > > quiescing > > > the queue upon an UA might still work, most of the time. Even if > > > we > > > were too late already, the sooner we stop the queue, the better. > > I think in most cases when something happens on an array side you > will see IO's being aborted. That might be a good time to start doing > TUR's and if these come back OK do a new inquiry. From a host side > there is only so much you can do. > > > > The current algorithm in multipath-tools needs to detect a path > > > going > > > down and being reinstated. The time interval during which a WWID > > > change > > > will go unnoticed is one or more path checker intervals, > > > typically on > > > the order of 5-30 seconds. If we could decrease this interval to > > > a > > > sub- > > > second or even millisecond range by blocking the queue in the > > > kernel > > > quickly, we'd have made a big step forward. > > > > Yes, and in many situations this may help. But in the general case > > we can't protect against a storage array misconfiguration, > > where something like this can happen. So I worry about people > > believing the host software will protect them against a mistake, > > when we can't really do that. > > My thought exactly. > > > All it takes is one I/O (a discard) to make a thorough mess of the > > LUN. > > > > -Ewan > > > > > Regards > > > Martin > > > > > > > -- > > dm-devel mailing list > > dm-de...@redhat.com > > https://listman.redhat.com/mailman/listinfo/dm-devel > > > > --dm-devel mailing listdm-de...@redhat.com > https://listman.redhat.com/mailman/listinfo/dm-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
--On Friday, April 30, 2021 11:39 AM -0400 Rick Winscot wrote: Early in the project it was decided to make the rootfs read-only... in an effort to improve durability in environments where power fluctuations might cause problems on the eMMC. At the same time, making logging (e.g. /var) persistent for debugging was added to requirements. Persistent storage would be achieved by mounting /var to a separate partition that is read-write. Does /etc need to be read-only? On my last server I decided to make /usr read-only but root is writable and /var is part of that. I put /home on its own partition. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot : > At this point, flush is attempting to re-route /run/log/journal to > /var/log/journal ... and the /var partition is not yet mounted. Units > generated for fstab in /run/systemd/generator that manage the mount have an > After=local-fs-pre.target which is too late. > > Other dependency errors appear in the log; all with the same root cause. By > the time [ a specified service ] that uses /var is ready, the partition has > not yet been mounted. My solution resolves the /var mount as soon as the > block device is seen by udev - which made all the dependency errors go away. Fwiw, I can't reproduce the problem. systemd-journal-flush.service is correctly started after /var has been mounted. In case you are interested, I attached a journalctl dump, /etc/fstab and systemd-analyze dump as well. As you can see, systemd-journal-flush.service has a proper After=var.mount ordering. I wonder if you have a dependency loop somewhere and systemd resolves this by removing that ordering. fstab Description: Binary data journal.txt.gz Description: application/gzip systemd-analyze.dump.txt.gz Description: application/gzip ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
15:31:18 localhost systemd[1]: Dependency failed for Flush Journal to Persistent Storage. ░░ Subject: A start job for unit systemd-journal-flush.service has failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit systemd-journal-flush.service has finished with a failure. ░░ ░░ The job identifier is 38 and the job result is dependency. At this point, flush is attempting to re-route /run/log/journal to /var/log/journal ... and the /var partition is not yet mounted. Units generated for fstab in /run/systemd/generator that manage the mount have an After=local-fs-pre.target which is too late. Other dependency errors appear in the log; all with the same root cause. By the time [ a specified service ] that uses /var is ready, the partition has not yet been mounted. My solution resolves the /var mount as soon as the block device is seen by udev - which made all the dependency errors go away. On Fri, Apr 30, 2021 at 1:46 PM Michael Biebl wrote: > Am Fr., 30. Apr. 2021 um 19:42 Uhr schrieb Rick Winscot > : > > > > systemd 247 > > Ok, thanks > > > /etc/systemd/journald.conf storage is persistent, > systemd-journal-flush.service has RequiresMountsFor=/var/log/journal. > > > > Mounting /var on a separate read-write partition handles the persistent > log requirement as well as offloading other read-write operations that can > no longer live on the rootfs... being read-only. > > Sure, but what is the actual problem? I do have systems with a > separate /var and don't remember any ordering issues / race > conditions. > Do you have any error messages / journal logs which show the issue? > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
Am Fr., 30. Apr. 2021 um 19:42 Uhr schrieb Rick Winscot : > > systemd 247 Ok, thanks > /etc/systemd/journald.conf storage is persistent, > systemd-journal-flush.service has RequiresMountsFor=/var/log/journal. > > Mounting /var on a separate read-write partition handles the persistent log > requirement as well as offloading other read-write operations that can no > longer live on the rootfs... being read-only. Sure, but what is the actual problem? I do have systems with a separate /var and don't remember any ordering issues / race conditions. Do you have any error messages / journal logs which show the issue? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
systemd 247 /etc/systemd/journald.conf storage is persistent, systemd-journal-flush.service has RequiresMountsFor=/var/log/journal. Mounting /var on a separate read-write partition handles the persistent log requirement as well as offloading other read-write operations that can no longer live on the rootfs... being read-only. On Fri, Apr 30, 2021 at 11:45 AM Michael Biebl wrote: > What is the actual problem you have with a separate /var and > systemd-journald? > For completeness sake, which systemd version do you have? > > Am Fr., 30. Apr. 2021 um 16:39 Uhr schrieb Rick Winscot > : > > > > We have an embedded product that uses a minimal Linux distribution > generated via Buildroot. > > > > Early in the project it was decided to make the rootfs read-only... in > an effort to improve durability in environments where power fluctuations > might cause problems on the eMMC. At the same time, making logging (e.g. > /var) persistent for debugging was added to requirements. Persistent > storage would be achieved by mounting /var to a separate partition that is > read-write. > > > > Several-hundred hours later... with many systemd-analyze reports and > various configurations tested, we have determined that managing the /var > mount with stadard services is not going to work due to tightly coupled and > precisely timed dependencies. Attempts with /etc/fstab and the systemd > generator are also unstable. > > > > Getting /var mounted in proximity to the initialization of > systemd-journald.service seemed illusive. > > > > Several days ago I found a post on Stackoverflow that tied into udev > triggers that seemed promising; resulting in the method outlined below. > Initial testing shows proper timing with all dependencies satisfied. > However, the solution feels... hackey. > > > > My question for anyone on the list, is the method outlined below a > reasonable solution to mounting /var early in the start-up cycle? > > > > Or... is there a better way? Some trimming > > > > > > Step One: Create a systemd service that mounts /var to the specified > partition > > Service: /etc/systemd/system/var.service > > > > [Unit] > > Description=service for mounting /var > > DefaultDependencies=no > > > > [Service] > > Type=oneshot > > RemainAfterExit=yes > > ExecStart=/bin/mount /dev/mmcblk0p6 > > > > > > Step Two: Add a nofail mount > > fstab:/etc/fstab > > > > /dev/root / auto rw 0 1 > > /dev/mmcblk0p6 /var ext4 rw,nofail 0 0 > > > > > > Step Three: Add a wants dependency on the mount in udev triggers (some > lines deleted for brevity) > > Service:/usr/lib/systemd/system/systemd-udev-trigger.service > > > > [Unit] > > ... > > Wants=systemd-udevd.service var.service > > > > [Service] > > ... > > ExecStart=udevadm trigger --type=subsystems --action=add ; > /usr/bin/udevadm trigger > > > > > > Finally, systemd-analyze plot shows that the mount works as desired. > > > > systemd-udev-trigger.service > > var.service > > dev-mmcblk0p6.device > > var.mount > > > > systemd-remount-fs.service > > systemd-journal-flush.service > > local-fs-pre.target > > > > ___ > > systemd-devel mailing list > > systemd-devel@lists.freedesktop.org > > https://lists.freedesktop.org/mailman/listinfo/systemd-devel > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] systemd-run / Failed to create bus connection: Input/output error
Hi guys. I'm do on my pretty vanilla, so I'd like to think, setup this: -> $ systemd-run --machine=qemu-8-c8kubernode1 /bin/cat /etc/centos-release Failed to create bus connection: Input/output error Someone would care to decipher that for me or/and shed bit more light on possible troubleshooting? many thanks, L. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
Am Freitag, dem 30.04.2021 um 10:39 -0400 schrieb Rick Winscot: > My question for anyone on the list, is the method outlined below a > reasonable solution to mounting /var early in the start-up cycle? > > Or... is there a better way? Some trimming > Hi Rick, by definition if you need to mount /var (or /usr for this argument), you need an initrd [1] which actually set up everything as you requires. Anything else has a tendency to break in unpleasant ways due to race conditions and such. You don't need much, just enough to set up everything required for the root and API file systems. BR Silvio [1] https://systemd.io/INITRD_INTERFACE/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
On Fri, 30 Apr 2021 at 16:45, Rick Winscot wrote: > > We have an embedded product that uses a minimal Linux distribution generated via Buildroot. > > Early in the project it was decided to make the rootfs read-only... in an effort to improve durability in environments where power fluctuations might cause problems on the eMMC. At the same time, making logging (e.g. /var) persistent for debugging was added to requirements. Persistent storage would be achieved by mounting /var to a separate partition that is read-write. > > Several-hundred hours later... with many systemd-analyze reports and various configurations tested, we have determined that managing the /var mount with stadard services is not going to work due to tightly coupled and precisely timed dependencies. Attempts with /etc/fstab and the systemd generator are also unstable. > > Getting /var mounted in proximity to the initialization of systemd-journald.service seemed illusive. > > Several days ago I found a post on Stackoverflow that tied into udev triggers that seemed promising; resulting in the method outlined below. Initial testing shows proper timing with all dependencies satisfied. However, the solution feels... hackey. > > My question for anyone on the list, is the method outlined below a reasonable solution to mounting /var early in the start-up cycle? > > Or... is there a better way? Some trimming > > > Step One: Create a systemd service that mounts /var to the specified partition > Service: /etc/systemd/system/var.service > > [Unit] > Description=service for mounting /var > DefaultDependencies=no > > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStart=/bin/mount /dev/mmcblk0p6 > > > Step Two: Add a nofail mount > fstab:/etc/fstab > > /dev/root / auto rw 0 1 > /dev/mmcblk0p6 /var ext4 rw,nofail 0 0 > > > Step Three: Add a wants dependency on the mount in udev triggers (some lines deleted for brevity) > Service:/usr/lib/systemd/system/systemd-udev-trigger.service > > [Unit] > ... > Wants=systemd-udevd.service var.service > > [Service] > ... > ExecStart=udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger > > > Finally, systemd-analyze plot shows that the mount works as desired. > > systemd-udev-trigger.service > var.service > dev-mmcblk0p6.device > var.mount > > systemd-remount-fs.service > systemd-journal-flush.service > local-fs-pre.target > > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel You don't need to synchronize with the actual journald unit - you need to synchronize with systemd-journal-flush.service which is what moves the journal from /run/log to /var/log. The default flush unit is ordered with systemd-tmpfiles-setup which is also somewhat early-boot - what I do is mask systemd-journal-flush.service, and provide my own without that ordering constraint. If you mount your /var via mount units/fstab generator, then RequiresMountsFor=/var/log/journal will take care of the ordering automagically. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] early mounts in systemd
What is the actual problem you have with a separate /var and systemd-journald? For completeness sake, which systemd version do you have? Am Fr., 30. Apr. 2021 um 16:39 Uhr schrieb Rick Winscot : > > We have an embedded product that uses a minimal Linux distribution generated > via Buildroot. > > Early in the project it was decided to make the rootfs read-only... in an > effort to improve durability in environments where power fluctuations might > cause problems on the eMMC. At the same time, making logging (e.g. /var) > persistent for debugging was added to requirements. Persistent storage would > be achieved by mounting /var to a separate partition that is read-write. > > Several-hundred hours later... with many systemd-analyze reports and various > configurations tested, we have determined that managing the /var mount with > stadard services is not going to work due to tightly coupled and precisely > timed dependencies. Attempts with /etc/fstab and the systemd generator are > also unstable. > > Getting /var mounted in proximity to the initialization of > systemd-journald.service seemed illusive. > > Several days ago I found a post on Stackoverflow that tied into udev triggers > that seemed promising; resulting in the method outlined below. Initial > testing shows proper timing with all dependencies satisfied. However, the > solution feels... hackey. > > My question for anyone on the list, is the method outlined below a reasonable > solution to mounting /var early in the start-up cycle? > > Or... is there a better way? Some trimming > > > Step One: Create a systemd service that mounts /var to the specified partition > Service: /etc/systemd/system/var.service > > [Unit] > Description=service for mounting /var > DefaultDependencies=no > > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStart=/bin/mount /dev/mmcblk0p6 > > > Step Two: Add a nofail mount > fstab:/etc/fstab > > /dev/root / auto rw 0 1 > /dev/mmcblk0p6 /var ext4 rw,nofail 0 0 > > > Step Three: Add a wants dependency on the mount in udev triggers (some lines > deleted for brevity) > Service:/usr/lib/systemd/system/systemd-udev-trigger.service > > [Unit] > ... > Wants=systemd-udevd.service var.service > > [Service] > ... > ExecStart=udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm > trigger > > > Finally, systemd-analyze plot shows that the mount works as desired. > > systemd-udev-trigger.service > var.service > dev-mmcblk0p6.device > var.mount > > systemd-remount-fs.service > systemd-journal-flush.service > local-fs-pre.target > > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] early mounts in systemd
We have an embedded product that uses a minimal Linux distribution generated via Buildroot. Early in the project it was decided to make the rootfs read-only... in an effort to improve durability in environments where power fluctuations might cause problems on the eMMC. At the same time, making logging (e.g. /var) persistent for debugging was added to requirements. Persistent storage would be achieved by mounting /var to a separate partition that is read-write. Several-hundred hours later... with many systemd-analyze reports and various configurations tested, we have determined that managing the /var mount with stadard services is not going to work due to tightly coupled and precisely timed dependencies. Attempts with /etc/fstab and the systemd generator are also unstable. Getting /var mounted in proximity to the initialization of systemd-journald.service seemed illusive. Several days ago I found a post on Stackoverflow that tied into udev triggers that seemed promising; resulting in the method outlined below. Initial testing shows proper timing with all dependencies satisfied. However, the solution feels... hackey. My question for anyone on the list, is the method outlined below a reasonable solution to mounting /var early in the start-up cycle? Or... is there a better way? Some trimming Step One: Create a systemd service that mounts /var to the specified partition Service: /etc/systemd/system/var.service [Unit] Description=service for mounting /var DefaultDependencies=no [Service] Type=oneshot RemainAfterExit=yes ExecStart=/bin/mount /dev/mmcblk0p6 Step Two: Add a nofail mount fstab:/etc/fstab /dev/root / auto rw 0 1 /dev/mmcblk0p6 /var ext4 rw,nofail 0 0 Step Three: Add a wants dependency on the mount in udev triggers (some lines deleted for brevity) Service:/usr/lib/systemd/system/systemd-udev-trigger.service [Unit] ... Wants=systemd-udevd.service var.service [Service] ... ExecStart=udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger Finally, systemd-analyze plot shows that the mount works as desired. systemd-udev-trigger.service var.service dev-mmcblk0p6.device var.mount systemd-remount-fs.service systemd-journal-flush.service local-fs-pre.target ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemctl reboot get terminated by signal 15
On Fr, 30.04.21 02:00, Pengpeng Sun (pengpe...@vmware.com) wrote: > > > Here is where our code calls “/sbin/telinit 6” to reboot Linux. > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvmware%2Fopen-vm-tools%2Fblob%2Fmaster%2Fopen-vm-tools%2FlibDeployPkg%2FlinuxDeployment.c%23L1466data=04%7C01%7Cpengpengs%40vmware.com%7C703b24bd11584d4d033d08d90b4a1ad9%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637553235200829094%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=DWWAKyPLV6dqnbJ37%2FFVqpAYapYvVC57reiYTSflFH4%3Dreserved=0 > > > This code is executed when systemd vmtoolsd.service starts, attached the > > > vmtoolsd.service file. > > > Not seeing this issue before /sbin/telinit becomes a softlink to > > > systemctl. > > > > vmtoolsd.service is probably asked to shutdown because of the system > > shutdown, and the forked off /sbin/telinit is part of that service, so > > it gets terminated too? > > Yes, this could be the reason. The issue is system does NOT always reboot > after > “uncaught signal 15” happens, while sometimes it does reboot during my local > testing. Is there a way/command to make sure system get rebooted? Check the logs? https://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1 Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
