Aria, I've had in our build stuff since the sles 12 days to put this in /etc/default/grub GRUB_DISABLE_LINUX_UUID=true
I think that might help your boot duplicate issues? My brain cells that contained why I did that seem to have been reused for another purpose though, so maybe it was something else. I know it had to do with clones. Marcy -----Original Message----- From: Linux on 390 Port <[email protected]> On Behalf Of Aria Bamdad Sent: Wednesday, October 13, 2021 2:00 PM To: [email protected] Subject: Re: [LINUX-390] Warning if upgrading SLES12 to SLES15 SP3 Peter, thank you again for an amazing analysis of what is happening. Your udev rules idea is an interesting one. I do use this to assign consistent device names to LTO tape drives on my TSM/Spectrum Protect server across boots. However, for this situation, I think it would be overkill. I think what is important is what I at least have learned from this process and what things to watch out for and avoid if/when one mounts or activates root partitions from one system onto another. The practice of resetting file system UUIDs after cloning may be something to consider doing going forward. Aria Quoting Peter Oberparleiter <[email protected]>: > On 08.10.2021 18:08, Aria Bamdad wrote: >> Quoting Peter Oberparleiter <[email protected]>: >> >> Peter, thank you for your excellent analysis and clarification of how >> this works. It now makes perfect sense to me. > > I'm glad that you found the information I provided to be helpful. > > [...] > > >> I tested the what you said above. Sure enough, if you have dasda1 as >> your root device for the active system, you will find properly defined >> device names in by-id, by-path and by-uuid for this disk. Then if you >> were to activate/enable a clone of that disk, say as dasdf1, what >> happens is that by-id and by-path are still correct (they point to >> dasdf1) but chzdev will replace the existing symbolic link in by-uuid >> pointing to dasda1 with a new one pointing to dasdf1 (as it should) so >> now you will overwrite the previous UUID symbolic link that pointed to >> dasda1 with one that points to dasdf1. > > Small correction: the links are not created by chzdev, but by udev which > is a core part of Linux OS distributions. > >> We no longer have one for >> dasda1 which is the true root file system since we can't have two >> files with the same name here. chzdev does this without any warning. >> Shouldn't it at least give a message or better yet, no replace the >> file in by-uuid? > > Unfortunately there's no easily consumable way for udev to issue > warnings like that. And chzdev cannot check a device's UUID without > activating it - and then it would already be too late. > >> Perhaps it would be good to present the same warning >> as it currently does when you try to disable/deactivate a disk and it >> detects conflicting UUIDs. In my opinion, that would be a good thing >> as it prompts the user that they are about to do something that may >> result in a system being in a inconsistent state. Not creating the >> link in by-uuid in situations like this would avoid the problem all >> together. > > Indeed udev *does* provide a level of control over how the situation > with conflicting UUIDs - or more generally: conflicting link targets - > is handled. A udev rule can assign a link-priority value to a device. If > two devices with the same symlink target are found, the device with the > higher link-priority value will take precedence. > > From my own experience writing a udev rule isn't always > straight-forward, so I'll provide an example to support the discussion. > This example assumes two DASDs, each with a single partition containing > an ext-filesystem: > > - 0.0.1000: dasdb1 > - 0.0.2000: dasdc1 > > By default all devices are assigned a link-priority of 0. To quote the > udev man page: > > If no link_priority is specified, the order of the devices > (and which one of them owns the link) is undefined. > > The usual effect though will be that the device that is registered last > will take precedence. > > The following sequence of commands demonstrates the problem by assigning > the same file system label first to dasdb1, then to dasdc1. The output > of readlink shows the resulting link target: > > $ e2label /dev/dasdb1 test > $ readlink /dev/disk/by-label/test > ../../dasdb1 > $ e2label /dev/dasdc1 test > $ readlink /dev/disk/by-label/test > ../../dasdc1 > > As you can see, dasdc1 overwrites the previous dasdb1 link target. > > Now if you create a udev rule that assigns a different link-priority, > the device with the higher value will take precedence. > > Here's an example udev rule file that assigns a link-priority of -1 to > all partitions on the DASD with device number 2000. > > ACTION=="add|change", KERNEL=="dasd*", ENV{DEVTYPE}=="partition", > ENV{ID_PATH}=="ccw-0.0.2000", OPTIONS="link_priority=-1" > > The rule can be activated by storing it as a single line to a file with > a name ending in .rules in /etc/udev/rules.d, e.g. > > /etc/udev/rules.d/99-dasd-link-prio.rules > > How this rule works: > > 1. ACTION=="add|change" > If a new device was registered, or an existing one was modified > 2. KERNEL=="dasd*" > and the device node name starts with "dasd" > 3. ENV{DEVTYPE}=="partition" > and the device type is a partition > 4. ENV{ID_PATH}=="ccw-0.0.2000" > and the device ID path that includes the device number matches > the specified value "ccw-0.0.2000" > 5. OPTIONS="link_priority=-1" > then assign a link-priority value of -1 which is lower than the > default value of 0 > > In our example setup, the rule will match the udev change events > generated when modifying the label for the file system on device dasdc1 > (device number 2000), and assign a lower-than-default priority of -1 to > it. As a result if you repeat the sequence from before, you'll find that > the symlink for dasdb1 is no longer overwritten: > > $ e2label /dev/dasdb1 test2 > $ readlink /dev/disk/by-label/test2 > ../../dasdb1 > $ e2label /dev/dasdc1 test2 > $ readlink /dev/disk/by-label/test2 > ../../dasdb1 > > The consumability of writing udev rules is of course limited, but the > flexibility of the concept might allow to work around the issues you > mentioned in specific setups. > >> Interestingly, at boot time, if you have two devices that have the >> same UUID (as in above) only the first device will end up having a >> by-uuid link created. Apparently, at boot, when the second device is >> created, it does not cause the by-uuid link of the first device to get >> replaced. This is assuming they second device is not created first >> and then the first but I don't know how I can tell. > > This may be due to the fact that during boot, the sequence of udev > events processed is somewhat different than during run-time: while > during run-time, kernel events are typically processed as they are > generated ("hotplug"), during boot some events are generated before udev > is running. To catch those, all events are replayed in a custom order > once udev is running. This is called "coldplug". > > Also note that udev rules intended for the device that provides the root > file system must be included in the initial RAM-disk, since all uevents > for the root device are processed while / is not yet mounted. > >>> dracut tries to determine the device number and persistent ID of the >>> block device that provides the root file system and stores the result >>> inside the initial RAM-disk. Since the persistent links are incorrect, >>> the wrong data is used during boot, even if the cloned disk is removed. >> >> This makes perfect sense now. I somehow think it is unlikely we can >> get dracut developers to change this behavior or at least check for >> it. Since on z/VM we do a lot of cloning of disks/severs, I think we >> may have a lot of disks lurking around with matching UUIDs. That's >> why I thought a message or change in behavior for chzdev would be >> helpful. Or by not creating the by-uuid links, we can still allow >> access to the device but not create confusion for dracut and other >> utilities that reference disks by uuid. > > With the flexibility of assigning link-priorities in udev rules it might > be possible to prevent the creation of conflicting symlinks altogether, > e.g. by assigning higher priority values to devices that are in use > (e.g. mounted). This requires some effort to implement though, so it > might be more feasible to change the cloning process to accommodate for > this obstacle in another way. Also adding udev rules wouldn't keep > chzdev from complaining that the cloned device is in use since it reads > the UUIDs directly from the devices. > >>> The evaluation of file system UUIDs by chzdev was specifically added to >>> support correct handling of multi-disk btrfs file systems. Also given >>> that a file system UUID is supposed to uniquely identify a file system, >>> I don't consider the current behavior a problem in chzdev. >> >> I think moving forward, I will implement a procedure to change the >> disk UUIDs after cloning. I don't see this mentioned in any of the >> past cookbooks but feel it is a prudent thing to do. > > This would surely be the most practical way. Depending on how file > systems are referenced on your systems, there might be multiple files > that you need to change though (e.g. /etc/fstab, the kernel parameter > line, the initial RAM-disk, etc.). > > > Regards, > Peter > > -- > Peter Oberparleiter > Linux on Z Development - IBM Germany > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or > visit > https://urldefense.com/v3/__http://www2.marist.edu/htbin/wlvindex?LINUX-390__;!!F9svGWnIaVPGSwU!_EBy2FlzaQrnubpLM8rARfBrcMsVYZKIfs6a5gEjWzd6qbHRhXYU-1Eq_GJwDdbcyT5_4PQ$ > ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit https://urldefense.com/v3/__http://www2.marist.edu/htbin/wlvindex?LINUX-390__;!!F9svGWnIaVPGSwU!_EBy2FlzaQrnubpLM8rARfBrcMsVYZKIfs6a5gEjWzd6qbHRhXYU-1Eq_GJwDdbcyT5_4PQ$ ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
