After some patching this weekend, we had a few servers go into dracut emergency mode. After a lot of pain and rescue system work, we found it didn't know about a couple of the devices in the VG group that housed some needed stuff. We found this in the dracut /run/initram/rdsosreport.txt you can get in the emergency shell.
So looking, the missing ones seemed to not be in the rd.cio_accept line in there. Brought the rescue system back up and eventually figured out that those seem to be coming out of /boot/zipl/active_devices.txt That seems to be updated when dasd_configure is run (or now chzdev -e - see previous email thread) I then ran that for each of the devices needed and then ran grub2-install and that got me past the missing devices problem. I then ran into dracut choking on the VG with these messages Read-only locking type set. Write locks are prohibited. Recovery of volume group "system" failed. Cannot process volume group system I solved that by bringing the rescue system back up and running vgscan and it reported: >> 19:14:44 WARNING: Inconsistent metadata found for VG system - updating to >> use >> version 23 So today I go looking at more servers and pretty much all of them don't have all the devices in active_devices.txt nor in rd.cio_accept as evidenced by the /var/log/zypp/history. Hmm. So maybe if I run grub2-install on one of them I can recreate the fail? Nope. Came up just fine. We have a case open with SUSE who has taken it to their level 3. But I'm left with so many questions here... Our builds are based on cloning and so I'm worried that this will strike more servers. SP4 now writes udev rules differently in /etc/udev/rules.d/ They start with 41 now for new disks. The SP3 and earlier generated ones start with 51. myserver:/etc/udev/rules.d # l total 64 drwxr-xr-x 2 root root 4096 Jun 8 19:04 ./ drwxr-xr-x 3 root root 4096 Jun 9 00:10 ../ -rw-r--r-- 1 root root 139 Jun 5 17:43 41-cio-ignore.rules -rw-r--r-- 1 root root 396 Jun 5 17:43 41-dasd-eckd-0.0.800f.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0101.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0102.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0103.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0104.rules -rw-r--r-- 1 root root 347 Sep 8 2016 51-dasd-0.0.8000.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff00.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff01.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff02.rules -rw-r--r-- 1 root root 538 Dec 15 2016 51-dasd-0.0.ff03.rules -rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.3000.rules -rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.4000.rules -rw-r--r-- 1 root root 594 Jun 8 19:04 70-persistent-net.rules There is also a new 41-cio-ignore.rules that looks like this # Generated by chzdev ACTION=="add", SUBSYSTEM=="subsystem", KERNEL=="ccw", RUN{program}+="/bin/sh -c 'echo free 800f > /proc/cio_ignore'" Does anyone know how dracut is really supposed to be told about devices? I was under the impression that all devices were allowed unless explicity in a cio_ignore when running under z/VM. At least that's the impression I get reading this https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_r_cio_ignore_cmd.html And what's up with LVM? If it needed to upgrade itself, why didn't it prior to this point? lvm2 was last patch a month ago. Should it have done it then? Or did it just become inconsistent when the devices decided to go AWOL? Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390