Re: Devices for dracut in sles 12 SP4
on its way. hopefully openable! thank you! -Original Message- From: Linux on 390 Port On Behalf Of Mark Post Sent: Tuesday, June 11, 2019 10:09 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] Devices for dracut in sles 12 SP4 On 6/11/19 1:05 PM, Marcy Cortes wrote: > If you can't, I can email you a copy. Please, do. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
Re: Devices for dracut in sles 12 SP4
On 6/11/19 1:05 PM, Marcy Cortes wrote: > If you can't, I can email you a copy. Please, do. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
Re: Devices for dracut in sles 12 SP4
Maybe its not the cio-ignore stuff. I just posted the whole long rescue/dracut console to the ticket. Can you look at it ? If you can't, I can email you a copy. -Original Message- From: Linux on 390 Port On Behalf Of Mark Post Sent: Tuesday, June 11, 2019 8:24 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] Devices for dracut in sles 12 SP4 On 6/10/19 3:52 PM, Marcy Cortes wrote: > Does anyone know how dracut is really supposed to be told about devices? I > was under the impression that all devices were allowed unless explicity in a > cio_ignore when running under z/VM. That's been the default for new installations on z/VM for a while now. However, if a system was upgraded from a version prior to that, the cio_ignore parameter will still be in your kernel parameters, and can cause problems like this. (Or, if someone chose to activate device blacklisting during a new install.) Check /proc/cmdline to see if it's there. If it is, remove it from /boot/grub2/grub.cfg, and /etc/default/grub so that the next time you reboot things should be more reasonable. > At least that's the impression I get reading this > https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_r_cio_ignore_cmd.html I'm not sure why you would get that impression from that page. The only references I see to z/VM on that page relate to adding devices to the cio_ignore list after booting, and what happens if that device is detached and later re-attached. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
Re: Devices for dracut in sles 12 SP4
On 6/11/19 11:51 AM, Cohen, Sam wrote: > I'm going to ask a religious questionif you're running under z/VM and > (ideally) giving the virtual machine only what it needs, why would you use > cio_ignore at all? That's exactly why Ihno and I successfully lobbied to get the default changed for not just z/VM, but KVM as well. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
Re: Devices for dracut in sles 12 SP4
I'm going to ask a religious questionif you're running under z/VM and (ideally) giving the virtual machine only what it needs, why would you use cio_ignore at all? Thanks, Sam (217) 862-9227 (office) (602) 327-2134 (cell) -Original Message- From: Linux on 390 Port On Behalf Of Mark Post Sent: Tuesday, June 11, 2019 8:24 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Devices for dracut in sles 12 SP4 On 6/10/19 3:52 PM, Marcy Cortes wrote: > Does anyone know how dracut is really supposed to be told about devices? I > was under the impression that all devices were allowed unless explicity in a > cio_ignore when running under z/VM. That's been the default for new installations on z/VM for a while now. However, if a system was upgraded from a version prior to that, the cio_ignore parameter will still be in your kernel parameters, and can cause problems like this. (Or, if someone chose to activate device blacklisting during a new install.) Check /proc/cmdline to see if it's there. If it is, remove it from /boot/grub2/grub.cfg, and /etc/default/grub so that the next time you reboot things should be more reasonable. > At least that's the impression I get reading this > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. > ibm.com%2Fsupport%2Fknowledgecenter%2Fen%2Flinuxonibm%2Fcom.ibm.linux. > z.lhdd%2Flhdd_r_cio_ignore_cmd.htmldata=02%7C01%7CSam.Cohen%40LRS > .COM%7C0a41595b36dc47ae1fa908d6ee813b90%7C62af9ccc42164ae2a1d306614c59 > c315%7C0%7C0%7C636958636152651551sdata=WvsnQZwpFYclPqprameZpTTDW8 > vaet45KuaGohzP31E%3Dreserved=0 I'm not sure why you would get that impression from that page. The only references I see to z/VM on that page relate to adding devices to the cio_ignore list after booting, and what happens if that device is detached and later re-attached. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww2.marist.edu%2Fhtbin%2Fwlvindex%3FLINUX-390data=02%7C01%7CSam.Cohen%40LRS.COM%7C0a41595b36dc47ae1fa908d6ee813b90%7C62af9ccc42164ae2a1d306614c59c315%7C0%7C0%7C636958636152661546sdata=EcmxBcaJl2MmxcM1DJXS%2FtGH5d9XvGC4y5WDA4lcbmI%3Dreserved=0 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
Re: Devices for dracut in sles 12 SP4
On 6/10/19 3:52 PM, Marcy Cortes wrote: > Does anyone know how dracut is really supposed to be told about devices? I > was under the impression that all devices were allowed unless explicity in a > cio_ignore when running under z/VM. That's been the default for new installations on z/VM for a while now. However, if a system was upgraded from a version prior to that, the cio_ignore parameter will still be in your kernel parameters, and can cause problems like this. (Or, if someone chose to activate device blacklisting during a new install.) Check /proc/cmdline to see if it's there. If it is, remove it from /boot/grub2/grub.cfg, and /etc/default/grub so that the next time you reboot things should be more reasonable. > At least that's the impression I get reading this > https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_r_cio_ignore_cmd.html I'm not sure why you would get that impression from that page. The only references I see to z/VM on that page relate to adding devices to the cio_ignore list after booting, and what happens if that device is detached and later re-attached. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
Devices for dracut in sles 12 SP4
Resending from other ID – maybe have been blocked for many (like me!) -- After some patching this weekend, we had a few servers go into dracut emergency mode. After a lot of pain and rescue system work, we found it didn't know about a couple of the devices in the VG group that housed some needed stuff. We found this in the dracut /run/initram/rdsosreport.txt you can get in the emergency shell. So looking, the missing ones seemed to not be in the rd.cio_accept line in there. Brought the rescue system back up and eventually figured out that those seem to be coming out of /boot/zipl/active_devices.txt That seems to be updated when dasd_configure is run (or now chzdev -e - see previous email thread) I then ran that for each of the devices needed and then ran grub2-install and that got me past the missing devices problem. I then ran into dracut choking on the VG with these messages Read-only locking type set. Write locks are prohibited. Recovery of volume group "system" failed. Cannot process volume group system I solved that by bringing the rescue system back up and running vgscan and it reported: >> 19:14:44 WARNING: Inconsistent metadata found for VG system - updating to >> use >> version 23 So today I go looking at more servers and pretty much all of them don't have all the devices in active_devices.txt nor in rd.cio_accept as evidenced by the /var/log/zypp/history. Hmm. So maybe if I run grub2-install on one of them I can recreate the fail? Nope. Came up just fine. We have a case open with SUSE who has taken it to their level 3. But I'm left with so many questions here... Our builds are based on cloning and so I'm worried that this will strike more servers. SP4 now writes udev rules differently in /etc/udev/rules.d/ They start with 41 now for new disks. The SP3 and earlier generated ones start with 51. myserver:/etc/udev/rules.d # l total 64 drwxr-xr-x 2 root root 4096 Jun 8 19:04 ./ drwxr-xr-x 3 root root 4096 Jun 9 00:10 ../ -rw-r--r-- 1 root root 139 Jun 5 17:43 41-cio-ignore.rules -rw-r--r-- 1 root root 396 Jun 5 17:43 41-dasd-eckd-0.0.800f.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0101.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0102.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0103.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0104.rules -rw-r--r-- 1 root root 347 Sep 8 2016 51-dasd-0.0.8000.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff00.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff01.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff02.rules -rw-r--r-- 1 root root 538 Dec 15 2016 51-dasd-0.0.ff03.rules -rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.3000.rules -rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.4000.rules -rw-r--r-- 1 root root 594 Jun 8 19:04 70-persistent-net.rules There is also a new 41-cio-ignore.rules that looks like this # Generated by chzdev ACTION=="add", SUBSYSTEM=="subsystem", KERNEL=="ccw", RUN{program}+="/bin/sh -c 'echo free 800f > /proc/cio_ignore'" Does anyone know how dracut is really supposed to be told about devices? I was under the impression that all devices were allowed unless explicity in a cio_ignore when running under z/VM. At least that's the impression I get reading this https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_r_cio_ignore_cmd.html And what's up with LVM? If it needed to upgrade itself, why didn't it prior to this point? lvm2 was last patch a month ago. Should it have done it then? Or did it just become inconsistent when the devices decided to go AWOL? Marcy -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
Devices for dracut in sles 12 SP4
After some patching this weekend, we had a few servers go into dracut emergency mode. After a lot of pain and rescue system work, we found it didn't know about a couple of the devices in the VG group that housed some needed stuff. We found this in the dracut /run/initram/rdsosreport.txt you can get in the emergency shell. So looking, the missing ones seemed to not be in the rd.cio_accept line in there. Brought the rescue system back up and eventually figured out that those seem to be coming out of /boot/zipl/active_devices.txt That seems to be updated when dasd_configure is run (or now chzdev -e - see previous email thread) I then ran that for each of the devices needed and then ran grub2-install and that got me past the missing devices problem. I then ran into dracut choking on the VG with these messages Read-only locking type set. Write locks are prohibited. Recovery of volume group "system" failed. Cannot process volume group system I solved that by bringing the rescue system back up and running vgscan and it reported: >> 19:14:44 WARNING: Inconsistent metadata found for VG system - updating to >> use >> version 23 So today I go looking at more servers and pretty much all of them don't have all the devices in active_devices.txt nor in rd.cio_accept as evidenced by the /var/log/zypp/history. Hmm. So maybe if I run grub2-install on one of them I can recreate the fail? Nope. Came up just fine. We have a case open with SUSE who has taken it to their level 3. But I'm left with so many questions here... Our builds are based on cloning and so I'm worried that this will strike more servers. SP4 now writes udev rules differently in /etc/udev/rules.d/ They start with 41 now for new disks. The SP3 and earlier generated ones start with 51. myserver:/etc/udev/rules.d # l total 64 drwxr-xr-x 2 root root 4096 Jun 8 19:04 ./ drwxr-xr-x 3 root root 4096 Jun 9 00:10 ../ -rw-r--r-- 1 root root 139 Jun 5 17:43 41-cio-ignore.rules -rw-r--r-- 1 root root 396 Jun 5 17:43 41-dasd-eckd-0.0.800f.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0101.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0102.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0103.rules -rw-r--r-- 1 root root 347 Aug 12 2016 51-dasd-0.0.0104.rules -rw-r--r-- 1 root root 347 Sep 8 2016 51-dasd-0.0.8000.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff00.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff01.rules -rw-r--r-- 1 root root 536 Dec 15 2016 51-dasd-0.0.ff02.rules -rw-r--r-- 1 root root 538 Dec 15 2016 51-dasd-0.0.ff03.rules -rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.3000.rules -rw-r--r-- 1 root root 1661 Aug 12 2016 51-qeth-0.0.4000.rules -rw-r--r-- 1 root root 594 Jun 8 19:04 70-persistent-net.rules There is also a new 41-cio-ignore.rules that looks like this # Generated by chzdev ACTION=="add", SUBSYSTEM=="subsystem", KERNEL=="ccw", RUN{program}+="/bin/sh -c 'echo free 800f > /proc/cio_ignore'" Does anyone know how dracut is really supposed to be told about devices? I was under the impression that all devices were allowed unless explicity in a cio_ignore when running under z/VM. At least that's the impression I get reading this https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lhdd/lhdd_r_cio_ignore_cmd.html And what's up with LVM? If it needed to upgrade itself, why didn't it prior to this point? lvm2 was last patch a month ago. Should it have done it then? Or did it just become inconsistent when the devices decided to go AWOL? Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390