Thanks Ray, We actually circumvented the problem earlier this afternoon by disabling multipathing.
Brought the system up in rescue mode, mounted /dev/sda3 (root) at /mnt, mounted /dev/sda1 to /mnt/boot, rebound proc, sys and dev then chroot /mnt. Edited /etc/sysconfig/kernel changing INITRD_MODULES="jbd ext3 zfcp dm-multipath" to INITRD_MODULES="jbd ext3 zfcp" then ran mkinitrd and zipl -V and rebooted successfully. However, we still don't understand why multipathing was broken. I have two other SuSE 11 guests running on this same zVM LPAR with multipathing enabled that weren't having any issues. None the less, just glad it's back. On Tue, Jan 12, 2010 at 4:01 PM, Raymond Higgs <rayhi...@us.ibm.com> wrote: > Linux on 390 Port <LINUX-390@VM.MARIST.EDU> wrote on 01/12/2010 10:09:09 > AM: > > > Joe Martin <widet.jmar...@gmail.com> > > Sent by: Linux on 390 Port <LINUX-390@VM.MARIST.EDU> > > > > 01/12/2010 10:09 AM > > > > Please respond to > > Linux on 390 Port <LINUX-390@VM.MARIST.EDU> > > > > To > > > > LINUX-390@VM.MARIST.EDU > > > > cc > > > > Subject > > > > Re: SLES11 boot problem > > > > *Did that change anything?* - We don't think so. > > *when you get dumped into the shell, can you see any of your devices in > > /dev, or /proc/partitions? * Yes. > > *Is the zfcp driver loaded (cat /proc/modules)?* Yes. > > > > > > We see the disk, with the right path and lun number for each partition. > > Here?s the output from the shell: > > > > $ cd > > by-path > > > > $ ls > > -l > > > > total > > 0 > > > > lrwxrwxrwx 1 root root 9 Jan 12 14:45 > > ccw-0.0.d000-zfcp-0x5006048ad52cea89:0x07 > > > > 01000000000000 -> > > ../../sda > > > > lrwxrwxrwx 1 root root 10 Jan 12 14:45 > > ccw-0.0.d000-zfcp-0x5006048ad52cea89:0x07 > > > > 01000000000000-part1 -> > > ../../sda1 > > > > lrwxrwxrwx 1 root root 10 Jan 12 14:45 > > ccw-0.0.d000-zfcp-0x5006048ad52cea89:0x07 > > > > 01000000000000-part2 -> > > ../../sda2 > > > > lrwxrwxrwx 1 root root 10 Jan 12 14:45 > > ccw-0.0.d000-zfcp-0x5006048ad52cea89:0x07 > > > > 01000000000000-part3 -> ../../sda3 > > > > > > > > cat partitions > > > > major minor #blocks name > > > > > > > > 8 0 20971200 sda > > > > 8 1 522112 sda1 > > > > 8 2 2096482 sda2 > > > > 8 3 18338197 sda3 > > > > 253 0 20971200 dm-0 > > > > 253 1 522112 dm-1 > > > > 253 2 2096482 dm-2 > > > > 253 3 18338197 dm-3 > > > > $ > > > > > > Here?s the cat /proc/modules output from the shell: > > > > cat > > /proc/modules > > > > scsi_dh_alua 10688 0 - Live > > 0x000003e000135000 > > > > scsi_dh_rdac 10196 0 - Live > > 0x000003e000131000 > > > > scsi_dh_hp_sw 6584 0 - Live > > 0x000003e000115000 > > > > scsi_dh_emc 11120 0 - Live > > 0x000003e0000b1000 > > > > dm_round_robin 4504 1 - Live > > 0x000003e000055000 > > > > sd_mod 48336 1 - Live > > 0x000003e00016c000 > > > > crc_t10dif 2288 1 sd_mod, Live > > 0x000003e000036000 > > > > dm_snapshot 25048 0 - Live > > 0x000003e000129000 > > > > dm_multipath 26736 2 dm_round_robin, Live > > 0x000003e000121000 > > > > dm_mod 97888 8 dm_snapshot,dm_multipath, Live > > 0x000003e0000e8000 > > > > scsi_dh 10036 5 > > scsi_dh_alua,scsi_dh_rdac,scsi_dh_hp_sw,scsi_dh_emc,dm_multipath, Live > > 0x000003e0000ad000 > > > > zfcp 137304 1 [permanent], Live 0x000003e000149000 > > > > > > qdio 65912 1 zfcp, Live > > 0x000003e000103000 > > > > scsi_transport_fc 60180 1 zfcp, Live > > 0x000003e000045000 > > > > scsi_tgt 17544 1 scsi_transport_fc, Live > > 0x000003e00003f000 > > > > scsi_mod 247096 9 > > scsi_dh_alua,scsi_dh_rdac,scsi_dh_hp_sw,scsi_dh_emc,sd_mod,scs > > > > i_dh,zfcp,scsi_transport_fc,scsi_tgt, Live > > 0x000003e00006f000 > > > > ext3 199376 0 - Live > > 0x000003e0000b6000 > > > > mbcache 11748 1 ext3, Live > > 0x000003e000039000 > > > > jbd 87624 1 ext3, Live 0x000003e000058000 > > > > $ > > > > May not be important but we had two other SLES11 guests in the same LPAR > > have the same problem (total of 3). zVM was restarted - the two other > > failing guests booted successfully. > > > > > > On Mon, Jan 11, 2010 at 3:30 PM, Mark Post <mp...@novell.com> wrote: > > > > > >>> On 1/11/2010 at 04:23 PM, Joe Martin <widet.jmar...@gmail.com> > wrote: > > > > I did catch the error in the fstab and put the mount for /boot after > the > > > > mount for /. > > > > > > Did that change anything? If not, when you get dumped into the shell, > can > > > you see any of your devices in /dev, or /proc/partitions? Is the zfcp > > > driver loaded (cat /proc/modules)? > > > > > > > > > Mark Post > > > > > > ---------------------------------------------------------------------- > > > For LINUX-390 subscribe / signoff / archive access instructions, > > > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 > or > > > visit > > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > > > > > > ---------------------------------------------------------------------- > > For LINUX-390 subscribe / signoff / archive access instructions, > > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > Joe, > > Here are some things that I would check: > > Is the pchid online on the SE for d000? > Is the chpid online on the SE for d000? > Did someone change the NPIV settings on the SE for the chpid for d000? > Were there any zoning changes? > Did someone change lun masking on your storage array, sometimes called > host connections? > Does vm think the chpid is online for d000? > Is d000 attached to your guest? > If you cd into /sys/bus/ccw/drivers/zfcp/0.0.d000, and "cat online", do > you see 1? > If you ls in /sys/bus/ccw/drivers/zfcp/0.0.d000, do you see > 0x5006048ad52cea89? Or if you zfcp_san_disc -b 0.0.d000 -W, do you see > 0x5006048ad52cea89? > If you ls in /sys/bus/ccw/drivers/zfcp/0.0.d000/0x5006048ad52cea89, do you > see 0x0701000000000000? Or if you zfcp_san_disc -b 0.0.d000 -p > 0x5006048ad52cea89 -L, do you see 0x0701000000000000? > Any hints in /var/log/messages? > > Hopefully one of these will get you closer to the problem. > > Ray Higgs > System z FCP Development > Bld. 706, B24 > 2455 South Road > Poughkeepsie, NY 12601 > (845) 435-8666, T/L 295-8666 > rayhi...@us.ibm.com > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
