Re: Where does Linux store path information for dasd devices?
As far as I know, zHPF is disabled. However, I do not know how to confirm that. We have 8 FICON paths to the DASD subsystem shared through switches to 2 z196 machines each having 8 FICON paths to the switches. Of the 8 paths, 4 have been offline from all 8 LPARs for a couple of months while we waited for IODF changes and physical cabling changes to move those 4 paths from old switches to newer switches. Last evening, I varied on the 4 paths that were moved to all 8 LPARs and verified that they were online to the DASD. I only spot checked the paths and not all 1800 volumes. After I confirmed that all 8 paths were online to the DASD, I took the other 4 paths offline. Shortly after that, the small subset of Linux guests began having problems. We only saw issues on about 20 of over 500 Linux guests. I am going to open a PMR and find out how to capture documentation to better understand what is happening. Thank you to all who suggested possible solutions. Rick Barlow -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Where does Linux store path information for dasd devices?
On 09.07.2013 13:39, Rick Barlow wrote: We have 8 FICON paths to the DASD subsystem shared through switches to 2 z196 machines each having 8 FICON paths to the switches. Of the 8 paths, 4 have been offline from all 8 LPARs for a couple of months while we waited for IODF changes and physical cabling changes to move those 4 paths from old switches to newer switches. Last evening, I varied on the 4 paths that were moved to all 8 LPARs and verified that they were online to the DASD. I only spot checked the paths and not all 1800 volumes. After I confirmed that all 8 paths were online to the DASD, I took the other 4 paths offline. Shortly after that, the small subset of Linux guests began having problems. We only saw issues on about 20 of over 500 Linux guests. The recommended procedure for performing maintenance on a CHPID looks like this: 1. Vary CHPID off in Linux (chchp -v 0 CHPID) 2. Vary path off in z/VM 3. Vary CHPID off in z/VM 4. Perform maintenance 5. Vary CHPID on in z/VM 6. Vary path on in z/VM 7. Vary CHPID on in Linux (chchp -v 1 CHPID) If necessary, step 7 can be used after the fact to tell Linux to update status information for all paths of all devices using that CHPID. Regards, Peter Oberparleiter -- Peter Oberparleiter Linux on System z Development - IBM Germany -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Where does Linux store path information for dasd devices?
This may not be of much help, but In our non mainframe environment where we present NAS as SCSI, our filesystems drop into read only when they experience time outs due to dasd controller performance and contention issues, or underlying netiwork issues. You can see it in the kernel message ring buffer. Perhaps you are experiencing similar problems with your new disk subsystem? On Tue, Jul 9, 2013 at 7:39 AM, Rick Barlow wrote: > As far as I know, zHPF is disabled. However, I do not know how to confirm > that. > > We have 8 FICON paths to the DASD subsystem shared through switches to 2 > z196 machines each having 8 FICON paths to the switches. Of the 8 paths, 4 > have been offline from all 8 LPARs for a couple of months while we waited > for IODF changes and physical cabling changes to move those 4 paths from > old switches to newer switches. Last evening, I varied on the 4 paths that > were moved to all 8 LPARs and verified that they were online to the DASD. > I only spot checked the paths and not all 1800 volumes. After I confirmed > that all 8 paths were online to the DASD, I took the other 4 paths > offline. Shortly after that, the small subset of Linux guests began having > problems. We only saw issues on about 20 of over 500 Linux guests. > > I am going to open a PMR and find out how to capture documentation to > better understand what is happening. > > Thank you to all who suggested possible solutions. > > Rick Barlow > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Linux on System z documentation for SLES 11 SP3 on developerWorks
Please refer to http://www.ibm.com/developerworks/linux/linux390/documentation_suse.html#sles11sp3 for new/updated documentation for SLES 11 SP3. * end of message Heinz-Werner Seeck (for Gerhard Hiller) Mit freundlichen Grüßen / Kind regards Heinz-Werner Seeck Project Manager for Linux on System z IBM Systems &Technology Group, Pure Systems & Modular Software Development Linux on System z Program Management Phone: +49-7031-16-4660 IBM Deutschland Mobile: +49-(0) 151-41920567 Schoenaicher Str. 220 E-Mail: heinz-werner_se...@de.ibm.com 71032 Boeblingen Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina Köderitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 <>
Some interesting bits about eth0:n
This may be old news to some but it has cost us several hours so I thought I'd pass it on. We run multiple blog sites on single RHEL6 servers (multiple servers with similar configurations), each blog has an IP address defined with a ifcfg-eth0:n interface. Works fine except when we recently changed lpars and the mac address changed and the switches keep their ARP cache for 4 hours. The blogs defined with eth0:n were not available via http or even a ping. We had to do an arping to make them available. Of course it took a couple of tries at midnight to discover this. Turns out the eth0:n interfaces don't send a gratuitous ARP to tell folks 'here I am'. Haven't found that documented anywhere yet. The 'correct' method is to add IPADDRn and NETMASKn to ifcfg-eth0. That way a gratuitous ARP is sent both at IPL and ifup time. Of course now if you issue an 'ifconfig' command to check on your interfaces you only see the main IP address for eth0. You need to issue an 'ip addr' command to see secondary addresses. To take an IP address down or to delete it use 'ip addr del / dev eth0 To add an IP address just add it to ifcfg-eth0 and do an 'ifup' Bobby Bauer Center for Information Technology National Institutes of Health Bethesda, MD 20892-5628 301-594-7474 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Some interesting bits about eth0:n
There are no mac entries in that files. Bobby Bauer Center for Information Technology National Institutes of Health Bethesda, MD 20892-5628 301-594-7474 From: Hornyak, Stanley (NIH/CIT) [E] Sent: Tuesday, July 09, 2013 10:15 AM To: Bauer, Bobby (NIH/CIT) [E]; 'Linux on 390 Port' Cc: Dickinson, Eric (NIH/CIT) [E]; Dussault, John (NIH/CIT) [E] Subject: RE: Some interesting bits about eth0:n You will need to remove the mac entries in /etc/udev/rules.d/70-persistent-net.rules When udev sees the new mac entry it automatically assigns a new eth device The alternative is to edit the file to match the mac address. Hope this helps Stanley V. Hornyak Unix Systems Administrator CIT Hosting Services Branch Building 12B, Room 2N207 12 South Dr Bethesda, MD 20892 Phone (301) 402-2627 Cell (240) 205-9474 From: Bauer, Bobby (NIH/CIT) [E] Sent: Tuesday, July 09, 2013 10:10 AM To: 'Linux on 390 Port' Cc: Dickinson, Eric (NIH/CIT) [E]; Dussault, John (NIH/CIT) [E]; Hornyak, Stanley (NIH/CIT) [E] Subject: Some interesting bits about eth0:n This may be old news to some but it has cost us several hours so I thought I'd pass it on. We run multiple blog sites on single RHEL6 servers (multiple servers with similar configurations), each blog has an IP address defined with a ifcfg-eth0:n interface. Works fine except when we recently changed lpars and the mac address changed and the switches keep their ARP cache for 4 hours. The blogs defined with eth0:n were not available via http or even a ping. We had to do an arping to make them available. Of course it took a couple of tries at midnight to discover this. Turns out the eth0:n interfaces don't send a gratuitous ARP to tell folks 'here I am'. Haven't found that documented anywhere yet. The 'correct' method is to add IPADDRn and NETMASKn to ifcfg-eth0. That way a gratuitous ARP is sent both at IPL and ifup time. Of course now if you issue an 'ifconfig' command to check on your interfaces you only see the main IP address for eth0. You need to issue an 'ip addr' command to see secondary addresses. To take an IP address down or to delete it use 'ip addr del / dev eth0 To add an IP address just add it to ifcfg-eth0 and do an 'ifup' Bobby Bauer Center for Information Technology National Institutes of Health Bethesda, MD 20892-5628 301-594-7474 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Some interesting bits about eth0:n
Bobby, Is this what you are looking for: RHEL6: A Gratuitous Arp Is Not Sent for any IP Alias when the Linux System Boots http://h10025.www1.hp.com/ewfrf/wc/document?cc=us&lc=en&dlc=en&docname=c03391733 Mike O'Reilly IBM Linux Change Team "Bauer, Bobby (NIH/CIT) [E]" LINUX-390@vm.marist.edu, Sent by: Linux on cc 390 Port Some interesting bits about eth0:n 07/09/2013 07:10 AM Please respond to Linux on 390 Port This may be old news to some but it has cost us several hours so I thought I'd pass it on. We run multiple blog sites on single RHEL6 servers (multiple servers with similar configurations), each blog has an IP address defined with a ifcfg-eth0:n interface. Works fine except when we recently changed lpars and the mac address changed and the switches keep their ARP cache for 4 hours. The blogs defined with eth0:n were not available via http or even a ping. We had to do an arping to make them available. Of course it took a couple of tries at midnight to discover this. Turns out the eth0:n interfaces don't send a gratuitous ARP to tell folks 'here I am'. Haven't found that documented anywhere yet. The 'correct' method is to add IPADDRn and NETMASKn to ifcfg-eth0. That way a gratuitous ARP is sent both at IPL and ifup time. Of course now if you issue an 'ifconfig' command to check on your interfaces you only see the main IP address for eth0. You need to issue an 'ip addr' command to see secondary addresses. To take an IP address down or to delete it use 'ip addr del / dev eth0 To add an IP address just add it to ifcfg-eth0 and do an 'ifup' Bobby Bauer Center for Information Technology National Institutes of Health Bethesda, MD 20892-5628 301-594-7474 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ <><><>
Re: Some interesting bits about eth0:n
You will need to remove the mac entries in /etc/udev/rules.d/70-persistent-net.rules When udev sees the new mac entry it automatically assigns a new eth device The alternative is to edit the file to match the mac address. Hope this helps Stanley V. Hornyak Unix Systems Administrator CIT Hosting Services Branch Building 12B, Room 2N207 12 South Dr Bethesda, MD 20892 Phone (301) 402-2627 Cell (240) 205-9474 From: Bauer, Bobby (NIH/CIT) [E] Sent: Tuesday, July 09, 2013 10:10 AM To: 'Linux on 390 Port' Cc: Dickinson, Eric (NIH/CIT) [E]; Dussault, John (NIH/CIT) [E]; Hornyak, Stanley (NIH/CIT) [E] Subject: Some interesting bits about eth0:n This may be old news to some but it has cost us several hours so I thought I'd pass it on. We run multiple blog sites on single RHEL6 servers (multiple servers with similar configurations), each blog has an IP address defined with a ifcfg-eth0:n interface. Works fine except when we recently changed lpars and the mac address changed and the switches keep their ARP cache for 4 hours. The blogs defined with eth0:n were not available via http or even a ping. We had to do an arping to make them available. Of course it took a couple of tries at midnight to discover this. Turns out the eth0:n interfaces don't send a gratuitous ARP to tell folks 'here I am'. Haven't found that documented anywhere yet. The 'correct' method is to add IPADDRn and NETMASKn to ifcfg-eth0. That way a gratuitous ARP is sent both at IPL and ifup time. Of course now if you issue an 'ifconfig' command to check on your interfaces you only see the main IP address for eth0. You need to issue an 'ip addr' command to see secondary addresses. To take an IP address down or to delete it use 'ip addr del / dev eth0 To add an IP address just add it to ifcfg-eth0 and do an 'ifup' Bobby Bauer Center for Information Technology National Institutes of Health Bethesda, MD 20892-5628 301-594-7474 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Some interesting bits about eth0:n
Upon conversation with Mr. Bauer, it appears that udev works differently for IBM Virtual machines, as there are no mac's in the file. In which case, the only other way is how he describes. Stanley V. Hornyak Unix Systems Administrator CIT Hosting Services Branch Building 12B, Room 2N207 12 South Dr Bethesda, MD 20892 Phone (301) 402-2627 Cell (240) 205-9474 From: Hornyak, Stanley (NIH/CIT) [E] Sent: Tuesday, July 09, 2013 10:15 AM To: Bauer, Bobby (NIH/CIT) [E]; 'Linux on 390 Port' Cc: Dickinson, Eric (NIH/CIT) [E]; Dussault, John (NIH/CIT) [E] Subject: RE: Some interesting bits about eth0:n You will need to remove the mac entries in /etc/udev/rules.d/70-persistent-net.rules When udev sees the new mac entry it automatically assigns a new eth device The alternative is to edit the file to match the mac address. Hope this helps Stanley V. Hornyak Unix Systems Administrator CIT Hosting Services Branch Building 12B, Room 2N207 12 South Dr Bethesda, MD 20892 Phone (301) 402-2627 Cell (240) 205-9474 From: Bauer, Bobby (NIH/CIT) [E] Sent: Tuesday, July 09, 2013 10:10 AM To: 'Linux on 390 Port' Cc: Dickinson, Eric (NIH/CIT) [E]; Dussault, John (NIH/CIT) [E]; Hornyak, Stanley (NIH/CIT) [E] Subject: Some interesting bits about eth0:n This may be old news to some but it has cost us several hours so I thought I'd pass it on. We run multiple blog sites on single RHEL6 servers (multiple servers with similar configurations), each blog has an IP address defined with a ifcfg-eth0:n interface. Works fine except when we recently changed lpars and the mac address changed and the switches keep their ARP cache for 4 hours. The blogs defined with eth0:n were not available via http or even a ping. We had to do an arping to make them available. Of course it took a couple of tries at midnight to discover this. Turns out the eth0:n interfaces don't send a gratuitous ARP to tell folks 'here I am'. Haven't found that documented anywhere yet. The 'correct' method is to add IPADDRn and NETMASKn to ifcfg-eth0. That way a gratuitous ARP is sent both at IPL and ifup time. Of course now if you issue an 'ifconfig' command to check on your interfaces you only see the main IP address for eth0. You need to issue an 'ip addr' command to see secondary addresses. To take an IP address down or to delete it use 'ip addr del / dev eth0 To add an IP address just add it to ifcfg-eth0 and do an 'ifup' Bobby Bauer Center for Information Technology National Institutes of Health Bethesda, MD 20892-5628 301-594-7474 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Where does Linux store path information for dasd devices?
Thank you Peter! Apparently, use of chchp on any path causes Linux to validate all of the CHPIDs. Using lscss, I found that many guests still show status like this: Device Subchan. DevType CU Type Use PIM PAM POM CHPIDs -- 0.0.01B0 0.0.000A 3390/0C 3990/E9 yes FF FF FF 444BA0B1 A1B06F4D After chchp -v 0 b1, the output changed to: Device Subchan. DevType CU Type Use PIM PAM POM CHPIDs -- 0.0.01B0 0.0.000A 3390/0C 3990/E9 yes FF E4 FF 444BA0B1 A1B06F4D Linux apparently found that 4 paths are actually offline. This circumvention whould help to avoid the problem we encountered. So, there are 2 issues remaining. 1) Of my 500+ Linux guests, only about 20 had issues that caused file system problems even though all show the issue using lscss. 2) Why doesn't Linux receive and handle pathing issues when they occur? It is not an unusual situation to take DASD paths off and back on. I am not going to pursue this further on the list. I will be opening a Service Request to investigate this further. Rick Barlow Nationwide -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Where does Linux store path information for dasd devices?
This is also why OPTION CHPIDVIRTUALIZATION ONE (and also the GLOBALOPTS statement) was invented, so that relocating guests don't see the real paths or care if the paths on the LPAR they move to change. The notes in the CP planning guide say "A single, virtualized CHPID is presented to the guest for dedicated devices and minidisks, independent of the real system CHPID. All online paths will still be used to support the guest I/O. This option is a requirement for a guest to be eligible for live guest relocation." and also "This option should be specified only when the guest is going to be relocated. In addition to virtualizing the CHPID numbers, CHPIDV ONE results in DASD path group ID virtualization." I'm not sure why it is suggested that this option is only used for relocating guests. It seems that it would have been useful to you if your Linux guests had this option in effect before you started changing the paths! On Tue, Jul 9, 2013 at 11:31 AM, Rick Barlow wrote: > Thank you Peter! > > Apparently, use of chchp on any path causes Linux to validate all of the > CHPIDs. > > Using lscss, I found that many guests still show status like this: > Device Subchan. DevType CU Type Use PIM PAM POM CHPIDs > -- > 0.0.01B0 0.0.000A 3390/0C 3990/E9 yes FF FF FF 444BA0B1 A1B06F4D > > After chchp -v 0 b1, the output changed to: > Device Subchan. DevType CU Type Use PIM PAM POM CHPIDs > -- > 0.0.01B0 0.0.000A 3390/0C 3990/E9 yes FF E4 FF 444BA0B1 A1B06F4D > > Linux apparently found that 4 paths are actually offline. This > circumvention whould help to avoid the problem we encountered. > > So, there are 2 issues remaining. > > 1) Of my 500+ Linux guests, only about 20 had issues that caused file > system problems even though all show the issue using lscss. > 2) Why doesn't Linux receive and handle pathing issues when they occur? It > is not an unusual situation to take DASD paths off and back on. > > I am not going to pursue this further on the list. I will be opening a > Service Request to investigate this further. > > Rick Barlow > Nationwide > > -- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > -- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- Bruce Hayden z/VM and Linux on System z ATS IBM, Endicott, NY -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
IBM to acquire CSL International
Cross posted to IBMVM and Linux-390 IBM Press Release today: http://www-03.ibm.com/press/us/en/pressrelease/41442.wss Marcy -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Where does Linux store path information for dasd devices?
On Tue, 9 Jul 2013, Rick Barlow wrote: > 2) Why doesn't Linux receive and handle pathing issues when they occur? It does (after receiving a notification from the hardware/firmware or hypervisor, or after beeing triggered by the admin). But it looks like it didn't receive a notification that something changed. Regards, Sebastian -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Sad new from Raleigh
Red Hat employee Seth Vidal, 36, was biking near his home in Durham around 9 p.m. Monday when he was hit from behind by a hit-and-run driver. He was pronounced dead at Duke University Hospital. Vidal worked for Red Hat as the lead developer of "Yum", an automated system updater used around the world. -Chip- -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
