Hi Martha,
On Wed, 15 Apr 2020 20:25:37 +0000
Martha McConaghy <martha.mccona...@marist.edu> wrote:
> I feel like Alice through the looking glass at this point. I was
> hopeful that mkinitrd would do the trick, but now I have a whole new
> problem. After bringing the disk online, adding it
> to /etc/dasd.conf, I ran mkinitrd and zipl. Then, I rebooted. I did
> NOT touch the LVM, or anything, at that point. This resulted in a
> bunch of dracut timeout errors and am now in dracut emergency mode.
> I'm about to give up and just give this darned thing a TB LUN and be
> done with it. But, this shouldn't be so hard for crying out loud.
> Its bugging me that I don't know what I'm doing wrong. The RedHat
> web site lets me log in, but won't show me their solution
> pages....arggg.....
>
> So, any thoughts?
hmm, this looks strange
have you changed the partition type to "LVM" in fdasd for the new DASD?
what does "lvm pvs" or "lvm vgs" in the emergency shell say?
If I see right, then kernel sees both the DASDs, so the problem will be
in LVM.
Dan
>
> Martha
>
> Pre-reboot:
>
> [root@opncl42 ~]# cio_ignore -l
> Ignored devices:
> =================
> [root@opncl42 ~]# chccwdev --online 0.0.0200
> Setting device 0.0.0200 online
> Done
> [root@opncl42 ~]# lsdasd
> Bus-ID Status Name Device Type BlkSz Size Blocks
> ==============================================================================
> 0.0.0150 active dasda 94:0 ECKD 4096 46068MB
> 11793420 0.0.0200 active dasdb 94:4 ECKD 4096
> 23033MB 5896620 [root@opncl42 ~]# vi /etc/dasd.conf
> [root@opncl42 ~]# mkinitrd
> -f /boot/initramfs-3.10.0-1127.el7.s390x.img 3.10.0-1127.el7.s390x
> [root@opncl42 ~]# zipl Using config file '/etc/zipl.conf'
> Building bootmap in '/boot'
> Building menu 'zipl-automatic-menu'
> Adding #1: IPL section '3.10.0-1127.el7.s390x' (default)
> Adding #2: IPL section '3.10.0-1127.el7.s390x_with_debugging'
> Adding #3: IPL section 'linux'
> Preparing boot device: dasda (0150).
> Done.
> [root@opncl42 ~]# reboot
>
> Post-reboot:
>
> Ý"Ý32m OK "Ý0m+ Reached target System Initialization.
> Ý 1.149462+ dasd-eckd 0.0.0150: A channel path to the device has
> become opera tional
> Ý 1.151741+ dasd-eckd 0.0.0150: New DASD 3390/0C (CU 3990/01) with
> 65519 cyli nders, 15 heads, 224 sectors
> Ý 1.153040+ dasd-eckd 0.0.0150: DASD with 4 KB/block, 47173680 KB
> total size, 48 KB/track, linux disk layout
> Ý 1.153145+ qeth: register layer 2 discipline
>
> Ý 1.154257+ qdio: 0.0.0702 OSA on SC 5 using AI:1 QEBSM:0 PRI:1
> TDD:1 SIGA:RW A
> Ý 1.155122+ dasda:VOL1/ 0X0150: dasda1 dasda2
> Ý 1.156584+ dasd-eckd 0.0.0200: A channel path to the device has
> become opera tional
> Ý 1.159045+ dasd-eckd 0.0.0200: New DASD 3390/0C (CU 3990/01) with
> 32759 cyli nders, 15 heads, 224 sectors
> Ý 1.159581+ dasd-eckd 0.0.0200: DASD with 4 KB/block, 23586480 KB
> total size, 48 KB/track, compatible disk layout
> Ý 1.160634+ dasdb:VOL1/ 0X0200: dasdb1
> Ý"Ý32m OK "Ý0m+ Started Show Plymouth Boot Screen.""
> Ý"Ý32m OK "Ý0m+ Reached target Paths.""
> Ý"Ý32m OK "Ý0m+ Started Forward Password Requests to Plymouth
> Directory Watch. ""
> Ý"Ý32m OK "Ý0m+ Reached target Basic System.""
> Ý 1.169428+ netif_napi_add() called with weight 128 on device eth%d
> Ý 1.169600+ qeth 0.0.0700: MAC address 02:01:42:00:00:08
> successfully registered on device eth0 Ý 1.169613+ qeth 0.0.0700:
> Device is a Virtual NIC QDIO card (level: V712) Ý 1.169613+ with
> link type Virt.NIC QDIO (portname: ) Ý 124.355856+
> dracut-initqueueÝ363+: Warning: dracut-initqueue timeout - starting
> timeout scripts"" Ý 124.862823+ dracut-initqueueÝ363+: Warning:
> dracut-initqueue timeout - starting timeout scripts"" Ý 125.366826+
> dracut-initqueueÝ363+: Warning: dracut-initqueue timeout - starting
> timeout scripts"" ...lots of timeout script errors....... Ý
> 184.424496+ dracut-initqueueÝ363+: Warning: dracut-initqueue timeout
> - starting timeout scripts"" Ý 184.424755+ dracut-initqueueÝ363+:
> Warning: Could not boot."" Ý 184.425825+ dracut-initqueueÝ363+:
> Warning: /dev/mapper/rhel_opncl41-root does not exist" Starting
> Dracut Emergency Shell..." Warning: /dev/mapper/rhel_opncl41-root
> does not exist
>
> Generating "/run/initramfs/rdsosreport.txt"
>
>
> Entering emergency mode. Exit the shell to continue.
> Type "journalctl" to view system logs.
> You might want to save "/run/initramfs/rdsosreport.txt" to a USB
> stick or /boot after mounting them and attach it to a bug report.
>
>
>
>
>
>
> Martha McConaghy
>
> Marist: System Architect/Technical Lead
>
> SHARE Association: Vice President
>
> Marist College IT
>
> Poughkeepsie, NY 12601
>
> ________________________________
> From: Linux on 390 Port <LINUX-390@VM.MARIST.EDU> on behalf of Marcy
> Cortes <marcy.d.cor...@wellsfargo.com> Sent: Wednesday, April 15,
> 2020 11:42 AM To: LINUX-390@VM.MARIST.EDU <LINUX-390@VM.MARIST.EDU>
> Subject: Re: Adding disk to root LVM with RHEL 7
>
> I'm kind of assuming RH7 works like sles11 and doesn't use grub2, but
> did you "mkinird; zipl" after getting that disk online?
>
> You can also "cat / run/initramfs/rdsosreport.txt" in that emergency
> mode to get more details.
>
>
>
>
> -----Original Message-----
> From: Linux on 390 Port <LINUX-390@VM.MARIST.EDU> On Behalf Of Rick
> Troth Sent: Wednesday, April 15, 2020 8:26 AM
> To: LINUX-390@VM.MARIST.EDU
> Subject: Re: [LINUX-390] Adding disk to root LVM with RHEL 7
>
> Looks like you did everything right: online, dasdfmt, fdasd, lvextend,
> vgextend, and using "1" instead of the whole disk (because it's ECKD
> and requires the VTOC).
>
> The first block where you get an error is 16380912. Maybe that's a
> clue? Looks like cylinder #109206 (150 4K blocks per ECKD cyl).
> Someone check my math? And the (new) root FS looks like 110100 cyls.
> The VG has two disks, correct? How big are they?
> So where is this troublesome block in the grand scheme?
>
> I don't recall resizing a filesystem *down*, but I think it can be
> done. I'm always nervous about that last block (or several) in any
> filesystem, any backing store (CKD, FBA, SCSI/SAN), anymode (LVM,
> partitioned, whole). Never know when the system or some program will
> walk to the end "just because".
>
> -- R; <><
>
>
> On 4/15/20 10:58 AM, Martha McConaghy wrote:
> > I have servers that I'm setting up with RHEL 7.8 on ECKD disk. I
> > installed it with root in an LVM, all of that worked fine. Now,
> > I'm testing adding a 2nd ECKD disk to the LVM as I know that will
> > be required in the future, but I'm running into a confusing
> > problem. Everything works as I expect and the disk space shows up
> > in / as it should. When I reboot the server, however, it goes to
> > heck and ends up in emergency mode. The most prominent feature of
> > the console are a bunch of I/O errors on device dm-2, which is the
> > disk I just added. Below are the steps I followed to add the
> > disk. Before doing this, I had activated the disk, added it
> > to /etc/dasd.conf and run zipl. After doing that, I rebooted to
> > ensure that the disk came up online, which it did. So, I'm
> > confident that the disk is there when the system rebooted the 2nd
> > time.
> >
> > I have a lot of colleagues who work with Linux all the time, but
> > don't usually have root as an LVM. They haven't seen this problem
> > before, but they work mainly on X. Could this be specific problem
> > to using ECKD? I have thought of going to an FBA LUN, but had
> > spare ECKD to use.
> >
> > Martha
> >
> >
> > [root@opncl42 ~]# cio_ignore -r 0.0.0200
> > [root@opncl42 ~]# chccwdev --online 0.0.0200
> > Setting device 0.0.0200 online
> > Done
> > [root@opncl42 ~]# lsdasd
> > Bus-ID Status Name Device Type BlkSz Size
> > Blocks
> > ==============================================================================
> > 0.0.0150 active dasda 94:0 ECKD 4096 46068MB
> > 11793420 0.0.0200 active dasdb 94:4 ECKD 4096
> > 23033MB 5896620 [root@opncl42 ~]# vi /etc/dasd.conf [root@opncl42
> > ~]# zipl Using config file '/etc/zipl.conf'
> > Building bootmap in '/boot'
> > Building menu 'zipl-automatic-menu'
> > Adding #1: IPL section '3.10.0-1127.el7.s390x' (default)
> > Adding #2: IPL section '3.10.0-1127.el7.s390x_with_debugging'
> > Adding #3: IPL section 'linux'
> > Preparing boot device: dasda (0150).
> > Done.
> > [root@opncl42 ~]# reboot
> > ---------------------------------------------------
> > [root@opncl42 ~]# lsdasd
> > Bus-ID Status Name Device Type BlkSz Size
> > Blocks
> > ==============================================================================
> > 0.0.0150 active dasda 94:0 ECKD 4096 46068MB
> > 11793420 0.0.0200 active dasdb 94:4 ECKD 4096
> > 23033MB 5896620
> >
> > [root@opncl42 ~]# dasdfmt -p -f /dev/dasdb
> > Please enter the blocksize of the formatting [4096]:
> > Drive Geometry: 32759 Cylinders * 15 Heads = 491385 Tracks
> >
> > I am going to format the device /dev/dasdb in the following way:
> > Device number of device : 0x200
> > Labelling device : yes
> > Disk label : VOL1
> > Disk identifier : 0X0200
> > Extent start (trk no) : 0
> > Extent end (trk no) : 491384
> > Compatible Disk Layout : yes
> > Blocksize : 4096
> > Mode : Full
> >
> > --->> ATTENTION! <<---
> > All data of that device will be lost.
> > Type "yes" to continue, no will leave the disk untouched: yes
> > Formatting the device. This may take a while (get yourself a
> > coffee). cyl 32759 of 32759 |#################################|
> > 100% [2m 43s] Finished formatting the device.
> > Rereading the partition table... ok
> > [root@opncl42 ~]# pvcreate /dev/dasdb1
> > Device /dev/dasdb1 not found.
> > [root@opncl42 ~]# fdasd /dev/dasdb
> > reading volume label ..: VOL1
> > reading vtoc ..........: ok
> >
> > Command action
> > m print this menu
> > p print the partition table
> > n add a new partition
> > d delete a partition
> > v change volume serial
> > t change partition type
> > r re-create VTOC and delete all partitions
> > u re-create VTOC re-using existing partition sizes
> > s show mapping (partition number - data set name)
> > q quit without saving changes
> > w write table to disk and exit
> >
> > Command (m for help): n
> > First track (1 track = 48 KByte) ([2]-491384):
> > Using default value 2
> > Last track or +size[c|k|M] (2-[491384]):
> > Using default value 491384
> >
> > Command (m for help): w
> > writing VTOC...
> > rereading partition table...
> > [root@opncl42 ~]# pvcreate /dev/dasdb1
> > Physical volume "/dev/dasdb1" successfully created.
> > [root@opncl42 ~]# vgdisplay
> > --- Volume group ---
> > VG Name rhel_opncl41
> > System ID
> > Format lvm2
> > Metadata Areas 1
> > Metadata Sequence No 3
> > VG Access read/write
> > VG Status resizable
> > MAX LV 0
> > Cur LV 2
> > Open LV 2
> > Max PV 0
> > Cur PV 1
> > Act PV 1
> > VG Size <44.50 GiB
> > PE Size 4.00 MiB
> > Total PE 11391
> > Alloc PE / Size 11381 / <44.46 GiB
> > Free PE / Size 10 / 40.00 MiB
> > VG UUID uEvApZ-Wrx7-7n47-f0f0-IQ6R-uU4K-JO8wAN
> >
> > [root@opncl42 ~]# vgextend rhel_opncl41 /dev/dasdb1
> > Volume group "rhel_opncl41" successfully extended
> > [root@opncl42 ~]# vgdisplay
> > --- Volume group ---
> > VG Name rhel_opncl41
> > System ID
> > Format lvm2
> > Metadata Areas 2
> > Metadata Sequence No 5
> > VG Access read/write
> > VG Status resizable
> > MAX LV 0
> > Cur LV 2
> > Open LV 2
> > Max PV 0
> > Cur PV 2
> > Act PV 2
> > VG Size <66.99 GiB
> > PE Size 4.00 MiB
> > Total PE 17149
> > Alloc PE / Size 11381 / <44.46 GiB
> > Free PE / Size 5768 / 22.53 GiB
> > VG UUID uEvApZ-Wrx7-7n47-f0f0-IQ6R-uU4K-JO8wAN
> >
> > [root@opncl42 ~]# lvdisplay
> > --- Logical volume ---
> > LV Path /dev/rhel_opncl41/swap
> > LV Name swap
> > VG Name rhel_opncl41
> > LV UUID QzWFVP-Z5hz-gRV2-ORnb-4rR6-pdhM-d77w3n
> > LV Write Access read/write
> > LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
> > LV Status available
> > # open 2
> > LV Size 4.50 GiB
> > Current LE 1152
> > Segments 1
> > Allocation inherit
> > Read ahead sectors auto
> > - currently set to 8192
> > Block device 253:1
> >
> > --- Logical volume ---
> > LV Path /dev/rhel_opncl41/root
> > LV Name root
> > VG Name rhel_opncl41
> > LV UUID G0Gy3F-lvt2-2uZy-3BKr-9yHL-uVqe-1lfycV
> > LV Write Access read/write
> > LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
> > LV Status available
> > # open 1
> > LV Size <39.96 GiB
> > Current LE 10229
> > Segments 1
> > Allocation inherit
> > Read ahead sectors auto
> > - currently set to 8192
> > Block device 253:0
> >
> > [root@opncl42 ~]# lvextend -l +5768 /dev/rhel_opncl41/root
> > Size of logical volume rhel_opncl41/root changed from <39.96 GiB
> > (10229 extents) to <62.49 GiB (15997 extents). Logical volume
> > rhel_opncl41/root successfully resized. [root@opncl42 ~]# lvdisplay
> > --- Logical volume ---
> > LV Path /dev/rhel_opncl41/swap
> > LV Name swap
> > VG Name rhel_opncl41
> > LV UUID QzWFVP-Z5hz-gRV2-ORnb-4rR6-pdhM-d77w3n
> > LV Write Access read/write
> > LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
> > LV Status available
> > # open 2
> > LV Size 4.50 GiB
> > Current LE 1152
> > Segments 1
> > Allocation inherit
> > Read ahead sectors auto
> > - currently set to 8192
> > Block device 253:1
> >
> > --- Logical volume ---
> > LV Path /dev/rhel_opncl41/root
> > LV Name root
> > VG Name rhel_opncl41
> > LV UUID G0Gy3F-lvt2-2uZy-3BKr-9yHL-uVqe-1lfycV
> > LV Write Access read/write
> > LV Creation host, time opncl41, 2020-04-14 20:25:30 -0400
> > LV Status available
> > # open 1
> > LV Size <62.49 GiB
> > Current LE 15997
> > Segments 2
> > Allocation inherit
> > Read ahead sectors auto
> > - currently set to 8192
> > Block device 253:0
> >
> > [root@opncl42 ~]# df -h
> > Filesystem Size Used
> > Avail Use% Mounted on
> > devtmpfs 9.8G 0
> > 9.8G 0% /dev
> > tmpfs 9.8G 0
> > 9.8G 0% /dev/shm
> > tmpfs 9.8G 0
> > 9.8G 0% /sys/fs/cgroup /dev/mapper/rhel_opncl41-root 40G
> > 2.0G 38G 5% / /dev/dasda1
> > 494M 136M 359M 28% /boot
> > tmpfs 2.0G 0
> > 2.0G 0% /run/user/1000
> >
> > [root@opncl42 ~]# fsadm resize /dev/mapper/rhel_opncl41-root
> > meta-data=/dev/mapper/rhel_opncl41-root isize=256 agcount=4,
> > agsize=2618624 blks = sectsz=4096 attr=2,
> > projid32bit=1 = crc=0 finobt=0
> > spinodes=0 data = bsize=4096
> > blocks=10474496, imaxpct=25 = sunit=0
> > swidth=0 blks naming =version 2 bsize=4096
> > ascii-ci=0 ftype=0 log =internal bsize=4096
> > blocks=5114, version=2 = sectsz=4096 sunit=1
> > blks, lazy-count=1 realtime =none extsz=4096
> > blocks=0, rtextents=0 data blocks changed from 10474496 to 16380928
> >
> > [root@opncl42 ~]# df -h
> > Filesystem Size Used
> > Avail Use% Mounted on
> > devtmpfs 9.8G 0
> > 9.8G 0% /dev
> > tmpfs 9.8G 0
> > 9.8G 0% /dev/shm
> > tmpfs 9.8G
> > 8.3M 9.8G 1% /run
> > tmpfs 9.8G 0
> > 9.8G 0% /sys/fs/cgroup /dev/mapper/rhel_opncl41-root 63G
> > 2.0G 61G 4% / /dev/dasda1
> > 494M 136M 359M 28% /boot
> > tmpfs 2.0G 0
> > 2.0G 0% /run/user/1000 [root@opncl42 ~]# reboot
> >
> > ----------------------------------------------------------------------------
> > Last part of console as it goes into emergency mode:
> >
> > 13 Ý Ý32m OK Ý0m¨ Started File System Check
> > on /dev/mapper/rhel_opncl41-root. 14 Ý Ý32m OK Ý0m¨ Started
> > dracut initqueue hook. 15 Mounting /sysroot...
> > 16 Ý Ý32m OK Ý0m¨ Reached target Remote File Systems (Pre).
> > 17 Ý Ý32m OK Ý0m¨ Reached target Remote File Systems.
> > 18 Ý 123.794007¨ Buffer I/O error on dev dm-2, logical block
> > 16380912, async page read 19 Ý Ý1;31mFAILED Ý0m¨ Failed to
> > mount /sysroot. 20 See 'systemctl status sysroot.mount' for details.
> > 21 Ý Ý1;33mDEPEND Ý0m¨ Dependency failed for Initrd Root File
> > System. 22 Ý Ý1;33mDEPEND Ý0m¨ Dependency failed for Reload
> > Configuration from the Real Root. 23 Ý Ý32m OK Ý0m¨ Stopped
> > dracut pre-udev hook. 24 Ý Ý32m OK Ý0m¨ Stopped dracut cmdline
> > hook. 25 Ý Ý32m OK Ý0m¨ Reached target Initrd File Systems.
> > 26 Ý Ý32m OK Ý0m¨ Stopped target Basic System.
> > 27 Ý Ý32m OK Ý0m¨ Stopped target System Initialization.
> > 28 Ý Ý32m OK Ý0m¨ Stopped dracut initqueue hook.
> > 29 Ý Ý32m OK Ý0m¨ Started Emergency Shell.
> > 30 Ý Ý32m OK Ý0m¨ Reached target Emergency Mode.
> > 31
> > 32 Generating "/run/initramfs/rdsosreport.txt"
> > 33 Ý 123.845595¨ Buffer I/O error on dev dm-2, logical block
> > 16380912, async page read
> > 34 Ý 123.846240¨ Buffer I/O error on dev dm-2, logical block
> > 16380912, async page read
> > 35
> > 36
> > 37 Entering emergency mode. Exit the shell to continue.
> > 38 Type "journalctl" to view system logs.
> > 39 You might want to save "/run/initramfs/rdsosreport.txt" to a USB
> > stick or / boot
> > 40 after mounting them and attach it to a bug report.
> > 41
> > 42
> > 43 :/#
> > ------------------------------------------------
> > Information from Journalctl while in emergency mode:
> >
> > ÝKApr 15 02:21:42 opncl42 systemdÝ1¨: Reached target Basic System.
> > Ým Apr 15 02:21:42 opncl42 kernel: Ý1;31mnetif_napi_add() called
> > with weight 128 o n devic Ým
> > Apr 15 02:21:42 opncl42 kernel: qeth 0.0.0700: MAC address
> > 02:01:42:00:00:08 suc Ým
> > Apr 15 02:21:42 opncl42 kernel: qeth 0.0.0700: Device is a Virtual
> > NIC QDIO card Ým
> > with link type Virt.NIC QDIO
> > (portname: ) Ým Apr 15 02:23:45 opncl42 dracut-initqueueÝ398¨:
> > Warning: dracut-initqueue timeout Ým
> > Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev
> > dm-2, logical blo ck 1638 Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Found
> > device /dev/mapper/rhel_opncl41-root. Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Starting File System Check
> > on /dev/mapper/rh Ým
> > Apr 15 02:23:45 opncl42 systemd-fsckÝ2026¨: fsck: error 2 (No such
> > file or direc Ým
> > Apr 15 02:23:45 opncl42 systemd-fsckÝ2026¨: Ý1;31mfsck failed with
> > error code 8 . Ý0m Ým
> > Apr 15 02:23:45 opncl42 systemd-fsckÝ2026¨: Ý1;39mIgnoring error.
> > Ý0m Ým Apr 15 02:23:45 opncl42 systemdÝ1¨: Started File System
> > Check on /dev/mapper/rhe Ým
> > Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev
> > dm-2, logical blo ck 1638 Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Started dracut initqueue hook.
> > Ým Apr 15 02:23:45 opncl42 systemdÝ1¨: Mounting /sysroot... Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Reached target Remote File
> > Systems (Pre). Ým
> >
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Reached target Remote File
> > Systems. Ým Apr 15 02:23:45 opncl42 mountÝ2042¨: mount: you must
> > specify the filesystem type Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;39msysroot.mount mount
> > process exited, c ode=exi Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;31mFailed to
> > mount /sysroot. Ý0m Ým Apr 15 02:23:45 opncl42 systemdÝ1¨:
> > Ý1;39mDependency failed for Initrd Root Fil e Syste Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Ý1;39mDependency failed for
> > Reload Configur ation f Ým
> > Ý7mlines 193-215 Ý27m ÝK ÝKApr 15 02:23:45 opncl42 systemdÝ1¨:
> > Ý1;39mJob ini Ým
> > Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev
> > dm-2, logical blo ck 1638 Ým
> > Apr 15 02:23:45 opncl42 systemdÝ1¨: Received SIGRTMIN+21 from PID
> > 406 (plymouthd Ým
> > Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev
> > dm-2, logical blo ck 1638 Ým
> > Apr 15 02:23:45 opncl42 kernel: Ý1;31mBuffer I/O error on dev
> > dm-2, logical blo ck 1638 Ým
> > Ý7mlines 211-233/233 (END) Ý27m ÝK ÝK ÝK Ý7mlines 211-233/233
> > (END) Ý27m ÝK
> >
> >
> >
> >
> >
> > Martha McConaghy
> >
> > Marist: System Architect/Technical Lead
> >
> > SHARE Association: Vice President
> >
> > Marist College IT
> >
> > Poughkeepsie, NY 12601
> >
> > ----------------------------------------------------------------------
> > For LINUX-390 subscribe / signoff / archive access instructions,
> > send email to lists...@vm.marist.edu with the message: INFO
> > LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
>
> --
> -- R; <><
>
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390
> or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390
> or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390
> or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
>
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www2.marist.edu/htbin/wlvindex?LINUX-390
