Jim:
Let me revise what I said earlier about pvfs2.ko. What I wrote holds if
you install using the orangefs-kmod-* rpm to install on your compute
nodes. If you use the vanilla orangefs-2.8.* rpm, then the pvfs2.ko is
located in
/opt/pvfs2/lib/modules/{2.6.18-92.1.13.el5/kernel/fs|2.6.32-220.13.1.el6_x86_64}/pvfs2/pvfs2.ko,
in which case, your install script will have to either move the pvfs2.ko
file into /lib/modules/... or use /sbin/insmod to install it.
Becky
On Thu, Jul 26, 2012 at 11:58 AM, Becky Ligon <[email protected]> wrote:
> Jim:
>
> I spent yesterday looking for ways that you might get the KP based on the
> screen shot that you provided. I cannot specifically point my finger at
> any one place in the code, but I can tell you that we have made many
> modifications to the kernel module and client-core in 2.8.6 that address
> KP's and corrupted data caused by the kernel module seen by other users
> going back as far as PVFS 2.6. I urge you to try 2.8.6 and see if you get
> the same KP's. You may be able to use 2.8.6 on your head node against the
> 2.8.5 servers that you are currently using, but first let me verify that
> there are no versioning issues between the two releases.
>
> Becky
>
> On Thu, Jul 26, 2012 at 11:49 AM, Becky Ligon <[email protected]> wrote:
>
>> Jim:
>>
>> The config files that you sent me are for older PVFS systems. We no
>> longer use the two-config-file approach anymore, and the config file should
>> have a data space and a storage space. Did you send me the wrong files?
>>
>> Becky
>>
>>
>> On Thu, Jul 26, 2012 at 11:48 AM, Becky Ligon <[email protected]> wrote:
>>
>>> Jim:
>>>
>>> Your 2.8.5 rpms install the kernel module in
>>> /lib/modules/2.6.18-92.1.13.el5/kernel/fs/pvfs2/pvfs2.ko and 2.8.6 into
>>> /lib/modules/2.6.32-220.13.1.el6.x86_64/kernel/fs/pvfs2/pvfs2.ko. Please
>>> verify that:
>>>
>>> /lib/modules/2.6.18-92.1.13.el5/modules.dep contains
>>> "kernel/fs/pvfs2/pvfs2.ko"
>>>
>>> /lib/modules/2.6.32-220.13.1.el6.x86_64/modules.dep contains
>>> "kernel/fs/pvfs2/pvfs2.ko"
>>>
>>> You might have to modify your rebuild scripts to execute a "depmod"
>>> AFTER the orangefs-kmod rpm is installed. Your scripts may already be
>>> doing this, and, if so, then the kernel module should be loaded after an
>>> install. You did mention that you thought the kernel module was being
>>> loaded properly. If that is the case, then adding the "LD_LIBRARY_PATH" to
>>> your OrangeFS init.d script should allow the client-core to fire up
>>> properly and then the following mount.
>>>
>>> Let me know if this works for you.
>>>
>>> Becky
>>>
>>> On Wed, Jul 25, 2012 at 5:50 PM, Jim Kusznir <[email protected]> wrote:
>>>
>>>> Here's the last file.
>>>>
>>>> On Wed, Jul 25, 2012 at 10:06 AM, Becky Ligon <[email protected]>
>>>> wrote:
>>>> > Jim:
>>>> >
>>>> > One more thing: can you send me the pvfs2-client.log files from the
>>>> nodes
>>>> > where a KP has occurred? If possible, I'd like the corresponding
>>>> > /var/log/messages log file from when the KP happened.
>>>> >
>>>> > Thanks,
>>>> > Becky
>>>> >
>>>> >
>>>> > On Wed, Jul 25, 2012 at 1:05 PM, Becky Ligon <[email protected]>
>>>> wrote:
>>>> >>
>>>> >> Jim:
>>>> >>
>>>> >> Can you also send me your PVFS server config file?
>>>> >>
>>>> >> Becky
>>>> >>
>>>> >>
>>>> >> On Wed, Jul 25, 2012 at 12:49 PM, Becky Ligon <[email protected]>
>>>> wrote:
>>>> >>>
>>>> >>> Jim:
>>>> >>>
>>>> >>> Can you send me the kmod-pvfs2-...rpm? I'd like to see how its
>>>> files are
>>>> >>> layed out.
>>>> >>>
>>>> >>> Thanks,
>>>> >>> Becky
>>>> >>>
>>>> >>>
>>>> >>> On Sat, Jul 21, 2012 at 4:46 PM, Jim Kusznir <[email protected]>
>>>> wrote:
>>>> >>>>
>>>> >>>> Hi Becky:
>>>> >>>>
>>>> >>>> Thanks for all your input. I was on travel and am currently
>>>> catching
>>>> >>>> up on e-mail, so here are answers to your questions:
>>>> >>>>
>>>> >>>> 1) this problem occurs on both my ROCKS 5.1 (CentOS 5.2) and ROCKS
>>>> 6
>>>> >>>> (CentOS 6.2) clusters identically.
>>>> >>>> 2) I can mount manually using the init script. It just will not
>>>> run
>>>> >>>> on boot. It tries, but fails with the error message supplied.
>>>> >>>> 3) The module is installed with a kmod-pvfs2-... rpm (as is
>>>> required
>>>> >>>> for ROCKS clusters...Any software to be installed on each node
>>>> needs
>>>> >>>> to be its own RPM). It appears to me that the module is being
>>>> loaded
>>>> >>>> successfully.
>>>> >>>> 4) Ok, that sounds plausible. I'll make those corrections and see
>>>> if
>>>> >>>> that fixes things.
>>>> >>>>
>>>> >>>> Of course, the mount on boot was one of two show-stopping issues.
>>>> The
>>>> >>>> second show-stopping issue is how many kernel panics are being
>>>> caused
>>>> >>>> by OrangeFS. I've been experiencing 3-8 KP's a week on a light to
>>>> >>>> moderate load on my cluster (24 nodes + head node, 3 pvfs nodes).
>>>> >>>>
>>>> >>>> My versions in use are: 2.8.5 (ROCKS 5.1), 2.8.6 (ROCKS 6). For my
>>>> >>>> users, I absolutely must have a "traditional filesystem interface"
>>>> >>>> (eg, MPI-IO or pvfs-* commands are not acceptable, they need to
>>>> work
>>>> >>>> on the files like they would for any other filesystem).
>>>> >>>>
>>>> >>>> --Jim
>>>> >>>>
>>>> >>>> On Fri, Jul 20, 2012 at 1:45 PM, Becky Ligon <[email protected]>
>>>> wrote:
>>>> >>>> > Jim:
>>>> >>>> >
>>>> >>>> > In your init script, you need to add the LD_LIBRARY_PATH
>>>> variable,
>>>> >>>> > since
>>>> >>>> > your pvfs library is not in a standard location:
>>>> >>>> >
>>>> >>>> > export LD_LIBRARY_PATH=/opt/pvfs2/lib:$LD_LIBRARY_PATH
>>>> >>>> >
>>>> >>>> > Remove the LD_PRELOAD. It is not needed here.
>>>> >>>> >
>>>> >>>> > Before "modprobe" will work, you have to run the command
>>>> "depmod" to
>>>> >>>> > update
>>>> >>>> > the modules list. The "make kmod_install" does not
>>>> automatically do
>>>> >>>> > this.
>>>> >>>> > NOTE: if you place the kernel module (pvfs2.ko) somewhere other
>>>> than
>>>> >>>> > /lib/modules/`uname -r`/kernel/fs/pvfs2, then you can't use
>>>> modprobe
>>>> >>>> > to load
>>>> >>>> > the module. Instead, use "/sbin/insmod <path>/pvfs2.ko". If
>>>> you are
>>>> >>>> > using
>>>> >>>> > the rpm spec that I gave you (and it looks like you are), then
>>>> >>>> > pvfs2.ko is
>>>> >>>> > located in /opt/pvfs2/lib/pvfs2.ko, in which case, you have to
>>>> use the
>>>> >>>> > "insmod" command to load it and the "rmmod" command to unload it.
>>>> >>>> >
>>>> >>>> > When you issue a "stop", your script does not stop the client
>>>> nor does
>>>> >>>> > it
>>>> >>>> > unload the kernel module. This will cause problems if you issue
>>>> a
>>>> >>>> > "start"
>>>> >>>> > by starting another pvfs2-client. I will send you the init
>>>> script
>>>> >>>> > that we
>>>> >>>> > use here. Maybe, you can modify it to accommodate your
>>>> environment.
>>>> >>>> > We
>>>> >>>> > have more checks in it than you have in yours.
>>>> >>>> >
>>>> >>>> > I am not familiar with how PVFS reacts to the "intr" option that
>>>> you
>>>> >>>> > specify
>>>> >>>> > in the mount command. What is its purpose?
>>>> >>>> >
>>>> >>>> > Becky
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > On Fri, Jul 20, 2012 at 3:27 PM, Becky Ligon <[email protected]
>>>> >
>>>> >>>> > wrote:
>>>> >>>> >>
>>>> >>>> >> Jim:
>>>> >>>> >>
>>>> >>>> >> I just realized that you have already sent me your init script.
>>>> Let
>>>> >>>> >> me
>>>> >>>> >> take a closer look at it.
>>>> >>>> >>
>>>> >>>> >> Becky
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >> On Fri, Jul 20, 2012 at 3:13 PM, Becky Ligon <
>>>> [email protected]>
>>>> >>>> >> wrote:
>>>> >>>> >>>
>>>> >>>> >>> Jim:
>>>> >>>> >>>
>>>> >>>> >>> I have successfully booted my CentOS 6.2 system (using
>>>> >>>> >>> 2.6.32-220.13.1.el6.x86_64) and started the PVFS2 server and
>>>> mounted
>>>> >>>> >>> the
>>>> >>>> >>> client. Thus, I can only guess that there is something in your
>>>> >>>> >>> environment
>>>> >>>> >>> causing the problem. Is it possible for you to mount the
>>>> client by
>>>> >>>> >>> issuing
>>>> >>>> >>> the commands manually once the system is running? Can you
>>>> send me a
>>>> >>>> >>> copy of
>>>> >>>> >>> your startup script for mounting the client from your
>>>> /etc/init.d
>>>> >>>> >>> directory?
>>>> >>>> >>>
>>>> >>>> >>> Becky
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>> >>> On Thu, Jul 19, 2012 at 12:58 PM, Becky Ligon <
>>>> [email protected]>
>>>> >>>> >>> wrote:
>>>> >>>> >>>>
>>>> >>>> >>>> Jim:
>>>> >>>> >>>>
>>>> >>>> >>>> I have been able to successfully mount-on-boot on a VM with
>>>> the
>>>> >>>> >>>> 2.6.32-220.13.1.el6.x86_64. However, I was using the
>>>> Scientific
>>>> >>>> >>>> Linux 6
>>>> >>>> >>>> distro and NOT CentOS 6.2. Next, I will try a CentOS 6.2
>>>> distro
>>>> >>>> >>>> and see
>>>> >>>> >>>> what happens with it.
>>>> >>>> >>>>
>>>> >>>> >>>> Becky
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>> On Wed, Jul 18, 2012 at 5:14 PM, Becky Ligon <
>>>> [email protected]>
>>>> >>>> >>>> wrote:
>>>> >>>> >>>>>
>>>> >>>> >>>>> Jim:
>>>> >>>> >>>>>
>>>> >>>> >>>>> Is the mount-on-boot issue just with your CentOS 6.2
>>>> environment?
>>>> >>>> >>>>> If
>>>> >>>> >>>>> so, which version of OrangeFS are you running?
>>>> >>>> >>>>>
>>>> >>>> >>>>> Becky
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>> On Wed, Jul 18, 2012 at 3:28 PM, Jim Kusznir <
>>>> [email protected]>
>>>> >>>> >>>>> wrote:
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> I cannot reproduce the pvfs2 crash on demand. I have not
>>>> yet
>>>> >>>> >>>>>> seen it
>>>> >>>> >>>>>> on centos 6, but I haven't placed centos6 into production
>>>> yet.
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> On my centos5 systems, its not reproducible on demand, but
>>>> it
>>>> >>>> >>>>>> seems to
>>>> >>>> >>>>>> happen with moderate file access from a few different
>>>> processes.
>>>> >>>> >>>>>> Sometimes scp'ing files to/from pvfs2 on the head node
>>>> (which is
>>>> >>>> >>>>>> a
>>>> >>>> >>>>>> pvfs2 client) will do it. This has happened since the
>>>> beginning
>>>> >>>> >>>>>> of
>>>> >>>> >>>>>> pvfs2 for me; on the compute nodes, I'm not sure if there's
>>>> more
>>>> >>>> >>>>>> than
>>>> >>>> >>>>>> one process, but since I updated to OrangeFS 2.8.5, I've
>>>> been
>>>> >>>> >>>>>> seeing
>>>> >>>> >>>>>> compute nodes KP with the previous screenshot (it did not
>>>> crash
>>>> >>>> >>>>>> (that
>>>> >>>> >>>>>> I'm aware of) prior to OrangeFS 2.8.5 on compute nodes).
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> Here's my /etc/init.d/pvfs2-client script:
>>>> >>>> >>>>>> ---------------
>>>> >>>> >>>>>> #!/bin/sh
>>>> >>>> >>>>>> #
>>>> >>>> >>>>>> # chkconfig: 2345 99 99
>>>> >>>> >>>>>> #
>>>> >>>> >>>>>> # description: mount pvfs2 filesystem
>>>> >>>> >>>>>> #
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> . /etc/rc.d/init.d/functions
>>>> >>>> >>>>>> #export LD_PRELOAD=/opt/db4/lib/
>>>> >>>> >>>>>> case "$1" in
>>>> >>>> >>>>>> start)
>>>> >>>> >>>>>> echo -n "Mounting PVFS2 Filesystem: "
>>>> >>>> >>>>>> modprobe pvfs2
>>>> >>>> >>>>>> /opt/pvfs2/sbin/pvfs2-client -p
>>>> >>>> >>>>>> /opt/pvfs2/sbin/pvfs2-client-core
>>>> >>>> >>>>>> mkdir -p /mnt/pvfs2
>>>> >>>> >>>>>> mount -t pvfs2 -o intr
>>>> tcp://pvfs2-io-0-0:3334/pvfs2-fs
>>>> >>>> >>>>>> /mnt/pvfs2
>>>> >>>> >>>>>> touch /var/lock/subsys/pvfs2-client
>>>> >>>> >>>>>> ;;
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> stop)
>>>> >>>> >>>>>> echo -n "Unmounting PVFS2 Filesystem: "
>>>> >>>> >>>>>> umount /mnt/pvfs2
>>>> >>>> >>>>>> rm -f /var/lock/subsys/pvfs2-client
>>>> >>>> >>>>>> ;;
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> restart)
>>>> >>>> >>>>>> $0 stop
>>>> >>>> >>>>>> $0 start
>>>> >>>> >>>>>> ;;
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> status)
>>>> >>>> >>>>>> status $NAME
>>>> >>>> >>>>>> ;;
>>>> >>>> >>>>>> *)
>>>> >>>> >>>>>> echo "Usage: $NAME {start|stop|restart|status}"
>>>> >>>> >>>>>> exit 1
>>>> >>>> >>>>>> esac
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> exit 0
>>>> >>>> >>>>>> ----------------
>>>> >>>> >>>>>> I've tried with the export commented and uncommented, no
>>>> >>>> >>>>>> difference.
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> --Jim
>>>> >>>> >>>>>>
>>>> >>>> >>>>>> On Wed, Jul 18, 2012 at 12:20 PM, Becky Ligon
>>>> >>>> >>>>>> <[email protected]>
>>>> >>>> >>>>>> wrote:
>>>> >>>> >>>>>> > Thanks, Jim.
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> > We are using 2.6.32-220.4.1.el6.x86_64 in our production
>>>> >>>> >>>>>> > environment. So, I
>>>> >>>> >>>>>> > should be able to setup a VM with your kernel version and
>>>> test.
>>>> >>>> >>>>>> > Can
>>>> >>>> >>>>>> > you
>>>> >>>> >>>>>> > give me a scenario to try in order to reproduce the
>>>> problem?
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> > I am also setting up a CENTOS 6 VM, so I can analyze the
>>>> >>>> >>>>>> > mount-with-boot
>>>> >>>> >>>>>> > issue.
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> > Becky
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> > On Wed, Jul 18, 2012 at 3:16 PM, Jim Kusznir
>>>> >>>> >>>>>> > <[email protected]>
>>>> >>>> >>>>>> > wrote:
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> [root@aeoltest torque]# rpm -qa |grep kernel
>>>> >>>> >>>>>> >> kernel-2.6.32-220.13.1.el6.x86_64
>>>> >>>> >>>>>> >> dracut-kernel-004-256.el6_2.1.noarch
>>>> >>>> >>>>>> >> kernel-devel-2.6.32-220.13.1.el6.x86_64
>>>> >>>> >>>>>> >> kernel-headers-2.6.32-220.13.1.el6.x86_64
>>>> >>>> >>>>>> >> kernel-firmware-2.6.32-220.13.1.el6.noarch
>>>> >>>> >>>>>> >> kernel-doc-2.6.32-220.13.1.el6.noarch
>>>> >>>> >>>>>> >> [root@aeoltest torque]# uname -a
>>>> >>>> >>>>>> >> Linux aeoltest.local 2.6.32-220.13.1.el6.x86_64 #1 SMP
>>>> Tue Apr
>>>> >>>> >>>>>> >> 17
>>>> >>>> >>>>>> >> 23:56:34 BST 2012 x86_64 x86_64 x86_64 GNU/Linux
>>>> >>>> >>>>>> >> [root@aeoltest torque]#
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >>
>>>> >>>> >>>>>> >> On Wed, Jul 18, 2012 at 12:10 PM, Becky Ligon
>>>> >>>> >>>>>> >> <[email protected]>
>>>> >>>> >>>>>> >> wrote:
>>>> >>>> >>>>>> >> > Jim:
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> > We are working on a few corrections to the user
>>>> library, as
>>>> >>>> >>>>>> >> > we
>>>> >>>> >>>>>> >> > speak,
>>>> >>>> >>>>>> >> > that
>>>> >>>> >>>>>> >> > were identified last week. Using LD_PRELOAD would
>>>> >>>> >>>>>> >> > definitely get
>>>> >>>> >>>>>> >> > around
>>>> >>>> >>>>>> >> > the
>>>> >>>> >>>>>> >> > kernel issues at hand, but I ask that you wait until
>>>> we have
>>>> >>>> >>>>>> >> > all
>>>> >>>> >>>>>> >> > of the
>>>> >>>> >>>>>> >> > current corrections in place before using it.
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> > I also have some questions for you. I am working the
>>>> issue
>>>> >>>> >>>>>> >> > with
>>>> >>>> >>>>>> >> > the
>>>> >>>> >>>>>> >> > "won't
>>>> >>>> >>>>>> >> > mount on boot" issue and would like to know the
>>>> specific
>>>> >>>> >>>>>> >> > kernel
>>>> >>>> >>>>>> >> > that you
>>>> >>>> >>>>>> >> > are
>>>> >>>> >>>>>> >> > using under CentOS 6.2.
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> > Thanks,
>>>> >>>> >>>>>> >> > Becky
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> > On Wed, Jul 18, 2012 at 3:01 PM, Jim Kusznir
>>>> >>>> >>>>>> >> > <[email protected]>
>>>> >>>> >>>>>> >> > wrote:
>>>> >>>> >>>>>> >> >>
>>>> >>>> >>>>>> >> >> I managed to get a screenshot of a ip-kvm with the
>>>> last
>>>> >>>> >>>>>> >> >> chunk of
>>>> >>>> >>>>>> >> >> a
>>>> >>>> >>>>>> >> >> pvfs-induced KP on a compute node; image attached.
>>>> >>>> >>>>>> >> >>
>>>> >>>> >>>>>> >> >> With respect to client access methods, perhaps I
>>>> should
>>>> >>>> >>>>>> >> >> switch
>>>> >>>> >>>>>> >> >> to a
>>>> >>>> >>>>>> >> >> user space solution. I remember hearing about an
>>>> >>>> >>>>>> >> >> LD_Preload
>>>> >>>> >>>>>> >> >> client
>>>> >>>> >>>>>> >> >> module (not using fuse, but being entirely
>>>> userspace). Is
>>>> >>>> >>>>>> >> >> that
>>>> >>>> >>>>>> >> >> "ready" with 2.8.6? If not, perhaps I need to switch
>>>> to
>>>> >>>> >>>>>> >> >> the
>>>> >>>> >>>>>> >> >> fuse
>>>> >>>> >>>>>> >> >> module...
>>>> >>>> >>>>>> >> >>
>>>> >>>> >>>>>> >> >> --Jim
>>>> >>>> >>>>>> >> >>
>>>> >>>> >>>>>> >> >> On Wed, Jul 18, 2012 at 11:46 AM, Andrew Savchenko
>>>> >>>> >>>>>> >> >> <[email protected]>
>>>> >>>> >>>>>> >> >> wrote:
>>>> >>>> >>>>>> >> >> > Hello Becky,
>>>> >>>> >>>>>> >> >> >
>>>> >>>> >>>>>> >> >> > On Wed, 18 Jul 2012 12:43:51 -0400 Becky Ligon
>>>> wrote:
>>>> >>>> >>>>>> >> >> >> Andrew:
>>>> >>>> >>>>>> >> >> >>
>>>> >>>> >>>>>> >> >> >> 2.8.6 does not fix the problem you were seeing with
>>>> >>>> >>>>>> >> >> >> question
>>>> >>>> >>>>>> >> >> >> marks
>>>> >>>> >>>>>> >> >> >> in
>>>> >>>> >>>>>> >> >> >> the
>>>> >>>> >>>>>> >> >> >> "ls" output, but we are working on it.
>>>> >>>> >>>>>> >> >> >>
>>>> >>>> >>>>>> >> >> >> Just FYI!
>>>> >>>> >>>>>> >> >> >
>>>> >>>> >>>>>> >> >> > Thanks for the warning. I'll keep sticking to the
>>>> fuse
>>>> >>>> >>>>>> >> >> > client
>>>> >>>> >>>>>> >> >> > during
>>>> >>>> >>>>>> >> >> > update then.
>>>> >>>> >>>>>> >> >> >
>>>> >>>> >>>>>> >> >> > Best regards,
>>>> >>>> >>>>>> >> >> > Andrew Savchenko
>>>> >>>> >>>>>> >> >> >
>>>> >>>> >>>>>> >> >> > _______________________________________________
>>>> >>>> >>>>>> >> >> > Pvfs2-users mailing list
>>>> >>>> >>>>>> >> >> > [email protected]
>>>> >>>> >>>>>> >> >> >
>>>> >>>> >>>>>> >> >> >
>>>> >>>> >>>>>> >> >> >
>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>>>> >>>> >>>>>> >> >> >
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> > --
>>>> >>>> >>>>>> >> > Becky Ligon
>>>> >>>> >>>>>> >> > OrangeFS Support and Development
>>>> >>>> >>>>>> >> > Omnibond Systems
>>>> >>>> >>>>>> >> > Anderson, South Carolina
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >> >
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> > --
>>>> >>>> >>>>>> > Becky Ligon
>>>> >>>> >>>>>> > OrangeFS Support and Development
>>>> >>>> >>>>>> > Omnibond Systems
>>>> >>>> >>>>>> > Anderson, South Carolina
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>> >
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>> --
>>>> >>>> >>>>> Becky Ligon
>>>> >>>> >>>>> OrangeFS Support and Development
>>>> >>>> >>>>> Omnibond Systems
>>>> >>>> >>>>> Anderson, South Carolina
>>>> >>>> >>>>>
>>>> >>>> >>>>>
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>> --
>>>> >>>> >>>> Becky Ligon
>>>> >>>> >>>> OrangeFS Support and Development
>>>> >>>> >>>> Omnibond Systems
>>>> >>>> >>>> Anderson, South Carolina
>>>> >>>> >>>>
>>>> >>>> >>>>
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>> >>> --
>>>> >>>> >>> Becky Ligon
>>>> >>>> >>> OrangeFS Support and Development
>>>> >>>> >>> Omnibond Systems
>>>> >>>> >>> Anderson, South Carolina
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >> --
>>>> >>>> >> Becky Ligon
>>>> >>>> >> OrangeFS Support and Development
>>>> >>>> >> Omnibond Systems
>>>> >>>> >> Anderson, South Carolina
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > --
>>>> >>>> > Becky Ligon
>>>> >>>> > OrangeFS Support and Development
>>>> >>>> > Omnibond Systems
>>>> >>>> > Anderson, South Carolina
>>>> >>>> >
>>>> >>>> >
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Becky Ligon
>>>> >>> OrangeFS Support and Development
>>>> >>> Omnibond Systems
>>>> >>> Anderson, South Carolina
>>>> >>>
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Becky Ligon
>>>> >> OrangeFS Support and Development
>>>> >> Omnibond Systems
>>>> >> Anderson, South Carolina
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Becky Ligon
>>>> > OrangeFS Support and Development
>>>> > Omnibond Systems
>>>> > Anderson, South Carolina
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Becky Ligon
>>> OrangeFS Support and Development
>>> Omnibond Systems
>>> Anderson, South Carolina
>>>
>>>
>>>
>>
>>
>> --
>> Becky Ligon
>> OrangeFS Support and Development
>> Omnibond Systems
>> Anderson, South Carolina
>>
>>
>>
>
>
> --
> Becky Ligon
> OrangeFS Support and Development
> Omnibond Systems
> Anderson, South Carolina
>
>
>
--
Becky Ligon
OrangeFS Support and Development
Omnibond Systems
Anderson, South Carolina
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
