FYI, from ROMIO's config.log, I am see the following line. conftest.c:58: error: too few arguments to function 'PVFS_sys_create'
The config.log file can be found in http://www.ece.northwestern.edu/~wkliao/romio-config.log The log files can be found in http://www.ece.northwestern.edu/~wkliao/var-log-messages http://www.ece.northwestern.edu/~wkliao/pvfs2-client.log http://www.ece.northwestern.edu/~wkliao/pvfs2-server.log The MPI command I ran is (so you know the file name): % mpiexec -n 2 coll_write /orangefs/wkliao/tmpfile.10 % ls -l /orangefs/wkliao/ ls: cannot access /orangefs/wkliao/tmpfile.10: No such file or directory total 0 ?????????? ? ? ? ? ? tmpfile.10 Wei-keng On Mar 30, 2015, at 5:22 PM, Becky Ligon wrote: > 2.6.23 is really old and I have not tried that particular version. > > Something is awry with your installation, I think. I discovered that when I > built mpich-3.1.4, the romio config was looking for pvfs2-config, a program > in the OFS bin directory that outputs compile time options. Make sure your > mpich installation has the appropriate OFS bin path set in your $PATH and > then rebuild your mpich installation...just to be sure. Make sure your OFS > installation has the static libraries. Since romio looks for the OFS compile > time options, it statically links them into the romio executables. > > I'm not sure why your version doesn't process the pvfs2 prefix correctly but > mine does? Try this on your 2.6.32-504 installation: > > On your client machine as root, "echo all > /proc/sys/pvfs2/kernel-debug; > echo all > /proc/sys/pvfs2/client-debug > > On the same machine, run your program with the pvfs2 prefix > > echo none > /proc/sys/pvfs2/kernel-debug; echo none > > /proc/sys/pvfs2/client-debug > > Send me /var/log/messages and your client log file (pvfs2-client.log), so I > can see what is/isn't passing through the kernel. > > Becky > > On Mon, Mar 30, 2015 at 5:48 PM, Wei-keng Liao <[email protected]> > wrote: > Hi, Becky > > Thanks for the update. > > I have another Linux box that runs an older Linux kernel 2.6.23.9-85.fc8. > When I tried your experiment 1 (i.e. ls a non-existent PVFS file, then > create the file using pvfs2-touch, and then trying to "ls" the file again) > I got this error below. > data ::~(4:33pm) #1010 ls /orangefs/wkliao/tmpfile > ls: cannot access /orangefs/wkliao/tmpfile: No such file or directory > data ::~(4:33pm) #1011 pvfs2-touch /orangefs/wkliao/tmpfile > data ::~(4:34pm) #1012 ls /orangefs/wkliao/tmpfile > ls: cannot access /orangefs/wkliao/tmpfile: No such file or directory > > Then I tried your experiment 2 as root. The same error occurs. > [root@data wkliao]# echo 3 > /proc/sys/vm/drop_caches > [root@data wkliao]# ls /orangefs/wkliao/tmpfile2 > ls: cannot access /orangefs/wkliao/tmpfile2: No such file or directory > [root@data wkliao]# pvfs2-touch /orangefs/wkliao/tmpfile2 > [root@data wkliao]# ls /orangefs/wkliao/tmpfile2 > ls: cannot access /orangefs/wkliao/tmpfile2: No such file or directory > [root@data wkliao]# cat /proc/sys/vm/drop_caches > 3 > > It looks like the problem also occurs in kernel 2.6.23.9-85.fc8 > > As for your additional info bullet 3, in my original email, I am > having problems with file name prefixed with "pvfs2:" and no prefix. > The only one ran correctly is when I used "ufs:" prefix. > > Wei-keng > > On Mar 30, 2015, at 3:48 PM, Becky Ligon wrote: > > > Wei-keng: > > > > Just wanted to give you an update: I have discovered that newer versions > > of the kernel, i.e., 3.10.0-123.20.1.el7.x86_64 distributed via CentOS 7.0 > > do not exhibit the problem that is found using kernel > > 2.6.32-504.12.2.el6.x86_64 distributed by Scientific Linux 6.6. As I > > mentioned before, kernel 2.6.32-358.14.1.el6.x86_64 distributed by > > Scientific Linux 6.4 didn't exhibit the problem behavior either. > > > > When I run your program coll_write without prefacing the /path/to/filename, > > mpi issues a lstat through the kernel, which passes the request to PVFS. > > PVFS returns a "file not found" and marks that filename as an invalid inode > > and stores this information in the kernel's dcache (directory cache). mpi > > then uses the PVFS libraries to complete the creation of the file. The > > libraries bypass the kernel, so the kernel doesn't know that this file has > > been created. Since the inode has been marked invalid in the kernel, the > > "ls" command sees this file as "file-not-found". This kernel behavior is > > wrong. The "ls" *SHOULD* be passed to the PVFS kernel module and on to the > > client core where a request for this file can be sent to the servers. > > Since I can't change the behavior of the kernel, I am experimenting to see > > if we need to mark the inode invalid in the first place. This is where my > > efforts have gotten me so far. > > > > Additional information: > > > > 1. I can reproduce this problem by issuing an "ls" on a non-existent PVFS > > file, then create the file using pvfs2-touch, and then trying to "ls" the > > file again. > > > > 2. If you drop the dcache ( echo 3 > /proc/sys/vm/drop_caches), you can > > "ls" the newly created file with no problem. > > > > 3. With your coll_write program, if you preface the file with pvfs2 or ufs > > ({pvfs2|ufs}:/path/to/filename), the system behaves properly. When the > > pvfs2 prefix is used, nothing goes through the kernel; everything goes > > through the libraries. When the ufs prefix is used, everything goes > > through the kernel and on to PVFS. > > > > > > Becky > > > > On Tue, Mar 17, 2015 at 6:10 PM, Becky Ligon <[email protected]> wrote: > > I'm heading out for the evening but will jump on the problem tomorrow! > > > > Thanks for finding this problem! > > > > Becky > > > > On Tue, Mar 17, 2015 at 6:07 PM, Wei-keng Liao > > <[email protected]> wrote: > > Alright! It is good to know. > > I will stop messing around my builds and wait for your good news. > > Thanks. > > > > Wei-keng > > > > On Mar 17, 2015, at 5:05 PM, Becky Ligon wrote: > > > > > [bligon@SL6 wkliao]$ ls -l /mnt/test/wkliao > > > ls: cannot access /mnt/test/wkliao/testfile: No such file or directory > > > total 0 > > > ?????????? ? ? ? ? ? testfile > > > > > > > > > On Tue, Mar 17, 2015 at 6:04 PM, Becky Ligon <[email protected]> wrote: > > > Okay. I just recreated your problem. There is something amuck with the > > > newer kernel. Let me work on it and get back to you! > > > > > > Becky > > > > > > On Tue, Mar 17, 2015 at 6:01 PM, Wei-keng Liao > > > <[email protected]> wrote: > > > % cat /etc/redhat-release > > > Red Hat Enterprise Linux Server release 6.6 (Santiago) > > > > > > > > > Wei-keng > > > > > > On Mar 17, 2015, at 5:01 PM, Becky Ligon wrote: > > > > > > > issue: cat /etc/redhat-release > > > > > > > > On Tue, Mar 17, 2015 at 6:00 PM, Becky Ligon <[email protected]> wrote: > > > > Yes, but are you running CentOS, SL, ????? > > > > > > > > On Tue, Mar 17, 2015 at 5:54 PM, Wei-keng Liao > > > > <[email protected]> wrote: > > > > The command uname -a shows > > > > > > > > Linux bigdata.eecs.northwestern.edu 2.6.32-504.8.1.el6.x86_64 #1 SMP > > > > Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux > > > > > > > > Wei-keng > > > > > > > > On Mar 17, 2015, at 4:52 PM, Becky Ligon wrote: > > > > > > > > > Good! > > > > > > > > > > I'm working on the getting the system up and running with the newer > > > > > kernel. > > > > > > > > > > Which distro are you using? > > > > > > > > > > Becky > > > > > > > > > > On Tue, Mar 17, 2015 at 5:51 PM, Wei-keng Liao > > > > > <[email protected]> wrote: > > > > > Versioning is not an issue, as my older version of orangefs is on a > > > > > different machine. > > > > > This machine is a fresh install for Orangefs, BerkeleyDB, and MPICH. > > > > > > > > > > Wei-keng > > > > > > > > > > On Mar 17, 2015, at 4:48 PM, Becky Ligon wrote: > > > > > > > > > > > Could it be that you have a versioning issue here? Somehow, you > > > > > > have a mix of 2.9.1 and some older version? > > > > > > > > > > > > I was told by our tester that he always tests with > > > > > > --enabled-shared. I will try it both ways. Maybe, that's not the > > > > > > case! > > > > > > > > > > > > Becky > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:43 PM, Wei-keng Liao > > > > > > <[email protected]> wrote: > > > > > > Here is from my /var/log/messages when I restart the pvfs2 > > > > > > server/client. > > > > > > > > > > > > Mar 16 12:20:50 bigdata kernel: pvfs2: module version 2.9.1- > > > > > > unloaded > > > > > > Mar 16 12:20:53 bigdata kernel: pvfs2: pvfs2_init called with debug > > > > > > mask: "none" (0x00000000) > > > > > > Mar 16 12:20:53 bigdata kernel: pvfs2: module version 2.9.1- loaded > > > > > > Mar 16 12:20:55 bigdata kernel: PVFS: kernel debug mask has been > > > > > > modified to "none" (0x00000000) > > > > > > Mar 16 12:20:55 bigdata kernel: PVFS: client debug mask has been > > > > > > modified to "none" (0x00000000) > > > > > > > > > > > > Are you saying that MPICH required PVFS2 built with --enable-shared > > > > > > option? > > > > > > I don't know about this, as I had an older version of pvfs2 running > > > > > > fine and it was > > > > > > not built with that option. > > > > > > > > > > > > Rob, do you know? > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > On Mar 17, 2015, at 4:35 PM, Becky Ligon wrote: > > > > > > > > > > > > > I'm also updating my kernel to 2.6.32-504.12.2.el6 and will rerun > > > > > > > my previous tests to see if the kernel is the problem. > > > > > > > > > > > > > > In /var/log/messages, you should also see a message like: > > > > > > > > > > > > > > Mar 17 15:15:42 SL6 kernel: pvfs2: module version 2.9.1- loaded > > > > > > > > > > > > > > If you are not seeing this message, then it appears the kernel > > > > > > > module did not get loaded. > > > > > > > > > > > > > > My understanding is that MPI requires the shared libraries. > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:27 PM, Wei-keng Liao > > > > > > > <[email protected]> wrote: > > > > > > > I tried to build pvfs2 with --enable-shared before and the same > > > > > > > occurred. > > > > > > > > > > > > > > I am going to try install orangefs on one computer only and not > > > > > > > using any symbolic link. > > > > > > > will let you know. > > > > > > > > > > > > > > I see the following from the pvfs2 client log file. > > > > > > > [D 03/16/2015 12:20:55] [INFO]: Mapping pointer 0x7ff8c0087000 > > > > > > > for I/O. > > > > > > > [D 03/16/2015 12:20:55] [INFO]: Mapping pointer 0x197e000 for I/O. > > > > > > > > > > > > > > I see the following from /var/log/message > > > > > > > Mar 16 10:19:35 bigdata kernel: pvfs2: module version 2.9.1- > > > > > > > unloaded > > > > > > > Mar 16 10:19:37 bigdata kernel: pvfs2: pvfs2_init called with > > > > > > > debug mask: "none" (0x00000000) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > On Mar 17, 2015, at 4:04 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > Try removing the symbolic link. Shut everything down; recreate > > > > > > > > your storage, and restart. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If that doesn't work, then I want you to rebuild the software > > > > > > > > using --enable-shared as one of your configure options. > > > > > > > > Recreate everything, including your storage. Before you start > > > > > > > > the client, you will have to add LD_LIBRARY_PATH to your > > > > > > > > environment and point it to the OFS installation lib directory. > > > > > > > > Recompile your mpi programs making sure you have the > > > > > > > > following set: > > > > > > > > > > > > > > > > C_INCLUDE_PATH contains the include directory for OFS and the > > > > > > > > LIBRARY_PATH includes the OFS lib directory. > > > > > > > > > > > > > > > > > > > > > > > > Are you seeing any errors in the pvfs2-client.log file or > > > > > > > > /var/log/messages? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If none of the above works, then we can turn on kernel > > > > > > > > debugging and/or client debugging to see where the problem is > > > > > > > > coming from. > > > > > > > > > > > > > > > > Let me know! > > > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 4:44 PM, Wei-keng Liao > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > The command "mount" shows /home directory is not nfs and there > > > > > > > > is no > > > > > > > > nfs client running on this machine (the metadata server). > > > > > > > > > > > /dev/sda7 on /home type ext4 (rw,usrquota) > > > > > > > > > > > > > > > > The command "df" also shows /home is not NSF mounted. > > > > > > > > bigdata::~(3:21pm) #1003 df > > > > > > > > Filesystem 1K-blocks Used Available Use% > > > > > > > > Mounted on > > > > > > > > /dev/sda2 30106576 8053816 20516760 29% / > > > > > > > > tmpfs 16414648 0 16414648 0% > > > > > > > > /dev/shm > > > > > > > > /dev/sda1 3997376 90920 3696744 3% /boot > > > > > > > > /dev/sda7 850331204 373303148 433827084 47% /home > > > > > > > > /dev/sda3 50264772 53104 47651668 1% /tmp > > > > > > > > /dev/sda6 9948012 875016 8560996 10% /var > > > > > > > > tcp://bigdata:3334/orangefs > > > > > > > > 3228520448 1581383680 1647136768 49% > > > > > > > > /orangefs > > > > > > > > > > > > > > > > > > > > > > > > My guess it is still the kernel module that does not behave > > > > > > > > correctly. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > On Mar 17, 2015, at 3:26 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > I also meant to say that if your storage is on a nfs mount, > > > > > > > > > move your filesystem onto local storage, no links or nfs > > > > > > > > > mounts. > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 4:14 PM, Becky Ligon > > > > > > > > > <[email protected]> wrote: > > > > > > > > > [bligon@SL6 wkliao]$ ls -l /mnt/test/wkliao > > > > > > > > > total 91804 > > > > > > > > > -rw-r--r-- 1 bligon bligon 31000000 Mar 17 16:06 testfile > > > > > > > > > -rw-r--r-- 1 bligon bligon 31000000 Mar 17 16:06 > > > > > > > > > testfile.pvfs2 > > > > > > > > > -rw-rw-rw- 1 bligon bligon 32000000 Mar 17 16:05 testfile.ufs > > > > > > > > > > > > > > > > > > > > > > > > > > > [bligon@SL6 wkliao]$ cat /proc/sys/pvfs2/acache/timeout-msecs > > > > > > > > > 60000 > > > > > > > > > [bligon@SL6 wkliao]$ cat /proc/sys/pvfs2/ncache/timeout-msecs > > > > > > > > > 60000 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I was able to run your programs and issue an "ls" with no > > > > > > > > > problems, even with timeouts set at 60 seconds. My > > > > > > > > > environment is slightly different: my distro is > > > > > > > > > 2.6.32-358.14.1.el6.x86_64. > > > > > > > > > > > > > > > > > > Is your storage on a NFS-mounted directory? You said in an > > > > > > > > > earlier post that /files1 was a symbolic link to /home. Does > > > > > > > > > /home reside on a NFS mounted filesystem? If not, then try > > > > > > > > > recreating your storage without the symbolic link. > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 4:00 PM, Becky Ligon > > > > > > > > > <[email protected]> wrote: > > > > > > > > > Got it. Thanks. > > > > > > > > > > > > > > > > > > I've installed mpich 3.1.4 on my local vm and have created a > > > > > > > > > filesystem just like yours. Now, I'm going to try running > > > > > > > > > your program and see what shakes out! > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 3:42 PM, Wei-keng Liao > > > > > > > > > <[email protected]> wrote: > > > > > > > > > Sorry, please try again. > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > On Mar 17, 2015, at 2:22 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > Wei-keng: > > > > > > > > > > > > > > > > > > > > I can't access coll_write.c. Permission denied. > > > > > > > > > > > > > > > > > > > > Can you grant me the appropriate permissions so I can copy > > > > > > > > > > the code? > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 5:31 PM, Becky Ligon > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > Thanks! Still working on it! > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 5:07 PM, Wei-keng Liao > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > How did you compile your programs? > > > > > > > > > > > > > > > > > > > > My test program can be found in > > > > > > > > > > http://www.ece.northwestern.edu/~wkliao/coll_write.c > > > > > > > > > > > > > > > > > > > > bigdata::~/TEST_PROG(3:57pm) #1053 make coll_write > > > > > > > > > > mpicc -g -o coll_write coll_write.c > > > > > > > > > > > > > > > > > > > > The command I ran: > > > > > > > > > > mpiexec -n 2 coll_write /orangefs/wkliao/testfile > > > > > > > > > > > > > > > > > > > > bigdata::~/TEST_PROG(3:58pm) #1055 mpicc --version > > > > > > > > > > gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11) > > > > > > > > > > Copyright (C) 2010 Free Software Foundation, Inc. > > > > > > > > > > This is free software; see the source for copying > > > > > > > > > > conditions. There is NO > > > > > > > > > > warranty; not even for MERCHANTABILITY or FITNESS FOR A > > > > > > > > > > PARTICULAR PURPOSE. > > > > > > > > > > > > > > > > > > > > bigdata::~/TEST_PROG(3:58pm) #1056 mpichversion > > > > > > > > > > MPICH Version: 3.1.4 > > > > > > > > > > MPICH Release date: Fri Feb 20 15:02:56 CST 2015 > > > > > > > > > > MPICH Device: ch3:nemesis > > > > > > > > > > MPICH configure: --enable-g=debug --disable-fast > > > > > > > > > > --enable-shared --enable-fc --enable-cxx --enable-romio > > > > > > > > > > --with-file-system=ufs+pvfs2 CC=gcc CXX=g++ FC=gfortran > > > > > > > > > > CFLAGS=-g -O0 FCFLAGS=-g -O0 > > > > > > > > > > MPICH CC: gcc -g -O0 -g -O0 > > > > > > > > > > MPICH CXX: g++ -g -O0 > > > > > > > > > > MPICH F77: gfortran -g -O0 > > > > > > > > > > MPICH FC: gfortran -g -O0 -g -O0 > > > > > > > > > > > > > > > > > > > > > Can you send me a listing of the /files1 directories > > > > > > > > > > > where your storage is located? > > > > > > > > > > > > > > > > > > > > /files1 is a symbolic link to the local disk mounted at > > > > > > > > > > /home > > > > > > > > > > > > > > > > > > > > bigdata::~(3:55pm) #1043 ls -l /files1 > > > > > > > > > > total 0 > > > > > > > > > > lrwxrwxrwx 1 root root 21 Mar 12 09:15 orangefs -> > > > > > > > > > > /home/files1/orangefs > > > > > > > > > > > > > > > > > > > > bigdata::~(3:55pm) #1044 ldd /usr/local/sbin/pvfs2-client > > > > > > > > > > linux-vdso.so.1 => (0x00007fff54db6000) > > > > > > > > > > librt.so.1 => /lib64/librt.so.1 (0x0000003fc2400000) > > > > > > > > > > libm.so.6 => /lib64/libm.so.6 (0x0000003fc1800000) > > > > > > > > > > libdl.so.2 => /lib64/libdl.so.2 (0x0000003fc2000000) > > > > > > > > > > libpthread.so.0 => /lib64/libpthread.so.0 > > > > > > > > > > (0x0000003fc1c00000) > > > > > > > > > > libssl.so.10 => /usr/lib64/libssl.so.10 > > > > > > > > > > (0x0000003fcb000000) > > > > > > > > > > libcrypto.so.10 => /usr/lib64/libcrypto.so.10 > > > > > > > > > > (0x0000003fc8400000) > > > > > > > > > > libc.so.6 => /lib64/libc.so.6 (0x0000003fc1400000) > > > > > > > > > > /lib64/ld-linux-x86-64.so.2 (0x0000003fc1000000) > > > > > > > > > > libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 > > > > > > > > > > (0x0000003fca800000) > > > > > > > > > > libkrb5.so.3 => /lib64/libkrb5.so.3 > > > > > > > > > > (0x0000003fcac00000) > > > > > > > > > > libcom_err.so.2 => /lib64/libcom_err.so.2 > > > > > > > > > > (0x0000003fc8c00000) > > > > > > > > > > libk5crypto.so.3 => /lib64/libk5crypto.so.3 > > > > > > > > > > (0x0000003fca400000) > > > > > > > > > > libz.so.1 => /lib64/libz.so.1 (0x0000003fc2800000) > > > > > > > > > > libkrb5support.so.0 => /lib64/libkrb5support.so.0 > > > > > > > > > > (0x0000003fc9800000) > > > > > > > > > > libkeyutils.so.1 => /lib64/libkeyutils.so.1 > > > > > > > > > > (0x0000003fc9000000) > > > > > > > > > > libresolv.so.2 => /lib64/libresolv.so.2 > > > > > > > > > > (0x0000003fc3400000) > > > > > > > > > > libselinux.so.1 => /lib64/libselinux.so.1 > > > > > > > > > > (0x0000003fc3000000) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > > > On Mar 16, 2015, at 3:49 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > > > Also: > > > > > > > > > > > > > > > > > > > > > > ldd ./pvfs2-client > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 4:47 PM, Becky Ligon > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > How did you compile your programs? > > > > > > > > > > > > > > > > > > > > > > Just checking everything. So far, all of your settings > > > > > > > > > > > look good. > > > > > > > > > > > > > > > > > > > > > > Can you send me a listing of the /files1 directories > > > > > > > > > > > where your storage is located? > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 4:32 PM, Wei-keng Liao > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > Here are the outputs. > > > > > > > > > > > > > > > > > > > > > > bigdata::~(3:27pm) #1030 /sbin/lsmod | grep pvfs > > > > > > > > > > > pvfs2 139546 2 > > > > > > > > > > > > > > > > > > > > > > bigdata::~(3:27pm) #1031 ps aux | grep pvfs > > > > > > > > > > > root 5775 0.5 0.0 321788 11228 ? Ssl 12:20 > > > > > > > > > > > 1:02 /usr/local/sbin/pvfs2-server --pidfile > > > > > > > > > > > /var/run/pvfs2.pid /etc/orangefs-server.conf > > > > > > > > > > > root 5791 0.0 0.0 39640 472 ? Ss 12:20 > > > > > > > > > > > 0:00 /usr/local/sbin/pvfs2-client -p > > > > > > > > > > > /usr/local/sbin/pvfs2-client-core --logfile > > > > > > > > > > > /files1/orangefs/client.log --logstamp=datetime -a 0 -n 0 > > > > > > > > > > > root 5792 0.2 0.0 79904 27532 ? SL 12:20 > > > > > > > > > > > 0:25 pvfs2-client-core --child -a 0 -n 0 --logtype file > > > > > > > > > > > -L /files1/orangefs/client.log --logstamp datetime > > > > > > > > > > > wkliao 7037 0.0 0.0 105316 864 pts/3 S+ 15:27 > > > > > > > > > > > 0:00 grep pvfs > > > > > > > > > > > > > > > > > > > > > > bigdata::~(3:27pm) #1032 cat /etc/pvfs2tab > > > > > > > > > > > tcp://bigdata:3334/orangefs /orangefs pvfs2 > > > > > > > > > > > default,noauto 0 0 > > > > > > > > > > > > > > > > > > > > > > bigdata::~(3:27pm) #1033 /bin/mount > > > > > > > > > > > /dev/sda2 on / type ext4 (rw) > > > > > > > > > > > proc on /proc type proc (rw) > > > > > > > > > > > sysfs on /sys type sysfs (rw) > > > > > > > > > > > devpts on /dev/pts type devpts (rw,gid=5,mode=620) > > > > > > > > > > > tmpfs on /dev/shm type tmpfs (rw) > > > > > > > > > > > /dev/sda1 on /boot type ext4 (rw) > > > > > > > > > > > /dev/sda7 on /home type ext4 (rw,usrquota) > > > > > > > > > > > /dev/sda3 on /tmp type ext4 (rw) > > > > > > > > > > > /dev/sda6 on /var type ext4 (rw) > > > > > > > > > > > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) > > > > > > > > > > > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) > > > > > > > > > > > nfsd on /proc/fs/nfsd type nfsd (rw) > > > > > > > > > > > tcp://bigdata:3334/orangefs on /orangefs type pvfs2 (rw) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > > > > > On Mar 16, 2015, at 3:21 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > > > > > Can you also issue: > > > > > > > > > > > > > > > > > > > > > > > > /bin/mount > > > > > > > > > > > > > > > > > > > > > > > > so, I can see if the filesystem is mounted correctly? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 4:13 PM, Becky Ligon > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > On your system where you have the client and server > > > > > > > > > > > > running, can you send me the output from the following > > > > > > > > > > > > commands: > > > > > > > > > > > > > > > > > > > > > > > > /sbin/lsmod | grep pvfs > > > > > > > > > > > > ps aux | grep pvfs > > > > > > > > > > > > cat /etc/pvfs2tab > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 4:01 PM, Becky Ligon > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > I'm running a test case to see if I can reproduce your > > > > > > > > > > > > problem. I'll get back with you shortly. > > > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 3:40 PM, Wei-keng Liao > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > Here you go. > > > > > > > > > > > > > > > > > > > > > > > > http://www.ece.northwestern.edu/~wkliao/orangefs-server.conf > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > > > > > > > On Mar 16, 2015, at 2:13 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > > > > > > > PVFS2_SERVER_CONF=/etc/orangefs-server.conf > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 3:10 PM, Wei-keng Liao > > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > I am not sure what file that is? > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 16, 2015, at 2:04 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Now send me a copy of your server config file. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 2:37 PM, Wei-keng Liao > > > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > The requested 2 files can be found in the following > > > > > > > > > > > > > > URLs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.ece.northwestern.edu/~wkliao/config.log > > > > > > > > > > > > > > http://www.ece.northwestern.edu/~wkliao/pvfs2-server > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 16, 2015, at 1:29 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you also send me your config.log file from > > > > > > > > > > > > > > > when you compiled the source? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 2:28 PM, Becky Ligon > > > > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > If you are experimenting with OrangeFS, then > > > > > > > > > > > > > > > having one metadata and 4 data servers is fine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you send me your pvfs2-server init file, the > > > > > > > > > > > > > > > one used with the /sbin/service command? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 1:27 PM, Wei-keng Liao > > > > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > On Mar 16, 2015, at 12:02 PM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-Keing: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Did you umount and mount the filesystem? If > > > > > > > > > > > > > > > > not, umount the filesystem, restart the client > > > > > > > > > > > > > > > > core, and then mount the filesystem again. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes. My restart command ran "/sbin/servive > > > > > > > > > > > > > > > pvfs2-server restart" > > > > > > > > > > > > > > > the script contains both client and server > > > > > > > > > > > > > > > commands, include client's umount and mount. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I also suggest that you define your environment > > > > > > > > > > > > > > > > so that each of your machines (bigdata, > > > > > > > > > > > > > > > > bigdata1, bigdata2, bigdata3) have their pvfs > > > > > > > > > > > > > > > > servers configured to handle both I/O and > > > > > > > > > > > > > > > > metadata. To do this, you will have to recreate > > > > > > > > > > > > > > > > the filesystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My Orangefs is newly created and the test program > > > > > > > > > > > > > > > is the first one run in parallel on it. > > > > > > > > > > > > > > > Isn't my configuration legit (one metadata sever > > > > > > > > > > > > > > > and 4 data servers) for orangefs setup? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Again, my mpi test program ran 2 processes > > > > > > > > > > > > > > > locally on the metadata server which is both data > > > > > > > > > > > > > > > server and client. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 16, 2015 at 11:30 AM, Wei-keng Liao > > > > > > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > HI, Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I tried the command option "-a 0 -n 0" and > > > > > > > > > > > > > > > > restart the client/server, but the same issue > > > > > > > > > > > > > > > > persists. > > > > > > > > > > > > > > > > pvfs2-ping command shows one metadata server > > > > > > > > > > > > > > > > and 4 data servers. > > > > > > > > > > > > > > > > I ran my test program on the metadata server. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > meta servers: > > > > > > > > > > > > > > > > tcp://bigdata:3334 Ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > data servers: > > > > > > > > > > > > > > > > tcp://bigdata:3334 Ok > > > > > > > > > > > > > > > > tcp://bigdata1:3334 Ok > > > > > > > > > > > > > > > > tcp://bigdata2:3334 Ok > > > > > > > > > > > > > > > > tcp://bigdata3:3334 Ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wei-keng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 16, 2015, at 8:26 AM, Becky Ligon wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Caching is still an issue if you have servers > > > > > > > > > > > > > > > > > on more than one machine and those servers > > > > > > > > > > > > > > > > > provide metadata. Even in a one-server > > > > > > > > > > > > > > > > > environment, it could make a difference. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The "ls" command uses the kernel module and > > > > > > > > > > > > > > > > > client core, which in turn use the caches, > > > > > > > > > > > > > > > > > while the pvfs2-ls command does not. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you don't have the appropriate sudo > > > > > > > > > > > > > > > > > permissions to modify the /proc filesystem, > > > > > > > > > > > > > > > > > then you can start the client with the caches > > > > > > > > > > > > > > > > > turned off. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Example: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ./pvfs2-client -a 0 -n 0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If you execute pvfs2-client --help, you will > > > > > > > > > > > > > > > > > see these options. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becky > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Mar 15, 2015, at 5:05 PM, Wei-keng Liao > > > > > > > > > > > > > > > > >> <[email protected]> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> I assume after 60 seconds, the client will > > > > > > > > > > > > > > > > >> flush the cache. > > > > > > > > > > > > > > > > >> Please note I am running orangefs client and > > > > > > > > > > > > > > > > >> server on the same machine. > > > > > > > > > > > > > > > > >> In this case, should caching become an issue? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> Long after 60 seconds of the file creation, > > > > > > > > > > > > > > > > >> command ls still could not find the file. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> I got permission denied when running the two > > > > > > > > > > > > > > > > >> echo commands you suggested. > > > > > > > > > > > > > > > > >> I DO have sudo permission. I also tried vi > > > > > > > > > > > > > > > > >> those files but got error of > > > > > > > > > > > > > > > > >> "/proc/sys/pvfs2/acache/timeout-msecs" E667: > > > > > > > > > > > > > > > > >> Fsync failed > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> Also, how do I set this automatically after > > > > > > > > > > > > > > > > >> system reboot? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> Wei-keng > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >>> On Mar 15, 2015, at 2:16 PM, Becky Ligon > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> Wei-keng: > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> This is most likely a caching issue with > > > > > > > > > > > > > > > > >>> the client. By default, we set the cache > > > > > > > > > > > > > > > > >>> to timeout after 60 seconds, which may be > > > > > > > > > > > > > > > > >>> too high in your environment. Or, you have > > > > > > > > > > > > > > > > >>> deleted and redefined a file using the same > > > > > > > > > > > > > > > > >>> name outside of the client where you are > > > > > > > > > > > > > > > > >>> seeing the question marks, in which case, > > > > > > > > > > > > > > > > >>> the cache would be wrong for that file. > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> To verify, turn off caching to see if this > > > > > > > > > > > > > > > > >>> resolves your problem: > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> As root on your client machine: > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> echo "0" > > > > > > > > > > > > > > > > > >>> /proc/sys/pvfs2/acache/timeout-msecs > > > > > > > > > > > > > > > > >>> echo "0" > > > > > > > > > > > > > > > > > >>> /proc/sys/pvfs2/ncache/timeout-msecs > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> If this change fixes your problem, try > > > > > > > > > > > > > > > > >>> setting the timeout-msecs to something more > > > > > > > > > > > > > > > > >>> appropriate for your environment. > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> Becky > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> On Sun, Mar 15, 2015 at 11:43 AM, Wei-keng > > > > > > > > > > > > > > > > >>> Liao <[email protected]> wrote: > > > > > > > > > > > > > > > > >>> Hi > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> I am having problems with OrangeFS 2.9.1 > > > > > > > > > > > > > > > > >>> and MPICH 3.1.4. > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> Here is my system settings: > > > > > > > > > > > > > > > > >>> Linux Kernel 2.6.32 > > > > > > > > > > > > > > > > >>> Berkeley DB version 6.1.19 > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> A simple MPI test program that calls > > > > > > > > > > > > > > > > >>> MPI_File_open and MPI_File_write_all > > > > > > > > > > > > > > > > >>> is used and ran two processes on the same > > > > > > > > > > > > > > > > >>> host. > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> The MPI commands I used with prefix file > > > > > > > > > > > > > > > > >>> names to force ADIO drivers: > > > > > > > > > > > > > > > > >>> mpiexec -n 2 coll_write > > > > > > > > > > > > > > > > >>> /orangefs/wkliao/testfile > > > > > > > > > > > > > > > > >>> mpiexec -n 2 coll_write > > > > > > > > > > > > > > > > >>> pvfs2:/orangefs/wkliao/testfile.pvfs2 > > > > > > > > > > > > > > > > >>> mpiexec -n 2 coll_write > > > > > > > > > > > > > > > > >>> ufs:/orangefs/wkliao/testfile.ufs > > > > > > > > > > > > > > > > >>> The first two will use the pvfs2 driver and > > > > > > > > > > > > > > > > >>> the 3rd the ufs driver. > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> Here is what I see when running "ls -l" and > > > > > > > > > > > > > > > > >>> "pvfs2-ls -l" commands. > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> % ls -l /orangefs/wkliao/ > > > > > > > > > > > > > > > > >>> ls: cannot access > > > > > > > > > > > > > > > > >>> /orangefs/wkliao/testfile: No such file or > > > > > > > > > > > > > > > > >>> directory > > > > > > > > > > > > > > > > >>> ls: cannot access > > > > > > > > > > > > > > > > >>> /orangefs/wkliao/testfile.pvfs2: No such > > > > > > > > > > > > > > > > >>> file or directory > > > > > > > > > > > > > > > > >>> total 31252 > > > > > > > > > > > > > > > > >>> ?????????? ? ? ? ? > > > > > > > > > > > > > > > > >>> ? testfile > > > > > > > > > > > > > > > > >>> ?????????? ? ? ? ? > > > > > > > > > > > > > > > > >>> ? testfile.pvfs2 > > > > > > > > > > > > > > > > >>> -rw------- 1 wkliao users 32000000 Mar 13 > > > > > > > > > > > > > > > > >>> 18:55 testfile.ufs > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> % pvfs2-ls -l /orangefs/wkliao/ > > > > > > > > > > > > > > > > >>> -rw-r--r-- 1 wkliao users > > > > > > > > > > > > > > > > >>> 31000000 2015-03-13 18:55 testfile > > > > > > > > > > > > > > > > >>> -rw------- 1 wkliao users > > > > > > > > > > > > > > > > >>> 32000000 2015-03-13 18:55 testfile.ufs > > > > > > > > > > > > > > > > >>> -rw-r--r-- 1 wkliao users > > > > > > > > > > > > > > > > >>> 31000000 2015-03-13 18:55 testfile.pvfs2 > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> My config.log file for building > > > > > > > > > > > > > > > > >>> orangefhttp://www.scientificlinux.org/downloads/sl-versions/sl6/s > > > > > > > > > > > > > > > > >>> can be found in this URL > > > > > > > > > > > > > > > > >>> http://www.ece.northwestern.edu/~wkliao/config.log > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> Wei-keng > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> _______________________________________________ > > > > > > > > > > > > > > > > >>> Pvfs2-users mailing list > > > > > > > > > > > > > > > > >>> [email protected] > > > > > > > > > > > > > > > > >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > > > > > Research Associate > > > > > > > > > > > > > > > > Clemson University > > > > > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > > > > Research Associate > > > > > > > > > > > > > > > Clemson University > > > > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > > > > Research Associate > > > > > > > > > > > > > > > Clemson University > > > > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > > > Research Associate > > > > > > > > > > > > > > Clemson University > > > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > > Research Associate > > > > > > > > > > > > > Clemson University > > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > Research Associate > > > > > > > > > > > > Clemson University > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > Research Associate > > > > > > > > > > > > Clemson University > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Becky Ligon > > > > > > > > > > > > Research Associate > > > > > > > > > > > > Clemson University > > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Becky Ligon > > > > > > > > > > > Research Associate > > > > > > > > > > > Clemson University > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Becky Ligon > > > > > > > > > > > Research Associate > > > > > > > > > > > Clemson University > > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Becky Ligon > > > > > > > > > > Research Associate > > > > > > > > > > Clemson University > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Becky Ligon > > > > > > > > > > Research Associate > > > > > > > > > > Clemson University > > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Becky Ligon > > > > > > > > > Research Associate > > > > > > > > > Clemson University > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Becky Ligon > > > > > > > > > Research Associate > > > > > > > > > Clemson University > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Becky Ligon > > > > > > > > > Research Associate > > > > > > > > > Clemson University > > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Becky Ligon > > > > > > > > Research Associate > > > > > > > > Clemson University > > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Becky Ligon > > > > > > > Research Associate > > > > > > > Clemson University > > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Becky Ligon > > > > > > Research Associate > > > > > > Clemson University > > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Becky Ligon > > > > > Research Associate > > > > > Clemson University > > > > > Clemson, SC > > > > > > > > > > > > > > > > > > > > -- > > > > Becky Ligon > > > > Research Associate > > > > Clemson University > > > > Clemson, SC > > > > > > > > > > > > > > > > -- > > > > Becky Ligon > > > > Research Associate > > > > Clemson University > > > > Clemson, SC > > > > > > > > > > > > > > > -- > > > Becky Ligon > > > Research Associate > > > Clemson University > > > Clemson, SC > > > > > > > > > > > > -- > > > Becky Ligon > > > Research Associate > > > Clemson University > > > Clemson, SC > > > > > > > > > > -- > > Becky Ligon > > Research Associate > > Clemson University > > Clemson, SC > > > > > > > > -- > > Becky Ligon > > Research Associate > > Clemson University > > Clemson, SC > > > > > -- > Becky Ligon > Research Associate > Clemson University > Clemson, SC _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
