On Mar 16, 2015, at 12:02 PM, Becky Ligon wrote: > Wei-Keing: > > Did you umount and mount the filesystem? If not, umount the filesystem, > restart the client core, and then mount the filesystem again.
Yes. My restart command ran "/sbin/servive pvfs2-server restart" the script contains both client and server commands, include client's umount and mount. > I also suggest that you define your environment so that each of your machines > (bigdata, bigdata1, bigdata2, bigdata3) have their pvfs servers configured to > handle both I/O and metadata. To do this, you will have to recreate the > filesystem. My Orangefs is newly created and the test program is the first one run in parallel on it. Isn't my configuration legit (one metadata sever and 4 data servers) for orangefs setup? Again, my mpi test program ran 2 processes locally on the metadata server which is both data server and client. Wei-keng > > Becky > > On Mon, Mar 16, 2015 at 11:30 AM, Wei-keng Liao > <[email protected]> wrote: > HI, Becky > > I tried the command option "-a 0 -n 0" and restart the client/server, but the > same issue persists. > pvfs2-ping command shows one metadata server and 4 data servers. > I ran my test program on the metadata server. > > meta servers: > tcp://bigdata:3334 Ok > > data servers: > tcp://bigdata:3334 Ok > tcp://bigdata1:3334 Ok > tcp://bigdata2:3334 Ok > tcp://bigdata3:3334 Ok > > > Wei-keng > > On Mar 16, 2015, at 8:26 AM, Becky Ligon wrote: > > > Caching is still an issue if you have servers on more than one machine and > > those servers provide metadata. Even in a one-server environment, it could > > make a difference. > > > > The "ls" command uses the kernel module and client core, which in turn use > > the caches, while the pvfs2-ls command does not. > > > > If you don't have the appropriate sudo permissions to modify the /proc > > filesystem, then you can start the client with the caches turned off. > > > > Example: > > > > ./pvfs2-client -a 0 -n 0 > > > > > > If you execute pvfs2-client --help, you will see these options. > > > > > > Becky > > > > Sent from my iPhone > > > >> On Mar 15, 2015, at 5:05 PM, Wei-keng Liao <[email protected]> > >> wrote: > >> > >> I assume after 60 seconds, the client will flush the cache. > >> Please note I am running orangefs client and server on the same machine. > >> In this case, should caching become an issue? > >> > >> Long after 60 seconds of the file creation, command ls still could not > >> find the file. > >> > >> I got permission denied when running the two echo commands you suggested. > >> I DO have sudo permission. I also tried vi those files but got error of > >> "/proc/sys/pvfs2/acache/timeout-msecs" E667: Fsync failed > >> > >> Also, how do I set this automatically after system reboot? > >> > >> Wei-keng > >> > >>> On Mar 15, 2015, at 2:16 PM, Becky Ligon wrote: > >>> > >>> Wei-keng: > >>> > >>> This is most likely a caching issue with the client. By default, we set > >>> the cache to timeout after 60 seconds, which may be too high in your > >>> environment. Or, you have deleted and redefined a file using the same > >>> name outside of the client where you are seeing the question marks, in > >>> which case, the cache would be wrong for that file. > >>> > >>> To verify, turn off caching to see if this resolves your problem: > >>> > >>> As root on your client machine: > >>> > >>> echo "0" > /proc/sys/pvfs2/acache/timeout-msecs > >>> echo "0" > /proc/sys/pvfs2/ncache/timeout-msecs > >>> > >>> If this change fixes your problem, try setting the timeout-msecs to > >>> something more appropriate for your environment. > >>> > >>> Becky > >>> > >>> On Sun, Mar 15, 2015 at 11:43 AM, Wei-keng Liao > >>> <[email protected]> wrote: > >>> Hi > >>> > >>> I am having problems with OrangeFS 2.9.1 and MPICH 3.1.4. > >>> > >>> Here is my system settings: > >>> Linux Kernel 2.6.32 > >>> Berkeley DB version 6.1.19 > >>> > >>> A simple MPI test program that calls MPI_File_open and MPI_File_write_all > >>> is used and ran two processes on the same host. > >>> > >>> The MPI commands I used with prefix file names to force ADIO drivers: > >>> mpiexec -n 2 coll_write /orangefs/wkliao/testfile > >>> mpiexec -n 2 coll_write pvfs2:/orangefs/wkliao/testfile.pvfs2 > >>> mpiexec -n 2 coll_write ufs:/orangefs/wkliao/testfile.ufs > >>> The first two will use the pvfs2 driver and the 3rd the ufs driver. > >>> > >>> Here is what I see when running "ls -l" and "pvfs2-ls -l" commands. > >>> > >>> % ls -l /orangefs/wkliao/ > >>> ls: cannot access /orangefs/wkliao/testfile: No such file or directory > >>> ls: cannot access /orangefs/wkliao/testfile.pvfs2: No such file or > >>> directory > >>> total 31252 > >>> ?????????? ? ? ? ? ? testfile > >>> ?????????? ? ? ? ? ? testfile.pvfs2 > >>> -rw------- 1 wkliao users 32000000 Mar 13 18:55 testfile.ufs > >>> > >>> % pvfs2-ls -l /orangefs/wkliao/ > >>> -rw-r--r-- 1 wkliao users 31000000 2015-03-13 18:55 testfile > >>> -rw------- 1 wkliao users 32000000 2015-03-13 18:55 > >>> testfile.ufs > >>> -rw-r--r-- 1 wkliao users 31000000 2015-03-13 18:55 > >>> testfile.pvfs2 > >>> > >>> My config.log file for building orangefs can be found in this URL > >>> http://www.ece.northwestern.edu/~wkliao/config.log > >>> > >>> Wei-keng > >>> > >>> _______________________________________________ > >>> Pvfs2-users mailing list > >>> [email protected] > >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > >> > > > > > -- > Becky Ligon > Research Associate > Clemson University > Clemson, SC _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
