hello there,
I have been trying to setup pvfs2 on a small cluster (16 servers, and
16 clients) running SUSE10SP2-64bit and I am running into some major
performance problems that are causing me to doubt my install. I am
hoping to get some help from this great users group.
The server side of things seems to be working great. I have 14 I/O
servers, and 2 metaDB servers. I don't see any errors at all. I can
run the pvfs2 native tools (ex: pvfs2-cp) and I am seeing some
fantastic results (500+ Mbs). The pvfs2-fs.conf is bone stock and is
as generated by pvfs2-genconfig.
When I use the native linux FS commands (ex: cp, rsync...) I am seeing
some dismal results that are 10-15 times slower then the pvfs2 FS
tools. The kernel driver build goes very smoothly, and I am not seeing
any errors. Here are the steps that I am taking:
cd /tmp
tar zxvf pvfs-2.8.1.tar.gz
cd pvfs-2.8.1/
./configure --prefix=/opt/pvfs2 --with-kernel=/tmp/linux
--disable-server --disable-karma
make kmod
make kmod_install
depmod -a
modprobe pvfs2
/opt/pvfs2/sbin/pvfs2-client -p /opt/pvfs2/sbin/pvfs2-client-core
mount -t pvfs2 tcp://lab1:3334/pvfs2-fs /mnt/pvfs2
I did an strace on the pvfs2-client process and I am seeing lots and
lots of retries:
readv(26, [{"p\27\0\0\2\0\0\0\4\0\0\0\0\0\0\0C\362\0\0d\0\0\0\244\1"...,
128}], 1) = 128
read(5, 0x7d4450, 8544) = -1 EAGAIN (Resource
temporarily unavailable)
getrusage(RUSAGE_SELF, {ru_utime={2, 960185}, ru_stime={7, 904494}, ...})
= 0
writev(5, [{"AQ\0\0", 4}, {")\5\3 ", 4}, {"5\316\0\0\0\0\0\0", 8},
{"\4\0\0\377\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0C\362"..., 8224}],
4) = 8240
poll([{fd=5, events=POLLIN, revents=POLLIN}], 1, 10) = 1
read(5, "AQ\0\0)\5\3 6\316\0\0\0\0\0\0\5\0\0\377\0\0\0\0C\362\0"...,
8544) = 8544
read(5, 0x7d1020, 8544) = -1 EAGAIN (Resource
temporarily unavailable)
getrusage(RUSAGE_SELF, {ru_utime={2, 960185}, ru_stime={7, 904494}, ...})
= 0
epoll_ctl(6, EPOLL_CTL_ADD, 26, {EPOLLIN|EPOLLERR|EPOLLHUP,
{u32=6084112, u64=6084112}}) = -1 EEXIST (File exists)
epoll_wait(6, {}, 16, 0) = 0
read(5, 0x7d2020, 8544) = -1 EAGAIN (Resource
temporarily unavailable)
writev(26,
[{"\277\312\0\0\2\0\0\0\246\267\0\0\0\0\0\0L\0\0\0\0\0\0\0"...,
24}, {"p\27\0\0\2\0\0\0\10\0\0\0\0\0\0\0C\362\0\0d\0\0\0\1\0\0"...,
76}], 2) = 100
epoll_wait(6, {{EPOLLIN, {u32=6084112, u64=6084112}}}, 16, 10) = 1
fcntl(26, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
recvfrom(26,
"\277\312\0\0\4\0\0\0\246\267\0\0\0\0\0\0\30\0\0\0\0\0\0"...,
24, MSG_PEEK|MSG_NOSIGNAL, NULL, NULL) = 24
fcntl(26, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
recvfrom(26,
"\277\312\0\0\4\0\0\0\246\267\0\0\0\0\0\0\30\0\0\0\0\0\0"...,
24, MSG_NOSIGNAL, NULL, NULL) = 24
readv(26, [{"p\27\0\0\2\0\0\0\10\0\0\0\0\0\0\0\331\323\17\0\0\0\0\0"...,
24}], 1) = 24
read(5, 0x7d2020, 8544) = -1 EAGAIN (Resource
temporarily unavailable)
epoll_ctl(6, EPOLL_CTL_ADD, 26, {EPOLLIN|EPOLLERR|EPOLLHUP,
{u32=6084112, u64=6084112}}) = -1 EEXIST (File exists)
epoll_wait(6, {}, 16, 0) = 0
read(5, <unfinished ...>
I appreciate any and all feedback!
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
