[Gluster-devel] glusterfs client problem
I'm noticing a problem with our test setup with regard to (reasonably) heavy read/write usage. the probelem we're having is that during an rsync of content, the sync bails due to the mount being lost with the following errors: rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trailers" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport endpoint is not connected (107) normal logging shows nothing either client or server-side, but running logging in DEBUG mode shows the following at the end of the client log right as it breaks: [Apr 02 13:25:11] [DEBUG/common-utils.c:213/gf_print_trace()] debug-backtrace:Got signal (11), printing backtrace [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(gf_print_trace+0x1f) [0x2a9556030f] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libc.so.6 [0x35b992e2b0] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libpthread.so.0(__pthread_mutex_destroy+0) [0x35ba807ab0] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/cluster/afr.so [0x2a958b840c] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so [0x2a957b06c2] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so [0x2a957b3196] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(epoll_iteration+0xf8) [0x2a955616f8] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:[glusterfs] [0x4031b7] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x35b991c3fb] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:[glusterfs] [0x402bba] the server log shows the following at the time it breaks: [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x510470 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1012 (fd=8) [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x510160 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1013 (fd=7) [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x502300 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1014 (fd=4) we're using 4 bricks in this setup and for the moment, just one client (would like to scale between 20-30 clients and 4-8 server bricks). the same behavior is observed when used with or without any combination of any of the performance translators as well as with or without file replication. alu, random, and round-robin schedulers were al
Re: [Gluster-devel] glusterfs client problem
Does this get better if you use the --bwlimit flag on rsync? On Apr 2, 2007, at 3:46 PM, Shawn Northart wrote: I'm noticing a problem with our test setup with regard to (reasonably) heavy read/write usage. the probelem we're having is that during an rsync of content, the sync bails due to the mount being lost with the following errors: rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trailers" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport endpoint is not connected (107) normal logging shows nothing either client or server-side, but running logging in DEBUG mode shows the following at the end of the client log right as it breaks: [Apr 02 13:25:11] [DEBUG/common-utils.c:213/gf_print_trace()] debug-backtrace:Got signal (11), printing backtrace [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0 (gf_print_trace+0x1f) [0x2a9556030f] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libc.so.6 [0x35b992e2b0] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libpthread.so.0(__pthread_mutex_destroy+0) [0x35ba807ab0] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0- pre2.2/xlator/cluster/afr.so [0x2a958b840c] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0- pre2.2/xlator/protocol/client.so [0x2a957b06c2] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0- pre2.2/xlator/protocol/client.so [0x2a957b3196] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0 (epoll_iteration+0xf8) [0x2a955616f8] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:[glusterfs] [0x4031b7] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x35b991c3fb] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:[glusterfs] [0x402bba] the server log shows the following at the time it breaks: [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x510470 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1012 (fd=8) [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x510160 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1013 (fd=7) [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x502300 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1014 (fd=4) we're using 4 bricks in this setup and for the moment, just one client (would like to scale between 20-30 clients and 4-8 server bricks). the same behavior is observed when used with or without any combination of
Re: [Gluster-devel] glusterfs client problem
Hi Shawn, Can you give us the exact rsync command you used? Thanks Krishna On 4/3/07, Shawn Northart <[EMAIL PROTECTED]> wrote: I'm noticing a problem with our test setup with regard to (reasonably) heavy read/write usage. the probelem we're having is that during an rsync of content, the sync bails due to the mount being lost with the following errors: rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trailers" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: Transport endpoint is not connected (107) rsync: recv_generator: mkdir "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport endpoint is not connected (107) rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport endpoint is not connected (107) normal logging shows nothing either client or server-side, but running logging in DEBUG mode shows the following at the end of the client log right as it breaks: [Apr 02 13:25:11] [DEBUG/common-utils.c:213/gf_print_trace()] debug-backtrace:Got signal (11), printing backtrace [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(gf_print_trace+0x1f) [0x2a9556030f] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libc.so.6 [0x35b992e2b0] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libpthread.so.0(__pthread_mutex_destroy+0) [0x35ba807ab0] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/cluster/afr.so [0x2a958b840c] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so [0x2a957b06c2] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so [0x2a957b3196] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(epoll_iteration+0xf8) [0x2a955616f8] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:[glusterfs] [0x4031b7] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:/lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x35b991c3fb] [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] debug-backtrace:[glusterfs] [0x402bba] the server log shows the following at the time it breaks: [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x510470 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1012 (fd=8) [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x510160 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1013 (fd=7) [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 [Apr 02 15:30:09] [DEBUG/protocol.c:244/gf_block_unserialize_transport()] libglusterfs/protocol:gf_block_unserialize_transport: full_read of header failed [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] protocol/server:cleaned up xl_private of 0x502300 [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] tcp/server:destroying transport object for 192.168.0.96:1014 (fd=4) we're using 4 bricks in this setup and for the moment, just one client (would like to scale between 20-30 clients and 4-8 server bricks). the same behavior is observed when used with or without any combination o
Re: [Gluster-devel] glusterfs client problem
sorry, forgot that one. the command used was: rsync -av --stats --progress --delete i haven't tried setting a bwlimit yet and i'd prefer not to have to if possible. i've got roughly 450GB of data i want to sync over and the faster i can do it, the better. i will try it just to see if it makes things any better.the network is all copper gig with both interfaces trunked and vlan'd (on both client and server). a couple of other things that just came to mind are that i didn't see this exact behavior during the initial rsync. i have three directories i'm trying to sync and when run concurrently, i would see the problem. when run one at a time, the sync would seem to complete without incident. the only difference in the command i ran was that i omitted the --delete flag. ~Shawn On Tue, 2007-04-03 at 11:07 +0530, Krishna Srinivas wrote: > Hi Shawn, > > Can you give us the exact rsync command you used? > > Thanks > Krishna > > On 4/3/07, Shawn Northart <[EMAIL PROTECTED]> wrote: > > I'm noticing a problem with our test setup with regard to (reasonably) > > heavy read/write usage. > > the probelem we're having is that during an rsync of content, the sync > > bails due to the mount being lost with the following errors: > > > > > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trailers" failed: > > Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport > > endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: > > Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: > > Transport endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" > > failed: Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: > > Transport endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" > > failed: Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport > > endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" > > failed: Transport endpoint is not connected (107) > > > > > > normal logging shows nothing either client or server-side, but running > > logging in DEBUG mode shows the following at the end of the client log > > right as it breaks: > > > > > > [Apr 02 13:25:11] [DEBUG/common-utils.c:213/gf_print_trace()] > > debug-backtrace:Got signal (11), printing backtrace > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(gf_print_trace+0x1f) > > [0x2a9556030f] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/lib64/tls/libc.so.6 [0x35b992e2b0] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/lib64/tls/libpthread.so.0(__pthread_mutex_destroy+0) > > [0x35ba807ab0] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/cluster/afr.so > > [0x2a958b840c] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so > > [0x2a957b06c2] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so > > [0x2a957b3196] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(epoll_iteration+0xf8) > > [0x2a955616f8] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:[glusterfs] [0x4031b7] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/lib64/tls/libc.so.6(__libc_start_main+0xdb) > > [0x35b991c3fb] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:[glusterfs] [0x402bba] > > > > > > > > the server log shows the following at the time it breaks: > > > > [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] > > libglusterfs:full_rw: 0 bytes r/w instead of 113 > > [Apr 02 15:30:09] > > [DEBUG/protocol.c:244/gf_block_unserialize_transport()] > > libglusterfs/protocol:gf_block_unserialize_transport: full_read of > > header failed > > [Apr 02 15:30:09] [DEBUG/proto-srv.c:2868/proto_srv_cleanup()] > > protocol/server:cleaned up xl_private of 0x510470 > > [Apr 02 15:30:09] [DEBUG/tcp-server.c:243/gf_transport_fini()] > > tcp/server:destroying transport object for 192.168.0.96:1012 (fd=8) > > [Apr 02 15:30:09] [ERROR/comm
Re: [Gluster-devel] glusterfs client problem
Hi Shawn, We committed a fix today which might have fixed your problem, can you check with the latest source? I tried with two instances of rsync but the problem did not reproduce. If you still see the problem can you give more detailed steps to reproduce the problem? Thanks Krishna On 4/4/07, Shawn Northart <[EMAIL PROTECTED]> wrote: sorry, forgot that one. the command used was: rsync -av --stats --progress --delete i haven't tried setting a bwlimit yet and i'd prefer not to have to if possible. i've got roughly 450GB of data i want to sync over and the faster i can do it, the better. i will try it just to see if it makes things any better.the network is all copper gig with both interfaces trunked and vlan'd (on both client and server). a couple of other things that just came to mind are that i didn't see this exact behavior during the initial rsync. i have three directories i'm trying to sync and when run concurrently, i would see the problem. when run one at a time, the sync would seem to complete without incident. the only difference in the command i ran was that i omitted the --delete flag. ~Shawn On Tue, 2007-04-03 at 11:07 +0530, Krishna Srinivas wrote: > Hi Shawn, > > Can you give us the exact rsync command you used? > > Thanks > Krishna > > On 4/3/07, Shawn Northart <[EMAIL PROTECTED]> wrote: > > I'm noticing a problem with our test setup with regard to (reasonably) > > heavy read/write usage. > > the probelem we're having is that during an rsync of content, the sync > > bails due to the mount being lost with the following errors: > > > > > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trailers" failed: > > Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: Transport > > endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember" failed: > > Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" failed: > > Transport endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/bardoux" > > failed: Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" failed: > > Transport endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/trialmember/images" > > failed: Transport endpoint is not connected (107) > > rsync: recv_generator: mkdir > > "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" failed: Transport > > endpoint is not connected (107) > > rsync: stat "/vol/vol0/sites/TESTSITE.com/htdocs/upgrade_trailers" > > failed: Transport endpoint is not connected (107) > > > > > > normal logging shows nothing either client or server-side, but running > > logging in DEBUG mode shows the following at the end of the client log > > right as it breaks: > > > > > > [Apr 02 13:25:11] [DEBUG/common-utils.c:213/gf_print_trace()] > > debug-backtrace:Got signal (11), printing backtrace > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(gf_print_trace+0x1f) [0x2a9556030f] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/lib64/tls/libc.so.6 [0x35b992e2b0] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/lib64/tls/libpthread.so.0(__pthread_mutex_destroy+0) > > [0x35ba807ab0] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/cluster/afr.so [0x2a958b840c] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so [0x2a957b06c2] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/glusterfs/1.3.0-pre2.2/xlator/protocol/client.so [0x2a957b3196] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/usr/local/glusterfs-mainline/lib/libglusterfs.so.0(epoll_iteration+0xf8) [0x2a955616f8] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:[glusterfs] [0x4031b7] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:/lib64/tls/libc.so.6(__libc_start_main+0xdb) > > [0x35b991c3fb] > > [Apr 02 13:25:11] [DEBUG/common-utils.c:215/gf_print_trace()] > > debug-backtrace:[glusterfs] [0x402bba] > > > > > > > > the server log shows the following at the time it breaks: > > > > [Apr 02 15:30:09] [ERROR/common-utils.c:54/full_rw()] > > libglusterfs:full_rw: 0 bytes r/w instead of 113 > > [Apr 02 15:30:09] > > [DEBUG/protocol.c:244/gf_block_unserialize_transport()] > > libglusterfs/protocol:gf_block_unserialize_transport:
Re: [Gluster-devel] glusterfs client problem
On Wed, 2007-04-04 at 21:42 +0530, Krishna Srinivas wrote: > > Hi Shawn, > > We committed a fix today which might have fixed your problem, can > you check with the latest source? > > I tried with two instances of rsync but the problem did not > reproduce. If you still see the problem can you give more > detailed steps to reproduce the problem? > > Thanks > Krishna thanks! so far it looks pretty good. i haven't had much of an opportunity to do any extensive testing yet as i've got a few other things pending at the moment. i'll try hammering on this a bit more in the next few days. thanks for your help. ~Shawn ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
