Thanks for the response. I have been having this problem for a while but
now that I test again I cant reproduce it... If it comes backup I will
run these commands thanks again..
On 2/11/14, 9:22 PM, Venky Shankar wrote:
Could you provide this information from the server (also the
client/server logs):
cat /proc/<pid>/task/*/stack
ls -l /proc/<pid>/task/*/fd
<pid> == process ID of glusterfsd
On Fri, Feb 7, 2014 at 6:14 PM, Barry Stetler <ba...@hivelocity.net
<mailto:ba...@hivelocity.net>> wrote:
We started using gluster in a custom solution recently. Basically
we setup a gluster volume that we mount at /home. Then we give
users their own home directory with a gluster quota. They have
FTP, SFTP and SSHMOUNT access. I first used vsftpd and notice that
ftp connections will not close and the load goes up ever couple of
days the funny thing is the load does not seem to slow down the
server. The load went up to 20 in the last two months but the
server was not slow at all... At first I thought it was vsftpd so
I changed to proftpd then the issue came back. If I reboot the
server it goes away for a few days depending on how many ftp
connections. I noticed in the dmesg log this making me think the
issue was some kind of leak in gluster-fuse
This is on CentOS 6.5 64 bit.
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task proftpd:7831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
proftpd D 0000000000000000 0 7831 4384 0x00000004
ffff88031f3bde38 0000000000000082 0000000000000000 ffffffff81051439
ffff88031f3bddc8 0000000300000000 ffff88031f3bddd8 ffff880335936040
ffff8803365cfab8 ffff88031f3bdfd8 000000000000fb88 ffff8803365cfab8
Call Trace:
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0213075>] fuse_request_send+0xe5/0x290 [fuse]
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0218b36>] fuse_flush+0x106/0x140 [fuse]
[<ffffffff8117e0bc>] filp_close+0x3c/0x90
[<ffffffff8117e1b5>] sys_close+0xa5/0x100
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
I appreciate any help I can get on this...
Thank you
--
Barry Stetler
Hivelocity Hosting
http://hivelocity.net
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
Barry Stetler
Hivelocity Hosting
http://hivelocity.net
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
