Dear Cephers,
We're using the ceph file system with the fuse client, and lately some
of our processes are getting stuck seemingly waiting for fuse
operations. At the same time, the cluster is healthy, no slow requests,
all OSDs up and running, and both the MDS and the fuse client think that
there are no pending operations. The situation is semi-reproducible.
When I run a various cluster jobs, some get stuck after a few hours of
correct operation. The cluster is on ceph 10.2.5 and 10.2.6, the fuse
clients are 10.2.6, but I have tried 10.2.5 and 10.2.3, all of which
have the same issue. This is on CentOS (7.2 for the clients, 7.3 for
the MDS/OSDs).
Here are some details:
The node with the stuck processes:
[root@worker1070 ~]# ps -auxwww | grep 30519
*apataki 30519 39.8 0.9 8728064 5257588 ? Dl 12:11 60:50
./Arepo param.txt 2 6*
[root@worker1070 ~]# cat /proc/30519/stack
*[<ffffffffa0a1d7bb>] fuse_file_aio_write+0xbb/0x340 [fuse]*
[<ffffffff811ddd3d>] do_sync_write+0x8d/0xd0
[<ffffffff811de55d>] vfs_write+0xbd/0x1e0
[<ffffffff811defff>] SyS_write+0x7f/0xe0
[<ffffffff816458c9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
[root@worker1070 ~]# ps -auxwww | grep 30533
*apataki 30533 39.8 0.9 8795316 5261308 ? Sl 12:11 60:55
./Arepo param.txt 2 6*
[root@worker1070 ~]# cat /proc/30533/stack
*[<ffffffffa0a12241>] wait_answer_interruptible+0x91/0xe0 [fuse]**
**[<ffffffffa0a12653>] __fuse_request_send+0x253/0x2c0 [fuse]**
**[<ffffffffa0a126d2>] fuse_request_send+0x12/0x20 [fuse]**
**[<ffffffffa0a1b966>] fuse_send_write+0xd6/0x110 [fuse]**
**[<ffffffffa0a1d45d>] fuse_perform_write+0x2ed/0x590 [fuse]**
**[<ffffffffa0a1d9a1>] fuse_file_aio_write+0x2a1/0x340 [fuse]**
**[<ffffffff811ddd3d>] do_sync_write+0x8d/0xd0*
[<ffffffff811de55d>] vfs_write+0xbd/0x1e0
[<ffffffff811defff>] SyS_write+0x7f/0xe0
[<ffffffff816458c9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
Presumably the second process is waiting on the first holding some lock ...
The fuse client on the node:
[root@worker1070 ~]# ceph daemon
/var/run/ceph/ceph-client.admin.asok status
{
"metadata": {
"ceph_sha1": "656b5b63ed7c43bd014bcafd81b001959d5f089f",
"ceph_version": "ceph version 10.2.6
(656b5b63ed7c43bd014bcafd81b001959d5f089f)",
"entity_id": "admin",
"hostname": "worker1070",
"mount_point": "\/mnt\/ceph",
"root": "\/"
},
"dentry_count": 40,
"dentry_pinned_count": 23,
"inode_count": 123,
"mds_epoch": 19041,
"osd_epoch": 462327,
"osd_epoch_barrier": 462326
}
[root@worker1070 ~]# ceph daemon
/var/run/ceph/ceph-client.admin.asok mds_sessions
{
"id": 3616543,
"sessions": [
{
"mds": 0,
"addr": "10.128.128.110:6800\/909443124",
"seq": 338,
"cap_gen": 0,
"cap_ttl": "2017-03-13 14:47:37.575229",
"last_cap_renew_request": "2017-03-13 14:46:37.575229",
"cap_renew_seq": 12694,
"num_caps": 713,
"state": "open"
}
],
"mdsmap_epoch": 19041
}
[root@worker1070 ~]# ceph daemon
/var/run/ceph/ceph-client.admin.asok mds_requests
{}
The overall cluster health and the MDS:
[root@cephosd000 ~]# ceph -s
cluster d7b33135-0940-4e48-8aa6-1d2026597c2f
health HEALTH_WARN
noscrub,nodeep-scrub,require_jewel_osds flag(s) set
monmap e17: 3 mons at
{hyperv029=10.4.36.179:6789/0,hyperv030=10.4.36.180:6789/0,hyperv031=10.4.36.181:6789/0}
election epoch 29148, quorum 0,1,2
hyperv029,hyperv030,hyperv031
fsmap e19041: 1/1/1 up {0=cephosd000=up:active}
osdmap e462328: 624 osds: 624 up, 624 in
flags noscrub,nodeep-scrub,require_jewel_osds
pgmap v44458747: 42496 pgs, 6 pools, 924 TB data, 272 Mobjects
2154 TB used, 1791 TB / 3946 TB avail
42496 active+clean
client io 86911 kB/s rd, 556 MB/s wr, 227 op/s rd, 303 op/s wr
[root@cephosd000 ~]# ceph daemon
/var/run/ceph/ceph-mds.cephosd000.asok ops
{
"ops": [],
"num_ops": 0
}
The odd thing is that if in this state I restart the MDS, the client
process wakes up and proceeds with its work without any errors. As if a
request was lost and somehow retransmitted/restarted when the MDS got
restarted and the fuse layer reconnected to it.
When I try to attach a gdb session to either of the client processes,
gdb just hangs. However, right after the MDS restart gdb attaches to
the process successfully, and shows that the getting stuck happened on
closing of a file. In fact, it looks like both processes were trying to
write to the same file opened with fopen("filename", "a") and close it:
(gdb) where
#0 0x00002aaaadc53abd in write () from /lib64/libc.so.6
#1 0x00002aaaadbe2383 in _IO_new_file_write () from /lib64/libc.so.6
#2 0x00002aaaadbe37ec in __GI__IO_do_write () from /lib64/libc.so.6
#3 0x00002aaaadbe30e0 in __GI__IO_file_close_it () from
/lib64/libc.so.6
#4 0x00002aaaadbd7020 in fclose@@GLIBC_2.2.5 () from /lib64/libc.so.6
...
It seems like the fuse client wasn't handling this case well, when two
processes write to the same file and close it perhaps? This is just a
speculation. Any ideas on how to proceed? Is there perhaps a known
issue related to this?
Thanks,
Andras
apat...@flatironinstitute.org
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
