On Tue, Dec 8, 2015 at 10:57 AM, Tom Christensen <pav...@gmail.com> wrote: > We aren't running NFS, but regularly use the kernel driver to map RBDs and > mount filesystems in same. We see very similar behavior across nearly all > kernel versions we've tried. In my experience only very few versions of the > kernel driver survive any sort of crush map change/update while something is > mapped. In fact in the last 2 years I think I've only seen this work on 1 > kernel version unfortunately its badly out of date and we can't run it in > our environment anymore, I think it was a 3.0 kernel version running on > ubuntu 12.04. We have just recently started trying to find a kernel that > will survive OSD outages or changes to the cluster. We're on ubuntu 14.04, > and have tried 3.16, 3.19.0-25, 4.3, and 4.2 without success in the last > week. We only map 1-3 RBDs per client machine at a time but we regularly > will get processes stuck in D state which are accessing the filesystem > inside the RBD and will have to hard reboot the RBD client machine. This is > always associated with a cluster change in some way, reweighting OSDs, > rebooting an OSD host, restarting an individual OSD, adding OSDs, and > removing OSDs all cause the kernel client to hang. If no change is made to > the cluster, the kernel client will be happy for weeks.
There are a couple of known bugs in the remap/resubmit area, but those are supposedly corner cases (like *all* the OSDs going down and then back up, etc). I had no idea it was that severe and goes that back. Apparently triggering it requires a heavier load, as we've never seen anything like that in our tests. For unrelated reasons, remap/resubmit code is getting entirely rewritten for kernel 4.5, so, if you've been dealing with this issue for the last two years (I don't remember seeing any tickets listing that many kernel versions and not mentioning NFS), I'm afraid the best course of action for you would be to wait for 4.5 to come out and try it. If you'd be willing to test out an early version on one of more of your client boxes, I can ping you when it's ready. I'll take a look at 3.0 vs 3.16 with an eye on remap code. Did you happen to try 3.10? It sounds like you can reproduce this pretty easily. Can you get it to lock up and do: # cat /sys/kernel/debug/ceph/*/osdmap # cat /sys/kernel/debug/ceph/*/osdc $ ceph status and bunch of times? I have a hunch that kernel client simply fails to request enough of new osdmaps after the cluster topology changes under load. Thanks, Ilya _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com