Hi Ilya,
Thanks for the clarification - very helpful.
I've lowered osd_map_messages_max to 10, and this resolves the issue
about the kernel being unhappy about large messages when the OSDMap
changes. One comment here though: you mentioned that Luminous uses 40
as the default, which is indeed the case. The documentation for
Luminous (and master), however, says that the default is 100.
One other follow-up question on the kernel client about something I've
been seeing while testing. Does the kernel client clean up when the MDS
asks due to cache pressure? On a machine I ran something that touches a
lot of files, so the kernel client accumulated over 4 million caps.
Many hours after all the activity finished (i.e. many hours after
anything accesses ceph on that node) the kernel client still holds
millions of caps, and the MDS periodically complains about clients not
responding to cache pressure. How is this supposed to be handled?
Obviously asking the kernel to drop caches via /proc/sys/vm/drop_caches
does a very thorough cleanup, but something in the middle would be better.
Andras
On 1/16/19 1:45 PM, Ilya Dryomov wrote:
On Wed, Jan 16, 2019 at 7:12 PM Andras Pataki
<apat...@flatironinstitute.org> wrote:
Hi Ilya/Kjetil,
I've done some debugging and tcpdump-ing to see what the interaction
between the kernel client and the mon looks like. Indeed -
CEPH_MSG_MAX_FRONT defined as 16Mb seems low for the default mon
messages for our cluster (with osd_mon_messages_max at 100). We have
about 3500 osd's, and the kernel advertises itself as older than
This is too big, especially for a fairly large cluster such as yours.
The default was reduced to 40 in luminous. Given about 3500 OSDs, you
might want to set it to 20 or even 10.
Luminous, so it gets full map updates. The FRONT message size on the
wire I saw was over 24Mb. I'll try setting osd_mon_messages_max to 30
and do some more testing, but from the debugging it definitely seems
like the issue.
Is the kernel driver really not up to date to be considered at least a
Luminous client by the mon (i.e. it has some feature really missing)? I
looked at the bits, and the MON seems to want is bit 59 in ceph features
shared by FS_BTIME, FS_CHANGE_ATTR, MSG_ADDR2. Can the kernel client be
used when setting require-min-compat to luminous (either with the 4.19.x
kernel or the Redhat/Centos 7.6 kernel)? Some background here would be
helpful.
Yes, the kernel client is missing support for that feature bit, however
4.13+ and RHEL 7.5+ _can_ be used with require-min-compat-client set to
luminous. See
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/027002.html
Thanks,
Ilya
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
