Hi, hoping someone can point me in the right direction. Some of my OSDs have a larger CPU usage (and ops latencies) than others. If I restart the OSD everything runs nicely for some time, then it creeps up.
1) most of my OSDs have ~40% CPU (core) usage (user+sys), some are closer to 80%. Restarting means the offending OSDs only use 40% again. 2) average latencies and CPU usage on the host are the same - so it’s not caused by the host that the OSD is running on 3) I can’t say exactly when or how the issue happens. I can’t even say if it’s the same OSDs. It seems it either happens when something heavy happens in a cluster (like dropping very old snapshots, rebalancing) and then doesn’t come back, or maybe it happens slowly over time and I can’t find it in the graphs. Looking at the graphs it seems to be the former. I have just one suspicion and that is the “fd cache size” - we have it set to 16384 but the open fds suggest there are more open files for the osd process (over 17K fds) - it varies by some hundreds between the osds. Maybe some are just slightly over the limit and the misses cause this? Restarting the OSD clears them (~2K) and they increase over time. I increased it to 32768 yesterday and it consistently nice now, but it might take another few days to manifest… Could this explain it? Any other tips? Thanks Jan _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
