Hi Wido, I came across this ancient ML entry with no responses and wanted to follow up with you to see if you recalled any solution to this. Copying the ceph-users list to preserve any replies that may result for archival.
I have a couple of boxes with 10x Micron 5100 SATA SSD’s, journaled on Micron 9100 NVMe SSD’s; ceph 10.2.7; Ubuntu 16.04 4.8 kernel. I have noticed now twice that I’ve had SSD’s flapping due to the fstrim eating up the io 100%. It eventually righted itself after a little less than 8 hours. Noout flag was set, so it didn’t create any unnecessary rebalance or whatnot. Timeline showing that only 1 OSD ever went down at a time, but they seemed to go down in a rolling fashion during the fstrim session. You can actually see in the OSD graph all 10 OSD’s on this node go down 1 by 1 over time. And the OSD’s were going down because of: > 2017-07-02 13:47:32.618752 7ff612721700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7ff5ecd0c700' had timed out after 15 > 2017-07-02 13:47:32.618757 7ff612721700 1 heartbeat_map is_healthy > 'FileStore::op_tp thread 0x7ff608d9e700' had timed out after 60 > 2017-07-02 13:47:32.618760 7ff612721700 1 heartbeat_map is_healthy > 'FileStore::op_tp thread 0x7ff608d9e700' had suicide timed out after 180 > 2017-07-02 13:47:32.624567 7ff612721700 -1 common/HeartbeatMap.cc: In > function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, > const char*, time_t)' thread 7ff612721700 time 2017-07-02 13:47:32.618784 > common/HeartbeatMap.cc: 86: FAILED assert(0 == "hit suicide timeout") I am curious if you were able to nice it or something similar to mitigate this issue? Oddly, I have similar machines with Samsung SM863a’s with Intel P3700 journals that do not appear to be affected by the fstrim load issue despite identical weekly cron jobs enabled. Only the Micron drives (newer) have had these issues. Appreciate any pointers, Reed > Wido den Hollander wido at 42on.com > <mailto:ceph-users%40lists.ceph.com?Subject=Re%3A%20%5Bceph-users%5D%20Watch%20for%20fstrim%20running%20on%20your%20Ubuntu%20systems&In-Reply-To=%3C5486BF08.3010505%4042on.com%3E> > Tue Dec 9 01:21:16 PST 2014 > Hi, > > Last sunday I got a call early in the morning that a Ceph cluster was > having some issues. Slow requests and OSDs marking each other down. > > Since this is a 100% SSD cluster I was a bit confused and started > investigating. > > It took me about 15 minutes to see that fstrim was running and was > utilizing the SSDs 100%. > > On Ubuntu 14.04 there is a weekly CRON which executes fstrim-all. It > detects all mountpoints which can be trimmed and starts to trim those. > > On the Intel SSDs used here it caused them to become 100% busy for a > couple of minutes. That was enough for them to no longer respond on > heartbeats, thus timing out and being marked down. > > Luckily we had the "out interval" set to 1800 seconds on that cluster, > so no OSD was marked as "out". > > fstrim-all does not execute fstrim with a ionice priority. From what I > understand, but haven't tested yet, is that running fstrim with ionice > -c Idle should solve this. > > It's weird that this issue didn't come up earlier on that cluster, but > after killing fstrim all problems we resolved and the cluster ran > happily again. > > So watch out for fstrim on early Sunday mornings on Ubuntu! > > -- > Wido den Hollander > 42on B.V. > Ceph trainer and consultant
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
