As far as I know, bluestore doesn't like super small sizes. Normally odd should stop doing funny things as full mark, but if device is too small it may be too late and bluefs run out of space.
Two things: 1. Don't use too small osd 2. Have a spare area on the drive. I usually reserve 1% for emergency extension (and to give ssd firmware a bit if space to breath). On Wed, Sep 23, 2020, 01:03 Ivan Kurnosov <zer...@zerkms.com> wrote: > Hi, > > this morning I woke up to a degraded test ceph cluster (managed by rook, > but it does not really change anything for the question I'm about to ask). > > After checking logs I have found that bluestore on one of the OSDs run out > of space. > > Some cluster details: > > ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus > (stable) > it runs on 3 little OSDs 10Gb each > > `ceph osd df` returned RAW USE of about 4.5GB on every node, happily > reporting about 5.5GB of AVAIL. > > Yet: > > ... > So, my question would be: how could I have prevented that? From monitoring > I have (prometheus) - OSDs are healthy, have plenty of space, yet they are > not. > > What command (and prometheus metric) would help me understand the actual > real bluestore use? Or am I missing something? > > Oh, and I "fixed" the cluster by expanding the broken osd.0 with a larger > 15GB volume. And 2 other OSDs still run on 10GB volumes. > > Thanks in advance for any thoughts. > > > -- > With best regards, Ivan Kurnosov > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
