On Thu, Oct 18, 2018 at 2:28 PM Graham Allan <g...@umn.edu> wrote: > Thanks Greg, > > This did get resolved though I'm not 100% certain why! > > For one of the suspect shards which caused crash on backfill, I > attempted to delete the associated via s3, late last week. I then > examined the filestore OSDs and the file shards were still present... > maybe for an hour following (after which I stopped looking). > > I left the cluster set to nobackfill over the weekend, during which time > all osds kept running; then on Monday morning re-enabled backfill. I > expected the osd to crash again. after which I could look into moving or > deleting the implicated backfill shards out of the way. Instead of which > it happily backfilled its way to cleanliness. > > I suppose it's possible the shards got deleted later in some kind of rgw > gc operation, and this could have cleared the problem? Unfortunately I > didn't look for them again before re-enabling backfill. I'm not sure if > that's how s3 object deletion works - does it make any sense? >
Yes, RGW generates garbage collection logs that are followed along and perform actual object deletes separately from when they're marked deleted in the S3 protocol. I don't know the details of the process but it's entirely plausible that it just went through and deleted all the bad objects during that time period. -Greg > > The only other thing I did late last week was notice that one of the > active osds for the pg seemed very slow to respond - the drive was > clearly failing. I was never getting any actual i/o errors at the user > or osd level, though it did trigger a 24-hour deathwatch SMART warning a > bit later. > > I exported the pg shard from the failing osd, and re-imported it to > another otherwise-evacuated osd. This was just for data safety; it seems > really unlikely this could be causing the other osds in the pg to crash... > > Graham > > On 10/15/2018 01:44 PM, Gregory Farnum wrote: > > > > > > On Thu, Oct 11, 2018 at 3:22 PM Graham Allan <g...@umn.edu > > > > As the osd crash implies, setting "nobackfill" appears to let all the > > osds keep running and the pg stays active and can apparently serve > data. > > > > If I track down the object referenced below in the object store, I > can > > download it without error via s3... though as I can't generate a > > matching etag, it may well be corrupt. > > > > Still I do wonder if deleting this object - either via s3, or maybe > > more > > likely directly within filestore, might permit backfill to continue. > > > > > > Yes, that is very likely! (...unless there are a bunch of other objects > > with the same issue.) > > > > I'm not immediately familiar with the crash asserts you're seeing, but > > it certainly looks like somehow the object data didn't quite get stored > > correctly as the metadata understands it. Perhaps a write got > > lost/missed on m+1 of the PG shards, setting the > > osd_find_best_info_ignore_history_les caused it to try and recover from > > what it had rather than following normal recovery procedures, and now > > it's not working. > > -Greg > > > -- > Graham Allan > Minnesota Supercomputing Institute - g...@umn.edu >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
