Your issue is different since not only do the omap digests of all replicas not match the omap digest from the auth object info but they are all different to each other.
What is min_size of pool 67 and what can you tell us about the events leading up to this? On Mon, Jul 16, 2018 at 7:06 PM, Matthew Vernon <m...@sanger.ac.uk> wrote: > Hi, > > Our cluster is running 10.2.9 (from Ubuntu; on 16.04 LTS), and we have a > pg that's stuck inconsistent; if I repair it, it logs "failed to pick > suitable auth object" (repair log attached, to try and stop my MUA > mangling it). > > We then deep-scrubbed that pg, at which point > rados list-inconsistent-obj 67.2e --format=json-pretty produces a bit of > output (also attached), which includes that all 3 osds have a zero-sized > object e.g. > > "osd": 1937, > "errors": [ > "omap_digest_mismatch_oi" > ], > "size": 0, > "omap_digest": "0x45773901", > "data_digest": "0xffffffff" > > All 3 osds have different omap_digest, but all have 0 size. Indeed, > looking on the OSD disks directly, each object is 0 size (i.e. they are > identical). > > This looks similar to one of the failure modes in > http://tracker.ceph.com/issues/21388 where the is a suggestion (comment > 19 from David Zafman) to do: > > rados -p default.rgw.buckets.index setomapval > .dir.861ae926-7ff0-48c5-86d6-a6ba8d0a7a14.7130858.6 temporary-key anything > [deep-scrub] > rados -p default.rgw.buckets.index rmomapkey > .dir.861ae926-7ff0-48c5-86d6-a6ba8d0a7a14.7130858.6 temporary-key > > Is this likely to be the correct approach here, to? And is there an > underlying bug in ceph that still needs fixing? :) > > Thanks, > > Matthew > > > > -- > The Wellcome Sanger Institute is operated by Genome Research > Limited, a charity registered in England with number 1021457 and a > company registered in England with number 2742969, whose registered > office is 215 Euston Road, London, NW1 2BE. > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
