I'm using filestore. I think the root cause is something getting stuck in the code. As such I went ahead and created a [1] bug tracker for this. Hopefully it gets some traction as I'm not particularly looking forward to messing with deleting PGs with the ceph-objectstore-tool in production.
[1] http://tracker.ceph.com/issues/23577 On Fri, Apr 6, 2018 at 11:40 AM Michael Sudnick <michael.sudn...@gmail.com> wrote: > I've tried a few more things to get a deep-scrub going on my PG. I tried > instructing the involved osds to scrub all their PGs and it looks like that > didn't do it. > > Do you have any documentation on the object-store-tool? What I've found > online talks about filestore and not bluestore. > > On 6 April 2018 at 09:27, David Turner <drakonst...@gmail.com> wrote: > >> I'm running into this exact same situation. I'm running 12.2.2 and I >> have an EC PG with a scrub error. It has the same output for [1] rados >> list-inconsistent-obj as mentioned before. This is the [2] full health >> detail. This is the [3] excerpt from the log from the deep-scrub that >> marked the PG inconsistent. The scrub happened when the PG was starting up >> after using ceph-objectstore-tool to split its filestore subfolders. This >> is using a script that I've used for months without any side effects. >> >> I have tried quite a few things to get this PG to deep-scrub or repair, >> but to no avail. It will not do anything. I have set every osd's >> osd_max_scrubs to 0 in the cluster, waited for all scrubbing and deep >> scrubbing to finish, then increased the 11 OSDs for this PG to 1 before >> issuing a deep-scrub. And it will sit there for over an hour without >> deep-scrubbing. My current testing of this is to set all osds to 1, >> increase all of the osds for this PG to 4, and then issue the repair... but >> similarly nothing happens. Each time I issue the deep-scrub or repair, the >> output correctly says 'instructing pg 145.2e3 on osd.234 to repair', but >> nothing shows up in the log for the OSD and the PG state stays >> 'active+clean+inconsistent'. >> >> My next step, unless anyone has a better idea, is to find the exact copy >> of the PG with the missing object, use object-store-tool to back up that >> copy of the PG and remove it. Then starting the OSD back up should >> backfill the full copy of the PG and be healthy again. >> >> >> >> [1] $ rados list-inconsistent-obj 145.2e3 >> No scrub information available for pg 145.2e3 >> error 2: (2) No such file or directory >> >> [2] $ ceph health detail >> HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent >> OSD_SCRUB_ERRORS 1 scrub errors >> PG_DAMAGED Possible data damage: 1 pg inconsistent >> pg 145.2e3 is active+clean+inconsistent, acting >> [234,132,33,331,278,217,55,358,79,3,24] >> >> [3] 2018-04-04 15:24:53.603380 7f54d1820700 0 log_channel(cluster) log >> [DBG] : 145.2e3 deep-scrub starts >> 2018-04-04 17:32:37.916853 7f54d1820700 -1 log_channel(cluster) log [ERR] >> : 145.2e3s0 deep-scrub 1 missing, 0 inconsistent objects >> 2018-04-04 17:32:37.916865 7f54d1820700 -1 log_channel(cluster) log [ERR] >> : 145.2e3 deep-scrub 1 errors >> >> On Mon, Apr 2, 2018 at 4:51 PM Michael Sudnick <michael.sudn...@gmail.com> >> wrote: >> >>> Hi Kjetil, >>> >>> I've tried to get the pg scrubbing/deep scrubbing and nothing seems to >>> be happening. I've tried it a few times over the last few days. My cluster >>> is recovering from a failed disk (which was probably the reason for the >>> inconsistency), do I need to wait for the cluster to heal before >>> repair/deep scrub works? >>> >>> -Michael >>> >>> On 2 April 2018 at 14:13, Kjetil Joergensen <kje...@medallia.com> wrote: >>> >>>> Hi, >>>> >>>> scrub or deep-scrub the pg, that should in theory get you back to >>>> list-inconsistent-obj spitting out what's wrong, then mail that info to the >>>> list. >>>> >>>> -KJ >>>> >>>> On Sun, Apr 1, 2018 at 9:17 AM, Michael Sudnick < >>>> michael.sudn...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> I have a small cluster with an inconsistent pg. I've tried ceph pg >>>>> repair multiple times to no luck. rados list-inconsistent-obj 49.11c >>>>> returns: >>>>> >>>>> # rados list-inconsistent-obj 49.11c >>>>> No scrub information available for pg 49.11c >>>>> error 2: (2) No such file or directory >>>>> >>>>> I'm a bit at a loss here as what to do to recover. That pg is part of >>>>> a cephfs_data pool with compression set to force/snappy. >>>>> >>>>> Does anyone have an suggestions? >>>>> >>>>> -Michael >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>> >>>> >>>> -- >>>> Kjetil Joergensen <kje...@medallia.com> >>>> SRE, Medallia Inc >>>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
