Well, after 4 days, this is probably moot. Hopefully it's finished backfilling, and your problem is gone.
If not, I believe that if you fix those backfill_toofull, the negative numbers will start approaching zero. I seem to recall that negative degraded is a special case of degraded, but I don't remember exactly, and can't find any references. I have seen it before, and it went away when my cluster became healthy. As long as you still have OSDs completing their backfilling, I'd let it run. If you get to the point that all of the backfills are done, and you're left with only wait_backfill+backfill_toofull, then you can bump osd_backfill_full_ratio, mon_osd_nearfull_ratio, and maybe osd_failsafe_nearfull_ratio. If you do, be careful, and only bump them just enough to let them start backfilling. If you set them to 0.99, bad things will happen. On Thu, Nov 13, 2014 at 7:57 AM, Fred Yang <frederic.y...@gmail.com> wrote: > Hi, > > The Ceph cluster we are running have few OSDs approaching to 95% 1+ weeks > ago so I ran a reweight to balance it out, in the meantime, instructing > application to purge data not required. But after large amount of data > purge issued from application side(all OSDs' usage dropped below 20%), the > cluster fall into this weird state for days, the "objects degraded" remain > negative for more than 7 days, I'm seeing some IOs going on on OSDs > consistently, but the number(negative) objects degraded does not change > much: > > 2014-11-13 10:43:07.237292 mon.0 [INF] pgmap v5935301: 44816 pgs: 44713 > active+clean, 1 active+backfilling, 20 active+remapped+wait_backfill, 27 > active+remapped+wait_backfill+backfill_toofull, 11 active+recovery_wait, 33 > active+remapped+backfilling, 11 active+wait_backfill+backfill_toofull; 1473 > GB data, 2985 GB used, 17123 GB / 20109 GB avail; 30172 kB/s wr, 58 op/s; > -13582/1468299 objects degraded (-0.925%) > 2014-11-13 10:43:08.248232 mon.0 [INF] pgmap v5935302: 44816 pgs: 44713 > active+clean, 1 active+backfilling, 20 active+remapped+wait_backfill, 27 > active+remapped+wait_backfill+backfill_toofull, 11 active+recovery_wait, 33 > active+remapped+backfilling, 11 active+wait_backfill+backfill_toofull; 1473 > GB data, 2985 GB used, 17123 GB / 20109 GB avail; 26459 kB/s wr, 51 op/s; > -13582/1468303 objects degraded (-0.925%) > > Any idea what might be happening here? It > seems active+remapped+wait_backfill+backfill_toofull stuck? > > osdmap e43029: 36 osds: 36 up, 36 in > pgmap v5935658: 44816 pgs, 32 pools, 1488 GB data, 714 kobjects > 3017 GB used, 17092 GB / 20109 GB avail > -13438/1475773 objects degraded (-0.911%) > 44713 active+clean > 1 active+backfilling > 20 active+remapped+wait_backfill > 27 active+remapped+wait_backfill+backfill_toofull > 11 active+recovery_wait > 33 active+remapped+backfilling > 11 active+wait_backfill+backfill_toofull > client io 478 B/s rd, 40170 kB/s wr, 80 op/s > > The cluster is running on v0.72.2, we are planning to upgrade cluster to > firefly, but I would like to get the cluster state clean first before the > upgrade. > > Thanks, > Fred > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
