i think the write data will also write to the osd.4 in this case. bc your osd.4 is not down, so the ceph don't think the pg have some osd down, and it will replicated the data to all osds in actingbackfill set.
Tarek Zegar <tze...@us.ibm.com> 于2019年6月7日周五 下午10:37写道： > Paul / All > > I'm not sure what warning your are referring to, I'm on Nautilus. The > point I'm getting at is if you weight out all OSD on a host with a cluster > of 3 OSD hosts with 3 OSD each, crush rule = host, then write to the > cluster, it *should* imo not just say remapped but undersized / degraded. > > See below, 1 out of the 3 OSD hosts has ALL it's OSD marked out and weight > = 0. When you write (say using FIO), the PGs *only* have 2 OSD in them (UP > set), which is pool min size. I don't understand why it's not saying > undersized/degraded, this seems like a bug. Who cares that the Acting Set > has the 3 original OSD in it, the actual data is only on 2 OSD, which is a > degraded state > > *root@hostadmin:~# ceph -s* > cluster: > id: 33d41932-9df2-40ba-8e16-8dedaa4b3ef6 > health: HEALTH_WARN > application not enabled on 1 pool(s) > > services: > mon: 1 daemons, quorum hostmonitor1 (age 29m) > mgr: hostmonitor1(active, since 31m) > osd: 9 osds: 9 up, 6 in; 100 remapped pgs > > data: > pools: 1 pools, 100 pgs > objects: 520 objects, 2.0 GiB > usage: 15 GiB used, 75 GiB / 90 GiB avail > pgs: 520/1560 objects misplaced (33.333%) > *100 active+clean+remapped* > > *root@hostadmin:~# ceph osd tree* > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 0.08817 root default > -3 0.02939 host hostosd1 > 0 hdd 0.00980 osd.0 up 1.00000 1.00000 > 3 hdd 0.00980 osd.3 up 1.00000 1.00000 > 6 hdd 0.00980 osd.6 up 1.00000 1.00000 > *-5 0.02939 host hostosd2* > * 1 hdd 0.00980 osd.1 up 0 1.00000* > * 4 hdd 0.00980 osd.4 up 0 1.00000* > * 7 hdd 0.00980 osd.7 up 0 1.00000* > -7 0.02939 host hostosd3 > 2 hdd 0.00980 osd.2 up 1.00000 1.00000 > 5 hdd 0.00980 osd.5 up 1.00000 1.00000 > 8 hdd 0.00980 osd.8 up 1.00000 1.00000 > > > *root@hostadmin:~# ceph osd df* > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS > STATUS > 0 hdd 0.00980 1.00000 10 GiB 1.7 GiB 765 MiB 12 KiB 1024 MiB 8.2 GiB 17.48 > 1.03 34 up > 3 hdd 0.00980 1.00000 10 GiB 1.7 GiB 765 MiB 12 KiB 1024 MiB 8.2 GiB 17.48 > 1.03 36 up > 6 hdd 0.00980 1.00000 10 GiB 1.6 GiB 593 MiB 4 KiB 1024 MiB 8.4 GiB 15.80 > 0.93 30 up > * 1 hdd 0.00980 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 up* > * 4 hdd 0.00980 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 up* > * 7 hdd 0.00980 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 100 up* > 2 hdd 0.00980 1.00000 10 GiB 1.5 GiB 525 MiB 8 KiB 1024 MiB 8.5 GiB 15.13 > 0.89 20 up > 5 hdd 0.00980 1.00000 10 GiB 1.9 GiB 941 MiB 4 KiB 1024 MiB 8.1 GiB 19.20 > 1.13 43 up > 8 hdd 0.00980 1.00000 10 GiB 1.6 GiB 657 MiB 8 KiB 1024 MiB 8.4 GiB 16.42 > 0.97 37 up > TOTAL 90 GiB 15 GiB 6.2 GiB 61 KiB 9.0 GiB 75 GiB 16.92 > MIN/MAX VAR: 0.89/1.13 STDDEV: 1.32 > Tarek Zegar > Senior SDS Engineer > Email *tze...@us.ibm.com* <email%20address> > Mobile *630.974.7172* > > > > > [image: Inactive hide details for Paul Emmerich ---06/07/2019 05:25:23 > AM---remapped no longer triggers a health warning in nautilus. Y]Paul > Emmerich ---06/07/2019 05:25:23 AM---remapped no longer triggers a health > warning in nautilus. Your data is still there, it's just on the > > From: Paul Emmerich <paul.emmer...@croit.io> > To: Tarek Zegar <tze...@us.ibm.com> > Cc: Ceph Users <ceph-users@lists.ceph.com> > Date: 06/07/2019 05:25 AM > Subject: [EXTERNAL] Re: [ceph-users] Reweight OSD to 0, why doesn't > report degraded if UP set under Pool Size > ------------------------------ > > > > remapped no longer triggers a health warning in nautilus. > > Your data is still there, it's just on the wrong OSD if that OSD is still > up and running. > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at *https://croit.io* > <https://croit.io> > > croit GmbH > Freseniusstr. 31h > 81247 München > *www.croit.io* <http://www.croit.io> > Tel: +49 89 1896585 90 > > > On Thu, Jun 6, 2019 at 10:48 PM Tarek Zegar <*tze...@us.ibm.com* > <tze...@us.ibm.com>> wrote: > > For testing purposes I set a bunch of OSD to 0 weight, this correctly > forces Ceph to not use said OSD. I took enough out such that the UP set > only had Pool min size # of OSD (i.e 2 OSD). > > Two Questions: > 1. Why doesn't the acting set eventually match the UP set and simply > point to [6,5] only > 2. Why are none of the PGs marked as undersized and degraded? The data > is only hosted on 2 OSD rather then Pool size (3), I would expect a > undersized warning and degraded for PG with data? > > Example PG: > PG 1.4d active+clean+remapped UP= [6,5] Acting = [6,5,4] > > OSD Tree: > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 0.08817 root default > -3 0.02939 host hostosd1 > 0 hdd 0.00980 osd.0 up 1.00000 1.00000 > 3 hdd 0.00980 osd.3 up 1.00000 1.00000 > 6 hdd 0.00980 osd.6 up 1.00000 1.00000 > -5 0.02939 host hostosd2 > 1 hdd 0.00980 osd.1 up 0 1.00000 > 4 hdd 0.00980 osd.4 up 0 1.00000 > 7 hdd 0.00980 osd.7 up 0 1.00000 > -7 0.02939 host hostosd3 > 2 hdd 0.00980 osd.2 up 1.00000 1.00000 > 5 hdd 0.00980 osd.5 up 1.00000 1.00000 > 8 hdd 0.00980 osd.8 up 0 1.00000 > > > > > _______________________________________________ > ceph-users mailing list > *ceph-users@lists.ceph.com* <ceph-users@lists.ceph.com> > *http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com* > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Thank you! HuangJun
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
