Re: [ceph-users] Degraded data redundancy: NUM pgs undersized
On 09/04/2018 09:47 AM, Jörg Kastning wrote: > My questions are: > > 1. What does active+undersized actually mean? I did not find anything > about it in the documentation on docs.ceph.com. http://docs.ceph.com/docs/master/rados/operations/pg-states/ active Ceph will process requests to the placement group. undersized The placement group has fewer copies than the configured pool replication level. Your crush map/rules and osds do not allow to have all pgs on three "independent" osds, so pgs have fewer copies than configured. > 2. Why are only 15 PGs were getting remapped after I've corrected the > mistake with the wrong pgp_num value? By pure chance 15 pgs are now actually replicated to all 3 osds, so they have enough copies (clean). But the placement is "wrong", it would like to move the data to different osds (remapped) if possible. It replicated to 2 osds in the initial placement but wasn't able to find a suitable third osd. Then by increasing pgp_num it recalculated the placement, again selected two osds and moved the data there. It won't remove the data from the "wrong" osd until it has a new place for it, so you end up with three copies, but remapped pgs. > 3. What's wrong here and what do I have to do to get the cluster back > to active+clean, again? I guess you want to have "two copies in dc1, one copy in dc2"? If you stay with only 3 osds that is the only way to distribute 3 objects anyway, so you don't need any crush rule. What your crush rule is currently expressing is "in the default root, select n buckets (where n is the pool size, 3 in this case) of type datacenter, select one leaf (meaning osd) in each datacenter". You only have 2 datacenter buckets, so that will only ever select 2 osds. If your cluster is going to grow to at least 2 osds in each dc, you can go with http://cephnotes.ksperis.com/blog/2017/01/23/crushmap-for-2-dc/ I would translate this crush rule as "in the default root, select 2 buckets of type datacenter, select n-1 (where n is the pool size, so here 3-1 = 2) leafs in each datacenter" You will need at least two osds in each dc for this, because it is random (with respect to the weights) in which dc the 2 copies will be placed and which gets the remaining copy. Best regards, Lothar -- Lothar Gesslein Linux Consultant Mail: gessl...@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Unexpected behaviour after monitors upgrade from Jewel to Luminous
adjusting msgr requires >> 2018-08-21 21:05:05.905473 7f10de377700 1 >> mon.clb-cph-frpar2-mon-01@0(leader).osd >> e114694 create_pending setting backfillfull_ratio = 0.9 >> 2018-08-21 21:05:05.940729 7f10de377700 0 log_channel(cluster) log [DBG] >> : osdmap e114694: 168 total, 168 up, 168 in >> 2018-08-21 21:05:05.947921 7f10de377700 0 log_channel(cluster) log [INF] >> : pgmap 6208 pgs: 204 peering, 6004 active+clean; 261 TB data, 385 TB used, >> 225 TB / 611 TB avail; 201 MB/s rd, 200 MB/s wr, 355 >> op/s >> 2018-08-21 21:05:06.912752 7f10de377700 1 >> mon.clb-cph-frpar2-mon-01@0(leader).osd >> e114695 e114695: 168 total, 168 up, 168 in >> 2018-08-21 21:05:06.913104 7f10de377700 0 >> mon.clb-cph-frpar2-mon-01@0(leader).osd >> e114695 crush map has features 432365734501548032, adjusting msgr requires >> 2018-08-21 21:05:06.928521 7f10de377700 1 >> mon.clb-cph-frpar2-mon-01@0(leader).osd >> e114695 create_pending setting backfillfull_ratio = 0.9 >> 2018-08-21 21:05:06.968793 7f10de377700 0 log_channel(cluster) log [DBG] >> : osdmap e114695: 168 total, 168 up, 168 in >> 2018-08-21 21:05:07.001112 7f10d93ce700 1 heartbeat_map reset_timeout >> 'Monitor::cpu_tp thread 0x7f10d93ce700' had timed out after 0 >> 2018-08-21 21:05:07.004274 7f10de377700 0 log_channel(cluster) log [INF] >> : pgmap 6208 pgs: 12 unknown, 114 remapped+peering, 5 activating, 481 >> peering, 5596 active+clean; 261 TB data, 385 TB used, 225 T >> B / 611 TB avail; 177 MB/s rd, 173 MB/s wr, 317 op/s >> 2018-08-21 21:05:07.025200 7f10d8bcd700 1 heartbeat_map reset_timeout >> 'Monitor::cpu_tp thread 0x7f10d8bcd700' had timed out after 0 >> 2018-08-21 21:05:07.948138 7f10de377700 1 >> mon.clb-cph-frpar2-mon-01@0(leader).osd >> e114696 e114696: 168 total, 168 up, 168 in >> 2018-08-21 21:05:07.948487 7f10de377700 0 >> mon.clb-cph-frpar2-mon-01@0(leader).osd >> e114696 crush map has features 432365734501548032, adjusting msgr requires >> 2018-08-21 21:05:07.965671 7f10de377700 1 >> mon.clb-cph-frpar2-mon-01@0(leader).osd >> e114696 create_pending setting backfillfull_ratio = 0.9 >> 2018-08-21 21:05:08.007585 7f10de377700 0 log_channel(cluster) log [DBG] >> : osdmap e114696: 168 total, 168 up, 168 in >> 2018-08-21 21:05:08.052153 7f10de377700 0 log_channel(cluster) log [INF] >> : pgmap 6208 pgs: 46 activating+remapped, 15 unknown, 305 remapped+peering, >> 11 activating, 697 peering, 8 active+remapped+backfil >> ling, 5126 active+clean; 261 TB data, 385 TB used, 225 TB / 611 TB avail; >> 115 MB/s rd, 182 MB/s wr, 292 op/s; 26681/519327262 objects misplaced >> (0.005%); 4856 kB/s, 3 objects/s recovering >> >> > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Lothar Gesslein Linux Consultant Mail: gessl...@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Error creating compat weight-set with mgr balancer plugin
On 07/24/2018 12:58 PM, Martin Overgaard Hansen wrote: > Creating a compat weight set manually with 'ceph osd crush weight-set > create-compat' gives me: Error EPERM: crush map contains one or more > bucket(s) that are not straw2 > > What changes do I need to implement to get the mgr balancer plugin > working? Thank. You will need to run osd crush set-all-straw-buckets-to-straw2 which exists since ceph mimic v13.0.1 as a handy shortcut to upgrade to straw2. The switch of the straw alorithm to the improved straw2 was introduced with hammer, but before this command you would have had to edit the crush map by hand. http://docs.ceph.com/docs/master/rados/operations/crush-map/ There is a new bucket type (straw2) supported. The new straw2 bucket type fixes several limitations in the original straw bucket. Specifically, the old straw buckets would change some mappings that should have changed when a weight was adjusted, while straw2 achieves the original goal of only changing mappings to or from the bucket item whose weight has changed. straw2 is the default for any newly created buckets. Migration impact: Changing a bucket type from straw to straw2 will result in a reasonably small amount of data movement, depending on how much the bucket item weights vary from each other. When the weights are all the same no data will move, and when item weights vary significantly there will be more movement. Best, Lothar -- Lothar Gesslein Linux Consultant B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
