Hi Asley,
We experience (using Hammer) a similar issue. Not that I have a perfect
solution to share, but I felt like mentioning a "me too". ;-)
On a side note: we configured correct weight per drive as well.
Regards,
Kees
On 29-12-16 11:54, Ashley Merrick wrote:
>
> Hello,
>
>
>
> I currently have 5 servers within my CEPH Cluster
>
>
>
> 2 x (10 * 8TB Disks)
>
> 3 x (10 * 4TB Disks)
>
>
>
> Currently seeing a larger difference in OSD use across the two
> separate server types, as well as within the server itself.
>
>
>
> For example on one 4TB server I have an OSD at 64% and one at 84%,
> where on the 8TB servers the OSD range from 49% to 64%, where the
> highest used OSD’s are on the 4TB.
>
>
>
> Each drive has a weight set correctly for the drive size and each
> server has the correct weight set, below is my crush map. Apart from
> running the command to adjust the re-weight is there anything I am
> doing wrong or should change for better spread of data, not looking
> for near perfect but where the 8TB drives are sitting at 64% max and
> 4TB are sitting at 80%’s causes a big inbalance.
>
>
>
> # begin crush map
>
> tunable choose_local_tries 0
>
> tunable choose_local_fallback_tries 0
>
> tunable choose_total_tries 50
>
> tunable chooseleaf_descend_once 1
>
> tunable chooseleaf_vary_r 1
>
> tunable straw_calc_version 1
>
> tunable allowed_bucket_algs 54
>
>
>
> # buckets
>
> host sn1 {
>
> id -2 # do not change unnecessarily
>
> # weight 72.800
>
> alg straw2
>
> hash 0 # rjenkins1
>
> item osd.0 weight 7.280
>
> item osd.1 weight 7.280
>
> item osd.3 weight 7.280
>
> item osd.4 weight 7.280
>
> item osd.2 weight 7.280
>
> item osd.5 weight 7.280
>
> item osd.6 weight 7.280
>
> item osd.7 weight 7.280
>
> item osd.8 weight 7.280
>
> item osd.9 weight 7.280
>
> }
>
> host sn3 {
>
> id -6 # do not change unnecessarily
>
> # weight 72.800
>
> alg straw2
>
> hash 0 # rjenkins1
>
> item osd.10 weight 7.280
>
> item osd.11 weight 7.280
>
> item osd.12 weight 7.280
>
> item osd.13 weight 7.280
>
> item osd.14 weight 7.280
>
> item osd.15 weight 7.280
>
> item osd.16 weight 7.280
>
> item osd.17 weight 7.280
>
> item osd.18 weight 7.280
>
> item osd.19 weight 7.280
>
> }
>
> host sn4 {
>
> id -7 # do not change unnecessarily
>
> # weight 36.060
>
> alg straw2
>
> hash 0 # rjenkins1
>
> item osd.20 weight 3.640
>
> item osd.21 weight 3.640
>
> item osd.22 weight 3.640
>
> item osd.23 weight 3.640
>
> item osd.24 weight 3.640
>
> item osd.25 weight 3.640
>
> item osd.26 weight 3.640
>
> item osd.27 weight 3.640
>
> item osd.28 weight 3.640
>
> item osd.29 weight 3.300
>
> }
>
> host sn5 {
>
> id -8 # do not change unnecessarily
>
> # weight 36.060
>
> alg straw2
>
> hash 0 # rjenkins1
>
> item osd.30 weight 3.640
>
> item osd.31 weight 3.640
>
> item osd.32 weight 3.640
>
> item osd.33 weight 3.640
>
> item osd.34 weight 3.640
>
> item osd.35 weight 3.640
>
> item osd.36 weight 3.640
>
> item osd.37 weight 3.640
>
> item osd.38 weight 3.640
>
> item osd.39 weight 3.640
>
> }
>
> host sn6 {
>
> id -9 # do not change unnecessarily
>
> # weight 36.060
>
> alg straw2
>
> hash 0 # rjenkins1
>
> item osd.40 weight 3.640
>
> item osd.41 weight 3.640
>
> item osd.42 weight 3.640
>
> item osd.43 weight 3.640
>
> item osd.44 weight 3.640
>
> item osd.45 weight 3.640
>
> item osd.46 weight 3.640
>
> item osd.47 weight 3.640
>
> item osd.48 weight 3.640
>
> item osd.49 weight 3.640
>
> }
>
> root default {
>
> id -1 # do not change unnecessarily
>
> # weight 253.780
>
> alg straw2
>
> hash 0 # rjenkins1
>
> item sn1 weight 72.800
>
> item sn3 weight 72.800
>
> item sn4 weight 36.060
>
> item sn5 weight 36.060
>
> item sn6 weight 36.060
>
> }
>
>
>
> # rules
>
> rule replicated_ruleset {
>
> ruleset 0
>
> type replicated
>
> min_size 1
>
> max_size 10
>
> step take default
>
> step chooseleaf firstn 0 type host
>
> step emit
>
> }
>
>
>
> Thanks,
>
> Ashley
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
