Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd
Hello Paul, Could you post your CRUSH map, crushtool -d caleb On Mon, Aug 6, 2012 at 1:01 PM, Yehuda Sadeh wrote: > > -- Forwarded message -- > From: Paul Pettigrew > Date: Sun, Aug 5, 2012 at 8:08 PM > Subject: RE: Crush not deliverying data uniformly -> HEALTH_ERR full osd > To: Yehuda Sadeh > Cc: "ceph-devel@vger.kernel.org" > > > Hi Yehuda, we have: > > root@dsanb1-coy:/mnt/ceph# ceph osd dump | grep ^pool > pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num > 1472 pgp_num 1472 last_change 1 owner 0 crash_replay_interval 45 > pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins > pg_num 1472 pgp_num 1472 last_change 1 owner 0 > pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num > 1472 pgp_num 1472 last_change 1 owner 0 > pool 3 'backup' rep size 1 crush_ruleset 3 object_hash rjenkins pg_num > 1472 pgp_num 1472 last_change 1 owner 0 > > > -Original Message- > From: ceph-devel-ow...@vger.kernel.org > [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Yehuda Sadeh > Sent: Monday, 6 August 2012 11:16 AM > To: Paul Pettigrew > Cc: ceph-devel@vger.kernel.org > Subject: Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd > > On Sun, Aug 5, 2012 at 5:16 PM, Paul Pettigrew > wrote: > > > > Hi Ceph community > > > > We are at the stage of performance capacity testing, where significant > > amounts of backup data is being written to Ceph. > > > > The issue we have, is that the underlying HDD's are not being > > populated > > (roughly) uniformly, and our Ceph system hits a brick wall after a > > couple of days our 30TB storage system is no longer able to operate > > after having only stored ~7TB. > > > > Basically, despite HDD's (1:1 ratio between OSD and HDD) all being the > > same storage size and weighting in the Crushmap, we have disks either: > > a) using 1% space; > > b) using 48%; or > > c) using 96% > > Too precise a split to be an accident. See below for more detail > > (osd11-22 not expected to get data, per our crushmap): > > > > > > ceph pg dump > > > > pool 0 24420 0 0 1024000 7302520 7302520 > > pool 1 57 0 0 0 127824767 5603518 5603518 > > pool 2 0 0 0 0 0 0 0 > > pool 3 1808757 0 0 0 7584377697985 1104048 1104048 > > sum1811256 0 0 0 7594745522752 14010086 > > 14010086 > > osdstat kbused kbavail kb hb in hb out > > 0 930606904 1021178408 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 1 1874428 1949525164 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 2 928811428 1022963676 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 3 929733676 1022051996 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 4 1719124 1949678844 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 5 1853452 1949545892 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 6 930979476 1020807132 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 7 1808968 1949590496 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 8 934035924 1017759100 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 9 1855955384 949274321953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 10 933572004 1018232340 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 11 2057096 953060760 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 12 2053512 953064656 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 13 2148732 972501316 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 14 2064640 972585104 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 15 1945388 972703468 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21] [] > > 16 2051708 972599412 976762584 > > [0,1,2,3,4,6,7,8,9,10,17,18,19,20,21] [] > > 17 2137632 952980216 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 18 2000124 953117508 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 19 2095124 972554492 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 20 1986800 972662640 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 21 2035204 972615332 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 22 1961412 972687788 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > sum7475488140 25609393172 33131684328 > > > > 2012-08-06 10:03:58.964716 7f06783bb700 0 -- 10.32.0.10:0/15147 > > send_keepalive con 0x223f690, no pipe. > > > > > > root@dsanb1-coy:~#
Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd
Hi Paul, What version of Ceph are you running, perhaps your issue could be related to an issue with the choose_local_tries parameter used in earlier versions of the CRUSH mapper code.. caleb On Mon, Aug 6, 2012 at 3:40 PM, Paul Pettigrew wrote: > Hi Caleb > Crushmap below, thanks! > Paul > > > > root@dsanb1-coy:~# cat crushfile.txt > # begin crush map > > # devices > device 0 osd.0 > device 1 osd.1 > device 2 osd.2 > device 3 osd.3 > device 4 osd.4 > device 5 osd.5 > device 6 osd.6 > device 7 osd.7 > device 8 osd.8 > device 9 osd.9 > device 10 osd.10 > device 11 osd.11 > device 12 osd.12 > device 13 osd.13 > device 14 osd.14 > device 15 osd.15 > device 16 osd.16 > device 17 osd.17 > device 18 osd.18 > device 19 osd.19 > device 20 osd.20 > device 21 osd.21 > device 22 osd.22 > > # types > type 0 osd > type 1 host > type 2 rack > type 3 zone > > # buckets > host dsanb1-coy { > id -2 # do not change unnecessarily > # weight 11.000 > alg straw > hash 0 # rjenkins1 > item osd.0 weight 2.000 > item osd.1 weight 2.000 > item osd.10 weight 2.000 > item osd.2 weight 2.000 > item osd.3 weight 2.000 > item osd.4 weight 2.000 > item osd.5 weight 2.000 > item osd.6 weight 2.000 > item osd.7 weight 2.000 > item osd.8 weight 2.000 > item osd.9 weight 2.000 > } > host dsanb2-coy { > id -4 # do not change unnecessarily > # weight 6.000 > alg straw > hash 0 # rjenkins1 > item osd.11 weight 1.000 > item osd.12 weight 1.000 > item osd.13 weight 1.000 > item osd.14 weight 1.000 > item osd.15 weight 1.000 > item osd.16 weight 1.000 > } > host dsanb3-coy { > id -5 # do not change unnecessarily > # weight 6.000 > alg straw > hash 0 # rjenkins1 > item osd.17 weight 1.000 > item osd.18 weight 1.000 > item osd.19 weight 1.000 > item osd.20 weight 1.000 > item osd.21 weight 1.000 > item osd.22 weight 1.000 > } > rack 2nrack { > id -3 # do not change unnecessarily > # weight 23.000 > alg straw > hash 0 # rjenkins1 > item dsanb1-coy weight 11.000 > item dsanb2-coy weight 6.000 > item dsanb3-coy weight 6.000 > } > zone default { > id -1 # do not change unnecessarily > # weight 23.000 > alg straw > hash 0 # rjenkins1 > item 2nrack weight 23.000 > } > rack 1nrack { > id -6 # do not change unnecessarily > # weight 11.000 > alg straw > hash 0 # rjenkins1 > item dsanb1-coy weight 11.000 > } > zone bak { > id -7 # do not change unnecessarily > # weight 23.000 > alg straw > hash 0 # rjenkins1 > item 1nrack weight 23.000 > } > > # rules > rule data { > ruleset 0 > type replicated > min_size 1 > max_size 10 > step take default > step chooseleaf firstn 0 type host > step emit > } > rule metadata { > ruleset 1 > type replicated > min_size 1 > max_size 10 > step take default > step chooseleaf firstn 0 type host > step emit > } > rule rbd { > ruleset 2 > type replicated > min_size 1 > max_size 10 > step take default > step chooseleaf firstn 0 type host > step emit > } > rule backup { > ruleset 3 > type replicated > min_size 1 > max_size 10 > step take bak > step chooseleaf firstn 0 type host > step emit > } > > # end crush map > > > -Original Message- > From: ceph-devel-ow...@vger.kernel.org > [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Caleb Miles > Sent: Tuesday, 7 August 2012 6:09 AM > To: ceph-devel@vger.kernel.org > Subject: Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd > > Hello Paul, > > Could you post your CRUSH map, crushtool -d > > caleb > > On Mon, Aug 6, 2012 at 1:01 PM, Yehuda Sadeh wrote: >> >> -- Forwarded message -- >> From: Paul Pettigrew >> Date: Sun, Aug 5, 2012 at 8:08 PM >> Subject: RE: Crush not deliverying data uniformly -> HEALTH_ERR full >> osd >> To: Yehuda Sadeh >> Cc:
chooseleaf_descend_once
Hello all, Here's what I've done to try and validate the new chooseleaf_descend_once tunable first described in commit f1a53c5e80a48557e63db9c52b83f39391bc69b8 in the wip-crush branch of ceph.git. First I set the new tunable to it's legacy value, disabled, tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 0 The map contains one thousand osd devices contained in one hundred hosts with the following data rule rule data { ruleset 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } I then simulate the creation of one million placement groups using the crushtool $ crushtool -i hundred.map --test --min-x 0 --max-x 99 --num-rep 3 --output-csv --weight 120 0.0 --weight 121 0.0 --weight 122 0.0 --weight 123 0.0 --weight 124 0.0 --weight 125 0.0 --weight 125 0.0 --weight 150 0.0 --weight 151 0.0 --weight 152 0.0 --weight 153 0.0 --weight 154 0.0 --weight 155 0.0 --weight 156 0.0 --weight 180 0.0 --weight 181 0.0 --weight 182 0.0 --weight 183 0.0 --weight 184 0.0 --weight 185 0.0 --weight 186 0.0 with the majority of devices in three hosts marked out. Then in (I)Python import scipy.stats as s import matplotlib.mlab as m data = m.csv2rec("data-device_utilization.csv") s.chisquare(data['number_of_objects_stored'], data['number_of_objects_expected']) which will output (122939.76474477499, 0.0) so that the chi squared value is 122939.795 and the p value is, rounded to, 0.0 and the observed placement distribution statistically differs from a uniform distribution. Repeating with the new tunable set to tunable chooseleaf_descend_once 1 I obtain the following result (998.97643161876761, 0.32151775131589833) so that the chi squared value is 998.976 and the p value is 0.32 and the observed placement distribution is statistically identical to the uniform distribution at the five and ten percent confidence levels, higher as well of course. The p value is the probability of obtaining a chi squared value more extreme than the statistic observed. Basically, from my rudimentary understanding of probability theory, that if you obtain a p value p < P then reject the null hypothesis, in our case that the observed placement distribution is drawn from the uniform distribution, at the P confidence level. Caleb -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: chooseleaf_descend_once
Hey Jim, Running the third test with tunable chooseleaf_descend_once 0 with no devices marked out yields the following result (999.827397, 0.48667056652539997) so chi squared value is 999 with a corresponding p value of 0.487 so that the placement distribution seems to be drawn from the uniform distribution as desired. Caleb -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: radosgw segfault in 0.56
Hi all, Created branch wip-3735 to capture Sylvain's patch. On Mon, Jan 7, 2013 at 7:21 AM, Sylvain Munaut wrote: > Hi, > >> As far as I know relying on SCRIPT_URI is rather dangerous since it's not >> always there. >> >> There better should be an if/else-satement surrounding that code having it >> defaulting to something else if SCRIPT_URI isn't available. > > I've opened a bug and proposed a patch setting the default value to "" > > http://tracker.newdream.net/issues/3735 > > Cheers, > > Sylvain > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html