Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd
Hello Paul, Could you post your CRUSH map, crushtool -d caleb On Mon, Aug 6, 2012 at 1:01 PM, Yehuda Sadeh wrote: > > -- Forwarded message -- > From: Paul Pettigrew > Date: Sun, Aug 5, 2012 at 8:08 PM > Subject: RE: Crush not deliverying data uniformly -> HEALTH_ERR full osd > To: Yehuda Sadeh > Cc: "ceph-devel@vger.kernel.org" > > > Hi Yehuda, we have: > > root@dsanb1-coy:/mnt/ceph# ceph osd dump | grep ^pool > pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num > 1472 pgp_num 1472 last_change 1 owner 0 crash_replay_interval 45 > pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins > pg_num 1472 pgp_num 1472 last_change 1 owner 0 > pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num > 1472 pgp_num 1472 last_change 1 owner 0 > pool 3 'backup' rep size 1 crush_ruleset 3 object_hash rjenkins pg_num > 1472 pgp_num 1472 last_change 1 owner 0 > > > -Original Message- > From: ceph-devel-ow...@vger.kernel.org > [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Yehuda Sadeh > Sent: Monday, 6 August 2012 11:16 AM > To: Paul Pettigrew > Cc: ceph-devel@vger.kernel.org > Subject: Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd > > On Sun, Aug 5, 2012 at 5:16 PM, Paul Pettigrew > wrote: > > > > Hi Ceph community > > > > We are at the stage of performance capacity testing, where significant > > amounts of backup data is being written to Ceph. > > > > The issue we have, is that the underlying HDD's are not being > > populated > > (roughly) uniformly, and our Ceph system hits a brick wall after a > > couple of days our 30TB storage system is no longer able to operate > > after having only stored ~7TB. > > > > Basically, despite HDD's (1:1 ratio between OSD and HDD) all being the > > same storage size and weighting in the Crushmap, we have disks either: > > a) using 1% space; > > b) using 48%; or > > c) using 96% > > Too precise a split to be an accident. See below for more detail > > (osd11-22 not expected to get data, per our crushmap): > > > > > > ceph pg dump > > > > pool 0 24420 0 0 1024000 7302520 7302520 > > pool 1 57 0 0 0 127824767 5603518 5603518 > > pool 2 0 0 0 0 0 0 0 > > pool 3 1808757 0 0 0 7584377697985 1104048 1104048 > > sum1811256 0 0 0 7594745522752 14010086 > > 14010086 > > osdstat kbused kbavail kb hb in hb out > > 0 930606904 1021178408 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 1 1874428 1949525164 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 2 928811428 1022963676 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 3 929733676 1022051996 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 4 1719124 1949678844 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 5 1853452 1949545892 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 6 930979476 1020807132 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 7 1808968 1949590496 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 8 934035924 1017759100 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 9 1855955384 949274321953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 10 933572004 1018232340 1953514584 > > [11,12,13,14,15,16,17,18,19,20,21,22] [] > > 11 2057096 953060760 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 12 2053512 953064656 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 13 2148732 972501316 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 14 2064640 972585104 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21,22] [] > > 15 1945388 972703468 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,17,18,19,20,21] [] > > 16 2051708 972599412 976762584 > > [0,1,2,3,4,6,7,8,9,10,17,18,19,20,21] [] > > 17 2137632 952980216 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 18 2000124 953117508 957230808 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 19 2095124 972554492 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 20 1986800 972662640 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 21 2035204 972615332 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > 22 1961412 972687788 976762584 > > [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] [] > > sum7475488140 25609393172 33131684328 > > > > 2012-08-06 10:03:58.964716 7f06783bb700 0 -- 10.32.0.10:0/15147 > > send_keepalive con 0x223f690, no pipe. > > > > > > root@dsanb1-coy:~#
Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd
Hi Paul,
What version of Ceph are you running, perhaps your issue could be
related to an issue with the choose_local_tries parameter used in
earlier versions of the CRUSH mapper code..
caleb
On Mon, Aug 6, 2012 at 3:40 PM, Paul Pettigrew
wrote:
> Hi Caleb
> Crushmap below, thanks!
> Paul
>
>
>
> root@dsanb1-coy:~# cat crushfile.txt
> # begin crush map
>
> # devices
> device 0 osd.0
> device 1 osd.1
> device 2 osd.2
> device 3 osd.3
> device 4 osd.4
> device 5 osd.5
> device 6 osd.6
> device 7 osd.7
> device 8 osd.8
> device 9 osd.9
> device 10 osd.10
> device 11 osd.11
> device 12 osd.12
> device 13 osd.13
> device 14 osd.14
> device 15 osd.15
> device 16 osd.16
> device 17 osd.17
> device 18 osd.18
> device 19 osd.19
> device 20 osd.20
> device 21 osd.21
> device 22 osd.22
>
> # types
> type 0 osd
> type 1 host
> type 2 rack
> type 3 zone
>
> # buckets
> host dsanb1-coy {
> id -2 # do not change unnecessarily
> # weight 11.000
> alg straw
> hash 0 # rjenkins1
> item osd.0 weight 2.000
> item osd.1 weight 2.000
> item osd.10 weight 2.000
> item osd.2 weight 2.000
> item osd.3 weight 2.000
> item osd.4 weight 2.000
> item osd.5 weight 2.000
> item osd.6 weight 2.000
> item osd.7 weight 2.000
> item osd.8 weight 2.000
> item osd.9 weight 2.000
> }
> host dsanb2-coy {
> id -4 # do not change unnecessarily
> # weight 6.000
> alg straw
> hash 0 # rjenkins1
> item osd.11 weight 1.000
> item osd.12 weight 1.000
> item osd.13 weight 1.000
> item osd.14 weight 1.000
> item osd.15 weight 1.000
> item osd.16 weight 1.000
> }
> host dsanb3-coy {
> id -5 # do not change unnecessarily
> # weight 6.000
> alg straw
> hash 0 # rjenkins1
> item osd.17 weight 1.000
> item osd.18 weight 1.000
> item osd.19 weight 1.000
> item osd.20 weight 1.000
> item osd.21 weight 1.000
> item osd.22 weight 1.000
> }
> rack 2nrack {
> id -3 # do not change unnecessarily
> # weight 23.000
> alg straw
> hash 0 # rjenkins1
> item dsanb1-coy weight 11.000
> item dsanb2-coy weight 6.000
> item dsanb3-coy weight 6.000
> }
> zone default {
> id -1 # do not change unnecessarily
> # weight 23.000
> alg straw
> hash 0 # rjenkins1
> item 2nrack weight 23.000
> }
> rack 1nrack {
> id -6 # do not change unnecessarily
> # weight 11.000
> alg straw
> hash 0 # rjenkins1
> item dsanb1-coy weight 11.000
> }
> zone bak {
> id -7 # do not change unnecessarily
> # weight 23.000
> alg straw
> hash 0 # rjenkins1
> item 1nrack weight 23.000
> }
>
> # rules
> rule data {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule metadata {
> ruleset 1
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule rbd {
> ruleset 2
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule backup {
> ruleset 3
> type replicated
> min_size 1
> max_size 10
> step take bak
> step chooseleaf firstn 0 type host
> step emit
> }
>
> # end crush map
>
>
> -Original Message-
> From: ceph-devel-ow...@vger.kernel.org
> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Caleb Miles
> Sent: Tuesday, 7 August 2012 6:09 AM
> To: ceph-devel@vger.kernel.org
> Subject: Re: Crush not deliverying data uniformly -> HEALTH_ERR full osd
>
> Hello Paul,
>
> Could you post your CRUSH map, crushtool -d
>
> caleb
>
> On Mon, Aug 6, 2012 at 1:01 PM, Yehuda Sadeh wrote:
>>
>> -- Forwarded message --
>> From: Paul Pettigrew
>> Date: Sun, Aug 5, 2012 at 8:08 PM
>> Subject: RE: Crush not deliverying data uniformly -> HEALTH_ERR full
>> osd
>> To: Yehuda Sadeh
>> Cc:
chooseleaf_descend_once
Hello all,
Here's what I've done to try and validate the new
chooseleaf_descend_once tunable first described in commit
f1a53c5e80a48557e63db9c52b83f39391bc69b8 in the wip-crush branch of
ceph.git.
First I set the new tunable to it's legacy value, disabled,
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 0
The map contains one thousand osd devices contained in one hundred hosts
with the following data rule
rule data {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
I then simulate the creation of one million placement groups using the
crushtool
$ crushtool -i hundred.map --test --min-x 0 --max-x 99 --num-rep 3
--output-csv --weight 120 0.0 --weight 121 0.0 --weight 122 0.0 --weight
123 0.0 --weight 124 0.0 --weight 125 0.0 --weight 125 0.0 --weight 150
0.0 --weight 151 0.0 --weight 152 0.0 --weight 153 0.0 --weight 154 0.0
--weight 155 0.0 --weight 156 0.0 --weight 180 0.0 --weight 181 0.0
--weight 182 0.0 --weight 183 0.0 --weight 184 0.0 --weight 185 0.0
--weight 186 0.0
with the majority of devices in three hosts marked out. Then in (I)Python
import scipy.stats as s
import matplotlib.mlab as m
data = m.csv2rec("data-device_utilization.csv")
s.chisquare(data['number_of_objects_stored'],
data['number_of_objects_expected'])
which will output
(122939.76474477499, 0.0)
so that the chi squared value is 122939.795 and the p value is, rounded
to, 0.0 and the observed placement distribution statistically differs
from a uniform distribution. Repeating with the new tunable set to
tunable chooseleaf_descend_once 1
I obtain the following result
(998.97643161876761, 0.32151775131589833)
so that the chi squared value is 998.976 and the p value is 0.32 and the
observed placement distribution is statistically identical to the
uniform distribution at the five and ten percent confidence levels,
higher as well of course. The p value is the probability of obtaining a
chi squared value more extreme than the statistic observed. Basically,
from my rudimentary understanding of probability theory, that if you
obtain a p value p < P then reject the null hypothesis, in our case that
the observed placement distribution is drawn from the uniform
distribution, at the P confidence level.
Caleb
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: chooseleaf_descend_once
Hey Jim, Running the third test with tunable chooseleaf_descend_once 0 with no devices marked out yields the following result (999.827397, 0.48667056652539997) so chi squared value is 999 with a corresponding p value of 0.487 so that the placement distribution seems to be drawn from the uniform distribution as desired. Caleb -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: radosgw segfault in 0.56
Hi all, Created branch wip-3735 to capture Sylvain's patch. On Mon, Jan 7, 2013 at 7:21 AM, Sylvain Munaut wrote: > Hi, > >> As far as I know relying on SCRIPT_URI is rather dangerous since it's not >> always there. >> >> There better should be an if/else-satement surrounding that code having it >> defaulting to something else if SCRIPT_URI isn't available. > > I've opened a bug and proposed a patch setting the default value to "" > > http://tracker.newdream.net/issues/3735 > > Cheers, > > Sylvain > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
