Re: [ceph-users] Altering crush-failure-domain
Thanks guys. Regards, Kees On 04-03-19 22:18, Smith, Eric wrote: > This will cause data migration. > > -Original Message- > From: ceph-users On Behalf Of Paul > Emmerich > Sent: Monday, March 4, 2019 2:32 PM > To: Kees Meijs > Cc: Ceph Users > Subject: Re: [ceph-users] Altering crush-failure-domain > > Yes, these parts of the profile are just used to create a crush rule. > You can change the crush rule like any other crush rule. > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Mar 4, 2019 at 8:13 PM Kees Meijs wrote: >> Hi Cephers, >> >> Documentation on >> http://docs.ceph.com/docs/master/rados/operations/erasure-code/ states: >> >> Choosing the right profile is important because it cannot be modified after >> the pool is created: a new pool with a different profile needs to be created >> and all objects from the previous pool moved to the new. >> >> >> Right, that makes sense. However, is it possible to "migrate" >> crush-failure-domain from osd to host, to rack and so on without copying >> pools? >> >> Regards, >> Kees >> >> -- >> https://nefos.nl/contact >> >> Nefos IT bv >> Ambachtsweg 25 (industrienummer 4217) >> 5627 BZ Eindhoven >> Nederland >> >> KvK 66494931 >> >> Aanwezig op maandag, dinsdag, woensdag en vrijdag >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw sync falling behind regularly
sure thing.
sv5-ceph-rgw1
zonegroup get
{
"id": "de6af748-1a2f-44a1-9d44-30799cf1313e",
"name": "us",
"api_name": "us",
"is_master": "true",
"endpoints": [
"http://sv5-ceph-rgw1.savagebeast.com:8080";
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04",
"zones": [
{
"id": "107d29a0-b732-4bf1-a26e-1f64f820e839",
"name": "dc11-prod",
"endpoints": [
"http://dc11-ceph-rgw1:8080";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
},
{
"id": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04",
"name": "sv5-corp",
"endpoints": [
"http://sv5-ceph-rgw1.savagebeast.com:8080";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
},
{
"id": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8",
"name": "sv3-prod",
"endpoints": [
"http://sv3-ceph-rgw1:8080";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd"
}
zone get
{
"id": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04",
"name": "sv5-corp",
"domain_root": "sv5-corp.rgw.meta:root",
"control_pool": "sv5-corp.rgw.control",
"gc_pool": "sv5-corp.rgw.log:gc",
"lc_pool": "sv5-corp.rgw.log:lc",
"log_pool": "sv5-corp.rgw.log",
"intent_log_pool": "sv5-corp.rgw.log:intent",
"usage_log_pool": "sv5-corp.rgw.log:usage",
"reshard_pool": "sv5-corp.rgw.log:reshard",
"user_keys_pool": "sv5-corp.rgw.meta:users.keys",
"user_email_pool": "sv5-corp.rgw.meta:users.email",
"user_swift_pool": "sv5-corp.rgw.meta:users.swift",
"user_uid_pool": "sv5-corp.rgw.meta:users.uid",
"system_key": {
"access_key": "access_key_redacted",
"secret_key": "secret_key_redacted"
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "sv5-corp.rgw.buckets.index",
"data_pool": "sv5-corp.rgw.buckets.data",
"data_extra_pool": "sv5-corp.rgw.buckets.non-ec",
"index_type": 0,
"compression": ""
}
}
],
"metadata_heap": "",
"tier_config": [],
"realm_id": "b3e2afe7-2254-494a-9a34-ce50358779fd"
}
sv3-ceph-rgw1
zonegroup get
{
"id": "de6af748-1a2f-44a1-9d44-30799cf1313e",
"name": "us",
"api_name": "us",
"is_master": "true",
"endpoints": [
"http://sv5-ceph-rgw1.savagebeast.com:8080";
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04",
"zones": [
{
"id": "107d29a0-b732-4bf1-a26e-1f64f820e839",
"name": "dc11-prod",
"endpoints": [
"http://dc11-ceph-rgw1:8080";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
},
{
"id": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04",
"name": "sv5-corp",
"endpoints": [
"http://sv5-ceph-rgw1.savagebeast.com:8080";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
},
{
"id": "331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8",
"name": "sv3-prod",
"endpoints": [
"http://sv3-ceph-rgw1:8080";
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"r
Re: [ceph-users] radosgw sync falling behind regularly
Christian,
Can you provide your zonegroup and zones configurations for all 3 rgw sites?
(run the commands for each site please)
Thanks,
From: Christian Rice
Sent: Monday, March 4, 2019 5:34 PM
To: Matthew H; ceph-users
Subject: Re: radosgw sync falling behind regularly
So we upgraded everything from 12.2.8 to 12.2.11, and things have gone to hell.
Lots of sync errors, like so:
sudo radosgw-admin sync error list
[
{
"shard_id": 0,
"entries": [
{
"id": "1_1549348245.870945_5163821.1",
"section": "data",
"name":
"dora/catalogmaker-redis:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.470/56fbc9685d609b4c8cdbd11dd60bf03bedcb613b438c663c9899d930b25f0405",
"timestamp": "2019-02-05 06:30:45.870945Z",
"info": {
"source_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04",
"error_code": 5,
"message": "failed to sync object(5) Input/output error"
}
},
…
radosgw logs are full of:
2019-03-04 14:32:58.039467 7f90e81eb700 0 data sync: ERROR: failed to read
remote data log info: ret=-2
2019-03-04 14:32:58.041296 7f90e81eb700 0 data sync: ERROR: init sync on
escarpment/escarpment:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.146 failed,
retcode=-2
2019-03-04 14:32:58.041662 7f90e81eb700 0 meta sync: ERROR:
RGWBackoffControlCR called coroutine returned -2
2019-03-04 14:32:58.042949 7f90e81eb700 0 data sync: WARNING: skipping data
log entry for missing bucket
escarpment/escarpment:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.146
2019-03-04 14:32:58.823501 7f90e81eb700 0 data sync: ERROR: failed to read
remote data log info: ret=-2
2019-03-04 14:32:58.825243 7f90e81eb700 0 meta sync: ERROR:
RGWBackoffControlCR called coroutine returned -2
dc11-ceph-rgw2:~$ sudo radosgw-admin sync status
realm b3e2afe7-2254-494a-9a34-ce50358779fd (savagebucket)
zonegroup de6af748-1a2f-44a1-9d44-30799cf1313e (us)
zone 107d29a0-b732-4bf1-a26e-1f64f820e839 (dc11-prod)
2019-03-04 14:26:21.351372 7ff7ae042e40 0 meta sync: ERROR: failed to fetch
mdlog info
metadata sync syncing
full sync: 0/64 shards
failed to fetch local sync status: (5) Input/output error
^C
Any advice? All three clusters on 12.2.11, Debian stretch.
From: Christian Rice
Date: Thursday, February 28, 2019 at 9:06 AM
To: Matthew H , ceph-users
Subject: Re: radosgw sync falling behind regularly
Yeah my bad on the typo, not running 12.8.8 ☺ It’s 12.2.8. We can upgrade and
will attempt to do so asap. Thanks for that, I need to read my release notes
more carefully, I guess!
From: Matthew H
Date: Wednesday, February 27, 2019 at 8:33 PM
To: Christian Rice , ceph-users
Subject: Re: radosgw sync falling behind regularly
Hey Christian,
I'm making a while guess, but assuming this is 12.2.8. If so, it it possible
that you can upgrade to 12.2.11? There's been rgw multisite bug fixes for
metadata syncing and data syncing ( both separate issues ) that you could be
hitting.
Thanks,
From: ceph-users on behalf of Christian
Rice
Sent: Wednesday, February 27, 2019 7:05 PM
To: ceph-users
Subject: [ceph-users] radosgw sync falling behind regularly
Debian 9; ceph 12.8.8-bpo90+1; no rbd or cephfs, just radosgw; three clusters
in one zonegroup.
Often we find either metadata or data sync behind, and it doesn’t look to ever
recover until…we restart the endpoint radosgw target service.
eg at 15:45:40:
dc11-ceph-rgw1:/var/log/ceph# radosgw-admin sync status
realm b3e2afe7-2254-494a-9a34-ce50358779fd (savagebucket)
zonegroup de6af748-1a2f-44a1-9d44-30799cf1313e (us)
zone 107d29a0-b732-4bf1-a26e-1f64f820e839 (dc11-prod)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is behind on 2 shards
behind shards: [19,41]
oldest incremental change not applied: 2019-02-27
14:42:24.0.408263s
data sync source: 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 (sv5-corp)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 (sv3-prod)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
so at 15:46:07:
dc11-ceph-rgw1:/var/log/ceph# sudo systemctl restart
ceph-radosgw@rgw.dc11-ceph-rgw1.service
and by the time I checked at 15:48:08:
dc11-ceph-rgw1:/var/log/cep
Re: [ceph-users] How to use STS Lite correctly?
Hello. I successfully created the role and attached the permission policy, but it still didn't work as expected. When I request the root path, it returns an HTTP 400 error: Request: POST / HTTP/1.1 Host: 192.168.199.81:8080 Accept-Encoding: identity Content-Length: 159 Content-Type: application/x-www-form-urlencoded; charset=utf-8 X-Amz-Date: 20190305T024604Z Authorization: AWS4-HMAC-SHA256 Credential=O966WM2NEUB232Z53VYG/20190305//sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=dfb51d46ca561fa7bf763ceaededf58afd17b3fe6293c4cc6dc4fccba24c95d1 User-Agent: Boto3/1.9.106 Python/2.7.15 Windows/7 Botocore/1.12.106 Action=AssumeRole&DurationSeconds=3600&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2Fapplication_abc%2Fcomponent_xyz%2Fcgtw-STS&Version=2011-06-15&RoleSessionName=Bob Response: InvalidArgument txf-005c7de2ea-1217e-default 1217e-default-default When I requested the /rgw path, it returned an HTTP 403 error: Request: POST /rgw HTTP/1.1 Host: 192.168.199.81:8080 Accept-Encoding: identity Content-Length: 159 Content-Type: application/x-www-form-urlencoded; charset=utf-8 X-Amz-Date: 20190305T024904Z Authorization: AWS4-HMAC-SHA256 Credential=O966WM2NEUB232Z53VYG/20190305//sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=d68e6f79ded8d06bef19fa0d9248d5c72bdfd08abbd61b54de887fba17474f6d User-Agent: Boto3/1.9.106 Python/2.7.15 Windows/7 Botocore/1.12.106 Action=AssumeRole&DurationSeconds=3600&RoleArn=arn%3Aaws%3Aiam%3A%3A%3Arole%2Fapplication_abc%2Fcomponent_xyz%2Fcgtw-STS&Version=2011-06-15&RoleSessionName=Bob Response: AccessDenied tx00010-005c7de39f-1217e-default 1217e-default-default Can you tell me if my request path is incorrect? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 14.1.0, No dashboard module
I have just spun up a small test environment to give the first RC a test run. Have managed to get a MON / MGR running fine on latest .dev packages on Ubuntu 18.04, however when I go to try enable the dashboard I get the following error. ceph mgr module enable dashboard Error ENOENT: all mgr daemons do not support module 'dashboard', pass --force to force enablement Trying with --force does nothing, checking the mgr log during boot the dashboard plugin is not listed along all the plugins available. I had a look through the tracker and commits since the RC1, however can't see this already mentioned, not sure if this is expected for RC1 or a bug. Thanks, Ashley ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Intel P4600 3.2TB U.2 form factor NVMe firmware problems causing dead disks
Hi, If QDV10130 pre-dates feb/march 2018, you may have suffered the same firmware bug as existed on the DC S4600 series. I'm under NDA so I can't bitch and moan about specifics, but your symptoms sounds very familiar. It's entirely possible that there's *something* about bluestore that has access patterns that differ from "regular filesystems", we burnt ourselves with the DC S4600, which were burnt in (I were told) - but probably the burn-in testing were done through filesystems rather than ceph/bluestore. Previously discussed around here http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-January/023835.html On Mon, Feb 18, 2019 at 7:44 AM David Turner wrote: > > We have 2 clusters of [1] these disks that have 2 Bluestore OSDs per disk > (partitioned), 3 disks per node, 5 nodes per cluster. The clusters are > 12.2.4 running CephFS and RBDs. So in total we have 15 NVMe's per cluster > and 30 NVMe's in total. They were all built at the same time and were > running firmware version QDV10130. On this firmware version we early on had > 2 disks failures, a few months later we had 1 more, and then a month after > that (just a few weeks ago) we had 7 disk failures in 1 week. > > The failures are such that the disk is no longer visible to the OS. This > holds true beyond server reboots as well as placing the failed disks into a > new server. With a firmware upgrade tool we got an error that pretty much > said there's no way to get data back and to RMA the disk. We upgraded all of > our remaining disks' firmware to QDV101D1 and haven't had any problems since > then. Most of our failures happened while rebalancing the cluster after > replacing dead disks and we tested rigorously around that use case after > upgrading the firmware. This firmware version seems to have resolved > whatever the problem was. > > We have about 100 more of these scattered among database servers and other > servers that have never had this problem while running the QDV10130 firmware > as well as firmwares between this one and the one we upgraded to. Bluestore > on Ceph is the only use case we've had so far with this sort of failure. > > Has anyone else come across this issue before? Our current theory is that > Bluestore is accessing the disk in a way that is triggering a bug in the > older firmware version that isn't triggered by more traditional filesystems. > We have a scheduled call with Intel to discuss this, but their preliminary > searches into the bugfixes and known problems between firmware versions > didn't indicate the bug that we triggered. It would be good to have some > more information about what those differences for disk accessing might be to > hopefully get a better answer from them as to what the problem is. > > > [1] > https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds/dc-p4600-series/dc-p4600-3-2tb-2-5inch-3d1.html > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Kjetil Joergensen SRE, Medallia Inc ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw sync falling behind regularly
So we upgraded everything from 12.2.8 to 12.2.11, and things have gone to hell.
Lots of sync errors, like so:
sudo radosgw-admin sync error list
[
{
"shard_id": 0,
"entries": [
{
"id": "1_1549348245.870945_5163821.1",
"section": "data",
"name":
"dora/catalogmaker-redis:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.470/56fbc9685d609b4c8cdbd11dd60bf03bedcb613b438c663c9899d930b25f0405",
"timestamp": "2019-02-05 06:30:45.870945Z",
"info": {
"source_zone": "1e27bf9c-3a2f-4845-85b6-33a24bbe1c04",
"error_code": 5,
"message": "failed to sync object(5) Input/output error"
}
},
…
radosgw logs are full of:
2019-03-04 14:32:58.039467 7f90e81eb700 0 data sync: ERROR: failed to read
remote data log info: ret=-2
2019-03-04 14:32:58.041296 7f90e81eb700 0 data sync: ERROR: init sync on
escarpment/escarpment:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.146 failed,
retcode=-2
2019-03-04 14:32:58.041662 7f90e81eb700 0 meta sync: ERROR:
RGWBackoffControlCR called coroutine returned -2
2019-03-04 14:32:58.042949 7f90e81eb700 0 data sync: WARNING: skipping data
log entry for missing bucket
escarpment/escarpment:1e27bf9c-3a2f-4845-85b6-33a24bbe1c04.18467.146
2019-03-04 14:32:58.823501 7f90e81eb700 0 data sync: ERROR: failed to read
remote data log info: ret=-2
2019-03-04 14:32:58.825243 7f90e81eb700 0 meta sync: ERROR:
RGWBackoffControlCR called coroutine returned -2
dc11-ceph-rgw2:~$ sudo radosgw-admin sync status
realm b3e2afe7-2254-494a-9a34-ce50358779fd (savagebucket)
zonegroup de6af748-1a2f-44a1-9d44-30799cf1313e (us)
zone 107d29a0-b732-4bf1-a26e-1f64f820e839 (dc11-prod)
2019-03-04 14:26:21.351372 7ff7ae042e40 0 meta sync: ERROR: failed to fetch
mdlog info
metadata sync syncing
full sync: 0/64 shards
failed to fetch local sync status: (5) Input/output error
^C
Any advice? All three clusters on 12.2.11, Debian stretch.
From: Christian Rice
Date: Thursday, February 28, 2019 at 9:06 AM
To: Matthew H , ceph-users
Subject: Re: radosgw sync falling behind regularly
Yeah my bad on the typo, not running 12.8.8 ☺ It’s 12.2.8. We can upgrade and
will attempt to do so asap. Thanks for that, I need to read my release notes
more carefully, I guess!
From: Matthew H
Date: Wednesday, February 27, 2019 at 8:33 PM
To: Christian Rice , ceph-users
Subject: Re: radosgw sync falling behind regularly
Hey Christian,
I'm making a while guess, but assuming this is 12.2.8. If so, it it possible
that you can upgrade to 12.2.11? There's been rgw multisite bug fixes for
metadata syncing and data syncing ( both separate issues ) that you could be
hitting.
Thanks,
From: ceph-users on behalf of Christian
Rice
Sent: Wednesday, February 27, 2019 7:05 PM
To: ceph-users
Subject: [ceph-users] radosgw sync falling behind regularly
Debian 9; ceph 12.8.8-bpo90+1; no rbd or cephfs, just radosgw; three clusters
in one zonegroup.
Often we find either metadata or data sync behind, and it doesn’t look to ever
recover until…we restart the endpoint radosgw target service.
eg at 15:45:40:
dc11-ceph-rgw1:/var/log/ceph# radosgw-admin sync status
realm b3e2afe7-2254-494a-9a34-ce50358779fd (savagebucket)
zonegroup de6af748-1a2f-44a1-9d44-30799cf1313e (us)
zone 107d29a0-b732-4bf1-a26e-1f64f820e839 (dc11-prod)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is behind on 2 shards
behind shards: [19,41]
oldest incremental change not applied: 2019-02-27
14:42:24.0.408263s
data sync source: 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 (sv5-corp)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 (sv3-prod)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
so at 15:46:07:
dc11-ceph-rgw1:/var/log/ceph# sudo systemctl restart
ceph-radosgw@rgw.dc11-ceph-rgw1.service
and by the time I checked at 15:48:08:
dc11-ceph-rgw1:/var/log/ceph# radosgw-admin sync status
realm b3e2afe7-2254-494a-9a34-ce50358779fd (savagebucket)
zonegroup de6af748-1a2f-44a1-9d44-30799cf1313e (us)
zone 107d29a0-b732-4bf1-a26e-1f64f820e839 (dc11-prod)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata i
Re: [ceph-users] Altering crush-failure-domain
This will cause data migration. -Original Message- From: ceph-users On Behalf Of Paul Emmerich Sent: Monday, March 4, 2019 2:32 PM To: Kees Meijs Cc: Ceph Users Subject: Re: [ceph-users] Altering crush-failure-domain Yes, these parts of the profile are just used to create a crush rule. You can change the crush rule like any other crush rule. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Mar 4, 2019 at 8:13 PM Kees Meijs wrote: > > Hi Cephers, > > Documentation on > http://docs.ceph.com/docs/master/rados/operations/erasure-code/ states: > > Choosing the right profile is important because it cannot be modified after > the pool is created: a new pool with a different profile needs to be created > and all objects from the previous pool moved to the new. > > > Right, that makes sense. However, is it possible to "migrate" > crush-failure-domain from osd to host, to rack and so on without copying > pools? > > Regards, > Kees > > -- > https://nefos.nl/contact > > Nefos IT bv > Ambachtsweg 25 (industrienummer 4217) > 5627 BZ Eindhoven > Nederland > > KvK 66494931 > > Aanwezig op maandag, dinsdag, woensdag en vrijdag > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Altering crush-failure-domain
Yes, these parts of the profile are just used to create a crush rule. You can change the crush rule like any other crush rule. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Mar 4, 2019 at 8:13 PM Kees Meijs wrote: > > Hi Cephers, > > Documentation on > http://docs.ceph.com/docs/master/rados/operations/erasure-code/ states: > > Choosing the right profile is important because it cannot be modified after > the pool is created: a new pool with a different profile needs to be created > and all objects from the previous pool moved to the new. > > > Right, that makes sense. However, is it possible to "migrate" > crush-failure-domain from osd to host, to rack and so on without copying > pools? > > Regards, > Kees > > -- > https://nefos.nl/contact > > Nefos IT bv > Ambachtsweg 25 (industrienummer 4217) > 5627 BZ Eindhoven > Nederland > > KvK 66494931 > > Aanwezig op maandag, dinsdag, woensdag en vrijdag > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Altering crush-failure-domain
Hi Cephers, Documentation on http://docs.ceph.com/docs/master/rados/operations/erasure-code/ states: > Choosing the right profile is important because it cannot be modified > after the pool is created: a new pool with a different profile needs > to be created and all objects from the previous pool moved to the new. Right, that makes sense. However, is it possible to "migrate" crush-failure-domain from osd to host, to rack and so on without copying pools? Regards, Kees -- https://nefos.nl/contact Nefos IT bv Ambachtsweg 25 (industrienummer 4217) 5627 BZ Eindhoven Nederland KvK 66494931 /Aanwezig op maandag, dinsdag, woensdag en vrijdag/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [Nfs-ganesha-devel] NFS-Ganesha CEPH_FSAL ceph.quota.max_bytes not enforced
On Mon, Mar 4, 2019 at 5:53 PM Jeff Layton wrote:
>
> On Mon, 2019-03-04 at 17:26 +, David C wrote:
> > Looks like you're right, Jeff. Just tried to write into the dir and am
> > now getting the quota warning. So I guess it was the libcephfs cache
> > as you say. That's fine for me, I don't need the quotas to be too
> > strict, just a failsafe really.
> >
>
> Actually, I said it was likely the NFS client cache. The Linux kernel is
> allowed to aggressively cache writes if you're doing buffered I/O. The
> NFS client has no concept of the quota here, so you'd only see
> enforcement once those writes start getting flushed back to the server.
>
Ah sorry, that makes a lot of sense!
>
>
> > Interestingly, if I create a new dir, set the same 100MB quota, I can
> > write multiple files with "dd if=/dev/zero of=1G bs=1M count=1024
> > oflag=direct". Wouldn't that bypass the cache? I have the following in
> > my ganesha.conf which I believe effectively disables Ganesha's
> > caching:
> >
> > CACHEINODE {
> > Dir_Chunk = 0;
> > NParts = 1;
> > Cache_Size = 1;
> > }
> >
>
> Using direct I/O like that should take the NFS client cache out of the
> picture. That said, cephfs quota enforcement is pretty "lazy". According
> to http://docs.ceph.com/docs/mimic/cephfs/quota/ :
>
> "Quotas are imprecise. Processes that are writing to the file system
> will be stopped a short time after the quota limit is reached. They will
> inevitably be allowed to write some amount of data over the configured
> limit. How far over the quota they are able to go depends primarily on
> the amount of time, not the amount of data. Generally speaking writers
> will be stopped within 10s of seconds of crossing the configured limit."
>
> You can write quite a bit of data in 10s of seconds (multiple GBs is not
> unreasonable here).
>
> > On Mon, Mar 4, 2019 at 2:50 PM Jeff Layton wrote:
>
> > > > > On Mon, 2019-03-04 at 09:11 -0500, Jeff Layton wrote:
> > This list has
> > > been deprecated. Please subscribe to the new devel list at
> > > lists.nfs-ganesha.org.
> > On Fri, 2019-03-01 at 15:49 +, David C
> > > wrote:
> > > This list has been deprecated. Please subscribe to the new
> > > devel list at lists.nfs-ganesha.org.
> > > Hi All
> > >
> > > Exporting
> > > cephfs with the CEPH_FSAL
> > >
> > > I set the following on a dir:
> > >
> >
> > > > setfattr -n ceph.quota.max_bytes -v 1 /dir
> > > setfattr -n
> > > ceph.quota.max_files -v 10 /dir
> > >
> > > From an NFSv4 client, the
> > > quota.max_bytes appears to be completely ignored, I can go GBs over
> > > the quota in the dir. The quota.max_files DOES work however, if I
> > > try and create more than 10 files, I'll get "Error opening file
> > > 'dir/new file': Disk quota exceeded" as expected.
> > >
> > > From a
> > > fuse-mount on the same server that is running nfs-ganesha, I've
> > > confirmed ceph.quota.max_bytes is enforcing the quota, I'm unable to
> > > copy more than 100MB into the dir.
> > >
> > > According to [1] and [2]
> > > this should work.
> > >
> > > Cluster is Luminous 12.2.10
> > >
> > > Package
> > > versions on nfs-ganesha server:
> > >
> > > nfs-ganesha-rados-grace-
> > > 2.7.1-0.1.el7.x86_64
> > > nfs-ganesha-2.7.1-0.1.el7.x86_64
> > > nfs-
> > > ganesha-vfs-2.7.1-0.1.el7.x86_64
> > > nfs-ganesha-ceph-2.7.1-
> > > 0.1.el7.x86_64
> > > libcephfs2-13.2.2-0.el7.x86_64
> > > ceph-fuse-
> > > 12.2.10-0.el7.x86_64
> > >
> > > My Ganesha export:
> > >
> > > EXPORT
> > > {
> >
> > > > Export_ID=100;
> > > Protocols = 4;
> > > Transports = TCP;
> >
> > > > Path = /;
> > > Pseudo = /ceph/;
> > > Access_Type = RW;
> > >
> > >Attr_Expiration_Time = 0;
> > > #Manage_Gids = TRUE;
> > >
> > > Filesystem_Id = 100.1;
> > > FSAL {
> > > Name = CEPH;
> > >
> > > }
> > > }
> > >
> > > My ceph.conf client section:
> > >
> > > [client]
> > >
> > >mon host = 10.10.10.210:6789, 10.10.10.211:6789,
> > > 10.10.10.212:6789
> > > client_oc_size = 8388608000
> > >
> > > #fuse_default_permission=0
> > > client_acl_type=posix_acl
> > >
> > >client_quota = true
> > > client_quota_df = true
> > >
> > >
> > > Related links:
> > >
> > > [1] http://tracker.ceph.com/issues/16526
> > > [2] https://github.com/nfs-ganesha/nfs-ganesha/issues/100
> > >
> > > Thanks
> > > David
> > >
> >
> > It looks like you're having ganesha do the mount as "client.admin", and
> > I suspect that that may allow you to bypass quotas? You may want to try
> > creating a cephx user with less privileges, have ganesha connect as that
> > user and see if it changes things?
> >
>
> Actually, this may be wrong info.
>
> How are you testing being able to write to the file past quota? Are you
> using O_DIRECT I/O? If not, then it may just be that you're seeing the
> effect of the NFS client caching writes.
>
> --
> Jeff Layton
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
h
Re: [ceph-users] [Nfs-ganesha-devel] NFS-Ganesha CEPH_FSAL ceph.quota.max_bytes not enforced
Looks like you're right, Jeff. Just tried to write into the dir and am now
getting the quota warning. So I guess it was the libcephfs cache as you
say. That's fine for me, I don't need the quotas to be too strict, just a
failsafe really.
Interestingly, if I create a new dir, set the same 100MB quota, I can write
multiple files with "dd if=/dev/zero of=1G bs=1M count=1024 oflag=direct".
Wouldn't that bypass the cache? I have the following in my ganesha.conf
which I believe effectively disables Ganesha's caching:
CACHEINODE {
Dir_Chunk = 0;
NParts = 1;
Cache_Size = 1;
}
Thanks,
On Mon, Mar 4, 2019 at 2:50 PM Jeff Layton wrote:
> On Mon, 2019-03-04 at 09:11 -0500, Jeff Layton wrote:
> > This list has been deprecated. Please subscribe to the new devel list at
> lists.nfs-ganesha.org.
> > On Fri, 2019-03-01 at 15:49 +, David C wrote:
> > > This list has been deprecated. Please subscribe to the new devel list
> at lists.nfs-ganesha.org.
> > > Hi All
> > >
> > > Exporting cephfs with the CEPH_FSAL
> > >
> > > I set the following on a dir:
> > >
> > > setfattr -n ceph.quota.max_bytes -v 1 /dir
> > > setfattr -n ceph.quota.max_files -v 10 /dir
> > >
> > > From an NFSv4 client, the quota.max_bytes appears to be completely
> ignored, I can go GBs over the quota in the dir. The quota.max_files DOES
> work however, if I try and create more than 10 files, I'll get "Error
> opening file 'dir/new file': Disk quota exceeded" as expected.
> > >
> > > From a fuse-mount on the same server that is running nfs-ganesha, I've
> confirmed ceph.quota.max_bytes is enforcing the quota, I'm unable to copy
> more than 100MB into the dir.
> > >
> > > According to [1] and [2] this should work.
> > >
> > > Cluster is Luminous 12.2.10
> > >
> > > Package versions on nfs-ganesha server:
> > >
> > > nfs-ganesha-rados-grace-2.7.1-0.1.el7.x86_64
> > > nfs-ganesha-2.7.1-0.1.el7.x86_64
> > > nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
> > > nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> > > libcephfs2-13.2.2-0.el7.x86_64
> > > ceph-fuse-12.2.10-0.el7.x86_64
> > >
> > > My Ganesha export:
> > >
> > > EXPORT
> > > {
> > > Export_ID=100;
> > > Protocols = 4;
> > > Transports = TCP;
> > > Path = /;
> > > Pseudo = /ceph/;
> > > Access_Type = RW;
> > > Attr_Expiration_Time = 0;
> > > #Manage_Gids = TRUE;
> > > Filesystem_Id = 100.1;
> > > FSAL {
> > > Name = CEPH;
> > > }
> > > }
> > >
> > > My ceph.conf client section:
> > >
> > > [client]
> > > mon host = 10.10.10.210:6789, 10.10.10.211:6789,
> 10.10.10.212:6789
> > > client_oc_size = 8388608000
> > > #fuse_default_permission=0
> > > client_acl_type=posix_acl
> > > client_quota = true
> > > client_quota_df = true
> > >
> > > Related links:
> > >
> > > [1] http://tracker.ceph.com/issues/16526
> > > [2] https://github.com/nfs-ganesha/nfs-ganesha/issues/100
> > >
> > > Thanks
> > > David
> > >
> >
> > It looks like you're having ganesha do the mount as "client.admin", and
> > I suspect that that may allow you to bypass quotas? You may want to try
> > creating a cephx user with less privileges, have ganesha connect as that
> > user and see if it changes things?
> >
>
> Actually, this may be wrong info.
>
> How are you testing being able to write to the file past quota? Are you
> using O_DIRECT I/O? If not, then it may just be that you're seeing the
> effect of the NFS client caching writes.
> --
> Jeff Layton
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to just delete PGs stuck incomplete on EC pool
Thanks for the suggestions. I've tried both -- setting osd_find_best_info_ignore_history_les = true and restarting all OSDs, as well as 'ceph osd-force-create-pg' -- but both still show incomplete PG_AVAILABILITY Reduced data availability: 2 pgs inactive, 2 pgs incomplete pg 18.c is incomplete, acting [32,48,58,40,13,44,61,59,30,27,43,37] (reducing pool ec84-hdd-zm min_size from 8 may help; search ceph.com/docs for 'incomplete') pg 18.1e is incomplete, acting [50,49,41,58,60,46,52,37,34,63,57,16] (reducing pool ec84-hdd-zm min_size from 8 may help; search ceph.com/docs for 'incomplete') The OSDs in down_osds_we_would_probe have already been marked lost When I ran the force-create-pg command, they went to peering for a few seconds, but then went back incomplete. Updated ceph pg 18.1e query https://pastebin.com/XgZHvJXu Updated ceph pg 18.c query https://pastebin.com/N7xdQnhX Any other suggestions? Thanks again, Daniel On Sat, Mar 2, 2019 at 3:44 PM Paul Emmerich wrote: > On Sat, Mar 2, 2019 at 5:49 PM Alexandre Marangone > wrote: > > > > If you have no way to recover the drives, you can try to reboot the OSDs > with `osd_find_best_info_ignore_history_les = true` (revert it afterwards), > you'll lose data. If after this, the PGs are down, you can mark the OSDs > blocking the PGs from become active lost. > > this should work for PG 18.1e, but not for 18.c. Try running "ceph osd > force-create-pg " to reset the PGs instead. > Data will obviously be lost afterwards. > > Paul > > > > > On Sat, Mar 2, 2019 at 6:08 AM Daniel K wrote: > >> > >> They all just started having read errors. Bus resets. Slow reads. Which > is one of the reasons the cluster didn't recover fast enough to compensate. > >> > >> I tried to be mindful of the drive type and specifically avoided the > larger capacity Seagates that are SMR. Used 1 SM863 for every 6 drives for > the WAL. > >> > >> Not sure why they failed. The data isn't critical at this point, just > need to get the cluster back to normal. > >> > >> On Sat, Mar 2, 2019, 9:00 AM wrote: > >>> > >>> Did they break, or did something went wronng trying to replace them? > >>> > >>> Jespe > >>> > >>> > >>> > >>> Sent from myMail for iOS > >>> > >>> > >>> Saturday, 2 March 2019, 14.34 +0100 from Daniel K >: > >>> > >>> I bought the wrong drives trying to be cheap. They were 2TB WD Blue > 5400rpm 2.5 inch laptop drives. > >>> > >>> They've been replace now with HGST 10K 1.8TB SAS drives. > >>> > >>> > >>> > >>> On Sat, Mar 2, 2019, 12:04 AM wrote: > >>> > >>> > >>> > >>> Saturday, 2 March 2019, 04.20 +0100 from satha...@gmail.com < > satha...@gmail.com>: > >>> > >>> 56 OSD, 6-node 12.2.5 cluster on Proxmox > >>> > >>> We had multiple drives fail(about 30%) within a few days of each > other, likely faster than the cluster could recover. > >>> > >>> > >>> Hov did so many drives break? > >>> > >>> Jesper > >> > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 13.2.4 odd memory leak?
Bloated to ~4 GB per OSD and you are on HDDs? 13.2.3 backported the cache auto-tuning which targets 4 GB memory usage by default. See https://ceph.com/releases/13-2-4-mimic-released/ The bluestore_cache_* options are no longer needed. They are replaced by osd_memory_target, defaulting to 4GB. BlueStore will expand and contract its cache to attempt to stay within this limit. Users upgrading should note this is a higher default than the previous bluestore_cache_size of 1GB, so OSDs using BlueStore will use more memory by default. For more details, see the BlueStore docs. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Mar 4, 2019 at 3:55 PM Steffen Winther Sørensen wrote: > > List Members, > > patched a centos 7 based cluster from 13.2.2 to 13.2.4 last monday, > everything appeared working fine. > > Only this morning I found all OSDs in the cluster to be bloated in memory > foot print, possible after weekend backup through MDS. > > Anyone else seeing possible memory leak in 13.2.4 OSD possible primarily when > using MDS? > > TIA > > /Steffen > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to use STS Lite correctly?
There are two steps that have to be performed before calling AssumeRole:
1. A role named S3Access needs to be created to which it is mandatory to
attach an assume role policy document. For example,
radosgw-admin role create --role-name=S3Access
--path=/application_abc/component_xyz/
--assume-role-policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Principal\":\{\"AWS\":\[\"arn:aws:iam:::user/TESTER\"\]\},\"Action\":\[\"sts:AssumeRole\"\]\}\]\}
2. A permission policy needs to be attached to the role, to allow all s3
operations using the temporary creds returned by the Assume role call. For
example,
radosgw-admin role-policy put --role-name=S3Access
--policy-name=Policy1
--policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Action\":\[\"s3:*\"\],\"Resource\":\"arn:aws:s3:::example_bucket\"\}\]\}
The documentation for above is at:
http://docs.ceph.com/docs/master/radosgw/role/
Thanks,
Pritha
On Mon, Mar 4, 2019 at 4:48 PM myxingkong wrote:
>
> I want to use the STS service to generate temporary credentials for use by
> third-party clients.
>
> I configured STS lite based on the documentation.
> http://docs.ceph.com/docs/master/radosgw/STSLite/
>
> This is my configuration file:
>
> [global]
> fsid = 42a7cae1-84d1-423e-93f4-04b0736c14aa
> mon_initial_members = admin, node1, node2, node3
> mon_host = 192.168.199.81,192.168.199.82,192.168.199.83,192.168.199.84
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
>
> osd pool default size = 2
>
> [client.rgw.admin]
> rgw sts key = "1234567890"
> rgw s3 auth use sts = true
>
> When I execute the getSessionToken method, return a 403 error:
>
>
> AccessDenied
> txd-005c7d07ed-3a3c-default
> 3a3c-default-default
>
>
> try:
> host = 'http://192.168.199.81:7480'
> access_key = '2324YFZ7QDEOSRL18QHR'
> secret_key = 'rL9FabxCOw5LDbrHtmykiGSCjzpKLmEs9WPiNjVJ'
>
> client = boto3.client('sts',
> aws_access_key_id = access_key,
> aws_secret_access_key = secret_key,
> endpoint_url = host)
> response = client.assume_role(
>
> RoleArn='arn:aws:iam:::role/application_abc/component_xyz/S3Access',
> RoleSessionName='Bob',
> DurationSeconds=3600
> )
> print response
> except:
> print traceback.format_exc()
>
> Who can tell me if my configuration or code is wrong?
>
> My version of ceph is: ceph version 14.1.0
> (adfd524c32325562f61c055a81dba4cb1b117e84) nautilus (dev)
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 13.2.4 odd memory leak?
List Members, patched a centos 7 based cluster from 13.2.2 to 13.2.4 last monday, everything appeared working fine. Only this morning I found all OSDs in the cluster to be bloated in memory foot print, possible after weekend backup through MDS. Anyone else seeing possible memory leak in 13.2.4 OSD possible primarily when using MDS? TIA /Steffen ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] How to use STS Lite correctly?
I want to use the STS service to generate temporary credentials for use by
third-party clients.
I configured STS lite based on the documentation.
http://docs.ceph.com/docs/master/radosgw/STSLite/
This is my configuration file:
[global]
fsid = 42a7cae1-84d1-423e-93f4-04b0736c14aa
mon_initial_members = admin, node1, node2, node3
mon_host = 192.168.199.81,192.168.199.82,192.168.199.83,192.168.199.84
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 2
[client.rgw.admin]
rgw sts key = "1234567890"
rgw s3 auth use sts = true
When I execute the getSessionToken method, return a 403 error:
AccessDenied
txd-005c7d07ed-3a3c-default
3a3c-default-default
try:
host = 'http://192.168.199.81:7480'
access_key = '2324YFZ7QDEOSRL18QHR'
secret_key = 'rL9FabxCOw5LDbrHtmykiGSCjzpKLmEs9WPiNjVJ'
client = boto3.client('sts',
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
endpoint_url = host)
response = client.assume_role(
RoleArn='arn:aws:iam:::role/application_abc/component_xyz/S3Access',
RoleSessionName='Bob',
DurationSeconds=3600
)
print response
except:
print traceback.format_exc()
Who can tell me if my configuration or code is wrong?
My version of ceph is: ceph version 14.1.0
(adfd524c32325562f61c055a81dba4cb1b117e84) nautilus (dev)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph tracker login failed
Fixed: use only user id (swamireddy) instead of full openID url. On Thu, Feb 28, 2019 at 7:04 PM M Ranga Swami Reddy wrote: > > I tried to login to ceph tracker - it failing with openID url.? > > I tried with my OpenID: > http://tracker.ceph.com/login > > my id: https://code.launchpad.net/~swamireddy ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Erasure coded pools and ceph failure domain setup
On 02/03/2019 01:02, Ravi Patel wrote:
Hello,
My question is how crush distributes chunks throughout the cluster with
erasure coded pools. Currently, we have 4 OSD nodes with 36 drives(OSD
daemons) per node. If we use ceph_failire_domaon=host, then we are
necessarily limited to k=3,m=1, or k=2,m=2. We would like to explore
k>3, m>2 modes of coding but are unsure how the crush rule set will
distribute the chunks if we set the crush_failure_domain to OSD
Ideally, we would like CRUSH to distribute the chunks hierarchically so
to spread them evenly across the nodes. For example, all chunks are on a
single node.
Are chunks evenly spread by default? If not, how might we go about
configuring them?
You can write your own CRUSH rules to distribute chunks hierarchically.
For example, you can have a k=6, m=2 code together with a rule that
guarantees that each node gets two chunks. This means that if you lose a
node you do not lose data (though depending on your min_size setting
your pool might be unavailable at that point until you replace the node
or add a new one and the chunks can be recovered). You would accomplish
this with a rule that looks like this:
rule ec8 {
id
type erasure
min_size 7
max_size 8
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default
step choose indep 4 type host
step chooseleaf indep 2 type osd
step emit
}
This means the rule will first pick 4 hosts, then pick 2 OSDs per host,
resulting in a total of 8 OSDs. This is appropriate for k=6 m=2 codes as
well as k=5 m=2 codes (that will just leave one random OSD unused),
hence min_size 7 max_size 8.
If you just set crush_failure_domain to OSD, then the rule will pick
random OSDs without regard for the hosts; you will be able to use
effectively any EC widths you want, but there will be no guarantees of
data durability if you lose a whole host.
--
Hector Martin (hec...@marcansoft.com)
Public Key: https://mrcn.st/pub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
