[ceph-users] how-to undo a "multisite" config
Hi Cephers. Quick question couldn't find a "how-to" or "docu"... not even sure if someone else ever had to do it... What would be the steps to make a (failed) multisite config change, exactly following - http://docs.ceph.com/docs/master/radosgw/multisite/ undone again? And as I'm on that topic now, any other WORKING documentation how to setup "multisite"? Thanks & regards Anton ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw global quotas - how to set in jewel?
Ah, thanks, I thought I was going crazy for a bit there! The global quota would be useful for us (now wanting to retroactively impose quotas on pre-existing users), but we can script a workaround instead. Thanks, Graham On 03/29/2017 10:17 AM, Casey Bodley wrote: Hi Graham, you're absolutely right. In jewel, these settings were moved into the period, but radosgw-admin doesn't have any commands to modify them. I opened a tracker issue for this at http://tracker.ceph.com/issues/19409. For now, it looks like you're stuck with the 'default quota' settings in ceph.conf. Thanks, Casey On 03/27/2017 03:13 PM, Graham Allan wrote: I'm following up to myself here, but I'd love to hear if anyone knows how the global quotas can be set in jewel's radosgw. I haven't found anything which has an effect - the documentation says to use: radosgw-admin region-map get > regionmap.json ...edit the json file radosgw-admin region-map set < regionmap.json but this has no effect on jewel. There doesn't seem to be any analogous function in the "period"-related commands which I think would be the right place to look for jewel. Am I missing something, or should I open a bug? Graham -- Graham Allan Minnesota Supercomputing Institute - g...@umn.edu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."
Interesting. I'm wondering what the -13 return code for the op execution in my debug output is (can't find in the source..) I just tried out setting the lifecycle with cyberduck and got this error, which is probably the other bug with AWSv4 auth, http://tracker.ceph.com/ issues/17076 Not sure if cyberduck can be forced to use V2. 2017-04-03 12:07:15.093235 7f5617024700 10 op=20RGWPutLC_ObjStore_S3 2017-04-03 12:07:15.093248 7f5617024700 2 req 14:0.000438:s3:PUT /bentest/:put_lifecycle:authorizing . 2017-04-03 12:07:15.093637 7f5617024700 10 delaying v4 auth 2017-04-03 12:07:15.093643 7f5617024700 10 ERROR: AWS4 completion for this operation NOT IMPLEMENTED 2017-04-03 12:07:15.093652 7f5617024700 10 failed to authorize request 2017-04-03 12:07:15.093658 7f5617024700 20 handler->ERRORHANDLER: err_no=-2201 new_err_no=-2201 2017-04-03 12:07:15.093844 7f5617024700 2 req 14:0.001034:s3:PUT /bentest/:put_lifecycle:op status=0 2017-04-03 12:07:15.093859 7f5617024700 2 req 14:0.001050:s3:PUT /bentest/:put_lifecycle:http status=501 2017-04-03 12:07:15.093884 7f5617024700 1 == req done req=0x7f561701e340 op status=0 http_status=501 == -Ben On Mon, Apr 3, 2017 at 7:16 AM, wrote: > ... hmm, "modify" gives no error and may be the option to use, but I don't > see anything related to an "expires" meta field > > [root s3cmd-master]# ./s3cmd --no-ssl --verbose modify s3://Test/INSTALL > --expiry-days=365 > INFO: Summary: 1 remote files to modify > modify: 's3://Test/INSTALL' > > [root s3cmd-master]# ./s3cmd --no-ssl --verbose info s3://Test/INSTALL > s3://Test/INSTALL (object): >File size: 3123 >Last mod: Mon, 03 Apr 2017 12:35:28 GMT >MIME type: text/plain >Storage: STANDARD >MD5 sum: 63834dbb20b32968505c4ebe768fc8c4 >SSE: none >policy: xmlns="http://s3.amazonaws.com/doc/2006-03-01/";> > Test1000< > /MaxKeys>false > INSTALL2017-04-03T12:35:28.533Z LastModified>"63834dbb20b32968505c4ebe768fc8 > c4"3123STANDARD StorageClass>666First > User > README.TXT2017-03-31T22:36:38.380Z LastModified>"708efc3b9184c8b112e36062804aca > 1e"88STANDARD StorageClass>666First > User >cors:none >ACL: First User: FULL_CONTROL >x-amz-meta-s3cmd-attrs: atime:1491218263/ctime: > 1490998096/gid:0/gname:root/md5:63834dbb20b32968505c4ebe768fc8 > c4/mode:33188/mtime:1488021707/uid:0/uname:root > > > *Gesendet:* Montag, 03. April 2017 um 14:13 Uhr > *Von:* ceph.nov...@habmalnefrage.de > *An:* ceph-users > > *Betreff:* Re: [ceph-users] Kraken release and RGW --> "S3 bucket > lifecycle API has been added. Note that currently it only supports object > expiration." > ... additional strange but a bit different info related to the "permission > denied" > > [root s3cmd-master]# ./s3cmd --no-ssl put INSTALL s3://Test/ > --expiry-days=5 > upload: 'INSTALL' -> 's3://Test/INSTALL' [1 of 1] > 3123 of 3123 100% in 0s 225.09 kB/s done > > [root s3cmd-master]# ./s3cmd info s3://Test/INSTALL > s3://Test/INSTALL (object): > File size: 3123 > Last mod: Mon, 03 Apr 2017 12:01:47 GMT > MIME type: text/plain > Storage: STANDARD > MD5 sum: 63834dbb20b32968505c4ebe768fc8c4 > SSE: none > policy: http://s3.amazonaws.com/doc/2006-03-01/";> > Test1000< > /MaxKeys>false > INSTALL2017-04-03T12:01:47.745Z LastModified>"63834dbb20b32968505c4ebe768fc8 > c4"3123STANDARD StorageClass>666First > User > README.TXT2017-03-31T22:36:38.380Z LastModified>"708efc3b9184c8b112e36062804aca > 1e"88STANDARD StorageClass>666First > User > cors: none > ACL: First User: FULL_CONTROL > x-amz-meta-s3cmd-attrs: atime:1491218263/ctime: > 1490998096/gid:0/gname:root/md5:63834dbb20b32968505c4ebe768fc8 > c4/mode:33188/mtime:1488021707/uid:0/uname:root > > [root s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/ --expiry-days=365 > ERROR: Access to bucket 'Test' was denied > ERROR: S3 error: 403 (AccessDenied) > > [root s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/INSTALL > --expiry-days=365 > ERROR: Parameter problem: Expecting S3 URI with just the bucket name set > instead of 's3://Test/INSTALL' > [root@mucsds26 s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/ > --expiry-days=365 > ERROR: Access to bucket 'Test' was denied > ERROR: S3 error: 403 (AccessDenied) > > [root s3cmd-master]# ./s3cmd --no-ssl la expire s3://Test > 2017-04-03 12:01 3123 s3://Test/INSTALL > 2017-03-31 22:36 88 s3://Test/README.TXT > > > > > Gesendet: Montag, 03. April 2017 um 12:31 Uhr > Von: ceph.nov...@habmalnefrage.de > An: "Ben Hines" , ceph-users > Betreff: Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle > API has been added. Note that currently it only supports object expiration." > Hi Cephers... > > I did set the "lifecycle" via Cyberduck.I do also get an error first, then > suddenly Cyberduck refreshes the window aand the lifecycle is there. > > I see the following when I check it via s3cmd (GitHub master version > because the regular installed version doesn't
Re: [ceph-users] deep-scrubbing
Thanks Sage. We have older version Ceph (Firefly version), where I could not see this behavior (is on Saturday, after enabling the deep-scrub @ 7AM hours, I could not see the deep-scrub started for Friday's PG deep-scrub)). Is the deep-scrub's performed based on the time-stamp? For ex: a PG deep-scrubbed on Friday 10AM and if its not deep-scrubed on the next Friday 10AM, then Saturday 10AM its will be performed (assumed that Saturday 7AM deep-scrub enabled). Thanks Swami On Mon, Apr 3, 2017 at 6:57 PM, Sage Weil wrote: > On Mon, 3 Apr 2017, M Ranga Swami Reddy wrote: >> + ceph-devel >> >> On Mon, Feb 27, 2017 at 3:54 PM, M Ranga Swami Reddy >> wrote: >> > Hello, >> > I use a ceph cluster and its show the deeps scrub's PG distribution as >> > below >> > from "ceph pg dump" command: >> > >> > >> >2000 Friday >> >1000 Saturday >> >4000 Sunday >> > == >> > >> > On Friday, I have disabled the deep-scrub due to some reason. If this case, >> > all Friday's PG deep-scrub will be performed on Saturday or is it will be >> > done on Next Friday? > > Saturday. > > ks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw leaking objects
On Mon, Apr 3, 2017 at 1:32 AM, Luis Periquito wrote: >> Right. The tool isn't removing objects (yet), because we wanted to >> have more confidence in the tool before having it automatically >> deleting all the found objects. The process currently is to manually >> move these objects to a different backup pool (via rados cp, rados >> rm), then when you're confident that no needed data was lost in the >> process remove the backup pool. In the future we'll automate that. > > My problem exactly. I don't have enough confidence in myself to just > delete a bunch of random objects... Any idea as to when will be > available such tool? Why random? The objects are the ones that the orphan tool pointed at. And the idea is to move these objects to a safe place before removal, so that even if the wrong objects are removed, they can be recovered. There is no current ETA for the tool, but the tool will probably have the same two steps as reflected here: 1. backup, 2. remove backup. Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 答复: 答复: rbd expord-diff aren't counting AioTruncate op correctly
Hi, actually, in our test, we created an image with the size 4MB which means it has only one object, then we write data into it, create a snap, clone it and create a snap of the new image, say "snap1", and then we send a "AioTruncate" to truncate the image to 1MB and create the second snap of the new image, say "snap2". Then, we use export-diff to calculate the "diff" between snap1 and snap2. By examing the result of export-diff, we confirmed that only the data that left in object are recorded into the export-diff's result, the truncate operation is lost. By the way, according to our understanding of the source code, it seems that the reason for this is that when doing copy-up, no clone is created even if there has already been some snapshot of the new image. So, if there's only one "WRITE" op is sent to the object, which leads to a "copy-up", before a new snapshot is created after which a export-diff is conducted, the export-diff will copy all the data in the HEAD object, which, in our case, is not the "diff" that we want. 发件人: Jason Dillaman [jdill...@redhat.com] 发送时间: 2017年4月3日 23:00 收件人: 许雪寒 Cc: ceph-users@lists.ceph.com 主题: Re: 答复: [ceph-users] rbd expord-diff aren't counting AioTruncate op correctly Are you referring to the class "AioTruncate"? If so, *how* did you issue the truncate against a specific object? Did you use the RBD API's discard method? Did you resize an image? I need a repeatable set of instructions. On Mon, Apr 3, 2017 at 10:58 AM, 许雪寒 wrote: > Hi, the operation we performed is AioTruncate. > > 发件人: Jason Dillaman [jdill...@redhat.com] > 发送时间: 2017年4月3日 22:11 > 收件人: 许雪寒 > Cc: ceph-users@lists.ceph.com > 主题: Re: [ceph-users] rbd expord-diff aren't counting AioTruncate op correctly > > On Fri, Mar 31, 2017 at 10:36 PM, 许雪寒 wrote: >> We created an image, write data into it, then protect it and clone a new >> image. Then we create a snapshot, snap1, of the cloned image, then truncate >> it and create a snapshot, snap2. In the diff of the two snapshot, we found >> that only the data that are not truncated is recorded in the diff, which >> means the truncate operation is not reflected by the diff. > > When you say "truncate" above, what specific operation are you > performing so that I can attempt to recreate it? > > > -- > Jason -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] List-Archive unavailable
accessible again, perhaps rebuild the mailman archive too (partially missing history from 03/29 to 04/02). On Sat, Apr 1, 2017 at 4:47 AM, Robert Sander wrote: > Hi, > > the list archive at http://lists.ceph.com/pipermail/ceph-users-ceph.com/ > is currently not available. Anybody knows what is going on there? > > Regards > -- > Robert Sander > Heinlein Support GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 93818 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Herbert ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 答复: rbd expord-diff aren't counting AioTruncate op correctly
Are you referring to the class "AioTruncate"? If so, *how* did you issue the truncate against a specific object? Did you use the RBD API's discard method? Did you resize an image? I need a repeatable set of instructions. On Mon, Apr 3, 2017 at 10:58 AM, 许雪寒 wrote: > Hi, the operation we performed is AioTruncate. > > 发件人: Jason Dillaman [jdill...@redhat.com] > 发送时间: 2017年4月3日 22:11 > 收件人: 许雪寒 > Cc: ceph-users@lists.ceph.com > 主题: Re: [ceph-users] rbd expord-diff aren't counting AioTruncate op correctly > > On Fri, Mar 31, 2017 at 10:36 PM, 许雪寒 wrote: >> We created an image, write data into it, then protect it and clone a new >> image. Then we create a snapshot, snap1, of the cloned image, then truncate >> it and create a snapshot, snap2. In the diff of the two snapshot, we found >> that only the data that are not truncated is recorded in the diff, which >> means the truncate operation is not reflected by the diff. > > When you say "truncate" above, what specific operation are you > performing so that I can attempt to recreate it? > > > -- > Jason -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 答复: rbd expord-diff aren't counting AioTruncate op correctly
Hi, the operation we performed is AioTruncate. 发件人: Jason Dillaman [jdill...@redhat.com] 发送时间: 2017年4月3日 22:11 收件人: 许雪寒 Cc: ceph-users@lists.ceph.com 主题: Re: [ceph-users] rbd expord-diff aren't counting AioTruncate op correctly On Fri, Mar 31, 2017 at 10:36 PM, 许雪寒 wrote: > We created an image, write data into it, then protect it and clone a new > image. Then we create a snapshot, snap1, of the cloned image, then truncate > it and create a snapshot, snap2. In the diff of the two snapshot, we found > that only the data that are not truncated is recorded in the diff, which > means the truncate operation is not reflected by the diff. When you say "truncate" above, what specific operation are you performing so that I can attempt to recreate it? -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."
... hmm, "modify" gives no error and may be the option to use, but I don't see anything related to an "expires" meta field
[root s3cmd-master]# ./s3cmd --no-ssl --verbose modify s3://Test/INSTALL --expiry-days=365
INFO: Summary: 1 remote files to modify
modify: 's3://Test/INSTALL'
[root s3cmd-master]# ./s3cmd --no-ssl --verbose info s3://Test/INSTALL
s3://Test/INSTALL (object):
File size: 3123
Last mod: Mon, 03 Apr 2017 12:35:28 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: 63834dbb20b32968505c4ebe768fc8c4
SSE: none
policy: Test1000falseINSTALL2017-04-03T12:35:28.533Z"63834dbb20b32968505c4ebe768fc8c4"3123STANDARD666First UserREADME.TXT2017-03-31T22:36:38.380Z"708efc3b9184c8b112e36062804aca1e"88STANDARD666First User
cors: none
ACL: First User: FULL_CONTROL
x-amz-meta-s3cmd-attrs: atime:1491218263/ctime:1490998096/gid:0/gname:root/md5:63834dbb20b32968505c4ebe768fc8c4/mode:33188/mtime:1488021707/uid:0/uname:root
Gesendet: Montag, 03. April 2017 um 14:13 Uhr
Von: ceph.nov...@habmalnefrage.de
An: ceph-users
Betreff: Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."
... additional strange but a bit different info related to the "permission denied"
[root s3cmd-master]# ./s3cmd --no-ssl put INSTALL s3://Test/ --expiry-days=5
upload: 'INSTALL' -> 's3://Test/INSTALL' [1 of 1]
3123 of 3123 100% in 0s 225.09 kB/s done
[root s3cmd-master]# ./s3cmd info s3://Test/INSTALL
s3://Test/INSTALL (object):
File size: 3123
Last mod: Mon, 03 Apr 2017 12:01:47 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: 63834dbb20b32968505c4ebe768fc8c4
SSE: none
policy: Test1000falseINSTALL2017-04-03T12:01:47.745Z"63834dbb20b32968505c4ebe768fc8c4"3123STANDARD666First UserREADME.TXT2017-03-31T22:36:38.380Z"708efc3b9184c8b112e36062804aca1e"88STANDARD666First User
cors: none
ACL: First User: FULL_CONTROL
x-amz-meta-s3cmd-attrs: atime:1491218263/ctime:1490998096/gid:0/gname:root/md5:63834dbb20b32968505c4ebe768fc8c4/mode:33188/mtime:1488021707/uid:0/uname:root
[root s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/ --expiry-days=365
ERROR: Access to bucket 'Test' was denied
ERROR: S3 error: 403 (AccessDenied)
[root s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/INSTALL --expiry-days=365
ERROR: Parameter problem: Expecting S3 URI with just the bucket name set instead of 's3://Test/INSTALL'
[root@mucsds26 s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/ --expiry-days=365
ERROR: Access to bucket 'Test' was denied
ERROR: S3 error: 403 (AccessDenied)
[root s3cmd-master]# ./s3cmd --no-ssl la expire s3://Test
2017-04-03 12:01 3123 s3://Test/INSTALL
2017-03-31 22:36 88 s3://Test/README.TXT
Gesendet: Montag, 03. April 2017 um 12:31 Uhr
Von: ceph.nov...@habmalnefrage.de
An: "Ben Hines" , ceph-users
Betreff: Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."
Hi Cephers...
I did set the "lifecycle" via Cyberduck.I do also get an error first, then suddenly Cyberduck refreshes the window aand the lifecycle is there.
I see the following when I check it via s3cmd (GitHub master version because the regular installed version doesn't offer the "getlifecycle" option):
[root s3cmd-master]# ./s3cmd getlifecycle s3://Test/README.txt
Cyberduck-nVWEhQwE
Enabled
1
Here is my S3 "user info":
[root ~]# radosgw-admin user info --uid=666
{
"user_id": "666",
"display_name": "First User",
"email": "a...@c.de",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "666",
"access_key": "abc ;)",
"secret_key": "abc def ;)"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw"
}
If someone has a working example how to set lifecycle via the s3cmd, I can try it and send the outcome...
Gesendet: Montag, 03. April 2017 um 01:43 Uhr
Von: "Ben Hines"
An: "Orit Wasserman"
Cc: ceph-users
Betreff: Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."
Hmm, Nope, not using tenants feature. The users/buckets were created on prior ceph versions, perhaps i'll try with a newly created user + bucket.
radosgw-admin user info --uid=foo
{
"user_id": "foo",
"display_name": "foo",
"email": "snip",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
{
Re: [ceph-users] rbd expord-diff aren't counting AioTruncate op correctly
On Fri, Mar 31, 2017 at 10:36 PM, 许雪寒 wrote: > We created an image, write data into it, then protect it and clone a new > image. Then we create a snapshot, snap1, of the cloned image, then truncate > it and create a snapshot, snap2. In the diff of the two snapshot, we found > that only the data that are not truncated is recorded in the diff, which > means the truncate operation is not reflected by the diff. When you say "truncate" above, what specific operation are you performing so that I can attempt to recreate it? -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Get/set/list rbd image using python librbd
We try to keep the C/C++ and Python APIs in-sync, but it looks like
these functions were missed and are not currently available via the
Python API. I created a tracker ticket [1] the issue. If you are
interested, feel free to contribute a pull request for the missing
APIs.
[1] http://tracker.ceph.com/issues/19451
On Sun, Apr 2, 2017 at 8:17 PM, Sayid Munawar wrote:
> Hi,
>
> Using rbd command line, we can set / get / list image-meta of an rbd image
> as described in the man page.
>
> # rbd image-meta list mypool/myimage
>
>
> How can we do the same using python librbdpy ? i can't find it in the
> documentation.
>
> with rados.Rados(conffile='my_ceph.conf') as cluster:
> with cluster.open_ioctx('mypool') as ioctx:
> with rbd.Image(ioctx, 'myimage') as image:
> image._some_method_to_set_metadata() ???
>
>
> Thank you
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] deep-scrubbing
On Mon, 3 Apr 2017, M Ranga Swami Reddy wrote: > + ceph-devel > > On Mon, Feb 27, 2017 at 3:54 PM, M Ranga Swami Reddy > wrote: > > Hello, > > I use a ceph cluster and its show the deeps scrub's PG distribution as below > > from "ceph pg dump" command: > > > > > >2000 Friday > >1000 Saturday > >4000 Sunday > > == > > > > On Friday, I have disabled the deep-scrub due to some reason. If this case, > > all Friday's PG deep-scrub will be performed on Saturday or is it will be > > done on Next Friday? Saturday. ks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Troubleshooting incomplete PG's
On Fri, 31 Mar 2017, nokia ceph wrote:
> Hello Brad,
> Many thanks of the info :)
>
> ENV:-- Kracken - bluestore - EC 4+1 - 5 node cluster : RHEL7
>
> What is the status of the down+out osd? Only one osd osd.6 down and out from
> cluster.
> What role did/does it play? Mostimportantly, is it osd.6? Yes, due to
> underlying I/O error issue we removed this device from the cluster.
Is the device completely destroyed or is it only returning errors
when reading certain data? It is likely that some (or all) of the
incomplete PGs can be extracted from the drive if the bad sector(s) don't
happen to affect those pgs. The ceph-objectstore-tool --op export command
can be used for this (extract it from the affected drive and add it to
some other osd).
> I put this parameter " osd_find_best_info_ignore_history_les = true" in
> ceph.conf, and find those 22 PG's were changed to "down+remapped" . Now all
> are reverted to "remapped+incomplete" state.
This is usually not a great idea unless you're out of options, by the way!
> #ceph pg stat 2> /dev/null
> v2731828: 4096 pgs: 1 incomplete, 21 remapped+incomplete, 4074 active+clean;
> 268 TB data, 371 TB used, 267 TB / 638 TB avail
>
> ## ceph -s
> 2017-03-30 19:02:14.350242 7f8b0415f700 -1 WARNING: the following dangerous
> and experimental features are enabled: bluestore,rocksdb
> 2017-03-30 19:02:14.366545 7f8b0415f700 -1 WARNING: the following dangerous
> and experimental features are enabled: bluestore,rocksdb
> cluster bd8adcd0-c36d-4367-9efe-f48f5ab5f108
> health HEALTH_ERR
> 22 pgs are stuck inactive for more than 300 seconds
> 22 pgs incomplete
> 22 pgs stuck inactive
> 22 pgs stuck unclean
> monmap e2: 5 mons
> at{au-adelaide=10.50.21.24:6789/0,au-brisbane=10.50.21.22:6789/0,au-canberra=
> 10.50.21.23:6789/0,au-melbourne=10.50.21.21:6789/0,au-sydney=10.50.21.20:67
> 89/0}
> election epoch 180, quorum 0,1,2,3,4
> au-sydney,au-melbourne,au-brisbane,au-canberra,au-adelaide
> mgr active: au-adelaide
> osdmap e6506: 117 osds: 117 up, 117 in; 21 remapped pgs
> flags sortbitwise,require_jewel_osds,require_kraken_osds
> pgmap v2731828: 4096 pgs, 1 pools, 268 TB data, 197 Mobjects
> 371 TB used, 267 TB / 638 TB avail
> 4074 active+clean
> 21 remapped+incomplete
> 1 incomplete
>
>
> ## ceph osd dump 2>/dev/null | grep cdvr
> pool 1 'cdvr_ec' erasure size 5 min_size 4 crush_ruleset 1 object_hash
> rjenkins pg_num 4096 pgp_num 4096 last_change 456 flags
> hashpspool,nodeep-scrub stripe_width 65536
>
> Inspecting affected PG 1.e4b
>
> # ceph pg dump 2> /dev/null | grep 1.e4b
> 1.e4b 50832 0 0 0 0 73013340821
> 10006 10006 remapped+incomplete 2017-03-30 14:14:26.297098 3844'161662
> 6506:325748 [113,66,15,73,103] 113 [NONE,NONE,NONE,73,NONE]
> 73 1643'139486 2017-03-21 04:56:16.683953 0'0 2017-02-21
> 10:33:50.012922
>
> When I trigger below command.
>
> #ceph pg force_create_pg 1.e4b
> pg 1.e4b now creating, ok
>
> As it went to creating state, no change after that. Can you explain why this
> PG showing null values after triggering "force_create_pg",?
>
> ]# ceph pg dump 2> /dev/null | grep 1.e4b
> 1.e4b 0 0 0 0 0 0
> 0 0 creating 2017-03-30 19:07:00.982178 0'0
> 0:0 [] -1 []
> -1 0'0 0.00 0'0
> 0.00
CRUSH isn't mapping the PG to any OSDs, so there is nowhere to create it,
it seems? What does 'ceph pg map ' show?
> Then I triggered below command
>
> # ceph pg repair 1.e4b
> Error EAGAIN: pg 1.e4b has no primary osd --<<
>
> Could you please provide answer for below queries.
>
> 1. How to fix this "incomplete+remapped" PG issue, here all OSD's were up
> and running and affected OSD marked out and removed from the cluster.
To recover the data, you need to find surviving shards of the PG.
ceph-objectstore-tool on the "failed" disk is one option, but since this
is a 4+2 code there should have been another copy that got lost along the
line... do you know where it is?
> 2. Will reduce min_size helps? currently it set to 4. Could you please
> explain what is the impact if we reduce min_size for the current config EC
> 4+1
You can't reduce it below 4 since it's a 4+2 code. By default we set it
as 5 (k+1) so that you won't write new data to the PG if a single
additional failure could lead you to lose those writes.
> 3. Is there any procedure to safely remove an affected PG? As per my
> understanding I'm aware about this command.
>
> ===
> #ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph --pgid 1.e4b --op
> remove
> ===
>
> Awaiting for your suggestions to proceed.
If you don'
Re: [ceph-users] deep-scrubbing
+ ceph-devel On Mon, Feb 27, 2017 at 3:54 PM, M Ranga Swami Reddy wrote: > Hello, > I use a ceph cluster and its show the deeps scrub's PG distribution as below > from "ceph pg dump" command: > > >2000 Friday >1000 Saturday >4000 Sunday > == > > On Friday, I have disabled the deep-scrub due to some reason. If this case, > all Friday's PG deep-scrub will be performed on Saturday or is it will be > done on Next Friday? > > Thanks > Swami ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."
... additional strange but a bit different info related to the "permission
denied"
[root s3cmd-master]# ./s3cmd --no-ssl put INSTALL s3://Test/ --expiry-days=5
upload: 'INSTALL' -> 's3://Test/INSTALL' [1 of 1]
3123 of 3123 100% in0s 225.09 kB/s done
[root s3cmd-master]# ./s3cmd info s3://Test/INSTALL
s3://Test/INSTALL (object):
File size: 3123
Last mod: Mon, 03 Apr 2017 12:01:47 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: 63834dbb20b32968505c4ebe768fc8c4
SSE: none
policy:http://s3.amazonaws.com/doc/2006-03-01/";>Test1000falseINSTALL2017-04-03T12:01:47.745Z"63834dbb20b32968505c4ebe768fc8c4"3123STANDARD666First
UserREADME.TXT2017-03-31T22:36:38.380Z"708efc3b9184c8b112e36062804aca1e"88STANDARD666First
User
cors:none
ACL: First User: FULL_CONTROL
x-amz-meta-s3cmd-attrs:
atime:1491218263/ctime:1490998096/gid:0/gname:root/md5:63834dbb20b32968505c4ebe768fc8c4/mode:33188/mtime:1488021707/uid:0/uname:root
[root s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/ --expiry-days=365
ERROR: Access to bucket 'Test' was denied
ERROR: S3 error: 403 (AccessDenied)
[root s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/INSTALL --expiry-days=365
ERROR: Parameter problem: Expecting S3 URI with just the bucket name set
instead of 's3://Test/INSTALL'
[root@mucsds26 s3cmd-master]# ./s3cmd --no-ssl expire s3://Test/
--expiry-days=365
ERROR: Access to bucket 'Test' was denied
ERROR: S3 error: 403 (AccessDenied)
[root s3cmd-master]# ./s3cmd --no-ssl la expire s3://Test
2017-04-03 12:01 3123 s3://Test/INSTALL
2017-03-31 22:3688 s3://Test/README.TXT
Gesendet: Montag, 03. April 2017 um 12:31 Uhr
Von: ceph.nov...@habmalnefrage.de
An: "Ben Hines" , ceph-users
Betreff: Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API
has been added. Note that currently it only supports object expiration."
Hi Cephers...
I did set the "lifecycle" via Cyberduck.I do also get an error first, then
suddenly Cyberduck refreshes the window aand the lifecycle is there.
I see the following when I check it via s3cmd (GitHub master version because
the regular installed version doesn't offer the "getlifecycle" option):
[root s3cmd-master]# ./s3cmd getlifecycle s3://Test/README.txt
http://s3.amazonaws.com/doc/2006-03-01/";>
Cyberduck-nVWEhQwE
Enabled
1
Here is my S3 "user info":
[root ~]# radosgw-admin user info --uid=666
{
"user_id": "666",
"display_name": "First User",
"email": "a...@c.de",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "666",
"access_key": "abc ;)",
"secret_key": "abc def ;)"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw"
}
If someone has a working example how to set lifecycle via the s3cmd, I can try
it and send the outcome...
Gesendet: Montag, 03. April 2017 um 01:43 Uhr
Von: "Ben Hines"
An: "Orit Wasserman"
Cc: ceph-users
Betreff: Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API
has been added. Note that currently it only supports object expiration."
Hmm, Nope, not using tenants feature. The users/buckets were created on prior
ceph versions, perhaps i'll try with a newly created user + bucket.
radosgw-admin user info --uid=foo
{
"user_id": "foo",
"display_name": "foo",
"email": "snip",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
{
"id": "foo:swift",
"permissions": "full-control"
}
],
"keys": [
{
"user": "foo:swift",
"access_key": "xxx",
"secret_key": ""
},
{
"user": "foo",
"access_key": "xxx",
"secret_key": ""
}
],
"swift_keys": [],
"caps": [
{
"type": "buckets",
"perm": "*"
},
{
"type": "metadata",
"perm": "*"
},
{
"type": "usage",
"perm": "*"
},
{
"type": "users",
"perm": "*"
},
{
"type": "zone",
"perm": "*"
}
],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_o
[ceph-users] Space accounting for snapshot objects
Hello, I noticed that when Ceph creates snapshot objects, it detects partial object change and creates an overlap record. This overlapping space is then subtracted from the total space occupied by objects. This makes sense to some extent, however, this is little inconsistent when a file storage backend is used. See the example: Create RBD device: > ./rbd create --size 16M rbd/image01 > dd if=/dev/urandom of=/dev/rbd1 bs=1M count=16 oflag=direct > sudo ./rbd map rbd/image01 --id dummy-id3 Pools stats: > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > rbd 0 16384k 0.0815459M 7 Create snapshot > ./rbd snap create rbd/image01@snap01 So far so good: > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > rbd 0 16384k 0.0815458M 7 Overwrite first object partially (leaving 3/4MiB overlap): > dd if=/dev/zero of=/dev/rbd1 bs=1M count=1 oflag=direct New object is created and only the non-overlapping space is added to USED: > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > rbd 0 17408k 0.0815450M 8 OTOH, we can see that whole object is duplicated when only the first quarter was duplicated: > -rw-r--r-- 1 mkoutny users 4.0M Mar 24 15:22 'osd1/current/0.2_head/rbd\udata.10276b8b4567.__4_3015A682__0' > -rw-r--r-- 1 mkoutny users 4.0M Mar 24 15:22 'osd1/current/0.2_head/rbd\udata.10276b8b4567.__head_3015A682__0' This was done with Ceph version v10.2.3. Is such behavior expected (i.e. it assumes backed will store data efficiently) or is it a bug? Thanks, Michal signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API has been added. Note that currently it only supports object expiration."
Hi Cephers...
I did set the "lifecycle" via Cyberduck.I do also get an error first, then
suddenly Cyberduck refreshes the window aand the lifecycle is there.
I see the following when I check it via s3cmd (GitHub master version because
the regular installed version doesn't offer the "getlifecycle" option):
[root s3cmd-master]# ./s3cmd getlifecycle s3://Test/README.txt
http://s3.amazonaws.com/doc/2006-03-01/";>
Cyberduck-nVWEhQwE
Enabled
1
Here is my S3 "user info":
[root ~]# radosgw-admin user info --uid=666
{
"user_id": "666",
"display_name": "First User",
"email": "a...@c.de",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "666",
"access_key": "abc ;)",
"secret_key": "abc def ;)"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw"
}
If someone has a working example how to set lifecycle via the s3cmd, I can try
it and send the outcome...
Gesendet: Montag, 03. April 2017 um 01:43 Uhr
Von: "Ben Hines"
An: "Orit Wasserman"
Cc: ceph-users
Betreff: Re: [ceph-users] Kraken release and RGW --> "S3 bucket lifecycle API
has been added. Note that currently it only supports object expiration."
Hmm, Nope, not using tenants feature. The users/buckets were created on prior
ceph versions, perhaps i'll try with a newly created user + bucket.
radosgw-admin user info --uid=foo
{
"user_id": "foo",
"display_name": "foo",
"email": "snip",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
{
"id": "foo:swift",
"permissions": "full-control"
}
],
"keys": [
{
"user": "foo:swift",
"access_key": "xxx",
"secret_key": ""
},
{
"user": "foo",
"access_key": "xxx",
"secret_key": ""
}
],
"swift_keys": [],
"caps": [
{
"type": "buckets",
"perm": "*"
},
{
"type": "metadata",
"perm": "*"
},
{
"type": "usage",
"perm": "*"
},
{
"type": "users",
"perm": "*"
},
{
"type": "zone",
"perm": "*"
}
],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "none"
}
On Sun, Apr 2, 2017 at 5:54 AM, Orit Wasserman
mailto:owass...@redhat.com]> wrote:
I see : acct_user=foo, acct_name=foo,
Are you using radosgw with tenants?
If not it could be the problem
Orit
On Sat, Apr 1, 2017 at 7:43 AM, Ben Hines
mailto:bhi...@gmail.com]> wrote:
I'm also trying to use lifecycles (via boto3) but i'm getting permission denied
trying to create the lifecycle. I'm bucket owner with full_control and
WRITE_ACP for good measure. Any ideas?
This is debug ms=20 debug radosgw=20
2017-03-31 21:28:18.382217 7f50d0010700 2 req 8:0.000693:s3:PUT
/bentest:put_lifecycle:verifying op permissions
2017-03-31 21:28:18.38 7f50d0010700 5 Searching permissions for
identity=RGWThirdPartyAccountAuthApplier() ->
RGWLocalAuthApplier(acct_user=foo, acct_name=foo, subuser=, perm_mask=15,
is_admin=) mask=56
2017-03-31 21:28:18.382232 7f50d0010700 5 Searching permissions for uid=foo
2017-03-31 21:28:18.382235 7f50d0010700 5 Found permission: 15
2017-03-31 21:28:18.382237 7f50d0010700 5 Searching permissions for group=1
mask=56
2017-03-31 21:28:18.382297 7f50d0010700 5 Found permission: 3
2017-03-31 21:28:18.382307 7f50d0010700 5 Searching permissions for group=2
mask=56
2017-03-31 21:28:18.382313 7f50d0010700 5 Permissions for group not found
2017-03-31 21:28:18.382318 7f50d0010700 5 Getting permissions
identity=RGWThirdPartyAccountAuthApplier() ->
RGWLocalAuthApplier(acct_user=foo, acct_name=foo, subuser=, perm_mask=15,
is_admin=) owner=foo perm=8
2017-03-31 21:28:18.382325 7f50d0010700 10
identity=RGWThirdPartyAccountAuthApplier() ->
RGWLoca
Re: [ceph-users] deep-scrubbing
I use a ceph cluster and its show the deeps scrub's PG distribution as below from "ceph pg dump" command: 2000 Friday 1000 Saturday 4000 Sunday == On Friday, I have disabled the deep-scrub due to some reason. If this case, all Friday's PG deep-scrub will be performed on Saturday or is it will be done on Next Friday? Thanks Swami On Mon, Feb 27, 2017 at 3:54 PM, M Ranga Swami Reddy wrote: > Hello, > I use a ceph cluster and its show the deeps scrub's PG distribution as > below from "ceph pg dump" command: > > >2000 Friday >1000 Saturday >4000 Sunday > == > > On Friday, I have disabled the deep-scrub due to some reason. If this > case, all Friday's PG deep-scrub will be performed on Saturday or is it > will be done on Next Friday? > > Thanks > Swami > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Flapping OSDs
Hi Vlad Is there anything in syslog on any of the hosts when this happens? Had a similar issue with a single node recently and it was caused by a firmware issue on a single ssd. That would cause the controller to reset and osds on that node would flap as a result. flashed the SSD with new FW and issue hasn't come up since. Brian On Mon, Apr 3, 2017 at 8:03 AM, Vlad Blando wrote: > Most of the time random and most of the time 1 at a time, but I also see > 2-3 that are down at the same time. > > The network seems fine, the bond seems fine, I just don't know where to > look anymore. My other option is to redo the server but that's the last > resort, as much as possible I don't want to. > > > > On Mon, Apr 3, 2017 at 2:24 PM, Maxime Guyot > wrote: > >> Hi Vlad, >> >> >> >> I am curious if those OSDs are flapping all at once? If a single host is >> affected I would consider the network connectivity (bottlenecks and >> misconfigured bonds can generate strange situations), storage controller >> and firmware. >> >> >> >> Cheers, >> >> Maxime >> >> >> >> *From: *ceph-users on behalf of Vlad >> Blando >> *Date: *Sunday 2 April 2017 16:28 >> *To: *ceph-users >> *Subject: *[ceph-users] Flapping OSDs >> >> >> >> Hi, >> >> >> >> One of my ceph nodes have flapping OSDs, network between nodes are fine, >> it's on a 10GBase-T network. I don't see anything wrong with the network, >> but these OSDs are going up/down. >> >> >> >> [root@avatar0-ceph4 ~]# ceph osd tree >> >> # idweight type name up/down reweight >> >> -1 174.7 root default >> >> -2 29.12 host avatar0-ceph2 >> >> 16 3.64osd.16 up 1 >> >> 17 3.64osd.17 up 1 >> >> 18 3.64osd.18 up 1 >> >> 19 3.64osd.19 up 1 >> >> 20 3.64osd.20 up 1 >> >> 21 3.64osd.21 up 1 >> >> 22 3.64osd.22 up 1 >> >> 23 3.64osd.23 up 1 >> >> -3 29.12 host avatar0-ceph0 >> >> 0 3.64osd.0 up 1 >> >> 1 3.64osd.1 up 1 >> >> 2 3.64osd.2 up 1 >> >> 3 3.64osd.3 up 1 >> >> 4 3.64osd.4 up 1 >> >> 5 3.64osd.5 up 1 >> >> 6 3.64osd.6 up 1 >> >> 7 3.64osd.7 up 1 >> >> -4 29.12 host avatar0-ceph1 >> >> 8 3.64osd.8 up 1 >> >> 9 3.64osd.9 up 1 >> >> 10 3.64osd.10 up 1 >> >> 11 3.64osd.11 up 1 >> >> 12 3.64osd.12 up 1 >> >> 13 3.64osd.13 up 1 >> >> 14 3.64osd.14 up 1 >> >> 15 3.64osd.15 up 1 >> >> -5 29.12 host avatar0-ceph3 >> >> 24 3.64osd.24 up 1 >> >> 25 3.64osd.25 up 1 >> >> 26 3.64osd.26 up 1 >> >> 27 3.64osd.27 up 1 >> >> 28 3.64osd.28 up 1 >> >> 29 3.64osd.29 up 1 >> >> 30 3.64osd.30 up 1 >> >> 31 3.64osd.31 up 1 >> >> -6 29.12 host avatar0-ceph4 >> >> 32 3.64osd.32 up 1 >> >> 33 3.64osd.33 up 1 >> >> 34 3.64osd.34 up 1 >> >> 35 3.64osd.35 up 1 >> >> 36 3.64osd.36 up 1 >> >> 37 3.64osd.37 up 1 >> >> 38 3.64osd.38 up 1 >> >> 39 3.64osd.39 up 1 >> >> -7 29.12 host avatar0-ceph5 >> >> 40 3.64osd.40 up 1 >> >> 41 3.64osd.41 up 1 >> >> 42 3.64osd.42 up 1 >> >> 43 3.64osd.43 up 1 >> >> 44 3.64osd.44 up 1 >> >> 45 3.64osd.45 up 1 >> >> 46 3.64osd.46 up 1 >> >> 47 3.64osd.47 up 1 >> >> [root@avatar0-ceph4 ~]# >> >> >> >> >> >> Here is my ceph.conf >> >> --- >> >> [root@avatar0-ceph4 ~]# cat /etc/ceph/ceph.conf >> >> [global] >> >> fsid = 2f0d1928-2ee5-4731-a259-64c0dc16110a >> >> mon_initial_members = avatar0-ceph0, avatar0-ceph1, avatar0-ceph2 >> >> mon_host = 172.40.40.100,172.40.40.101,172.40.40.102 >> >> auth_cluster_required = cephx >> >> auth_service_required = cephx >> >> auth_client_required = cephx >> >> f
Re: [ceph-users] radosgw leaking objects
> Right. The tool isn't removing objects (yet), because we wanted to > have more confidence in the tool before having it automatically > deleting all the found objects. The process currently is to manually > move these objects to a different backup pool (via rados cp, rados > rm), then when you're confident that no needed data was lost in the > process remove the backup pool. In the future we'll automate that. My problem exactly. I don't have enough confidence in myself to just delete a bunch of random objects... Any idea as to when will be available such tool? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Flapping OSDs
Most of the time random and most of the time 1 at a time, but I also see 2-3 that are down at the same time. The network seems fine, the bond seems fine, I just don't know where to look anymore. My other option is to redo the server but that's the last resort, as much as possible I don't want to. On Mon, Apr 3, 2017 at 2:24 PM, Maxime Guyot wrote: > Hi Vlad, > > > > I am curious if those OSDs are flapping all at once? If a single host is > affected I would consider the network connectivity (bottlenecks and > misconfigured bonds can generate strange situations), storage controller > and firmware. > > > > Cheers, > > Maxime > > > > *From: *ceph-users on behalf of Vlad > Blando > *Date: *Sunday 2 April 2017 16:28 > *To: *ceph-users > *Subject: *[ceph-users] Flapping OSDs > > > > Hi, > > > > One of my ceph nodes have flapping OSDs, network between nodes are fine, > it's on a 10GBase-T network. I don't see anything wrong with the network, > but these OSDs are going up/down. > > > > [root@avatar0-ceph4 ~]# ceph osd tree > > # idweight type name up/down reweight > > -1 174.7 root default > > -2 29.12 host avatar0-ceph2 > > 16 3.64osd.16 up 1 > > 17 3.64osd.17 up 1 > > 18 3.64osd.18 up 1 > > 19 3.64osd.19 up 1 > > 20 3.64osd.20 up 1 > > 21 3.64osd.21 up 1 > > 22 3.64osd.22 up 1 > > 23 3.64osd.23 up 1 > > -3 29.12 host avatar0-ceph0 > > 0 3.64osd.0 up 1 > > 1 3.64osd.1 up 1 > > 2 3.64osd.2 up 1 > > 3 3.64osd.3 up 1 > > 4 3.64osd.4 up 1 > > 5 3.64osd.5 up 1 > > 6 3.64osd.6 up 1 > > 7 3.64osd.7 up 1 > > -4 29.12 host avatar0-ceph1 > > 8 3.64osd.8 up 1 > > 9 3.64osd.9 up 1 > > 10 3.64osd.10 up 1 > > 11 3.64osd.11 up 1 > > 12 3.64osd.12 up 1 > > 13 3.64osd.13 up 1 > > 14 3.64osd.14 up 1 > > 15 3.64osd.15 up 1 > > -5 29.12 host avatar0-ceph3 > > 24 3.64osd.24 up 1 > > 25 3.64osd.25 up 1 > > 26 3.64osd.26 up 1 > > 27 3.64osd.27 up 1 > > 28 3.64osd.28 up 1 > > 29 3.64osd.29 up 1 > > 30 3.64osd.30 up 1 > > 31 3.64osd.31 up 1 > > -6 29.12 host avatar0-ceph4 > > 32 3.64osd.32 up 1 > > 33 3.64osd.33 up 1 > > 34 3.64osd.34 up 1 > > 35 3.64osd.35 up 1 > > 36 3.64osd.36 up 1 > > 37 3.64osd.37 up 1 > > 38 3.64osd.38 up 1 > > 39 3.64osd.39 up 1 > > -7 29.12 host avatar0-ceph5 > > 40 3.64osd.40 up 1 > > 41 3.64osd.41 up 1 > > 42 3.64osd.42 up 1 > > 43 3.64osd.43 up 1 > > 44 3.64osd.44 up 1 > > 45 3.64osd.45 up 1 > > 46 3.64osd.46 up 1 > > 47 3.64osd.47 up 1 > > [root@avatar0-ceph4 ~]# > > > > > > Here is my ceph.conf > > --- > > [root@avatar0-ceph4 ~]# cat /etc/ceph/ceph.conf > > [global] > > fsid = 2f0d1928-2ee5-4731-a259-64c0dc16110a > > mon_initial_members = avatar0-ceph0, avatar0-ceph1, avatar0-ceph2 > > mon_host = 172.40.40.100,172.40.40.101,172.40.40.102 > > auth_cluster_required = cephx > > auth_service_required = cephx > > auth_client_required = cephx > > filestore_xattr_use_omap = true > > osd_pool_default_size = 2 > > osd_pool_default_min_size = 1 > > cluster_network = 172.50.50.0/24 > > public_network = 172.40.40.0/24 > > max_open_files = 131072 > > mon_clock_drift_allowed = .15 > > mon_clock_drift_warn_backoff = 30 > > mon_osd_down_out_interval = 300 > > mon_osd_report_timeout = 300 > > mon_osd_min_down_reporters = 3 > > > > > > [osd] > > filestore_merge_threshold = 40 > > filestore_split_multiple = 8 > > osd_op_threads = 8 > > osd_max_backfills = 1 > > osd_recovery_op_priority = 1 > > osd_recovery_max_active = 1 > > > > [clie
