[ceph-users] Re: Current best practice for migrating from one EC profile to another?

[Zhenshi Zhou]

I'm facing the same issue. My cluster will have an expansion and I wanna
modify the ec profile too. What I can think of is to create a new profile
and a new pool, and then migrate the data from the old pool to the new one.
Finally rename the pools as I can use the new pool just like nothing
happened.
And of course, the access will be shut down while migrating. I'm looking
for a more efficient way, which has no need to stop the clients, to deal
with this.

Janne Johansson  于2020年7月29日周三 上午1:43写道：

> Den tis 28 juli 2020 kl 18:50 skrev David Orman :
>
> > Hi,
> >
> > As we expand our cluster (adding nodes), we'd like to take advantage of
> > better EC profiles enabled by higher server/rack counts. I understand, as
> > Ceph currently exists (15.2.4), there is no way to live-migrate from one
> EC
> > profile to another on an existing pool, for example, from 4+2 to 17+3
> when
> > going from 7 nodes to 21. Is this correct?
> >
>
> ..depends on how brave you are, but mostly it's true.
>
> Mostly people stick to the configured EC size(s) on the pools even if you
> buy
> more nodes, the EC profile doesn't have to be close to the number of nodes,
> it
> can well be 4+2 even with 21 nodes, it will just choose 6 random nodes out
> of
> the 21 for each PG and spread out evenly in your new larger cluster.
>
> Also, 17+3 means 20 nodes are involved in every reasonably sized IO, and
> that
> may not be optimal, both in terms of CPU usage, and of course that if your
> previous
> speed was ok when 6 nodes were needed to reassemble a piece of data, you
> can now do 3 of those 4+2-IOs in parallel if your network allows.
>
> Also, with 17+3 on 21 nodes, then a two node failure means it is
> continually degraded*
> until a host comes back up, but with k+m being < 15 would mean that the
> cluster can
> repair itself onto those ~15 nodes and still be in full swing even if up to
> 6 nodes would
> be lost, however improbable such a failure would be.
>
> *) So the +3 normally means three hosts could fail, but having only 19
> hosts on a 17+3
>would mean it can never reach active+clean for any PG, since there
> aren't 20 hosts to
> use. You wouldn't lose data, but all PGs would be degraded/undersized
> meaning you
> run the whole cluster at 17+2 at best.
>
> --
> May the most significant bit of your life be positive.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: S3 bucket lifecycle not deleting old objects

[Robin H. Johnson]

On Tue, Jul 28, 2020 at 01:28:14PM +, Alex Hussein-Kershaw wrote:
> Hello,
> 
> I have a problem that old versions of S3 objects are not being deleted. Can 
> anyone advise as to why? I'm using Ceph 14.2.9.
How many objects are in the bucket? If it's a lot, then you may run into
RGW's lifecycle performance limitations: listing each bucket is a very
slow operation for lifecycle prior to improvements make in later
versions (Octopus with maybe a backport to Nautilius?)

If the bucket doesn't have a lot of operations, you could try running
the 'radosgw-admin lc process' directly, with debug logging, and see
where it gets bogged down.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm and disk partitions

[Jason Borden]

Hi Robert!

Thanks for answering my question. I take it you're working a lot with Ceph 
these days! On my pre-octopus clusters I did use LVM backed by partitions, but 
I always kind of wondered if it was a good practice or not as it added an 
additional layer and obscures the underlying disk topology. Then on this new 
octopus cluster I wanted to use the new cephadm approach for management and it 
seems to steer you away from using partitions or LVM directly, thus my 
question. I don't really have the option to not use partitions in this 
particular instance. I was merely curious if there was a particular reason that 
cephadm doesn't consider partitions (or LVM) as being "available" devices. All 
the storage in this cluster is the same so no need to split metadata on to 
faster storage in my instance. Anyway, it's good to hear from you. Hope you and 
your family are doing well.

Thanks,
Jason
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Usable space vs. Overhead

[Alan Johnson]

A k=3, m=3 scheme would be 3:3 = 50% , you get to use 4 bytes out of 6 bytes = 
4:6 = 2:3 = 66.6%?

From: David Orman 
Sent: Wednesday, July 29, 2020 2:17 AM
To: Alan Johnson (System) 
Cc: ceph-users 
Subject: Re: [ceph-users] Usable space vs. Overhead

[CAUTION: External Mail]
That's what the formula on the ceph link arrives at, a 2/3 or 66.66% overhead. 
But if a 4 byte object is split into 4x1 byte chunks data (4 bytes total) + 2x 
1 byte chunks parity (2 bytes total), you arrive at 6 bytes, which is 50% more 
than 4 bytes. So 50% overhead, vs. 33.33% overhead as the other formula arrives 
at. I'm curious what I'm missing.

On Tue, Jul 28, 2020 at 3:40 PM Alan Johnson 
mailto:al...@supermicro.com>> wrote:
It would be 4/(4+2) = 4/6 =2/3  or k/(k+m)?

-Original Message-
From: David Orman mailto:orma...@corenode.com>>
Sent: Tuesday, July 28, 2020 9:32 PM
To: ceph-users mailto:ceph-users@ceph.io>>
Subject: [ceph-users] Usable space vs. Overhead

[CAUTION: External Mail]

I'm having a hard time understanding the EC usable space vs. raw.

https://urldefense.com/v3/__https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/__;!!B4Ndrdkg3tRaKVT9!79nn4ZG7ADJCY7JEhJwbPvHUn8dvmzAYz9_z-BUG_7Pe0uUETMW_AwDPmgiU4dc$
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space, but 
that's not lining up with what i'd expect just from k data chunks + m parity 
chunks.

So, for example, k=4, m=2. you'd expect every 4 byte object written would 
consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server 
cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw 
storage.

What am I missing here?
___
ceph-users mailing list -- ceph-users@ceph.io To 
unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Usable space vs. Overhead

[David Orman]

That's what the formula on the ceph link arrives at, a 2/3 or 66.66%
overhead. But if a 4 byte object is split into 4x1 byte chunks data (4
bytes total) + 2x 1 byte chunks parity (2 bytes total), you arrive at 6
bytes, which is 50% more than 4 bytes. So 50% overhead, vs. 33.33% overhead
as the other formula arrives at. I'm curious what I'm missing.

On Tue, Jul 28, 2020 at 3:40 PM Alan Johnson  wrote:

> It would be 4/(4+2) = 4/6 =2/3  or k/(k+m)?
>
> -Original Message-
> From: David Orman 
> Sent: Tuesday, July 28, 2020 9:32 PM
> To: ceph-users 
> Subject: [ceph-users] Usable space vs. Overhead
>
> [CAUTION: External Mail]
>
> I'm having a hard time understanding the EC usable space vs. raw.
>
>
> https://urldefense.com/v3/__https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/__;!!B4Ndrdkg3tRaKVT9!79nn4ZG7ADJCY7JEhJwbPvHUn8dvmzAYz9_z-BUG_7Pe0uUETMW_AwDPmgiU4dc$
> indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space,
> but that's not lining up with what i'd expect just from k data chunks + m
> parity chunks.
>
> So, for example, k=4, m=2. you'd expect every 4 byte object written would
> consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server
> cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw
> storage.
>
> What am I missing here?
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm and disk partitions

[Robert LeBlanc]

Jason,

Using partitions won't get you into trouble, but depending on the
version of Ceph you are using, you may want to leverage LVM instead of
partitions. For our Filestore cluster, we had two partitions on NVMe
to get more performance and it worked fine. I'm using LVM to carve out
NVMe drives for RocksDB and then an NVMe pool for CephFS metadata. The
problem with some of these configurations is that the tools don't set
them up well, so I did the partitioning/LVM setup with a script I
wrote, then just had Ceph use the final partition/LV.

Hope that helps,
Robert LeBlanc

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1

On Sat, Jul 25, 2020 at 8:59 AM Jason Borden  wrote:
>
> Greetings,
>
> I have a question regarding the use of cephadm and disk partitions. I have 
> noticed that the cephadm documentation mentions that a device cannot have 
> partitions to be considered "available" for use. In my situation I don't want 
> to use a device with partitions, but rather a partition itself as an osd. 
> I've noticed that partitions do not show up when using `ceph orch device ls`. 
> I've also noticed that partitions can still be used as osds by running 
> something like `ceph orch daemon add osd node1:/dev/sda4`. My question is 
> should I? Am I going to run into trouble by using a partition for an osd 
> instead of a full device?
>
> Thanks,
> Jason
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Zabbix module Octopus 15.2.3

[Reed Dier]

I'm going to resurrect this thread to throw my hat in the ring as I am having 
this issue as well.

I just moved to 15.2.4 on Ubuntu 18.04/bionic, and Zabbix is 5.0.2.
> $ ceph zabbix config-show
> Error EINVAL: Traceback (most recent call last):
>   File "/usr/share/ceph/mgr/mgr_module.py", line 1167, in _handle_command
> return self.handle_command(inbuf, cmd)
>   File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command
> return 0, json.dumps(self.config, index=4, sort_keys=True), ''
>   File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
> **kw).encode(obj)
> TypeError: __init__() got an unexpected keyword argument 'index'

Which looks to be exactly the same as your error.

I also appear to be seeing some weird issue with the Zabbix template as it 
pertains to the discovery and daemon/pool stats.

I made sure that I pulled the latest XML from the ceph repo here: 
https://github.com/ceph/ceph/blob/master/src/pybind/mgr/zabbix/zabbix_template.xml
 


When I run the discovery command, it looks like it runs ok?
> $ ceph zabbix discovery
> Sending discovery data to Zabbix

And if I pass the --verbose flag I get this at the end

> better match: 1.5 > 0.5: zabbix config-set  
> better match: 1.5 > 1.5: zabbix config-show
> better match: 1.5 > 1.5: zabbix send
> better match: 2.5 > 1.5: zabbix discovery
> bestcmds_sorted:
> [{'flags': 8,
>   'help': 'Discovering Zabbix data',
>   'module': 'mgr',
>   'perm': 'r',
>   'sig': [argdesc(, req=True, name=prefix, 
> n=1, numseen=0, prefix=zabbix),
>   argdesc(, req=True, name=prefix, 
> n=1, numseen=0, prefix=discovery)]}]
> Submitting command:  {'prefix': 'zabbix discovery', 'target': ('mon-mgr', '')}
> submit ['{"prefix": "zabbix discovery", "target": ["mon-mgr", ""]}'] to 
> mon-mgr
> Sending discovery data to Zabbix

The pools get discovered correctly, however the pool % used doesn't work, and 
the R/W iops/BW are off by a magnitude of 10^9 I think.

[fs-metadata] Pool Percent Used
07/27/2020 06:17:22 PM
0%

[fs-metadata] Pool RAW Used
07/28/2020 05:07:22 PM
180 Gb
+2.52 Mb
[fs-metadata] Pool Read bandwidth
07/28/2020 05:07:22 PM
4.3 Tbytes
+560.13 Kbytes
[fs-metadata] Pool Read operations
07/28/2020 05:07:22 PM
460.18 Mops
+170 ops
[fs-metadata] Pool Used
07/28/2020 05:07:22 PM
180 Gb
+2.52 Mb
[fs-metadata] Pool Write bandwidth
07/28/2020 05:07:22 PM
1.43 Tbytes
+337.92 Kbytes
[fs-metadata] Pool Write operations
07/28/2020 05:07:22 PM
62.04 Mops
+99 ops

However, typical patterns are this:
> pool fs-metadata id 16
>   client io 3.9 KiB/s rd, 3.3 KiB/s wr, 2 op/s rd, 0 op/s wr

And Zabbix shows this error for the "Pool Percent Used" Item.
> Value of type "string" is not suitable for value type "Numeric (unsigned)". 
> Value "0.014200450852513313"
So it looks like this should be a float, and thats a pretty easy change.

The other, much bigger, issue I am seeing is with the discovery for OSD's.

It appears like its descending the crush tree and selecting the crush failure 
domains.

For one tree, it is chassis, for another it is host.
And the OSD values it is showing "[osd.-##]" correspond directly to the crush 
ID number.
Table of the Items below.

Ceph OSD discovery: [osd.-18] OSD fill
Triggers 2
ceph.[osd.-18,osd_fill]

90d
365d
Zabbix trapper
Ceph CRUSH [ssd]
Enabled
Ceph OSD discovery: [osd.-18] OSD in

ceph.[osd.-18,in]

90d
365d
Zabbix trapper
Ceph CRUSH [ssd]
Enabled
Ceph OSD discovery: [osd.-18] OSD latency apply

ceph.[osd.-18,osd_latency_apply]

90d
365d
Zabbix trapper
Ceph CRUSH [ssd]
Enabled
Ceph OSD discovery: [osd.-18] OSD latency commit

ceph.[osd.-18,osd_latency_commit]

90d
365d
Zabbix trapper
Ceph CRUSH [ssd]
Enabled
Ceph OSD discovery: [osd.-18] OSD PGs

ceph.[osd.-18,num_pgs]

90d
365d
Zabbix trapper
Ceph CRUSH [ssd]
Enabled
Ceph OSD discovery: [osd.-18] OSD up
Triggers 1
ceph.[osd.-18,up]

90d
365d
Zabbix trapper
Ceph CRUSH [ssd]
Enabled
Ceph OSD discovery: [osd.-55] OSD fill
Triggers 2
ceph.[osd.-55,osd_fill]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD discovery: [osd.-55] OSD in

ceph.[osd.-55,in]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD discovery: [osd.-55] OSD latency apply

ceph.[osd.-55,osd_latency_apply]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD discovery: [osd.-55] OSD latency commit

ceph.[osd.-55,osd_latency_commit]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD discovery: [osd.-55] OSD PGs

ceph.[osd.-55,num_pgs]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD discovery: [osd.-55] OSD up
Triggers 1
ceph.[osd.-55,up]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD discovery: [osd.-56] OSD fill
Triggers 2
ceph.[osd.-56,osd_fill]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD discovery: [osd.-56] OSD in

ceph.[osd.-56,in]

90d
365d
Zabbix trapper
Ceph CRUSH [default]
Enabled
Ceph OSD 

[ceph-users] Re: Usable space vs. Overhead

[Alan Johnson]

It would be 4/(4+2) = 4/6 =2/3  or k/(k+m)?

-Original Message-
From: David Orman  
Sent: Tuesday, July 28, 2020 9:32 PM
To: ceph-users 
Subject: [ceph-users] Usable space vs. Overhead

[CAUTION: External Mail]

I'm having a hard time understanding the EC usable space vs. raw.

https://urldefense.com/v3/__https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/__;!!B4Ndrdkg3tRaKVT9!79nn4ZG7ADJCY7JEhJwbPvHUn8dvmzAYz9_z-BUG_7Pe0uUETMW_AwDPmgiU4dc$
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space, but 
that's not lining up with what i'd expect just from k data chunks + m parity 
chunks.

So, for example, k=4, m=2. you'd expect every 4 byte object written would 
consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server 
cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw 
storage.

What am I missing here?
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Usable space vs. Overhead

[David Orman]

I'm having a hard time understanding the EC usable space vs. raw.

https://ceph.io/geen-categorie/ceph-erasure-coding-overhead-in-a-nutshell/
indicates "nOSD * k / (k+m) * OSD Size" is how you calculate usable space,
but that's not lining up with what i'd expect just from k data chunks + m
parity chunks.

So, for example, k=4, m=2. you'd expect every 4 byte object written would
consume 6 bytes, so 50% overhead. however, the prior formula in a 7 server
cluster, using 4+2 encoding, would indicate 66.67% usable capacity vs. raw
storage.

What am I missing here?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Current best practice for migrating from one EC profile to another?

[Janne Johansson]

Den tis 28 juli 2020 kl 18:50 skrev David Orman :

> Hi,
>
> As we expand our cluster (adding nodes), we'd like to take advantage of
> better EC profiles enabled by higher server/rack counts. I understand, as
> Ceph currently exists (15.2.4), there is no way to live-migrate from one EC
> profile to another on an existing pool, for example, from 4+2 to 17+3 when
> going from 7 nodes to 21. Is this correct?
>

..depends on how brave you are, but mostly it's true.

Mostly people stick to the configured EC size(s) on the pools even if you
buy
more nodes, the EC profile doesn't have to be close to the number of nodes,
it
can well be 4+2 even with 21 nodes, it will just choose 6 random nodes out
of
the 21 for each PG and spread out evenly in your new larger cluster.

Also, 17+3 means 20 nodes are involved in every reasonably sized IO, and
that
may not be optimal, both in terms of CPU usage, and of course that if your
previous
speed was ok when 6 nodes were needed to reassemble a piece of data, you
can now do 3 of those 4+2-IOs in parallel if your network allows.

Also, with 17+3 on 21 nodes, then a two node failure means it is
continually degraded*
until a host comes back up, but with k+m being < 15 would mean that the
cluster can
repair itself onto those ~15 nodes and still be in full swing even if up to
6 nodes would
be lost, however improbable such a failure would be.

*) So the +3 normally means three hosts could fail, but having only 19
hosts on a 17+3
   would mean it can never reach active+clean for any PG, since there
aren't 20 hosts to
use. You wouldn't lose data, but all PGs would be degraded/undersized
meaning you
run the whole cluster at 17+2 at best.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] S3 bucket lifecycle not deleting old objects

[Alex Hussein-Kershaw]

Hello,

I have a problem that old versions of S3 objects are not being deleted. Can 
anyone advise as to why? I'm using Ceph 14.2.9.

I expect old versions of S3 objects to be deleted after 3 days as per my 
lifecycle config on the bucket:

{
"Rules": [
{
"Status": "Enabled",
"Prefix": "",
"NoncurrentVersionExpiration": {
"NoncurrentDays": 3
},
"Expiration": {
"ExpiredObjectDeleteMarker": true
},
"ID": "S3 scsdata bucket: Tidy up old versions"
}
]
}

I have an object with 3 versions below (and is much older than 3 days):

[root@hera hera_sdc] /usr/bin> aws s3api --endpoint http://127.3.3.3:7480 
list-object-versions --bucket hera-scsdata --key 
84/46/2020060508501821902143658709-Subscriber | grep -B 4 -A 6 
84/46/2020060508501821902143658709-Subscriber
"LastModified": "2020-06-05T08:58:19.644Z",
"VersionId": "FUdZIehBu3sgRbNJSmZwj3VHWs1ednH",
"ETag": "\"a18286c50a7323efe58497eb97d6dc9d\"",
"StorageClass": "STANDARD",
"Key": "84/46/2020060508501821902143658709-Subscriber",
"Owner": {
"DisplayName": "hera EAS S3 user",
"ID": "hera"
},
"IsLatest": true,
"Size": 4440
--
"LastModified": "2020-06-05T08:58:17.943Z",
"VersionId": "JVKGMJQS-l7xKQuqdfn4QsEY5WLEosj",
"ETag": "\"87e9953af436b702afb80d457f1d73cb\"",
"StorageClass": "STANDARD",
"Key": "84/46/2020060508501821902143658709-Subscriber",
"Owner": {
"DisplayName": "hera EAS S3 user",
"ID": "hera"
},
"IsLatest": false,
"Size": 4408
--
"LastModified": "2020-06-05T08:50:19.167Z",
"VersionId": "-RSNSCDvGj83f4DZ11s8YZ2KaxT8T.a",
"ETag": "\"a68ec68ce825e009ee9a70cfdae9c794\"",
"StorageClass": "STANDARD",
"Key": "84/46/2020060508501821902143658709-Subscriber",
"Owner": {
"DisplayName": "hera EAS S3 user",
"ID": "hera"
},
"IsLatest": false,
"Size": 4256
--
],
"NextKeyMarker": "85/49/20200604163626B4C712312312302641-Subscriber",
"MaxKeys": 1000,
"Prefix": "",
"KeyMarker": "84/46/2020060508501821902143658709-Subscriber",
"DeleteMarkers": [
{
"Owner": {
"DisplayName": "hera EAS S3 user",
"ID": "hera"
},

So those objects still being present seems to be in conflict with the config I 
have set?

Thanks,
Alex
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Current best practice for migrating from one EC profile to another?

[David Orman]

Hi,

As we expand our cluster (adding nodes), we'd like to take advantage of
better EC profiles enabled by higher server/rack counts. I understand, as
Ceph currently exists (15.2.4), there is no way to live-migrate from one EC
profile to another on an existing pool, for example, from 4+2 to 17+3 when
going from 7 nodes to 21. Is this correct?

How are people accomplishing migrations such as this one (or 4+2 to 9+3,
for example) with minimal disruption to services that utilize RBDs sitting
on top of these pools? I found:
https://ceph.io/geen-categorie/ceph-pool-migration/ , which requires
effectively shutting down access during migration (which is doable, but not
ideal) - and from what I've read - has some potential downsides
(specifically referencing the cppool method).

The pools are all EC for data, and an rbd pool exists for metadata.

Thank you!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: July Ceph Science User Group Virtual Meeting

[Mike Perez]

And here's the recording:

https://youtu.be/m-ogTC8J7Y4

On 7/17/20 5:55 AM, Kevin Hrpcek wrote:

Hey all,

We will be having a Ceph science/research/big cluster call on 
Wednesday July 22nd. If anyone wants to discuss something specific 
they can add it to the pad linked below. If you have questions or 
comments you can contact me.


This is an informal open call of community members mostly from 
hpc/htc/research environments where we discuss whatever is on our 
minds regarding ceph. Updates, outages, features, maintenance, 
etc...there is no set presenter but I do attempt to keep the 
conversation lively.


https://pad.ceph.com/p/Ceph_Science_User_Group_20200722

We try to keep it to an hour or less.

Ceph calendar event details:

July 22, 2020
14:00 UTC
4pm Central European
9am Central US

Description: Main pad for discussions: 
https://pad.ceph.com/p/Ceph_Science_User_Group_Index

Meetings will be recorded and posted to the Ceph Youtube channel.
To join the meeting on a computer or mobile phone: 
https://bluejeans.com/908675367?src=calendarLink

To join from a Red Hat Deskphone or Softphone, dial: 84336.
Connecting directly from a room system?
    1.) Dial: 199.48.152.152 or bjn.vc
    2.) Enter Meeting ID: 908675367
Just want to dial in on your phone?
    1.) Dial one of the following numbers: 408-915-6466 (US)
    See all numbers: https://www.redhat.com/en/conference-numbers
    2.) Enter Meeting ID: 908675367
    3.) Press #
Want to test your video connection? https://bluejeans.com/111


Kevin


--

Mike Perez

He/Him

Ceph Community Manager

Red Hat Los Angeles 

thin...@redhat.com 
M: 1-951-572-2633  IM: IRC Freenode/OFTC: thingee

494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA

@Thingee 
  


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: repeatable crash in librbd1

[Jason Dillaman]

On Tue, Jul 28, 2020 at 11:39 AM Jason Dillaman  wrote:
>
> On Tue, Jul 28, 2020 at 11:19 AM Johannes Naab
>  wrote:
> >
> > On 2020-07-28 15:52, Jason Dillaman wrote:
> > > On Tue, Jul 28, 2020 at 9:44 AM Johannes Naab
> > >  wrote:
> > >>
> > >> On 2020-07-28 14:49, Jason Dillaman wrote:
> >  VM in libvirt with:
> >  
> >  
> >    
> >    
> >  
> >    
> >    
> >  209715200
> >  209715200
> >  5000
> >  5000
> >  314572800
> >  314572800
> >  7500
> >  7500
> >  60
> >  60
> >  60
> >  60
> >    
> >  
> >  
> > 
> >  workload:
> >  
> >  fio --rw=write --name=test --size=10M
> >  timeout 30s fio --rw=write --name=test --size=20G
> >  timeout 3m fio --rw=write --name=test --size=20G --direct=1
> >  timeout 1m fio --rw=randrw --name=test --size=20G --direct=1
> >  timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G 
> >  --direct=1
> >  # the backtraces are then observed while the following command is 
> >  running
> >  fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test 
> >  --size=1G --direct=1
> > >>>
> > >>> I'm not sure I understand this workload. Are you running these 6 "fio"
> > >>> processes sequentially or concurrently? Does it only crash on that
> > >>> last one? Do you have "exclusive-lock" enabled on the image since
> > >>> "--numjobs 8" would cause lots of lock fighting if it was enabled.
> > >>
> > >> The workload is a virtual machine with the above libvirt device
> > >> configuration. Within that virtual machine, the workload is run
> > >> sequentially (as script crash.sh) on the xfs formatted device.
> > >>
> > >> I.e. librbd/ceph should only the one qemu process, which is then running
> > >> the workload.
> > >>
> > >> Only the last fio invocation causes the problems.
> > >> When skipping some (I did not test it exhaustively) of the fio
> > >> invocations, the crash is no longer reliably triggered.
> > >
> > > Hmm, all those crash backtraces are in
> > > "AioCompletion::complete_event_socket", but QEMU does not have any
> > > code that utilizes the event socket notification system. AFAIK, only
> > > the fio librbd engine has integrated with that callback system.
> > >
> >
> >
> > The host is an Ubuntu 20.04 with minor backports in libvirt 
> > (6.0.0-0ubuntu8.1)
> > and qemu (1:4.2-3ubuntu6.3) for specific CPU IDs, and the ceph.com librbd1.
> >
> >
> > Upon further testing, changing the libvirt device configuration to:
> >
> > > 
> >
> > (adding cache='none' amd io='native'), did not yet resurface the crash.
> >
> > Based on my understanding, cache='writeback' and io='thread' are the
> > defaults when not otherwise configured. However, I do not yet fully
> > understand the dependencies between those options.
>
> The "io=native" vs "io=threads" is a no-op for all by file-based IO.

"for all non file-based"

> The "cache" setting will just configure the librbd in-memory cache
> mode (disabled, write-through, or write-back).
>
> > Are the libvirt  and librbd caches distinct, or do
> > they refer to the same cache
> > (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008486.html)?
> >
> >
> > >>> Are all the crashes seg faults? They all seem to hint that the
> > >>> internal ImageCtx instance was destroyed somehow while there was still
> > >>> in-flight IO. If the crashes appeared during the "timeout XYZ fio ..."
> > >>> calls, I would think it's highly likely that "fio" is incorrectly
> > >>> closing the RBD image while there was still in-flight IO via its
> > >>> signal handler.
> > >>
> > >> They are all segfaults of the qemu process, captured on the host system.
> > >> librbd should not see any image open/closing during the workload run
> > >> within the VM.
> > >> The `timeout` is used to approximate the initial (manual) workload
> > >> generation, which caused a crash.
> > >>
> > >
> > >
> >
>
>
> --
> Jason



-- 
Jason
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: repeatable crash in librbd1

[Jason Dillaman]

On Tue, Jul 28, 2020 at 11:19 AM Johannes Naab
 wrote:
>
> On 2020-07-28 15:52, Jason Dillaman wrote:
> > On Tue, Jul 28, 2020 at 9:44 AM Johannes Naab
> >  wrote:
> >>
> >> On 2020-07-28 14:49, Jason Dillaman wrote:
>  VM in libvirt with:
>  
>  
>    
>    
>  
>    
>    
>  209715200
>  209715200
>  5000
>  5000
>  314572800
>  314572800
>  7500
>  7500
>  60
>  60
>  60
>  60
>    
>  
>  
> 
>  workload:
>  
>  fio --rw=write --name=test --size=10M
>  timeout 30s fio --rw=write --name=test --size=20G
>  timeout 3m fio --rw=write --name=test --size=20G --direct=1
>  timeout 1m fio --rw=randrw --name=test --size=20G --direct=1
>  timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1
>  # the backtraces are then observed while the following command is running
>  fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test 
>  --size=1G --direct=1
> >>>
> >>> I'm not sure I understand this workload. Are you running these 6 "fio"
> >>> processes sequentially or concurrently? Does it only crash on that
> >>> last one? Do you have "exclusive-lock" enabled on the image since
> >>> "--numjobs 8" would cause lots of lock fighting if it was enabled.
> >>
> >> The workload is a virtual machine with the above libvirt device
> >> configuration. Within that virtual machine, the workload is run
> >> sequentially (as script crash.sh) on the xfs formatted device.
> >>
> >> I.e. librbd/ceph should only the one qemu process, which is then running
> >> the workload.
> >>
> >> Only the last fio invocation causes the problems.
> >> When skipping some (I did not test it exhaustively) of the fio
> >> invocations, the crash is no longer reliably triggered.
> >
> > Hmm, all those crash backtraces are in
> > "AioCompletion::complete_event_socket", but QEMU does not have any
> > code that utilizes the event socket notification system. AFAIK, only
> > the fio librbd engine has integrated with that callback system.
> >
>
>
> The host is an Ubuntu 20.04 with minor backports in libvirt (6.0.0-0ubuntu8.1)
> and qemu (1:4.2-3ubuntu6.3) for specific CPU IDs, and the ceph.com librbd1.
>
>
> Upon further testing, changing the libvirt device configuration to:
>
> > 
>
> (adding cache='none' amd io='native'), did not yet resurface the crash.
>
> Based on my understanding, cache='writeback' and io='thread' are the
> defaults when not otherwise configured. However, I do not yet fully
> understand the dependencies between those options.

The "io=native" vs "io=threads" is a no-op for all by file-based IO.
The "cache" setting will just configure the librbd in-memory cache
mode (disabled, write-through, or write-back).

> Are the libvirt  and librbd caches distinct, or do
> they refer to the same cache
> (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008486.html)?
>
>
> >>> Are all the crashes seg faults? They all seem to hint that the
> >>> internal ImageCtx instance was destroyed somehow while there was still
> >>> in-flight IO. If the crashes appeared during the "timeout XYZ fio ..."
> >>> calls, I would think it's highly likely that "fio" is incorrectly
> >>> closing the RBD image while there was still in-flight IO via its
> >>> signal handler.
> >>
> >> They are all segfaults of the qemu process, captured on the host system.
> >> librbd should not see any image open/closing during the workload run
> >> within the VM.
> >> The `timeout` is used to approximate the initial (manual) workload
> >> generation, which caused a crash.
> >>
> >
> >
>


-- 
Jason
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: repeatable crash in librbd1

[Johannes Naab]

On 2020-07-28 15:52, Jason Dillaman wrote:
> On Tue, Jul 28, 2020 at 9:44 AM Johannes Naab
>  wrote:
>>
>> On 2020-07-28 14:49, Jason Dillaman wrote:
 VM in libvirt with:
 
 
   
   
 
   
   
 209715200
 209715200
 5000
 5000
 314572800
 314572800
 7500
 7500
 60
 60
 60
 60
   
 
 

 workload:
 
 fio --rw=write --name=test --size=10M
 timeout 30s fio --rw=write --name=test --size=20G
 timeout 3m fio --rw=write --name=test --size=20G --direct=1
 timeout 1m fio --rw=randrw --name=test --size=20G --direct=1
 timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1
 # the backtraces are then observed while the following command is running
 fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test 
 --size=1G --direct=1
>>>
>>> I'm not sure I understand this workload. Are you running these 6 "fio"
>>> processes sequentially or concurrently? Does it only crash on that
>>> last one? Do you have "exclusive-lock" enabled on the image since
>>> "--numjobs 8" would cause lots of lock fighting if it was enabled.
>>
>> The workload is a virtual machine with the above libvirt device
>> configuration. Within that virtual machine, the workload is run
>> sequentially (as script crash.sh) on the xfs formatted device.
>>
>> I.e. librbd/ceph should only the one qemu process, which is then running
>> the workload.
>>
>> Only the last fio invocation causes the problems.
>> When skipping some (I did not test it exhaustively) of the fio
>> invocations, the crash is no longer reliably triggered.
> 
> Hmm, all those crash backtraces are in
> "AioCompletion::complete_event_socket", but QEMU does not have any
> code that utilizes the event socket notification system. AFAIK, only
> the fio librbd engine has integrated with that callback system.
> 


The host is an Ubuntu 20.04 with minor backports in libvirt (6.0.0-0ubuntu8.1)
and qemu (1:4.2-3ubuntu6.3) for specific CPU IDs, and the ceph.com librbd1.


Upon further testing, changing the libvirt device configuration to:

> 

(adding cache='none' amd io='native'), did not yet resurface the crash.

Based on my understanding, cache='writeback' and io='thread' are the
defaults when not otherwise configured. However, I do not yet fully
understand the dependencies between those options.

Are the libvirt  and librbd caches distinct, or do
they refer to the same cache
(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008486.html)?


>>> Are all the crashes seg faults? They all seem to hint that the
>>> internal ImageCtx instance was destroyed somehow while there was still
>>> in-flight IO. If the crashes appeared during the "timeout XYZ fio ..."
>>> calls, I would think it's highly likely that "fio" is incorrectly
>>> closing the RBD image while there was still in-flight IO via its
>>> signal handler.
>>
>> They are all segfaults of the qemu process, captured on the host system.
>> librbd should not see any image open/closing during the workload run
>> within the VM.
>> The `timeout` is used to approximate the initial (manual) workload
>> generation, which caused a crash.
>>
> 
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Weird buckets in a new cluster causing broken dashboard functionality

[Eugen König]

Hi all,

I'm currently experience some strange behavior in our cluster: the dashboards 
object gateway "buckets" submenu is broken and I'm getting 503 errors (however, 
"Users" and "Daemons" work flawlessly). Looking into the mgr log gives me 
following error:

2020-07-24T12:38:12.695+0200 7f42150f3700  0 [dashboard ERROR rest_client] RGW 
REST API failed GET req status: 404
2020-07-24T12:38:12.843+0200 7f42130ef700  0 [dashboard ERROR request] 
[10.1.0.133:38454] [GET] [500] [0.039s] [eugen] [513.0B] 
/api/rgw/bucket/91e22800581543b5be4654f7b9b0c7cc_102020-07-24T12:38:12.843+0200 
7f42130ef700  0 [dashboard ERROR request] [b'{"status": "500 Internal Server 
Error", "detail": "The server encountered an unexpected condition which 
prevented it from fulfilling the request.", "request_id": 
"00a3c78a-d96f-4f8b-b3c4-f24eac99f4a1"} 



   ']
So it took me to the point to look into buckets and I got this list with weird 
bucket names (even with dash, but it's actually not allowed):

root@ceph1-40-10~# radosgw-admin buckets list
[
"12345",
"deployment",
"91e22800581543b5be4654f7b9b0c7cc_5", "91e22800581543b5be4654f7b9b0c7cc_6", 
"91e22800581543b5be4654f7b9b0c7cc_11", 
"91e22800581543b5be4654f7b9b0c7cc_8", "91e22800581543b5be4654f7b9b0c7cc_3", 
"91e22800581543b5be4654f7b9b0c7cc_17",
[...]
]

If I try to perform some operations on these buckets, I get an error:
root@ceph1-40-10~# radosgw-admin bucket rm 
--bucket=91e22800581543b5be4654f7b9b0c7cc_15 --purge-objects
2020-07-28T12:40:05.392+0200 7f500024a080 -1 ERROR: unable to remove bucket(2) 
No such file or directory
I'm able to change the owner and even rename it, but anything else is ending in 
the same error above. This also seems to break the buckets functionality in the 
dashboard.

My question is: what are those buckets, where do they come from and what is 
stored in there? And why one is not able to perform any ops on them?  

Any ideas?

Best,
eugen
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph mgr memory leak

[XuYun]

We have the same mgr memory leak problem. I doubt it’s related to the PID which 
is used to identify peer address.
Maybe you cloud try to set the ‘PidMode’ to ‘host’ in your deployment.

> 2020年7月28日 上午2:44，Frank Ritchie  写道：
> 
> Hi all,
> 
> When running containerized Ceph (Nautilus) is anyone else seeing a
> constant memory leak in the ceph-mgr pod with constant ms_handle_reset
> errors in the logs for the backup mgr instance?
> 
> ---
> 0 client.0 ms_handle_reset on v2:172.29.1.13:6848/1
> 0 client.0 ms_handle_reset on v2:172.29.1.13:6848/1
> 0 client.0 ms_handle_reset on v2:172.29.1.13:6848/1
> ---
> 
> I see a couple of related reports with no activity:
> 
> 
> https://tracker.ceph.com/issues/36471
> https://tracker.ceph.com/issues/40260
> 
> and one related merge that doesn't seem to have corrected the issue:
> 
> https://github.com/ceph/ceph/pull/24233
> 
> thx
> Frank
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: repeatable crash in librbd1

[Jason Dillaman]

On Tue, Jul 28, 2020 at 9:44 AM Johannes Naab
 wrote:
>
> On 2020-07-28 14:49, Jason Dillaman wrote:
> >> VM in libvirt with:
> >> 
> >> 
> >>   
> >>   
> >> 
> >>   
> >>   
> >> 209715200
> >> 209715200
> >> 5000
> >> 5000
> >> 314572800
> >> 314572800
> >> 7500
> >> 7500
> >> 60
> >> 60
> >> 60
> >> 60
> >>   
> >> 
> >> 
> >>
> >> workload:
> >> 
> >> fio --rw=write --name=test --size=10M
> >> timeout 30s fio --rw=write --name=test --size=20G
> >> timeout 3m fio --rw=write --name=test --size=20G --direct=1
> >> timeout 1m fio --rw=randrw --name=test --size=20G --direct=1
> >> timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1
> >> # the backtraces are then observed while the following command is running
> >> fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test 
> >> --size=1G --direct=1
> >
> > I'm not sure I understand this workload. Are you running these 6 "fio"
> > processes sequentially or concurrently? Does it only crash on that
> > last one? Do you have "exclusive-lock" enabled on the image since
> > "--numjobs 8" would cause lots of lock fighting if it was enabled.
>
> The workload is a virtual machine with the above libvirt device
> configuration. Within that virtual machine, the workload is run
> sequentially (as script crash.sh) on the xfs formatted device.
>
> I.e. librbd/ceph should only the one qemu process, which is then running
> the workload.
>
> Only the last fio invocation causes the problems.
> When skipping some (I did not test it exhaustively) of the fio
> invocations, the crash is no longer reliably triggered.

Hmm, all those crash backtraces are in
"AioCompletion::complete_event_socket", but QEMU does not have any
code that utilizes the event socket notification system. AFAIK, only
the fio librbd engine has integrated with that callback system.

> > Are all the crashes seg faults? They all seem to hint that the
> > internal ImageCtx instance was destroyed somehow while there was still
> > in-flight IO. If the crashes appeared during the "timeout XYZ fio ..."
> > calls, I would think it's highly likely that "fio" is incorrectly
> > closing the RBD image while there was still in-flight IO via its
> > signal handler.
>
> They are all segfaults of the qemu process, captured on the host system.
> librbd should not see any image open/closing during the workload run
> within the VM.
> The `timeout` is used to approximate the initial (manual) workload
> generation, which caused a crash.
>


-- 
Jason
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: repeatable crash in librbd1

[Johannes Naab]

On 2020-07-28 14:49, Jason Dillaman wrote:
>> VM in libvirt with:
>> 
>> 
>>   
>>   
>> 
>>   
>>   
>> 209715200
>> 209715200
>> 5000
>> 5000
>> 314572800
>> 314572800
>> 7500
>> 7500
>> 60
>> 60
>> 60
>> 60
>>   
>> 
>> 
>>
>> workload:
>> 
>> fio --rw=write --name=test --size=10M
>> timeout 30s fio --rw=write --name=test --size=20G
>> timeout 3m fio --rw=write --name=test --size=20G --direct=1
>> timeout 1m fio --rw=randrw --name=test --size=20G --direct=1
>> timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1
>> # the backtraces are then observed while the following command is running
>> fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test 
>> --size=1G --direct=1
> 
> I'm not sure I understand this workload. Are you running these 6 "fio"
> processes sequentially or concurrently? Does it only crash on that
> last one? Do you have "exclusive-lock" enabled on the image since
> "--numjobs 8" would cause lots of lock fighting if it was enabled.

The workload is a virtual machine with the above libvirt device
configuration. Within that virtual machine, the workload is run
sequentially (as script crash.sh) on the xfs formatted device.

I.e. librbd/ceph should only the one qemu process, which is then running
the workload.

Only the last fio invocation causes the problems.
When skipping some (I did not test it exhaustively) of the fio
invocations, the crash is no longer reliably triggered.

> Are all the crashes seg faults? They all seem to hint that the
> internal ImageCtx instance was destroyed somehow while there was still
> in-flight IO. If the crashes appeared during the "timeout XYZ fio ..."
> calls, I would think it's highly likely that "fio" is incorrectly
> closing the RBD image while there was still in-flight IO via its
> signal handler.

They are all segfaults of the qemu process, captured on the host system.
librbd should not see any image open/closing during the workload run
within the VM.
The `timeout` is used to approximate the initial (manual) workload
generation, which caused a crash.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] slow ops on one osd makes all my buckets unavailable

[Zhenshi Zhou]

Hi,

My harbor registry uses ceph object storage to save the images. But
I couldn't pull/push images from harbor a few moments ago. Ceph was
in warning health status in the same time.

The cluster just had a warning message said that osd.24 has slow ops.
I check the ceph-osd.24.log, and showed as below:























*2020-07-28 19:01:40.599 7f907a39c700 -1 osd.24 4324 get_health_metrics
reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]
ondisk+write+known_if_redirected e4324)2020-07-28 19:01:41.558 7f907a39c700
-1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is
osd_op(client.166289.0:34144787 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]
ondisk+write+known_if_redirected e4324)2020-07-28 19:01:42.579 7f907a39c700
-1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is
osd_op(client.166289.0:34144787 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]
ondisk+write+known_if_redirected e4324)2020-07-28 19:01:43.566 7f907a39c700
-1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is
osd_op(client.166289.0:34144787 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]
ondisk+write+known_if_redirected e4324)2020-07-28 19:01:44.588 7f907a39c700
-1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is
osd_op(client.166289.0:34144787 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]
ondisk+write+known_if_redirected e4324)2020-07-28 19:01:45.627 7f907a39c700
-1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is
osd_op(client.166289.0:34144787 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]
ondisk+write+known_if_redirected e4324)2020-07-28 19:01:46.674 7f907a39c700
-1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is
osd_op(client.166289.0:34144787 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]
ondisk+write+known_if_redirected e4324)2020-07-28 19:01:47.701 7f907a39c700
-1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is
osd_op(client.166289.0:34144852 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected
e4324)2020-07-28 19:01:48.729 7f907a39c700 -1 osd.24 4324
get_health_metrics reporting 1 slow ops, oldest is
osd_op(client.166289.0:34144852 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected
e4324)2020-07-28 19:01:49.729 7f907a39c700 -1 osd.24 4324
get_health_metrics reporting 2 slow ops, oldest is
osd_op(client.166289.0:34144852 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected
e4324)2020-07-28 19:01:50.889 7f907a39c700 -1 osd.24 4324
get_health_metrics reporting 2 slow ops, oldest is
osd_op(client.166289.0:34144852 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)..*
*..*

*2020-07-28 21:03:35.053 7f907a39c700 -1 osd.24 4324 get_health_metrics
reporting 46 slow ops, oldest is osd_op(client.166298.0:34904067 17.4f
17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call
rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)*


After restarted osd.24, the cluster became health again, so was harbor.
What confuse me is that why my harbor couldn't get data from its bucket
while the log indicated that there was a client block on the other bucket.
I don't think some slow ops on an osd has any bad effect on all buckets.

Any idea is appreciated. Thanks
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: repeatable crash in librbd1

[Jason Dillaman]

On Tue, Jul 28, 2020 at 7:19 AM Johannes Naab
 wrote:
>
> Hi,
>
> we observe crashes in librbd1 on specific workloads in virtual machines
> on Ubuntu 20.04 hosts with librbd1=15.2.4-1focal.
>
> The changes in
> https://github.com/ceph/ceph/commit/50694f790245ca90a3b8a644da7b128a7a148cc6
> could be related, but do not easily apply against v15.2.4.

I don't think that commit is related. That commit fixed an issue in
the python-rbd test suite where the "wait_for_complete_and_cb" API
method could return before the callback was invoked when running with
multiple IO threads (Octopus is locked to a single thread).

> We have collected several backtraces, using a reliable local
> reproducer.
>
> (This is a resent of the message stuck in the moderation queue since 
> yesterday.)
>
> Best regards
> Johannes
>
>
> VM in libvirt with:
> 
> 
>   
>   
> 
>   
>   
> 209715200
> 209715200
> 5000
> 5000
> 314572800
> 314572800
> 7500
> 7500
> 60
> 60
> 60
> 60
>   
> 
> 
>
> workload:
> 
> fio --rw=write --name=test --size=10M
> timeout 30s fio --rw=write --name=test --size=20G
> timeout 3m fio --rw=write --name=test --size=20G --direct=1
> timeout 1m fio --rw=randrw --name=test --size=20G --direct=1
> timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1
> # the backtraces are then observed while the following command is running
> fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test 
> --size=1G --direct=1

I'm not sure I understand this workload. Are you running these 6 "fio"
processes sequentially or concurrently? Does it only crash on that
last one? Do you have "exclusive-lock" enabled on the image since
"--numjobs 8" would cause lots of lock fighting if it was enabled.

Are all the crashes seg faults? They all seem to hint that the
internal ImageCtx instance was destroyed somehow while there was still
in-flight IO. If the crashes appeared during the "timeout XYZ fio ..."
calls, I would think it's highly likely that "fio" is incorrectly
closing the RBD image while there was still in-flight IO via its
signal handler.

> 
>
>
> observed stack traces
>
> three times:
> 
> #0  librbd::io::AioCompletion::complete_event_socket 
> (this=this@entry=0x557f633e9400) at ./src/common/event_socket.h:32
> #1  0x7ffb9740ba34 in 
> librbd::io::AioCompletion::complete_external_callback 
> (this=this@entry=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:262
> #2  0x7ffb9740ce98 in librbd::io::AioCompletion::complete 
> (this=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:104
> #3  0x7ffb9740d1a0 in librbd::io::AioCompletion::complete_request 
> (this=0x557f633f7600, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229
> #4  0x7ffb9742fdca in librbd::io::ReadResult::C_ObjectReadRequest::finish 
> (this=0x7ffb68364a60, r=4096) at ./src/librbd/io/ReadResult.cc:155
> #5  0x7ffb9728334d in Context::complete (this=0x7ffb68364a60, 
> r=) at ./src/include/Context.h:77
> #6  0x7ffb9742c7d9 in 
> librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x7ffb6832bed0, 
> r=) at ./src/librbd/io/ObjectDispatchSpec.cc:32
> #7  0x7ffb9742c735 in 
> librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=, 
> r=) at ./src/librbd/io/ObjectDispatchSpec.cc:23
> #8  0x7ffb9755e942 in librbd::io::ObjectRequest::finish 
> (this=this@entry=0x7ffb683394a0, r=r@entry=0) at ./src/include/Context.h:78
> #9  0x7ffb97562e3b in 
> librbd::io::ObjectReadRequest::handle_read_object 
> (this=0x7ffb683394a0, r=0) at ./src/log/SubsystemMap.h:72
> #10 0x7ffb970f4177 in librados::C_AioComplete::finish 
> (this=0x7ffb68362e30, r=) at 
> ./src/librados/AioCompletionImpl.h:140
> #11 0x7ffb970afd1d in Context::complete (this=0x7ffb68362e30, 
> r=) at ./src/include/Context.h:77
> #12 0x7ffb8765791d in Finisher::finisher_thread_entry 
> (this=0x557f63387a30) at ./src/common/Finisher.cc:66
> #13 0x7ffb9946f609 in start_thread (arg=) at 
> pthread_create.c:477
> #14 0x7ffb99396103 in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
> 
>
> once:
> 
> #0  0x7fe7a972169d in 
> std::atomic  boost::lockfree::allocator > >::node> >::load 
> (__m=std::memory_order_acquire, this=) at 
> /usr/include/c++/9/atomic:250
> #1  boost::lockfree::queue boost::lockfree::allocator > >::do_push 
> (t=, this=) at 
> ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:311
> #2  boost::lockfree::queue boost::lockfree::allocator > >::push (t=, 
> this=) at 
> ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280
> #3  librbd::io::AioCompletion::complete_event_socket 
> (this=this@entry=0x5613db630440) at ./src/librbd/io/AioCompletion.cc:276
> #4  0x7fe7a9721a34 in 
> librbd::io::AioCompletion::complete_external_callback 
> (this=this@entry=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:262
> #5  0x7fe7a9722e98 

[ceph-users] Server error when trying to view this list in browser

[biohazd]

Hi

I often get "Server error - An error occurred while processing your request."  
when tryignt o view this lis in Fireofx, is it a known issues?

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/new
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Push config to all hosts

[Cem Zafer]

Thanks Ricardo for clarification.
Regards.

On Mon, Jul 27, 2020 at 2:50 PM Ricardo Marques  wrote:

> Hi Cem,
>
> Since https://github.com/ceph/ceph/pull/35576 you will be able to tell
> cephadm to keep your `/etc/ceph/ceph.conf` updated in all hosts by runnig:
>
> # ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true
>
> But this feature was not released yet, so you will have to wait for
> v15.2.5.
>
>
> Ricardo Marques
>
> --
> *From:* Cem Zafer 
> *Sent:* Monday, June 29, 2020 6:37 AM
> *To:* ceph-users@ceph.io 
> *Subject:* [ceph-users] Push config to all hosts
>
> Hi,
> What is the best method(s) to push ceph.conf to all hosts in octopus
> (15.x)?
> Thanks.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 6 hosts fail cephadm check (15.2.4)

[Ml Ml]

Hello,

i get:

[WRN] CEPHADM_HOST_CHECK_FAILED: 6 hosts fail cephadm check
host ceph01 failed check: Failed to connect to ceph01 (ceph01).
Check that the host is reachable and accepts connections using the
cephadm SSH key
you may want to run:
> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
> mgr/cephadm/ssh_identity_key) root@ceph01
host ceph02 failed check: Failed to connect to ceph02 (10.10.1.2).
Check that the host is reachable and accepts connections using the
cephadm SSH key
you may want to run:
> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
> mgr/cephadm/ssh_identity_key) root@ceph02
host ceph03 failed check: Failed to connect to ceph03 (10.10.1.3).
Check that the host is reachable and accepts connections using the
cephadm SSH key
you may want to run:
> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
> mgr/cephadm/ssh_identity_key) root@ceph03
host ceph04 failed check: Failed to connect to ceph04 (10.10.1.4).
Check that the host is reachable and accepts connections using the
cephadm SSH key
you may want to run:
> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
> mgr/cephadm/ssh_identity_key) root@ceph04
host ceph05 failed check: Failed to connect to ceph05 (10.10.1.5).
Check that the host is reachable and accepts connections using the
cephadm SSH key
you may want to run:
> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
> mgr/cephadm/ssh_identity_key) root@ceph05
host ceph06 failed check: Failed to connect to ceph06 (10.10.1.6).
Check that the host is reachable and accepts connections using the
cephadm SSH key


on ceph01 i run:
ceph cephadm get-ssh-config > /tmp/ceph.conf
ceph config-key get mgr/cephadm/ssh_identity_key > /tmp/ceph.key
chmod 600 /tmp/ceph.key
ssh -F /tmp/ceph.conf -i /tmp/ceph.key root@ceph01 (which works)

So i can not understand the errors above.

root@ceph01:~# ceph versions
{
"mon": {
"ceph version 15.2.1
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 3
},
"mgr": {
"ceph version 15.2.1
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 3
},
"osd": {
"ceph version 15.2.1
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 56
},
"mds": {
"ceph version 15.2.1
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 1
},
"overall": {
"ceph version 15.2.1
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 63
}
}

root@ceph01:~# dpkg -l |grep ceph
ii  ceph-base   15.2.4-1~bpo10+1
   amd64common ceph daemon libraries and management tools
ii  ceph-common 15.2.4-1~bpo10+1
   amd64common utilities to mount and interact with a ceph
storage cluster
ii  ceph-deploy 2.0.1
   all  Ceph-deploy is an easy to use configuration tool
ii  ceph-fuse   15.2.4-1~bpo10+1
   amd64FUSE-based client for the Ceph distributed file system
ii  ceph-grafana-dashboards 15.2.4-1~bpo10+1
   all  grafana dashboards for the ceph dashboard
ii  ceph-mds15.2.4-1~bpo10+1
   amd64metadata server for the ceph distributed file system
ii  ceph-mgr15.2.4-1~bpo10+1
   amd64manager for the ceph distributed storage system
ii  ceph-mgr-cephadm15.2.4-1~bpo10+1
   all  cephadm orchestrator module for ceph-mgr
ii  ceph-mgr-dashboard  15.2.4-1~bpo10+1
   all  dashboard module for ceph-mgr
ii  ceph-mgr-diskprediction-cloud   15.2.4-1~bpo10+1
   all  diskprediction-cloud module for ceph-mgr
ii  ceph-mgr-diskprediction-local   15.2.4-1~bpo10+1
   all  diskprediction-local module for ceph-mgr
ii  ceph-mgr-k8sevents  15.2.4-1~bpo10+1
   all  kubernetes events module for ceph-mgr
ii  ceph-mgr-modules-core   15.2.4-1~bpo10+1
   all  ceph manager modules which are always enabled
ii  ceph-mon15.2.4-1~bpo10+1
   amd64monitor server for the ceph storage system
ii  ceph-osd15.2.4-1~bpo10+1
   amd64OSD server for the ceph storage system
ii  cephadm 15.2.4-1~bpo10+1
   amd64cephadm utility to bootstrap ceph daemons with systemd
and containers
ii  libcephfs1  10.2.11-2
   amd64Ceph distributed file system client library
ii  libcephfs2  15.2.4-1~bpo10+1
   amd64Ceph distributed file system client library
ii  python-ceph-argparse14.2.8-1
   all  Python 2 utility libraries for Ceph CLI
ii  python3-ceph-argparse   15.2.4-1~bpo10+1
   all  Python 3 utility libraries for Ceph CLI
ii  

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

[Mariusz Gronczewski]

Dnia 2020-07-27, o godz. 21:31:33
"Robin H. Johnson"  napisał(a):

> On Mon, Jul 27, 2020 at 08:02:23PM +0200, Mariusz Gronczewski wrote:
> > Hi,
> > 
> > I've got a problem on Octopus (15.2.3, debian packages) install,
> > bucket S3 index shows a file:
> > 
> > s3cmd ls s3://upvid/255/38355 --recursive
> > 2020-07-27 17:48  50584342
> > 
> > s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4
> > 
> > radosgw-admin bi list also shows it
> > 
> > {
> > "type": "plain",
> > "idx":
> > "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4",
> > "entry": { "name":
> > "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4",
> > "instance": "", "ver": {
> > "pool": 11,
> > "epoch": 853842
> > },
> > "locator": "",
> > "exists": "true",
> > "meta": {
> > "category": 1,
> > "size": 50584342,
> > "mtime": "2020-07-27T17:48:27.203008Z",
> > "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7",
> > "storage_class": "",
> > "owner": "filmweb-app",
> > "owner_display_name": "filmweb app user",
> > "content_type": "",
> > "accounted_size": 50584342,
> > "user_data": "",
> > "appendable": "false"
> > },
> > "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP",
> > "flags": 0,
> > "pending_map": [],
> > "versioned_epoch": 0
> > }
> > },
> > 
> > but trying to download it via curl (I've set permissions to public0
> > only gets me  
> Does the RADOS object for this still exist?
> 
> try:
> radosgw-admin object stat --bucket ... --object
> '255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4'
> 
> If that doesn't return, then the backing object is gone, and you have
> a stale index entry that can be cleaned up in most cases with check
> bucket.
> For cases where that doesn't fix it, my recommended way to fix it is
> write a new 0-byte object to the same name, then delete it.
> 


it does exist:

{
"name":
"255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4",
"size": 50584342, "policy": {
"acl": {...},
"owner": {...}
},
"etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7",
"tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP",
"manifest": {
"objs": [],
"obj_size": 50584342,
"explicit_objs": "false",
"head_size": 0,
"max_head_size": 0,
"prefix":
"255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2",
"rules": [ {
"key": 0,
"val": {
"start_part_num": 1,
"start_ofs": 0,
"part_size": 8388608,
"stripe_max_size": 4194304,
"override_prefix": ""
}
},
{
"key": 50331648,
"val": {
"start_part_num": 7,
"start_ofs": 50331648,
"part_size": 252694,
"stripe_max_size": 4194304,
"override_prefix": ""
}
}
],
"tail_instance": "",
"tail_placement": {
"bucket": {
"name": "upvid",
"marker":
"88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id":
"88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"placement_rule": "default-placement"
},
"begin_iter": {
"part_ofs": 0,
"stripe_ofs": 0,
"ofs": 0,
"stripe_size": 4194304,
"cur_part_id": 1,
"cur_stripe": 0,
"cur_override_prefix": "",
"location": {
"placement_rule": "default-placement",
"obj": {
"bucket": {
"name": "upvid",
"marker":
"88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id":
"88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"key": {
"name":
"255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.1",
"instance": "", "ns": "multipart"
}
},

[ceph-users] repeatable crash in librbd1

[Johannes Naab]

Hi,

we observe crashes in librbd1 on specific workloads in virtual machines
on Ubuntu 20.04 hosts with librbd1=15.2.4-1focal.

The changes in
https://github.com/ceph/ceph/commit/50694f790245ca90a3b8a644da7b128a7a148cc6
could be related, but do not easily apply against v15.2.4.

We have collected several backtraces, using a reliable local
reproducer.

(This is a resent of the message stuck in the moderation queue since yesterday.)

Best regards
Johannes


VM in libvirt with:


  
  

  
  
209715200
209715200
5000
5000
314572800
314572800
7500
7500
60
60
60
60
  



workload:

fio --rw=write --name=test --size=10M
timeout 30s fio --rw=write --name=test --size=20G
timeout 3m fio --rw=write --name=test --size=20G --direct=1
timeout 1m fio --rw=randrw --name=test --size=20G --direct=1
timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1
# the backtraces are then observed while the following command is running
fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test 
--size=1G --direct=1



observed stack traces

three times:

#0  librbd::io::AioCompletion::complete_event_socket 
(this=this@entry=0x557f633e9400) at ./src/common/event_socket.h:32
#1  0x7ffb9740ba34 in librbd::io::AioCompletion::complete_external_callback 
(this=this@entry=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:262
#2  0x7ffb9740ce98 in librbd::io::AioCompletion::complete 
(this=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:104
#3  0x7ffb9740d1a0 in librbd::io::AioCompletion::complete_request 
(this=0x557f633f7600, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229
#4  0x7ffb9742fdca in librbd::io::ReadResult::C_ObjectReadRequest::finish 
(this=0x7ffb68364a60, r=4096) at ./src/librbd/io/ReadResult.cc:155
#5  0x7ffb9728334d in Context::complete (this=0x7ffb68364a60, r=) at ./src/include/Context.h:77
#6  0x7ffb9742c7d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish 
(this=0x7ffb6832bed0, r=) at 
./src/librbd/io/ObjectDispatchSpec.cc:32
#7  0x7ffb9742c735 in 
librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=, 
r=) at ./src/librbd/io/ObjectDispatchSpec.cc:23
#8  0x7ffb9755e942 in librbd::io::ObjectRequest::finish 
(this=this@entry=0x7ffb683394a0, r=r@entry=0) at ./src/include/Context.h:78
#9  0x7ffb97562e3b in 
librbd::io::ObjectReadRequest::handle_read_object 
(this=0x7ffb683394a0, r=0) at ./src/log/SubsystemMap.h:72
#10 0x7ffb970f4177 in librados::C_AioComplete::finish (this=0x7ffb68362e30, 
r=) at ./src/librados/AioCompletionImpl.h:140
#11 0x7ffb970afd1d in Context::complete (this=0x7ffb68362e30, r=) at ./src/include/Context.h:77
#12 0x7ffb8765791d in Finisher::finisher_thread_entry (this=0x557f63387a30) 
at ./src/common/Finisher.cc:66
#13 0x7ffb9946f609 in start_thread (arg=) at 
pthread_create.c:477
#14 0x7ffb99396103 in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95


once:

#0  0x7fe7a972169d in 
std::atomic > >::node> >::load 
(__m=std::memory_order_acquire, this=) at 
/usr/include/c++/9/atomic:250
#1  boost::lockfree::queue > >::do_push 
(t=, this=) at 
./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:311
#2  boost::lockfree::queue > >::push (t=, 
this=) at 
./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280
#3  librbd::io::AioCompletion::complete_event_socket 
(this=this@entry=0x5613db630440) at ./src/librbd/io/AioCompletion.cc:276
#4  0x7fe7a9721a34 in librbd::io::AioCompletion::complete_external_callback 
(this=this@entry=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:262
#5  0x7fe7a9722e98 in librbd::io::AioCompletion::complete 
(this=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:104
#6  0x7fe7a97231a0 in librbd::io::AioCompletion::complete_request 
(this=0x5613dbbfa2c0, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229
#7  0x7fe7a9745dca in librbd::io::ReadResult::C_ObjectReadRequest::finish 
(this=0x5613dc6a5c90, r=4096) at ./src/librbd/io/ReadResult.cc:155
#8  0x7fe7a959934d in Context::complete (this=0x5613dc6a5c90, r=) at ./src/include/Context.h:77
#9  0x7fe7a97427d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish 
(this=0x5613db8bae40, r=) at 
./src/librbd/io/ObjectDispatchSpec.cc:32
#10 0x7fe7a9742735 in 
librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=, 
r=) at ./src/librbd/io/ObjectDispatchSpec.cc:23
#11 0x7fe7a9874942 in librbd::io::ObjectRequest::finish 
(this=this@entry=0x5613dbbfc900, r=r@entry=0) at ./src/include/Context.h:78
#12 0x7fe7a9878e3b in 
librbd::io::ObjectReadRequest::handle_read_object 
(this=0x5613dbbfc900, r=0) at ./src/log/SubsystemMap.h:72
#13 0x7fe7a940a177 in librados::C_AioComplete::finish (this=0x5613db85e6d0, 
r=) at ./src/librados/AioCompletionImpl.h:140
#14 0x7fe7a93c5d1d in Context::complete (this=0x5613db85e6d0, 

[ceph-users] Not able to access radosgw S3 bucket creation with AWS java SDK. Caused by: java.net.UnknownHostException: issue.

[sathvik vutukuri]

Hi All,

radosgw-admin is configured in ceph-deploy, created a few buckets from the
Ceph dashboard, but when accessing through Java AWS S3 code to create a new
bucket i am facing the below issue..

Exception in thread "main" com.amazonaws.SdkClientException: Unable to
execute HTTP request: firstbucket.rgwhost
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
at
com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)
at
com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5950)
at
com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1812)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1772)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1710)
at org.S3.App.main(App.java:71)
Caused by: java.net.UnknownHostException: firstbucket.rgwhost
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at
com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
at
com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy3.connect(Unknown Source)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
... 15 more






-- 
Thanks,
Vutukuri Sathvik,
8197748291.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: please help me fix iSCSI Targets not available

[David Thuong]

i user ceph octopus v15
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error

[Philipp Hocke]

Hi,

did you try following the documentation on that one?

https://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#pgs-inconsistent

Best regards

Philipp

On 7/24/20 12:47 PM, Fabrizio Cuseo wrote:
> Hello, I use ceph with proxmox, release 14.2.9, with bluestore OSD.
>
> I had a problem in a replica 2 pool (i know, is dangerous), with an 
> unexpected clone.
> I have deleted the RBD image that used the damaged object (with his 
> snapshot), and now ceph can't trim and clean.
>
> How can I restore the clean and health status ? 
>
> Regards, Fabrizio
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

-- 
Philipp Hocke 
Leiter Linux Systemadministration 
CM4ALL GmbH
Im Mediapark 6A - 50670 Köln / Cologne
Phone +49-(0)221-6601-0
Fax +49-(0)221-6601-1011
E-Mail: philipp.ho...@cm4all.com
Internet: www.cm4all.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cluster became unresponsive: e5 handle_auth_request failed to assign global_id

[Илья Борисович Волошин]

No, they are stored locally on ESXi data storage on top of hardware RAID5
built with SAS/SATA (different hardware on hosts).

Also, I've tried going back to the snapshot taken just after all monitors
and OSDs were added to cluster. The host boots fine and is working as it
should, however, after the next reboot this problem appears (no changes to
configuration were made).

And another thing - even if docker container for mgr is running and gives
no errors in logs neither inside the container nor on the parent host the
mgr doesn't bind to any ports it should: 6800, 6801 and 8443 for dashboard.
Not sure if it is the reason or the consequence of this problem.


вт, 28 июл. 2020 г. в 11:37, Anthony D'Atri :

> Are your mon DBs on SSDs?
>
> > On Jul 27, 2020, at 7:28 AM, Илья Борисович Волошин <
> i.volos...@simplesolution.pro> wrote:
> >
> > Here are all the active ports on mon1 (with the exception of sshd and
> ntpd):
> >
> > # netstat -npl
> > Proto Recv-Q Send-Q Local Address   Foreign Address State
> >PID/Program name
> > tcp0  0 :3300  0.0.0.0:*   LISTEN
> > 1582/ceph-mon
> > tcp0  0 :6789  0.0.0.0:*
>  LISTEN
> > 1582/ceph-mon
> > tcp6   0  0 :::9093 :::*
> LISTEN
> > 908/alertmanager
> > tcp6   0  0 :::9094 :::*
> LISTEN
> > 908/alertmanager
> > tcp6   0  0 :::9095 :::*
> LISTEN
> > 896/prometheus
> > tcp6   0  0 :::9100 :::*
> LISTEN
> > 906/node_exporter
> > tcp6   0  0 :::3000 :::*
> LISTEN
> > 882/grafana-server
> > udp6   0  0 :::9094 :::*
> > 908/alertmanager
> >
> > I've tried telnet from mon1 host, can connect to 3300 and 6789:
> >
> > # telnet  3300
>
> > Trying ...
> > Connected to .
> > Escape character is '^]'.
> > ceph v2
> >
> > # telnet  6789
> > Trying ...
> > Connected to .
> > Escape character is '^]'.
> > ceph v027QQ
> >
> > 6800 and 6801 refuse connection:
> >
> > # telnet  6800
> > Trying ...
> > telnet: Unable to connect to remote host: Connection refused
> >
> > I don't see any errors in the log related to failures to bind... and all
> > CEPH systemd services are running as far as I can tell:
> >
> > # systemctl list-units -a | grep ceph
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5@alertmanager.mon1.service
> >loadedactive   running   Ceph
> > alertmanager.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5@crash.mon1.service
> > loadedactive   running   Ceph crash.mon1
> > for e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5@grafana.mon1.service
> > loadedactive   running   Ceph
> grafana.mon1
> > for e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5@mgr.mon1.peevkl.service
> >loadedactive   running   Ceph
> > mgr.mon1.peevkl for e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5@mon.mon1.service
> > loadedactive   running   Ceph mon.mon1
> for
> > e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5@node-exporter.mon1.service
> > loadedactive   running   Ceph
> > node-exporter.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5@prometheus.mon1.service
> >loadedactive   running   Ceph
> > prometheus.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  system-ceph\x2de30397f0\x2dcc32\x2d11ea\x2d8c8e\x2d000c29469cd5.slice
> >loadedactive   active
> > system-ceph\x2de30397f0\x2dcc32\x2d11ea\x2d8c8e\x2d000c29469cd5.slice
> >  ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5.target
> > loadedactive   activeCeph cluster
> > e30397f0-cc32-11ea-8c8e-000c29469cd5
> >  ceph.target
> >loadedactive   activeAll Ceph clusters
> > and services
> >
> > Here are currently active docker images:
> >
> > # docker ps
> > CONTAINER IDIMAGECOMMAND
> > CREATED STATUS  PORTS   NAMES
> > dfd8dbeccf1eceph/ceph:v15"/usr/bin/ceph-mgr -…"
> > 41 minutes ago  Up 41 minutes
> > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-mgr.mon1.peevkl
> > 9452d1db7ffbceph/ceph:v15"/usr/bin/ceph-mon -…"
>  3
> > hours ago Up 3 hours
> > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-mon.mon1
> > 703ec4a43824prom/prometheus:v2.18.1  "/bin/prometheus --c…"
>  3
> > hours ago Up 3 hours
> > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-prometheus.mon1
> > d816ec5e645fceph/ceph:v15"/usr/bin/ceph-crash…"
>  3
> > 

[ceph-users] Re: 6 hosts fail cephadm check (15.2.4)

[Sebastian Wagner]

Looks as if your cluster is still running 15.2.1.

Have a look at https://docs.ceph.com/docs/master/cephadm/upgrade/

Am 28.07.20 um 09:57 schrieb Ml Ml:
> Hello,
> 
> i get:
> 
> [WRN] CEPHADM_HOST_CHECK_FAILED: 6 hosts fail cephadm check
> host ceph01 failed check: Failed to connect to ceph01 (ceph01).
> Check that the host is reachable and accepts connections using the
> cephadm SSH key
> you may want to run:
>> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
>> mgr/cephadm/ssh_identity_key) root@ceph01
> host ceph02 failed check: Failed to connect to ceph02 (10.10.1.2).
> Check that the host is reachable and accepts connections using the
> cephadm SSH key
> you may want to run:
>> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
>> mgr/cephadm/ssh_identity_key) root@ceph02
> host ceph03 failed check: Failed to connect to ceph03 (10.10.1.3).
> Check that the host is reachable and accepts connections using the
> cephadm SSH key
> you may want to run:
>> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
>> mgr/cephadm/ssh_identity_key) root@ceph03
> host ceph04 failed check: Failed to connect to ceph04 (10.10.1.4).
> Check that the host is reachable and accepts connections using the
> cephadm SSH key
> you may want to run:
>> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
>> mgr/cephadm/ssh_identity_key) root@ceph04
> host ceph05 failed check: Failed to connect to ceph05 (10.10.1.5).
> Check that the host is reachable and accepts connections using the
> cephadm SSH key
> you may want to run:
>> ssh -F =(ceph cephadm get-ssh-config) -i =(ceph config-key get 
>> mgr/cephadm/ssh_identity_key) root@ceph05
> host ceph06 failed check: Failed to connect to ceph06 (10.10.1.6).
> Check that the host is reachable and accepts connections using the
> cephadm SSH key
> 
> 
> on ceph01 i run:
> ceph cephadm get-ssh-config > /tmp/ceph.conf
> ceph config-key get mgr/cephadm/ssh_identity_key > /tmp/ceph.key
> chmod 600 /tmp/ceph.key
> ssh -F /tmp/ceph.conf -i /tmp/ceph.key root@ceph01 (which works)
> 
> So i can not understand the errors above.
> 
> root@ceph01:~# ceph versions
> {
> "mon": {
> "ceph version 15.2.1
> (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 3
> },
> "mgr": {
> "ceph version 15.2.1
> (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 3
> },
> "osd": {
> "ceph version 15.2.1
> (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 56
> },
> "mds": {
> "ceph version 15.2.1
> (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 1
> },
> "overall": {
> "ceph version 15.2.1
> (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 63
> }
> }
> 
> root@ceph01:~# dpkg -l |grep ceph
> ii  ceph-base   15.2.4-1~bpo10+1
>amd64common ceph daemon libraries and management tools
> ii  ceph-common 15.2.4-1~bpo10+1
>amd64common utilities to mount and interact with a ceph
> storage cluster
> ii  ceph-deploy 2.0.1
>all  Ceph-deploy is an easy to use configuration tool
> ii  ceph-fuse   15.2.4-1~bpo10+1
>amd64FUSE-based client for the Ceph distributed file system
> ii  ceph-grafana-dashboards 15.2.4-1~bpo10+1
>all  grafana dashboards for the ceph dashboard
> ii  ceph-mds15.2.4-1~bpo10+1
>amd64metadata server for the ceph distributed file system
> ii  ceph-mgr15.2.4-1~bpo10+1
>amd64manager for the ceph distributed storage system
> ii  ceph-mgr-cephadm15.2.4-1~bpo10+1
>all  cephadm orchestrator module for ceph-mgr
> ii  ceph-mgr-dashboard  15.2.4-1~bpo10+1
>all  dashboard module for ceph-mgr
> ii  ceph-mgr-diskprediction-cloud   15.2.4-1~bpo10+1
>all  diskprediction-cloud module for ceph-mgr
> ii  ceph-mgr-diskprediction-local   15.2.4-1~bpo10+1
>all  diskprediction-local module for ceph-mgr
> ii  ceph-mgr-k8sevents  15.2.4-1~bpo10+1
>all  kubernetes events module for ceph-mgr
> ii  ceph-mgr-modules-core   15.2.4-1~bpo10+1
>all  ceph manager modules which are always enabled
> ii  ceph-mon15.2.4-1~bpo10+1
>amd64monitor server for the ceph storage system
> ii  ceph-osd15.2.4-1~bpo10+1
>amd64OSD server for the ceph storage system
> ii  cephadm 15.2.4-1~bpo10+1
>amd64cephadm utility to bootstrap ceph daemons with systemd
> and containers
> ii  libcephfs1  10.2.11-2
>amd64Ceph distributed file system client