[ceph-users] ceph df reports incorrect stats

[Frank Schilder]

Dear fellow cephers,

we got a problem with ceph df: ceph df reports incorrect USED. It would be 
great if someone could look at this, if a ceph operator doesn't discover this 
issue, they might run out of space without noticing.

This has been reported before but didn't get much attention:

https://www.spinics.net/lists/ceph-users/msg74602.html
https://www.spinics.net/lists/ceph-users/msg74630.html

The symptom: STORED=USED in output of ceph df. All reports I know of are for 
octopus clusters, but I suspect newer versions are affected as well. I don't 
have a reproducer yet (still lacking a test cluster).

Here is a correct usage report:

==> logs/health_231203.log <==
--- RAW STORAGE ---
CLASS SIZE AVAILUSED RAW USED  %RAW USED
hdd13 PiB  7.8 PiB  4.8 PiB   4.8 PiB  38.29
 
--- POOLS ---
POOL   ID  PGS   STORED   OBJECTS  USED %USED  MAX AVAIL
con-fs2-data   14  2048  1.1 PiB  402.93M  1.2 PiB  20.953.7 PiB
con-fs2-data2  19  8192  2.7 PiB1.10G  3.4 PiB  42.783.3 PiB


Here is an incorrect one:

==> logs/health_231204.log <==
--- RAW STORAGE ---
CLASS SIZE AVAILUSED RAW USED  %RAW USED
hdd13 PiB  7.8 PiB  4.8 PiB   4.8 PiB  38.06
 
--- POOLS ---
POOL   ID  PGS   STORED   OBJECTS  USED %USED  MAX AVAIL
con-fs2-data   14  2048  1.1 PiB  402.93M  1.1 PiB  18.823.6 PiB
con-fs2-data2  19  8192  2.7 PiB1.10G  2.7 PiB  37.093.3 PiB


That the first report is correct and not the second is supported by the output 
of ceph osd df tree, showing a use of 4.6PB in alignment with the first output 
of ceph df. Note that the date of the ceph osd df tree output is identical to 
the date of the incorrect ceph df output, hence, ceph osd df tree is *not* 
affected by this issue:

==> ceph osd df tree 231204 <===
SIZE  RAW USE  DATA OMAP META AVAILNAME 

  12 PiB  4.6 PiB  4.6 PiB  2.2 TiB   19 TiB  7.5 PiB  datacenter 
ContainerSquare
   0 B  0 B  0 B  0 B  0 B  0 B  room CON-161-A
  12 PiB  4.6 PiB  4.6 PiB  2.2 TiB   19 TiB  7.5 PiB  room CON-161-A1  
 


In our case, the problem showed up out of nowhere. Here the log snippet for the 
time window within which the flip happened (compare the lines for 
con-fs2-data?-pools):

==> logs/health_231203.log <==
ceph status/df/pool stats/health detail at 16:30:03:
  cluster:
health: HEALTH_OK
 
  services:
mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M)
mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02
mds: con-fs2:8 4 up:standby 8 up:active
osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w)
 
  task status:
 
  data:
pools:   14 pools, 25065 pgs
objects: 2.23G objects, 4.0 PiB
usage:   5.0 PiB used, 8.1 PiB / 13 PiB avail
pgs: 25035 active+clean
 29active+clean+scrubbing+deep
 1 active+clean+scrubbing
 
  io:
client:   215 MiB/s rd, 140 MiB/s wr, 2.34k op/s rd, 1.89k op/s wr
 
--- RAW STORAGE ---
CLASS SIZE AVAILUSED RAW USED  %RAW USED
fs_meta51 TiB   45 TiB  831 GiB   6.0 TiB  11.84
hdd13 PiB  7.8 PiB  4.8 PiB   4.8 PiB  38.08
rbd_data  283 TiB  171 TiB  111 TiB   112 TiB  39.44
rbd_perf   42 TiB   22 TiB   20 TiB20 TiB  48.60
TOTAL  13 PiB  8.1 PiB  4.9 PiB   5.0 PiB  38.04
 
--- POOLS ---
POOL   ID  PGS   STORED   OBJECTS  USED %USED  MAX AVAIL
sr-rbd-meta-one 1   128   13 GiB   16.57k   38 GiB   0.03 39 TiB
sr-rbd-data-one 2  4096  121 TiB   32.06M  108 TiB  48.08 88 TiB
sr-rbd-one-stretch  3   160  262 GiB   68.81k  573 GiB   0.48 39 TiB
con-rbd-meta-hpc-one750   12 KiB   45  372 KiB  09.2 TiB
con-rbd-data-hpc-one8   150   24 GiB6.10k   24 GiB  03.6 PiB
sr-rbd-data-one-hdd11  1024  137 TiB   35.95M  193 TiB  46.57166 TiB
con-fs2-meta1  12   512  554 GiB   76.76M  2.2 TiB   7.266.9 TiB
con-fs2-meta2  13  4096  0 B  574.23M  0 B  06.9 TiB
con-fs2-data   14  2048  1.1 PiB  402.93M  1.2 PiB  21.093.6 PiB
con-fs2-data-ec-ssd17   256  700 GiB7.27M  706 GiB   2.44 22 TiB
ms-rbd-one 18   256  805 GiB  210.92k  1.4 TiB   1.18 39 TiB
con-fs2-data2  19  8192  2.7 PiB1.10G  3.4 PiB  42.963.3 PiB
sr-rbd-data-one-perf   20  4096  6.8 TiB1.81M   20 TiB  57.095.1 TiB
device_health_metrics  21 1  1.4 GiB1.11k  4.2 GiB  0 39 TiB


ceph status/df/pool stats/health detail at 16:30:10:
  cluster:
health: HEALTH_OK
 
  services:
mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age 3M)
mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03, ceph-02
mds: con-fs2:8 4 up:standby 8 up:active
osd: 1284 osds: 1279 up (since 14h), 1

[ceph-users] About delete old bucket lifecycle

[VÔ VI]

Hi community,

I am have multiple bucket was delete but lifecycle of bucket still exist,
how i can delete it with radosgw-admin, because user can't access to bucket
for delete lifecycle. User for this bucket does not exist.

root@ceph:~# radosgw-admin lc list
[
{
"bucket": ":r30203:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.8",
"started": "Wed, 06 Dec 2023 10:43:55 GMT",
"status": "COMPLETE"
},
{
"bucket": ":r30304:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.13",
"started": "Wed, 06 Dec 2023 10:43:54 GMT",
"status": "COMPLETE"
},
{
"bucket":
":ec3204cam04:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.1",
"started": "Wed, 06 Dec 2023 10:44:30 GMT",
"status": "COMPLETE"
},
{
"bucket": ":r30105:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.5",
"started": "Wed, 06 Dec 2023 10:44:40 GMT",
"status": "COMPLETE"
},
{
"bucket": ":r30303:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.14",
"started": "Wed, 06 Dec 2023 10:44:40 GMT",
"status": "COMPLETE"
},
{
"bucket":
":ec3201cam02:f3fec4b6-a248-4f3f-be75-b8055e61233a.56439.2",
"started": "Wed, 06 Dec 2023 10:43:56 GMT",
"status": "COMPLETE"
},
Thanks to the community.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: About delete old bucket lifecycle

[Matt Benjamin]

Hi,

I think running "lc reshard fix" will fix this.

Matt


On Wed, Dec 6, 2023 at 5:48 AM VÔ VI  wrote:

> Hi community,
>
> I am have multiple bucket was delete but lifecycle of bucket still exist,
> how i can delete it with radosgw-admin, because user can't access to bucket
> for delete lifecycle. User for this bucket does not exist.
>
> root@ceph:~# radosgw-admin lc list
> [
> {
> "bucket": ":r30203:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.8",
> "started": "Wed, 06 Dec 2023 10:43:55 GMT",
> "status": "COMPLETE"
> },
> {
> "bucket": ":r30304:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.13",
> "started": "Wed, 06 Dec 2023 10:43:54 GMT",
> "status": "COMPLETE"
> },
> {
> "bucket":
> ":ec3204cam04:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.1",
> "started": "Wed, 06 Dec 2023 10:44:30 GMT",
> "status": "COMPLETE"
> },
> {
> "bucket": ":r30105:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.5",
> "started": "Wed, 06 Dec 2023 10:44:40 GMT",
> "status": "COMPLETE"
> },
> {
> "bucket": ":r30303:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.14",
> "started": "Wed, 06 Dec 2023 10:44:40 GMT",
> "status": "COMPLETE"
> },
> {
> "bucket":
> ":ec3201cam02:f3fec4b6-a248-4f3f-be75-b8055e61233a.56439.2",
> "started": "Wed, 06 Dec 2023 10:43:56 GMT",
> "status": "COMPLETE"
> },
> Thanks to the community.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>

-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph df reports incorrect stats

[Bailey Allison]

Hey Frank,

+1 to this, we've seen it a few times now. I've attached an output of ceph
df from an internal cluster we have with the same issue.

[root@Cluster1 ~]# ceph df
--- RAW STORAGE ---
CLASS  SIZE AVAILUSED RAW USED  %RAW USED
fast_nvme  596 GiB  595 GiB   50 MiB   1.0 GiB   0.18
hdd653 TiB  648 TiB  5.3 TiB   5.4 TiB   0.82
nvme   1.6 TiB  1.6 TiB  251 MiB   5.2 GiB   0.32
ssd 24 TiB   22 TiB  2.8 TiB   2.8 TiB  11.65
TOTAL  680 TiB  671 TiB  8.1 TiB   8.2 TiB   1.21

--- POOLS ---
POOL   ID  PGS   STORED   OBJECTS  USED %USED  MAX
AVAIL
device_health_metrics   1 1  166 MiB  116  166 MiB  0127
TiB
cephfs_data 3   128  214 GiB  767.78k  214 GiB   0.05127
TiB
cephfs_metadata 432  967 MiB1.19k  967 MiB  06.2
TiB
cephfs_42   5   512  1.4 TiB  530.33k  1.4 TiB   0.36254
TiB
cephfs_53   6  1024  1.7 TiB  435.12k  1.7 TiB   0.43238
TiB
rbd 9  1024  1.6 TiB  446.52k  1.6 TiB   7.999.4
TiB
cephfs_ssd 14   512   10 GiB2.64k   10 GiB   0.056.2
TiB
cephfs_42_ssd  15   512   13 GiB   18.07k   13 GiB  0254
TiB
default.rgw.log2232  3.4 KiB  207  3.4 KiB  0127
TiB
default.rgw.meta   2332  1.1 KiB7  1.1 KiB  0127
TiB
.rgw.root  2432  1.3 KiB4  1.3 KiB  0127
TiB
default.rgw.control2532  0 B8  0 B  0127
TiB
default.rgw.buckets.index  27   256  1.4 MiB   11  1.4 MiB  06.2
TiB
default.rgw.buckets.data   28   256   79 MiB   10.00k   79 MiB  0127
TiB

Regards,

Bailey

> -Original Message-
> From: Frank Schilder 
> Sent: December 6, 2023 5:22 AM
> To: ceph-users@ceph.io
> Subject: [ceph-users] ceph df reports incorrect stats
> 
> Dear fellow cephers,
> 
> we got a problem with ceph df: ceph df reports incorrect USED. It would be
> great if someone could look at this, if a ceph operator doesn't discover
this
> issue, they might run out of space without noticing.
> 
> This has been reported before but didn't get much attention:
> 
> https://www.spinics.net/lists/ceph-users/msg74602.html
> https://www.spinics.net/lists/ceph-users/msg74630.html
> 
> The symptom: STORED=USED in output of ceph df. All reports I know of are
> for octopus clusters, but I suspect newer versions are affected as well. I
don't
> have a reproducer yet (still lacking a test cluster).
> 
> Here is a correct usage report:
> 
> ==> logs/health_231203.log <==
> --- RAW STORAGE ---
> CLASS SIZE AVAILUSED RAW USED  %RAW USED
> hdd13 PiB  7.8 PiB  4.8 PiB   4.8 PiB  38.29
> 
> --- POOLS ---
> POOL   ID  PGS   STORED   OBJECTS  USED %USED  MAX
AVAIL
> con-fs2-data   14  2048  1.1 PiB  402.93M  1.2 PiB  20.953.7
PiB
> con-fs2-data2  19  8192  2.7 PiB1.10G  3.4 PiB  42.783.3
PiB
> 
> 
> Here is an incorrect one:
> 
> ==> logs/health_231204.log <==
> --- RAW STORAGE ---
> CLASS SIZE AVAILUSED RAW USED  %RAW USED
> hdd13 PiB  7.8 PiB  4.8 PiB   4.8 PiB  38.06
> 
> --- POOLS ---
> POOL   ID  PGS   STORED   OBJECTS  USED %USED  MAX
AVAIL
> con-fs2-data   14  2048  1.1 PiB  402.93M  1.1 PiB  18.823.6
PiB
> con-fs2-data2  19  8192  2.7 PiB1.10G  2.7 PiB  37.093.3
PiB
> 
> 
> That the first report is correct and not the second is supported by the
output
> of ceph osd df tree, showing a use of 4.6PB in alignment with the first
output
> of ceph df. Note that the date of the ceph osd df tree output is identical
to
> the date of the incorrect ceph df output, hence, ceph osd df tree is *not*
> affected by this issue:
> 
> ==> ceph osd df tree 231204 <===
> SIZE  RAW USE  DATA OMAP META AVAILNAME
>   12 PiB  4.6 PiB  4.6 PiB  2.2 TiB   19 TiB  7.5 PiB  datacenter
ContainerSquare
>0 B  0 B  0 B  0 B  0 B  0 B  room CON-161-A
>   12 PiB  4.6 PiB  4.6 PiB  2.2 TiB   19 TiB  7.5 PiB  room CON-161-A1
> 
> 
> In our case, the problem showed up out of nowhere. Here the log snippet
> for the time window within which the flip happened (compare the lines for
> con-fs2-data?-pools):
> 
> ==> logs/health_231203.log <==
> ceph status/df/pool stats/health detail at 16:30:03:
>   cluster:
> health: HEALTH_OK
> 
>   services:
> mon: 5 daemons, quorum ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 (age
> 3M)
> mgr: ceph-25(active, since 2M), standbys: ceph-26, ceph-01, ceph-03,
> ceph-02
> mds: con-fs2:8 4 up:standby 8 up:active
> osd: 1284 osds: 1279 up (since 14h), 1279 in (since 2w)
> 
>   task status:
> 
>   data:
> pools:   14 pools, 25065 pgs
> objects: 2.23G objects, 4.0 PiB
> usage:   5.0 PiB used, 8.1 PiB / 13 PiB avail
> pgs: 25035 acti

[ceph-users] Re: EC Profiles & DR

[Patrick Begou]

Le 06/12/2023 à 00:11, Rich Freeman a écrit :

On Tue, Dec 5, 2023 at 6:35 AM Patrick Begou
  wrote:

Ok, so I've misunderstood the meaning of failure domain. If there is no
way to request using 2 osd/node and node as failure domain, with 5 nodes
k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a
raid1  setup. A little bit better than replication in the point of view
of global storage capacity.


I'm not sure what you mean by requesting 2osd/node.  If the failure
domain is set to the host, then by default k/m refer to hosts, and the
PGs will be spread across all OSDs on all hosts, but with any
particular PG only being present on one OSD on each host.  You can get
fancy with device classes and crush rules and such and be more
specific with how they're allocated, but that would be the typical
behavior.

Since k/m refer to hosts, then k+m must be less than or equal to the
number of hosts or you'll have a degraded pool because there won't be
enough hosts to allocate them all.  It won't ever stack them across
multiple OSDs on the same host with that configuration.

k=2,m=2 with min=3 would require at least 4 hosts (k+m), and would
allow you to operate degraded with a single host down, and the PGs
would become inactive but would still be recoverable with two hosts
down.  While strictly speaking only 4 hosts are required, you'd do
better to have more than that since then the cluster can immediately
recover from a loss, assuming you have sufficient space.  As you say
it is no more space-efficient than RAID1 or size=2, and it suffers
write amplification for modifications, but it does allow recovery
after the loss of up to two hosts, and you can operate degraded with
one host down which allows for somewhat high availability.


Hi Rich,

My understood was that k and m were for EC chunks not hosts. 🙁 Of 
course if k and m are hosts the best choice would be k=2 and m=2.


When Christian wrote:
/For example if you run an EC=4+2 profile on 3 hosts you can structure 
your crushmap so that you have 2 chunks per host. This means even if one 
host is down you are still guaranteed to have 4 chunks available./


This is that I had thought before (and using 5 nodes instead of 3 as the 
Christian's example). But it does not match what you explain if k and m 
are nodes.


I'm a little bit confused with crushmap settings.

Patrick
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: EC Profiles & DR

[Curt]

Hi Patrick,

Yes K and M are chunks, but the default crush map is a chunk per host,
which is probably the best way to do it, but I'm no expert. I'm not sure
why you would want to do a crush map with 2 chunks per host and min size 4
as it' s just asking for trouble at some point, in my opinion.  Anyway,
take a look at this post if your interested in doing 2 chunks per host it
will give you an idea of crushmap setup,
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NB3M22GNAC7VNWW7YBVYTH6TBZOYLTWA/
.

Regards,
Curt


On Wed, Dec 6, 2023 at 6:26 PM Patrick Begou <
patrick.be...@univ-grenoble-alpes.fr> wrote:

> Le 06/12/2023 à 00:11, Rich Freeman a écrit :
> > On Tue, Dec 5, 2023 at 6:35 AM Patrick Begou
> >   wrote:
> >> Ok, so I've misunderstood the meaning of failure domain. If there is no
> >> way to request using 2 osd/node and node as failure domain, with 5 nodes
> >> k=3+m=1 is not secure enough and I will have to use k=2+m=2, so like a
> >> raid1  setup. A little bit better than replication in the point of view
> >> of global storage capacity.
> >>
> > I'm not sure what you mean by requesting 2osd/node.  If the failure
> > domain is set to the host, then by default k/m refer to hosts, and the
> > PGs will be spread across all OSDs on all hosts, but with any
> > particular PG only being present on one OSD on each host.  You can get
> > fancy with device classes and crush rules and such and be more
> > specific with how they're allocated, but that would be the typical
> > behavior.
> >
> > Since k/m refer to hosts, then k+m must be less than or equal to the
> > number of hosts or you'll have a degraded pool because there won't be
> > enough hosts to allocate them all.  It won't ever stack them across
> > multiple OSDs on the same host with that configuration.
> >
> > k=2,m=2 with min=3 would require at least 4 hosts (k+m), and would
> > allow you to operate degraded with a single host down, and the PGs
> > would become inactive but would still be recoverable with two hosts
> > down.  While strictly speaking only 4 hosts are required, you'd do
> > better to have more than that since then the cluster can immediately
> > recover from a loss, assuming you have sufficient space.  As you say
> > it is no more space-efficient than RAID1 or size=2, and it suffers
> > write amplification for modifications, but it does allow recovery
> > after the loss of up to two hosts, and you can operate degraded with
> > one host down which allows for somewhat high availability.
> >
> Hi Rich,
>
> My understood was that k and m were for EC chunks not hosts. 🙁 Of
> course if k and m are hosts the best choice would be k=2 and m=2.
>
> When Christian wrote:
> /For example if you run an EC=4+2 profile on 3 hosts you can structure
> your crushmap so that you have 2 chunks per host. This means even if one
> host is down you are still guaranteed to have 4 chunks available./
>
> This is that I had thought before (and using 5 nodes instead of 3 as the
> Christian's example). But it does not match what you explain if k and m
> are nodes.
>
> I'm a little bit confused with crushmap settings.
>
> Patrick
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: EC Profiles & DR

[Frank Schilder]

Hi,

the post linked in the previous message is a good source for different 
approaches.

To provide some first-hand experience, I was operating a pool with a 6+2 EC 
profile on 4 hosts for a while (until we got more hosts) and the "subdivide a 
physical host into 2 crush-buckets" approach is actually working best (I 
basically tried all the approaches described in the linked post and they all 
had pitfalls).

Procedure is more or less:

- add second (logical) host bucket for each physical host by suffixing the host 
name with "-B" (ceph osd crush add-bucket   )
- move half the OSDs per host to this new host bucket (ceph osd crush move 
osd.ID host=HOSTNAME-B)
- make this location persist reboot of the OSDs (ceph config set osd.ID 
crush_location host=HOSTNAME-B")

This will allow you to move OSDs back easily when you get more hosts and can 
afford the recommended 1 shard per host. It will also show which and where OSDs 
are moved to with a simple "ceph config dump | grep crush_location". Bets of 
all, you don't have to fiddle around with crush maps and hope they do what you 
want. Just use failure domain host and you are good. No more than 2 host 
buckets per physical host means no more than 2 shards per physical host with 
default placement rules.

I was operating this set-up with min_size=6 and feeling bad about it due to the 
reduced maintainability (risk of data loss during maintenance). Its not great 
really, but sometimes there is no way around it. I was happy when I got the 
extra hosts.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Curt 
Sent: Wednesday, December 6, 2023 3:56 PM
To: Patrick Begou
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: EC Profiles & DR

Hi Patrick,

Yes K and M are chunks, but the default crush map is a chunk per host,
which is probably the best way to do it, but I'm no expert. I'm not sure
why you would want to do a crush map with 2 chunks per host and min size 4
as it' s just asking for trouble at some point, in my opinion.  Anyway,
take a look at this post if your interested in doing 2 chunks per host it
will give you an idea of crushmap setup,
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NB3M22GNAC7VNWW7YBVYTH6TBZOYLTWA/
.

Regards,
Curt
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CLT Meeting minutes 2023-12-06

[Laura Flores]

Closing the loop (blocked waiting for Neha's input): how are we using Gibba
on a day-to-day basis? Is it only used for checking reef point releases?

   - To be discussed again next week, as Neha had a conflict

[Nizam] http://old.ceph.com/pgcalc is not working anymore, is there any
replacement for pgcalc page in the new ceph site?

   - Last attempt at a fix https://github.com/ceph/ceph.io/issues/265
   - Nizam will take a look; the PR needs some CSS work
   - We may not even need this link due to the autoscaler, etc.
   - Perhaps add some kind of "banner" to the pgcalc page to make it clear
   that users should look to the autoscaler

Update on 18.2.1 bluestore issue?

   - Fix has been raised; also not as serious as initially suspected
   - Fix is in testing

-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CLT Meeting minutes 2023-12-06

[Travis Nielsen]

On Wed, Dec 6, 2023 at 8:26 AM Laura Flores  wrote:

> Closing the loop (blocked waiting for Neha's input): how are we using
> Gibba on a day-to-day basis? Is it only used for checking reef point
> releases?
>
>- To be discussed again next week, as Neha had a conflict
>
> [Nizam] http://old.ceph.com/pgcalc is not working anymore, is there any
> replacement for pgcalc page in the new ceph site?
>
>- Last attempt at a fix https://github.com/ceph/ceph.io/issues/265
>- Nizam will take a look; the PR needs some CSS work
>- We may not even need this link due to the autoscaler, etc.
>- Perhaps add some kind of "banner" to the pgcalc page to make it
>clear that users should look to the autoscaler
>
> Update on 18.2.1 bluestore issue?
>
>- Fix has been raised; also not as serious as initially suspected
>- Fix is in testing
>
> Is this the final blocking issue? Is the release imminent or perhaps
another week or two? Depending on the timeline we would prefer to depend on
it in Rook's next release, thanks.



> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
>
>
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: EC Profiles & DR

[Rich Freeman]

On Wed, Dec 6, 2023 at 9:25 AM Patrick Begou
 wrote:
>
> My understood was that k and m were for EC chunks not hosts. 🙁 Of
> course if k and m are hosts the best choice would be k=2 and m=2.

A few others have already replied - as they said if the failure domain
is set to host then it will put only one chunk per host for each PG.

You can get fancy to alter this behavior (something that hadn't
originally occurred to me), but you'll need to use care to ensure that
your host redundancy is what you want it to be.  The main reason to do
that I would think would either be as an interim configuration, or if
you want to have a different level of disk redundancy than host
redundancy.  There might be some use cases I'm not thinking of, but it
is definitely atypical.

In any case, if you start putting multiple chunks for a PG on a single
host, then you'll have to use care to ensure that you can achieve
whatever goals you have for host failures and high availability.
You'll lose more replicas when a single host goes down.

If anything, I think the more common pattern I've seen is to have even
more distribution of chunks than the host level, such as distributing
them by racks/switches/PDUs/datacenters/etc.

Personally if I had 5 nodes and they were balanced, I'd probably just
run k=2, m=2, or even just size=3, and set the failure domain to host.
I avoid manually editing crush maps but my needs are fairly
straightforward - I'm just using hdd/ssd device classes and every pool
is on one or the other with a host failure domain.  I'm also using
rook which abstracts it a bit further, but it isn't doing anything too
fancy with the actual pools besides mapping them to k8s entities and
running the various daemons.

--
Rich
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: EC Profiles & DR

[Patrick Begou]

Le 06/12/2023 à 16:21, Frank Schilder a écrit :

Hi,

the post linked in the previous message is a good source for different 
approaches.

To provide some first-hand experience, I was operating a pool with a 6+2 EC profile on 4 
hosts for a while (until we got more hosts) and the "subdivide a physical host into 
2 crush-buckets" approach is actually working best (I basically tried all the 
approaches described in the linked post and they all had pitfalls).

Procedure is more or less:

- add second (logical) host bucket for each physical host by suffixing the host name with "-B" 
(ceph osd crush add-bucket   )
- move half the OSDs per host to this new host bucket (ceph osd crush move 
osd.ID host=HOSTNAME-B)
- make this location persist reboot of the OSDs (ceph config set osd.ID 
crush_location host=HOSTNAME-B")

This will allow you to move OSDs back easily when you get more hosts and can afford the 
recommended 1 shard per host. It will also show which and where OSDs are moved to with a 
simple "ceph config dump | grep crush_location". Bets of all, you don't have to 
fiddle around with crush maps and hope they do what you want. Just use failure domain 
host and you are good. No more than 2 host buckets per physical host means no more than 2 
shards per physical host with default placement rules.

I was operating this set-up with min_size=6 and feeling bad about it due to the 
reduced maintainability (risk of data loss during maintenance). Its not great 
really, but sometimes there is no way around it. I was happy when I got the 
extra hosts.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Curt
Sent: Wednesday, December 6, 2023 3:56 PM
To: Patrick Begou
Cc:ceph-users@ceph.io
Subject: [ceph-users] Re: EC Profiles & DR

Hi Patrick,

Yes K and M are chunks, but the default crush map is a chunk per host,
which is probably the best way to do it, but I'm no expert. I'm not sure
why you would want to do a crush map with 2 chunks per host and min size 4
as it' s just asking for trouble at some point, in my opinion.  Anyway,
take a look at this post if your interested in doing 2 chunks per host it
will give you an idea of crushmap setup,
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/NB3M22GNAC7VNWW7YBVYTH6TBZOYLTWA/
.

Regards,
Curt


Thanks all for this details that clarify many things for me.

Rich, yes I'm starting with 5 nodes and 4 HDD/node to set up the first 
Ceph cluster in the laboratory and my goal is to increase this cluster 
(may be up to  10 nodes) and to add storage in the nodes (until 12 OSD 
per node). It is a starting point for capacitif storage connected to my 
two clusters (400 cores + 256 cores).


Thanks Franck for these details, as a newbie Iwould never have thought 
to this strategy. In my mind, this is the best way for starting the 
first setup and moving to a more standard configuration later. I've all 
the template now, just have to dive deeper in the details to build it.


Patrick
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Assistance Needed with Ceph Cluster Slow Ops Issue

[Peter]

Dear all,


I am reaching out regarding an issue with our Ceph cluster that has been 
recurring every six hours. Upon investigating the problem using the "ceph 
daemon dump_historic_slow_ops" command, I observed that the issue appears to be 
related to slow operations, specifically getting stuck at "Waiting for RW 
Locks." The wait times often range from one to two seconds.

Our cluster use SAS SSD disks from Samsung for the storage pool in question. 
While these disks are of high quality and should provide sufficient speed, the 
problem persists. The slow ops occurrence is consistent every six hours.

I would greatly appreciate any insights or suggestions you may have to address 
and resolve this issue. If there are specific optimizations or configurations 
that could improve the situation, please advise.


below are some output:

root@lasas003:~# ceph -v
ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable)


"events": [

{
"event": "initiated",
"time": "2023-12-06T08:34:18.501644-0800",
"duration": 0
},
{
"event": "throttled",
"time": "2023-12-06T08:34:18.501644-0800",
"duration": 3.067e-06
},
{
"event": "header_read",
"time": "2023-12-06T08:34:18.501647-0800",
"duration": 3.5428e-06
},
{
"event": "all_read",
"time": "2023-12-06T08:34:18.501650-0800",
"duration": 9.3397e-07
},
{
"event": "dispatched",
"time": "2023-12-06T08:34:18.501651-0800",
"duration": 3.2832e-06
},
{
"event": "queued_for_pg",
"time": "2023-12-06T08:34:18.501654-0800",
"duration": 1.381993999001
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:19.883648-0800",
"duration": 5.7982e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:19.883654-0800",
"duration": 4.248471164998
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:24.132125-0800",
"duration": 1.0667e-05
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:24.132136-0800",
"duration": 2.159352784002
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:26.291489-0800",
"duration": 3.292e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:26.291492-0800",
"duration": 0.4391816470001
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:26.730674-0800",
"duration": 5.1526e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:26.730679-0800",
"duration": 1.053151686999
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:27.783831-0800",
"duration": 5.1328e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:27.783836-0800",
"duration": 1.232525088
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.016361-0800",
"duration": 3.844e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.016365-0800",
"duration": 0.00513857003
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.021503-0800",
 

[ceph-users] Re: Assistance Needed with Ceph Cluster Slow Ops Issue

[Boris]

Hi Peter,

try to set the cluster to nosnaptrim

If this helps, you might need to upgrade to pacific, because you are hit by the 
pg dups bug. 

See: https://www.clyso.com/blog/how-to-identify-osds-affected-by-pg-dup-bug/


Mit freundlichen Grüßen
 - Boris Behrens

> Am 06.12.2023 um 19:01 schrieb Peter :
> 
> Dear all,
> 
> 
> I am reaching out regarding an issue with our Ceph cluster that has been 
> recurring every six hours. Upon investigating the problem using the "ceph 
> daemon dump_historic_slow_ops" command, I observed that the issue appears to 
> be related to slow operations, specifically getting stuck at "Waiting for RW 
> Locks." The wait times often range from one to two seconds.
> 
> Our cluster use SAS SSD disks from Samsung for the storage pool in question. 
> While these disks are of high quality and should provide sufficient speed, 
> the problem persists. The slow ops occurrence is consistent every six hours.
> 
> I would greatly appreciate any insights or suggestions you may have to 
> address and resolve this issue. If there are specific optimizations or 
> configurations that could improve the situation, please advise.
> 
> 
> below are some output:
> 
> root@lasas003:~# ceph -v
> ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus 
> (stable)
> 
> 
> "events": [
> 
>{
>"event": "initiated",
>"time": "2023-12-06T08:34:18.501644-0800",
>"duration": 0
>},
>{
>"event": "throttled",
>"time": "2023-12-06T08:34:18.501644-0800",
>"duration": 3.067e-06
>},
>{
>"event": "header_read",
>"time": "2023-12-06T08:34:18.501647-0800",
>"duration": 3.5428e-06
>},
>{
>"event": "all_read",
>"time": "2023-12-06T08:34:18.501650-0800",
>"duration": 9.3397e-07
>},
>{
>"event": "dispatched",
>"time": "2023-12-06T08:34:18.501651-0800",
>"duration": 3.2832e-06
>},
>{
>"event": "queued_for_pg",
>"time": "2023-12-06T08:34:18.501654-0800",
>"duration": 1.381993999001
>},
>{
>"event": "reached_pg",
>"time": "2023-12-06T08:34:19.883648-0800",
>"duration": 5.7982e-06
>},
>{
>"event": "waiting for rw locks",
>"time": "2023-12-06T08:34:19.883654-0800",
>"duration": 4.248471164998
>},
>{
>"event": "reached_pg",
>"time": "2023-12-06T08:34:24.132125-0800",
>"duration": 1.0667e-05
>},
>{
>"event": "waiting for rw locks",
>"time": "2023-12-06T08:34:24.132136-0800",
>"duration": 2.159352784002
>},
>{
>"event": "reached_pg",
>"time": "2023-12-06T08:34:26.291489-0800",
>"duration": 3.292e-06
>},
>{
>"event": "waiting for rw locks",
>"time": "2023-12-06T08:34:26.291492-0800",
>"duration": 0.4391816470001
>},
>{
>"event": "reached_pg",
>"time": "2023-12-06T08:34:26.730674-0800",
>"duration": 5.1526e-06
>},
>{
>"event": "waiting for rw locks",
>"time": "2023-12-06T08:34:26.730679-0800",
>"duration": 1.053151686999
>},
>{
>"event": "reached_pg",
>"time": "2023-12-06T08:34:27.783831-0800",
>"duration": 5.1328e-06
>},
>{
>"event": "waiting for rw locks",
>"time": "2023-12-06T08:34:27.783836-0800",
>"duration": 1.232525088
>},
>{
>"event": "reached_pg",
>"time": "2023-12-06T08:34:29.016361-0800",

[ceph-users] Re: Assistance Needed with Ceph Cluster Slow Ops Issue

[Peter]

Thank you for pointing this out. I did check my cluster by using the article 
given command, it over 17 million PG dups over each OSDs.
May I know if the snaptrim activity takes place every six hours?  If I disable 
the snaptrim, will it stop the slow ops temporarily before my performing 
version upgrade?
If I want to upgrade my Ceph, it will take time to analysis the environment. 
Can I have work around quickly for delete OSD then create it again for zeroized 
the log times? or manually delete the OSD log?




From: Boris 
Sent: Wednesday, December 6, 2023 10:13
To: Peter 
Cc: ceph-users@ceph.io 
Subject: Re: [ceph-users] Assistance Needed with Ceph Cluster Slow Ops Issue

Hi Peter,

try to set the cluster to nosnaptrim

If this helps, you might need to upgrade to pacific, because you are hit by the 
pg dups bug.

See: https://www.clyso.com/blog/how-to-identify-osds-affected-by-pg-dup-bug/


Mit freundlichen Grüßen
 - Boris Behrens

Am 06.12.2023 um 19:01 schrieb Peter :

Dear all,


I am reaching out regarding an issue with our Ceph cluster that has been 
recurring every six hours. Upon investigating the problem using the "ceph 
daemon dump_historic_slow_ops" command, I observed that the issue appears to be 
related to slow operations, specifically getting stuck at "Waiting for RW 
Locks." The wait times often range from one to two seconds.

Our cluster use SAS SSD disks from Samsung for the storage pool in question. 
While these disks are of high quality and should provide sufficient speed, the 
problem persists. The slow ops occurrence is consistent every six hours.

I would greatly appreciate any insights or suggestions you may have to address 
and resolve this issue. If there are specific optimizations or configurations 
that could improve the situation, please advise.


below are some output:

root@lasas003:~# ceph -v
ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable)


"events": [

   {
   "event": "initiated",
   "time": "2023-12-06T08:34:18.501644-0800",
   "duration": 0
   },
   {
   "event": "throttled",
   "time": "2023-12-06T08:34:18.501644-0800",
   "duration": 3.067e-06
   },
   {
   "event": "header_read",
   "time": "2023-12-06T08:34:18.501647-0800",
   "duration": 3.5428e-06
   },
   {
   "event": "all_read",
   "time": "2023-12-06T08:34:18.501650-0800",
   "duration": 9.3397e-07
   },
   {
   "event": "dispatched",
   "time": "2023-12-06T08:34:18.501651-0800",
   "duration": 3.2832e-06
   },
   {
   "event": "queued_for_pg",
   "time": "2023-12-06T08:34:18.501654-0800",
   "duration": 1.381993999001
   },
   {
   "event": "reached_pg",
   "time": "2023-12-06T08:34:19.883648-0800",
   "duration": 5.7982e-06
   },
   {
   "event": "waiting for rw locks",
   "time": "2023-12-06T08:34:19.883654-0800",
   "duration": 4.248471164998
   },
   {
   "event": "reached_pg",
   "time": "2023-12-06T08:34:24.132125-0800",
   "duration": 1.0667e-05
   },
   {
   "event": "waiting for rw locks",
   "time": "2023-12-06T08:34:24.132136-0800",
   "duration": 2.159352784002
   },
   {
   "event": "reached_pg",
   "time": "2023-12-06T08:34:26.291489-0800",
   "duration": 3.292e-06
   },
   {
   "event": "waiting for rw locks",
   "time": "2023-12-06T08:34:26.291492-0800",
   "duration": 0.4391816470001
   },
   {
   "event": "reached_pg",
   "time": "2023-12-06T08:34:26.730674-0800",
   "duration": 5.1526e-06
   },
   {
   "event": "waiting for rw locks",
   "time": "2023-12-06T08:34:26.730679-0800",
   "duration": 1.053151686999
   },
   {
   "event": "reached_pg",
  

[ceph-users] Re: Assistance Needed with Ceph Cluster Slow Ops Issue

[Boris]

Snaptrim is the process of removing a snapshot and reclaiming diskspace after 
it got deleted. I don't know how the ceph internals work, but it helped for us. 

You can try to move the snaptrim into specific timeframes and limit it to one 
per osd. Also sleeping (3s worked for us) between the deletion helped to 
alleviate the pressure from the OSDs. Please check the documentation for the 
exact config parameters. 

I don't know if octopus got the tooling to remove the pg log dups. 

Check "ceph-objectstore-tool --help" if the --op parameter accepts 
trim-pg-log-dups

croit made a blog post how to remove these dups: 
https://croit.io/blog/oom-killer-osds

But be warned: Checking an 8TB Samsung SAS SSD with 300pgs (not trimming) took 
4 hours. 
You can skip the listing and counting and just start the trim directly. But the 
OSD needs to be offline for it. 

What we did when we were hit with this issue:
Stop snaptrim, update to pacific, do an offline rocksdb compression before the 
OSDs start after the upgrade, start the OSDs and hate our lives while they 
started, wait a week, slowly start the snaptrim and hope for the best. :-)

Mit freundlichen Grüßen
 - Boris Behrens

> Am 06.12.2023 um 20:17 schrieb Peter :
> 
> 
> Thank you for pointing this out. I did check my cluster by using the article 
> given command, it over 17 million PG dups over each OSDs. 
> May I know if the snaptrim activity takes place every six hours?  If I 
> disable the snaptrim, will it stop the slow ops temporarily before my 
> performing version upgrade?
> If I want to upgrade my Ceph, it will take time to analysis the environment. 
> Can I have work around quickly for delete OSD then create it again for 
> zeroized the log times? or manually delete the OSD log?
> 
> 
> 
> From: Boris 
> Sent: Wednesday, December 6, 2023 10:13
> To: Peter 
> Cc: ceph-users@ceph.io 
> Subject: Re: [ceph-users] Assistance Needed with Ceph Cluster Slow Ops Issue
>  
> Hi Peter,
> 
> try to set the cluster to nosnaptrim
> 
> If this helps, you might need to upgrade to pacific, because you are hit by 
> the pg dups bug. 
> 
> See: https://www.clyso.com/blog/how-to-identify-osds-affected-by-pg-dup-bug/
> 
> 
> Mit freundlichen Grüßen
>  - Boris Behrens
> 
>>> Am 06.12.2023 um 19:01 schrieb Peter :
>>> 
>> Dear all,
>> 
>> 
>> I am reaching out regarding an issue with our Ceph cluster that has been 
>> recurring every six hours. Upon investigating the problem using the "ceph 
>> daemon dump_historic_slow_ops" command, I observed that the issue appears to 
>> be related to slow operations, specifically getting stuck at "Waiting for RW 
>> Locks." The wait times often range from one to two seconds.
>> 
>> Our cluster use SAS SSD disks from Samsung for the storage pool in question. 
>> While these disks are of high quality and should provide sufficient speed, 
>> the problem persists. The slow ops occurrence is consistent every six hours.
>> 
>> I would greatly appreciate any insights or suggestions you may have to 
>> address and resolve this issue. If there are specific optimizations or 
>> configurations that could improve the situation, please advise.
>> 
>> 
>> below are some output:
>> 
>> root@lasas003:~# ceph -v
>> ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus 
>> (stable)
>> 
>> 
>> "events": [
>> 
>>{
>>"event": "initiated",
>>"time": "2023-12-06T08:34:18.501644-0800",
>>"duration": 0
>>},
>>{
>>"event": "throttled",
>>"time": "2023-12-06T08:34:18.501644-0800",
>>"duration": 3.067e-06
>>},
>>{
>>"event": "header_read",
>>"time": "2023-12-06T08:34:18.501647-0800",
>>"duration": 3.5428e-06
>>},
>>{
>>"event": "all_read",
>>"time": "2023-12-06T08:34:18.501650-0800",
>>"duration": 9.3397e-07
>>},
>>{
>>"event": "dispatched",
>>"time": "2023-12-06T08:34:18.501651-0800",
>>"duration": 3.2832e-06
>>},
>>{
>>"event": "queued_for_pg",
>>"time": "2023-12-06T08:34:18.501654-0800",
>>"duration": 1.381993999001
>>},
>>{
>>"event": "reached_pg",
>>"time": "2023-12-06T08:34:19.883648-0800",
>>"duration": 5.7982e-06
>>},
>>{
>>"event": "waiting for rw locks",
>>  

[ceph-users] Re: About delete old bucket lifecycle

[VÔ VI]

Hi Matt,

Thanks Matt but it's not working for me, after run radosgw-admin lc reshard
fix don't change anything

Vào Th 4, 6 thg 12, 2023 vào lúc 20:18 Matt Benjamin 
đã viết:

> Hi,
>
> I think running "lc reshard fix" will fix this.
>
> Matt
>
>
> On Wed, Dec 6, 2023 at 5:48 AM VÔ VI  wrote:
>
>> Hi community,
>>
>> I am have multiple bucket was delete but lifecycle of bucket still exist,
>> how i can delete it with radosgw-admin, because user can't access to
>> bucket
>> for delete lifecycle. User for this bucket does not exist.
>>
>> root@ceph:~# radosgw-admin lc list
>> [
>> {
>> "bucket": ":r30203:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.8",
>> "started": "Wed, 06 Dec 2023 10:43:55 GMT",
>> "status": "COMPLETE"
>> },
>> {
>> "bucket": ":r30304:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.13",
>> "started": "Wed, 06 Dec 2023 10:43:54 GMT",
>> "status": "COMPLETE"
>> },
>> {
>> "bucket":
>> ":ec3204cam04:f3fec4b6-a248-4f3f-be75-b8055e61233a.31736.1",
>> "started": "Wed, 06 Dec 2023 10:44:30 GMT",
>> "status": "COMPLETE"
>> },
>> {
>> "bucket": ":r30105:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.5",
>> "started": "Wed, 06 Dec 2023 10:44:40 GMT",
>> "status": "COMPLETE"
>> },
>> {
>> "bucket": ":r30303:f3fec4b6-a248-4f3f-be75-b8055e61233a.33081.14",
>> "started": "Wed, 06 Dec 2023 10:44:40 GMT",
>> "status": "COMPLETE"
>> },
>> {
>> "bucket":
>> ":ec3201cam02:f3fec4b6-a248-4f3f-be75-b8055e61233a.56439.2",
>> "started": "Wed, 06 Dec 2023 10:43:56 GMT",
>> "status": "COMPLETE"
>> },
>> Thanks to the community.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
>
> --
>
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-821-5101
> fax.  734-769-8938
> cel.  734-216-5309
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph 17.2.7 to 18.2.0 issues

[Eugen Block]

Hi, did you unmount your clients after the cluster poweroff? You could  
also enable debug logs in mds to see more information. Are there any  
blocked requests? You can query the mds daemon via cephadm shell or  
with ad admin keyring like this:


# ceph tell mds.cephfs.storage.lgmyqv dump_blocked_ops

Regards,
Eugen


Zitat von pclark6...@outlook.com:


Hi All,

Recently I upgraded my cluster from Quincy to Reef. Everything  
appeared to go smoothly and without any issues arising.
I was forced to poweroff the cluster, performing the ususal  
procedures beforehand and everything appears to have come back fine.  
Every service reports green across the board except


If i try to copy any files from a cephfs mountpoint whether kernel  
or fuse the actual copy will hang. ls/stat etc all work which  
indicates metadata appears fine but copying always hangs.


I can copy objects direct using the rados toolset which indicates  
the underlying data exists.


The system itself reports no errors and thinks its healthy.

The entire cluster and cephfs clients are all Rocky9.

Any advice would be much appreciatd. I'd find this easier to deal  
with if the cluster actually gave me an error

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph 17.2.7 to 18.2.0 issues

[Venky Shankar]

On Thu, Dec 7, 2023 at 12:49 PM Eugen Block  wrote:
>
> Hi, did you unmount your clients after the cluster poweroff?

If this is the case, then a remount would kick things back working.

> You could
> also enable debug logs in mds to see more information. Are there any
> blocked requests? You can query the mds daemon via cephadm shell or
> with ad admin keyring like this:
>
> # ceph tell mds.cephfs.storage.lgmyqv dump_blocked_ops
>
> Regards,
> Eugen
>
>
> Zitat von pclark6...@outlook.com:
>
> > Hi All,
> >
> > Recently I upgraded my cluster from Quincy to Reef. Everything
> > appeared to go smoothly and without any issues arising.
> > I was forced to poweroff the cluster, performing the ususal
> > procedures beforehand and everything appears to have come back fine.
> > Every service reports green across the board except
> >
> > If i try to copy any files from a cephfs mountpoint whether kernel
> > or fuse the actual copy will hang. ls/stat etc all work which
> > indicates metadata appears fine but copying always hangs.
> >
> > I can copy objects direct using the rados toolset which indicates
> > the underlying data exists.
> >
> > The system itself reports no errors and thinks its healthy.
> >
> > The entire cluster and cephfs clients are all Rocky9.
> >
> > Any advice would be much appreciatd. I'd find this easier to deal
> > with if the cluster actually gave me an error
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io