[ceph-users] Re: ceph orch upgrade to 18.2.1 seems stuck on MDS?

[Nigel Williams]

On Wed, 7 Feb 2024 at 20:00, Nigel Williams 
wrote:

>
> and just MDS left to do but upgrade has been sitting for hours on this
>
> resolved by rebooting a single host...still not sure why this fixed it
other than it had a standby MDS that would not upgrade?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Snapshot automation/scheduling for rbd?

[Jayanth Reddy]

Right. IIUC, disk snapshots are disabled in the global settings and I believe 
they also warn you that it can not produce crash-consistent snapshots. I 
believe the snapshots can be taken like but not sure if a pause or fs freeze is 
involved.

AFAIK, you'll have to initiate snapshot for each volume regardless root or data.

I've not tried with full VM snapshot which might be helpful in this scenario, 
but I think it doesn't work either due to the limitations imposed. To track a 
block device, there is a path parameter in every volume detail on CloudStack 
similar to this "e93016ae-e572-4b15-8df7-a08c787568d4". This should directly 
point to /. If you have access 
to the Ceph dashboard, you should be able to view those snapshots under the 
block device in an easier way or you may also use rbd cli.

As a general note, bypassing the Cloud orchestrator and taking snapshots 
manually using Ceph client tools will likely cause disruption in the cloud 
orchestrator workflow. For example, if you take snapshots that are unmanaged by 
cloudstack and try to perform CRUD op on the storage blockdev in WebUI or API, 
you'll see unexpected errors.

Thanks


From: Jeremy Hansen 
Sent: Monday, February 5, 2024 10:29:06 pm
To: ceph-users@ceph.io ; Jayanth Reddy 

Subject: Re: [ceph-users] Re: Snapshot automation/scheduling for rbd?

Thanks. I think the only issue with doing snapshots via Cloudstack is 
potentially having to pause an instance for an extended period of time. I 
haven’t tested this yet but based on the docs, I think kvm has to be paused 
regardless.

What about added volumes?  Does an instance have to pause of you’re only 
snapshotting added volumes and not the root disk?

Couple of questions.  If I snapshot an rbd image from the ceph side, does that 
require an instance pause and is there a graceful way, perhaps through the api 
to do the full mapping of instance volumes -> Ceph block image named?  So I can 
understand what block images belong to which Cloudstack instance. I never 
understood how to properly trace a volume from instance to Ceph image.

Thanks!



On Saturday, Feb 03, 2024 at 10:47 AM, Jayanth Reddy 
mailto:jayanthreddy5...@gmail.com>> wrote:
Hi,
For CloudStack with RBD, you should be able to control the snapshot placement 
using the global setting "snapshot.backup.to.secondary". Setting this to false 
makes snapshots be placed directly on Ceph instead of secondary storage. See if 
you can perform recurring snapshots. I know that there are limitations with KVM 
and disk snapshots but good to give it a try.

Thanks


Get Outlook for Android

From: Jeremy Hansen 
Sent: Saturday, February 3, 2024 11:39:19 PM
To: ceph-users@ceph.io 
Subject: [ceph-users] Re: Snapshot automation/scheduling for rbd?

Am I just off base here or missing something obvious?

Thanks



On Thursday, Feb 01, 2024 at 2:13 AM, Jeremy Hansen 
mailto:jer...@skidrow.la>> wrote:
Can rbd image snapshotting be scheduled like CephFS snapshots?  Maybe I missed 
it in the documentation but it looked like scheduling snapshots wasn’t a 
feature for block images.  I’m still running Pacific. We’re trying to devise a 
sufficient backup plan for Cloudstack and other things residing in Ceph.

Thanks.
-jeremy




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Help: Balancing Ceph OSDs with different capacity

[Jasper Tan]

Hi Anthony and everyone else

We have found the issue. Because the new 20x 14 TiB OSDs were onboarded
onto a single node, there was not only an imbalance in the capacity of each
OSD but also between the nodes (other nodes each have around 15x 1.7TiB).
Furthermore, CRUSH rule sets default failure domain to host with 3x
replication. This means that 1 of the copies will reside on a PG within the
node with 20x 14TiB while 2/3 of the replicated copies are forced to be on
the other nodes with 1.7TiB regardless of the weight as there are no other
alternatives. Changing the failure domain from host to osd resolved the
issue and I was able to achieve perfect balance at the cost of redundancy.
Moving forward we will physically rearrange the OSDs on each node.

Thanks
Jasper Tan

On Thu, Feb 8, 2024 at 3:29 AM Anthony D'Atri  wrote:

>
>
> > I have recently onboarded new OSDs into my Ceph Cluster. Previously, I
> had
> > 44 OSDs of 1.7TiB each and was using it for about a year. About 1 year
> ago,
> > we onboarded an additional 20 OSDs of 14TiB each.
>
> That's a big difference in size.  I suggest increasing
> mon_max_pg_per_osd  to 1000 -- that will help avoid unpleasantness when a
> component fails, including PGs or OSDs that won't activate.
>
> > However I observed that many of the data were still being written onto
> the
> > original 1.7TiB OSDs instead of the 14TiB ones. Overtime, this caused a
> > bottleneck as the 1.7TiB OSDs reached nearfull capacity.
>
> Please share your Ceph release, `ceph osd tree`, and and what your pool
> definitions and CRUSH rules look like.
>
>
> > I have tried to perform a reweight (both crush reweight and reweight) to
> > reduce the number of PGs on each 1.7TiB. This worked temporarily but
> > resulted in many objects being misplaced and PGs being in a Warning
> state.
>
> Misplaced objects are natural in such an operation.  With recent Ceph
> releases you shouldn't have to do this.  You have the balancer module
> enabled?
>
> > Subsequently I have also tried using crush-compat balancer mode instead
> of
> > upmap but did not see significant improvement. The latest changes I made
> > was to change backfill-threshold to 0.85, hoping that PGs will no longer
> be
> > assigned to OSDs that are >85% utilization.
>
> No, that'll just stop backfill from happening.  That ratio is for a
> different purpose.
>
>
> > However, this did not change the situation much as I see many OSDs above
> >85% utilization today.
> >
> > Attached is a report from ceph report command.
>
> Attachments don't make it through to the list.
>
> I suspect that what you're seeing is a misalignment of your CRUSH rules
> and your cluster topology:
>
> * Maybe your 1.7 TB OSDs are the ssd deviceclass and the 14 TB SSDs are
> the hdd device class.  If your CRUSH rule(s) specify the ssd device class,
> they won't use the new OSDs
> * Say you have failure domain = host, and all the 14TB OSDs are on one or
> two hosts.  Your CRUSH rules may force the smaller OSDs to be selected for
> PGs to satisfy anti-affinity
> * Similarly if you have rack failure domain and all the larger OSDs are in
> the same rack.
>
>
>

--

-- 


--


*The contents of this e-mail message and any attachments are 
confidential and are intended solely
for addressee. The information may 
also be legally privileged. This transmission is sent in trust, for
the 
sole purpose of delivery to the intended recipient. If you have received 
this transmission in error,
any use, reproduction or dissemination of this 
transmission is strictly prohibited. If you are not the
intended recipient, 
please immediately NOTIFY the sender by reply e-mail or phone and DELETE
this message and its attachments, if any.*
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: PG stuck at recovery

[Kai Stian Olstad]

You don't say anything about the Ceph version you are running.
I had an similar issue with 17.2.7, and is seams to be an issue with mclock,
when I switch to wpq everything worked again.

You can read more about it here
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/IPHBE3DLW5ABCZHSNYOBUBSI3TLWVD22/#OE3QXLAJIY6NU7PNMGHP47UK2CBZJPUG

- 
Kai Stian Olstad



On Tue, Feb 06, 2024 at 06:35:26AM -, LeonGao  wrote:

Hi community

We have a new Ceph cluster deployment with 100 nodes. When we are draining an 
OSD host from the cluster, we see a small amount of PGs that cannot make any 
progress to the end. From the logs and metrics, it seems like the recovery 
progress is stuck (0 recovery ops for several days). Would like to get some 
ideas on this. Re-peering and OSD restart do resolve to mitigate the issue but 
we want to get to the root cause of it as draining and recovery happen 
frequently.

I have put some debugging information below. Any help is appreciated, thanks!

ceph -s
   pgs: 4210926/7380034104 objects misplaced (0.057%)
41198 active+clean
71active+remapped+backfilling
12active+recovering

One of the stuck PG:
6.38f1   active+remapped+backfilling [313,643,727] 313  
   [313,643,717] 313

PG query result:

ceph pg 6.38f1 query
{
   "snap_trimq": "[]",
   "snap_trimq_len": 0,
   "state": "active+remapped+backfilling",
   "epoch": 246856,
   "up": [
   313,
   643,
   727
   ],
   "acting": [
   313,
   643,
   717
   ],
   "backfill_targets": [
   "727"
   ],
   "acting_recovery_backfill": [
   "313",
   "643",
   "717",
   "727"
   ],
   "info": {
   "pgid": "6.38f1",
   "last_update": "212333'38916",
   "last_complete": "212333'38916",
   "log_tail": "80608'37589",
   "last_user_version": 38833,
   "last_backfill": "MAX",
   "purged_snaps": [],
   "history": {
   "epoch_created": 3726,
   "epoch_pool_created": 3279,
   "last_epoch_started": 243987,
   "last_interval_started": 243986,
   "last_epoch_clean": 220174,
   "last_interval_clean": 220173,
   "last_epoch_split": 3726,
   "last_epoch_marked_full": 0,
   "same_up_since": 238347,
   "same_interval_since": 243986,
   "same_primary_since": 3728,
   "last_scrub": "212333'38916",
   "last_scrub_stamp": "2024-01-29T13:43:10.654709+",
   "last_deep_scrub": "212333'38916",
   "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+",
   "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+",
   "prior_readable_until_ub": 0
   },
   "stats": {
   "version": "212333'38916",
   "reported_seq": 413425,
   "reported_epoch": 246856,
   "state": "active+remapped+backfilling",
   "last_fresh": "2024-02-05T21:14:40.838785+",
   "last_change": "2024-02-03T22:33:43.052272+",
   "last_active": "2024-02-05T21:14:40.838785+",
   "last_peered": "2024-02-05T21:14:40.838785+",
   "last_clean": "2024-02-03T04:26:35.168232+",
   "last_became_active": "2024-02-03T22:31:16.037823+",
   "last_became_peered": "2024-02-03T22:31:16.037823+",
   "last_unstale": "2024-02-05T21:14:40.838785+",
   "last_undegraded": "2024-02-05T21:14:40.838785+",
   "last_fullsized": "2024-02-05T21:14:40.838785+",
   "mapping_epoch": 243986,
   "log_start": "80608'37589",
   "ondisk_log_start": "80608'37589",
   "created": 3726,
   "last_epoch_clean": 220174,
   "parent": "0.0",
   "parent_split_bits": 14,
   "last_scrub": "212333'38916",
   "last_scrub_stamp": "2024-01-29T13:43:10.654709+",
   "last_deep_scrub": "212333'38916",
   "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+",
   "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+",
   "objects_scrubbed": 17743,
   "log_size": 1327,
   "log_dups_size": 3000,
   "ondisk_log_size": 1327,
   "stats_invalid": false,
   "dirty_stats_invalid": false,
   "omap_stats_invalid": false,
   "hitset_stats_invalid": false,
   "hitset_bytes_stats_invalid": false,
   "pin_stats_invalid": false,
   "manifest_stats_invalid": false,
   "snaptrimq_len": 0,
   "last_scrub_duration": 29,
   "scrub_schedule": "queued for deep scrub",
   "scrub_duration": 28.69984447599,
   "objects_trimmed": 0,
   "snaptrim_duration": 0,
   "stat_sum": {
   "num_bytes": 67167022618,
   "num_objects": 17743,
   "num_object_clones": 0,
   "num_object_copies": 53229,
   

[ceph-users] Re: Help: Balancing Ceph OSDs with different capacity

[Anthony D'Atri]

> I have recently onboarded new OSDs into my Ceph Cluster. Previously, I had
> 44 OSDs of 1.7TiB each and was using it for about a year. About 1 year ago,
> we onboarded an additional 20 OSDs of 14TiB each.

That's a big difference in size.  I suggest increasing  mon_max_pg_per_osd  to 
1000 -- that will help avoid unpleasantness when a component fails, including 
PGs or OSDs that won't activate.

> However I observed that many of the data were still being written onto the
> original 1.7TiB OSDs instead of the 14TiB ones. Overtime, this caused a
> bottleneck as the 1.7TiB OSDs reached nearfull capacity.

Please share your Ceph release, `ceph osd tree`, and and what your pool 
definitions and CRUSH rules look like.


> I have tried to perform a reweight (both crush reweight and reweight) to
> reduce the number of PGs on each 1.7TiB. This worked temporarily but
> resulted in many objects being misplaced and PGs being in a Warning state.

Misplaced objects are natural in such an operation.  With recent Ceph releases 
you shouldn't have to do this.  You have the balancer module enabled?

> Subsequently I have also tried using crush-compat balancer mode instead of
> upmap but did not see significant improvement. The latest changes I made
> was to change backfill-threshold to 0.85, hoping that PGs will no longer be
> assigned to OSDs that are >85% utilization.

No, that'll just stop backfill from happening.  That ratio is for a different 
purpose.


> However, this did not change the situation much as I see many OSDs above >85% 
> utilization today.
> 
> Attached is a report from ceph report command.

Attachments don't make it through to the list.

I suspect that what you're seeing is a misalignment of your CRUSH rules and 
your cluster topology:

* Maybe your 1.7 TB OSDs are the ssd deviceclass and the 14 TB SSDs are the hdd 
device class.  If your CRUSH rule(s) specify the ssd device class, they won't 
use the new OSDs
* Say you have failure domain = host, and all the 14TB OSDs are on one or two 
hosts.  Your CRUSH rules may force the smaller OSDs to be selected for PGs to 
satisfy anti-affinity
* Similarly if you have rack failure domain and all the larger OSDs are in the 
same rack.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Help: Balancing Ceph OSDs with different capacity

[Dan van der Ster]

Hi Jasper,

I suggest to disable all the crush-compat and reweighting approaches.
They rarely work out.

The state of the art is:

ceph balancer on
ceph balancer mode upmap
ceph config set mgr mgr/balancer/upmap_max_deviation 1

Cheers, Dan

--
Dan van der Ster
CTO

Clyso GmbH
p: +49 89 215252722 | a: Vancouver, Canada
w: https://clyso.com | e: dan.vanders...@clyso.com

Try our Ceph Analyzer!: https://analyzer.clyso.com/
We are hiring: https://www.clyso.com/jobs/




On Wed, Feb 7, 2024 at 11:05 AM Jasper Tan  wrote:
>
> Hi
>
> I have recently onboarded new OSDs into my Ceph Cluster. Previously, I had
> 44 OSDs of 1.7TiB each and was using it for about a year. About 1 year ago,
> we onboarded an additional 20 OSDs of 14TiB each.
>
> However I observed that many of the data were still being written onto the
> original 1.7TiB OSDs instead of the 14TiB ones. Overtime, this caused a
> bottleneck as the 1.7TiB OSDs reached nearfull capacity.
>
> I have tried to perform a reweight (both crush reweight and reweight) to
> reduce the number of PGs on each 1.7TiB. This worked temporarily but
> resulted in many objects being misplaced and PGs being in a Warning state.
>
> Subsequently I have also tried using crush-compat balancer mode instead of
> upmap but did not see significant improvement. The latest changes I made
> was to change backfill-threshold to 0.85, hoping that PGs will no longer be
> assigned to OSDs that are >85% utilization. However, this did not change
> the situation much as I see many OSDs above >85% utilization today.
>
> Attached is a report from ceph report command. For now I have OUT-ed two of
> my OSDs which have reached 95% capacity. I would greatly appreciate it if
> someone can provide advice on this matter.
>
> Thanks
> Jasper Tan
>
>
> --
>
> --
>
>
> --
>
>
> *The contents of this e-mail message and any attachments are
> confidential and are intended solely
> for addressee. The information may
> also be legally privileged. This transmission is sent in trust, for
> the
> sole purpose of delivery to the intended recipient. If you have received
> this transmission in error,
> any use, reproduction or dissemination of this
> transmission is strictly prohibited. If you are not the
> intended recipient,
> please immediately NOTIFY the sender by reply e-mail or phone and DELETE
> this message and its attachments, if any.*
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] PG stuck at recovery

[LeonGao]

Hi community

We have a new Ceph cluster deployment with 100 nodes. When we are draining an 
OSD host from the cluster, we see a small amount of PGs that cannot make any 
progress to the end. From the logs and metrics, it seems like the recovery 
progress is stuck (0 recovery ops for several days). Would like to get some 
ideas on this. Re-peering and OSD restart do resolve to mitigate the issue but 
we want to get to the root cause of it as draining and recovery happen 
frequently.

I have put some debugging information below. Any help is appreciated, thanks!

ceph -s
pgs: 4210926/7380034104 objects misplaced (0.057%)
 41198 active+clean
 71active+remapped+backfilling
 12active+recovering

One of the stuck PG:
6.38f1   active+remapped+backfilling [313,643,727] 313  
   [313,643,717] 313

PG query result:

ceph pg 6.38f1 query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+remapped+backfilling",
"epoch": 246856,
"up": [
313,
643,
727
],
"acting": [
313,
643,
717
],
"backfill_targets": [
"727"
],
"acting_recovery_backfill": [
"313",
"643",
"717",
"727"
],
"info": {
"pgid": "6.38f1",
"last_update": "212333'38916",
"last_complete": "212333'38916",
"log_tail": "80608'37589",
"last_user_version": 38833,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 3726,
"epoch_pool_created": 3279,
"last_epoch_started": 243987,
"last_interval_started": 243986,
"last_epoch_clean": 220174,
"last_interval_clean": 220173,
"last_epoch_split": 3726,
"last_epoch_marked_full": 0,
"same_up_since": 238347,
"same_interval_since": 243986,
"same_primary_since": 3728,
"last_scrub": "212333'38916",
"last_scrub_stamp": "2024-01-29T13:43:10.654709+",
"last_deep_scrub": "212333'38916",
"last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+",
"last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+",
"prior_readable_until_ub": 0
},
"stats": {
"version": "212333'38916",
"reported_seq": 413425,
"reported_epoch": 246856,
"state": "active+remapped+backfilling",
"last_fresh": "2024-02-05T21:14:40.838785+",
"last_change": "2024-02-03T22:33:43.052272+",
"last_active": "2024-02-05T21:14:40.838785+",
"last_peered": "2024-02-05T21:14:40.838785+",
"last_clean": "2024-02-03T04:26:35.168232+",
"last_became_active": "2024-02-03T22:31:16.037823+",
"last_became_peered": "2024-02-03T22:31:16.037823+",
"last_unstale": "2024-02-05T21:14:40.838785+",
"last_undegraded": "2024-02-05T21:14:40.838785+",
"last_fullsized": "2024-02-05T21:14:40.838785+",
"mapping_epoch": 243986,
"log_start": "80608'37589",
"ondisk_log_start": "80608'37589",
"created": 3726,
"last_epoch_clean": 220174,
"parent": "0.0",
"parent_split_bits": 14,
"last_scrub": "212333'38916",
"last_scrub_stamp": "2024-01-29T13:43:10.654709+",
"last_deep_scrub": "212333'38916",
"last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+",
"last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+",
"objects_scrubbed": 17743,
"log_size": 1327,
"log_dups_size": 3000,
"ondisk_log_size": 1327,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 29,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 28.69984447599,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 67167022618,
"num_objects": 17743,
"num_object_clones": 0,
"num_object_copies": 53229,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 12401,
"num_objects_unfound": 0,
"num_objects_dirty": 17743,
"num_whiteouts": 0,
"num_read": 16640,

[ceph-users] Help: Balancing Ceph OSDs with different capacity

[Jasper Tan]

Hi

I have recently onboarded new OSDs into my Ceph Cluster. Previously, I had
44 OSDs of 1.7TiB each and was using it for about a year. About 1 year ago,
we onboarded an additional 20 OSDs of 14TiB each.

However I observed that many of the data were still being written onto the
original 1.7TiB OSDs instead of the 14TiB ones. Overtime, this caused a
bottleneck as the 1.7TiB OSDs reached nearfull capacity.

I have tried to perform a reweight (both crush reweight and reweight) to
reduce the number of PGs on each 1.7TiB. This worked temporarily but
resulted in many objects being misplaced and PGs being in a Warning state.

Subsequently I have also tried using crush-compat balancer mode instead of
upmap but did not see significant improvement. The latest changes I made
was to change backfill-threshold to 0.85, hoping that PGs will no longer be
assigned to OSDs that are >85% utilization. However, this did not change
the situation much as I see many OSDs above >85% utilization today.

Attached is a report from ceph report command. For now I have OUT-ed two of
my OSDs which have reached 95% capacity. I would greatly appreciate it if
someone can provide advice on this matter.

Thanks
Jasper Tan


--

-- 


--


*The contents of this e-mail message and any attachments are 
confidential and are intended solely
for addressee. The information may 
also be legally privileged. This transmission is sent in trust, for
the 
sole purpose of delivery to the intended recipient. If you have received 
this transmission in error,
any use, reproduction or dissemination of this 
transmission is strictly prohibited. If you are not the
intended recipient, 
please immediately NOTIFY the sender by reply e-mail or phone and DELETE
this message and its attachments, if any.*
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD Image Returning 'Unknown Filesystem LVM2_member' On Mount - Help Please

[Gilles Mocellin]

Le dimanche 4 février 2024, 09:29:04 CET duluxoz a écrit :
> Hi Cedric,
> 
> That's what I thought - the access method shouldn't make a difference.
> 
> No, no lvs details at all - I mean, yes, the osds show up with the lvs 
> command on the ceph node(s), but not on the individual pools/images (on 
> the ceph node or the client) - this is, of course, that I'm doing this 
> right (and there's no guarantee of that).
> 
> To clarify: entering `lvs` on the client (which has the rbd image 
> "attached" as /dev/rbd0) returns nothing, and `lvs` on any of the ceph 
> nodes only returns the data for each OSD/HDD.
[...]

Hello,

I think that /dev/rbd* devices are flitered "out" or not filter "in" by the 
fiter 
option in the devices section of /etc/lvm/lvm.conf.

So pvscan (pvs, vgs and lvs) don't look at your device.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Accumulation of removed_snaps_queue After Deleting Snapshots in Ceph RBD

[localhost Liam]

Hello,

I'm encountering an issue with Ceph when using it as the backend storage for 
OpenStack Cinder. Specifically, after deleting RBD snapshots through Cinder, 
I've noticed a significant increase in the removed_snaps_queue entries within 
the corresponding Ceph pool. It seems to affect the pool's performance and 
space efficiency.

I understand that snapshot deletion in Cinder is an asynchronous operation, and 
Ceph itself uses a lazy deletion mechanism to handle snapshot removal. However, 
even after allowing sufficient time, the entries in removed_snaps_queue do not 
decrease as expected.

I have several questions for the community:

Are there recommended methods or best practices for managing or reducing 
entries in removed_snaps_queue?
Is there any tool or command that can safely clear these residual snapshot 
entries without affecting the integrity of active snapshots and data?
Is this issue known, and are there any bug reports or plans for fixes related 
to it?
Thank you very much for your assistance!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph error connecting to the cluster

[arimbidhea3]

hello, i was tried to create osd but when i run ceph command the output is like 
this :

root@pod-deyyaa-ceph1:~# sudo ceph -s
2024-02-02T16:01:23.627+0700 7fc762f37640  0 monclient(hunting): authenticate 
timed out after 300
[errno 110] RADOS timed out (error connecting to the cluster)


can anyone give me a hint or help me to solve this
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific 16.2.15 QE validation status

[Yuri Weinstein]

We are still working through the remaining issues and will do a full
cycle of testing soon.

Adam, the issues mentioned by Ilya below require some response and
resolution, pls take a look

> rbd looks good overall but we are missing iSCSI coverage due to
> https://tracker.ceph.com/issues/64126


On Wed, Jan 31, 2024 at 6:40 AM Ilya Dryomov  wrote:
>
> On Tue, Jan 30, 2024 at 9:24 PM Yuri Weinstein  wrote:
> >
> > Update.
> > Seeking approvals/reviews for:
> >
> > rados - Radek, Laura, Travis, Ernesto, Adam King
> > rgw - Casey
> > fs - Venky
> > rbd - Ilya
>
> Hi Yuri,
>
> rbd looks good overall but we are missing iSCSI coverage due to
> https://tracker.ceph.com/issues/64126:
>
> https://pulpito.ceph.com/yuriw-2024-01-25_17:27:10-rbd-pacific-release-distro-default-smithi/7532167
> https://pulpito.ceph.com/yuriw-2024-01-26_17:05:20-rbd-pacific-release-distro-default-smithi/7535193
>
> Adam, did you get a chance to look into it?
>
> > krbd - Ilya
>
> Please do another rerun for krbd -- I want to see one of those
> remaining jobs pass.
>
> Thanks,
>
> Ilya
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific 16.2.15 QE validation status

[Konstantin Shalygin]

> 
> On Feb 7, 2024, at 16:59, Zakhar Kirpichenko  wrote:
> 
> Indeed, it looks like it's been recently reopened. Thanks for this!

Hi,

It was merged yesterday

Thanks for the right noise,
k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problems adding a new host via orchestration.

[Eugen Block]

I still don't have an explanation or other ideas, but I was able to  
add a Rocky Linux 9 host to my existing quincy cluster based on  
openSUSE (don't have pacific in this environment) quite fast and easy.  
It is a fresh Rocky Install, only added cephadm and podman packages,  
copied the ceph.pub key over and ran 'ceph orch host add ...'  
successfully.


Zitat von Gary Molenkamp :

I confirmed selinux is disabled on all existing and new hosts.  
Likewise, python3.6 is installed on all as well.  (3.9.16 on RL8,  
3.9.18 on RL9).


I am running 16.2.12 on all containers, so it may be worth updating  
to 16.2.14 to ensure I'm on the latest Pacific release.


Gary


On 2024-02-05 08:17, Curt wrote:



You don't often get email from light...@gmail.com. Learn why this  
is important 



I don't use rocky, so stab in the dark and probably not the issue,  
but could selinux be blocking the process?  Really long shot, but  
python3 is in the standard location? So if you run python3  
--version as your ceph user it returns?


Probably not much help, but figured I'd throw it out there.

On Mon, 5 Feb 2024, 16:54 Gary Molenkamp,  wrote:

   I have verified the server's expected hostname (with `hostname`)
   matches
   the hostname I am trying to use.
   Just to be sure, I also ran:
    cephadm check-host --expect-hostname 
   and it returns:
    Hostname "" matches what is expected.

   On the current admin server where I am trying to add the host, the
   host
   is reachable, the shortname even matches proper IP with dns search
   order.
   Likewise, on the server where the mgr is running, I am able to
   confirm
   reachability and DNS resolution for the new server as well.

   I thought this may be a DNS/name resolution issue as well, but I
   don't
   see any errors in my setup wrt to host naming.

   Thanks
   Gary


   On 2024-02-03 06:46, Eugen Block wrote:
   > Hi,
   >
   > I found this blog post [1] which reports the same error message. It
   > seems a bit misleading because it appears to be about DNS. Can
   you check
   >
   > cephadm check-host --expect-hostname 
   >
   > Or is that what you already tried? It's not entirely clear how you
   > checked the hostname.
   >
   > Regards,
   > Eugen
   >
   > [1]
   >

https://blog.mousetech.com/ceph-distributed-file-system-for-the-enterprise/ceph-bogus-error-cannot-allocate-memory/

   >
   > Zitat von Gary Molenkamp :
   >
   >> Happy Friday all.  I was hoping someone could point me in the
   right
   >> direction or clarify any limitations that could be impacting an
   issue
   >> I am having.
   >>
   >> I'm struggling to add a new set of hosts to my ceph cluster using
   >> cephadm and orchestration.  When trying to add a host:
   >>     "ceph orch host add  172.31.102.41 --labels _admin"
   >> returns:
   >>     "Error EINVAL: Can't communicate with remote host
   >> `172.31.102.41`, possibly because python3 is not installed there:
   >> [Errno 12] Cannot allocate memory"
   >>
   >> I've verified that the ceph ssh key works to the remote host,
   host's
   >> name matches that returned from `hostname`, python3 is
   installed, and
   >> "/usr/sbin/cephadm prepare-host" on the new hosts returns "host is
   >> ok".    In addition, the cluster ssh key works between hosts
   and the
   >> existing hosts are able to ssh in using the ceph key.
   >>
   >> The existing ceph cluster is Pacific release using docker based
   >> containerization on RockyLinux8 base OS.  The new hosts are
   >> RockyLinux9 based, with the cephadm being installed from Quincy
   release:
   >>         ./cephadm add-repo --release quincy
   >>         ./cephadm install
   >> I did try installing cephadm from the Pacific release by
   changing the
   >> repo to el8,  but that did not work either.
   >>
   >> Is there a limitation is mixing RL8 and RL9 container hosts under
   >> Pacific?  Does this same limitation exist under Quincy? Is there a
   >> python version dependency?
   >> The reason for RL9 on the new hosts is to stage upgrading the OS's
   >> for the cluster.  I did this under Octopus for moving from
   Centos7 to
   >> RL8.
   >>
   >> Thanks and I appreciate any feedback/pointers.
   >> Gary
   >>
   >>
   >> I've added the log trace here in case that helps (from `ceph
   log last
   >> cephadm`)
   >>
   >>
   >>
   >> 2024-02-02T14:22:32.610048+ mgr.storage01.oonvfl
   (mgr.441023307)
   >> 4957871 : cephadm [ERR] Can't communicate with remote host
   >> `172.31.102.41`, possibly because python3 is not installed there:
   >> [Errno 12] Cannot allocate memory
   >> Traceback (most recent call last):
   >>   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1524, in
   >> _remote_connection
   >>     conn, connr = self.mgr._get_connection(addr)
   >>   File "/usr/share/ceph/mgr/cephadm/module.py", line 1370, in
   >> _get_connection
   >>     sudo=True if self.ssh_user != 'root' else False)
   >>   File 

[ceph-users] Re: pacific 16.2.15 QE validation status

[Zakhar Kirpichenko]

Indeed, it looks like it's been recently reopened. Thanks for this!

/Z

On Wed, 7 Feb 2024 at 15:43, David Orman  wrote:

> That tracker's last update indicates it's slated for inclusion.
>
> On Thu, Feb 1, 2024, at 10:47, Zakhar Kirpichenko wrote:
> > Hi,
> >
> > Please consider not leaving this behind:
> https://github.com/ceph/ceph/pull/55109
> >
> > It's a serious bug, which potentially affects a whole node stability if
> > the affected mgr is colocated with OSDs. The bug was known for quite a
> > while and really shouldn't be left unfixed.
> >
> > /Z
> >
> > On Thu, 1 Feb 2024 at 18:45, Nizamudeen A  wrote:
> >> Thanks Laura,
> >>
> >> Raised a PR for  https://tracker.ceph.com/issues/57386
> >> https://github.com/ceph/ceph/pull/55415
> >>
> >>
> >> On Thu, Feb 1, 2024 at 5:15 AM Laura Flores  wrote:
> >>
> >> > I reviewed the rados suite. @Adam King ,
> @Nizamudeen A
> >> >  would appreciate a look from you, as there are some
> >> > orchestrator and dashboard trackers that came up.
> >> >
> >> > pacific-release, 16.2.15
> >> >
> >> > Failures:
> >> > 1. https://tracker.ceph.com/issues/62225
> >> > 2. https://tracker.ceph.com/issues/64278
> >> > 3. https://tracker.ceph.com/issues/58659
> >> > 4. https://tracker.ceph.com/issues/58658
> >> > 5. https://tracker.ceph.com/issues/64280 -- new tracker, worth a
> look
> >> > from Orch
> >> > 6. https://tracker.ceph.com/issues/63577
> >> > 7. https://tracker.ceph.com/issues/63894
> >> > 8. https://tracker.ceph.com/issues/64126
> >> > 9. https://tracker.ceph.com/issues/63887
> >> > 10. https://tracker.ceph.com/issues/61602
> >> > 11. https://tracker.ceph.com/issues/54071
> >> > 12. https://tracker.ceph.com/issues/57386
> >> > 13. https://tracker.ceph.com/issues/64281
> >> > 14. https://tracker.ceph.com/issues/49287
> >> >
> >> > Details:
> >> > 1. pacific upgrade test fails on 'ceph versions | jq -e' command -
> >> > Ceph - RADOS
> >> > 2. Unable to update caps for client.iscsi.iscsi.a - Ceph -
> Orchestrator
> >> > 3. mds_upgrade_sequence: failure when deploying node-exporter -
> Ceph -
> >> > Orchestrator
> >> > 4. mds_upgrade_sequence: Error: initializing source
> >> > docker://prom/alertmanager:v0.20.0 - Ceph - Orchestrator
> >> > 5. mgr-nfs-upgrade test times out from failed cephadm daemons -
> Ceph -
> >> > Orchestrator
> >> > 6. cephadm: docker.io/library/haproxy: toomanyrequests: You have
> >> > reached your pull rate limit. You may increase the limit by
> authenticating
> >> > and upgrading: https://www.docker.com/increase-rate-limit - Ceph -
> >> > Orchestrator
> >> > 7. qa: cephadm failed with an error code 1, alertmanager
> container not
> >> > found. - Ceph - Orchestrator
> >> > 8. ceph-iscsi build was retriggered and now missing
> >> > package_manager_version attribute - Ceph
> >> > 9. Starting alertmanager fails from missing container - Ceph -
> >> > Orchestrator
> >> > 10. pacific: cls/test_cls_sdk.sh: Health check failed: 1 pool(s)
> do
> >> > not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS
> >> > 11. rados/cephadm/osds: Invalid command: missing required
> parameter
> >> > hostname() - Ceph - Orchestrator
> >> > 12. cephadm/test_dashboard_e2e.sh: Expected to find content:
> '/^foo$/'
> >> > within the selector: 'cd-modal .badge' but never did - Ceph - Mgr -
> >> > Dashboard
> >> > 13. Failed to download key at
> >> > http://download.ceph.com/keys/autobuild.asc: Request failed:  >> > error [Errno 101] Network is unreachable> - Infrastructure
> >> > 14. podman: setting cgroup config for procHooks process caused:
> Unit
> >> > libpod-$hash.scope not found - Ceph - Orchestrator
> >> >
> >> > On Wed, Jan 31, 2024 at 1:41 PM Casey Bodley 
> wrote:
> >> >
> >> >> On Mon, Jan 29, 2024 at 4:39 PM Yuri Weinstein 
> >> >> wrote:
> >> >> >
> >> >> > Details of this release are summarized here:
> >> >> >
> >> >> > https://tracker.ceph.com/issues/64151#note-1
> >> >> >
> >> >> > Seeking approvals/reviews for:
> >> >> >
> >> >> > rados - Radek, Laura, Travis, Ernesto, Adam King
> >> >> > rgw - Casey
> >> >>
> >> >> rgw approved, thanks
> >> >>
> >> >> > fs - Venky
> >> >> > rbd - Ilya
> >> >> > krbd - in progress
> >> >> >
> >> >> > upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
> >> >> > upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)
> >> >> >
> >> >> > upgrade/pacific-x (quincy) - in progress
> >> >> > upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)
> >> >> >
> >> >> > ceph-volume - Guillaume
> >> >> >
> >> >> > TIA
> >> >> > YuriW
> >> >> > ___
> >> >> > ceph-users mailing list -- ceph-users@ceph.io
> >> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >> >> >
> >> >> ___
> >> >> Dev mailing list -- d...@ceph.io
> >> >> To unsubscribe send an email to dev-le...@ceph.io
> >> >>
> >> >
> 

[ceph-users] Re: pacific 16.2.15 QE validation status

[David Orman]

That tracker's last update indicates it's slated for inclusion.

On Thu, Feb 1, 2024, at 10:47, Zakhar Kirpichenko wrote:
> Hi, 
>
> Please consider not leaving this behind: 
> https://github.com/ceph/ceph/pull/55109
>
> It's a serious bug, which potentially affects a whole node stability if 
> the affected mgr is colocated with OSDs. The bug was known for quite a 
> while and really shouldn't be left unfixed. 
>
> /Z
>
> On Thu, 1 Feb 2024 at 18:45, Nizamudeen A  wrote:
>> Thanks Laura,
>> 
>> Raised a PR for  https://tracker.ceph.com/issues/57386
>> https://github.com/ceph/ceph/pull/55415
>> 
>> 
>> On Thu, Feb 1, 2024 at 5:15 AM Laura Flores  wrote:
>> 
>> > I reviewed the rados suite. @Adam King , @Nizamudeen A
>> >  would appreciate a look from you, as there are some
>> > orchestrator and dashboard trackers that came up.
>> >
>> > pacific-release, 16.2.15
>> >
>> > Failures:
>> > 1. https://tracker.ceph.com/issues/62225
>> > 2. https://tracker.ceph.com/issues/64278
>> > 3. https://tracker.ceph.com/issues/58659
>> > 4. https://tracker.ceph.com/issues/58658
>> > 5. https://tracker.ceph.com/issues/64280 -- new tracker, worth a look
>> > from Orch
>> > 6. https://tracker.ceph.com/issues/63577
>> > 7. https://tracker.ceph.com/issues/63894
>> > 8. https://tracker.ceph.com/issues/64126
>> > 9. https://tracker.ceph.com/issues/63887
>> > 10. https://tracker.ceph.com/issues/61602
>> > 11. https://tracker.ceph.com/issues/54071
>> > 12. https://tracker.ceph.com/issues/57386
>> > 13. https://tracker.ceph.com/issues/64281
>> > 14. https://tracker.ceph.com/issues/49287
>> >
>> > Details:
>> > 1. pacific upgrade test fails on 'ceph versions | jq -e' command -
>> > Ceph - RADOS
>> > 2. Unable to update caps for client.iscsi.iscsi.a - Ceph - Orchestrator
>> > 3. mds_upgrade_sequence: failure when deploying node-exporter - Ceph -
>> > Orchestrator
>> > 4. mds_upgrade_sequence: Error: initializing source
>> > docker://prom/alertmanager:v0.20.0 - Ceph - Orchestrator
>> > 5. mgr-nfs-upgrade test times out from failed cephadm daemons - Ceph -
>> > Orchestrator
>> > 6. cephadm: docker.io/library/haproxy: toomanyrequests: You have
>> > reached your pull rate limit. You may increase the limit by authenticating
>> > and upgrading: https://www.docker.com/increase-rate-limit - Ceph -
>> > Orchestrator
>> > 7. qa: cephadm failed with an error code 1, alertmanager container not
>> > found. - Ceph - Orchestrator
>> > 8. ceph-iscsi build was retriggered and now missing
>> > package_manager_version attribute - Ceph
>> > 9. Starting alertmanager fails from missing container - Ceph -
>> > Orchestrator
>> > 10. pacific: cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do
>> > not have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS
>> > 11. rados/cephadm/osds: Invalid command: missing required parameter
>> > hostname() - Ceph - Orchestrator
>> > 12. cephadm/test_dashboard_e2e.sh: Expected to find content: '/^foo$/'
>> > within the selector: 'cd-modal .badge' but never did - Ceph - Mgr -
>> > Dashboard
>> > 13. Failed to download key at
>> > http://download.ceph.com/keys/autobuild.asc: Request failed: > > error [Errno 101] Network is unreachable> - Infrastructure
>> > 14. podman: setting cgroup config for procHooks process caused: Unit
>> > libpod-$hash.scope not found - Ceph - Orchestrator
>> >
>> > On Wed, Jan 31, 2024 at 1:41 PM Casey Bodley  wrote:
>> >
>> >> On Mon, Jan 29, 2024 at 4:39 PM Yuri Weinstein 
>> >> wrote:
>> >> >
>> >> > Details of this release are summarized here:
>> >> >
>> >> > https://tracker.ceph.com/issues/64151#note-1
>> >> >
>> >> > Seeking approvals/reviews for:
>> >> >
>> >> > rados - Radek, Laura, Travis, Ernesto, Adam King
>> >> > rgw - Casey
>> >>
>> >> rgw approved, thanks
>> >>
>> >> > fs - Venky
>> >> > rbd - Ilya
>> >> > krbd - in progress
>> >> >
>> >> > upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
>> >> > upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)
>> >> >
>> >> > upgrade/pacific-x (quincy) - in progress
>> >> > upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)
>> >> >
>> >> > ceph-volume - Guillaume
>> >> >
>> >> > TIA
>> >> > YuriW
>> >> > ___
>> >> > ceph-users mailing list -- ceph-users@ceph.io
>> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> >> >
>> >> ___
>> >> Dev mailing list -- d...@ceph.io
>> >> To unsubscribe send an email to dev-le...@ceph.io
>> >>
>> >
>> >
>> > --
>> >
>> > Laura Flores
>> >
>> > She/Her/Hers
>> >
>> > Software Engineer, Ceph Storage 
>> >
>> > Chicago, IL
>> >
>> > lflo...@ibm.com | lflo...@redhat.com 
>> > M: +17087388804
>> >
>> >
>> >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to 

[ceph-users] Re: Direct ceph mount on desktops

[Tim Holloway]

Followup.

Desktop system went to sleep overnight. I woke up to this:


HEALTH_WARN 1 client(s) laggy due to laggy OSDs; 1 clients failing to
respond to capability release; 1 MDSs report slow requests
[WRN] MDS_CLIENTS_LAGGY: 1 client(s) laggy due to laggy OSDs
mds.ceefs.www2.lzjqgd(mds.0): Client 7513643 is laggy; not evicted
because some OSD(s) is/are laggy
[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to
capability release
mds.ceefs.www2.lzjqgd(mds.0): Client a64.mousetech.com: failing to
respond to capability release client_id: 7513643
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
mds.ceefs.www2.lzjqgd(mds.0): 1 slow requests are blocked > 30 secs

a64 is the sleeping desktop.

I restarted the the ceph target and a whole bunch of stuck pg's popped
up, but the whole w2 machine is apparently scrambled and even a reboot
hasn't made its processes happy so I'm going to have to go in and fix
them one by one. May be unrelated to the original laggy OSD problem,
though. Just wanted to mention it for completeness.

Also, the desktop has weirdly managed to mount both native ceph and the
ceph NFS at the same mount point. Most likely because I'd added a
pre/post sleep script to unmount/remount ceph when the system went into
or out of suspension and I lost track of it so I haven't fixed it.

Again, this is mostly just for background not related to ceph's own
internals, although I had wondered if the ceph mount was happening
slowly and the suspension wasn't waiting properly for it to complete.

On Tue, 2024-02-06 at 13:00 -0500, Patrick Donnelly wrote:
> On Tue, Feb 6, 2024 at 12:09 PM Tim Holloway 
> wrote:
> > 
> > Back when I was battline Octopus, I had problems getting ganesha's
> > NFS
> > to work reliably. I resolved this by doing a direct (ceph) mount on
> > my
> > desktop machine instead of an NFS mount.
> > 
> > I've since been plagued by ceph "laggy OSD" complaints that appear
> > to
> > be due to a non-responsive client and I'm suspecting that the
> > client in
> > question is the desktop machine when it's suspended while the ceph
> > mount is in effect.
> 
> You should not see "laggy OSD" messages due to a client becoming
> unresponsive.
> 
> > So the question is: Should ceph native mounts be used on general
> > client
> > machines which may hibernate or otherwise go offline?
> 
> The mounts will eventually be evicted (generally) by the MDS if the
> machine hibernates/suspends. There are mechanisms for the mount to
> recover (see "recover_session" in the mount.ceph man page).  Any
> dirty
> data would be lost.
> 
> As for whether you should have clients that hibernate, it's not
> ideal.
> It could conceivably create problems if client machines hibernate
> longer than the blocklist duration (after eviction by the MDS).
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph orch upgrade to 18.2.1 seems stuck on MDS?

[Nigel Williams]

kicked off
 ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.1

and just MDS left to do but upgrade has been sitting for hours on this

root@rdx-00:~# ceph orch upgrade status
{
"target_image": "
quay.io/ceph/ceph@sha256:a4e86c750cc11a8c93453ef5682acfa543e3ca08410efefa30f520b54f41831f
",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [
"osd",
"mgr",
"mon",
"crash"
],
"progress": "1267/1306 daemons upgraded",
"message": "Currently upgrading osd daemons",
"is_paused": false
}
root@rdx-00:~#

message seems erroneous as all the OSDs seem to be done.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io