[ceph-users] How to speed up rgw lifecycle

Hi community, My ceph cluster is using s3 with three pools and obj/s approximately 4.5k obj/s and the rgw lifecycle delete per pool is only 60-70 objects/s How can I speed up the lc rgw process? 60 70 objects/s is too slow Thanks a lot ___ ceph-users

[ceph-users] the image used size becomes 0 after export/import with snapshot

Hi, I have an image with a snapshot and some changes after snapshot. ``` $ rbd du backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 NAME PROVISIONED USED

[ceph-users] Re: osdmaptool target & deviation calculation

Hi, This deviation is very soft. If u wanna do real upmaps you should use deviation 1 k Sent from my iPhone > On Nov 27, 2023, at 21:39, Robert Hish wrote: > > The result is many many OSDs with a deviation well above the > upmap_max_deviation which is at default: 5

[ceph-users] Re: reef 18.2.1 QE Validation status

On Tue, Nov 21, 2023 at 10:35 PM Venky Shankar wrote: > > Hi Yuri, > > On Fri, Nov 10, 2023 at 1:22 PM Venky Shankar wrote: > > > > Hi Yuri, > > > > On Fri, Nov 10, 2023 at 4:55 AM Yuri Weinstein wrote: > > > > > > I've updated all approvals and merged PRs in the tracker and it looks > > > like

[ceph-users] Re: How balancer module balance data

Hi, For the reason you observed, I normally set upmap_max_deviation = 1 on all clusters I get my hands on. Cheers, Dan -- Dan van der Ster CTO Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanders...@clyso.com We are hiring: https://www.clyso.com/jobs/

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

Hi, looks like line I got : # ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-888 --deep 1 _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x73a13c49, expected 0x21d59f5e, device location [0x268e81~1000], logical extent 0x3~1000, object

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

So those options are not consistent with the error in the video I linked. I am not entirely sure how to proceed with your OSDs (how many are impacted?) but you may want to try injecting an older osdmap epoch fetched from the mon in your osdmap injection: try rewinding 1 epoch at a time from the

[ceph-users] Re: understand "extent"

On Sat, Nov 25, 2023 at 4:19 AM Tony Liu wrote: > > Hi, > > The context is RBD on bluestore. I did check extent on Wiki. > I see "extent" when talking about snapshot and export/import. > For example, when create a snapshot, we mark extents. When > there is write to marked extents, we will make a

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

it's: "bluestore_compression_algorithm": "snappy" "bluestore_compression_mode": "none" On 11/27/23 20:13, Wesley Dillingham wrote: How about these two options: bluestore_compression_algorithm bluestore_compression_mode Thanks. Respectfully, *Wes Dillingham* w...@wesdillingham.com

[ceph-users] Re: Rook-Ceph OSD Deployment Error

Sounds like you're hitting a known issue with v17.2.7. https://github.com/rook/rook/issues/13136 The fix will be in v18.2.1 if it's an option to upgrade to Reef. If not, you'll need to use v17.2.6 until the fix comes out for quincy in v17.2.8. Travis On Thu, Nov 23, 2023 at 4:06 PM P

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

What I was getting at was to see if this was somehow related to the bug described here https://www.youtube.com/watch?v=_4HUR00oCGo It should not be given the version of ceph you are using but the CRC error you are seeing is similar. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

The options Wes listed are for data, not RocksDB. > On Nov 27, 2023, at 1:59 PM, Denis Polom wrote: > > Hi, > > no we don't: > > "bluestore_rocksdb_options": >

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

How about these two options: bluestore_compression_algorithm bluestore_compression_mode Thanks. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Mon, Nov 27, 2023 at 2:01 PM Denis Polom wrote: > Hi, > > no we don't: > >

[ceph-users] Re: About number of osd node can be failed with erasure code 3+2

With a k+m which is 3+2 each RADOS object is broken into 5 shards. By default the pool will have a min_size of k+1 (4 in this case). Which means you can lose 1 shard and still be >= min_size. If one host goes down and you use a host-based failure domain (default) you will lose 1 shard out of all

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

Hi, no we don't: "bluestore_rocksdb_options":

[ceph-users] Re: Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

Hi Giuseppe, There are likely one or two clients whose op is blocking the reconnect/replay. If you increase debug_mds perhaps you can find the guilty client and disconnect it / block it from mounting. Or for a more disruptive recovery you can try this "Deny all reconnect to clients " option:

[ceph-users] [MDS] mds stuck in laggy state, CephFS unusable

Hi, we're having a peculiar issue which we found out during HA/DR testing in our Ceph cluster. Basic info about cluster: Version: Quincy (17.2.6) 5 nodes configured in stretch cluster (2 DCs with one arbiter node which is also admin node for the cluster) On every node beside the admin node we

[ceph-users] osdmaptool target & deviation calculation

Question about the osdmaptool deviation calculations; For instance, - osdmaptool omap --upmap output.txt --upmap-pool cephfs_data-rep3 --upmap-max 1000 --upmap-deviation 5 osdmaptool: osdmap file 'omap' writing upmap command output to: output.txt checking for upmap cleanups upmap,

[ceph-users] About number of osd node can be failed with erasure code 3+2

Hi Groups, Recently I was setting up a ceph cluster with 10 nodes 144 osd, and I use S3 for it with pool erasure code EC3+2 on it. I have a question, how many osd nodes can fail with erasure code 3+2 with cluster working normal (read, write)? and can i choose better erasure code ec7+3, 8+2

[ceph-users] CloudStack and Ceph Day 2024

Hello Ceph users, Together with ShapeBlue and Adyen, 42on is organizing a CloudStack and Ceph Day; this time in Amsterdam, The Netherlands. We are planning this for February 8 | 2024. We want to create a technical event that shares updates on both technologies, as well as 'use cases',

[ceph-users] Re: RadosGW public HA traffic - best practices?

An easy setup if you use PowerDNS is to establish LUA records on the gateway: https://doc.powerdns.com/authoritative/lua-records/ ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How balancer module balance data

Hello, We are running a pacific 16.2.10 cluster and enabled the balancer module, here is the configuration: [root@ceph-1 ~]# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.052548", "last_optimize_started": "Fri Nov 17 17:09:57 2023", "mode": "upmap",

[ceph-users] blustore osd nearfull but no pgs on it

Hi, after a massive rebalance(tunables) my small SSD-OSDs are getting full, i changed my crush rules so there are actual no pgs/pools on it, but the disks stay full: ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus (stable) ID CLASS WEIGHT REWEIGHT SIZE   

[ceph-users] ceph-volume lvm new-db throws errors

Hi, ceph-volume lvm new-db does not work as expected due to missing /var/lib/ceph/osd/ceph-OSDID. However, the database device seems to be added to the OSD. Until now, it is unclear to me if the DB device was actually successfully added to the OSD or this is a bug in ceph version 17.2.7 The

[ceph-users] What is the maximum number of Rados gateway objects in one cluster using the bucket index and in one bucket?

Hello. We are using Ceph storage to test whether we can run the service by uploading and saving more than 40 billion files. So I'd like to check the contents below. 1) Maximum number of Rados gateway objects that can be stored in one cluster using the bucket index 2) Maximum number of Rados

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

Curious if you are using bluestore compression? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Mon, Nov 27, 2023 at 10:09 AM Denis Polom wrote: > Hi > > we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7 >

[ceph-users] Re: MDS_DAMAGE in 17.2.7 / Cannot delete affected files

Hello Sebastian, On Fri, Nov 24, 2023 at 8:49 AM Sebastian Knust wrote: > > Hi, > > After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into > MDS_DAMAGE state. We had some prior issues with faulty kernel clients > not releasing capabilities, therefore the update might just be a

[ceph-users] Re: Does cephfs ensure close-to-open consistency after enabling lazyio?

No. You must call lazyio_synchronize. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to

[ceph-users] OSDs failing to start due to crc32 and osdmap error

Hi we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7 cluster. Some OSDs on that node are running fine, but some failing to start. Looks like crc32 checksum error, and failing to get OSD map. I found a some discussions on that but nothing helped. I've also tried to

[ceph-users] MDS stuck in up:rejoin

Hi all, For about a week our CephFS has experienced issues with its MDS. Currently the MDS is stuck in "up:rejoin" Issues become apparent when simple commands like "mv foo bar/" hung. I unmounted CephFS offline on the clients, evicted those remaining, and then issued ceph config set mds.0

[ceph-users] Re: import/export with --export-format 2

Hi, I can't comment on why #3 shouldn't be used, but a quick test shows that the image is not really usable in that case. I created a partition on the src-image (1 GB), filled it up with around 500 MB of data and then did the same export you did: rbd export --export-format 2 src-image -

[ceph-users] Re: Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

Hi David, Thanks a lot for your reply. Yes we have heavy load from clients on the same subtree. We have multiple MDSs that were setup with the hope to distribute the load among them, but this is not really happening, in moments of high load we see most of the load on one MDS. We don't use

[ceph-users] Experience with deduplication

Hi Developers, What is the status of the deduplication for objectsore? I see it under the dev area only since octopus even with the latest release. https://docs.ceph.com/en/octopus/dev/deduplication/ Is it something that can be used in production? Thank you

[ceph-users] Re: Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

Hi Guiseppe, Wouldn't you have clients who heavily load the MDS with concurrent access on the same trees ? Perhaps, also, look at the stability of all your clients (even if there are many) [dmesg -T, ...] How are your 4 active MDS configured (pinning?) ? Probably nothing to do but normal for 2

[ceph-users] Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

Hi, We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are having CephFS issues. For example this morning: “”” [root@naret-monitor01 ~]# ceph -s cluster: id: 63334166-d991-11eb-99de-40a6b72108d0 health: HEALTH_WARN 1 filesystem is degraded

[ceph-users] Ceph/daemon container lvm tools don’t work

Hi team, I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re migrating away from CentOS Stream 8. As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK running on the same hosts (OSDs+NOVA/CINDER). That prohibits me to let CEPH nodes running on CentOS

[ceph-users] Re: Where is a simple getting started guide for a very basic cluster?

Looking up the "manual installation" parts might help, if you can't get the container stuff going for $reasons. Den mån 27 nov. 2023 kl 00:45 skrev Leo28C : > > I'm pulling my hair trying to get a simple cluster going. I first tried > Gluster but I have an old system that can't handle the latest