[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

Hi, You need revert your packages from Quincy to Pacific. `dnf downgrade ceph-mon` command should help with this k Sent from my iPhone > On Oct 11, 2023, at 03:22, Waywatcher wrote: > > I am unable to get any of the current monitors to run. They all fail to start

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

I think the problem is that there's no MONs running now in the cluster because he upgraded them without heading the warning to update the database first. Are you suggesting he deploy new Pacific MONs using levelDB.then update them to rocksdb after the CEPH cluster recovers? A bit confused by

[ceph-users] Re: Hardware recommendations for a Ceph cluster

On Mon, 9 Oct 2023 at 14:24, Anthony D'Atri wrote: > > > > AFAIK the standing recommendation for all flash setups is to prefer fewer > > but faster cores > > Hrm, I think this might depend on what you’re solving for. This is the > conventional wisdom for MDS for sure. My sense is that OSDs can

[ceph-users] Re: Hardware recommendations for a Ceph cluster

Anthony, Thank you very much for your comments; they were very helpful. It made me reconsider some aspects of the configuration, and it also helped me see that I wasn't too far off in general. I'll respond to some of your suggestions, explaining my reasons. > Indeed, I know from experience

[ceph-users] Re: Unable to fix 1 Inconsistent PG

In case it's not obvious I forgot a space: "rados list-inconsistent-obj 15.f4f" Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Tue, Oct 10, 2023 at 4:55 PM Wesley Dillingham wrote: > You likely have a failing disk, what does

[ceph-users] Re: Unable to fix 1 Inconsistent PG

You likely have a failing disk, what does "rados list-inconsistent-obj15.f4f" return? It should identify the failing osd. Assuming "ceph osd ok-to-stop " returns in the affirmative for that osd, you likely need to stop the associated osd daemon, then mark it out "ceph osd out wait for it to

[ceph-users] Re: CephFS: convert directory into subvolume

Hello, I'm following this tread and the original. I'm trying to convert directories into subvolumes. Where I'm stuck is how you move a directory into the subvolume root directory. I have a volume 'tank' and it's mounted on the host as '/mnt/tank' I have subfolders '/mnt/tank/database',

[ceph-users] Re: Ceph 18: Unable to delete image after imcomplete migration "image being migrated"

Thanks Eugen. root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+ 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory

[ceph-users] Unable to fix 1 Inconsistent PG

Hello All, Greetings. We've a Ceph Cluster with the version *ceph version 14.2.16-402-g7d47dbaf4d (7d47dbaf4d0960a2e910628360ae36def84ed913) nautilus (stable) === Issues: 1 pg in inconsistent state and does not recover. # ceph -s cluster: id:

[ceph-users] cephadm, cannot use ECDSA key with quincy

Hi ceph users, We have a few clusters with quincy 17.2.6 and we are preparing to migrate from ceph-deploy to cephadm for better management. We are using Ubuntu20 with latest updates (latest openssh). While testing the migration to cephadm on a test cluster with octopus (v16 latest) we had no

[ceph-users] Re: Remove empty orphaned PGs not mapped to a pool

@Eugen We have seen the same problems 8 years ago. I can only recommend never to use cache tiering in production. At Cephalocon this was part of my talk and as far as I remember cache tiering will also disappear from ceph soon. Cache tiering has been deprecated in the Reef release as it has

[ceph-users] Re: snap_schedule works after 1 hour of scheduling

Hi Milind, Thank you for your response. Please find the logs attached, as instructed. Thanks and Regards, Kushagra Gupta On Thu, Oct 5, 2023 at 12:09 PM Milind Changire wrote: > this is really odd > > Please run following commands and send over their outputs: > # ceph status > # ceph fs

[ceph-users] Re: snap_schedule works after 1 hour of scheduling

Hi Milind,Team Thank you for your response @Milind Changire >>The only thing I can think of is a stale mgr that wasn't restarted >>after an upgrade. >>Was an upgrade performed lately ? Yes an upgrade was performed after which we faced this. But we were facing this issue previously as well.

[ceph-users] Re: Ceph 18: Unable to delete image after imcomplete migration "image being migrated"

Thanks again Eugen. Looking at my command history it does look like I did execute the migration but didn't commit it. I wasn't surprised to see it in the trash based on the doc you mentioned, I only tried the restore as a desperate measure to clean up my mess. It doesn't help that I messed

[ceph-users] Re: Ceph 18: Unable to delete image after imcomplete migration "image being migrated"

Hi, I just re-read the docs on rbd migration [1], haven't done that in a while, and it states the following: Note that the source image will be moved to the RBD trash to avoid mistaken usage during the migration process So it was expected that your source image was in the trash during

[ceph-users] Announcing go-ceph v0.24.0

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.24.0 Changes include fixes to the rgw admin and rbd packages. More details are available at the link

[ceph-users] Re: Ceph 18: Unable to delete image after imcomplete migration "image being migrated"

Thanks Eugen. root@hcn03:~# rbd status infra-pool/sophosbuild 2023-10-10T09:44:21.234+ 7f1675c524c0 -1 librbd::Migration: open_images: failed to open destination image images/65d188c5f5a34: (2) No such file or directory rbd: getting migration status failed: (2) No such file or directory

[ceph-users] Re: slow recovery with Quincy

Hi Ben, Please see this thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PWHG6QJ6N2TJEYD2U4AXJAJ23CRPJG4E/#7ZMBM23GXYFIGY52ZWJDY5NUSYSDSYL6 for possible workaround. 发自我的 iPad 在 2023年10月10日，22:26，Ben 写道： Dear cephers, with one osd down(200GB/9.1TB data), rebalance

[ceph-users] Re: Nothing provides libthrift-0.14.0.so()(64bit)

we're tracking this in https://tracker.ceph.com/issues/61882. my understanding is that we're just waiting for the next quincy point release builds to resolve this On Tue, Oct 10, 2023 at 11:07 AM Graham Derryberry wrote: > > I have just started adding a ceph client on a rocky 9 system to our

[ceph-users] Re: Copying big objects (>5GB) doesn't work after upgrade to Quincy on S3

hi Arvydas, it looks like this change corresponds to https://tracker.ceph.com/issues/48322 and https://github.com/ceph/ceph/pull/38234. the intent was to enforce the same limitation as AWS S3 and force clients to use multipart copy instead. this limit is controlled by the config option

[ceph-users] Re: Nothing provides libthrift-0.14.0.so()(64bit)

I have just started adding a ceph client on a rocky 9 system to our ceph cluster (we're on quincy 17.2.6) and just discovered that epel 9 now provides thrift-0.15.0-2.el9 not thrift-0.14.0-7.el9 as of June 21 2023. So the Nothing provides libthrift-0.14.0.so()(64bit) error has returned!

[ceph-users] Copying big objects (>5GB) doesn't work after upgrade to Quincy on S3

Hi all, after upgrading our cluster from Nautilus -> Pacific -> Quincy we noticed we can't copy bigger objects anymore via S3. An error we get: "Aws::S3::Errors::EntityTooLarge (Aws::S3::Errors::EntityTooLarge)" After some tests we have following findings: * Problems starts for objects bigger

[ceph-users] slow recovery with Quincy

Dear cephers, with one osd down(200GB/9.1TB data), rebalance takes 3 hours still in progress. Client bandwidth can go as high as 200MB/s. With little client request throughput, recovery goes at couple MB/s. I wonder if there is configuration to polish for improvement. It runs with quincy 17.2.5,

[ceph-users] Re: outdated mds slow requests

Hi, It get cleared by restarting ceph client with issues. It works. to do that, you would umount problematic cephfs volume and remount. All ceph warning is gone in couple minutes, trimming well now. Indeed I wouldn't restart mds unless I had to. Many thanks for help, Ben Eugen Block

[ceph-users] Re: cephadm, cannot use ECDSA key with quincy

The CA signed keys working in pacific was sort of accidental. We found out that it was a working use case in pacific but not in quincy earlier this year, which resulted in this tracker https://tracker.ceph.com/issues/62009. That has since been implemented in main, and backported to the reef branch

[ceph-users] Re: cephadm, cannot use ECDSA key with quincy

Hi! If is because old ssh client was replaced with asyncssh ( https://github.com/ceph/ceph/pull/51899) and only ported to reef, when will be added to quincy? For us is a blocker as we cannot move to cephadm anymore, as we planned for Q4. Is there a workarround? Thank you for your efforts! Paul

[ceph-users] Re: Ceph 18: Unable to delete image after imcomplete migration "image being migrated"

Hi, there are a couple of things I would check before migrating all images. What's the current 'rbd status infra-pool/sophosbuild'? You probably don't have an infinite number of pools so I would also check if any of the other pools contains an image with the same name, just in case you

[ceph-users] Re: outdated mds slow requests

Hi, 2, restart problematic mds with trimming behind issue: 3,4,5: mds will start up quickly, won't they? investigating... this one you should be able to answer better than the rest of us. You probably have restarted MDS daemons before, I would assume. Just don't restart them all at once but

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Any input from anyone, please? It's another thing that seems to be rather poorly documented: it's unclear what to expect, what 'normal' behavior should be, and what can be done about the huge amount of writes by monitors. /Z On Mon, 9 Oct 2023 at 12:40, Zakhar Kirpichenko wrote: > Hi, > >

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

Hi, For this upgrade you need at least some mon's up, then you can redeploy your pacific mon's to rocksdb k Sent from my iPhone > On Oct 10, 2023, at 02:01, Waywatcher wrote: > > I upgraded my CEPH cluster without properly following the mon upgrade so > they were no longer on leveldb. > >