[ceph-users] Re: Odd 10-minute delay before recovery IO begins

I think you are experiencing the mon_osd_down_out_interval https://docs.ceph.com/en/latest/rados/configuration/mon-osd-interaction/#confval-mon_osd_down_out_interval Ceph waits 10 minutes before marking a down osd as out for the reasons you mention, but this would have been the case in nautilus a

[ceph-users] Re: Very slow snaptrim operations blocking client I/O

I hit this issue once on a nautilus cluster and changed the OSD parameter bluefs_buffered_io = true (was set at false). I believe the default of this parameter was switched from false to true in release 14.2.20, however, perhaps you could still check what your osds are configured with in regard to

[ceph-users] Re: deep scrub and long backfilling

In general it is safe and during long running remapping and backfill situations I enable it. You can enable it with: "ceph config set osd osd_scrub_during_recovery true" If you have any problems you think are caused by the change, undo it: Stop scrubs asap: "ceph osd set nodeep-scrub" "ceph osd

[ceph-users] Re: v16.2.12 Pacific (hot-fix) released

A few questions: - Will the 16.2.12 packages be "corrected" and reuploaded to the ceph.com mirror? or will 16.2.13 become what 16.2.12 was supposed to be? - Was the osd activation regression introduced in 16.2.11 (or does 16.2.10 have it as well)? - Were the hotfxes in 16.2.12 just related to pe

[ceph-users] Re: For suggestions and best practices on expanding Ceph cluster and removing old nodes

Get on nautilus first and (perhaps even go to pacific) before expansion. Primarily for the reason that starting in nautilus degraded data recovery will be prioritized over remapped data recovery. As you phase out old hardware and phase in new hardware you will have a very large amount of backfill

[ceph-users] Re: Ceph recovery

Assuming size=3 and min_size=2 It will run degraded (read/write capable) until a third host becomes available at which point it will backfill the third copy on the third host. It will be unable to create the third copy of data if no third host exists. If an additional host is lost the data will bec

[ceph-users] Re: Upgrade Ceph cluster + radosgw from 14.2.18 to latest 15

Curious, why not go to Pacific? You can upgrade up to 2 major releases in a go. The upgrade process to pacific is here: https://docs.ceph.com/en/latest/releases/pacific/#upgrading-non-cephadm-clusters The upgrade to Octopus is here: https://docs.ceph.com/en/latest/releases/octopus/#upgrading-from-

[ceph-users] Re: Upgrade Ceph cluster + radosgw from 14.2.18 to latest 15

I have upgraded dozens of clusters 14 -> 16 using the methods described in the docs, and when followed precisely no issues have arisen. I would suggest moving to a release that is receiving backports still (pacific or quincy). The important aspects are only doing one system at a time. In the case o

[ceph-users] Re: ceph Pacific - MDS activity freezes when one the MDSs is restarted

There was a memory issue with standby-replay that may have been resolved since and fix is in 16.2.10 (not sure), the suggestion at the time was to avoid standby-replay. Perhaps a dev can chime in on that status. Your MDSs look pretty inactive. I would consider scaling them down (potentially to sin

[ceph-users] Re: `ceph features` on Nautilus still reports "luminous"

Fairly confident this is normal. I just checked a pacific cluster and they all report luminous as well. I think some of the backstory of this is luminous is the release where up-maps were released and there hasnt been a reason to increment the features release of subsequent daemons. To be honest I

[ceph-users] Re: PGs stuck undersized and not scrubbed

When PGs are degraded they won't scrub, further, if an OSD is involved with recovery of another PG it wont accept scrubs either so that is the likely explanation of your not-scrubbed-in time issue. Its of low concern. Are you sure that recovery is not progressing? I see: "7349/147534197 objects de

[ceph-users] Re: The pg_num from 1024 reduce to 32 spend much time, is there way to shorten the time？

Can you send along the responses from "ceph df detail" and ceph "ceph osd pool ls detail" Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Tue, Jun 6, 2023 at 1:03 PM Eugen Block wrote: > I suspect the target_max_misplaced_ratio (

[ceph-users] Re: ceph.conf and two different ceph clusters

You need to use the --id and --cluster options of the rbd command and maintain a .conf file for each cluster. /etc/ceph/clusterA.conf /etc/ceph/clusterB.conf /etc/ceph/clusterA.client.userA.keyring /etc/ceph/clusterB.client.userB.keyring now use the rbd commands as such: rbd --id userA --cluste

[ceph-users] Re: mon log file grows huge

At what level do you have logging set to for your mons? That is a high volume of logs for the mon to generate. You can ask all the mons to print their debug logging level with: "ceph tell mon.* config get debug_mon" The default is 1/5 What is the overall status of your cluster? Is it healthy?

[ceph-users] Re: librbd hangs during large backfill

Did your automation / process allow for stalls in between changes to allow peering to complete? My hunch is you caused a very large peering storm (during peering a PG is inactive) which in turn caused your VMs to panic. If the RBDs are unmapped and re-mapped does it still continue to struggle? Res

[ceph-users] Re: cannot repair a handful of damaged pg's

A repair is just a type of scrub and it is also limited by osd_max_scrubs which in pacific is 1. If another scrub is occurring on any OSD in the PG it wont start. do "ceph osd set noscrub" and "ceph osd set nodeep-scrub" wait for all scrubs to stop (a few seconds probably) Then issue the pg repa

[ceph-users] Re: Unable to fix 1 Inconsistent PG

You likely have a failing disk, what does "rados list-inconsistent-obj15.f4f" return? It should identify the failing osd. Assuming "ceph osd ok-to-stop " returns in the affirmative for that osd, you likely need to stop the associated osd daemon, then mark it out "ceph osd out wait for it to backf

[ceph-users] Re: Unable to fix 1 Inconsistent PG

In case it's not obvious I forgot a space: "rados list-inconsistent-obj 15.f4f" Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Oct 10, 2023 at 4:55 PM Wesley Dillingham wrote: > You likely have a

[ceph-users] Re: Unable to fix 1 Inconsistent PG

If I recall correctly When the acting or up_set of an PG changes the scrub information is lost. This was likely lost when you stopped osd.238 and changed the sets. I do not believe based on your initial post you need to be using the objectstore tool currently. Inconsistent PGs are a common occurre

[ceph-users] Re: Unable to fix 1 Inconsistent PG

finish backfill 4 - issue the pg repair Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Wed, Oct 11, 2023 at 4:38 PM Wesley Dillingham wrote: > If I recall correctly When the acting or up_set of an PG changes

[ceph-users] Re: How do you handle large Ceph object storage cluster?

Well you are probably in the top 1% of cluster size. I would guess that trying to cut your existing cluster in half while not encountering any downtime as you shuffle existing buckets between old cluster and new cluster would be harder than redirecting all new buckets (or users) to a second cluster

[ceph-users] owner locked out of bucket via bucket policy

I have a bucket which got injected with bucket policy which locks the bucket even to the bucket owner. The bucket now cannot be accessed (even get its info or delete bucket policy does not work) I have looked in the radosgw-admin command for a way to delete a bucket policy but do not see anything.

[ceph-users] Re: owner locked out of bucket via bucket policy

ls with awscli to delete or overwrite this > bucket policy > > On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham > wrote: > > > > I have a bucket which got injected with bucket policy which locks the > > bucket even to the bucket owner. The bucket now cannot be accessed

[ceph-users] Re: owner locked out of bucket via bucket policy

Thank you, this has worked to remove the policy. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley wrote: > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham > wrote: > >

[ceph-users] Re: owner locked out of bucket via bucket policy

option for now. >> >> Thanks, >> Jayanth >> -- >> *From:* Jayanth Reddy >> *Sent:* Tuesday, November 7, 2023 11:59:38 PM >> *To:* Casey Bodley >> *Cc:* Wesley Dillingham ; ceph-users < >> ceph-users@ceph.io>;

[ceph-users] per-rbd snapshot limitation

looking into how to limit snapshots at the ceph level for RBD snapshots. Ideally ceph would enforce an arbitrary number of snapshots allowable per rbd. Reading the man page for rbd command I see this option: https://docs.ceph.com/en/quincy/man/8/rbd/#cmdoption-rbd-limit --limit Specifies the lim

[ceph-users] Re: per-rbd snapshot limitation

Perfect, thank you. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Wed, Nov 15, 2023 at 1:00 PM Ilya Dryomov wrote: > On Wed, Nov 15, 2023 at 5:57 PM Wesley Dillingham > wrote: > > > > looking into h

[ceph-users] Re: per-rbd snapshot limitation

; On Wed, Nov 15, 2023 at 1:14 PM David C. wrote: > rbd create testpool/test3 --size=100M > rbd snap limit set testpool/test3 --limit 3 > > > Le mer. 15 nov. 2023 à 17:58, Wesley Dillingham a > écrit : > >> looking into how to limit snapshots at the ceph level for RBD

[ceph-users] Re: blustore osd nearfull but no pgs on it

Please send along a pastebin of "ceph status" and "ceph osd df tree" and "ceph df detail" also "ceph tell osd.158 status" Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Fri, Nov 17, 2023 at 6:20 PM Debian wrote: > thx for your r

[ceph-users] Re: blustore osd nearfull but no pgs on it

The large amount of osdmaps is what i was suspecting. "ceph tell osd.158 status" (or any osd other than 158) would show us how many osdmaps the osds are currently holding on to. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Mon,

[ceph-users] Re: Why is min_size of erasure pools set to k+1

" if min_size is k and you lose an OSD during recovery after a failure of m OSDs, data will become unavailable" In that situation data wouldnt become unavailable it would be lost. Having a min_size of k+1 provides a buffer between data being active+writeable and where data is lost. That inbetween

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

Curious if you are using bluestore compression? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Mon, Nov 27, 2023 at 10:09 AM Denis Polom wrote: > Hi > > we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7 > cl

[ceph-users] Re: About number of osd node can be failed with erasure code 3+2

With a k+m which is 3+2 each RADOS object is broken into 5 shards. By default the pool will have a min_size of k+1 (4 in this case). Which means you can lose 1 shard and still be >= min_size. If one host goes down and you use a host-based failure domain (default) you will lose 1 shard out of all PG

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

ions=2,max_total_wal_size=1073741824", > thx > > On 11/27/23 19:17, Wesley Dillingham wrote: > > Curious if you are using bluestore compression? > > Respectfully, > > *Wes Dillingham* > w...@wesdillingham.com > LinkedIn <http://www.linkedin.com/in/wesleydilling

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

> > > > "bluestore_rocksdb_options": > "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactio

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

> it's: > > "bluestore_compression_algorithm": "snappy" > > "bluestore_compression_mode": "none" > > > On 11/27/23 20:13, Wesley Dillingham wrote: > > How about these two options: > > bluestore_compression_algorithm > bluestore_compression_mode &g

[ceph-users] Re: Best Practice for OSD Balancing

It's a complicated topic and there is no one answer, it varies for each cluster and depends. You have a good lay of the land. I just wanted to mention that the correct "foundation" for equally utilized OSDs within a cluster relies on two important factors: - Symmetry of disk/osd quantity and capa

[ceph-users] Re: Logging control

"ceph daemon" commands need to be run local to the machine where the daemon is running. So in this case if you arent on the node where osd.1 lives it wouldnt work. "ceph tell" should work anywhere there is a client.admin key. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn

[ceph-users] Re: Is there a way to find out which client uses which version of ceph?

You can ask the monitor to dump its sessions (which should expose the IPs and the release / features) you can then track down by IP those with the undesirable features/release ceph daemon mon.`hostname -s` sessions Assuming your mon is named after the short hostname, you may need to do this for e

[ceph-users] Re: 17.2.7: Backfilling deadlock / stall / stuck / standstill

I faced a similar issue. The PG just would never finish recovery. Changing all OSDs in the PG to "osd_op_queue wpq" and then restarting them serially ultimately allowed the PG to recover. Seemed to be some issue with mclock. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn

[ceph-users] Re: 6 pgs not deep-scrubbed in time

Respond back with "ceph versions" output If your sole goal is to eliminate the not scrubbed in time errors you can increase the aggressiveness of scrubbing by setting: osd_max_scrubs = 2 The default in pacific is 1. if you are going to start tinkering manually with the pg_num you will want to tu

[ceph-users] Re: 6 pgs not deep-scrubbed in time

I now concur you should increase the pg_num as a first step for this cluster. Disable the pg autoscaler for and increase the volumes pool to pg_num 256. Then likely re-asses and make the next power of 2 jump to 512 and probably beyond. Keep in mind this is not going to fix your short term deep-scr

[ceph-users] Re: 6 pgs not deep-scrubbed in time

.0 TiB 3.4 TiB 1.6 TiB 8.5 MiB 4.8 > GiB 5.7 TiB 37.19 1.07 29 up osd.43 > 46hdd9.02330 1.0 9.0 TiB 3.1 TiB 1.4 TiB 8.4 MiB 4.4 > GiB 5.9 TiB 34.85 1.00 23 up osd.46 > TOTAL 433 TiB 151 TiB67 TiB

[ceph-users] Re: 6 pgs not deep-scrubbed in time

32 870 GiB 170.73k 2.5 TiB 1.13 72 TiB > testbench 10 32 0 B0 0 B 0 72 TiB > > On Tue, Jan 30, 2024 at 5:05 PM Wesley Dillingham > wrote: > >> It will take a couple weeks to a couple months to complete is my best >> gue

[ceph-users] Re: 6 pgs not deep-scrubbed in time

actually it seems the issue I had in mind was fixed in 16.2.11 so you should be fine. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Jan 30, 2024 at 10:34 AM Wesley Dillingham wrote: > You may want to consider upg

[ceph-users] Re: 6 pgs not deep-scrubbed in time

set norebalance' >>> > 'ceph osd set nodown' >>> > 'ceph osd set pause' >>> > 'ceph osd set nodeep-scrub' >>> > 'ceph osd set noscrub' >>> > >>> > >>> > Would like to ask

[ceph-users] Re: Ceph-storage slack access

At the very bottom of this page is a link https://ceph.io/en/community/connect/ Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Wed, Mar 6, 2024 at 11:45 AM Matthew Vernon wrote: > Hi, > > How does one get an invite to the ceph-s

[ceph-users] Re: activating+undersized+degraded+remapped

Please share "ceph osd tree" and "ceph osd df tree" I suspect you have not enough hosts to satisfy the EC On Sat, Mar 16, 2024, 8:04 AM Deep Dish wrote: > Hello > > I found myself in the following situation: > > [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive > > pg 4.3d is

[ceph-users] Re: activating+undersized+degraded+remapped

vice. With EC the > > default min_size is k+1, you could reduce the min_size to 5 > > temporarily, this might bring the PGs back online. But the long term > > fix is to have all required OSDs up and have enough OSDs to sustain an > > outage. > > > > Zitat von Wesle

[ceph-users] Re: Mounting A RBD Via Kernal Modules

I suspect this may be a network / firewall issue between the client and one OSD-server. Perhaps the 100MB RBD didn't have an object mapped to a PG with the primary on this problematic OSD host but the 2TB RBD does. Just a theory. Respectfully, *Wes Dillingham* LinkedIn

[ceph-users] Slow ops during recovery for RGW index pool only when degraded OSD is primary

I am fighting an issue on an 18.2.0 cluster where a restart of an OSD which supports the RGW index pool causes crippling slow ops. If the OSD is marked with primary-affinity of 0 prior to the OSD restart no slow ops are observed. If the OSD has a primary affinity of 1 slow ops occur. The slow ops o

[ceph-users] Re: Slow ops during recovery for RGW index pool only when degraded OSD is primary

Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> w...@wesdillingham.com On Wed, Apr 3, 2024 at 1:38 PM Joshua Baergen wrote: > We've had success using osd_async_recovery_min_cost=0 to drastically > reduce slow ops during index recovery. > > Josh >

[ceph-users] Re: PG inconsistent

check your ceph.log on the mons for "stat mismatch" and grep for the PG in question for potentially more information. Additionally "rados list-inconsistent-obj {pgid}" will often show which OSD and objects are implicated for the inconsistency. If the acting set has changed since the scrub (for exa

[ceph-users] Re: Setting S3 bucket policies with multi-tenants

Did you actually get this working? I am trying to replicate your steps but am not being successful doing this with multi-tenant. Respectfully, *Wes Dillingham* LinkedIn w...@wesdillingham.com On Wed, Nov 1, 2023 at 12:52 PM Thomas Bennett wrote:

[ceph-users] Re: Remove an OSD with hardware issue caused rgw 503

What you want to do is to stop the OSD (and all its copies of data it contains) by stopping the OSD service immediately. The downside of this approach is it causes the PGs on that OSD to be degraded. But the upside is the OSD which has bad hardware is immediately no longer participating in any cli

[ceph-users] Re: Ceph reef and (slow) backfilling - how to speed it up

In our case it was with a EC pool as well. I believe the PG state was degraded+recovering / recovery_wait and iirc the PGs just simply sat in the recovering state without any progress (degraded PG object count did not decline). A repeer of the PG was attempted but no success there. A restart of all

[ceph-users] Re: ceph orch osd rm --zap --replace leaves cluster in odd state

What is the state of your PGs? could you post "ceph -s" I believe (but a bit of an assumption after encountering something similar myself) that under the hood cephadm is using the "ceph osd safe-to-destroy osd.X" command and when OSD.X is no longer running and not all PGs are active+clean (for ins

[ceph-users] Re: Adding new OSDs - also adding PGs?

It depends on the cluster. In general I would say if your PG count is already good in terms of PG-per-OSD (say between 100 and 200 each) add capacity and then re-evaluate your PG count after. If you have a lot of time before the gear will be racked and could undergo some PG splits before the new g

[ceph-users] Re: Incomplete PGs. Ceph Consultant Wanted

Perhaps Ceph itself should also have a warning pop up (in "ceph -s", "ceph health detail" etc) when replica and min_size=1 or in an EC if min_size < k+1. Of course it could be muted but it would give an operator pause initially when setting that. I think a lot of people assume replica size=2 is saf

[ceph-users] Mon's falling out of quorum, require rebuilding. Rebuilt with only V2 address.

We have had multiple clusters experiencing the following situation over the past few months on both 14.2.6 and 14.2.11. On a few instances it seemed random , in a second situation we had temporary networking disruption, in a third situation we accidentally made some osd changes which caused certain

[ceph-users] Running Mons on msgrv2/3300 only.

We rebuilt all of our mons in one cluster such that they bind only to port 3300 with msgrv2. Previous to this we were binding to both 6789 and 3300. All of our server and client components are sufficiently new (14.2.x) and we haven’t observed any disruption but I am inquiring if this may be prob

[ceph-users] Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"

We have also had this issue multiple times in 14.2.11 On Tue, Dec 8, 2020, 5:11 PM wrote: > I have same issue. My cluster runing 14.2.11 versions. What is your > version ceph? > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send

[ceph-users] Re: Monitors not starting, getting "e3 handle_auth_request failed to assign global_id"

/ packet inspection security technology being run on the servers. Perhaps you've made similar updates. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Dec 8, 2020 at 7:46 PM Wesley Dillingham wrote: > We have

[ceph-users] Re: bug ceph auth

is /var/lib/ceph/bootstrap-osd/ in existence and writeable? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Wed, Jul 14, 2021 at 8:35 AM Marc wrote: > > > > [@t01 ~]# ceph auth get client.bootstrap-osd -o > ​/var/lib/ceph/bootstr

[ceph-users] Re: bug ceph auth

Do you get the same error if you just do "ceph auth get client.bootstrap-osd" i.e. does client.bootstrap exist as a user? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Wed, Jul 14, 2021 at 1:56 PM Wesley Dillingh

[ceph-users] Re: ceph osd continously fails

Can you send the results of "ceph daemon osd.0 status" and maybe do that for a couple of osd ids ? You may need to target ones which are currently running. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Wed, Aug 11, 2021 at 9:51 A

[ceph-users] Re: erasure coded pool PG stuck inconsistent on ceph Pacific 15.2.13

That response is typically indicative of a pg whose OSD sets has changed since it was last scrubbed (typically from a disk failing). Are you sure its actually getting scrubbed when you issue the scrub? For example you can issue: "ceph pg query" and look for "last_deep_scrub_stamp" which will tel

[ceph-users] Re: erasure coded pool PG stuck inconsistent on ceph Pacific 15.2.13

't > run at all. Could the deepscrubbing process be stuck elsewhere? > On 11/18/21 3:29 PM, Wesley Dillingham wrote: > > That response is typically indicative of a pg whose OSD sets has changed > since it was last scrubbed (typically from a disk failing). > > Are you sure

[ceph-users] Re: erasure coded pool PG stuck inconsistent on ceph Pacific 15.2.13

You may also be able to use an upmap (or the upmap balancer) to help make room for you on the osd which is too full. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Fri, Nov 19, 2021 at 1:14 PM Wesley Dillingham wrote:

[ceph-users] Re: Experience reducing size 3 to 2 on production cluster?

I would avoid doing this. Size 2 is not where you want to be. Maybe you can give more details about your cluster size and shape and what you are trying to accomplish and another solution could be proposed. The contents of "ceph osd tree " and "ceph df" would help. Respectfully, *Wes Dillingham* w

[ceph-users] Re: Is it normal Ceph reports "Degraded data redundancy" in normal use?

If you mark an osd "out" but not down / you dont stop the daemon do the PGs go remapped or do they go degraded then as well? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Thu, Apr 14, 2022 at 5:15 AM Kai Stian Olstad wrote: > O

[ceph-users] Aggressive Bluestore Compression Mode for client data only?

I would like to use bluestore compression (probably zstd level 3) to compress my clients data unless the incompressible hint is set (aggressive mode) but I do no want to expose myself to the bug experienced in this Cern talk (Ceph bug of the year) https://www.youtube.com/watch?v=_4HUR00oCGo where t

[ceph-users] Re: Erasure-coded PG stuck in the failed_repair state

In my experience: "No scrub information available for pg 11.2b5 error 2: (2) No such file or directory" is the output you get from the command when the up or acting osd set has changed since the last deep-scrub. Have you tried to run a deep scrub (ceph pg deep-scrub 11.2b5) on the pg and then try

[ceph-users] Re: Migration Nautilus to Pacifi : Very high latencies (EC profile)

We have a newly-built pacific (16.2.7) cluster running 8+3 EC jerasure ~250 OSDS across 21 hosts which has significantly lower than expected IOPS. Only doing about 30 IOPS per spinning disk (with appropriately sized SSD bluestore db) around ~100 PGs per OSD. Have around 100 CephFS (ceph fuse 16.2.7

[ceph-users] Re: Migration Nautilus to Pacifi : Very high latencies (EC profile)

;mds_max_purge_ops_per_pg": "0.10", with some success but still experimenting with how we can reduce the throughput impact from osd slow ops. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Mon, May 16, 2022 at

[ceph-users] Re: Migration Nautilus to Pacifi : Very high latencies (EC profile)

What was the largest cluster that you upgraded that didn't exhibit the new issue in 16.2.8 ? Thanks. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Tue, May 17, 2022 at 10:24 AM David Orman wrote: > We had an issue with our orig

[ceph-users] Re: Trouble getting cephadm to deploy iSCSI gateway

Well I dont use either the dashboard or the cephadm/containerized deployment but do use ceph-iscsi. The fact that your two gateways are not "up" might indicate that they havent been added to the target IQN yet. Once you can get into gwcli and create an iqn and associate your gateways with it, I gue

[ceph-users] Re: Drained OSDs are still ACTIVE_PRIMARY - casuing high IO latency on clients

This sounds similar to an inquiry I submitted a couple years ago [1] whereby I discovered that the choose_acting function does not consider primary affinity when choosing the primary osd. I had made the assumption it would when developing my procedure for replacing failing disks. After that discove

[ceph-users] Re: Cluster healthy, but 16.2.7 osd daemon upgrade says its unsafe to stop them?

What does "ceph osd pool ls detail" say? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Thu, May 26, 2022 at 11:24 AM Sarunas Burdulis < saru...@math.dartmouth.edu> wrote: > Running > > `ceph osd ok-to-stop 0` > > shows: > > {"ok

[ceph-users] Re: Cluster healthy, but 16.2.7 osd daemon upgrade says its unsafe to stop them?

26, 2022 at 2:22 PM Sarunas Burdulis wrote: > On 5/26/22 14:09, Wesley Dillingham wrote: > > What does "ceph osd pool ls detail" say? > > $ ceph osd pool ls detail > pool 0 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_nu

[ceph-users] Re: Slow delete speed through the s3 API

Is it just your deletes which are slow or writes and read as well? On Thu, Jun 2, 2022, 4:09 PM J-P Methot wrote: > I'm following up on this as we upgraded to Pacific 16.2.9 and deletes > are still incredibly slow. The pool rgw is using is a fairly small > erasure coding pool set at 8 + 3. Is th

[ceph-users] Changes to Crush Weight Causing Degraded PGs instead of Remapped

I have a brand new Cluster 16.2.9 running bluestore with 0 client activity. I am modifying some crush weights to move PGs off of a host for testing purposes but the result is that the PGs go into a degraded+remapped state instead of simply a remapped state. This is a strange result to me as in prev

[ceph-users] Re: Changes to Crush Weight Causing Degraded PGs instead of Remapped

ght osd.1 0.0 ? > > Istvan Szabo > Senior Infrastructure Engineer > --- > Agoda Services Co., Ltd. > e: istvan.sz...@agoda.com > ------- > > On 2022. Jun 14., at 0

[ceph-users] Re: Changes to Crush Weight Causing Degraded PGs instead of Remapped

that in a quick search. > > [1] > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/H4L5VNQJKIDXXNY2TINEGUGOYLUTT5UL/ > > Zitat von Wesley Dillingham : > > > Thanks for the reply. I believe regarding "0" vs "0.0" its the same > > difference.

[ceph-users] Re: Changes to Crush Weight Causing Degraded PGs instead of Remapped

cache={type=binned_lru} L P" \ reshard Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Tue, Jun 14, 2022 at 11:31 AM Wesley Dillingham wrote: > I have made https://tracker.ceph.com/issues/56046 regarding the iss

[ceph-users] PGs stuck deep-scrubbing for weeks - 16.2.9

We have two clusters one 14.2.22 -> 16.2.7 -> 16.2.9 Another 16.2.7 -> 16.2.9 Both with a multi disk (spinner block / ssd block.db) and both CephFS around 600 OSDs each with combo of rep-3 and 8+3 EC data pools. Examples of stuck scrubbing PGs from all of the pools. They have generally been behi

[ceph-users] Re: PGs stuck deep-scrubbing for weeks - 16.2.9

; fix, see if it fits what you've encountered: >> >> https://github.com/ceph/ceph/pull/46727 (backport to Pacific here: >> https://github.com/ceph/ceph/pull/46877 ) >> https://tracker.ceph.com/issues/54172 >> >> On Fri, Jul 15, 2022 at 8:52 AM Wesley Dilling

[ceph-users] rh8 krbd mapping causes no match of type 1 in addrvec problem decoding monmap, -2

I have a strange error when trying to map via krdb on a RH (alma8) release / kernel 4.18.0-372.13.1.el8_6.x86_64 using ceph client version 14.2.22 (cluster is 14.2.16) the rbd map causes the following error in dmesg: [Tue Jul 19 07:45:00 2022] libceph: no match of type 1 in addrvec [Tue Jul 19 07

[ceph-users] Re: rh8 krbd mapping causes no match of type 1 in addrvec problem decoding monmap, -2

:00 AM Wesley Dillingham wrote: > I have a strange error when trying to map via krdb on a RH (alma8) release > / kernel 4.18.0-372.13.1.el8_6.x86_64 using ceph client version 14.2.22 > (cluster is 14.2.16) > > the rbd map causes the following error in dmesg: > > [Tue Jul 19

[ceph-users] Re: rh8 krbd mapping causes no match of type 1 in addrvec problem decoding monmap, -2

com/in/wesleydillingham> On Tue, Jul 19, 2022 at 12:51 PM Ilya Dryomov wrote: > On Tue, Jul 19, 2022 at 5:01 PM Wesley Dillingham > wrote: > > > > I have a strange error when trying to map via krdb on a RH (alma8) > release > > / kernel 4.18.0-372.13.1.el8_6.x86_64

[ceph-users] Re: rh8 krbd mapping causes no match of type 1 in addrvec problem decoding monmap, -2

2 at 9:12 PM Wesley Dillingham > wrote: > > > > > > from ceph.conf: > > > > mon_host = 10.26.42.172,10.26.42.173,10.26.42.174 > > > > map command: > > rbd --id profilerbd device map win-rbd-test/originalrbdfromsnap > > > > [root@a2tlom

[ceph-users] Using cloudbase windows RBD / wnbd with pre-pacific clusters

I understand that the client side code available from cloudbase started being distributed with pacific and now quincy client code but is there any particular reason it shouldn't work in conjunction with a nautilus, for instance, cluster. We have seen some errors when trying to do IO with mapped RB

[ceph-users] Re: Quincy full osd(s)

Can you send along the return of "ceph osd pool ls detail" and "ceph health detail" On Sun, Jul 24, 2022, 1:00 AM Nigel Williams wrote: > With current 17.2.1 (cephadm) I am seeing an unusual HEALTH_ERR > Adding files to a new empty cluster, replica 3 (crush is by host), OSDs > became 95% full an

[ceph-users] Re: Map RBD to multiple nodes (line NFS)

You probably want CephFS instead RBD. Overview here: https://docs.ceph.com/en/quincy/cephfs/ Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Mon, Jul 25, 2022 at 11:00 AM Thomas Schneider <74cmo...@gmail.com> wrote: > Hi, > > I ha

[ceph-users] Re: Full cluster, new OSDS not being used

Can you please send the output of "ceph osd tree" Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Tue, Aug 23, 2022 at 10:53 AM Wyll Ingersoll < wyllys.ingers...@keepertech.com> wrote: > > We have a large cluster with a many osds

[ceph-users] Re: Full cluster, new OSDS not being used

are very lightly used > at this point, only a few PGs have been assigned to them, though more than > zero and the number does appear to be slowly (very slowly) growing so > recovery is happening but very very slowly. > > > > > -- > *From:* W

[ceph-users] Re: Full cluster, new OSDS not being used

have increased backfill settings, but can you elaborate on > "injecting upmaps" ? > ---------- > *From:* Wesley Dillingham > *Sent:* Tuesday, August 23, 2022 1:44 PM > *To:* Wyll Ingersoll > *Cc:* ceph-users@ceph.io > *Subject:* Re: [ceph-users] Ful

[ceph-users] Re: Increasing number of unscrubbed PGs

what does "ceph pg ls scrubbing" show? Do you have PGs that have been stuck in a scrubbing state for a long period of time (many hours,days,weeks etc). This will show in the "SINCE" column. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

I haven't read through this entire thread so forgive me if already mentioned: What is the parameter "bluefs_buffered_io" set to on your OSDs? We once saw a terrible slowdown on our OSDs during snaptrim events and setting bluefs_buffered_io to true alleviated that issue. That was on a nautilus clus

[ceph-users] Re: Power outage recovery

What does "ceph status" "ceph health detail" etc show, currently? Based on what you have said here my thought is you have created a new monitor quorum and as such all auth details from the old cluster are lost including any and all mgr cephx auth keys, so what does the log for the mgr say? How man

  1   2   >