[ceph-users] Re: [CEPH] OSD Memory Usage

Hello, Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is over target. Does Mem Use include cache, I am right? NAMEHOST PORTSSTATUS REFRESHED AGE MEM USE MEM LIM VERSIONIMAGE ID CONTAINER ID osd.7

[ceph-users] Re: iSCSI GW trusted IPs

Hi, I don't have a solution for you, I just wanted to make you aware of this note in the docs: Warning The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features. Here's some more

[ceph-users] Re: [CEPH] OSD Memory Usage

You're most welcome! I'd say that real leak issues are very rare. For example, these are my OSDs with memory target=16GB which have been running for quite a while, as you can see they don't exceed 16 GB RSS: PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 92298

[ceph-users] Re: [CEPH] OSD Memory Usage

Hello. Thank you very much for your explanation. Because I thought that osd_memory_target will help me limit OSD memory usage which will help prevent memory leak - I tried google and many people talked about memory leak. A nice man, @Anthony D'Atri , on this forum helped me to understand that

[ceph-users] Re: [CEPH] OSD Memory Usage

Hi, osd_memory_target is a "target", i.e. an OSD make an effort to consume up to the specified amount of RAM, but won't consume less than required for its operation and caches, which have some minimum values such as for example osd_memory_cache_min, bluestore_cache_size, bluestore_cache_size_hdd,

[ceph-users] Re: migrate wal/db to block device

Hi Igor, The immediate answer is to use "ceph-volume lvm zap" on the db LV after running the migrate. But for the longer term I think the "lvm zap" should be included in the "lvm migrate" process. I.e. this works to migrate a separate wal/db to the block device: # # WARNING! DO NOT ZAP

[ceph-users] Re: Different behaviors for ceph kernel client in limiting IOPS when data pool enters `nearfull`?

Hi Matt, On 11/15/23 02:40, Matt Larson wrote: On CentOS 7 systems with the CephFS kernel client, if the data pool has a `nearfull` status there is a slight reduction in write speeds (possibly 20-50% fewer IOPS). On a similar Rocky 8 system with the CephFS kernel client, if the data pool has

[ceph-users] remove spurious data

Hi, I’d like to remove some “spurious" data: root@nerffs03:/# ceph df --- RAW STORAGE --- CLASS SIZEAVAILUSED RAW USED %RAW USED hdd1.0 PiB 1.0 PiB 47 GiB47 GiB 0 TOTAL 1.0 PiB 1.0 PiB 47 GiB47 GiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS

[ceph-users] rasize= in ceph.conf some section?

Hello, I'm trying to make it easy to distribute the expected config settings for my ceph volumes for other admin groups. Is there a place I can set rasize in the ceph.conf where the client would pick it up? The NFS world these days has nfsmount.conf that I've grown very fond of. I'm a bit

[ceph-users] ceph -s very slow in my rdma eviroment

Today I run some ceph -s in my rdma enviroment. but very slow. After perf ceph -s to FlameGraph. I find almost all the time is spent on compact zone. Has anyone encountered it? my enviroment . rpm -qa | grep ibverb libibverbs-41mlnx1-OFED.4.1.0.1.1.41102.x86_64

[ceph-users] planning upgrade from pacific to quincy

Hi All (apologies if you get this twice, I suspect mails from my @science.ru.nl account get dropped by most receiving mail servers, due to the strict DMARC policies in place) after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we

[ceph-users] Issue with using the block device inside a pod.

Hi Team, Components: Kubernetes, Ceph Problem statement: We are trying to integrate Ceph with kubernetes. We are unable to utilize the block volume mode in the pod. Description: OS: Almalinux 8.8 Ceph version: 18.2 Kubernetes version: 1.28.2 We have deployed a single node kubernetes cluster

[ceph-users] Re: Ceph Allocation - used space is unreasonably higher than stored space

Thank you Igor, Yeah the 25K waste per rados object seems reasonable, couple of questions though: 1. Is the story of blobs re-using empty sub-sections of already allocated "min_alloc_size"ed blocks, just for RBD/FS? I read some blogs about onode->extent->blob->min_alloc->pextent->disk flow

[ceph-users] Re: CephFS mirror very slow (maybe for small files?)

Hi Jos, I have tried adding multiple daemons but it seems only 1 is active, and there is no improvement in throughput. On further reading, you suggestion conflicts with the docs

[ceph-users] Re: CephFS mirror very slow (maybe for small files?)

Thankyou Jos. I will try the multiple daemons to see how that helps. It looks like I need to wait for the fix [1] to be in a release (currently pending review) before I can apply it. Stuart ___ ceph-users mailing list -- ceph-users@ceph.io To

[ceph-users] Re: reef 18.2.1 QE Validation status

Yes, cephadm has some tests for monitoring that should be enough to ensure basic functionality is working properly. The rest of the changes in the PR are for rook orchestrator. On Tue, Nov 14, 2023 at 5:04 AM Nizamudeen A wrote: > dashboard changes are minimal and approved. and since the

[ceph-users] Upgrading From RHCS v4 to OSS Ceph

Hi everyone, I have a storage cluster running RHCS v4 (old, I know) and am looking to upgrade it soon. I would also like to migrate from RHCS to the open source version of Ceph at some point, as our support contract with RedHat for Ceph is likely going to not be renewed going forward. I was

[ceph-users] Re: reef 18.2.1 QE Validation status

Hi Yuri, I've just backported to reef several fixes that I introduced in the last months for the rook orchestrator. Most of them are fixes for dashboard issues/crashes that only happen on Rook environments. The PR [1] has all the changes and it was merged into reef this morning. We really need

[ceph-users] Re: Ceph Allocation - used space is unreasonably higher than stored space

Igor Fedotov wrote: > Hi Motahare, > > On 13/11/2023 14:44, Motahare S wrote: > > Hello everyone, > > > > Recently we have noticed that the results of "ceph df" stored and used > > space does not match; as the amount of stored data *1.5 (ec factor) is > > still like 5TB away from used

[ceph-users] [CEPH] OSD Memory Usage

Hello, I am using a CEPH cluster. After monitoring it, I set: ceph config set osd osd_memory_target_autotune false ceph config set osd osd_memory_target 1G Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB.

[ceph-users] Re: Reinitialize rgw garbage collector

Hello Michael, Did you receive any help on this ? We are issuing the same problem without solution for now. Regard -- Pierre ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.1 QE Validation status

rados approved! Details are here: https://tracker.ceph.com/projects/rados/wiki/REEF#1821-Review. On Mon, Nov 6, 2023 at 10:33 PM Yuri Weinstein wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/63443#note-1 > > Seeking approvals/reviews for: > > smoke -

[ceph-users] Re: reef 18.2.1 QE Validation status

On Tue, Nov 7, 2023 at 4:12 PM Adam King wrote: > I think the orch code itself is doing fine, but a bunch of tests are > failing due to https://tracker.ceph.com/issues/63151. I think that's > likely related to the ganesha build we have included in the container and > if we want nfs over rgw to

[ceph-users] Re: Large size differences between pgs

Namely, the problem what I am trying to solve is that with such a large cluster I will lose a lot of  capacity like unused. I have deviation set to value 1 at the balancer, that is, if I'm not mistaken +-1pg per OSD and then due to the size dispersion between the largest and smallest PGs on the

[ceph-users] Large size differences between pgs

Hi,is it possible decrease large size differences between pgs? I have 5PB cluster and differences between smalest and bigest pgs are somewhere about 25GB.thanks,Svoboda Miroslav ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an

[ceph-users] Re: reef 18.2.1 QE Validation status

The tests were re-run with Guillaume's changes and are passing now! Thanks, Travis On Wed, Nov 15, 2023 at 1:19 PM Yuri Weinstein wrote: > Sounds like it's a must to be added. > > When the reef backport PR can be merged? > > On Wed, Nov 15, 2023 at

[ceph-users] Re: reef 18.2.1 QE Validation status

Sounds like it's a must to be added. When the reef backport PR can be merged? On Wed, Nov 15, 2023 at 12:13 PM Travis Nielsen wrote: > > Thanks Guiilaume and Redo for tracking down this issue. After talking more > with Guillaume I now realized that not all the tests were using the expected >

[ceph-users] Re: reef 18.2.1 QE Validation status

Thanks Guiilaume and Redo for tracking down this issue. After talking more with Guillaume I now realized that not all the tests were using the expected latest-reef-devel label so Rook tests were incorrectly showing green for Reef. :( Now that I ran the tests again in the test PR

[ceph-users] Re: reef 18.2.1 QE Validation status

Hi Yuri, (thanks) Indeed, we had a regression in ceph-volume impacting rook scenarios which was supposed to be fixed by [1]. It turns out rook's CI didn't catch that fix wasn't enough for some reason (I believe the CI run wasn't using the right image, Travis might confirm or give more

[ceph-users] Re: reef 18.2.1 QE Validation status

This is on behalf of Guillaume. We have one more last mites issue that may have to be included https://tracker.ceph.com/issues/63545 https://github.com/ceph/ceph/pull/54514 Travis, Redo, Guillaume will provide more context and details. We are assessing the situation as 18.2.1 has been built and

[ceph-users] Re: Debian 12 support

On 11/15/23 19:52, Daniel Baumann wrote: > for 18.2.0, there's only one trivial thing needed: > https://git.progress-linux.org/packages/graograman-backports-extras/ceph/commit/?id=ed59c69244ec7b81ec08f7a2d1a1f0a90e765de0 or, for mainline inclusion, an alternative depends would be suitable too:

[ceph-users] Re: per-rbd snapshot limitation

I don't think this parameter exists (today) Le mer. 15 nov. 2023 à 19:25, Wesley Dillingham a écrit : > Are you aware of any config item that can be set (perhaps in the ceph.conf > or config db) so the limit is enforced immediately at creation time without > needing to set it for each rbd? > >

[ceph-users] Re: Debian 12 support

On 11/15/23 19:31, Gregory Farnum wrote: > There are versioning and dependency issues for 18.2.0, there's only one trivial thing needed: https://git.progress-linux.org/packages/graograman-backports-extras/ceph/commit/?id=ed59c69244ec7b81ec08f7a2d1a1f0a90e765de0 then, the packages build

[ceph-users] Re: Debian 12 support

There are versioning and dependency issues (both of packages, and compiler toolchain pieces) which mean that the existing reef releases do not build on Debian. Our upstream support for Debian has always been inconsistent because we don’t have anybody dedicated or involved enough in both Debian and

[ceph-users] Re: per-rbd snapshot limitation

Are you aware of any config item that can be set (perhaps in the ceph.conf or config db) so the limit is enforced immediately at creation time without needing to set it for each rbd? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On

[ceph-users] Re: Join us for the User + Dev Monthly Meetup - November 16!

Hi Ceph users and developers, I wanted to inform you about a change in the agenda for tomorrow's User + Dev meeting. Our originally scheduled speaker, Christian Theune, needs to reschedule his presentation on "Operational Reliability and Flexibility in Ceph Upgrades" to a later date. However,

[ceph-users] Re: per-rbd snapshot limitation

rbd create testpool/test3 --size=100M rbd snap limit set testpool/test3 --limit 3 Le mer. 15 nov. 2023 à 17:58, Wesley Dillingham a écrit : > looking into how to limit snapshots at the ceph level for RBD snapshots. > Ideally ceph would enforce an arbitrary number of snapshots allowable per >

[ceph-users] Re: per-rbd snapshot limitation

Perfect, thank you. Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn On Wed, Nov 15, 2023 at 1:00 PM Ilya Dryomov wrote: > On Wed, Nov 15, 2023 at 5:57 PM Wesley Dillingham > wrote: > > > > looking into how to limit snapshots at

[ceph-users] Re: per-rbd snapshot limitation

On Wed, Nov 15, 2023 at 5:57 PM Wesley Dillingham wrote: > > looking into how to limit snapshots at the ceph level for RBD snapshots. > Ideally ceph would enforce an arbitrary number of snapshots allowable per > rbd. > > Reading the man page for rbd command I see this option: >

[ceph-users] per-rbd snapshot limitation

looking into how to limit snapshots at the ceph level for RBD snapshots. Ideally ceph would enforce an arbitrary number of snapshots allowable per rbd. Reading the man page for rbd command I see this option: https://docs.ceph.com/en/quincy/man/8/rbd/#cmdoption-rbd-limit --limit Specifies the

[ceph-users] Ceph Leadership Team Meeting Minutes Nov 15, 2023

Hi Cephers, These are the topics discussed today: - 18.2.1 - Almost ready, packages built/signed - Plan to release on Monday - Last minute PR for Rook - Lab update to be finished by tomorrow - Finalize CDM APAC time - Review

[ceph-users] Re: iSCSI GW trusted IPs

I just setup iscisi on a reef cluster and I couldn’t add targets properly until I put in the username and password entered for the gateways via the "Discovery Authentication" button at the top of the targets page in the iscsi area. I don’t remember if the quincy console had that though. In my

[ceph-users] Re: migrate wal/db to block device

Hi Igor, On Wed, Nov 15, 2023 at 12:30:57PM +0300, Igor Fedotov wrote: Hi Chris, haven't checked you actions thoroughly but migration to be done on a down OSD which is apparently not the case here. May be that's a culprit and we/you somehow missed the relevant error during the migration

[ceph-users] iSCSI GW trusted IPs

Hi, I’m configuring the iSCSI GW services on a quincy 17.2.3 cluster. I brought almost everything up and running (using cephadm), but I’m stuck in a configuration detail: if I check the gateway status in the Block -> iSCSI -> Overview section of the dashboard, they’re showing “Down”

[ceph-users] planning upgrade from pacific to quincy

Hi All after a long while being in health_err state (due to an unfound object, which we eventually decided to "forget"), we are now planning to upgrade our cluster which is running Pacific (at least on the mons/mdss/osds, the gateways are by accident running quincy already). The installation

[ceph-users] Re: migrate wal/db to block device

Oh right, I responded from my mobile phone and missed the examples. Thanks for the clarification! OP did stop the OSD according to his output: $ cephadm  unit --fsid ${fsid} --name osd.${osdid} stop But there might have been an error anyway, I guess. Zitat von Igor Fedotov : Hi Eugen,

[ceph-users] How to configure something like osd_deep_scrub_min_interval?

Hi folks, I am fighting a bit with odd deep-scrub behavior on HDDs and discovered a likely cause of why the distribution of last_deep_scrub_stamps is so weird. I wrote a small script to extract a histogram of scrubs by "days not scrubbed" (more precisely, intervals not scrubbed; see code) to

[ceph-users] Re: migrate wal/db to block device

Hi Chris, haven't checked you actions thoroughly but migration to be done on a down OSD which is apparently not the case here. May be that's a culprit and we/you somehow missed the relevant error during the migration process? Thanks, Igor On 11/15/2023 5:33 AM, Chris Dunlop wrote: Hi,

[ceph-users] Re: migrate wal/db to block device

Hi Eugen, this scenario is supported, see the last example on the relevant doc page: Moves BlueFS data from main, DB and WAL devices to main device, WAL and DB are removed: ceph-volume lvm migrate --osd-id 1 --osd-fsid--from db wal --target vgname/data Thanks, Igor On

[ceph-users] Re: RGW: user modify default_storage_class does not work

Thanks for your reply. You are right, newly-created bucket will now have "placement_rule": "default-placement/COLD". But then I have another question that can we specify the default storage class when creating a new bucket? I found a way to set placement but not with storage class:

[ceph-users] Re: Stretch mode size

No it’s not too late, it will take some time till we get there. So thanks for the additional input, I am aware of the MON communication. Zitat von Sake Ceph : Don't forget with stretch mode, osds only communicate with mons in the same DC and the tiebreaker only communicate with the other

[ceph-users] Re: migrate wal/db to block device

Hi, AFAIU, you can’t migrate back to the slow device. It’s either migrating from the slow device to a fast device or remove between fast devices. I’m not aware that your scenario was considered in that tool. The docs don’t specifically say that, but they also don’t mention going back to

[ceph-users] Re: Stretch mode size

Don't forget with stretch mode, osds only communicate with mons in the same DC and the tiebreaker only communicate with the other mons (to prevent split brain scenarios). Little late response, but I wanted you to know this :) ___ ceph-users mailing