[ceph-users] Re: OSDs are not utilized evenly

Hi Joseph replying to autoscaler question - no I don't use it. On 11/4/22 22:45, Joseph Mundackal wrote: Hi Denis, can you share the following data points? ceph osd df tree (to see how the osd's are distributed) ceph osd crush rule dump (to see what your ec rule looks like) ceph osd pool ls

[ceph-users] all monitors deleted, state recovered using documentation .. at what point to start osds ?

Hi, Unfortunately, all 3 monitors were lost. I followed this -> https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-mon/#mon-store-recovery-using-osds and it is in the current state now. id: 234c6a96-8101-49d1-b354-1110e759d572 health: HEALTH_WARN mon is

[ceph-users] Re: Recent ceph.io Performance Blog Posts

On 11/8/22 14:59, Marc wrote: 2. https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/ Very nice! Thanks Marc! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send

[ceph-users] Re: Recent ceph.io Performance Blog Posts

> 2. > https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/ > Very nice! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Recent ceph.io Performance Blog Posts

Hi Folks, I thought I would mention that I've released a couple of performance articles on the Ceph blog recently that might be of interest to people: 1. https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/ 2.

[ceph-users] Re: CephFS constant high write I/O to the metadata pool

Hi Milind, Here's are the output of top and a pstack backtrace: https://gist.github.com/olliRJL/5f483c6bc4ad50178c8c9871370b26d3 https://gist.github.com/olliRJL/b83a743eca098c05d244e5c1def9046c I uploaded the debug log using ceph-post-file - hope someone can access that :) ceph-post-file:

[ceph-users] Re: scanning RGW S3 bucket contents

Hi David, Using bucket notifications sounds like the right approach here. One option with the policy would be to move the suspicious objects to a quarantine bucket, so they are not lost in case of false positives, but not harmful either. I did something similar, but with inline processing using

[ceph-users] Re: How to ... alertmanager and prometheus

Thank you both very much! I have understood things better now. I'm not sure, though, whether all URIs are adjusted properly when changing the placement of the services. Still testing... Am 08.11.22 um 17:13 schrieb Redouane Kachach Elhichou: Welcome Eugen, There are some ongoing efforts to

[ceph-users] Re: Ceph Virtual 2022 Day 5 is starting!

Hi Stefan, Sorry - you beat me to it! Two links below: https://sfo3.digitaloceanspaces.com/do-matt-sfo3/How%20we%20Operate%20Ceph%20at%20Scale.pdf https://ams3.digitaloceanspaces.com/do-matt-ams3/How%20we%20Operate%20Ceph%20at%20Scale.pdf Thanks, Matt On 2022-11-08 12:01, Stefan Kooman

[ceph-users] Re: HELP NEEDED : cephadm adopt osd crash

You can either provide an image with the adopt command (—image) or you configure it globally with ceph config set (I don’t have the exact command right now). Which image does it fail to pull? You should see that in cephadm.log. Does that node with osd.17 have access to the image repo?

[ceph-users] HELP NEEDED : cephadm adopt osd crash

Hi, We've already convert two PRODUCTION storage nodes on Octopus to cephadm without problem. On the third one, we succeeded to convert only one OSD. [root@server4 osd]# cephadm adopt --style legacy --name osd.0 Found online OSD at //var/lib/ceph/osd/ceph-0/fsid objectstore_type is bluestore

[ceph-users] Re: How to ... alertmanager and prometheus

Welcome Eugen, There are some ongoing efforts to make the whole prometheus stack config more dynamic by using the http sd configuration [1]. In fact part of the changes are already in main but they will not be available till the next Ceph official release.

[ceph-users] Re: Ceph Virtual 2022 Day 5 is starting!

On 11/8/22 15:10, Mike Perez wrote: Hi everyone, Ceph Virtual 2022 is starting! Today's topic is Scale. We will hear from Matt Vandermeulen about how Digital Ocean, a Ceph Foundation Premier member, scales Ceph for their needs. Unfortunately, our other scheduled presentation for today, "Ceph

[ceph-users] Re: RGW at all (re)deploying from scratch

Hello Fabio, we use cephadm orchestrator after upgrading to Quincy, As you said, after complete RGW redeploy, new default.rgw.(xyz) and .rgw.root pools appeared and now everything works fine. It's also described here: https://docs.ceph.com/en/quincy/radosgw/pools/ But one question. If we deploy

[ceph-users] Re: How to ... alertmanager and prometheus

I somehow missed the HA part in [1], thanks for pointing that out. Zitat von Redouane Kachach Elhichou : If you are running quincy and using cephadm then you can have more instances of prometheus (and other monitoring daemons) running in HA mode by increasing the number of daemons as in [1]:

[ceph-users] Re: How to ... alertmanager and prometheus

If you are running quincy and using cephadm then you can have more instances of prometheus (and other monitoring daemons) running in HA mode by increasing the number of daemons as in [1]: from a cephadm shell (to run 2 instances of prometheus and altertmanager): > ceph orch apply prometheus

[ceph-users] scanning RGW S3 bucket contents

Hi, Historically our RGW S3 service has been mostly used by staff to publish data to the world and external partners. However there's an upcoming project to allow external partners/collaborators to upload bulk data to us. Depending on the trust relationship with the uploaders, I can imagine

[ceph-users] Re: How to ... alertmanager and prometheus

Hi, the only information I found so far was this statement from the redhat docs [1]: When multiple services of the same type are deployed, a highly-available setup is deployed. I tried to do that in a virtual test environment (16.2.7) and it seems to work as expected. ses7-host1:~ #

[ceph-users] Re: How to check available storage with EC and different sized OSD's ?

with a m value of 1 if you lost a single OSD/failure domain you'd end up with a read only pg or cluster. usually you need at least k+1 to survive a failure domain failure depending on your min_size setting. The other thing you need to take into consideration is that the m value is for both

[ceph-users] Re: How to check available storage with EC and different sized OSD's ?

I guess it's not an issue in larger scenarios, but I hope there's some feature to inform user that pool is not safe. And what is the general rule? If k+m = #OSDs than do not use disks of different size? P. W dniu 8.11.2022 o 15:25, Paweł Kowalski pisze: Hi, I've set up a minimal EC

[ceph-users] How to check available storage with EC and different sized OSD's ?

Hi, I've set up a minimal EC setup - 3 OSDs, k=2, m=1: root@skarb:~# ceph osd df ID  CLASS    WEIGHT   REWEIGHT  SIZE RAW USE  DATA OMAP META AVAIL    %USE   VAR   PGS  STATUS [...]  9  low_hdd  2.72849   1.0  2.7 TiB  632 GiB  631 GiB  121 KiB  1.6 GiB  2.1 TiB  22.62 

[ceph-users] Ceph Virtual 2022 Day 5 is starting!

Hi everyone, Ceph Virtual 2022 is starting! Today's topic is Scale. We will hear from Matt Vandermeulen about how Digital Ocean, a Ceph Foundation Premier member, scales Ceph for their needs. Unfortunately, our other scheduled presentation for today, "Ceph and 6G: Are we ready for zettabytes?" is

[ceph-users] Re: MDS Performance and PG/PGP value

On 10/7/22 16:50, Yoann Moulin wrote: By the way, since I have set PG=256, I have much less SLOW requests than before, even I still have, the impact on my users has been reduced a lot. # zgrep -c -E 'WRN.*(SLOW_OPS|SLOW_REQUEST|MDS_SLOW_METADATA_IO)' floki.log.4.gz floki.log.3.gz

[ceph-users] Re: Ceph Virtual 2022 Begins Today!

On Tue, Nov 8, 2022 at 1:25 PM Stefan Kooman wrote: > > On 11/3/22 14:05, Mike Perez wrote: > > Hi everyone, > > > > Today is the first of our series in Ceph Virtual 2022! Our agenda will > > include a Ceph project update, community update, and telemetry talk by > > Yaarit Hatuka. Join us today

[ceph-users] Re: Ceph Virtual 2022 Begins Today!

On 11/3/22 14:05, Mike Perez wrote: Hi everyone, Today is the first of our series in Ceph Virtual 2022! Our agenda will include a Ceph project update, community update, and telemetry talk by Yaarit Hatuka. Join us today at 10:00 AM EDT / 14:00 UTC Meeting link: https://bluejeans.com/908675367

[ceph-users] Re: TOO_MANY_PGS after upgrade from Nautilus to Octupus

Hi Patrick, just disable the autoscaler everywhere (per pool and global). It is completely ignorant to load distribution considerations, IO patterns, object sizes and so on. If you know what you are doing, you will do better with little effort. You might want to take a look why it wants to

[ceph-users] Re: RGW at all (re)deploying from scratch

Hi Christoph, I’m using rgw at pacific release 16.2.10 but I obtain occasionally the same 500 error due to “invalid constraint” but it seems to be completely random error.. About the pools: yes, these are the defaults pools that were created by rgw during the first deploy so if you don’t use

[ceph-users] Re: Make Ceph available over VPN?

Den mån 7 nov. 2022 kl 13:04 skrev Sagittarius-A Black Hole : > I'm using the Tailscale VPN, which is based on wireguard and want to > make the Ceph cluster available offsite. > Now I have seen that ceph does not bind the mon to the tailscale > interface, so it is not available over it. Nmap shows

[ceph-users] RGW at all (re)deploying from scratch

Dear list members, we have a really old prod. cluster, meanwhile running Quincy(cephadm) with RBD and CephFS very well. :-) Now we would like to establish S3 object store for a customer, but it seems that our existing RGW stack of Pools and RadosGateways are somewhat broken due to lots of

[ceph-users] TOO_MANY_PGS after upgrade from Nautilus to Octupus

Hi, We are currently upgrading our cluster from Nautilus to Octupus. After upgrade of the mons and mgrs, we get warnings about the number of PGS. Which parameter did change during upgrade to explain those new warnings. Nothing else was changed. Is it risky to change the pgs/pool as proposed

[ceph-users] Re: Make Ceph available over VPN?

On 08.11.22 00:17, Sagittarius-A Black Hole wrote: All the ceph nodes are part of the vpn network, so all of the nodes can be reached: in tailscale, each host gets an additional vpn ip and can be reached over tailscale from the individual client systems (laptops) when out of the office. Ah,