[ceph-users] Re: [Suspicious newsletter] Weird performance issue with long heartbeat and slow ops warnings

Hi, We have a quite serious issue regarding slow ops. In our case DB team used the cluster to read and write in the same pool at the same time and it made the cluster useless. When we ran fio, we realised that ceph doesn't like the read and write at the same time in the same pool, so we tested

[ceph-users] Re: [Suspicious newsletter] Weird performance issue with long heartbeat and slow ops warnings

Thanks Istvan. I did some more investigation and what I found is that if I run FIO with 100% write on an already warm volume, then the performance degradation doesn't happen. In other words, 100% write OPS on an empty volume causes performance degradation while subsequent read/writes on a volume

[ceph-users] pg active+clean but can not handle io

Hi all: I have a ceph cluster, the version is 12.2.12. this is my ceph osd tree: [root@node-1 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -25 2.78760 root rack-test -26 0.92920 rack rack_1

[ceph-users] Re: Ceph OIDC Integration

Hello, If it is possible for the uid that has been used for LDAP users to be the same for OIDC users (which is based off the 'sub' field of the OpenID connect token), then there are no extra migration steps needed. Which version of Ceph are you using? In octopus, offline token validation has

[ceph-users] Bluestore migration: per-osd device copy

Hi, The docs have scant detail on doing a migration to bluestore using a per-osd device copy: https://docs.ceph.com/en/latest/rados/operations/bluestore-migration/#per-osd-device-copy This mentions "using the copy function of ceph-objectstore-tool", but ceph-objectstore-tool doesn't have a

[ceph-users] Re: el6 / centos6 rpm's for luminous?

Ok thanks Dan for letting me know. -Original Message- Cc: ceph-users Subject: Re: [ceph-users] el6 / centos6 rpm's for luminous? We had built some rpms locally for ceph-fuse, but AFAIR luminous needs systemd so the server rpms would be difficult. -- dan > > > Nobody ever used

[ceph-users] Re: Multisite replication speed

With a long distance link I would definitely look into switching to BBR for your congestion control as your first step. Well, your _first_ step is to do an iperf and establish a baseline A quick search and this link seems to explain it not-too-bad

[ceph-users] Re: Wipe an Octopus install

Thanks. PDSH will help too. Sam Liston (sam.lis...@utah.edu) === Center for High Performance Computing 155 S. 1452 E. Rm 405 Salt Lake City, Utah 84112 (801)232-6932 === On Oct 8, 2020, at 8:10 AM, Eugen Block

[ceph-users] Multisite replication speed

Hello everybody, We have two Ceph object clusters replicating over a very long-distance WAN link. Our version of Ceph is 14.2.10. Currently, replication speed seems to be capped around 70 MiB/s even if there's a 10Gb WAN link between the two clusters. The clusters themselves don't seem to

[ceph-users] Re: Wipe an Octopus install

Ah, if you run 'cephadm rm-cluster --fsid ...' on each node it will remove all containers and configs (ceph-salt comes in handy with this). You'll still have to wipe the drives though, but nevertheless it's a little quicker than doing it all manually. Zitat von Samuel Taylor Liston :

[ceph-users] Fwd: pool pgp_num not updated

Just making sure this makes the list: Mac Wynkoop -- Forwarded message - From: 胡 玮文 Date: Wed, Oct 7, 2020 at 9:00 PM Subject: Re: pool pgp_num not updated To: Mac Wynkoop Hi, You can read about this behavior at

[ceph-users] Re: pool pgp_num not updated

OK, great. We'll keep tabs on it for now then and try again once we're fully rebalanced. Mac Wynkoop, Senior Datacenter Engineer *NetDepot.com:* Cloud Servers; Delivered Houston | Atlanta | NYC | Colorado Springs 1-844-25-CLOUD Ext 806 On Thu, Oct 8, 2020 at 2:08 AM Eugen Block wrote: >

[ceph-users] Re: Error "Operation not permitted" using rbd pool init command

Haha, how could I miss that? :) Anyway, I really appriciate the help and I can confirm that it was the issue. Thanks! Best regards, Fredrik Från: Janne Johansson Skickat: den 8 oktober 2020 13:01 Till: floda Kopia: ceph-users@ceph.io Ämne: Re: [ceph-users]

[ceph-users] Re: Error "Operation not permitted" using rbd pool init command

Den tors 8 okt. 2020 kl 10:25 skrev floda : > Hi guys, > I run the commands as the Linux root user and as the Ceph user > client.admin (I have turned off apparmor and other hardening things as > well). The chep user client.admin has the following setup in its keyring: > [client.admin] > key =

[ceph-users] Re: el6 / centos6 rpm's for luminous?

We had built some rpms locally for ceph-fuse, but AFAIR luminous needs systemd so the server rpms would be difficult. -- dan On Thu, Oct 8, 2020 at 11:12 AM Marc Roos wrote: > > > Nobody ever used luminous on el6? > > > ___ > ceph-users mailing list

[ceph-users] el6 / centos6 rpm's for luminous?

Nobody ever used luminous on el6? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] What are mon.-safe containers?

Hi, When I run `ceph orch ps` I see a couple of containers running on our MON nodes whose names end with the `-safe` suffix, and I was wondering what they are? I couldn't find information about it in https://docs.ceph.com This cluster is running Ceph 15.2.5, recently upgraded from 15.2.4 Many

[ceph-users] Re: Wipe an Octopus install

I honestly do not get what the problem is. Just yum remove the rpm's, dd your osd drives, if there is something left in /var/lib/ceph, /etc/ceph, rm -R -f * those. Do a find / -iname "*ceph*" if there is still something there. -Original Message- To: Samuel Taylor Liston Cc:

[ceph-users] Re: Wipe an Octopus install

Hm, not really, that command only remove the ceph.conf on my admin node. So the same as already reported in [2]. Zitat von Samuel Taylor Liston : Eugen, That sounds promising. I missed that in the man. Thanks for pointing it out. Sam Liston (sam.lis...@utah.edu)

[ceph-users] Re: pool pgp_num not updated

Yes, after your cluster has recovered you'll be able to increase pgp_num. Or your change will be applied automatically since you already set it, I'm not sure but you'll see. Zitat von Mac Wynkoop : Well, backfilling sure, but will it allow me to actually change the pgp_num as more space

[ceph-users] Weird performance issue with long heartbeat and slow ops warnings

Hello, I have a ceph cluster running 14.2.11. I am running benchmark tests with FIO concurrently on ~2000 volumes of 10G each. During the time initial warm-up FIO creates a 10G file on each volume before it runs the actual read/write I/O operations. During this time, I start seeing the Ceph