[ceph-users] Re: Setting up Hashicorp Vault for Encryption with Ceph

[Stefan Kooman]

On 15-04-2024 16:43, Michael Worsham wrote:

Is there a how-to document available on how to setup Hashicorp's Vault for 
Ceph, preferably in a HA state?


See [1] on how to do this on kubernetes.

AFAIK there is no documentation / integration on using Vault with 
Cephadm / packages.




Due to some encryption needs, we need to set up LUKS, OSD encryption AND Ceph 
bucket encryption as well. Yes, we know there will be a performance hit, but 
the encrypt-everything is a hard requirement for our business needs since we 
have government and healthcare-related contracts.


You might want to use secure mode for communication between clients / 
daemons as well [2].


Gr. Stefan

[1]: 
https://rook.io/docs/rook/latest-release/Storage-Configuration/Advanced/key-management-system/
[2]: 
https://rook.io/docs/rook/latest-release/Storage-Configuration/Advanced/key-management-system/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Have a problem with haproxy/keepalived/ganesha/docker

[Eugen Block]

Hi,

I believe I can confirm your suspicion, I have a test cluster on Reef  
18.2.1 and deployed nfs without HAProxy but with keepalived [1].
Stopping the active NFS daemon doesn't trigger anything, the MGR  
notices that it's stopped at some point, but nothing else seems to  
happen. I didn't wait too long, but I suspect that it's not supposed  
to work like that. I'll check tracker for existing issues.


Thanks,
Eugen


[1]  
https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy


Zitat von Ruslan Nurabayev :

Hello! I've installed my 5-node CEPH cluster next install NFS server  
by command:
ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip  
192.168.171.48/26 --ingress-mode haproxy-protocol.
I don't understand fully how this must be works but when i stop NFS  
daemon even on one of this nodes I've see that writing on NFS shares  
is disappear (testing via vdbench).
As i understand it is wrong and IO from stopped daemon must  
switching to another NFS daemon without any impact on IO.
Can someone help me with troubleshoot this issue? Or explain how  
done full-fledged Active-Active HA NFS Cluster for production use.


Thanks!



Руслан Нурабаев
Старший инженер
Сектор ИТ платформы
Отдел развития опорной сети
Департамент развития сети

  +77012119272
  ruslan.nuraba...@kcell.kz







-Original Message-
From: ceph-users-requ...@ceph.io 
Sent: Thursday, April 11, 2024 15:07
To: Ruslan Nurabayev 
Subject: Welcome to the "ceph-users" mailing list

[You don't often get email from ceph-users-requ...@ceph.io. Learn  
why this is important at  
https://aka.ms/LearnAboutSenderIdentification ]


Welcome to the "ceph-users" mailing list!

To post to this list, send your email to:

  ceph-users@ceph.io

You can unsubscribe or make adjustments to your options via email by  
sending a message to:


  ceph-users-requ...@ceph.io

with the word 'help' in the subject or body (don't include the  
quotes), and you will get back a message with instructions.  You  
will need your password to change your options, but for security  
purposes, this password is not included here.  If you have forgotten  
your password you will need to reset it via the web UI.





Осы хабарлама және онымен берілетін кез келген файлдар құпия болып
табылады және олар мекенжайда көрсетілген жеке немесе заңды тұлғалардың
пайдалануына ғана арналған. Егер сіз болжамды алушы болып табылмайтын
болсаңыз, осы арқылы осындай ақпаратты кез келген таратуға, жіберуге,
көшіруге немесе пайдалануға қатаң тыйым салынатыны және осы электрондық
хабарлама дереу жойылуға тиіс екендігін хабарлаймыз.
KCELL осы хабарламадағы кез келген ақпараттың дәлдігіне немесе
толықтығына қатысты ешқандай кепілдік бермейді және сол арқылы онда
қамтылған ақпарат үшін немесе оны беру, қабылдау, сақтау немесе қандай да
бір түрде пайдалану үшін кез келген жауапкершілікті болдырмайды. Осы
хабарламада айтылған пікірлер тек жіберушіге ғана тиесілі және KCELL
пікірін де білдіруі міндетті емес. Бұл электрондық хабарлама барлық
танымал компьютерлік вирустарға тексерілді.

Данное сообщение и любые передаваемые с ним файлы являются
конфиденциальными и предназначены исключительно для использования
физическими или юридическими лицами, которым они адресованы. Если вы не
являетесь предполагаемым получателем, настоящим уведомляем о том,
что любое распространение, пересылка, копирование или использование такой
информации строго запрещено, и данное электронное сообщение должно
быть немедленно удалено.
KCELL не дает никаких гарантий относительно точности или полноты любой
информации, содержащейся в данном сообщении, и тем самым исключает
любую ответственность за информацию, содержащуюся в нем, или за ее
передачу, прием, хранение или использование каким-либо образом. Мнения,
выраженные в данном сообщении, принадлежат только отправителю и
не обязательно отражают мнение KCELL.
Данное электронное сообщение было проверено на наличие всех известных
компьютерных вирусов.

This e-mail and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you are not the intended recipient you are hereby notified
that any dissemination, forwarding, copying or use of any of the
information is strictly prohibited, and the e-mail should immediately be
deleted.
KCELL makes no warranty as to the accuracy or completeness of any
information contained in this message and hereby excludes any liability
of any kind for the information contained therein or for the information
transmission, reception, storage or use of such in any way
whatsoever. The opinions expressed in this message belong to sender alone
and may

[ceph-users] Re: [EXTERN] cephFS on CentOS7

[Dietmar Rieder]

Hello,

we a run CentOS 7.9 client to access cephfs on a Ceph Reef (18.2.2) 
Cluster and it works just fine using the kernel client that comes with 
CentOS 7.9 + updates.


Best
  Dietmar

On 4/15/24 16:17, Dario Graña wrote:

Hello everyone!

We deployed a platform with Ceph Quincy and now we need to give access to
some old nodes with CentOS7 until 30/07/2024. I found two approaches, the
first one, deploying Ganesha NFS and bringing access through the NFS
protocol. The second one is to use an older cephfs client, specifically the
Octopus client.
I would like to know if there is a third option and which the community
would recommend.
Thanks in advance.

Regards!




OpenPGP_signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Have a problem with haproxy/keepalived/ganesha/docker

[Robert Sander]

Hi,

On 16.04.24 10:49, Eugen Block wrote:


I believe I can confirm your suspicion, I have a test cluster on Reef 
18.2.1 and deployed nfs without HAProxy but with keepalived [1].
Stopping the active NFS daemon doesn't trigger anything, the MGR notices 
that it's stopped at some point, but nothing else seems to happen.


There is currently no failover for NFS.

The ingress service (haproxy + keepalived) that cephadm deploys for an 
NFS cluster does not have a health check configured. Haproxy does not 
notice if a backend NFS server dies. This does not matter as there is no 
failover and the NFS client cannot be "load balanced" to another backend 
NFS server.


There is no use to configure an ingress service currently without failover.

The NFS clients have to remount the NFS share in case of their current 
NFS server dies anyway.


Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Have a problem with haproxy/keepalived/ganesha/docker

[Eugen Block]

Hm, no, I can't confirm it yet. I missed something in the config, the  
failover happens and a new nfs daemon is deployed on a different node.  
But I still see client interruptions so I'm gonna look into that first.


Zitat von Eugen Block :


Hi,

I believe I can confirm your suspicion, I have a test cluster on  
Reef 18.2.1 and deployed nfs without HAProxy but with keepalived [1].
Stopping the active NFS daemon doesn't trigger anything, the MGR  
notices that it's stopped at some point, but nothing else seems to  
happen. I didn't wait too long, but I suspect that it's not supposed  
to work like that. I'll check tracker for existing issues.


Thanks,
Eugen


[1]  
https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy


Zitat von Ruslan Nurabayev :

Hello! I've installed my 5-node CEPH cluster next install NFS  
server by command:
ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip  
192.168.171.48/26 --ingress-mode haproxy-protocol.
I don't understand fully how this must be works but when i stop NFS  
daemon even on one of this nodes I've see that writing on NFS  
shares is disappear (testing via vdbench).
As i understand it is wrong and IO from stopped daemon must  
switching to another NFS daemon without any impact on IO.
Can someone help me with troubleshoot this issue? Or explain how  
done full-fledged Active-Active HA NFS Cluster for production use.


Thanks!



Руслан Нурабаев
Старший инженер
Сектор ИТ платформы
Отдел развития опорной сети
Департамент развития сети

 +77012119272
 ruslan.nuraba...@kcell.kz







-Original Message-
From: ceph-users-requ...@ceph.io 
Sent: Thursday, April 11, 2024 15:07
To: Ruslan Nurabayev 
Subject: Welcome to the "ceph-users" mailing list

[You don't often get email from ceph-users-requ...@ceph.io. Learn  
why this is important at  
https://aka.ms/LearnAboutSenderIdentification ]


Welcome to the "ceph-users" mailing list!

To post to this list, send your email to:

 ceph-users@ceph.io

You can unsubscribe or make adjustments to your options via email  
by sending a message to:


 ceph-users-requ...@ceph.io

with the word 'help' in the subject or body (don't include the  
quotes), and you will get back a message with instructions.  You  
will need your password to change your options, but for security  
purposes, this password is not included here.  If you have  
forgotten your password you will need to reset it via the web UI.





Осы хабарлама және онымен берілетін кез келген файлдар құпия болып
табылады және олар мекенжайда көрсетілген жеке немесе заңды тұлғалардың
пайдалануына ғана арналған. Егер сіз болжамды алушы болып табылмайтын
болсаңыз, осы арқылы осындай ақпаратты кез келген таратуға, жіберуге,
көшіруге немесе пайдалануға қатаң тыйым салынатыны және осы электрондық
хабарлама дереу жойылуға тиіс екендігін хабарлаймыз.
KCELL осы хабарламадағы кез келген ақпараттың дәлдігіне немесе
толықтығына қатысты ешқандай кепілдік бермейді және сол арқылы онда
қамтылған ақпарат үшін немесе оны беру, қабылдау, сақтау немесе қандай да
бір түрде пайдалану үшін кез келген жауапкершілікті болдырмайды. Осы
хабарламада айтылған пікірлер тек жіберушіге ғана тиесілі және KCELL
пікірін де білдіруі міндетті емес. Бұл электрондық хабарлама барлық
танымал компьютерлік вирустарға тексерілді.

Данное сообщение и любые передаваемые с ним файлы являются
конфиденциальными и предназначены исключительно для использования
физическими или юридическими лицами, которым они адресованы. Если вы не
являетесь предполагаемым получателем, настоящим уведомляем о том,
что любое распространение, пересылка, копирование или использование такой
информации строго запрещено, и данное электронное сообщение должно
быть немедленно удалено.
KCELL не дает никаких гарантий относительно точности или полноты любой
информации, содержащейся в данном сообщении, и тем самым исключает
любую ответственность за информацию, содержащуюся в нем, или за ее
передачу, прием, хранение или использование каким-либо образом. Мнения,
выраженные в данном сообщении, принадлежат только отправителю и
не обязательно отражают мнение KCELL.
Данное электронное сообщение было проверено на наличие всех известных
компьютерных вирусов.

This e-mail and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you are not the intended recipient you are hereby notified
that any dissemination, forwarding, copying or use of any of the
information is strictly prohibited, and the e-mail should immediately be
deleted.
KCELL makes no warranty as to the accuracy or completeness of any
information contained in this message and her

[ceph-users] Re: Have a problem with haproxy/keepalived/ganesha/docker

[Eugen Block]

Ah, okay, thanks for the hint. In that case what I see is expected.

Zitat von Robert Sander :


Hi,

On 16.04.24 10:49, Eugen Block wrote:


I believe I can confirm your suspicion, I have a test cluster on  
Reef 18.2.1 and deployed nfs without HAProxy but with keepalived [1].
Stopping the active NFS daemon doesn't trigger anything, the MGR  
notices that it's stopped at some point, but nothing else seems to  
happen.


There is currently no failover for NFS.

The ingress service (haproxy + keepalived) that cephadm deploys for  
an NFS cluster does not have a health check configured. Haproxy does  
not notice if a backend NFS server dies. This does not matter as  
there is no failover and the NFS client cannot be "load balanced" to  
another backend NFS server.


There is no use to configure an ingress service currently without failover.

The NFS clients have to remount the NFS share in case of their  
current NFS server dies anyway.


Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Have a problem with haproxy/keepalived/ganesha/docker

[Frank Schilder]

Question about HA here: I understood the documentation of the fuse NFS client 
such that the connection state of all NFS clients is stored on ceph in rados 
objects and, if using a floating IP, the NFS clients should just recover from a 
short network timeout.

Not sure if this is what should happen with this specific HA set-up in the 
original request, but a fail-over of the NFS server ought to be handled 
gracefully by starting a new one up with the IP of the down one. Or not?

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Eugen Block 
Sent: Tuesday, April 16, 2024 11:24 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: Have a problem with haproxy/keepalived/ganesha/docker

Ah, okay, thanks for the hint. In that case what I see is expected.

Zitat von Robert Sander :

> Hi,
>
> On 16.04.24 10:49, Eugen Block wrote:
>>
>> I believe I can confirm your suspicion, I have a test cluster on
>> Reef 18.2.1 and deployed nfs without HAProxy but with keepalived [1].
>> Stopping the active NFS daemon doesn't trigger anything, the MGR
>> notices that it's stopped at some point, but nothing else seems to
>> happen.
>
> There is currently no failover for NFS.
>
> The ingress service (haproxy + keepalived) that cephadm deploys for
> an NFS cluster does not have a health check configured. Haproxy does
> not notice if a backend NFS server dies. This does not matter as
> there is no failover and the NFS client cannot be "load balanced" to
> another backend NFS server.
>
> There is no use to configure an ingress service currently without failover.
>
> The NFS clients have to remount the NFS share in case of their
> current NFS server dies anyway.
>
> Regards
> --
> Robert Sander
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> http://www.heinlein-support.de
>
> Tel: 030 / 405051-43
> Fax: 030 / 405051-19
>
> Zwangsangaben lt. §35a GmbHG:
> HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
> Geschäftsführer: Peer Heinlein -- Sitz: Berlin
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Announcing go-ceph v0.27.0

[John Mulligan]

We are happy to announce another release of the go-ceph API library. This is a
regular release following our every-two-months release cadence.

https://github.com/ceph/go-ceph/releases/tag/v0.27.0

The library includes bindings that aim to play a similar role to the "pybind"
python bindings in the ceph tree but for the Go language. The library also
includes additional APIs that can be used to administer cephfs, rbd, rgw, and 
other subsystems.
There are already a few consumers of this library in the wild, including the
ceph-csi project.


-- 
John Mulligan

phlogistonj...@asynchrono.us
jmulli...@redhat.com


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

And approval is needed for:

fs - Venky approved?
powercycle - seems fs related, Venky, Brad PTL

On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein  wrote:
>
> Still waiting for approvals:
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
>
> We plan not to upgrade the LRC to 18.2.3 as we are very close to the
> first squid RC and will be using it for this purpose.
> Please speak up if this may present any issues.
>
> Thx
>
> On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein  wrote:
> >
> > Details of this release are summarized here:
> >
> > https://tracker.ceph.com/issues/65393#note-1
> > Release Notes - TBD
> > LRC upgrade - TBD
> >
> > Seeking approvals/reviews for:
> >
> > smoke - infra issues, still trying, Laura PTL
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > rgw - Casey approved?
> > fs - Venky approved?
> > orch - Adam King approved?
> >
> > krbd - Ilya approved
> > powercycle - seems fs related, Venky, Brad PTL
> >
> > ceph-volume - will require
> > https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > Guillaume is fixing it.
> >
> > TIA
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERN] Re: Ceph 16.2.x mon compactions, disk writes

[Dietmar Rieder]

Hi Zakhar, hello List,

I just wanted to follow up on this and ask a few quesitions:

Did you noticed any downsides with your compression settings so far?
Do you have all mons now on compression?
Did release updates go through without issues?
Do you know if this works also with reef (we see massive writes as well 
there)?


Can you briefly tabulate the commands you used to persistently set the 
compression options?


Thanks so much,

  Dietmar


On 10/18/23 06:14, Zakhar Kirpichenko wrote:

Many thanks for this, Eugen! I very much appreciate yours and Mykola's
efforts and insight!

Another thing I noticed was a reduction of RocksDB store after the
reduction of the total PG number by 30%, from 590-600 MB:

65M 3675511.sst
65M 3675512.sst
65M 3675513.sst
65M 3675514.sst
65M 3675515.sst
65M 3675516.sst
65M 3675517.sst
65M 3675518.sst
62M 3675519.sst

to about half of the original size:

-rw-r--r-- 1 167 167  7218886 Oct 13 16:16 3056869.log
-rw-r--r-- 1 167 167 67250650 Oct 13 16:15 3056871.sst
-rw-r--r-- 1 167 167 67367527 Oct 13 16:15 3056872.sst
-rw-r--r-- 1 167 167 63268486 Oct 13 16:15 3056873.sst

Then when I restarted the monitors one by one before adding compression,
RocksDB store reduced even further. I am not sure why and what exactly got
automatically removed from the store:

-rw-r--r-- 1 167 167   841960 Oct 18 03:31 018779.log
-rw-r--r-- 1 167 167 67290532 Oct 18 03:31 018781.sst
-rw-r--r-- 1 167 167 53287626 Oct 18 03:31 018782.sst

Then I have enabled LZ4 and LZ4HC compression in our small production
cluster (6 nodes, 96 OSDs) on 3 out of 5
monitors: compression=kLZ4Compression,bottommost_compression=kLZ4HCCompression.
I specifically went for LZ4 and LZ4HC because of the balance between
compression/decompression speed and impact on CPU usage. The compression
doesn't seem to affect the cluster in any negative way, the 3 monitors with
compression are operating normally. The effect of the compression on
RocksDB store size and disk writes is quite noticeable:

Compression disabled, 155 MB store.db, ~125 MB RocksDB sst, and ~530 MB
writes over 5 minutes:

-rw-r--r-- 1 167 167  4227337 Oct 18 03:58 3080868.log
-rw-r--r-- 1 167 167 67253592 Oct 18 03:57 3080870.sst
-rw-r--r-- 1 167 167 57783180 Oct 18 03:57 3080871.sst

# du -hs
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/;
iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
155M
  /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
2471602 be/4 167   6.05 M473.24 M  0.00 %  0.16 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [rocksdb:low0]
2471633 be/4 167 188.00 K 40.91 M  0.00 %  0.02 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [ms_dispatch]
2471603 be/4 167  16.00 K 24.16 M  0.00 %  0.01 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [rocksdb:high0]

Compression enabled, 60 MB store.db, ~23 MB RocksDB sst, and ~130 MB of
writes over 5 minutes:

-rw-r--r-- 1 167 167  5766659 Oct 18 03:56 3723355.log
-rw-r--r-- 1 167 167 22240390 Oct 18 03:56 3723357.sst

# du -hs
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph03/store.db/;
iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
60M
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph03/store.db/
2052031 be/4 1671040.00 K 83.48 M  0.00 %  0.01 % ceph-mon -n
mon.ceph03 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [rocksdb:low0]
2052062 be/4 167   0.00 B 40.79 M  0.00 %  0.01 % ceph-mon -n
mon.ceph03 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [ms_dispatch]
2052032 be/4 167  16.00 K  4.68 M  0.00 %  0.00 % ceph-mon -n
mon.ceph03 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [rocksdb:high0]
2052052 be/4 167  44.00 K  0.00 B  0.00 %  0.00 % ceph-mon -n
mon.ceph03 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-

[ceph-users] Re: [EXTERN] Re: Ceph 16.2.x mon compactions, disk writes

[Zakhar Kirpichenko]

Hi,

>Did you noticed any downsides with your compression settings so far?

None, at least on our systems. Except the part that I haven't found a way
to make the settings persist.

>Do you have all mons now on compression?

I have 3 out of 5 monitors with compression and 2 without it. The 2
monitors with uncompressed RocksDB have much larger disks which do not
suffer from writes as much as the other 3. I keep them uncompressed "just
in case", i.e. for the unlikely event if the 3 monitors with compressed
RocksDB fail or have any issues specifically because of the compression. I
have to say that this hasn't happened yet, and this precaution may be
unnecessary.

>Did release updates go through without issues?

In our case, container updates overwrite the monitors' configurations and
reset RocksDB options, thus each updated monitor runs with no RocksDB
compression until it is added back manually. Other than that, I have not
encountered any issues related to compression during the updates.

>Do you know if this works also with reef (we see massive writes as well
there)?

Unfortunately, I can't comment on Reef as we're still using Pacific.

/Z

On Tue, 16 Apr 2024 at 18:08, Dietmar Rieder 
wrote:

> Hi Zakhar, hello List,
>
> I just wanted to follow up on this and ask a few quesitions:
>
> Did you noticed any downsides with your compression settings so far?
> Do you have all mons now on compression?
> Did release updates go through without issues?
> Do you know if this works also with reef (we see massive writes as well
> there)?
>
> Can you briefly tabulate the commands you used to persistently set the
> compression options?
>
> Thanks so much,
>
>Dietmar
>
>
> On 10/18/23 06:14, Zakhar Kirpichenko wrote:
> > Many thanks for this, Eugen! I very much appreciate yours and Mykola's
> > efforts and insight!
> >
> > Another thing I noticed was a reduction of RocksDB store after the
> > reduction of the total PG number by 30%, from 590-600 MB:
> >
> > 65M 3675511.sst
> > 65M 3675512.sst
> > 65M 3675513.sst
> > 65M 3675514.sst
> > 65M 3675515.sst
> > 65M 3675516.sst
> > 65M 3675517.sst
> > 65M 3675518.sst
> > 62M 3675519.sst
> >
> > to about half of the original size:
> >
> > -rw-r--r-- 1 167 167  7218886 Oct 13 16:16 3056869.log
> > -rw-r--r-- 1 167 167 67250650 Oct 13 16:15 3056871.sst
> > -rw-r--r-- 1 167 167 67367527 Oct 13 16:15 3056872.sst
> > -rw-r--r-- 1 167 167 63268486 Oct 13 16:15 3056873.sst
> >
> > Then when I restarted the monitors one by one before adding compression,
> > RocksDB store reduced even further. I am not sure why and what exactly
> got
> > automatically removed from the store:
> >
> > -rw-r--r-- 1 167 167   841960 Oct 18 03:31 018779.log
> > -rw-r--r-- 1 167 167 67290532 Oct 18 03:31 018781.sst
> > -rw-r--r-- 1 167 167 53287626 Oct 18 03:31 018782.sst
> >
> > Then I have enabled LZ4 and LZ4HC compression in our small production
> > cluster (6 nodes, 96 OSDs) on 3 out of 5
> > monitors:
> compression=kLZ4Compression,bottommost_compression=kLZ4HCCompression.
> > I specifically went for LZ4 and LZ4HC because of the balance between
> > compression/decompression speed and impact on CPU usage. The compression
> > doesn't seem to affect the cluster in any negative way, the 3 monitors
> with
> > compression are operating normally. The effect of the compression on
> > RocksDB store size and disk writes is quite noticeable:
> >
> > Compression disabled, 155 MB store.db, ~125 MB RocksDB sst, and ~530 MB
> > writes over 5 minutes:
> >
> > -rw-r--r-- 1 167 167  4227337 Oct 18 03:58 3080868.log
> > -rw-r--r-- 1 167 167 67253592 Oct 18 03:57 3080870.sst
> > -rw-r--r-- 1 167 167 57783180 Oct 18 03:57 3080871.sst
> >
> > # du -hs
> > /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/;
> > iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
> > 155M
> >   /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
> > 2471602 be/4 167   6.05 M473.24 M  0.00 %  0.16 % ceph-mon -n
> > mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
> > --default-log-to-stderr=true --default-log-stderr-prefix=debug
> >   --default-mon-cluster-log-to-file=false
> > --default-mon-cluster-log-to-stderr=true [rocksdb:low0]
> > 2471633 be/4 167 188.00 K 40.91 M  0.00 %  0.02 % ceph-mon -n
> > mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
> > --default-log-to-stderr=true --default-log-stderr-prefix=debug
> >   --default-mon-cluster-log-to-file=false
> > --default-mon-cluster-log-to-stderr=true [ms_dispatch]
> > 2471603 be/4 167  16.00 K 24.16 M  0.00 %  0.01 % ceph-mon -n
> > mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
> > --default-log-to-stderr=true --default-log-stderr-prefix=debug
> >   --default-mon-cluster-log-to-file=false
> > --default-mon-cluster-log-to-stderr=true [rocksdb:high0]
> >
> > Compression enabled, 60 MB store.db, ~23 MB RocksDB sst, an

[ceph-users] Re: [EXTERN] Re: Ceph 16.2.x mon compactions, disk writes

[Eugen Block]

You can use the extra container arguments I pointed out a few months  
ago. Those work in my test clusters, although I haven’t enabled that  
in production yet. But it shouldn’t make a difference if it’s a test  
cluster or not. 😉


Zitat von Zakhar Kirpichenko :


Hi,


Did you noticed any downsides with your compression settings so far?


None, at least on our systems. Except the part that I haven't found a way
to make the settings persist.


Do you have all mons now on compression?


I have 3 out of 5 monitors with compression and 2 without it. The 2
monitors with uncompressed RocksDB have much larger disks which do not
suffer from writes as much as the other 3. I keep them uncompressed "just
in case", i.e. for the unlikely event if the 3 monitors with compressed
RocksDB fail or have any issues specifically because of the compression. I
have to say that this hasn't happened yet, and this precaution may be
unnecessary.


Did release updates go through without issues?


In our case, container updates overwrite the monitors' configurations and
reset RocksDB options, thus each updated monitor runs with no RocksDB
compression until it is added back manually. Other than that, I have not
encountered any issues related to compression during the updates.


Do you know if this works also with reef (we see massive writes as well

there)?

Unfortunately, I can't comment on Reef as we're still using Pacific.

/Z

On Tue, 16 Apr 2024 at 18:08, Dietmar Rieder 
wrote:


Hi Zakhar, hello List,

I just wanted to follow up on this and ask a few quesitions:

Did you noticed any downsides with your compression settings so far?
Do you have all mons now on compression?
Did release updates go through without issues?
Do you know if this works also with reef (we see massive writes as well
there)?

Can you briefly tabulate the commands you used to persistently set the
compression options?

Thanks so much,

   Dietmar


On 10/18/23 06:14, Zakhar Kirpichenko wrote:
> Many thanks for this, Eugen! I very much appreciate yours and Mykola's
> efforts and insight!
>
> Another thing I noticed was a reduction of RocksDB store after the
> reduction of the total PG number by 30%, from 590-600 MB:
>
> 65M 3675511.sst
> 65M 3675512.sst
> 65M 3675513.sst
> 65M 3675514.sst
> 65M 3675515.sst
> 65M 3675516.sst
> 65M 3675517.sst
> 65M 3675518.sst
> 62M 3675519.sst
>
> to about half of the original size:
>
> -rw-r--r-- 1 167 167  7218886 Oct 13 16:16 3056869.log
> -rw-r--r-- 1 167 167 67250650 Oct 13 16:15 3056871.sst
> -rw-r--r-- 1 167 167 67367527 Oct 13 16:15 3056872.sst
> -rw-r--r-- 1 167 167 63268486 Oct 13 16:15 3056873.sst
>
> Then when I restarted the monitors one by one before adding compression,
> RocksDB store reduced even further. I am not sure why and what exactly
got
> automatically removed from the store:
>
> -rw-r--r-- 1 167 167   841960 Oct 18 03:31 018779.log
> -rw-r--r-- 1 167 167 67290532 Oct 18 03:31 018781.sst
> -rw-r--r-- 1 167 167 53287626 Oct 18 03:31 018782.sst
>
> Then I have enabled LZ4 and LZ4HC compression in our small production
> cluster (6 nodes, 96 OSDs) on 3 out of 5
> monitors:
compression=kLZ4Compression,bottommost_compression=kLZ4HCCompression.
> I specifically went for LZ4 and LZ4HC because of the balance between
> compression/decompression speed and impact on CPU usage. The compression
> doesn't seem to affect the cluster in any negative way, the 3 monitors
with
> compression are operating normally. The effect of the compression on
> RocksDB store size and disk writes is quite noticeable:
>
> Compression disabled, 155 MB store.db, ~125 MB RocksDB sst, and ~530 MB
> writes over 5 minutes:
>
> -rw-r--r-- 1 167 167  4227337 Oct 18 03:58 3080868.log
> -rw-r--r-- 1 167 167 67253592 Oct 18 03:57 3080870.sst
> -rw-r--r-- 1 167 167 57783180 Oct 18 03:57 3080871.sst
>
> # du -hs
> /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/;
> iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
> 155M
>   /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
> 2471602 be/4 167   6.05 M473.24 M  0.00 %  0.16 % ceph-mon -n
> mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
> --default-log-to-stderr=true --default-log-stderr-prefix=debug
>   --default-mon-cluster-log-to-file=false
> --default-mon-cluster-log-to-stderr=true [rocksdb:low0]
> 2471633 be/4 167 188.00 K 40.91 M  0.00 %  0.02 % ceph-mon -n
> mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
> --default-log-to-stderr=true --default-log-stderr-prefix=debug
>   --default-mon-cluster-log-to-file=false
> --default-mon-cluster-log-to-stderr=true [ms_dispatch]
> 2471603 be/4 167  16.00 K 24.16 M  0.00 %  0.01 % ceph-mon -n
> mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
> --default-log-to-stderr=true --default-log-stderr-prefix=debug
>   --default-mon-cluster-log-to-file=false
> --default-mon-cluster-lo

[ceph-users] Re: [EXTERN] Re: Ceph 16.2.x mon compactions, disk writes

[Eugen Block]

Sorry, I meant extra-entrypoint-arguments:

https://www.spinics.net/lists/ceph-users/msg79251.html

Zitat von Eugen Block :

You can use the extra container arguments I pointed out a few months  
ago. Those work in my test clusters, although I haven’t enabled that  
in production yet. But it shouldn’t make a difference if it’s a test  
cluster or not. 😉


Zitat von Zakhar Kirpichenko :


Hi,


Did you noticed any downsides with your compression settings so far?


None, at least on our systems. Except the part that I haven't found a way
to make the settings persist.


Do you have all mons now on compression?


I have 3 out of 5 monitors with compression and 2 without it. The 2
monitors with uncompressed RocksDB have much larger disks which do not
suffer from writes as much as the other 3. I keep them uncompressed "just
in case", i.e. for the unlikely event if the 3 monitors with compressed
RocksDB fail or have any issues specifically because of the compression. I
have to say that this hasn't happened yet, and this precaution may be
unnecessary.


Did release updates go through without issues?


In our case, container updates overwrite the monitors' configurations and
reset RocksDB options, thus each updated monitor runs with no RocksDB
compression until it is added back manually. Other than that, I have not
encountered any issues related to compression during the updates.


Do you know if this works also with reef (we see massive writes as well

there)?

Unfortunately, I can't comment on Reef as we're still using Pacific.

/Z

On Tue, 16 Apr 2024 at 18:08, Dietmar Rieder 
wrote:


Hi Zakhar, hello List,

I just wanted to follow up on this and ask a few quesitions:

Did you noticed any downsides with your compression settings so far?
Do you have all mons now on compression?
Did release updates go through without issues?
Do you know if this works also with reef (we see massive writes as well
there)?

Can you briefly tabulate the commands you used to persistently set the
compression options?

Thanks so much,

  Dietmar


On 10/18/23 06:14, Zakhar Kirpichenko wrote:

Many thanks for this, Eugen! I very much appreciate yours and Mykola's
efforts and insight!

Another thing I noticed was a reduction of RocksDB store after the
reduction of the total PG number by 30%, from 590-600 MB:

65M 3675511.sst
65M 3675512.sst
65M 3675513.sst
65M 3675514.sst
65M 3675515.sst
65M 3675516.sst
65M 3675517.sst
65M 3675518.sst
62M 3675519.sst

to about half of the original size:

-rw-r--r-- 1 167 167  7218886 Oct 13 16:16 3056869.log
-rw-r--r-- 1 167 167 67250650 Oct 13 16:15 3056871.sst
-rw-r--r-- 1 167 167 67367527 Oct 13 16:15 3056872.sst
-rw-r--r-- 1 167 167 63268486 Oct 13 16:15 3056873.sst

Then when I restarted the monitors one by one before adding compression,
RocksDB store reduced even further. I am not sure why and what exactly

got

automatically removed from the store:

-rw-r--r-- 1 167 167   841960 Oct 18 03:31 018779.log
-rw-r--r-- 1 167 167 67290532 Oct 18 03:31 018781.sst
-rw-r--r-- 1 167 167 53287626 Oct 18 03:31 018782.sst

Then I have enabled LZ4 and LZ4HC compression in our small production
cluster (6 nodes, 96 OSDs) on 3 out of 5
monitors:

compression=kLZ4Compression,bottommost_compression=kLZ4HCCompression.

I specifically went for LZ4 and LZ4HC because of the balance between
compression/decompression speed and impact on CPU usage. The compression
doesn't seem to affect the cluster in any negative way, the 3 monitors

with

compression are operating normally. The effect of the compression on
RocksDB store size and disk writes is quite noticeable:

Compression disabled, 155 MB store.db, ~125 MB RocksDB sst, and ~530 MB
writes over 5 minutes:

-rw-r--r-- 1 167 167  4227337 Oct 18 03:58 3080868.log
-rw-r--r-- 1 167 167 67253592 Oct 18 03:57 3080870.sst
-rw-r--r-- 1 167 167 57783180 Oct 18 03:57 3080871.sst

# du -hs
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/;
iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
155M
  /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
2471602 be/4 167   6.05 M473.24 M  0.00 %  0.16 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [rocksdb:low0]
2471633 be/4 167 188.00 K 40.91 M  0.00 %  0.02 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cluster-log-to-stderr=true [ms_dispatch]
2471603 be/4 167  16.00 K 24.16 M  0.00 %  0.01 % ceph-mon -n
mon.ceph04 -f --setuser ceph --setgroup ceph --default-log-to-file=false
--default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-mon-cluster-log-to-file=false
--default-mon-cl

[ceph-users] Ceph Community Management Update

[Josh Durgin]

Hi everyone,

I’d like to extend a warm thank you to Mike Perez for his years of service
as community manager for Ceph. He is changing focuses now to engineering.

The Ceph Foundation board decided to use services from the Linux Foundation
to fulfill some community management responsibilities, rather than rely on
a single member organization employing a community manager. The Linux
Foundation will assist with Ceph Foundation membership and governance
matters.

Please welcome Noah Lehman (cc’d) as our social media and marketing point
person - for anything related to this area, including the Ceph YouTube
channel, please reach out to him.

Ceph days will continue to be organized and funded by organizations around
the world, with the help of the Ceph Ambassadors (
https://ceph.io/en/community/ambassadors/). Gaurav Sitlani (cc’d) will help
organize the ambassadors going forward.

For other matters, please contact coun...@ceph.io and we’ll direct the
matter to the appropriate people.

Thanks,
Neha Ojha, Dan van der Ster, Josh Durgin
Ceph Executive Council
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERN] Re: Ceph 16.2.x mon compactions, disk writes

[Zakhar Kirpichenko]

I remember that I found the part which said "if something goes wrong,
monitors will fail" rather discouraging :-)

/Z

On Tue, 16 Apr 2024 at 18:59, Eugen Block  wrote:

> Sorry, I meant extra-entrypoint-arguments:
>
> https://www.spinics.net/lists/ceph-users/msg79251.html
>
> Zitat von Eugen Block :
>
> > You can use the extra container arguments I pointed out a few months
> > ago. Those work in my test clusters, although I haven’t enabled that
> > in production yet. But it shouldn’t make a difference if it’s a test
> > cluster or not. 😉
> >
> > Zitat von Zakhar Kirpichenko :
> >
> >> Hi,
> >>
> >>> Did you noticed any downsides with your compression settings so far?
> >>
> >> None, at least on our systems. Except the part that I haven't found a
> way
> >> to make the settings persist.
> >>
> >>> Do you have all mons now on compression?
> >>
> >> I have 3 out of 5 monitors with compression and 2 without it. The 2
> >> monitors with uncompressed RocksDB have much larger disks which do not
> >> suffer from writes as much as the other 3. I keep them uncompressed
> "just
> >> in case", i.e. for the unlikely event if the 3 monitors with compressed
> >> RocksDB fail or have any issues specifically because of the
> compression. I
> >> have to say that this hasn't happened yet, and this precaution may be
> >> unnecessary.
> >>
> >>> Did release updates go through without issues?
> >>
> >> In our case, container updates overwrite the monitors' configurations
> and
> >> reset RocksDB options, thus each updated monitor runs with no RocksDB
> >> compression until it is added back manually. Other than that, I have not
> >> encountered any issues related to compression during the updates.
> >>
> >>> Do you know if this works also with reef (we see massive writes as well
> >> there)?
> >>
> >> Unfortunately, I can't comment on Reef as we're still using Pacific.
> >>
> >> /Z
> >>
> >> On Tue, 16 Apr 2024 at 18:08, Dietmar Rieder <
> dietmar.rie...@i-med.ac.at>
> >> wrote:
> >>
> >>> Hi Zakhar, hello List,
> >>>
> >>> I just wanted to follow up on this and ask a few quesitions:
> >>>
> >>> Did you noticed any downsides with your compression settings so far?
> >>> Do you have all mons now on compression?
> >>> Did release updates go through without issues?
> >>> Do you know if this works also with reef (we see massive writes as well
> >>> there)?
> >>>
> >>> Can you briefly tabulate the commands you used to persistently set the
> >>> compression options?
> >>>
> >>> Thanks so much,
> >>>
> >>>   Dietmar
> >>>
> >>>
> >>> On 10/18/23 06:14, Zakhar Kirpichenko wrote:
>  Many thanks for this, Eugen! I very much appreciate yours and Mykola's
>  efforts and insight!
> 
>  Another thing I noticed was a reduction of RocksDB store after the
>  reduction of the total PG number by 30%, from 590-600 MB:
> 
>  65M 3675511.sst
>  65M 3675512.sst
>  65M 3675513.sst
>  65M 3675514.sst
>  65M 3675515.sst
>  65M 3675516.sst
>  65M 3675517.sst
>  65M 3675518.sst
>  62M 3675519.sst
> 
>  to about half of the original size:
> 
>  -rw-r--r-- 1 167 167  7218886 Oct 13 16:16 3056869.log
>  -rw-r--r-- 1 167 167 67250650 Oct 13 16:15 3056871.sst
>  -rw-r--r-- 1 167 167 67367527 Oct 13 16:15 3056872.sst
>  -rw-r--r-- 1 167 167 63268486 Oct 13 16:15 3056873.sst
> 
>  Then when I restarted the monitors one by one before adding
> compression,
>  RocksDB store reduced even further. I am not sure why and what exactly
> >>> got
>  automatically removed from the store:
> 
>  -rw-r--r-- 1 167 167   841960 Oct 18 03:31 018779.log
>  -rw-r--r-- 1 167 167 67290532 Oct 18 03:31 018781.sst
>  -rw-r--r-- 1 167 167 53287626 Oct 18 03:31 018782.sst
> 
>  Then I have enabled LZ4 and LZ4HC compression in our small production
>  cluster (6 nodes, 96 OSDs) on 3 out of 5
>  monitors:
> >>> compression=kLZ4Compression,bottommost_compression=kLZ4HCCompression.
>  I specifically went for LZ4 and LZ4HC because of the balance between
>  compression/decompression speed and impact on CPU usage. The
> compression
>  doesn't seem to affect the cluster in any negative way, the 3 monitors
> >>> with
>  compression are operating normally. The effect of the compression on
>  RocksDB store size and disk writes is quite noticeable:
> 
>  Compression disabled, 155 MB store.db, ~125 MB RocksDB sst, and ~530
> MB
>  writes over 5 minutes:
> 
>  -rw-r--r-- 1 167 167  4227337 Oct 18 03:58 3080868.log
>  -rw-r--r-- 1 167 167 67253592 Oct 18 03:57 3080870.sst
>  -rw-r--r-- 1 167 167 57783180 Oct 18 03:57 3080871.sst
> 
>  # du -hs
> 
> /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/;
>  iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
>  155M
> 
>  /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
>  247160

[ceph-users] Re: [EXTERN] Re: Ceph 16.2.x mon compactions, disk writes

[Eugen Block]

I understand, I just wanted to point out to be careful (as you always  
should be with critical services like MONs). If you apply it to one  
mon first you will see if it works or not. Then apply it to the other  
ones.


Zitat von Zakhar Kirpichenko :


I remember that I found the part which said "if something goes wrong,
monitors will fail" rather discouraging :-)

/Z

On Tue, 16 Apr 2024 at 18:59, Eugen Block  wrote:


Sorry, I meant extra-entrypoint-arguments:

https://www.spinics.net/lists/ceph-users/msg79251.html

Zitat von Eugen Block :

> You can use the extra container arguments I pointed out a few months
> ago. Those work in my test clusters, although I haven’t enabled that
> in production yet. But it shouldn’t make a difference if it’s a test
> cluster or not. 😉
>
> Zitat von Zakhar Kirpichenko :
>
>> Hi,
>>
>>> Did you noticed any downsides with your compression settings so far?
>>
>> None, at least on our systems. Except the part that I haven't found a
way
>> to make the settings persist.
>>
>>> Do you have all mons now on compression?
>>
>> I have 3 out of 5 monitors with compression and 2 without it. The 2
>> monitors with uncompressed RocksDB have much larger disks which do not
>> suffer from writes as much as the other 3. I keep them uncompressed
"just
>> in case", i.e. for the unlikely event if the 3 monitors with compressed
>> RocksDB fail or have any issues specifically because of the
compression. I
>> have to say that this hasn't happened yet, and this precaution may be
>> unnecessary.
>>
>>> Did release updates go through without issues?
>>
>> In our case, container updates overwrite the monitors' configurations
and
>> reset RocksDB options, thus each updated monitor runs with no RocksDB
>> compression until it is added back manually. Other than that, I have not
>> encountered any issues related to compression during the updates.
>>
>>> Do you know if this works also with reef (we see massive writes as well
>> there)?
>>
>> Unfortunately, I can't comment on Reef as we're still using Pacific.
>>
>> /Z
>>
>> On Tue, 16 Apr 2024 at 18:08, Dietmar Rieder <
dietmar.rie...@i-med.ac.at>
>> wrote:
>>
>>> Hi Zakhar, hello List,
>>>
>>> I just wanted to follow up on this and ask a few quesitions:
>>>
>>> Did you noticed any downsides with your compression settings so far?
>>> Do you have all mons now on compression?
>>> Did release updates go through without issues?
>>> Do you know if this works also with reef (we see massive writes as well
>>> there)?
>>>
>>> Can you briefly tabulate the commands you used to persistently set the
>>> compression options?
>>>
>>> Thanks so much,
>>>
>>>   Dietmar
>>>
>>>
>>> On 10/18/23 06:14, Zakhar Kirpichenko wrote:
 Many thanks for this, Eugen! I very much appreciate yours and Mykola's
 efforts and insight!

 Another thing I noticed was a reduction of RocksDB store after the
 reduction of the total PG number by 30%, from 590-600 MB:

 65M 3675511.sst
 65M 3675512.sst
 65M 3675513.sst
 65M 3675514.sst
 65M 3675515.sst
 65M 3675516.sst
 65M 3675517.sst
 65M 3675518.sst
 62M 3675519.sst

 to about half of the original size:

 -rw-r--r-- 1 167 167  7218886 Oct 13 16:16 3056869.log
 -rw-r--r-- 1 167 167 67250650 Oct 13 16:15 3056871.sst
 -rw-r--r-- 1 167 167 67367527 Oct 13 16:15 3056872.sst
 -rw-r--r-- 1 167 167 63268486 Oct 13 16:15 3056873.sst

 Then when I restarted the monitors one by one before adding
compression,
 RocksDB store reduced even further. I am not sure why and what exactly
>>> got
 automatically removed from the store:

 -rw-r--r-- 1 167 167   841960 Oct 18 03:31 018779.log
 -rw-r--r-- 1 167 167 67290532 Oct 18 03:31 018781.sst
 -rw-r--r-- 1 167 167 53287626 Oct 18 03:31 018782.sst

 Then I have enabled LZ4 and LZ4HC compression in our small production
 cluster (6 nodes, 96 OSDs) on 3 out of 5
 monitors:
>>> compression=kLZ4Compression,bottommost_compression=kLZ4HCCompression.
 I specifically went for LZ4 and LZ4HC because of the balance between
 compression/decompression speed and impact on CPU usage. The
compression
 doesn't seem to affect the cluster in any negative way, the 3 monitors
>>> with
 compression are operating normally. The effect of the compression on
 RocksDB store size and disk writes is quite noticeable:

 Compression disabled, 155 MB store.db, ~125 MB RocksDB sst, and ~530
MB
 writes over 5 minutes:

 -rw-r--r-- 1 167 167  4227337 Oct 18 03:58 3080868.log
 -rw-r--r-- 1 167 167 67253592 Oct 18 03:57 3080870.sst
 -rw-r--r-- 1 167 167 57783180 Oct 18 03:57 3080871.sst

 # du -hs

/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/;
 iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
 155M

 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/
 2471602 be/

[ceph-users] Re: reef 18.2.3 QE validation status

[Laura Flores]

On behalf of @Radoslaw Zarzynski , rados approved.

Below is the summary of the rados suite failures, divided by component. @Adam
King  @Venky Shankar  PTAL at the
orch and cephfs failures to see if they are blockers.

Failures, unrelated:

RADOS:
1. https://tracker.ceph.com/issues/65183 - Overriding an EC pool needs
the "--yes-i-really-mean-it" flag in addition to "force"
3. https://tracker.ceph.com/issues/62992 - Heartbeat crash in
reset_timeout and clear_timeout
4. https://tracker.ceph.com/issues/58893 - test_map_discontinuity:
AssertionError: wait_for_clean: failed before timeout expired
5. https://tracker.ceph.com/issues/61774 - centos 9 testing reveals
rocksdb "Leak_StillReachable" memory leak in mons
7. https://tracker.ceph.com/issues/62776 - rados: cluster [WRN] overall
HEALTH_WARN - do not have an application enabled
8. https://tracker.ceph.com/issues/59196 - ceph_test_lazy_omap_stats
segfault while waiting for active+clean

Orchestrator:
1. https://tracker.ceph.com/issues/64208 - test_cephadm.sh: Container
version mismatch causes job to fail.

CephFS:
1. https://tracker.ceph.com/issues/64946 - qa: unable to locate package
libcephfs1

Teuthology:
1. https://tracker.ceph.com/issues/64727 - suites/dbench.sh: Socket
exception: No route to host (113)

On Tue, Apr 16, 2024 at 9:22 AM Yuri Weinstein  wrote:

> And approval is needed for:
>
> fs - Venky approved?
> powercycle - seems fs related, Venky, Brad PTL
>
> On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein 
> wrote:
> >
> > Still waiting for approvals:
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
> >
> > We plan not to upgrade the LRC to 18.2.3 as we are very close to the
> > first squid RC and will be using it for this purpose.
> > Please speak up if this may present any issues.
> >
> > Thx
> >
> > On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein 
> wrote:
> > >
> > > Details of this release are summarized here:
> > >
> > > https://tracker.ceph.com/issues/65393#note-1
> > > Release Notes - TBD
> > > LRC upgrade - TBD
> > >
> > > Seeking approvals/reviews for:
> > >
> > > smoke - infra issues, still trying, Laura PTL
> > >
> > > rados - Radek, Laura approved? Travis?  Nizamudeen?
> > >
> > > rgw - Casey approved?
> > > fs - Venky approved?
> > > orch - Adam King approved?
> > >
> > > krbd - Ilya approved
> > > powercycle - seems fs related, Venky, Brad PTL
> > >
> > > ceph-volume - will require
> > >
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > > Guillaume is fixing it.
> > >
> > > TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Venky Shankar]

On Tue, Apr 16, 2024 at 7:52 PM Yuri Weinstein  wrote:
>
> And approval is needed for:
>
> fs - Venky approved?

Could not get to this today. Will be done tomorrow.

> powercycle - seems fs related, Venky, Brad PTL
>
> On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein  wrote:
> >
> > Still waiting for approvals:
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
> >
> > We plan not to upgrade the LRC to 18.2.3 as we are very close to the
> > first squid RC and will be using it for this purpose.
> > Please speak up if this may present any issues.
> >
> > Thx
> >
> > On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein  wrote:
> > >
> > > Details of this release are summarized here:
> > >
> > > https://tracker.ceph.com/issues/65393#note-1
> > > Release Notes - TBD
> > > LRC upgrade - TBD
> > >
> > > Seeking approvals/reviews for:
> > >
> > > smoke - infra issues, still trying, Laura PTL
> > >
> > > rados - Radek, Laura approved? Travis?  Nizamudeen?
> > >
> > > rgw - Casey approved?
> > > fs - Venky approved?
> > > orch - Adam King approved?
> > >
> > > krbd - Ilya approved
> > > powercycle - seems fs related, Venky, Brad PTL
> > >
> > > ceph-volume - will require
> > > https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > > Guillaume is fixing it.
> > >
> > > TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io



-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Adam King]

>
> Orchestrator:
> 1. https://tracker.ceph.com/issues/64208 - test_cephadm.sh: Container
> version mismatch causes job to fail.
>

Not a blocker issue. Just a problem with the test itself that will be fixed
by https://github.com/ceph/ceph/pull/56714


On Tue, Apr 16, 2024 at 1:39 PM Laura Flores  wrote:

> On behalf of @Radoslaw Zarzynski , rados approved.
>
> Below is the summary of the rados suite failures, divided by component. @Adam
> King  @Venky Shankar  PTAL at the
> orch and cephfs failures to see if they are blockers.
>
> Failures, unrelated:
>
> RADOS:
> 1. https://tracker.ceph.com/issues/65183 - Overriding an EC pool
> needs the "--yes-i-really-mean-it" flag in addition to "force"
> 3. https://tracker.ceph.com/issues/62992 - Heartbeat crash in
> reset_timeout and clear_timeout
> 4. https://tracker.ceph.com/issues/58893 - test_map_discontinuity:
> AssertionError: wait_for_clean: failed before timeout expired
> 5. https://tracker.ceph.com/issues/61774 - centos 9 testing reveals
> rocksdb "Leak_StillReachable" memory leak in mons
> 7. https://tracker.ceph.com/issues/62776 - rados: cluster [WRN]
> overall HEALTH_WARN - do not have an application enabled
> 8. https://tracker.ceph.com/issues/59196 - ceph_test_lazy_omap_stats
> segfault while waiting for active+clean
>
> Orchestrator:
> 1. https://tracker.ceph.com/issues/64208 - test_cephadm.sh: Container
> version mismatch causes job to fail.
>
> CephFS:
> 1. https://tracker.ceph.com/issues/64946 - qa: unable to locate
> package libcephfs1
>
> Teuthology:
> 1. https://tracker.ceph.com/issues/64727 - suites/dbench.sh: Socket
> exception: No route to host (113)
>
> On Tue, Apr 16, 2024 at 9:22 AM Yuri Weinstein 
> wrote:
>
>> And approval is needed for:
>>
>> fs - Venky approved?
>> powercycle - seems fs related, Venky, Brad PTL
>>
>> On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein 
>> wrote:
>> >
>> > Still waiting for approvals:
>> >
>> > rados - Radek, Laura approved? Travis?  Nizamudeen?
>> >
>> > ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
>> >
>> > We plan not to upgrade the LRC to 18.2.3 as we are very close to the
>> > first squid RC and will be using it for this purpose.
>> > Please speak up if this may present any issues.
>> >
>> > Thx
>> >
>> > On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein 
>> wrote:
>> > >
>> > > Details of this release are summarized here:
>> > >
>> > > https://tracker.ceph.com/issues/65393#note-1
>> > > Release Notes - TBD
>> > > LRC upgrade - TBD
>> > >
>> > > Seeking approvals/reviews for:
>> > >
>> > > smoke - infra issues, still trying, Laura PTL
>> > >
>> > > rados - Radek, Laura approved? Travis?  Nizamudeen?
>> > >
>> > > rgw - Casey approved?
>> > > fs - Venky approved?
>> > > orch - Adam King approved?
>> > >
>> > > krbd - Ilya approved
>> > > powercycle - seems fs related, Venky, Brad PTL
>> > >
>> > > ceph-volume - will require
>> > >
>> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
>> > > Guillaume is fixing it.
>> > >
>> > > TIA
>> ___
>> Dev mailing list -- d...@ceph.io
>> To unsubscribe send an email to dev-le...@ceph.io
>>
>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph User Dev Meeting next week: Ceph Users Feedback Survey Results

[Neha Ojha]

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Leadership Team Meeting, 2024-04-08

[Satoru Takeuchi]

2024年4月9日(火) 8:06 Laura Flores :

> I've added them!
>

I have a question about the role of github milestones.

The two PRs in the v18.2.3 milestone are essential for debian package users
as I described before. Are these PRs release blocker for v18.2.3 release?
Or is the merging work done as best-effort?

https://github.com/ceph/ceph/pull/56541

I want to know this because of two reasons.

First, I'd like to upgrade my v17.2 cluster to v18.2. It's blocked for now
due to the above-mentioned two PRs. I don't want to suspend upgrading by
the release of v18.2.4.

Second, I'd like to plan the timing of my clusters as precise as possible.
To accomplish this, which PRs will be in the next stable release is the
essential information. Although I've read official documents, PRs, issues,
and slack channels to know the information for several years, I couldn't
find the strict role of milestones yet.

Of course, I read the following document.

https://github.com/ceph/ceph/blob/dbd161df82a5b6079484649873438185aa7e1b76/SubmittingPatches-backports.rst#reviewing-testing-and-merging-of-backport-prs
>
Once your backport PR is open and the Milestone is set properly, the Stable
Releases and Backports team will take care of getting the PR reviewed and
tested. Once the PR is reviewed and tested, it will be merged.

However, these PRs are not reviewed yet and the milestone is not mentioned
in neither the release tracker issue nor the thread for the v18.2.3 release.

https://tracker.ceph.com/issues/65393

Best,
Satoru



> cc @Yuri Weinstein 
>
> On Mon, Apr 8, 2024 at 5:39 PM Satoru Takeuchi 
> wrote:
>
>> 2024年4月9日(火) 0:43 Laura Flores :
>>
>>> Hi all,
>>>
>>> Today we discussed:
>>>
>>> 2024/04/08
>>>
>>>- [Zac] CQ#4 is going out this week -
>>>https://pad.ceph.com/p/ceph_quarterly_2024_04
>>>
>>>
>>>- Last chance to review!
>>>
>>>
>>>- [Zac] IcePic Initiative - context-sensitive help - do we regard
>>>the docs as a part of the online help?
>>>
>>>
>>>- https://pad.ceph.com/p/2024_04_08_cephadm_context_sensitive_help
>>>
>>>
>>>- docs.ceph.com should be main source of truth; can link to this or
>>>reference it generally as "see docs.ceph.com"
>>>
>>>
>>>- Squid RC status
>>>
>>>
>>>- Blockers tracked in: https://pad.ceph.com/p/squid-upgrade-failures
>>>
>>>
>>>- rgw: topic changes merged to main, but introduced some test
>>>failures. account changes blocked on topics
>>>
>>>
>>>- Non-blocker for RC0
>>>
>>>
>>>- centos 9 containerization (status unknown?)
>>>
>>>
>>>- Non-blocker for RC0
>>>
>>>
>>>- Follow up with Dan / Guillaume
>>>
>>>
>>>- RADOS has one outstanding blocker awaiting QA
>>>
>>>
>>>- Failing to register new account at Ceph tracker - error 404.
>>>
>>>
>>>- Likely related to Redmine upgrade over the weekend
>>>
>>>
>>>- Pacific eol:
>>>
>>>
>>>- Action item: in https://docs.ceph.com/en/latest/releases/, move to
>>>"archived"
>>>
>>>
>>>- 18.2.3
>>>
>>>
>>>- one or two PRs from cephfs left
>>>
>>>
>>>- Milestone: https://github.com/ceph/ceph/milestone/19
>>>
>>>
>> Could you add the following PRs to 18.2.3 milestone? Without these PRs,
>> debian package users
>> can't use metrics from ceph-exporter at all.
>>
>> reef: debian: add ceph-exporter package #56541
>> https://github.com/ceph/ceph/pull/56541
>>
>> reef: debian: add missing bcrypt to ceph-mgr .requires to fix resulting
>> package dependencies #54662
>> https://github.com/ceph/ceph/pull/54662
>>
>> Thanks,
>> Satoru
>>
>>
>>>
>>> Thanks,
>>> Laura
>>> --
>>>
>>> Laura Flores
>>>
>>> She/Her/Hers
>>>
>>> Software Engineer, Ceph Storage 
>>>
>>> Chicago, IL
>>>
>>> lflo...@ibm.com | lflo...@redhat.com 
>>> M: +17087388804
>>>
>>>
>>> ___
>>> Dev mailing list -- d...@ceph.io
>>> To unsubscribe send an email to dev-le...@ceph.io
>>>
>>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How to make config changes stick for MDS?

[Erich Weiler]

Hi All,

I'm having a crazy time getting config items to stick on my MDS daemons. 
 I'm running Reef 18.2.1 on RHEL 9 and the daemons are running in 
podman, I used cephadm to deploy the daemons.


I can adjust the config items in runtime, like so:

ceph tell mds.slugfs.pr-md-01.xdtppo config set mds_bal_interval -1

But for the life of me I cannot get that to stick when I restart the MDS 
daemon.


I've tried adding this to /etc/ceph/ceph.conf in the host server:

[mds]
mds_bal_interval = -1

But that doesn't get picked up on daemon restart.  I also added the same 
config segment to /etc/ceph/ceph.conf *inside* the container, no dice, 
still doesn't stick.  I even tried adding it to 
/var/lib/ceph//config/ceph.conf and it *still* doesn't stick 
across daemon restarts.


Does anyone know how I can get MDS config items to stick across daemon 
reboots when the daemon is running in podman under RHEL?


Thanks much!

-erich
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How to make config changes stick for MDS?

[Stefan Kooman]

On 17-04-2024 05:23, Erich Weiler wrote:

Hi All,

I'm having a crazy time getting config items to stick on my MDS daemons. 
  I'm running Reef 18.2.1 on RHEL 9 and the daemons are running in 
podman, I used cephadm to deploy the daemons.


I can adjust the config items in runtime, like so:

ceph tell mds.slugfs.pr-md-01.xdtppo config set mds_bal_interval -1

But for the life of me I cannot get that to stick when I restart the MDS 
daemon.


I've tried adding this to /etc/ceph/ceph.conf in the host server:

[mds]
     mds_bal_interval = -1

But that doesn't get picked up on daemon restart.  I also added the same 
config segment to /etc/ceph/ceph.conf *inside* the container, no dice, 
still doesn't stick.  I even tried adding it to 
/var/lib/ceph//config/ceph.conf and it *still* doesn't stick 
across daemon restarts.


Does anyone know how I can get MDS config items to stick across daemon 
reboots when the daemon is running in podman under RHEL?


ceph config set mds mds_bal_interval -1. See [1]

Gr. Stefan

[1]: 
https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#monitor-configuration-database

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERN] Re: Ceph 16.2.x mon compactions, disk writes

[Dietmar Rieder]

Hi Zakhar,

thanks so much for the information.

Best
  D.Rieder

On 4/16/24 17:40, Zakhar Kirpichenko wrote:

Hi,

 >Did you noticed any downsides with your compression settings so far?

None, at least on our systems. Except the part that I haven't found a 
way to make the settings persist.


 >Do you have all mons now on compression?

I have 3 out of 5 monitors with compression and 2 without it. The 2 
monitors with uncompressed RocksDB have much larger disks which do not 
suffer from writes as much as the other 3. I keep them uncompressed 
"just in case", i.e. for the unlikely event if the 3 monitors with 
compressed RocksDB fail or have any issues specifically because of the 
compression. I have to say that this hasn't happened yet, and this 
precaution may be unnecessary.


 >Did release updates go through without issues?

In our case, container updates overwrite the monitors' configurations 
and reset RocksDB options, thus each updated monitor runs with no 
RocksDB compression until it is added back manually. Other than that, I 
have not encountered any issues related to compression during the updates.


 >Do you know if this works also with reef (we see massive writes as 
well there)?


Unfortunately, I can't comment on Reef as we're still using Pacific.

/Z

On Tue, 16 Apr 2024 at 18:08, Dietmar Rieder > wrote:


Hi Zakhar, hello List,

I just wanted to follow up on this and ask a few quesitions:

Did you noticed any downsides with your compression settings so far?
Do you have all mons now on compression?
Did release updates go through without issues?
Do you know if this works also with reef (we see massive writes as well
there)?

Can you briefly tabulate the commands you used to persistently set the
compression options?

Thanks so much,

    Dietmar


On 10/18/23 06:14, Zakhar Kirpichenko wrote:
 > Many thanks for this, Eugen! I very much appreciate yours and
Mykola's
 > efforts and insight!
 >
 > Another thing I noticed was a reduction of RocksDB store after the
 > reduction of the total PG number by 30%, from 590-600 MB:
 >
 > 65M     3675511.sst
 > 65M     3675512.sst
 > 65M     3675513.sst
 > 65M     3675514.sst
 > 65M     3675515.sst
 > 65M     3675516.sst
 > 65M     3675517.sst
 > 65M     3675518.sst
 > 62M     3675519.sst
 >
 > to about half of the original size:
 >
 > -rw-r--r-- 1 167 167  7218886 Oct 13 16:16 3056869.log
 > -rw-r--r-- 1 167 167 67250650 Oct 13 16:15 3056871.sst
 > -rw-r--r-- 1 167 167 67367527 Oct 13 16:15 3056872.sst
 > -rw-r--r-- 1 167 167 63268486 Oct 13 16:15 3056873.sst
 >
 > Then when I restarted the monitors one by one before adding
compression,
 > RocksDB store reduced even further. I am not sure why and what
exactly got
 > automatically removed from the store:
 >
 > -rw-r--r-- 1 167 167   841960 Oct 18 03:31 018779.log
 > -rw-r--r-- 1 167 167 67290532 Oct 18 03:31 018781.sst
 > -rw-r--r-- 1 167 167 53287626 Oct 18 03:31 018782.sst
 >
 > Then I have enabled LZ4 and LZ4HC compression in our small production
 > cluster (6 nodes, 96 OSDs) on 3 out of 5
 > monitors:
compression=kLZ4Compression,bottommost_compression=kLZ4HCCompression.
 > I specifically went for LZ4 and LZ4HC because of the balance between
 > compression/decompression speed and impact on CPU usage. The
compression
 > doesn't seem to affect the cluster in any negative way, the 3
monitors with
 > compression are operating normally. The effect of the compression on
 > RocksDB store size and disk writes is quite noticeable:
 >
 > Compression disabled, 155 MB store.db, ~125 MB RocksDB sst, and
~530 MB
 > writes over 5 minutes:
 >
 > -rw-r--r-- 1 167 167  4227337 Oct 18 03:58 3080868.log
 > -rw-r--r-- 1 167 167 67253592 Oct 18 03:57 3080870.sst
 > -rw-r--r-- 1 167 167 57783180 Oct 18 03:57 3080871.sst
 >
 > # du -hs
 >
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/;
 > iotop -ao -bn 2 -d 300 2>&1 | grep ceph-mon
 > 155M
 > 
  /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph04/store.db/

 > 2471602 be/4 167           6.05 M    473.24 M  0.00 %  0.16 %
ceph-mon -n
 > mon.ceph04 -f --setuser ceph --setgroup ceph
--default-log-to-file=false
 > --default-log-to-stderr=true --default-log-stderr-prefix=debug
 >   --default-mon-cluster-log-to-file=false
 > --default-mon-cluster-log-to-stderr=true [rocksdb:low0]
 > 2471633 be/4 167         188.00 K     40.91 M  0.00 %  0.02 %
ceph-mon -n
 > mon.ceph04 -f --setuser ceph --setgroup ceph
--default-log-to-file=false
 > --default-log-to-stderr=true --default-log-stderr-prefix=debug
 >   --default-mon-cluster-log-to-file=false
 > --default-mon-cluster-log-to-st