[ceph-users] Re: Slow recovery on Quincy

[Nigel Williams]

We're on 17.2.5 and had the default value (5.2), but changing it didn't
seem to impact recovery speed:

root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
5.20
root@rdx-00:/# ceph config show osd.0 osd_op_queue
mclock_scheduler
root@rdx-00:/# ceph config set osd osd_mclock_cost_per_byte_usec_hdd 0.4
root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
0.40
root@rdx-00:/#

We have about 500TB of misplaced objects moving about so we are keen to
speed it up; was there another parameter that needed to be changed too?
thanks.


On Mon, 22 May 2023 at 23:24, David Orman  wrote:

> Someone who's got data regarding this should file a bug report, it sounds
> like a quick fix for defaults if this holds true.
>
> On Sat, May 20, 2023, at 00:59, Hector Martin wrote:
> > On 17/05/2023 03.07, 胡 玮文 wrote:
> >> Hi Sake,
> >>
> >> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd”
> to 0.1 (default is 2.6) and get about 15 times backfill speed, without
> significant affect client IO. This parameter seems calculated wrongly, from
> the description 5e-3 should be a reasonable value for HDD (corresponding to
> 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to
> increase the recovery speed. So I suspect the original author just convert
> the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code.
> >>
> >> But all this may be not important in the next version. I see the
> relevant code is rewritten, and this parameter is now removed.
> >>
> >> high_recovery_ops profile works very poorly for us. It increase the
> average latency of client IO from 50ms to about 1s.
> >>
> >> Weiwen Hu
> >>
> >
> > Thank you for this, that parameter indeed seems completely wrong
> > (assuming it means what it says on the tin). After changing that my
> > Quincy cluster is no recovering at a much more reasonable speed.
> >
> > - Hector
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Encryption per user Howto

[Christian Wuerdig]

Hm, this thread is confusing
in the context of S3 client-side encryption means - the user is responsible
to encrypt the data with their own keys before submitting it. As far as I'm
aware, client-side encryption doesn't require any specific server support -
it's a function of the client SDK used which provides the convenience of
encrypting your data before upload and decryptiing it after download -
https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingClientSideEncryption.html
But you can always encrypt your data and then upload it via RGW, there is
nothing anywhere that prevents that since uploaded objects are just a
sequence of bytes, meta data won't be encrypted then

You can also do server-side encryption by bringing your own keys -
https://docs.ceph.com/en/quincy/radosgw/encryption/#customer-provided-keys

I suspect you're asking for server-side encryption with keys managed by
ceph on a per-user basis?


On Tue, 23 May 2023 at 03:28, huxia...@horebdata.cn 
wrote:

> Hi, Stefan,
>
> Thanks a lot for the message. It seems that client-side encryption (or per
> use) is still on the way and not ready yet for today.
>
> Are there  practical methods to implement encryption for CephFS with
> today' technique? e.g using LUKS or other tools?
>
> Kind regards,
>
>
> Samuel
>
>
>
>
> huxia...@horebdata.cn
>
> From: Stefan Kooman
> Date: 2023-05-22 17:19
> To: Alexander E. Patrakov; huxia...@horebdata.cn
> CC: ceph-users
> Subject: Re: [ceph-users] Re: Encryption per user Howto
> On 5/21/23 15:44, Alexander E. Patrakov wrote:
> > Hello Samuel,
> >
> > On Sun, May 21, 2023 at 3:48 PM huxia...@horebdata.cn
> >  wrote:
> >>
> >> Dear Ceph folks,
> >>
> >> Recently one of our clients approached us with a request on encrpytion
> per user, i.e. using individual encrytion key for each user and encryption
> files and object store.
> >>
> >> Does anyone know (or have experience) how to do with CephFS and Ceph
> RGW?
> >
> > For CephFS, this is unachievable.
>
> For a couple of years already, work is being done to have fscrypt
> support for CephFS [1]. When that work ends up in mainline kernel (and
> distro kernels at some point) this will be possible.
>
> Gr. Stefan
>
> [1]: https://lwn.net/Articles/829448/
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Deleting a CephFS volume

[Patrick Donnelly]

Hi Conrad,

On Wed, May 17, 2023 at 2:41 PM Conrad Hoffmann  wrote:
>
> On 5/17/23 18:07, Stefan Kooman wrote:
> > On 5/17/23 17:29, Conrad Hoffmann wrote:
> >> Hi all,
> >>
> >> I'm having difficulties removing a CephFS volume that I set up for
> >> testing. I've been through this with RBDs, so I do know about
> >> `mon_allow_pool_delete`. However, it doesn't help in this case.
> >>
> >> It is a cluster with 3 monitors. You can find a console log of me
> >> verifying that `mon_allow_pool_delete` is indeed true on all monitors
> >> but still fail to remove the volume here:
> >
> > That's not just a volume, that's the whole filesystem. If that's what
> > you want to do ... I see the MDS daemon is still up. IIRC there should
> > be no MDS running if you want to delete the fs. Can you stop the MDS
> > daemon and try again.
>
> That sort of got me in the right direction, but I am still confused. I
> don't think I understand the difference between a volume and a
> filesystem. I think I followed [1] when I set this up. It says to use
> `ceph fs volume create`. I went ahead and ran it again, and it certainly
> creates something that shows up in both `ceph fs ls` and `ceph fs volume
> ls`. Also, [2] says "FS volumes, an abstraction for CephFS file
> systems", so I guess they are the same thing?

Yes.

> At any rate, shutting down the MDS did _not_ help with `ceph fs volume
> rm` (it failed with the same error message), but it _did_ help with
> `ceph fs rm`, which then worked. Hard to make sense of, but I am pretty
> sure the error message I was seeing is pretty non-sensical in that
> context. Under what circumstance will `ceph fs volume rm` even work if
> it fails to delete a volume I just created?

`fs rm` just removes the file system from the monitor maps. You still
have the data pools lying around which is what the `volume rm` command
is complaining about.

Try:

ceph config set global mon_allow_pool_delete true
ceph fs volume rm ...

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS crashes to damaged metadata

[Patrick Donnelly]

On Mon, May 15, 2023 at 8:55 AM Stefan Kooman  wrote:
>
> On 12/15/22 15:31, Stolte, Felix wrote:
> > Hi Patrick,
> >
> > we used your script to repair the damaged objects on the weekend and it 
> > went smoothly. Thanks for your support.
> >
> > We adjusted your script to scan for damaged files on a daily basis, runtime 
> > is about 6h. Until thursday last week, we had exactly the same 17 Files. On 
> > thursday at 13:05 a snapshot was created and our active mds crashed once at 
> > this time (snapshot was created):
>
> Are you willing to share this script? I would like to use it to scan our
> CephFS before upgrading to 16.2.13. Do you run this script when the
> filesystem is online / active?

The original script is here:
https://github.com/ceph/ceph/blob/main/src/tools/cephfs/first-damage.py

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS crashes to damaged metadata

[Patrick Donnelly]

Hi Felix,

On Sat, May 13, 2023 at 9:18 AM Stolte, Felix  wrote:
>
> Hi Patrick,
>
> we have been running one daily snapshot since december and our cephfs crashed 
> 3 times because of this https://tracker.ceph.com/issues/38452
>
> We currentliy have 19 files with corrupt metadata found by your 
> first-damage.py script. We isolated the these files from access by users and 
> are waiting for a fix before we remove them with your script (or maybe a new 
> way?)

No other fix is anticipated at this time. Probably one will be
developed after the cause is understood.

> Today we upgraded our cluster from 16.2.11 and 16.2.13. After Upgrading the 
> mds  servers, cluster health went to ERROR MDS_DAMAGE. 'ceph tells mds 0 
> damage ls‘ is showing me the same files as your script (initially only a 
> part, after a cephfs scrub all of them).

This is expected. Once the dentries are marked damaged, the MDS won't
allow operations on those files (like those triggering tracker
#38452).

> I noticed "mds: catch damage to CDentry’s first member before persisting 
> (issue#58482, pr#50781, Patrick Donnelly)“ in the change logs for 16.2.13  
> and like to ask you the following questions:
>
> a) can we repair the damaged files online now instead of bringing down the 
> whole fs and using the python script?

Not yet.

> b) should we set one of the new mds options in our specific case to avoid our 
> fileserver crashing because of the wrong snap ids?

Have your MDS crashed or just marked the dentries damaged? If you can
reproduce a crash with detailed logs (debug_mds=20), that would be
incredibly helpful.

> c) will your patch prevent wrong snap ids in the future?

It will prevent persisting the damage.


-- 
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Encryption per user Howto

[huxia...@horebdata.cn]

Hi, Stefan,

Thanks a lot for the message. It seems that client-side encryption (or per use) 
is still on the way and not ready yet for today. 

Are there  practical methods to implement encryption for CephFS with today' 
technique? e.g using LUKS or other tools?

Kind regards,


Samuel




huxia...@horebdata.cn
 
From: Stefan Kooman
Date: 2023-05-22 17:19
To: Alexander E. Patrakov; huxia...@horebdata.cn
CC: ceph-users
Subject: Re: [ceph-users] Re: Encryption per user Howto
On 5/21/23 15:44, Alexander E. Patrakov wrote:
> Hello Samuel,
> 
> On Sun, May 21, 2023 at 3:48 PM huxia...@horebdata.cn
>  wrote:
>>
>> Dear Ceph folks,
>>
>> Recently one of our clients approached us with a request on encrpytion per 
>> user, i.e. using individual encrytion key for each user and encryption  
>> files and object store.
>>
>> Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
> 
> For CephFS, this is unachievable.
 
For a couple of years already, work is being done to have fscrypt 
support for CephFS [1]. When that work ends up in mainline kernel (and 
distro kernels at some point) this will be possible.
 
Gr. Stefan
 
[1]: https://lwn.net/Articles/829448/
 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Encryption per user Howto

[Stefan Kooman]

On 5/21/23 15:44, Alexander E. Patrakov wrote:

Hello Samuel,

On Sun, May 21, 2023 at 3:48 PM huxia...@horebdata.cn
 wrote:


Dear Ceph folks,

Recently one of our clients approached us with a request on encrpytion per 
user, i.e. using individual encrytion key for each user and encryption  files 
and object store.

Does anyone know (or have experience) how to do with CephFS and Ceph RGW?


For CephFS, this is unachievable.


For a couple of years already, work is being done to have fscrypt 
support for CephFS [1]. When that work ends up in mainline kernel (and 
distro kernels at some point) this will be possible.


Gr. Stefan

[1]: https://lwn.net/Articles/829448/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Encryption per user Howto

[Casey Bodley]

rgw supports the 3 flavors of S3 Server-Side Encryption, along with
the PutBucketEncryption api for per-bucket default encryption. you can
find the docs in https://docs.ceph.com/en/quincy/radosgw/encryption/

On Mon, May 22, 2023 at 10:49 AM huxia...@horebdata.cn
 wrote:
>
> Dear Alexander,
>
> Thanks a lot for helpful comments and insights. Regarding CephFS and RGW, Per 
> user seems to be daunting and complex.
>
> What if encryption on the server side without per user requirment? would it 
> be relatively easy to achieve, and how?
>
> best regards,
>
> Samuel
>
>
>
>
>
> huxia...@horebdata.cn
>
> From: Alexander E. Patrakov
> Date: 2023-05-21 15:44
> To: huxia...@horebdata.cn
> CC: ceph-users
> Subject: Re: [ceph-users] Encryption per user Howto
> Hello Samuel,
>
> On Sun, May 21, 2023 at 3:48 PM huxia...@horebdata.cn
>  wrote:
> >
> > Dear Ceph folks,
> >
> > Recently one of our clients approached us with a request on encrpytion per 
> > user, i.e. using individual encrytion key for each user and encryption  
> > files and object store.
> >
> > Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
>
> For CephFS, this is unachievable.
>
> For RGW, please use Vault for storing encryption keys. Don't forget
> about the proper high-availability setup. Use an AppRole to manage
> tokens. Use Vault Agent as a proxy that adds the token to requests
> issued by RGWs. Then create a bucket for each user and set the
> encryption policy for this bucket using the PutBucketEncryption API
> that is available through AWS CLI. Either SSE-S3 or SSE-KMS will work
> for you. SSE-S3 is easier to manage. Each object will then be
> encrypted using a different key derived from its name and a per-bucket
> master key which never leaves Vault.
>
> Note that users will be able to create additional buckets by
> themselves, and they won't be encrypted, so tell them either not to do
> that or to encrypt the new buckets similarly.
>
> --
> Alexander E. Patrakov
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Encryption per user Howto

[huxia...@horebdata.cn]

Dear Alexander,

Thanks a lot for helpful comments and insights. Regarding CephFS and RGW, Per 
user seems to be daunting and complex. 

What if encryption on the server side without per user requirment? would it be 
relatively easy to achieve, and how?

best regards,

Samuel





huxia...@horebdata.cn
 
From: Alexander E. Patrakov
Date: 2023-05-21 15:44
To: huxia...@horebdata.cn
CC: ceph-users
Subject: Re: [ceph-users] Encryption per user Howto
Hello Samuel,
 
On Sun, May 21, 2023 at 3:48 PM huxia...@horebdata.cn
 wrote:
>
> Dear Ceph folks,
>
> Recently one of our clients approached us with a request on encrpytion per 
> user, i.e. using individual encrytion key for each user and encryption  files 
> and object store.
>
> Does anyone know (or have experience) how to do with CephFS and Ceph RGW?
 
For CephFS, this is unachievable.
 
For RGW, please use Vault for storing encryption keys. Don't forget
about the proper high-availability setup. Use an AppRole to manage
tokens. Use Vault Agent as a proxy that adds the token to requests
issued by RGWs. Then create a bucket for each user and set the
encryption policy for this bucket using the PutBucketEncryption API
that is available through AWS CLI. Either SSE-S3 or SSE-KMS will work
for you. SSE-S3 is easier to manage. Each object will then be
encrypted using a different key derived from its name and a per-bucket
master key which never leaves Vault.
 
Note that users will be able to create additional buckets by
themselves, and they won't be encrypted, so tell them either not to do
that or to encrypt the new buckets similarly.
 
-- 
Alexander E. Patrakov
 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery on Quincy

[David Orman]

Someone who's got data regarding this should file a bug report, it sounds like 
a quick fix for defaults if this holds true.

On Sat, May 20, 2023, at 00:59, Hector Martin wrote:
> On 17/05/2023 03.07, 胡 玮文 wrote:
>> Hi Sake,
>> 
>> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” to 
>> 0.1 (default is 2.6) and get about 15 times backfill speed, without 
>> significant affect client IO. This parameter seems calculated wrongly, from 
>> the description 5e-3 should be a reasonable value for HDD (corresponding to 
>> 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to 
>> increase the recovery speed. So I suspect the original author just convert 
>> the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code.
>> 
>> But all this may be not important in the next version. I see the relevant 
>> code is rewritten, and this parameter is now removed.
>> 
>> high_recovery_ops profile works very poorly for us. It increase the average 
>> latency of client IO from 50ms to about 1s.
>> 
>> Weiwen Hu
>> 
>
> Thank you for this, that parameter indeed seems completely wrong
> (assuming it means what it says on the tin). After changing that my
> Quincy cluster is no recovering at a much more reasonable speed.
>
> - Hector
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Advices : RGW and Citrix ADC

[BEAUDICHON Hubert (Acoss)]

Hi everyone,

Is there any advices, best practices for a Citrix ADC VIP configuration in 
front of a dual RadosGW (same zone) ?
I find many documentations for HAProxy but nothing for this hardware.

For information, the CEPH cluster version is Pacific 16.2.10.

Thanks for your help.

Regards,

Hubert


[cid:image001.png@01D98CA0.D075D720]
Hubert Beaudichon

Secteur Infrastructures / Gamme CET-Infrastructures
SDAT UT AUTOMATION ET CLOUD

hubert.beaudic...@acoss.fr
01 82 30 77 03 * 06 00 00 00 00
36 rue de Valmy * 93108 Montreuil CEDEX

www.urssaf.fr
[cid:image002.png@01D98CA1.F1ADE8F0][cid:image003.png@01D98CA1.F1ADE8F0][cid:image002.png@01D98CA1.F1ADE8F0][cid:image005.png@01D98CA0.D075D720]


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Colocated RGW using same port, but different IPs

[Ondřej Kukla]

Hello,

As a part of performance/HW utilization testing I would like to compare how a 
VM with single RGW instance compares to a VM with 2 or more colocated RGWs.

I’m deploying Quincy using Ansible but I’m struggling with setting the config 
to use the same port across all RGW instances, but to have a different IP as a 
beast endpoint.

I would like to ask if anyone has any suggestion on this topic. Is this even 
supported scenario or it is better to have the instances using different ports?

Any help is much appreciated.

Kind regards,

Ondrej
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io