[ceph-users] Re: How to use hardware

[Simon Kepp]

I know that your question is regarding the service servers, but may I ask,
why you are planning to place so many OSDs ( 300) on so few OSD hosts( 6)
(= 50 OSDs per node)?
This is possible to do, but sounds like the nodes were designed for
scale-up rather than a scale-out architecture like ceph. Going with such
"fat nodes" is doable, but will significantly limit performance,
reliability and availability, compared to distributing the same OSDs
on more thinner nodes.

Best regards,
Simon Kepp

Founder/CEO
Kepp Technologies

On Fri, Nov 17, 2023 at 10:59 AM Albert Shih  wrote:

> Hi everyone,
>
> In the purpose to deploy a medium size of ceph cluster (300 OSD) we have 6
> bare-metal server for the OSD, and 5 bare-metal server for the service
> (MDS, Mon, etc.)
>
> Those 5 bare-metal server have each 48 cores and 256 Gb.
>
> What would be the smartest way to use those 5 server, I see two way :
>
>   first :
>
> Server 1 : MDS,MON, grafana, prometheus, webui
> Server 2:  MON
> Server 3:  MON
> Server 4 : MDS
> Server 5 : MDS
>
>   so 3 MDS, 3 MON. and we can loose 2 servers.
>
>   Second
>
> KVM on each server
>   Server 1 : 3 VM : One for grafana & CIe, and 1 MDS, 2 MON
>   other server : 1 MDS, 1 MON
>
>   in total :  5 MDS, 5 MON and we can loose 4 servers.
>
> So on paper it's seem the second are smarter, but it's also more complex,
> so my question are «is it worth the complexity to have 5 MDS/MON for 300
> OSD».
>
> Important : The main goal of this ceph cluster are not to get the maximum
> I/O speed, I would not say the speed is not a factor, but it's not the main
> point.
>
> Regards.
>
>
> --
> Albert SHIH 嶺 
> Observatoire de Paris
> France
> Heure locale/Local time:
> ven. 17 nov. 2023 10:49:27 CET
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: blustore osd nearfull but no pgs on it

[Wesley Dillingham]

Please send along a pastebin of "ceph status" and "ceph osd df tree" and
"ceph df detail" also "ceph tell osd.158 status"

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Fri, Nov 17, 2023 at 6:20 PM Debian  wrote:

> thx for your reply, it shows nothing,... there are no pgs on the osd,...
>
> best regards
>
> On 17.11.23 23:09, Eugen Block wrote:
> > After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should
> > show you which PGs are created there and then you’ll know which pool
> > they belong to, then check again the crush rule for that pool. You can
> > paste the outputs here.
> >
> > Zitat von Debian :
> >
> >> Hi,
> >>
> >> after a massive rebalance(tunables) my small SSD-OSDs are getting
> >> full, i changed my crush rules so there are actual no pgs/pools on
> >> it, but the disks stay full:
> >>
> >> ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6)
> >> nautilus (stable)
> >>
> >> ID CLASS WEIGHT REWEIGHT SIZERAW USE DATAOMAP
> >> META AVAIL%USE  VAR  PGS STATUS TYPE NAME
> >> 158   ssd0.21999  1.0 224 GiB 194 GiB 193 GiB  22 MiB 1002
> >> MiB   30 GiB 86.68 1.49   0 up osd.158
> >>
> >> inferring bluefs devices from bluestore path
> >> 1 : device size 0x37e440 : own 0x[1ad3f0~23c60] =
> >> 0x23c60 : using 0x3963(918 MiB) : bluestore has
> >> 0x46e2d(18 GiB) available
> >>
> >> when i recreate the osd the osd gets full again
> >>
> >> any suggestion?
> >>
> >> thx & best regards
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: blustore osd nearfull but no pgs on it

[Debian]

thx for your reply, it shows nothing,... there are no pgs on the osd,...

best regards

On 17.11.23 23:09, Eugen Block wrote:
After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should 
show you which PGs are created there and then you’ll know which pool 
they belong to, then check again the crush rule for that pool. You can 
paste the outputs here.


Zitat von Debian :


Hi,

after a massive rebalance(tunables) my small SSD-OSDs are getting 
full, i changed my crush rules so there are actual no pgs/pools on 
it, but the disks stay full:


ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) 
nautilus (stable)


ID CLASS WEIGHT REWEIGHT SIZE    RAW USE DATA    OMAP 
META AVAIL    %USE  VAR  PGS STATUS TYPE NAME
158   ssd    0.21999  1.0 224 GiB 194 GiB 193 GiB  22 MiB 1002 
MiB   30 GiB 86.68 1.49   0 up osd.158


inferring bluefs devices from bluestore path
1 : device size 0x37e440 : own 0x[1ad3f0~23c60] = 
0x23c60 : using 0x3963(918 MiB) : bluestore has 
0x46e2d(18 GiB) available


when i recreate the osd the osd gets full again

any suggestion?

thx & best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: blustore osd nearfull but no pgs on it

[Eugen Block]

After you create the OSD, run ‚ceph pg ls-by-osd {OSD}‘, it should  
show you which PGs are created there and then you’ll know which pool  
they belong to, then check again the crush rule for that pool. You can  
paste the outputs here.


Zitat von Debian :


Hi,

after a massive rebalance(tunables) my small SSD-OSDs are getting  
full, i changed my crush rules so there are actual no pgs/pools on  
it, but the disks stay full:


ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6)  
nautilus (stable)


ID CLASS WEIGHT REWEIGHT SIZE    RAW USE DATA    OMAP  
META AVAIL    %USE  VAR  PGS STATUS TYPE NAME
158   ssd    0.21999  1.0 224 GiB 194 GiB 193 GiB  22 MiB 1002  
MiB   30 GiB 86.68 1.49   0 up osd.158


inferring bluefs devices from bluestore path
1 : device size 0x37e440 : own 0x[1ad3f0~23c60] =  
0x23c60 : using 0x3963(918 MiB) : bluestore has  
0x46e2d(18 GiB) available


when i recreate the osd the osd gets full again

any suggestion?

thx & best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] blustore osd nearfull but no pgs on it

[Debian]

Hi,

after a massive rebalance(tunables) my small SSD-OSDs are getting full, 
i changed my crush rules so there are actual no pgs/pools on it, but the 
disks stay full:


ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus 
(stable)


ID CLASS WEIGHT REWEIGHT SIZE    RAW USE DATA    OMAP META 
AVAIL    %USE  VAR  PGS STATUS TYPE NAME
158   ssd    0.21999  1.0 224 GiB 194 GiB 193 GiB  22 MiB 1002 MiB   
30 GiB 86.68 1.49   0 up osd.158


inferring bluefs devices from bluestore path
1 : device size 0x37e440 : own 0x[1ad3f0~23c60] = 
0x23c60 : using 0x3963(918 MiB) : bluestore has 0x46e2d(18 
GiB) available


when i recreate the osd the osd gets full again

any suggestion?

thx & best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.1 QE Validation status

[Yuri Weinstein]

Dev Lead, pls review/edit/approve the release notes:
https://github.com/ceph/ceph/pull/54506

TIA

On Thu, Nov 16, 2023 at 10:03 AM Travis Nielsen  wrote:
>
> Rook already ran the tests against Guillaume's change directly, it looks good 
> to us. I don't see a new latest-reef-devel image tag yet, but will plan on 
> rerunning the tests when that tag is updated.
>
> Thanks,
> Travis
>
> On Thu, Nov 16, 2023 at 8:27 AM Adam King  wrote:
>>
>> Guillaume ran that patch through the orch suite earlier today before 
>> merging. I think we should be okay on that front. The issue it's fixing was 
>> also particular to rook iirc, which teuthology doesn't cover.
>>
>> On Thu, Nov 16, 2023 at 10:18 AM Yuri Weinstein  wrote:
>>>
>>> OK I will start building.
>>>
>>> Travis, Adam King - any need to rerun any suites?
>>>
>>> On Thu, Nov 16, 2023 at 7:14 AM Guillaume Abrioux  wrote:
>>> >
>>> > Hi Yuri,
>>> >
>>> >
>>> >
>>> > Backport PR [2] for reef has been merged.
>>> >
>>> >
>>> >
>>> > Thanks,
>>> >
>>> >
>>> >
>>> > [2] https://github.com/ceph/ceph/pull/54514/files
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Guillaume Abrioux
>>> >
>>> > Software Engineer
>>> >
>>> >
>>> >
>>> > From: Guillaume Abrioux 
>>> > Date: Wednesday, 15 November 2023 at 21:02
>>> > To: Yuri Weinstein , Nizamudeen A , 
>>> > Guillaume Abrioux , Travis Nielsen 
>>> > 
>>> > Cc: Adam King , Redouane Kachach 
>>> > , dev , ceph-users 
>>> > Subject: Re: [EXTERNAL] [ceph-users] Re: reef 18.2.1 QE Validation status
>>> >
>>> > Hi Yuri, (thanks)
>>> >
>>> >
>>> >
>>> > Indeed, we had a regression in ceph-volume impacting rook scenarios which 
>>> > was supposed to be fixed by [1].
>>> >
>>> > It turns out rook's CI didn't catch that fix wasn't enough for some 
>>> > reason (I believe the CI run wasn't using the right image, Travis might 
>>> > confirm or give more details).
>>> >
>>> > Another patch [2] is needed in order to fix this regression.
>>> >
>>> >
>>> >
>>> > Let me know if more details are needed.
>>> >
>>> >
>>> >
>>> > Thanks,
>>> >
>>> >
>>> >
>>> > [1] 
>>> > https://github.com/ceph/ceph/pull/54429/commits/ee26074a5e7e90b4026659bf3adb1bc973595e91
>>> >
>>> > [2] https://github.com/ceph/ceph/pull/54514/files
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Guillaume Abrioux
>>> >
>>> > Software Engineer
>>> >
>>> >
>>> >
>>> > 
>>> >
>>> > From: Yuri Weinstein 
>>> > Sent: 15 November 2023 20:23
>>> > To: Nizamudeen A ; Guillaume Abrioux 
>>> > ; Travis Nielsen 
>>> > Cc: Adam King ; Redouane Kachach 
>>> > ; dev ; ceph-users 
>>> > Subject: [EXTERNAL] [ceph-users] Re: reef 18.2.1 QE Validation status
>>> >
>>> >
>>> >
>>> > This is on behalf of Guillaume.
>>> >
>>> > We have one more last mites issue that may have to be included
>>> > https://tracker.ceph.com/issues/63545  
>>> > https://github.com/ceph/ceph/pull/54514
>>> >
>>> > Travis, Redo, Guillaume will provide more context and details.
>>> >
>>> > We are assessing the situation as 18.2.1 has been built and signed.
>>> >
>>> > On Tue, Nov 14, 2023 at 11:07 AM Yuri Weinstein  
>>> > wrote:
>>> > >
>>> > > OK thx!
>>> > >
>>> > > We have completed the approvals.
>>> > >
>>> > > On Tue, Nov 14, 2023 at 9:13 AM Nizamudeen A  wrote:
>>> > > >
>>> > > > dashboard approved. Failure known and unrelated!
>>> > > >
>>> > > > On Tue, Nov 14, 2023, 22:34 Adam King  wrote:
>>> > > >>
>>> > > >> orch approved.  After reruns, orch/cephadm was just hitting two 
>>> > > >> known (nonblocker) issues and orch/rook teuthology suite is known to 
>>> > > >> not be functional currently.
>>> > > >>
>>> > > >> On Tue, Nov 14, 2023 at 10:33 AM Yuri Weinstein 
>>> > > >>  wrote:
>>> > > >>>
>>> > > >>> Build 4 with https://github.com/ceph/ceph/pull/54224  was built and 
>>> > > >>> I
>>> > > >>> ran the tests below and asking for approvals:
>>> > > >>>
>>> > > >>> smoke - Laura
>>> > > >>> rados/mgr - PASSED
>>> > > >>> rados/dashboard - Nizamudeen
>>> > > >>> orch - Adam King
>>> > > >>>
>>> > > >>> See Build 4 runs - https://tracker.ceph.com/issues/63443#note-1
>>> > > >>>
>>> > > >>> On Tue, Nov 14, 2023 at 12:21 AM Redouane Kachach 
>>> > > >>>  wrote:
>>> > > >>> >
>>> > > >>> > Yes, cephadm has some tests for monitoring that should be enough 
>>> > > >>> > to ensure basic functionality is working properly. The rest of 
>>> > > >>> > the changes in the PR are for rook orchestrator.
>>> > > >>> >
>>> > > >>> > On Tue, Nov 14, 2023 at 5:04 AM Nizamudeen A  
>>> > > >>> > wrote:
>>> > > >>> >>
>>> > > >>> >> dashboard changes are minimal and approved. and since the 
>>> > > >>> >> dashboard change is related to the
>>> > > >>> >> monitoring stack (prometheus..) which is something not covered 
>>> > > >>> >> in the dashboard test suites, I don't think running it is 
>>> > > >>> >> necessary.
>>> > > >>> >> But maybe the cephadm suite has some monitoring stack related 
>>> > > >>> >> testings written?
>>> > > >>> >>
>>> > > >>> >> On Tue, Nov 14, 2023 at 1:10 AM Yuri 

[ceph-users] Re: RadosGW public HA traffic - best practices?

[David Orman]

I apologize, I somehow missed you cannot do BGP. I don't know of a better 
solution for you if this is the case. You'll just want to make sure to do 
graceful shutdowns of haproxy when necessary to do maintenance work to avoid 
severing active connections. At some point, though, timeouts will likely 
happens, so the impact won't be non-zero, but it also won't be catastrophic.

David

On Fri, Nov 17, 2023, at 10:09, David Orman wrote:
> Use BGP/ECMP with something like exabgp on the haproxy servers.
>
> David
>
> On Fri, Nov 17, 2023, at 04:09, Boris Behrens wrote:
>> Hi,
>> I am looking for some experience on how people make their RGW public.
>>
>> Currently we use the follow:
>> 3 IP addresses that get distributed via keepalived between three HAproxy
>> instances, which then balance to three RGWs.
>> The caveat is, that keepalived is PITA to get working in distributing a set
>> of IP addresses, and it doesn't scale very well (up and down).
>> The upside is, that it is really stable and customer nearly never have an
>> availability problem. And we have 3 IPs that make some sort of LB. It
>> serves up to 24Gbit in peak times, when all those backup jobs are running
>> at night.
>>
>> But today I thought, what will happen if I just ditch the keepalived and
>> configure thos addresses static to the haproxy hosts?
>> How bad will the impact to a customer if I reboot one haproxy? Is there an
>> easier, more scalable way if I want to spread the load even further without
>> having an ingress HW LB (what I don't have)?
>>
>> I have a lot of hosts that would be able to host some POD with a haproxy
>> and a RGW as container together, or even host the RGW alone in a container.
>> It would just need to bridge two networks.
>> But I currently do not have a way to use BGP to have one IP address split
>> between a set of RGW instances.
>>
>> So long story short:
>> What are your easy setups to serve public RGW traffic with some sort of HA
>> and LB (without using a big HW LB that is capable of 100GBit traffic)?
>> And have you experienced problems when you do not shift around IP addresses.
>>
>> Cheers
>>  Boris
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW public HA traffic - best practices?

[David Orman]

Use BGP/ECMP with something like exabgp on the haproxy servers.

David

On Fri, Nov 17, 2023, at 04:09, Boris Behrens wrote:
> Hi,
> I am looking for some experience on how people make their RGW public.
>
> Currently we use the follow:
> 3 IP addresses that get distributed via keepalived between three HAproxy
> instances, which then balance to three RGWs.
> The caveat is, that keepalived is PITA to get working in distributing a set
> of IP addresses, and it doesn't scale very well (up and down).
> The upside is, that it is really stable and customer nearly never have an
> availability problem. And we have 3 IPs that make some sort of LB. It
> serves up to 24Gbit in peak times, when all those backup jobs are running
> at night.
>
> But today I thought, what will happen if I just ditch the keepalived and
> configure thos addresses static to the haproxy hosts?
> How bad will the impact to a customer if I reboot one haproxy? Is there an
> easier, more scalable way if I want to spread the load even further without
> having an ingress HW LB (what I don't have)?
>
> I have a lot of hosts that would be able to host some POD with a haproxy
> and a RGW as container together, or even host the RGW alone in a container.
> It would just need to bridge two networks.
> But I currently do not have a way to use BGP to have one IP address split
> between a set of RGW instances.
>
> So long story short:
> What are your easy setups to serve public RGW traffic with some sort of HA
> and LB (without using a big HW LB that is capable of 100GBit traffic)?
> And have you experienced problems when you do not shift around IP addresses.
>
> Cheers
>  Boris
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm user on cephadm rpm package

[David C.]

If you provision the binary (python script) cephadm yourself and your
users, you should be able to do without the cephadm rpm.



Le ven. 17 nov. 2023 à 14:04, Luis Domingues  a
écrit :

> So I guess I need to install the cephadm rpm packages on all my machines
> then?
>
> I like the idea of not having a root user, and in fact we do it on our
> clusters. But as we need to push ssh keys to the user config, so we manage
> users outside of ceph, during OS provisioning.
> So it look a little bit redundant to have cephadm package to create that
> user, when we need to figure out how to enable cephadm's access to the
> machines.
>
> Anyway, thanks for your reply.
>
> Luis Domingues
> Proton AG
>
>
> On Friday, 17 November 2023 at 13:55, David C. 
> wrote:
>
>
> > Hi,
> >
> > You can use the cephadm account (instead of root) to control machines
> with
> > the orchestrator.
> >
> >
> > Le ven. 17 nov. 2023 à 13:30, Luis Domingues luis.doming...@proton.ch a
> >
> > écrit :
> >
> > > Hi,
> > >
> > > I noticed when installing the cephadm rpm package, to bootstrap a
> cluster
> > > for example, that a user cephadm was created. But I do not see it used
> > > anywhere.
> > >
> > > What is the purpose of creating a user on the machine we install the
> local
> > > binary of cephadm?
> > >
> > > Luis Domingues
> > > Proton AG
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm user on cephadm rpm package

[Luis Domingues]

So I guess I need to install the cephadm rpm packages on all my machines then?

I like the idea of not having a root user, and in fact we do it on our 
clusters. But as we need to push ssh keys to the user config, so we manage 
users outside of ceph, during OS provisioning. 
So it look a little bit redundant to have cephadm package to create that user, 
when we need to figure out how to enable cephadm's access to the machines.

Anyway, thanks for your reply.

Luis Domingues
Proton AG


On Friday, 17 November 2023 at 13:55, David C.  wrote:


> Hi,
> 
> You can use the cephadm account (instead of root) to control machines with
> the orchestrator.
> 
> 
> Le ven. 17 nov. 2023 à 13:30, Luis Domingues luis.doming...@proton.ch a
> 
> écrit :
> 
> > Hi,
> > 
> > I noticed when installing the cephadm rpm package, to bootstrap a cluster
> > for example, that a user cephadm was created. But I do not see it used
> > anywhere.
> > 
> > What is the purpose of creating a user on the machine we install the local
> > binary of cephadm?
> > 
> > Luis Domingues
> > Proton AG
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm user on cephadm rpm package

[David C.]

Hi,

You can use the cephadm account (instead of root) to control machines with
the orchestrator.


Le ven. 17 nov. 2023 à 13:30, Luis Domingues  a
écrit :

> Hi,
>
> I noticed when installing the cephadm rpm package, to bootstrap a cluster
> for example, that a user cephadm was created. But I do not see it used
> anywhere.
>
> What is the purpose of creating a user on the machine we install the local
> binary of cephadm?
>
> Luis Domingues
> Proton AG
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephadm user on cephadm rpm package

[Luis Domingues]

Hi,

I noticed when installing the cephadm rpm package, to bootstrap a cluster for 
example, that a user cephadm was created. But I do not see it used anywhere.

What is the purpose of creating a user on the machine we install the local 
binary of cephadm?

Luis Domingues
Proton AG
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: No SSL Dashboard working after installing mgr crt|key with RSA/4096 secp384r1

[Eugen Block]

I was able to reproduce the error with a self-signed elliptic curves  
based certificate. But I also got out of it by removing cert and key:


quincy-1:~ # ceph config-key rm mgr/dashboard/key
key deleted
quincy-1:~ # ceph config-key rm mgr/dashboard/crt
key deleted

Then I failed the mgr just to be sure:

quincy-1:~ # ceph mgr fail
quincy-1:~ # ceph config-key get mgr/dashboard/crt
Error ENOENT:

And then I configured the previous key, did a mgr fail and now the  
dashboard is working again.


Zitat von Eugen Block :


Hi,

did you get your dashboard back in the meantime? I don't have an  
answer regarding the certificate based on elliptic curves but since  
you wrote:



So we tried to go back to the original state by removing CRT anf KEY but
without success. The new key seems to stuck into the config


how did you try to remove it? I would just assume that this should work:

$ ceph config-key rm mgr/dashboard/cert

Do you get an error message when removing it or does the mgr log  
anything when you try to remove it which fails?

Also which ceph version is this?

Thanks,
Eugen

Zitat von "Ackermann, Christoph" :


Hello all,

today i got a new certificate for our internal domain based on  RSA/4096
secp384r1. After inserting  CRT and Key i got both "...updated" messages.
After checking the dashboard i got an empty page and this error:

  health: HEALTH_ERR
  Module 'dashboard' has failed: key type unsupported

So we tried to go back to the original state by removing CRT anf KEY but
without success. The new key seems to stuck into the config

[root@ceph ~]# ceph config-key get mgr/dashboard/crt
-BEGIN CERTIFICATE-
MIIFqTCCBJGgAwIBAgIMB5tjLSz264Ic8zeHMA0GCSqGSIb3DQEBCwUAMEwxCzAJ
[...]
ItzkEzq4SZ6V1Jhuf4bFlOMBVAKgAwZ90gXlguoiFFQu5+NIqNljZ8Jz7d0jhH43
e3zhm5sn21+eIqRbiQ==
-END CERTIFICATE-

[root@ceph ~]# ceph config-key get mgr/dashboard/key

*Error ENOENT: *

We tried to generate a self signed Cert but no luck. It looks like manger
stays in an intermediate state. The only way to get back the dashboard is
to disable SSL  and use plain http.

Can somebody explain this behaviour?  Maybe secp384r1 elliptic curves
aren't supported? How can we clean up SSL configuration?

Thanks,
Christoph Ackermann

Ps we checked some Information like
https://tracker.ceph.com/issues/57924#change-227744 and others  but  no
luck...
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem while upgrade 17.2.6 to 17.2.7

[David C.]

Hi,

don't you have a traceback below that ?

You probably have a communication problem (ssl ? ) between the dashboard
and the rgw.

Maybe check the settings: ceph dashboard get-rgw-api-*
=>
https://docs.ceph.com/en/quincy/mgr/dashboard/#enabling-the-object-gateway-management-frontend




Le ven. 17 nov. 2023 à 11:22, Jean-Marc FONTANA 
a écrit :

> Hello, everyone,
>
> There's nothing cephadm.log in /var/log/ceph.
>
> To get something else, we tried what David C. proposed (thanks to him !!)
> and found:
>
> nov. 17 10:53:54 svtcephmonv3 ceph-mgr[727]: [balancer ERROR root] execute
> error: r = -1, detail = min_compat_client jewel < luminous, which is
> required for pg-upmap. Try 'ceph osd set-require-min-compat-client
> luminous' before using the new interface
> nov. 17 10:54:54 svtcephmonv3 ceph-mgr[727]: [balancer ERROR root] execute
> error: r = -1, detail = min_compat_client jewel < luminous, which is
> required for pg-upmap. Try 'ceph osd set-require-min-compat-client
> luminous' before using the new interface
> nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR exception]
> Internal Server Error
> nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request]
> [:::192.168.114.32:53414] [GET] [500] [0.026s] [testadmin] [513.0B]
> /api/rgw/daemon
> nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request]
> [b'{"status": "500 Internal Server Error", "detail": "The server
> encountered an unexpected condition which prevented it from fulfilling the
> request.", "request_id":
> "961b2a25-5c14-4c67-a82a-431f08684f80"}
> ']
> nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR exception]
> Internal Server Error
> nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request]
> [:::192.168.114.32:53409] [GET] [500] [0.012s] [testadmin] [513.0B]
> /api/rgw/daemon
> nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request]
> [b'{"status": "500 Internal Server Error", "detail": "The server
> encountered an unexpected condition which prevented it from fulfilling the
> request.", "request_id": "baf41a81-1e6b-4422-97a7-bd96b832dc5a"}
>
> The error about min_compat_client has been fixed with the suggested
> command ( that is a nice result :) ),
> but the web interface still keeps on going on error.
>
> Thanks for your helping,
>
> JM
> Le 17/11/2023 à 07:33, Nizamudeen A a écrit :
>
> Hi,
>
> I think it should be in /var/log/ceph/ceph-mgr..log, probably you
> can reproduce this error again and hopefully
> you'll be able to see a python traceback or something related to rgw in the
> mgr logs.
>
> Regards
>
> On Thu, Nov 16, 2023 at 7:43 PM Jean-Marc FONTANA  
> 
> wrote:
>
>
> Hello,
>
> These are the last lines of /var/log/ceph/cephadm.log of the active mgr
> machine after an error occured.
> As I don't feel this will be very helpfull, would you please tell us where
> to look ?
>
> Best regards,
>
> JM Fontana
>
> 2023-11-16 14:45:08,200 7f341eae8740 DEBUG
> 
> cephadm ['--timeout', '895', 'gather-facts']
> 2023-11-16 14:46:10,406 7fca81386740 DEBUG
> 
> cephadm ['--timeout', '895', 'gather-facts']
> 2023-11-16 14:47:12,594 7fd48f814740 DEBUG
> 
> cephadm ['--timeout', '895', 'gather-facts']
> 2023-11-16 14:48:14,857 7fd0b24b1740 DEBUG
> 
> cephadm ['--timeout', '895', 'check-host']
> 2023-11-16 14:48:14,990 7fd0b24b1740 INFO podman (/usr/bin/podman) version
> 3.0.1 is present
> 2023-11-16 14:48:14,992 7fd0b24b1740 INFO systemctl is present
> 2023-11-16 14:48:14,993 7fd0b24b1740 INFO lvcreate is present
> 2023-11-16 14:48:15,041 7fd0b24b1740 INFO Unit chrony.service is enabled
> and running
> 2023-11-16 14:48:15,043 7fd0b24b1740 INFO Host looks OK
> 2023-11-16 14:48:15,655 7f36b81fd740 DEBUG
> 
> cephadm ['--image', 
> 'quay.io/ceph/ceph@sha256:56984a149e89ce282e9400ca53371ff7df74b1c7f5e979b6ec651b751931483a',
> '--timeout', '895', 'ls']
> 2023-11-16 14:48:17,662 7f17bfc28740 DEBUG
> 
> cephadm ['--timeout', '895', 'gather-facts']
> 2023-11-16 14:49:20,131 7fc8a9cc1740 DEBUG
> 
> cephadm ['--timeout', '895', 'gather-facts']
> 2023-11-16 14:50:22,284 7f1a6a7eb740 DEBUG
> 
> cephadm ['--timeout', '895', 'gather-facts']
> 2023-11-16 14:51:24,505 7f1798dd5740 DEBUG
> 
> cephadm ['--timeout', '895', 'gather-facts']
> 2023-11-16 14:52:26,574 

[ceph-users] Re: How to use hardware

[David C.]

Hi Albert ,

5 instead of 3 mon will allow you to limit the impact if you break a mon
(for example, with the file system full)

5 instead of 3 MDS, this makes sense if the workload can be distributed
over several trees in your file system. Sometimes it can also make sense to
have several FSs in order to limit the consequences of an infrastructure
with several active MDSs.

Concerning performance, if you see a node that is too busy which impacts
the cluster, you can always think about relocating certain services.



Le ven. 17 nov. 2023 à 11:00, Albert Shih  a écrit :

> Hi everyone,
>
> In the purpose to deploy a medium size of ceph cluster (300 OSD) we have 6
> bare-metal server for the OSD, and 5 bare-metal server for the service
> (MDS, Mon, etc.)
>
> Those 5 bare-metal server have each 48 cores and 256 Gb.
>
> What would be the smartest way to use those 5 server, I see two way :
>
>   first :
>
> Server 1 : MDS,MON, grafana, prometheus, webui
> Server 2:  MON
> Server 3:  MON
> Server 4 : MDS
> Server 5 : MDS
>
>   so 3 MDS, 3 MON. and we can loose 2 servers.
>
>   Second
>
> KVM on each server
>   Server 1 : 3 VM : One for grafana & CIe, and 1 MDS, 2 MON
>   other server : 1 MDS, 1 MON
>
>   in total :  5 MDS, 5 MON and we can loose 4 servers.
>
> So on paper it's seem the second are smarter, but it's also more complex,
> so my question are «is it worth the complexity to have 5 MDS/MON for 300
> OSD».
>
> Important : The main goal of this ceph cluster are not to get the maximum
> I/O speed, I would not say the speed is not a factor, but it's not the main
> point.
>
> Regards.
>
>
> --
> Albert SHIH 嶺 
> Observatoire de Paris
> France
> Heure locale/Local time:
> ven. 17 nov. 2023 10:49:27 CET
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem while upgrade 17.2.6 to 17.2.7

[Jean-Marc FONTANA]

Hello, everyone,

There's nothing cephadm.log in /var/log/ceph.

To get something else, we tried what David C. proposed (thanks to him 
!!) and found:


nov. 17 10:53:54 svtcephmonv3 ceph-mgr[727]: [balancer ERROR root] 
execute error: r = -1, detail = min_compat_client jewel < luminous, 
which is required for pg-upmap. Try 'ceph osd 
set-require-min-compat-client luminous' before using the new interface
nov. 17 10:54:54 svtcephmonv3 ceph-mgr[727]: [balancer ERROR root] 
execute error: r = -1, detail = min_compat_client jewel < luminous, 
which is required for pg-upmap. Try 'ceph osd 
set-require-min-compat-client luminous' before using the new interface
nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR exception] 
Internal Server Error
nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request] 
[:::192.168.114.32:53414] [GET] [500] [0.026s] [testadmin] [513.0B] 
/api/rgw/daemon
nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request] 
[b'{"status": "500 Internal Server Error", "detail": "The server 
encountered an unexpected condition which prevented it from fulfilling 
the request.", "request_id": "961b2a25-5c14-4c67-a82a-431f08684f80"} ']
nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR exception] 
Internal Server Error
nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request] 
[:::192.168.114.32:53409] [GET] [500] [0.012s] [testadmin] [513.0B] 
/api/rgw/daemon
nov. 17 10:55:56 svtcephmonv3 ceph-mgr[727]: [dashboard ERROR request] 
[b'{"status": "500 Internal Server Error", "detail": "The server 
encountered an unexpected condition which prevented it from fulfilling 
the request.", "request_id": "baf41a81-1e6b-4422-97a7-bd96b832dc5a"}


The error about min_compat_client has been fixed with the suggested 
command ( that is a nice result :) ),

but the web interface still keeps on going on error.

Thanks for your helping,

JM

Le 17/11/2023 à 07:33, Nizamudeen A a écrit :

Hi,

I think it should be in /var/log/ceph/ceph-mgr..log, probably you
can reproduce this error again and hopefully
you'll be able to see a python traceback or something related to rgw in the
mgr logs.

Regards

On Thu, Nov 16, 2023 at 7:43 PM Jean-Marc FONTANA
wrote:


Hello,

These are the last lines of /var/log/ceph/cephadm.log of the active mgr
machine after an error occured.
As I don't feel this will be very helpfull, would you please tell us where
to look ?

Best regards,

JM Fontana

2023-11-16 14:45:08,200 7f341eae8740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:46:10,406 7fca81386740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:47:12,594 7fd48f814740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:48:14,857 7fd0b24b1740 DEBUG

cephadm ['--timeout', '895', 'check-host']
2023-11-16 14:48:14,990 7fd0b24b1740 INFO podman (/usr/bin/podman) version
3.0.1 is present
2023-11-16 14:48:14,992 7fd0b24b1740 INFO systemctl is present
2023-11-16 14:48:14,993 7fd0b24b1740 INFO lvcreate is present
2023-11-16 14:48:15,041 7fd0b24b1740 INFO Unit chrony.service is enabled
and running
2023-11-16 14:48:15,043 7fd0b24b1740 INFO Host looks OK
2023-11-16 14:48:15,655 7f36b81fd740 DEBUG

cephadm ['--image', '
quay.io/ceph/ceph@sha256:56984a149e89ce282e9400ca53371ff7df74b1c7f5e979b6ec651b751931483a',
'--timeout', '895', 'ls']
2023-11-16 14:48:17,662 7f17bfc28740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:49:20,131 7fc8a9cc1740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:50:22,284 7f1a6a7eb740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:51:24,505 7f1798dd5740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:52:26,574 7f0185a55740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:53:28,630 7f9bc3fff740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:54:30,673 7fc3752d0740 DEBUG

cephadm ['--timeout', '895', 'gather-facts']
2023-11-16 14:55:32,662 7fd3865f8740 DEBUG

[ceph-users] RadosGW public HA traffic - best practices?

[Boris Behrens]

Hi,
I am looking for some experience on how people make their RGW public.

Currently we use the follow:
3 IP addresses that get distributed via keepalived between three HAproxy
instances, which then balance to three RGWs.
The caveat is, that keepalived is PITA to get working in distributing a set
of IP addresses, and it doesn't scale very well (up and down).
The upside is, that it is really stable and customer nearly never have an
availability problem. And we have 3 IPs that make some sort of LB. It
serves up to 24Gbit in peak times, when all those backup jobs are running
at night.

But today I thought, what will happen if I just ditch the keepalived and
configure thos addresses static to the haproxy hosts?
How bad will the impact to a customer if I reboot one haproxy? Is there an
easier, more scalable way if I want to spread the load even further without
having an ingress HW LB (what I don't have)?

I have a lot of hosts that would be able to host some POD with a haproxy
and a RGW as container together, or even host the RGW alone in a container.
It would just need to bridge two networks.
But I currently do not have a way to use BGP to have one IP address split
between a set of RGW instances.

So long story short:
What are your easy setups to serve public RGW traffic with some sort of HA
and LB (without using a big HW LB that is capable of 100GBit traffic)?
And have you experienced problems when you do not shift around IP addresses.

Cheers
 Boris
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How to use hardware

[Albert Shih]

Hi everyone, 

In the purpose to deploy a medium size of ceph cluster (300 OSD) we have 6
bare-metal server for the OSD, and 5 bare-metal server for the service
(MDS, Mon, etc.)

Those 5 bare-metal server have each 48 cores and 256 Gb. 

What would be the smartest way to use those 5 server, I see two way : 

  first : 

Server 1 : MDS,MON, grafana, prometheus, webui
Server 2:  MON
Server 3:  MON
Server 4 : MDS
Server 5 : MDS

  so 3 MDS, 3 MON. and we can loose 2 servers.

  Second 

KVM on each server
  Server 1 : 3 VM : One for grafana & CIe, and 1 MDS, 2 MON
  other server : 1 MDS, 1 MON

  in total :  5 MDS, 5 MON and we can loose 4 servers. 

So on paper it's seem the second are smarter, but it's also more complex,
so my question are «is it worth the complexity to have 5 MDS/MON for 300
OSD». 

Important : The main goal of this ceph cluster are not to get the maximum
I/O speed, I would not say the speed is not a factor, but it's not the main
point. 

Regards.


-- 
Albert SHIH 嶺 
Observatoire de Paris
France
Heure locale/Local time:
ven. 17 nov. 2023 10:49:27 CET
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: No SSL Dashboard working after installing mgr crt|key with RSA/4096 secp384r1

[Eugen Block]

Hi,

did you get your dashboard back in the meantime? I don't have an  
answer regarding the certificate based on elliptic curves but since  
you wrote:



So we tried to go back to the original state by removing CRT anf KEY but
without success. The new key seems to stuck into the config


how did you try to remove it? I would just assume that this should work:

$ ceph config-key rm mgr/dashboard/cert

Do you get an error message when removing it or does the mgr log  
anything when you try to remove it which fails?

Also which ceph version is this?

Thanks,
Eugen

Zitat von "Ackermann, Christoph" :


Hello all,

today i got a new certificate for our internal domain based on  RSA/4096
secp384r1. After inserting  CRT and Key i got both "...updated" messages.
After checking the dashboard i got an empty page and this error:

   health: HEALTH_ERR
   Module 'dashboard' has failed: key type unsupported

So we tried to go back to the original state by removing CRT anf KEY but
without success. The new key seems to stuck into the config

[root@ceph ~]# ceph config-key get mgr/dashboard/crt
-BEGIN CERTIFICATE-
MIIFqTCCBJGgAwIBAgIMB5tjLSz264Ic8zeHMA0GCSqGSIb3DQEBCwUAMEwxCzAJ
[...]
ItzkEzq4SZ6V1Jhuf4bFlOMBVAKgAwZ90gXlguoiFFQu5+NIqNljZ8Jz7d0jhH43
e3zhm5sn21+eIqRbiQ==
-END CERTIFICATE-

[root@ceph ~]# ceph config-key get mgr/dashboard/key

*Error ENOENT: *

We tried to generate a self signed Cert but no luck. It looks like manger
stays in an intermediate state. The only way to get back the dashboard is
to disable SSL  and use plain http.

Can somebody explain this behaviour?  Maybe secp384r1 elliptic curves
aren't supported? How can we clean up SSL configuration?

Thanks,
Christoph Ackermann

Ps we checked some Information like
https://tracker.ceph.com/issues/57924#change-227744 and others  but  no
luck...
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Issue with using the block device inside a pod.

[Eugen Block]

Hi,

can you share the auth caps for your k8s client?

ceph auth get client.

And maybe share the yaml files as well (redact sensitive data) so we  
can get a full picture.


Zitat von Kushagr Gupta :


Hi Team,

Components:
Kubernetes, Ceph

Problem statement:
We are trying to integrate Ceph with kubernetes.
We are unable to utilize the block volume mode in the pod.

Description:
OS: Almalinux 8.8
Ceph version: 18.2
Kubernetes version: 1.28.2

We have deployed a single node kubernetes cluster and a 3 node ceph cluster.
We are trying to integrate kubernetes with ceph using csi-rbd plugin.
We have used the following link to do the same:
https://docs.ceph.com/en/reef/rbd/rbd-kubernetes/

We are strugling with the usage of the pvc we created in the block mode.
Kindly refer the "raw-block-pvc-4.yaml".
We created a pod using the file "raw-block-pod-4.yaml".

The pod was created successfully. The device path we used was /dev/xvda.
We are unable to use this path inside the pod. Its not gettting shown in
the lsblk of the pod and its not even shown in the df -kh of the pod.
When we try to mount /dev/xvda on /tmp but we get permission denied error.
We get a similar error when we try to make partiotion of the disk. The
partition gets created but partiotion table is not updated.
At the same time we are able to format the disk using the command mkfs.ext4
/dev/xvda.
Kindly refer the output:
"
[root@kube-cluster-1 ceph_provisioner]# k exec -it
pod-with-raw-block-volume-2 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future
version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.4# hostname
pod-with-raw-block-volume-2
sh-4.4# df -kh
Filesystem  Size  Used Avail Use% Mounted on
overlay  70G   14G   57G  19% /
tmpfs64M 0   64M   0% /dev
tmpfs32G 0   32G   0% /sys/fs/cgroup
/dev/mapper/almalinux-root   70G   14G   57G  19% /etc/hosts
shm  64M 0   64M   0% /dev/shm
tmpfs63G   12K   63G   1% /run/secrets/
kubernetes.io/serviceaccount
tmpfs32G 0   32G   0% /proc/acpi
tmpfs32G 0   32G   0% /proc/scsi
tmpfs32G 0   32G   0% /sys/firmware
sh-4.4# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop07:00 1G  0 loop
loop17:10   500G  0 loop
sda  8:00 446.6G  0 disk
|-sda1   8:1200M  0 part
|-sda2   8:2  1M  0 part
|-sda3   8:3  1G  0 part
`-sda4   8:40 445.4G  0 part
sdb  8:16   0 893.3G  0 disk
rbd0   252:00 1G  0 disk
rbd1   252:16   0   500G  0 disk
rbd2   252:32   0   100G  0 disk
sh-4.4# ls /dev
core  fd  full  mqueue  null  ptmx  pts  random  shm  stderr  stdin  stdout
 termination-log  tty  urandom  xvda  zero
sh-4.4#
"

Altough we are able to see a volume corresponding to the PV.
But we are not able to use the block device.

Could anyone help me out.

Thanks and Regards,
Kushagra Gupta
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Large size differences between pgs

[Eugen Block]

Hi,

if you could share some more info about your cluster you might get a  
better response. For example, 'ceph osd df tree' could be helpful to  
get an impression how many PGs you currently have. You can inspect the  
'ceph pg dump' output and look for the column "BYTES" which tells you  
how large a PG is. It's not uncommon to have PGs sizes of 50 GB or  
more, one of our customers had 250 GB per PG because they couldn't  
tell how much data would be stored in their pools so the initial  
pg_num values didn't change for a while.
So getting back to your cluster, assuming that your PGs have a size of  
around 25 GB it would be expected that with a deviation of 1 PG the  
difference would be 25 GB. Depending on the current PG distribution  
you could consider splitting them, but note the risks of having too  
many PGs per OSD.


Regards,
Eugen

Zitat von Miroslav Svoboda :

Namely, the problem what I am trying to solve is that with such a  
large cluster I will lose a lot of  capacity like unused. I have  
deviation set to value 1 at the balancer, that is, if I'm not  
mistaken +-1pg per OSD and then due to the size dispersion between  
the largest and smallest PGs on the one OSDs host.Svoboda Miroslav
 Původní zpráva Od: Anthony D'Atri  
 Datum: 15.11.23  21:54  (GMT+01:00) Komu:  
Miroslav Svoboda  Předmět: Re:  
[ceph-users] Large size differences between pgs How are you  
determining PG size?> On Nov 15, 2023, at 15:46, Miroslav Svoboda  
 wrote:> > Hi,is it possible decrease  
large size differences between pgs? I have 5PB cluster and  
differences between smalest and bigest pgs are somewhere about  
25GB.thanks,Svoboda Miroslav>  
___> ceph-users mailing  
list -- ceph-users@ceph.io> To unsubscribe send an email to  
ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Different behaviors for ceph kernel client in limiting IOPS when data pool enters `nearfull`?

[Xiubo Li]

On 11/17/23 00:41, Ilya Dryomov wrote:

On Thu, Nov 16, 2023 at 5:26 PM Matt Larson  wrote:

Ilya,

  Thank you for providing these discussion threads on the Kernel fixes for 
where there was a change and details on this affects the clients.

  What is the expected behavior in CephFS client when there are multiple data 
pools in the CephFS? Does having 'nearfull' in any data pool in the CephFS then 
trigger the synchronous writes for clients even if they would be writing to a 
CephFS location mapped to a non-nearfull data pool? I.e. is 'nearfull' / sync 
behavior global across the same CephFS filesystem?

I would expect it to apply only to the pool in question (i.e. not be
global), but let's get Xiubo or someone else working on CephFS to
confirm.


It seems just before mimic when any pool is nearfull it will affect the 
whole cephfs, and after since mimic the 'CEPH_OSDMAP_NEARFULL' has been 
deprecated, so it will depends on each pool.


-#define CEPH_OSDMAP_NEARFULL (1<<0)  /* sync writes (near 
ENOSPC) */

-#define CEPH_OSDMAP_FULL (1<<1)  /* no data writes (ENOSPC) */
+#define CEPH_OSDMAP_NEARFULL (1<<0)  /* sync writes (near 
ENOSPC), deprecated since mimic*/
+#define CEPH_OSDMAP_FULL (1<<1)  /* no data writes 
(ENOSPC), deprecated since mimic */


Thanks

- Xiubo



Thanks,

 Ilya


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io