[ceph-users] Ambigous mds behind on trimming and slowps issue on ceph 17.2.5 with rook 1.10.8 operator

[Akash Warkhade]

Guys,

We were facing cephFs volume mount issue and ceph status it was showing
 mds slow requests
 Mds behind on trimming

After restarting mds pods it was resolved
But wanted to know Root caus of this
It was started after 2 hours of one of the active mds was crashed
So does that an active mds crash can cause this issue ?


Please provide your inputs anyone
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific Bug?

[Alex]

Thank you.
Appreciate the prompt response.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific Bug?

[Adam King]

Does seem like a bug, actually in more than just this command. The `ceph
orch host ls` with the --label and/or --host-pattern flag just piggybacks
off of the existing filtering done for placements in service specs. I've
just taken a look and you actually can create the same behavior with the
placement of an actual service, for example, with

[ceph: root@vm-00 /]# ceph orch host ls
HOST   ADDR LABELS  STATUS
vm-00  192.168.122.7_admin
vm-01  192.168.122.171  foo
vm-02  192.168.122.147  foo
3 hosts in cluster

and spec

[ceph: root@vm-00 /]# cat ne.yaml
service_type: node-exporter
service_name: node-exporter
placement:
  host_pattern: 'vm-0[0-1]'

you get the expected placement on vm-00 and vm-01

[ceph: root@vm-00 /]# ceph orch ps --daemon-type node-exporter
NAME HOST   PORTS   STATUS REFRESHED  AGE  MEM USE
 MEM LIM  VERSION  IMAGE ID  CONTAINER ID
node-exporter.vm-00  vm-00  *:9100  running (23s)17s ago  23s3636k
   -  1.5.00da6a335fe13  f83e88caa7e0
node-exporter.vm-01  vm-01  *:9100  running (21h) 2m ago  21h16.1M
   -  1.5.00da6a335fe13  a5153c378449

but if I add label to the placement, while still leaving in the host pattern

[ceph: root@vm-00 /]# cat ne.yaml
service_type: node-exporter
service_name: node-exporter
placement:
  label: foo
  host_pattern: 'vm-0[0-1]'

you would expect to only get vm-01 at this point, as it's the only host
that matches both pieces of the placement, but instead you get both vm-01
and vm-02

[ceph: root@vm-00 /]# ceph orch ps --daemon-type node-exporter
NAME HOST   PORTS   STATUS REFRESHED  AGE  MEM USE
 MEM LIM  VERSION  IMAGE ID  CONTAINER ID
node-exporter.vm-01  vm-01  *:9100  running (21h) 4m ago  21h16.1M
   -  1.5.00da6a335fe13  a5153c378449
node-exporter.vm-02  vm-02  *:9100  running (23s)18s ago  23s5410k
   -  1.5.00da6a335fe13  ddd1e643e341

Looking at the scheduling implementation, it seems currently it selects
candidates based on attributes in this order: Explicit host list, label,
host pattern (with some additional handling for count that happens in all
cases). When it finds the first thing in that list, in this case the label,
that is present in the placement, it uses that to select the candidates and
then bails out without any additional filtering on the host pattern
attribute. Since the placement spec validation doesn't allow applying specs
with both host_pattern/label and an explicit host list, this case with the
label and host pattern is the only one you can hit where this is an issue,
and I guess was just overlooked. Will take a look at making a patch to fix
this.

On Tue, Feb 13, 2024 at 7:09 PM Alex  wrote:

> Hello Ceph Gurus!
>
> I'm running Ceph Pacific version.
> if I run
> ceph orch host ls --label osds
> shows all hosts label osds
> or
> ceph orch host ls --host-pattern host1
> shows just host1
> it works as expected
>
> But combining the two the label tag seems to "take over"
>
> ceph orch host ls --label osds --host-pattern host1
> 6 hosts in cluster who had label osds whose hostname matched host1
> shows all host with the label osds instead of only host1.
> So at first the flags seem to act like an OR instead of an AND.
>
> ceph orch host ls --label osds --host-pattern foo
> 6 hosts in cluster who had label osds whose hostname matched foo
> even though "foo" doesn't even exist
>
> ceph orch host ls --label bar --host-pattern host1
> 0 hosts in cluster who had label bar whose hostname matched host1
> if the label and host combo was an OR this should have worked
> there is no label bar but host1 exists so it just disregards the
> host-pattern.
>
> This started because the osd deployment task had both label and
> host_pattern.
> The cluster was attempting to deploy OSDS on all the servers with the
> given tag instead of the one host we needed,
> which caused it to go into warning state.
> If I ran
> ceph orch ls --export --service_name host1
> it also showed both tags and host_pattern.
> unmanaged: false
> placement:
>   host_pattern:
>   label:
> The issue persisted until I removed the label tag.
>
> Thanks.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific Bug?

[Eugen Block]

Hi,

I don't think this is just limited to Pacific, besides the overriding  
nature of the label parameter, Reef also shows some weird results  
(it's an upgraded test cluster so don't mind the hostnames):


nautilus:~ # ceph orch host ls
HOST   ADDR LABELS  STATUS
nautilus   192.168.168.111  mon,osd,mds,_admin
nautilus2  192.168.168.112  mon,osd,mds,_admin
nautilus3  192.168.168.113  mon,osd,mds,_admin
3 hosts in cluster

I have clearly 3 hosts all with the same labels. But selecting a  
specific label doesn't list all the hosts:


nautilus:~ # ceph orch host ls --label osd
HOST   ADDR LABELS  STATUS
nautilus   192.168.168.111  mon,osd,mds,_admin
nautilus2  192.168.168.112  mon,osd,mds,_admin
2 hosts in cluster who had label osd

nautilus:~ # ceph orch host ls --label mon
HOST   ADDR LABELS  STATUS
nautilus   192.168.168.111  mon,osd,mds,_admin
nautilus2  192.168.168.112  mon,osd,mds,_admin
2 hosts in cluster who had label mon

Maybe I misunderstand the term "host_pattern", but I would expect that  
if I select "nautilus" as host-pattern all three hosts should be  
listed, but only the one matching the exact hostname shows up:


nautilus:~ # ceph orch host ls --host-pattern nautilus
HOST  ADDR LABELS  STATUS
nautilus  192.168.168.111  mon,osd,mds,_admin
1 hosts in cluster whose hostname matched nautilus

I haven't checked tracker.ceph.com for existing reports, have you?

Thanks,
Eugen

Zitat von Alex :


Hello Ceph Gurus!

I'm running Ceph Pacific version.
if I run
ceph orch host ls --label osds
shows all hosts label osds
or
ceph orch host ls --host-pattern host1
shows just host1
it works as expected

But combining the two the label tag seems to "take over"

ceph orch host ls --label osds --host-pattern host1
6 hosts in cluster who had label osds whose hostname matched host1
shows all host with the label osds instead of only host1.
So at first the flags seem to act like an OR instead of an AND.

ceph orch host ls --label osds --host-pattern foo
6 hosts in cluster who had label osds whose hostname matched foo
even though "foo" doesn't even exist

ceph orch host ls --label bar --host-pattern host1
0 hosts in cluster who had label bar whose hostname matched host1
if the label and host combo was an OR this should have worked
there is no label bar but host1 exists so it just disregards the  
host-pattern.


This started because the osd deployment task had both label and host_pattern.
The cluster was attempting to deploy OSDS on all the servers with the
given tag instead of the one host we needed,
which caused it to go into warning state.
If I ran
ceph orch ls --export --service_name host1
it also showed both tags and host_pattern.
unmanaged: false
placement:
  host_pattern:
  label:
The issue persisted until I removed the label tag.

Thanks.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: concept of ceph and 2 datacenters

[Anthony D'Atri]

Notably, the tiebreaker should be in a third location.

> On Feb 14, 2024, at 05:16, Peter Sabaini  wrote:
> 
> On 14.02.24 06:59, Vladimir Sigunov wrote:
>> Hi Ronny,
>> This is a good starting point for your design.
>> https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
>> 
>> My personal experience says that 2 DC Ceph deployment could suffer from a 
>> 'split brain' situation. If you have any chance to create a 3 DC 
>> configuration, I would suggest to consider it. It could be more expensive, 
>> but it definitely will be more reliable and fault tolerant.
> 
> The docs you linked mention[0] a tiebreaker monitor (which could be a VM / in 
> the cloud) -- have you used something like this?
> 
> [0] 
> https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#limitations-of-stretch-mode
> 
> cheers,
> peter.
> 
>> Sincerely,
>> Vladimir
>> 
>> Get Outlook for Android
>> 
>> From: ronny.lipp...@spark5.de 
>> Sent: Tuesday, February 13, 2024 6:50:50 AM
>> To: ceph-users@ceph.io 
>> Subject: [ceph-users] concept of ceph and 2 datacenters
>> 
>> hi there,
>> i have a design/concept question, to see, whats outside and which kind
>> of redundancy you use.
>> 
>> actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby
>> clone.
>> but, rbd mirror is not application consistend. so we cannot be sure,
>> that all vms (kvm/proxy) are running.
>> we also waste a lot of hardware.
>> 
>> so now, we think about one big cluster over the two datacenters (two
>> buildings).
>> 
>> my queston is, do you care about ceph redundancy or is one ceph with
>> backups enough for you?
>> i know, with ceph, we are aware of hdd or server failure. but, are
>> software failures a real scenario?
>> 
>> would be great, to get some ideas from you.
>> also about the bandwidth between the 2 datacenters.
>> we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd).
>> 
>> thanks for help, my minds are rotating.
>> 
>> kind regards,
>> ronny
>> 
>> 
>> --
>> Ronny Lippold
>> System Administrator
>> 
>> --
>> Spark 5 GmbH
>> Rheinstr. 97
>> 64295 Darmstadt
>> Germany
>> --
>> Fon: +49-6151-8508-050
>> Fax: +49-6151-8508-111
>> Mail: ronny.lipp...@spark5.de
>> Web: https://www.spark5.de
>> --
>> Geschäftsführer: Henning Munte, Michael Mylius
>> Amtsgericht Darmstadt, HRB 7809
>> --
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: concept of ceph and 2 datacenters

[Peter Sabaini]

On 14.02.24 06:59, Vladimir Sigunov wrote:
> Hi Ronny,
> This is a good starting point for your design.
> https://docs.ceph.com/en/latest/rados/operations/stretch-mode/
> 
> My personal experience says that 2 DC Ceph deployment could suffer from a 
> 'split brain' situation. If you have any chance to create a 3 DC 
> configuration, I would suggest to consider it. It could be more expensive, 
> but it definitely will be more reliable and fault tolerant.

The docs you linked mention[0] a tiebreaker monitor (which could be a VM / in 
the cloud) -- have you used something like this?

[0] 
https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#limitations-of-stretch-mode

cheers,
peter.
 
> Sincerely,
> Vladimir
> 
> Get Outlook for Android
> 
> From: ronny.lipp...@spark5.de 
> Sent: Tuesday, February 13, 2024 6:50:50 AM
> To: ceph-users@ceph.io 
> Subject: [ceph-users] concept of ceph and 2 datacenters
> 
> hi there,
> i have a design/concept question, to see, whats outside and which kind
> of redundancy you use.
> 
> actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby
> clone.
> but, rbd mirror is not application consistend. so we cannot be sure,
> that all vms (kvm/proxy) are running.
> we also waste a lot of hardware.
> 
> so now, we think about one big cluster over the two datacenters (two
> buildings).
> 
> my queston is, do you care about ceph redundancy or is one ceph with
> backups enough for you?
> i know, with ceph, we are aware of hdd or server failure. but, are
> software failures a real scenario?
> 
> would be great, to get some ideas from you.
> also about the bandwidth between the 2 datacenters.
> we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd).
> 
> thanks for help, my minds are rotating.
> 
> kind regards,
> ronny
> 
> 
> --
> Ronny Lippold
> System Administrator
> 
> --
> Spark 5 GmbH
> Rheinstr. 97
> 64295 Darmstadt
> Germany
> --
> Fon: +49-6151-8508-050
> Fax: +49-6151-8508-111
> Mail: ronny.lipp...@spark5.de
> Web: https://www.spark5.de
> --
> Geschäftsführer: Henning Munte, Michael Mylius
> Amtsgericht Darmstadt, HRB 7809
> --
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io