[ceph-users] Re: [cephadm] mgr: no daemons active

[Satish Patel]

Now when I run "ceph orch ps" it works but the following command throws an
error.  Trying to bring up second mgr using ceph orch apply mgr command but
didn't help

root@ceph1:/ceph-disk# ceph version
ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus
(stable)

root@ceph1:/ceph-disk# ceph orch ls
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1212, in _handle_command
return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in
handle_command
return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 320, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 102, in

wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 91, in wrapper
return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/module.py", line 503, in
_list_services
raise_if_exception(completion)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 642, in
raise_if_exception
raise e
AssertionError: cephadm

On Fri, Sep 2, 2022 at 1:32 AM Satish Patel  wrote:

> nevermind, i found doc related that and i am able to get 1 mgr up -
> https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-a-mgr-daemon
>
>
> On Fri, Sep 2, 2022 at 1:21 AM Satish Patel  wrote:
>
>> Folks,
>>
>> I am having little fun time with cephadm and it's very annoying to deal
>> with it
>>
>> I have deployed a ceph cluster using cephadm on two nodes. Now when i was
>> trying to upgrade and noticed hiccups where it just upgraded a single mgr
>> with 16.2.10 but not other so i started messing around and somehow I
>> deleted both mgr in the thought that cephadm will recreate them.
>>
>> Now i don't have any single mgr so my ceph orch command hangs forever and
>> looks like a chicken egg issue.
>>
>> How do I recover from this? If I can't run the ceph orch command, I won't
>> be able to redeploy my mgr daemons.
>>
>> I am not able to find any mgr in the following command on both nodes.
>>
>> $ cephadm ls | grep mgr
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [cephadm] mgr: no daemons active

[Satish Patel]

nevermind, i found doc related that and i am able to get 1 mgr up -
https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-a-mgr-daemon


On Fri, Sep 2, 2022 at 1:21 AM Satish Patel  wrote:

> Folks,
>
> I am having little fun time with cephadm and it's very annoying to deal
> with it
>
> I have deployed a ceph cluster using cephadm on two nodes. Now when i was
> trying to upgrade and noticed hiccups where it just upgraded a single mgr
> with 16.2.10 but not other so i started messing around and somehow I
> deleted both mgr in the thought that cephadm will recreate them.
>
> Now i don't have any single mgr so my ceph orch command hangs forever and
> looks like a chicken egg issue.
>
> How do I recover from this? If I can't run the ceph orch command, I won't
> be able to redeploy my mgr daemons.
>
> I am not able to find any mgr in the following command on both nodes.
>
> $ cephadm ls | grep mgr
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: how to fix mds stuck at dispatched without restart ads

[Xiubo Li]

On 9/1/22 4:29 PM, zxcs wrote:

Thanks a lot, xiubo!!!


this time we still restarted mds fix this due to user urgent need list 
/path/to/A/, i will try to mds debug log if we hit it again.


Also, haven’t try flush mds journal before, any side effect to do 
this? This cephfs cluster is a production environment, we need very 
careful to do anything.


No side effect. The MDS will also do the journal flush sooner or later. 
But I am afraid this won't help for this issue, but still could have a try.





And I read this bug details https://tracker.ceph.com/issues/50840 
 , from pre-mail mentions it 
fixed in 15.2.17, we using ceps-deploy to deploy(upgrade) cephfs, 
seems it latest ceph version is 15.2.16? Will update if we can fixed 
after upgrade.


Will try to flush ads journal option when we hit this bug next time(if 
no user urgent need list directory). Seems it can 100% recurrent these 
days. Thanks All!




If possible please enable the 'debug_mds = 10' and 'debug_ms = 1'.

Thanks!




Thanks,
zx



2022年8月31日 15:23，Xiubo Li > 写道：



On 8/31/22 2:43 PM, zxcs wrote:

Hi, experts

we have a cephfs(15.2.13) cluster with kernel mount, and when we 
read from 2000+ processes to one ceph path(called /path/to/A/), then 
all of the process hung, and ls -lrth /path/to/A/ always stuck, but 
list other directory are health( /path/to/B/),


health detail always report mds has slow request.  And then we need 
to restart the mds fix this issue.


How can we fix this without restart mds(restart mds always impact 
other users)?


Any suggestions are welcome! Thanks a ton!

from this dump_ops_in_flight:

"description": "client_request(client.100807215:2856632 getattr 
AsLsXsFs #0x200978a3326 2022-08-31T09:36:30.444927+0800 
caller_id=2049, caller_gid=2049})",

"initiated_at": "2022-08-31T09:36:30.454570+0800",
"age": 17697.012491966001,
"duration": 17697.012805568,
"type_data": {
"flag_point": "dispatched",
"reqid": "client. 100807215:2856632",
"op_type": "client_request",
"client_info":
"client": "client.100807215",
"tid": 2856632
"events":
"time": "2022-08-31T09:36:30.454570+0800",
"event": "initiated"

"time": "2022-08-31T09:36:30.454572+0800",
"event": "throttled"

"time": "2022-08-31T09:36:30.454570+0800",
"event": "header read"

"time": "2022-08-31T09:36:30.454580+0800",
'event": "all_read"
"time": "2022-08-31T09:36:30.454604+0800",
"event": "dispatched"
}

AFAIK there is no easy way to do this. At least we need to know why 
it gets stuck and where. From your above and the previous mail 
thread, it should stuck in getattr request and sounds like a smiliar 
issue withhttps://tracker.ceph.com/issues/50840 
.


If it's not, it should be a new bug and could you create one tracker 
and provide the mds side debug logs.


Maybe you can try to flush the mds journal to see what will happen ?

- Xiubo




Thanks,
Xiong
___
ceph-users mailing list --ceph-users@ceph.io 
To unsubscribe send an email toceph-users-le...@ceph.io 




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [cephadm] mgr: no daemons active

[Satish Patel]

Folks,

I am having little fun time with cephadm and it's very annoying to deal
with it

I have deployed a ceph cluster using cephadm on two nodes. Now when i was
trying to upgrade and noticed hiccups where it just upgraded a single mgr
with 16.2.10 but not other so i started messing around and somehow I
deleted both mgr in the thought that cephadm will recreate them.

Now i don't have any single mgr so my ceph orch command hangs forever and
looks like a chicken egg issue.

How do I recover from this? If I can't run the ceph orch command, I won't
be able to redeploy my mgr daemons.

I am not able to find any mgr in the following command on both nodes.

$ cephadm ls | grep mgr
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Satish Patel]

Great, thanks!

Don't ask me how many commands I have typed to fix my issue. Finally I did
it. Basically i fix /etc/hosts and then i remove mgr service using
following command

ceph orch daemon rm mgr.ceph1.xmbvsb

And cephadm auto deployed a new working mgr.  I found ceph orch ps was
hanging and the solution I found was to restart all ceph daemon using
( systemctl restart ceph.target ) command.

root@ceph1:/ceph-disk# ceph orch ps
NAME HOST   PORTSSTATUS REFRESHED  AGE  MEM
USE  MEM LIM  VERSION  IMAGE ID  CONTAINER ID
alertmanager.ceph1   ceph1   running (12m) 9m ago   2w
 16.0M-  0.20.0   0881eb8f169f  d064a0177439
crash.ceph1  ceph1   running (49m) 9m ago   2w
 7963k-  15.2.17  93146564743f  550b088467e4
crash.ceph2  ceph2   running (35m) 9m ago  13d
 7287k-  15.2.17  93146564743f  c4b5b3327fa5
grafana.ceph1ceph1   running (14m) 9m ago   2w
 34.9M-  6.7.4557c83e11646  46048ebff031
mgr.ceph1.hxsfrs ceph1  *:8443,9283  running (13m) 9m ago  13m
327M-  15.2.17  93146564743f  4c5169890e9d
mgr.ceph2.hmbdla ceph2   running (35m) 9m ago  13d
435M-  16.2.10  0d668911f040  361d58a423cd
mon.ceph1ceph1   running (49m) 9m ago   2w
 85.5M2048M  15.2.17  93146564743f  a5f055953256
node-exporter.ceph1  ceph1   running (14m) 9m ago   2w
 32.9M-  0.18.1   e5a616e4b9cf  833cc2e6c9ed
node-exporter.ceph2  ceph2   running (13m) 9m ago  13d
 33.9M-  0.18.1   e5a616e4b9cf  30d15dde3860
osd.0ceph1   running (49m) 9m ago  13d
355M4096M  15.2.17  93146564743f  6e9bee5c211e
osd.1ceph1   running (49m) 9m ago  13d
372M4096M  15.2.17  93146564743f  09b8616bc096
osd.2ceph2   running (35m) 9m ago  13d
287M4096M  15.2.17  93146564743f  20f75a1b5221
osd.3ceph2   running (35m) 9m ago  13d
300M4096M  15.2.17  93146564743f  c57154355b03
prometheus.ceph1 ceph1   running (12m) 9m ago   2w
 89.5M-  2.18.1   de242295e225  b5ff35307ac0

Now I am going to start an upgrade process next. I will keep you posted to
see how it goes.

On Thu, Sep 1, 2022 at 10:06 PM Adam King  wrote:

> I'm not sure exactly what needs to be done to fix that, but I'd imagine
> just editing the /etc/hosts file on all your hosts to be correct would be
> the start (the cephadm shell would have taken its /etc/hosts off of
> whatever host you ran the shell from). Unfortunately I'm not much of a
> networking expert and if you have some sort of DNS stuff going on for your
> local network I'm not too sure what to do there, but if it's possible just
> fixing the /etc/hosts entries will resolve things. Either way, once you've
> got the networking fixed so ssh-ing to the hosts works as expected with the
> IPs you might need to  re-add one or both of the hosts to the cluster with
> the correct IP as well ( "ceph orch host add  "). I believe
> if you just run the orch host add command again with a different IP but the
> same hostname it will just change the IP cephadm has stored for the host.
> If that isn't working, running "ceph orch host rm  --force"
> beforehand should make it work (if you just remove the host with --force it
> shouldn't touch the host's daemons and should therefore be a relatively
> sage operation). In the end, the IP cephadm lists for each host in "ceph
> orch host ls" must be an IP that allows correctly ssh-ing to the host.
>
> On Thu, Sep 1, 2022 at 9:17 PM Satish Patel  wrote:
>
>> Hi Adam,
>>
>> You are correct, look like it was a naming issue in my /etc/hosts file.
>> Is there a way to correct it?
>>
>> If you see i have ceph1 two time. :(
>>
>> 10.73.0.191 ceph1.example.com ceph1
>> 10.73.0.192 ceph2.example.com ceph1
>>
>> On Thu, Sep 1, 2022 at 8:06 PM Adam King  wrote:
>>
>>> the naming for daemons is a bit different for each daemon type, but for
>>> mgr daemons it's always "mgr..".  The daemons
>>> cephadm will be able to find for something like a daemon redeploy are
>>> pretty much always whatever is reported in "ceph orch ps". Given that
>>> "mgr.ceph1.xmbvsb" isn't listed there, it's not surprising it said it
>>> couldn't find it.
>>>
>>> There is definitely something very odd going on here. It looks like the
>>> crash daemons as well are reporting a duplicate "crash.ceph2" on both ceph1
>>> and ceph2. Going back to your original orch ps output from the first email,
>>> it seems that every daemon seems to have a duplicate and none of the actual
>>> daemons listed in the "cephadm ls" on ceph1 are actually being reported in
>>> the orch ps output. I think something may have gone wrong with the host and
>>> networking setup here and it seems to be reporting ceph2 daemons as the
>>> daemons for both 

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Adam King]

I'm not sure exactly what needs to be done to fix that, but I'd imagine
just editing the /etc/hosts file on all your hosts to be correct would be
the start (the cephadm shell would have taken its /etc/hosts off of
whatever host you ran the shell from). Unfortunately I'm not much of a
networking expert and if you have some sort of DNS stuff going on for your
local network I'm not too sure what to do there, but if it's possible just
fixing the /etc/hosts entries will resolve things. Either way, once you've
got the networking fixed so ssh-ing to the hosts works as expected with the
IPs you might need to  re-add one or both of the hosts to the cluster with
the correct IP as well ( "ceph orch host add  "). I believe
if you just run the orch host add command again with a different IP but the
same hostname it will just change the IP cephadm has stored for the host.
If that isn't working, running "ceph orch host rm  --force"
beforehand should make it work (if you just remove the host with --force it
shouldn't touch the host's daemons and should therefore be a relatively
sage operation). In the end, the IP cephadm lists for each host in "ceph
orch host ls" must be an IP that allows correctly ssh-ing to the host.

On Thu, Sep 1, 2022 at 9:17 PM Satish Patel  wrote:

> Hi Adam,
>
> You are correct, look like it was a naming issue in my /etc/hosts file. Is
> there a way to correct it?
>
> If you see i have ceph1 two time. :(
>
> 10.73.0.191 ceph1.example.com ceph1
> 10.73.0.192 ceph2.example.com ceph1
>
> On Thu, Sep 1, 2022 at 8:06 PM Adam King  wrote:
>
>> the naming for daemons is a bit different for each daemon type, but for
>> mgr daemons it's always "mgr..".  The daemons
>> cephadm will be able to find for something like a daemon redeploy are
>> pretty much always whatever is reported in "ceph orch ps". Given that
>> "mgr.ceph1.xmbvsb" isn't listed there, it's not surprising it said it
>> couldn't find it.
>>
>> There is definitely something very odd going on here. It looks like the
>> crash daemons as well are reporting a duplicate "crash.ceph2" on both ceph1
>> and ceph2. Going back to your original orch ps output from the first email,
>> it seems that every daemon seems to have a duplicate and none of the actual
>> daemons listed in the "cephadm ls" on ceph1 are actually being reported in
>> the orch ps output. I think something may have gone wrong with the host and
>> networking setup here and it seems to be reporting ceph2 daemons as the
>> daemons for both ceph1 and ceph2 as if trying to connect to ceph1 ends up
>> connecting to ceph2. The only time I've seen anything like this was when I
>> made a mistake and setup a virtual IP on one host that was the same as the
>> actual IP for another host on the cluster and cephadm basically ended up
>> ssh-ing to the same host via both IPs (the one that was supposed to be for
>> host A and host B where the virtual IP matching host B was setup on host
>> A). I doubt you're in that exact situation, but I think we need to look
>> very closely at the networking setup here. I would try opening up a cephadm
>> shell and ssh-ing to each of the two hosts by the IP listed in "ceph orch
>> host ls" and make sure you actually get to the correct host and it has the
>> correct hostname. Given the output, I wouldn't be surprised if trying to
>> connect to ceph1's IP landed you on ceph2 or vice versa. I will say I found
>> it a bit odd originally when I saw the two IPs were 10.73.0.192 and
>> 10.73.3.192. There's nothing necessarily wrong with that, but typically IPs
>> on the host are more likely to differ at the end than in the middle (e.g.
>> 192.168.122.1 and 192.168.122.2 rather than 192.168.1.122 and
>> 192.168.2.122) and it did make me wonder if a mistake had occurred in the
>> networking. Either way, there's clearly something making it think ceph2's
>> daemons are on both ceph1 and ceph2 and some sort of networking issue is
>> the only thing I'm aware of currently that causes something like that.
>>
>> On Thu, Sep 1, 2022 at 6:30 PM Satish Patel  wrote:
>>
>>> Hi Adam,
>>>
>>> I have also noticed a very strange thing which is Duplicate name in the
>>> following output.  Is this normal?  I don't know how it got here. Is there
>>> a way I can rename them?
>>>
>>> root@ceph1:~# ceph orch ps
>>> NAME HOST   PORTSSTATUS  REFRESHED  AGE
>>>  MEM USE  MEM LIM  VERSIONIMAGE ID  CONTAINER ID
>>> alertmanager.ceph1   ceph1  *:9093,9094  starting--
>>>-- 
>>> crash.ceph2  ceph1   running (13d) 10s ago  13d
>>>10.0M-  15.2.1793146564743f  0a009254afb0
>>> crash.ceph2  ceph2   running (13d) 10s ago  13d
>>>10.0M-  15.2.1793146564743f  0a009254afb0
>>> grafana.ceph1ceph1  *:3000   starting--
>>>-- 
>>> mgr.ceph2.hmbdla ceph1   running (103m)10s ago  

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Satish Patel]

Hi Adam,

You are correct, look like it was a naming issue in my /etc/hosts file. Is
there a way to correct it?

If you see i have ceph1 two time. :(

10.73.0.191 ceph1.example.com ceph1
10.73.0.192 ceph2.example.com ceph1

On Thu, Sep 1, 2022 at 8:06 PM Adam King  wrote:

> the naming for daemons is a bit different for each daemon type, but for
> mgr daemons it's always "mgr..".  The daemons
> cephadm will be able to find for something like a daemon redeploy are
> pretty much always whatever is reported in "ceph orch ps". Given that
> "mgr.ceph1.xmbvsb" isn't listed there, it's not surprising it said it
> couldn't find it.
>
> There is definitely something very odd going on here. It looks like the
> crash daemons as well are reporting a duplicate "crash.ceph2" on both ceph1
> and ceph2. Going back to your original orch ps output from the first email,
> it seems that every daemon seems to have a duplicate and none of the actual
> daemons listed in the "cephadm ls" on ceph1 are actually being reported in
> the orch ps output. I think something may have gone wrong with the host and
> networking setup here and it seems to be reporting ceph2 daemons as the
> daemons for both ceph1 and ceph2 as if trying to connect to ceph1 ends up
> connecting to ceph2. The only time I've seen anything like this was when I
> made a mistake and setup a virtual IP on one host that was the same as the
> actual IP for another host on the cluster and cephadm basically ended up
> ssh-ing to the same host via both IPs (the one that was supposed to be for
> host A and host B where the virtual IP matching host B was setup on host
> A). I doubt you're in that exact situation, but I think we need to look
> very closely at the networking setup here. I would try opening up a cephadm
> shell and ssh-ing to each of the two hosts by the IP listed in "ceph orch
> host ls" and make sure you actually get to the correct host and it has the
> correct hostname. Given the output, I wouldn't be surprised if trying to
> connect to ceph1's IP landed you on ceph2 or vice versa. I will say I found
> it a bit odd originally when I saw the two IPs were 10.73.0.192 and
> 10.73.3.192. There's nothing necessarily wrong with that, but typically IPs
> on the host are more likely to differ at the end than in the middle (e.g.
> 192.168.122.1 and 192.168.122.2 rather than 192.168.1.122 and
> 192.168.2.122) and it did make me wonder if a mistake had occurred in the
> networking. Either way, there's clearly something making it think ceph2's
> daemons are on both ceph1 and ceph2 and some sort of networking issue is
> the only thing I'm aware of currently that causes something like that.
>
> On Thu, Sep 1, 2022 at 6:30 PM Satish Patel  wrote:
>
>> Hi Adam,
>>
>> I have also noticed a very strange thing which is Duplicate name in the
>> following output.  Is this normal?  I don't know how it got here. Is there
>> a way I can rename them?
>>
>> root@ceph1:~# ceph orch ps
>> NAME HOST   PORTSSTATUS  REFRESHED  AGE
>>  MEM USE  MEM LIM  VERSIONIMAGE ID  CONTAINER ID
>> alertmanager.ceph1   ceph1  *:9093,9094  starting--
>>  -- 
>> crash.ceph2  ceph1   running (13d) 10s ago  13d
>>  10.0M-  15.2.1793146564743f  0a009254afb0
>> crash.ceph2  ceph2   running (13d) 10s ago  13d
>>  10.0M-  15.2.1793146564743f  0a009254afb0
>> grafana.ceph1ceph1  *:3000   starting--
>>  -- 
>> mgr.ceph2.hmbdla ceph1   running (103m)10s ago  13d
>>   518M-  16.2.100d668911f040  745245c18d5e
>> mgr.ceph2.hmbdla ceph2   running (103m)10s ago  13d
>>   518M-  16.2.100d668911f040  745245c18d5e
>> node-exporter.ceph2  ceph1   running (7h)  10s ago  13d
>>  70.2M-  0.18.1 e5a616e4b9cf  d0ba04bb977c
>> node-exporter.ceph2  ceph2   running (7h)  10s ago  13d
>>  70.2M-  0.18.1 e5a616e4b9cf  d0ba04bb977c
>> osd.2ceph1   running (19h) 10s ago  13d
>>   901M4096M  15.2.1793146564743f  e286fb1c6302
>> osd.2ceph2   running (19h) 10s ago  13d
>>   901M4096M  15.2.1793146564743f  e286fb1c6302
>> osd.3ceph1   running (19h) 10s ago  13d
>>  1006M4096M  15.2.1793146564743f  d3ae5d9f694f
>> osd.3ceph2   running (19h) 10s ago  13d
>>  1006M4096M  15.2.1793146564743f  d3ae5d9f694f
>> osd.5ceph1   running (19h) 10s ago   9d
>>   222M4096M  15.2.1793146564743f  405068fb474e
>> osd.5ceph2   running (19h) 10s ago   9d
>>   222M4096M  15.2.1793146564743f  405068fb474e
>> prometheus.ceph1 ceph1  *:9095   running (15s) 10s ago  15s
>>  30.6M-

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Adam King]

the naming for daemons is a bit different for each daemon type, but for mgr
daemons it's always "mgr..".  The daemons cephadm
will be able to find for something like a daemon redeploy are pretty much
always whatever is reported in "ceph orch ps". Given that
"mgr.ceph1.xmbvsb" isn't listed there, it's not surprising it said it
couldn't find it.

There is definitely something very odd going on here. It looks like the
crash daemons as well are reporting a duplicate "crash.ceph2" on both ceph1
and ceph2. Going back to your original orch ps output from the first email,
it seems that every daemon seems to have a duplicate and none of the actual
daemons listed in the "cephadm ls" on ceph1 are actually being reported in
the orch ps output. I think something may have gone wrong with the host and
networking setup here and it seems to be reporting ceph2 daemons as the
daemons for both ceph1 and ceph2 as if trying to connect to ceph1 ends up
connecting to ceph2. The only time I've seen anything like this was when I
made a mistake and setup a virtual IP on one host that was the same as the
actual IP for another host on the cluster and cephadm basically ended up
ssh-ing to the same host via both IPs (the one that was supposed to be for
host A and host B where the virtual IP matching host B was setup on host
A). I doubt you're in that exact situation, but I think we need to look
very closely at the networking setup here. I would try opening up a cephadm
shell and ssh-ing to each of the two hosts by the IP listed in "ceph orch
host ls" and make sure you actually get to the correct host and it has the
correct hostname. Given the output, I wouldn't be surprised if trying to
connect to ceph1's IP landed you on ceph2 or vice versa. I will say I found
it a bit odd originally when I saw the two IPs were 10.73.0.192 and
10.73.3.192. There's nothing necessarily wrong with that, but typically IPs
on the host are more likely to differ at the end than in the middle (e.g.
192.168.122.1 and 192.168.122.2 rather than 192.168.1.122 and
192.168.2.122) and it did make me wonder if a mistake had occurred in the
networking. Either way, there's clearly something making it think ceph2's
daemons are on both ceph1 and ceph2 and some sort of networking issue is
the only thing I'm aware of currently that causes something like that.

On Thu, Sep 1, 2022 at 6:30 PM Satish Patel  wrote:

> Hi Adam,
>
> I have also noticed a very strange thing which is Duplicate name in the
> following output.  Is this normal?  I don't know how it got here. Is there
> a way I can rename them?
>
> root@ceph1:~# ceph orch ps
> NAME HOST   PORTSSTATUS  REFRESHED  AGE
>  MEM USE  MEM LIM  VERSIONIMAGE ID  CONTAINER ID
> alertmanager.ceph1   ceph1  *:9093,9094  starting--
>  -- 
> crash.ceph2  ceph1   running (13d) 10s ago  13d
>  10.0M-  15.2.1793146564743f  0a009254afb0
> crash.ceph2  ceph2   running (13d) 10s ago  13d
>  10.0M-  15.2.1793146564743f  0a009254afb0
> grafana.ceph1ceph1  *:3000   starting--
>  -- 
> mgr.ceph2.hmbdla ceph1   running (103m)10s ago  13d
>   518M-  16.2.100d668911f040  745245c18d5e
> mgr.ceph2.hmbdla ceph2   running (103m)10s ago  13d
>   518M-  16.2.100d668911f040  745245c18d5e
> node-exporter.ceph2  ceph1   running (7h)  10s ago  13d
>  70.2M-  0.18.1 e5a616e4b9cf  d0ba04bb977c
> node-exporter.ceph2  ceph2   running (7h)  10s ago  13d
>  70.2M-  0.18.1 e5a616e4b9cf  d0ba04bb977c
> osd.2ceph1   running (19h) 10s ago  13d
>   901M4096M  15.2.1793146564743f  e286fb1c6302
> osd.2ceph2   running (19h) 10s ago  13d
>   901M4096M  15.2.1793146564743f  e286fb1c6302
> osd.3ceph1   running (19h) 10s ago  13d
>  1006M4096M  15.2.1793146564743f  d3ae5d9f694f
> osd.3ceph2   running (19h) 10s ago  13d
>  1006M4096M  15.2.1793146564743f  d3ae5d9f694f
> osd.5ceph1   running (19h) 10s ago   9d
>   222M4096M  15.2.1793146564743f  405068fb474e
> osd.5ceph2   running (19h) 10s ago   9d
>   222M4096M  15.2.1793146564743f  405068fb474e
> prometheus.ceph1 ceph1  *:9095   running (15s) 10s ago  15s
>  30.6M- 514e6a882f6e  65a0acfed605
> prometheus.ceph1 ceph2  *:9095   running (15s) 10s ago  15s
>  30.6M- 514e6a882f6e  65a0acfed605
>
> I found the following example link which has all different names, how does
> cephadm decide naming?
>
>
> https://achchusnulchikam.medium.com/deploy-ceph-cluster-with-cephadm-on-centos-8-257b300e7b42
>
> On Thu, Sep 

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Satish Patel]

Hi Adam,

I have also noticed a very strange thing which is Duplicate name in the
following output.  Is this normal?  I don't know how it got here. Is there
a way I can rename them?

root@ceph1:~# ceph orch ps
NAME HOST   PORTSSTATUS  REFRESHED  AGE
 MEM USE  MEM LIM  VERSIONIMAGE ID  CONTAINER ID
alertmanager.ceph1   ceph1  *:9093,9094  starting--
   -- 
crash.ceph2  ceph1   running (13d) 10s ago  13d
 10.0M-  15.2.1793146564743f  0a009254afb0
crash.ceph2  ceph2   running (13d) 10s ago  13d
 10.0M-  15.2.1793146564743f  0a009254afb0
grafana.ceph1ceph1  *:3000   starting--
   -- 
mgr.ceph2.hmbdla ceph1   running (103m)10s ago  13d
518M-  16.2.100d668911f040  745245c18d5e
mgr.ceph2.hmbdla ceph2   running (103m)10s ago  13d
518M-  16.2.100d668911f040  745245c18d5e
node-exporter.ceph2  ceph1   running (7h)  10s ago  13d
 70.2M-  0.18.1 e5a616e4b9cf  d0ba04bb977c
node-exporter.ceph2  ceph2   running (7h)  10s ago  13d
 70.2M-  0.18.1 e5a616e4b9cf  d0ba04bb977c
osd.2ceph1   running (19h) 10s ago  13d
901M4096M  15.2.1793146564743f  e286fb1c6302
osd.2ceph2   running (19h) 10s ago  13d
901M4096M  15.2.1793146564743f  e286fb1c6302
osd.3ceph1   running (19h) 10s ago  13d
 1006M4096M  15.2.1793146564743f  d3ae5d9f694f
osd.3ceph2   running (19h) 10s ago  13d
 1006M4096M  15.2.1793146564743f  d3ae5d9f694f
osd.5ceph1   running (19h) 10s ago   9d
222M4096M  15.2.1793146564743f  405068fb474e
osd.5ceph2   running (19h) 10s ago   9d
222M4096M  15.2.1793146564743f  405068fb474e
prometheus.ceph1 ceph1  *:9095   running (15s) 10s ago  15s
 30.6M- 514e6a882f6e  65a0acfed605
prometheus.ceph1 ceph2  *:9095   running (15s) 10s ago  15s
 30.6M- 514e6a882f6e  65a0acfed605

I found the following example link which has all different names, how does
cephadm decide naming?

https://achchusnulchikam.medium.com/deploy-ceph-cluster-with-cephadm-on-centos-8-257b300e7b42

On Thu, Sep 1, 2022 at 6:20 PM Satish Patel  wrote:

> Hi Adam,
>
> Getting the following error, not sure why it's not able to find it.
>
> root@ceph1:~# ceph orch daemon redeploy mgr.ceph1.xmbvsb
> Error EINVAL: Unable to find mgr.ceph1.xmbvsb daemon(s)
>
> On Thu, Sep 1, 2022 at 5:57 PM Adam King  wrote:
>
>> what happens if you run `ceph orch daemon redeploy mgr.ceph1.xmbvsb`?
>>
>> On Thu, Sep 1, 2022 at 5:12 PM Satish Patel  wrote:
>>
>>> Hi Adam,
>>>
>>> Here is requested output
>>>
>>> root@ceph1:~# ceph health detail
>>> HEALTH_WARN 4 stray daemon(s) not managed by cephadm
>>> [WRN] CEPHADM_STRAY_DAEMON: 4 stray daemon(s) not managed by cephadm
>>> stray daemon mon.ceph1 on host ceph1 not managed by cephadm
>>> stray daemon osd.0 on host ceph1 not managed by cephadm
>>> stray daemon osd.1 on host ceph1 not managed by cephadm
>>> stray daemon osd.4 on host ceph1 not managed by cephadm
>>>
>>>
>>> root@ceph1:~# ceph orch host ls
>>> HOST   ADDR LABELS  STATUS
>>> ceph1  10.73.0.192
>>> ceph2  10.73.3.192  _admin
>>> 2 hosts in cluster
>>>
>>>
>>> My cephadm ls  saying mgr is in error state
>>>
>>> {
>>> "style": "cephadm:v1",
>>> "name": "mgr.ceph1.xmbvsb",
>>> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
>>> "systemd_unit":
>>> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb",
>>> "enabled": true,
>>> "state": "error",
>>> "container_id": null,
>>> "container_image_name": "quay.io/ceph/ceph:v15",
>>> "container_image_id": null,
>>> "version": null,
>>> "started": null,
>>> "created": "2022-09-01T20:59:49.314347Z",
>>> "deployed": "2022-09-01T20:59:48.718347Z",
>>> "configured": "2022-09-01T20:59:49.314347Z"
>>> },
>>>
>>>
>>> Getting error
>>>
>>> root@ceph1:~# cephadm unit --fsid f270ad9e-1f6f-11ed-b6f8-a539d87379ea
>>> --name mgr.ceph1.xmbvsb start
>>> stderr Job for
>>> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service
>>> failed because the control process exited with error code.
>>> stderr See "systemctl status
>>> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service" and
>>> "journalctl -xe" for details.
>>> Traceback (most recent call last):
>>>   File "/usr/sbin/cephadm", line 6250, in 
>>> r = args.func()
>>>   File "/usr/sbin/cephadm", line 1357, in _infer_fsid
>>> return func()
>>>   File "/usr/sbin/cephadm", line 3727, in command_unit
>>> 

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Satish Patel]

Hi Adam,

Getting the following error, not sure why it's not able to find it.

root@ceph1:~# ceph orch daemon redeploy mgr.ceph1.xmbvsb
Error EINVAL: Unable to find mgr.ceph1.xmbvsb daemon(s)

On Thu, Sep 1, 2022 at 5:57 PM Adam King  wrote:

> what happens if you run `ceph orch daemon redeploy mgr.ceph1.xmbvsb`?
>
> On Thu, Sep 1, 2022 at 5:12 PM Satish Patel  wrote:
>
>> Hi Adam,
>>
>> Here is requested output
>>
>> root@ceph1:~# ceph health detail
>> HEALTH_WARN 4 stray daemon(s) not managed by cephadm
>> [WRN] CEPHADM_STRAY_DAEMON: 4 stray daemon(s) not managed by cephadm
>> stray daemon mon.ceph1 on host ceph1 not managed by cephadm
>> stray daemon osd.0 on host ceph1 not managed by cephadm
>> stray daemon osd.1 on host ceph1 not managed by cephadm
>> stray daemon osd.4 on host ceph1 not managed by cephadm
>>
>>
>> root@ceph1:~# ceph orch host ls
>> HOST   ADDR LABELS  STATUS
>> ceph1  10.73.0.192
>> ceph2  10.73.3.192  _admin
>> 2 hosts in cluster
>>
>>
>> My cephadm ls  saying mgr is in error state
>>
>> {
>> "style": "cephadm:v1",
>> "name": "mgr.ceph1.xmbvsb",
>> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
>> "systemd_unit":
>> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb",
>> "enabled": true,
>> "state": "error",
>> "container_id": null,
>> "container_image_name": "quay.io/ceph/ceph:v15",
>> "container_image_id": null,
>> "version": null,
>> "started": null,
>> "created": "2022-09-01T20:59:49.314347Z",
>> "deployed": "2022-09-01T20:59:48.718347Z",
>> "configured": "2022-09-01T20:59:49.314347Z"
>> },
>>
>>
>> Getting error
>>
>> root@ceph1:~# cephadm unit --fsid f270ad9e-1f6f-11ed-b6f8-a539d87379ea
>> --name mgr.ceph1.xmbvsb start
>> stderr Job for
>> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service
>> failed because the control process exited with error code.
>> stderr See "systemctl status
>> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service" and
>> "journalctl -xe" for details.
>> Traceback (most recent call last):
>>   File "/usr/sbin/cephadm", line 6250, in 
>> r = args.func()
>>   File "/usr/sbin/cephadm", line 1357, in _infer_fsid
>> return func()
>>   File "/usr/sbin/cephadm", line 3727, in command_unit
>> call_throws([
>>   File "/usr/sbin/cephadm", line 1119, in call_throws
>> raise RuntimeError('Failed command: %s' % ' '.join(command))
>> RuntimeError: Failed command: systemctl start
>> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb
>>
>>
>> How do I remove and re-deploy mgr?
>>
>> On Thu, Sep 1, 2022 at 4:54 PM Adam King  wrote:
>>
>>> cephadm deploys the containers with --rm so they will get removed if you
>>> stop them. As for getting the 2nd mgr back, if it still lists the 2nd one
>>> in `ceph orch ps` you should be able to do a `ceph orch daemon redeploy
>>> ` where  should match the name given in
>>> the orch ps output for the one that isn't actually up. If it isn't listed
>>> there, given you have a count of 2, cephadm should deploy another one. I do
>>> see in the orch ls output you posted that it says the mgr service has "2/2"
>>> running which implies it believes a 2nd mgr is present (and you would
>>> therefore be able to try the daemon redeploy if that daemon isn't actually
>>> there).
>>>
>>> Is it still reporting the duplicate osds in orch ps? I see in the
>>> cephadm ls output on ceph1 that osd.2 isn't being reported, which was
>>> reported as being on ceph1 in the orch ps output in your original message
>>> in this thread. I'm interested in what `ceph health detail` is reporting
>>> now as well, as it says there are 4 stray daemons. Also, the `ceph orch
>>> host ls` output just to get a better grasp of the topology of this cluster.
>>>
>>> On Thu, Sep 1, 2022 at 3:50 PM Satish Patel 
>>> wrote:
>>>
 Adam,

 I have posted a question related to upgrading earlier and this thread
 is related to that, I have opened a new one because I found that error in
 logs and thought the upgrade may be stuck because of duplicate OSDs.

 root@ceph1:~# ls -l /var/lib/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/
 total 44
 drwx-- 3 nobody nogroup 4096 Aug 19 05:37 alertmanager.ceph1
 drwx-- 3167 167 4096 Aug 19 05:36 crash
 drwx-- 2167 167 4096 Aug 19 05:37 crash.ceph1
 drwx-- 4998 996 4096 Aug 19 05:37 grafana.ceph1
 drwx-- 2167 167 4096 Aug 19 05:36 mgr.ceph1.xmbvsb
 drwx-- 3167 167 4096 Aug 19 05:36 mon.ceph1
 drwx-- 2 nobody nogroup 4096 Aug 19 05:37 node-exporter.ceph1
 drwx-- 2167 167 4096 Aug 19 17:55 osd.0
 drwx-- 2167 167 4096 Aug 19 18:03 osd.1
 drwx-- 2167 167 4096 Aug 31 05:20 osd.4
 drwx-- 4 nobody nogroup 4096 Aug 19 05:38 prometheus.ceph1

 Here is the output of cephadm 

[ceph-users] Re: Remove corrupt PG

[Jesper Lykkegaard Karlsen]

Well not the total solution after all.
There is still some metadata and header structure left that I still cannot 
delete with ceph-objectstore-tool —op remove. 
It makes a core dump. 

I think I need to declare the OSD lost anyway to the through this. 
Unless somebody have a better suggestion?

Best, 
Jesper
--
Jesper Lykkegaard Karlsen
Scientific Computing
Centre for Structural Biology
Department of Molecular Biology and Genetics
Aarhus University
Universitetsbyen 81
8000 Aarhus C

E-mail: je...@mbg.au.dk
Tlf:+45 50906203

> On 1 Sep 2022, at 22.01, Jesper Lykkegaard Karlsen  wrote:
> 
> To answer my own question. 
> 
> The removal of the  corrupt PG, could be fixed by doing ceph-objectstore-tool 
> fuse mount-thingy. 
> Then from the mount point, delete everything in the PGs head directory. 
> 
> This took only a few seconds (compared to 7.5 days) and after unmount and 
> restart of the OSD it came back online. 
> 
> Best, 
> Jesper
> 
> --
> Jesper Lykkegaard Karlsen
> Scientific Computing
> Centre for Structural Biology
> Department of Molecular Biology and Genetics
> Aarhus University
> Universitetsbyen 81
> 8000 Aarhus C
> 
> E-mail: je...@mbg.au.dk
> Tlf:+45 50906203
> 
>> On 31 Aug 2022, at 20.53, Jesper Lykkegaard Karlsen  wrote:
>> 
>> Hi all, 
>> 
>> I wanted to move a PG to an empty OSD, so I could do repairs on it without 
>> the whole OSD, which is full of other PG’s, would be effected with extensive 
>> downtime. 
>> 
>> Thus, I exported the PG with ceph-objectstore-tool, an after successful 
>> export I removed it. Unfortunately, the remove command was interrupted 
>> midway. 
>> This resulted in a PG that could not be remove with “ceph-objectstore-tool 
>> —op remove ….”, since the header is gone. 
>> Worse is that the OSD does not boot, due to it can see objects from the 
>> removed PG, but cannot access them. 
>> 
>> I have tried to remove the individual objects in that PG (also with 
>> objectstore-tool), but this process is extremely slow. 
>> When looping over the >65,000 object, each remove takes ~10 sec and is very 
>> compute intensive, which is approximately 7.5 days. 
>> 
>> Is the a faster way to get around this? 
>> 
>> Mvh. Jesper
>> 
>> --
>> Jesper Lykkegaard Karlsen
>> Scientific Computing
>> Centre for Structural Biology
>> Department of Molecular Biology and Genetics
>> Aarhus University
>> Universitetsbyen 81
>> 8000 Aarhus C
>> 
>> E-mail: je...@mbg.au.dk
>> Tlf:+45 50906203
>> 
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Adam King]

what happens if you run `ceph orch daemon redeploy mgr.ceph1.xmbvsb`?

On Thu, Sep 1, 2022 at 5:12 PM Satish Patel  wrote:

> Hi Adam,
>
> Here is requested output
>
> root@ceph1:~# ceph health detail
> HEALTH_WARN 4 stray daemon(s) not managed by cephadm
> [WRN] CEPHADM_STRAY_DAEMON: 4 stray daemon(s) not managed by cephadm
> stray daemon mon.ceph1 on host ceph1 not managed by cephadm
> stray daemon osd.0 on host ceph1 not managed by cephadm
> stray daemon osd.1 on host ceph1 not managed by cephadm
> stray daemon osd.4 on host ceph1 not managed by cephadm
>
>
> root@ceph1:~# ceph orch host ls
> HOST   ADDR LABELS  STATUS
> ceph1  10.73.0.192
> ceph2  10.73.3.192  _admin
> 2 hosts in cluster
>
>
> My cephadm ls  saying mgr is in error state
>
> {
> "style": "cephadm:v1",
> "name": "mgr.ceph1.xmbvsb",
> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
> "systemd_unit":
> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb",
> "enabled": true,
> "state": "error",
> "container_id": null,
> "container_image_name": "quay.io/ceph/ceph:v15",
> "container_image_id": null,
> "version": null,
> "started": null,
> "created": "2022-09-01T20:59:49.314347Z",
> "deployed": "2022-09-01T20:59:48.718347Z",
> "configured": "2022-09-01T20:59:49.314347Z"
> },
>
>
> Getting error
>
> root@ceph1:~# cephadm unit --fsid f270ad9e-1f6f-11ed-b6f8-a539d87379ea
> --name mgr.ceph1.xmbvsb start
> stderr Job for
> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service failed
> because the control process exited with error code.
> stderr See "systemctl status
> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service" and
> "journalctl -xe" for details.
> Traceback (most recent call last):
>   File "/usr/sbin/cephadm", line 6250, in 
> r = args.func()
>   File "/usr/sbin/cephadm", line 1357, in _infer_fsid
> return func()
>   File "/usr/sbin/cephadm", line 3727, in command_unit
> call_throws([
>   File "/usr/sbin/cephadm", line 1119, in call_throws
> raise RuntimeError('Failed command: %s' % ' '.join(command))
> RuntimeError: Failed command: systemctl start
> ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb
>
>
> How do I remove and re-deploy mgr?
>
> On Thu, Sep 1, 2022 at 4:54 PM Adam King  wrote:
>
>> cephadm deploys the containers with --rm so they will get removed if you
>> stop them. As for getting the 2nd mgr back, if it still lists the 2nd one
>> in `ceph orch ps` you should be able to do a `ceph orch daemon redeploy
>> ` where  should match the name given in
>> the orch ps output for the one that isn't actually up. If it isn't listed
>> there, given you have a count of 2, cephadm should deploy another one. I do
>> see in the orch ls output you posted that it says the mgr service has "2/2"
>> running which implies it believes a 2nd mgr is present (and you would
>> therefore be able to try the daemon redeploy if that daemon isn't actually
>> there).
>>
>> Is it still reporting the duplicate osds in orch ps? I see in the cephadm
>> ls output on ceph1 that osd.2 isn't being reported, which was reported as
>> being on ceph1 in the orch ps output in your original message in this
>> thread. I'm interested in what `ceph health detail` is reporting now as
>> well, as it says there are 4 stray daemons. Also, the `ceph orch host ls`
>> output just to get a better grasp of the topology of this cluster.
>>
>> On Thu, Sep 1, 2022 at 3:50 PM Satish Patel  wrote:
>>
>>> Adam,
>>>
>>> I have posted a question related to upgrading earlier and this thread is
>>> related to that, I have opened a new one because I found that error in logs
>>> and thought the upgrade may be stuck because of duplicate OSDs.
>>>
>>> root@ceph1:~# ls -l /var/lib/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/
>>> total 44
>>> drwx-- 3 nobody nogroup 4096 Aug 19 05:37 alertmanager.ceph1
>>> drwx-- 3167 167 4096 Aug 19 05:36 crash
>>> drwx-- 2167 167 4096 Aug 19 05:37 crash.ceph1
>>> drwx-- 4998 996 4096 Aug 19 05:37 grafana.ceph1
>>> drwx-- 2167 167 4096 Aug 19 05:36 mgr.ceph1.xmbvsb
>>> drwx-- 3167 167 4096 Aug 19 05:36 mon.ceph1
>>> drwx-- 2 nobody nogroup 4096 Aug 19 05:37 node-exporter.ceph1
>>> drwx-- 2167 167 4096 Aug 19 17:55 osd.0
>>> drwx-- 2167 167 4096 Aug 19 18:03 osd.1
>>> drwx-- 2167 167 4096 Aug 31 05:20 osd.4
>>> drwx-- 4 nobody nogroup 4096 Aug 19 05:38 prometheus.ceph1
>>>
>>> Here is the output of cephadm ls
>>>
>>> root@ceph1:~# cephadm ls
>>> [
>>> {
>>> "style": "cephadm:v1",
>>> "name": "alertmanager.ceph1",
>>> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
>>> "systemd_unit":
>>> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@alertmanager.ceph1",
>>> "enabled": true,
>>> "state": "running",
>>> 

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Satish Patel]

Hi Adam,

Here is requested output

root@ceph1:~# ceph health detail
HEALTH_WARN 4 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 4 stray daemon(s) not managed by cephadm
stray daemon mon.ceph1 on host ceph1 not managed by cephadm
stray daemon osd.0 on host ceph1 not managed by cephadm
stray daemon osd.1 on host ceph1 not managed by cephadm
stray daemon osd.4 on host ceph1 not managed by cephadm


root@ceph1:~# ceph orch host ls
HOST   ADDR LABELS  STATUS
ceph1  10.73.0.192
ceph2  10.73.3.192  _admin
2 hosts in cluster


My cephadm ls  saying mgr is in error state

{
"style": "cephadm:v1",
"name": "mgr.ceph1.xmbvsb",
"fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
"systemd_unit":
"ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb",
"enabled": true,
"state": "error",
"container_id": null,
"container_image_name": "quay.io/ceph/ceph:v15",
"container_image_id": null,
"version": null,
"started": null,
"created": "2022-09-01T20:59:49.314347Z",
"deployed": "2022-09-01T20:59:48.718347Z",
"configured": "2022-09-01T20:59:49.314347Z"
},


Getting error

root@ceph1:~# cephadm unit --fsid f270ad9e-1f6f-11ed-b6f8-a539d87379ea
--name mgr.ceph1.xmbvsb start
stderr Job for
ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service failed
because the control process exited with error code.
stderr See "systemctl status
ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb.service" and
"journalctl -xe" for details.
Traceback (most recent call last):
  File "/usr/sbin/cephadm", line 6250, in 
r = args.func()
  File "/usr/sbin/cephadm", line 1357, in _infer_fsid
return func()
  File "/usr/sbin/cephadm", line 3727, in command_unit
call_throws([
  File "/usr/sbin/cephadm", line 1119, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: systemctl start
ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@mgr.ceph1.xmbvsb


How do I remove and re-deploy mgr?

On Thu, Sep 1, 2022 at 4:54 PM Adam King  wrote:

> cephadm deploys the containers with --rm so they will get removed if you
> stop them. As for getting the 2nd mgr back, if it still lists the 2nd one
> in `ceph orch ps` you should be able to do a `ceph orch daemon redeploy
> ` where  should match the name given in
> the orch ps output for the one that isn't actually up. If it isn't listed
> there, given you have a count of 2, cephadm should deploy another one. I do
> see in the orch ls output you posted that it says the mgr service has "2/2"
> running which implies it believes a 2nd mgr is present (and you would
> therefore be able to try the daemon redeploy if that daemon isn't actually
> there).
>
> Is it still reporting the duplicate osds in orch ps? I see in the cephadm
> ls output on ceph1 that osd.2 isn't being reported, which was reported as
> being on ceph1 in the orch ps output in your original message in this
> thread. I'm interested in what `ceph health detail` is reporting now as
> well, as it says there are 4 stray daemons. Also, the `ceph orch host ls`
> output just to get a better grasp of the topology of this cluster.
>
> On Thu, Sep 1, 2022 at 3:50 PM Satish Patel  wrote:
>
>> Adam,
>>
>> I have posted a question related to upgrading earlier and this thread is
>> related to that, I have opened a new one because I found that error in logs
>> and thought the upgrade may be stuck because of duplicate OSDs.
>>
>> root@ceph1:~# ls -l /var/lib/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/
>> total 44
>> drwx-- 3 nobody nogroup 4096 Aug 19 05:37 alertmanager.ceph1
>> drwx-- 3167 167 4096 Aug 19 05:36 crash
>> drwx-- 2167 167 4096 Aug 19 05:37 crash.ceph1
>> drwx-- 4998 996 4096 Aug 19 05:37 grafana.ceph1
>> drwx-- 2167 167 4096 Aug 19 05:36 mgr.ceph1.xmbvsb
>> drwx-- 3167 167 4096 Aug 19 05:36 mon.ceph1
>> drwx-- 2 nobody nogroup 4096 Aug 19 05:37 node-exporter.ceph1
>> drwx-- 2167 167 4096 Aug 19 17:55 osd.0
>> drwx-- 2167 167 4096 Aug 19 18:03 osd.1
>> drwx-- 2167 167 4096 Aug 31 05:20 osd.4
>> drwx-- 4 nobody nogroup 4096 Aug 19 05:38 prometheus.ceph1
>>
>> Here is the output of cephadm ls
>>
>> root@ceph1:~# cephadm ls
>> [
>> {
>> "style": "cephadm:v1",
>> "name": "alertmanager.ceph1",
>> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
>> "systemd_unit":
>> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@alertmanager.ceph1",
>> "enabled": true,
>> "state": "running",
>> "container_id":
>> "97403cf9799711461216b7f83e88c574da2b631c7c65233ebd82d8a216a48924",
>> "container_image_name": "quay.io/prometheus/alertmanager:v0.20.0
>> ",
>> "container_image_id":
>> "0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0",
>> "version": 

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Adam King]

cephadm deploys the containers with --rm so they will get removed if you
stop them. As for getting the 2nd mgr back, if it still lists the 2nd one
in `ceph orch ps` you should be able to do a `ceph orch daemon redeploy
` where  should match the name given in
the orch ps output for the one that isn't actually up. If it isn't listed
there, given you have a count of 2, cephadm should deploy another one. I do
see in the orch ls output you posted that it says the mgr service has "2/2"
running which implies it believes a 2nd mgr is present (and you would
therefore be able to try the daemon redeploy if that daemon isn't actually
there).

Is it still reporting the duplicate osds in orch ps? I see in the cephadm
ls output on ceph1 that osd.2 isn't being reported, which was reported as
being on ceph1 in the orch ps output in your original message in this
thread. I'm interested in what `ceph health detail` is reporting now as
well, as it says there are 4 stray daemons. Also, the `ceph orch host ls`
output just to get a better grasp of the topology of this cluster.

On Thu, Sep 1, 2022 at 3:50 PM Satish Patel  wrote:

> Adam,
>
> I have posted a question related to upgrading earlier and this thread is
> related to that, I have opened a new one because I found that error in logs
> and thought the upgrade may be stuck because of duplicate OSDs.
>
> root@ceph1:~# ls -l /var/lib/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/
> total 44
> drwx-- 3 nobody nogroup 4096 Aug 19 05:37 alertmanager.ceph1
> drwx-- 3167 167 4096 Aug 19 05:36 crash
> drwx-- 2167 167 4096 Aug 19 05:37 crash.ceph1
> drwx-- 4998 996 4096 Aug 19 05:37 grafana.ceph1
> drwx-- 2167 167 4096 Aug 19 05:36 mgr.ceph1.xmbvsb
> drwx-- 3167 167 4096 Aug 19 05:36 mon.ceph1
> drwx-- 2 nobody nogroup 4096 Aug 19 05:37 node-exporter.ceph1
> drwx-- 2167 167 4096 Aug 19 17:55 osd.0
> drwx-- 2167 167 4096 Aug 19 18:03 osd.1
> drwx-- 2167 167 4096 Aug 31 05:20 osd.4
> drwx-- 4 nobody nogroup 4096 Aug 19 05:38 prometheus.ceph1
>
> Here is the output of cephadm ls
>
> root@ceph1:~# cephadm ls
> [
> {
> "style": "cephadm:v1",
> "name": "alertmanager.ceph1",
> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
> "systemd_unit":
> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@alertmanager.ceph1",
> "enabled": true,
> "state": "running",
> "container_id":
> "97403cf9799711461216b7f83e88c574da2b631c7c65233ebd82d8a216a48924",
> "container_image_name": "quay.io/prometheus/alertmanager:v0.20.0",
> "container_image_id":
> "0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0",
> "version": "0.20.0",
> "started": "2022-08-19T16:59:02.461978Z",
> "created": "2022-08-19T03:37:16.403605Z",
> "deployed": "2022-08-19T03:37:15.815605Z",
> "configured": "2022-08-19T16:59:02.117607Z"
> },
> {
> "style": "cephadm:v1",
> "name": "grafana.ceph1",
> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
> "systemd_unit":
> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@grafana.ceph1",
> "enabled": true,
> "state": "running",
> "container_id":
> "c7136aea8349a37dd9b320acd926c4bcbed95bc4549779e9580ed4290edc2117",
> "container_image_name": "quay.io/ceph/ceph-grafana:6.7.4",
> "container_image_id":
> "557c83e11646f123a27b5e4b62ac6c45e7bb8b2e90d6044034d0db5b7019415c",
> "version": "6.7.4",
> "started": "2022-08-19T03:38:05.481992Z",
> "created": "2022-08-19T03:37:46.823604Z",
> "deployed": "2022-08-19T03:37:46.239604Z",
> "configured": "2022-08-19T03:38:05.163603Z"
> },
> {
> "style": "cephadm:v1",
> "name": "osd.1",
> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
> "systemd_unit": "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@osd.1",
> "enabled": true,
> "state": "running",
> "container_id":
> "51586b775bda0485c8b27b8401ac2430570e6f42cb7e12bae3eea05064f1fd20",
> "container_image_name": "quay.io/ceph/ceph:v15",
> "container_image_id":
> "93146564743febec815d6a764dad93fc07ce971e88315403ac508cb5da6d35f4",
> "version": "15.2.17",
> "started": "2022-08-19T16:03:10.612432Z",
> "created": "2022-08-19T16:03:09.765746Z",
> "deployed": "2022-08-19T16:03:09.141746Z",
> "configured": "2022-08-31T02:53:34.224643Z"
> },
> {
> "style": "cephadm:v1",
> "name": "prometheus.ceph1",
> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
> "systemd_unit":
> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@prometheus.ceph1",
> "enabled": true,
> "state": "running",
> "container_id":
> "ba305236e5db9f2095b23b86a2340924909e9e8e54e5cdbe1d51c14dc4c8587a",
> "container_image_name": 

[ceph-users] Re: Remove corrupt PG

[Jesper Lykkegaard Karlsen]

To answer my own question. 

The removal of the  corrupt PG, could be fixed by doing ceph-objectstore-tool 
fuse mount-thingy. 
Then from the mount point, delete everything in the PGs head directory. 

This took only a few seconds (compared to 7.5 days) and after unmount and 
restart of the OSD it came back online. 

Best, 
Jesper

--
Jesper Lykkegaard Karlsen
Scientific Computing
Centre for Structural Biology
Department of Molecular Biology and Genetics
Aarhus University
Universitetsbyen 81
8000 Aarhus C

E-mail: je...@mbg.au.dk
Tlf:+45 50906203

> On 31 Aug 2022, at 20.53, Jesper Lykkegaard Karlsen  wrote:
> 
> Hi all, 
> 
> I wanted to move a PG to an empty OSD, so I could do repairs on it without 
> the whole OSD, which is full of other PG’s, would be effected with extensive 
> downtime. 
> 
> Thus, I exported the PG with ceph-objectstore-tool, an after successful 
> export I removed it. Unfortunately, the remove command was interrupted 
> midway. 
> This resulted in a PG that could not be remove with “ceph-objectstore-tool 
> —op remove ….”, since the header is gone. 
> Worse is that the OSD does not boot, due to it can see objects from the 
> removed PG, but cannot access them. 
> 
> I have tried to remove the individual objects in that PG (also with 
> objectstore-tool), but this process is extremely slow. 
> When looping over the >65,000 object, each remove takes ~10 sec and is very 
> compute intensive, which is approximately 7.5 days. 
> 
> Is the a faster way to get around this? 
> 
> Mvh. Jesper
> 
> --
> Jesper Lykkegaard Karlsen
> Scientific Computing
> Centre for Structural Biology
> Department of Molecular Biology and Genetics
> Aarhus University
> Universitetsbyen 81
> 8000 Aarhus C
> 
> E-mail: je...@mbg.au.dk
> Tlf:+45 50906203
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Satish Patel]

Adam,

I have posted a question related to upgrading earlier and this thread is
related to that, I have opened a new one because I found that error in logs
and thought the upgrade may be stuck because of duplicate OSDs.

root@ceph1:~# ls -l /var/lib/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/
total 44
drwx-- 3 nobody nogroup 4096 Aug 19 05:37 alertmanager.ceph1
drwx-- 3167 167 4096 Aug 19 05:36 crash
drwx-- 2167 167 4096 Aug 19 05:37 crash.ceph1
drwx-- 4998 996 4096 Aug 19 05:37 grafana.ceph1
drwx-- 2167 167 4096 Aug 19 05:36 mgr.ceph1.xmbvsb
drwx-- 3167 167 4096 Aug 19 05:36 mon.ceph1
drwx-- 2 nobody nogroup 4096 Aug 19 05:37 node-exporter.ceph1
drwx-- 2167 167 4096 Aug 19 17:55 osd.0
drwx-- 2167 167 4096 Aug 19 18:03 osd.1
drwx-- 2167 167 4096 Aug 31 05:20 osd.4
drwx-- 4 nobody nogroup 4096 Aug 19 05:38 prometheus.ceph1

Here is the output of cephadm ls

root@ceph1:~# cephadm ls
[
{
"style": "cephadm:v1",
"name": "alertmanager.ceph1",
"fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
"systemd_unit":
"ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@alertmanager.ceph1",
"enabled": true,
"state": "running",
"container_id":
"97403cf9799711461216b7f83e88c574da2b631c7c65233ebd82d8a216a48924",
"container_image_name": "quay.io/prometheus/alertmanager:v0.20.0",
"container_image_id":
"0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0",
"version": "0.20.0",
"started": "2022-08-19T16:59:02.461978Z",
"created": "2022-08-19T03:37:16.403605Z",
"deployed": "2022-08-19T03:37:15.815605Z",
"configured": "2022-08-19T16:59:02.117607Z"
},
{
"style": "cephadm:v1",
"name": "grafana.ceph1",
"fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
"systemd_unit":
"ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@grafana.ceph1",
"enabled": true,
"state": "running",
"container_id":
"c7136aea8349a37dd9b320acd926c4bcbed95bc4549779e9580ed4290edc2117",
"container_image_name": "quay.io/ceph/ceph-grafana:6.7.4",
"container_image_id":
"557c83e11646f123a27b5e4b62ac6c45e7bb8b2e90d6044034d0db5b7019415c",
"version": "6.7.4",
"started": "2022-08-19T03:38:05.481992Z",
"created": "2022-08-19T03:37:46.823604Z",
"deployed": "2022-08-19T03:37:46.239604Z",
"configured": "2022-08-19T03:38:05.163603Z"
},
{
"style": "cephadm:v1",
"name": "osd.1",
"fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
"systemd_unit": "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@osd.1",
"enabled": true,
"state": "running",
"container_id":
"51586b775bda0485c8b27b8401ac2430570e6f42cb7e12bae3eea05064f1fd20",
"container_image_name": "quay.io/ceph/ceph:v15",
"container_image_id":
"93146564743febec815d6a764dad93fc07ce971e88315403ac508cb5da6d35f4",
"version": "15.2.17",
"started": "2022-08-19T16:03:10.612432Z",
"created": "2022-08-19T16:03:09.765746Z",
"deployed": "2022-08-19T16:03:09.141746Z",
"configured": "2022-08-31T02:53:34.224643Z"
},
{
"style": "cephadm:v1",
"name": "prometheus.ceph1",
"fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
"systemd_unit":
"ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@prometheus.ceph1",
"enabled": true,
"state": "running",
"container_id":
"ba305236e5db9f2095b23b86a2340924909e9e8e54e5cdbe1d51c14dc4c8587a",
"container_image_name": "quay.io/prometheus/prometheus:v2.18.1",
"container_image_id":
"de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490",
"version": "2.18.1",
"started": "2022-08-19T16:59:03.538981Z",
"created": "2022-08-19T03:38:01.567604Z",
"deployed": "2022-08-19T03:38:00.983603Z",
"configured": "2022-08-19T16:59:03.193607Z"
},
{
"style": "cephadm:v1",
"name": "node-exporter.ceph1",
"fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
"systemd_unit":
"ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@node-exporter.ceph1",
"enabled": true,
"state": "running",
"container_id":
"00bf3ad29cce79e905e8533648ef38cbd232990fa9616aff1c0020b7b66d0cc0",
"container_image_name": "quay.io/prometheus/node-exporter:v0.18.1",
"container_image_id":
"e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87",
"version": "0.18.1",
"started": "2022-08-19T03:37:55.232032Z",
"created": "2022-08-19T03:37:47.711604Z",
"deployed": "2022-08-19T03:37:47.155604Z",
"configured": "2022-08-19T03:37:47.711604Z"
},
{
"style": "cephadm:v1",
"name": "osd.0",
"fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea",
"systemd_unit": 

[ceph-users] Ceph Mgr/Dashboard Python depedencies: a new approach

[Ernesto Puerta]

Hi all,

For the Reef release, Dashboard team wants to deliver Python packages
embedded in the ceph-dashboard distro package. [1] This will make it
unnecessary to depend on distro Python packages and allow us to:

   - *Use new/er Python packages:* that are not available in all distros
   (e.g.: fastapi), or which are available but in very disparate versions (and
   hence with different feature sets, resulting in spaghetti code and
   complicating test matrices),
   - *Simplify upgrades to newer Python versions*: ceph-mgr package
   currently depends on another 42 python packages, so everytime we move to a
   newer minor version of Python we have to ensure that a total of almost 300
   new packages are available in the 7 supported distro-releases. Sometimes
   the distro maintainers provide those, but it's not infrequent that Ceph
   devels have to maintain themselves these missing packages,
   - *Improve security and quality*: contrary to common belief, the distro
   package approach does not intrinsically ensure a more secure, maintained
   environment, just a stable one. Many non-standard Python packages come from
   non-base repos (like Universe in Ubuntu, or EPEL in CentOS), which have
   pretty lax/no maintenance commitments at all. By controlling the versions
   of the Python deps, we can run tools like Dependabot to monitor for CVEs
   and security upgrades, and also perform our tests on a single-version
   stack. This is a paradoxical situation in which more duties translate into
   less (more effective) work.
   - *Summing up: spend more time exposing Ceph features in the Dashboard*,
   and less time implementing functionality that can be provided by these
   external libraries or dealing with cross-distro/environment issues.

We also find that this approach is consistent with widely-adopted best
practices for reproducible runtime environments (Python virtualenvs,
containers, etc.).

There're a couple of open points and clarifications:

   - A discussion about whether to give precedence, in the ceph-mgr
   sys.path, to the distro/site packages or to the embedded ones. The former
   approach would allow admins to override the embedded versions and provide
   their own versions, but it's very likely to result in issues (e.g.: distro
   packages provide an older version of a transitive dependency), so we would
   prefer the latter, in which only the Python standard libraries are imported
   via distro packages.
   - The embedded deps will still be under the control of the distro
   package management system. The only downside (or upside) is that these deps
   can't be shared/used by other Python modules. However, this also happens
   with containers, and is actually perceived as a positive outcome if you
   favor isolation/reproducibility over disk space savings.

[1] https://github.com/ceph/ceph/pull/47501

We'd like to hear your comments, concerns, and suggestions.

Thank you!

Kind Regards,


Ernesto Puerta

He / Him / His

Principal Software Engineer, Ceph

Red Hat 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: The next quincy point release

[Ilya Dryomov]

On Thu, Sep 1, 2022 at 8:19 PM Yuri Weinstein  wrote:
>
> I have several PRs that are ready for merge but failing "make check"
>
> https://github.com/ceph/ceph/pull/47650 (main related to quincy)
> https://github.com/ceph/ceph/pull/47057
> https://github.com/ceph/ceph/pull/47621
> https://github.com/ceph/ceph/pull/47056
>
> The only way known now is to restart "make check" until it passes
> (which is not too efficient as you might guess)
> We suspect that something has changed on Jenkins nodes, but not sure.
>
> Anybody can help in resolving this, please?

FWIW this is the ticket for the main offender:

https://tracker.ceph.com/issues/57116

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm upgrade from octopus to pasific stuck

[Adam King]

Does "ceph orch upgrade status" give any insights (e.g. an error message of
some kind)? If not, maybe you could try looking at
https://tracker.ceph.com/issues/56485#note-2 because it seems like a
similar issue and I see you're using --ceph-version (which we need to fix,
sorry about that).

On Wed, Aug 31, 2022 at 10:58 PM Satish Patel  wrote:

> Hi,
>
> I have a small cluster in the lab which has only two nodes. I have a single
> monitor and two OSD nodes.
>
> Running upgrade but somehow it stuck after upgrading mgr
>
> ceph orch upgrade start --ceph-version 16.2.10
>
> root@ceph1:~# ceph -s
>   cluster:
> id: f270ad9e-1f6f-11ed-b6f8-a539d87379ea
> health: HEALTH_WARN
> 5 stray daemon(s) not managed by cephadm
>
>   services:
> mon: 1 daemons, quorum ceph1 (age 22m)
> mgr: ceph1.xmbvsb(active, since 21m), standbys: ceph2.hmbdla
> osd: 6 osds: 6 up (since 23h), 6 in (since 8d)
>
>   data:
> pools:   6 pools, 161 pgs
> objects: 20.53k objects, 85 GiB
> usage:   173 GiB used, 826 GiB / 1000 GiB avail
> pgs: 161 active+clean
>
>   io:
> client:   0 B/s rd, 2.7 KiB/s wr, 0 op/s rd, 0 op/s wr
>
>   progress:
> Upgrade to quay.io/ceph/ceph:v16.2.10 (0s)
>   []
>
>
> root@ceph1:~# ceph health detail
> HEALTH_WARN 5 stray daemon(s) not managed by cephadm
> [WRN] CEPHADM_STRAY_DAEMON: 5 stray daemon(s) not managed by cephadm
> stray daemon mgr.ceph1.xmbvsb on host ceph1 not managed by cephadm
> stray daemon mon.ceph1 on host ceph1 not managed by cephadm
> stray daemon osd.0 on host ceph1 not managed by cephadm
> stray daemon osd.1 on host ceph1 not managed by cephadm
> stray daemon osd.4 on host ceph1 not managed by cephadm
>
>
> root@ceph1:~# ceph log last cephadm
> 2022-09-01T02:46:12.020993+ mgr.ceph1.xmbvsb (mgr.254112) 437 : cephadm
> [INF] refreshing ceph2 facts
> 2022-09-01T02:47:12.016303+ mgr.ceph1.xmbvsb (mgr.254112) 469 : cephadm
> [INF] refreshing ceph1 facts
> 2022-09-01T02:47:12.431002+ mgr.ceph1.xmbvsb (mgr.254112) 470 : cephadm
> [INF] refreshing ceph2 facts
> 2022-09-01T02:48:12.424640+ mgr.ceph1.xmbvsb (mgr.254112) 501 : cephadm
> [INF] refreshing ceph1 facts
> 2022-09-01T02:48:12.839790+ mgr.ceph1.xmbvsb (mgr.254112) 502 : cephadm
> [INF] refreshing ceph2 facts
> 2022-09-01T02:49:12.836875+ mgr.ceph1.xmbvsb (mgr.254112) 534 : cephadm
> [INF] refreshing ceph1 facts
> 2022-09-01T02:49:13.210871+ mgr.ceph1.xmbvsb (mgr.254112) 535 : cephadm
> [INF] refreshing ceph2 facts
> 2022-09-01T02:50:13.207635+ mgr.ceph1.xmbvsb (mgr.254112) 566 : cephadm
> [INF] refreshing ceph1 facts
> 2022-09-01T02:50:13.615722+ mgr.ceph1.xmbvsb (mgr.254112) 568 : cephadm
> [INF] refreshing ceph2 facts
>
>
>
> root@ceph1:~# ceph orch ps
> NAME
>  HOST   STATUS REFRESHED  AGE  VERSIONIMAGE NAME
>  IMAGE ID  CONTAINER ID
> cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d
>  ceph1  stopped3m ago -  
>  
> cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d
>  ceph2  stopped3m ago -  
>  
> crash.ceph2
> ceph1  running (12d)  3m ago 12d  15.2.17quay.io/ceph/ceph:v15
> 93146564743f  0a009254afb0
> crash.ceph2
> ceph2  running (12d)  3m ago 12d  15.2.17quay.io/ceph/ceph:v15
> 93146564743f  0a009254afb0
> mgr.ceph2.hmbdla
>  ceph1  running (43m)  3m ago 12d  16.2.10
> quay.io/ceph/ceph:v16.2.10
>0d668911f040  6274723c35f7
> mgr.ceph2.hmbdla
>  ceph2  running (43m)  3m ago 12d  16.2.10
> quay.io/ceph/ceph:v16.2.10
>0d668911f040  6274723c35f7
> node-exporter.ceph2
> ceph1  running (23m)  3m ago 12d  0.18.1
> quay.io/prometheus/node-exporter:v0.18.1  e5a616e4b9cf  7a6217cb1a9e
> node-exporter.ceph2
> ceph2  running (23m)  3m ago 12d  0.18.1
> quay.io/prometheus/node-exporter:v0.18.1  e5a616e4b9cf  7a6217cb1a9e
> osd.2
> ceph1  running (23h)  3m ago 12d  15.2.17quay.io/ceph/ceph:v15
> 93146564743f  e286fb1c6302
> osd.2
> ceph2  running (23h)  3m ago 12d  15.2.17quay.io/ceph/ceph:v15
> 93146564743f  e286fb1c6302
> osd.3
> ceph1  running (23h)  3m ago 12d  15.2.17quay.io/ceph/ceph:v15
> 93146564743f  d3ae5d9f694f
> osd.3
> ceph2  running (23h)  3m ago 12d  15.2.17quay.io/ceph/ceph:v15
> 93146564743f  d3ae5d9f694f
> osd.5
> ceph1  running (23h)  3m ago 8d   15.2.17quay.io/ceph/ceph:v15
> 93146564743f  405068fb474e
> osd.5
> ceph2  running (23h)  3m ago 8d   15.2.17quay.io/ceph/ceph:v15
> 93146564743f  405068fb474e
>
>
>
> What could be wrong here and how to debug issue, cephadm is new to me so
> not sure where to look for logs
> ___
> ceph-users mailing list -- 

[ceph-users] Re: [cephadm] Found duplicate OSDs

[Adam King]

Are there any extra directories in /var/lib/ceph or /var/lib/ceph/
that appear to be for those OSDs on that host? When cephadm builds the info
it uses for "ceph orch ps" it's actually scraping those directories. The
output of "cephadm ls" on the host with the duplicates could also
potentially have some insights.

On Thu, Sep 1, 2022 at 12:15 PM Satish Patel  wrote:

> Folks,
>
> I am playing with cephadm and life was good until I started upgrading from
> octopus to pacific. My upgrade process stuck after upgrading mgr and in
> logs now i can see following error
>
> root@ceph1:~# ceph log last cephadm
> 2022-09-01T14:40:45.739804+ mgr.ceph2.hmbdla (mgr.265806) 8 :
> cephadm [INF] Deploying daemon grafana.ceph1 on ceph1
> 2022-09-01T14:40:56.115693+ mgr.ceph2.hmbdla (mgr.265806) 14 :
> cephadm [INF] Deploying daemon prometheus.ceph1 on ceph1
> 2022-09-01T14:41:11.856725+ mgr.ceph2.hmbdla (mgr.265806) 25 :
> cephadm [INF] Reconfiguring alertmanager.ceph1 (dependencies
> changed)...
> 2022-09-01T14:41:11.861535+ mgr.ceph2.hmbdla (mgr.265806) 26 :
> cephadm [INF] Reconfiguring daemon alertmanager.ceph1 on ceph1
> 2022-09-01T14:41:12.927852+ mgr.ceph2.hmbdla (mgr.265806) 27 :
> cephadm [INF] Reconfiguring grafana.ceph1 (dependencies changed)...
> 2022-09-01T14:41:12.940615+ mgr.ceph2.hmbdla (mgr.265806) 28 :
> cephadm [INF] Reconfiguring daemon grafana.ceph1 on ceph1
> 2022-09-01T14:41:14.056113+ mgr.ceph2.hmbdla (mgr.265806) 33 :
> cephadm [INF] Found duplicate OSDs: osd.2 in status running on ceph1,
> osd.2 in status running on ceph2
> 2022-09-01T14:41:14.056437+ mgr.ceph2.hmbdla (mgr.265806) 34 :
> cephadm [INF] Found duplicate OSDs: osd.5 in status running on ceph1,
> osd.5 in status running on ceph2
> 2022-09-01T14:41:14.056630+ mgr.ceph2.hmbdla (mgr.265806) 35 :
> cephadm [INF] Found duplicate OSDs: osd.3 in status running on ceph1,
> osd.3 in status running on ceph2
>
>
> Not sure from where duplicate names came and how that happened. In
> following output i can't see any duplication
>
> root@ceph1:~# ceph osd tree
> ID  CLASS  WEIGHT   TYPE NAME   STATUS  REWEIGHT  PRI-AFF
> -1 0.97656  root default
> -3 0.48828  host ceph1
>  4hdd  0.09769  osd.4   up   1.0  1.0
>  0ssd  0.19530  osd.0   up   1.0  1.0
>  1ssd  0.19530  osd.1   up   1.0  1.0
> -5 0.48828  host ceph2
>  5hdd  0.09769  osd.5   up   1.0  1.0
>  2ssd  0.19530  osd.2   up   1.0  1.0
>  3ssd  0.19530  osd.3   up   1.0  1.0
>
>
> But same time i can see duplicate OSD number in ceph1 and ceph2
>
>
> root@ceph1:~# ceph orch ps
> NAME HOST   PORTSSTATUS REFRESHED  AGE
>  MEM USE  MEM LIM  VERSION  IMAGE ID  CONTAINER ID
> alertmanager.ceph1   ceph1  *:9093,9094  running (20s) 2s ago  20s
>17.1M-   ba2b418f427c  856a4fe641f1
> alertmanager.ceph1   ceph2  *:9093,9094  running (20s) 3s ago  20s
>17.1M-   ba2b418f427c  856a4fe641f1
> crash.ceph2  ceph1   running (12d) 2s ago  12d
>10.0M-  15.2.17  93146564743f  0a009254afb0
> crash.ceph2  ceph2   running (12d) 3s ago  12d
>10.0M-  15.2.17  93146564743f  0a009254afb0
> grafana.ceph1ceph1  *:3000   running (18s) 2s ago  19s
>47.9M-  8.3.5dad864ee21e9  7d7a70b8ab7f
> grafana.ceph1ceph2  *:3000   running (18s) 3s ago  19s
>47.9M-  8.3.5dad864ee21e9  7d7a70b8ab7f
> mgr.ceph2.hmbdla ceph1   running (13h) 2s ago  12d
> 506M-  16.2.10  0d668911f040  6274723c35f7
> mgr.ceph2.hmbdla ceph2   running (13h) 3s ago  12d
> 506M-  16.2.10  0d668911f040  6274723c35f7
> node-exporter.ceph2  ceph1   running (91m) 2s ago  12d
>60.7M-  0.18.1   e5a616e4b9cf  d0ba04bb977c
> node-exporter.ceph2  ceph2   running (91m) 3s ago  12d
>60.7M-  0.18.1   e5a616e4b9cf  d0ba04bb977c
> osd.2ceph1   running (12h) 2s ago  12d
> 867M4096M  15.2.17  93146564743f  e286fb1c6302
> osd.2ceph2   running (12h) 3s ago  12d
> 867M4096M  15.2.17  93146564743f  e286fb1c6302
> osd.3ceph1   running (12h) 2s ago  12d
> 978M4096M  15.2.17  93146564743f  d3ae5d9f694f
> osd.3ceph2   running (12h) 3s ago  12d
> 978M4096M  15.2.17  93146564743f  d3ae5d9f694f
> osd.5ceph1   running (12h) 2s ago   8d
> 225M4096M  15.2.17  93146564743f  405068fb474e
> osd.5ceph2   running (12h) 3s ago   8d
> 225M4096M  15.2.17  93146564743f  405068fb474e
> prometheus.ceph1 ceph1  *:9095 

[ceph-users] [cephadm] Found duplicate OSDs

[Satish Patel]

Folks,

I am playing with cephadm and life was good until I started upgrading from
octopus to pacific. My upgrade process stuck after upgrading mgr and in
logs now i can see following error

root@ceph1:~# ceph log last cephadm
2022-09-01T14:40:45.739804+ mgr.ceph2.hmbdla (mgr.265806) 8 :
cephadm [INF] Deploying daemon grafana.ceph1 on ceph1
2022-09-01T14:40:56.115693+ mgr.ceph2.hmbdla (mgr.265806) 14 :
cephadm [INF] Deploying daemon prometheus.ceph1 on ceph1
2022-09-01T14:41:11.856725+ mgr.ceph2.hmbdla (mgr.265806) 25 :
cephadm [INF] Reconfiguring alertmanager.ceph1 (dependencies
changed)...
2022-09-01T14:41:11.861535+ mgr.ceph2.hmbdla (mgr.265806) 26 :
cephadm [INF] Reconfiguring daemon alertmanager.ceph1 on ceph1
2022-09-01T14:41:12.927852+ mgr.ceph2.hmbdla (mgr.265806) 27 :
cephadm [INF] Reconfiguring grafana.ceph1 (dependencies changed)...
2022-09-01T14:41:12.940615+ mgr.ceph2.hmbdla (mgr.265806) 28 :
cephadm [INF] Reconfiguring daemon grafana.ceph1 on ceph1
2022-09-01T14:41:14.056113+ mgr.ceph2.hmbdla (mgr.265806) 33 :
cephadm [INF] Found duplicate OSDs: osd.2 in status running on ceph1,
osd.2 in status running on ceph2
2022-09-01T14:41:14.056437+ mgr.ceph2.hmbdla (mgr.265806) 34 :
cephadm [INF] Found duplicate OSDs: osd.5 in status running on ceph1,
osd.5 in status running on ceph2
2022-09-01T14:41:14.056630+ mgr.ceph2.hmbdla (mgr.265806) 35 :
cephadm [INF] Found duplicate OSDs: osd.3 in status running on ceph1,
osd.3 in status running on ceph2


Not sure from where duplicate names came and how that happened. In
following output i can't see any duplication

root@ceph1:~# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME   STATUS  REWEIGHT  PRI-AFF
-1 0.97656  root default
-3 0.48828  host ceph1
 4hdd  0.09769  osd.4   up   1.0  1.0
 0ssd  0.19530  osd.0   up   1.0  1.0
 1ssd  0.19530  osd.1   up   1.0  1.0
-5 0.48828  host ceph2
 5hdd  0.09769  osd.5   up   1.0  1.0
 2ssd  0.19530  osd.2   up   1.0  1.0
 3ssd  0.19530  osd.3   up   1.0  1.0


But same time i can see duplicate OSD number in ceph1 and ceph2


root@ceph1:~# ceph orch ps
NAME HOST   PORTSSTATUS REFRESHED  AGE
 MEM USE  MEM LIM  VERSION  IMAGE ID  CONTAINER ID
alertmanager.ceph1   ceph1  *:9093,9094  running (20s) 2s ago  20s
   17.1M-   ba2b418f427c  856a4fe641f1
alertmanager.ceph1   ceph2  *:9093,9094  running (20s) 3s ago  20s
   17.1M-   ba2b418f427c  856a4fe641f1
crash.ceph2  ceph1   running (12d) 2s ago  12d
   10.0M-  15.2.17  93146564743f  0a009254afb0
crash.ceph2  ceph2   running (12d) 3s ago  12d
   10.0M-  15.2.17  93146564743f  0a009254afb0
grafana.ceph1ceph1  *:3000   running (18s) 2s ago  19s
   47.9M-  8.3.5dad864ee21e9  7d7a70b8ab7f
grafana.ceph1ceph2  *:3000   running (18s) 3s ago  19s
   47.9M-  8.3.5dad864ee21e9  7d7a70b8ab7f
mgr.ceph2.hmbdla ceph1   running (13h) 2s ago  12d
506M-  16.2.10  0d668911f040  6274723c35f7
mgr.ceph2.hmbdla ceph2   running (13h) 3s ago  12d
506M-  16.2.10  0d668911f040  6274723c35f7
node-exporter.ceph2  ceph1   running (91m) 2s ago  12d
   60.7M-  0.18.1   e5a616e4b9cf  d0ba04bb977c
node-exporter.ceph2  ceph2   running (91m) 3s ago  12d
   60.7M-  0.18.1   e5a616e4b9cf  d0ba04bb977c
osd.2ceph1   running (12h) 2s ago  12d
867M4096M  15.2.17  93146564743f  e286fb1c6302
osd.2ceph2   running (12h) 3s ago  12d
867M4096M  15.2.17  93146564743f  e286fb1c6302
osd.3ceph1   running (12h) 2s ago  12d
978M4096M  15.2.17  93146564743f  d3ae5d9f694f
osd.3ceph2   running (12h) 3s ago  12d
978M4096M  15.2.17  93146564743f  d3ae5d9f694f
osd.5ceph1   running (12h) 2s ago   8d
225M4096M  15.2.17  93146564743f  405068fb474e
osd.5ceph2   running (12h) 3s ago   8d
225M4096M  15.2.17  93146564743f  405068fb474e
prometheus.ceph1 ceph1  *:9095   running (8s)  2s ago   8s
   30.4M-   514e6a882f6e  9031dbe30cae
prometheus.ceph1 ceph2  *:9095   running (8s)  3s ago   8s
   30.4M-   514e6a882f6e  9031dbe30cae


Is this a bug or did I do something wrong? any workaround to get out
from this condition?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] More recovery pain

[Wyll Ingersoll]

We are in the middle of a massive recovery event and our monitor DBs keep 
exploding to the point that they fill their disk partition (800GB disk).   We 
cannot compact it because there is no room on the device for compaction to 
happen.  We cannot add another disk at this time either.  We destroyed rebuilt 
one of them and now it won't come back into quorum.

What other parameters can we set to limit the growth of the monitor DB during 
recovery?
Are there tools to manually purge the LOGM messages from the monitor DB?

We disabled clog_to_monitor already.



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Fwd: Active-Active MDS RAM consumption

[Gerdriaan Mulder]

Hi Kamil,

Seems like this issue 
 
where the MDS loads all direntries into memory. You should probably take 
a look at the mds_oft_prefetch_dirfrags setting (which changed default 
from true to false in 16.2.8).


Best regards,
Gerdriaan Mulder

On 01/09/2022 14.39, Kamil Madac wrote:

Hi Ceph Community

One of my customer has an issue with the MDS cluster. Ceph cluster is
deployed with cephadm and is in version 16.2.7. As soon as MDS is switched
from Active-Standby to Active-Active-Standby, MDS daemon starts to consume
a lot of RAM. After some time it consumes 48GB RAM, and container engine
kills it. Same thing happens then on the second node which is killed after
some time as well, and the situation repeats again.

When the MDS cluster is switched back to Active-Backup MDS configuration
the situation stabilizes.

mds_cache_memory_limit is set to 4294967296, which is the default value. No
health warning about high cache consumption is generated.

Is that known behavior, and can it be solved by some reconfiguration?

Can someone give us a hint on what to check, debug or tune?

Thank you.

Kamil Madac
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: The next quincy point release

[Venky Shankar]

On Tue, Aug 30, 2022 at 10:48 PM Yuri Weinstein  wrote:
>
> I have several PRs in testing:
>
> https://github.com/ceph/ceph/labels/wip-yuri2-testing
> https://github.com/ceph/ceph/labels/wip-yuri-testing (needs fs review)
>
> Assuming they were merged, anything else is a must to be added to the
> next quincy point release?

I tagged ~3 backport PRs for Q -- would be good to have them merged.

>
> Dev leads ^^^
>
> Thx
> YuriW
>
> On Mon, Aug 29, 2022 at 1:26 PM Yuri Weinstein  wrote:
> >
> > Update:
> >
> > It's taking longer than we planned to start QE for this point release
> > due to various reasons.
> > We still want to test and release it ASAP.
> >
> > Sorry for the delay.
> > YuriW
> >
> > On Mon, Aug 15, 2022 at 8:00 AM Yuri Weinstein  wrote:
> > >
> > > We plan to start QE validation for the next quincy point release this 
> > > week.
> > >
> > > Dev leads please tag all PRs needed to be included ("needs-qa") ASAP
> > > so they can be tested and merged on time.
> > >
> > > Thx
> > > YuriW
>
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 
Cheers,
Venky

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Leadership Team Meeting Minutes - August 31, 2022

[Venky Shankar]

On Thu, Sep 1, 2022 at 10:12 AM Neha Ojha  wrote:
>
> Hi everyone,
>
> Here are the topics discussed in today's meeting.
>
> - David Galoway's last CLT meeting, mixed emotions but we wish David
> all the best for all his future endeavors
> - Tracker upgrade postponed for now
> - OVH payment logistics handover to Patrick
> - Discussion on cephadm documentation
> - ceph-mgr deps via pip vs distro packages
> (https://github.com/ceph/ceph/pull/47501#pullrequestreview-1082126162),
> more discussion to follow in the user list
> - Further discussion on coverity reboot
> (https://scan.coverity.com/projects/ceph), works with quincy but not
> with main, other suggestions https://clang.llvm.org/extra/clang-tidy/
> and https://clang-analyzer.llvm.org/
> - Tech leads to update the Ceph Backlog Trello to reflect Reef features
> - The next Quincy point release is being planned. Yuri needs to be
> made aware of any outstanding fixes as soon as possible.

I've tagged cephfs related Q backport as needs-qa. Would be good to
get those merged.

(And, we might have a couple of more incoming)

>
> Thanks,
> Neha
>
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 
Cheers,
Venky

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Fwd: Active-Active MDS RAM consumption

[Kamil Madac]

Hi Ceph Community

One of my customer has an issue with the MDS cluster. Ceph cluster is
deployed with cephadm and is in version 16.2.7. As soon as MDS is switched
from Active-Standby to Active-Active-Standby, MDS daemon starts to consume
a lot of RAM. After some time it consumes 48GB RAM, and container engine
kills it. Same thing happens then on the second node which is killed after
some time as well, and the situation repeats again.

When the MDS cluster is switched back to Active-Backup MDS configuration
the situation stabilizes.

mds_cache_memory_limit is set to 4294967296, which is the default value. No
health warning about high cache consumption is generated.

Is that known behavior, and can it be solved by some reconfiguration?

Can someone give us a hint on what to check, debug or tune?

Thank you.

Kamil Madac
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Questions about the QA process and the data format of both OSD and MON

[Neha Ojha]

Hi Satoru,

Apologies for the delay in responding to your questions.

In the case of https://github.com/ceph/ceph/pull/45963, we caught the
bug in an upgrade test (as described in
https://tracker.ceph.com/issues/55444) and not in the rados test
suite. Our upgrade test suites are meant to be run before we do point
releases, to ensure backward compatibility with N-2 and N-1 releases.
As I mentioned in https://tracker.ceph.com/issues/55444#note-12, the
same test scenario was run as a part of the rados suite before the
original PR was merged, but we did not catch this bug then, because
the we were testing the patch against pacific, where resharding is on
by default. When the same test scenario was run as a part of the
upgrade test, resharding was not on by default, so we caught this
issue. We are have minimal upgrade tests within the rados suite, but
unfortunately, they weren't enough to catch this particular issue.

Rest of the answers are inline.

On Fri, Aug 26, 2022 at 2:52 AM Satoru Takeuchi
 wrote:
>
> Could anyone answe this question? There are many questions but it's of
> course really helpful to know the answer of just one question.
>
> The summary of my questions.
>
> - a. About QA process
>   - a.1 The nunber of test cases differ between the QA for merging a PR and
> the QA for release?
>- a.2 If a.1 is correct, is it possible to chenge the CI system to run
> all test cases in both QAs?

Our testing framework, teuthology uses a concept of subsets to
determine the number of jobs scheduled in a run, you can find detailed
information about it in
https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro/#nested-subsets.
I have already explained what was different in the case of the bug you
referred to. We can certainly encourage more use of our upgrade suites
before merging critical PRs.

>- a.3 How to run Teuthology in my local environment?

At this point, we have the ability to run some tests locally using
teuthology, Junior (cc'ed here) did a presentation on this topic,
which was recorded here: https://www.youtube.com/watch?v=wZHcg0oVzhY.

> - b. Is there any way to know the data format of each OSD and MON (e.g.
> created in version A with cobfiguration B) by issuing Ceph commands?

I am not quite sure I understand your question.

Hope this helps!

Thanks,
Neha
>
> The detail is in the following quoted description.
>
> 2022年8月19日(金) 15:39 Satoru Takeuchi :
>
> > Hi,
> >
> > As I described in another mail(*1), my development Ceph cluster was
> > corrupted when using problematic binary.
> > When I upgraded to v16.2.7 + some patches (*2) + PR#45963 patch,
> > unfound pgs and inconsistent pgs appeared. In the end, I deleted this
> > cluster.
> >
> >   pacific: bluestore: set upper and lower bounds on rocksdb omap iterators
> >   https://github.com/ceph/ceph/pull/45963
> >
> > This problem happened because PR#45963 causes data corruption about OSDs
> > which were created in octopus or older.
> >
> > This patch was reverted, and the correct version (PR#46096) was applied
> > later.
> >
> >   pacific: revival and backport of fix for RocksDB optimized iterators
> >   https://github.com/ceph/ceph/pull/46096
> >
> > It's mainly because I applied the not-well-tested patch carelessly. To
> > prevent the same
> > a mistake from happening again, let me ask some questions.
> >
> > a. About QA process
> >a.1 In my understanding, the test cases differ between the QA for
> > merging
> >  a PR and the QA for release. For example, the upgrade test was
> > run only
> >  in the release QA process. Is my understanding correct?
> >  I thought so because the bug in #45963 was not detected in
> > the QA for merging
> >  but was detected in the QA for release.
> >a.2 If a.1 is correct, is it possible to run all test cases in both
> > QA? I guess that some
> > time-consuming tests are skipped to improve efficient development.
> >a.3 Is there any detailed document about how to run Teuthology in
> > the user's local environment?
> >  Once I tried this by reading the official document, it didn't
> > work well.
> >
> >
> > https://docs.ceph.com/en/quincy/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro/#how-to-run-integration-tests
> >
> >  At that time, Teuthology failed to connect to
> > paddles.front.sepia.ceph.com, which wasn't written in this document.
> >
> >  ```
> >  requests.exceptions.ConnectionError:
> > HTTPConnectionPool(host='paddles.front.sepia.ceph.com', port=80): Max
> > retries exceeded with url: /nodes/?machine_type=vps=1 (Caused by
> > NewConnectionError(' > 0x7fc945880490>: Failed to establish a new connection: [Errno 110]
> > Connection timed out'))
> >  ```
> > b. To minimize the risk, I'd like to use the newest data format of
> > both OSD and MON as possible.
> > More precisely, I'd like to re-create all 

[ceph-users] Re: how to speed up hundreds of millions small files read base on cephfs?

[Maged Mokhtar]

Hi, experts,

We are using cephfs(15.2.*) with kernel mount on our production environment. 
And these days when we do massive read from cluster(multi processes),  ceph 
health always report slow ops for some osds(build with hdd(8TB) which using ssd 
as db cache).

our cluster have more read than write request.

health log like below:
100 slow ops, oldest one blocked for 114 sec, [osd.* ...] has slow ops (SLOW 
_OPS)

  
my question is does there any best practices to process hundreds of millions small files(means 100kb-300kb each file and 1+ files in each directory, also more than 5000 directory)?


Small files are slow on any HDD system, each HDD can only do around 100 
ops  per sec. Some things to try, not that some may involve data copy-ing:


-If your workload logic involves more processing on recent files, may 
have 2 pools, 1 ssd pool for recent files and a larger hdd for less 
accessed archived files.


-if you can group files to be processed in groups, maybe store them in 
larger lumps like via tar files or even re-structure their data in 
SQLite , then you would modify the processing application to tar/untar 
the goup, or access data via SQLite.


-it may help to reduce the read_ahead_kb on your HHD devices to reduce 
un-needed load.


-Using dm-cache on the HDD may help, though our experience with it is 
not great (we use dm-writecache instead but is geared for speeding 
writes), it should cache more recent read objects on ssd, but its 
promotion algorithms may not match your workload pattern, maybe try it 
first in a lab with similar workload pattern.


-Using Ceph cache tier may help, though our experience with it is not 
great, its support is also deprecated


-The file sizes, average 150kb, are not large but also not extremely 
small, you could lower the application concurrency/processes so not to 
stress the disk % busy over say 80%, with 150kb size you should get 
around 10 MB/s read speeds from your HDD. Having too much processes 
could actually slow things.


-You may want to lower your scrub rates or increase the scrub window, if 
you have a lot of small files this will already be stressing your HDDs.


-Any Ceph recovery healing with small files on HDD will also slow things 
down but it is something to bear in mind not too much we can do.


/maged

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs crush - Since Pacific

[Igor Fedotov]

Hi Wissem,

given the log output it looks like suicide timeout has been fired. From 
my experience this is often observed when DB performance is degraded 
after bulk removals. And offline compaction should provide some relief. 
At least temporarily... But if deletes are ongoing (e.g. due to cluster 
rebuilding) another compaction round might be needed.



Thanks,

Igor

On 8/31/2022 2:37 PM, Wissem MIMOUNA wrote:

Hi Igor ,
I attached the full log file found beside the crash report on the 
concerned ceph osd server.

Thank you for your time J
Hi Wissem,
sharing OSD log snippet preceding the crash (e.g. prior 20K lines) could
be helpful and hopefully will provide more insigh - there might be some
errors/assertion details and/or other artefacts...
Thanks,
Igor


--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us athttps://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web:https://croit.io  | YouTube:https://goo.gl/PGE1Bx
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] how to speed up hundreds of millions small files read base on cephfs?

[zxcs]

Hi, experts,

We are using cephfs(15.2.*) with kernel mount on our production environment. 
And these days when we do massive read from cluster(multi processes),  ceph 
health always report slow ops for some osds(build with hdd(8TB) which using ssd 
as db cache). 

our cluster have more read than write request. 

health log like below:
100 slow ops, oldest one blocked for 114 sec, [osd.* ...] has slow ops (SLOW 
_OPS)

 
my question is does there any best practices to process hundreds of millions 
small files(means 100kb-300kb each file and 1+ files in each directory, 
also more than 5000 directory)? A
 
Any config we can tune or any patch we can apply try to speed up the read(more 
important than write) and any other file system we could try (we also not sure 
cephfs is the best choice to store such huge small files )?

Please experts shed some light here! We really need your are help here!

Any suggestions are welcome! Thanks in advance!~

Thanks,
zx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Wide variation in osd_mclock_max_capacity_iops_hdd

[Sridhar Seshasayee]

Hello Vladimir,

I have noticed that our osd_mclock_max_capacity_iops_hdd
> varies widely for OSDs on identical drives in identical
> machines (from ~600 to ~2800).
>

The IOPS shouldn't vary widely if the drives are of similar age and running
the same workloads. The osd_mclock_max_capacity_iops_hdd is determined
using Ceph's osd bench during osd boot-up. From our testing on HDDs, the
tool shows fairly consistent numbers (+/- a few 10s of IOPS) for a given
HDD.
For more details please see:
https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#osd-capacity-determination-automated

In your case, it would be good to take another look at the subset of HDDs
showing degraded/lower IOPS performance and check if there are any
underlying issues. You could run the osd bench against the affected osds as
described in the link below or your preferred tool to get a better
understanding:

https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#benchmarking-test-steps-using-osd-bench


> Is such great variation a problem? What effect on
> performance does this have?
>

The mClock profiles use the IOPS capacity of each osd to allocate IOPS
resources to ops like client, recovery, scrubs and so on. Therefore, a
lower IOPS
capacity will have an impact on these ops and therefore it would make sense
to
check the health of the HDDs that are showing lower than expected IOPS
numbers.
-Sridhar
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: how to fix mds stuck at dispatched without restart ads

[zxcs]

Thanks a lot, xiubo!!!


this time we still restarted mds fix this due to user urgent need list 
/path/to/A/, i will try to mds debug log if we hit it again.

Also, haven’t try flush mds journal before, any side effect to do this? This 
cephfs cluster is a production environment, we need very careful to do anything.

And I read this bug details  https://tracker.ceph.com/issues/50840 
 , from pre-mail mentions it fixed in 
15.2.17, we using ceps-deploy to deploy(upgrade) cephfs, seems it latest ceph 
version is 15.2.16? Will update if we can fixed after upgrade. 

Will try to flush ads journal option when we hit this bug next time(if no user 
urgent need list directory). Seems it can 100% recurrent these days. Thanks All!




Thanks,
zx



> 2022年8月31日 15:23，Xiubo Li  写道：
> 
> 
> On 8/31/22 2:43 PM, zxcs wrote:
>> Hi, experts
>> 
>> we have a cephfs(15.2.13) cluster with kernel mount, and when we read from 
>> 2000+ processes to one ceph path(called /path/to/A/), then all of the 
>> process hung, and ls -lrth /path/to/A/ always stuck, but list other 
>> directory are health( /path/to/B/),
>> 
>> health detail always report mds has slow request.  And then we need to 
>> restart the mds fix this issue.
>> 
>> How can we fix this without restart mds(restart mds always impact other 
>> users)?
>> 
>> Any suggestions are welcome! Thanks a ton!
>> 
>> from this dump_ops_in_flight:
>> 
>> "description": "client_request(client.100807215:2856632 getattr AsLsXsFs 
>> #0x200978a3326 2022-08-31T09:36:30.444927+0800 caller_id=2049, 
>> caller_gid=2049})",
>> "initiated_at": "2022-08-31T09:36:30.454570+0800",
>> "age": 17697.012491966001,
>> "duration": 17697.012805568,
>> "type_data": {
>> "flag_point": "dispatched",
>> "reqid": "client. 100807215:2856632",
>> "op_type": "client_request",
>> "client_info":
>> "client": "client.100807215",
>> "tid": 2856632
>> "events":
>> "time": "2022-08-31T09:36:30.454570+0800",
>> "event": "initiated"
>> 
>> "time": "2022-08-31T09:36:30.454572+0800",
>> "event": "throttled"
>> 
>> "time": "2022-08-31T09:36:30.454570+0800",
>> "event": "header read"
>> 
>> "time": "2022-08-31T09:36:30.454580+0800",
>> 'event": "all_read"
>> "time": "2022-08-31T09:36:30.454604+0800",
>> "event": "dispatched"
>> }
>> 
> AFAIK there is no easy way to do this. At least we need to know why it gets 
> stuck and where. From your above and the previous mail thread, it should 
> stuck in getattr request and sounds like a smiliar issue with 
> https://tracker.ceph.com/issues/50840 .
> 
> If it's not, it should be a new bug and could you create one tracker and 
> provide the mds side debug logs.
> 
> Maybe you can try to flush the mds journal to see what will happen ?
> 
> - Xiubo
> 
> 
>> 
>> Thanks,
>> Xiong
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io 
>> To unsubscribe send an email to ceph-users-le...@ceph.io 
>> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io