[ceph-users] Re: 'ceph fs status' no longer works?

[Eugen Block]

Yeah, I knew it was something trivial, I just checked my notes but  
didn't have anything written down. I agree, it's not a big deal but  
shouldn't be necessary.


Zitat von Erich Weiler :


Excellent!  Restarting all the MDS daemons fixed it.  Thank you.

This kinda feels like a bug.

-erich

On 5/2/24 12:44 PM, Bandelow, Gunnar wrote:

Hi Erich,

im not sure about this specific error message, but "ceph fs status"  
sometimes did fail me end of last year/in the beginning of the year.


Restarting ALL mon, mgr AND mds fixed it at the time.

Best regards,
Gunnar


===

Gunnar Bandelow (dipl. phys.)

Universitätsrechenzentrum (URZ)
Universität Greifswald
Felix-Hausdorff-Straße 18
17489 Greifswald
Germany


--- Original Nachricht ---
*Betreff: *[ceph-users] Re: 'ceph fs status' no longer works?
*Von: *"Erich Weiler" mailto:wei...@soe.ucsc.edu>>
*An: *"Eugen Block" mailto:ebl...@nde.ag>>,  
ceph-users@ceph.io 

*Datum: *02-05-2024 21:05



   Hi Eugen,

   Thanks for the tip!  I just ran:

   ceph orch daemon restart mgr.pr-md-01.jemmdf

   (my specific mgr instance)

   And it restarted my primary mgr daemon, and in the process failed over
   to my standby mgr daemon on another server.  That went smoothly.

   Unfortunately, I still cannot get 'ceph fs status' to work (on any
   node)...

   # ceph fs status
   Error EINVAL: Traceback (most recent call last):
   File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in
   _handle_command
 return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
   File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
 return self.func(mgr, **kwargs)
   File "/usr/share/ceph/mgr/status/module.py", line 109, in
   handle_fs_status
 assert metadata
   AssertionError

   -erich

   On 5/2/24 11:07 AM, Eugen Block wrote:
> Yep, seen this a couple of times during upgrades. I’ll have to
   check my
> notes if I wrote anything down for that. But try a mgr failover
   first,
> that could help.
>
> Zitat von Erich Weiler mailto:wei...@soe.ucsc.edu>>:
>
>> Hi All,
>>
>> For a while now I've been using 'ceph fs status' to show current
   MDS
>> active servers, filesystem status, etc.  I recently took down my
   MDS
>> servers and added RAM to them (one by one, so the filesystem stayed
>> online).  After doing that with my four MDS servers (I had two
   active
>> and two standby), all looks OK, 'ceph -s' reports HEALTH_OK.  
But when

>> I do 'ceph fs status' now, I get this:
>>
>> # ceph fs status
>> Error EINVAL: Traceback (most recent call last):
>>   File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in
   _handle_command
>>     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
>>   File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
>>     return self.func(mgr, **kwargs)
>>   File "/usr/share/ceph/mgr/status/module.py", line 109, in
>> handle_fs_status
>>     assert metadata
>> AssertionError
>>
>> This is on ceph 18.2.1 reef.  This is very odd - can anyone
   think of a
>> reason why 'ceph fs status' would stop working after taking each of
>> the servers down for maintenance?
>>
>> The filesystem is online and working just fine however.  This ceph
>> instance is deployed via the cephadm method on RHEL 9.3, so the
>> everything is containerized in podman.
>>
>> Thanks again,
>> erich
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
   
>> To unsubscribe send an email to ceph-users-le...@ceph.io
   
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
   
> To unsubscribe send an email to ceph-users-le...@ceph.io
   
   ___
   ceph-users mailing list -- ceph-users@ceph.io
   
   To unsubscribe send an email to ceph-users-le...@ceph.io
   


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 'ceph fs status' no longer works?

[Erich Weiler]

Excellent!  Restarting all the MDS daemons fixed it.  Thank you.

This kinda feels like a bug.

-erich

On 5/2/24 12:44 PM, Bandelow, Gunnar wrote:

Hi Erich,

im not sure about this specific error message, but "ceph fs status" 
sometimes did fail me end of last year/in the beginning of the year.


Restarting ALL mon, mgr AND mds fixed it at the time.

Best regards,
Gunnar


===

Gunnar Bandelow (dipl. phys.)

Universitätsrechenzentrum (URZ)
Universität Greifswald
Felix-Hausdorff-Straße 18
17489 Greifswald
Germany


--- Original Nachricht ---
*Betreff: *[ceph-users] Re: 'ceph fs status' no longer works?
*Von: *"Erich Weiler" mailto:wei...@soe.ucsc.edu>>
*An: *"Eugen Block" mailto:ebl...@nde.ag>>, 
ceph-users@ceph.io 

*Datum: *02-05-2024 21:05



Hi Eugen,

Thanks for the tip!  I just ran:

ceph orch daemon restart mgr.pr-md-01.jemmdf

(my specific mgr instance)

And it restarted my primary mgr daemon, and in the process failed over
to my standby mgr daemon on another server.  That went smoothly.

Unfortunately, I still cannot get 'ceph fs status' to work (on any
node)...

# ceph fs status
Error EINVAL: Traceback (most recent call last):
    File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in
_handle_command
  return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
    File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
  return self.func(mgr, **kwargs)
    File "/usr/share/ceph/mgr/status/module.py", line 109, in
handle_fs_status
  assert metadata
AssertionError

-erich

On 5/2/24 11:07 AM, Eugen Block wrote:
 > Yep, seen this a couple of times during upgrades. I’ll have to
check my
 > notes if I wrote anything down for that. But try a mgr failover
first,
 > that could help.
 >
 > Zitat von Erich Weiler mailto:wei...@soe.ucsc.edu>>:
 >
 >> Hi All,
 >>
 >> For a while now I've been using 'ceph fs status' to show current
MDS
 >> active servers, filesystem status, etc.  I recently took down my
MDS
 >> servers and added RAM to them (one by one, so the filesystem stayed
 >> online).  After doing that with my four MDS servers (I had two
active
 >> and two standby), all looks OK, 'ceph -s' reports HEALTH_OK. 
But when

 >> I do 'ceph fs status' now, I get this:
 >>
 >> # ceph fs status
 >> Error EINVAL: Traceback (most recent call last):
 >>   File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in
_handle_command
 >>     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
 >>   File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
 >>     return self.func(mgr, **kwargs)
 >>   File "/usr/share/ceph/mgr/status/module.py", line 109, in
 >> handle_fs_status
 >>     assert metadata
 >> AssertionError
 >>
 >> This is on ceph 18.2.1 reef.  This is very odd - can anyone
think of a
 >> reason why 'ceph fs status' would stop working after taking each of
 >> the servers down for maintenance?
 >>
 >> The filesystem is online and working just fine however.  This ceph
 >> instance is deployed via the cephadm method on RHEL 9.3, so the
 >> everything is containerized in podman.
 >>
 >> Thanks again,
 >> erich
 >> ___
 >> ceph-users mailing list -- ceph-users@ceph.io

 >> To unsubscribe send an email to ceph-users-le...@ceph.io

 >
 >
 > ___
 > ceph-users mailing list -- ceph-users@ceph.io

 > To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io

To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 'ceph fs status' no longer works?

[Bandelow, Gunnar]

Hi Erich,

im not sure about this specific error message, but "ceph fs status"
sometimes did fail me end of last year/in the beginning of the year. 


Restarting ALL mon, mgr AND mds fixed it at the time.

Best regards,
Gunnar


===


Gunnar Bandelow (dipl. phys.)


Universitätsrechenzentrum (URZ)
Universität Greifswald

Felix-Hausdorff-Straße 18
17489 GreifswaldGermany
 

--- Original Nachricht ---
Betreff: [ceph-users] Re: 'ceph fs status' no longer works?
Von: "Erich Weiler" 
An: "Eugen Block" , ceph-users@ceph.io
Datum: 02-05-2024 21:05






Hi Eugen,

Thanks for the tip!  I just ran:

ceph orch daemon restart mgr.pr-md-01.jemmdf

(my specific mgr instance)

And it restarted my primary mgr daemon, and in the process failed over

to my standby mgr daemon on another server.  That went smoothly.

Unfortunately, I still cannot get 'ceph fs status' to work (on any
node)...

# ceph fs status
Error EINVAL: Traceback (most recent call last):
   File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in
_handle_command
 return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd,
inbuf)
   File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
 return self.func(mgr, **kwargs)
   File "/usr/share/ceph/mgr/status/module.py", line 109, in 
handle_fs_status
 assert metadata
AssertionError

-erich

On 5/2/24 11:07 AM, Eugen Block wrote:
> Yep, seen this a couple of times during upgrades. I’ll have to
check my 
> notes if I wrote anything down for that. But try a mgr failover
first, 
> that could help.
> 
> Zitat von Erich Weiler :
> 
>> Hi All,
>>
>> For a while now I've been using 'ceph fs status' to show current
MDS 
>> active servers, filesystem status, etc.  I recently took down my
MDS 
>> servers and added RAM to them (one by one, so the filesystem stayed

>> online).  After doing that with my four MDS servers (I had two
active 
>> and two standby), all looks OK, 'ceph -s' reports HEALTH_OK.  But
when 
>> I do 'ceph fs status' now, I get this:
>>
>> # ceph fs status
>> Error EINVAL: Traceback (most recent call last):
>>   File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in
_handle_command
>>     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd,
inbuf)
>>   File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
>>     return self.func(mgr, **kwargs)
>>   File "/usr/share/ceph/mgr/status/module.py", line 109, in 
>> handle_fs_status
>>     assert metadata
>> AssertionError
>>
>> This is on ceph 18.2.1 reef.  This is very odd - can anyone think
of a 
>> reason why 'ceph fs status' would stop working after taking each of

>> the servers down for maintenance?
>>
>> The filesystem is online and working just fine however.  This ceph

>> instance is deployed via the cephadm method on RHEL 9.3, so the 
>> everything is containerized in podman.
>>
>> Thanks again,
>> erich
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] NVME node disks maxed out during rebalance after adding to existing cluster

[Szabo, Istvan (Agoda)]

Hi,

I have slow heartbeat in front and back with the extra node added to the 
cluster and this occasionally causing slow ops and failed osd reports.

I'm extending our cluster with +3 relatively differently configured servers 
compared to the original 12.
Our cluster (latest octopus) is an objectstore cluster with 12x identical node 
(8x15.3 TB SSD inside with 4osd on each, 512GB mem, 96 vcore cpu ...) and it 
hosts a 4:2 EC data pool.
The +3 nodes - currently done only 1 and now the 2nd is in progress but have 
the issue during rebalance - have 8x 15.3TB NVME drives with 4x osd on each.

The NVME drive specification:https://i.ibb.co/BVmLKnf/currentnvme.png
[https://i.ibb.co/BVmLKnf/currentnvme.png]


The old server SSD spec: https://i.ibb.co/dkD3VKx/oldssd.png
[https://i.ibb.co/dkD3VKx/oldssd.png]

Iostat on new nvme: https://i.ibb.co/PF0hrVW/iostat.png
[https://i.ibb.co/PF0hrVW/iostat.png]

Rebalance is ongoing with the slowest option like max_backfill/op priority/max 
recovery = 1
But it generates a very huge iowait, seems like the disk not fast enough (but 
why in the previous node didn't have this issue)?
Here is the metrics about the disk which is running the backfill/rebalance now 
(FYI we have 3.2B objects in the cluster):
https://i.ibb.co/LNXCRbj/disks.png
[https://i.ibb.co/LNXCRbj/disks.png]

Wonder what I'm missing or how this can happen?

Here you can see gigantic latencies, failed osds and slow ops:
https://i.ibb.co/Jn0sj9g/laten.png
[https://i.ibb.co/Jn0sj9g/laten.png]
Thank you for your help


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 'ceph fs status' no longer works?

[Erich Weiler]

Hi Eugen,

Thanks for the tip!  I just ran:

ceph orch daemon restart mgr.pr-md-01.jemmdf

(my specific mgr instance)

And it restarted my primary mgr daemon, and in the process failed over 
to my standby mgr daemon on another server.  That went smoothly.


Unfortunately, I still cannot get 'ceph fs status' to work (on any node)...

# ceph fs status
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/status/module.py", line 109, in 
handle_fs_status

assert metadata
AssertionError

-erich

On 5/2/24 11:07 AM, Eugen Block wrote:
Yep, seen this a couple of times during upgrades. I’ll have to check my 
notes if I wrote anything down for that. But try a mgr failover first, 
that could help.


Zitat von Erich Weiler :


Hi All,

For a while now I've been using 'ceph fs status' to show current MDS 
active servers, filesystem status, etc.  I recently took down my MDS 
servers and added RAM to them (one by one, so the filesystem stayed 
online).  After doing that with my four MDS servers (I had two active 
and two standby), all looks OK, 'ceph -s' reports HEALTH_OK.  But when 
I do 'ceph fs status' now, I get this:


# ceph fs status
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/status/module.py", line 109, in 
handle_fs_status

    assert metadata
AssertionError

This is on ceph 18.2.1 reef.  This is very odd - can anyone think of a 
reason why 'ceph fs status' would stop working after taking each of 
the servers down for maintenance?


The filesystem is online and working just fine however.  This ceph 
instance is deployed via the cephadm method on RHEL 9.3, so the 
everything is containerized in podman.


Thanks again,
erich
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Reconstructing an OSD server when the boot OS is corrupted

[Murilo Morais]

Em qui., 2 de mai. de 2024 às 06:20, Matthew Vernon 
escreveu:

> On 24/04/2024 13:43, Bailey Allison wrote:
>
> > A simple ceph-volume lvm activate should get all of the OSDs back up and
> > running once you install the proper packages/restore the ceph config
> > file/etc.,
>
> What's the equivalent procedure in a cephadm-managed cluster?
>

ceph cephadm osd activate 


>
> Thanks,
>
> Matthew
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 'ceph fs status' no longer works?

[Eugen Block]

Yep, seen this a couple of times during upgrades. I’ll have to check  
my notes if I wrote anything down for that. But try a mgr failover  
first, that could help.


Zitat von Erich Weiler :


Hi All,

For a while now I've been using 'ceph fs status' to show current MDS  
active servers, filesystem status, etc.  I recently took down my MDS  
servers and added RAM to them (one by one, so the filesystem stayed  
online).  After doing that with my four MDS servers (I had two  
active and two standby), all looks OK, 'ceph -s' reports HEALTH_OK.   
But when I do 'ceph fs status' now, I get this:


# ceph fs status
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
assert metadata
AssertionError

This is on ceph 18.2.1 reef.  This is very odd - can anyone think of  
a reason why 'ceph fs status' would stop working after taking each  
of the servers down for maintenance?


The filesystem is online and working just fine however.  This ceph  
instance is deployed via the cephadm method on RHEL 9.3, so the  
everything is containerized in podman.


Thanks again,
erich
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 'ceph fs status' no longer works?

[Erich Weiler]

Hi All,

For a while now I've been using 'ceph fs status' to show current MDS 
active servers, filesystem status, etc.  I recently took down my MDS 
servers and added RAM to them (one by one, so the filesystem stayed 
online).  After doing that with my four MDS servers (I had two active 
and two standby), all looks OK, 'ceph -s' reports HEALTH_OK.  But when I 
do 'ceph fs status' now, I get this:


# ceph fs status
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/status/module.py", line 109, in 
handle_fs_status

assert metadata
AssertionError

This is on ceph 18.2.1 reef.  This is very odd - can anyone think of a 
reason why 'ceph fs status' would stop working after taking each of the 
servers down for maintenance?


The filesystem is online and working just fine however.  This ceph 
instance is deployed via the cephadm method on RHEL 9.3, so the 
everything is containerized in podman.


Thanks again,
erich
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: service:mgr [ERROR] "Failed to apply:

[Eugen Block]

Can you please paste the output of the following command?

ceph orch host ls

Zitat von "Roberto Maggi @ Debian" :


Hi you all,

it is a couple of days I'm facing this problem.

Although I already destroyed the cluster a couple of times I  
continuously get these error


I instruct ceph to place 3 daemons

ceph orch apply mgr 3  
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"


but

root@cephstage01:/# ceph -s
  cluster:
    id: 0827effc-302a-45a6-bb97-758d2a9e6835
    health: HEALTH_ERR
    Failed to apply 6 service(s):  
alertmanager,grafana,mgr,mon,node-exporter,prometheus

    failed to probe daemons or devices
    2 mgr modules have failed
    1 mgr modules have recently crashed
    OSD count 0 < osd_pool_default_size 3

  services:
    mon: 2 daemons, quorum cephstage01,cephstagedatanode01 (age 21m)
    mgr: cephstage01.ukwnlo(active, since 5h), standbys:  
cephstagedatanode01.vqbwbz

    osd: 0


root@cephstage01:/# ceph orch ls --service_name mgr --format yaml
service_type: mgr
service_name: mgr
placement:
  hosts:
  - cephstage01:10.20.20.81
  - cephstage02:10.20.20.82
  - cephstage03:10.20.20.83
status:
  created: '2024-05-02T13:52:41.346547Z'
  last_refresh: '2024-05-02T14:06:31.823024Z'
  running: 2
  size: 1
events:
- 2024-05-02T13:51:49.993145Z service:mgr [INFO] "service was created"
- '2024-05-02T13:53:45.566950Z service:mgr [ERROR] "Failed to apply:  
Cannot place
   on cephstage02, cephstage03:  
Unknown hosts"'




and when I check those hosts

root@cephstage01:/# ceph cephadm check-host cephstage03
cephstage03 (None) ok
docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit chrony.service is enabled and running
Hostname "cephstage03" matches what is expected.
Host looks OK

they look fine.


every time I recreate the cluster it gives me the same errors: does  
anybody have any idea?



thanks in advance




























___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Day NYC 2024 Slides

[Laura Flores]

I was notified that my attachment may be stripped out of the email in some
cases.

Here is a link to the presentation in GitHub:
https://github.com/ljflores/ceph_user_dev_monthly_meeting/blob/main/Launch%20of%20Ceph%20User%20Council%20.pdf

Hopefully that works better for some people.

Thanks,
Laura

On Mon, Apr 29, 2024 at 11:27 AM Laura Flores  wrote:

> Attached is a copy of the "Launch of the Ceph User Council" slides.
>
> On Sat, Apr 27, 2024 at 8:12 AM Matt Vandermeulen 
> wrote:
>
>> Hi folks!
>>
>> Thanks for a great Ceph Day event in NYC! I wanted to make sure I posted
>> my slides before I forget (and encourage others to do the same). Feel
>> free to reach out in the Ceph Slack
>> https://ceph.io/en/community/connect/
>>
>> How we Operate Ceph at Scale (DigitalOcean):
>>
>> -
>>
>> https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20How%20we%20Operate%20Ceph%20at%20Scale.pdf
>> -
>>
>> https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20How%20we%20Operate%20Ceph%20at%20Scale.pdf
>>
>> Discards in Ceph (DigitalOcean):
>>
>> -
>>
>> https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20Discards%20Lightning%20Talk.pdf
>> -
>>
>> https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20Discards%20Lightning%20Talk.pdf
>>
>> Matt
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
>
>
>

-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm custom crush location hooks

[Eugen Block]

Thank you very much for the quick response! I will take a look first  
thing tomorrow and try that in a test cluster. But I agree, it would  
be helpful to have a way with cephadm to apply these hooks without  
these workarounds. I'll check if there's a tracker issue for that, and  
create one if necessary.


Thanks!
Eugen

Zitat von Wyll Ingersoll :

I've found the crush location hook script code to be problematic in  
the containerized/cephadm world.


Our workaround is to place the script in a common place on each OSD  
node, such as /etc/crush/crushhook.sh, and then make a link from  
/rootfs -> /, and set the configuration value so that the path to  
the hook script starts with /rootfs.  The container that the OSDs  
run in has access to /rootfs and this hack allows them to all view  
the crush script without having to manually modify unit files.


For example:

  1.
put crushhook script on the host OS in /etc/crush/crushhook.sh
  2.
make a link on the host os:   $ cd /; sudo ln -s / /rootfs
  3.
ceph config set osd crush_location_hook /rootfs/etc/crush/crushhook.sh


The containers see "/rootfs" and will then be able to access your  
script.  Be aware though that if your script requires any sort of  
elevated access, it may fail because the hook runs as ceph:ceph in a  
minimal container so not all functions are available.  I had to add  
lots of debug output and logging in mine (it's rather complicated)  
to figure out what was going on when it was running.


I would love to see the "crush_location_hook" script be something  
that can be stored in the config entirely instead of as a link,  
similar to how the ssl certificates for RGW or the dashboard are  
stored (ceph config-key set ...).   The current situation is not  
ideal.






From: Eugen Block 
Sent: Thursday, May 2, 2024 10:23 AM
To: ceph-users@ceph.io 
Subject: [ceph-users] cephadm custom crush location hooks

Hi,

we've been using custom crush location hooks for some OSDs [1] for
years. Since we moved to cephadm, we always have to manually edit the
unit.run file of those OSDs because the path to the script is not
mapped into the containers. I don't want to define custom location
hooks for all OSDs globally in the OSD spec, even if those are limited
to two hosts only in our case. But I'm not aware of a method to target
only specific OSDs to have some files mapped into the container [2].
Is my assumption correct that we'll have to live with the manual
intervention until we reorganize our osd tree? Or did I miss something?

Thanks!
Eugen

[1]
https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-location-hooks
[2]
https://docs.ceph.com/en/latest/cephadm/services/#mounting-files-with-extra-container-arguments
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm custom crush location hooks

[Wyll Ingersoll]

I've found the crush location hook script code to be problematic in the 
containerized/cephadm world.

Our workaround is to place the script in a common place on each OSD node, such 
as /etc/crush/crushhook.sh, and then make a link from /rootfs -> /, and set the 
configuration value so that the path to the hook script starts with /rootfs.  
The container that the OSDs run in has access to /rootfs and this hack allows 
them to all view the crush script without having to manually modify unit files.

For example:

  1.
put crushhook script on the host OS in /etc/crush/crushhook.sh
  2.
make a link on the host os:   $ cd /; sudo ln -s / /rootfs
  3.
ceph config set osd crush_location_hook /rootfs/etc/crush/crushhook.sh


The containers see "/rootfs" and will then be able to access your script.  Be 
aware though that if your script requires any sort of elevated access, it may 
fail because the hook runs as ceph:ceph in a minimal container so not all 
functions are available.  I had to add lots of debug output and logging in mine 
(it's rather complicated) to figure out what was going on when it was running.

I would love to see the "crush_location_hook" script be something that can be 
stored in the config entirely instead of as a link, similar to how the ssl 
certificates for RGW or the dashboard are stored (ceph config-key set ...).   
The current situation is not ideal.





From: Eugen Block 
Sent: Thursday, May 2, 2024 10:23 AM
To: ceph-users@ceph.io 
Subject: [ceph-users] cephadm custom crush location hooks

Hi,

we've been using custom crush location hooks for some OSDs [1] for
years. Since we moved to cephadm, we always have to manually edit the
unit.run file of those OSDs because the path to the script is not
mapped into the containers. I don't want to define custom location
hooks for all OSDs globally in the OSD spec, even if those are limited
to two hosts only in our case. But I'm not aware of a method to target
only specific OSDs to have some files mapped into the container [2].
Is my assumption correct that we'll have to live with the manual
intervention until we reorganize our osd tree? Or did I miss something?

Thanks!
Eugen

[1]
https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-location-hooks
[2]
https://docs.ceph.com/en/latest/cephadm/services/#mounting-files-with-extra-container-arguments
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephadm custom crush location hooks

[Eugen Block]

Hi,

we've been using custom crush location hooks for some OSDs [1] for  
years. Since we moved to cephadm, we always have to manually edit the  
unit.run file of those OSDs because the path to the script is not  
mapped into the containers. I don't want to define custom location  
hooks for all OSDs globally in the OSD spec, even if those are limited  
to two hosts only in our case. But I'm not aware of a method to target  
only specific OSDs to have some files mapped into the container [2].
Is my assumption correct that we'll have to live with the manual  
intervention until we reorganize our osd tree? Or did I miss something?


Thanks!
Eugen

[1]  
https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-location-hooks
[2]  
https://docs.ceph.com/en/latest/cephadm/services/#mounting-files-with-extra-container-arguments

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] service:mgr [ERROR] "Failed to apply:

[Roberto Maggi @ Debian]

Hi you all,

it is a couple of days I'm facing this problem.

Although I already destroyed the cluster a couple of times I 
continuously get these error


I instruct ceph to place 3 daemons

ceph orch apply mgr 3 
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"


but

root@cephstage01:/# ceph -s
  cluster:
    id: 0827effc-302a-45a6-bb97-758d2a9e6835
    health: HEALTH_ERR
    Failed to apply 6 service(s): 
alertmanager,grafana,mgr,mon,node-exporter,prometheus

    failed to probe daemons or devices
    2 mgr modules have failed
    1 mgr modules have recently crashed
    OSD count 0 < osd_pool_default_size 3

  services:
    mon: 2 daemons, quorum cephstage01,cephstagedatanode01 (age 21m)
    mgr: cephstage01.ukwnlo(active, since 5h), standbys: 
cephstagedatanode01.vqbwbz

    osd: 0


root@cephstage01:/# ceph orch ls --service_name mgr --format yaml
service_type: mgr
service_name: mgr
placement:
  hosts:
  - cephstage01:10.20.20.81
  - cephstage02:10.20.20.82
  - cephstage03:10.20.20.83
status:
  created: '2024-05-02T13:52:41.346547Z'
  last_refresh: '2024-05-02T14:06:31.823024Z'
  running: 2
  size: 1
events:
- 2024-05-02T13:51:49.993145Z service:mgr [INFO] "service was created"
- '2024-05-02T13:53:45.566950Z service:mgr [ERROR] "Failed to apply: 
Cannot place
   on cephstage02, cephstage03: 
Unknown hosts"'




and when I check those hosts

root@cephstage01:/# ceph cephadm check-host cephstage03
cephstage03 (None) ok
docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit chrony.service is enabled and running
Hostname "cephstage03" matches what is expected.
Host looks OK

they look fine.


every time I recreate the cluster it gives me the same errors: does 
anybody have any idea?



thanks in advance




























___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph reef and (slow) backfilling - how to speed it up

[Wesley Dillingham]

In our case it was with a EC pool as well. I believe the PG state was
degraded+recovering / recovery_wait and iirc the PGs just simply sat in the
recovering state without any progress (degraded PG object count did not
decline). A repeer of the PG was attempted but no success there. A restart
of all the OSDs for the given PGs was attempted under mclock. That didnt
work. Switching to wpq for all OSDS in the given PG did resolve the issue.
This was on a 17.2.7 cluster.

Respectfully,

*Wes Dillingham*
LinkedIn 
w...@wesdillingham.com




On Thu, May 2, 2024 at 9:54 AM Sridhar Seshasayee 
wrote:

> >
> > Multiple people -- including me -- have also observed backfill/recovery
> > stop completely for no apparent reason.
> >
> > In some cases poking the lead OSD for a PG with `ceph osd down` restores,
> > in other cases it doesn't.
> >
> > Anecdotally this *may* only happen for EC pools on HDDs but that sample
> > size is small.
> >
> >
> Thanks for the information. We will try and reproduce this locally with EC
> pools and investigate this further.
> I will revert with a tracker for this.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph reef and (slow) backfilling - how to speed it up

[Sridhar Seshasayee]

>
> Multiple people -- including me -- have also observed backfill/recovery
> stop completely for no apparent reason.
>
> In some cases poking the lead OSD for a PG with `ceph osd down` restores,
> in other cases it doesn't.
>
> Anecdotally this *may* only happen for EC pools on HDDs but that sample
> size is small.
>
>
Thanks for the information. We will try and reproduce this locally with EC
pools and investigate this further.
I will revert with a tracker for this.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph reef and (slow) backfilling - how to speed it up

[Anthony D'Atri]

>> For our customers we are still disabling mclock and using wpq. Might be
>> worth trying.
>> 
>> 
> Could you please elaborate a bit on the issue(s) preventing the
> use of mClock. Is this specific to only the slow backfill rate and/or other
> issue?
> 
> This feedback would help prioritize the improvements in those areas.

Multiple people -- including me -- have also observed backfill/recovery stop 
completely for no apparent reason.

In some cases poking the lead OSD for a PG with `ceph osd down` restores, in 
other cases it doesn't.

Anecdotally this *may* only happen for EC pools on HDDs but that sample size is 
small.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: After dockerized ceph cluster to Pacific, the fsid changed in the output of 'ceph -s'

[Eugen Block]

Hi,

did you maybe have some test clusters leftovers on the hosts so  
cephadm might have picked up the wrong FSID?
Does that mean that you adopted all daemons and only afterwards looked  
into ceph -s? I would have adopted the first daemon and checked  
immediately if everything still was as expected.

What you can't do is injecting a different FSID directly into the cluster:

quincy-1:~ # ceph config set mon fsid 
Error EINVAL: fsid is special and cannot be stored by the mon

I assume you would have to update the monmap, but I've never tried  
that. Something like:


- stop one mon
- cephadm shell --name mon.
- ceph-monstore-tool /var/lib/ceph/mon/ceph-host1/ get monmap -- --out  
monmap.bin (as backup)
- monmaptool --create --fsid  --addv host1  
[v2:IP:3300/0,v1:IP:6789/0] --addv host2 [v2:IP:3300/0,v1:IP:6789/0]  
--addv host3 [v2:IP:3300/0,v1:IP:6789/0] --set-min-mon-release 17  
monmap.new

- ceph-mon -i host1 --inject-monmap monmap.new

This is just an example from a quincy test cluster, I have no idea if  
that will work if the other two MONs still have the other FSID. If  
that works, I assume the orchestrator would reconfigure the rest  
automatically, but again, I don't know if that will work. If you  
decide to try this approach, let us know how it went.


Regards,
Eugen

Zitat von wjsherry...@outlook.com:


Hello,
I had a problem after I finished the 'cephadm adopt' from services  
to docker containers for mon and mgr. The fsid of `ceph -s` is not  
the same as the /etc/ceph/ceph.conf. The ceph.conf is correct, but  
`ceph -s` is incorrect. I followed the  
https://docs.ceph.com/en/quincy/cephadm/adoption/

```
2024-04-25T19:49:17.460652+ mgr.cloud-lab-test-mon01 (mgr.65113)  
109 : cephadm [ERR] cephadm exited with an error code: 1,  
stderr:ERROR: fsid does not match ceph.conf

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in  
_remote_connection

yield (conn, connr)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an  
error code: 1, stderr:ERROR: fsid does not match ceph.conf

#ceph health detail shows the warning:
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
```

Does anyone else have any ideas?

Thanks,
Sherry
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph reef and (slow) backfilling - how to speed it up

[Mark Nelson]

Hi Sridhar,

(Very!) Slow backfill was one issue, but if I recall we hit a case where 
backfill wasn't completing at all until we reverted to WPQ.  I was 
getting hammered with other stuff at the time so I don't quite remember 
the details, but Dan might.  I think this was in Quincy after the more 
recent updates landed in 17.2.6+ though.


Mark

On 5/2/24 00:05, Sridhar Seshasayee wrote:

Hi Mark,

On Thu, May 2, 2024 at 3:18 AM Mark Nelson  wrote:


For our customers we are still disabling mclock and using wpq. Might be
worth trying.



Could you please elaborate a bit on the issue(s) preventing the
use of mClock. Is this specific to only the slow backfill rate and/or other
issue?

This feedback would help prioritize the improvements in those areas.

Thanks,
-Sridhar
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Best Regards,
Mark Nelson
Head of Research and Development

Clyso GmbH
p: +49 89 21552391 12 | a: Minnesota, USA
w: https://clyso.com | e: mark.nel...@clyso.com

We are hiring: https://www.clyso.com/jobs/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Unable to add new OSDs

[Eugen Block]

Hi,

is the cluster healthy? Sometimes a degraded state prevents the  
orchestrator from doing its work. Then I would fail the mgr (ceph mgr  
fail), this seems to be necessary lots of times. Then keep an eye on  
the active mgr log as well as the cephadm.log locally on the host  
where the OSDs need to be created.


Regards,
Eugen

Zitat von c...@mikesoffice.com:

I'm trying to add a new storage host into a Ceph cluster (quincy  
17.2.6). The machine has boot drives, one free SSD and 10 HDDs. The  
plan is to have each HDD be an OSD with a DB on a equal size lvm of  
the SDD. This machine is newer but otherwise similar to other  
machines already in the cluster that are setup and running the same  
way. But I've been unable to add OSDs and unable to figure out why,  
or fix it. I have some experience, but I'm not an expert and could  
be missing something obvious. If anyone has any suggestions, I would  
appreciate it.


I've tried to add OSDs a couple different ways.

Via the dashboard, this has worked fine for previous machines. And  
it appears to succeed and gives no errors that I can find looking in  
/var/log/ceph and dashboard logs. The OSDs are never created. In  
fact, the drives still show up as available in Physical Disks and I  
can do the same creation procedure repeatedly.


I've tried creating it in cephadm shell with the following, which  
has also worked in the past:
ceph orch daemon add osd  
stor04.fqdn:data_devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi,/dev/sdj,/dev/sdk,db_devices=/dev/sda,osds_per_device=1
The command just hangs. Again I wasn't able to find any obvious  
errors. Although this one did seem to cause some slow op errors from  
the monitors that required restarting a monitor. And it could cause  
errors with the dashboard locking up and having to restart the  
manager as well.


And I've tried setting 'ceph orch apply osd --all-available-devices  
--unmanaged=false' to let Ceph automatically add the drives. In the  
past, this would cause Ceph to automatically add the drives as OSDs  
but without having associated DBs on the SSD. The SSD would just be  
another OSD. This time it appears to have no affect and similar to  
the above, I wasn't able to find any obvious error feedback.


-Mike
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Reconstructing an OSD server when the boot OS is corrupted

[Matthew Vernon]

On 24/04/2024 13:43, Bailey Allison wrote:


A simple ceph-volume lvm activate should get all of the OSDs back up and
running once you install the proper packages/restore the ceph config
file/etc.,


What's the equivalent procedure in a cephadm-managed cluster?

Thanks,

Matthew
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RBD Mirroring with Journaling and Snapshot mechanism

[V A Prabha]

Dear Eugen,
We have a scenario of DC and DR replication, and planned to explore RBD
mirroring with both Journaling and Snapshot mechanism.
I have a 5 TB storage at Primary DC and 5 TB storage at DR site with 2 different
ceph clusters configured.

Please clarify the following queries

1. With One way mirroring, the failover works fine in both journaling and
snapshot mechanism and we are able to promote the workload from DR site. How
does Failback work? We wanted to move the contents from DR to DC but it fails.
In journaling mechanism, it deletes the entire volume and recreates it afresh
which does not solve our problem.
2. How does incremental replication work from DR to DC?
3. Does Two-way mirroring help this situation. According to me, in this method,
it is for 2 different clouds with 2 different storages and replicating both the
clouds workloads? Does Failback work in this scenario ?
Please help us / guide us to deploy this solution

Regards
V.A.Prabha


Thanks & Regards,
Ms V A Prabha / श्रीमती प्रभा वी ए
Joint Director / संयुक्त निदेशक
Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास
केन्द्र(सी-डैक)
Tidel Park”, 8th Floor, “D” Block, (North ) / “टाइडल पार्क”,8वीं मंजिल,
“डी” ब्लॉक, (उत्तर और दक्षिण)
No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई
Taramani / तारामणि
Chennai / चेन्नई – 600113
Ph.No.:044-22542226/27
Fax No.: 044-22542294

[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Unable to add new OSDs

[Bogdan Adrian Velica]

Hi,

I would suggest wiping the disks first with "wipefs -af /dev/_your_disk" or
"sgdisk --zap-all /dev/your_disk" and try again. Try only one disk first.
Is the host visible by running the command: "ceph orch host ls". Is the
FQDN name correct? If so, does the following command return any errors?
"ceph cephadm check-host **"
I don't think this is the case but the disks are visible on the new host
correct? Use "lsblk" or "fdisk -l" commands.

Thank you,
Bogdan Velica
croit.io

On Thu, May 2, 2024 at 7:04 AM  wrote:

> I'm trying to add a new storage host into a Ceph cluster (quincy 17.2.6).
> The machine has boot drives, one free SSD and 10 HDDs. The plan is to have
> each HDD be an OSD with a DB on a equal size lvm of the SDD. This machine
> is newer but otherwise similar to other machines already in the cluster
> that are setup and running the same way. But I've been unable to add OSDs
> and unable to figure out why, or fix it. I have some experience, but I'm
> not an expert and could be missing something obvious. If anyone has any
> suggestions, I would appreciate it.
>
> I've tried to add OSDs a couple different ways.
>
> Via the dashboard, this has worked fine for previous machines. And it
> appears to succeed and gives no errors that I can find looking in
> /var/log/ceph and dashboard logs. The OSDs are never created. In fact, the
> drives still show up as available in Physical Disks and I can do the same
> creation procedure repeatedly.
>
> I've tried creating it in cephadm shell with the following, which has also
> worked in the past:
> ceph orch daemon add osd
> stor04.fqdn:data_devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi,/dev/sdj,/dev/sdk,db_devices=/dev/sda,osds_per_device=1
> The command just hangs. Again I wasn't able to find any obvious errors.
> Although this one did seem to cause some slow op errors from the monitors
> that required restarting a monitor. And it could cause errors with the
> dashboard locking up and having to restart the manager as well.
>
> And I've tried setting 'ceph orch apply osd --all-available-devices
> --unmanaged=false' to let Ceph automatically add the drives. In the past,
> this would cause Ceph to automatically add the drives as OSDs but without
> having associated DBs on the SSD. The SSD would just be another OSD. This
> time it appears to have no affect and similar to the above, I wasn't able
> to find any obvious error feedback.
>
> -Mike
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph client cluster compatibility

[Konstantin Shalygin]

Hi,

Yes, like it always do


k
Sent from my iPhone

> On 2 May 2024, at 07:09, Nima AbolhassanBeigi 
>  wrote:
> 
> We are trying to upgrade our OS version from ubuntu 18.04 to ubuntu 22.04.
> Our ceph cluster version is 16.2.13 (pacific).
> 
> The problem is that the ubuntu packages for the ceph pacific release will
> not be supported for ubuntu 22.04. We were wondering if the ceph client
> (version 18.2, reef) on ubuntu 22.04 can work with lower version clusters.
> 
> Thanks in advance
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io