[ceph-users] Re: Pacific unable to configure NFS-Ganesha

[Robert Sander]

Hi,

I forgot to mention that CephFS is enabled and working.

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

https://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 93818 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: bug in ceph-volume create

[Philip Brown]

O
sigh.
Thank you very much.
That actually makes sense, and isnt so bad after all.
Makes me surprised why I got no answers on my prior related question, couple 
weeks ago, about what the proper way to replace an HDD in a failed hybrid OSD.

At least I know now.

You guys might consider a feature request of doing some kind of check on long 
device path names getting passed in, to see if the util should complain to the 
user, "hey use the other syntax".





- Original Message -
From: "Jeff Bailey" 
To: "ceph-users" 
Sent: Monday, April 5, 2021 1:00:18 PM
Subject: [ceph-users] Re: bug in ceph-volume create

On 4/5/2021 3:49 PM, Philip Brown wrote:
>
> As soon as you have an HDD fail... you will need to recreate the OSD.. and 
> you are then stuck. Because you cant use batch mode for it...
> and you cant do it more granularly, with
>
>ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdg 
> --block.db /dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here


This isn't a bug.  You're specifying the LV incorrectly.  Just use


--block.db ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd


without the /dev at the front. 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs: Migrating Data to a new Data Pool

[Oliver Freyermuth]

Am 05.04.21 um 21:27 schrieb Peter Woodman:
yeah, but you don't want to have those reference objects in an EC pool, 

that's iiuc been explicitly disallowed in newer versions, as it's a performance 
suck. so leaving them in the replicated pool is good :)

I know, but that's quite workload-dependent. We actually fare quite well with 
our existing EC-only data pool on HDDs,
keeping only the metadata on a small replicated NVMe pool. If most of your 
workloads are write-once, read-many, and of that, most are streaming reads,
this may be exactly down your alley ;-).
IIUC, newer versions disallow this only to clarify it might not be what you 
want performance-wise,
but they still allow to override this if you know it is what you want.

So in our case, indeed an EC-to-EC-migration would be what we'd look at (once 
we add more and more servers).
This could either be solved by "EC profile migration" (if it was possible),
or, more generally, by adding the possibility to migrate the primary data 
pool of an existing FS.


But as I understand it, for now, none of that is possible just yet, and the only 
"big hammer" would be to create a new FS and copy things over.
CephFS mirroring (of snapshots) in Pacific may make this easier by allowing to 
reduce any actual downtime for users, but any other solution would be much 
appreciated.

Cheers,
Oliver



On Mon, Apr 5, 2021 at 2:55 PM Oliver Freyermuth mailto:freyerm...@physik.uni-bonn.de>> wrote:

Hi,

that really looks like a useful tool, thanks for mentioning this on 

the list :-).


However, I'd also love to learn about a different way ­— 

as documentation states:

   "You may notice that object counts in your primary data pool (the one 
passed to fs new) continue to increase, even if files are being created in
the pool you added."
https://docs.ceph.com/en/latest/cephfs/file-layouts/ 

So I think while this will migrate the bulk of the data, it will never be
a full migration the way CephFS seems to be implemented.
Especially for growing EC clusters, it would be helpful to be able to 
migrate to a different, more space-efficient EC profile as the number of hosts 
increases.

We're not at this point yet, but one day we'll surely be. Right now, the only 
"complete migration" approach seems to be to create a new FS, and migrate 
things over...
Am I right?

Cheers,
         Oliver


Am 05.04.21 um 19:22 schrieb Peter Woodman:
 > hi, i made a tool to do this. it’s rough around the edges and has some
 > known bugs with symlinks as parent paths but it checks all file layouts
to
 > see if they match the directory layout they’re in, and if not, makes them
 > so by copying and replacing. so to ‘migrate’ set 

your directory layouts and

 > then run this tool to move everything to the right places.
 >
 > i’m unaware of another way of doing this so if there is someone
tell me!
 >
 > https://git.sr.ht/~pjjw/cephfs-layout-tool 

 >
 > On Sun, Apr 4, 2021 at 5:43 PM mailto:c...@fionera.de>> wrote:
 >
 >> Hello everyone,
 >>
 >> I currently have a CephFS running with about 60TB of Data. I created it
 >> with a replicated pool as default pool, an erasure coded one as
 >> additional data pool like it is described in the docs. Now I want to
 >> migrate the data from the replicated pool, to the new erasure coded one.
 >> I couldn't find any docs and was wondering if its even possible
 >> currently.
 >>
 >> Thank you very much,
 >> Fionera
 >> ___
 >> ceph-users mailing list -- ceph-users@ceph.io 

 >> To unsubscribe send an email to ceph-users-le...@ceph.io 

 >>
 > ___
 > ceph-users mailing list -- ceph-users@ceph.io 
 > To unsubscribe send an email to ceph-users-le...@ceph.io 

 >





smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: bug in ceph-volume create

[Jeff Bailey]

On 4/5/2021 3:49 PM, Philip Brown wrote:

I would file this as a potential bug.. but it takes too long to get approved, 
and tracker.ceph.com doesnt have straightfoward google signin enabled :-/


I believe that with the new lvm mandate, ceph-volume should not be complaining about 
"missing PARTUUID".
This is stopping me from using my system.

Details on how to recreate:

1. have a system with 1 SSD and multiple HDDS
2. create a buncha OSDs with your preferred frontend, which will eventualy come 
down to

ceph-volume lvm batch --bluestore /dev/ssddevice  /dev/sdA ... /dev/sdX

THIS will work great. batch mode will appropriately carve up the SSD device 
into multiple LVMs, and allocate one of them to be a DB device for each of the 
HDDs.

3. try to repair/replace an HDD


As soon as you have an HDD fail... you will need to recreate the OSD.. and you 
are then stuck. Because you cant use batch mode for it...
and you cant do it more granularly, with

   ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdg --block.db 
/dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here



This isn't a bug.  You're specifying the LV incorrectly.  Just use


--block.db ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd


without the /dev at the front.  The /dev path gets treated like a normal 
block device.





because ceph-volume will complain that,

   blkid could not detect a PARTUUID for device: 
/dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here


but the lvm IS NOT SUPPOSED TO HAVE A PARTUUID.
Which is provable first all by the fact that it isnt a partition. But secondly, 
that none of the other block-db LVMs it created on the SSD in batch mode, have 
an partuuid either!!

So kindly quit checking for something that isnt supposed to be there in the 
first place?!


(This is a bug all the way back in nautilus, through latest, I believe)




--
Philip Brown| Sr. Linux System Administrator | Medata, Inc.
5 Peters Canyon Rd Suite 250
Irvine CA 92606
Office 714.918.1310| Fax 714.918.1325
pbr...@medata.com| www.medata.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] bug in ceph-volume create

[Philip Brown]

I would file this as a potential bug.. but it takes too long to get approved, 
and tracker.ceph.com doesnt have straightfoward google signin enabled :-/


I believe that with the new lvm mandate, ceph-volume should not be complaining 
about "missing PARTUUID".
This is stopping me from using my system.

Details on how to recreate:

1. have a system with 1 SSD and multiple HDDS
2. create a buncha OSDs with your preferred frontend, which will eventualy come 
down to

   ceph-volume lvm batch --bluestore /dev/ssddevice  /dev/sdA ... /dev/sdX

THIS will work great. batch mode will appropriately carve up the SSD device 
into multiple LVMs, and allocate one of them to be a DB device for each of the 
HDDs.

3. try to repair/replace an HDD


As soon as you have an HDD fail... you will need to recreate the OSD.. and you 
are then stuck. Because you cant use batch mode for it...
and you cant do it more granularly, with

  ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdg --block.db 
/dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here

because ceph-volume will complain that,

  blkid could not detect a PARTUUID for device: 
/dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here


but the lvm IS NOT SUPPOSED TO HAVE A PARTUUID.
Which is provable first all by the fact that it isnt a partition. But secondly, 
that none of the other block-db LVMs it created on the SSD in batch mode, have 
an partuuid either!!

So kindly quit checking for something that isnt supposed to be there in the 
first place?!


(This is a bug all the way back in nautilus, through latest, I believe)




--
Philip Brown| Sr. Linux System Administrator | Medata, Inc. 
5 Peters Canyon Rd Suite 250 
Irvine CA 92606 
Office 714.918.1310| Fax 714.918.1325 
pbr...@medata.com| www.medata.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs: Migrating Data to a new Data Pool

[Peter Woodman]

yeah, but you don't want to have those reference objects in an EC pool,
that's iiuc been explicitly disallowed in newer versions, as it's a
performance suck. so leaving them in the replicated pool is good :)

On Mon, Apr 5, 2021 at 2:55 PM Oliver Freyermuth <
freyerm...@physik.uni-bonn.de> wrote:

> Hi,
>
> that really looks like a useful tool, thanks for mentioning this on the
> list :-).
>
> However, I'd also love to learn about a different way ­— as documentation
> states:
>   "You may notice that object counts in your primary data pool (the one
> passed to fs new) continue to increase, even if files are being created in
> the pool you added."
>   https://docs.ceph.com/en/latest/cephfs/file-layouts/
> So I think while this will migrate the bulk of the data, it will never be
> a full migration the way CephFS seems to be implemented.
> Especially for growing EC clusters, it would be helpful to be able to
> migrate to a different, more space-efficient EC profile as the number of
> hosts increases.
>
> We're not at this point yet, but one day we'll surely be. Right now, the
> only "complete migration" approach seems to be to create a new FS, and
> migrate things over...
> Am I right?
>
> Cheers,
> Oliver
>
>
> Am 05.04.21 um 19:22 schrieb Peter Woodman:
> > hi, i made a tool to do this. it’s rough around the edges and has some
> > known bugs with symlinks as parent paths but it checks all file layouts
> to
> > see if they match the directory layout they’re in, and if not, makes them
> > so by copying and replacing. so to ‘migrate’ set your directory layouts
> and
> > then run this tool to move everything to the right places.
> >
> > i’m unaware of another way of doing this so if there is someone
> tell me!
> >
> > https://git.sr.ht/~pjjw/cephfs-layout-tool
> >
> > On Sun, Apr 4, 2021 at 5:43 PM  wrote:
> >
> >> Hello everyone,
> >>
> >> I currently have a CephFS running with about 60TB of Data. I created it
> >> with a replicated pool as default pool, an erasure coded one as
> >> additional data pool like it is described in the docs. Now I want to
> >> migrate the data from the replicated pool, to the new erasure coded one.
> >> I couldn't find any docs and was wondering if its even possible
> >> currently.
> >>
> >> Thank you very much,
> >> Fionera
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific unable to configure NFS-Ganesha

[Daniel Gryniewicz]

In order to enable NFS via Ganesha, you will need either an RGW or a 
CephFS.  Within the context of a Ceph deployment, Ganesha cannot export 
anything it's own, it just exports either RGW or CephFS.


Daniel

On 4/5/21 1:43 PM, Robert Sander wrote:

Hi,

I have a test cluster now running on Pacific with the cephadm
orchestrator and upstream container images.

In the Dashboard on the services tab I created a new service for NFS.
The containers got deployed.

But when I go to the NFS tab and try to create a new NFS share the
Dashboard only returns a 500 error:

Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.146+ 
7f64468d1700  0 [dashboard ERROR exception] Internal Server Error
Apr 05 19:38:49 ceph01 bash[35064]: Traceback (most recent call last):
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in 
dashboard_exception_handler
Apr 05 19:38:49 ceph01 bash[35064]: return handler(*args, **kwargs)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
Apr 05 19:38:49 ceph01 bash[35064]: return self.callable(*self.args, 
**self.kwargs)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner
Apr 05 19:38:49 ceph01 bash[35064]: ret = func(*args, **kwargs)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/controllers/nfsganesha.py", line 265, in fsals
Apr 05 19:38:49 ceph01 bash[35064]: return Ganesha.fsals_available()
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/ganesha.py", line 154, in 
fsals_available
Apr 05 19:38:49 ceph01 bash[35064]: if 
RgwClient.admin_instance().is_service_online() and \
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 301, in 
admin_instance
Apr 05 19:38:49 ceph01 bash[35064]: return 
RgwClient.instance(daemon_name=daemon_name)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 241, in instance
Apr 05 19:38:49 ceph01 bash[35064]: RgwClient._daemons = _get_daemons()
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 53, in _get_daemons
Apr 05 19:38:49 ceph01 bash[35064]: raise NoRgwDaemonsException
Apr 05 19:38:49 ceph01 bash[35064]: 
dashboard.services.rgw_client.NoRgwDaemonsException: No RGW service is running.
Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+ 
7f64468d1700  0 [dashboard ERROR request] [:::10.0.44.42:39898] [GET] [500] 
[0.030s] [admin] [513.0B] /ui-api/nfs-ganesha/fsals
Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+ 7f64468d1700  0 [dashboard ERROR request] [b'{"status": 
"500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from 
fulfilling the request.", "request_id": "e89b8519-352f-4e44-a364-6e6faf9dc533"}

']

I have no radosgateways in that cluster (currently). There are the pools
for radosgw (.rgw.root etc) but no running instance.

Regards


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs: Migrating Data to a new Data Pool

[Oliver Freyermuth]

Hi,

that really looks like a useful tool, thanks for mentioning this on the list 
:-).

However, I'd also love to learn about a different way ­— as documentation 
states:
 "You may notice that object counts in your primary data pool (the one passed to fs new) continue to increase, even if files are being created in 
the pool you added."

 https://docs.ceph.com/en/latest/cephfs/file-layouts/
So I think while this will migrate the bulk of the data, it will never be 
a full migration the way CephFS seems to be implemented.

Especially for growing EC clusters, it would be helpful to be able to migrate 
to a different, more space-efficient EC profile as the number of hosts 
increases.

We're not at this point yet, but one day we'll surely be. Right now, the only 
"complete migration" approach seems to be to create a new FS, and migrate 
things over...
Am I right?

Cheers,
Oliver


Am 05.04.21 um 19:22 schrieb Peter Woodman:

hi, i made a tool to do this. it’s rough around the edges and has some
known bugs with symlinks as parent paths but it checks all file layouts 

to

see if they match the directory layout they’re in, and if not, makes them
so by copying and replacing. so to ‘migrate’ set your directory layouts and
then run this tool to move everything to the right places.

i’m unaware of another way of doing this so if there is someone 

tell me!


https://git.sr.ht/~pjjw/cephfs-layout-tool

On Sun, Apr 4, 2021 at 5:43 PM  wrote:


Hello everyone,

I currently have a CephFS running with about 60TB of Data. I created it
with a replicated pool as default pool, an erasure coded one as
additional data pool like it is described in the docs. Now I want to
migrate the data from the replicated pool, to the new erasure coded one.
I couldn't find any docs and was wondering if its even possible
currently.

Thank you very much,
Fionera
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io





smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: understanding orchestration and cephadm

[Gary Molenkamp]

Thanks Sage,

I opted to move to an explicit placement map of candidate hostnames and 
a replica count rather than using labels.  This is a testing cluster of 
VMs to experiment before updating the production system.


The only reason I was starting with 2 on the test cluster is that my 
production ceph has a few mons that are custom built docker containers 
that will give some grief in the adoption process.  My plan is to remove 
those mons before converting to cephadm.   On further thought, it will 
probably be safer to add a few normal mons to the OSD servers before 
adopting to cephadm, to ensure I have more than 3 before the conversion.


I like the work done with cephadm and orchestration as compared to the 
older ceph-deploy.  I do have a few questions/challenges on OSD 
deployment that I will put in a new post.


Cheers.
Gary



On 2021-03-31 5:00 p.m., Sage Weil wrote:

Hi Gary,

It looks like everything you did is fine.  I think the "problem" is
that cephadm has/had some logic that tried to leave users with an odd
number of monitors.  I'm pretty sure this is why two of them were
removed.

This code has been removed in pacific, and should probably be
backported to octopus.

There is nothing wrong with an even number of mons.  The only number
you might want to avoid is 2 because a failure of either monitor will
cause the cluster to lose quorum and become unavailable (quorum
requires > N/2, which in a 2-mon case means both mons).  As far as
availability goes that is probably not ideal, but as far as durability
goes, it's extremely useful to have a duplicate copy of the mon data
so that losing a single disk doesn't destroy the cluster metadata (and
require a complicated recovery process).

In any case, generally speaking, nobody should worry about having an
even number of monitors.  Focus instead of getting >2 so you can
tolerate at least one mon failure and keep the cluster running.


On Wed, Mar 31, 2021 at 10:14 AM Gary Molenkamp  wrote:

A nautilus cluster with two mons (I know this is not correct for
quorum), a mgr, and a handful of osds.  I went though the adoption

Any number of monitors is correct.  Less than 3 is not recommended.

sage


--
Gary Molenkamp  Computer Science/Science Technology Services
Systems Administrator   University of Western Ontario
molen...@uwo.ca http://www.csd.uwo.ca
(519) 661-2111 x86882   (519) 661-3566
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW failed to start after upgrade to pacific

[Robert Sander]

Hi,

Am 04.04.21 um 15:22 schrieb 胡 玮文:

> bash[9823]: debug 2021-04-04T13:01:04.995+ 7ff80f172440 -1 static int 
> rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, 
> std::unique_ptr*, optional_yield, 
> std::optional, 
> std::optional >, bool, uint64_t, uint64_t):925 
> create_meta failed: r=-5
> bash[9823]: debug 2021-04-04T13:01:04.995+ 7ff80f172440 -1 int 
> RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, 
> librados::v14_2_0::Rados*): Error when starting backend: Input/output error
> bash[9823]: debug 2021-04-04T13:01:04.995+ 7ff80f172440  0 ERROR: failed 
> to start datalog_rados service ((5) Input/output error
> bash[9823]: debug 2021-04-04T13:01:04.995+ 7ff80f172440  0 ERROR: failed 
> to init services (ret=(5) Input/output error)

I see the same issues on an upgraded cluster.

Regards
-- 
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Pacific unable to configure NFS-Ganesha

[Robert Sander]

Hi,

I have a test cluster now running on Pacific with the cephadm
orchestrator and upstream container images.

In the Dashboard on the services tab I created a new service for NFS.
The containers got deployed.

But when I go to the NFS tab and try to create a new NFS share the
Dashboard only returns a 500 error:

Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.146+ 
7f64468d1700  0 [dashboard ERROR exception] Internal Server Error
Apr 05 19:38:49 ceph01 bash[35064]: Traceback (most recent call last):
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in 
dashboard_exception_handler
Apr 05 19:38:49 ceph01 bash[35064]: return handler(*args, **kwargs)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
Apr 05 19:38:49 ceph01 bash[35064]: return self.callable(*self.args, 
**self.kwargs)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner
Apr 05 19:38:49 ceph01 bash[35064]: ret = func(*args, **kwargs)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/controllers/nfsganesha.py", line 265, in fsals
Apr 05 19:38:49 ceph01 bash[35064]: return Ganesha.fsals_available()
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/ganesha.py", line 154, in 
fsals_available
Apr 05 19:38:49 ceph01 bash[35064]: if 
RgwClient.admin_instance().is_service_online() and \
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 301, in 
admin_instance
Apr 05 19:38:49 ceph01 bash[35064]: return 
RgwClient.instance(daemon_name=daemon_name)
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 241, in instance
Apr 05 19:38:49 ceph01 bash[35064]: RgwClient._daemons = _get_daemons()
Apr 05 19:38:49 ceph01 bash[35064]:   File 
"/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 53, in _get_daemons
Apr 05 19:38:49 ceph01 bash[35064]: raise NoRgwDaemonsException
Apr 05 19:38:49 ceph01 bash[35064]: 
dashboard.services.rgw_client.NoRgwDaemonsException: No RGW service is running.
Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+ 
7f64468d1700  0 [dashboard ERROR request] [:::10.0.44.42:39898] [GET] [500] 
[0.030s] [admin] [513.0B] /ui-api/nfs-ganesha/fsals
Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+ 
7f64468d1700  0 [dashboard ERROR request] [b'{"status": "500 Internal Server 
Error", "detail": "The server encountered an unexpected condition which 
prevented it from fulfilling the request.", "request_id": 
"e89b8519-352f-4e44-a364-6e6faf9dc533"} 



   ']

I have no radosgateways in that cluster (currently). There are the pools
for radosgw (.rgw.root etc) but no running instance.

Regards
-- 
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG: 
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] which is definitive: /var/lib/ceph symlinks or ceph-volume?

[Philip Brown]

I am in a situation where I see conflicting information.
On the one hand, 
ls -l /var/lib/ceph/osd/ceph-7 
shows a symlink for block device, but no block.db

On the other hand,

ceph-volume lvm list
claims that there is a separate db device registered for osd 7

how can I know which one is correct?

(This is currently ceph nautilus)





--
Philip Brown| Sr. Linux System Administrator | Medata, Inc. 
5 Peters Canyon Rd Suite 250 
Irvine CA 92606 
Office 714.918.1310| Fax 714.918.1325 
pbr...@medata.com| www.medata.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs: Migrating Data to a new Data Pool

[Peter Woodman]

hi, i made a tool to do this. it’s rough around the edges and has some
known bugs with symlinks as parent paths but it checks all file layouts to
see if they match the directory layout they’re in, and if not, makes them
so by copying and replacing. so to ‘migrate’ set your directory layouts and
then run this tool to move everything to the right places.

i’m unaware of another way of doing this so if there is someone tell me!

https://git.sr.ht/~pjjw/cephfs-layout-tool

On Sun, Apr 4, 2021 at 5:43 PM  wrote:

> Hello everyone,
>
> I currently have a CephFS running with about 60TB of Data. I created it
> with a replicated pool as default pool, an erasure coded one as
> additional data pool like it is described in the docs. Now I want to
> migrate the data from the replicated pool, to the new erasure coded one.
> I couldn't find any docs and was wondering if its even possible
> currently.
>
> Thank you very much,
> Fionera
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW S3 user.rgw.olh.pending - Can not overwrite on 0 byte objects rgw sync leftovers.

[by morphin]

Hello. I was have one-way multisite S3 cluster and we've seen issues
with rgw-sync due to sharding problems and I've stopped the multisite
sync. This is not the topic just a knowledge about my story.
I have some leftover 0 byte objects in destination and I'm trying to
overwrite them with Rclone "path to path". But somehow I can not
overwrite these objects. If I delete with rclone or rados rm and do
rclone copy again, I got the result below. Rclone gives error but the
object is created again "0 byte" with pending attrs.  Why is this
happening?
I think somehow I need to clean these objects and copy from source
again but how?

What is "user.rgw.olh.pending" ?


[root@SRV1]# radosgw-admin --id radosgw.prod1 object stat
--bucket=mybucket
--object=images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
{
"name": "images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f",
"size": 0,
"tag": "713li30rvcrjfwhctx894mj7vf1wa1a8",
"attrs": {
"user.rgw.manifest": "",
"user.rgw.olh.idtag": "v1m9jy4cjck38ptel09qebsbb10pe2af",
"user.rgw.olh.info": "\u0001\u0001�",
"user.rgw.olh.pending.606b04728gs23ecq11b3i3l1":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.606b0472bfhdzxeb9wesd8t7":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.606b0472fv06t1dob3vmo4da":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.606b0472lql6c9o88rt211r9":
"\u0001\u0001\u0008",
"user.rgw.olh.ver": ""
}
}
[root@SRV1]# rados listxattr -p prod.rgw.buckets.data
c106b26b-xxx--xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
user.rgw.idtag
user.rgw.olh.idtag
user.rgw.olh.info
user.rgw.olh.ver
[root@SRV1]# rados -p prod.rgw.buckets.data stat
c106b26b-xxx--xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
prod.rgw.buckets.data/c106b26b-xxx--xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
mtime 2021-04-05 17:10:55.00, size 0
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: "unable to find any IP address in networks"

[Stephen Smith6]

Surprisingly enough - I figured this out moments after sending this. Setting public_network = 0.0.0.0/0 seems to work.
 
- Original message -From: "Stephen Smith6" To: ceph-users@ceph.ioCc:Subject: [EXTERNAL] [ceph-users] "unable to find any IP address in networks"Date: Mon, Apr 5, 2021 9:22 AM  
Hey folks - I have a unique networking scenario I'm trying to understand. I'm using cephadm to deploy an Octopus based cluster and I'm trying to add monitors however when running "ceph orch apply mon " I'm seeing the following error in my cephadm.log on the node I'm trying to make a monitor: "unable to find any IP address in networks '10.12.0.0/16' interfaces ''". There is an interface with an address in the 10.12.0.0/16 CIDR however it has a different subnet mask (We're using ECMP routing). I've tried pre-setting public_bind_addr, public_addr, and public_network_interface before running "ceph orch apply mon " all to no avail. Any thoughts on how to work around / address this check using cephadm? 

___ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-le...@ceph.io
 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] "unable to find any IP address in networks"

[Stephen Smith6]

Hey folks - I have a unique networking scenario I'm trying to understand. I'm using cephadm to deploy an Octopus based cluster and I'm trying to add monitors however when running "ceph orch apply mon " I'm seeing the following error in my cephadm.log on the node I'm trying to make a monitor: "unable to find any IP address in networks '10.12.0.0/16' interfaces ''". There is an interface with an address in the 10.12.0.0/16 CIDR however it has a different subnet mask (We're using ECMP routing). I've tried pre-setting public_bind_addr, public_addr, and public_network_interface before running "ceph orch apply mon " all to no avail. Any thoughts on how to work around / address this check using cephadm?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: First 6 nodes cluster with Octopus

[mabi]

‐‐‐ Original Message ‐‐‐
On Saturday, April 3, 2021 11:22 PM, David Orman  wrote:

> We use cephadm + podman for our production clusters, and have had a
> great experience. You just need to know how to operate with
> containers, so make sure to do some reading about how containers work.
> We're using Octopus 15.2.10 (started with earlier 15.2.x and have
> upgraded). We will be upgrading to Pacific in the future.

Hi David and thanks for your experience feedback.

Regarding doing some container reading, I already know Docker quite well but 
never used podman. What kind of reading do you recommend to do beforehand? 
Anything specific on docs.ceph.com? or just more general reading on the podman 
website?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Increase of osd space usage on cephfs heavy load

[Olivier AUDRY]

hello

when I run my borgbackup over cephfs volume (10 subvolumes for 1.5To) I
can see a big increase of osd space usage and 2 or 3 osd goes near
full, or full, then out and finally the cluster goes in error.

Any tips to prevent this ?

My cluster is cephv15 with :

9 nodes :

each node run : 2x6to hdd and 2x600to ssd
the cephfs got data on hdd and metadata on ssd.
the cephfs md cache is : 32Go

128pg for data and metadata (this is has been setup by auto balancer)

Perhaps I can fix the pg num for each of cephfs pool and prevent
autobalancer to run for them.

what do you think ?

thx you for your help and advices.

UPDATE : I increase the pg number to 256 for data and 1024 for metadata

Here the df during the backup started since 30min

POOL  ID  STORED   OBJECTS  USED %USED  MAX AVAIL
cephfs-metadata   12  183 GiB  514.68k  550 GiB   7.162.3 TiB

Before the backup the stored was 20GiB

oau
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm:: how to change the image for services

[胡 玮文]

> 在 2021年4月5日，20:48，Adrian Sevcenco  写道：
> 
> On 4/5/21 3:27 PM, 胡 玮文 wrote:
 在 2021年4月5日，19:29，Adrian Sevcenco  写道：
>>> 
>>> Hi! How/where can i change the image configured for a service?
>>> I tried to modify /var/lib/ceph///unit.{image,run}
>>> but after restarting
>>> ceph orch ps shows that the service use the same old image.
>> Hi Adrian,
> Hi!
> 
>> Try “ceph config set  container_image ” where  
>> can be global, osd, osd.1, etc. Then redeploy 
> yup, i did
> 
>> it with “ceph orch” series commands.
> well, already tried .. unfortunately because this is a test "cluster" on my 
> desktop
> i have a single mgr, and it seems that there is a requirement of having a 
> second standby mgr.

Ceph is designed to be a distributed system. I think set up some VMs to 
experiment with ceph can be easier.

> i tried to start a second mgr, but because of listening port clash i can't... 
> and i see not such option
> in ceph config show
> 
> so, i was hopping to be able to make the service (podman) to restart with the 
> new image ..
> and no joy so far ..
> 
> Thanks a lot!
> Adrian
> 
>>> What other configuration locations are there for the ceph components
>>> beside /etc/ceph (which is quite sparse) and /var/lib/ceph/ ?
>> Monitor configuration database.
>> https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#monitor-configuration-database
>>  
>> 
>> Weiwen Hu
>>> Thank you!
>>> Adrian
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm:: how to change the image for services

[Adrian Sevcenco]

On 4/5/21 3:27 PM, 胡 玮文 wrote:



在 2021年4月5日，19:29，Adrian Sevcenco  写道：

Hi! How/where can i change the image configured for a service?
I tried to modify /var/lib/ceph///unit.{image,run}
but after restarting
ceph orch ps shows that the service use the same old image.


Hi Adrian,

Hi!

Try “ceph config set  container_image ” where  can be global, osd, osd.1, etc. Then redeploy 

yup, i did


it with “ceph orch” series commands.

well, already tried .. unfortunately because this is a test "cluster" on my 
desktop
i have a single mgr, and it seems that there is a requirement of having a 
second standby mgr.

i tried to start a second mgr, but because of listening port clash i can't... 
and i see not such option
in ceph config show

so, i was hopping to be able to make the service (podman) to restart with the 
new image ..
and no joy so far ..

Thanks a lot!
Adrian


What other configuration locations are there for the ceph components
beside /etc/ceph (which is quite sparse) and /var/lib/ceph/ ?


Monitor configuration database.
https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#monitor-configuration-database 



Weiwen Hu

Thank you!
Adrian




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm:: how to change the image for services

[胡 玮文]

在 2021年4月5日，19:29，Adrian Sevcenco  写道：

Hi! How/where can i change the image configured for a service?
I tried to modify /var/lib/ceph///unit.{image,run}
but after restarting
ceph orch ps shows that the service use the same old image.

Hi Adrian,

Try “ceph config set  container_image ” where  can 
be global, osd, osd.1, etc. Then redeploy it with “ceph orch” series commands.

What other configuration locations are there for the ceph components
beside /etc/ceph (which is quite sparse) and /var/lib/ceph/ ?

Monitor configuration database.
https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#monitor-configuration-database

Weiwen Hu
Thank you!
Adrian

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephadm:: how to change the image for services

[Adrian Sevcenco]

Hi! How/where can i change the image configured for a service?
I tried to modify /var/lib/ceph///unit.{image,run}
but after restarting
ceph orch ps shows that the service use the same old image.

What other configuration locations are there for the ceph components
beside /etc/ceph (which is quite sparse) and /var/lib/ceph/ ?

Thank you!
Adrian



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Real world Timings of PG states

[Nico Schottelius]

Good morning,

I was wondering if there are any timing indications as to how long a PG
should "usually" stay in a certain state?

For instance, how long should a pg stay in

- peering (seconds - minutes?)
- activating (seconds?)
- srubbing (+deep)

The scrub process obviously depends on the number of objects in the PG,
however is the same true for peering and activation? Since Nautilus we
see longer (minutes long) peering states in the cluster, which we did
not see before.

Thanks for your input and hav a good start into the week!

Best regards,

Nico


--
Sustainable and modern Infrastructures by ungleich.ch
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] cephadm upgrade to pacific

[Peter Childs]

I am attempting to upgrade a Ceph Upgrade cluster that was deployed with
Octopus 15.2.8 and upgraded to 15.2.10 successfully. I'm not attempting to
upgrade to 16.2.0 Pacific, and it is not going very well.

 I am using cephadm. It looks to have upgraded the managers and stopped,
and not moved on to the monitors or anything else. I've attempted stopping
the upgrade and restarting it, with debug on and I'm not seeing anything to
say why it is not progressing any further.

I've also tried rebooting machines and failing the managers over with
no success. I'm currently thinking its stuck attempting to upgrade a
manager that does not exist.

Its a test cluster of 16 nodes, bit of a proof of concept, so if I've got
something terribly wrong I'm happy to look at deploying, (running on top of
CentOS 7 but I'm fast heading to using something else) (apart from anything
its not really a production ready system yet)

Just not sure where cephadm upgrade has crashed in 16.2.0

Thanks in advance

Peter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Is metadata on SSD or bluestore cache better?

[Robert Sander]

Am 04.04.21 um 22:52 schrieb Kai Börnert:

> a) Make SSD only pools for the cephfs metadata
> 
> b) Give every OSD a SSD for the bluestore cache

I would go with both. Depending on how much budget you have for SSD they
could be used in a mixed scenario where you have three to four Block.DB
volumes and one OSD on each SSD.

HDD only SSDs are quite slow. If you do not have enough SSDs for them go
with an SSD only cephfs metadata pool.

Regards
-- 
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Upgrade and lost osds Operation not permitted

[Behzad Khoshbakhti]

running as ceph user and not root.
Following is the startup configuration which can be found via the
https://paste.ubuntu.com/p/2kV8KhrRfV/.
[Unit]
Description=Ceph object storage daemon osd.%i
PartOf=ceph-osd.target
After=network-online.target local-fs.target time-sync.target
Before=remote-fs-pre.target ceph-osd.target
Wants=network-online.target local-fs.target time-sync.target
remote-fs-pre.target ceph-osd.target

[Service]
Environment=CLUSTER=ceph
EnvironmentFile=-/etc/default/ceph
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph
--setgroup ceph
ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER}
--id %i
LimitNOFILE=1048576
LimitNPROC=1048576
LockPersonality=true
MemoryDenyWriteExecute=true
# Need NewPrivileges via `sudo smartctl`
NoNewPrivileges=false
PrivateTmp=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
# flushing filestore requires access to /proc/sys/vm/drop_caches
ProtectKernelTunables=false
ProtectSystem=full
Restart=on-failure
RestartSec=10
RestrictSUIDSGID=true
StartLimitBurst=3
StartLimitInterval=30min
TasksMax=infinity

[Install]
WantedBy=ceph-osd.target

When I issue the following command, the ceph osd starts successfully.
However, when it is failed when launching from systemctl.
root@osd03:~# /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph
--setgroup ceph
2021-04-05T11:24:08.823+0430 7f91772c5f00 -1 osd.2 496 log_to_monitors
{default=true}
2021-04-05T11:24:09.943+0430 7f916f7b9700 -1 osd.2 496 set_numa_affinity
unable to identify public interface 'ens160' numa node: (0) Success


On Mon, Apr 5, 2021, 10:51 AM Behzad Khoshbakhti 
wrote:

> running as ceph user
>
> On Mon, Apr 5, 2021, 10:49 AM Anthony D'Atri 
> wrote:
>
>> Running as root, or as ceph?
>>
>> > On Apr 4, 2021, at 3:51 AM, Behzad Khoshbakhti 
>> wrote:
>> >
>> > It worth mentioning as I issue the following command, the Ceph OSD
>> starts
>> > and joins the cluster:
>> > /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup
>> ceph
>> >
>> >
>> >
>> > On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
>> khoshbakh...@gmail.com>
>> > wrote:
>> >
>> >> Hi all,
>> >>
>> >> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
>> >> manual upgrade using the precompiled packages, the OSDs was down with
>> the
>> >> following messages:
>> >>
>> >> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
>> >> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
>> >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
>> prime-osd-dir
>> >> --dev
>> >>
>> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> >> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>> >> Running command: /usr/bin/ln -snf
>> >>
>> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> >> /var/lib/ceph/osd/ceph-2/block
>> >> Running command: /usr/bin/chown -h ceph:ceph
>> /var/lib/ceph/osd/ceph-2/block
>> >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
>> >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> >> Running command: /usr/bin/systemctl enable
>> >> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> >> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>> >> Running command: /usr/bin/systemctl start ceph-osd@2
>> >> --> ceph-volume lvm activate successful for osd ID: 2
>> >>
>> >> Content of /var/log/ceph/ceph-osd.2.log
>> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 set uid:gid to 64045:64045
>> >> (ceph:ceph)
>> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 ceph version 16.2.0
>> >> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
>> >> ceph-osd, pid 5484
>> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00  0 pidfile_write: ignore
>> empty
>> >> --pid-file
>> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
>> >> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to
>> open
>> >> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
>> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1  *** ERROR: unable to open
>> >> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
>> directory*
>> >>
>> >>
>> >> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
>> >> â— ceph-osd@2.service - Ceph object storage daemon osd.2
>> >> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
>> >> vendor preset: enabled)
>> >> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
>> >> +0430; 50s ago
>> >>Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
>> >> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
>> >>Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
>> --id