[ceph-users] 3 DC with 4+5 EC not quite working

[Torkil Svensgaard]

We are looking to create a 3 datacenter 4+5 erasure coded pool but can't 
quite get it to work. Ceph version 17.2.7. These are the hosts (there 
will eventually be 6 hdd hosts in each datacenter):


-33  886.00842  datacenter 714
 -7  209.93135  host ceph-hdd1 


-69   69.86389  host ceph-flash1
 -6  188.09579  host ceph-hdd2 

 -3  233.57649  host ceph-hdd3 


-12  184.54091  host ceph-hdd4
-34  824.47168  datacenter DCN
-73   69.86389  host ceph-flash2
 -2  201.78067  host ceph-hdd5 

-81  288.26501  host ceph-hdd6 

-31  264.56207  host ceph-hdd7 


-36 1284.48621  datacenter TBA
-77   69.86389  host ceph-flash3
-21  190.83224  host ceph-hdd8 

-29  199.08838  host ceph-hdd9 

-11  193.85382  host ceph-hdd10 

 -9  237.28154  host ceph-hdd11 

-26  187.19536  host ceph-hdd12 


 -4  206.37102  host ceph-hdd13

We did this:

ceph osd erasure-code-profile set DRCMR_k4m5_datacenter_hdd 
plugin=jerasure k=4 m=5 technique=reed_sol_van crush-root=default 
crush-failure-domain=datacenter crush-device-class=hdd


ceph osd pool create cephfs.hdd.data erasure DRCMR_k4m5_datacenter_hdd
ceph osd pool set cephfs.hdd.data allow_ec_overwrites true
ceph osd pool set cephfs.hdd.data pg_autoscale_mode warn

Didn't quite work:

"
[WARN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg 
incomplete
pg 33.0 is creating+incomplete, acting 
[104,219,NONE,NONE,NONE,41,NONE,NONE,NONE] (reducing pool 
cephfs.hdd.data min_size from 5 may help; search ceph.com/docs for 
'incomplete')

"

I then manually changed the crush rule from this:

"
rule cephfs.hdd.data {
id 7
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step chooseleaf indep 0 type datacenter
step emit
}
"

To this:

"
rule cephfs.hdd.data {
id 7
type erasure
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class hdd
step choose indep 0 type datacenter
step chooseleaf indep 3 type host
step emit
}
"

Based on some testing and dialogue I had with Red Hat support last year 
when we were on RHCS, and it seemed to work. Then:


ceph fs add_data_pool cephfs cephfs.hdd.data
ceph fs subvolumegroup create hdd --pool_layout cephfs.hdd.data

I started copying data to the subvolume and increased pg_num a couple of 
times:


ceph osd pool set cephfs.hdd.data pg_num 256
ceph osd pool set cephfs.hdd.data pg_num 2048

But at some point it failed to activate new PGs eventually leading to this:

"
[WARN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.cephfs.ceph-flash1.agdajf(mds.0): 64 slow metadata IOs are 
blocked > 30 secs, oldest blocked for 25455 secs

[WARN] MDS_TRIM: 1 MDSs behind on trimming
mds.cephfs.ceph-flash1.agdajf(mds.0): Behind on trimming 
(997/128) max_segments: 128, num_segments: 997

[WARN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive
pg 33.6f6 is stuck inactive for 8h, current state 
activating+remapped, last acting [50,79,116,299,98,219,164,124,421]
pg 33.6fa is stuck inactive for 11h, current state 
activating+undersized+degraded+remapped, last acting 
[17,408,NONE,196,223,290,73,39,11]
pg 33.705 is stuck inactive for 11h, current state 
activating+undersized+degraded+remapped, last acting 
[33,273,71,NONE,411,96,28,7,161]
pg 33.721 is stuck inactive for 7h, current state 
activating+remapped, last acting [283,150,209,423,103,325,118,142,87]
pg 33.726 is stuck inactive for 11h, current state 
activating+undersized+degraded+remapped, last acting 
[234,NONE,416,121,54,141,277,265,19]
[WARN] PG_DEGRADED: Degraded data redundancy: 1818/1282640036 objects 
degraded (0.000%), 3 pgs degraded, 3 pgs undersized
pg 33.6fa is stuck undersized for 7h, current state 
activating+undersized+degraded+remapped, last acting 
[17,408,NONE,196,223,290,73,39,11]
pg 33.705 is stuck undersized for 7h, current state 
activating+undersized+degraded+remapped, last acting 
[33,273,71,NONE,411,96,28,7,161]
pg 33.726 is stuck undersized for 7h, current state 
activating+undersized+degraded+remapped, last acting 
[234,NONE,416,121,54,141,277,265,19]

[WARN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups
Pool cephfs.hdd.data has 1024 placement groups, should have 2048
[WARN] SLOW_OPS: 3925 slow ops, oldest one blocked for 26012 sec, 
daemons [osd.17,osd.234,osd.283,osd.33,osd.50] have slow ops.

"

We had thought it would only affect that particular data pool but 
eventually everything ground to a halt, also RBD on unrelated replicated 
pools. Looking at it now I guess that was because the 5 OSDs were 

[ceph-users] Re: Is there any way to merge an rbd image's full backup and a diff?

[Satoru Takeuchi]

Hi Ilya,

2023年12月18日(月) 9:14 Satoru Takeuchi :
>
> Hi Ilya,
>
> > > Yes, it's possible. It's one of a workaround I thought. Then the
> > > backup data are as follows:
> > >
> > > a. The full backup taken at least 14 days ago.
> > > b. The latest 14 days backup data
> >
> > I think it would be:
> >
> > a. A full backup (taken potentially months ago, exact age doesn't
> >really matter)
> > b. Differential #1 - diff from the full backup to the 14 days old version
> > c. Differential #2 - diff from the 14 days old to the 13 days old version
> > d. Differential #3 - diff from the 13 days old to the 12 days old version
> > ...
> >
> > Every day after a new differential is taken, (b) and (c) would be
> > merged, keeping the number of differentials constant.
> >
> > >
> > > In this case, I concern the effect if (a) becomes too old. However, it
> > > might be a groundless fear.
> >
> > I would suggest re-capturing the full backup from time to time, just as
> > a precaution against something going wrong with a backup based on a too
> > long series of (merged) differentials.  It might be groundless concern,
> > but then you can't be too careful when it comes to backups.

I also got an advice from MykolaI and succeeded to merge the full
backup with the oldest diff.
The key point is capturing the full backup as a diff data.

My verification is:

1. Create an empty RBD image.
2. Update this image.
3. Capture a full backup F, by `rbd export-diff`.
4. Update this image again.
5. Capture a diff D by `rbd export-diff`
6. Create a new image which size is the same as an RBD image created at step1.
7. Import F by `rbd import-diff` and the contents is correct.
8. Import D by `rbd import-diff` and the contents is correct.
9. Merge F and D by `rbd merge-diff` and create D2.
10. Create an another new image.
11. Import D2 by `rbd import-diff` and the contents is as expected.

Thanks,
Satoru
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [v18.2.1] problem with wrong osd device symlinks after upgrade to 18.2.1

[Reto Gysi]

Hi Eugen

LV tags seem to look ok to me.

LV_tags:
-
root@zephir:~# lvs -a -o +devices,tags  | egrep 'osd1|  LV' | grep -v osd12

 LV VG
   Attr   LSizePool Origin
Data%  Meta%  Move Log Cpy%Sync Convert Devices
  LV Tags



 osd-block-cdd02721-6876-4db8-bdb2-12ac6c70127c
ceph-dec5bd7c-d84f-40d9-ba14-6bd8aadf2957 -wi-ao   16.37t
/dev/sde(0)
  ceph.block_device=/dev/ceph-de
c5bd7c-d84f-40d9-ba14-6bd8aadf2957/osd-block-cdd02721-6876-4db8-bdb2-12ac6c70127c,ceph.block_uuid=tHvTi7-nkde-JCvt-USL1-52V7-ejIi-J14rkb,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=27923302-87a5-11ec-ac5b-976d21a49941,ceph.cluster_name=ceph,c
eph.crush_device_class=,ceph.db_device=/dev/optane/ceph-db-osd1,ceph.db_uuid=NfBaWG-ZnB2-1RV0-zRqJ-EbUe-v7ds-230OW9,ceph.encrypted=0,ceph.osd_fsid=cdd02721-6876-4db8-bdb2-12ac6c70127c,ceph.osd_id=1,ceph.osdspec_affinity=,ceph.type=block,ceph.v
do=0
 ceph-db-osd1   optane
   rwi-aor---   50.00g
   100.00
  ceph-db-osd1_rimage_0(0),ceph-db-osd1_rimage_1(0)
ceph.block_device=/dev/ceph-de
c5bd7c-d84f-40d9-ba14-6bd8aadf2957/osd-block-cdd02721-6876-4db8-bdb2-12ac6c70127c,ceph.block_uuid=tHvTi7-nkde-JCvt-USL1-52V7-ejIi-J14rkb,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=27923302-87a5-11ec-ac5b-976d21a49941,ceph.cluster_name=ceph,c
eph.crush_device_class=,ceph.db_device=/dev/optane/ceph-db-osd1,ceph.db_uuid=NfBaWG-ZnB2-1RV0-zRqJ-EbUe-v7ds-230OW9,ceph.encrypted=0,ceph.osd_fsid=cdd02721-6876-4db8-bdb2-12ac6c70127c,ceph.osd_id=1,ceph.osdspec_affinity=,ceph.type=db,ceph.vdo=
0
 [ceph-db-osd1_rimage_0]optane
   iwi-aor---   50.00g
/dev/sdi(9216)




 [ceph-db-osd1_rimage_0]optane
   iwi-aor---   50.00g
/dev/sdi(82518)




 [ceph-db-osd1_rimage_0]optane
   iwi-aor---   50.00g
/dev/sdi(55297)




 [ceph-db-osd1_rimage_1]optane
   iwi-aor---   50.00g
/dev/sdj(1)




 [ceph-db-osd1_rmeta_0] optane
   ewi-aor---4.00m
/dev/sdi(46080)




 [ceph-db-osd1_rmeta_1] optane
   ewi-aor---4.00m
/dev/sdj(0)




root@zephir:~#
--


root@zephir:~# ceph osd metadata osd.1
{
   "id": 1,
   "arch": "x86_64",
   "back_addr": "[v2:
192.168.0.3:6832/1767446902,v1:192.168.0.3:6833/1767446902]",
   "back_iface": "",
   "bluefs": "1",
   "bluefs_db_access_mode": "blk",
   "bluefs_db_block_size": "4096",
   "bluefs_db_dev_node": "/dev/dm-13",
   "bluefs_db_devices": "sdi,sdj",
   "bluefs_db_driver": "KernelDevice",
   "bluefs_db_optimal_io_size": "0",
   "bluefs_db_partition_path": "/dev/dm-13",
   "bluefs_db_rotational": "0",
   "bluefs_db_size": "53687091200",
   "bluefs_db_support_discard": "1",
   "bluefs_db_type": "ssd",
   "bluefs_dedicated_db": "1",
   "bluefs_dedicated_wal": "0",
   "bluefs_single_shared_device": "0",
   "bluestore_bdev_access_mode": "blk",
   "bluestore_bdev_block_size": "4096",
   "bluestore_bdev_dev_node": "/dev/dm-1",
   "bluestore_bdev_devices": "sde",
   "bluestore_bdev_driver": "KernelDevice",
   "bluestore_bdev_optimal_io_size": "0",
   "bluestore_bdev_partition_path": "/dev/dm-1",
   "bluestore_bdev_rotational": "1",
   "bluestore_bdev_size": "18000203743232",
   "bluestore_bdev_support_discard": "0",
   "bluestore_bdev_type": "hdd",
   "bluestore_min_alloc_size": "4096",
   "ceph_release": "reef",
   "ceph_version": "ceph version 18.2.1
(7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)",
   "ceph_version_short": "18.2.1",
   "ceph_version_when_created": "ceph version 17.2.6
(d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)",
   "container_hostname": "zephir",
   "container_image": "quay.io/ceph/ceph:v18.2.1",
   "cpu": "AMD Ryzen Threadripper PRO 3975WX 32-Cores",
   "created_at": "2023-05-08T16:42:46.313648Z",
   "default_device_class": "hdd",
   "device_ids":
"sde=ATA_ST18000NM000J-2T_ZR53Z55N,sdi=NVME_INTEL_SSDPF21Q40_PHAL1505013J400BGN,sdj=NVME_INTEL_SSDPF21Q80_PHAL2165005L800CGN",

   "device_paths":

[ceph-users] Re: [v18.2.1] problem with wrong osd device symlinks after upgrade to 18.2.1

[Reto Gysi]

Ok, I found the problem I think.
The problem is that the LVM osd with the LVM raid1 block.db is activated by
RAWActivate instead of LVMActivate, which I think is wrong.

furthermore if /dev/optante/ceph-db-osd1 is a raid1 LV,
ceph_volume.device.raw.list reports:
>>> foo = raw_list.direct_report(dev)
('ignoring child device /dev/mapper/optane-ceph--db--osd1 whose parent
/dev/dm-51 is a BlueStore OSD.', "device is likely a phantom Atari
partition. device info: {'NAME': '/dev/mapper/optane-ceph--db--osd1',
'KNAME': '/dev/dm-13', 'PKNAME': '/
dev/dm-51', 'MAJ:MIN': '254:13', 'FSTYPE': 'ceph_bluestore', 'MOUNTPOINT':
'', 'LABEL': '', 'UUID': '', 'RO': '0', 'RM': '0', 'MODEL': '', 'SIZE':
'50G', 'STATE': 'running', 'OWNER': 'ceph', 'GROUP': 'ceph', 'MODE':
'brw-rw', 'ALIGNMENT':
'0', 'PHY-SEC': '512', 'LOG-SEC': '512', 'ROTA': '0', 'SCHED': '', 'TYPE':
'lvm', 'DISC-ALN': '0', 'DISC-GRAN': '512B', 'DISC-MAX': '4G', 'DISC-ZERO':
'0', 'PARTLABEL': ''}")
and the block.db is wrong.
>>> print(foo['cdd02721-6876-4db8-bdb2-12ac6c70127c'])
{'osd_uuid': 'cdd02721-6876-4db8-bdb2-12ac6c70127c', 'type': 'bluestore',
'osd_id': 1, 'ceph_fsid': '27923302-87a5-11ec-ac5b-976d21a49941', 'device':
'/dev/mapper/ceph--dec5bd7c--d84f--40d9--ba14--6bd8aadf2957-osd--block--cdd02721--6876--4db8-
-bdb2--12ac6c70127c', 'device_db':
'/dev/mapper/optane-ceph--db--osd1_rimage_1'}

one possible fix:
diff --git a/src/ceph-volume/ceph_volume/devices/raw/list.py
b/src/ceph-volume/ceph_volume/devices/raw/list.py
index 794bb18c103..5f874050f7c 100644
--- a/src/ceph-volume/ceph_volume/devices/raw/list.py
+++ b/src/ceph-volume/ceph_volume/devices/raw/list.py
@@ -112,6 +112,9 @@ class List(object):
result = {}
logger.debug('inspecting devices: {}'.format(devs))
for info_device in info_devices:
+if info_device['TYPE'] == 'lvm':
+# lvm devices are not raw devices
+continue
bs_info = _get_bluestore_info(info_device['NAME'])
if bs_info is None:
# None is also returned in the rare event that there is an
issue reading info from


I saw that in commit comments that a previous change like this got
reverted, because of some problems
--
commit 401bb755020bc0962a2a8038d626b3bc4ec4fff4
Author: Guillaume Abrioux 
Date:   Tue Nov 7 14:39:50 2023 +0100

   ceph-volume: Revert "ceph-volume: fix raw list for lvm devices"

   This reverts commit e5e429617c1c27dcd631171f65d30571e32f7266.
   This commit introduced a regression, see linked tracker for details.

   Fixes: https://tracker.ceph.com/issues/63391

   Signed-off-by: Guillaume Abrioux 
   (cherry picked from commit 916a22ef031953056771eceb1f49cab7eb746978)

diff --git a/src/ceph-volume/ceph_volume/devices/raw/list.py
b/src/ceph-volume/ceph_volume/devices/raw/list.py
index 0f801701b80..c86353b90ec 100644
--- a/src/ceph-volume/ceph_volume/devices/raw/list.py
+++ b/src/ceph-volume/ceph_volume/devices/raw/list.py
@@ -69,7 +69,7 @@ class List(object):
def generate(self, devs=None):
logger.debug('Listing block devices via lsblk...')
info_devices = disk.lsblk_all(abspath=True)
-if devs is None or devs == []:
+if not devs or not any(devs):
# If no devs are given initially, we want to list ALL devices
including children and
# parents. Parent disks with child partitions may be the
appropriate device to return if
# the parent disk has a bluestore header, but children may be
the most appropriate
@@ -89,9 +89,6 @@ class List(object):
# determine whether a parent is bluestore, we should err on the
side of not reporting
# the child so as not to give a false negative.
info_device = [info for info in info_devices if info['NAME'] ==
dev][0]
-if info_device['TYPE'] == 'lvm':
-# lvm devices are not raw devices
-continue
if 'PKNAME' in info_device and info_device['PKNAME'] != "":
parent = info_device['PKNAME']
try:


Regards,

Reto

Am Sa., 6. Jan. 2024 um 17:22 Uhr schrieb Reto Gysi :

> Hi ceph community
>
> I noticed the following problem after upgrading my ceph instance on Debian
> 12.4 from 17.2.7 to 18.2.1:
>
> I had placed bluestore block.db for hdd osd's on raid1/mirrored logical
> volumes on 2 nvme devices, so that if a single block.db nvme device fails,
> that not all hdd osds fail.
> That worked fine under 17.2.7 and had no problems during host/osd restarts.
> During the upgrade to 18.2.1 the osd's wouldn't with the block.db on
> mirrored lv wouldn't start anymore because the block.db symlink was updated
> to pointing to the wrong device mapper device, and the osd startup failed
> with error message that block.db device is busy.
>
> OSD1:
> 2024-01-05T19:56:43.592+ 7fdde9f43640 -1
> 

[ceph-users] [quincy 17.2.7] ceph orchestrator not doing anything

[Boris]

Happy new year everybody.

I just found out that the orchestrator in one of our clusters is not doing
anything.

What I tried until now:
- disabling / enabling cephadm (no impact)
- restarting hosts (no impact)
- starting upgrade to same version (no impact)
- starting downgrade (no impact)
- forcefully removing hosts and adding them again (now I have no daemons
anymore)
- applying new configurations (no impact)

The orchestrator just does nothing.
Cluster itself is fine.

I also checked the SSH connecability from all hosts to all hosts (
https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors)

The logs always show a message like "took the task" but then nothing
happens.

Cheers
 Boris
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific bluestore_volume_selection_policy

[Igor Fedotov]

Hi Reed,

no much sense to attach the logs to the mentioned tickets - the problem 
with the assertion is well-known and has been already fixed.


Your current issue is weird config update behavior which prevents from 
applying the work around. Feel free to open ticket about that but I 
don't think it's an efficient way - IIUC the problem isn't common and 
likely caused by something specific to your setup. Which rather means 
the fix wouldn't appear soon enough. Unfortunately that's not my area of 
expertise either so I'm of little help here as well.


Nevertheless If I troubleshoot  this config update issue I'd start the 
investigation by trying different parameters/daemons/hosts. Are you able 
to tune any parameter at all? Is it doable at different host or OSD?


Not to mention that you might just try to restart monitors first ;)


Thanks,

Igor

On 10/01/2024 21:38, Reed Dier wrote:

Hi Igor,

That’s correct (shown below).
Would it be helpful for me to add logs/uploaded crash UUID’s to 53906 
, 53907 
, 54209 
, 62928 
, 63110 
, 63161 
, 63352 
?
Or maybe open a new tracker to track that the parameter change isn’t 
being properly persisted or whatever appears to be happening?


Thanks,
Reed

/build/ceph-16.2.14/src/os/bluestore/BlueStore.h: 3870: FAILED 
ceph_assert(cur >= p.length)


 ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) 
pacific (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x152) [0x55d51970a987]

 2: /usr/bin/ceph-osd(+0xad3b8f) [0x55d51970ab8f]
 3: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t 
const&)+0x112) [0x55d519e040f2]
 4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, 
unsigned long)+0x69d) [0x55d519ea0fad]
 5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xaa) 
[0x55d519ea14ea]

 6: (BlueFS::fsync(BlueFS::FileWriter*)+0x7d) [0x55d519ec61ed]
 7: (BlueRocksWritableFile::Sync()+0x19) [0x55d519ed5a59]
 8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions 
const&, rocksdb::IODebugContext*)+0x52) [0x55d51a3e37ce]
 9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x216) 
[0x55d51a5eddac]

 10: (rocksdb::WritableFileWriter::Sync(bool)+0x17b) [0x55d51a5ed785]
 11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup 
const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned 
long)+0x39a) [0x55d51a441bf8]
 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, 
rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, 
unsigned long, bool, unsigned long*, unsigned long, 
rocksdb::PreReleaseCallback*)+0x135e) [0x55d51a43d96c]
 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, 
rocksdb::WriteBatch*)+0x5d) [0x55d51a43c56f]
 14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, 
std::shared_ptr)+0x85) [0x55d51a388635]
 15: 
(RocksDBStore::submit_transaction_sync(std::shared_ptr)+0x9b) 
[0x55d51a38904b]

 16: (BlueStore::_kv_sync_thread()+0x22bc) [0x55d519e016dc]
 17: (BlueStore::KVSyncThread::entry()+0x11) [0x55d519e2de71]
 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f490cf23609]
 19: clone()

     0> 2024-01-10T11:39:05.922-0500 7f48f978d700 -1 *** Caught 
signal (Aborted) **

 in thread 7f48f978d700 thread_name:bstore_kv_sync

 ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) 
pacific (stable)

 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f490cf2f420]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x1ad) [0x55d51970a9e2]

 5: /usr/bin/ceph-osd(+0xad3b8f) [0x55d51970ab8f]
 6: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t 
const&)+0x112) [0x55d519e040f2]
 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, 
unsigned long)+0x69d) [0x55d519ea0fad]
 8: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xaa) 
[0x55d519ea14ea]

 9: (BlueFS::fsync(BlueFS::FileWriter*)+0x7d) [0x55d519ec61ed]
 10: (BlueRocksWritableFile::Sync()+0x19) [0x55d519ed5a59]
 11: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions 
const&, rocksdb::IODebugContext*)+0x52) [0x55d51a3e37ce]
 12: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x216) 
[0x55d51a5eddac]

 13: (rocksdb::WritableFileWriter::Sync(bool)+0x17b) [0x55d51a5ed785]
 14: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup 
const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned 
long)+0x39a) [0x55d51a441bf8]
 15: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, 
rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, 
unsigned long, bool, unsigned long*, unsigned long, 
rocksdb::PreReleaseCallback*)+0x135e) [0x55d51a43d96c]
 16: (rocksdb::DBImpl::Write(rocksdb::WriteOptions 

[ceph-users] Re: Unable to execute radosgw command using cephx users on client side

[Eugen Block]

Hi,

I don't really have any solution, but it appears to require rwx  
permissions at least for the rgw tag:


caps osd = "allow rwx tag rgw *=*

This was the only way I got the radosgw-admin commands to work in my  
limited test attempts. Maybe someone else has more insights. My  
interpretation of these error messages ("failed to update source  
index") is that it actually requires to update something:


2024-01-03T17:34:06.498+ 7f915ece1fc0  0 ERROR: failed reading  
data (obj=default.rgw.log:bucket.sync-source-hints.), r=-1
2024-01-03T17:34:06.498+ 7f915ece1fc0  0 ERROR: failed to update  
sources index for bucket=:[]) r=-1


But as I said, I might misinterpret things, so I hope someone else can  
chime in here.


Regards,
Eugen

Zitat von Alam Mohammad :


Hello,

In our Ceph cluster we encountered issues while attempting to  
execute "radosgw-admin" command on client side using cephx user  
having read only permission. Whenever we are executing  
"radosgw-admin user list" command it is throwing an error.


"ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef  
(stable)"


We have performed below steps in our environment
Case-1 : First we created cephx user with below privileges

client.rgw.username
key: <---key--->
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow r  tag rgw *=*

on client side we copied keyring and ceph.conf file
What we noticed on client machine all general command like "ceph  
-s", "ceph health detail" "ceph df" running fine, even  
"radosgw-admin zonegroup list --id=rgw.username," command returned  
the expected output, but when attempting commands like  
"radosgw-admin user list," "radosgw-admin bucket list," or  
"radosgw-admin user info," errors were encountered.

Below are the outputs that is throwing

root@control:~# radosgw-admin user list --id=rgw.username
2024-01-03T17:34:06.498+ 7f915ece1fc0  0 ERROR: failed reading  
data (obj=default.rgw.log:bucket.sync-source-hints.), r=-1
2024-01-03T17:34:06.498+ 7f915ece1fc0  0 ERROR: failed to update  
sources index for bucket=:[]) r=-1
2024-01-03T17:34:06.498+ 7f915ece1fc0  0 ERROR: failed to  
initialize bucket sync policy handler: get_bucket_sync_hints() on  
bucket=-- returned r=-1
2024-01-03T17:34:06.498+ 7f915ece1fc0 -1 ERROR: could not  
initialize zone policy handler for zone=default
2024-01-03T17:34:06.498+ 7f915ece1fc0  0 ERROR: failed to start  
notify service ((1) Operation not permitted
2024-01-03T17:34:06.498+ 7f915ece1fc0  0 ERROR: failed to init  
services (ret=(1) Operation not permitted)

couldn't init storage provider

Case- 2 : In this case we granted read permissions to the rgw data  
pool and index pool for the user,

client.rgw.username
key: 
caps: [mgr] allow r
caps: [mon] allow r
caps: [osd] allow r pool=default.rgw.log
Despite this, while general commands worked perfectly fine on the  
client side, but "radosgw-admin" commands still failed to execute.


Here is the output
root@control:~# radosgw-admin user list --id=rgw.username
2024-01-03T17:43:38.071+ 7f8b5a8bffc0  0 failed reading realm  
info: ret -1 (1) Operation not permitted
2024-01-03T17:43:38.071+ 7f8b5a8bffc0  0 ERROR: failed to start  
notify service ((1) Operation not permitted
2024-01-03T17:43:38.071+ 7f8b5a8bffc0  0 ERROR: failed to init  
services (ret=(1) Operation not permitted)

couldn't init storage provider

Have I overlooked anything in the process?
Any guidance or insight would be greatly appreciated.

Thanks,
Mohammad Saif
Ceph Enthusiast







In the first step, we created a CephX user named client.rgw.saif  
with read permissions for the manager (mgr), monitor (mon), and  
object storage daemon (osd) components, along with specific RGW  
capabilities. On the client side, we successfully copied the keyring  
and ceph.conf, and certain commands, such as radosgw-admin zonegroup  
list --id=rgw.username,

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [v18.2.1] problem with wrong osd device symlinks after upgrade to 18.2.1

[Eugen Block]

Hi,

I don't really have any advice but I'm curious how the LV tags look  
like (lvs -o lv_tags). Do they point to the correct LVs for the  
block.db? Does the 'ceph osd metadata ' show anything weird? Is  
there something useful in the ceph-volume.log  
(/var/log/ceph/{FSID}/ceph-volume.log)?


Regards,
Eugen

Zitat von Reto Gysi :


Hi ceph community

I noticed the following problem after upgrading my ceph instance on Debian
12.4 from 17.2.7 to 18.2.1:

I had placed bluestore block.db for hdd osd's on raid1/mirrored logical
volumes on 2 nvme devices, so that if a single block.db nvme device fails,
that not all hdd osds fail.
That worked fine under 17.2.7 and had no problems during host/osd restarts.
During the upgrade to 18.2.1 the osd's wouldn't with the block.db on
mirrored lv wouldn't start anymore because the block.db symlink was updated
to pointing to the wrong device mapper device, and the osd startup failed
with error message that block.db device is busy.

OSD1:
2024-01-05T19:56:43.592+ 7fdde9f43640 -1
bluestore(/var/lib/ceph/osd/ceph-1) _minimal_open_bluefs add block
device(/var/lib/ceph/osd/ceph-1/block.db) returned: (16) Device or resource
busy
2024-01-05T19:56:43.592+ 7fdde9f43640 -1
bluestore(/var/lib/ceph/osd/ceph-1) _open_db failed to prepare db
environment:
2024-01-05T19:56:43.592+ 7fdde9f43640  1 bdev(0x55a2d5014000
/var/lib/ceph/osd/ceph-1/block) close
2024-01-05T19:56:43.892+ 7fdde9f43640 -1 osd.1 0 OSD:init: unable to
mount object store

the symlink was updated to point to
lrwxrwxrwx 1 ceph ceph  111 Jan  5 20:57 block ->
/dev/mapper/ceph--dec5bd7c--d84f--40d9--ba14--6bd8aadf2957-osd--block--cdd02721--6876--4db8--bdb2--12ac6c70127c

lrwxrwxrwx 1 ceph ceph   48 Jan  5 20:57 block.db ->
/dev/mapper/optane-ceph--db--osd1_rimage_1_iorig

the correct symlink would have been:
lrwxrwxrwx 1 ceph ceph  111 Jan  5 20:57 block ->
/dev/mapper/ceph--dec5bd7c--d84f--40d9--ba14--6bd8aadf2957-osd--block--cdd02721--6876--4db8--bdb2--12ac6c70127c

lrwxrwxrwx 1 ceph ceph   48 Jan  5 20:57 block.db ->
/dev/mapper/optane-ceph--db--osd1


To continue with the upgrade I converted one by one all the block.db lvm
logical volumes back to linear volumes, and fixed the symlinks manually.
converting the lv's back to linear was necessary, because even when I fixed
the symlink manually, after a osd restart the symlink would be created
wrong again if the block.db would point to a raid1 lv.

Here's any example how the symlink looked before an osd was touched by the
18.2.1 upgrade:
OSD2:
lrwxrwxrwx 1 ceph ceph   93 Jan  4 03:38 block ->
/dev/ceph-17a894d6-3a64-4e5e-9fa0-8dd3b5f4bf33/osd-block-3cd7a5af-9002-47a7-b4c2-540381d53be7

lrwxrwxrwx 1 ceph ceph   24 Jan  4 03:38 block.db ->
/dev/optane/ceph-db-osd2


Here's what the output of lvs -a -o +devices looked like for OSD1 block.db
device when it was an raid1 lv:

  LV VG
   Attr   LSizePool Origin
Data%  Meta%  Move Log Cpy%Sync Convert Devices

 ceph-db-osd1   optane
   rwi-a-r---   44.00g
   100.00
  ceph-db-osd1_rimage_0(0),ceph-db-osd1_rimage_1(0)

 [ceph-db-osd1_rimage_0]optane
   gwi-aor---   44.00g
 [ceph-db-osd1_rimage_0_iorig] 100.00
  ceph-db-osd1_rimage_0_iorig(0)
 [ceph-db-osd1_rimage_0_imeta]  optane
   ewi-ao  428.00m

 /dev/sdg(55482)

 [ceph-db-osd1_rimage_0_imeta]  optane
   ewi-ao  428.00m

 /dev/sdg(84566)

 [ceph-db-osd1_rimage_0_iorig]  optane
   -wi-ao   44.00g

 /dev/sdg(9216)

 [ceph-db-osd1_rimage_0_iorig]  optane
   -wi-ao   44.00g

 /dev/sdg(82518)

 [ceph-db-osd1_rimage_1]optane
   gwi-aor---   44.00g
 [ceph-db-osd1_rimage_1_iorig] 100.00
  ceph-db-osd1_rimage_1_iorig(0)
 [ceph-db-osd1_rimage_1_imeta]  optane
   ewi-ao  428.00m

 /dev/sdj(55392)

 [ceph-db-osd1_rimage_1_imeta]  optane
   ewi-ao  428.00m

 /dev/sdj(75457)

 [ceph-db-osd1_rimage_1_iorig]  optane
   -wi-ao   44.00g

 /dev/sdj(9218)

 [ceph-db-osd1_rimage_1_iorig]  optane
   -wi-ao   44.00g

 /dev/sdj(73409)

 [ceph-db-osd1_rmeta_0] optane
   ewi-aor---4.00m

 /dev/sdg(55388)

 [ceph-db-osd1_rmeta_1] optane

[ceph-users] Re: Stuck in upgrade process to reef

[Igor Fedotov]

Hi Jan,

unfortunately this wasn't very helpful. Moreover the log looks a bit 
messy - looks like a mixture of outputs from multiple running instances 
or something. I'm not an expert in using containerized setups though.


Could you please simplify things by running ceph-osd process manually 
like you did for ceph-objectstore-tool. And enforce log output to a 
file. Command line should look somewhat the following:


ceph-osd -i 0 --log-to-file --log-file  --debug-bluestore 
5/20 --debug-prioritycache 10


Please don't forget to run repair prior to that.


Also you haven't answered my questions about custom [memory] settings 
and RAM usage during OSD startup. It would be nice to hear some feedback.



Thanks,

Igor

On 11/01/2024 16:47, Jan Marek wrote:

Hi Igor,

I've tried to start osd.1 with debug_prioritycache and
debug_bluestore 5/20, see attached file...

Sincerely
Jan

Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov:

Hi Jan,

indeed this looks like some memory allocation problem - may be OSD's RAM
usage threshold reached or something?

Curious if you have any custom OSD settings or may be any memory caps for
Ceph containers?

Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10
and try to start OSD once again. Please monitor process RAM usage along the
process and share the resulting log.


Thanks,

Igor

On 10/01/2024 11:20, Jan Marek wrote:

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-volume fails in all recent releases with IndexError

[Eugen Block]

Hi,

I don't use rook but I haven't seen this issue yet in any of my test  
clusters (from octopus to reef). Althouth I don't redeploy OSDs all  
the time, I do set up fresh (single-node) clusters once or twice a  
week with different releases without any ceph-volume issues. Just to  
confirm I just recreated one cluster with 3 OSDs and separate rocksdb.  
Maybe it's a rook issue?


Regards,
Eugen


Zitat von Nico Schottelius :


Hello dear fellow ceph users,

it seems that for some months all current ceph releases (16.x, 17.x,
18.x) are having a bug in ceph-volume that causes disk
activation to fail with the error "IndexError: list index out of range"
(details below, [0]).

It also seems there is already a fix for it available [1], but also that
it hasn't been merged into any official release [2,3,4].

This has started to affect more and more nodes in our clusters and thus
I was wondering if others are also seeing this issue and whether anyone
knows whether it is planned to create a new release based on this soon?

Best regards,

Nico


[0]
kubectl -n rook-ceph logs -c activate rook-ceph-osd-30-6558b7cf69-5cbbl
+ OSD_ID=30
+ CEPH_FSID=bd3061a0-ecf3-4af6-9017-51b63c90b526
+ OSD_UUID=319e5756-318c-46a0-b7e9-429e39069302
+ OSD_STORE_FLAG=--bluestore
+ OSD_DATA_DIR=/var/lib/ceph/osd/ceph-30
+ CV_MODE=raw
+ DEVICE=/dev/sdf
+ cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf
+ python3 -c '
import configparser

config = configparser.ConfigParser()
config.read('\''/etc/ceph/ceph.conf'\'')

if not config.has_section('\''global'\''):
config['\''global'\''] = {}

if not config.has_option('\''global'\'','\''fsid'\''):
config['\''global'\'']['\''fsid'\''] = '\''\''

with open('\''/etc/ceph/ceph.conf'\'', '\''w'\'') as configfile:
config.write(configfile)
'
+ ceph -n client.admin auth get-or-create osd.30 mon 'allow profile  
osd' mgr 'allow profile osd' osd 'allow *' -k  
/etc/ceph/admin-keyring-store/keyring

[osd.30]
key = ...
+ [[ raw == \l\v\m ]]
++ mktemp
+ OSD_LIST=/tmp/tmp.OpZRJJOcrX
+ ceph-volume raw list /dev/sdf
Traceback (most recent call last):
  File "/usr/sbin/ceph-volume", line 11, in 
load_entry_point('ceph-volume==1.0.0', 'console_scripts',  
'ceph-volume')()
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line  
41, in __init__

self.main(self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py",  
line 59, in newfunc

return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line  
153, in main

terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py",  
line 194, in dispatch

instance.main()
  File  
"/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py",  
line 32, in main

terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py",  
line 194, in dispatch

instance.main()
  File  
"/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py",  
line 166, in main

self.list(args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py",  
line 16, in is_root

return func(*a, **kw)
  File  
"/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py",  
line 122, in list

report = self.generate(args.device)
  File  
"/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py",  
line 91, in generate

info_device = [info for info in info_devices if info['NAME'] == dev][0]
IndexError: list index out of range

[1] https://github.com/ceph/ceph/pull/49954
[2] https://github.com/ceph/ceph/pull/54705
[3] https://github.com/ceph/ceph/pull/54706
[4] https://github.com/ceph/ceph/pull/54707

--
Sustainable and modern Infrastructures by ungleich.ch
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Rack outage test failing when nodes get integrated again

[Frank Schilder]

Hi Steve,

I also observed that setting mon_osd_reporter_subtree_level to anything else 
than host leads to incorrect behavior.

In our case, I actually observed the opposite. I had 
mon_osd_reporter_subtree_level=datacenter (we have 3 DCs in the crush tree). 
After cutting off a single host with ifdown - also a network cut-off albeit not 
via firewall rules, I observed that not all OSDs on that host were marked down 
(neither was the host), leading to blocked IO. I didn't wait for very long 
(only a few minutes, less than 5), because its a production system. I also 
didn't find the time to file a tracker issue. I observed this with mimic, but 
since you report it for Pacific I'm pretty sure its affecting all versions.

My guess is that this is not part of the CI testing, at least not in a way that 
covers network cut-off.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Steve Baker 
Sent: Thursday, January 11, 2024 8:45 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Rack outage test failing when nodes get integrated again

Hi, we're currently testing a ceph (v 16.2.14) cluster, 3 mon nodes, 6 osd
nodes à 8 nvme ssd osds distributed over 3 racks. Daemons are deployed in
containers with cephadm / podman. We got 2 pools on it, one with 3x
replication and min_size=2, one with an EC (k3m3). With 1 mon node and 2
osd nodes in each rack, the crush rules are configured in a way (for the 3x
pool chooseleaf_firstn rack, for the ec pool choose_indep 3 rack /
chooseleaf_indep 2 host), so that a full rack can go down while the cluster
stays accessible for client operations. Other options we have set are
mon_osd_down_out_subtree_limit=host so that in case of a host/rack outage
the cluster does not automatically start to backfill, but will continue to
run in a degraded state until human interaction comes to fix it. Also we
set mon_osd_reporter_subtree_level=rack.

We tested - while under (synthetic test-)client load - what happens if we
take a full rack (one mon node and 2 osd nodes) out of the cluster. We did
that using iptables to block the nodes of the rack from other nodes of the
cluster (global and cluster network), as well as from the clients. As
expected, the remainder of the cluster continues to run in a degraded state
without to start any backfilling or recovery processes. All client requests
gets served while the rack is out.

But then a strange thing happens when we take the rack (1mon node, 2 osd
nodes) back into the cluster again by deleting all firewall rules with
iptables -F at once. Some osds get integrated in the cluster again
immediatelly but some others remain in state "down" for exactly 10 minutes.
These osds that stay down for the 10 minutes are in a state where they
still seem to not be able to reach other osd nodes (see heartbeat_check
logs below). After these 10 minutes have passed, these osds come up as well
but then at exactly that time, many PGs get stuck in peering state and
other osds that were all the time in the cluster get slow requests and the
cluster blocks client traffic (I think it's just the PGs stuck in peering
soaking all the client threads). Then, exactly 45 minutes after the nodes
of the rack were made reachable by iptables -F again, the situation
recovers, peering succeeds and client load gets handled again.

We have repeated this test several times and it's always exactly the same
10 min "down interval" and a 45 min affected client requests. When we
integrate the nodes into the cluster again one after another with a delay
of some minutes inbetween, this does not happen at all. I wonder what's
happening there. It must be some kind of split-brain situation having to do
with blocking the nodes using iptables but not rebooting them completelly.
The 10 min and 45 min intervals I described occure every time. For the 10
minutes, some osds stay down after the hosts got integrated again. It's not
all of the 16 osds from the 2 osd hosts that got integrated again but just
some of them. Which ones varies randomly. Sometimes it's also only just
one. We also observerd, the longer the hosts were out of the cluster, the
more osds are affected. Then even after they get up again after 10 minutes,
it takes another 45 minutes until the stuck peering situation resolves.
Also during these 45 minutes, we see slow ops on osds thet remained into
the cluster.


See here some OSD logs that get written after the reintegration:


2024-01-04T08:25:03.856+ 7f369132b700 -1 monclient:
_check_auth_rotating possible clock skew, rotating keys expired way too
early (before 2024-01-04T07:25:03.860426+)
2024-01-04T08:25:06.556+ 7f3682882700  0 log_channel(cluster) log [WRN]
: Monitor daemon marked osd.0 down, but it is still running
2024-01-04T08:25:06.556+ 7f3682882700  0 log_channel(cluster) log [DBG]
: map 

[ceph-users] Re: Sending notification after multiple objects are created in a ceph bucket.

[Yuval Lifshitz]

Lokendra and Kushagra,
We don't have such an enhancement on the roadmap. Would think of 2 options:
(1) implement the special logic using lua scripting. We have an example on
how to send notifications to a NATS broker from lua [1]. you can easily
adjust that to kafka. the 2 main drawbacks with this solution are:
* performance and latency, everything done in the lua script is
synchronous with the S3 operation
* lua scripts have the "global" RGW table for aggregations. however, this
is local to an RGW and not shared across RGWs. in addition, the information
there is not persistent, and will be lost if the RGW crash/restart
(2) implement that externally to the RGW. e.g. use bucket notifications to
trigger a serverless function (probably backed with some db) that does the
aggregation and send the notifications according to the logic that
you want. see [2]

also, there is the possibility of developing that feature and contributing
it to ceph. note that it would be a considerably more effort than the
options above.

Yuval

[1] https://github.com/ceph/ceph/blob/main/examples/rgw/lua/nats_adapter.md
[2] https://www.youtube.com/watch?v=1E5Q-ErF2sI


On Thu, Jan 11, 2024 at 11:08 AM Lokendra Rathour 
wrote:

> Facing a similar situation, any support would be helpful.
>
> -Lokendra
>
>
>
> On Tue, Jan 9, 2024 at 10:47 PM Kushagr Gupta <
> kushagrguptasps@gmail.com>
> wrote:
>
> > Hi Team,
> >
> > Features used: Rados gateway, ceph S3 buckets
> >
> > We are trying to create a data pipeline using the S3 buckets capability
> > and rado gateway in ceph.
> > Our endpoint is a kafka topic.
> >
> > Currently, as soon as an object is created, a notification is sent to
> > kafka topic.
> > Our goal is to create multiple object first and then send the
> notification.
> > Also if there is a way such that we can assign objects to a group and
> > collectively send notification on an event on that group.
> >
> > Could anyone please help me?
> >
> > Thanks and Regards,
> > Kushagra Gupta
> >
>
>
> --
> ~ Lokendra
> skype: lokendrarathour
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Sending notification after multiple objects are created in a ceph bucket.

[Lokendra Rathour]

Facing a similar situation, any support would be helpful.

-Lokendra



On Tue, Jan 9, 2024 at 10:47 PM Kushagr Gupta 
wrote:

> Hi Team,
>
> Features used: Rados gateway, ceph S3 buckets
>
> We are trying to create a data pipeline using the S3 buckets capability
> and rado gateway in ceph.
> Our endpoint is a kafka topic.
>
> Currently, as soon as an object is created, a notification is sent to
> kafka topic.
> Our goal is to create multiple object first and then send the notification.
> Also if there is a way such that we can assign objects to a group and
> collectively send notification on an event on that group.
>
> Could anyone please help me?
>
> Thanks and Regards,
> Kushagra Gupta
>


-- 
~ Lokendra
skype: lokendrarathour
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io