[ceph-users] Re: Can not activate some OSDs after upgrade (bad crc on label)
Hi, On Tue, Dec 19, 2023 at 9:16 PM Huseyin Cotuk wrote: > After comparing the first blocks of running and failed OSDs, we found that HW > crash caused a corruption on the first 23 bytes of block devices. > > First few bytes of the block device of a failed OSD contains: > > : > 0010: 0a63 3965 6533 6566 362d ...c9ee3ef6- A small correction, just in case it might be a clue for someone to understand how it could have happened at all. As you may see, 22 bytes (not 23) were erased, i.e. "bluestore block device" safety header but not the ending "\n" (0x0a) byte which is also a part of this header. We advised to restore the full header (23 bytes) just for safety but 22 bytes would be enough too. -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: backfill_wait preventing deep scrubs
On Thu, Sep 21, 2023 at 1:57 PM Frank Schilder wrote: > > Hi all, > > I replaced a disk in our octopus cluster and it is rebuilding. I noticed that > since the replacement there is no scrubbing going on. Apparently, an OSD > having a PG in backfill_wait state seems to block deep scrubbing all other > PGs on that OSD as well - at least this is how it looks. > > Some numbers: the pool in question has 8192 PGs with EC 8+3 and ca 850 OSDs. > A total of 144 PGs needed backfilling (were remapped after replacing the > disk). After about 2 days we are down to 115 backfill_wait + 3 backfilling. > It will take a bit more than a week to complete. > > There is plenty of time and IOP/s available to deep-scrub PGs on the side, > but since the backfill started there is zero scrubbing/deep scrubbing going > on and "PGs not deep scrubbed in time" messages are piling up. > > Is there a way to allow (deep) scrub in this situation? ceph config set osd osd_scrub_during_recovery true -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: MGR executes config rm all the time
Hi Frank. It should be fixed in recent versions. See [1] and the
related backport tickets.
[1] https://tracker.ceph.com/issues/57726
On Sun, Sep 10, 2023 at 11:16 AM Frank Schilder wrote:
>
> Hi all,
>
> I recently started observing that our MGR seems to execute the same "config
> rm" commands over and over again, in the audit log:
>
> 9/10/23 10:03:24 AM[INF]from='mgr.252911336 ' entity='mgr.ceph-25'
> cmd=[{"prefix":"config
> rm","who":"mgr","name":"mgr/rbd_support/ceph-25/mirror_snapshot_schedule"}]:
> dispatch
>
> 9/10/23 10:03:24 AM[INF]from='mgr.252911336 192.168.32.68:0/63'
> entity='mgr.ceph-25' cmd=[{"prefix":"config
> rm","who":"mgr","name":"mgr/rbd_support/ceph-25/mirror_snapshot_schedule"}]:
> dispatch
>
> 9/10/23 10:03:19 AM[INF]from='mgr.252911336 ' entity='mgr.ceph-25'
> cmd=[{"prefix":"config
> rm","who":"mgr","name":"mgr/rbd_support/ceph-25/trash_purge_schedule"}]:
> dispatch
>
> 9/10/23 10:03:19 AM[INF]from='mgr.252911336 192.168.32.68:0/63'
> entity='mgr.ceph-25' cmd=[{"prefix":"config
> rm","who":"mgr","name":"mgr/rbd_support/ceph-25/trash_purge_schedule"}]:
> dispatch
>
> 9/10/23 10:02:24 AM[INF]from='mgr.252911336 ' entity='mgr.ceph-25'
> cmd=[{"prefix":"config
> rm","who":"mgr","name":"mgr/rbd_support/ceph-25/mirror_snapshot_schedule"}]:
> dispatch
>
> We don't have mirroring, ts a single cluster. What is going on here and how
> can I stop that? I already restarted all MGR daemons to no avail.
>
> Thanks and best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
--
Mykola Golub
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: rbd export-diff/import-diff hangs
On Mon, Aug 28, 2023 at 6:21 AM Tony Liu wrote:
>
> It's export-diff from an in-use image, both from-snapshot and to-snapshot
> exist.
> The same from-snapshot exists in import image, which is the to-snapshot from
> last diff.
> export/import is used for local backup, rbd-mirroring is used for remote
> backup.
Just to make it clear, do you mean you are running export-diff for an
image that is being mirrored
(snapshot based)?
> Looking for options to get more info to troubleshoot.
I would split "rbd export-diff | rbd import-diff" into two commands:
rbd export-diff > image.diff
rbd import-diff < image.diff
and see if it gets stuck for the first one, so we are sure the
export-diff is the issue here.
The next step would be enabling rbd debug, something like this:
rbd export-diff .. --debug-rbd=20
--log-file=/tmp/{image}_{from_snap}_{to_snap}.log
--log-to-stderr=false
Hope it will not use too much space and you will be able to get a log
for a getting stuck case.
Then please provide the log for review somehow. Also, notice the time
when you interrupt the hanging
export-diff.
--
Mykola Golub
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 16.2.7 pacific QE validation status, RC1 available for testing
On Tue, Nov 30, 2021 at 01:57:14PM +0530, Deepika Upadhyay wrote: > krbd, rbd lgtm! > >- krbd: > > <https://tracker.ceph.com/issues/51845> > <https://tracker.ceph.com/issues/53427> > <https://tracker.ceph.com/issues/53418> > <https://tracker.ceph.com/issues/52059> > > >- rbd > > <https://tracker.ceph.com/issues/53112> > <https://tracker.ceph.com/issues/52503> > <https://tracker.ceph.com/issues/51531> > > > @Mykola adding existing issues found on these runs, can you take another > look and approve? I don't have much experience with krbd tests but the failures indeed look like known issues, not related to pacific update. rbd - LGTM. Thanks. -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: rbd-nbd crashes Error: failed to read nbd request header: (33) Numerical argument out of domain
On Wed, May 19, 2021 at 11:32:04AM +0800, Zhi Zhang wrote: > On Wed, May 19, 2021 at 11:19 AM Zhi Zhang > wrote: > > > > > On Tue, May 18, 2021 at 10:58 PM Mykola Golub > > wrote: > > > > > > Could you please provide the full rbd-nbd log? If it is too large for > > > the attachment then may be via some public url? > > > > ceph.rbd-client.log.bz2 > > <https://drive.google.com/file/d/1TuiGOrVAgKIJ3BUmiokG0cU12fnlQ3GR/view?usp=drive_web> > > > > I uploaded it to google driver. Pls check it out. > > We found the reader_entry thread got zero byte when trying to read the nbd > request header, then rbd-nbd exited and closed the socket. But we haven't > figured out why read zero byte? Ok. I was hoping to find some hint in the log, why the read from the kernel could return without data, but I don't see it. >From experience it could happen when the rbd-nbd got stack or was too slow so the kernel failed after timeout, but it looked different in the logs AFAIR. Anyway you can try increasing the timeout using rbd-nbd --timeout (--io-timeout in newer versions) option. The default is 30 sec. If it does not help, probably you will find a clue increasing the kernel debug level for nbd (it seems it is possible to do). -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: rbd-nbd crashes Error: failed to read nbd request header: (33) Numerical argument out of domain
Could you please provide the full rbd-nbd log? If it is too large for the attachment then may be via some public url? -- Mykola Golub On Tue, May 18, 2021 at 03:04:51PM +0800, Zhi Zhang wrote: > Hi guys, > > We are recently testing rbd-nbd using ceph N version. After map rbd > image, mkfs and mount the nbd device, the rbd-nbd and dmesg will show > following errors when doing some read/write testing. > > rbd-nbd log: > > 2021-05-18 11:35:08.034 7efdb8ff9700 20 []rbd-nbd: reader_entry: > waiting for nbd request > ... > 2021-05-18 11:35:08.066 7efdb8ff9700 -1 []rbd-nbd: failed to read nbd > request header: (33) Numerical argument out of domain > 2021-05-18 11:35:08.066 7efdb3fff700 20 []rbd-nbd: writer_entry: no io > requests, terminating > 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: > 0x564a2be2b3c0 unregister_update_watcher: handle=0 > 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: > 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: handle=0 > 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: > 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: completing > unregister > 2021-05-18 11:35:08.066 7efdea8d1a00 10 []rbd-nbd: ~NBDServer: terminating > 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: > 0x564a2be2b3c0 close > > dmesg: > > [Tue May 18 11:35:07 2021] EXT4-fs (nbd0): mounted filesystem with > ordered data mode. Opts: discard > [Tue May 18 11:35:07 2021] block nbd0: shutting down sockets > [Tue May 18 11:35:09 2021] blk_update_request: I/O error, dev nbd0, > sector 75592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0 > > client host info: > > centos7.x > kernel 5.4.109 > > > It looks like the kernel nbd device shutdown its socket for some > reason, but we haven't figured it out. BTW, we have tried to turn > on/off rbd cache, use different fs ext4/xfs, use ec pool or replicated > pool, but the error remains. It is more frequent for us to reproduce > when batch map, mkfs and mount rbd-nbd on different hosts > simultaneously. > > Thanks for any suggestions. > > Regards, > Zhi Zhang (David) > Contact: zhang.david2...@gmail.com > zhangz.da...@outlook.com > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Erasure-coded Block Device Image Creation With qemu-img - Help
On Wed, Mar 17, 2021 at 04:29:10PM +1100, duluxoz wrote: > ``` > > rbd create -s 1T --data-pool my_pool my_pool_metadata/my_data > > ``` > > First Question: Is this correct? Yes > > Second Question: What is the qemu-img equivalent command - is it: > > ``` > > qemu-img create -f rbd rbd:--data-pool my_pool my_pool_metadata/my_data 1T > > ``` To make qemu-img use the data pool you can set rbd_default_data_pool param in ceph.conf, or use CEPH_ARGS environment variable. Something like this: ``` CEPH_ARGS=--rbd_default_data_pool=my_pool qemu-img create -f rbd rbd:my_pool_metadata/my_data 1T ``` -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: MON slow ops and growing MON store
On Thu, Feb 25, 2021 at 08:58:01PM +0100, Janek Bevendorff wrote:
> On the first MON, the command doesn’t even return, but I was able to
> get a dump from the one I restarted most recently. The oldest ops
> look like this:
>
> {
> "description": "log(1000 entries from seq 17876238 at
> 2021-02-25T15:13:20.306487+0100)",
> "initiated_at": "2021-02-25T20:40:34.698932+0100",
> "age": 183.762551121,
> "duration": 183.762599201,
The mon stores cluster log messages in the mon db. You mentioned
problems with osds flooding with log messages. It looks like related.
If you still observe the db growth you may try temporarily disable
clog_to_monitors, i.e. set for all osds:
clog_to_monitors = false
And see if it stops growing after this and if it helps with the slow
ops (it might make sense to restar mons if some look like get
stuck). You can apply the config option on the fly (without restarting
the osds, e.g with injectargs), but when re-enabling back you will
have to restart the osds to avoid crashes due to this bug [1].
[1] https://tracker.ceph.com/issues/48946
--
Mykola Golub
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Erasure coded calculation
On Thu, Feb 25, 2021 at 10:55:05AM +, Simon Sutter wrote: > The output of ceph df detail is: > --- RAW STORAGE --- > CLASS SIZE AVAILUSED RAW USED %RAW USED > hdd109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 > TOTAL 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 > > --- POOLS --- > POOL ID PGS STORED OBJECTS %USED MAX AVAIL > device_health_metrics 11 51 MiB 48 0 30 TiB > rep_data_fs 2 32 14 KiB3.41k 0 30 TiB > rep_meta_fs 3 32 227 MiB1.72k 0 30 TiB > ec_bkp14 32 4.2 TiB1.10M 6.11 67 TiB > > So ec_bkp1 uses 4.2TiB an there are 67TiB free usable Storage. > This means total EC usable storage would be 71.2TiB. > But calculating with the 109TiB RAW storage, shouldn't it be 81.75? The "MAX AVAIL" is not "free", i.e. it is not a difference between total and used. It is estimated by the cluster as "how much data user will be able to store additionally until one of osds is filled up". Consider an example, when by some reason all osds are 10% full and one 50% full. The cluster will "assume" that if you store additional data it will be distributed the same way, i.e. that one osd will continue to store more than others, so when all other osds will 20% full that one will be 100% full and the cluster is not usable. I.e. basically "MAX AVAIL" in this example is (N - 1) * 10% + 1 * 50%, instead of (N - 1) * 90% + 1 * 50%, which whould you expect for "free". To make "MAX AVAIL" match exactly "free", you have to have a perfectly balanced cluster. Look at `ceph osd df` output to see how well data is balanced in your case. -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Question about per MDS journals
On Thu, Feb 25, 2021 at 09:59:41AM +, John Spray wrote: > osdc/Journaler is for RDB, client/Journaler is for CephFS. Actually, src/journal/(Journaler.h) is for RBD (it is more generic, but currently is used by RBD only). And src/osdc/Journaler.h is for cephfs. > > > On Thu, Feb 25, 2021 at 8:26 AM 조규진 wrote: > > > > Hi, John. > > > > Thanks for your kind reply! > > > > While i'm checking the code that you recommend to check and other .cc files > > about journal, I find that there is two Journaler class. > > One is at "src/osdc/Journaler.h" and the other one is at > > "src/journal/Journaler.h". > > If you don't mind, could you tell me which one is for MDS journal? and the > > differences between them? > > > > Thanks. > > kyujin > > > > 2021년 2월 25일 (목) 오전 1:15, John Spray 님이 작성: > >> > >> On Wed, Feb 24, 2021 at 9:10 AM 조규진 wrote: > >> > > >> > Hi. > >> > > >> > I'm a newbie in CephFS and I have some questions about how per-MDS > >> > journals > >> > work. > >> > In Sage's paper (osdi '06), I read that each MDSs has its own journal and > >> > it lazily flushes metadata modifications on OSD cluster. > >> > What I'm wondering is that some directory operations like rename work > >> > with > >> > multiple metadata and It may work on two or more MDSs and their journals, > >> > so I think it needs some mechanisms to construct a transaction that works > >> > on multiple journals like some distributed transaction mechanisms. > >> > > >> > Could anybody explains how per-MDS journals work in such directory > >> > operations? or recommends some references about it? > >> > >> Your intuition is correct: these transactions span multiple MDS journals. > >> > >> The code for this stuff is somewhat long, in src/mds/Server.cc, but > >> here are a couple of pointers if you're interested in untangling it: > >> - Server::handle_client_rename is the entry point > >> - The MDS which handles the client request sends MMDSPeerRequest > >> messages to peers in rename_prepare_witness, and waits for > >> acknowledgements before writing EUpdate events to its journal > >> - The peer(s) write EPeerUpdate(OP_PREPARE) events to their journals > >> during prepare, and EPeerUpdate(OP_COMMIT) after the first MDS has > >> completed. > >> > >> John > >> > >> > >> > >> > > >> > Thanks. > >> > kyujin. > >> > ___ > >> > ceph-users mailing list -- ceph-users@ceph.io > >> > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Data Missing with RBD-Mirror
On Mon, Feb 22, 2021 at 11:37:52AM -0500, Vikas Rana wrote: > That is correct. On Prod we do have 22TB and on DR we only have 5.5TB But did you check that you really have missing files/data? Just to make sure it is not just some issue with how data is stored/counted in different clusters. Assuming you did and data is missing, then the only way to proceed I think is to issue resync, running on the secondary site: rbd mirror image resync cifs/research_data Note, it would mean recreating the image and resyncing all 22TB though. And it would be nice to add some monitoring to be able to detect the moment if the issue happens again, and report it to the tracker attaching the rbd-mirror logs. -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Data Missing with RBD-Mirror
On Mon, Feb 22, 2021 at 09:41:44AM -0500, Vikas Rana wrote: > # rbd journal info -p cifs --image research_data > rbd journal '11cb6c2ae8944a': > header_oid: journal.11cb6c2ae8944a > object_oid_prefix: journal_data.17.11cb6c2ae8944a. > order: 24 (16MiB objects) > splay_width: 4 Eh, I asked for a wrong command. Actually I wanted to see `rbd journal status`. Anyway, I have that info in mirror status below, which looks like up to date now. > We restarted the rbd-mirror process on the DR side > # rbd --cluster cephdr mirror pool status cifs --verbose > health: OK > images: 1 total > 1 replaying > > research_data: > global_id: 69656449-61b8-446e-8b1e-6cf9bd57d94a > state: up+replaying > description: replaying, master_position=[object_number=396351, tag_tid=4, > entry_tid=455084955], mirror_position=[object_number=396351, tag_tid=4, > entry_tid=455084955], entries_behind_master=0 > last_update: 2021-02-19 15:36:30 And I suppose, after creating and replaying a snapshot, you still see files missing on the secondary after mounting it? -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Data Missing with RBD-Mirror
On Thu, Feb 18, 2021 at 03:28:11PM +, Eugen Block wrote: > Hi, > > was there an interruption between those sites? > > > last_update: 2021-01-29 15:10:13 > > If there was an interruption you'll probably need to resync those images. If your results shown below are not from that past then yes, it looks like the rbd-mirror (at least the image replayer) got stuck for some reason long time ago. Then I can't see though how could you mount a newly created snap, because it would not be replayed. Probably you had a snapshot with such name previously, it was replayed, then the rbd-mirror got stuck, the snapshot was deleted on the primary and a new one created recently. And on the secondary you was still seeing and mounting the old snapshot? This would also explain why you were able to mount it -- if data is really missing I expect you are not able to mount the fs due to corruption. If the rbd-mirror just got stuck then you probably don't need to resync. Just restarting the rbd-mirror should make it to start replaying again. Though taking how long it was not replaying, if the journal is very large, the resync might be faster. You can try: rbd journal info -p cifs --image research_data to see how large the journal is currently (the difference in the master and the rbd-mirror client positions). And if this is really the case that rbd-mirror got stuck, any additional info you could provide (rbd-mirror logs, the core dump) might be helpful for fixing the bug. It is can be reported right to the tracker. What version are you running BTW? -- Mykola Golub > Zitat von Vikas Rana : > > > Hi Friends, > > > > > > > > We have a very weird issue with rbd-mirror replication. As per the command > > output, we are in sync but the OSD usage on DR side doesn't match the Prod > > Side. > > > > On Prod, we are using close to 52TB but on DR side we are only 22TB. > > > > We took a snap on Prod and mounted the snap on DR side and compared the data > > and we found lot of missing data. Please see the output below. > > > > > > > > Please help us resolve this issue or point us in right direction. > > > > > > > > Thanks, > > > > -Vikas > > > > > > > > DR# rbd --cluster cephdr mirror pool status cifs --verbose > > > > health: OK > > > > images: 1 total > > > > 1 replaying > > > > > > > > research_data: > > > > global_id: 69656449-61b8-446e-8b1e-6cf9bd57d94a > > > > state: up+replaying > > > > description: replaying, master_position=[object_number=390133, tag_tid=4, > > entry_tid=447832541], mirror_position=[object_number=390133, tag_tid=4, > > entry_tid=447832541], entries_behind_master=0 > > > > last_update: 2021-01-29 15:10:13 > > > > > > > > DR# ceph osd pool ls detail > > > > pool 5 'cifs' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins > > pg_num 128 pgp_num 128 last_change 1294 flags hashpspool stripe_width 0 > > application rbd > > > > removed_snaps [1~5] > > > > > > > > > > > > PROD# ceph df detail > > > > POOLS: > > > > NAMEID QUOTA OBJECTS QUOTA BYTES USED%USED > > MAX AVAIL OBJECTS DIRTY READWRITE RAW USED > > > > cifs17 N/A N/A 26.0TiB 30.10 > > 60.4TiB 6860550 6.86M 873MiB 509MiB 52.1TiB > > > > > > > > DR# ceph df detail > > > > POOLS: > > > > NAMEID QUOTA OBJECTS QUOTA BYTES USED%USED > > MAX AVAIL OBJECTS DIRTY READWRITE RAW USED > > > > cifs5 N/A N/A 11.4TiB 15.78 > > 60.9TiB 3043260 3.04M 2.65MiB 431MiB 22.8TiB > > > > > > > > > > > > > > > > PROD#:/vol/research_data# du -sh * > > > > 11T Flab1 > > > > 346GKLab > > > > 1.5TMore > > > > 4.4TReLabs > > > > 4.0TWLab > > > > > > > > DR#:/vol/research_data# du -sh * > > > > 2.6TFlab1 > > > > 14G KLab > > > > 52K More > > > > 8.0KRLabs > > > > 202MWLab > > > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: rbd move between pools
On Thu, Feb 18, 2021 at 09:34:00AM +, Marc wrote: > > > > > There IS an 'rbd mv', and also 'rbd cp' does the exact copy. > > Or am I missing something in the question? > > > > Yes my bad. I am blind sometimes. Since I learned that a mv in > cephfs between different data pools, leaves still data in the source > pool. Seeing 'deep cp', 'cp' and 'import/export' I thought maybe ask > first, before I destroy something. The `rbd mv` is just an alias for `rbd rename`, so it can be used just to rename an image within a pool. Which I suppose is not what you want. `rbd cp` will copy only one image snapshot (or the image head) to the destination. `rbd deep cp` will copy all image snapshots and the image head. It looks like you want `rbd deep cp` for your task but from your example I am not sure I understand what problems you have with it? The new format 'import/export' can also be used for copying all snapshots. And there is `rbd migration` [1] which among other things may be used as "deep move" between the pools. May be this is what you want? [1] https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [Ceph-qa] Using rbd-nbd tool in Ceph development cluster
On Mon, Nov 16, 2020 at 12:19:35PM +0100, Bobby wrote: > My question is: Can we use this *rbd-nbd* tool in the Ceph cluster? By Ceph > cluster I mean the development cluster we build through *vstart.sh* script. > I am quite sure we could use it. I have this script running. I can *start* > and *stop* the cluster. But I am struggling to use this rbd-nbd tool in > the development cluster which we build through vstart.sh script. Sure. Running this from the build directory should just work: sudo ./bin/rbd-nbd map $pool/$image ./bin/rbd-nbd list-mapped sudo ./bin/rbd-nbd unmap $pool/$image Doesn't it work for you? -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Mgr stability
On Wed, Aug 14, 2019 at 12:12:36PM -0500, Reed Dier wrote: > My main metrics source is the influx plugin, but I enabled the > prometheus plugin to get access to the per-rbd image metrics. I may > disable prometheus and see if that yields better stability, until > possibly the influx plugin gets updated to support those metric > exports. Before disabling the prometheus plugin, could you try just disabling per-rbd image metrics (i.e. set rbd_stats_pools param to empty)? Per-rbd images stats is a new feature and might be heavy depending on your cluster size and image count, so it would be nice to check this first. I also see you have rbd_support module enabled. It would be good to have it temporary disabled during this experiment too. -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: rbd image usage per osd
On Fri, Aug 09, 2019 at 12:44:42PM -0400, Frank R wrote: > I have an all RBD pool/cluster. I am interested in tracking how much disk > space is being used by each RBD image on every OSD drive. > > The OSDs are Filestore. > > Does anyone know of any existing scripts that accomplish this task? > > If not, what commands can be used to generate this info? 1) use `rbd info $pool/$image` to get object name prefix for this image (block_name_prefix field); 2) use `rados -p $pool ls | grep "^$block_name_prefix"` to generate a list of all existing objects for this image; 3) for every object from the list use `rados -p $pool stat $object` to get its size; 4) for every object from the list use `ceph osd map $pool $object` to get its location; 5) group obtained data by osd. Though I expect it will be very slow. For Filestore, instead of using `rados` and `ceph osd map` you could ssh to every osd node and use e.g. find+stat to get necessary data for all files (objects) with $block_name_prefix in their name. -- Mykola Golub ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
