[ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data

Hi Istvan: I think this issue may relate to this problem. Currently I am using tenant buckets now. https://tracker.ceph.com/issues/50785 Hope this will help you. Szabo, Istvan (Agoda) 于2021年5月11日周二 上午10:47写道： > Ok, will be challenging with an 800 millions object bucket 😃 But I might > give a

[ceph-users] Re: after upgrade to 16.2.3 16.2.4 and after adding few hdd's OSD's started to fail 1 by 1.

Thanks for fast responses. For now I m running with bluestore_allocator = bitmap and it doesnt crash anymore. (ps other messages didint appear for some reason) On 2021-05-15 02:10, Igor Fedotov wrote: This looks similar to #50656 indeed. Hopefully will fix that next week. Thanks, Igor

[ceph-users] Re: Limit memory of ceph-mgr

‐‐‐ Original Message ‐‐‐ On Monday, May 17, 2021 4:31 AM, Anthony D'Atri wrote: > You’re running on so small a node that 3.6GB is a problem?? Yes, I have hardware constraints based on the hardware where my hardware has maximum 8 GB of RAM as it is a Raspberry Pi 4. I am doing a proof of

[ceph-users] Re: Process for adding a separate block.db to an osd

Hi, sorry for replying to this old thread: I tried to add a block.db to an OSD but now the OSD can not start with the error: Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]: -7> 2021-05-17 09:50:38.362 7fc48ec94a80 -1 rocksdb: Corruption: CURRENT file does not end with newline Mai 17 09:5

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

What happens if we are using buffered_io and the machine restared due to some power failure? Everything that was in the cache will be lost or how ceph handle this? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.s

[ceph-users] Re: Process for adding a separate block.db to an osd

Hi Boris, could you please share full OSD startup log and file listing for '/var/lib/ceph/osd/ceph-68'? Thanks, Igor On 5/17/2021 1:09 PM, Boris Behrens wrote: Hi, sorry for replying to this old thread: I tried to add a block.db to an OSD but now the OSD can not start with the error: Mai

[ceph-users] Re: Process for adding a separate block.db to an osd

Hi Igor, I posted it on pastebin: https://pastebin.com/Ze9EuCMD Cheers Boris Am Mo., 17. Mai 2021 um 12:22 Uhr schrieb Igor Fedotov : > Hi Boris, > > could you please share full OSD startup log and file listing for > '/var/lib/ceph/osd/ceph-68'? > > > Thanks, > > Igor > > On 5/17/2021 1:09 PM,

[ceph-users] Re: Process for adding a separate block.db to an osd

You might want to check file structure at new DB using bluestore-tools's bluefs-export command: ceph-bluestore-tool --path --command bluefs-export --out needs to have enough free space enough to fit DB data. Once exported - does   contain valid BlueFS directory structure - multiple .sst

[ceph-users] Re: Process for adding a separate block.db to an osd

Like this? [root@s3db10 export-bluefs]# ls * db: 018215.sst 018444.sst 018839.sst 019074.sst 019174.sst 019372.sst 019470.sst 019675.sst 019765.sst 019882.sst 019918.sst 019961.sst 019997.sst 020022.sst 020042.sst 020061.sst 020073.sst 018216.sst 018445.sst 018840.sst 019075.sst

[ceph-users] Re: Process for adding a separate block.db to an osd

On 5/17/2021 2:53 PM, Boris Behrens wrote: Like this? Yeah. so DB dir structure is more or less O but db/CURRENT looks corrupted. It should contain something like: MANIFEST-020081 Could you please remove (or even just rename) block.db symlink and do the export again? Preferably to preserv

[ceph-users] Re: Process for adding a separate block.db to an osd

Here is the new output. I kept both for now. [root@s3db10 export-bluefs2]# ls * db: 018215.sst 018444.sst 018839.sst 019074.sst 019210.sst 019381.sst 019560.sst 019755.sst 019849.sst 019888.sst 019958.sst 019995.sst 020007.sst 020042.sst 020067.sst 020098.sst 020115.sst 018216.sst

[ceph-users] Re: Process for adding a separate block.db to an osd

Would you try fsck without standalone DB? On 5/17/2021 3:39 PM, Boris Behrens wrote: Here is the new output. I kept both for now. [root@s3db10 export-bluefs2]# ls * db: 018215.sst 018444.sst 018839.sst 019074.sst 019210.sst 019381.sst 019560.sst 019755.sst 019849.sst 019888.sst 01995

[ceph-users] Re: Process for adding a separate block.db to an osd

The FSCK looks good: [root@s3db10 export-bluefs2]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-68 fsck fsck success Am Mo., 17. Mai 2021 um 14:39 Uhr schrieb Boris Behrens : > Here is the new output. I kept both for now. > > [root@s3db10 export-bluefs2]# ls * > db: > 018215.sst 018444.ss

[ceph-users] Re: Process for adding a separate block.db to an osd

See my last mail :) Am Mo., 17. Mai 2021 um 14:52 Uhr schrieb Igor Fedotov : > Would you try fsck without standalone DB? > > On 5/17/2021 3:39 PM, Boris Behrens wrote: > > Here is the new output. I kept both for now. > > > > [root@s3db10 export-bluefs2]# ls * > > db: > > 018215.sst 018444.sst 0

[ceph-users] Re: Process for adding a separate block.db to an osd

If you haven't had successful OSD.68 starts with standalone DB I think it's safe to revert previous DB adding and just retry it. At first suggest to run bluefs-bdev-new-db command only and then do fsck again. If it's OK - proceed with bluefs migrate followed by another fsck. And then finalize

[ceph-users] Re: Process for adding a separate block.db to an osd

I have no idea why, but it worked. As the fsck went well, I just re did the bluefs-bdev-new-db and now the OSD is back up, with a block.db device. Thanks a lot Am Mo., 17. Mai 2021 um 15:28 Uhr schrieb Igor Fedotov : > If you haven't had successful OSD.68 starts with standalone DB I think > it'

[ceph-users] Re: Pool has been deleted before snaptrim finished

Highly likely you're facing "degraded" RocksDB caused by bulk data removal. This applies to both your original snaptrimm issue and the current flapping OSDs. Following tickets reffer to snaptrim issue: https://tracker.ceph.com/issues/50511 https://tracker.ceph.com/issues/47446 As a workarou

[ceph-users] Pool has been deleted before snaptrim finished

Hi, We decided to delete the pool before the snaptrim finished after 4 days waiting. Now we have bigger issue, many osd started to flap, 2 of them cannot even restart due after. Did some bluestore fsck on the not started osds and has many messages like this inside: 2021-05-17 18:37:07.176203 7

[ceph-users] Re: After a huge amount of snaphot delete many snaptrim+snaptrim_wait pgs

Yes, but before I update need to have a healthy cluster, don't really want to update if it is not healthy to carry issue over. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Re: RBD as a boot image [was: libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping]

Markus Kienast writes: > Hi Nico, > > we are already doing exactly that: > > Loading initrd via iPXE > which contains the necessary modules and scripts to boot an RBD boot dev. > Works just fine. Interesting and very good to hear. How do you handle kernel differences (loaded kernel vs. module

[ceph-users] Re: RBD as a boot image [was: libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping]

Am Mo., 17. Mai 2021 um 20:28 Uhr schrieb Nico Schottelius < nico.schottel...@ungleich.ch>: > > > Markus Kienast writes: > > > Hi Nico, > > > > we are already doing exactly that: > > > > Loading initrd via iPXE > > which contains the necessary modules and scripts to boot an RBD boot dev. > > Work

[ceph-users] Does dynamic resharding block I/Os by design?

Hi, I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not? I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS o

[ceph-users] Re: Does dynamic resharding block I/Os by design?

2021年5月18日(火) 9:23 Satoru Takeuchi : > > Hi, > > I have a Ceph cluster used for RGW and RBD. I found that all I/Os to > RGW seemed to be > blocked while dynamic resharding. Could you tell me whether this > behavior is by design or not? > > I attached a graph which means I/O seemed to be blocked. He