[ceph-users] How to speed up rgw lifecycle

[VÔ VI]

Hi community,

My ceph cluster is using s3 with three pools and obj/s approximately 4.5k
obj/s and the rgw lifecycle delete per pool is only 60-70 objects/s

How can I speed up the lc rgw process? 60 70 objects/s is too slow

Thanks a lot
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] the image used size becomes 0 after export/import with snapshot

[Tony Liu]

Hi,

I have an image with a snapshot and some changes after snapshot.
```
$ rbd du backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26  
NAME
PROVISIONED  USED   
f0408e1e-06b6-437b-a2b5-70e3751d0a26@snapshot-eb085877-7557-4620-9c01-c5587b857029
   10 GiB  2.4 GiB
f0408e1e-06b6-437b-a2b5-70e3751d0a26
 10 GiB  2.4 GiB
 
 10 GiB  4.8 GiB
```
If there is no changes after snapshot, the image line will show 0 used.

I did export and import.
```
$ rbd export --export-format 2 backup/f0408e1e-06b6-437b-a2b5-70e3751d0a26 - | 
rbd import --export-format 2 - backup/test
Exporting image: 100% complete...done.
Importing image: 100% complete...done.
```

When check the imported image, the image line shows 0 used.
```
$ rbd du backup/test
NAMEPROVISIONED  USED   
test@snapshot-eb085877-7557-4620-9c01-c5587b857029   10 GiB  2.4 GiB
test 10 GiB  0 B
  10 GiB  2.4 GiB
```
Any clues how that happened? I'd expect the same du as the source.

I tried another quick test. It works fine.
```
$ rbd create backup/test-src --size 10G
$ sudo rbd map backup/test-src
/dev/rbd0
$ echo "hello" | sudo tee /dev/rbd0
hello
$ rbd du backup/test-src
NAME  PROVISIONED  USED 
test-src   10 GiB  4 MiB
$ rbd snap create backup/test-src@snap-1
Creating snap: 100% complete...done.
$ rbd du backup/test-src  
NAME PROVISIONED  USED 
test-src@snap-1   10 GiB  4 MiB
test-src  10 GiB0 B
   10 GiB  4 MiB
$ echo "world" | sudo tee /dev/rbd0
world
$ rbd du backup/test-src
NAME PROVISIONED  USED 
test-src@snap-1   10 GiB  4 MiB
test-src  10 GiB  4 MiB
   10 GiB  8 MiB
$ rbd export --export-format 2 backup/test-src - | rbd import --export-format 2 
- backup/test-dst
Exporting image: 100% complete...done.
Importing image: 100% complete...done.
$ rbd du backup/test-dst
NAME PROVISIONED  USED 
test-dst@snap-1   10 GiB  4 MiB
test-dst  10 GiB  4 MiB
   10 GiB  8 MiB
```

Thanks!
Tony
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: osdmaptool target & deviation calculation

[Konstantin Shalygin]

Hi,

This deviation is very soft. If u wanna do real upmaps you should use deviation 
1

k

Sent from my iPhone

> On Nov 27, 2023, at 21:39, Robert Hish  wrote:
> 
> The result is many many OSDs with a deviation well above the 
> upmap_max_deviation which is at default: 5
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.1 QE Validation status

[Venky Shankar]

On Tue, Nov 21, 2023 at 10:35 PM Venky Shankar  wrote:
>
> Hi Yuri,
>
> On Fri, Nov 10, 2023 at 1:22 PM Venky Shankar  wrote:
> >
> > Hi Yuri,
> >
> > On Fri, Nov 10, 2023 at 4:55 AM Yuri Weinstein  wrote:
> > >
> > > I've updated all approvals and merged PRs in the tracker and it looks
> > > like we are ready for gibba, LRC upgrades pending approval/update from
> > > Venky.
> >
> > The smoke test failure is caused by missing (kclient) patches in
> > Ubuntu 20.04 that certain parts of the fs suite (via smoke tests) rely
> > on. More details here
> >
> > https://tracker.ceph.com/issues/63488#note-8
> >
> > The kclient tests in smoke pass with other distro's and the fs suite
> > tests have been reviewed and look good. Run details are here
> >
> > https://tracker.ceph.com/projects/cephfs/wiki/Reef#07-Nov-2023
> >
> > The smoke failure is noted as a known issue for now. Consider this run
> > as "fs approved".
>
> We need an additional change to be tested for inclusion into this reef
> release. This one
>
> https://github.com/ceph/ceph/pull/54407
>
> The issue showed up when upgrading the LRC.

This was discussed in the CLT meeting last week and the overall
agreement was to be able to extend our tests to validate the fix which
showed up in quincy run (smoke suite) but not in reef. I've sent a
change regarding the same:

https://github.com/ceph/ceph/pull/54677

I'll update when it's ready to be included for testing.

>
> >
> > >
> > > On Thu, Nov 9, 2023 at 1:31 PM Radoslaw Zarzynski  
> > > wrote:
> > > >
> > > > rados approved!
> > > >
> > > > Details are here: 
> > > > https://tracker.ceph.com/projects/rados/wiki/REEF#1821-Review.
> > > >
> > > > On Mon, Nov 6, 2023 at 10:33 PM Yuri Weinstein  
> > > > wrote:
> > > > >
> > > > > Details of this release are summarized here:
> > > > >
> > > > > https://tracker.ceph.com/issues/63443#note-1
> > > > >
> > > > > Seeking approvals/reviews for:
> > > > >
> > > > > smoke - Laura, Radek, Prashant, Venky (POOL_APP_NOT_ENABLE failures)
> > > > > rados - Neha, Radek, Travis, Ernesto, Adam King
> > > > > rgw - Casey
> > > > > fs - Venky
> > > > > orch - Adam King
> > > > > rbd - Ilya
> > > > > krbd - Ilya
> > > > > upgrade/quincy-x (reef) - Laura PTL
> > > > > powercycle - Brad
> > > > > perf-basic - Laura, Prashant (POOL_APP_NOT_ENABLE failures)
> > > > >
> > > > > Please reply to this email with approval and/or trackers of known
> > > > > issues/PRs to address them.
> > > > >
> > > > > TIA
> > > > > YuriW
> > > > > ___
> > > > > Dev mailing list -- d...@ceph.io
> > > > > To unsubscribe send an email to dev-le...@ceph.io
> > > > >
> > > >
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
> >
> > --
> > Cheers,
> > Venky
>
>
>
> --
> Cheers,
> Venky



-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How balancer module balance data

[Dan van der Ster]

Hi,

For the reason you observed, I normally set upmap_max_deviation = 1 on
all clusters I get my hands on.

Cheers, Dan

--
Dan van der Ster
CTO

Clyso GmbH
p: +49 89 215252722 | a: Vancouver, Canada
w: https://clyso.com | e: dan.vanders...@clyso.com

We are hiring: https://www.clyso.com/jobs/
Try our Ceph Analyzer! https://analyzer.clyso.com/

On Mon, Nov 27, 2023 at 10:30 AM  wrote:
>
> Hello,
>
> We are running a pacific 16.2.10 cluster and enabled the balancer module, 
> here is the configuration:
>
> [root@ceph-1 ~]# ceph balancer status
> {
> "active": true,
> "last_optimize_duration": "0:00:00.052548",
> "last_optimize_started": "Fri Nov 17 17:09:57 2023",
> "mode": "upmap",
> "optimize_result": "Unable to find further optimization, or pool(s) 
> pg_num is decreasing, or distribution is already perfect",
> "plans": []
> }
> [root@ceph-1 ~]# ceph balancer eval
> current cluster score 0.017742 (lower is better)
>
> Here is the balancer configuration of upmap_max_deviation:
> # ceph config get mgr mgr/balancer/upmap_max_deviation
> 5
>
> We have two different types of OSDS, one is 7681G and another is 3840G. When 
> I checked our PG distribution on each type of OSD, I found the PG 
> distribution is not evenly, for the 7681G OSDs, the OSD distribution varies 
> from 136 to 158; while for the 3840G OSDs, it varies from 60 to 83, seems the 
> upmap_max_deviation is almost +/- 10. So I just wondering if this is expected 
> or do I need to change the upmap_max_deviation to a smaller value.
>
> Thanks for answering my question.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Denis Polom]

Hi,

looks like line I got :

# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-888 --deep 1

_verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 
0x73a13c49, expected 0x21d59f5e, device location [0x268e81~1000], 
logical extent 0x3~1000, object #-1:2b46bd33:::osdmap.927580:0#


was most interesting and was giving number of corrupted epoch

following brings OSDs back to life:
# ceph osd getmap -o osdmap.927580 927580
# CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool 
--data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap.927580


thx!

On 11/27/23 20:59, Wesley Dillingham wrote:

So those options are not consistent with the error in the video I linked.

I am not entirely sure how to proceed with your OSDs (how many are 
impacted?)


but you may want to try injecting an older osdmap epoch fetched from 
the mon in your osdmap injection:


try rewinding 1 epoch at a time from the current and see if that gets 
them to start.


Proceed with caution, I would test this as well.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 2:36 PM Denis Polom  wrote:

it's:

"bluestore_compression_algorithm": "snappy"

"bluestore_compression_mode": "none"


On 11/27/23 20:13, Wesley Dillingham wrote:

How about these two options:

bluestore_compression_algorithm
bluestore_compression_mode

Thanks.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 2:01 PM Denis Polom
 wrote:

Hi,

no we don't:

"bluestore_rocksdb_options":

"compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",

thx

On 11/27/23 19:17, Wesley Dillingham wrote:

Curious if you are using bluestore compression?

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 10:09 AM Denis Polom
 wrote:

Hi

we have issue to start some OSDs on one node on our Ceph
Quincy 17.2.7
cluster. Some OSDs on that node are running fine, but
some failing to start.

Looks like crc32 checksum error, and failing to get OSD
map. I found a
some discussions on that but nothing helped.

I've also tried to insert current OSD map but that ends
with error:

# CEPH_ARGS="--bluestore-ignore-data-csum"
ceph-objectstore-tool
--data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap
--file osdmap
osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.

Log is bellow

Any ideas please?

Thank you


 From log file:

2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling
back to public
interface

2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad
crc32c/0x1000
checksum at blob offset 0x0, got 0xb1701b42, expected
0x9ee5ece2, device
location [0x1~1000], logical extent 0x0~1000, object
#-1:7b3f43c4:::osd_superblock:0#

2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0
failed to load
OSD map for epoch 927580, got 0 bytes

/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
2023-11-27T16:01:51.443522+0100
/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED
ceph_assert(ret)
  ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
(stable)
  1: (ceph::__ceph_assert_fail(char const*, char const*,
int, char
const*)+0x14f) [0x561ad07d2624]
  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
  3: (OSD::init()+0x4026) [0x561ad08e5a86]
  4: main()
  5: __libc_start_main()
  6: _start()
*** Caught signal (Aborted) **
  in thread 7f3f17aa13c0 thread_name:ceph-osd
2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
2023-11-27T16:01:51.443522+0100
/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED
ceph_assert(ret)

  ceph version 17.2.7

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Wesley Dillingham]

So those options are not consistent with the error in the video I linked.

I am not entirely sure how to proceed with your OSDs (how many are
impacted?)

but you may want to try injecting an older osdmap epoch fetched from the
mon in your osdmap injection:

try rewinding 1 epoch at a time from the current and see if that gets them
to start.

Proceed with caution, I would test this as well.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 2:36 PM Denis Polom  wrote:

> it's:
>
> "bluestore_compression_algorithm": "snappy"
>
> "bluestore_compression_mode": "none"
>
>
> On 11/27/23 20:13, Wesley Dillingham wrote:
>
> How about these two options:
>
> bluestore_compression_algorithm
> bluestore_compression_mode
>
> Thanks.
>
> Respectfully,
>
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn 
>
>
> On Mon, Nov 27, 2023 at 2:01 PM Denis Polom  wrote:
>
>> Hi,
>>
>> no we don't:
>>
>> "bluestore_rocksdb_options":
>> "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",
>> thx
>>
>> On 11/27/23 19:17, Wesley Dillingham wrote:
>>
>> Curious if you are using bluestore compression?
>>
>> Respectfully,
>>
>> *Wes Dillingham*
>> w...@wesdillingham.com
>> LinkedIn 
>>
>>
>> On Mon, Nov 27, 2023 at 10:09 AM Denis Polom 
>> wrote:
>>
>>> Hi
>>>
>>> we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7
>>> cluster. Some OSDs on that node are running fine, but some failing to
>>> start.
>>>
>>> Looks like crc32 checksum error, and failing to get OSD map. I found a
>>> some discussions on that but nothing helped.
>>>
>>> I've also tried to insert current OSD map but that ends with error:
>>>
>>> # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
>>> --data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap
>>> osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.
>>>
>>> Log is bellow
>>>
>>> Any ideas please?
>>>
>>> Thank you
>>>
>>>
>>>  From log file:
>>>
>>> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public
>>> interface
>>>
>>> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
>>> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
>>> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device
>>> location [0x1~1000], logical extent 0x0~1000, object
>>> #-1:7b3f43c4:::osd_superblock:0#
>>>
>>> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load
>>> OSD map for epoch 927580, got 0 bytes
>>>
>>> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
>>> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
>>> 2023-11-27T16:01:51.443522+0100
>>> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>>> (stable)
>>>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x14f) [0x561ad07d2624]
>>>   2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>>   3: (OSD::init()+0x4026) [0x561ad08e5a86]
>>>   4: main()
>>>   5: __libc_start_main()
>>>   6: _start()
>>> *** Caught signal (Aborted) **
>>>   in thread 7f3f17aa13c0 thread_name:ceph-osd
>>> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
>>> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
>>> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
>>> 2023-11-27T16:01:51.443522+0100
>>> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>>>
>>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>>> (stable)
>>>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x14f) [0x561ad07d2624]
>>>   2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>>   3: (OSD::init()+0x4026) [0x561ad08e5a86]
>>>   4: main()
>>>   5: __libc_start_main()
>>>   6: _start()
>>>
>>>
>>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>>> (stable)
>>>   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>>>   2: gsignal()
>>>   3: abort()
>>>   4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x1b7) [0x561ad07d268c]
>>>   5: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>>   6: (OSD::init()+0x4026) [0x561ad08e5a86]
>>>   7: main()
>>>   8: __libc_start_main()
>>>   9: _start()
>>> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted)
>>> **
>>>   in thread 7f3f17aa13c0 thread_name:ceph-osd
>>>
>>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>>> (stable)
>>>   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>>>   2: gsignal()
>>>   3: abort()
>>>   4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> 

[ceph-users] Re: understand "extent"

[Ilya Dryomov]

On Sat, Nov 25, 2023 at 4:19 AM Tony Liu  wrote:
>
> Hi,
>
> The context is RBD on bluestore. I did check extent on Wiki.
> I see "extent" when talking about snapshot and export/import.
> For example, when create a snapshot, we mark extents. When
> there is write to marked extents, we will make a copy.
> I also know that user data on block device maps to objects.

Hi Tony,

An extent as simply a byte range, often defined by an offset where it
starts and a length (denoted as offset~length).

> How "extent" and "object" are related?

In the context of RBD there are several types of extents, but the two
main ones are image extents and object extents.  An image extent is at
an offset into the image, while an object extent is at an offset into
a particular object.  Any range in the image can be referred to using
either type of extent.  For example, assuming default striping:

  image extent 9437184~4096

and

  object 2 extent 1048576~4096

refer to the same 4096-byte block (objects are 0-indexed).  Similarly:

  image extent 12578816~8192

and

  object 2 extent 4190208~4096
  object 3 extent 0~4096

refer to the same 8192-byte block, but in this case the image extent
corresponds to two object extents.

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Denis Polom]

it's:

"bluestore_compression_algorithm": "snappy"

"bluestore_compression_mode": "none"


On 11/27/23 20:13, Wesley Dillingham wrote:

How about these two options:

bluestore_compression_algorithm
bluestore_compression_mode

Thanks.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 2:01 PM Denis Polom  wrote:

Hi,

no we don't:

"bluestore_rocksdb_options":

"compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",

thx

On 11/27/23 19:17, Wesley Dillingham wrote:

Curious if you are using bluestore compression?

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 10:09 AM Denis Polom
 wrote:

Hi

we have issue to start some OSDs on one node on our Ceph
Quincy 17.2.7
cluster. Some OSDs on that node are running fine, but some
failing to start.

Looks like crc32 checksum error, and failing to get OSD map.
I found a
some discussions on that but nothing helped.

I've also tried to insert current OSD map but that ends with
error:

# CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
--data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap
--file osdmap
osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.

Log is bellow

Any ideas please?

Thank you


 From log file:

2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to
public
interface

2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad
crc32c/0x1000
checksum at blob offset 0x0, got 0xb1701b42, expected
0x9ee5ece2, device
location [0x1~1000], logical extent 0x0~1000, object
#-1:7b3f43c4:::osd_superblock:0#

2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed
to load
OSD map for epoch 927580, got 0 bytes

/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
2023-11-27T16:01:51.443522+0100
/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
  ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
(stable)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int,
char
const*)+0x14f) [0x561ad07d2624]
  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
  3: (OSD::init()+0x4026) [0x561ad08e5a86]
  4: main()
  5: __libc_start_main()
  6: _start()
*** Caught signal (Aborted) **
  in thread 7f3f17aa13c0 thread_name:ceph-osd
2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
2023-11-27T16:01:51.443522+0100
/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)

  ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
(stable)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int,
char
const*)+0x14f) [0x561ad07d2624]
  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
  3: (OSD::init()+0x4026) [0x561ad08e5a86]
  4: main()
  5: __libc_start_main()
  6: _start()


  ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
(stable)
  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)
[0x7f3f1814b420]
  2: gsignal()
  3: abort()
  4: (ceph::__ceph_assert_fail(char const*, char const*, int,
char
const*)+0x1b7) [0x561ad07d268c]
  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
  6: (OSD::init()+0x4026) [0x561ad08e5a86]
  7: main()
  8: __libc_start_main()
  9: _start()
2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught
signal (Aborted) **
  in thread 7f3f17aa13c0 thread_name:ceph-osd

  ceph version 17.2.7
(b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
(stable)
  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)
[0x7f3f1814b420]
  2: gsignal()
  3: abort()
  4: (ceph::__ceph_assert_fail(char const*, char const*, int,
char
const*)+0x1b7) [0x561ad07d268c]
  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
  6: (OSD::init()+0x4026) [0x561ad08e5a86]
  7: main()
  8: __libc_start_main()
  9: _start()
 

[ceph-users] Re: Rook-Ceph OSD Deployment Error

[Travis Nielsen]

Sounds like you're hitting a known issue with v17.2.7.
https://github.com/rook/rook/issues/13136

The fix will be in v18.2.1 if it's an option to upgrade to Reef. If not,
you'll need to use v17.2.6 until the fix comes out for quincy in v17.2.8.

Travis

On Thu, Nov 23, 2023 at 4:06 PM P Wagner-Beccard <
wagner-kerschbau...@schaffroth.eu> wrote:

> Hi Mailing-Lister's,
>
> I am reaching out for assistance regarding a deployment issue I am facing
> with Ceph on a 4 node RKE2 cluster. We are attempting to deploy Ceph via
> the rook helm chart, but we are encountering an issue that apparently seems
> related to a known bug (https://tracker.ceph.com/issues/61597).
>
> During the OSD preparation phase, the deployment consistently fails with an
> IndexError: list index out of range. The logs indicate a problem occurs
> when configuring new Disks, specifically using /dev/dm-3 as a metadata
> device. It's important to note that /dev/dm-3 is an LVM on top of an mdadm
> RAID, which might or might not be contributing to this issue. (I swear,
> this setup worked already)
>
> Here is a snippet of the error from the deployment logs:
> > 2023-11-23 23:11:30.196913 D | exec: IndexError: list index out of range
> > 2023-11-23 23:11:30.236962 C | rookcmd: failed to configure devices:
> failed to initialize osd: failed ceph-volume report: exit status 1
> https://paste.openstack.org/show/bileqRFKbolrBlTqszmC/
>
> We have attempted different configurations, including specifying devices
> explicitly and using the useAllDevices: true option with a specified
> metadata device (/dev/dm-3 or the /dev/pv_md0/lv_md0 path). However, the
> issue persists across multiple configurations.
>
> tested configurations are as follows:
>
> Explicit device specification:
>
> ```yaml
> nodes:
>   - name: "ceph01.maas"
> devices:
>   - name: /dev/dm-1
>   - name: /dev/dm-2
>   - name: "sdb"
> config:
>   metadataDevice: "/dev/dm-3"
>   - name: "sdc"
> config:
>   metadataDevice: "/dev/dm-3"
> ```
>
> General device specification with metadata device:
> ```yaml
> storage:
>   useAllNodes: true
>   useAllDevices: true
>   config:
> metadataDevice: /dev/dm-3
> ```
>
> I would greatly appreciate any insights or recommendations on how to
> proceed or work around this issue.
> Is there a halfway decent way to apply the fix or maybe a workaround that
> we can apply to successfully deploy Ceph in our environment?
>
> Kind regards,
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Wesley Dillingham]

What I was getting at was to see if this was somehow related to the bug
described here https://www.youtube.com/watch?v=_4HUR00oCGo

It should not be given the version of ceph you are using but the CRC error
you are seeing is similar.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 2:19 PM Anthony D'Atri  wrote:

> The options Wes listed are for data, not RocksDB.
>
> > On Nov 27, 2023, at 1:59 PM, Denis Polom  wrote:
> >
> > Hi,
> >
> > no we don't:
> >
> > "bluestore_rocksdb_options":
> "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",
> >
> > thx
> >
> > On 11/27/23 19:17, Wesley Dillingham wrote:
> >> Curious if you are using bluestore compression?
> >>
> >> Respectfully,
> >>
> >> *Wes Dillingham*
> >> w...@wesdillingham.com
> >> LinkedIn 
> >>
> >>
> >> On Mon, Nov 27, 2023 at 10:09 AM Denis Polom 
> wrote:
> >>
> >>Hi
> >>
> >>we have issue to start some OSDs on one node on our Ceph Quincy
> >>17.2.7
> >>cluster. Some OSDs on that node are running fine, but some failing
> >>to start.
> >>
> >>Looks like crc32 checksum error, and failing to get OSD map. I
> >>found a
> >>some discussions on that but nothing helped.
> >>
> >>I've also tried to insert current OSD map but that ends with error:
> >>
> >># CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
> >>--data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap
> >>osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.
> >>
> >>Log is bellow
> >>
> >>Any ideas please?
> >>
> >>Thank you
> >>
> >>
> >> From log file:
> >>
> >>2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public
> >>interface
> >>
> >>2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
> >>bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
> >>checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
> >>device
> >>location [0x1~1000], logical extent 0x0~1000, object
> >>#-1:7b3f43c4:::osd_superblock:0#
> >>
> >>2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load
> >>OSD map for epoch 927580, got 0 bytes
> >>
> >>/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> >>OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> >>2023-11-27T16:01:51.443522+0100
> >>/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
> >>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>quincy
> >>(stable)
> >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>const*)+0x14f) [0x561ad07d2624]
> >>  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>  3: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>  4: main()
> >>  5: __libc_start_main()
> >>  6: _start()
> >>*** Caught signal (Aborted) **
> >>  in thread 7f3f17aa13c0 thread_name:ceph-osd
> >>2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
> >>/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> >>OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> >>2023-11-27T16:01:51.443522+0100
> >>/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
> >>
> >>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>quincy
> >>(stable)
> >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>const*)+0x14f) [0x561ad07d2624]
> >>  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>  3: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>  4: main()
> >>  5: __libc_start_main()
> >>  6: _start()
> >>
> >>
> >>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>quincy
> >>(stable)
> >>  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
> >>  2: gsignal()
> >>  3: abort()
> >>  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>const*)+0x1b7) [0x561ad07d268c]
> >>  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>  6: (OSD::init()+0x4026) [0x561ad08e5a86]
> >>  7: main()
> >>  8: __libc_start_main()
> >>  9: _start()
> >>2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal
> >>(Aborted) **
> >>  in thread 7f3f17aa13c0 thread_name:ceph-osd
> >>
> >>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
> >>quincy
> >>(stable)
> >>  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
> >>  2: gsignal()
> >>  3: abort()
> >>  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >>const*)+0x1b7) [0x561ad07d268c]
> >>  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
> >>  

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Anthony D'Atri]

The options Wes listed are for data, not RocksDB.

> On Nov 27, 2023, at 1:59 PM, Denis Polom  wrote:
> 
> Hi,
> 
> no we don't:
> 
> "bluestore_rocksdb_options": 
> "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",
> 
> thx
> 
> On 11/27/23 19:17, Wesley Dillingham wrote:
>> Curious if you are using bluestore compression?
>> 
>> Respectfully,
>> 
>> *Wes Dillingham*
>> w...@wesdillingham.com
>> LinkedIn 
>> 
>> 
>> On Mon, Nov 27, 2023 at 10:09 AM Denis Polom  wrote:
>> 
>>Hi
>> 
>>we have issue to start some OSDs on one node on our Ceph Quincy
>>17.2.7
>>cluster. Some OSDs on that node are running fine, but some failing
>>to start.
>> 
>>Looks like crc32 checksum error, and failing to get OSD map. I
>>found a
>>some discussions on that but nothing helped.
>> 
>>I've also tried to insert current OSD map but that ends with error:
>> 
>># CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
>>--data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap
>>osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.
>> 
>>Log is bellow
>> 
>>Any ideas please?
>> 
>>Thank you
>> 
>> 
>> From log file:
>> 
>>2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public
>>interface
>> 
>>2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
>>bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
>>checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
>>device
>>location [0x1~1000], logical extent 0x0~1000, object
>>#-1:7b3f43c4:::osd_superblock:0#
>> 
>>2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load
>>OSD map for epoch 927580, got 0 bytes
>> 
>>/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
>>OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
>>2023-11-27T16:01:51.443522+0100
>>/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
>>quincy
>>(stable)
>>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>const*)+0x14f) [0x561ad07d2624]
>>  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>  3: (OSD::init()+0x4026) [0x561ad08e5a86]
>>  4: main()
>>  5: __libc_start_main()
>>  6: _start()
>>*** Caught signal (Aborted) **
>>  in thread 7f3f17aa13c0 thread_name:ceph-osd
>>2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
>>/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
>>OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
>>2023-11-27T16:01:51.443522+0100
>>/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>> 
>>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
>>quincy
>>(stable)
>>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>const*)+0x14f) [0x561ad07d2624]
>>  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>  3: (OSD::init()+0x4026) [0x561ad08e5a86]
>>  4: main()
>>  5: __libc_start_main()
>>  6: _start()
>> 
>> 
>>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
>>quincy
>>(stable)
>>  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>>  2: gsignal()
>>  3: abort()
>>  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>const*)+0x1b7) [0x561ad07d268c]
>>  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>  6: (OSD::init()+0x4026) [0x561ad08e5a86]
>>  7: main()
>>  8: __libc_start_main()
>>  9: _start()
>>2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal
>>(Aborted) **
>>  in thread 7f3f17aa13c0 thread_name:ceph-osd
>> 
>>  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
>>quincy
>>(stable)
>>  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>>  2: gsignal()
>>  3: abort()
>>  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>const*)+0x1b7) [0x561ad07d268c]
>>  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>  6: (OSD::init()+0x4026) [0x561ad08e5a86]
>>  7: main()
>>  8: __libc_start_main()
>>  9: _start()
>>  NOTE: a copy of the executable, or `objdump -rdS ` is
>>needed to interpret this.
>> 
>> 
>>   -558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to
>>public interface
>> 
>> -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
>>bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
>>checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
>>device
>>location [0x1~1000], logical extent 0x0~1000, object
>>

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Wesley Dillingham]

How about these two options:

bluestore_compression_algorithm
bluestore_compression_mode

Thanks.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 2:01 PM Denis Polom  wrote:

> Hi,
>
> no we don't:
>
> "bluestore_rocksdb_options":
> "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",
> thx
>
> On 11/27/23 19:17, Wesley Dillingham wrote:
>
> Curious if you are using bluestore compression?
>
> Respectfully,
>
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn 
>
>
> On Mon, Nov 27, 2023 at 10:09 AM Denis Polom  wrote:
>
>> Hi
>>
>> we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7
>> cluster. Some OSDs on that node are running fine, but some failing to
>> start.
>>
>> Looks like crc32 checksum error, and failing to get OSD map. I found a
>> some discussions on that but nothing helped.
>>
>> I've also tried to insert current OSD map but that ends with error:
>>
>> # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
>> --data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap
>> osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.
>>
>> Log is bellow
>>
>> Any ideas please?
>>
>> Thank you
>>
>>
>>  From log file:
>>
>> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public
>> interface
>>
>> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
>> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
>> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device
>> location [0x1~1000], logical extent 0x0~1000, object
>> #-1:7b3f43c4:::osd_superblock:0#
>>
>> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load
>> OSD map for epoch 927580, got 0 bytes
>>
>> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
>> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
>> 2023-11-27T16:01:51.443522+0100
>> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>> (stable)
>>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x14f) [0x561ad07d2624]
>>   2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>   3: (OSD::init()+0x4026) [0x561ad08e5a86]
>>   4: main()
>>   5: __libc_start_main()
>>   6: _start()
>> *** Caught signal (Aborted) **
>>   in thread 7f3f17aa13c0 thread_name:ceph-osd
>> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
>> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
>> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
>> 2023-11-27T16:01:51.443522+0100
>> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>>
>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>> (stable)
>>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x14f) [0x561ad07d2624]
>>   2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>   3: (OSD::init()+0x4026) [0x561ad08e5a86]
>>   4: main()
>>   5: __libc_start_main()
>>   6: _start()
>>
>>
>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>> (stable)
>>   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>>   2: gsignal()
>>   3: abort()
>>   4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x1b7) [0x561ad07d268c]
>>   5: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>   6: (OSD::init()+0x4026) [0x561ad08e5a86]
>>   7: main()
>>   8: __libc_start_main()
>>   9: _start()
>> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted)
>> **
>>   in thread 7f3f17aa13c0 thread_name:ceph-osd
>>
>>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
>> (stable)
>>   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>>   2: gsignal()
>>   3: abort()
>>   4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x1b7) [0x561ad07d268c]
>>   5: ceph-osd(+0xc2e836) [0x561ad07d2836]
>>   6: (OSD::init()+0x4026) [0x561ad08e5a86]
>>   7: main()
>>   8: __libc_start_main()
>>   9: _start()
>>   NOTE: a copy of the executable, or `objdump -rdS ` is
>> needed to interpret this.
>>
>>
>>-558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to
>> public interface
>>
>>  -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
>> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
>> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device
>> location [0x1~1000], logical extent 0x0~1000, object
>> #-1:7b3f43c4:::osd_superblock:0#
>>
>>  -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed
>> to load OSD map for epoch 927580, got 0 bytes
>>
>>  -1> 2023-11-27T16:01:51.443+0100 

[ceph-users] Re: About number of osd node can be failed with erasure code 3+2

[Wesley Dillingham]

With a k+m which is 3+2 each RADOS object is broken into 5 shards. By
default the pool will have a min_size of k+1 (4 in this case). Which means
you can lose 1 shard and still be >= min_size. If one host goes down and
you use a host-based failure domain (default) you will lose 1 shard out of
all PGs on that host. You will now be at min_size and so still
readable/writeable. If you lose another host you will now be below min_size
with 3 healthy shards for some subset of PG (those common to the 2 hosts)
will be inactive and therefore not read/writeable. As you can see, the
higher your M the more disks/hosts you can lose before dropping below
min_size.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 1:36 PM  wrote:

> Hi Groups,
>
> Recently I was setting up a ceph cluster with 10 nodes 144 osd, and I use
> S3 for it with pool erasure code EC3+2 on it.
>
> I have a question, how many osd nodes can fail with erasure code 3+2 with
> cluster working normal (read, write)? and can i choose better erasure code
> ec7+3, 8+2 etc..?
>
> With the erasure code algorithm, it only ensures no data loss, but does
> not guarantee that the cluster operates normally and does not block IO when
> osd nodes down. Is that right?
>
> Thanks to the community.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Denis Polom]

Hi,

no we don't:

"bluestore_rocksdb_options": 
"compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824",


thx

On 11/27/23 19:17, Wesley Dillingham wrote:

Curious if you are using bluestore compression?

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 10:09 AM Denis Polom  wrote:

Hi

we have issue to start some OSDs on one node on our Ceph Quincy
17.2.7
cluster. Some OSDs on that node are running fine, but some failing
to start.

Looks like crc32 checksum error, and failing to get OSD map. I
found a
some discussions on that but nothing helped.

I've also tried to insert current OSD map but that ends with error:

# CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
--data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap
osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.

Log is bellow

Any ideas please?

Thank you


 From log file:

2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public
interface

2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
device
location [0x1~1000], logical extent 0x0~1000, object
#-1:7b3f43c4:::osd_superblock:0#

2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load
OSD map for epoch 927580, got 0 bytes

/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
2023-11-27T16:01:51.443522+0100
/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
quincy
(stable)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x14f) [0x561ad07d2624]
  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
  3: (OSD::init()+0x4026) [0x561ad08e5a86]
  4: main()
  5: __libc_start_main()
  6: _start()
*** Caught signal (Aborted) **
  in thread 7f3f17aa13c0 thread_name:ceph-osd
2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
2023-11-27T16:01:51.443522+0100
/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)

  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
quincy
(stable)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x14f) [0x561ad07d2624]
  2: ceph-osd(+0xc2e836) [0x561ad07d2836]
  3: (OSD::init()+0x4026) [0x561ad08e5a86]
  4: main()
  5: __libc_start_main()
  6: _start()


  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
quincy
(stable)
  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
  2: gsignal()
  3: abort()
  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1b7) [0x561ad07d268c]
  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
  6: (OSD::init()+0x4026) [0x561ad08e5a86]
  7: main()
  8: __libc_start_main()
  9: _start()
2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal
(Aborted) **
  in thread 7f3f17aa13c0 thread_name:ceph-osd

  ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2)
quincy
(stable)
  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
  2: gsignal()
  3: abort()
  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1b7) [0x561ad07d268c]
  5: ceph-osd(+0xc2e836) [0x561ad07d2836]
  6: (OSD::init()+0x4026) [0x561ad08e5a86]
  7: main()
  8: __libc_start_main()
  9: _start()
  NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.


   -558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to
public interface

 -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2,
device
location [0x1~1000], logical extent 0x0~1000, object
#-1:7b3f43c4:::osd_superblock:0#

 -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0
failed
to load OSD map for epoch 927580, got 0 bytes

 -1> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
2023-11-27T16:01:51.443522+0100
/build/ceph-17.2.7/src/osd/OSD.h: 696: 

[ceph-users] Re: Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

[Dan van der Ster]

Hi Giuseppe,

There are likely one or two clients whose op is blocking the reconnect/replay.
If you increase debug_mds perhaps you can find the guilty client and
disconnect it / block it from mounting.

Or for a more disruptive recovery you can try this "Deny all reconnect
to clients " option:
https://docs.ceph.com/en/reef/cephfs/troubleshooting/#avoiding-recovery-roadblocks

Hope that helps,

Dan


On Mon, Nov 27, 2023 at 1:08 AM Lo Re Giuseppe  wrote:
>
> Hi,
> We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are 
> having CephFS issues.
> For example this morning:
> “””
> [root@naret-monitor01 ~]# ceph -s
>   cluster:
> id: 63334166-d991-11eb-99de-40a6b72108d0
> health: HEALTH_WARN
> 1 filesystem is degraded
> 3 clients failing to advance oldest client/flush tid
> 3 MDSs report slow requests
> 6 pgs not scrubbed in time
> 29 daemons have recently crashed
> …
> “””
>
> The ceph orch, ceph crash and ceph fs status commands were hanging.
>
> After a “ceph mgr fail” those commands started to respond.
> Then I have noticed that there was one mds with most of the slow operations,
>
> “””
> [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests
> mds.cephfs.naret-monitor01.nuakzo(mds.0): 18 slow requests are blocked > 
> 30 secs
> mds.cephfs.naret-monitor01.uvevbf(mds.1): 1683 slow requests are blocked 
> > 30 secs
> mds.cephfs.naret-monitor02.exceuo(mds.2): 1 slow requests are blocked > 
> 30 secs
> “””
>
> Then I tried to restart it with
>
> “””
> [root@naret-monitor01 ~]# ceph orch daemon restart 
> mds.cephfs.naret-monitor01.uvevbf
> Scheduled to restart mds.cephfs.naret-monitor01.uvevbf on host 
> 'naret-monitor01'
> “””
>
> After the cephfs entered into this situation:
> “””
> [root@naret-monitor01 ~]# ceph fs status
> cephfs - 198 clients
> ==
> RANK STATE   MDS  ACTIVITY DNS
> INOS   DIRS   CAPS
> 0   active cephfs.naret-monitor01.nuakzo  Reqs:0 /s  17.2k  16.2k 
>  1892   14.3k
> 1   active cephfs.naret-monitor02.ztdghf  Reqs:0 /s  28.1k  10.3k 
>   752   6881
> 2clientreplay  cephfs.naret-monitor02.exceuo 63.0k  6491  
>   541 66
> 3   active cephfs.naret-monitor03.lqppte  Reqs:0 /s  16.7k  13.4k 
>  8233990
>   POOL  TYPE USED  AVAIL
>cephfs.cephfs.meta metadata  5888M  18.5T
>cephfs.cephfs.data   data 119G   215T
> cephfs.cephfs.data.e_4_2data2289G  3241T
> cephfs.cephfs.data.e_8_3data9997G   470T
>  STANDBY MDS
> cephfs.naret-monitor03.eflouf
> cephfs.naret-monitor01.uvevbf
> MDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) 
> reef (stable)
> “””
>
> The file system is totally unresponsive (we can mount it on client nodes but 
> any operations like a simple ls hangs).
>
> During the night we had a lot of mds crashes, I can share the content.
>
> Does anybody have an idea on how to tackle this problem?
>
> Best,
>
> Giuseppe
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [MDS] mds stuck in laggy state, CephFS unusable

[kvesligaj]

Hi,

we're having a peculiar issue which we found out during HA/DR testing in our 
Ceph cluster.

Basic info about cluster: 
Version: Quincy (17.2.6)
5 nodes configured in stretch cluster (2 DCs with one arbiter node which is 
also admin node for the cluster)
On every node beside the admin node we have OSD and MON services. We have 3 MGR 
instances in cluster.

Specific thing that we wanted to test is multiple CephFS with each having 
multiple MDS (with HA in mind). 
We deployed MDS on every node, increased max_mds to 2 for every CephFS and 
other two MDS-es are in standby-replay mode (they are automatically configured 
during CephFS creation to follow specific CephFS - join_fscid).

We did multiple tests and when we have only one CephFS it behaves as expected 
(two MDS are in up:active state and clients can connect and interact with 
CephFS as if nothing has happened).

When we test with multiple CephFS (two for example) and we shutdown two nodes 
one of MDS is stuck in up:active laggy state and when this happens the CephFS 
for which this happens is unusable, client hangs and it is stuck like that 
until we power on other DC. This happens even when there are no clients 
connected to this specific CephFS.

We can provide additional logs and do any tests necessary. We checked the usual 
culprits and our nodes don't show any excessive CPU or memory usage.

We would appreciate any help.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] osdmaptool target & deviation calculation

[Robert Hish]

Question about the osdmaptool deviation calculations;

For instance, 

-
osdmaptool omap --upmap output.txt --upmap-pool cephfs_data-rep3 --upmap-max 
1000 --upmap-deviation 5 
osdmaptool: osdmap file 'omap'
writing upmap command output to: output.txt
checking for upmap cleanups
upmap, max-count 1000, max deviation 5
 limiting to pools cephfs_data-rep3 ([30])
pools cephfs_data-rep3 
prepared 0/1000 changes
Unable to find further optimization, or distribution is already perfect
-

The evaluated pool is all-on-hdd, and the pool was created with PGS > number of 
hdd OSDs in the cluster. So each hdd OSD is being used at least once by this 
pool.

Is it correct to assume that the osdmaptool is relying on the equations set at
ceph-17.2.5/src/osd/OSDMap.cc:5143

5143   // This function calculates the 2 maps osd_deviation and deviation_osd 
which 
5144   // hold the deviation between the current number of PGs which map to an 
OSD 
5145   // and the optimal number. ...

# pgs_per_weight
# ceph-17.2.5/src/osd/OSDMap.cc:4806
4806   float pgs_per_weight = total_pgs / osd_weight_total;

# target
# ceph-17.2.5/src/osd/OSDMap.cc:5156
5156 float target = osd_weight.at(oid) * pgs_per_weight;

# deviation
# ceph-17.2.5/src/osd/OSDMap.cc:5157
5157 float deviation = (float)opgs.size() - target;

And so for pgs_per_weight I calculate

ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") .pgs ] | 
add'
divided by
ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") 
.crush_weight ] | add'

(each hdd OSD in this cluster has identical weight)

target = osd_weight.at(oid) * pgs_per_weight

I calculate deviation for each osd

deviation = opgs.size - target
where, opgs.size = the number of PGs at an OSD. i.e. The value of $19 for each 
$1, in  `ceph osd df hdd | awk '{ print $1 " " $19 }'`

The result is many many OSDs with a deviation well above the 
upmap_max_deviation which is at default: 5

So I am wondering if I am miscalculating something, or if I'm not aware of 
further things the osdmaptool is considering when formulating upmap suggestions?

-Robert
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] About number of osd node can be failed with erasure code 3+2

[tranphong079]

Hi Groups,

Recently I was setting up a ceph cluster with 10 nodes 144 osd, and I use S3 
for it with pool erasure code EC3+2 on it.

I have a question, how many osd nodes can fail with erasure code 3+2 with 
cluster working normal (read, write)? and can i choose better erasure code 
ec7+3, 8+2 etc..?

With the erasure code algorithm, it only ensures no data loss, but does not 
guarantee that the cluster operates normally and does not block IO when osd 
nodes down. Is that right?

Thanks to the community.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CloudStack and Ceph Day 2024

[42on - Michiel Manten]

Hello Ceph users,

Together with ShapeBlue and Adyen, 42on is organizing a CloudStack and Ceph 
Day; this time in Amsterdam, The Netherlands. We are planning this for February 
8 | 2024.

We want to create a technical event that shares updates on both technologies, 
as well as 'use cases', stories, challenges and perhaps some crazy ideas or 
configurations. Let’s share information and make Ceph even bigger and better!

I am still looking for some speakers who would love to share something about 
their Ceph infrastructure or configuration. As small as it might seem to you, 
any ideas are welcome and if you are in doubt about the subject, please message 
or call me to discuss. I would love to hear about your ideas.

To RSVP and more information can be found here: 
https://www.eventbrite.nl/e/cloudstack-and-ceph-day-netherlands-2024-tickets-700177167757?utm-campaign=social=attendeeshare=discovery=listing=cp=ebdsshcopyurl

We hope to see you in Amsterdam once more.

Sincerely,

Michiel Manten
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW public HA traffic - best practices?

[Félix Barbeira]

An easy setup if you use PowerDNS is to establish LUA records on the gateway:

https://doc.powerdns.com/authoritative/lua-records/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How balancer module balance data

[bryansoong21]

Hello,

We are running a pacific 16.2.10 cluster and enabled the balancer module, here 
is the configuration:

[root@ceph-1 ~]# ceph balancer status
{
"active": true,
"last_optimize_duration": "0:00:00.052548",
"last_optimize_started": "Fri Nov 17 17:09:57 2023",
"mode": "upmap",
"optimize_result": "Unable to find further optimization, or pool(s) pg_num 
is decreasing, or distribution is already perfect",
"plans": []
}
[root@ceph-1 ~]# ceph balancer eval
current cluster score 0.017742 (lower is better)

Here is the balancer configuration of upmap_max_deviation:
# ceph config get mgr mgr/balancer/upmap_max_deviation
5

We have two different types of OSDS, one is 7681G and another is 3840G. When I 
checked our PG distribution on each type of OSD, I found the PG distribution is 
not evenly, for the 7681G OSDs, the OSD distribution varies from 136 to 158; 
while for the 3840G OSDs, it varies from 60 to 83, seems the 
upmap_max_deviation is almost +/- 10. So I just wondering if this is expected 
or do I need to change the upmap_max_deviation to a smaller value.

Thanks for answering my question.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] blustore osd nearfull but no pgs on it

[Debian]

Hi,

after a massive rebalance(tunables) my small SSD-OSDs are getting full, 
i changed my crush rules so there are actual no pgs/pools on it, but the 
disks stay full:


ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus 
(stable)


ID CLASS WEIGHT REWEIGHT SIZE    RAW USE DATA    OMAP META 
AVAIL    %USE  VAR  PGS STATUS TYPE NAME
158   ssd    0.21999  1.0 224 GiB 194 GiB 193 GiB  22 MiB 1002 MiB   
30 GiB 86.68 1.49   0 up osd.158


inferring bluefs devices from bluestore path
1 : device size 0x37e440 : own 0x[1ad3f0~23c60] = 
0x23c60 : using 0x3963(918 MiB) : bluestore has 0x46e2d(18 
GiB) available


when i recreate the osd the osd gets full again

any suggestion?

thx & best regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph-volume lvm new-db throws errors

[Giuliano Maggi]

Hi,

ceph-volume lvm new-db does not work as expected due to missing 
/var/lib/ceph/osd/ceph-OSDID. However, the database device seems to be added to 
the OSD.
Until now, it is unclear to me if the DB device was actually successfully added 
to the OSD or this is a bug in ceph version 17.2.7

The detailed explanation  is in a github ticket: 
https://github.com/ceph/ceph/pull/52941#issuecomment-1814251890

does anyone else have this issue ?
Any feedback is welcome of course.

Thanks,
Giuliano.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] What is the maximum number of Rados gateway objects in one cluster using the bucket index and in one bucket?

[steve jung]

Hello. 

We are using Ceph storage to test whether we can run the service by uploading 
and saving more than 40 billion files.

So I'd like to check the contents below.

1) Maximum number of Rados gateway objects that can be stored in one cluster 
using the bucket index
2) Maximum number of Rados gateway objects that can be stored in one bucket

Although we have referred to the limitations on the number of Rados gateway 
objects mentioned in existing documents, it seems theoretically unlimited

If you have operated the number of files at the level we think in actual 
services or products, we would appreciate it if you could share them.

Below are related documents and related settings values.

> Related documents
- https://documentation.suse.com/ses/5.5/html/ses-all/cha-ceph-gw.html
- 
https://www.ibm.com/docs/en/storage-ceph/6?topic=resharding-limitations-bucket-index
- https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/

> Related config
- rgw_dynamic_resharding: true
- rgw_max_objs_per_shard: 10
- rgw_max_dynamic_shards : 65521
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs failing to start due to crc32 and osdmap error

[Wesley Dillingham]

Curious if you are using bluestore compression?

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Nov 27, 2023 at 10:09 AM Denis Polom  wrote:

> Hi
>
> we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7
> cluster. Some OSDs on that node are running fine, but some failing to
> start.
>
> Looks like crc32 checksum error, and failing to get OSD map. I found a
> some discussions on that but nothing helped.
>
> I've also tried to insert current OSD map but that ends with error:
>
> # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool
> --data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap
> osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.
>
> Log is bellow
>
> Any ideas please?
>
> Thank you
>
>
>  From log file:
>
> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public
> interface
>
> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device
> location [0x1~1000], logical extent 0x0~1000, object
> #-1:7b3f43c4:::osd_superblock:0#
>
> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load
> OSD map for epoch 927580, got 0 bytes
>
> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> 2023-11-27T16:01:51.443522+0100
> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
> (stable)
>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x14f) [0x561ad07d2624]
>   2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>   3: (OSD::init()+0x4026) [0x561ad08e5a86]
>   4: main()
>   5: __libc_start_main()
>   6: _start()
> *** Caught signal (Aborted) **
>   in thread 7f3f17aa13c0 thread_name:ceph-osd
> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> 2023-11-27T16:01:51.443522+0100
> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>
>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
> (stable)
>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x14f) [0x561ad07d2624]
>   2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>   3: (OSD::init()+0x4026) [0x561ad08e5a86]
>   4: main()
>   5: __libc_start_main()
>   6: _start()
>
>
>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
> (stable)
>   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>   2: gsignal()
>   3: abort()
>   4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x1b7) [0x561ad07d268c]
>   5: ceph-osd(+0xc2e836) [0x561ad07d2836]
>   6: (OSD::init()+0x4026) [0x561ad08e5a86]
>   7: main()
>   8: __libc_start_main()
>   9: _start()
> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) **
>   in thread 7f3f17aa13c0 thread_name:ceph-osd
>
>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
> (stable)
>   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
>   2: gsignal()
>   3: abort()
>   4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x1b7) [0x561ad07d268c]
>   5: ceph-osd(+0xc2e836) [0x561ad07d2836]
>   6: (OSD::init()+0x4026) [0x561ad08e5a86]
>   7: main()
>   8: __libc_start_main()
>   9: _start()
>   NOTE: a copy of the executable, or `objdump -rdS ` is
> needed to interpret this.
>
>
>-558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to
> public interface
>
>  -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1
> bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000
> checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device
> location [0x1~1000], logical extent 0x0~1000, object
> #-1:7b3f43c4:::osd_superblock:0#
>
>  -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed
> to load OSD map for epoch 927580, got 0 bytes
>
>  -1> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1
> /build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef
> OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time
> 2023-11-27T16:01:51.443522+0100
> /build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
>
>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy
> (stable)
>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x14f) [0x561ad07d2624]
>   2: ceph-osd(+0xc2e836) [0x561ad07d2836]
>   3: (OSD::init()+0x4026) [0x561ad08e5a86]
>   4: main()
>   5: __libc_start_main()
>   6: _start()
>
>
>   0> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal
> (Aborted) **
>   in thread 7f3f17aa13c0 thread_name:ceph-osd
>
>   ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy

[ceph-users] Re: MDS_DAMAGE in 17.2.7 / Cannot delete affected files

[Patrick Donnelly]

Hello Sebastian,

On Fri, Nov 24, 2023 at 8:49 AM Sebastian Knust
 wrote:
>
> Hi,
>
> After updating from 17.2.6 to 17.2.7 with cephadm, our cluster went into
> MDS_DAMAGE state. We had some prior issues with faulty kernel clients
> not releasing capabilities, therefore the update might just be a
> coincidence.
>
> `ceph tell mds.cephfs:0 damage ls` lists 56 affected files all with
> these general details:
>
> {
>  "damage_type": "dentry",
>  "id": 123456,
>  "ino": 1234567890,
>  "frag": "*",
>  "dname": "some-filename.ext",
>  "snap_id": "head",
>  "path": "/full/path/to/file"
> }
>
> The behaviour upon trying to access file information in the (Kernel
> mounted) filesystem is a bit inconsistent. Generally, the first `stat`
> call seems to result in "Input/output error", the next call provides all
> `stat` data as expected from an undamaged file. The file can be read
> with `cat` with full and correct content (verified with backup) once the
> stat call succeeds.
>
> Scrubbing the affected subdirectories with `ceph tell mds.cephfs:0 scrub
> start /path/to/dir/ recursive,repair,force` does not fix the issue.
>
> Trying to delete the file results in an "Input/output error". If the
> stat calls beforehand succeeded, this also crashes the active MDS with
> these messages in the system journal:
> > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: 
> > mds.0.cache.den(0x10012271195 DisplaySettings.json) newly corrupt dentry to 
> > be committed: [dentry 
> > #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
> >  [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 
> > state=1073741824 | inodepin=1 0x56413e1e2780]
> > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: log_channel(cluster) 
> > log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry 
> > #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
> >  [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012271197 
> > state=1073741824 | inodepin=1 0x56413e1e2780]
> > Nov 24 14:21:15 iceph-18.servernet 
> > ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
> >  2023-11-24T13:21:15.654+ 7f3fdcde0700 -1 mds.0.cache.den(0x10012271195 
> > DisplaySettings.json) newly corrupt dentry to be committed: [dentry 
> > #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
> >  [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x1001>
> > Nov 24 14:21:15 iceph-18.servernet 
> > ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
> >  2023-11-24T13:21:15.654+ 7f3fdcde0700 -1 log_channel(cluster) log 
> > [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry 
> > #0x1/homes/huser/d3data/transfer/hortkrass/FLIMSIM/2023-04-12-irf-characterization/2-qwp-no-extra-filter-pc-off-tirf-94-tirf-cursor/DisplaySettings.json
> >  [1000275c4a0,head] auth (dversion lock) pv=0 v=225 ino=0x10012>
> > Nov 24 14:21:15 iceph-18.servernet 
> > ceph-eafd0514-3644-11eb-bc6a-3cecef2330fa-mds-cephfs-iceph-18-ujfqnd[1946838]:
> >  
> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
> >  In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 
> > time 2023-11-24T13:21:15.655088+
> > Nov 24 14:21:15 iceph-18.servernet ceph-mds[1946861]: 
> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
> >  In function 'void MDSRank::abort(std::string_view)' thread 7f3fdcde0700 
> > time 2023-11-24T13:21:15.655088+
> >   
> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDSRank.cc:
> >  937: ceph_abort_msg("abort() called")
> >
> >ceph version 17.2.7 
> > (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)
> >1: 
> > (ceph::__ceph_abort(char const*, int, char const*, 
> > std::__cxx11::basic_string, 
> > std::allocator > const&)+0xd7) [0x7f3fe5a1cb03]
> >2: 
> > (MDSRank::abort(std::basic_string_view 
> > >)+0x7d) [0x5640f2e6fa2d]
> >3: 
> > (CDentry::check_corruption(bool)+0x740) 

[ceph-users] Re: Does cephfs ensure close-to-open consistency after enabling lazyio?

[Patrick Donnelly]

No. You must call lazyio_synchronize.

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OSDs failing to start due to crc32 and osdmap error

[Denis Polom]

Hi

we have issue to start some OSDs on one node on our Ceph Quincy 17.2.7 
cluster. Some OSDs on that node are running fine, but some failing to start.


Looks like crc32 checksum error, and failing to get OSD map. I found a 
some discussions on that but nothing helped.


I've also tried to insert current OSD map but that ends with error:

# CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool 
--data-path /var/lib/ceph/osd/ceph-888/ --op set-osdmap --file osdmap

osdmap (#-1:20684533:::osdmap.931991:0#) does not exist.

Log is bellow

Any ideas please?

Thank you


From log file:

2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to public 
interface


2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 
bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 
checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device 
location [0x1~1000], logical extent 0x0~1000, object 
#-1:7b3f43c4:::osd_superblock:0#


2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed to load 
OSD map for epoch 927580, got 0 bytes


/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef 
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 
2023-11-27T16:01:51.443522+0100

/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)
 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy 
(stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x14f) [0x561ad07d2624]

 2: ceph-osd(+0xc2e836) [0x561ad07d2836]
 3: (OSD::init()+0x4026) [0x561ad08e5a86]
 4: main()
 5: __libc_start_main()
 6: _start()
*** Caught signal (Aborted) **
 in thread 7f3f17aa13c0 thread_name:ceph-osd
2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 
/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef 
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 
2023-11-27T16:01:51.443522+0100

/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)

 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy 
(stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x14f) [0x561ad07d2624]

 2: ceph-osd(+0xc2e836) [0x561ad07d2836]
 3: (OSD::init()+0x4026) [0x561ad08e5a86]
 4: main()
 5: __libc_start_main()
 6: _start()


 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy 
(stable)

 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x1b7) [0x561ad07d268c]

 5: ceph-osd(+0xc2e836) [0x561ad07d2836]
 6: (OSD::init()+0x4026) [0x561ad08e5a86]
 7: main()
 8: __libc_start_main()
 9: _start()
2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal (Aborted) **
 in thread 7f3f17aa13c0 thread_name:ceph-osd

 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy 
(stable)

 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x1b7) [0x561ad07d268c]

 5: ceph-osd(+0xc2e836) [0x561ad07d2836]
 6: (OSD::init()+0x4026) [0x561ad08e5a86]
 7: main()
 8: __libc_start_main()
 9: _start()
 NOTE: a copy of the executable, or `objdump -rdS ` is 
needed to interpret this.



  -558> 2023-11-27T16:01:47.691+0100 7f3f17aa13c0 -1 Falling back to 
public interface


    -5> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 
bluestore(/var/lib/ceph/osd/ceph-888) _verify_csum bad crc32c/0x1000 
checksum at blob offset 0x0, got 0xb1701b42, expected 0x9ee5ece2, device 
location [0x1~1000], logical extent 0x0~1000, object 
#-1:7b3f43c4:::osd_superblock:0#


    -2> 2023-11-27T16:01:51.439+0100 7f3f17aa13c0 -1 osd.888 0 failed 
to load OSD map for epoch 927580, got 0 bytes


    -1> 2023-11-27T16:01:51.443+0100 7f3f17aa13c0 -1 
/build/ceph-17.2.7/src/osd/OSD.h: In function 'OSDMapRef 
OSDService::get_map(epoch_t)' thread 7f3f17aa13c0 time 
2023-11-27T16:01:51.443522+0100

/build/ceph-17.2.7/src/osd/OSD.h: 696: FAILED ceph_assert(ret)

 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy 
(stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x14f) [0x561ad07d2624]

 2: ceph-osd(+0xc2e836) [0x561ad07d2836]
 3: (OSD::init()+0x4026) [0x561ad08e5a86]
 4: main()
 5: __libc_start_main()
 6: _start()


 0> 2023-11-27T16:01:51.447+0100 7f3f17aa13c0 -1 *** Caught signal 
(Aborted) **

 in thread 7f3f17aa13c0 thread_name:ceph-osd

 ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy 
(stable)

 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f1814b420]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x1b7) [0x561ad07d268c]

 5: ceph-osd(+0xc2e836) [0x561ad07d2836]
 6: (OSD::init()+0x4026) [0x561ad08e5a86]
 7: main()
 8: __libc_start_main()
 9: _start()
 NOTE: a copy of the executable, or `objdump -rdS ` is 
needed to interpret this.



  -562> 

[ceph-users] MDS stuck in up:rejoin

[Eric Tittley]

Hi all,

For about a week our CephFS has experienced issues with its MDS.

Currently the MDS is stuck in "up:rejoin"

Issues become apparent when simple commands like "mv foo bar/" hung.

I unmounted CephFS offline on the clients, evicted those remaining, and then 
issued

ceph config set mds.0 mds_wipe_sessions true
ceph config set mds.1 mds_wipe_sessions true

which allowed me to delete the hung requests.

I've lost the exact commands I used, but something like
rados -p cephfs_metadata ls | grep mds
rados rm -p cephfs_metadata mds0_openfiles.0

etc

This allowed the MDS to get to "up:rejoin" where it has been stuck ever since 
which is getting on five days.

# ceph mds stat
cephfs:1/1 {0=cephfs.ceph00.uvlkrw=up:rejoin} 2 up:standby



root@ceph00:/var/log/ceph/a614303a-5eb5-11ed-b492-011f01e12c9a# ceph -s
 cluster:
   id: a614303a-5eb5-11ed-b492-011f01e12c9a
   health: HEALTH_WARN
   1 filesystem is degraded
   1 pgs not deep-scrubbed in time
   2 pool(s) do not have an application enabled
   1 daemons have recently crashed

 services:
   mon: 3 daemons, quorum ceph00,ceph01,ceph02 (age 57m)
   mgr: ceph01.lvdgyr(active, since 2h), standbys: ceph00.gpwpgs
   mds: 1/1 daemons up, 2 standby
   osd: 91 osds: 90 up (since 78m), 90 in (since 112m)

 data:
   volumes: 0/1 healthy, 1 recovering
   pools:   5 pools, 1539 pgs
   objects: 138.83M objects, 485 TiB
   usage:   971 TiB used, 348 TiB / 1.3 PiB avail
   pgs: 1527 active+clean
12   active+clean+scrubbing+deep

 io:
   client:   3.1 MiB/s rd, 3.16k op/s rd, 0 op/s wr


# ceph --version
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)


I've tried failing the MDS so it switches.  Rebooted a couple of times.
I've added more OSDs to the metadata pool and took one out as I thought it might be a bad 
metadata OSD (The "recently crashed" daemon).

The error logs are full of
(prefix to all are:
Nov 27 14:02:44 ceph00 bash[2145]: debug 2023-11-27T14:02:44.619+ 7f74e845e700 
 1 -- [v2:192.168.1.128:6800/2157301677,v1:192.168.1.128:6801/2157301677] --> 
[v2:192.168.1.133:6896/4289132926,v1:192.168.1.133:6897/4289132926]
)

crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 
tx=0).send_message enqueueing message m=0x559be00adc00 type=42 
osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.:head [getxattr parent 
in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio 
e32465) v8
crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 
tx=0).write_message sending message m=0x559be00adc00 seq=8142643 
osd_op(mds.0.36244:8142873 3.ff 3:ff5b34d6:::1.:head [getxattr parent 
in=6b] snapc 0=[] ondisk+read+known_if_redirected+full_force+supports_pool_eio 
e32465) v8
crc :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 
tx=0).handle_message got 154 + 0 + 30 byte message. envelope type=43 src osd.89 
off 0
crc :-1 s=READ_MESSAGE_COMPLETE pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp 
rx=0 tx=0).handle_message received message m=0x559be01f4480 seq=8142643 
from=osd.89 type=43 osd_op_reply(8142873 1. [getxattr (30) out=30b] 
v0'0 uv560123 ondisk = 0) v8
osd_op_reply(8142873 1. [getxattr (30) out=30b] v0'0 uv560123 ondisk = 
0) v8  154+0+30 (crc 0 0 0) 0x559be01f4480 con 0x559be00ad800
osd_op(unknown.0.36244:8142874 3.ff 3:ff5b34d6:::1.:head [getxattr 
parent in=6b] snapc 0=[] 
ondisk+read+known_if_redirected+full_force+supports_pool_eio e32465) v8 -- 
0x559be2caec00 con 0x559be00ad800




Repeating multiple times a second (and filling /var)
Prior to taking one of the cephfs_metadata OSDs offline, these came from 
communications from ceph00 to the node hosting the suspected bad OSD.
Now they are between ceph00 and the host of the replacement metadata OSD.

Does anyone have any suggestion on how to get the MDS to switch from "up:rejoin" to 
"up:active"?

Is there any way to debug this, to determine what issue really is? I'm unable 
to interpret the debug log.

Cheers,
Eric


Dr Eric Tittley
Research Computing Officer www.roe.ac.uk/~ert
Institute for Astronomy Royal Observatory, Edinburgh




The University of Edinburgh is a charitable body, registered in Scotland, with 
registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh 
Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: import/export with --export-format 2

[Eugen Block]

Hi,

I can't comment on why #3 shouldn't be used, but a quick test shows  
that the image is not really usable in that case. I created a  
partition on the src-image (1 GB), filled it up with around 500 MB of  
data and then did the same export you did:


rbd export --export-format 2 src-image - | rbd import - dst-image

dst-image has around 500 MB total size and does not have a partition,  
so it's broken. Apparently, if you use "--export-format 2" to export  
you should also use the same flag for import. But the result is the  
same if you omit the flag for both export and import.
I'm not sure if that should be documented, one could argue that you  
should use the same flags if you want to use the dest-image the same  
way as the src-image. You could create a tracker issue in the category  
"documentation" and see what Zac thinks of it (added him in CC):


https://tracker.ceph.com/projects/ceph

Zitat von Tony Liu :


Hi,

src-image is 1GB (provisioned size). I did the following 3 tests.

1. rbd export src-image - | rbd import - dst-image
2. rbd export --export-format 2 src-image - | rbd import  
--export-format 2 - dst-image

3. rbd export --export-format 2 src-image - | rbd import - dst-image

With #1 and #2, dst-image size (rbd info) is the same as src-image,  
which is expected.
With #3, dst-image size (rbd info) is close to used size (rbd du),  
not the provisioned
size of src-image. I'm not sure if this image is actually useable  
when write into it.


The questions is that, is #3 not supposed to be used at all?
I checked doc, didn't see something like "--export-format 2 has to  
be used for

importing the image which is exported with --export-format 2 option".
Any comments?


Thanks!
Tony
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

[Lo Re Giuseppe]

Hi David,
Thanks a lot for your reply.
Yes we have heavy load from clients on the same subtree. We have multiple MDSs 
that were setup with the hope to distribute the load among them, but this is 
not really happening, in moments of high load we see most of the load on one 
MDS.
We don't use pinning today.
We have placed >1 MDSs in the same servers as we noticed that the memory 
consumption was allowing this.
Right now I have scaled down the MDS services  to 1 as I have learnt that the 
use of multiple MDSs could have been a risky move. Though it was working not 
bad until we upgraded from 17.2.5 up to 17.2.7 and now 18.2.0.
I'll look more in the client stability as per your suggestion.

Best,

Giuseppe

On 27.11.2023, 10:41, "David C." mailto:david.cas...@aevoo.fr>> wrote:


Hi Guiseppe,


Wouldn't you have clients who heavily load the MDS with concurrent access
on the same trees ?


Perhaps, also, look at the stability of all your clients (even if there are
many) [dmesg -T, ...]


How are your 4 active MDS configured (pinning?) ?


Probably nothing to do but normal for 2 MDS to be on the same host
"monitor-02" ?





Cordialement,


*David CASIER*









Le lun. 27 nov. 2023 à 10:09, Lo Re Giuseppe mailto:giuseppe.l...@cscs.ch>> a
écrit :


> Hi,
> We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are
> having CephFS issues.
> For example this morning:
> “””
> [root@naret-monitor01 ~]# ceph -s
> cluster:
> id: 63334166-d991-11eb-99de-40a6b72108d0
> health: HEALTH_WARN
> 1 filesystem is degraded
> 3 clients failing to advance oldest client/flush tid
> 3 MDSs report slow requests
> 6 pgs not scrubbed in time
> 29 daemons have recently crashed
> …
> “””
>
> The ceph orch, ceph crash and ceph fs status commands were hanging.
>
> After a “ceph mgr fail” those commands started to respond.
> Then I have noticed that there was one mds with most of the slow
> operations,
>
> “””
> [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests
> mds.cephfs.naret-monitor01.nuakzo(mds.0): 18 slow requests are blocked
> > 30 secs
> mds.cephfs.naret-monitor01.uvevbf(mds.1): 1683 slow requests are
> blocked > 30 secs
> mds.cephfs.naret-monitor02.exceuo(mds.2): 1 slow requests are blocked
> > 30 secs
> “””
>
> Then I tried to restart it with
>
> “””
> [root@naret-monitor01 ~]# ceph orch daemon restart
> mds.cephfs.naret-monitor01.uvevbf
> Scheduled to restart mds.cephfs.naret-monitor01.uvevbf on host
> 'naret-monitor01'
> “””
>
> After the cephfs entered into this situation:
> “””
> [root@naret-monitor01 ~]# ceph fs status
> cephfs - 198 clients
> ==
> RANK STATE MDS ACTIVITY DNS
> INOS DIRS CAPS
> 0 active cephfs.naret-monitor01.nuakzo Reqs: 0 /s 17.2k
> 16.2k 1892 14.3k
> 1 active cephfs.naret-monitor02.ztdghf Reqs: 0 /s 28.1k
> 10.3k 752 6881
> 2 clientreplay cephfs.naret-monitor02.exceuo 63.0k
> 6491 541 66
> 3 active cephfs.naret-monitor03.lqppte Reqs: 0 /s 16.7k
> 13.4k 8233 990
> POOL TYPE USED AVAIL
> cephfs.cephfs.meta metadata 5888M 18.5T
> cephfs.cephfs.data data 119G 215T
> cephfs.cephfs.data.e_4_2 data 2289G 3241T
> cephfs.cephfs.data.e_8_3 data 9997G 470T
> STANDBY MDS
> cephfs.naret-monitor03.eflouf
> cephfs.naret-monitor01.uvevbf
> MDS version: ceph version 18.2.0
> (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)
> “””
>
> The file system is totally unresponsive (we can mount it on client nodes
> but any operations like a simple ls hangs).
>
> During the night we had a lot of mds crashes, I can share the content.
>
> Does anybody have an idea on how to tackle this problem?
>
> Best,
>
> Giuseppe
> ___
> ceph-users mailing list -- ceph-users@ceph.io 
> To unsubscribe send an email to ceph-users-le...@ceph.io 
> 
>
___
ceph-users mailing list -- ceph-users@ceph.io 
To unsubscribe send an email to ceph-users-le...@ceph.io 




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Experience with deduplication

[Szabo, Istvan (Agoda)]

Hi Developers,

What is the status of the deduplication for objectsore? I see it under the dev 
area only since octopus even with the latest release.
https://docs.ceph.com/en/octopus/dev/deduplication/

Is it something that can be used in production?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

[David C.]

Hi Guiseppe,

Wouldn't you have clients who heavily load the MDS with concurrent access
on the same trees ?

Perhaps, also, look at the stability of all your clients (even if there are
many) [dmesg -T, ...]

How are your 4 active MDS configured (pinning?) ?

Probably nothing to do but normal for 2 MDS to be on the same host
"monitor-02" ?



Cordialement,

*David CASIER*





Le lun. 27 nov. 2023 à 10:09, Lo Re Giuseppe  a
écrit :

> Hi,
> We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are
> having CephFS issues.
> For example this morning:
> “””
> [root@naret-monitor01 ~]# ceph -s
>   cluster:
> id: 63334166-d991-11eb-99de-40a6b72108d0
> health: HEALTH_WARN
> 1 filesystem is degraded
> 3 clients failing to advance oldest client/flush tid
> 3 MDSs report slow requests
> 6 pgs not scrubbed in time
> 29 daemons have recently crashed
> …
> “””
>
> The ceph orch, ceph crash and ceph fs status commands were hanging.
>
> After a “ceph mgr fail” those commands started to respond.
> Then I have noticed that there was one mds with most of the slow
> operations,
>
> “””
> [WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests
> mds.cephfs.naret-monitor01.nuakzo(mds.0): 18 slow requests are blocked
> > 30 secs
> mds.cephfs.naret-monitor01.uvevbf(mds.1): 1683 slow requests are
> blocked > 30 secs
> mds.cephfs.naret-monitor02.exceuo(mds.2): 1 slow requests are blocked
> > 30 secs
> “””
>
> Then I tried to restart it with
>
> “””
> [root@naret-monitor01 ~]# ceph orch daemon restart
> mds.cephfs.naret-monitor01.uvevbf
> Scheduled to restart mds.cephfs.naret-monitor01.uvevbf on host
> 'naret-monitor01'
> “””
>
> After the cephfs entered into this situation:
> “””
> [root@naret-monitor01 ~]# ceph fs status
> cephfs - 198 clients
> ==
> RANK STATE   MDS  ACTIVITY DNS
> INOS   DIRS   CAPS
> 0   active cephfs.naret-monitor01.nuakzo  Reqs:0 /s  17.2k
> 16.2k  1892   14.3k
> 1   active cephfs.naret-monitor02.ztdghf  Reqs:0 /s  28.1k
> 10.3k   752   6881
> 2clientreplay  cephfs.naret-monitor02.exceuo 63.0k
> 6491541 66
> 3   active cephfs.naret-monitor03.lqppte  Reqs:0 /s  16.7k
> 13.4k  8233990
>   POOL  TYPE USED  AVAIL
>cephfs.cephfs.meta metadata  5888M  18.5T
>cephfs.cephfs.data   data 119G   215T
> cephfs.cephfs.data.e_4_2data2289G  3241T
> cephfs.cephfs.data.e_8_3data9997G   470T
>  STANDBY MDS
> cephfs.naret-monitor03.eflouf
> cephfs.naret-monitor01.uvevbf
> MDS version: ceph version 18.2.0
> (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)
> “””
>
> The file system is totally unresponsive (we can mount it on client nodes
> but any operations like a simple ls hangs).
>
> During the night we had a lot of mds crashes, I can share the content.
>
> Does anybody have an idea on how to tackle this problem?
>
> Best,
>
> Giuseppe
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Issue with CephFS (mds stuck in clientreplay status) since upgrade to 18.2.0.

[Lo Re Giuseppe]

Hi,
We have upgraded one ceph cluster from 17.2.7 to 18.2.0. Since then we are 
having CephFS issues.
For example this morning:
“””
[root@naret-monitor01 ~]# ceph -s
  cluster:
id: 63334166-d991-11eb-99de-40a6b72108d0
health: HEALTH_WARN
1 filesystem is degraded
3 clients failing to advance oldest client/flush tid
3 MDSs report slow requests
6 pgs not scrubbed in time
29 daemons have recently crashed
…
“””

The ceph orch, ceph crash and ceph fs status commands were hanging.

After a “ceph mgr fail” those commands started to respond.
Then I have noticed that there was one mds with most of the slow operations,

“””
[WRN] MDS_SLOW_REQUEST: 3 MDSs report slow requests
mds.cephfs.naret-monitor01.nuakzo(mds.0): 18 slow requests are blocked > 30 
secs
mds.cephfs.naret-monitor01.uvevbf(mds.1): 1683 slow requests are blocked > 
30 secs
mds.cephfs.naret-monitor02.exceuo(mds.2): 1 slow requests are blocked > 30 
secs
“””

Then I tried to restart it with

“””
[root@naret-monitor01 ~]# ceph orch daemon restart 
mds.cephfs.naret-monitor01.uvevbf
Scheduled to restart mds.cephfs.naret-monitor01.uvevbf on host 'naret-monitor01'
“””

After the cephfs entered into this situation:
“””
[root@naret-monitor01 ~]# ceph fs status
cephfs - 198 clients
==
RANK STATE   MDS  ACTIVITY DNSINOS  
 DIRS   CAPS
0   active cephfs.naret-monitor01.nuakzo  Reqs:0 /s  17.2k  16.2k  
1892   14.3k
1   active cephfs.naret-monitor02.ztdghf  Reqs:0 /s  28.1k  10.3k   
752   6881
2clientreplay  cephfs.naret-monitor02.exceuo 63.0k  6491
541 66
3   active cephfs.naret-monitor03.lqppte  Reqs:0 /s  16.7k  13.4k  
8233990
  POOL  TYPE USED  AVAIL
   cephfs.cephfs.meta metadata  5888M  18.5T
   cephfs.cephfs.data   data 119G   215T
cephfs.cephfs.data.e_4_2data2289G  3241T
cephfs.cephfs.data.e_8_3data9997G   470T
 STANDBY MDS
cephfs.naret-monitor03.eflouf
cephfs.naret-monitor01.uvevbf
MDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) 
reef (stable)
“””

The file system is totally unresponsive (we can mount it on client nodes but 
any operations like a simple ls hangs).

During the night we had a lot of mds crashes, I can share the content.

Does anybody have an idea on how to tackle this problem?

Best,

Giuseppe
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph/daemon container lvm tools don’t work

[Gaël THEROND]

Hi team,

I’m experimenting a bit CentOS Stream 9 on our infrastructure as we’re
migrating away from CentOS Stream 8.

As our deployment model is an hyperconverged one, I have CEPH and OPENSTACK
running on the same hosts (OSDs+NOVA/CINDER).

That prohibits me to let CEPH nodes running on CentOS Stream 8.

However, I’ve noticed an issue related to LVM, which is a bit annoying,
when running CentOS Stream 8 based ceph/daemon container over a CentOS
Stream 9 node.

Indeed, when the ceph-volume lvm batch is processed and perform the lvm
subcommand process call, the pvcreate do not persist anymore on the host
and so consequently, when you reboot, all ceph related disks disappear as
LVM can’t find their PVID or device references. Even a pvscan doesn’t find
them.

The only workaround that we found out right now is to pvcreate the host
disks, pvremove them, dd them and then run the installation process again,
from my understanding it kinda warmup the host lvm system and consequently
let it keep up with the ceph-volume run later on.

For additional information:

We deploy the cluster using ceph-ansible stable-6.0 branch (pacific).

The container come from quay.io and is 6.0.11 CentOS Stream 8 release.

This release is working perfectly on a CentOS Stream 8 based host.

One weird thing that we catched is that on a CentOS Stream 8 host using
this CentOS Stream 8 based image, lvm/dm are creating new dm block devices
named after ceph vg/lv where on a CentOS Stream 9 based host, lvm/dm create
the new vg/lv as symlink to dm-xY dm block devices.

And last question, is there any planned build of the ceph/daemon:6.0.11 or
greater image based on CentOS Stream 9 ?

Thanks a lot!

PS: I did forget the [ceph-users] tag at first post and don’t know if it’s
something important for the mailing distribution.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Where is a simple getting started guide for a very basic cluster?

[Janne Johansson]

Looking up the "manual installation" parts might help, if you can't
get the container stuff going for $reasons.

Den mån 27 nov. 2023 kl 00:45 skrev Leo28C :
>
> I'm pulling my hair trying to get a simple cluster going. I first tried
> Gluster but I have an old system that can't handle the latest version, so I
> resorted to Ceph. However, I can't get my cluster to work. I tried to find
> tutorials but everything uses tools on top of Ceph, whereas I'm trying to
> use it barebones. (I don't trust nfs or samba for this purpose)
>
> The three systems I'm trying to cluster together have a variety of Linux
> distros, and one of them is discontinued, so I can't use a lot of newer
> stuff. I need to make do with the commands `ceph`, `ceph-mon`, and
> `ceph-volume`. I do *not* have access to `ceph-disk` which I ran into a lot
> online.
>
> ChatGPT guided me up to a certain point before she started making stuff up,
> so I need to start from scratch. I just need to share a directory (or
> rather a block device in the case of Ceph) across these three systems that
> already have a secure connection to one another. Can somebody please help
> me out?
>
> Thanks in advance
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io