The other fixes landed to nautilus and later releases
I suggest to you upgrade to nautilus as soon as possible, this is very stable 
release (14.2.22)


k
Sent from my iPhone

> On 14 Jun 2022, at 12:13, tao song <alansong1...@gmail.com> wrote:
> 
> 
>   Thanks , we have backport some PR to 12.2.12,but the problem remain.Are 
> there any other fixes?
> eg:
>> os/bluestore: apply garbage collection against excessive blob count growth
>>  https://github.com/ceph/ceph/pull/28229 
>> AND the ceph-bluestore-tool fsck/repair
>> https://github.com/ceph/ceph/pull/38050 
> 
> 
> Konstantin Shalygin <k0...@k0ste.ru> 于2022年6月14日周二 15:37写道:
>> Hi,
>> 
>> Many of fixes for "zombie blobs" landed in last Nautilus release
>> I suggest to upgrade to last Nautilus version
>> 
>> 
>> k
>> 
>> > On 14 Jun 2022, at 10:23, tao song <alansong1...@gmail.com> wrote:
>> > 
>> > I have a old Cluster 12.2.12 running bluestore ,use iscsi + RBD in EC
>> > pools(k:m=2:1) with ec_overwrites flags. Multiple OSD crashes occurred due
>> > to assert (0 == "no available blob id").
>> > The problems occur periodically when the RBD volume is cyclically
>> > overwritten.
>> > 
>> > 2022-05-24 22:08:19.950550 7fcb41894700  1 osd.171 pg_epoch: 47676
>> >> pg[4.1467s2( v 44365'8455207 (44045'8453660,44365'8455207]
>> >> local-lis/les=44415/44416 n=21866 ec=16123/349 lis/c 47665/44415 les/c/f
>> >> 47666/44416/4714 47676/47676/39511)
>> >> [115,16,171]/[115,2147483647,2147483647]p115(0) r=-1 lpr=47676
>> >> pi=[44415,47676)/2 crt=44260'8455206 lcod 0'0 remapped NOTIFY mbc={}]
>> >> state<Start>: transitioning to Stray
>> >> 2022-05-24 22:08:20.834007 7fcb41894700  1 osd.171 pg_epoch: 47677
>> >> pg[4.1467s2( v 44365'8455207 (44045'8453660,44365'8455207]
>> >> local-lis/les=44415/44416 n=21866 ec=16123/349 lis/c 47665/44415 les/c/f
>> >> 47666/44416/4714 47676/47677/39511) [115,16,171]p115(0) r=2 lpr=47677
>> >> pi=[44415,47677)/2 crt=44260'8455206 lcod 0'0 unknown NOTIFY mbc={}]
>> >> start_peering_interval up [115,16,171] -> [115,16,171], acting
>> >> [115,2147483647,2147483647] -> [115,16,171], acting_primary 115(0) -> 115,
>> >> up_primary 115(0) -> 115, role -1 -> 2, features acting 
>> >> 4611087853746454523
>> >> upacting 4611087853746454523
>> >> 2022-05-24 22:08:20.834073 7fcb41894700  1 osd.171 pg_epoch: 47677
>> >> pg[4.1467s2( v 44365'8455207 (44045'8453660,44365'8455207]
>> >> local-lis/les=44415/44416 n=21866 ec=16123/349 lis/c 47665/44415 les/c/f
>> >> 47666/44416/4714 47676/47677/39511) [115,16,171]p115(0) r=2 lpr=47677
>> >> pi=[44415,47677)/2 crt=44260'8455206 lcod 0'0 unknown NOTIFY mbc={}]
>> >> state<Start>: transitioning to Stray
>> >> 2022-05-24 22:08:22.097055 7fcb3a085700 -1
>> >> /ceph-12.2.12/src/os/bluestore/BlueStore.cc: In function 'bid_t
>> >> BlueStore::ExtentMap::allocate_spanning_blob_id()' thread 7fcb3a085700 
>> >> time
>> >> 2022-05-24 22:08:22.091806
>> >> /ceph-12.2.12/src/os/bluestore/BlueStore.cc: 2083: FAILED assert(0 == "no
>> >> available blob id")
>> >> 
>> >> ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous
>> >> (stable)
>> >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> >> const*)+0x110) [0x560e41dbd520]
>> >> 2: (()+0x8fce4e) [0x560e41c15e4e]
>> >> 3: (BlueStore::ExtentMap::reshard(KeyValueDB*,
>> >> std::shared_ptr<KeyValueDB::TransactionImpl>)+0x13da) [0x560e41c6fc6a]
>> >> 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*,
>> >> std::shared_ptr<KeyValueDB::TransactionImpl>)+0x1ab) [0x560e41c7131b]
>> >> 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*,
>> >> std::vector<ObjectStore::Transaction,
>> >> std::allocator<ObjectStore::Transaction> >&,
>> >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3fd)
>> >> [0x560e41c8cc4d]
>> >> 6:
>> >> (PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction,
>> >> std::allocator<ObjectStore::Transaction> >&,
>> >> boost::intrusive_ptr<OpRequest>)+0x65) [0x560e419efac5]
>> >> 7: (ECBackend::handle_sub_write(pg_shard_t,
>> >> boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&,
>> >> Context*)+0x631) [0x560e41b18331]
>> >> 8: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x349)
>> >> [0x560e41b29ba9]
>> >> 9: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50)
>> >> [0x560e41a255f0]
>> >> 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&,
>> >> ThreadPool::TPHandle&)+0x59c) [0x560e4198f97c]
>> >> 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
>> >> boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9)
>> >> [0x560e4180af59]
>> >> 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest>
>> >> const&)+0x57) [0x560e41a9ac27]
>> >> 13: (OSD::ShardedOpWQ::_process(unsigned int,
>> >> ceph::heartbeat_handle_d*)+0xfce) [0x560e4183a20e]
>> >> 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x83f)
>> >> [0x560e41dc304f]
>> >> 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x560e41dc4fe0]
>> >> 16: (()+0x7dd5) [0x7fcb5913fdd5]
>> >> 17: (clone()+0x6d) [0x7fcb5822fead]
>> >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
>> >> to interpret this.
>> >> 
>> > 
>> > Some would not restart with "no available blob id" assertion.We adjust the
>> > following parameters to to ensure that the OSD can be started
>> >  bluestore_extent_map_shard_target_size=2000 default 500,
>> >  bluestore_extent_map_shard_target_size_slop=0.300000 default 0.200000,
>> > 
>> > 
>> >> We found several related bugs :
>> >> https://tracker.ceph.com/issues/48216
>> >> https://tracker.ceph.com/issues/38272
>> > 
>> > The PR :
>> > 
>> > os/bluestore: apply garbage collection against excessive blob count growth
>> > 
>> > https://github.com/ceph/ceph/pull/28229
>> >> we have backport the PR to 12.2.12,but  it didn't solve the problem.
>> > 
>> > The workaround that works is to fsck / repair the stopped OSD :
>> >> ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<osd_id> --command
>> >> repair
>> >> 
>> > But it's not a long term solution.
>> >> I have seen a PR merged in 2019 here :
>> >> https://github.com/ceph/ceph/pull/28229
>> > 
>> > The fsck log:
>> > 
>> >> 2022-06-11 14:33:00.524108 7ff94ce7eec0 -1
>> >> bluestore(/var/lib/ceph/osd/ceph-162/) fsck error:
>> >> 2#4:fbdd648a:::rbd_data.3.3404c86b8b4567.0000000001977896:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x5567bd482690 spanning 7
>> >> blob([!~40000] csum crc32c/0x1000) use_tracker(0x4*0x10000 0x[0,0,0,0])
>> >> SharedBlob(0x5567bd482150 sbid 0x0))
>> >> 2022-06-11 14:33:00.620716 7ff94ce7eec0 -1
>> >> bluestore(/var/lib/ceph/osd/ceph-162/) fsck error:
>> >> 2#4:fbdda2c7:::rbd_data.3.3404d16b8b4567.0000000000dd0d7c:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x5567b1790230 spanning 0
>> >> blob([!~50000] csum crc32c/0x1000) use_tracker(0x5*0x10000 0x[0,0,0,0,0])
>> >> SharedBlob(0x5567b17902a0 sbid 0x0))
>> >> 2022-06-11 14:33:00.659399 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)   fsck error:
>> >> 2#4:fbddcdc9:::rbd_data.3.3404d76b8b4567.00000000005c3915:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x5568059181c0 spanning 0
>> >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0])
>> >> SharedBlob(0x556805919ce0 sbid 0x0))
>> >> 2022-06-11 14:33:00.666271 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)   fsck error:
>> >> 2#4:fbddd3fa:::rbd_data.3.3404d16b8b4567.00000000008d9fcf:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x55675655ea80 spanning 1
>> >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0])
>> >> SharedBlob(0x55675655eaf0 sbid 0x0))
>> >> 2022-06-11 14:33:00.830832 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)  fsck error:
>> >> 2#4:fbde5cdd:::rbd_data.3.3404d76b8b4567.0000000000aff911:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x556810246690 spanning 0
>> >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0])
>> >> SharedBlob(0x5567aff68f50 sbid 0x0))
>> >> 2022-06-11 14:33:00.875893 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)  fsck error:
>> >> 2#4:fbde8f23:::rbd_data.3.3404c86b8b4567.000000000151a7dc:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x5567f2a7b650 spanning 8
>> >> blob([!~50000] csum crc32c/0x1000) use_tracker(0x5*0x10000 0x[0,0,0,0,0])
>> >> SharedBlob(0x5567f2a7b810 sbid 0x0))
>> >> 2022-06-11 14:33:00.894370 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)   fsck error:
>> >> 2#4:fbdea5c3:::rbd_data.3.3404d16b8b4567.0000000000dcecaf:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x5567b5faa5b0 spanning 0
>> >> blob([!~50000] csum crc32c/0x1000) use_tracker(0x5*0x10000 0x[0,0,0,0,0])
>> >> SharedBlob(0x5567b5fabf10 sbid 0x0))
>> >> 2022-06-11 14:33:00.930450 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)   fsck error:
>> >> 2#4:fbdebbe9:::rbd_data.3.3404c86b8b4567.0000000000be844c:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x556737a395e0 spanning 8
>> >> blob([!~20000] csum crc32c/0x1000) use_tracker(0x2*0x10000 0x[0,0])
>> >> SharedBlob(0x556737b7dc70 sbid 0x0))
>> >> 2022-06-11 14:33:01.132212 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)   fsck error:
>> >> 2#4:fbdf864f:::rbd_data.3.3404c86b8b4567.0000000001d29b47:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x55673c858620 spanning 7
>> >> blob([!~40000] csum crc32c/0x1000) use_tracker(0x4*0x10000 0x[0,0,0,0])
>> >> SharedBlob(0x55673c354a80 sbid 0x0))
>> >> 2022-06-11 14:33:01.157017 7ff94ce7eec0 -1 bluestore
>> >> (/var/lib/ceph/osd/ceph-162/)   fsck error:
>> >> 2#4:fbdfa612:::rbd_data.3.3404d16b8b4567.0000000000a73d63:head# - 1 zombie
>> >> spanning blob(s) found, the first one: Blob(0x55673f6fcf5
>> > 
>> > 
>> > What is the root cause of zombie blobs? How can we avoid this problem?
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> 
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to