Hi Igor
the reason why i tested differekt rocksdb options is that was having a
really bad write performance with the default settings (30-60mb/s) on the
cluster...
actually i have 200mb/s read and 180mb/s write performance
now i dont now which of the booth settings are the good ones
Another
Hi Igor,
Thx for checking the logs.. but what the hell is going on here? :-)
Yes its true i tested the and created the osd´s with three
different rockdb options.
I can not understand why the osd dont have the same rockdb option, because
i have created ALL OSDs new after set and test those
Hello Igor
"Does single OSD startup (after if's experiencing "unable to allocate)
takes 20 mins as well?"
A: YES
Here the example log of the startup and recovery of a problematic osd.
https://paste.ubuntu.com/p/2WVJbg7cBy/
Here the example log of a problematic osd
hmm... so it looks like RocksDB still doesn't perform WAL cleanup during
regular operation but applies it on OSD startup
Does single OSD startup (after if's experiencing "unable to allocate)
takes 20 mins as well?
Could you please share OSD log containing both that long startup and
Hi Igor,
"And was osd.2 redeployed AFTER settings had been reset to defaults ?"
A: YES
"Anything particular about current cluster use cases?"
A: we are using it temporary as a iscsi target for a vmware esxi cluster
with 6 hosts. We created two 10tb iscsi images/luns for vmware, because the
other
And was osd.2 redeployed AFTER settings had been reset to defaults ?
Anything particular about current cluster use cases?
E.g. is it a sort of regular usage (with load flukes and peak) or may be
some permanently running stress load testing. The latter might tend to
hold the resources and e.g.
Hi Igor,
yes the same problem is on osd.2
we have 3 OSD Nodes... Each Node has 20 Bluestore OSDs ... in total we have
60 OSDs
i checked right now one node... and 15 of 20 OSDs have this problem and
error in the log.
the settings that you have complained some emails ago .. i have reverted
them
Hi Igor,
yesss
Am Do., 7. Okt. 2021 um 19:24 Uhr schrieb Igor Fedotov <
igor.fedo...@croit.io>:
> Hey Jose,
>
> so you applied default settings and compacted osd.8, right?
>
> This helped for a short time and now it's back to the same state, right?
>
>
> Igor
>
>
> On 10/7/2021 12:46 PM, José
And does redeployed osd.2 expose the same issue (or at least DB/WAL
disbalance) again? Were settings reverted to defaults for it as well?
Thanks
Igor
On 10/7/2021 12:46 PM, José H. Freidhof wrote:
Good morning,
i checked today the osd.8 and the log shows again the same error
bluefs
Hey Jose,
so you applied default settings and compacted osd.8, right?
This helped for a short time and now it's back to the same state, right?
Igor
On 10/7/2021 12:46 PM, José H. Freidhof wrote:
Good morning,
i checked today the osd.8 and the log shows again the same error
bluefs
Good morning,
i checked today the osd.8 and the log shows again the same error
bluefs _allocate unable to allocate 0x10 on bdev 0, allocator name
bluefs-wal, allocator type hybrid, capacity 0xb4000, block size
0x10, free 0xff000, fragmentation 0, allocated 0x0
any idea why that could
Hi Igor,
today i repaired one osd node and all osd´s on the node, creating them new
again
after that i waited for the rebalance/recovery process and the cluster was
healthy after some hours..
i notices that the osd.2 does not have any more this error in the log.
but i noticed it now on the
On 10/6/2021 4:25 PM, José H. Freidhof wrote:
hi,
no risk no fun okay
I have reset the settings you mentioned to standard.
what you exactly mean with taking offline the osd? ceph orch daemon stop
osd.2? or mark down?
"daemon stop" is enough. You might want to set noout flag before that
hi,
no risk no fun okay
I have reset the settings you mentioned to standard.
what you exactly mean with taking offline the osd? ceph orch daemon stop
osd.2? or mark down?
for the command which path i use? you mean:
bluestore-kv /var/lib/ceph/$fsid/osd.2 compact???
Igor Fedotov schrieb am
On 10/6/2021 2:16 PM, José H. Freidhof wrote:
Hi Igor,
yes i have some osd settings set :-) here are my ceph config dump. those
settings are from a redhat document for bluestore devices
maybe it is that setting causing this problem? "advanced
mon_compact_on_trimfalse"???
OMG!!!
No -
Hi Igor,
yes i have some osd settings set :-) here are my ceph config dump. those
settings are from a redhat document for bluestore devices
maybe it is that setting causing this problem? "advanced
mon_compact_on_trimfalse"???
i will test it this afternoon... at the moment are everything
Jose,
In fact 48GB is a way too much for WAL drive - usually the write ahead
log tend to be 2-4 GBs.
But in your case it's ~150GB, while DB itself is very small (146MB!!!):
WAL 45 GiB 111 GiB 0 B 0 B 0 B
154 GiB 2400
DB 0 B 164
Hello Igor,
yes the volume is nvme wal partitions for the bluestore devicegroups are
only 48gb each
on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb (WAL)
on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb (rock.db)
on each osd node are 20 hdd with 5.5tb with 1
Hey Jose,
it looks like your WAL volume is out of space which looks weird given
its capacity = 48Gb.
Could you please share the output of the following commands:
ceph daemon osd.N bluestore bluefs device info
ceph daemon osd.N bluefs stats
Thanks,
Igor
On 10/6/2021 12:24 PM, José H.
19 matches
Mail list logo
