Hi Igor,
Am 05.05.20 um 16:10 schrieb Igor Fedotov:
> Hi Stefan,
>
> so (surprise!) some DB access counters show a significant difference, e.g.
>
> "kv_flush_lat": {
> "avgcount": 1423,
> "sum": 0.000906419,
> "avgtime": 0.000000636
> },
> "kv_sync_lat": {
> "avgcount": 1423,
> "sum": 0.712888091,
> "avgtime": 0.000500975
> },
> vs.
>
> "kv_flush_lat": {
> "avgcount": 1146,
> "sum": 3.346228802,
> "avgtime": 0.002919920
> },
> "kv_sync_lat": {
> "avgcount": 1146,
> "sum": 3.754915016,
> "avgtime": 0.003276540
> },
>
> Also for bluefs:
> "bytes_written_sst": 0,
> vs.
> "bytes_written_sst": 59785361,
>
> Could you please rerun these benchmark/perf counter gathering steps a couple
> more times and check if the difference is persistent.
I reset all perf counters and ran the bench 10 times on each osd.
OSD 38:
bench: wrote 12 MiB in blocks of 4 KiB in 1.22796 sec at 9.5 MiB/sec
2.44k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.26407 sec at 9.3 MiB/sec
2.37k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.24987 sec at 9.4 MiB/sec
2.40k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.37125 sec at 8.5 MiB/sec
2.19k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.25549 sec at 9.3 MiB/sec
2.39k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.24358 sec at 9.4 MiB/sec
2.41k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.24208 sec at 9.4 MiB/sec
2.42k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.2433 sec at 9.4 MiB/sec
2.41k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.26548 sec at 9.3 MiB/sec
2.37k IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 1.31509 sec at 8.9 MiB/sec
2.28k IOPS
kv_flush_lat.sum: 8.955978864
kv_sync_lat.sum: 10.869536503
bytes_written_sst: 0
OSD 0:
bench: wrote 12 MiB in blocks of 4 KiB in 5.71447 sec at 2.1 MiB/sec 524
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 6.18679 sec at 1.9 MiB/sec 484
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 6.69068 sec at 1.8 MiB/sec 448
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 7.06413 sec at 1.7 MiB/sec 424
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 7.50321 sec at 1.6 MiB/sec 399
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 6.86882 sec at 1.7 MiB/sec 436
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 7.11702 sec at 1.6 MiB/sec 421
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 7.10497 sec at 1.6 MiB/sec 422
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 6.69801 sec at 1.7 MiB/sec 447
IOPS
bench: wrote 12 MiB in blocks of 4 KiB in 7.13588 sec at 1.6 MiB/sec 420
IOPS
kv_flush_lat.sum: 0.003866224
kv_sync_lat.sum: 2.667407139
bytes_written_sst: 34904457
> If that's particularly true for "kv_flush_lat" counter - please rerun with
> debug-bluefs set to 20 and collect OSD logs for both cases
Yes it's still true for kv_flush_lat - see above. Where to upload / put
those logs?
greets,
Stefan
>
> Thanks,
> Igor
>
> On 5/5/2020 11:46 AM, Stefan Priebe - Profihost AG wrote:
>> Hello Igor,
>>
>> Am 30.04.20 um 15:52 schrieb Igor Fedotov:
>>> 1) reset perf counters for the specific OSD
>>>
>>> 2) run bench
>>>
>>> 3) dump perf counters.
>> This is OSD 0:
>>
>> # ceph tell osd.0 bench -f plain 12288000 4096
>> bench: wrote 12 MiB in blocks of 4 KiB in 6.70482 sec at 1.7 MiB/sec 447
>> IOPS
>>
>> https://pastebin.com/raw/hbKcU07g
>>
>> This is OSD 38:
>>
>> # ceph tell osd.38 bench -f plain 12288000 4096
>> bench: wrote 12 MiB in blocks of 4 KiB in 2.01763 sec at 5.8 MiB/sec
>> 1.49k IOPS
>>
>> https://pastebin.com/raw/Tx2ckVm1
>>
>>> Collecting disks' (both main and db) activity with iostat would be nice
>>> too. But please either increase benchmark duration or reduce iostat
>>> probe period to 0.1 or 0.05 second
>> This gives me:
>>
>> # ceph tell osd.38 bench -f plain 122880000 4096
>> Error EINVAL: 'count' values greater than 12288000 for a block size of 4
>> KiB, assuming 100 IOPS, for 30 seconds, can cause ill effects on osd.
>> Please adjust 'osd_bench_small_size_max_iops' with a higher value if you
>> wish to use a higher 'count'.
>>
>> Stefan
>>
>>> Thanks,
>>>
>>> Igor
>>>
>>> On 4/28/2020 8:42 PM, Stefan Priebe - Profihost AG wrote:
>>>> HI Igor,
>>>>
>>>> but the performance issue is still present even on the recreated OSD.
>>>>
>>>> # ceph tell osd.38 bench -f plain 12288000 4096
>>>> bench: wrote 12 MiB in blocks of 4 KiB in 1.63389 sec at 7.2 MiB/sec
>>>> 1.84k IOPS
>>>>
>>>> vs.
>>>>
>>>> # ceph tell osd.10 bench -f plain 12288000 4096
>>>> bench: wrote 12 MiB in blocks of 4 KiB in 10.7454 sec at 1.1 MiB/sec 279
>>>> IOPS
>>>>
>>>> both baked by the same SAMSUNG SSD as block.db.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>> Am 28.04.20 um 19:12 schrieb Stefan Priebe - Profihost AG:
>>>>> Hi Igore,
>>>>> Am 27.04.20 um 15:03 schrieb Igor Fedotov:
>>>>>> Just left a comment at https://tracker.ceph.com/issues/44509
>>>>>>
>>>>>> Generally bdev-new-db performs no migration, RocksDB might
>>>>>> eventually do
>>>>>> that but no guarantee it moves everything.
>>>>>>
>>>>>> One should use bluefs-bdev-migrate to do actual migration.
>>>>>>
>>>>>> And I think that's the root cause for the above ticket.
>>>>> perfect - this removed all spillover in seconds.
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Igor
>>>>>>
>>>>>> On 4/24/2020 2:37 PM, Stefan Priebe - Profihost AG wrote:
>>>>>>> No not a standalone Wal I wanted to ask whether bdev-new-db migrated
>>>>>>> dB and Wal from hdd to ssd.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>>> Am 24.04.2020 um 13:01 schrieb Igor Fedotov <ifedo...@suse.de>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Unless you have 3 different types of disks beyond OSD (e.g. HDD, SSD,
>>>>>>>> NVMe) standalone WAL makes no sense.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/24/2020 1:58 PM, Stefan Priebe - Profihost AG wrote:
>>>>>>>>> Is Wal device missing? Do I need to run *bluefs-bdev-new-db and
>>>>>>>>> Wal?*
>>>>>>>>>
>>>>>>>>> Greets,
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>>> Am 24.04.2020 um 11:32 schrieb Stefan Priebe - Profihost AG
>>>>>>>>>> <s.pri...@profihost.ag>:
>>>>>>>>>>
>>>>>>>>>> Hi Igor,
>>>>>>>>>>
>>>>>>>>>> there must be a difference. I purged osd.0 and recreated it.
>>>>>>>>>>
>>>>>>>>>> Now it gives:
>>>>>>>>>> ceph tell osd.0 bench
>>>>>>>>>> {
>>>>>>>>>> "bytes_written": 1073741824,
>>>>>>>>>> "blocksize": 4194304,
>>>>>>>>>> "elapsed_sec": 8.1554735639999993,
>>>>>>>>>> "bytes_per_sec": 131659040.46819863,
>>>>>>>>>> "iops": 31.389961354303033
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> What's wrong wiht adding a block.db device later?
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>> Am 23.04.20 um 20:34 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>>> Hi,
>>>>>>>>>>> if the OSDs are idle the difference is even more worse:
>>>>>>>>>>> # ceph tell osd.0 bench
>>>>>>>>>>> {
>>>>>>>>>>> "bytes_written": 1073741824,
>>>>>>>>>>> "blocksize": 4194304,
>>>>>>>>>>> "elapsed_sec": 15.396707875000001,
>>>>>>>>>>> "bytes_per_sec": 69738403.346825853,
>>>>>>>>>>> "iops": 16.626931034761871
>>>>>>>>>>> }
>>>>>>>>>>> # ceph tell osd.38 bench
>>>>>>>>>>> {
>>>>>>>>>>> "bytes_written": 1073741824,
>>>>>>>>>>> "blocksize": 4194304,
>>>>>>>>>>> "elapsed_sec": 6.8903985170000004,
>>>>>>>>>>> "bytes_per_sec": 155831599.77624846,
>>>>>>>>>>> "iops": 37.153148597776521
>>>>>>>>>>> }
>>>>>>>>>>> Stefan
>>>>>>>>>>> Am 23.04.20 um 14:39 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> Am 23.04.20 um 14:06 schrieb Igor Fedotov:
>>>>>>>>>>>>> I don't recall any additional tuning to be applied to new DB
>>>>>>>>>>>>> volume. And assume the hardware is pretty the same...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you still have any significant amount of data spilled over
>>>>>>>>>>>>> for these updated OSDs? If not I don't have any valid
>>>>>>>>>>>>> explanation for the phenomena.
>>>>>>>>>>>> just the 64k from here:
>>>>>>>>>>>> https://tracker.ceph.com/issues/44509
>>>>>>>>>>>>
>>>>>>>>>>>>> You might want to try "ceph osd bench" to compare OSDs under
>>>>>>>>>>>>> pretty the same load. Any difference observed
>>>>>>>>>>>> Servers are the same HW. OSD Bench is:
>>>>>>>>>>>> # ceph tell osd.0 bench
>>>>>>>>>>>> {
>>>>>>>>>>>> "bytes_written": 1073741824,
>>>>>>>>>>>> "blocksize": 4194304,
>>>>>>>>>>>> "elapsed_sec": 16.091414781000001,
>>>>>>>>>>>> "bytes_per_sec": 66727620.822242722,
>>>>>>>>>>>> "iops": 15.909104543266945
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> # ceph tell osd.36 bench
>>>>>>>>>>>> {
>>>>>>>>>>>> "bytes_written": 1073741824,
>>>>>>>>>>>> "blocksize": 4194304,
>>>>>>>>>>>> "elapsed_sec": 10.023828538,
>>>>>>>>>>>> "bytes_per_sec": 107118933.6419194,
>>>>>>>>>>>> "iops": 25.539143953780986
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> OSD 0 is a Toshiba MG07SCA12TA SAS 12G
>>>>>>>>>>>> OSD 36 is a Seagate ST12000NM0008-2H SATA 6G
>>>>>>>>>>>>
>>>>>>>>>>>> SSDs are all the same like the rest of the HW. But both drives
>>>>>>>>>>>> should give the same performance from their specs. The only other
>>>>>>>>>>>> difference is that OSD 36 was directly created with the block.db
>>>>>>>>>>>> device (Nautilus 14.2.7) and OSD 0 (14.2.8) does not.
>>>>>>>>>>>>
>>>>>>>>>>>> Stefan
>>>>>>>>>>>>
>>>>>>>>>>>>> On 4/23/2020 8:35 AM, Stefan Priebe - Profihost AG wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> is there anything else needed beside running:
>>>>>>>>>>>>>> ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD}
>>>>>>>>>>>>>> bluefs-bdev-new-db --dev-target /dev/vgroup/lvdb-1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I did so some weeks ago and currently i'm seeing that all osds
>>>>>>>>>>>>>> originally deployed with --block-db show 10-20% I/O waits while
>>>>>>>>>>>>>> all those got converted using ceph-bluestore-tool show 80-100%
>>>>>>>>>>>>>> I/O waits.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also is there some tuning available to use more of the SSD? The
>>>>>>>>>>>>>> SSD (block-db) is only saturated at 0-2%.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Greets,
>>>>>>>>>>>>>> Stefan
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>>>>>>>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
