Hi Igor,
"And was osd.2 redeployed AFTER settings had been reset to defaults ?"
A: YES
"Anything particular about current cluster use cases?"
A: we are using it temporary as a iscsi target for a vmware esxi cluster
with 6 hosts. We created two 10tb iscsi images/luns for vmware, because the
other datastore are at 90%.
We plan in the future, after ceph is working right, and stable to install
openstack and kvm and we want to convert all vms into rbd images.
Like i told you is a three osd nodes cluster with 32 cores and 256gb ram
and two 10g bond network cards on a 10g network
"E.g. is it a sort of regular usage (with load flukes and peak) or may be
some permanently running stress load testing. The latter might tend to hold
the resources and e.g. prevent from internal house keeping...
A: Its a SAN for vmware and there are running 43 VMs at the moment... at
the daytime is more stress on the disks because the people are working and
in the afternoon the iops goes down because the users are at home
noting speculative...
There is something else that i noticed... if i reboot one osd with
20osds then it takes 20min to come up... if i tail the logs of the osd i
can see a lot of " recovery log mode 2" on all osd
after the 20min the osd comes one after one up and the waldb are small and
no error in the logs about bluefs _allocate unable to allocate...
it seems that the problem is rocking up after a longer time (12h)
Am Fr., 8. Okt. 2021 um 15:24 Uhr schrieb Igor Fedotov <
igor.fedo...@croit.io>:
> And was osd.2 redeployed AFTER settings had been reset to defaults ?
>
> Anything particular about current cluster use cases?
>
> E.g. is it a sort of regular usage (with load flukes and peak) or may be
> some permanently running stress load testing. The latter might tend to hold
> the resources and e.g. prevent from internal house keeping...
>
> Igor
>
>
> On 10/8/2021 12:16 AM, José H. Freidhof wrote:
>
> Hi Igor,
>
> yes the same problem is on osd.2
>
> we have 3 OSD Nodes... Each Node has 20 Bluestore OSDs ... in total we
> have 60 OSDs
> i checked right now one node... and 15 of 20 OSDs have this problem and
> error in the log.
>
> the settings that you have complained some emails ago .. i have reverted
> them to default.
>
> ceph.conf file:
>
> [global]
> fsid = 462c44b4-eed6-11eb-8b2c-a1ad45f88a97
> mon_host = [v2:10.50.50.21:3300/0,v1:10.50.50.21:6789/0] [v2:
> 10.50.50.22:3300/0,v1:10.50.50.22:6789/0] [v2:
> 10.50.50.20:3300/0,v1:10.50.50.20:6789/0]
> log file = /var/log/ceph/$cluster-$type-$id.log
> max open files = 131072
> mon compact on trim = False
> osd deep scrub interval = 137438953472
> osd max scrubs = 16
> osd objectstore = bluestore
> osd op threads = 2
> osd scrub load threshold = 0.01
> osd scrub max interval = 137438953472
> osd scrub min interval = 137438953472
> perf = True
> rbd readahead disable after bytes = 0
> rbd readahead max bytes = 4194304
> throttler perf counter = False
>
> [client]
> rbd cache = False
>
>
> [mon]
> mon health preluminous compat = True
> mon osd down out interval = 300
>
> [osd]
> bluestore cache autotune = 0
> bluestore cache kv ratio = 0.2
> bluestore cache meta ratio = 0.8
> bluestore extent map shard max size = 200
> bluestore extent map shard min size = 50
> bluestore extent map shard target size = 100
> bluestore rocksdb options =
> compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB
> osd map share max epochs = 100
> osd max backfills = 5
> osd op num shards = 8
> osd op num threads per shard = 2
> osd min pg log entries = 10
> osd max pg log entries = 10
> osd pg log dups tracked = 10
> osd pg log trim min = 10
>
>
>
> root@cd133-ceph-osdh-01:~# ceph config dump
> WHO MASK
> LEVEL OPTION VALUE
>
>
>
>
>
> RO
> global
> basic container_image
> docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb
>
>
>
>
> *
> global
> advanced leveldb_max_open_files 131072
>
>
>
>
> global
> advanced mon_compact_on_trim false
>
>
>
>
> global
> dev ms_crc_data false
>
>
>
>
> global
> advanced osd_deep_scrub_interval 1209600.000000
>
>
>
>
> global
> advanced osd_max_scrubs 16
>
>
>
>
> global
> advanced osd_scrub_load_threshold 0.010000
>
>
>
>
> global
> advanced osd_scrub_max_interval 1209600.000000
>
>
>
>
> global
> advanced osd_scrub_min_interval 86400.000000
>
>
>
>
> global
> advanced perf true
>
>
>
>
> global
> advanced rbd_readahead_disable_after_bytes 0
>
>
>
>
> global
> advanced rbd_readahead_max_bytes 4194304
>
>
>
>
> global
> advanced throttler_perf_counter false
>
>
>
>
> mon
> advanced auth_allow_insecure_global_id_reclaim false
>
>
>
>
> mon
> advanced cluster_network 10.50.50.0/24
>
>
>
>
>
> *
> mon
> advanced mon_osd_down_out_interval 300
>
>
>
>
> mon
> advanced public_network 10.50.50.0/24
>
>
>
>
>
> *
> mgr
> advanced mgr/cephadm/container_init True
>
>
>
>
>
> *
> mgr
> advanced mgr/cephadm/device_enhanced_scan true
>
>
>
>
>
> *
> mgr
> advanced mgr/cephadm/migration_current 2
>
>
>
>
>
> *
> mgr
> advanced mgr/cephadm/warn_on_stray_daemons false
>
>
>
>
>
> *
> mgr
> advanced mgr/cephadm/warn_on_stray_hosts false
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/10.50.50.21/server_addr
>
>
>
>
>
> *
>
>
>
> *
> mgr
> advanced mgr/dashboard/camdatadash/ssl_server_port 8443
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/cd133-ceph-mon-01/server_addr
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/dasboard/server_port 80
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/dashboard/server_addr 10.251.133.161
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/dashboard/ssl_server_port 8443
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/server_addr 0.0.0.0
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/server_port 8080
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/ssl false
>
>
>
>
>
> *
> mgr
> advanced mgr/dashboard/ssl_server_port 8443
>
>
>
>
>
> *
> mgr
> advanced mgr/orchestrator/orchestrator cephadm
>
>
>
>
> mgr
> advanced mgr/prometheus/server_addr 0.0.0.0
>
>
>
>
>
> *
> mgr
> advanced mgr/telemetry/channel_ident true
>
>
>
>
>
> *
> mgr
> advanced mgr/telemetry/enabled true
>
>
>
>
>
> *
> mgr
> advanced mgr/telemetry/last_opt_revision 3
>
>
>
>
>
> *
> osd
> dev bluestore_cache_autotune true
>
>
>
>
> osd
> dev bluestore_cache_kv_ratio 0.200000
>
>
>
>
> osd
> dev bluestore_cache_meta_ratio 0.800000
>
>
>
>
> osd
> dev bluestore_cache_size 2147483648
>
>
>
>
> osd
> dev bluestore_cache_size_hdd 2147483648
>
>
>
>
> osd
> dev bluestore_extent_map_shard_max_size 200
>
>
>
>
> osd
> dev bluestore_extent_map_shard_min_size 50
>
>
>
>
> osd
> dev bluestore_extent_map_shard_target_size 100
>
>
>
>
> osd
> advanced bluestore_rocksdb_options
>
> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
> *
> osd
> advanced mon_osd_cache_size 1024
>
>
>
>
> osd
> dev ms_crc_data false
>
>
>
>
> osd
> advanced osd_map_share_max_epochs 5
>
>
>
>
> osd
> advanced osd_max_backfills 1
>
>
>
>
> osd
> dev osd_max_pg_log_entries 10
>
>
>
>
> osd
> dev osd_memory_cache_min 3000000000
>
>
>
>
> osd host:cd133-ceph-osdh-01
> basic osd_memory_target 5797322096
>
>
>
>
> osd host:cd133k-ceph-osdh-01
> basic osd_memory_target 9402402385
>
>
>
>
> osd host:cd88-ceph-osdh-01
> basic osd_memory_target 5797322096
>
>
>
>
> osd
> advanced osd_memory_target_autotune true
>
>
>
>
> osd
> dev osd_min_pg_log_entries 10
>
>
>
>
> osd
> advanced osd_op_num_shards 8
>
>
>
>
>
> *
> osd
> advanced osd_op_num_threads_per_shard 2
>
>
>
>
>
> *
> osd
> dev osd_pg_log_dups_tracked 10
>
>
>
>
> osd
> dev osd_pg_log_trim_min 10
>
>
>
>
> osd
> advanced osd_recovery_max_active 3
>
>
>
>
> osd
> advanced osd_recovery_max_single_start 1
>
>
>
>
> osd
> advanced osd_recovery_sleep 0.000000
>
>
>
>
> client
> advanced rbd_cache false
>
>
>
>
>
>
>
> Am Do., 7. Okt. 2021 um 19:27 Uhr schrieb Igor Fedotov <
> igor.fedo...@croit.io>:
>
>> And does redeployed osd.2 expose the same issue (or at least DB/WAL
>> disbalance) again? Were settings reverted to defaults for it as well?
>>
>>
>> Thanks
>>
>> Igor
>> On 10/7/2021 12:46 PM, José H. Freidhof wrote:
>>
>> Good morning,
>>
>> i checked today the osd.8 and the log shows again the same error
>> bluefs _allocate unable to allocate 0x100000 on bdev 0, allocator name
>> bluefs-wal, allocator type hybrid, capacity 0xb40000000, block size
>> 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>
>> any idea why that could be?
>>
>> Am Mi., 6. Okt. 2021 um 22:23 Uhr schrieb José H. Freidhof <
>> harald.freid...@googlemail.com>:
>>
>>> Hi Igor,
>>>
>>> today i repaired one osd node and all osd´s on the node, creating them
>>> new again....
>>> after that i waited for the rebalance/recovery process and the cluster
>>> was healthy after some hours..
>>>
>>> i notices that the osd.2 does not have any more this error in the log.
>>> but i noticed it now on the same node on osd.8... so i did the test that
>>> you suggested on osd.8
>>>
>>> it took nearly 20minutes to compact those db´s on the bluestore, but it
>>> helped... the problem on osd.8 is gone...
>>>
>>>
>>> *so the problem that i have with the alloc on the wal device seems to be
>>> random on different nodes and osd´s and looks like it comes, stay a while
>>> and disappears after a longer while... *
>>>
>>> here are the results that you suggested:
>>>
>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
>>> {
>>> "dev": {
>>> "device": "BDEV_WAL",
>>> "total": 48318377984,
>>> "free": 1044480,
>>> "bluefs_used": 48317333504
>>> },
>>> "dev": {
>>> "device": "BDEV_DB",
>>> "total": 187904811008,
>>> "free": 79842762752,
>>> "bluefs_used": 108062048256
>>> },
>>> "dev": {
>>> "device": "BDEV_SLOW",
>>> "total": 6001172414464,
>>> "free": 5510727389184,
>>> "bluefs_used": 0,
>>> "bluefs max available": 5508815847424
>>> }
>>> }
>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
>>> 1 : device size 0x2bbfffe000 : using 0x1931500000(101 GiB)
>>> 2 : device size 0x57541c00000 : using 0x7235e3e000(457 GiB)
>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472
>>> Usage matrix:
>>> DEV/LEV WAL DB SLOW * *
>>> REAL FILES
>>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B
>>> 9.7 MiB 1
>>> WAL 45 GiB 100 GiB 0 B 0 B 0 B
>>> 144 GiB 2319
>>> DB 0 B 276 MiB 0 B 0 B 0 B
>>> 249 MiB 47
>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> 0 B 0
>>> TOTALS 45 GiB 109 GiB 0 B 0 B 0 B
>>> 0 B 2367
>>> MAXIMUMS:
>>> LOG 304 MiB 7.9 GiB 0 B 0 B 0 B
>>> 20 MiB
>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
>>> 192 GiB
>>> DB 0 B 762 MiB 0 B 0 B 0 B
>>> 738 MiB
>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> 0 B
>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B
>>> 0 B
>>>
>>> ---
>>>
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.464+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x400000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.472+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.480+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.500+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.576+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.624+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.636+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.884+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.968+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 4 rocksdb:
>>> [db_impl/db_impl_write.cc:1668] [L] New memtable created with log file:
>>> #13656. Immutable memtables: 1.
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.992+0000 7f4a9483a700 1 bluefs _allocate unable to
>>> allocate 0x100000 on bdev 0, allocator name bluefs-wal, allocator type
>>> hybrid, capacity 0xb40000000, block size 0x100000, free 0xff000,
>>> fragmentation 0, allocated 0x0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: (Original Log Time
>>> 2021/10/06-19:51:34.996331) [db_impl/db_impl_compaction_flush.cc:2198]
>>> Calling FlushMemTableToOutputFile with column family [L], flush slots
>>> available 1, compaction slots available 1, flush slots scheduled 1,
>>> compaction slots scheduled 0
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
>>> [L] [JOB 8859] Flushing memtable with next log file: 13655
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:321]
>>> [L] [JOB 8859] Flushing memtable with next log file: 13656
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
>>> {"time_micros": 1633549894998273, "job": 8859, "event": "flush_started",
>>> "num_memtables": 2, "num_entries": 3662, "num_deletes": 0,
>>> "total_data_size": 130482337, "memory_usage": 132976224, "flush_reason":
>>> "Write Buffer Full"}
>>> Oct 06 21:51:34 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:34.996+0000 7f4aab067700 4 rocksdb: [flush_job.cc:350]
>>> [L] [JOB 8859] Level-0 flush table #13657: started
>>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: EVENT_LOG_v1
>>> {"time_micros": 1633549895008271, "cf_name": "L", "job": 8859, "event":
>>> "table_file_creation", "file_number": 13657, "file_size": 2952537,
>>> "table_properties": {"data_size": 2951222, "index_size": 267,
>>> "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key":
>>> 0, "index_value_is_delta_encoded": 0, "filter_size": 197, "raw_key_size":
>>> 1120, "raw_average_key_size": 16, "raw_value_size": 2950151,
>>> "raw_average_value_size": 42145, "num_data_blocks": 9, "num_entries": 70,
>>> "num_deletions": 61, "num_merge_operands": 0, "num_range_deletions": 0,
>>> "format_version": 0, "fixed_key_len": 0, "filter_policy":
>>> "rocksdb.BuiltinBloomFilter", "column_family_name": "L",
>>> "column_family_id": 10, "comparator": "leveldb.BytewiseComparator",
>>> "merge_operator": "nullptr", "prefix_extractor_name": "nullptr",
>>> "property_collectors": "[]", "compression": "NoCompression",
>>> "compression_options": "window_bits=-14; level=32767; strategy=0;
>>> max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time":
>>> 1633549854, "oldest_key_time": 1633549854, "file_creation_time":
>>> 1633549894}}
>>> Oct 06 21:51:35 cd88-ceph-osdh-01 bash[6328]: debug
>>> 2021-10-06T19:51:35.004+0000 7f4aab067700 4 rocksdb: [flush_job.cc:401]
>>> [L] [JOB 8859] Level-0 flush table #13657: 2952537 bytes OK
>>>
>>> ---
>>>
>>> root@cd88-ceph-osdh-01:~# ceph osd set noout
>>> root@cd88-ceph-osdh-01:~# ceph orch daemon stop osd.8
>>> root@cd88-ceph-osdh-01:~# ceph orch ps
>>> ...
>>> osd.7 cd133-ceph-osdh-01
>>> running (4h) 44s ago - 2738M 5528M 16.2.5 6933c2a0b7dd
>>> 8a98ae61f0eb
>>> osd.8 cd88-ceph-osdh-01
>>> stopped 5s ago - - 5528M <unknown> <unknown>
>>> <unknown>
>>> osd.9 cd133k-ceph-osdh-01
>>> running (3d) 5m ago - 4673M 8966M 16.2.5 6933c2a0b7dd
>>> 0ff7584b1808
>>> ...
>>>
>>> ---
>>>
>>> root@cd88-ceph-osdh-01:~# ceph-kvstore-tool bluestore-kv
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/ compact
>>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 0
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_db_and_around read-only:0 repair:0
>>> 2021-10-06T21:53:50.559+0200 7f87bde3c240 1 bdev(0x5644f056c800
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056c800
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
>>> discard not supported
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
>>> rotational discard not supported
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device
>>> bdev 1 path
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db size 175
>>> GiB
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d000
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
>>> discard not supported
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device
>>> bdev 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>>> size 5.5 TiB
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bdev(0x5644f056d400
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
>>> non-rotational discard supported
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs add_block_device
>>> bdev 0 path
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal size 45
>>> GiB
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs mount
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>>> 0, allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
>>> block size 0x100000
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>>> 1, allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
>>> block size 0x100000
>>> 2021-10-06T21:53:50.563+0200 7f87bde3c240 1 bluefs _init_alloc shared,
>>> id 2, capacity 0x57541c00000, block size 0x10000
>>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1 bluefs mount
>>> shared_bdev_used = 0
>>> 2021-10-06T21:53:50.655+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _prepare_db_environment set db_paths to db,178509578240
>>> db.slow,5701113793740
>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_db opened rocksdb path db options
>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_super_meta old nid_max 167450
>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_super_meta old blobid_max 30720
>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_super_meta freelist_type bitmap
>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_super_meta ondisk_format 4 compat_ondisk_format 3
>>> 2021-10-06T22:01:32.715+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_super_meta min_alloc_size 0x1000
>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist init
>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1 freelist _read_cfg
>>> 2021-10-06T22:01:33.347+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _init_alloc opening allocation metadata
>>> 2021-10-06T22:01:41.031+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _init_alloc loaded 5.0 TiB in 37191 extents, allocator type hybrid,
>>> capacity 0x57541c00000, block size 0x1000, free 0x502f8f9a000,
>>> fragmentation 2.76445e-05
>>> 2021-10-06T22:01:41.039+0200 7f87bde3c240 1 bluefs umount
>>> 2021-10-06T22:01:41.043+0200 7f87bde3c240 1 bdev(0x5644f056d400
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
>>> 2021-10-06T22:01:43.623+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
>>> 2021-10-06T22:01:54.727+0200 7f87bde3c240 1 bdev(0x5644f056d000
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d000
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) open
>>> size 187904819200 (0x2bc0000000, 175 GiB) block_size 4096 (4 KiB)
>>> rotational discard not supported
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device
>>> bdev 1 path
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db size 175
>>> GiB
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open path
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) open size
>>> 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational
>>> discard not supported
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device
>>> bdev 2 path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block
>>> size 5.5 TiB
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>>> path /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bdev(0x5644f056d400
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) open
>>> size 48318382080 (0xb40000000, 45 GiB) block_size 4096 (4 KiB)
>>> non-rotational discard supported
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs add_block_device
>>> bdev 0 path
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal size 45
>>> GiB
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs mount
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>>> 0, allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
>>> block size 0x100000
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc new, id
>>> 1, allocator name bluefs-db, allocator type hybrid, capacity 0x2bc0000000,
>>> block size 0x100000
>>> 2021-10-06T22:01:54.995+0200 7f87bde3c240 1 bluefs _init_alloc shared,
>>> id 2, capacity 0x57541c00000, block size 0x10000
>>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1 bluefs mount
>>> shared_bdev_used = 0
>>> 2021-10-06T22:01:55.079+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _prepare_db_environment set db_paths to db,178509578240
>>> db.slow,5701113793740
>>> 2021-10-06T22:09:36.519+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/)
>>> _open_db opened rocksdb path db options
>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824
>>> 2021-10-06T22:09:54.067+0200 7f87bde3c240 1
>>> bluestore(/var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8/) umount
>>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bluefs umount
>>> 2021-10-06T22:09:54.079+0200 7f87bde3c240 1 bdev(0x5644f056d400
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.wal) close
>>> 2021-10-06T22:09:56.612+0200 7f87bde3c240 1 bdev(0x5644f056d000
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block.db) close
>>> 2021-10-06T22:10:07.520+0200 7f87bde3c240 1 bdev(0x5644f056cc00
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
>>> 2021-10-06T22:10:07.688+0200 7f87bde3c240 1 freelist shutdown
>>> 2021-10-06T22:10:07.692+0200 7f87bde3c240 1 bdev(0x5644f056c800
>>> /var/lib/ceph/462c44b4-eed6-11eb-8b2c-a1ad45f88a97/osd.8//block) close
>>>
>>> ---
>>>
>>> root@cd88-ceph-osdh-01:~# ceph orch daemon start osd.8
>>>
>>> ---
>>>
>>> root@cd88-ceph-osdh-01:/# ceph -s
>>> cluster:
>>> id: 462c44b4-eed6-11eb-8b2c-a1ad45f88a97
>>> health: HEALTH_OK
>>>
>>> services:
>>> mon: 3 daemons, quorum
>>> cd133-ceph-mon-01,cd88-ceph-mon-01,cd133k-ceph-mon-01 (age 15h)
>>> mgr: cd133-ceph-mon-01.mzapob(active, since 15h), standbys:
>>> cd133k-ceph-mon-01.imikwh
>>> osd: 60 osds: 60 up (since 2m), 60 in (since 3h)
>>> rgw: 4 daemons active (2 hosts, 1 zones)
>>> tcmu-runner: 10 portals active (2 hosts)
>>>
>>> data:
>>> pools: 6 pools, 361 pgs
>>> objects: 2.46M objects, 8.0 TiB
>>> usage: 33 TiB used, 304 TiB / 338 TiB avail
>>> pgs: 361 active+clean
>>>
>>> io:
>>> client: 45 MiB/s rd, 50 MiB/s wr, 921 op/s rd, 674 op/s wr
>>>
>>>
>>>
>>> ---
>>>
>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluestore bluefs device info
>>> {
>>> "dev": {
>>> "device": "BDEV_WAL",
>>> "total": 48318377984,
>>>
>>> * "free": 41354784768, * "bluefs_used": 6963593216
>>> },
>>> "dev": {
>>> "device": "BDEV_DB",
>>> "total": 187904811008,
>>> "free": 187302928384,
>>> "bluefs_used": 601882624
>>> },
>>> "dev": {
>>> "device": "BDEV_SLOW",
>>> "total": 6001172414464,
>>> "free": 5507531620352,
>>> "bluefs_used": 0,
>>> "bluefs max available": 5505566572544
>>> }
>>> }
>>>
>>> ---
>>>
>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.8 bluefs stats
>>> 0 : device size 0xb3ffff000 : using 0x1a0c00000(6.5 GiB)
>>> 1 : device size 0x2bbfffe000 : using 0x23e00000(574 MiB)
>>> 2 : device size 0x57541c00000 : using 0x72f0803000(460 GiB)
>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472
>>> Usage matrix:
>>> DEV/LEV WAL DB SLOW * *
>>> REAL FILES
>>> LOG 12 MiB 18 MiB 0 B 0 B 0 B
>>> 10 MiB 0
>>> WAL 6.5 GiB 0 B 0 B 0 B 0 B
>>> 6.4 GiB 102
>>> DB 0 B 573 MiB 0 B 0 B 0 B
>>> 557 MiB 22
>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> 0 B 0
>>> TOTALS 6.5 GiB 591 MiB 0 B 0 B 0 B
>>> 0 B 125
>>> MAXIMUMS:
>>> LOG 12 MiB 18 MiB 0 B 0 B 0 B
>>> 17 MiB
>>> WAL 45 GiB 101 GiB 0 B 0 B 0 B
>>> 145 GiB
>>> DB 0 B 688 MiB 0 B 0 B 0 B
>>> 670 MiB
>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> 0 B
>>> TOTALS 45 GiB 101 GiB 0 B 0 B 0 B
>>> 0 B
>>>
>>> ----
>>>
>>>
>>> Here are the osd.2... the problem disapeared from alone
>>> very strange...
>>>
>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
>>> 0 : device size 0xb3ffff000 : using 0x7bcc00000(31 GiB)
>>> 1 : device size 0x2bbfffe000 : using 0x458c00000(17 GiB)
>>> 2 : device size 0x57541c00000 : using 0x5cd3665000(371 GiB)
>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>>> db_total:178509578240, slow_total:5701113793740, db_avail:103884521472
>>> Usage matrix:
>>> DEV/LEV WAL DB SLOW * *
>>> REAL FILES
>>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B
>>> 10 MiB 1
>>> WAL 31 GiB 17 GiB 0 B 0 B 0 B
>>> 48 GiB 765
>>> DB 0 B 193 MiB 0 B 0 B 0 B
>>> 175 MiB 30
>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> 0 B 0
>>> TOTALS 32 GiB 21 GiB 0 B 0 B 0 B
>>> 0 B 796
>>> MAXIMUMS:
>>> LOG 920 MiB 4.0 GiB 0 B 0 B 0 B
>>> 17 MiB
>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
>>> 192 GiB
>>> DB 0 B 762 MiB 0 B 0 B 0 B
>>> 741 MiB
>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>> 0 B
>>> TOTALS 45 GiB 153 GiB 0 B 0 B 0 B
>>> 0 B
>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device info
>>> {
>>> "dev": {
>>> "device": "BDEV_WAL",
>>> "total": 48318377984,
>>> "free": 15043915776,
>>> "bluefs_used": 33274462208
>>> },
>>> "dev": {
>>> "device": "BDEV_DB",
>>> "total": 187904811008,
>>> "free": 169235963904,
>>> "bluefs_used": 18668847104
>>> },
>>> "dev": {
>>> "device": "BDEV_SLOW",
>>> "total": 6001172414464,
>>> "free": 5602453327872,
>>> "bluefs_used": 0,
>>> "bluefs max available": 5600865222656
>>> }
>>> }
>>>
>>>
>>>
>>>
>>>
>>>
>>> Am Mi., 6. Okt. 2021 um 18:11 Uhr schrieb Igor Fedotov <
>>> igor.fedo...@croit.io>:
>>>
>>>>
>>>> On 10/6/2021 4:25 PM, José H. Freidhof wrote:
>>>> > hi,
>>>> >
>>>> > no risk no fun 😂 okay
>>>> > I have reset the settings you mentioned to standard.
>>>> >
>>>> > what you exactly mean with taking offline the osd? ceph orch daemon
>>>> stop
>>>> > osd.2? or mark down?
>>>> "daemon stop" is enough. You might want to set noout flag before that
>>>> though...
>>>> >
>>>> > for the command which path i use? you mean:
>>>> >
>>>> > bluestore-kv /var/lib/ceph/$fsid/osd.2 compact???
>>>> yep
>>>> >
>>>> >
>>>> > Igor Fedotov <ifedo...@suse.de> schrieb am Mi., 6. Okt. 2021, 13:33:
>>>> >
>>>> >> On 10/6/2021 2:16 PM, José H. Freidhof wrote:
>>>> >>> Hi Igor,
>>>> >>>
>>>> >>> yes i have some osd settings set :-) here are my ceph config dump.
>>>> those
>>>> >>> settings are from a redhat document for bluestore devices
>>>> >>> maybe it is that setting causing this problem? "advanced
>>>> >>> mon_compact_on_trim false"???
>>>> >> OMG!!!
>>>> >>
>>>> >> No - mon_compact_on_trim has nothing to deal with bluestore.
>>>> >>
>>>> >> Highly likely it's bluestore_rocksdb_options which hurts...
>>>> >> Documentations tend to fall behind the best practices.... I would
>>>> >> strongly discourage you from using non-default settings unless it's
>>>> >> absolutely clear why this is necessary.
>>>> >>
>>>> >> Even at the first glance the following settings (just a few ones I'm
>>>> >> completely aware) are suboptimal/non-recommended:
>>>> >>
>>>> >> rocksdb_perf
>>>> >>
>>>> >> bluefs_sync_write
>>>> >>
>>>> >> bluefs_csum_type
>>>> >>
>>>> >>
>>>> >> Not to mention bluestore_rocksdb_options which hasn't got much
>>>> adoption
>>>> >> so far and apparently greatly alters rocksdb behavior...
>>>> >>
>>>> >>
>>>> >> So I would suggest to revert rocksdb options back to default, run the
>>>> >> compaction and if it succeeds monitor the OSD for a while. Then if it
>>>> >> works fine - apply the same for others
>>>> >>
>>>> >>
>>>> >> Hope this helps,
>>>> >>
>>>> >> Igor
>>>> >>
>>>> >>
>>>> >>
>>>> >>> i will test it this afternoon... at the moment are everything semi
>>>> >>> prodcuctive and i need to repair one osd node.. because i think of
>>>> this
>>>> >>> reason the osds crashed on the node and the osd container crashes
>>>> with a
>>>> >>> dump while coming up now.
>>>> >>> need first to replicate all between all three nodes and then i can
>>>> take
>>>> >>> offline the osd.2.and test your command. i will inform you later...
>>>> >>>
>>>> >>> root@cd88-ceph-osdh-01:/# ceph config dump
>>>> >>> WHO MASK
>>>> >>> LEVEL OPTION VALUE
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> RO
>>>> >>> global
>>>> >>> advanced leveldb_max_open_files 131072
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced mon_compact_on_trim false
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> dev ms_crc_data false
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced osd_deep_scrub_interval
>>>> 1209600.000000
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced osd_max_scrubs 16
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced osd_scrub_load_threshold 0.010000
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced osd_scrub_max_interval
>>>> 1209600.000000
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced osd_scrub_min_interval
>>>> 86400.000000
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced perf true
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced rbd_readahead_disable_after_bytes 0
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced rbd_readahead_max_bytes 4194304
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced rocksdb_perf true
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> global
>>>> >>> advanced throttler_perf_counter false
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> mon
>>>> >>> advanced auth_allow_insecure_global_id_reclaim false
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> mon
>>>> >>> advanced cluster_network
>>>> 10.50.50.0/24
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> mon
>>>> >>> advanced mon_osd_down_out_interval 300
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> mon
>>>> >>> advanced public_network
>>>> 10.50.50.0/24
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> mgr
>>>> >>> advanced mgr/cephadm/container_init True
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> mgr
>>>> >>> advanced mgr/cephadm/device_enhanced_scan true
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> mgr
>>>> >>> advanced mgr/cephadm/migration_current 2
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> mgr
>>>> >>> advanced mgr/cephadm/warn_on_stray_daemons false
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> mgr
>>>> >>> advanced mgr/cephadm/warn_on_stray_hosts false
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> osd
>>>> >>> advanced bluefs_sync_write true
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_cache_autotune true
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_cache_kv_ratio 0.200000
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_cache_meta_ratio 0.800000
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_cache_size 2147483648
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_cache_size_hdd 2147483648
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced bluestore_csum_type none
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_extent_map_shard_max_size 200
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_extent_map_shard_min_size 50
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev bluestore_extent_map_shard_target_size 100
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced bluestore_rocksdb_options
>>>> >>>
>>>> >>
>>>>
>>>> compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
>>>> >>> *
>>>> >>> osd
>>>> >>> advanced mon_osd_cache_size 1024
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev ms_crc_data false
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced osd_map_share_max_epochs 5
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced osd_max_backfills 1
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev osd_max_pg_log_entries 10
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev osd_memory_cache_min 3000000000
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >> host:cd133-ceph-osdh-01
>>>> >>> basic osd_memory_target 5797322383
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >> host:cd133k-ceph-osdh-01
>>>> >>> basic osd_memory_target 9402402385
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> host:cd88-ceph-osdh-01
>>>> >>> basic osd_memory_target 5797322096
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced osd_memory_target_autotune true
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev osd_min_pg_log_entries 10
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced osd_op_num_shards 8
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> osd
>>>> >>> advanced osd_op_num_threads_per_shard 2
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> *
>>>> >>> osd
>>>> >>> dev osd_pg_log_dups_tracked 10
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> dev osd_pg_log_trim_min 10
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced osd_recovery_max_active 3
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced osd_recovery_max_single_start 1
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> osd
>>>> >>> advanced osd_recovery_sleep 0.000000
>>>> >>>
>>>> >>>
>>>> >>> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor Fedotov <
>>>> ifedo...@suse.de
>>>> >>> :
>>>> >>>
>>>> >>>> Jose,
>>>> >>>>
>>>> >>>> In fact 48GB is a way too much for WAL drive - usually the write
>>>> ahead
>>>> >> log
>>>> >>>> tend to be 2-4 GBs.
>>>> >>>>
>>>> >>>> But in your case it's ~150GB, while DB itself is very small
>>>> (146MB!!!):
>>>> >>>>
>>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
>>>> >>>> 154 GiB 2400
>>>> >>>>
>>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
>>>> >>>> 146 MiB 30
>>>> >>>>
>>>> >>>>
>>>> >>>> which means that there are some issues with RocksDB's WAL
>>>> processing,
>>>> >>>> which needs some troubleshooting...
>>>> >>>>
>>>> >>>> Curious if other OSDs are suffering from the same and whether you
>>>> have
>>>> >> any
>>>> >>>> custom settings for your OSD(s)?
>>>> >>>>
>>>> >>>> Additionally you might want to try the following command to
>>>> compact this
>>>> >>>> specific OSD manually and check if this would normalize the DB
>>>> layout -
>>>> >> the
>>>> >>>> majority of data has to be at DB level not WAL. Please share the
>>>> >> resulting
>>>> >>>> layout (reported by "ceph daemon osd.2 bluefs stats" command)
>>>> after the
>>>> >>>> compaction is fulfiled and OSD is restarted.
>>>> >>>>
>>>> >>>> The compaction command to be applied on an offline OSD:
>>>> >> "ceph-kvstore-tool
>>>> >>>> bluestore-kv <path-to-osd> compact"
>>>> >>>>
>>>> >>>> Even if the above works great please refrain from applying that
>>>> >> compaction
>>>> >>>> to every OSD - let's see how that "compacted" OSD evolves.Would
>>>> WAL grow
>>>> >>>> again or not?
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>>
>>>> >>>> Igor
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On 10/6/2021 1:35 PM, José H. Freidhof wrote:
>>>> >>>>
>>>> >>>> Hello Igor,
>>>> >>>>
>>>> >>>> yes the volume is nvme wal partitions for the bluestore
>>>> devicegroups
>>>> >> are
>>>> >>>> only 48gb each
>>>> >>>>
>>>> >>>> on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb
>>>> (WAL)
>>>> >>>> on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb
>>>> >> (rock.db)
>>>> >>>> on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db)
>>>> >>>>
>>>> >>>> each blustore have 1 partition nvme,ssd and hdd like described in
>>>> the
>>>> >>>> documentation
>>>> >>>>
>>>> >>
>>>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
>>>> >>>> is this to small or can i adjust the max allocation on the wal nvme
>>>> >> device
>>>> >>>> in the ceph configuration?
>>>> >>>> i know that the ssd and nvme are to small for those 5.5tb disk...
>>>> its 1%
>>>> >>>> only ot the rotation disk.
>>>> >>>> i am new in ceph and still or always learning, but we are in a
>>>> little
>>>> >>>> hurry because our other datastores are old and full.
>>>> >>>>
>>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs
>>>> device
>>>> >> info
>>>> >>>> {
>>>> >>>> "dev": {
>>>> >>>> "device": "BDEV_WAL",
>>>> >>>> "total": 48318377984,
>>>> >>>> "free": 1044480,
>>>> >>>> "bluefs_used": 48317333504
>>>> >>>> },
>>>> >>>> "dev": {
>>>> >>>> "device": "BDEV_DB",
>>>> >>>> "total": 187904811008,
>>>> >>>> "free": 68757217280,
>>>> >>>> "bluefs_used": 119147593728
>>>> >>>> },
>>>> >>>> "dev": {
>>>> >>>> "device": "BDEV_SLOW",
>>>> >>>> "total": 6001172414464,
>>>> >>>> "free": 5624912359424,
>>>> >>>> "bluefs_used": 0,
>>>> >>>> "bluefs max available": 5624401231872
>>>> >>>> }
>>>> >>>> }
>>>> >>>> root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
>>>> >>>> 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
>>>> >>>> 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB)
>>>> >>>> 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB)
>>>> >>>> RocksDBBlueFSVolumeSelector: wal_total:45902462976,
>>>> >> db_total:178509578240,
>>>> >>>> slow_total:5701113793740, db_avail:103884521472
>>>> >>>> Usage matrix:
>>>> >>>> DEV/LEV WAL DB SLOW * *
>>>> >>>> REAL FILES
>>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
>>>> >>>> 7.5 MiB 1
>>>> >>>> WAL 45 GiB 111 GiB 0 B 0 B 0 B
>>>> >>>> 154 GiB 2400
>>>> >>>> DB 0 B 164 MiB 0 B 0 B 0 B
>>>> >>>> 146 MiB 30
>>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>>> >> 0
>>>> >>>> B 0
>>>> >>>> TOTALS 45 GiB 113 GiB 0 B 0 B 0 B
>>>> >> 0
>>>> >>>> B 2431
>>>> >>>> MAXIMUMS:
>>>> >>>> LOG 124 MiB 2.3 GiB 0 B 0 B 0 B
>>>> >> 17
>>>> >>>> MiB
>>>> >>>> WAL 45 GiB 149 GiB 0 B 0 B 0 B
>>>> >>>> 192 GiB
>>>> >>>> DB 0 B 762 MiB 0 B 0 B 0 B
>>>> >>>> 741 MiB
>>>> >>>> SLOW 0 B 0 B 0 B 0 B 0 B
>>>> >> 0 B
>>>> >>>> TOTALS 45 GiB 150 GiB 0 B 0 B 0 B
>>>> >> 0 B
>>>> >>>> Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov <
>>>> >> ifedo...@suse.de>:
>>>> >>>>> Hey Jose,
>>>> >>>>>
>>>> >>>>> it looks like your WAL volume is out of space which looks weird
>>>> given
>>>> >>>>> its capacity = 48Gb.
>>>> >>>>>
>>>> >>>>> Could you please share the output of the following commands:
>>>> >>>>>
>>>> >>>>> ceph daemon osd.N bluestore bluefs device info
>>>> >>>>>
>>>> >>>>> ceph daemon osd.N bluefs stats
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Thanks,
>>>> >>>>>
>>>> >>>>> Igor
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On 10/6/2021 12:24 PM, José H. Freidhof wrote:
>>>> >>>>>> Hello together
>>>> >>>>>>
>>>> >>>>>> we have a running ceph pacific 16.2.5 cluster and i found this
>>>> >> messages
>>>> >>>>> in
>>>> >>>>>> the service logs of the osd daemons.
>>>> >>>>>>
>>>> >>>>>> we have three osd nodes .. each node has 20osds as bluestore with
>>>> >>>>>> nvme/ssd/hdd
>>>> >>>>>>
>>>> >>>>>> is this a bug or maybe i have some settings wrong?
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:25.821+0000
>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>>> bdev
>>>> >> 0,
>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>>> >> 0xb40000000,
>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:29.857+0000
>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>>> bdev
>>>> >> 0,
>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>>> >> 0xb40000000,
>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.073+0000
>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x400000 on
>>>> bdev
>>>> >> 0,
>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>>> >> 0xb40000000,
>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.405+0000
>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>>> bdev
>>>> >> 0,
>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>>> >> 0xb40000000,
>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.465+0000
>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>>> bdev
>>>> >> 0,
>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>>> >> 0xb40000000,
>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.529+0000
>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>>> bdev
>>>> >> 0,
>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>>> >> 0xb40000000,
>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>>> >>>>>> 7f38eebd4700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] New
>>>> >>>>> memtable
>>>> >>>>>> created with log file: #9588. Immutable memtables: 1.
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>>> >>>>>> 7f38eebd4700 1 bluefs _allocate unable to allocate 0x100000 on
>>>> bdev
>>>> >> 0,
>>>> >>>>>> allocator name bluefs-wal, allocator type hybrid, capacity
>>>> >> 0xb40000000,
>>>> >>>>>> block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.547575)
>>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:2198] Calling
>>>> >>>>>> FlushMemTableToOutputFile with column family [L], flush slots
>>>> >> available
>>>> >>>>> 1,
>>>> >>>>>> compaction slots available 1, flush slots scheduled 1, compaction
>>>> >> slots
>>>> >>>>>> scheduled 0
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709]
>>>> Flushing
>>>> >>>>>> memtable with next log file: 9587
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:321] [L] [JOB 5709]
>>>> Flushing
>>>> >>>>>> memtable with next log file: 9588
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850547916,
>>>> >>>>>> "job": 5709, "event": "flush_started", "num_memtables": 2,
>>>> >>>>> "num_entries":
>>>> >>>>>> 4146, "num_deletes": 0, "total_data_size": 127203926,
>>>> "memory_usage":
>>>> >>>>>> 130479920, "flush_reason": "Write Buffer Full"}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:350] [L] [JOB 5709]
>>>> Level-0
>>>> >> flush
>>>> >>>>>> table #9589: started
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850559292,
>>>> >>>>>> "cf_name": "L", "job": 5709, "event": "table_file_creation",
>>>> >>>>> "file_number":
>>>> >>>>>> 9589, "file_size": 3249934, "table_properties": {"data_size":
>>>> 3247855,
>>>> >>>>>> "index_size": 1031, "index_partitions": 0,
>>>> "top_level_index_size": 0,
>>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
>>>> >>>>>> "filter_size": 197, "raw_key_size": 1088,
>>>> "raw_average_key_size": 16,
>>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 47739,
>>>> >>>>>> "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32,
>>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0,
>>>> "format_version":
>>>> >> 0,
>>>> >>>>>> "fixed_key_len": 0, "filter_policy":
>>>> "rocksdb.BuiltinBloomFilter",
>>>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
>>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
>>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
>>>> >>>>>> "compression": "NoCompression", "compression_options":
>>>> >> "window_bits=-14;
>>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0;
>>>> zstd_max_train_bytes=0;
>>>> >>>>>> enabled=0; ", "creation_time": 1633511730, "oldest_key_time":
>>>> >>>>> 1633511730,
>>>> >>>>>> "file_creation_time": 1633511850}}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: [flush_job.cc:401] [L] [JOB 5709]
>>>> Level-0
>>>> >> flush
>>>> >>>>>> table #9589: 3249934 bytes OK
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.559362)
>>>> >>>>>> [memtable_list.cc:447] [L] Level-0 commit table #9589 started
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.559583)
>>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable
>>>> #1
>>>> >> done
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.559586)
>>>> >>>>>> [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable
>>>> #2
>>>> >> done
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.559601)
>>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709,
>>>> "event":
>>>> >>>>>> "flush_finished", "output_compression": "NoCompression",
>>>> "lsm_state":
>>>> >>>>> [8,
>>>> >>>>>> 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f3905c02700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.559638)
>>>> >>>>>> [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary:
>>>> files[8 1
>>>> >>>>> 0 0
>>>> >>>>>> 0 0 0] max score 1.00
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1676] [L]
>>>> [JOB
>>>> >>>>> 5710]
>>>> >>>>>> Compacting 8@0 + 1@1 files to L1, score 1.00
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1680] [L]
>>>> >>>>> Compaction
>>>> >>>>>> start summary: Base version 3090 Base level 0, inputs:
>>>> [9589(3173KB)
>>>> >>>>>> 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB)
>>>> >>>>> 9570(405KB)
>>>> >>>>>> 9567(29KB)], [9564(1115KB)]
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850559956,
>>>> >>>>>> "job": 5710, "event": "compaction_started", "compaction_reason":
>>>> >>>>>> "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576,
>>>> 9573,
>>>> >>>>> 9570,
>>>> >>>>>> 9567], "files_L1": [9564], "score": 1, "input_data_size":
>>>> 19542092}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1349] [L]
>>>> [JOB
>>>> >>>>> 5710]
>>>> >>>>>> Generated table #9590: 36 keys, 3249524 bytes
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850582987,
>>>> >>>>>> "cf_name": "L", "job": 5710, "event": "table_file_creation",
>>>> >>>>> "file_number":
>>>> >>>>>> 9590, "file_size": 3249524, "table_properties": {"data_size":
>>>> 3247449,
>>>> >>>>>> "index_size": 1031, "index_partitions": 0,
>>>> "top_level_index_size": 0,
>>>> >>>>>> "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
>>>> >>>>>> "filter_size": 197, "raw_key_size": 576, "raw_average_key_size":
>>>> 16,
>>>> >>>>>> "raw_value_size": 3246252, "raw_average_value_size": 90173,
>>>> >>>>>> "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0,
>>>> >>>>>> "num_merge_operands": 0, "num_range_deletions": 0,
>>>> "format_version":
>>>> >> 0,
>>>> >>>>>> "fixed_key_len": 0, "filter_policy":
>>>> "rocksdb.BuiltinBloomFilter",
>>>> >>>>>> "column_family_name": "L", "column_family_id": 10, "comparator":
>>>> >>>>>> "leveldb.BytewiseComparator", "merge_operator": "nullptr",
>>>> >>>>>> "prefix_extractor_name": "nullptr", "property_collectors": "[]",
>>>> >>>>>> "compression": "NoCompression", "compression_options":
>>>> >> "window_bits=-14;
>>>> >>>>>> level=32767; strategy=0; max_dict_bytes=0;
>>>> zstd_max_train_bytes=0;
>>>> >>>>>> enabled=0; ", "creation_time": 1633471854, "oldest_key_time": 0,
>>>> >>>>>> "file_creation_time": 1633511850}}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: [compaction/compaction_job.cc:1415] [L]
>>>> [JOB
>>>> >>>>> 5710]
>>>> >>>>>> Compacted 8@0 + 1@1 files to L1 => 3249524 bytes
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.583469)
>>>> >>>>>> [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 0
>>>> 0 0 0
>>>> >>>>> 0]
>>>> >>>>>> max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files in(8,
>>>> 1)
>>>> >>>>> out(1)
>>>> >>>>>> MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2)
>>>> write-amplify(0.2)
>>>> >>>>> OK,
>>>> >>>>>> records in: 376, records dropped: 340 output_compression:
>>>> >> NoCompression
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: (Original Log Time
>>>> >> 2021/10/06-09:17:30.583498)
>>>> >>>>>> EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710,
>>>> "event":
>>>> >>>>>> "compaction_finished", "compaction_time_micros": 23098,
>>>> >>>>>> "compaction_time_cpu_micros": 20039, "output_level": 1,
>>>> >>>>> "num_output_files":
>>>> >>>>>> 1, "total_output_size": 3249524, "num_input_records": 376,
>>>> >>>>>> "num_output_records": 36, "num_subcompactions": 1,
>>>> >> "output_compression":
>>>> >>>>>> "NoCompression", "num_single_delete_mismatches": 0,
>>>> >>>>>> "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0,
>>>> 0,
>>>> >> 0]}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583615,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9589}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583648,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9586}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583675,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9583}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583709,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9580}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583739,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9576}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583769,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9573}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583804,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9570}
>>>> >>>>>> cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
>>>> >>>>>> 7f38fb3ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros":
>>>> >> 1633511850583835,
>>>> >>>>>> "job": 5710, "event": "table_file_deletion", "file_number": 9567}
>>>> >>>>>> _______________________________________________
>>>> >>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> >>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>> >>>> --
>>>>
>>>
>>>
>
> --
>
> Mit freundlichen Grüßen,
>
> -
>
> José H. Freidhof
>
> Reyerhütterstrasse 130b
> 41065 Mönchengladbach
> eMail: harald.freid...@gmail.com
> mobil: +49 (0) 1523 – 717 7801
>
>
--
Mit freundlichen Grüßen,
-
José H. Freidhof
Reyerhütterstrasse 130b
41065 Mönchengladbach
eMail: harald.freid...@gmail.com
mobil: +49 (0) 1523 – 717 7801
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
