This is for osd.0 (more below) bluestore(/var/lib/ceph/osd/ceph-0) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x1a128a93, expected 0x90407f75, device location [0x5826c10000~1000], logical extent 0x0~1000 bluestore(/var/lib/ceph/osd/ceph-12) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x100ac314, expected 0x90407f75, device location [0x15a0170000~1000], logical extent 0x0~1000, bluestore(/var/lib/ceph/osd/ceph-9) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb40b26a7, expected 0x90407f75, device location [0x2daea0000~1000], logical extent 0x0~1000,
I could not get this to work. ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-12 --log-file out --debug-bluestore 30 --no-log-to-stderr too many positional options have been specified on the command line ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-12 --out-dir /tmp > /tmp/fsck.out 2>&1 2017-08-09 11:01:34.208832 7fafd0171b80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore 2017-08-09 11:01:34.208927 7fafd0171b80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore action fsck 2017-08-09 11:01:34.238734 7fafd0171b80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore 2017-08-09 11:01:34.238937 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) fsck (shallow) start 2017-08-09 11:01:34.239001 7fafd0171b80 1 bdev create path /var/lib/ceph/osd/ceph-12/block type kernel 2017-08-09 11:01:34.239013 7fafd0171b80 1 bdev(0x7fafd2ac9200 /var/lib/ceph/osd/ceph-12/block) open path /var/lib/ceph/osd/ceph-12/block 2017-08-09 11:01:34.239263 7fafd0171b80 1 bdev(0x7fafd2ac9200 /var/lib/ceph/osd/ceph-12/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational 2017-08-09 11:01:34.239549 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) _set_cache_sizes max 0.5 < ratio 0.99 2017-08-09 11:01:34.239577 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 2017-08-09 11:01:34.239659 7fafd0171b80 1 bdev create path /var/lib/ceph/osd/ceph-12/block type kernel 2017-08-09 11:01:34.239664 7fafd0171b80 1 bdev(0x7fafd2ac8e00 /var/lib/ceph/osd/ceph-12/block) open path /var/lib/ceph/osd/ceph-12/block 2017-08-09 11:01:34.239815 7fafd0171b80 1 bdev(0x7fafd2ac8e00 /var/lib/ceph/osd/ceph-12/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational 2017-08-09 11:01:34.239827 7fafd0171b80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-12/block size 3725 GB 2017-08-09 11:01:34.239857 7fafd0171b80 1 bluefs mount 2017-08-09 11:01:35.239635 7fafd0171b80 0 set rocksdb option compaction_readahead_size = 2097152 2017-08-09 11:01:35.239654 7fafd0171b80 0 set rocksdb option compression = kNoCompression 2017-08-09 11:01:35.239677 7fafd0171b80 0 set rocksdb option max_write_buffer_number = 4 2017-08-09 11:01:35.239683 7fafd0171b80 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-08-09 11:01:35.239686 7fafd0171b80 0 set rocksdb option recycle_log_file_num = 4 2017-08-09 11:01:35.239690 7fafd0171b80 0 set rocksdb option writable_file_max_buffer_size = 0 2017-08-09 11:01:35.239694 7fafd0171b80 0 set rocksdb option write_buffer_size = 268435456 2017-08-09 11:01:35.239725 7fafd0171b80 0 set rocksdb option compaction_readahead_size = 2097152 2017-08-09 11:01:35.239730 7fafd0171b80 0 set rocksdb option compression = kNoCompression 2017-08-09 11:01:35.239735 7fafd0171b80 0 set rocksdb option max_write_buffer_number = 4 2017-08-09 11:01:35.239739 7fafd0171b80 0 set rocksdb option min_write_buffer_number_to_merge = 1 2017-08-09 11:01:35.239742 7fafd0171b80 0 set rocksdb option recycle_log_file_num = 4 2017-08-09 11:01:35.239746 7fafd0171b80 0 set rocksdb option writable_file_max_buffer_size = 0 2017-08-09 11:01:35.239749 7fafd0171b80 0 set rocksdb option write_buffer_size = 268435456 2017-08-09 11:01:35.239948 7fafd0171b80 4 rocksdb: RocksDB version: 5.4.0 2017-08-09 11:01:35.239959 7fafd0171b80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2017-08-09 11:01:35.239961 7fafd0171b80 4 rocksdb: Compile date Jul 17 2017 2017-08-09 11:01:35.239964 7fafd0171b80 4 rocksdb: DB SUMMARY 2017-08-09 11:01:35.240004 7fafd0171b80 4 rocksdb: CURRENT file: CURRENT 2017-08-09 11:01:35.240006 7fafd0171b80 4 rocksdb: IDENTITY file: IDENTITY 2017-08-09 11:01:35.240010 7fafd0171b80 4 rocksdb: MANIFEST file: MANIFEST-001083 size: 1868 Bytes 2017-08-09 11:01:35.240013 7fafd0171b80 4 rocksdb: SST files in db dir, Total Num: 15, files: 000529.sst 000733.sst 000904.sst 000905.sst 000906.sst 000907.sst 000908.sst 000909.sst 000910.sst 2017-08-09 11:01:35.240015 7fafd0171b80 4 rocksdb: Write Ahead Log file in db: 001084.log size: 160853870 ; 2017-08-09 11:01:35.240018 7fafd0171b80 4 rocksdb: Options.error_if_exists: 0 2017-08-09 11:01:35.240019 7fafd0171b80 4 rocksdb: Options.create_if_missing: 0 2017-08-09 11:01:35.240019 7fafd0171b80 4 rocksdb: Options.paranoid_checks: 1 2017-08-09 11:01:35.240029 7fafd0171b80 4 rocksdb: Options.env: 0x7fafd2c6bd40 2017-08-09 11:01:35.240030 7fafd0171b80 4 rocksdb: Options.info_log: 0x7fafd2c6bf00 2017-08-09 11:01:35.240031 7fafd0171b80 4 rocksdb: Options.max_open_files: -1 2017-08-09 11:01:35.240032 7fafd0171b80 4 rocksdb: Options.max_file_opening_threads: 16 2017-08-09 11:01:35.240033 7fafd0171b80 4 rocksdb: Options.use_fsync: 0 2017-08-09 11:01:35.240034 7fafd0171b80 4 rocksdb: Options.max_log_file_size: 0 2017-08-09 11:01:35.240035 7fafd0171b80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 2017-08-09 11:01:35.240037 7fafd0171b80 4 rocksdb: Options.log_file_time_to_roll: 0 2017-08-09 11:01:35.240037 7fafd0171b80 4 rocksdb: Options.keep_log_file_num: 1000 2017-08-09 11:01:35.240038 7fafd0171b80 4 rocksdb: Options.recycle_log_file_num: 4 2017-08-09 11:01:35.240039 7fafd0171b80 4 rocksdb: Options.allow_fallocate: 1 2017-08-09 11:01:35.240040 7fafd0171b80 4 rocksdb: Options.allow_mmap_reads: 0 2017-08-09 11:01:35.240041 7fafd0171b80 4 rocksdb: Options.allow_mmap_writes: 0 2017-08-09 11:01:35.240051 7fafd0171b80 4 rocksdb: Options.use_direct_reads: 0 2017-08-09 11:01:35.240053 7fafd0171b80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2017-08-09 11:01:35.240054 7fafd0171b80 4 rocksdb: Options.create_missing_column_families: 0 2017-08-09 11:01:35.240055 7fafd0171b80 4 rocksdb: Options.db_log_dir: 2017-08-09 11:01:35.240056 7fafd0171b80 4 rocksdb: Options.wal_dir: db 2017-08-09 11:01:35.240062 7fafd0171b80 4 rocksdb: Options.table_cache_numshardbits: 6 2017-08-09 11:01:35.240063 7fafd0171b80 4 rocksdb: Options.max_subcompactions: 1 2017-08-09 11:01:35.240064 7fafd0171b80 4 rocksdb: Options.max_background_flushes: 1 2017-08-09 11:01:35.240065 7fafd0171b80 4 rocksdb: Options.WAL_ttl_seconds: 0 2017-08-09 11:01:35.240067 7fafd0171b80 4 rocksdb: Options.WAL_size_limit_MB: 0 2017-08-09 11:01:35.240068 7fafd0171b80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2017-08-09 11:01:35.240069 7fafd0171b80 4 rocksdb: Options.is_fd_close_on_exec: 1 2017-08-09 11:01:35.240070 7fafd0171b80 4 rocksdb: Options.advise_random_on_open: 1 2017-08-09 11:01:35.240072 7fafd0171b80 4 rocksdb: Options.db_write_buffer_size: 0 2017-08-09 11:01:35.240073 7fafd0171b80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2017-08-09 11:01:35.240074 7fafd0171b80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 2017-08-09 11:01:35.240075 7fafd0171b80 4 rocksdb: Options.compaction_readahead_size: 2097152 2017-08-09 11:01:35.240076 7fafd0171b80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2017-08-09 11:01:35.240076 7fafd0171b80 4 rocksdb: Options.writable_file_max_buffer_size: 0 2017-08-09 11:01:35.240077 7fafd0171b80 4 rocksdb: Options.use_adaptive_mutex: 0 2017-08-09 11:01:35.240078 7fafd0171b80 4 rocksdb: Options.rate_limiter: (nil) 2017-08-09 11:01:35.240080 7fafd0171b80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2017-08-09 11:01:35.240081 7fafd0171b80 4 rocksdb: Options.bytes_per_sync: 0 2017-08-09 11:01:35.240081 7fafd0171b80 4 rocksdb: Options.wal_bytes_per_sync: 0 2017-08-09 11:01:35.240082 7fafd0171b80 4 rocksdb: Options.wal_recovery_mode: 0 2017-08-09 11:01:35.240083 7fafd0171b80 4 rocksdb: Options.enable_thread_tracking: 0 2017-08-09 11:01:35.240084 7fafd0171b80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2017-08-09 11:01:35.240085 7fafd0171b80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2017-08-09 11:01:35.240086 7fafd0171b80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2017-08-09 11:01:35.240087 7fafd0171b80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2017-08-09 11:01:35.240088 7fafd0171b80 4 rocksdb: Options.row_cache: None 2017-08-09 11:01:35.240089 7fafd0171b80 4 rocksdb: Options.wal_filter: None 2017-08-09 11:01:35.240090 7fafd0171b80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2017-08-09 11:01:35.240091 7fafd0171b80 4 rocksdb: Options.base_background_compactions: 1 2017-08-09 11:01:35.240092 7fafd0171b80 4 rocksdb: Options.max_background_compactions: 1 2017-08-09 11:01:35.240093 7fafd0171b80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2017-08-09 11:01:35.240094 7fafd0171b80 4 rocksdb: Options.delayed_write_rate : 16777216 2017-08-09 11:01:35.240095 7fafd0171b80 4 rocksdb: Options.max_total_wal_size: 0 2017-08-09 11:01:35.240096 7fafd0171b80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2017-08-09 11:01:35.240097 7fafd0171b80 4 rocksdb: Options.stats_dump_period_sec: 600 2017-08-09 11:01:35.240099 7fafd0171b80 4 rocksdb: Compression algorithms supported: 2017-08-09 11:01:35.240101 7fafd0171b80 4 rocksdb: Snappy supported: 0 2017-08-09 11:01:35.240102 7fafd0171b80 4 rocksdb: Zlib supported: 0 2017-08-09 11:01:35.240103 7fafd0171b80 4 rocksdb: Bzip supported: 0 2017-08-09 11:01:35.240104 7fafd0171b80 4 rocksdb: LZ4 supported: 0 2017-08-09 11:01:35.240105 7fafd0171b80 4 rocksdb: ZSTD supported: 0 2017-08-09 11:01:35.240108 7fafd0171b80 4 rocksdb: Fast CRC32 supported: 1 2017-08-09 11:01:35.246606 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-001083 2017-08-09 11:01:35.246707 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: 2017-08-09 11:01:35.246712 7fafd0171b80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2017-08-09 11:01:35.246721 7fafd0171b80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor 2017-08-09 11:01:35.246724 7fafd0171b80 4 rocksdb: Options.compaction_filter: None 2017-08-09 11:01:35.246726 7fafd0171b80 4 rocksdb: Options.compaction_filter_factory: None 2017-08-09 11:01:35.246727 7fafd0171b80 4 rocksdb: Options.memtable_factory: SkipListFactory 2017-08-09 11:01:35.246728 7fafd0171b80 4 rocksdb: Options.table_factory: BlockBasedTable 2017-08-09 11:01:35.246752 7fafd0171b80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fafd2aaa1f0) cache_index_and_filter_blocks: 0 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x7fafd2b03048 block_cache_name: LRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 filter_policy: nullptr whole_key_filtering: 1 format_version: 2 2017-08-09 11:01:35.246758 7fafd0171b80 4 rocksdb: Options.write_buffer_size: 268435456 2017-08-09 11:01:35.246759 7fafd0171b80 4 rocksdb: Options.max_write_buffer_number: 4 2017-08-09 11:01:35.246761 7fafd0171b80 4 rocksdb: Options.compression: NoCompression 2017-08-09 11:01:35.246762 7fafd0171b80 4 rocksdb: Options.bottommost_compression: Disabled 2017-08-09 11:01:35.246763 7fafd0171b80 4 rocksdb: Options.prefix_extractor: nullptr 2017-08-09 11:01:35.246764 7fafd0171b80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2017-08-09 11:01:35.246765 7fafd0171b80 4 rocksdb: Options.num_levels: 7 2017-08-09 11:01:35.246766 7fafd0171b80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2017-08-09 11:01:35.246767 7fafd0171b80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2017-08-09 11:01:35.246768 7fafd0171b80 4 rocksdb: Options.compression_opts.window_bits: -14 2017-08-09 11:01:35.246769 7fafd0171b80 4 rocksdb: Options.compression_opts.level: -1 2017-08-09 11:01:35.246770 7fafd0171b80 4 rocksdb: Options.compression_opts.strategy: 0 2017-08-09 11:01:35.246771 7fafd0171b80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2017-08-09 11:01:35.246772 7fafd0171b80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2017-08-09 11:01:35.246775 7fafd0171b80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2017-08-09 11:01:35.246777 7fafd0171b80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2017-08-09 11:01:35.246778 7fafd0171b80 4 rocksdb: Options.target_file_size_base: 67108864 2017-08-09 11:01:35.246779 7fafd0171b80 4 rocksdb: Options.target_file_size_multiplier: 1 2017-08-09 11:01:35.246790 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2017-08-09 11:01:35.246793 7fafd0171b80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 2017-08-09 11:01:35.246794 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2017-08-09 11:01:35.246799 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2017-08-09 11:01:35.246801 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2017-08-09 11:01:35.246802 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2017-08-09 11:01:35.246803 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2017-08-09 11:01:35.246804 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2017-08-09 11:01:35.246805 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2017-08-09 11:01:35.246806 7fafd0171b80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2017-08-09 11:01:35.246807 7fafd0171b80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2017-08-09 11:01:35.246808 7fafd0171b80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2017-08-09 11:01:35.246809 7fafd0171b80 4 rocksdb: Options.arena_block_size: 33554432 2017-08-09 11:01:35.246810 7fafd0171b80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2017-08-09 11:01:35.246811 7fafd0171b80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2017-08-09 11:01:35.246812 7fafd0171b80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2017-08-09 11:01:35.246813 7fafd0171b80 4 rocksdb: Options.disable_auto_compactions: 0 2017-08-09 11:01:35.246814 7fafd0171b80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2017-08-09 11:01:35.246824 7fafd0171b80 4 rocksdb: Options.compaction_pri: kByCompensatedSize 2017-08-09 11:01:35.246826 7fafd0171b80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2017-08-09 11:01:35.246827 7fafd0171b80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2017-08-09 11:01:35.246828 7fafd0171b80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2017-08-09 11:01:35.246829 7fafd0171b80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2017-08-09 11:01:35.246831 7fafd0171b80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2017-08-09 11:01:35.246832 7fafd0171b80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2017-08-09 11:01:35.246833 7fafd0171b80 4 rocksdb: Options.table_properties_collectors: 2017-08-09 11:01:35.246834 7fafd0171b80 4 rocksdb: Options.inplace_update_support: 0 2017-08-09 11:01:35.246835 7fafd0171b80 4 rocksdb: Options.inplace_update_num_locks: 10000 2017-08-09 11:01:35.246836 7fafd0171b80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2017-08-09 11:01:35.246837 7fafd0171b80 4 rocksdb: Options.memtable_huge_page_size: 0 2017-08-09 11:01:35.246839 7fafd0171b80 4 rocksdb: Options.bloom_locality: 0 2017-08-09 11:01:35.246840 7fafd0171b80 4 rocksdb: Options.max_successive_merges: 0 2017-08-09 11:01:35.246841 7fafd0171b80 4 rocksdb: Options.optimize_filters_for_hits: 0 2017-08-09 11:01:35.246842 7fafd0171b80 4 rocksdb: Options.paranoid_file_checks: 0 2017-08-09 11:01:35.246843 7fafd0171b80 4 rocksdb: Options.force_consistency_checks: 0 2017-08-09 11:01:35.246844 7fafd0171b80 4 rocksdb: Options.report_bg_io_stats: 0 2017-08-09 11:01:35.473935 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-001083 succeeded,manifest_file_number is 1083, next_file_number is 1085, last_sequence is 53818783, log_number is 0,prev_log_number is 0,max_column_family is 0 2017-08-09 11:01:35.473953 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 1082 2017-08-09 11:01:35.474064 7fafd0171b80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502269295474050, "job": 1, "event": "recovery_started", "log_files": [1084]} 2017-08-09 11:01:35.474071 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl_open.cc:482] Recovering log #1084 mode 0 2017-08-09 11:01:36.905449 7fafd0171b80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502269296905427, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 1085, "file_size": 1123741, "table_properties": {"data_size": 1103795, "index_size": 19072, "filter_size": 0, "raw_key_size": 247455, "raw_average_key_size": 52, "raw_value_size": 1007202, "raw_average_value_size": 215, "num_data_blocks": 277, "num_entries": 4674, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "225"}} 2017-08-09 11:01:36.907614 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/version_set.cc:2395] Creating manifest 1086 2017-08-09 11:01:36.949137 7fafd0171b80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1502269296949132, "job": 1, "event": "recovery_finished"} 2017-08-09 11:01:36.983107 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x7fafd2da2000 2017-08-09 11:01:36.983155 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_nu mber_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writa ble_file_max_buffer_size=0,compaction_readahead_size=2097152 2017-08-09 11:01:37.003047 7fafd0171b80 1 freelist init 2017-08-09 11:01:37.027518 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) _open_alloc opening allocation metadata 2017-08-09 11:01:41.875967 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) _open_alloc loaded 3718 G in 3275 extents 2017-08-09 11:01:42.524478 7fafd0171b80 1 bluefs fsck 2017-08-09 11:01:42.524502 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) fsck walking object keyspace 2017-08-09 11:02:01.008048 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) fsck checking shared_blobs 2017-08-09 11:02:01.008594 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: found stray shared blob data for sbid 0xc959 2017-08-09 11:02:01.008811 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: shared blob 0xcadc ref_map ref_map(0x15a0170000~80000=2) != expected ref_map(0x15a0170000~80000=1) 2017-08-09 11:02:01.008827 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# extent 0x15a0170000~80000 or a subset is already allocated 2017-08-09 11:02:01.009713 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: found stray shared blob data for sbid 0xcdb3 2017-08-09 11:02:01.009735 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: shared blob 0xcdd4 ref_map ref_map(0x7a14b00000~10000=2,0x81357c0000~70000=2) != expected ref_map(0x7a14b00000~10000=1,0x81357c0000~70000=1) 2017-08-09 11:02:01.009741 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) #17:6d8fbe37:::rbd_data.1f114174b0dc51.000000000000006b:4# extent 0x7a14b00000~10000 or a subset is already allocated 2017-08-09 11:02:01.009748 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) #17:6d8fbe37:::rbd_data.1f114174b0dc51.000000000000006b:4# extent 0x81357c0000~70000 or a subset is already allocated 2017-08-09 11:02:01.010672 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: found stray shared blob data for sbid 0xd08b 2017-08-09 11:02:01.016220 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: actual store_statfs(0x3a12ef00000/0x3a37b2d1000, stored 0x1db028d2b/0x1dd4c0000, compress 0x0/0x0/0x0) != expected store_statfs(0x3a12ef00000/0x3a37b2d1000, stored 0x1db028d2b/0x1dd5c0000, compress 0x0/0x0/0x0) 2017-08-09 11:02:01.016249 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) fsck checking for stray omap data 2017-08-09 11:02:01.334870 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) fsck checking deferred events 2017-08-09 11:02:01.334934 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) fsck checking freelist vs allocated 2017-08-09 11:02:03.091036 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: free extent 0x159f970000~810000 intersects allocated blocks 2017-08-09 11:02:03.091055 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: free extent 0x15a01b0000~20000 intersects allocated blocks 2017-08-09 11:02:03.091058 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: free extent 0x15a01e0000~10000 intersects allocated blocks 2017-08-09 11:02:25.466606 7fafd0171b80 -1 bluestore(/var/lib/ceph/osd/ceph-12) fsck error: leaked some space;65536 bytes leaked 2017-08-09 11:02:25.513900 7fafd0171b80 1 freelist shutdown 2017-08-09 11:02:25.514045 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work 2017-08-09 11:02:25.516412 7fafd0171b80 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343] Shutdown complete 2017-08-09 11:02:25.851561 7fafd0171b80 1 bluefs umount 2017-08-09 11:02:25.872039 7fafd0171b80 1 bdev(0x7fafd2ac8e00 /var/lib/ceph/osd/ceph-12/block) close 2017-08-09 11:02:26.107504 7fafd0171b80 1 bdev(0x7fafd2ac9200 /var/lib/ceph/osd/ceph-12/block) close 2017-08-09 11:02:26.259819 7fafd0171b80 1 bluestore(/var/lib/ceph/osd/ceph-12) fsck finish with 13 errors in 52.020867 seconds ======================================================================== ========================= objectstore.out ======================================================================== ========================= data section offset=1048576 len=1048576 data section offset=2097152 len=1048576 data section offset=3145728 len=1048576 attrs size 2 omap map size 0 Read #17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head# size=4194304 object_info: 17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head(5387'35157 client.2096993.0:78941 dirty|data_digest|omap_digest s 4194304 uv 35356 dd f53dff2e od ffffffff alloc_hint [4194304 4194304 0]) data section offset=0 len=1048576 data section offset=1048576 len=1048576 data section offset=2097152 len=1048576 data section offset=3145728 len=1048576 attrs size 2 omap map size 0 Read #17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head# size=4194304 object_info: 17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head(5163'7136 client.2074638.1:483264 dirty|data_digest|omap_digest s 4194304 uv 7418 dd 43d61c5d od ffffffff alloc_hint [4194304 4194304 0]) data section offset=0 len=1048576 data section offset=1048576 len=1048576 data section offset=2097152 len=1048576 data section offset=3145728 len=1048576 attrs size 2 omap map size 0 Read #17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head# size=4194304 object_info: 17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head(5236'7640 client.2074638.1:704364 dirty|data_digest|omap_digest s 4194304 uv 7922 dd 3bcff64d od ffffffff alloc_hint [4194304 4194304 0]) data section offset=0 len=1048576 data section offset=1048576 len=1048576 data section offset=2097152 len=1048576 data section offset=3145728 len=1048576 attrs size 2 omap map size 0 Read #17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head# size=4194304 object_info: 17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head(5387'35553 client.2096993.0:123721 dirty|data_digest|omap_digest s 4194304 uv 35752 dd f9bc0fbd od ffffffff alloc_hint [4194304 4194304 0]) data section offset=0 len=1048576 data section offset=1048576 len=1048576 data section offset=2097152 len=1048576 data section offset=3145728 len=1048576 attrs size 2 omap map size 0 Read #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# size=4194304 object_info: 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4(5390'56613 client.2096907.1:3222443 dirty|omap_digest s 4194304 uv 55477 od ffffffff alloc_hint [0 0 0]) export_files error -5 2017-08-09 11:41:25.273003 7f26db8ae100 -1 bluestore(/var/lib/ceph/osd/ceph-0) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x1a128a93, expected 0x90407f75, device location [0x5826c10000~1000], logical extent 0x0~1000, object #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# 2017-08-09 11:41:25.286087 7f26db8ae100 1 bluestore(/var/lib/ceph/osd/ceph-0) umount 2017-08-09 11:41:25.455280 7f26db8ae100 1 freelist shutdown 2017-08-09 11:41:25.455386 7f26db8ae100 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work 2017-08-09 11:41:25.471514 7f26db8ae100 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_AR CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343] Shutdown complete 2017-08-09 11:41:25.686088 7f26db8ae100 1 bluefs umount 2017-08-09 11:41:25.705389 7f26db8ae100 1 bdev(0x7f26de472e00 /var/lib/ceph/osd/ceph-0/block) close 2017-08-09 11:41:25.944548 7f26db8ae100 1 bdev(0x7f26de2b3a00 /var/lib/ceph/osd/ceph-0/block) close -----Original Message----- From: Sage Weil [mailto:s...@newdream.net] Sent: woensdag 9 augustus 2017 4:44 To: Brad Hubbard Cc: Marc Roos; ceph-users Subject: Re: [ceph-users] Pg inconsistent / export_files error -5 On Wed, 9 Aug 2017, Brad Hubbard wrote: > Wee > > On Wed, Aug 9, 2017 at 12:41 AM, Marc Roos <m.r...@f1-outsourcing.eu> wrote: > > > > > > > > The --debug indeed comes up with something > > bluestore(/var/lib/ceph/osd/ceph-12) _verify_csum bad crc32c/0x1000 > > checksum at blob offset 0x0, got 0x100ac314, expected 0x90407f75, > > device location [0x15a0170000~1000], logical extent 0x0~1000, > > bluestore(/var/lib/ceph/osd/ceph-9) _verify_csum bad crc32c/0x1000 > > checksum at blob offset 0x0, got 0xb40b26a7, expected 0x90407f75, > > device location [0x2daea0000~1000], logical extent 0x0~1000, What about the 3rd OSD? It would be interesting to capture the fsck output for one of these. Stop the OSD, and then run ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-12 --log-file out \ --debug-bluestore 30 --no-log-to-stderr That'll generate a pretty huge log, but should include dumps of onode metadata and will hopefully include something else with the checksum of 0x100ac314 so we can get some clue as to where the bad data came from. Thanks! sage > > > > I dont know how to interpret this, but am I correct to understand > > that data has been written across the cluster to these 3 osd's and > > all 3 have somehow received something different? > > Did you run this command on OSD 0? What was the output in that case? > > Possibly, all we currently know for sure is that the crc32c checksum > for the object on OSDs 12 and 9 do not match the expected checksum > according to the code when we attempt to read the object > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4#. There > seems to be some history behind this based on your previous emails > regarding these OSDs (12,9,0, and possibly 13) could you give us as > much detail as possible about how this issue came about and what you > have done in the interim to try to resolve it? > > When was the first indication there was a problem with pg 17.36? Did > this correspond with any significant event? > > Are these OSDs all on separate hosts? > > It's possible ceph-bluestore-tool may help here but I would hold off > on that option until we understand the issue better. > > > > > > > > size=4194304 object_info: > > 17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head(5387'351 > > 57 > > client.2096993.0:78941 dirty|data_digest|omap_digest s 4194304 uv > > 35356 dd f53dff2e od ffffffff alloc_hint [4194304 4194304 0]) data > > section offset=0 > > len=1048576 data section offset=1048576 len=1048576 data section > > offset=2097152 len=1048576 data section offset=3145728 len=1048576 > > attrs size > > 2 omap map size 0 Read > > #17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head# > > size=4194304 > > object_info: > > 17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head(5163'713 > > 6 > > client.2074638.1:483264 dirty|data_digest|omap_digest s 4194304 uv > > 7418 dd 43d61c5d od ffffffff alloc_hint [4194304 4194304 0]) data > > section offset=0 > > len=1048576 data section offset=1048576 len=1048576 data section > > offset=2097152 len=1048576 data section offset=3145728 len=1048576 > > attrs size > > 2 omap map size 0 Read > > #17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head# > > size=4194304 > > object_info: > > 17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head(5236'764 > > 0 > > client.2074638.1:704364 dirty|data_digest|omap_digest s 4194304 uv > > 7922 dd 3bcff64d od ffffffff alloc_hint [4194304 4194304 0]) data > > section offset=0 > > len=1048576 data section offset=1048576 len=1048576 data section > > offset=2097152 len=1048576 data section offset=3145728 len=1048576 > > attrs size > > 2 omap map size 0 Read > > #17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head# > > size=4194304 > > object_info: > > 17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head(5387'355 > > 53 > > client.2096993.0:123721 dirty|data_digest|omap_digest s 4194304 uv > > 35752 dd f9bc0fbd od ffffffff alloc_hint [4194304 4194304 0]) data > > section offset=0 > > len=1048576 data section offset=1048576 len=1048576 data section > > offset=2097152 len=1048576 data section offset=3145728 len=1048576 > > attrs size > > 2 omap map size 0 Read > > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# > > size=4194304 > > object_info: > > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4(5390'56613 > > client.2096907.1:3222443 dirty|omap_digest s 4194304 uv 55477 od > > ffffffff alloc_hint [0 0 0]) 2017-08-08 15:57:45.078348 7fad08fa4100 > > -1 > > bluestore(/var/lib/ceph/osd/ceph-12) _verify_csum bad crc32c/0x1000 > > checksum at blob offset 0x0, got 0x100ac314, expected 0x90407f75, > > device location [0x15a0170000~1000], logical extent 0x0~1000, object > > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# > > export_files error > > -5 2017-08-08 15:57:45.081279 7fad08fa4100 1 > > bluestore(/var/lib/ceph/osd/ceph-12) umount 2017-08-08 > > 15:57:45.150210 7fad08fa4100 1 freelist shutdown 2017-08-08 > > 15:57:45.150307 7fad08fa4100 4 > > rocksdb: > > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL > > E_AR > > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele > > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217] > > Shutdown: > > canceling all background work 2017-08-08 15:57:45.152099 > > 7fad08fa4100 4 > > rocksdb: > > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL > > E_AR > > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele > > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343] > > Shutdown complete 2017-08-08 15:57:45.184742 7fad08fa4100 1 bluefs > > umount 2017-08-08 > > 15:57:45.203674 7fad08fa4100 1 bdev(0x7fad0b260e00 > > /var/lib/ceph/osd/ceph-12/block) close 2017-08-08 15:57:45.442499 > > 7fad08fa4100 > > 1 bdev(0x7fad0b0a5a00 /var/lib/ceph/osd/ceph-12/block) close > > > > grep -i export_files strace.out -C 10 > > > > 814 16:08:19.261144 futex(0x7fffea9378c0, FUTEX_WAKE_PRIVATE, 1) = > > 0 <0.000010> 6814 16:08:19.261242 futex(0x7f4832bb60bc, > > FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f4832bb60b8, {FUTEX_OP_SET, 0, > > FUTEX_OP_CMP_GT, 1}) = 1 <0.000012> > > 6814 16:08:19.261281 madvise(0x7f4843bf0000, 524288, MADV_DONTNEED > > <unfinished ...> 6815 16:08:19.261382 <... futex resumed> ) = 0 > > <14.990766> > > 6814 16:08:19.261412 <... madvise resumed> ) = 0 <0.000123> 6814 > > 16:08:19.261446 madvise(0x7f4843b70000, 1048576, MADV_DONTNEED > > <unfinished ...> 6815 16:08:19.261474 futex(0x7f4832bb6038, > > FUTEX_WAKE_PRIVATE, 1 <unfinished ...> 6814 16:08:19.261535 <... > > madvise resumed> ) = 0 <0.000067> > > 6815 16:08:19.261557 <... futex resumed> ) = 0 <0.000069> 6815 > > 16:08:19.261647 futex(0x7f4832bb60bc, FUTEX_WAIT_PRIVATE, 45, NULL > > <unfinished ...> 6814 16:08:19.261700 write(2</dev/pts/0>, > > "export_files error ", 19) = > > 19 <0.000024> 6814 16:08:19.261774 write(2</dev/pts/0>, "-5", 2) = > > 2 <0.000018> 6814 16:08:19.261841 write(2</dev/pts/0>, "\n", 1) = 1 > > <0.000016> > > 6814 16:08:19.262191 madvise(0x7f4839106000, 16384, MADV_DONTNEED) > > = 0 <0.000015> 6814 16:08:19.262229 madvise(0x7f483914e000, 16384, > > MADV_DONTNEED) = 0 <0.000012> 6814 16:08:19.262295 > > madvise(0x7f48389e6000, 49152, > > MADV_DONTNEED) = 0 <0.000013> 6814 16:08:19.262498 > > madvise(0x7f48390ea000, 16384, MADV_DONTNEED) = 0 <0.000013> 6814 > > 16:08:19.262538 madvise(0x7f48390ce000, 16384, MADV_DONTNEED) = 0 > > <0.000012> 6814 16:08:19.262580 madvise(0x7f483c228000, 24576, > > MADV_DONTNEED) = 0 <0.000012> > > 6814 16:08:19.263047 madvise(0x7f48393d8000, 16384, MADV_DONTNEED) > > = 0 <0.000013> 6814 16:08:19.263081 madvise(0x7f48393d8000, 32768, > > MADV_DONTNEED) = 0 <0.000016> > > > > > > I was curious how this would compare to the osd.9 > > > > object_info: > > 17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head(5236'764 > > 0 > > client.2074638.1:704364 dirty|data_digest|omap_digest s 4194304 uv > > 7922 dd 3bcff64d od ffffffff alloc_hint [4194304 4194304 0]) data > > section offset=0 > > len=1048576 data section offset=1048576 len=1048576 data section > > offset=2097152 len=1048576 data section offset=3145728 len=1048576 > > attrs size > > 2 omap map size 0 Read > > #17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head# > > size=4194304 > > object_info: > > 17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head(5387'355 > > 53 > > client.2096993.0:123721 dirty|data_digest|omap_digest s 4194304 uv > > 35752 dd f9bc0fbd od ffffffff alloc_hint [4194304 4194304 0]) data > > section offset=0 > > len=1048576 data section offset=1048576 len=1048576 data section > > offset=2097152 len=1048576 data section offset=3145728 len=1048576 > > attrs size > > 2 omap map size 0 Read > > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# > > size=4194304 > > object_info: > > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4(5390'56613 > > client.2096907.1:3222443 dirty|omap_digest s 4194304 uv 55477 od > > ffffffff alloc_hint [0 0 0]) 2017-08-08 16:22:00.893216 7f94e10f5100 > > -1 > > bluestore(/var/lib/ceph/osd/ceph-9) _verify_csum bad crc32c/0x1000 > > checksum at blob offset 0x0, got 0xb40b26a7, expected 0x90407f75, > > device location [0x2daea0000~1000], logical extent 0x0~1000, object > > #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# > > export_files error > > -5 2017-08-08 16:22:00.895439 7f94e10f5100 1 > > bluestore(/var/lib/ceph/osd/ceph-9) umount 2017-08-08 > > 16:22:00.963774 7f94e10f5100 1 freelist shutdown 2017-08-08 > > 16:22:00.963861 7f94e10f5100 4 > > rocksdb: > > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL > > E_AR > > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele > > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:217] > > Shutdown: > > canceling all background work 2017-08-08 16:22:00.968438 > > 7f94e10f5100 4 > > rocksdb: > > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABL > > E_AR > > CH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/rele > > ase/ 12.1.1/rpm/el7/BUILD/ceph-12.1.1/src/rocksdb/db/db_impl.cc:343] > > Shutdown complete 2017-08-08 16:22:00.984583 7f94e10f5100 1 bluefs > > umount 2017-08-08 > > 16:22:01.026784 7f94e10f5100 1 bdev(0x7f94e3670e00 > > /var/lib/ceph/osd/ceph-9/block) close 2017-08-08 16:22:01.243361 > > 7f94e10f5100 > > 1 bdev(0x7f94e34b5a00 /var/lib/ceph/osd/ceph-9/block) close > > > > > > 23555 16:26:31.336061 io_getevents(139955679129600, 1, 16, > > <unfinished ...> > > 23552 16:26:31.336081 futex(0x7ffe7e4c9210, FUTEX_WAKE_PRIVATE, 1) = > > 0 <0.000155> 23552 16:26:31.336452 futex(0x7f49fb4d20bc, > > FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f49fb4d20b8, {FUTEX_OP_SET, 0, > > FUTEX_OP_CMP_GT, 1}) = 1 <0.000129> > > 23553 16:26:31.336637 <... futex resumed> ) = 0 <16.434259> 23553 > > 16:26:31.336758 futex(0x7f49fb4d2038, FUTEX_WAKE_PRIVATE, 1 > > <unfinished ...> > > 23552 16:26:31.336801 madvise(0x7f4a0cafa000, 2555904, MADV_DONTNEED > > <unfinished ...> 23553 16:26:31.336915 <... futex resumed> ) = 0 > > <0.000113> > > 23552 16:26:31.336959 <... madvise resumed> ) = 0 <0.000148> 23553 > > 16:26:31.337040 futex(0x7f49fb4d20bc, FUTEX_WAIT_PRIVATE, 55, NULL > > <unfinished ...> 23552 16:26:31.337070 madvise(0x7f4a0ca7a000, > > 3080192, MADV_DONTNEED) = 0 <0.000180> 23552 16:26:31.337424 > > write(2</dev/pts/1>, "export_files error ", > > 19) = 19 <0.000104> 23552 16:26:31.337615 write(2</dev/pts/1>, "-5", > > 2) = 2 <0.000017> 23552 16:26:31.337674 write(2</dev/pts/1>, "\n", > > 1) = 1 <0.000037> > > 23552 16:26:31.338270 madvise(0x7f4a01ae4000, 16384, MADV_DONTNEED) > > = 0 <0.000020> 23552 16:26:31.338320 madvise(0x7f4a018cc000, 49152, > > MADV_DONTNEED) = 0 <0.000014> 23552 16:26:31.338561 > > madvise(0x7f4a0770a000, 24576, > > MADV_DONTNEED) = 0 <0.000015> 23552 16:26:31.339161 > > madvise(0x7f4a02102000, 16384, MADV_DONTNEED) = 0 <0.000015> 23552 > > 16:26:31.339201 madvise(0x7f4a02132000, 16384, MADV_DONTNEED) = 0 > > <0.000013> 23552 > > 16:26:31.339235 madvise(0x7f4a02102000, 32768, MADV_DONTNEED) = 0 > > <0.000014> > > 23552 16:26:31.339331 madvise(0x7f4a01df8000, 16384, MADV_DONTNEED) > > = 0 <0.000019> 23552 16:26:31.339372 madvise(0x7f4a01df8000, 32768, > > MADV_DONTNEED) = 0 <0.000013> > > > > > > -----Original Message----- From: Brad Hubbard > > [mailto:bhubb...@redhat.com] > > Sent: 07 August 2017 02:34 To: Marc Roos Cc: ceph-users Subject: Re: > > [ceph-users] Pg inconsistent / export_files error -5 > > > > > > > > On Sat, Aug 5, 2017 at 1:21 AM, Marc Roos <m.r...@f1-outsourcing.eu> wrote: > >> > >> I have got a placement group inconsistency, and saw some manual > >> where you can export and import this on another osd. But I am > >> getting an export error on every osd. > >> > >> What does this export_files error -5 actually mean? I thought 3 > >> copies > > > > #define EIO 5 /* I/O error */ > > > >> should be enough to secure your data. > >> > >> > >>> PG_DAMAGED Possible data damage: 1 pg inconsistent pg 17.36 is > >>> active+clean+inconsistent, acting [9,0,12] > >> > >> > >>> 2017-08-04 05:39:51.534489 7f2f623d6700 -1 log_channel(cluster) > >>> log > >> [ERR] : 17.36 soid 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4: > >> failed to pick suitable object info > >>> 2017-08-04 05:41:12.715393 7f2f623d6700 -1 log_channel(cluster) > >>> log > >> [ERR] : 17.36 deep-scrub 3 errors > >>> 2017-08-04 15:21:12.445799 7f2f623d6700 -1 log_channel(cluster) > >>> log > >> [ERR] : 17.36 soid 17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4: > >> failed to pick suitable object info > >>> 2017-08-04 15:22:35.646635 7f2f623d6700 -1 log_channel(cluster) > >>> log > >> [ERR] : 17.36 repair 3 errors, 0 fixed > >> > >> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --pgid > >> 17.36 --op export --file /tmp/recover.17.36 > > > > Can you run this command under strace like so? > > > > # strace -fvttyyTo /tmp/strace.out -s 1024 ceph-objectstore-tool > > --data-path > > /var/lib/ceph/osd/ceph-12 --pgid 17.36 --op export --file > > /tmp/recover.17.36 > > > > Then see if you can find which syscall is returning EIO. > > > > # grep "= \-5" /tmp/strace.out > > > >> > >> ... Read > >> #17:6c9f811c:::rbd_data.1b42f52ae8944a.0000000000001a32:head# Read > >> #17:6ca035fc:::rbd_data.1fff61238e1f29.000000000000b31a:head# Read > >> #17:6ca0b4f8:::rbd_data.1fff61238e1f29.0000000000006fcc:head# Read > >> #17:6ca0ffbc:::rbd_data.1fff61238e1f29.000000000000a214:head# Read > >> #17:6ca10b29:::rbd_data.1fff61238e1f29.0000000000009923:head# Read > >> #17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.00000000000011b4:head# Read > >> #17:6ca13bed:::rbd_data.1f114174b0dc51.00000000000002c6:head# Read > >> #17:6ca1a791:::rbd_data.1fff61238e1f29.000000000000f101:head# Read > >> #17:6ca1f70a:::rbd_data.1f114174b0dc51.0000000000000974:4# > >> export_files error > >> -5 > > > > Running the command with "--debug" appended will give more output > > which may shed more light as well. > > > >> _______________________________________________ ceph-users mailing > >> list ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- Cheers, Brad > > > > > > > > -- > Cheers, > Brad > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com