HI Igor , Here the assert line and the 10 line that goes before and after .
-4> 2019-06-21 10:54:45.161493 7f689291ed00 20 bluefs _read left 0x8000 len 0x8000 -3> 2019-06-21 10:54:45.161497 7f689291ed00 20 bluefs _read got 32768 -2> 2019-06-21 10:54:45.161533 7f689291ed00 10 bluefs _read h 0x55bab9dbfc00 0x6d00000~8000 from file(ino 165 size 0xf5775f4 mtime 2019-06-19 02:49:24.017808 bdev 1 allocated f700000 extents [1:0xbfeb00000+100000,1:0xbfec00000+100000,1:0xbfed00000+100000,1:0xbfee00000+100000,1:0xbfef00000+100000,1:0xbff000000+100000,1:0xbff100000+100000,1:0xbff200000+100000,1:0xbff300000+100000,1:0xbff400000+100000,1:0xbff500000+100000,1:0xbff600000+100000,1:0xbff700000+100000,1:0xbff800000+100000,1:0xbff900000+100000,1:0xbffa00000+100000,1:0xbffb00000+100000,1:0xbffc00000+100000,1:0xbffd00000+100000,1:0xbffe00000+100000,1:0xbfff00000+100000,1:0xc00000000+100000,1:0xc00100000+100000,1:0xc00200000+100000,1:0xc00300000+100000,1:0xc00400000+100000,1:0xc00500000+100000,1:0xc00600000+100000,1:0xc00700000+100000,1:0xc00800000+100000,1:0xc00900000+100000,1:0xc00a00000+100000,1:0xc00b00000+100000,1:0xc00c00000+100000,1:0xc00d00000+100000,1:0xc00e00000+100000,1:0xc00f00000+100000,1:0xc01000000+100000,1:0xc01100000+100000,1:0xc01200000+100000,1:0xc01300000+100000,1:0xc01400000+100000,1:0xc01500000+100000,1:0xc01600000+100000,1:0xc01700000+100000,1:0xc01800000+100000,1:0xc01900000+100000,1:0xc01a00000+100000,1:0xc01b00000+100000,1:0xc01c00000+100000,1:0xc01d00000+100000,1:0xc01e00000+100000,1:0xc01f00000+100000,1:0xc02000000+100000,1:0xc02100000+100000,1:0xc02200000+100000,1:0xc02300000+100000,1:0xc02400000+100000,1:0xc02500000+100000,1:0xc02600000+100000,1:0xc02700000+100000,1:0xc02800000+100000,1:0xc02900000+100000,1:0xc02a00000+100000,1:0xc02b00000+100000,1:0xc02c00000+100000,1:0xc02d00000+100000,1:0xc02e00000+100000,1:0xc02f00000+100000,1:0xc03000000+100000,1:0xc03100000+100000,1:0xc03200000+100000,1:0xc03300000+100000,1:0xc03400000+100000,1:0xc03500000+100000,1:0xc03600000+100000,1:0xc03700000+100000,1:0xc03800000+100000,1:0xc03900000+100000,1:0xc03a00000+100000,1:0xc03b00000+100000,1:0xc03c00000+100000,1:0xc03d00000+100000,1:0xc03e00000+100000,1:0xc03f00000+100000,1:0xc04000000+100000,1:0xc04100000+100000,1:0xc04200000+100000,1:0xc04300000+100000,1:0xc04400000+100000,1:0xc04500000+100000,1:0xc04600000+100000,1:0xc04700000+100000,1:0xc04800000+100000,1:0xc04900000+100000,1:0xc04a00000+100000,1:0xc04b00000+100000,1:0xc04c00000+100000,1:0xc04d00000+100000,1:0xc04e00000+100000,1:0xc04f00000+100000,1:0xc05000000+100000,1:0xc05100000+100000,1:0xc05200000+100000,1:0xc05300000+100000,1:0xc05400000+100000,1:0xc05500000+100000,1:0xc05600000+100000,1:0xc05700000+100000,1:0xc05800000+100000,1:0xc05900000+100000,1:0xc05a00000+100000,1:0xc05b00000+100000,1:0xc05c00000+100000,1:0xc05d00000+100000,1:0xc05e00000+100000,1:0xc05f00000+100000,1:0xc06000000+100000,1:0xc06100000+100000,1:0xc06200000+100000,1:0xc06300000+100000,1:0xc06400000+100000,1:0xc06500000+100000,1:0xc06600000+100000,1:0xc06700000+100000,1:0xc06800000+100000,1:0xc06900000+100000,1:0xc06a00000+100000,1:0xc06b00000+100000,1:0xc06c00000+100000,1:0xc06d00000+100000,1:0xc06e00000+100000,1:0xc06f00000+100000,1:0xc07000000+100000,1:0xc07100000+100000,1:0xc07200000+100000,1:0xc07300000+100000,1:0xc07400000+100000,1:0xc07500000+100000,1:0xc07600000+100000,1:0xc07700000+100000,1:0xc07800000+100000,1:0xc07900000+100000,1:0xc07a00000+100000,1:0xc07b00000+100000,1:0xc07c00000+100000,1:0xc07d00000+100000,1:0xc07e00000+100000,1:0xc07f00000+100000,1:0xc08000000+100000,1:0xc08100000+100000,1:0xc08200000+100000,1:0xc08300000+100000,1:0xc08400000+100000,1:0xc08500000+100000,1:0xc08600000+100000,1:0xc08700000+100000,1:0xc08800000+100000,1:0xc08900000+100000,1:0xc08a00000+100000,1:0xc08b00000+100000,1:0xc08c00000+100000,1:0xc08d00000+100000,1:0xc08e00000+100000,1:0xc08f00000+100000,1:0xc09000000+100000,1:0xc09100000+100000,1:0xc09200000+100000,1:0xc09300000+100000,1:0xc09400000+100000,1:0xc09500000+100000,1:0xc09600000+100000,1:0xc09700000+100000,1:0xc09800000+100000,1:0xc09900000+100000,1:0xc09a00000+100000,1:0xc09b00000+100000,1:0xc09c00000+100000,1:0xc09d00000+100000,1:0xc09e00000+100000,1:0xc09f00000+100000,1:0xc0a000000+100000,1:0xc0a100000+100000,1:0xc0a200000+100000,1:0xc0a300000+100000,1:0xc0a400000+100000,1:0xc0a500000+100000,1:0xc0a600000+100000,1:0xc0a700000+100000,1:0xc0a800000+100000,1:0xc0a900000+100000,1:0xc0aa00000+100000,1:0xc0ab00000+100000,1:0xc0ac00000+100000,1:0xc0ad00000+100000,1:0xc0ae00000+100000,1:0xc0af00000+100000,1:0xc0b000000+100000,1:0xc0b100000+100000,1:0xc0b200000+100000,1:0xc0b300000+100000,1:0xc0b400000+100000,1:0xc0b500000+100000,1:0xc0b600000+100000,1:0xc0b700000+100000,1:0xc0b800000+100000,1:0xc0b900000+100000,1:0xc0ba00000+100000,1:0xc0bb00000+100000,1:0xc0bc00000+100000,1:0xc0bd00000+100000,1:0xc0be00000+100000,1:0xc0bf00000+100000,1:0xc0c000000+100000,1:0xc0c100000+100000,1:0xc0c200000+100000,1:0xc0c300000+100000,1:0xc0c400000+100000,1:0xc0c500000+100000,1:0xc0c600000+100000,1:0xc0c700000+100000,1:0xc0c800000+100000,1:0xc0c900000+100000,1:0xc0ca00000+100000,1:0xc0cb00000+100000,1:0xc0cc00000+100000,1:0xc0cd00000+100000,1:0xc0ce00000+100000,1:0xc0cf00000+100000,1:0xc0d000000+100000,1:0xc0d100000+100000,1:0xc0d200000+100000,1:0xc0d300000+100000,1:0xc0d400000+100000,1:0xc0d500000+100000,1:0xc0d600000+100000,1:0xc0d700000+100000,1:0xc0d800000+100000,1:0xc20e00000+100000,1:0xc20f00000+100000,1:0xc21000000+100000,1:0xc21100000+100000,1:0xc21200000+100000,1:0xc21300000+100000,1:0xc21400000+100000,1:0xc21500000+100000,1:0xc21700000+100000]) -1> 2019-06-21 10:54:45.161646 7f689291ed00 20 bluefs _read fetching 0x0~100000 of 1:0xc05800000+100000 0> 2019-06-21 10:54:45.193256 7f689291ed00 -1 /home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7f689291ed00 time 2019-06-21 10:54:45.181081 /home/abuild/rpmbuild/BUILD/ceph-12.2.10-551-gbb089269ea/src/os/bluestore/BlueFS.cc: 976: FAILED assert(r == 0) ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x10e) [0x55baaff3172e] 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55baafeaaf44] 3: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55baafecde16] 4: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55bab028d84c] 5: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55bab0206628] 6: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55bab02066f4] -- ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable) 1: (()+0xa589a9) [0x55baafef49a9] 2: (()+0x10c10) [0x7f6890c20c10] 3: (gsignal()+0x37) [0x7f688fbe6f67] 4: (abort()+0x13a) [0x7f688fbe833a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x55baaff318a0] 6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55baafeaaf44] 7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55baafecde16] 8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55bab028d84c] 9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55bab0206628] 10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55bab02066f4] -- ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable) 1: (()+0xa589a9) [0x55baafef49a9] 2: (()+0x10c10) [0x7f6890c20c10] 3: (gsignal()+0x37) [0x7f688fbe6f67] 4: (abort()+0x13a) [0x7f688fbe833a] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x280) [0x55baaff318a0] 6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0xcb4) [0x55baafeaaf44] 7: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*, char*)+0x36) [0x55baafecde16] 8: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*, char*)+0x17c) [0x55bab028d84c] 9: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xc8) [0x55bab0206628] 10: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*)+0x34) [0x55bab02066f4] Is that one ? Am I correct ? Best Regards, Saulo Augusto Silva Em sex, 21 de jun de 2019 às 13:03, Saulo Silva <sauloaugustosi...@gmail.com> escreveu: > Hi Igor , > > I am looking at the log and I am not sure what is the exactly line that I > shuold send . > I tried to > tail -f /var/log/ceph/ceph-osd.6.log | grep -i assertion -A 5 > > But no valid result returned . > What you be the regex to get this specific line ? > I also could send the entire log . > > Best Regards, > > Saulo Augusto Silva > > Em sex, 21 de jun de 2019 às 11:39, Igor Fedotov <ifedo...@suse.de> > escreveu: > >> Saulo, >> >> please share a few log lines immediately before the assertion, not the >> starting ones. >> >> >> Thanks, >> >> Igor >> On 6/21/2019 5:37 PM, Saulo Silva wrote: >> >> Hi Igor , thanks for helping , >> >> Here part of the log : >> >> head ceph-osd.6.log -n80 >> 2019-06-21 10:50:56.090891 7f462db84d00 0 set uid:gid to 167:167 >> (ceph:ceph) >> 2019-06-21 10:50:56.090910 7f462db84d00 0 ceph version >> 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous >> (stable), process ceph-osd, pid 18014 >> 2019-06-21 10:50:56.116848 7f462db84d00 0 pidfile_write: ignore empty >> --pid-file >> 2019-06-21 10:50:56.127199 7f462db84d00 0 load: jerasure load: lrc load: >> isa >> 2019-06-21 10:50:56.435977 7f462db84d00 10 bluefs add_block_device bdev 1 >> path /var/lib/ceph/osd/ceph-6/block >> 2019-06-21 10:50:56.436187 7f462db84d00 1 bluefs add_block_device bdev 1 >> path /var/lib/ceph/osd/ceph-6/block size 99.9GiB >> 2019-06-21 10:50:56.436221 7f462db84d00 1 bluefs mount >> 2019-06-21 10:50:56.436225 7f462db84d00 10 bluefs _open_super >> 2019-06-21 10:50:56.436866 7f462db84d00 10 bluefs _open_super superblock >> 15 >> 2019-06-21 10:50:56.436893 7f462db84d00 10 bluefs _open_super log_fnode >> file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents >> [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.436904 7f462db84d00 20 bluefs _init_alloc >> 2019-06-21 10:50:56.436913 7f462db84d00 10 bluefs _replay >> 2019-06-21 10:50:56.436919 7f462db84d00 10 bluefs _replay log_fnode >> file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated 500000 extents >> [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.436923 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x0~1000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.436928 7f462db84d00 20 bluefs _read fetching >> 0x0~100000 of 1:0xc9bf00000+100000 >> 2019-06-21 10:50:56.458689 7f462db84d00 20 bluefs _read left 0x100000 len >> 0x1000 >> 2019-06-21 10:50:56.458701 7f462db84d00 20 bluefs _read got 4096 >> 2019-06-21 10:50:56.458704 7f462db84d00 20 bluefs _replay need 0x1000 >> more bytes >> 2019-06-21 10:50:56.458705 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x1000~1000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.458710 7f462db84d00 20 bluefs _read left 0xff000 len >> 0x1000 >> 2019-06-21 10:50:56.458711 7f462db84d00 20 bluefs _read got 4096 >> 2019-06-21 10:50:56.458717 7f462db84d00 10 bluefs _replay 0x0: txn(seq 1 >> len 0x1c73 crc 0xfd2d3aef) >> 2019-06-21 10:50:56.458719 7f462db84d00 20 bluefs _replay 0x0: op_init >> 2019-06-21 10:50:56.458721 7f462db84d00 20 bluefs _replay 0x0: >> op_alloc_add 1:0xbfd100000~ffd00000 >> 2019-06-21 10:50:56.458735 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 1012 size 0x41ef34e mtime 2019-05-22 >> 19:13:16.240226 bdev 1 allocated 4200000 extents [1:0xceda00000+4200000]) >> 2019-06-21 10:50:56.458765 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 167 size 0xf4da0c3 mtime 2019-05-23 >> 13:11:26.154047 bdev 1 allocated f700000 extents >> [1:0xc0d900000+100000,1:0xc11a00000+100000,1:0xc11b00000+100000,1:0xc11c00000+100000,1:0xc11d00000+100000,1:0xc11e00000+100000,1:0xc11f00000+100000,1:0xc12000000+100000,1:0xc12100000+100000,1:0xc12200000+100000,1:0xc12300000+100000,1:0xc12400000+100000,1:0xc12500000+100000,1:0xc12600000+100000,1:0xc12700000+100000,1:0xc12800000+100000,1:0xc12900000+100000,1:0xc12a00000+100000,1:0xc12b00000+100000,1:0xc12c00000+100000,1:0xc12d00000+100000,1:0xc12e00000+100000,1:0xc12f00000+100000,1:0xc13000000+100000,1:0xc13100000+100000,1:0xc13200000+100000,1:0xc13300000+100000,1:0xc13400000+100000,1:0xc13500000+100000,1:0xc13600000+100000,1:0xc13700000+100000,1:0xc13800000+100000,1:0xc13900000+100000,1:0xc13a00000+100000,1:0xc13b00000+100000,1:0xc13c00000+100000,1:0xc13d00000+100000,1:0xc13e00000+100000,1:0xc13f00000+100000,1:0xc14000000+100000,1:0xc14100000+100000,1:0xc14200000+100000,1:0xc14300000+100000,1:0xc14400000+100000,1:0xc14500000+100000,1:0xc14600000+100000,1:0xc14700000+100000,1:0xc14800000+100000,1:0xc14900000+100000,1:0xc14a00000+100000,1:0xc14b00000+100000,1:0xc14c00000+100000,1:0xc14d00000+100000,1:0xc14e00000+100000,1:0xc14f00000+100000,1:0xc15000000+100000,1:0xc15100000+100000,1:0xc15200000+100000,1:0xc15300000+100000,1:0xc15400000+100000,1:0xc15500000+100000,1:0xc15600000+100000,1:0xc15700000+100000,1:0xc15800000+100000,1:0xc15900000+100000,1:0xc15a00000+100000,1:0xc15b00000+100000,1:0xc15c00000+100000,1:0xc15d00000+100000,1:0xc15e00000+100000,1:0xc15f00000+100000,1:0xc16000000+100000,1:0xc16100000+100000,1:0xc16200000+100000,1:0xc16300000+100000,1:0xc16400000+100000,1:0xc16500000+100000,1:0xc16600000+100000,1:0xc16700000+100000,1:0xc16800000+100000,1:0xc16900000+100000,1:0xc16a00000+100000,1:0xc16b00000+100000,1:0xc16c00000+100000,1:0xc16d00000+100000,1:0xc16e00000+100000,1:0xc16f00000+100000,1:0xc17000000+100000,1:0xc17100000+100000,1:0xc17200000+100000,1:0xc17300000+100000,1:0xc17400000+100000,1:0xc17500000+100000,1:0xc17600000+100000,1:0xc17700000+100000,1:0xc17800000+100000,1:0xc17900000+100000,1:0xc17a00000+100000,1:0xc17b00000+100000,1:0xc17c00000+100000,1:0xc17d00000+100000,1:0xc17e00000+100000,1:0xc17f00000+100000,1:0xc18000000+100000,1:0xc18100000+100000,1:0xc18200000+100000,1:0xc18300000+100000,1:0xc18400000+100000,1:0xc18500000+100000,1:0xc18600000+100000,1:0xc18700000+100000,1:0xc18800000+100000,1:0xc18900000+100000,1:0xc18a00000+100000,1:0xc18b00000+100000,1:0xc18c00000+100000,1:0xc18d00000+100000,1:0xc18e00000+100000,1:0xc18f00000+100000,1:0xc19000000+100000,1:0xc19100000+100000,1:0xc19200000+100000,1:0xc19300000+100000,1:0xc19400000+100000,1:0xc19500000+100000,1:0xc19600000+100000,1:0xc19700000+100000,1:0xc19800000+100000,1:0xc19900000+100000,1:0xc19a00000+100000,1:0xc19b00000+100000,1:0xc19c00000+100000,1:0xc19d00000+100000,1:0xc19e00000+100000,1:0xc19f00000+100000,1:0xc1a000000+100000,1:0xc1a100000+100000,1:0xc1a200000+100000,1:0xc1a300000+100000,1:0xc1a400000+100000,1:0xc1a500000+100000,1:0xc1a600000+100000,1:0xc1a700000+100000,1:0xc1a800000+100000,1:0xc1a900000+100000,1:0xc1aa00000+100000,1:0xc1ab00000+100000,1:0xc1ac00000+100000,1:0xc1ad00000+100000,1:0xc1ae00000+100000,1:0xc1af00000+100000,1:0xc1b000000+100000,1:0xc1b100000+100000,1:0xc1b200000+100000,1:0xc1b300000+100000,1:0xc1b400000+100000,1:0xc1b500000+100000,1:0xc1b600000+100000,1:0xc1b700000+100000,1:0xc1b800000+100000,1:0xc1b900000+100000,1:0xc1ba00000+100000,1:0xc1bb00000+100000,1:0xc1bc00000+100000,1:0xc1bd00000+100000,1:0xc1be00000+100000,1:0xc1bf00000+100000,1:0xc1c000000+100000,1:0xc1c100000+100000,1:0xc1c200000+100000,1:0xc1c300000+100000,1:0xc1c400000+100000,1:0xc1c500000+100000,1:0xc1c600000+100000,1:0xc1c700000+100000,1:0xc1c800000+100000,1:0xc1c900000+100000,1:0xc1ca00000+100000,1:0xc1cb00000+100000,1:0xc1cc00000+100000,1:0xc1cd00000+100000,1:0xc1ce00000+100000,1:0xc1cf00000+100000,1:0xc1d000000+100000,1:0xc1d100000+100000,1:0xc1d200000+100000,1:0xc1d300000+100000,1:0xc1d400000+100000,1:0xc1d500000+100000,1:0xc1d600000+100000,1:0xc1d700000+100000,1:0xc1d800000+100000,1:0xc1d900000+100000,1:0xc1da00000+100000,1:0xc1db00000+100000,1:0xc1dc00000+100000,1:0xc1dd00000+100000,1:0xc1de00000+100000,1:0xc1df00000+100000,1:0xc1e000000+100000,1:0xc1e100000+100000,1:0xc1e200000+100000,1:0xc1e300000+100000,1:0xc1e400000+100000,1:0xc1e500000+100000,1:0xc1e600000+100000,1:0xc1e700000+100000,1:0xc1e800000+100000,1:0xc1e900000+100000,1:0xc1ea00000+100000,1:0xc1eb00000+100000,1:0xc1ec00000+100000,1:0xc1ed00000+100000,1:0xc1ee00000+100000,1:0xc1ef00000+100000,1:0xc1f000000+100000,1:0xc1f100000+100000,1:0xc1f200000+100000,1:0xc1f300000+100000,1:0xc1f400000+100000,1:0xc1f500000+100000,1:0xc1f600000+100000,1:0xc1f700000+100000,1:0xc1f800000+100000,1:0xc1f900000+100000,1:0xc1fa00000+100000,1:0xc1fb00000+100000,1:0xc1fc00000+100000,1:0xc1fd00000+100000,1:0xc1fe00000+100000,1:0xc1ff00000+100000,1:0xc20000000+100000,1:0xc20100000+100000,1:0xc20200000+100000,1:0xc20300000+100000,1:0xc20400000+100000,1:0xc20500000+100000,1:0xc20600000+100000,1:0xc20700000+100000,1:0xc20800000+100000,1:0xc20900000+100000,1:0xc20a00000+100000,1:0xc20b00000+100000,1:0xc20c00000+100000,1:0xc20d00000+100000,1:0xc21600000+100000,1:0xc21800000+100000]) >> 2019-06-21 10:50:56.458874 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 1016 size 0x0 mtime 2019-05-23 13:11:26.173532 >> bdev 1 allocated 0 extents []) >> 2019-06-21 10:50:56.458883 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 165 size 0xf4eb0ff mtime 2019-05-23 >> 13:11:26.154138 bdev 1 allocated f700000 extents >> [1:0xbfeb00000+100000,1:0xbfec00000+100000,1:0xbfed00000+100000,1:0xbfee00000+100000,1:0xbfef00000+100000,1:0xbff000000+100000,1:0xbff100000+100000,1:0xbff200000+100000,1:0xbff300000+100000,1:0xbff400000+100000,1:0xbff500000+100000,1:0xbff600000+100000,1:0xbff700000+100000,1:0xbff800000+100000,1:0xbff900000+100000,1:0xbffa00000+100000,1:0xbffb00000+100000,1:0xbffc00000+100000,1:0xbffd00000+100000,1:0xbffe00000+100000,1:0xbfff00000+100000,1:0xc00000000+100000,1:0xc00100000+100000,1:0xc00200000+100000,1:0xc00300000+100000,1:0xc00400000+100000,1:0xc00500000+100000,1:0xc00600000+100000,1:0xc00700000+100000,1:0xc00800000+100000,1:0xc00900000+100000,1:0xc00a00000+100000,1:0xc00b00000+100000,1:0xc00c00000+100000,1:0xc00d00000+100000,1:0xc00e00000+100000,1:0xc00f00000+100000,1:0xc01000000+100000,1:0xc01100000+100000,1:0xc01200000+100000,1:0xc01300000+100000,1:0xc01400000+100000,1:0xc01500000+100000,1:0xc01600000+100000,1:0xc01700000+100000,1:0xc01800000+100000,1:0xc01900000+100000,1:0xc01a00000+100000,1:0xc01b00000+100000,1:0xc01c00000+100000,1:0xc01d00000+100000,1:0xc01e00000+100000,1:0xc01f00000+100000,1:0xc02000000+100000,1:0xc02100000+100000,1:0xc02200000+100000,1:0xc02300000+100000,1:0xc02400000+100000,1:0xc02500000+100000,1:0xc02600000+100000,1:0xc02700000+100000,1:0xc02800000+100000,1:0xc02900000+100000,1:0xc02a00000+100000,1:0xc02b00000+100000,1:0xc02c00000+100000,1:0xc02d00000+100000,1:0xc02e00000+100000,1:0xc02f00000+100000,1:0xc03000000+100000,1:0xc03100000+100000,1:0xc03200000+100000,1:0xc03300000+100000,1:0xc03400000+100000,1:0xc03500000+100000,1:0xc03600000+100000,1:0xc03700000+100000,1:0xc03800000+100000,1:0xc03900000+100000,1:0xc03a00000+100000,1:0xc03b00000+100000,1:0xc03c00000+100000,1:0xc03d00000+100000,1:0xc03e00000+100000,1:0xc03f00000+100000,1:0xc04000000+100000,1:0xc04100000+100000,1:0xc04200000+100000,1:0xc04300000+100000,1:0xc04400000+100000,1:0xc04500000+100000,1:0xc04600000+100000,1:0xc04700000+100000,1:0xc04800000+100000,1:0xc04900000+100000,1:0xc04a00000+100000,1:0xc04b00000+100000,1:0xc04c00000+100000,1:0xc04d00000+100000,1:0xc04e00000+100000,1:0xc04f00000+100000,1:0xc05000000+100000,1:0xc05100000+100000,1:0xc05200000+100000,1:0xc05300000+100000,1:0xc05400000+100000,1:0xc05500000+100000,1:0xc05600000+100000,1:0xc05700000+100000,1:0xc05800000+100000,1:0xc05900000+100000,1:0xc05a00000+100000,1:0xc05b00000+100000,1:0xc05c00000+100000,1:0xc05d00000+100000,1:0xc05e00000+100000,1:0xc05f00000+100000,1:0xc06000000+100000,1:0xc06100000+100000,1:0xc06200000+100000,1:0xc06300000+100000,1:0xc06400000+100000,1:0xc06500000+100000,1:0xc06600000+100000,1:0xc06700000+100000,1:0xc06800000+100000,1:0xc06900000+100000,1:0xc06a00000+100000,1:0xc06b00000+100000,1:0xc06c00000+100000,1:0xc06d00000+100000,1:0xc06e00000+100000,1:0xc06f00000+100000,1:0xc07000000+100000,1:0xc07100000+100000,1:0xc07200000+100000,1:0xc07300000+100000,1:0xc07400000+100000,1:0xc07500000+100000,1:0xc07600000+100000,1:0xc07700000+100000,1:0xc07800000+100000,1:0xc07900000+100000,1:0xc07a00000+100000,1:0xc07b00000+100000,1:0xc07c00000+100000,1:0xc07d00000+100000,1:0xc07e00000+100000,1:0xc07f00000+100000,1:0xc08000000+100000,1:0xc08100000+100000,1:0xc08200000+100000,1:0xc08300000+100000,1:0xc08400000+100000,1:0xc08500000+100000,1:0xc08600000+100000,1:0xc08700000+100000,1:0xc08800000+100000,1:0xc08900000+100000,1:0xc08a00000+100000,1:0xc08b00000+100000,1:0xc08c00000+100000,1:0xc08d00000+100000,1:0xc08e00000+100000,1:0xc08f00000+100000,1:0xc09000000+100000,1:0xc09100000+100000,1:0xc09200000+100000,1:0xc09300000+100000,1:0xc09400000+100000,1:0xc09500000+100000,1:0xc09600000+100000,1:0xc09700000+100000,1:0xc09800000+100000,1:0xc09900000+100000,1:0xc09a00000+100000,1:0xc09b00000+100000,1:0xc09c00000+100000,1:0xc09d00000+100000,1:0xc09e00000+100000,1:0xc09f00000+100000,1:0xc0a000000+100000,1:0xc0a100000+100000,1:0xc0a200000+100000,1:0xc0a300000+100000,1:0xc0a400000+100000,1:0xc0a500000+100000,1:0xc0a600000+100000,1:0xc0a700000+100000,1:0xc0a800000+100000,1:0xc0a900000+100000,1:0xc0aa00000+100000,1:0xc0ab00000+100000,1:0xc0ac00000+100000,1:0xc0ad00000+100000,1:0xc0ae00000+100000,1:0xc0af00000+100000,1:0xc0b000000+100000,1:0xc0b100000+100000,1:0xc0b200000+100000,1:0xc0b300000+100000,1:0xc0b400000+100000,1:0xc0b500000+100000,1:0xc0b600000+100000,1:0xc0b700000+100000,1:0xc0b800000+100000,1:0xc0b900000+100000,1:0xc0ba00000+100000,1:0xc0bb00000+100000,1:0xc0bc00000+100000,1:0xc0bd00000+100000,1:0xc0be00000+100000,1:0xc0bf00000+100000,1:0xc0c000000+100000,1:0xc0c100000+100000,1:0xc0c200000+100000,1:0xc0c300000+100000,1:0xc0c400000+100000,1:0xc0c500000+100000,1:0xc0c600000+100000,1:0xc0c700000+100000,1:0xc0c800000+100000,1:0xc0c900000+100000,1:0xc0ca00000+100000,1:0xc0cb00000+100000,1:0xc0cc00000+100000,1:0xc0cd00000+100000,1:0xc0ce00000+100000,1:0xc0cf00000+100000,1:0xc0d000000+100000,1:0xc0d100000+100000,1:0xc0d200000+100000,1:0xc0d300000+100000,1:0xc0d400000+100000,1:0xc0d500000+100000,1:0xc0d600000+100000,1:0xc0d700000+100000,1:0xc0d800000+100000,1:0xc20e00000+100000,1:0xc20f00000+100000,1:0xc21000000+100000,1:0xc21100000+100000,1:0xc21200000+100000,1:0xc21300000+100000,1:0xc21400000+100000,1:0xc21500000+100000,1:0xc21700000+100000]) >> 2019-06-21 10:50:56.458949 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 166 size 0x1033 mtime 2019-03-01 11:57:12.151139 >> bdev 1 allocated 100000 extents [1:0xbfea00000+100000]) >> 2019-06-21 10:50:56.458963 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 5 size 0x25 mtime 2018-11-16 09:47:02.220665 bdev >> 1 allocated 100000 extents [1:0xbfd700000+100000]) >> 2019-06-21 10:50:56.458970 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 1014 size 0x14e71f1 mtime 2019-05-22 >> 23:10:22.033008 bdev 1 allocated 1500000 extents [1:0xcf5c00000+1500000]) >> 2019-06-21 10:50:56.458975 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 163 size 0x1204c mtime 2019-05-23 07:49:27.039370 >> bdev 1 allocated 100000 extents [1:0xbfe800000+100000]) >> 2019-06-21 10:50:56.458978 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 2 size 0x0 mtime 2018-11-16 09:47:02.202743 bdev >> 0 allocated 0 extents []) >> 2019-06-21 10:50:56.458982 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 1013 size 0x3f3b207 mtime 2019-05-22 >> 19:13:18.907005 bdev 1 allocated 4000000 extents [1:0xcf1c00000+4000000]) >> 2019-06-21 10:50:56.458997 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 146 size 0x1033 mtime 2018-12-28 19:12:38.946579 >> bdev 1 allocated 100000 extents [1:0xc66200000+100000]) >> 2019-06-21 10:50:56.459003 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 1015 size 0x7ff641 mtime 2019-05-23 >> 07:49:26.959694 bdev 1 allocated 800000 extents [1:0xcf7100000+800000]) >> 2019-06-21 10:50:56.459007 7f462db84d00 20 bluefs _replay 0x0: >> op_file_update file(ino 164 size 0x10 mtime 2019-03-01 11:57:12.126941 >> bdev 1 allocated 100000 extents [1:0xbfe900000+100000]) >> 2019-06-21 10:50:56.459013 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_create db >> 2019-06-21 10:50:56.459016 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/001610.sst to 1012 >> 2019-06-21 10:50:56.459020 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/001611.sst to 1013 >> 2019-06-21 10:50:56.459030 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/001613.sst to 1014 >> 2019-06-21 10:50:56.459035 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/001614.log to 165 >> 2019-06-21 10:50:56.459037 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/001615.sst to 1015 >> 2019-06-21 10:50:56.459039 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/001616.log to 167 >> 2019-06-21 10:50:56.459041 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/001617.sst to 1016 >> 2019-06-21 10:50:56.459043 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/CURRENT to 164 >> 2019-06-21 10:50:56.459044 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/IDENTITY to 5 >> 2019-06-21 10:50:56.459046 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/LOCK to 2 >> 2019-06-21 10:50:56.459048 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/MANIFEST-000148 to 163 >> 2019-06-21 10:50:56.459050 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/OPTIONS-000120 to 146 >> 2019-06-21 10:50:56.459052 7f462db84d00 20 bluefs _replay 0x0: >> op_dir_link db/OPTIONS-000151 to 166 >> 2019-06-21 10:50:56.459054 7f462db84d00 20 bluefs _replay 0x0: op_jump >> seq 332730 offset 0x100000 >> 2019-06-21 10:50:56.459056 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x2000~fe000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.459059 7f462db84d00 20 bluefs _read left 0xfe000 len >> 0xfe000 >> 2019-06-21 10:50:56.459060 7f462db84d00 20 bluefs _read got 1040384 >> 2019-06-21 10:50:56.459062 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x100000~1000 from file(ino 1 size 0x100000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.459115 7f462db84d00 20 bluefs _read fetching >> 0x0~100000 of 1:0xc9bb00000+400000 >> 2019-06-21 10:50:56.475215 7f462db84d00 20 bluefs _read left 0x100000 len >> 0x1000 >> 2019-06-21 10:50:56.475226 7f462db84d00 20 bluefs _read got 4096 >> 2019-06-21 10:50:56.475233 7f462db84d00 10 bluefs _replay 0x100000: >> txn(seq 332731 len 0x27 crc 0x86332d59) >> 2019-06-21 10:50:56.475248 7f462db84d00 20 bluefs _replay 0x100000: >> op_file_update file(ino 1016 size 0x7e2ee4 mtime 2019-05-23 >> 13:11:26.235932 bdev 1 allocated 800000 extents [1:0xc9c000000+800000]) >> 2019-06-21 10:50:56.475258 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x101000~1000 from file(ino 1 size 0x101000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.475262 7f462db84d00 20 bluefs _read left 0xff000 len >> 0x1000 >> 2019-06-21 10:50:56.475263 7f462db84d00 20 bluefs _read got 4096 >> 2019-06-21 10:50:56.475265 7f462db84d00 10 bluefs _replay 0x101000: >> txn(seq 332732 len 0x26 crc 0x6ee93342) >> 2019-06-21 10:50:56.475267 7f462db84d00 20 bluefs _replay 0x101000: >> op_file_update file(ino 163 size 0x12096 mtime 2019-05-23 13:11:26.314998 >> bdev 1 allocated 100000 extents [1:0xbfe800000+100000]) >> 2019-06-21 10:50:56.475271 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x102000~1000 from file(ino 1 size 0x102000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.475274 7f462db84d00 20 bluefs _read left 0xfe000 len >> 0x1000 >> 2019-06-21 10:50:56.475275 7f462db84d00 20 bluefs _read got 4096 >> 2019-06-21 10:50:56.475278 7f462db84d00 10 bluefs _replay 0x102000: >> txn(seq 332733 len 0xca5 crc 0x8634e0dc) >> 2019-06-21 10:50:56.475285 7f462db84d00 20 bluefs _replay 0x102000: >> op_file_update file(ino 167 size 0xf4dd172 mtime 2019-05-23 >> 16:05:23.255290 bdev 1 allocated f700000 extents >> [1:0xc0d900000+100000,1:0xc11a00000+100000,1:0xc11b00000+100000,1:0xc11c00000+100000,1:0xc11d00000+100000,1:0xc11e00000+100000,1:0xc11f00000+100000,1:0xc12000000+100000,1:0xc12100000+100000,1:0xc12200000+100000,1:0xc12300000+100000,1:0xc12400000+100000,1:0xc12500000+100000,1:0xc12600000+100000,1:0xc12700000+100000,1:0xc12800000+100000,1:0xc12900000+100000,1:0xc12a00000+100000,1:0xc12b00000+100000,1:0xc12c00000+100000,1:0xc12d00000+100000,1:0xc12e00000+100000,1:0xc12f00000+100000,1:0xc13000000+100000,1:0xc13100000+100000,1:0xc13200000+100000,1:0xc13300000+100000,1:0xc13400000+100000,1:0xc13500000+100000,1:0xc13600000+100000,1:0xc13700000+100000,1:0xc13800000+100000,1:0xc13900000+100000,1:0xc13a00000+100000,1:0xc13b00000+100000,1:0xc13c00000+100000,1:0xc13d00000+100000,1:0xc13e00000+100000,1:0xc13f00000+100000,1:0xc14000000+100000,1:0xc14100000+100000,1:0xc14200000+100000,1:0xc14300000+100000,1:0xc14400000+100000,1:0xc14500000+100000,1:0xc14600000+100000,1:0xc14700000+100000,1:0xc14800000+100000,1:0xc14900000+100000,1:0xc14a00000+100000,1:0xc14b00000+100000,1:0xc14c00000+100000,1:0xc14d00000+100000,1:0xc14e00000+100000,1:0xc14f00000+100000,1:0xc15000000+100000,1:0xc15100000+100000,1:0xc15200000+100000,1:0xc15300000+100000,1:0xc15400000+100000,1:0xc15500000+100000,1:0xc15600000+100000,1:0xc15700000+100000,1:0xc15800000+100000,1:0xc15900000+100000,1:0xc15a00000+100000,1:0xc15b00000+100000,1:0xc15c00000+100000,1:0xc15d00000+100000,1:0xc15e00000+100000,1:0xc15f00000+100000,1:0xc16000000+100000,1:0xc16100000+100000,1:0xc16200000+100000,1:0xc16300000+100000,1:0xc16400000+100000,1:0xc16500000+100000,1:0xc16600000+100000,1:0xc16700000+100000,1:0xc16800000+100000,1:0xc16900000+100000,1:0xc16a00000+100000,1:0xc16b00000+100000,1:0xc16c00000+100000,1:0xc16d00000+100000,1:0xc16e00000+100000,1:0xc16f00000+100000,1:0xc17000000+100000,1:0xc17100000+100000,1:0xc17200000+100000,1:0xc17300000+100000,1:0xc17400000+100000,1:0xc17500000+100000,1:0xc17600000+100000,1:0xc17700000+100000,1:0xc17800000+100000,1:0xc17900000+100000,1:0xc17a00000+100000,1:0xc17b00000+100000,1:0xc17c00000+100000,1:0xc17d00000+100000,1:0xc17e00000+100000,1:0xc17f00000+100000,1:0xc18000000+100000,1:0xc18100000+100000,1:0xc18200000+100000,1:0xc18300000+100000,1:0xc18400000+100000,1:0xc18500000+100000,1:0xc18600000+100000,1:0xc18700000+100000,1:0xc18800000+100000,1:0xc18900000+100000,1:0xc18a00000+100000,1:0xc18b00000+100000,1:0xc18c00000+100000,1:0xc18d00000+100000,1:0xc18e00000+100000,1:0xc18f00000+100000,1:0xc19000000+100000,1:0xc19100000+100000,1:0xc19200000+100000,1:0xc19300000+100000,1:0xc19400000+100000,1:0xc19500000+100000,1:0xc19600000+100000,1:0xc19700000+100000,1:0xc19800000+100000,1:0xc19900000+100000,1:0xc19a00000+100000,1:0xc19b00000+100000,1:0xc19c00000+100000,1:0xc19d00000+100000,1:0xc19e00000+100000,1:0xc19f00000+100000,1:0xc1a000000+100000,1:0xc1a100000+100000,1:0xc1a200000+100000,1:0xc1a300000+100000,1:0xc1a400000+100000,1:0xc1a500000+100000,1:0xc1a600000+100000,1:0xc1a700000+100000,1:0xc1a800000+100000,1:0xc1a900000+100000,1:0xc1aa00000+100000,1:0xc1ab00000+100000,1:0xc1ac00000+100000,1:0xc1ad00000+100000,1:0xc1ae00000+100000,1:0xc1af00000+100000,1:0xc1b000000+100000,1:0xc1b100000+100000,1:0xc1b200000+100000,1:0xc1b300000+100000,1:0xc1b400000+100000,1:0xc1b500000+100000,1:0xc1b600000+100000,1:0xc1b700000+100000,1:0xc1b800000+100000,1:0xc1b900000+100000,1:0xc1ba00000+100000,1:0xc1bb00000+100000,1:0xc1bc00000+100000,1:0xc1bd00000+100000,1:0xc1be00000+100000,1:0xc1bf00000+100000,1:0xc1c000000+100000,1:0xc1c100000+100000,1:0xc1c200000+100000,1:0xc1c300000+100000,1:0xc1c400000+100000,1:0xc1c500000+100000,1:0xc1c600000+100000,1:0xc1c700000+100000,1:0xc1c800000+100000,1:0xc1c900000+100000,1:0xc1ca00000+100000,1:0xc1cb00000+100000,1:0xc1cc00000+100000,1:0xc1cd00000+100000,1:0xc1ce00000+100000,1:0xc1cf00000+100000,1:0xc1d000000+100000,1:0xc1d100000+100000,1:0xc1d200000+100000,1:0xc1d300000+100000,1:0xc1d400000+100000,1:0xc1d500000+100000,1:0xc1d600000+100000,1:0xc1d700000+100000,1:0xc1d800000+100000,1:0xc1d900000+100000,1:0xc1da00000+100000,1:0xc1db00000+100000,1:0xc1dc00000+100000,1:0xc1dd00000+100000,1:0xc1de00000+100000,1:0xc1df00000+100000,1:0xc1e000000+100000,1:0xc1e100000+100000,1:0xc1e200000+100000,1:0xc1e300000+100000,1:0xc1e400000+100000,1:0xc1e500000+100000,1:0xc1e600000+100000,1:0xc1e700000+100000,1:0xc1e800000+100000,1:0xc1e900000+100000,1:0xc1ea00000+100000,1:0xc1eb00000+100000,1:0xc1ec00000+100000,1:0xc1ed00000+100000,1:0xc1ee00000+100000,1:0xc1ef00000+100000,1:0xc1f000000+100000,1:0xc1f100000+100000,1:0xc1f200000+100000,1:0xc1f300000+100000,1:0xc1f400000+100000,1:0xc1f500000+100000,1:0xc1f600000+100000,1:0xc1f700000+100000,1:0xc1f800000+100000,1:0xc1f900000+100000,1:0xc1fa00000+100000,1:0xc1fb00000+100000,1:0xc1fc00000+100000,1:0xc1fd00000+100000,1:0xc1fe00000+100000,1:0xc1ff00000+100000,1:0xc20000000+100000,1:0xc20100000+100000,1:0xc20200000+100000,1:0xc20300000+100000,1:0xc20400000+100000,1:0xc20500000+100000,1:0xc20600000+100000,1:0xc20700000+100000,1:0xc20800000+100000,1:0xc20900000+100000,1:0xc20a00000+100000,1:0xc20b00000+100000,1:0xc20c00000+100000,1:0xc20d00000+100000,1:0xc21600000+100000,1:0xc21800000+100000]) >> 2019-06-21 10:50:56.475353 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x103000~1000 from file(ino 1 size 0x103000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.475356 7f462db84d00 20 bluefs _read left 0xfd000 len >> 0x1000 >> 2019-06-21 10:50:56.475357 7f462db84d00 20 bluefs _read got 4096 >> 2019-06-21 10:50:56.475359 7f462db84d00 10 bluefs _replay 0x103000: >> txn(seq 332734 len 0xca5 crc 0xa28aacab) >> 2019-06-21 10:50:56.475366 7f462db84d00 20 bluefs _replay 0x103000: >> op_file_update file(ino 167 size 0xf4def05 mtime 2019-05-23 >> 16:05:23.261981 bdev 1 allocated f700000 extents >> [1:0xc0d900000+100000,1:0xc11a00000+100000,1:0xc11b00000+100000,1:0xc11c00000+100000,1:0xc11d00000+100000,1:0xc11e00000+100000,1:0xc11f00000+100000,1:0xc12000000+100000,1:0xc12100000+100000,1:0xc12200000+100000,1:0xc12300000+100000,1:0xc12400000+100000,1:0xc12500000+100000,1:0xc12600000+100000,1:0xc12700000+100000,1:0xc12800000+100000,1:0xc12900000+100000,1:0xc12a00000+100000,1:0xc12b00000+100000,1:0xc12c00000+100000,1:0xc12d00000+100000,1:0xc12e00000+100000,1:0xc12f00000+100000,1:0xc13000000+100000,1:0xc13100000+100000,1:0xc13200000+100000,1:0xc13300000+100000,1:0xc13400000+100000,1:0xc13500000+100000,1:0xc13600000+100000,1:0xc13700000+100000,1:0xc13800000+100000,1:0xc13900000+100000,1:0xc13a00000+100000,1:0xc13b00000+100000,1:0xc13c00000+100000,1:0xc13d00000+100000,1:0xc13e00000+100000,1:0xc13f00000+100000,1:0xc14000000+100000,1:0xc14100000+100000,1:0xc14200000+100000,1:0xc14300000+100000,1:0xc14400000+100000,1:0xc14500000+100000,1:0xc14600000+100000,1:0xc14700000+100000,1:0xc14800000+100000,1:0xc14900000+100000,1:0xc14a00000+100000,1:0xc14b00000+100000,1:0xc14c00000+100000,1:0xc14d00000+100000,1:0xc14e00000+100000,1:0xc14f00000+100000,1:0xc15000000+100000,1:0xc15100000+100000,1:0xc15200000+100000,1:0xc15300000+100000,1:0xc15400000+100000,1:0xc15500000+100000,1:0xc15600000+100000,1:0xc15700000+100000,1:0xc15800000+100000,1:0xc15900000+100000,1:0xc15a00000+100000,1:0xc15b00000+100000,1:0xc15c00000+100000,1:0xc15d00000+100000,1:0xc15e00000+100000,1:0xc15f00000+100000,1:0xc16000000+100000,1:0xc16100000+100000,1:0xc16200000+100000,1:0xc16300000+100000,1:0xc16400000+100000,1:0xc16500000+100000,1:0xc16600000+100000,1:0xc16700000+100000,1:0xc16800000+100000,1:0xc16900000+100000,1:0xc16a00000+100000,1:0xc16b00000+100000,1:0xc16c00000+100000,1:0xc16d00000+100000,1:0xc16e00000+100000,1:0xc16f00000+100000,1:0xc17000000+100000,1:0xc17100000+100000,1:0xc17200000+100000,1:0xc17300000+100000,1:0xc17400000+100000,1:0xc17500000+100000,1:0xc17600000+100000,1:0xc17700000+100000,1:0xc17800000+100000,1:0xc17900000+100000,1:0xc17a00000+100000,1:0xc17b00000+100000,1:0xc17c00000+100000,1:0xc17d00000+100000,1:0xc17e00000+100000,1:0xc17f00000+100000,1:0xc18000000+100000,1:0xc18100000+100000,1:0xc18200000+100000,1:0xc18300000+100000,1:0xc18400000+100000,1:0xc18500000+100000,1:0xc18600000+100000,1:0xc18700000+100000,1:0xc18800000+100000,1:0xc18900000+100000,1:0xc18a00000+100000,1:0xc18b00000+100000,1:0xc18c00000+100000,1:0xc18d00000+100000,1:0xc18e00000+100000,1:0xc18f00000+100000,1:0xc19000000+100000,1:0xc19100000+100000,1:0xc19200000+100000,1:0xc19300000+100000,1:0xc19400000+100000,1:0xc19500000+100000,1:0xc19600000+100000,1:0xc19700000+100000,1:0xc19800000+100000,1:0xc19900000+100000,1:0xc19a00000+100000,1:0xc19b00000+100000,1:0xc19c00000+100000,1:0xc19d00000+100000,1:0xc19e00000+100000,1:0xc19f00000+100000,1:0xc1a000000+100000,1:0xc1a100000+100000,1:0xc1a200000+100000,1:0xc1a300000+100000,1:0xc1a400000+100000,1:0xc1a500000+100000,1:0xc1a600000+100000,1:0xc1a700000+100000,1:0xc1a800000+100000,1:0xc1a900000+100000,1:0xc1aa00000+100000,1:0xc1ab00000+100000,1:0xc1ac00000+100000,1:0xc1ad00000+100000,1:0xc1ae00000+100000,1:0xc1af00000+100000,1:0xc1b000000+100000,1:0xc1b100000+100000,1:0xc1b200000+100000,1:0xc1b300000+100000,1:0xc1b400000+100000,1:0xc1b500000+100000,1:0xc1b600000+100000,1:0xc1b700000+100000,1:0xc1b800000+100000,1:0xc1b900000+100000,1:0xc1ba00000+100000,1:0xc1bb00000+100000,1:0xc1bc00000+100000,1:0xc1bd00000+100000,1:0xc1be00000+100000,1:0xc1bf00000+100000,1:0xc1c000000+100000,1:0xc1c100000+100000,1:0xc1c200000+100000,1:0xc1c300000+100000,1:0xc1c400000+100000,1:0xc1c500000+100000,1:0xc1c600000+100000,1:0xc1c700000+100000,1:0xc1c800000+100000,1:0xc1c900000+100000,1:0xc1ca00000+100000,1:0xc1cb00000+100000,1:0xc1cc00000+100000,1:0xc1cd00000+100000,1:0xc1ce00000+100000,1:0xc1cf00000+100000,1:0xc1d000000+100000,1:0xc1d100000+100000,1:0xc1d200000+100000,1:0xc1d300000+100000,1:0xc1d400000+100000,1:0xc1d500000+100000,1:0xc1d600000+100000,1:0xc1d700000+100000,1:0xc1d800000+100000,1:0xc1d900000+100000,1:0xc1da00000+100000,1:0xc1db00000+100000,1:0xc1dc00000+100000,1:0xc1dd00000+100000,1:0xc1de00000+100000,1:0xc1df00000+100000,1:0xc1e000000+100000,1:0xc1e100000+100000,1:0xc1e200000+100000,1:0xc1e300000+100000,1:0xc1e400000+100000,1:0xc1e500000+100000,1:0xc1e600000+100000,1:0xc1e700000+100000,1:0xc1e800000+100000,1:0xc1e900000+100000,1:0xc1ea00000+100000,1:0xc1eb00000+100000,1:0xc1ec00000+100000,1:0xc1ed00000+100000,1:0xc1ee00000+100000,1:0xc1ef00000+100000,1:0xc1f000000+100000,1:0xc1f100000+100000,1:0xc1f200000+100000,1:0xc1f300000+100000,1:0xc1f400000+100000,1:0xc1f500000+100000,1:0xc1f600000+100000,1:0xc1f700000+100000,1:0xc1f800000+100000,1:0xc1f900000+100000,1:0xc1fa00000+100000,1:0xc1fb00000+100000,1:0xc1fc00000+100000,1:0xc1fd00000+100000,1:0xc1fe00000+100000,1:0xc1ff00000+100000,1:0xc20000000+100000,1:0xc20100000+100000,1:0xc20200000+100000,1:0xc20300000+100000,1:0xc20400000+100000,1:0xc20500000+100000,1:0xc20600000+100000,1:0xc20700000+100000,1:0xc20800000+100000,1:0xc20900000+100000,1:0xc20a00000+100000,1:0xc20b00000+100000,1:0xc20c00000+100000,1:0xc20d00000+100000,1:0xc21600000+100000,1:0xc21800000+100000]) >> 2019-06-21 10:50:56.475433 7f462db84d00 10 bluefs _read h 0x5632d7074b80 >> 0x104000~1000 from file(ino 1 size 0x104000 mtime 0.000000 bdev 0 allocated >> 500000 extents [1:0xc9bf00000+100000,1:0xc9bb00000+400000]) >> 2019-06-21 10:50:56.475436 7f462db84d00 20 bluefs _read left 0xfc000 len >> 0x1000 >> 2019-06-21 10:50:56.475438 7f462db84d00 20 bluefs _read got 4096 >> 2019-06-21 10:50:56.475440 7f462db84d00 10 bluefs _replay 0x104000: >> txn(seq 332735 len 0xca5 crc 0x4715a5c6) >> >> The entire file as 17M and I can send with necessary , >> >> Saulo Augusto Silva >> >> Em sex, 21 de jun de 2019 às 06:42, Igor Fedotov <ifedo...@suse.de> >> escreveu: >> >>> Hi Saulo, >>> >>> looks like disk I/O error. >>> >>> Will you set debug_bluefs to 20 and collect the log, then share a few >>> lines prior to the assertion? >>> >>> Checking smartctl output might be a good idea too. >>> >>> Thanks, >>> >>> Igor >>> On 6/21/2019 11:30 AM, Saulo Silva wrote: >>> >>> Hi, >>> >>> After a power failure all OSD´s from a pool are fail with the following >>> error : >>> >>> -5> 2019-06-20 13:32:58.886299 7f146bcb2d00 4 rocksdb: >>> [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/version_set.cc:2859] >>> Recovered from manifest file:db/MANIFEST-003373 >>> succeeded,manifest_file_number is 3373, next_file_number is 3598, >>> last_sequence is 319489940, log_number is 0,prev_log_number is >>> 0,max_column_family is 0 >>> >>> -4> 2019-06-20 13:32:58.886330 7f146bcb2d00 4 rocksdb: >>> [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/version_set.cc:2867] >>> Column family [default] (ID 0), log number is 3594 >>> >>> -3> 2019-06-20 13:32:58.886401 7f146bcb2d00 4 rocksdb: EVENT_LOG_v1 >>> {"time_micros": 1561048378886391, "job": 1, "event": "recovery_started", >>> "log_files": [3592, 3594]} >>> -2> 2019-06-20 13:32:58.886407 7f146bcb2d00 4 rocksdb: >>> [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/db_impl_open.cc:482] >>> Recovering log #3592 mode 0 >>> -1> 2019-06-20 13:33:06.629066 7f146bcb2d00 4 rocksdb: >>> [/home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/rocksdb/db/db_impl_open.cc:482] >>> Recovering log #3594 mode 0 >>> 0> 2019-06-20 13:33:10.086512 7f146bcb2d00 -1 >>> /home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/os/bluestore/BlueFS.cc: >>> In function 'int BlueFS::_read(BlueFS::FileReader*, >>> BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' >>> thread 7f146bcb2d00 time 2019-06-20 13:33:10.073021 >>> /home/abuild/rpmbuild/BUILD/ceph-12.2.12-573-g67074fa839/src/os/bluestore/BlueFS.cc: >>> 996: FAILED assert(r == 0) >>> >>> All of OSD only read 2 log and return the error . >>> Is it possible to delete the rocksdb log and start the OSD again? >>> >>> Best Regards, >>> >>> Saulo Augusto Silva >>> >>> _______________________________________________ >>> ceph-users mailing >>> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com