Hi, I am quite new to ceph and just try to set up a ceph cluster. Initially I used ceph-deploy for this but when I tried to create a BlueStore osd ceph-deploy fails. Next I tried the direct way on one of the OSD-nodes using ceph-volume to create the osd, but this also fails. Below you can see what ceph-volume says.
I ensured that there was no left over lvm VG and LV on the disk sdg before I started the osd creation for this disk. The very same error happens also on other disks not just for /dev/sdg. All the disk have 4TB in size and the linux system is Ubuntu 18.04 and finally ceph is installed in version 13.2.4-1bionic from this repo: https://download.ceph.com/debian-mimic. There is a VG and two LV's on the system for the ubuntu system itself that is installed on two separate disks configured as software raid1 and lvm on top of the raid. But I cannot imagine that this might do any harm to cephs osd creation. Does anyone have an idea what might be wrong? Thanks for hints Rainer root@ceph1:~# wipefs -fa /dev/sdg root@ceph1:~# ceph-volume lvm prepare --bluestore --data /dev/sdg Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 14d041d6-0beb-4056-8df2-3920e2febce0 Running command: /sbin/vgcreate --force --yes ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b /dev/sdg stdout: Physical volume "/dev/sdg" successfully created. stdout: Volume group "ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b" successfully created Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b stdout: Logical volume "osd-block-14d041d6-0beb-4056-8df2-3920e2febce0" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 --> Absolute path not found for executable: restorecon --> Ensure $PATH environment variable contains common executable locations Running command: /bin/chown -h ceph:ceph /dev/ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b/osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 Running command: /bin/chown -R ceph:ceph /dev/dm-8 Running command: /bin/ln -s /dev/ceph-1433ffd0-0a80-481a-91f5-d7a47b78e17b/osd-block-14d041d6-0beb-4056-8df2-3920e2febce0 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: got monmap epoch 1 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAAY2VcU968HxAAvYWMaJZmriUc4H9bCCp8XQ== stdout: creating /var/lib/ceph/osd/ceph-0/keyring added entity osd.0 auth auth(auid = 18446744073709551615 key=AQAAY2VcU968HxAAvYWMaJZmriUc4H9bCCp8XQ== with 0 caps) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 14d041d6-0beb-4056-8df2-3920e2febce0 --setuser ceph --setgroup ceph stderr: 2019-02-14 13:45:54.788 7f3fcecb3240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7f3fcecb3240 time 2019-02-14 13:45:54.841130 stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: FAILED assert((uint64_t)r == len) stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3fc60d33e2] stderr: 2: (()+0x26d5a7) [0x7f3fc60d35a7] stderr: 3: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 5: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 6: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 8: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 10: (main()+0x4222) [0x561370cf7462] stderr: 11: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 12: (_start()+0x2a) [0x561370dc095a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: 2019-02-14 13:45:54.840 7f3fcecb3240 -1 /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7f3fcecb3240 time 2019-02-14 13:45:54.841130 stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: FAILED assert((uint64_t)r == len) stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3fc60d33e2] stderr: 2: (()+0x26d5a7) [0x7f3fc60d35a7] stderr: 3: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 5: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 6: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 8: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 10: (main()+0x4222) [0x561370cf7462] stderr: 11: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 12: (_start()+0x2a) [0x561370dc095a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: -15> 2019-02-14 13:45:54.788 7f3fcecb3240 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid stderr: 0> 2019-02-14 13:45:54.840 7f3fcecb3240 -1 /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7f3fcecb3240 time 2019-02-14 13:45:54.841130 stderr: /build/ceph-13.2.4/src/os/bluestore/KernelDevice.cc: 821: FAILED assert((uint64_t)r == len) stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f3fc60d33e2] stderr: 2: (()+0x26d5a7) [0x7f3fc60d35a7] stderr: 3: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 4: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 5: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 6: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 7: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 8: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 9: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 10: (main()+0x4222) [0x561370cf7462] stderr: 11: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 12: (_start()+0x2a) [0x561370dc095a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: *** Caught signal (Aborted) ** stderr: in thread 7f3fcecb3240 thread_name:ceph-osd stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) stderr: 1: (()+0x92aa40) [0x561371357a40] stderr: 2: (()+0x12890) [0x7f3fc47d7890] stderr: 3: (gsignal()+0xc7) [0x7f3fc36b2e97] stderr: 4: (abort()+0x141) [0x7f3fc36b4801] stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f3fc60d3530] stderr: 6: (()+0x26d5a7) [0x7f3fc60d35a7] stderr: 7: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 8: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 14: (main()+0x4222) [0x561370cf7462] stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 16: (_start()+0x2a) [0x561370dc095a] stderr: 2019-02-14 13:45:54.840 7f3fcecb3240 -1 *** Caught signal (Aborted) ** stderr: in thread 7f3fcecb3240 thread_name:ceph-osd stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) stderr: 1: (()+0x92aa40) [0x561371357a40] stderr: 2: (()+0x12890) [0x7f3fc47d7890] stderr: 3: (gsignal()+0xc7) [0x7f3fc36b2e97] stderr: 4: (abort()+0x141) [0x7f3fc36b4801] stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f3fc60d3530] stderr: 6: (()+0x26d5a7) [0x7f3fc60d35a7] stderr: 7: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 8: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 14: (main()+0x4222) [0x561370cf7462] stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 16: (_start()+0x2a) [0x561370dc095a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. stderr: 0> 2019-02-14 13:45:54.840 7f3fcecb3240 -1 *** Caught signal (Aborted) ** stderr: in thread 7f3fcecb3240 thread_name:ceph-osd stderr: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) stderr: 1: (()+0x92aa40) [0x561371357a40] stderr: 2: (()+0x12890) [0x7f3fc47d7890] stderr: 3: (gsignal()+0xc7) [0x7f3fc36b2e97] stderr: 4: (abort()+0x141) [0x7f3fc36b4801] stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x250) [0x7f3fc60d3530] stderr: 6: (()+0x26d5a7) [0x7f3fc60d35a7] stderr: 7: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*, IOContext*, bool)+0x4a7) [0x561371346817] stderr: 8: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x435) [0x5613713065c5] stderr: 9: (BlueFS::_replay(bool, bool)+0x214) [0x56137130c434] stderr: 10: (BlueFS::mount()+0x1f1) [0x561371310c81] stderr: 11: (BlueStore::_open_db(bool, bool)+0x17cd) [0x56137123704d] stderr: 12: (BlueStore::mkfs()+0x805) [0x561371267fe5] stderr: 13: (OSD::mkfs(CephContext*, ObjectStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, uuid_d, int)+0x1b0) [0x561370e10480] stderr: 14: (main()+0x4222) [0x561370cf7462] stderr: 15: (__libc_start_main()+0xe7) [0x7f3fc3695b97] stderr: 16: (_start()+0x2a) [0x561370dc095a] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 14d041d6-0beb-4056-8df2-3920e2febce0 --setuser ceph --setgroup ceph -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
