Hello everyone,
I'm trying to add an OSD node to my current cluster. I created an lvm volume
for this node to use for OSD.
My current Ceph version is 14.2.6 and it runs on an RHEL 7 OS.
However, I got error when trying to activate the node. I'm confused with the
output. I tried to see what really happened, but I don't know how to proceed to
pinpoint the issues.
I appreciate so much for any help to clear the confusion.
* Why did the "mon getmap" command send the result to /dev/stderr? I saw
the monmap file having value and it looked like it got the monmap, is there any
way to check if the monmap having problem?
* What caused "_read_fsid unparsable uuid"?
Sincerely,
Hai
[x...@xxx.com@xxx430 ~]$ sudo ceph-volume --cluster aap-storage lvm create
--data /dev/kubernetes/ceph-osd
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring -i - osd new
c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/aap-storage-5
Running command: /sbin/restorecon /var/lib/ceph/osd/aap-storage-5
Running command: /bin/chown -h ceph:ceph /dev/kubernetes/ceph-osd
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/ln -s /dev/kubernetes/ceph-osd
/var/lib/ceph/osd/aap-storage-5/block
Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring mon getmap -o
/var/lib/ceph/osd/aap-storage-5/activate.monmap
stderr: got monmap epoch 8
Running command: /bin/ceph-authtool /var/lib/ceph/osd/aap-storage-5/keyring
--create-keyring --name osd.5 --add-key AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA==
stdout: creating /var/lib/ceph/osd/aap-storage-5/keyring
added entity osd.5 auth(key=AQB2RAhgnkcLIBAAdcdR5N4YKzSJmfoA6G6XvA==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5/
Running command: /bin/ceph-osd --cluster aap-storage --osd-objectstore
bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/aap-storage-5/activate.monmap
--keyfile - --osd-data /var/lib/ceph/osd/aap-storage-5/ --osd-uuid
c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90 --setuser ceph --setgroup ceph
stderr: 2021-01-20 15:55:52.319 7f0970d3ba80 -1
bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid
--> ceph-volume lvm prepare successful for: kubernetes/ceph-osd
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5
Running command: /bin/ceph-bluestore-tool --cluster=aap-storage prime-osd-dir
--dev /dev/kubernetes/ceph-osd --path /var/lib/ceph/osd/aap-storage-5
--no-mon-config
Running command: /bin/ln -snf /dev/kubernetes/ceph-osd
/var/lib/ceph/osd/aap-storage-5/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/aap-storage-5/block
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/aap-storage-5
Running command: /bin/systemctl enable
ceph-volume@lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90
stderr: Created symlink from
/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-5-c84f74b3-9a3e-4cd9-a4ce-4c1819a56f90.service
to /usr/lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@5
stderr: Created symlink from
/run/systemd/system/ceph-osd.target.wants/ceph-osd@5.service to
/usr/lib/systemd/system/ceph-osd@.service.
Running command: /bin/systemctl start ceph-osd@5
stderr: Job for ceph-osd@5.service failed because the control process exited
with error code. See "systemctl status ceph-osd@5.service" and "journalctl -xe"
for details.
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster aap-storage --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/aap-storage.keyring osd purge-new osd.5
--yes-i-really-mean-it
stderr: purged osd.5
--> RuntimeError: command returned non-zero exit status: 1
So I break the procedure into steps, and proceed them one by one. I saw the
problem happen with the bluestore.
[x...@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k
/etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs \
> --monmap /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/activate.monmap --osd-uuid
> $UUID --no-mon-config \
> --osd-data /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/ --setuser ceph --setgroup
> ceph
2021-01-21 16:59:14.907 7f724e882a80 -1
bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to
open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied
2021-01-21 16:59:14.908 7f724e882a80 -1
bluestore(/var/lib/ceph/osd/aap-storage-5//block) _read_bdev_label failed to
open /var/lib/ceph/osd/aap-storage-5//block: (13) Permission denied
2021-01-21 16:59:14.908 7f724e882a80 -1
bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid
2021-01-21 16:59:14.908 7f724e882a80 -1
bluestore(/var/lib/ceph/osd/aap-storage-5/) _setup_block_symlink_or_file failed
to open block file: (13) Permission denied
2021-01-21 16:59:14.908 7f724e882a80 -1
bluestore(/var/lib/ceph/osd/aap-storage-5/) mkfs failed, (13) Permission denied
2021-01-21 16:59:14.908 7f724e882a80 -1 OSD::mkfs: ObjectStore::mkfs failed
with error (13) Permission denied
2021-01-21 16:59:14.908 7f724e882a80 -1 ** ERROR: error creating empty object
store in /var/lib/ceph/osd/aap-storage-5/: (13) Permission denied
Because of the permission problem, so I avoid using the ceph user for the ceph
service, however, the problem still persisted.
[x...@xxx.com@xxx430 ~]$ sudo ceph-osd -c /etc/ceph/${CLUSTER_NAME}.conf -k
/etc/ceph/${CLUSTER_NAME}.client.admin.keyring -i $ID --mkfs --monmap
/var/lib/ceph/osd/${CLUSTER_NAME}-$ID/activate.monmap --osd-uuid $UUID
--no-mon-config --osd-data /var/lib/ceph/osd/${CLUSTER_NAME}-$ID/
2021-01-21 17:00:08.583 7f3acc546a80 -1
bluestore(/var/lib/ceph/osd/aap-storage-5/) _read_fsid unparsable uuid
--
KPMG IT Service GmbH
Sitz/Registergericht: Berlin/Amtsgericht Charlottenburg, HRB 87521 B
Geschäftsführer: Hans-Christian Schwieger, Helmar Symmank
Aufsichtsratsvorsitzender: WP StB Klaus Becker
Allgemeine Informationen zur Datenverarbeitung im Rahmen unserer allgemeinen
Geschäftstätigkeit sowie im Mandatsverhältnis gemäß EU
Datenschutz-Grundverordnung sind hier
<https://home.kpmg.com/content/dam/kpmg/de/pdf/Themen/2018/datenschutzinformationen-fuer-betroffene-kpmg-it-service-gmbh.pdf>
abrufbar.
Die Information in dieser E-Mail ist vertraulich und kann dem Berufsgeheimnis
unterliegen. Sie ist ausschließlich für den Adressaten bestimmt. Jeglicher
Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist
untersagt. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist
Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das
Ergreifen oder Unterlassen von Maßnahmen im Vertrauen auf erlangte Information
untersagt. In dieser E-Mail enthaltene Meinungen oder Empfehlungen unterliegen
den Bedingungen des jeweiligen Mandatsverhältnisses mit dem Adressaten.
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorized. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance on
it, is prohibited and may be unlawful. Any opinions or advice contained in this
e-mail are subject to the terms and conditions expressed in the governing KPMG
client engagement letter.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
