Thanks all,
Here is a link to our our command being executed: https://pastebin.com/iy8iSaKH Here are the results from the command Executed with debug enabled (after a zap with destroy) [root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data /dev/sdu Running command: ceph-authtool --gen-print-key Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 664894a8-530a-4557-b2f4-1af5b391f2b7 --> Was unable to complete a new OSD, will rollback changes --> OSD will be fully purged from the cluster, because the ID was generated Running command: ceph osd purge osd.140 --yes-i-really-mean-it stderr: purged osd.140 Traceback (most recent call last): File "/sbin/ceph-volume", line 6, in <module> main.Volume() File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, in __init__ self.main(self.argv) File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch instance.main() File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 38, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch instance.main() File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", line 74, in main self.create(args) File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 217, in safe_prepare self.prepare(args) File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 283, in prepare block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid) File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 193, in prepare_device if api.get_vg(vg_name=vg_name): File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 334, in get_vg return vgs.get(vg_name=vg_name, vg_tags=vg_tags) File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 429, in get raise MultipleVGsError(vg_name) ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc Rhian Resnick Associate Director Middleware and HPC Office of Information Technology Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> ________________________________ From: Alfredo Deza <ad...@redhat.com> Sent: Saturday, April 14, 2018 8:45 AM To: Rhian Resnick Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Error Creating OSD On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick <rresn...@fau.edu<mailto:rresn...@fau.edu>> wrote: Evening, When attempting to create an OSD we receive the following error. [ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore --data /dev/sdu Running command: ceph-authtool --gen-print-key Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c8cb8cff-dad9-48b8-8d77-6f130a4b629d --> Was unable to complete a new OSD, will rollback changes --> OSD will be fully purged from the cluster, because the ID was generated Running command: ceph osd purge osd.140 --yes-i-really-mean-it stderr: purged osd.140 --> MultipleVGsError: Got more than 1 result looking for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc Any hints on what to do? This occurs when we attempt to create osd's on this node. Can you use a paste site and get the /var/log/ceph/ceph-volume.log contents? Also, if you could try the same command but with: CEPH_VOLUME_DEBUG=1 I think you are hitting two issues here: 1) Somehow `osd new` is not completing and failing 2) The `purge` command to wipe out the LV is getting multiple LV's and cannot make sure to match the one it used. #2 definitely looks like something we are doing wrong, and #1 can have a lot of different causes. The logs would be tremendously helpful! Rhian Resnick Associate Director Middleware and HPC Office of Information Technology Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222 [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
