On Sat, Apr 14, 2018 at 5:17 PM, Rhian Resnick <rresn...@fau.edu> wrote:
> Afternoon, > > > Happily, I resolved this issue. > > > Running vgdisplay showed that ceph-volume tried to create a disk on failed > disk. (We didn't know we had a bad did so this is information that was new > to us) and when the command failed it left three bad volume groups. Since > you cannot rename them you need to use the following command to delete > them. > We actually tried to guard against this problem, but we failed in this specific situation where multiple VGs exist. I've created http://tracker.ceph.com/issues/23757 to follow up with this, thanks for reporting it! > > vgdisplay to find the bad volume groups > > vgremove --select vg_uuid=your uuid -f # -f forces it to be removed > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road > <https://maps.google.com/?q=777+Glades+Road&entry=gmail&source=g>, CM22, > Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image: image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > > ------------------------------ > *From:* Rhian Resnick > *Sent:* Saturday, April 14, 2018 12:47 PM > *To:* Alfredo Deza > > *Cc:* ceph-users@lists.ceph.com > *Subject:* Re: [ceph-users] Error Creating OSD > > > Thanks all, > > > Here is a link to our our command being executed: https://pastebin. > com/iy8iSaKH > > > Here are the results from the command > > > Executed with debug enabled (after a zap with destroy) > > > [root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data > /dev/sdu > Running command: ceph-authtool --gen-print-key > Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > 664894a8-530a-4557-b2f4-1af5b391f2b7 > --> Was unable to complete a new OSD, will rollback changes > --> OSD will be fully purged from the cluster, because the ID was generated > Running command: ceph osd purge osd.140 --yes-i-really-mean-it > stderr: purged osd.140 > Traceback (most recent call last): > File "/sbin/ceph-volume", line 6, in <module> > main.Volume() > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, > in __init__ > self.main(self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line > 59, in newfunc > return f(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, > in main > terminal.dispatch(self.mapper, subcommand_args) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line > 182, in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", > line 38, in main > terminal.dispatch(self.mapper, self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line > 182, in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", > line 74, in main > self.create(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line > 16, in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", > line 26, in create > prepare_step.safe_prepare(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", > line 217, in safe_prepare > self.prepare(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line > 16, in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", > line 283, in prepare > block_lv = self.prepare_device(args.data, 'block', cluster_fsid, > osd_fsid) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", > line 193, in prepare_device > if api.get_vg(vg_name=vg_name): > File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line > 334, in get_vg > return vgs.get(vg_name=vg_name, vg_tags=vg_tags) > File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line > 429, in get > raise MultipleVGsError(vg_name) > ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking > for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc > > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road > <https://maps.google.com/?q=777+Glades+Road&entry=gmail&source=g>, CM22, > Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image: image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > > ------------------------------ > *From:* Alfredo Deza <ad...@redhat.com> > *Sent:* Saturday, April 14, 2018 8:45 AM > *To:* Rhian Resnick > *Cc:* ceph-users@lists.ceph.com > *Subject:* Re: [ceph-users] Error Creating OSD > > > > On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick <rresn...@fau.edu> wrote: > > Evening, > > When attempting to create an OSD we receive the following error. > > [ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore > --data /dev/sdu > Running command: ceph-authtool --gen-print-key > Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > c8cb8cff-dad9-48b8-8d77-6f130a4b629d > --> Was unable to complete a new OSD, will rollback changes > --> OSD will be fully purged from the cluster, because the ID was generated > Running command: ceph osd purge osd.140 --yes-i-really-mean-it > stderr: purged osd.140 > --> MultipleVGsError: Got more than 1 result looking for volume group: > ceph-6a2e8f21-bca2-492b-8869-eecc995216cc > > Any hints on what to do? This occurs when we attempt to create osd's on > this node. > > > Can you use a paste site and get the /var/log/ceph/ceph-volume.log > contents? Also, if you could try the same command but with: > > CEPH_VOLUME_DEBUG=1 > > I think you are hitting two issues here: > > 1) Somehow `osd new` is not completing and failing > 2) The `purge` command to wipe out the LV is getting multiple LV's and > cannot make sure to match the one it used. > > #2 definitely looks like something we are doing wrong, and #1 can have a > lot of different causes. The logs would be tremendously helpful! > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road > <https://maps.google.com/?q=777+Glades+Road&entry=gmail&source=g>, CM22, > Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image: image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
