You should pass underlying device instead of DM volume to ceph-volume. On Jun 12, 2018, 15:41 +0300, Alfredo Deza <ad...@redhat.com>, wrote: > On Tue, Jun 12, 2018 at 7:04 AM, Vadim Bulst <vadim.bu...@uni-leipzig.de> > wrote: > > I cannot release this lock! This is an expansion shelf connected with two > > cables to the controller. If there is no multipath management, the os would > > see every disk at least twice. Ceph has to deal with it somehow. I guess I'm > > not the only one who has a setup like this. > > > > Do you have an LV on top of that dm? We don't support multipath devices: > > http://docs.ceph.com/docs/master/ceph-volume/lvm/prepare/#multipath-support > > > Best, > > > > Vadim > > > > > > > > > > On 12.06.2018 12:55, Alfredo Deza wrote: > > > > > > On Tue, Jun 12, 2018 at 6:47 AM, Vadim Bulst <vadim.bu...@uni-leipzig.de> > > > wrote: > > > > > > > > Hi Alfredo, > > > > > > > > thanks for your help. Yust to make this clear /dev/dm-0 is the name of > > > > my > > > > multipath disk: > > > > > > > > root@polstor01:/home/urzadmin# ls -la /dev/disk/by-id/ | grep dm-0 > > > > lrwxrwxrwx 1 root root 10 Jun 12 07:50 dm-name-35000c500866f8947 -> > > > > ../../dm-0 > > > > lrwxrwxrwx 1 root root 10 Jun 12 07:50 dm-uuid-mpath-35000c500866f8947 > > > > -> > > > > ../../dm-0 > > > > lrwxrwxrwx 1 root root 10 Jun 12 07:50 scsi-35000c500866f8947 -> > > > > ../../dm-0 > > > > lrwxrwxrwx 1 root root 10 Jun 12 07:50 wwn-0x5000c500866f8947 -> > > > > ../../dm-0 > > > > > > > > If I run pvdisplay this device is not listed. > > > > > > Either way, you should not use dm devices directly. If this is a > > > multipath disk, then you must use that other name instead of /dev/dm-* > > > > > > I am not sure what kind of setup you have, but that mapper must > > > release its lock so that you can zap. We ensure that works with LVM, I > > > am not sure > > > how to do that in your environment. > > > > > > For example, with dmcrypt you get into similar issues, that is why we > > > check crypsetup, so that we can make dmcrypt release that device > > > before zapping. > > > > > > > > Cheers, > > > > > > > > Vadim > > > > > > > > > > > > > > > > On 12.06.2018 12:40, Alfredo Deza wrote: > > > > > > > > > > On Tue, Jun 12, 2018 at 4:37 AM, Vadim Bulst > > > > > <vadim.bu...@uni-leipzig.de> > > > > > wrote: > > > > > > > > > > > > no change: > > > > > > > > > > > > > > > > > > root@polstor01:/home/urzadmin# ceph-volume lvm zap --destroy > > > > > > /dev/dm-0 > > > > > > --> Zapping: /dev/dm-0 > > > > > > > > > > This is the problem right here. Your script is using the dm device > > > > > that belongs to an LV. > > > > > > > > > > What you want to do here is destroy/zap the LV. Not the dm device that > > > > > belongs to the LV. > > > > > > > > > > To make this clear in the future, I've created: > > > > > http://tracker.ceph.com/issues/24504 > > > > > > > > > > > > > > > > > > > > > Running command: /sbin/cryptsetup status /dev/mapper/ > > > > > > stdout: /dev/mapper/ is inactive. > > > > > > --> Skipping --destroy because no associated physical volumes are > > > > > > found > > > > > > for > > > > > > /dev/dm-0 > > > > > > Running command: wipefs --all /dev/dm-0 > > > > > > stderr: wipefs: error: /dev/dm-0: probing initialization failed: > > > > > > Device > > > > > > or > > > > > > resource busy > > > > > > --> RuntimeError: command returned non-zero exit status: 1 > > > > > > > > > > > > > > > > > > On 12.06.2018 09:03, Linh Vu wrote: > > > > > > > > > > > > ceph-volume lvm zap --destroy $DEVICE > > > > > > > > > > > > ________________________________ > > > > > > From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of > > > > > > Vadim > > > > > > Bulst <vadim.bu...@uni-leipzig.de> > > > > > > Sent: Tuesday, 12 June 2018 4:46:44 PM > > > > > > To: ceph-users@lists.ceph.com > > > > > > Subject: Re: [ceph-users] Filestore -> Bluestore > > > > > > > > > > > > > > > > > > Thanks Sergey. > > > > > > > > > > > > Could you specify your answer a bit more? When I look into the > > > > > > manpage > > > > > > of > > > > > > ceph-volume I couldn't find an option named "--destroy". > > > > > > > > > > > > I just like to make clear - this script has already migrated several > > > > > > servers. The problem is appearing when it should migrate devices in > > > > > > the > > > > > > expansion shelf. > > > > > > > > > > > > "--> RuntimeError: Cannot use device (/dev/dm-0). A vg/lv path or an > > > > > > existing device is needed" > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Vadim > > > > > > > > > > > > > > > > > > I would say the handling of devices > > > > > > On 11.06.2018 23:58, Sergey Malinin wrote: > > > > > > > > > > > > “Device or resource busy” error rises when no “--destroy” option is > > > > > > passed > > > > > > to ceph-volume. > > > > > > On Jun 11, 2018, 22:44 +0300, Vadim Bulst > > > > > > <vadim.bu...@uni-leipzig.de>, > > > > > > wrote: > > > > > > > > > > > > Dear Cephers, > > > > > > > > > > > > I'm trying to migrate our OSDs to Bluestore using this little > > > > > > script: > > > > > > > > > > > > #!/bin/bash > > > > > > HOSTNAME=$(hostname -s) > > > > > > OSDS=`ceph osd metadata | jq -c '[.[] | select(.osd_objectstore | > > > > > > contains("filestore")) ]' | jq '[.[] | select(.hostname | > > > > > > contains("'${HOSTNAME}'")) ]' | jq '.[].id'` > > > > > > IFS=' ' read -a OSDARRAY <<<$OSDS > > > > > > for OSD in "${OSDARRAY[@]}"; do > > > > > > DEV=/dev/`ceph osd metadata | jq -c '.[] | select(.id=='${OSD}') | > > > > > > .backend_filestore_dev_node' | sed 's/"//g'` > > > > > > echo "=== Migrating OSD nr ${OSD} on device ${DEV} ===" > > > > > > ceph osd out ${OSD} > > > > > > while ! ceph osd safe-to-destroy ${OSD} ; do echo "waiting for > > > > > > full > > > > > > evacuation"; sleep 60 ; done > > > > > > systemctl stop ceph-osd@${OSD} > > > > > > umount /var/lib/ceph/osd/ceph-${OSD} > > > > > > /usr/sbin/ceph-volume lvm zap ${DEV} > > > > > > ceph osd destroy ${OSD} --yes-i-really-mean-it > > > > > > /usr/sbin/ceph-volume lvm create --bluestore --data ${DEV} > > > > > > --osd-id ${OSD} > > > > > > done > > > > > > > > > > > > Unfortunately - under normal circumstances this works flawlessly. In > > > > > > our > > > > > > case we have expansion shelfs connected as multipath devices to our > > > > > > nodes. > > > > > > > > > > > > /usr/sbin/ceph-volume lvm zap ${DEV} is breaking with an error: > > > > > > > > > > > > OSD(s) 1 are safe to destroy without reducing data durability. > > > > > > --> Zapping: /dev/dm-0 > > > > > > Running command: /sbin/cryptsetup status /dev/mapper/ > > > > > > stdout: /dev/mapper/ is inactive. > > > > > > Running command: wipefs --all /dev/dm-0 > > > > > > stderr: wipefs: error: /dev/dm-0: probing initialization failed: > > > > > > Device or resource busy > > > > > > --> RuntimeError: command returned non-zero exit status: 1 > > > > > > destroyed osd.1 > > > > > > Running command: /usr/bin/ceph-authtool --gen-print-key > > > > > > Running command: /usr/bin/ceph --cluster ceph --name > > > > > > client.bootstrap-osd --keyring > > > > > > /var/lib/ceph/bootstrap-osd/ceph.keyring > > > > > > osd tree -f json > > > > > > Running command: /usr/bin/ceph --cluster ceph --name > > > > > > client.bootstrap-osd --keyring > > > > > > /var/lib/ceph/bootstrap-osd/ceph.keyring > > > > > > -i - osd new 74f6ff02-d027-4fc6-9b93-3a96d753 > > > > > > 5c8f 1 > > > > > > --> Was unable to complete a new OSD, will rollback changes > > > > > > --> OSD will be destroyed, keeping the ID because it was provided > > > > > > with > > > > > > --osd-id > > > > > > Running command: ceph osd destroy osd.1 --yes-i-really-mean-it > > > > > > stderr: destroyed osd.1 > > > > > > > > > > > > --> RuntimeError: Cannot use device (/dev/dm-0). A vg/lv path or an > > > > > > existing device is needed > > > > > > > > > > > > > > > > > > Does anybody know how to solve this problem? > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Vadim > > > > > > > > > > > > -- > > > > > > Vadim Bulst > > > > > > > > > > > > Universität Leipzig / URZ > > > > > > 04109 Leipzig, Augustusplatz 10 > > > > > > > > > > > > phone: +49-341-97-33380 > > > > > > mail: vadim.bu...@uni-leipzig.de > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list > > > > > > ceph-users@lists.ceph.com > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > -- > > > > > > Vadim Bulst > > > > > > > > > > > > Universität Leipzig / URZ > > > > > > 04109 Leipzig, Augustusplatz 10 > > > > > > > > > > > > phone: ++49-341-97-33380 > > > > > > mail: vadim.bu...@uni-leipzig.de > > > > > > > > > > > > > > > > > > -- > > > > > > Vadim Bulst > > > > > > > > > > > > Universität Leipzig / URZ > > > > > > 04109 Leipzig, Augustusplatz 10 > > > > > > > > > > > > phone: ++49-341-97-33380 > > > > > > mail: vadim.bu...@uni-leipzig.de > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list > > > > > > ceph-users@lists.ceph.com > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > -- > > > > Vadim Bulst > > > > > > > > Universität Leipzig / URZ > > > > 04109 Leipzig, Augustusplatz 10 > > > > > > > > phone: ++49-341-97-33380 > > > > mail: vadim.bu...@uni-leipzig.de > > > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- > > Vadim Bulst > > > > Universität Leipzig / URZ > > 04109 Leipzig, Augustusplatz 10 > > > > phone: ++49-341-97-33380 > > mail: vadim.bu...@uni-leipzig.de > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
