Re: [ceph-users] "no valid command found" when running "ceph-deploy osd create"
There should be useful logs from ceph-volume in /var/log/ceph/ceph-volume.log that might show a bit more here. I would also try the command that fails directly on the server (sans ceph-deploy) to see what is it that is actually failing. Seems like the ceph-deploy log output is a bit out of order (some race condition here maybe) On Sun, Sep 2, 2018 at 2:53 AM, David Wahler wrote: > Hi all, > > I'm attempting to get a small Mimic cluster running on ARM, starting > with a single node. Since there don't seem to be any Debian ARM64 > packages in the official Ceph repository, I had to build from source, > which was fairly straightforward. > > After installing the .deb packages that I built and following the > quick-start guide > (http://docs.ceph.com/docs/mimic/start/quick-ceph-deploy/), things > seemed to be working fine at first, but I got this error when > attempting to create an OSD: > > rock64@rockpro64-1:~/my-cluster$ ceph-deploy osd create --data > /dev/storage/bluestore rockpro64-1 > [ceph_deploy.conf][DEBUG ] found configuration file at: > /home/rock64/.cephdeploy.conf > [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy osd > create --data /dev/storage/bluestore rockpro64-1 > [ceph_deploy.cli][INFO ] ceph-deploy options: > [ceph_deploy.cli][INFO ] verbose : False > [ceph_deploy.cli][INFO ] bluestore : None > [ceph_deploy.cli][INFO ] cd_conf : > > [ceph_deploy.cli][INFO ] cluster : ceph > [ceph_deploy.cli][INFO ] fs_type : xfs > [ceph_deploy.cli][INFO ] block_wal : None > [ceph_deploy.cli][INFO ] default_release : False > [ceph_deploy.cli][INFO ] username : None > [ceph_deploy.cli][INFO ] journal : None > [ceph_deploy.cli][INFO ] subcommand: create > [ceph_deploy.cli][INFO ] host : rockpro64-1 > [ceph_deploy.cli][INFO ] filestore : None > [ceph_deploy.cli][INFO ] func : osd at 0x7fa9ca0c80> > [ceph_deploy.cli][INFO ] ceph_conf : None > [ceph_deploy.cli][INFO ] zap_disk : False > [ceph_deploy.cli][INFO ] data : > /dev/storage/bluestore > [ceph_deploy.cli][INFO ] block_db : None > [ceph_deploy.cli][INFO ] dmcrypt : False > [ceph_deploy.cli][INFO ] overwrite_conf: False > [ceph_deploy.cli][INFO ] dmcrypt_key_dir : > /etc/ceph/dmcrypt-keys > [ceph_deploy.cli][INFO ] quiet : False > [ceph_deploy.cli][INFO ] debug : False > [ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data > device /dev/storage/bluestore > [rockpro64-1][DEBUG ] connection detected need for sudo > [rockpro64-1][DEBUG ] connected to host: rockpro64-1 > [rockpro64-1][DEBUG ] detect platform information from remote host > [rockpro64-1][DEBUG ] detect machine type > [rockpro64-1][DEBUG ] find the location of an executable > [ceph_deploy.osd][INFO ] Distro info: debian buster/sid sid > [ceph_deploy.osd][DEBUG ] Deploying osd to rockpro64-1 > [rockpro64-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf > [rockpro64-1][WARNIN] osd keyring does not exist yet, creating one > [rockpro64-1][DEBUG ] create a keyring file > [rockpro64-1][DEBUG ] find the location of an executable > [rockpro64-1][INFO ] Running command: sudo /usr/sbin/ceph-volume > --cluster ceph lvm create --bluestore --data /dev/storage/bluestore > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph-authtool --gen-print-key > [rockpro64-1][WARNIN] --> RuntimeError: command returned non-zero > exit status: 22 > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph --cluster ceph > --name client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > 4903fff3-550c-4ce3-aa7d-97193627c6c0 > [rockpro64-1][DEBUG ] --> Was unable to complete a new OSD, will > rollback changes > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph --cluster ceph > --name client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 > --yes-i-really-mean-it > [rockpro64-1][DEBUG ] stderr: no valid command found; 10 closest matches: > [rockpro64-1][DEBUG ] osd tier add-cache > [rockpro64-1][DEBUG ] osd tier remove-overlay > [rockpro64-1][DEBUG ] osd out [...] > [rockpro64-1][DEBUG ] osd in [...] > [rockpro64-1][DEBUG ] osd down [...] > [rockpro64-1][DEBUG ] stderr: osd unset > full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|nodeep-scrub|notieragent|nosnaptrim > [rockpro64-1][DEBUG ] osd require-osd-release luminous|mimic > {--yes-i-really-mean-it} > [rockpro64-1][DEBUG ] osd erasure-code-profile ls > [rockpro64-1][DEBUG ] osd set > full|pause|noup|nodown|noout|noin|nobackf
Re: [ceph-users] "no valid command found" when running "ceph-deploy osd create"
Ah, ceph-volume.log pointed out the actual problem: RuntimeError: Cannot use device (/dev/storage/bluestore). A vg/lv path or an existing device is needed When I changed "--data /dev/storage/bluestore" to "--data storage/bluestore", everything worked fine. I agree that the ceph-deploy logs are a bit confusing. I submitted a PR to add a brief note to the quick-start guide, in case anyone else makes the same mistake: https://github.com/ceph/ceph/pull/23879 Thanks for the assistance! -- David On Sun, Sep 2, 2018 at 7:44 AM Alfredo Deza wrote: > > There should be useful logs from ceph-volume in > /var/log/ceph/ceph-volume.log that might show a bit more here. > > I would also try the command that fails directly on the server (sans > ceph-deploy) to see what is it that is actually failing. Seems like > the ceph-deploy log output is a bit out of order (some race condition > here maybe) > > > On Sun, Sep 2, 2018 at 2:53 AM, David Wahler wrote: > > Hi all, > > > > I'm attempting to get a small Mimic cluster running on ARM, starting > > with a single node. Since there don't seem to be any Debian ARM64 > > packages in the official Ceph repository, I had to build from source, > > which was fairly straightforward. > > > > After installing the .deb packages that I built and following the > > quick-start guide > > (http://docs.ceph.com/docs/mimic/start/quick-ceph-deploy/), things > > seemed to be working fine at first, but I got this error when > > attempting to create an OSD: > > > > rock64@rockpro64-1:~/my-cluster$ ceph-deploy osd create --data > > /dev/storage/bluestore rockpro64-1 > > [ceph_deploy.conf][DEBUG ] found configuration file at: > > /home/rock64/.cephdeploy.conf > > [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy osd > > create --data /dev/storage/bluestore rockpro64-1 > > [ceph_deploy.cli][INFO ] ceph-deploy options: > > [ceph_deploy.cli][INFO ] verbose : False > > [ceph_deploy.cli][INFO ] bluestore : None > > [ceph_deploy.cli][INFO ] cd_conf : > > > > [ceph_deploy.cli][INFO ] cluster : ceph > > [ceph_deploy.cli][INFO ] fs_type : xfs > > [ceph_deploy.cli][INFO ] block_wal : None > > [ceph_deploy.cli][INFO ] default_release : False > > [ceph_deploy.cli][INFO ] username : None > > [ceph_deploy.cli][INFO ] journal : None > > [ceph_deploy.cli][INFO ] subcommand: create > > [ceph_deploy.cli][INFO ] host : rockpro64-1 > > [ceph_deploy.cli][INFO ] filestore : None > > [ceph_deploy.cli][INFO ] func : > osd at 0x7fa9ca0c80> > > [ceph_deploy.cli][INFO ] ceph_conf : None > > [ceph_deploy.cli][INFO ] zap_disk : False > > [ceph_deploy.cli][INFO ] data : > > /dev/storage/bluestore > > [ceph_deploy.cli][INFO ] block_db : None > > [ceph_deploy.cli][INFO ] dmcrypt : False > > [ceph_deploy.cli][INFO ] overwrite_conf: False > > [ceph_deploy.cli][INFO ] dmcrypt_key_dir : > > /etc/ceph/dmcrypt-keys > > [ceph_deploy.cli][INFO ] quiet : False > > [ceph_deploy.cli][INFO ] debug : False > > [ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data > > device /dev/storage/bluestore > > [rockpro64-1][DEBUG ] connection detected need for sudo > > [rockpro64-1][DEBUG ] connected to host: rockpro64-1 > > [rockpro64-1][DEBUG ] detect platform information from remote host > > [rockpro64-1][DEBUG ] detect machine type > > [rockpro64-1][DEBUG ] find the location of an executable > > [ceph_deploy.osd][INFO ] Distro info: debian buster/sid sid > > [ceph_deploy.osd][DEBUG ] Deploying osd to rockpro64-1 > > [rockpro64-1][DEBUG ] write cluster configuration to > > /etc/ceph/{cluster}.conf > > [rockpro64-1][WARNIN] osd keyring does not exist yet, creating one > > [rockpro64-1][DEBUG ] create a keyring file > > [rockpro64-1][DEBUG ] find the location of an executable > > [rockpro64-1][INFO ] Running command: sudo /usr/sbin/ceph-volume > > --cluster ceph lvm create --bluestore --data /dev/storage/bluestore > > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph-authtool > > --gen-print-key > > [rockpro64-1][WARNIN] --> RuntimeError: command returned non-zero > > exit status: 22 > > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph --cluster ceph > > --name client.bootstrap-osd --keyring > > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > > 4903fff3-550c-4ce3-aa7d-97193627c6c0 > > [rockpro64-1][DEBUG ] --> Was unable to complete a new OSD, will > > rollback changes > > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph --cluster ceph > > --name client.bootstrap-osd --keyring > > /var/lib/ceph/bootst
Re: [ceph-users] "no valid command found" when running "ceph-deploy osd create"
On Sun, Sep 2, 2018 at 12:00 PM, David Wahler wrote: > Ah, ceph-volume.log pointed out the actual problem: > > RuntimeError: Cannot use device (/dev/storage/bluestore). A vg/lv path > or an existing device is needed That is odd, is it possible that the error log wasn't the one that matched what you saw on ceph-deploy's end? Usually ceph-deploy will just receive whatever ceph-volume produced. > > When I changed "--data /dev/storage/bluestore" to "--data > storage/bluestore", everything worked fine. > > I agree that the ceph-deploy logs are a bit confusing. I submitted a > PR to add a brief note to the quick-start guide, in case anyone else > makes the same mistake: https://github.com/ceph/ceph/pull/23879 > Thanks for the PR! > Thanks for the assistance! > > -- David > > On Sun, Sep 2, 2018 at 7:44 AM Alfredo Deza wrote: >> >> There should be useful logs from ceph-volume in >> /var/log/ceph/ceph-volume.log that might show a bit more here. >> >> I would also try the command that fails directly on the server (sans >> ceph-deploy) to see what is it that is actually failing. Seems like >> the ceph-deploy log output is a bit out of order (some race condition >> here maybe) >> >> >> On Sun, Sep 2, 2018 at 2:53 AM, David Wahler wrote: >> > Hi all, >> > >> > I'm attempting to get a small Mimic cluster running on ARM, starting >> > with a single node. Since there don't seem to be any Debian ARM64 >> > packages in the official Ceph repository, I had to build from source, >> > which was fairly straightforward. >> > >> > After installing the .deb packages that I built and following the >> > quick-start guide >> > (http://docs.ceph.com/docs/mimic/start/quick-ceph-deploy/), things >> > seemed to be working fine at first, but I got this error when >> > attempting to create an OSD: >> > >> > rock64@rockpro64-1:~/my-cluster$ ceph-deploy osd create --data >> > /dev/storage/bluestore rockpro64-1 >> > [ceph_deploy.conf][DEBUG ] found configuration file at: >> > /home/rock64/.cephdeploy.conf >> > [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy osd >> > create --data /dev/storage/bluestore rockpro64-1 >> > [ceph_deploy.cli][INFO ] ceph-deploy options: >> > [ceph_deploy.cli][INFO ] verbose : False >> > [ceph_deploy.cli][INFO ] bluestore : None >> > [ceph_deploy.cli][INFO ] cd_conf : >> > >> > [ceph_deploy.cli][INFO ] cluster : ceph >> > [ceph_deploy.cli][INFO ] fs_type : xfs >> > [ceph_deploy.cli][INFO ] block_wal : None >> > [ceph_deploy.cli][INFO ] default_release : False >> > [ceph_deploy.cli][INFO ] username : None >> > [ceph_deploy.cli][INFO ] journal : None >> > [ceph_deploy.cli][INFO ] subcommand: create >> > [ceph_deploy.cli][INFO ] host : rockpro64-1 >> > [ceph_deploy.cli][INFO ] filestore : None >> > [ceph_deploy.cli][INFO ] func : > > osd at 0x7fa9ca0c80> >> > [ceph_deploy.cli][INFO ] ceph_conf : None >> > [ceph_deploy.cli][INFO ] zap_disk : False >> > [ceph_deploy.cli][INFO ] data : >> > /dev/storage/bluestore >> > [ceph_deploy.cli][INFO ] block_db : None >> > [ceph_deploy.cli][INFO ] dmcrypt : False >> > [ceph_deploy.cli][INFO ] overwrite_conf: False >> > [ceph_deploy.cli][INFO ] dmcrypt_key_dir : >> > /etc/ceph/dmcrypt-keys >> > [ceph_deploy.cli][INFO ] quiet : False >> > [ceph_deploy.cli][INFO ] debug : False >> > [ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data >> > device /dev/storage/bluestore >> > [rockpro64-1][DEBUG ] connection detected need for sudo >> > [rockpro64-1][DEBUG ] connected to host: rockpro64-1 >> > [rockpro64-1][DEBUG ] detect platform information from remote host >> > [rockpro64-1][DEBUG ] detect machine type >> > [rockpro64-1][DEBUG ] find the location of an executable >> > [ceph_deploy.osd][INFO ] Distro info: debian buster/sid sid >> > [ceph_deploy.osd][DEBUG ] Deploying osd to rockpro64-1 >> > [rockpro64-1][DEBUG ] write cluster configuration to >> > /etc/ceph/{cluster}.conf >> > [rockpro64-1][WARNIN] osd keyring does not exist yet, creating one >> > [rockpro64-1][DEBUG ] create a keyring file >> > [rockpro64-1][DEBUG ] find the location of an executable >> > [rockpro64-1][INFO ] Running command: sudo /usr/sbin/ceph-volume >> > --cluster ceph lvm create --bluestore --data /dev/storage/bluestore >> > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph-authtool >> > --gen-print-key >> > [rockpro64-1][WARNIN] --> RuntimeError: command returned non-zero >> > exit status: 22 >> > [rockpro64-1][DEBUG ] Running command: /usr/bin/ceph --cluster ceph >> > --n
Re: [ceph-users] "no valid command found" when running "ceph-deploy osd create"
On Sun, Sep 2, 2018 at 1:31 PM Alfredo Deza wrote: > > On Sun, Sep 2, 2018 at 12:00 PM, David Wahler wrote: > > Ah, ceph-volume.log pointed out the actual problem: > > > > RuntimeError: Cannot use device (/dev/storage/bluestore). A vg/lv path > > or an existing device is needed > > That is odd, is it possible that the error log wasn't the one that > matched what you saw on ceph-deploy's end? > > Usually ceph-deploy will just receive whatever ceph-volume produced. I tried again, running ceph-volume directly this time, just to see if I had mixed anything up. It looks like ceph-deploy is correctly reporting the output of ceph-volume. The problem is that ceph-volume only writes the relevant error message to the log file, and not to its stdout/stderr. Console output: rock64@rockpro64-1:~/my-cluster$ sudo ceph-volume --cluster ceph lvm create --bluestore --data /dev/storage/foobar Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e7dd6d45-b556-461c-bad1-83d98a5a1afa --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it stderr: no valid command found; 10 closest matches: [...etc...] ceph-volume.log: [2018-09-02 18:49:21,415][ceph_volume.main][INFO ] Running command: ceph-volume --cluster ceph lvm create --bluestore --data /dev/storage/foobar [2018-09-02 18:49:21,423][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key [2018-09-02 18:49:26,664][ceph_volume.process][INFO ] stdout AQCxMIxb+SezJRAAGAP/HHtHLVbciSQnZ/c/qw== [2018-09-02 18:49:26,668][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e7dd6d45-b556-461c-bad1-83d98a5a1afa [2018-09-02 18:49:27,685][ceph_volume.process][INFO ] stdout 1 [2018-09-02 18:49:27,686][ceph_volume.process][INFO ] Running command: /bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/storage/foobar [2018-09-02 18:49:27,707][ceph_volume.process][INFO ] stdout NAME="storage-foobar" KNAME="dm-1" MAJ:MIN="253:1" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="100G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw" ALIGNMENT="0" PHY-SEC="4096" LOG-SEC="512" ROTA="1" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" PKNAME="" PARTLABEL="" [2018-09-02 18:49:27,708][ceph_volume.process][INFO ] Running command: /bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/storage/foobar [2018-09-02 18:49:27,720][ceph_volume.process][INFO ] stdout NAME="storage-foobar" KNAME="dm-1" MAJ:MIN="253:1" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="100G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw" ALIGNMENT="0" PHY-SEC="4096" LOG-SEC="512" ROTA="1" SCHED="" TYPE="lvm" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" PKNAME="" PARTLABEL="" [2018-09-02 18:49:27,720][ceph_volume.devices.lvm.prepare][ERROR ] lvm prepare was unable to complete Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", line 216, in safe_prepare self.prepare(args) File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", line 283, in prepare block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid) File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", line 206, in prepare_device raise RuntimeError(' '.join(error)) RuntimeError: Cannot use device (/dev/storage/foobar). A vg/lv path or an existing device is needed [2018-09-02 18:49:27,722][ceph_volume.devices.lvm.prepare][INFO ] will rollback OSD ID creation [2018-09-02 18:49:27,723][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it [2018-09-02 18:49:28,425][ceph_volume.process][INFO ] stderr no valid command found; 10 closest matches: [...etc...] -- David > > > > When I changed "--data /dev/storage/bluestore" to "--data > > storage/bluestore", everything worked fine. > > > > I agree that the ceph-deploy logs are a bit confusing. I submitted a > > PR to add a brief note to the quick-start guide, in case
Re: [ceph-users] "no valid command found" when running "ceph-deploy osd create"
On Sun, Sep 2, 2018 at 3:01 PM, David Wahler wrote: > On Sun, Sep 2, 2018 at 1:31 PM Alfredo Deza wrote: >> >> On Sun, Sep 2, 2018 at 12:00 PM, David Wahler wrote: >> > Ah, ceph-volume.log pointed out the actual problem: >> > >> > RuntimeError: Cannot use device (/dev/storage/bluestore). A vg/lv path >> > or an existing device is needed >> >> That is odd, is it possible that the error log wasn't the one that >> matched what you saw on ceph-deploy's end? >> >> Usually ceph-deploy will just receive whatever ceph-volume produced. > > I tried again, running ceph-volume directly this time, just to see if > I had mixed anything up. It looks like ceph-deploy is correctly > reporting the output of ceph-volume. The problem is that ceph-volume > only writes the relevant error message to the log file, and not to its > stdout/stderr. > > Console output: > > rock64@rockpro64-1:~/my-cluster$ sudo ceph-volume --cluster ceph lvm > create --bluestore --data /dev/storage/foobar > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > e7dd6d45-b556-461c-bad1-83d98a5a1afa > --> Was unable to complete a new OSD, will rollback changes > Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 > --yes-i-really-mean-it > stderr: no valid command found; 10 closest matches: > [...etc...] > > ceph-volume.log: > > [2018-09-02 18:49:21,415][ceph_volume.main][INFO ] Running command: > ceph-volume --cluster ceph lvm create --bluestore --data > /dev/storage/foobar > [2018-09-02 18:49:21,423][ceph_volume.process][INFO ] Running > command: /usr/bin/ceph-authtool --gen-print-key > [2018-09-02 18:49:26,664][ceph_volume.process][INFO ] stdout > AQCxMIxb+SezJRAAGAP/HHtHLVbciSQnZ/c/qw== > [2018-09-02 18:49:26,668][ceph_volume.process][INFO ] Running > command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > e7dd6d45-b556-461c-bad1-83d98a5a1afa > [2018-09-02 18:49:27,685][ceph_volume.process][INFO ] stdout 1 > [2018-09-02 18:49:27,686][ceph_volume.process][INFO ] Running > command: /bin/lsblk --nodeps -P -o > NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL > /dev/storage/foobar > [2018-09-02 18:49:27,707][ceph_volume.process][INFO ] stdout > NAME="storage-foobar" KNAME="dm-1" MAJ:MIN="253:1" FSTYPE="" > MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="100G" > STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw" > ALIGNMENT="0" PHY-SEC="4096" LOG-SEC="512" ROTA="1" SCHED="" > TYPE="lvm" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" > PKNAME="" PARTLABEL="" > [2018-09-02 18:49:27,708][ceph_volume.process][INFO ] Running > command: /bin/lsblk --nodeps -P -o > NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL > /dev/storage/foobar > [2018-09-02 18:49:27,720][ceph_volume.process][INFO ] stdout > NAME="storage-foobar" KNAME="dm-1" MAJ:MIN="253:1" FSTYPE="" > MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="" SIZE="100G" > STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw" > ALIGNMENT="0" PHY-SEC="4096" LOG-SEC="512" ROTA="1" SCHED="" > TYPE="lvm" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" > PKNAME="" PARTLABEL="" > [2018-09-02 18:49:27,720][ceph_volume.devices.lvm.prepare][ERROR ] lvm > prepare was unable to complete > Traceback (most recent call last): > File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", > line 216, in safe_prepare > self.prepare(args) > File "/usr/lib/python2.7/dist-packages/ceph_volume/decorators.py", > line 16, in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", > line 283, in prepare > block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid) > File "/usr/lib/python2.7/dist-packages/ceph_volume/devices/lvm/prepare.py", > line 206, in prepare_device > raise RuntimeError(' '.join(error)) > RuntimeError: Cannot use device (/dev/storage/foobar). A vg/lv path or > an existing device is needed > [2018-09-02 18:49:27,722][ceph_volume.devices.lvm.prepare][INFO ] > will rollback OSD ID creation > [2018-09-02 18:49:27,723][ceph_volume.process][INFO ] Running > command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 > --yes-i-really-mean-it > [2018-09-02 18:49:28,425][ceph_volume.process][INFO ] stderr no valid > command found; 10 closest matches: > [...etc...] This is a bug. Thanks for di