I recommend you file a tracker issue at http://tracker.ceph.com/ with all
details( ceph version, steps you ran and output hiding out anything you
dont want to put), I doubt its a ceph-deploy issue
but we can try in our lab to replicate it.
On Thu, Jul 6, 2017 at 5:25 AM, Martin Emrich <martin.emr...@empolis.com>
wrote:
> Hi!
>
> I changed the partitioning scheme to use a "real" primary partition
> instead of a logical volume. Ceph-deploy seems run fine now, but the OSD
> does not start.
>
> I see lots of these in the journal:
>
> Jul 06 13:53:42 sh[9768]: 0> 2017-07-06 13:53:42.794027 7fcf9918fb80 -1
> *** Caught signal (Aborted) **
> Jul 06 13:53:42 sh[9768]: in thread 7fcf9918fb80 thread_name:ceph-osd
> Jul 06 13:53:42 sh[9768]: ceph version 12.1.0 (
> 262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
> Jul 06 13:53:42 sh[9768]: 1: (()+0x9cd6af) [0x7fcf99b776af]
> Jul 06 13:53:42 sh[9768]: 2: (()+0xf370) [0x7fcf967d9370]
> Jul 06 13:53:42 sh[9768]: 3: (gsignal()+0x37) [0x7fcf958031d7]
> Jul 06 13:53:42 sh[9768]: 4: (abort()+0x148) [0x7fcf958048c8]
> Jul 06 13:53:42 sh[9768]: 5: (ceph::__ceph_assert_fail(char const*, char
> const*, int, char const*)+0x284) [0x7fcf99bb5394]
> Jul 06 13:53:42 sh[9768]: 6: (BitMapAreaIN::reserve_blocks(long)+0xb6)
> [0x7fcf99b6c486]
> Jul 06 13:53:42 sh[9768]: 7: (BitMapAllocator::reserve(unsigned
> long)+0x80) [0x7fcf99b6a240]
> Jul 06 13:53:42 sh[9768]: 8: (BlueFS::_allocate(unsigned char, unsigned
> long, std::vector<bluefs_extent_t,
> mempool::pool_allocator<(mempool::pool_index_t)9,
> bluefs_extent_t> >*)+0xee) [0x7fcf99b31c\
> 0e]
> Jul 06 13:53:42 sh[9768]: 9:
> (BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&,
> unsigned long, unsigned long)+0xbc4) [0x7fcf99b38be4]
> Jul 06 13:53:42 sh[9768]: 10: (BlueFS::sync_metadata()+0x215)
> [0x7fcf99b3d725]
> Jul 06 13:53:42 sh[9768]: 11: (BlueFS::umount()+0x74) [0x7fcf99b3dc44]
> Jul 06 13:53:42 sh[9768]: 12: (BlueStore::_open_db(bool)+0x579)
> [0x7fcf99a62859]
> Jul 06 13:53:42 sh[9768]: 13: (BlueStore::fsck(bool)+0x39b)
> [0x7fcf99a9581b]
> Jul 06 13:53:42 sh[9768]: 14: (BlueStore::mkfs()+0x1168) [0x7fcf99a6d118]
> Jul 06 13:53:42 sh[9768]: 15: (OSD::mkfs(CephContext*, ObjectStore*,
> std::string const&, uuid_d, int)+0x29b) [0x7fcf9964b75b]
> Jul 06 13:53:42 sh[9768]: 16: (main()+0xf83) [0x7fcf99590573]
> Jul 06 13:53:42 sh[9768]: 17: (__libc_start_main()+0xf5) [0x7fcf957efb35]
> Jul 06 13:53:42 sh[9768]: 18: (()+0x4826e6) [0x7fcf9962c6e6]
> Jul 06 13:53:42 sh[9768]: NOTE: a copy of the executable, or `objdump
> -rdS <executable>` is needed to interpret this.
> Jul 06 13:53:42 sh[9768]: Traceback (most recent call last):
> Jul 06 13:53:42 sh[9768]: File "/usr/sbin/ceph-disk", line 9, in <module>
> Jul 06 13:53:42 sh[9768]: load_entry_point('ceph-disk==1.0.0',
> 'console_scripts', 'ceph-disk')()
>
>
> Also interesting is the message "-1 rocksdb: Invalid argument: db: does
> not exist (create_if_missing is false)"... Looks to me as if ceph-deploy
> did not create the RocksDB?
>
> So still no success with bluestore :(
>
> Thanks,
>
> Martin
>
> -----Ursprüngliche Nachricht-----
> Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von
> Martin Emrich
> Gesendet: Dienstag, 4. Juli 2017 22:02
> An: Loris Cuoghi <loris.cuo...@artificiale.net>; ceph-users@lists.ceph.com
> Betreff: Re: [ceph-users] How to set up bluestore manually?
>
> Hi!
>
> After getting some other stuff done, I finally got around to continuing
> here.
>
> I set up a whole new cluster with ceph-deploy, but adding the first OSD
> fails:
>
> ceph-deploy osd create --bluestore ${HOST}:/dev/sdc --block-wal
> /dev/cl/ceph-waldb-sdc --block-db /dev/cl/ceph-waldb-sdc .
> .
> .
> [WARNIN] get_partition_dev: Try 9/10 : partition 1 for
> /dev/cl/ceph-waldb-sdc does not exist in /sys/block/dm-2 [WARNIN]
> get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid path is
> /sys/dev/block/253:2/dm/uuid [WARNIN] get_dm_uuid: get_dm_uuid
> /dev/cl/ceph-waldb-sdc uuid is LVM-2r0bGcoyMB0VnWeGGS77eOD5IOu8wA
> PN3wPX4OWSS1XGkYZYoziXhfAFMjJf4FJR
> [WARNIN]
> [WARNIN] get_partition_dev: Try 10/10 : partition 1 for
> /dev/cl/ceph-waldb-sdc does not exist in /sys/block/dm-2 [WARNIN]
> get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid path is
> /sys/dev/block/253:2/dm/uuid [WARNIN] get_dm_uuid: get_dm_uuid
> /dev/cl/ceph-waldb-sdc uuid is LVM-2r0bGcoyMB0VnWeGGS77eOD5IOu8wA
> PN3wPX4OWSS1XGkYZYoziXhfAFMjJf4FJR
> [WARNIN]
> [WARNIN] Traceback (most recent call last):
> [WARNIN] File "/usr/sbin/ceph-disk", line 9, in <module>
> [WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts',
> 'ceph-disk')()
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 5687, in run
> [WARNIN] main(sys.argv[1:])
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 5638, in main
> [WARNIN] args.func(args)
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 2004, in main
> [WARNIN] Prepare.factory(args).prepare()
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 1993, in prepare
> [WARNIN] self._prepare()
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 2074, in _prepare
> [WARNIN] self.data.prepare(*to_prepare_list)
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 2807, in prepare
> [WARNIN] self.prepare_device(*to_prepare_list)
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 2983, in prepare_device
> [WARNIN] to_prepare.prepare()
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 2216, in prepare
> [WARNIN] self.prepare_device()
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 2310, in prepare_device
> [WARNIN] partition = device.get_partition(num)
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 1714, in get_partition
> [WARNIN] dev = get_partition_dev(self.path, num)
> [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py",
> line 717, in get_partition_dev
> [WARNIN] (pnum, dev, error_msg))
> [WARNIN] ceph_disk.main.Error: Error: partition 1 for
> /dev/cl/ceph-waldb-sdc does not appear to exist in /sys/block/dm-2 [ERROR
> ] RuntimeError: command returned non-zero exit status: 1
> [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-disk -v
> prepare --block.wal /dev/cl/ceph-waldb-sdc --block.db
> /dev/cl/ceph-waldb-sdc --bluestore --cluster ceph --fs-type xfs -- /dev/sdc
> [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs
>
> I found some open issues about this: http://tracker.ceph.com/issues/6042,
> and http://tracker.ceph.com/issues/5461.. Could this be related?
>
> Cheers,
>
> Martin
>
>
> -----Ursprüngliche Nachricht-----
> Von: Loris Cuoghi [mailto:loris.cuo...@artificiale.net]
> Gesendet: Montag, 3. Juli 2017 15:48
> An: Martin Emrich <martin.emr...@empolis.com>
> Cc: ceph-users@lists.ceph.com; Vasu Kulkarni <vakul...@redhat.com>
> Betreff: Re: [ceph-users] How to set up bluestore manually?
>
> Le Mon, 3 Jul 2017 12:32:20 +0000,
> Martin Emrich <martin.emr...@empolis.com> a écrit :
>
> > Hi!
> >
> > Thanks for the super-fast response!
> >
> > That did work somehow... Here's my commandline (As Bluestore seems to
> > still require a Journal,
>
> No, it doesn't. :D
>
> > I repurposed the SSD partitions for it and put the DB/WAL on the
> > spinning disk):
>
> On the contrary, Bluestore's DB/WAL are good candidates for low-latency
> storage like an SSD.
>
> >
> > ceph-deploy osd create --bluestore
> > <hostname>:/dev/sdc:/dev/mapper/cl-ceph_journal_sdc
>
> Just
> ceph-deploy osd create --bluestore ${hostname}:/device/path
>
> should be sufficent to create a device composed of:
>
> * 1 small (~100 MB) XFS partition
> * 1 big (remaining space) partition formatted as bluestore
>
> Additional options like:
>
> --block-wal /path/to/ssd/partition
> --block-db /path/to/ssd/partition
>
> allow having SSD-backed WAL and DB.
>
> > But it created two (!) new OSDs instead of one, and placed them under
> > the default CRUSH rule (thus making my cluster doing stuff; they
> > should be under a different rule)...
>
> Default stuff is applied... by default :P
>
> Take a good read:
>
> http://docs.ceph.com/docs/master/rados/operations/crush-map/
>
> In particular, on how editing an existing CRUSH map:
>
> http://docs.ceph.com/docs/master/rados/operations/crush-
> map/#editing-a-crush-map
>
> -- Loris
>
> > Did I do something wrong or are
> > the two OSDs part of the bluestore concept? If yes, how to handle them
> > in the CRUSH map (I have different categories of OSD hosts for
> > different use cases, split by appropriate CRUSH rules).
> >
> > Thanks
> >
> > Martin
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Loris Cuoghi [mailto:loris.cuo...@artificiale.net]
> > Gesendet: Montag, 3. Juli 2017 13:39
> > An: Martin Emrich <martin.emr...@empolis.com>
> > Cc: Vasu Kulkarni <vakul...@redhat.com>; ceph-users@lists.ceph.com
> > Betreff: Re: [ceph-users] How to set up bluestore manually?
> >
> > Le Mon, 3 Jul 2017 11:30:04 +0000,
> > Martin Emrich <martin.emr...@empolis.com> a écrit :
> >
> > > Hi!
> > >
> > > Thanks for the hint, but I get this error:
> > >
> > > [ceph_deploy][ERROR ] ConfigError: Cannot load config: [Errno 2] No
> > > such file or directory: 'ceph.conf'; has `ceph-deploy new` been run
> > > in this directory?
> > >
> > > Obviously, ceph-deploy only works if the cluster has been managed
> > > with ceph-deply all along (and I won’t risk messing with my cluster
> > > by attempting to “retrofit” ceph-deply to it)… I’ll setup a single
> > > VM cluster to squeeze out the necessary commands and check back…
> >
> > No need to start with ceph-deploy in order to use it :)
> >
> > You can:
> >
> > create a working directory (e.g. ~/ceph-deploy)
> > - cd ~/ceph-deploy
> > - copy the ceph.conf from /etc/ceph/ceph.conf in the working directory
> > - execute ceph-deploy gatherkeys to obtain the necessary keys
> >
> > et voilà ;)
> >
> > Give it a try!
> >
> > -- Loris
> >
> > >
> > > Regards,
> > >
> > > Martin
> > >
> > > Von: Vasu Kulkarni [mailto:vakul...@redhat.com]
> > > Gesendet: Freitag, 30. Juni 2017 17:58
> > > An: Martin Emrich <martin.emr...@empolis.com>
> > > Cc: ceph-users@lists.ceph.com
> > > Betreff: Re: [ceph-users] How to set up bluestore manually?
> > >
> > >
> > >
> > > On Fri, Jun 30, 2017 at 8:31 AM, Martin Emrich
> > > <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> wrote:
> > > Hi!
> > >
> > > I’d like to set up new OSDs with bluestore: the real data (“block”)
> > > on a spinning disk, and DB+WAL on a SSD partition.
> > >
> > > But I do not use ceph-deploy, and never used ceph-disk (I set up the
> > > filestore OSDs manually). Google tells me that ceph-disk does not
> > > (yet) support splitting the components across multiple block
> > > devices, I also had no luck while attempting it anyways. (using Ceph
> > > 12.1 RC1)
> > >
> > > why not give ceph-deploy a chance, It has options to specify for db
> > > and wal in command line. you dont have to worry about what options
> > > to pass to ceph-disk as it encapsulates that
> > >
> > > ex: ceph-deploy osd create --bluestore --block-wal /dev/nvme0n1
> > > --block-db /dev/nvme0n1 p30:sdb
> > >
> > >
> > >
> > > I just can’t find documentation on how to set up a bluestore OSD
> > > manually:
> > >
> > >
> > > * How do I “prepare” the block, block.wal and block.db
> > > blockdevices? Just directing ceph-osd to the block devices via
> > > ceph.conf does not seem to be enough.
> > > * Do bluestore OSDs still use/need a separate journal
> > > file/device? Or ist that replaced by the WAL?
> > >
> > > http://docs.ceph.com/docs/master/ also has not very much information
> > > on using bluestore, is this documentation deprecated?
> > >
> > > The document has fallen behind for manual deployment and ceph-deploy
> > > usage for bluestore as well, but if you use the above command, it
> > > will clearly throw out what it is doing and how ceph-disk is being
> > > called, those are essentially manual steps that should go in
> > > document. I think we have a tracker for update which is pending for
> > > sometime.
> > >
> > >
> > > Thanks for any hints,
> > >
> > > Martin
> > >
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> >
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
