Makes sense, however I have seen online that it states you can safely use against the raw disk aka /dev/sdd and it will create two new partitions in the free space and not touch any existing partitions to allow you to use one device for multiple OSD's.
Number Start End Size File system Name Flags 1 1049kB 1075MB 1074MB ceph block.db 2 1075MB 1679MB 604MB ceph block.wal ,Ashley ________________________________ From: Martin Emrich <martin.emr...@empolis.com> Sent: 07 July 2017 08:12:24 To: Ashley Merrick; Vasu Kulkarni Cc: ceph-users@lists.ceph.com Subject: AW: [ceph-users] How to set up bluestore manually? You seem to use a whole disk, but I used a partition, which cannot (sanely) be partitioned further. But could you tell me with which sizes for sdd1 and sdd2 you ended up? Thanks, Martin Von: Ashley Merrick [mailto:ash...@amerrick.co.uk] Gesendet: Freitag, 7. Juli 2017 09:08 An: Martin Emrich <martin.emr...@empolis.com>; Vasu Kulkarni <vakul...@redhat.com> Cc: ceph-users@lists.ceph.com Betreff: Re: [ceph-users] How to set up bluestore manually? I can run the following command with no issue however and have done for multiple OSD's which work fine, it just creates an sdd1 and sdd2 ceph-disk prepare --bluestore /dev/sdg --block.wal /dev/sdd --block.db /dev/sdd ,Ashley ________________________________ From: ceph-users <ceph-users-boun...@lists.ceph.com<mailto:ceph-users-boun...@lists.ceph.com>> on behalf of Martin Emrich <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> Sent: 07 July 2017 08:03:44 To: Vasu Kulkarni Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] How to set up bluestore manually? Hi! It looks like I found the problem: The example suggested I can share the same block device for both WAL and DB, but apparently this is not the case. Analyzing the log output of ceph-deploy and the OSD hosts logs, I am making progress in discovering the exact steps to set up a bluestore OSD without convenience tools like ceph-deploy and ceph-disk. The error below occurs during “ceph-osd –mkfs” if both block.db and block.wal point to the same block device. If I use different block devices, creating the bluestore fs works fine. Nevertheless I created a ticket: http://tracker.ceph.com/issues/20540 One more question: What are the size requirements for the WAL and the DB? Cheers, Martin Von: Vasu Kulkarni [mailto:vakul...@redhat.com] Gesendet: Donnerstag, 6. Juli 2017 20:45 An: Martin Emrich <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> Cc: Loris Cuoghi <loris.cuo...@artificiale.net<mailto:loris.cuo...@artificiale.net>>; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Betreff: Re: [ceph-users] How to set up bluestore manually? I recommend you file a tracker issue at http://tracker.ceph.com/ with all details( ceph version, steps you ran and output hiding out anything you dont want to put), I doubt its a ceph-deploy issue but we can try in our lab to replicate it. On Thu, Jul 6, 2017 at 5:25 AM, Martin Emrich <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> wrote: Hi! I changed the partitioning scheme to use a "real" primary partition instead of a logical volume. Ceph-deploy seems run fine now, but the OSD does not start. I see lots of these in the journal: Jul 06 13:53:42 sh[9768]: 0> 2017-07-06 13:53:42.794027 7fcf9918fb80 -1 *** Caught signal (Aborted) ** Jul 06 13:53:42 sh[9768]: in thread 7fcf9918fb80 thread_name:ceph-osd Jul 06 13:53:42 sh[9768]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev) Jul 06 13:53:42 sh[9768]: 1: (()+0x9cd6af) [0x7fcf99b776af] Jul 06 13:53:42 sh[9768]: 2: (()+0xf370) [0x7fcf967d9370] Jul 06 13:53:42 sh[9768]: 3: (gsignal()+0x37) [0x7fcf958031d7] Jul 06 13:53:42 sh[9768]: 4: (abort()+0x148) [0x7fcf958048c8] Jul 06 13:53:42 sh[9768]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x7fcf99bb5394] Jul 06 13:53:42 sh[9768]: 6: (BitMapAreaIN::reserve_blocks(long)+0xb6) [0x7fcf99b6c486] Jul 06 13:53:42 sh[9768]: 7: (BitMapAllocator::reserve(unsigned long)+0x80) [0x7fcf99b6a240] Jul 06 13:53:42 sh[9768]: 8: (BlueFS::_allocate(unsigned char, unsigned long, std::vector<bluefs_extent_t, mempool::pool_allocator<(mempool::pool_index_t)9, bluefs_extent_t> >*)+0xee) [0x7fcf99b31c\ 0e] Jul 06 13:53:42 sh[9768]: 9: (BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, unsigned long, unsigned long)+0xbc4) [0x7fcf99b38be4] Jul 06 13:53:42 sh[9768]: 10: (BlueFS::sync_metadata()+0x215) [0x7fcf99b3d725] Jul 06 13:53:42 sh[9768]: 11: (BlueFS::umount()+0x74) [0x7fcf99b3dc44] Jul 06 13:53:42 sh[9768]: 12: (BlueStore::_open_db(bool)+0x579) [0x7fcf99a62859] Jul 06 13:53:42 sh[9768]: 13: (BlueStore::fsck(bool)+0x39b) [0x7fcf99a9581b] Jul 06 13:53:42 sh[9768]: 14: (BlueStore::mkfs()+0x1168) [0x7fcf99a6d118] Jul 06 13:53:42 sh[9768]: 15: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x29b) [0x7fcf9964b75b] Jul 06 13:53:42 sh[9768]: 16: (main()+0xf83) [0x7fcf99590573] Jul 06 13:53:42 sh[9768]: 17: (__libc_start_main()+0xf5) [0x7fcf957efb35] Jul 06 13:53:42 sh[9768]: 18: (()+0x4826e6) [0x7fcf9962c6e6] Jul 06 13:53:42 sh[9768]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 06 13:53:42 sh[9768]: Traceback (most recent call last): Jul 06 13:53:42 sh[9768]: File "/usr/sbin/ceph-disk", line 9, in <module> Jul 06 13:53:42 sh[9768]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')() Also interesting is the message "-1 rocksdb: Invalid argument: db: does not exist (create_if_missing is false)"... Looks to me as if ceph-deploy did not create the RocksDB? So still no success with bluestore :( Thanks, Martin -----Ursprüngliche Nachricht----- Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com<mailto:ceph-users-boun...@lists.ceph.com>] Im Auftrag von Martin Emrich Gesendet: Dienstag, 4. Juli 2017 22:02 An: Loris Cuoghi <loris.cuo...@artificiale.net<mailto:loris.cuo...@artificiale.net>>; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Betreff: Re: [ceph-users] How to set up bluestore manually? Hi! After getting some other stuff done, I finally got around to continuing here. I set up a whole new cluster with ceph-deploy, but adding the first OSD fails: ceph-deploy osd create --bluestore ${HOST}:/dev/sdc --block-wal /dev/cl/ceph-waldb-sdc --block-db /dev/cl/ceph-waldb-sdc . . . [WARNIN] get_partition_dev: Try 9/10 : partition 1 for /dev/cl/ceph-waldb-sdc does not exist in /sys/block/dm-2 [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid path is /sys/dev/block/253:2/dm/uuid [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid is LVM-2r0bGcoyMB0VnWeGGS77eOD5IOu8wAPN3wPX4OWSS1XGkYZYoziXhfAFMjJf4FJR [WARNIN] [WARNIN] get_partition_dev: Try 10/10 : partition 1 for /dev/cl/ceph-waldb-sdc does not exist in /sys/block/dm-2 [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid path is /sys/dev/block/253:2/dm/uuid [WARNIN] get_dm_uuid: get_dm_uuid /dev/cl/ceph-waldb-sdc uuid is LVM-2r0bGcoyMB0VnWeGGS77eOD5IOu8wAPN3wPX4OWSS1XGkYZYoziXhfAFMjJf4FJR [WARNIN] [WARNIN] Traceback (most recent call last): [WARNIN] File "/usr/sbin/ceph-disk", line 9, in <module> [WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')() [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5687, in run [WARNIN] main(sys.argv[1:]) [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5638, in main [WARNIN] args.func(args) [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2004, in main [WARNIN] Prepare.factory(args).prepare() [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1993, in prepare [WARNIN] self._prepare() [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2074, in _prepare [WARNIN] self.data.prepare(*to_prepare_list) [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2807, in prepare [WARNIN] self.prepare_device(*to_prepare_list) [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2983, in prepare_device [WARNIN] to_prepare.prepare() [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2216, in prepare [WARNIN] self.prepare_device() [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2310, in prepare_device [WARNIN] partition = device.get_partition(num) [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1714, in get_partition [WARNIN] dev = get_partition_dev(self.path, num) [WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 717, in get_partition_dev [WARNIN] (pnum, dev, error_msg)) [WARNIN] ceph_disk.main.Error: Error: partition 1 for /dev/cl/ceph-waldb-sdc does not appear to exist in /sys/block/dm-2 [ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-disk -v prepare --block.wal /dev/cl/ceph-waldb-sdc --block.db /dev/cl/ceph-waldb-sdc --bluestore --cluster ceph --fs-type xfs -- /dev/sdc [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs I found some open issues about this: http://tracker.ceph.com/issues/6042, and http://tracker.ceph.com/issues/5461.. Could this be related? Cheers, Martin -----Ursprüngliche Nachricht----- Von: Loris Cuoghi [mailto:loris.cuo...@artificiale.net<mailto:loris.cuo...@artificiale.net>] Gesendet: Montag, 3. Juli 2017 15:48 An: Martin Emrich <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>; Vasu Kulkarni <vakul...@redhat.com<mailto:vakul...@redhat.com>> Betreff: Re: [ceph-users] How to set up bluestore manually? Le Mon, 3 Jul 2017 12:32:20 +0000, Martin Emrich <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> a écrit : > Hi! > > Thanks for the super-fast response! > > That did work somehow... Here's my commandline (As Bluestore seems to > still require a Journal, No, it doesn't. :D > I repurposed the SSD partitions for it and put the DB/WAL on the > spinning disk): On the contrary, Bluestore's DB/WAL are good candidates for low-latency storage like an SSD. > > ceph-deploy osd create --bluestore > <hostname>:/dev/sdc:/dev/mapper/cl-ceph_journal_sdc Just ceph-deploy osd create --bluestore ${hostname}:/device/path should be sufficent to create a device composed of: * 1 small (~100 MB) XFS partition * 1 big (remaining space) partition formatted as bluestore Additional options like: --block-wal /path/to/ssd/partition --block-db /path/to/ssd/partition allow having SSD-backed WAL and DB. > But it created two (!) new OSDs instead of one, and placed them under > the default CRUSH rule (thus making my cluster doing stuff; they > should be under a different rule)... Default stuff is applied... by default :P Take a good read: http://docs.ceph.com/docs/master/rados/operations/crush-map/ In particular, on how editing an existing CRUSH map: http://docs.ceph.com/docs/master/rados/operations/crush-map/#editing-a-crush-map -- Loris > Did I do something wrong or are > the two OSDs part of the bluestore concept? If yes, how to handle them > in the CRUSH map (I have different categories of OSD hosts for > different use cases, split by appropriate CRUSH rules). > > Thanks > > Martin > > -----Ursprüngliche Nachricht----- > Von: Loris Cuoghi > [mailto:loris.cuo...@artificiale.net<mailto:loris.cuo...@artificiale.net>] > Gesendet: Montag, 3. Juli 2017 13:39 > An: Martin Emrich > <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> > Cc: Vasu Kulkarni <vakul...@redhat.com<mailto:vakul...@redhat.com>>; > ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> > Betreff: Re: [ceph-users] How to set up bluestore manually? > > Le Mon, 3 Jul 2017 11:30:04 +0000, > Martin Emrich <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> a > écrit : > > > Hi! > > > > Thanks for the hint, but I get this error: > > > > [ceph_deploy][ERROR ] ConfigError: Cannot load config: [Errno 2] No > > such file or directory: 'ceph.conf'; has `ceph-deploy new` been run > > in this directory? > > > > Obviously, ceph-deploy only works if the cluster has been managed > > with ceph-deply all along (and I won’t risk messing with my cluster > > by attempting to “retrofit” ceph-deply to it)… I’ll setup a single > > VM cluster to squeeze out the necessary commands and check back… > > No need to start with ceph-deploy in order to use it :) > > You can: > > create a working directory (e.g. ~/ceph-deploy) > - cd ~/ceph-deploy > - copy the ceph.conf from /etc/ceph/ceph.conf in the working directory > - execute ceph-deploy gatherkeys to obtain the necessary keys > > et voilà ;) > > Give it a try! > > -- Loris > > > > > Regards, > > > > Martin > > > > Von: Vasu Kulkarni [mailto:vakul...@redhat.com<mailto:vakul...@redhat.com>] > > Gesendet: Freitag, 30. Juni 2017 17:58 > > An: Martin Emrich > > <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>> > > Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> > > Betreff: Re: [ceph-users] How to set up bluestore manually? > > > > > > > > On Fri, Jun 30, 2017 at 8:31 AM, Martin Emrich > > <martin.emr...@empolis.com<mailto:martin.emr...@empolis.com><mailto:martin.emr...@empolis.com<mailto:martin.emr...@empolis.com>>> > > wrote: > > Hi! > > > > I’d like to set up new OSDs with bluestore: the real data (“block”) > > on a spinning disk, and DB+WAL on a SSD partition. > > > > But I do not use ceph-deploy, and never used ceph-disk (I set up the > > filestore OSDs manually). Google tells me that ceph-disk does not > > (yet) support splitting the components across multiple block > > devices, I also had no luck while attempting it anyways. (using Ceph > > 12.1 RC1) > > > > why not give ceph-deploy a chance, It has options to specify for db > > and wal in command line. you dont have to worry about what options > > to pass to ceph-disk as it encapsulates that > > > > ex: ceph-deploy osd create --bluestore --block-wal /dev/nvme0n1 > > --block-db /dev/nvme0n1 p30:sdb > > > > > > > > I just can’t find documentation on how to set up a bluestore OSD > > manually: > > > > > > * How do I “prepare” the block, block.wal and block.db > > blockdevices? Just directing ceph-osd to the block devices via > > ceph.conf does not seem to be enough. > > * Do bluestore OSDs still use/need a separate journal > > file/device? Or ist that replaced by the WAL? > > > > http://docs.ceph.com/docs/master/ also has not very much information > > on using bluestore, is this documentation deprecated? > > > > The document has fallen behind for manual deployment and ceph-deploy > > usage for bluestore as well, but if you use the above command, it > > will clearly throw out what it is doing and how ceph-disk is being > > called, those are essentially manual steps that should go in > > document. I think we have a tracker for update which is pending for > > sometime. > > > > > > Thanks for any hints, > > > > Martin > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com><mailto:ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
