No, it's exactly what I told you it was. "bluestore_bdev_partition_path" is the data path. In all of my scenarios my DB and Data are on the same partition, hence mine are the same. Your DB and WAL are on a different partition from your Data... so your DB partition is different... Whatever your misunderstanding is about where/why your cluster's usage is higher/different than you think it is, it has nothing to do with where your DB and WAL partitions are.
There is a overhead just for having a FS on the disk. In this case that FS is bluestore. You can look at [1] this ML thread from a while ago where I mentioned a brand new cluster with no data in it and the WAL partitions on separate disks that it was using about 1.1GB of data per OSD. [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-March/025246.html On Mon, Oct 22, 2018 at 4:51 PM Robert Stanford <rstanford8...@gmail.com> wrote: > > That's very helpful, thanks. In your first case above your > bluefs_db_partition_path and bluestore_bdev_partition path are the same. > Though I have a different data and db drive, mine are different. Might > this explain something? My root concern is that there is more utilization > on the cluster than what's in the pools, the excess equal to about wal size > * number of osds... > > On Mon, Oct 22, 2018 at 3:35 PM David Turner <drakonst...@gmail.com> > wrote: > >> My DB doesn't have a specific partition anywhere, but there's still a >> symlink for it to the data partition. On my home cluster with all DB, WAL, >> and Data on the same disk without any partitions specified there is a block >> symlink but no block.wal symlink. >> >> For the cluster with a specific WAL partition, but no DB partition, my >> OSD paths looks like [1] this. For my cluster with everything on the same >> disk, my OSD paths look like [2] this. Unless you have a specific path for >> "bluefs_wal_partition_path" then it's going to find itself on the same >> partition as the db. >> >> [1] $ ceph osd metadata 5 | grep path >> "bluefs_db_partition_path": "/dev/dm-29", >> "bluefs_wal_partition_path": "/dev/dm-41", >> "bluestore_bdev_partition_path": "/dev/dm-29", >> >> [2] $ ceph osd metadata 5 | grep path >> "bluefs_db_partition_path": "/dev/dm-5", >> "bluestore_bdev_partition_path": "/dev/dm-5", >> >> On Mon, Oct 22, 2018 at 4:21 PM Robert Stanford <rstanford8...@gmail.com> >> wrote: >> >>> >>> Let me add, I have no block.wal file (which the docs suggest should be >>> there). >>> http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ >>> >>> On Mon, Oct 22, 2018 at 3:13 PM Robert Stanford <rstanford8...@gmail.com> >>> wrote: >>> >>>> >>>> We're out of sync, I think. You have your DB on your data disk so >>>> your block.db symlink points to that disk, right? There is however no wal >>>> symlink? So how would you verify your WAL actually lived on your NVMe? >>>> >>>> On Mon, Oct 22, 2018 at 3:07 PM David Turner <drakonst...@gmail.com> >>>> wrote: >>>> >>>>> And by the data disk I mean that I didn't specify a location for the >>>>> DB partition. >>>>> >>>>> On Mon, Oct 22, 2018 at 4:06 PM David Turner <drakonst...@gmail.com> >>>>> wrote: >>>>> >>>>>> Track down where it says they point to? Does it match what you >>>>>> expect? It does for me. I have my DB on my data disk and my WAL on a >>>>>> separate NVMe. >>>>>> >>>>>> On Mon, Oct 22, 2018 at 3:21 PM Robert Stanford < >>>>>> rstanford8...@gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> David - is it ensured that wal and db both live where the symlink >>>>>>> block.db points? I assumed that was a symlink for the db, but >>>>>>> necessarily >>>>>>> for the wal, because it can live in a place different than the db. >>>>>>> >>>>>>> On Mon, Oct 22, 2018 at 2:18 PM David Turner <drakonst...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> You can always just go to /var/lib/ceph/osd/ceph-{osd-num}/ and >>>>>>>> look at where the symlinks for block and block.wal point to. >>>>>>>> >>>>>>>> On Mon, Oct 22, 2018 at 12:29 PM Robert Stanford < >>>>>>>> rstanford8...@gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> That's what they say, however I did exactly this and my cluster >>>>>>>>> utilization is higher than the total pool utilization by about the >>>>>>>>> number >>>>>>>>> of OSDs * wal size. I want to verify that the wal is on the SSDs too >>>>>>>>> but >>>>>>>>> I've asked here and no one seems to know a way to verify this. Do >>>>>>>>> you? >>>>>>>>> >>>>>>>>> Thank you, R >>>>>>>>> >>>>>>>>> On Mon, Oct 22, 2018 at 5:22 AM Maged Mokhtar < >>>>>>>>> mmokh...@petasan.org> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> If you specify a db on ssd and data on hdd and not explicitly >>>>>>>>>> specify a >>>>>>>>>> device for wal, wal will be placed on same ssd partition with db. >>>>>>>>>> Placing only wal on ssd or creating separate devices for wal and >>>>>>>>>> db are >>>>>>>>>> less common setups. >>>>>>>>>> >>>>>>>>>> /Maged >>>>>>>>>> >>>>>>>>>> On 22/10/18 09:03, Fyodor Ustinov wrote: >>>>>>>>>> > Hi! >>>>>>>>>> > >>>>>>>>>> > For sharing SSD between WAL and DB what should be placed on >>>>>>>>>> SSD? WAL or DB? >>>>>>>>>> > >>>>>>>>>> > ----- Original Message ----- >>>>>>>>>> > From: "Maged Mokhtar" <mmokh...@petasan.org> >>>>>>>>>> > To: "ceph-users" <ceph-users@lists.ceph.com> >>>>>>>>>> > Sent: Saturday, 20 October, 2018 20:05:44 >>>>>>>>>> > Subject: Re: [ceph-users] Drive for Wal and Db >>>>>>>>>> > >>>>>>>>>> > On 20/10/18 18:57, Robert Stanford wrote: >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Our OSDs are BlueStore and are on regular hard drives. Each OSD >>>>>>>>>> has a partition on an SSD for its DB. Wal is on the regular hard >>>>>>>>>> drives. >>>>>>>>>> Should I move the wal to share the SSD with the DB? >>>>>>>>>> > >>>>>>>>>> > Regards >>>>>>>>>> > R >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > _______________________________________________ >>>>>>>>>> > ceph-users mailing list [ mailto:ceph-users@lists.ceph.com | >>>>>>>>>> ceph-users@lists.ceph.com ] [ >>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com | >>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ] >>>>>>>>>> > >>>>>>>>>> > you should put wal on the faster device, wal and db could share >>>>>>>>>> the same ssd partition, >>>>>>>>>> > >>>>>>>>>> > Maged >>>>>>>>>> > >>>>>>>>>> > _______________________________________________ >>>>>>>>>> > ceph-users mailing list >>>>>>>>>> > ceph-users@lists.ceph.com >>>>>>>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>>>>> > _______________________________________________ >>>>>>>>>> > ceph-users mailing list >>>>>>>>>> > ceph-users@lists.ceph.com >>>>>>>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list >>>>>>>>>> ceph-users@lists.ceph.com >>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list >>>>>>>>> ceph-users@lists.ceph.com >>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>>>>> >>>>>>>>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
