Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

2018-04-11 Thread Gary Verhulp 

On 4/7/18 4:21 PM, Alfredo Deza wrote:

On Sat, Apr 7, 2018 at 11:59 AM, Gary Verhulp  wrote:





I’m trying to create bluestore osds with separate --block.wal --block.db
devices on a write intensive SSD



I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db and
wal





I seems to me the osd uuid is getting changed and I’m only able to start the
last OSD



Do I need to create a new partition or logical volume on the SSD for each
OSD?

Correct! This is what is needed for each OSD. You are re-using the
same partitions for the other OSD which is why you are getting the
following message:



2018-04-06 19:45:43.730515 7fe91a9cfd00 -1 bluestore(/dev/sda1)
_check_or_set_bdev_label bdev /dev/sda1 fsid
eb6cbcb3-f644-4973-b745-0e4389ef719c does not match our fsid
9d7a103a-f590-4842-bd3d-e9da27c3fb09






I’m sure this is a simple fail in my understanding of how it is supposed to
be provisioned.

Any advice would be appreciated.



Thanks,

Gary





[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc
--block.wal /dev/sda2 --block.db /dev/sda1

Running command: sudo vgcreate --force --yes
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034 /dev/sdc

stdout: Physical volume "/dev/sdc" successfully created.

stdout: Volume group "ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034"
successfully created

Running command: sudo lvcreate --yes -l 100%FREE -n
osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034

stdout: Logical volume "osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09"
created.

Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1

Running command: chown -R ceph:ceph /dev/dm-2

Running command: sudo ln -s
/dev/ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034/osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
/var/lib/ceph/osd/ceph-1/block

Running command: sudo ceph --cluster ceph --name client.bootstrap-osd
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-1/activate.monmap

stderr: got monmap epoch 1

Running command: ceph-authtool /var/lib/ceph/osd/ceph-1/keyring
--create-keyring --name osd.1 --add-key
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==

stdout: creating /var/lib/ceph/osd/ceph-1/keyring

stdout: added entity osd.1 auth auth(auid = 18446744073709551615
key=AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g== with 0 caps)

Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring

Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/

Running command: chown -R ceph:ceph /dev/sda2

Running command: chown -R ceph:ceph /dev/sda1

Running command: sudo ceph-osd --cluster ceph --osd-objectstore bluestore
--mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --key
 --bluestore-block-wal-path
/dev/sda2 --bluestore-block-db-path /dev/sda1 --osd-data
/var/lib/ceph/osd/ceph-1/ --osd-uuid 9d7a103a-f590-4842-bd3d-e9da27c3fb09
--setuser ceph --setgroup ceph

stderr: 2018-04-06 19:41:44.519662 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding

stderr: 2018-04-06 19:41:44.520939 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding

stderr: 2018-04-06 19:41:44.521190 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
label at offset 102: buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
end of struct encoding

stderr: 2018-04-06 19:41:44.521454 7f734f2e4d00 -1
bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid

stderr: 2018-04-06 19:41:47.307648 7f734f2e4d00 -1 key
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==

stderr: 2018-04-06 19:41:48.068161 7f734f2e4d00 -1 created object store
/var/lib/ceph/osd/ceph-1/ for osd.1 fsid
1ff50434-64ad-42bd-9a70-1968e4a9a813





[root@osdhost osd]# ceph-bluestore-tool show-label --dev /dev/sda1

{

 "/dev/sda1": {

 "osd_uuid": "9d7a103a-f590-4842-bd3d-e9da27c3fb09",

 "size": 200043171840,

 "btime": "2018-04-06 19:41:44.523894",

 "description": "bluefs db"

 }

}



[root@osdhost  osd]# ceph-volume lvm prepare --bluestore --data /dev/sdd
--block.wal /dev/sda2 --block.db /dev/sda1

Running command: sudo vgcreate --force --yes
ceph-cc91203d-de5c-4d27-8c48-a58663075e67 /dev/sdd

stdout: Physical volume "/dev/sdd" successfully created.

stdout: Volume group "ceph-cc91203d-de5c-4d27-8c48-a58663075e67"
successfully created

Running command: sudo lvcreate --yes -l 100%FREE -n
osd-block-eb6cbcb3-f644-4973-b745-0e4389ef719c
ceph-cc91203d-de5c-4d27-8c48-a58663075e67

stdout: Logical volume

Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

2018-04-10 Thread Richard Hesketh 
No, you shouldn't invoke it that way, you should just not specify a WAL device 
at all if you want it to be stored with the DB - if not otherwise specified the 
WAL is automatically stored with the other metadata on the DB device. You 
should do something like:

ceph-volume lvm prepare --bluestore --data /dev/sdc --block.db /dev/sda1

Rich

On 09/04/18 09:19, Hervé Ballans wrote:
> Hi,
> 
> Just a little question regarding this operation :
> 
> [root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc 
> --block.wal /dev/sda2 --block.db /dev/sda1
> 
> On a previous post, I understood that if both wal and db are stored on the 
> same separate device, then we could use a single partition for both...which 
> means we could do :
> 
> # ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal /dev/sda1 
> --block.db /dev/sda1
> 
> and so on with other uniq wal/db partition for other OSD...
> 
> Did I get that correctly ?
> 
> Thanks,
> 
> Hervé
> 
> 
> Le 07/04/2018 à 17:59, Gary Verhulp a écrit :
>>
>> I’m trying to create bluestore osds with separate --block.wal --block.db 
>> devices on a write intensive SSD
>>
>> I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db and 
>> wal
>>
>>  
>>
>> I seems to me the osd uuid is getting changed and I’m only able to start the 
>> last OSD
>>
>> Do I need to create a new partition or logical volume on the SSD for each 
>> OSD?
>>
>> I’m sure this is a simple fail in my understanding of how it is supposed to 
>> be provisioned.
>>
>> Any advice would be appreciated.
>>
>> Thanks,
>>
>> Gary



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

2018-04-09 Thread Hervé Ballans 

Hi,

Just a little question regarding this operation :

[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc 
--block.wal /dev/sda2 --block.db /dev/sda1


On a previous post, I understood that if both wal and db are stored on 
the same separate device, then we could use a single partition for 
both...which means we could do :


# ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal 
/dev/sda1 --block.db /dev/sda1


and so on with other uniq wal/db partition for other OSD...

Did I get that correctly ?

Thanks,

Hervé


Le 07/04/2018 à 17:59, Gary Verhulp a écrit :


I’m trying to create bluestore osds with separate --block.wal 
--block.db devices on a write intensive SSD


I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for 
db and wal


I seems to me the osd uuid is getting changed and I’m only able to 
start the last OSD


Do I need to create a new partition or logical volume on the SSD for 
each OSD?


I’m sure this is a simple fail in my understanding of how it is 
supposed to be provisioned.


Any advice would be appreciated.

Thanks,

Gary

[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data 
/dev/sdc --block.wal /dev/sda2 --block.db /dev/sda1




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

2018-04-07 Thread Alfredo Deza 
On Sat, Apr 7, 2018 at 11:59 AM, Gary Verhulp  wrote:
>
>
>
>
>
> I’m trying to create bluestore osds with separate --block.wal --block.db
> devices on a write intensive SSD
>
>
>
> I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db and
> wal
>
>
>
>
>
> I seems to me the osd uuid is getting changed and I’m only able to start the
> last OSD
>
>
>
> Do I need to create a new partition or logical volume on the SSD for each
> OSD?

Correct! This is what is needed for each OSD. You are re-using the
same partitions for the other OSD which is why you are getting the
following message:



2018-04-06 19:45:43.730515 7fe91a9cfd00 -1 bluestore(/dev/sda1)
_check_or_set_bdev_label bdev /dev/sda1 fsid
eb6cbcb3-f644-4973-b745-0e4389ef719c does not match our fsid
9d7a103a-f590-4842-bd3d-e9da27c3fb09



>
>
>
> I’m sure this is a simple fail in my understanding of how it is supposed to
> be provisioned.
>
> Any advice would be appreciated.
>
>
>
> Thanks,
>
> Gary
>
>
>
>
>
> [root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc
> --block.wal /dev/sda2 --block.db /dev/sda1
>
> Running command: sudo vgcreate --force --yes
> ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034 /dev/sdc
>
> stdout: Physical volume "/dev/sdc" successfully created.
>
> stdout: Volume group "ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034"
> successfully created
>
> Running command: sudo lvcreate --yes -l 100%FREE -n
> osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
> ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034
>
> stdout: Logical volume "osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09"
> created.
>
> Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1
>
> Running command: chown -R ceph:ceph /dev/dm-2
>
> Running command: sudo ln -s
> /dev/ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034/osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09
> /var/lib/ceph/osd/ceph-1/block
>
> Running command: sudo ceph --cluster ceph --name client.bootstrap-osd
> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
> /var/lib/ceph/osd/ceph-1/activate.monmap
>
> stderr: got monmap epoch 1
>
> Running command: ceph-authtool /var/lib/ceph/osd/ceph-1/keyring
> --create-keyring --name osd.1 --add-key
> AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==
>
> stdout: creating /var/lib/ceph/osd/ceph-1/keyring
>
> stdout: added entity osd.1 auth auth(auid = 18446744073709551615
> key=AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g== with 0 caps)
>
> Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring
>
> Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/
>
> Running command: chown -R ceph:ceph /dev/sda2
>
> Running command: chown -R ceph:ceph /dev/sda1
>
> Running command: sudo ceph-osd --cluster ceph --osd-objectstore bluestore
> --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --key
>  --bluestore-block-wal-path
> /dev/sda2 --bluestore-block-db-path /dev/sda1 --osd-data
> /var/lib/ceph/osd/ceph-1/ --osd-uuid 9d7a103a-f590-4842-bd3d-e9da27c3fb09
> --setuser ceph --setgroup ceph
>
> stderr: 2018-04-06 19:41:44.519662 7f734f2e4d00 -1
> bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
> label at offset 102: buffer::malformed_input: void
> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
> end of struct encoding
>
> stderr: 2018-04-06 19:41:44.520939 7f734f2e4d00 -1
> bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
> label at offset 102: buffer::malformed_input: void
> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
> end of struct encoding
>
> stderr: 2018-04-06 19:41:44.521190 7f734f2e4d00 -1
> bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode
> label at offset 102: buffer::malformed_input: void
> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
> end of struct encoding
>
> stderr: 2018-04-06 19:41:44.521454 7f734f2e4d00 -1
> bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
>
> stderr: 2018-04-06 19:41:47.307648 7f734f2e4d00 -1 key
> AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==
>
> stderr: 2018-04-06 19:41:48.068161 7f734f2e4d00 -1 created object store
> /var/lib/ceph/osd/ceph-1/ for osd.1 fsid
> 1ff50434-64ad-42bd-9a70-1968e4a9a813
>
>
>
>
>
> [root@osdhost osd]# ceph-bluestore-tool show-label --dev /dev/sda1
>
> {
>
> "/dev/sda1": {
>
> "osd_uuid": "9d7a103a-f590-4842-bd3d-e9da27c3fb09",
>
> "size": 200043171840,
>
> "btime": "2018-04-06 19:41:44.523894",
>
> "description": "bluefs db"
>
> }
>
> }
>
>
>
> [root@osdhost  osd]# ceph-volume lvm prepare --bluestore --data /dev/sdd
> --block.wal /dev/sda2 --block.db /dev/sda1
>
> Running command: sudo vgcreate --force --yes
> ceph-cc91203d-de5c-4d27-8c48-a58663075e67 /dev/sdd
>
> stdout: Physical volume "/dev/sdd" successfully created.
>
> stdout: Volume group

[ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

2018-04-07 Thread Gary Verhulp 


I’m trying to create bluestore osds with separate --block.wal --block.db 
devices on a write intensive SSD


I’ve split the SSD (/dev/sda) into two partditions sda1 and sda2 for db 
and wal


I seems to me the osd uuid is getting changed and I’m only able to start 
the last OSD


Do I need to create a new partition or logical volume on the SSD for 
each OSD?


I’m sure this is a simple fail in my understanding of how it is supposed 
to be provisioned.


Any advice would be appreciated.

Thanks,

Gary

[root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc 
--block.wal /dev/sda2 --block.db /dev/sda1


Running command: sudo vgcreate --force --yes 
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034 /dev/sdc


stdout: Physical volume "/dev/sdc" successfully created.

stdout: Volume group "ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034" 
successfully created


Running command: sudo lvcreate --yes -l 100%FREE -n 
osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09 
ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034


stdout: Logical volume "osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09" 
created.


Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1

Running command: chown -R ceph:ceph /dev/dm-2

Running command: sudo ln -s 
/dev/ceph-5a6b8ab6-ca12-4855-9a5a-a3a54c249034/osd-block-9d7a103a-f590-4842-bd3d-e9da27c3fb09 
/var/lib/ceph/osd/ceph-1/block


Running command: sudo ceph --cluster ceph --name client.bootstrap-osd 
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o 
/var/lib/ceph/osd/ceph-1/activate.monmap


stderr: got monmap epoch 1

Running command: ceph-authtool /var/lib/ceph/osd/ceph-1/keyring 
--create-keyring --name osd.1 --add-key 
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==


stdout: creating /var/lib/ceph/osd/ceph-1/keyring

stdout: added entity osd.1 auth auth(auid = 18446744073709551615 
key=AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g== with 0 caps)


Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring

Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/

Running command: chown -R ceph:ceph /dev/sda2

Running command: chown -R ceph:ceph /dev/sda1

Running command: sudo ceph-osd --cluster ceph --osd-objectstore 
bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap 
--key  
--bluestore-block-wal-path /dev/sda2 --bluestore-block-db-path /dev/sda1 
--osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid 
9d7a103a-f590-4842-bd3d-e9da27c3fb09 --setuser ceph --setgroup ceph


stderr: 2018-04-06 19:41:44.519662 7f734f2e4d00 -1 
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to 
decode label at offset 102: buffer::malformed_input: void 
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode 
past end of struct encoding


stderr: 2018-04-06 19:41:44.520939 7f734f2e4d00 -1 
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to 
decode label at offset 102: buffer::malformed_input: void 
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode 
past end of struct encoding


stderr: 2018-04-06 19:41:44.521190 7f734f2e4d00 -1 
bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to 
decode label at offset 102: buffer::malformed_input: void 
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode 
past end of struct encoding


stderr: 2018-04-06 19:41:44.521454 7f734f2e4d00 -1 
bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid


stderr: 2018-04-06 19:41:47.307648 7f734f2e4d00 -1 key 
AQDjL8haKmzYOhAAM7ehRUUgF/n4x/Ybu7VR/g==


stderr: 2018-04-06 19:41:48.068161 7f734f2e4d00 -1 created object store 
/var/lib/ceph/osd/ceph-1/ for osd.1 fsid 
1ff50434-64ad-42bd-9a70-1968e4a9a813


[root@osdhost osd]# ceph-bluestore-tool show-label --dev /dev/sda1

{

"/dev/sda1": {

"osd_uuid": "9d7a103a-f590-4842-bd3d-e9da27c3fb09",

"size": 200043171840,

"btime": "2018-04-06 19:41:44.523894",

"description": "bluefs db"

    }

}

[root@osdhost  osd]# ceph-volume lvm prepare --bluestore --data /dev/sdd 
--block.wal /dev/sda2 --block.db /dev/sda1


Running command: sudo vgcreate --force --yes 
ceph-cc91203d-de5c-4d27-8c48-a58663075e67 /dev/sdd


stdout: Physical volume "/dev/sdd" successfully created.

stdout: Volume group "ceph-cc91203d-de5c-4d27-8c48-a58663075e67" 
successfully created


Running command: sudo lvcreate --yes -l 100%FREE -n 
osd-block-eb6cbcb3-f644-4973-b745-0e4389ef719c 
ceph-cc91203d-de5c-4d27-8c48-a58663075e67


stdout: Logical volume "osd-block-eb6cbcb3-f644-4973-b745-0e4389ef719c" 
created.


Running command: sudo mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-6

Running command: chown -R ceph:ceph /dev/dm-8

Running command: sudo ln -s 
/dev/ceph-cc91203d-de5c-4d27-8c48-a58663075e67/osd-block-eb6cbcb3-f644-4973-b745-0e4389ef719c 
/var/lib/ceph/osd/ceph-6/block


Running command: sudo ceph --cluster ceph --name client.bootstrap-osd 
--keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o