Re: [ceph-users] Using RBD with LVM
On 09/25/2013 02:00 AM, John-Paul Robinson wrote: Hi, I'm exploring a configuration with multiple Ceph block devices used with LVM. The goal is to provide a way to grow and shrink my file systems while they are on line. I've created three block devices: $ sudo ./ceph-ls | grep home jpr-home-lvm-p01: 102400 MB jpr-home-lvm-p02: 102400 MB jpr-home-lvm-p03: 102400 MB And have them mapped into my kernel (3.2.0-23-generic #36-Ubuntu SMP): $ sudo rbd showmapped id pool imagesnap device 0 rbd jpr-test-vol01 -/dev/rbd0 1 rbd jpr-home-lvm-p01 -/dev/rbd1 2 rbd jpr-home-lvm-p02 -/dev/rbd2 3 rbd jpr-home-lvm-p03 -/dev/rbd3 In order to use them with LVM, I need to define them as physical volumes. But when I run this command I get an unexpected error: $ sudo pvcreate /dev/rbd1 Device /dev/rbd1 not found (or ignored by filtering). Try this: $ sudo pvcreate -vvv /dev/rbd1 It has something to do with LVM filtering RBD devices away, you might need to add them manually in /etc/lvm/lvm.conf I've seen this before and fixed it, but I forgot what the root cause was. Wido I am able to use other RBD on this same machine to create file systems directly and mount them: $ df -h /mnt-test Filesystem Size Used Avail Use% Mounted on /dev/rbd050G 885M 47G 2% /mnt-test Is there a reason that the /dev/rbd[1-2] devices can't be initialized as physical volumes in LVM? Thanks, ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Wido den Hollander 42on B.V. Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Using RBD with LVM
Thanks. This fixed the problem. BTW, after adding this line I still got the same error on my pvcreate but then I ran pvcreate -vvv and found that it was ignorning my /dev/rbd1 device because it had detected a partition signature (which I had added in an earlier attempt to work around this ignored issue). I deleted the partion and the pvcreate worked on all my RBD devices. A basic recipe for creating an LVM volume is: for i in 1 2 3 do rbd create user1-home-lvm-p0$i --size 102400 rbd map user1-home-lvm-p0$i pvcreate user1-home-lvm-p0$i done vgcreate user1-home-vg \ /dev/rbd/rbd/user1-home-lvm-p01 \ /dev/rbd/rbd/user1-home-lvm-p02 \ /dev/rbd/rbd/user1-home-lvm-p03 lvcreate -nuser1-home-lv -l%100FREE user1-home-vg mkfs.ext4 /dev/user1-home-vg/user1-home-lv mount /dev/user1-home-vg/user1-home-lv /somewhere ~jpr On 09/24/2013 07:58 PM, Mandell Degerness wrote: You need to add a line to /etc/lvm/lvm.conf: types = [ rbd, 1024 ] It should be in the devices section of the file. On Tue, Sep 24, 2013 at 5:00 PM, John-Paul Robinson j...@uab.edu wrote: Hi, I'm exploring a configuration with multiple Ceph block devices used with LVM. The goal is to provide a way to grow and shrink my file systems while they are on line. I've created three block devices: $ sudo ./ceph-ls | grep home jpr-home-lvm-p01: 102400 MB jpr-home-lvm-p02: 102400 MB jpr-home-lvm-p03: 102400 MB And have them mapped into my kernel (3.2.0-23-generic #36-Ubuntu SMP): $ sudo rbd showmapped id pool imagesnap device 0 rbd jpr-test-vol01 -/dev/rbd0 1 rbd jpr-home-lvm-p01 -/dev/rbd1 2 rbd jpr-home-lvm-p02 -/dev/rbd2 3 rbd jpr-home-lvm-p03 -/dev/rbd3 In order to use them with LVM, I need to define them as physical volumes. But when I run this command I get an unexpected error: $ sudo pvcreate /dev/rbd1 Device /dev/rbd1 not found (or ignored by filtering). I am able to use other RBD on this same machine to create file systems directly and mount them: $ df -h /mnt-test Filesystem Size Used Avail Use% Mounted on /dev/rbd050G 885M 47G 2% /mnt-test Is there a reason that the /dev/rbd[1-2] devices can't be initialized as physical volumes in LVM? Thanks, ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Using RBD with LVM
Thanks. After fixing the issue with the types entry in lvm.conf, I discovered the -vvv option which helped me detect a the second cause for the ignored error: pvcreate saw a partition signature and skipped the device. The -vvv is s good flag. :) ~jpr On 09/25/2013 01:52 AM, Wido den Hollander wrote: Try this: $ sudo pvcreate -vvv /dev/rbd1 It has something to do with LVM filtering RBD devices away, you might need to add them manually in /etc/lvm/lvm.conf I've seen this before and fixed it, but I forgot what the root cause was. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Using RBD with LVM
You need to add a line to /etc/lvm/lvm.conf: types = [ rbd, 1024 ] It should be in the devices section of the file. On Tue, Sep 24, 2013 at 5:00 PM, John-Paul Robinson j...@uab.edu wrote: Hi, I'm exploring a configuration with multiple Ceph block devices used with LVM. The goal is to provide a way to grow and shrink my file systems while they are on line. I've created three block devices: $ sudo ./ceph-ls | grep home jpr-home-lvm-p01: 102400 MB jpr-home-lvm-p02: 102400 MB jpr-home-lvm-p03: 102400 MB And have them mapped into my kernel (3.2.0-23-generic #36-Ubuntu SMP): $ sudo rbd showmapped id pool imagesnap device 0 rbd jpr-test-vol01 -/dev/rbd0 1 rbd jpr-home-lvm-p01 -/dev/rbd1 2 rbd jpr-home-lvm-p02 -/dev/rbd2 3 rbd jpr-home-lvm-p03 -/dev/rbd3 In order to use them with LVM, I need to define them as physical volumes. But when I run this command I get an unexpected error: $ sudo pvcreate /dev/rbd1 Device /dev/rbd1 not found (or ignored by filtering). I am able to use other RBD on this same machine to create file systems directly and mount them: $ df -h /mnt-test Filesystem Size Used Avail Use% Mounted on /dev/rbd050G 885M 47G 2% /mnt-test Is there a reason that the /dev/rbd[1-2] devices can't be initialized as physical volumes in LVM? Thanks, ~jpr ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
