Re: about attaching rbd volume from instance on KVM
Hi Christian, Thank you for reply ! Could I ask a version of libvirt which is your patch applied ? I'm trying it with libvirt0.9.8-2ubuntu9, but there is same error. (I'm using blow package: https://launchpad.net/ubuntu/precise/+source/libvirt/0.9.8-2ubuntu9) root@compute04:~# dpkg -l | grep libvirt-bin ii libvirt-bin0.9.8-2ubuntu9 programs for the libvirt library /var/log/libvirtd/libvirt.log 2012-02-08 23:33:53.800+: 4663: error : qemuMonitorTextAddDrive:2828 : operation failed: open disk image file failed 2012-02-08 23:33:53.800+: 4663: error : virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink rbd/testvolume: No such file or directory 2012-02-08 23:33:54.212+: 4663: warning : qemuDomainAttachPciDiskDevice:287 : Unable to restore security label on rbd/testvolume 2012/2/6 Christian Brunner christ...@brunner-muc.de: Libvirt is trying to set security lables even for network shares. This will not work. I think this is fixed in newer libvirt versions. For older versions you can try this patch: http://www.redhat.com/archives/libvir-list/2011-May/msg01446.html Regards, Christian 2012/2/4 Masuko Tomoya tomoya.mas...@gmail.com: Hi, all. I'm trying to attach rbd volume from instance on KVM. But I have problem. Could you help me ? --- I tried to attach rbd volume on ceph01 to instance on compute1 with virsh command. root@compute1:~# virsh attach-device test-ub16 /root/testvolume.xml error: Failed to attach device from /root/testvolume.xml error: cannot resolve symlink rbd/testvolume: No such file or directory /var/log/messages Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error : qemuMonitorTextAddDevice:2417 : operation failed: adding virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4 device failed: Device needs media, but drive is empty#015#012Device 'virtio-blk-pci' could not be initialized#015#012 Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning : qemuDomainAttachPciDiskDevice:188 : qemuMonitorAddDevice failed on file=rbd:rbd/testvolume,if=none,id=drive-virtio-disk4,format=raw (virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4) Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error : virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink rbd/testvolume: No such file or directory Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning : qemuDomainAttachPciDiskDevice:229 : Unable to restore security label on rbd/testvolume there is no log in /var/log/ceph/mon.0.log of host ceph01. --- My environment is below. *There are two servers. All server are ubuntu 10.10 x86_64. *ceph01: single server configured ceph.(version: 0.41-1maverick) *compute1: kvm hypervisor -librados2 and librbd1 packages are installed. (version: 0.41-1maverick) -qemu-kvm is 0.14.0-rc1. I built qemu with rbd enable. the output of run 'qemu-img' show 'rbd' at supported formats field. (I built qemu reffering this page. http://ceph.newdream.net/wiki/QEMU-RBD) -apparmor is disable. -libvirt is 0.8.8 -there is ceph.conf on compute1. root@compute1:~# ls -l /etc/ceph/ total 20 -rw-r--r-- 1 root root 508 2012-02-03 14:38 ceph.conf -rw--- 1 root root 63 2012-02-03 17:04 keyring.admin -rw--- 1 root root 63 2012-02-03 14:38 keyring.bin -rw--- 1 root root 56 2012-02-03 14:38 keyring.mds.0 -rw--- 1 root root 56 2012-02-03 14:38 keyring.osd.0 = -contents of ceph.conf is below. root@compute1:~# cat /etc/ceph/ceph.conf [global] auth supported = cephx keyring = /etc/ceph/keyring.bin [mon] mon data = /data/data/mon$id debug ms = 1 [mon.0] host = ceph01 mon addr = 10.68.119.191:6789 [mds] keyring = /etc/ceph/keyring.$name [mds.0] host = ceph01 [osd] keyring = /etc/ceph/keyring.$name osd data = /data/osd$id osd journal = /data/osd$id/journal osd journal size = 512 osd class tmp = /var/lib/ceph/tmp debug osd = 20 debug ms = 1 debug filestore = 20 [osd.0] host = ceph01 btrfs devs = /dev/sdb1 === *conten of keyring.admin is below root@compute1:~# cat /etc/ceph/keyring.admin [client.admin] key = AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg== === *output of run 'ceph auth list' root@ceph01:/etc/ceph# ceph auth list 2012-02-03 20:34:59.507451 mon - [auth,list] 2012-02-03 20:34:59.508785 mon.0 - 'installed auth entries: mon. key: AQDFeCxPiK04IxAAslDBNkrOGKWxcbCh2iysqg== mds.0 key: AQDFeCxPsJ+LGhAAJ3/rmkAtGXSv/eHh0yXgww== caps: [mds] allow caps: [mon] allow rwx caps: [osd] allow * osd.0 key: AQDFeCxPoEK+ExAAecD7+tWgpIRoZx2AT7Jwbg== caps: [mon] allow rwx caps: [osd] allow * client.admin key: AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg== caps: [mds] allow caps: [mon] allow
Re: couldn't use rbd
: 0.38-1maverick)
*apparmor is disable.
apparmor shouldn't matter if you have libvirt 0.9.9 or newer.
I use libvirt 0.8.3 (latest version for maverick), so I disabled apparmor.
2012/2/2 Josh Durgin josh.dur...@dreamhost.com:
On 02/02/2012 01:49 AM, Masuko Tomoya wrote:
Hi, all.
When I execute rbd command, it is not success.
root@ceph01:~# rbd list
(no response)
/var/log/ceph/mon.0.log
-
2012-02-02 17:58:19.801762 7ff4bbfb1710 -- 10.68.119.191:6789/0==
client.? 10.68.119.191:0/1002580 1 auth(proto 0 30 bytes) v1
56+0+0 (625540289 0 0) 0x1619a00 con 0x1615a00
2012-02-02 17:58:19.801919 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x1619c00 con 0x1615a00
2012-02-02 17:58:19.802505 7ff4bbfb1710 -- 10.68.119.191:6789/0==
client.? 10.68.119.191:0/1002580 2 auth(proto 2 32 bytes) v1
58+0+0 (346146289 0 0) 0x161fc00 con 0x1615a00
2012-02-02 17:58:19.802673 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x1619a00 con 0x1615a00
2012-02-02 17:58:19.803473 7ff4bbfb1710 -- 10.68.119.191:6789/0==
client.? 10.68.119.191:0/1002580 3 auth(proto 2 165 bytes) v1
191+0+0 (3737796417 0 0) 0x1619600 con 0x1615a00
2012-02-02 17:58:19.803745 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x161fc00 con 0x1615a00
2012-02-02 17:58:19.804425 7ff4bbfb1710 -- 10.68.119.191:6789/0==
client.? 10.68.119.191:0/1002580 4 mon_subscribe({monmap=0+}) v2
23+0+0 (1620593354 0 0) 0x1617380 con 0x1615a00
2012-02-02 17:58:19.804488 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- mon_map v1 -- ?+0 0x1635700 con 0x1615a00
2012-02-02 17:58:19.804517 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.? 10.68.119.191:0/1002580 -- mon_subscribe_ack(300s) v1 -- ?+0
0x163d780
2012-02-02 17:58:19.804550 7ff4bbfb1710 -- 10.68.119.191:6789/0==
client.4112 10.68.119.191:0/1002580 5
mon_subscribe({monmap=0+,osdmap=0}) v2 42+0+0 (982583713 0 0)
0x1617a80 con 0x1615a00
2012-02-02 17:58:19.804578 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- mon_map v1 -- ?+0 0x1617380 con 0x1615a00
2012-02-02 17:58:19.804656 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.? 10.68.119.191:0/1002580 -- osd_map(3..3 src has 1..3) v1 --
?+0 0x1619600
2012-02-02 17:58:19.804744 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.4112 10.68.119.191:0/1002580 -- mon_subscribe_ack(300s) v1 --
?+0 0x163d900
2012-02-02 17:58:19.804778 7ff4bbfb1710 -- 10.68.119.191:6789/0==
client.4112 10.68.119.191:0/1002580 6
mon_subscribe({monmap=0+,osdmap=0}) v2 42+0+0 (982583713 0 0)
0x16178c0 con 0x1615a00
2012-02-02 17:58:19.804811 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- mon_map v1 -- ?+0 0x1617a80 con 0x1615a00
2012-02-02 17:58:19.804855 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.? 10.68.119.191:0/1002580 -- osd_map(3..3 src has 1..3) v1 --
?+0 0x1619400
2012-02-02 17:58:19.804884 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.4112 10.68.119.191:0/1002580 -- mon_subscribe_ack(300s) v1 --
?+0 0x161d300
-
No problems there.
BTW, I could execute rados lspools.
root@ceph01:~# rados lspools
data
metadata
rbd
This might mean the rbd image list object can't be read for some
reason, or the rbd tool is doing something weird that the rados tool
isn't. Can you share the output of 'ceph -s' and
'rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20 --debug-monc 20
--debug-auth 20'?
You can run 'rados lspools' with those options as well and compare.
I would like to use rbd volume and attach it as virtual device for VM
guest on KVM.
Could you advice to me ?
My environment is below.
*ceph cluster is configured on single server.
*server is ubuntu 10.10 maverick.
*ceph, librados2 and librbd1 packages are installed.
(version: 0.38-1maverick)
*apparmor is disable.
apparmor shouldn't matter if you have libvirt 0.9.9 or newer.
*root@ceph01:/# ls -l /etc/ceph
total 16
-rw-r--r-- 1 root root 340 2012-02-02 17:28 ceph.conf
-rw--- 1 root root 92 2012-02-02 17:28 client.admin.keyring
-rw--- 1 root root 85 2012-02-02 17:28 mds.0.keyring
-rw--- 1 root root 85 2012-02-02 17:28 osd.0.keyring
*/var/lib/ceph/tmp is exists.
root@ceph01:/var/log# ls -l /var/lib/ceph/
total 4
drwxrwxrwx 2 root root 4096 2011-11-11 09:28 tmp
*/etc/ceph/ceph.conf
[global]
auth supported = cephx
keyring = /etc/ceph/$name.keyring
[mon]
mon data = /data/data/mon$id
debug ms = 1
[mon.0]
host = ceph01
mon addr = 10.68.119.191:6789
[mds]
[mds.0]
host = ceph01
[osd]
osd data = /data/osd$id
osd journal = /data/osd$id/journal
osd journal size = 512
osd class tmp = /var/lib/ceph/tmp
[osd.0]
host = ceph01
Re: couldn't use rbd
avail, 15358 MB total, peers []/[])
2012-02-04 04:29:28.424886 7fe0e12fe710 osd.0 32 update_osd_stat
osd_stat(1400 KB used, 15357 MB avail, 15358 MB total, peers []/[])
2012-02-04 04:29:28.424957 7fe0e12fe710 osd.0 32 heartbeat:
osd_stat(1400 KB used, 15357 MB avail, 15358 MB total, peers []/[])
2012-02-04 04:29:28.647769 7fe0edc18710 osd.0 32 tick
2012-02-04 04:29:28.647971 7fe0edc18710 osd.0 32 scrub_should_schedule
loadavg 0 max 0.5 = no, randomly backing off
2012-02-04 04:29:29.648746 7fe0edc18710 osd.0 32 tick
2012-02-04 04:29:29.648955 7fe0edc18710 osd.0 32 scrub_should_schedule
loadavg 0 max 0.5 = no, randomly backing off
2012/2/3 Josh Durgin josh.dur...@dreamhost.com:
On 02/03/2012 12:51 AM, Masuko Tomoya wrote:
Hi Josh,
Thank you for reply !
This might mean the rbd image list object can't be read for some
reason, or the rbd tool is doing something weird that the rados tool
isn't. Can you share the output of 'ceph -s' and
'rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20 --debug-monc 20
--debug-auth 20'?
The output of 'ceph -s' is below.
---
root@ceph01:~# ceph -s
2012-02-03 17:01:47.881960 pg v33: 6 pgs: 6 active+clean+degraded;
0 KB data, 1056 KB used, 15357 MB / 15358 MB avail
2012-02-03 17:01:47.882583 mds e9: 1/1/1 up {0=0=up:creating}
2012-02-03 17:01:47.882733 osd e21: 1 osds: 1 up, 1 in
2012-02-03 17:01:47.883042 log 2012-02-03 16:35:11.183897 osd.0
10.68.119.191:6801/2912 12 : [WRN] map e19 wrongly marked me down or
wrong addr
2012-02-03 17:01:47.883144 mon e1: 1 mons at {0=10.68.119.191:6789/0}
The output of 'rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20
--debug-monc 20 --debug-auth 20' is below.
--
root@ceph01:~# rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20
--debug-monc 20 --debug-auth 20
snip
2012-02-03 17:02:10.971391 7f88cbb91720 client.4106.objecter op_submit
oid rbd_directory @2 [read 0~0] tid 1 osd.0
2012-02-03 17:02:10.971465 7f88cbb91720 client.4106.objecter send_op 1 to
osd.0
2012-02-03 17:02:10.971533 7f88cbb91720 -- 10.68.119.191:0/1003500 --
10.68.119.191:6801/2912 -- osd_op(client.4106.0:1 rbd_directory [read
0~0] 2.30a98c1c) v1 -- ?+0 0x24664c0 con 0x24661b0
Everything above here is normal - the rbd tool connected to the
monitors, got the monmap and osdmap, and sent a request to read the
'rbd_directory' object.
snip
2012-02-03 17:02:25.969338 7f88c7261710 client.4106.objecter tid 1 on
osd.0 is laggy
This means the osd isn't responding to the read. Check the osd log for
errors. If there's nothing obvious, add this to the osd section of your
ceph.conf and restart the osd:
debug osd = 20
debug ms = 1
debug filestore = 20
Then run 'rbd ls' and look at what happens after
'osd_op.*rbd_directory' appears in the osd log.
rados lspools log
I compared those logs and found there is differences.
'rbd list'
2012-02-03 17:02:10.971770 7f88c9366710 -- 10.68.119.191:0/1003500==
mon.0 10.68.119.191:6789/0 10 osd_map(21..21 src has 1..21) v2
1284+0+0 (473305567 0 0) 0x24655a0 con 0x24603e0
2012-02-03 17:02:10.971789 7f88c9366710 client.4106.objecter
handle_osd_map ignoring epochs [21,21]= 21
2012-02-03 17:02:10.971801 7f88c9366710 client.4106.objecter
dump_active .. 0 homeless
2012-02-03 17:02:10.971815 7f88c9366710 client.4106.objecter 1
2.30a98c1c osd.0 rbd_directory [read 0~0]
--(snip)--
'rados lspools'
2012-02-03 17:11:52.866072 7f9c5764b710 -- 10.68.119.191:0/1003868==
mon.0 10.68.119.191:6789/0 7 osd_map(21..21 src has 1..21) v2
1284+0+0 (473305567 0 0) 0x770a70 con 0x771440
2012-02-03 17:11:52.866103 7f9c5764b710 client.4107.objecter
handle_osd_map got epochs [21,21] 0
2012-02-03 17:11:52.866111 7f9c5764b710 client.4107.objecter
handle_osd_map decoding full epoch 21
2012-02-03 17:11:52.866272 7f9c5764b710 client.4107.objecter
dump_active .. 0 homeless
data
metadata
rbd
--(snip)--
What do these logs mean ?
The difference is that 'rbd ls' talks to the monitors and osds, while 'rados
lspools' just needs to talk to the monitors. The objecter dump_active part
is listing in-flight osd requests.
*ceph cluster is configured on single server.
*server is ubuntu 10.10 maverick.
*ceph, librados2 and librbd1 packages are installed.
(version: 0.38-1maverick)
*apparmor is disable.
apparmor shouldn't matter if you have libvirt 0.9.9 or newer.
I use libvirt 0.8.3 (latest version for maverick), so I disabled apparmor.
2012/2/2 Josh Durginjosh.dur...@dreamhost.com:
On 02/02/2012 01:49 AM, Masuko Tomoya wrote:
Hi, all.
When I execute rbd command, it is not success.
root@ceph01:~# rbd list
(no response)
/var/log/ceph/mon.0.log
-
2012-02-02 17:58:19.801762 7ff4bbfb1710 -- 10.68.119.191:6789/0==
client.? 10.68.119.191:0/1002580 1 auth(proto 0 30 bytes) v1
56+0+0 (625540289 0 0) 0x1619a00 con 0x1615a00
2012-02-02 17:58:19.801919 7ff4bbfb1710 -- 10.68.119.191:6789/0
Re: couldn't use rbd
Hi,
The output of 'ceph pg dump' is below.
root@ceph01:~# ceph pg dump
2012-02-04 07:50:15.453151 mon - [pg,dump]
2012-02-04 07:50:15.453734 mon.0 - 'dumped all in format plain' (0)
version 63
last_osdmap_epoch 37
last_pg_scan 1
full_ratio 0.95
nearfull_ratio 0.85
pg_stat objects mip degrunf kb bytes log
disklog state v reportedup acting last_scrub
1.1p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:25:55.383343
0.0p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'32 [0] [0] 0'0
2012-02-03 10:25:51.380648
1.0p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:25:53.381291
0.1p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'32 [0] [0] 0'0
2012-02-03 10:25:52.380881
2.0p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:25:59.387441
2.1p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:26:04.392778
pool 0 0 0 0 0 0 0 0 0
pool 1 0 0 0 0 0 0 0 0
pool 2 0 0 0 0 0 0 0 0
sum0 0 0 0 0 0 0 0
osdstat kbused kbavail kb hb in hb out
0 15681572603215727600[] []
sum15681572603215727600
2012/2/3 Josh Durgin josh.dur...@dreamhost.com:
On 02/03/2012 02:14 PM, Masuko Tomoya wrote:
Hi Josh,
Thank you for your comments.
debug osd = 20
debug ms = 1
debug filestore = 20
I added this to the osd section of ceph.conf and ran /etc/init.d/ceph
stopstart.
The output of OSD.log when 'rbd list' was executed is below.
-
2012-02-04 04:29:22.457990 7fe0e08fb710 osd.0 32
OSD::ms_verify_authorizer name=client.admin auid=0
2012-02-04 04:29:22.458041 7fe0e08fb710 osd.0 32 new session
0x24f5240 con=0x24d4dc0 addr=10.68.119.191:0/1005110
2012-02-04 04:29:22.458069 7fe0e08fb710 osd.0 32 session 0x24f5240
has caps osdcaps(pools={} default allow= default_deny=)
2012-02-04 04:29:22.458415 7fe0e6c0a710 -- 10.68.119.191:6801/4992==
client.4201 10.68.119.191:0/1005110 1 osd_op(client.4201.0:1
rbd_directory [read 0~0] 2.30a98c1c) v3 143+0+0 (3720164172 0 0)
0x24d8900 con 0x24d4dc0
2012-02-04 04:29:22.458442 7fe0e6c0a710 osd.0 32 _dispatch 0x24d8900
osd_op(client.4201.0:1 rbd_directory [read 0~0] 2.30a98c1c) v3
2012-02-04 04:29:22.458463 7fe0e6c0a710 osd.0 32
require_same_or_newer_map 32 (i am 32) 0x24d8900
2012-02-04 04:29:22.458487 7fe0e6c0a710 osd.0 32 _share_map_incoming
client.4201 10.68.119.191:0/1005110 32
2012-02-04 04:29:22.458507 7fe0e6c0a710 osd.0 32 hit non-existent pg
2.0, waiting
The pg should have been created already. What's the output of 'ceph pg
dump'?
2012/2/3 Josh Durginjosh.dur...@dreamhost.com:
On 02/03/2012 12:51 AM, Masuko Tomoya wrote:
Hi Josh,
Thank you for reply !
This might mean the rbd image list object can't be read for some
reason, or the rbd tool is doing something weird that the rados tool
isn't. Can you share the output of 'ceph -s' and
'rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20 --debug-monc
20
--debug-auth 20'?
The output of 'ceph -s' is below.
---
root@ceph01:~# ceph -s
2012-02-03 17:01:47.881960 pg v33: 6 pgs: 6 active+clean+degraded;
0 KB data, 1056 KB used, 15357 MB / 15358 MB avail
2012-02-03 17:01:47.882583 mds e9: 1/1/1 up {0=0=up:creating}
2012-02-03 17:01:47.882733 osd e21: 1 osds: 1 up, 1 in
2012-02-03 17:01:47.883042 log 2012-02-03 16:35:11.183897 osd.0
10.68.119.191:6801/2912 12 : [WRN] map e19 wrongly marked me down or
wrong addr
2012-02-03 17:01:47.883144 mon e1: 1 mons at {0=10.68.119.191:6789/0}
The output of 'rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20
--debug-monc 20 --debug-auth 20' is below.
--
root@ceph01:~# rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20
--debug-monc 20 --debug-auth 20
snip
2012-02-03 17:02:10.971391 7f88cbb91720 client.4106.objecter op_submit
oid rbd_directory @2 [read 0~0] tid 1 osd.0
2012-02-03 17:02:10.971465 7f88cbb91720 client.4106.objecter send_op 1
to
osd.0
2012-02-03 17:02:10.971533 7f88cbb91720 -- 10.68.119.191:0/1003500 --
10.68.119.191:6801/2912 -- osd_op(client.4106.0:1 rbd_directory [read
0~0] 2.30a98c1c) v1 -- ?+0 0x24664c0 con 0x24661b0
Everything above here is normal - the rbd tool connected to the
monitors, got the monmap and osdmap, and sent a request to read the
'rbd_directory' object.
snip
2012-02-03 17:02:25.969338 7f88c7261710 client.4106.objecter tid 1 on
osd.0 is laggy
Re: couldn't use rbd
Hi,
I upgraded ceph to 0.41 and re-running mkcephfs.
I found my issue is fixed.
-
root@ceph01:~# rbd list
pool rbd doesn't contain rbd images
root@ceph01:~# rbd create test --size 1024
root@ceph01:~# rbd list
test
-
Josh, thank you for your advices.
2012/2/3 Josh Durgin josh.dur...@dreamhost.com:
On 02/03/2012 02:54 PM, Masuko Tomoya wrote:
Hi,
The output of 'ceph pg dump' is below.
root@ceph01:~# ceph pg dump
2012-02-04 07:50:15.453151 mon- [pg,dump]
2012-02-04 07:50:15.453734 mon.0 - 'dumped all in format plain' (0)
version 63
last_osdmap_epoch 37
last_pg_scan 1
full_ratio 0.95
nearfull_ratio 0.85
pg_stat objects mip degr unf kb bytes log
disklog state v reported up acting last_scrub
1.1p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:25:55.383343
0.0p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'32 [0] [0] 0'0
2012-02-03 10:25:51.380648
1.0p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:25:53.381291
0.1p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'32 [0] [0] 0'0
2012-02-03 10:25:52.380881
2.0p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:25:59.387441
2.1p0 0 0 0 0 0 0 0 0
active+clean+degraded 0'0 34'30 [0] [0] 0'0
2012-02-03 10:26:04.392778
pool 0 0 0 0 0 0 0 0 0
pool 1 0 0 0 0 0 0 0 0
pool 2 0 0 0 0 0 0 0 0
sum 0 0 0 0 0 0 0 0
osdstat kbused kbavail kb hb in hb out
0 1568 15726032 15727600 [] []
sum 1568 15726032 15727600
You hit a bug in 0.38 that made the default crushmap for one osd
contain no pgs. This was fixed in 0.39, so I'd suggest upgrading and
re-running mkcephfs.
2012/2/3 Josh Durginjosh.dur...@dreamhost.com:
On 02/03/2012 02:14 PM, Masuko Tomoya wrote:
Hi Josh,
Thank you for your comments.
debug osd = 20
debug ms = 1
debug filestore = 20
I added this to the osd section of ceph.conf and ran /etc/init.d/ceph
stopstart.
The output of OSD.log when 'rbd list' was executed is below.
-
2012-02-04 04:29:22.457990 7fe0e08fb710 osd.0 32
OSD::ms_verify_authorizer name=client.admin auid=0
2012-02-04 04:29:22.458041 7fe0e08fb710 osd.0 32 new session
0x24f5240 con=0x24d4dc0 addr=10.68.119.191:0/1005110
2012-02-04 04:29:22.458069 7fe0e08fb710 osd.0 32 session 0x24f5240
has caps osdcaps(pools={} default allow= default_deny=)
2012-02-04 04:29:22.458415 7fe0e6c0a710 -- 10.68.119.191:6801/4992==
client.4201 10.68.119.191:0/1005110 1 osd_op(client.4201.0:1
rbd_directory [read 0~0] 2.30a98c1c) v3 143+0+0 (3720164172 0 0)
0x24d8900 con 0x24d4dc0
2012-02-04 04:29:22.458442 7fe0e6c0a710 osd.0 32 _dispatch 0x24d8900
osd_op(client.4201.0:1 rbd_directory [read 0~0] 2.30a98c1c) v3
2012-02-04 04:29:22.458463 7fe0e6c0a710 osd.0 32
require_same_or_newer_map 32 (i am 32) 0x24d8900
2012-02-04 04:29:22.458487 7fe0e6c0a710 osd.0 32 _share_map_incoming
client.4201 10.68.119.191:0/1005110 32
2012-02-04 04:29:22.458507 7fe0e6c0a710 osd.0 32 hit non-existent pg
2.0, waiting
The pg should have been created already. What's the output of 'ceph pg
dump'?
2012/2/3 Josh Durginjosh.dur...@dreamhost.com:
On 02/03/2012 12:51 AM, Masuko Tomoya wrote:
Hi Josh,
Thank you for reply !
This might mean the rbd image list object can't be read for some
reason, or the rbd tool is doing something weird that the rados tool
isn't. Can you share the output of 'ceph -s' and
'rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20 --debug-monc
20
--debug-auth 20'?
The output of 'ceph -s' is below.
---
root@ceph01:~# ceph -s
2012-02-03 17:01:47.881960 pg v33: 6 pgs: 6 active+clean+degraded;
0 KB data, 1056 KB used, 15357 MB / 15358 MB avail
2012-02-03 17:01:47.882583 mds e9: 1/1/1 up {0=0=up:creating}
2012-02-03 17:01:47.882733 osd e21: 1 osds: 1 up, 1 in
2012-02-03 17:01:47.883042 log 2012-02-03 16:35:11.183897 osd.0
10.68.119.191:6801/2912 12 : [WRN] map e19 wrongly marked me down or
wrong addr
2012-02-03 17:01:47.883144 mon e1: 1 mons at
{0=10.68.119.191:6789/0}
The output of 'rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20
--debug-monc 20 --debug-auth 20' is below.
--
root@ceph01:~# rbd ls --log-to-stderr --debug-ms 1 --debug-objecter 20
--debug-monc 20 --debug-auth 20
snip
2012-02-03 17:02:10.971391 7f88cbb91720 client.4106.objecter op_submit
oid
[no subject]
Hi, all. I'm trying to attach rbd volume from instance on KVM. But I have problem. Could you help me ? --- I tried to attach rbd volume on ceph01 to instance on compute1 with virsh command. root@compute1:~# virsh attach-device test-ub16 /root/testvolume.xml error: Failed to attach device from /root/testvolume.xml error: cannot resolve symlink rbd/testvolume: No such file or directory /var/log/messages Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error : qemuMonitorTextAddDevice:2417 : operation failed: adding virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4 device failed: Device needs media, but drive is empty#015#012Device 'virtio-blk-pci' could not be initialized#015#012 Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning : qemuDomainAttachPciDiskDevice:188 : qemuMonitorAddDevice failed on file=rbd:rbd/testvolume,if=none,id=drive-virtio-disk4,format=raw (virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4) Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error : virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink rbd/testvolume: No such file or directory Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning : qemuDomainAttachPciDiskDevice:229 : Unable to restore security label on rbd/testvolume there is no log in /var/log/ceph/mon.0.log of host ceph01. --- My environment is below. *There are two servers. All server are ubuntu 10.10 x86_64. *ceph01: single server configured ceph.(version: 0.41-1maverick) *compute1: kvm hypervisor -librados2 and librbd1 packages are installed. (version: 0.41-1maverick) -qemu-kvm is 0.14.0-rc1. I built qemu with rbd enable. the output of run 'qemu-img' show 'rbd' at supported formats field. (I built qemu reffering this page. http://ceph.newdream.net/wiki/QEMU-RBD) -apparmor is disable. -libvirt is 0.8.8 -there is ceph.conf on compute1. root@compute1:~# ls -l /etc/ceph/ total 20 -rw-r--r-- 1 root root 508 2012-02-03 14:38 ceph.conf -rw--- 1 root root 63 2012-02-03 17:04 keyring.admin -rw--- 1 root root 63 2012-02-03 14:38 keyring.bin -rw--- 1 root root 56 2012-02-03 14:38 keyring.mds.0 -rw--- 1 root root 56 2012-02-03 14:38 keyring.osd.0 = -contents of ceph.conf is below. root@compute1:~# cat /etc/ceph/ceph.conf [global] auth supported = cephx keyring = /etc/ceph/keyring.bin [mon] mon data = /data/data/mon$id debug ms = 1 [mon.0] host = ceph01 mon addr = 10.68.119.191:6789 [mds] keyring = /etc/ceph/keyring.$name [mds.0] host = ceph01 [osd] keyring = /etc/ceph/keyring.$name osd data = /data/osd$id osd journal = /data/osd$id/journal osd journal size = 512 osd class tmp = /var/lib/ceph/tmp debug osd = 20 debug ms = 1 debug filestore = 20 [osd.0] host = ceph01 btrfs devs = /dev/sdb1 === *conten of keyring.admin is below root@compute1:~# cat /etc/ceph/keyring.admin [client.admin] key = AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg== === *output of run 'ceph auth list' root@ceph01:/etc/ceph# ceph auth list 2012-02-03 20:34:59.507451 mon - [auth,list] 2012-02-03 20:34:59.508785 mon.0 - 'installed auth entries: mon. key: AQDFeCxPiK04IxAAslDBNkrOGKWxcbCh2iysqg== mds.0 key: AQDFeCxPsJ+LGhAAJ3/rmkAtGXSv/eHh0yXgww== caps: [mds] allow caps: [mon] allow rwx caps: [osd] allow * osd.0 key: AQDFeCxPoEK+ExAAecD7+tWgpIRoZx2AT7Jwbg== caps: [mon] allow rwx caps: [osd] allow * client.admin key: AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg== caps: [mds] allow caps: [mon] allow * caps: [osd] allow * ' (0) *xml file is below. root@compute1:~# cat /root/testvolume.xml disk type='network' device='disk' driver name='qemu' type='raw'/ source protocol='rbd' name='rbd/testvolume' host name='10.68.119.191' port='6789'/ /source target dev='vde' bus='virtio'/ /disk *testvolume is on rados pools. root@compute1:~# qemu-img info rbd:rbd/testvolume image: rbd:rbd/testvolume file format: raw virtual size: 1.0G (1073741824 bytes) disk size: unavailable Waiting for reply, Tomoya. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
about attaching rbd volume from instance on KVM
Hi, all. I'm trying to attach rbd volume from instance on KVM. But I have problem. Could you help me ? --- I tried to attach rbd volume on ceph01 to instance on compute1 with virsh command. root@compute1:~# virsh attach-device test-ub16 /root/testvolume.xml error: Failed to attach device from /root/testvolume.xml error: cannot resolve symlink rbd/testvolume: No such file or directory /var/log/messages Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error : qemuMonitorTextAddDevice:2417 : operation failed: adding virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4 device failed: Device needs media, but drive is empty#015#012Device 'virtio-blk-pci' could not be initialized#015#012 Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning : qemuDomainAttachPciDiskDevice:188 : qemuMonitorAddDevice failed on file=rbd:rbd/testvolume,if=none,id=drive-virtio-disk4,format=raw (virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4) Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: error : virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink rbd/testvolume: No such file or directory Feb 3 20:14:48 compute1 libvirtd: 20:14:48.717: 3234: warning : qemuDomainAttachPciDiskDevice:229 : Unable to restore security label on rbd/testvolume there is no log in /var/log/ceph/mon.0.log of host ceph01. --- My environment is below. *There are two servers. All server are ubuntu 10.10 x86_64. *ceph01: single server configured ceph.(version: 0.41-1maverick) *compute1: kvm hypervisor -librados2 and librbd1 packages are installed. (version: 0.41-1maverick) -qemu-kvm is 0.14.0-rc1. I built qemu with rbd enable. the output of run 'qemu-img' show 'rbd' at supported formats field. (I built qemu reffering this page. http://ceph.newdream.net/wiki/QEMU-RBD) -apparmor is disable. -libvirt is 0.8.8 -there is ceph.conf on compute1. root@compute1:~# ls -l /etc/ceph/ total 20 -rw-r--r-- 1 root root 508 2012-02-03 14:38 ceph.conf -rw--- 1 root root 63 2012-02-03 17:04 keyring.admin -rw--- 1 root root 63 2012-02-03 14:38 keyring.bin -rw--- 1 root root 56 2012-02-03 14:38 keyring.mds.0 -rw--- 1 root root 56 2012-02-03 14:38 keyring.osd.0 = -contents of ceph.conf is below. root@compute1:~# cat /etc/ceph/ceph.conf [global] auth supported = cephx keyring = /etc/ceph/keyring.bin [mon] mon data = /data/data/mon$id debug ms = 1 [mon.0] host = ceph01 mon addr = 10.68.119.191:6789 [mds] keyring = /etc/ceph/keyring.$name [mds.0] host = ceph01 [osd] keyring = /etc/ceph/keyring.$name osd data = /data/osd$id osd journal = /data/osd$id/journal osd journal size = 512 osd class tmp = /var/lib/ceph/tmp debug osd = 20 debug ms = 1 debug filestore = 20 [osd.0] host = ceph01 btrfs devs = /dev/sdb1 === *conten of keyring.admin is below root@compute1:~# cat /etc/ceph/keyring.admin [client.admin] key = AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg== === *output of run 'ceph auth list' root@ceph01:/etc/ceph# ceph auth list 2012-02-03 20:34:59.507451 mon - [auth,list] 2012-02-03 20:34:59.508785 mon.0 - 'installed auth entries: mon. key: AQDFeCxPiK04IxAAslDBNkrOGKWxcbCh2iysqg== mds.0 key: AQDFeCxPsJ+LGhAAJ3/rmkAtGXSv/eHh0yXgww== caps: [mds] allow caps: [mon] allow rwx caps: [osd] allow * osd.0 key: AQDFeCxPoEK+ExAAecD7+tWgpIRoZx2AT7Jwbg== caps: [mon] allow rwx caps: [osd] allow * client.admin key: AQDFeCxPyBlNIRAAxS1DcRHpMXRpcjY/GNMwYg== caps: [mds] allow caps: [mon] allow * caps: [osd] allow * ' (0) *xml file is below. root@compute1:~# cat /root/testvolume.xml disk type='network' device='disk' driver name='qemu' type='raw'/ source protocol='rbd' name='rbd/testvolume' host name='10.68.119.191' port='6789'/ /source target dev='vde' bus='virtio'/ /disk *testvolume is on rados pools. root@compute1:~# qemu-img info rbd:rbd/testvolume image: rbd:rbd/testvolume file format: raw virtual size: 1.0G (1073741824 bytes) disk size: unavailable Waiting for reply, Tomoya. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
couldn't use rbd
Hi, all.
When I execute rbd command, it is not success.
root@ceph01:~# rbd list
(no response)
/var/log/ceph/mon.0.log
-
2012-02-02 17:58:19.801762 7ff4bbfb1710 -- 10.68.119.191:6789/0 ==
client.? 10.68.119.191:0/1002580 1 auth(proto 0 30 bytes) v1
56+0+0 (625540289 0 0) 0x1619a00 con 0x1615a00
2012-02-02 17:58:19.801919 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x1619c00 con 0x1615a00
2012-02-02 17:58:19.802505 7ff4bbfb1710 -- 10.68.119.191:6789/0 ==
client.? 10.68.119.191:0/1002580 2 auth(proto 2 32 bytes) v1
58+0+0 (346146289 0 0) 0x161fc00 con 0x1615a00
2012-02-02 17:58:19.802673 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x1619a00 con 0x1615a00
2012-02-02 17:58:19.803473 7ff4bbfb1710 -- 10.68.119.191:6789/0 ==
client.? 10.68.119.191:0/1002580 3 auth(proto 2 165 bytes) v1
191+0+0 (3737796417 0 0) 0x1619600 con 0x1615a00
2012-02-02 17:58:19.803745 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x161fc00 con 0x1615a00
2012-02-02 17:58:19.804425 7ff4bbfb1710 -- 10.68.119.191:6789/0 ==
client.? 10.68.119.191:0/1002580 4 mon_subscribe({monmap=0+}) v2
23+0+0 (1620593354 0 0) 0x1617380 con 0x1615a00
2012-02-02 17:58:19.804488 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- mon_map v1 -- ?+0 0x1635700 con 0x1615a00
2012-02-02 17:58:19.804517 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.? 10.68.119.191:0/1002580 -- mon_subscribe_ack(300s) v1 -- ?+0
0x163d780
2012-02-02 17:58:19.804550 7ff4bbfb1710 -- 10.68.119.191:6789/0 ==
client.4112 10.68.119.191:0/1002580 5
mon_subscribe({monmap=0+,osdmap=0}) v2 42+0+0 (982583713 0 0)
0x1617a80 con 0x1615a00
2012-02-02 17:58:19.804578 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- mon_map v1 -- ?+0 0x1617380 con 0x1615a00
2012-02-02 17:58:19.804656 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.? 10.68.119.191:0/1002580 -- osd_map(3..3 src has 1..3) v1 --
?+0 0x1619600
2012-02-02 17:58:19.804744 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.4112 10.68.119.191:0/1002580 -- mon_subscribe_ack(300s) v1 --
?+0 0x163d900
2012-02-02 17:58:19.804778 7ff4bbfb1710 -- 10.68.119.191:6789/0 ==
client.4112 10.68.119.191:0/1002580 6
mon_subscribe({monmap=0+,osdmap=0}) v2 42+0+0 (982583713 0 0)
0x16178c0 con 0x1615a00
2012-02-02 17:58:19.804811 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
10.68.119.191:0/1002580 -- mon_map v1 -- ?+0 0x1617a80 con 0x1615a00
2012-02-02 17:58:19.804855 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.? 10.68.119.191:0/1002580 -- osd_map(3..3 src has 1..3) v1 --
?+0 0x1619400
2012-02-02 17:58:19.804884 7ff4bbfb1710 -- 10.68.119.191:6789/0 --
client.4112 10.68.119.191:0/1002580 -- mon_subscribe_ack(300s) v1 --
?+0 0x161d300
-
BTW, I could execute rados lspools.
root@ceph01:~# rados lspools
data
metadata
rbd
I would like to use rbd volume and attach it as virtual device for VM
guest on KVM.
Could you advice to me ?
My environment is below.
*ceph cluster is configured on single server.
*server is ubuntu 10.10 maverick.
*ceph, librados2 and librbd1 packages are installed.
(version: 0.38-1maverick)
*apparmor is disable.
*root@ceph01:/# ls -l /etc/ceph
total 16
-rw-r--r-- 1 root root 340 2012-02-02 17:28 ceph.conf
-rw--- 1 root root 92 2012-02-02 17:28 client.admin.keyring
-rw--- 1 root root 85 2012-02-02 17:28 mds.0.keyring
-rw--- 1 root root 85 2012-02-02 17:28 osd.0.keyring
*/var/lib/ceph/tmp is exists.
root@ceph01:/var/log# ls -l /var/lib/ceph/
total 4
drwxrwxrwx 2 root root 4096 2011-11-11 09:28 tmp
*/etc/ceph/ceph.conf
[global]
auth supported = cephx
keyring = /etc/ceph/$name.keyring
[mon]
mon data = /data/data/mon$id
debug ms = 1
[mon.0]
host = ceph01
mon addr = 10.68.119.191:6789
[mds]
[mds.0]
host = ceph01
[osd]
osd data = /data/osd$id
osd journal = /data/osd$id/journal
osd journal size = 512
osd class tmp = /var/lib/ceph/tmp
[osd.0]
host = ceph01
btrfs devs = /dev/sdb1
Waiting for your reply,
Tomoya.
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
