Re: [ceph-users] Ceph + Xen - RBD io hang
The suspicious line in /var/log/debug (see the pastebin below) and that
'blkid' was the culprit keeping the device open looked like juicy clues:
kernel: vbd vbd-51728: 16 Device in use; refusing to close
Search results:
https://www.redhat.com/archives/libguestfs/2012-February/msg00023.html
https://rwmj.wordpress.com/2012/01/19/udev-unexpectedness/#content
blkid is started up by udevd after block device changes. The theory is
while blkid runs it holds the device open, and after pygrub, xenops
fails to close the device if blkid hasn't finished by then.
The above links suggest to avoid the condition by running 'udevadm
settle' before closing the device.
For a cheap hack, I added 'subprocess.call(("/sbin/udevadm", "settle"))'
at the end of the pygrub script.
Several VMs *in a row* successfully started with no failures, for the
first time since I pulled the host's root filesystem off the OSDs.
In terms of the udev theory, perhaps the extra load on the disk shared
by both OS and OSD was slowing things down enough that the race
condition was rarely triggered. After putting the OS on a separate
disk, the system was noticeably snappier, triggering this race condition
to where it was difficult to boot two VMs in a row (annoying when a
single failure required the machine to be power-cycled to recover!).
John
On 10/28/2013 02:01 PM, John Morris wrote:
> I'm encountering a problem with RBD-backed Xen. During a VM boot,
> pygrub attaches the VM's root VDI to dom0. This hangs with these
> messages in the debug log:
>
> Oct 27 21:19:59 xen27 kernel:
> vbd vbd-51728: 16 Device in use; refusing to close
> Oct 27 21:19:59 xen27 xenopsd-xenlight:
> [xenops] waiting for backend to close
> Oct 27 21:19:59 xen27 kernel:
> qemu-system-i38[2899]: segfault at 7fac042e4000 ip 7fac0447b129
> sp 7fffe7028630 error 4 in qemu-system-i386[7fac042ed000+309000]
>
> More details here:
>
> http://pastebin.ca/2472234
>
> - Scientific Linux 6
> - 64-bit, Phenom CPU
> - Ceph from RPM ceph-0.67.4-0.el6.x86_64
> - XenAPI from Dave Scott's technology preview
> - two btrfs-backed OSDs with journals on separate drives
> - various kernels, incl. 3.4.6 from Dave Scott's repo and 3.11.6
> from elrepo.org.
>
> This thread (whose Subject: I borrowed) describes what I'm seeing quite
> well, but no resolution was posted:
>
> http://comments.gmane.org/gmane.comp.file-systems.ceph.user/3636
>
> In my case, udevd starts a 'blkid' process that holds /dev/xvdb open.
> Like in James's case, any interaction with the device will hang, and
> usually can't be killed. This same problem prevents the machine from
> completing shutdown.
>
> In that thread, Sylvain Munaut says the OSD and kernel driver shouldn't
> be run in the same host. I believe my setup does not violate that,
> since the rbd kernel module is not loaded, and instead the device is
> attached through the xen_blkfront module instead.
>
> Thanks-
>
> John
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Hi James, That doesn't sound like a fun one to debug. I'll try your messaging stack size tweak after the current (super ugly) hack experiment, to be described next Thanks- John On 10/28/2013 11:11 PM, James Harper wrote: > Maybe nothing to do with your issue, but I was having problems using librbd > with blktap, and ended up adding: > > [client] > ms rwthread stack bytes = 8388608 > > to my config. This is a workaround, not a fix though (IMHO) as there is > nothing to indicate that librbd is running out of stack space, rather that > stack is being clobbered and this works around it. I spent a fair bit of time > trying to debug it but could never pin it down. > > James > >> -Original Message- >> From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users- >> boun...@lists.ceph.com] On Behalf Of John Morris >> Sent: Tuesday, 29 October 2013 6:01 AM >> To: ceph-users@lists.ceph.com >> Subject: [ceph-users] Ceph + Xen - RBD io hang >> >> I'm encountering a problem with RBD-backed Xen. During a VM boot, >> pygrub attaches the VM's root VDI to dom0. This hangs with these >> messages in the debug log: >> >> Oct 27 21:19:59 xen27 kernel: >> vbd vbd-51728: 16 Device in use; refusing to close >> Oct 27 21:19:59 xen27 xenopsd-xenlight: >> [xenops] waiting for backend to close >> Oct 27 21:19:59 xen27 kernel: >> qemu-system-i38[2899]: segfault at 7fac042e4000 ip 7fac0447b129 >> sp 7fffe7028630 error 4 in qemu-system-i386[7fac042ed000+309000] >> >> More details here: >> >> http://pastebin.ca/2472234 >> >> - Scientific Linux 6 >> - 64-bit, Phenom CPU >> - Ceph from RPM ceph-0.67.4-0.el6.x86_64 >> - XenAPI from Dave Scott's technology preview >> - two btrfs-backed OSDs with journals on separate drives >> - various kernels, incl. 3.4.6 from Dave Scott's repo and 3.11.6 >> from elrepo.org. >> >> This thread (whose Subject: I borrowed) describes what I'm seeing quite >> well, but no resolution was posted: >> >> http://comments.gmane.org/gmane.comp.file-systems.ceph.user/3636 >> >> In my case, udevd starts a 'blkid' process that holds /dev/xvdb open. >> Like in James's case, any interaction with the device will hang, and >> usually can't be killed. This same problem prevents the machine from >> completing shutdown. >> >> In that thread, Sylvain Munaut says the OSD and kernel driver shouldn't >> be run in the same host. I believe my setup does not violate that, >> since the rbd kernel module is not loaded, and instead the device is >> attached through the xen_blkfront module instead. >> >> Thanks- >> >> John >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Maybe nothing to do with your issue, but I was having problems using librbd with blktap, and ended up adding: [client] ms rwthread stack bytes = 8388608 to my config. This is a workaround, not a fix though (IMHO) as there is nothing to indicate that librbd is running out of stack space, rather that stack is being clobbered and this works around it. I spent a fair bit of time trying to debug it but could never pin it down. James > -Original Message- > From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users- > boun...@lists.ceph.com] On Behalf Of John Morris > Sent: Tuesday, 29 October 2013 6:01 AM > To: ceph-users@lists.ceph.com > Subject: [ceph-users] Ceph + Xen - RBD io hang > > I'm encountering a problem with RBD-backed Xen. During a VM boot, > pygrub attaches the VM's root VDI to dom0. This hangs with these > messages in the debug log: > > Oct 27 21:19:59 xen27 kernel: > vbd vbd-51728: 16 Device in use; refusing to close > Oct 27 21:19:59 xen27 xenopsd-xenlight: > [xenops] waiting for backend to close > Oct 27 21:19:59 xen27 kernel: > qemu-system-i38[2899]: segfault at 7fac042e4000 ip 7fac0447b129 > sp 7fffe7028630 error 4 in qemu-system-i386[7fac042ed000+309000] > > More details here: > > http://pastebin.ca/2472234 > > - Scientific Linux 6 > - 64-bit, Phenom CPU > - Ceph from RPM ceph-0.67.4-0.el6.x86_64 > - XenAPI from Dave Scott's technology preview > - two btrfs-backed OSDs with journals on separate drives > - various kernels, incl. 3.4.6 from Dave Scott's repo and 3.11.6 > from elrepo.org. > > This thread (whose Subject: I borrowed) describes what I'm seeing quite > well, but no resolution was posted: > > http://comments.gmane.org/gmane.comp.file-systems.ceph.user/3636 > > In my case, udevd starts a 'blkid' process that holds /dev/xvdb open. > Like in James's case, any interaction with the device will hang, and > usually can't be killed. This same problem prevents the machine from > completing shutdown. > > In that thread, Sylvain Munaut says the OSD and kernel driver shouldn't > be run in the same host. I believe my setup does not violate that, > since the rbd kernel module is not loaded, and instead the device is > attached through the xen_blkfront module instead. > > Thanks- > > John > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Hi John, On 10/28/2013 08:55 PM, John Wilkins wrote: > John, > > I've never installed anything on Scientific Linux. SL6 is binary compatible with other EL6 varieties. This host uses the same repos as CentOS for Dave Scott's xenserver-core technology preview: EPEL6, Dave's XenServer + Ceph technology preview, Dave's xenserver-core snapshot repo, and the ceph.com Dumpling repo. > Are you sure that QEMU has RBD support? > > I have some wip-doc text, which I'm going to move around shortly. You > can see the yum install requirements here: > > http://ceph.com/docs/wip-doc-install/install/yum-priorities/ > http://ceph.com/docs/wip-doc-install/install/install-qemu/#rpm-packages The SL6 qemu-img package does not have RBD support, but I did find those repos yesterday, and installed the RBD-capable qemu-img package from there. There are no other obvious qemu-related packages (as in 'rpm -qa | grep qemu') originating from SL6, and just one from Dave's repo, 'libvirt-daemon-driver-qemu'. > Make sure you can run QEMU commands on RBD first, since that's a dependency. > > http://ceph.com/docs/wip-doc-install/rbd/qemu-rbd/ Since installing the ceph.com package, the 'qemu-img -f rbd' commands work. Before, the SL6 version 'qemu-img -f rbd' commands complained about rbd being an unknown format. However, xapi was able to launch VMs both before and after the qemu-img package update, with no effect on the intermittent hangs. The 'xe vm-start' hanging problem is not shared by the 'xe vdi-*' commands (or the updated 'qemu-img -f rbd *' commands), which have no trouble manipulating rbd-backed vdis at all. Anything I might've missed in there? Thanks for the tips! John > On Mon, Oct 28, 2013 at 12:01 PM, John Morris wrote: >> I'm encountering a problem with RBD-backed Xen. During a VM boot, >> pygrub attaches the VM's root VDI to dom0. This hangs with these >> messages in the debug log: >> >> Oct 27 21:19:59 xen27 kernel: >> vbd vbd-51728: 16 Device in use; refusing to close >> Oct 27 21:19:59 xen27 xenopsd-xenlight: >> [xenops] waiting for backend to close >> Oct 27 21:19:59 xen27 kernel: >> qemu-system-i38[2899]: segfault at 7fac042e4000 ip 7fac0447b129 >> sp 7fffe7028630 error 4 in qemu-system-i386[7fac042ed000+309000] >> >> More details here: >> >> http://pastebin.ca/2472234 >> >> - Scientific Linux 6 >> - 64-bit, Phenom CPU >> - Ceph from RPM ceph-0.67.4-0.el6.x86_64 >> - XenAPI from Dave Scott's technology preview >> - two btrfs-backed OSDs with journals on separate drives >> - various kernels, incl. 3.4.6 from Dave Scott's repo and 3.11.6 >> from elrepo.org. >> >> This thread (whose Subject: I borrowed) describes what I'm seeing quite >> well, but no resolution was posted: >> >> http://comments.gmane.org/gmane.comp.file-systems.ceph.user/3636 >> >> In my case, udevd starts a 'blkid' process that holds /dev/xvdb open. >> Like in James's case, any interaction with the device will hang, and >> usually can't be killed. This same problem prevents the machine from >> completing shutdown. >> >> In that thread, Sylvain Munaut says the OSD and kernel driver shouldn't >> be run in the same host. I believe my setup does not violate that, >> since the rbd kernel module is not loaded, and instead the device is >> attached through the xen_blkfront module instead. >> >> Thanks- >> >> John >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
John, I've never installed anything on Scientific Linux. Are you sure that QEMU has RBD support? I have some wip-doc text, which I'm going to move around shortly. You can see the yum install requirements here: http://ceph.com/docs/wip-doc-install/install/yum-priorities/ http://ceph.com/docs/wip-doc-install/install/install-qemu/#rpm-packages Make sure you can run QEMU commands on RBD first, since that's a dependency. http://ceph.com/docs/wip-doc-install/rbd/qemu-rbd/ On Mon, Oct 28, 2013 at 12:01 PM, John Morris wrote: > I'm encountering a problem with RBD-backed Xen. During a VM boot, > pygrub attaches the VM's root VDI to dom0. This hangs with these > messages in the debug log: > > Oct 27 21:19:59 xen27 kernel: > vbd vbd-51728: 16 Device in use; refusing to close > Oct 27 21:19:59 xen27 xenopsd-xenlight: > [xenops] waiting for backend to close > Oct 27 21:19:59 xen27 kernel: > qemu-system-i38[2899]: segfault at 7fac042e4000 ip 7fac0447b129 > sp 7fffe7028630 error 4 in qemu-system-i386[7fac042ed000+309000] > > More details here: > > http://pastebin.ca/2472234 > > - Scientific Linux 6 > - 64-bit, Phenom CPU > - Ceph from RPM ceph-0.67.4-0.el6.x86_64 > - XenAPI from Dave Scott's technology preview > - two btrfs-backed OSDs with journals on separate drives > - various kernels, incl. 3.4.6 from Dave Scott's repo and 3.11.6 > from elrepo.org. > > This thread (whose Subject: I borrowed) describes what I'm seeing quite > well, but no resolution was posted: > > http://comments.gmane.org/gmane.comp.file-systems.ceph.user/3636 > > In my case, udevd starts a 'blkid' process that holds /dev/xvdb open. > Like in James's case, any interaction with the device will hang, and > usually can't be killed. This same problem prevents the machine from > completing shutdown. > > In that thread, Sylvain Munaut says the OSD and kernel driver shouldn't > be run in the same host. I believe my setup does not violate that, > since the rbd kernel module is not loaded, and instead the device is > attached through the xen_blkfront module instead. > > Thanks- > > John > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- John Wilkins Senior Technical Writer Intank john.wilk...@inktank.com (415) 425-9599 http://inktank.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph + Xen - RBD io hang
I'm encountering a problem with RBD-backed Xen. During a VM boot, pygrub attaches the VM's root VDI to dom0. This hangs with these messages in the debug log: Oct 27 21:19:59 xen27 kernel: vbd vbd-51728: 16 Device in use; refusing to close Oct 27 21:19:59 xen27 xenopsd-xenlight: [xenops] waiting for backend to close Oct 27 21:19:59 xen27 kernel: qemu-system-i38[2899]: segfault at 7fac042e4000 ip 7fac0447b129 sp 7fffe7028630 error 4 in qemu-system-i386[7fac042ed000+309000] More details here: http://pastebin.ca/2472234 - Scientific Linux 6 - 64-bit, Phenom CPU - Ceph from RPM ceph-0.67.4-0.el6.x86_64 - XenAPI from Dave Scott's technology preview - two btrfs-backed OSDs with journals on separate drives - various kernels, incl. 3.4.6 from Dave Scott's repo and 3.11.6 from elrepo.org. This thread (whose Subject: I borrowed) describes what I'm seeing quite well, but no resolution was posted: http://comments.gmane.org/gmane.comp.file-systems.ceph.user/3636 In my case, udevd starts a 'blkid' process that holds /dev/xvdb open. Like in James's case, any interaction with the device will hang, and usually can't be killed. This same problem prevents the machine from completing shutdown. In that thread, Sylvain Munaut says the OSD and kernel driver shouldn't be run in the same host. I believe my setup does not violate that, since the rbd kernel module is not loaded, and instead the device is attached through the xen_blkfront module instead. Thanks- John ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Hi, On wo, 2013-08-28 at 10:10 +0200, Sylvain Munaut wrote: > Hi, > > > I am doing some experimentation with Ceph and Xen (on the same host) and I'm > > experiencing some problems with the rbd device that I'm using as the block > > device. My environment is: > > AFAIK running the OSD and the kernel driver inside the same host is > not recommended at all ... I can confirm that this seems to be the case, with RBD as well as CephFS. As soon as you use the kernel-based driver to access them on the same node as where the OSD data-dirs are mounted for that cluster, things slow to a crawl. For CephFS, using the Fuse-driver solves that problem. For RBD, you want to avoid going through the kernel-based block-driver. There is a technical reason for this, but I'll leave that for others to explain. > > Personally I run storage node and compute node on the same physical > hosts, but the OSD is isolated into a DomU to avoid the possible > 'paging / memory' issue of running rbd.ko and osd on the same host. > (here they have pinned cpu and memory, so no interactions). I, too, run compute and shared storage on the same physical hardware on some clusters. Works for me, even under reasonably high loading. I use RBD directly from KVM, which avoids the host-OS kernel and therefore your problem altogether. This also avoids runaway-conditions on the nodes themselves under unexpected overload or failure of the shared storage, with the VMs usually just blocking until the problem is cleared and then continuing without any I/O-errors, with the host-OS remaining workable too. Regards, Oliver ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Le mercredi 28 août 2013 à 10:07 +0200, Sylvain Munaut a écrit : > Hi, > > > I use Ceph 0.61.8 and Xen 4.2.2 (Debian) in production, and can't use > > kernel 3.10.* on dom0, which hang very soon. But it's visible in kernel > > logs of the dom0, not the domU. > > Weird. I'm using 3.10.0 without issue here. What's the issue you're hitting ? > > Cheers, > > Sylvain > Hi, this one : http://tracker.ceph.com/issues/5760 it seems to be related to snapshots. Olivier ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Hi, > I am doing some experimentation with Ceph and Xen (on the same host) and I'm > experiencing some problems with the rbd device that I'm using as the block > device. My environment is: AFAIK running the OSD and the kernel driver inside the same host is not recommended at all ... Personally I run storage node and compute node on the same physical hosts, but the OSD is isolated into a DomU to avoid the possible 'paging / memory' issue of running rbd.ko and osd on the same host. (here they have pinned cpu and memory, so no interactions). Alternatively you could give the xen blktap rbd userspace driver a shot. Cheers, Sylvain ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Hi, > I use Ceph 0.61.8 and Xen 4.2.2 (Debian) in production, and can't use > kernel 3.10.* on dom0, which hang very soon. But it's visible in kernel > logs of the dom0, not the domU. Weird. I'm using 3.10.0 without issue here. What's the issue you're hitting ? Cheers, Sylvain ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Sage Weil wrote:
Can you post the contents of the hung task warning so we can see where it
is stuck?
The messages in the domU are:
[35284.544137] INFO: task dd:4903 blocked for more than 120 seconds.
[35284.544148] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[35284.544312] INFO: task flush-202:0:4904 blocked for more than 120
seconds.
[35284.544315] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[35284.544450] INFO: task blkid:4906 blocked for more than 120 seconds.
[35284.544453] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
As per the suggestion of Olivier I tried with 3.9.11 for dom0 but with
this kernel I didn't get any writes to succeed from the domU at all.
The complete set of commands that I executed are given below.
Regards,
James
dom0# uname -r
3.9.11
dom0# xl info
...
xen_major : 4
xen_minor : 3
xen_extra : .0
...
dom0# rbd --version
ceph version 0.67.2 (eb4380dd036a0b644c6283869911d615ed729ac8)
dom0# ceph status
cluster b85e8b00-baca-4d21-95d6-4b45f652dfd9
health HEALTH_OK
monmap e1: 1 mons at {a=ip-this-node:6789/0}, election epoch 1,
quorum 0 a
osdmap e1590: 4 osds: 4 up, 4 in
pgmap v43252: 576 pgs: 576 active+clean; 33416 MB data, 72315 MB
used, 1971 GB / 2048 GB avail
mdsmap e30676: 1/1/1 up {0=a=up:active}
dom0# mount | grep osd
/dev/mapper/sysvg-ceph_osd0 on /var/lib/ceph/osd/ceph-0 type btrfs
(rw,noatime)
/dev/mapper/sysvg-ceph_osd1 on /var/lib/ceph/osd/ceph-1 type btrfs
(rw,noatime)
dom0# rbd create --size 32768 --image-format 1 ubuntu-test
dom0# rbd info ubuntu-test
rbd image 'ubuntu-test':
size 32768 MB in 8192 objects
order 22 (4096 KB objects)
block_name_prefix: rb.0.422a.2ae8944a
format: 1
dom0# rbd map rbd/ubuntu-test
dom0# rbd showmapped
id pool image snap device
1 rbd ubuntu-test -/dev/rbd1
dom0# for i in $(seq 0 1023) ; do echo $i ; dd if=/dev/rbd1 of=/dev/null
bs=1M count=64 skip=$(($i * 4)) ; done
...
1023
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.430964 s, 156 MB/s
dom0# for i in $(seq 0 1023) ; do echo $i ; dd if=/dev/zero of=/dev/rbd1
bs=1M count=64 seek=$(($i * 4)) ; done
...
1023
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.0559911 s, 1.2 GB/s
dom0# xl create pv_ceph.cfg
dom0# for i in $(seq 0 1023) ; do echo $i ; dd if=/dev/rbd1 of=/dev/null
bs=1M count=64 skip=$(($i * 4)) ; done
...
1023
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.0651528 s, 1.0 GB/s
dom0# for i in $(seq 0 1023) ; do echo $i ; dd if=/dev/zero of=/dev/rbd1
bs=1M count=64 seek=$(($i * 4)) ; done
...
1023
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.705422 s, 95.1 MB/s
# The last two tests are to show that the rbd device is
readable/writeable while passed through the domU until the domU tries a
write
domU# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 13.04
Release:13.04
Codename: raring
domU# uname -r
3.8.0-19-generic
domU# for i in $(seq 0 1023) ; do echo $i ; dd if=/dev/xvda of=/dev/null
bs=1M count=64 skip=$(($i * 4)) ; done
...
1023
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 1.1602 s, 57.8 MB/s
# The domU has no problems reading from the virtual block device
domU# for i in $(seq 0 1023) ; do echo $i ; dd if=/dev/zero of=/dev/xvda
bs=1M count=64 seek=$(($i * 4)) ; done
0
[35284.544137] INFO: task dd:4903 blocked for more than 120 seconds.
[35284.544148] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[35284.544312] INFO: task flush-202:0:4904 blocked for more than 120
seconds.
[35284.544315] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[35284.544450] INFO: task blkid:4906 blocked for more than 120 seconds.
[35284.544453] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
# repeated every two minutes, a similar test with a 3.10.9 dom0 usually
managed a small number of iterations of the loop
dom0# dmesg | tail
# these messages start before trying the write test in domU but perhaps
they are important
[43395.355813] libceph: osd3 ip-other-node:6804 socket closed (con state
OPEN)
[43395.358203] rbd: obj_request 8800474793c0 was already done
[44295.663754] libceph: osd3 ip-other-node:6804 socket closed (con state
OPEN)
[44295.666140] rbd: obj_request 8800474793c0 was already done
[45195.971648] libceph: osd3 ip-other-node:6804 socket closed (con state
OPEN)
[45195.974044] rbd: obj_request 8800474793c0 was already done
dom0# for i in $(seq 0 1023) ; do echo $i ; dd if=/dev/rbd1 of=/dev/null
bs=1M count=64 skip=$(($i * 4)) ; done
# dd is now hung, kernel debug options are not enabled in the config but
I would expect
# similar messages to domU if they were. I can recompile the dom0
kernel to includ
Re: [ceph-users] Ceph + Xen - RBD io hang
Hi James, Can you post the contents of the hung task warning so we can see where it is stuck? Thanks! sage On Tue, 27 Aug 2013, James Dingwall wrote: > Hi, > > I am doing some experimentation with Ceph and Xen (on the same host) and I'm > experiencing some problems with the rbd device that I'm using as the block > device. My environment is: > > 2 node Ceph 0.67.2 cluster, 4x OSD (btrfs) and 1x mon > Xen 4.3.0 > Kernel 3.10.9 > > The domU I'm trying to build is from the Ubuntu 13.04 desktop release. When I > pass through the rbd (format 1 or 2) device as phy:/dev/rbd/rbd/ubuntu-test > then the domU has no problems reading data from it, the test I ran was: > > for i in $(seq 0 1023) ; do > dd if=/dev/xvda of=/dev/null bs=4k count=1024 skip=$(($i * 4)) > done > > However writing data causes the domU to hang while while i is still in single > figures but it doesn't seem consistent about the exact value. > for i in $(seq 0 1023) ; do > dd of=/dev/xvda of=/dev/zero bs=4k count=1024 seek=$(($i * 4)) > done > > eventually the kernel in the domU will print a hung task warning. I have > tried the domU as pv and hvm (with xen_platform_pci = 1 and 0) but have the > same behaviour in both cases. Once this state is triggered on the rbd device > then any interaction with it in dom0 will result in the same hang. I'm > assuming that there is some unfavourable interaction between ceph/rbd and > blkback but I haven't found anything in the dom0 logs so I would like to know > if anyone has some suggestions about where to start trying to hunt this down. > > Thanks, > James > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph + Xen - RBD io hang
Hi, I use Ceph 0.61.8 and Xen 4.2.2 (Debian) in production, and can't use kernel 3.10.* on dom0, which hang very soon. But it's visible in kernel logs of the dom0, not the domU. Anyway, you should probably re-try with kernel 3.9.11 for the dom0 (I also use 3.10.9 in domU). Olivier Le mardi 27 août 2013 à 11:46 +0100, James Dingwall a écrit : > Hi, > > I am doing some experimentation with Ceph and Xen (on the same host) and > I'm experiencing some problems with the rbd device that I'm using as the > block device. My environment is: > > 2 node Ceph 0.67.2 cluster, 4x OSD (btrfs) and 1x mon > Xen 4.3.0 > Kernel 3.10.9 > > The domU I'm trying to build is from the Ubuntu 13.04 desktop release. > When I pass through the rbd (format 1 or 2) device as > phy:/dev/rbd/rbd/ubuntu-test then the domU has no problems reading data > from it, the test I ran was: > > for i in $(seq 0 1023) ; do > dd if=/dev/xvda of=/dev/null bs=4k count=1024 skip=$(($i * 4)) > done > > However writing data causes the domU to hang while while i is still in > single figures but it doesn't seem consistent about the exact value. > for i in $(seq 0 1023) ; do > dd of=/dev/xvda of=/dev/zero bs=4k count=1024 seek=$(($i * 4)) > done > > eventually the kernel in the domU will print a hung task warning. I > have tried the domU as pv and hvm (with xen_platform_pci = 1 and 0) but > have the same behaviour in both cases. Once this state is triggered on > the rbd device then any interaction with it in dom0 will result in the > same hang. I'm assuming that there is some unfavourable interaction > between ceph/rbd and blkback but I haven't found anything in the dom0 > logs so I would like to know if anyone has some suggestions about where > to start trying to hunt this down. > > Thanks, > James > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph + Xen - RBD io hang
Hi, I am doing some experimentation with Ceph and Xen (on the same host) and I'm experiencing some problems with the rbd device that I'm using as the block device. My environment is: 2 node Ceph 0.67.2 cluster, 4x OSD (btrfs) and 1x mon Xen 4.3.0 Kernel 3.10.9 The domU I'm trying to build is from the Ubuntu 13.04 desktop release. When I pass through the rbd (format 1 or 2) device as phy:/dev/rbd/rbd/ubuntu-test then the domU has no problems reading data from it, the test I ran was: for i in $(seq 0 1023) ; do dd if=/dev/xvda of=/dev/null bs=4k count=1024 skip=$(($i * 4)) done However writing data causes the domU to hang while while i is still in single figures but it doesn't seem consistent about the exact value. for i in $(seq 0 1023) ; do dd of=/dev/xvda of=/dev/zero bs=4k count=1024 seek=$(($i * 4)) done eventually the kernel in the domU will print a hung task warning. I have tried the domU as pv and hvm (with xen_platform_pci = 1 and 0) but have the same behaviour in both cases. Once this state is triggered on the rbd device then any interaction with it in dom0 will result in the same hang. I'm assuming that there is some unfavourable interaction between ceph/rbd and blkback but I haven't found anything in the dom0 logs so I would like to know if anyone has some suggestions about where to start trying to hunt this down. Thanks, James ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
