Re: [Qemu-devel] [PATCH v3 00/10] curl: fix curl read
On Tue, May 21, 2013 at 09:54:15AM +0800, Fam Zheng wrote: On Mon, 05/20 09:49, Richard W.M. Jones wrote: On Mon, May 20, 2013 at 09:41:06AM +0100, Richard W.M. Jones wrote: On Mon, May 20, 2013 at 03:03:34PM +0800, Fam Zheng wrote: CURL library API has changed, the current curl driver is not working. This patch rewrites the use of API as well as the structure of internal states. I tried this, but it segfaults: Program terminated with signal 11, Segmentation fault. That stack trace was wrong. I was testing against the version of libcurl in Fedora which is known to be broken. Here is the stack trace, this time really running against curl-7_30_0-147-gae26ee3: Program terminated with signal 11, Segmentation fault. #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f63d48ba340) at block/curl.c:240 240 size_t aio_base = acb-sector_num * SECTOR_SIZE; Looks like a memory corrupt (QLIST head is invalid pointer). But I can't reproduce here with your steps. Can you try qemu-io? $LD_LIBRARY_PATH=~/d/curl/lib/.libs ~/d/qemu/qemu-io http://192.168.0.249/scratch/winxp.img -c 'read 0 512' This command is successful: $ LD_LIBRARY_PATH=~/d/curl/lib/.libs ~/d/qemu/qemu-io http://192.168.0.249/scratch/winxp.img -c 'read 0 512' read 512/512 bytes at offset 0 512 bytes, 1 ops; 0. sec (32.552 MiB/sec and 6.6667 ops/sec) $ echo $? 0 Here's another go with guestfish: $ ulimit -c unlimited $ LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 LIBGUESTFS_BACKEND=direct LIBGUESTFS_QEMU=~/d/qemu/qemu.wrapper LD_LIBRARY_PATH=~/d/curl/lib/.libs PATH=~/d/qemu:$PATH ./run ./fish/guestfish -a http://192.168.0.249/scratch/winxp.img -i [...] [00159ms] /home/rjones/d/qemu/qemu.wrapper \ -global virtio-blk-pci.scsi=off \ -nodefconfig \ -nodefaults \ -nographic \ -device virtio-scsi-pci,id=scsi \ -drive file=http://192.168.0.249/scratch/winxp.img,id=hd0,if=none \ -device scsi-hd,drive=hd0 \ -drive file=/home/rjones/d/libguestfs/tmp/.guestfs-1000/root.15535,snapshot=on,id=appliance,if=none,cache=unsafe \ -device scsi-hd,drive=appliance \ -machine accel=kvm:tcg \ -m 500 \ -no-reboot \ -no-hpet \ -device virtio-serial \ -serial stdio \ -device sga \ -chardev socket,path=/home/rjones/d/libguestfs/tmp/libguestfsk9fu9P/guestfsd.sock,id=channel0 \ -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \ -kernel /home/rjones/d/libguestfs/tmp/.guestfs-1000/kernel.15535 \ -initrd /home/rjones/d/libguestfs/tmp/.guestfs-1000/initrd.15535 \ -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color'libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages libguestfs: child_cleanup: 0x1db0090: child process died libguestfs: sending SIGTERM to process 15600 libguestfs: error: /home/rjones/d/qemu/qemu.wrapper killed by signal 11 (Segmentation fault), see debug messages above libguestfs: error: guestfs_launch failed, see earlier error messages libguestfs: trace: launch = -1 (error) [...] $ file /tmp/core.15600 /tmp/core.15600: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/home/rjones/d/qemu/x86_64-softmmu/qemu-system-x86_64 -L /home/rjones/d/qemu/pc' $ gdb /home/rjones/d/qemu/x86_64-softmmu/qemu-system-x86_64 /tmp/core.15600 [stack trace is the same as before] #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f4d3c769360) at block/curl.c:240 240 size_t aio_base = acb-sector_num * SECTOR_SIZE; (gdb) print acb $1 = (CURLAIOCB *) 0x7575757575757575 Looks like use-after-free? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Re: [Qemu-devel] [PATCH v3 00/10] curl: fix curl read
On Tue, 05/21 08:39, Richard W.M. Jones wrote: On Tue, May 21, 2013 at 09:54:15AM +0800, Fam Zheng wrote: On Mon, 05/20 09:49, Richard W.M. Jones wrote: On Mon, May 20, 2013 at 09:41:06AM +0100, Richard W.M. Jones wrote: On Mon, May 20, 2013 at 03:03:34PM +0800, Fam Zheng wrote: CURL library API has changed, the current curl driver is not working. This patch rewrites the use of API as well as the structure of internal states. I tried this, but it segfaults: Program terminated with signal 11, Segmentation fault. That stack trace was wrong. I was testing against the version of libcurl in Fedora which is known to be broken. Here is the stack trace, this time really running against curl-7_30_0-147-gae26ee3: Program terminated with signal 11, Segmentation fault. #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f63d48ba340) at block/curl.c:240 240 size_t aio_base = acb-sector_num * SECTOR_SIZE; Looks like a memory corrupt (QLIST head is invalid pointer). But I can't reproduce here with your steps. Can you try qemu-io? $LD_LIBRARY_PATH=~/d/curl/lib/.libs ~/d/qemu/qemu-io http://192.168.0.249/scratch/winxp.img -c 'read 0 512' This command is successful: $ LD_LIBRARY_PATH=~/d/curl/lib/.libs ~/d/qemu/qemu-io http://192.168.0.249/scratch/winxp.img -c 'read 0 512' read 512/512 bytes at offset 0 512 bytes, 1 ops; 0. sec (32.552 MiB/sec and 6.6667 ops/sec) $ echo $? 0 Here's another go with guestfish: $ ulimit -c unlimited $ LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 LIBGUESTFS_BACKEND=direct LIBGUESTFS_QEMU=~/d/qemu/qemu.wrapper LD_LIBRARY_PATH=~/d/curl/lib/.libs PATH=~/d/qemu:$PATH ./run ./fish/guestfish -a http://192.168.0.249/scratch/winxp.img -i [...] [00159ms] /home/rjones/d/qemu/qemu.wrapper \ -global virtio-blk-pci.scsi=off \ -nodefconfig \ -nodefaults \ -nographic \ -device virtio-scsi-pci,id=scsi \ -drive file=http://192.168.0.249/scratch/winxp.img,id=hd0,if=none \ -device scsi-hd,drive=hd0 \ -drive file=/home/rjones/d/libguestfs/tmp/.guestfs-1000/root.15535,snapshot=on,id=appliance,if=none,cache=unsafe \ -device scsi-hd,drive=appliance \ -machine accel=kvm:tcg \ -m 500 \ -no-reboot \ -no-hpet \ -device virtio-serial \ -serial stdio \ -device sga \ -chardev socket,path=/home/rjones/d/libguestfs/tmp/libguestfsk9fu9P/guestfsd.sock,id=channel0 \ -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \ -kernel /home/rjones/d/libguestfs/tmp/.guestfs-1000/kernel.15535 \ -initrd /home/rjones/d/libguestfs/tmp/.guestfs-1000/initrd.15535 \ -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color'libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages libguestfs: child_cleanup: 0x1db0090: child process died libguestfs: sending SIGTERM to process 15600 libguestfs: error: /home/rjones/d/qemu/qemu.wrapper killed by signal 11 (Segmentation fault), see debug messages above libguestfs: error: guestfs_launch failed, see earlier error messages libguestfs: trace: launch = -1 (error) [...] $ file /tmp/core.15600 /tmp/core.15600: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/home/rjones/d/qemu/x86_64-softmmu/qemu-system-x86_64 -L /home/rjones/d/qemu/pc' $ gdb /home/rjones/d/qemu/x86_64-softmmu/qemu-system-x86_64 /tmp/core.15600 [stack trace is the same as before] #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f4d3c769360) at block/curl.c:240 240 size_t aio_base = acb-sector_num * SECTOR_SIZE; (gdb) print acb $1 = (CURLAIOCB *) 0x7575757575757575 Looks like use-after-free? Yes, thank you a lot. Will post another version to fix this. -- Fam
Re: [Qemu-devel] [PATCH v3 00/10] curl: fix curl read
On Mon, May 20, 2013 at 03:03:34PM +0800, Fam Zheng wrote: CURL library API has changed, the current curl driver is not working. This patch rewrites the use of API as well as the structure of internal states. I tried this, but it segfaults: Program terminated with signal 11, Segmentation fault. #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f09d2975340) at block/curl.c:240 240 size_t aio_base = acb-sector_num * SECTOR_SIZE; Missing separate debuginfos, use: debuginfo-install SDL-1.2.15-3.fc18.x86_64 bluez-libs-4.101-6.fc18.x86_64 brlapi-0.5.6-12.fc18.x86_64 celt051-0.5.1.3-5.fc18.x86_64 ceph-devel-0.56.3-1.fc18.x86_64 ceph-libs-0.56.3-1.fc18.x86_64 cryptopp-5.6.2-2.fc18.x86_64 cyrus-sasl-lib-2.1.25-2.fc18.x86_64 leveldb-1.7.0-4.fc18.x86_64 libfdt-1.3.0-5.fc18.x86_64 libseccomp-1.0.1-0.fc18.x86_64 libselinux-2.1.12-7.3.fc18.x86_64 libusbx-1.0.14-1.fc18.x86_64 nss-mdns-0.10-11.fc18.x86_64 snappy-1.0.5-2.fc18.x86_64 spice-server-0.12.2-3.fc18.x86_64 usbredir-0.6-1.fc18.x86_64 xen-libs-4.2.2-3.fc18.x86_64 (gdb) bt #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f09d2975340) at block/curl.c:240 #1 0x7f09cc7ee0e8 in Curl_client_write (conn=conn@entry=0x7f09d2996c80, type=type@entry=1, ptr=0x7f09d298e8f0 , len=2046) at sendf.c:449 #2 0x7f09cc801c52 in readwrite_data (done=0x770dac77, didwhat=synthetic pointer, k=0x7f09d298e080, conn=0x7f09d2996c80, data= 0x7f09d298e050) at transfer.c:705 #3 Curl_readwrite (conn=0x7f09d2996c80, done=done@entry=0x770dac77) at transfer.c:1023 #4 0x7f09cc80a4d2 in multi_runsingle (multi=multi@entry=0x7f09d29815b0, now=..., easy=0x7f09d29756a0) at multi.c:1430 #5 0x7f09cc80b559 in multi_socket (multi=multi@entry=0x7f09d29815b0, checkall=checkall@entry=false, s=10, ev_bitmask=3, running_handles=running_handles@entry=0x770dad88) at multi.c:2140 #6 0x7f09cc80b64f in curl_multi_socket_action (multi_handle= 0x7f09d29815b0, s=optimized out, ev_bitmask=optimized out, running_handles=running_handles@entry=0x770dad88) at multi.c:2258 #7 0x7f09d0077043 in curl_fd_handler (arg=0x7f09d2997410) at block/curl.c:265 #8 0x7f09d005abbb in aio_dispatch (ctx=0x7f09d296e3f0, ctx=0x7f09d296e3f0) at aio-posix.c:149 #9 0x7f09d005b0b1 in aio_poll (ctx=0x7f09d296e3f0, blocking=blocking@entry=true) at aio-posix.c:248 #10 0x7f09d019f9c9 in qemu_aio_wait () at main-loop.c:484 #11 0x7f09d0070b65 in bdrv_rwv_co (bs=bs@entry=0x7f09d29746e0, sector_num=sector_num@entry=0, qiov=qiov@entry=0x770daea0, is_write=is_write@entry=false) at block.c:2215 #12 0x7f09d0070c90 in bdrv_rw_co (is_write=false, nb_sectors=optimized out, buf=optimized out, sector_num=0, bs= 0x7f09d29746e0) at block.c:2234 #13 bdrv_read (bs=bs@entry=0x7f09d29746e0, sector_num=sector_num@entry=0, buf=buf@entry=0x770db190 3\300\216м, nb_sectors=nb_sectors@entry=4) at block.c:2241 #14 0x7f09d0070d82 in bdrv_pread (bs=0x7f09d29746e0, offset=offset@entry= 0, buf=buf@entry=0x770db190, count1=count1@entry=2048) at block.c:2303 #15 0x7f09d0071460 in find_image_format (pdrv=synthetic pointer, filename=0x7f09d2971b40 http://192.168.0.249/scratch/winxp.img;, bs=optimized out) at block.c:533 #16 bdrv_open (bs=0x7f09d2971cc0, filename=filename@entry= 0x7f09d2971b40 http://192.168.0.249/scratch/winxp.img;, options= 0x7f09d29726a0, options@entry=0x7f09d2970980, flags=8258, drv=drv@entry= 0x0) at block.c:1047 #17 0x7f09d00a4093 in drive_init (all_opts=0x7f09d296d420, block_default_type=IF_NONE) at blockdev.c:698 #18 0x7f09d020e54b in drive_init_func (opts=optimized out, opaque=optimized out) at vl.c:1117 #19 0x7f09d03386f3 in qemu_opts_foreach (list=optimized out, func=func@entry=0x7f09d020e530 drive_init_func, opaque=opaque@entry= 0x7f09d07318f0 pc_i440fx_machine_v1_5+48, abort_on_failure=abort_on_failure@entry=1) at util/qemu-option.c:1162 #20 0x7f09d0055b89 in main (argc=optimized out, argv=optimized out, envp=optimized out) at vl.c:4201 I am using qemu from git + your patches, and curl-7_30_0-147-gae26ee3. - - - - A second, unrelated problem with the curl driver is to do with the way it resolves hostnames. I have a host which has IPv6 and IPv4 records in DNS (ie. and A). For some reason the IPv6 address isn't reachable, but this doesn't matter for most clients since they fall back to using the IPv4 address after a brief timeout. However the qemu curl driver does *not* fallback. It gives up after the IPv6 address fails. It'd be nice if this problem could be fixed too. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc.
Re: [Qemu-devel] [PATCH v3 00/10] curl: fix curl read
On Mon, May 20, 2013 at 09:41:06AM +0100, Richard W.M. Jones wrote: On Mon, May 20, 2013 at 03:03:34PM +0800, Fam Zheng wrote: CURL library API has changed, the current curl driver is not working. This patch rewrites the use of API as well as the structure of internal states. I tried this, but it segfaults: Program terminated with signal 11, Segmentation fault. That stack trace was wrong. I was testing against the version of libcurl in Fedora which is known to be broken. Here is the stack trace, this time really running against curl-7_30_0-147-gae26ee3: Program terminated with signal 11, Segmentation fault. #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f63d48ba340) at block/curl.c:240 240 size_t aio_base = acb-sector_num * SECTOR_SIZE; Missing separate debuginfos, use: debuginfo-install SDL-1.2.15-3.fc18.x86_64 bluez-libs-4.101-6.fc18.x86_64 brlapi-0.5.6-12.fc18.x86_64 celt051-0.5.1.3-5.fc18.x86_64 ceph-devel-0.56.3-1.fc18.x86_64 ceph-libs-0.56.3-1.fc18.x86_64 cryptopp-5.6.2-2.fc18.x86_64 cyrus-sasl-lib-2.1.25-2.fc18.x86_64 leveldb-1.7.0-4.fc18.x86_64 libfdt-1.3.0-5.fc18.x86_64 libseccomp-1.0.1-0.fc18.x86_64 libselinux-2.1.12-7.3.fc18.x86_64 libusbx-1.0.14-1.fc18.x86_64 nss-mdns-0.10-11.fc18.x86_64 snappy-1.0.5-2.fc18.x86_64 spice-server-0.12.2-3.fc18.x86_64 usbredir-0.6-1.fc18.x86_64 xen-libs-4.2.2-3.fc18.x86_64 (gdb) bt #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f63d48ba340) at block/curl.c:240 #1 0x7f63cef51cc8 in Curl_client_write () from /home/rjones/d/curl/lib/.libs/libcurl.so.4 #2 0x7f63cef697ef in Curl_readwrite () from /home/rjones/d/curl/lib/.libs/libcurl.so.4 #3 0x7f63cef710b0 in multi_runsingle () from /home/rjones/d/curl/lib/.libs/libcurl.so.4 #4 0x7f63cef720f7 in multi_socket () from /home/rjones/d/curl/lib/.libs/libcurl.so.4 #5 0x7f63cef721df in curl_multi_socket_action () from /home/rjones/d/curl/lib/.libs/libcurl.so.4 #6 0x7f63d27d9043 in curl_fd_handler (arg=0x7f63d48e16c0) at block/curl.c:265 #7 0x7f63d27bcbbb in aio_dispatch (ctx=0x7f63d48b33f0, ctx=0x7f63d48b33f0) at aio-posix.c:149 #8 0x7f63d27bd0b1 in aio_poll (ctx=0x7f63d48b33f0, blocking=blocking@entry=true) at aio-posix.c:248 #9 0x7f63d29019c9 in qemu_aio_wait () at main-loop.c:484 #10 0x7f63d27d2b65 in bdrv_rwv_co (bs=bs@entry=0x7f63d48b96e0, sector_num=sector_num@entry=0, qiov=qiov@entry=0x7fff2c127e10, is_write=is_write@entry=false) at block.c:2215 #11 0x7f63d27d2c90 in bdrv_rw_co (is_write=false, nb_sectors=optimized out, buf=optimized out, sector_num=0, bs= 0x7f63d48b96e0) at block.c:2234 #12 bdrv_read (bs=bs@entry=0x7f63d48b96e0, sector_num=sector_num@entry=0, buf=buf@entry=0x7fff2c128100 3\300\216м, nb_sectors=nb_sectors@entry=4) at block.c:2241 #13 0x7f63d27d2d82 in bdrv_pread (bs=0x7f63d48b96e0, offset=offset@entry= 0, buf=buf@entry=0x7fff2c128100, count1=count1@entry=2048) at block.c:2303 #14 0x7f63d27d3460 in find_image_format (pdrv=synthetic pointer, filename=0x7f63d48b6b40 http://192.168.0.249/scratch/winxp.img;, bs=optimized out) at block.c:533 #15 bdrv_open (bs=0x7f63d48b6cc0, filename=filename@entry= 0x7f63d48b6b40 http://192.168.0.249/scratch/winxp.img;, options= 0x7f63d48b76a0, options@entry=0x7f63d48b5980, flags=8258, drv=drv@entry= 0x0) at block.c:1047 #16 0x7f63d2806093 in drive_init (all_opts=0x7f63d48b2420, block_default_type=IF_NONE) at blockdev.c:698 #17 0x7f63d297054b in drive_init_func (opts=optimized out, opaque=optimized out) at vl.c:1117 #18 0x7f63d2a9a6f3 in qemu_opts_foreach (list=optimized out, func=func@entry=0x7f63d2970530 drive_init_func, opaque=opaque@entry= 0x7f63d2e938f0 pc_i440fx_machine_v1_5+48, abort_on_failure=abort_on_failure@entry=1) at util/qemu-option.c:1162 #19 0x7f63d27b7b89 in main (argc=optimized out, argv=optimized out, envp=optimized out) at vl.c:4201 --- For completeness: (1) qemu from git (without your patches) works. (2) I'm testing using the following command: $ LD_LIBRARY_PATH=~/d/curl/lib/.libs \ LIBGUESTFS_BACKEND=direct \ LIBGUESTFS_QEMU=~/d/qemu/qemu.wrapper \ ./run ./fish/guestfish -a http://192.168.0.249/scratch/winxp.img -i -v where: (a) ~/d/libguestfs contains libguestfs from git (b) ~/d/curl contains curl-7_30_0-147-gae26ee3 (c) http://192.168.0.249/scratch/winxp.img is a Windows XP image (d) qemu.wrapper is: -- #!/bin/sh - qemudir=/home/rjones/d/qemu exec $qemudir/x86_64-softmmu/qemu-system-x86_64 -L $qemudir/pc-bios $@ -- Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers.
Re: [Qemu-devel] [PATCH v3 00/10] curl: fix curl read
On Mon, 05/20 09:49, Richard W.M. Jones wrote: On Mon, May 20, 2013 at 09:41:06AM +0100, Richard W.M. Jones wrote: On Mon, May 20, 2013 at 03:03:34PM +0800, Fam Zheng wrote: CURL library API has changed, the current curl driver is not working. This patch rewrites the use of API as well as the structure of internal states. I tried this, but it segfaults: Program terminated with signal 11, Segmentation fault. That stack trace was wrong. I was testing against the version of libcurl in Fedora which is known to be broken. Here is the stack trace, this time really running against curl-7_30_0-147-gae26ee3: Program terminated with signal 11, Segmentation fault. #0 curl_read_cb (ptr=optimized out, size=optimized out, nmemb=optimized out, opaque=0x7f63d48ba340) at block/curl.c:240 240 size_t aio_base = acb-sector_num * SECTOR_SIZE; Looks like a memory corrupt (QLIST head is invalid pointer). But I can't reproduce here with your steps. Can you try qemu-io? $LD_LIBRARY_PATH=~/d/curl/lib/.libs ~/d/qemu/qemu-io http://192.168.0.249/scratch/winxp.img -c 'read 0 512' Thanks. -- Fam
