Bug#616159: qemu-kvm: SEGV inside vnc tight encoding

2011-03-04 Thread Philippe Latu 
Hello,

My post was not relevant. sorry for the noise.

I am going to file a new bug.

-- 
- Philippe Latu
philippe.latu(at)linux-france.org
http://www.linux-france.org/~platu
(GPG|PGP) KeyId 0x742A6424 - http://www.linux-france.org/~platu/key.asc


signature.asc
Description: This is a digitally signed message part.

Bug#616159: qemu-kvm: SEGV inside vnc tight encoding

2011-03-03 Thread Roland Dreier 
Philippe, I believe your problems are a separate bug (which I think is a
bad interaction between the qemu VGA BIOS and grub2).



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Bug#616159: qemu-kvm: SEGV inside vnc tight encoding

2011-03-03 Thread Roland Dreier 
I think I've found the bug.  The attached patch (already sent upstream
at http://lists.nongnu.org/archive/html/qemu-devel/2011-03/msg00270.html
and http://patchwork.ozlabs.org/patch/85359/) should fix the problem
(which occurs after 2GB of network traffic on a tight-compressed VNC
connection)

>From e51c7b1ab05d4a6fe4d153b2769290d37e077479 Mon Sep 17 00:00:00 2001
From: Roland Dreier 
Date: Thu, 03 Mar 2011 16:39:35 -0800
Subject: [PATCH] vnc: tight: Fix crash after 2GB of output

If one leaves a VNC session with tight compression running for long
enough, Qemu crashes.  This is because of the computation

bytes = zstream->total_out - previous_out;

in tight_compress_data, where zstream->total_out is a uLong but
previous_out is an int.  As soon as zstream->total_out gets past
INT_MAX (ie 2GB), previous_out becomes negative and therefore the
result of the subtraction, bytes, becomes a huge positive number that
causes havoc for obvious reasons when passed as a length to
vnc_write().

The fix for this is simple: keep previous_out as a uLong too, which
avoids any problems with sign conversion or truncation.

Signed-off-by: Roland Dreier 
---
 ui/vnc-enc-tight.c |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c
index af45edd..59ec0e3 100644
--- a/ui/vnc-enc-tight.c
+++ b/ui/vnc-enc-tight.c
@@ -829,7 +829,7 @@ static int tight_compress_data(VncState *vs, int stream_id, size_t bytes,
int level, int strategy)
 {
 z_streamp zstream = &vs->tight.stream[stream_id];
-int previous_out;
+uLong previous_out;
 
 if (bytes < VNC_TIGHT_MIN_TO_COMPRESS) {
 vnc_write(vs, vs->tight.tight.buffer, vs->tight.tight.offset);

Bug#616159: qemu-kvm: SEGV inside vnc tight encoding

2011-03-03 Thread Michael Tokarev 
03.03.2011 00:31, Roland Dreier wrote:
> Package: qemu-kvm
> Version: 0.14.0+dfsg-1~tls
> Severity: important
> 
> I'm running a 32-bit guest (Xubuntu 10.04 to be precise) on a
> 64-bit host with the command line:
> 
> kvm -drive file=xubuntu.img,if=virtio,format=qcow2,boot=on -net 
> nic,model=virtio -net user -m 1024 -vga vmware -usbdevice tablet -smp 2 -vnc 
> :1

Roland, can you please tell me which vnc client are you
using, and with what options?  I use VNC alot here, but
never encountered this bug so far.

Thanks!

/mjt



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Bug#616159: qemu-kvm: SEGV inside vnc tight encoding

2011-03-02 Thread Roland Dreier 
Package: qemu-kvm
Version: 0.14.0+dfsg-1~tls
Severity: important

I'm running a 32-bit guest (Xubuntu 10.04 to be precise) on a
64-bit host with the command line:

kvm -drive file=xubuntu.img,if=virtio,format=qcow2,boot=on -net 
nic,model=virtio -net user -m 1024 -vga vmware -usbdevice tablet -smp 2 -vnc :1

and after a while, kvm crashes.  I get the following traceback from gdb:

Program received signal SIGSEGV, Segmentation fault.
0x7494aaf1 in memcpy () from /lib/libc.so.6
(gdb) bt
#0  0x7494aaf1 in memcpy () from /lib/libc.so.6
#1  0x004c0381 in buffer_append (buffer=0x184d2d8, data=0x1da6000, 
len=4294427045)
at /usr/include/bits/string3.h:52
#2  0x004c9bff in tight_compress_data (vs=0x1843230, stream_id=,
bytes=4294967717, level=, strategy=)
at ui/vnc-enc-tight.c:864
#3  0x004c9d2d in send_full_color_rect (vs=0x1843230, x=,
y=, w=, h=48) at 
ui/vnc-enc-tight.c:925
#4  0x004cb9cf in send_sub_rect_nojpeg (vs=0x1843230, x=432, y=441, 
w=,
h=) at ui/vnc-enc-tight.c:1462
#5  send_sub_rect (vs=0x1843230, x=432, y=441, w=, 
h=)
at ui/vnc-enc-tight.c:1530
#6  0x004ccd95 in tight_send_framebuffer_update (vs=0x1843230, x=,
y=, w=, h=)
at ui/vnc-enc-tight.c:1675
#7  0x004cec90 in vnc_job_add_rect (job=0x184d480, x=31088640, 
y=-540251, w=33525740, h=0)
at ui/vnc-jobs-sync.c:64
#8  0x004c07fa in vnc_update_client (vs=, 
has_dirty=)
at ui/vnc.c:909
#9  0x004c0a2f in vnc_refresh (opaque=0x17ba9b0) at ui/vnc.c:2326
#10 0x004d017e in qemu_run_timers (clock=) at 
qemu-timer.c:503
#11 0x004d01f6 in qemu_run_all_timers () at qemu-timer.c:634
#12 0x0041edc9 in main_loop_wait (nonblocking=)
at /home/roland/qemu-kvm-0.14.0+dfsg/vl.c:1401
#13 0x00438747 in kvm_main_loop () at 
/home/roland/qemu-kvm-0.14.0+dfsg/qemu-kvm.c:1589
#14 0x0041fe8a in main_loop (argc=17, argv=, 
envp=)
at /home/roland/qemu-kvm-0.14.0+dfsg/vl.c:1429
#15 main (argc=17, argv=, envp=)
at /home/roland/qemu-kvm-0.14.0+dfsg/vl.c:3201

I notice that the len values look like they are negative numbers
that have become huge unsigned 32-bit values...
-- Package-specific info:


/proc/cpuinfo:

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5650  @ 2.67GHz
stepping: 2
cpu MHz : 2667.171
cache size  : 12288 KB
physical id : 0
siblings: 12
core id : 0
cpu cores   : 6
apicid  : 0
initial apicid  : 0
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb 
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology 
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 
ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts 
tpr_shadow vnmi flexpriority ept vpid
bogomips: 5334.34
clflush size: 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5650  @ 2.67GHz
stepping: 2
cpu MHz : 2667.171
cache size  : 12288 KB
physical id : 0
siblings: 12
core id : 1
cpu cores   : 6
apicid  : 2
initial apicid  : 2
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb 
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology 
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 
ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts 
tpr_shadow vnmi flexpriority ept vpid
bogomips: 5333.47
clflush size: 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5650  @ 2.67GHz
stepping: 2
cpu MHz : 2667.171
cache size  : 12288 KB
physical id : 0
siblings: 12
core id : 2
cpu cores   : 6
apicid  : 4
initial apicid  : 4
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb 
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology 
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 
ssse3