Re: Network hangs when communicating with host

[Will Deacon]

[apologies for the delay -- I've been off for a week and am catching up
 on email]

On Tue, Oct 20, 2015 at 09:58:51AM -0400, Sasha Levin wrote:
> On 10/20/2015 09:42 AM, Dmitry Vyukov wrote:
> > I now have another issue. My binary fails to mmap a file within lkvm
> > sandbox. The same binary works fine on host and in qemu. I've added
> > strace into sandbox script, and here is the output:
> > 
> > [pid   837] openat(AT_FDCWD, "syzkaller-shm048878722", O_RDWR|O_CLOEXEC) = 5
> > [pid   837] mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 5,
> > 0) = -1 EINVAL (Invalid argument)
> > 
> > I don't see anything that can potentially cause EINVAL here. Is it
> > possible that lkvm somehow affects kernel behavior here?
> > 
> > I run lkvm as:
> > 
> > $ taskset 1 /kvmtool/lkvm sandbox --disk syz-0 --mem=2048 --cpus=2
> > --kernel /arch/x86/boot/bzImage --network mode=user --sandbox
> > /workdir/kvm/syz-0.sh
> 
> It's possible that something in the virtio-9p layer is broken. I'll give
> it a look in the evening.

I ended up with the patch below, but it's really ugly and I didn't get
round to posting it.

Will

--->8

>From 7cbcdfef1b9f094db4bf75676f22339f3164e103 Mon Sep 17 00:00:00 2001
From: Will Deacon 
Date: Fri, 17 Apr 2015 17:31:36 +0100
Subject: [PATCH] kvmtool: virtio-net: fix VIRTIO_NET_F_MRG_RXBUF usage in rx
 thread

When merging virtio-net buffers using the VIRTIO_NET_F_MRG_RXBUF feature,
the first buffer added to the used ring should indicate the total number
of buffers used to hold the packet. Unfortunately, kvmtool has a number
of issues when constructing these merged buffers:

  - Commit 5131332e3f1a ("kvmtool: convert net backend to support
bi-endianness") introduced a strange loop counter, which resulted in
hdr->num_buffers being set redundantly the first time round

  - When adding the buffers to the ring, we actually add them one-by-one,
allowing the guest to see the header before we've inserted the rest
of the data buffers...

  - ... which is made worse because we non-atomically increment the
num_buffers count in the header each time we insert a new data buffer

Consequently, the guest quickly becomes confused in its net rx code and
the whole thing grinds to a halt. This is easily exemplified by trying
to boot a root filesystem over NFS, which seldom succeeds.

This patch resolves the issues by allowing us to insert items into the
used ring without updating the index. Once the full payload has been
added and num_buffers corresponds to the total size, we *then* publish
the buffers to the guest.

Cc: Marc Zyngier 
Cc: Sasha Levin 
Signed-off-by: Will Deacon 
---
 include/kvm/virtio.h |  2 ++
 virtio/core.c| 32 +---
 virtio/net.c | 31 +++
 3 files changed, 42 insertions(+), 23 deletions(-)

diff --git a/include/kvm/virtio.h b/include/kvm/virtio.h
index 768ee9668d44..8324ba7d38be 100644
--- a/include/kvm/virtio.h
+++ b/include/kvm/virtio.h
@@ -112,6 +112,8 @@ static inline bool virt_queue__available(struct virt_queue 
*vq)
return virtio_guest_to_host_u16(vq, vq->vring.avail->idx) != 
vq->last_avail_idx;
 }
 
+void virt_queue__used_idx_advance(struct virt_queue *queue, u16 jump);
+struct vring_used_elem * virt_queue__set_used_elem_no_update(struct virt_queue 
*queue, u32 head, u32 len, u16 offset);
 struct vring_used_elem *virt_queue__set_used_elem(struct virt_queue *queue, 
u32 head, u32 len);
 
 bool virtio_queue__should_signal(struct virt_queue *vq);
diff --git a/virtio/core.c b/virtio/core.c
index 3b6e4d7cd045..d6ac289d450e 100644
--- a/virtio/core.c
+++ b/virtio/core.c
@@ -21,22 +21,17 @@ const char* virtio_trans_name(enum virtio_trans trans)
return "unknown";
 }
 
-struct vring_used_elem *virt_queue__set_used_elem(struct virt_queue *queue, 
u32 head, u32 len)
+void virt_queue__used_idx_advance(struct virt_queue *queue, u16 jump)
 {
-   struct vring_used_elem *used_elem;
u16 idx = virtio_guest_to_host_u16(queue, queue->vring.used->idx);
 
-   used_elem   = >vring.used->ring[idx % queue->vring.num];
-   used_elem->id   = virtio_host_to_guest_u32(queue, head);
-   used_elem->len  = virtio_host_to_guest_u32(queue, len);
-
/*
 * Use wmb to assure that used elem was updated with head and len.
 * We need a wmb here since we can't advance idx unless we're ready
 * to pass the used element to the guest.
 */
wmb();
-   idx++;
+   idx += jump;
queue->vring.used->idx = virtio_host_to_guest_u16(queue, idx);
 
/*
@@ -45,6 +40,29 @@ struct vring_used_elem *virt_queue__set_used_elem(struct 
virt_queue *queue, u32
 * an updated idx.
 */
wmb();
+}
+
+struct vring_used_elem *
+virt_queue__set_used_elem_no_update(struct virt_queue *queue, u32 head,
+   u32 

Re: Network hangs when communicating with host

[Sasha Levin]

On 10/20/2015 09:42 AM, Dmitry Vyukov wrote:
> I now have another issue. My binary fails to mmap a file within lkvm
> sandbox. The same binary works fine on host and in qemu. I've added
> strace into sandbox script, and here is the output:
> 
> [pid   837] openat(AT_FDCWD, "syzkaller-shm048878722", O_RDWR|O_CLOEXEC) = 5
> [pid   837] mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 5,
> 0) = -1 EINVAL (Invalid argument)
> 
> I don't see anything that can potentially cause EINVAL here. Is it
> possible that lkvm somehow affects kernel behavior here?
> 
> I run lkvm as:
> 
> $ taskset 1 /kvmtool/lkvm sandbox --disk syz-0 --mem=2048 --cpus=2
> --kernel /arch/x86/boot/bzImage --network mode=user --sandbox
> /workdir/kvm/syz-0.sh

It's possible that something in the virtio-9p layer is broken. I'll give
it a look in the evening.


Thanks,
Sasha
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network hangs when communicating with host

[Dmitry Vyukov]

I now have another issue. My binary fails to mmap a file within lkvm
sandbox. The same binary works fine on host and in qemu. I've added
strace into sandbox script, and here is the output:

[pid   837] openat(AT_FDCWD, "syzkaller-shm048878722", O_RDWR|O_CLOEXEC) = 5
[pid   837] mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 5,
0) = -1 EINVAL (Invalid argument)

I don't see anything that can potentially cause EINVAL here. Is it
possible that lkvm somehow affects kernel behavior here?

I run lkvm as:

$ taskset 1 /kvmtool/lkvm sandbox --disk syz-0 --mem=2048 --cpus=2
--kernel /arch/x86/boot/bzImage --network mode=user --sandbox
/workdir/kvm/syz-0.sh








On Mon, Oct 19, 2015 at 4:20 PM, Sasha Levin  wrote:
> On 10/19/2015 05:28 AM, Dmitry Vyukov wrote:
>> On Mon, Oct 19, 2015 at 11:22 AM, Andre Przywara  
>> wrote:
>>> Hi Dmitry,
>>>
>>> On 19/10/15 10:05, Dmitry Vyukov wrote:
 On Fri, Oct 16, 2015 at 7:25 PM, Sasha Levin  
 wrote:
> On 10/15/2015 04:20 PM, Dmitry Vyukov wrote:
>> Hello,
>>
>> I am trying to run a program in lkvm sandbox so that it communicates
>> with a program on host. I run lkvm as:
>>
>> ./lkvm sandbox --disk sandbox-test --mem=2048 --cpus=4 --kernel
>> /arch/x86/boot/bzImage --network mode=user -- /my_prog
>>
>> /my_prog then connects to a program on host over a tcp socket.
>> I see that host receives some data, sends some data back, but then
>> my_prog hangs on network read.
>>
>> To localize this I wrote 2 programs (attached). ping is run on host
>> and pong is run from lkvm sandbox. They successfully establish tcp
>> connection, but after some iterations both hang on read.
>>
>> Networking code in Go runtime is there for more than 3 years, widely
>> used in production and does not have any known bugs. However, it uses
>> epoll edge-triggered readiness notifications that known to be tricky.
>> Is it possible that lkvm contains some networking bug? Can it be
>> related to the data races in lkvm I reported earlier today?
>>>
>>> Just to let you know:
>>> I think we have seen networking issues in the past - root over NFS had
>>> issues IIRC. Will spent some time on debugging this and it looked like a
>>> race condition in kvmtool's virtio implementation. I think pinning
>>> kvmtool's virtio threads to one host core made this go away. However
>>> although he tried hard (even by Will's standards!) he couldn't find a
>>> the real root cause or a fix at the time he looked at it and we found
>>> other ways to work around the issues (using virtio-blk or initrd's).
>>>
>>> So it's quite possible that there are issues. I haven't had time yet to
>>> look at your sanitizer reports, but it looks like a promising approach
>>> to find the root cause.
>>
>>
>> Thanks, Andre!
>>
>> ping/pong does not hang within at least 5 minutes when I run lkvm
>> under taskset 1.
>>
>> And, yeah, this pretty strongly suggests a data race. ThreadSanitizer
>> can point you to the bug within a minute, so you just need to say
>> "aha! it is here". Or maybe not. There are no guarantees. But if you
>> already spent significant time on this, then checking the reports
>> definitely looks like a good idea.
>
> Okay, that's good to know.
>
> I have a few busy days, but I'll definitely try to clear up these reports
> as they seem to be pointing to real issues.
>
>
> Thanks,
> Sasha
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network hangs when communicating with host

[Andre Przywara]

Hi Dmitry,

On 19/10/15 10:05, Dmitry Vyukov wrote:
> On Fri, Oct 16, 2015 at 7:25 PM, Sasha Levin  wrote:
>> On 10/15/2015 04:20 PM, Dmitry Vyukov wrote:
>>> Hello,
>>>
>>> I am trying to run a program in lkvm sandbox so that it communicates
>>> with a program on host. I run lkvm as:
>>>
>>> ./lkvm sandbox --disk sandbox-test --mem=2048 --cpus=4 --kernel
>>> /arch/x86/boot/bzImage --network mode=user -- /my_prog
>>>
>>> /my_prog then connects to a program on host over a tcp socket.
>>> I see that host receives some data, sends some data back, but then
>>> my_prog hangs on network read.
>>>
>>> To localize this I wrote 2 programs (attached). ping is run on host
>>> and pong is run from lkvm sandbox. They successfully establish tcp
>>> connection, but after some iterations both hang on read.
>>>
>>> Networking code in Go runtime is there for more than 3 years, widely
>>> used in production and does not have any known bugs. However, it uses
>>> epoll edge-triggered readiness notifications that known to be tricky.
>>> Is it possible that lkvm contains some networking bug? Can it be
>>> related to the data races in lkvm I reported earlier today?

Just to let you know:
I think we have seen networking issues in the past - root over NFS had
issues IIRC. Will spent some time on debugging this and it looked like a
race condition in kvmtool's virtio implementation. I think pinning
kvmtool's virtio threads to one host core made this go away. However
although he tried hard (even by Will's standards!) he couldn't find a
the real root cause or a fix at the time he looked at it and we found
other ways to work around the issues (using virtio-blk or initrd's).

So it's quite possible that there are issues. I haven't had time yet to
look at your sanitizer reports, but it looks like a promising approach
to find the root cause.

Cheers,
Andre.

>>>
>>> I am on commit 3695adeb227813d96d9c41850703fb53a23845eb.
>>
>> Hey Dmitry,
>>
>> How long does it take to reproduce? I've been running ping/pong as you've
>> described and it looks like it doesn't get stuck (read/writes keep going
>> on both sides).
>>
>> Can you share your guest kernel config maybe?
> 
> 
> Humm it my setup it happens within a minute or so.
> 
> My kernel is not completely standard, but it works with qemu without
> any problems.
> It is not trivial to reproduce, but FWIW I on commit
> f9fbf6b72ffaaca8612979116c872c9d5d9cc1f5 of
> https://github.com/dvyukov/linux/commits/coverage branch. Config file
> is attached. Then, I build it with custom gcc: revision 228818 +
> https://codereview.appspot.com/267910043 patch. This is all per
> https://github.com/google/syzkaller instructions.
> 
> I run lkvm as:
> ./lkvm sandbox --disk sandbox-test --mem=2048 --cpus=4 --kernel
> /arch/x86/boot/bzImage --network mode=user -- /pong
> 
> kvmtool is on 3695adeb227813d96d9c41850703fb53a23845eb.
> 
> Just tried to do the same with qemu, it does not hang.
> 
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network hangs when communicating with host

[Dmitry Vyukov]

On Mon, Oct 19, 2015 at 11:22 AM, Andre Przywara  wrote:
> Hi Dmitry,
>
> On 19/10/15 10:05, Dmitry Vyukov wrote:
>> On Fri, Oct 16, 2015 at 7:25 PM, Sasha Levin  wrote:
>>> On 10/15/2015 04:20 PM, Dmitry Vyukov wrote:
 Hello,

 I am trying to run a program in lkvm sandbox so that it communicates
 with a program on host. I run lkvm as:

 ./lkvm sandbox --disk sandbox-test --mem=2048 --cpus=4 --kernel
 /arch/x86/boot/bzImage --network mode=user -- /my_prog

 /my_prog then connects to a program on host over a tcp socket.
 I see that host receives some data, sends some data back, but then
 my_prog hangs on network read.

 To localize this I wrote 2 programs (attached). ping is run on host
 and pong is run from lkvm sandbox. They successfully establish tcp
 connection, but after some iterations both hang on read.

 Networking code in Go runtime is there for more than 3 years, widely
 used in production and does not have any known bugs. However, it uses
 epoll edge-triggered readiness notifications that known to be tricky.
 Is it possible that lkvm contains some networking bug? Can it be
 related to the data races in lkvm I reported earlier today?
>
> Just to let you know:
> I think we have seen networking issues in the past - root over NFS had
> issues IIRC. Will spent some time on debugging this and it looked like a
> race condition in kvmtool's virtio implementation. I think pinning
> kvmtool's virtio threads to one host core made this go away. However
> although he tried hard (even by Will's standards!) he couldn't find a
> the real root cause or a fix at the time he looked at it and we found
> other ways to work around the issues (using virtio-blk or initrd's).
>
> So it's quite possible that there are issues. I haven't had time yet to
> look at your sanitizer reports, but it looks like a promising approach
> to find the root cause.


Thanks, Andre!

ping/pong does not hang within at least 5 minutes when I run lkvm
under taskset 1.

And, yeah, this pretty strongly suggests a data race. ThreadSanitizer
can point you to the bug within a minute, so you just need to say
"aha! it is here". Or maybe not. There are no guarantees. But if you
already spent significant time on this, then checking the reports
definitely looks like a good idea.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network hangs when communicating with host

[Sasha Levin]

On 10/19/2015 05:28 AM, Dmitry Vyukov wrote:
> On Mon, Oct 19, 2015 at 11:22 AM, Andre Przywara  
> wrote:
>> Hi Dmitry,
>>
>> On 19/10/15 10:05, Dmitry Vyukov wrote:
>>> On Fri, Oct 16, 2015 at 7:25 PM, Sasha Levin  wrote:
 On 10/15/2015 04:20 PM, Dmitry Vyukov wrote:
> Hello,
>
> I am trying to run a program in lkvm sandbox so that it communicates
> with a program on host. I run lkvm as:
>
> ./lkvm sandbox --disk sandbox-test --mem=2048 --cpus=4 --kernel
> /arch/x86/boot/bzImage --network mode=user -- /my_prog
>
> /my_prog then connects to a program on host over a tcp socket.
> I see that host receives some data, sends some data back, but then
> my_prog hangs on network read.
>
> To localize this I wrote 2 programs (attached). ping is run on host
> and pong is run from lkvm sandbox. They successfully establish tcp
> connection, but after some iterations both hang on read.
>
> Networking code in Go runtime is there for more than 3 years, widely
> used in production and does not have any known bugs. However, it uses
> epoll edge-triggered readiness notifications that known to be tricky.
> Is it possible that lkvm contains some networking bug? Can it be
> related to the data races in lkvm I reported earlier today?
>>
>> Just to let you know:
>> I think we have seen networking issues in the past - root over NFS had
>> issues IIRC. Will spent some time on debugging this and it looked like a
>> race condition in kvmtool's virtio implementation. I think pinning
>> kvmtool's virtio threads to one host core made this go away. However
>> although he tried hard (even by Will's standards!) he couldn't find a
>> the real root cause or a fix at the time he looked at it and we found
>> other ways to work around the issues (using virtio-blk or initrd's).
>>
>> So it's quite possible that there are issues. I haven't had time yet to
>> look at your sanitizer reports, but it looks like a promising approach
>> to find the root cause.
> 
> 
> Thanks, Andre!
> 
> ping/pong does not hang within at least 5 minutes when I run lkvm
> under taskset 1.
> 
> And, yeah, this pretty strongly suggests a data race. ThreadSanitizer
> can point you to the bug within a minute, so you just need to say
> "aha! it is here". Or maybe not. There are no guarantees. But if you
> already spent significant time on this, then checking the reports
> definitely looks like a good idea.

Okay, that's good to know.

I have a few busy days, but I'll definitely try to clear up these reports
as they seem to be pointing to real issues.


Thanks,
Sasha

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Network hangs when communicating with host

[Sasha Levin]

On 10/15/2015 04:20 PM, Dmitry Vyukov wrote:
> Hello,
> 
> I am trying to run a program in lkvm sandbox so that it communicates
> with a program on host. I run lkvm as:
> 
> ./lkvm sandbox --disk sandbox-test --mem=2048 --cpus=4 --kernel
> /arch/x86/boot/bzImage --network mode=user -- /my_prog
> 
> /my_prog then connects to a program on host over a tcp socket.
> I see that host receives some data, sends some data back, but then
> my_prog hangs on network read.
> 
> To localize this I wrote 2 programs (attached). ping is run on host
> and pong is run from lkvm sandbox. They successfully establish tcp
> connection, but after some iterations both hang on read.
> 
> Networking code in Go runtime is there for more than 3 years, widely
> used in production and does not have any known bugs. However, it uses
> epoll edge-triggered readiness notifications that known to be tricky.
> Is it possible that lkvm contains some networking bug? Can it be
> related to the data races in lkvm I reported earlier today?
> 
> I am on commit 3695adeb227813d96d9c41850703fb53a23845eb.

Hey Dmitry,

How long does it take to reproduce? I've been running ping/pong as you've
described and it looks like it doesn't get stuck (read/writes keep going
on both sides).

Can you share your guest kernel config maybe?


Thanks,
Sasha
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Network hangs when communicating with host

[Dmitry Vyukov]

Hello,

I am trying to run a program in lkvm sandbox so that it communicates
with a program on host. I run lkvm as:

./lkvm sandbox --disk sandbox-test --mem=2048 --cpus=4 --kernel
/arch/x86/boot/bzImage --network mode=user -- /my_prog

/my_prog then connects to a program on host over a tcp socket.
I see that host receives some data, sends some data back, but then
my_prog hangs on network read.

To localize this I wrote 2 programs (attached). ping is run on host
and pong is run from lkvm sandbox. They successfully establish tcp
connection, but after some iterations both hang on read.

Networking code in Go runtime is there for more than 3 years, widely
used in production and does not have any known bugs. However, it uses
epoll edge-triggered readiness notifications that known to be tricky.
Is it possible that lkvm contains some networking bug? Can it be
related to the data races in lkvm I reported earlier today?

I am on commit 3695adeb227813d96d9c41850703fb53a23845eb.

Thank you
package main

import (
	"log"
	"net"
)

func main() {
	c, err := net.Dial("tcp", "192.168.33.1:39921")
	if err != nil {
		log.Fatalf("failed to dial: %v", err)
	}
	log.Printf("connected")
	for {
		var buf [1]byte
		n, err := c.Write(buf[:])
		if err != nil || n != 1 {
			log.Fatalf("failed to write: %v", err)
		}
		log.Printf("write")
		n, err = c.Read(buf[:])
		if err != nil || n != 1 {
			log.Fatalf("failed to read: %v", err)
		}
		log.Printf("read")
	}
}
package main

import (
	"log"
	"net"
)

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:39921")
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	log.Printf("listening")
	c, err := ln.Accept()
	if err != nil {
		log.Fatalf("failed to accept: %v", err)
	}
	log.Printf("connected")
	for {
		var buf [1]byte
		n, err := c.Read(buf[:])
		if err != nil || n != 1 {
			log.Fatalf("failed to read: %v", err)
		}
		log.Printf("read")
		n, err = c.Write(buf[:])
		if err != nil || n != 1 {
			log.Fatalf("failed to write: %v", err)
		}
		log.Printf("write")
	}
}