[PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets
Hyper-V Sockets (hv_sock) supplies a byte-stream based communication mechanism between the host and the guest. It's somewhat like TCP over VMBus, but the transportation layer (VMBus) is much simpler than IP. With Hyper-V Sockets, applications between the host and the guest can talk to each other directly by the traditional BSD-style socket APIs. Hyper-V Sockets is only available on new Windows hosts, like Windows Server 2016. More info is in this article "Make your own integration services": https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/develop/make_mgmt_service The patch implements the necessary support in the guest side by introducing a new socket address family AF_HYPERV. Signed-off-by: Dexuan Cui Cc: "K. Y. Srinivasan" Cc: Haiyang Zhang Cc: Vitaly Kuznetsov Cc: Cathy Avery --- You can also get the patch here: https://github.com/dcui/linux/commits/decui/hv_sock/net-next/20160620_v12 For the change log before v12, please see https://lkml.org/lkml/2016/5/15/31 In v12, the changes are mainly the following: 1) remove the module params as David suggested. 2) use 5 exact pages for VMBus send/recv rings, respectively. The host side's design of the feature requires 5 exact pages for recv/send rings respectively -- this is suboptimal considering memory consumption, however unluckily we have to live with it, before the host comes up with a new design in the future. :-( 3) remove the per-connection static send/recv buffers Instead, we allocate and free the buffers dynamically only when we recv/send data. This means: when a connection is idle, no memory is consumed as recv/send buffers at all. Looking forward to your comments! MAINTAINERS |2 + include/linux/hyperv.h | 14 + include/linux/socket.h |4 +- include/net/af_hvsock.h | 59 ++ include/uapi/linux/hyperv.h | 25 + net/Kconfig |1 + net/Makefile|1 + net/hv_sock/Kconfig | 10 + net/hv_sock/Makefile|3 + net/hv_sock/af_hvsock.c | 1514 +++ 10 files changed, 1632 insertions(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 50f69ba..6eaa26f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5514,7 +5514,9 @@ F:drivers/pci/host/pci-hyperv.c F: drivers/net/hyperv/ F: drivers/scsi/storvsc_drv.c F: drivers/video/fbdev/hyperv_fb.c +F: net/hv_sock/ F: include/linux/hyperv.h +F: include/net/af_hvsock.h F: tools/hv/ F: Documentation/ABI/stable/sysfs-bus-vmbus diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h index 50f493e..95d159e 100644 --- a/include/linux/hyperv.h +++ b/include/linux/hyperv.h @@ -1509,4 +1509,18 @@ static inline void commit_rd_index(struct vmbus_channel *channel) } +struct vmpipe_proto_header { + u32 pkt_type; + u32 data_size; +}; + +#define HVSOCK_HEADER_LEN (sizeof(struct vmpacket_descriptor) + \ +sizeof(struct vmpipe_proto_header)) + +/* See 'prev_indices' in hv_ringbuffer_read(), hv_ringbuffer_write() */ +#define PREV_INDICES_LEN (sizeof(u64)) + +#define HVSOCK_PKT_LEN(payload_len)(HVSOCK_HEADER_LEN + \ + ALIGN((payload_len), 8) + \ + PREV_INDICES_LEN) #endif /* _HYPERV_H */ diff --git a/include/linux/socket.h b/include/linux/socket.h index b5cc5a6..0b68b58 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -202,8 +202,9 @@ struct ucred { #define AF_VSOCK 40 /* vSockets */ #define AF_KCM 41 /* Kernel Connection Multiplexor*/ #define AF_QIPCRTR 42 /* Qualcomm IPC Router */ +#define AF_HYPERV 43 /* Hyper-V Sockets */ -#define AF_MAX 43 /* For now.. */ +#define AF_MAX 44 /* For now.. */ /* Protocol families, same as address families. */ #define PF_UNSPEC AF_UNSPEC @@ -251,6 +252,7 @@ struct ucred { #define PF_VSOCK AF_VSOCK #define PF_KCM AF_KCM #define PF_QIPCRTR AF_QIPCRTR +#define PF_HYPERV AF_HYPERV #define PF_MAX AF_MAX /* Maximum queue length specifiable by listen. */ diff --git a/include/net/af_hvsock.h b/include/net/af_hvsock.h new file mode 100644 index 000..20d23d5 --- /dev/null +++ b/include/net/af_hvsock.h @@ -0,0 +1,59 @@ +#ifndef __AF_HVSOCK_H__ +#define __AF_HVSOCK_H__ + +#include +#include +#include + +/* The host side's design of the feature requires 5 exact pages for recv/send + * rings respectively -- this is suboptimal considering memory consumption, + * however unluckily we have to live with it, before the host comes up with + * a better new design in the future. + */ +#define RINGBUFFER_HVSOCK_RCV_SIZE (PAGE_SIZE * 5) +#define RINGBUFFER_HVSOCK_SND_SIZE (PAGE_SIZE * 5) + +#define sk_to_hvsock(__sk) ((struct hvsock_sock *)(__sk)) +#define hvsock_to_s
Re: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets
From: Dexuan Cui Date: Fri, 24 Jun 2016 07:45:24 + > + while ((ret = vmalloc(size)) == NULL) > + ssleep(1); This is completely, and entirely, unacceptable. If the allocation fails, you return an error and release your resources. You don't just loop forever waiting for it to succeed.
RE: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets
> From: David Miller [mailto:da...@davemloft.net] > Sent: Tuesday, June 28, 2016 17:34 > To: Dexuan Cui > Cc: gre...@linuxfoundation.org; netdev@vger.kernel.org; linux- > ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de; > a...@canonical.com; jasow...@redhat.com; vkuzn...@redhat.com; > cav...@redhat.com; KY Srinivasan ; Haiyang Zhang > ; j...@perches.com > Subject: Re: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets > > From: Dexuan Cui > Date: Fri, 24 Jun 2016 07:45:24 + > > > + while ((ret = vmalloc(size)) == NULL) > > + ssleep(1); > > This is completely, and entirely, unacceptable. > > If the allocation fails, you return an error and release > your resources. > > You don't just loop forever waiting for it to succeed. Hi David, I agree this is ugly... The idea here is: IMO the syscalls sys_read()/write() shoudn't return -ENOMEM, so I have to make sure the buffer allocation succeeds? I tried to use kmalloc with __GFP_NOFAIL, but I hit a warning in in mm/page_alloc.c: WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); What error code do you think I should return? EAGAIN, ERESTARTSYS, or something else? May I have your suggestion? Thanks! -- Dexuan
Re: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets
From: Dexuan Cui Date: Tue, 28 Jun 2016 09:59:21 + > The idea here is: IMO the syscalls sys_read()/write() shoudn't return > -ENOMEM, so I have to make sure the buffer allocation succeeds? You have to fail if resources cannot be allocated.
RE: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets
> From: David Miller [mailto:da...@davemloft.net] > Sent: Tuesday, June 28, 2016 21:45 > To: Dexuan Cui > Cc: gre...@linuxfoundation.org; netdev@vger.kernel.org; linux- > ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de; > a...@canonical.com; jasow...@redhat.com; vkuzn...@redhat.com; > cav...@redhat.com; KY Srinivasan ; Haiyang Zhang > ; j...@perches.com > Subject: Re: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets > > From: Dexuan Cui > Date: Tue, 28 Jun 2016 09:59:21 + > > > The idea here is: IMO the syscalls sys_read()/write() shoudn't return > > -ENOMEM, so I have to make sure the buffer allocation succeeds? > > You have to fail if resources cannot be allocated. OK, I'll try to fix this, probably by returning -EAGAIN or -ERESTARTSYS. I'll report back ASAP. Thanks, -- Dexuan
Re: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets
On 06/28/2016 02:59 AM, Dexuan Cui wrote: The idea here is: IMO the syscalls sys_read()/write() shoudn't return -ENOMEM, so I have to make sure the buffer allocation succeeds? I tried to use kmalloc with __GFP_NOFAIL, but I hit a warning in in mm/page_alloc.c: WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); What error code do you think I should return? EAGAIN, ERESTARTSYS, or something else? May I have your suggestion? Thanks! What happens as far as errno is concerned when an application makes a read() call against a (say TCP) socket associated with a connection which has been reset? Is it limited to those errno values listed in the read() manpage, or does it end-up getting an errno value from those listed in the recv() manpage? Or, perhaps even one not (presently) listed in either? rick jones
RE: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets
> From: Rick Jones [mailto:rick.jon...@hpe.com] > Sent: Tuesday, June 28, 2016 23:43 > To: Dexuan Cui ; David Miller > Cc: gre...@linuxfoundation.org; netdev@vger.kernel.org; linux- > ker...@vger.kernel.org; de...@linuxdriverproject.org; o...@aepfle.de; > a...@canonical.com; jasow...@redhat.com; vkuzn...@redhat.com; > cav...@redhat.com; KY Srinivasan ; Haiyang Zhang > ; j...@perches.com > Subject: Re: [PATCH v12 net-next 1/1] hv_sock: introduce Hyper-V Sockets > > On 06/28/2016 02:59 AM, Dexuan Cui wrote: > > The idea here is: IMO the syscalls sys_read()/write() shoudn't return > > -ENOMEM, so I have to make sure the buffer allocation succeeds? > > > > I tried to use kmalloc with __GFP_NOFAIL, but I hit a warning in > > in mm/page_alloc.c: > > WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); > > > > What error code do you think I should return? > > EAGAIN, ERESTARTSYS, or something else? > > > > May I have your suggestion? Thanks! > > What happens as far as errno is concerned when an application makes a > read() call against a (say TCP) socket associated with a connection > which has been reset? I suppose it is ECONNRESET (Connection reset by peer). > Is it limited to those errno values listed in the > read() manpage, or does it end-up getting an errno value from those > listed in the recv() manpage? Or, perhaps even one not (presently) > listed in either? > > rick jones Actually "man read/write" says "Other errors may occur, depending on the object connected to fd". "man send/recv" indeed lists ENOMEM. Considering AF_HYPERV is a new socket type, ENOMEM seems OK to me and I'm going to post a new version of the patch. In the long run, I think we should add a new API in the VMBus driver, allowing data copy from VMBus ringbuffer into user mode buffer directly. This way, we can even eliminate this temporary buffer. Thanks, -- Dexuan