Re: [Xen-devel] [DRAFT v5] PV Calls protocol design document (former XenSock)

[Stefano Stabellini]

Thank you!

On Sun, 21 Aug 2016, Christopher Clark wrote:
> The PV Calls (formerly XenSock) protocol design has recently attracted 
> interest within the OpenXT software
> development community, as it is a novel interdomain communication mechanism 
> and protocol.
> 
> We have a longstanding and active interest in interdomain communication 
> transport. v4v is a protocol and mechanism
> originally designed at Citrix for XenClient and XenClient XT, now in use by 
> OpenXT, and has been deployed in
> production systems for several years at this point. It has benefitted from 
> previous reviews by the Xen Community,
> and continued engagement from its original designers, and is our selection of 
> protocol and implementation to meet
> the particular software requirements of our project. Members of the OpenXT 
> community also have interdomain
> protocol experience through being involved in the development of both vchan 
> and the XSM architecture.
> 
> The PV Calls (formerly XenSock) work described in this thread is interesting 
> and quite a different technology,
> evidently with a different set of design constraints and focus on solving a 
> different problem: it enables the
> insertion of VM boundaries and separation in-between components that are 
> communicating via the POSIX socket API,
> which could not otherwise be so isolated from each other.
> 
> In contrast, v4v prioritizes isolation between VMs over seamless integration 
> of existing components, preferring no
> memory sharing between VMs, and mandatory access control enforced on 
> communication channels by the hypervisor.
> 
> So while PV Calls is not a fit for the needs of OpenXT, this proposed 
> approach looks very good for the problem
> that it aims to address. It is complimentary to the other mechanisms 
> available in the Xen ecosystem.
> 
> I would like to convey my positive support for the PV Calls proposal.
> 
> Christopher Clark
> BAE Systems, OpenXT Project
> http://openxt.org
> 
> 
> On Thu, Aug 4, 2016 at 10:17 AM, Stefano Stabellini  
> wrote:
>   Hi all,
> 
>   This is the design document of the PV Calls protocol. You can find
>   prototypes of the Linux frontend and backend drivers here:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git 
> pvcalls-5
> 
>   To use them, make sure to enable CONFIG_PVCALLS in your kernel config
>   and add "pvcalls=1" to the command line of your DomU Linux kernel. You
>   also need the toolstack to create the initial xenstore nodes for the
>   protocol. To do that, please apply the attached patch to libxl (the
>   patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU config
>   file.
> 
>   Note that previous versions of the protocols were named XenSock. It has
>   been renamed for clarity of scope and to avoid confusion with hv_sock
>   and vsock, which are used for inter-VMs communications.
> 
>   Cheers,
> 
>   Stefano
> 
>   Changes in v5:
>   - clarify text
>   - rename id to req_id
>   - rename sockid to id
>   - move id to request and response specific fields
>   - add version node to xenstore
> 
>   Changes in v4:
>   - rename xensock to pvcalls
> 
>   Changes in v3:
>   - add a dummy element to struct xen_xensock_request to make sure the
>     size of the struct is the same on both x86_32 and x86_64
> 
>   Changes in v2:
>   - add max-dataring-page-order
>   - add "Publish backend features and transport parameters" to backend
>     xenbus workflow
>   - update new cmd values
>   - update xen_xensock_request
>   - add backlog parameter to listen and binary layout
>   - add description of new data ring format (interface+data)
>   - modify connect and accept to reflect new data ring format
>   - add link to POSIX docs
>   - add error numbers
>   - add address format section and relevant numeric definitions
>   - add explicit mention of unimplemented commands
>   - add protocol node name
>   - add xenbus shutdown diagram
>   - add socket operation
> 
>   ---
> 
>   # PV Calls Protocol version 1
> 
>   ## Rationale
> 
>   PV Calls is a paravirtualized protocol that allows the implementation of
>   a set of POSIX functions in a different domain. The PV Calls frontend
>   sends POSIX function calls to the backend, which implements them and
>   returns a value to the frontend.
> 
>   This version of the document covers networking function calls, such as
>   connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but
>   the protocol is meant to be easily extended to cover different sets of
>   calls. Unimplemented commands return ENOTSUPP.
> 
>   PV Calls provide the following benefits:
>   * full visibility of the guest behavior on the backend domain, allowing
>     for inexpensive filtering and manipulation of 

Re: [Xen-devel] [DRAFT v5] PV Calls protocol design document (former XenSock)

[Christopher Clark]

The PV Calls (formerly XenSock) protocol design has recently attracted
interest within the OpenXT software development community, as it is a novel
interdomain communication mechanism and protocol.

We have a longstanding and active interest in interdomain communication
transport. v4v is a protocol and mechanism originally designed at Citrix
for XenClient and XenClient XT, now in use by OpenXT, and has been deployed
in production systems for several years at this point. It has benefitted
from previous reviews by the Xen Community, and continued engagement from
its original designers, and is our selection of protocol and implementation
to meet the particular software requirements of our project. Members of the
OpenXT community also have interdomain protocol experience through being
involved in the development of both vchan and the XSM architecture.

The PV Calls (formerly XenSock) work described in this thread is
interesting and quite a different technology, evidently with a different
set of design constraints and focus on solving a different problem: it
enables the insertion of VM boundaries and separation in-between components
that are communicating via the POSIX socket API, which could not otherwise
be so isolated from each other.

In contrast, v4v prioritizes isolation between VMs over seamless
integration of existing components, preferring no memory sharing between
VMs, and mandatory access control enforced on communication channels by the
hypervisor.

So while PV Calls is not a fit for the needs of OpenXT, this proposed
approach looks very good for the problem that it aims to address. It is
complimentary to the other mechanisms available in the Xen ecosystem.

I would like to convey my positive support for the PV Calls proposal.

Christopher Clark
BAE Systems, OpenXT Project
http://openxt.org


On Thu, Aug 4, 2016 at 10:17 AM, Stefano Stabellini 
wrote:

> Hi all,
>
> This is the design document of the PV Calls protocol. You can find
> prototypes of the Linux frontend and backend drivers here:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git
> pvcalls-5
>
> To use them, make sure to enable CONFIG_PVCALLS in your kernel config
> and add "pvcalls=1" to the command line of your DomU Linux kernel. You
> also need the toolstack to create the initial xenstore nodes for the
> protocol. To do that, please apply the attached patch to libxl (the
> patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU config
> file.
>
> Note that previous versions of the protocols were named XenSock. It has
> been renamed for clarity of scope and to avoid confusion with hv_sock
> and vsock, which are used for inter-VMs communications.
>
> Cheers,
>
> Stefano
>
> Changes in v5:
> - clarify text
> - rename id to req_id
> - rename sockid to id
> - move id to request and response specific fields
> - add version node to xenstore
>
> Changes in v4:
> - rename xensock to pvcalls
>
> Changes in v3:
> - add a dummy element to struct xen_xensock_request to make sure the
>   size of the struct is the same on both x86_32 and x86_64
>
> Changes in v2:
> - add max-dataring-page-order
> - add "Publish backend features and transport parameters" to backend
>   xenbus workflow
> - update new cmd values
> - update xen_xensock_request
> - add backlog parameter to listen and binary layout
> - add description of new data ring format (interface+data)
> - modify connect and accept to reflect new data ring format
> - add link to POSIX docs
> - add error numbers
> - add address format section and relevant numeric definitions
> - add explicit mention of unimplemented commands
> - add protocol node name
> - add xenbus shutdown diagram
> - add socket operation
>
> ---
>
> # PV Calls Protocol version 1
>
> ## Rationale
>
> PV Calls is a paravirtualized protocol that allows the implementation of
> a set of POSIX functions in a different domain. The PV Calls frontend
> sends POSIX function calls to the backend, which implements them and
> returns a value to the frontend.
>
> This version of the document covers networking function calls, such as
> connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but
> the protocol is meant to be easily extended to cover different sets of
> calls. Unimplemented commands return ENOTSUPP.
>
> PV Calls provide the following benefits:
> * full visibility of the guest behavior on the backend domain, allowing
>   for inexpensive filtering and manipulation of any guest calls
> * excellent performance
>
> Specifically, PV Calls for networking offer these advantages:
> * guest networking works out of the box with VPNs, wireless networks and
>   any other complex configurations on the host
> * guest services listen on ports bound directly to the backend domain IP
>   addresses
> * localhost becomes a secure namespace for inter-VMs communications
>
>
> ## Design
>
> ### Xenstore
>
> The frontend and the backend connect to each other exchanging information
> via
> 

Re: [Xen-devel] [DRAFT v5] PV Calls protocol design document (former XenSock)

[Stefano Stabellini]

Ping?

On Thu, 4 Aug 2016, Stefano Stabellini wrote:
> Hi all,
> 
> This is the design document of the PV Calls protocol. You can find
> prototypes of the Linux frontend and backend drivers here:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git pvcalls-5
> 
> To use them, make sure to enable CONFIG_PVCALLS in your kernel config
> and add "pvcalls=1" to the command line of your DomU Linux kernel. You
> also need the toolstack to create the initial xenstore nodes for the
> protocol. To do that, please apply the attached patch to libxl (the
> patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU config
> file.
> 
> Note that previous versions of the protocols were named XenSock. It has
> been renamed for clarity of scope and to avoid confusion with hv_sock
> and vsock, which are used for inter-VMs communications.
> 
> Cheers,
> 
> Stefano
> 
> Changes in v5:
> - clarify text
> - rename id to req_id
> - rename sockid to id
> - move id to request and response specific fields
> - add version node to xenstore
> 
> Changes in v4:
> - rename xensock to pvcalls
> 
> Changes in v3:
> - add a dummy element to struct xen_xensock_request to make sure the
>   size of the struct is the same on both x86_32 and x86_64
> 
> Changes in v2:
> - add max-dataring-page-order
> - add "Publish backend features and transport parameters" to backend
>   xenbus workflow
> - update new cmd values
> - update xen_xensock_request
> - add backlog parameter to listen and binary layout
> - add description of new data ring format (interface+data)
> - modify connect and accept to reflect new data ring format
> - add link to POSIX docs
> - add error numbers
> - add address format section and relevant numeric definitions
> - add explicit mention of unimplemented commands
> - add protocol node name
> - add xenbus shutdown diagram
> - add socket operation
> 
> ---
> 
> # PV Calls Protocol version 1
> 
> ## Rationale
> 
> PV Calls is a paravirtualized protocol that allows the implementation of
> a set of POSIX functions in a different domain. The PV Calls frontend
> sends POSIX function calls to the backend, which implements them and
> returns a value to the frontend.
> 
> This version of the document covers networking function calls, such as
> connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but
> the protocol is meant to be easily extended to cover different sets of
> calls. Unimplemented commands return ENOTSUPP.
> 
> PV Calls provide the following benefits:
> * full visibility of the guest behavior on the backend domain, allowing
>   for inexpensive filtering and manipulation of any guest calls
> * excellent performance
> 
> Specifically, PV Calls for networking offer these advantages:
> * guest networking works out of the box with VPNs, wireless networks and
>   any other complex configurations on the host
> * guest services listen on ports bound directly to the backend domain IP
>   addresses
> * localhost becomes a secure namespace for inter-VMs communications
> 
> 
> ## Design
> 
> ### Xenstore
> 
> The frontend and the backend connect to each other exchanging information via
> xenstore. The toolstack creates front and back nodes with state
> XenbusStateInitialising. The protocol node name is **pvcalls**. There can only
> be one PV Calls frontend per domain.
> 
>  Frontend XenBus Nodes
> 
> port
>  Values: 
> 
>  The identifier of the Xen event channel used to signal activity
>  in the ring buffer.
> 
> ring-ref
>  Values: 
> 
>  The Xen grant reference granting permission for the backend to map
>  the sole page in a single page sized ring buffer.
> 
>  Backend XenBus Nodes
> 
> version
>  Values: 
> 
>  Protocol version supported by the backend.
> 
> max-dataring-page-order
>  Values: 
> 
>  The maximum supported size of the data ring in units of lb(machine
>  pages). (e.g. 0 == 1 page,  1 = 2 pages, 2 == 4 pages, etc.).
> 
>  State Machine
> 
> Initialization:
> 
> *Front*   *Back*
> XenbusStateInitialising   XenbusStateInitialising
> - Query virtual device- Query backend device
>   properties.   identification data.
> - Setup OS device instance.   - Publish backend features
> - Allocate and initialize the   and transport parameters
>   request ring.  |
> - Publish transport parameters   |
>   that will be in effect during  V
>   this connection.XenbusStateInitWait
>  |
>  |
>  V
>XenbusStateInitialised
> 
>   - Query frontend transport 
> parameters.
>   - Connect to the request ring and
>

[Xen-devel] [DRAFT v5] PV Calls protocol design document (former XenSock)

[Stefano Stabellini]

Hi all,

This is the design document of the PV Calls protocol. You can find
prototypes of the Linux frontend and backend drivers here:

git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git pvcalls-5

To use them, make sure to enable CONFIG_PVCALLS in your kernel config
and add "pvcalls=1" to the command line of your DomU Linux kernel. You
also need the toolstack to create the initial xenstore nodes for the
protocol. To do that, please apply the attached patch to libxl (the
patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU config
file.

Note that previous versions of the protocols were named XenSock. It has
been renamed for clarity of scope and to avoid confusion with hv_sock
and vsock, which are used for inter-VMs communications.

Cheers,

Stefano

Changes in v5:
- clarify text
- rename id to req_id
- rename sockid to id
- move id to request and response specific fields
- add version node to xenstore

Changes in v4:
- rename xensock to pvcalls

Changes in v3:
- add a dummy element to struct xen_xensock_request to make sure the
  size of the struct is the same on both x86_32 and x86_64

Changes in v2:
- add max-dataring-page-order
- add "Publish backend features and transport parameters" to backend
  xenbus workflow
- update new cmd values
- update xen_xensock_request
- add backlog parameter to listen and binary layout
- add description of new data ring format (interface+data)
- modify connect and accept to reflect new data ring format
- add link to POSIX docs
- add error numbers
- add address format section and relevant numeric definitions
- add explicit mention of unimplemented commands
- add protocol node name
- add xenbus shutdown diagram
- add socket operation

---

# PV Calls Protocol version 1

## Rationale

PV Calls is a paravirtualized protocol that allows the implementation of
a set of POSIX functions in a different domain. The PV Calls frontend
sends POSIX function calls to the backend, which implements them and
returns a value to the frontend.

This version of the document covers networking function calls, such as
connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but
the protocol is meant to be easily extended to cover different sets of
calls. Unimplemented commands return ENOTSUPP.

PV Calls provide the following benefits:
* full visibility of the guest behavior on the backend domain, allowing
  for inexpensive filtering and manipulation of any guest calls
* excellent performance

Specifically, PV Calls for networking offer these advantages:
* guest networking works out of the box with VPNs, wireless networks and
  any other complex configurations on the host
* guest services listen on ports bound directly to the backend domain IP
  addresses
* localhost becomes a secure namespace for inter-VMs communications


## Design

### Xenstore

The frontend and the backend connect to each other exchanging information via
xenstore. The toolstack creates front and back nodes with state
XenbusStateInitialising. The protocol node name is **pvcalls**. There can only
be one PV Calls frontend per domain.

 Frontend XenBus Nodes

port
 Values: 

 The identifier of the Xen event channel used to signal activity
 in the ring buffer.

ring-ref
 Values: 

 The Xen grant reference granting permission for the backend to map
 the sole page in a single page sized ring buffer.

 Backend XenBus Nodes

version
 Values: 

 Protocol version supported by the backend.

max-dataring-page-order
 Values: 

 The maximum supported size of the data ring in units of lb(machine
 pages). (e.g. 0 == 1 page,  1 = 2 pages, 2 == 4 pages, etc.).

 State Machine

Initialization:

*Front*   *Back*
XenbusStateInitialising   XenbusStateInitialising
- Query virtual device- Query backend device
  properties.   identification data.
- Setup OS device instance.   - Publish backend features
- Allocate and initialize the   and transport parameters
  request ring.  |
- Publish transport parameters   |
  that will be in effect during  V
  this connection.XenbusStateInitWait
 |
 |
 V
   XenbusStateInitialised

  - Query frontend transport parameters.
  - Connect to the request ring and
event channel.
 |
 |
 V
 XenbusStateConnected

 - Query backend device properties.
 - Finalize OS virtual