On Wed, 23 Nov 2022 16:27:00 +0100, Jan Kiszka <jan.kis...@siemens.com> wrote: > On 16.11.22 03:13, Xuan Zhuo wrote: > > On Mon, 14 Nov 2022 22:30:53 +0100, Jan Kiszka <jan.kis...@siemens.com> > > wrote: > >> On 18.10.22 09:32, Jan Kiszka wrote: > >>> On 17.10.22 09:47, Xuan Zhuo wrote: > >>>> Hello everyone, > >>>> > >>>> # Background > >>>> > >>>> Nowadays, there is a common scenario to accelerate communication between > >>>> different VMs and containers, including light weight virtual machine > >>>> based > >>>> containers. One way to achieve this is to colocate them on the same host. > >>>> However, the performance of inter-VM communication through network stack > >>>> is not > >>>> optimal and may also waste extra CPU cycles. This scenario has been > >>>> discussed > >>>> many times, but still no generic solution available [1] [2] [3]. > >>>> > >>>> With pci-ivshmem + SMC(Shared Memory Communications: [4]) based PoC[5], > >>>> We found that by changing the communication channel between VMs from TCP > >>>> to SMC > >>>> with shared memory, we can achieve superior performance for a common > >>>> socket-based application[5]: > >>>> - latency reduced by about 50% > >>>> - throughput increased by about 300% > >>>> - CPU consumption reduced by about 50% > >>>> > >>>> Since there is no particularly suitable shared memory management solution > >>>> matches the need for SMC(See ## Comparison with existing technology), > >>>> and virtio > >>>> is the standard for communication in the virtualization world, we want to > >>>> implement a virtio-ism device based on virtio, which can support > >>>> on-demand > >>>> memory sharing across VMs, containers or VM-container. To match the > >>>> needs of SMC, > >>>> the virtio-ism device need to support: > >>>> > >>>> 1. Dynamic provision: shared memory regions are dynamically allocated and > >>>> provisioned. > >>>> 2. Multi-region management: the shared memory is divided into regions, > >>>> and a peer may allocate one or more regions from the same shared > >>>> memory > >>>> device. > >>>> 3. Permission control: The permission of each region can be set > >>>> seperately. > >>>> > >>>> # Virtio ism device > >>>> > >>>> ISM devices provide the ability to share memory between different guests > >>>> on a > >>>> host. A guest's memory got from ism device can be shared with multiple > >>>> peers at > >>>> the same time. This shared relationship can be dynamically created and > >>>> released. > >>>> > >>>> The shared memory obtained from the device is divided into multiple ism > >>>> regions > >>>> for share. ISM device provides a mechanism to notify other ism region > >>>> referrers > >>>> of content update events. > >>>> > >>>> # Usage (SMC as example) > >>>> > >>>> Maybe there is one of possible use cases: > >>>> > >>>> 1. SMC calls the interface ism_alloc_region() of the ism driver to > >>>> return the > >>>> location of a memory region in the PCI space and a token. > >>>> 2. The ism driver mmap the memory region and return to SMC with the token > >>>> 3. SMC passes the token to the connected peer > >>>> 3. the peer calls the ism driver interface ism_attach_region(token) to > >>>> get the location of the PCI space of the shared memory > >>>> > >>>> > >>>> # About hot plugging of the ism device > >>>> > >>>> Hot plugging of devices is a heavier, possibly failed, > >>>> time-consuming, and > >>>> less scalable operation. So, we don't plan to support it for now. > >>>> > >>>> # Comparison with existing technology > >>>> > >>>> ## ivshmem or ivshmem 2.0 of Qemu > >>>> > >>>> 1. ivshmem 1.0 is a large piece of memory that can be seen by all > >>>> devices that > >>>> use this VM, so the security is not enough. > >>>> > >>>> 2. ivshmem 2.0 is a shared memory belonging to a VM that can be > >>>> read-only by all > >>>> other VMs that use the ivshmem 2.0 shared memory device, which also > >>>> does not > >>>> meet our needs in terms of security. > >>> > >>> This is addressed by establishing separate links between VMs (modeled > >>> with separate devices). That is a trade-off between simplicity of the > >>> model and convenience, for sure. > >> > >> BTW, simplicity can also brings security because it reduces the trusted > >> code base. > >> > >> Another feature of ivshmem-v2 is permitting direct access to essential > >> resources of the device from /unprivileged/ userspace, including to the > >> event triggering registers. Is your model designed for that as well? It > >> not only permits VM-to-VM, it actually makes app-to-app (in VMs) very > >> cheap. > > > > Yes, there are two actual application scenarios or design goals: > > > > * As we mentioned above, docking with SMC inside the linux kernel to > > achieve high-speed communication. > > * Virtio-ISM also has an interface below /dev. Ordinary users can also > > directly > > obtain the SHM region resources to achieve sharing with other APPs on > > other > > VMs. > > > > https://github.com/fengidri/linux-kernel-virtio-ism/commit/55a8ed21344e26f574dd81b0213b0d61d80e2ecb > > https://github.com/fengidri/linux-kernel-virtio-ism/commit/6518739f9a9a36f25d5709da940b7a7938f8e0ee > > > > An example for the missing detach notification in ISM.
Yes, I agree. I also think this is a good point. We should introduce the management function of life status in the next version. For example, attach, detach, permissions change notification. I think it may be a more appropriate way to use virtio-virtqueue to receive these messages. The current ISM Spec defines vq that can be used to receive a shm update. We can add some new events. struct event { u32 ev_type; // update event, detach event or attach event ...... } > > And the model of ivshmem-v2 permits syscall-free notification (sending, > not IRQ-based receiving, obviously). On reception, it avoids one vmexit > to throttle incoming events if you have continuously firing sender in > combination with a non-reacting unprivileged receiver task. I guess that you are talking about the SHM update event. We did not introduce similar models in the ISM framework, because we hope that the user will realize such a mechanism in the shared memory, such as SMC. If the user uses the shared memory at user space, I think it is also convenient to use a part of the shared memory as a notification area. > > Would be great to combine the best of all worlds here. But specifically > missing life-cycle management on the detach side makes the ISM not > better than legacy ivshmem IHMO. I will add such a mechanism in the next version. Thanks. > > Jan > > -- > Siemens AG, Technology > Competence Center Embedded Linux > --------------------------------------------------------------------- To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org