On Tue, Aug 18, 2026 at 05:38:38PM +0800, Huiba Li wrote: > Hi, > > I would like to propose adding a QEMU block driver for the > overlaybd image format, and I am looking for early feedback on > scope and design before sending patches. > > What overlaybd is > ----------------- > Overlaybd is an open-source image format ( > https://containerd.github.io/overlaybd/) > hosted as a sub-project of containerd (CNCF graduated). A disk > image is a stack of read-only snapshot blobs plus a writable top > file, in a layout compatible with OCI image registries: a base > image is stored once and shared by every derivative image, and > data is fetched on demand rather than pulled in full before boot.
This looks interesting. I've pondered whether the layered and immutable image trends in recent years can be used effectively for virtual machines. I will take a look at the specs you linked to understand the internals better, but in the meantime some answers to your questions below... > > Two properties distinguish it from qcow2/VHDX/VMDK-style chains: > > * Lookup cost is O(1) in chain depth. At open time the per-file > indices are merged into a single in-memory index (an > extent-based log-structured merge tree), so a read never walks > backing files, and per-file index caches do not multiply with > snapshots. > > * The merged index is small (about 300 KB average for 50+ GB > images in our production data), and because extents are > variable-length, writes land at 512-byte granularity with no > cluster copy-on-write. > > The format is openly specified: > https://github.com/containerd/overlaybd/tree/main/docs/specs/lsmt.md > https://github.com/containerd/overlaybd/tree/main/docs/specs/zfile.md > > Reference implementation: > https://github.com/containerd/overlaybd > > Overlaybd has been in production at Alibaba for years (Taobao, > Tmall, Alibaba Cloud, Function Compute) and is used by Azure AKS > Artifact Streaming, Databricks serverless compute, and several > microVM-based sandbox platforms. The design is documented in two > USENIX ATC papers (DADI, ATC'20; FaaSNet, ATC'21). I am one of the > overlaybd/DADI authors. > > Why a QEMU driver > ----------------- > Today overlaybd images are served to VMs through a kernel blk dev > and a userspace daemon. That fits container workloads well, but > a native QEMU driver would let any QEMU-based stack (libvirt, > KubeVirt, OpenStack, plain qemu-system-*) boot overlaybd images > directly, with QEMU's usual amenities (backing files, block jobs, > throttling, migration) and no extra daemon on the host. This is > beneficial to agent sandboxes. > > Proposed implementation > ----------------------- > A new block format driver, block/overlaybd.c, written in C from the > open spec (like the existing vmdk/vhdx drivers) rather than binding > the C++ reference implementation: > > * v1: read-only support. Local layer files, index merge at open, > raw and ZFile blobs (lz4/zstd are already optional QEMU > dependencies). Writes can be served by a qcow2/raw file on top, > exactly like booting a read-only backing image today. > * v2: native writable layer, and lazy fetching of remote blobs — > either composed over the existing curl driver or with a small > built-in HTTP fetcher; guidance welcome. > > Questions > --------- > 1. Is a new in-tree format driver acceptable in principle, given > the format is stable, openly specified, and deployed at scale? Yes. There are other options like writing a FUSE, iSCSI, or NBD daemon. Linux ublk (https://www.kernel.org/doc/html/latest/block/ublk.html) is a newer option with potentially better performance than other daemon approaches. The advantage of a block driver is that it has better integration with QEMU and may avoid the need for privileges. The downside is that it's extra work to integrate with QEMU (and add libvirt support) that non-QEMU use cases don't benefit from the QEMU block driver. The choice is yours. There is no fundamental blocker from the QEMU side if you're willing to write the QEMU code and the format is an open spec. > 2. Is read-only-first a reasonable merge scope for the initial > series (with iotests and a docs/interop/ spec page)? Yes. > 3. For remote blobs: compose over block/curl, or fetch in-driver > with a local cache? QEMU's block layer is designed around graphs of block driver nodes. Reusing the curl block driver would be a natural choice that avoids code duplication. However, if you hit issues and it becomes clear that directly calling libcurl is a better solution, then that's an option too. > 4. CLI surface: point -drive at the OCI-style JSON image config > (which lists the layer blobs), or expose per-layer options? Both :). Per-layer options allow for run-time (re)configuration or hotplug using blockdev-add and similar monitor commands. Users may prefer to point QEMU at a JSON file rather than building long command-line options. > > I will write the iotests and the docs/interop format page, and add > myself to MAINTAINERS in the first series. You do not need to document the on-disk format in docs/interop/ if you already maintain the spec in https://github.com/containerd/overlaybd.git. It would be fine to include a comment in block/overlaybd.c with the spec URLs. If it helps we could have a call with Kevin Wolf in about 2 weeks to discuss any topics you have in more depth. Stefan
signature.asc
Description: PGP signature
