Hi Paolo,
thanks for this series!
Some comments below.
On 7/1/24 07:58, Paolo Bonzini wrote:
Hi all,
this is an example of what some bindings code for QEMU would look like.
Note that some parts of this code are barely expected to compile, and
are probably full of bugs, but still should look like finished code
(in fact, because they compile, the type system parts should be okay;
though a conditional "should" is required).
This code is not integrated in the QEMU source tree, because again it
is just a example of what kind of Rust code would exist to handle the
C<->Rust FFI. The translation of a handful of C structs and function
prototypes is done by hand rather than with bindgen, in particular.
The patches are organized as follows:
Patches 1-2 introduce the skeleton for the rest of the code and are
not particularly interesting, since that skeleton would be provided
by the patches that introduce Rust usage in QEMU.
Patches 3-4 define common code to handle conversion of data structures
between Rust and C. I couldn't find an existing crate to do this,
though there are similar concepts in glib-rs. The crate defines
variants of Clone, From and Into that convert a Rust struct to a
C pointer, for example:
let s = "abc".clone_to_foreign(); // mallocs a NULL-terminated copy
let p = s.as_ptr();
drop(s); // p is freed now
or the other way round:
let s = String::cloned_from_foreign(p); // p is a *const c_char
let t: String = p.into_native(); // p is a *mut c_char and is
free()d
The second patch defines what you need to convert strings from and to
C. It's used by tests but also by a couple of QOM functions implemented
below; it lets you forget the differences between String, &str, CString,
&CStr and Cow<'_, str>.
This is the only complete part of the skeleton (in fact I wrote it
a couple years ago), and it comes with testcases that pass in both
"#[test]" and "doctest" (i.e. snippets integrated in the documentation
comments) styles.
Patch 5 (mostly complete except for &error_abort support and missing
tests) allows using the above functionality for the QEMU Error*.
Patches 6-8 provide an example of how to use QOM from Rust. They
define all the functionality to:
- handle reference counting
- handle typesafe casting (i.e. code doesn't compile if an upcast
is invalid, or if a downcast cannot possibly succeed).
- faking inheritance since Rust doesn't support it.
While I started with some implementation ideas in glib-rs, this has
evolved quite a bit and doesn't really look much like glib-rs anymore.
I provided a handful of functions to wrap C APIs. For example:
object_property_add_child(obj, "foo", object_new(TYPE_BLAH))
becomes
obj.property_add_child("foo", Blah::new());
... except that the former leaks a reference and the latter does not
(gotcha :)). The idea is that these functions would be written, in
general, as soon as Rust code uses them, but I included a few as an
example of using the various abstractions for typecasting, reference
counting, and converting from/to C data strcutures.
A large part of this code is of the "it compiles and it looks nice, it
must be perfect" kind. Rust is _very_ picky on type safety, in case you
didn't know. One day we're all going to be programming in Haskell,
without even noticing it.
Patches 9-10 deal with how to define new subclasses in Rust. They are
a lot less polished and less ready. There is probably a lot of polish
that could be applied to make the code look nicer, but I guess there is
always time to make the code look nicer until the details are sorted out.
The things that I considered here are:
- splitting configuration from runtime state. Configuration is
immutable throughout the lifetime of the object, and holds the
value of user-configured properties. State will typically be a
Mutex<> or RefCell<> since the QOM bindings make wide use of interior
mutability---almost all functions are declared as &self---following
the lead of glib-rs.
- automatic generation of instance_init and instance_finalize. For
this I opted to introduce a new initialization step that is tailored to
Rust, called instance_mem_init(), that is executed before the default
value of properties is set. This makes sure that user code only ever
sees valid values for the whole struct, including after a downcast;
it matters because some Rust types (notably references) cannot be
initialized to a zero-bytes pattern. The default implementation of
instance_mem_init is simply a memset(), since the callback replaces the
memset(obj, 0, type->instance_size);
line in object_initialize_with_type(). I have prototyped this change
in QEMU already.
- generation of C vtables from safe code that is written in Rust. I
chose a trait that only contains associated constants as a way to
access the vtables generically. For example:
impl ObjectImpl for TestDevice {
const UNPARENT: Option<fn(&TestDevice)> = Some(TestDevice::unparent);
}
impl DeviceImpl for TestDevice {
const REALIZE: Option<fn(&TestDevice) -> Result<()>> =
Some(TestDevice::realize);
}
This works and it seems like a style that (in the future) we could apply
macro or even procedural macro magic to.
- generation of qdev property tables. While only boolean properties are
implemented here, one idea that I experimented with, is that the
default value of properties is derived from the ConstDefault trait.
(ConstDefault is provided by the const_default external crate). Again,
this is material for future conversion to procedural macros.
I absolutely didn't look at vmstate, but it shouldn't be too different
from properties, at least for the common cases.
Patches 11-14 finally are an example of the changes that are needed
to respect a minimum supported Rust version consistent with what is in
Debian Bullseye. It's not too bad, especially since the current version
of the QOM bindings does not require generic associated types anymore.
Why am I posting this? Because this kind of glue code is the ultimate
source of technical debt. It is the thing that we should be scared of
when introducing a new language in QEMU. It makes it harder to change C
code, and it is hard to change once Rust code becomes more widespread.
If we think a C API is not fully baked, we probably shouldn't write
Rust code that uses it (including bindings code). If we think a Rust
API is not fully baked, we probably shouldn't add too much Rust code
that uses it.
Shouldn't we focus more on how we want to write a QOM device in Rust
instead, by making abstraction of existing C implementation first?
Once we have satisfying idea, we could evaluate how it maps to existing
implementation, and which compromise should be made for efficiency.
It seems that the current approach is to stick to existing model, and
derive Rust bindings from this.
I agree this glue can be a source of technical debt, but on the other
side, it should be easy to refactor it, if we decided first what the
"clean and idiomatic" Rust API should look like.
A compromise where you have to manually translate some structs, or copy
memory to traverse languages borders at some point, could be worth the
safety and expressiveness.
We should have an idea of what this glue code looks like, in order to make
an informed choice. If we think we're not comfortable with reviewing it,
well, we should be ready to say so and stick with C until we are.
While it is important that this glue code is maintainable and looks
great, I don't think it should be the main reason to accept usage of a
new language.
The alternative could be to use Rust without this kind of binding. I
think it's a bad idea. It removes many of the advantages of Rust
(which are exemplified by the above object_property_add_child one-liner),
and it also introduces _new_ kinds of memory errors, since Rust has
its own undefined behavior conditions that are not there in C/C++.
For example:
impl Struct {
pub fn f(&self) {
call_some_c_function(Self::g, self as *const Self as *mut _);
}
fn do_g(&mut self) {
...
}
extern "C" fn g(ptr: *mut Self) {
unsafe { &mut *ptr }.do_g();
}
}
is invalid because a &mut reference (exclusive) is alive at the same time
as a & reference (shared). It is left as an exercise to the reader to
figure out all the possible ways in which we can shoot our own feet,
considering the pervasive use of callbacks in QEMU.
I agree with this point that you made on the original RFC series from
Manos. Just calling qemu C API through unsafe code is not making the
best possible usage of Rust.
With respect to callbacks, that's something that is missing in this
prototype. Fortunately, that's also something that will be tackled very
soon if the PL011 example is merged, because memory regions and character
devices both introduce them. Also, as I understand it, Rust code using
callbacks is not particularly nice anyway, though it is of course doable.
Instead, this exercise are about being able to write *nice* Rust code,
with all the advantages provided by the language, and the cost of
writing/maintaining the glue code that makes it possible. I expect
that we'll use a technique similar to the extern_c crate (it's 20 lines of
code; https://docs.rs/crate/extern-c/) to convert something that implements
Fn(), including a member function, into an extern "C" function.
It could be an interesting thing to explore. Is the best solution to use
specific traits (no callbacks involved from Rust code), or should we
have Rust closures that mimic this behavior.
We could have a specific trait with functions to handle various kind of
events. And the glue code could be responsible to translate this to
callbacks for the C side (and calling Rust code accordingly, eventually
serializing this on a single thread to avoid any race issues).
Anyhow: I think we can do it, otherwise I would not have written 2000 lines
of code (some of it two or three times). But if people are now scared and
think we shouldn't, well, that's also a success of its own kind.
Paolo
Paolo Bonzini (14):
add skeleton
set expectations
rust: define traits and pointer wrappers to convert from/to C
representations
rust: add tests for util::foreign
rust: define wrappers for Error
rust: define wrappers for basic QOM concepts
rust: define wrappers for methods of the QOM Object class
rust: define wrappers for methods of the QOM Device class
rust: add idiomatic bindings to define Object subclasses
rust: add idiomatic bindings to define Device subclasses
rust: replace std::ffi::c_char with libc::c_char
rust: replace c"" literals with cstr crate
rust: introduce alternative to offset_of!
rust: use version of toml_edit that does not require new Rust
