Re: [Libguestfs] [libnbd PATCH] rust: Use mio::poll instead of requiring epoll

[Tage Johansson]

On 9/4/2023 8:23 PM, Eric Blake wrote:

On Mon, Sep 04, 2023 at 04:07:54PM +, Tage Johansson wrote:

From: Eric Blake 

CI shows our async handle fails to build on FreeBSD and MacOS (where
epoll() is not available as a syscall, and therefore not available as
a Rust crate).  We can instead accomplish the same level-probing
effects by doing a zero-timeout poll with mio (which defers under the
hood to epoll on Linux, and kqueue on BSD).

Fixes: 223a9965 ("rust: async: Create an async friendly handle type")
CC: Tage Johansson 
Signed-off-by: Eric Blake 
---
  rust/Cargo.toml  |  3 ++-
  rust/src/async_handle.rs | 40 +++-
  2 files changed, 25 insertions(+), 18 deletions(-)

-// Use epoll to check the current read/write availability on the fd.
+// Use mio poll to check the current read/write availability on the fd.
  // This is needed because Tokio supports only edge-triggered
  // notifications but Libnbd requires level-triggered notifications.
-let mut revent = epoll::Event { data: 0, events: 0 };
  // Setting timeout to 0 means that it will return immediately.
-epoll::wait(epfd, 0, std::slice::from_mut( revent))?;
-let revents = Events::from_bits(revent.events).unwrap();
-if !revents.contains(Events::EPOLLIN) {
-ready_guard.clear_ready_matching(IoReady::READABLE);
-}
-if !revents.contains(Events::EPOLLOUT) {
-ready_guard.clear_ready_matching(IoReady::WRITABLE);
+// mio states that it is OS-dependent on whether a single event
+// can be both readable and writable, but we survive just fine
+// if we only see one direction even when both are available.
+poll.registry().register(
+ SourceFd(),
+Token(0),
+MioInterest::READABLE | MioInterest::WRITABLE,
+)?;
+poll.poll( events, Some(Duration::ZERO))?;

Why do we want 'poll.poll()?;', that is, to fail this function if the
poll returns an error?  We _expect_ poll to sometimes return an error
(namely, the fact that it timed out) if there is nothing pending on
the fd, at which point we WANT to successfully clear the ready_guard
for both read and write, rather than to error out of this function.



You are right. I thought that the poll() call would return Ok(()) upon 
timeout, but according to the documentation:


Currently if the timeout elapses without any readiness events 
triggering this will return Ok(()). However we’re not guaranteeing 
this behaviour as this depends on the OS.


So I guess it is best to ignore any errors from the poll call as in your 
patch.



--

Best regards,

Tage



+for event in  {
+if !event.is_readable() {
+ready_guard.clear_ready_matching(IoReady::READABLE);
+}
+if !event.is_writable() {
+ready_guard.clear_ready_matching(IoReady::WRITABLE);
+}
  }
  ready_guard.retain_ready();
+poll.registry().deregister( SourceFd())?;

Furthermore, if the regsiter()/deregister() should always occur in
pairs, the early poll()? exit breaks that assumption (I don't know if
Rust has enough smarts to automatically clean up the unpaired
registration).



___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH] rust: Use mio::poll instead of requiring epoll

[Tage Johansson]

From: Eric Blake 

CI shows our async handle fails to build on FreeBSD and MacOS (where
epoll() is not available as a syscall, and therefore not available as
a Rust crate).  We can instead accomplish the same level-probing
effects by doing a zero-timeout poll with mio (which defers under the
hood to epoll on Linux, and kqueue on BSD).

Fixes: 223a9965 ("rust: async: Create an async friendly handle type")
CC: Tage Johansson 
Signed-off-by: Eric Blake 
---
 rust/Cargo.toml  |  3 ++-
 rust/src/async_handle.rs | 40 +++-
 2 files changed, 25 insertions(+), 18 deletions(-)

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 0879b34..678848a 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -49,9 +49,10 @@ thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
 tokio = { optional = true, version = "1.29.1", default-features = false, 
features = ["rt", "sync", "net"] }
-epoll = "4.3.3"
+mio = { optional = true, version = "0.8.0" }
 
 [features]
+tokio = ["dep:tokio", "dep:mio"]
 default = ["log", "tokio"]
 
 [dev-dependencies]
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
index 6793ce9..8f3e8df 100644
--- a/rust/src/async_handle.rs
+++ b/rust/src/async_handle.rs
@@ -35,9 +35,11 @@ use crate::sys;
 use crate::Handle;
 use crate::{Error, FatalErrorKind, Result};
 use crate::{AIO_DIRECTION_BOTH, AIO_DIRECTION_READ, AIO_DIRECTION_WRITE};
-use epoll::Events;
+use mio::unix::SourceFd;
+use mio::{Events, Interest as MioInterest, Poll, Token};
 use std::sync::Arc;
 use std::sync::Mutex;
+use std::time::Duration;
 use tokio::io::{unix::AsyncFd, Interest, Ready as IoReady};
 use tokio::sync::Notify;
 use tokio::task;
@@ -176,13 +178,8 @@ async fn polling_task(handle_data: ) -> 
Result<(), FatalErrorKind> {
 } = handle_data;
 let fd = handle.aio_get_fd().map_err(Error::to_fatal)?;
 let tokio_fd = AsyncFd::new(fd)?;
-let epfd = epoll::create(false)?;
-epoll::ctl(
-epfd,
-epoll::ControlOptions::EPOLL_CTL_ADD,
-fd,
-epoll::Event::new(Events::EPOLLIN | Events::EPOLLOUT, 42),
-)?;
+let mut events = Events::with_capacity(1);
+let mut poll = Poll::new()?;
 
 // The following loop does approximately the following things:
 //
@@ -248,19 +245,28 @@ async fn polling_task(handle_data: ) -> 
Result<(), FatalErrorKind> {
 }
 drop(pending_cmds_lock);
 
-// Use epoll to check the current read/write availability on the fd.
+// Use mio poll to check the current read/write availability on the fd.
 // This is needed because Tokio supports only edge-triggered
 // notifications but Libnbd requires level-triggered notifications.
-let mut revent = epoll::Event { data: 0, events: 0 };
 // Setting timeout to 0 means that it will return immediately.
-epoll::wait(epfd, 0, std::slice::from_mut( revent))?;
-let revents = Events::from_bits(revent.events).unwrap();
-if !revents.contains(Events::EPOLLIN) {
-ready_guard.clear_ready_matching(IoReady::READABLE);
-}
-if !revents.contains(Events::EPOLLOUT) {
-ready_guard.clear_ready_matching(IoReady::WRITABLE);
+// mio states that it is OS-dependent on whether a single event
+// can be both readable and writable, but we survive just fine
+// if we only see one direction even when both are available.
+poll.registry().register(
+ SourceFd(),
+Token(0),
+MioInterest::READABLE | MioInterest::WRITABLE,
+)?;
+poll.poll( events, Some(Duration::ZERO))?;
+for event in  {
+if !event.is_readable() {
+ready_guard.clear_ready_matching(IoReady::READABLE);
+}
+if !event.is_writable() {
+ready_guard.clear_ready_matching(IoReady::WRITABLE);
+}
 }
 ready_guard.retain_ready();
+poll.registry().deregister( SourceFd())?;
 }
 }

base-commit: 9afb980d05d6144129c899285e44779757a380e8
prerequisite-patch-id: 8d1779610795021ed5a3d0973ddf9ef854cd6a24
prerequisite-patch-id: 1f0f11f11ac9b1ff0be08f5aa0a9904ba4de
prerequisite-patch-id: dc4af7343c57a4f99dc82918c07470030e542747
prerequisite-patch-id: 8e8f7a043c80d6c24e883967f5bd952a64db1228
prerequisite-patch-id: ba7b3482e2e16f76b5f285daeeda30b31a841912
prerequisite-patch-id: 219a9595e550a8caf43d466dcb2b044114e1b7bf
prerequisite-patch-id: 3de46c9673221bff1d897970aa983b3f8e6cab74
prerequisite-patch-id: 4235d5e174fce05b9a947b3b838bebf968f0fa6a
prerequisite-patch-id: 07773355d5718e0593c4030a8f035fc11fea3715
prerequisite-patch-id: f023deea8b706706e3c2980403064d90a254af3c
prerequisite-patch-id: dd415a31fa127d90bcc7d993a2659e39

Re: [Libguestfs] [libnbd PATCH 2/2] rust: Use mio::poll instead of requiring epoll

[Tage Johansson]

On 9/1/2023 3:41 PM, Eric Blake wrote:

On Fri, Sep 01, 2023 at 08:35:58AM +, Tage Johansson wrote:

On 8/30/2023 10:11 PM, Eric Blake wrote:

CI shows our async handle fails to build on FreeBSD and MacOS (where
epoll() is not available as a syscall, and therefore not available as
a Rust crate).  We can instead accomplish the same level-probing
effects by doing a zero-timeout poll with mio (which defers under the
hood to epoll on Linux, and kqueue on BSD).


The problem with mio is that it uses edge triggered notifications. According
to this page <https://docs.rs/mio/latest/mio/struct.Poll.html>:


Draining readiness
Once a readiness event is received, the corresponding operation must be
performed repeatedly until it returns variant
std::io::error::ErrorKind::WouldBlock. Unless this is done, there is no
guarantee that another readiness event will be delivered, even if
further data is received for the event source.

Would that still be true even if we deregister the interest after our
one-shot poll, and reregister it immediately before-hand?  In other
words, even if registration sets up edge triggers for future
notifications, the fact that we are only doing a one-shot query seems
like it would be sufficient to use our one-shot as a level probe
(assuming that the registration starts by learning the current state
of the fd before waiting for any future edges).

I agree that once we get a notification after a .poll(), the normal
assumption is that we must loop and consume all data before calling
.poll again (because the .poll would block until the next edge - but
if we didn't consume to the blocking point, there is no next edge).
But since we are NOT looping, nor doing any consumption work, our
timeout of 0 is trying to behave as an instant level check, and if
adding a reregister/deregister wrapper around the .poll makes it more
reliable (by wiping out all prior events and re-priming the
registration to start with the current level), that may be all the
more we need.



It should work. I tried to do that and it works on my machine at least. 
Although it might be a bit uggly, it may be a short term solution. I 
will send your patch with these modifications soon as reply to this message.




The motivation behind using epoll in addition to tokio in the first place
was to check if fd is readable/writable without necessarily knowing if the
last read/write operation returned [ErrorKind::WouldBlock]. And it doesn't
seems that mio full fills that requirenment.

Do we need to expand the libnbd API itself to make it more obvious
whether our state machine completed because it hit a EWOULDBLOCK
scenario?  In general, once you kick the state machine (by sending a
new command, or calling one of the two aio_notify calls), the state
machine tries to then process as much data as possible until it blocks
again, rather than stopping after just processing a single
transaction.  That is, if libnbd isn't already designed to work with
edge triggers, how much harder would it be to modify it (perhaps by
adding a new API) so that it DOES work nicely with edge triggers?



This would be the cleanest and best solution I think. Adding two methods 
to the handle: aio_read_blocked() and aio_write_blocked() which return 
true iff the last read/write operation blocked. It could be two bool 
fields on the handle struct which are set to false by default and reset 
to false whenever a read/write operation is performed without blocking. 
It is very important that aio_read_blocked() will return true only if 
the last read operation blocked, (and the same thing for 
aio_write_blocked()).




If we can solve _that_ problem, then the Rust async handle wouldn't
need to use epoll, mio, or anything else; tokio's edge trigger
behavior would already be sufficient on its own.



Yes, this is true. It would be more efficient, easier to understand, and 
require less dependencies.



--

Best regards,

Tage


___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH] rust: Cleanups in examples/concurrent-read-write.rs.

[Tage Johansson]

This patch makes some small cleanups in
rust/examples/concurrent-read-write.rs. Specificly, it refrases one
comment, removes some outcommented code, and removes a completely
redundent if statement. It also replaces a hard coded number with the
index of the task as seed to an RNG.
---
 rust/examples/concurrent-read-write.rs | 11 ---
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
index 4858f76..589cefd 100644
--- a/rust/examples/concurrent-read-write.rs
+++ b/rust/examples/concurrent-read-write.rs
@@ -3,11 +3,10 @@
 //!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
 //! Or connect over a URI:
 //! nbdkit -U - memory 100M \
-//!   --run 'cargo run --example concurrent-read-write -- $uri'
+//!   --run 'cargo run --example concurrent-read-write -- "$uri"'
 //!
-//! This will read and write randomly over the first megabyte of the
-//! plugin using multi-conn, multiple threads and multiple requests in
-//! flight on each thread.
+//! This will read and write randomly over the plugin using multi-conn,
+//! multiple threads and multiple requests in flight on each thread.
 
 #![deny(warnings)]
 use rand::prelude::*;
@@ -111,12 +110,11 @@ async fn run_thread(
 nbd.connect_unix(socket_or_uri).await?;
 }
 
-let mut rng = SmallRng::seed_from_u64(44 as u64);
+let mut rng = SmallRng::seed_from_u64(task_idx as u64);
 
 // Issue commands.
 let mut stats = Stats::default();
 let mut join_set = JoinSet::new();
-//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
 while stats.requests < NR_CYCLES || !join_set.is_empty() {
 while stats.requests < NR_CYCLES && join_set.len() < MAX_IN_FLIGHT {
 // If we want to issue another request, do so.  Note that we reuse
@@ -144,6 +142,5 @@ async fn run_thread(
 join_set.join_next().await.unwrap().unwrap()?;
 }
 
-if task_idx == 0 {}
 Ok(stats)
 }

base-commit: 9afb980d05d6144129c899285e44779757a380e8
prerequisite-patch-id: 8d1779610795021ed5a3d0973ddf9ef854cd6a24
prerequisite-patch-id: 1f0f11f11ac9b1ff0be08f5aa0a9904ba4de
prerequisite-patch-id: dc4af7343c57a4f99dc82918c07470030e542747
prerequisite-patch-id: 8e8f7a043c80d6c24e883967f5bd952a64db1228
prerequisite-patch-id: ba7b3482e2e16f76b5f285daeeda30b31a841912
prerequisite-patch-id: 219a9595e550a8caf43d466dcb2b044114e1b7bf
prerequisite-patch-id: 3de46c9673221bff1d897970aa983b3f8e6cab74
prerequisite-patch-id: 4235d5e174fce05b9a947b3b838bebf968f0fa6a
prerequisite-patch-id: 07773355d5718e0593c4030a8f035fc11fea3715
prerequisite-patch-id: f023deea8b706706e3c2980403064d90a254af3c
prerequisite-patch-id: 8639c6cc4fec58f4761771c5d8a9476d538c6251
prerequisite-patch-id: 9bc660aed54a6266b014993ff0f388a26ac2982a
-- 
2.42.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v9 6/7] rust: async: Add an example

[Tage Johansson]

On 9/1/2023 10:41 PM, Eric Blake wrote:

On Sat, Aug 26, 2023 at 11:29:59AM +, Tage Johansson wrote:

This patch adds an example using the asynchronous Rust bindings.
---
  rust/Cargo.toml|   1 +
  rust/examples/concurrent-read-write.rs | 149 +
  rust/run-tests.sh.in   |   2 +
  3 files changed, 152 insertions(+)
  create mode 100644 rust/examples/concurrent-read-write.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index c49f9f2..0879b34 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -58,5 +58,6 @@ default = ["log", "tokio"]
  anyhow = "1.0.72"
  once_cell = "1.18.0"
  pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
  tempfile = "3.6.0"
  tokio = { version = "1.29.1", default-features = false, features = ["rt-multi-thread", 
"macros"] }
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..4858f76
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,149 @@
+//! Example usage with nbdkit:
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//! Or connect over a URI:
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $uri'

Should be "$uri" here (to avoid accidental shell globbing surprises).


+//!
+//! This will read and write randomly over the first megabyte of the

This says first megabyte...[1]



If I understand my code correctly, it is actually reading and writing 
randomly over the entire memory.



+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;

1024 isn't much larger than 512.  It looks like you borrowed heavily
from examples/threaded-reads-and-writes.c, but that used 1M as the
large buffer.



The reason for this is that we can't read and write to a shared buffer 
simultaneously in safe Rust. So the buffer gets created on the fly for 
each read/write operation. And creating a 1M buffer so many times seemd 
like a bit too much memory comsumtion.




+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+
+// Check if the user provided a URI or a unix socket.
+let socket_or_uri = args[1].to_str().unwrap();
+if socket_or_uri.contains("://") {
+nbd.connect_uri(socket_or_uri)?;
+} else {
+nbd.connect_unix(socket_or_uri)?;
+}
+
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket_or_uri.to_owned(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were requi

[Libguestfs] [libnbd PATCH 2/2] rust: Add missing files to EXTRA_DIST

[Tage Johansson]

---
 rust/Makefile.am | 1 +
 1 file changed, 1 insertion(+)

diff --git a/rust/Makefile.am b/rust/Makefile.am
index 027097a..c5933e5 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -36,6 +36,7 @@ source_files = \
examples/connect-command.rs \
examples/get-size.rs \
examples/fetch-first-sector.rs \
+   examples/concurrent-read-write.rs \
libnbd-sys/Cargo.toml \
libnbd-sys/build.rs \
libnbd-sys/src/lib.rs \
-- 
2.42.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH 1/2] generator: Add string_starts_with functions to utils.ml

[Tage Johansson]

Previously, the file Rust.ml used the function String.starts_with. But
this function is not available in older versions of OCaml. So an
identical function (string_starts_with) has been created in utils.ml to
be used instead.
---
 generator/utils.ml  | 11 +++
 generator/utils.mli |  3 +++
 2 files changed, 14 insertions(+)

diff --git a/generator/utils.ml b/generator/utils.ml
index 96c15ac..9f97aa7 100644
--- a/generator/utils.ml
+++ b/generator/utils.ml
@@ -570,3 +570,14 @@ let camel_case name =
   a ^ String.uppercase_ascii (Str.first_chars x 1) ^
   String.lowercase_ascii (Str.string_after x 1)
   ) "" xs
+
+(* Copied from OCaml's stdlib because it was not available in earlier versions
+   of OCaml. *)
+let string_starts_with ~prefix s =
+  let len_s = String.length s
+  and len_pre = String.length prefix in
+  let rec aux i =
+if i = len_pre then true
+else if String.unsafe_get s i <> String.unsafe_get prefix i then false
+else aux (i + 1)
+  in len_s >= len_pre && aux 0
diff --git a/generator/utils.mli b/generator/utils.mli
index c7681a9..d2def0b 100644
--- a/generator/utils.mli
+++ b/generator/utils.mli
@@ -80,3 +80,6 @@ val pod2text : cache_key -> cache_value
 
 (* Convert C function name to upper-camel-case name. *)
 val camel_case : string -> string
+
+(* Same as [String.starts_with] in later versions of OCaml. *)
+val string_starts_with : prefix:string -> string -> bool
-- 
2.42.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH 0/2] (Attempt to) fix some Rust CI failures

[Tage Johansson]

These are two minor fixes to the Rust bindings to fix some CI failures.

--
Best regards,
Tage


Tage Johansson (2):
  generator: Add string_starts_with functions to utils.ml
  rust: Add missing files to EXTRA_DIST

 generator/utils.ml  | 11 +++
 generator/utils.mli |  3 +++
 rust/Makefile.am|  1 +
 3 files changed, 15 insertions(+)


base-commit: 9afb980d05d6144129c899285e44779757a380e8
prerequisite-patch-id: 8d1779610795021ed5a3d0973ddf9ef854cd6a24
prerequisite-patch-id: 1f0f11f11ac9b1ff0be08f5aa0a9904ba4de
prerequisite-patch-id: dc4af7343c57a4f99dc82918c07470030e542747
prerequisite-patch-id: 8e8f7a043c80d6c24e883967f5bd952a64db1228
prerequisite-patch-id: ba7b3482e2e16f76b5f285daeeda30b31a841912
prerequisite-patch-id: 219a9595e550a8caf43d466dcb2b044114e1b7bf
prerequisite-patch-id: 3de46c9673221bff1d897970aa983b3f8e6cab74
prerequisite-patch-id: 4235d5e174fce05b9a947b3b838bebf968f0fa6a
prerequisite-patch-id: 07773355d5718e0593c4030a8f035fc11fea3715
prerequisite-patch-id: f023deea8b706706e3c2980403064d90a254af3c
--
2.42.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH 2/2] rust: Use mio::poll instead of requiring epoll

[Tage Johansson]

On 8/30/2023 10:11 PM, Eric Blake wrote:

CI shows our async handle fails to build on FreeBSD and MacOS (where
epoll() is not available as a syscall, and therefore not available as
a Rust crate).  We can instead accomplish the same level-probing
effects by doing a zero-timeout poll with mio (which defers under the
hood to epoll on Linux, and kqueue on BSD).



The problem with mio is that it uses edge triggered notifications. 
According to this page <https://docs.rs/mio/latest/mio/struct.Poll.html>:



Draining readiness
Once a readiness event is received, the corresponding operation must 
be performed repeatedly until it returns variant 
std::io::error::ErrorKind::WouldBlock. Unless this is done, there is 
no guarantee that another readiness event will be delivered, even if 
further data is received for the event source.


The motivation behind using epoll in addition to tokio in the first 
place was to check if fd is readable/writable without necessarily 
knowing if the last read/write operation returned 
[ErrorKind::WouldBlock]. And it doesn't seems that mio full fills that 
requirenment.



I tried using the polling <https://docs.rs/polling/latest/polling/> 
crate, which uses epoll and kqueue under the hood. but for some reason 
the disconnect call failes, at least on my Linux machine. I have know 
idea what's the problem.



Another idea would be that we create our own abstraction upon epoll and 
kqueue. But I am unable to test any kqueue implementation, I think that 
I can not even compile it on my machine.



It would of course also be possible to disable the Rust bindings (or at 
least the asynchronous API) on BSD and MacOS, but that would not be a 
good solution I think.



--

Best regards,

Tage




Fixes: 223a9965 ("rust: async: Create an async friendly handle type")
CC: Tage Johansson
Signed-off-by: Eric Blake
---
  rust/Cargo.toml  |  2 +-
  rust/src/async_handle.rs | 46 +---
  2 files changed, 30 insertions(+), 18 deletions(-)

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 0879b34c..391c80e9 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -49,7 +49,7 @@ thiserror = "1.0.40"
  log = { version = "0.4.19", optional = true }
  libc = "0.2.147"
  tokio = { optional = true, version = "1.29.1", default-features = false, features = ["rt", 
"sync", "net"] }
-epoll = "4.3.3"
+mio = "0.8.0"

  [features]
  default = ["log", "tokio"]
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
index ab28b910..35b1c0d2 100644
--- a/rust/src/async_handle.rs
+++ b/rust/src/async_handle.rs
@@ -35,9 +35,11 @@ use crate::sys;
  use crate::Handle;
  use crate::{Error, FatalErrorKind, Result};
  use crate::{AIO_DIRECTION_BOTH, AIO_DIRECTION_READ, AIO_DIRECTION_WRITE};
-use epoll::Events;
+use mio::unix::SourceFd;
+use mio::{Events, Interest as MioInterest, Poll, Token};
  use std::sync::Arc;
  use std::sync::Mutex;
+use std::time::Duration;
  use tokio::io::{unix::AsyncFd, Interest, Ready as IoReady};
  use tokio::sync::Notify;
  use tokio::task;
@@ -176,12 +178,12 @@ async fn polling_task(handle_data: ) -> 
Result<(), FatalErrorKind> {
  } = handle_data;
  let fd = handle.aio_get_fd().map_err(Error::to_fatal)?;
  let tokio_fd = AsyncFd::new(fd)?;
-let epfd = epoll::create(false)?;
-epoll::ctl(
-epfd,
-epoll::ControlOptions::EPOLL_CTL_ADD,
-fd,
-epoll::Event::new(Events::EPOLLIN | Events::EPOLLOUT, 42),
+let mut events = Events::with_capacity(1);
+let mut poll = Poll::new()?;
+poll.registry().register(
+ SourceFd(),
+Token(0),
+MioInterest::READABLE | MioInterest::WRITABLE,
  )?;

  // The following loop does approximately the following things:
@@ -248,19 +250,29 @@ async fn polling_task(handle_data: ) -> 
Result<(), FatalErrorKind> {
  }
  drop(pending_cmds_lock);

-// Use epoll to check the current read/write availability on the fd.
+// Use mio poll to check the current read/write availability on the fd.
  // This is needed because Tokio supports only edge-triggered
  // notifications but Libnbd requires level-triggered notifications.
-let mut revent = epoll::Event { data: 0, events: 0 };
  // Setting timeout to 0 means that it will return immediately.
-epoll::wait(epfd, 0, std::slice::from_mut( revent))?;
-let revents = Events::from_bits(revent.events).unwrap();
-if !revents.contains(Events::EPOLLIN) {
-ready_guard.clear_ready_matching(IoReady::READABLE);
-}
-if !revents.contains(Events::EPOLLOUT) {
-ready_guard.clear_ready_matching(IoReady::WRITABLE);
-}
+// mio states that it is OS-dependent on whether a single event
+// can be both readable and wri

Re: [Libguestfs] [libnbd PATCH v9 5/7] rust: async: Add a couple of integration tests

[Tage Johansson]

On 8/30/2023 10:19 PM, Eric Blake wrote:

On Wed, Aug 30, 2023 at 12:16:07PM -0500, Eric Blake wrote:

error[E0425]: cannot find value `EPOLL_CTL_ADD` in crate `libc`

https://www.reddit.com/r/rust/comments/65kflg/does_rust_have_native_epoll_support/
mentions how epoll is Linux-specific, and has comments about tokio
trying to build on top of mio in order to be platform-independent (mio
uses epoll on Linux where it is available, but does not require epoll
on other platforms).

So I spent a couple hours experimenting with whether I could use
mio::poll instead of epoll, and at least got things compiling on
FreeBSD with the tests still passing on Linux (I'm not quite set up to
actually prove that the tests work on FreeBSD).  Review appreciated:

https://listman.redhat.com/archives/libguestfs/2023-August/032464.html



Interesting. I tried with the polling 
 crate in order to get a cross 
platform version of epoll. But the disconnect call failes for some 
reason then.



I'll try to take a look at mio later today.


--

Best regards,
Tage

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v9 7/7] rust: Check that compilation works with default features disabled

[Tage Johansson]

The Rust crate has some default features, (tokio and log), to enable the
asynchronous API and logging facilities. This patch extends the test
suite to check that the crate compiles even with these features
disabled. The crate did in fact not compile without those features,
so the patch adds a "#[allow(unused)]" attribute to the utils
module to mittigate an error.
---
 rust/run-tests.sh.in | 1 +
 rust/src/lib.rs  | 1 +
 2 files changed, 2 insertions(+)

diff --git a/rust/run-tests.sh.in b/rust/run-tests.sh.in
index 661c018..b24bb7e 100755
--- a/rust/run-tests.sh.in
+++ b/rust/run-tests.sh.in
@@ -26,6 +26,7 @@ requires @NBDKIT@ floppy --version
 requires @NBDKIT@ memory --version
 
 if [ -z "$VG" ]; then
+@CARGO@ c --no-default-features
 @CARGO@ test -- --nocapture
 @CARGO@ run --example connect-command
 @NBDKIT@ -U - memory 1M \
diff --git a/rust/src/lib.rs b/rust/src/lib.rs
index 56316b4..1e1ca2e 100644
--- a/rust/src/lib.rs
+++ b/rust/src/lib.rs
@@ -25,6 +25,7 @@ mod bindings;
 mod error;
 mod handle;
 pub mod types;
+#[allow(unused)]
 mod utils;
 #[cfg(feature = "tokio")]
 pub use async_bindings::*;
-- 
2.42.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v9 6/7] rust: async: Add an example

[Tage Johansson]

This patch adds an example using the asynchronous Rust bindings.
---
 rust/Cargo.toml|   1 +
 rust/examples/concurrent-read-write.rs | 149 +
 rust/run-tests.sh.in   |   2 +
 3 files changed, 152 insertions(+)
 create mode 100644 rust/examples/concurrent-read-write.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index c49f9f2..0879b34 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -58,5 +58,6 @@ default = ["log", "tokio"]
 anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
 tempfile = "3.6.0"
 tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..4858f76
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,149 @@
+//! Example usage with nbdkit:
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//! Or connect over a URI:
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $uri'
+//!
+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+
+// Check if the user provided a URI or a unix socket.
+let socket_or_uri = args[1].to_str().unwrap();
+if socket_or_uri.contains("://") {
+nbd.connect_uri(socket_or_uri)?;
+} else {
+nbd.connect_unix(socket_or_uri)?;
+}
+
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket_or_uri.to_owned(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket_or_uri: String,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// futures must be `'static`, hence we wrap the handle in an [Arc].
+let nbd = Arc::new(libnbd::AsyncHandle::new()?);
+
+// Check if the user provided a URI or a unix socket.
+if socket_or_uri.contains("://") {
+nbd.connect_uri(socket_or_uri).await?;
+} else {
+nbd.connect_unix(socket_or_uri).await?;
+}
+
+let mut rng = SmallRng::seed_from_u64(44 as u64);
+
+// Issue commands.
+let mut stats = Stats::default();
+let mut join_set = JoinSet::new();
+//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+while stats.requests < 

[Libguestfs] [libnbd PATCH v9 4/7] rust: async: Use the modifies_fd flag to exclude calls

[Tage Johansson]

All handle calls which has the modifies_fd flag set to true will be
excluded from AsyncHandle (the asynchronous handle in the rust
bindings). This is a better approach then listing all calls that should
be excluded in Rust.ml explicetly.
---
 generator/Rust.ml | 29 -
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 3ee53bc..c86f9f7 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -574,18 +574,17 @@ let generate_rust_bindings () =
 
 let excluded_handle_calls : NameSet.t =
   NameSet.of_list
-[
-  "aio_get_fd";
-  "aio_get_direction";
-  "aio_notify_read";
-  "aio_notify_write";
-  "clear_debug_callback";
-  "get_debug";
-  "poll";
-  "poll2";
-  "set_debug";
-  "set_debug_callback";
-]
+  @@ [
+   "aio_get_fd";
+   "aio_get_direction";
+   "clear_debug_callback";
+   "get_debug";
+   "set_debug";
+   "set_debug_callback";
+ ]
+  @ (handle_calls
+|> List.filter (fun (_, { modifies_fd }) -> modifies_fd)
+|> List.map (fun (name, _) -> name))
 
 (* A mapping with names as keys. *)
 module NameMap = Map.Make (String)
@@ -615,11 +614,7 @@ let async_handle_calls : (string * call * async_kind) 
NameMap.t =
 let sync_handle_calls : call NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
-  |> List.filter (fun (name, _) ->
- (not (NameMap.mem name async_handle_calls))
- && not
-  (String.starts_with ~prefix:"aio_" name
-  && NameMap.mem (strip_aio name) async_handle_calls))
+  |> List.filter (fun (n, _) -> not (NameMap.mem n async_handle_calls))
   |> List.to_seq |> NameMap.of_seq
 
 (* Get the Rust type for an argument in the asynchronous API. Like
-- 
2.42.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v9 3/7] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

Add a flag (modifies_fd) to the call structure in generator/API.ml
which is set to true if the handle call may do something with the
file descriptor. That is, it is true for all calls which are or may
call aio_notify_*, including all synchronous commands like
nbd_connect or nbd_opt_go.

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to true will be
excluded from that handle. The asynchronous (aio_*) functions will be
used instead.
---
 generator/API.ml  | 32 
 generator/API.mli |  7 +++
 2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 99fcb82..b0f5e2a 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
   is_locked : bool;
   may_set_error : bool;
   async_kind : async_kind option;
+  modifies_fd: bool;
   mutable first_version : int * int;
 }
 and arg =
@@ -255,6 +256,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  permitted_states = [];
  is_locked = true; may_set_error = true;
  async_kind = None;
+ modifies_fd = false;
  first_version = (0, 0) }
 
 (* Calls.
@@ -1162,6 +1164,7 @@ Return true if option negotiation mode was enabled on 
this handle.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -1189,6 +1192,7 @@ although older servers will instead have killed the 
connection.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 Request that the server finish negotiation, gracefully if possible, then
@@ -1202,6 +1206,7 @@ enabled option mode.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to initiate TLS";
 longdesc = "\
 Request that the server initiate a secure TLS connection, by
@@ -1240,6 +1245,7 @@ established, as reported by 
L.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to enable structured replies";
 longdesc = "\
 Request that the server use structured replies, by sending
@@ -1266,6 +1272,7 @@ later calls to this function return false.";
 default_call with
 args = [ Closure list_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to list all exports during negotiation";
 longdesc = "\
 Request that the server list all exports that it supports.  This can
@@ -1307,6 +1314,7 @@ description is set with I<-D>.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server for information about an export";
 longdesc = "\
 Request that the server supply information about the export name
@@ -1338,6 +1346,7 @@ corresponding L would succeed.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using implicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1393,6 +1402,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using explicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1443,6 +1453,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using implicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1502,6 +1513,7 @@ no contexts are reported, or may fail but have a 
non-empty list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using explicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1731,6 

[Libguestfs] [libnbd PATCH v9 5/7] rust: async: Add a couple of integration tests

[Tage Johansson]

Add a couple of integration tests as rust/tests/test_async_*.rs. They
are very similar to the tests for the synchronous API.
---
 rust/Cargo.toml   |   1 +
 rust/tests/test_async_100_handle.rs   |  25 +++
 rust/tests/test_async_200_connect_command.rs  |  26 +++
 rust/tests/test_async_210_opt_abort.rs|  32 
 rust/tests/test_async_220_opt_list.rs |  86 ++
 rust/tests/test_async_230_opt_info.rs | 122 ++
 rust/tests/test_async_240_opt_list_meta.rs| 150 ++
 .../test_async_245_opt_list_meta_queries.rs   |  94 +++
 rust/tests/test_async_250_opt_set_meta.rs | 125 +++
 .../test_async_255_opt_set_meta_queries.rs| 110 +
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 +++
 rust/tests/test_async_460_block_status.rs |  98 
 rust/tests/test_async_620_stats.rs|  69 
 15 files changed, 1121 insertions(+)
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index a9b5988..c49f9f2 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -59,3 +59,4 @@ anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
 tempfile = "3.6.0"
+tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/tests/test_async_100_handle.rs 
b/rust/tests/test_async_100_handle.rs
new file mode 100644
index 000..e50bad9
--- /dev/null
+++ b/rust/tests/test_async_100_handle.rs
@@ -0,0 +1,25 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+//! Just check that we can link with libnbd and create a handle.
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_nbd_handle_new() {
+let _ = libnbd::AsyncHandle::new().unwrap();
+}
diff --git a/rust/tests/test_async_200_connect_command.rs 
b/rust/tests/test_async_200_connect_command.rs
new file mode 100644
index 000..2f4e7cc
--- /dev/null
+++ b/rust/tests/test_async_200_connect_command.rs
@@ -0,0 +1,26 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_connect_command() {
+let nbd = libnbd::AsyncHandle::new().unwrap();
+nbd.connect_command(&["nbdkit", "-s", "--exit-with-parent", "-v", "null"])
+.await
+.unwrap();
+}
diff --git a/rust/tests/test_async_210_opt_abort.rs 
b/rust/tests/test_a

[Libguestfs] [libnbd PATCH v9 1/7] generator: Add information about asynchronous handle calls

[Tage Johansson]

A new field (async_kind) is added to the call data type in
generator/API.ml*. The purpose is to tell if a certain handle call is
an asynchronous command and if so how one can know when it is
completed. The motivation for this is that all asynchronous commands
on the AsyncHandle in the Rust bindings make use of Rust's
[`async fn`s](https://doc.rust-lang.org/std/keyword.async.html).
But to implement such an async fn, the API needs to know when the
command completed, either by a completion callback or via a change
of state.
---
 generator/API.ml  | 32 
 generator/API.mli | 11 +++
 2 files changed, 43 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 72c8165..99fcb82 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -32,6 +32,7 @@ type call = {
   permitted_states : permitted_state list;
   is_locked : bool;
   may_set_error : bool;
+  async_kind : async_kind option;
   mutable first_version : int * int;
 }
 and arg =
@@ -102,6 +103,9 @@ and permitted_state =
 | Negotiating
 | Connected
 | Closed | Dead
+and async_kind =
+| WithCompletionCallback
+| ChangesState of string * bool
 and link =
 | Link of string
 | SectionLink of string
@@ -250,6 +254,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  see_also = [];
  permitted_states = [];
  is_locked = true; may_set_error = true;
+ async_kind = None;
  first_version = (0, 0) }
 
 (* Calls.
@@ -2802,6 +2807,7 @@ wait for an L.";
 default_call with
 args = [ SockAddrAndLen ("addr", "addrlen") ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Begin connecting to the NBD server.  The C and C
@@ -2814,6 +2820,7 @@ parameters specify the address of the socket to connect 
to.
 default_call with
 args = [ String "uri" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to an NBD URI";
 longdesc = "\
 Begin connecting to the NBD URI C.  Parameters behave as
@@ -2827,6 +2834,7 @@ documented in L.
 default_call with
 args = [ Path "unixsocket" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a Unix domain socket";
 longdesc = "\
 Begin connecting to the NBD server over Unix domain socket
@@ -2841,6 +2849,7 @@ L.
 default_call with
 args = [ UInt32 "cid"; UInt32 "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over AF_VSOCK socket";
 longdesc = "\
 Begin connecting to the NBD server over the C
@@ -2854,6 +2863,7 @@ L.
 default_call with
 args = [ String "hostname"; String "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a TCP port";
 longdesc = "\
 Begin connecting to the NBD server listening on C.
@@ -2866,6 +2876,7 @@ Parameters behave as documented in L.
 default_call with
 args = [ Fd "sock" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect directly to a connected socket";
 longdesc = "\
 Begin connecting to the connected socket C.
@@ -2878,6 +2889,7 @@ Parameters behave as documented in 
L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it over
@@ -2891,6 +2903,7 @@ L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect using systemd socket activation";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it using
@@ -2907,6 +2920,7 @@ L.
 optargs = [ OClosure completion_closure ];
 ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some WithCompletionCallback;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -2930,6 +2944,7 @@ when L returns true.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 

[Libguestfs] [libnbd PATCH v9 2/7] rust: async: Create an async friendly handle type

[Tage Johansson]

quot;let (ret_tx, ret_rx) = oneshot::channel::>();\n";
+  pr "let mut ret_tx = Some(ret_tx);\n";
+  pr "let completion_predicate = \n";
+  pr " move |handle: , res: <()>| {\n";
+  pr "  let ret = if let Err(_) = res {\n";
+  pr "res.clone()\n";
+  pr "  } else {\n";
+  pr "if handle.%s() != %s { return false; }\n" predicate value;
+  pr "else { Ok(()) }\n";
+  pr "  };\n";
+  pr "  ret_tx.take().unwrap().send(ret).ok();\n";
+  pr "  true\n";
+  pr "};\n";
+  pr "self.add_command(completion_predicate)?;\n";
+  pr "ret_rx.await.unwrap()\n";
+  pr "}\n\n"
+
+(* Print an impl with all handle calls. *)
+let print_rust_async_handle_impls () =
+  pr "impl AsyncHandle {\n";
+  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
+  async_handle_calls
+  |> NameMap.iter (fun name (aio_name, call, async_kind) ->
+ match async_kind with
+ | WithCompletionCallback ->
+ print_rust_async_handle_call_with_completion_cb name aio_name
+   call
+ | ChangesState (predicate, value) ->
+ print_rust_async_handle_call_changing_state name aio_name call
+   (predicate, value));
+  pr "}\n\n"
+
+let print_rust_async_imports () =
+  pr "use crate::{*, types::*};\n";
+  pr "use os_socketaddr::OsSocketAddr;\n";
+  pr "use std::ffi::*;\n";
+  pr "use std::mem;\n";
+  pr "use std::net::SocketAddr;\n";
+  pr "use std::os::fd::{AsRawFd, OwnedFd};\n";
+  pr "use std::os::unix::prelude::*;\n";
+  pr "use std::path::PathBuf;\n";
+  pr "use std::ptr;\n";
+  pr "use std::sync::Arc;\n";
+  pr "use tokio::sync::oneshot;\n";
+  pr "\n"
+
+let generate_rust_async_bindings () =
+  generate_header CStyle ~copyright:"Tage Johansson";
+  pr "\n";
+  print_rust_async_imports ();
+  print_rust_async_handle_impls ()
diff --git a/generator/Rust.mli b/generator/Rust.mli
index 450e4ca..0960170 100644
--- a/generator/Rust.mli
+++ b/generator/Rust.mli
@@ -18,3 +18,5 @@
 
 (* Print all flag-structs, enums, constants and handle calls in Rust code. *)
 val generate_rust_bindings : unit -> unit
+
+val generate_rust_async_bindings : unit -> unit
diff --git a/generator/generator.ml b/generator/generator.ml
index 8c9a585..11bec4d 100644
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -69,3 +69,5 @@ let () =
 RustSys.generate_rust_sys_bindings;
   output_to ~formatter:(Some Rustfmt) "rust/src/bindings.rs"
 Rust.generate_rust_bindings;
+  output_to ~formatter:(Some Rustfmt) "rust/src/async_bindings.rs"
+Rust.generate_rust_async_bindings;
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 01555de..a9b5988 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -48,9 +48,11 @@ os_socketaddr = "0.2.4"
 thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
+tokio = { optional = true, version = "1.29.1", default-features = false, 
features = ["rt", "sync", "net"] }
+epoll = "4.3.3"
 
 [features]
-default = ["log"]
+default = ["log", "tokio"]
 
 [dev-dependencies]
 anyhow = "1.0.72"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index 7098c9a..2b5b85b 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -19,6 +19,7 @@ include $(top_srcdir)/subdir-rules.mk
 
 generator_built = \
libnbd-sys/src/generated.rs \
+   src/async_bindings.rs \
src/bindings.rs \
$(NULL)
 
@@ -30,6 +31,7 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   src/async_handle.rs \
examples/connect-command.rs \
examples/get-size.rs \
examples/fetch-first-sector.rs \
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
new file mode 100644
index 000..ab28b91
--- /dev/null
+++ b/rust/src/async_handle.rs
@@ -0,0 +1,266 @@
+// nbd client library in userspace
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; 

[Libguestfs] [libnbd PATCH v9 0/7] Rust Bindings for Libnbd

[Tage Johansson]

Compared to v8, this patch series corrects some grammatical errors in
patch 1 and 2. Some unrelated formatting has also been removed from
patch 2. Finally, a patch has been added wich extends the Rust test
suite by checking that the Rust crate compiles even with all features
disabled.

--
Best regards,
Tage

Tage Johansson (7):
  generator: Add information about asynchronous handle calls
  rust: async: Create an async friendly handle type
  generator: Add `modifies_fd` flag to the [call] structure
  rust: async: Use the modifies_fd flag to exclude calls
  rust: async: Add a couple of integration tests
  rust: async: Add an example
  rust: Check that compilation works with default features disabled

 generator/API.ml  |  64 +
 generator/API.mli |  18 ++
 generator/Rust.ml | 231 +++
 generator/Rust.mli|   2 +
 generator/generator.ml|   2 +
 rust/Cargo.toml   |   6 +-
 rust/Makefile.am  |   2 +
 rust/examples/concurrent-read-write.rs| 149 ++
 rust/run-tests.sh.in  |   3 +
 rust/src/async_handle.rs  | 266 ++
 rust/src/lib.rs   |   9 +
 rust/src/utils.rs |   9 +
 rust/tests/test_async_100_handle.rs   |  25 ++
 rust/tests/test_async_200_connect_command.rs  |  26 ++
 rust/tests/test_async_210_opt_abort.rs|  32 +++
 rust/tests/test_async_220_opt_list.rs |  86 ++
 rust/tests/test_async_230_opt_info.rs | 122 
 rust/tests/test_async_240_opt_list_meta.rs| 150 ++
 .../test_async_245_opt_list_meta_queries.rs   |  94 +++
 rust/tests/test_async_250_opt_set_meta.rs | 125 
 .../test_async_255_opt_set_meta_queries.rs| 110 
 rust/tests/test_async_400_pread.rs|  40 +++
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 
 rust/tests/test_async_460_block_status.rs |  98 +++
 rust/tests/test_async_620_stats.rs|  69 +
 scripts/git.orderfile |   1 +
 27 files changed, 1881 insertions(+), 1 deletion(-)
 create mode 100644 rust/examples/concurrent-read-write.rs
 create mode 100644 rust/src/async_handle.rs
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs


base-commit: f2dac1102884e3dea1cfb33479b34dd689fbb670
prerequisite-patch-id: ce5d2f65bb12ecda61c97fdf22255e188016b3fc
prerequisite-patch-id: cb5e3f05b600a4953e2a77bd53067bb51903aecd
prerequisite-patch-id: 7613cb6ebcc41fb45da587fc9487eb6c643a14a4
prerequisite-patch-id: 397ff0bea47242cf549a894ce519b3702f072c44
prerequisite-patch-id: e0024d76b8f30c22981fa7410d3ba6935aa3fde4
prerequisite-patch-id: 8228f280bd7dc331ba6e59b3b39209ae3eca0ed3
prerequisite-patch-id: 0bfbfc74216958994f80c7363e4473090bd8bab3
prerequisite-patch-id: 02d5d810150c2042a7b4dbd4f4991fa3f69a7039
prerequisite-patch-id: c55d4346bf391f61d0f836d4d2a8818366d2a701
prerequisite-patch-id: b79c88f8e664526bff63d22faa2b8b4e068d6d85
prerequisite-patch-id: a596c70f96938e496b92d4c99e3e8004c4b3c725
prerequisite-patch-id: d6bcb838a1875541f3f125b95f346c21a7d614ea
--
2.42.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v8 06/10] rust: async: Create an async friendly handle type

[Tage Johansson]

On 8/24/2023 11:55 PM, Eric Blake wrote:

On Sun, Aug 20, 2023 at 02:16:25PM +, Tage Johansson wrote:

Create another handle type: AsyncHandle, which makes use of Rust's
builtin asynchronous functions (see
<https://doc.rust-lang.org/std/keyword.async.html>) and runs on top of
the Tokio runtime (see<https://docs.rs/tokio>). For every asynchronous
command, like aio_connect(), a corresponding `async` method is created
on the handle. In this case it would be:
 async fn connect(...) -> Result<(), ...>
When called, it will poll the file descriptor until the command is
complete, and then return with a result. All the synchronous
counterparts (like nbd_connect()) are excluded from this handle type
as they are unnecessary and since they might interfear with the polling
made by the Tokio runtime. For more details about how the asynchronous
commands are executed, please see the comments in
rust/src/async_handle.rs.
---
  generator/Rust.ml| 249 +++-
  generator/Rust.mli   |   2 +
  generator/generator.ml   |   2 +
  rust/Cargo.toml  |   4 +-
  rust/Makefile.am |   2 +
  rust/src/async_handle.rs | 268 +++
  rust/src/lib.rs  |   8 ++
  rust/src/utils.rs|   9 ++
  scripts/git.orderfile|   1 +
  9 files changed, 538 insertions(+), 7 deletions(-)
  create mode 100644 rust/src/async_handle.rs

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 431c814..1bc81f0 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -61,11 +61,12 @@ let print_rust_flags { flag_prefix; flags } =
  let rec to_upper_snake_case s =
let s = String.uppercase_ascii s in
let s = explode s in
-  let s = filter_map (
-function
-|'-' -> Some "_" | ':' -> None
-| ch -> Some (String.make 1 ch)
-  ) s in
+  let s =
+filter_map
+  (function
+| '-' -> Some "_" | ':' -> None | ch -> Some (String.make 1 ch))
+  s
+  in
String.concat "" s

This looks like it is just reformatting.  While cleanup patches are
okay (and I trust Rich's take on OCaml style more than my own), it's
cleaner to do them in separate patches.

  
  (* Split a string into a list of chars.  In later OCaml we could

@@ -75,7 +76,7 @@ and explode str =
let r = ref [] in
for i = 0 to String.length str - 1 do
  let c = String.unsafe_get str i in
-r := c :: !r;
+r := c :: !r
done;
List.rev !r
  
@@ -564,3 +565,239 @@ let generate_rust_bindings () =

pr "impl Handle {\n";
List.iter print_rust_handle_method handle_calls;
pr "}\n\n"
+
+(*)
+(* The rest of the file conserns the asynchronous API.   *)

concerns


+(*   *)
+(* See the comments in rust/src/async_handle.rs for more *)
+(* information about how it works.   *)
+(*)
+
+let excluded_handle_calls : NameSet.t =
+  NameSet.of_list
+[
+  "aio_get_fd";
+  "aio_get_direction";
+  "aio_notify_read";
+  "aio_notify_write";
+  "clear_debug_callback";
+  "get_debug";
+  "poll";
+  "poll2";
+  "set_debug";
+  "set_debug_callback";
+]
+
+(* A mapping with names as keys. *)
+module NameMap = Map.Make (String)
+
+(* Strip "aio_" from the beginning of a string. *)
+let strip_aio name : string =
+  if String.starts_with ~prefix:"aio_" name then
+String.sub name 4 (String.length name - 4)
+  else failwithf "Asynchronous call %s must begin with aio_" name
+
+(* A map with all asynchronous handle calls. The keys are names with "aio_"
+   stripped, the values are a tuple with the actual name (with "aio_"), the
+   [call] and the [async_kind]. *)
+let async_handle_calls : ((string * call) * async_kind) NameMap.t =

Do we need a 2-deep nested tuple, or can we use (string * call * async_kind)?


+  handle_calls
+  |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
+  |> List.filter_map (fun (name, call) ->
+ call.async_kind
+ |> Option.map (fun async_kind ->
+(strip_aio name, ((name, call), async_kind
+  |> List.to_seq |> NameMap.of_seq
+
+(* A mapping with all synchronous (not asynchronous) handle calls. Excluded
+   are also all synchronous calls that has an asynchronous counterpart. So if

s/has/have/


+   "foo" is the name of a handle call and an asynchronous call "aio_foo"
+   exists, then "foo" will not b in this map. *)

s/b /be /


+let sync_handle_calls : call NameMap.t =
+  handle_calls
+  |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handl

[Libguestfs] [libnbd PATCH v8 10/10] rust: async: Add an example

[Tage Johansson]

This patch adds an example using the asynchronous Rust bindings.
---
 rust/Cargo.toml|   1 +
 rust/examples/concurrent-read-write.rs | 149 +
 rust/run-tests.sh.in   |   2 +
 3 files changed, 152 insertions(+)
 create mode 100644 rust/examples/concurrent-read-write.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index c49f9f2..0879b34 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -58,5 +58,6 @@ default = ["log", "tokio"]
 anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
 tempfile = "3.6.0"
 tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..4858f76
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,149 @@
+//! Example usage with nbdkit:
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//! Or connect over a URI:
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $uri'
+//!
+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+
+// Check if the user provided a URI or a unix socket.
+let socket_or_uri = args[1].to_str().unwrap();
+if socket_or_uri.contains("://") {
+nbd.connect_uri(socket_or_uri)?;
+} else {
+nbd.connect_unix(socket_or_uri)?;
+}
+
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket_or_uri.to_owned(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket_or_uri: String,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// futures must be `'static`, hence we wrap the handle in an [Arc].
+let nbd = Arc::new(libnbd::AsyncHandle::new()?);
+
+// Check if the user provided a URI or a unix socket.
+if socket_or_uri.contains("://") {
+nbd.connect_uri(socket_or_uri).await?;
+} else {
+nbd.connect_unix(socket_or_uri).await?;
+}
+
+let mut rng = SmallRng::seed_from_u64(44 as u64);
+
+// Issue commands.
+let mut stats = Stats::default();
+let mut join_set = JoinSet::new();
+//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+while stats.requests < 

[Libguestfs] [libnbd PATCH v8 09/10] rust: async: Add a couple of integration tests

[Tage Johansson]

Add a couple of integration tests as rust/tests/test_async_*.rs. They
are very similar to the tests for the synchronous API.
---
 rust/Cargo.toml   |   1 +
 rust/tests/test_async_100_handle.rs   |  25 +++
 rust/tests/test_async_200_connect_command.rs  |  26 +++
 rust/tests/test_async_210_opt_abort.rs|  32 
 rust/tests/test_async_220_opt_list.rs |  86 ++
 rust/tests/test_async_230_opt_info.rs | 122 ++
 rust/tests/test_async_240_opt_list_meta.rs| 150 ++
 .../test_async_245_opt_list_meta_queries.rs   |  94 +++
 rust/tests/test_async_250_opt_set_meta.rs | 125 +++
 .../test_async_255_opt_set_meta_queries.rs| 110 +
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 +++
 rust/tests/test_async_460_block_status.rs |  98 
 rust/tests/test_async_620_stats.rs|  69 
 15 files changed, 1121 insertions(+)
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index a9b5988..c49f9f2 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -59,3 +59,4 @@ anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
 tempfile = "3.6.0"
+tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/tests/test_async_100_handle.rs 
b/rust/tests/test_async_100_handle.rs
new file mode 100644
index 000..e50bad9
--- /dev/null
+++ b/rust/tests/test_async_100_handle.rs
@@ -0,0 +1,25 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+//! Just check that we can link with libnbd and create a handle.
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_nbd_handle_new() {
+let _ = libnbd::AsyncHandle::new().unwrap();
+}
diff --git a/rust/tests/test_async_200_connect_command.rs 
b/rust/tests/test_async_200_connect_command.rs
new file mode 100644
index 000..2f4e7cc
--- /dev/null
+++ b/rust/tests/test_async_200_connect_command.rs
@@ -0,0 +1,26 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_connect_command() {
+let nbd = libnbd::AsyncHandle::new().unwrap();
+nbd.connect_command(&["nbdkit", "-s", "--exit-with-parent", "-v", "null"])
+.await
+.unwrap();
+}
diff --git a/rust/tests/test_async_210_opt_abort.rs 
b/rust/tests/test_a

[Libguestfs] [libnbd PATCH v8 06/10] rust: async: Create an async friendly handle type

[Tage Johansson]

   ret_rx.await.unwrap()\n";
+  pr "}\n\n"
+
+(* Print a Rust function for an asynchronous handle call which signals
+   completion by changing state. The predicate is a call like
+   "aio_is_connecting" which should get the value (like false) for the call to
+   be complete. *)
+let print_rust_async_handle_call_changing_state name (aio_name, call)
+(predicate, value) =
+  let value = if value then "true" else "false" in
+  print_rust_handle_call_comment call;
+  pr "pub async fn %s(, %s) -> SharedResult<()>\n" name
+(rust_async_handle_call_args call);
+  pr "{\n";
+  print_ffi_call aio_name "self.data.handle.handle" call;
+  pr "?;\n";
+  pr "let (ret_tx, ret_rx) = oneshot::channel::>();\n";
+  pr "let mut ret_tx = Some(ret_tx);\n";
+  pr "let completion_predicate = \n";
+  pr " move |handle: , res: <()>| {\n";
+  pr "  let ret = if let Err(_) = res {\n";
+  pr "res.clone()\n";
+  pr "  } else {\n";
+  pr "if handle.%s() != %s { return false; }\n" predicate value;
+  pr "else { Ok(()) }\n";
+  pr "  };\n";
+  pr "  ret_tx.take().unwrap().send(ret).ok();\n";
+  pr "  true\n";
+  pr "};\n";
+  pr "self.add_command(completion_predicate)?;\n";
+  pr "ret_rx.await.unwrap()\n";
+  pr "}\n\n"
+
+(* Print an impl with all handle calls. *)
+let print_rust_async_handle_impls () =
+  pr "impl AsyncHandle {\n";
+  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
+  NameMap.iter
+(fun name (call, async_kind) ->
+  match async_kind with
+  | WithCompletionCallback ->
+  print_rust_async_handle_call_with_completion_cb name call
+  | ChangesState (predicate, value) ->
+  print_rust_async_handle_call_changing_state name call
+(predicate, value))
+async_handle_calls;
+  pr "}\n\n"
+
+let print_rust_async_imports () =
+  pr "use crate::{*, types::*};\n";
+  pr "use os_socketaddr::OsSocketAddr;\n";
+  pr "use std::ffi::*;\n";
+  pr "use std::mem;\n";
+  pr "use std::net::SocketAddr;\n";
+  pr "use std::os::fd::{AsRawFd, OwnedFd};\n";
+  pr "use std::os::unix::prelude::*;\n";
+  pr "use std::path::PathBuf;\n";
+  pr "use std::ptr;\n";
+  pr "use std::sync::Arc;\n";
+  pr "use tokio::sync::oneshot;\n";
+  pr "\n"
+
+let generate_rust_async_bindings () =
+  generate_header CStyle ~copyright:"Tage Johansson";
+  pr "\n";
+  print_rust_async_imports ();
+  print_rust_async_handle_impls ()
diff --git a/generator/Rust.mli b/generator/Rust.mli
index 450e4ca..0960170 100644
--- a/generator/Rust.mli
+++ b/generator/Rust.mli
@@ -18,3 +18,5 @@
 
 (* Print all flag-structs, enums, constants and handle calls in Rust code. *)
 val generate_rust_bindings : unit -> unit
+
+val generate_rust_async_bindings : unit -> unit
diff --git a/generator/generator.ml b/generator/generator.ml
index 8c9a585..11bec4d 100644
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -69,3 +69,5 @@ let () =
 RustSys.generate_rust_sys_bindings;
   output_to ~formatter:(Some Rustfmt) "rust/src/bindings.rs"
 Rust.generate_rust_bindings;
+  output_to ~formatter:(Some Rustfmt) "rust/src/async_bindings.rs"
+Rust.generate_rust_async_bindings;
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 01555de..a9b5988 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -48,9 +48,11 @@ os_socketaddr = "0.2.4"
 thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
+tokio = { optional = true, version = "1.29.1", default-features = false, 
features = ["rt", "sync", "net"] }
+epoll = "4.3.3"
 
 [features]
-default = ["log"]
+default = ["log", "tokio"]
 
 [dev-dependencies]
 anyhow = "1.0.72"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index 7098c9a..2b5b85b 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -19,6 +19,7 @@ include $(top_srcdir)/subdir-rules.mk
 
 generator_built = \
libnbd-sys/src/generated.rs \
+   src/async_bindings.rs \
src/bindings.rs \
$(NULL)
 
@@ -30,6 +31,7 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   src/async_handle.rs \
examples/connect-command.rs \
examples/get-size.rs \
examples/fetch-first-sector.rs \
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
new file mode 100644
index 000..4223b80
--- /dev/null
+++ b/rust/src/async_handle.rs
@@ -0,0 +1,268 @@
+/

[Libguestfs] [libnbd PATCH v8 07/10] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

Add a flag (modifies_fd) to the call structure in generator/API.ml
which is set to true if the handle call may do something with the
file descriptor. That is, it is true for all calls which are or may
call aio_notify_*, including all synchronous commands like
nbd_connect or nbd_opt_go.

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to true will be
excluded from that handle. The asynchronous (aio_*) functions will be
used instead.
---
 generator/API.ml  | 32 
 generator/API.mli |  7 +++
 2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 99fcb82..b0f5e2a 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
   is_locked : bool;
   may_set_error : bool;
   async_kind : async_kind option;
+  modifies_fd: bool;
   mutable first_version : int * int;
 }
 and arg =
@@ -255,6 +256,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  permitted_states = [];
  is_locked = true; may_set_error = true;
  async_kind = None;
+ modifies_fd = false;
  first_version = (0, 0) }
 
 (* Calls.
@@ -1162,6 +1164,7 @@ Return true if option negotiation mode was enabled on 
this handle.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -1189,6 +1192,7 @@ although older servers will instead have killed the 
connection.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 Request that the server finish negotiation, gracefully if possible, then
@@ -1202,6 +1206,7 @@ enabled option mode.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to initiate TLS";
 longdesc = "\
 Request that the server initiate a secure TLS connection, by
@@ -1240,6 +1245,7 @@ established, as reported by 
L.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to enable structured replies";
 longdesc = "\
 Request that the server use structured replies, by sending
@@ -1266,6 +1272,7 @@ later calls to this function return false.";
 default_call with
 args = [ Closure list_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to list all exports during negotiation";
 longdesc = "\
 Request that the server list all exports that it supports.  This can
@@ -1307,6 +1314,7 @@ description is set with I<-D>.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server for information about an export";
 longdesc = "\
 Request that the server supply information about the export name
@@ -1338,6 +1346,7 @@ corresponding L would succeed.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using implicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1393,6 +1402,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using explicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1443,6 +1453,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using implicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1502,6 +1513,7 @@ no contexts are reported, or may fail but have a 
non-empty list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using explicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1731,6 

[Libguestfs] [libnbd PATCH v8 02/10] rust: Correct license field in Cargo.toml

[Tage Johansson]

Previously, the license field in rust/Cargo.toml was wrongly set to
"LGPL-2.1-only". This commit changes it to the correct
"LGPL-2.1-or-later", which conforms to the license preamble in the
source files.
---
 rust/Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 9ac6e53..01555de 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -25,7 +25,7 @@ version = "0.1.0"
 edition = "2021"
 rust-version = "1.69"
 description = "Rust bindings for libnbd, a client library for controlling 
block devices over a network."
-license = "LGPL-2.1-only"
+license = "LGPL-2.1-or-later"
 keywords = ["libnbd", "block-device", "network"]
 categories = ["api-bindings", "emulators", "virtualization"]
 
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v8 08/10] rust: async: Use the modifies_fd flag to exclude calls

[Tage Johansson]

All handle calls which has the modifies_fd flag set to true will be
excluded from AsyncHandle (the asynchronous handle in the rust
bindings). This is a better approach then listing all calls that should
be excluded in Rust.ml explicetly.
---
 generator/Rust.ml | 60 +--
 1 file changed, 27 insertions(+), 33 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 1bc81f0..323ccf1 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -575,18 +575,17 @@ let generate_rust_bindings () =
 
 let excluded_handle_calls : NameSet.t =
   NameSet.of_list
-[
-  "aio_get_fd";
-  "aio_get_direction";
-  "aio_notify_read";
-  "aio_notify_write";
-  "clear_debug_callback";
-  "get_debug";
-  "poll";
-  "poll2";
-  "set_debug";
-  "set_debug_callback";
-]
+  @@ [
+   "aio_get_fd";
+   "aio_get_direction";
+   "clear_debug_callback";
+   "get_debug";
+   "set_debug";
+   "set_debug_callback";
+ ]
+  @ (handle_calls
+|> List.filter (fun (_, { modifies_fd }) -> modifies_fd)
+|> List.map (fun (name, _) -> name))
 
 (* A mapping with names as keys. *)
 module NameMap = Map.Make (String)
@@ -597,16 +596,16 @@ let strip_aio name : string =
 String.sub name 4 (String.length name - 4)
   else failwithf "Asynchronous call %s must begin with aio_" name
 
-(* A map with all asynchronous handle calls. The keys are names with "aio_"
-   stripped, the values are a tuple with the actual name (with "aio_"), the
-   [call] and the [async_kind]. *)
+(* A map with all asynchronous handle calls. The keys are names, the values
+   are tuples of: the name with "aio_" stripped, the [call] and the
+   [async_kind]. *)
 let async_handle_calls : ((string * call) * async_kind) NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
   |> List.filter_map (fun (name, call) ->
  call.async_kind
  |> Option.map (fun async_kind ->
-(strip_aio name, ((name, call), async_kind
+(name, ((strip_aio name, call), async_kind
   |> List.to_seq |> NameMap.of_seq
 
 (* A mapping with all synchronous (not asynchronous) handle calls. Excluded
@@ -616,11 +615,7 @@ let async_handle_calls : ((string * call) * async_kind) 
NameMap.t =
 let sync_handle_calls : call NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
-  |> List.filter (fun (name, _) ->
- (not (NameMap.mem name async_handle_calls))
- && not
-  (String.starts_with ~prefix:"aio_" name
-  && NameMap.mem (strip_aio name) async_handle_calls))
+  |> List.filter (fun (n, _) -> not (NameMap.mem n async_handle_calls))
   |> List.to_seq |> NameMap.of_seq
 
 (* Get the Rust type for an argument in the asynchronous API. Like
@@ -660,7 +655,7 @@ let print_rust_sync_handle_call name call =
 (* Print the Rust function for an asynchronous handle call with a completion
callback. (Note that "callback" might be abbreviated with "cb" in the
following code. *)
-let print_rust_async_handle_call_with_completion_cb name (aio_name, call) =
+let print_rust_async_handle_call_with_completion_cb aio_name (name, call) =
   (* An array of all optional arguments. Useful because we need to deel with
  the index of the completion callback. *)
   let optargs = Array.of_list call.optargs in
@@ -741,7 +736,7 @@ let print_rust_async_handle_call_with_completion_cb name 
(aio_name, call) =
completion by changing state. The predicate is a call like
"aio_is_connecting" which should get the value (like false) for the call to
be complete. *)
-let print_rust_async_handle_call_changing_state name (aio_name, call)
+let print_rust_async_handle_call_changing_state aio_name (name, call)
 (predicate, value) =
   let value = if value then "true" else "false" in
   print_rust_handle_call_comment call;
@@ -770,16 +765,15 @@ let print_rust_async_handle_call_changing_state name 
(aio_name, call)
 (* Print an impl with all handle calls. *)
 let print_rust_async_handle_impls () =
   pr "impl AsyncHandle {\n";
-  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
-  NameMap.iter
-(fun name (call, async_kind) ->
-  match async_kind with
-  | WithCompletionCallback ->
-  print_rust_async_handle_call_with_completion_cb name call
-  | ChangesState (predicate, value) ->
-  print_rust_async_handle_call_changing_state name call
-(predicate, value))
-async_handle_calls;
+  sync_handle_calls |> NameMap.iter print_rust_sync_handle_call;
+  async_handle_calls
+  |> NameMap.iter (fun name (call, async_kind) ->
+ match async_kind with
+ | WithCompletionCallback ->
+ print_rust_async_handle_call_with_completion_cb name call
+ | ChangesState (predicate, value) ->
+ print_rust_async_handle_call_changing_state 

[Libguestfs] [libnbd PATCH v8 03/10] rust: Format some files to conform to rustfmt and rustfmt.toml

[Tage Johansson]

Some Rust files were previously not formatted according to rustfmt and
the top-level rustfmt.toml. This commit correct the formatting in those
files.
---
 rust/examples/connect-command.rs   |  1 -
 rust/tests/test_200_connect_command.rs | 11 ++-
 rust/tests/test_300_get_size.rs|  1 -
 rust/tests/test_620_stats.rs   | 11 ++-
 4 files changed, 4 insertions(+), 20 deletions(-)

diff --git a/rust/examples/connect-command.rs b/rust/examples/connect-command.rs
index db4adbe..4ad6143 100644
--- a/rust/examples/connect-command.rs
+++ b/rust/examples/connect-command.rs
@@ -1,7 +1,6 @@
 //! This example shows how to run an NBD server
 //! (nbdkit) as a subprocess of libnbd.
 
-
 fn main() -> libnbd::Result<()> {
 // Create the libnbd handle.
 let handle = libnbd::Handle::new()?;
diff --git a/rust/tests/test_200_connect_command.rs 
b/rust/tests/test_200_connect_command.rs
index 8338650..963f334 100644
--- a/rust/tests/test_200_connect_command.rs
+++ b/rust/tests/test_200_connect_command.rs
@@ -17,16 +17,9 @@
 
 #![deny(warnings)]
 
-
 #[test]
 fn test_connect_command() {
 let nbd = libnbd::Handle::new().unwrap();
-nbd.connect_command(&[
-"nbdkit",
-"-s",
-"--exit-with-parent",
-"-v",
-"null",
-])
-.unwrap();
+nbd.connect_command(&["nbdkit", "-s", "--exit-with-parent", "-v", "null"])
+.unwrap();
 }
diff --git a/rust/tests/test_300_get_size.rs b/rust/tests/test_300_get_size.rs
index c830164..bed9d87 100644
--- a/rust/tests/test_300_get_size.rs
+++ b/rust/tests/test_300_get_size.rs
@@ -17,7 +17,6 @@
 
 #![deny(warnings)]
 
-
 #[test]
 fn test_get_size() {
 let nbd = libnbd::Handle::new().unwrap();
diff --git a/rust/tests/test_620_stats.rs b/rust/tests/test_620_stats.rs
index 134d59a..0b6f4e2 100644
--- a/rust/tests/test_620_stats.rs
+++ b/rust/tests/test_620_stats.rs
@@ -17,7 +17,6 @@
 
 #![deny(warnings)]
 
-
 #[test]
 fn test_stats() {
 let nbd = libnbd::Handle::new().unwrap();
@@ -31,14 +30,8 @@ fn test_stats() {
 // Connection performs handshaking, which increments stats.
 // The number of bytes/chunks here may grow over time as more features get
 // automatically negotiated, so merely check that they are non-zero.
-nbd.connect_command(&[
-"nbdkit",
-"-s",
-"--exit-with-parent",
-"-v",
-"null",
-])
-.unwrap();
+nbd.connect_command(&["nbdkit", "-s", "--exit-with-parent", "-v", "null"])
+.unwrap();
 
 let bs1 = nbd.stats_bytes_sent();
 let cs1 = nbd.stats_chunks_sent();
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v8 05/10] generator: Add information about asynchronous handle calls

[Tage Johansson]

A new field (async_kind) is added to the call data type in
generator/API.ml*. The purpose is to tell if a certain handle call is
an asynchronous command and if so how one can know when it is
completed. The motivation for this is that all asynchronous commands
on the AsyncHandle in the Rust bindings makes use of Rust's
[`async fn`s](https://doc.rust-lang.org/std/keyword.async.html).
But to implement such an async fn, the API needs to know when the
command completed, either by a completion callback or via a change
of state.
---
 generator/API.ml  | 32 
 generator/API.mli | 11 +++
 2 files changed, 43 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 72c8165..99fcb82 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -32,6 +32,7 @@ type call = {
   permitted_states : permitted_state list;
   is_locked : bool;
   may_set_error : bool;
+  async_kind : async_kind option;
   mutable first_version : int * int;
 }
 and arg =
@@ -102,6 +103,9 @@ and permitted_state =
 | Negotiating
 | Connected
 | Closed | Dead
+and async_kind =
+| WithCompletionCallback
+| ChangesState of string * bool
 and link =
 | Link of string
 | SectionLink of string
@@ -250,6 +254,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  see_also = [];
  permitted_states = [];
  is_locked = true; may_set_error = true;
+ async_kind = None;
  first_version = (0, 0) }
 
 (* Calls.
@@ -2802,6 +2807,7 @@ wait for an L.";
 default_call with
 args = [ SockAddrAndLen ("addr", "addrlen") ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Begin connecting to the NBD server.  The C and C
@@ -2814,6 +2820,7 @@ parameters specify the address of the socket to connect 
to.
 default_call with
 args = [ String "uri" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to an NBD URI";
 longdesc = "\
 Begin connecting to the NBD URI C.  Parameters behave as
@@ -2827,6 +2834,7 @@ documented in L.
 default_call with
 args = [ Path "unixsocket" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a Unix domain socket";
 longdesc = "\
 Begin connecting to the NBD server over Unix domain socket
@@ -2841,6 +2849,7 @@ L.
 default_call with
 args = [ UInt32 "cid"; UInt32 "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over AF_VSOCK socket";
 longdesc = "\
 Begin connecting to the NBD server over the C
@@ -2854,6 +2863,7 @@ L.
 default_call with
 args = [ String "hostname"; String "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a TCP port";
 longdesc = "\
 Begin connecting to the NBD server listening on C.
@@ -2866,6 +2876,7 @@ Parameters behave as documented in L.
 default_call with
 args = [ Fd "sock" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect directly to a connected socket";
 longdesc = "\
 Begin connecting to the connected socket C.
@@ -2878,6 +2889,7 @@ Parameters behave as documented in 
L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it over
@@ -2891,6 +2903,7 @@ L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect using systemd socket activation";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it using
@@ -2907,6 +2920,7 @@ L.
 optargs = [ OClosure completion_closure ];
 ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some WithCompletionCallback;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -2930,6 +2944,7 @@ when L returns true.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 

[Libguestfs] [libnbd PATCH v8 04/10] rust: Make it possible to run examples with a URI

[Tage Johansson]

Previously, the examples fetch-first-sector and get-size in
rust/examples only accepted a unix socket as argument. This commit makes
it possible to provide a URI as well.
---
 rust/examples/fetch-first-sector.rs | 15 +++
 rust/examples/get-size.rs   | 20 +++-
 2 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/rust/examples/fetch-first-sector.rs 
b/rust/examples/fetch-first-sector.rs
index 9efb47a..b035ccb 100644
--- a/rust/examples/fetch-first-sector.rs
+++ b/rust/examples/fetch-first-sector.rs
@@ -7,6 +7,9 @@
 //!
 //! nbdkit -U - floppy . \
 //!   --run 'cargo run --example fetch-first-sector -- $unixsocket'
+//! Or with a URI:
+//! nbdkit -U - floppy . \
+//!   --run 'cargo run --example fetch-first-sector -- $uri'
 //!
 //! The nbdkit floppy plugin creates an MBR disk so the
 //! first sector is the partition table.
@@ -21,11 +24,15 @@ fn main() -> anyhow::Result<()> {
 if args.len() != 2 {
 anyhow::bail!("Usage: {:?} socket", args[0]);
 }
-let socket = [1];
 
-// Connect to the NBD server over a
-// Unix domain socket.
-nbd.connect_unix(socket)?;
+// Check if the user provided a URI or a unix socket.
+let socket_or_uri = args[1].to_str().unwrap();
+if socket_or_uri.contains("://") {
+nbd.connect_uri(socket_or_uri)?;
+} else {
+// Connect to the NBD server over a Unix domain socket.
+nbd.connect_unix(socket_or_uri)?;
+}
 
 // Read the first sector synchronously.
 let mut buf = [0; 512];
diff --git a/rust/examples/get-size.rs b/rust/examples/get-size.rs
index 7f31df5..7af8e9f 100644
--- a/rust/examples/get-size.rs
+++ b/rust/examples/get-size.rs
@@ -5,6 +5,12 @@
 //!
 //! nbdkit -U - memory 1M \
 //!   --run 'cargo run --example get-size -- $unixsocket'
+//! Or with a URI:
+//! nbdkit -U - memory 1M \
+//!   --run 'cargo run --example get-size -- $uri'
+//!
+//! Or connect over a URI:
+//! cargo run --example get-size -- nbd://hostname:port
 
 use std::env;
 
@@ -15,15 +21,19 @@ fn main() -> anyhow::Result<()> {
 if args.len() != 2 {
 anyhow::bail!("Usage: {:?} socket", args[0]);
 }
-let socket = [1];
 
-// Connect to the NBD server over a
-// Unix domain socket.
-nbd.connect_unix(socket)?;
+// Check if the user provided a URI or a unix socket.
+let socket_or_uri = args[1].to_str().unwrap();
+if socket_or_uri.contains("://") {
+nbd.connect_uri(socket_or_uri)?;
+} else {
+// Connect to the NBD server over a Unix domain socket.
+nbd.connect_unix(socket_or_uri)?;
+}
 
 // Read the size in bytes and print it.
 let size = nbd.get_size()?;
-println!("{:?}: size = {size} bytes", socket);
+println!("{:?}: size = {size} bytes", socket_or_uri);
 
 Ok(())
 }
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v8 00/10] Rust Bindings for Libnbd

[Tage Johansson]

This is the 8th version of the Rust bindings for libnbd. Compared to v7,
it (amongst others) removes the cblifetime and cbcount fields in API.ml.
It also contains patches for improving the examples, reformatting rust
files, and correcting the license field in Cargo.toml.


Best regards,
Tage


Tage Johansson (10):
  rust: Specify minimum Rust version in Cargo.toml
  rust: Correct license field in Cargo.toml
  rust: Format some files to conform to rustfmt and rustfmt.toml
  rust: Make it possible to run examples with a URI
  generator: Add information about asynchronous handle calls
  rust: async: Create an async friendly handle type
  generator: Add `modifies_fd` flag to the [call] structure
  rust: async: Use the modifies_fd flag to exclude calls
  rust: async: Add a couple of integration tests
  rust: async: Add an example

 generator/API.ml  |  64 +
 generator/API.mli |  18 ++
 generator/Rust.ml | 243 +++-
 generator/Rust.mli|   2 +
 generator/generator.ml|   2 +
 rust/Cargo.toml   |  12 +-
 rust/Makefile.am  |   2 +
 rust/cargo_test/Cargo.toml|   1 +
 rust/examples/concurrent-read-write.rs| 149 ++
 rust/examples/connect-command.rs  |   1 -
 rust/examples/fetch-first-sector.rs   |  15 +-
 rust/examples/get-size.rs |  20 +-
 rust/libnbd-sys/Cargo.toml|   1 +
 rust/run-tests.sh.in  |   2 +
 rust/src/async_handle.rs  | 268 ++
 rust/src/lib.rs   |   8 +
 rust/src/utils.rs |   9 +
 rust/tests/test_200_connect_command.rs|  11 +-
 rust/tests/test_220_opt_list.rs   |   2 +-
 rust/tests/test_240_opt_list_meta.rs  |   2 +-
 rust/tests/test_245_opt_list_meta_queries.rs  |   2 +-
 rust/tests/test_250_opt_set_meta.rs   |   2 +-
 rust/tests/test_255_opt_set_meta_queries.rs   |   2 +-
 rust/tests/test_300_get_size.rs   |   1 -
 rust/tests/test_460_block_status.rs   |   2 +-
 rust/tests/test_620_stats.rs  |  11 +-
 rust/tests/test_async_100_handle.rs   |  25 ++
 rust/tests/test_async_200_connect_command.rs  |  26 ++
 rust/tests/test_async_210_opt_abort.rs|  32 +++
 rust/tests/test_async_220_opt_list.rs |  86 ++
 rust/tests/test_async_230_opt_info.rs | 122 
 rust/tests/test_async_240_opt_list_meta.rs| 150 ++
 .../test_async_245_opt_list_meta_queries.rs   |  94 ++
 rust/tests/test_async_250_opt_set_meta.rs | 125 
 .../test_async_255_opt_set_meta_queries.rs| 110 +++
 rust/tests/test_async_400_pread.rs|  40 +++
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 
 rust/tests/test_async_460_block_status.rs |  98 +++
 rust/tests/test_async_620_stats.rs|  69 +
 scripts/git.orderfile |   1 +
 41 files changed, 1930 insertions(+), 43 deletions(-)
 create mode 100644 rust/examples/concurrent-read-write.rs
 create mode 100644 rust/src/async_handle.rs
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs


base-commit: f2dac1102884e3dea1cfb33479b34dd689fbb670
prerequisite-patch-id: ce5d2f65bb12ecda61c97fdf22255e188016b3fc
prerequisite-patch-id: cb5e3f05b600a4953e2a77bd53067bb51903aecd
prerequisite-patch-id: 7613cb6ebcc41fb45da587fc9487eb6c643a14a4
prerequisite-patch-id: 397ff0bea47242cf549a894ce519b3702f072c44
prerequisite-patch-id: d6bcb838a1875541f3f125b95f346c21a7d614ea
--
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v8 01/10] rust: Specify minimum Rust version in Cargo.toml

[Tage Johansson]

Specify the minimum Rust version (1.69) in rust/Cargo.toml. Previously,
trying to compile with an older Rust toolchain caused various
compilation errors which were hard to interpret. Now the cause of the
error will be much more clear.
---
 rust/Cargo.toml  | 4 
 rust/cargo_test/Cargo.toml   | 1 +
 rust/libnbd-sys/Cargo.toml   | 1 +
 rust/tests/test_220_opt_list.rs  | 2 +-
 rust/tests/test_240_opt_list_meta.rs | 2 +-
 rust/tests/test_245_opt_list_meta_queries.rs | 2 +-
 rust/tests/test_250_opt_set_meta.rs  | 2 +-
 rust/tests/test_255_opt_set_meta_queries.rs  | 2 +-
 rust/tests/test_460_block_status.rs  | 2 +-
 9 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 04e371e..9ac6e53 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -20,7 +20,10 @@
 [workspace.package]
 authors = ["Tage Johansson"]
 version = "0.1.0"
+# Make sure that the values of the edition and rust-version fields in
+# rust_test/Cargo.toml matches the values here.
 edition = "2021"
+rust-version = "1.69"
 description = "Rust bindings for libnbd, a client library for controlling 
block devices over a network."
 license = "LGPL-2.1-only"
 keywords = ["libnbd", "block-device", "network"]
@@ -31,6 +34,7 @@ name = "libnbd"
 authors.workspace = true
 version.workspace = true
 edition.workspace = true
+rust-version.workspace = true
 description.workspace = true
 license.workspace = true
 keywords.workspace = true
diff --git a/rust/cargo_test/Cargo.toml b/rust/cargo_test/Cargo.toml
index 9f9d478..ffe88c1 100644
--- a/rust/cargo_test/Cargo.toml
+++ b/rust/cargo_test/Cargo.toml
@@ -20,4 +20,5 @@
 [package]
 name = "cargo_test"
 edition = "2021"
+rust-version = "1.69"
 version = "0.1.0"
diff --git a/rust/libnbd-sys/Cargo.toml b/rust/libnbd-sys/Cargo.toml
index 3fa581f..3f09f1b 100644
--- a/rust/libnbd-sys/Cargo.toml
+++ b/rust/libnbd-sys/Cargo.toml
@@ -18,6 +18,7 @@
 [package]
 name = "libnbd-sys"
 version.workspace = true
+rust-version.workspace = true
 edition.workspace = true
 description.workspace = true
 license.workspace = true
diff --git a/rust/tests/test_220_opt_list.rs b/rust/tests/test_220_opt_list.rs
index 180a95b..28d4db9 100644
--- a/rust/tests/test_220_opt_list.rs
+++ b/rust/tests/test_220_opt_list.rs
@@ -66,7 +66,7 @@ impl ConnTester {
 .push(String::from_utf8(name.to_owned()).unwrap());
 0
 })?;
-let exports = Arc::into_inner(exports).unwrap().into_inner().unwrap();
+let exports = Arc::try_unwrap(exports).unwrap().into_inner().unwrap();
 assert_eq!(exports.len(), count as usize);
 assert_eq!(exports.len(), expected_exports.len());
 for (export, ) in exports.iter().zip(expected_exports) {
diff --git a/rust/tests/test_240_opt_list_meta.rs 
b/rust/tests/test_240_opt_list_meta.rs
index 5598458..cc0310d 100644
--- a/rust/tests/test_240_opt_list_meta.rs
+++ b/rust/tests/test_240_opt_list_meta.rs
@@ -43,7 +43,7 @@ fn list_meta_ctxs(nbd: ::Handle) -> 
libnbd::Result {
 }
 0
 })?;
-let info = Arc::into_inner(info).unwrap().into_inner().unwrap();
+let info = Arc::try_unwrap(info).unwrap().into_inner().unwrap();
 assert_eq!(info.count, replies);
 Ok(info)
 }
diff --git a/rust/tests/test_245_opt_list_meta_queries.rs 
b/rust/tests/test_245_opt_list_meta_queries.rs
index da5c674..d6409e5 100644
--- a/rust/tests/test_245_opt_list_meta_queries.rs
+++ b/rust/tests/test_245_opt_list_meta_queries.rs
@@ -46,7 +46,7 @@ fn list_meta_ctxs(
 }
 0
 })?;
-let info = Arc::into_inner(info).unwrap().into_inner().unwrap();
+let info = Arc::try_unwrap(info).unwrap().into_inner().unwrap();
 assert_eq!(info.count, replies);
 Ok(info)
 }
diff --git a/rust/tests/test_250_opt_set_meta.rs 
b/rust/tests/test_250_opt_set_meta.rs
index c7a8144..26d82e5 100644
--- a/rust/tests/test_250_opt_set_meta.rs
+++ b/rust/tests/test_250_opt_set_meta.rs
@@ -44,7 +44,7 @@ fn set_meta_ctxs(nbd: ::Handle) -> 
libnbd::Result {
 }
 0
 })?;
-let info = Arc::into_inner(info).unwrap().into_inner().unwrap();
+let info = Arc::try_unwrap(info).unwrap().into_inner().unwrap();
 assert_eq!(info.count, replies);
 Ok(info)
 }
diff --git a/rust/tests/test_255_opt_set_meta_queries.rs 
b/rust/tests/test_255_opt_set_meta_queries.rs
index 143a2f1..87f8d6a 100644
--- a/rust/tests/test_255_opt_set_meta_queries.rs
+++ b/rust/tests/test_255_opt_set_meta_queries.rs
@@ -47,7 +47,7 @@ fn set_meta_ctxs_queries(
 }
 0
 })?;
-let info = Arc::into_inner(info).unwrap().into_inner().unwrap();
+let info = Arc::try_unwrap(info).unwrap().into_inner().unwrap();
 assert_eq!(info.count, repli

Re: [Libguestfs] [libnbd PATCH v7 0/9] Rust Bindings for Libnbd

[Tage Johansson]

On 8/17/2023 6:09 PM, Richard W.M. Jones wrote:

On Thu, Aug 17, 2023 at 09:29:47AM -0500, Eric Blake wrote:

On Thu, Aug 17, 2023 at 04:13:36AM +, Tage Johansson wrote:

It is possible to add a minimum version requirement in Cargo.toml, I guess I
should do that to make the errors a bit more clear.

It's okay if older platforms can't build Rust bindings, but anything
we can do to make it so that configure detects that gracefully up
front (and the rest of the build still succeeds), rather than
falling apart later during make when the missing feature turns into
a compiler error, is a good thing.  And whatever we do, it doesn't
hurt if we amend README to call out the minimum Rust version we are
willing to support.

^ This is the important bit.  ./configure should disable the Rust
bindings if they can't be built (for any reason, but including Rust
being too old).  This avoids people hitting problems later on.  We
don't control the environment where people try to compile libnbd.

Rich.



This is absolutely possible. I can make so that the compilation of the 
test-crate in rust/cargo_test will fail if the rust version is to old 
and thereby disabling rust bindings in ./configure.



It should be fairly easy to support rust 1.69.0.


I will try to send out a new patch series as soon as possible, but I am 
on some sort of vacation this weekend so it might not be until next week.



--

Best regards,

Tage


___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v5 02/12] rust: Add a couple of integration tests

[Tage Johansson]

On 8/16/2023 11:35 PM, Eric Blake wrote:

On Thu, Aug 03, 2023 at 03:36:06PM +, Tage Johansson wrote:

A couple of integration tests are added in rust/tests. They are mostly
ported from the OCaml tests.
---

Overall, this looked like a nice counterpart to the OCaml unit tests,
and I was able to easily figure out how to amend my own tests in for
the unit tests I added in my 64-bit extension work.  Keeping similar
unit test numbers across language bindings has been a big boon :-)

Style question:


+++ b/rust/tests/test_250_opt_set_meta.rs
+
+/// A struct with information about set meta contexts.
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct CtxInfo {
+/// Whether the meta context "base:allocation" is set.
+has_alloc: bool,
+/// The number of set meta contexts.
+count: u32,
+}

Here, you use a trailing comma,


+
+fn set_meta_ctxs(nbd: ::Handle) -> libnbd::Result {
+let info = Arc::new(Mutex::new(CtxInfo {
+has_alloc: false,
+count: 0,
+}));
+let info_clone = info.clone();
+let replies = nbd.opt_set_meta_context(move |ctx| {
+let mut info = info_clone.lock().unwrap();
+info.count += 1;
+if ctx == CONTEXT_BASE_ALLOCATION {
+info.has_alloc = true;
+}
+0
+})?;
+let info = Arc::into_inner(info).unwrap().into_inner().unwrap();
+assert_eq!(info.count, replies);
+Ok(info)
+}
+

...

+
+// nbdkit does not match wildcard for SET, even though it does for LIST
+nbd.clear_meta_contexts().unwrap();
+nbd.add_meta_context("base:").unwrap();
+assert_eq!(
+set_meta_ctxs().unwrap(),
+CtxInfo {
+count: 0,
+has_alloc: false
+}

whereas here, you did not.  Does it make a difference?  'make check'
still passes, and rustfmt doesn't complain, if I temporarily add a
trailing comma here.



No, it doesn't make a difference. I nearly always runs rustfmt on my 
code but it doesn't seem that rustfmt cares about that comma.




I also note that 'rustfmt --check rust/tests/*.rs' flags a few style
suggestions.  I had started work on automating gofmt for all
checked-in *.go files; maybe I should revive that patch and extend it
to also automate rustfmt on all checked-in *.rs files.  Here's what
rustfmt suggested to me (rustfmt 1.5.2-stable ( ) on Fedora 38):

Diff in /home/eblake/libnbd/rust/tests/test_200_connect_command.rs at line 17:
  
  #![deny(warnings)]
  
-

  #[test]
  fn test_connect_command() {
  let nbd = libnbd::Handle::new().unwrap();
Diff in /home/eblake/libnbd/rust/tests/test_200_connect_command.rs at line 24:
-nbd.connect_command(&[
-"nbdkit",
-"-s",
-"--exit-with-parent",
-"-v",
-"null",
-])
-.unwrap();
+nbd.connect_command(&["nbdkit", "-s", "--exit-with-parent", "-v", "null"])
+.unwrap();
  }
  
Diff in /home/eblake/libnbd/rust/tests/test_300_get_size.rs at line 17:
  
  #![deny(warnings)]
  
-

  #[test]
  fn test_get_size() {
  let nbd = libnbd::Handle::new().unwrap();
Diff in /home/eblake/libnbd/rust/tests/test_620_stats.rs at line 17:
  
  #![deny(warnings)]
  
-

  #[test]
  fn test_stats() {
  let nbd = libnbd::Handle::new().unwrap();
Diff in /home/eblake/libnbd/rust/tests/test_620_stats.rs at line 31:
  // Connection performs handshaking, which increments stats.
  // The number of bytes/chunks here may grow over time as more features get
  // automatically negotiated, so merely check that they are non-zero.
-nbd.connect_command(&[
-"nbdkit",
-"-s",
-"--exit-with-parent",
-"-v",
-"null",
-])
-.unwrap();
+nbd.connect_command(&["nbdkit", "-s", "--exit-with-parent", "-v", "null"])
+.unwrap();
  
  let bs1 = nbd.stats_bytes_sent();

  let cs1 = nbd.stats_chunks_sent();



Automating rustfmt on all checked-in rust files sounds like a good idea. 
It is strange that those file were not proprly formatted since I just 
mentioned that I nearly always runs rustfmt on every edit. For the files 
which are not upstream yet, (test_async_*), I will fix the formatting in 
the next patch series. For the other files I will create an extra patch 
to correct the formatting.



--

Best regards,

Tage


___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v7 0/9] Rust Bindings for Libnbd

[Tage Johansson]

On 8/16/2023 9:11 PM, Eric Blake wrote:

On Thu, Aug 10, 2023 at 11:24:27AM +, Tage Johansson wrote:

This is the 7th version of the Rust bindings for Libnbd. It is more or
less identical to the 6th version without the already merged patches.

Best regards,
Tage


I still hope to do more review of both merged patches and this one (in
part, to learn more about Rust myself).  But my first observation is
that you currently have several build failures on different platforms.
Here's some CI links summarizing the types of failures I'm seeing,
when I try to turn on --enable-rust as part of our CI coverage:



All of these errors are due to the Rustc version is too old. At least 
rustc 1.70.0 is required. Is it possible to update rust in the CI 
machines, perhaps with rustup?



It is possible to add a minimum version requirement in Cargo.toml, I 
guess I should do that to make the errors a bit more clear.




--

Best regards,

Tage



On Debian 12 (similar on Ubuntu):
https://gitlab.com/ebblake/libnbd/-/jobs/4886374776
line 261:
Get:105 http://deb.debian.org/debian bookworm/main amd64 cargo amd64 
0.66.0+ds1-1 [3419 kB]
line 3717..:
  Documenting libnbd v0.1.0 (/builds/ebblake/libnbd/rust)
error[E0432]: unresolved imports `std::os::fd::AsRawFd`, 
`std::os::fd::OwnedFd`, `std::os::fd::RawFd`
   --> src/bindings.rs:30:19
|
30 | use std::os::fd::{AsRawFd, OwnedFd, RawFd};
|   ^^^  ^^^  ^ no `RawFd` in `os::fd`
|   ||
|   |no `OwnedFd` in `os::fd`
|   no `AsRawFd` in `os::fd`
error[E0412]: cannot find type `c_uint` in this scope
--> src/bindings.rs:120:30
 |
120 |   where F: FnMut(&[u8], u64, c_uint,  c_int) -> c_int + Send + Sync
 |  ^^ not found in this scope
 |
help: consider importing one of these items
 |
24  | use core::ffi::c_uint;
 |
24  | use libc::c_uint;
 |
24  | use std::os::raw::c_uint;
 |
...
Line 5248..:
error[E0603]: module `fd` is private
   --> src/bindings.rs:30:14
|
30 | use std::os::fd::{AsRawFd, OwnedFd, RawFd};
|  ^^ private module
|
note: the module `fd` is defined here
Line 5270..:
Compiling libnbd v0.1.0 (/builds/ebblake/libnbd/rust)
error[E0432]: unresolved imports `std::os::fd::AsRawFd`, 
`std::os::fd::OwnedFd`, `std::os::fd::RawFd`
   --> src/bindings.rs:30:19
|
30 | use std::os::fd::{AsRawFd, OwnedFd, RawFd};
|   ^^^  ^^^  ^ no `RawFd` in `os::fd`
|   ||
|   |no `OwnedFd` in `os::fd`
|   no `AsRawFd` in `os::fd`
Line 7120..:
error[E0599]: no method named `cast_mut` found for raw pointer `*const i8` in 
the current scope
 --> src/bindings.rs:4231:48
  |
4231 |   queries_ffi_c_strs.iter().map(|x| x.as_ptr().cast_mut()).collect();
  | help: there is 
an associated function with a similar name: `as_mut`
  |
  = note: try using `<*const T>::as_ref()` to get a reference to the type 
behind the pointer: https://doc.rust-lang.org/std/primitive.pointer.html#method.as_ref
  = note: using `<*const T>::as_ref()` on a pointer which is unaligned or 
points to invalid or uninitialized memory is undefined behavior


On to the next one. Centos-stream-8 (and almalinux):
https://gitlab.com/ebblake/libnbd/-/jobs/4886374755
Line 18: cargox86_64  
1.69.0-1.module_el8+430+506bc849  appstream   4.9 M
https://gitlab.com/ebblake/libnbd/-/jobs/4886374755/artifacts/external_file/rust/test-suite.log
Compiling libnbd v0.1.0 (/builds/ebblake/libnbd/rust)
Compiling tempfile v3.7.1
error[E0658]: use of unstable library feature 'arc_into_inner'
   --> tests/test_245_opt_list_meta_queries.rs:49:16
|
49 | let info = Arc::into_inner(info).unwrap().into_inner().unwrap();
|^^^
|
= note: see issue #106894 <https://github.com/rust-lang/rust/issues/106894> 
for more information





___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v7 9/9] rust: async: Add an example

[Tage Johansson]

On 8/15/2023 11:00 AM, Richard W.M. Jones wrote:

On Thu, Aug 10, 2023 at 11:24:36AM +, Tage Johansson wrote:

This patch adds an example using the asynchronous Rust bindings.
---
  rust/Cargo.toml|   1 +
  rust/examples/concurrent-read-write.rs | 135 +
  rust/run-tests.sh.in   |   2 +
  3 files changed, 138 insertions(+)
  create mode 100644 rust/examples/concurrent-read-write.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index d001248..4332783 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -54,5 +54,6 @@ default = ["log", "tokio"]
  anyhow = "1.0.72"
  once_cell = "1.18.0"
  pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
  tempfile = "3.6.0"
  tokio = { version = "1.29.1", default-features = false, features = ["rt-multi-thread", 
"macros"] }
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..a1c3e8a
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,135 @@
+//! Example usage with nbdkit:
+//!
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'

It would be nice to make the example use a URI (ie.  nbd_connect_uri).
Is that possible?



Yes, of course.



+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+nbd.connect_unix()?;

ie. here ^^^


+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket.clone().into(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket: PathBuf,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// futures must be `'static`, hence we wrap the handle in an [Arc].
+let nbd = Arc::new(libnbd::AsyncHandle::new()?);
+nbd.connect_unix(socket).await?;
+
+let mut rng = SmallRng::seed_from_u64(44 as u64);
+
+// Issue commands.
+let mut stats = Stats::default();
+let mut join_set = JoinSet::new();
+//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+while stats.requests < NR_CYCLES || !join_set.is_empty() {
+while sta

Re: [Libguestfs] [libnbd PATCH v7 9/9] rust: async: Add an example

[Tage Johansson]

On 8/15/2023 11:07 AM, Richard W.M. Jones wrote:

So what do I think about the patch series as a whole ...  (in
particular, the patches I didn't add Reviewed-by tags to).

It would be much nicer IMHO if we didn't have to define callback
lifetimes in this way, since they were not intended to be classified
into async_kind / cblifetime / cbcount, and this might limit our
options for new ABIs in future.

I see two ways to go here:

(1) (Easier for now, problems in future) Rename async_kind, cblifetime
and cbcount as rust_async_kind, rust_cblifetime, rust_cbcount, which
would in some sense limit the scope of getting these right to the Rust
bindings.

This defers the pain til later (maybe never, if we never added an ABI
which didn't satisfy these constraints).

(2) (Harder for now, no problems in future) Use a reference count in
the Rust bindings, which is how the other bindings work.  It makes the
Rust bindings more awkward to use, but does more accurately express
he actual intention of the API.

Discuss ...



I want to emphasize the different purposes of async_kind, cblifetime and 
cbcount:


- async_kind: Tells what calls are asynchronous or not and how they

mark completion. It has nothing to do with reference counting and is

  required for the asynchronous Rust bindings to work.

- cblifetime: Tells for how long a closure might be used. Without this

  the user would need to use a reference counter (Arc) and a Mutex. The

  API would be a bit more verbose/awkward to use, but it would

  definitely be possible.

- cbcount: Tells for how many times the closure might be called. Without

  this, all closures would be FnMut, including completion callbacks.

  I don't think it would hurt too much to remove this.


So unless someone comes up with a better solution, I think we have to 
keep the async_kind field. Although it would make sense to rename it to 
rust_async_kind.



Personally I think the other two makes sense as well, but they can 
definitely be removed if that's what the community wants.



--

Best regards,

Tage



Rich.



___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH] rust: Add cargo as an optional build dependency in README

[Tage Johansson]

This patch adds cargo to the list of optional build dependencies in
top-level README.md.
---
 README.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/README.md b/README.md
index c716661..5ce27bd 100644
--- a/README.md
+++ b/README.md
@@ -106,6 +106,8 @@ ## Building from source
 * FUSE 3 to build the nbdfuse program.
 * Linux >= 6.0 and ublksrv library to build nbdublk program.
 * go and cgo, for compiling the golang bindings and tests.
+* cargo with a recent stable toolchain is required to build
+  the Rust bindings.
 * bash-completion >= 1.99 for tab completion.

 Optional, only needed to run the test suite:

base-commit: 33a47171653931b7e255e33930697a55eae1493b
prerequisite-patch-id: ff317be8e27608697ee070388502566ecf8546bb
prerequisite-patch-id: 6a68a5da00c78e039972118bfde68cf87d7db6af
prerequisite-patch-id: a6ae1f1d90ca8cb69d17c977428855acadd47608
prerequisite-patch-id: 053dc904d579f2d065228c1c5780109871e9cd66
prerequisite-patch-id: 99eb277dfb04af7930cc827d85fd011fc54bdd4c
prerequisite-patch-id: 0b4159540024f935140d070146d89f95b96576fa
prerequisite-patch-id: f320498a380789b51bf65b603c0627167453352c
prerequisite-patch-id: f9aea5f724ac167744aa72456340c370a43611a2
prerequisite-patch-id: 9ff0b41ad9fd00d5d67de92d574255b46cdf150a
prerequisite-patch-id: 1fd2bcd42012e5d0ab10f63a1849f2ccc706e6d6
prerequisite-patch-id: 7c5b4b59f2765e8c255857a0834805d19cecf65d
prerequisite-patch-id: 4d7bd8e07e4710e3420c1ee71502f0fd0da91ea7
prerequisite-patch-id: 5ed2a56efbc9554261f875cd299dd2b7483c78c8
prerequisite-patch-id: 52d475de3ab033859d6bd87996078ae7b3385695
prerequisite-patch-id: c6a05c89340bed6471de1d74ef95acb5b6ac2c25
prerequisite-patch-id: 097dd7285726e45b02493fc306fd3017748d50e2
prerequisite-patch-id: 359900c28144cf2059e23a2911ea9f9f9ca5db23
prerequisite-patch-id: 4db98f7b211c0de9a4095b300970e1973cf0716c
prerequisite-patch-id: 0bb320af5109c1c21e5b76d44e6ec1e7e685fd9f
prerequisite-patch-id: 205525d8ea09e77ea13f43d0720153ed5904dbcd
prerequisite-patch-id: f76cdc6ceca68268df92341985068388f25291ff
prerequisite-patch-id: 84cb140c8f0dd089ca8e9567cc2117bf38c9e558
prerequisite-patch-id: b2c3285d05fd56a258d3ec47d7d4cdcf06a57014
prerequisite-patch-id: 8938eab7a42f8a7ed82c9372be9bf29c2991787f
prerequisite-patch-id: 7694233787dd758add8c30e69965dfd1ffee7012
prerequisite-patch-id: 770993f2baac576e163abd73c2d4af3d8841b032
prerequisite-patch-id: e85690fa670dce4f6eb84b01c1b16700540789df
prerequisite-patch-id: 72e384aafbe3d1bec34371a2f4a862ff2db0b8ca
prerequisite-patch-id: 45e0dc7a06e0845f7ca992f663afcea8e9bc4ba0
prerequisite-patch-id: 0c3ac3d6f776a2e09e45534f3f74eaeb672dd003
prerequisite-patch-id: 0fc920e92dd6a36d97b4e60a68060031fe274a3b
prerequisite-patch-id: 8467ff99068b5ca29b74813ffefe1cd5689196d1
prerequisite-patch-id: cf92490b4d427b4fb036fdd5cc682a43be82d937
prerequisite-patch-id: 6518b46cd0dcfc9aec830b0532ae1fbfd3f9bbea
prerequisite-patch-id: 8e75ca2543d6e901a472b32a3f0daf81e79c090c
prerequisite-patch-id: 97ec755ec74a802d1b4efeec8be66dcdb9ac739d
prerequisite-patch-id: 95dd4160a43fa2bea8fbf614c912263cd5ad840d
prerequisite-patch-id: f219538637190d9340f30acef9b3d24125d3bb69
prerequisite-patch-id: c11ebccd6db5c175ed40b13916f9b1affd8179d8
prerequisite-patch-id: 6c6ada8eaf72221c82d11778ba6dd05eec60bdb1
prerequisite-patch-id: e1eb6d1aa978f8daeb5caa4281f46a29465e0fc9
prerequisite-patch-id: 39725e9d285db482e5314f4c1e2a19b80906f2ed
prerequisite-patch-id: 01e0ec094775996b95f74898fe5ff9840ca2dc0c
prerequisite-patch-id: a7c24c91cb12dc41a332cf8b820c8548f0d8e850
prerequisite-patch-id: 821c9c061ab5b577483794a635ba5b38dd07fd70
prerequisite-patch-id: b11622efc2554e1a2fae9fc7f0c957cb08d0ff87
prerequisite-patch-id: 1983f74a0c2991333ff04a2099bc2f15509f7444
prerequisite-patch-id: 7e7f28602431481b0817980238039264781e114e
prerequisite-patch-id: 7d662d655ccdd81d3c2e6e7ddd69f527bffdd19b
prerequisite-patch-id: d6bcb838a1875541f3f125b95f346c21a7d614ea
prerequisite-patch-id: 403572023dc08169c3f5052d5602ffe06885e3e8
prerequisite-patch-id: 0a6846e107ecffde51c1436b4b81256c7c459000
prerequisite-patch-id: e42c3f5f2ba4afcc038acd9ac4442dee4a443117
prerequisite-patch-id: 56d2f65eaa08a569286e2cd55b27ac2ca9838952
prerequisite-patch-id: c603d778c9ed6e2eaf211909085f89261d094694
prerequisite-patch-id: 33346c3ac9b34b83c6c310ead5b8b71b08f84af7
prerequisite-patch-id: b75aca15d5568618a38235cc8353bf22c6a340d2
prerequisite-patch-id: 7d7de645a10c4afef98c4408a6c0efe120632e9b
prerequisite-patch-id: ef892445bd153ebadbcde3fea8d42a289f594023
--
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v5 00/12] Rust Bindings for Libnbd

[Tage Johansson]

On 8/11/2023 10:54 PM, Eric Blake wrote:

On Thu, Aug 03, 2023 at 03:36:04PM +, Tage Johansson wrote:

A new version of the Rust bindings for Libnbd. Most changes have been
mentioned as replies to the comments I got on v4. But the most significant
change is probably that the patches have been reordered.

Best regards,
Tage


Tage Johansson (12):
   rust: create basic Rust bindings
   rust: Add a couple of integration tests
   rust: Make it possible to run tests with Valgrind
   rust: Add some examples
   generator: Add information about asynchronous handle calls
   generator: Add information about the lifetime of closures
   rust: Use more specific closure traits
   rust: async: Create an async friendly handle type
   generator: Add `modifies_fd` flag to the [call] structure
   rust: async: Use the modifies_fd flag to exclude calls
   rust: async: Add a couple of integration tests
   rust: async: Add an example

  .gitignore|  10 +
  .ocamlformat  |   4 +
  Makefile.am   |   2 +
  configure.ac  |  30 +
  generator/API.ml  |  84 ++

...

  rust/tests/test_log/mod.rs|  86 ++
  rustfmt.toml  |  19 +
  scripts/git.orderfile |  12 +
  68 files changed, 4851 insertions(+)

None of these touched top-level README.md, which should now list cargo
and any other new optional build dependencies that must be installed
for enabling Rust bindings.



True, I'll add that in an upcoming patch.


An unrelated question: Should the patch which adds cargo as an optional 
build dependency to README be a reply to this message or should I send 
it as a new thread? I'll send it as a new thread now, but what is the 
best way to do it?



--

Best regards,

Tage

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v5 01/12] rust: create basic Rust bindings

[Tage Johansson]

On 8/11/2023 3:41 PM, Richard W.M. Jones wrote:

On Fri, Aug 11, 2023 at 08:22:34AM -0500, Eric Blake wrote:

On Thu, Aug 03, 2023 at 03:36:05PM +, Tage Johansson wrote:

This commit creates basic Rust bindings in the rust directory.
The bindings are generated by generator/Rust.ml and
generator/RustSys.ml.
---
+++ b/rust/Cargo.toml
@@ -0,0 +1,49 @@
+# nbd client library in userspace
+# Copyright Tage Johansson
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.

This says LGPLv2+...


+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+[workspace]
+
+[workspace.package]
+authors = ["Tage Johansson"]
+version = "0.1.0"
+edition = "2021"
+description = "Rust bindings for libnbd, a client library for controlling block 
devices over a network."
+license = "LGPL-2.1-only"

...but this does not.  Why the discrepancy?  It is a disservice to
clients to have a more restrictive license for the Rust bindings than
for the rest of libnbd.

I agree with Eric that we should fix this (it seems like a simple
oversight rather than a deliberate thing).  If Tage you give me the OK
I will change it in the upstream repo.



You have the OK to change it in the upstream repo. It should be 
"GPL-2.1-or-later" according to this site 
<https://spdx.github.io/license-list-data/LGPL-2.1-or-later.html>.



Best regards,

Tage



Rich.
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v5 01/12] rust: create basic Rust bindings

[Tage Johansson]

On 8/11/2023 2:00 PM, Eric Blake wrote:

On Thu, Aug 03, 2023 at 03:36:05PM +, Tage Johansson wrote:

This commit creates basic Rust bindings in the rust directory.
The bindings are generated by generator/Rust.ml and
generator/RustSys.ml.
---
--- /dev/null
+++ b/generator/RustSys.ml
+(** Print the struct for a closure. *)
+let print_closure_struct { cbname; cbargs } =
+  pr "#[repr(C)]\n";
+  pr "#[derive(Debug, Clone, Copy)]\n";
+  pr "pub struct nbd_%s_callback {\n" cbname;
+  pr "pub callback: \n";
+  pr "  Option c_int>,\n"
+(cbargs |> List.map cbarg_types |> List.flatten |> String.concat ", ");
+  pr "pub user_data: *mut c_void,\n";
+  pr "pub free: Option,\n";
+  pr "}\n"

Why is 'callback' an Option<> rather than a mandatory argument?  I get
that 'free' must be an Option<> (because it corresponds to an
OClosure, which is an optional callback), but 'callback' is a
mandatory function pointer in the C API; why would it ever be
acceptable to pass None instead of Some?



It uses the "nullable pointer optimization" 
<https://doc.rust-lang.org/nomicon/ffi.html#the-nullable-pointer-optimization> 
which makes an Option where T is a non nullable type be represented 
with no extra space, and the None variant is represented by NULL.



Keep in mind that this is only the internal FFI bindings and the Option 
will never be part of the public interface.



Best regards,

Tage

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v7 4/9] rust: Use more specific closure traits

[Tage Johansson]

For closures with cbcount = CBOnce, FnOnce will be used instead of
FnMut. Moreover, closures in synchronous commands with
cblifetime = CBCommand will not need to live for the static lifetime.
See [here](https://doc.rust-lang.org/std/ops/trait.FnOnce.html) for more
information about the advantages of using `FnOnce` when possible.
---
 generator/Rust.ml  | 44 
 rust/src/handle.rs |  2 ++
 rust/src/types.rs  |  2 ++
 3 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 431c814..c409961 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -129,7 +129,7 @@ let rust_cbarg_name : cbarg -> string = function
   | CBArrayAndLen (arg, _) | CBMutable arg -> rust_arg_name arg
 
 (* Get the Rust type for an argument. *)
-let rec rust_arg_type : arg -> string = function
+let rec rust_arg_type ?(async_kind = None) : arg -> string = function
   | Bool _ -> "bool"
   | Int _ -> "c_int"
   | UInt _ -> "c_uint"
@@ -149,15 +149,18 @@ let rec rust_arg_type : arg -> string = function
   | BytesOut _ -> " [u8]"
   | BytesPersistIn _ -> "&'static [u8]"
   | BytesPersistOut _ -> "&'static mut [u8]"
-  | Closure { cbargs } -> "impl " ^ rust_closure_trait cbargs
+  | Closure { cbargs; cbcount } -> (
+  match async_kind with
+  | Some _ -> "impl " ^ rust_closure_trait cbargs cbcount
+  | None -> "impl " ^ rust_closure_trait cbargs cbcount ~lifetime:None)
 
 (* Get the Rust closure trait for a callback, That is `Fn*(...) -> ...)`. *)
-and rust_closure_trait ?(lifetime = Some "'static") cbargs : string =
+and rust_closure_trait ?(lifetime = Some "'static") cbargs cbcount : string =
   let rust_cbargs = String.concat ", " (List.map rust_cbarg_type cbargs)
-  and lifetime_constraint =
-match lifetime with None -> "" | Some x -> " + " ^ x
-  in
-  "FnMut(" ^ rust_cbargs ^ ") -> c_int + Send + Sync" ^ lifetime_constraint
+  and closure_type =
+match cbcount with CBOnce -> "FnOnce" | CBMany -> "FnMut"
+  and lifetime = match lifetime with None -> "" | Some x -> " + " ^ x in
+  sprintf "%s(%s) -> c_int + Send + Sync%s" closure_type rust_cbargs lifetime
 
 (* Get the Rust type for a callback argument. *)
 and rust_cbarg_type : cbarg -> string = function
@@ -171,8 +174,8 @@ and rust_cbarg_type : cbarg -> string = function
   | CBMutable arg -> " " ^ rust_arg_type arg
 
 (* Get the type of a rust optional argument. *)
-let rust_optarg_type : optarg -> string = function
-  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x))
+let rust_optarg_type ?(async_kind = None) : optarg -> string = function
+  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x) ~async_kind)
   | OFlags (name, flags, _) ->
   sprintf "Option<%s>" (rust_arg_type (Flags (name, flags)))
 
@@ -437,8 +440,8 @@ let ffi_ret_to_rust call =
closure data, and a free function for the closure data. This struct is what
will be sent to a C function taking the closure as an argument. In fact,
the struct itself is generated by rust-bindgen. *)
-let print_rust_closure_to_raw_fn { cbname; cbargs } =
-  let closure_trait = rust_closure_trait cbargs ~lifetime:None in
+let print_rust_closure_to_raw_fn { cbname; cbargs; cbcount } =
+  let closure_trait = rust_closure_trait cbargs cbcount ~lifetime:None in
   let ffi_cbargs_names = List.flatten (List.map ffi_cbarg_names cbargs) in
   let ffi_cbargs_types = List.flatten (List.map ffi_cbarg_types cbargs) in
   let rust_cbargs_names = List.map rust_cbarg_name cbargs in
@@ -453,16 +456,24 @@ let print_rust_closure_to_raw_fn { cbname; cbargs } =
(List.map2 (sprintf "%s: %s") ffi_cbargs_names ffi_cbargs_types));
   pr "  where F: %s\n" closure_trait;
   pr "{\n";
-  pr "let callback_ptr = data as *mut F;\n";
-  pr "let callback =  *callback_ptr;\n";
+  (match cbcount with
+  | CBMany ->
+  pr "let callback_ptr = data as *mut F;\n";
+  pr "let callback =  *callback_ptr;\n"
+  | CBOnce ->
+  pr "let callback_ptr = data as *mut Option;\n";
+  pr "let callback_option:  Option =  *callback_ptr;\n";
+  pr "let callback: F = callback_option.take().unwrap();\n");
   List.iter ffi_cbargs_to_rust cbargs;
   pr "callback(%s)\n" (String.concat ", " rust_cbargs_names);
   pr "}\n";
-  pr "let callback_data = Box::into_raw(Box::new(f));\n";
+  pr "let callback_data = Box::into_raw(Box::new(%s));\n"
+(match cbcount with CBMany -> "f" | CBOnce -> "Some(f)");
   pr "sys::nbd_%s_callback {\n" cbname;
   pr "callback: Some(call_closure::),\n";
   pr "user_data: callback_data as *mut _,\n";
-  pr "free: Some(utils::drop_data::),\n";
+  pr "free: Some(utils::drop_data::<%s>),\n"
+(match cbcount with CBMany -> "F" | CBOnce -> "Option");
   pr "}\n";
   pr "}\n";
   pr "\n"
@@ -519,7 +530,8 @@ let print_rust_handle_method (name, call) =
   let 

[Libguestfs] [libnbd PATCH v7 5/9] rust: async: Create an async friendly handle type

[Tage Johansson]

 name
+(rust_async_handle_call_args call);
+  pr "{\n";
+  print_ffi_call aio_name "self.data.handle.handle" call;
+  pr "?;\n";
+  pr "let (ret_tx, ret_rx) = oneshot::channel::>();\n";
+  pr "let mut ret_tx = Some(ret_tx);\n";
+  pr "let completion_predicate = \n";
+  pr " move |handle: , res: <()>| {\n";
+  pr "  let ret = if let Err(_) = res {\n";
+  pr "res.clone()\n";
+  pr "  } else {\n";
+  pr "if handle.%s() != %s { return false; }\n" predicate value;
+  pr "else { Ok(()) }\n";
+  pr "  };\n";
+  pr "  ret_tx.take().unwrap().send(ret).ok();\n";
+  pr "  true\n";
+  pr "};\n";
+  pr "self.add_command(completion_predicate)?;\n";
+  pr "ret_rx.await.unwrap()\n";
+  pr "}\n\n"
+
+(* Print an impl with all handle calls. *)
+let print_rust_async_handle_impls () =
+  pr "impl AsyncHandle {\n";
+  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
+  NameMap.iter
+(fun name (call, async_kind) ->
+  match async_kind with
+  | WithCompletionCallback ->
+  print_rust_async_handle_call_with_completion_cb name call
+  | ChangesState (predicate, value) ->
+  print_rust_async_handle_call_changing_state name call
+(predicate, value))
+async_handle_calls;
+  pr "}\n\n"
+
+let print_rust_async_imports () =
+  pr "use crate::{*, types::*};\n";
+  pr "use os_socketaddr::OsSocketAddr;\n";
+  pr "use std::ffi::*;\n";
+  pr "use std::mem;\n";
+  pr "use std::net::SocketAddr;\n";
+  pr "use std::os::fd::{AsRawFd, OwnedFd};\n";
+  pr "use std::os::unix::prelude::*;\n";
+  pr "use std::path::PathBuf;\n";
+  pr "use std::ptr;\n";
+  pr "use std::sync::Arc;\n";
+  pr "use tokio::sync::oneshot;\n";
+  pr "\n"
+
+let generate_rust_async_bindings () =
+  generate_header CStyle ~copyright:"Tage Johansson";
+  pr "\n";
+  print_rust_async_imports ();
+  print_rust_async_handle_impls ()
diff --git a/generator/Rust.mli b/generator/Rust.mli
index 450e4ca..0960170 100644
--- a/generator/Rust.mli
+++ b/generator/Rust.mli
@@ -18,3 +18,5 @@
 
 (* Print all flag-structs, enums, constants and handle calls in Rust code. *)
 val generate_rust_bindings : unit -> unit
+
+val generate_rust_async_bindings : unit -> unit
diff --git a/generator/generator.ml b/generator/generator.ml
index 8c9a585..11bec4d 100644
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -69,3 +69,5 @@ let () =
 RustSys.generate_rust_sys_bindings;
   output_to ~formatter:(Some Rustfmt) "rust/src/bindings.rs"
 Rust.generate_rust_bindings;
+  output_to ~formatter:(Some Rustfmt) "rust/src/async_bindings.rs"
+Rust.generate_rust_async_bindings;
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 04e371e..e076826 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -44,9 +44,11 @@ os_socketaddr = "0.2.4"
 thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
+tokio = { optional = true, version = "1.29.1", default-features = false, 
features = ["rt", "sync", "net"] }
+epoll = "4.3.3"
 
 [features]
-default = ["log"]
+default = ["log", "tokio"]
 
 [dev-dependencies]
 anyhow = "1.0.72"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index 295254d..9fbe9fe 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -19,6 +19,7 @@ include $(top_srcdir)/subdir-rules.mk
 
 generator_built = \
libnbd-sys/src/generated.rs \
+   src/async_bindings.rs \
src/bindings.rs \
$(NULL)
 
@@ -30,6 +31,7 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   src/async_handle.rs \
examples/connect-command.rs \
examples/get-size.rs \
examples/fetch-first-sector.rs \
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
new file mode 100644
index 000..4223b80
--- /dev/null
+++ b/rust/src/async_handle.rs
@@ -0,0 +1,268 @@
+// nbd client library in userspace
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should 

[Libguestfs] [libnbd PATCH v7 7/9] rust: async: Use the modifies_fd flag to exclude calls

[Tage Johansson]

All handle calls which has the modifies_fd flag set to true will be
excluded from AsyncHandle (the asynchronous handle in the rust
bindings). This is a better approach then listing all calls that should
be excluded in Rust.ml explicetly.
---
 generator/Rust.ml | 60 +--
 1 file changed, 27 insertions(+), 33 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index b7d9bcf..39ad22a 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -586,18 +586,17 @@ let generate_rust_bindings () =
 
 let excluded_handle_calls : NameSet.t =
   NameSet.of_list
-[
-  "aio_get_fd";
-  "aio_get_direction";
-  "aio_notify_read";
-  "aio_notify_write";
-  "clear_debug_callback";
-  "get_debug";
-  "poll";
-  "poll2";
-  "set_debug";
-  "set_debug_callback";
-]
+  @@ [
+   "aio_get_fd";
+   "aio_get_direction";
+   "clear_debug_callback";
+   "get_debug";
+   "set_debug";
+   "set_debug_callback";
+ ]
+  @ (handle_calls
+|> List.filter (fun (_, { modifies_fd }) -> modifies_fd)
+|> List.map (fun (name, _) -> name))
 
 (* A mapping with names as keys. *)
 module NameMap = Map.Make (String)
@@ -608,16 +607,16 @@ let strip_aio name : string =
 String.sub name 4 (String.length name - 4)
   else failwithf "Asynchronous call %s must begin with aio_" name
 
-(* A map with all asynchronous handle calls. The keys are names with "aio_"
-   stripped, the values are a tuple with the actual name (with "aio_"), the
-   [call] and the [async_kind]. *)
+(* A map with all asynchronous handle calls. The keys are names, the values
+   are tuples of: the name with "aio_" stripped, the [call] and the
+   [async_kind]. *)
 let async_handle_calls : ((string * call) * async_kind) NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
   |> List.filter_map (fun (name, call) ->
  call.async_kind
  |> Option.map (fun async_kind ->
-(strip_aio name, ((name, call), async_kind
+(name, ((strip_aio name, call), async_kind
   |> List.to_seq |> NameMap.of_seq
 
 (* A mapping with all synchronous (not asynchronous) handle calls. Excluded
@@ -627,11 +626,7 @@ let async_handle_calls : ((string * call) * async_kind) 
NameMap.t =
 let sync_handle_calls : call NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
-  |> List.filter (fun (name, _) ->
- (not (NameMap.mem name async_handle_calls))
- && not
-  (String.starts_with ~prefix:"aio_" name
-  && NameMap.mem (strip_aio name) async_handle_calls))
+  |> List.filter (fun (n, _) -> not (NameMap.mem n async_handle_calls))
   |> List.to_seq |> NameMap.of_seq
 
 (* Get the Rust type for an argument in the asynchronous API. Like
@@ -676,7 +671,7 @@ let print_rust_sync_handle_call name call =
 (* Print the Rust function for an asynchronous handle call with a completion
callback. (Note that "callback" might be abbreviated with "cb" in the
following code. *)
-let print_rust_async_handle_call_with_completion_cb name (aio_name, call) =
+let print_rust_async_handle_call_with_completion_cb aio_name (name, call) =
   (* An array of all optional arguments. Useful because we need to deel with
  the index of the completion callback. *)
   let optargs = Array.of_list call.optargs in
@@ -756,7 +751,7 @@ let print_rust_async_handle_call_with_completion_cb name 
(aio_name, call) =
completion by changing state. The predicate is a call like
"aio_is_connecting" which should get the value (like false) for the call to
be complete. *)
-let print_rust_async_handle_call_changing_state name (aio_name, call)
+let print_rust_async_handle_call_changing_state aio_name (name, call)
 (predicate, value) =
   let value = if value then "true" else "false" in
   print_rust_handle_call_comment call;
@@ -785,16 +780,15 @@ let print_rust_async_handle_call_changing_state name 
(aio_name, call)
 (* Print an impl with all handle calls. *)
 let print_rust_async_handle_impls () =
   pr "impl AsyncHandle {\n";
-  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
-  NameMap.iter
-(fun name (call, async_kind) ->
-  match async_kind with
-  | WithCompletionCallback ->
-  print_rust_async_handle_call_with_completion_cb name call
-  | ChangesState (predicate, value) ->
-  print_rust_async_handle_call_changing_state name call
-(predicate, value))
-async_handle_calls;
+  sync_handle_calls |> NameMap.iter print_rust_sync_handle_call;
+  async_handle_calls
+  |> NameMap.iter (fun name (call, async_kind) ->
+ match async_kind with
+ | WithCompletionCallback ->
+ print_rust_async_handle_call_with_completion_cb name call
+ | ChangesState (predicate, value) ->
+ print_rust_async_handle_call_changing_state 

[Libguestfs] [libnbd PATCH v7 0/9] Rust Bindings for Libnbd

[Tage Johansson]

This is the 7th version of the Rust bindings for Libnbd. It is more or
less identical to the 6th version without the already merged patches.

Best regards,
Tage


Tage Johansson (9):
  rust: Make it possible to run tests with Valgrind
  generator: Add information about asynchronous handle calls
  generator: Add information about the lifetime of closures
  rust: Use more specific closure traits
  rust: async: Create an async friendly handle type
  generator: Add `modifies_fd` flag to the [call] structure
  rust: async: Use the modifies_fd flag to exclude calls
  rust: async: Add a couple of integration tests
  rust: async: Add an example

 generator/API.ml  |  84 ++
 generator/API.mli |  35 +++
 generator/Rust.ml | 278 +-
 generator/Rust.mli|   2 +
 generator/generator.ml|   2 +
 rust/Cargo.toml   |   6 +-
 rust/Makefile.am  |   5 +
 rust/examples/concurrent-read-write.rs| 135 +
 rust/run-tests.sh.in  |  18 +-
 rust/src/async_handle.rs  | 268 +
 rust/src/handle.rs|   2 +
 rust/src/lib.rs   |   8 +
 rust/src/types.rs |   2 +
 rust/tests/test_async_100_handle.rs   |  25 ++
 rust/tests/test_async_200_connect_command.rs  |  33 +++
 rust/tests/test_async_210_opt_abort.rs|  32 ++
 rust/tests/test_async_220_opt_list.rs |  81 +
 rust/tests/test_async_230_opt_info.rs | 122 
 rust/tests/test_async_240_opt_list_meta.rs| 147 +
 .../test_async_245_opt_list_meta_queries.rs   |  91 ++
 rust/tests/test_async_250_opt_set_meta.rs | 122 
 .../test_async_255_opt_set_meta_queries.rs| 107 +++
 rust/tests/test_async_400_pread.rs|  40 +++
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 
 rust/tests/test_async_460_block_status.rs |  92 ++
 rust/tests/test_async_620_stats.rs|  76 +
 scripts/git.orderfile |   1 +
 28 files changed, 1934 insertions(+), 23 deletions(-)
 create mode 100644 rust/examples/concurrent-read-write.rs
 create mode 100644 rust/src/async_handle.rs
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs


base-commit: 33a47171653931b7e255e33930697a55eae1493b
prerequisite-patch-id: ff317be8e27608697ee070388502566ecf8546bb
prerequisite-patch-id: 6a68a5da00c78e039972118bfde68cf87d7db6af
prerequisite-patch-id: a6ae1f1d90ca8cb69d17c977428855acadd47608
prerequisite-patch-id: 053dc904d579f2d065228c1c5780109871e9cd66
prerequisite-patch-id: 99eb277dfb04af7930cc827d85fd011fc54bdd4c
prerequisite-patch-id: 0b4159540024f935140d070146d89f95b96576fa
prerequisite-patch-id: f320498a380789b51bf65b603c0627167453352c
prerequisite-patch-id: f9aea5f724ac167744aa72456340c370a43611a2
prerequisite-patch-id: 9ff0b41ad9fd00d5d67de92d574255b46cdf150a
prerequisite-patch-id: 1fd2bcd42012e5d0ab10f63a1849f2ccc706e6d6
prerequisite-patch-id: 7c5b4b59f2765e8c255857a0834805d19cecf65d
prerequisite-patch-id: 4d7bd8e07e4710e3420c1ee71502f0fd0da91ea7
prerequisite-patch-id: 5ed2a56efbc9554261f875cd299dd2b7483c78c8
prerequisite-patch-id: 52d475de3ab033859d6bd87996078ae7b3385695
prerequisite-patch-id: c6a05c89340bed6471de1d74ef95acb5b6ac2c25
prerequisite-patch-id: 097dd7285726e45b02493fc306fd3017748d50e2
prerequisite-patch-id: 359900c28144cf2059e23a2911ea9f9f9ca5db23
prerequisite-patch-id: 4db98f7b211c0de9a4095b300970e1973cf0716c
prerequisite-patch-id: 0bb320af5109c1c21e5b76d44e6ec1e7e685fd9f
prerequisite-patch-id: 205525d8ea09e77ea13f43d0720153ed5904dbcd
prerequisite-patch-id: f76cdc6ceca68268df92341985068388f25291ff
prerequisite-patch-id: 84cb140c8f0dd089ca8e9567cc2117bf38c9e558
prerequisite-patch-id: b2c3285d05fd56a258d3ec47d7d4cdcf06a57014
prerequisite-patch-id: 8938eab7a42f8a7ed82c9372be9bf29c2991787f
prerequisite-patch-id: 7694233787dd758add8c30e69965dfd1ffee7012
prerequisite-patch-id

[Libguestfs] [libnbd PATCH v7 3/9] generator: Add information about the lifetime of closures

[Tage Johansson]

Add two new fields, cblifetime and cbcount, to the `closure` type
in generator/API.ml*. cblifetime tells if the closure may only be used
for as long as the command is in flight or if the closure may be used
until the handle is destructed. cbcount tells whether the closure may
be called many times or just once.

This information is needed in the Rust bindings for:
a) Knowing if the closure trait should be FnMut or FnOnce
   (see ).
b) Knowing for what lifetime the closure should be valid. A closure that
   may be called after the function invokation has returned must live
   for the `'static` lietime. But static closures are inconveniant for
   the user since they can't effectively borrow any local data. So it is
   good if this restriction is relaxed when it is not needed.
---
 generator/API.ml  | 20 
 generator/API.mli | 17 +
 2 files changed, 37 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 99fcb82..41a6dd1 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -77,6 +77,8 @@ and ret =
 and closure = {
   cbname : string;
   cbargs : cbarg list;
+  cblifetime : cblifetime;
+  cbcount : cbcount
 }
 and cbarg =
 | CBArrayAndLen of arg * string
@@ -87,6 +89,12 @@ and cbarg =
 | CBString of string
 | CBUInt of string
 | CBUInt64 of string
+and cblifetime =
+| CBCommand
+| CBHandle
+and cbcount =
+| CBOnce
+| CBMany
 and enum = {
   enum_prefix : string;
   enums : (string * int) list
@@ -141,20 +149,28 @@ the handle from the NBD protocol handshake."
 (* Closures. *)
 let chunk_closure = {
   cbname = "chunk";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBBytesIn ("subbuf", "count");
  CBUInt64 "offset"; CBUInt "status";
  CBMutable (Int "error") ]
 }
 let completion_closure = {
   cbname = "completion";
+  cblifetime = CBCommand;
+  cbcount = CBOnce;
   cbargs = [ CBMutable (Int "error") ]
 }
 let debug_closure = {
   cbname = "debug";
+  cblifetime = CBHandle;
+  cbcount = CBMany;
   cbargs = [ CBString "context"; CBString "msg" ]
 }
 let extent_closure = {
   cbname = "extent";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "metacontext";
  CBUInt64 "offset";
  CBArrayAndLen (UInt32 "entries",
@@ -163,10 +179,14 @@ let extent_closure = {
 }
 let list_closure = {
   cbname = "list";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name"; CBString "description" ]
 }
 let context_closure = {
   cbname = "context";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name" ]
 }
 let all_closures = [ chunk_closure; completion_closure;
diff --git a/generator/API.mli b/generator/API.mli
index 361132d..ff85849 100644
--- a/generator/API.mli
+++ b/generator/API.mli
@@ -94,6 +94,12 @@ and ret =
 and closure = {
   cbname : string; (** name of callback function *)
   cbargs : cbarg list; (** all closures return int for now *)
+  (** An upper bound of the lifetime of the closure. Either it will be used for
+  as long as the command is in flight or it may be used until the handle
+  is destructed. *)
+  cblifetime : cblifetime;
+  (** Whether the callback may only be called once or many times. *)
+  cbcount : cbcount;
 }
 and cbarg =
 | CBArrayAndLen of arg * string (** array + number of entries *)
@@ -104,6 +110,17 @@ and cbarg =
 | CBString of string   (** like String *)
 | CBUInt of string (** like UInt *)
 | CBUInt64 of string   (** like UInt64 *)
+and cblifetime =
+| CBCommand (** The closure may only be used until the command is retired.
+(E.G., completion callback or list callback.) *)
+| CBHandle  (** The closure might be used until the handle is descructed.
+(E.G., debug callback.) *)
+and cbcount =
+| CBOnce (** The closure will be used 0 or 1 time if the aio_* call returned an
+ error and exactly once if the call succeeded.
+ (E.g., completion callback.) *)
+| CBMany (** The closure may be used any number of times.
+ (E.g., list callback.) *)
 and enum = {
   enum_prefix : string;(** prefix of each enum variant *)
   enums : (string * int) list (** enum names and their values in C *)
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v7 8/9] rust: async: Add a couple of integration tests

[Tage Johansson]

Add a couple of integration tests as rust/tests/test_async_*.rs. They
are very similar to the tests for the synchronous API.
---
 rust/Cargo.toml   |   1 +
 rust/tests/test_async_100_handle.rs   |  25 +++
 rust/tests/test_async_200_connect_command.rs  |  33 
 rust/tests/test_async_210_opt_abort.rs|  32 
 rust/tests/test_async_220_opt_list.rs |  81 ++
 rust/tests/test_async_230_opt_info.rs | 122 +++
 rust/tests/test_async_240_opt_list_meta.rs| 147 ++
 .../test_async_245_opt_list_meta_queries.rs   |  91 +++
 rust/tests/test_async_250_opt_set_meta.rs | 122 +++
 .../test_async_255_opt_set_meta_queries.rs| 107 +
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 +++
 rust/tests/test_async_460_block_status.rs |  92 +++
 rust/tests/test_async_620_stats.rs|  76 +
 15 files changed, 1112 insertions(+)
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index e076826..d001248 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -55,3 +55,4 @@ anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
 tempfile = "3.6.0"
+tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/tests/test_async_100_handle.rs 
b/rust/tests/test_async_100_handle.rs
new file mode 100644
index 000..e50bad9
--- /dev/null
+++ b/rust/tests/test_async_100_handle.rs
@@ -0,0 +1,25 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+//! Just check that we can link with libnbd and create a handle.
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_nbd_handle_new() {
+let _ = libnbd::AsyncHandle::new().unwrap();
+}
diff --git a/rust/tests/test_async_200_connect_command.rs 
b/rust/tests/test_async_200_connect_command.rs
new file mode 100644
index 000..8a3f497
--- /dev/null
+++ b/rust/tests/test_async_200_connect_command.rs
@@ -0,0 +1,33 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+#![deny(warnings)]
+
+
+#[tokio::test]
+async fn test_async_connect_command() {
+let nbd = libnbd::AsyncHandle::new().unwrap();
+nbd.connect_command(&[
+"nbdkit",
+"-s",
+"--exit-with-parent",
+"-v",
+"null",
+])
+.await
+.unwrap();
+}
diff --git a/rust/tests/tes

[Libguestfs] [libnbd PATCH v7 9/9] rust: async: Add an example

[Tage Johansson]

This patch adds an example using the asynchronous Rust bindings.
---
 rust/Cargo.toml|   1 +
 rust/examples/concurrent-read-write.rs | 135 +
 rust/run-tests.sh.in   |   2 +
 3 files changed, 138 insertions(+)
 create mode 100644 rust/examples/concurrent-read-write.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index d001248..4332783 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -54,5 +54,6 @@ default = ["log", "tokio"]
 anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
 tempfile = "3.6.0"
 tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..a1c3e8a
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,135 @@
+//! Example usage with nbdkit:
+//!
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//!
+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+nbd.connect_unix()?;
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket.clone().into(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket: PathBuf,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// futures must be `'static`, hence we wrap the handle in an [Arc].
+let nbd = Arc::new(libnbd::AsyncHandle::new()?);
+nbd.connect_unix(socket).await?;
+
+let mut rng = SmallRng::seed_from_u64(44 as u64);
+
+// Issue commands.
+let mut stats = Stats::default();
+let mut join_set = JoinSet::new();
+//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+while stats.requests < NR_CYCLES || !join_set.is_empty() {
+while stats.requests < NR_CYCLES && join_set.len() < MAX_IN_FLIGHT {
+// If we want to issue another request, do so.  Note that we reuse
+// the same buffer for multiple in-flight requests.  It doesn't
+// matter here because we're just trying to write random stuff,
+// but that would be Very Bad in a real application.
+// Simulate a mix of large and small requests.
+let size = if 

[Libguestfs] [libnbd PATCH v7 1/9] rust: Make it possible to run tests with Valgrind

[Tage Johansson]

Make it possible to run Rust tests with Valgrind with
`make check-valgrind` in the rust directory.
---
 rust/Makefile.am |  3 +++
 rust/run-tests.sh.in | 16 ++--
 2 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/rust/Makefile.am b/rust/Makefile.am
index 6609eec..295254d 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -92,6 +92,9 @@ TESTS_ENVIRONMENT = \
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
 
+check-valgrind:
+   LIBNBD_VALGRIND=1 $(MAKE) check
+
 clean-local:
$(CARGO) clean
$(CARGO) clean --manifest-path cargo_test/Cargo.toml
diff --git a/rust/run-tests.sh.in b/rust/run-tests.sh.in
index afa83dc..3ebf9a1 100755
--- a/rust/run-tests.sh.in
+++ b/rust/run-tests.sh.in
@@ -25,9 +25,13 @@ requires @NBDKIT@ --version
 requires @NBDKIT@ floppy --version
 requires @NBDKIT@ memory --version
 
-@CARGO@ test -- --nocapture
-@CARGO@ run --example connect-command
-@NBDKIT@ -U - memory 1M \
- --run '@CARGO@ run --example get-size -- $unixsocket'
-@NBDKIT@ -U - floppy . \
- --run '@CARGO@ run --example fetch-first-sector -- $unixsocket'
+if [ -z "$VG" ]; then
+@CARGO@ test -- --nocapture
+@CARGO@ run --example connect-command
+@NBDKIT@ -U - memory 1M \
+ --run '@CARGO@ run --example get-size -- $unixsocket'
+@NBDKIT@ -U - floppy . \
+ --run '@CARGO@ run --example fetch-first-sector -- $unixsocket'
+else
+@CARGO@ test --config "target.'cfg(all())'.runner = \"$VG\"" -- --nocapture
+fi
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v7 6/9] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

Add a flag (modifies_fd) to the call structure in generator/API.ml
which is set to true if the handle call may do something with the
file descriptor. That is, it is true for all calls which are or may
call aio_notify_*, including all synchronous commands like
nbd_connect or nbd_opt_go.

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to true will be
excluded from that handle. The asynchronous (aio_*) functions will be
used instead.
---
 generator/API.ml  | 32 
 generator/API.mli |  7 +++
 2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 41a6dd1..ada2b06 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
   is_locked : bool;
   may_set_error : bool;
   async_kind : async_kind option;
+  modifies_fd: bool;
   mutable first_version : int * int;
 }
 and arg =
@@ -275,6 +276,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  permitted_states = [];
  is_locked = true; may_set_error = true;
  async_kind = None;
+ modifies_fd = false;
  first_version = (0, 0) }
 
 (* Calls.
@@ -1182,6 +1184,7 @@ Return true if option negotiation mode was enabled on 
this handle.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -1209,6 +1212,7 @@ although older servers will instead have killed the 
connection.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 Request that the server finish negotiation, gracefully if possible, then
@@ -1222,6 +1226,7 @@ enabled option mode.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to initiate TLS";
 longdesc = "\
 Request that the server initiate a secure TLS connection, by
@@ -1260,6 +1265,7 @@ established, as reported by 
L.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to enable structured replies";
 longdesc = "\
 Request that the server use structured replies, by sending
@@ -1286,6 +1292,7 @@ later calls to this function return false.";
 default_call with
 args = [ Closure list_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to list all exports during negotiation";
 longdesc = "\
 Request that the server list all exports that it supports.  This can
@@ -1327,6 +1334,7 @@ description is set with I<-D>.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server for information about an export";
 longdesc = "\
 Request that the server supply information about the export name
@@ -1358,6 +1366,7 @@ corresponding L would succeed.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using implicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1413,6 +1422,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using explicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1463,6 +1473,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using implicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1522,6 +1533,7 @@ no contexts are reported, or may fail but have a 
non-empty list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using explicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1751,6 

[Libguestfs] [libnbd PATCH v7 2/9] generator: Add information about asynchronous handle calls

[Tage Johansson]

A new field (async_kind) is added to the call data type in
generator/API.ml*. The purpose is to tell if a certain handle call is
an asynchronous command and if so how one can know when it is
completed. The motivation for this is that all asynchronous commands
on the AsyncHandle in the Rust bindings makes use of Rust's
[`async fn`s](https://doc.rust-lang.org/std/keyword.async.html).
But to implement such an async fn, the API needs to know when the
command completed, either by a completion callback or via a change
of state.
---
 generator/API.ml  | 32 
 generator/API.mli | 11 +++
 2 files changed, 43 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 72c8165..99fcb82 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -32,6 +32,7 @@ type call = {
   permitted_states : permitted_state list;
   is_locked : bool;
   may_set_error : bool;
+  async_kind : async_kind option;
   mutable first_version : int * int;
 }
 and arg =
@@ -102,6 +103,9 @@ and permitted_state =
 | Negotiating
 | Connected
 | Closed | Dead
+and async_kind =
+| WithCompletionCallback
+| ChangesState of string * bool
 and link =
 | Link of string
 | SectionLink of string
@@ -250,6 +254,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  see_also = [];
  permitted_states = [];
  is_locked = true; may_set_error = true;
+ async_kind = None;
  first_version = (0, 0) }
 
 (* Calls.
@@ -2802,6 +2807,7 @@ wait for an L.";
 default_call with
 args = [ SockAddrAndLen ("addr", "addrlen") ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Begin connecting to the NBD server.  The C and C
@@ -2814,6 +2820,7 @@ parameters specify the address of the socket to connect 
to.
 default_call with
 args = [ String "uri" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to an NBD URI";
 longdesc = "\
 Begin connecting to the NBD URI C.  Parameters behave as
@@ -2827,6 +2834,7 @@ documented in L.
 default_call with
 args = [ Path "unixsocket" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a Unix domain socket";
 longdesc = "\
 Begin connecting to the NBD server over Unix domain socket
@@ -2841,6 +2849,7 @@ L.
 default_call with
 args = [ UInt32 "cid"; UInt32 "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over AF_VSOCK socket";
 longdesc = "\
 Begin connecting to the NBD server over the C
@@ -2854,6 +2863,7 @@ L.
 default_call with
 args = [ String "hostname"; String "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a TCP port";
 longdesc = "\
 Begin connecting to the NBD server listening on C.
@@ -2866,6 +2876,7 @@ Parameters behave as documented in L.
 default_call with
 args = [ Fd "sock" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect directly to a connected socket";
 longdesc = "\
 Begin connecting to the connected socket C.
@@ -2878,6 +2889,7 @@ Parameters behave as documented in 
L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it over
@@ -2891,6 +2903,7 @@ L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect using systemd socket activation";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it using
@@ -2907,6 +2920,7 @@ L.
 optargs = [ OClosure completion_closure ];
 ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some WithCompletionCallback;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -2930,6 +2944,7 @@ when L returns true.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 

Re: [Libguestfs] [libnbd PATCH v6 04/13] rust: Make it possible to run tests with Valgrind

[Tage Johansson]

On 8/4/2023 3:04 PM, Richard W.M. Jones wrote:

On Fri, Aug 04, 2023 at 11:34:07AM +, Tage Johansson wrote:

Make it possible to run Rust tests with Valgrind with
`make check-valgrind` in the rust directory.
---
  rust/Makefile.am | 3 +++
  rust/run-tests.sh.in | 6 +-
  2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/rust/Makefile.am b/rust/Makefile.am
index 1e63724..bb2e6f0 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -89,6 +89,9 @@ TESTS_ENVIRONMENT = \
  LOG_COMPILER = $(top_builddir)/run
  TESTS = run-tests.sh
  
+check-valgrind:

+   LIBNBD_VALGRIND=1 $(MAKE) check
+
  clean-local:
$(CARGO) clean
$(CARGO) clean --manifest-path cargo_test/Cargo.toml
diff --git a/rust/run-tests.sh.in b/rust/run-tests.sh.in
index d45b1bf..f7db344 100755
--- a/rust/run-tests.sh.in
+++ b/rust/run-tests.sh.in
@@ -23,4 +23,8 @@ set -x
  
  requires nbdkit --version
  
-@CARGO@ test -- --nocapture

+if [ -z "$VG" ]; then
+@CARGO@ test -- --nocapture
+else
+@CARGO@ test --config "target.'cfg(all())'.runner = \"$VG\"" -- --nocapture
+fi

So the tests don't actually pass, with unfortunately quite a lot of
errors.  Do they pass for you?



Strange, "make check-valgrind" works without errors for me. Could it be 
that we have different Valgrind installations? Or else I have no idea 
what would be the difference.




You may need to add a valgrind/rust.suppressions file.





Does rust require a runtime mode where it is forced to free up
allocations on exit (as not all languages will bother with that, since
it is not necessary ordinarily).



I am quite sure the answer is no.


Best regards,

Tage



Rich.



___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v6 08/13] rust: Use more specific closure traits

[Tage Johansson]

For closures with cbcount = CBOnce, FnOnce will be used instead of
FnMut. Moreover, closures in synchronous commands with
cblifetime = CBCommand will not need to live for the static lifetime.
See [here](https://doc.rust-lang.org/std/ops/trait.FnOnce.html) for more
information about the advantages of using `FnOnce` when possible.
---
 generator/Rust.ml  | 44 
 rust/src/handle.rs |  2 ++
 rust/src/types.rs  |  2 ++
 3 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index daf0e8a..052f61b 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -114,7 +114,7 @@ let rust_cbarg_name : cbarg -> string = function
   | CBArrayAndLen (arg, _) | CBMutable arg -> rust_arg_name arg
 
 (* Get the Rust type for an argument. *)
-let rec rust_arg_type : arg -> string = function
+let rec rust_arg_type ?(async_kind = None) : arg -> string = function
   | Bool _ -> "bool"
   | Int _ -> "c_int"
   | UInt _ -> "c_uint"
@@ -134,15 +134,18 @@ let rec rust_arg_type : arg -> string = function
   | BytesOut _ -> " [u8]"
   | BytesPersistIn _ -> "&'static [u8]"
   | BytesPersistOut _ -> "&'static mut [u8]"
-  | Closure { cbargs } -> "impl " ^ rust_closure_trait cbargs
+  | Closure { cbargs; cbcount } -> (
+  match async_kind with
+  | Some _ -> "impl " ^ rust_closure_trait cbargs cbcount
+  | None -> "impl " ^ rust_closure_trait cbargs cbcount ~lifetime:None)
 
 (* Get the Rust closure trait for a callback, That is `Fn*(...) -> ...)`. *)
-and rust_closure_trait ?(lifetime = Some "'static") cbargs : string =
+and rust_closure_trait ?(lifetime = Some "'static") cbargs cbcount : string =
   let rust_cbargs = String.concat ", " (List.map rust_cbarg_type cbargs)
-  and lifetime_constraint =
-match lifetime with None -> "" | Some x -> " + " ^ x
-  in
-  "FnMut(" ^ rust_cbargs ^ ") -> c_int + Send + Sync" ^ lifetime_constraint
+  and closure_type =
+match cbcount with CBOnce -> "FnOnce" | CBMany -> "FnMut"
+  and lifetime = match lifetime with None -> "" | Some x -> " + " ^ x in
+  sprintf "%s(%s) -> c_int + Send + Sync%s" closure_type rust_cbargs lifetime
 
 (* Get the Rust type for a callback argument. *)
 and rust_cbarg_type : cbarg -> string = function
@@ -156,8 +159,8 @@ and rust_cbarg_type : cbarg -> string = function
   | CBMutable arg -> " " ^ rust_arg_type arg
 
 (* Get the type of a rust optional argument. *)
-let rust_optarg_type : optarg -> string = function
-  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x))
+let rust_optarg_type ?(async_kind = None) : optarg -> string = function
+  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x) ~async_kind)
   | OFlags (name, flags, _) ->
   sprintf "Option<%s>" (rust_arg_type (Flags (name, flags)))
 
@@ -422,8 +425,8 @@ let ffi_ret_to_rust call =
closure data, and a free function for the closure data. This struct is what
will be sent to a C function taking the closure as an argument. In fact,
the struct itself is generated by rust-bindgen. *)
-let print_rust_closure_to_raw_fn { cbname; cbargs } =
-  let closure_trait = rust_closure_trait cbargs ~lifetime:None in
+let print_rust_closure_to_raw_fn { cbname; cbargs; cbcount } =
+  let closure_trait = rust_closure_trait cbargs cbcount ~lifetime:None in
   let ffi_cbargs_names = List.flatten (List.map ffi_cbarg_names cbargs) in
   let ffi_cbargs_types = List.flatten (List.map ffi_cbarg_types cbargs) in
   let rust_cbargs_names = List.map rust_cbarg_name cbargs in
@@ -438,16 +441,24 @@ let print_rust_closure_to_raw_fn { cbname; cbargs } =
(List.map2 (sprintf "%s: %s") ffi_cbargs_names ffi_cbargs_types));
   pr "  where F: %s\n" closure_trait;
   pr "{\n";
-  pr "let callback_ptr = data as *mut F;\n";
-  pr "let callback =  *callback_ptr;\n";
+  (match cbcount with
+  | CBMany ->
+  pr "let callback_ptr = data as *mut F;\n";
+  pr "let callback =  *callback_ptr;\n"
+  | CBOnce ->
+  pr "let callback_ptr = data as *mut Option;\n";
+  pr "let callback_option:  Option =  *callback_ptr;\n";
+  pr "let callback: F = callback_option.take().unwrap();\n");
   List.iter ffi_cbargs_to_rust cbargs;
   pr "callback(%s)\n" (String.concat ", " rust_cbargs_names);
   pr "}\n";
-  pr "let callback_data = Box::into_raw(Box::new(f));\n";
+  pr "let callback_data = Box::into_raw(Box::new(%s));\n"
+(match cbcount with CBMany -> "f" | CBOnce -> "Some(f)");
   pr "sys::nbd_%s_callback {\n" cbname;
   pr "callback: Some(call_closure::),\n";
   pr "user_data: callback_data as *mut _,\n";
-  pr "free: Some(utils::drop_data::),\n";
+  pr "free: Some(utils::drop_data::<%s>),\n"
+(match cbcount with CBMany -> "F" | CBOnce -> "Option");
   pr "}\n";
   pr "}\n";
   pr "\n"
@@ -504,7 +515,8 @@ let print_rust_handle_method (name, call) =
   let 

[Libguestfs] [libnbd PATCH v6 13/13] rust: async: Add an example

[Tage Johansson]

This patch adds an example using the asynchronous Rust bindings.
---
 rust/Cargo.toml|   1 +
 rust/examples/concurrent-read-write.rs | 135 +
 rust/run-tests.sh.in   |   2 +
 3 files changed, 138 insertions(+)
 create mode 100644 rust/examples/concurrent-read-write.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index d001248..4332783 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -54,5 +54,6 @@ default = ["log", "tokio"]
 anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
 tempfile = "3.6.0"
 tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..a1c3e8a
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,135 @@
+//! Example usage with nbdkit:
+//!
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//!
+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+nbd.connect_unix()?;
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket.clone().into(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket: PathBuf,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// futures must be `'static`, hence we wrap the handle in an [Arc].
+let nbd = Arc::new(libnbd::AsyncHandle::new()?);
+nbd.connect_unix(socket).await?;
+
+let mut rng = SmallRng::seed_from_u64(44 as u64);
+
+// Issue commands.
+let mut stats = Stats::default();
+let mut join_set = JoinSet::new();
+//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+while stats.requests < NR_CYCLES || !join_set.is_empty() {
+while stats.requests < NR_CYCLES && join_set.len() < MAX_IN_FLIGHT {
+// If we want to issue another request, do so.  Note that we reuse
+// the same buffer for multiple in-flight requests.  It doesn't
+// matter here because we're just trying to write random stuff,
+// but that would be Very Bad in a real application.
+// Simulate a mix of large and small requests.
+let size = if 

[Libguestfs] [libnbd PATCH v6 12/13] rust: async: Add a couple of integration tests

[Tage Johansson]

Add a couple of integration tests as rust/tests/test_async_*.rs. They
are very similar to the tests for the synchronous API.
---
 rust/Cargo.toml   |   1 +
 rust/tests/test_async_100_handle.rs   |  25 +++
 rust/tests/test_async_200_connect_command.rs  |  33 
 rust/tests/test_async_210_opt_abort.rs|  32 
 rust/tests/test_async_220_opt_list.rs |  81 ++
 rust/tests/test_async_230_opt_info.rs | 122 +++
 rust/tests/test_async_240_opt_list_meta.rs| 147 ++
 .../test_async_245_opt_list_meta_queries.rs   |  91 +++
 rust/tests/test_async_250_opt_set_meta.rs | 122 +++
 .../test_async_255_opt_set_meta_queries.rs| 107 +
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 +++
 rust/tests/test_async_460_block_status.rs |  92 +++
 rust/tests/test_async_620_stats.rs|  76 +
 15 files changed, 1112 insertions(+)
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index e076826..d001248 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -55,3 +55,4 @@ anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
 tempfile = "3.6.0"
+tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/tests/test_async_100_handle.rs 
b/rust/tests/test_async_100_handle.rs
new file mode 100644
index 000..e50bad9
--- /dev/null
+++ b/rust/tests/test_async_100_handle.rs
@@ -0,0 +1,25 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+//! Just check that we can link with libnbd and create a handle.
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_nbd_handle_new() {
+let _ = libnbd::AsyncHandle::new().unwrap();
+}
diff --git a/rust/tests/test_async_200_connect_command.rs 
b/rust/tests/test_async_200_connect_command.rs
new file mode 100644
index 000..8a3f497
--- /dev/null
+++ b/rust/tests/test_async_200_connect_command.rs
@@ -0,0 +1,33 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+#![deny(warnings)]
+
+
+#[tokio::test]
+async fn test_async_connect_command() {
+let nbd = libnbd::AsyncHandle::new().unwrap();
+nbd.connect_command(&[
+"nbdkit",
+"-s",
+"--exit-with-parent",
+"-v",
+"null",
+])
+.await
+.unwrap();
+}
diff --git a/rust/tests/tes

[Libguestfs] [libnbd PATCH v6 11/13] rust: async: Use the modifies_fd flag to exclude calls

[Tage Johansson]

All handle calls which has the modifies_fd flag set to true will be
excluded from AsyncHandle (the asynchronous handle in the rust
bindings). This is a better approach then listing all calls that should
be excluded in Rust.ml explicetly.
---
 generator/Rust.ml | 60 +--
 1 file changed, 27 insertions(+), 33 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 6ec6376..c08920d 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -571,18 +571,17 @@ let generate_rust_bindings () =
 
 let excluded_handle_calls : NameSet.t =
   NameSet.of_list
-[
-  "aio_get_fd";
-  "aio_get_direction";
-  "aio_notify_read";
-  "aio_notify_write";
-  "clear_debug_callback";
-  "get_debug";
-  "poll";
-  "poll2";
-  "set_debug";
-  "set_debug_callback";
-]
+  @@ [
+   "aio_get_fd";
+   "aio_get_direction";
+   "clear_debug_callback";
+   "get_debug";
+   "set_debug";
+   "set_debug_callback";
+ ]
+  @ (handle_calls
+|> List.filter (fun (_, { modifies_fd }) -> modifies_fd)
+|> List.map (fun (name, _) -> name))
 
 (* A mapping with names as keys. *)
 module NameMap = Map.Make (String)
@@ -593,16 +592,16 @@ let strip_aio name : string =
 String.sub name 4 (String.length name - 4)
   else failwithf "Asynchronous call %s must begin with aio_" name
 
-(* A map with all asynchronous handle calls. The keys are names with "aio_"
-   stripped, the values are a tuple with the actual name (with "aio_"), the
-   [call] and the [async_kind]. *)
+(* A map with all asynchronous handle calls. The keys are names, the values
+   are tuples of: the name with "aio_" stripped, the [call] and the
+   [async_kind]. *)
 let async_handle_calls : ((string * call) * async_kind) NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
   |> List.filter_map (fun (name, call) ->
  call.async_kind
  |> Option.map (fun async_kind ->
-(strip_aio name, ((name, call), async_kind
+(name, ((strip_aio name, call), async_kind
   |> List.to_seq |> NameMap.of_seq
 
 (* A mapping with all synchronous (not asynchronous) handle calls. Excluded
@@ -612,11 +611,7 @@ let async_handle_calls : ((string * call) * async_kind) 
NameMap.t =
 let sync_handle_calls : call NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
-  |> List.filter (fun (name, _) ->
- (not (NameMap.mem name async_handle_calls))
- && not
-  (String.starts_with ~prefix:"aio_" name
-  && NameMap.mem (strip_aio name) async_handle_calls))
+  |> List.filter (fun (n, _) -> not (NameMap.mem n async_handle_calls))
   |> List.to_seq |> NameMap.of_seq
 
 (* Get the Rust type for an argument in the asynchronous API. Like
@@ -661,7 +656,7 @@ let print_rust_sync_handle_call name call =
 (* Print the Rust function for an asynchronous handle call with a completion
callback. (Note that "callback" might be abbreviated with "cb" in the
following code. *)
-let print_rust_async_handle_call_with_completion_cb name (aio_name, call) =
+let print_rust_async_handle_call_with_completion_cb aio_name (name, call) =
   (* An array of all optional arguments. Useful because we need to deel with
  the index of the completion callback. *)
   let optargs = Array.of_list call.optargs in
@@ -741,7 +736,7 @@ let print_rust_async_handle_call_with_completion_cb name 
(aio_name, call) =
completion by changing state. The predicate is a call like
"aio_is_connecting" which should get the value (like false) for the call to
be complete. *)
-let print_rust_async_handle_call_changing_state name (aio_name, call)
+let print_rust_async_handle_call_changing_state aio_name (name, call)
 (predicate, value) =
   let value = if value then "true" else "false" in
   print_rust_handle_call_comment call;
@@ -770,16 +765,15 @@ let print_rust_async_handle_call_changing_state name 
(aio_name, call)
 (* Print an impl with all handle calls. *)
 let print_rust_async_handle_impls () =
   pr "impl AsyncHandle {\n";
-  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
-  NameMap.iter
-(fun name (call, async_kind) ->
-  match async_kind with
-  | WithCompletionCallback ->
-  print_rust_async_handle_call_with_completion_cb name call
-  | ChangesState (predicate, value) ->
-  print_rust_async_handle_call_changing_state name call
-(predicate, value))
-async_handle_calls;
+  sync_handle_calls |> NameMap.iter print_rust_sync_handle_call;
+  async_handle_calls
+  |> NameMap.iter (fun name (call, async_kind) ->
+ match async_kind with
+ | WithCompletionCallback ->
+ print_rust_async_handle_call_with_completion_cb name call
+ | ChangesState (predicate, value) ->
+ print_rust_async_handle_call_changing_state 

[Libguestfs] [libnbd PATCH v6 10/13] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

Add a flag (modifies_fd) to the call structure in generator/API.ml
which is set to true if the handle call may do something with the
file descriptor. That is, it is true for all calls which are or may
call aio_notify_*, including all synchronous commands like
nbd_connect or nbd_opt_go.

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to true will be
excluded from that handle. The asynchronous (aio_*) functions will be
used instead.
---
 generator/API.ml  | 32 
 generator/API.mli |  7 +++
 2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 41a6dd1..ada2b06 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
   is_locked : bool;
   may_set_error : bool;
   async_kind : async_kind option;
+  modifies_fd: bool;
   mutable first_version : int * int;
 }
 and arg =
@@ -275,6 +276,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  permitted_states = [];
  is_locked = true; may_set_error = true;
  async_kind = None;
+ modifies_fd = false;
  first_version = (0, 0) }
 
 (* Calls.
@@ -1182,6 +1184,7 @@ Return true if option negotiation mode was enabled on 
this handle.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -1209,6 +1212,7 @@ although older servers will instead have killed the 
connection.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 Request that the server finish negotiation, gracefully if possible, then
@@ -1222,6 +1226,7 @@ enabled option mode.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to initiate TLS";
 longdesc = "\
 Request that the server initiate a secure TLS connection, by
@@ -1260,6 +1265,7 @@ established, as reported by 
L.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to enable structured replies";
 longdesc = "\
 Request that the server use structured replies, by sending
@@ -1286,6 +1292,7 @@ later calls to this function return false.";
 default_call with
 args = [ Closure list_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to list all exports during negotiation";
 longdesc = "\
 Request that the server list all exports that it supports.  This can
@@ -1327,6 +1334,7 @@ description is set with I<-D>.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server for information about an export";
 longdesc = "\
 Request that the server supply information about the export name
@@ -1358,6 +1366,7 @@ corresponding L would succeed.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using implicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1413,6 +1422,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using explicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1463,6 +1473,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using implicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1522,6 +1533,7 @@ no contexts are reported, or may fail but have a 
non-empty list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using explicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1751,6 

[Libguestfs] [libnbd PATCH v6 03/13] rust: Add a couple of integration tests

[Tage Johansson]

A couple of integration tests are added in rust/tests. They are mostly
ported from the OCaml tests.
---
 rust/Cargo.toml  |   4 +
 rust/Makefile.am |  22 +++
 rust/run-tests.sh.in |   4 +-
 rust/tests/nbdkit_pattern/mod.rs |  28 
 rust/tests/test_100_handle.rs|  25 
 rust/tests/test_110_defaults.rs  |  33 +
 rust/tests/test_120_set_non_defaults.rs  |  53 +++
 rust/tests/test_130_private_data.rs  |  28 
 rust/tests/test_140_explicit_close.rs|  31 
 rust/tests/test_200_connect_command.rs   |  32 
 rust/tests/test_210_opt_abort.rs |  31 
 rust/tests/test_220_opt_list.rs  |  86 +++
 rust/tests/test_230_opt_info.rs  | 120 +++
 rust/tests/test_240_opt_list_meta.rs | 147 +++
 rust/tests/test_245_opt_list_meta_queries.rs |  93 
 rust/tests/test_250_opt_set_meta.rs  | 123 
 rust/tests/test_255_opt_set_meta_queries.rs  | 109 ++
 rust/tests/test_300_get_size.rs  |  35 +
 rust/tests/test_400_pread.rs |  39 +
 rust/tests/test_405_pread_structured.rs  |  79 ++
 rust/tests/test_410_pwrite.rs|  58 
 rust/tests/test_460_block_status.rs  |  92 
 rust/tests/test_620_stats.rs |  75 ++
 rust/tests/test_log/mod.rs   |  86 +++
 24 files changed, 1432 insertions(+), 1 deletion(-)
 create mode 100644 rust/tests/nbdkit_pattern/mod.rs
 create mode 100644 rust/tests/test_100_handle.rs
 create mode 100644 rust/tests/test_110_defaults.rs
 create mode 100644 rust/tests/test_120_set_non_defaults.rs
 create mode 100644 rust/tests/test_130_private_data.rs
 create mode 100644 rust/tests/test_140_explicit_close.rs
 create mode 100644 rust/tests/test_200_connect_command.rs
 create mode 100644 rust/tests/test_210_opt_abort.rs
 create mode 100644 rust/tests/test_220_opt_list.rs
 create mode 100644 rust/tests/test_230_opt_info.rs
 create mode 100644 rust/tests/test_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_300_get_size.rs
 create mode 100644 rust/tests/test_400_pread.rs
 create mode 100644 rust/tests/test_405_pread_structured.rs
 create mode 100644 rust/tests/test_410_pwrite.rs
 create mode 100644 rust/tests/test_460_block_status.rs
 create mode 100644 rust/tests/test_620_stats.rs
 create mode 100644 rust/tests/test_log/mod.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index c745972..b498930 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -47,3 +47,7 @@ libc = "0.2.147"
 
 [features]
 default = ["log"]
+
+[dev-dependencies]
+once_cell = "1.18.0"
+tempfile = "3.6.0"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index b39bd32..1e63724 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -36,6 +36,27 @@ source_files = \
cargo_test/Cargo.toml \
cargo_test/src/lib.rs \
cargo_test/README.md \
+   tests/nbdkit_pattern/mod.rs \
+   tests/test_100_handle.rs \
+   tests/test_110_defaults.rs \
+   tests/test_120_set_non_defaults.rs \
+   tests/test_130_private_data.rs \
+   tests/test_140_explicit_close.rs \
+   tests/test_200_connect_command.rs \
+   tests/test_210_opt_abort.rs \
+   tests/test_220_opt_list.rs \
+   tests/test_230_opt_info.rs \
+   tests/test_240_opt_list_meta.rs \
+   tests/test_245_opt_list_meta_queries.rs \
+   tests/test_250_opt_set_meta.rs \
+   tests/test_255_opt_set_meta_queries.rs \
+   tests/test_300_get_size.rs \
+   tests/test_400_pread.rs \
+   tests/test_405_pread_structured.rs \
+   tests/test_410_pwrite.rs \
+   tests/test_460_block_status.rs \
+   tests/test_620_stats.rs \
+   tests/test_log/mod.rs \
run-tests.sh.in \
$(NULL)
 
@@ -63,6 +84,7 @@ TESTS_ENVIRONMENT = \
LIBNBD_DEBUG=1 \
$(MALLOC_CHECKS) \
abs_top_srcdir=$(abs_top_srcdir) \
+   CARGO=$(CARGO) \
$(NULL)
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
diff --git a/rust/run-tests.sh.in b/rust/run-tests.sh.in
index f4d220c..d45b1bf 100755
--- a/rust/run-tests.sh.in
+++ b/rust/run-tests.sh.in
@@ -1,6 +1,6 @@
 #!/bin/bash -
 # nbd client library in userspace
-# Copyright Red Hat
+# Copyright Tage Johansson
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -21,4 +21,6 @@
 set -e
 set -x
 
+requires nbdkit --version
+
 @CARGO@ test -- --nocapture
diff --git a/rust/tests/nbdkit_pattern/mod.rs b/rust/tests/nbdkit_pattern/mod.rs
new

[Libguestfs] [libnbd PATCH v6 09/13] rust: async: Create an async friendly handle type

[Tage Johansson]

 name
+(rust_async_handle_call_args call);
+  pr "{\n";
+  print_ffi_call aio_name "self.data.handle.handle" call;
+  pr "?;\n";
+  pr "let (ret_tx, ret_rx) = oneshot::channel::>();\n";
+  pr "let mut ret_tx = Some(ret_tx);\n";
+  pr "let completion_predicate = \n";
+  pr " move |handle: , res: <()>| {\n";
+  pr "  let ret = if let Err(_) = res {\n";
+  pr "res.clone()\n";
+  pr "  } else {\n";
+  pr "if handle.%s() != %s { return false; }\n" predicate value;
+  pr "else { Ok(()) }\n";
+  pr "  };\n";
+  pr "  ret_tx.take().unwrap().send(ret).ok();\n";
+  pr "  true\n";
+  pr "};\n";
+  pr "self.add_command(completion_predicate)?;\n";
+  pr "ret_rx.await.unwrap()\n";
+  pr "}\n\n"
+
+(* Print an impl with all handle calls. *)
+let print_rust_async_handle_impls () =
+  pr "impl AsyncHandle {\n";
+  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
+  NameMap.iter
+(fun name (call, async_kind) ->
+  match async_kind with
+  | WithCompletionCallback ->
+  print_rust_async_handle_call_with_completion_cb name call
+  | ChangesState (predicate, value) ->
+  print_rust_async_handle_call_changing_state name call
+(predicate, value))
+async_handle_calls;
+  pr "}\n\n"
+
+let print_rust_async_imports () =
+  pr "use crate::{*, types::*};\n";
+  pr "use os_socketaddr::OsSocketAddr;\n";
+  pr "use std::ffi::*;\n";
+  pr "use std::mem;\n";
+  pr "use std::net::SocketAddr;\n";
+  pr "use std::os::fd::{AsRawFd, OwnedFd};\n";
+  pr "use std::os::unix::prelude::*;\n";
+  pr "use std::path::PathBuf;\n";
+  pr "use std::ptr;\n";
+  pr "use std::sync::Arc;\n";
+  pr "use tokio::sync::oneshot;\n";
+  pr "\n"
+
+let generate_rust_async_bindings () =
+  generate_header CStyle ~copyright:"Tage Johansson";
+  pr "\n";
+  print_rust_async_imports ();
+  print_rust_async_handle_impls ()
diff --git a/generator/Rust.mli b/generator/Rust.mli
index 450e4ca..0960170 100644
--- a/generator/Rust.mli
+++ b/generator/Rust.mli
@@ -18,3 +18,5 @@
 
 (* Print all flag-structs, enums, constants and handle calls in Rust code. *)
 val generate_rust_bindings : unit -> unit
+
+val generate_rust_async_bindings : unit -> unit
diff --git a/generator/generator.ml b/generator/generator.ml
index c62b0c4..237e5c5 100644
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -64,3 +64,4 @@ let () =
 
   output_to ~formatter:(Some Rustfmt) "rust/libnbd-sys/src/generated.rs" 
RustSys.generate_rust_sys_bindings;
   output_to ~formatter:(Some Rustfmt) "rust/src/bindings.rs" 
Rust.generate_rust_bindings;
+  output_to ~formatter:(Some Rustfmt) "rust/src/async_bindings.rs" 
Rust.generate_rust_async_bindings;
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 04e371e..e076826 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -44,9 +44,11 @@ os_socketaddr = "0.2.4"
 thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
+tokio = { optional = true, version = "1.29.1", default-features = false, 
features = ["rt", "sync", "net"] }
+epoll = "4.3.3"
 
 [features]
-default = ["log"]
+default = ["log", "tokio"]
 
 [dev-dependencies]
 anyhow = "1.0.72"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index 295254d..9fbe9fe 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -19,6 +19,7 @@ include $(top_srcdir)/subdir-rules.mk
 
 generator_built = \
libnbd-sys/src/generated.rs \
+   src/async_bindings.rs \
src/bindings.rs \
$(NULL)
 
@@ -30,6 +31,7 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   src/async_handle.rs \
examples/connect-command.rs \
examples/get-size.rs \
examples/fetch-first-sector.rs \
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
new file mode 100644
index 000..4223b80
--- /dev/null
+++ b/rust/src/async_handle.rs
@@ -0,0 +1,268 @@
+// nbd client library in userspace
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GN

[Libguestfs] [libnbd PATCH v6 06/13] generator: Add information about asynchronous handle calls

[Tage Johansson]

A new field (async_kind) is added to the call data type in
generator/API.ml*. The purpose is to tell if a certain handle call is
an asynchronous command and if so how one can know when it is
completed. The motivation for this is that all asynchronous commands
on the AsyncHandle in the Rust bindings makes use of Rust's
[`async fn`s](https://doc.rust-lang.org/std/keyword.async.html).
But to implement such an async fn, the API needs to know when the
command completed, either by a completion callback or via a change
of state.
---
 generator/API.ml  | 32 
 generator/API.mli | 11 +++
 2 files changed, 43 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 72c8165..99fcb82 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -32,6 +32,7 @@ type call = {
   permitted_states : permitted_state list;
   is_locked : bool;
   may_set_error : bool;
+  async_kind : async_kind option;
   mutable first_version : int * int;
 }
 and arg =
@@ -102,6 +103,9 @@ and permitted_state =
 | Negotiating
 | Connected
 | Closed | Dead
+and async_kind =
+| WithCompletionCallback
+| ChangesState of string * bool
 and link =
 | Link of string
 | SectionLink of string
@@ -250,6 +254,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  see_also = [];
  permitted_states = [];
  is_locked = true; may_set_error = true;
+ async_kind = None;
  first_version = (0, 0) }
 
 (* Calls.
@@ -2802,6 +2807,7 @@ wait for an L.";
 default_call with
 args = [ SockAddrAndLen ("addr", "addrlen") ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Begin connecting to the NBD server.  The C and C
@@ -2814,6 +2820,7 @@ parameters specify the address of the socket to connect 
to.
 default_call with
 args = [ String "uri" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to an NBD URI";
 longdesc = "\
 Begin connecting to the NBD URI C.  Parameters behave as
@@ -2827,6 +2834,7 @@ documented in L.
 default_call with
 args = [ Path "unixsocket" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a Unix domain socket";
 longdesc = "\
 Begin connecting to the NBD server over Unix domain socket
@@ -2841,6 +2849,7 @@ L.
 default_call with
 args = [ UInt32 "cid"; UInt32 "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over AF_VSOCK socket";
 longdesc = "\
 Begin connecting to the NBD server over the C
@@ -2854,6 +2863,7 @@ L.
 default_call with
 args = [ String "hostname"; String "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a TCP port";
 longdesc = "\
 Begin connecting to the NBD server listening on C.
@@ -2866,6 +2876,7 @@ Parameters behave as documented in L.
 default_call with
 args = [ Fd "sock" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect directly to a connected socket";
 longdesc = "\
 Begin connecting to the connected socket C.
@@ -2878,6 +2889,7 @@ Parameters behave as documented in 
L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it over
@@ -2891,6 +2903,7 @@ L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect using systemd socket activation";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it using
@@ -2907,6 +2920,7 @@ L.
 optargs = [ OClosure completion_closure ];
 ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some WithCompletionCallback;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -2930,6 +2944,7 @@ when L returns true.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 

[Libguestfs] [libnbd PATCH v6 05/13] rust: Add some examples

[Tage Johansson]

This patch adds a few examples in rust/examples/. The examples are
compiled and run as part of the test suite.
---
 rust/Cargo.toml |  2 ++
 rust/Makefile.am|  3 +++
 rust/examples/connect-command.rs| 39 +
 rust/examples/fetch-first-sector.rs | 38 
 rust/examples/get-size.rs   | 29 +
 rust/run-tests.sh.in|  7 ++
 scripts/git.orderfile   |  1 +
 7 files changed, 119 insertions(+)
 create mode 100644 rust/examples/connect-command.rs
 create mode 100644 rust/examples/fetch-first-sector.rs
 create mode 100644 rust/examples/get-size.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index b498930..04e371e 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -49,5 +49,7 @@ libc = "0.2.147"
 default = ["log"]
 
 [dev-dependencies]
+anyhow = "1.0.72"
 once_cell = "1.18.0"
+pretty-hex = "0.3.0"
 tempfile = "3.6.0"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index bb2e6f0..295254d 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -30,6 +30,9 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   examples/connect-command.rs \
+   examples/get-size.rs \
+   examples/fetch-first-sector.rs \
libnbd-sys/Cargo.toml \
libnbd-sys/build.rs \
libnbd-sys/src/lib.rs \
diff --git a/rust/examples/connect-command.rs b/rust/examples/connect-command.rs
new file mode 100644
index 000..db4adbe
--- /dev/null
+++ b/rust/examples/connect-command.rs
@@ -0,0 +1,39 @@
+//! This example shows how to run an NBD server
+//! (nbdkit) as a subprocess of libnbd.
+
+
+fn main() -> libnbd::Result<()> {
+// Create the libnbd handle.
+let handle = libnbd::Handle::new()?;
+
+// Run nbdkit as a subprocess.
+let args = [
+"nbdkit",
+// You must use ‘-s’ (which tells nbdkit to serve
+// a single connection on stdin/stdout).
+"-s",
+// It is recommended to use ‘--exit-with-parent’
+// to ensure nbdkit is always cleaned up even
+// if the main program crashes.
+"--exit-with-parent",
+// Use this to enable nbdkit debugging.
+"-v",
+// The nbdkit plugin name - this is a RAM disk.
+"memory",
+"size=1M",
+];
+handle.connect_command()?;
+
+// Write some random data to the first sector.
+let wbuf: Vec = (0..512).into_iter().map(|i| (i % 13) as u8).collect();
+handle.pwrite(, 0, None)?;
+
+// Read the first sector back.
+let mut rbuf = [0; 512];
+handle.pread( rbuf, 0, None)?;
+
+// What was read must be exactly the same as what was written.
+assert_eq!(wbuf.as_slice(), rbuf.as_slice());
+
+Ok(())
+}
diff --git a/rust/examples/fetch-first-sector.rs 
b/rust/examples/fetch-first-sector.rs
new file mode 100644
index 000..9efb47a
--- /dev/null
+++ b/rust/examples/fetch-first-sector.rs
@@ -0,0 +1,38 @@
+//! This example shows how to connect to an NBD server
+//! and fetch and print the first sector (usually the
+//! boot sector or partition table or filesystem
+//! superblock).
+//!
+//! You can test it with nbdkit like this:
+//!
+//! nbdkit -U - floppy . \
+//!   --run 'cargo run --example fetch-first-sector -- $unixsocket'
+//!
+//! The nbdkit floppy plugin creates an MBR disk so the
+//! first sector is the partition table.
+
+use pretty_hex::pretty_hex;
+use std::env;
+
+fn main() -> anyhow::Result<()> {
+let nbd = libnbd::Handle::new()?;
+
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// Connect to the NBD server over a
+// Unix domain socket.
+nbd.connect_unix(socket)?;
+
+// Read the first sector synchronously.
+let mut buf = [0; 512];
+nbd.pread( buf, 0, None)?;
+
+// Print the sector in hexdump like format.
+print!("{}", pretty_hex());
+
+Ok(())
+}
diff --git a/rust/examples/get-size.rs b/rust/examples/get-size.rs
new file mode 100644
index 000..7f31df5
--- /dev/null
+++ b/rust/examples/get-size.rs
@@ -0,0 +1,29 @@
+//! This example shows how to connect to an NBD
+//! server and read the size of the disk.
+//!
+//! You can test it with nbdkit like this:
+//!
+//! nbdkit -U - memory 1M \
+//!   --run 'cargo run --example get-size -- $unixsocket'
+
+use std::env;
+
+fn main() -> anyhow::Result<()> {
+let nbd = libnbd::Handle::new()?;
+
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// Connect to the NBD server over a
+// Unix domain socket.
+nbd.connect_unix(socket)?;
+
+// Read the size in bytes and print it.
+let size = nbd.get_size()?;
+println!("{:?}: size = {size} bytes", socket);
+
+Ok(())
+}
diff --git a/rust/run-tests.sh.in 

[Libguestfs] [libnbd PATCH v6 02/13] rust: create basic Rust bindings

[Tage Johansson]

generator/Makefile
  golang/Makefile
  golang/examples/Makefile
+ rust/Makefile
  include/Makefile
  info/Makefile
  interop/Makefile
@@ -717,6 +746,7 @@ echo
 echo "Language bindings:"
 echo
 feature "Go"test "x$HAVE_GOLANG_TRUE" = "x"
+feature "Rust"  test "x$HAVE_RUST_TRUE" = "x"
 feature "OCaml" test "x$HAVE_OCAML_TRUE" = "x"
 feature "Python"test "x$HAVE_PYTHON_TRUE" = "x"
 
diff --git a/generator/Makefile.am b/generator/Makefile.am
index c3d53b2..5e148be 100644
--- a/generator/Makefile.am
+++ b/generator/Makefile.am
@@ -60,6 +60,10 @@ sources = \
OCaml.ml \
GoLang.mli \
GoLang.ml \
+   RustSys.mli \
+   RustSys.ml \
+   Rust.mli \
+   Rust.ml \
generator.ml \
$(NULL)
 
diff --git a/generator/Rust.ml b/generator/Rust.ml
new file mode 100644
index 000..daf0e8a
--- /dev/null
+++ b/generator/Rust.ml
@@ -0,0 +1,551 @@
+(* hey emacs, this is OCaml code: -*- tuareg -*- *)
+(* nbd client library in userspace: generator
+ * Copyright Tage Johansson
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ *)
+
+(* Rust language bindings. *)
+
+open Printf
+open API
+open Utils
+
+(* The type for a set of names. *)
+module NameSet = Set.Make (String)
+
+(* List of handle calls which should not be part of the public API. This could
+   for instance be `set_debug` and `set_debug_callback` which are handled
+   separately by the log crate *)
+let hidden_handle_calls : NameSet.t =
+  NameSet.of_list
+[ "get_debug"; "set_debug"; "set_debug_callback"; "clear_debug_callback" ]
+
+let print_rust_constant (name, value) =
+  pr "pub const %s: u32 = %d;\n" name value
+
+let print_rust_enum enum =
+  pr "#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]\n";
+  pr "#[repr(isize)]";
+  pr "pub enum %s {\n" (camel_case enum.enum_prefix);
+  List.iter
+(fun (name, num) -> pr "%s = %d,\n" (camel_case name) num)
+enum.enums;
+  pr "}\n\n"
+
+(* Print a Rust struct for a set of flags. *)
+let print_rust_flags { flag_prefix; flags } =
+  pr "bitflags! {\n";
+  pr "#[repr(C)]\n";
+  pr "#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]\n";
+  pr "pub struct %s: u32 {\n" (camel_case flag_prefix);
+  List.iter
+(fun (name, value) -> pr "const %s = %d;\n" name value)
+flags;
+  pr "}\n";
+  pr "}\n\n"
+
+(* Convert a string to upper snake case. *)
+let to_upper_snake_case s =
+  s |> String.uppercase_ascii |> String.to_seq
+  |> Seq.filter_map (fun ch ->
+ match ch with '-' -> Some '_' | ':' -> None | ch -> Some ch)
+  |> String.of_seq
+
+(* Print metadata namespaces. *)
+let print_metadata_namespace (ns, ctxts) =
+  pr "pub const NAMESPACE_%s: &[u8] = b\"%s:\";\n" (to_upper_snake_case ns) ns;
+  ctxts
+  |> List.iter (fun (ctxt, consts) ->
+ let s = ns ^ ":" ^ ctxt in
+ pr "pub const CONTEXT_%s_%s: &[u8] = b\"%s\";\n"
+   (to_upper_snake_case ns)
+   (to_upper_snake_case ctxt)
+   s;
+ consts
+ |> List.iter (fun (n, i) ->
+pr "pub const %s: u32 = %d;\n" (to_upper_snake_case n) i))
+
+(* Get the name of a rust argument. *)
+let rust_arg_name : arg -> string = function
+  | Bool n
+  | Int n
+  | UInt n
+  | UIntPtr n
+  | UInt32 n
+  | Int64 n
+  | UInt64 n
+  | SizeT n
+  | String n
+  | StringList n
+  | Path n
+  | Fd n
+  | Enum (n, _)
+  | Flags (n, _)
+  | SockAddrAndLen (n, _)
+  | BytesIn (n, _)
+  | BytesPersistIn (n, _)
+  | BytesOut (n, _)
+  | BytesPersistOut (n, _)
+  | Closure { cbname = n } ->
+  n
+
+(* Get the name of a rust optional argument. *)
+let rust_optarg_name : optarg -> string = function
+  | OClosure { cbname = n } | OFlags (n, _, _) -> n
+
+(* Get the

[Libguestfs] [libnbd PATCH v6 00/13] Rust Bindings for Libnbd

[Tage Johansson]

This is the 6th version of the Rust bindings. It solves a problem with
make clean and includes one more patch which I for some reason thought
was already upstream.

Best regards,
Tage


Tage Johansson (13):
  generator: Add an optional `formatter` argument to the [output_to]
function in generator/utils.mli. This defaults to [None] and the
only code formatter supported so far is Rustfmt.
  rust: create basic Rust bindings
  rust: Add a couple of integration tests
  rust: Make it possible to run tests with Valgrind
  rust: Add some examples
  generator: Add information about asynchronous handle calls
  generator: Add information about the lifetime of closures
  rust: Use more specific closure traits
  rust: async: Create an async friendly handle type
  generator: Add `modifies_fd` flag to the [call] structure
  rust: async: Use the modifies_fd flag to exclude calls
  rust: async: Add a couple of integration tests
  rust: async: Add an example

 .gitignore|  10 +
 .ocamlformat  |   4 +
 Makefile.am   |   2 +
 configure.ac  |  30 +
 generator/API.ml  |  84 ++
 generator/API.mli |  35 +
 generator/Makefile.am |   4 +
 generator/Rust.ml | 797 ++
 generator/Rust.mli|  22 +
 generator/RustSys.ml  | 167 
 generator/RustSys.mli |  19 +
 generator/generator.ml|   4 +
 generator/utils.ml|  13 +-
 generator/utils.mli   |   8 +-
 rust/Cargo.toml   |  59 ++
 rust/Makefile.am  | 106 +++
 rust/cargo_test/Cargo.toml|  23 +
 rust/cargo_test/README.md |   3 +
 rust/cargo_test/src/lib.rs|  31 +
 rust/examples/concurrent-read-write.rs| 135 +++
 rust/examples/connect-command.rs  |  39 +
 rust/examples/fetch-first-sector.rs   |  38 +
 rust/examples/get-size.rs |  29 +
 rust/libnbd-sys/Cargo.toml|  32 +
 rust/libnbd-sys/build.rs  |  26 +
 rust/libnbd-sys/src/lib.rs|  19 +
 rust/run-tests.sh.in  |  39 +
 rust/src/async_handle.rs  | 268 ++
 rust/src/error.rs | 157 
 rust/src/handle.rs|  67 ++
 rust/src/lib.rs   |  36 +
 rust/src/types.rs |  20 +
 rust/src/utils.rs |  23 +
 rust/tests/nbdkit_pattern/mod.rs  |  28 +
 rust/tests/test_100_handle.rs |  25 +
 rust/tests/test_110_defaults.rs   |  33 +
 rust/tests/test_120_set_non_defaults.rs   |  53 ++
 rust/tests/test_130_private_data.rs   |  28 +
 rust/tests/test_140_explicit_close.rs |  31 +
 rust/tests/test_200_connect_command.rs|  32 +
 rust/tests/test_210_opt_abort.rs  |  31 +
 rust/tests/test_220_opt_list.rs   |  86 ++
 rust/tests/test_230_opt_info.rs   | 120 +++
 rust/tests/test_240_opt_list_meta.rs  | 147 
 rust/tests/test_245_opt_list_meta_queries.rs  |  93 ++
 rust/tests/test_250_opt_set_meta.rs   | 123 +++
 rust/tests/test_255_opt_set_meta_queries.rs   | 109 +++
 rust/tests/test_300_get_size.rs   |  35 +
 rust/tests/test_400_pread.rs  |  39 +
 rust/tests/test_405_pread_structured.rs   |  79 ++
 rust/tests/test_410_pwrite.rs |  58 ++
 rust/tests/test_460_block_status.rs   |  92 ++
 rust/tests/test_620_stats.rs  |  75 ++
 rust/tests/test_async_100_handle.rs   |  25 +
 rust/tests/test_async_200_connect_command.rs  |  33 +
 rust/tests/test_async_210_opt_abort.rs|  32 +
 rust/tests/test_async_220_opt_list.rs |  81 ++
 rust/tests/test_async_230_opt_info.rs | 122 +++
 rust/tests/test_async_240_opt_list_meta.rs| 147 
 .../test_async_245_opt_list_meta_queries.rs   |  91 ++
 rust/tests/test_async_250_opt_set_meta.rs | 122 +++
 .../test_async_255_opt_set_meta_queries.rs| 107 +++
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 ++
 rust/tests/test_async_460_block_status.rs |  92 ++
 rust/tests/test_async_620_stats.rs|  76 ++
 rust/tests/test_log/mod.rs|  86 ++
 rustfmt.toml  |  19 +
 scripts/git.orderfile |  12 +
 70 files changed, 4891 insertions(+), 3 deletions(-)
 create mode 100644 .ocamlformat
 create mode 100644

[Libguestfs] [libnbd PATCH v6 04/13] rust: Make it possible to run tests with Valgrind

[Tage Johansson]

Make it possible to run Rust tests with Valgrind with
`make check-valgrind` in the rust directory.
---
 rust/Makefile.am | 3 +++
 rust/run-tests.sh.in | 6 +-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/rust/Makefile.am b/rust/Makefile.am
index 1e63724..bb2e6f0 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -89,6 +89,9 @@ TESTS_ENVIRONMENT = \
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
 
+check-valgrind:
+   LIBNBD_VALGRIND=1 $(MAKE) check
+
 clean-local:
$(CARGO) clean
$(CARGO) clean --manifest-path cargo_test/Cargo.toml
diff --git a/rust/run-tests.sh.in b/rust/run-tests.sh.in
index d45b1bf..f7db344 100755
--- a/rust/run-tests.sh.in
+++ b/rust/run-tests.sh.in
@@ -23,4 +23,8 @@ set -x
 
 requires nbdkit --version
 
-@CARGO@ test -- --nocapture
+if [ -z "$VG" ]; then
+@CARGO@ test -- --nocapture
+else
+@CARGO@ test --config "target.'cfg(all())'.runner = \"$VG\"" -- --nocapture
+fi
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v6 07/13] generator: Add information about the lifetime of closures

[Tage Johansson]

Add two new fields, cblifetime and cbcount, to the `closure` type
in generator/API.ml*. cblifetime tells if the closure may only be used
for as long as the command is in flight or if the closure may be used
until the handle is destructed. cbcount tells whether the closure may
be called many times or just once.

This information is needed in the Rust bindings for:
a) Knowing if the closure trait should be FnMut or FnOnce
   (see ).
b) Knowing for what lifetime the closure should be valid. A closure that
   may be called after the function invokation has returned must live
   for the `'static` lietime. But static closures are inconveniant for
   the user since they can't effectively borrow any local data. So it is
   good if this restriction is relaxed when it is not needed.
---
 generator/API.ml  | 20 
 generator/API.mli | 17 +
 2 files changed, 37 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 99fcb82..41a6dd1 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -77,6 +77,8 @@ and ret =
 and closure = {
   cbname : string;
   cbargs : cbarg list;
+  cblifetime : cblifetime;
+  cbcount : cbcount
 }
 and cbarg =
 | CBArrayAndLen of arg * string
@@ -87,6 +89,12 @@ and cbarg =
 | CBString of string
 | CBUInt of string
 | CBUInt64 of string
+and cblifetime =
+| CBCommand
+| CBHandle
+and cbcount =
+| CBOnce
+| CBMany
 and enum = {
   enum_prefix : string;
   enums : (string * int) list
@@ -141,20 +149,28 @@ the handle from the NBD protocol handshake."
 (* Closures. *)
 let chunk_closure = {
   cbname = "chunk";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBBytesIn ("subbuf", "count");
  CBUInt64 "offset"; CBUInt "status";
  CBMutable (Int "error") ]
 }
 let completion_closure = {
   cbname = "completion";
+  cblifetime = CBCommand;
+  cbcount = CBOnce;
   cbargs = [ CBMutable (Int "error") ]
 }
 let debug_closure = {
   cbname = "debug";
+  cblifetime = CBHandle;
+  cbcount = CBMany;
   cbargs = [ CBString "context"; CBString "msg" ]
 }
 let extent_closure = {
   cbname = "extent";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "metacontext";
  CBUInt64 "offset";
  CBArrayAndLen (UInt32 "entries",
@@ -163,10 +179,14 @@ let extent_closure = {
 }
 let list_closure = {
   cbname = "list";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name"; CBString "description" ]
 }
 let context_closure = {
   cbname = "context";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name" ]
 }
 let all_closures = [ chunk_closure; completion_closure;
diff --git a/generator/API.mli b/generator/API.mli
index 361132d..ff85849 100644
--- a/generator/API.mli
+++ b/generator/API.mli
@@ -94,6 +94,12 @@ and ret =
 and closure = {
   cbname : string; (** name of callback function *)
   cbargs : cbarg list; (** all closures return int for now *)
+  (** An upper bound of the lifetime of the closure. Either it will be used for
+  as long as the command is in flight or it may be used until the handle
+  is destructed. *)
+  cblifetime : cblifetime;
+  (** Whether the callback may only be called once or many times. *)
+  cbcount : cbcount;
 }
 and cbarg =
 | CBArrayAndLen of arg * string (** array + number of entries *)
@@ -104,6 +110,17 @@ and cbarg =
 | CBString of string   (** like String *)
 | CBUInt of string (** like UInt *)
 | CBUInt64 of string   (** like UInt64 *)
+and cblifetime =
+| CBCommand (** The closure may only be used until the command is retired.
+(E.G., completion callback or list callback.) *)
+| CBHandle  (** The closure might be used until the handle is descructed.
+(E.G., debug callback.) *)
+and cbcount =
+| CBOnce (** The closure will be used 0 or 1 time if the aio_* call returned an
+ error and exactly once if the call succeeded.
+ (E.g., completion callback.) *)
+| CBMany (** The closure may be used any number of times.
+ (E.g., list callback.) *)
 and enum = {
   enum_prefix : string;(** prefix of each enum variant *)
   enums : (string * int) list (** enum names and their values in C *)
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v6 01/13] generator: Add an optional `formatter` argument to the [output_to] function in generator/utils.mli. This defaults to [None] and the only code formatter supported s

[Tage Johansson]

---
 generator/utils.ml  | 13 +++--
 generator/utils.mli |  8 +++-
 2 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/generator/utils.ml b/generator/utils.ml
index 61cce87..201fb54 100644
--- a/generator/utils.ml
+++ b/generator/utils.ml
@@ -419,7 +419,10 @@ let files_equal n1 n2 =
   | 1 -> false
   | i -> failwithf "%s: failed with error code %d" cmd i
 
-let output_to filename k =
+type formatter =
+  | Rustfmt
+
+let output_to ?(formatter = None) filename k =
   lineno := 1; col := 0;
   let filename_new = filename ^ ".new" in
   let c = open_out filename_new in
@@ -427,7 +430,13 @@ let output_to filename k =
   k ();
   close_out c;
   chan := NoOutput;
-
+  (match formatter with
+  | Some Rustfmt ->
+(match system (sprintf "rustfmt %s" filename_new) with
+  | WEXITED 0 -> ()
+  | WEXITED i -> failwith (sprintf "Rustfmt failed with exit code %d" i)
+  | _ -> failwith "Rustfmt was killed or stopped by a signal.");
+  | None -> ());
   (* Is the new file different from the current file? *)
   if Sys.file_exists filename && files_equal filename filename_new then
 unlink filename_new (* same, so skip it *)
diff --git a/generator/utils.mli b/generator/utils.mli
index c4d47a3..d97d43a 100644
--- a/generator/utils.mli
+++ b/generator/utils.mli
@@ -52,7 +52,13 @@ val files_equal : string -> string -> bool
 val generate_header :
   ?extra_sources:string list -> ?copyright:string -> comment_style -> unit
 
-val output_to : string -> (unit -> 'a) -> unit
+(** Type of code formatter. *)
+type formatter =
+  | Rustfmt
+
+(** Redirect stdout to a file. Possibly formatting the code. *)
+val output_to : ?formatter:formatter option -> string -> (unit -> 'a) -> unit
+
 val pr : ('a, unit, string, unit) format4 -> 'a
 val pr_wrap : ?maxcol:int -> char -> (unit -> 'a) -> unit
 val pr_wrap_cstr : ?maxcol:int -> (unit -> 'a) -> unit
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v5 01/12] rust: create basic Rust bindings

[Tage Johansson]

On 8/4/2023 10:54 AM, Richard W.M. Jones wrote:

I applied the first 4 patches to my local repo.
'make clean' now fails at:

   error: failed to load manifest for dependency `libnbd-sys`
   Caused by:
 failed to parse manifest at 
`/home/rjones/d/libnbd/rust/libnbd-sys/Cargo.toml`
   Caused by:
 no targets specified in the manifest
 either src/lib.rs, src/main.rs, a [lib] section, or [[bin]] section must 
be present
   make[1]: *** [Makefile:1057: clean-local] Error 101
   make[1]: Leaving directory '/home/rjones/d/libnbd/rust'

I'm not sure what that means.  rust/libnbd-sys/Cargo.toml exists.



It is because rust/libnbd-sys/src/lib.rs is generated and hence does not 
exist before running make the first time. I have solved that in v6.




./configure works which is good:

   checking for cargo... cargo
   checking for rustfmt... rustfmt
   checking if cargo is usable... yes
   ...
 Rust ... yes

However the generator fails to compile with:

   make[3]: Entering directory '/home/rjones/d/libnbd/generator'
   ocamlc.opt -g -annot -safe-string -warn-error 
+C+D+E+F+L+M+P+S+U+V+Y+Z+X+52-3 -I . -I . \
 -I +str str.cma -I +unix unix.cma utils.mli utils.ml state_machine.mli 
state_machine.ml API.mli API.ml state_machine_generator.mli 
state_machine_generator.ml C.mli C.ml Python.mli Python.ml OCaml.mli OCaml.ml 
GoLang.mli GoLang.ml RustSys.mli RustSys.ml Rust.mli Rust.ml generator.ml  -o 
generator
File "generator.ml", line 65, characters 2-11:
   65 |   output_to ~formatter:(Some Rustfmt) "rust/libnbd-sys/src/lib.rs" 
RustSys.generate_rust_sys_bindings;
  ^
   Error: This function has type string -> (unit -> 'weak1) -> unit
  It is applied to too many arguments; maybe you forgot a `;'.

This confused me for a while, but I think it's because this series
depends on another series or needs to be rebased?

I removed the ~formatter parameter to get it to compile.



Hmmm, I think there is a patch I thought was upstream, which doesn't 
seem to be so. I'll include that patch in v6.




The next problem is:

   /home/rjones/d/libnbd/run cargo build
  Compiling libnbd v0.1.0 (/home/rjones/d/libnbd/rust)
   error: expected one of `:`, `;`, or `=`, found `-`
  --> src/bindings.rs:116:29
   |
   116 | pub const CONTEXT_QEMU_DIRTY-BITMAP:: &[u8] = b"qemu:dirty-bitmap:";
   | ^ expected one of `:`, `;`, or `=`



I've fixed that as well. I was not testing on the latest changes 
upstream so I didn't encounter that error before.




   error[E0599]: no method named `set_debug_callback` found for struct `Handle` 
in the current scope
 --> src/handle.rs:38:21
  |
   23 | pub struct Handle {
  | - method `set_debug_callback` not found for this struct
   ...
   38 | nbd.set_debug_callback(|func_name, msg| {
  | ^^ method not found in `Handle`
   [and some more]

It might be an idea to run this command to check that the series is
bisectable:

   git rebase -i HEAD~12 -x 'make clean && make && make check'

It will generate a series of rebase commands which look like:

   pick 7475e8560a rust: create basic Rust bindings
   exec make clean && make && make check
   pick c48c7eee0f rust: Add a couple of integration tests
   exec make clean && make && make check
   pick 16debe7848 rust: Make it possible to run tests with Valgrind
   exec make clean && make && make check
   [etc]

which will do a full build cycle after every patch to make sure they
all work incrementally.



I will try that.


Best regards,

Tage



Rich.



___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v5 12/12] rust: async: Add an example

[Tage Johansson]

This patch adds an example using the asynchronous Rust bindings.
---
 rust/Cargo.toml|   1 +
 rust/examples/concurrent-read-write.rs | 135 +
 rust/run-tests.sh.in   |   2 +
 3 files changed, 138 insertions(+)
 create mode 100644 rust/examples/concurrent-read-write.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index d001248..4332783 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -54,5 +54,6 @@ default = ["log", "tokio"]
 anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
 tempfile = "3.6.0"
 tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..a1c3e8a
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,135 @@
+//! Example usage with nbdkit:
+//!
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//!
+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+nbd.connect_unix()?;
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket.clone().into(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket: PathBuf,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// futures must be `'static`, hence we wrap the handle in an [Arc].
+let nbd = Arc::new(libnbd::AsyncHandle::new()?);
+nbd.connect_unix(socket).await?;
+
+let mut rng = SmallRng::seed_from_u64(44 as u64);
+
+// Issue commands.
+let mut stats = Stats::default();
+let mut join_set = JoinSet::new();
+//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+while stats.requests < NR_CYCLES || !join_set.is_empty() {
+while stats.requests < NR_CYCLES && join_set.len() < MAX_IN_FLIGHT {
+// If we want to issue another request, do so.  Note that we reuse
+// the same buffer for multiple in-flight requests.  It doesn't
+// matter here because we're just trying to write random stuff,
+// but that would be Very Bad in a real application.
+// Simulate a mix of large and small requests.
+let size = if 

[Libguestfs] [libnbd PATCH v5 09/12] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

Add a flag (modifies_fd) to the call structure in generator/API.ml
which is set to true if the handle call may do something with the
file descriptor. That is, it is true for all calls which are or may
call aio_notify_*, including all synchronous commands like
nbd_connect or nbd_opt_go.

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to true will be
excluded from that handle. The asynchronous (aio_*) functions will be
used instead.
---
 generator/API.ml  | 32 
 generator/API.mli |  7 +++
 2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 42b9eec..6e1670d 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
   is_locked : bool;
   may_set_error : bool;
   async_kind : async_kind option;
+  modifies_fd: bool;
   mutable first_version : int * int;
 }
 and arg =
@@ -274,6 +275,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  permitted_states = [];
  is_locked = true; may_set_error = true;
  async_kind = None;
+ modifies_fd = false;
  first_version = (0, 0) }
 
 (* Calls.
@@ -1181,6 +1183,7 @@ Return true if option negotiation mode was enabled on 
this handle.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -1208,6 +1211,7 @@ although older servers will instead have killed the 
connection.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 Request that the server finish negotiation, gracefully if possible, then
@@ -1221,6 +1225,7 @@ enabled option mode.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to initiate TLS";
 longdesc = "\
 Request that the server initiate a secure TLS connection, by
@@ -1259,6 +1264,7 @@ established, as reported by 
L.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to enable structured replies";
 longdesc = "\
 Request that the server use structured replies, by sending
@@ -1285,6 +1291,7 @@ later calls to this function return false.";
 default_call with
 args = [ Closure list_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to list all exports during negotiation";
 longdesc = "\
 Request that the server list all exports that it supports.  This can
@@ -1326,6 +1333,7 @@ description is set with I<-D>.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server for information about an export";
 longdesc = "\
 Request that the server supply information about the export name
@@ -1357,6 +1365,7 @@ corresponding L would succeed.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using implicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1412,6 +1421,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using explicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1462,6 +1472,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using implicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1521,6 +1532,7 @@ no contexts are reported, or may fail but have a 
non-empty list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using explicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1750,6 

[Libguestfs] [libnbd PATCH v5 03/12] rust: Make it possible to run tests with Valgrind

[Tage Johansson]

Make it possible to run Rust tests with Valgrind with
`make check-valgrind` in the rust directory.
---
 rust/Makefile.am | 3 +++
 rust/run-tests.sh.in | 6 +-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/rust/Makefile.am b/rust/Makefile.am
index db8fc66..f9830d0 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -89,6 +89,9 @@ TESTS_ENVIRONMENT = \
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
 
+check-valgrind:
+   LIBNBD_VALGRIND=1 $(MAKE) check
+
 clean-local:
$(CARGO) clean
$(CARGO) clean --manifest-path cargo_test/Cargo.toml
diff --git a/rust/run-tests.sh.in b/rust/run-tests.sh.in
index d45b1bf..f7db344 100755
--- a/rust/run-tests.sh.in
+++ b/rust/run-tests.sh.in
@@ -23,4 +23,8 @@ set -x
 
 requires nbdkit --version
 
-@CARGO@ test -- --nocapture
+if [ -z "$VG" ]; then
+@CARGO@ test -- --nocapture
+else
+@CARGO@ test --config "target.'cfg(all())'.runner = \"$VG\"" -- --nocapture
+fi
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v5 06/12] generator: Add information about the lifetime of closures

[Tage Johansson]

Add two new fields, cblifetime and cbcount, to the `closure` type
in generator/API.ml*. cblifetime tells if the closure may only be used
for as long as the command is in flight or if the closure may be used
until the handle is destructed. cbcount tells whether the closure may
be called many times or just once.

This information is needed in the Rust bindings for:
a) Knowing if the closure trait should be FnMut or FnOnce
   (see ).
b) Knowing for what lifetime the closure should be valid. A closure that
   may be called after the function invokation has returned must live
   for the `'static` lietime. But static closures are inconveniant for
   the user since they can't effectively borrow any local data. So it is
   good if this restriction is relaxed when it is not needed.
---
 generator/API.ml  | 20 
 generator/API.mli | 17 +
 2 files changed, 37 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index f90a6fa..42b9eec 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -77,6 +77,8 @@ and ret =
 and closure = {
   cbname : string;
   cbargs : cbarg list;
+  cblifetime : cblifetime;
+  cbcount : cbcount
 }
 and cbarg =
 | CBArrayAndLen of arg * string
@@ -87,6 +89,12 @@ and cbarg =
 | CBString of string
 | CBUInt of string
 | CBUInt64 of string
+and cblifetime =
+| CBCommand
+| CBHandle
+and cbcount =
+| CBOnce
+| CBMany
 and enum = {
   enum_prefix : string;
   enums : (string * int) list
@@ -141,20 +149,28 @@ the handle from the NBD protocol handshake."
 (* Closures. *)
 let chunk_closure = {
   cbname = "chunk";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBBytesIn ("subbuf", "count");
  CBUInt64 "offset"; CBUInt "status";
  CBMutable (Int "error") ]
 }
 let completion_closure = {
   cbname = "completion";
+  cblifetime = CBCommand;
+  cbcount = CBOnce;
   cbargs = [ CBMutable (Int "error") ]
 }
 let debug_closure = {
   cbname = "debug";
+  cblifetime = CBHandle;
+  cbcount = CBMany;
   cbargs = [ CBString "context"; CBString "msg" ]
 }
 let extent_closure = {
   cbname = "extent";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "metacontext";
  CBUInt64 "offset";
  CBArrayAndLen (UInt32 "entries",
@@ -163,10 +179,14 @@ let extent_closure = {
 }
 let list_closure = {
   cbname = "list";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name"; CBString "description" ]
 }
 let context_closure = {
   cbname = "context";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name" ]
 }
 let all_closures = [ chunk_closure; completion_closure;
diff --git a/generator/API.mli b/generator/API.mli
index 361132d..ff85849 100644
--- a/generator/API.mli
+++ b/generator/API.mli
@@ -94,6 +94,12 @@ and ret =
 and closure = {
   cbname : string; (** name of callback function *)
   cbargs : cbarg list; (** all closures return int for now *)
+  (** An upper bound of the lifetime of the closure. Either it will be used for
+  as long as the command is in flight or it may be used until the handle
+  is destructed. *)
+  cblifetime : cblifetime;
+  (** Whether the callback may only be called once or many times. *)
+  cbcount : cbcount;
 }
 and cbarg =
 | CBArrayAndLen of arg * string (** array + number of entries *)
@@ -104,6 +110,17 @@ and cbarg =
 | CBString of string   (** like String *)
 | CBUInt of string (** like UInt *)
 | CBUInt64 of string   (** like UInt64 *)
+and cblifetime =
+| CBCommand (** The closure may only be used until the command is retired.
+(E.G., completion callback or list callback.) *)
+| CBHandle  (** The closure might be used until the handle is descructed.
+(E.G., debug callback.) *)
+and cbcount =
+| CBOnce (** The closure will be used 0 or 1 time if the aio_* call returned an
+ error and exactly once if the call succeeded.
+ (E.g., completion callback.) *)
+| CBMany (** The closure may be used any number of times.
+ (E.g., list callback.) *)
 and enum = {
   enum_prefix : string;(** prefix of each enum variant *)
   enums : (string * int) list (** enum names and their values in C *)
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v5 05/12] generator: Add information about asynchronous handle calls

[Tage Johansson]

A new field (async_kind) is added to the call data type in
generator/API.ml*. The purpose is to tell if a certain handle call is
an asynchronous command and if so how one can know when it is
completed. The motivation for this is that all asynchronous commands
on the AsyncHandle in the Rust bindings makes use of Rust's
[`async fn`s](https://doc.rust-lang.org/std/keyword.async.html).
But to implement such an async fn, the API needs to know when the
command completed, either by a completion callback or via a change
of state.
---
 generator/API.ml  | 32 
 generator/API.mli | 11 +++
 2 files changed, 43 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 5fcb0e1..f90a6fa 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -32,6 +32,7 @@ type call = {
   permitted_states : permitted_state list;
   is_locked : bool;
   may_set_error : bool;
+  async_kind : async_kind option;
   mutable first_version : int * int;
 }
 and arg =
@@ -102,6 +103,9 @@ and permitted_state =
 | Negotiating
 | Connected
 | Closed | Dead
+and async_kind =
+| WithCompletionCallback
+| ChangesState of string * bool
 and link =
 | Link of string
 | SectionLink of string
@@ -249,6 +253,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  see_also = [];
  permitted_states = [];
  is_locked = true; may_set_error = true;
+ async_kind = None;
  first_version = (0, 0) }
 
 (* Calls.
@@ -2798,6 +2803,7 @@ wait for an L.";
 default_call with
 args = [ SockAddrAndLen ("addr", "addrlen") ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Begin connecting to the NBD server.  The C and C
@@ -2810,6 +2816,7 @@ parameters specify the address of the socket to connect 
to.
 default_call with
 args = [ String "uri" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to an NBD URI";
 longdesc = "\
 Begin connecting to the NBD URI C.  Parameters behave as
@@ -2823,6 +2830,7 @@ documented in L.
 default_call with
 args = [ Path "unixsocket" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a Unix domain socket";
 longdesc = "\
 Begin connecting to the NBD server over Unix domain socket
@@ -2837,6 +2845,7 @@ L.
 default_call with
 args = [ UInt32 "cid"; UInt32 "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over AF_VSOCK socket";
 longdesc = "\
 Begin connecting to the NBD server over the C
@@ -2850,6 +2859,7 @@ L.
 default_call with
 args = [ String "hostname"; String "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a TCP port";
 longdesc = "\
 Begin connecting to the NBD server listening on C.
@@ -2862,6 +2872,7 @@ Parameters behave as documented in L.
 default_call with
 args = [ Fd "sock" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect directly to a connected socket";
 longdesc = "\
 Begin connecting to the connected socket C.
@@ -2874,6 +2885,7 @@ Parameters behave as documented in 
L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it over
@@ -2887,6 +2899,7 @@ L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect using systemd socket activation";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it using
@@ -2903,6 +2916,7 @@ L.
 optargs = [ OClosure completion_closure ];
 ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some WithCompletionCallback;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -2926,6 +2940,7 @@ when L returns true.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 

[Libguestfs] [libnbd PATCH v5 01/12] rust: create basic Rust bindings

[Tage Johansson]

generator/Makefile
  golang/Makefile
  golang/examples/Makefile
+ rust/Makefile
  include/Makefile
  info/Makefile
  interop/Makefile
@@ -717,6 +746,7 @@ echo
 echo "Language bindings:"
 echo
 feature "Go"test "x$HAVE_GOLANG_TRUE" = "x"
+feature "Rust"  test "x$HAVE_RUST_TRUE" = "x"
 feature "OCaml" test "x$HAVE_OCAML_TRUE" = "x"
 feature "Python"test "x$HAVE_PYTHON_TRUE" = "x"
 
diff --git a/generator/Makefile.am b/generator/Makefile.am
index c3d53b2..5e148be 100644
--- a/generator/Makefile.am
+++ b/generator/Makefile.am
@@ -60,6 +60,10 @@ sources = \
OCaml.ml \
GoLang.mli \
GoLang.ml \
+   RustSys.mli \
+   RustSys.ml \
+   Rust.mli \
+   Rust.ml \
generator.ml \
$(NULL)
 
diff --git a/generator/Rust.ml b/generator/Rust.ml
new file mode 100644
index 000..88434c3
--- /dev/null
+++ b/generator/Rust.ml
@@ -0,0 +1,548 @@
+(* hey emacs, this is OCaml code: -*- tuareg -*- *)
+(* nbd client library in userspace: generator
+ * Copyright Tage Johansson
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ *)
+
+(* Rust language bindings. *)
+
+open Printf
+open API
+open Utils
+
+(* The type for a set of names. *)
+module NameSet = Set.Make (String)
+
+(* List of handle calls which should not be part of the public API. This could
+   for instance be `set_debug` and `set_debug_callback` which are handled
+   separately by the log crate *)
+let hidden_handle_calls : NameSet.t =
+  NameSet.of_list
+[ "get_debug"; "set_debug"; "set_debug_callback"; "clear_debug_callback" ]
+
+let print_rust_constant (name, value) =
+  pr "pub const %s: u32 = %d;\n" name value
+
+let print_rust_enum enum =
+  pr "#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]\n";
+  pr "#[repr(isize)]";
+  pr "pub enum %s {\n" (camel_case enum.enum_prefix);
+  List.iter
+(fun (name, num) -> pr "%s = %d,\n" (camel_case name) num)
+enum.enums;
+  pr "}\n\n"
+
+(* Print a Rust struct for a set of flags. *)
+let print_rust_flags { flag_prefix; flags } =
+  pr "bitflags! {\n";
+  pr "#[repr(C)]\n";
+  pr "#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]\n";
+  pr "pub struct %s: u32 {\n" (camel_case flag_prefix);
+  List.iter
+(fun (name, value) -> pr "const %s = %d;\n" name value)
+flags;
+  pr "}\n";
+  pr "}\n\n"
+
+(* Print metadata namespaces. *)
+let print_metadata_namespace (ns, ctxts) =
+  pr "pub const NAMESPACE_%s: &[u8] = b\"%s:\";\n"
+(String.uppercase_ascii ns)
+ns;
+  List.iter
+(fun (ctxt, consts) ->
+  let s = ns ^ ":" ^ ctxt in
+  pr "pub const CONTEXT_%s_%s: &[u8] = b\"%s\";\n"
+(String.uppercase_ascii ns)
+(String.uppercase_ascii ctxt)
+s;
+  List.iter
+(fun (n, i) ->
+  pr "pub const %s: u32 = %d;\n" (String.uppercase_ascii n) i)
+consts)
+ctxts
+
+(* Get the name of a rust argument. *)
+let rust_arg_name : arg -> string = function
+  | Bool n
+  | Int n
+  | UInt n
+  | UIntPtr n
+  | UInt32 n
+  | Int64 n
+  | UInt64 n
+  | SizeT n
+  | String n
+  | StringList n
+  | Path n
+  | Fd n
+  | Enum (n, _)
+  | Flags (n, _)
+  | SockAddrAndLen (n, _)
+  | BytesIn (n, _)
+  | BytesPersistIn (n, _)
+  | BytesOut (n, _)
+  | BytesPersistOut (n, _)
+  | Closure { cbname = n } ->
+  n
+
+(* Get the name of a rust optional argument. *)
+let rust_optarg_name : optarg -> string = function
+  | OClosure { cbname = n } | OFlags (n, _, _) -> n
+
+(* Get the name of a Rust closure argument. *)
+let rust_cbarg_name : cbarg -> string = function
+  | CBInt n | CBUInt n | CBInt64 n | CBUInt64 n | CBString n | CBBytesIn (n, _)
+->
+  n
+  | CBArrayAndLen (arg, _) | CBMutable arg -> rust_arg_name arg
+
+(* Get the Rust type for 

[Libguestfs] [libnbd PATCH v5 02/12] rust: Add a couple of integration tests

[Tage Johansson]

A couple of integration tests are added in rust/tests. They are mostly
ported from the OCaml tests.
---
 rust/Cargo.toml  |   4 +
 rust/Makefile.am |  22 +++
 rust/run-tests.sh.in |   4 +-
 rust/tests/nbdkit_pattern/mod.rs |  28 
 rust/tests/test_100_handle.rs|  25 
 rust/tests/test_110_defaults.rs  |  33 +
 rust/tests/test_120_set_non_defaults.rs  |  53 +++
 rust/tests/test_130_private_data.rs  |  28 
 rust/tests/test_140_explicit_close.rs|  31 
 rust/tests/test_200_connect_command.rs   |  32 
 rust/tests/test_210_opt_abort.rs |  31 
 rust/tests/test_220_opt_list.rs  |  86 +++
 rust/tests/test_230_opt_info.rs  | 120 +++
 rust/tests/test_240_opt_list_meta.rs | 147 +++
 rust/tests/test_245_opt_list_meta_queries.rs |  93 
 rust/tests/test_250_opt_set_meta.rs  | 123 
 rust/tests/test_255_opt_set_meta_queries.rs  | 109 ++
 rust/tests/test_300_get_size.rs  |  35 +
 rust/tests/test_400_pread.rs |  39 +
 rust/tests/test_405_pread_structured.rs  |  79 ++
 rust/tests/test_410_pwrite.rs|  58 
 rust/tests/test_460_block_status.rs  |  92 
 rust/tests/test_620_stats.rs |  75 ++
 rust/tests/test_log/mod.rs   |  86 +++
 24 files changed, 1432 insertions(+), 1 deletion(-)
 create mode 100644 rust/tests/nbdkit_pattern/mod.rs
 create mode 100644 rust/tests/test_100_handle.rs
 create mode 100644 rust/tests/test_110_defaults.rs
 create mode 100644 rust/tests/test_120_set_non_defaults.rs
 create mode 100644 rust/tests/test_130_private_data.rs
 create mode 100644 rust/tests/test_140_explicit_close.rs
 create mode 100644 rust/tests/test_200_connect_command.rs
 create mode 100644 rust/tests/test_210_opt_abort.rs
 create mode 100644 rust/tests/test_220_opt_list.rs
 create mode 100644 rust/tests/test_230_opt_info.rs
 create mode 100644 rust/tests/test_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_300_get_size.rs
 create mode 100644 rust/tests/test_400_pread.rs
 create mode 100644 rust/tests/test_405_pread_structured.rs
 create mode 100644 rust/tests/test_410_pwrite.rs
 create mode 100644 rust/tests/test_460_block_status.rs
 create mode 100644 rust/tests/test_620_stats.rs
 create mode 100644 rust/tests/test_log/mod.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index c745972..b498930 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -47,3 +47,7 @@ libc = "0.2.147"
 
 [features]
 default = ["log"]
+
+[dev-dependencies]
+once_cell = "1.18.0"
+tempfile = "3.6.0"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index 8615794..db8fc66 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -36,6 +36,27 @@ source_files = \
cargo_test/Cargo.toml \
cargo_test/src/lib.rs \
cargo_test/README.md \
+   tests/nbdkit_pattern/mod.rs \
+   tests/test_100_handle.rs \
+   tests/test_110_defaults.rs \
+   tests/test_120_set_non_defaults.rs \
+   tests/test_130_private_data.rs \
+   tests/test_140_explicit_close.rs \
+   tests/test_200_connect_command.rs \
+   tests/test_210_opt_abort.rs \
+   tests/test_220_opt_list.rs \
+   tests/test_230_opt_info.rs \
+   tests/test_240_opt_list_meta.rs \
+   tests/test_245_opt_list_meta_queries.rs \
+   tests/test_250_opt_set_meta.rs \
+   tests/test_255_opt_set_meta_queries.rs \
+   tests/test_300_get_size.rs \
+   tests/test_400_pread.rs \
+   tests/test_405_pread_structured.rs \
+   tests/test_410_pwrite.rs \
+   tests/test_460_block_status.rs \
+   tests/test_620_stats.rs \
+   tests/test_log/mod.rs \
run-tests.sh.in \
$(NULL)
 
@@ -63,6 +84,7 @@ TESTS_ENVIRONMENT = \
LIBNBD_DEBUG=1 \
$(MALLOC_CHECKS) \
abs_top_srcdir=$(abs_top_srcdir) \
+   CARGO=$(CARGO) \
$(NULL)
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
diff --git a/rust/run-tests.sh.in b/rust/run-tests.sh.in
index f4d220c..d45b1bf 100755
--- a/rust/run-tests.sh.in
+++ b/rust/run-tests.sh.in
@@ -1,6 +1,6 @@
 #!/bin/bash -
 # nbd client library in userspace
-# Copyright Red Hat
+# Copyright Tage Johansson
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -21,4 +21,6 @@
 set -e
 set -x
 
+requires nbdkit --version
+
 @CARGO@ test -- --nocapture
diff --git a/rust/tests/nbdkit_pattern/mod.rs b/rust/tests/nbdkit_pattern/mod.rs
new

[Libguestfs] [libnbd PATCH v5 00/12] Rust Bindings for Libnbd

[Tage Johansson]

A new version of the Rust bindings for Libnbd. Most changes have been
mentioned as replies to the comments I got on v4. But the most significant
change is probably that the patches have been reordered.

Best regards,
Tage


Tage Johansson (12):
  rust: create basic Rust bindings
  rust: Add a couple of integration tests
  rust: Make it possible to run tests with Valgrind
  rust: Add some examples
  generator: Add information about asynchronous handle calls
  generator: Add information about the lifetime of closures
  rust: Use more specific closure traits
  rust: async: Create an async friendly handle type
  generator: Add `modifies_fd` flag to the [call] structure
  rust: async: Use the modifies_fd flag to exclude calls
  rust: async: Add a couple of integration tests
  rust: async: Add an example

 .gitignore|  10 +
 .ocamlformat  |   4 +
 Makefile.am   |   2 +
 configure.ac  |  30 +
 generator/API.ml  |  84 ++
 generator/API.mli |  35 +
 generator/Makefile.am |   4 +
 generator/Rust.ml | 794 ++
 generator/Rust.mli|  22 +
 generator/RustSys.ml  | 167 
 generator/RustSys.mli |  19 +
 generator/generator.ml|   4 +
 rust/Cargo.toml   |  59 ++
 rust/Makefile.am  | 106 +++
 rust/cargo_test/Cargo.toml|  23 +
 rust/cargo_test/README.md |   3 +
 rust/cargo_test/src/lib.rs|  31 +
 rust/examples/concurrent-read-write.rs| 135 +++
 rust/examples/connect-command.rs  |  39 +
 rust/examples/fetch-first-sector.rs   |  38 +
 rust/examples/get-size.rs |  29 +
 rust/libnbd-sys/Cargo.toml|  32 +
 rust/libnbd-sys/build.rs  |  26 +
 rust/libnbd-sys/src/.keep |   0
 rust/run-tests.sh.in  |  39 +
 rust/src/async_handle.rs  | 268 ++
 rust/src/error.rs | 157 
 rust/src/handle.rs|  67 ++
 rust/src/lib.rs   |  36 +
 rust/src/types.rs |  20 +
 rust/src/utils.rs |  23 +
 rust/tests/nbdkit_pattern/mod.rs  |  28 +
 rust/tests/test_100_handle.rs |  25 +
 rust/tests/test_110_defaults.rs   |  33 +
 rust/tests/test_120_set_non_defaults.rs   |  53 ++
 rust/tests/test_130_private_data.rs   |  28 +
 rust/tests/test_140_explicit_close.rs |  31 +
 rust/tests/test_200_connect_command.rs|  32 +
 rust/tests/test_210_opt_abort.rs  |  31 +
 rust/tests/test_220_opt_list.rs   |  86 ++
 rust/tests/test_230_opt_info.rs   | 120 +++
 rust/tests/test_240_opt_list_meta.rs  | 147 
 rust/tests/test_245_opt_list_meta_queries.rs  |  93 ++
 rust/tests/test_250_opt_set_meta.rs   | 123 +++
 rust/tests/test_255_opt_set_meta_queries.rs   | 109 +++
 rust/tests/test_300_get_size.rs   |  35 +
 rust/tests/test_400_pread.rs  |  39 +
 rust/tests/test_405_pread_structured.rs   |  79 ++
 rust/tests/test_410_pwrite.rs |  58 ++
 rust/tests/test_460_block_status.rs   |  92 ++
 rust/tests/test_620_stats.rs  |  75 ++
 rust/tests/test_async_100_handle.rs   |  25 +
 rust/tests/test_async_200_connect_command.rs  |  33 +
 rust/tests/test_async_210_opt_abort.rs|  32 +
 rust/tests/test_async_220_opt_list.rs |  81 ++
 rust/tests/test_async_230_opt_info.rs | 122 +++
 rust/tests/test_async_240_opt_list_meta.rs| 147 
 .../test_async_245_opt_list_meta_queries.rs   |  91 ++
 rust/tests/test_async_250_opt_set_meta.rs | 122 +++
 .../test_async_255_opt_set_meta_queries.rs| 107 +++
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 ++
 rust/tests/test_async_460_block_status.rs |  92 ++
 rust/tests/test_async_620_stats.rs|  76 ++
 rust/tests/test_log/mod.rs|  86 ++
 rustfmt.toml  |  19 +
 scripts/git.orderfile |  12 +
 68 files changed, 4851 insertions(+)
 create mode 100644 .ocamlformat
 create mode 100644 generator/Rust.ml
 create mode 100644 generator/Rust.mli
 create mode 100644 generator/RustSys.ml
 create mode 100644 generator/RustSys.mli
 create mode 100644 rust/Cargo.toml
 create mode 100644 rust/Makefile.am
 create mode 100644 rust/cargo_test/Cargo.toml
 create mode 100644 rust

[Libguestfs] [libnbd PATCH v5 10/12] rust: async: Use the modifies_fd flag to exclude calls

[Tage Johansson]

All handle calls which has the modifies_fd flag set to true will be
excluded from AsyncHandle (the asynchronous handle in the rust
bindings). This is a better approach then listing all calls that should
be excluded in Rust.ml explicetly.
---
 generator/Rust.ml | 60 +--
 1 file changed, 27 insertions(+), 33 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index f6016d4..fad991a 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -568,18 +568,17 @@ let generate_rust_bindings () =
 
 let excluded_handle_calls : NameSet.t =
   NameSet.of_list
-[
-  "aio_get_fd";
-  "aio_get_direction";
-  "aio_notify_read";
-  "aio_notify_write";
-  "clear_debug_callback";
-  "get_debug";
-  "poll";
-  "poll2";
-  "set_debug";
-  "set_debug_callback";
-]
+  @@ [
+   "aio_get_fd";
+   "aio_get_direction";
+   "clear_debug_callback";
+   "get_debug";
+   "set_debug";
+   "set_debug_callback";
+ ]
+  @ (handle_calls
+|> List.filter (fun (_, { modifies_fd }) -> modifies_fd)
+|> List.map (fun (name, _) -> name))
 
 (* A mapping with names as keys. *)
 module NameMap = Map.Make (String)
@@ -590,16 +589,16 @@ let strip_aio name : string =
 String.sub name 4 (String.length name - 4)
   else failwithf "Asynchronous call %s must begin with aio_" name
 
-(* A map with all asynchronous handle calls. The keys are names with "aio_"
-   stripped, the values are a tuple with the actual name (with "aio_"), the
-   [call] and the [async_kind]. *)
+(* A map with all asynchronous handle calls. The keys are names, the values
+   are tuples of: the name with "aio_" stripped, the [call] and the
+   [async_kind]. *)
 let async_handle_calls : ((string * call) * async_kind) NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
   |> List.filter_map (fun (name, call) ->
  call.async_kind
  |> Option.map (fun async_kind ->
-(strip_aio name, ((name, call), async_kind
+(name, ((strip_aio name, call), async_kind
   |> List.to_seq |> NameMap.of_seq
 
 (* A mapping with all synchronous (not asynchronous) handle calls. Excluded
@@ -609,11 +608,7 @@ let async_handle_calls : ((string * call) * async_kind) 
NameMap.t =
 let sync_handle_calls : call NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
-  |> List.filter (fun (name, _) ->
- (not (NameMap.mem name async_handle_calls))
- && not
-  (String.starts_with ~prefix:"aio_" name
-  && NameMap.mem (strip_aio name) async_handle_calls))
+  |> List.filter (fun (n, _) -> not (NameMap.mem n async_handle_calls))
   |> List.to_seq |> NameMap.of_seq
 
 (* Get the Rust type for an argument in the asynchronous API. Like
@@ -658,7 +653,7 @@ let print_rust_sync_handle_call name call =
 (* Print the Rust function for an asynchronous handle call with a completion
callback. (Note that "callback" might be abbreviated with "cb" in the
following code. *)
-let print_rust_async_handle_call_with_completion_cb name (aio_name, call) =
+let print_rust_async_handle_call_with_completion_cb aio_name (name, call) =
   (* An array of all optional arguments. Useful because we need to deel with
  the index of the completion callback. *)
   let optargs = Array.of_list call.optargs in
@@ -738,7 +733,7 @@ let print_rust_async_handle_call_with_completion_cb name 
(aio_name, call) =
completion by changing state. The predicate is a call like
"aio_is_connecting" which should get the value (like false) for the call to
be complete. *)
-let print_rust_async_handle_call_changing_state name (aio_name, call)
+let print_rust_async_handle_call_changing_state aio_name (name, call)
 (predicate, value) =
   let value = if value then "true" else "false" in
   print_rust_handle_call_comment call;
@@ -767,16 +762,15 @@ let print_rust_async_handle_call_changing_state name 
(aio_name, call)
 (* Print an impl with all handle calls. *)
 let print_rust_async_handle_impls () =
   pr "impl AsyncHandle {\n";
-  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
-  NameMap.iter
-(fun name (call, async_kind) ->
-  match async_kind with
-  | WithCompletionCallback ->
-  print_rust_async_handle_call_with_completion_cb name call
-  | ChangesState (predicate, value) ->
-  print_rust_async_handle_call_changing_state name call
-(predicate, value))
-async_handle_calls;
+  sync_handle_calls |> NameMap.iter print_rust_sync_handle_call;
+  async_handle_calls
+  |> NameMap.iter (fun name (call, async_kind) ->
+ match async_kind with
+ | WithCompletionCallback ->
+ print_rust_async_handle_call_with_completion_cb name call
+ | ChangesState (predicate, value) ->
+ print_rust_async_handle_call_changing_state 

[Libguestfs] [libnbd PATCH v5 04/12] rust: Add some examples

[Tage Johansson]

This patch adds a few examples in rust/examples/. The examples are
compiled and run as part of the test suite.
---
 rust/Cargo.toml |  2 ++
 rust/Makefile.am|  3 +++
 rust/examples/connect-command.rs| 39 +
 rust/examples/fetch-first-sector.rs | 38 
 rust/examples/get-size.rs   | 29 +
 rust/run-tests.sh.in|  7 ++
 scripts/git.orderfile   |  1 +
 7 files changed, 119 insertions(+)
 create mode 100644 rust/examples/connect-command.rs
 create mode 100644 rust/examples/fetch-first-sector.rs
 create mode 100644 rust/examples/get-size.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index b498930..04e371e 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -49,5 +49,7 @@ libc = "0.2.147"
 default = ["log"]
 
 [dev-dependencies]
+anyhow = "1.0.72"
 once_cell = "1.18.0"
+pretty-hex = "0.3.0"
 tempfile = "3.6.0"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index f9830d0..bdfe2ca 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -30,6 +30,9 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   examples/connect-command.rs \
+   examples/get-size.rs \
+   examples/fetch-first-sector.rs \
libnbd-sys/Cargo.toml \
libnbd-sys/build.rs \
libnbd-sys/src/.keep \
diff --git a/rust/examples/connect-command.rs b/rust/examples/connect-command.rs
new file mode 100644
index 000..db4adbe
--- /dev/null
+++ b/rust/examples/connect-command.rs
@@ -0,0 +1,39 @@
+//! This example shows how to run an NBD server
+//! (nbdkit) as a subprocess of libnbd.
+
+
+fn main() -> libnbd::Result<()> {
+// Create the libnbd handle.
+let handle = libnbd::Handle::new()?;
+
+// Run nbdkit as a subprocess.
+let args = [
+"nbdkit",
+// You must use ‘-s’ (which tells nbdkit to serve
+// a single connection on stdin/stdout).
+"-s",
+// It is recommended to use ‘--exit-with-parent’
+// to ensure nbdkit is always cleaned up even
+// if the main program crashes.
+"--exit-with-parent",
+// Use this to enable nbdkit debugging.
+"-v",
+// The nbdkit plugin name - this is a RAM disk.
+"memory",
+"size=1M",
+];
+handle.connect_command()?;
+
+// Write some random data to the first sector.
+let wbuf: Vec = (0..512).into_iter().map(|i| (i % 13) as u8).collect();
+handle.pwrite(, 0, None)?;
+
+// Read the first sector back.
+let mut rbuf = [0; 512];
+handle.pread( rbuf, 0, None)?;
+
+// What was read must be exactly the same as what was written.
+assert_eq!(wbuf.as_slice(), rbuf.as_slice());
+
+Ok(())
+}
diff --git a/rust/examples/fetch-first-sector.rs 
b/rust/examples/fetch-first-sector.rs
new file mode 100644
index 000..9efb47a
--- /dev/null
+++ b/rust/examples/fetch-first-sector.rs
@@ -0,0 +1,38 @@
+//! This example shows how to connect to an NBD server
+//! and fetch and print the first sector (usually the
+//! boot sector or partition table or filesystem
+//! superblock).
+//!
+//! You can test it with nbdkit like this:
+//!
+//! nbdkit -U - floppy . \
+//!   --run 'cargo run --example fetch-first-sector -- $unixsocket'
+//!
+//! The nbdkit floppy plugin creates an MBR disk so the
+//! first sector is the partition table.
+
+use pretty_hex::pretty_hex;
+use std::env;
+
+fn main() -> anyhow::Result<()> {
+let nbd = libnbd::Handle::new()?;
+
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// Connect to the NBD server over a
+// Unix domain socket.
+nbd.connect_unix(socket)?;
+
+// Read the first sector synchronously.
+let mut buf = [0; 512];
+nbd.pread( buf, 0, None)?;
+
+// Print the sector in hexdump like format.
+print!("{}", pretty_hex());
+
+Ok(())
+}
diff --git a/rust/examples/get-size.rs b/rust/examples/get-size.rs
new file mode 100644
index 000..7f31df5
--- /dev/null
+++ b/rust/examples/get-size.rs
@@ -0,0 +1,29 @@
+//! This example shows how to connect to an NBD
+//! server and read the size of the disk.
+//!
+//! You can test it with nbdkit like this:
+//!
+//! nbdkit -U - memory 1M \
+//!   --run 'cargo run --example get-size -- $unixsocket'
+
+use std::env;
+
+fn main() -> anyhow::Result<()> {
+let nbd = libnbd::Handle::new()?;
+
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// Connect to the NBD server over a
+// Unix domain socket.
+nbd.connect_unix(socket)?;
+
+// Read the size in bytes and print it.
+let size = nbd.get_size()?;
+println!("{:?}: size = {size} bytes", socket);
+
+Ok(())
+}
diff --git a/rust/run-tests.sh.in 

[Libguestfs] [libnbd PATCH v5 11/12] rust: async: Add a couple of integration tests

[Tage Johansson]

Add a couple of integration tests as rust/tests/test_async_*.rs. They
are very similar to the tests for the synchronous API.
---
 rust/Cargo.toml   |   1 +
 rust/tests/test_async_100_handle.rs   |  25 +++
 rust/tests/test_async_200_connect_command.rs  |  33 
 rust/tests/test_async_210_opt_abort.rs|  32 
 rust/tests/test_async_220_opt_list.rs |  81 ++
 rust/tests/test_async_230_opt_info.rs | 122 +++
 rust/tests/test_async_240_opt_list_meta.rs| 147 ++
 .../test_async_245_opt_list_meta_queries.rs   |  91 +++
 rust/tests/test_async_250_opt_set_meta.rs | 122 +++
 .../test_async_255_opt_set_meta_queries.rs| 107 +
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 +++
 rust/tests/test_async_460_block_status.rs |  92 +++
 rust/tests/test_async_620_stats.rs|  76 +
 15 files changed, 1112 insertions(+)
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index e076826..d001248 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -55,3 +55,4 @@ anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
 tempfile = "3.6.0"
+tokio = { version = "1.29.1", default-features = false, features = 
["rt-multi-thread", "macros"] }
diff --git a/rust/tests/test_async_100_handle.rs 
b/rust/tests/test_async_100_handle.rs
new file mode 100644
index 000..e50bad9
--- /dev/null
+++ b/rust/tests/test_async_100_handle.rs
@@ -0,0 +1,25 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+//! Just check that we can link with libnbd and create a handle.
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_nbd_handle_new() {
+let _ = libnbd::AsyncHandle::new().unwrap();
+}
diff --git a/rust/tests/test_async_200_connect_command.rs 
b/rust/tests/test_async_200_connect_command.rs
new file mode 100644
index 000..8a3f497
--- /dev/null
+++ b/rust/tests/test_async_200_connect_command.rs
@@ -0,0 +1,33 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+#![deny(warnings)]
+
+
+#[tokio::test]
+async fn test_async_connect_command() {
+let nbd = libnbd::AsyncHandle::new().unwrap();
+nbd.connect_command(&[
+"nbdkit",
+"-s",
+"--exit-with-parent",
+"-v",
+"null",
+])
+.await
+.unwrap();
+}
diff --git a/rust/tests/tes

[Libguestfs] [libnbd PATCH v5 07/12] rust: Use more specific closure traits

[Tage Johansson]

For closures with cbcount = CBOnce, FnOnce will be used instead of
FnMut. Moreover, closures in synchronous commands with
cblifetime = CBCommand will not need to live for the static lifetime.
See [here](https://doc.rust-lang.org/std/ops/trait.FnOnce.html) for more
information about the advantages of using `FnOnce` when possible.
---
 generator/Rust.ml  | 44 
 rust/src/handle.rs |  2 ++
 rust/src/types.rs  |  2 ++
 3 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 88434c3..cd31b60 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -111,7 +111,7 @@ let rust_cbarg_name : cbarg -> string = function
   | CBArrayAndLen (arg, _) | CBMutable arg -> rust_arg_name arg
 
 (* Get the Rust type for an argument. *)
-let rec rust_arg_type : arg -> string = function
+let rec rust_arg_type ?(async_kind = None) : arg -> string = function
   | Bool _ -> "bool"
   | Int _ -> "c_int"
   | UInt _ -> "c_uint"
@@ -131,15 +131,18 @@ let rec rust_arg_type : arg -> string = function
   | BytesOut _ -> " [u8]"
   | BytesPersistIn _ -> "&'static [u8]"
   | BytesPersistOut _ -> "&'static mut [u8]"
-  | Closure { cbargs } -> "impl " ^ rust_closure_trait cbargs
+  | Closure { cbargs; cbcount } -> (
+  match async_kind with
+  | Some _ -> "impl " ^ rust_closure_trait cbargs cbcount
+  | None -> "impl " ^ rust_closure_trait cbargs cbcount ~lifetime:None)
 
 (* Get the Rust closure trait for a callback, That is `Fn*(...) -> ...)`. *)
-and rust_closure_trait ?(lifetime = Some "'static") cbargs : string =
+and rust_closure_trait ?(lifetime = Some "'static") cbargs cbcount : string =
   let rust_cbargs = String.concat ", " (List.map rust_cbarg_type cbargs)
-  and lifetime_constraint =
-match lifetime with None -> "" | Some x -> " + " ^ x
-  in
-  "FnMut(" ^ rust_cbargs ^ ") -> c_int + Send + Sync" ^ lifetime_constraint
+  and closure_type =
+match cbcount with CBOnce -> "FnOnce" | CBMany -> "FnMut"
+  and lifetime = match lifetime with None -> "" | Some x -> " + " ^ x in
+  sprintf "%s(%s) -> c_int + Send + Sync%s" closure_type rust_cbargs lifetime
 
 (* Get the Rust type for a callback argument. *)
 and rust_cbarg_type : cbarg -> string = function
@@ -153,8 +156,8 @@ and rust_cbarg_type : cbarg -> string = function
   | CBMutable arg -> " " ^ rust_arg_type arg
 
 (* Get the type of a rust optional argument. *)
-let rust_optarg_type : optarg -> string = function
-  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x))
+let rust_optarg_type ?(async_kind = None) : optarg -> string = function
+  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x) ~async_kind)
   | OFlags (name, flags, _) ->
   sprintf "Option<%s>" (rust_arg_type (Flags (name, flags)))
 
@@ -419,8 +422,8 @@ let ffi_ret_to_rust call =
closure data, and a free function for the closure data. This struct is what
will be sent to a C function taking the closure as an argument. In fact,
the struct itself is generated by rust-bindgen. *)
-let print_rust_closure_to_raw_fn { cbname; cbargs } =
-  let closure_trait = rust_closure_trait cbargs ~lifetime:None in
+let print_rust_closure_to_raw_fn { cbname; cbargs; cbcount } =
+  let closure_trait = rust_closure_trait cbargs cbcount ~lifetime:None in
   let ffi_cbargs_names = List.flatten (List.map ffi_cbarg_names cbargs) in
   let ffi_cbargs_types = List.flatten (List.map ffi_cbarg_types cbargs) in
   let rust_cbargs_names = List.map rust_cbarg_name cbargs in
@@ -435,16 +438,24 @@ let print_rust_closure_to_raw_fn { cbname; cbargs } =
(List.map2 (sprintf "%s: %s") ffi_cbargs_names ffi_cbargs_types));
   pr "  where F: %s\n" closure_trait;
   pr "{\n";
-  pr "let callback_ptr = data as *mut F;\n";
-  pr "let callback =  *callback_ptr;\n";
+  (match cbcount with
+  | CBMany ->
+  pr "let callback_ptr = data as *mut F;\n";
+  pr "let callback =  *callback_ptr;\n"
+  | CBOnce ->
+  pr "let callback_ptr = data as *mut Option;\n";
+  pr "let callback_option:  Option =  *callback_ptr;\n";
+  pr "let callback: F = callback_option.take().unwrap();\n");
   List.iter ffi_cbargs_to_rust cbargs;
   pr "callback(%s)\n" (String.concat ", " rust_cbargs_names);
   pr "}\n";
-  pr "let callback_data = Box::into_raw(Box::new(f));\n";
+  pr "let callback_data = Box::into_raw(Box::new(%s));\n"
+(match cbcount with CBMany -> "f" | CBOnce -> "Some(f)");
   pr "sys::nbd_%s_callback {\n" cbname;
   pr "callback: Some(call_closure::),\n";
   pr "user_data: callback_data as *mut _,\n";
-  pr "free: Some(utils::drop_data::),\n";
+  pr "free: Some(utils::drop_data::<%s>),\n"
+(match cbcount with CBMany -> "F" | CBOnce -> "Option");
   pr "}\n";
   pr "}\n";
   pr "\n"
@@ -501,7 +512,8 @@ let print_rust_handle_method (name, call) =
   let 

[Libguestfs] [libnbd PATCH v5 08/12] rust: async: Create an async friendly handle type

[Tage Johansson]

 name
+(rust_async_handle_call_args call);
+  pr "{\n";
+  print_ffi_call aio_name "self.data.handle.handle" call;
+  pr "?;\n";
+  pr "let (ret_tx, ret_rx) = oneshot::channel::>();\n";
+  pr "let mut ret_tx = Some(ret_tx);\n";
+  pr "let completion_predicate = \n";
+  pr " move |handle: , res: <()>| {\n";
+  pr "  let ret = if let Err(_) = res {\n";
+  pr "res.clone()\n";
+  pr "  } else {\n";
+  pr "if handle.%s() != %s { return false; }\n" predicate value;
+  pr "else { Ok(()) }\n";
+  pr "  };\n";
+  pr "  ret_tx.take().unwrap().send(ret).ok();\n";
+  pr "  true\n";
+  pr "};\n";
+  pr "self.add_command(completion_predicate)?;\n";
+  pr "ret_rx.await.unwrap()\n";
+  pr "}\n\n"
+
+(* Print an impl with all handle calls. *)
+let print_rust_async_handle_impls () =
+  pr "impl AsyncHandle {\n";
+  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
+  NameMap.iter
+(fun name (call, async_kind) ->
+  match async_kind with
+  | WithCompletionCallback ->
+  print_rust_async_handle_call_with_completion_cb name call
+  | ChangesState (predicate, value) ->
+  print_rust_async_handle_call_changing_state name call
+(predicate, value))
+async_handle_calls;
+  pr "}\n\n"
+
+let print_rust_async_imports () =
+  pr "use crate::{*, types::*};\n";
+  pr "use os_socketaddr::OsSocketAddr;\n";
+  pr "use std::ffi::*;\n";
+  pr "use std::mem;\n";
+  pr "use std::net::SocketAddr;\n";
+  pr "use std::os::fd::{AsRawFd, OwnedFd};\n";
+  pr "use std::os::unix::prelude::*;\n";
+  pr "use std::path::PathBuf;\n";
+  pr "use std::ptr;\n";
+  pr "use std::sync::Arc;\n";
+  pr "use tokio::sync::oneshot;\n";
+  pr "\n"
+
+let generate_rust_async_bindings () =
+  generate_header CStyle ~copyright:"Tage Johansson";
+  pr "\n";
+  print_rust_async_imports ();
+  print_rust_async_handle_impls ()
diff --git a/generator/Rust.mli b/generator/Rust.mli
index 450e4ca..0960170 100644
--- a/generator/Rust.mli
+++ b/generator/Rust.mli
@@ -18,3 +18,5 @@
 
 (* Print all flag-structs, enums, constants and handle calls in Rust code. *)
 val generate_rust_bindings : unit -> unit
+
+val generate_rust_async_bindings : unit -> unit
diff --git a/generator/generator.ml b/generator/generator.ml
index f5ef7cc..2118446 100644
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -64,3 +64,4 @@ let () =
 
   output_to ~formatter:(Some Rustfmt) "rust/libnbd-sys/src/lib.rs" 
RustSys.generate_rust_sys_bindings;
   output_to ~formatter:(Some Rustfmt) "rust/src/bindings.rs" 
Rust.generate_rust_bindings;
+  output_to ~formatter:(Some Rustfmt) "rust/src/async_bindings.rs" 
Rust.generate_rust_async_bindings;
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 04e371e..e076826 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -44,9 +44,11 @@ os_socketaddr = "0.2.4"
 thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
+tokio = { optional = true, version = "1.29.1", default-features = false, 
features = ["rt", "sync", "net"] }
+epoll = "4.3.3"
 
 [features]
-default = ["log"]
+default = ["log", "tokio"]
 
 [dev-dependencies]
 anyhow = "1.0.72"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index bdfe2ca..e553d8f 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -19,6 +19,7 @@ include $(top_srcdir)/subdir-rules.mk
 
 generator_built = \
libnbd-sys/src/lib.rs \
+   src/async_bindings.rs \
src/bindings.rs \
$(NULL)
 
@@ -30,6 +31,7 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   src/async_handle.rs \
examples/connect-command.rs \
examples/get-size.rs \
examples/fetch-first-sector.rs \
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
new file mode 100644
index 000..4223b80
--- /dev/null
+++ b/rust/src/async_handle.rs
@@ -0,0 +1,268 @@
+// nbd client library in userspace
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GN

Re: [Libguestfs] [libnbd PATCH v4 11/11] rust: Add some examples

[Tage Johansson]

On 8/2/2023 6:53 PM, Richard W.M. Jones wrote:

On Wed, Aug 02, 2023 at 12:40:56PM +, Tage Johansson wrote:

This patch adds a few examples in rust/examples/. The examples are
compiled and run as part of the test suite.
---
  rust/Makefile.am   |   3 +
  rust/examples/concurrent-read-write.rs | 135 +
  rust/examples/connect-command.rs   |  39 +++
  rust/examples/fetch-first-sector.rs|  38 +++
  rust/examples/get-size.rs  |  29 ++
  rust/run-tests.sh  |   7 ++
  6 files changed, 251 insertions(+)
  create mode 100644 rust/examples/concurrent-read-write.rs
  create mode 100644 rust/examples/connect-command.rs
  create mode 100644 rust/examples/fetch-first-sector.rs
  create mode 100644 rust/examples/get-size.rs

diff --git a/rust/Makefile.am b/rust/Makefile.am
index b954b22..d75163d 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -32,6 +32,9 @@ source_files = \
src/types.rs \
src/utils.rs \
src/async_handle.rs \
+   examples/connect-command.rs \
+   examples/get-size.rs \
+   examples/fetch-first-sector.rs \

This doesn't list all the source files, it is missing
examples/concurrent-read-write.rs.

If you split out examples/connect-command.rs, examples/get-size.rs and
examples/fetch-first-sector.rs into a separate patch (since those
don't depend on asynch), and moved that patch earlier in the sequence,
then it could go upstream earlier.



Yes, I have done that.


Best regards,

Tage



libnbd-sys/Cargo.toml \
libnbd-sys/build.rs \
$(NULL)
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..a1c3e8a
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,135 @@
+//! Example usage with nbdkit:
+//!
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//!
+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+nbd.connect_unix()?;
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket.clone().into(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket: PathBuf,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// fu

Re: [Libguestfs] [libnbd PATCH v4 08/11] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

On 8/2/2023 6:39 PM, Richard W.M. Jones wrote:

On Wed, Aug 02, 2023 at 12:40:53PM +, Tage Johansson wrote:

Add a flag (`modifies_fd`) to the [call] structure in generator/API.ml
which is set to [true] if the handle call may do something with the
file descriptor. That is, it is [true] for all calls which are or may
call [aio_notify_*], including all synchronous commands like
[nbd_connect] or [nbd_opt_go].

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to [true] will be
excluded from that handle. The asynchronous (`aio_*`) functions will be
used instead.

Isn't this just saying that the call is synchronous?

Rich.



Kind of, it is true for all synchronous commands and the functions 
aio_notify_read() and aio_notify_write(). I could mention those 
explicitly though and rename the flag to something like 
"is_synchronous", if you want?



Best regards,

Tage



---
  generator/API.ml  | 32 
  generator/API.mli |  7 +++
  2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 42b9eec..6e1670d 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
is_locked : bool;
may_set_error : bool;
async_kind : async_kind option;
+  modifies_fd: bool;
mutable first_version : int * int;
  }
  and arg =
@@ -274,6 +275,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
   permitted_states = [];
   is_locked = true; may_set_error = true;
   async_kind = None;
+ modifies_fd = false;
   first_version = (0, 0) }
  
  (* Calls.

@@ -1181,6 +1183,7 @@ Return true if option negotiation mode was enabled on this 
handle.";
  default_call with
  args = []; ret = RErr;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "end negotiation and move on to using an export";
  longdesc = "\
  Request that the server finish negotiation and move on to serving the
@@ -1208,6 +1211,7 @@ although older servers will instead have killed the 
connection.";
  default_call with
  args = []; ret = RErr;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "end negotiation and close the connection";
  longdesc = "\
  Request that the server finish negotiation, gracefully if possible, then
@@ -1221,6 +1225,7 @@ enabled option mode.";
  default_call with
  args = []; ret = RBool;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "request the server to initiate TLS";
  longdesc = "\
  Request that the server initiate a secure TLS connection, by
@@ -1259,6 +1264,7 @@ established, as reported by 
L.";
  default_call with
  args = []; ret = RBool;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "request the server to enable structured replies";
  longdesc = "\
  Request that the server use structured replies, by sending
@@ -1285,6 +1291,7 @@ later calls to this function return false.";
  default_call with
  args = [ Closure list_closure ]; ret = RInt;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "request the server to list all exports during negotiation";
  longdesc = "\
  Request that the server list all exports that it supports.  This can
@@ -1326,6 +1333,7 @@ description is set with I<-D>.";
  default_call with
  args = []; ret = RErr;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "request the server for information about an export";
  longdesc = "\
  Request that the server supply information about the export name
@@ -1357,6 +1365,7 @@ corresponding L would succeed.";
  default_call with
  args = [ Closure context_closure ]; ret = RInt;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "list available meta contexts, using implicit query list";
  longdesc = "\
  Request that the server list available meta contexts associated with
@@ -1412,6 +1421,7 @@ a server may send a lengthy list.";
  default_call with
  args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
  permitted_states = [ Negotiating ];
+modifies_fd = true;
  shortdesc = "list available meta contexts, using explicit query list";
  longdesc = "\
  Request that the server list available meta contexts associated with
@@ -1462,6 +1472,7 @@ a server may send a lengthy list.";
  default_call with

Re: [Libguestfs] [libnbd PATCH v4 04/11] rust: Use more specific closure traits

[Tage Johansson]

On 8/2/2023 6:32 PM, Richard W.M. Jones wrote:

On Wed, Aug 02, 2023 at 12:40:49PM +, Tage Johansson wrote:

For closures with `cb,count = CBOnce`, `FnOnce` will be used instead of

I think there's an extra ',' here.

Previous comments about use of markdown apply too.


`FnMut`. Moreover, closures in synchronous commands with
`cblifetime = CBCommand` will not need to live for the static lifetime.
See [here](https://doc.rust-lang.org/std/ops/trait.FnOnce.html) for more
information about the advantages of using `FnOnce` when possible.
---
  generator/Rust.ml  | 44 
  rust/src/handle.rs |  2 ++
  rust/src/types.rs  |  2 ++
  3 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 3117980..d3225eb 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -111,7 +111,7 @@ let rust_cbarg_name : cbarg -> string = function
| CBArrayAndLen (arg, _) | CBMutable arg -> rust_arg_name arg
  
  (* Get the Rust type for an argument. *)

-let rec rust_arg_type : arg -> string = function
+let rec rust_arg_type ?(async_kind = None) : arg -> string = function
| Bool _ -> "bool"
| Int _ -> "c_int"
| UInt _ -> "c_uint"
@@ -131,15 +131,18 @@ let rec rust_arg_type : arg -> string = function
| BytesOut _ -> " [u8]"
| BytesPersistIn _ -> "&'static [u8]"
| BytesPersistOut _ -> "&'static mut [u8]"
-  | Closure { cbargs } -> "impl " ^ rust_closure_trait cbargs
+  | Closure { cbargs; cbcount } -> (
+  match async_kind with
+  | Some _ -> "impl " ^ rust_closure_trait cbargs cbcount
+  | None -> "impl " ^ rust_closure_trait cbargs cbcount ~lifetime:None)
  
  (* Get the Rust closure trait for a callback, That is `Fn*(...) -> ...)`. *)

-and rust_closure_trait ?(lifetime = Some "'static") cbargs : string =
+and rust_closure_trait ?(lifetime = Some "'static") cbargs cbcount : string =
let rust_cbargs = String.concat ", " (List.map rust_cbarg_type cbargs)
-  and lifetime_constraint =
-match lifetime with None -> "" | Some x -> " + " ^ x
-  in
-  "FnMut(" ^ rust_cbargs ^ ") -> c_int + Send + Sync" ^ lifetime_constraint
+  and closure_type =
+match cbcount with CBOnce -> "FnOnce" | CBMany -> "FnMut"
+  and lifetime = match lifetime with None -> "" | Some x -> " + " ^ x in
+  sprintf "%s(%s) -> c_int + Send + Sync%s" closure_type rust_cbargs lifetime
  
  (* Get the Rust type for a callback argument. *)

  and rust_cbarg_type : cbarg -> string = function
@@ -153,8 +156,8 @@ and rust_cbarg_type : cbarg -> string = function
| CBMutable arg -> " " ^ rust_arg_type arg
  
  (* Get the type of a rust optional argument. *)

-let rust_optarg_type : optarg -> string = function
-  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x))
+let rust_optarg_type ?(async_kind = None) : optarg -> string = function
+  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x) ~async_kind)
| OFlags (name, flags, _) ->
sprintf "Option<%s>" (rust_arg_type (Flags (name, flags)))
  
@@ -419,8 +422,8 @@ let ffi_ret_to_rust (call : call) =

 closure data, and a free function for the closure data. This struct is what
 will be sent to a C function taking the closure as an argument. In fact,
 the struct itself is generated by rust-bindgen. *)
-let print_rust_closure_to_raw_fn ({ cbname; cbargs } : closure) =
-  let closure_trait = rust_closure_trait cbargs ~lifetime:None in
+let print_rust_closure_to_raw_fn ({ cbname; cbargs; cbcount } : closure) =
+  let closure_trait = rust_closure_trait cbargs cbcount ~lifetime:None in
let ffi_cbargs_names = List.flatten (List.map ffi_cbarg_names cbargs) in
let ffi_cbargs_types = List.flatten (List.map ffi_cbarg_types cbargs) in
let rust_cbargs_names = List.map rust_cbarg_name cbargs in
@@ -435,16 +438,24 @@ let print_rust_closure_to_raw_fn ({ cbname; cbargs } : 
closure) =
 (List.map2 (sprintf "%s: %s") ffi_cbargs_names ffi_cbargs_types));
pr "  where F: %s\n" closure_trait;
pr "{\n";
-  pr "let callback_ptr = data as *mut F;\n";
-  pr "let callback =  *callback_ptr;\n";
+  (match cbcount with
+  | CBMany ->
+  pr "let callback_ptr = data as *mut F;\n";
+  pr "let callback =  *callback_ptr;\n"
+  | CBOnce ->
+  pr "let callback_ptr = data as *mut Option;\n";
+  pr "let callback_option:  Option =  *callback_ptr;\n";
+  pr "let callback: F = callback_option.take().

[Libguestfs] [libnbd PATCH v4 10/11] rust: async: Add a couple of integration tests

[Tage Johansson]

Add a couple of integration tests as rust/tests/test_async_*.rs. They
are very similar to the tests for the synchronous API.
---
 rust/Cargo.toml   |   1 +
 rust/tests/test_async_100_handle.rs   |  25 +++
 rust/tests/test_async_200_connect_command.rs  |  33 
 rust/tests/test_async_210_opt_abort.rs|  32 
 rust/tests/test_async_220_opt_list.rs |  81 ++
 rust/tests/test_async_230_opt_info.rs | 122 +++
 rust/tests/test_async_240_opt_list_meta.rs| 147 ++
 .../test_async_245_opt_list_meta_queries.rs   |  91 +++
 rust/tests/test_async_250_opt_set_meta.rs | 122 +++
 .../test_async_255_opt_set_meta_queries.rs| 107 +
 rust/tests/test_async_400_pread.rs|  40 +
 rust/tests/test_async_405_pread_structured.rs |  84 ++
 rust/tests/test_async_410_pwrite.rs   |  59 +++
 rust/tests/test_async_460_block_status.rs |  92 +++
 rust/tests/test_async_620_stats.rs|  76 +
 15 files changed, 1112 insertions(+)
 create mode 100644 rust/tests/test_async_100_handle.rs
 create mode 100644 rust/tests/test_async_200_connect_command.rs
 create mode 100644 rust/tests/test_async_210_opt_abort.rs
 create mode 100644 rust/tests/test_async_220_opt_list.rs
 create mode 100644 rust/tests/test_async_230_opt_info.rs
 create mode 100644 rust/tests/test_async_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_async_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_async_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_async_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_async_400_pread.rs
 create mode 100644 rust/tests/test_async_405_pread_structured.rs
 create mode 100644 rust/tests/test_async_410_pwrite.rs
 create mode 100644 rust/tests/test_async_460_block_status.rs
 create mode 100644 rust/tests/test_async_620_stats.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index c7e461a..d7973ff 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -52,6 +52,7 @@ epoll = "4.3.3"
 default = ["log", "tokio"]
 
 [dev-dependencies]
+anyhow = "1.0.72"
 once_cell = "1.18.0"
 pretty-hex = "0.3.0"
 rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
diff --git a/rust/tests/test_async_100_handle.rs 
b/rust/tests/test_async_100_handle.rs
new file mode 100644
index 000..e50bad9
--- /dev/null
+++ b/rust/tests/test_async_100_handle.rs
@@ -0,0 +1,25 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+//! Just check that we can link with libnbd and create a handle.
+
+#![deny(warnings)]
+
+#[tokio::test]
+async fn test_async_nbd_handle_new() {
+let _ = libnbd::AsyncHandle::new().unwrap();
+}
diff --git a/rust/tests/test_async_200_connect_command.rs 
b/rust/tests/test_async_200_connect_command.rs
new file mode 100644
index 000..8a3f497
--- /dev/null
+++ b/rust/tests/test_async_200_connect_command.rs
@@ -0,0 +1,33 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+#![deny(warnings)]
+
+
+#[tokio::test]
+async fn test_async_connect_command() {
+let nbd = libnbd::AsyncHandle::new().unwrap();
+nbd.connect_command(&[
+"nbdkit",
+"-s",
+"--exit-with-parent",
+"-v",
+"null"

[Libguestfs] [libnbd PATCH v4 03/11] generator: Add information about the lifetime of closures

[Tage Johansson]

Add two new fields, `cblifetime` and `cbcount`, to the `closure` type
in generator/API.ml*. `cblifetime` tells if the closure may only be used
for as long as the command is in flight or if the closure may be used
until the handle is destructed. `cbcount` tells whether the closure may
be called many times or just once.

This information is needed in the Rust bindings for:
a) Knowing if the closure trait should be `FnMut` or `FnOnce`
   (see ).
b) Knowing for what lifetime the closure should be valid. A closure that
   may be called after the function invokation has returned must live
   for the `'static` lietime. But static closures are inconveniant for
   the user since they can't effectively borrow any local data. So it is
   good if this restriction is relaxed when it is not needed.
---
 generator/API.ml  | 20 
 generator/API.mli | 17 +
 2 files changed, 37 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index f90a6fa..42b9eec 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -77,6 +77,8 @@ and ret =
 and closure = {
   cbname : string;
   cbargs : cbarg list;
+  cblifetime : cblifetime;
+  cbcount : cbcount
 }
 and cbarg =
 | CBArrayAndLen of arg * string
@@ -87,6 +89,12 @@ and cbarg =
 | CBString of string
 | CBUInt of string
 | CBUInt64 of string
+and cblifetime =
+| CBCommand
+| CBHandle
+and cbcount =
+| CBOnce
+| CBMany
 and enum = {
   enum_prefix : string;
   enums : (string * int) list
@@ -141,20 +149,28 @@ the handle from the NBD protocol handshake."
 (* Closures. *)
 let chunk_closure = {
   cbname = "chunk";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBBytesIn ("subbuf", "count");
  CBUInt64 "offset"; CBUInt "status";
  CBMutable (Int "error") ]
 }
 let completion_closure = {
   cbname = "completion";
+  cblifetime = CBCommand;
+  cbcount = CBOnce;
   cbargs = [ CBMutable (Int "error") ]
 }
 let debug_closure = {
   cbname = "debug";
+  cblifetime = CBHandle;
+  cbcount = CBMany;
   cbargs = [ CBString "context"; CBString "msg" ]
 }
 let extent_closure = {
   cbname = "extent";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "metacontext";
  CBUInt64 "offset";
  CBArrayAndLen (UInt32 "entries",
@@ -163,10 +179,14 @@ let extent_closure = {
 }
 let list_closure = {
   cbname = "list";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name"; CBString "description" ]
 }
 let context_closure = {
   cbname = "context";
+  cblifetime = CBCommand;
+  cbcount = CBMany;
   cbargs = [ CBString "name" ]
 }
 let all_closures = [ chunk_closure; completion_closure;
diff --git a/generator/API.mli b/generator/API.mli
index 361132d..ff85849 100644
--- a/generator/API.mli
+++ b/generator/API.mli
@@ -94,6 +94,12 @@ and ret =
 and closure = {
   cbname : string; (** name of callback function *)
   cbargs : cbarg list; (** all closures return int for now *)
+  (** An upper bound of the lifetime of the closure. Either it will be used for
+  as long as the command is in flight or it may be used until the handle
+  is destructed. *)
+  cblifetime : cblifetime;
+  (** Whether the callback may only be called once or many times. *)
+  cbcount : cbcount;
 }
 and cbarg =
 | CBArrayAndLen of arg * string (** array + number of entries *)
@@ -104,6 +110,17 @@ and cbarg =
 | CBString of string   (** like String *)
 | CBUInt of string (** like UInt *)
 | CBUInt64 of string   (** like UInt64 *)
+and cblifetime =
+| CBCommand (** The closure may only be used until the command is retired.
+(E.G., completion callback or list callback.) *)
+| CBHandle  (** The closure might be used until the handle is descructed.
+(E.G., debug callback.) *)
+and cbcount =
+| CBOnce (** The closure will be used 0 or 1 time if the aio_* call returned an
+ error and exactly once if the call succeeded.
+ (E.g., completion callback.) *)
+| CBMany (** The closure may be used any number of times.
+ (E.g., list callback.) *)
 and enum = {
   enum_prefix : string;(** prefix of each enum variant *)
   enums : (string * int) list (** enum names and their values in C *)
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v4 11/11] rust: Add some examples

[Tage Johansson]

This patch adds a few examples in rust/examples/. The examples are
compiled and run as part of the test suite.
---
 rust/Makefile.am   |   3 +
 rust/examples/concurrent-read-write.rs | 135 +
 rust/examples/connect-command.rs   |  39 +++
 rust/examples/fetch-first-sector.rs|  38 +++
 rust/examples/get-size.rs  |  29 ++
 rust/run-tests.sh  |   7 ++
 6 files changed, 251 insertions(+)
 create mode 100644 rust/examples/concurrent-read-write.rs
 create mode 100644 rust/examples/connect-command.rs
 create mode 100644 rust/examples/fetch-first-sector.rs
 create mode 100644 rust/examples/get-size.rs

diff --git a/rust/Makefile.am b/rust/Makefile.am
index b954b22..d75163d 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -32,6 +32,9 @@ source_files = \
src/types.rs \
src/utils.rs \
src/async_handle.rs \
+   examples/connect-command.rs \
+   examples/get-size.rs \
+   examples/fetch-first-sector.rs \
libnbd-sys/Cargo.toml \
libnbd-sys/build.rs \
$(NULL)
diff --git a/rust/examples/concurrent-read-write.rs 
b/rust/examples/concurrent-read-write.rs
new file mode 100644
index 000..a1c3e8a
--- /dev/null
+++ b/rust/examples/concurrent-read-write.rs
@@ -0,0 +1,135 @@
+//! Example usage with nbdkit:
+//!
+//! nbdkit -U - memory 100M \
+//!   --run 'cargo run --example concurrent-read-write -- $unixsocket'
+//!
+//! This will read and write randomly over the first megabyte of the
+//! plugin using multi-conn, multiple threads and multiple requests in
+//! flight on each thread.
+
+#![deny(warnings)]
+use rand::prelude::*;
+use std::env;
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::task::JoinSet;
+
+/// Number of simultaneous connections to the NBD server.
+///
+/// Note that some servers only support a limited number of
+/// simultaneous connections, and/or have a configurable thread pool
+/// internally, and if you exceed those limits then something will break.
+const NR_MULTI_CONN: usize = 8;
+
+/// Number of commands that can be "in flight" at the same time on each
+/// connection.  (Therefore the total number of requests in flight may
+/// be up to NR_MULTI_CONN * MAX_IN_FLIGHT).
+const MAX_IN_FLIGHT: usize = 16;
+
+/// The size of large reads and writes, must be > 512.
+const BUFFER_SIZE: usize = 1024;
+
+/// Number of commands we issue (per [task][tokio::task]).
+const NR_CYCLES: usize = 32;
+
+/// Statistics gathered during the run.
+#[derive(Debug, Default)]
+struct Stats {
+/// The total number of requests made.
+requests: usize,
+}
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+let args = env::args_os().collect::>();
+if args.len() != 2 {
+anyhow::bail!("Usage: {:?} socket", args[0]);
+}
+let socket = [1];
+
+// We begin by making a connection to the server to get the export size
+// and ensure that it supports multiple connections and is writable.
+let nbd = libnbd::Handle::new()?;
+nbd.connect_unix()?;
+let export_size = nbd.get_size()?;
+anyhow::ensure!(
+(BUFFER_SIZE as u64) < export_size,
+"export is {export_size}B, must be larger than {BUFFER_SIZE}B"
+);
+anyhow::ensure!(
+!nbd.is_read_only()?,
+"error: this NBD export is read-only"
+);
+anyhow::ensure!(
+nbd.can_multi_conn()?,
+"error: this NBD export does not support multi-conn"
+);
+drop(nbd); // Close the connection.
+
+// Start the worker tasks, one per connection.
+let mut tasks = JoinSet::new();
+for i in 0..NR_MULTI_CONN {
+tasks.spawn(run_thread(i, socket.clone().into(), export_size));
+}
+
+// Wait for the tasks to complete.
+let mut stats = Stats::default();
+while !tasks.is_empty() {
+let this_stats = tasks.join_next().await.unwrap().unwrap()?;
+stats.requests += this_stats.requests;
+}
+
+// Make sure the number of requests that were required matches what
+// we expect.
+assert_eq!(stats.requests, NR_MULTI_CONN * NR_CYCLES);
+
+Ok(())
+}
+
+async fn run_thread(
+task_idx: usize,
+socket: PathBuf,
+export_size: u64,
+) -> anyhow::Result {
+// Start a new connection to the server.
+// We shall spawn many commands concurrently on different tasks and those
+// futures must be `'static`, hence we wrap the handle in an [Arc].
+let nbd = Arc::new(libnbd::AsyncHandle::new()?);
+nbd.connect_unix(socket).await?;
+
+let mut rng = SmallRng::seed_from_u64(44 as u64);
+
+// Issue commands.
+let mut stats = Stats::default();
+let mut join_set = JoinSet::new();
+//tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+while stats.requests < NR_CYCLES || !join_set.is_empty() {
+while stats.requests < NR_CYCLES && join_set.len() < MAX_IN_FLIGHT {
+// If we want to issue another request, do 

[Libguestfs] [libnbd PATCH v4 07/11] rust: async: Create an async friendly handle type

[Tage Johansson]

e_call_args call);
+  pr "{\n";
+  print_ffi_call aio_name "self.data.handle.handle" call;
+  pr "?;\n";
+  pr "let (ret_tx, ret_rx) = oneshot::channel::>();\n";
+  pr "let mut ret_tx = Some(ret_tx);\n";
+  pr "let completion_predicate = \n";
+  pr " move |handle: , res: <()>| {\n";
+  pr "  let ret = if let Err(_) = res {\n";
+  pr "res.clone()\n";
+  pr "  } else {\n";
+  pr "if handle.%s() != %s { return false; }\n" predicate value;
+  pr "else { Ok(()) }\n";
+  pr "  };\n";
+  pr "  ret_tx.take().unwrap().send(ret).ok();\n";
+  pr "  true\n";
+  pr "};\n";
+  pr "self.add_command(completion_predicate)?;\n";
+  pr "ret_rx.await.unwrap()\n";
+  pr "}\n\n"
+
+(* Print an impl with all handle calls. *)
+let print_rust_async_handle_impls () =
+  pr "impl AsyncHandle {\n";
+  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
+  NameMap.iter
+(fun name (call, async_kind) ->
+  match async_kind with
+  | WithCompletionCallback ->
+  print_rust_async_handle_call_with_completion_cb name call
+  | ChangesState (predicate, value) ->
+  print_rust_async_handle_call_changing_state name call
+(predicate, value))
+async_handle_calls;
+  pr "}\n\n"
+
+let print_rust_async_imports () =
+  pr "use crate::{*, types::*};\n";
+  pr "use os_str_bytes::OsStringBytes as _;\n";
+  pr "use os_socketaddr::OsSocketAddr;\n";
+  pr "use std::ffi::*;\n";
+  pr "use std::mem;\n";
+  pr "use std::net::SocketAddr;\n";
+  pr "use std::os::fd::{AsRawFd, OwnedFd};\n";
+  pr "use std::path::PathBuf;\n";
+  pr "use std::ptr;\n";
+  pr "use std::sync::Arc;\n";
+  pr "use tokio::sync::oneshot;\n";
+  pr "\n"
+
+let generate_rust_async_bindings () =
+  generate_header CStyle ~copyright:"Tage Johansson";
+  pr "\n";
+  print_rust_async_imports ();
+  print_rust_async_handle_impls ()
diff --git a/generator/Rust.mli b/generator/Rust.mli
index 450e4ca..0960170 100644
--- a/generator/Rust.mli
+++ b/generator/Rust.mli
@@ -18,3 +18,5 @@
 
 (* Print all flag-structs, enums, constants and handle calls in Rust code. *)
 val generate_rust_bindings : unit -> unit
+
+val generate_rust_async_bindings : unit -> unit
diff --git a/generator/generator.ml b/generator/generator.ml
index f5ef7cc..2118446 100644
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -64,3 +64,4 @@ let () =
 
   output_to ~formatter:(Some Rustfmt) "rust/libnbd-sys/src/lib.rs" 
RustSys.generate_rust_sys_bindings;
   output_to ~formatter:(Some Rustfmt) "rust/src/bindings.rs" 
Rust.generate_rust_bindings;
+  output_to ~formatter:(Some Rustfmt) "rust/src/async_bindings.rs" 
Rust.generate_rust_async_bindings;
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index f74c3ac..c7e461a 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -45,10 +45,15 @@ thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
 byte-strings = "0.3.1"
+tokio = { optional = true, version = "1.29.1", default-features = false, 
features = ["rt", "sync", "net"] }
+epoll = "4.3.3"
 
 [features]
-default = ["log"]
+default = ["log", "tokio"]
 
 [dev-dependencies]
 once_cell = "1.18.0"
+pretty-hex = "0.3.0"
+rand = { version = "0.8.5", default-features = false, features = ["small_rng", 
"min_const_gen"] }
 tempfile = "3.6.0"
+tokio = { version = "1.29.1", default-features = false, features = ["macros", 
"rt-multi-thread"] }
diff --git a/rust/Makefile.am b/rust/Makefile.am
index 19dbf02..b954b22 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -19,6 +19,7 @@ include $(top_srcdir)/subdir-rules.mk
 
 generator_built = \
libnbd-sys/src/lib.rs \
+   src/async_bindings.rs \
src/bindings.rs \
$(NULL)
 
@@ -30,6 +31,7 @@ source_files = \
src/handle.rs \
src/types.rs \
src/utils.rs \
+   src/async_handle.rs \
libnbd-sys/Cargo.toml \
libnbd-sys/build.rs \
$(NULL)
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
new file mode 100644
index 000..4223b80
--- /dev/null
+++ b/rust/src/async_handle.rs
@@ -0,0 +1,268 @@
+// nbd client library in userspace
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation;

[Libguestfs] [libnbd PATCH v4 05/11] rust: Add a couple of integration tests

[Tage Johansson]

A couple of integration tests are added in rust/tests. They are mostly
ported from the OCaml tests.
---
 rust/Cargo.toml  |   4 +
 rust/Makefile.am |   1 +
 rust/run-tests.sh|   6 +-
 rust/tests/nbdkit_pattern/mod.rs |  28 
 rust/tests/test_100_handle.rs|  25 
 rust/tests/test_110_defaults.rs  |  33 +
 rust/tests/test_120_set_non_defaults.rs  |  53 +++
 rust/tests/test_130_private_data.rs  |  28 
 rust/tests/test_140_explicit_close.rs|  31 
 rust/tests/test_200_connect_command.rs   |  32 
 rust/tests/test_210_opt_abort.rs |  31 
 rust/tests/test_220_opt_list.rs  |  86 +++
 rust/tests/test_230_opt_info.rs  | 120 +++
 rust/tests/test_240_opt_list_meta.rs | 147 +++
 rust/tests/test_245_opt_list_meta_queries.rs |  93 
 rust/tests/test_250_opt_set_meta.rs  | 123 
 rust/tests/test_255_opt_set_meta_queries.rs  | 109 ++
 rust/tests/test_300_get_size.rs  |  35 +
 rust/tests/test_400_pread.rs |  39 +
 rust/tests/test_405_pread_structured.rs  |  79 ++
 rust/tests/test_410_pwrite.rs|  58 
 rust/tests/test_460_block_status.rs  |  92 
 rust/tests/test_620_stats.rs |  75 ++
 rust/tests/test_log/mod.rs   |  86 +++
 24 files changed, 1412 insertions(+), 2 deletions(-)
 create mode 100644 rust/tests/nbdkit_pattern/mod.rs
 create mode 100644 rust/tests/test_100_handle.rs
 create mode 100644 rust/tests/test_110_defaults.rs
 create mode 100644 rust/tests/test_120_set_non_defaults.rs
 create mode 100644 rust/tests/test_130_private_data.rs
 create mode 100644 rust/tests/test_140_explicit_close.rs
 create mode 100644 rust/tests/test_200_connect_command.rs
 create mode 100644 rust/tests/test_210_opt_abort.rs
 create mode 100644 rust/tests/test_220_opt_list.rs
 create mode 100644 rust/tests/test_230_opt_info.rs
 create mode 100644 rust/tests/test_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_300_get_size.rs
 create mode 100644 rust/tests/test_400_pread.rs
 create mode 100644 rust/tests/test_405_pread_structured.rs
 create mode 100644 rust/tests/test_410_pwrite.rs
 create mode 100644 rust/tests/test_460_block_status.rs
 create mode 100644 rust/tests/test_620_stats.rs
 create mode 100644 rust/tests/test_log/mod.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index e81360b..f74c3ac 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -48,3 +48,7 @@ byte-strings = "0.3.1"
 
 [features]
 default = ["log"]
+
+[dev-dependencies]
+once_cell = "1.18.0"
+tempfile = "3.6.0"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index 5160e9e..251a139 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -58,6 +58,7 @@ TESTS_ENVIRONMENT = \
LIBNBD_DEBUG=1 \
$(MALLOC_CHECKS) \
abs_top_srcdir=$(abs_top_srcdir) \
+   CARGO=$(CARGO) \
$(NULL)
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
diff --git a/rust/run-tests.sh b/rust/run-tests.sh
index 7a0bc85..005000e 100755
--- a/rust/run-tests.sh
+++ b/rust/run-tests.sh
@@ -1,6 +1,6 @@
 #!/bin/bash -
 # nbd client library in userspace
-# Copyright Red Hat
+# Copyright Tage Johansson
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -21,4 +21,6 @@
 set -e
 set -x
 
-cargo test
+requires nbdkit --version
+
+$CARGO test -- --nocapture
diff --git a/rust/tests/nbdkit_pattern/mod.rs b/rust/tests/nbdkit_pattern/mod.rs
new file mode 100644
index 000..5f4069e
--- /dev/null
+++ b/rust/tests/nbdkit_pattern/mod.rs
@@ -0,0 +1,28 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+use once_cell::sync::Lazy;
+
+/// The byte pattern as described in nbdkit-PATT

[Libguestfs] [libnbd PATCH v4 04/11] rust: Use more specific closure traits

[Tage Johansson]

For closures with `cb,count = CBOnce`, `FnOnce` will be used instead of
`FnMut`. Moreover, closures in synchronous commands with
`cblifetime = CBCommand` will not need to live for the static lifetime.
See [here](https://doc.rust-lang.org/std/ops/trait.FnOnce.html) for more
information about the advantages of using `FnOnce` when possible.
---
 generator/Rust.ml  | 44 
 rust/src/handle.rs |  2 ++
 rust/src/types.rs  |  2 ++
 3 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 3117980..d3225eb 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -111,7 +111,7 @@ let rust_cbarg_name : cbarg -> string = function
   | CBArrayAndLen (arg, _) | CBMutable arg -> rust_arg_name arg
 
 (* Get the Rust type for an argument. *)
-let rec rust_arg_type : arg -> string = function
+let rec rust_arg_type ?(async_kind = None) : arg -> string = function
   | Bool _ -> "bool"
   | Int _ -> "c_int"
   | UInt _ -> "c_uint"
@@ -131,15 +131,18 @@ let rec rust_arg_type : arg -> string = function
   | BytesOut _ -> " [u8]"
   | BytesPersistIn _ -> "&'static [u8]"
   | BytesPersistOut _ -> "&'static mut [u8]"
-  | Closure { cbargs } -> "impl " ^ rust_closure_trait cbargs
+  | Closure { cbargs; cbcount } -> (
+  match async_kind with
+  | Some _ -> "impl " ^ rust_closure_trait cbargs cbcount
+  | None -> "impl " ^ rust_closure_trait cbargs cbcount ~lifetime:None)
 
 (* Get the Rust closure trait for a callback, That is `Fn*(...) -> ...)`. *)
-and rust_closure_trait ?(lifetime = Some "'static") cbargs : string =
+and rust_closure_trait ?(lifetime = Some "'static") cbargs cbcount : string =
   let rust_cbargs = String.concat ", " (List.map rust_cbarg_type cbargs)
-  and lifetime_constraint =
-match lifetime with None -> "" | Some x -> " + " ^ x
-  in
-  "FnMut(" ^ rust_cbargs ^ ") -> c_int + Send + Sync" ^ lifetime_constraint
+  and closure_type =
+match cbcount with CBOnce -> "FnOnce" | CBMany -> "FnMut"
+  and lifetime = match lifetime with None -> "" | Some x -> " + " ^ x in
+  sprintf "%s(%s) -> c_int + Send + Sync%s" closure_type rust_cbargs lifetime
 
 (* Get the Rust type for a callback argument. *)
 and rust_cbarg_type : cbarg -> string = function
@@ -153,8 +156,8 @@ and rust_cbarg_type : cbarg -> string = function
   | CBMutable arg -> " " ^ rust_arg_type arg
 
 (* Get the type of a rust optional argument. *)
-let rust_optarg_type : optarg -> string = function
-  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x))
+let rust_optarg_type ?(async_kind = None) : optarg -> string = function
+  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x) ~async_kind)
   | OFlags (name, flags, _) ->
   sprintf "Option<%s>" (rust_arg_type (Flags (name, flags)))
 
@@ -419,8 +422,8 @@ let ffi_ret_to_rust (call : call) =
closure data, and a free function for the closure data. This struct is what
will be sent to a C function taking the closure as an argument. In fact,
the struct itself is generated by rust-bindgen. *)
-let print_rust_closure_to_raw_fn ({ cbname; cbargs } : closure) =
-  let closure_trait = rust_closure_trait cbargs ~lifetime:None in
+let print_rust_closure_to_raw_fn ({ cbname; cbargs; cbcount } : closure) =
+  let closure_trait = rust_closure_trait cbargs cbcount ~lifetime:None in
   let ffi_cbargs_names = List.flatten (List.map ffi_cbarg_names cbargs) in
   let ffi_cbargs_types = List.flatten (List.map ffi_cbarg_types cbargs) in
   let rust_cbargs_names = List.map rust_cbarg_name cbargs in
@@ -435,16 +438,24 @@ let print_rust_closure_to_raw_fn ({ cbname; cbargs } : 
closure) =
(List.map2 (sprintf "%s: %s") ffi_cbargs_names ffi_cbargs_types));
   pr "  where F: %s\n" closure_trait;
   pr "{\n";
-  pr "let callback_ptr = data as *mut F;\n";
-  pr "let callback =  *callback_ptr;\n";
+  (match cbcount with
+  | CBMany ->
+  pr "let callback_ptr = data as *mut F;\n";
+  pr "let callback =  *callback_ptr;\n"
+  | CBOnce ->
+  pr "let callback_ptr = data as *mut Option;\n";
+  pr "let callback_option:  Option =  *callback_ptr;\n";
+  pr "let callback: F = callback_option.take().unwrap();\n");
   List.iter ffi_cbargs_to_rust cbargs;
   pr "callback(%s)\n" (String.concat ", " rust_cbargs_names);
   pr "}\n";
-  pr "let callback_data = Box::into_raw(Box::new(f));\n";
+  pr "let callback_data = Box::into_raw(Box::new(%s));\n"
+(match cbcount with CBMany -> "f" | CBOnce -> "Some(f)");
   pr "sys::nbd_%s_callback {\n" cbname;
   pr "callback: Some(call_closure::),\n";
   pr "user_data: callback_data as *mut _,\n";
-  pr "free: Some(utils::drop_data::),\n";
+  pr "free: Some(utils::drop_data::<%s>),\n"
+(match cbcount with CBMany -> "F" | CBOnce -> "Option");
   pr "}\n";
   pr "}\n";
   pr "\n"
@@ -501,7 +512,8 @@ let 

[Libguestfs] [libnbd PATCH v4 08/11] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

Add a flag (`modifies_fd`) to the [call] structure in generator/API.ml
which is set to [true] if the handle call may do something with the
file descriptor. That is, it is [true] for all calls which are or may
call [aio_notify_*], including all synchronous commands like
[nbd_connect] or [nbd_opt_go].

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to [true] will be
excluded from that handle. The asynchronous (`aio_*`) functions will be
used instead.
---
 generator/API.ml  | 32 
 generator/API.mli |  7 +++
 2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 42b9eec..6e1670d 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
   is_locked : bool;
   may_set_error : bool;
   async_kind : async_kind option;
+  modifies_fd: bool;
   mutable first_version : int * int;
 }
 and arg =
@@ -274,6 +275,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  permitted_states = [];
  is_locked = true; may_set_error = true;
  async_kind = None;
+ modifies_fd = false;
  first_version = (0, 0) }
 
 (* Calls.
@@ -1181,6 +1183,7 @@ Return true if option negotiation mode was enabled on 
this handle.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -1208,6 +1211,7 @@ although older servers will instead have killed the 
connection.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 Request that the server finish negotiation, gracefully if possible, then
@@ -1221,6 +1225,7 @@ enabled option mode.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to initiate TLS";
 longdesc = "\
 Request that the server initiate a secure TLS connection, by
@@ -1259,6 +1264,7 @@ established, as reported by 
L.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to enable structured replies";
 longdesc = "\
 Request that the server use structured replies, by sending
@@ -1285,6 +1291,7 @@ later calls to this function return false.";
 default_call with
 args = [ Closure list_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to list all exports during negotiation";
 longdesc = "\
 Request that the server list all exports that it supports.  This can
@@ -1326,6 +1333,7 @@ description is set with I<-D>.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server for information about an export";
 longdesc = "\
 Request that the server supply information about the export name
@@ -1357,6 +1365,7 @@ corresponding L would succeed.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using implicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1412,6 +1421,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using explicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1462,6 +1472,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using implicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1521,6 +1532,7 @@ no contexts are reported, or may fail but have a 
non-empty list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using explicit query list";
 longdesc = "\
 Request that the server supply all recognized meta 

[Libguestfs] [libnbd PATCH v4 02/11] generator: Add information about asynchronous handle calls

[Tage Johansson]

A new field `async_kind` is added to the `call` data type in
generator/API.ml*. The purpose is to tell if a certain handle call is
an asynchronous command and if so how one can know when it is
completed. The motivation for this is that all asynchronous commands
on the `AsyncHandle` in the Rust bindings makes use of Rust's
[`async fn`s](https://doc.rust-lang.org/std/keyword.async.html).
But to implement such an `async fn`, the API needs to know when the
command completed, either by a completion callback or via a change
of state.
---
 generator/API.ml  | 32 
 generator/API.mli | 11 +++
 2 files changed, 43 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 5fcb0e1..f90a6fa 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -32,6 +32,7 @@ type call = {
   permitted_states : permitted_state list;
   is_locked : bool;
   may_set_error : bool;
+  async_kind : async_kind option;
   mutable first_version : int * int;
 }
 and arg =
@@ -102,6 +103,9 @@ and permitted_state =
 | Negotiating
 | Connected
 | Closed | Dead
+and async_kind =
+| WithCompletionCallback
+| ChangesState of string * bool
 and link =
 | Link of string
 | SectionLink of string
@@ -249,6 +253,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  see_also = [];
  permitted_states = [];
  is_locked = true; may_set_error = true;
+ async_kind = None;
  first_version = (0, 0) }
 
 (* Calls.
@@ -2798,6 +2803,7 @@ wait for an L.";
 default_call with
 args = [ SockAddrAndLen ("addr", "addrlen") ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Begin connecting to the NBD server.  The C and C
@@ -2810,6 +2816,7 @@ parameters specify the address of the socket to connect 
to.
 default_call with
 args = [ String "uri" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to an NBD URI";
 longdesc = "\
 Begin connecting to the NBD URI C.  Parameters behave as
@@ -2823,6 +2830,7 @@ documented in L.
 default_call with
 args = [ Path "unixsocket" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a Unix domain socket";
 longdesc = "\
 Begin connecting to the NBD server over Unix domain socket
@@ -2837,6 +2845,7 @@ L.
 default_call with
 args = [ UInt32 "cid"; UInt32 "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over AF_VSOCK socket";
 longdesc = "\
 Begin connecting to the NBD server over the C
@@ -2850,6 +2859,7 @@ L.
 default_call with
 args = [ String "hostname"; String "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a TCP port";
 longdesc = "\
 Begin connecting to the NBD server listening on C.
@@ -2862,6 +2872,7 @@ Parameters behave as documented in L.
 default_call with
 args = [ Fd "sock" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect directly to a connected socket";
 longdesc = "\
 Begin connecting to the connected socket C.
@@ -2874,6 +2885,7 @@ Parameters behave as documented in 
L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it over
@@ -2887,6 +2899,7 @@ L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect using systemd socket activation";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it using
@@ -2903,6 +2916,7 @@ L.
 optargs = [ OClosure completion_closure ];
 ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some WithCompletionCallback;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -2926,6 +2940,7 @@ when L returns true.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 

[Libguestfs] [libnbd PATCH v4 09/11] rust: async: Use the `modifies_fd` flag to exclude calls

[Tage Johansson]

All handle calls which has the `modifies_fd` flag set to [true] will be
excluded from `AsyncHandle` (the asynchronous handle in the rust
bindings). This is a better approach then listing all calls that should
be excluded in Rust.ml explicetly.
---
 generator/Rust.ml | 63 ++-
 1 file changed, 30 insertions(+), 33 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index dc82d46..5d26c73 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -567,19 +567,21 @@ let generate_rust_bindings () =
 (*)
 
 let excluded_handle_calls : NameSet.t =
+  let modifies_fd =
+handle_calls
+|> List.filter (fun (_, { modifies_fd }) -> modifies_fd)
+|> List.map (fun (name, _) -> name)
+  in
   NameSet.of_list
-[
-  "aio_get_fd";
-  "aio_get_direction";
-  "aio_notify_read";
-  "aio_notify_write";
-  "clear_debug_callback";
-  "get_debug";
-  "poll";
-  "poll2";
-  "set_debug";
-  "set_debug_callback";
-]
+(modifies_fd
+@ [
+"aio_get_fd";
+"aio_get_direction";
+"clear_debug_callback";
+"get_debug";
+"set_debug";
+"set_debug_callback";
+  ])
 
 (* A mapping with names as keys. *)
 module NameMap = Map.Make (String)
@@ -590,16 +592,16 @@ let strip_aio name : string =
 String.sub name 4 (String.length name - 4)
   else failwithf "Asynchronous call %s must begin with aio_" name
 
-(* A map with all asynchronous handle calls. The keys are names with "aio_"
-   stripped, the values are a tuple with the actual name (with "aio_"), the
-   [call] and the [async_kind]. *)
+(* A map with all asynchronous handle calls. The keys are names, the values
+   are tuples of: the name with "aio_" stripped, the [call] and the
+   [async_kind]. *)
 let async_handle_calls : ((string * call) * async_kind) NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
   |> List.filter_map (fun (name, call) ->
  call.async_kind
  |> Option.map (fun async_kind ->
-(strip_aio name, ((name, call), async_kind
+(name, ((strip_aio name, call), async_kind
   |> List.to_seq |> NameMap.of_seq
 
 (* A mapping with all synchronous (not asynchronous) handle calls. Excluded
@@ -609,11 +611,7 @@ let async_handle_calls : ((string * call) * async_kind) 
NameMap.t =
 let sync_handle_calls : call NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
-  |> List.filter (fun (name, _) ->
- (not (NameMap.mem name async_handle_calls))
- && not
-  (String.starts_with ~prefix:"aio_" name
-  && NameMap.mem (strip_aio name) async_handle_calls))
+  |> List.filter (fun (n, _) -> not (NameMap.mem n async_handle_calls))
   |> List.to_seq |> NameMap.of_seq
 
 (* Get the Rust type for an argument in the asynchronous API. Like
@@ -658,7 +656,7 @@ let print_rust_sync_handle_call (name : string) (call : 
call) =
 (* Print the Rust function for an asynchronous handle call with a completion
callback. (Note that "callback" might be abbreviated with "cb" in the
following code. *)
-let print_rust_async_handle_call_with_completion_cb name (aio_name, call) =
+let print_rust_async_handle_call_with_completion_cb aio_name (name, call) =
   (* An array of all optional arguments. Useful because we need to deel with
  the index of the completion callback. *)
   let optargs = Array.of_list call.optargs in
@@ -738,7 +736,7 @@ let print_rust_async_handle_call_with_completion_cb name 
(aio_name, call) =
completion by changing state. The predicate is a call like
"aio_is_connecting" which should get the value (like false) for the call to
be complete. *)
-let print_rust_async_handle_call_changing_state name (aio_name, call)
+let print_rust_async_handle_call_changing_state aio_name (name, call)
 (predicate, value) =
   let value = if value then "true" else "false" in
   print_rust_handle_call_comment call;
@@ -767,16 +765,15 @@ let print_rust_async_handle_call_changing_state name 
(aio_name, call)
 (* Print an impl with all handle calls. *)
 let print_rust_async_handle_impls () =
   pr "impl AsyncHandle {\n";
-  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
-  NameMap.iter
-(fun name (call, async_kind) ->
-  match async_kind with
-  | WithCompletionCallback ->
-  print_rust_async_handle_call_with_completion_cb name call
-  | ChangesState (predicate, value) ->
-  print_rust_async_handle_call_changing_state name call
-(predicate, value))
-async_handle_calls;
+  sync_handle_calls |> NameMap.iter print_rust_sync_handle_call;
+  async_handle_calls
+  |> NameMap.iter (fun name (call, async_kind) ->
+ match async_kind with
+ | WithCompletionCallback ->
+ 

[Libguestfs] [libnbd PATCH v4 06/11] rust: Make it possible to run tests with Valgrind

[Tage Johansson]

Make it possible to run Rust tests with Valgrind with
`make check-valgrind` in the rust directory.
---
 rust/Makefile.am  | 3 +++
 rust/run-tests.sh | 6 +-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/rust/Makefile.am b/rust/Makefile.am
index 251a139..19dbf02 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -63,6 +63,9 @@ TESTS_ENVIRONMENT = \
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
 
+check-valgrind:
+   LIBNBD_VALGRIND=1 $(MAKE) check
+
 endif
 
 clean-local:
diff --git a/rust/run-tests.sh b/rust/run-tests.sh
index 005000e..da7852a 100755
--- a/rust/run-tests.sh
+++ b/rust/run-tests.sh
@@ -23,4 +23,8 @@ set -x
 
 requires nbdkit --version
 
-$CARGO test -- --nocapture
+if [ -z "$VG" ]; then
+$CARGO test -- --nocapture
+else
+$CARGO test --config "target.'cfg(all())'.runner = \"$VG\"" -- --nocapture
+fi
-- 
2.41.0

___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH v3 01/10] rust: create basic Rust bindings

[Tage Johansson]

On 7/27/2023 9:43 AM, Richard W.M. Jones wrote:

On Wed, Jul 26, 2023 at 09:58:03AM +, Tage Johansson wrote:

I have never used rpm, but if you are using Fedora, the following command
should be enough:

     dnf install clang-devel

We need to work with whatever environment we find ourselves in.  We
can't just fail because of a missing dependency, but instead should
disable the rust bindings (we can however fail at ./configure time if
--enable-rust was given explicitly but is not possible).

This is the reason for having ./configure at all.


According to this link.


 I seem to remember in some project we had a configure-time test to
 check if rust/cargo was working, compiling a simple test program, but
 I can't find that right now.

 Rich.


That was actually this project. I removed it because we both thought it was
unnecessary. And I actually think it wouldn't catch this problem anyway because
this is directly related to rust-bindgen.

Since this seems to be caused by rust-bindgen, which we don't need,
can we get rid of bindgen now?



I'll work on it this week.  I found a bug in the asynchronous API which 
took most of my time previous week. Think I have solved it now, but the 
implementation is slightly changed.



Best regards,

Tage



Rich.



___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

Re: [Libguestfs] [libnbd PATCH 04/10] generator: Add information about the lifetime of closures

[Tage Johansson]

On 7/25/2023 8:03 PM, Stefan Hajnoczi wrote:

On Mon, Jul 24, 2023 at 09:04:15PM +0100, Richard W.M. Jones wrote:

On Mon, Jul 24, 2023 at 03:22:56PM -0400, Stefan Hajnoczi wrote:

On Fri, Jul 21, 2023 at 11:37:09AM +0100, Richard W.M. Jones wrote:

On Fri, Jul 21, 2023 at 10:13:02AM +, Tage Johansson wrote:

On 7/19/2023 4:35 PM, Richard W.M. Jones wrote:

On Wed, Jul 19, 2023 at 09:09:48AM +, Tage Johansson wrote:

Add a new field `cbkind` to the `closure` type in generator/API.ml*.
It tells how many times the closure may be invoked and for how long time
it might be used. More specifically, it can take any of these values:
- `CBOnceCommand`: The closure may only be called once and shall not
   be called after the command is retired.
- `CBManyCommand`: The closure may be called any number of times but
   not after the command is retired.
- `CBManyHandle`: The closure may be called any number of times before
   the handle is destructed.
This information is needed in the Rust bindings for:
a) Knowing if the closure trait should be `FnMut` or `FnOnce`
(see<https://doc.rust-lang.org/std/ops/trait.FnOnce.html>).
b) Knowing for what lifetime the closure should be valid. A closure that
may be called after the function invokation has returned must live
for the `'static` lifetime. But static closures are inconveniant for
the user since they can't effectively borrow any local data. So it is
good if this restriction is relaxed when it is not needed.
---
  generator/API.ml  | 11 +++
  generator/API.mli | 13 +
  2 files changed, 24 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index f90a6fa..086b2f9 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -77,6 +77,7 @@ and ret =
  and closure = {
cbname : string;
cbargs : cbarg list;
+  cbkind : cbkind;

I'm dubious about the premise of this patch, but let's at least call
it "cblifetime" since that's what it is expressing.


The difference in code for the user might be something like the following:

With only static lifetimes, a call to `opt_list` might look like this:

```Rust

use std::sync::{Arc, Mutex};    // Collect all exports in this list.


// Collect all exports in this list.
let exports = Arc::new(Mutex::new(Vec::new()));
let exports_clone = exports.clone();
let count = nbd.opt_list(move |name, _| {
     exports_clone.lock().unwrap().push(name.to_owned());
     0
})?;
let exports = Arc::into_inner(exports).unwrap().into_inner().unwrap();
assert_eq!(export.as_c_str(), expected);
```


And with custom lifetimes:

```Rust

// Collect all exports in this list.
let mut exports = Vec::new();
let count = nbd.opt_list(|name, _| {
     exports.push(name.to_owned());
     0
})?;
assert_eq!(exports.len(), count as usize);
```


Not only is the latter shorter and easier to read, it is also more
efficient. But it is not strictly necessary, and I can remove it if
you want.

Stefan - any thoughts on this?

 From my point of view the issue is that attempting to categorize
libnbd callbacks according to their lifetime complicates the API
description and might shut down (or make complicated) future more
complex patterns of callback use.

The performance issue is not very critical given that we're already
going through a C library to Rust layer.  A reference count doesn't
seem like a big deal to me.

If the generated Rust API involves closures then dealing with Fn,
FnOnce, FnMut is necessary.

It may be more natural to use the Iterator trait or other Rust features
instead of closures in some cases. Doing so might allow you to avoid
dealing with FnOnce, Fn, and FnMut while also making the Rust API nicer.

Are the generated API docs available somewhere so I can get an
understanding of the Rust API?

I don't think we're publishing those for Rust yet.  Tage ..?

The C API docs are published.  An example might be the
nbd_block_status API where (because we get a potentially long list of
extents from the server, and we get them asynchronously) we call back
to a function that the caller provides:

   https://libguestfs.org/nbd_block_status.3.html

   
https://gitlab.com/nbdkit/libnbd/-/blob/5c2fc3cc7e14146d000b65b191e70d9a0585a395/info/map.c#L73
   
https://gitlab.com/nbdkit/libnbd/-/blob/5c2fc3cc7e14146d000b65b191e70d9a0585a395/info/map.c#L108

This is how OCaml binds it ('fun meta _ entries err ->' is the
callback closure):

   https://gitlab.com/nbdkit/libnbd/-/blob/master/ocaml/examples/extents.ml#L17

Not all callbacks work like nbd_block_status though.  It is possible
to register a callback which can be called much later, after the
function that registered it has returned.

In C each callback has an associated '.free' function.  This is
guaranteed to be called exactly once, after the callback itself will
no longer be called.  It can be used to free the callback data; or for
GC'd bindings like OCaml, to dereference the global root so it will be
garbage collected.

General di

Re: [Libguestfs] [libnbd PATCH 04/10] generator: Add information about the lifetime of closures

[Tage Johansson]

On 7/24/2023 9:22 PM, Stefan Hajnoczi wrote:

On Fri, Jul 21, 2023 at 11:37:09AM +0100, Richard W.M. Jones wrote:

On Fri, Jul 21, 2023 at 10:13:02AM +, Tage Johansson wrote:

On 7/19/2023 4:35 PM, Richard W.M. Jones wrote:

On Wed, Jul 19, 2023 at 09:09:48AM +, Tage Johansson wrote:

Add a new field `cbkind` to the `closure` type in generator/API.ml*.
It tells how many times the closure may be invoked and for how long time
it might be used. More specifically, it can take any of these values:
- `CBOnceCommand`: The closure may only be called once and shall not
   be called after the command is retired.
- `CBManyCommand`: The closure may be called any number of times but
   not after the command is retired.
- `CBManyHandle`: The closure may be called any number of times before
   the handle is destructed.
This information is needed in the Rust bindings for:
a) Knowing if the closure trait should be `FnMut` or `FnOnce`
(see <https://doc.rust-lang.org/std/ops/trait.FnOnce.html>).
b) Knowing for what lifetime the closure should be valid. A closure that
may be called after the function invokation has returned must live
for the `'static` lifetime. But static closures are inconveniant for
the user since they can't effectively borrow any local data. So it is
good if this restriction is relaxed when it is not needed.
---
  generator/API.ml  | 11 +++
  generator/API.mli | 13 +
  2 files changed, 24 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index f90a6fa..086b2f9 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -77,6 +77,7 @@ and ret =
  and closure = {
cbname : string;
cbargs : cbarg list;
+  cbkind : cbkind;

I'm dubious about the premise of this patch, but let's at least call
it "cblifetime" since that's what it is expressing.


The difference in code for the user might be something like the following:

With only static lifetimes, a call to `opt_list` might look like this:

```Rust

use std::sync::{Arc, Mutex};    // Collect all exports in this list.


// Collect all exports in this list.
let exports = Arc::new(Mutex::new(Vec::new()));
let exports_clone = exports.clone();
let count = nbd.opt_list(move |name, _| {
     exports_clone.lock().unwrap().push(name.to_owned());
     0
})?;
let exports = Arc::into_inner(exports).unwrap().into_inner().unwrap();
assert_eq!(export.as_c_str(), expected);
```


And with custom lifetimes:

```Rust

// Collect all exports in this list.
let mut exports = Vec::new();
let count = nbd.opt_list(|name, _| {
     exports.push(name.to_owned());
     0
})?;
assert_eq!(exports.len(), count as usize);
```


Not only is the latter shorter and easier to read, it is also more
efficient. But it is not strictly necessary, and I can remove it if
you want.

Stefan - any thoughts on this?

 From my point of view the issue is that attempting to categorize
libnbd callbacks according to their lifetime complicates the API
description and might shut down (or make complicated) future more
complex patterns of callback use.

The performance issue is not very critical given that we're already
going through a C library to Rust layer.  A reference count doesn't
seem like a big deal to me.

If the generated Rust API involves closures then dealing with Fn,
FnOnce, FnMut is necessary.

It may be more natural to use the Iterator trait or other Rust features
instead of closures in some cases. Doing so might allow you to avoid
dealing with FnOnce, Fn, and FnMut while also making the Rust API nicer.

Are the generated API docs available somewhere so I can get an
understanding of the Rust API?

Thanks,
Stefan



The make script generates the docs. After running make the docs is 
availlable from rust/target/doc/libnbd/index.html.


Just remember that you currently need LLVM installed for the bindings to 
build.



Best regards,

Tage


___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs

[Libguestfs] [libnbd PATCH v3 10/10] rust: async: Use the `modifies_fd` flag to exclude calls

[Tage Johansson]

All handle calls which has the `modifies_fd` flag set to [true] will be
excluded from `AsyncHandle` (the asynchronous handle in the rust
bindings). This is a better approach then listing all calls that should
be excluded in Rust.ml explicetly.
---
 generator/Rust.ml | 63 ++-
 1 file changed, 30 insertions(+), 33 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index adc7ffa..4d81e3f 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -597,19 +597,21 @@ let generate_rust_bindings () =
 (*)
 
 let excluded_handle_calls : NameSet.t =
+  let modifies_fd =
+handle_calls
+|> List.filter (fun (_, { modifies_fd }) -> modifies_fd)
+|> List.map (fun (name, _) -> name)
+  in
   NameSet.of_list
-[
-  "aio_get_fd";
-  "aio_get_direction";
-  "aio_notify_read";
-  "aio_notify_write";
-  "clear_debug_callback";
-  "get_debug";
-  "poll";
-  "poll2";
-  "set_debug";
-  "set_debug_callback";
-]
+(modifies_fd
+@ [
+"aio_get_fd";
+"aio_get_direction";
+"clear_debug_callback";
+"get_debug";
+"set_debug";
+"set_debug_callback";
+  ])
 
 (* A mapping with names as keys. *)
 module NameMap = Map.Make (String)
@@ -620,16 +622,16 @@ let strip_aio name : string =
 String.sub name 4 (String.length name - 4)
   else failwithf "Asynchronous call %s must begin with aio_" name
 
-(* A map with all asynchronous handle calls. The keys are names with "aio_"
-   stripped, the values are a tuple with the actual name (with "aio_"), the
-   [call] and the [async_kind]. *)
+(* A map with all asynchronous handle calls. The keys are names, the values
+   are tuples of: the name with "aio_" stripped, the [call] and the
+   [async_kind]. *)
 let async_handle_calls : ((string * call) * async_kind) NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
   |> List.filter_map (fun (name, call) ->
  call.async_kind
  |> Option.map (fun async_kind ->
-(strip_aio name, ((name, call), async_kind
+(name, ((strip_aio name, call), async_kind
   |> List.to_seq |> NameMap.of_seq
 
 (* A mapping with all synchronous (not asynchronous) handle calls. Excluded
@@ -639,11 +641,7 @@ let async_handle_calls : ((string * call) * async_kind) 
NameMap.t =
 let sync_handle_calls : call NameMap.t =
   handle_calls
   |> List.filter (fun (n, _) -> not (NameSet.mem n excluded_handle_calls))
-  |> List.filter (fun (name, _) ->
- (not (NameMap.mem name async_handle_calls))
- && not
-  (String.starts_with ~prefix:"aio_" name
-  && NameMap.mem (strip_aio name) async_handle_calls))
+  |> List.filter (fun (n, _) -> not (NameMap.mem n async_handle_calls))
   |> List.to_seq |> NameMap.of_seq
 
 (* Get the Rust type for an argument in the asynchronous API. Like
@@ -688,7 +686,7 @@ let print_rust_sync_handle_call (name : string) (call : 
call) =
 (* Print the Rust function for an asynchronous handle call with a completion
callback. (Note that "callback" might be abbreviated with "cb" in the
following code. *)
-let print_rust_async_handle_call_with_completion_cb name (aio_name, call) =
+let print_rust_async_handle_call_with_completion_cb aio_name (name, call) =
   (* An array of all optional arguments. Useful because we need to deel with
  the index of the completion callback. *)
   let optargs = Array.of_list call.optargs in
@@ -760,7 +758,7 @@ let print_rust_async_handle_call_with_completion_cb name 
(aio_name, call) =
completion by changing state. The predicate is a call like
"aio_is_connecting" which should get the value (like false) for the call to
be complete. *)
-let print_rust_async_handle_call_changing_state name (aio_name, call)
+let print_rust_async_handle_call_changing_state aio_name (name, call)
 (predicate, value) =
   let value = if value then "true" else "false" in
   print_rust_handle_call_comment call;
@@ -784,16 +782,15 @@ let print_rust_async_handle_call_changing_state name 
(aio_name, call)
 (* Print an impl with all handle calls. *)
 let print_rust_async_handle_impls () =
   pr "impl AsyncHandle {\n";
-  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
-  NameMap.iter
-(fun name (call, async_kind) ->
-  match async_kind with
-  | WithCompletionCallback ->
-  print_rust_async_handle_call_with_completion_cb name call
-  | ChangesState (predicate, value) ->
-  print_rust_async_handle_call_changing_state name call
-(predicate, value))
-async_handle_calls;
+  sync_handle_calls |> NameMap.iter print_rust_sync_handle_call;
+  async_handle_calls
+  |> NameMap.iter (fun name (call, async_kind) ->
+ match async_kind with
+ | WithCompletionCallback ->
+ 

[Libguestfs] [libnbd PATCH v3 02/10] generator: Add information about asynchronous handle calls

[Tage Johansson]

A new field `async_kind` is added to the `call` data type in
generator/API.ml*. The purpose is to tell if a certain handle call is
an asynchronous command and if so how one can know when it is
completed. The motivation for this is that all asynchronous commands
on the `AsyncHandle` in the Rust bindings makes use of Rust's
[`async fn`s](https://doc.rust-lang.org/std/keyword.async.html).
But to implement such an `async fn`, the API needs to know when the
command completed, either by a completion callback or via a change
of state.
---
 generator/API.ml  | 32 
 generator/API.mli | 11 +++
 2 files changed, 43 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 5fcb0e1..f90a6fa 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -32,6 +32,7 @@ type call = {
   permitted_states : permitted_state list;
   is_locked : bool;
   may_set_error : bool;
+  async_kind : async_kind option;
   mutable first_version : int * int;
 }
 and arg =
@@ -102,6 +103,9 @@ and permitted_state =
 | Negotiating
 | Connected
 | Closed | Dead
+and async_kind =
+| WithCompletionCallback
+| ChangesState of string * bool
 and link =
 | Link of string
 | SectionLink of string
@@ -249,6 +253,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  see_also = [];
  permitted_states = [];
  is_locked = true; may_set_error = true;
+ async_kind = None;
  first_version = (0, 0) }
 
 (* Calls.
@@ -2798,6 +2803,7 @@ wait for an L.";
 default_call with
 args = [ SockAddrAndLen ("addr", "addrlen") ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Begin connecting to the NBD server.  The C and C
@@ -2810,6 +2816,7 @@ parameters specify the address of the socket to connect 
to.
 default_call with
 args = [ String "uri" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to an NBD URI";
 longdesc = "\
 Begin connecting to the NBD URI C.  Parameters behave as
@@ -2823,6 +2830,7 @@ documented in L.
 default_call with
 args = [ Path "unixsocket" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a Unix domain socket";
 longdesc = "\
 Begin connecting to the NBD server over Unix domain socket
@@ -2837,6 +2845,7 @@ L.
 default_call with
 args = [ UInt32 "cid"; UInt32 "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over AF_VSOCK socket";
 longdesc = "\
 Begin connecting to the NBD server over the C
@@ -2850,6 +2859,7 @@ L.
 default_call with
 args = [ String "hostname"; String "port" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server over a TCP port";
 longdesc = "\
 Begin connecting to the NBD server listening on C.
@@ -2862,6 +2872,7 @@ Parameters behave as documented in L.
 default_call with
 args = [ Fd "sock" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect directly to a connected socket";
 longdesc = "\
 Begin connecting to the connected socket C.
@@ -2874,6 +2885,7 @@ Parameters behave as documented in 
L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect to the NBD server";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it over
@@ -2887,6 +2899,7 @@ L.
 default_call with
 args = [ StringList "argv" ]; ret = RErr;
 permitted_states = [ Created ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "connect using systemd socket activation";
 longdesc = "\
 Run the command as a subprocess and begin connecting to it using
@@ -2903,6 +2916,7 @@ L.
 optargs = [ OClosure completion_closure ];
 ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some WithCompletionCallback;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -2926,6 +2940,7 @@ when L returns true.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+async_kind = Some (ChangesState ("aio_is_connecting", false));
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 

[Libguestfs] [libnbd PATCH v3 04/10] rust: Use more specific closure traits

[Tage Johansson]

For closures with `cb,count = CBOnce`, `FnOnce` will be used instead of
`FnMut`. Moreover, closures in synchronous commands with
`cblifetime = CBCommand` will not need to live for the static lifetime.
See [here](https://doc.rust-lang.org/std/ops/trait.FnOnce.html) for more
information about the advantages of using `FnOnce` when possible.
---
 generator/Rust.ml  | 44 
 rust/src/error.rs  | 13 ++---
 rust/src/handle.rs |  2 ++
 rust/src/types.rs  |  2 ++
 4 files changed, 42 insertions(+), 19 deletions(-)

diff --git a/generator/Rust.ml b/generator/Rust.ml
index 8e2ca13..a4b5257 100644
--- a/generator/Rust.ml
+++ b/generator/Rust.ml
@@ -113,7 +113,7 @@ let rust_cbarg_name : cbarg -> string = function
   | CBArrayAndLen (arg, _) | CBMutable arg -> rust_arg_name arg
 
 (* Get the Rust type for an argument. *)
-let rec rust_arg_type : arg -> string = function
+let rec rust_arg_type ?(async_kind = None) : arg -> string = function
   | Bool _ -> "bool"
   | Int _ -> "c_int"
   | UInt _ -> "c_uint"
@@ -133,15 +133,18 @@ let rec rust_arg_type : arg -> string = function
   | BytesOut _ -> " [u8]"
   | BytesPersistIn _ -> "&'static [u8]"
   | BytesPersistOut _ -> "&'static mut [u8]"
-  | Closure { cbargs } -> "impl " ^ rust_closure_trait cbargs
+  | Closure { cbargs; cbcount } -> (
+  match async_kind with
+  | Some _ -> "impl " ^ rust_closure_trait cbargs cbcount
+  | None -> "impl " ^ rust_closure_trait cbargs cbcount ~lifetime:None)
 
 (* Get the Rust closure trait for a callback, That is `Fn*(...) -> ...)`. *)
-and rust_closure_trait ?(lifetime = Some "'static") cbargs : string =
+and rust_closure_trait ?(lifetime = Some "'static") cbargs cbcount : string =
   let rust_cbargs = String.concat ", " (List.map rust_cbarg_type cbargs)
-  and lifetime_constraint =
-match lifetime with None -> "" | Some x -> " + " ^ x
-  in
-  "FnMut(" ^ rust_cbargs ^ ") -> c_int + Send + Sync" ^ lifetime_constraint
+  and closure_type =
+match cbcount with CBOnce -> "FnOnce" | CBMany -> "FnMut"
+  and lifetime = match lifetime with None -> "" | Some x -> " + " ^ x in
+  sprintf "%s(%s) -> c_int + Send + Sync%s" closure_type rust_cbargs lifetime
 
 (* Get the Rust type for a callback argument. *)
 and rust_cbarg_type : cbarg -> string = function
@@ -155,8 +158,8 @@ and rust_cbarg_type : cbarg -> string = function
   | CBMutable arg -> " " ^ rust_arg_type arg
 
 (* Get the type of a rust optional argument. *)
-let rust_optarg_type : optarg -> string = function
-  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x))
+let rust_optarg_type ?(async_kind = None) : optarg -> string = function
+  | OClosure x -> sprintf "Option<%s>" (rust_arg_type (Closure x) ~async_kind)
   | OFlags (name, flags, _) ->
   sprintf "Option<%s>" (rust_arg_type (Flags (name, flags)))
 
@@ -448,8 +451,8 @@ let ffi_ret_to_rust (call : call) =
closure data, and a free function for the closure data. This struct is what
will be sent to a C function taking the closure as an argument. In fact,
the struct itself is generated by rust-bindgen. *)
-let print_rust_closure_to_raw_fn ({ cbname; cbargs } : closure) =
-  let closure_trait = rust_closure_trait cbargs ~lifetime:None in
+let print_rust_closure_to_raw_fn ({ cbname; cbargs; cbcount } : closure) =
+  let closure_trait = rust_closure_trait cbargs cbcount ~lifetime:None in
   let ffi_cbargs_names = List.flatten (List.map ffi_cbarg_names cbargs) in
   let ffi_cbargs_types = List.flatten (List.map ffi_cbarg_types cbargs) in
   let rust_cbargs_names = List.map rust_cbarg_name cbargs in
@@ -464,16 +467,24 @@ let print_rust_closure_to_raw_fn ({ cbname; cbargs } : 
closure) =
(List.map2 (sprintf "%s: %s") ffi_cbargs_names ffi_cbargs_types));
   pr "  where F: %s\n" closure_trait;
   pr "{\n";
-  pr "let callback_ptr = data as *mut F;\n";
-  pr "let callback =  *callback_ptr;\n";
+  (match cbcount with
+  | CBMany ->
+  pr "let callback_ptr = data as *mut F;\n";
+  pr "let callback =  *callback_ptr;\n"
+  | CBOnce ->
+  pr "let callback_ptr = data as *mut Option;\n";
+  pr "let callback_option:  Option =  *callback_ptr;\n";
+  pr "let callback: F = callback_option.take().unwrap();\n");
   List.iter ffi_cbargs_to_rust cbargs;
   pr "callback(%s)\n" (String.concat ", " rust_cbargs_names);
   pr "}\n";
-  pr "let callback_data = Box::into_raw(Box::new(f));\n";
+  pr "let callback_data = Box::into_raw(Box::new(%s));\n"
+(match cbcount with CBMany -> "f" | CBOnce -> "Some(f)");
   pr "sys::nbd_%s_callback {\n" cbname;
   pr "callback: Some(call_closure::),\n";
   pr "user_data: callback_data as *mut _,\n";
-  pr "free: Some(utils::drop_data::),\n";
+  pr "free: Some(utils::drop_data::<%s>),\n"
+(match cbcount with CBMany -> "F" | CBOnce -> "Option");
   pr "}\n";
   pr "}\n";
 

[Libguestfs] [libnbd PATCH v3 09/10] generator: Add `modifies_fd` flag to the [call] structure

[Tage Johansson]

Add a flag (`modifies_fd`) to the [call] structure in generator/API.ml
which is set to [true] if the handle call may do something with the
file descriptor. That is, it is [true] for all calls which are or may
call [aio_notify_*], including all synchronous commands like
[nbd_connect] or [nbd_opt_go].

The motivation for this is that the asynchronous handle in the Rust
bindings uses its own loop for polling, modifying the file descriptor
outside of this loop may cause unexpected behaviour. Including that the
handle may hang. All commands which set this flag to [true] will be
excluded from that handle. The asynchronous (`aio_*`) functions will be
used instead.
---
 generator/API.ml  | 32 
 generator/API.mli |  7 +++
 2 files changed, 39 insertions(+)

diff --git a/generator/API.ml b/generator/API.ml
index 42b9eec..6e1670d 100644
--- a/generator/API.ml
+++ b/generator/API.ml
@@ -33,6 +33,7 @@ type call = {
   is_locked : bool;
   may_set_error : bool;
   async_kind : async_kind option;
+  modifies_fd: bool;
   mutable first_version : int * int;
 }
 and arg =
@@ -274,6 +275,7 @@ let default_call = { args = []; optargs = []; ret = RErr;
  permitted_states = [];
  is_locked = true; may_set_error = true;
  async_kind = None;
+ modifies_fd = false;
  first_version = (0, 0) }
 
 (* Calls.
@@ -1181,6 +1183,7 @@ Return true if option negotiation mode was enabled on 
this handle.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and move on to using an export";
 longdesc = "\
 Request that the server finish negotiation and move on to serving the
@@ -1208,6 +1211,7 @@ although older servers will instead have killed the 
connection.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "end negotiation and close the connection";
 longdesc = "\
 Request that the server finish negotiation, gracefully if possible, then
@@ -1221,6 +1225,7 @@ enabled option mode.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to initiate TLS";
 longdesc = "\
 Request that the server initiate a secure TLS connection, by
@@ -1259,6 +1264,7 @@ established, as reported by 
L.";
 default_call with
 args = []; ret = RBool;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to enable structured replies";
 longdesc = "\
 Request that the server use structured replies, by sending
@@ -1285,6 +1291,7 @@ later calls to this function return false.";
 default_call with
 args = [ Closure list_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server to list all exports during negotiation";
 longdesc = "\
 Request that the server list all exports that it supports.  This can
@@ -1326,6 +1333,7 @@ description is set with I<-D>.";
 default_call with
 args = []; ret = RErr;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "request the server for information about an export";
 longdesc = "\
 Request that the server supply information about the export name
@@ -1357,6 +1365,7 @@ corresponding L would succeed.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using implicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1412,6 +1421,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "list available meta contexts, using explicit query list";
 longdesc = "\
 Request that the server list available meta contexts associated with
@@ -1462,6 +1472,7 @@ a server may send a lengthy list.";
 default_call with
 args = [ Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using implicit query list";
 longdesc = "\
 Request that the server supply all recognized meta contexts
@@ -1521,6 +1532,7 @@ no contexts are reported, or may fail but have a 
non-empty list.";
 default_call with
 args = [ StringList "queries"; Closure context_closure ]; ret = RInt;
 permitted_states = [ Negotiating ];
+modifies_fd = true;
 shortdesc = "select specific meta contexts, using explicit query list";
 longdesc = "\
 Request that the server supply all recognized meta 

[Libguestfs] [libnbd PATCH v3 05/10] rust: Add a couple of integration tests

[Tage Johansson]

A couple of integration tests are added in rust/tests. They are mostly
ported from the OCaml tests.
---
 rust/Cargo.toml  |   4 +
 rust/Makefile.am |   1 +
 rust/run-tests.sh|   6 +-
 rust/tests/nbdkit_pattern/mod.rs |  28 
 rust/tests/test_100_handle.rs|  25 +++
 rust/tests/test_110_defaults.rs  |  33 
 rust/tests/test_120_set_non_defaults.rs  |  56 +++
 rust/tests/test_130_private_data.rs  |  28 
 rust/tests/test_140_explicit_close.rs|  31 
 rust/tests/test_200_connect_command.rs   |  33 
 rust/tests/test_210_opt_abort.rs |  39 +
 rust/tests/test_220_opt_list.rs  |  85 +++
 rust/tests/test_230_opt_info.rs  | 124 +++
 rust/tests/test_240_opt_list_meta.rs | 151 +++
 rust/tests/test_245_opt_list_meta_queries.rs |  98 
 rust/tests/test_250_opt_set_meta.rs  | 124 +++
 rust/tests/test_255_opt_set_meta_queries.rs  | 111 ++
 rust/tests/test_300_get_size.rs  |  36 +
 rust/tests/test_400_pread.rs |  40 +
 rust/tests/test_405_pread_structured.rs  |  80 ++
 rust/tests/test_410_pwrite.rs|  62 
 rust/tests/test_460_block_status.rs  |  96 
 rust/tests/test_620_stats.rs |  76 ++
 rust/tests/test_log/mod.rs   |  86 +++
 24 files changed, 1451 insertions(+), 2 deletions(-)
 create mode 100644 rust/tests/nbdkit_pattern/mod.rs
 create mode 100644 rust/tests/test_100_handle.rs
 create mode 100644 rust/tests/test_110_defaults.rs
 create mode 100644 rust/tests/test_120_set_non_defaults.rs
 create mode 100644 rust/tests/test_130_private_data.rs
 create mode 100644 rust/tests/test_140_explicit_close.rs
 create mode 100644 rust/tests/test_200_connect_command.rs
 create mode 100644 rust/tests/test_210_opt_abort.rs
 create mode 100644 rust/tests/test_220_opt_list.rs
 create mode 100644 rust/tests/test_230_opt_info.rs
 create mode 100644 rust/tests/test_240_opt_list_meta.rs
 create mode 100644 rust/tests/test_245_opt_list_meta_queries.rs
 create mode 100644 rust/tests/test_250_opt_set_meta.rs
 create mode 100644 rust/tests/test_255_opt_set_meta_queries.rs
 create mode 100644 rust/tests/test_300_get_size.rs
 create mode 100644 rust/tests/test_400_pread.rs
 create mode 100644 rust/tests/test_405_pread_structured.rs
 create mode 100644 rust/tests/test_410_pwrite.rs
 create mode 100644 rust/tests/test_460_block_status.rs
 create mode 100644 rust/tests/test_620_stats.rs
 create mode 100644 rust/tests/test_log/mod.rs

diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index e81360b..f74c3ac 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -48,3 +48,7 @@ byte-strings = "0.3.1"
 
 [features]
 default = ["log"]
+
+[dev-dependencies]
+once_cell = "1.18.0"
+tempfile = "3.6.0"
diff --git a/rust/Makefile.am b/rust/Makefile.am
index e8a916f..d353949 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -60,6 +60,7 @@ TESTS_ENVIRONMENT = \
LIBNBD_DEBUG=1 \
$(MALLOC_CHECKS) \
abs_top_srcdir=$(abs_top_srcdir) \
+   CARGO=$(CARGO) \
$(NULL)
 LOG_COMPILER = $(top_builddir)/run
 TESTS = run-tests.sh
diff --git a/rust/run-tests.sh b/rust/run-tests.sh
index 7a0bc85..005000e 100755
--- a/rust/run-tests.sh
+++ b/rust/run-tests.sh
@@ -1,6 +1,6 @@
 #!/bin/bash -
 # nbd client library in userspace
-# Copyright Red Hat
+# Copyright Tage Johansson
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -21,4 +21,6 @@
 set -e
 set -x
 
-cargo test
+requires nbdkit --version
+
+$CARGO test -- --nocapture
diff --git a/rust/tests/nbdkit_pattern/mod.rs b/rust/tests/nbdkit_pattern/mod.rs
new file mode 100644
index 000..5f4069e
--- /dev/null
+++ b/rust/tests/nbdkit_pattern/mod.rs
@@ -0,0 +1,28 @@
+// libnbd Rust test case
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+use once_cell::sync::Lazy;
+
+/// The byte pattern as described in nbdkit-PATTERN-plug

[Libguestfs] [libnbd PATCH v3 07/10] rust: async: Create an async friendly handle type

[Tage Johansson]

 match res_rx.recv().await {\n";
+  pr "Ok(Ok(())) | Err(RecvError::Lagged(_)) => (),\n";
+  pr "Ok(err @ Err(_)) => return err,\n";
+  pr "Err(RecvError::Closed) => unreachable!(),\n";
+  pr "  }\n";
+  pr "}\n";
+  pr "Ok(())\n";
+  pr "}\n\n"
+
+(* Print an impl with all handle calls. *)
+let print_rust_async_handle_impls () =
+  pr "impl AsyncHandle {\n";
+  NameMap.iter print_rust_sync_handle_call sync_handle_calls;
+  NameMap.iter
+(fun name (call, async_kind) ->
+  match async_kind with
+  | WithCompletionCallback ->
+  print_rust_async_handle_call_with_completion_cb name call
+  | ChangesState (predicate, value) ->
+  print_rust_async_handle_call_changing_state name call
+(predicate, value))
+async_handle_calls;
+  pr "}\n\n"
+
+let print_rust_async_imports () =
+  pr "use crate::{*, types::*};\n";
+  pr "use os_str_bytes::OsStringBytes as _;\n";
+  pr "use os_socketaddr::OsSocketAddr;\n";
+  pr "use std::ffi::*;\n";
+  pr "use std::mem;\n";
+  pr "use std::net::SocketAddr;\n";
+  pr "use std::os::fd::{AsRawFd, OwnedFd};\n";
+  pr "use std::path::PathBuf;\n";
+  pr "use std::ptr;\n";
+  pr "use std::sync::Arc;\n";
+  pr "use tokio::sync::{oneshot, broadcast::error::RecvError};\n";
+  pr "\n"
+
+let generate_rust_async_bindings () =
+  generate_header CStyle ~copyright:"Tage Johansson";
+  pr "\n";
+  print_rust_async_imports ();
+  print_rust_async_handle_impls ()
diff --git a/generator/Rust.mli b/generator/Rust.mli
index 450e4ca..0960170 100644
--- a/generator/Rust.mli
+++ b/generator/Rust.mli
@@ -18,3 +18,5 @@
 
 (* Print all flag-structs, enums, constants and handle calls in Rust code. *)
 val generate_rust_bindings : unit -> unit
+
+val generate_rust_async_bindings : unit -> unit
diff --git a/generator/generator.ml b/generator/generator.ml
index 67b9502..dc9eb80 100644
--- a/generator/generator.ml
+++ b/generator/generator.ml
@@ -63,3 +63,4 @@ let () =
   output_to "golang/wrappers.h" GoLang.generate_golang_wrappers_h;
 
   output_to ~formatter:(Some Rustfmt) "rust/src/bindings.rs" 
Rust.generate_rust_bindings;
+  output_to ~formatter:(Some Rustfmt) "rust/src/async_bindings.rs" 
Rust.generate_rust_async_bindings;
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index f74c3ac..c3e27b3 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -45,6 +45,7 @@ thiserror = "1.0.40"
 log = { version = "0.4.19", optional = true }
 libc = "0.2.147"
 byte-strings = "0.3.1"
+tokio = { version = "1.29.1", default-features = false, features = ["rt", 
"sync", "net", "macros"] }
 
 [features]
 default = ["log"]
diff --git a/rust/Makefile.am b/rust/Makefile.am
index cc17bb9..a6fd9b1 100644
--- a/rust/Makefile.am
+++ b/rust/Makefile.am
@@ -18,6 +18,7 @@
 include $(top_srcdir)/subdir-rules.mk
 
 generator_built = \
+   src/async_bindings.rs \
src/bindings.rs \
$(NULL)
 
diff --git a/rust/src/async_handle.rs b/rust/src/async_handle.rs
new file mode 100644
index 000..346c4ef
--- /dev/null
+++ b/rust/src/async_handle.rs
@@ -0,0 +1,222 @@
+// nbd client library in userspace
+// Copyright Tage Johansson
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+// This module implements an asynchronous handle working on top of the
+// [Tokio](https://tokio.rs) runtime. When the handle is created,
+// a "polling task" is spawned on the Tokio runtime. The purpose of that
+// "polling task" is to call `aio_notify_*` when appropriate. It shares a
+// reference to the handle as well as some channels with the handle in the
+// [HandleData] struct. The "polling task" is sleeping when no command is in
+// flight, but wakes up as soon as any command is issued.
+//
+// The commands are implemented as
+// [`async fn`s](https://doc.rust-lang.org/std/keyword.async.html)
+// in async_bindings.rs. There are two types of commands: Those