On Tue, 2021-05-04 at 12:57 +0200, Marc Lehmann wrote:
> Thanks for trying out the iouring backend.
>
> On Wed, Apr 28, 2021 at 11:24:49AM -0400, Olivier Langlois
> <oliv...@trillion01.com> wrote:
> > I believe that in order to achieve the performance gain that io_uring
> > can deliver, you would need to service I/O through io_uring as well
> > to
> > save on the associated system call cost instead of just using
> > io_uring
> > for polling.
>
> iouring being quite a bit slower than epoll was my own experience. That
> and it being too buggy for general use (and obviously no movement of
> the
> maintainer to fix things) made me kind of give up on this as a libev
> backend. It can only work in some case,s or when you embed it into
> another
> event loop. And then you can just embed iouring into an epoll loop to
> get
> the best of both worlds.
I tend to disagree on the future of this new API. It seems to have a
lot of potential. It has a lot of visibility, Linus himself seems to be
heavily involved in it. From my point of view, they do put a lot of
efforts and resources into its development. I do skim through the
kernel release notes at each release and as of right now io_uring is
one of the most actively developped kernel component. With that kind of
effort, I have a hard time imagining something else than a bright
future for it.
I am about to upgrade my system to 5.12.1 with a fresh compile
including nohzfull option. (but Mellanox mlx5 driver code is broken in
this release!) According to Jens io_uring is 10% faster previous
iteration...
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.12-Faster-IO_uring
I wish he would care to explain what changes are responsible for this
boost. I did try to find out and I couldn't see what could explain this
improvement.
As far as I can tell, it is only code clean up and optimizations but
without more details, I find the claims hard to believe...
and 5.13 appears to be a game changer for libev users... It will have
multishot poll support in it:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=625434dafdd97372d15de21972be4b682709e854
>
> > but if adding a function specific to libev io_uring backend that
> > would
> > let the watcher code perform their i/o requests through io_uring was
> > thinkable, that could be the performance holy grail for libev users
> > by
> > only making 1 system call to service the 64 i/o operations.
>
> The obvious way to do this would be to expose request submission. There
> are
> two problems to solve though: a) how to identify/distinguish those
> requests
> from libev's own ones and b) libev can tear down and open a new iouring
> at
> any time, and this is hard to synchronise with external users.
>
> It might be possible to do some kind of iouring watcher.
>
> However, in the need, you can also use your own iouring and embed it
> into
> libev. This will also take care of extra system calls, and is rather
> clean
> and works with e.g. epoll as well, and might even be faster (certainly
> more correct, as iouring is too buggy as a generic event backend).
I have a working proof of concept modified libev that supports async
reading.
The project that I did use as a source of inspiration is:
https://github.com/frevib/io_uring-echo-server/
Here are the big lines of how it works:
1. I create a new eventmask bit.
2. Reuse the ev_io watcher type that I extend using its data pointer
with an agreed content (struct ev_io_uring_buffer_read_params) to
support the new event type
3. Refine the sqe user_data usage by encoding an event type in it
4. Add a io_uring loop specific function to return back the buffers to
io_uring when the user code is done with it.
I have tested it and it works fine. Here are some random comments about
the experience:
1. Async reading only works on blocking fds (It returns EAGAIN
otherwise). This makes the transition of existing multiplexing code
harder (I wish there was a flag for io_uring to disregard O_NONBLOCK
flag for this case)
2. I have started using liburing for the io_uring loop implementation.
I understand that this is something you cannot afford to do to not
force a new dependency on your users but I do not have this constraint
and when I saw how passing a timout to io_uring_enter syscall was done,
I just gave up the idea of doing my code in the right way. My goal is
to reach a working proof-of-concept prototype ASAP. Not fixing bugs in
complex boilerplate code. I guess that the resulting code can still
have some learning value despite not being pullable as-is
3. ev_io_uring code is currently littered with printf. I am currently
trying to fix an odd behavior observed from io_uring:
https://lore.kernel.org/io-uring/8992f5f989808798ad2666b0a3ef8ae8d777b7de.ca...@trillion01.com/T/#u
imho, this should be of interest to you because AFAIK, the original
libev 4.33 must be plagued by the same issue...
I will attach the mods to this email.
My app is mostly one sided on the receive side so Async reading is the
low hanging performance fruit. My end goal is having an async io_uring
openssl BIO module 100% fed by libev.
I guess once I have the reading part done, starting doing the same for
output should be easy to do.
>
> > In the meantime, I took care of one of the TODO item. That is using a
> > single mmap() when possible. It is essentially code from liburing
> > adapted to libev coding style...
>
> Thanks, when I come around to implement this I will certainly take
> advantage of your work, although this is currently on the back burner
> due
> to the issues with iouring.
>
> Would it be possible to re-send the patch properly though? The version
> you
> sent is completely garbled because there are extra spurious newlines
> all
> over the place.
Yes. I am going to do that. Check this email attachement.
>
> > By switching from epoll backend to io_uring one, my process CPU usage
> > did drop from 20-30% to below 5%. It seems too good to be true!
> > What I suspect happening is that my socket option SO_BUSY_POLL
> > setting
> > might not be honored by io_uring.
>
> That indeed sounds too good to be true. In my tests iouring is
> consistently
> slower, although I can imagine that in workloads which are very heavy
> on
> syscalls (e.g. epoll_ctl) this might change.
>
> On the other hand, epoll now has a mode where it can also queue things
> with few syscalls, and as much as I hate epoll, since iouring is going
> down the same road as linux aio (buggy, never getting fixed), it is
> probably the way to go for the future.
I was a big epoll user 10 years ago and well served by it. Ive stopped
following its development when I delegated the implementation details
to good libraries like libev. I wasn't aware at all that epoll was now
allowing its users to queue things.
>
> On Wed, Apr 28, 2021 at 11:31:47AM -0400, Olivier Langlois
> <oliv...@trillion01.com> wrote:
> > Here is a last quick sidenote concerning my CPU usage observation.
> >
> > CPU usage reported by top is now below 5% when using io_uring backend
> > but it seems like the CPU is spent by something else inside the
> > kernel
> > as my average load did pass from 2.5 to ~3.1...
>
> I would expect that all other things being similar, that things take
> more
> cpu, as iouring seems to be vastly less efficient as epoll (e.g. its
> use of
> hash tables instead of a simple array lookup for everything is bound to
> slow
> things down).
>
> It might be possible that this is improved in future versions of the
> kernel, but I am doubtful that it can ever reach epoll speeds,
> especialy
> if a queueing system is used for epoll as well (which libev does not
> yet
> implement).
>
> All of this points ot being the right solution to use iouring for the
> things
> only it can do (I/O) and using epoll with a submission queue for the
> rest.
>
This is a good point. I guess that the only way to find out is to try
out those ideas. my bet isn't only on the queuing mechanic but also the
elimination of system calls for performing the I/O that io_uring can
offer.
I'll go through the experiment. We will see if it is beneficial... and
adjust until the bext combination is found...
--- ev_iouring.c.orig 2021-04-28 09:20:26.425089573 -0400
+++ ev_iouring.c 2021-05-05 09:27:16.199395115 -0400
@@ -61,7 +61,7 @@
* applications, to the detriment of everybody else who just wants
* an event loop. but, umm, ok, if that's all, it could be worse.
* (from what I gather from the author Jens Axboe, it simply didn't
- * occur to him, and he made good on it by adding an unlimited nuber
+ * occur to him, and he made good on it by adding an unlimited number
* of timeouts later :).
* h) initially there was a hardcoded limit of 4096 outstanding events.
* later versions not only bump this to 32k, but also can handle
@@ -79,7 +79,7 @@
*/
/* TODO: use internal TIMEOUT */
-/* TODO: take advantage of single mmap, NODROP etc. */
+/* TODO: take advantage of NODROP etc. */
/* TODO: resize cq/sq size independently */
#include <sys/timerfd.h>
@@ -320,7 +320,8 @@
close (iouring_fd);
if (iouring_sq_ring != MAP_FAILED) munmap (iouring_sq_ring, iouring_sq_ring_size);
- if (iouring_cq_ring != MAP_FAILED) munmap (iouring_cq_ring, iouring_cq_ring_size);
+ if (iouring_cq_ring != MAP_FAILED && iouring_cq_ring != iouring_sq_ring)
+ munmap (iouring_cq_ring, iouring_cq_ring_size);
if (iouring_sqes != MAP_FAILED) munmap (iouring_sqes , iouring_sqes_size );
if (ev_is_active (&iouring_tfd_w))
@@ -343,7 +344,7 @@
iouring_cq_ring = MAP_FAILED;
iouring_sqes = MAP_FAILED;
- if (!have_monotonic) /* cannot really happen, but what if11 */
+ if (!have_monotonic) /* cannot really happen, but what if!! */
return -1;
for (;;)
@@ -357,7 +358,7 @@
return -1; /* we failed */
#if TODO
- if ((~params.features) & (IORING_FEAT_NODROP | IORING_FEATURE_SINGLE_MMAP | IORING_FEAT_SUBMIT_STABLE))
+ if ((~params.features) & (IORING_FEAT_NODROP | IORING_FEAT_SUBMIT_STABLE))
return -1; /* we require the above features */
#endif
@@ -378,10 +379,20 @@
iouring_cq_ring_size = params.cq_off.cqes + params.cq_entries * sizeof (struct io_uring_cqe);
iouring_sqes_size = params.sq_entries * sizeof (struct io_uring_sqe);
+ if (params.features & IORING_FEAT_SINGLE_MMAP)
+ {
+ if (iouring_cq_ring_size > iouring_sq_ring_size)
+ iouring_sq_ring_size = iouring_cq_ring_size;
+ iouring_cq_ring_size = iouring_sq_ring_size;
+ }
+
iouring_sq_ring = mmap (0, iouring_sq_ring_size, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_POPULATE, iouring_fd, IORING_OFF_SQ_RING);
- iouring_cq_ring = mmap (0, iouring_cq_ring_size, PROT_READ | PROT_WRITE,
- MAP_SHARED | MAP_POPULATE, iouring_fd, IORING_OFF_CQ_RING);
+ if (params.features & IORING_FEAT_SINGLE_MMAP)
+ iouring_cq_ring = iouring_sq_ring;
+ else
+ iouring_cq_ring = mmap (0, iouring_cq_ring_size, PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_POPULATE, iouring_fd, IORING_OFF_CQ_RING);
iouring_sqes = mmap (0, iouring_sqes_size, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_POPULATE, iouring_fd, IORING_OFF_SQES);
@@ -634,7 +645,7 @@
iouring_poll (EV_P_ ev_tstamp timeout)
{
/* if we have events, no need for extra syscalls, but we might have to queue events */
- /* we also clar the timeout if there are outstanding fdchanges */
+ /* we also clear the timeout if there are outstanding fdchanges */
/* the latter should only happen if both the sq and cq are full, most likely */
/* because we have a lot of event sources that immediately complete */
/* TODO: fdchacngecnt is always 0 because fd_reify does not have two buffers yet */
--- ev_vars.h.orig 2021-05-01 05:54:44.227350081 -0400
+++ ev_vars.h 2021-05-01 03:35:33.347647544 -0400
@@ -119,32 +119,8 @@
#endif
#if EV_USE_IOURING || EV_GENWRAP
-VARx(int, iouring_fd)
-VARx(unsigned, iouring_to_submit);
+VARx(struct io_uring, iouring_ring)
VARx(int, iouring_entries)
-VARx(int, iouring_max_entries)
-VARx(void *, iouring_sq_ring)
-VARx(void *, iouring_cq_ring)
-VARx(void *, iouring_sqes)
-VARx(uint32_t, iouring_sq_ring_size)
-VARx(uint32_t, iouring_cq_ring_size)
-VARx(uint32_t, iouring_sqes_size)
-VARx(uint32_t, iouring_sq_head)
-VARx(uint32_t, iouring_sq_tail)
-VARx(uint32_t, iouring_sq_ring_mask)
-VARx(uint32_t, iouring_sq_ring_entries)
-VARx(uint32_t, iouring_sq_flags)
-VARx(uint32_t, iouring_sq_dropped)
-VARx(uint32_t, iouring_sq_array)
-VARx(uint32_t, iouring_cq_head)
-VARx(uint32_t, iouring_cq_tail)
-VARx(uint32_t, iouring_cq_ring_mask)
-VARx(uint32_t, iouring_cq_ring_entries)
-VARx(uint32_t, iouring_cq_overflow)
-VARx(uint32_t, iouring_cq_cqes)
-VARx(ev_tstamp, iouring_tfd_to)
-VARx(int, iouring_tfd)
-VARx(ev_io, iouring_tfd_w)
#endif
#if EV_USE_KQUEUE || EV_GENWRAP
--- ev_wrap.h.orig 2021-05-01 05:54:45.400674504 -0400
+++ ev_wrap.h 2021-05-01 03:35:45.474258916 -0400
@@ -44,32 +44,8 @@
#define invoke_cb ((loop)->invoke_cb)
#define io_blocktime ((loop)->io_blocktime)
#define iocp ((loop)->iocp)
-#define iouring_cq_cqes ((loop)->iouring_cq_cqes)
-#define iouring_cq_head ((loop)->iouring_cq_head)
-#define iouring_cq_overflow ((loop)->iouring_cq_overflow)
-#define iouring_cq_ring ((loop)->iouring_cq_ring)
-#define iouring_cq_ring_entries ((loop)->iouring_cq_ring_entries)
-#define iouring_cq_ring_mask ((loop)->iouring_cq_ring_mask)
-#define iouring_cq_ring_size ((loop)->iouring_cq_ring_size)
-#define iouring_cq_tail ((loop)->iouring_cq_tail)
#define iouring_entries ((loop)->iouring_entries)
-#define iouring_fd ((loop)->iouring_fd)
-#define iouring_max_entries ((loop)->iouring_max_entries)
-#define iouring_sq_array ((loop)->iouring_sq_array)
-#define iouring_sq_dropped ((loop)->iouring_sq_dropped)
-#define iouring_sq_flags ((loop)->iouring_sq_flags)
-#define iouring_sq_head ((loop)->iouring_sq_head)
-#define iouring_sq_ring ((loop)->iouring_sq_ring)
-#define iouring_sq_ring_entries ((loop)->iouring_sq_ring_entries)
-#define iouring_sq_ring_mask ((loop)->iouring_sq_ring_mask)
-#define iouring_sq_ring_size ((loop)->iouring_sq_ring_size)
-#define iouring_sq_tail ((loop)->iouring_sq_tail)
-#define iouring_sqes ((loop)->iouring_sqes)
-#define iouring_sqes_size ((loop)->iouring_sqes_size)
-#define iouring_tfd ((loop)->iouring_tfd)
-#define iouring_tfd_to ((loop)->iouring_tfd_to)
-#define iouring_tfd_w ((loop)->iouring_tfd_w)
-#define iouring_to_submit ((loop)->iouring_to_submit)
+#define iouring_ring ((loop)->iouring_ring)
#define kqueue_changecnt ((loop)->kqueue_changecnt)
#define kqueue_changemax ((loop)->kqueue_changemax)
#define kqueue_changes ((loop)->kqueue_changes)
--- ev.h.orig 2021-05-01 05:54:45.074010317 -0400
+++ ev.h 2021-05-01 10:13:25.507256778 -0400
@@ -223,6 +223,8 @@
EV_NONE = 0x00, /* no events */
EV_READ = 0x01, /* ev_io detected read will not block */
EV_WRITE = 0x02, /* ev_io detected write will not block */
+ EV_IOURING_BUFSEL_READ
+ = 0x04,
EV__IOFDSET = 0x80, /* internal use only */
EV_IO = EV_READ, /* alias for type-detection */
EV_TIMER = 0x00000100, /* timer timed out */
@@ -326,6 +328,15 @@
int events; /* ro */
} ev_io;
+typedef struct ev_io_uring_buffer_read_params
+{
+ void *data;
+ size_t max_message_size;
+ int bytes_read;
+ unsigned gid;
+ unsigned short bid;
+} ev_io_uring_buffer_read_params;
+
/* invoked after a specific time, repeatable (based on monotonic clock) */
/* revent EV_TIMEOUT */
typedef struct ev_timer
@@ -831,6 +842,8 @@
EV_API_DECL void ev_async_send (EV_P_ ev_async *w) EV_NOEXCEPT;
# endif
+EV_API_DECL int ev_iouring_provide_buffers(EV_P_ void *addr, int len, int nr, int bgid, int bid, int block) EV_NOEXCEPT;
+
#if EV_COMPAT3
#define EVLOOP_NONBLOCK EVRUN_NOWAIT
#define EVLOOP_ONESHOT EVRUN_ONCE
--- ev.c.orig 2021-05-01 05:54:44.564014191 -0400
+++ ev.c 2021-05-01 10:09:08.722021390 -0400
@@ -493,6 +493,7 @@
#if EV_USE_IOURING
# include <sys/syscall.h>
+# include <liburing.h>
# if !SYS_io_uring_setup && __linux && !__alpha
# define SYS_io_uring_setup 425
# define SYS_io_uring_enter 426
@@ -2394,7 +2395,7 @@
{
int i;
- /* most backends do not modify the fdchanges list in backend_modfiy.
+ /* most backends do not modify the fdchanges list in backend_modify.
* except io_uring, which has fixed-size buffers which might force us
* to handle events in backend_modify, causing fdchanges to be amended,
* which could result in an endless loop.
@@ -4337,7 +4338,7 @@
return;
assert (("libev: ev_io_start called with negative fd", fd >= 0));
- assert (("libev: ev_io_start called with illegal event mask", !(w->events & ~(EV__IOFDSET | EV_READ | EV_WRITE))));
+ assert (("libev: ev_io_start called with illegal event mask", !(w->events & ~(EV__IOFDSET | EV_READ | EV_WRITE | EV_IOURING_BUFSEL_READ))));
#if EV_VERIFY >= 2
assert (("libev: ev_io_start called on watcher with invalid fd", fd_valid (fd)));
@@ -5621,6 +5622,15 @@
}
#endif
+int ev_iouring_provide_buffers(EV_P_ void *addr, int len, int nr, int bgid, int bid, int block) EV_NOEXCEPT
+{
+#if EV_USE_IOURING
+ if (backend == EVBACKEND_IOURING)
+ return iouring_provide_buffers(EV_A_ addr, len, nr, bgid, bid, block);
+#endif
+ return -1;
+}
+
#if EV_MULTIPLICITY
#include "ev_wrap.h"
#endif
--- ev_iouring.c.orig 2021-04-28 09:20:26.425089573 -0400
+++ ev_iouring.c 2021-05-04 13:41:20.497145326 -0400
@@ -61,7 +61,7 @@
* applications, to the detriment of everybody else who just wants
* an event loop. but, umm, ok, if that's all, it could be worse.
* (from what I gather from the author Jens Axboe, it simply didn't
- * occur to him, and he made good on it by adding an unlimited nuber
+ * occur to him, and he made good on it by adding an unlimited number
* of timeouts later :).
* h) initially there was a hardcoded limit of 4096 outstanding events.
* later versions not only bump this to 32k, but also can handle
@@ -79,167 +79,51 @@
*/
/* TODO: use internal TIMEOUT */
-/* TODO: take advantage of single mmap, NODROP etc. */
+/* TODO: take advantage of NODROP etc. */
/* TODO: resize cq/sq size independently */
-#include <sys/timerfd.h>
-#include <sys/mman.h>
#include <poll.h>
-#include <stdint.h>
-#define IOURING_INIT_ENTRIES 32
+#define IOURING_INIT_ENTRIES 256
-/*****************************************************************************/
-/* syscall wrapdadoop - this section has the raw api/abi definitions */
-
-#include <linux/fs.h>
-#include <linux/types.h>
-
-/* mostly directly taken from the kernel or documentation */
-
-struct io_uring_sqe
-{
- __u8 opcode;
- __u8 flags;
- __u16 ioprio;
- __s32 fd;
- union {
- __u64 off;
- __u64 addr2;
- };
- __u64 addr;
- __u32 len;
- union {
- __kernel_rwf_t rw_flags;
- __u32 fsync_flags;
- __u16 poll_events;
- __u32 sync_range_flags;
- __u32 msg_flags;
- __u32 timeout_flags;
- __u32 accept_flags;
- __u32 cancel_flags;
- __u32 open_flags;
- __u32 statx_flags;
- };
- __u64 user_data;
- union {
- __u16 buf_index;
- __u64 __pad2[3];
- };
-};
-
-struct io_uring_cqe
-{
- __u64 user_data;
- __s32 res;
- __u32 flags;
-};
-
-struct io_sqring_offsets
-{
- __u32 head;
- __u32 tail;
- __u32 ring_mask;
- __u32 ring_entries;
- __u32 flags;
- __u32 dropped;
- __u32 array;
- __u32 resv1;
- __u64 resv2;
-};
-
-struct io_cqring_offsets
-{
- __u32 head;
- __u32 tail;
- __u32 ring_mask;
- __u32 ring_entries;
- __u32 overflow;
- __u32 cqes;
- __u64 resv[2];
-};
-
-struct io_uring_params
-{
- __u32 sq_entries;
- __u32 cq_entries;
- __u32 flags;
- __u32 sq_thread_cpu;
- __u32 sq_thread_idle;
- __u32 features;
- __u32 resv[4];
- struct io_sqring_offsets sq_off;
- struct io_cqring_offsets cq_off;
+enum {
+ IOURING_POLL,
+ IOURING_PROV_BUF,
+ IOURING_BUF_READ
};
-#define IORING_SETUP_CQSIZE 0x00000008
-
-#define IORING_OP_POLL_ADD 6
-#define IORING_OP_POLL_REMOVE 7
-#define IORING_OP_TIMEOUT 11
-#define IORING_OP_TIMEOUT_REMOVE 12
-
-/* relative or absolute, reference clock is CLOCK_MONOTONIC */
-struct iouring_kernel_timespec
-{
- int64_t tv_sec;
- long long tv_nsec;
-};
-
-#define IORING_TIMEOUT_ABS 0x00000001
-
-#define IORING_ENTER_GETEVENTS 0x01
-
-#define IORING_OFF_SQ_RING 0x00000000ULL
-#define IORING_OFF_CQ_RING 0x08000000ULL
-#define IORING_OFF_SQES 0x10000000ULL
-
-#define IORING_FEAT_SINGLE_MMAP 0x00000001
-#define IORING_FEAT_NODROP 0x00000002
-#define IORING_FEAT_SUBMIT_STABLE 0x00000004
-
-inline_size
-int
-evsys_io_uring_setup (unsigned entries, struct io_uring_params *params)
+inline_speed
+void *
+iouring_build_user_data(char type, int fd, uint32_t egen)
{
- return ev_syscall2 (SYS_io_uring_setup, entries, params);
+ return (void *)((uint32_t)fd | ((__u64)(egen && 0x00ffffff) << 32 ) |
+ ((__u64)type << 56));
}
-inline_size
-int
-evsys_io_uring_enter (int fd, unsigned to_submit, unsigned min_complete, unsigned flags, const sigset_t *sig, size_t sigsz)
+inline_speed
+void
+iouring_decode_user_data(uint64_t data, char *type, int *fd, uint32_t *egen)
{
- return ev_syscall6 (SYS_io_uring_enter, fd, to_submit, min_complete, flags, sig, sigsz);
+ *type = data >> 56;
+ *fd = data & 0xffffffffU;
+ *egen = (data >> 32) & 0x00ffffffU;
}
/*****************************************************************************/
/* actual backed implementation */
-/* we hope that volatile will make the compiler access this variables only once */
-#define EV_SQ_VAR(name) *(volatile unsigned *)((char *)iouring_sq_ring + iouring_sq_ ## name)
-#define EV_CQ_VAR(name) *(volatile unsigned *)((char *)iouring_cq_ring + iouring_cq_ ## name)
-
-/* the index array */
-#define EV_SQ_ARRAY ((unsigned *)((char *)iouring_sq_ring + iouring_sq_array))
-
-/* the submit/completion queue entries */
-#define EV_SQES ((struct io_uring_sqe *) iouring_sqes)
-#define EV_CQES ((struct io_uring_cqe *)((char *)iouring_cq_ring + iouring_cq_cqes))
inline_speed
int
iouring_enter (EV_P_ ev_tstamp timeout)
{
int res;
-
+ struct __kernel_timespec ts;
+ struct io_uring_cqe *cqe_ptr;
+ EV_TS_SET(ts, timeout);
EV_RELEASE_CB;
- res = evsys_io_uring_enter (iouring_fd, iouring_to_submit, 1,
- timeout > EV_TS_CONST (0.) ? IORING_ENTER_GETEVENTS : 0, 0, 0);
-
- assert (("libev: io_uring_enter did not consume all sqes", (res < 0 || res == iouring_to_submit)));
-
- iouring_to_submit = 0;
+ res = io_uring_wait_cqe_timeout(&iouring_ring, &cqe_ptr, &ts);
EV_ACQUIRE_CB;
@@ -254,80 +138,34 @@
struct io_uring_sqe *
iouring_sqe_get (EV_P)
{
- unsigned tail;
-
+ struct io_uring_sqe *sqe = NULL;
+
for (;;)
{
- tail = EV_SQ_VAR (tail);
-
- if (ecb_expect_true (tail + 1 - EV_SQ_VAR (head) <= EV_SQ_VAR (ring_entries)))
- break; /* whats the problem, we have free sqes */
-
- /* queue full, need to flush and possibly handle some events */
-
-#if EV_FEATURE_CODE
- /* first we ask the kernel nicely, most often this frees up some sqes */
- int res = iouring_enter (EV_A_ EV_TS_CONST (0.));
-
- ECB_MEMORY_FENCE_ACQUIRE; /* better safe than sorry */
-
- if (res >= 0)
- continue; /* yes, it worked, try again */
-#endif
+ sqe = io_uring_get_sqe(&iouring_ring);
+ if (sqe)
+ break;
/* some problem, possibly EBUSY - do the full poll and let it handle any issues */
+ printf("do full poll\n");
iouring_poll (EV_A_ EV_TS_CONST (0.));
/* iouring_poll should have done ECB_MEMORY_FENCE_ACQUIRE for us */
}
/*assert (("libev: io_uring queue full after flush", tail + 1 - EV_SQ_VAR (head) <= EV_SQ_VAR (ring_entries)));*/
- return EV_SQES + (tail & EV_SQ_VAR (ring_mask));
-}
-
-inline_size
-struct io_uring_sqe *
-iouring_sqe_submit (EV_P_ struct io_uring_sqe *sqe)
-{
- unsigned idx = sqe - EV_SQES;
-
- EV_SQ_ARRAY [idx] = idx;
- ECB_MEMORY_FENCE_RELEASE;
- ++EV_SQ_VAR (tail);
- /*ECB_MEMORY_FENCE_RELEASE; /* for the time being we assume this is not needed */
- ++iouring_to_submit;
+ return sqe;
}
/*****************************************************************************/
-/* when the timerfd expires we simply note the fact,
- * as the purpose of the timerfd is to wake us up, nothing else.
- * the next iteration should re-set it.
- */
-static void
-iouring_tfd_cb (EV_P_ struct ev_io *w, int revents)
-{
- iouring_tfd_to = EV_TSTAMP_HUGE;
-}
-
/* called for full and partial cleanup */
ecb_cold
static int
iouring_internal_destroy (EV_P)
{
- close (iouring_tfd);
- close (iouring_fd);
-
- if (iouring_sq_ring != MAP_FAILED) munmap (iouring_sq_ring, iouring_sq_ring_size);
- if (iouring_cq_ring != MAP_FAILED) munmap (iouring_cq_ring, iouring_cq_ring_size);
- if (iouring_sqes != MAP_FAILED) munmap (iouring_sqes , iouring_sqes_size );
-
- if (ev_is_active (&iouring_tfd_w))
- {
- ev_ref (EV_A);
- ev_io_stop (EV_A_ &iouring_tfd_w);
- }
+ io_uring_queue_exit(&iouring_ring);
}
ecb_cold
@@ -336,80 +174,12 @@
{
struct io_uring_params params = { 0 };
- iouring_to_submit = 0;
-
- iouring_tfd = -1;
- iouring_sq_ring = MAP_FAILED;
- iouring_cq_ring = MAP_FAILED;
- iouring_sqes = MAP_FAILED;
-
- if (!have_monotonic) /* cannot really happen, but what if11 */
+ if (!have_monotonic) /* cannot really happen, but what if!! */
return -1;
- for (;;)
- {
- iouring_fd = evsys_io_uring_setup (iouring_entries, ¶ms);
-
- if (iouring_fd >= 0)
- break; /* yippie */
-
- if (errno != EINVAL)
- return -1; /* we failed */
-
-#if TODO
- if ((~params.features) & (IORING_FEAT_NODROP | IORING_FEATURE_SINGLE_MMAP | IORING_FEAT_SUBMIT_STABLE))
- return -1; /* we require the above features */
-#endif
-
- /* EINVAL: lots of possible reasons, but maybe
- * it is because we hit the unqueryable hardcoded size limit
- */
-
- /* we hit the limit already, give up */
- if (iouring_max_entries)
- return -1;
-
- /* first time we hit EINVAL? assume we hit the limit, so go back and retry */
- iouring_entries >>= 1;
- iouring_max_entries = iouring_entries;
- }
-
- iouring_sq_ring_size = params.sq_off.array + params.sq_entries * sizeof (unsigned);
- iouring_cq_ring_size = params.cq_off.cqes + params.cq_entries * sizeof (struct io_uring_cqe);
- iouring_sqes_size = params.sq_entries * sizeof (struct io_uring_sqe);
-
- iouring_sq_ring = mmap (0, iouring_sq_ring_size, PROT_READ | PROT_WRITE,
- MAP_SHARED | MAP_POPULATE, iouring_fd, IORING_OFF_SQ_RING);
- iouring_cq_ring = mmap (0, iouring_cq_ring_size, PROT_READ | PROT_WRITE,
- MAP_SHARED | MAP_POPULATE, iouring_fd, IORING_OFF_CQ_RING);
- iouring_sqes = mmap (0, iouring_sqes_size, PROT_READ | PROT_WRITE,
- MAP_SHARED | MAP_POPULATE, iouring_fd, IORING_OFF_SQES);
-
- if (iouring_sq_ring == MAP_FAILED || iouring_cq_ring == MAP_FAILED || iouring_sqes == MAP_FAILED)
+ if (io_uring_queue_init_params(iouring_entries, &iouring_ring, ¶ms) < 0)
return -1;
- iouring_sq_head = params.sq_off.head;
- iouring_sq_tail = params.sq_off.tail;
- iouring_sq_ring_mask = params.sq_off.ring_mask;
- iouring_sq_ring_entries = params.sq_off.ring_entries;
- iouring_sq_flags = params.sq_off.flags;
- iouring_sq_dropped = params.sq_off.dropped;
- iouring_sq_array = params.sq_off.array;
-
- iouring_cq_head = params.cq_off.head;
- iouring_cq_tail = params.cq_off.tail;
- iouring_cq_ring_mask = params.cq_off.ring_mask;
- iouring_cq_ring_entries = params.cq_off.ring_entries;
- iouring_cq_overflow = params.cq_off.overflow;
- iouring_cq_cqes = params.cq_off.cqes;
-
- iouring_tfd = timerfd_create (CLOCK_MONOTONIC, TFD_CLOEXEC);
-
- if (iouring_tfd < 0)
- return iouring_tfd;
-
- iouring_tfd_to = EV_TSTAMP_HUGE;
-
return 0;
}
@@ -423,10 +193,6 @@
ev_syserr ("(libev) io_uring_setup");
fd_rearm_all (EV_A);
-
- ev_io_stop (EV_A_ &iouring_tfd_w);
- ev_io_set (EV_A_ &iouring_tfd_w, iouring_tfd, EV_READ);
- ev_io_start (EV_A_ &iouring_tfd_w);
}
/*****************************************************************************/
@@ -434,59 +200,54 @@
static void
iouring_modify (EV_P_ int fd, int oev, int nev)
{
+ /*
+ * Equal events means that the fd got closed and reopen.
+ */
+ int equalEvents = oev == nev;
+ int oep = oev & (EV_READ|EV_WRITE);
+ int nep = nev & (EV_READ|EV_WRITE);
if (oev)
{
- /* we assume the sqe's are all "properly" initialised */
- struct io_uring_sqe *sqe = iouring_sqe_get (EV_A);
- sqe->opcode = IORING_OP_POLL_REMOVE;
- sqe->fd = fd;
- /* Jens Axboe notified me that user_data is not what is documented, but is
- * some kind of unique ID that has to match, otherwise the request cannot
- * be removed. Since we don't *really* have that, we pass in the old
- * generation counter - if that fails, too bad, it will hopefully be removed
- * at close time and then be ignored. */
- sqe->addr = (uint32_t)fd | ((__u64)(uint32_t)anfds [fd].egen << 32);
- sqe->user_data = (uint64_t)-1;
- iouring_sqe_submit (EV_A_ sqe);
+ if (oep && oep^nep)
+ {
+ struct io_uring_sqe *sqe = iouring_sqe_get (EV_A);
+ printf("%d %d remove %d %u\n", oev, nev, fd, (uint32_t)anfds [fd].egen);
+ io_uring_prep_poll_remove(sqe, iouring_build_user_data(IOURING_POLL, fd, anfds [fd].egen));
+// io_uring_sqe_set_data(sqe, iouring_build_user_data(IOURING_POLL, fd, anfds [fd].egen));
- /* increment generation counter to avoid handling old events */
- ++anfds [fd].egen;
+ /* increment generation counter to avoid handling old events */
+ ++anfds [fd].egen;
+ }
+ else if (equalEvents)
+ {
+ ++anfds [fd].egen;
+ }
}
if (nev)
{
- struct io_uring_sqe *sqe = iouring_sqe_get (EV_A);
- sqe->opcode = IORING_OP_POLL_ADD;
- sqe->fd = fd;
- sqe->addr = 0;
- sqe->user_data = (uint32_t)fd | ((__u64)(uint32_t)anfds [fd].egen << 32);
- sqe->poll_events =
- (nev & EV_READ ? POLLIN : 0)
- | (nev & EV_WRITE ? POLLOUT : 0);
- iouring_sqe_submit (EV_A_ sqe);
- }
-}
-
-inline_size
-void
-iouring_tfd_update (EV_P_ ev_tstamp timeout)
-{
- ev_tstamp tfd_to = mn_now + timeout;
-
- /* we assume there will be many iterations per timer change, so
- * we only re-set the timerfd when we have to because its expiry
- * is too late.
- */
- if (ecb_expect_false (tfd_to < iouring_tfd_to))
- {
- struct itimerspec its;
-
- iouring_tfd_to = tfd_to;
- EV_TS_SET (its.it_interval, 0.);
- EV_TS_SET (its.it_value, tfd_to);
-
- if (timerfd_settime (iouring_tfd, TFD_TIMER_ABSTIME, &its, 0) < 0)
- assert (("libev: iouring timerfd_settime failed", 0));
+ if (nep && (oep^nep || equalEvents))
+ {
+ struct io_uring_sqe *sqe = iouring_sqe_get (EV_A);
+ io_uring_prep_poll_add(sqe, fd, (nev & EV_READ ? POLLIN : 0) | (nev & EV_WRITE ? POLLOUT : 0));
+ io_uring_sqe_set_data(sqe, iouring_build_user_data(IOURING_POLL, fd, anfds [fd].egen));
+ if (fd == 85)
+ printf("%d %d add %d %u\n", oev, nev, fd, (uint32_t)anfds [fd].egen);
+ }
+ if ((nev & EV_IOURING_BUFSEL_READ) && (equalEvents || !(oev & EV_IOURING_BUFSEL_READ)))
+ {
+ ev_io *w = (ev_io *)anfds [fd].head;
+ ev_io_uring_buffer_read_params *params = (ev_io_uring_buffer_read_params *)w->data;
+/*
+ printf("iouring_modify EV_IOURING_BUFSEL_READ buf_group: %d, size: %d\n",
+ params->gid, params->max_message_size);
+*/
+ struct io_uring_sqe *sqe = iouring_sqe_get (EV_A);
+ io_uring_prep_read(sqe, fd, NULL, params->max_message_size, 0);
+ io_uring_sqe_set_flags(sqe, IOSQE_BUFFER_SELECT);
+ sqe->buf_group = params->gid;
+ io_uring_sqe_set_data(sqe, iouring_build_user_data(IOURING_BUF_READ, fd, anfds [fd].egen));
+ }
}
}
@@ -494,58 +255,100 @@
void
iouring_process_cqe (EV_P_ struct io_uring_cqe *cqe)
{
- int fd = cqe->user_data & 0xffffffffU;
- uint32_t gen = cqe->user_data >> 32;
+ char type;
+ int fd;
+ uint32_t gen;
int res = cqe->res;
+ assert (("libev: bufs in automatic buffer selection empty, this should not happen...", res != -ENOBUFS));
+
/* user_data -1 is a remove that we are not atm. interested in */
if (cqe->user_data == (uint64_t)-1)
return;
- assert (("libev: io_uring fd must be in-bounds", fd >= 0 && fd < anfdmax));
-
- /* documentation lies, of course. the result value is NOT like
- * normal syscalls, but like linux raw syscalls, i.e. negative
- * error numbers. fortunate, as otherwise there would be no way
- * to get error codes at all. still, why not document this?
- */
-
- /* ignore event if generation doesn't match */
- /* other than skipping removal events, */
- /* this should actually be very rare */
- if (ecb_expect_false (gen != (uint32_t)anfds [fd].egen))
- return;
+ iouring_decode_user_data(cqe->user_data, &type, &fd, &gen);
- if (ecb_expect_false (res < 0))
- {
- /*TODO: EINVAL handling (was something failed with this fd)*/
-
- if (res == -EBADF)
- {
- assert (("libev: event loop rejected bad fd", res != -EBADF));
- fd_kill (EV_A_ fd);
- }
- else
- {
- errno = -res;
- ev_syserr ("(libev) IORING_OP_POLL_ADD");
- }
-
- return;
+ switch (type) {
+ case IOURING_POLL:
+ assert (("libev: io_uring fd must be in-bounds", fd >= 0 && fd < anfdmax));
+
+ /* documentation lies, of course. the result value is NOT like
+ * normal syscalls, but like linux raw syscalls, i.e. negative
+ * error numbers. fortunate, as otherwise there would be no way
+ * to get error codes at all. still, why not document this?
+ */
+ if (fd == 85)
+ printf("85 gen %d res %d\n", (uint32_t)gen, res);
+
+ /* ignore event if generation doesn't match */
+ /* other than skipping removal events, */
+ /* this should actually be very rare */
+ if (ecb_expect_false (gen != (uint32_t)anfds [fd].egen & 0x00ffffff))
+ return;
+
+ if (ecb_expect_false (res < 0))
+ {
+ /*TODO: EINVAL handling (was something failed with this fd)*/
+
+ if (res == -EBADF)
+ {
+ assert (("libev: event loop rejected bad fd", res != -EBADF));
+ fd_kill (EV_A_ fd);
+ }
+ else
+ {
+ char buf[128];
+ sprintf(buf, "(libev) IORING_OP_POLL_ADD fd: %d, errno: %d", fd, -res);
+ errno = -res;
+ ev_syserr (buf);
+ }
+
+ return;
+ }
+
+ /* feed events, we do not expect or handle POLLNVAL */
+ fd_event (EV_A_ fd,
+ (res & (POLLOUT | POLLERR | POLLHUP) ? EV_WRITE : 0)
+ | (res & (POLLIN | POLLERR | POLLHUP) ? EV_READ : 0)
+ );
+
+ /* io_uring is oneshot, so we need to re-arm the fd next iteration */
+ /* this also means we usually have to do at least one syscall per iteration */
+ anfds [fd].events &= ~(EV_WRITE|EV_READ);
+ fd_change (EV_A_ fd, EV_ANFD_REIFY);
+ break;
+ case IOURING_PROV_BUF:
+ assert (("libev: IOURING_PROV_BUF failed", res >= 0));
+ if (ecb_expect_false (res < 0)) {
+ ev_syserr (strerror(-res));
+ }
+ break;
+ case IOURING_BUF_READ:
+ assert (("libev: io_uring fd must be in-bounds", fd >= 0 && fd < anfdmax));
+ /* ignore event if generation doesn't match */
+ /* other than skipping removal events, */
+ /* this should actually be very rare */
+ if (ecb_expect_false (gen != (uint32_t)anfds [fd].egen & 0x00ffffff))
+ return;
+ if (res <= 0) {
+// printf("iouring_process_cqe EV_IOURING_BUFSEL_READ: %s (%d)\n", strerror(-res), -res);
+ // connection closed or error
+ fd_event (EV_A_ fd, EV_WRITE|EV_READ|EV_ERROR);
+ } else {
+ // bytes have been read into bufs, now add write to socket sqe
+// printf("iouring_process_cqe EV_IOURING_BUFSEL_READ\n");
+ ev_io *w = (ev_io *)anfds [fd].head;
+ if (w) {
+ ev_io_uring_buffer_read_params *params = (ev_io_uring_buffer_read_params *)w->data;
+ params->bid = cqe->flags >> 16;
+ params->bytes_read = res;
+ fd_event (EV_A_ fd, EV_IOURING_BUFSEL_READ);
+ }
+ anfds [fd].events &= ~(EV_IOURING_BUFSEL_READ);
+ fd_change (EV_A_ fd, EV_ANFD_REIFY);
+ }
+ break;
}
-
- /* feed events, we do not expect or handle POLLNVAL */
- fd_event (
- EV_A_
- fd,
- (res & (POLLOUT | POLLERR | POLLHUP) ? EV_WRITE : 0)
- | (res & (POLLIN | POLLERR | POLLHUP) ? EV_READ : 0)
- );
-
- /* io_uring is oneshot, so we need to re-arm the fd next iteration */
- /* this also means we usually have to do at least one syscall per iteration */
- anfds [fd].events = 0;
- fd_change (EV_A_ fd, EV_ANFD_REIFY);
}
/* called when the event queue overflows */
@@ -568,64 +371,31 @@
fd_rearm_all (EV_A);
/* we double the size until we hit the hard-to-probe maximum */
- if (!iouring_max_entries)
- {
- iouring_entries <<= 1;
- iouring_fork (EV_A);
- }
- else
- {
- /* we hit the kernel limit, we should fall back to something else.
- * we can either poll() a few times and hope for the best,
- * poll always, or switch to epoll.
- * TODO: is this necessary with newer kernels?
- */
-
- iouring_internal_destroy (EV_A);
-
- /* this should make it so that on return, we don't call any uring functions */
- iouring_to_submit = 0;
-
- for (;;)
- {
- backend = epoll_init (EV_A_ 0);
-
- if (backend)
- break;
-
- ev_syserr ("(libev) iouring switch to epoll");
- }
- }
+ iouring_entries <<= 1;
+ iouring_fork (EV_A);
}
/* handle any events in the completion queue, return true if there were any */
static int
iouring_handle_cq (EV_P)
{
- unsigned head, tail, mask;
-
- head = EV_CQ_VAR (head);
- ECB_MEMORY_FENCE_ACQUIRE;
- tail = EV_CQ_VAR (tail);
-
- if (head == tail)
- return 0;
+ unsigned completed = 0;
+ unsigned head;
+ struct io_uring_cqe *cqe;
/* it can only overflow if we have events, yes, yes? */
- if (ecb_expect_false (EV_CQ_VAR (overflow)))
+ if (ecb_expect_false (IO_URING_READ_ONCE(*iouring_ring.cq.koverflow)))
{
iouring_overflow (EV_A);
return 1;
}
- mask = EV_CQ_VAR (ring_mask);
-
- do
- iouring_process_cqe (EV_A_ &EV_CQES [head++ & mask]);
- while (head != tail);
-
- EV_CQ_VAR (head) = head;
- ECB_MEMORY_FENCE_RELEASE;
+ io_uring_for_each_cqe(&iouring_ring, head, cqe)
+ {
+ completed++;
+ iouring_process_cqe(EV_A_ cqe);
+ }
+ io_uring_cq_advance(&iouring_ring, completed);
return 1;
}
@@ -634,18 +404,16 @@
iouring_poll (EV_P_ ev_tstamp timeout)
{
/* if we have events, no need for extra syscalls, but we might have to queue events */
- /* we also clar the timeout if there are outstanding fdchanges */
+ /* we also clear the timeout if there are outstanding fdchanges */
/* the latter should only happen if both the sq and cq are full, most likely */
/* because we have a lot of event sources that immediately complete */
- /* TODO: fdchacngecnt is always 0 because fd_reify does not have two buffers yet */
+ /* TODO: fdchangecnt is always 0 because fd_reify does not have two buffers yet */
if (iouring_handle_cq (EV_A) || fdchangecnt)
timeout = EV_TS_CONST (0.);
- else
- /* no events, so maybe wait for some */
- iouring_tfd_update (EV_A_ timeout);
/* only enter the kernel if we have something to submit, or we need to wait */
- if (timeout || iouring_to_submit)
+ unsigned to_submit = io_uring_sq_ready(&iouring_ring);
+ if (timeout || to_submit)
{
int res = iouring_enter (EV_A_ timeout);
@@ -654,6 +422,8 @@
/* ignore */;
else if (errno == EBUSY)
/* cq full, cannot submit - should be rare because we flush the cq first, so simply ignore */;
+ else if (errno == ETIME)
+ /* timeout */;
else
ev_syserr ("(libev) iouring setup");
else
@@ -661,12 +431,36 @@
}
}
+static int
+iouring_provide_buffers(EV_P_ void *addr, int len, int nr, int bgid, int bid, int block)
+{
+ int res;
+ // register buffers for buffer selection
+ struct io_uring_sqe *sqe;
+ struct io_uring_cqe *cqe;
+
+ sqe = io_uring_get_sqe(&iouring_ring);
+ io_uring_prep_provide_buffers(sqe, addr, len, nr, bgid, bid);
+ io_uring_submit(&iouring_ring);
+ if (block) {
+ io_uring_wait_cqe(&iouring_ring, &cqe);
+ res = cqe->res;
+ if (ecb_expect_false (cqe->res < 0)) {
+ ev_syserr("(libev) iouring_provide_buffers failed");
+ }
+ io_uring_cqe_seen(&iouring_ring, cqe);
+ }
+ else {
+ res = 0;
+ }
+ return res;
+}
+
inline_size
int
iouring_init (EV_P_ int flags)
{
- iouring_entries = IOURING_INIT_ENTRIES;
- iouring_max_entries = 0;
+ iouring_entries = IOURING_INIT_ENTRIES;
if (iouring_internal_init (EV_A) < 0)
{
@@ -674,11 +468,6 @@
return 0;
}
- ev_io_init (&iouring_tfd_w, iouring_tfd_cb, iouring_tfd, EV_READ);
- ev_set_priority (&iouring_tfd_w, EV_MINPRI);
- ev_io_start (EV_A_ &iouring_tfd_w);
- ev_unref (EV_A); /* watcher should not keep loop alive */
-
backend_modify = iouring_modify;
backend_poll = iouring_poll;
_______________________________________________
libev mailing list
libev@lists.schmorp.de
http://lists.schmorp.de/mailman/listinfo/libev
