On Sat, May 23, 2026 at 12:57 PM Helge Deller <[email protected]> wrote:
> On 5/23/26 03:19, Pierrick Bouvier wrote: > > On 5/22/2026 5:40 PM, Warner Losh wrote: > >> > >> > >> On Fri, May 22, 2026 at 6:04 PM Pierrick Bouvier > >> <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> On 5/18/2026 2:27 PM, Warner Losh wrote: > >> > Upstream the file, thread, socket and remaining signal system > >> calls (too > >> > numerous to list here). > >> > > >> > This series is an ambitous use of claude to help me upstream all > the > >> > remaining system calls. I've batched them together in what I > think are > >> > reasonable chunks, and had claude do the grunt work of copying > the > >> code > >> > over and attributing the commits from our complex branching > >> history. The > >> > chopping up was a bit arbitrary, but I think it's good. The > commit > >> > messages may be a little weak, but may also be OK. I've also > double > >> > checked the style and made fixes upstream for them as well. > Claude > >> also > >> > reviewed all these changes and found a few bugs that I've fixed. > I've > >> > personally read through them all and haven't found anything > glaring. I > >> > fixed all the bugs that were found, in most cases differently > than > >> > claude's suggestions. > >> > > >> > I've added 'Assisted-by: Claude...' to all these commits to > reflect my > >> > leaning on Claude so hard. This use falls within the 'non- > >> creative' use > >> > that the Qemu project has said is OK. If that's not the right > thing to > >> > do, I can remove them. > >> > > >> > This leaves sysctl translation, the powerpc architecture support, > >> > coredumps and a transition to 'truss' based system call tracing. > With > >> > these changes applied we'er down from a high of about 30k lines > of > >> diffs > >> > to only 5k (not counting genereted, but checked in files in > >> blitz). The > >> > changes are 8k, so maybe a bit ambitious from that perspective as > >> well. > >> > > >> > There's a few lines over 80 that I've not cleaned up. Let me > know if > >> > that's a problem. The other warnings are about adding files, and > >> there's > >> > no new MAINTAINERS entry needed. And the 'arch dependent defines > >> should > >> > be avoided' are needed to cope with different FreeBSD build > systems > >> > having different system calls. > >> > > >> > Note, this is called out below too, but in v2 I've folded back > all the > >> > fixes based on some out-of-band feedback I recieved to do this > the > >> > normal way and the qemu project isn't interested in the fixes to > >> fixes. > >> > > >> > This is a big experiment, in many ways, for me, so I'm > interested in > >> > whatever feedback you may have to make things better in the > future. > >> > > >> > Signed-off-by: Warner Losh <[email protected] <mailto: > [email protected]>> > >> > --- > >> > Changes in v2: > >> > - don't write offsets on errors for copy_file_range > >> > - pipe2: Check the raw system call value directly > >> > - setsockopt: Check target_to_host_ip_mreq return value > >> > - <several places>: Check return value of host_to_target_sockaddr > >> > - do_bsd_setsockopt: Read in the socket option > >> > - ntp_gettime: write back time on non-errors, not just 0 > >> > - do_freebsd_sigtimedwait check t2h_freebsd_timespec return > >> > - do_freebsd_swapcontext writing incorrectly, readonly data (0 to > >> user_unlock) > >> > - do_thr_set_name needs get_errno() > >> > - Have ntp_adjtime write back the structure > >> > - kevent_freebsd11 no need to explicitly mask, datatypes do this > >> for us > >> > - do_freebsd_setcontext just reads the mcontext, 0 in unlock_user > >> > - target_to_host_sigevent is read-only, so pass 0 in unlock_user > >> > - target_to_host_ip_mreq needs tswap32 for int32_t > >> > - use get_errno on kenv return values to report errors better > >> > - kevent: only copy the number of events returns, not whole > buffer > >> > - Copy tolerance field of timex > >> > - freebsd_umtx_sem_wake use VERIFY_READ instead of write to read > >> struct > >> > - Avoid double g2h_untagged mistakes in !_UMTX_OPTIMIZED > >> > - Fixed additional missing unlock in freebsd_rw_rdlock and > >> freebsd_rw_wrlock in !_UMTX_OPTIMIZED > >> > - Squashed all the bug commit back to the appropriate earleir > patch > >> > - Link to v1: https://lore.kernel.org/qemu-devel/20260515- > >> [email protected] <https://lore.kernel.org/ > >> qemu-devel/[email protected]> > >> > > >> > --- > >> > Warner Losh (37): > >> > bsd-user: catchup to locking / mapping routines in > bsd-misc. > >> > bsd-user: Rename cpu_env to env throughout bsd-user > >> > bsd-user: Add bsd-signal.h with sigaction through sigreturn > >> > bsd-user: Add signal shims sigwait through kill and > os-signal.h > >> > bsd-user: Add signal system call dispatch > >> > bsd-user: Add poll, lseek, pipe, and swap system call shims > >> > bsd-user: Add os-file.h with pipe2, chflagsat, close_range, > >> and more > >> > bsd-user: Add file operation system call dispatch > >> > bsd-user: Add bsd-socket.h with bind through getsockname > >> > bsd-user: Add socket shims socketpair through shutdown > >> > bsd-user: Add os-socket.h with sendrecvmsg and message > >> structures > >> > bsd-user: Add do_bsd_setsockopt and socket option > definitions > >> > bsd-user: Add do_bsd_getsockopt > >> > bsd-user: Add FreeBSD socket helpers and sockaddr > conversion > >> > bsd-user: Add os-socket.c with cmsg conversion functions > >> > bsd-user: Add socket system call dispatch > >> > bsd-user: Add os-time.h with clock and time-of-day > functions > >> > bsd-user: Add utimes, futimes, and ktimer functions to > os-time.h > >> > bsd-user: Add select, pselect, and ppoll to os-time.h > >> > bsd-user: Add kqueue and kevent functions to os-time.h > >> > bsd-user: Add sigtimedwait, itimer, and futimens to > os-time.h > >> > bsd-user: Add os-time.c and time-related type definitions > >> > bsd-user: Add time system call dispatch > >> > bsd-user: Add thread, umtx, and ACL type definitions > >> > bsd-user: Add cpu_copy and make init_task_state non-static > >> > bsd-user: Add os-thread.h with thr and context functions > >> > bsd-user: Add do_freebsd__umtx_op to os-thread.h > >> > bsd-user: Add os-thread.c with umtx, mutex, and thread > creation > >> > bsd-user: Add thread system call dispatch > >> > bsd-user: Add os-extattr.h with file and fd extattr > functions > >> > bsd-user: Add link and list extattr functions to > os-extattr.h > >> > bsd-user: Add ACL system call shims to os-extattr.h > >> > bsd-user: Add os-extattr.c with ACL conversion functions > >> > bsd-user: Add extended attribute and ACL system call > dispatch > >> > bsd-user: Add scheduler and cpuset functions to os-misc.h > >> > bsd-user: Add kmod, posix, and misc functions to os-misc.h > >> > bsd-user: Add scheduler, cpuset, kmod, and misc syscall > dispatch > >> > > >> > bsd-user/aarch64/target.h | 2 +- > >> > bsd-user/arm/target.h | 2 +- > >> > bsd-user/bsd-file.h | 164 +++- > >> > bsd-user/bsd-mem.h | 4 +- > >> > bsd-user/bsd-proc.h | 24 +- > >> > bsd-user/bsd-signal.h | 231 ++++++ > >> > bsd-user/bsd-socket.c | 98 +++ > >> > bsd-user/bsd-socket.h | 283 +++++++ > >> > bsd-user/freebsd/meson.build | 8 +- > >> > bsd-user/freebsd/os-extattr.c | 116 +++ > >> > bsd-user/freebsd/os-extattr.h | 662 ++++++++++++++++ > >> > bsd-user/freebsd/os-file.h | 129 ++++ > >> > bsd-user/freebsd/os-misc.h | 470 +++++++++++- > >> > bsd-user/freebsd/os-proc.c | 4 +- > >> > bsd-user/freebsd/os-proc.h | 26 +- > >> > bsd-user/freebsd/os-signal.h | 20 + > >> > bsd-user/freebsd/os-socket.c | 234 ++++++ > >> > bsd-user/freebsd/os-socket.h | 770 +++++++++++++++++++ > >> > bsd-user/freebsd/os-stat.h | 8 +- > >> > bsd-user/freebsd/os-sys.c | 4 +- > >> > bsd-user/freebsd/os-syscall.c | 790 +++++++++++++++++-- > >> > bsd-user/freebsd/os-thread.c | 1682 > ++++++++++++++++++++++++++++ > >> +++++++++++++ > >> > bsd-user/freebsd/os-thread.h | 596 +++++++++++++++ > >> > bsd-user/freebsd/os-time.c | 346 +++++++++ > >> > bsd-user/freebsd/os-time.h | 983 ++++++++++++++++++++++++ > >> > bsd-user/freebsd/qemu-os.h | 48 ++ > >> > bsd-user/i386/target.h | 2 +- > >> > bsd-user/main.c | 32 +- > >> > bsd-user/meson.build | 1 + > >> > bsd-user/qemu-bsd.h | 26 + > >> > bsd-user/qemu.h | 48 +- > >> > bsd-user/riscv/target.h | 2 +- > >> > bsd-user/signal-common.h | 4 + > >> > bsd-user/signal.c | 67 ++ > >> > bsd-user/syscall_defs.h | 458 +++++++++-- > >> > bsd-user/x86_64/target.h | 2 +- > >> > 36 files changed, 8150 insertions(+), 196 deletions(-) > >> > --- > >> > base-commit: edcc429e9e41a8e0e415dcdab6aa52ad17ef8889 > >> > change-id: 20260504-misc-2026q2-a5af31ac032a > >> > > >> > Best regards, > >> > >> After reviewing (what I can) two thirds of the series, I don't > feel very > >> at ease with validating the rest. Not that there is problem in > itself, > >> but it's a massive add, and I'm not sure how we can ensure it's > >> implemented correctly without any test exercising it. > >> It's not particular to bsd-user, linux-user does not have test for > all > >> syscalls implemented neither. > >> > >> > >> Yes. I'd like a low-level test like that. The upstream bsd-user has > been too > >> incomplete to run real programs prior to this series. And there's still > some > >> issues that remain after these system calls (which is next up on my > list: I > >> think elfload needs some patches for PIE). > >> > >> > >> However, because we are adding most of it at once, it's a bit > different. > >> Dropping 8150 new lines at once without any test looks a bit > complicated > >> to me. > >> > >> A few questions about the series in itself: > >> - Where does most of the implementation comes from? Was it > downstream on > >> FreeBSD side, or written from scratch for this series? > >> > >> > >> We've had almost all of this code, modulo some big fixes I've added, in > >> blitz branch > >> for years. Stacey Son implemented most of this a decade ago now and had > some > >> issue upstream at the time, IIRC, but I have only sketchy details of the > >> project then. > >> > >> We use it inside our poudriere jails to build thousands of ports to test > >> it. Though there are issues with many ports, enough work to keep going. > > So, this seems to be similar to what I do, and it's actually a good test > case for linux-user. > I'm running some debian buildd servers which use qemu-hppa to build debian > packages > for the hppa architecture. With that setup I found quite some bugs in > qemu-user > and now it's pretty stable. > > >> I also run a number of random commands in the jail that's nothing but > >> cross-arch binaries > >> as well. > >> > > > > That's good to hear this has been used downstream for a long time. That > > was my feeling after our conversation, but just wanted to double check > this. > > > >> Sometimes I run the Kyua tests, but it's a bit hard to read the output > >> because many are broken, > >> many can't work in qemu or in a jail, and there's some that are broken > >> for a given arch. We don't > >> have good baselines to compare against and I've not spent the time to > >> get this into good shape. > >> > >> Not the most formal set of things we do, but it works. > >> > >> > >> - How did you validate it works, beyond it compiles? > >> > >> > >> Generally on the blitz side, all of the above. On the qemu-project side, > >> I usually just do the bit > >> about messing around in the jail to test the basics. > >> > >> I'm happy cleaving this massive set into some smaller sets if that would > >> help, but that still wouldn't > >> answer the 'how the heck do we test it'. > >> > > > > It would require a massive effort for no benefit, since it doesn't > > increase testability of the whole. > > > >> I've also had claude do code reviews of this branch. It's found maybe > >> 100 bugs that I've fixed > >> (maybe 25 more between v2 and v3). The vast majority of the bugs are > >> lock/unlock issues. Either > >> leaks on error paths, failure to check for TARGET_EFAULT in places or > >> passing args that don't > >> matter unless you compile CONFIG_DEBUG_REMAP to indicate copying, etc. > >> Some real bugs > >> have been found, including one that might lead to a multi-threaded hang. > >> > > > > This is better than nothing, and it's a common usage I have with Claude > > too. I just hope the industry does not settle on "let the AI review" > > instead of test/cover approach. > > > >> I have been putting off dealing with the PIE issue which blocks building > >> most ports at the moment, > >> but there's two pull requests to address it in blitz. When I tried them > >> I hit some problems, but > >> maybe I've fixed those in other ways. If I fixed that, I'd know that the > >> base code works, but > >> maybe all the pieces aren't in qemu-project yet. > >> > >> Based on all that, what's your advice? > >> > > > > In an ideal world, I would advise to upstream the minimum to be able to > > run hello-world. Once done, add tests for all existing syscalls. > > Then add new syscalls with their tests. I tend to think that good code > > is not the one being the most elegant/performant, but the one that you > > can modify with trust in what you did, which means covered by a test. > Hello World was upstreamed about 3 years ago now. The rest has been insanely slow to get going... > > In the real world, seems like we don't really have another option than > > taking it, except if you're ready to spend 6 months to add tests for > > this series. I'll let other reviewers finish the work for this series. > > > > I'm not against the pragmatic approach of taking it, and let it be > > tested on the field. > > +1 from me. > Yea, I'm trying to see how close to being able to build something with Poudriere (the FreeBSD building tool that has this tightly integrated) was can get. Right now I'm running into umtx hangs, which are a joy to debug. > > It would be unfair considering how the rest of QEMU is not always tested. > > True. > Yea... I have unfair feelings from time to time, but I've just decided to push back on the suggestions that feel that way that don't really help. I'm working towards having blitz being able to run poudriere again... Somewhere between 9.2 and now something broke things, but I think I got those things fixed, and then again around 10.1 or 10.2 there was another. It's one reason I want it in upstream: we can then have some minimal acceptance test (that's less than a full poudriere bulk build, since that's hours of CPU time) that will hopefully keep things from breaking... Even just 'install freebsd/$ARCH into a jail and build hello world' would catch a huge class of bugs. Some basic locking tests would catch the rest of the big-ticket items. Warner > Helge >
