This is an automated email from the ASF dual-hosted git repository. acassis pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nuttx.git
commit 067e30e14f3e64a3b3d0bc6e63e23da2ce8ca39b Author: Marco Casaroli <[email protected]> AuthorDate: Sun Aug 9 15:13:39 2026 +0200 arch/x86_64: Build fork() children from the caller's syscall frame. In a kernel build vfork() is reached through a system call, so the return address and stack pointer the architecture's entry point can see for itself are the kernel's, not the caller's. A child built from those resumes at a kernel address, which is why x86_64 selected the fork family only for the flat build. x86_64_syscall() now publishes the caller's frame in xcp.sregs for the duration of the stub call, and x86_64_fork() builds the child from it: x86_64_fork_syscall() when xcp.sregs is non-NULL, so that the child returns from the very same `syscall' instruction as the parent, in user mode, on its own stack; x86_64_fork_direct() otherwise, which is the flat build and any kernel thread that calls the entry point as a plain function. The discriminator is xcp.sregs rather than TCB_FLAG_SYSCALL, which arm64 and RISC-V use: that flag also defers signal actions, x86_64 has never raised it, and its kernel-build signal path does not survive being made to -- a pre-existing problem that does not belong to this work. Two properties of SYSCALL/SYSRET shape the child's frame. The instruction leaves the caller's RIP and RFLAGS in RCX and R11 rather than on a stack, so they are moved into the RIP and RFLAGS slots of the interrupt frame the child is resumed from; and the hardware never records the caller's CS and SS at all, SYSRETQ reconstructing them from IA32_STAR, so the child's are filled in with the user code and data selectors at RPL 3. The frame is therefore not copied wholesale: the extended state and the general registers are inherited, while the segment registers and the thread pointer stay as up_initial_state() left them, the child's stack being a fresh allocation the parent's FS base does not describe. x86_64_fork_relocfp() is new and is not optional here. A function returns with `leave', which feeds the frame pointer into the stack pointer, so relocating only the RBP the child resumes with gets it exactly one frame: the next return loads a saved RBP still pointing into the parent's stack. With that in place ARCH_X86_64 can select ARCH_HAVE_VFORK unconditionally. Build-verified on qemu-intel64:knsh_romfs and qemu-intel64:ostest. NuttX on qemu-intel64 requires tsc-deadline and pcid, which TCG does not implement, so it cannot be run on this host. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Marco Casaroli <[email protected]> --- Documentation/guides/fork_vfork_migration.rst | 17 +- arch/Kconfig | 2 +- arch/x86_64/include/intel64/irq.h | 12 + arch/x86_64/src/common/x86_64_fork.c | 439 ++++++++++++++++++++++---- arch/x86_64/src/common/x86_64_syscall.c | 35 +- 5 files changed, 446 insertions(+), 59 deletions(-) diff --git a/Documentation/guides/fork_vfork_migration.rst b/Documentation/guides/fork_vfork_migration.rst index 63e5f682a03..fa9a305d3c8 100644 --- a/Documentation/guides/fork_vfork_migration.rst +++ b/Documentation/guides/fork_vfork_migration.rst @@ -153,7 +153,7 @@ exception frame when it traps -- ``xcp.sregs`` is the field that exists for this -- and build the child from that instead, while a kernel thread that calls the entry point directly still takes the ordinary path. -Three architectures do it, and they are worth copying: +Four architectures do it, and they are worth copying: * RISC-V: ``riscv_swint.c`` stores the frame in ``xcp.sregs``, and ``riscv_fork.c`` rebuilds the child from it. @@ -177,6 +177,21 @@ Three architectures do it, and they are worth copying: ``arm_fork()`` runs, its PC, CPSR and SP are the kernel's; the caller's are read from where ``arm_syscall()`` put them -- ``syscall[0].sysreturn``, ``syscall[0].cpsr`` and ``ustkptr``. +* x86_64: ``x86_64_syscall()`` stores the frame in ``xcp.sregs``, and + ``x86_64_fork()`` dispatches to ``x86_64_fork_syscall()`` or + ``x86_64_fork_direct()``. The discriminator here is ``xcp.sregs`` itself + being non-NULL, because raising ``TCB_FLAG_SYSCALL`` would also defer signal + actions -- something x86_64 has never done and its kernel-build signal path + does not currently survive. Two properties of ``SYSCALL``/``SYSRET`` shape + the child's frame: the instruction leaves the caller's RIP and RFLAGS in + RCX and R11 rather than on a stack, so they have to be moved into the RIP + and RFLAGS slots of the interrupt frame the child is resumed from; and the + hardware never records the caller's CS and SS at all -- ``SYSRETQ`` + reconstructs them from ``IA32_STAR`` -- so the child's have to be filled in + with the user code and data selectors at RPL 3. For the same reason the + saved frame is not copied wholesale: only the extended state and the + general registers are inherited, and the segment registers and thread + pointer come from the frame ``up_initial_state()`` built for the child. Nothing else is required: the ``up_fork()`` entry point and the libc wrapper are already there and become live automatically. diff --git a/arch/Kconfig b/arch/Kconfig index 3944e83d5a5..60039f1991e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -147,7 +147,7 @@ config ARCH_X86_64 select PCI_LATE_DRIVERS_REGISTER if PCI select ARCH_TOOLCHAIN_GNU select ARCH_HAVE_BACKTRACE - select ARCH_HAVE_VFORK if !BUILD_KERNEL + select ARCH_HAVE_VFORK select ARCH_HAVE_SETJMP select ARCH_HAVE_PERF_EVENTS select ARCH_HAVE_POWEROFF diff --git a/arch/x86_64/include/intel64/irq.h b/arch/x86_64/include/intel64/irq.h index 24fd088d29c..84a04a4436f 100644 --- a/arch/x86_64/include/intel64/irq.h +++ b/arch/x86_64/include/intel64/irq.h @@ -544,6 +544,18 @@ struct xcptcontext uint64_t *regs; +#ifdef CONFIG_LIB_SYSCALL + /* The register context of the user code that is currently in a system + * call, as x86_64_syscall_entry() saved it on the kernel stack. This is + * what the caller of a system call was doing, as opposed to xcp.regs, + * which during a system call describes the kernel side of it. + * x86_64_fork() needs it to build a child from the caller rather than + * from the stub. + */ + + uint64_t *sregs; +#endif + #ifdef CONFIG_ARCH_ADDRENV # ifdef CONFIG_ARCH_KERNEL_STACK /* In this configuration, all syscalls execute from an internal kernel diff --git a/arch/x86_64/src/common/x86_64_fork.c b/arch/x86_64/src/common/x86_64_fork.c index 05cd26016e8..ef5cfb83e45 100644 --- a/arch/x86_64/src/common/x86_64_fork.c +++ b/arch/x86_64/src/common/x86_64_fork.c @@ -35,6 +35,7 @@ #include <nuttx/sched.h> #include <nuttx/arch.h> +#include <arch/arch.h> #include <arch/irq.h> #include "x86_64_fork.h" @@ -42,57 +43,156 @@ #include "sched/sched.h" /**************************************************************************** - * Public Functions + * Pre-processor Definitions ****************************************************************************/ +#ifdef CONFIG_LIB_SYSCALL + +/* Requested privilege level 3 in a segment selector. A caller that reached + * here through the `syscall' instruction was in user mode, and SYSRETQ is + * going to put it back there with the selectors IA32_STAR describes -- see + * x86_64_fork_syscall(). + */ + +# define X86_GDT_RPL_USER 3 + +#endif + /**************************************************************************** - * Name: x86_64_fork + * Private Functions + ****************************************************************************/ + +/**************************************************************************** + * Name: x86_64_fork_stacktop * * Description: - * The fork() function has the same effect as posix fork(), except that the - * behavior is undefined if the process created by fork() either modifies - * any data other than a variable of type pid_t used to store the return - * value from fork(), or returns from the function in which fork() was - * called, or calls any other function before successfully calling _exit() - * or one of the exec family of functions. + * The high end of the part of a task's stack that a fork copies: the + * bottom of the register save area that up_initial_state() reserved at the + * very top of the stack. The save area itself is not stack and must not + * be copied over -- it is where the child's own resume frame is built. * - * The overall sequence is: + * Input Parameters: + * tcb - The task whose stack is in question * - * 1) User code calls fork(). fork() collects context information and - * transfers control up x86_64_fork(). - * 2) x86_64_fork() and calls nxtask_setup_fork(). - * 3) nxtask_setup_fork() allocates and configures the child task's TCB. - * This consists of: - * - Allocation of the child task's TCB. - * - Initialization of file descriptors and streams - * - Configuration of environment variables - * - Allocate and initialize the stack - * - Setup the input parameters for the task. - * - Initialization of the TCB (including call to up_initial_state()) - * 4) x86_64_fork() provides any additional operating context. It must: - * - Initialize special values in any CPU registers that were not - * already configured by up_initial_state() - * 5) x86_64_fork() then calls nxtask_start_fork() - * 6) nxtask_start_fork() then executes the child thread. + * Returned Value: + * The address one past the last byte of stack that is copied. * - * nxtask_abort_fork() may be called if an error occurs between steps 3 and - * 6. + ****************************************************************************/ + +static uint64_t x86_64_fork_stacktop(struct tcb_s *tcb) +{ + return (uint64_t)XCP_ALIGN_DOWN((uintptr_t)tcb->stack_base_ptr + + tcb->adj_stack_size - XCPTCONTEXT_SIZE); +} + +/**************************************************************************** + * Name: x86_64_fork_reloc + * + * Description: + * Carry one address from the parent's stack over to the child's copy of + * it. Addresses outside the copied region are returned unchanged: they + * point somewhere the child shares with the parent, or somewhere that has + * no counterpart at all. + * + * Input Parameters: + * addr - The address to relocate + * rsp - The parent's stack pointer where the primitive was called, + * which is the low end of the region that was copied + * stacktop - The high end of the region that was copied + * offset - The distance from the parent's stack to the child's copy + * + * Returned Value: + * The relocated address. + * + ****************************************************************************/ + +static uint64_t x86_64_fork_reloc(uint64_t addr, uint64_t rsp, + uint64_t stacktop, uint64_t offset) +{ + if (addr >= rsp && addr < stacktop) + { + return addr + offset; + } + + return addr; +} + +/**************************************************************************** + * Name: x86_64_fork_relocfp + * + * Description: + * Relocate the saved frame-pointer chain inside the child's copy of the + * parent's stack. + * + * This is x86_64-specific and it is not optional. A function returns here + * with `leave', which is `mov %rbp,%rsp' followed by `pop %rbp': the + * frame pointer feeds the *stack* pointer. Relocating only the RBP the + * child resumes with therefore gets it exactly one frame; the moment it + * returns through the next one it loads a saved RBP that still points + * into the parent's stack, and from then on the child runs on the + * parent's stack. It looks like the child is working -- it is even at the + * right offset -- until something returns through a slot the parent has + * since reused. + * + * The other architectures with this fork path do not need it: they return + * through a link register, so a stale frame pointer spoils a backtrace and + * nothing else. + * + * The walk stops at the first link that leaves the copied region -- the + * outermost frame's saved RBP does -- and refuses to move backwards, so a + * corrupt chain terminates it rather than looping. + * + * Input Parameters: + * rbp - The parent's frame pointer where the primitive was called + * rsp - The parent's stack pointer, the low end of the copied region + * stacktop - The high end of the copied region + * offset - The distance from the parent's stack to the child's copy + * + ****************************************************************************/ + +static void x86_64_fork_relocfp(uint64_t rbp, uint64_t rsp, + uint64_t stacktop, uint64_t offset) +{ + while (rbp >= rsp && rbp < stacktop) + { + uint64_t *slot = (uint64_t *)(rbp + offset); + uint64_t next = *slot; + + if (next <= rbp || next >= stacktop) + { + break; + } + + *slot = next + offset; + rbp = next; + } +} + +/**************************************************************************** + * Name: x86_64_fork_direct + * + * Description: + * Clone a caller that reached up_fork() by an ordinary function call, so + * that the register snapshot fork.S took describes the caller itself. + * That is the case in a flat build, and for a kernel thread in any build. + * + * The child has no exception frame to inherit, so one is synthesised: it + * resumes at the caller's return address, in the caller's own segments, + * with the callee-saved registers the caller had. * * Input Parameters: * vfork - true for vfork(), false for fork() - * context - Caller context information saved by up_fork() + * parent - The calling task's TCB + * context - Caller context information saved by fork.S * * Returned Value: - * Upon successful completion, fork() returns 0 to the child process and - * returns the process ID of the child process to the parent process. - * Otherwise, -1 is returned to the parent, no child process is created, - * and errno is set to indicate the error. + * The process ID of the child, or ERROR on failure. * ****************************************************************************/ -pid_t x86_64_fork(bool vfork, const struct fork_s *context) +static pid_t x86_64_fork_direct(bool vfork, struct tcb_s *parent, + const struct fork_s *context) { - struct tcb_s *parent = this_task(); struct tcb_s *child; uint64_t newsp; uint64_t newfp; @@ -120,15 +220,13 @@ pid_t x86_64_fork(bool vfork, const struct fork_s *context) sinfo("TCBs: Parent=%p Child=%p\n", parent, child); - /* How much of the parent's stack was utilized? The ARM uses - * a push-down stack so that the current stack pointer should - * be lower than the initial, adjusted stack pointer. The - * stack usage should be the difference between those two. + /* How much of the parent's stack was utilized? x86_64 uses a push-down + * stack so that the current stack pointer should be lower than the + * initial, adjusted stack pointer. The stack usage should be the + * difference between those two. */ - stacktop = (uint64_t)XCP_ALIGN_DOWN((uintptr_t)parent->stack_base_ptr + - parent->adj_stack_size - - XCPTCONTEXT_SIZE); + stacktop = x86_64_fork_stacktop(parent); DEBUGASSERT(stacktop > context->rsp); stackutil = stacktop - context->rsp; @@ -161,25 +259,17 @@ pid_t x86_64_fork(bool vfork, const struct fork_s *context) * effort: the child is entitled to use them, and it does. */ - newtop = (uint64_t)XCP_ALIGN_DOWN((uintptr_t)child->stack_base_ptr + - child->adj_stack_size - - XCPTCONTEXT_SIZE); - - newsp = newtop - stackutil; + newtop = x86_64_fork_stacktop(child); + newsp = newtop - stackutil; memcpy((void *)newsp, (const void *)context->rsp, stackutil); /* Was there a frame pointer in place before? */ - if (context->rbp >= context->rsp && context->rbp < stacktop) - { - uint32_t frameutil = stacktop - context->rbp; - newfp = newtop - frameutil; - } - else - { - newfp = context->rbp; - } + newfp = x86_64_fork_reloc(context->rbp, context->rsp, stacktop, + newtop - stacktop); + x86_64_fork_relocfp(context->rbp, context->rsp, stacktop, + newtop - stacktop); sinfo("Old stack top:%08" PRIx64 " RSP:%08" PRIx64 " RBP:%08" PRIx64 "\n", stacktop, context->rsp, context->rbp); @@ -213,3 +303,242 @@ pid_t x86_64_fork(bool vfork, const struct fork_s *context) return nxtask_start_fork(child, vfork); } + +#ifdef CONFIG_LIB_SYSCALL + +/**************************************************************************** + * Name: x86_64_fork_syscall + * + * Description: + * Clone a caller that reached up_fork() through a system call. The + * register snapshot fork.S took is useless here: it describes the + * kernel-side stub, so a child built from it would resume at a kernel + * address on a kernel stack. What the caller was actually doing is the + * frame x86_64_syscall_entry() saved and x86_64_syscall() recorded in + * xcp.sregs; the child is built from that. + * + * The child therefore returns from the very same `syscall' instruction as + * the parent, in user mode, differing only in that it sees 0 as the return + * value and runs on its own stack. + * + * Two details of the SYSCALL/SYSRET pair shape this: + * + * 1. `syscall' does not save the caller's RIP and RFLAGS on a stack; it + * leaves them in RCX and R11, which is where the saved frame has them. + * The child is resumed by IRETQ (x86_64_fullcontextrestore()), so they + * have to be moved into the RIP and RFLAGS slots of its frame. + * 2. The hardware never tells the kernel which CS and SS the caller had -- + * SYSRETQ reconstructs them from IA32_STAR -- so those slots of the + * saved frame hold nothing, and the child's have to be filled with the + * selectors SYSRETQ would have produced, which is where the parent is + * about to return to. + * + * Everything the frame does hold -- the general registers and the extended + * (FPU/SSE) state -- is inherited. Everything it does not is taken from + * the frame up_initial_state() built for the child, so that the child + * keeps its own segment registers and, importantly, its own thread + * pointer: the child's stack is a fresh allocation at a different virtual + * address, so the parent's FS base does not describe it. + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * parent - The calling task's TCB + * + * Returned Value: + * The process ID of the child, or ERROR on failure. + * + ****************************************************************************/ + +static pid_t x86_64_fork_syscall(bool vfork, struct tcb_s *parent) +{ + uint64_t *sregs = parent->xcp.sregs; + struct tcb_s *child; + uint64_t newsp; + uint64_t newtop; + uint64_t offset; + uint64_t stacktop; + uint64_t stackutil; + uint64_t rsp; + uint64_t rip; + + DEBUGASSERT(sregs != NULL); + + /* Where the caller was and what it was doing */ + + rsp = sregs[REG_RSP]; + rip = sregs[REG_RCX]; + + sinfo("syscall frame [%p]: RSP:%08" PRIx64 " RIP:%08" PRIx64 "\n", + sregs, rsp, rip); + + /* Allocate and initialize a TCB for the child task. The child resumes at + * the instruction after the `syscall', which is where the parent resumes + * too. + */ + + child = nxtask_setup_fork((start_t)rip, vfork); + if (!child) + { + serr("ERROR: nxtask_setup_fork failed\n"); + return (pid_t)ERROR; + } + + sinfo("TCBs: Parent=%p Child=%p\n", parent, child); + + /* Give the child the part of the parent's stack that is in use, copied to + * the same place in its own stack. The copy is aligned with the top of + * each stack rather than the bottom, so a single offset carries any + * address in the copied region from one to the other. + */ + + stacktop = x86_64_fork_stacktop(parent); + DEBUGASSERT(stacktop > rsp); + stackutil = stacktop - rsp; + + newtop = x86_64_fork_stacktop(child); + newsp = newtop - stackutil; + offset = newtop - stacktop; + + memcpy((void *)newsp, (const void *)rsp, stackutil); + + sinfo("Old stack top:%08" PRIx64 " RSP:%08" PRIx64 "\n", stacktop, rsp); + sinfo("New stack top:%08" PRIx64 " RSP:%08" PRIx64 "\n", newtop, newsp); + + /* Inherit the parent's extended (FPU/SSE) state, which + * x86_64_syscall_entry saved at the front of the frame, exactly where + * the child's belongs. + */ + + memcpy(child->xcp.regs, sregs, XCPTCONTEXT_XMM_AREA_SIZE); + + /* Inherit the general registers. RCX and R11 are included deliberately: + * SYSRETQ leaves the return address in RCX and RFLAGS in R11, so the + * parent resumes with those values and the child must too. + */ + + child->xcp.regs[REG_RBX] = sregs[REG_RBX]; + child->xcp.regs[REG_R8] = sregs[REG_R8]; + child->xcp.regs[REG_R9] = sregs[REG_R9]; + child->xcp.regs[REG_R10] = sregs[REG_R10]; + child->xcp.regs[REG_R11] = sregs[REG_R11]; + child->xcp.regs[REG_R12] = sregs[REG_R12]; + child->xcp.regs[REG_R13] = sregs[REG_R13]; + child->xcp.regs[REG_R14] = sregs[REG_R14]; + child->xcp.regs[REG_R15] = sregs[REG_R15]; + child->xcp.regs[REG_RCX] = sregs[REG_RCX]; + child->xcp.regs[REG_RDX] = sregs[REG_RDX]; + child->xcp.regs[REG_RSI] = sregs[REG_RSI]; + child->xcp.regs[REG_RDI] = sregs[REG_RDI]; + + /* The frame pointer moves with the stack it points into */ + + child->xcp.regs[REG_RBP] = x86_64_fork_reloc(sregs[REG_RBP], rsp, + stacktop, offset); + x86_64_fork_relocfp(sregs[REG_RBP], rsp, stacktop, offset); + + /* Build the interrupt frame the child is resumed from. RIP and RFLAGS + * come out of RCX and R11, and the selectors are the ones SYSRETQ derives + * from IA32_STAR: CS is the user code segment and SS the user data + * segment, both at RPL 3. See x86_64_cpu_priv_set(), which programs + * IA32_STAR. + */ + + child->xcp.regs[REG_RAX] = 0; + child->xcp.regs[REG_RIP] = rip; + child->xcp.regs[REG_RFLAGS] = sregs[REG_R11]; + child->xcp.regs[REG_RSP] = newsp; + child->xcp.regs[REG_CS] = X86_GDT_USERCODE_SEL | X86_GDT_RPL_USER; + child->xcp.regs[REG_SS] = X86_GDT_USERDATA_SEL | X86_GDT_RPL_USER; + +#ifdef CONFIG_ARCH_KERNEL_STACK + /* The child's own user stack pointer, for the signal dispatch path */ + + child->xcp.ustkptr = (uintptr_t *)newsp; +#endif + + /* And, finally, start the child task. On a failure, nxtask_start_fork() + * will discard the TCB by calling nxtask_abort_fork(). + */ + + return nxtask_start_fork(child, vfork); +} + +#endif /* CONFIG_LIB_SYSCALL */ + +/**************************************************************************** + * Public Functions + ****************************************************************************/ + +/**************************************************************************** + * Name: x86_64_fork + * + * Description: + * The common x86_64 worker behind up_fork(). vfork() and fork() snapshot + * the caller's registers identically; `vfork' says which primitive was + * called, and is passed straight through to nxtask_setup_fork(), which is + * where the memory semantics are decided. + * + * The overall sequence is: + * + * 1) User code calls vfork() or fork(). The libc wrapper enters + * up_fork(), which collects context information and transfers control + * to x86_64_fork(). + * 2) x86_64_fork() calls nxtask_setup_fork(). + * 3) nxtask_setup_fork() allocates and configures the child task's TCB. + * This consists of: + * - Allocation of the child task's TCB. + * - Initialization of file descriptors and streams + * - Configuration of environment variables + * - Allocate and initialize the stack + * - Setup the input parameters for the task. + * - Initialization of the TCB (including call to up_initial_state()) + * 4) x86_64_fork() provides any additional operating context. It must: + * - Initialize special values in any CPU registers that were not + * already configured by up_initial_state() + * 5) x86_64_fork() then calls nxtask_start_fork() + * 6) nxtask_start_fork() then executes the child thread. + * + * nxtask_abort_fork() may be called if an error occurs between steps 3 and + * 6. + * + * Everything above is common to the two ways this can be reached, which + * differ only in where the caller's registers are to be found -- see + * x86_64_fork_direct() and x86_64_fork_syscall(). + * + * Input Parameters: + * vfork - true for vfork(), false for fork() + * context - Caller context information saved by fork.S + * + * Returned Value: + * Upon successful completion, 0 is returned to the child and the process + * ID of the child is returned to the parent. Otherwise, -1 is returned to + * the parent, no child is created, and errno is set to indicate the error. + * + ****************************************************************************/ + +pid_t x86_64_fork(bool vfork, const struct fork_s *context) +{ + struct tcb_s *parent = this_task(); + +#ifdef CONFIG_LIB_SYSCALL + /* A non-NULL xcp.sregs means a system call is in progress: + * x86_64_syscall() publishes the caller's frame there for the duration of + * the call and nowhere else. So this was reached from a kernel-side stub, + * and the caller to clone is the user task that trapped, not the code that + * called into fork.S. + * + * arm64 and RISC-V discriminate on TCB_FLAG_SYSCALL instead. x86_64 + * cannot: that flag also defers signal actions, which x86_64 has never + * done and which its kernel-build signal path does not currently survive + * -- see the note in x86_64_syscall(). xcp.sregs says exactly what is + * needed here and means nothing to anyone else. + */ + + if (parent->xcp.sregs != NULL) + { + return x86_64_fork_syscall(vfork, parent); + } +#endif + + return x86_64_fork_direct(vfork, parent, context); +} diff --git a/arch/x86_64/src/common/x86_64_syscall.c b/arch/x86_64/src/common/x86_64_syscall.c index 848f625c3ce..196453f5d6a 100644 --- a/arch/x86_64/src/common/x86_64_syscall.c +++ b/arch/x86_64/src/common/x86_64_syscall.c @@ -313,15 +313,42 @@ uint64_t *x86_64_syscall(uint64_t *regs) #ifdef CONFIG_LIB_SYSCALL int nbr = cmd - CONFIG_SYS_RESERVED; syscall_stub_t stub = (syscall_stub_t)g_stublookup[nbr]; - -#ifdef CONFIG_ARCH_KERNEL_STACK struct tcb_s *rtcb = nxsched_self(); + uint64_t *sregs; +#ifdef CONFIG_ARCH_KERNEL_STACK /* Store reference to user RSP for signals */ rtcb->xcp.saved_ursp = regs[REG_RSP]; #endif + /* Publish the caller's register context. up_fork() has to clone + * the caller rather than the stub that is about to invoke it, and + * this frame is the only description of it -- see + * x86_64_fork_syscall(), which also takes a non-NULL xcp.sregs as + * its "reached here through a system call" discriminator. + * + * It is saved and restored rather than simply set and cleared: + * x86_64_syscall_entry() has an explicit path for a nested system + * call, and when the inner one returns the outer one must still be + * described by its own frame. + * + * Note what is deliberately *not* done here. arm64 and RISC-V + * also raise TCB_FLAG_SYSCALL across the stub call, which defers + * any signal action until the system call returns. x86_64 has + * never set it, and making it do so is not free: the deferred + * action then has to be picked up by nxsig_unmask_pendingsignal() + * on the way out, and the signal dispatch path of an x86_64 kernel + * build does not survive that today -- it faults in + * x86_64_syscall_entry()'s return path with RSP == 0. That is a + * pre-existing bug in a configuration nothing has exercised, and + * fixing it does not belong to the fork/vfork work; so this + * records the frame and changes nothing else. + */ + + sregs = rtcb->xcp.sregs; + rtcb->xcp.sregs = regs; + /* Re-enable interrupts if enabled before. * Current task RFLAGS are stored in R11. */ @@ -334,6 +361,10 @@ uint64_t *x86_64_syscall(uint64_t *regs) /* Call syscall function and store return value in RAX register */ regs[REG_RAX] = stub(nbr, arg1, arg2, arg3, arg4, arg5, arg6); + + /* The system call is now done */ + + rtcb->xcp.sregs = sregs; #else svcerr("ERROR: Bad SYS call: %" PRId32 "\n", cmd); #endif
