> you do one PTRACE_SYSCALL + WAITPID and then PEEKUSER %rax getting -ENOSYS. > -ENOSYS is a syscall return value which should be returned after the _second_ > PTRACE_SYSCALL - on the syscall exit, not on the syscall entry, shouldn't be? > I would expect PEEKUSER %rax should still return -23 on first PTRACE_SYSCALL.
The details of registers at syscall tracing points vary by machine. What follows is about x86 and other machines will not necessarily have analogous rules. > Also on the syscall entry POKEUSER %rax should set the syscall number > (SYS_open etc.), not the result value (-ENOTTY). On i386, it has always been that at entry tracing, eax = -ENOSYS and orig_eax = incoming %eax value (syscall #). Here, ptrace can change the value of orig_eax to affect what syscall the thread will attempt. If the orig_eax value is invalid (< 0 or > __NR_syscall_max), then no syscall is attempted and %eax is returned as it is. This means that for an invalid syscall # in orig_eax--whether originally there or put there by ptrace while at the entry stop--whatever register state (eax included) that ptrace left there after the entry tracing stop will be what the user sees. Thus you can use syscall entry tracing to do what PTRACE_SYSEMU does, which is to let the debugger intercept and simulate system call attempts precisely how it chooses. This is simpler than tweaking at both entry and exit tracing just to jump around the syscall and set the eax value you want. This matters even more in utrace, because you can request entry tracing only and not exit tracing (so it stays quite simple). On current x86_64 kernels, 32-bit tasks fail to behave this way. When orig_eax is invalid, they clobber eax back to -ENOSYS upon resuming from syscall entry tracing. 64-bit tasks never behaved the i386 way before, they always clobbered rax to -ENOSYS. I've sent patches upstream that change x86_64 to match the i386 behavior for both 32-bit tasks (fixing the compatibility regression) and for 64-bit tasks (so you can treat x86_64 uniformly with i386). I think these will go into 2.6.26, but will not go in before that. Thanks, Roland
