When running trace-cmd record, I encounter multiple "No such device"
errors due to the splice syscall failing with ENODEV for
/sys/kernel/tracing/per_cpu/cpuX/trace_pipe_raw files corresponding to
CPUs that are possible but not present.
Environment:
- Operating System: Red Hat Enterprise Linux 9.5 (Plow)
- Kernel: Linux 6.12.0-55.9.1.el10_0.ppc64le
- Architecture: ppc64le
- Command Executed: trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
- System CPU Configuration:
- /sys/devices/system/cpu/possible: 0-223
- /sys/devices/system/cpu/present: 0-63
- /sys/kernel/tracing/per_cpu/: Contains directories cpu0 to cpu223
Steps to Reproduce:
1. On a system with a mismatch between possible and present CPUs (e.g.,
possible: 0-223, present: 0-63).
2. Ensure /sys/kernel/tracing/per_cpu/ contains directories for all possible
CPUs (e.g., cpu0 to cpu223).
3. Run the command:
strace --decode-fds -kk -o strace/trace-cmd -ff -- trace-cmd record -o
/dev/null -v -e sched_switch -- sleep 1
4. Observe the output and strace logs.
Expected Behavior: trace-cmd should only attempt to access
trace_pipe_raw files for CPUs that are present (e.g., cpu0 to cpu63),
completing the recording without errors.
Actual Behavior:
trace-cmd fails with multiple errors:
trace-cmd: No such device
recorder error in splice input
The strace output shows splice syscalls failing with ENODEV for
trace_pipe_raw files of non-present CPUs
(e.g., /sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw).
Example:
trace-cmd.47140:splice(5</sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw>,
NULL, 7<pipe:[96498]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
Analysis:
- trace-cmd iterates over all directories in
/sys/kernel/tracing/per_cpu/ (which includes cpu0 to cpu223 for all
possible CPUs).
- It attempts to invoke splice on trace_pipe_raw for each CPU, including
those not present (e.g., cpu64 to cpu223).
- The splice syscall fails with ENODEV for non-present CPUs.
- Stack trace (from strace -kk):
splice(5</sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw>, ...)
> /usr/lib64/glibc-hwcaps/power10/libc.so.6(splice+0x74)
> /usr/bin/trace-cmd(splice_data+0x3b)
> /usr/bin/trace-cmd(tracecmd_start_recording+0x93)
> /usr/bin/trace-cmd(create_recorder+0x14f)
> /usr/bin/trace-cmd(start_threads+0x81b)
> /usr/bin/trace-cmd(record_trace.isra.0+0xf5f)
> /usr/bin/trace-cmd(trace_record+0x47)
> /usr/bin/trace-cmd(main+0xff)
# strace --decode-fds -kk -o strace/trace-cmd -ff -- trace-cmd record -o
/dev/null -v -e sched_switch -- sleep 1
trace-cmd: No such device
trace-cmd: No such device
recorder error in splice input recorder error in splice input
recorder error in splice inputtrace-cmd: No such device
recorder error in splice input
trace-cmd: No such device
<snipped>
recorder error in splice input
trace-cmd: No such device
recorder error in splice input
recorder error in splice inputtrace-cmd: No such device
recorder error in splice inputtrace-cmd: No such device
recorder error in splice input
trace-cmd: No such device
trace-cmd: No such device
recorder error in splice input
recorder error in splice inputtrace-cmd: No such device
recorder error in splice input
trace-cmd: No such device
recorder error in splice inputtrace-cmd: No such device
recorder error in splice inputtrace-cmd: No such device
<snipped>
recorder error in splice input
recorder error in splice inputtrace-cmd: No such device
recorder error in splice inputtrace-cmd: No such device
recorder error in splice inputtrace-cmd: No such device
recorder error in splice input
CPU0 data recorded at offset=0x10000
0 bytes in size
CPU1 data recorded at offset=0x10000
0 bytes in size
<snipped>
CPU10 data recorded at offset=0x10000
0 bytes in size
CPU11 data recorded at offset=0x10000
<snipped>
0 bytes in size
CPU63 data recorded at offset=0x10000
0 bytes in size
CPU64 data recorded at offset=0x10000
0 bytes in size
<snipped?
CPU213 data recorded at offset=0x10000
0 bytes in size
<snipped>
CPU222 data recorded at offset=0x10000
0 bytes in size
CPU223 data recorded at offset=0x10000
0 bytes in size
# grep -i "such device" * | grep splice
trace-cmd.46981:splice(5</sys/kernel/tracing/per_cpu/cpu64/trace_pipe_raw>,
NULL, 7<pipe:[86367]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46982:splice(5</sys/kernel/tracing/per_cpu/cpu65/trace_pipe_raw>,
NULL, 7<pipe:[82351]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46983:splice(5</sys/kernel/tracing/per_cpu/cpu66/trace_pipe_raw>,
NULL, 7<pipe:[81717]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46984:splice(5</sys/kernel/tracing/per_cpu/cpu67/trace_pipe_raw>,
NULL, 7<pipe:[1932]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46985:splice(5</sys/kernel/tracing/per_cpu/cpu68/trace_pipe_raw>,
NULL, 7<pipe:[92520]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46986:splice(5</sys/kernel/tracing/per_cpu/cpu69/trace_pipe_raw>,
NULL, 7<pipe:[93494]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
<snipped>
trace-cmd.47136:splice(5</sys/kernel/tracing/per_cpu/cpu219/trace_pipe_raw>,
NULL, 7<pipe:[19742]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47137:splice(5</sys/kernel/tracing/per_cpu/cpu220/trace_pipe_raw>,
NULL, 7<pipe:[29868]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47138:splice(5</sys/kernel/tracing/per_cpu/cpu221/trace_pipe_raw>,
NULL, 7<pipe:[42069]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47139:splice(5</sys/kernel/tracing/per_cpu/cpu222/trace_pipe_raw>,
NULL, 7<pipe:[81737]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47140:splice(5</sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw>,
NULL, 7<pipe:[96498]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
