I am not that expert on Debian nor Linux Kernel, but did you try
benchmark the IPC and compare the result to same CPU to rule out any
insulation issues?
also, you can try to run the program using gamemode (package), which
optimize performance.
following from the their github repo:
Currently GameMode includes support for optimisations including:
* CPU governor
* I/O priority
* Process niceness
* Kernel scheduler (|SCHED_ISO|)
* Screensaver inhibiting
* GPU performance mode (NVIDIA and AMD), GPU overclocking (NVIDIA)
* CPU core pinning or parking
* Custom scripts
