I had a possibly related symptom described in https://forums.debian.net/viewtopic.php?p=743616#p743616 with my fix described in https://forums.debian.net/viewtopic.php?p=743676#p743676.
In short, a USB device seems to have caused issues with the kernel that led to the increased network latency. For me, removing the USB device completely fixed my network latency problems.
