I had occasion a couple of days ago to try to block-copy a very large filesystem from a xen dom0 to another machine across a fast local network.
I tried this: sysctl -w kern.sbmax=10000000 sysctl -w net.inet.tcp.sendbuf_auto=0 dd if=/dev/rsd0g bs=1048576 | ttcp -t -b 2097152 -l 131072 -fm -p 9001 172.0.0.1 On the far end, I had simply the corresponding ttcp -r job (with corresponding socket/tcp sysctls in place) redirected to a file. The underlying hardware is a RAID5 of 300GB SAS drives on an Areca 1680. The Ethernet controller in use is a 'wm'. All supported offload functionality is turned on (turning it off just slows things down). Shockingly, I saw the system go to 0% idle time, with 45-55% "Interrupt" and the rest "System". Interrupts per second were a comparatively low 1500, about 500 disk and 1000 network. Throughput was horrible -- about 35MB/sec. I rebooted the system to a GENERIC kernel; throughput was 100MB/sec and peaked at 200MB/sec when I piped the data through gzip -1. The system was mostly idle with under 2% "Interrupt" time. Interrupts/sec were significantly higher due to both the higher data rate and about 800 TLB shootdown interrupts/sec (the GENERIC kernel runs MP on this system). Kernels are both netbsd-7 branch from just around the time of the release, XEN3_DOM0 and GENERIC. Neither kernel has DEBUG nor DIAGNOSTIC nor KMEMSTATS. Do we have some kind of terrible problem that causes a 20% increase in interrupt time when running under Xen? Note the Areca driver (arcmsr) is a SCSI driver so it is giant-locked, but the XEN dom0 is UP, so this is probably a red herring. I am going to install a different disk controller and see what happens, but I am not totally optimistic. Does anyone have any idea what might be going on? -- Thor Lancelot Simon t...@panix.com "We cannot usually in social life pursue a single value or a single moral aim, untroubled by the need to compromise with others." - H.L.A. Hart