Here's a few tests I ran. Hardware is a Dell optiplex 755 (core 2 duo, WDC WD20EARS-00MVWB0, on Intel AHCI):
Xen45/netbsd-6: borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 53.957 secs (121459680 bytes/sec) systat says: 11.4% Sy 0.8% Us 0.0% Ni 2.4% In 85.4% Id 1850 ops/s, 1850 irq/s on ioapic0 pin 18 netbsd-6 bare metal, booted with -1 (as XEN3_DOM0 is not SMP yet): borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 53.882 secs (121628744 bytes/sec) systat says: 2.6% Sy 0.0% Us 0.0% Ni 1.8% In 95.6% Id about 1850 ops/s, 1850 irq/s on ioapic0 pin 18 xen45/netbsd-7: borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 54.258 secs (120785874 bytes/sec) systat says: 15.8% Sy 0.8% Us 0.0% Ni 0.6% In 82.8% Id pages about 1850 ops/s, 1850 irq/s on ioapic0 pin 18 netbsd-7 bare metal, booted with -1 (as XEN3_DOM0 is not SMP yet): borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 54.175 secs (120970927 bytes/sec) systat says: 3.6% Sy 0.2% Us 0.0% Ni 1.4% In 94.8% Id about 1850 ops/s, 1850 irq/s on ioapic0 pin 18 xen31/netbsd-7: borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out systat says: 15.6% Sy 0.2% Us 0.0% Ni 0.8% In 83.4% Id about 1850 ops/s, 1850 irq/s on ioapic0 pin 18 No, I'm not seeing twice the interrupts as you see with the SCSI controllers. I suspect it could be interrupt coalescing, which may kick in with a bare-metal kernel and not with the Xen one, as the Xen one is slower. For I/Os, Xen is not slower in my case but uses much more system time. This is probably because on this hardware, the limiting factor is the hard disk and not the CPU. This time is not spent in hard interrupt handlers, I actually think it's MMU-related (page remapping or invalidations). It makes sense, as in the Xen/PV case these operations needs an hypercall. There's no big differences between netbsd-6 and netbsd-7, or Xen versions. I re-ran tests with the attached program, which should avoid reading data from disk (the data should be in the disks's cache, so we're benchmarking the SATA link/controller, and the data move between kernel and userland). On Xen/netbsd-7: 23.4% Sy 0.4% Us 0.0% Ni 0.8% In 75.4% Id borneo:/home/bouyer#./tst /dev/rwd0d 100000 37753400 us, 165.548004 MB/s UP bare-metal: 4.0% Sy 0.0% Us 0.0% Ni 3.6% In 92.4% Id borneo:/home/bouyer#./tst /dev/rwd0d 100000 30945314 us, 201.969190 MB/s Here we have a true difference (also, Xen/netbsd-7 can't keep the disk more than 70% busy, while bare metal is more than 99% - maybe because interrupt latency is higher with Xen). The result is the same with different Xen versions. Maybe some improvement is possible in pmap.c, this needs to be looked at. But I don't think we'll get more than a few %. Maybe also a XEN3PAE_DOM0 would do better (with amd64/xen the kernel runs unprivileged and has a completely separate vm space from userland, while i386PAE kernel runs in ring 1 and shares its vm space with userland - this makes a big difference in pmap). I never tried, but it should be possible to run a amd64 Xen with a i386pae dom0. The real fix would be to run the dom0 kernel as a HVM domain on hardware that supports it. But this requires quite a bit of work. -- Manuel Bouyer <bou...@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference
#include <fcntl.h> #include <unistd.h> #include <sys/time.h> main(int argc, char **argv) { static char buf[64*1024]; int fd, i; struct timeval tv0, tv1; int t; fd = open(argv[1], O_RDONLY, 0); if (fd < 0) { perror("open"); exit(1); } if (gettimeofday(&tv0, NULL) < 0) { perror("gettimeofday"); exit(1); } for (i = 0; i < atoi(argv[2]); i++) { if (read(fd, buf, sizeof(buf)) != sizeof(buf)) { perror("read"); exit(1); } if (lseek(fd, 0, SEEK_SET) < 0) { perror("seek"); exit(1); } } if (gettimeofday(&tv1, NULL) < 0) { perror("gettimeofday"); exit(1); } t = (tv1.tv_sec - tv0.tv_sec) * 1000000; t = t + tv1.tv_usec - tv0.tv_usec; printf("%d us, %f MB/s\n", t, ((double)64 * (double)i / 1024) / ((double)t / 1000000)); exit(0); }