Module Name: src Committed By: pooka Date: Fri May 28 16:44:14 UTC 2010
Modified Files: src/sys/rump/librump/rumpkern: rump.c scheduler.c threads.c Log Message: Improve the CPU scheduler for a host MP system with multithreaded access. The old scheduler had a global freelist which caused a cache crisis with multiple host threads trying to schedule a virtual CPU simultaneously. The rump scheduler is different from a normal thread scheduler, so it has different requirements. First, we schedule a CPU for a thread (which we get from the host scheduler) instead of scheduling a thread onto a CPU. Second, scheduling points are at every entry/exit to/from the rump kernel, including (but not limited to) syscall entry points and hypercalls. This means scheduling happens a lot more frequently than in a normal kernel. For every lwp, cache the previously used CPU. When scheduling, attempt to reuse the same CPU. If we get it, we can use it directly without any memory barriers or expensive locks. If the CPU is taken, migrate. Use a lock/wait only in the slowpath. Be very wary of walking the entire CPU array because that does not lead to a happy cacher. The migration algorithm could probably benefit from improved heuristics and tuning. Even as such, with the new scheduler an application which has two threads making rlimit syscalls in a tight loop experiences almost 400% speedup. The exact speedup is difficult to pinpoint, though, since the old scheduler caused very jittery results due to cache contention. Also, the rump version is now 70% faster than the counterpart which calls the host kernel. To generate a diff of this commit: cvs rdiff -u -r1.171 -r1.172 src/sys/rump/librump/rumpkern/rump.c cvs rdiff -u -r1.14 -r1.15 src/sys/rump/librump/rumpkern/scheduler.c cvs rdiff -u -r1.8 -r1.9 src/sys/rump/librump/rumpkern/threads.c Please note that diffs are not public domain; they are subject to the copyright notices on the relevant files.