On Friday 22 June 2007 16:02:59 Duncan wrote: > Was it you who posted about this before, or someone else? If it wasn't > you, take a look back thru the list a couple months, as it did come up > previously. You may have someone to compare notes with. =8^)
Nope, t'was I, and I've now tried lots more things but with no success. > Separate processes or separate threads? Two CPUs (um, two separate > sockets) or two cores on the same CPU/socket? Twin sockets, single Opteron 246 cores. And the BOINC scheduler sets off separate processes; they have distinct PIDs and on the old motherboard ran on distinct CPUs. > There are [...] differences in architecture between AMD (with its onboard > memory controller and closer cooperation, both between cores and between > CPUs on separate sockets, due to the direct Hypertransport links) and > Intel (with its off-chip controller and looser inter-core, inter- chip, > and inter-socket cooperation). There's also differences in the way you > can configure both the memory (thru the BIOS) and the kernel, for separate > NUMA access or unified view memory. If these settings don't match your > actual physical layout, efficiency will be less than peak, either because > there won't be enough resistance to a relatively high cost switching > between CPUs/cores and memory, so they'll switch off frequently > with little reason, incurring expensive delays each time, or because > there's too much resistance and to much favor placed on what the kernel > thinks is local vs remote memory, when it's all the same, and there is in > fact very little cost to switching cores/CPUs. This board has eight DIMM sockets, four arranged next to each CPU socket and associated with it electrically and logically. I have four 1GB DIMMs, each in (I hope) the right pair of sockets in each bank of four. In other words, each CPU has 2GB of local RAM. I suppose I could buy as much RAM again and fill up all the sockets :-) > If it's AMD with its onboard memory controllers, two sockets means two > controllers, and you'll also want to consider NUMA, tho you can disable it > and interleave your memory if you wish, for a unified memory view and > higher bandwidth, but at the tradeoff of higher latency and less efficient > memory access when separate tasks (each running on a CPU) both want to use > memory at the same time. NUMA is switched on in BIOS and kernel config. I still find some of the BIOS settings mysterious though, so perhaps I don't have it set up right. I have tried the two sets of defaults, failsafe and optimised, but with no effect on this problem. > In particular, you'll want to pay attention to the following kernel config > settings under Processor type and features: > > 1) Symmetric multiprocessing support (CONFIG_SMP). You probably have > this set right or you'd not be using multiple CPUs/cores. Yep. > 2) Under SMP, /possibly/ SMT (CONFIG_SCHED_SMT), tho for Intel only, and > on the older Hyperthreading Netburst arch models. Nope. > 3) Still under SMP, Multi-core scheduler support (CONFIG_SCHED_MC), if > you have true dual cores. Again, note that the first "dual core" Intel > units were simply two separate CPUs in the same package, so you probably > do NOT want this for them. Nope. > 4) Non Uniform Memory Access (NUMA) Support (CONFIG_NUMA) [...] You > probably DO want this on AMD multi-socket Opteron systems, BUT note that > there may be BIOS settings for this as well. It won't work so efficiently > if the BIOS setting doesn't agree with the kernel setting. Yep. > 5) If you have NUMA support enabled, you'll also want either Old style > AMD Opteron NUMA detection (CONFIG_K8_NUMA) or (preferred) ACPI NUMA > detection (CONFIG_X86_64_ACPI_NUMA). Tried both together and each separately. No difference. > 6) Make sure you do *NOT* have NUMA emulation (CONFIG_NUMA_EMU) enabled. No point in emulating something that's present. :-) > What I'm wondering, of course, is whether you have NUMA turned on when > you shouldn't, or don't have core scheduling turned on when you should, > thus artificially increasing the resistance to switching cores/cpus and > causing the stickiness. I don't think so. > If [...] you were correct when you said BOINC starts two > separate /processes/ (not threads), I'm sure I was correct. > or if BOINC happens to use the older/heavier Linux threads model (which > again will cause the threads to show up as separate processes), I can't be quite certain this isn't happening, but I'm nearly so. > There are two scheduling utility packages that include utilities to tie > processes to one or more specific processors. > > sys-process/schedutils is what I have installed. It's a collection of > separate utilities, including taskset, by which I can tell the kernel > which CPUs I want specific processes to run on. > If you prefer a single do-it-all scheduler-tool, perhaps easier to learn > if you plan to fiddle with more than simply which CPU a process runs on, > and want to learn it all at once, sys-process/schedtool may be more your > style. I'll look into those - thanks. -- Rgds Peter Humphrey Linux Counter 5290, Aug 93 -- [EMAIL PROTECTED] mailing list
