Re: Getting # Physical CPUs
dsimcha wrote: When I do parallel programming in D, it's nice to have a sane default for the number of threads. The ideal number is the number of cores that can see the current address space. core.cpuid provides coresPerCPU, but it doesn't appear to provide nPhysialCPUs or something similar, which would allow the total number of physical CPUs in the system to be determined. Is there any easy way to determine this in D2? coresPerCPU is a misnomer. It's actually the total number of cores. Will fix.
Re: Getting # Physical CPUs
== Quote from Don (nos...@nospam.com)'s article
> dsimcha wrote:
> > When I do parallel programming in D, it's nice to have a sane default for
> > the
> > number of threads. The ideal number is the number of cores that can see the
> > current address space. core.cpuid provides coresPerCPU, but it doesn't
> > appear
> > to provide nPhysialCPUs or something similar, which would allow the total
> > number of physical CPUs in the system to be determined. Is there any easy
> > way
> > to determine this in D2?
> coresPerCPU is a misnomer. It's actually the total number of cores. Will
> fix.
Ok, then maybe this is a bug and belongs in bugzilla. Here's a little test
program I wrote:
import core.cpuid, std.stdio;
void main() {
writeln("Cores: ", coresPerCPU, " Threads: ", threadsPerCPU);
}
This prints the following on the machine in question:
Cores: 1 Threads: 2
Here's a cat/proc/cpuinfo on the same machine:
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name :Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping: 1
cpu MHz : 3657.825
cache size : 1024 KB
physical id : 0
siblings: 2
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips: 7321.79
clflush size: 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name :Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping: 1
cpu MHz : 3657.825
cache size : 1024 KB
physical id : 4
siblings: 2
core id : 4
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips: 7314.82
clflush size: 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name :Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping: 1
cpu MHz : 3657.825
cache size : 1024 KB
physical id : 3
siblings: 2
core id : 3
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips: 7315.15
clflush size: 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name :Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping: 1
cpu MHz : 3657.825
cache size : 1024 KB
physical id : 7
siblings: 2
core id : 7
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips: 7315.01
clflush size: 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name :Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping: 1
cpu MHz : 3657.825
cache size : 1024 KB
physical id : 0
siblings: 2
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor
ds_cpl est tm2 cid cx16 xtpr
bogomips: 7315.01
clflush size: 64
cache_alignment : 128
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name :Intel(R) Xeon(TM) MP CPU 3.66GHz
stepping: 1
cpu MHz : 3657.825
cache size : 1024 KB
physical id : 4
siblings: 2
core id : 4
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse t
Re: Getting # Physical CPUs
dsimcha Wrote:
> When I do parallel programming in D, it's nice to have a sane default for the
> number of threads. The ideal number is the number of cores that can see the
> current address space. core.cpuid provides coresPerCPU, but it doesn't appear
> to provide nPhysialCPUs or something similar, which would allow the total
> number of physical CPUs in the system to be determined. Is there any easy way
> to determine this in D2?
dsimcha,
This is a relatively difficult problem in general to do portably due to
hardware differences, topology differences, changes to hardware, OS variations.
Even the pthreads library doesn't reliably implement it in a portable manner.
I came to the conclusion that the people most motivated to keep up to date on a
portable CPU core topology are the national supercomputing labs. INRIA and
various US labs came up with "Portable Hardware Locality" library. It gives
you *everything* you need to discover the number of CPU sockets, memory
architecture, number of cores per socket, control cpu affinity etc.
The HWLoc C libraries are written by the open-mpi project here:
http://www.open-mpi.org/projects/hwloc/
I created an interface to their C library called HWLoc.d and I haven't
published it anywhere. I just needed it for my multi-core Dendrite system.
Give me an email you want it sent to and it's yours. I could post it here, but
it would be a lot of noise in the discussion. I'll probably post it on dsource
later.
My license on all of my Dendrite code is zlib (which means you can do almost
anything except claim that you wrote the original) and the open-mpi C libraries
are new BSD, so have at it.
You'll obviously have to download and install the hwloc c libraries also.
Here's some output from "hwhello.d" on a 2 socket system:
*** Objects at level 0
CPU Set: 0003
Index 0: System(1002MB)
*** Objects at level 1
CPU Set: 0001
Index 0: Socket#0
CPU Set: 0002
Index 1: Socket#2
*** Objects at level 2
CPU Set: 0001
Index 0: L2(2048KB)
CPU Set: 0002
Index 1: L2(2048KB)
*** Objects at level 3
CPU Set: 0001
Index 0: L1(16KB)
CPU Set: 0002
Index 1: L1(16KB)
*** Objects at level 4
CPU Set: 0001
Index 0: Core#0
CPU Set: 0002
Index 1: Core#0
*** Objects at level 5
CPU Set: 0001
Index 0: P#0
CPU Set: 0002
Index 1: P#1
*** Printing overall tree
System(1002MB)
Socket#0
L2(2048KB)
L1(16KB)
Core#0
P#0
Socket#2
L2(2048KB)
L1(16KB)
Core#0
P#1
*** 2 socket(s)
depth is now: 4
number objects at this depth: 2
ob 0: 0001
ob 0: 0002
Last CPU Set: 0002
Singleified CPU Set: 0002
burning CPU
... and here's the code snippet from hwloc.d that generates the last part of
that output:
/* Get last level. */
obj = hwloc_get_obj_by_depth(topology, depth,
hwloc_get_nbobjs_by_depth(topology, depth) - 1);
if (obj) {
/* Get a copy of its cpuset that we may modify. */
cpuset = hwloc_cpuset_dup(obj.cpuset);
// convert it to a string
nchars = hwloc_cpuset_snprintf(string.ptr, string.sizeof, cpuset);
Stdout("Last CPU Set: ")(string[0..nchars]).newline;
/* Get only one logical processor (in case the core is
SMT/hyperthreaded). */
hwloc_cpuset_singlify(cpuset);
nchars = hwloc_cpuset_snprintf(string.ptr, string.sizeof, cpuset);
Stdout("Singleified CPU Set: ")(string[0..nchars]).newline;
/* And try to bind ourself there. */
if (hwloc_set_cpubind(topology, cpuset, 0)) {
char* str;
nchars = hwloc_cpuset_asprintf(&str, obj.cpuset);
Stdout.formatln("Couldn't bind to cpuset {}", str[0..nchars]);
free(str);
}
Stdout("burning CPU").newline;
/* do something we can see in htop */
for (j=0; j<1; j++) f+=f/2;
/* Free our cpuset copy */
hwloc_cpuset_free(cpuset);
}
finally here's the output from the 'lstopo' utility in the hwloc distro
that displays the topology visually (mine is a 4 socket, hexacore NUMA machine.
Use a fixed font to view it).
┌┐
│ System(4096MB) │
││
│
┌┐
│
│ │
┌┐
│ │
│ │ │ Node#0(1024MB)
│ │ │
│ │
└┘
│ │
│ │
│ │
│ │ ┌──┐
┌──┐
┌──┐
┌──┐
┌──┐
┌──┐ │ │
│ │ │ P#0 │ │ P#1 │ │ P#2
│ │ P#3 │ │ P#4 │ │ P#5 │
│ │
│ │ └──┘
└──┘
└──┘
└──┘
└──┘
└──┘ │ │
│
└
Re: Getting # Physical CPUs
== Quote from eris (jvbur...@gmail.com)'s article > This is a relatively difficult problem in general to do portably due to > hardware differences, topology differences, changes to hardware, OS variations. Even the pthreads library doesn't reliably implement it in a portable manner. > I came to the conclusion that the people most motivated to keep up to date on > a portable CPU core topology are the national supercomputing labs. INRIA and various US labs came up with "Portable Hardware Locality" library. It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc. > The HWLoc C libraries are written by the open-mpi project here: http://www.open-mpi.org/projects/hwloc/ I appreciate the help, but honestly, if detecting this properly requires adding dependencies to my projects, I'm happier with the simple workaround of having a manual command line switch to specify the number of CPUs. The projects in question are internal research projects, not things that are going to be released on the computer-illiterate masses. It would be nice to not have to manually specify such a parameter on every run, but not nice enough to be worth introducing a dependency.
Re: Getting # Physical CPUs
dsimcha wrote: == Quote from Don (nos...@nospam.com)'s article dsimcha wrote: When I do parallel programming in D, it's nice to have a sane default for the number of threads. The ideal number is the number of cores that can see the current address space. core.cpuid provides coresPerCPU, but it doesn't appear to provide nPhysialCPUs or something similar, which would allow the total number of physical CPUs in the system to be determined. Is there any easy way to determine this in D2? coresPerCPU is a misnomer. It's actually the total number of cores. Will fix. Ok, then maybe this is a bug and belongs in bugzilla. Here's a little test program I wrote: [snip] Thanks, that's definitely a bug. The code in core.cpuid has not been tested on the most recent CPUs (Intel added a totally new method) and their documentation is quite convoluted. It's hard to get it right without an actual machine.
Re: Getting # Physical CPUs
== Quote from Don (nos...@nospam.com)'s article > [snip] > Thanks, that's definitely a bug. The code in core.cpuid has not been > tested on the most recent CPUs (Intel added a totally new method) and > their documentation is quite convoluted. It's hard to get it right > without an actual machine. Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462
Re: Getting # Physical CPUs
dsimcha wrote: == Quote from Don (nos...@nospam.com)'s article [snip] Thanks, that's definitely a bug. The code in core.cpuid has not been tested on the most recent CPUs (Intel added a totally new method) and their documentation is quite convoluted. It's hard to get it right without an actual machine. Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462 Please check if the latest druntime commit fixes this.
Re: Getting # Physical CPUs
== Quote from Don (nos...@nospam.com)'s article > dsimcha wrote: > > == Quote from Don (nos...@nospam.com)'s article > >> [snip] > >> Thanks, that's definitely a bug. The code in core.cpuid has not been > >> tested on the most recent CPUs (Intel added a totally new method) and > >> their documentation is quite convoluted. It's hard to get it right > >> without an actual machine. > > > > Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462 > Please check if the latest druntime commit fixes this. Thanks. Unfortunately I can't test this because the Linux build script for Phobos is broken on my machine in some inscrutable way. Frankly, my success rate at building stuff from other people's make files is well under 50%. Make is just a horrible technology that needs to die a horrible death. We should be eating our own dogfood and using rdmd for build scripts. Here's the error message I'm getting. I know basically nothing about make except that it's a build system and that it almost never works, so I can't even begin to debug this. Here's the error message I've been getting, on a freshly unpacked 2.047 directory on some ancient Linux distro that my sys admin insists on using: $ make -flinux.mak make --no-print-directory -f OS=posix BUILD=release make[1]: OS=posix: No such file or directory make[1]: *** No rule to make target `OS=posix'. Stop. make: *** [release] Error 2 And here's the error I get when I try on a different machine w/ a more modern distro (this one is probably due to lack of 64 bit libs): $ make -flinux.mak make --no-print-directory -f linux.mak OS=posix BUILD=release cc -c -m32 -O3 etc/c/zlib/adler32.c -ogenerated/posix/release/etc/c/zlib/adler32.o cc -c -m32 -O3 etc/c/zlib/compress.c -ogenerated/posix/release/etc/c/zlib/compress.o cc -c -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o In file included from /usr/include/features.h:378, from /usr/include/string.h:26, from etc/c/zlib/zutil.h:23, from etc/c/zlib/crc32.c:29: /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1 make: *** [release] Error 2
Re: Getting # Physical CPUs
dsimcha wrote: == Quote from Don (nos...@nospam.com)'s article dsimcha wrote: == Quote from Don (nos...@nospam.com)'s article [snip] Thanks, that's definitely a bug. The code in core.cpuid has not been tested on the most recent CPUs (Intel added a totally new method) and their documentation is quite convoluted. It's hard to get it right without an actual machine. Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462 Please check if the latest druntime commit fixes this. Thanks. Unfortunately I can't test this because the Linux build script for Phobos is broken on my machine in some inscrutable way. Frankly, my success rate at building stuff from other people's make files is well under 50%. Make is just a horrible technology that needs to die a horrible death. We should be eating our own dogfood and using rdmd for build scripts. I agree. In this case, core.cpuid is completely stand-alone. So you could just copy it into another directory and change the module statement.
Re: Getting # Physical CPUs
== Quote from Don (nos...@nospam.com)'s article > dsimcha wrote: > > == Quote from Don (nos...@nospam.com)'s article > >> dsimcha wrote: > >>> == Quote from Don (nos...@nospam.com)'s article > [snip] > Thanks, that's definitely a bug. The code in core.cpuid has not been > tested on the most recent CPUs (Intel added a totally new method) and > their documentation is quite convoluted. It's hard to get it right > without an actual machine. > >>> Bug 4462. http://d.puremagic.com/issues/show_bug.cgi?id=4462 > >> Please check if the latest druntime commit fixes this. > > > > Thanks. Unfortunately I can't test this because the Linux build script for > > Phobos > > is broken on my machine in some inscrutable way. Frankly, my success rate > > at > > building stuff from other people's make files is well under 50%. Make is > > just a > > horrible technology that needs to die a horrible death. We should be > > eating our > > own dogfood and using rdmd for build scripts. > I agree. > In this case, core.cpuid is completely stand-alone. So you could just > copy it into another directory and change the module statement. Great idea. Unfortunately that still doesn't fix it. I get different wrong information on some machines, but it's still wrong, and the specific example I posted to Bugzilla hasn't changed at all.
Re: Getting # Physical CPUs
As a side note, why is there both a std.cpuid and a core.cpuid? Does std use core? In that case, why not import std.cpuid?
Re: Getting # Physical CPUs
Philippe Sigaud wrote: As a side note, why is there both a std.cpuid and a core.cpuid? Does std use core? No. In that case, why not import std.cpuid? std.cpuid is deprecated.
Re: Getting # Physical CPUs
dsimcha wrote: Here's the error message I'm getting. I know basically nothing about make except that it's a build system and that it almost never works, so I can't even begin to debug this. Here's the error message I've been getting, on a freshly unpacked 2.047 directory on some ancient Linux distro that my sys admin insists on using: $ make -flinux.mak make --no-print-directory -f OS=posix BUILD=release make[1]: OS=posix: No such file or directory make[1]: *** No rule to make target `OS=posix'. Stop. make: *** [release] Error 2 The "OS=posix" sets the macro OS to the value posix, it does not set the target. This has been a feature of make since at least the 1980's, earlier than Linux even existed. So I'm astonished you're seeing this error. And here's the error I get when I try on a different machine w/ a more modern distro (this one is probably due to lack of 64 bit libs): $ make -flinux.mak make --no-print-directory -f linux.mak OS=posix BUILD=release cc -c -m32 -O3 etc/c/zlib/adler32.c -ogenerated/posix/release/etc/c/zlib/adler32.o cc -c -m32 -O3 etc/c/zlib/compress.c -ogenerated/posix/release/etc/c/zlib/compress.o cc -c -m32 -O3 etc/c/zlib/crc32.c -ogenerated/posix/release/etc/c/zlib/crc32.o In file included from /usr/include/features.h:378, from /usr/include/string.h:26, from etc/c/zlib/zutil.h:23, from etc/c/zlib/crc32.c:29: /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1 make: *** [release] Error 2 This is most likely because you have not gotten the 32 bit dev system installed on your 64 bit system (it usually is not installed by the default linux install). Which one do you have? BTW, using a script rather than make wouldn't have helped you with the second issue.
Re: Getting # Physical CPUs
== Quote from Walter Bright (newshou...@digitalmars.com)'s article > > And here's the error I get when I try on a different machine w/ a more > > modern > > distro (this one is probably due to lack of 64 bit libs): > > > > $ make -flinux.mak > > make --no-print-directory -f linux.mak OS=posix BUILD=release > > cc -c -m32 -O3 etc/c/zlib/adler32.c -ogenerated/posix/release/etc/c/zlib/adler32.o > > cc -c -m32 -O3 etc/c/zlib/compress.c -ogenerated/posix/release/etc/c/zlib/compress.o > > cc -c -m32 -O3 etc/c/zlib/crc32.c > > -ogenerated/posix/release/etc/c/zlib/crc32.o > > In file included from /usr/include/features.h:378, > > from /usr/include/string.h:26, > > from etc/c/zlib/zutil.h:23, > > from etc/c/zlib/crc32.c:29: > > /usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or > > directory > > make[1]: *** [generated/posix/release/etc/c/zlib/crc32.o] Error 1 > > make: *** [release] Error 2 > This is most likely because you have not gotten the 32 bit dev system > installed > on your 64 bit system (it usually is not installed by the default linux > install). Which one do you have? > BTW, using a script rather than make wouldn't have helped you with the second > issue. Yeah, in my original most I meant to say it was probably due to lack of 32-bit libs, which I don't have root privileges to install on the machine in question. It can see the same file systems via NFS as some machines that do have 32-bit libs installed, so my usual kludge is to always use a machine w/ 32-bit libs for building. Even so, I only figured this out b/c I was previously aware of the problem. The point isn't that make failed here, it's that it failed with an absolutely inscrutable error message. I wouldn't have even been able to begin guessing what was wrong if I didn't already know about the 32-bit lib issue.
Re: Getting # Physical CPUs
Walter Bright, el 15 de julio a las 11:40 me escribiste: > dsimcha wrote: > >Here's the error message I'm getting. I know basically nothing about make > >except > >that it's a build system and that it almost never works, so I can't even > >begin to > >debug this. Here's the error message I've been getting, on a freshly > >unpacked > >2.047 directory on some ancient Linux distro that my sys admin insists on > >using: > > > >$ make -flinux.mak > >make --no-print-directory -f OS=posix BUILD=release > >make[1]: OS=posix: No such file or directory > >make[1]: *** No rule to make target `OS=posix'. Stop. > >make: *** [release] Error 2 > > The "OS=posix" sets the macro OS to the value posix, it does not set > the target. This has been a feature of make since at least the > 1980's, earlier than Linux even existed. So I'm astonished you're > seeing this error. This is even standard POSIX: http://www.opengroup.org/onlinepubs/009695399/utilities/make.html (see the OPERANDS section) -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ -- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) -- - Que hacés, ratita? - Espero un ratito...
Re: Getting # Physical CPUs
Walter Bright Wrote: > > > > $ make -flinux.mak > > make --no-print-directory -f OS=posix BUILD=release > > make[1]: OS=posix: No such file or directory > > make[1]: *** No rule to make target `OS=posix'. Stop. > > make: *** [release] Error 2 > > The "OS=posix" sets the macro OS to the value posix, it does not set the > target. > This has been a feature of make since at least the 1980's, earlier than Linux > even existed. So I'm astonished you're seeing this error. Looks to me like a macro isn't being set. In the first output line it has the argument '-f ' there are two spaces after requesting to look in a file. So the file name it sees is OS=posix
Re: Getting # Physical CPUs
On 07/14/2010 08:55 PM, dsimcha wrote: == Quote from eris (jvbur...@gmail.com)'s article This is a relatively difficult problem in general to do portably due to hardware differences, topology differences, changes to hardware, OS variations. Even the pthreads library doesn't reliably implement it in a portable manner. I came to the conclusion that the people most motivated to keep up to date on a portable CPU core topology are the national supercomputing labs. INRIA and various US labs came up with "Portable Hardware Locality" library. It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc. The HWLoc C libraries are written by the open-mpi project here: http://www.open-mpi.org/projects/hwloc/ I appreciate the help, but honestly, if detecting this properly requires adding dependencies to my projects, I'm happier with the simple workaround of having a manual command line switch to specify the number of CPUs. The projects in question are internal research projects, not things that are going to be released on the computer-illiterate masses. It would be nice to not have to manually specify such a parameter on every run, but not nice enough to be worth introducing a dependency. I can't imagine how this would not be a required part of the core library. For a language that claims to be thread savvy, knowing the number of cpus and the number of cores, is simply obligatory homework. An extra point: the code that identifies them, should not ever assume that all cores are identical. Nor that they have identical access to machine resources. The day that someone invents the 'unequal cores paradigm', where cores of dissimilar power are included in the same computer, should not expose us with our pants down. (A case in point, at bootup, the Linux core already enumerates and evaluates each found core individually.)
Re: Getting # Physical CPUs
Georg Wrede wrote: On 07/14/2010 08:55 PM, dsimcha wrote: == Quote from eris (jvbur...@gmail.com)'s article This is a relatively difficult problem in general to do portably due to hardware differences, topology differences, changes to hardware, OS variations. Even the pthreads library doesn't reliably implement it in a portable manner. I came to the conclusion that the people most motivated to keep up to date on a portable CPU core topology are the national supercomputing labs. INRIA and various US labs came up with "Portable Hardware Locality" library. It gives you *everything* you need to discover the number of CPU sockets, memory architecture, number of cores per socket, control cpu affinity etc. The HWLoc C libraries are written by the open-mpi project here: http://www.open-mpi.org/projects/hwloc/ I appreciate the help, but honestly, if detecting this properly requires adding dependencies to my projects, I'm happier with the simple workaround of having a manual command line switch to specify the number of CPUs. The projects in question are internal research projects, not things that are going to be released on the computer-illiterate masses. It would be nice to not have to manually specify such a parameter on every run, but not nice enough to be worth introducing a dependency. I can't imagine how this would not be a required part of the core library. For a language that claims to be thread savvy, knowing the number of cpus and the number of cores, is simply obligatory homework. An extra point: the code that identifies them, should not ever assume that all cores are identical. Nor that they have identical access to machine resources. The day that someone invents the 'unequal cores paradigm', where cores of dissimilar power are included in the same computer, should not expose us with our pants down. It really depends on what the purpose is. If you want to determine the precise core topology, the available information is heavily OS-dependent. Note that there's potentially a large difference between the number of cores in the machine, versus the number of cores which the OS makes available to your app. Generally the second number is the one which matters. (A case in point, at bootup, the Linux core already enumerates and evaluates each found core individually.) Of course it does. It's trivial when you're an OS and have unrestricted access to the machine. An app is severely limited to what it can get from the OS. Currently core.cpuid doesn't make any OS calls at all. I think std.cpuid should be replaced with a new module std.sysinfo, which determines more features (such as available RAM).
Re: Getting # Physical CPUs
On Thu, 15 Jul 2010 14:40:57 -0400, Walter Bright wrote: dsimcha wrote: Here's the error message I'm getting. I know basically nothing about make except that it's a build system and that it almost never works, so I can't even begin to debug this. Here's the error message I've been getting, on a freshly unpacked 2.047 directory on some ancient Linux distro that my sys admin insists on using: $ make -flinux.mak make --no-print-directory -f OS=posix BUILD=release make[1]: OS=posix: No such file or directory make[1]: *** No rule to make target `OS=posix'. Stop. make: *** [release] Error 2 The "OS=posix" sets the macro OS to the value posix, it does not set the target. This has been a feature of make since at least the 1980's, earlier than Linux even existed. So I'm astonished you're seeing this error. The issue is the -f is passed a blank argument. Probably in linux.mak, it's calling something like this: make --no-print-directory -f $(SUB_MAKE_FILE) OS=$(OS) BUILD=release or something like that, and SUB_MAKE_FILE isn't being set for some reason. I think make is interpreting that OS=posix as a makefile to use. -Steve
Re: Getting # Physical CPUs
On 07/16/2010 04:48 AM, Don wrote: Georg Wrede wrote: On 07/14/2010 08:55 PM, dsimcha wrote: It would be nice to not have to manually specify such a parameter on every run, but not nice enough to be worth introducing a dependency. I can't imagine how this would not be a required part of the core library. For a language that claims to be thread savvy, knowing the number of cpus and the number of cores, is simply obligatory homework. An extra point: the code that identifies them, should not ever assume that all cores are identical. Nor that they have identical access to machine resources. The day that someone invents the 'unequal cores paradigm', where cores of dissimilar power are included in the same computer, should not expose us with our pants down. It really depends on what the purpose is. If you want to determine the precise core topology, the available information is heavily OS-dependent. Note that there's potentially a large difference between the number of cores in the machine, versus the number of cores which the OS makes available to your app. Generally the second number is the one which matters. True. What my app needs D to tell it, is to enumerate and specify the processors (and of course any other variable resources) available to this particular instance of running program. (A case in point, at bootup, the Linux core already enumerates and evaluates each found core individually.) Of course it does. It's trivial when you're an OS and have unrestricted access to the machine. An app is severely limited to what it can get from the OS. Currently core.cpuid doesn't make any OS calls at all. I think std.cpuid should be replaced with a new module std.sysinfo, which determines more features (such as available RAM). Agreed. Runtime initialization code should find out the resources available to the current app instance. Anything else can (or should) be available from /proc/... (or, whatever equivalents Bill G or Steve J have decided), and the app is free to roam around in that directory tree.
