Re: [Rd] Detecting physical CPUs in detectCores() on Linux platforms
On 8 August 2023 at 13:17, Simon Urbanek wrote: | To be honest I think the motivation of this thread is dubious at best: it is a bad idea to use detectCore() blindly to specify parallelization and we explicitly say it's a bad idea - any sensible person will set it according to the demands, the hardware and the task. The number of cores is only partially relevant - e.g. if any I/O is involved you want to oversubscribe the CPU. If you have other users you want to only use a fraction etc. That doesn't mean that the we couldn't do a better job, but if you have to use detectCores() then you are already in trouble to start with. As I often say, "life's a bitch and then you die". Detecting hardware capabilties at run-time is no small task. When 1 1/2 decades ago I filled in maintaining slurm for Debian (and interfacing open-mpi) that latter project added an entire library for that scope (libhwloc). For all I know it is still best practices. So in that sense no point grepping / reading /proc/cpuinfo as that has clear limits. Dirk -- dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Detecting physical CPUs in detectCores() on Linux platforms
> On 8/08/2023, at 12:07 PM, Dirk Eddelbuettel wrote:
>
>
> On 8 August 2023 at 11:21, Simon Urbanek wrote:
> | First, detecting HT vs cores is not necessarily possible in general, Linux
> may assign core id to each HT depending on circumstances:
> |
> | $ grep 'cpu cores' /proc/cpuinfo | uniq
> | cpu cores : 32
> | $ grep 'model name' /proc/cpuinfo | uniq
> | model name : Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz
> |
> | and you can look up that Xenon 6142 has 16 cores.
> |
> | Second, instead of "awk"ward contortions it's easily done in R with
> something like
> |
> | d=read.dcf("/proc/cpuinfo")
> | sum(as.integer(tapply(
> | d[,grep("cpu cores",colnames(d))],
> | d[,grep("physical id",colnames(d))], `[`, 1)))
> |
> | which avoids subprocesses, quoting hell and all such issues...
>
> Love the use of read.dcf("/proc/cpuinfo") !!
>
> On my box a simpler
>
>> d <- read.dcf("/proc/cpuinfo")
>> as.integer(unique(d[, grep("cpu cores",colnames(d))]))
> [1] 6
>>
>
I don't think that works on NUMA/SMP machines - you need to add all the cores
for each CPU (that's why the above splits by physical id which is unique per
cpu). On a dual-cpu machine:
> as.integer(unique(d[, grep("cpu cores",colnames(d))]))
[1] 32
> sum(as.integer(tapply(
d[,grep("cpu cores",colnames(d))],
d[,grep("physical id",colnames(d))], `[`, 1)))
[1] 64
Also things get quite fun on VMs as they can cobble together quite a few
virtual CPUs regardless of the underlying hardware.
To be honest I think the motivation of this thread is dubious at best: it is a
bad idea to use detectCore() blindly to specify parallelization and we
explicitly say it's a bad idea - any sensible person will set it according to
the demands, the hardware and the task. The number of cores is only partially
relevant - e.g. if any I/O is involved you want to oversubscribe the CPU. If
you have other users you want to only use a fraction etc. That doesn't mean
that the we couldn't do a better job, but if you have to use detectCores() then
you are already in trouble to start with.
Cheers,
Simon
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Detecting physical CPUs in detectCores() on Linux platforms
On 8 August 2023 at 11:21, Simon Urbanek wrote:
| First, detecting HT vs cores is not necessarily possible in general, Linux
may assign core id to each HT depending on circumstances:
|
| $ grep 'cpu cores' /proc/cpuinfo | uniq
| cpu cores : 32
| $ grep 'model name' /proc/cpuinfo | uniq
| model name: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz
|
| and you can look up that Xenon 6142 has 16 cores.
|
| Second, instead of "awk"ward contortions it's easily done in R with something
like
|
| d=read.dcf("/proc/cpuinfo")
| sum(as.integer(tapply(
| d[,grep("cpu cores",colnames(d))],
| d[,grep("physical id",colnames(d))], `[`, 1)))
|
| which avoids subprocesses, quoting hell and all such issues...
Love the use of read.dcf("/proc/cpuinfo") !!
On my box a simpler
> d <- read.dcf("/proc/cpuinfo")
> as.integer(unique(d[, grep("cpu cores",colnames(d))]))
[1] 6
>
does the right thing.
Dirk
--
dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Detecting physical CPUs in detectCores() on Linux platforms
First, detecting HT vs cores is not necessarily possible in general, Linux may
assign core id to each HT depending on circumstances:
$ grep 'cpu cores' /proc/cpuinfo | uniq
cpu cores : 32
$ grep 'model name' /proc/cpuinfo | uniq
model name : Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz
and you can look up that Xenon 6142 has 16 cores.
Second, instead of "awk"ward contortions it's easily done in R with something
like
d=read.dcf("/proc/cpuinfo")
sum(as.integer(tapply(
d[,grep("cpu cores",colnames(d))],
d[,grep("physical id",colnames(d))], `[`, 1)))
which avoids subprocesses, quoting hell and all such issues...
Cheers,
Simon
> On 8/08/2023, at 12:47 AM, Julian Hniopek wrote:
>
> On Mon, 2023-08-07 at 07:12 -0500, Dirk Eddelbuettel wrote:
>>
>> On 7 August 2023 at 08:48, Nils Kehrein wrote:
>>> I recently noticed that `detectCores()` ignores the `logical=FALSE`
>>> argument on Linux platforms. This means that the function will
>>> always
>>> return the number of logical CPUs, i.e. it will count the number of
>>> threads
>>> that theoretically can run in parallel due to e.g. hyper-threading.
>>> Unfortunately, this can result in issues in high-performance
>>> computing use
>>> cases where hyper-threading might degrade performance instead of
>>> improving
>>> it.
>>>
>>> Currently, src/library/parallel/R/detectCores.R uses the following
>>> R/shell
>>> code fragment to identify the number of logical CPUs:
>>> linux = 'grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l'
>>>
>>> As far as I understand, one could derive the number of online
>>> physical CPUs
>>> by parsing the contents of /sys/devices/system/cpu/* but that seems
>>> rather
>>> cumbersome. Instead, could we amend the R code with the following
>>> line?
>>> linux = if(logical) 'grep "^processor" /proc/cpuinfo 2>/dev/null |
>>> wc -l'
>>> else 'lscpu -b --parse="CORE" | tail -n +5 | sort -u | wc -l'
>>
>> That's good but you also need to at protect this from `lscpu` being
>> in the
>> path. Maybe `if (logical && nzchar(Sys.which("lscpu")))` ?
>>
>> Dirk
>>
> Alternatively, using only on POSIX utils which should be in the path of
> all Linux Systems and /proc/cpuinfo:
>
> awk '/^physical id/{PHYS_ID=$NF; next} /^cpu cores/{print PHYS_ID"
> "$NF;}' /proc/cpuinfo 2>/dev/null | sort | uniq | awk '{sum+=$NF;} END
> {print sum}'.
>
> Parses /proc/cpuinfo for the number of physical cores and physical id
> in each CPU. Only returns unique combinations of physical id (i.e.
> Socket) and core numbers. Then sums up the number of cores for each
> physicalid to get the total amount of physical cores.
>
> Something I had lying around. Someone with better awk skills could
> probably do sorting and filtering in awk as well to save on pipes.
> Works on single and multisocket AMD/Intel from my experience.
>
> Julian
>>>
>>> This solution uses `lscpu` from `sys-utils`. The -b switch makes
>>> sure that
>>> only online CPUs/cores are listed and due to the --parse="CORE",
>>> the output
>>> will contain only a single column with logical core ids. It seems
>>> to do the
>>> job in my view, but there might be edge cases for exotic CPU
>>> topologies
>>> that I am not aware of.
>>>
>>> Thank you, Nils
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> __
>>> R-devel@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Detecting physical CPUs in detectCores() on Linux platforms
On Mon, 2023-08-07 at 07:12 -0500, Dirk Eddelbuettel wrote:
>
> On 7 August 2023 at 08:48, Nils Kehrein wrote:
> > I recently noticed that `detectCores()` ignores the `logical=FALSE`
> > argument on Linux platforms. This means that the function will
> > always
> > return the number of logical CPUs, i.e. it will count the number of
> > threads
> > that theoretically can run in parallel due to e.g. hyper-threading.
> > Unfortunately, this can result in issues in high-performance
> > computing use
> > cases where hyper-threading might degrade performance instead of
> > improving
> > it.
> >
> > Currently, src/library/parallel/R/detectCores.R uses the following
> > R/shell
> > code fragment to identify the number of logical CPUs:
> > linux = 'grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l'
> >
> > As far as I understand, one could derive the number of online
> > physical CPUs
> > by parsing the contents of /sys/devices/system/cpu/* but that seems
> > rather
> > cumbersome. Instead, could we amend the R code with the following
> > line?
> > linux = if(logical) 'grep "^processor" /proc/cpuinfo 2>/dev/null |
> > wc -l'
> > else 'lscpu -b --parse="CORE" | tail -n +5 | sort -u | wc -l'
>
> That's good but you also need to at protect this from `lscpu` being
> in the
> path. Maybe `if (logical && nzchar(Sys.which("lscpu")))` ?
>
> Dirk
>
Alternatively, using only on POSIX utils which should be in the path of
all Linux Systems and /proc/cpuinfo:
awk '/^physical id/{PHYS_ID=$NF; next} /^cpu cores/{print PHYS_ID"
"$NF;}' /proc/cpuinfo 2>/dev/null | sort | uniq | awk '{sum+=$NF;} END
{print sum}'.
Parses /proc/cpuinfo for the number of physical cores and physical id
in each CPU. Only returns unique combinations of physical id (i.e.
Socket) and core numbers. Then sums up the number of cores for each
physicalid to get the total amount of physical cores.
Something I had lying around. Someone with better awk skills could
probably do sorting and filtering in awk as well to save on pipes.
Works on single and multisocket AMD/Intel from my experience.
Julian
> >
> > This solution uses `lscpu` from `sys-utils`. The -b switch makes
> > sure that
> > only online CPUs/cores are listed and due to the --parse="CORE",
> > the output
> > will contain only a single column with logical core ids. It seems
> > to do the
> > job in my view, but there might be edge cases for exotic CPU
> > topologies
> > that I am not aware of.
> >
> > Thank you, Nils
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Detecting physical CPUs in detectCores() on Linux platforms
On 7 August 2023 at 08:48, Nils Kehrein wrote:
| I recently noticed that `detectCores()` ignores the `logical=FALSE`
| argument on Linux platforms. This means that the function will always
| return the number of logical CPUs, i.e. it will count the number of threads
| that theoretically can run in parallel due to e.g. hyper-threading.
| Unfortunately, this can result in issues in high-performance computing use
| cases where hyper-threading might degrade performance instead of improving
| it.
|
| Currently, src/library/parallel/R/detectCores.R uses the following R/shell
| code fragment to identify the number of logical CPUs:
| linux = 'grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l'
|
| As far as I understand, one could derive the number of online physical CPUs
| by parsing the contents of /sys/devices/system/cpu/* but that seems rather
| cumbersome. Instead, could we amend the R code with the following line?
| linux = if(logical) 'grep "^processor" /proc/cpuinfo 2>/dev/null | wc -l'
| else 'lscpu -b --parse="CORE" | tail -n +5 | sort -u | wc -l'
That's good but you also need to at protect this from `lscpu` being in the
path. Maybe `if (logical && nzchar(Sys.which("lscpu")))` ?
Dirk
| This solution uses `lscpu` from `sys-utils`. The -b switch makes sure that
| only online CPUs/cores are listed and due to the --parse="CORE", the output
| will contain only a single column with logical core ids. It seems to do the
| job in my view, but there might be edge cases for exotic CPU topologies
| that I am not aware of.
|
| Thank you, Nils
|
| [[alternative HTML version deleted]]
|
| __
| R-devel@r-project.org mailing list
| https://stat.ethz.ch/mailman/listinfo/r-devel
--
dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
