On 2018/09/17 19:47, Scott Cheloha wrote:
> Hi,
>
> The new sysctl(2) for determining how many CPUs the scheduler has
> access to at a particular moment is HW_NCPUONLINE.
>
> A lot of software in ports uses HW_NCPU for this purpose, so this
> software may need to be patched. Caveat: I'm not a performance
> expert. It may be the case that e.g. spawning more threads than
> CPUs does not degrade performance for some software. It may be the
> case that this brings other software to its knees.
>
> If there is not enough time to do this auditing before release,
> that's on me. But I figure you need to start somewhere.
>
> Some background:
>
> We opted not to change HW_NCPU to return the number of online CPUs
> in order to preserve the kernel ABI. HW_NCPU has always been an
> immutable value after boot. Monitoring software that uses HW_NCPU
> to allocate resources might break if the value of HW_NCPU were to
> change while it is running. I think this risk outweighs the cost
> of having some software perform suboptimally until patched to use
> HW_NCPUONLINE.
I agree with thus decision, there are two categories of program:
- those using HW_NCPU for stats - programs like top(1)
- those using it to decide how many threads/processes to run
(and maybe some doing both).
I think the "stats" category should continue to use HW_NCPU, partly
these programs are often not setup to cope with changing numbers of
CPUs, partly because it's useful to check that the "offline" (not used
for scheduling) CPUs are indeed quiescent. So I 100% think it was the
right thing to add a new sysctl rather than change HW_NCPU semantics.
I think the "number of parallel processes" category should be changed
but don't think it's urgent.
Some are in the built code - some are just in the build system (which
should not be used for ports anyway - this should purely be controlled
from ports via the MAKE_JOBS variable).
For C, sysconf is probably the best interface to use here, its simpler
to implement than direct sysctl calls, and has better cross-OS support
on current OS versions.
#include
int ncpu;
ncpu = sysconf(_SC_NPROCESSORS_ONLN);
For C++ >= 11, afaik the standard is std::thread::hardware_concurrency().
> As for the choice of HW_NCPUONLINE over some of Apple's sysctl(2)
> extensions, I guess I had a kneejerk reaction to the lack of
> documentation and opted to copy what NetBSD did.
I am wondering if we should #define HW_AVAILCPU HW_NCPUONLINE, but it
depends on how the programs use it. Definitely needs looking over source.
> --
>
> I don't have a fully extracted ports checkout, but I was able to
> grep the distfiles I do have. I think I have many, but not all
> of them.
>
> It'd be a good idea for someone with a proper ports extract
> to look for ports using HW_NCPU.
I suggest Debian codesearch. I have extracted ports source on my
workstation but searching over it generally results in many short
freezes (no keyboard/mouse response), it's very frustrating to use the
machine when this goes on so I am not doing many of these searches at
the moment.
> That said, here is what I was able to find. I don't know how
> or why they are using HW_NCPU.
>
> I trimmed the version numbers and paths here. I can just send
> the raw grep if that's more useful.
>
> boost
> chromium
> collectd
> coreutils
> darktable
> ffmpeg-git
> firefox-52.7.3esr
> freeradius-server
> gkrellm
> glib
> hexchat
> hugin
> iridium-browser
> libgtop
> nim
> p7zip
> radare2
> sslsplit
> supertuxkart
> texlive
> valgrind
> webkitgtk
> x264-snapshot
> xvidcore
> 0ad
> MoarVM
> SuperTux
> Vitallium
> beats
> blender
> blink1
> cargo/libc
> cargo/num_cpus
> cmake
> conky
> desmume
> distcc
> doxygen
> e17/eina
> facter
> fpc/fpcbuild
> freerdp
> fzf
> gc
> gcc/avr32-gcc
> ghc/cabal-install
> git
> go
> guile
> hmmer
> htop
> jamvm
> kamailio
> kore
> lcdproc
> libmemcached
> libstatgrab
> libuv
> libvirt
> lscpu
> mariadb
> mercurial
> minetest
> mongodb
> monit
> mozilla/tor-browser
> myrddin
> net-snmp
> node
> nspr
> octave
> opencv
> openmpi
> openttd/openttd
> otp
> pinot
> posixtestsuite
> psutil
> pyuv
> qt-creator-opensource
> qt-everywhere-opensource
> retroarch
> rust/rustc
> rust/rustc-bootstrap-amd64
> samba
> sendmail
> suricata
> symon
> syslog-ng
> tmux-mem-cpu-load
> tor-browser/v7.0.6-esr52.4.0
> vice
> wkhtmltopdf-qt
> x265
> xemacs-sumo
> xscreensaver
> xz
> zabbix
