[OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Ross Burton
Hi, Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of cores available unless told otherwise. This was a good idea six years ago[1] but some modern machines are moving to very large core counts. For example, 88 core dual Xeons are fairly common. A ThunderX2 has 256 cores (2 sockets

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Alexander Kanavin
I'd rather teach bitbake to abstain from starting new tasks when I/O or CPU gets tight. Alex On Thu, 3 Dec 2020 at 18:48, Ross Burton wrote: > Hi, > > Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of cores > available unless told otherwise. This was a good idea six years > ago[

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Andre McCurdy
On Thu, Dec 3, 2020 at 9:48 AM Ross Burton wrote: > > Hi, > > Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of cores > available unless told otherwise. This was a good idea six years > ago[1] but some modern machines are moving to very large core counts. > > For example, 88 core d

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Paul Barker
On Thu, 3 Dec 2020 at 17:48, Ross Burton wrote: > > Hi, > > Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of cores > available unless told otherwise. This was a good idea six years > ago[1] but some modern machines are moving to very large core counts. > > For example, 88 core dua

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Konrad Weihmann
+1 for Alex's comment. As I/O (and potentially RAM) is clearly the bottleneck here, limitation to some arbitrary value doesn't address the issue at the right end. I would rather see real resource management. From my point of view there is a huge difference if I package like 128 shell scripts in

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Khem Raj
On Thu, Dec 3, 2020 at 9:48 AM Ross Burton wrote: > > Hi, > > Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of cores > available unless told otherwise. This was a good idea six years > ago[1] but some modern machines are moving to very large core counts. > > For example, 88 core d

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Richard Purdie
On Thu, 2020-12-03 at 17:48 +, Ross Burton wrote: > Hi, > > Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of > cores > available unless told otherwise. This was a good idea six years > ago[1] but some modern machines are moving to very large core counts. > > For example, 88 c

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Mikko Rapeli
Hi, On Thu, Dec 03, 2020 at 05:48:16PM +, Ross Burton wrote: > Hi, > > Currently, BB_NUMBER_THREADS and PARALLEL_MAKE use the number of cores > available unless told otherwise. This was a good idea six years > ago[1] but some modern machines are moving to very large core counts. > > For exa

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-03 Thread Mikko Rapeli
On Thu, Dec 03, 2020 at 07:20:11PM +0100, Alexander Kanavin wrote: > I'd rather teach bitbake to abstain from starting new tasks when I/O or CPU > gets tight. And memory! -Mikko -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#145287): https://list

Re: [OE-core] Automatic cap on BB_NUMBER_THREADS and PARALLEL_MAKE

2020-12-04 Thread Jose Quaresma
Alexander Kanavin escreveu no dia quinta, 3/12/2020 à(s) 18:20: > I'd rather teach bitbake to abstain from starting new tasks when I/O or > CPU gets tight. > This is definitely the best approach in my view. however more complex to implement. Quaresma > Alex > > On Thu, 3 Dec 2020 at 18:48, Ro