Re: [yocto] What are the key factors for yocto build speed?

2020-03-21 Thread Martin Jansa
On 1600AF with 16G RAM and nvme drive it took ~ 38 hours, if you were running all 4 builds (as test.sh does), you can still finish the tests by running only the remaining build steps. You can send me even partial results and I'll let you know if your times are expected or not (it should be very

Re: [yocto] What are the key factors for yocto build speed?

2020-03-21 Thread Oliver Westermann
On Wed, Mar 18, 2020 at 08:09 AM, Mike Looijmans wrote: > > Of course he's on a tight budget. He wouldn't need to ask for advice > otherwise... When has a company ever given you a card blanche for equipment? :D There is a budget and I just need to write down a good argument to spend it on

Re: [yocto] What are the key factors for yocto build speed?

2020-03-21 Thread Oliver Westermann
On Wed, Mar 18, 2020 at 08:38 AM, Martin Jansa wrote: > > > If you want to compare how terrible your current VM compares with some > other builders, you can use: > https://github.com/shr-project/test-oe-build-time Thanks for all the replies here, had some long days so no earlier answers. I did

Re: [yocto] What are the key factors for yocto build speed?

2020-03-20 Thread Adrian Bunk
On Fri, Mar 20, 2020 at 03:58:35PM +0100, Mike Looijmans wrote: >... > (because octave is virtually impossible to cross-compile). >... The recipe in meta-openembedded cross-compiles without problem. cu Adrian -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply

Re: Re: [yocto] What are the key factors for yocto build speed?

2020-03-20 Thread Philip Balister
ia > Lists.Yoctoproject.Org > Sent: Wednesday, March 18, 2020 11:52 AM > To: mike.looijm...@topic.nl<mailto:mike.looijm...@topic.nl> > Cc: yocto@lists.yoctoproject.org<mailto:yocto@lists.yoctoproject.org> > Subject: Re: [yocto] What are the key factors for yocto build speed

Re: [yocto] What are the key factors for yocto build speed?

2020-03-20 Thread Mike Looijmans
On 19-03-2020 18:21, Adrian Bunk wrote: On Thu, Mar 19, 2020 at 05:07:17PM +0100, Mike Looijmans wrote: ... With both parallelization options to "16", I might end up with 16 compile tasks running 16 compile threads each, i.e. 256 running processes. ... This is a bug:

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Khem Raj
On Thu, Mar 19, 2020 at 9:07 AM Mike Looijmans wrote: > > On 19-03-2020 12:04, Richard Purdie via Lists.Yoctoproject.Org wrote: > >> , fetch, configure, package and rootfs tasks. > > > > Sadly these tasks are much harder. > > It would be really great if some sort of "weight" could be attached to

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Yann Dirson
Le jeu. 19 mars 2020 à 18:04, Richard Purdie < richard.pur...@linuxfoundation.org> a écrit : > On Thu, 2020-03-19 at 17:29 +0100, Yann Dirson wrote: > > > > > > Le jeu. 19 mars 2020 à 17:07, Mike Looijmans > a écrit : > > > On 19-03-2020 12:04, Richard Purdie via Lists.Yoctoproject.Org wrote: >

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Richard Purdie
On Thu, 2020-03-19 at 19:21 +0200, Adrian Bunk wrote: > On Thu, Mar 19, 2020 at 05:07:17PM +0100, Mike Looijmans wrote: > > ... > > With both parallelization options > > to "16", I might end up with 16 compile tasks running 16 compile > > threads > > each, i.e. 256 running processes. > > ... > >

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Adrian Bunk
On Thu, Mar 19, 2020 at 05:07:17PM +0100, Mike Looijmans wrote: >... > With both parallelization options > to "16", I might end up with 16 compile tasks running 16 compile threads > each, i.e. 256 running processes. >... This is a bug: http://bugzilla.yoctoproject.org/show_bug.cgi?id=13306 I

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Richard Purdie
On Thu, 2020-03-19 at 17:29 +0100, Yann Dirson wrote: > > > Le jeu. 19 mars 2020 à 17:07, Mike Looijmans a > écrit : > > On 19-03-2020 12:04, Richard Purdie via Lists.Yoctoproject.Org wrote: > > >> , fetch, configure, package and rootfs tasks. > > > > > > Sadly these tasks are much harder. >

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Yann Dirson
Le jeu. 19 mars 2020 à 17:07, Mike Looijmans a écrit : > On 19-03-2020 12:04, Richard Purdie via Lists.Yoctoproject.Org wrote: > >> , fetch, configure, package and rootfs tasks. > > > > Sadly these tasks are much harder. > > It would be really great if some sort of "weight" could be attached to

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Mike Looijmans
On 19-03-2020 12:04, Richard Purdie via Lists.Yoctoproject.Org wrote: , fetch, configure, package and rootfs tasks. Sadly these tasks are much harder. It would be really great if some sort of "weight" could be attached to a task. This relates to memory usage. My system has 16 cores but

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Richard Purdie
On Thu, 2020-03-19 at 11:43 +, mikko.rap...@bmw.de wrote: > On Thu, Mar 19, 2020 at 11:04:26AM +, Richard Purdie wrote: > > Recipe parsing should hit 100% CPU, its one of the few places we > > can do > > that. > > I'm not fully aware what bitbake does before starting task execution. >

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Mikko Rapeli
On Thu, Mar 19, 2020 at 11:04:26AM +, Richard Purdie wrote: > On Thu, 2020-03-19 at 08:05 +, Mikko Rapeli wrote: > > Once this is done, IO still happens when anything calls sync() and > > fsync() and worst offenders are package management tools. In yocto > > builds, package manager actions

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Richard Purdie
On Thu, 2020-03-19 at 08:05 +, Mikko Rapeli wrote: > Once this is done, IO still happens when anything calls sync() and > fsync() and worst offenders are package management tools. In yocto > builds, package manager actions to flush to disk are always useless > since rootfs images are going to

Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Mikko Rapeli
On Wed, Mar 18, 2020 at 10:56:50PM +, Ross Burton wrote: > On 18/03/2020 14:09, Mike Looijmans wrote: > > Harddisk speed has very little impact on your build time. It helps with > > the "setscene" parts, but doesn't affect actual compile time at all. I > > recall someone did a build from RAM

Re: Re: [yocto] What are the key factors for yocto build speed?

2020-03-19 Thread Mike Looijmans
Sent: Wednesday, March 18, 2020 11:52 AM To: mike.looijm...@topic.nl <mailto:mike.looijm...@topic.nl> Cc: yocto@lists.yoctoproject.org <mailto:yocto@lists.yoctoproject.org> Subject: Re: [yocto] What are the key factors for yocto build speed? On Wed, Mar 18, 2020 at 04:09:39PM +0100, Mi

Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Ross Burton
On 18/03/2020 14:09, Mike Looijmans wrote: Harddisk speed has very little impact on your build time. It helps with the "setscene" parts, but doesn't affect actual compile time at all. I recall someone did a build from RAM disks only on a rig, and it was only about 1 minute faster on a one hour

Re: Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Srini
@lists.yoctoproject.org Subject: Re: Re: [yocto] What are the key factors for yocto build speed? 4 hours seems extremely long to me for a 8 core system, but I have not tried this in a while. Were you removing all the sources and re-downloading them for every build? From: mailto:yocto

Re: Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Yann Dirson
oading > them for every build? > > > > *From: * on behalf of Srini < > rsriniva...@abiomed.com> > *Date: *Wednesday, March 18, 2020 at 10:13 AM > *To: *"mikko.rap...@bmw.de" , " > mike.looijm...@topic.nl" > *Cc: *"yocto@lists.yoctoproject.org

Re: Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread David Stewart
ijm...@topic.nl" Cc: "yocto@lists.yoctoproject.org" Subject: Re: Re: [yocto] What are the key factors for yocto build speed? My own experience (pardon me if already discussed) Fought the build times for several months - ending up eventually at 8 cores (but specifying 16 threa

Re: Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Srini
- From: yocto@lists.yoctoproject.org On Behalf Of Mikko Rapeli via Lists.Yoctoproject.Org Sent: Wednesday, March 18, 2020 11:52 AM To: mike.looijm...@topic.nl Cc: yocto@lists.yoctoproject.org Subject: Re: [yocto] What are the key factors for yocto build speed? On Wed, Mar 18, 2020 at 04:09:39PM

Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Mikko Rapeli
On Wed, Mar 18, 2020 at 04:09:39PM +0100, Mike Looijmans wrote: > On 18-03-2020 15:49, Adrian Bunk via Lists.Yoctoproject.Org wrote: > > On Wed, Mar 18, 2020 at 10:12:26AM -0400, Jean-Marie Lemetayer wrote: > > > ... > > > For example one of our build servers is using: > > > - AMD Ryzen 9 3900X >

Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Mike Looijmans
On 18-03-2020 15:49, Adrian Bunk via Lists.Yoctoproject.Org wrote: On Wed, Mar 18, 2020 at 10:12:26AM -0400, Jean-Marie Lemetayer wrote: ... For example one of our build servers is using: - AMD Ryzen 9 3900X ... - 32Go DDR4 3200 MHZ CL14 ... It is a really good price / build time ratio

Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Mike Looijmans
On 18-03-2020 15:09, Mike Looijmans via Lists.Yoctoproject.Org wrote: Big ryzen is a good choice. My home rig is a Ryzen 1900, with only 8GB RAM. It's way faster at OE yocto builds than the i7 at work that has 32GB RAM installed. Sorry - wrong number. My rig does not have a 1900, but an

Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Alexander Kanavin
I have to say that AMD aggressively pursuing core count increase on consumer level CPUs is awesome news for the YP. Previously, you had to buy some hugely overpriced Xeons or similar to be able to work efficiently, or rely on CI doing builds for you which makes interactive development complicated.

Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Jean-Marie Lemetayer
Hi, In my company we have tested some "big Ryzen" configurations to speed up Yocto builds. The conclusion of these tests is that the build time is almost only related to the CPU score: https://www.cpubenchmark.net/high_end_cpus.html The speed (overclock) and size of the RAM does not influence

Re: [yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Paul Barker
On Wed, 18 Mar 2020 at 13:01, Mikko Rapeli wrote: > > On Wed, Mar 18, 2020 at 05:52:37AM -0700, Oliver Westermann wrote: > > Hey, > > > > We're currently using a VM on Windows and it's a lot slower than the native > > linux build (which is expected). > > We're looking into getting a dedicated

[yocto] What are the key factors for yocto build speed?

2020-03-18 Thread Oliver Westermann
Hey, We're currently using a VM on Windows and it's a lot slower than the native linux build (which is expected). We're looking into getting a dedicated build server for our team (basically a self-build tower PC). Any suggestions what to put in that build to get the most out of it? Currently