On Fri, Jan 4, 2013 at 2:47 PM, Alex J Lennon <ajlen...@dynamicdevices.co.uk> wrote: > On 04/01/2013 22:43, Mark Hatle wrote: >> On 1/4/13 2:56 PM, Alex J Lennon wrote: >>> >>> Can anybody advise on whether bitbake currently supports offloading of >>> build tasks onto multiple systems? Perhaps cloud based? >> >> I see Chris Larson has already answer the main question here.. but I've got >> some >> additional insight I can add. >> >>> I'm thinking that it would be more efficient for me if I could bring up >>> a number of Amazon EC2 servers (or similar) then have bitbake >>> parallelise the build onto those servers to significantly reduce my >>> build times? >> >> In the past I've had people ask me questions about remote builds, parallel >> building packages, etc. In the past when I've attempted to figure out if it >> would be useful, the cost of the data transfer time always comes into play. >> How >> long does it take to simply perform the task locally vs go to the remote >> machine, perform the task, and get the data back. For a lot of the tasks >> that >> OE does, the remote tasks will end up being slower on a reasonably modern i7 >> (or >> similar Xeon) machine. I'm not sure using something like a cloud server >> would >> end up helping due to this. >> > > Thanks for your thoughts Mark. > > Another idea that occurred was that it might be interesting if there > were some metrics available somewhere on recipe build times (are timing > metrics currently generated in any detail?).
Yes, they're generated by buildstats.bbclass and should be available within TMPDIR/buildstats. You can also use scripts/pybootchartgui.py to get a visual representation of the timeline of a build. -b > > Using that as a starting point it might then be possible to determine > how best to parallelise tasks? > >> From experience I'd say that if anything you'd likely want to modify the >> bitbake scheduler with some type of locality information. I.e. if you >> extract >> the sources on a node, you most likely want to patch, configure, compile, >> install, QA and package on that node as well.... but that also means a way to >> establish node specific tmp/work space is needed for performance as well. >> >>> I see bitbake supports a level of task parallelisation on a single box. >>> >>> Can parallelisation of build onto multiple systems be achieved? >> >> All of the above are possible, but I suspect there is a lot of design and >> refactoring required to come up with a system that is both efficient and >> fast. >> >>> Is it something that should even be a goal? >> >> It's not currently a goal. But it does get asked about every now and then. >> It >> would be nice to have something more then anecdotal information about if it >> would be a good idea or not. This is the type of thing that I think would >> make >> an excellent research project. (Hey and if it makes things able to be built >> for >> more people, faster, and cheaper great! If not, we'd have a more informed >> response to questions like this.) >> >> --Mark >> >>> Cheers, >>> >>> Alex >>> >>> _______________________________________________ >>> yocto mailing list >>> yocto@yoctoproject.org >>> https://lists.yoctoproject.org/listinfo/yocto >>> >> >> _______________________________________________ >> yocto mailing list >> yocto@yoctoproject.org >> https://lists.yoctoproject.org/listinfo/yocto >> > > _______________________________________________ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto -- Elizabeth Flanagan Yocto Project Build and Release _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto