Re: Rust, pkgsrc
On Tue, Nov 06, 2018 at 07:50:18PM +0100, Havard Eidnes wrote: > The last version I managed to build on one of my NetBSD/macppc > 8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was > 1.29.2, the build took nearly 29 hours wallclock time. This one > doesn't have a tmpfs, and has a single file system with ~40G > free, so it also didn't run into any barriers on the /tmp front > either. I recently did build (thanks to Havard's bootstrap kit) a native version of 1.30 on sparc64 on a 8GB RAM dual 1.5GHz machine, everything on SSD plus /tmp on tmpfs. I did not do proper timing, but it finished over night (started early when I left the office and probably only because I came in late the next day). Martin
Re: Rust, pkgsrc
> Yes. I had a similar problem. The build would fill up the > /tmp/ directory and die from exhausted resources. I had /tmp/ > created with tmpfs and had a constraint of 64M. The answer for > me was to create /tmp in /etc/fstab with tmpfs and no size > constraint. Then Rust would build, but it still took a long > time. Yes, rust is an absolute Pig resource-wise. Not only does it carry a copy of llvm inside itself (it most probably has to), it also carries a nubmer of other packages inside. Plus, the build builds most parts at least twice if I've been able to observe correctly. When building on NetBSD/amd64 8.0, I noticed that the work/ directory after a "make" consumes in the order of 10G disk space, possibly more when cross-building (my current work/ is 13G, which is an unfinished cross-build because I hit build issues...). It is conceivable that the storage could be reduced somewhat (but probably not by much?) by tweaking src/bootstrap/boostrap.py to say -Cdebuginfo=0 instead of 2 for RUSTFLAGS. So that you have an idea what to expect: My amd64 build host has 8G real memory, and a 2G tmpfs /tmp, and ... it didn't run out of space anywhere :) On this particular host (i7 3rd-gen, 4 real cores, 8 w/HT, pkgsrc and system on SSD) the build of 1.29.2 completed in a little over 2 hours wall-clock time, csh's "time" report at the end of the build was 40468.007u 1958.277s 2:04:42.18 567.0% 0+0k 10302+100556io 129329pf+0w So, yes, the build makes fairly good use of the multiple cores; notice the 567.0%, which, if I've understood correctly, indicates approx. 5.7* parallelism on average. The last version I managed to build on one of my NetBSD/macppc 8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was 1.29.2, the build took nearly 29 hours wallclock time. This one doesn't have a tmpfs, and has a single file system with ~40G free, so it also didn't run into any barriers on the /tmp front either. Regards, - Håvard
Re: Rust, pkgsrc
On 05/11/2018 16:34, bch wrote: The latest rust (1.30?) supporting the latest Firefox is *brutal* to build. I’ve blown (and then resized) /tmp multiple times, and am now exhausted on /usr for its build artifacts, before it’s even actually installed. Does anybody have tips or tricks for dealing with rust-building (which has always been terribly painful CPU-wise), or should I just move to prebuilt packages? I think I’ve never seen a piece of software as horrible to build as rust... If you are on x86 and have access to a bigger system for building on have you considered running pbulk in a chroot environment or using pkg_comp. I've save myself a lot of time on my x86 systems by doing all the builds on a relatively well specced system and then just installing binary packages on the smaller ones. Do you use MAKEJOBS on your pkgsrc builds? If you do then add MAKE_JOBS_SAFE=no to the rust package Makefile. Without that I've found that make ends up spawning multiple rustc processes each of which internally then spawn even more threads causing ridiculous cpu overcommit. If you aren't using it and you are on a small system I don't know what to suggest. Even with that tweak compiling rust on a 4core (8 threads) 4GHz 64bit CPU with 16GB of RAM is still painfully slow and ends up maxing out the CPU for most of the build. Mike -bch
Re: Rust, pkgsrc
Hi, bch wrote: The latest rust (1.30?) supporting the latest Firefox is *brutal* to build. I’ve blown (and then resized) /tmp multiple times, and am now exhausted on /usr for its build artifacts, before it’s even actually installed. Does anybody have tips or tricks for dealing with rust-building (which has always been terribly painful CPU-wise), or should I just move to prebuilt packages? I think I’ve never seen a piece of software as horrible to build as rust... it is terrible and I am very sorry our beloved Firefox was gone that way. It is making non-mainstreambuilds very difficult. I have issues with it not only on NetBSD, but also on Gentoo Linux. are you on 32bit or 64bit? The former has more issues! second, playing with binutils helped me out on Linux.. older version worked and the latest one worked too, but there was a series that did not. Also, RAM helps... more than 2GB seems to make thhings friendlier, 1.5GB is about the limit. Riccardo
Re: Rust, pkgsrc
Yes. I had a similar problem. The build would fill up the /tmp/ directory and die from exhausted resources. I had /tmp/ created with tmpfs and had a constraint of 64M. The answer for me was to create /tmp in /etc/fstab with tmpfs and no size constraint. Then Rust would build, but it still took a long time. On Tue, 6 Nov 2018 at 04:42, bch wrote: > The latest rust (1.30?) supporting the latest Firefox is *brutal* to > build. I’ve blown (and then resized) /tmp multiple times, and am now > exhausted on /usr for its build artifacts, before it’s even actually > installed. > > Does anybody have tips or tricks for dealing with rust-building (which has > always been terribly painful CPU-wise), or should I just move to prebuilt > packages? I think I’ve never seen a piece of software as horrible to build > as rust... > > > -bch >
