Re: Large builds with poudriere
On 2021-May-20, at 21:10, Chris wrote: > On 2021-05-20 13:47, Mark Millard wrote: >> Chris portmaster at bsdforge.com wrote on >> Thu May 20 20:09:29 UTC 2021 : >>> On 2021-05-20 12:21, Kevin Oberman wrote: >>> > . . . >>> >> >>> > You can greatly reduce the build-time for devel/llvm* by changing the >>> > config to BE_NATIVE to avoid building backends for all FreeBSD supported >>> > platforms. Obviously this is not acceptable for many cases, but if you >>> > never cross-compile for other platforms, it's a really big win. >>> If I understand correctly. Isn't your suggestion synonymous with >>> WITHOUT_CROSS_COMPILER? >> So far as I know, devel/llvm* builds make no use of WITHOUT_CROSS_COMPILER >> definitions ( documented in src.conf ) or what I mention below (also from >> src.conf documentation). >> As for what is analogous in src.conf's documentation for system >> builds: BE_NATIVE in an devel/llvm* for a given context would be >> analogous to using a specific mix of: >>WITHOUT_LLVM_TARGET_AARCH64 vs. WITH_LLVM_TARGET_AARCH64 >>WITHOUT_LLVM_TARGET_ARM vs. WITH_LLVM_TARGET_ARM >>WITHOUT_LLVM_TARGET_MIPSvs. WITH_LLVM_TARGET_MIPS >>WITHOUT_LLVM_TARGET_POWERPC vs. WITH_LLVM_TARGET_POWERPC >>WITHOUT_LLVM_TARGET_RISCV vs. WITH_LLVM_TARGET_RISCV >>WITHOUT_LLVM_TARGET_X86 vs. WITH_LLVM_TARGET_X86 >> Some of the mixes would simulate the various devel/llvm* ports >> BE_NATIVE's intent for various platforms. (BE_NATIVE does not >> work in all contexts, last I tried it anyway.) > Ahh. My mistake I guess. I somehow got the idea that defining > WITHOUT_CROSS_COMPILER caused llvm to build only what's needed > build/bootstrap for the arch it's (currently) running on. Which > would, of course. Slim down the buildtime/installed footprint. (I'm not trying to have my wording below cover older gcc based FreeBSD alternatives.) 13.x of FreeBSD building 14.0 (for example) generally uses a bootstrap compiler even for the same architecture. A difference is the default target of the two compilers, such as: aarch64-unknown-freebsd13.0 vs. aarch64-unknown-freebsd14.0 even if that were the only difference at the time. (A different form of "cross"?) devel/llvm* does not have this bootstrapping issue that is driven by FreeBSD definitions for building FreeBSD --instead of by processor architecture. WITHOUT_CROSS_COMPILER for a FreeBSD system build causes: WITHOUT_CLANG_BOOTSTRAP WITHOUT_ELFTOOLCHAIN_BOOTSTRAP WITHOUT_LLD_BOOTSTRAP and so can not be used for the 13.x to 14.0 example: such is not an example of "close" if I understand right: WITHOUT_CROSS_COMPILER Set to not build any cross compiler in the cross-tools stage of buildworld. When compiling a different version of FreeBSD than what is installed on the system, provide an alternate compiler with XCC to ensure success. When compiling with an identical version of FreeBSD to the host, this option may be safely used. This option may also be safe when the host version of FreeBSD is close to the sources being built, but all bets are off if there have been any changes to the toolchain between the versions. When set, it enforces these options: WITHOUT_CLANG_BOOTSTRAP WITHOUT_ELFTOOLCHAIN_BOOTSTRAP WITHOUT_LLD_BOOTSTRAP (One might have compilers around for XCC and such symbol use but that is a different kind of context.) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On 2021-05-20 13:47, Mark Millard wrote: Chris portmaster at bsdforge.com wrote on Thu May 20 20:09:29 UTC 2021 : On 2021-05-20 12:21, Kevin Oberman wrote: > . . . >> > You can greatly reduce the build-time for devel/llvm* by changing the > config to BE_NATIVE to avoid building backends for all FreeBSD supported > platforms. Obviously this is not acceptable for many cases, but if you > never cross-compile for other platforms, it's a really big win. If I understand correctly. Isn't your suggestion synonymous with WITHOUT_CROSS_COMPILER? So far as I know, devel/llvm* builds make no use of WITHOUT_CROSS_COMPILER definitions ( documented in src.conf ) or what I mention below (also from src.conf documentation). As for what is analogous in src.conf's documentation for system builds: BE_NATIVE in an devel/llvm* for a given context would be analogous to using a specific mix of: WITHOUT_LLVM_TARGET_AARCH64 vs. WITH_LLVM_TARGET_AARCH64 WITHOUT_LLVM_TARGET_ARM vs. WITH_LLVM_TARGET_ARM WITHOUT_LLVM_TARGET_MIPSvs. WITH_LLVM_TARGET_MIPS WITHOUT_LLVM_TARGET_POWERPC vs. WITH_LLVM_TARGET_POWERPC WITHOUT_LLVM_TARGET_RISCV vs. WITH_LLVM_TARGET_RISCV WITHOUT_LLVM_TARGET_X86 vs. WITH_LLVM_TARGET_X86 Some of the mixes would simulate the various devel/llvm* ports BE_NATIVE's intent for various platforms. (BE_NATIVE does not work in all contexts, last I tried it anyway.) Ahh. My mistake I guess. I somehow got the idea that defining WITHOUT_CROSS_COMPILER caused llvm to build only what's needed build/bootstrap for the arch it's (currently) running on. Which would, of course. Slim down the buildtime/installed footprint. Thanks for the clarification, Mark. --Chris === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On Thu, May 20, 2021 at 03:09:32PM -0700, Mark Millard wrote:
> Kevin Oberman rkoberman at gmail.com wrote on
> Thu May 20 21:37:28 UTC 2021 :
>
> > On Thu, May 20, 2021 at 12:48 PM Mark Millard wrote:
> >
> > > Kevin Oberman rkoberman at gmail.com wrote on
> > > Thu May 20 19:21:24 UTC 2021 :
> > >
> > > > You can greatly reduce the build-time for devel/llvm* by changing the
> > > > config to BE_NATIVE to avoid building backends for all FreeBSD supported
> > > > platforms. Obviously this is not acceptable for many cases, but if you
> > > > never cross-compile for other platforms, it's a really big win.
> > >
> > >
> > > Unfortunately, using something like (llvm10 is just one example,
> > > llvm11 showed the same sort of problem at the time):
> > >
> > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:_FILE_COMPLETE_OPTIONS_LIST=BE_AMDGPU
> > > CLANG DOCS EXTRAS LIT LLD LLDB LLD_LINK OPENMP PYCLANG BE_FREEBSD
> > > BE_NATIVE
> > > BE_STANDARD
> > >
> > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_AMDGPU
> > >
> > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_FREEBSD
> > >
> > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_NATIVE
> > >
> > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_STANDARD
> > >
> > > does not work for all platforms/targets. On a Cortex-A57
> > > this lead to:
> > >
> > > Registered Targets:
> > > amdgcn - AMD GCN GPUs
> > > r600 - AMD GPUs HD2XXX-HD6XXX
> > >
> > > In other words, aarch64 was missing. I had to pick
> > > BE_STANDARD or BE_FREEBSD to get something that
> > > would target aarch64 on aarch64.
> > >
> > > ===
> > > Mark Millard
> > > marklmi at yahoo.com
> > > ( dsl-only.net went
> > > away in early 2018-Mar)
> > >
> > Looks like the Makefile might need some work. I see stuff for handling
> > aach64/arm64, so it SHOULD work, but there are things I don't understand
> > about AARCH64 to figure it all out. Still, it should be detected.
> >
> > Out of curiosity, if you do a "make -C /usr/ports/devel/llvm10 config",
> > the line for BE_NATIVE should show the architecture you are running on. If
> > it's missing/something else, maybe you should ask brooks@ about it.
>
>
> On two types of Cortex-A72 context
> # make -C /usr/ports/devel/llvm10 config
>
> produced:
>
> BE_NATIVE Backend(s) for this architecture ()
>
> The same for each of:
>
> # make -C /usr/ports/devel/llvm80 config
> # make -C /usr/ports/devel/llvm90 config
> # make -C /usr/ports/devel/llvm11 config
> # make -C /usr/ports/devel/llvm12 config
>
> But this turns out to be because:
>
> # make -C /usr/ports/devel/llvm10 -V ARCH
> aarch64
>
> yet the Makefiles have a test for arm64 instead:
>
> .elif ${ARCH} == arm64
> _NATIVE_BACKENDS= AAarch64
Ah, that's the key point. I've fixed the ARCH value and fixed the
spelling of AArch64 in older ports where it was wrong.
-- Brooks
signature.asc
Description: PGP signature
Re: Large builds with poudriere
Kevin Oberman rkoberman at gmail.com wrote on
Thu May 20 21:37:28 UTC 2021 :
> On Thu, May 20, 2021 at 12:48 PM Mark Millard wrote:
>
> > Kevin Oberman rkoberman at gmail.com wrote on
> > Thu May 20 19:21:24 UTC 2021 :
> >
> > > You can greatly reduce the build-time for devel/llvm* by changing the
> > > config to BE_NATIVE to avoid building backends for all FreeBSD supported
> > > platforms. Obviously this is not acceptable for many cases, but if you
> > > never cross-compile for other platforms, it's a really big win.
> >
> >
> > Unfortunately, using something like (llvm10 is just one example,
> > llvm11 showed the same sort of problem at the time):
> >
> > /usr/local/etc/poudriere.d/options/devel_llvm10/options:_FILE_COMPLETE_OPTIONS_LIST=BE_AMDGPU
> > CLANG DOCS EXTRAS LIT LLD LLDB LLD_LINK OPENMP PYCLANG BE_FREEBSD BE_NATIVE
> > BE_STANDARD
> >
> > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_AMDGPU
> >
> > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_FREEBSD
> >
> > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_NATIVE
> >
> > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_STANDARD
> >
> > does not work for all platforms/targets. On a Cortex-A57
> > this lead to:
> >
> > Registered Targets:
> > amdgcn - AMD GCN GPUs
> > r600 - AMD GPUs HD2XXX-HD6XXX
> >
> > In other words, aarch64 was missing. I had to pick
> > BE_STANDARD or BE_FREEBSD to get something that
> > would target aarch64 on aarch64.
> >
> > ===
> > Mark Millard
> > marklmi at yahoo.com
> > ( dsl-only.net went
> > away in early 2018-Mar)
> >
> Looks like the Makefile might need some work. I see stuff for handling
> aach64/arm64, so it SHOULD work, but there are things I don't understand
> about AARCH64 to figure it all out. Still, it should be detected.
>
> Out of curiosity, if you do a "make -C /usr/ports/devel/llvm10 config",
> the line for BE_NATIVE should show the architecture you are running on. If
> it's missing/something else, maybe you should ask brooks@ about it.
On two types of Cortex-A72 context
# make -C /usr/ports/devel/llvm10 config
produced:
BE_NATIVE Backend(s) for this architecture ()
The same for each of:
# make -C /usr/ports/devel/llvm80 config
# make -C /usr/ports/devel/llvm90 config
# make -C /usr/ports/devel/llvm11 config
# make -C /usr/ports/devel/llvm12 config
But this turns out to be because:
# make -C /usr/ports/devel/llvm10 -V ARCH
aarch64
yet the Makefiles have a test for arm64 instead:
.elif ${ARCH} == arm64
_NATIVE_BACKENDS= AAarch64
FYI (from one of the test environments):
# uname -apKU
FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0
releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 21:53:20 PDT 2021
root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
arm64 aarch64 1300139 1300139
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On Thu, May 20, 2021 at 12:48 PM Mark Millard wrote: > Kevin Oberman rkoberman at gmail.com wrote on > Thu May 20 19:21:24 UTC 2021 : > > > You can greatly reduce the build-time for devel/llvm* by changing the > > config to BE_NATIVE to avoid building backends for all FreeBSD supported > > platforms. Obviously this is not acceptable for many cases, but if you > > never cross-compile for other platforms, it's a really big win. > > > Unfortunately, using something like (llvm10 is just one example, > llvm11 showed the same sort of problem at the time): > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:_FILE_COMPLETE_OPTIONS_LIST=BE_AMDGPU > CLANG DOCS EXTRAS LIT LLD LLDB LLD_LINK OPENMP PYCLANG BE_FREEBSD BE_NATIVE > BE_STANDARD > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_AMDGPU > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_FREEBSD > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_NATIVE > > /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_STANDARD > > does not work for all platforms/targets. On a Cortex-A57 > this lead to: > > Registered Targets: > amdgcn - AMD GCN GPUs > r600 - AMD GPUs HD2XXX-HD6XXX > > In other words, aarch64 was missing. I had to pick > BE_STANDARD or BE_FREEBSD to get something that > would target aarch64 on aarch64. > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > Looks like the Makefile might need some work. I see stuff for handling aach64/arm64, so it SHOULD work, but there are things I don't understand about AARCH64 to figure it all out. Still, it should be detected. Out of curiosity, if you do a "make -C /usr/ports/devel/llvm10 config", the line for BE_NATIVE should show the architecture you are running on. If it's missing/something else, maybe you should ask brooks@ about it. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkober...@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
Chris portmaster at bsdforge.com wrote on Thu May 20 20:09:29 UTC 2021 : > On 2021-05-20 12:21, Kevin Oberman wrote: > > . . . > >> > > You can greatly reduce the build-time for devel/llvm* by changing the > > config to BE_NATIVE to avoid building backends for all FreeBSD supported > > platforms. Obviously this is not acceptable for many cases, but if you > > never cross-compile for other platforms, it's a really big win. > > If I understand correctly. Isn't your suggestion synonymous with > WITHOUT_CROSS_COMPILER? So far as I know, devel/llvm* builds make no use of WITHOUT_CROSS_COMPILER definitions ( documented in src.conf ) or what I mention below (also from src.conf documentation). As for what is analogous in src.conf's documentation for system builds: BE_NATIVE in an devel/llvm* for a given context would be analogous to using a specific mix of: WITHOUT_LLVM_TARGET_AARCH64 vs. WITH_LLVM_TARGET_AARCH64 WITHOUT_LLVM_TARGET_ARM vs. WITH_LLVM_TARGET_ARM WITHOUT_LLVM_TARGET_MIPSvs. WITH_LLVM_TARGET_MIPS WITHOUT_LLVM_TARGET_POWERPC vs. WITH_LLVM_TARGET_POWERPC WITHOUT_LLVM_TARGET_RISCV vs. WITH_LLVM_TARGET_RISCV WITHOUT_LLVM_TARGET_X86 vs. WITH_LLVM_TARGET_X86 Some of the mixes would simulate the various devel/llvm* ports BE_NATIVE's intent for various platforms. (BE_NATIVE does not work in all contexts, last I tried it anyway.) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On 2021-05-20 12:21, Kevin Oberman wrote: On Thu, May 20, 2021 at 8:08 AM Chris wrote: On 2021-05-20 06:15, Ronald Klop wrote: > On 5/20/21 2:47 PM, Fernando Apesteguía wrote: >> El jue., 20 may. 2021 14:45, Alexis Praga >> escribió: >> >>> >>> Hi guys, >>> >>> I've recently been using poudriere to test news ports but this resulted in >>> building two large ports, namely: >>> - devel/llvm [1] >>> - lang/rust [2] >>> >>> Now, lang/rust takes 2 hourse to compile. So testing for 13.0, 12.2 and >>> 11.4 means around 6 hours in total, a lot of time for my desktop >>> computer. Especially if the dependencies are upgraded later on. >>> >>> Do you have any tips to deal with such long build time ? Apart from >>> have a dedicated server :) >>> >> >> Patience :-) >> >> Package seeding in poudriere is a long awaited feature. >> > > > You can manually put the rust/llvm package in the All packages directory and > poudriere will use that. It is a manual repeating task though. So the > package > seeding is indeed a nice feature. I can't believe I didn't already know that. :-/ Thanks, Ronald. Today you are a hero! :-) --Chris > > Ronald. > > > > > >>> Thanks, >>> >>> >>> [1] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233010 >>> [2] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255233 >>> >>> -- >>> Alexis Praga You can greatly reduce the build-time for devel/llvm* by changing the config to BE_NATIVE to avoid building backends for all FreeBSD supported platforms. Obviously this is not acceptable for many cases, but if you never cross-compile for other platforms, it's a really big win. If I understand correctly. Isn't your suggestion synonymous with WITHOUT_CROSS_COMPILER? Thanks --Chris -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkober...@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
Kevin Oberman rkoberman at gmail.com wrote on Thu May 20 19:21:24 UTC 2021 : > You can greatly reduce the build-time for devel/llvm* by changing the > config to BE_NATIVE to avoid building backends for all FreeBSD supported > platforms. Obviously this is not acceptable for many cases, but if you > never cross-compile for other platforms, it's a really big win. Unfortunately, using something like (llvm10 is just one example, llvm11 showed the same sort of problem at the time): /usr/local/etc/poudriere.d/options/devel_llvm10/options:_FILE_COMPLETE_OPTIONS_LIST=BE_AMDGPU CLANG DOCS EXTRAS LIT LLD LLDB LLD_LINK OPENMP PYCLANG BE_FREEBSD BE_NATIVE BE_STANDARD /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_AMDGPU /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_FREEBSD /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_SET+=BE_NATIVE /usr/local/etc/poudriere.d/options/devel_llvm10/options:OPTIONS_FILE_UNSET+=BE_STANDARD does not work for all platforms/targets. On a Cortex-A57 this lead to: Registered Targets: amdgcn - AMD GCN GPUs r600 - AMD GPUs HD2XXX-HD6XXX In other words, aarch64 was missing. I had to pick BE_STANDARD or BE_FREEBSD to get something that would target aarch64 on aarch64. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On Thu, May 20, 2021 at 8:08 AM Chris wrote: > On 2021-05-20 06:15, Ronald Klop wrote: > > On 5/20/21 2:47 PM, Fernando Apesteguía wrote: > >> El jue., 20 may. 2021 14:45, Alexis Praga > >> escribió: > >> > >>> > >>> Hi guys, > >>> > >>> I've recently been using poudriere to test news ports but this > resulted in > >>> building two large ports, namely: > >>> - devel/llvm [1] > >>> - lang/rust [2] > >>> > >>> Now, lang/rust takes 2 hourse to compile. So testing for 13.0, 12.2 and > >>> 11.4 means around 6 hours in total, a lot of time for my desktop > >>> computer. Especially if the dependencies are upgraded later on. > >>> > >>> Do you have any tips to deal with such long build time ? Apart from > >>> have a dedicated server :) > >>> > >> > >> Patience :-) > >> > >> Package seeding in poudriere is a long awaited feature. > >> > > > > > > You can manually put the rust/llvm package in the All packages directory > and > > poudriere will use that. It is a manual repeating task though. So the > > package > > seeding is indeed a nice feature. > I can't believe I didn't already know that. :-/ > Thanks, Ronald. Today you are a hero! :-) > > --Chris > > > > Ronald. > > > > > > > > > > > >>> Thanks, > >>> > >>> > >>> [1] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233010 > >>> [2] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255233 > >>> > >>> -- > >>> Alexis Praga > You can greatly reduce the build-time for devel/llvm* by changing the config to BE_NATIVE to avoid building backends for all FreeBSD supported platforms. Obviously this is not acceptable for many cases, but if you never cross-compile for other platforms, it's a really big win. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkober...@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On 2021-May-20, at 11:11, Mark Millard wrote: > Andrea Venturoli ml at netfence.it wrote on > Thu May 20 14:54:50 UTC 2021 : > >> On 5/20/21 4:11 PM, Alexis Praga wrote: >>> >>> Thanks for the suggestions. >>> At least, if your machine is powerful enough, you can edit your /usr/local/etc/poudriere.conf and list these big ports in ALLOW_MAKE_JOBS_PACKAGES. >>> I'm using ALLOW_MAKE_JOBS=yes already so it won't help unfortunately, I >>> think. >> >> Those are not the same things. Check the docs. >> >> Briefly, ALLOW_MAKE_JOBS will allow building several packages at once >> (one per core) with one core dedicated to each. > > Nope: /usr/local/etc/poudriere.conf.sample reports that it enables > more than one process per cpu: > > # By default MAKE_JOBS is disabled to allow only one process per cpu > # Use the following to allow it anyway > # ALLOW_MAKE_JOBS=yes My wording was poor by being incomplete: ALLOW_MAKE_JOBS allows each builder to have as many processes as there are cores/processors. So if multiple builders are allowed (and happen to be in use), then the total number of processes ready to run for building can be bigger than the number of cores/processors, something that shows up in the load averages. How many builders are allowed can be controlled with: # parallel build support. # # By default poudriere uses hw.ncpu to determine the number of builders. # You can override this default by changing PARALLEL_JOBS here, or # by specifying the -J flag to bulk/testport. # # Example to define PARALLEL_JOBS to one single job # PARALLEL_JOBS=1 Looking in /usr/ports/Mk/bsd.port.mk there is also the following, that is part of where some of the "MAKE_JOBS" terminology comes from: # MAKE_JOBS_UNSAFE # - Disallow multiple jobs even when user set a global override. # To be used with known bad ports. # DISABLE_MAKE_JOBS # - Set to disable the multiple jobs feature. User settable. # MAKE_JOBS_NUMBER # - Override the number of make jobs to be used. User settable. # MAKE_JOBS_NUMBER_LIMIT # - Set a limit for maximum number of make jobs allowed to be # used. >> ALLOW_MAKE_JOBS_PACKAGES will let a single package build using more cores. > > > Even when MAKE_JOBS is otherwise disabled: > > # List of packages that will always be allowed to use MAKE_JOBS > # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports > # which holdup the rest of the queue to build more quickly. > #ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*" === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
Andrea Venturoli ml at netfence.it wrote on Thu May 20 14:54:50 UTC 2021 : > On 5/20/21 4:11 PM, Alexis Praga wrote: > > > > Thanks for the suggestions. > > > >> At least, if your machine is powerful enough, you can edit your > >> /usr/local/etc/poudriere.conf and list these big ports in > >> ALLOW_MAKE_JOBS_PACKAGES. > > I'm using ALLOW_MAKE_JOBS=yes already so it won't help unfortunately, I > > think. > > Those are not the same things. Check the docs. > > Briefly, ALLOW_MAKE_JOBS will allow building several packages at once > (one per core) with one core dedicated to each. Nope: /usr/local/etc/poudriere.conf.sample reports that it enables more than one process per cpu: # By default MAKE_JOBS is disabled to allow only one process per cpu # Use the following to allow it anyway # ALLOW_MAKE_JOBS=yes > ALLOW_MAKE_JOBS_PACKAGES will let a single package build using more cores. Even when MAKE_JOBS is otherwise disabled: # List of packages that will always be allowed to use MAKE_JOBS # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports # which holdup the rest of the queue to build more quickly. #ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*" === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On 2021-05-20 06:15, Ronald Klop wrote: On 5/20/21 2:47 PM, Fernando Apesteguía wrote: El jue., 20 may. 2021 14:45, Alexis Praga escribió: Hi guys, I've recently been using poudriere to test news ports but this resulted in building two large ports, namely: - devel/llvm [1] - lang/rust [2] Now, lang/rust takes 2 hourse to compile. So testing for 13.0, 12.2 and 11.4 means around 6 hours in total, a lot of time for my desktop computer. Especially if the dependencies are upgraded later on. Do you have any tips to deal with such long build time ? Apart from have a dedicated server :) Patience :-) Package seeding in poudriere is a long awaited feature. You can manually put the rust/llvm package in the All packages directory and poudriere will use that. It is a manual repeating task though. So the package seeding is indeed a nice feature. I can't believe I didn't already know that. :-/ Thanks, Ronald. Today you are a hero! :-) --Chris Ronald. Thanks, [1] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233010 [2] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255233 -- Alexis Praga ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" signature.asc Description: OpenPGP digital signature
Re: Large builds with poudriere
On 2021-05-20 05:47, Fernando Apesteguía wrote: El jue., 20 may. 2021 14:45, Alexis Praga escribió: Hi guys, I've recently been using poudriere to test news ports but this resulted in building two large ports, namely: - devel/llvm [1] - lang/rust [2] Now, lang/rust takes 2 hourse to compile. So testing for 13.0, 12.2 and 11.4 means around 6 hours in total, a lot of time for my desktop computer. Especially if the dependencies are upgraded later on. Do you have any tips to deal with such long build time ? Apart from have a dedicated server :) What? You don't already have a dedicated build server? ;-) Patience :-) Package seeding in poudriere is a long awaited feature. So where is the pkg install llvm && rust before starting the build job option? In all honesty. This is my sole reason for not choosing poudriere. Unless I'm performing MAINTAINER work on my ports. Even with 8 cores, 16 threads @4.2Ghz && 128Gb RAM. It's a daunting task. I generally use jail(8) otherwise. --Chris Thanks, [1] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233010 [2] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255233 -- Alexis Praga ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" signature.asc Description: OpenPGP digital signature
Re: Large builds with poudriere
On 5/20/21 4:11 PM, Alexis Praga wrote: Thanks for the suggestions. At least, if your machine is powerful enough, you can edit your /usr/local/etc/poudriere.conf and list these big ports in ALLOW_MAKE_JOBS_PACKAGES. I'm using ALLOW_MAKE_JOBS=yes already so it won't help unfortunately, I think. Those are not the same things. Check the docs. Briefly, ALLOW_MAKE_JOBS will allow building several packages at once (one per core) with one core dedicated to each. ALLOW_MAKE_JOBS_PACKAGES will let a single package build using more cores. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
On Thu, 20 May 2021 16:11:21 +0200, Alexis Praga stated: >Thanks for the suggestions. > >> At least, if your machine is powerful enough, you can edit your >> /usr/local/etc/poudriere.conf and list these big ports in >> ALLOW_MAKE_JOBS_PACKAGES. >I'm using ALLOW_MAKE_JOBS=yes already so it won't help unfortunately, >I think. > >> You can manually put the rust/llvm package in the All packages >> directory and poudriere will use that. It is a manual repeating task >> though. So the package seeding is indeed a nice feature. >I just tried to copy rust-1.52-1.txz from one jail "all" to another >"all" but it's building anyway. > > >Regarding package seeding, a github issue mentions it's already >implemented but not working for some package (like llvm): >https://github.com/freebsd/poudriere/issues/319 > >-- >Alexis Praga Have you tried "devel/ccache"? I have it incorporated into poudriere and it makes quite a difference. The first time you build a large package, it won't make any difference; however, on subsequent builds, it cut the build time significantly on my system. -- Carmel ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
Thanks for the suggestions. > At least, if your machine is powerful enough, you can edit your > /usr/local/etc/poudriere.conf and list these big ports in > ALLOW_MAKE_JOBS_PACKAGES. I'm using ALLOW_MAKE_JOBS=yes already so it won't help unfortunately, I think. > You can manually put the rust/llvm package in the All packages directory and > poudriere will use that. It is a manual > repeating task though. So the package seeding is indeed a nice feature. I just tried to copy rust-1.52-1.txz from one jail "all" to another "all" but it's building anyway. Regarding package seeding, a github issue mentions it's already implemented but not working for some package (like llvm): https://github.com/freebsd/poudriere/issues/319 -- Alexis Praga ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
Le jeu. 20 mai 21 à 14:47:35 +0200, Fernando Apesteguía écrivait : > > Hi guys, Hello, > > I've recently been using poudriere to test news ports but this resulted in > > building two large ports, namely: > > - devel/llvm [1] > > - lang/rust [2] > > > > Now, lang/rust takes 2 hourse to compile. So testing for 13.0, 12.2 and > > 11.4 means around 6 hours in total, a lot of time for my desktop > > computer. Especially if the dependencies are upgraded later on. > > > > Do you have any tips to deal with such long build time ? Apart from > > have a dedicated server :) > > > > Patience :-) At least, if your machine is powerful enough, you can edit your /usr/local/etc/poudriere.conf and list these big ports in ALLOW_MAKE_JOBS_PACKAGES. > Package seeding in poudriere is a long awaited feature. + 1 -- Th. Thomas. signature.asc Description: PGP signature
Re: Large builds with poudriere
On 5/20/21 2:47 PM, Fernando Apesteguía wrote: El jue., 20 may. 2021 14:45, Alexis Praga escribió: Hi guys, I've recently been using poudriere to test news ports but this resulted in building two large ports, namely: - devel/llvm [1] - lang/rust [2] Now, lang/rust takes 2 hourse to compile. So testing for 13.0, 12.2 and 11.4 means around 6 hours in total, a lot of time for my desktop computer. Especially if the dependencies are upgraded later on. Do you have any tips to deal with such long build time ? Apart from have a dedicated server :) Patience :-) Package seeding in poudriere is a long awaited feature. You can manually put the rust/llvm package in the All packages directory and poudriere will use that. It is a manual repeating task though. So the package seeding is indeed a nice feature. Ronald. Thanks, [1] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233010 [2] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255233 -- Alexis Praga ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Large builds with poudriere
El jue., 20 may. 2021 14:45, Alexis Praga escribió: > > Hi guys, > > I've recently been using poudriere to test news ports but this resulted in > building two large ports, namely: > - devel/llvm [1] > - lang/rust [2] > > Now, lang/rust takes 2 hourse to compile. So testing for 13.0, 12.2 and > 11.4 means around 6 hours in total, a lot of time for my desktop > computer. Especially if the dependencies are upgraded later on. > > Do you have any tips to deal with such long build time ? Apart from > have a dedicated server :) > Patience :-) Package seeding in poudriere is a long awaited feature. > Thanks, > > > [1] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233010 > [2] For https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255233 > > -- > Alexis Praga > ___ > freebsd-ports@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" > ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
