Re: poudriere: howto build ports with 'make -j x'?

[Ultima]

 Chromium and Libreoffice are two programs that are massive and have the
longest compile times in the ports tree along with eclipse. If the system
only has 2 cpus I would expect those to take at least 4-8 hours minimum. If
they are also building in parallel you can expect that to also increase. My
system takes 40 minutes for Chromium and 50 for Libreoffice, and thats with
jobs=40 on 2.0 cores.

Sorry if my comment was confusing, this variable's use-case is better to
limit the amount of jobs per jail on a given system. Enabling
ALLOW_MAKE_JOBS=yes will set the makejobs on all jails to maximum amount of
cpu's, in you're case 2.

The reason I made my comment, if makejobs > 64 some ports will begin to
fail to build due to too many makejobs. Also, for multiple threaded cpu's
it may increase performance setting it lower when running many jails in
parallel.

> These I do run as single steps each from a shell script

> poudriere bulk -j freebsd-head -p ports-head www/firefox
> poudriere bulk -j freebsd-head -p ports-head www/chromium

It may run faster, but only because they are not building in parallel.

On Thu, Aug 11, 2016 at 3:49 AM, Matthias Apitz  wrote:

> El día Saturday, August 06, 2016 a las 02:13:39PM -0400, Ultima escribió:
>
> > If you plan on running build in parallel it maybe necessary to also
> > set MAKE_JOBS_NUMBER_LIMIT= into the make.conf file as well.
>
> I have a list of some 270 ports which poudriere should work through with
> 3 builders, resulting in some 1700 ports to build:
>
> poudriere bulk -f /usr/local/etc/poudriere-list -J 3 -j freebsd-head -p
> ports-head
>
> This list does not include some ports which are verry time consuming on
> my hardware (2 CPU 3.7 GHz) like
>
> www/firefox
> www/chromium
> editors/openoffice-devel
> editors/libreoffice
> ...
>
> These I do run as single steps each from a shell script
>
> poudriere bulk -j freebsd-head -p ports-head www/firefox
> poudriere bulk -j freebsd-head -p ports-head www/chromium
> ...
>
> matthias
> --
> Matthias Apitz, ✉ g...@unixarea.de, ⌂ http://www.unixarea.de/  ☎
> +49-176-38902045
> "Ohne die Mauer hätte es Krieg gegeben" Fritz Streletz u.a.
> "Sin el Muro hubiese habido guerra."
>
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: poudriere: howto build ports with 'make -j x'?

[Matthias Apitz]

El día Saturday, August 06, 2016 a las 02:13:39PM -0400, Ultima escribió:

> If you plan on running build in parallel it maybe necessary to also
> set MAKE_JOBS_NUMBER_LIMIT= into the make.conf file as well.

I have a list of some 270 ports which poudriere should work through with
3 builders, resulting in some 1700 ports to build:

poudriere bulk -f /usr/local/etc/poudriere-list -J 3 -j freebsd-head -p 
ports-head

This list does not include some ports which are verry time consuming on
my hardware (2 CPU 3.7 GHz) like

www/firefox
www/chromium
editors/openoffice-devel
editors/libreoffice
...

These I do run as single steps each from a shell script

poudriere bulk -j freebsd-head -p ports-head www/firefox
poudriere bulk -j freebsd-head -p ports-head www/chromium
...

matthias
-- 
Matthias Apitz, ✉ g...@unixarea.de, ⌂ http://www.unixarea.de/  ☎ 
+49-176-38902045
"Ohne die Mauer hätte es Krieg gegeben" Fritz Streletz u.a.
"Sin el Muro hubiese habido guerra."
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: poudriere: howto build ports with 'make -j x'?

[Ultima]

If you plan on running build in parallel it maybe necessary to also
set MAKE_JOBS_NUMBER_LIMIT= into the make.conf file as well.

On Sat, Aug 6, 2016 at 11:49 AM, Michael Grimm  wrote:

> Christian Schwarz  wrote:
>
> > From my poudriere.conf:
> >
> >  # By default MAKE_JOBS is disabled to allow only one process per cpu
> >  # Use the following to allow it anyway
> >  ALLOW_MAKE_JOBS=no
> >
> >  # List of packages that will always be allowed to use MAKE_JOBS
> >  # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
> >  # which holdup the rest of the queue to build more quickly.
> >  ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py* perl5.20"
> >
> > Hope this helps!
>
> Yes, it helped. Especially me being blind to find that in the first place
> :-(
>
> Thanks to all who helped me finding these knobs.
>
> Regards,
> Michael
>
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
>
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: poudriere: howto build ports with 'make -j x'?

[Michael Grimm]

Christian Schwarz  wrote:

> From my poudriere.conf:
> 
>  # By default MAKE_JOBS is disabled to allow only one process per cpu
>  # Use the following to allow it anyway
>  ALLOW_MAKE_JOBS=no
> 
>  # List of packages that will always be allowed to use MAKE_JOBS
>  # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
>  # which holdup the rest of the queue to build more quickly.
>  ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py* perl5.20"
> 
> Hope this helps!

Yes, it helped. Especially me being blind to find that in the first place :-(

Thanks to all who helped me finding these knobs.

Regards,
Michael

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: poudriere: howto build ports with 'make -j x'?

[Christian Schwarz]

>From my poudriere.conf:

  # By default MAKE_JOBS is disabled to allow only one process per cpu
  # Use the following to allow it anyway
  ALLOW_MAKE_JOBS=no
  
  # List of packages that will always be allowed to use MAKE_JOBS
  # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
  # which holdup the rest of the queue to build more quickly.
  ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py* perl5.20"

Hope this helps!

--
Christian Schwarz
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: poudriere: howto build ports with 'make -j x'?

[Kubilay Kocak]

On 7/08/2016 1:16 AM, Michael Grimm wrote:
> Hi,
> 
> I am used to build my ports by poudriere in parallel (-J x). But I do want to 
> build specific ports in parallel per se (make -j x) just to speed up build 
> times of huge ports like mariadb101-server.
> 
> I did try to add an MAKEOPTS="-j8" into poudriere's specific 
> jailname-make.conf file, and I did try to use a jailname-setfile-make.conf as 
> well. But that didn't work for me.
> 
> What would be the correct way to achieve what I want?
> 
> Thanks and regards,
> Michael
> 

LOCALBASE/etc/poudriere.conf has some settings for this:

# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
#ALLOW_MAKE_JOBS=yes

# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
ALLOW_MAKE_JOBS_PACKAGES="pkg gcc cmake py*"

MAKE_JOBS is a ports framework variable (see Mk/bsd.port.mk)

./koobs


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: poudriere: howto build ports with 'make -j x'?

[Tobias Kortkamp]

Hi Michael,

you can set ALLOW_MAKE_JOBS_PACKAGES in /usr/local/etc/poudriere.conf to
do that. 

# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
#ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"

and MAKE_JOBS_NUMBER in jailname-make.conf to set the number of jobs you
want.

On Sat, Aug 6, 2016, at 17:16, Michael Grimm wrote:
> Hi,
> 
> I am used to build my ports by poudriere in parallel (-J x). But I do
> want to build specific ports in parallel per se (make -j x) just to speed
> up build times of huge ports like mariadb101-server.
> 
> I did try to add an MAKEOPTS="-j8" into poudriere's specific
> jailname-make.conf file, and I did try to use a
> jailname-setfile-make.conf as well. But that didn't work for me.
> 
> What would be the correct way to achieve what I want?
> 
> Thanks and regards,
> Michael
> 
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

poudriere: howto build ports with 'make -j x'?

[Michael Grimm]

Hi,

I am used to build my ports by poudriere in parallel (-J x). But I do want to 
build specific ports in parallel per se (make -j x) just to speed up build 
times of huge ports like mariadb101-server.

I did try to add an MAKEOPTS="-j8" into poudriere's specific jailname-make.conf 
file, and I did try to use a jailname-setfile-make.conf as well. But that 
didn't work for me.

What would be the correct way to achieve what I want?

Thanks and regards,
Michael

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"