Re: dpb build box performance suggestions.

[lists]

On Wed, 16 Dec 2015 23:45:43 -0500 Andre Smagin  wrote:

> On Wed, 16 Dec 2015 23:15:29 +
> Tati Chevron  wrote:
> 
> > Really, have a look at the dependencies for ImageMagick, and ask yourself
> > who really uses djvu, for example.  Removing it and ghostscript reduces
> > the dependencies from:  
> 
> Plenty of people read books in djvu format and use ImageMagick to work
> with it.

Plenty of people read books, indeed.  Also plenty of people use the
packages available in OpenBSD and focus on actual work done, sometimes
even with the help of books.  And plenty of people preferred OpenBSD
on the desktop for having the packages before FreeBSD thus not having to
waste good part of their life time attending to building them on their
own.  For personal use, it makes much more sense, than at work,
though.  Interesting enough, common pitfall is time spent at work in non
productive activities somehow misses the notion of personal time wasted.

That's exactly why the recommended system was a fast desktop, and not a
virtual instance on a laptop.  Hint included, you can find multiple
uses for the build machine when you're done building (in more than one
meaning of "done").  It's alright to roll your packages, please don't
say we need not have some software because you hate waiting for it to be
built, we don't have to wait ;-)

Another tip is to build the packages on one of the machines where
you'll be using them, and export that to the others.  One can even use
ubiquitous machines instead of expensive purpose built embedded
systems, small power savings are not comparable with human time, before
going large scale.  This goes to say that having bought such systems,
fine, playing and tinkering, cool, but considering to buy them fresh...
not that trendy any more, given they are pricier and somewhat
constrained.

As naddy@ said: just get going and don't try to over-engineer the
perfect solution at first.

> There are many old and valuable, but long out of print books
> that were scanned and encoded to djvu format a decade or more ago.

Exactly right.

> Converting such books to pdf format using open source tools is usually
> difficult without drastically reducing the quality or increasing the
> file size two- or threefold. And when you do decide to convert, you
> need the ImageMagick or similar software.

And ImageMagick is also useful in other cases even as a stand alone
tool kit.  PHP land is not fun to be into and fails to realise UNIX
ways.

> I am grateful to OpenBSD developers and porters for supporting various
> seemingly obscure dependencies and software packages, even though they
> may seem to be useless to the majority of the users.

Thank you for spelling it out, Andre, and to OpenBSD developers for
providing us one smooth and efficient desktop environment and an
abundance or ready to install packages.

Re: dpb build box performance suggestions.

[Tati Chevron]

[ massive dependency list snipped... ]

Thank you!!! This hits the nail on the head. One of the twenty four
things I currently want is editors/emacs,gtk2. That wants ImageMagick... I
stopped the dpb build this morning at I=417 ports. As far as I'm
concerned that's off the chain. I'm trying to decide between figuring
out who the big players are in my dependency chain or just going with
editors/emacs,no_x11 and using tramp and or git when I want bells and
whistles emacs.


This is why I never leave the ports build system running unattended on
a machine where it can find it's way out to the internet :-).

If you want to retain some control over what's installed automatically
as dependencies, you have several choices.  The easiest way I have found
is to collect the distfiles that you might need in advance, using the
-F option to dpb, and then looking through what it downloads to see if
actually doing the build will require installing anything obnoxious.

If it does, examine the individual ports makefiles and see if it's an
unnecessary, (for you), dependency, that can be eliminated.

When you are ready to do the build, if the machine has outside
connectivity, using -f 0 should stop it from downloading additional
source, and only using what is available in it's own distfiles directory.
That way, it will stop with an error rather then pull more stuff in.

--
Tati Chevron
Perl and FORTRAN specialist.
SWABSIT development and migration department.
http://www.swabsit.com

dpb build box performance suggestions.

[Christopher Sean Hilton]

I'm trying to dpb to maintain a small set of packages for a handfull
of OpenBSD boxes that I run. These boxes will all be single purpose
servers of some type or another. Many of them will run with limited
disk space and memory on Soekris hardware. What resources do I want on
my dpb/build box to make it fast?

My dpb/build box is a VMWare virtual machine on a host with SSD
storage. Tweaking the number of available CPU's, the memory, or the
type of storage is relatively simple further, I can split the task and
have a fast build VM and an install virtual machine which shares httpd
available storage via NFS.

Thanks in advance for any help/advice.

[demime 1.01d removed an attachment of type application/pgp-signature which had 
a name of signature.asc]

Re: dpb build box performance suggestions.

[Tati Chevron]

On Wed, Dec 16, 2015 at 01:01:51PM -0500, Christopher Sean Hilton wrote:

I'm trying to dpb to maintain a small set of packages for a handfull
of OpenBSD boxes that I run. These boxes will all be single purpose
servers of some type or another. Many of them will run with limited
disk space and memory on Soekris hardware. What resources do I want on
my dpb/build box to make it fast?


What do you mean by, 'fast'?

Our couple of build machines are both fairly standard core i5 boxes with
16 gb of RAM, and Corsair SSDs.  The RAM seems to make more difference
than anything else, because you can set the work directory to a ramdisk,
and do the entire build without touching the disk.

I wrote a short paper on tweaking the ports build system here:

http://www.gotati.com/papers/customising_openbsd_ports.html


My dpb/build box is a VMWare virtual machine on a host with SSD
storage. Tweaking the number of available CPU's, the memory, or the
type of storage is relatively simple


It's probably best to experiment and see what works best for your workload.
Much depends on the specific ports you are building, whether they will
benefit most from RAM or CPU, for example.

Also, be aware that some ports have a mass of unnecessary dependencies,
and that tweaking this can reduce the build time substantially, especially
if you are building the same packages repeatedly for some reason.

--
Tati Chevron
Perl and FORTRAN specialist.
SWABSIT development and migration department.
http://www.swabsit.com

Re: dpb build box performance suggestions.

[Christian Weisgerber]

On 2015-12-16, Christopher Sean Hilton  wrote:

> I'm trying to dpb to maintain a small set of packages for a handfull
> of OpenBSD boxes that I run. These boxes will all be single purpose
> servers of some type or another. Many of them will run with limited
> disk space and memory on Soekris hardware. What resources do I want on
> my dpb/build box to make it fast?

I wouldn't overthink it.

The number one limit is CPU.  Faster CPU, better.

Regarding memory, you want to avoid going into swap.  On amd64, the
biggest pig in the build is lang/pypy which requires ~4GB.  The
second biggest ones are the Mozillas, which take ~2GB during linking.
(binutils 2.17 may have shaved off a few hundred MB there, I haven't
really payed attention.)  The vast majority of ports take far, far
less memory.  So your memory requirements really depend on how many
of those big ports you will end up building in parallel.  With
ncpu*2GB but a minimum of 4GB you should be on the safe side.

Disk doesn't matter much.  If you run off magnetic disk, you want
to use soft updates at least for the work and log directories.

Probably the biggest question is how many cores to use for the
build.  At the low level, our SMP scales poorly.  More cores are
faster than fewer cores, but also mean that ever more CPU goes into
spinning on the big lock instead of compiling.  At the high level,
dpb's ability to distribute build jobs to all cores is limited by
the numer of ports and their interdependencies.  It works well for
building the whole ports tree, but if you only do a "small set of
packages", the build may have to wait for some port that everything
else depends on.

Probably the biggest hint for building small sets of packages on
more than one core is to increase dpb's parallel property (-p flag)
to ncpu, from its default of ncpu/2, so you won't see half the cores
idling while the build blocks on something like gcc/4.9 or llvm.

Anyway, as I said above, don't overthink it.  Do an initial build
with modest resources, see how it goes.

-- 
Christian "naddy" Weisgerber  na...@mips.inka.de

Re: dpb build box performance suggestions.

[Christian Weisgerber]

On 2015-12-16, Tati Chevron  wrote:

> Our couple of build machines are both fairly standard core i5 boxes with
> 16 gb of RAM, and Corsair SSDs.  The RAM seems to make more difference
> than anything else, because you can set the work directory to a ramdisk,
> and do the entire build without touching the disk.

Have you done actual comparisons?  With SSDs, I don't expect a
significant difference.  (There is none for doing a "make build"
of the base system.)

-- 
Christian "naddy" Weisgerber  na...@mips.inka.de

Re: dpb build box performance suggestions.

[Michael McConville]

Christian Weisgerber wrote:
> On 2015-12-16, Tati Chevron  wrote:
> 
> > Our couple of build machines are both fairly standard core i5 boxes
> > with 16 gb of RAM, and Corsair SSDs.  The RAM seems to make more
> > difference than anything else, because you can set the work
> > directory to a ramdisk, and do the entire build without touching the
> > disk.
> 
> Have you done actual comparisons?  With SSDs, I don't expect a
> significant difference.  (There is none for doing a "make build" of
> the base system.)

FWIW: A few days ago, we added additional zeroing to tmpfs and were
discussing its performance. Another dev mentioned that their SSD is
already typically faster than tmpfs.

Re: dpb build box performance suggestions.

[Tati Chevron]

On Wed, Dec 16, 2015 at 06:00:10PM -0500, Michael McConville wrote:

Christian Weisgerber wrote:

On 2015-12-16, Tati Chevron  wrote:

> Our couple of build machines are both fairly standard core i5 boxes
> with 16 gb of RAM, and Corsair SSDs.  The RAM seems to make more
> difference than anything else, because you can set the work
> directory to a ramdisk, and do the entire build without touching the
> disk.

Have you done actual comparisons?  With SSDs, I don't expect a
significant difference.  (There is none for doing a "make build" of
the base system.)


FWIW: A few days ago, we added additional zeroing to tmpfs and were
discussing its performance. Another dev mentioned that their SSD is
already typically faster than tmpfs.


We have WRKOBJDIR on an 8 GB MFS partition, that uses half of the 16 GB
installed RAM.

An observation: we are not running a GENERIC kernel on these boxes, so
results will be skewed.  In addition, one of the build machines is
running with softraid crypto on all of it's mounted filesystems.  This
is for no reason other than to stress the softraid code.

--
Tati Chevron
Perl and FORTRAN specialist.
SWABSIT development and migration department.
http://www.swabsit.com

Re: dpb build box performance suggestions.

[lists]

On Wed, 16 Dec 2015 18:58:48 + Tati Chevron 
wrote:

> On Wed, Dec 16, 2015 at 01:01:51PM -0500, Christopher Sean Hilton wrote:
> >I'm trying to dpb to maintain a small set of packages for a handfull
> >of OpenBSD boxes that I run. These boxes will all be single purpose
> >servers of some type or another. Many of them will run with limited
> >disk space and memory on Soekris hardware. What resources do I want on
> >my dpb/build box to make it fast?  
> 
> What do you mean by, 'fast'?

The dictionary antonym of slow.  What do you mean by mean?

> Our couple of build machines are both fairly standard core i5 boxes with
> 16 gb of RAM, and Corsair SSDs.

This season the sweet spot goes towards i7 4770 / 4790 with 32 GB RAM.
The logic behind this is the fastest desktop CPU for around $300-$350
price point with as much RAM as it can hold.  DDR4 is not mainstream,
yet, and this goes for some time.  As for storage, the lowest size of
Intel 3500 / 3510 SSDs and the 2-3 GB HGST HDDs, both 1x per system.
No RAID, or just pass-through.  You can spare the SSD cost, but don't
use silly gamer/enthusiast desktop class jagged metal style covered
parts that hide a flaky controller board (go for the Data Centre
optimised versions from Intel, really discard advice from gamers).
Don't forget 12 cm fans and a reliable 350-400 W PSU (cheap and almost
OK goes for Seasonic), and have a cold standby PSU unit just in case.
Oh yes, pick a Supermicro motherboard with a dedicated IPMI LAN port,
you'll find use for it.

> The RAM seems to make more difference than anything else, because you
> can set the work directory to a ramdisk, and do the entire build
> without touching the disk.

Nice point.
 
> I wrote a short paper on tweaking the ports build system here:
> 
> http://www.gotati.com/papers/customising_openbsd_ports.html
> 
> >My dpb/build box is a VMWare virtual machine on a host with SSD
> >storage. Tweaking the number of available CPU's, the memory, or the
> >type of storage is relatively simple  
> 
> It's probably best to experiment and see what works best for your workload.
> Much depends on the specific ports you are building, whether they will
> benefit most from RAM or CPU, for example.

Or both.  Drop VMWare on the floor NOW, if you need virtualisation use
generic QEMU/KVM in any recent Linux distribution of your choice and
plan to wipe it clean after you're done fiddling with it.  Yes, really
seriously remove the virtualisation for a build machine, go bare metal.
Try without hyperthreading for a comparison.  Before you notice and get
to complain you need VM for something just use the native OpenBSD
hypervisor.

> Also, be aware that some ports have a mass of unnecessary dependencies,
> and that tweaking this can reduce the build time substantially, especially
> if you are building the same packages repeatedly for some reason.

Use a "virtual" axe ;-) virtually "axing" around.

Re: dpb build box performance suggestions.

[Tati Chevron]

Or both.  Drop VMWare on the floor NOW, if you need virtualisation use
generic QEMU/KVM in any recent Linux distribution of your choice and
plan to wipe it clean after you're done fiddling with it.  Yes, really
seriously remove the virtualisation for a build machine, go bare metal.
Try without hyperthreading for a comparison.  Before you notice and get
to complain you need VM for something just use the native OpenBSD
hypervisor.


Our build machines both run on bare metal.  To be honest, once you've
pulled the entire set of source distfiles for one release, you don't even
need much in the way of connectivity to stay up to date.


From the way the OP described the setup, it does look like he intends to

run the build machine remotely, as a VPS.  I wouldn't recommend using a
VPS as a build machine, as you need CPU and RAM with little connectivity,
which is the opposite of what most VPS providers will offer.  Our build
machines are on-site, and we just send the resulting binary packages
wherever they need to go.


Also, be aware that some ports have a mass of unnecessary dependencies,
and that tweaking this can reduce the build time substantially, especially
if you are building the same packages repeatedly for some reason.


Use a "virtual" axe ;-) virtually "axing" around.


Really, have a look at the dependencies for ImageMagick, and ask yourself
who really uses djvu, for example.  Removing it and ghostscript reduces
the dependencies from:

5.8-release:

# make print-build-depends

This port requires package(s) "bzip2-1.0.6p1 libusb1-1.0.9p9 lynx-2.8.9pl6
gmp-5.0.2p3 jbigkit-2.1 metaauto-1.0p1 autoconf-2.13p3 libelf-0.8.13p3
xz-5.2.1 png-1.6.17 help2man-1.47.1 autoconf-2.69p1 libffi-3.1p0
automake-1.14.1 automake-1.11.6p1 autoconf-2.63p0 autoconf-2.67p0
libltdl-2.4.2p1 libtool-2.4.2p0 hicolor-icon-theme-0.15
p5-XML-NamespaceSupport-1.11p0 p5-XML-SAX-0.96p1 giflib-5.1.1 imake-cf-1.0.5
imake-1.0.7 jpeg-9a tiff-4.0.4 lcms2-2.6p1 libwebp-0.4.3 libwmf-0.2.8.4p3
libiconv-1.14p3 gettext-0.19.5.1 gettext-tools-0.19.5.1 gmake-4.1p0
gnugetopt-1.1.6 libpaper-1.1.24.4 libnettle-3.1.1 icu4c-55.1p0 nspr-4.10.8
fftw3-common-3.2.2p1 fftw3-3.2.2p3 bash-4.3.39p0 libgpg-error-1.19
libgcrypt-1.6.3 libidn-1.32 curl-7.43.0 re2c-0.14.3 unzip-6.0p7
jasper-1.900.1p2 iso8879-1986p0 docbook-dsssl-1.79 ghostscript-fonts-8.11p3
p5-XML-Parser-2.44 xmltoman-0.4 intltool-0.51.0p0 pcre-8.37p1 libtasn1-4.5
p11-kit-0.22.1p1 gnutls-3.3.16 groff-1.22.3p2 gdbm-1.11p0 tcl-8.5.18 tk-8.5.18
ijs-0.35p2 libsigsegv-2.10p2 m4-1.4.17 bison-2.3p2 db-4.6.21p1v0 python-2.7.10
libxml-2.9.2p1 netpbm-10.35.96 docbook-4.5p1 jbig2dec-0.11 libxml-2.9.2p1
py-libxml-2.9.2p0 libxslt-1.1.28p2 docbook-xsl-1.68.1p5 xmlto-0.0.26p0
dbus-1.8.20v0 glib2-2.44.1 vala-0.28.0 dconf-0.24.0p1 shared-mime-info-1.4
libcroco-0.6.8p2 dbus-glib-0.104p0v0 dbus-1.8.20v0
dbus-daemon-launch-helper-1.8.20 docbook2x-0.8.8p1 py-setuptools-3.4.4p2v0
py-MarkupSafe-0.23 py-jinja2-2.7.3p0 py-tz-2015.4 py-babel-1.3p2
py-pygments-2.0.1 py-docutils-0.12 py-sphinx-1.2.3p0 py-crypto-2.6.1p0
py-beaker-1.6.2p3 py-mako-0.9.1p1 ninja-1.5.3p0 scons-2.3.5p0 jsoncpp-0.10.5
mozjs17-17.0p2 lzo2-2.09 cairo-1.14.2 gobject-introspection-1.44.0
gdk-pixbuf-2.30.8p1 atk-2.16.0 at-spi2-core-2.16.0p0 at-spi2-atk-2.16.0p0
polkit-0.113p3 consolekit-0.4.6p14 colord-1.2.11 json-glib-1.0.4
gsettings-desktop-schemas-3.16.1 libarchive-3.1.2 cmake-3.2.3p1
graphite2-1.2.4p0 harfbuzz-1.0.1 pango-1.36.8 librsvg-2.40.9 libproxy-0.4.11p3
glib2-networking-2.44.0 libsoup-2.50.0 librest-0.7.93p0 libdaemon-0.14p1
avahi-0.6.31p19 cups-libs-2.0.4 ghostscript-9.07p2 transfig-3.2.5ap0
gtk-update-icon-cache-3.16.6 djvulibre-3.5.27" to build.

# make print-run-depends

This port requires package(s) "hicolor-icon-theme-0.15 bzip2-1.0.6p1
jbigkit-2.1 xz-5.2.1 dbus-1.8.20v0 dbus-daemon-launch-helper-1.8.20
dbus-1.8.20v0 giflib-5.1.1 jpeg-9a tiff-4.0.4 lcms2-2.6p1 libdaemon-0.14p1
jasper-1.900.1p2 libiconv-1.14p3 libxml-2.9.2p1 gettext-0.19.5.1 gdbm-1.11p0
libltdl-2.4.2p1 ghostscript-fonts-8.11p3 pcre-8.37p1 libtasn1-4.5
fftw3-common-3.2.2p1 fftw3-3.2.2p3 ijs-0.35p2 gmp-5.0.2p3 libnettle-3.1.1
png-1.6.17 netpbm-10.35.96 jbig2dec-0.11 libwebp-0.4.3 libwmf-0.2.8.4p3
libffi-3.1p0 python-2.7.10 p11-kit-0.22.1p1 gnutls-3.3.16 libelf-0.8.13p3
glib2-2.44.1 avahi-0.6.31p19 cups-libs-2.0.4 ghostscript-9.07p2
transfig-3.2.5ap0 shared-mime-info-1.4 gdk-pixbuf-2.30.8p1
gtk-update-icon-cache-3.16.6 djvulibre-3.5.27" to run.

Removing djvu and ghostscript, (almost no loss of functionality):

# make print-build-depends

This port requires package(s) "bzip2-1.0.6p1 jbigkit-2.1 metaauto-1.0p1
xz-5.2.1 png-1.6.17 help2man-1.47.1 autoconf-2.69p1 libffi-3.1p0
autoconf-2.67p0 libltdl-2.4.2p1 giflib-5.1.1 jpeg-9a tiff-4.0.4 lcms2-2.6p1
libwebp-0.4.3 libwmf-0.2.8.4p3 libiconv-1.14p3 gettext-0.19.5.1
gettext-tools-0.19.5.1 gmake-4.1p0 fftw3-common-3.2.2p1 fftw3-3.2.2p3
unzip-6.0p7 jasper-1.900.1p2 groff-1.22.3p2 gdbm-1.11p0 tcl-8.5.18 tk-8.5.18
db-4.6.21p1v0 

Re: dpb build box performance suggestions.

[Tati Chevron]

On Wed, Dec 16, 2015 at 10:43:43PM +, Christian Weisgerber wrote:

On 2015-12-16, Tati Chevron  wrote:


Our couple of build machines are both fairly standard core i5 boxes with
16 gb of RAM, and Corsair SSDs.  The RAM seems to make more difference
than anything else, because you can set the work directory to a ramdisk,
and do the entire build without touching the disk.


Have you done actual comparisons?  With SSDs, I don't expect a
significant difference.  (There is none for doing a "make build"
of the base system.)


If you're really interested I could probably produce some benchmarks from
one of our build machines, but I did notice a modest increase in performance
doing everything in RAM.

--
Tati Chevron
Perl and FORTRAN specialist.
SWABSIT development and migration department.
http://www.swabsit.com

Re: dpb build box performance suggestions.

[Andre Smagin]

On Wed, 16 Dec 2015 23:15:29 +
Tati Chevron  wrote:

> Really, have a look at the dependencies for ImageMagick, and ask yourself
> who really uses djvu, for example.  Removing it and ghostscript reduces
> the dependencies from:

Plenty of people read books in djvu format and use ImageMagick to work
with it. There are many old and valuable, but long out of print books
that were scanned and encoded to djvu format a decade or more ago.
Converting such books to pdf format using open source tools is usually
difficult without drastically reducing the quality or increasing the
file size two- or threefold. And when you do decide to convert, you
need the ImageMagick or similar software.

I am grateful to OpenBSD developers and porters for supporting various
seemingly obscure dependencies and software packages, even though they
may seem to be useless to the majority of the users.

--
Andre

Re: dpb build box performance suggestions.

[Christopher Sean Hilton]

On Wed, Dec 16, 2015 at 11:15:29PM +, Tati Chevron wrote:
> >Or both.  Drop VMWare on the floor NOW, if you need virtualisation use
> >generic QEMU/KVM in any recent Linux distribution of your choice and
> >plan to wipe it clean after you're done fiddling with it.  Yes, really
> >seriously remove the virtualisation for a build machine, go bare metal.
> >Try without hyperthreading for a comparison.  Before you notice and get
> >to complain you need VM for something just use the native OpenBSD
> >hypervisor.
>
> Our build machines both run on bare metal.  To be honest, once you've
> pulled the entire set of source distfiles for one release, you don't even
> need much in the way of connectivity to stay up to date.
> 

Virtual is the only option but I'm not trying to mirror the entire ports
collection. I'm trying run a puppet/package server for a tiny fleet of
soekris boxen.

> From the way the OP described the setup, it does look like he intends to
> run the build machine remotely, as a VPS.  I wouldn't recommend using a
> VPS as a build machine, as you need CPU and RAM with little connectivity,
> which is the opposite of what most VPS providers will offer.  Our build
> machines are on-site, and we just send the resulting binary packages
> wherever they need to go.
>

It's not remote. It runs as one virtual server of two on a 2010
MacPro. My host is modest. It's a 2.8GHz Zeon with 24Gb of RAM and
0.5Tb of SSD. My ports list is equally modest. I generally run OpenBSD
as a server role. If I were to build an OpenBSD desktop, I would rely
on project's mirrors. There's a good argument to be made that me using
dpb is a fools errand but I like to rely on myself. My ports list is
equally modest at 24 ports right now. I expect it to grow but not by much.

> >>Also, be aware that some ports have a mass of unnecessary dependencies,
> >>and that tweaking this can reduce the build time substantially, especially
> >>if you are building the same packages repeatedly for some reason.
> >
> >Use a "virtual" axe ;-) virtually "axing" around.
> 
> Really, have a look at the dependencies for ImageMagick, and ask yourself
> who really uses djvu, for example.  Removing it and ghostscript reduces
> the dependencies from:
> 
> 5.8-release:
> 
> # make print-build-depends

[ massive dependency list snipped... ]

Thank you!!! This hits the nail on the head. One of the twenty four
things I currently want is editors/emacs,gtk2. That wants ImageMagick... I
stopped the dpb build this morning at I=417 ports. As far as I'm
concerned that's off the chain. I'm trying to decide between figuring
out who the big players are in my dependency chain or just going with
editors/emacs,no_x11 and using tramp and or git when I want bells and
whistles emacs.  

-- 
Chris

 __o  "All I was trying to do was get home from work."
   _`\<,_   -Rosa Parks
___(*)/_(*)_
Christopher Sean Hilton[chris/at/vindaloo/dot/com]