Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Dhananjay Balan]

On Mon, Aug 20, 2018 at 05:48:04AM -0700, Cy Schubert wrote:

> Make sure the archs are the same. Make sure the src and obj pathnames 
> on both servers are exactly the same. Make sure /usr/src (or wherever 
> you put it) is the same, it too needs to be rsynced.
> 
> The tiniest differences will lead to this error.

Starting from a make clean and rsyncing fixed the problems, I can
install and boot my laptop from the build on my server.

Thanks!
Dhananjay Balan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Cy Schubert]

In message <20180820062511.yaa2rimgmwpv3hxx@kazhap>, Dhananjay Balan 
writes:
> On Sat, Aug 18, 2018 at 04:36:15PM -0700, Cy Schubert wrote:
> > You can use NFS or rsync. Make sure the src and obj path names are 
> > exactly the same on all thes servers. I used to use NFS until a year or 
> > two ago when I started using different patches on different machines. 
> > On occasion I've used rsync followed by a NO_CLEAN, WORLDFAST, KERNFAST 
> > build on the target machine or simply installkernel and installworld.
> > 
>
> I had a quick try of this, with rsync. installkernel worked properly
> and I can boot, howver installworld fails with lot of $PROGRAM not
> found erroer for env, mktemp etc.
>
> Thanks for all the suggestions I will grok at the installworld a bit
> more, and will also give PkgBase a try.

Make sure the archs are the same. Make sure the src and obj pathnames 
on both servers are exactly the same. Make sure /usr/src (or wherever 
you put it) is the same, it too needs to be rsynced.

The tiniest differences will lead to this error.

Don't do this _during_ a build on the source machine, and other common 
sense things.


-- 
Cheers,
Cy Schubert 
FreeBSD UNIX: Web:  http://www.FreeBSD.org

The need of the many outweighs the greed of the few.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Dhananjay Balan]

On Sat, Aug 18, 2018 at 04:36:15PM -0700, Cy Schubert wrote:
> You can use NFS or rsync. Make sure the src and obj path names are 
> exactly the same on all thes servers. I used to use NFS until a year or 
> two ago when I started using different patches on different machines. 
> On occasion I've used rsync followed by a NO_CLEAN, WORLDFAST, KERNFAST 
> build on the target machine or simply installkernel and installworld.
> 

I had a quick try of this, with rsync. installkernel worked properly
and I can boot, howver installworld fails with lot of $PROGRAM not
found erroer for env, mktemp etc.

Thanks for all the suggestions I will grok at the installworld a bit
more, and will also give PkgBase a try.

-
Dhananjay Balan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Shane Ambler]

On 19/8/18 5:31 pm, Matthew Seaman wrote:
> On 19/08/2018 01:55, Shane Ambler wrote:
>>> I run 12-CURRENT on few machines, some more powerful that other (all
>>> of them x86_64, march varies). 
> 
>> You can use freebsd-update by setting up your own update server
>> https://www.freebsd.org/doc/en/articles/freebsd-update-server
> 
> freebsd-upgrade(8) only works for releases, and not 12-CURRENT as the OP
> specified.

While I haven't used it to make releases from current, I would be
surprised if it doesn't work for current. The default config for
release.sh is to checkout and build HEAD, so without setup, current
would be used when building for the update server.

I do expect an incremental numbering scheme would need to be used to
differentiate each build, patch level builds should fit that.


-- 
FreeBSD - the place to B...Software Developing

Shane Ambler

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[blubee blubeeme]

On Sun, Aug 19, 2018 at 6:30 PM O. Hartmann  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Am Sun, 19 Aug 2018 00:34:20 +0200
> Dhananjay Balan  schrieb:
>
> > Hi,
> >
> > I run 12-CURRENT on few machines, some more powerful that other (all
> > of them x86_64, march varies).
> >
> > Is there is a way to avoid building CURRENT on all machines? Rather
> > than building everywhere, can I just build it on the big server that I
> > have and copy and update my laptop?
> >
> > -
> > Dhananjay Balan
> [...]
>
> Yes, you can.
> We do this via a custom build and creating packages as this is introduced
> at
>
> https://wiki.freebsd.org/PkgBase
>
> But beware! As many others have already stated, take care about to use the
> least common
> denominator of achitectural specifications amongst your pools! This means
> to not use any
> kind of optimizations for a specific CPU type for pkg-base distributed
> builds!
>
> Because we build the local OS for fast/server machines always(!) with
> optimisations, the
> pkg-base builds are done in a separate way - which is very easy if you've
> once understood
> the really neat and great build framework of FreeBSD's!
>
> Instead of building the traditional (probably optimised) system from
> /usr/src
> and /usr/obj, now you've to consider a separate path like
> /pool/sources/CURRENT/src (our
> way, since we also try to build packages and object trees for 11.X), then
> setup a small
> build script that essentially sets
>
> MAKEOBJDIRPREFIX
> KERNCONFDIR
> KERNCONF
>
> and especially
>
> __MAKE_CONF
> SRCCONF
> SRC_ENV_CONF
>
> if you use usually optimisations for the native target build on the
> building host and
> more generic binaries for the Intel x86 crap for redistribution.
>
> Doing so, we also perform builds for some ARM64 based experimental boxes.
>
> The next step is then up to you how to distribute. We copy all the pkg
> stuff coming out
> of the build cycle to a folder which is accessible by pkg via HTTP(S) - so
> www/apache24
> is our platform to redistribute the binaries over the network and even to
> remote sites
> (beware of the security implications!). You also can distribute the
> obj-folder (/usr/obj,
> or, if using another approach, like /pool/sources/CURRENT/obj) via NFS.
>
> Once you've understood the (easy to learn) concept of building the source
> tree, creating
> the packages for pkg-base and having dealt with the more labor for the
> setting of a
> distribution server, you can use the most potent server/box on you network
> for building
> dedicated distributions
>
> Even a "Release" build is possible as long as there are not pitfalls like
> they occured
> during the transition from 11.X to more 12-CURRENT spcific development
> (i.e. LINT).
>
> If you use pkg-base as mentioned above, be aware to setup a proper
> pkg.conf file as
> introduced on the Wiki and please be also aware of the fact, that there is
> at the moment
> no sharp separation between pkg-base and oridnary pkg for packages - so
> take the
> warinig serious, that pkg delete -a will not only kill all packages, it
> also will kill
> all packages installed for the base system!
>
> Once installed you'll see how fast compared to source build the pkg-base
> installation
> is (although I still prefer source build, optimised builds ...).
>
> And by the way: depending on the sophistication of your build script for
> dedicated
> tree builds as mentioned, via manipulations of __MAKE_CONF, SRCCONF you're
> able to also
> build optimised binary trees for different CPU types, but you have to deal
> yourself
> with the fact that you've to put the binaries into a proper place and then
> delegate the
> URI in pkg.conf to the correct branch of your tree. The ABI environmental
> variable
> doesn't take care of the "set" you may have used to optimise. But this is
> something
> you're able to deal with easily yourself after having setup your basic
> build
> environment.
>
> Regards,
>
> oh
>
> - --
> O. Hartmann
>
> Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
> Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).
> -BEGIN PGP SIGNATURE-
>
> iLUEARMKAB0WIQQZVZMzAtwC2T/86TrS528fyFhYlAUCW3lGDwAKCRDS528fyFhY
> lMBeAf4nVIFEAgegbZyKDZvTQQD/9Q8mN+IY8aJ7Fzza23KgWQsM3ZP59Orh0mi2
> PA94ywn5hOjfTgzdYcp1lq3MCE84AgCBGAWAMTag87ru89JHm75NLsgDvEjPpYgG
> 9hW1Vrdoj9EeiR2HTv2ncTc/AkFPNhdyhe+EJqUg01rtAd9sdmdM
> =/rzO
> -END PGP SIGNATURE-
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>

I would say setup a poudriere build bot on your most powerful machine to
build and test all the applications and tools that your weaker machines
need, then setup a custom pkg repo so once poudriere builds all the ports
and create packages from them.

Then you use the custom pkg url to update your 

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[O. Hartmann]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Am Sun, 19 Aug 2018 00:34:20 +0200
Dhananjay Balan  schrieb:

> Hi,
> 
> I run 12-CURRENT on few machines, some more powerful that other (all
> of them x86_64, march varies). 
> 
> Is there is a way to avoid building CURRENT on all machines? Rather
> than building everywhere, can I just build it on the big server that I
> have and copy and update my laptop?
> 
> -
> Dhananjay Balan
[...]

Yes, you can.
We do this via a custom build and creating packages as this is introduced at

https://wiki.freebsd.org/PkgBase

But beware! As many others have already stated, take care about to use the 
least common
denominator of achitectural specifications amongst your pools! This means to 
not use any
kind of optimizations for a specific CPU type for pkg-base distributed builds!

Because we build the local OS for fast/server machines always(!) with 
optimisations, the
pkg-base builds are done in a separate way - which is very easy if you've once 
understood
the really neat and great build framework of FreeBSD's!

Instead of building the traditional (probably optimised) system from /usr/src
and /usr/obj, now you've to consider a separate path like 
/pool/sources/CURRENT/src (our
way, since we also try to build packages and object trees for 11.X), then setup 
a small
build script that essentially sets 
 
MAKEOBJDIRPREFIX
KERNCONFDIR
KERNCONF

and especially

__MAKE_CONF
SRCCONF
SRC_ENV_CONF

if you use usually optimisations for the native target build on the building 
host and
more generic binaries for the Intel x86 crap for redistribution.

Doing so, we also perform builds for some ARM64 based experimental boxes.

The next step is then up to you how to distribute. We copy all the pkg stuff 
coming out
of the build cycle to a folder which is accessible by pkg via HTTP(S) - so 
www/apache24
is our platform to redistribute the binaries over the network and even to 
remote sites
(beware of the security implications!). You also can distribute the obj-folder 
(/usr/obj,
or, if using another approach, like /pool/sources/CURRENT/obj) via NFS.

Once you've understood the (easy to learn) concept of building the source tree, 
creating
the packages for pkg-base and having dealt with the more labor for the setting 
of a
distribution server, you can use the most potent server/box on you network for 
building
dedicated distributions

Even a "Release" build is possible as long as there are not pitfalls like they 
occured
during the transition from 11.X to more 12-CURRENT spcific development (i.e. 
LINT).

If you use pkg-base as mentioned above, be aware to setup a proper pkg.conf 
file as
introduced on the Wiki and please be also aware of the fact, that there is at 
the moment
no sharp separation between pkg-base and oridnary pkg for packages - so take the
warinig serious, that pkg delete -a will not only kill all packages, it also 
will kill
all packages installed for the base system!

Once installed you'll see how fast compared to source build the pkg-base 
installation
is (although I still prefer source build, optimised builds ...).

And by the way: depending on the sophistication of your build script for 
dedicated
tree builds as mentioned, via manipulations of __MAKE_CONF, SRCCONF you're able 
to also
build optimised binary trees for different CPU types, but you have to deal 
yourself
with the fact that you've to put the binaries into a proper place and then 
delegate the
URI in pkg.conf to the correct branch of your tree. The ABI environmental 
variable
doesn't take care of the "set" you may have used to optimise. But this is 
something
you're able to deal with easily yourself after having setup your basic build
environment.

Regards,

oh   

- -- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).
-BEGIN PGP SIGNATURE-

iLUEARMKAB0WIQQZVZMzAtwC2T/86TrS528fyFhYlAUCW3lGDwAKCRDS528fyFhY
lMBeAf4nVIFEAgegbZyKDZvTQQD/9Q8mN+IY8aJ7Fzza23KgWQsM3ZP59Orh0mi2
PA94ywn5hOjfTgzdYcp1lq3MCE84AgCBGAWAMTag87ru89JHm75NLsgDvEjPpYgG
9hW1Vrdoj9EeiR2HTv2ncTc/AkFPNhdyhe+EJqUg01rtAd9sdmdM
=/rzO
-END PGP SIGNATURE-
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Miroslav Lachman]

Dhananjay Balan wrote on 2018/08/19 00:34:

Hi,

I run 12-CURRENT on few machines, some more powerful that other (all
of them x86_64, march varies).

Is there is a way to avoid building CURRENT on all machines? Rather
than building everywhere, can I just build it on the big server that I
have and copy and update my laptop?


Yes and we are using it (for STABLE, but it works for all versions).
You can build it on you build server and export it with NFS.

https://www.freebsd.org/cgi/man.cgi?query=development=7

Miroslav Lachman
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Matthew Seaman]

On 19/08/2018 01:55, Shane Ambler wrote:
>> I run 12-CURRENT on few machines, some more powerful that other (all
>> of them x86_64, march varies). 

> You can use freebsd-update by setting up your own update server
> https://www.freebsd.org/doc/en/articles/freebsd-update-server

freebsd-upgrade(8) only works for releases, and not 12-CURRENT as the OP
specified.

Cheers,

Matthew





signature.asc
Description: OpenPGP digital signature

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Shane Ambler]

On 19/8/18 8:04 am, Dhananjay Balan wrote:
> Hi,
> 
> I run 12-CURRENT on few machines, some more powerful that other (all
> of them x86_64, march varies). 
> 
> Is there is a way to avoid building CURRENT on all machines? Rather
> than building everywhere, can I just build it on the big server that I
> have and copy and update my laptop?

You can use freebsd-update by setting up your own update server
https://www.freebsd.org/doc/en/articles/freebsd-update-server
This can then be paired with poudriere for pkgs - see handbook 4.6

Also be careful setting the CPUTYPE, it has to be the lowest denominator
so that no cpu features are used on a machine without them. This is more
important with poudriere as many ports will configure to the current cpu
features, I think base always targets the lowest cpu specs.

-- 
FreeBSD - the place to B...Software Developing

Shane Ambler

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Mike Appleby]

On Sat, Aug 18, 2018 at 5:34 PM, Dhananjay Balan  wrote:
> Hi,
>
> I run 12-CURRENT on few machines, some more powerful that other (all
> of them x86_64, march varies).
>
> Is there is a way to avoid building CURRENT on all machines? Rather
> than building everywhere, can I just build it on the big server that I
> have and copy and update my laptop?
>

I've never actually tried it myself, but the following section of the
handbook
outlines one possibility for doing this via NFS:

https://www.freebsd.org/doc/handbook/small-lan.html
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[David Wolfskill]

On Sun, Aug 19, 2018 at 12:34:20AM +0200, Dhananjay Balan wrote:
> Hi,
> 
> I run 12-CURRENT on few machines, some more powerful that other (all
> of them x86_64, march varies). 
> 
> Is there is a way to avoid building CURRENT on all machines? Rather
> than building everywhere, can I just build it on the big server that I
> have and copy and update my laptop?
> 

There are ways (note plural) to do this kind of thing, yes.  Depending
on a lot of factors, some may be more feasible than others.

While the machines for which I do something similar run stable (at
present, stable/11), I believe that the approach I use, and which is
described at  may
(at least) be a useful case study.  There is also a log (of sorts) of
the successful updates at
.

Note that I explicitly avoid optimizations that would not work for all
of the machines in question.  That's probably obvious, but :-)

As Cy Schubert pointed out, paths need to be set up to look identical;
setting that up initially may be a little more "interesting" than one
might prefer, but in my experience, once that's done, things Just Work
(at least, using the approach I use).

I'll skip trying to mention other approaches, as I don't actually use
them.

Peace,
david
-- 
David H. Wolfskill  da...@catwhisker.org
Trump is gaslighting us: https://www.bbc.com/news/world-us-canada-44959300

See http://www.catwhisker.org/~david/publickey.gpg for my public key.


signature.asc
Description: PGP signature

Re: Sharing compiled builds between multiple 12-CURRENT boxes.

[Cy Schubert]

In message <20180818223420.rjisst4vuxzxbcrl@kazhap>, Dhananjay Balan 
writes:
> Hi,
>
> I run 12-CURRENT on few machines, some more powerful that other (all
> of them x86_64, march varies). 
>
> Is there is a way to avoid building CURRENT on all machines? Rather
> than building everywhere, can I just build it on the big server that I
> have and copy and update my laptop?

You can use NFS or rsync. Make sure the src and obj path names are 
exactly the same on all thes servers. I used to use NFS until a year or 
two ago when I started using different patches on different machines. 
On occasion I've used rsync followed by a NO_CLEAN, WORLDFAST, KERNFAST 
build on the target machine or simply installkernel and installworld.


-- 
Cheers,
Cy Schubert 
FreeBSD UNIX: Web:  http://www.FreeBSD.org

The need of the many outweighs the greed of the few.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Sharing compiled builds between multiple 12-CURRENT boxes.

[Dhananjay Balan]

Hi,

I run 12-CURRENT on few machines, some more powerful that other (all
of them x86_64, march varies). 

Is there is a way to avoid building CURRENT on all machines? Rather
than building everywhere, can I just build it on the big server that I
have and copy and update my laptop?

-
Dhananjay Balan



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"