Re: /usr/src/release/release.sh -> ports -> fetch pkg -> Bad system call (core dumped)
On 4/8/2020 9:05 PM, Tomasz CEDRO wrote: > On Sun, Apr 5, 2020 at 7:53 PM Ruslan Garipov wrote: >>> I wrongly assumed that release will simply update this svn repo that I >>> am working on.. but it fetches HEAD.. so I was trying to build >>> 13/HEAD/CURRENT on 12/STABLE/RELEASE that have different ABI thus bad >>> syscall.. and I need CURRENT to build CURRENT, right? :-) >> I believe in order to build the source tree you just need a compatible >> toolchain. So you can build the source tree for 13.0-CURRENT on >> 12.1-RELEASE system. But you need CURRENT to **run** userland with ABI >> from the CURRENT. > > Long story short I need 13-CURRENT to create a 13-CURRENT release with > release(7) because on 12-STABLE/RELEASE chroot with 13-CURRENT does > not work. Of course, because release(7) runs built userland; in your case it runs userland with FreeBSD 13 ABI on FreeBSD 12.x host. > I need to see how it works with `make release` :-) This is what I meant when saying "you can build the source tree for 13.0-CURRENT on 12.1-RELEASE"; sorry if I've confused you. Check ``Building with release.sh on -STABLE'' thread[1] on the forum. May be it can help you. > > >> Once again: for native build `make release` may be quite easy and fast. >> release(7) guarantees "absolutely clean build environment". > > Will try that `make release` as well, also for 13-CURRENT on > 12-STABLE, thank you! Also will compare the time cost with the one > presented below :-) > > > Creating a "release" also has this advantage that I have memstick > images that I can boot from directly, test what I need, even no need > to install :-) > > Here are my build times, for a release for AMD64/12-STABLE, I did not > select DOC nor PORTS to build, only Kernel and Base :-) > > BUILD HOST (~$500): HP PROLIANT DL380G7X (HDD 1TB, 128GB RAM, 24vCPU = > 2 x Intel Xeon X5660 @2.8GHz = 2 PACKAGES x 6 CORES x 2 HW THREADS ) > INTERNET: 600/60MBit > > time make clean buildkernel (16x speedup with SMP): > 2472.027u 291.657s 46:34.54 98.8% 38592+3123k > 195123+3315182io 80167pf+0w > -j122355.140u 282.807s 3:40.12 1198.4% 38509+3121k > 190142+3361605io 71356pf+0w > -j243672.641u 382.254s 2:54.85 2319.0% 39837+3149k > 218567+3361605io 64857pf+0w > -j483722.707u 352.940s 3:07.10 2178.3% 40328+3169k > 214359+3361607io 66526pf+0w > > time make clean buildworld (12.5x speedup with SMP): > 32223.598u 1940.942s 9:28:07.59 100.2% 46920+3430k > 563493+3074126io 234706pf+0w > -j1233644.438u 1818.147s 54:46.68 1078.9% 47324+3449k > 485257+3255853io 236896pf+0w > -j2450728.732u 2556.953s 45:06.66 1968.6% 47556+3455k > 560905+3255862io 229433pf+0w > -j4851129.166u 2454.564s 44:38.12 2000.7% 47630+3458k > 525602+3255854io 210237pf+0w > > release.sh (seems to auto-adjust -j optimization and below 2h looks > really cool): Yes, release(7) sets number of make(1) jobs to the number of available CPUs (hw.ncpu) for buildworld target, and half of that number for buildkernel one. "2h looks really cool" because release(7) may build chrooted build environment, where it then may build port(s) and builds the target. My Xeons usually spends 1.5 hours on clean release(7) (with 16 make(1) jobs). > 94556.462u 5583.273s 1:43:14.24 1616.6% 50234+3336k > 1457023+16921354io 615421pf+0w > -j12 94477.001u 5577.692s 1:42:59.60 1619.1% 50251+3337k > 1463964+16932952io 617689pf+0w > -j24 94480.274u 5574.117s 1:43:38.89 1608.8% 50213+3336k > 1468685+16931170io 617425pf+0w > -j48 94588.653u 5569.909s 1:43:10.62 1617.9% 50211+3336k > 1460546+16925145io 611229pf+0w > > These times may be even smaller when SSD is used instead of HDD. For me it almost changed nothing, only CPU resources counted. > Also > I wonder how to utilize all RAM to speed up the compilation because I > can see 8..34GB utilization while the rest of 128GB remains unused. By installing additional CPU(s) which will allow you to run more make(1) jobs in parallel. My -j16 builds consume about 16-18 GB (at peaks). > I > saw some ramdisk tricks I need to try that as well :-) > > Best regards :-) > Tomek > [1] https://forums.freebsd.org/threads/building-with-release-sh-on-stable.70889/ ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: /usr/src/release/release.sh -> ports -> fetch pkg -> Bad system call (core dumped)
On 4/5/2020 10:05 PM, Tomasz CEDRO wrote: > On Sun, Apr 5, 2020 at 4:59 PM Ruslan Garipov wrote: >> I'm sorry, I forgot to ask how do you call /usr/src/release/release.sh? >> Do you pass a configuration file to this script? >> >> By default /usr/src/release/release.sh checks out the source tree for >> the CURRENT branch (svn://svn.FreeBSD.org/base/head@rHEAD). In this >> case (if you doesn't change it) chrooted environment definitely will >> fail to run on STABLE and/or RELEASE. >> >> May be it's easy for you to use `make release` directly. > > Case solved! =) > > I wrongly assumed that release will simply update this svn repo that I > am working on.. but it fetches HEAD.. so I was trying to build > 13/HEAD/CURRENT on 12/STABLE/RELEASE that have different ABI thus bad > syscall.. and I need CURRENT to build CURRENT, right? :-) I believe in order to build the source tree you just need a compatible toolchain. So you can build the source tree for 13.0-CURRENT on 12.1-RELEASE system. But you need CURRENT to **run** userland with ABI from the CURRENT. In order to build, for example, 12.1-RELEASE image with release(7) you should assign the SRCBRANCH variable to "base/release/12.1.0@rHEAD", and for 12.0-STABLE: SRCBRANCH="base/stable/12@rHEAD". Either in your configuration file for release(7) or directly in your shell: env SRCBRANCH="base/release/12.1.0@rHEAD" /usr/src/release/release.sh > > I will provide a release.conf, make.conf, src.conf and maybe KERNCONF > if I need something beyond GENERIC. For now I just need to work with > 12-STABLE. Good hint! :-) Sure. Just as a note, by default (when the caller doesn't provide a configuration file to release(7), or the file provided doesn't exists), release(7) builds GENERIC kernel and uses no make.conf and src.conf. Once again: for native build `make release` may be quite easy and fast. release(7) guarantees "absolutely clean build environment". > > Thank you Ruslan!! :-) > > Tomek > > ps/2: Can I provide a patch that will print out what actually is being > fetched by release.sh? That could save some time for first time users > :-) Why not :-) For me reading /usr/src/release/release.sh and default configs for ARM saved me a lot of time. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: /usr/src/release/release.sh -> ports -> fetch pkg -> Bad system call (core dumped)
On 4/5/2020 6:52 PM, Tomasz CEDRO wrote:
> Hello Ruslav and thank you for your feedback! :-)
>
> On Sun, Apr 5, 2020 at 3:07 PM Ruslan Garipov wrote:
>> On 4/4/2020 7:50 PM, Tomasz CEDRO wrote:
>>> 1. Is it a good build / testing environment? Maybe there is a simpler
>>> / better way to cross compile binaries and test on another machine?
>>> Both are using 12.1-RELEASE AMD64 installations so far. All /usr/local
>>> should work both with 12.1-RELEASE and 12-STABLE right?
>> Both machines have the same architecture, therefore it is not a cross
>> build, I believe. For my direct builds (both build and consumer
>> machines are x86-64) I use the procedure described in the handbook
>> (``23.6. Tracking for Multiple Machines''[1]).
>
> I know that method, thank you :-) But I also want to try out the
> binary release, which seems a bit more flexible to have just
> everything in one place, may be used to install on an external machine
> without NFS access, etc :-)
>
> It would be also nice to know the time cost of those two methods, so I
> want to verify :-)
>
>
>>> 3. During /usr/src/release/release.sh I get following error as pasted
>>> below. Does release.sh update /usr/ports just as it snaps from svn or
>>> it will use the /usr/porst that are just there and I need to provide
>>> /usr/ports in a state that will be bindled into a /scratch release?
>> A quote from release(7) man page:
>>
>> release.sh checks out the src/, ports/, and doc/ trees to CHROOTDIR...
>>
>> Therefore, release(7) "ignores" /usr/ports and uses
>> ${CHROOTDIR}/usr/ports. My build machine doesn't have access to the
>> Internet, therefore, I have to define the PORTS_UPDATE_SKIP variable and
>> provide ports tree into the ${CHROOTDIR}/usr/ports before
>> I will call /usr/src/release/release.sh.
>
> Okay, so the build uses totally separate CHROOT in /scratch? I wonder
> if that "Bad System Call" is caused by my Host tools or the CHROOT
> tools?
>
> It looks like the /scratch has its own compiled in tools not a copy
> from my host?
>
> # diff -u /usr/bin/fetch /scratch/usr/bin/fetch
> Binary files /usr/bin/fetch and /scratch/usr/bin/fetch differ
I'm sorry, I forgot to ask how do you call /usr/src/release/release.sh?
Do you pass a configuration file to this script?
By default /usr/src/release/release.sh checks out the source tree for
the CURRENT branch (svn://svn.FreeBSD.org/base/head@rHEAD). In this
case (if you doesn't change it) chrooted environment definitely will
fail to run on STABLE and/or RELEASE.
May be it's easy for you to use `make release` directly.
>
>
>>> ===> docproj-2.0_14 depends on file: /usr/local/sbin/pkg - not found
>>> ===> License BSD2CLAUSE accepted by the user
>>> ===> Fetching all distfiles required by pkg-1.14.2 for building
>>> ===> Extracting for pkg-1.14.2
>>> ===> License BSD2CLAUSE accepted by the user
>>> ===> Fetching all distfiles required by pkg-1.14.2 for building
>>> => SHA256 Checksum mismatch for freebsd-pkg-1.14.2_GH0.tar.gz.
>>> ===> Refetch for 1 more times files: freebsd-pkg-1.14.2_GH0.tar.gz
>>> ===> License BSD2CLAUSE accepted by the user
>>> => freebsd-pkg-1.14.2_GH0.tar.gz doesn't seem to exist in /tmp/distfiles/.
>>> => Attempting to fetch
>>> https://codeload.github.com/freebsd/pkg/tar.gz/1.14.2?dummy=/freebsd-pkg-1.14.2_GH0.tar.gz
>>> freebsd-pkg-1.14.2_GH0.tar.gz Bad system call (core dumped)
>> /usr/src/release/release.sh defines DISTDIR=/tmp/distfiles when it
>> installs the textproc/docproj port or a port from the ${EMBEDDEDPORTS}.
>>
>> As for why fetch(1) fails with bad system call under chrooted
>> environment -- I don't know. I failed on a port fetching only if I
>> hadn't provided all necessary distfiles. You have checksum error
>> message which is causing refetching of the ports-mgmt/pkg port.
>> Therefore, I believe
>> ${CHROOTDIR}/tmp/distfiles/freebsd-pkg-1.14.2_GH0.tar.gz exists on your
>> file system (remained from a previous fetch try?)... May be you should
>> try fetch(1) from the chrooted environment manually, to get any content?
>
> This "Bad System Call" stops me from proceeding. I did place by hand
> the required package in the right place, then it built ok, then I got
> that "Bad System Call" again on install :-(
>
> How can I get the debug symbols in /scratch binaries?
>
> So far I can just show:
> [New LWP 100764]
> Core was generated by `/usr/bin/fetch -Fpr -S 3405355
> htt
Re: /usr/src/release/release.sh -> ports -> fetch pkg -> Bad system call (core dumped)
On 4/5/2020 6:52 PM, Tomasz CEDRO wrote:
> Hello Ruslav and thank you for your feedback! :-)
>
> On Sun, Apr 5, 2020 at 3:07 PM Ruslan Garipov wrote:
>> On 4/4/2020 7:50 PM, Tomasz CEDRO wrote:
>>> 1. Is it a good build / testing environment? Maybe there is a simpler
>>> / better way to cross compile binaries and test on another machine?
>>> Both are using 12.1-RELEASE AMD64 installations so far. All /usr/local
>>> should work both with 12.1-RELEASE and 12-STABLE right?
>> Both machines have the same architecture, therefore it is not a cross
>> build, I believe. For my direct builds (both build and consumer
>> machines are x86-64) I use the procedure described in the handbook
>> (``23.6. Tracking for Multiple Machines''[1]).
>
> I know that method, thank you :-) But I also want to try out the
> binary release, which seems a bit more flexible to have just
> everything in one place, may be used to install on an external machine
> without NFS access, etc :-)
>
> It would be also nice to know the time cost of those two methods, so I
> want to verify :-)
>
>
>>> 3. During /usr/src/release/release.sh I get following error as pasted
>>> below. Does release.sh update /usr/ports just as it snaps from svn or
>>> it will use the /usr/porst that are just there and I need to provide
>>> /usr/ports in a state that will be bindled into a /scratch release?
>> A quote from release(7) man page:
>>
>> release.sh checks out the src/, ports/, and doc/ trees to CHROOTDIR...
>>
>> Therefore, release(7) "ignores" /usr/ports and uses
>> ${CHROOTDIR}/usr/ports. My build machine doesn't have access to the
>> Internet, therefore, I have to define the PORTS_UPDATE_SKIP variable and
>> provide ports tree into the ${CHROOTDIR}/usr/ports before
>> I will call /usr/src/release/release.sh.
>
> Okay, so the build uses totally separate CHROOT in /scratch?
Yes.
> I wonder
> if that "Bad System Call" is caused by my Host tools or the CHROOT
> tools?
By the chrooted environment within the ${CHROOTDIR} I believe.
Otherwise you would get the same error on "host".
>
> It looks like the /scratch has its own compiled in tools not a copy
> from my host?
Yes, release(7) builds and installs clean userland into the ${CHROOTDIR}
(DESTDIR=${CHROOTDIR}), and doesn't copy it from the "host".
>
> # diff -u /usr/bin/fetch /scratch/usr/bin/fetch
> Binary files /usr/bin/fetch and /scratch/usr/bin/fetch differ
>
>
>>> ===> docproj-2.0_14 depends on file: /usr/local/sbin/pkg - not found
>>> ===> License BSD2CLAUSE accepted by the user
>>> ===> Fetching all distfiles required by pkg-1.14.2 for building
>>> ===> Extracting for pkg-1.14.2
>>> ===> License BSD2CLAUSE accepted by the user
>>> ===> Fetching all distfiles required by pkg-1.14.2 for building
>>> => SHA256 Checksum mismatch for freebsd-pkg-1.14.2_GH0.tar.gz.
>>> ===> Refetch for 1 more times files: freebsd-pkg-1.14.2_GH0.tar.gz
>>> ===> License BSD2CLAUSE accepted by the user
>>> => freebsd-pkg-1.14.2_GH0.tar.gz doesn't seem to exist in /tmp/distfiles/.
>>> => Attempting to fetch
>>> https://codeload.github.com/freebsd/pkg/tar.gz/1.14.2?dummy=/freebsd-pkg-1.14.2_GH0.tar.gz
>>> freebsd-pkg-1.14.2_GH0.tar.gz Bad system call (core dumped)
>> /usr/src/release/release.sh defines DISTDIR=/tmp/distfiles when it
>> installs the textproc/docproj port or a port from the ${EMBEDDEDPORTS}.
>>
>> As for why fetch(1) fails with bad system call under chrooted
>> environment -- I don't know. I failed on a port fetching only if I
>> hadn't provided all necessary distfiles. You have checksum error
>> message which is causing refetching of the ports-mgmt/pkg port.
>> Therefore, I believe
>> ${CHROOTDIR}/tmp/distfiles/freebsd-pkg-1.14.2_GH0.tar.gz exists on your
>> file system (remained from a previous fetch try?)... May be you should
>> try fetch(1) from the chrooted environment manually, to get any content?
>
> This "Bad System Call" stops me from proceeding. I did place by hand
> the required package in the right place, then it built ok, then I got
> that "Bad System Call" again on install :-(
>
> How can I get the debug symbols in /scratch binaries?
You need to provide custom src.conf or/and make.conf to release(7).
I, actually, provide make.conf, src.conf and kernel config (not for the
debug symbols; just for tuning the target release).
>
> So far I can just show:
> [New LWP 100764]
> Core was genera
Re: /usr/src/release/release.sh -> ports -> fetch pkg -> Bad system call (core dumped)
On 4/4/2020 7:50 PM, Tomasz CEDRO wrote:
> Hello world :-)
>
> I would like to build a 12-STABLE (/usr/src contains
> svn.freebsd.org/base/stable/12) locally on strong machine (24CPU 127GB
> RAM 12-1-RELEASE AMD64), then test changes on my local machine
> (panasonic toughbook i5 laptop 12.1-RELEASE AMD64). This will be used
> for testing kernel patches and driver development/fixes.
>
> The goal is to have separate zroot/ROOT/stable to select and act as
> the FreeBSD base. So far I have zroot/ROOT/default to use FreeBSD
> 12.1-RELEASE. I would like to switch between those to on boot to have
> one base system stable for working and another base system for testing
> on real environment.
>
> I noticed that simple copy of /boot/kernel does not work on my target
> machine. Thus I am trying to create a whole release, put a separate
> system base, then on boot select different zfs container base to boot
> from. I just love ZFS for that! I may even use snapshots to log and
> rollback changes.
>
> Questions:
>
> 1. Is it a good build / testing environment? Maybe there is a simpler
> / better way to cross compile binaries and test on another machine?
> Both are using 12.1-RELEASE AMD64 installations so far. All /usr/local
> should work both with 12.1-RELEASE and 12-STABLE right?
Both machines have the same architecture, therefore it is not a cross
build, I believe. For my direct builds (both build and consumer
machines are x86-64) I use the procedure described in the handbook
(``23.6. Tracking for Multiple Machines''[1]).
>
> 2. When that works, I would like to cross-compile for ARM in a similar
> manner, then attach pyOCD + GDB to debug ARM target. I guess that
> should work too as above?
>
> 3. During /usr/src/release/release.sh I get following error as pasted
> below. Does release.sh update /usr/ports just as it snaps from svn or
> it will use the /usr/porst that are just there and I need to provide
> /usr/ports in a state that will be bindled into a /scratch release?
A quote from release(7) man page:
release.sh checks out the src/, ports/, and doc/ trees to CHROOTDIR...
Therefore, release(7) "ignores" /usr/ports and uses
${CHROOTDIR}/usr/ports. My build machine doesn't have access to the
Internet, therefore, I have to define the PORTS_UPDATE_SKIP variable and
provide ports tree into the ${CHROOTDIR}/usr/ports before
I will call /usr/src/release/release.sh.
>
> ===> docproj-2.0_14 depends on file: /usr/local/sbin/pkg - not found
> ===> License BSD2CLAUSE accepted by the user
> ===> Fetching all distfiles required by pkg-1.14.2 for building
> ===> Extracting for pkg-1.14.2
> ===> License BSD2CLAUSE accepted by the user
> ===> Fetching all distfiles required by pkg-1.14.2 for building
> => SHA256 Checksum mismatch for freebsd-pkg-1.14.2_GH0.tar.gz.
> ===> Refetch for 1 more times files: freebsd-pkg-1.14.2_GH0.tar.gz
> ===> License BSD2CLAUSE accepted by the user
> => freebsd-pkg-1.14.2_GH0.tar.gz doesn't seem to exist in /tmp/distfiles/.
> => Attempting to fetch
> https://codeload.github.com/freebsd/pkg/tar.gz/1.14.2?dummy=/freebsd-pkg-1.14.2_GH0.tar.gz
> freebsd-pkg-1.14.2_GH0.tar.gz Bad system call (core dumped)
/usr/src/release/release.sh defines DISTDIR=/tmp/distfiles when it
installs the textproc/docproj port or a port from the ${EMBEDDEDPORTS}.
As for why fetch(1) fails with bad system call under chrooted
environment -- I don't know. I failed on a port fetching only if I
hadn't provided all necessary distfiles. You have checksum error
message which is causing refetching of the ports-mgmt/pkg port.
Therefore, I believe
${CHROOTDIR}/tmp/distfiles/freebsd-pkg-1.14.2_GH0.tar.gz exists on your
file system (remained from a previous fetch try?)... May be you should
try fetch(1) from the chrooted environment manually, to get any content?
> => Attempting to fetch
> http://distcache.FreeBSD.org/ports-distfiles/freebsd-pkg-1.14.2_GH0.tar.gz
> freebsd-pkg-1.14.2_GH0.tar.gz Bad system call (core dumped)
> => Couldn't fetch it - please try to retrieve this
> => port manually into /tmp/distfiles/ and try again.
> *** Error code 1
>
> Stop.
>
> Any hints and comments are welcome :-)
> Tomek
>
[1]
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/small-lan.html
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
