Re: [gentoo-user] Binary package server questions
On Tuesday 21 Feb 2017 21:26:20 Walter Dnes wrote: > On Tue, Feb 21, 2017 at 10:19:16PM +, Mick wrote > > > Perhaps I do not understand ... why should the chrooted system need > > to use different flags? > > See > https://gcc.gnu.org/onlinedocs/gcc-4.9.4/gcc/i386-and-x86-64-Options.html#i > 386-and-x86-64-Options > > The desktop is "-march=ivybridge" and the netbook is "-march=bonnell". > Neither of them can run the other's "-march=native" code. "ivybridge" > does not have MOVBE, while "bonnell" does not have SSE4.1, SSE4.2, > POPCNT, AVX, AES, PCLMUL, FSGSBASE, RDRND and F16C. gcc will gladly > build for whatever Intel cpu you tell it to. I can build "bonnell" code > on the "ivybridge". But I can't run it on the "ivybridge" machine. > > As a compromise, I suppose I could declare the chroot "-march=core2". > "core2" is "bonnell" minus MOVBE, so both the netbook and the desktop > could run that code, with the netbook getting some, but not all, of the > possible optimization. > > I have a "hot backup" to my desktop, which has a "silvermont" cpu. > That's a newer Atom cpu, and it can run "bonnell" code, no problem. But > the "ivybridge" machine is not that old, and I prefer to keep my > machines until they start dying. I probably misunderstood your intent. I thought you have 3 PCs and want to build binary packages for the oldest/slowest PC only, within a chroot on the fastest PC. The 2 faster PCs build their own packages and do not need a chroot build environment. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Binary package server questions
On Tue, Feb 21, 2017 at 10:19:16PM +, Mick wrote > Perhaps I do not understand ... why should the chrooted system need > to use different flags? See https://gcc.gnu.org/onlinedocs/gcc-4.9.4/gcc/i386-and-x86-64-Options.html#i386-and-x86-64-Options The desktop is "-march=ivybridge" and the netbook is "-march=bonnell". Neither of them can run the other's "-march=native" code. "ivybridge" does not have MOVBE, while "bonnell" does not have SSE4.1, SSE4.2, POPCNT, AVX, AES, PCLMUL, FSGSBASE, RDRND and F16C. gcc will gladly build for whatever Intel cpu you tell it to. I can build "bonnell" code on the "ivybridge". But I can't run it on the "ivybridge" machine. As a compromise, I suppose I could declare the chroot "-march=core2". "core2" is "bonnell" minus MOVBE, so both the netbook and the desktop could run that code, with the netbook getting some, but not all, of the possible optimization. I have a "hot backup" to my desktop, which has a "silvermont" cpu. That's a newer Atom cpu, and it can run "bonnell" code, no problem. But the "ivybridge" machine is not that old, and I prefer to keep my machines until they start dying. -- Walter Dnes I don't run "desktop environments"; I run useful applications
Re: [gentoo-user] Binary package server questions
On Tuesday 21 Feb 2017 16:31:31 Walter Dnes wrote: > To repeat, the basic question I'm asking is how do I set up a "dual > mode" in the chroot so that... > - when the chroot is updating *ITSELF* it builds stuff "-march=native" > and with its own CPU_FLAGS_X86, etc > - when the chroot is building for the Atom, it uses "-march=bonnell" and > the Atom's CPU_FLAGS_X86 and stashes binaries in /usr/portage/packages Perhaps I do not understand ... why should the chrooted system need to use different flags? Does it have a purpose other than building binpkg for the Atom? > Right now, I think the easiest approach is to go with 2 versions of > make.conf, and a wrapper script that copies in the appropriate one > before launching "emerge". Well, yes if the the chroot is used to other things and it is not meant to merely duplicate a build environment for the Atom, but on more powerful hardware. I use a chroot on a modern 64bit PC with 8 cores and 16GB RAM, to build 32bit binaries for a really old 32bit Pentium 4 PC. The CFLAGS in the chroot are: CFLAGS="-march=prescott -O2 -pipe -fomit-frame-pointer" while the host OS is nothing of the sort. This set up has served me nicely without an NFS, because the host OS is not always on when the Pentium 4 PC is in use. I use rsync/cp for moving the built binaries over to the old PC before I install them there. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Binary package server questions
On Tue, 21 Feb 2017 16:31:31 -0500, Walter Dnes wrote: > To repeat, the basic question I'm asking is how do I set up a "dual > mode" in the chroot so that... > - when the chroot is updating *ITSELF* it builds stuff "-march=native" > and with its own CPU_FLAGS_X86, etc > - when the chroot is building for the Atom, it uses "-march=bonnell" and > the Atom's CPU_FLAGS_X86 and stashes binaries in /usr/portage/packages > > Right now, I think the easiest approach is to go with 2 versions of > make.conf, and a wrapper script that copies in the appropriate one > before launching "emerge". So you want to build packages for two different architectures? In that case, you should set up two chroots, or containers. -- Neil Bothwick Whats the difference between a magician and a brothel? One has a cunning array of stunts, pgpQSkrf7ahrS.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Binary package server questions
On Tue, Feb 21, 2017 at 06:02:24AM +, Mick wrote > You'll need to run in 32bit mode when chrooting of course: > > linux32 chroot /mnt/Atom_Build_env /bin/bash > source /etc/profile > export PS1="(Atom_Build) $PS1" I'm already doing something similar with a 32-bit CentOS 6 chroot on a 64-bit no-multilib Gentoo host. No need to reboot. I previously used a QEMU VM. It has disadvantages... - you need dedicated extra space on the disk image for a safety margin - the VM needs its own 5 gigabytes of swap space - the VM has overhead. Building Pale Moon from source took 35 minutes in the VM. In the chroot, it gets done in 22 or 23 minutes. To repeat, the basic question I'm asking is how do I set up a "dual mode" in the chroot so that... - when the chroot is updating *ITSELF* it builds stuff "-march=native" and with its own CPU_FLAGS_X86, etc - when the chroot is building for the Atom, it uses "-march=bonnell" and the Atom's CPU_FLAGS_X86 and stashes binaries in /usr/portage/packages Right now, I think the easiest approach is to go with 2 versions of make.conf, and a wrapper script that copies in the appropriate one before launching "emerge". -- Walter Dnes I don't run "desktop environments"; I run useful applications
Re: [gentoo-user] Binary package server questions
On Tue, 21 Feb 2017 02:38:21 -0500, Walter Dnes wrote: > > Oh, and you don't need a package server, just export PKGDIR via NFS > > and mount it on the netbook. > > I see nfs as being more complex with kernel settings required for > client and server, not to mention config files all over the place. > Gentoo has python as part of the system. To fire up a very simple > binary package webserver from a commandline (xterm/whatever)... That's a fair point. I use NFS because PKGDIR was on a fileserver that was always on, the build host may not have been on at the time. Both approaches have their merits, but an NFS client needs no extra config files (and the server only has one) just a line in fstab. -- Neil Bothwick Macro: (n.) a series of keystrokes used to simulate a missing but essential command. pgpKmsBZZLATr.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Binary package server questions
On Tue, 21 Feb 2017 06:02:24 +, Mick wrote: > On Tuesday 21 Feb 2017 00:22:51 Neil Bothwick wrote: > > If the chroot is identical to your netbooks's install in terms of > > *FLAGS, USE, @world etc, then yes. I used to do it this way when I > > had an Atom netbook. I even build for a low memory 486 system in the > > same way. > > You'll need to run in 32bit mode when chrooting of course: > > linux32 chroot /mnt/Atom_Build_env /bin/bash > source /etc/profile > export PS1="(Atom_Build) $PS1" I'm pretty sure I didn't do this. CHOST was set to 32 bit so I got 32 bit binaries, but full use of the 64 bits for compiling. > > Oh, and you don't need a package server, just export PKGDIR via NFS > > and mount it on the netbook. > > Or, if you can't be bothered with the extra work to set up NFS, copy > the contents of the PKGDIR from the chroot'ed system to the Atom after > you finished building all the chroot'ed binary packages, then emerge > world in the Atom. Setting up the NFS share is a one off task, copying PKGDIR would have to be done every time. Also, the Atom netbooks usually had very limited storage, and PKGDIR can get very big. -- Neil Bothwick The considered application of terror is also a form of communication. pgpVbZ0tzpLXe.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Binary package server questions
On Tue, Feb 21, 2017 at 12:22:51AM +, Neil Bothwick wrote > If the chroot is identical to your netbooks's install in terms of > *FLAGS, USE, @world etc, then yes. I used to do it this way when I had an > Atom netbook. I even build for a low memory 486 system in the same way. Unfortunately, the cpus are different enough that CFLAGS, CXXFLAGS, and CPU_FLAGS_X86 are different. See the web page... https://gcc.gnu.org/onlinedocs/gcc-4.9.4/gcc/i386-and-x86-64-Options.html#i386-and-x86-64-Options I actually use "-march=native" in make.conf, but this translates as... * The netbook cpu equals "-march=bonnell" (first-generation Atom). * My current desktop equals "-march=ivybridge" * My "hot backup" machine equals "-march=silvermont" The natively-compiled code on the netbook will not run on my desktop because the Ivybridge cpu doesn't support MOVBE instructions. The netbook has 2 gigs of ram. I estimate 12 hours if I * changed CFLAGS to "-march=core2" and adjusted CPU_FLAGS_X86 * and ran "emerge -e @world" on the netbook. That would produce "lowest common denominator" code that would run on both the netbook and my desktop. Then I could rsync the contents of the netbook into what would become a chroot directory on the desktop. After that, I'd have to change to "-march=native", adjust CPU_FLAGS_X86, and then "emerge -e @world" on both the netbook and the desktop's chroot. A better option would be to * rsync the contents of the netbook to the Silvermont. It's a newer Atom-family cpu, and can handle MOVBE instructions. * change CFLAGS to "-march=silvermont -mno-movbe" and "emerge -e @world" on the Silvermont. It has a newer, more powerful cpu than netbook, and also 8 gigs of ram, versus the netbook's 2 gigs. A full rebuild won't take anywhere near as long. * rsync the contents of the Silvermont's chroot directory to the Ivybridge desktop. > Oh, and you don't need a package server, just export PKGDIR via NFS > and mount it on the netbook. I see nfs as being more complex with kernel settings required for client and server, not to mention config files all over the place. Gentoo has python as part of the system. To fire up a very simple binary package webserver from a commandline (xterm/whatever)... In python 2.x cd /usr/portage/packages python -m SimpleHTTPServer In python 3.x cd /usr/portage/packages python3 -m http.server ...where "" is the desired port number to listen on. In both cases the default port is 8000 if not specified. Note that only root can open privileged ports in the range 0..1023. -- Walter Dnes I don't run "desktop environments"; I run useful applications
Re: [gentoo-user] Binary package server questions
On Tuesday 21 Feb 2017 00:22:51 Neil Bothwick wrote: > On Mon, 20 Feb 2017 18:34:47 -0500, Walter Dnes wrote: > > Reading https://wiki.gentoo.org/wiki/Binary_package_guide still leaves > > > > me uncertain. I have an ancient 32-bit Atom netbook. I've installed > > uclibc-ng Gentoo on it. Building big packages on it is a pain. I can > > do an identical install in a QEMU VM, and distcc into it. But that > > doesn't catch all compiling work. > > > > What I'd like to do is build binaries in a chroot on my desktop, > > > > assuming a 32-bit uclibc-ng chroot on a 64-bit glibc host is possible. > > Because the cpus are different, I would need to use different CFLAGS > > (and CXXFLAGS) variables for when the host updates its own files, versus > > when it builds files for the netbook. > > If the chroot is identical to your netbooks's install in terms of > *FLAGS, USE, @world etc, then yes. I used to do it this way when I had an > Atom netbook. I even build for a low memory 486 system in the same way. You'll need to run in 32bit mode when chrooting of course: linux32 chroot /mnt/Atom_Build_env /bin/bash source /etc/profile export PS1="(Atom_Build) $PS1" > > Finally, is it possible for the client (the netbook) to notify the > > > > host that it needs certain packages built? I plan to run with > > "--getbinpkgonly" on the netbook. > > You don't need to if the systems are the same. Set both systems to use > the same $PKGDIR, set FEATURES=buildpkg in the chroot and do a world > update. Then do the same update on the netbook but with -K. > > I used a script to control this that basically synced world and most > of /etc/portage before entering the chroot, although I later switched to > using containers as they make life so much easier. > > Oh, and you don't need a package server, just export PKGDIR via NFS and > mount it on the netbook. Or, if you can't be bothered with the extra work to set up NFS, copy the contents of the PKGDIR from the chroot'ed system to the Atom after you finished building all the chroot'ed binary packages, then emerge world in the Atom. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Binary package server questions
On Mon, 20 Feb 2017 18:34:47 -0500, Walter Dnes wrote: > Reading https://wiki.gentoo.org/wiki/Binary_package_guide still leaves > me uncertain. I have an ancient 32-bit Atom netbook. I've installed > uclibc-ng Gentoo on it. Building big packages on it is a pain. I can > do an identical install in a QEMU VM, and distcc into it. But that > doesn't catch all compiling work. > > What I'd like to do is build binaries in a chroot on my desktop, > assuming a 32-bit uclibc-ng chroot on a 64-bit glibc host is possible. > Because the cpus are different, I would need to use different CFLAGS > (and CXXFLAGS) variables for when the host updates its own files, versus > when it builds files for the netbook. If the chroot is identical to your netbooks's install in terms of *FLAGS, USE, @world etc, then yes. I used to do it this way when I had an Atom netbook. I even build for a low memory 486 system in the same way. > Finally, is it possible for the client (the netbook) to notify the > host that it needs certain packages built? I plan to run with > "--getbinpkgonly" on the netbook. You don't need to if the systems are the same. Set both systems to use the same $PKGDIR, set FEATURES=buildpkg in the chroot and do a world update. Then do the same update on the netbook but with -K. I used a script to control this that basically synced world and most of /etc/portage before entering the chroot, although I later switched to using containers as they make life so much easier. Oh, and you don't need a package server, just export PKGDIR via NFS and mount it on the netbook. -- Neil Bothwick Artificial Intelligence usually beats real stupidity. pgpA1Z4f_R6U4.pgp Description: OpenPGP digital signature
[gentoo-user] Binary package server questions
Reading https://wiki.gentoo.org/wiki/Binary_package_guide still leaves me uncertain. I have an ancient 32-bit Atom netbook. I've installed uclibc-ng Gentoo on it. Building big packages on it is a pain. I can do an identical install in a QEMU VM, and distcc into it. But that doesn't catch all compiling work. What I'd like to do is build binaries in a chroot on my desktop, assuming a 32-bit uclibc-ng chroot on a 64-bit glibc host is possible. Because the cpus are different, I would need to use different CFLAGS (and CXXFLAGS) variables for when the host updates its own files, versus when it builds files for the netbook. Finally, is it possible for the client (the netbook) to notify the host that it needs certain packages built? I plan to run with "--getbinpkgonly" on the netbook. -- Walter Dnes I don't run "desktop environments"; I run useful applications