Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On Tue, Apr 23, 2019 at 09:36:50PM +0100, Ken Moffat via lfs-dev wrote: > A better tool (Python, GPL3) might be 'diffoscope'. > https://diffoscope.org/ > > Git is at https://salsa.debian.org/reproducible-builds/diffoscope > but I'm not sure about any working links to released tarballs. > Hmm, I'm still very iffy with the back-end of a cold, can't read text unless it spells things out in black and white. Try https://diffoscope.org/archive/ -- With a few red lights, a few old bits, we made the place to sweat. No matter what we get out of this, I know, I know we'll never forget. Smoke on the water, a fire in the sky. Smoke, on the water. -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On Tue, Apr 23, 2019 at 08:04:29PM +0100, Ken Moffat via lfs-dev wrote: > On Tue, Apr 23, 2019 at 07:41:24PM +0100, Ken Moffat via lfs-dev wrote: > > > > But related to that I found various posts. One mentioned using a > > seeded random number so that builds were deterministic. Having > > suffered from a lack of entropy last year (on some of my machines > > with SSDs, not enough entropy to run all my BLFS bootscripts - > > solved by adding weak entropy from haveged), and given general > > upstream efforts to harden builds and increase randomness, I think > > that the "can it build itself" tests might be possible, but only in > > a very constrained environment where special measures are taken to > > see randomness. However, the problem of building one package > > To _seed_ randomness > > > repeatably is somewhat simpler than building the base LFS system > > repeatably, and that might be impractical. > > > A better tool (Python, GPL3) might be 'diffoscope'. https://diffoscope.org/ Git is at https://salsa.debian.org/reproducible-builds/diffoscope but I'm not sure about any working links to released tarballs. ĸen -- With a few red lights, a few old bits, we made the place to sweat. No matter what we get out of this, I know, I know we'll never forget. Smoke on the water, a fire in the sky. Smoke, on the water. -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On Tue, Apr 23, 2019 at 07:41:24PM +0100, Ken Moffat via lfs-dev wrote: > > But related to that I found various posts. One mentioned using a > seeded random number so that builds were deterministic. Having > suffered from a lack of entropy last year (on some of my machines > with SSDs, not enough entropy to run all my BLFS bootscripts - > solved by adding weak entropy from haveged), and given general > upstream efforts to harden builds and increase randomness, I think > that the "can it build itself" tests might be possible, but only in > a very constrained environment where special measures are taken to > see randomness. However, the problem of building one package To _seed_ randomness > repeatably is somewhat simpler than building the base LFS system > repeatably, and that might be impractical. > -- With a few red lights, a few old bits, we made the place to sweat. No matter what we get out of this, I know, I know we'll never forget. Smoke on the water, a fire in the sky. Smoke, on the water. -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On Wed, Apr 17, 2019 at 04:50:54AM +, DJ Lucas via lfs-dev wrote: > On 4/12/2019 7:47 AM, Pierre Labastie via lfs-dev wrote: > > On 12/04/2019 12:37, DJ Lucas via lfs-dev wrote: > > > > > Comparison does not work anymore because of some randomization in code > > generation... Maybe, It could be disabled by some switch, though. We may > > look at what is done for comparing the second and third build of gcc at > > the end of a bootstrap. > > No, you can't do direct 1:1 comparisons, but you can still compare them > using other tools and attributes. You can't guarantee that they are > identical (they aren't), but you should be able to glean reasonable > assurance that they are functionally equivalent by maybe comparing the > symbol table for executables, and using a tool like abidiff for libraries > (shot in the dark, I haven't actually verified either is a viable test > case). > > > > > > > I'm sure there are lots of holes in the above being that I woke with > > > it, but thought I'd throw it out there anyway. I think that ticks > > > off all of the boxes identified in the earlier thread without > > > introducing more exotic flags, or even existing hacks we currently > > > use for the toolchain. Thoughts? > > > > It is certainly worth trying, but who will be able to find the time for > > doing this? I'm working steadily towards a complete jhalfs automated > > LFS/BLFS (some kind of reference build), trying not to ask my fellow > > editors to modify their habits, and it takes almost all my free time. > > You are absolutely right, I think Ken was alluding to the same. We need new > blood to go and experiment. Look at the changes Xi has put in, has caught me > depending on old assumptions more than once. Jeremy coming back to assist > with JHALFS in whatever capacity he sees fit. More capable eyes are just > plain helpful. Just putting the ideas out there for now, maybe somebody > finds it worth the time, maybe even somebody outside of the core devs will > find it an interesting experiment...or not. :-/ > > --DJ I'm now coming back to this to summarise the reason why I gave up trying to use farce to compare an initial build against the results of using that to build itself: unexplained variation with more-recent toolchain (this was 10 years, or maybe more, ago and might have been documented on cross-lfs rather than lfs). Since then, the concept of reproducible builds has been created (check that, using the same toolchain and scripts, the distributed binary packages match). It has some of the same issues, and apparently tools have been created to help with some of the issues. Certainly, ASLR (put variables in random places, and therefore use different addresses when accessing them) is a large part of this (see wikipedia). But related to that I found various posts. One mentioned using a seeded random number so that builds were deterministic. Having suffered from a lack of entropy last year (on some of my machines with SSDs, not enough entropy to run all my BLFS bootscripts - solved by adding weak entropy from haveged), and given general upstream efforts to harden builds and increase randomness, I think that the "can it build itself" tests might be possible, but only in a very constrained environment where special measures are taken to see randomness. However, the problem of building one package repeatably is somewhat simpler than building the base LFS system repeatably, and that might be impractical. ĸen -- With a few red lights, a few old bits, we made the place to sweat. No matter what we get out of this, I know, I know we'll never forget. Smoke on the water, a fire in the sky. Smoke, on the water. -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On 4/12/2019 7:47 AM, Pierre Labastie via lfs-dev wrote: On 12/04/2019 12:37, DJ Lucas via lfs-dev wrote: Comparison does not work anymore because of some randomization in code generation... Maybe, It could be disabled by some switch, though. We may look at what is done for comparing the second and third build of gcc at the end of a bootstrap. No, you can't do direct 1:1 comparisons, but you can still compare them using other tools and attributes. You can't guarantee that they are identical (they aren't), but you should be able to glean reasonable assurance that they are functionally equivalent by maybe comparing the symbol table for executables, and using a tool like abidiff for libraries (shot in the dark, I haven't actually verified either is a viable test case). I'm sure there are lots of holes in the above being that I woke with it, but thought I'd throw it out there anyway. I think that ticks off all of the boxes identified in the earlier thread without introducing more exotic flags, or even existing hacks we currently use for the toolchain. Thoughts? It is certainly worth trying, but who will be able to find the time for doing this? I'm working steadily towards a complete jhalfs automated LFS/BLFS (some kind of reference build), trying not to ask my fellow editors to modify their habits, and it takes almost all my free time. You are absolutely right, I think Ken was alluding to the same. We need new blood to go and experiment. Look at the changes Xi has put in, has caught me depending on old assumptions more than once. Jeremy coming back to assist with JHALFS in whatever capacity he sees fit. More capable eyes are just plain helpful. Just putting the ideas out there for now, maybe somebody finds it worth the time, maybe even somebody outside of the core devs will find it an interesting experiment...or not. :-/ --DJ -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On 12/04/2019 20:24, Jeremy Huntwork via lfs-dev wrote: > On Fri, Apr 12, 2019 at 2:13 PM Pierre Labastie via lfs-dev > wrote: >> >> I'm certainly interested. You can make it as a patch set, I have a private >> git >> repo where I can run "git am", which I then synchronize with the svn local >> copy, which can then be checked in svn.linuxfromscratch.org. I do all my >> development on the private git repo, creating and deleting branches as >> needed. >> So much easier than subversion... But of course, it works only for a lonely >> dev. > > Yeah I set it up for myself similarly using a local git repo. It's > been years since I worked with Subversion and yes, I also much prefer > git's workflow. > > I'll see if I can send it somewhere for you. This thread probably > isn't good though. Sorry for hijacking it. Let me know where you'd > like it. > Can you post to alfs-discuss? Pierre -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On Fri, Apr 12, 2019 at 2:13 PM Pierre Labastie via lfs-dev wrote: > > I'm certainly interested. You can make it as a patch set, I have a private git > repo where I can run "git am", which I then synchronize with the svn local > copy, which can then be checked in svn.linuxfromscratch.org. I do all my > development on the private git repo, creating and deleting branches as needed. > So much easier than subversion... But of course, it works only for a lonely > dev. Yeah I set it up for myself similarly using a local git repo. It's been years since I worked with Subversion and yes, I also much prefer git's workflow. I'll see if I can send it somewhere for you. This thread probably isn't good though. Sorry for hijacking it. Let me know where you'd like it. JH -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On 12/04/2019 19:06, Jeremy Huntwork via lfs-dev wrote: > On Fri, Apr 12, 2019 at 8:48 AM Pierre Labastie via lfs-dev > wrote: >> >> It is certainly worth trying, but who will be able to find the time for >> doing this? I'm working steadily towards a complete jhalfs automated >> LFS/BLFS (some kind of reference build), trying not to ask my fellow >> editors to modify their habits, and it takes almost all my free time. > > On a somewhat related note, I was spending some time the other day > poking around with jhalfs. I'm impressed by the work you have all put > into it over the years. I started running it through shellcheck > (https://www.shellcheck.net/) to see what it might have to say. > Interestingly, it exposed at least a couple of subtle bugs in the code > that could be fixed. I don't think they're anything major, but it > might be useful to fix them. > > I hesitated to post my changes/findings anywhere though because once I > started modifying the things shellcheck found, it led me to other > optimizations and then my changes were starting to become significant. > I can share them if you like, but I worry that a gigantic patch of > changes would be a little overwhelming :) > I'm certainly interested. You can make it as a patch set, I have a private git repo where I can run "git am", which I then synchronize with the svn local copy, which can then be checked in svn.linuxfromscratch.org. I do all my development on the private git repo, creating and deleting branches as needed. So much easier than subversion... But of course, it works only for a lonely dev. I'm installing shellcheck... I'm certainly not easy with bash! I wish that a similar tool exist for the XSLT language, because I've learned it by myself, from the W3C doc, and I'm never sure I come to the right/best solution. Pierre -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On Fri, Apr 12, 2019 at 8:48 AM Pierre Labastie via lfs-dev wrote: > > It is certainly worth trying, but who will be able to find the time for > doing this? I'm working steadily towards a complete jhalfs automated > LFS/BLFS (some kind of reference build), trying not to ask my fellow > editors to modify their habits, and it takes almost all my free time. On a somewhat related note, I was spending some time the other day poking around with jhalfs. I'm impressed by the work you have all put into it over the years. I started running it through shellcheck (https://www.shellcheck.net/) to see what it might have to say. Interestingly, it exposed at least a couple of subtle bugs in the code that could be fixed. I don't think they're anything major, but it might be useful to fix them. I hesitated to post my changes/findings anywhere though because once I started modifying the things shellcheck found, it led me to other optimizations and then my changes were starting to become significant. I can share them if you like, but I worry that a gigantic patch of changes would be a little overwhelming :) JH -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] very clean method - was: Re: [RFC] Use GCC -ffile-prefix-map option to simplify instruction of Glibc
On 12/04/2019 12:37, DJ Lucas via lfs-dev wrote: On March 28, 2019 7:00:29 PM CDT, Ken Moffat via lfs-dev wrote: It is disappointing that glibc links to static libgcc. What we really need is a new generation of experimenters to replace Greg and Ryan. I like the sound of the "very clean method", but I'm not sure that it helps, particularly if someone uses an archived old version of tools. But on the other hand, it is possible that I saw actions like that getting mentioned because of problems which resulted, so perhaps anyone doing that will get to keep both the pieces. Ken, thank you for making mention of this. An interesting way to wake, maybe even a tiny bit disturbing, but I've been doing LFS for a really long time. :-) Just a fleeting idea after waking at 4:00 AM, that I've obviously no time to explore, but wanted to get it out there in case somebody would like to explore. Anyway, why not make a real chroot environment? We can still use /tools for the first environment, so it's not that drastic of a change, we just populate it via DESTDIR installs, built for the host with our modified temporary vendor field. We'd chroot into it just like we do now, and do the same (with adjusted triplet) for our real new root up to the point that we have the tools needed on the real new root to proceed with the native build, then exit our tools chroot environment, and chroot into the target. Thanks for mentioning that: I've tried that before becoming involved in jhalfs (and then in blfs). One of the problems I had at the time (around 2010) was that not all packages in chapter 5 are cross-compile friendly, so they needed hacks. It may be better today. But as you say, it eliminates a lot of the tweaks we have to do right now. Off the top of my head, chapter five might grow a bit, but we'll eliminate the separation of libstdc++ and the double build of gcc. Whatever you do, the minimum number of builds is two: one to build a cross-compiler, and one using the cross-compiler to build a native one. But maybe the rebuild of gcc in chapter 6 wouldn't be needed, although a good bootstrapping procedure would require a third one, which compares well with the second one. I'm not sure for libraries (when I was doing that, gcc was written in pure C), but the problem is that building libstdc++ (cross-compiled) needs a cross-compiled glibc, which can itself only be built after building the cross-compiler... So separating libstdc++ might still be necessary. We'd likely have to tighten up the host requirements a bit too. What we gain is real paths throughout the build, a more mainline build method (but also less hacks to the build machinery), a build method that more readily mimics what distros do, and we get to introduce a useful implementation of DESTDIR for a fair number of packages (yes it'd still be repetitive). It also makes the work that went into Greg, and later, Ken's comparison builds much easier by being able to do comparisons easily at any point we choose (per package), and it gives us the opportunity to reinvent the book a bit. For that last point, I'm envisioning introducing tough topics as they are needed, rather than being locked away in an earlier chapter where they might be overlooked. Comparison does not work anymore because of some randomization in code generation... Maybe, It could be disabled by some switch, though. We may look at what is done for comparing the second and third build of gcc at the end of a bootstrap. I'm sure there are lots of holes in the above being that I woke with it, but thought I'd throw it out there anyway. I think that ticks off all of the boxes identified in the earlier thread without introducing more exotic flags, or even existing hacks we currently use for the toolchain. Thoughts? It is certainly worth trying, but who will be able to find the time for doing this? I'm working steadily towards a complete jhalfs automated LFS/BLFS (some kind of reference build), trying not to ask my fellow editors to modify their habits, and it takes almost all my free time. Pierre -- http://lists.linuxfromscratch.org/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
