Re: Turning TESTS on by default
On Mon, Jun 09, 2014 at 07:27:43PM -0400, Glen Barber wrote: On Tue, Jun 10, 2014 at 02:15:06AM +0300, Konstantin Belousov wrote: On Sun, Jun 08, 2014 at 04:36:45PM -0400, Glen Barber wrote: On Sun, Jun 08, 2014 at 01:33:31PM -0700, Adrian Chadd wrote: Now, as for how much bloat it adds to the CD images... Enough that it will be turned off for the CDs. I wanted to answer that we should split the question into two, one is whether to enable tests installation by default for buildworld (and I answered no there), another is whether to ship tests with the installation media, both on releases and snapshots. I think that we should do the later, if not on CD, then on DVD and for ftp installs. This would mean that there is one more customization for the release build process, but I do not think that the release build structure requires significant changes. We are dangerously teetering every time an option to add just one more thing to the CD is committed to being unable to fall below the 685MB size (which I understand is what FreeBSDMall needs for the CDs they produce). So, as our src/ and ports/ trees grow, the room we have left is shrinking fast. I think it is not feasible to ship the tests on the CD, but I do agree they can be shipped on the DVD. Julio has submitted a review request for this to make a tests.txz distribution, which can then be excluded for the bootonly.iso and disc1.iso, but kept for the dvd.iso. https://phabric.freebsd.org/D198 I am building a test release build with this change now, but I am aware of one immediate problem that will be encountered - if we default WITH_TESTS=1, the running userland on the bootonly.iso and disc1.iso (used for the installer) will also include the tests. This is because we do not evaluate the value of WITH_TESTS=, but only that it is defined. This is also a problem for the debug.txz with WITH_DEBUG_FILES=1 for the buildworld/buildkernel, but WITHOUT_DEBUG_FILES=1 for the bootonly.iso and disc1.iso creation, because we cannot set both. Some of the later commits to a projects/ branch to work on allowing the debug.txz to be fetched from FTP (or via the netinstall) illustrates the problem and some very hackish workarounds I have tried. https://svn.freebsd.org/base/projects/release-debugdist/ After reviewing Julio's proposed changes, I realize that there is a way to avoid populating /usr/tests when building the on-disc filesystem. I think this will also apply to the WITH_DEBUG_FILES=1 case mentioned. It seems I cannot apply an update to another differential review (or I am just doing it wrong), so have created a new differential here: https://phabric.freebsd.org/D202 Glen pgpJ5_VnwsRSz.pgp Description: PGP signature
Re: Turning TESTS on by default
On Tue, Jun 10, 2014 at 4:54 PM, Glen Barber g...@freebsd.org wrote: After reviewing Julio's proposed changes, I realize that there is a way to avoid populating /usr/tests when building the on-disc filesystem. I think this will also apply to the WITH_DEBUG_FILES=1 case mentioned. It seems I cannot apply an update to another differential review (or I am just doing it wrong), so have created a new differential here: https://phabric.freebsd.org/D202 Thanks for testing and for the feedback. All raised issues should now be resolved: - installworld now works again with read-only src and object trees. - The test suite is shipped as a separate tests.txz file that users can choose to not install. - Release media do not include the test suite in them. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Sun, Jun 08, 2014 at 04:36:45PM -0400, Glen Barber wrote: On Sun, Jun 08, 2014 at 01:33:31PM -0700, Adrian Chadd wrote: Now, as for how much bloat it adds to the CD images... Enough that it will be turned off for the CDs. I wanted to answer that we should split the question into two, one is whether to enable tests installation by default for buildworld (and I answered no there), another is whether to ship tests with the installation media, both on releases and snapshots. I think that we should do the later, if not on CD, then on DVD and for ftp installs. This would mean that there is one more customization for the release build process, but I do not think that the release build structure requires significant changes. pgpdcois8TFWO.pgp Description: PGP signature
Re: Turning TESTS on by default
On Tue, Jun 10, 2014 at 02:15:06AM +0300, Konstantin Belousov wrote: On Sun, Jun 08, 2014 at 04:36:45PM -0400, Glen Barber wrote: On Sun, Jun 08, 2014 at 01:33:31PM -0700, Adrian Chadd wrote: Now, as for how much bloat it adds to the CD images... Enough that it will be turned off for the CDs. I wanted to answer that we should split the question into two, one is whether to enable tests installation by default for buildworld (and I answered no there), another is whether to ship tests with the installation media, both on releases and snapshots. I think that we should do the later, if not on CD, then on DVD and for ftp installs. This would mean that there is one more customization for the release build process, but I do not think that the release build structure requires significant changes. We are dangerously teetering every time an option to add just one more thing to the CD is committed to being unable to fall below the 685MB size (which I understand is what FreeBSDMall needs for the CDs they produce). So, as our src/ and ports/ trees grow, the room we have left is shrinking fast. I think it is not feasible to ship the tests on the CD, but I do agree they can be shipped on the DVD. Julio has submitted a review request for this to make a tests.txz distribution, which can then be excluded for the bootonly.iso and disc1.iso, but kept for the dvd.iso. https://phabric.freebsd.org/D198 I am building a test release build with this change now, but I am aware of one immediate problem that will be encountered - if we default WITH_TESTS=1, the running userland on the bootonly.iso and disc1.iso (used for the installer) will also include the tests. This is because we do not evaluate the value of WITH_TESTS=, but only that it is defined. This is also a problem for the debug.txz with WITH_DEBUG_FILES=1 for the buildworld/buildkernel, but WITHOUT_DEBUG_FILES=1 for the bootonly.iso and disc1.iso creation, because we cannot set both. Some of the later commits to a projects/ branch to work on allowing the debug.txz to be fetched from FTP (or via the netinstall) illustrates the problem and some very hackish workarounds I have tried. https://svn.freebsd.org/base/projects/release-debugdist/ Glen pgpSuN4LmHasa.pgp Description: PGP signature
Re: Turning TESTS on by default
On Fri, Jun 6, 2014 at 11:33 PM, Konstantin Belousov kostik...@gmail.com wrote: It is the same as the debugging kernel. The INVARIANTS, WITNESS, DEBUG and DIAGNOSTIC options are not enabled for the user consumption. No; this is not the same at all. All the options you mention introduce a performance penalty on the system at *run* time and the _only_ way to get rid of such penalty completely is to disable the options at *build* time (sysctl doesn't make the cut because the system would still need to check a boolean value). Because these toggles affect run-time performance, and because they are intended for developers only, they are not the default on release builds. Building the test suite has no impact whatsoever on the system at run-time. The test suite is just a bunch of files in a self-contained location. We ought to ship such files as tests.tgz to give users a very easy way of not installing them though. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Sat, Jun 7, 2014 at 3:53 PM, Konstantin Belousov kostik...@gmail.com wrote: Could somebody point out a popular software system that spills the tests or other developer-only[*] stuff into the production install ? I immediately remember the perl and its modules which have very extensive test suite, but the test suite is not installed. [*] As is, developers of the system, not developers utilizing the product as the base. That's the key you are missing: the FreeBSD test suite, just as the NetBSD test suite, have been designed from the very beginning as a tool for *both* developers and end users. Most test suites out there, if not all, do not have such a goal and thus it is uninteresting, and likely harmful, to install them (if only because it is non-trivial to execute them in a controlled manner). The FreeBSD test suite is intended to let users (especially sysadmins) routinely assess the functionality of their system, especially right after deployment and during system upgrades. This approach is partly intended to cover the fact that the FreeBSD project cannot possibly have[*] a large-enough testing farm to validate most hardware/software combinations run by our users. It is also partly intended to inspire confidence in the system by our users by being very transparent about what works and what may possibly not work. NetBSD encourages you to do test runs (even from cron) in its afterboot(8) manual page. * Mind you: companies such as Microsoft have extremely large test environments and, yet, they continuously fail to catch specific failures in some weird system combination. It is just a fact of the complexity of OSes. The difference between them and us is that they cannot afford to ship their test suite because it's a very important asset of theirs and because of the potential PR implications of users uncovering failures here and there (my guesses though). I recommend reading The Old New Thing blog for insightful details on all the testing that Windows goes thorough, for example. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Sun, 8 Jun 2014, Julio Merino wrote: On Fri, Jun 6, 2014 at 11:33 PM, Konstantin Belousov kostik...@gmail.com wrote: It is the same as the debugging kernel. The INVARIANTS, WITNESS, DEBUG and DIAGNOSTIC options are not enabled for the user consumption. No; this is not the same at all. All the options you mention introduce a performance penalty on the system at *run* time and the _only_ way to get rid of such penalty completely is to disable the options at *build* time (sysctl doesn't make the cut because the system would still need to check a boolean value). Because these toggles affect run-time performance, and because they are intended for developers only, they are not the default on release builds. Building the test suite has no impact whatsoever on the system at run-time. The test suite is just a bunch of files in a self-contained location. We ought to ship such files as tests.tgz to give users a very easy way of not installing them though. How much of an impact does the test suite have in terms of disk space (probably not much) and buildworld time (possibly not much, either)? Users being able to report the results of tests could demystify a lot of bug reports. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Sun, Jun 8, 2014 at 1:50 PM, Warren Block wbl...@wonkity.com wrote: How much of an impact does the test suite have in terms of disk space (probably not much) It clocks at 11MB now on amd64. and buildworld time (possibly not much, either)? Haven't measured. Note, though, that the majority of the existing tests are shell-scripts so they do not take any build time. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On 8 June 2014 11:52, Julio Merino j...@freebsd.org wrote: On Sun, Jun 8, 2014 at 1:50 PM, Warren Block wbl...@wonkity.com wrote: How much of an impact does the test suite have in terms of disk space (probably not much) It clocks at 11MB now on amd64. and buildworld time (possibly not much, either)? Haven't measured. Note, though, that the majority of the existing tests are shell-scripts so they do not take any build time. We can easily just package it up as a tests distfile, rather than including it in base. Now, as for how much bloat it adds to the CD images... -a ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Sun, Jun 08, 2014 at 01:33:31PM -0700, Adrian Chadd wrote: Now, as for how much bloat it adds to the CD images... Enough that it will be turned off for the CDs. Glen pgpSgQfS5INoO.pgp Description: PGP signature
Re: Turning TESTS on by default
On Fri, Jun 6, 2014 at 9:33 PM, Konstantin Belousov kostik...@gmail.com wrote: On Fri, Jun 06, 2014 at 03:14:52PM -0400, Julio Merino wrote: Hello all, TL;DR - I plan to turn the TESTS src.conf knob ON by default on Tuesday once I have been able to perform enough sanity-checks of the build and all of them pass. The impact of this is that the FreeBSD Test Suite (see tests(7)) will be built and installed by default under /usr/tests/ along with the private atf libraries and binaries. There should be no other changes and this should be transparent to everyone. If this happens to break the world in any way, we can trivially roll the change back to fix the fallout. Some details TESTS was never intended to be disabled by default. However, the original patches that were committed months ago related to this feature broke the build and the easiest way out (instead of reverting the commits) was to set the knob to disabled. Unfortunately, it stayed that way even after the discovered problems were fixed. I am confident enough now that we have ironed out all major issues that this might introduce, so it is about time to enable TESTS by default again in HEAD. The benefits of this are that 1) we allow end users (especially consumers of binary releases!) to run the tests out of the box, as it has always been intended; and 2) we will be able to run the official release builds through testing via Jenkins, instead of having to issue custom builds. This is very weird and unprobable. Users cannot care less about running the test suite, they use OS to run applications. IMO enabling installation of the stuff that bloats the system but have no practical use for the system consumer should not be allowed by default. I disagree. Sure, some users won't care. Probably even most users won't care. But some of our users are active supporters of FreeBSD. They evangelize, they file PRs, and they help other users on the forums. Those users will run the tests. Some of them will find bugs that we didn't, because they'll be using different hardware and different configurations. Plus, shipping a test suite exudes an aura of quality (if the tests pass, that is). So I think that we should install the tests, but in a separate installation set, just like games. -Alan It is the same as the debugging kernel. The INVARIANTS, WITNESS, DEBUG and DIAGNOSTIC options are not enabled for the user consumption. There was a similar argument to disable compiling the profiling static libraries, which probably should be reconsidered since lib*_p.a is absolutely useless on current toolchain and hardware. Actual change: https://phabric.freebsd.org/D188 I will update this thread when the change is committed and/or with any updates. Please let me know if I'm missing anything. Cheers ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Sat, Jun 07, 2014 at 12:31:01PM -0600, Alan Somers wrote: On Fri, Jun 6, 2014 at 9:33 PM, Konstantin Belousov kostik...@gmail.com wrote: On Fri, Jun 06, 2014 at 03:14:52PM -0400, Julio Merino wrote: Hello all, TL;DR - I plan to turn the TESTS src.conf knob ON by default on Tuesday once I have been able to perform enough sanity-checks of the build and all of them pass. The impact of this is that the FreeBSD Test Suite (see tests(7)) will be built and installed by default under /usr/tests/ along with the private atf libraries and binaries. There should be no other changes and this should be transparent to everyone. If this happens to break the world in any way, we can trivially roll the change back to fix the fallout. Some details TESTS was never intended to be disabled by default. However, the original patches that were committed months ago related to this feature broke the build and the easiest way out (instead of reverting the commits) was to set the knob to disabled. Unfortunately, it stayed that way even after the discovered problems were fixed. I am confident enough now that we have ironed out all major issues that this might introduce, so it is about time to enable TESTS by default again in HEAD. The benefits of this are that 1) we allow end users (especially consumers of binary releases!) to run the tests out of the box, as it has always been intended; and 2) we will be able to run the official release builds through testing via Jenkins, instead of having to issue custom builds. This is very weird and unprobable. Users cannot care less about running the test suite, they use OS to run applications. IMO enabling installation of the stuff that bloats the system but have no practical use for the system consumer should not be allowed by default. I disagree. Sure, some users won't care. Probably even most users won't care. But some of our users are active supporters of FreeBSD. They evangelize, they file PRs, and they help other users on the forums. Those users will run the tests. Some of them will find bugs that we didn't, because they'll be using different hardware and different configurations. Plus, shipping a test suite exudes an aura of quality (if the tests pass, that is). So I think that we should install the tests, but in a separate installation set, just like games. I would agree with your arguments, and in fact not bother with the proposal at all, if most systems were installed using installer. I am very much confident that significant part of the population is installed or updated using make build/installworld. If somebody cares to run tests, she certainly cares enough to be able to turn the knob on. Otherwise, the tests take sometimes precious space on / or /usr, for nothing. Could somebody point out a popular software system that spills the tests or other developer-only[*] stuff into the production install ? I immediately remember the perl and its modules which have very extensive test suite, but the test suite is not installed. [*] As is, developers of the system, not developers utilizing the product as the base. pgpxZPyer6WX4.pgp Description: PGP signature
Turning TESTS on by default
Hello all, TL;DR - I plan to turn the TESTS src.conf knob ON by default on Tuesday once I have been able to perform enough sanity-checks of the build and all of them pass. The impact of this is that the FreeBSD Test Suite (see tests(7)) will be built and installed by default under /usr/tests/ along with the private atf libraries and binaries. There should be no other changes and this should be transparent to everyone. If this happens to break the world in any way, we can trivially roll the change back to fix the fallout. Some details TESTS was never intended to be disabled by default. However, the original patches that were committed months ago related to this feature broke the build and the easiest way out (instead of reverting the commits) was to set the knob to disabled. Unfortunately, it stayed that way even after the discovered problems were fixed. I am confident enough now that we have ironed out all major issues that this might introduce, so it is about time to enable TESTS by default again in HEAD. The benefits of this are that 1) we allow end users (especially consumers of binary releases!) to run the tests out of the box, as it has always been intended; and 2) we will be able to run the official release builds through testing via Jenkins, instead of having to issue custom builds. Actual change: https://phabric.freebsd.org/D188 I will update this thread when the change is committed and/or with any updates. Please let me know if I'm missing anything. Cheers ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Fri, Jun 06, 2014 at 03:14:52PM -0400, Julio Merino wrote: TL;DR - I plan to turn the TESTS src.conf knob ON by default on Tuesday once I have been able to perform enough sanity-checks of the build and all of them pass. The impact of this is that the FreeBSD Test Suite (see tests(7)) will be built and installed by default under /usr/tests/ along with the private atf libraries and binaries. There should be no other changes and this should be transparent to everyone. If this happens to break the world in any way, we can trivially roll the change back to fix the fallout. Some details TESTS was never intended to be disabled by default. However, the original patches that were committed months ago related to this feature broke the build and the easiest way out (instead of reverting the commits) was to set the knob to disabled. Unfortunately, it stayed that way even after the discovered problems were fixed. I am confident enough now that we have ironed out all major issues that this might introduce, so it is about time to enable TESTS by default again in HEAD. The benefits of this are that 1) we allow end users (especially consumers of binary releases!) to run the tests out of the box, as it has always been intended; and 2) we will be able to run the official release builds through testing via Jenkins, instead of having to issue custom builds. Actual change: https://phabric.freebsd.org/D188 I will update this thread when the change is committed and/or with any updates. Please let me know if I'm missing anything. This is certainly useful, but please fix installworld from a read-only (e.g. NFS) /usr/obj first. I reported this a while ago in http://lists.freebsd.org/pipermail/freebsd-testing/2014-May/000384.html build Kyuafile.auto during buildworld, not installworld. This message includes patches, although they are a bit ugly. -- Jilles Tjoelker ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Fri, Jun 6, 2014 at 3:50 PM, Jilles Tjoelker jil...@stack.nl wrote: This is certainly useful, but please fix installworld from a read-only (e.g. NFS) /usr/obj first. I reported this a while ago in http://lists.freebsd.org/pipermail/freebsd-testing/2014-May/000384.html build Kyuafile.auto during buildworld, not installworld. This message includes patches, although they are a bit ugly. OK; that's a blocker and had forgotten about it. Will fix. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On 7/06/2014 5:14 AM, Julio Merino wrote: Hello all, TL;DR - I plan to turn the TESTS src.conf knob ON by default on Tuesday once I have been able to perform enough sanity-checks of the build and all of them pass. The impact of this is that the FreeBSD Test Suite (see tests(7)) will be built and installed by default under /usr/tests/ along with the private atf libraries and binaries. There should be no other changes and this should be transparent to everyone. If this happens to break the world in any way, we can trivially roll the change back to fix the fallout. Some details TESTS was never intended to be disabled by default. However, the original patches that were committed months ago related to this feature broke the build and the easiest way out (instead of reverting the commits) was to set the knob to disabled. Unfortunately, it stayed that way even after the discovered problems were fixed. I am confident enough now that we have ironed out all major issues that this might introduce, so it is about time to enable TESTS by default again in HEAD. The benefits of this are that 1) we allow end users (especially consumers of binary releases!) to run the tests out of the box, as it has always been intended; and 2) we will be able to run the official release builds through testing via Jenkins, instead of having to issue custom builds. Actual change: https://phabric.freebsd.org/D188 I will update this thread when the change is committed and/or with any updates. Please let me know if I'm missing anything. Cheers ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Julio, Awesome! I look forward to automated review lint checks that test for addition of tests, coverage++ and issue ID's in a changeset :] On a related note, how challenging might it be to generate and expose coverage metrics? This is not to say they are a perfect measure of anything in particular, but ought to provide us a good high level idea of important candidate areas that would benefit from coverage and don't currently. -- Koobs ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Turning TESTS on by default
On Fri, Jun 06, 2014 at 03:14:52PM -0400, Julio Merino wrote: Hello all, TL;DR - I plan to turn the TESTS src.conf knob ON by default on Tuesday once I have been able to perform enough sanity-checks of the build and all of them pass. The impact of this is that the FreeBSD Test Suite (see tests(7)) will be built and installed by default under /usr/tests/ along with the private atf libraries and binaries. There should be no other changes and this should be transparent to everyone. If this happens to break the world in any way, we can trivially roll the change back to fix the fallout. Some details TESTS was never intended to be disabled by default. However, the original patches that were committed months ago related to this feature broke the build and the easiest way out (instead of reverting the commits) was to set the knob to disabled. Unfortunately, it stayed that way even after the discovered problems were fixed. I am confident enough now that we have ironed out all major issues that this might introduce, so it is about time to enable TESTS by default again in HEAD. The benefits of this are that 1) we allow end users (especially consumers of binary releases!) to run the tests out of the box, as it has always been intended; and 2) we will be able to run the official release builds through testing via Jenkins, instead of having to issue custom builds. This is very weird and unprobable. Users cannot care less about running the test suite, they use OS to run applications. IMO enabling installation of the stuff that bloats the system but have no practical use for the system consumer should not be allowed by default. It is the same as the debugging kernel. The INVARIANTS, WITNESS, DEBUG and DIAGNOSTIC options are not enabled for the user consumption. There was a similar argument to disable compiling the profiling static libraries, which probably should be reconsidered since lib*_p.a is absolutely useless on current toolchain and hardware. Actual change: https://phabric.freebsd.org/D188 I will update this thread when the change is committed and/or with any updates. Please let me know if I'm missing anything. Cheers ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org pgpzdDDc5mujo.pgp Description: PGP signature
