Re: possible solution for -Otarget recurse
Hi Paul. On 05/05/2013 08:19 AM, Paul Smith wrote: > On Sun, 2013-05-05 at 00:44 +0200, Stefano Lattarini wrote: >> The test 'features/output-sync' now fails for me: >> >> Test timed out after 6 seconds >> Error running /storage/home/stefano/src/gnu/make/tests/../make \ >> (expected 0; got 14): /storage/home/stefano/src/gnu/make/tests/../make \ >> -f work/features/output-sync.mk.1 -j --output-sync=target >> Caught signal 14! >> FAILED (4/6 passed) >> >> Can you reproduce the failure? If not, let me know which information you >> need to debug this, and I'll try to provide them. > > It didn't fail for me. However, it's possible that the 6 second timeout > is just a little too short for reliability. > Sorry to add this only now, but I realized the failure is only reproducible if I run the testsuite with "make -j", as in "make -j8 check; and even in that case, the failure is racy. With a bare "make check", things work for me as well. On the other hand, increasing the parallelism even more, other tests start to fail as well: -*-*-*- $ make check -j16 Making check in glob make[1]: Entering directory `/storage/home/stefano/src/gnu/make/glob' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/glob' Making check in config make[1]: Entering directory `/storage/home/stefano/src/gnu/make/config' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/config' Making check in po make[1]: Entering directory `/storage/home/stefano/src/gnu/make/po' make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/po' Making check in doc make[1]: Entering directory `/storage/home/stefano/src/gnu/make/doc' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/doc' make[1]: Entering directory `/storage/home/stefano/src/gnu/make' make loadavg make[2]: Entering directory `/storage/home/stefano/src/gnu/make' make[2]: Leaving directory `/storage/home/stefano/src/gnu/make' make check-local make[2]: Entering directory `/storage/home/stefano/src/gnu/make' cd tests && perl ./run_make_tests.pl -srcdir /storage/home/stefano/src/gnu/make -make ../make The system uptime program believes the load average to be: uptime 11:01:21 up 47 min, 4 users, load average: 0.03, 0.19, 0.21 The GNU load average checking code thinks: ./loadavg 1-minute: 0.03 5-minute: 0.19 15-minute: 0.21 -- Running tests for GNU make on Linux laptop 3.2.0-4-amd64 x86_64 GNU Make 3.82.90 -- Clearing work... Finding tests... features/archives ... ok (8 passed) features/comments ... ok (1 passed) features/conditionals ... ok (4 passed) features/default_names .. ok (3 passed) features/double_colon ... *** Test died (features/double_colon): test_driver.pl: 468 -> scripts/features/double_colon: 77: Couldn't touch foo: Is a directory FAILED (4/4 passed) features/echoing ok (4 passed) features/errors . *** Test died (features/errors): test_driver.pl: 468 -> scripts/features/errors: 41: Couldn't touch foo: Is a directory FAILED (0/0 passed) features/escape . FAILED (7/8 passed) features/export . ok (12 passed) features/include Error running /storage/home/stefano/src/gnu/make/tests/../make (expected 0; got 512): /storage/home/stefano/src/gnu/make/tests/../make -f work/features/include.mk.4 Error running /storage/home/stefano/src/gnu/make/tests/../make (expected 0; got 512): /storage/home/stefano/src/gnu/make/tests/../make -f work/features/include.mk.5 FAILED (5/10 passed) features/load ... ok (6 passed) features/loadapi ok (3 passed) features/mult_rules . ok (2 passed) features/mult_targets ... ok (2 passed) features/order_only . FAILED (6/10 passed) features/output-sync ok (6 passed) features/override ... ok (4 passed) features/parallelism ok (10 passed) features/patspecific_vars ... ok (10 passed) features/patternrules ... ok (10 passed) features/quoting ok (1 passed) features/
load directive - module parameters
One thing that's worth thinking about in advance about the load directive is whether there needs to be a way to pass parameters into modules. In my case I need for the module to be able to know whether it's allowed to generate debug output or not. I can think of many other areas in which I might want to connect to an external resource like a web server or database or whatever and would need to provide the module with information. load mymodule.so debug load mymodule.so debug=1 load mymodule.so 1 Obvously this wouldn't work because load allows you do do this: load fred.so bob.so alice.so An unsympathetic answer would be: DEBUG:=1 load mymodule.so But imagine telling a programmer in any other language that this was how to instantiate a class or call a function? Other possibilities: load mymodule.so[1] fred.so[0] Filenames can actually have [] characters in them so it's not very desirable. If the answer is target specific variables then I give up but I will avoid saying why because it will only start a debate. :-) I also notice that there is a setup() but no teardown() or equivalent. Any module which might have some state will need such a hook so that it can deallocate resources cleanly. Cheers, Tim -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
On Sun, 2013-05-05 at 11:11 +0200, Stefano Lattarini wrote: > Sorry to add this only now, but I realized the failure is only > reproducible if I run the testsuite with "make -j", as in "make -j8 > check; and even in that case, the failure is racy. With a bare "make > check", things work for me as well. On the other hand, increasing the > parallelism even more, other tests start to fail as well: The test suite definitely cannot be run in parallel. However this should not happen (and does not happen in my environment when I run the commands above) because the test harness cleans out the environment, which will remove any of the MAKEFLAGS or MFLAGS variables that might tell the make to run in parallel when it's not expected. Can you examine your shell configuration files etc. to see if they're setting MAKEFLAGS or MFLAGS? Although if that's true then the tests should fail all the time. Can you verify that there don't seem to be any leftover test files in the tests directory? Sometimes if something doesn't get cleaned up correctly that can cause future builds to fail. However if that were the case then "make check" without -j would fail as well. I don't have an explanation for this. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
> From: Paul Smith > Cc: bug-make@gnu.org > Date: Sat, 04 May 2013 17:51:05 -0400 > > Eli, I did some cleanup in job.c to try to make things simpler and > reduce duplication. I tried to be careful but it's quite possible I did > something to disrupt the Windows version again. It's up to you if you > want to fix any problems now or wait until I solve the above two issues > and look at it all at the same time. There's nothing to fix, as things still seem to work. Thanks! However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. The way they were together before is how programmers are used to see this stuff on Windows; splitting them will not something people will expect. I'm sure you can see this in quite a few packages out there. Here's a random example from GMP's gmp.h, slightly edited for clarity: #define __GMP_DECLSPEC_EXPORT __declspec(__dllexport__) #define __GMP_DECLSPEC_IMPORT __declspec(__dllimport__) #if __GMP_LIBGMP_DLL #if __GMP_WITHIN_GMP /* compiling to go into a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_EXPORT #else /* compiling to go into an application which will link to a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_IMPORT #endif #else /* all other cases */ #define __GMP_DECLSPEC #endif If you didn't like the symbol "MAIN", then we can use any other symbol, like BUILDING_GMAKE or whatever. But having this split in two is not something I'd recommend. I think it's bad for maintenance, if nothing else. > There will be more disruption I think. Looking forward to it ;-) > One other thing: I changed the pump function to read from a FD but write > to a FILE*, because all our other uses of stdout/stderr use FILE* and > it's not wise to mix them. It works fine. While I was in there I > noticed the handling of the text/binary mode. I wonder if this is not > quite right. It seems to me that since we're going to be writing to > stdout/stderr anyway, if we're going to set the mode at all we should be > setting the mode on the temporary file to match the mode on > stdout/stderr, before we write to it, rather than setting the mode on > stdout/stderr to match the temporary file. > > What do you think? Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. That wouldn't be right. The issue here is that Make has no idea what mode will be used by its children. We redirect the children's standard streams to a file, but we cannot force them to use this or that mode (nor should we, because that is up to the child program). Since we have no idea, and we must copy to our stdout/stderr everything the child wrote, we must assume the worst. And the worst is that the child did use binary mode, and as result wrote there some bytes that cannot be read in text mode without corrupting child's output. Examples include the ^Z byte, which stops text-mode reads and writes, and lone CR characters that get stripped by text-mode reads to disappear without a trace. So we use binary mode to read from the temporary file, because that's the only way to guarantee that everything the child wrote will be read verbatim. We then need to write that stuff to our output in the same mode, to produce identical output, as if the intermediate temporary file never happened. Does this make sense? If you want a practical example to illustrate this conundrum, just ask. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: > However, I wonder what was the reason for splitting the definition of > GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. I wanted to separate that in-make variant out so that users would never see it or have the possibility of accidentally using it, so I moved it into our internal headers which are never installed anywhere outside our source tree and would not be included in a GNU make binary package. This is slightly more potential maintenance on our part internally but is much safer for the user which is a tradeoff I'll almost always choose. However, if you really want it back the way it was please do choose a more unique name than "MAIN". Something prefixed with "GMK_" at least. > > One other thing: I changed the pump function to read from a FD but write > > to a FILE*, because all our other uses of stdout/stderr use FILE* and > > it's not wise to mix them. It works fine. While I was in there I > > noticed the handling of the text/binary mode. I wonder if this is not > > quite right. It seems to me that since we're going to be writing to > > stdout/stderr anyway, if we're going to set the mode at all we should be > > setting the mode on the temporary file to match the mode on > > stdout/stderr, before we write to it, rather than setting the mode on > > stdout/stderr to match the temporary file. > > > > What do you think? > > Make never changes the I/O mode of its standard streams. And it > shouldn't, since everything Make itself writes or reads is always > plain text. Since the standard streams always start in text mode, > your suggestion boils down to make input from the temporary file be > always in text mode. Well, I assumed that something that invoked make could set the mode and then make could inherit that mode. I don't know if that's how it works or not in Windows. And of course I doubt anyone does that. I understand your point. I just wonder if this difference might end up being visible to the user in some way. But, we'll leave it as-is. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
> From: Paul Smith
> Cc: bug-make@gnu.org
> Date: Sun, 05 May 2013 12:56:48 -0400
>
> On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote:
> > However, I wonder what was the reason for splitting the definition of
> > GMK_EXPORT in two, and putting each part in a different file.
>
> Well, because the gnumake.h file is intended to be installed into the
> user's /usr/include or similar, and included in a binary package build
> of make such as RPM or DEB or whatever, and be included by the user's
> code, and when it's included there it should NEVER (IIUC) be using the
> in-make variant.
That's true, but it's common to have the same header be included when
building the application or library which provides the exported
services.
> However, if you really want it back the way it was please do choose a
> more unique name than "MAIN". Something prefixed with "GMK_" at least.
OK, will do.
> > Make never changes the I/O mode of its standard streams. And it
> > shouldn't, since everything Make itself writes or reads is always
> > plain text. Since the standard streams always start in text mode,
> > your suggestion boils down to make input from the temporary file be
> > always in text mode.
>
> Well, I assumed that something that invoked make could set the mode and
> then make could inherit that mode. I don't know if that's how it works
> or not in Windows. And of course I doubt anyone does that.
The text/binary mode cannot be inherited the way file descriptors are.
That mode is entirely private to each application, and the startup
code provided by MS (to which MinGW is compatible) unconditionally
initializes it to text mode. Each application decides on its own how
it should treat its standard streams; most leave them at the default
text mode, but some switch to binary. Examples of the latter are
ports of 'cat' ("cat foo > bar" should create a file that is identical
to 'foo'), carefully done ports of 'tr' and Sed (otherwise you cannot
edit in or out CR characters), etc.
> I understand your point. I just wonder if this difference might end up
> being visible to the user in some way.
If the users will see any difference, it means (barring bugs in Make)
the child program used binary I/O where it would use text I/O had its
stdout/stderr not been redirected. But this belongs to the broader
class of problems with -O, not unlike the colored screen output that
was discussed here a few days ago.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
On 05/05/2013 05:30 PM, Paul Smith wrote: > On Sun, 2013-05-05 at 11:11 +0200, Stefano Lattarini wrote: >> Sorry to add this only now, but I realized the failure is only >> reproducible if I run the testsuite with "make -j", as in "make -j8 >> check; and even in that case, the failure is racy. With a bare "make >> check", things work for me as well. On the other hand, increasing the >> parallelism even more, other tests start to fail as well: > > The test suite definitely cannot be run in parallel. > I realize that (and it's no issue, considering how fast the make testsuite is); but I often run a "make -j8 check" just after running "./configure" (requires less typing than "make -j8 && make check"), and that's where things broke down. > However this > should not happen (and does not happen in my environment when I run the > commands above) because the test harness cleans out the environment, > which will remove any of the MAKEFLAGS or MFLAGS variables that might > tell the make to run in parallel when it's not expected. > > Can you examine your shell configuration files etc. to see if they're > setting MAKEFLAGS or MFLAGS? > Definitely not. > Although if that's true then the tests should fail all the time. > > Can you verify that there don't seem to be any leftover test files in > the tests directory? Sometimes if something doesn't get cleaned up > correctly that can cause future builds to fail. However if that were > the case then "make check" without -j would fail as well. > > I don't have an explanation for this. > It's no big deal; I can just run "make check" and live happily. Also, I don't have the mental bandwidth to try to debug this ATM, sorry. Let's return to the issue if I get bitten again. Or maybe you'll manage to reproduce the issue somehow. In the meantime, thanks for your help and your attention, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
On Sun, May 5, 2013 at 2:49 PM, Stefano Lattarini wrote: > On 05/05/2013 05:30 PM, Paul Smith wrote: >> On Sun, 2013-05-05 at 11:11 +0200, Stefano Lattarini wrote: >>> Sorry to add this only now, but I realized the failure is only >>> reproducible if I run the testsuite with "make -j", as in "make -j8 >>> check; and even in that case, the failure is racy. With a bare "make >>> check", things work for me as well. On the other hand, increasing the >>> parallelism even more, other tests start to fail as well: >> >> The test suite definitely cannot be run in parallel. >> > I realize that (and it's no issue, considering how fast the make testsuite > is); but I often run a "make -j8 check" just after running "./configure" > (requires less typing than "make -j8 && make check"), and that's where > things broke down. Remember that GNU make has a .NOTPARALLEL special target to suppress parallelism. Maybe a makefile in the test suite would benefit from one of those? -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Output sync completed (?)
Hi all. I've recently pushed changes to solve the last open issues that
I'm aware of with the --output-sync feature:
* If command line printing is not suppressed ("@" is not used) the
command line is attached to the output.
* Extraneous enter/leave lines are not printed any longer.
* I renamed the options to -Oline, -Otarget (not changed) and
-Orecurse.
* I took another stab at documentation; please check the "Parallel
Output" node and see if it is more understandable now. Some
picky details of -Otarget I didn't spell out but hopefully it's
more clear.
Please test the heck out of this with the different modes in your most
difficult build environments, and let me know if you see any anomalies.
I'm also interested to know if the enter/leave stuff is really correct
now; it is in my testing but it sort of happened more by accident than
real planning.
We are right on the cusp of a release candidate for the next GNU make
version. If you have the ability to build and test from Git feel free
to start your testing early.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
