Re: [bug #40639] GNU Make with profiling information

[Reinier Post]

On Wed Nov 20 22:47:20 2013, eddy.petri...@gmail.com (Eddy Petrișor) wrote:
  On Tue Nov 19 22:29:22 2013, Reinier Post wrote:
  Follow-up Comment #1, bug #40639 (project make):
  
  I have created a cleaned up rebased branch that contains only changes for 
  the
  profiling feature so it can be easier to review.
  
  https://github.com/eddyp/make-profiler/tree/profile-rebase
  
  Please note this branch will be rebased. Use 'git pull --rebase' if you 
  need
  to resync.
  
  Looks very useful!
  
  Can't this functionality be provided by a wrapper $SHELL?
  Sure, it's an extra exec(),
  but it will keep the make code base simpler.
 
 I'm not sure if I understand what you mean. Do you mean wrapping all target 
 invokes in a $SHELL?

Yes; something line /usr/bin/time.

 If that's the case, you don't get the actual/real-time measurement, and that 
 might skew your profiling information.

True.  Hnow how much of a difference does it make?

 I have some build systems that can take easily over 12 hours to do a full 
 build (several thousands of elfs in recursive
 make calls - I know that's bad, I am working on it), so any extra time waste 
 just for the measurement can mean
 significant delays in getting the results.

If it's 15 minutes, they are not the thing you want to optimize first.
The build is probably running overnight anyway.

Can you measure it?

-- 
Reinier Post
TU EIndhoven

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

VMS port

[Pavel Fedin]

 Hello!

 I am restarting work on spawn-patch for Cygwin. Actually, i have the very
first version working, but want to try to do some face-lift and get rid of
some #ifdef's.
 My first question is: is VMS port maintained, or dead long ago ?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: VMS port

[Paul Smith]

On Mon, 2013-11-25 at 13:48 +0400, Pavel Fedin wrote:
  I am restarting work on spawn-patch for Cygwin. Actually, i have the very
 first version working, but want to try to do some face-lift and get rid of
 some #ifdef's.
  My first question is: is VMS port maintained, or dead long ago ?

The VMS port is actively and capably maintained by Hartmut Becker.  The
ChangeLog shows he provided VMS fixes for 4.0 as recently as September.

It's easier if patches are targeted for specific results, so it's best
not to include major refactorings, like removing ports, in the same
patch.


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

RE: VMS port

[Pavel Fedin]

 Hello!

 The VMS port is actively and capably maintained by Hartmut Becker.  The
 ChangeLog shows he provided VMS fixes for 4.0 as recently as September.

 Ah, i see. Thanks for pointing at.

 It's easier if patches are targeted for specific results, so it's best
 not to include major refactorings, like removing ports, in the same
 patch.

 I know, this is what i'm going to do. Actually, i want to fix output-sync for 
spawn()-based flavor. This includes EMX, DOS and potentially Cygwin. Currently 
output-sync option will not work in that ports, because the related fragment:
--- cut ---
  /* Divert child output if output_sync in use.  */
  if (child-output.syncout)
{
  if (child-output.out = 0)
outfd = child-output.out;
  if (child-output.err = 0)
errfd = child-output.err;
}
--- cut ---
 is for some reason under #else branch only (where fork() is used). I think 
that the best way to fix this is to move this fragment out of #ifdef's. But 
above there's one more #ifdef VMS fragment, which also seems to ignore 
child-output settings. I see that VMS version of 'child_execute_job' gets the 
complete 'child' structure, i have checked it but it seems to ignore these new 
members.
 So i've got one more question - may be this fragment should become completely 
generic ? Just with some note that VMS code should also implement handling for 
it.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: VMS port

[Eli Zaretskii]

 From: Pavel Fedin p.fe...@samsung.com
 Date: Mon, 25 Nov 2013 16:44:20 +0400
 Cc: bug-make@gnu.org
 
  Actually, i want to fix output-sync for spawn()-based flavor. This includes 
 EMX, DOS and potentially Cygwin. Currently output-sync option will not work 
 in that ports, because the related fragment:
 --- cut ---
   /* Divert child output if output_sync in use.  */
   if (child-output.syncout)
 {
   if (child-output.out = 0)
 outfd = child-output.out;
   if (child-output.err = 0)
 errfd = child-output.err;
 }
 --- cut ---

Please leave DOS out of this discussion: it doesn't support parallel
execution (and never will), and therefore output-sync is a no-op
there.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #40225] Deterministic output ordering

[Frank Heckenbach]

Paul D. Smith wrote:

 I don't think Frank's suggestion below will work (or maybe I misunderstood
 it).  It's not enough to remember the order in which target recipes were
 started, because in parallel builds the order of starting targets can be
 different.

 [...]

 What we'd need to do is remember the order in which make _would_ have built
 the targets, if running in a serial way.  In order for that to work, we'd need
 to keep an ordered list of the targets we visit as we walk the dependency
 graph, and we'd need to convince ourselves that even with parallelism enabled
 we will walk the graph the same way without skipping any nodes (that is, we
 need to be sure that we won't jump to another branch of the graph because we
 know that this branch is blocked by a target currently being built).
 
 These changes would need to be made above the job layer, in remake.c which is
 where the dependency graph is walked.  Frank is correct that we'd need to
 separate the output dump function out and hoist it up a layer, so that it
 could be done in the proper order.

I see. In this case (I'd been waiting for a definitive statement
from you) I agree it's quite complicated and the better way to
achieve the goal may be to implement tree structured log files as
was suggested before (so external tools can concatenate them in a
deterministic order, e.g. alphabetically, or parse them file by file
in the first place).

As I wrote before, this might not be so hard to implement by using
the output-sync infrastructure, just changing it so the log files
are not created as temporary files, but as actual log files with
names based on the target, and later not dumped and deleted. Of
course, there's some devils in the details, like how to reliably
name those log files, making sure they're unique, whether to combine
or separate stdout and stderr, where to put make's output messages
that don't apply to a particular target etc.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: VMS port

[hb]

On 11/25/2013 01:44 PM, Pavel Fedin wrote:
  I know, this is what i'm going to do. Actually, i want to fix output-sync 
 for spawn()-based flavor. This includes EMX, DOS and potentially Cygwin. 
 Currently output-sync option will not work in that ports, because the related 
 fragment:
 --- cut ---
   /* Divert child output if output_sync in use.  */
   if (child-output.syncout)
 {
   if (child-output.out = 0)
 outfd = child-output.out;
   if (child-output.err = 0)
 errfd = child-output.err;
 }
 --- cut ---
  is for some reason under #else branch only (where fork() is used). I think 
 that the best way to fix this is to move this fragment out of #ifdef's. But 
 above there's one more #ifdef VMS fragment, which also seems to ignore 
 child-output settings. I see that VMS version of 'child_execute_job' gets 
 the complete 'child' structure, i have checked it but it seems to ignore 
 these new members.
  So i've got one more question - may be this fragment should become 
 completely generic ? Just with some note that VMS code should also implement 
 handling for it.

The VMS port does not support parallel execution of recipes. So the
output-sync (although accepted) is ignored.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

mingw-w64 build breaks and warnings

[Stephan T. Lavavej]

Hi,

make's HEAD is currently broken for mingw-w64, and is also emitting several
warnings.

I'm building http://git.savannah.gnu.org/cgit/make.git/commit/?id=f5f5adb
with build_w32.bat gcc.

The attached make-mingw-w64.patch fixes all errors and warnings, and
shouldn't affect other platforms.


Here are the build breaks:

#1:
job.c: In function 'free_child':
job.c:1015:11: warning: passing argument 1 of 'strlen' makes pointer from
integer without a cast [enabled by default]
   OSN (fatal, NILF,
   ^
In file included from c:\mingw\x86_64-w64-mingw32\include\io.h:10:0,
 from c:\mingw\x86_64-w64-mingw32\include\sys\stat.h:14,
 from makeint.h:71,
 from job.c:17:
c:\mingw\x86_64-w64-mingw32\include\string.h:54:18: note: expected 'const
char *' but argument is of type 'DWORD'
   size_t __cdecl strlen(const char *_Str);
  ^

job.c: In function 'new_job':
job.c:1962:13: warning: passing argument 1 of 'strlen' makes pointer from
integer without a cast [enabled by default]
 OSN (fatal, NILF,
 ^
In file included from c:\mingw\x86_64-w64-mingw32\include\io.h:10:0,
 from c:\mingw\x86_64-w64-mingw32\include\sys\stat.h:14,
 from makeint.h:71,
 from job.c:17:
c:\mingw\x86_64-w64-mingw32\include\string.h:54:18: note: expected 'const
char *' but argument is of type 'DWORD'
   size_t __cdecl strlen(const char *_Str);
  ^

main.c: In function 'main':
main.c:1984:11: warning: passing argument 1 of 'strlen' makes pointer from
integer without a cast [enabled by default]
   OSN (fatal, NILF,
   ^
In file included from c:\mingw\x86_64-w64-mingw32\include\io.h:10:0,
 from c:\mingw\x86_64-w64-mingw32\include\sys\stat.h:14,
 from makeint.h:71,
 from main.c:17:
c:\mingw\x86_64-w64-mingw32\include\string.h:54:18: note: expected 'const
char *' but argument is of type 'DWORD'
   size_t __cdecl strlen(const char *_Str);
  ^

This was introduced by
http://git.savannah.gnu.org/cgit/make.git/commit/?id=757849c . Each of these
lines is saying (Error %ld: %s), which corresponds to ONS() (I'm assuming
that's Output Number, then String).

#2:
w32err.c: In function 'map_windows32_error_to_string':
w32err.c:70:3: warning: passing argument 2 of 'fatal' makes integer from
pointer without a cast [enabled by default]
   fatal(NILF, szMessageBuffer);
   ^
In file included from w32err.c:19:0:
../../makeint.h:433:6: note: expected 'size_t' but argument is of type 'char
*'
 void fatal (const gmk_floc *flocp, size_t length, const char *fmt, ...)
  ^
w32err.c:70:3: error: too few arguments to function 'fatal'
   fatal(NILF, szMessageBuffer);
   ^
In file included from w32err.c:19:0:
../../makeint.h:433:6: note: declared here
 void fatal (const gmk_floc *flocp, size_t length, const char *fmt, ...)
  ^

This call wasn't updated by 757849c. Since it has no arguments for the
ellipsis, I believe it needs plain O().


Then there is a severe warning:

#3:
main.c: In function 'prepare_mutex_handle_string':
main.c:796:7: warning: format '%x' expects argument of type 'unsigned int',
but argument 3 has type 'sync_handle_t' [-Wformat=]
   sprintf (sync_mutex, 0x%x, handle);
   ^

job.h typedefs sync_handle_t to intptr_t for WINDOWS32 (otherwise it's int).
When intptr_t is 64-bit, %x is definitely wrong.

Note that this sprintf() is guarded by #ifdef WINDOWS32 above, so we don't
need to worry about other platforms. Using the MS length modifier, %Ix
will work for both 32-bit and 64-bit builds.

Also note that job.c 197 is already using this length modifier in sprintf
(pidstring, %Id, pid);.


Then there are innocuous warnings:

#4:
function.c: In function 'func_shell_base':
function.c:1625:7: warning: variable 'errfd' set but not used
[-Wunused-but-set-variable]
   int errfd;
   ^

#5:
getopt.c: In function '_getopt_internal':
getopt.c:679:8: warning: suggest explicit braces to avoid ambiguous 'else'
[-Wparentheses]
 if (opterr)
^

#6:
job.c: In function 'reap_children':
job.c:666:9: warning: label 'remote_status_lose' defined but not used
[-Wunused-label]
 remote_status_lose:
 ^

#7:
job.c: In function 'construct_command_argv_internal':
job.c:2667:9: warning: variable 'end' set but not used
[-Wunused-but-set-variable]
   char *end;
 ^

These were all easy to fix.

Thanks,
STL


make-mingw-w64.patch
Description: Binary data
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Some more refactor suggestions

[Pavel Fedin]

 Hello! I would like to suggest another code refactor, which could make the
code easier to maintain.
 This is a result of my study of spawn() vs fork() differences and ways to
implement runtime switching between spawn() and fork() in a simple way. The
idea is to unify child_execute_job() function.
 The unification can be done in the following way:
1. The function will always return child's PID.
2. On UNIX the function will call fork() in order to create a child by
itself.
3. job_fds[] and job_rfd can be handled in spawn() way in both cases.
Currently the differences are:
   a) fork() version calls fork(), then closes descriptors in the child
explicitly.
   b) spawn() version sets FD_CLOEXEC flags for these fd's, then calls
spawn(). This effectively means that these descriptors are not inherited by
the child. After spawn()ing FD_CLOEXEC flag is reset.
   I suggest that (b) approach can also be used on UNIX without bad effects.
We could FD_CLONEXEC for relevand fds, then call child_execute_job() which
does fork(). The effect should be the same as closing fd's in the child
manually. I suggest that parent process can even skip resetting this flag
since Make AFAIK never replaces itself with some other process which needs
to inherit all internal file descriptors. So, the code can be further
simplified.
 Any thoughts / pros / contras ?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia



___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make