Re: Parallel job instance identifiers?

[Christian Hujer]

Hi,

I'm afraid there's a lot to consider.
> That would require starting up a server first, and somehow getting it to
report its port# to us.
> But the server configs must know port#s in advance because they need to
talk to each other.
Bootstrapping from leaf servers up the dependency tree should be possible.
Cyclic dependencies could get in the way.
Cyclic dependencies usually are a bad idea.
But that's off-topic.

> Currently we use the range of 9000-9020. On a build/test machine we don't
worry about conflicts,
> a developer has no business running conflicting software at the same time
as building/testing
> our software. And of course, the baseport 9000 can be moved at will.

Well, well. I don't subscribe to the idea that anyone can dictate what a
developer would run at the same time and what constitutes such software.
IDEs might use ports.
The system itself uses ports.
Other software might use ports.
The developer might want to build/test two version of the software in
parallel at the same time to do comparative analysis and comparative
debugging.
The definition of the port range for ephemeral ports could change.
There's just so much that could go wrong.
In my experience, assuming the availability of ports leads to flaky tests.
Developers can lose a lot of time to get tests running if they first have
to resolve port conflicts.

I get that you want to use job identifiers for this. But the mechanism is
just not made for that.
That the job identifier is a number is only by coincidence.
Job identifiers are identifiers first, and numbers only coincidentally.
If someone decides, for whatever reason, to replace them with UUIDs, so
what?
I'm not saying that using UUIDs would be a good idea in this case, it's
just about making the point about the difference between an identifier and
a number.

But I totally get that you need something that helps you to get unique
numbers.
Have you considered using a simple shell script that counts up and writes
the count into a named pipe, and then having each job read one line from
that named pipe to get "your" number?
This should do the job:

$ mkfifo counter
$ for ((uniqueNumber = 0; uniqueNumber < 1000; uniqueNumber++)) ; do echo
$uniqueNumber >counter ; done &

You can then get a - for that run - unique number using
$ head -n 1 counter

It should be easy to do this from a Makefile.
If you do this directly from the Makefile, this might require
SHELL:=/bin/bash, recursive make, and an exit handler.

Here is how you could use an exit handler from within a Makefile:
https://stackoverflow.com/questions/28597794/how-can-i-clean-up-after-an-error-in-a-makefile/52159940#52159940

I hope that this helps you and provides you with a pathway solution that
allows you even more control without accidental coupling to internal
features of make.
And while I think that using make is great, this solution would even work
if you use something else than make.

Q'aplah!


On Sat, Aug 14, 2021 at 5:57 PM Howard Chu  wrote:

> Christian Hujer wrote:
> > Hi,
> >
> > Have you considered using port 0 instead to tap into the ephemeral port
> range and communicate that port somehow?
>
> That would require starting up a server first, and somehow getting it to
> report its port# to us.
> But the server configs must know port#s in advance because they need to
> talk to each other.
> >
> > From what I understand, you want to use the job id as an offset to a
> base port to choose a port from a range, right? That sounds non-portable to
> me, spelling
> > all sorts of ports conflict trouble.
>
> Currently we use the range of 9000-9020. On a build/test machine we don't
> worry about conflicts,
> a developer has no business running conflicting software at the same time
> as building/testing
> our software. And of course, the baseport 9000 can be moved at will.
> >
> > Christian
> >
> > On Fri, Aug 13, 2021, 18:44 Howard Chu via Bug reports and discussion
> for GNU make mailto:bug-make@gnu.org>> wrote:
> >
> > In my original jobserver implementation, I filled the job pipe with
> > up to 256 bytes, numbers from 0-255. The idea being that then make
> > could distinguish each job with a unique ID number. That idea got
> > thrown away when we decided to raise the limit to PIPEBUF (4096 on
> > traditional Unix, much larger now on Linux).
> >
> > I'm looking for a way to expose a job ID number to the individual
> > jobs, in a consecutive range from 1-[number of jobs]. Not just unique
> > numbers, for that we could just use $$ already. The purpose is to
> > e.g. assign non-overlapping network port numbers when firing off a
> > number of client/server test scripts in parallel.
> >
> > Is there

Re: Parallel job instance identifiers?

[Christian Hujer]

Hi,

Have you considered using port 0 instead to tap into the ephemeral port
range and communicate that port somehow?

>From what I understand, you want to use the job id as an offset to a base
port to choose a port from a range, right? That sounds non-portable to me,
spelling all sorts of ports conflict trouble.

Christian

On Fri, Aug 13, 2021, 18:44 Howard Chu via Bug reports and discussion for
GNU make  wrote:

> In my original jobserver implementation, I filled the job pipe with
> up to 256 bytes, numbers from 0-255. The idea being that then make
> could distinguish each job with a unique ID number. That idea got
> thrown away when we decided to raise the limit to PIPEBUF (4096 on
> traditional Unix, much larger now on Linux).
>
> I'm looking for a way to expose a job ID number to the individual
> jobs, in a consecutive range from 1-[number of jobs]. Not just unique
> numbers, for that we could just use $$ already. The purpose is to
> e.g. assign non-overlapping network port numbers when firing off a
> number of client/server test scripts in parallel.
>
> Is there anything that could do this now, exposed as a Make variable?
> --
>   -- Howard Chu
>   CTO, Symas Corp.   http://www.symas.com
>   Director, Highland Sun http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/
>
>

Suggestion: Modernization of the include path

[Christian Hujer]

Hello everyone,


I suggest that
a) $HOME/.local/include is effectively added to the include_directories, as
if it were inserted in default_include_directories before /usr/gnu/include.
b) Change function src/read.c/eval_makefile() to loop over .INCLUDE_DIRS
instead of include_directories when searching for another Makefile.

Rationale for a):
The $HOME/.local directory tree is a directory tree for user-installed
software established beside /usr and /usr/local.
By supporting $HOME/.local/include as a default include directory, users of
make could share modules by "installing" them into their
$HOME/.local/include directory without needing administrator privileges
(required for installing in /usr/include or /usr/local/include or
/usr/gnu/include), and without appending -I ~/.local/include to each of
their make calls which would be inconvenient.

This feature looks portable:
* The entry $HOME/.local/include can be assumed on all POSIXy systems.
  No harm is done if that directory is part of the include path but doesn't
exist.
  This assumes that the mechanism that scans the include paths for include
files will continue to silently ignore non-existent directories.
* On Windows, there is a corresponding directory, I can create a VM if the
details are needed.
* On Amiga OS, the variable $HOME is optional and usually only set if
something like MuFS or GNU is used in an extended configuration.
  Make could check if $HOME is set and include the entry based on that
condition.

Rationale for b):
INCLUDE_DIRS+=${HOME}/.local/include has no effect and does not print any
error message, which is not friendly towards users of make.
Something should either work, or when it doesn't, it should print an error
message.


Best regards,
-- 
Christian Hujer
CEO/CTO
Nelkinda Software Craft Pvt Ltd
📧 Christian Hujer  | 🐦 @nelkinda
<https://twitter.com/nelkinda> | 🌐 http://nelkinda.com
☏ 🇮🇳 ✆ +91 77 2003 6661 <+917720036661>

[bug #29074] -include target fails to issue Error in 3.81

[Christian Hujer]

Follow-up Comment #5, bug #29074 (project make):

I agree with Hack Lee.

There was a bug before 3.81.

You wouldn't want make to fall over on an
-include $(sources:%.c=%.d)
in the Makefile because of a source error when you're actually running
make clean

The behavior of GNUmake 3.81 is correct.

___

Reply to this item at:

  

___
  Nachricht geschickt von/durch Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make