[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Carol Willing]

Carol Willing  added the comment:

@iritkatriel Thanks for the follow up. I'm going to close this as I haven't 
seen any issues with -j on MacOS.

--
resolution:  -> not a bug
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Irit Katriel]

Irit Katriel  added the comment:

I closed PR 13186 because the change in it was already committed under 
issue36464. Can this issue be closed now as well or is there anything left to 
look into?

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Cheryl Sabella]

Cheryl Sabella  added the comment:

Even though this mentions Python 2, the pull request is against master.

--
nosy: +cheryl.sabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Cheryl Sabella]

Change by Cheryl Sabella :


--
versions: +Python 3.10 -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[sandy-lcq]

Change by sandy-lcq :


--
keywords: +patch
pull_requests: +13100
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Martin Panter]

Martin Panter  added the comment:

On Linux, Gnu’s “install” command is happy if the target directory already 
exists; it just changes the mode (-m) etc. So the race isn’t a big deal.

This is like the race I described (theoretical at the time) at 
, although that is about $(LIBPC) 
not $(LIBDIR). I suggested adding a separate target (a bit like Paul’s 
“make-directories” suggestion; see my -4.patch in that bug), but it didn’t get 
much support.

A more conservative patch would be a phony target that retains the existing “if 
/ echo / install” logic.

--
nosy: +martin.panter
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Paul Smith]

Paul Smith  added the comment:

I've tried on both MacOS 10.12 and 10.14.  I'm using GNU make 4.2.1 (built 
myself, not the one that comes with Xcode).

I have not tried Python3 builds.  I agree with you that -jN probably has little 
impact on the install step, but the build infrastructure I'm using simply 
provides it by default on both the build and install steps.

All my builds are completely from scratch and scripted; I start with a tarball, 
unpack it, then do an out-of-tree builds that run configure, then run "make" 
then "make install".

Note that these commands are _themselves_ invoked from a makefile the controls 
the build so they are recursive make invocations and the parent make was 
started with -j8 or so, causing the child makes to inherit that parallelism.

I doubt much of that is relevant though.  Here's the issue:

  install: commoninstall ...

  commoninstall: ... altbininstall ... libainstall ...

  altbininstall: $(BUILDPYTHON)

  libainstall: all python-config

So, these two targets altbininstall and libainstall are both prerequisites of 
commoninstall, which is a prerequisite of install.

Since there's no dependency relationship between altbininstall and libainstall, 
make is free to run them both at the same time if you invoke it with 
parallelism enabled.

Both of these targets try to create directories; altbininstall uses:

@for i in $(BINDIR) $(LIBDIR); \
do \
if test ! -d $(DESTDIR)$$i; then \
echo "Creating directory $$i"; \
$(INSTALL) -d -m $(DIRMODE) $(DESTDIR)$$i; \
elsetrue; \
fi; \
done

and libainstall uses:

@for i in $(LIBDIR) $(LIBP) $(LIBPL) $(LIBPC); \
do \
if test ! -d $(DESTDIR)$$i; then \
echo "Creating directory $$i"; \
$(INSTALL) -d -m $(DIRMODE) $(DESTDIR)$$i; \
elsetrue; \
fi; \
done

You can see that both try to create the directory $(LIBDIR).  They do test for 
directory existence first but that's not sufficient when running in parallel: 
if they are running at the same time they might both test at the same time and 
succeed, then one will create the directory first.

Because the rule uses install -d, which (unlike mkdir -p for example) will fail 
if the directory already exists, you have a potential problem here.

If the Python 3 makefile has similar target constructs it will have the same 
issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Carol Willing]

Carol Willing  added the comment:

Thanks @madscientist for filing an issue. It would be helpful to have a bit 
more detail on what specific MacOS version you are building on as well as the 
specific commands that you are executing.

Does this also happen if you run `make clean` before running a build?

--
nosy: +willingc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Ned Deily]

Ned Deily  added the comment:

What version of macOS do you see this with and to what kind of macOS file 
system are you installing?  Also, is the failure seem with Python 3.x installs? 
 For what it's worth, I've never seen this and I do a *lot* of make installs on 
macOS but, while I usually do a make -j3 for the build steps, I rarely use -j 
on install steps as I doubt it has much positive effect there.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36464] Python 2.7 build install fails intermittently with -j on MacOS

[Paul Smith]

New submission from Paul Smith :

Maybe no one cares anymore, but I've discovered that if I run make with -j the 
installation sometimes fails with this error:

install: mkdir /Users/build/.../dist/python/x86_64-darwin/lib: File exists

I believe it's because the targets altbininstall and libainstall as well as 
$(DESTSHARED) ($(BINLIBDEST)/lib-dynload) all contain a for loop which tries to 
create $(LIBDIR).  The makefile uses the install -d command to create 
directories and this command will fail if the directory already exists.

I haven't investigated the full dependency chain but at least two of the above 
targets don't have a relationship that forces make to avoid running them both 
at the same time.

Maybe a better solution would be to create a separate target like 
make-directories or something that creates all the directories and have the 
other targets depend on that one target.  Or something.

As it is, my MacOS builds fail about 1 in 5 times or similar.

Interestingly my Linux builds never fail.  Not sure if install works 
differently on Linux, or the timing is just different there.

--
components: Build
messages: 339078
nosy: madscientist
priority: normal
severity: normal
status: open
title: Python 2.7 build install fails intermittently with -j on MacOS
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com