[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

Thanks for your help with this Xavier.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Roundup Robot]

Roundup Robot added the comment:

New changeset 66e40df31fac by Martin Panter in branch '3.5':
Issue #22359: Disable running cross-compiled _freeze_importlib and pgen
https://hg.python.org/cpython/rev/66e40df31fac

New changeset c36b9ef8ff39 by Martin Panter in branch 'default':
Issue #22359: Merge cross-compilation fix from 3.5
https://hg.python.org/cpython/rev/c36b9ef8ff39

New changeset 0f7a299c6d50 by Martin Panter in branch '2.7':
Issue #22359: Avoid recursive $(MAKE); disable running cross-compiled pgen
https://hg.python.org/cpython/rev/0f7a299c6d50

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

_freeze_importlib and pgen are cross-built in this patch. Patch tested with a 
run of the testsuite after a cross-build.

--
Added file: http://bugs.python.org/file42554/crossbuild-sources-readonly_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

For the record, to fix this in 2.7 will involve backporting revision 
c2a53aa27cad, plus Xavier’s patch (except the freeze_importlib parts which is 
not relevant to 2.7).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

I see. I guess it would keep the makefile simpler if we cross-compiled both 
programs, and never used them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

> Why do you make the linking of _freeze_importlib conditional, but always 
> build $(PGEN)?

Yes, this is not consistent. The cross-build is correct when both are linked or 
when both are not linked. And $(PGENOBJS) are cross-compiled in both cases.

Should both programs not be created ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

Looks pretty good. One more question: Why do you make the linking of 
_freeze_importlib conditional, but always build $(PGEN)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

Martin, thanks for the review and the suggestions.

The attached patch makes the following changes:
* do not modify the source files so as not to break builds made from
  read-only build sources (as per issue 15819)
* the cross-build copies the graminit.[ch] files to the build directory
* fix tab indentation

Patch tested by checking that the graminit.[ch] time stamps do not
change in the source directory afer a build and after a cross-build,
and by running the test suite both natively and on android emulator.

--
Added file: http://bugs.python.org/file42530/crossbuild-sources-readonly.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Changes by Martin Panter :


--
dependencies:  -When cross-compiling, don’t try to execute binaries
status: languishing -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

> But I am a bit worried at the new makefile syntax [ifeq directive and 
> $(findstring) function]. I suspect it is Gnu specific, and that Python tries 
> to support other versions of Make as well.

Yes, they are both GNU extensions :(

To avoid modifying configure.ac since the information is already available, the 
recipes could be modified with the following conditional:

cross_compiling=$$(echo $(PYTHON_FOR_BUILD) | sed 
"s/.*_PYTHON_HOST_PLATFORM.*/yes/"); \
if [ "$$cross_compiling" = "yes" ]; then \
  touch $@;
else \
  # The original recipes.
fi

So the built binaries $(PGEN) and Programs/_freeze_importlib would be dummy 
empty files, but that would not prevent Programs/_freeze_importlib.o, 
$(LIBRARY_OBJS_OMIT_FROZEN) and $(PGENOBJS) to be needlessly cross-compiled.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

Here is a patch implementing my idea to make file reneration a separate step, 
not automatically run during the normal build process. But it is sounding like 
other people don’t like this idea, so Xavier’s approach might be less 
controversial even if it is more complicated.

--
Added file: http://bugs.python.org/file42169/separate-regen.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

Thanks for you work Xavier. Making it conditional on cross compilation is 
another approach I hadn’t thought of. But I am a bit worried at the new 
makefile syntax [ifeq directive and $(findstring) function]. I suspect it is 
Gnu specific, and that Python tries to support other versions of Make as well. 
Koobs: does Free BSD accept this makefile syntax?

Much as I hate the configure.ac stuff, I guess it could be used to enable and 
disable conditional stuff in the makefile, so that might be another option.

Whatever the end solution is, I think we need a big comment in the makefile 
pointing out the quirks and different people’s interests:

* Reliable way to update generated files when necessary
* Shared prerequisites unsafe with recursive concurrent $(MAKE)
* Reasons to avoid regenerating files (not out of date, cross compilation, less 
build steps and prerequisites; see python-dev)
* Building with read-only sources (see revision 67ed8a6905c3, then r87558)

--
keywords:  -easy
stage:  -> patch review
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

This patch improves changeset c2a53aa27cad by allowing cross-compilation:

The following checks have been made:
  * successfull cross-compilation of python3.4 after retrofiting changeset
c2a53aa27cad to python3.4 since I don't have a working cross-compilation
setup for python3.6 at the moment
  * successfull native compilation of python3.6 (default branch)
  * Python/graminit.c is re-generated after touching Parser/pgenmain.c on a
native build

--
nosy: +xdegaye
Added file: http://bugs.python.org/file42164/cross-chgeset-c2a53aa27cad.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
stage: commit review - 

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[koobs]

koobs added the comment:

Initial commit to default is done. Backport to 2.7  3.4 branches remains to do

--
components:  -Cross-Build
keywords: +easy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[koobs]

koobs added the comment:

Add 3.5 to list of versions merging required for

--
versions: +Python 3.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[koobs]

koobs added the comment:

martin: yes, if the incorrect usage of recursive make is in the 2.7 and 3.4 
branches and the fix is relevant to the issues I reported. Of course all 
subsequent changes (fixing cross-builds) should be applied there too.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

My math and cmath issue seems to be completely separate. I opened Issue 24421 
for that, so sorry for the noise here :)

I think the patch here should be applied (or adapted if necessary) to the 3.4 
and 2.7 branches. If it breaks cross compilation, then the cross compilation 
technique needs improving instead; perhaps see Issue 22625 for that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

koobs: It sounds like applying the fix for this bug to the Python 2 branch 
might satisfy you, right? It only seems to have been applied to 3.5+ so far.

Matt: It sounds like you are trying to pre-compile $(PGEN) for the host, and 
then rely on it not being rebuilt. Maybe there is a better way, like changing 
the $(PGEN) command line to use a $(HOST_CC) or something. (I’m not familiar 
with the build system; certainly not when cross compiling.) But you could also 
try and identify the dependency that is causing $(PGEN) to be rebuilt, for 
example with Gnu Make:

$ make --debug=b Parser/pgen
Updating goal targets
   Prerequisite `Include/Python.h' is newer than target `Parser/acceler.o'.
  Must remake target `Parser/acceler.o'.
[. . .]
 Prerequisite `Parser/acceler.o' is newer than target `Parser/pgen'.
[. . .]
Must remake target `Parser/pgen'.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[koobs]

Changes by koobs koobs.free...@gmail.com:


--
nosy: +koobs

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[koobs]

koobs added the comment:

We've (FreeBSD) seen and received several random and intermittent reports of 
failures during this stage of the build for all versions of Python, resulting 
in errors such as, among others:

Parser/pgen.o: file not recognized: File truncated
*** [Parser/pgen] Error code 1

pgenmain.c:(.text+0x258): undefined reference to `_Py_pgen'
*** [Parser/pgen] Error code 1
1 error
*** [Include/graminit.h] Error code 2

See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200622

Given the existing patch breaks cross-builds, is there an alternative or safer 
way to make this section of the build more robust in order to make progress?

--
versions: +Python 2.7, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

FWIW two times recently I saw strange intermittent build errors. I use “make 
-j2” on Linux. The one time that I investigated was a Python module (math or 
cmath perhaps?) failing to build because some dependency was zero bytes, and 
looking through the build log, I saw duplicate compiler commands lines printed 
out one after the other to build that dependency.

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[koobs]

Changes by koobs koobs.free...@gmail.com:


--
stage: resolved - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Matthias Klose]

Matthias Klose added the comment:

reopening, breaks cross builds. I'll have a look

--
resolution: fixed - 
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Matt Frank]

Matt Frank added the comment:

Sorry, I'm complaining.  Cross builds broke.  Please see issue22809.

--
nosy: +WanderingLogic

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Roundup Robot]

Roundup Robot added the comment:

New changeset c2a53aa27cad by Antoine Pitrou in branch 'default':
Issue #22359: Remove incorrect uses of recursive make.  Patch by Jonas Wagner.
https://hg.python.org/cpython/rev/c2a53aa27cad

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Antoine Pitrou]

Antoine Pitrou added the comment:

Ok, I've pushed the patch. Let's see if anyone complains.

--
resolution:  - fixed
stage: patch review - resolved
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Antoine Pitrou]

Antoine Pitrou added the comment:

Martin, Matthias, do you think this would break any legitimate use?

--
nosy: +loewis, pitrou
stage:  - patch review
type: compile error - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Jonas Wagner]

New submission from Jonas Wagner:

The attached patch fixes issues with Python's Makefile, which manifest when 
doing parallel builds. The Makefile invoked make recursively for some 
targets. This caused some files (which were depended upon by multiple targets) 
to be built by both the original make and the sub-make.

Besides duplicate work, this caused failed builds with non-threadsafe compilers.

The proposed patch removes recursive calls to make, and instead builds all 
targets in the same make process.

--
components: Build
files: makefile_parallel.patch
keywords: patch
messages: 226563
nosy: Sjlver
priority: normal
severity: normal
status: open
title: Remove incorrect uses of recursive make
type: compile error
versions: Python 3.5
Added file: http://bugs.python.org/file36570/makefile_parallel.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Berker Peksag]

Changes by Berker Peksag berker.pek...@gmail.com:


--
nosy: +doko

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Brett Cannon]

Changes by Brett Cannon br...@python.org:


--
nosy: +brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22359
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com