[issue22625] When cross-compiling, don’t try to execute binaries

[Martin Panter]

Martin Panter added the comment:

In Issue 22359, Xavier has posted a patch that looks like a reasonable solution 
to both cross compiling and allowing parallel make.

--
resolution:  -> duplicate
status: open -> closed
superseder:  -> Remove incorrect uses of recursive make

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Alex Willmer]

Alex Willmer added the comment:

On 14 March 2016 at 01:05, Robert Collins  wrote:
> There are three platforms in play: target, host, build.
>
> Host is the platform where what you build should run on.
> build is the platform we are building on.
> target is the platform where the *output* of the build thing itself should 
> run on. Baby steps though: lets assume target==host always.

To be 100% explicit: CPython doesn't need to worry about the third
one, the target platform. That only matters when the thing being
compiled, will itself cross-compile/process binaries at runtime e.g.
gcc, binutils. So if

 - I'm on Linux and
 - I want to compile a gcc that runs on BeOS
 - that in turn compiles binaries for TempleOS

only then would target matter.

--
nosy: +moreati

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Martin Panter]

Martin Panter added the comment:

Thanks Robert. Russell also pointed out on python-dev that the patches at Issue 
23670 show how he is doing cross compilation, via a new high-level iOS/Makefile 
file. Here is a cut-down version of the parts that I find relevant:

# Build for the native build host
host/bin/python*:
make -C .. distclean
cd .. && ./configure --prefix=host ...
make -C .. Programs/_freeze_importlib
# Main makefile is patched to use this copy (3.5+ only):
cp ../Programs/_freeze_importlib .
make -C .. install
tar czf host.tar.gz host && rm -rf host

# Build for a foreign target host
ios-%.tar.gz: host/bin/python*
make -C .. distclean
tar xzf host.tar.gz
cd .. && PATH=host/bin:$(PATH) ./configure \
--host=%-apple-ios --build=$(BUILD_OS_ID) CC=% LD=% \
--prefix=$* ...
PATH=host/bin:$(PATH) make -C .. install
tar czf $*.tar.gz $* && rm -rf $*

Something I just realized is that the output of pgen is actually committed to 
the Mercurial repository. It seems that Russell’s cross-compilation was relying 
on this to avoid rerunning pgen. Same with the output of _freeze_importlib, 
except Russell is working around that differently. It does not seem right to 
have files in the repository that are also generated by the normal build 
process.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Robert Collins]

Robert Collins added the comment:

So in general: 
https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/System-Type.html#System-Type
 and 
https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Hosts-and-Cross_002dCompilation.html
 

There are three platforms in play: target, host, build.

Host is the platform where what you build should run on.
build is the platform we are building on.
target is the platform where the *output* of the build thing itself should run 
on. Baby steps though: lets assume target==host always.

Now, the pathological case of building things is the canadian cross - 
https://en.wikipedia.org/wiki/Cross_compiler#Canadian_Cross

Note here that you actually build multiple different entire compilers, - and 
thats what we need here.

We need to build two python's. 

One, for build, which pgen and _freeze_importlib can depend on.

One, for host, which is the output, and can depend on the output of running 
pgen and _freeze_importlib

I don't have examples of Makefile parameterisation to support this offhand, but 
gcc would be the obvious (if perhaps overwhelming) place to look at it.

The key things I'd expect are that:
 - when host==build, the dependencies and outputs are identical, so we only 
build one copy of Python and everything else.
 - when host!=build, we get a chain - the host Python -> pgenoutput -> pgen -> 
build Python -> pgenoutput

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Martin Panter]

Martin Panter added the comment:

In  I 
suggested adding some makefile settings to manually override the pgen and 
_frozen_importlib programs. I imagine the changes would look something like 
cross-override.patch. Can someone who does cross compiling see if it is any 
use, and/or also indicate what commands or process they use to cross compile?

--
versions:  -Python 3.4
Added file: http://bugs.python.org/file42143/cross-override.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Alex Willmer]

Changes by Alex Willmer :


--
nosy: +Alex.Willmer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[ƦOB COASTN]

ƦOB COASTN added the comment:

Similar patch used for 3.5
Additionally required:
-Python/importlib_external.h: $(srcdir)/Lib/importlib/_bootstrap_external.py 
Programs/_freeze_importlib
+Python/importlib_external.h: $(srcdir)/Lib/importlib/_bootstrap_external.py
+   $(MAKE) Programs/_freeze_importlib
./Programs/_freeze_importlib \
$(srcdir)/Lib/importlib/_bootstrap_external.py 
Python/importlib_external.h

--
nosy: +mancoast

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Robert Collins]

Changes by Robert Collins robe...@robertcollins.net:


--
nosy: +rbcollins

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Martin Panter]

Martin Panter added the comment:

So is there somewhere that documents how cross compilation is meant to be done? 
I tried looking at https://docs.python.org/2/using/unix.html#building-python, 
https://hg.python.org/cpython/file/2.7/README, and “./configure --help”, but 
found no hints for cross compilation.

If it is just word-of-mouth, would someone mind explaining to me how it is 
meant to be done?

The configure script lists CC as a “C compiler command”. But for cross 
compilation it sounds like you would need to differentiate host and target 
compilers. From the errors that have been posted, it sounds like you guys may 
be setting CC to the target compiler, causing the build to try and use the 
target compiler to build host programs.

--
nosy: +vadmium
stage:  - test needed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Ethan Furman]

Changes by Ethan Furman et...@stoneleaf.us:


--
nosy:  -ethan.furman

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[koobs]

koobs added the comment:

Incorrect recursive use of make will be fixed in default, 3.5, 3.4 (?), 2.7 in 
issue 22359, reflect the versions correctly here so they're not forgotten.

--
nosy: +koobs
versions: +Python 2.7, Python 3.4, Python 3.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Matthias Klose]

Matthias Klose added the comment:

there are two approaches here, one to check in generated files and try to avoid 
the rebuild, or completely fix the cross build. I think the patch for the 
parallel build just was a bit over-eager

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Russell Keith-Magee]

Russell Keith-Magee added the comment:

I'm looking into this issue because of issue23670 (iOS support). 

Am I correct in assuming that the right fix here is to identify a 
$(CC_FOR_BUILD) analog for $(PYTHON_FOR_BUILD) that will identify the build 
host's CC, enabling a build-host native $(PGEN) and _freeze_importlib to be 
compiled?

--
nosy: +freakboy3742

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Ethan Furman]

Changes by Ethan Furman et...@stoneleaf.us:


--
nosy: +ethan.furman

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Link Mauve]

New submission from Link Mauve:

```
% make
./Programs/_freeze_importlib \
./Lib/importlib/_bootstrap.py Python/importlib.h
./Programs/_freeze_importlib: ./Programs/_freeze_importlib: cannot execute 
binary file
Makefile:710: recipe for target 'Python/importlib.h' failed
make: *** [Python/importlib.h] Error 126
```

I tried `make touch` as it was suggested to me on #python-dev, but it didn’t 
fix that issue.

--
components: Cross-Build
messages: 229261
nosy: Link Mauve
priority: normal
severity: normal
status: open
title: When cross-compiling, don’t try to execute binaries
type: compile error
versions: Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Georg Brandl]

Georg Brandl added the comment:

You're right, this can be avoided (although you will also have to patch the 
extension module build, which uses the just-built python executable).

This problem occurs twice in the build process, once for pgen and once for 
_freeze_importlib.  The problem is that the Makefile rules for the generated 
files depend on the binary (which is of course not checked in and cannot be 
touched by make touch).

Attached patch should fix this.

--
keywords: +patch
nosy: +benjamin.peterson, georg.brandl, pitrou
Added file: http://bugs.python.org/file36902/makefile-regen-fix.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Link Mauve]

Link Mauve added the comment:

Thanks, this patch worked fine. :)

I used to build pgen natively first, then make distclean and rebuild until it 
failed, and then put the native version back here, but this works much better!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Antoine Pitrou]

Antoine Pitrou added the comment:

This was removed in c2a53aa27cad, to fix #22359.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22625] When cross-compiling, don’t try to execute binaries

[Georg Brandl]

Georg Brandl added the comment:

Indeed, that is an issue :(

--
nosy: +doko

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22625
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com