[issue22699] Module source files not found when cross-compiling

[Isuru Fernando]

Change by Isuru Fernando :


--
keywords: +patch
nosy: +isuruf
nosy_count: 8.0 -> 9.0
pull_requests: +26810
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28397

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

> the multiarch approach allows you to install libraries and headers for many 
> architectures in the same installation/chroot

That makes sense, but I'm not using it. So presumably I've added a configure 
option that I didn't need (either "--host=x86_64-my-linux" or "--build=x86_64").

I'm building against a Yocto SDK (like described at 
https://www.yoctoproject.org/docs/2.1/sdk-manual/sdk-manual.html#sdk-installing-the-sdk),
 and the --host option identifies the sysroot.

When I add just --host, I also have to add --build, and IIRC only x86_64 
worked. Could that have enabled the multiarch settings? Or it is always enabled 
because I'm building on an Ubuntu VM?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Matthias Klose]

Matthias Klose  added the comment:

the multiarch approach allows you to install libraries and headers for many 
architectures in the same installation/chroot.  This can then be used to 
cross-build stuff.

you can check that on WSL with having Debian or Ubuntu installed. A short 
reference is https://wiki.debian.org/Multiarch/HOWTO
you don't need to build packages, you can build upstream sources as well, just 
use apt to install the required packages for the target architecture.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

> I think the patch is wrong.  why would you want to differentiate between 
> native and cross builds?

Because the multiarch headers are incompatible with my SDK's headers, and 
because of the -I ordering they override mine. I tried swapping the order of 
calls without a difference, but excluding them entirely made it work.

I'll happily take suggestions for a better patch. I don't even know what the 
multiarch headers are for, but I don't seem to need them here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Matthias Klose]

Matthias Klose  added the comment:

I think the patch is wrong.  why would you want to differentiate between native 
and cross builds?  There should be a generic way to pick up those for both 
cross and native builds.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

The values that are used are (apparently) correct, but most of sysconfig is 
not. That's very confusing, and makes it difficult to debug. I'm glad I get to 
maintain the *other* monster (MSBuild) :)

I would much rather sysconfig had a single variable pointing directly to 
whatever source of information it needs to tell everything about any Python 
install, whether that's _sysconfigdata or something else.

But I've also got my stuff working with only a minimal patch, so I'm not too 
concerned here. I'll go invest my time in improving the Windows 
cross-compilation support.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

So it's possible that my first few attempts didn't have a matched build of 
Python on the host - the cross-build certainly relies on mixing the installed 
runtime with the source stdlib, so that could have been an early cause of 
issues. I can only suspect that it was a factor in all the earlier reports too. 
Maybe we should detect this situation and fail faster?

The final problem in my case is that the multiarch paths conflict with the 
cross-compiling paths. The change below to configure_compiler() got me through 
a full build:

 if CROSS_COMPILING:
 self.add_cross_compiling_paths()
-self.add_multiarch_paths()
+else:
+self.add_multiarch_paths()
 self.add_ldflags_cppflags()

I have no idea whether this is a safe change in general. Anyone able to weigh 
in?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Xavier de Gaye]

Xavier de Gaye  added the comment:

> Is that epic line automatically generated?

When cross compiling it is generated by the configure script and configure 
replaces the right hand side of "PYTHON_FOR_BUILD=@PYTHON_FOR_BUILD@" of 
Makefile.pre.in with the generated value into Makefile.pre.

I think you may be missing the point that when this 'epic line' is being used 
then PYTHONPATH points to the location of the sysconfig module that has been 
newly built for the target platform, and so the values should be correct.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

Is that epic line automatically generated? It didn't work for me for some 
reason (was trying to launch the built Python, not the host).

Even so, I got to the point of filling in all those values manually while 
testing sysconfig, and while I'm 100% sure sysconfig still gave some wrong 
values, I'm 99% sure they weren't being fixed up by setup.py. Need to get back 
on my work machine to check.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Steve wrote:
> In my build, I've set PYTHON_FOR_BUILD=python3.8

This will not work, PYTHON_FOR_BUILD when cross-compiling is set by configure 
as:

PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) 
_PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f 
pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib 
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) 
'$interp

FWIW there is no problem cross-compiling Python and the standard library 
extension modules on Android, see for example the 'termux' Android application 
and its Python build.

On the other hand there are problems when cross-compiling third party extension 
modules due to:
* PYTHON_FOR_BUILD must be set correctly.
* distutils is still refering to sysconfig values of the build system instead 
of the target system.
But setup.py handles this correctly and does not need to be modified.

--
nosy: +xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Ned Deily]

Change by Ned Deily :


--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

So ultimately this is a major sysconfig design flaw, and it likely requires a 
rewrite.

The mix of information sources between _sysconfigdata (even when overriding the 
name and PYTHONPATH) and the Makefile, and the order in which you can rely them 
to be loaded (you can't) and the fact that INSTALL_SCHEMES depends on _neither_ 
(it only uses the running interpreter's sys module) means you can't possibly 
get a coherent view of the settings for anything other than the running 
interpreter.

Since we need something else for cross-compiling, we should either fix 
sysconfig so that it's possible to override the source of information reliably, 
or we avoid using it completely for building the CPython extension modules. The 
former would likely help third-party cross-compilation more.

I'm not sure who needs to be in on this, so I'll post to python-dev before 
going any further.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

Perhaps sysconfig needs an overhaul to be able to handle _PYTHON_PROJECT_BASE 
properly? I see so many values in there that are taken from the running 
interpreter (sys.prefix, for example) that ought to come from the 
cross-compilation target. Wouldn't surprise me if the host's CFLAGS are leaking 
through and adding those extra includes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22699] Module source files not found when cross-compiling

[Steve Dower]

Steve Dower  added the comment:

Trimmed a bit from setup.py here, but I think this is the cause:

def build_extensions(self):
self.srcdir = sysconfig.get_config_var('srcdir')
self.srcdir = os.path.abspath(self.srcdir)

[SNIP]

# Fix up the autodetected modules, prefixing all the source files
# with Modules/.
moddirlist = [os.path.join(self.srcdir, 'Modules')]

In my build, I've set PYTHON_FOR_BUILD=python3.8, and the contents of 
moddirlist is '/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/Modules'. So it's 
picking the wrong source directory.

I replaced the second line of the function above with:
self.srcdir = os.path.dirname(os.path.abspath(__file__))

Which made things a little better (I can launch my build now), but all the 
-I/usr/* options need to be replaced as well or there are no dynamically loaded 
modules.

--
title: cross-compilation of Python3.4 -> Module source files not found when 
cross-compiling

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com