[Python-Dev] Re: [Python-Help] Unable to bootstrap Python 3 install
Hi Christian, Some of the answers to your questions may possibly be found in PR 1629: bpo-30386: Add a build infrastructure for Android https://github.com/python/cpython/pull/1629 Please note that the purpose of this PR was not to give clear instructions on how to cross-compile android, but to add an android buildbot to the Python buildbots, hence its relative complexity. The README created by this PR provide some of the links that you are looking for: https://github.com/python/cpython/blob/0a51515a634872275aaeafb2e8a198289e6cf3d3/Android/README.rst > we would like to have clear instructions how to > reproduce the problem ourselves It is difficult to miss the irony in this sentence because at the time of this PR I was a Python core developer and the 'we' in your sentence would have included me then. So at this time 'we' had all the knowledge to fulfill your requests. The irony is that the reason why I resigned from being a core developer was because of the handling of this PR. Xavier de Gaye On Thu, Jun 16, 2022 at 7:54 PM Christian Heimes wrote: > On 16/06/2022 16.29, Xavier de Gaye wrote: > > Victor Stinner wrote: > > > https://github.com/python/cpython/issues/66913 > > <https://github.com/python/cpython/issues/66913> doesn't explain how to > > reproduce the issue, it only gives some info about what doesn't work. > > > I don't know how to reproduce the issue. > > > > Yes this issue does explain how to reproduce the problem. It even shows > > the backtrace printed when attempting to cross-build Android x86_64 on a > > linux x86_64. > > To reproduce the issue, just cross-build python for android x86_64 > > following the well documented process by the Android team. > > It is not clear how this backtrace can be missed when reading the issue ! > > > > There is also a patch provided in this issue that is straightforward and > > that does not involve any change on distutils, only the Makefile and > > configure. > > What Victor means that we would like to have clear instructions how to > reproduce the problem ourselves. Could you please provide step by step > instructions how I could set up a build environment on a X86_64 PC with > a standard Linux distro (Fedora, Debian/Ubuntu)? > > What packages have to be installed? Do I have to download any extra > packages? How do I have to set up my build environment? Which commands > do I have to execute? Is there a container image available that comes > with everything pre-installed? > > You mentioned well-documented process by the Android team. Could you > please provide links to the relevant documents? > > Christian > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EKJ2PFUFUCV3DOP55UAIRFYECX4NYLIU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-Help] Unable to bootstrap Python 3 install
A Google search on "andoid x86_64" gives about 10,900,000 results, showing
that cross-compiling is quite common for the case where the build system
and the host system have the same PLATFORM_TRIPLET as when building android
x86_64 on an x86_64. So the argument that this problem is not worth fixing
because no one has to cross-compile in this environment, IMHO this argument
is not valid.
Victor Stinner wrote:
> https://github.com/python/cpython/issues/66913 doesn't explain how to
reproduce the issue, it only gives some info about what doesn't work.
> I don't know how to reproduce the issue.
Yes this issue does explain how to reproduce the problem. It even shows the
backtrace printed when attempting to cross-build Android x86_64 on a linux
x86_64.
To reproduce the issue, just cross-build python for android x86_64
following the well documented process by the Android team.
It is not clear how this backtrace can be missed when reading the issue !
There is also a patch provided in this issue that is straightforward and
that does not involve any change on distutils, only the Makefile and
configure.
Dave wrote:
> Xavier de Gaye reworked the build system, apparently fixing the problem
with this pull request in Dec. 2019:
https://github.com/python/cpython/pull/17420
PR 17420 is about cross-compiling third-party extension modules. If your
problem is just cross-compiling Python you should stick to a solution that
only fixes the problem described in the first paragraph of my first comment
in https://github.com/python/cpython/issues/66913
Best regards.
Xavier de Gaye
On Wed, Jun 15, 2022 at 5:49 PM Gregory P. Smith wrote:
> -cc:help (bcc)
>
> FWIW the thing to do to move this forward is to open a new PR. Patches
> attached to an email are lost like tears in the rain.
>
> What you describe of cross compilation where host and target triples
> appear the same but use different libraries is a valid cross compilation
> case. But doesn't surprise me that there are bugs given most people never
> have such a setup. Even in an "every build should be a cross build" world
> (which we sadly don't have) it being broken may not show up.
>
> -gps
>
> On Wed, Jun 15, 2022 at 8:28 AM Victor Stinner
> wrote:
>
>> Hi,
>>
>> While this problem is causing you a lot of troubles, I never had to
>> cross compile Python, and I guess that it's the case for most people.
>> Changing the Python build system and distutils is stressful since it
>> can break Python for the majority of users, rather than leaving the
>> minority of users with an old bug. So far, nobody was brave enough to
>> "break" the system for cross compilation.
>>
>> Moreover, as far as I know, cross compiling Python works for the
>> common case: different platform triplet. Only the corner case of the
>> same triple is causing troubles.
>>
>> https://github.com/python/cpython/issues/66913 doesn't explain how to
>> reproduce the issue, it only gives some info about what doesn't work.
>> I don't know how to reproduce the issue. Please comment the issue.
>>
>> To cross compile Python, I found these documentations:
>>
>> *
>> https://docs.python.org/dev/using/configure.html#cross-compiling-options
>> * WASM: https://github.com/python/cpython/blob/main/Tools/wasm/README.md
>>
>> Currently, setup.py checks for:
>>
>> CROSS_COMPILING = ("_PYTHON_HOST_PLATFORM" in os.environ)
>>
>> Victor
>>
>>
>> On Tue, Jun 14, 2022 at 1:49 AM Dave Blanchard wrote:
>> > Here's what's happening. This is a very old problem reported ages ago
>> which has never been fixed. If I set the target arch to i686 (on an x86_64
>> build system) Python will cross-compile and bootstrap just fine. But if the
>> host and build triple are the same, then the problem will occur. Python
>> seems to be ASSuming that if the build and host triple match (in this case,
>> both being 'x86_64-linux-gnu') therefore the host and build libraries are
>> binary-compatible--which is NOT actually the case when one is
>> cross-compiling to a different libc, for example. Actually it's kind of
>> brain dead how this is implemented.
>> >
>> > Some prior discussions/years-old bug reports:
>> >
>> > https://bugs.python.org/issue39399
>> > https://bugs.python.org/issue22724
>> > https://bugs.gentoo.org/705970
>> >
>> > In those links, various hacks are attempted with various degrees of
>> success, but then Xavier de Gaye reworked the build system, apparently
>> fixing the problem with this pull request in Dec. 2019:
>> >
>> > https://github.com/python/cpython/pull/17420
>> >
>> > Unf
[Python-Dev] Re: Jump on C by PyEval_SetTrace Python 3.7.7 - SOLUTION
On 3/23/20 11:17 PM, Leandro Müller wrote: Hello Victor Stinner. Thanks. Your tip helped me a lot. I understood that I need to get position on bycode of the line. So I create the funcion to get position inside of the bycode. Now I can to jump the correct position on trace. frame->f_lasti = checkBycodePosition(frame->f_code, 11); There are cases where it is invalid to change f_lineno to avoid a python stack corruption. See the comments of the f_lineno setter frame_setlineno() in Objects/frameobject.c, the comments at the start of the function and inside the code itself. Xavier ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/J55XW45WYEI4F5BUX43OQGPDC7S22MCO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Why doesn't venv also install python3*-config?
On 1/9/20 3:18 PM, Victor Stinner wrote: Is the "ARM64" python-config (shell script) executed on the x86-64 host? A cross-compilation means that there is probably no build framework on the target platform and therefore the build configuration of the cross-compilation of Python is not very useful there. Which build command rely on python-config? Is it to cross-compile 3rd party C extensions on the host? It is useful when cross-compiling an application that embeds Python. And I have a hunch that this may be also useful to fix the cross-compilation of third-party extension modules :) Xavier ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2DRE2EEMJ4IYNC52ZV6HA4FNNIK5PRL4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Why doesn't venv also install python3*-config?
On 1/8/20 5:53 PM, Victor Stinner wrote: You may get the wrong information if you pick the wrong python-config script :-( IMHO we should add a new module (problem: how should it be called? pyconfig?) The shell script python-config has been introduced by bpo issue 16235 named "Add python-config.sh for use during cross compilation" in order "to behave exactly the same as python-config.py except it doesn't depend on a Python interpreter" as stated in the OP. So it does not make sense to say that one may pick the wrong python-config script. Its purpose is to provide cross-compilation configuration data independently of any Python interpreter running natively on the build machine used to do the cross-compilation. FWIW the shell script python-config is generated from Misc/python-config.sh by configure and updated by the Makefile. Xavier ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UNIJPEYGZHDW2UGBK4JNI5TEDY3MDDPF/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] PEP 558: Defined semantics for locals()
Currently f_locals is documented as readonly [1]. The PEP says: * "Don't change what isn't broken": the current tracing mode problems are caused by a requirement that's specific to tracing mode (support for external rebinding of function local variable references), so it made sense to also restrict any related fixes to tracing mode However, actually attempting to implement and document that dynamic approach highlighted the fact that it makes for a really subtle runtime state dependent behaviour distinction in how ``frame.f_locals`` works, and creates several new edge cases around how ``f_locals`` behaves as trace functions are added and removed. Accordingly, the design was switched to the current one, where ``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always a dynamic snapshot, which is both simpler to implement and easier to explain. Do these edge cases still exist when f_locals write access is restricted to code executed by the tracing function (which is more restrictive than 'tracing mode') ? We can use the condition frame->f_trace not NULL and tstate->tracing true (tstate being a pointer to the PyThreadState structure) to know when code is executed by the tracing function [2]: * The condition on tstate->tracing allows to figure out if we are running a frame executed by the trace function as opposed to a frame that is being traced or a frame executed in 'regular operation'. * The condition on frame->f_trace removes the ambiguity whether tstate->tracing is set by a tracing function or by a profiling function. [1] In section 'Frame objects' at https://github.com/python/cpython/blob/master/Doc/reference/datamodel.rst#the-standard-type-hierarchy [2] Except that frame->f_trace is NULL in a 'PyTrace_CALL' trace event, so f_locals would remain readonly in that case. But this event is always followed by a 'PyTrace_LINE' event anyway so this limitation is not important IMO. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] duplicate method names in tests
The last post [1] in issue bpo-16079 lists the methods in Lib/test that have duplicate names and that should be fixed. Xavier [1] https://bugs.python.org/issue16079#msg340168 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bedevere pipelines hang on github
Thanks. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] bedevere pipelines hang on github
bedevere/issue-number and bedevere/news are not triggered for some reason at https://github.com/python/cpython/pull/12708 and hang forever with "Expected — Waiting for status to be reported ". Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbottest on Android emulator with docker
Timeout (360 seconds) reached; failed to start emulator
---> Device ready.
---> Install Python on the emulator.
/home/pydev/build/python-native/python -B
/home/pydev/abifa/Android/tools/install.py
error: device 'emulator-5556' not found
I can reproduce this error after removing the kvm kernel modules (I am on
ArchLinux and ArchLinux kernels provide the required kernel modules to support
KVM virtualization). So my fault, forgot to explain that kvm is required to
run this docker image, kvm is not required though when the docker image is
built for other architectures (non x86_64), but it is very very slow then.
To fix the problem one must install the Linux distribution packages that
provide KVM virtualization. Whatever the Linux distribution it may also be
necessary to enable virtualization support in the BIOS.
I will enter an issue and update the documentation at
https://gitlab.com/xdegaye/abifa.
Thanks Stephane for the report.
Xavier
On 2/19/19 4:52 PM, Stephane Wirtel wrote:
Hi Xavier,
I get this exception
Timeout (360 seconds) reached; failed to start emulator
---> Device ready.
---> Install Python on the emulator.
/home/pydev/build/python-native/python -B
/home/pydev/abifa/Android/tools/install.py
error: device 'emulator-5556' not found
unzip -q /home/pydev/dist/python3.8-android-24-x86_64-stdlib.zip -d
/tmp/tmpt_v_gdbo
/home/pydev/android/android-sdk/platform-tools/adb -s emulator-5556 shell mkdir
-p /data/local/tmp/python
Command "('/home/pydev/android/android-sdk/platform-tools/adb', '-s',
'emulator-5556', 'shell', 'mkdir -p /data/local/tmp/python')" returned non-zero exit
status 1
/home/pydev/abifa/Android/emulator.mk:57: recipe for target '_install' failed
make: *** [_install] Error 1
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is distutils.util.get_platform() the "current" or the "target" platform
[Any reason for dropping python-dev?]
Sorry. just clicked the wrong button.
And the answer is a resounding "yes, it returns the target platform"? It seems
you're saying this, but the wording of your email sounds just enough of a question that
I'm not sure whether you are definitively answering it or not.
The answer is yes indeed, it returns the target platform.
This is a listing of the non-obvious steps that lead to this conclusion.
Xavier
On 2/19/19 8:45 PM, Steve Dower wrote:
[Any reason for dropping python-dev?]
On 19Feb2019 1139, Xavier de Gaye wrote:
Is distutils.util.get_platform() the "current" or the "target" platform ?
I *think* you're answering this below, yes?
* When cross-compiling on posix platforms using autoconf, configure is
run with the '--host=host-type' [1] command line option to specify
the target platform.
* The AC_CANONICAL_HOST macro is used by configure.ac to get the
canonical variables `host' and `host_cpu' [2].
Those variables are used to compute _PYTHON_HOST_PLATFORM.
* The Makefile generated by configure runs setup.py,
generate-posix-vars, etc... on the build platform using
PYTHON_FOR_BUILD, a native python interpreter that is set
to run with the _PYTHON_HOST_PLATFORM environment variable.
* get_platform() in setup.py and in Lib/distutils/util.py returns the
value of _PYTHON_HOST_PLATFORM when cross-compiling.
So the process of cross-compilation on posix platforms has
get_platform() return the target ('host' in autoconf terminology) platform.
[1]
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Specifying-Target-Triplets.html
[2]
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Canonicalizing.html
And the answer is a resounding "yes, it returns the target platform"? It seems
you're saying this, but the wording of your email sounds just enough of a question that
I'm not sure whether you are definitively answering it or not.
Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbottest on Android emulator with docker
https://gitlab.com/xdegaye/abifa
The table of content of the first url in my initial post gives also a
link to that repository.under the name 'Repository'.
Xavier
On Tue, Feb 19, 2019 at 4:54 PM Stephane Wirtel wrote:
>
> Hi Xavier,
>
> I get this exception
>
> Timeout (360 seconds) reached; failed to start emulator
> ---> Device ready.
> ---> Install Python on the emulator.
> /home/pydev/build/python-native/python -B
> /home/pydev/abifa/Android/tools/install.py
> error: device 'emulator-5556' not found
> unzip -q /home/pydev/dist/python3.8-android-24-x86_64-stdlib.zip -d
> /tmp/tmpt_v_gdbo
> /home/pydev/android/android-sdk/platform-tools/adb -s emulator-5556 shell
> mkdir -p /data/local/tmp/python
> Command "('/home/pydev/android/android-sdk/platform-tools/adb', '-s',
> 'emulator-5556', 'shell', 'mkdir -p /data/local/tmp/python')" returned
> non-zero exit status 1
> /home/pydev/abifa/Android/emulator.mk:57: recipe for target '_install' failed
> make: *** [_install] Error 1
>
>
> Is there a repository with a bug tracker because I would not want to
> pullute this ML with that.
>
> Thank you,
>
> Stéphane
>
>
> --
> Stéphane Wirtel - https://wirtel.be - @matrixise
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/xdegaye%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] buildbottest on Android emulator with docker
The following command runs the buildbottest on an Android emulator with docker (it will use a little bit more than 11 GB): $ docker run -it --privileged xdegaye/abifa:r14b-24-x86_64-master This command does: * pull an image from the Docker hub (only the first time that the command is run, note that this a 2 GB download !) and start a container * pull the latest changes from the GitHub cpython repository and cross-compile python * start an Android emulator and install python on it * run the buildbottest target of cpython Makefile The image is built from a Dockerfile [2]. This same image can also be used with the 'bash' command line argument to enter bash in the container and run python interactively on the emulator [1]. If the 'docker run' command also sets a bind mount to a local cpython repository, then it is possible to develop/debug/fix python on the emulator running in this container using one's own clone of cpython. [1] documentation at https://xdegaye.gitlab.io/abifa/docker.html [2] Dockerfile at https://gitlab.com/xdegaye/abifa/blob/master/docker/Dockerfile.r14b-24-x86_64-master Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 compiled listcomp can't see local var - bug or feature?
On 06/06/2018 03:51 PM, Nick Coghlan wrote: > On 6 June 2018 at 15:31, Rob Cliffe via Python-Dev mailto:python-dev@python.org>> wrote: > ... > *In other words, it looks as if in Python 3.6.5, the compiled list comprehension** > **can "see" a pre-existing global variable but not a local one.* > > Yes, this is expected behaviour - the two-namespace form of exec (which is what you get implicitly when you use it inside a function body) is similar to a class body, and hence nested functions > (including the implicit ones created for comprehensions) can't see the top level local variables. In issue 13557 [1] Amaury gives the following explanation by quoting the documentation [2] "Free variables are not resolved in the nearest enclosing namespace, but in the global namespace" and hints at the same solution that is proposed by Nick. FWIW in issue 21161 [3] folks have been bitten by this when trying to run a list comprehension in pdb. [1] https://bugs.python.org/issue13557 [2] http://docs.python.org/py3k/reference/executionmodel.html#interaction-with-dynamic-features [3] https://bugs.python.org/issue21161 Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/14/2017 02:59 PM, Victor Stinner wrote: > It seems like Android is evolving quickly, would say quicker than > Python releases. I'm asking if it's a good idea to put a recipe aside > the Python source code for one specific Android API version? Would it > still make sense to build for NDK v14 in 2 or 5 years? NDK 14 has been released in march 2017 and the latest release is NDK 16. There are sometimes major changes between releases and I think it is critical to ensure that the builds all use the same NDK release for that reason. Supporting another NDK release is just a substitution in one of the files of the build system and I am sure that in 2 or 5 years there would have been a core developer smart enough to make that substitution (this could even have been me, I will only be 71 years old in 5 years :-)). Anyway if this is a problem, this should have been discussed in a review of the PR. There are concerns, including a concern raised by me, about supporting Android with that build system or to supporting Android at all. It has been interesting and gratifying to work on this build system and to get the Python test suite running on Android without failures. Given these concerns and the lack of interest in the support of Android it is time for me to switch to something else, maybe improve the bdb module, why not ? Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/13/2017 10:56 PM, Victor Stinner wrote: > I looked at your scripts to build Android but I failed to use them. You failed because you did not read the README and tried to improvise. I will change the documentation and the build process to make it simpler for those that do not have the time to RTFM :-) I have documented the correct steps in the PR especially for you (they are just a summary of what is in the README) after you reported that failure. So you can follow these steps any time now and use correctly the build system and report the results here. This would be only fair, I think. BTW You keep saying they are scripts when the build is actually driven by Makefiles (with the '.mk' extension). > Anyway, I'm not sure why these scripts have to be part of the CPython > git repository. > > Technically, is there a reason to put it aside the source code and > Unix build scripts (configure/Makefile/setup)? > Your https://github.com/python/cpython/pull/1629 only adds new files > without touching existing files. > > I suggest to create new Git project. It may be in the python > organization, or you may start with your GitHub account. The 'Mac' build system has its own subdirectory in the source tree and it makes sense as it is the reference build system for this platform. I do not see why this should be different for Android. > Cross-compilation is hard, and I'm not sure that it's possible to > build a single recipe for all Android API versions, all configuration, > any set of libraries, etc. For Android, it seems like each developer > might want a subtle different configuration which might not be easy to > support. You are mistaken, this proposal does not suggest that we are going to support "all Android API versions, all configuration, any set of libraries, etc.", quite the opposite actually. The proposal is an Android build system for a specific API, a set of architectures using the NDK r14 toolchain and a set of optional external libraries. The build system enforces the use of NDK r14 for example (as you have painfuly experienced). Python is tested on emulators so that there is no interference with vendor specific additions found on the Android devices or with installed PlayStore applications. > Having a separated Git project would allow people to contribute more > easily, experiment their fork, etc. > > What do you think? Certainly not. We, core-devs, are very happy that no one is experimenting with our build system, it is complex enough as it is. The same goes for this Android build system. Your suggestion seems to be driven by the failure you have experienced with this new build system and the fact that a user is also reporting a failure. The origin of this other failure is unclear because I cannot reproduce it even though all the components used for the build are well defined and identical for everyone: the NDK includes the clang compiler, the libraries and the headers, the external libraries are downloaded by the build system, all the users use identical tools (same versions) and the same source code and the only difference may be with some utility tools such as sed, awk, etc... This is a bad start for this proposal and it would have been fair to inform me that you were working in irc collaboration with this other user in testing the build system. On the other hand, these problems may have some positive consequences since it allows us to be aware of the fact that the bpo audience may change if we support Android and that this may be a problem. Android attracts all kind of developers that do not have the average expertise of unix developers and more importantly that do not have the same motivations and the same etiquette. I am now concerned by the fact that the quality of the bug reports on bpo may dramatically decrease if we adopt this proposal. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/11/2017 04:14 PM, Victor Stinner wrote: > I'm asking for precise hardware specifications since Red Hat may be > able to provide one through the https://osci.io/ program. Is it acceptable to run the arm buildbots only on a weekly basis ? For all the architectures (x86_64, armv7 and arm64), the tests are run with the same Python code, built with the same tools and running on the same operating system that runs on emulators using the same configuration. If we do not take into account the ctypes issue, there is only one issue (issue 26939 [1]) among all the Android issues that is specific to arm, and it is not even really an arm issue but an issue related to the slowness of the emulator that has been fixed by increasing in a test the switch interval with sys.setswitchinterval(). Xavier [1] https://bugs.python.org/issue26939 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What does Android support mean?
On 12/11/2017 03:58 PM, Carl Bordum Hansen wrote: > > I've been lurking at your progress with android support for about a year, and now that it is closing in I simply have to ask: what does it actually mean that android is supported? That Android apps > will be easy to develop in Python? That I can easily run Python script or a REPL on my phone? Hello Carl, It means that we will make sure that changes made to Python to fix it or to improve it will not break on Android for the supported API level and architectures. One can already run Python scripts or the interactive interpreter and install packages with pip using termux, a great application. Kivy can be used to develop in Python apps that work on Android and you may be interested in the Python Mobile SIG [1] mailing list or in the VOC [2] transpiler that converts Python code into Java bytecode. Xavier [1] https://www.python.org/community/sigs/current/mobile-sig/ [2] https://github.com/pybee/voc ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/11/2017 03:40 PM, Victor Stinner wrote: > 2017-12-11 14:58 GMT+01:00 Xavier de Gaye <xdeg...@gmail.com>: >> The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x >> + maintenance version)) emulators simultaneously, so with an eight core cpu, >> that will be 6 cores running at 100%. The armv7 and arm64 buildbot may be >> set to run only daily but the tests last a long time on these architectures >> anyway. > > What do you mean by "maintenance version"? Do you want to add Android > support to Python 2.7 and 3.6 as well? I would prefer to only support > Android since the master branch. Yes, today only the master branch, and when 3.8 is released then both 3.7 and 3.8 would be supported. That is what I (implicitly) meant. > Would it be possible to only run an emulator to run a test, and then > stop it? So we could test many combo in the same host? I may not understand your question. An emulator runs with an AVD image (Android Virtual Device) that is specific to the API and architecture (armv7, x86_64, ...), and that contains the data (file systems) and the configuration. The AVD image is created once and for all for each of the combinations (API, architecture). You start the emulator with the AVD that matches the type of the built python (API, architecture) and install that build. Multiple emulators can run simultaneously except that the build system imposes the restriction that no two identical (API, architecture) emulators can run concurrently except if they are not used for the same Python version (emulators are identified by their port numbers by the Android tools and the build system allocates those ports statically). Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/11/2017 12:56 PM, Victor Stinner wrote: > 2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdeg...@gmail.com>: >> * Given the cpu resources required to run the test suite on the arm >> emulators, >>it may be difficult to find a contributed buildbot worker. So it remains >> to >>find the hardware to run these buildbots. > > Do you have the hardware to host such worker? Or are you looking for a > host somewhere? Which kind of hardware are you looking for? CPU, > memory, network bandwidth, etc. I cannot host the buildbots or any buildbot for that matter. I can maintain them. The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x + maintenance version)) emulators simultaneously, so with an eight core cpu, that will be 6 cores running at 100%. The armv7 and arm64 buildbot may be set to run only daily but the tests last a long time on these architectures anyway. >> *API 24* >>* API 21 is the first version to provide usable support for wide >> characters >> and where SELinux is run in enforcing mode. > > Some people are looking for API 19 support. Would it be doable, or > would it require too many changes? I know that people are running > heavily patched Python 2.7 and 3.5 on Android with API 19. > > I'm not asking for a "full support" for API 19, but more if it would > be possible to get a "best effort" level of support, like accept > patches if someone writes them. Not sure that python can be built on API 19. What I remember about API 19 at the time I started this project, is that wide characters support is not usable. If you look at the Android version history [1] on Wikipedia referred to in my initial post, the Kit Kat (API 19) share is 16 % now and will probably be 8 % next year. Another point to consider is that working on a change specific to Android is tedious: the test case must be ok on the build platform and on the emulator. The emulator must be started and an installation made from scratch, and after few file modifications on the emulator there is no 'git status' command to tell exactly what change you are running and you must re-install from scratch. Is there a way to browse these patches to get a better idea of the changes involved ? Xavier [1] https://en.wikipedia.org/wiki/Android_version_history ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/10/2017 11:07 PM, Ned Deily wrote: > On the one hand, it would be nice to have. But on the other, it does add a large non-zero burden to all core developers and to the release teams, to the minimum extent of trying to make sure that all ongoing changes don't break platform support. Yes this platform has few quirks especially when SELinux is involved and some tests are harder to write because of that :-( Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/10/2017 10:27 PM, Skip Montanaro wrote: > I'm not familiar with software development on/for Android, but > wouldn't official support also involve suitable package creation or > does that just fall out for free from the build-for-emulator PR? > > Skip > The build-for-emulator PR allows building for a device but the packaging of an Android application into an installable APK (a zip file actually) is the responsability of the java application developer. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
On 12/10/2017 09:27 PM, Brett Cannon wrote: > While the note from a technical standpoint is interest, Xavier, I don't quite see what needs to be done to support Android at this point. Are you simply asking we add Android API 24 as an official > platform? Or permission to add your note to the Misc/ directory? Basically what are you wanting to see happen? :) This is mainly a proposal written in the form of a PEP and open for discussion, not a PEP because it is not required by PEP 11 but it may become one. I guess the discussion may be about the choice of the API level, or whether the buildbot should run on emulators or on devices (not currently supported by the build system) or on any other subject. What is left to be done to support Android at this point is to merge the PR that implements the build system, make the change in the buildbot buildmaster-config repo and test it on a worker and then find the hardware to run all the workers for these architectures and set it up. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Support of the Android platform
The following note is a proposal to add the support of the Android platform. The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst Motivations === * Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_. Current status == * The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots. Proposal Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b. *API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode. * API 22 introduces an annoying bug on the linker that prints something like this when python is started:: ``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffe arg 0x14554``. The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables. * API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64. *x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker. *NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now). Android idiosyncrasies == * The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8. Android build system The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features: * To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing. The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example. The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules: * ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures. The following extension modules are disabled
Re: [Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523
On 03/25/2017 08:57 PM, Terry Reedy wrote: > On 3/25/2017 8:56 AM, Serhiy Storchaka wrote: >> On 25.03.17 12:04, Victor Stinner wrote: >>> https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ >>> >>> >>> "What changed in Python 3.6 to allow this? >>> >>> The new frame evaluation API was introduced to CPython in PEP 523 and it >>> allows to specify a per-interpreter function pointer to handle the >>> evaluation of frames." >>> >>> Nice! >> >> Awesome! Any chance that pdb can utilize similar technique? Or this >> doesn't make sense for pdb? > > According to the bdb.Bdb docstring, pdb implements a command-line user interface on top of bdb, while bdb.Bdb "takes care of the details of the trace facility". idlelib.debugger similarly implements > a GUI user interface on top of bdb. I am sure that there are other debuggers that build directly or indirectly (via pdb) on bdb. So the question is whether bdb can be enhanced or augmented with a > C-coded _bdb or other new module. > > As I understand it, sys.settrace results in an execution break and function call at each point in the bytecode corresponding to the beginning of a (logical?) line. This add much overhead. In return, > a trace-based debugger allows one to flexibly control stop and go execution either with preset breakpoints* or with interactive commands: step (one line), step into (a function frame), step over (a > function frame), or go to next breakpoint. The last is implemented by the debugger automatically stepping at each break call unless the line is in the existing breakpoint list. > > * Breakpoints can be defined either in an associated editor or with breakpoint commands in the debugger when execution is stopped. > > PEP 523 envisioned an alternate non-trace implementation of 'go to next breakpoint' by a debugger going "as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the > bytecode." > https://www.python.org/dev/peps/pep-0523/#debugging > > A debugger doing this could either eliminate the other 'go' commands (easiest) or implement them by either setting temporary breakpoints or temporarily turning tracing on. > > I presume it should be possible to make bdb.Bdb use bytecode breakpoints or add a new class with a similar API. Then any bdb-based debugger to be modified to make the speedup available. pdb-clone, an extension to pdb, gets about those same performance gains over pdb while still using sys.settrace(). pdb-clone runs at a speed of less than twice the speed of the interpreter when pdb runs at about 80 times the speed of the interpreter. See some performance measurements at https://bitbucket.org/xdegaye/pdb-clone/wiki/Performances.md Given those results, it is not clear how one would get a boost of a factor 40 by implementing PEP 523 for the pdb debugger as pdb could already be very close to the speed of the interpreter mostly by implementing in a C extension module the bdb.Bdb methods that check whether the debugger should take control. Setting a trace function with sys.settrace() adds the following incompressible overhead: * 15-20 % overhead: computed goto are not used in the ceval loop when tracing is active. * The trace function receives all the PyTrace_LINE events (even when the frame f_trace is NULL :(). The interpreter calls _PyCode_CheckLineNumber() for each of these events and the processing in this function is the one that is costly. An optimization is done in pdb-clone that swaps the trace function with a profiler function whenever possible (i.e. when there is no need to trace the lines of the function) to avoid calling _PyCode_CheckLineNumber() (the profiler still gets PyTrace_C_CALL events but there is not such overhead with these events). The performance gain obtained with this scheme is about 30%. I think that the main point here is not whether to switch from sys.settrace() to PEP 523, but first to implement the stop_here() bdb.Bdb method in a C extension module. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] attributes of Python code objects
In PR 218 [1], INADA Naoki wrote: > I think attributes of Python's code object is implementation detail, even > though PyPy follows. > But it's not big problem until there are some Python implementation > having different code implementation. Python's code object attributes are described in the Data Model [2] section of the documentation. 1) Is this description normative (to be followed by all the implementations of the Python language)? 2) Is the description of co_lnotab in [3] normative ? Xavier [1] https://github.com/python/cpython/pull/218 [2] https://docs.python.org/3/reference/datamodel.html [3] https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the Android platform
On 01/14/2017 09:56 PM, Victor Stinner wrote: > Great job! Thank you and Chi Hsuan Yen! > > Did you get feedback from users? Maybe from the Kivy community? > > Victor > > Le 14 janv. 2017 18:31, "Xavier de Gaye" a écrit : > > Only a few minor issues are left to be fixed before the support of the > Android platform may be considered. See the current status at msg285493 [1] > in the Android meta-issue 26865 [2]. Yes Chi Hsuan Yen is a great contributor to this project. AFAIK Kivy uses the CrystaX NDK [1] for Python 3 instead of the Android NDK, mainly because the Android NDK did not support wide character until API level 21. Xavier [1] https://www.crystax.net/en ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] support of the Android platform
Only a few minor issues are left to be fixed before the support of the Android platform may be considered. See the current status at msg285493 [1] in the Android meta-issue 26865 [2]. Xavier [1] http://bugs.python.org/issue26865#msg285493 [2] http://bugs.python.org/issue26865 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
On 04/26/2016 07:12 PM, Stefan Krah wrote: > Xavier de Gaye gmail.com> writes: >> This code, or part of it, could be used to setup a buildbot and in this case >> there would not be any conflict between the GPL v3 license and the Python >> license, I think. I don't see how it can be combined with Python 3. > > For the patches on the tracker I just went by your contributor agreement. > I didn't check the lineage of the patches. Can I assume that either you > are re-licensing GPL-stuff written by yourself to the PSF (which is a > perfectly valid use case of the agreement) or rewriting from scratch? Yes, I am re-licensing GPL code to the PSF for all the patches written by me in the issues listed on http://bugs.python.org/issue26865#msg264310. I have only rewritten the patches from scratch in the following issues: issue #26849: android does not support versioning in SONAME (using a switch case on ac_sys_system) issue #26854: missing header on android for the ossaudiodev module (actually it's difficult to rewrite such an obvious patch) issue #26855: add platform.android_ver() for android (using configparser; Chi Hsuan Yen is proposing a more complete approach) Fixes for those three issues can also be found in other projects porting python3 to android, the ones that I know of are: * Python 3 Android at https://github.com/yan12125/python3-android, author Chi Hsuan Yen * python-for-android at https://github.com/kuri65536/python-for-android, author shimoda dragon I also browsed rapidly issue 23496 and could not find any overlap with my patches. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
On 04/26/2016 04:14 PM, Ethan Furman wrote: > On 04/24/2016 12:20 AM, Xavier de Gaye wrote: > >> [1] https://bitbucket.org/xdegaye/pyona/src > > The license: > --- > This software is licensed under the GNU General Public License version 3 or later. > --- > > > Will combining your code with Python 3 be a problem? This code, or part of it, could be used to setup a buildbot and in this case there would not be any conflict between the GPL v3 license and the Python license, I think. I don't see how it can be combined with Python 3. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
On 04/26/2016 04:02 AM, Nick Coghlan wrote: > On 26 April 2016 at 06:25, Xavier de Gaye <xdeg...@gmail.com <mailto:xdeg...@gmail.com>> wrote: > > On 04/25/2016 04:53 PM, Stefan Krah wrote: > > Eric Snow gmail.com <http://gmail.com>> writes: > >> On Sun, Apr 24, 2016 at 1:20 AM, Xavier de Gaye gmail.com <http://gmail.com>> > > wrote: > >>> Starting with API level 21 (Android 5.0), the build of python3 with the > >>> official android toolchains (that is, without resorting to external > > > >> How does this relate tohttp://bugs.python.org/issue23496? > > > > As I understand, that issue seems abandoned and the patches are > > (despite core devs asking otherwise) against 3.4. > > > > > > If Xavier is willing to do so, I think it would be best to start over > > with a new issue that integrates his work into 3.6. > > I will enter a new issue that lists all the new issues and the other already > existing issues that, would have they been fixed, would have allowed a > successfull cross-build and the same test suite results as described in my > previous post. > > > Thanks for this, Xavier! > > Once you have that, in addition to posting the link back here, you may also want to ping the Mobile SIG list: https://www.python.org/community/sigs/current/mobile-sig/ Issue 26865 [1] lists issues that may have to be fixed in the perspective of a future support of the android platform. Xavier [1] http://bugs.python.org/issue26865 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
On 04/25/2016 04:53 PM, Stefan Krah wrote: > Eric Snow gmail.com> writes: >> On Sun, Apr 24, 2016 at 1:20 AM, Xavier de Gaye gmail.com> > wrote: >>> Starting with API level 21 (Android 5.0), the build of python3 with the >>> official android toolchains (that is, without resorting to external > >> How does this relate to http://bugs.python.org/issue23496? > > As I understand, that issue seems abandoned and the patches are > (despite core devs asking otherwise) against 3.4. > > > If Xavier is willing to do so, I think it would be best to start over > with a new issue that integrates his work into 3.6. I will enter a new issue that lists all the new issues and the other already existing issues that, would have they been fixed, would have allowed a successfull cross-build and the same test suite results as described in my previous post. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
On 04/25/2016 04:36 PM, Eric Snow wrote: > On Sun, Apr 24, 2016 at 1:20 AM, Xavier de Gaye <xdeg...@gmail.com> wrote: >> Starting with API level 21 (Android 5.0), the build of python3 with the >> official android toolchains (that is, without resorting to external >> libraries >> for wide character support) runs correctly. With the set of patches >> described >> in the patches/Makefile file at [1], the cpython test suite runs[2] on the >> android x86 and armv7 emulators with only few errors[3]. Those errors are >> listed with their corresponding error messages, this may give a raw idea of >> the effort needed to support this platform. > > How does this relate to http://bugs.python.org/issue23496? The patches in issue 23496 address the native compilation of Android 4.4.2 on an android device using a port of gcc on this device. Some of these patches are not needed anymore on Android 5.0 and it seems that the kbox_fix.patch is needed because the KBOX application is used to build python in issue 23496. The existing issues that are relevant to the android platform are, I think: issue #26723: Add an option to skip _decimal module issue #22747: Interpreter fails in initialize on systems where HAVE_LANGINFO_H is undefined issue #16353: add function to os module for getting path to default shell issue #20306: Lack of pw_gecos field in Android's struct passwd causes cross-compilation for the pwd module to fail Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
On 04/24/2016 11:50 AM, Stefan Krah wrote: > Xavier de Gaye gmail.com> writes: >> Starting with API level 21 (Android 5.0), the build of python3 with the >> official android toolchains (that is, without resorting to external libraries >> for wide character support) runs correctly. With the set of patches described >> in the patches/Makefile file at [1], the cpython test suite runs[2] on the >> android x86 and armv7 emulators with only few errors[3]. Those errors are >> listed with their corresponding error messages, this may give a raw idea of >> the effort needed to support this platform. >> >> Xavier >> >> [1] https://bitbucket.org/xdegaye/pyona/src >> [2] To reproduce these results, follow the instructions found in INSTALL >> at https://bitbucket.org/xdegaye/pyona/wiki/install >> [3] https://bitbucket.org/xdegaye/pyona/wiki/testsuite > > > This looks great, very clean! As I understand the patches, the locale.h and > langinfo.h problems are solved. Do you think the following issues on the > Python bug tracker could be closed? > > > http://bugs.python.org/issue20305 > http://bugs.python.org/issue22747 > http://bugs.python.org/issue17905 Thanks. A fix is still needed because Android does not HAVE_LANGINFO_H. I have tried to answer your question directly in those issues. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] support of the android platform
Starting with API level 21 (Android 5.0), the build of python3 with the official android toolchains (that is, without resorting to external libraries for wide character support) runs correctly. With the set of patches described in the patches/Makefile file at [1], the cpython test suite runs[2] on the android x86 and armv7 emulators with only few errors[3]. Those errors are listed with their corresponding error messages, this may give a raw idea of the effort needed to support this platform. Xavier [1] https://bitbucket.org/xdegaye/pyona/src [2] To reproduce these results, follow the instructions found in INSTALL at https://bitbucket.org/xdegaye/pyona/wiki/install [3] https://bitbucket.org/xdegaye/pyona/wiki/testsuite ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in build system for cross-platform builds
On 03/14/2016 05:34 PM, Xavier de Gaye wrote: > Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect > uses of recursive make. The changeset added executable binaries as > prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)). > This broke cross-compilation: > * the executables do not exist and must be cross-compiled > * then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since >the prerequisites are newer > * but the executables cannot run on the build system > > Actually the files need not be re-generated as their timestamps have been > setup for that purpose with 'make touch'. So a solution to the problem > introduced by this changeset when cross-compiling could be to remove the > binaries as prerequisites of these rules and include the recipe of their > corresponding rules, the one used to build the executable, into the recipes of > the original rule. Also IMHO the Programs/_freeze_importlib.c can be used > instead of Programs/_freeze_importlib.o as a prerequisite in the > Python/importlib.h rule. > > [1] https://hg.python.org/cpython/rev/c2a53aa27cad/ > [2] http://bugs.python.org/issue22359 The pgen dependencies are lost when following my previous suggestion, which is wrong. I have uploaded a patch at issue 22359 that uses a conditional to change the rules, based on whether a cross-compilation is being run. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in build system for cross-platform builds
On 03/14/2016 02:26 PM, R. David Murray wrote: > > The repo-timestamp problem is addressed by the 'make touch' target. > > And yes, checking in these platform-independent artifacts is very > intentional: less to build, fewer external dependencies in the build > process...you don't need to *have* python to *build* python, which you > would have to if they were not checked in. Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect uses of recursive make. The changeset added executable binaries as prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)). This broke cross-compilation: * the executables do not exist and must be cross-compiled * then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since the prerequisites are newer * but the executables cannot run on the build system Actually the files need not be re-generated as their timestamps have been setup for that purpose with 'make touch'. So a solution to the problem introduced by this changeset when cross-compiling could be to remove the binaries as prerequisites of these rules and include the recipe of their corresponding rules, the one used to build the executable, into the recipes of the original rule. Also IMHO the Programs/_freeze_importlib.c can be used instead of Programs/_freeze_importlib.o as a prerequisite in the Python/importlib.h rule. [1] https://hg.python.org/cpython/rev/c2a53aa27cad/ [2] http://bugs.python.org/issue22359 Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issues not responded to.
On 07/31/2015 06:42 AM, Zachary Ware wrote: On Thu, Jul 30, 2015 at 8:21 PM, Brett Cannon br...@python.org wrote: Best thing I can think of is to post the Roundup search you did to find those 400 so thoseof us who can help can just start whittling them away. You could also share it with core-mentorship and explain we need help evaluating these issues with the caveat we have no idea how difficult it is to do the evaluation. Here's a query: https://bugs.python.org/issue?@action=search@columns=title,id,creator,activity,actor,status@sort=activitystatus=-1,1,3,4message_count=1 This is nice, thanks. Note that this is missing the cases where more than one message was required, for example to send two attachments (a script as the use case, and a patch). Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
On 02/06/2015 11:48 PM, Francis Giraldeau wrote: 2015-02-06 6:04 GMT-05:00 Armin Rigo: Hi, On 6 February 2015 at 08:24, Maciej Fijalkowski fij...@gmail.com mailto:fij...@gmail.com wrote: I don't think it's safe to assume f_code is properly filled by the time you might read it, depending a bit where you find the frame object. Are you sure it's not full of garbage? Yes, before discussing how to do the utf8 decoding, we should realize that it is really unsafe code starting from the line before. From a signal handler you're only supposed to read data that was written to volatile fields. So even PyEval_GetFrame(), which is done by reading the thread state's frame field, is not safe: this is not a volatile. This means that the compiler is free to do crazy things like *first* write into this field and *then* initialize the actual content of the frame. The uninitialized content may be garbage, not just NULLs. Thanks for these comments. Of course accessing frames withing a signal handler is racy. I confirm that code encoded in non-ascii is not accessible from the uft8 buffer pointer. However, a call to PyUnicode_AsUTF8() encodes the data and caches it in the unicode object. Later access returns the byte buffer without memory allocation and re-encoding. I think it is possible to solve both safety problems by registering a handler with PyPyEval_SetProfile(). On function entry, the handler will call PyUnicode_AsUTF8() on the required frame members to make sure the utf8 encoded string is available. Then, we increment the refcount of the frame and assign it to a thread local pointer. On function return, the refcount is decremented. These operations occurs in the normal context and they are not racy. The signal handler will use the thread local frame pointer instead of calling PyEval_GetFrame(). Does that sounds good? You could call Py_AddPendingCall() from your signal handler and access the frame members from the function scheduled by Py_AddPendingCall(). Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Wave module support for floating point data
On 03/02/2014 12:04 AM, Terry Reedy wrote: On 3/1/2014 3:25 PM, Antoine Pitrou wrote: On Sat, 01 Mar 2014 15:08:00 -0500 Terry Reedy wrote: On 3/1/2014 2:57 PM, Sebastian Kraft wrote: Hi everybody, more than a year ago I have submitted a patch to enhance the Wave module with read/write support for floating point data. http://bugs.python.org/issue16525 Up till now this patch has not been applied nor did I get feedback if anything needs to be enhanced or changed before it can be committed. I have never been in contact with your development process and Python core developers, so please tell me what I should improve... Please subscribe to core-mentorship list and post your question there. I don't understand this response. You seem to be assuming that Sebastian is asking for guidance, I am reading what he posted, which ended with please tell me what I should improve This sort of question-request is routinely posted, in much the same words, on core-mentorship, and routinely gets a response there. The core-mentorship current archive is only available to the list members as stated at the core-mentorship web site at http://pythonmentors.com/. Curious to know why. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] f_lineno accessors
In issues 7238 [1], 16482 [2], 17697 [3] and 17277 [4], the line number may be incorrect when the global trace function has been removed but not the frame f_trace function. A simple test (see below) in issue 17288 [5] crashes the interpreter when setting f_lineno in a generator from a return trace function. All those issues are few months old. There is a patch at issue 17277 [4] that fixes the first 4 issues. There is also a patch for issue 17288 [5]. ## Setting f_lineno in a generator ## $ cat jump.py def gen(): for i in range(1): yield i lineno = 4 for i in gen(): pass $ python -m pdb jump.py /tmp/jump.py(1)module() - def gen(): (Pdb) import sys; print(sys.version) 3.4.0a1+ (default:975d1e180689, Sep 6 2013, 09:26:12) [GCC 4.3.2] (Pdb) break 3 Breakpoint 1 at /tmp/jump.py:3 (Pdb) continue /tmp/jump.py(3)gen() - yield i (Pdb) step --Return-- /tmp/jump.py(3)gen()-0 - yield i (Pdb) jump 2 /tmp/jump.py(2)gen()-0 - for i in range(1): (Pdb) continue Segmentation fault [1] http://bugs.python.org/issue7238 [2] http://bugs.python.org/issue16482 [3] http://bugs.python.org/issue17697 [4] http://bugs.python.org/issue17277 [5] http://bugs.python.org/issue17288 Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Test the test suite?
It happens that few tests are also never run because of name conflicts. See issue 16056. Xavier On Wed, Aug 28, 2013 at 1:37 PM, Victor Stinner victor.stin...@gmail.com wrote: Hi, I just noticed that tests using @requires_freebsd_version and @requires_linux_version decorator from test.support are never run since this commit (almost 2 years ago): changeset: 72618:3b1859f80e6d user:Charles-François Natali neolo...@free.fr date:Mon Oct 03 19:40:37 2011 +0200 files: Lib/test/support.py description: Introduce support.requires_freebsd_version decorator. ... raise unittest.SkipTest( -Linux kernel %s or higher required, not %s -% (min_version_txt, version_txt)) -return func(*args, **kw) -wrapper.min_version = min_version +%s version %s or higher required, not %s +% (sysname, min_version_txt, version_txt)) I don't want to blame Charles-François, nobody saw the issue during 2 years! No, my question is: how can we detect that a test is never run? Do we need test covertage on the test suite? Or inject faults in the code to test the test suite? Any other idea? I fixed the decorators in Python 3.3 (84debb4abd50) and 3.4 (f98fd5712b0e). Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/xdegaye%40gmail.com -- Xavier Les Chemins de Lokoti: http://lokoti.alwaysdata.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can we triple quoted string as a comment?
On Tue, Mar 26, 2013 at 2:28 PM, R. David Murray wrote: On Mon, 25 Mar 2013 18:16:47 -0700, Raymond Hettinger wrote: If you're editing with Emacs, it is really easy to reflow paragraphs and to insert or remove multiline comments each prefixed with #. But with other editors, it can be a PITA and a multiline string is the easiest to maintain and works well when cutting-and-pasting the comments from somewhere else. Just FYI it is also very easy in vim: gq plus whatever movement prefix suits the situation. And to comment out multiple lines in vim, each prefixed with #, see :help v_b_I and :help v_b_I_example. Xavier ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The step command of pdb is broken
On Mon, Apr 30, 2012 at 12:31 PM, Xavier de Gaye wrote: Issue http://bugs.python.org/issue13183 raises the point that the step command of pdb is broken. This issue is 6 months old. A patch and test case have been proposed. Other pdb commands are also broken for the same reason (no trace function setup in the targeted caller frame). A new http://bugs.python.org/issue14728 has been submitted with a proposed patch for these commands and the corresponding test cases. The patch removes a while loop from the fast path, and that should also provide an improvement of the performance of Pdb. Xavier ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] The step command of pdb is broken
Issue http://bugs.python.org/issue13183 raises the point that the step
command of pdb is broken. This issue is 6 months old. A patch and test
case have been proposed. The 'Lifecycle of a Patch' at
http://docs.python.org/devguide/patch.html says
quote
If your patch has not received any notice from reviewers (i.e., no
comment made) after a substantial amount of time then you may email
python-dev@python.org asking for someone to take a look at your patch.
/quote
I am the author of pyclewn, a Vim front end to pdb and gdb, and I
would be grateful for any progress on this issue.
The following pdb session shows the problem when running the three
modules main.py, foo.py and bar.py. After the second step command, pdb
does not stop (as it should) at lines foo.py:5 and foo.py:6, nor does
it stop to print the return value of increment().
=
main.py
1 import foo
2
3 result = foo.increment(100)
4 print('result', result)
foo.py
1 import bar
2
3 def increment(arg):
4 v = bar.value()
5 result = arg + v
6 return result
bar.py
1 def value():
2 return 5
=
$ python -m pdb main.py
/path_to/main.py(1)module()
- import foo
(Pdb) import sys; sys.version
'3.3.0a2+ (default:2c27093fd11f, Apr 30 2012, 10:51:35) \n[GCC 4.3.2]'
(Pdb) break bar.py:2
Breakpoint 1 at /path_to/bar.py:2
(Pdb) continue
/path_to/bar.py(2)value()
- return 5
(Pdb) step
--Return--
/path_to/bar.py(2)value()-5
- return 5
(Pdb) step
/path_to/main.py(4)module()
- print('result', result)
(Pdb)
=
Xavier
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The step command of pdb is broken
On Mon, Apr 30, 2012 at 6:57 PM, Senthil Kumaran wrote: Irrespective of this - Issue13183 seems to be an easy to verify bug in 3.2 and 3.3. I think, it would most visible if you were to use a full screen debugger and you will notice that the return call indicator has jumped to the next statement (skipping return) when returning. I guess, that's why Xavier (pyclewn author) noted it. The fix seems fine too. I have just requested an additional info and this particular one could be fixed. Thanks for your help on this issue Senthil. Xavier ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
