Re: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ...
Test if the provided system C/C++ compiler supports OpenMP and do not use the openmp compiler flag to fallback to a sequential build if not. -- Olivier Grisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
2016-01-22 3:47 GMT+01:00 Chris Barker - NOAA Federal : > > Maybe the community will spring forth and do that -- I'm skeptical because I > tried to to that for years for OS-X and it was just too much to do. And the > infrastructure was there. > > Before pip and wheel there were mpkgs on OS-X, and repo's of toms for Linux > years before that -- but always the result of a couple people's heroic > efforts. > > Maybe the infrastructure has improved, and the community grown enough, that > this will all work. We'll see. I think the infrastructure has improved. For instance I invested some time and effort to provide a template configuration to build C/C++ compiled extensions for windows wheels on top of the free AppVeyor.com CI platform [1]. Since then this build configuration has been integrated in the python packaging documentation [2] and I had the opportunity to present that work at PyCon [3] (and PyCon FR) last year. Now the community of project maintainers has picked it up. I can count more than 300 projects using this build setup on github. I have very little work to do to help them maintain that infra-strucuture nowadays. Even the configuration upgrade to make it work with MSVC 2015 / Python 3.5 was contributed to my repo before I could find the time to investigate the issue my-self. My point is that once we have clearly defined best-practices for packaging and convenient tools to build the packages automatically and test that they work as expected (e.g. free hosted CI that support running an old centos-based docker container), I am rather confident that the community will do the work. It's mostly a matter of providing good tools and education resources (documentation and example configuration templates). [1] https://github.com/ogrisel/python-appveyor-demo/ [2] https://www.appveyor.com/docs/packaging-artifacts [3] https://www.youtube.com/watch?v=d-p6lJJObLU -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 513: A Platform Tag for Portable Linux Built Distributions Version
Maybe we could devise some syntax for /etc/python/compatibility.conf to state that the manylinux1 entry is only valid for Python interpreters such as distutils.util.get_platform() == 'linux-x86_64'? -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 513: A Platform Tag for Portable Linux Built Distributions Version
2016-01-26 11:41 GMT+01:00 Olivier Grisel : > Maybe we could devise some syntax for /etc/python/compatibility.conf > to state that the manylinux1 entry is only valid for Python > interpreters such as distutils.util.get_platform() == 'linux-x86_64'? Actually this won't work. I just try to debootstrap ubuntu trusty i386 on top of ubuntu trusty amd64 and I get the following behavior in python3 inside the i386 chroot: >>> from distutils.util import get_platform >>> get_platform() 'linux-x86_64' >>> import platform >>> platform.machine() 'x86_64' >>> import sys >>> sys.maxsize > 2**32 False So this is actually a 32 bit Python declared as running on a linux-x86_64 platform (even though everything is i386 inside the chroot)... I get the exact same behavior when installing the 32 bit miniconda on ubuntu trusty amd64. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
Strong +1 for all Donald's proposals. -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Re: Opinions on requiring younger glibc in manylinux1 wheel?
> I would say there's value in having two official manylinux flavors at once, for example manylinux2010 for maximum compatibility (it's already 8 years old as far as requirements go!) and manylinux2016 for recent systems compatibility. Later, manylinux2022 gets released as the "recent systems compatibility" standard and manylinux2016 becomes the "maximum compatibility" flavor. That's an interesting proposition. Would pip be able to automatically select the most recent compatible wheel when two are available on PyPI? -- Olivier -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KT4IWSHYRF2MWRHSXQE3OYDY6RV4TWCJ/
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
It seems to work, I merged the change in the master of python-appveyor-demo. Thanks! -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
2014-09-24 23:52 GMT+02:00 Paul Moore : > On 24 September 2014 03:45, Jonathan J. Helmus wrote: >> Some of us from the Scientific Python side of development have been >> using appveyor to build Windows wheels for a few projects. A demo from one >> of developers of scikit-learn gives a good overview of the process we have >> been using [1]. > > This is excellent. Many thanks for the pointer - you've clearly > managed to solve some of the more annoying problems that I have been > hitting. (I'd claim that I was getting there, but you've saved me the > effort :-)) > > One thing I have done is request the Appveyor team to add 64-bit > Pythons to their build environments, which they have done, so that now > there should be no need to install your own copy of Python (at least > for 2.7, 3.3 and 3.4). Under which path? Could you please issue a PR to: https://github.com/ogrisel/python-appveyor-demo to show how to leverage pre-installed versions of Python? > I've copied Olivier in here as the author of the demo project, but > would you mind if I used this as the basis of a document covering how > to build wheels for your project using Appveyor? Obviously, I'd give > you full credit. I'm thinking of including it as a section in the > Python packaging guide, or maybe as a separate HOWTO document. Feel free to reuse any of my work for your document. The license of the scripts in python-appveyor-demo is CC0, no attribution required. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
2014-09-25 0:09 GMT+02:00 Paul Moore : > On 24 September 2014 22:58, Olivier Grisel wrote: >> Under which path? > > It's now documented in > http://www.appveyor.com/docs/installed-software, but C:\PythonXY and > C:\PythonXY-x64. Nice, thanks: I will try it now. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
Thank you very Steve for pushing that installer out, this is very appreciated. What is the story for project maintainers who want to also support Python 3.3+ (for 32 bit and 64 bit python) for their project with binary wheels for windows? At the moment it's possible to use the Windows SDK as documented here: http://scikit-learn.org/dev/install.html#building-on-windows However getting VC Express + Windows SDK is hard and slow to setup and cannot be scripted in a CI environment. In the mean time, it's possible to use CI environments that already feature all the necessary versions of the VC compilers and libraries such as appveyor.com, see this demo project: https://github.com/ogrisel/python-appveyor-demo https://ci.appveyor.com/project/ogrisel/python-appveyor-demo -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
2014-09-30 18:07 GMT+02:00 Steve Dower : > Paul Moore wrote: >> On 30 September 2014 16:56, Olivier Grisel wrote: >>> What is the story for project maintainers who want to also support >>> Python 3.3+ (for 32 bit and 64 bit python) for their project with >>> binary wheels for windows? >> >> It would be so easy at this point to ask "What's the chance of a similarly >> packaged >> version of VS2010 for Python 3.2/3.3/3.4?" But I really don't want Steve to >> get into >> any trouble with people saying "now look what you've started" :-) > > :-) > > The answer is basically no chance - the slippery slope was considered and > shut down. > > If VC14 slips significantly and we have to stick with VC10 for Python 3.5, > then I'll make the case again and see what we get, but for now the future > story is to upgrade. Luckily, 3.3->3.5 is not going to be as hard as 2.7->3.5. Alright, sounds good. Thanks again. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making a wheel platform-specific?
2014-10-28 14:20 GMT+01:00 Antoine Pitrou : > > If I rename the file manually, is there an easy (CLI-based) way of > uploading it to PyPI? I had the same issue for a tool that collects wheels generated by various CI platforms to prepare a binary release. https://github.com/ogrisel/wheelhouse-uploader Here is the hack I did (a distutils command extension): https://github.com/ogrisel/wheelhouse-uploader/blob/master/wheelhouse_uploader/cmd.py#L54 -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP425 - Wheel Binary Package Compatibility
2014-10-28 15:04 GMT+01:00 Paul Moore : > On 28 October 2014 10:13, Matthias Hafner wrote: > > (in reverse order, easiest to hardest :-)) > >> Why does that work on MacOS, btw? Are all library versions fixed there for >> one version of OSX? > > I believe (I'm not 100% sure as I'm a Windows user not an OSX user) > that PyPI only supports binaries compatible with the official > python.org binaries of Python. So library version decisions are moot. > This means that homebrew or "the other ones" (sorry :-)) users can't > avail themselves of PyPI-hosted binaries yet, until the wider library > compatibility issues are resolved. Homebrew-installed Python and the system Python that comes by default on OSX are binary compatible with wheels generated with Python from the official OSX installer from python.org. In current versions of pip you "just" have to rename the wheel filenames to get them picked up from PyPI on those platforms. This explains the strange names of the OSX wheels for numpy for instance: numpy-1.9.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl numpy-1.9.0-cp33-cp33m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl numpy-1.9.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl https://pypi.python.org/pypi/numpy macosx_10_9_x86_64 is the platform tag used by Homebrew-installed Python while macosx_10_6_intel is the platform tag of the python.org Python. Off-course if your python extension link to non-system libraries, you have to embed them in the wheel, for instance using: https://pypi.python.org/pypi/delocate -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP425 - Wheel Binary Package Compatibility
Here is a test matrix that checks binary compatibility of OSX wheels for many projects of the SciPy stack: https://github.com/MacPython/scipy-stack-osx-testing -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP425 - Wheel Binary Package Compatibility
2014-10-28 16:22 GMT+01:00 Olivier Grisel : > Here is a test matrix that checks binary compatibility of OSX wheels > for many projects of the SciPy stack: > > https://github.com/MacPython/scipy-stack-osx-testing And here are the results: https://travis-ci.org/MacPython/scipy-stack-osx-testing -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files
+1 to the idea in general. as well. Here is another use case to motivate distro-aware platform tags for linux wheels: speeding up continuous integration. Many Python projects related to the SciPy ecosystem by sharing use the following rackspace cloud storage container to mutualize build artifacts: http://wheels.scikit-image.org/ This makes it really easy to setup build chains to test libraries with against recent (or even development) version of they dependencies that cannot be apt-get installed on travis hosts for instance. Building from source is often not an option for a project that have a dependency on scipy for instance, the scipy build time itself would eat most of the travis job allowed time and would not leave enough time. And it would be a complete waste of resource to rebuild scipy over and over again. Unfortunately, all the wheels you find on http://wheels.scikit-image.org with the "linux_x86_64" are only expected to work on the Ubuntu Precise platform (which is used by travis). This ambiguous platform tag is really annoying: - it prevents people who would like to setup CI for other linux platforms (e.g. docker based CI engines that would test wheels on redhat or centos) to share the same distribution infrastructure, - the day travis changes its base distro we will have no way to detect which wheels were built on the old distro and which wheels are built on the newer version of the distro. Note that those wheels are not meant to be used by our end-users, only by CI tools for testing or by developers who would like to quickly reproduce without having to rebuild everything from scratch. If the experiment show that such distro tagged wheels are actually as stable as their are supposed too (which can be mostly tested via CI), then we could discuss later about the opportunity to distribute Linux wheels on PyPI for the end users but to me this is not a priority at all. -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Amend PEP 440 with Wider Feedback on Release Candidates
Since PEP 440 was formally accepted in August 2014, would it make sense to add a change log to document the amendment of the PEP, for instance in the appendix (maybe with a link to the diff in hg)? -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pycon] Packaging-related discussions
Which specific room are you in? The board says 512dh. -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pycon] Packaging-related discussions
512f -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: "help needed"
+1 overall to Nick' suggestions. -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Released: pip 7.0 and virtualenv 13.0
Congrats on the release, wheel caching for sdist is a great new feature! -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform
I just checked and indeed the python exec installed by miniconda does not work on Alpine linux (launch via docker from the gliderlabs/alpine image): # ldd /root/miniconda3/pkgs/python-3.4.3-0/bin/python3.4 /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000) libpython3.4m.so.1.0 => /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0 (0x7f26bd153000) libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000) libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000) libutil.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000) libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000) libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f26bd5fe000) Error relocating /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0: __finite: symbol not found Error relocating /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0: __rawmemchr: symbol not found Error relocating /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0: __isinff: symbol not found Error relocating /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0: __isnan: symbol not found Error relocating /root/miniconda3/pkgs/python-3.4.3-0/bin/../lib/libpython3.4m.so.1.0: __isinf: symbol not found We could still have a platform or ABI tag for linux that would include some libc information to ensure that it points to a compatible glibc and provide a reference docker image to build such wheels. We could assume that wheel binary packages should not link to any .so file from the system besides the libc. I think the packages that have specific kernel ABI requirements are rare enough to be explicitly left outside of this first effort and let the user build those from source directly. Is there an easy way to introspect a binary to detect such kernel dependencies? -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
2015-07-17 18:50 GMT+02:00 Marcus Smith : > > >> I think Linux wheel support is almost useless unless the pypa stack >> provides _something_ to handle non-python dependencies. > > > I wouldn't say useless, but I tend to agree with this sentiment. > > I'm thinking the only way to really "compete" with the ease of Conda (for > non-python dependencies) is to shift away from wheels, and instead focus on > making it easier to create native distro packages (i.e. rpm, deb etc...that > can easily depend on non-python dependencies) for python applications, and > moreover that these packages should be "parallel installable" with the > system packages, i.e. they should depend on virtual environments, not the > system python. +1 for being able to work in isolation of the system packages (and without admin rights). This is precisely the killer feature of conda (and virtualenv to some extent): users do not need to rely on interaction with sys admins to get up and running to setup a developer environment. Furthermore they can get as many cheap environments in parallel to develop and reproduce bugs with various versions of libraries or Python it-self. However I don't see why you would not be able to ship your non-Python dependencies as wheels. Surely it should be possible to package stateless libraries like OpenBLAS, libxml/libxsql, llvm runtimes, qt and the like as wheels. Shipping wheels for services such as database servers like postgresql is out of the scope in my opinion. For such admin sys tasks such as managing running stateful services, system packages or docker containers + orchestration are the way to go. Still wheels should be able to address the "setup parallel dev environments" use case. When I say "developer environment" I also include "datascientists environment" that rely on ipython notebook + scipy stack libraries. Best, -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP for dependencies on libraries like BLAS (was: Re: Working toward Linux wheel support)
2015-08-12 20:51 GMT-04:00 Nathaniel Smith : > > I'm a bit surprised to hear that such a PEP is needed. We (= numpy devs) > have actively been making plans to ship a BLAS wheel on windows, and AFAICT > this is totally doable now -- the blocker is windows toolchain issues, not > pypa-related infrastructure. > > Specifically the idea is to have a wheel that contains the shared library as > a regular old data file, plus a stub python package that knows how to find > this data file and how to make it accessible to the linker. So > numpy/__init__.py would start by calling: > > import pyopenblas1 > # on Linux modifies LD_LIBRARY_PATH, > # on Windows uses ctypes to preload... whatever > pyopenblas1.enable() > > and then get on with things, or the build system might do: > > import pyopenblas1 > pyopenblas1.get_header_directories() > pyopenblas1.get_linker_directories() > > This doesn't help if you want to declare dependencies on external, system > managed libraries and have those be automatically somehow provided or > checked for, but to me that sounds like an impossible boil-the-ocean project > anyway, while the above is trivial and should just work. +1 -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributing .c and .pyd modules
You can use the wheel package format to generate platform specific packages for Windows. Have a look at the python packaging documentation: https://packaging.python.org In particular: https://packaging.python.org/en/latest/distributing.html#wheels -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
