Re: [Distutils] Handling the binary dependency management problem
On 6 December 2013 17:10, Thomas Heller thel...@ctypes.org wrote: Am 06.12.2013 06:47, schrieb Nick Coghlan: Hmm, I just had an idea for how to do the runtime selection thing. It actually shouldn't be that hard, so long as the numpy folks are OK with a bit of __path__ manipulation in package __init__ modules. Manipulation of __path__ at runtime usually makes it harder for modulefinder to find all the required modules. Not usually, always. That's why http://docs.python.org/2/library/modulefinder#modulefinder.AddPackagePath exists :) However, the interesting problem in this case is that we want to package 3 different versions of the modules, choosing one of them at runtime, and modulefinder definitely *won't* cope with that. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On 6 December 2013 17:21, Ralf Gommers ralf.gomm...@gmail.com wrote: On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan ncogh...@gmail.com wrote: With that approach, the existing wheel model would work (no need for a variant system), and numpy installations could be freely moved between machines (or shared via a network directory). Hmm, taking a compile flag and encoding it in the package layout seems like a fundamentally wrong approach. And in order to not litter the source tree and all installs with lots of empty dirs, the changes to __init__.py will have to be made at build time based on whether you're building Windows binaries or something else. Path manipulation is usually fragile as well. So I suspect this is not going to fly. In the absence of the perfect solution (i.e. picking the right variant out of no SSE, SSE2, SSE3 automatically), would it be a reasonable compromise to standardise on SSE2 as lowest acceptable common denominator? Users with no sse capability at all or that wanted to take advantage of the SSE3 optimisations, would need to grab one of the Windows installers or something from conda, but for a lot of users, a pip install numpy that dropped the SSE2 version onto their system would be just fine, and a much lower barrier to entry than well, first install this other packaging system that doesn't interoperate with your OS package manager at all Are we letting perfect be the enemy of better, here? (punting on the question for 6 months and seeing if we can deal with the install-time variant problem in pip 1.6 is certainly an option, but if we don't *need* to wait that long...) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
How does conda handle SSE vs SSE2 vs SSE3? I’m digging through it’s source code and just installed numpy with it and I can’t seem to find any handling of that? On Dec 6, 2013, at 7:33 AM, Nick Coghlan ncogh...@gmail.com wrote: On 6 December 2013 17:21, Ralf Gommers ralf.gomm...@gmail.com wrote: On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan ncogh...@gmail.com wrote: With that approach, the existing wheel model would work (no need for a variant system), and numpy installations could be freely moved between machines (or shared via a network directory). Hmm, taking a compile flag and encoding it in the package layout seems like a fundamentally wrong approach. And in order to not litter the source tree and all installs with lots of empty dirs, the changes to __init__.py will have to be made at build time based on whether you're building Windows binaries or something else. Path manipulation is usually fragile as well. So I suspect this is not going to fly. In the absence of the perfect solution (i.e. picking the right variant out of no SSE, SSE2, SSE3 automatically), would it be a reasonable compromise to standardise on SSE2 as lowest acceptable common denominator? Users with no sse capability at all or that wanted to take advantage of the SSE3 optimisations, would need to grab one of the Windows installers or something from conda, but for a lot of users, a pip install numpy that dropped the SSE2 version onto their system would be just fine, and a much lower barrier to entry than well, first install this other packaging system that doesn't interoperate with your OS package manager at all Are we letting perfect be the enemy of better, here? (punting on the question for 6 months and seeing if we can deal with the install-time variant problem in pip 1.6 is certainly an option, but if we don't *need* to wait that long...) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 5:47 AM, Nick Coghlan ncogh...@gmail.com wrote: On 6 December 2013 11:52, Donald Stufft don...@stufft.io wrote: On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: What would really be best is run-time selection of the appropriate lib -- it would solve this problem, and allow users to re-distribute working binaries via py2exe, etc. And not require opening a security hole in wheels... Not sure how hard that would be to do, though. Install time selectors probably isn’t a huge deal as long as there’s a way to force a particular variant to install and to disable the executing code. Hmm, I just had an idea for how to do the runtime selection thing. It actually shouldn't be that hard, so long as the numpy folks are OK with a bit of __path__ manipulation in package __init__ modules. As Ralf, I think it is overkill. The problem of SSE vs non SSE is because of one library, ATLAS, which as IMO the design flaw of being arch specific. I always hoped we could get away from this when I built those special installers for numpy :) MKL does not have this issue, and now that openblas (under a BSD license) can be used as well, we can alleviate this for deployment. Building a deployment story for this is not justified. David Specifically, what could be done is this: - all of the built SSE level dependent modules would move out of their current package directories into a suitable named subdirectory (say _nosse, _sse2, _sse3) - in the __init__.py file for each affected subpackage, you would have a snippet like: numpy._add_sse_subdir(__path__) where _add_sse_subdir would be something like: def _add_sse_subdir(search_path): if len(search_path) 1: return # Assume the SSE dependent dir has already been added # Could likely do this SSE availability check once at import time if _have_sse3(): sub_dir = _sse3 elif _have_sse2(): sub_dir = _sse2 else: sub_dir = _nosse main_dir = search_path[0] search_path.append(os.path.join(main_dir, sub_dir) With that approach, the existing wheel model would work (no need for a variant system), and numpy installations could be freely moved between machines (or shared via a network directory). To avoid having the implicit namespace packages in 3.3+ cause any problems with this approach, the SSE subdirectories should contain __init__.py files that explicitly raise ImportError. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 12:44 PM, Donald Stufft don...@stufft.io wrote: How does conda handle SSE vs SSE2 vs SSE3? I’m digging through it’s source code and just installed numpy with it and I can’t seem to find any handling of that? I can't speak for conda, but @enthought, we solve it by using the MKL, which selects the right implementation at runtime. Linux distributions have system to cope with it (the hwcap capabtility of ld), but even there few packages use it. Atlas, libc are the ones I am aware of. And this breaks anyway when you use static linking obviously. David On Dec 6, 2013, at 7:33 AM, Nick Coghlan ncogh...@gmail.com wrote: On 6 December 2013 17:21, Ralf Gommers ralf.gomm...@gmail.com wrote: On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan ncogh...@gmail.com wrote: With that approach, the existing wheel model would work (no need for a variant system), and numpy installations could be freely moved between machines (or shared via a network directory). Hmm, taking a compile flag and encoding it in the package layout seems like a fundamentally wrong approach. And in order to not litter the source tree and all installs with lots of empty dirs, the changes to __init__.py will have to be made at build time based on whether you're building Windows binaries or something else. Path manipulation is usually fragile as well. So I suspect this is not going to fly. In the absence of the perfect solution (i.e. picking the right variant out of no SSE, SSE2, SSE3 automatically), would it be a reasonable compromise to standardise on SSE2 as lowest acceptable common denominator? Users with no sse capability at all or that wanted to take advantage of the SSE3 optimisations, would need to grab one of the Windows installers or something from conda, but for a lot of users, a pip install numpy that dropped the SSE2 version onto their system would be just fine, and a much lower barrier to entry than well, first install this other packaging system that doesn't interoperate with your OS package manager at all Are we letting perfect be the enemy of better, here? (punting on the question for 6 months and seeing if we can deal with the install-time variant problem in pip 1.6 is certainly an option, but if we don't *need* to wait that long...) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
Am 06.12.2013 13:22, schrieb Nick Coghlan: On 6 December 2013 17:10, Thomas Heller thel...@ctypes.org wrote: Am 06.12.2013 06:47, schrieb Nick Coghlan: Hmm, I just had an idea for how to do the runtime selection thing. It actually shouldn't be that hard, so long as the numpy folks are OK with a bit of __path__ manipulation in package __init__ modules. Manipulation of __path__ at runtime usually makes it harder for modulefinder to find all the required modules. Not usually, always. That's why http://docs.python.org/2/library/modulefinder#modulefinder.AddPackagePath exists :) Well, as the py2exe author and the (inactive, I admit) modulefinder maintainer I already know this. However, the interesting problem in this case is that we want to package 3 different versions of the modules, choosing one of them at runtime, and modulefinder definitely *won't* cope with that. The new importlib implementation in python3.3 offers a lot a new possibilities, probably not all of them have been explored yet. For example, I have written a ModuleMapper object that, when inserted into sys.meta_path, allows transparent mapping of module names between Python2 and Python3 - no need to use six. And the new modulefinder(*) that I've written works great with that. Thomas (*) which will be part of py2exe for python3, but it is too late for python3.4. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On 6 December 2013 13:06, David Cournapeau courn...@gmail.com wrote: As Ralf, I think it is overkill. The problem of SSE vs non SSE is because of one library, ATLAS, which as IMO the design flaw of being arch specific. I always hoped we could get away from this when I built those special installers for numpy :) MKL does not have this issue, and now that openblas (under a BSD license) can be used as well, we can alleviate this for deployment. Building a deployment story for this is not justified. Oh, okay that's great. How hard would it be to get openblas numpy wheels up and running? Would they be compatible with the existing scipy etc. binaries? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Binary dependency management, round 2 :)
On 4 December 2013 21:10, Nick Coghlan ncogh...@gmail.com wrote: == Regarding conda == In terms of providing an answer to the question Where does conda fit in the scheme of packaging tools?, my conclusion from the thread is that once a couple of security related issues are fixed (think PyPI before the rubygems.org compromise for the current state of conda's security model), and once the Python 3.3 compatibility issue is addressed on Windows, it would be reasonable to recommend it as one of the current options for getting hold of pre-built versions of the scientific Python stack. I think this is important enough to warrant a NumPy and the Scientific Python stack section in the user guide (with Linux distro packages, Windows installers and conda all discussed as options): https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific I created a draft of this new section at https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Binary dependency management, round 2 :)
On 6 December 2013 13:54, Nick Coghlan ncogh...@gmail.com wrote: On 4 December 2013 21:10, Nick Coghlan ncogh...@gmail.com wrote: == Regarding conda == In terms of providing an answer to the question Where does conda fit in the scheme of packaging tools?, my conclusion from the thread is that once a couple of security related issues are fixed (think PyPI before the rubygems.org compromise for the current state of conda's security model), and once the Python 3.3 compatibility issue is addressed on Windows, it would be reasonable to recommend it as one of the current options for getting hold of pre-built versions of the scientific Python stack. I think this is important enough to warrant a NumPy and the Scientific Python stack section in the user guide (with Linux distro packages, Windows installers and conda all discussed as options): https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific I created a draft of this new section at https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff It's probably worth listing each of the full scientific Python distributions on this page (or just linking to it), rather than just Anaconda: http://www.scipy.org/install.html Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Binary dependency management, round 2 :)
On 7 December 2013 00:05, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 6 December 2013 13:54, Nick Coghlan ncogh...@gmail.com wrote: On 4 December 2013 21:10, Nick Coghlan ncogh...@gmail.com wrote: I think this is important enough to warrant a NumPy and the Scientific Python stack section in the user guide (with Linux distro packages, Windows installers and conda all discussed as options): https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific I created a draft of this new section at https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff It's probably worth listing each of the full scientific Python distributions on this page (or just linking to it), rather than just Anaconda: http://www.scipy.org/install.html Done! I added a new section on SciPy distributions and retitled to the conda section to emphasise the open source package manager over the distribution (since the main point of that section relates to the fact that you can use conda *outside* Anaconda, even if there are still some rough edges to that approach at this point in time). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Thu, Dec 5, 2013 at 11:21 PM, Ralf Gommers ralf.gomm...@gmail.comwrote: Hmm, taking a compile flag and encoding it in the package layout seems like a fundamentally wrong approach. well, it's pretty ugly hack, but sometimes an ugly hack that does the job is better than nothing. IIUC, the Intel MKL libs do some sort of dynamic switching at run time too -- and that is a great feature. And in order to not litter the source tree and all installs with lots of empty dirs, where lots what, 3? Is that so bad in a project the size of numpy? the changes to __init__.py will have to be made at build time based on whether you're building Windows binaries or something else. That might in fact be nicer than the litter, but also may be a less robust and more annoying way to do it. Path manipulation is usually fragile as well. My first instinct was that you'd re-name directories on the fly, which might be more robust, but wouldn't work in any kind of secure environment. so a no-go. But could you elaborate on the fragile nature of sys.path manipulation? What might go wrong there? Also, it's not out of the question that once such a system was in place, that it could be used on systems other than Windows -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 1:33 PM, Nick Coghlan ncogh...@gmail.com wrote: On 6 December 2013 17:21, Ralf Gommers ralf.gomm...@gmail.com wrote: On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan ncogh...@gmail.com wrote: With that approach, the existing wheel model would work (no need for a variant system), and numpy installations could be freely moved between machines (or shared via a network directory). Hmm, taking a compile flag and encoding it in the package layout seems like a fundamentally wrong approach. And in order to not litter the source tree and all installs with lots of empty dirs, the changes to __init__.py will have to be made at build time based on whether you're building Windows binaries or something else. Path manipulation is usually fragile as well. So I suspect this is not going to fly. In the absence of the perfect solution (i.e. picking the right variant out of no SSE, SSE2, SSE3 automatically), would it be a reasonable compromise to standardise on SSE2 as lowest acceptable common denominator? Maybe, yes. It's hard to figure out the impact of this, but I'll bring it up on the numpy list. If no one has a good way to get some statistics on cpu's that don't support these instruction sets, it may be worth a try for one of the Python versions and see how many users will run into the issue. On accident we've released an incorrect binary once before by the way (scipy 0.8.0 for Python 2.5) and that was a problem fairly quickly: https://github.com/scipy/scipy/issues/1697. That was 2010 though. Users with no sse capability at all or that wanted to take advantage of the SSE3 optimisations, would need to grab one of the Windows installers or something from conda, but for a lot of users, a pip install numpy that dropped the SSE2 version onto their system would be just fine, and a much lower barrier to entry than well, first install this other packaging system that doesn't interoperate with your OS package manager at all Well, for most Windows users grabbing a .exe and clicking on it is a lower barrier that opening a console and typing pip install numpy:) Are we letting perfect be the enemy of better, here? (punting on the question for 6 months and seeing if we can deal with the install-time variant problem in pip 1.6 is certainly an option, but if we don't *need* to wait that long...) Let's first get the OS X wheels up, that can be done now. And then see what is decided on the numpy list for the compromise you propose above. Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 2:48 PM, Oscar Benjamin oscar.j.benja...@gmail.comwrote: On 6 December 2013 13:06, David Cournapeau courn...@gmail.com wrote: As Ralf, I think it is overkill. The problem of SSE vs non SSE is because of one library, ATLAS, which as IMO the design flaw of being arch specific. I always hoped we could get away from this when I built those special installers for numpy :) MKL does not have this issue, and now that openblas (under a BSD license) can be used as well, we can alleviate this for deployment. Building a deployment story for this is not justified. Oh, okay that's great. How hard would it be to get openblas numpy wheels up and running? Would they be compatible with the existing scipy etc. binaries? OpenBLAS is still pretty buggy compared to ATLAS (although performance in many cases seems to be on par); I don't think that will be well received for the official releases. We actually did discuss it as an alternative for Accelerate on OS X, but there was quite a bit of opposition. Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Binary dependency management, round 2 :)
On Fri, Dec 6, 2013 at 6:22 AM, Nick Coghlan ncogh...@gmail.com wrote: I created a draft of this new section at https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff looks good, thanks! ONe note: In particular, bootstrapping conda via ``pip install conda`` and then running the ``conda init`` command provides access to all of the pre-built binaries that Continuum Analytics have created for the free version of the Anaconda distribution. I've been chatting off list with Travis, and while it does appear that Anaconda is more compatibly with the python.org installers than I had thought, there are still some pretty significant rough edges, particularly with the OS-X build. But it does look pretty promising. Travis pointed out that I had been pretty critical of endorsing conda in this thread. He is right. That came from two things: 1) years of frustration with the mess of python packaging, that I thought we were finally resolving with binary wheels. 2) recent bad experience with Anaconda and teaching python to newbies. I'm pretty sure we all want one way to do it -- i.e. we can just tell people to install a python.org build, then use pip install to get everything else, than may simply not be practical, and apparently we're not as close as I thought. But as I think back over the last few years, I realize that I've been recommending the python.org binaries across the board, because it is the basis of a number of different package approaches. So I can tell my students: - try pip install - if that doesn't work, look in the Gohlke repo - if that doesn't work, look for a binary on the package web page. - if that doesn't work, follow the build instructions on the package web page And while it would be great if pip install always worked, this really isn't so bad, and it's not so bad because, in fact, most of the maintainers of complex packages have been targeting python.org binaries for years. So if we can't have wheels for everything, we can do pretty well if the other options (e.g. conda) are python.org compatible. That's not quite the case for Anaconda now, but it's not that far off, and Travis is interested in making it better. I'm not going to have time to poke at this myself for at least a few weeks, but at some point, maybe we can, for instance, try to convert the wxPython binaries to a conda package, or just figure out a post-install hack that will let it work with Anaconda (the current wxPython binaries for OS-X already have *.pth file magic to let both the python.org and Apple binaries use the same package). So nice work all -- it seem this is not getting as ugly as I feared. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 4:33 AM, Nick Coghlan ncogh...@gmail.com wrote: In the absence of the perfect solution (i.e. picking the right variant out of no SSE, SSE2, SSE3 automatically), would it be a reasonable compromise to standardise on SSE2 as lowest acceptable common denominator? +1 Users with no sse capability at all or that wanted to take advantage of the SSE3 optimisations, would need to grab one of the Windows installers or something from conda, but for a lot of users, a pip install numpy that dropped the SSE2 version onto their system would be just fine, and a much lower barrier to entry than well, first install this other packaging system that doesn't interoperate with your OS package manager at all exactly -- for example, I work with a web dev that could really use Matplotlib for a little task -- if I could tell him to pip install matplotlib, he's do it, but he just sees it as too much hassle at the point... Are we letting perfect be the enemy of better, here? I think so, yes. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 5:50 PM, Chris Barker chris.bar...@noaa.gov wrote: On Fri, Dec 6, 2013 at 5:06 AM, David Cournapeau courn...@gmail.comwrote: As Ralf, I think it is overkill. The problem of SSE vs non SSE is because of one library, ATLAS, which as IMO the design flaw of being arch specific. yup -- really designed for the end user to built it themselves MKL does not have this issue, and now that openblas (under a BSD license) can be used as well, we can alleviate this for deployment. Building a deployment story for this is not justified. So Openblas has run-time selection of the right binary? very cool! So are we done here? Not that I know of, but you can easily build one for a given architecture, which is essentially impossible to do with Atlas reliably. I did not know about openblas instabilities, though. I guess we will have to do some more testing. David -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 5:06 AM, David Cournapeau courn...@gmail.com wrote: As Ralf, I think it is overkill. The problem of SSE vs non SSE is because of one library, ATLAS, which as IMO the design flaw of being arch specific. yup -- really designed for the end user to built it themselves MKL does not have this issue, and now that openblas (under a BSD license) can be used as well, we can alleviate this for deployment. Building a deployment story for this is not justified. So Openblas has run-time selection of the right binary? very cool! So are we done here? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling the binary dependency management problem
On Fri, Dec 6, 2013 at 5:16 AM, Thomas Heller thel...@ctypes.org wrote: Am 06.12.2013 13:22, schrieb Nick Coghlan: Manipulation of __path__ at runtime usually makes it harder for modulefinder to find all the required modules. Not usually, always. That's why http://docs.python.org/2/library/modulefinder#modulefinder.AddPackagePath exists :) Well, as the py2exe author and the (inactive, I admit) modulefinder maintainer I already know this. modulefinder fails often enough that Ive never been able ot package a non-trivial app without a bit of force-include all of this package, (and don't-include this other thing!). So while too bad, this should not be considered deal breaker. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] More control over uploading an sdist and egg to PyPI.
We use setuptools for our library. I'm trying to figure out a way to upload an sdist and an egg to PyPI without having to put my PyPI credentials in a file. Please let me know if this is not the right mailing list for this question. Our setup is that the entire team has access to the source code repository, but only the release managers have access to the credentials needed to make releases to PyPI (they're in a fake home directory). To do a release, you run: cd project-dir HOME=/release-manager-secrets/fake-pypi-home-dir \ python setup.py bdist_egg sdist upload This works ok, but we would prefer to not have our PyPI credentials on the filesystem (unencrypted) at all. I'd like to run setup.py bdist_egg sdist first and then run a separate command that uploaded the egg and sdist. This command would ask for my credentials on stdin. How can I write such a program? Is there a PyPI HTTP API I can use? Does setuptools have a public interface that I can call? Thanks! - Kannan ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Binary dependency management, round 2 :)
On 7 December 2013 03:44, Chris Barker chris.bar...@noaa.gov wrote: On Fri, Dec 6, 2013 at 6:22 AM, Nick Coghlan ncogh...@gmail.com wrote: I created a draft of this new section at https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff looks good, thanks! With some additional suggestions from Oscar Benjamin and Ned Deilly, this is now live on the user guide: https://python-packaging-user-guide.readthedocs.org/en/latest/additional.html#installing-scientific-packages ONe note: In particular, bootstrapping conda via ``pip install conda`` and then running the ``conda init`` command provides access to all of the pre-built binaries that Continuum Analytics have created for the free version of the Anaconda distribution. I've been chatting off list with Travis, and while it does appear that Anaconda is more compatibly with the python.org installers than I had thought, there are still some pretty significant rough edges, particularly with the OS-X build. But it does look pretty promising. Travis pointed out that I had been pretty critical of endorsing conda in this thread. He is right. That came from two things: 1) years of frustration with the mess of python packaging, that I thought we were finally resolving with binary wheels. We are, but I think there's still a great deal of value in *curated* platforms. PyPI works rather well for a relatively anarchic free-for-all, but we aren't going to completely get rid of the case of somebody depending on another package that has a binary extension, but doesn't publish a wheel file until we have an integrated build farm that generates wheels from uploaded sdists automatically (such a thing is definitely on the long term wish list, but there are a great many higher priority problems ahead of it on the todo list). 2) recent bad experience with Anaconda and teaching python to newbies. I'm pretty sure we all want one way to do it -- i.e. we can just tell people to install a python.org build, then use pip install to get everything else, than may simply not be practical, and apparently we're not as close as I thought. If we can resolve the numpy situation (either with OpenBLAS, or by nominating ATLAS SSE2 as good enough for now), then we'll be in a pretty good place for Windows and Mac OS X. Other *nix systems will still be a challenge, since there may be more C ABI variations to cover, and the current definition of the platform tag doesn't do a good job of handling variation across Linux distros. But as I think back over the last few years, I realize that I've been recommending the python.org binaries across the board, because it is the basis of a number of different package approaches. So I can tell my students: - try pip install - if that doesn't work, look in the Gohlke repo - if that doesn't work, look for a binary on the package web page. - if that doesn't work, follow the build instructions on the package web page And while it would be great if pip install always worked, this really isn't so bad, and it's not so bad because, in fact, most of the maintainers of complex packages have been targeting python.org binaries for years. So if we can't have wheels for everything, we can do pretty well if the other options (e.g. conda) are python.org compatible. That's not quite the case for Anaconda now, but it's not that far off, and Travis is interested in making it better. The Anaconda folks just pushed a new release today which should fix the binary extension compatibility issue for 3.3 on Windows. I'm not aware of the situation on Mac OS X, but they were quite responsive to the ticket Paul filed about the compatibility problem on Windows. I'm not going to have time to poke at this myself for at least a few weeks, but at some point, maybe we can, for instance, try to convert the wxPython binaries to a conda package, or just figure out a post-install hack that will let it work with Anaconda (the current wxPython binaries for OS-X already have *.pth file magic to let both the python.org and Apple binaries use the same package). I believe the conda folks are also working on automatically falling back to pip - if conda doesn't find a package in the conda repos, use pip to go hunting on PyPI instead. With the PyPI package count outnumbering the curated Anaconda package list by a few orders of magnitude (~34k vs a hundred or so) such an approach should resolve a lot of conda install doesn't work for package X issues :) So nice work all -- it seem this is not getting as ugly as I feared. Yeah, I see a great deal of potential benefit to users in conda as a cross platform platform, so from an upstream tooling perspective, I think it makes sense for us to treat it similarly to how we treat other platforms. The main difference is that there's additional integration in the other direction in the conda case. I think these threads have also made it clear
