Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
2014-09-30 18:07 GMT+02:00 Steve Dower steve.do...@microsoft.com: Paul Moore wrote: On 30 September 2014 16:56, Olivier Grisel olivier.gri...@ensta.org wrote: What is the story for project maintainers who want to also support Python 3.3+ (for 32 bit and 64 bit python) for their project with binary wheels for windows? It would be so easy at this point to ask What's the chance of a similarly packaged version of VS2010 for Python 3.2/3.3/3.4? But I really don't want Steve to get into any trouble with people saying now look what you've started :-) :-) The answer is basically no chance - the slippery slope was considered and shut down. If VC14 slips significantly and we have to stick with VC10 for Python 3.5, then I'll make the case again and see what we get, but for now the future story is to upgrade. Luckily, 3.3-3.5 is not going to be as hard as 2.7-3.5. Alright, sounds good. Thanks again. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] advice re: packaging tasks
Hi Chris, On 09/30/2014 05:44 PM, Chris Jerdonek wrote: 2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of? Perhaps this is stating the obvious, but the simplest solution here is just to switch your README to rst format. It's not that different from Markdown, GitHub/BitBucket will parse it and display it nicely, etc. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
Thank you all for the precious info. Here are my observations: - We are merely writing extension modules with third party dependant code packaged in a dll. In my mind, this use case is not the exception, and would not necessarily warrant the use of a full blown solution like conda. Our deployed environments are self-contained. - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). - On the other hand, if you run pip wheel on the same package, the dlls are not placed in the archive. Is this a pip bug ? In an ideal world, the scripts directory would be called bin, like the unix counter-part, and any dependency, being startup scripts or dlls could be installed in the bin/ environment global space. This path would be added to the python startup sequence (in order to not rely on the env's activate). I feel that the current state of affairs is not that far, because setup.py bdist_wheel works now. Knowing that there are alternatives on the way (in metadata 2.0 ?) and workarounds, we will go with our current wheel solution using setup.py bdist_wheel. If the bdist_wheel command ever loses the package binary files in scripts dir we have alternatives (listed in order of dependability): 1) add the dependent dlls to every package that needs it (Steve's answer https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html concurs that the dependent dll would be loaded only once) 2) use technique to modify path in the first __init__.py (https://mail.python.org/pipermail/distutils-sig/2014-September/024962.html) 3) statically link the dependent library: not very good in the face of sharing code and having multiple copies in different extension modules with its state etc. Does the community still think this is a I would not design my solution like yours use-case ? The extension modules are a really good way to accelerate python, so they are bound to be constructed with other dependent libraries. It is not only a sdist world :-), particularly on Windows. Once again thanks, D. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
David Genest wrote: 1) add the dependent dlls to every package that needs it (Steve's answer https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html concurs that the dependent dll would be loaded only once) This is the best approach regardless of what else works/doesn't work. Doing this will ensure that your extension module loads the correct dependencies regardless of what other packages or versions are installed in the same environment. The Scripts folder is not necessarily on the path for all users, and nor is the Python directory itself. Depending on how you launch Python, you may get DLLs loaded from the Python directory or the Scripts directory, but either way you should _always_ get DLLs loaded from the same directory as the extension module. Neither directory is intended to be a dumping ground for all the dependencies used by packages. I see no reason 'pip wheel' should produce different wheels to bdist_wheel (but perhaps there is one?), so I would consider this a bug. Cheers, Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 1 October 2014 17:44, David Genest david.gen...@ubisoft.com wrote: - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). It sounds like you're using an old version of wheel. The --skip-scripts argument was removed (and skipping scripts made the default) in 0.23.0. - On the other hand, if you run pip wheel on the same package, the dlls are not placed in the archive. Is this a pip bug ? No, this is not a pip bug. Scripts are omitted from wheels and generated on install from the metadata. DLLs aren't scripts, and putting them into the scripts list in setup.py will cause them to be treated inappropriately (as you see). Knowing that there are alternatives on the way (in metadata 2.0 ?) and workarounds, we will go with our current wheel solution using setup.py bdist_wheel. You're likely to hit issues pretty soon, I suspect. You're using undefined (and generally strongly discouraged) behaviour, I'm afraid. But there *is* an intention to allow wheels to specify more possible locations for files to be installed (along the lines of the autoconf directory classes), so the appropriate binary directory should be a location you can specify in a supported manner in the longer term. If the bdist_wheel command ever loses the package binary files in scripts dir we have alternatives (listed in order of dependability): 1) add the dependent dlls to every package that needs it (Steve's answer https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html concurs that the dependent dll would be loaded only once) 2) use technique to modify path in the first __init__.py (https://mail.python.org/pipermail/distutils-sig/2014-September/024962.html) 3) statically link the dependent library: not very good in the face of sharing code and having multiple copies in different extension modules with its state etc. Does the community still think this is a I would not design my solution like yours use-case ? The extension modules are a really good way to accelerate python, so they are bound to be constructed with other dependent libraries. It is not only a sdist world :-), particularly on Windows. I certainly wouldn't recommend using undefined behaviour like you are. Personally, I'd probably have designed my system around a single interface package that contained the DLL alongside a Python extension wrapping it. Other packages in your system could then depend on that one, and the DLLs would only be stored in one place. Other packages access the DLL via the extension (extensions can publish a C API for that purpose). But that's with hindsight, and learning the lessons from the issues you're having, so I wouldn't expect you to have known that in advance! Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On Wed, Oct 1, 2014 at 5:44 PM, David Genest david.gen...@ubisoft.com wrote: Thank you all for the precious info. Here are my observations: - We are merely writing extension modules with third party dependant code packaged in a dll. In my mind, this use case is not the exception, and would not necessarily warrant the use of a full blown solution like conda. Our deployed environments are self-contained. - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). One issue with PATH is its global settings: if you copy your dll there and any other program the user may want to use, you will get into trouble. This is mostly an issue for well known 3rd party DLLs, though. - On the other hand, if you run pip wheel on the same package, the dlls are not placed in the archive. Is this a pip bug ? In an ideal world, the scripts directory would be called bin, like the unix counter-part, and any dependency, being startup scripts or dlls could be installed in the bin/ environment global space. This path would be added to the python startup sequence (in order to not rely on the env's activate). I feel that the current state of affairs is not that far, because setup.py bdist_wheel works now. Knowing that there are alternatives on the way (in metadata 2.0 ?) and workarounds, we will go with our current wheel solution using setup.py bdist_wheel. If the bdist_wheel command ever loses the package binary files in scripts dir we have alternatives (listed in order of dependability): 1) add the dependent dlls to every package that needs it (Steve's answer https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html concurs that the dependent dll would be loaded only once) Note Steve's observation regarding manifests. It is not an uncommon issue when integrating with 3rd party libs in my experience (where renaming the dll is often impractical). David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On Wed, Oct 1, 2014 at 9:44 AM, David Genest david.gen...@ubisoft.com wrote: - We are merely writing extension modules with third party dependent code packaged in a dll. In my mind, this use case is not the exception, and would not necessarily warrant the use of a full blown solution like conda. agreed -- it is not rare, so yes, it would be nice if the core python (pypa) systems addressed it. But like David said, Windows makes this really hard... - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). If this is the PATH only for that environment, then this is probably fine. But one of the biggest sources of dll hell is that the same PATH is used for executables and dlls, and that dlls placed next to executables will be found. this means that any old app could find any old dll on teh PATH, and that there are a lot of dll on teh PATH. So putting dlls into the python scripts or bin dir is a bad idea in general -- who know what apps may find them? Couple this with the (absolutely incomprehensible to me) habit of folks to use short (still 8.3) names for dlls, without much version info, and you really have a mess. So if you do put your dlls into the Script dir -- do please give them nice long descriptive names! But isn't there a library or somethign directory where other python dlls are that could be used instead? then you could get clashes between python extensions, but it wouldn't clash with anything else on the system. In an ideal world, the scripts directory would be called bin, like the unix counter-part, why does the name matter at all? and any dependency, being startup scripts or dlls could be installed in the bin/ environment global space. This path would be added to the python startup sequence (in order to not rely on the env's activate). ouch -- no dlls and top level scripts don't belong in the same place, period. ANother option is to make a python package that has little other than that dll in it, then yoru packaged list it as a dependency, and I _THINK_ there is some relative path magic that you can do so that your other extensions can find it. Anyone know what Anaconda does on Windows? 1) add the dependent dlls to every package that needs it (Steve's answer https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html concurs that the dependent dll would be loaded only once) If Steve is correct, which he probably is -- this is a great way to go. Alternatively, alter my suggestion above a bit -- and have your dll package have a tiny extension that does nothing but link the dll in. then everything that depends on that dll will have a import the_funny_dll_package line at the top -- and this ends up looking just like a regular old python dependency. Again, make sure to use a descriptive enough name for the dll so that it doesn't clash with other packages (Not yours) that may use the same (similar) dll. Does the community still think this is a I would not design my solution like yours use-case ? The extension modules are a really good way to accelerate python, so they are bound to be constructed with other dependent libraries. It is not only a sdist world :-), particularly on Windows. this is common problem we'd all love to be able to solve! (and conda does help) and sdist doesn't help anyway -- folks need to build and install it somehow anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On Wed, Oct 1, 2014 at 1:35 PM, Paul Moore p.f.mo...@gmail.com wrote: On 1 October 2014 17:44, David Genest david.gen...@ubisoft.com wrote: - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). It sounds like you're using an old version of wheel. The --skip-scripts argument was removed (and skipping scripts made the default) in 0.23.0. - On the other hand, if you run pip wheel on the same package, the dlls are not placed in the archive. Is this a pip bug ? No, this is not a pip bug. Scripts are omitted from wheels and generated on install from the metadata. DLLs aren't scripts, and putting them into the scripts list in setup.py will cause them to be treated inappropriately (as you see). You are confusing generated entry_points script wrappers with the setup(scripts=...) scripts. The scripts=... scripts should never be skipped, even with --skip-scripts, they should work the same as they always have. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 1 October 2014 21:06, Daniel Holth dho...@gmail.com wrote: You are confusing generated entry_points script wrappers with the setup(scripts=...) scripts. The scripts=... scripts should never be skipped, even with --skip-scripts, they should work the same as they always have. Sorry, you're right. But the legacy (non entry-point) scripts are certainly fragile, and I'd recommend avoiding them. Even for actual scripts, and *certainly* as a hack to get things in the Scripts directory... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 2 Oct 2014 06:12, Paul Moore p.f.mo...@gmail.com wrote: On 1 October 2014 21:06, Daniel Holth dho...@gmail.com wrote: You are confusing generated entry_points script wrappers with the setup(scripts=...) scripts. The scripts=... scripts should never be skipped, even with --skip-scripts, they should work the same as they always have. Sorry, you're right. But the legacy (non entry-point) scripts are certainly fragile, and I'd recommend avoiding them. Even for actual scripts, and *certainly* as a hack to get things in the Scripts directory... Note that PEP 459 currently proposes preserving this capability as python.commands.prebuilt, so I personally consider it reasonable as a way of packaging arbitrary executables and non-entry-point based scripts. The main problem with using it for DLLs is the potential for DLL hell that you and others have mentioned, as version management on DLLs installed into shared directories can get very messy. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 1 October 2014 23:10, Nick Coghlan ncogh...@gmail.com wrote: Sorry, you're right. But the legacy (non entry-point) scripts are certainly fragile, and I'd recommend avoiding them. Even for actual scripts, and *certainly* as a hack to get things in the Scripts directory... Note that PEP 459 currently proposes preserving this capability as python.commands.prebuilt, so I personally consider it reasonable as a way of packaging arbitrary executables and non-entry-point based scripts. Existing code tends to try to rewrite shebang lines on scripts. And maybe on Unix add the executable bit. The wheel spec (installing a wheel, the Spread step, item c) actually requires that script rewriting is done, so in theory that makes the scripts subdirectory in a wheel unsuitable. In practice of course, DLLs will be fine as they don't start with #!python. The python.commands.prebuilt spec doesn't say whether #! rewrites are done. If they are, they have the same issue as scripts. If they aren't, the current wheel spec doesn't offer an appropriate area to put them. Basically, it needs a wheel spec update to have an area which holds files that are copied unchanged to the destination Scripts folder. But this is entirely standards-style nitpicking. The existing scripts support is fine for DLLs, and will work as expected for every practical purpose. That's why I called it a hack - it works, but it's unofficial. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
Note that PEP 459 currently proposes preserving this capability as python.commands.prebuilt, so I personally consider it reasonable as a way of packaging arbitrary executables and non-entry-point based scripts. Yes, this will prove valuable (for other things than dlls, admittedly). The main problem with using it for DLLs is the potential for DLL hell that you and others have mentioned, as version management on DLLs installed into shared directories can get very messy. We control our environment and package only what is needed in it. This makes a micro system in which everything is controlled and isolated, even the global dlls (to the virtual env) I wanted to install. They become only accessible to the activated environment. I don’t see how it can become DLL hell in this situation. But hearing many fine and useful comments on the thread made me change my mind and package the dependency near the extension module. I still find that something like pep 459 will be really useful. To the list: Thanks for all you input. D. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
