Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 April 2015 at 00:48, Steve Dower steve.do...@microsoft.com wrote: On the Start Menu suggestion, I think that's a horrible idea. Pip is not the system package manager and it shouldn't be changing the system. Unversioned script launchers are in the same category, but aren't quite as offensive. I know it's only a hypothetical, but I'd much rather it didn't get repeated so often that it actually happens. There are better tools for making app installers, as opposed to package installers. Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. Just to expand on another point in my mail - I'd like *anyone* to provide an example of a genuine use case for something they think should be a required installer extension. I'm not sure such a thing actually exists... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 April 2015 at 08:30, Tim Golden m...@timgolden.me.uk wrote: Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. Thanks for involving Mark in this. While pywin32 isn't the only project with a postinstall script, it's one of the most complex that I know of, and a good example to work from when looking at what projects need. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). To be clear, I don't have that much of a problem with pywin32. I don't use it myself, these days, but that's because (a) it's a very big, monolithic dependency, and (b) it's not usable directly with pip. The problem I have with it is that a lot of projects use it for simple access to the Win32 API (uses which can easily be handled by ctypes, possibly with slightly more messy code) and that means that they inherit the pywin32 problems. So I advise against pywin32 because of that, *not* because I think it's a problem itself, when used for situations where there isn't a better alternative. The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? To be honest, for the cases I encounter frequently, these requirements don't come up. So my experience here goes back to the days when I used pywin32 to write COM servers and services, which was quite a while ago. From what I recall, pywin32 has the following steps in its postinstall: 1. Create start menu entries. My view is that this should simply be dropped. Python packages should never be adding start menu entries. Steve Dower has confirmed he agrees with this view earlier on this thread. 2. Move the pywin32 DLLs to the system directory. I don't see any way this is compatible with per-user or virtualenv installs, so I don't know how to address this, other than again dropping the step. I've no idea why this is necessary, or precisely which parts of pywin32 require it (I've a recollection from a long time ago that services written in Python was the explanation, but that's all I know). But presumably such use cases already break with a per-user Python install? 3. Registering the ActiveX COM DLLs. I believe this is mostly obsolete technology these days (who still uses ActiveX Scripting in anything other than VBScript or maybe a bit of JScript?) I'd drop this and make it a step that the user has to do manually if they want it. In place of it, pywin32 could provide an entry point to register the DLLs (python -m pywin32 --register-dlls or something). Presumably users who need it would understand the implications, and how to avoid registering multiple environments or forgetting to unregister before dropping an environment, etc. That sort of pitfall isn't something Python should try to solve automatically via pre- and post- install scripts. 4. Registering help files. I never understood how that worked or why it was needed. So again, I'd say just drop it. Have I missed anything else? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]
Tim, As a long time user, I think you're right on the money. My only concern is how to manage the transition in user experience, as moving to what you've described (which I totally approve of, if it's feasible) will be a significant change, and may break user expectations. I think maybe the best thing to do is to change the existing binary installer package to: - install the included wheel in the system python environment - run the various post-install scripts (py -m) - install pythonwin, along with start menu icons, etc. Those that want to use pywin32 in a virtualenv (or just without all the system changes) could simply install the wheel (or even an sdist, perhaps) from the command line using pip, and then perform whatever other steps they want manually. This would allow those who are installing using the installer package (which I assume is almost everybody, right?) to get a similar experience to the current one, while those wanting more control (use in virtualenvs, etc) to have that as well. I think the changes described have the potential to be a big win. On Thu, Apr 16, 2015 at 5:11 AM, Tim Golden m...@timgolden.me.uk wrote: [cc-ing Mark H as he indicated he was open to be kept in the loop; also changed the title to reflect the shift of conversation] On 16/04/2015 09:21, Paul Moore wrote: On 16 April 2015 at 08:30, Tim Golden m...@timgolden.me.uk wrote: Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. Thanks for involving Mark in this. While pywin32 isn't the only project with a postinstall script, it's one of the most complex that I know of, and a good example to work from when looking at what projects need. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). To be clear, I don't have that much of a problem with pywin32. I don't use it myself, these days, but that's because (a) it's a very big, monolithic dependency, and (b) it's not usable directly with pip. The problem I have with it is that a lot of projects use it for simple access to the Win32 API (uses which can easily be handled by ctypes, possibly with slightly more messy code) and that means that they inherit the pywin32 problems. So I advise against pywin32 because of that, *not* because I think it's a problem itself, when used for situations where there isn't a better alternative. The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? To be honest, for the cases I encounter frequently, these requirements don't come up. So my experience here goes back to the days when I used pywin32 to write COM servers and services, which was quite a while ago. From what I recall, pywin32 has the following steps in its postinstall: 1. Create start menu entries. My view is that this should simply be dropped. Python packages should never be adding start menu entries. Steve Dower has confirmed he agrees with this view earlier on this thread. 2. Move the pywin32 DLLs to the system directory. I don't see any way this is compatible with per-user or virtualenv installs, so I don't know how to address this, other than again dropping the step. I've no idea why this is necessary, or precisely which parts of pywin32 require it (I've a recollection from a long time ago that services written in Python was the explanation, but that's all I know). But presumably such use cases already break with a per-user Python install? 3. Registering the ActiveX COM DLLs. I believe this is mostly obsolete technology these days (who still uses ActiveX Scripting in anything other than VBScript or maybe a bit of JScript?) I'd drop this and make it a step that the user has to do manually if they want it. In place of it, pywin32 could provide an entry point to register the DLLs (python -m pywin32
[Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]
[cc-ing Mark H as he indicated he was open to be kept in the loop; also changed the title to reflect the shift of conversation] On 16/04/2015 09:21, Paul Moore wrote: On 16 April 2015 at 08:30, Tim Golden m...@timgolden.me.uk wrote: Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. Thanks for involving Mark in this. While pywin32 isn't the only project with a postinstall script, it's one of the most complex that I know of, and a good example to work from when looking at what projects need. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). To be clear, I don't have that much of a problem with pywin32. I don't use it myself, these days, but that's because (a) it's a very big, monolithic dependency, and (b) it's not usable directly with pip. The problem I have with it is that a lot of projects use it for simple access to the Win32 API (uses which can easily be handled by ctypes, possibly with slightly more messy code) and that means that they inherit the pywin32 problems. So I advise against pywin32 because of that, *not* because I think it's a problem itself, when used for situations where there isn't a better alternative. The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? To be honest, for the cases I encounter frequently, these requirements don't come up. So my experience here goes back to the days when I used pywin32 to write COM servers and services, which was quite a while ago. From what I recall, pywin32 has the following steps in its postinstall: 1. Create start menu entries. My view is that this should simply be dropped. Python packages should never be adding start menu entries. Steve Dower has confirmed he agrees with this view earlier on this thread. 2. Move the pywin32 DLLs to the system directory. I don't see any way this is compatible with per-user or virtualenv installs, so I don't know how to address this, other than again dropping the step. I've no idea why this is necessary, or precisely which parts of pywin32 require it (I've a recollection from a long time ago that services written in Python was the explanation, but that's all I know). But presumably such use cases already break with a per-user Python install? 3. Registering the ActiveX COM DLLs. I believe this is mostly obsolete technology these days (who still uses ActiveX Scripting in anything other than VBScript or maybe a bit of JScript?) I'd drop this and make it a step that the user has to do manually if they want it. In place of it, pywin32 could provide an entry point to register the DLLs (python -m pywin32 --register-dlls or something). Presumably users who need it would understand the implications, and how to avoid registering multiple environments or forgetting to unregister before dropping an environment, etc. That sort of pitfall isn't something Python should try to solve automatically via pre- and post- install scripts. 4. Registering help files. I never understood how that worked or why it was needed. So again, I'd say just drop it. Really, pywin32 is several things: a set of libraries (win32api, win32file, etc.); some system-level support for various things (COM registration, Service support etc.); and a development/editing environment (pythonwin). I see this ending up as (respectively): as venv-friendly wheel; a py -m script of the kind Paul suggests; and an installable app with the usual start menu icons etc. In my copious spare time I'll at least try to visit the pywin32 codebase to see how viable all this is. Feel free to challenge my thoughts on the matter. TJG ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]
This seems like a good time to remind everyone that wheel convert can turn bdist_wininst .exe's to wheels. Both formats are designed to preserve all the distutils file categories. In the future it would be nice if the bdist_wininst .exe wrapper used wheel instead of its own format. Then a single file would both be the Windows installer and a valid wheel (except for the extension). On Thu, Apr 16, 2015 at 9:43 AM, Kevin Horn kevin.h...@gmail.com wrote: Tim, As a long time user, I think you're right on the money. My only concern is how to manage the transition in user experience, as moving to what you've described (which I totally approve of, if it's feasible) will be a significant change, and may break user expectations. I think maybe the best thing to do is to change the existing binary installer package to: - install the included wheel in the system python environment - run the various post-install scripts (py -m) - install pythonwin, along with start menu icons, etc. Those that want to use pywin32 in a virtualenv (or just without all the system changes) could simply install the wheel (or even an sdist, perhaps) from the command line using pip, and then perform whatever other steps they want manually. This would allow those who are installing using the installer package (which I assume is almost everybody, right?) to get a similar experience to the current one, while those wanting more control (use in virtualenvs, etc) to have that as well. I think the changes described have the potential to be a big win. On Thu, Apr 16, 2015 at 5:11 AM, Tim Golden m...@timgolden.me.uk wrote: [cc-ing Mark H as he indicated he was open to be kept in the loop; also changed the title to reflect the shift of conversation] On 16/04/2015 09:21, Paul Moore wrote: On 16 April 2015 at 08:30, Tim Golden m...@timgolden.me.uk wrote: Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. Thanks for involving Mark in this. While pywin32 isn't the only project with a postinstall script, it's one of the most complex that I know of, and a good example to work from when looking at what projects need. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). To be clear, I don't have that much of a problem with pywin32. I don't use it myself, these days, but that's because (a) it's a very big, monolithic dependency, and (b) it's not usable directly with pip. The problem I have with it is that a lot of projects use it for simple access to the Win32 API (uses which can easily be handled by ctypes, possibly with slightly more messy code) and that means that they inherit the pywin32 problems. So I advise against pywin32 because of that, *not* because I think it's a problem itself, when used for situations where there isn't a better alternative. The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? To be honest, for the cases I encounter frequently, these requirements don't come up. So my experience here goes back to the days when I used pywin32 to write COM servers and services, which was quite a while ago. From what I recall, pywin32 has the following steps in its postinstall: 1. Create start menu entries. My view is that this should simply be dropped. Python packages should never be adding start menu entries. Steve Dower has confirmed he agrees with this view earlier on this thread. 2. Move the pywin32 DLLs to the system directory. I don't see any way this is compatible with per-user or virtualenv installs, so I don't know how to address this, other than again dropping the step. I've no idea why this is necessary, or precisely which parts of pywin32 require it (I've a recollection from a long time ago that services written in Python was the explanation,
Re: [Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]
On 16 April 2015 at 14:43, Kevin Horn kevin.h...@gmail.com wrote: Those that want to use pywin32 in a virtualenv (or just without all the system changes) could simply install the wheel (or even an sdist, perhaps) from the command line using pip, and then perform whatever other steps they want manually. Just as a data point, converting the existing wininst installer for pywin32 to a wheel (using wheel convert), and installing that via pip, is entirely workable (for the win32api/win32file type basic functionality). The pypiwin32 project (started by Glyph as a way of providing pywin32 wheels from PyPI) includes wheels for Python 3.x which I built that way, so it's certainly seen some level of use. The wheels are probably unnecessarily big, as they'll include all of pythonwin, and the ActiveX Scripting and service creation support, which I guess won't work in that configuration, but they are a good starting point for checking precisely what will work unmodified from a wheel. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16/04/2015 08:08, Paul Moore wrote: On 16 April 2015 at 00:48, Steve Dower steve.do...@microsoft.com wrote: On the Start Menu suggestion, I think that's a horrible idea. Pip is not the system package manager and it shouldn't be changing the system. Unversioned script launchers are in the same category, but aren't quite as offensive. I know it's only a hypothetical, but I'd much rather it didn't get repeated so often that it actually happens. There are better tools for making app installers, as opposed to package installers. Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? TJG ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]
On 16 April 2015 at 11:11, Tim Golden m...@timgolden.me.uk wrote: Really, pywin32 is several things: a set of libraries (win32api, win32file, etc.); some system-level support for various things (COM registration, Service support etc.); and a development/editing environment (pythonwin). That sounds about right. I see this ending up as (respectively): as venv-friendly wheel; a py -m script of the kind Paul suggests; and an installable app with the usual start menu icons etc. Again, yes, that seems reasonable. Personally, for the uses I see of pywin32, it would make sense to split it into a number of separate wheels (win32api, win32file, ...) to reduce the dependency footprint for projects that only use one or two functions out of the whole thing, but honestly ctypes is probably still a better approach for that scenario, so the benefit of such a split is likely minimal. In my copious spare time I'll at least try to visit the pywin32 codebase to see how viable all this is. Feel free to challenge my thoughts on the matter. I think you're going in the right direction. The hardest parts are likely to be where the Windows architecture interferes (COM registration and services). Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Wed, Apr 15, 2015 at 2:23 PM, Paul Moore p.f.mo...@gmail.com wrote: In the PEP, there's a concept of optional vs required extensions. See https://www.python.org/dev/peps/pep-0426/#required-extension-handling. This is crucial - I've no problem if a particular extension is used by a project, as long as it's optional. I won't install it, so it's fine. It seems to me that pip *has* to ignore missing optional extensions, for this reason. Of course, that introduces the converse problem, which is how would people who might want that extension to be activated, know that a project used it? Exactly -- we do want pip install to just work... But I worry that some people may have a more liberal definition of required than I do. They probably do -- if they want things to just work We have the same problem with optional dependencies. For instance, for iPython to work, you don't need much. but if you want the ipython notebook to work, you need tornado, zeromq, who knows what else. But people want it to just work -- and just work be default, so you want all that optional stuff to go in by default. I expect this is the same with wheel installer extensions. To use your example, for instance. People want to do: pip install sphinx and then have the sphinx-quickstart utility ready to go. by default. So scripts need to be installed by default. The trade-off between convenience and control/security is tough. Based on the above, it's possibly valid to allow required extensions to be auto-installed. It *is* a vector for unexpected code execution, but maybe that's OK. If even required extensions aren't auto installed, then we can just toss out the whole idea of automatic dependency management. (which I personally wouldn't mind, actually, but I'm weird that way) But maybe we need some real use cases to talk about -- I agree with others in this thread that the Start menu isn't a good example. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: help needed
On Apr 14, 2015 7:15 PM, Nick Coghlan ncogh...@gmail.com wrote: On 14 April 2015 at 11:19, Trishank Karthik Kuppusamy trish...@nyu.edu wrote: On 14 April 2015 at 11:16, Brett Cannon br...@python.org wrote: I agree. Even something as simple as a boolean that triggers a banner saying this project is looking for a new maintainer would be useful both from the perspective of project owners who want to move on or from the perspective of users who can't tell if a project is maintained based on how long it has been since a project uploaded a new version (which is why I think someone suggested sending an annual email asking for a human action to say alive and kicking to help determine if a project is completely abandoned). Yeah, I think Guido said something to this effect in his keynote. Yep, Guido's keynote was the genesis of the thread. For folks that haven't seen it, the specific points of concern raised were: * seeking a new maintainer from amongst their users * seeking help with enabling Python 3 support Past suggestions for social features have related to providing users with a standard way to reach maintainers and each other, and I'd prefer to leave maintainers in full control of that aspect of the maintainer experience. I'm not alone in feeling that way, hence why such features tend not to be viewed especially positively. If only there was a way to add RDFa metadata to the a tags in the HTML output of a pypa/readme-rendered long_description. FOAF RDFa and/or a /users/username https://warehouse.readthedocs.org/application/ Recently I learned about pyramid_autodoc. The one thing that *only* PyPI can provide is the combination of a publication channel for maintainers to reach their user base without either side needing to share contact information they aren't already sharing, together with the creation of the clear understanding that providing sustaining engineering for a piece of software represents a significant time commitment that users benefiting from an open source maintainer's generosity should respect. This thread regarding maintainers being able to more clearly communicate maintenance status to users also relates to my blog post ( http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html) regarding the fact that folks that: a) don't personally need to ensure software they maintain works on old versions of Python; and b) aren't getting paid to ensure it works on old versions of Python; c) shouldn't feel obliged to provide such support for free Supporting legacy platforms is generally tedious work that isn't inherently interesting or rewarding. Folks that want such legacy platform support should thus be expecting to have to pay for it, and demanding it for free is unreasonable. The perception that open source software is provided by magic internet pixies that don't need to eat (or at the very least to be thanked for the time their generosity has saved us) is unfortunately widespread and pernicious [1], and PyPI is in a position to help shift that situation to one where open source maintainers at least have the opportunity to clearly explain the sustaining engineering model backing their software while deflecting any criticism for the mere existence of such explanations onto the PyPI maintainers rather than having to cope with any negative feedback themselves. So, is this a new ENUM field and something over and above mtime? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: help needed
On Tue, Apr 14, 2015 at 8:34 PM Ben Finney ben+pyt...@benfinney.id.au wrote: Nick Coghlan ncogh...@gmail.com writes: Yep, Guido's keynote was the genesis of the thread. I can't find it online, can you give a URL so we can see the talk? https://www.youtube.com/watch?v=G-uKNd5TSBw Past suggestions for social features have related to providing users with a standard way to reach maintainers and each other, and I'd prefer to leave maintainers in full control of that aspect of the maintainer experience. I'm not alone in feeling that way, hence why such features tend not to be viewed especially positively. Thanks for this detailed response differentiating this proposal from previous ones, it's exactly what I was asking for. -- \ “For mad scientists who keep brains in jars, here's a tip: why | `\ not add a slice of lemon to each jar, for freshness?” —Jack | _o__) Handey | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: help needed
On Apr 14, 2015 7:15 PM, Nick Coghlan ncogh...@gmail.com wrote: [...] The perception that open source software is provided by magic internet pixies that don't need to eat (or at the very least to be thanked for the time their generosity has saved us) https://en.wikipedia.org/wiki/Business_models_for_open-source_software https://gist.github.com/ndarville/4295324 ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]
On Thu, Apr 16, 2015 at 9:58 AM, Paul Moore p.f.mo...@gmail.com wrote: On 16 April 2015 at 14:43, Kevin Horn kevin.h...@gmail.com wrote: Those that want to use pywin32 in a virtualenv (or just without all the system changes) could simply install the wheel (or even an sdist, perhaps) from the command line using pip, and then perform whatever other steps they want manually. Just as a data point, converting the existing wininst installer for pywin32 to a wheel (using wheel convert), and installing that via pip, is entirely workable (for the win32api/win32file type basic functionality). The pypiwin32 project (started by Glyph as a way of providing pywin32 wheels from PyPI) includes wheels for Python 3.x which I built that way, so it's certainly seen some level of use. The wheels are probably unnecessarily big, as they'll include all of pythonwin, and the ActiveX Scripting and service creation support, which I guess won't work in that configuration, but they are a good starting point for checking precisely what will work unmodified from a wheel. For people interested in a lightweight alternative to pywin32, we have the pywin32ctypes project, which started as a way to get access to win32 credentials without depending on DLL (to avoid file locking issues with inplace updates). The project is on github (https://github.com/enthought/pywin32-ctypes), and is already used by a few projects. We support both cffi and ctypes backends (the former to work out of the box on cpython, the latter to work on pypy). David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 Apr 2015 03:08, Paul Moore p.f.mo...@gmail.com wrote: Just to expand on another point in my mail - I'd like *anyone* to provide an example of a genuine use case for something they think should be a required installer extension. I'm not sure such a thing actually exists... The constraints extension in PEP 459 recommends flagging extension processing as required, otherwise it's possible for unaware installers to silently skip the compatibility checks: https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension Installers offering the ability to opt in to ignoring environmental constraints is one thing, ignoring them through lack of understanding the extension is something else entirely. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 Apr 2015 14:34, Paul Moore p.f.mo...@gmail.com wrote: By the way. I just did a check through PEPs 426 and 459. Neither one currently defines a postinstall script metadata item or extension, which is interesting given that this discussion started from the question of how postinstall actions would be supported. There *have* been discussions in the past, and I could have sworn they ended up in a PEP somewhere, but maybe I was wrong. Arbitrary postinstall operations haven't ended up in a PEP yet because *I* don't like them. Anyone that has experienced Windows rot where uninstallers fail to clean up properly after themselves has seen first hand the consequences of delegating trust to development teams without the ability to set any minimum expectations for their quality assurance processes. (One way of looking at Linux distro packaging policies is to view them as a code review process applied to Turing complete software build and installation programs, while container tech like Docker is a way of isolating apps from the host system) Trigger based programming is hard at the best of times, and it doesn't get easier when integrating arbitrary pieces of software written by different people at different times in different contexts. On the other hand, I *am* prepared to build in an escape hatch that lets folks disagree with me, and I'll just not install their software. As far as *pip* goes, whether or not to add a plugin system to handle additional metadata extensions would be up to the pip devs. As a user, my main request if the pip devs decided to add such a plugin system would be that extension handlers couldn't be implicitly installed as a dependency of another package. If folks want their installs to just work, they shouldn't be marking non-standard metadata extensions as mandatory :) Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pywin32 on wheels [was: Beyond wheels 1.0: helping downstream, FHS and more]
As already mentioned in this thread, most of the postinstall stuff is needed only for a subset of users - mainly those who want to write COM objects or Windows Services (and also people who want the shortcuts etc). pywin32 itself should be close to portable - eg, setup.py install doesn't run the postinstall script but leaves a largely functioning pywin32 install. So I think it should be relatively easy to achieve for pywin32 to work in a virtual env without running any of the post-install scripts, and I'd support any consolidation of the setup process to support this effort. Cheers, Mark On 16/04/2015 8:11 PM, Tim Golden wrote: [cc-ing Mark H as he indicated he was open to be kept in the loop; also changed the title to reflect the shift of conversation] On 16/04/2015 09:21, Paul Moore wrote: On 16 April 2015 at 08:30, Tim Golden m...@timgolden.me.uk wrote: Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. Thanks for involving Mark in this. While pywin32 isn't the only project with a postinstall script, it's one of the most complex that I know of, and a good example to work from when looking at what projects need. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). To be clear, I don't have that much of a problem with pywin32. I don't use it myself, these days, but that's because (a) it's a very big, monolithic dependency, and (b) it's not usable directly with pip. The problem I have with it is that a lot of projects use it for simple access to the Win32 API (uses which can easily be handled by ctypes, possibly with slightly more messy code) and that means that they inherit the pywin32 problems. So I advise against pywin32 because of that, *not* because I think it's a problem itself, when used for situations where there isn't a better alternative. The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? To be honest, for the cases I encounter frequently, these requirements don't come up. So my experience here goes back to the days when I used pywin32 to write COM servers and services, which was quite a while ago. From what I recall, pywin32 has the following steps in its postinstall: 1. Create start menu entries. My view is that this should simply be dropped. Python packages should never be adding start menu entries. Steve Dower has confirmed he agrees with this view earlier on this thread. 2. Move the pywin32 DLLs to the system directory. I don't see any way this is compatible with per-user or virtualenv installs, so I don't know how to address this, other than again dropping the step. I've no idea why this is necessary, or precisely which parts of pywin32 require it (I've a recollection from a long time ago that services written in Python was the explanation, but that's all I know). But presumably such use cases already break with a per-user Python install? 3. Registering the ActiveX COM DLLs. I believe this is mostly obsolete technology these days (who still uses ActiveX Scripting in anything other than VBScript or maybe a bit of JScript?) I'd drop this and make it a step that the user has to do manually if they want it. In place of it, pywin32 could provide an entry point to register the DLLs (python -m pywin32 --register-dlls or something). Presumably users who need it would understand the implications, and how to avoid registering multiple environments or forgetting to unregister before dropping an environment, etc. That sort of pitfall isn't something Python should try to solve automatically via pre- and post- install scripts. 4. Registering help files. I never understood how that worked or why it was needed. So again, I'd say just drop it. Really, pywin32 is several things: a set of libraries (win32api, win32file, etc.); some system-level support for various things (COM registration, Service support etc.); and a
Re: [Distutils] name of the dependency problem
Okay, I tried to summarize the discussion and most of my thoughts on that issue. https://github.com/pypa/pip/issues/988 I'll post anything further I have to say there. I hope to get a student to measure the extent of the problem... Thanks, Justin On Wed, Apr 15, 2015 at 2:44 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-04-15 12:09:04 +0100 (+0100), Robin Becker wrote: After again finding that pip doesn't have a correct dependency resolution solution a colleage and I discussed the nature of the problem. [...] Before the discussion of possible solutions heads too far afield, it's worth noting that this was identified years ago and has a pending feature request looking for people pitching in on implementation. It's perhaps better discussed at https://github.com/pypa/pip/issues/988 so as to avoid too much repetition. -- Jeremy Stanley ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: help needed
On 16 April 2015 at 17:42, Wes Turner wes.tur...@gmail.com wrote: On Apr 14, 2015 7:15 PM, Nick Coghlan ncogh...@gmail.com wrote: [...] The perception that open source software is provided by magic internet pixies that don't need to eat (or at the very least to be thanked for the time their generosity has saved us) https://en.wikipedia.org/wiki/Business_models_for_open-source_software https://gist.github.com/ndarville/4295324 Right, there *are* for-profit business models and non-profit fundraising models that can support sustainable development and maintenance of open source software. However, it can also be hard to tell the difference between supported and unsupported software until low level infrastructure shifts like Python 3 or Linux containerisation come along - in those cases, the software without a good sustaining development story runs a high risk of getting trapped in the old model. Unfortunately, Do you know and understand the sustaining engineering model for all of your dependencies? is a question most of us will be forced to say No to, even those of us that really should know better. It's very easy to assume that a popular open source project has a well-funded sustainable development process backing it without actually checking that that assumption is accurate. When I first started working there, I used to think Boeing's risk management folks were overly paranoid for demanding to know the answer to that question before agreeing to depend on a new supplier, but I eventually came to understand that it's mostly a matter of being able to quantify risk - if you have 10 key dependencies, each with a 10% chance of going away in a given time period, then you end up facing a 2/3rds chance of having to replace at least one of those components with an alternative. As a result, these days *I* tend to be the one wanting to know the long term sustaining engineering plans for new services and dependencies (and sometimes a service or dependency will be valuable enough to be worth taking a chance on). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 April 2015 at 17:58, Chris Barker chris.bar...@noaa.gov wrote:
We have the same problem with optional dependencies.
For instance, for iPython to work, you don't need much. but if you want the
ipython notebook to work, you need tornado, zeromq, who knows what else. But
people want it to just work -- and just work be default, so you want all
that optional stuff to go in by default.
But none of those are installed by default with ipython - they are
covered by extras. If you want them, you ask for them. Thanks to
extras, ipython offers a nice shortcut for you - pip install
ipython[all] - but you still have to ask for them.
I expect this is the same with wheel installer extensions. To use your
example, for instance. People want to do:
pip install sphinx
and then have the sphinx-quickstart utility ready to go. by default. So
scripts need to be installed by default.
Yes, the script wrapper extension is a much better example of a
nobody would ever want this off extension. But is it really
required? The distribution would work fine if scripts weren't
installed. My understanding of the required/optional distinction in
the PEP is that an extension is required if the installation would be
*broken* if that extension wasn't supported. And having to type
python -m something rather than just something isn't broken, it's
just an inconvenience.
In practice, I'd assume script wrappers would be an extension built
into pip, so it would always be available. But that's different from
required. Installing via wheel install (which doesn't support
generating script wrappers) still works.
Actually, I just checked PEP 459, which says of the python.commands
extension: Build tools SHOULD mark this extension as requiring
handling by installers. So I stand corrected - script wrappers should
be considered mandatory. In practice, though, what that means is that
pip will be fine (as it'll have support built in) and wheel will be a
non-compliant installer (as it neither supports generating wrappers,
nor does it give an error when asked to create them - maybe an error
will get added, but I doubt it as wheel install isn't intended to be a
full installer). I've no idea at this point what distil or
easy_install will do, much less any other installers out there.
The trade-off between convenience and control/security is tough.
Certainly. But that's about what is available by default, and how the
installer (pip) handles the user interface for this package says that
it gives you some extra functionality if you have extensions X, Y, and
Z. There's no convenience or UI implication with required extensions
- if they aren't available, the installer refuses to install the
distribution. Simple as that. Maybe pip could try to locate and
download mandatory extensions before deciding whether to fail the
install, but the package metadata doesn't say how to find such
installer plugins (and it *can't* - because the plugin would be
different for pip, easy-install, distil or wheel, so all it can say is
I need plugin foo and the installer has to know the rest). That's an
installation program quality of implementation issue though.
Given that a random project could add metadata
extensions: {
foocorp.randomstuff: {
installer_must_handle: true,
foocorp.localdata: Some random stuff
}
}
there is no way that pip has any means of discovering where to get
code to handle the foocorp.randomstuff extension from. So in the
general case, auto-installing required extension support just isn't
possible. At best, pip could have a registry of plugins that support
standard extensions (i.e. those covered by a PEP) but I'd expect that
we'd just build them into pip (as we don't have a plugin interface at
the moment).
Based on the above, it's possibly valid to allow required extensions
to be auto-installed. It *is* a vector for unexpected code execution,
but maybe that's OK.
If even required extensions aren't auto installed, then we can just toss out
the whole idea of automatic dependency management. (which I personally
wouldn't mind, actually, but I'm weird that way)
I disagree. It's no different conceptually than the fact that if you
don't have a C compiler, you can't install a package that contains C
code and only comes in sdist format today. The UI in pip for noticing
and saying you need a C compiler is terrible (you get a build error
which might mention that you don't have the compiler, if you're lucky
:-)). And yet people survive. So a clear error saying package X needs
a handler for extension Y to install is a major improvement over the
current situation. (I know C compilers are build-step and extensions
are install-step, but right now the user experience doesn't clearly
distinguish these, so the analogy holds).
Whether users want pip to go one step further and auto-install the
plugin is unknown at this point. So far, it seems that the only people
who have expressed an opinion (you and I)
