Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 18 April 2015 at 18:19, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: (your quote trimming's a bit over-enthusiastic, you lost the attribution here) python -m something rather than just something isn't broken, it's just an inconvenience. Tell that to a newbie. This is EXACTLY the kind of thing that should just work. It's a huge quality of implementation issue, certainly - any installer that doesn't include script generation built in is going to be as annoying as hell to a user. But they do exist (wheel install, for instance) and the resulting installation works, even if a newcomer would hate it. So it's not mandatory in the sense that no functionality is lost. But this is a moot point, as PEP 459 says the python.commands extension SHOULD be marked as required. And wheel install would technically be in violation of PEP 426, as it doesn't handle script wrappers and it doesn't fail when a package needs them (only technically, because PEP 426 isn't finalised yet, and wheel install could be updated to support it). But I'd already said most of that - you just pulled that one point out of context. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
The wheel installer does call setuptools to generate console script wrappers. On Apr 18, 2015 1:36 PM, Paul Moore p.f.mo...@gmail.com wrote: On 18 April 2015 at 18:19, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: (your quote trimming's a bit over-enthusiastic, you lost the attribution here) python -m something rather than just something isn't broken, it's just an inconvenience. Tell that to a newbie. This is EXACTLY the kind of thing that should just work. It's a huge quality of implementation issue, certainly - any installer that doesn't include script generation built in is going to be as annoying as hell to a user. But they do exist (wheel install, for instance) and the resulting installation works, even if a newcomer would hate it. So it's not mandatory in the sense that no functionality is lost. But this is a moot point, as PEP 459 says the python.commands extension SHOULD be marked as required. And wheel install would technically be in violation of PEP 426, as it doesn't handle script wrappers and it doesn't fail when a package needs them (only technically, because PEP 426 isn't finalised yet, and wheel install could be updated to support it). But I'd already said most of that - you just pulled that one point out of context. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 18 April 2015 at 13:36, Paul Moore p.f.mo...@gmail.com wrote: On 18 April 2015 at 18:19, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: (your quote trimming's a bit over-enthusiastic, you lost the attribution here) python -m something rather than just something isn't broken, it's just an inconvenience. Tell that to a newbie. This is EXACTLY the kind of thing that should just work. It's a huge quality of implementation issue, certainly - any installer that doesn't include script generation built in is going to be as annoying as hell to a user. But they do exist (wheel install, for instance) and the resulting installation works, even if a newcomer would hate it. So it's not mandatory in the sense that no functionality is lost. But this is a moot point, as PEP 459 says the python.commands extension SHOULD be marked as required. And wheel install would technically be in violation of PEP 426, as it doesn't handle script wrappers and it doesn't fail when a package needs them (only technically, because PEP 426 isn't finalised yet, and wheel install could be updated to support it). It's not in violation, that's the whole point of saying SHOULD, rather than MUST. Please don't lose that distinction - if users start demanding that developers always implement SHOULDs, they're misreading the spec, and are going to make life miserable for a lot of people by making unreasonable demands on their time. As a specification author, SHOULD is a way for us to say most users are likely to want this, so you should probably do it if you don't have a strong preference, but not all users will want it, so certain tools may choose not to do it for reasons that are too context specific for us to go into in a general purpose specification. The MUSTs are the things you may not personally care about, but other people do care about, will break if you get this wrong part of the specs, the SHOULDs are this is probably a good thing to do, but you may disagree :) Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
For the most part, I think it's all been said. What should and shouldn't be installed by default is really specific extension dependent, not much point in speculating. But a comment or two: having to type python -m something rather than just something isn't broken, it's just an inconvenience. Tell that to a newbie. This is EXACTLY the kind of thing that should just work. Maybe a three-tier system: 1) mandatory -- can't install without it 2) default -- try to install it be default if possible 3) optional: only install if specifically asked for And this isn't just about extensions -- for instance, the all stuff in iPython would be well served by level 2 It's no different conceptually than the fact that if you don't have a C compiler, you can't install a package that contains C Sure it is -- a C complier is a system tool, and the whole point of binary wheels is that the end user doesn't need one. -CHB ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 18 April 2015 at 20:02, Nick Coghlan ncogh...@gmail.com wrote: It's a huge quality of implementation issue, certainly - any installer that doesn't include script generation built in is going to be as annoying as hell to a user. But they do exist (wheel install, for instance) and the resulting installation works, even if a newcomer would hate it. So it's not mandatory in the sense that no functionality is lost. But this is a moot point, as PEP 459 says the python.commands extension SHOULD be marked as required. And wheel install would technically be in violation of PEP 426, as it doesn't handle script wrappers and it doesn't fail when a package needs them (only technically, because PEP 426 isn't finalised yet, and wheel install could be updated to support it). It's not in violation, that's the whole point of saying SHOULD, rather than MUST. Please don't lose that distinction - if users start demanding that developers always implement SHOULDs, they're misreading the spec, and are going to make life miserable for a lot of people by making unreasonable demands on their time. As a specification author, SHOULD is a way for us to say most users are likely to want this, so you should probably do it if you don't have a strong preference, but not all users will want it, so certain tools may choose not to do it for reasons that are too context specific for us to go into in a general purpose specification. The MUSTs are the things you may not personally care about, but other people do care about, will break if you get this wrong part of the specs, the SHOULDs are this is probably a good thing to do, but you may disagree :) Sorry - I missed quoting one relevant bit of PEP 426. What I was trying to say was that wheel install violates the MUST in PEP 426 (that installers MUST support a required extension or report an error when installing). But Daniel pointed out that I'm wrong anyway - wheel install *does* support generating script wrappers, using setuptools to do so. My apologies for not checking my facts more carefully, and for my confusing wording. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 April 2015 at 00:48, Steve Dower steve.do...@microsoft.com wrote: On the Start Menu suggestion, I think that's a horrible idea. Pip is not the system package manager and it shouldn't be changing the system. Unversioned script launchers are in the same category, but aren't quite as offensive. I know it's only a hypothetical, but I'd much rather it didn't get repeated so often that it actually happens. There are better tools for making app installers, as opposed to package installers. Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. Just to expand on another point in my mail - I'd like *anyone* to provide an example of a genuine use case for something they think should be a required installer extension. I'm not sure such a thing actually exists... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 April 2015 at 08:30, Tim Golden m...@timgolden.me.uk wrote: Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. Thanks for involving Mark in this. While pywin32 isn't the only project with a postinstall script, it's one of the most complex that I know of, and a good example to work from when looking at what projects need. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). To be clear, I don't have that much of a problem with pywin32. I don't use it myself, these days, but that's because (a) it's a very big, monolithic dependency, and (b) it's not usable directly with pip. The problem I have with it is that a lot of projects use it for simple access to the Win32 API (uses which can easily be handled by ctypes, possibly with slightly more messy code) and that means that they inherit the pywin32 problems. So I advise against pywin32 because of that, *not* because I think it's a problem itself, when used for situations where there isn't a better alternative. The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? To be honest, for the cases I encounter frequently, these requirements don't come up. So my experience here goes back to the days when I used pywin32 to write COM servers and services, which was quite a while ago. From what I recall, pywin32 has the following steps in its postinstall: 1. Create start menu entries. My view is that this should simply be dropped. Python packages should never be adding start menu entries. Steve Dower has confirmed he agrees with this view earlier on this thread. 2. Move the pywin32 DLLs to the system directory. I don't see any way this is compatible with per-user or virtualenv installs, so I don't know how to address this, other than again dropping the step. I've no idea why this is necessary, or precisely which parts of pywin32 require it (I've a recollection from a long time ago that services written in Python was the explanation, but that's all I know). But presumably such use cases already break with a per-user Python install? 3. Registering the ActiveX COM DLLs. I believe this is mostly obsolete technology these days (who still uses ActiveX Scripting in anything other than VBScript or maybe a bit of JScript?) I'd drop this and make it a step that the user has to do manually if they want it. In place of it, pywin32 could provide an entry point to register the DLLs (python -m pywin32 --register-dlls or something). Presumably users who need it would understand the implications, and how to avoid registering multiple environments or forgetting to unregister before dropping an environment, etc. That sort of pitfall isn't something Python should try to solve automatically via pre- and post- install scripts. 4. Registering help files. I never understood how that worked or why it was needed. So again, I'd say just drop it. Have I missed anything else? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16/04/2015 08:08, Paul Moore wrote: On 16 April 2015 at 00:48, Steve Dower steve.do...@microsoft.com wrote: On the Start Menu suggestion, I think that's a horrible idea. Pip is not the system package manager and it shouldn't be changing the system. Unversioned script launchers are in the same category, but aren't quite as offensive. I know it's only a hypothetical, but I'd much rather it didn't get repeated so often that it actually happens. There are better tools for making app installers, as opposed to package installers. Sorry - I agree it's an awful idea. Older wininst installers such as the pywin32 (and I think the PyQT one) one do this, I wanted to use it as an example of abuse of postinstall scripts that should *not* be perpetuated in any new scheme. FWIW I've just had a to-and-fro by email with Mark Hammond. I gather that he's now given Glyph access to the PyPI hg setup for pywin32. He's also happy to consider changes to the setup process to support wheel/virtualenv/postinstall improvements. There's been a side discussion on the pywin32 list about which versions of Python pywin32 should continue to support going forward, which obviously interacts with the idea of making it wheel/virtualenv-friendly. I'm not sure what Glyph's plan is at this point -- doubtless he can speak for himself. I gather from Paul's comments earlier that he's not a particular fan of pywin32. If the thing seems to have legs, I'm happy to coordinate changes to the setup. (I am, technically, a pywin32 committer although I've never made use of that fact). The particular issue I'm not sure about is: how does Paul see pywin32's postinstall steps working when they *are* needed, ie when someone wants to install pywin32 as a wheel and wants the COM registration to happen? Or is that a question of: run these steps manually once pip's completed? TJG ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Wed, Apr 15, 2015 at 2:23 PM, Paul Moore p.f.mo...@gmail.com wrote: In the PEP, there's a concept of optional vs required extensions. See https://www.python.org/dev/peps/pep-0426/#required-extension-handling. This is crucial - I've no problem if a particular extension is used by a project, as long as it's optional. I won't install it, so it's fine. It seems to me that pip *has* to ignore missing optional extensions, for this reason. Of course, that introduces the converse problem, which is how would people who might want that extension to be activated, know that a project used it? Exactly -- we do want pip install to just work... But I worry that some people may have a more liberal definition of required than I do. They probably do -- if they want things to just work We have the same problem with optional dependencies. For instance, for iPython to work, you don't need much. but if you want the ipython notebook to work, you need tornado, zeromq, who knows what else. But people want it to just work -- and just work be default, so you want all that optional stuff to go in by default. I expect this is the same with wheel installer extensions. To use your example, for instance. People want to do: pip install sphinx and then have the sphinx-quickstart utility ready to go. by default. So scripts need to be installed by default. The trade-off between convenience and control/security is tough. Based on the above, it's possibly valid to allow required extensions to be auto-installed. It *is* a vector for unexpected code execution, but maybe that's OK. If even required extensions aren't auto installed, then we can just toss out the whole idea of automatic dependency management. (which I personally wouldn't mind, actually, but I'm weird that way) But maybe we need some real use cases to talk about -- I agree with others in this thread that the Start menu isn't a good example. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 Apr 2015 03:08, Paul Moore p.f.mo...@gmail.com wrote: Just to expand on another point in my mail - I'd like *anyone* to provide an example of a genuine use case for something they think should be a required installer extension. I'm not sure such a thing actually exists... The constraints extension in PEP 459 recommends flagging extension processing as required, otherwise it's possible for unaware installers to silently skip the compatibility checks: https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension Installers offering the ability to opt in to ignoring environmental constraints is one thing, ignoring them through lack of understanding the extension is something else entirely. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 Apr 2015 14:34, Paul Moore p.f.mo...@gmail.com wrote: By the way. I just did a check through PEPs 426 and 459. Neither one currently defines a postinstall script metadata item or extension, which is interesting given that this discussion started from the question of how postinstall actions would be supported. There *have* been discussions in the past, and I could have sworn they ended up in a PEP somewhere, but maybe I was wrong. Arbitrary postinstall operations haven't ended up in a PEP yet because *I* don't like them. Anyone that has experienced Windows rot where uninstallers fail to clean up properly after themselves has seen first hand the consequences of delegating trust to development teams without the ability to set any minimum expectations for their quality assurance processes. (One way of looking at Linux distro packaging policies is to view them as a code review process applied to Turing complete software build and installation programs, while container tech like Docker is a way of isolating apps from the host system) Trigger based programming is hard at the best of times, and it doesn't get easier when integrating arbitrary pieces of software written by different people at different times in different contexts. On the other hand, I *am* prepared to build in an escape hatch that lets folks disagree with me, and I'll just not install their software. As far as *pip* goes, whether or not to add a plugin system to handle additional metadata extensions would be up to the pip devs. As a user, my main request if the pip devs decided to add such a plugin system would be that extension handlers couldn't be implicitly installed as a dependency of another package. If folks want their installs to just work, they shouldn't be marking non-standard metadata extensions as mandatory :) Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 16 April 2015 at 17:58, Chris Barker chris.bar...@noaa.gov wrote:
We have the same problem with optional dependencies.
For instance, for iPython to work, you don't need much. but if you want the
ipython notebook to work, you need tornado, zeromq, who knows what else. But
people want it to just work -- and just work be default, so you want all
that optional stuff to go in by default.
But none of those are installed by default with ipython - they are
covered by extras. If you want them, you ask for them. Thanks to
extras, ipython offers a nice shortcut for you - pip install
ipython[all] - but you still have to ask for them.
I expect this is the same with wheel installer extensions. To use your
example, for instance. People want to do:
pip install sphinx
and then have the sphinx-quickstart utility ready to go. by default. So
scripts need to be installed by default.
Yes, the script wrapper extension is a much better example of a
nobody would ever want this off extension. But is it really
required? The distribution would work fine if scripts weren't
installed. My understanding of the required/optional distinction in
the PEP is that an extension is required if the installation would be
*broken* if that extension wasn't supported. And having to type
python -m something rather than just something isn't broken, it's
just an inconvenience.
In practice, I'd assume script wrappers would be an extension built
into pip, so it would always be available. But that's different from
required. Installing via wheel install (which doesn't support
generating script wrappers) still works.
Actually, I just checked PEP 459, which says of the python.commands
extension: Build tools SHOULD mark this extension as requiring
handling by installers. So I stand corrected - script wrappers should
be considered mandatory. In practice, though, what that means is that
pip will be fine (as it'll have support built in) and wheel will be a
non-compliant installer (as it neither supports generating wrappers,
nor does it give an error when asked to create them - maybe an error
will get added, but I doubt it as wheel install isn't intended to be a
full installer). I've no idea at this point what distil or
easy_install will do, much less any other installers out there.
The trade-off between convenience and control/security is tough.
Certainly. But that's about what is available by default, and how the
installer (pip) handles the user interface for this package says that
it gives you some extra functionality if you have extensions X, Y, and
Z. There's no convenience or UI implication with required extensions
- if they aren't available, the installer refuses to install the
distribution. Simple as that. Maybe pip could try to locate and
download mandatory extensions before deciding whether to fail the
install, but the package metadata doesn't say how to find such
installer plugins (and it *can't* - because the plugin would be
different for pip, easy-install, distil or wheel, so all it can say is
I need plugin foo and the installer has to know the rest). That's an
installation program quality of implementation issue though.
Given that a random project could add metadata
extensions: {
foocorp.randomstuff: {
installer_must_handle: true,
foocorp.localdata: Some random stuff
}
}
there is no way that pip has any means of discovering where to get
code to handle the foocorp.randomstuff extension from. So in the
general case, auto-installing required extension support just isn't
possible. At best, pip could have a registry of plugins that support
standard extensions (i.e. those covered by a PEP) but I'd expect that
we'd just build them into pip (as we don't have a plugin interface at
the moment).
Based on the above, it's possibly valid to allow required extensions
to be auto-installed. It *is* a vector for unexpected code execution,
but maybe that's OK.
If even required extensions aren't auto installed, then we can just toss out
the whole idea of automatic dependency management. (which I personally
wouldn't mind, actually, but I'm weird that way)
I disagree. It's no different conceptually than the fact that if you
don't have a C compiler, you can't install a package that contains C
code and only comes in sdist format today. The UI in pip for noticing
and saying you need a C compiler is terrible (you get a build error
which might mention that you don't have the compiler, if you're lucky
:-)). And yet people survive. So a clear error saying package X needs
a handler for extension Y to install is a major improvement over the
current situation. (I know C compilers are build-step and extensions
are install-step, but right now the user experience doesn't clearly
distinguish these, so the analogy holds).
Whether users want pip to go one step further and auto-install the
plugin is unknown at this point. So far, it seems that the only people
who have expressed an opinion (you and I)
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Tue, Apr 14, 2015 at 8:57 PM, Kevin Horn kevin.h...@gmail.com wrote: Personally, I'm not a fan of auto-installing, I'm with Paul on this one. It seems to me that auto-installing the extension would destroy most of the advantages of distributing the extensions separately. Exactly -- I actually tossed that one out there because I wanted to know what folks were thinking, but also a bit of bait ;-) -- we've got a conflict here: 1) These possible extensions are potentially dangerous, etc, and should be well reviewed and not just tossed in there. 2) People (and I'm one of them) really, really want pip install to just work. (or conda install or enpkg, or...). If it's going to just work, then it needs to find and install the extensions auto-magically, and then we're really not very far from running arbitrary code... Would that be that different than the fact that installing a given package automatically installs all sorts of other packages -- and most of us don't give that a good review before running install... (I just was showing off Shinx to a class last night -- quite amazing what gets brought in with a pip install of sphinx (including pytz -- I have no idea why). But at the end of the day, I don't care much either. I'm trusting that the Sphinx folks aren't doing something ridiculous or dangerous. Which brings us back to the review of extensions thing -- I think it's less about the end user checking it out and making a decision about it, but about the package builder doing that. I have a package I want to be easy to install on Windows -- so I go look for an extension that does the Start Menu, etc. Indeed, that kind of thing 'should be part of pip and/or wheel, but it would probably be more successful if it were done as third party extensions -- perhaps over the years, the ones that rise to the top of usefulness can become standards. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On the Start Menu suggestion, I think that's a horrible idea. Pip is not the system package manager and it shouldn't be changing the system. Unversioned script launchers are in the same category, but aren't quite as offensive. I know it's only a hypothetical, but I'd much rather it didn't get repeated so often that it actually happens. There are better tools for making app installers, as opposed to package installers. Cheers, Steve Top-posted from my Windows Phone From: Paul Mooremailto:p.f.mo...@gmail.com Sent: 4/15/2015 17:24 To: Chris Barkermailto:chris.bar...@noaa.gov Cc: distutils-sigmailto:distutils-sig@python.org Subject: Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more On 15 April 2015 at 21:40, Chris Barker chris.bar...@noaa.gov wrote: Which brings us back to the review of extensions thing -- I think it's less about the end user checking it out and making a decision about it, but about the package builder doing that. I have a package I want to be easy to install on Windows -- so I go look for an extension that does the Start Menu, etc. Indeed, that kind of thing 'should be part of pip and/or wheel, but it would probably be more successful if it were done as third party extensions -- perhaps over the years, the ones that rise to the top of usefulness can become standards. In the PEP, there's a concept of optional vs required extensions. See https://www.python.org/dev/peps/pep-0426/#required-extension-handling. This is crucial - I've no problem if a particular extension is used by a project, as long as it's optional. I won't install it, so it's fine. It seems to me that pip *has* to ignore missing optional extensions, for this reason. Of course, that introduces the converse problem, which is how would people who might want that extension to be activated, know that a project used it? Critical extensions, on the other hand, are precisely that - the install won't run without them. I'd hope that critical extensions will only be used for things where the installation will be useless without them. But I worry that some people may have a more liberal definition of required than I do. To be honest, I can't think of *anything* that I'd consider a required extension. Console script wrappers aren't essential, for example (you can use python -m pip even if pip.exe isn't present). More generally, none of the extensions in PEP 459 seem essential, in this sense. Start menu entry writers wouldn't be essential, nor would COM registration extensions necessarily be (most of pywin32's functionality works fine if the COM components aren't registered). Beyond that I'm struggling to think of things that might be extensions. So, as long as the optional vs required distinction is respected, people are conservative about deeming something as essential, and a missing optional extension doesn't stop an install, then I don't see extensions as being a big issue. Based on the above, it's possibly valid to allow required extensions to be auto-installed. It *is* a vector for unexpected code execution, but maybe that's OK. Paul PS The idea of a Start Menu entries has come up a lot here. To be clear, I *don't* actually think such a thing is a good idea (far from it - I think it's a pretty lousy idea) but it is a good example of something that people think they ought to do, but in practice users have widely differing views on whet they prefer or will use, and a developer with limited experience could easily create a dreadful user experience without meaning to (developer here could either mean the extension developer, or the package developer using the extension - both have opportunities to make a horrible mess...) So it's a good straw man for an extension that some people will love and others will hate :-) PPS I'm inclined to think that the PEP should say Installation tools MUST NOT fail if installer_must_handle is set to false for an extension that the tool cannot process. Installation tools SHOULD NOT attempt to install plugins or similar optional functionality to handle an extension with installer_must_handle set to false, except with explicit approval from the end user. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 15 April 2015 at 21:40, Chris Barker chris.bar...@noaa.gov wrote: Which brings us back to the review of extensions thing -- I think it's less about the end user checking it out and making a decision about it, but about the package builder doing that. I have a package I want to be easy to install on Windows -- so I go look for an extension that does the Start Menu, etc. Indeed, that kind of thing 'should be part of pip and/or wheel, but it would probably be more successful if it were done as third party extensions -- perhaps over the years, the ones that rise to the top of usefulness can become standards. In the PEP, there's a concept of optional vs required extensions. See https://www.python.org/dev/peps/pep-0426/#required-extension-handling. This is crucial - I've no problem if a particular extension is used by a project, as long as it's optional. I won't install it, so it's fine. It seems to me that pip *has* to ignore missing optional extensions, for this reason. Of course, that introduces the converse problem, which is how would people who might want that extension to be activated, know that a project used it? Critical extensions, on the other hand, are precisely that - the install won't run without them. I'd hope that critical extensions will only be used for things where the installation will be useless without them. But I worry that some people may have a more liberal definition of required than I do. To be honest, I can't think of *anything* that I'd consider a required extension. Console script wrappers aren't essential, for example (you can use python -m pip even if pip.exe isn't present). More generally, none of the extensions in PEP 459 seem essential, in this sense. Start menu entry writers wouldn't be essential, nor would COM registration extensions necessarily be (most of pywin32's functionality works fine if the COM components aren't registered). Beyond that I'm struggling to think of things that might be extensions. So, as long as the optional vs required distinction is respected, people are conservative about deeming something as essential, and a missing optional extension doesn't stop an install, then I don't see extensions as being a big issue. Based on the above, it's possibly valid to allow required extensions to be auto-installed. It *is* a vector for unexpected code execution, but maybe that's OK. Paul PS The idea of a Start Menu entries has come up a lot here. To be clear, I *don't* actually think such a thing is a good idea (far from it - I think it's a pretty lousy idea) but it is a good example of something that people think they ought to do, but in practice users have widely differing views on whet they prefer or will use, and a developer with limited experience could easily create a dreadful user experience without meaning to (developer here could either mean the extension developer, or the package developer using the extension - both have opportunities to make a horrible mess...) So it's a good straw man for an extension that some people will love and others will hate :-) PPS I'm inclined to think that the PEP should say Installation tools MUST NOT fail if installer_must_handle is set to false for an extension that the tool cannot process. Installation tools SHOULD NOT attempt to install plugins or similar optional functionality to handle an extension with installer_must_handle set to false, except with explicit approval from the end user. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 14 April 2015 at 06:37, David Cournapeau courn...@gmail.com wrote: pywin32 is one of the most used package in the python ecosystem, and its post install script is not trivial. And yet pywin32's postinstall script is completely virtualenv-hostile. It registers start menu entries (not applicable when installing in a virtualenv), registers itself as a COM server (once again, one per machine), adds registry entries (again, virtualenv-hostile), moves installed files into the Windows system directory (ditto) etc. And yet for many actual uses of pywin32, installing as a wheel without running the postinstall is sufficient. With the exception of writing COM servers in Python (and maybe writing services, but I thing cx_Freeze lets you do that without pywin32), pretty much every use *I* have seen of pywin32 can be replaced with ctypes or cffi with no loss of functionality. I'd argue that pywin32 is a perfect example of a project where *not* supporting postinstall scripts would be a good idea, as it would encourage the project to find a way to implement the same functionality in a way that's compatible with current practices (virtualenv, tox, etc). Or it would encourage other projects to stop depending on pywin32 (which is actually what is happening, many projects now use ctypes and similar in place of pywin32-using code, to avoid the problems pywin32 causes for them). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Tue, Apr 14, 2015 at 8:41 AM, Nick Coghlan ncogh...@gmail.com wrote: The main two language independent solutions I've identified for this general user level package management problem in the Fedora Environments Stacks context ( https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement ) are conda (http://conda.pydata.org/) and Nix (https://nixos.org/nix/about.html), cool -- I hadn't seem nix before. From a Python upstream perspective, Nix falls a long way behind conda due to the fact that Nix currently relies on Cygwin for Windows support - The other thing that's nice about conda is that while it was designed for the general case, it has a lot of python-specific features. Being a Python guy -- I llke that ;-) -- it may not work nearly as well for Ruby or what have you -- I wouldn't know. The point where I draw the line is supporting *dynamic* linking between modules - I'm confused -- you don't want a system to be able to install ONE version of a lib that various python packages can all link to? That's really the key use-case for me that's the capability I view as defining the boundary between enabling an add-on ecosystem for a programming language runtime and providing a comprehensive software development platform :) Well, with it's target audience being scientific programmers, conda IS trying to give you a comprehensive software development platform We're talking about Python here -- it's a development tool. It turns out that for scientific development, pure python is simply not enough -- hence the need for conda and friends. I guess this is what it comes down to -- I'm all for adding a few features to wheel -- it would be nice to be abel to pip install most of what I, and people like me, need. But maybe it's not possible -- you can solve the shared lib problem, and the scripts problem, and maybe the menu entires problem, but eventually, you end up with I want to use numba -- and then you need LLVM, etc. -- and pretty soon you are building a tool that provides a comprehensive software development platform. ;-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 14 April 2015 at 17:10, Chris Barker chris.bar...@noaa.gov wrote: If there’s a plugin that understands this extension installed, let it do something before you actually move the files into place”. This let’s Wheels themselves still be declarative and moves the responsibility of implementing these bits into their own PyPI projects that can be versioned and independently upgraded and such. We’d probably need some method of marking an extension as “critical” (e.g. bail out and don’t install this Wheel if you don’t have something that knows how to handle it) and then non critical extensions just get ignored if we don’t know how to handle it. Could an extension be -- run this arbitrary Python script ? The main point (as I see it) of an extension is that it's distributed independently of the packages that use it. So you get to decide to use an extension (and by inference audit it if you want) *before* it gets run as part of an installation. Extensions get peer review by the community, and bad ones get weeded out, in a way that just having a chunk of code in your setup.py or the postinstall section of your wheel doesn't. We've got a full featured scripting language (with batteries included!) -- isn't that all the extension you need? Up to a point yes. It's the independent review and quality control aspects that matter to me. Or is this about security? We don't want to let a package do virtually anything on install? Security is one aspect, and one that a lot of people will pick up on immediately. But there's also portability. And code quality. And consistency. I'd be much happier installing a project that used a well-known start menu manager extension than one that just used custom code. I'd be willing to assume that the author of the extension had thought about Unix/Windows compatibility, how to handle use in a virtualenv, handling user preferences (such as the end user *not wanting* shortcuts), etc etc. And I could look at the extension project's issue tracker to see how happy I was with the state of the project. Of course, if the project I want to install makes using the extension mandatory for the install to work, I still don't have a real choice - I accept the extension or I can't use the code I want - but there's an extra level of transparency involved. And hopefully most extensions will be optional, in practice. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 13 April 2015 at 12:56, Chris Barker chris.bar...@noaa.gov wrote: Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to simply pip install pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda. The main two language independent solutions I've identified for this general user level package management problem in the Fedora Environments Stacks context (https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement) are conda (http://conda.pydata.org/) and Nix (https://nixos.org/nix/about.html), backed up by Pulp for plugin-based format independent repository management (http://www.pulpproject.org/). From a Python upstream perspective, Nix falls a long way behind conda due to the fact that Nix currently relies on Cygwin for Windows support - it's interesting to me for Fedora because Nix ticks a lot of boxes from a system administrator perspective that conda doesn't (in particular, system administrators can more easily track what users have installed, and ensure that packages are updated appropriately in the face of security updates in dependencies). I definitely see value in Python upstream formats being able to bundle additional files like config files, desktop integration files, service definition files, statically linked extensions modules, etc, in a way that not only supports direct installation onto end user machines, but also conversion into platform specific formats (whether that platform is an operating system, or a cross-platform platform like nix, canopy or conda). The point where I draw the line is supporting *dynamic* linking between modules - that's the capability I view as defining the boundary between enabling an addon ecosystem for a programming language runtime and providing a comprehensive software development platform :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 13 April 2015 at 22:29, Donald Stufft don...@stufft.io wrote: So a possible way for this to work is in a PEP 426 world, simply define a twisted.plugins extension that says, in a declarative way, “hey when you install this Wheel, if there’s a plugin that understands this extension installed, let it do something before you actually move the files into place”. This let’s Wheels themselves still be declarative and moves the responsibility of implementing these bits into their own PyPI projects that can be versioned and independently upgraded and such. We’d probably need some method of marking an extension as “critical” (e.g. bail out and don’t install this Wheel if you don’t have something that knows how to handle it) and then non critical extensions just get ignored if we don’t know how to handle it. Right, this is the intent of the Required extension handling feature: https://www.python.org/dev/peps/pep-0426/#required-extension-handling If a package flags an extension as installer_must_handle, then attempts to install that package are supposed to fail if the installer doesn't recognise the extension. Otherwise, installers are free to ignore extensions they don't understand. So meta-installers like canopy could add their own extensions to their generated wheel files, flag those extensions as required, and other installers would correctly reject those wheels as unsupported. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
If there’s a plugin that understands this extension installed, let it do something before you actually move the files into place”. This let’s Wheels themselves still be declarative and moves the responsibility of implementing these bits into their own PyPI projects that can be versioned and independently upgraded and such. We’d probably need some method of marking an extension as “critical” (e.g. bail out and don’t install this Wheel if you don’t have something that knows how to handle it) and then non critical extensions just get ignored if we don’t know how to handle it. Could an extension be -- run this arbitrary Python script ? We've got a full featured scripting language (with batteries included!) -- isn't that all the extension you need? Or is this about security? We don't want to let a package do virtually anything on install? -CHB Right, this is the intent of the Required extension handling feature: https://www.python.org/dev/peps/pep-0426/#required-extension-handling If a package flags an extension as installer_must_handle, then attempts to install that package are supposed to fail if the installer doesn't recognise the extension. Otherwise, installers are free to ignore extensions they don't understand. So meta-installers like canopy could add their own extensions to their generated wheel files, flag those extensions as required, and other installers would correctly reject those wheels as unsupported. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Tue, Apr 14, 2015 at 4:19 PM, Paul Moore p.f.mo...@gmail.com wrote: On 14 April 2015 at 22:02, Chris Barker chris.bar...@noaa.gov wrote: Personally, I'm not a fan of auto-installing, so I'd hope for something more like pip would fail to install if a required extension were missing. The user would then install the extension and redo the install. But that may be a minority opinion - it's a bit like setup_requires in principle, and people seem to prefer that to be auto-installed. (lurker surfaces) I'm with Paul on this one. It seems to me that auto-installing the extension would destroy most of the advantages of distributing the extensions separately. I _might_ not hate it if pip prompted the user and _then_ installed, but then again, I might. (lurker sinks back into the depths) -- Kevin Horn ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 14 April 2015 at 22:02, Chris Barker chris.bar...@noaa.gov wrote: - pip auto-installs it (if not already there) when the user goes to install the wheel. Personally, I'm not a fan of auto-installing, so I'd hope for something more like pip would fail to install if a required extension were missing. The user would then install the extension and redo the install. But that may be a minority opinion - it's a bit like setup_requires in principle, and people seem to prefer that to be auto-installed. Paul Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Tue, Apr 14, 2015 at 9:46 AM, Paul Moore p.f.mo...@gmail.com wrote: Could an extension be -- run this arbitrary Python script ? The main point (as I see it) of an extension is that it's distributed independently of the packages that use it. So you get to decide to use an extension (and by inference audit it if you want) *before* it gets run as part of an installation. OK, I think this is getting clearer to me now -- an Extension is (I suppose arbitrary) block of python code, but what goes into the wheel is not the code, but rather a declarative configuration for the extension. then at install-time, the actual code that runs is separate from the wheel, which gives the end user greater control, plus these nifty features Extensions get peer review by the community, and bad ones get weeded out, the independent review and quality control there's also portability. And code quality. And consistency. And I'll add that this would promote code re-use and DRY. I'd be much happier installing a project that used a well-known start menu manager extension So where would that code live? and how would it be managed? I'm thinking: - In package on PyPi like anything else - a specification in install_requires - pip auto-installs it (if not already there) when the user goes to install the wheel. Is that the idea? Of course, if the project I want to install makes using the extension mandatory for the install to work, I still don't have a real choice - I accept the extension or I can't use the code I want - well, you can't easily auto-install it anyway -- you could still do a source install, presumably. but there's an extra level of transparency involved. And hopefully most extensions will be optional, in practice. There's a bit to think about in the API/UI here. If an installation_extension is used by a package, and it's specified in install_requires, then it's going to get auto-magically installed an used with a regular old pip install. If we are worried about code review and users being in control of what extensions they use, then how to we make it obvious that a given extension is in use, but optional, and how to turn it off if you want? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. We've used eggs for that purpose for almost 10 years at Enthought, and we did not need that many extensions on top of the egg format after all. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). IMO, given that wheels do most of what's needed, it is worth supporting most simple usecases (compiled libraries required by well known extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to quite horrible custom hacks to support those cases. Hope that clarifies the intent, David I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to simply pip install pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda. BTW, while you can't generally install a conda package in/for another python, you can generally install a wheel in a conda pythonThere are a few issues with
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to simply pip install pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda. BTW, while you can't generally install a conda package in/for another python, you can generally install a wheel in a conda pythonThere are a few issues with pip/setuptools trying to resolve dependencies while not knowing about conda packages, but it does mostly work. Not sure that helped the discussion -- but I've been wrestling with this for a while, so thought I'd get my thoughts out there. I've always thought of wheel as solving only the Python-specific problem. Providing relocatable Python-specific packaging without trying to solve the intractable problem of non-Python dependencies. The strategy works best the more you are targeting Python as your platform and not a specific OS or distribution - sometimes it works well, other times not at all. Obviously if you need a specific build of PostgreSQL wheel isn't going to help you. With enough hacks you could make it work but are we ready to pip install kde? I don't think so. Personally I'm happy to let other tools solve the problem of C-level
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. We've used eggs for that purpose for almost 10 years at Enthought, and we did not need that many extensions on top of the egg format after all. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). IMO, given that wheels do most of what's needed, it is worth supporting most simple usecases (compiled libraries required by well known extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to quite horrible custom hacks to support those cases. Hope that clarifies the intent, David Then it sounds like I should read about the Enthought egg extensions. It's something else than just defining a separate pypi name for just the libxml.so without the python bits? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 1:19 PM, David Cournapeau courn...@gmail.com wrote: This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows Are these inside or outside the python installation? I'm more than a bit wary of a wheel that would install stuff outside of the sandbox of the python install. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. hmm -- true. I guess where it seems to get more complicated is beyond the wheel (or conda, or...) package itself, to the dependency management, installation tools, etc. But perhaps you are suggesting that we can extend wheel to support a bt more stuff, and leave the rest of the system as separate problem? i.e. Canopy can have it's own find, install, manage-dependency tool, but that it can use the wheel format for the packages themselves? I don't see why not -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
I would advise against using or even reading about our egg extensions, as the implementation is full of legacy (we've been doing this many years :) ): http://enstaller.readthedocs.org/en/master/reference/egg_format.html This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows - a few extensions to the actual binary format (adding support for symlinks is the only one I can think of ATM). Everything else is legacy you really don't want to know (see here if you still want to http://enstaller.readthedocs.org/en/master/reference/egg_format.html) David On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth dho...@gmail.com wrote: On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. We've used eggs for that purpose for almost 10 years at Enthought, and we did not need that many extensions on top of the egg format
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 5:25 PM, Chris Barker chris.bar...@noaa.gov wrote: On Mon, Apr 13, 2015 at 1:19 PM, David Cournapeau courn...@gmail.com wrote: This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows Are these inside or outside the python installation? I'm more than a bit wary of a wheel that would install stuff outside of the sandbox of the python install. I would always install things relative to sys.prefix, for exactly the reasons you mention. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. hmm -- true. I guess where it seems to get more complicated is beyond the wheel (or conda, or...) package itself, to the dependency management, installation tools, etc. But perhaps you are suggesting that we can extend wheel to support a bt more stuff, and leave the rest of the system as separate problem? i.e. Canopy can have it's own find, install, manage-dependency tool, but that it can use the wheel format for the packages themselves? Exactly ! David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
Seems like you could extend wheel to do that easily. On Apr 13, 2015 4:19 PM, David Cournapeau courn...@gmail.com wrote: I would advise against using or even reading about our egg extensions, as the implementation is full of legacy (we've been doing this many years :) ): http://enstaller.readthedocs.org/en/master/reference/egg_format.html This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows - a few extensions to the actual binary format (adding support for symlinks is the only one I can think of ATM). Everything else is legacy you really don't want to know (see here if you still want to http://enstaller.readthedocs.org/en/master/reference/egg_format.html) David On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth dho...@gmail.com wrote: On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Apr 13, 2015, at 10:17 PM, Robert Collins robe...@robertcollins.net
wrote:
On 14 April 2015 at 09:35, David Cournapeau courn...@gmail.com wrote:
...
One of the earlier things mentioned here - {pre,post}{install,remove}
scripts - raises a red flag for me.
In Debian at least, the underlying system has the ability to run such
turing complete scripts, and they are a rich source of bugs - both
correctness and performance related.
Nowadays nearly all such scripts are machine generated from higher
level representations such as 'this should be the default command' or
'configure X if Y is installed', but because the plumbing is turing
complete, they all need to be executed, which slows down
install/upgrade paths, and any improvement to the tooling requires a
version bump on *all* the packages using it - because effectively the
package is itself a compiled artifact.
I'd really prefer it if we keep wheels 100% declarative, and instead
focus on defining appropriate metadata for the things you need to
accomplish {pre,post}{install,remove} of a package.
A possible way to implement {pre,post}{install,remove} scripts is to
instead turn them into extensions. One example is that Twisted uses
a setup.py hack to regenerate a cache file of all of the registered
plugins. This needs to happen at install time due to permission issues.
Currently you can’t get this speedup when installing something that
uses twisted plugins and installing via Wheel.
So a possible way for this to work is in a PEP 426 world, simply define
a twisted.plugins extension that says, in a declarative way, “hey when
you install this Wheel, if there’s a plugin that understands this extension
installed, let it do something before you actually move the files into
place”. This let’s Wheels themselves still be declarative and moves the
responsibility of implementing these bits into their own PyPI projects
that can be versioned and independently upgraded and such. We’d probably
need some method of marking an extension as “critical” (e.g. bail out and
don’t install this Wheel if you don’t have something that knows how to handle
it) and then non critical extensions just get ignored if we don’t know
how to handle it.
Popular extensions could possibly be added directly to pip at some point
if a lot of people are using them (or even moved from third party extension
to officially supported extension).
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 14 April 2015 at 09:35, David Cournapeau courn...@gmail.com wrote:
...
One of the earlier things mentioned here - {pre,post}{install,remove}
scripts - raises a red flag for me.
In Debian at least, the underlying system has the ability to run such
turing complete scripts, and they are a rich source of bugs - both
correctness and performance related.
Nowadays nearly all such scripts are machine generated from higher
level representations such as 'this should be the default command' or
'configure X if Y is installed', but because the plumbing is turing
complete, they all need to be executed, which slows down
install/upgrade paths, and any improvement to the tooling requires a
version bump on *all* the packages using it - because effectively the
package is itself a compiled artifact.
I'd really prefer it if we keep wheels 100% declarative, and instead
focus on defining appropriate metadata for the things you need to
accomplish {pre,post}{install,remove} of a package.
-Rob
--
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 10:17 PM, Robert Collins robe...@robertcollins.net
wrote:
On 14 April 2015 at 09:35, David Cournapeau courn...@gmail.com wrote:
...
One of the earlier things mentioned here - {pre,post}{install,remove}
scripts - raises a red flag for me.
That's indeed a good a priori. I myself removed a lot of those scripts
because of the fragility. Anything that needs to run on a end-user machine
can fail, and writing idempotent scripts is hard.
Unfortunately, pure declarative does not really cut it if you want cross
platform support. Sure, you may be able to deal with menu entries,
environment variables, etc... in a cross-platform manner with a significant
effort, but what about COM registration ? pywin32 is one of the most used
package in the python ecosystem, and its post install script is not
trivial. Another difficult case if when a package needs some specific
configuration to run at all, and that configuration requires values known
at install time only (e.g. sys.prefix, as in the iris package).
I'd really prefer it if we keep wheels 100% declarative, and instead
focus on defining appropriate metadata for the things you need to
accomplish {pre,post}{install,remove} of a package.
What about a way for wheels to specify whether their
post,pre/install,remove scripts are declarative or not, with support for
the most common tasks, with an escape, but opt-in mechanism ? This way it
could be a matter of policy to refuse packages that require non-declarative
scripts.
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft don...@stufft.io wrote: On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote: Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento ( https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. To my knowledge, (1) was purposely punted until a later revision of Wheel just to make it easier to land the “basic” wheel. Great. Was there any proposal made to support it at all ? Or should I just work from scratch there ? I think (2) is a reasonable thing as long as we can map it sanely on all platforms. Yes. We support all platforms at Enthought, and Windows is important for us ! I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python More generally, we could be modifying python in a way that is not forward compatible with upstream python: a binary that works on our python may not work on the python from python.org (though the opposite is true). It would be nice if one could make sure pip will not try to install those eggs when installed on top of a python that does not advertise itself as compatible. David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Beyond wheels 1.0: helping downstream, FHS and more
Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento ( https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. thanks, David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote: Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. To my knowledge, (1) was purposely punted until a later revision of Wheel just to make it easier to land the “basic” wheel. I think (2) is a reasonable thing as long as we can map it sanely on all platforms. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to simply pip install pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda. BTW, while you can't generally install a conda package in/for another python, you can generally install a wheel in a conda pythonThere are a few issues with pip/setuptools trying to resolve dependencies while not knowing about conda packages, but it does mostly work. Not sure that helped the discussion -- but I've been wrestling with this for a while, so thought I'd get my thoughts out there. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
#1 is pretty straightforward. An entry-point format Python
pre/post/etc. script may do.
I have some ideas for the FHS, though I fear it's full of bikesheds:
1. Allow all GNU directory variables as .data/* subdirectories
(https://www.gnu.org/prep/standards/html_node/Directory-Variables.html).
The distutils names will continue to be allowed.
packagename-1.0.data/mandir/...
2. Make data_files useful again. Interpolate path variables into
distutils data_files using $template syntax. (Only allow at
beginning?)
data_files=[('$mandir/xyz', ['manfile', 'other_man_file'])
In addition to $bindir, $mandir, etc. it will be important to allow
the package name and version to be interpolated into the install
directories.
Inside the wheel archive, you will get
packagename-1.0.data/mandir/manfile and
packagename-1.0.data/mandir/other_man_file
3. Write the install paths (the mapping from $bindir, $mandir, $prefix
etc. to the actual paths used) to one or more of a .py, .json, or
.dist-info/* based on new metadata in WHEEL:
install-paths-to: wheel/_paths.py
It is critical that this be allowed to work without requiring the end
user to look for it with pkg_resources or its pals. It's also good to
only write it if the installed package actually needs to locate its
file categories after it has been installed.
This will also be written inside the wheel itself with relative paths
to the .data/ directory.
4. Allow configurable custom paths. The GNU paths could be
configured relative to the distutils paths as a default. We might let
the user add additional paths with a configuration dict.
paths = {
foo : $bar/${quux},
bar : ${baz}/more/stuff,
baz : ${quux}/again,
quux: larry
}
5. On Windows, no one will really care where most of these files go,
but they probably won't mind if they are installed into separate
directories. Come up with sensible locations for the most important
categories.
On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft don...@stufft.io wrote:
On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote:
Hi there,
During pycon, Nick mentioned there was interest in updating the wheel format
to support downstream distributions. Nick mentioned Linux distributions, but
I would like to express interest for other kind of downstream distributors
like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work
for Enthought).
Right now, wheels have the following limitations for us:
1. lack of post/pre install/removing
2. more fine-grained installation scheme
3. lack of clarify on which tags vendors should use for custom wheels: some
packages we provide would not be installable on normal python, and it
would be nice to have a scheme to avoid confusion there as well.
At least 1. and 2. are of interest not just for us.
Regarding 2., it looks like anything in the wheel_name.data/data directory
will be placed as is in sys.prefix by pip. This is how distutils scheme is
defined ATM, but I am not sure whether that's by design or accident ?
I would suggest to use something close to autotools, with some tweaks to
work well on windows. I implemented something like this in my project bento
(https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
but we could of course tweak that.
For 1., I believe it was a conscious decision not to include them in wheel
1.0 ? Would it make sense to start a discussion to add it to wheel ?
I will be at the pycon sprints until wednesday evening, so that we can flesh
some concrete proposal first, if there is enough interest.
As a background: at Enthought, we have been using eggs to distribute
binaries of python packages and other packages (e.g. C libraries, compiled
binaries, etc...) for a very long time. We had our own extensions to the egg
format to support this, but I want to get out of eggs so as to make our own
software more compatible with where the community is going. I would also
like to avoid making ad-hoc extensions to wheels for our own purposes.
To my knowledge, (1) was purposely punted until a later revision of Wheel
just to make it easier to land the “basic” wheel.
I think (2) is a reasonable thing as long as we can map it sanely on all
platforms.
I’m not sure what (3) means exactly. What is a “normal” Python, do you
modify Python in a way that breaks the ABI but which isn’t reflected in the
standard ABI tag?
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 13 April 2015 at 16:02, Daniel Holth dho...@gmail.com wrote: #1 is pretty straightforward. An entry-point format Python pre/post/etc. script may do. There's metadata 2.0 information for this. It would be sensible to follow that definition where it applies, but otherwise yes, this shouldn't be hard. Some thoughts, though: 1. Some thought should be put into how we ensure that pre/post install/remove scripts are cross-platform. It would be a shame if a wheel was unusable on Windows for no reason other than that the postinstall script was written as a bash script. Or on Unix because the postinstall script tried to write Windows start menu items. 2. It's worth considering appropriate use of such scripts. The Windows start menu example is relevant here - I can easily imagine users requesting something like that for a wheel they want to install into the system Python, but it's completely inappropriate for installing into a virtualenv. To an extent, there's nothing we can (or maybe even should) do about this - projects that include inappropriate install scripts will get issues raised or will lose users, and the problem is self-correcting to an extent, but it's probably worth including in the implementation, some work to add appropriate documentation to the packaging user guide about best practices for pre/post-install/remove scripts (hmm, a glossary entry with a good name for these beasts would also be helpful :-)) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 10:54 AM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft don...@stufft.io wrote: On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote: Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. To my knowledge, (1) was purposely punted until a later revision of Wheel just to make it easier to land the “basic” wheel. Great. Was there any proposal made to support it at all ? Or should I just work from scratch there ? I think (2) is a reasonable thing as long as we can map it sanely on all platforms. Yes. We support all platforms at Enthought, and Windows is important for us ! I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python More generally, we could be modifying python in a way that is not forward compatible with upstream python: a binary that works on our python may not work on the python from python.org (though the opposite is true). It would be nice if one could make sure pip will not try to install those eggs when installed on top of a python that does not advertise itself as compatible We need a hook to alter pip's list of compatible tags (pip.pep425tags), and to alter the default tags used by bdist_wheel when creating wheels. One sensible proposal for special wheels is to just use a truncated hash of the platform description (a random hex string) in place of the wheel platform tag. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
