Re: [Distutils] Q about best practices now (or near future)
On Sun, Jul 21, 2013 at 6:44 PM, Nick Coghlan wrote: > > On 22 Jul 2013 01:46, "PJ Eby" wrote: >> >> Now that I'm thinking about it some more, one of the motivating use >> cases for extras in entry points was startup performance in >> plugin-heavy GUI applications like Chandler. The use of extras allows >> for late-loading of additions to sys.path. IOW, it's intended more >> for a situation where not only are the entry points imported late, but >> you also want as few plugins as possible on sys.path to start with, in >> order to have fast startup. > > I'm working with Eric Snow on a scheme that I hope will allow > module-specific path entries that aren't processed at interpreter startup > and never get added to sys.path at all (even if you import the module). > Assuming we can get it to work the way I hope (which is still a "maybe" at > this point in time), it should then be possible to backport it to earlier > versions as a metaimporter. I haven't had a chance to look at that proposal at more than surface depth, but my immediate concern with it is that it seems to be at the wrong level of abstraction for the packaging system, i.e., just because you can import a module, doesn't mean you can get at its project metadata (e.g., how would you find its exports, or even know what distribution it belonged to?). (Also, I don't actually see how it would be useful or relevant to the use case we're talking about; it seems maybe orthogonal at best.) > OK, so as Daniel suggested, it's more like an export/entry-point specific > "requires" field, but limited to the extras of the current distribution. Correct: at the time, it seemed a lot simpler to me than supporting arbitrary requirements, and allows for more DRY, since entry points might share some requirements. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 22 Jul 2013 01:46, "PJ Eby" wrote: > > On Sat, Jul 20, 2013 at 10:54 PM, Nick Coghlan wrote: > > Extras should just be a way to ask "are these optional dependencies present > > on this system?", without needing to worry about how they got there. > > Technically, they are a way to ask "can you get this for me?", since > pkg_resources' API allows you to specify an installer callback when > you ask to load an entry point. This means that an installer tool can > dynamically obtain any extras it needs, not just check for their > installation. > > To put it another way, it's not "exported only if extra is available", > it's "exported, but make sure you have this first." A subtle > difference, but important to the original use cases (see below). Ah, yes, I see the distinction (and it does make this notion conceptually simpler). > > > > For now, I'll switch export specifiers back to the concise > > "modulename:qualname" entry point format and add "Do we need to > > support the exported-only-if-extra-is-available feature?" as an open > > question. My current thinking is that the point you made about script > > wrappers (putting the wrapper in separate distribution and depending > > on that from an extra) applies to other plugins as well. > > Now that I'm thinking about it some more, one of the motivating use > cases for extras in entry points was startup performance in > plugin-heavy GUI applications like Chandler. The use of extras allows > for late-loading of additions to sys.path. IOW, it's intended more > for a situation where not only are the entry points imported late, but > you also want as few plugins as possible on sys.path to start with, in > order to have fast startup. I'm working with Eric Snow on a scheme that I hope will allow module-specific path entries that aren't processed at interpreter startup and never get added to sys.path at all (even if you import the module). Assuming we can get it to work the way I hope (which is still a "maybe" at this point in time), it should then be possible to backport it to earlier versions as a metaimporter. > The other use case is similar, in that a plugin-heavy environment with > self-upgrading abilities can defer *installation* of parts of a > plug-in until it is actually used. (Which is why EntryPoint objects > have a .require() method separate from .load() - you can loop over a > relevant set of entry points to pre-test or pre-ensure that they're > all available and dependencies are installed before importing any of > them, even though .load() will also do that for a single entry point.) OK, so as Daniel suggested, it's more like an export/entry-point specific "requires" field, but limited to the extras of the current distribution. > For the specific case of the meta build system itself, these use cases > may be moot. For the overall use of exports, however, the use cases > are still valuable for plugin-heavy apps. (Specifically, applications > that use lots of plugins written by different people, and don't want > to have to import everything at startup.) > > Indeed, this is the original use case for exports in the first place: > it's a plugin system that doesn't require importing any plugins until > you actually need a particular plugin's functionality. Extras just > expand that slightly to "don't require installing things or putting > them on sys.path until you need their functionality". OK, I understand the use case now. If I can come up with a relatively simple way to explain it, I'll keep it in the proposed metadata, otherwise I'll leave it to metadata extensions to handle the more sophisticated version where an export depends on an extra. Cheers, Nick. > > Heck, if pip itself were split into two distributions, one of which > were a command line script declared with an extra, pointing into the > second distribution, it'd have dynamic bootstrapping. (Were it not > for the part where it would need pip available to *do* the > bootstrapping, of course. ;-) ) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sun, Jul 21, 2013 at 12:10 PM, Paul Moore wrote: > On 21 July 2013 16:46, PJ Eby wrote: >> >> Now that I'm thinking about it some more, one of the motivating use >> cases for extras in entry points was startup performance in >> plugin-heavy GUI applications like Chandler. The use of extras allows >> for late-loading of additions to sys.path. IOW, it's intended more >> for a situation where not only are the entry points imported late, but >> you also want as few plugins as possible on sys.path to start with, in >> order to have fast startup. > > > This type of complexity is completely outside of my experience. So I'm going > to have to defer to people who understand the relevant scenarios to assess > any proposed solutions. > > But could I make a general plea for an element of "keep the simple cases > simple" in both the PEP and the implementations, here? I think it's critical > that we make sure that the 99% of users[1] who want to do nothing more than > bundle up an app with a few dependencies can both understand the mechanisms > for doing so, and can use them straight out of the box. > > Paul > > [1] Yes, that number is made up - but to put it into context, I don't > believe I've ever used a distribution from PyPI with entry points depending > on extras. In fact, the only case I know of where I've seen extras in *any* > context is in wheel, and I've never used them even there. The extras system is simple and more importantly ubiquitous with tens of thousands of releases taking advantage of it. The proposed system of also having separate kinds of build and test dependencies with their own extras hasn't been demonstrated. Entry points having extras is an extension of entry points having simple distribution-level dependencies "entry point depends on beaglevote" -> "entry point depends on beaglevote[doghouse]". I can see how perhaps in the setuptools case it may have been more straightforward to include extras rather than to exclude them. Someone else may want to check again but the last time I checked all pypi-hosted distributions for "entry points depending on extras" I found none. It would be pretty safe to leave this particular feature out. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 21 July 2013 16:46, PJ Eby wrote: > Now that I'm thinking about it some more, one of the motivating use > cases for extras in entry points was startup performance in > plugin-heavy GUI applications like Chandler. The use of extras allows > for late-loading of additions to sys.path. IOW, it's intended more > for a situation where not only are the entry points imported late, but > you also want as few plugins as possible on sys.path to start with, in > order to have fast startup. > This type of complexity is completely outside of my experience. So I'm going to have to defer to people who understand the relevant scenarios to assess any proposed solutions. But could I make a general plea for an element of "keep the simple cases simple" in both the PEP and the implementations, here? I think it's critical that we make sure that the 99% of users[1] who want to do nothing more than bundle up an app with a few dependencies can both understand the mechanisms for doing so, and can use them straight out of the box. Paul [1] Yes, that number is made up - but to put it into context, I don't believe I've ever used a distribution from PyPI with entry points depending on extras. In fact, the only case I know of where I've seen extras in *any* context is in wheel, and I've never used them even there. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sat, Jul 20, 2013 at 10:54 PM, Nick Coghlan wrote: > Extras should just be a way to ask "are these optional dependencies present > on this system?", without needing to worry about how they got there. Technically, they are a way to ask "can you get this for me?", since pkg_resources' API allows you to specify an installer callback when you ask to load an entry point. This means that an installer tool can dynamically obtain any extras it needs, not just check for their installation. To put it another way, it's not "exported only if extra is available", it's "exported, but make sure you have this first." A subtle difference, but important to the original use cases (see below). > For now, I'll switch export specifiers back to the concise > "modulename:qualname" entry point format and add "Do we need to > support the exported-only-if-extra-is-available feature?" as an open > question. My current thinking is that the point you made about script > wrappers (putting the wrapper in separate distribution and depending > on that from an extra) applies to other plugins as well. Now that I'm thinking about it some more, one of the motivating use cases for extras in entry points was startup performance in plugin-heavy GUI applications like Chandler. The use of extras allows for late-loading of additions to sys.path. IOW, it's intended more for a situation where not only are the entry points imported late, but you also want as few plugins as possible on sys.path to start with, in order to have fast startup. The other use case is similar, in that a plugin-heavy environment with self-upgrading abilities can defer *installation* of parts of a plug-in until it is actually used. (Which is why EntryPoint objects have a .require() method separate from .load() - you can loop over a relevant set of entry points to pre-test or pre-ensure that they're all available and dependencies are installed before importing any of them, even though .load() will also do that for a single entry point.) For the specific case of the meta build system itself, these use cases may be moot. For the overall use of exports, however, the use cases are still valuable for plugin-heavy apps. (Specifically, applications that use lots of plugins written by different people, and don't want to have to import everything at startup.) Indeed, this is the original use case for exports in the first place: it's a plugin system that doesn't require importing any plugins until you actually need a particular plugin's functionality. Extras just expand that slightly to "don't require installing things or putting them on sys.path until you need their functionality". Heck, if pip itself were split into two distributions, one of which were a command line script declared with an extra, pointing into the second distribution, it'd have dynamic bootstrapping. (Were it not for the part where it would need pip available to *do* the bootstrapping, of course. ;-) ) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
From: Paul Moore > On 18 July 2013 08:57, Nick Coghlan wrote: >> Shipping an msi installer for pip (perhaps bundling with setuptools) >> would also be an acceptable alternative. > > -1. > > I would suggest that this approach, if it were considered seriously, should be > reviewed carefully by someone who understands MSI installers (not me!). > Specifically, if I install pip via an MSI, then use "python -m pip install -U > pip", will the "Add/Remove Programs" entry created by the MSI still uninstall > cleanly? Broken uninstall options and incomplete package removals are a > perennial problem on Windows, usually caused by messing with installed files > outside control of the installer. > Paul Also -1, and I've spent quite a lot of time writing MSIs recently... It could be solved, but wheels are a better fix for the problems that people solve with MSIs. MSIs are also useless when virtualenvs are involved, since there's basically a guarantee that its metadata will get out of sync with reality as soon as someone deletes the virtualenv. IMHO bundling pip (and all dependencies) with the installer is best. Any bootstrap script hitting the internet will need to pin the version, so you may as well include a zip of the files and extract them on install. That way you'll always get a pip that can upgrade itself, and if you do a repair install you'll get a working pip back. Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
