Re: [Distutils] Adding entry points into Distutils ?
On Tue, May 5, 2009 at 1:57 AM, Ian Bicking i...@colorstudy.com wrote: Not strong, but I have a few issues with how they are currently defined: * There's the issue of activated and unactivated eggs, of course, but I guess that will be moot since there's no activation with just distutils? Yes * There's no idea of explicitly enabling an entry point, simply installing a package makes the entry point show up. Implicit plugins make me uncomfortable. I don't see entry points as plugins, but rather the registering of a given piece of code, under a unique name. If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ? The way I see entry points is potential plugins, an application can decide to consume, and turn into a real plugin when it uses it. And an entry point that would be disabled is an entry point that is not used from the application A point of view, but might be used in the application B. (unless I misunderstood the concept of group) So enabling/disabling an entry point and keeping track of the activation state should be done by the host application ihmo. * There's not much in the way of entry point documentation built into the system. The __doc__ of the entry point objects is one way, but there's no way to document entry point groups. very good point, -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
2009/5/5 P.J. Eby p...@telecommunity.com: At 06:57 PM 5/4/2009 -0500, Ian Bicking wrote: * I'm uncomfortable with the way entry points are scanned. I haven't looked close enough to back it up with numbers, but I think there's a noticeable performance degradation when the number of installed packages becomes large. (Given the algorithm this would be expected.) It's linear in the number of entry_points.txt files, yes, but in most apps it should occur at most once, since pkg_resources has a single WorkingSet object holding Distribution objects which cache their entry point data upon first access. There are all sorts of ways you could make different tradeoffs, but this particular set of tradeoffs was optimized for a single-application environment, rather than a massive global shared site-packages where there are plugins for every application on the system. It was also optimized for the zipimport case, because you can tell whether a project has entry points from its cached zip directory, that's needed at startup anyhow. Would it make sense then to maintain an global index of all entry points, that would be updated upon installation / uninstallation (if it's added like we wrote in PEP 376) rather than scanning the paths everytime ? we could have one index file per site-package-like directory, and discover index files rather than all directories / zip files. Extra paths added in sys.path would be omited but I don't see it as a problem -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On May 5, 2009, at 4:49 AM, Tarek Ziadé wrote: On Tue, May 5, 2009 at 1:57 AM, Ian Bicking i...@colorstudy.com wrote: Not strong, but I have a few issues with how they are currently defined: * There's the issue of activated and unactivated eggs, of course, but I guess that will be moot since there's no activation with just distutils? Yes * There's no idea of explicitly enabling an entry point, simply installing a package makes the entry point show up. Implicit plugins make me uncomfortable. I don't see entry points as plugins, but rather the registering of a given piece of code, under a unique name. I don't understand that. I thought the purpose of entry points was to register code such as plugins so that applications didn't have to be manually configured. I think I'm with Ian on that one: Explicit is better than implicit. If I have to turn on the plugin, then what benefit does an entry point registry give me? I could just as easily provide that information to the application directly. If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ? The user who wants the application to consume the plugin. The way I see entry points is potential plugins, an application can decide to consume, and turn into a real plugin when it uses it. And an entry point that would be disabled is an entry point that is not used from the application A point of view, but might be used in the application B. But if it's not being used by A, why should A see it at all? Doug ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On Tue, May 5, 2009 at 1:57 PM, Doug Hellmann doug.hellm...@gmail.com wrote: If I have to turn on the plugin, then what benefit does an entry point registry give me? I don't understand this sentence, since you say later that you want the user to manually turn a plugin on for an application to soncume it. If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ? The user who wants the application to consume the plugin. I am confused with the role of this man in the middle. In my mind there are plugins on one side, and host applications that consumes them if they wish on they other side. Do you have a use case we can share for mutual comprehension ? And an entry point that would be disabled is an entry point that is not used from the application A point of view, but might be used in the application B. But if it's not being used by A, why should A see it at all? A, B or any app can browse all entry points. Entry points are defined by the (group, name) couple. Basically if I want to create a pluggable feature called myfeature, the group will be myfeature and plugins that implement that feature will register themselves under that group. Then my application will browse and consume entry points for the group myfeature and do whatever they want with them. In pseudo code: plugins = iter_entry_points(group=myfeature) So if A doesn't need the plugins that are under the group myfeature, it will just ignore the entry points that are in this group. e.g. ignore the group. Maybe A will consume entry_points that are under another group. But I have never browsed *all* entry points from an application. I think the best practice for entry points is to use the most explicit group names possible, but having plugins that can be consumed by several applications is a win ihmo. For instance, if I need to write a specific extensible installation script, I'll probably see if I can consume zc.buildout recipes through the zc.buildout.recipe group. Doug -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On May 5, 2009, at 8:15 AM, Tarek Ziadé wrote: On Tue, May 5, 2009 at 1:57 PM, Doug Hellmann doug.hellm...@gmail.com wrote: If I have to turn on the plugin, then what benefit does an entry point registry give me? I don't understand this sentence, since you say later that you want the user to manually turn a plugin on for an application to soncume it. I want each application to be configured separately, rather than having a global registry of plugins. So having a plugin registry *library* in stdlib may make sense (so that apps can build their registry databases in a consistent way), but automatically registering entry points just because a package is installed does not. If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ? The user who wants the application to consume the plugin. I am confused with the role of this man in the middle. In my mind there are plugins on one side, and host applications that consumes them if they wish on they other side. Do you have a use case we can share for mutual comprehension ? I think we have a different view about how the plugins should work. It sounds like you're advocating a model where all plugins are registered globally, an application can search the global registry for plugins based on categories, and then some administrator enables/ disables them locally for each app. I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort. So if A doesn't need the plugins that are under the group myfeature, it will just ignore the entry points that are in this group. e.g. ignore the group. Maybe A will consume entry_points that are under another group. But I have never browsed *all* entry points from an application. That depends on how the database of entry points is maintained, but I'll grant that point. I think the best practice for entry points is to use the most explicit group names possible, but having plugins that can be consumed by several applications is a win ihmo. For instance, if I need to write a specific extensible installation script, I'll probably see if I can consume zc.buildout recipes through the zc.buildout.recipe group. Doug -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On May 5, 2009, at 9:33 AM, Tarek Ziadé wrote: On Tue, May 5, 2009 at 2:41 PM, Doug Hellmann doug.hellm...@gmail.com wrote: I am confused with the role of this man in the middle. In my mind there are plugins on one side, and host applications that consumes them if they wish on they other side. Do you have a use case we can share for mutual comprehension ? I think we have a different view about how the plugins should work. It sounds like you're advocating a model where all plugins are registered globally, an application can search the global registry for plugins based on categories, and then some administrator enables/disables them locally for each app. I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort. That is basically how host applications are dealing with entry points: an explicit configuration. The implicit part is just happening when you look for the plugins. Let's take a real example : I am working on a program called Atomisator which can be extended through plugins. For example people can provide a plugin to read data that are located somewhere (like a rss feed), and return a sequence of entries. So in atomisator, I have a configuration file where I decide which plugin to use to read my data: [atomisator] reader = rss When reading it, Atomisator knows that it needs the rss plugin, and will browse in entry points with a specific group called atomisator.rss. If it finds it, it uses it, otherwise it throws an exception. (Install it!) That allows people to create their own plugins in separate packages, and use them by tweaking the configuration file. The only think that entry point provided here is an automatic registration at installation time of the rss plugins, so my Atomisatior application can discover then load it at run time. I don't think I understand the difference between the step you're calling discover, scanning the registry, and actually loading the plugin. Does discovering the plugin involve importing any of its code? So in your way of seeing thing, you'd rather see this registration mechanism at the application level, but the you need to provide specific installation instructions for people that want to add plugins. e.g. put your package in this /plugins directory for example. Sort of. I see the installation and configuration of entry points as 2 steps: 1. some variation of python setup.py install 2. update the configuration of app to tell it where to find the plugin Some applications may provide automated tools for step 2, and I could see great benefit to making the loading and registration of entry points part of the standard library, as long as the registration is maintained per-application instead of site-wide. I am advocating that the entry point mechanism is handy because it relies on an existing mechanism : install your package and it allows plugin sharing amongst applications. How then do I install a package but *not* have it available for an application? Suppose, for example, that I have several copies of a web app installed for different customers, but I want to prevent some of them from using an advanced plugin of some sort. But I see the caveats you are explaining, and I understand them now; So, what if we didn't have these entry points installed globally when the package is installed, but just a configuration file in the host application level to point them ? a configuration file that reunites all entry points an application uses. For the Atomisator example: [atomisator.reader] rss = somepackage.somemodule:MyPluginClass Yes! We can figure out the exact spelling, but we're talking about the same thing now. If we use dotted notation all the way (somepackage.somemodule.MyPluginClass) then it's simple to just import the thing directly. This would let the application consume the plugins pointed by this configuration file and this would remove the implicit part you don't like. I'd be very happy with such a plugin system on my side. If you follow the model of the logging module and provide an easy load plugins from an ini file, then all python apps have the feature of making it easy to load plugins, developers have a standard API to code to, and administrators don't have to worry about plugins sneaking into an app. And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg That sounds good. Doug ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
At 08:41 AM 5/5/2009 -0400, Doug Hellmann wrote: I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort. Note that this is not incompatible with entry points; an application can simply treat entry points as a list of *available* plugins, rather than as a list of *active* plugins. Chandler, for example, does this for user-level plugins. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On May 5, 2009, at 10:38 AM, P.J. Eby wrote: At 08:41 AM 5/5/2009 -0400, Doug Hellmann wrote: I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort. Note that this is not incompatible with entry points; an application can simply treat entry points as a list of *available* plugins, rather than as a list of *active* plugins. Chandler, for example, does this for user-level plugins. That's good. I wasn't sure if registering an entry point caused it to be automatically loaded in an app that expressed interest in it, or if there was a step the app could take to verify that it was desirable code before doing any imports. Earlier messages in this thread made me think there was no protection from a registered plugin. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On Tue, May 5, 2009 at 4:29 PM, Doug Hellmann doug.hellm...@gmail.com wrote
I don't think I understand the difference between the step you're calling
discover, scanning the registry, and actually loading the plugin. Does
discovering the plugin involve importing any of its code?
No, like Phillip said somewhere in the thread.
The discovery pseudo-code is:
entry_points = []
for path in paths:
if path is egg-info:
entry_points.append(load('entry_points.txt'))
Then each entry point is just a string that locates the plugins
(package.modul.class for example)
The real import occurs only explicitely when you do entry_point.load()
a configuration file that reunites all entry points an application
uses. For the Atomisator example:
[atomisator.reader]
rss = somepackage.somemodule:MyPluginClass
Yes! We can figure out the exact spelling, but we're talking about the same
thing now. If we use dotted notation all the way
(somepackage.somemodule.MyPluginClass) then it's simple to just import the
thing directly.
I think is simpler with the somepackage.somemodule:MyPluginClass notation
This is how setuptools does roughly:
parts = somepackage.somemodule:MyPluginClass.split(':')
module = __import__(parts[0])
plugin = getattr(module, parts[1])
And this would fit I think in Distutils needs since we can configure
it through three levels of configuration files
distutils.cfg, pydistutils.cfg and setup.cfg
That sounds good.
The only caveat I see though, is that the host app has to know the
exact location
of each plugin in the code of the third party app, whereas entry points provide
this information through the discovery API.
I could leave with both notation I believe:
location = somepackage.somemodule:MyPluginClass
*or*
location = iter_entry_points('atomisator.reader', 'rss')
--
Tarek Ziadé | http://ziade.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
2009/5/5 Sebastien Binet seb.bi...@gmail.com: hi, That allows people to create their own plugins in separate packages, and use them by tweaking the configuration file. The only think that entry point provided here is an automatic registration at installation time of the rss plugins, so my Atomisatior application can discover then load it at run time. So in your way of seeing thing, you'd rather see this registration mechanism at the application level, but the you need to provide specific installation instructions for people that want to add plugins. e.g. put your package in this /plugins directory for example. wouldn't this be tackled by providing a plugin-discoverer plugin ? kind of like the new import hooks of pep-302 [1] where you have module finders separated from module loaders, or like this library I have been using in my field [2] That would be better to use these hooks rather than __import__ I believe, but I don't think it changes the problem (eg locate the plugin) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On May 5, 2009, at 11:05 AM, Tarek Ziadé wrote: On Tue, May 5, 2009 at 4:29 PM, Doug Hellmann doug.hellm...@gmail.com wrote a configuration file that reunites all entry points an application uses. For the Atomisator example: [atomisator.reader] rss = somepackage.somemodule:MyPluginClass Yes! We can figure out the exact spelling, but we're talking about the same thing now. If we use dotted notation all the way (somepackage.somemodule.MyPluginClass) then it's simple to just import the thing directly. I think is simpler with the somepackage.somemodule:MyPluginClass notation Good point, I was remembering the import syntax incorrectly. And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg That sounds good. The only caveat I see though, is that the host app has to know the exact location of each plugin in the code of the third party app, whereas entry points provide this information through the discovery API. True. On the other hand, that encourages standard locations like mylib.yourapp.entry_point and mylib.anotherapp.entry_point. I think most of my concerns about the global registry are taken care of by the fact that discovering a plugin doesn't involve any imports of unknown code. I still prefer per-app, explicit configuration of entry points, and think we could build a system to support that. I would like to see *some* variation of this in the standard library, though, because I have several uses for it. Doug ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On Tue, May 05, 2009 at 05:05:28PM +0200, Tarek Ziadé wrote: On Tue, May 5, 2009 at 4:29 PM, Doug Hellmann doug.hellm...@gmail.com wrote a configuration file that reunites all entry points an application uses. For the Atomisator example: [atomisator.reader] rss = somepackage.somemodule:MyPluginClass [...] And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg That sounds good. So there would be a configuration file for each application that needs it? I like this a lot more then the global entry-point registry too (it avoids name collisions for entry points too). But how can a python setup.py install know where to find this configuration file to add it's plugin? Or should this be an explicit manual step? It might be nice to have a --register-plugins option to the install command though if possible. Something else to keep into account is the FHS, I can imagine GNU/Linux distributions would want to place a configuration file somewhere else, like in /etc/PROJECT.conf instead of /usr/lib/pythonX.Y/site-packages/PROJECT.egg-info/plugin_registry (pathnames are fairly random examples). The only thing I can think of is somehow having a file in .egg-info telling you where the plugin configuration file is, just like was proposed for all types of data files earlier on this list. But I think by now I'm taking this too far and the install command of distutils should not be able to register plugins for random projects automatically. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
At 09:27 PM 5/5/2009 +0100, Floris Bruynooghe wrote: But how can a python setup.py install know where to find this configuration file to add it's plugin? It doesn't. The whole point of having two stages -- discovery and activation -- is that discovery is an automatic side-effect of installation, whereas activation can be controlled by an application-specific policy or configuration file. Entry points (as currently implemented) are discovery; the application then determines which entry points to actually load or invoke, using its own configuration (or command-line options, or menu selections, or automatic policy, or whatever else). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools, namespace packages, --single-version-externally-managed
Any ideas on this? Phillip?
On Fri, May 1, 2009 at 5:07 PM, Ian Bicking i...@colorstudy.com wrote:
So, a bit of a problem came up with pip and namespace packages. Here's my
understanding of what's going on:
When you install a namespace package with pip, it uses install
--single-version-externally-managed, and generally the namespace directory
is empty and there's a *.nspkg.pth file that has this:
import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('NAMESPACE',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m =
not ie and sys.modules.setdefault('zope',new.module('zope')); mp = (m or [])
and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
So the lack of an __init__.py file doesn't really matter, because it's
created right there, and has its __path__ added to. But there's a problem
when there's another namespace package elsewhere on the path, that wasn't
installed with pip (or Setuptools) and uses pkgutils.extend_path(__path__,
__name__). This doesn't get imported because of that .pth file, and the
.pth file doesn't itself use extend_path, so the path isn't searched. This
is currently happening with Zope packages installed with plain distutils,
then another package installed with the zope namespace elsewhere with pip.
(When using easy_install, I think pkg_resource.declare_namespace comes into
play somewhere, and this seems to handle this case, but I'm not sure why the
installation is different with pip.)
So... what should pip be doing differently to make this work?
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools, namespace packages, --single-version-externally-managed
At 04:35 PM 5/5/2009 -0500, Ian Bicking wrote:
Any ideas on this? Phillip?
On Fri, May 1, 2009 at 5:07 PM, Ian Bicking
mailto:i...@colorstudy.comi...@colorstudy.com wrote:
So, a bit of a problem came up with pip and namespace
packages. Here's my understanding of what's going on:
When you install a namespace package with pip, it uses install
--single-version-externally-managed, and generally the namespace
directory is empty and there's a *.nspkg.pth file that has this:
import sys,new,os; p =
os.path.join(sys._getframe(1).f_locals['sitedir'],
*('NAMESPACE',)); ie =
os.path.exists(os.path.join(p,'__init__.py')); m = not ie and
sys.modules.setdefault('zope',new.module('zope')); mp = (m or [])
and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
So the lack of an __init__.py file doesn't really matter, because
it's created right there, and has its __path__ added to. But
there's a problem when there's another namespace package elsewhere
on the path, that wasn't installed with pip (or Setuptools) and uses
pkgutils.extend_path(__path__, __name__). This doesn't get
imported because of that .pth file, and the .pth file doesn't itself
use extend_path, so the path isn't searched. This is currently
happening with Zope packages installed with plain distutils, then
another package installed with the zope namespace elsewhere with
pip. (When using easy_install, I think
pkg_resource.declare_namespace comes into play somewhere, and this
seems to handle this case, but I'm not sure why the installation is
different with pip.)
So... what should pip be doing differently to make this work?
Heck if I know. IIUC, this should work as long as the namespace
declaration gets invoked at some point.
But I don't know of any way to ensure that, except for pip to write
its own __init__.py (invoking declare_namespace()) and gets rid of
the nspkg.pth file.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
Tarek Ziadé ziade.ta...@gmail.com writes:
I think is simpler with the somepackage.somemodule:MyPluginClass notation
This is how setuptools does roughly:
parts = somepackage.somemodule:MyPluginClass.split(':')
[…]
Using the standard import notation is no more difficult:
parts = somepackage.somemodule.MyPluginClass.rsplit('.', 1)
parts
['somepackage.somemodule', 'MyPluginClass']
versus:
parts = somepackage.somemodule:MyPluginClass.split(':', 1)
parts
['somepackage.somemodule', 'MyPluginClass']
I don't see any advantage, in the context of this discussion, to having
an additional, incompatible naming for full-path-to-a-class.
--
\ “The lift is being fixed for the day. During that time we |
`\regret that you will be unbearable.” —hotel, Bucharest |
_o__) |
Ben Finney
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote: I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class. Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
