Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Nathaniel Smith's message of 2017-10-20 13:41:03 -0700:
> On Oct 19, 2017 11:10, "Donald Stufft"  wrote:
> 
> 
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to “hook” into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
> 
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].

Having post-install and pre-uninstall hooks should be sufficient for
updating a cache, assuming the hook could be given enough information
about the thing being manipulated to probe for whatever data it
needs.

> In this hypothetical system, how do installers like pip find the list of
> hooks to call? By looking up an entrypoint? (Sorry if this was discussed
> downthread; I didn't see it but I admit I only skimmed.)

That's how I would expect it to work. Using setuptools most likely?
That would mean that other plugin systems would have to provide one
setuptools plugin to hook into the installer to build a lookup
cache, but the actual plugins wouldn't have to use setuptools for
anything.

Doug
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Thomas Kluyver's message of 2017-10-20 19:37:45 +0100:
> On Fri, Oct 20, 2017, at 07:24 PM, Doug Hellmann wrote:
> > I have been trying to find time to do something like that within
> > stevedore for a while to solve some client-side startup performance
> > issues with the OpenStack client. I would be happy to help add it
> > to entrypoints instead and use it from there.
> > 
> > Thomas, please me know how I can help.
> 
> Thanks Doug! For starters, I'd be interested to hear any plans you have
> for how to tackle caching, or any thoughts you have on the rough plan I
> described before. If you're happy with the concepts, I'll have a go at
> implementing it. I'll probably consider it experimental until there's a
> hooks mechanism to trigger rebuilding the cache when packages are
> installed or uninstalled.
> 
> Thomas

I assumed that the user loading the plugins might not be able to
write to any of the directories on sys.path (aside from "." and we
don't want to put a cache file there), so my plan was to build the
cache the first time entry points were scanned and use appdirs [1]
to pick a cache location specific to the user.  I thought I would
use the value of sys.path as a string (joining the paths together
with a separator of some sort) to create a hash for the cache file
ID. Some of that may be obviated if we assume a setuptools hook that
lets us update the cache(s) when a package is installed.

I also thought I'd provide a command line tool to generate the cache
just in case it became corrupted or if someone wanted to update it
by hand for some other reason, similar to Nick's locate/updatedb
parallel UX example (and re-reading your email, I see you mention this,
too).

I hadn't gone as far as deciding on a file format, but sqlite, JSON,
and INI (definitely something built-in) were all on my mind.  I
planned to see if we would actually gain enough of a boost just by
placing a separate file for each dist in a single cache directory,
rather than trying to merge everything into one file. In addition
to eliminating the concurrency issue, that approach might have the
additional benefit of simplifying operating system packages, because
they could just add a new file to the package instead of having to
run a command to update the cache when a package was installed (if
the file is the same format as entry_points.txt but with a different
name, that's even simpler since it's just a copy of a file that
will already be available during packaging).

Your idea of having a cache file per directory on sys.path is also
interesting, though I have to admit I'm not familiar enough with
the import machinery to know if it's easy to determine the containing
directory for a dist to find the right cache to update. I am
interested in hearing more details about what you planned there.

I would also like to compare the performance of a few approaches
(1 file per sys.path hash using INI, JSON, and sqlite; one file per
entry on sys.path using the same formats) using a significant number
of plugins (~100?) before we decide.

I agree with your statement in the original email that applications
should be able to disable the cache. I'm not sure it makes sense
to have a mode that only reads from a cache, but I may just not see
the use case for that.

What's our next step?

Doug

[1] https://pypi.python.org/pypi/appdirs/1.4.3
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Nick Coghlan's message of 2017-10-20 14:42:09 +1000:
> On 20 October 2017 at 02:14, Thomas Kluyver  wrote:
> 
> > On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
> > > I’m in favor, although one question I guess is whether it should be a a
> > > PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
> > > without (2), its just another file in the .dist-info directory and that
> > > doesn’t actually need standardized at all). I don’t think that this will
> > > be a very controversial PEP though, and should be pretty easy.
> >
> > I have opened a PR to document what is already there, without adding any
> > new features. I think this is worth doing even if we don't change
> > anything, since it's a de-facto standard used for different tools to
> > interact.
> >
> > https://github.com/pypa/python-packaging-user-guide/pull/390
> >
> > We can still write a PEP for caching if necessary.
> >
> 
> +1 for that approach (PR for the status quo, PEP for a shared metadata
> caching design) from me
> 
> Making the status quo more discoverable is valuable in its own right, and
> the only decisions we'll need to make for that are terminology
> clarification ones, not interoperability ones (this isn't like PEP 440 or
> 508 where we actually thought some of the default setuptools behaviour was
> slightly incorrect and wanted to change it).
> 
> Figuring out a robust cross-platform network-file-system-tolerant metadata
> caching design on the other hand is going to be hard, and as Donald
> suggests, the right ecosystem level solution might be to define
> install-time hooks for package installation operations.
> 
> > > I’m also in favor of this. Although I would suggest SQLite rather than a
> > > JSON file for the primary reason being that a JSON file isn’t
> > > multiprocess safe without being careful (and possibly introducing
> > > locking) whereas SQLite has already solved that problem.
> >
> > SQLite was actually my first thought, but from experience in Jupyter &
> > IPython I'm wary of it - its built-in locking does not work well over
> > NFS, and it's easy to corrupt the database. I think careful use of
> > atomic writing can be more reliable (though that has given us some
> > problems too).
> >
> > That may be easier if there's one cache per user, though - we can
> > perhaps try to store it somewhere that's not NFS.
> >
> 
> I'm wondering if rather than jumping straight to a PEP, it may make sense
> to instead initially pursue this idea as a *non-*standard, implementation
> dependent thing specific to the "entrypoints" project. There are a *lot* of
> challenges to be taken into account for a truly universal metadata caching
> design, and it would be easy to fall into the trap of coming up with a
> design so complex that nobody can realistically implement it.
> 
> Specifically, I'm thinking of a usage model along the lines of the
> updatedb/locate pair on *nix systems: `locate` gives you access to very
> fast searches of your filesystem, but it *doesn't* try to automagically
> keeps its indexes up to date. Instead, refreshing the indexes is handled by
> `updatedb`, and you can either rely on that being run automatically in a
> cron job, or else force an update with `sudo updatedb` when you want to use
> `locate`.
> 
> For a project like entrypoints, what that might look like is that at
> *runtime*, you may implement a reasonably fast "cache freshness check",
> where you scanned the mtime of all the sys.path entries, and compared those
> to the mtime of the cache. If the cache looks up to date, then cool,
> otherwise emit a warning about the stale metadata cache, and then bypass it.
> 
> The entrypoints project itself could then expose a
> `refresh-entrypoints-cache` command that could start out only supporting
> virtual environments, and then extend to per-user caching, and then finally
> (maybe) consider whether or not it wanted to support installation-wide
> caches (with the extra permissions management and cross-process and
> cross-system coordination that may imply).
> 
> Such an approach would also tie in nicely with Donald's suggestion of
> reframing the ecosystem level question as "How should the entrypoints
> project request that 'refresh-entrypoints-cache' be run after every package
> installation or removal operation?", which in turn would integrate nicely
> with things like RPM file triggers (where the system `pip` package could
> set a file trigger that arranged for any properly registered Python package
> installation plugins to be run for every modification to site-packages
> while still appropriately managing the risk of running arbitrary code with
> elevated privileges)
> 
> Cheers,
> Nick.
> 

I have been trying to find time to do something like that within
stevedore for a while to solve some client-side startup performance
issues with the OpenStack client. I would be happy to help add it
to entrypoints instead and use it from there.

Thomas, 

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Wes Turner's message of 2017-10-20 10:41:02 -0400:
> On Friday, October 20, 2017, Donald Stufft  wrote:
> 
> >
> >
> > On Oct 20, 2017, at 9:35 AM, Nick Coghlan  > > wrote:
> >
> > On 20 October 2017 at 23:19, Donald Stufft  > > wrote:
> >
> >> One that I was helping someone debug just the other day is that they’re
> >> super non-debuggable and the behavior when you have two things providing
> >> the same entry point is basically  (If I remember correctly, the
> >> behavior is that the first thing found is the one that “wins”, which means
> >> the ordering of sys.path and the names of the projects supply it is what
> >> determines it). This got exposed to the end user that they installed
> >> something that they thought was going to add support for something, but
> >> which silently did nothing because two different project happened to pick
> >> the same name for their entry point (not the group, it was two things
> >> providing plugins for the same system).
> >>
> >
> > While I agree with this, I think that's a combination of pkg_resources
> > itself being hard to debug in general, and the fact that pkg_resources
> > doesn't clearly define the semantics of how it resolves name conflicts
> > within an entry point group - as far as I know, it's largely an accident of
> > implementation.
> >
> > The interoperability spec is going to state that conflict resolution when
> > the same name within a group is declared by multiple packages is the
> > responsibility of the group consumer, so documenting the format should
> > actually improve this situation, since it allows for the development of
> > competing conflict resolution strategies in different runtime libraries.
> >
> >
> > I think it makes it *worse*, because now the behavior isn’t just a
> > entrypoints weirdness, but now it changes based on which runtime library
> > you use (which isn’t something that end users are likely to have much
> > insight into) and it represents a footgun that package authors are unlikely
> > to be aware of. If mycoolentrypointslib comes out that is faster, but
> > changes some subtle behavior like this it’ll break people, but that is
> > unlikely going to be an effect that people expect to happen just because
> > they switched between two things both implementing the same standard.
> >
> > So effectively this means that not only is the fact you’re using
> > entrypoints part of your API, but now which entry point library you’re
> > using at runtime is now also part of your API.
> >
> 
> When should the check for duplicate entry points occur?
> 
> - At on_install() time (+1)
> - At runtime
> 
> Is a sys.path-like OrderedDict preemptive strategy preferable or just as
> dangerous as importlib?

Having "duplicate" entry points is not necessarily an error. It's
a different usage pattern.  The semantics of dropping a named plugin
into a namespace are defined by the application and plugin-point.
Please do not build assumptions about uniqueness into the underlying
implementation.

The stevedore library wraps up pkg_resources with several such
patterns. For example, it supports "give me all of the plugins in
a namespace" (find all the extensions to your app), "give me all
of the plugins named $name in a namespace" (find the hooks for a
specific event defined by the app), and "give me *the* plugin named
$name in a namespace" (load a driver for talking to a backend).

https://docs.openstack.org/stevedore/latest/reference/index.html

Doug
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Entry points: specifying and caching

[Doug Hellmann]

Excerpts from Thomas Kluyver's message of 2017-10-18 15:52:00 +0100:
> We're increasingly using entry points in Jupyter to help integrate
> third-party components. This brings up a couple of things that I'd like
> to do:
> 
> 1. Specification
> 
> As far as I know, there's no document describing the details of entry
> points; it's a de-facto standard established by setuptools. It seems to
> work quite well, but it's worth writing down what is unofficially
> standardised. I would like to see a document on
> https://packaging.python.org/specifications/ saying:
> 
> - Where build tools should put entry points in wheels
> - Where entry points live in installed distributions
> - The file format (including allowed characters, case sensitivity...)
> 
> I guess I'm volunteering to write this, although if someone else wants
> to, don't let me stop you. ;-)
> 
> I'd also be happy to hear that I'm wrong, that this specification
> already exists somewhere. If it does, can we add a link from
> https://packaging.python.org/specifications/ ?

I've always used the setuptools documentation as a reference. Are you
suggesting moving that information to a different location to
allow/encourage other tools to implement it as a standard?

> 2. Caching
> 
> "There are only two hard problems in computer science: cache
> invalidation, naming things, and off-by-one errors"
> 
> I know that caching is going to make things more complex, but at present
> a scan of available entry points requires a stat() for every installed
> package, plus open()+read()+parse for every installed package that
> provides entry points. This doesn't scale well, especially on spinning
> hard drives. By eliminating a call to pygments which caused an entry
> points scan, we cut the cold-start time of IPython almost in half on one
> HDD system (11s -> 6s; PR 10859).
> 
> As packaging improves, the trend is to break functionality into more,
> smaller packages, which is only going to make this worse (though I hope
> we never end up with a left-pad package ;-). Caching could allow entry
> points to be used in places where the current performance penalty is too
> much.
> 
> I envisage a cache working something like this:
> - Each directory on sys.path can have a cache file, e.g.
> 'entry-points.json'
> - I suggest JSON because Python can parse it efficiently, and it's not
> intended to be directly edited by humans. Other options? SQLite? Does
> someone want to do performance comparisons?
> - There is a command to scan all packages in a directory and build the
> cache file
> - After an install tool (e.g. pip) has added/removed packages from a
> directory, it should call that command to rebuild the cache.
> - A second command goes through all directories on sys.path and rebuilds
> their cache files - this lets the user rebuild caches if something has
> gone wrong.
> - Applications looking for entry points can choose from a range of
> behaviours depending on how important accuracy and performance are. E.g.
> ignore all caches, only use caches, use caches for directories where
> they exist, or try caches first and then scan packages if a key is
> missing.
> 
> In the best case, when the caches exist and you trust them, loading them
> would cost one set of filesystem operations per sys.path entry, rather
> than per package.
> 
> Thanks,
> Thomas

We've run into similar issues in some applications I work on. I had
intended to implement a caching layer within stevedore
(https://docs.openstack.org/stevedore/latest/) as a first step for
experimenting with approaches, but I would be happy to collaborate on
something further upstream if there's interest.

Doug
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Multiple pips to go along with my multiple Pythons

[Doug Hellmann]

Excerpts from Skip Montanaro's message of 2015-05-07 09:58:35 -0500:
 Thanks for the responses. I eventually decided to just download and
 expand the tarball, then execute
 
 PY setup.py clean -a
 PY setup.py install
 
 for each of my PYs. I then deleted the pip, pip2 and pip3 items,
 leaving only the pipXY versions. That avoids any confusion. While it's
 a bit more work on my part, it achieved the desired result - nothing
 but a bunch of unambiguous pipXY programs.
 
 Skip

Depending on your OS, you might be interested in the ansible role I
created for doing this on my dev VMs:
http://doughellmann.com/2015/03/07/ansible-roles-for-python-developers.html

It's meant for Ubuntu now, but I would be happy to have updates to make
it work for other platforms.

Doug
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] entry points PEP

[Doug Hellmann]

On Jul 19, 2013, at 6:22 PM, Monty Taylor monty.tay...@gmail.com wrote:

 Yeah. Not moving to zc.buildout for anything. I believe it will be a better 
 option to just write by-hand scripts that get installed that just do:
 
 from nova.rootwrap import cmd
 
 return cmd.main(sys.argv)
 
 or something. Basically, a tiny boiler-plate script that does the same thing 
 as a console_scripts entry point thing without loading the module in question 
 via pkg_resources.

We could also have pbr do this when it builds the sdist.

Doug

 
 
 On Fri, Jul 19, 2013 at 3:10 PM, Joe Gordon joe.gord...@gmail.com wrote:
 
 
 
 On Fri, Jul 19, 2013 at 1:42 PM, PJ Eby p...@telecommunity.com wrote:
 On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon joe.gord...@gmail.com wrote:
  When I try importing pkg_resources in our development environment it is very
  slow:
 
 Use zc.buildout to install the application you're invoking, and then
 it won't need to import pkg_resources.  (Unless the actual app uses
 it.)
 
 It looks like zc.buildout is not an option as we are already heavily invested 
 in using  pip and virtualenv.
 
 
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] time for packaging summit at pycon

[Doug Hellmann]

OK, thanks!

On Feb 27, 2013, at 10:54 PM, Nick Coghlan wrote:

 On Thu, Feb 28, 2013 at 12:32 AM, Doug Hellmann doug.hellm...@gmail.com 
 wrote:
 The Packaging Summit page [1] lists Friday evening but doesn't specify a 
 time or location, as far as I can tell. Have those been set, yet?
 
 Doug
 
 [1] 
 https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/
 
 Laurens has a tentative schedule linked from the main open spaces page
 (https://us.pycon.org/2013/community/openspaces/)
 
 I've added the relevant details to the mini-summit page (room 206, 7-9 pm)
 
 It's only a couple of hours, because my aim is mostly to encourage
 information sharing through lightning talks about some of the key
 projects involved in packaging and distribution, rather than making
 any actual firm decisions about anything. I hope the presentations
 will stimulate an ongoing discussion throughout the conference,
 following on into the sprints and post-conference development.
 
 Cheers,
 Nick.
 
 
 
 -- 
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] time for packaging summit at pycon

[Doug Hellmann]

The Packaging Summit page [1] lists Friday evening but doesn't specify a time 
or location, as far as I can tell. Have those been set, yet?

Doug

[1] 
https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] need guidance on state-of-the-art for entry points

[Doug Hellmann]

Dear Fellowship,

I am starting a new project which will be heavily extensible using plugins 
defined in other people's packages over which I have little or no control. I 
was planning to use setuptools entry points to manage the plugin discovery, 
since it has worked so well for virtualenvwrapper. However, in the process of 
searching for current documentation to reference in the project documentation, 
I got lost and confused about what is currently supported and what is 
deprecated.

Can someone tell me the currently approved way to work with a modern version of 
entry points? Should I continue to use distribute/setuptools as before? Should 
I move to distutils2 and some new metadata in the setup.cfg file? Should I be 
using Tarek's extensions package? If so, how do I tell distutils2 to run the 
registration code during installation?

Thanks,
Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] need guidance on state-of-the-art for entry points

[Doug Hellmann]

OK, thanks!

On Apr 25, 2012, at 1:39 PM, Ronny Pfannschmidt wrote:

 Hi doug,
 
 currently just distribute/setuptools is the way to go,
 later on packaging/ds2 will probably provide a replacement,
 but thats currently not usable/feature complete
 
 -- Ronny
 
 On 04/25/2012 07:33 PM, Doug Hellmann wrote:
 Dear Fellowship,
 
 I am starting a new project which will be heavily extensible using plugins 
 defined in other people's packages over which I have little or no control. I 
 was planning to use setuptools entry points to manage the plugin discovery, 
 since it has worked so well for virtualenvwrapper. However, in the process 
 of searching for current documentation to reference in the project 
 documentation, I got lost and confused about what is currently supported and 
 what is deprecated.
 
 Can someone tell me the currently approved way to work with a modern version 
 of entry points? Should I continue to use distribute/setuptools as before? 
 Should I move to distutils2 and some new metadata in the setup.cfg file? 
 Should I be using Tarek's extensions package? If so, how do I tell 
 distutils2 to run the registration code during installation?
 
 Thanks,
 Doug
 
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig
 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] problem with sdist for nested packages

[Doug Hellmann]

On Sep 8, 2010, at 12:56 PM, P.J. Eby wrote:

 At 12:00 PM 9/8/2010 -0400, Doug Hellmann wrote:
 Instead of explicitly listing packages, I use find_package_data() and then 
 get the package names from that result.  The odd thing is I get a package 
 PyMOTW.ElementTree but not PyMOTW.xml.etree.ElementTree.  Maybe I'm 
 misunderstanding/misusing find_package_data(), since that nested package is 
 the only one I'm having problems with.
 
 You want find_packages():
 
 http://peak.telecommunity.com/DevCenter/setuptools#using-find-packages

That took care of it. Thanks for the help!

Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] problem with sdist for nested packages

[Doug Hellmann]

I'm having some problems creating an sdist for a nested package where some of 
the levels only include __init__.py and the subdirectories.  The full list of 
files is fairly long, but here's a snippet to give you an idea of the layout 
for the inputs:

PyMOTW/xml
PyMOTW/xml/__init__.py
PyMOTW/xml/etree
PyMOTW/xml/etree/__init__.py
PyMOTW/xml/etree/ElementTree
PyMOTW/xml/etree/ElementTree/__init__.py
PyMOTW/xml/etree/ElementTree/ElementTree_create.py
PyMOTW/xml/etree/ElementTree/...

When I build the sdist, I get:

build/lib/PyMOTW/xml
build/lib/PyMOTW/xml/etree
build/lib/PyMOTW/xml/etree/ElementTree
build/lib/PyMOTW/xml/etree/ElementTree/__init__.py
build/lib/PyMOTW/xml/etree/ElementTree/ElementTree_create.py
build/lib/PyMOTW/xml/etree/ElementTree/...

The __init__.py at the xml and xml/etree levels are not included.  As a 
result, the sdist won't install because it wants to populate a 
PyMOTW.ElementTree package but the directory doesn't exist.  I also tried 
adding other .py files at those levels in the directory structure (in case an 
empty directory was being ignored), but that had no effect.

The traceback I get trying to install the sdist is:

Traceback (most recent call last):
  File setup.py, line 7, in module
paver.tasks.main()
  File paver-minilib.zip/paver/tasks.py, line 621, in main
  File paver-minilib.zip/paver/tasks.py, line 604, in _launch_pavement
  File paver-minilib.zip/paver/tasks.py, line 569, in _process_commands
  File paver-minilib.zip/paver/setuputils.py, line 146, in __call__
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
 line 972, in run_command
cmd_obj.run()
  File 
/Users/dhellmann/.virtualenvs/pymotwtest/lib/python2.7/site-packages/distribute-0.6.10-py2.7.egg/setuptools/command/install.py,
 line 71, in run
_install.run(self)
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py,
 line 563, in run
self.run_command('build')
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py,
 line 326, in run_command
self.distribution.run_command(command)
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
 line 972, in run_command
cmd_obj.run()
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py,
 line 127, in run
self.run_command(cmd_name)
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py,
 line 326, in run_command
self.distribution.run_command(command)
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
 line 972, in run_command
cmd_obj.run()
  File 
/Users/dhellmann/.virtualenvs/pymotwtest/lib/python2.7/site-packages/distribute-0.6.10-py2.7.egg/setuptools/command/build_py.py,
 line 77, in run
self.build_packages()
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py,
 line 366, in build_packages
modules = self.find_package_modules(package, package_dir)
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py,
 line 218, in find_package_modules
self.check_package(package, package_dir)
  File 
/Users/dhellmann/.virtualenvs/pymotwtest/lib/python2.7/site-packages/distribute-0.6.10-py2.7.egg/setuptools/command/build_py.py,
 line 192, in check_package
init_py = _build_py.check_package(self, package, package_dir)
  File 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py,
 line 191, in check_package
package directory '%s' does not exist % package_dir)
distutils.errors.DistutilsFileError: package directory 'PyMOTW/ElementTree' 
does not exist

Has anyone seen this problem before?

Thanks,
Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] problem with sdist for nested packages

[Doug Hellmann]

Forgot to include version info:

Python 2.7
distribute 0.6.10
Paver 1.0.3

Doug

On Sep 8, 2010, at 7:34 AM, Doug Hellmann wrote:

 I'm having some problems creating an sdist for a nested package where some of 
 the levels only include __init__.py and the subdirectories.  The full list of 
 files is fairly long, but here's a snippet to give you an idea of the layout 
 for the inputs:
 
 PyMOTW/xml
 PyMOTW/xml/__init__.py
 PyMOTW/xml/etree
 PyMOTW/xml/etree/__init__.py
 PyMOTW/xml/etree/ElementTree
 PyMOTW/xml/etree/ElementTree/__init__.py
 PyMOTW/xml/etree/ElementTree/ElementTree_create.py
 PyMOTW/xml/etree/ElementTree/...
 
 When I build the sdist, I get:
 
 build/lib/PyMOTW/xml
 build/lib/PyMOTW/xml/etree
 build/lib/PyMOTW/xml/etree/ElementTree
 build/lib/PyMOTW/xml/etree/ElementTree/__init__.py
 build/lib/PyMOTW/xml/etree/ElementTree/ElementTree_create.py
 build/lib/PyMOTW/xml/etree/ElementTree/...
 
 The __init__.py at the xml and xml/etree levels are not included.  As a 
 result, the sdist won't install because it wants to populate a 
 PyMOTW.ElementTree package but the directory doesn't exist.  I also tried 
 adding other .py files at those levels in the directory structure (in case an 
 empty directory was being ignored), but that had no effect.
 
 The traceback I get trying to install the sdist is:
 
 Traceback (most recent call last):
  File setup.py, line 7, in module
paver.tasks.main()
  File paver-minilib.zip/paver/tasks.py, line 621, in main
  File paver-minilib.zip/paver/tasks.py, line 604, in _launch_pavement
  File paver-minilib.zip/paver/tasks.py, line 569, in _process_commands
  File paver-minilib.zip/paver/setuputils.py, line 146, in __call__
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
  line 972, in run_command
cmd_obj.run()
  File 
 /Users/dhellmann/.virtualenvs/pymotwtest/lib/python2.7/site-packages/distribute-0.6.10-py2.7.egg/setuptools/command/install.py,
  line 71, in run
_install.run(self)
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py,
  line 563, in run
self.run_command('build')
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py,
  line 326, in run_command
self.distribution.run_command(command)
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
  line 972, in run_command
cmd_obj.run()
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py,
  line 127, in run
self.run_command(cmd_name)
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py,
  line 326, in run_command
self.distribution.run_command(command)
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py,
  line 972, in run_command
cmd_obj.run()
  File 
 /Users/dhellmann/.virtualenvs/pymotwtest/lib/python2.7/site-packages/distribute-0.6.10-py2.7.egg/setuptools/command/build_py.py,
  line 77, in run
self.build_packages()
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py,
  line 366, in build_packages
modules = self.find_package_modules(package, package_dir)
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py,
  line 218, in find_package_modules
self.check_package(package, package_dir)
  File 
 /Users/dhellmann/.virtualenvs/pymotwtest/lib/python2.7/site-packages/distribute-0.6.10-py2.7.egg/setuptools/command/build_py.py,
  line 192, in check_package
init_py = _build_py.check_package(self, package, package_dir)
  File 
 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py,
  line 191, in check_package
package directory '%s' does not exist % package_dir)
 distutils.errors.DistutilsFileError: package directory 'PyMOTW/ElementTree' 
 does not exist
 
 Has anyone seen this problem before?
 
 Thanks,
 Doug
 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] problem with sdist for nested packages

[Doug Hellmann]

On Sep 8, 2010, at 7:37 AM, Eric Smith wrote:

 On 9/8/10 7:34 AM, Doug Hellmann wrote:
 I'm having some problems creating an sdist for a nested package where some 
 of the levels only include __init__.py and the subdirectories.  The full 
 list of files is fairly long, but here's a snippet to give you an idea of 
 the layout for the inputs:
 
 snip
 
 Has anyone seen this problem before?
 
 What does your setup.py look like?

I'm using Paver, which wraps setuptools.  My pavement.py file is pasted below.  

Instead of explicitly listing packages, I use find_package_data() and then get 
the package names from that result.  The odd thing is I get a package 
PyMOTW.ElementTree but not PyMOTW.xml.etree.ElementTree.  Maybe I'm 
misunderstanding/misusing find_package_data(), since that nested package is the 
only one I'm having problems with.  It seems to work fine for all of the other 
subdirectories of PyMOTW, but this is the only one where the real code is more 
than one level deep.

Doug



#!/usr/bin/env python
# encoding: utf-8
#
# Copyright (c) 2008 Doug Hellmann All rights reserved.
#



# Standard library
import os
import sys
import tabnanny
import traceback
import types

# Set up Paver
import paver
import paver.doctools
import paver.misctasks
from paver.path import path
from paver.easy import *
import paver.setuputils
paver.setuputils.install_distutils_tasks()
try:
from sphinxcontrib import paverutils
except:
paverutils = None

# TODO
# - move these variables to options?

# What project are we building?
PROJECT = 'PyMOTW'

# What version is this?
VERSION = '1.124'

# The sphinx templates expect the VERSION in the shell environment
os.environ['VERSION'] = VERSION

# What is the current module being documented?
MODULE = path('module').text().rstrip()
os.environ['MODULE'] = MODULE

# Read the long description to give to setup
README = path('README.txt').text()

# Scan the input for package information
# to grab any data files (text, images, etc.) 
# associated with sub-packages.
PACKAGE_DATA = paver.setuputils.find_package_data(PROJECT, 
  package=PROJECT,
  only_in_packages=True,
  )

options(
setup=Bunch(
name = PROJECT,
version = VERSION,

description = 'Python Module of the Week Examples: ' + MODULE,
long_description = README,

author = 'Doug Hellmann',
author_email = 'doug.hellm...@gmail.com',

url = 'http://www.doughellmann.com/PyMOTW/',
download_url = 'http://www.doughellmann.com/downloads/%s-%s.tar.gz' % \
(PROJECT, VERSION),

classifiers = [ 'Development Status :: 5 - Production/Stable',
'Environment :: Console',
'Intended Audience :: Developers',
'Intended Audience :: Education',
'License :: OSI Approved :: BSD License',
'Operating System :: POSIX',
'Programming Language :: Python',
'Topic :: Software Development',
],

platforms = ('Any',),
keywords = ('python', 'PyMOTW', 'documentation'),

# It seems wrong to have to list recursive packages explicitly.
packages = sorted(PACKAGE_DATA.keys()),
package_data=PACKAGE_DATA,
zip_safe=False,

scripts=['motw'],

),

sdist = Bunch(
),

sdistext = Bunch(
outdir='~/Desktop',
),

sphinx = Bunch(
sourcedir=PROJECT,
docroot = '.',
builder = 'html',
doctrees='sphinx/doctrees',
confdir = 'sphinx',
template_args = { 'module':MODULE }
),

html = Bunch(
builddir='%s/docs' % PROJECT,
outdir='%s/docs' % PROJECT,
templates='pkg',
),

text = Bunch(
builddir='%s/docs' % PROJECT,
outdir='%s/docs' % PROJECT,
templates='pkg',
builder='text',
),

website=Bunch(
templates = 'web',
builddir = 'web',

# What server hosts the website?
server = 'www.doughellmann.com',
server_path = '/var/www/doughellmann/DocumentRoot/PyMOTW/',

# What template should be used for the web site HTML?
template_source = 
'~/Devel/doughellmann/doughellmann/templates/base.html',
template_dest = 'sphinx/templates/web/base.html',
),

sitemap_gen=Bunch(
# Where is the config file for sitemap_gen.py?
config='sitemap_gen_config.xml',
),

pdf=Bunch(
templates='pkg',
builddir='web',
builder='latex',
),

blog=Bunch(
sourcedir=path(PROJECT)/MODULE,
builddir='blog_posts',
outdir='blog_posts',
confdir='sphinx/blog',
doctrees='blog_posts

Re: [Distutils] problem with sdist for nested packages

[Doug Hellmann]

On Sep 8, 2010, at 12:56 PM, P.J. Eby wrote:

 At 12:00 PM 9/8/2010 -0400, Doug Hellmann wrote:
 Instead of explicitly listing packages, I use find_package_data() and then 
 get the package names from that result.  The odd thing is I get a package 
 PyMOTW.ElementTree but not PyMOTW.xml.etree.ElementTree.  Maybe I'm 
 misunderstanding/misusing find_package_data(), since that nested package is 
 the only one I'm having problems with.
 
 You want find_packages():
 
 http://peak.telecommunity.com/DevCenter/setuptools#using-find-packages

Thanks, I'll give that a try.

Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Distribute bootstrap question

[Doug Hellmann]

I just released my first new piece of software using Distribute, and  
I'm still confused about whether I have the packaging correct.  I  
think not, because a user has reported receiving a SandboxViolation  
error when trying to install the package with easy_install (bug report  
at: http://bitbucket.org/dhellmann/ical2org/issue/1/unable-to-install-ical2org-with) 
.  The relevant portion of the error message seems to be:


Searching for distribute
Reading http://pypi.python.org/simple/distribute/
Reading http://packages.python.org/distribute
Best match: distribute 0.6.10
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.10.tar.gz#md5 
=99fb4b3e4ef0861bba11aa1905e89fed

Processing distribute-0.6.10.tar.gz
Running distribute-0.6.10/setup.py -q bdist_egg --dist-dir /tmp/ 
easy_install-aO9HwB/distribute-0.6.10/egg-dist-tmp-woULY8

Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /System/Library/Frameworks/ 
Python.framework/Versions/2.5/Extras/lib/python

Non-egg installation
Removing elements out of the way...
Renaming /System/Library/Frameworks/Python.framework/Versions/2.5/ 
Extras/lib/python/setuptools-0.6c7-py2.5.egg-info into /System/Library/ 
Frameworks/Python.framework/Versions/2.5/Extras/lib/python/ 
setuptools-0.6c7-py2.5.egg-info.OLD.1263800301.89
error: SandboxViolation: mkdir('/System/Library/Frameworks/ 
Python.framework/Versions/2.5/Extras/lib/python/setuptools-0.6c7- 
py2.5.egg-info',) {}


That looks like the script is trying to install Distribute itself,  
based I suppose on having it in the install_requires list in my  
setup.py.  I added distribute to install_requires after on an apparent  
misreading of the instructions on http://pypi.python.org/pypi/ 
distribute.


So, my questions:

1. While researching the problem this morning, I came across the  
instructions at http://packages.python.org/distribute/setuptools.html#using-setuptools-without-bundling-it 
, which seem like a much more sensible way to bootstrap the  
installation of Distribute than install_requires.  Should I have used  
that instead of install_requires?


2. If I don't require Distribute for installation, will my package  
work with the older setuptools (assuming I'm not using any Distribute- 
specific features, which I don't think I am, but I'm not sure I know  
what those are).  In a nutshell, do I need to force my user to install  
Distribute at all before using my package?  And if so, what is the  
preferred way of doing that?


Thanks,
Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 6, 2009, at 9:38 PM, Wheat wrote:


I'll also mention my most common use-case for using entry_points is
installing
console_scripts using zc.recipe.egg.


I'm curious about that because I've never understood the benefit of  
using entry points for console scripts.  Why not just list the script  
as a part of the package and have it installed normally?



Also, using console_scripts for entry_points means that there is a
second
way of specifying scripts, since Distutils already has the 'scripts'
metadata field. I would say that the 'scripts' field should then be
deprecated, but perhaps there are reason's beyond breaking backwards
compatability for keeping that field around?


I would argue the other way.  Why force authors of console scripts to  
deal with entry points instead of just installing the script as-is?


Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 7, 2009, at 8:54 AM, Paul Moore wrote:


2009/5/7 Doug Hellmann doug.hellm...@gmail.com:
I would argue the other way.  Why force authors of console scripts  
to deal

with entry points instead of just installing the script as-is?


Please explain as-is with reference to ensuring that the script
works cross-platform. I think the benefit of entry points for scripts
is that it generates appropriate wrappers to allow use on all
platforms.


I write a python script call hello.py like this:

#!/usr/bin/env python

def main():
print 'hello!'

if __name__ == '__main__':
main()

Why make me define an entry point for that?  I can just put it in /usr/ 
bin or somewhere in the path on Windows and call it as hello.py.


Does setuptools give me something extra for Windows?  I'm not a  
regular Windows user, so it's likely that there are features I don't  
know about.



Having said that, I find setuptools entry points to be
over-engineered, and the Windows wrappers (in particular, the fact
that they are not version-independent) to be somewhat clumsy. But as a
concept, I like the idea of having a way of specifying that a script
is intended as an executable, and having distutils do the job of
generating whatever platform cruft is required [1] to make that work.


That's the part I'm not clear about.  What cruft is needed anywhere?


Of course, for even remotely modern Python versions, I'd argue
strongly that packages shouldn't be including console scripts, but
should rather be supplying modules that can be run as scripts, via the
-m argument to python. Users can then build aliases, shell scripts, or
whatever is appropriate based on that.

Paul.

[1] And note especially that .bat files are *not* suitable wrappers on
Windows, in spite of the fact that they are commonly used. Their
biggest disadvantage is that they don't nest.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 7, 2009, at 10:03 AM, Paul Moore wrote:


2009/5/7 Doug Hellmann doug.hellm...@gmail.com:


Does setuptools give me something extra for Windows?  I'm not a  
regular
Windows user, so it's likely that there are features I don't know  
about.


I don't think so, as such. It gives Unix and Windows users who care
(== not me, and clearly not you, either) the ability to call the
command hello rather than hello.py.



Hmm, yeah, I see that as useful.  I don't work with Windows a lot  
myself, but I don't want to explicitly drop support there, either.


Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 7, 2009, at 10:20 AM, Tres Seaver wrote:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Eric Smith wrote:

Doug Hellmann wrote:

On May 7, 2009, at 8:54 AM, Paul Moore wrote:


2009/5/7 Doug Hellmann doug.hellm...@gmail.com:
I would argue the other way.  Why force authors of console  
scripts to

deal
with entry points instead of just installing the script as-is?

Please explain as-is with reference to ensuring that the script
works cross-platform. I think the benefit of entry points for  
scripts

is that it generates appropriate wrappers to allow use on all
platforms.

I write a python script call hello.py like this:

   #!/usr/bin/env python

   def main():
   print 'hello!'

   if __name__ == '__main__':
   main()

Why make me define an entry point for that?  I can just put it in
/usr/bin or somewhere in the path on Windows and call it as  
hello.py.


Does setuptools give me something extra for Windows?  I'm not a  
regular
Windows user, so it's likely that there are features I don't know  
about.


Yes. It creates a .exe wrapper [1]. By using entry points, I don't  
need
to care what the target system is. Also, /usr/bin/env might invoke  
the

wrong python.


Exactly:  using entry points for console scripts guarantees that the
python into which the corresponding distribution is installed is the  
one
used to run the script, which is *highly* desirable.  Otherwise, you  
end

up with the just install everything in the system Python's
site-packages mess.


pip installs my scripts into a virtualenv without any issue and  
without using entry points, AFAICT.


I guess if we move to requiring entry points and disallowing simple  
script distribution I'll need to find another way to package  
virtualenvwrapper.  Since it's a bash script, it doesn't have entry  
points.  I've been using setuptools to package it so it can be  
installed via easy_install, since it is a Python development tool.


Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 7, 2009, at 5:53 PM, Ben Finney wrote:


Doug Hellmann doug.hellm...@gmail.com writes:


I write a python script call hello.py like this:

#!/usr/bin/env python

def main():
print 'hello!'

if __name__ == '__main__':
main()

Why make me define an entry point for that?  I can just put it in
/usr/ bin or somewhere in the path on Windows and call it as
hello.py.


To address the issue raised elsewhere in this thread:

Why should the name of a *command* include the suffix ‘.py’? That just
begs for the situation down the line where one of these commands is
being used widely, by the name ‘hello.py’, and then someone wants to
provide an alternative implementation in another language.


I personally don't care if it includes the .py suffix or not, most of  
the time.  I tend not to include it in my own code, and don't get bent  
out of shape if someone else's tool does use it.  Of course, as I've  
been reminded in this thread not including the .py (and not using  
entry points) means my scripts don't work as-is when installed on  
Windows.


Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 7, 2009, at 7:28 PM, P.J. Eby wrote:


At 10:38 AM 5/7/2009 -0400, Doug Hellmann wrote:

pip installs my scripts into a virtualenv without any issue and
without using entry points, AFAICT.

I guess if we move to requiring entry points and disallowing simple
script distribution I'll need to find another way to package
virtualenvwrapper.  Since it's a bash script, it doesn't have entry
points.  I've been using setuptools to package it so it can be
installed via easy_install, since it is a Python development tool.


Setuptools still supports classic scripts, and I don't see any  
reason to remove that support.  People do package non-Python scripts  
with their projects, after all.


easy_install basically examines such scripts to see if they end  
with .py, have a #! line with 'python' in it, or are valid Python  
source code.  If any of the above are true, it makes a Python script  
wrapper, otherwise it assumes the script is some other language and  
leaves it alone.


That would do what I want.  I only brought it up because another  
poster in this thread mentioned eliminating scripts an requiring  
console apps to use entry points.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 5, 2009, at 10:50 PM, P.J. Eby wrote:


At 12:03 PM 5/6/2009 +1000, Ben Finney wrote:
I don't see any advantage, in the context of this discussion, to  
having an additional, incompatible naming for full-path-to-a-class.


Setuptools doesn't limit an entry point to being a class, function,  
or other top-level name within a module.  It can be a method of a  
class, or an attribute of an attribute.  The ':' removes any  
ambiguity as to which part of the name is the module, and which  
parts are attributes within that module.


Is that level of complexity useful in practice?  I can understand how  
it came to be implemented, but is it actually used by any projects?


Doug


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 5, 2009, at 4:49 AM, Tarek Ziadé wrote:

On Tue, May 5, 2009 at 1:57 AM, Ian Bicking i...@colorstudy.com  
wrote:
Not strong, but I have a few issues with how they are currently  
defined:


* There's the issue of activated and unactivated eggs, of course,  
but I
guess that will be moot since there's no activation with just  
distutils?


Yes

* There's no idea of explicitly enabling an entry point, simply  
installing a

package makes the entry point show up.  Implicit plugins make me
uncomfortable.


I don't see entry points as plugins, but rather the registering of a
given piece of code,
under a unique name.


I don't understand that.  I thought the purpose of entry points was to  
register code such as plugins so that applications didn't have to be  
manually configured.  I think I'm with Ian on that one: Explicit is  
better than implicit.  If I have to turn on the plugin, then what  
benefit does an entry point registry give me?  I could just as easily  
provide that information to the application directly.



If you add explicit enabling, who will do it ? the package that has
the entry point ?
The applications that consumes them ?


The user who wants the application to consume the plugin.


The way I see entry points is potential plugins, an application can
decide to consume,
and turn into a real plugin when it uses it.


And an entry point that would be disabled is an entry point that  
is not used
from the application A point of view, but might be used in the  
application B.


But if it's not being used by A, why should A see it at all?

Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 5, 2009, at 8:15 AM, Tarek Ziadé wrote:

On Tue, May 5, 2009 at 1:57 PM, Doug Hellmann  
doug.hellm...@gmail.com wrote:


 If I have to turn on the plugin, then what benefit does an entry  
point registry give me?


I don't understand this sentence, since you say later that you want  
the user to

manually turn a plugin on for an application to soncume it.


I want each application to be configured separately, rather than  
having a global registry of plugins.  So having a plugin registry  
*library* in stdlib may make sense (so that apps can build their  
registry databases in a consistent way), but automatically registering  
entry points just because a package is installed does not.



If you add explicit enabling, who will do it ? the package that has
the entry point ?
The applications that consumes them ?


The user who wants the application to consume the plugin.



I am confused with the role of this man in the middle. In my mind
there are plugins on one side,
and host applications that consumes them if they wish on they other  
side.


Do you have a use case we can share for mutual comprehension ?


I think we have a different view about how the plugins should work.   
It sounds like you're advocating a model where all plugins are  
registered globally, an application can search the global registry for  
plugins based on categories, and then some administrator enables/ 
disables them locally for each app.


I don't want new functionality available to an application just  
because someone has permission to install a package somewhere in the  
PYTHONPATH.  I would rather have plugins added to an app through an  
explicit configuration step of some sort.



So if A doesn't need the plugins that are under the group myfeature,
it will just ignore the entry points
that are in this group. e.g. ignore the group.

Maybe A will consume entry_points that are under another group. But I
have never browsed *all* entry points
from an application.


That depends on how the database of entry points is maintained, but  
I'll grant that point.



I think the best practice for entry points is to use the most explicit
group names possible, but having plugins
that can be consumed by several applications is a win ihmo.

For instance, if I need to write a specific extensible installation
script, I'll probably see if I can consume
zc.buildout recipes through the zc.buildout.recipe group.




Doug






--
Tarek Ziadé | http://ziade.org


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 5, 2009, at 9:33 AM, Tarek Ziadé wrote:

On Tue, May 5, 2009 at 2:41 PM, Doug Hellmann  
doug.hellm...@gmail.com wrote:

I am confused with the role of this man in the middle. In my mind
there are plugins on one side,
and host applications that consumes them if they wish on they  
other side.


Do you have a use case we can share for mutual comprehension ?


I think we have a different view about how the plugins should  
work.  It
sounds like you're advocating a model where all plugins are  
registered
globally, an application can search the global registry for plugins  
based on
categories, and then some administrator enables/disables them  
locally for

each app.

I don't want new functionality available to an application just  
because
someone has permission to install a package somewhere in the  
PYTHONPATH.  I
would rather have plugins added to an app through an explicit  
configuration

step of some sort.


That is basically how host applications are dealing with entry points:
an explicit configuration. The implicit part is just happening when
you look for the plugins.

Let's take a real example : I am working on a program called  
Atomisator

which can be extended through plugins.

For example people can provide a plugin to read data that are located
somewhere (like a rss feed), and return
a sequence of entries.

So in atomisator, I have a configuration file where I decide which
plugin to use to read my data:


  [atomisator]

  reader = rss


When reading it, Atomisator knows that it needs the rss plugin, and
will browse in entry points
with a specific group called atomisator.rss. If it finds it, it uses
it, otherwise it throws an exception. (Install it!)



That allows people to create their own plugins in separate packages,
and use them by tweaking the configuration
file. The only think that entry point provided here is an automatic
registration at installation time of the rss plugins,
so my Atomisatior application can discover then load it at run time.


I don't think I understand the difference between the step you're  
calling discover, scanning the registry, and actually loading the  
plugin.  Does discovering the plugin involve importing any of its  
code?



So in your way of seeing thing, you'd rather see this registration
mechanism at the application level,
but the you need to provide specific installation instructions for
people that want to add plugins.
e.g. put your package in this /plugins directory for example.


Sort of.  I see the installation and configuration of entry points as  
2 steps:


1. some variation of python setup.py install
2. update the configuration of app to tell it where to find the plugin

Some applications may provide automated tools for step 2, and I could  
see great benefit to making the loading and registration of entry  
points part of the standard library, as long as the registration is  
maintained per-application instead of site-wide.



I am advocating that the entry point mechanism is handy because it
relies on an existing mechanism : install your package
and it allows plugin sharing amongst applications.


How then do I install a package but *not* have it available for an  
application?  Suppose, for example, that I have several copies of a  
web app installed for different customers, but I want to prevent some  
of them from using an advanced plugin of some sort.



But I see the caveats you are explaining, and I understand them now;

So, what if we didn't have these entry points installed globally when
the package is installed,
but just a configuration file in the host application level to point  
them ?


a configuration file that reunites all entry points an application
uses. For the Atomisator example:

 [atomisator.reader]
 rss = somepackage.somemodule:MyPluginClass


Yes!  We can figure out the exact spelling, but we're talking about  
the same thing now.  If we use dotted notation all the way  
(somepackage.somemodule.MyPluginClass) then it's simple to just  
import the thing directly.



This would let the application consume the plugins pointed by this
configuration file
and this would remove the implicit part you don't like. I'd be very
happy with such a plugin system on my side.


If you follow the model of the logging module and provide an easy  
load plugins from an ini file, then all python apps have the feature  
of making it easy to load plugins, developers have a standard API to  
code to, and administrators don't have to worry about plugins  
sneaking into an app.



And this would fit I think in Distutils needs since we can configure
it through three levels of configuration files
distutils.cfg, pydistutils.cfg and setup.cfg


That sounds good.

Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 5, 2009, at 10:38 AM, P.J. Eby wrote:


At 08:41 AM 5/5/2009 -0400, Doug Hellmann wrote:

I don't want new functionality available to an application just
because someone has permission to install a package somewhere in the
PYTHONPATH.  I would rather have plugins added to an app through an
explicit configuration step of some sort.


Note that this is not incompatible with entry points; an application  
can simply treat entry points as a list of *available* plugins,  
rather than as a list of *active* plugins.  Chandler, for example,  
does this for user-level plugins.


That's good.  I wasn't sure if registering an entry point caused it to  
be automatically loaded in an app that expressed interest in it, or if  
there was a step the app could take to verify that it was desirable  
code before doing any imports.  Earlier messages in this thread made  
me think there was no protection from a registered plugin.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding entry points into Distutils ?

[Doug Hellmann]

On May 5, 2009, at 11:05 AM, Tarek Ziadé wrote:

On Tue, May 5, 2009 at 4:29 PM, Doug Hellmann  
doug.hellm...@gmail.com wrote



a configuration file that reunites all entry points an application
uses. For the Atomisator example:

 [atomisator.reader]
 rss = somepackage.somemodule:MyPluginClass


Yes!  We can figure out the exact spelling, but we're talking about  
the same

thing now.  If we use dotted notation all the way
(somepackage.somemodule.MyPluginClass) then it's simple to just  
import the

thing directly.


I think is simpler with the somepackage.somemodule:MyPluginClass  
notation


Good point, I was remembering the import syntax incorrectly.


And this would fit I think in Distutils needs since we can configure
it through three levels of configuration files
distutils.cfg, pydistutils.cfg and setup.cfg


That sounds good.


The only caveat I see though, is that the host app has to know the
exact location
of each plugin in the code of the third party app, whereas entry  
points provide

this information through the discovery API.


True.  On the other hand, that encourages standard locations like  
mylib.yourapp.entry_point and mylib.anotherapp.entry_point.


I think most of my concerns about the global registry are taken care  
of by the fact that discovering a plugin doesn't involve any imports  
of unknown code.  I still prefer per-app, explicit configuration of  
entry points, and think we could build a system to support that.  I  
would like to see *some* variation of this in the standard library,  
though, because I have several uses for it.


Doug

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig