At 12:42 PM 9/22/2006 -0700, Josiah Carlson wrote: > > You might as well suggest that each environment > > consist of a single large zipfile containing the packages in question: > this > > would actually be *more* practical (and fast!) in terms of Python startup, > > and is no different from having a database with respect to the need for > > installation and uninstallation to modify a central file! > >We should remember that the sizes of databases that (I expect) will be >common, we are talking about maybe 30k if a user has installed every >package in pypi. And after the initial query, everything will be stored >in a dictionary or dictionary-like object, offering faster query times >than even a zip file
Measure it. Be sure to include the time to import SQLite vs. the time to import the zipimport module. >SQLite is pretty fast. And for startup, we are really only performing a >single query per database "SELECT * FROM package_registry". It will end >up reading the entire database, but these databases will be generally >small, perhaps a few dozen rows, maybe a few thousand if we have set up >a bunch of installation-time application environments. Again, seriously, compare this against a zipfile. You'll find that there's absolutely no comparison between reading this and reading a zipfile central directory -- which also results in an in-memory cache that can then be used to seek() directly to the module. >Actually, I'm offering a way of *registering* a package with the >repository from the command line. I'm of the opinion that setting the >environment via command line for the subsequent Python runs is a bad >idea, but then again, I have been using wxPython's wxversion method for >a while to select which wxPython installation I want to use, and find >things like: > > import wxversion > wxversion.ensureMinimal('2.6-unicode', optionsRequired=True) > >To be exactly the amount of control I want, where I want it. Well, that's already easy to do for arbitrary packages and arbitrary versions with setuptools. Eggs installed in "multi-version" mode are added to sys.path at runtime if/when they are requested. >With a package registry (perhaps as I have been describing, perhaps >something different), all of the disparate ways of choosing a version of >a library during import can be removed in favor of a single mechanism. >This single mechanism could handle things like the wxPython >'ensureMinimal', perhaps even 'ensure exact' or 'use latest'. This discussion is mostly making me realize that sys.path is exactly the right thing to have, and that the only thing that actually need fixing is universal .pth support, and maybe some utility functions for better sys.path manipulation within .pth files. I suggest that there is no way an arbitrary "registry" implementation is going to be faster than reading lines from a text file. > > Setuptools works around this by installing an enhancement for the 'site' > > module that extends .pth support to include all PYTHONPATH > > directories. The enhancement delegates to the original site module after > > recording data about sys.path that the site module destroys at startup. > >But wasn't there a recent discussion describing how keeping persistant >environment variables is a PITA both during install and runtime? Yes, exactly. >Extending .pth files to PYTHONPATH seems to me like a hack meant to work >around the fact that Python doesn't have a package registry. And really, >all of the current sys.path + .pth + PYTHONPATH stuff could be subsumed >into a *single* mechanism. Sure -- I suggest that the single mechanism is none other than *sys.path*. The .pth files, PYTHONPATH, and a new command-line option merely being ways to set it. All of the discussion that's taken place here has sufficed at this point to convince me that sys.path isn't broken at all, and doesn't need fixing. Some tweaks to 'site' and maybe a new command-line option will suffice to clean everything up quite nicely. I say this because all of the version and dependency management things that people are asking about can already be achieved by setuptools, so clearly the underlying machinery is fine. It wasn't until this message of yours that I realized that you are trying to solve a bunch of problems that are quite solvable within the existing machinery. I was mainly interested in cleaning up the final awkwardness that's effectively caused by lack of .pth support for the startup script directory. > > I'm not sure of that, since I don't yet know how your approach would deal > > with namespace packages, which are distributed in pieces and assembled > > later. For example, many PEAK and Zope distributions live in the peak.* > > and zope.* package namespaces, but are installed separately, and glued > > together via __path__ changes (see the pkgutil docs). > > packages.register('zope', '/path/to/zope') > >And if the installation path is different: > > packages.register('zope.subpackage', '/different/path/to/subpackage/') > >Otherwise the importer will know where the zope (or peak) package exists >in the filesystem (or otherwise), and search it whenever 'from zope >import ...' is performed. If you're talking about replacing the current import machinery, you would have to leave this to Py3K, otherwise all you've done is add a *new* import hook, i.e. a "sys.package_loaders" dictionary or some such. If you wanted something like that now, of course, you could slap an importer into sys.meta_path that then did a lookup in sys.package_loaders. Getting this mechanism bootstrapped, however, is left as an exercise for the reader. ;) Note, by the way, that it might be quite possible to do away with everything but sys.meta_path in Py3K, prepopulated with such an importer (along with ones to support builtin and frozen modules). You could then import a backward-compatibility module that would add support for sys.path and for package __path__ attributes, by adding a new entry to sys.meta_path. But this is strictly a pipe dream where Python 2.x is concerned. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com