Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
Le lundi 06 octobre 2008 à 21:33 -0400, Phillip J. Eby a écrit : Does this mean actively avoiding an API that would allow developers to access certain types of data files (I'm thinking of the discussion about locale data and not putting anything else but .py/.pyc/.pyo files in packages) or merely making sure the existing way (of shipping data files in packages and finding them by os.path and __file__) keeps working? I would actively avoid it for a BUILDS 1.0 spec, because on any platform where the people building installation tools care about relocating these files, symlinks are available, so both sides can be happy without needing a new API. I fail to see what this would bring over the current situation, then. That is, unless I have misunderstood Josselin and Toshio, I understand symlinks to currently be an acceptable compromise. (For example, Debian uses them to relocate .pyc files currently.) Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers don’t do it. Furthermore, you seem to be unaware of the amount of abuse that was produced by the mere idea of using __file__ to locate data. It is not only about loading a few image files, let me show you some cases. Case 1: pastescript. This module ships templates of python modules, that are named with the .py extension but are not usable as is from the interpreter, and shipped in the modules directory. How in the world can our tools know that they shouldn’t be byte-compiled and that they should be relocated to somewhere in /usr/share/$package? Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions. Case 3: GStreamer. The modules installation directory is detected at compile time and then hardcoded in the modules themselves so that some subdirectories can be added to sys.path upon loading. In the end, the modules are not relocatable. I keep wondering what Python module developers will invent next to complicate packaging even more. Frankly, we’d lose much less hair if such things were simply impossible. At least, it should be documented that such practice is wrong, so that we can actually consider it buggy and make our tools simpler. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
On Tue, 7 Oct 2008 13:12:03 +0100, Floris Bruynooghe [EMAIL PROTECTED] wrote: On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote: On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions. The facts you have presented above are not correct. It would be helpful if you could correct them in that case. I don't think the details of the plugin system are relevant to the topic under discussion here. The installation requirements are not unusual for the most part - that a directory full of .py files be copied to the install location, just as any package would be. The one unusual thing is that non-Twisted package X might want to copy files into twisted/plugins/ (if it is providing plugins). For details, see the Twisted plugin documentation: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html Or this discussion in the Debian issue tracker: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=474630 I would be happy to answer any questions not addressed in the documentation, but perhaps off-list if they are not directly relevant to distribution topics. Jean-Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Downloading betas?
On Oct 7, 2008, at 5:41 AM, [EMAIL PROTECTED] wrote: I have been using ZODB 3.6.0 for SpamBayes for awhile. I noticed today that ZODB 3.8.0 is available from PyPI, so I ran easy_install -U ZODB3 It determined that 3.8.1b8 was the best match. Why didn't it stop with 3.8.0 instead of installing a version which was clearly marked as beta? Is there a way to tell it, don't do that? No. Note that buildout has a prefer-final option to prefer final releases over non-final releases. Jim -- Jim Fulton Zope Corporation ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
On Tue, 07 Oct 2008 15:19:57 +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Le mardi 07 octobre 2008 à 08:58 -0400, Jean-Paul Calderone a écrit : I don't think the details of the plugin system are relevant to the topic under discussion here. The installation requirements are not unusual for the most part - that a directory full of .py files be copied to the install location, just as any package would be. The one unusual thing is that non-Twisted package X might want to copy files into twisted/plugins/ (if it is providing plugins). This is not unusual, but other frameworks that do the same don’t break when you add namespace packages. In fact, the very fact that you can’t add namespace packages shows that you are abusing the python modules directory: you are shipping a twisted/plugins/foo/foo.py file which is not meant to be available as a twisted.plugins.foo.foo module. I think you meant shipping a twisted/plugins/foo.py file which is not meant to be available as a twisted.plugins.foo module, since as far as I know, no one is putting packages (as opposed to modules) into the twisted.plugins package (and I don't think it works, either). The ability to install several incompatible versions of Twisted on the system is the reason I was given by glyph; if there is another reason, I’d be happy to hear the explanation, but the result is the same: we had to code a special case for Twisted. Above correction aside, your understanding of the system is still flawed (I am fully aware that this is likely due to our failure to document the system sufficiently, so please don't interpret that as an attack). I think that you also misunderstood the explanation that Glyph provided. In your earlier message, you wrote this: The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. However, no such feature is offered by the Twisted plugin system. :) The issue which arose when Debian packaged Twisted and (for example) a new version of Nevow which provides a plugin to Twisted is that Nevow's plugin was installed to a location which was not the location intended or expected by the Twisted developers or the Nevow developers. The expectation of the Twisted developers was that the installed directory structure for Twisted files would look like this: /usr/lib/python2.5/site-packages/twisted/ /usr/lib/python2.5/site-packages/twisted/__init__.py /usr/lib/python2.5/site-packages/twisted/plugins/ /usr/lib/python2.5/site-packages/twisted/plugins/__init__.py (obviously this is only part of what a Twisted installation would look like) The expectations of the Nevow developers was that a file included in Nevow, nevow_widget.py, would be installed to /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py On Debian, difficulties were encountered because, while Twisted was installed more or less to the expected location, the file in Nevow was instead installed to /var/lib/python-support/python2.5/twisted/plugins/nevow_widget.py and the infrastructure for installing the file to this location also creates /var/lib/python-support/python2.5/twisted/ /var/lib/python-support/python2.5/twisted/__init__.py /var/lib/python-support/python2.5/twisted/plugins/ /var/lib/python-support/python2.5/twisted/plugins/__init__.py This is problematic because there are now two Twisted packages in the module search path. Python's module system can't do anything with the second one (which one is second depends on the order of sys.path). If the version created for nevow_widget.py came first, the Twisted package itself would basically be completely broken. Fortunately, it doesn't, so that doesn't happen. However, since nevow_widget.py is in the second version, the only way it can be imported is if some extra help is supplied to the import system. In previous releases of Twisted, this help was given by twisted/plugins/ __init__.py. It included an extremely primitive implementation of the concept of namespace packages (_extremely_ primitive). It would find all directories named twisted/plugins which were in the module search path and add them to its own __path__. The purpose of this feature wasn't to support installations, however. The purpose of this feature was as a developer convenience. It allows developers to edit plugins in a non- installed location (for example, their home directory) and then use them without copying them to the installed Twisted's twisted/plugins/ directory. In a more recent release of Twisted, this feature was tweaked slightly. The reason for the change is that there was a significant bug in the old implementation of namespace packages. If a developer was using their own copy of Twisted (for example, an SVN checkout in their home directory) instead of the installed version, the namespace package implementation would still load plugins from the *installed* version of Twisted. In the best possible case, this gave you two copies of
Re: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals
On Sun, Oct 5, 2008 at 12:04 PM, zooko [EMAIL PROTECTED] wrote: 5/ ideally, they should be one and only one version of a given package in an OS-based installation -1 -- This is the strong preference of the folks who package software for OSes -- Debian, Fedora, etc. -- but it is not necessarily the choice of the users who use their OSes. It is best for the Python packaging standards to be agnostic towards this, or at least to support both this desideratum and its opposite. I can see this as an exponentional problem for packagers, but let's think about it: - How you would handle several version of the same package in Python then ? - How each application would pick the right version ? - How would you decide which version is the one by default ? That is the core of the problem. The -m feature of setuptools is nice, but it activates one version at a time, and this is globlal to Python unless each application is handling the version switch, wich is pretty heavy. A programmable sys.path ? Tarek. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
Le mardi 07 octobre 2008 à 08:58 -0400, Jean-Paul Calderone a écrit : I don't think the details of the plugin system are relevant to the topic under discussion here. The installation requirements are not unusual for the most part - that a directory full of .py files be copied to the install location, just as any package would be. The one unusual thing is that non-Twisted package X might want to copy files into twisted/plugins/ (if it is providing plugins). This is not unusual, but other frameworks that do the same don’t break when you add namespace packages. In fact, the very fact that you can’t add namespace packages shows that you are abusing the python modules directory: you are shipping a twisted/plugins/foo/foo.py file which is not meant to be available as a twisted.plugins.foo.foo module. The ability to install several incompatible versions of Twisted on the system is the reason I was given by glyph; if there is another reason, I’d be happy to hear the explanation, but the result is the same: we had to code a special case for Twisted. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Availability of setuptools installer for python2.6?
2008/10/7 Phillip J. Eby [EMAIL PROTECTED]: This is a really frustrating aspect of setuptools, that pure-Python packages produce version-specific installers. Actually, that's not setuptools' fault in this case; I specifically make the .exe's version-specific because they have different contents. Different versions of Python include different distutils commands, and setuptools needs to install different things. So even though it's pure Python (ha!) it is still Python-version specific. Not sure I follow this. I see this in bdist_wininst installers, so distutils commands shouldn't be relevant (?) But I'll freely admit I'm naive over this stuff, so if I need to be enlightened, please do so! Eggs contain bytecode, and bytecode is Python version-specific. bdist_wininst recompiles bytecode at install-time, so that's not relevant for me. I can see it would be for eggs, but I'm talking about installers, here. Hmm, I just went looking for a specific example, and I see that the one I thought I remembered, Genshi, includes some C code I'd missed. Maybe I'm misinterpreting what I thought I'd seen. But I do wish people would continue to distribute bdist_wininst installers for pure python packages, rather than just version-specific .egg files. Never mind. We've done this discussion to death in the past. Let's just say that the move from bdist_wininst to eggs in some areas, is making an early switch to Python 2.6 harder for me on Windows than the equivalent switch to 2.5 was... Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions. Hi Josselin, The facts you have presented above are not correct. Jean-Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote: On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions. The facts you have presented above are not correct. It would be helpful if you could correct them in that case. Cheers Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Downloading betas?
I have been using ZODB 3.6.0 for SpamBayes for awhile. I noticed today that ZODB 3.8.0 is available from PyPI, so I ran easy_install -U ZODB3 It determined that 3.8.1b8 was the best match. Why didn't it stop with 3.8.0 instead of installing a version which was clearly marked as beta? Is there a way to tell it, don't do that? Thx, Skip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
Le mardi 07 octobre 2008 à 10:02 -0400, Jean-Paul Calderone a écrit : The expectations of the Nevow developers was that a file included in Nevow, nevow_widget.py, would be installed to /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py This expectation is wrong. You’re shipping it as a Python module, the only thing you can expect is to be able to import it. This wouldn’t happen if you were shipping plugins in a specific, private, plugins directory. The ideal fix, from my perspective as a Twisted developer, is to install the nevow_widget.py file into /usr/lib/python2.5/site-packages/twisted/plugins/ As explained on the debian-python list (see http://lists.debian.org/debian-python/2008/05/msg00032.html), there are strong reasons for not putting managed modules in the same directory as modules included in the Python distribution. So you can’t expect modules to be in a specific directory (but this shouldn’t be a problem since these are not modules, hmmm?). (I have some ideas for keeping all files at the same place and moving only .pyc files to /var, but it requires patching the interpreter and the Debian Python maintainer disagrees, so we’ll have to stick to the current situation for now.) along with all the other Twisted plugin files (perhaps using a symlink, I can't think of any reason why that would make a difference, particularly since all the other .py files in that area are symlinks). In other words, Debian took advantage of a feature which was intended to be used only as a convenience for development and was never intended to assist installation. The fact that we use a .pth to specify the different module hierarchy is an implementation detail but is indeed abusing this feature; we should be shipping a modified setup.py instead. However this has nothing to do with the inability of Twisted to cope with multiple module paths. Let’s take another example that is not Debian specific: what if I want to install a plugin to /usr/local? The fact that you are abusing the python modules directory forces me to install it to /usr/local/python2.5/site-packages/twisted/plugins instead of e.g. /usr/local/share/twisted/plugins, where a normal application or library would be looking for its plugins. And there, I have to deal with namespaces and other python module specificities that should be irrelevant for plugins. In the beginning, Debian only expected the Python modules directory to contain, well, Python modules. It seems even that is too much to expect, since module developers are taking advantage of a set of features (module file location) that were initially meant for introspection. Have I succeeded in explaining why the Twisted plugin system isn't making any unusual requirements of the installation system nor using the Python module system in an unusual way? Not really. You have succeeded in explaining what is the exact nature of the unusual requirements you make, though. Thanks for that, as it is much clearer than the original explanations I received. Can I expect to see any improvements to the Debian packaging of Twisted and projects which supply plugins to Twisted? :) Until you fix your plugin system, we are bound to treat Twisted packages as a special case. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
2008/10/7 Phillip J. Eby [EMAIL PROTECTED]: In the case of BUILDS, I propose to do the same: define a standard whose cost/benefit ratios are ideally balanced for each participant. This does not, by the way, mean that everybody ends up with the same cost/benefit ratio; it simply means that the cost/benefit ratios are best for those people whose participation is most required for the standard to be widely adopted. You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers. I'd argue (you may differ) that the most significant area where you missed the mark on user benefits with easy_install and setuptools is the lack of easy *uninstall* and easy *list* options. Most of the issues I hear from users about setuptools (filtered by my prejudices, admittedly) is that there's no management options (which brings in the system packagers, and their concerns). Can I suggest that this be included somehow in the new spec, so that metadata is available to make wtiting uninstallers and listers as easy as writing installers? Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Distutils and PyPI : P4-Sprint in D.C.
On Tue, Oct 7, 2008 at 10:35 AM, Tarek Ziadé [EMAIL PROTECTED] wrote: We are going to have a P4-sprint (pre-pre-pre-PEP sprint) in D.C. during the Plone Conference. Very cool. Given the venue, should people expect that they're welcome in person even if not associated with the Plone conference? I don't know if I'll be able to steal some time from my family, but there's a possibility. -Fred -- Fred L. Drake, Jr.fdrake at gmail.com Chaos is the score upon which reality is written. --Henry Miller ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Twisted packaging in Debian (was Re: Pre-pre-PEP: Requirements for the Python BUILDS Specification)
On Tue, 07 Oct 2008 17:11:19 +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Le mardi 07 octobre 2008 à 10:02 -0400, Jean-Paul Calderone a écrit : The expectations of the Nevow developers was that a file included in Nevow, nevow_widget.py, would be installed to /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py This expectation is wrong. You’re shipping it as a Python module, the only thing you can expect is to be able to import it. Leaving aside whether or not I think that is a reasonable restriction to impose, I don't see the relevance. The only reason the module can be imported on Debian now is because of the special measures included (by upstream developers, not by Debian) in twisted/plugins/__init__.py. I can't understand how this could be part of Debian's Python packaging policy, so perhaps I'm missing something very fundamental. [snip - specific, private, plugins directory which I may not understand, but doubt is relevant] The ideal fix, from my perspective as a Twisted developer, is to install the nevow_widget.py file into /usr/lib/python2.5/site-packages/twisted/plugins/ As explained on the debian-python list (see http://lists.debian.org/debian-python/2008/05/msg00032.html), there are strong reasons for not putting managed modules in the same directory as modules included in the Python distribution. So you can’t expect modules to be in a specific directory (but this shouldn’t be a problem since these are not modules, hmmm?). They're modules. They get imported. That makes them modules, right? [EMAIL PROTECTED]:~$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type help, copyright, credits or license for more information. import twisted.plugins.nevow_widget Do we have the same definition of module? I chose the phrase The ideal fix, from my perspective as a Twisted developer very intentionally. If the file ends up somewhere else but it's still an importable module, fine - if, from a Debian packager's perspective that's better, fine. I don't believe that the current situation is better from a Debian packager's perspective, though, since it involves carrying a patch that reverts an upstream bugfix. [snip - .pyc files in /var - not directly relevant] [snip - use of .pth file - not directly relevant] However this has nothing to do with the inability of Twisted to cope with multiple module paths. Let’s take another example that is not Debian specific: what if I want to install a plugin to /usr/local? The fact that you are abusing the python modules directory forces me to install it to /usr/local/python2.5/site-packages/twisted/plugins instead of e.g. /usr/local/share/twisted/plugins, where a normal application or library would be looking for its plugins. And there, I have to deal with namespaces and other python module specificities that should be irrelevant for plugins. No, this is wrong. Install it to /usr/local/share/ if you want. It will work just fine. Why do you think it won't? I'd rather you _didn't_, at least not without a couple other changes, and I don't understand why you would _want_ to, but it's possible. In the beginning, Debian only expected the Python modules directory to contain, well, Python modules. It seems even that is too much to expect, since module developers are taking advantage of a set of features (module file location) that were initially meant for introspection. I assume you're talking about the Python modules directory in the install target (not, say, in an upstream VCS). Twisted plugins are modules, so Debian's expectations (be they valid or otherwise, I make no judgement) aren't violated by them (oops, except for dropin.cache files, but we haven't really talked about those yet and I don't think they're actually contentious). Have I succeeded in explaining why the Twisted plugin system isn't making any unusual requirements of the installation system nor using the Python module system in an unusual way? Not really. You have succeeded in explaining what is the exact nature of the unusual requirements you make, though. Thanks for that, as it is much clearer than the original explanations I received. I'm glad this was clearer. I'm not sure I've yet succeeded in entirely explaining the system. I hope this follow up goes a little bit further toward that goal. [snip - I can't expect to see any improvements yet] Jean-Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Twisted packaging in Debian (was Re: Pre-pre-PEP: Requirements for the Python BUILDS Specification)
Le mardi 07 octobre 2008 à 11:44 -0400, Jean-Paul Calderone a écrit : They're modules. They get imported. That makes them modules, right? Fine. I hadn’t understood that they are actually meant to be imported as is (which makes you right in your assertions that the explanations weren’t completely clear to me yet). If they are modules, meant to be imported, they actually have their place in the modules directory, so please disregard what I’ve said about private directories. So, if they are modules, they sometimes need namespace packages to be imported properly. However you intentionally do not import modules that have associated namespace packages, which is the exact opposite of what we expect for other modules. Let me comment on your previous email with that in mind. In previous releases of Twisted, this help was given by twisted/plugins/ __init__.py. It included an extremely primitive implementation of the concept of namespace packages (_extremely_ primitive). It would find all directories named twisted/plugins which were in the module search path and add them to its own __path__. The purpose of this feature wasn't to support installations, however. The purpose of this feature was as a developer convenience. It allows developers to edit plugins in a non- installed location (for example, their home directory) and then use them without copying them to the installed Twisted's twisted/plugins/ directory. OK, I get that now. You need to do that because importing plugins.something in the twisted/ directory will only look for subdirectories of the same installation path. In a more recent release of Twisted, this feature was tweaked slightly. The reason for the change is that there was a significant bug in the old implementation of namespace packages. If a developer was using their own copy of Twisted (for example, an SVN checkout in their home directory) instead of the installed version, the namespace package implementation would still load plugins from the *installed* version of Twisted. In the best possible case, this gave you two copies of every plugin. In the worst case, the installed version of Twisted wasn't compatible with the other version of Twisted and the plugins would raise exceptions. About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions. Otherwise - and this is something that will hit distributions hard - you may install a an incompatible version of, say, Nevow, and get your installation fucked up. However, in the end, I think you’ve been hit by an issue that was only sporadically met until now: managing files belonging to the same module hierarchy with two different tools doesn’t always work. And this is something that we need to manage inside Debian (which is more a political problem than a technical one). Do we have the same definition of module? I think so. I chose the phrase The ideal fix, from my perspective as a Twisted developer very intentionally. If the file ends up somewhere else but it's still an importable module, fine - if, from a Debian packager's perspective that's better, fine. I don't believe that the current situation is better from a Debian packager's perspective, though, since it involves carrying a patch that reverts an upstream bugfix. In the end, I don’t think this patch was added as there is another workaround. Either way, it may break again if you change the way you look for plugins, so you are right to expect that they land in the same directory. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Availability of setuptools installer for python2.6?
At 09:58 AM 10/7/2008 +0100, Paul Moore wrote: 2008/10/7 Phillip J. Eby [EMAIL PROTECTED]: This is a really frustrating aspect of setuptools, that pure-Python packages produce version-specific installers. Actually, that's not setuptools' fault in this case; I specifically make the .exe's version-specific because they have different contents. Different versions of Python include different distutils commands, and setuptools needs to install different things. So even though it's pure Python (ha!) it is still Python-version specific. Not sure I follow this. I see this in bdist_wininst installers, so distutils commands shouldn't be relevant (?) I'm saying that setuptools' own bdist_wininst installer is version specific because setuptools itself includes different code (and different script names, e.g. easy_install-2.x) depending on the Python version. So, even though setuptools is pure Python, its bdist_wininst files are version-specific. This does not have anything to do with packages that simply *use* setuptools; if you make a bdist_wininst of some non-setuptools package, it will not be version specific unless you include C code. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
At 10:04 AM 10/7/2008 +0100, Paul Moore wrote: 2008/10/7 Phillip J. Eby [EMAIL PROTECTED]: In the case of BUILDS, I propose to do the same: define a standard whose cost/benefit ratios are ideally balanced for each participant. This does not, by the way, mean that everybody ends up with the same cost/benefit ratio; it simply means that the cost/benefit ratios are best for those people whose participation is most required for the standard to be widely adopted. You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers. I'd argue (you may differ) that the most significant area where you missed the mark on user benefits with easy_install and setuptools is the lack of easy *uninstall* and easy *list* options. Well, I'd certainly agree that those features are desirable. But I didn't miss the mark, in that those features were not part of my mark. ;-) The goal was to get widespread adoption by developers, and thus the primary target audience was developers. Any features that attracted users were there only insofar as the benefit to users would specifically drive adoption by developers. And that means that features that only matter *after* you have things installed (i.e., easy uninstall and list) were of relatively little importance in the initial feature set. It was only necessary to provide the *possibility* of uninstall and list features, and then allow others to scratch the resulting itches. Now, for BUILDS, the situation is different. Just as setuptools' competitor was distutils, BUILDS' competitor is setuptools. So, BUILDS has to offer developers a switching benefit. And, unlike the situation before, now developers are much more likely to have a lot of packages they'd like to list or uninstall, so such features have considerably more value now, than they did then, which makes them worth putting some effort into. Also, since that switching benefit is largely going to come from having better management tools, the management tools need to be easy to build (so that people will build them). However, since system packagers are experiencing pain right now wrt Python packages, there is at least some benefit to their participating in the process. Back when setuptools was developed, people from the distros weren't hanging out here in the distutils-sig... or at least, they never answered any of my calls for interested parties. Most of the issues I hear from users about setuptools (filtered by my prejudices, admittedly) is that there's no management options (which brings in the system packagers, and their concerns). Can I suggest that this be included somehow in the new spec, so that metadata is available to make wtiting uninstallers and listers as easy as writing installers? Since the single most important part of BUILDS (and the part that should be done first) is an installation manifest for packaging tools to know what files are of what kind and where, it should be easy for uninstallers to work. Regarding listers, there are already a number of them available in the Cheeseshop, and if you're working with Python 2.5+ (and not using an older distro that deletes .egg-info files), they should also work with distutils-installed packages. The idea behind the installation manifest and metadata (the IL and DS of BUILDS) is that it should be possible for anyone to write their own management tools for specific scenarios, and not have to rely on a bunch of implementation-specific and not-very-well-documented stuff. (It should also be possible, given the IL part, to make a distutils and/or setuptools command that generates this data from a sufficiently well-behaved setup.py.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote: Le lundi 06 octobre 2008 à 21:33 -0400, Phillip J. Eby a écrit : Does this mean actively avoiding an API that would allow developers to access certain types of data files (I'm thinking of the discussion about locale data and not putting anything else but .py/.pyc/.pyo files in packages) or merely making sure the existing way (of shipping data files in packages and finding them by os.path and __file__) keeps working? I would actively avoid it for a BUILDS 1.0 spec, because on any platform where the people building installation tools care about relocating these files, symlinks are available, so both sides can be happy without needing a new API. I fail to see what this would bring over the current situation, then. That is, unless I have misunderstood Josselin and Toshio, I understand symlinks to currently be an acceptable compromise. (For example, Debian uses them to relocate .pyc files currently.) Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers donât do it. Which is why the idea for the BUILDS spec to include a way for automated tools to do it, so that you won't have to do it manually. Or are you saying that that isn't an improvement over the current situation? Yes, it's true that I'm saying that developers should not be *required* to add the extra data to their packages, but that they should be *able* to, and if it is trivial to add the extra data, most should accept patches or respond to requests to do so. Right now, it's not even *possible* for them to do so, however. Also, I wasn't aware of those peculiarities regarding Twisted and GStreamer -- neither one uses setuptools. But certainly they should make interesting test cases for any proposed standard, wrt to finding ways to satisfy both parties. Please understand that I'm not saying we must be 100% backward compatible with 100% of the packages; we just need to be compatible enough with enough of the packages for network effects to drive the adoption of the rest. At the same time, the people I'd most like to see on the PEP team from the developer-user side would definitely include folks from Twisted, Numpy, Enthought, and Zope, as they are the folks who have most stressed the distutils to the limits and beyond. Just getting you and them in the same room so to speak seems to already be producing some benefits. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals
At 10:07 AM 10/7/2008 -0400, Tarek Ziadé wrote: The -m feature of setuptools is nice, but it activates one version at a time, and this is globlal to Python unless each application is handling the version switch, wich is pretty heavy. With or without the -m switch, scripts installed by setuptools will find the version they are specified to use, without the user needing to do anything. So, you can have a default version of an egg (used by the interpreter and non-setuptools scripts), and then some non-default versions that are used by scripts. zc.buildout and virtualenv also have their own ways of accomplishing the same thing, e.g., by hardcoding paths in an installed script. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Distutils and PyPI : P4-Sprint in D.C.
Hey all, We are going to have a P4-sprint (pre-pre-pre-PEP sprint) in D.C. during the Plone Conference. The idea is to try to bring the discussions that have been going on in the mailing lists into the next stage. Please join us if you are interested, even if you are not in D.C. http://www.openplans.org/projects/plone-conference-2008-dc/distribute Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
Le mardi 07 octobre 2008 à 14:40 -0400, Phillip J. Eby a écrit : At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote: Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers don’t do it. Which is why the idea for the BUILDS spec to include a way for automated tools to do it, so that you won't have to do it manually. If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way. Or are you saying that that isn't an improvement over the current situation? It is. I didn’t understand you wanted to automate the symlinks creation. It indeed means less burden on the maintainers, but it would be a shame to keep the same mess in binary packages, while having a tool to do that would allow to make things cleaner. Yes, it's true that I'm saying that developers should not be *required* to add the extra data to their packages, but that they should be *able* to, and if it is trivial to add the extra data, most should accept patches or respond to requests to do so. Right now, it's not even *possible* for them to do so, however. It is possible - if they use autoconf instead. BTW, I would consider it a good approach to try bringing BUILDS on par with the autotools capabilities. These tools have serious drawbacks, but they were written with the ability to work for distributors in mind. At the same time, the people I'd most like to see on the PEP team from the developer-user side would definitely include folks from Twisted, Numpy, Enthought, and Zope, as they are the folks who have most stressed the distutils to the limits and beyond. Just getting you and them in the same room so to speak seems to already be producing some benefits. Looks like a sane approach. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals
On Tue, Oct 7, 2008 at 2:42 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 10:07 AM 10/7/2008 -0400, Tarek Ziadé wrote: The -m feature of setuptools is nice, but it activates one version at a time, and this is globlal to Python unless each application is handling the version switch, wich is pretty heavy. With or without the -m switch, scripts installed by setuptools will find the version they are specified to use, without the user needing to do anything. So, you can have a default version of an egg (used by the interpreter and non-setuptools scripts), and then some non-default versions that are used by scripts. zc.buildout and virtualenv also have their own ways of accomplishing the same thing, e.g., by hardcoding paths in an installed script. in a plain python setup, If foo 1.2 is the default, and a package wants use foo 1.4, it needs to specifically call pkg_resources.require() in the code, to activate it in sys.path before importing foo in the code. Since each package can list with setuptools its dependencies with versions in install_requires, how hard would it be to automatically call the right require() calls when the package is used ? Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals
Tarek Ziadé wrote: On Tue, Oct 7, 2008 at 2:42 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 10:07 AM 10/7/2008 -0400, Tarek Ziadé wrote: The -m feature of setuptools is nice, but it activates one version at a time, and this is globlal to Python unless each application is handling the version switch, wich is pretty heavy. With or without the -m switch, scripts installed by setuptools will find the version they are specified to use, without the user needing to do anything. So, you can have a default version of an egg (used by the interpreter and non-setuptools scripts), and then some non-default versions that are used by scripts. zc.buildout and virtualenv also have their own ways of accomplishing the same thing, e.g., by hardcoding paths in an installed script. in a plain python setup, If foo 1.2 is the default, and a package wants use foo 1.4, it needs to specifically call pkg_resources.require() in the code, to activate it in sys.path before importing foo in the code. Since each package can list with setuptools its dependencies with versions in install_requires, how hard would it be to automatically call the right require() calls when the package is used ? require() is recursive, so as long as the original script is explicitly loaded (e.g., from a binary script, or something that loads eggs/entry points) then the proper versions will be loaded. Though as far as I know, pkg_resources won't remove other versions of the egg from the path, so it only works if there are no active versions of the eggs. Which isn't how many people install packages, so this feature of require() doesn't get used for much of anything (at least that I've seen). I'll also note that the require in setuptools-generated scripts causes pretty frequent problems for people, all to support this multi-version feature that no one really uses. An example of an easy way to cause the problem, if you do: python setup.py develop; svn up; python setup.py egg_info it'll break any scripts, or if you install a script in an unusual location, or use $PYTHONPATH but don't set $PATH so that you get an unexpected script that doesn't match your libraries -- since pyinstall is using --single-version-externally-managed, I kind of wish I could easily turn off the require() as well (I could monkeypatch setuptools to remove it, but I've been burned by going down that path before). -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals
At 02:58 PM 10/7/2008 -0400, Tarek Ziadé wrote: On Tue, Oct 7, 2008 at 2:42 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 10:07 AM 10/7/2008 -0400, Tarek Ziadé wrote: The -m feature of setuptools is nice, but it activates one version at a time, and this is globlal to Python unless each application is handling the version switch, wich is pretty heavy. With or without the -m switch, scripts installed by setuptools will find the version they are specified to use, without the user needing to do anything. So, you can have a default version of an egg (used by the interpreter and non-setuptools scripts), and then some non-default versions that are used by scripts. zc.buildout and virtualenv also have their own ways of accomplishing the same thing, e.g., by hardcoding paths in an installed script. in a plain python setup, If foo 1.2 is the default, and a package wants use foo 1.4, it needs to specifically call pkg_resources.require() in the code, to activate it in sys.path before importing foo in the code. You can't un-default the default, actually. If there's a default, it can't be replaced once pkg_resources has been imported. Since each package can list with setuptools its dependencies with versions in install_requires, how hard would it be to automatically call the right require() calls when the package is used ? This is already done by setuptools-generated scripts. Same for zc.buildout and virtualenv, they just do it differently. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
At 08:58 PM 10/7/2008 +0200, Josselin Mouette wrote: Le mardi 07 octobre 2008 à 14:40 -0400, Phillip J. Eby a écrit : At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote: Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers donât do it. . Which is why the idea for the BUILDS spec to include a way for automated tools to do it, so that you won't have to do it manually. If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way. I'm not sure I follow you. What I'm proposing is that: 1. A BUILDS manifest for a project should list the files included in the project by what kind of file they are, as well as by inter-file relationships (e.g., these 3 files are part of Python package 'foo.bar' within this project, and they are static data used at runtime but not read or written). 2. BUILDS-compatible install tools will be free to install those files wherever they like (or create .debs, .rpms, etc. that do), and use symlinks so that the installed project can find them in the places that the developer wanted to find them. Is this what you're asking for? Because it's what I've been trying to say we should do. Or are you saying that that isn't an improvement over the current situation? It is. I didnât understand you wanted to automate the symlinks creation. It indeed means less burden on the maintainers, but it would be a shame to keep the same mess in binary packages, while having a tool to do that would allow to make things cleaner. I'm not sure I understand you here, either. I'm saying that a BUILDS-compatible install tools for Debian and RPM systems should create system packages with the stuff wherever the OS people want it to go, *and* symlinks so the Python code can find it. And that this should be an automated process, to the extent possible/practical. For example, if Debian needs metadata that's not in the core BUILDS metadata, it should be possible within BUILDS to include it, so that it can be contributed upstream. Likewise, it should be possible for a BUILDS install tool to use a third-party supplement to get that data, so that when you're initially porting a project, you could write a file with the additional data while doing the initial port, then submit that file upstream. Yes, it's true that I'm saying that developers should not be *required* to add the extra data to their packages, but that they should be *able* to, and if it is trivial to add the extra data, most should accept patches or respond to requests to do so. Right now, it's not even *possible* for them to do so, however. It is possible - if they use autoconf instead. Sure, but who's going to convince them to do *that*? ;-) Less flippantly, our goal here is to have something that has a better cost/benefit ratio for adoption by a Python developer who has not seen any pressing need for autoconf in their life to date. And if somebody wants to make a BUILDS tool to generate autoconf sources from a project manifest, then great! BTW, I would consider it a good approach to try bringing BUILDS on par with the autotools capabilities. These tools have serious drawbacks, but they were written with the ability to work for distributors in mind. Sure. I like stealing designs and requirements definition from mature projects, even if you might argue that I chose poor role models for setuptools. (Java and OSGi.) However, since autotools compatibility isn't a goal in itself, I think I'll leave it to those people who have interest/experience to propose what specific autotools features they'd like to see in BUILDS, and the use cases they have for them. If I were to just skim the docs trying to figure that out, I might think that something cool in it is actually useful, or vice versa. ;-) So, it's better here I think to hear from experienced users such as yourself. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
2008/10/7 Phillip J. Eby [EMAIL PROTECTED]: At 10:04 AM 10/7/2008 +0100, Paul Moore wrote: 2008/10/7 Phillip J. Eby [EMAIL PROTECTED]: You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers. I'd argue (you may differ) that the most significant area where you missed the mark on user benefits with easy_install and setuptools is the lack of easy *uninstall* and easy *list* options. Well, I'd certainly agree that those features are desirable. But I didn't miss the mark, in that those features were not part of my mark. ;-) The goal was to get widespread adoption by developers, and thus the primary target audience was developers. Any features that attracted users were there only insofar as the benefit to users would specifically drive adoption by developers. OK, I was assuming that when you said I only considered developers and users, you were implying that you were aiming to provide for users benefits (and hence, the user benefit of package management - uninstall and list - was important). As you say, though, your key aim was to drive adoption, so making *installation* easy for users is important, but making *uninstall* easy isn't, as it's not something related to adoption - rather the opposite :-) Of course, as a user, my perspectives are different :-) And as a user who is increasingly resistant to setuptools-enabled packages precisely because they don't address user-level issues, I suspect that catering for user level requirements (in addition to system packagers', and the already covered developers) is more relevant this time round. My feeling, by the way, is that system packagers are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers) whereas on Windows, users are the more relevant group (as there isn't really a significant system packager role in the first place). Whether this, coupled with the notoriously difficult task of getting Windows and Linux/Unix users to understand each other's perspectives, is an important fact, only time will tell. Thanks for taking the time to clarify for me. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Twisted packaging in Debian
Josselin Mouette [EMAIL PROTECTED] writes: About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions. You've referred to this a few times now without explaining exactly what you expect the developer to do. So, just in case: I presume you mean that, in the case where a program's plugins are not compatible with plugins for previous versions of the same program, the developer should always seek (and attempt to install) plugins for version N in a differently-named directory from plugins for version M and version P. That is, that a directory expected to contain plugins for version N should contain *only* plugins for version N, and never for other versions. Yes? -- \ “Pinky, are you pondering what I'm pondering?” “I think so, | `\ Brain, but isn't that why they invented tube socks?” —_Pinky | _o__) and The Brain_ | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Simultaneous multi-version support (was: [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals)
Ian Bicking [EMAIL PROTECTED] writes: I'll also note that the require in setuptools-generated scripts causes pretty frequent problems for people, all to support this multi-version feature that no one really uses. I agree heartily that it seems to cause more trouble than it's worth — for my assessment of its worth, anyway. Is it true that “no-one” (to some epsilon value) actually uses this feature? -- \ “The best ad-libs are rehearsed.” —Graham Kennedy | `\ | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Simultaneous multi-version support
Ben Finney wrote: Ian Bicking [EMAIL PROTECTED] writes: I'll also note that the require in setuptools-generated scripts causes pretty frequent problems for people, all to support this multi-version feature that no one really uses. I agree heartily that it seems to cause more trouble than it's worth — for my assessment of its worth, anyway. Is it true that “no-one” (to some epsilon value) actually uses this feature? There is one person on enthought-dev who does this for everything. He says it keeps him honest about his dependencies. And consequently keeps us at Enthought honest about ours. I typically have multiple versions of things installed, and switch between them, but I never use pkg_resources.require() to do so. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
Le mardi 07 octobre 2008 à 16:51 -0400, Phillip J. Eby a écrit : If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way. I'm not sure I follow you. What I'm proposing is that: 1. A BUILDS manifest for a project should list the files included in the project by what kind of file they are, as well as by inter-file relationships (e.g., these 3 files are part of Python package 'foo.bar' within this project, and they are static data used at runtime but not read or written). 2. BUILDS-compatible install tools will be free to install those files wherever they like (or create .debs, .rpms, etc. that do), and use symlinks so that the installed project can find them in the places that the developer wanted to find them. Is this what you're asking for? Because it's what I've been trying to say we should do. What I am afraid of is that, by adding just another layer, you will introduce new problems while fixing existing ones. Currently we already have a hard time maintaining a symlink farm, and adding a second symlink farm on top of it is not going to make things more understandable (currently, package maintainers already have a hard time understanding how to deal with Python modules) nor more reliable. The first kind of issues I can think of is that dpkg handles symlinks to directories very badly, so you’re likely to run into issues that have to be dealt with by hand by maintainers. I haven’t thought about it much, but I have the feeling that other things will break. In the end, if you are designing a new packaging specification and a new tool, I think you’d better take this as an occasion to do the right thing instead of adding this new layer. For example, if Debian needs metadata that's not in the core BUILDS metadata, it should be possible within BUILDS to include it, so that it can be contributed upstream. Likewise, it should be possible for a BUILDS install tool to use a third-party supplement to get that data, so that when you're initially porting a project, you could write a file with the additional data while doing the initial port, then submit that file upstream. If there is a need for Debian-specific metadata, it will go in the debian/ directory. It looks interesting to integrate it upstream only if it is relevant for other distributors. Less flippantly, our goal here is to have something that has a better cost/benefit ratio for adoption by a Python developer who has not seen any pressing need for autoconf in their life to date. And if somebody wants to make a BUILDS tool to generate autoconf sources from a project manifest, then great! No, no. Please don’t :) But if you can express in the project manifest the same flexibility as with autoconf (or even the tenth of it, we don’t need *that* much flexibility), that should be enough. BTW, I would consider it a good approach to try bringing BUILDS on par with the autotools capabilities. These tools have serious drawbacks, but they were written with the ability to work for distributors in mind. Sure. I like stealing designs and requirements definition from mature projects, even if you might argue that I chose poor role models for setuptools. (Java and OSGi.) However, since autotools compatibility isn't a goal in itself, I think I'll leave it to those people who have interest/experience to propose what specific autotools features they'd like to see in BUILDS, and the use cases they have for them. If I were to just skim the docs trying to figure that out, I might think that something cool in it is actually useful, or vice versa. ;-) So, it's better here I think to hear from experienced users such as yourself. There is no use in being compatible with the autotools, the point is about stealing good ideas. The first thing that comes to mind, and that we already discussed at large, is the ability to specify at install time where data, documentation, manual pages and so on have to go. Another thing that I consider a good design (when expunged of the useless stuff in it) is the config.h file. It is a file that is generated at configure time and which contains macros you may need inside your code: package version, installation directories, enabled features, etc. What some Python projects using autoconf are doing, and I’d like to see in BUILDS, is the generation of a config.py file at installation time. It would be useful to specify install-time stuff like installation directories, optional features and installation parameters. For example it could contain: version = 1.2.3 builddate = Tue 7 Oct 2008, 23:12 +0100 templates_dir = /usr/share/foobar/templates has_libamazing = True # _foobar.so is built with libamazing support page_footer = Powered by foobar For Python-only
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote: Le mardi 07 octobre 2008 à 16:51 -0400, Phillip J. Eby a écrit : If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way. I'm not sure I follow you. What I'm proposing is that: 1. A BUILDS manifest for a project should list the files included in the project by what kind of file they are, as well as by inter-file relationships (e.g., these 3 files are part of Python package 'foo.bar' within this project, and they are static data used at runtime but not read or written). 2. BUILDS-compatible install tools will be free to install those files wherever they like (or create .debs, .rpms, etc. that do), and use symlinks so that the installed project can find them in the places that the developer wanted to find them. Is this what you're asking for? Because it's what I've been trying to say we should do. What I am afraid of is that, by adding just another layer, you will introduce new problems while fixing existing ones. Currently we already have a hard time maintaining a symlink farm, and adding a second symlink farm on top of it is not going to make things more understandable (currently, package maintainers already have a hard time understanding how to deal with Python modules) nor more reliable. I guess I'm confused a bit here, since the idea is not to have maintainers need to understand anything but how to be able to tell upstream, hey, you didn't configure file X correctly. At least, that's what I'd hope would be all that's needed. :-) The first kind of issues I can think of is that dpkg handles symlinks to directories very badly, so youâre likely to run into issues that have to be dealt with by hand by maintainers. I havenât thought about it much, but I have the feeling that other things will break. Actually, until I saw the Twisted plugins directory use case, I hadn't actually seen any use cases for symlinks to directories that couldn't be handled by using a directory of symlinks instead. Are there any other Python packages you know of that would need a symlink to a directory? In the end, if you are designing a new packaging specification and a new tool, The BUILDS manifest is neither packaging nor a tool; it's just information about a bunch of files. ;-) We don't want to design *a* tool, we want to make it possible for tools to be written. Currently, there are a whole bunch of installation and maintenance tools out there built on top of setuptools, but they aren't necessarily very compatible with each other, and building tools that approach things very differently from setuptools is hard. The idea of a specification is to allow installation tools to be built that apply different policies on different platforms, not to build The One True Tool[tm]. In other words, the goal isn't to do the right thing, but to allow various useful things to be done. For example, if Debian needs metadata that's not in the core BUILDS metadata, it should be possible within BUILDS to include it, so that it can be contributed upstream. Likewise, it should be possible for a BUILDS install tool to use a third-party supplement to get that data, so that when you're initially porting a project, you could write a file with the additional data while doing the initial port, then submit that file upstream. If there is a need for Debian-specific metadata, it will go in the debian/ directory. It looks interesting to integrate it upstream only if it is relevant for other distributors. If I have to explicitly mark my locale files for Debian to be happy, then that *is* Debian-specific metadata from my perspective as a cross-platform developer. That is, I didn't say anything about metadata that was unique to Debian, just metadata needed by a Debian BUILDS installer to be happy, as opposed to a generic BUILDS installer that doesn't care about the FHS. (For example, an Enthought installer for Windows that's also installing its own Python interpreter might want to consume BUILDS but it sure as heck wouldn't be following FHS standards; it'd have its own.) There is no use in being compatible with the autotools, the point is about stealing good ideas. The first thing that comes to mind, and that we already discussed at large, is the ability to specify at install time where data, documentation, manual pages and so on have to go. The idea of the BUILDS install locations (IL) manifest is just that it lists the contents of the project distribution, and describes the logical locations that things go in. So, part of the BUILDS spec would be what categories exist for the above. It would then be the responsibility of people building BUILDS-consuming install tools to define the actual locations. I say it that way mainly because I'm not
Re: [Distutils] Twisted packaging in Debian
On Wed, 08 Oct 2008 00:03:16 +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Le mercredi 08 octobre 2008 à 08:44 +1100, Ben Finney a écrit : Josselin Mouette [EMAIL PROTECTED] writes: About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions. You've referred to this a few times now without explaining exactly what you expect the developer to do. So, just in case: I presume you mean that, in the case where a program's plugins are not compatible with plugins for previous versions of the same program, the developer should always seek (and attempt to install) plugins for version N in a differently-named directory from plugins for version M and version P. That is, that a directory expected to contain plugins for version N should contain *only* plugins for version N, and never for other versions. Yes. Sorry for omitting this detailed explanation. What if we consider the scenario without two different versions? The plugin duplication issue happens if you have a site-wide installation of Twisted X.Y and an extra (eg, per-user) copy of Twisted X.Y, the same version, explicitly in PYTHONPATH. Versioning the plugin directory wouldn't help here, because both copies would appear to be the right version for whichever instance of the install gets used. It might be possible to avoid duplication by inspecting the names carefully and avoiding loading anything that looks like it was already loaded. But I can still think of problem cases - the site-wide installation has plugins X, Y, and Z and my personal installation has plugins A, B, and C, and I use my personal installation with the express intent of not having X, Y, and Z made available. If the site-wide plugins directory is considered, then I get all 6 - not what I wanted. Jean-Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
Phillip J. Eby wrote: Sorry, what is pkg-config? http://pkg-config.freedesktop.org/wiki/ David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals
Tarek Ziadé wrote: On Sun, Oct 5, 2008 at 12:04 PM, zooko [EMAIL PROTECTED] wrote: 5/ ideally, they should be one and only one version of a given package in an OS-based installation -1 -- This is the strong preference of the folks who package software for OSes -- Debian, Fedora, etc. -- but it is not necessarily the choice of the users who use their OSes. It is best for the Python packaging standards to be agnostic towards this, or at least to support both this desideratum and its opposite. It is not just a strong preference of the Linux guys, it is proper software engineering. It is fine for developers to have several versions installed at the same time, but we should really discourage developers to depend on the possibility to deploy several versions of the same package, at least in the site-packages owned by the system. The problem is not just for linux guys: by enabling everyone to deploy several versions, you encourage people not to care about API compatibility, and then quikcly, in the dependencies, you will have depends on foo = 1.2 (foo 1.3), which quickly means it will fail because a package A depends on B and C, and B depends on foo 1.2 and C 1.3. By encouraging multiple versions, you are encouraging this kind of failures all the time. If people want to try several versions, there is virtual env and co (or just installing several python interpreters). It should be a *developer only* convenience as much as possible. We can have a system to control imported versions, but without support from python interpreter, it will be unreliable, and keeps breaking as it does now. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
