Re: [Python-Dev] PEP 376 - Open questions
On Fri, 10 Jul 2009 01:56:36 -0700, Paul Moore p.f.mo...@gmail.com wrote: One thing that did occur to me based on this - do we want the format to support designation of files (such as config files) that *shouldn't* be uninstalled along with everything else? Or are we happy with not mentioning the file in RECORD at all as the means of supporting that use case? No. In my view, the uninstall feature is a relatively minor issue - people who want uninstall facilities generally use system packages (or complain that easy_install doesn't support uninstallation :-)). They aren't using setup.py install. (Of course, my experience is minimal in this regard, so maybe I'm wrong). Remember that the PEP says Distutils will provide a **very basic** uninstall function (my emphasis). Unless I'm misunderstanding, no-one is intending to use the new distutils uninstall feature in their own package manager (bdist_wininst, RPM, apt, ...) Perhaps distutils can also provide an `uninstall` hook that application can use to do some pre-processing? import distutils.installer distutils.installer.register_uninstall_hoook(my_uninstall_hook) I imagine that packages like pywin32 may have to register DLLs (install time) and unregister them (uninstall time). Just a wild idea .. I understand that this may not be related to the current PEP. :-) -srid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Thu, 09 Jul 2009 01:22:19 -0700, Tarek Ziadé ziade.ta...@gmail.com wrote: On Thu, Jul 9, 2009 at 3:42 AM, Sridhar Ratnakumarsridh...@activestate.com wrote: Other than easy_install/pip, there is also PyPM which is being developed at ActiveState. PyPM is the Python package manager much like what ppm is for ActivePerl. Great ! besides the RECORD file, anything remark on the PEP from a PyPM point of view ? I'll shortly send an email I composed regarding this. Once Paul has finished to work on the PEP 302 part of the prototype, maybe we could try it out on PyPM to see how it fits ? I took a look at PEP 302 and Paul's new implemention in pkgutil2.py http://bitbucket.org/tarek/pep376/src/tip/pkgutil2.py And yes, I'd happy to try the new pkgutil API on PyPM .. keeping interoperability with pip/easy_install installed packages in mind. -srid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/9 Eric Smith e...@trueblade.com: P.J. Eby wrote: ISTM that the problem that it solves is uninstall in the absence of the original installer. Or uninstall where the installer is setup.py install, actually. I think we need to move away from setup.py install. It's the antithesis of static metadata. setup.py needs to go away. If we have a static metadata file called, say setup.info, then I'd like to see: bdist_rpm setup.info produce a .rpm bdist_msi setup.info produce a .msi bdist_deb setup.info produce a .deb bdist_egg setup.info produce a .egg and so on. There might be a library (and I call dibs on the name distlib :) that provides support routines to parse setup.info, but there's no framework involved. And no need for a plugin system. +1. Now who's going to design write it? It's the conflation of build a package with install a package that distutils and setuptools has given us (via setup.py) that leads to us never making progress. Let's just focus on describe a package and let the packagers handle build a package and let the installers handle install and uninstall a package. Then the packagers (and their associated installers) can evolve independently of each other. There still needs to be some sort of package build system. I do *not* want to see a situation where users can't easily build their own packages. Not all systems have all Python packages packaged up by official system packagers... Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Thu, Jul 9, 2009 at 3:07 AM, P.J. Ebyp...@telecommunity.com wrote: At 09:38 AM 7/9/2009 +0900, Stephen J. Turnbull wrote: Eric Smith writes: But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. ISTM that the problem that it solves is uninstall in the absence of the original installer. Or uninstall where the installer is setup.py install, actually. Right, and for instance, Fedora packagers are using it to re-package eggs http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools with the --root option, so they will get a RECORD file and will be able to set a value for the INSTALLER one, ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Thu, Jul 9, 2009 at 3:42 AM, Sridhar Ratnakumarsridh...@activestate.com wrote: Other than easy_install/pip, there is also PyPM which is being developed at ActiveState. PyPM is the Python package manager much like what ppm is for ActivePerl. Great ! besides the RECORD file, anything remark on the PEP from a PyPM point of view ? Once Paul has finished to work on the PEP 302 part of the prototype, maybe we could try it out on PyPM to see how it fits ? Eventually, I'd like PEP 376 to support system packagers too. So for example, if you did apt-get install python-pyqt4, then running pip install python-pyqt4 should return without installing anything .. as RECORD will be part of the .deb previously installed. As for generating the RECORD file, I vote for generating it during install time (w/ absolute paths). That should be possible for all python setup.py install-based systems e.g. stating that if python-pyqt4 is already installed and if the INSTALLER file contains another name than pip, pip should not try to do anything with it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/9 Tarek Ziadé ziade.ta...@gmail.com: On Thu, Jul 9, 2009 at 3:42 AM, Sridhar Ratnakumarsridh...@activestate.com wrote: Other than easy_install/pip, there is also PyPM which is being developed at ActiveState. PyPM is the Python package manager much like what ppm is for ActivePerl. Great ! besides the RECORD file, anything remark on the PEP from a PyPM point of view ? Once Paul has finished to work on the PEP 302 part of the prototype, maybe we could try it out on PyPM to see how it fits ? Note: the pkgutil2 implementation I pushed to Tarek's bitbucket repository a day or so ago includes PEP 302 support (using a few new finder methods, as proposed). It's ready to go now. It's as compliant with the PEP as possible given the outstanding questions (I've implemented some things on the basis of what I thought was consensus here, even though the PEP has not changed yet to reflect that - using metadata rather than egginfo in method names, and removing the local= parameter from some of the methods, being the 2 I can recall immediately). I'd like to add a test case using a non-standard PEP 302 importer, but that's a small detail. There are bound to be holes in the implementation, especially around the areas of the open questions that started this thread. I'd appreciate feedback, but I'm not expecting to make code changes until the PEP is changed - it's too much like trying to hit a moving target at the moment! Paul. PS I've added distutils-sig back into the cc list. Please keep PEP 376 in the subject of any subsequent threads on that list if you want my input, as I don't follow all the traffic on there. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Eric Smith wrote: I really don't get this use case of multiple installers trying to install the same package. There's just no way that running yum install twisted and apt-get install twisted and pip install twisted are going to coexist with each other. The best they can do is say a file I'm trying to install already exists. Why try for anything else? If that use case doesn't click for you, just focus on the distutils.uninstall use case. At the moment, there is no automatic method for uninstalling a distutils installed package. This PEP is mainly about fixing that situation. However, if we're going to go to all the effort of having an index of distutils-installed packages and the metadata needed to uninstall them, it makes sense to have an official API for querying that datastore. The fact that that API then provides support for Did someone else install this already? is really a sideshow. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/9 Paul Moore p.f.mo...@gmail.com: I'd like to add a test case using a non-standard PEP 302 importer, but that's a small detail. I've just pushed a PEP 302 importer test case. The level of boilerplate needed is a bit painful, but it's fine as a prototype. When we get consensus on some of the remaining details, I'll tidy up the implementation and streamline the importer requirements at that time. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
P.J. Eby wrote: At 11:20 PM 7/8/2009 -0400, Eric Smith wrote: P.J. Eby wrote: ISTM that the problem that it solves is uninstall in the absence of the original installer. Or uninstall where the installer is setup.py install, actually. I think we need to move away from setup.py install. It's the antithesis of static metadata. Please note that that's entirely out of scope for the PEP at hand. Yes. I'm just trying to point out that the information in the PEP is applicable only to the set of installers that want to work together in some integrated way. It doesn't apply to all installers, and I think that's the bigger problem. But I grant you that the rest of my mail is veering off-topic, and there's a more appropriate thread now on distutils-sig. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/9 Eric Smith e...@trueblade.com: Yes. I'm just trying to point out that the information in the PEP is applicable only to the set of installers that want to work together in some integrated way. It doesn't apply to all installers, and I think that's the bigger problem. Interoperability standards are only as useful as the number of applications that conform to them. The set of installers that will *not* provide PEP 376 compatible metadata is an important measure of the usefulness of the PEP. Some metadata (the INSTALLER file and the proposed PREFIXES file) must be created by the installer for the PEP 376 APIs to work. Simple cases (relocatable packages with all package data maintained inside the package directory) will work fine even without installer co-operation, as long as the installer copies the whole structure unchanged. But: - some cases are not simple, and it's not clear to me how useful nearly always accurate data will be - I understand that some packagers deliberately pick apart Python distributions to conform to OS filesystem layout standards Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Paul Moore wrote: - some cases are not simple, and it's not clear to me how useful nearly always accurate data will be Since the accuracy can always be checked against the filesystem, I think the metadata is useful even if not 100% reliable. Applications that need the extra assurance (such as the distutils uninstaller itself) can check the filesytem and complain when it differs from what it is claimed in the package metadata. One thing that did occur to me based on this - do we want the format to support designation of files (such as config files) that *shouldn't* be uninstalled along with everything else? Or are we happy with not mentioning the file in RECORD at all as the means of supporting that use case? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Wed, Jul 8, 2009 at 7:09 AM, P.J. Ebyp...@telecommunity.com wrote: At 11:30 PM 7/7/2009 +0200, Tarek Ziadé wrote: On Tue, Jul 7, 2009 at 10:31 PM, P.J. Ebyp...@telecommunity.com wrote: At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote: When I started to work on this I didn't realize the gigantic amount of work and coordination it requires No one expects the package inquisition. ;-) Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ? It's a Monty Python reference, i.e. no one expects the Spanish Inquisition. In the Spanish Inquisition sketch, a character is asked several questions he can't answer, and then complains that he wasn't expecting some kind of Spanish inquisition, at which point three scarlet-robed cardinals burst into the room and exclaim, NO ONE expects the Spanish Inquisition! So I punned on that in order to imply that nobody who takes on the job of packaging expects to be asked the kind of detailed, unanswerable questions that come with the territory of packaging, or to be subjected to torture by third parties merely for trying to do something useful. And, by further implication, I was thus expressing camaraderie with you on your initiation into the wonderful world of trying to satisfy everyone in Pythonland, and our shared experience in the area of if I'd really known what I was getting into, I probably wouldn't have. ;-) Hehe very nice, thanks for the explanation :) (thanks to the others too) -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Tarek Ziadé wrote: On Wed, Jul 8, 2009 at 7:09 AM, P.J. Ebyp...@telecommunity.com wrote: So I punned on that in order to imply that nobody who takes on the job of packaging expects to be asked the kind of detailed, unanswerable questions that come with the territory of packaging, or to be subjected to torture by third parties merely for trying to do something useful. And, by further implication, I was thus expressing camaraderie with you on your initiation into the wonderful world of trying to satisfy everyone in Pythonland, and our shared experience in the area of if I'd really known what I was getting into, I probably wouldn't have. ;-) Hehe very nice, thanks for the explanation :) (thanks to the others too) I think is exchange actually a good reminder that the bar for acceptance of PEP 376 shouldn't be Addresses every possible packaging issue we can come up with. That would be setting expectations far too high because packaging cross-platform is so messy. A more appropriate bar is probably: a) Better that what we have now; and b) Provides possible avenues for even more improvement in the future For example, I quite like the concept behind the various ideas for location or prefix definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above. Once some kind prefix definition mechanism is in place in the metadata, it becomes possible to leverage it to categorise files for the benefit of system packagers while still allowing simple internal-use-only packages to just bundle everything together. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Nick Coghlan ncoghlan at gmail.com writes: I think is exchange actually a good reminder that the bar for acceptance of PEP 376 shouldn't be Addresses every possible packaging issue we can come up with. That would be setting expectations far too high because packaging cross-platform is so messy. +1 For example, I quite like the concept behind the various ideas for location or prefix definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above. I don't understand the point. Since RECORD is specific to a particular machine and installation (it is generated at install time), what would you gain by adding a fake genericity to its entries? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Antoine Pitrou wrote: Nick Coghlan ncoghlan at gmail.com writes: For example, I quite like the concept behind the various ideas for location or prefix definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above. I don't understand the point. Since RECORD is specific to a particular machine and installation (it is generated at install time), what would you gain by adding a fake genericity to its entries? My understanding is that RECORD isn't generated at install time for the bdist_* commands -it's generated at the time the binary distribution is created. Hence the problems raised with the idea of having absolute paths in RECORD - if they were written directly into RECORD they would reflect where the files were when the distribution was created, not where they end up when it is installed. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/8 Nick Coghlan ncogh...@gmail.com: Antoine Pitrou wrote: Nick Coghlan ncoghlan at gmail.com writes: For example, I quite like the concept behind the various ideas for location or prefix definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above. I don't understand the point. Since RECORD is specific to a particular machine and installation (it is generated at install time), what would you gain by adding a fake genericity to its entries? My understanding is that RECORD isn't generated at install time for the bdist_* commands -it's generated at the time the binary distribution is created. Hence the problems raised with the idea of having absolute paths in RECORD - if they were written directly into RECORD they would reflect where the files were when the distribution was created, not where they end up when it is installed. This is a key open question, IMHO. No-one has done any work on updating the core bdist_xxx commands to the new structure, and the PEP is silent on the matter, so it's still all speculation. The current best guess assumption is that, given that bdist_xxx uses a setup.py install --root=xxx incantation to generate the structure to package, the RECORD can be picked up from this. There are a number of issues with this assumption, though - it doesn't handle absolute locations well (there are existing bugs), paths may be affected by the environment at the time the installer is built, and it requires RECORD to be relocatable. An alternative solution would be for the bdist_xxx commands to ignore the RECORD file generated in the temp area, and build its own on the target machine when the installer is run. This is conceptuially far cleaner, it's in line with the implicit assumptions in the PEP, and it avoids a lot of the problems with path formats (the filenames in RECORD could be absolute local pathnames, and that's the end of the story). But - and it's a big but - this dumps RECORD file handling on the bdist_xxx command, and they all need to implement the same code, resulting in a lot of duplication and opportunity for inconsistency. I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc. There's also Phillip commenting from the setuptools POV and MAL who has mentioned his build process. Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable? Subsequent question - if it's to be relocatable, how will that work? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Paul Moore p.f.moore at gmail.com writes: I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc. I think any bdist_xxx command which targets externally handled package formats (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or dpkg, you will uninstall with the same tool, won't you? Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable? This is quite an uninformed opinion, but a relocatable RECORD looks like a can of worms to me. As for creating RECORD at install time, I don't understand the argument about duplicate code. Distutils can provide useful primitives for generating RECORD, so that interested subcommands don't have a lot of additional work to do. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/8 Antoine Pitrou solip...@pitrou.net: Paul Moore p.f.moore at gmail.com writes: I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc. I think any bdist_xxx command which targets externally handled package formats (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or dpkg, you will uninstall with the same tool, won't you? I thought the purpose of RECORD was (also) so that *other* installs could check who owned an existing file, before potentially overwriting it. That seems like it's the reasoning behind the uses() function. But the PEP is very light on practical applications, so it's hard to tell what the methods are intended to be used for. Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable? This is quite an uninformed opinion, but a relocatable RECORD looks like a can of worms to me. As for creating RECORD at install time, I don't understand the argument about duplicate code. Distutils can provide useful primitives for generating RECORD, so that interested subcommands don't have a lot of additional work to do. The bdist_wininst code is in C, and the bdist_msi code is driven by the MSI data format. I don't know what scripting options exist for .deb and .rpm files. Different languages, so no code sharing seems possible. Maybe I'm wrong, and something could be done along the lines of a pkgutil.generate_record_file() function which could be run from a post-install script written in Python, but I don't know how plausible or robust that would be... Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Antoine Pitrou wrote: Paul Moore p.f.moore at gmail.com writes: I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc. I think any bdist_xxx command which targets externally handled package formats (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or dpkg, you will uninstall with the same tool, won't you? I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all. For bdist_rpm (which I'm contemplating taking over, but no promises), the only thing that needs to be done with the metadata is be able to build a .spec file. Once the .spec file is generated, rpmbuild (or equivalent) will produce the .rpm files. I image that .deb's are similar, but I'm no expert. Which brings up the static metadata argument I was making at PyCon. I want there to be enough static metadata that the new bdist_rpm can read so that it produces a well-behaved .spec file. I need to know the intent of some of the files (which are documentation, which are config files, which are scripts, which are libraries, etc.) and not much else. I envision the new bdist_rpm as a standalone tool, not a distutils extension. Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable? This is quite an uninformed opinion, but a relocatable RECORD looks like a can of worms to me. Agreed. As for creating RECORD at install time, I don't understand the argument about duplicate code. Distutils can provide useful primitives for generating RECORD, so that interested subcommands don't have a lot of additional work to do. I envision a distlib that contains new support code that's not part of the current distutils morass. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/8 Eric Smith e...@trueblade.com: [Any reason you didn't respond to the list? I'd like to continue the discussion there. But I'm currently replying just to you.] No, I just clicked the wrong button. Sorry. Redirecting back to the list. Paul Moore wrote: 2009/7/8 Eric Smith e...@trueblade.com: I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all. I thought RECORD was to ensure that you can install with RPM, and *not* have any other random uninstaller remove your files (because you populate RECORD, and the other uninstaller uses pkgutil.get_file_users(path) to find out that you care about the file, and hence avoid deleting it). I really don't think this is a use case worth supporting. A single file managed by multiple installers is a mess waiting to happen. RPM won't let you do it. About the only use case I can think of is namespace packages, and that can be handled by factoring out the __init__.py file into it's own install package. Personally, I think you're right. But I have essentially zero experience with the non-Windows use cases, so I don't feel qualified to judge. It seems to me that a lot of the fancy detecting what other people do stuff is motivated by setuptools. I'm not going to comment on that, other than to say that it may be that setuptools is generating more problems than it's solving... Basically, other install methods can't read your (RPM) database, and you can't read their database. RECORD is a simple common format, providing just enough information for everyone to communicate the message I manage this file. The RPM uninstaller won't look at the RECORD file. I think the best we can say is that _if_ an installer wants to share state about what's installed with other installers, this is how it's done. OK. I'd be stunned if bdist_wininst or bdist_msi ends up checking RECORD. So that's 3 not interested votes. Who's going to use the APIs based around the RECORD file? Anyone? Seriously - who is actually going to USE any of the APIs being proposed in PEP 376? If no-one's interested, we should stop right now. Do any of the people from the summit read this list? If not, can anyone contact them and get some feedback? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 01:13 PM 7/8/2009 +0100, Paul Moore wrote: An alternative solution would be for the bdist_xxx commands to ignore the RECORD file generated in the temp area, and build its own on the target machine when the installer is run. This is conceptuially far cleaner, it's in line with the implicit assumptions in the PEP, and it avoids a lot of the problems with path formats (the filenames in RECORD could be absolute local pathnames, and that's the end of the story). But - and it's a big but - this dumps RECORD file handling on the bdist_xxx command, and they all need to implement the same code, resulting in a lot of duplication and opportunity for inconsistency. The PREFIXES approach I suggested doesn't have these problems; the bdist installer only needs to write a PREFIXES file, and doesn't need to touch RECORD (which would be generated by the install --root). In fact, since PREFIXES is strictly advisory, it can be generated by the install --root and left alone at installation time. And, if it's generated without the --root prefix included, RPMs would have a correct PREFIXES file without further manipulation. Plus, the bdist_wininst .exe installer can easily write a PREFIXES file, since it has to know the prefixes in order to do an installation in the first place. (We still need to hash out an initial list of prefixes, of course.) At 08:59 AM 7/8/2009 -0400, Eric Smith wrote: I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all. This is a misunderstanding. The purpose is to let an *installer* (like easy_install) know that these files belong to some other installer, and not allow overwriting them. That's why there's also an INSTALLER file. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 03:57 PM 7/8/2009 +0100, Paul Moore wrote: Who's going to use the APIs based around the RECORD file? Anyone? The distutils uninstall facility, for starters. easy_install and pip also will, eventually. In all three cases, the use will be in order to avoid overwriting files belonging to a system package manager, and in order to uninstall/upgrade their own files. (I can't personally commit that pip will need this; but it's currently the only installer besides the distutils that installs things in this format, and would therefore need this to prevent overwriting things.) Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES appropriately for both development and actual installations. Seriously - who is actually going to USE any of the APIs being proposed in PEP 376? If no-one's interested, we should stop right now. Do any of the people from the summit read this list? If not, can anyone contact them and get some feedback? Package management people aren't likely to be doing anything with this stuff directly; if they need a list of installed files they're probably using --root or --record at package build time, not needing to sniff stuff and installation time, and they already have installation facilities. Their only real need in relation to this will be to set an option to generate an INSTALLER file that says you can't uninstall their packages manually, thereby preventing the package from being uninstalled by the distutils utility, or any other well-behaved PEP 376-based package management tool. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Wed, Jul 8, 2009 at 5:11 PM, P.J. Ebyp...@telecommunity.com wrote: At 01:13 PM 7/8/2009 +0100, Paul Moore wrote: An alternative solution would be for the bdist_xxx commands to ignore the RECORD file generated in the temp area, and build its own on the target machine when the installer is run. This is conceptuially far cleaner, it's in line with the implicit assumptions in the PEP, and it avoids a lot of the problems with path formats (the filenames in RECORD could be absolute local pathnames, and that's the end of the story). But - and it's a big but - this dumps RECORD file handling on the bdist_xxx command, and they all need to implement the same code, resulting in a lot of duplication and opportunity for inconsistency. The PREFIXES approach I suggested doesn't have these problems; the bdist installer only needs to write a PREFIXES file, and doesn't need to touch RECORD (which would be generated by the install --root). In fact, since PREFIXES is strictly advisory, it can be generated by the install --root and left alone at installation time. And, if it's generated without the --root prefix included, RPMs would have a correct PREFIXES file without further manipulation. Plus, the bdist_wininst .exe installer can easily write a PREFIXES file, since it has to know the prefixes in order to do an installation in the first place. (We still need to hash out an initial list of prefixes, of course.) Instead of building this list of prefixes from scratch, let's use the ones that are *actually* used when the install command is called, have a look at the list I've suggested : http://mail.python.org/pipermail/python-dev/2009-July/090514.html And let's drop other absolute paths, ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/8 P.J. Eby p...@telecommunity.com: At 03:57 PM 7/8/2009 +0100, Paul Moore wrote: Who's going to use the APIs based around the RECORD file? Anyone? The distutils uninstall facility, for starters. easy_install and pip also will, eventually. Is pip getting this via setuptools, or will it be independently changing its code? (Who's the pip contact who could confirm this directly?) In all three cases, the use will be in order to avoid overwriting files belonging to a system package manager, and in order to uninstall/upgrade their own files. To uninstall/upgrade their own files counts as an installer-specific detail in the sense that Eric stated it. To avoid overwriting files belonging to a system package manager: That sounds like we're looking at a use case where people use a mixture of tools to install packages into the system Python. Personally, I never do this (I use bdist_wininst exclusively). The cases I can imagine are: - no OS-specific package available, and the user can't build his own, so he uses setup.py install. - user using easy_install as well as the OS package In case (1), I'd imagine the user is careful enough (given that he's doing everything manually) to avoid installing something that's already present. Case (2) is setuptools-related (see below for setuptools-related comments). (I can't personally commit that pip will need this; but it's currently the only installer besides the distutils that installs things in this format, and would therefore need this to prevent overwriting things.) Not the only one - MAL mentioned his (internal use?) packaging process. Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES appropriately for both development and actual installations. If no-one else is using them, that's an internal setuptools detail again. Or it's for interoperability between setuptools and distutils (and other packagers don't benefit unless they are using PEP 376 APIs). Seriously - who is actually going to USE any of the APIs being proposed in PEP 376? If no-one's interested, we should stop right now. Do any of the people from the summit read this list? If not, can anyone contact them and get some feedback? Package management people aren't likely to be doing anything with this stuff directly; if they need a list of installed files they're probably using --root or --record at package build time, not needing to sniff stuff and installation time, and they already have installation facilities. Their only real need in relation to this will be to set an option to generate an INSTALLER file that says you can't uninstall their packages manually, thereby preventing the package from being uninstalled by the distutils utility, or any other well-behaved PEP 376-based package management tool. So if I follow this, there's a use for this in making sure the new distutils uninstall option doesn't break something it shouldn't, but as the uninstall uses RECORD, it won't be (accidentally) usable on distributions that didn't follow PEP 376 and create a RECORD file. There's also a use for this in terms of setuptools/easy_install interoperability. I'm now going to take of my PEP supporter hat, and put on my End user hat. Please understand that the following is *not* unbiased, whereas the rest of what I've said *is* intended to be reasonably impartial. If the only driver for this PEP is setuptools, then I'm -1 on it. Unless someone working on a packaging tool *other* than setuptools (or setuptools-derived projects) speaks up and says I have code of my own which uses distutils, and I would benefit as follows from PEP 376 and I will be changing my code to conform to PEP 376 I think the PEP should be rejected, and the energy put into some other PEP which will benefit the wider (non-setuptools) community. OK, end of the opinion bit. I'll go back to trying to keep things impartial in future mails. Paul. PS Note - I like the idea of the PEP in theory. It's the practical benefits that I'm trying to establish. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
P.J. Eby wrote: At 08:59 AM 7/8/2009 -0400, Eric Smith wrote: I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all. This is a misunderstanding. The purpose is to let an *installer* (like easy_install) know that these files belong to some other installer, and not allow overwriting them. That's why there's also an INSTALLER file. I really don't see this use case. Supporting multiple installers for the same file (or even just trying to prevent two installers from writing the same file)? Wouldn't you be better off just saying an installer can't overwrite any existing file? Likewise, I don't see a use case for installing with one installer and uninstalling with another. Put those two together, and the mechanism that an installer uses to record what files it installed is a private implementation detail. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 05:22 PM 7/8/2009 +0100, Paul Moore wrote: 2009/7/8 P.J. Eby p...@telecommunity.com: At 03:57 PM 7/8/2009 +0100, Paul Moore wrote: Who's going to use the APIs based around the RECORD file? Anyone? The distutils uninstall facility, for starters. easy_install and pip also will, eventually. Is pip getting this via setuptools, or will it be independently changing its code? (Who's the pip contact who could confirm this directly?) It would likely be changing its code directly, with respect to supporting overwrite-avoidance and uninstall-before-upgrade scenarios. (Generation support would likely come from setuptools, dependent on who implemented first.) Ian Bicking is pip's author. In case (1), I'd imagine the user is careful enough (given that he's doing everything manually) to avoid installing something that's already present. Obviously, you've never tried doing this yourself. ;-) My guess would actually be that in Unixy environments these days, people either use virtualenv or buildouts to manage such things. Buildout handles uninstall-before-update (for eggs, anyway, I'm not sure about other things) and virtualenv makes it easy to create throwaway Python installs. So if I follow this, there's a use for this in making sure the new distutils uninstall option doesn't break something it shouldn't, but as the uninstall uses RECORD, it won't be (accidentally) usable on distributions that didn't follow PEP 376 and create a RECORD file. That's an interesting point. But for non-distutils tools, it'd be nice to be able to find out whether another package manager is administering some particular file. For that matter, the distutils themselves should (ultimately) support uninstall-before-upgrade and avoid overwriting another package manager's files. If the only driver for this PEP is setuptools, then I'm -1 on it. If it were being driven by setuptools, I'd have just implemented it myself and presented it as a fait accompli. I can't speak to Tarek's motives, but I assume that, as stated in the PEP, the primary driver is supporting the distutils being able to uninstall things, and secondarily to allow other tools to be built on top of the API. a packaging tool *other* than setuptools (or setuptools-derived projects) Is there really such a thing? ;-) AFAIK, every published tool for managing Python projects is either distutils-based or setuptools-based. (Things like scons and pymake and various other project build tools don't seem to fall under packaging tool in this sense.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 01:58 PM 7/8/2009 -0400, Eric Smith wrote: I really don't see this use case. Supporting multiple installers for the same file (or even just trying to prevent two installers from writing the same file)? Wouldn't you be better off just saying an installer can't overwrite any existing file? Likewise, I don't see a use case for installing with one installer and uninstalling with another. Put those two together, and the mechanism that an installer uses to record what files it installed is a private implementation detail. You'll have to ask Tarek about that. As I've said in another email, if I were doing this for setuptools I'd just go and do it; it's already on my backlog of things to do. The point of the PEP is to provide an uninstall facility for the distutils, and to expose those records via an API so that other tools can be built on top of it. Whether anything other than distutils-based tools will actually use the format directly is entirely irrelevant to the PEP's merits, since bdist_* tools and platform-specific distutils installers may need to read or write the format. Hence, a PEP is appropriate to describe a format and API for the distutils family of tools to operate on. The specific proposed format is based on an existing de facto standard for installed package metadata, so that the many existing packaging tools based on that other format and existing APIs can easily read or write this data, and so that the API can also be used as a way to detect the presence of already-installed distributions of software -- even ones installed other ways. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Hehe very nice, thanks for the explanation :) (thanks to the others too) I think is exchange actually a good reminder that the bar for acceptance of PEP 376 shouldn't be Addresses every possible packaging issue we can come up with. That would be setting expectations far too high because packaging cross-platform is so messy. OTOH, it *is* expected that the PEP explicitly lists aspects that it didn't resolve, so that any reader can find out whether something was discussed and rejected, or not discussed. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/8 P.J. Eby p...@telecommunity.com: If it were being driven by setuptools, I'd have just implemented it myself and presented it as a fait accompli. I can't speak to Tarek's motives, but I assume that, as stated in the PEP, the primary driver is supporting the distutils being able to uninstall things, and secondarily to allow other tools to be built on top of the API. My understanding is that all of the various distutils PEPs were driven by the packaging summit ay PyCon. The struggle here seems to be to find *anyone* from that summit who will now comment on the discussion :-( a packaging tool *other* than setuptools (or setuptools-derived projects) Is there really such a thing? ;-) I wonder... :-) AFAIK, every published tool for managing Python projects is either distutils-based or setuptools-based. (Things like scons and pymake and various other project build tools don't seem to fall under packaging tool in this sense.) The big driver seems to be things that take distutils packages and build Linux packages (debs, rpms, ...) from them. Unfortunately, I know nothing about these tools, and no-one seems to be speaking up on the subject. Maybe this was all thrashed out on the distutils SIG, and consensus reached there (I only skim that list). If so, maybe this list is the wrong one to be discussing changes to the PEP on. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Paul Moore wrote: 2009/7/8 P.J. Eby p...@telecommunity.com: If it were being driven by setuptools, I'd have just implemented it myself and presented it as a fait accompli. I can't speak to Tarek's motives, but I assume that, as stated in the PEP, the primary driver is supporting the distutils being able to uninstall things, and secondarily to allow other tools to be built on top of the API. My understanding is that all of the various distutils PEPs were driven by the packaging summit ay PyCon. The struggle here seems to be to find *anyone* from that summit who will now comment on the discussion :-( I was there, and I've been commenting! There might have been more discussion after the language summit and the one open space event I went to. But the focus as I recall was static metadata and version specification. When I originally brought up static metadata at the summit, I meant metadata describing the sources in the distribution, so that we can get rid of setup.py's. From that metadata, I want to be able to generate .debs, .rpms, .eggs, etc. But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. I'd like to get back to the metadata that describes the source files. That's where the real value lies, in my opinion. I'll try and work on a post to distutils-sig explaining my thinking. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/8 Eric Smith e...@trueblade.com: I was there, and I've been commenting! Sorry, I hadn't realised that. Thanks for the correction. There might have been more discussion after the language summit and the one open space event I went to. But the focus as I recall was static metadata and version specification. When I originally brought up static metadata at the summit, I meant metadata describing the sources in the distribution, so that we can get rid of setup.py's. From that metadata, I want to be able to generate .debs, .rpms, .eggs, etc. But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. I'd like to get back to the metadata that describes the source files. That's where the real value lies, in my opinion. I'll try and work on a post to distutils-sig explaining my thinking. OK, that helps a lot. I see how your postings fit into things a little better now. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Thu, Jul 9, 2009 at 7:07 AM, Eric Smithe...@trueblade.com wrote: Paul Moore wrote: 2009/7/8 P.J. Eby p...@telecommunity.com: If it were being driven by setuptools, I'd have just implemented it myself and presented it as a fait accompli. I can't speak to Tarek's motives, but I assume that, as stated in the PEP, the primary driver is supporting the distutils being able to uninstall things, and secondarily to allow other tools to be built on top of the API. My understanding is that all of the various distutils PEPs were driven by the packaging summit ay PyCon. The struggle here seems to be to find *anyone* from that summit who will now comment on the discussion :-( I was there, and I've been commenting! There might have been more discussion after the language summit and the one open space event I went to. But the focus as I recall was static metadata and version specification. When I originally brought up static metadata at the summit, I meant metadata describing the sources in the distribution, so that we can get rid of setup.py's. From that metadata, I want to be able to generate .debs, .rpms, .eggs, etc. I agree wholeheartedly. Getting rid of setup.py for most packages should be a goal IMHO. Most packages don't need anything fancy, and static metadata are so much easier to use compared to setup.py/distutils for 3rd party interop. There was a discussion about how to describe/find the list of files to form a distribution (for the different sdist/bdist_* commands), but no agreement was reached. Some people strongly defend the setuptools feature to get the list of files from the source control system, in particular. http://mail.python.org/pipermail/distutils-sig/2009-April/011226.html David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Eric Smith writes: But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. ISTM that the problem that it solves is uninstall in the absence of the original installer. Am I to understand that you don't think that use case is important? Or that there's another way to do this? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote: But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. But doesn't this metadata give any two installers a common language to use to communicate, instead of having every pair of installers create their own private communication method? Personally, I like to be able to look at a package and say What did that install? Or contrary-wise, look at a file and say What package installed that? There are few things worse than discovering a bunch of mysterious executable files on your system that you don't remember installing, and then spending a few paranoid hours trying to determine whether you've stumbled across a root kit or virus or whether they have a legitimate reason to be there. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 09:38 AM 7/9/2009 +0900, Stephen J. Turnbull wrote: Eric Smith writes: But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. ISTM that the problem that it solves is uninstall in the absence of the original installer. Or uninstall where the installer is setup.py install, actually. Am I to understand that you don't think that use case is important? Or that there's another way to do this? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Wed, 08 Jul 2009 09:22:52 -0700, Paul Moore p.f.mo...@gmail.com wrote: If the only driver for this PEP is setuptools, then I'm -1 on it. Unless someone working on a packaging tool *other* than setuptools (or setuptools-derived projects) speaks up and says I have code of my own which uses distutils, and I would benefit as follows from PEP 376 and I will be changing my code to conform to PEP 376 I think the PEP should be rejected, and the energy put into some other PEP which will benefit the wider (non-setuptools) community. Other than easy_install/pip, there is also PyPM which is being developed at ActiveState. PyPM is the Python package manager much like what ppm is for ActivePerl. We build packages using 'setup.py install --root', but when it comes installation on client .. there is a separate database of installed packages that has the following information for each package installed: - metadata (name, version, author, etc..) - installed_files (list of files installed on client) So pypm remove Foo will simply remove all the 'installed_files' of Foo and remove Foo from the package database. The package database is currently an sqlite-based one. PEP 376 definitely will benefit PyPM besides benefiting setuptools/pip. If all package managers rely on common API to 1) retrieve metadata of installed packages 2) write metadata into site-packages), then it is very possible that one package manager does not mess up with the installation of others. For example: $ pip install IPython $ pypm install ipython Without PEP 376, pypm will overwrite pip's installation. But with PEP 376, as pip writes the RECORD file, pypm will reuse it (via the API) and co-operate with it. Eventually, I'd like PEP 376 to support system packagers too. So for example, if you did apt-get install python-pyqt4, then running pip install python-pyqt4 should return without installing anything .. as RECORD will be part of the .deb previously installed. As for generating the RECORD file, I vote for generating it during install time (w/ absolute paths). -srid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Steven D'Aprano wrote: On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote: But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. But doesn't this metadata give any two installers a common language to use to communicate, instead of having every pair of installers create their own private communication method? I really don't get this use case of multiple installers trying to install the same package. There's just no way that running yum install twisted and apt-get install twisted and pip install twisted are going to coexist with each other. The best they can do is say a file I'm trying to install already exists. Why try for anything else? Personally, I like to be able to look at a package and say What did that install? Or contrary-wise, look at a file and say What package installed that? I completely support that. All of the system installers I know of support this (at least on Linux; I'm familiar with the msi database layout (as exposed by Orca) on Windows, but I've never looked to see if they record the results of installing an msi). I just see this as an issue within each installer. Why would the installers need to communicate this with each other? There are few things worse than discovering a bunch of mysterious executable files on your system that you don't remember installing, and then spending a few paranoid hours trying to determine whether you've stumbled across a root kit or virus or whether they have a legitimate reason to be there. We can't solve the problem of here's an arbitrary file on the system, where did it come from?. There are many, many installers outside of our control that might have installed something. Ruby has installers, Perl has installers, I'm sure there are hundreds of such installers. The user might have even used wget to install something. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
P.J. Eby wrote: ISTM that the problem that it solves is uninstall in the absence of the original installer. Or uninstall where the installer is setup.py install, actually. I think we need to move away from setup.py install. It's the antithesis of static metadata. setup.py needs to go away. If we have a static metadata file called, say setup.info, then I'd like to see: bdist_rpm setup.info produce a .rpm bdist_msi setup.info produce a .msi bdist_deb setup.info produce a .deb bdist_egg setup.info produce a .egg and so on. There might be a library (and I call dibs on the name distlib :) that provides support routines to parse setup.info, but there's no framework involved. And no need for a plugin system. There might be support routines used to build setup.info. For those who think it's reasonable that the list of source files comes from your version control system, build a small command that produces setup.info from your vcs. If you need the version number to come from some other source, write a small command that reads that and sticks it into a setup.info. But the source distribution would be defined by a (possibly generated) setup.info. The metadata in setup.info would be extensible. If setup.info contains information that's needed to build a .rpm that's not needed for a .deb, then that's fine. We can define the minimum information needed and an extension mechanism. As time goes by, some of the extensions might become standardized. Then for each binary format, there's an installer (or more than one!), and it might handle dependency detection and automatic fetching, or it might not. Some of these exist (apt-get, yum, msi), some might need some tweaking (setuptools, pip) and others might need to be written from scratch. But the installer, its database of what it has installed, and how it handles uninstalls (if it does) is really not the issue. It's the conflation of build a package with install a package that distutils and setuptools has given us (via setup.py) that leads to us never making progress. Let's just focus on describe a package and let the packagers handle build a package and let the installers handle install and uninstall a package. Then the packagers (and their associated installers) can evolve independently of each other. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Eventually, I'd like PEP 376 to support system packagers too. So for example, if you did apt-get install python-pyqt4, then running pip install python-pyqt4 should return without installing anything .. as RECORD will be part of the .deb previously installed. As for generating the RECORD file, I vote for generating it during install time (w/ absolute paths). I think we should explicitly not support this. What if pip and apt-get (or rpm, or others) install the same package in different places because of system conventions (/usr vs. /usr/local vs. /opt, say)? There's no way we're ever going to get this right, and it's not worth complicating our lives over it. Seriously: Is there some real world use case I'm missing? Does any existing install system support this? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 11:20 PM 7/8/2009 -0400, Eric Smith wrote: P.J. Eby wrote: ISTM that the problem that it solves is uninstall in the absence of the original installer. Or uninstall where the installer is setup.py install, actually. I think we need to move away from setup.py install. It's the antithesis of static metadata. Please note that that's entirely out of scope for the PEP at hand. That being said, the rest of your proposal is strikingly similar to a proposal I previously floated on the distutils-sig for a concept called BUILDS. The main difference is that I suggested that the spec should include a standard interface for running build operations that would produce the manifest (equivalent to your setup.info), and that distutils and setuptools should provide setup.py commands to generate said manifest, to allow for a seamless transition. There was very little comment on the proposal, perhaps because it involves a lot of work that most people are sane enough not to sign up for. ;-) (Or more to the point, it's the sort of thing that never gets off the ground for design by consensus. The best way to make something like your proposal to happen is to go off and build it, and get it to take over. If it can't win substantial market share on its own merits, it probably doesn't deserve to be blessed as a standard.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 11:28 PM 7/8/2009 -0400, Eric Smith wrote: Eventually, I'd like PEP 376 to support system packagers too. So for example, if you did apt-get install python-pyqt4, then running pip install python-pyqt4 should return without installing anything .. as RECORD will be part of the .deb previously installed. As for generating the RECORD file, I vote for generating it during install time (w/ absolute paths). I think we should explicitly not support this. What if pip and apt-get (or rpm, or others) install the same package in different places because of system conventions (/usr vs. /usr/local vs. /opt, say)? There's no way we're ever going to get this right, and it's not worth complicating our lives over it. Seriously: Is there some real world use case I'm missing? Does any existing install system support this? The use case described above is supported right now by easy_install and pip; it doesn't require a RECORD file, though. An .egg-info file or an .egg-info directory with a PKG-INFO is enough to prevent duplicate installation. I don't know how pip handles file overwrites; easy_install OTOH never overwrites anything that's not in an .egg or a .pth. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On 03:28 am, e...@trueblade.com wrote: Eventually, I'd like PEP 376 to support system packagers too. So for example, if you did apt-get install python-pyqt4, then running pip install python-pyqt4 should return without installing anything .. as RECORD will be part of the .deb previously installed. As for generating the RECORD file, I vote for generating it during install time (w/ absolute paths). I think we should explicitly not support this. What if pip and apt-get (or rpm, or others) install the same package in different places because of system conventions (/usr vs. /usr/local vs. /opt, say)? There's no way we're ever going to get this right, and it's not worth complicating our lives over it. As I understand it, the point of the PEP is just to *provide the information* that the package installers need to get it right, not to actually implement the interaction with system package managers. Seriously: Is there some real world use case I'm missing? Does any existing install system support this? System package managers already provide .egg-info metadata, so this is hardly unprecedented. The antecedent of your this is somewhat ambiguous, so I'm not sure if any package manager provides the exact feature that you have in mind. If you're talking about different package managers installing stuff on the same system, you can look a bit farther afield: 'alien' allows you to install RPM and debian packages on the same system, with some reconciliation of dependencies and other metadata. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Paul Moore p.f.mo...@gmail.com: 2009/7/6 P.J. Eby p...@telecommunity.com: At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: As promised, here are some open questions on PEP 376. - Will the public API names be changed from *egginfo* to *metadata*? +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff as.) Thanks. I think this one is pretty non-controversial. +1, I'll change the PEP accordingly; - How will bdist_wininst/bdist_msi/bdist_rpm be updated? bdist_wininst, bdist_dumb, and various others use 'install --root' pattern to generate files for installation, which means that they would transparently end up writing a correct RECORD file, except for the inclusion of incorrect absolute paths for non-libdir-relative files. However, if we used the all relative in RECORD, with a base in INSTALLER, these cases could transparently be treated as another instance of install directory relocation. I don't know if bdist_msi does a --root install before generating the .msi; if it does, then it should work the same way. Yes, that sounds like a plausible approach. As I said earlier in the previous thread, all these commands rely on a call to the install command, so bdist_msi does too. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Paul Moore p.f.mo...@gmail.com:
2009/7/6 Nick Coghlan ncogh...@gmail.com:
I'd add one more question to the list: is allowing backslash separated
names in the RECORD file actually a good idea, or would it be better to
always use forward slashes?
They do always use forward slashes.
For the other questions, I don't have anything much to add to PJE's
comments, except that the all relative paths idea won't work due to
the Windows drive letter issue (i.e. if an installer puts files in
C:\Program Files, there is no guarantee that a relative path between
site-packages and Program Files even exists if Python is installed on a
different drive).
The big question, though, is can an installer actually *do* that in
practical terms?
- There are *no* guaranteed absolute locations on Windows, so any such
oddly-located file would require user interaction to work. Certainly
bdist_wininst and bdist_msi don't do that.
- My experiments indicate that bdist_{wininst,msi} are broken with
respect to absolute paths anyway: they do a --root install to a
temporary directory (and the absolute paths don't end up in there) and
then package up that temporary directory.
yes that's unfortunately the case for all windows-based installation.
wether it's a bdist call to the install command, to create a binary package,
wether it's an installation.
c:\something or d:\something will be installed in sys.prefix\something.
I will add an issue for distutils for this, probably ending up raising
an exception when we hit this case, because I don't see how these
paths can work.
Unless we define a drive that contains the python installation maybe, or
the Program Files directory
would that make sense from a win32 point of view ?
I still want to see a real life example that demonstrates that there
is a genuine issue here. We're spending a lot of energy and complexity
trying to design a solution to a problem that actually doesn't appear
to exist in practice...
(To be honest, I'd be fairly confident in saying that absolute paths
can be ignored on Windows, subject to some corner cases that I haven't
thought through yet. My worry is that I don't know what Unix and Mac
users might do, so I can't just wish away the issue because it can't
arise on Windows. Can a Unix/Mac user offer a real-world example on
their own system?)
I know some people are writing to /etc to add their configuration file
on the system,
So a real-world example under linux would be:
setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
That is basically how the examples are shown at:
http://docs.python.org/distutils/setupscript.html#installing-additional-files
But this is already os-specific, and exists because distutils doesn't
have a way (yet)
to express systems locations independantly from their physical location,
like what the RPM system does with %VARIABLES.
So another way to handle this maybe, like I have added with $PREFIX
and $EXEC_PREFIX
would be to nominate a list of variables that every python environment
has (querying
modules like sys) and let the developers use them as root locations
for some files.
- sys.prefix
- sys.exec_prefix
- some elements returned by distutils.sysconfig.get_config_vars()
(distutils.sysconfig queries the python Makefile amongst others)
- ...
Semi-related: distutils.sysconfig should be removed from distutils,
and be a standalone module in the sdtlib for example.
Regards
Tarek
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Tarek Ziadé ziade.ta...@gmail.com:
Unless we define a drive that contains the python installation maybe, or
the Program Files directory
would that make sense from a win32 point of view ?
I can't imagine that it would be useful in practice.
I know some people are writing to /etc to add their configuration file
on the system,
So a real-world example under linux would be:
setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
That is basically how the examples are shown at:
http://docs.python.org/distutils/setupscript.html#installing-additional-files
Thanks. Yes, that makes for a good example.
But this is already os-specific, and exists because distutils doesn't
have a way (yet)
to express systems locations independantly from their physical location,
like what the RPM system does with %VARIABLES.
So another way to handle this maybe, like I have added with $PREFIX
and $EXEC_PREFIX would be to nominate a list of variables that every
python environment has (querying modules like sys) and let the developers
use them as root locations for some files.
- sys.prefix
- sys.exec_prefix
- some elements returned by distutils.sysconfig.get_config_vars()
(distutils.sysconfig queries the python Makefile amongst others)
- ...
This adds extra complexity to the RECORD format, for little practical
benefit that I can see.
From the POV of core distutils:
- Windows users use bdist_wininst and/or bdist_msi, and absolute
locations are a lost cause.
- Presumably, some people use bdist_rpm (I don't know if there are
other ways of creating RPMs).
- Everyone else uses setup.py install or a 3rd party tool.
- PEP 302 style loaders aren't relevant as the core only uses the
filesystem (not even zip files).
- Only 3rd party tools will consume this data.
So, we need input from developers of 3rd party tools here. Phillip has
stated the case for setuptools, from his POV having everything
relative to the install location which is stored elsewhere (in the
installer file) is fine. I'd like to know whether he needs
upwards-pointing relative paths like ../../../../xx.py, but that's a
small detail.
So - do any other potential users of the PEP 376 metadata want to speak up?
At the moment, it feels like we're designing things more or less in a vacuum.
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Tue, Jul 7, 2009 at 10:33 AM, Paul Moorep.f.mo...@gmail.com wrote:
2009/7/7 Tarek Ziadé ziade.ta...@gmail.com:
Unless we define a drive that contains the python installation maybe, or
the Program Files directory
would that make sense from a win32 point of view ?
I can't imagine that it would be useful in practice.
I know some people are writing to /etc to add their configuration file
on the system,
So a real-world example under linux would be:
setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
That is basically how the examples are shown at:
http://docs.python.org/distutils/setupscript.html#installing-additional-files
Thanks. Yes, that makes for a good example.
But this is already os-specific, and exists because distutils doesn't
have a way (yet)
to express systems locations independantly from their physical location,
like what the RPM system does with %VARIABLES.
So another way to handle this maybe, like I have added with $PREFIX
and $EXEC_PREFIX would be to nominate a list of variables that every
python environment has (querying modules like sys) and let the developers
use them as root locations for some files.
- sys.prefix
- sys.exec_prefix
- some elements returned by distutils.sysconfig.get_config_vars()
(distutils.sysconfig queries the python Makefile amongst others)
- ...
This adds extra complexity to the RECORD format, for little practical
benefit that I can see.
From the POV of core distutils:
- Windows users use bdist_wininst and/or bdist_msi, and absolute
locations are a lost cause.
- Presumably, some people use bdist_rpm (I don't know if there are
other ways of creating RPMs).
They are othe ways to generate RPMs.
There are also debian packaging scripts
- Everyone else uses setup.py install or a 3rd party tool.
- PEP 302 style loaders aren't relevant as the core only uses the
filesystem (not even zip files).
- Only 3rd party tools will consume this data.
So, we need input from developers of 3rd party tools here. Phillip has
stated the case for setuptools, from his POV having everything
relative to the install location which is stored elsewhere (in the
installer file) is fine. I'd like to know whether he needs
upwards-pointing relative paths like ../../../../xx.py, but that's a
small detail.
So - do any other potential users of the PEP 376 metadata want to speak up?
At the moment, it feels like we're designing things more or less in a vacuum.
I am CC'ing the people that worked with us for the versionning matters,
they can speak from a Fedora and Ubuntu POV, I am also adding Jim for
zc.buildout,
he can provide precious input.
(I am sorry if some people get the mail twice)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On 6 Jul, 2009, at 20:38, Paul Moore wrote: - Should distribution names be case insensitive on case insensitive filesystems? For comparison, module/package names are always case sensitive even on case insensitive systems. I'd then go for case sensitive names for distributions as well, just to be consistent. But isn't the actual name in the PKG-INFO file anyway? Do you mean really case insensive filesystems like DOS' FAT, or case preserving (HFS+ on OSX, NTFS on Windows)? For the latter you can at least get the original name back using os.readdir. BTW. Actually determining if you are working with a case-sensitive fileystem requires filesystem access, os.path.normcase is hopelessly naieve in that respect. I'm reguarly dealing with NFS mounted filesystems on OSX systems, which means that parts of a path are case preserving (the HFS+ filesystem upto the mount point) and other parts are truly case sensitive (paths on the NFS mounted filesystem originating on a Linux server). Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Paul Moore wrote:
I know some people are writing to /etc to add their configuration file
on the system,
So a real-world example under linux would be:
setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
That is basically how the examples are shown at:
http://docs.python.org/distutils/setupscript.html#installing-additional-files
Thanks. Yes, that makes for a good example.
But this is already os-specific, and exists because distutils doesn't
have a way (yet)
to express systems locations independantly from their physical location,
like what the RPM system does with %VARIABLES.
So another way to handle this maybe, like I have added with $PREFIX
and $EXEC_PREFIX would be to nominate a list of variables that every
python environment has (querying modules like sys) and let the developers
use them as root locations for some files.
I think you have to differentiate between packages and applications.
Packages will usually have their own way of getting configured,
either by passing parameters via some API or pointing the package
to a configuration file. They may come with some example config
files, but should normally don't interfere with the system
configuration. I.e. putting a Python package configuration into /etc
does not really sound like a good idea.
Applications tend to ship everything needed to run the application
together with the installer and typically use a system-dependent
installer rather than a distutils based approach. These then
place config files in the usual default dirs of the system.
This is out-of-scope for PEP 376.
Then you have tools like zc.buildout which basically build
an application in some directory at install time. Since only
these tools know what they are doing, the whole uninstall
mechanism also lies in their hands. Again, PEP 376 would only
apply to the dynamic package installation part, but not to the
complete application build.
Another aspect to consider is that config files should normally
not be uninstalled automatically. The user should either be asked
whether she wants to keep the files or the uninstaller should leave
them untouched and issue a warning that certain files were not
uninstalled.
Summarizing, I think it's better not to record config files
and other user-edited files in the RECORD file.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Jul 07 2009)
Python/Zope Consulting and Support ...http://www.egenix.com/
mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free !
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 M.-A. Lemburg m...@egenix.com: I think you have to differentiate between packages and applications. Agreed. I believe that only packages should be considered here. Applications are the focus of tools like py2exe on Windows, and (AIUI) things like workingenv. These tools should (will) have their own approaches. The only cases I know where there is reason for a package to store paths outside the package directory are: - executable scripts, which go in sys.prefix/Scripts on Windows, and ??? on Unix/MAC OS - supporting files (MoinMoin puts its HTML documents etc in sys.prefix/share on Windows, cx_Oracle puts its documentation in sys.prefix/cx_Oracle-doc) Executable files could probably be superseded by using python -m, but compatibility and users' preference for having a real script probably means they aren't going away in the near future. Support files are getting put in the package directory more often these days, but stuff that needs to be found by non-Python tools is arguably still better in a more discoverable location. For Windows, having a few distinguished locations under the installation root (scripts, share, doc) would probably do. For Unix and Mac OS, I have no opinion (but I suspect that absolute paths like /usr/local/bin might be the norm there). [...] Summarizing, I think it's better not to record config files and other user-edited files in the RECORD file. The RECORD file should contain precisely those files that are created as part of the install. That's ultimately the point of the file (for ownership queries and uninstallation). Hmm, on the other hand, if foo.py is in the RECORD file, the uninstaller should uninstall foo.pyc and foo.pyo as well. And a query as to whether the distribution owns foo.pyc should return True. How will this be handled? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Paul Moore wrote:
2009/7/6 Nick Coghlan ncogh...@gmail.com:
- There are *no* guaranteed absolute locations on Windows, so any such
oddly-located file would require user interaction to work. Certainly
bdist_wininst and bdist_msi don't do that.
- My experiments indicate that bdist_{wininst,msi} are broken with
respect to absolute paths anyway: they do a --root install to a
temporary directory (and the absolute paths don't end up in there) and
then package up that temporary directory.
I still want to see a real life example that demonstrates that there
is a genuine issue here. We're spending a lot of energy and complexity
trying to design a solution to a problem that actually doesn't appear
to exist in practice...
(To be honest, I'd be fairly confident in saying that absolute paths
can be ignored on Windows, subject to some corner cases that I haven't
thought through yet. My worry is that I don't know what Unix and Mac
users might do, so I can't just wish away the issue because it can't
arise on Windows. Can a Unix/Mac user offer a real-world example on
their own system?)
I thought installing pywin32 based COM objects still involved messing
with the Windows directory, but MS may have improved that in more recent
OS versions. It's been years since I played with win32com objects, and
even then it was just idle experimentation that didn't get very far so I
could easily be misremembering.
For *nix, the obvious use case is installing scripts somewhere like
/usr/bin or /usr/local/bin.
One option is to punt on this whole issue and say if people want to
install stuff outside the Python module heirarchy they should create
their own OS-specific package to manage it (i.e. leave the non-relative
paths to be managed by APT or a Windows installer or whatever).
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Paul Moore p.f.mo...@gmail.com writes: The only cases I know where there is reason for a package to store paths outside the package directory are: I think the *only* files that actually belong in the package directory are the Python modules inside that package. Other files need to be easily, and automatically, separable for purposes of installation. - executable scripts, which go in sys.prefix/Scripts on Windows os.path.join(sys.prefix, 'Scripts') and ??? on Unix/MAC OS Depending on whether the developer designates the programs for sysadmin-only use or not: os.path.join(sys.prefix, 'bin') os.path.join(sys.prefix, 'sbin') - supporting files (MoinMoin puts its HTML documents etc in sys.prefix/share on Windows, cx_Oracle puts its documentation in sys.prefix/cx_Oracle-doc) This category, it seems to me, needs to be expanded with metadata that allows “purpose”-based tagging, so that platform-specific standards can be applied using those purposes to determine the correct location for these non-Python-module files. Support files are getting put in the package directory more often these days Only, IMO, because there's no way of flagging it as anything but arbitrary “data”. Examples of purpose-based classifications that need to be distinctly declarable: executable program, importable module source, platform-independent compiled module, platform-dependent compiled module, documentation, run-time variable data, static data, configuration, and so on. All of these (and others I've forgotten) should be possible for the developer to declare in distribution metadata, and the installer can then use those declarations to make the files go to platform-specific locations, not the Python package directories. For Windows, having a few distinguished locations under the installation root (scripts, share, doc) would probably do. For Unix and Mac OS, I have no opinion (but I suspect that absolute paths like /usr/local/bin might be the norm there). The lines are drawn in different places on each platform; we don't want to have the union of all these different platform-specific locations in the standard, and likewise we don't want to leave any of them with second-class support. I think it's not the developer's burden to decide *where* such files go; rather, they should be declaring only the *purpose* of these files in the distribution metadata, and it's up to the site-specific installer (possibly as configured by the installing user) to decide the location of each file by its declared purpose. -- \ “I can picture in my mind a world without war, a world without | `\ hate. And I can picture us attacking that world, because they'd | _o__) never expect it.” —Jack Handey | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Ben Finney ben+pyt...@benfinney.id.au: [... lots of interesting stuff deleted ...] I think it's not the developer's burden to decide *where* such files go; rather, they should be declaring only the *purpose* of these files in the distribution metadata, and it's up to the site-specific installer (possibly as configured by the installing user) to decide the location of each file by its declared purpose. That's a whole different PEP, though. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Tue, 7 Jul 2009 at 13:05, Paul Moore wrote: 2009/7/7 Ben Finney ben+pyt...@benfinney.id.au: [... lots of interesting stuff deleted ...] I think it's not the developer's burden to decide *where* such files go; rather, they should be declaring only the *purpose* of these files in the distribution metadata, and it's up to the site-specific installer (possibly as configured by the installing user) to decide the location of each file by its declared purpose. That's a whole different PEP, though. Which one? It seems to me that supporting this is implicit in the language summit goals of (1) having distutils be better support infrastructure for system packaging utilities and (2) needing a way to deal with resource files that might be installed in a specific place on the target system by the system packager. I'll grant that I'm reading between the lines, it isn't an explicitly stated goal. But it was the direction my mind went when I read Tarek's notes, given that the first stated goal is standardize more metadata. But I'm not one of the people involved in system packaging tools, so I'll leave it to them to say how useful/important this is. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Tue, Jul 7, 2009 at 2:52 PM, R. David Murrayrdmur...@bitdance.com wrote: On Tue, 7 Jul 2009 at 13:05, Paul Moore wrote: 2009/7/7 Ben Finney ben+pyt...@benfinney.id.au: [... lots of interesting stuff deleted ...] I think it's not the developer's burden to decide *where* such files go; rather, they should be declaring only the *purpose* of these files in the distribution metadata, and it's up to the site-specific installer (possibly as configured by the installing user) to decide the location of each file by its declared purpose. That's a whole different PEP, though. Which one? It seems to me that supporting this is implicit in the language summit goals of (1) having distutils be better support infrastructure for system packaging utilities and (2) needing a way to deal with resource files that might be installed in a specific place on the target system by the system packager. I'll grant that I'm reading between the lines, it isn't an explicitly stated goal. But it was the direction my mind went when I read Tarek's notes, given that the first stated goal is standardize more metadata. Yes but the topic is so wide that it has to be cut in several PEP, and things have to be done gradually So far: - PEP 376 : standard for the metadata format and location + query APIs - PEP 345 : standard for the metadata *content* - work in progress too (there's a branch with new fields waiting) - PEP 386 : standard for version comparisons topics that are not yet in PEP are grouped on the wiki page (under current work) with notes : http://wiki.python.org/moin/Distutils When I started to work on this I didn't realize the gigantic amount of work and coordination it requires, and I do understand now the current state. At first I was trying to coordinate interested people to work on each topic mentioned there in parallel. (like we did a bit after the summit) But at the end, it seems that having everyone interested in packaging matters focusing on the less number of possible topics pays more. PEP 376 is just a piece of the puzzle but I am confident it will speed up other tasks since it raises our common ground knowledge. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote: When I started to work on this I didn't realize the gigantic amount of work and coordination it requires No one expects the package inquisition. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Paul Moore p.f.mo...@gmail.com: The RECORD file should contain precisely those files that are created as part of the install. That's ultimately the point of the file (for ownership queries and uninstallation). Hmm, on the other hand, if foo.py is in the RECORD file, the uninstaller should uninstall foo.pyc and foo.pyo as well. And a query as to whether the distribution owns foo.pyc should return True. How will this be handled? It's planned to list them as well in RECORD, since install calls a sub command that build thems (install_lib), So the same rules apply than the .py ones. But there's a special case : if the --no-compile or the --no-optimize option is used, then the pyc|pyo files are not added. Which means they will be created afterwards when the module is used on the target system. So the pyc|pyo files could be removed when they are present besides the py file that is being removed. Although, it will still be required to write them in the RECORD file by the install command when the --no-compile or the --no-optimize options are *not* used. So they are properly detected and removed when the py files are not distributed in binary distributions, but just pyc files. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Nick Coghlan ncogh...@gmail.com:
For *nix, the obvious use case is installing scripts somewhere like
/usr/bin or /usr/local/bin.
Using distutils' scripts option, they will end up in : sys.exec_path/bin
Another use case I've found in a distro I've installed this afternoon :
setup(..., data_files=[('/usr/share/man/man1/', ['SOMEFILE'])], ...)
That's not the most elegant way to add a man page, but for someone
who doesn't bather with APT or whatever, it works to build a binary
distribution.
One option is to punt on this whole issue and say if people want to
install stuff outside the Python module heirarchy they should create
their own OS-specific package to manage it (i.e. leave the non-relative
paths to be managed by APT or a Windows installer or whatever).
If so, what do we do with the data_files option in distutils ?
If it's used with absolute paths, files can be installed anywhere on
the system, and we want to track them.
Even if we don't uninstall them automatically, they should be tracked
so a third-party
uninstaller can deal with them properly.
Or do we change this distutils feature and state that the directories
used in data_files
will always be relative to sys.prefix ?
That would bring us back to three cases in the RECORD:
- files located under sys.prefix, but not located under site-packages
- files located under sys.exec_prefix, but not located under site-packages
- files located under site-packages
Where site-packages is the directory that contains .egg-info
directory of the distribution
(that's basically the current PEP state, beside the absolute paths
case we would need to remove)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Tue, Jul 7, 2009 at 10:31 PM, P.J. Ebyp...@telecommunity.com wrote: At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote: When I started to work on this I didn't realize the gigantic amount of work and coordination it requires No one expects the package inquisition. ;-) Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Tue, 7 Jul 2009 at 23:30, Tarek Ziad? wrote: On Tue, Jul 7, 2009 at 10:31 PM, P.J. Ebyp...@telecommunity.com wrote: At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote: When I started to work on this I didn't realize the gigantic amount of work and coordination it requires No one expects the package inquisition. ?;-) Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ? See the recent thread on python-committers about mandatory Monty Python cultural education for committers :) See also http://www.youtube.com/watch?v=gldlyTjXk9A --David___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Tarek Ziadé wrote: On Tue, Jul 7, 2009 at 10:31 PM, P.J. Ebyp...@telecommunity.com wrote: At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote: When I started to work on this I didn't realize the gigantic amount of work and coordination it requires No one expects the package inquisition. ;-) Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ? It's a Monty Python reference: http://en.wikipedia.org/wiki/The_Spanish_Inquisition_(Monty_Python) http://www.youtube.com/watch?v=gldlyTjXk9A Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
Tarek Ziadé writes: On Tue, Jul 7, 2009 at 10:31 PM, P.J. Ebyp...@telecommunity.com wrote: No one expects the package inquisition. ;-) Sorry, i've looked in the english dictionary but I don't get this one. I think that far more important than PEP 385 conversion of Roundup and other utilities to recognize Mercurial revision identifiers is a filter for Python channels that recognizes Monty Python references and automatically inserts YouTube video URLs for this essential background. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 11:30 PM 7/7/2009 +0200, Tarek Ziadé wrote: On Tue, Jul 7, 2009 at 10:31 PM, P.J. Ebyp...@telecommunity.com wrote: At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote: When I started to work on this I didn't realize the gigantic amount of work and coordination it requires No one expects the package inquisition. ;-) Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ? It's a Monty Python reference, i.e. no one expects the Spanish Inquisition. In the Spanish Inquisition sketch, a character is asked several questions he can't answer, and then complains that he wasn't expecting some kind of Spanish inquisition, at which point three scarlet-robed cardinals burst into the room and exclaim, NO ONE expects the Spanish Inquisition! So I punned on that in order to imply that nobody who takes on the job of packaging expects to be asked the kind of detailed, unanswerable questions that come with the territory of packaging, or to be subjected to torture by third parties merely for trying to do something useful. And, by further implication, I was thus expressing camaraderie with you on your initiation into the wonderful world of trying to satisfy everyone in Pythonland, and our shared experience in the area of if I'd really known what I was getting into, I probably wouldn't have. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: As promised, here are some open questions on PEP 376. - Will the public API names be changed from *egginfo* to *metadata*? +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff as.) - What precisely are the use cases for absolute path names? Concrete examples are needed. With the current spec, some things can go wrong (e.g., see below), so we need real use cases to know how to address this. The purpose is to be able to remove scripts, data files, documentation, etc. that's stored in some place that's not package-relative, and survive a move of the package directory itself. For example, suppose I install with --prefix=~, and have files in ~/bin and ~/lib. I then move ~/lib/python2.7 to ~/pylib. If the scripts were installed with a relative path, this would break the link to the script still living in ~/bin. Now that I'm thinking about this, it occurs to me that there is another way to address this besides mixing relative and absolute paths in RECORD, and that is to make *all* the paths relative, and include an original installation location in INSTALLER. Then, it would be possible to detect whether the installation location had been moved and handle it accordingly... for some definition of accordingly. I agree that further discussion is probably needed to resolve this. - How will bdist_wininst/bdist_msi/bdist_rpm be updated? bdist_wininst, bdist_dumb, and various others use 'install --root' pattern to generate files for installation, which means that they would transparently end up writing a correct RECORD file, except for the inclusion of incorrect absolute paths for non-libdir-relative files. However, if we used the all relative in RECORD, with a base in INSTALLER, these cases could transparently be treated as another instance of install directory relocation. I don't know if bdist_msi does a --root install before generating the .msi; if it does, then it should work the same way. - Can distutils be made to install files in places the current RECORD file spec can't handle? (I think the answer is yes). What happens then? Surely an absolute path is sufficient? - Should distribution names be case insensitive on case insensitive filesystems? For comparison, module/package names are always case sensitive even on case insensitive systems. IMO, they should be case-insensitive *regardless* of filesystem case, and I believe PyPI has already begun disallowing packages with only case-distinct names (e.g. 'Zope' vs. 'ZoPe') to be registered. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
(I cancelled sending this the first time, so apologies if a half-written version turns up) Paul Moore wrote: As promised, here are some open questions on PEP 376. I'd add one more question to the list: is allowing backslash separated names in the RECORD file actually a good idea, or would it be better to always use forward slashes? My reason for this question is what happens if (for example) a bdist_* installer is generated on Linux and then used on Windows or vice-versa? If the expected RECORD format is different on the two platforms, won't it get things wrong? For the other questions, I don't have anything much to add to PJE's comments, except that the all relative paths idea won't work due to the Windows drive letter issue (i.e. if an installer puts files in C:\Program Files, there is no guarantee that a relative path between site-packages and Program Files even exists if Python is installed on a different drive). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/6 P.J. Eby p...@telecommunity.com: At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: As promised, here are some open questions on PEP 376. - Will the public API names be changed from *egginfo* to *metadata*? +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff as.) Thanks. I think this one is pretty non-controversial. - What precisely are the use cases for absolute path names? Concrete examples are needed. With the current spec, some things can go wrong (e.g., see below), so we need real use cases to know how to address this. The purpose is to be able to remove scripts, data files, documentation, etc. that's stored in some place that's not package-relative, and survive a move of the package directory itself. For example, suppose I install with --prefix=~, and have files in ~/bin and ~/lib. I then move ~/lib/python2.7 to ~/pylib. If the scripts were installed with a relative path, this would break the link to the script still living in ~/bin. Now that I'm thinking about this, it occurs to me that there is another way to address this besides mixing relative and absolute paths in RECORD, and that is to make *all* the paths relative, and include an original installation location in INSTALLER. Then, it would be possible to detect whether the installation location had been moved and handle it accordingly... for some definition of accordingly. I agree that further discussion is probably needed to resolve this. I'm not 100% sure I follow your logic here. If you're renaming directories after install, without involving the package manager, I think you probably deserve whatever you get :-) Having said that, your proposal (all paths relative, plus an install location) is similar to something I mentioned earlier (but better explained :-)) The only case it doesn't cater for is when an installed file can't be expressed as relative to the install location (which would involve being on a different drive on Windows). That's likely to never work, so I doubt it counts in any case. I've got some concerns about the possibility of having locations above the install location - something like ../../scripts/driver.bat. I'm not sure if such things will work (consider if the install location is a zipfile or egg) but nor am I sure anything generates such names. As you say, more discussion (and use cases!) would help here. - How will bdist_wininst/bdist_msi/bdist_rpm be updated? bdist_wininst, bdist_dumb, and various others use 'install --root' pattern to generate files for installation, which means that they would transparently end up writing a correct RECORD file, except for the inclusion of incorrect absolute paths for non-libdir-relative files. However, if we used the all relative in RECORD, with a base in INSTALLER, these cases could transparently be treated as another instance of install directory relocation. I don't know if bdist_msi does a --root install before generating the .msi; if it does, then it should work the same way. Yes, that sounds like a plausible approach. - Can distutils be made to install files in places the current RECORD file spec can't handle? (I think the answer is yes). What happens then? Surely an absolute path is sufficient? Current spec says relative or $PREFIX/... or $EXEC_PREFIX/... I think that was based on earlier discussion, but it's hampered by the fact that nobody's pointing at a real case of a distribution that actually tries to *install* files in odd places. (Actually, something that installs a driver script, like twisted's twistd or nose's nosetests, would do - I'll see what I can find tomorrow). - Should distribution names be case insensitive on case insensitive filesystems? For comparison, module/package names are always case sensitive even on case insensitive systems. IMO, they should be case-insensitive *regardless* of filesystem case, and I believe PyPI has already begun disallowing packages with only case-distinct names (e.g. 'Zope' vs. 'ZoPe') to be registered. OK, that's another option :-) It's not hard to change the code, so as long as a final decision is reached at some point, we're OK here... Thanks for the comments, Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/6 Nick Coghlan ncogh...@gmail.com:
I'd add one more question to the list: is allowing backslash separated
names in the RECORD file actually a good idea, or would it be better to
always use forward slashes?
They do always use forward slashes.
For the other questions, I don't have anything much to add to PJE's
comments, except that the all relative paths idea won't work due to
the Windows drive letter issue (i.e. if an installer puts files in
C:\Program Files, there is no guarantee that a relative path between
site-packages and Program Files even exists if Python is installed on a
different drive).
The big question, though, is can an installer actually *do* that in
practical terms?
- There are *no* guaranteed absolute locations on Windows, so any such
oddly-located file would require user interaction to work. Certainly
bdist_wininst and bdist_msi don't do that.
- My experiments indicate that bdist_{wininst,msi} are broken with
respect to absolute paths anyway: they do a --root install to a
temporary directory (and the absolute paths don't end up in there) and
then package up that temporary directory.
I still want to see a real life example that demonstrates that there
is a genuine issue here. We're spending a lot of energy and complexity
trying to design a solution to a problem that actually doesn't appear
to exist in practice...
(To be honest, I'd be fairly confident in saying that absolute paths
can be ignored on Windows, subject to some corner cases that I haven't
thought through yet. My worry is that I don't know what Unix and Mac
users might do, so I can't just wish away the issue because it can't
arise on Windows. Can a Unix/Mac user offer a real-world example on
their own system?)
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
