Re: [Distutils] People want CPAN :-)
David Lyon wrote: Even if you implement PyPI + PEP 381 + PEP 381 tomorrow I promise you that you won't be anywhere close to CPAN. It's a much more serious challenge than perphaps you realise.. I keep reading and I keep hearing you and others saying this, but as someone who has never used CPAN, I'm not seeing the large number of specific implementable tasks that CPAN clearly has and PyPI clearly does not. Without handwaving please, in a technical sense what does CPAN have that is so wonderful other than the items mentioned so far of: - buildbot on the server - mirrored. hierarchical servers - one widely known and accepted way of doing packaging If we implement those, if we implement PEP 381, in what specific way will we still fall *far short* as you suggest? I can't fix the social aspects so those are not of use to me. Only architecture. BTW at the recent pyTexas regional conference we had a good group discussion about packaging, with people offering analysis from the Perl, Java, Ruby and Haskell communities. It seems each language still only covers part of the solution, albeit different parts. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [issue89] easy_install silently drop symlinks when auto-extracting tarball source distributions
P.J. Eby wrote: At 12:01 AM 11/2/2009 +0100, Martin v. Löwis wrote: It's actually Jeff Rush who is in charge of this tracker instance. I can't change the setup without him agreeing. So, Jeff, where else should the roundup instance for setuptools send its notifications? In over 1 year, there are now a grand total of 89 such messages (at most) that have been sent to the list, vs. 5 messages just sent now (6 when you count this one) discussing whether or not they should continue to be sent here. So far, there has also only been one complaint or request to change this, as far as I'm aware. (Personally, I like them just fine where they are. Among other things it makes it easy for someone to answer the ticket via email, which is one of the reasons I asked for it to be set up this way in the first place.) I'm happy with the existing arrangement - I think sending low-traffic bug reports to a separate list is a bad idea because few folk will actually see them. If the traffic level increases, then we can consider moving notifications to a separate list hosted at python.org. With our own listserver I see no reason to use Google Groups though as there are people who refuse to get an account there. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools 0.6c10 release imminent; please test
sstein...@gmail.com wrote: On Oct 12, 2009, at 7:48 PM, P.J. Eby wrote: At 07:28 PM 10/12/2009 -0400, sstein...@gmail.com wrote: In any case, the update is not intended for people who are happy to have Distribute, but the people who are unhappy about having to switch, or deal with its workarounds... or just wish the whole discussion would go away. Thanks P.J. very much for this release of setuptools. I'm one of those who'd prefer not to switch to distribute until it has matured and built up a reputation for responsible stewardship of the technology. When it does, great I'm there but until then setuptools does what I need without hassle. The fall-down was in the testing done before the Python release and I'm sure more testing will be done in that area before the 2.6.4 bugfix release. Not really - according to the emails on this list, the incompatibility was found _before_ the 2.6.3 release and it was decided so what let things break. More testing would _not_ have prevented this problem. Too little, too late, no thanks, I'll just be sticking with Distribute from now on. Several developers and an open development process vs a lone coder with a closed codebase? That's not really a choice at all... Sorry, it doesn't look like anyone wants to play with you any more; you can just keep your ball. Please don't speak for others - it's rude. Only speak for yourself. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Package install failures in 2.6.3 - setuptools vs Distribute
Lennart Regebro wrote: 2009/10/3 Ned Deily n...@acm.org: This is not a good experience for users. Unless I'm missing something (and I hope I am), this issue really can't be hand-waved away. It's unfortunate that this comes in a minor release. Very unfortunate, as in, it should NOT have happened. And *especially* without any announcement on python.org or mention on the python-committers list of something this major. But at the same time we can hardly avoid fixing bugs just because setuptools isn't getting updated at the moment. It's a lose-lose situation. Setuptools is hardly a minor software tool. Considering that setuptools is a major focus and key technology of this group, those making changes to distutils should have tested against setuptools and devised a solution providing backward compatibility, along with deprecation warnings. Lennart and Tarek, I know you at least are familiar with this valuable practice in Zope for years. At the least, those making changes to Distutils should, after testing against Setuptools, widely announced this was coming. It does not reflect positively on the Distribute team and even hints of under-the-table intentionality in forcing the community to abandon use of setuptools. Distribute should win because of superior technology not forced migration. As I see it this will speed up adaptation of Distribute. Word will spread. It's not ideal, but then it's not a perfect world. Distribute is very new and there are many folk who will not be adopting it until it has been out for quite some time. It is a fact of the conservative nature of some development teams. If setuptools were an optional, little-used technology in Python it would not be a big deal but it isn't. It is not appropriate to -force- users to adopt a particular branch of a fork, except perhaps in the move from Python 2.x to 3.x where major changes are anticipated and there was no setuptools for 3.x anyway. Considering that 2.6.3 is messed up in other ways, like displaying the SVN rc1 tag and a broken logging module, and probably will be re-released at 2.6.4 very shortly, perhaps we can get this -at least- into the Python docs and maybe introduce backward compatible hooks into distutils to support setuptools. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtualenv and buildout integration ?
Yonsy Solis wrote: we recently began tu use virtualenv and buildout for development and deployments in some projects (more one internal effort to convince the ppl heads to use this for the next projects) by now i have two ways to work. 1) create a venv and inside this create the buildout dir project. 2) create a venv and in the same venv directory create the buildout tree. by now both options require to do two operations, first create the venv, activate this and after this create the buildout and run this. i am thinking about develop an recipe/extension for buildout to: - create a virtualenv - activate the venv - rerun the buildout using the ptyhon venv (bypassing the venv part) Not quite the approach you propose but (in a french Monty Python and the Holy Grail accent) We're already got one. ;-) Like you I need to initialize new projects often and tired of repeating various steps. One day I noticed that the virtualenv command is extensible so I extended it to init a buildout after creating a sandbox. I also had it drop in some default files like a buildout.cfg with my favorite options, license/README.txt files and the buildout bootstrap.py that I always forget about. The extension also changes the default sandbox option to --no-site-packages since I use that most often. You can find it at: https://www.dfwpython.org/repo/Projects/sandbox/mk_sandbox.py To create your new command, run once with your chosen version of Python: $ python2.6 mk_sandbox.py $ cp sandbox /usr/local/bin/ now anytime I need a readymade project I run: $ sandbox projectx $ cd projectx (edit buildout.cfg) $ bin/buildout (make some changes) $ hg ci It has some new command-line options as well. I find it quite useful during presentations when you need to spin up a workarea to illustrate some concept spontaneously. $ sandbox --help Usage: sandbox [OPTIONS] DEST_DIR Options: --version show program's version number and exit -h, --helpshow this help message and exit -v, --verbose Increase verbosity -q, --quiet Decrease verbosity -p PYTHON_EXE, --python=PYTHON_EXE The Python interpreter to use, e.g., --python=python2.5 will use the python2.5 interpreter to create the new environment. The default is the interpreter that virtualenv was installed with (/usr/bin/python2.6) --clear Clear out the non-root install and start from scratch --unzip-setuptoolsUnzip Setuptools when installing it --relocatable Make an EXISTING virtualenv environment relocatable. This fixes up scripts and makes all .pth files relative --site-packages Give access to the global site-packages dir to the virtual environment --no-buildout Omit installing zc.buildout into the sandbox --no-mercurialOmit initializing a Mercurial repository in the sandbox -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Static metadata using setup.cfg
David Lyon wrote: On Mon, 17 Aug 2009 09:34:56 +0200, Tarek Ziadé ziade.ta...@gmail.com wrote: the long_description field is expressed as a multi-line field following the config parser convention so no problem for it (see my example) The long_description field is also a reStructuredText field and as such, for the developer to preview it during its composure, it needs to be left-aligned. While a developer -could- cut/paste it into a separate text file each time he wants to test preview it, that would be annoying. Although there's another change we need to apply and decide : being able to express a list of values, for fields like keywords or classifiers. [Files] file1 = abc.txt file2 = def.txt You can get all the keys in [Files] in a list in the configparser class. Many of these ideas have already been expressed in a buildout.cfg file that also uses ConfigParser. Look there for ideas on how to do lists, order things, substitute fields. However if you put too much expressive power into the metadata file, you make it hard to programatically reason about what is means, which is the whole point of having a static metadata file. If you put too little, people will stick with setup.py. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Cache PYTHONPATH? (Re: make unzipped eggs be the default)
David Lyon wrote: Third party libraries are rarely so big that they need to be compressed to save disk space.. on any of the systems that i know about anyway.. Hi David. Not just your post but others here are making assumptions on your own working environment. Yes there are systems you need to save disk space on, yes there are systems where you care about I/O performance. These are embedded systems. Python has a strong and growing following on small devices such as phones (OpenMoko), music players, settops, netbooks, OLPC/XO and such. If you haven't been following it, the Python-on-a-Chip initiative formed from several projects took place at PyCon 2009. The language is in a position to become the standard control language for devices, if we don't hobble it by assuming Python is always run on a full-blown desktop. This attitude of allowing Python to always grow larger is prevalent on the core developers list as well, where they are removing the ability to compile Python selectively to drop out those portions not needed on a platform. The attitude there was if the embedded folks want a stripped down version they can create and maintain it themselves, redoing work already done years ago. But they won't -- they'll chose the path of least resistance and choose a more lightweight language. Pardon the rant. I just get frustrated when people believe that the path forward is faster and bigger systems on our desktops when actually desktops are dying and will be rare in ten years. Let's keep Python lean and flexible so it takes up residence in the infrastructure instead. And the benefit of defaulting to zipped eggs is that it enforces on the developer the discipline of writing his packages to use pkg_resources instead of file I/O, to retain the future option of alternate packaging formats. Developers know, especially those using test-driven-development, that if you don't regularly test against an environment, your code will gradually rot and no longer run in that environment. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Removing the hack from setup.py - Setup.py as a Class, not a Procedure
David Lyon wrote: On Tue, 21 Jul 2009 14:12:48 -0400, P.J. Eby p...@telecommunity.com wrote: The concept of package data is data in the same directory as the package's Python code. Using it to install libraries is a bit of a hack in the first place. Too right it's a hack - haha. I'm glad that I'm not the only one who's noticed. You're misreading him - the hack is using an attribute of package_data, which is intended for describing data files residing within a package directory, in a way in which it was not intended to describe data _outside_ a package directory. That's a definition of a hack. If an egg wants to splatter data files outside the Python package area, other than scripts in /usr/local/bin/, I'd consider that very bad behavior. My reasons are that (1) such splattering is usually very OS/platform-specific, (2) the files splattered may have overwritten files under the control of a native system packaging system, with no way to recover the originals and (3) there is no real metadata or directory proximity linking the splattered files to a particular Python distribution. These properties have a destabilizing effect on a system over time. Now it's somewhat restrictive. And my guess + experience is that it can take about 3+ days to get a typical setup.py to work properly because of all the hacks. (For a newcomer) Setuptools is not the hack (by the definition of 'hack') -- using setuptools in unusual ways is the hack. Most often the problems I see people have are they do not grok the problem that setuptools is solving and want it to solve some other problem. When used for the purpose intended, setuptools is quite good. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specifying version information once
A. Cavallo wrote: No, I would say “too much” here is importing a module not guaranteed to be installed, for creating something as simple as a version. I'd prefer the above to be:: $ cat _version.py # This is an auto-generated file. Use magicbean to update. version = 0.9.33 $ Or my preferred way: $cat foobar/__init__.py __version__ = 0.9.33 and import foobar should not trigger code execution anyway IMHO so $ python -c 'import foobar; print foobar.__version__' 0.9.33 That doesn't work in all cases. Your example is of an external query of the version of an installed module. You also need to query the version -before- it is installed (during the packaging phase) on sys.path, and also from -within- the foobar module itself. Your code does not handle those cases. An attempt to 'import foobar; print foobar.__version__' from a setup.py inside foobar won't find foobar. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout/setuptools setup_requires and test_requires
Chris Withers wrote: Sorry, still not sure I follow you... Are you saying that any packages specified in setup_requires and test_requires will be installed in the path of the package that specified them rather than the same location as all other eggs? If so, that would seem to be a bug, no? Nope, a feature. Packages needed -only- for packaging (setup_requires) and testing (test_requires) should not be generally installed. They are installed in a special place for use just during those particular phases of operation. No need to have them around in production. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 376 - critique - why more sub-directories within python aren't needed...
David Lyon wrote: I seriously question the need for a new .EGG_INFO directory... Given that a typical site-packages directory doesn't have many files in it anyway. Typically, it will contain 20+ project/package subdirectories or .EGG files/directories. ;-) Mine has 213 files/directories, and I use virtualenv and buildout to keep the number down. I also use Gentoo Linux which has distro-packaged lots of Python-based apps, leading to more entries in site-packages than might otherwise be expected in a virtualenv/buildout environment. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deprecate MANIFEST.in
David Cournapeau wrote: No other distribution mechanism that I know of uses magic to detect which files to distribute. Neither debian tools, nor rpm tools, nor autotools. Not being explicit about files for distribution is a terrible idea; a packaging tool should be explicit, to avoid as much as possible to do something unexpected. While true that the rpm tools do not magically detect which files to distribute, there are many that wish it did and have tried various approaches to adding it. The following except from the RPM book speaks of the various, failed approaches. Still it speaks of a essential need on the part of developers to automate this aspect of packaging, even if the perfect solution has not yet been found. If we're going to be explicit on what to include, I would argue we should NOT allow wildcards, since in subsequent uses of sdist by another person, it could cause files to be included that otherwise would not be there. The only way to be sure precisely the same set of files is always included is to explicitly list them one by one. A big pain. --- cut here --- excerpt from Maximum RPM book How Do You Create the File List? Since RPM automates so many aspects of software installation, it's easy to fall into the trap of assuming that RPM does everything for you. Not so! One task that is still a manual process is creating the file list. While it may seem at first glance, that it could be automated somehow, it's actually a more difficult problem than it seems. Since the majority of an application's files are installed by its makefile, RPM has no control over that part of the build process, and therefore, cannot automatically determine which files should be part of the package. Some people have attempted to use a modified version of install that logs the name of every file it installs. But not every makefile uses install, or if it does, uses it sporadically. Another approach tried was to obtain a list of every file on the build system, immediately before and after a build, and use the differences as the file list. While this approach will certainly find every file that the application installed, it can also pick up extraneous files, such as system logs, files in /tmp, and the like. The only way to begin to make this approach workable would be to do nothing else on the build system, which is highly inconvenient. This approach also precludes building more than one package on the system at any given time. At present, the best way to create the file list is to read the makefile to see what files it installs, verify this against the files installed on the build system, and create the list. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [issue67] You can't tell easy_install not to talk to the network
Glyph Lefkowitz wrote: New submission from Glyph Lefkowitz gl...@twistedmatrix.com: If I want to use easy_install for nice dependency resolution, there's no way (that I can figure out, at least) to tell it never talk to the network, use only this local directory where you can find dependencies. -- messages: 260 nosy: glyph priority: feature status: unread title: You can't tell easy_install not to talk to the network Sure you can. To tell it to pull ONLY from a local directory, use: easy_install -H None -f /tmp/cachedir SQLObject and to *generate* that directory when you -are- online, use: easy_install -zmaxd /tmp/cachedir SQLObject It's the '--allow-hosts=None' or '-H None' that does the magic. BTW my slides and handouts from the tutorial I ran at PyCon about distutils, setuptools and buildout are linked to at the bottom of the page: http://us.pycon.org/2009/tutorials/schedule/1PM3/ It's broken into four sub-topics of virtualenv, distutils, setuptools and buildout and the handout text includes many tips and techniques that are not often well known like the one you asked about. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] GUI for Python package Management
Ben Finney wrote: (Could you please set your name in your From field? Thanks.) david.l...@preisshare.net writes: From the other perspective, it is a fairly trivial task for the gui to track what packages it has installed - through it's own configuration file and generate a manifest from that. You mean, have packages tracked separately depending on which program was used to install them? That seems like a recipe for madness. Agreed, manifest generation -has- to be done in the backend. Also any such tracking of what is installed has to address the use case of sandboxes into which different things are installed concurrently. And it needs to handle that some of those sandboxes are connected via wormholes to the system Python site-packages. I'm talking about the 'virtualenv' command and its command-line option whether to let the system site-packages be available as a last resort when searching for packages. And having experience with Red Hat RPMs, you also should have a tool to rebuild the registry of packages, say from .egg metadata, should the registry be corrupted. Not an easy problem at all to get right. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Annoucing distribute project
Phillip J. Eby wrote: At 10:54 PM 9/24/2008 +0200, Christophe Combelles wrote: there are a LOT of other improvements that are already wanted, such as support for other (d)vcs, Note, by the way, that the reason most of the items listed as feature or wish on the tracker are in those statuses are because nobody has proposed a design for critique. Usually, it's just wishes for magic ponies (e.g. uninstall, post-install hooks, etc.) without any discussion of how to deal with the edge cases, interactions, or other negative consequences of said features. (magic pony manure?) I sincerely hope that things will speed-up with this fork. I imagine they might speed up, but likely at the cost of stability. If anybody knew enough to be able to add these features in a safe way, then they knew enough to be able to contribute patches to setuptools (after first proposing how they would handle all the nasty edge cases). I agree, in that we must not add features without careful thought as to their impact and cross-platform support. Python is a conservative community and we've seen value in the role of an gatekeeper who sees the bigger picture, as Guido and others for other projects. Of course, a gatekeeper who is too-strict isn't good either - there must be balance. But to the distribute project, because of the nature of standardizing package technology, running this is a huge responsibility. Please do not start adding all features asked for without considering their tradeoffs. One issue with something like setuptools is that packaging APIs get immortalized in setup.py files that are not easy to go back and change, so the APIs must carry heavy backward compatibility baggage. You may want to consider using PEPs as a basis for discussing change requests. They're not just for the core Python language. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Issues Tracker for Setuptools
We now have a specific issues tracker for setuptools at: http://bugs.python.org/setuptools/ and [EMAIL PROTECTED] (for email issue postings) using the Roundup tracker software also used for Python bug tracking. Note that the trailing slash is NOT optional. For now the tracker echoes bug reports onto the distutils list, for community awareness. If this turns out to be a problem, I'll create a separate distutils-bugs list but wanted to avoid having folks sign up for yet-another-list if I would avoid it. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
Paul Moore wrote: On 20/03/2008, zooko [EMAIL PROTECTED] wrote: I'll chime in here, too. I really want to like setuptools/easy_install, but I don't. I'll try to be specific in my reasons, in the hope that they can be addressed. I know some of these are known about, but one of my meta-dislikes of setuptools is that known issues never seem to get addressed (I know, patches accepted, but I haven't got the time either...) Clearly explained problems with the existing arrangement is valuable as well as patches. Thanks for taking the time to help us see your viewpoint. 1. No integration with the system packager (Windows, in my case). If I do easy_install nose, then nose does not show up in add/remove programs. That significantly affects the way I manage my PC. Part of this stems from stretching of the original mission of setuptools, to install modules for Python, into a general-purpose application installation tool. The buildout tool is more suited for application installation, although not ideal yet. In your scenario, what happens when one egg pulls in another and another, until you have a hundred entries in your add/remove menu? And you don't understand the inter-relationship of those so you cannot do a clean uninstall? Similarly, or what do you want to appear in that add/remove menu when you are using independent sandboxes with various applications in them, some of which are accessible only to specific users who are not admins on that box? 3. The pkg_resources documentation (in particular, that's the one I've tried to follow) is extremely hard to read. Partly this is just style, but it's partly because it is couched in very unfamiliar terms (distributions, working sets, interfaces, providers, etc). It's also *huge*. A tutorial style overview, supported by API detail, would be far better. We'll get better docs. Of course, that module contains roughly five different sets of functionality, some of which can be used unrelated to the others so it's not just one API. 4. Hard to use with limited connectivity. At work, I *only* have access to the internet via Internet Explorer (MS based proxy). There are workarounds, but ultimately download an installer, then run it is a far simpler approach for me. This is hard to address using independent eggs re setuptools but fits buildout which provides for deployment of a set of related eggs as a single entity. I'll add it as a use case and see what we can do though. 5. Auto-discovery doesn't always work. I'm sorry, I really can't recall the example at the moment, but sometimes easy_install says it can't find a package I *know* is available. I've hit a few of these myself, although it wasn't an issue with the auto-discovery mechanism but rather quality problems with PyPI itself. Some of the eggs only had binary distributions provided, and they were not for my platform so couldn't be used. Better error messages in this area would help, with encouragement to nag the original author to provide better data on PyPI. 6. Splitting the community. Windows users rely heavily on binary installers (at least, I do). We're starting to get a situation where some projects provide .egg files, and some provide traditional (bdist_wininst/bdist_msi) installers. This is bad. One way to do it, and all that :-) Reporting and author nagging facilities built into PyPI could help encourage more consistent behavior. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Dave Peterson wrote: I agree. Let's get that setuptools wiki started and start documenting some of these ideas as a roadmap so that anyone who wants to help out has an idea of what to work on, or factor into what they're currently working on. Anyway, since Enthought is already scratching, I'm fine with the idea of building a standard way to do it that is driven by human-readable data. We just need to setup the process to allow that to happen. So far I haven't seen any responses from you in regards to the setup of an issue/patch tracker, wiki, process to open up the number of commiters, etc. that gives me any confidence I'm not heading off down the wrong path somehow. Perhaps I'm too cautious? Dave, I'm in the process of getting a tracker for setuptools, and I'll work on the wiki shortly, although we have the PackagingBOF wiki for idea collection at the moment. Give me a couple of days, including travel from PyCon. I'm fired up to make this happen. Actually, I wonder if instead of trying to enhance setuptools for post-install, if maybe we should be looking at buildout recipes and maybe having a way for setuptools dependencies to point to buildout specs. IIRC, buildout specs can be remotely retrieved from a single URL, too. I'll need to read up more on buildout to understand this, but my understanding was that buildout was not something a user ran to install an app, but rather something the developer ran to build and publish an app. The end result of a 'production' buildout is to generate a large tarball or rpm that included everything, right? If so, this goes directly against what Enthought was aiming for, which was to allow delivery of bug-fixes and minor updates in a large app by downloading only smaller units instead of a huge monolithic re-install of everything. Your view of a fine-grained application bundle with the ability to dynamically download updated eggs without re-pulling the entire thing is an interesting contrast to Paul's view of a more monolithic application for easier add/remove/uninstall completeness. Supporting both usage models is going to be a challenge but I think is feasible with some thought. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: I'm actually happy to hear that there's this much energy available -- hopefully some of it can be harnessed towards positive solutions. When I began developing setuptools, I often asked for the input of packagers, developers, etc., through the distutils-sig... and was met with overwhelming silence. So the fact that there is now a group of people who are ready to work for some solutions seems like a positive change, to me. I can appreciate how frustrating silence is when you call for input. Let's see if we can keep the volunteer energy going this time around. It's hard to make design decisions regarding itches you don't personally have, and which other people won't help scratch. Unfortunately, a lot of the proposals from packaging system people have been of the form of, fix this for us by breaking things for other people. Not all of them, though. Many have been very helpful, contributing troubleshooting help and good patches. That some of those good patches took nearly a year to get into setuptools (some from Fedora just got into 0.6c8 that were sent to me almost a year ago) is because I'm the only person reviewing setuptools patches, and I've spent only a few days in the last year doing focused development work on setuptools (as opposed to answering questions about it on the SIG). It's never a good thing when people's patches sit around, regardless of where they come from. But that's not the same thing as *rejecting* the patches. I and others appreciate your call for more patches on various topics. However a long delay in applying them will discourage contribution. Are you open to giving certain others patch view/commit privileges to setuptools? I'd be willing to help out, and keep a carefully balanced hand in what is accepted. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] The Breaking of distutils and PyPI for Python 3000?
As I'm digging into packaging issues here at PyCon, a couple of Python 3000 related matters occur to me. As I'm new to the Python 3000 development, if these have already been addressed in prior discussions, I apologize for your time. 1. What is the plan for PyPI when Python 3.0 comes out and dependencies start getting satisfied from distribution across the great divide, e.g. a 3.0-specific package pulls from PyPI a 2.x-specific package to meet some need? Are there plans to fork PyPI, apply special tags to uploads or what? While binary distributions are tagged with the Python version, source distributions are not. And of course a dependency expression as it stands today for SomePackage 2.4 may pull 3.0 to satisfy it. 2. There have been attempts over the years to fix distutils, with the last one being in 2006 by Anthony Baxter. He stated that a major hurdle was the strong demand to respect backward compatibility and he finally gave up. One of the purposes of Python 3.0 was the freedom to break backward compatibility for the sake of doing the right thing. So is it now permissible to give distutils a good reworking and stop letting compatibility issues hold us back? -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Request for Input re Packaging
In researching the state of packaging, I've been reading the archives and all the bug reports filed against distutils. I'd like though to get some examples of particularly troublesome uses of setup.py, to pull together and propose some changes to make their use case a bit easier. So far such cases I've been made aware of are Twisted, numpy and SciPy. If you know of a tough case where the developer had to jump through hoops to make it work, please point me to it. I'd also like to get suggestions of improvements to PyPI, which I've not seen much discussion about. A few I've collected are: - move to https/ssl - improvements re package signing - internal parsing/aggregation of metadata for better queries, and to stop using the filename for version/platform/etc. information. - moving of requirements logic from client into PyPI, where it has db access to the dependency, resolves what packages are needed and delivers a list back to the client for prompting the user for permission, similar to how yum interacts today. - a db lint-picking walker that looks for problems on PyPI, such as binary distros w/o a source distro, lack of binaries for those platforms often without compilers, failure to provide a link to a version repo for use with project==dev. - some auto-generated reports of access statistics and the mix of distutils vs setuptools, those who registered w/o uploading, and perhaps if we get a new classifier assigned, some idea of Python 2.x vs 3.x packages. Last, some of the issues with distutils/setuptools can be solved with zc.buildout. If you have found zc.buildout lacking, please tell me where it fell short so we can see if anything can be done. Thanks for your involvement, -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Marius Gedminas wrote: On Mon, Mar 17, 2008 at 08:37:30PM -0400, Phillip J. Eby wrote: At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote: People also want a greater variety of file_finders to be included with setuptools. Instead of just CVS and SVN, they want it to comprehend Mercurial, Bazaar, Git and so forth. Did you point them to the Cheeseshop? There are plugins already available for all the systems you mentioned, plus Darcs and Monotone. If you mean included as in bundled, this doesn't make a whole lot of sense to me. They knew there were plugins out there, of various quality and availability but wanted them bundled. ;-) It's a pain to track them down. Perhaps if the RPM format were broken out from setuptools, as the inclusion of some formats leads them to believe the set is just incomplete, not intentionally sparse. I'd think that if you're using setuptools as a developer (the only reason you need the file finders, since source distributions include a prebuilt manifest), you'd not have a problem saying easy_install setuptools-git or adding a setup_requires='setuptools-git' line to your setup.py. (Although the latter would only be needed for *development*, not deployment.) setup_requires looks like a solution, but it requires extra attention from the developers who write the setup.py. Writing a setup.py is already quite complicated -- I usually end up copying an existing one and modifying it. As a compromise, of making new formats easily available but not bundled, and not requiring special action within setup.py, setuptools could treat --formats=dpkg as an implicit setup_requires= and pull it from PyPI. And the --list-formats option could query PyPI for the possibilities, just as --list-classifiers does today. If would require a few standards in keywording/classifying those format eggs but we already need those standards for other projects, such as locating recipes for buildout and plugins for trac. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
Guido van Rossum wrote: On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore [EMAIL PROTECTED] wrote: I'm +lots on someone giving a clear explanation of the meaning and interrelationship of the various terms involved in this discussion (setuptools, easy_install, pkg_resources, eggs, package managers as distinct from setuptools, etc etc) so that the discussion gets some much-needed clarity :-( Right. But finding someone who can explain all this is apparently hard. All the owners of package managers seem busy... In preparing for my PyCon 2008 tutorial on eggs and buildout, I spent three full-time weeks carefully going over sources for distutils, setuptools and buildout to discover those aspects not documented. I can explain how they work, although I'm not sure this is the correct forum. I'd like to first offer my slides from my tutorial, 150 of them with detailed handout notes on many of them. http://wiki.python.org/moin/buildout/pycon2008_tutorial I'm happy to answer questions after that. I'm in the Hanada B room for OLPC at PyCon and on IRC #pycon. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Capsule Summary of Some Packaging/Deployment Technology Concerns
I was in a Packaging BoF yesterday and, although not very relevant to the packager bootstrap thread, Guido has asked me to post some of the concerns. The BoF drew about 15 people, many of whom were packagers for Red Hat, Ubuntu and such. Everyone had strong expressions of frustration with the status quo and most had tried to resolve their issues but had their patches rejected. I am not taking either side and whether those rejections were justified I cannot say, but the general feeling of their concerns intentionally not being addressed isn't healthy. Several had abandoned setuptools, deeming it a failed solution and others called for a fork. To start, I am not a leader of the group nor do I claim I accurately captured and expressed all their concerns. I apologize to those in the BoF for any misrepresentations. 1. Many felt the existing dependency resolver was not correct. They wanted a full tree traversal resulting in an intersection of all restrictions, instead of a first-acceptable-solution approach taking now, which can result in top-level dependencies not being enforced upon lower levels. The latter is faster however. One solution would be to make the resolver pluggable. 2. People want a solution for the handling of documentation. The distutils module has had commented out sections related to this for several years. 3. A more flexible internal handing of the different types of files is needed. Currently the code, data, lib, etc. files are aggregated at build time and people would like them to be kept separate until install/packaging time. They also want greater flexibility in the kinds of files identified for packaging. There is currently a single plugin entrypoint for file_finding, so people have resorted to abusing the setuptools function find_packages() again and again with different include/exclude args. A solution is to expand the set of entrypoints into finer grained categories. They also want a way to expand the set of categories rather than a fixed set, which can be easily done with entrypoint groups and names. People also want a greater variety of file_finders to be included with setuptools. Instead of just CVS and SVN, they want it to comprehend Mercurial, Bazaar, Git and so forth. 4. They want an uninstall setuptools command. Adding one to remove a specific egg isn't difficult but correctly removing those dependencies that came in with that egg, without breaking later installs can be tricky. This is complicated because there isn't a single global package namespace to manage, when you factor in virtualenv and buildout sandboxes and per-user package areas. This differs from how RPMs and .debs are viewed. 5. There was concern over the .pth mechanism used by setuptools re activation. First, there is a (perceived) performance issue with increasingly adding every ZIP file explicitly onto sys.path. This may or may not be a red herring. The other is the use of a single .pth file to control the list of activated packages. Those who produce distributions would prefer a magic directory into which links to distributions could be dropped, similar to the current best practices for Linux, with /etc/conf.d/, /etc/profile.d/, /etc/xinetd.d/ and so forth. 6. There is a need for more extensibility hooks. People want places to plug in special handling. For example: a) setuptools has a --record option to capture the list of files installed for use by subsequent packaging tools. Some want that list to be available to a setuptools plugin. b) some want hooks for post-build/post-install actions, instead of the current approach of writing a custom build class that handles it all. 7. Many wanted to ability to install files anywhere in the install tree and not just under the Python package. Under distutils this was possible but it was removed in setuptools for security reasons. Custom code can still be written to do this explicitly but this is not popular. Neither setuptools nor distutils has the ability to rename files at install time. A fair question is whether it is the job of setuptools (or any Python packaging solution) to cover all these bases. The risk of not doing the job is that some of those in attendance were rolling their own solutions which do not play well with packages installed using other means, not seeing them. Distutils has intentionally tried to -not- be a general replacement packaging solution, with its support of the bdist command for various platform-specific distribution formats. We should continue not trying to replace platform-specific packaging technologies but perhaps improve our control of their creation. As mentioned, some of these concerns can be resolved by adding customization-pressure-release entrypoints to setuptools, and some can be handled with much
Re: [Distutils] Question re zc.buildout Name Expansion Rules
Jim Fulton wrote:
On Jan 29, 2008, at 9:52 AM, Jeff Rush wrote:
Suppose I have something like:
[FreeTDS_installation]
recipe = zc.recipe.cmmi
url = blah
odbcinst_ini =
[FreeTDS]
Driver = ${FreeTDS_installation:location}/lib/libtdsodbc.so
1) Since the variable expansion is referring to an attribute of its
own .cfg section, is there a way to omit the section name?
Something like (but this doesn't work):
Driver = ${location}/lib/libtdsodbc.so
It just seems extra work to keep putting my own section name
in, and is dangerous if I cut/paste lines to move about.
Yup. I do plan to add a syntax for this.
Thanks.
2) The initial case above fails to parse because at the time
the ${} expression is reached during read-in time, the part
section has not yet been added to the global registry of
part sections. Hence I get a FreeTDS_installation
undefined error.
It just seems there ought to be a way for a part section
to refer to its own location attribute.
I agree that this would be helpful. (This is going to be tricky to
implement, but that's a separate issue.) Would you mind adding a feature
request at:
https://blueprints.launchpad.net/zc.buildout/
I have done so.
-Jeff
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Question re zc.buildout Name Expansion Rules
In using buildout, I've come across a couple of limitations and I'm wondering
if there are workarounds.
Suppose I have something like:
[FreeTDS_installation]
recipe = zc.recipe.cmmi
url = blah
odbcinst_ini =
[FreeTDS]
Driver = ${FreeTDS_installation:location}/lib/libtdsodbc.so
1) Since the variable expansion is referring to an attribute of its
own .cfg section, is there a way to omit the section name?
Something like (but this doesn't work):
Driver = ${location}/lib/libtdsodbc.so
It just seems extra work to keep putting my own section name
in, and is dangerous if I cut/paste lines to move about.
2) The initial case above fails to parse because at the time
the ${} expression is reached during read-in time, the part
section has not yet been added to the global registry of
part sections. Hence I get a FreeTDS_installation
undefined error.
It just seems there ought to be a way for a part section
to refer to its own location attribute.
Thanks for any tips,
-Jeff
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
