Re: [Python-Dev] Packaging and binary distributions for Python 3.3
Hi Paul, Thanks for raising this during the development phase. > I see that the Packaging documentation is now more complete (at least > at docs.python.org) - I don't know if it's deemed fully complete yet, > but I scanned the documentation and "Installing Python Projects" looks > pretty much converted (and very good!!), but "Distributing Python > Projects" still has quite a lot of distutils-related text in, and I > need to read more deeply to understand if that's because it remains > unchanged, or if it is still to be updated. The basic structure is in place (distributing/installing/library reference), but the docs are far from up-to-date. I have nearly finished a first patch that fixes a ton of markup issues and updates distutils idioms (setup.py snippets e.g.) to packaging ones (setup.cfg snippets, using pysetup, etc.), it’s already a thousand changed lines. Then I will work on another patch to move things around, consolidate, expand and rephrase. See http://bugs.python.org/issue12779 if you want to follow along and review patches. > But one thing struck me - the "Installing Python Projects" document > talks about source distributions, but not much about binary > distributions. This is inherited from distutils docs, not a deliberate choice. We just haven’t thought much, if at all, about binary distributions. > On Windows, binary distributions are significantly more important than > on Unix, because not all users have easy access to a compiler, and > more importantly, C library dependencies can be difficult to build, > hard to set up, and generally a pain to deal with. Are there that many distributions with extension modules? sdists should work well even on Windows for pure Python projects. > I don't know what format packaging favours. As a direct distutils descendant, can create bdist_wininst and bdist_msi. For installing, I was not aware of the problem you reported (“does not interact well with pysetup”); can you give more info? I’m guessing it boils down to the fact that Windows binary installers are meant to be clicked by users, not managed with command-line tools. IIRC the current behavior in pysetup is to favor source distributions, but bdists should probably be favored for non-pure distributions on Windows. > So there will be a need for a pysetup-friendly binary format. > I assume that the egg format will fill this role - or is that not the > case? What is the current thinking on binary distribution formats for > Python 3.3? First, we don’t want to include wholesale support for setuptools eggs in packaging. We have options to support egg/egg-info metadata in the PEP 376 implementation (packaging.database), because we need that to provide a useful tool for users and help them switch, but eggs are another matter. After all, if setuptools and then pkg_resources were turned down for inclusion in Python 2.5, it’s not now that we have packaging that we’ll change our mind and just bless eggs. What we can do however is to see what bdist_egg does and define a new bdist command inspired by it, but without zipping, pkg_resource calls, etc. > The main reason I am asking is that I would like to write an article > (or maybe a series of articles) for Python Insider, introducing the > new packaging facilities from the point of view of an end user with > straightforward needs (whether a package user just looking to manage a > set of installed packages, or a module author who just wants to > publish his code in a form that satisfies as many people as possible). That’s excellent. I too thought about writing something about packaging for that blog, but an outside end-user viewpoint like yours would best match the readership. I can write a shorter piece just for packaging tool developers (i.e. how to use packaging as a library), or you can write that one too and act as a tester for our doc and API. > What I'd hope to do is, as well as showing people all the nice things > they can expect to see in Python 3.3, to also start package authors > thinking about what they need to do to support their users under the > new system. Yes! We need feedback to provide a much better tool than distutils, before the API is locked by backward compatibility rules. I actually wanted to talk about that, so let me take the opportunity. What if we released packaging in Python 3.3 (and distutils2 1.0 on PyPI) as a not-quite-final release? (Something like Python 3.0, which was not considered a real version and not supported as much as the other ones.) The goal would be to expose it to a large range of users to get bug reports and feature requests, but without locking us forever into one API or design, which was the death of distutils a year ago. The idea is not to scare people with warnings that we’ll break APIs on a whim, but that we keep the option to change parts of packaging and release a 2.0 with Python 3.4, with documented changes from 3.3. Opinions? Regards ___ Pytho
Re: [Python-Dev] [Python-ideas] PEP 3101 (Advanced string formatting) base 36 integer presentation type
Le 08/10/2011 17:14, Victor Stinner a écrit : Le 08/10/2011 15:03, Antoine Pitrou a écrit : On Fri, 07 Oct 2011 21:14:44 -0600 Jeffrey wrote: I would like to suggest adding an integer presentation type for base 36 to PEP 3101. I can't imagine that it would be a whole lot more difficult than the existing types. Python's built-in long integers provide a nice way to prototype and demonstrate cryptographic operations, especially with asymmetric cryptography. (Alice and Bob stuff.) Built-in functions provide modular reduction, modular exponentiation, and lots of nice number theory stuff that supports a variety of protocols and algorithms. A frequent need is to represent a message by a number. Base 36 provides a way to represent all 26 letters in a semi-standard way, and simple string transformations can efficiently make zeros into spaces or vice versa. Why base 36 rather than, say, base 64 or even base 80? Base 85 is the most efficient base to format IPv6 addresses! http://tools.ietf.org/html/rfc1924 And Python doesn't provide builtin function for this base! Victor Oops, I answered to the wrong mailing list. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 9 October 2011 08:15, Éric Araujo wrote: >> But one thing struck me - the "Installing Python Projects" document >> talks about source distributions, but not much about binary >> distributions. > This is inherited from distutils docs, not a deliberate choice. We just > haven’t thought much, if at all, about binary distributions. > >> On Windows, binary distributions are significantly more important than >> on Unix, because not all users have easy access to a compiler, and >> more importantly, C library dependencies can be difficult to build, >> hard to set up, and generally a pain to deal with. > Are there that many distributions with extension modules? sdists should > work well even on Windows for pure Python projects. To be honest, I'm not that sure. I tend to find that many of the ones I want to use have binary dependencies, but maybe I'm atypical :-) Looking at my installations, I see: - database drivers (cx_Oracle, in my case) - lxml - pywin32 - pyQT - pyzmq (that's just for playing a bit with IPython, so doesn't really count...) - I've also used in the past PIL mod_python (mod_wsgi more recently) and wxPython, but don't these days because they either aren't available for Python 3, or no binaries are available and building them is a pain. I've hit others in the past, but mainly just in idle hacking, so I don't depend on them as such (and can't really remember which). >> I don't know what format packaging favours. > As a direct distutils descendant, can create bdist_wininst and > bdist_msi. For installing, I was not aware of the problem you reported > (“does not interact well with pysetup”); can you give more info? I’m > guessing it boils down to the fact that Windows binary installers are > meant to be clicked by users, not managed with command-line tools. Precisely that (and nothing really more). The pysetup features for uninstalling packages aren't going to work with bdist_wininst/bdist_msi (that's an assumption, I haven't tried them but I can't see how they would, and it'd certainly be a lot of marginally-useful effort to do even if it were possible). The virtual environment stuff also wouldn't work that well with the installers, because they wouldn't have any way of finding which environments existed to ask where to install to. The same problem exists with virtualenv. (Again this is speculation backed by a small amount of playing with virtualenv, so I may be wrong here). > IIRC the current behavior in pysetup is to favor source distributions, > but bdists should probably be favored for non-pure distributions on Windows. > >> So there will be a need for a pysetup-friendly binary format. >> I assume that the egg format will fill this role - or is that not the >> case? What is the current thinking on binary distribution formats for >> Python 3.3? > First, we don’t want to include wholesale support for setuptools eggs in > packaging. We have options to support egg/egg-info metadata in the PEP > 376 implementation (packaging.database), because we need that to provide > a useful tool for users and help them switch, but eggs are another > matter. After all, if setuptools and then pkg_resources were turned > down for inclusion in Python 2.5, it’s not now that we have packaging > that we’ll change our mind and just bless eggs. What we can do however > is to see what bdist_egg does and define a new bdist command inspired by > it, but without zipping, pkg_resource calls, etc. It may be that the bdist_dumb format would be OK. I haven't checked it out (to be honest, I don't think it's ever been used much). I could have a play with that and see if it did the job (or could be made to). Like you say, eggs have a lot of extra infrastructure that wouldn't be needed here. >> The main reason I am asking is that I would like to write an article >> (or maybe a series of articles) for Python Insider, introducing the >> new packaging facilities from the point of view of an end user with >> straightforward needs (whether a package user just looking to manage a >> set of installed packages, or a module author who just wants to >> publish his code in a form that satisfies as many people as possible). > That’s excellent. I too thought about writing something about packaging > for that blog, but an outside end-user viewpoint like yours would best > match the readership. I can write a shorter piece just for packaging > tool developers (i.e. how to use packaging as a library), or you can > write that one too and act as a tester for our doc and API. Let's see how things go. My goal is to evangelise packaging so that people writing packages I use will create binary builds to save me the effort (:-)) - but anything else I can do, if I have the time, I'm happy to chip in with. >> What I'd hope to do is, as well as showing people all the nice things >> they can expect to see in Python 3.3, to also start package authors >> thinking about what they need to do to support their users under the >> new system. > Ye
Re: [Python-Dev] Bring new features to older python versions
On 08/10/2011 20:38, Maciej Fijalkowski wrote: On Sat, Oct 8, 2011 at 8:35 PM, "Martin v. Löwis" wrote: The first one is about licensing. What I would be doing is basically copy&paste pieces of the python stdlib modules (including tests) and, where needed, adjust them so that they work with older python versions. Would this represent problem? You have a "nonexclusive, royalty-free, world-wide license to ..." "prepare derivative works, distribute, and otherwise use Python alone or in any derivative version," so: no, this is no problem ... "provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee." My second doubt is about morality. Although this might be useful to those people who are forced to use older python versions, on the other hand it might represent an incentive for not upgrading (and there will be python 3.X features as well). Don't worry about that. I'm not sure how many people would be interested in your approach in the first place - if I have to support old versions of Python, I personally just don't use newer features, and don't even have the desire to do so. If I want to use newer features, I decide to drop support for older versions. That I get both with a hack as such a module is just something that I *personally* would never consider (there are other reasons for me to consider hacks like this, such as when supporting multiple versions is just not feasible, but I wouldn't use a hack for convenience reasons). People that do feel the same way as you have probably started their own emulation layers already, so by publishing your emulation layer, it's not getting worse. Regards, Martin Most programs I know have it's own imperfect version of such thing, so I would definitely use it. Not everyone can drop support for older versions of python at will. Ditto. unittest2 and the mock test suite both have a subset of this in for some of the newer Python standard library features they use (plus putting back into Python 3 some of the things that disappeared like callable and apply). All the best, Michael Foord Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On Sun, Oct 9, 2011 at 3:15 AM, Éric Araujo wrote: > After all, if setuptools and then pkg_resources were turned > down for inclusion in Python 2.5, it’s not now that we have packaging that we’ll change our mind and just bless eggs. Actually, that's not what happened. I withdrew the approved-by-Guido, announced-at-PyCon, and already-in-progress implementation, both because of the lack of package management features, and because of support concerns raised by Fredrik Lundh. (At that time, the EggFormats doc didn't exist, and there were not as many people familiar with the design or code as there are now.) For the full statement, see: http://mail.python.org/pipermail/python-dev/2006-April/064145.html (The withdrawal is after a lot of background on the history of setuptools and what it was designed for.) In any case, it definitely wasn't the case that eggs or setuptools were rejected for 2.5; they were withdrawn for reasons that didn't have anything to do with the format itself. (And, ironically enough, AFAIK the new packaging module uses code that's actually based on the bits of setuptools Fredrik was worried about supporting... but at least there now are more people providing that support.) What we can do however > is to see what bdist_egg does and define a new bdist command inspired by > it, but without zipping, pkg_resource calls, etc. > Why? If you just want a dumb bdist format, there's already bdist_dumb. Conversely, if you want a smarter format, why reinvent wheels? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On Sun, Oct 9, 2011 at 9:31 PM, PJ Eby wrote: ... >> What we can do however >> is to see what bdist_egg does and define a new bdist command inspired by >> it, but without zipping, pkg_resource calls, etc. > > Why? If you just want a dumb bdist format, there's already bdist_dumb. > Conversely, if you want a smarter format, why reinvent wheels? Just to make sure we're on the same page here. PEP 376 provide the installation format for the 'future' -- http://www.python.org/dev/peps/pep-0376/ Introducing back another *installation* format would be against the goal we've initially had with PEP 376 : have a single installation format all tools out there would support, for the sake of standardization of interoperability. (and for consumers in other communities) So, while 'eggs' are interesting as plugins for a given application (that was the initial use case right ?), please do not consider them as an installation format for Python. Now for a binary archive, that would get installed ala PEP 376, why not ? I'd just be curious to have someone list the advantage of having a project released that way besides the "importable as-is" feature. Cheers Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 9 October 2011 20:47, Tarek Ziadé wrote: > On Sun, Oct 9, 2011 at 9:31 PM, PJ Eby wrote: > ... >>> What we can do however >>> is to see what bdist_egg does and define a new bdist command inspired by >>> it, but without zipping, pkg_resource calls, etc. >> >> Why? If you just want a dumb bdist format, there's already bdist_dumb. >> Conversely, if you want a smarter format, why reinvent wheels? > > Just to make sure we're on the same page here. > > PEP 376 provide the installation format for the 'future' -- > http://www.python.org/dev/peps/pep-0376/ [...] > Now for a binary archive, that would get installed ala PEP 376, why > not ? I'd just be curious to have someone list the advantage of having > a project released that way besides the "importable as-is" feature. Agreed. I'm not looking at a new binary installed format - PEP 376 covers this fine. What I am looking at is how/if users without a compiler can get a file that contains all the bits they need to install a distribution. My expectation would be that the user would type pysetup install some_binary_format_file.zip and have that file unpacked and all the "bits" put in the appropriate place. Basically just like installing from a source archive - pysetup install project-1.0.tar.gz - but skipping the compile steps because the compiler output files are present. That may need some extra intelligence in pysetup if it doesn't have this feature already (I sort of assumed it would, but that's likely because of my interest in binary formats) but if not it shouldn't be hard to add - just unzip the bits into the right place, or something similar. As regards the format, bdist_dumb is about the right level - but having just checked it has some problems (which if I recall, have been known for some time, and are why bdist_dumb doesn't get used). Specifically, bdist_dumb puts the location of site-packages ON THE BUILD SYSTEM into the archive, making it useless for direct unzipping on a target system which has Python installed somewhere else. See the following for an example: PS D:\Data\python-sample> unzip -l .\dist\PackageName-1.0.win32.zip Archive: ./dist/PackageName-1.0.win32.zip Length EAs ACLsDate TimeName --- 6656 0 0 09/10/11 20:56 Apps/Python32/Lib/site-packages/hello.pyd 208 0 0 09/10/11 20:56 Apps/Python32/Lib/site-packages/PackageName-1.0-py3.2.egg-info - - --- 6864 0 0 2 files It should be simple enough to fix this in bdist_dumb, although a new name might be needed if backward compatibility of the old broken format matters... If pysetup doesn't have support for binary archives at all, then I'm happy to take a look at what might be involved in adding this. But I don't know the code at all, and I have little time, so I'm rather hoping I won't need to... Paul. PS The problem for me is that if pysetup only handles source builds, it's STILL annoyingly incomplete for my requirements (and possibly many Windows users') So I feel this is a hole that needs to be filled before 3.3 is released, or pysetup won't be suitable as "the way to do all packaging in Python". ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 9 Oct 2011, at 21:14, Paul Moore wrote: > On 9 October 2011 20:47, Tarek Ziadé wrote: >> On Sun, Oct 9, 2011 at 9:31 PM, PJ Eby wrote: >> ... What we can do however is to see what bdist_egg does and define a new bdist command inspired by it, but without zipping, pkg_resource calls, etc. >>> >>> Why? If you just want a dumb bdist format, there's already bdist_dumb. >>> Conversely, if you want a smarter format, why reinvent wheels? >> >> Just to make sure we're on the same page here. >> >> PEP 376 provide the installation format for the 'future' -- >> http://www.python.org/dev/peps/pep-0376/ > [...] >> Now for a binary archive, that would get installed ala PEP 376, why >> not ? I'd just be curious to have someone list the advantage of having >> a project released that way besides the "importable as-is" feature. > > Agreed. I'm not looking at a new binary installed format - PEP 376 > covers this fine. What I am looking at is how/if users without a > compiler can get a file that contains all the bits they need to > install a distribution. Just to agree with Paul, a typical Windows Python user will not be able to install a non-binary version of a distribution that includes C code. Even on the Mac it is common to distribute binaries. Michael > > My expectation would be that the user would type pysetup install > some_binary_format_file.zip and have that file unpacked and all the > "bits" put in the appropriate place. Basically just like installing > from a source archive - pysetup install project-1.0.tar.gz - but > skipping the compile steps because the compiler output files are > present. That may need some extra intelligence in pysetup if it > doesn't have this feature already (I sort of assumed it would, but > that's likely because of my interest in binary formats) but if not it > shouldn't be hard to add - just unzip the bits into the right place, > or something similar. > > As regards the format, bdist_dumb is about the right level - but > having just checked it has some problems (which if I recall, have been > known for some time, and are why bdist_dumb doesn't get used). > Specifically, bdist_dumb puts the location of site-packages ON THE > BUILD SYSTEM into the archive, making it useless for direct unzipping > on a target system which has Python installed somewhere else. > > See the following for an example: > > PS D:\Data\python-sample> unzip -l .\dist\PackageName-1.0.win32.zip > Archive: ./dist/PackageName-1.0.win32.zip > Length EAs ACLsDate TimeName > --- > 6656 0 0 09/10/11 20:56 > Apps/Python32/Lib/site-packages/hello.pyd > 208 0 0 09/10/11 20:56 > Apps/Python32/Lib/site-packages/PackageName-1.0-py3.2.egg-info > - - --- > 6864 0 0 2 files > > It should be simple enough to fix this in bdist_dumb, although a new > name might be needed if backward compatibility of the old broken > format matters... > > If pysetup doesn't have support for binary archives at all, then I'm > happy to take a look at what might be involved in adding this. But I > don't know the code at all, and I have little time, so I'm rather > hoping I won't need to... > > Paul. > > PS The problem for me is that if pysetup only handles source builds, > it's STILL annoyingly incomplete for my requirements (and possibly > many Windows users') So I feel this is a hole that needs to be filled > before 3.3 is released, or pysetup won't be suitable as "the way to do > all packaging in Python". > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On Sun, Oct 9, 2011 at 4:14 PM, Paul Moore wrote: > As regards the format, bdist_dumb is about the right level - but > having just checked it has some problems (which if I recall, have been > known for some time, and are why bdist_dumb doesn't get used). > Specifically, bdist_dumb puts the location of site-packages ON THE > BUILD SYSTEM into the archive, making it useless for direct unzipping > on a target system which has Python installed somewhere else. > I don't know about the case for packaging/distutils2, but I know that in original distutils, you can work around this by making bdist_dumb call the install commands with different arguments. That is, it's a relatively shallow flaw in bdist_dumb. bdist_wininst, for example, is basically a zipped bdist_dumb with altered install arguments and an .exe header tacked on the front. (Along with a little extra data crammed in between the two.) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
