Re: Amend Debian Python Proposal to Include More Python Metadata?
> On Jan 22, 2016, at 7:18 PM, Scott Kitterman wrote: > > The Zen of Python says, among other things, "There should be one-- and > preferably only one --obvious way to do it". Build systems seem to me like a > great place to apply that. We have a sliding scale of complexity in what a project needs from it’s build system that generally breaks down into a few major classes of projects: 1) Projects that are pure python, single source, whose only real “compilation” is shuffling files to the correct location. 2) Projects that are pure python, but require some sort of generation step as part of the build process (2to3, etc). 3) Projects that have some basic C code that needs to be compiled, but which doesn’t link against anything special besides Python itself. 4) Projects that have some basic C code that needs to be compiled, but which links against other libraries like OpenSSL, libpq, etc. 5) Projects that have some basic C code, that needs to be compiled, that links against other libraries, and needs to be able to conditionally link against different libraries based on the capabilities and what is available in the system. 6) Extremely complex projects that need to link against many different libraries, possibly hard to build, possibly not C (e.g., Numpy with it’s blas libraries, Fortran, etc). The problem with a one size fits all solution is that it’s very difficult to actually cover all of these cases in a way that is not horrible to actually use for each particular case. For an example, there is currently a build system called flit which doesn’t support anything but building straight to wheels (because we don’t have any sort of sdist 2.0 or anything yet). It doesn’t attempt to solve anything but the first class of users up there, and because of that it is able to create a very simple and easy to use packaging experience for authors, you just add a __version__ = “the version” to the top level of your package, and then drop in a flit.ini that looks something like: [metadata] module=foobar author=Sir Robin author-email=ro...@camelot.uk home-page=http://github.com/sirrobin/foobar # If you want command line scripts, this is how to declare them. # If not, you can leave this section out completely. [scripts] # foobar:main means the script will do: from foobar import main; main() foobar=foobar:main And that’s it. You’re done. From there flit can do the rest of the work for you because it didn’t need to concern itself with trying to work on anything complex. In addition to the above types of problems, you also have other things like what the “source of truth” is for your metadata. A common thing that people want is to be able to not have to duplicate their version in multiple locations (sometimes that even extends to the tags in the version control), however it’s not currently possible to do that in a particularly easy way. You have systems like pbr, setuptools_scm, versioner, etc that all do it but which all rely on some level of terribleness to deal with the sort of weird inverted control flow we have. Once you get to all of the possible options for things people reasonably want to do, you quickly end up in a place where the only reasonable solution is the full complexity of a programming language like what we have in setup.py. However, that has it’s own problems which we’ve discovered over two decades of doing that :) It tends to end up with people doing badly tested adhoc code that they cargo cult from project to project and when there’s a problem there ends up being very little understanding why some code did that since it had been copy/pasted around so much. Pip has to go to a lot of effort to try and work around common mistakes people make in their setup.py files (like, depending on it being invoked with the project root as CURDIR) which most people will just never notice their slightly broken setup.py. So basically, by allowing multiple build systems we will enable a world where your bog standard pure python project can get an extremely simple tool/ux and projects with crazy hard requirements like Numpy can get something more complex, without the two groups of users fighting each other over simplicity + limited vs complexity + powerful. In the end, I think it’s likely we’ll end up with 2-3 really popular build tools, one that is for the complex projects, one that is for the simpler projects with some basic C needs, and *maybe* one that is for pure python (but that may be able to be handled by the basic C needs one) though there will be a long tail I’m sure. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Request to Join
On 21 January 2016 at 07:10, Harlan Lieberman-Berg wrote: > I'd like to request to join this team to help maintain backports of the Let's > Encrypt dependencies, especially python-sphinx. > > I've read and agree to the Python Modules Team Policy. My Alioth username is > hlieberman-guest. I can't accept your request (not a DPMT admin), but I think you'd be a fine addition to the team (having worked with you previously). :) ♥, - Tianon 4096R / B42F 6819 007F 00F8 8E36 4FD4 036A 9C25 BF35 7DD4
Re: Amend Debian Python Proposal to Include More Python Metadata?
On January 22, 2016 6:27:08 PM EST, Donald Stufft wrote: > >> On Jan 22, 2016, at 6:04 PM, Scott Kitterman >wrote: >> >> On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote: >> ... >>> We already have an option like this, the —root option which will >just append >>> a different prefix to all of the installation paths. So essentially >instead >>> of invoking ``python setup.py install —root /tmp/something/`` which >is what >>> I think you’re doing now, you’d do ``pip install —root >/tmp/something/ >>> —no-deps .`` and it’d just work similarly to what you have now, >except pip >>> would be responsible for interacting with the Python build system. >> ... >> >> Why would pip interacting with the Python build system instead of >setuptools >> be better? >> >> Scott K > >setuptools *is* the build system right now (or distutils if you’re >still using >that). So the benefit to Debian right now would be: We maintain the >distutils/setuptools hacks to make them suck less so you don’t have to! > >Longer term, we want to enable people to go hog wild and invent their >own >build systems instead of trying to do it all one size fits all with >everyone >miserable because it doesn’t solve anyones problems nicely. At that >point the >benefit to Debian becomes that instead of having to implement the >standard >build tool interface that these tools implement (Which will most likely >be >geared towards produces wheels, not towards putting files on disk in an >installed location) Debian can just let us maintain that bit of code >too. > >Of course, if Debian would prefer to interact directly with these >tools, it’s >totally fine to do that. The whole point of us doing this work and >trying to >define formats and APIs and not bless implementations is so anyone who >wants >to can slot in their own tool in the process instead of being forced to >use >some blessed tool. You probably expected something like this back, but here goes anyway... The Zen of Python says, among other things, "There should be one-- and preferably only one --obvious way to do it". Build systems seem to me like a great place to apply that. Build system proliferation is a nightmare. In the last several months I've gone through roughly all of the several hundred packages in the Debian archive that build (or were packed like they build) compiled code (e.g. python C extensions, library bindings, etc.). I was working on two goals: 1) Making sure they built with python3.5 2) Adding support for building with multiple python3 versions For the former, the issue were sometimes buil system related, but for the latter that was almost always the case. When packages used setup.py (and it made no difference if it was distutills or setuputils) it was virtually always easy. For anything else ranged from moderately complex to not worth the trouble. For the things I need it for in the distro, setup.py (with the changes we've made) works really well. Why do we need a bazillion options for Python build tools? To close with another quote from the Zen, "Special cases aren't special enough to break the rules." Scott K P.S. I worry about having to go through waf again [1]. That was painful. [1] https://lists.debian.org/debian-dak/2012/01/msg9.html
Re: Amend Debian Python Proposal to Include More Python Metadata?
> On Jan 22, 2016, at 6:04 PM, Scott Kitterman wrote: > > On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote: > ... >> We already have an option like this, the —root option which will just append >> a different prefix to all of the installation paths. So essentially instead >> of invoking ``python setup.py install —root /tmp/something/`` which is what >> I think you’re doing now, you’d do ``pip install —root /tmp/something/ >> —no-deps .`` and it’d just work similarly to what you have now, except pip >> would be responsible for interacting with the Python build system. > ... > > Why would pip interacting with the Python build system instead of setuptools > be better? > > Scott K setuptools *is* the build system right now (or distutils if you’re still using that). So the benefit to Debian right now would be: We maintain the distutils/setuptools hacks to make them suck less so you don’t have to! Longer term, we want to enable people to go hog wild and invent their own build systems instead of trying to do it all one size fits all with everyone miserable because it doesn’t solve anyones problems nicely. At that point the benefit to Debian becomes that instead of having to implement the standard build tool interface that these tools implement (Which will most likely be geared towards produces wheels, not towards putting files on disk in an installed location) Debian can just let us maintain that bit of code too. Of course, if Debian would prefer to interact directly with these tools, it’s totally fine to do that. The whole point of us doing this work and trying to define formats and APIs and not bless implementations is so anyone who wants to can slot in their own tool in the process instead of being forced to use some blessed tool. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Amend Debian Python Proposal to Include More Python Metadata?
> On Jan 22, 2016, at 6:02 PM, Barry Warsaw wrote: > > On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote: > >> Forget that pip can fetch files from PyPI and install them for a moment and >> consider the command ``pip install .``. Fundamentally this is similar to the >> command ``make install`` right? > > Please remind me what the long term plan for this is. Let's say it's 2021 and > you and your amazing team of dozens of distutil-sig developers have been > cranking away the whole time. > > What tool will be used to build Python packages? Whatever people want! At least, that’s my hope. Ideally we get away from the idea that any particular build system is the right build system (because the needs of a pure python thing like six.py is vastly different from Numpy which needs Fortran and other crazy things). My end goal is, as long as your build system can consume a sdist (or maybe a repository too, still up in the air, but some kind of source input) and produce a wheel (whether this is actually a zipped up wheel file or a properly formatted directory is also still up in the air). Then you have a good enough build system and that is all it needs to do, it doesn’t need to also implement installing and what not, that’s some other tool’s job. > > What tool will be used to upload (built/source?) Python packages to > WarehouseNG? Either twine, or we’ll fold that functionality into pip. > > What tool will be used to download Python packages from WarehouseNG? Whatever tool you want! We’ll have defined formats so as long as your tool speaks those formats it can install them. In the case of already “compiled” (pure Python or not) binary .whl files it can just download them, unzip, move into the correct location + install deps and such. For source files (in a not-yet thought out or done sdist 2.0 thing) it can download the sdist 2.0 file, install the build dependencies, invoke the build tool for this package (from above), get the wheel it produced, and then install that as if it had downloaded it from PyPI. > > What tool will be used to install packages from WarehouseNG into an importable > location? Same answer here. > > Cheers, > -Barry - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote: ... > We already have an option like this, the —root option which will just append > a different prefix to all of the installation paths. So essentially instead > of invoking ``python setup.py install —root /tmp/something/`` which is what > I think you’re doing now, you’d do ``pip install —root /tmp/something/ > —no-deps .`` and it’d just work similarly to what you have now, except pip > would be responsible for interacting with the Python build system. ... Why would pip interacting with the Python build system instead of setuptools be better? Scott K signature.asc Description: This is a digitally signed message part.
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote: >Forget that pip can fetch files from PyPI and install them for a moment and >consider the command ``pip install .``. Fundamentally this is similar to the >command ``make install`` right? Please remind me what the long term plan for this is. Let's say it's 2021 and you and your amazing team of dozens of distutil-sig developers have been cranking away the whole time. What tool will be used to build Python packages? What tool will be used to upload (built/source?) Python packages to WarehouseNG? What tool will be used to download Python packages from WarehouseNG? What tool will be used to install packages from WarehouseNG into an importable location? Cheers, -Barry pgpPQkjaWvhbD.pgp Description: OpenPGP digital signature
Re: Python Policy: Things to consider for Stretch
On Jan 21, 2016, at 10:47 AM, Scott Kitterman wrote: >I've taken a run through the current Python Policy to see where I think it >needs to be updated for Stretch. Thanks Scott for the badly needed update. Some comments, apologies for the lack of good quoting, or if I've read the diff incorrectly. 2.5 Module Path "Public Python modules must be installed in the system Python modules directory, /usr/lib/python./dist-packages. Public Python 3 modules must be installed in /usr/lib/python3/dist-packages." I think this could use a bit of disambiguation, sice "system Python modules" could mean either Python generically, in which case the second sentence is in conflict. Do you mean "system Python 2 modules"? If so, let's say that. Also, since Python 2.7 is all there will be from now on, why not say that explicitly? A good reason not to would be because Policy needs to cover older releases where there can be multiple versions of Python 2. But then later in your diff you seem to mention python2.7 as being associated with /usr/local/lib/python2./dist-packages. So maybe say /usr/local/lib/python2.7/dist-packages here too? There really won't be any other value for than 7. 3.4 Specifying Supported Versions "The optional `X-Python-Version (preferred) or `XS-Python-Version` field..." It's a little confusing because we're now saying we prefer not to have either field. How about "(previously preferred)" or just drop the parenthetical? 3.6 Provides s/substituation/substitution/ B.2. dh_python2 and dh_python3 Again, I think here you want to say "Python2 and Python3" to disambiguate between generic Python. Other than that, +1 Cheers, -Barry pgplZMuJy6II7.pgp Description: OpenPGP digital signature
Re: Amend Debian Python Proposal to Include More Python Metadata?
> On Jan 22, 2016, at 5:30 PM, Barry Warsaw wrote: > > Hey Donald, thanks for starting this conversation. I for one am super > appreciative of all the consideration you give for Debian's little slice of > the world. > > There's a lot to unpack in this thread, and I'm a little under the weather[1], > so hopefully this makes sense. > > Big +1 for recording the files that get installed via the .egg-info/RECORD > file. As you know, I've been working on dirtbike, which is a "rewheeling" > tool to turn an installed package into a .whl. While Debian tightly controls > via policy the set of wheels we'll allow into the archive, dirtbike has code > for parsing the RECORD file. Unfortunately this is never exercised in > practice because we don't have RECORD files - at least not for the packages we > care about[2]. > > Big +1 for using setuptools everywhere. By my count, of the packages that I > happen to randomly have installed on my Ubuntu 16.04 system, I have 67 > .egg-info files and 113 .egg-info directories. I'd rather have .egg-info > directories everywhere. > > +1 for a lintian warning if distutils is used. I guess I'm +0 on forcing that > through pybuild because it'll be unobvious and mysterious, and kind of lets > upstreams off the hook. I'd mildly prefer to patch packages that use > distutils because that's much more discoverable, but I can appreciate that > that's a lot of work we'd be imposing on maintainers, so I won't argue this > too much (other than to say that if pybuild forces it, let's definitely > document this in its manpage!). > > On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote: > >> Currently --record includes the .pyc files which is both unneeded and bad. >> Before this gets added either in setuptools or by us, this needed to be >> fixed. > > +1 for (I think) another reason than has already been discussed. We won't be > generating .egg-info directories on the end-user's machine, but instead the > machine the package is built on. That could be a maintainer's own system or a > central build machine depending on various conditions. But because the pycs > are generated on the end-user's machine, we actually don't know what pycs will > be generated when the debs are installed, so the egg-info/RECORD file *can't* > contain them, at least not accurately. FWIW It appears that if you do setup.py install —no-compile to tell distutils not to compile the .pyc during .deb build time, I believe that as is it will just omit the .pyc files. Not sure if that’s workable for Debian or not. > > On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote: > >> Regardless of what happens in this thread, pip is going to stop mucking >> around in files that are owned by some other tool without some sort of >> opt-in/--force style flag *and* we're going to be restructuring things to try >> and guide people away from using pip outside of a virtual environment or >> through the user of --user as well. > > Of course, I'd still like --user to be the default[3]. I think that's still > the eventual goal for pip, but isn't yet implemented because $priorities. Yes, still a goal (it’s not as simple as —user will be the default, but that’s the outcome for a non root account in a system Python). > >> A more controversial way that comes with possibly some extra benefits (which >> Debian may not care about) is to use ``pip`` itself as the build tool rather >> than directly invoking setup.py. In this pip would be responsible for mucking >> around with the distutils/setuptools mess instead of that needing to be >> handled by Debian specific tooling. > > I'd like to better understand how this would work. IIUC, the Fedora ecosystem > is making or already has made this switch, but I don't know how it works. > Obviously, we don't want to install wheels into > /usr/lib/python3/dist-packages. I haven’t actually personally seen if they’ve done it, but the Fedora maintainer of the python-pip told me they were doing it, and they had a patch to pip to add a feature they needed for it. Forget that pip can fetch files from PyPI and install them for a moment and consider the command ``pip install .``. Fundamentally this is similar to the command ``make install`` right? It builds the files and then installs them into the final location. As I understand it, the standard way for a project like Debian to install a project using a “regular” Makefile install like that is to do ``make DESTDIR=/tmp/some-tempdir install`` so that it invokes the build system, sets all the paths in the resulting build system to the final destination, but then right before it actually copies files to the final paths it just appends a directory in front of them to let you stage the files in a different location instead. We already have an option like this, the —root option which will just append a different prefix to all of the installation paths. So essentially instead of invoking ``python setup.py install —root /tmp/something/`` which is what I
Re: Amend Debian Python Proposal to Include More Python Metadata?
Hey Donald, thanks for starting this conversation. I for one am super appreciative of all the consideration you give for Debian's little slice of the world. There's a lot to unpack in this thread, and I'm a little under the weather[1], so hopefully this makes sense. Big +1 for recording the files that get installed via the .egg-info/RECORD file. As you know, I've been working on dirtbike, which is a "rewheeling" tool to turn an installed package into a .whl. While Debian tightly controls via policy the set of wheels we'll allow into the archive, dirtbike has code for parsing the RECORD file. Unfortunately this is never exercised in practice because we don't have RECORD files - at least not for the packages we care about[2]. Big +1 for using setuptools everywhere. By my count, of the packages that I happen to randomly have installed on my Ubuntu 16.04 system, I have 67 .egg-info files and 113 .egg-info directories. I'd rather have .egg-info directories everywhere. +1 for a lintian warning if distutils is used. I guess I'm +0 on forcing that through pybuild because it'll be unobvious and mysterious, and kind of lets upstreams off the hook. I'd mildly prefer to patch packages that use distutils because that's much more discoverable, but I can appreciate that that's a lot of work we'd be imposing on maintainers, so I won't argue this too much (other than to say that if pybuild forces it, let's definitely document this in its manpage!). On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote: >Currently --record includes the .pyc files which is both unneeded and bad. >Before this gets added either in setuptools or by us, this needed to be >fixed. +1 for (I think) another reason than has already been discussed. We won't be generating .egg-info directories on the end-user's machine, but instead the machine the package is built on. That could be a maintainer's own system or a central build machine depending on various conditions. But because the pycs are generated on the end-user's machine, we actually don't know what pycs will be generated when the debs are installed, so the egg-info/RECORD file *can't* contain them, at least not accurately. On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote: >Regardless of what happens in this thread, pip is going to stop mucking >around in files that are owned by some other tool without some sort of >opt-in/--force style flag *and* we're going to be restructuring things to try >and guide people away from using pip outside of a virtual environment or >through the user of --user as well. Of course, I'd still like --user to be the default[3]. I think that's still the eventual goal for pip, but isn't yet implemented because $priorities. >A more controversial way that comes with possibly some extra benefits (which >Debian may not care about) is to use ``pip`` itself as the build tool rather >than directly invoking setup.py. In this pip would be responsible for mucking >around with the distutils/setuptools mess instead of that needing to be >handled by Debian specific tooling. I'd like to better understand how this would work. IIUC, the Fedora ecosystem is making or already has made this switch, but I don't know how it works. Obviously, we don't want to install wheels into /usr/lib/python3/dist-packages. FTR, I am still working on updating Debian's pip. I'm slowly shaving the yak, but there are still a few things to do. If you want to help, get in touch. >[3] Import pkg_resources is not the fasest thing in the world, it has to scan >every directory on sys.path looking for things to activate and it comes with >a bunch of side effects. This happens implicitly for any project using >console scripts. Which frankly sucks. It's also fragile. Every once in a while a broken package gets uploaded that breaks pkg_resources, and it's not easy to debug or repair. I really hope Brett can fix this when/if he builds this functionality into the stdlib. Cheers, -Barry [1] double meaning: I have a cold and we're in the early stages of an historic snow storm! [2] dirtbike has two fallbacks, both of which use `dpkg -L` to get the list of installed files. The first fallback uses `dpkg -S` to find the binary package that contains the Python package's .egg-info file/directory (doesn't matter for this purpose). pkg_resources doesn't have one of those. The second fallback imports the Python package and looks for the binary package containing the top-level directory. It's all rather ugly, and I'd like to just use the .egg-info/RECORD file, but I suspect I'll still need the import fallback for pkg_resources. [3] https://github.com/pypa/pip/issues/1668 pgpcYL6d9Pl7r.pgp Description: OpenPGP digital signature
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Friday, January 22, 2016 01:47:28 PM Fred Drake wrote: > On Fri, Jan 22, 2016 at 1:41 PM, Scott Kitterman wrote: > > For Debian it's bad because we don't ship the .pyc files in the package > > they are managed locally by the installed python system. They are also > > unnecessary because setuptools/pip/python is smart enough to relate .pyc > > files to their .py files without them being listed. > > The .pyc files aren't shipped in the .deb, or they aren't installed > when the .deb is installed? They aren't shipped in the .deb, so the list of files shipped in the .deb shouldn't include them. > If they're installed, I think they should be listed. If not installed > when the package is installed, then they definitely shouldn't be > listed. No I don't think they should be. There's no need for that and it may change. Then you have a maintenance nightmare. Currently we have both python3.4 and python3.5 as supported versions so as part of installation *.cpython-34.pyc and *.cpython-35.pyc are generated on the local file system. As the python3 versions change, those will change and Python knows how to deal with that as long as the .py files are present. Scott K > > Part of the goal here is to make sure that regardless of the installation > > method, there's a list of files that were installed. Since Debian > > packages > > don't install the .pyc files, listing them is wrong. > > I agree that listing them if they aren't installed is wrong. > > > -Fred
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Fri, Jan 22, 2016 at 1:41 PM, Scott Kitterman wrote: > For Debian it's bad because we don't ship the .pyc files in the package they > are managed locally by the installed python system. They are also unnecessary > because setuptools/pip/python is smart enough to relate .pyc files to their > .py > files without them being listed. The .pyc files aren't shipped in the .deb, or they aren't installed when the .deb is installed? If they're installed, I think they should be listed. If not installed when the package is installed, then they definitely shouldn't be listed. > Part of the goal here is to make sure that regardless of the installation > method, there's a list of files that were installed. Since Debian packages > don't install the .pyc files, listing them is wrong. I agree that listing them if they aren't installed is wrong. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Friday, January 22, 2016 01:32:45 PM Fred Drake wrote: > On Fri, Jan 22, 2016 at 12:40 PM, Scott Kitterman wrote: > > Currently --record includes the .pyc files which is both unneeded and bad. > > Before this gets added either in setuptools or by us, this needed to be > > fixed. > Why is this bad? Isn't the point that the record file lists the files > installed by the installation process? If the pyc files are installed > as part of the package, they should be listed. For Debian it's bad because we don't ship the .pyc files in the package they are managed locally by the installed python system. They are also unnecessary because setuptools/pip/python is smart enough to relate .pyc files to their .py files without them being listed. Part of the goal here is to make sure that regardless of the installation method, there's a list of files that were installed. Since Debian packages don't install the .pyc files, listing them is wrong. Scott K P.S. I'm subscribed to the list, so there's no need to CC me.
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Fri, Jan 22, 2016 at 12:40 PM, Scott Kitterman wrote: > Currently --record includes the .pyc files which is both unneeded and bad. > Before this gets added either in setuptools or by us, this needed to be fixed. Why is this bad? Isn't the point that the record file lists the files installed by the installation process? If the pyc files are installed as part of the package, they should be listed. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Friday, January 22, 2016 12:11:27 PM Donald Stufft wrote: ... > 3) It slipped my mind that you have to pass an additional flag to setuptools > right now to get the full file list (pip passes that flag unconditionally) > however I'm going to poke setuptools to see about getting them to add the > record file unconditionally to the .egg-info directory so it doesn't > require the --record flag. (Although Debian could add it earlier if they > wanted, but it's fine to wait for setuptools to change here). ... Just to document this (since I think it's significant and worth memorializing) from IRC: Currently --record includes the .pyc files which is both unneeded and bad. Before this gets added either in setuptools or by us, this needed to be fixed. Scott K signature.asc Description: This is a digitally signed message part.
Re: Amend Debian Python Proposal to Include More Python Metadata?
> let's make a deal. If you will make sure pip doesn't touch system files > (and others will not crucify me for this) - I will make sure pybuild > uses above line (if setuptools is not detected in setup.py but is listed > in Build-Depends). FTR: (after talking about it on IRC) the "deal" was about my free time (for implementing this) and not about not accepting patches (and Donald already mentioned he wants to implement the cookie I want so I simply wanted him to give it higher priority) -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: Amend Debian Python Proposal to Include More Python Metadata?
> On Jan 22, 2016, at 11:51 AM, Scott Kitterman wrote:
>
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
>>> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski wrote:
>>>
>>> to be honest, I still don't know what you're asking for. What do you
>>> want us to do? Patch 2.7's distutils?
>>
>> Essentially, ensure that setuptools not distutils is used in a setup.py.
>> There are generally three kinds of setup.py files:
>>
>> 1) Ones that use setuptools unconditionally - These ones you just leave
>> alone, they are already correct and you should already have a build depends
>> on python-setuptools. 2) Ones that conditionally use setuptools - These
>> ones you just need to satisfy whatever condition the setup.py uses to
>> enable setuptools. Typically this is just checking if setuptools is
>> importable but sometimes they use environment variables or similar. 3) Ones
>> that use distutils unconditionally - These ones you switch to making them
>> use setuptools instead of distutils.
>>
>> Now, that’s the high level overview, there’s an easier, more automatic way
>> that could maybe just be added to pybuild (Not sure exactly how pybuild
>> works) where instead of invoking the setup.py as:
>>
>>python setup.py install (or whatever commands/args you’re passing)
>>
>> You do it as (taken from pip):
>>
>>python -c "import setuptools,
>> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
>> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
>> (or whatever commands/args you’re passing).
>>
>> The thing is kind of ugly, but that will install things using setuptools
>> (just like pip does) regardless of if it imports setuptools or distutils in
>> it’s setup.py file.
>
> I tried this and it works, but what's the big deal?
>
> It provides a PKG-INFO file that has identical content to the old egg-info
> file,
> an empty dependency links file, and a top_level.txt file with the one line in
> it.
>
> Why is this better (I'm not a huge upstream developer of Python stuff, but I
> do
> do some and I don't see what this gets me)?
>
> Scott K
This a fair question:
1) distutils is effectively frozen and all improvements to the build system
are either going to be inside of setuptools, or will be opt in build systems
that (probably) won't rely on setup.py at all. This includes newer metadata
formats and the like as well.
2) That top_level.txt is an important file, without it we have no idea that
$SITE-PACKAGES/requests is in any way associated with the metadata file
requests-2.9.1.egg-info, that top_level.txt tells us what the top level
import names are for a particular project.
3) It slipped my mind that you have to pass an additional flag to setuptools
right now to get the full file list (pip passes that flag unconditionally)
however I'm going to poke setuptools to see about getting them to add the
record file unconditionally to the .egg-info directory so it doesn't
require the --record flag. (Although Debian could add it earlier if they
wanted, but it's fine to wait for setuptools to change here).
4) Switching it to a directory allows us to add additional files/information to
the Python level metadata, like the sort of divergent thread about getting
pip to stop mucking with OS owned files, the most likely path forward is to
drop an INSTALLER file inside the .egg-info directory that has some special
value ("system"? "os"? Not sure) which pip will take as a signal that it
really should not touch.
5) In pip, because of #2 and #3 a regular distutils installed package (which
gives you an .egg-info file, not directory) we have now way when we're told
to install a package named "foo" to associate that with any files on disk.
This means that we're currently lying when we tell a user we've uninstalled
a distutils installed package, we just remove the .egg-info file but leave
all the other files behind. We've deprecated the ability to do this, and we
tried to remove it all together in the recent pip 8.0 release, but we had to
roll it back because it caused too much breakage, largely due to OS packages
that were using distutils installed metadata.
On the surface, it may seem counterproductive for Debian to help pip get
better metadata about what files belong to a package so that we can
uninstall it, but we're going to solve the "people modify system files with
pip when they shouldn't do that" more directly and there exist use cases
where it's perfectly fine to break your system in this way, because it's an
emphereal system (think CI servers) where you just want to get the newest
version of a package (maybe just one package) into the system in order to
run some tests before you throw the whole server away or something equally
as reasonable.
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signatu
Re: Amend Debian Python Proposal to Include More Python Metadata?
[Paul Tagliamonte, 2016-01-22] > I'm sure if you had a real and honest conversation with Donnald, there'd > be middleground. I've never found him to be the sort to bully or ignore > technical arguments. and why we cannot find a middle ground here? Did I say something bad about Donald? Quite the contrary, I think he's the best guy to talk to regarding pip/setuptools/distutils. > "Give me a thing I want and I'll fix this broken thing otherwise it'll > remain broken" is *NOT* how we do things in Debian. Flat out. This > isn't a game of poker, and the archive is not yours to bargain with. no, it's: "I already use too many hacks, but I will add another one if you bribe me" ;P There's nothing "broken" on our side, so I don't have to fix anything (other than educating our users to not shoot themselves in the foot) I want to help Daniel (I will even send a pull request if he wants), all I need is something for me in return (and I'm also OK with adding "Q: why don't you allow me to touch system files? A: because evil Piotr made me" to pip's FAQ) > I've never used my position on the ftpteam to bully someone into doing a > thing, and I seriously hope that was a bad joke about using your > position in DPMT/PAPT to get what you want. it's my time and I will do in my time whatever I want. If I want to add a hack to pybuild in my free time, I will do it when I want, you will not bully me into doing something you think is right! -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: Amend Debian Python Proposal to Include More Python Metadata?
> On Jan 22, 2016, at 11:18 AM, Piotr Ożarowski wrote: > > let's make a deal. If you will make sure pip doesn't touch system files > (and others will not crucify me for this) - I will make sure pybuild > uses above line (if setuptools is not detected in setup.py but is listed > in Build-Depends). Regardless of what happens in this thread, pip is going to stop mucking around in files that are owned by some other tool without some sort of opt-in/--force style flag *and* we're going to be restructuring things to try and guide people away from using pip outside of a virtual environment or through the user of --user as well. The main reason why that hasn't happened yet isn't because I'm holding those things hostage, it's because I'm one person and my time is spread amongst several really important tools and I have to prioritize things in accordance to how bad it's on fire. In addition, pip is a cross platform program and we have to ensure that whatever we do works in a cross platform manner. You mentioned /usr/local/.../site-packages/ before, but plenty of systems don't have a /usr/local/.../site-packages/ that their Python respects because that is a Debian specific patch, so if we refused to install to /usr/.../site-packages/ we'd break installing at the system level on all platforms but Debian derived ones. What we need is a cross platform way to determine if a particular installed Python package is "owned" by the system or if it's something we're allowed to modify. That's going to require some effort to work out the exact best way, but in my head the way forward on that is to use the metadata directory (the one I'm asking y'all to start using) and add an additional piece of metadata inside of it that just says "Hey, This is a system install" that we can inspect and then take additional logic (like refusing to touch it without a --force) based on that. The change I'm asking for here helps make that possible (among other things). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
> > On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski wrote:
> >
> > to be honest, I still don't know what you're asking for. What do you
> > want us to do? Patch 2.7's distutils?
>
> Essentially, ensure that setuptools not distutils is used in a setup.py.
> There are generally three kinds of setup.py files:
>
> 1) Ones that use setuptools unconditionally - These ones you just leave
> alone, they are already correct and you should already have a build depends
> on python-setuptools. 2) Ones that conditionally use setuptools - These
> ones you just need to satisfy whatever condition the setup.py uses to
> enable setuptools. Typically this is just checking if setuptools is
> importable but sometimes they use environment variables or similar. 3) Ones
> that use distutils unconditionally - These ones you switch to making them
> use setuptools instead of distutils.
>
> Now, that’s the high level overview, there’s an easier, more automatic way
> that could maybe just be added to pybuild (Not sure exactly how pybuild
> works) where instead of invoking the setup.py as:
>
> python setup.py install (or whatever commands/args you’re passing)
>
> You do it as (taken from pip):
>
> python -c "import setuptools,
> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
> (or whatever commands/args you’re passing).
>
> The thing is kind of ugly, but that will install things using setuptools
> (just like pip does) regardless of if it imports setuptools or distutils in
> it’s setup.py file.
I tried this and it works, but what's the big deal?
It provides a PKG-INFO file that has identical content to the old egg-info
file,
an empty dependency links file, and a top_level.txt file with the one line in
it.
Why is this better (I'm not a huge upstream developer of Python stuff, but I do
do some and I don't see what this gets me)?
Scott K
signature.asc
Description: This is a digitally signed message part.
Re: Amend Debian Python Proposal to Include More Python Metadata?
On Fri, Jan 22, 2016 at 05:18:36PM +0100, Piotr Ożarowski wrote: > let's make a deal. If you will make sure pip doesn't touch system files > (and others will not crucify me for this) - I will make sure pybuild > uses above line (if setuptools is not detected in setup.py but is listed > in Build-Depends). Oh *come on*. That's insane. I'm sure if you had a real and honest conversation with Donnald, there'd be middleground. I've never found him to be the sort to bully or ignore technical arguments. "Give me a thing I want and I'll fix this broken thing otherwise it'll remain broken" is *NOT* how we do things in Debian. Flat out. This isn't a game of poker, and the archive is not yours to bargain with. Your argument is basically: - Thing X being broken is preventing Y from doing Z - We do Z on our platform - So, I'll fix X when you make Y stop doing Z pip is used outside Debian, and another special case is tech debt. pip touching the system sucks, but that's unrelated to the fact that thing X sucks. Why these issues are intertwined is confusing me, perhaps dstufft would have been better being dishonest and not mentioned pip in his mail. I've never used my position on the ftpteam to bully someone into doing a thing, and I seriously hope that was a bad joke about using your position in DPMT/PAPT to get what you want. Cheers, Paul signature.asc Description: PGP signature
Re: Amend Debian Python Proposal to Include More Python Metadata?
[Donald Stufft, 2016-01-22]
> Now, that’s the high level overview, there’s an easier, more automatic
> way that could maybe just be added to pybuild (Not sure exactly how
> pybuild works) where instead of invoking the setup.py as:
>
> python setup.py install (or whatever commands/args you’re passing)
>
> You do it as (taken from pip):
>
> python -c "import setuptools,
> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install (or
> whatever commands/args you’re passing).
let's make a deal. If you will make sure pip doesn't touch system files
(and others will not crucify me for this) - I will make sure pybuild
uses above line (if setuptools is not detected in setup.py but is listed
in Build-Depends).
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: Amend Debian Python Proposal to Include More Python Metadata?
> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski wrote:
>
> to be honest, I still don't know what you're asking for. What do you
> want us to do? Patch 2.7's distutils?
Essentially, ensure that setuptools not distutils is used in a setup.py. There
are generally three kinds of setup.py files:
1) Ones that use setuptools unconditionally - These ones you just leave alone,
they are already correct and you should already have a build depends on
python-setuptools.
2) Ones that conditionally use setuptools - These ones you just need to satisfy
whatever condition the setup.py uses to enable setuptools. Typically this is
just checking if setuptools is importable but sometimes they use environment
variables or similar.
3) Ones that use distutils unconditionally - These ones you switch to making
them use setuptools instead of distutils.
Now, that’s the high level overview, there’s an easier, more automatic way that
could maybe just be added to pybuild (Not sure exactly how pybuild works) where
instead of invoking the setup.py as:
python setup.py install (or whatever commands/args you’re passing)
You do it as (taken from pip):
python -c "import setuptools,
tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install (or
whatever commands/args you’re passing).
The thing is kind of ugly, but that will install things using setuptools (just
like pip does) regardless of if it imports setuptools or distutils in it’s
setup.py file.
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
Re: Amend Debian Python Proposal to Include More Python Metadata?
Hi, [Donald Stufft, 2016-01-21] > I'd like to suggest a change to the Debian Policy around Python packages that > will help enable the world of Python packaging to continue to progress > forward. [...] > I have a series of improvements that I'd like to make to the packaging > toolchain that will sort of build on one another, but which is not going to > function correctly with the distutils style metadata and I'm hoping that I can > convince y'all to make it policy to default to generating one of the other two > kinds (with varying methods, more on that later). Isn't that something for distutils-sig or python-dev? We install whatever setup.py creates, no matter if it's egg-info dir, file or something else. [...] > Now, I know that upgrading OS provided packages using pip is less than optimal it's more than that, it's something that makes me scream: kill it! kill it with fire!!! (but since we also allow /bin/rm in Debian, I guess we have to tolerate pip ;) > and I would greatly prefer that people did not do it (and I'm generally in > agreement) however if we don't enable people to do it, they'll just continue > to > use an old version of pip and file bugs. It's a non starter for pip to make it > impossible to do. did I mention fire? ;) [...] > An additional benefit here is that by switching to using the directory based > options, we can add additional metadata files to the installed projects, much > like the INSTALLER file from PEP376 (IIRC). This file will likely be the path > to having pip refuse to touch OS owned files all together without some sort of > --force flag to override the safety switch. why pip cannot do that right now? IMO pip should be allowed to touch 3 types of paths only: /usr/local/pythonX.Y/, ~/.local/lib/pythonX.Y/ and virtual envs. [...] > So without getting into the actual *method* of doing this (of which there are > several different options with different trade offs) does this sound like > something at all that Debian would be interested in? to be honest, I still don't know what you're asking for. What do you want us to do? Patch 2.7's distutils? -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
