[Distutils] ANN: distlib 0.3.4 released on PyPI
I've recently released version 0.3.4 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #153: Raise warnings in get_distributions() if bad metadata seen, but keep going. * Fixed #154: Determine Python versions correctly for Python >= 3.10. * Updated launcher executables with changes to handle duplication logic. Code relating to support for Python 2.6 was also removed (support for Python 2.6 was dropped in an earlier release, but supporting code wasn't removed until now). A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.4/ [2] https://distlib.readthedocs.io/en/0.3.4/ [3] https://bitbucket.org/pypa/distlib/issues/new -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/RPPTGLXNTI4V6YX3SLZYGXV4MCLAW2NK/
[Distutils] ANN: distlib 0.3.3 released on PyPI
I've recently released version 0.3.3 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #152: Removed splituser() function which wasn't used and is deprecated. * Fixed #149: Handle version comparisons correctly in environment markers. * Add ARM-64 launchers and support code to use them. Thanks to Niyas Sait and Adrian Vladu for their contributions. * Fixed #148: Handle a single trailing comma following a version. Thanks to Blazej Floch for the report and suggested fix. * Fixed #150: Fix incorrect handling of epochs. * Reverted handling of tags for Python >= 3.10 (use 310 rather than 3_10). This is because PEP 641 was rejected. * Added a GitHub Actions workflow to perform tests. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.3/ [2] https://distlib.readthedocs.io/en/0.3.3/ [3] https://bitbucket.org/pypa/distlib/issues/new -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/WRCX727JNHMV65MWQS4H7GU3AM7K7TMP/
[Distutils] ANN: distlib 0.3.2 released on PyPI
I've recently released version 0.3.2 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #139: improved handling of errors related to the test PyPI server. * Fixed #140: allowed "Obsoletes" in more scenarios, to better handle faulty metadata already on PyPI. * Fixed #141: removed unused regular expression. * Fixed #143: removed normcase() to avoid some problems on Windows. * Fixed #146: added entry for SourcelessFileLoader to the finder registry. * Fixed #147: permission bits are now preserved on POSIX when installing from a wheel. * Made the generation of scripts more configurable. * Added support for manylinux wheel tags. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.2/ [2] https://distlib.readthedocs.io/en/0.3.2/ [3] https://bitbucket.org/pypa/distlib/issues/new -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/UHGHLUK4YMN3LZK4KH4THARAWHN6ZUUM/
[Distutils] ANN: distlib 0.3.1 released on PyPI
I've recently released version 0.3.1 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #132: Added documentation to help with relative interpreter paths. Thanks to Paul Kienzle for the patch. * Fixed #134: Allowed specifying a different target Python version when generating scripts. * Fixed #135: Exposed the ``enquote_executable`` function previously implemented as an internal function. * Addressed #138: Improved metadata support for newer metadata versions. Thanks to James Tocknell for the patch. * Changed the output of flags in entry point definitions in wheels. Thanks to frostming (明希) for the patch. * Stopped writing JSON metadata. Only old-style metadata is written now. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Note: Mailman3 might mishandle some of the links below. In that case, just copy and paste the links into your browser address bar - that should work. Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.1/ [2] https://distlib.readthedocs.io/en/0.3.1/ [3] https://bitbucket.org/pypa/distlib/issues/new -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/QA57KVMRJEXQF3K5QH2ONCAQHIJIOTWD/
[Distutils] Re: July 1 deadline for Mercurial repositories on Bitbucket
FYI distlib has moved to Git, while remaining on BitBucket. The move was made in January 2020. https://bitbucket.org/pypa/distlib/src/master/ Regards, Vinay Sajip On Tuesday, 16 June 2020, 03:45:52 BST, distutils-sig-requ...@python.org wrote: Send Distutils-SIG mailing list submissions to distutils-sig@python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman3/lists/distutils-sig.python.org/ or, via email, send a message with subject or body 'help' to distutils-sig-requ...@python.org You can reach the person managing the list at distutils-sig-ow...@python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Distutils-SIG digest..." Today's Topics: 1. July 1 deadline for Mercurial repositories on Bitbucket (Sumana Harihareswara) 2. Re: July 1 deadline for Mercurial repositories on Bitbucket (Cooper Lees) 3. Re: July 1 deadline for Mercurial repositories on Bitbucket (Sumana Harihareswara) 4. Re: July 1 deadline for Mercurial repositories on Bitbucket (Sumana Harihareswara) -- Date: Mon, 15 Jun 2020 17:29:42 -0400 From: Sumana Harihareswara Subject: [Distutils] July 1 deadline for Mercurial repositories on Bitbucket To: DistUtils mailing list Message-ID: <372de837-9aa7-f489-bb42-0d3077ad5...@changeset.nyc> Content-Type: text/plain; charset=utf-8; format=flowed TL;DR: distlib_hg and bandersnatch, you have a July 1st deadline to switch to Git or move off Bitbucket. https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket "we've decided to remove Mercurial support from Bitbucket Cloud and its API." "Mercurial features and repositories will be officially removed from Bitbucket and its API on July 1, 2020." Most of https://bitbucket.org/pypa/ is Mercurial repos. Maintainers of those repositories need to switch them to Git or move them off Bitbucket by July 1st. distlib_hg and bandersnatch are the active Mercurial repos that most clearly need to switch or move. A few repos in bitbucket.org/pypa have explicit "moved to GitHub" notices on them. And I assume its pkg_resources , setuptools, import_resources, pylauncher, and bootstrap projects are now defunct, but it's probably worth archiving them somehow anyway, for historical reference. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc -- Date: Mon, 15 Jun 2020 14:34:02 -0700 From: Cooper Lees Subject: [Distutils] Re: July 1 deadline for Mercurial repositories on Bitbucket To: Sumana Harihareswara Cc: DistUtils mailing list Message-ID: Content-Type: multipart/alternative; boundary="Apple-Mail=_108E09A8-924C-4637-B2BF-854F5C6C3608" --Apple-Mail=_108E09A8-924C-4637-B2BF-854F5C6C3608 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Bandersnatch moved over 2 years ago and has been displaying the move on = the repository this whole time via the README.md. I just deleted it from BitBucket. As we have been for the last 2 years, = all bandersnatch action can be found here: = https://github.com/pypa/bandersnatch = <https://github.com/pypa/bandersnatch> Cooper > On Jun 15, 2020, at 2:29 PM, Sumana Harihareswara = wrote: >=20 > TL;DR: distlib_hg and bandersnatch, you have a July 1st deadline to = switch to Git or move off Bitbucket. >=20 > https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket >=20 > "we've decided to remove Mercurial support from Bitbucket Cloud and = its API." >=20 > "Mercurial features and repositories will be officially removed from = Bitbucket and its API on July 1, 2020." >=20 > Most of https://bitbucket.org/pypa/ is Mercurial repos. Maintainers of = those repositories need to switch them to Git or move them off Bitbucket = by July 1st. >=20 > distlib_hg and bandersnatch are the active Mercurial repos that most = clearly need to switch or move. >=20 > A few repos in bitbucket.org/pypa have explicit "moved to GitHub" = notices on them. And I assume its pkg_resources , setuptools, = import_resources, pylauncher, and bootstrap projects are now defunct, = but it's probably worth archiving them somehow anyway, for historical = reference. >=20 > --=20 > Sumana Harihareswara > Changeset Consulting > https://changeset.nyc > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at = https://mail.python.org/archives/list/distutils-sig@python.org/message/QOS= UVOF6RY4EVFBFZRHL4WAENXVKH5YL/ --Apple-Mail=_108E09A8-924C-4637-B2BF-854F5C6C3608 Content-Transfer-Encoding: q
[Distutils] ANN: distlib 0.3.0 released on PyPI
I've recently released version 0.3.0 of distlib on PyPI [1]. For newcomers,distlib is a library of packaging functionality which is intended to beusable as the basis for third-party packaging tools. The main changes in this release are as follows: * Partially addressed #102: modules attribute of InstalledDistribution was incorrectly computed as a list of bytes, rather than a list of str. This has now been corrected. * Updated Locator._get_digest to check PyPI JSON responses for a "digests" dictionary before trying "algo_digest" keys. Thanks to Jeffery To for the patch. * Fixed #123: Improved error message if a resource isn't found. * Fixed #124: Stopped norm-casing the executable written into shebangs, as it doesn't work for some non-ASCII paths. * Fixed #125: Updated launchers with versions that correctly report errors containing non-ASCII characters. The updated launchers now also support relative paths (see http://bit.ly/2JxmOoi for more information). * Fixed #127: Allowed hyphens in flags in export specifications. * Changed Python version handling to accommodate versions like e.g. 3.10 (no longer assume a version X.Y where X and Y are single digits). A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements,please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.0/[2] https://distlib.readthedocs.io/en/0.3.0/[3] https://bitbucket.org/pypa/distlib/issues/new -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/JWLL47OXOEP4GZ44MNSZAJK5ZD2XJ2WL/
[Distutils] ANN: distlib 0.2.9 released on PyPI
I've recently released version 0.2.9 of distlib on PyPI [1]. For newcomers,distlib is a library of packaging functionality which is intended to beusable as the basis for third-party packaging tools. The main changes in this release are as follows: * Updated default PyPI URL to https://pypi.org/pypi. * Relaxed metadata format checks to ignore 'Provides'. * Fixed #33, #34: simplified script template. * Fixed #115: Relaxed check for '..' in wheel archive entries by not checking filename parts, only directory segments. * Fixed #116: Corrected parsing of credentials from URLs. * Fixed #122: Skipped entries in archive entries ending with '/' (directories) when verifying or installing. * Commented out Disqus comment section in documentation. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements,please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.2.9/[2] https://distlib.readthedocs.io/en/latest/overview.html#change-log-for-distlib[3] https://bitbucket.org/pypa/distlib/issues/new -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/SYOKX5FUZNS5QRGRQDMZR32LPOS6DOLH/
[Distutils] Custom installation commands in setup.py
Are custom installation commands in setup.py no longer respected by setuptools? For example, the pybind11 project has a custom InstallHeaders command class in its setup.py, which is passed to the setup() call. When setup is imported from setuptools, the custom command class never gets invoked. When setup is imported from distutils.core, the custom command class is invoked. What's the reason for the disparity - can someone please enlighten me? Regards, Vinay Sajip-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/WJW62DPQVWWXGBHDT3IQARISRDHQBITV/
[Distutils] Re: PyPI not showing latest version?
It's coming up correctly on both the search page and in pip install now :-) $ curl -I https://pypi.org/simple/python-gnupg/HTTP/1.1 200 OKCache-Control: max-age=600, publicContent-Security-Policy: default-src 'none'; sandbox allow-top-navigationContent-Type: text/html; charset=UTF-8ETag: "YEnWUk5DPw++o3kMUQi+rQ"Referrer-Policy: origin-when-cross-originServer: nginx/1.13.9X-PyPI-Last-Serial: 3957873Content-Length: 5154Accept-Ranges: bytesDate: Wed, 13 Jun 2018 15:20:21 GMTAge: 1804Connection: keep-aliveX-Served-By: cache-iad2131-IAD, cache-lhr6324-LHRX-Cache: HIT, MISSX-Cache-Hits: 1, 0X-Timer: S1528903221.367847,VS0,VE80Vary: Accept-Encoding, Accept-EncodingStrict-Transport-Security: max-age=31536000; includeSubDomains; preloadX-Frame-Options: denyX-XSS-Protection: 1; mode=blockX-Content-Type-Options: nosniffX-Permitted-Cross-Domain-Policies: none Regards, Vinay Sajip On Wednesday, 13 June 2018, 16:07:33 BST, Ernest W. Durbin III wrote: On June 13, 2018 at 11:03:05 AM, Vinay Sajip via Distutils-SIG (distutils-sig@python.org) wrote: I just uploaded python-gnupg 0.4.3 to PyPI using Twine. Searchstill shows the previous version: https://pypi.org/search/?q=python-gnupg =>0.4.2 Search index updates are not immediate and are only used by the web search form and pip search. pip installs use the simple index. However, clicking on the link brings up the page for the latest version: https://pypi.org/project/python-gnupg/ => 0.4.3 This indicates that the cache purge for the project worked, which should also purge https://pypi.org/simple/python-gnupg/ But pip install is also wrongly picking up 0.4.2. What's the expected delay between uploading a new version and having it be available via pip? I would have expected more or less immediately. All systems are showing as operational. Indeed, new uploads should be available for pip install very quickly. I’m seeing 0.4.3 when I request the simple index. Can you provide the output of `curl -I https://pypi.org/simple/python-gnupg/` It is possible that you are hitting a stale cache in PyPI's CDN. Regards, Vinay Sajip -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HGJUP5NCCW4LMYVBJBL3XAWSA3CFAHY4/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/XSQBYQF6CFOK35PMM3YOP6G556YBVRBC/
[Distutils] PyPI not showing latest version?
I just uploaded python-gnupg 0.4.3 to PyPI using Twine. Search still shows the previous version: https://pypi.org/search/?q=python-gnupg => 0.4.2 However, clicking on the link brings up the page for the latest version: https://pypi.org/project/python-gnupg/ => 0.4.3 But pip install is also wrongly picking up 0.4.2. What's the expected delay between uploading a new version and having it be available via pip? I would have expected more or less immediately. All systems are showing as operational. Regards, Vinay Sajip -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HGJUP5NCCW4LMYVBJBL3XAWSA3CFAHY4/
[Distutils] ANN: distlib 0.2.7 released on PyPI
I've just released version 0.2.7 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Addressed #102: InstalledDistributions now have a modules attribute which is a list of top-level modules as read from top_level.txt, if that is in the distribution info. * Fixed #103: Now https downloads are preferred to those over http. Thanks to Saulius Žemaitaitis for the patch. * Fixed #104: Updated launcher binaries to properly handle interpreter paths with spaces. Thanks to Atsushi Odagiri for the diagnosis and fix. * Fixed #105: cache_from_source is now imported from importlib.util where available. * Added support for PEP 566 / Metadata 2.1. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.2.7/ [2] https://goo.gl/M3kQzR [3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] ANN: distlib 0.2.6 released on PyPI
I've just released version 0.2.6 of distlib on PyPI [1]. For newcomers,distlib is a library of packaging functionality which is intended to beusable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #99: Updated to handle a case where sys.getfilesystemencoding() returns None. * Fixed #97: Eliminated a crash in EggInfoDistribution.list_distinfo_files() which was caused by trying to open a non-existent file. * Fixed #96: SimpleScrapingLocator no longer fails prematurely when scraping links due to invalid versions. * Improved error messages issued when interpreting markers. * Improved the shebangs written into installed scripts when the interpreter path is very long or contains spaces (to cater for a limitation in shebang line parsing on Linux). * Updated launcher binaries. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions forimprovements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.2.6[2] https://goo.gl/M3kQzR[3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] ANN: distlib 0.2.5 released on PyPI
I've just released version 0.2.5 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Changed regular expressions to be compatible with 3.6 as regards escape sequences. * Closed some resource leaks related to XML-RPC proxies. * Changed wheel processing to look for metadata in metadata.json as well as pydist.json. * Updated requirement and marker parsing to be compatible with PEP 508. * Made downloadability a factor in scoring URLs for preferences. * Removed Python 2.6 from the support list. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.2.5 [2] https://goo.gl/M3kQzR [3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
> the full METADATA format is documented in the pre-JSON revision of PEP 426. Can you confirm which exact revision in the PEPs repo you mean? I could guess at 0451397. That version does not refer to a field "Requires" (rather, the more recent "Requires-Dist"). Your conversion function reads the existing PKG-INFO, updates the Metadata-Version, and adds "Provides-Dist" and "Requires-Dist". It does not check whether the result conforms to that version of the PEP. For example, in the presence of "Requires" in PKG-INFO, you add "Requires-Dist", possibly leading to an ambiguity, because they sort of mean the same thing but could contain conflicting information (for example, different version constraints). The python-dateutils wheel which Jim referred to contained both "Requires" and "Requires-Dist" fields in its METADATA file, and, faced with a metadata set with both fields, the old packaging code used by distlib to handle the different metadata versions raised a "Unknown metadata set" error. In the face of ambiguity, it's refusing the temptation to guess :-) If the conversion function adds "Requires-Dist" but doesn't remove "Requires", I'm not sure it conforms to that version of the PEP. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
> Managing backwards compatibility is probably the single most important thing > we can do here. > There are almost 800,000 files on PyPI that someone can download and install, > telling all > of them they need to switch to some new system or things are going to break > for them is > simply not tenable. I agree. But if packaging is going at some point to break out of allowing completely bespoke code to run at installation time (i.e. executable code like a free-for-all setup.py, vs. something declarative and thus more restrictive) then IMO you have to sacrifice 100% backwards compatibility. See my comment in my other post about the ability to install old releases - I made that a goal of my experiments with the parallel metadata, to not require anything other than a declarative setup() in order to be able to install stuff using just the metadata, so that nobody has to switch anything in a big-bang style, but could transition over to a newer system at their leisure. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
> The technical problem with PEP 426 is that unless you want to throw away pypi > and start over, > all tools need to understand the old METADATA files regardless. It might not be as bad as that. For example, that IMO was the mistake behind the original concept of distutils2 - it was never going to fly as it required everyone to switch over to distutils2's way of doing things, and wouldn't be able to deal with old releases etc. For a time, I maintained a pretty extensive parallel set of metadata, based on just the data passed to setup() by packages using distutils/setuptools. This included not just the data for installation but even the data for package build, where it was purely declarative at the arguments-to-setup() level. Where a package didn't do completely bespoke things in setup() - like create new files, move files around etc. then the parallel set of metadata would allow installation of even old releases, without executing any setuptools code at all. I've not had the bandwidth to keep working on distlib and the metadata (example [1]), and the volume of new stuff going onto PyPI meant I didn't have time to keep on top of it. But the approach had some promise, in my view, and certainly showed that purely declarative packages (which didn't use e.g. custom build and install distutils/setuptools commands) could be installed using a completely different tool [than distutils/setuptools] without package authors having to change anything (beyond staying purely declarative). The distil documentation [2] shows installing a number of distributions (existing releases) from PyPI with better dependency resolution than pip does now, and without "throwing away PyPI". Anyway, I guess it's water under the bridge. Regards, Vinay Sajip [1] https://www.red-dove.com/pypi/projects/J/Jinja2/package-2.7.3.json [2] https://distil.readthedocs.io/en/0.1.0/installing.html#installing-distributions ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
> I thought the current status was that it's called metadata.json > exactly *because* it's not standardized, and you *shouldn't* look at > it? Well, it was work-in-progress-standardised according to PEP 426 (since sometimes implementations have to work in parallel with working out the details of specifications). Given that PEP 426 wasn't done and dusted but being progressed, I would have thought it perfectly acceptable to use "pydist.json", as the only things that would be affected would be packaging tools working to the PEP. > It's too bad that the JSON thing didn't work out, but I think we're > better off working on better specifying the one source of truth > everything already uses (METADATA) instead of bringing in *new* > partially-incompatible-and-poorly-specified formats. When you say "everything already uses", do you mean setuptools and wheel? If nobody else is allowed to play, that's one thing. But otherwise, there need to be standards for interoperability. The METADATA file, now - exactly which standard does it follow? The one in the dateutil wheel that Jim referred to doesn't appear to conform to any of the metadata PEPs. It was rejected by old metadata code in distlib (which came of out the Python 3.3 era "packaging" package - not to be confused with Donald's of the same name - which is strict in its interpretation of those earlier PEPs). The METADATA format (key-value) is not really flexible enough for certain things which were in PEP 426 (e.g. dependency descriptions), and for these JSON seems a reasonable fit. There's no technical reason why "the JSON thing didn't work out", as far as I can see - it was just given up on for a more incremental approach (which has got no new PEPs other than 440, AFAICT). I understand that social reasons are often more important than technical reasons when it comes to success or failure of an approach; I'm just not sure that in this case, it wasn't given up on too early. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib and wheel metadata
> Nope. Honestly, though, I wish there was *one* *library* that defined the > standard, > which was the case for setuptools for a while (yeah, I know, the warts, > really, I know) > because I really don't think there's a desire to innovate or a reason for > competition > at this level. In the case of wheel, perhaps it makes sense for that > implementation to > be authoritative. The problem, to me, is not whether it is authoritative - it's more that it's ad hoc, just like setuptools in some areas. For example, the decision to use "metadata.json" rather than "pydist.json" is arbitrary, and could change in the future, and anyone who relies on how things work now will have to play catch-up when that happens. That's sometimes just too much work for volunteer activity - dig into what the problem is, put through a fix (for now), rinse and repeat - all the while, little or no value is really added. In theory this is an "infrastructure" area where a single blessed implementation might be OK, but these de facto tools don't do everything one wants, so interoperability remains important. There's no reason why we shouldn't look to innovate even in this area - there's some talk of a GSoC project now to look at dependency resolution for pip - something that I had sort-of working in the distil tool long ago (as a proof of concept) [1]. We've gotten so used to how pip and setuptools work, and because they are "good enough", there is a real failure of imagination to see how things might be done better. Regards, Vinay Sajip [1] https://distil.readthedocs.io/en/0.1.0/overview.html#actual-improvements ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] distlib and wheel metadata
> humpty in term uses uses distlib which seems to mishandle wheel> metadata. >(For example, it chokes if there's extra distribution meta and > makes it impossible for buildout to install python-dateutil from a wheel.) I looked into the "mishandling". It's that the other tools don't adhere to [the current state of] PEP 426 as closely as distlib does. For example, wheel writes JSON metadata to metadata.json in the .dist-info directory, whereas PEP 426 calls for that data to be in pydist.json. The non-JSON metadata in the wheel (the METADATA file) does not strictly adhere to any of the metadata PEPs 241, 314, 345 or 426 (it has a mixture of incompatible fields). I can change distlib to look for metadata.json, and relax the rules to be more liberal regarding which fields to accept, but adhering to the PEP isn't mishandling things, as I see it. Work on distlib has slowed right down since around the time when PEP 426 was deferred indefinitely, and there seems to be little interest in progressing via metadata or other standardisation - we have to go by what the de facto tools (setuptools, wheel) choose to do. It's not an ideal situation, and incompatibilities can crop up, as you've seen. Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] JetBrains licences
I accidentally cc-ed this list when sending email to a Python committer about JetBrains licences. Please don't take up one of these licences unless you are a Python committer; I will remove access from any person who is not named in https://hg.python.org/committers.txt I apologise for any confusion I have caused. Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PyCharm / JetBrains licence
Dear Christian, You can access the PyCharm and other JetBrains products licence at the following link: https://account.jetbrains.com/a/53fr5cnq If you don't already have a JetBrains account, you will need to create one to activate the licence. Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] deprecating pip install --target
Robert Collins robertcollins.net> writes: > > This is fairly broken - it doesn't handle platlib vs purelib (see pip > PR 3450), doesn't handle data files, or any other layout. Donald says > pip uses it to maintain the _vendor subtrees only, which doesn't seem > like a particularly strong use case. > > Certainly the help description for it is misleading - since what we're > actually doing is copying only a subset of what the package installed > - so at a minimum we need to be much clearer about what it does. > > But, I think it would be better to deprecate it and remove it... so > I'm pinging here to see if anyone can explain a sensible use case for > it in the context of pip :) I use it in pretty much the same way as Paul mentioned - I wouldn't like it to go unless something equivalent is available. Updating the help / documentation for it to better reflect what it does would be uncontroversial for me, but I see no strong reason for deprecation and removal. As Paul suggests, it can get stricter about what it'll handle. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] ANN: distlib 0.2.2 released on PyPI
I've just released version 0.2.2 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed issue #81: Added support for detecting distributions installed by wheel versions >= 0.23 (which use metadata.json rather than pydist.json). * Updated default PyPI URL to https://pypi.python.org/pypi * Updated to use different formatting for description field for V1.1 metadata. * Corrected “classifier” to “classifiers” in the mapping for V1.0 metadata. * Improved support for Jython when quoting executables in output scripts. * Fixed issue #77: Made the internal URL used for extended metadata fetches configurable via a module attribute. * Fixed issue #78: Improved entry point parsing to handle leading spaces in ini-format files. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.2.2 [2] https://goo.gl/M3kQzR [3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Remove distutils, was: red, green, refactor ...
From: Nick Coghlan <ncogh...@gmail.com> > We still need a migration path to modern metadata standards for > everyone using distutils and setuptools - that's the side of things > that caused major problems for both distribute and distutils2. I've thought about the migration path with distil, which uses a declarative metadata format (superset of PEP-426 as it was a while ago) where the actual metadata is extracted from setup.py files in releases already on PyPI. This works well for simple distributions which don't do anything fancy in setup.py, such as extending build commands to create files on the fly which are then installed etc. In my early testing, distil (which uses no distutils or setuptools code) could install and package distributions which were pure-Python as well as package, build and install ones with C extensions and even Fortran extensions. I haven't done much work on it for a while due to other commitments, but when I had the time to work on it, I noted precious little interest from distutils-sig (yes, I know we're all volunteers, but one person can only do so much). IMO the distil approach comes the closest to a least-work migration path: with distutils2 / packaging that was pulled from 3.3, almost everyone would have to convert their releases to setup.cfg, with limited support for automating this and no support for existing stuff on PyPI, which made it a non-starter. Plus, setup.cfg never supported build-side metadata IIRC, only installation-side. With distil, the metadata for build and installation is *automatically* extracted from setup.py with no work required from packagers e.g. for existing releases, and the extraction only fails if the setup.py code does certain operations which can't be inferred from the actual call to setup() in setup.py (I know that those hard-to-infer operations occur in a lot of releases - but that's the nature of setup.py, where anything goes - I still wanted to see how much progress could be made for the simple cases). The metadata extracted has been useful in other contexts, e.g. Brett Cannon's caniusepython3 uses it to work out package dependencies. With distil, the entire dependency tree is calculated before any package is downloaded at all - something people seem to want - but, it seems, not to the extent of actually doing anything concrete, even to try out code and report problems ... distil isn't perfect and I've said it's more like a proof of concept, but it's quite usable for the "non-fancy" distributions. As long as people are free to put whatever code they want in setup.py, we're not really going to have a simple migration path, because a sizeable number of packagers will carry on doing so, and that'll be hard to convert to declarative metadata automagically (not to mention the situation with existing releases). And as long as there is so much inertia stopping interested parties from trying out new approaches, it seems like the pace of development of better tools will continue to be glacial. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PyPI search RPC not working as expected
The PyPI XML-RPC API seems not to be working as expected. The following
simple script:
from pprint import pprint
import sys
try:
from xmlrpclib import ServerProxy
except ImportError:
from xmlrpc.client import ServerProxy
rpc_proxy = ServerProxy('https://pypi.python.org/pypi')
if len(sys.argv) 2:
pkg = 'sarge'
else:
pkg = sys.argv[1]
pprint(rpc_proxy.search({'name': pkg}))
Returns an empty list when passed the command line argument tatterdema (it
should match my project tatterdemalion), whereas it does return a
non-empty list when passed sarg (matching my project sarge), or when
passed jobswo (matching my project jobsworth). My project ragamuffin
also fails to show up if passed to the script.
Can anyone shed any light on this? Have there been any changes to how the
search is supposed to work?
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] ANN: distlib 0.2.1 released on PyPI
I've just released version 0.2.1 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Fixed issue #58: Return a Distribution instance or None from locate(). Fixed issue #59: Skipped special keys when looking for versions. Improved behaviour of PyPIJSONLocator to be analogous to that of other locators. Added resource iterator functionality. Fixed issue #71: Updated launchers to decode shebangs using UTF-8. This allows non-ASCII pathnames to be correctly handled. Ensured that the executable written to shebangs is normcased. Changed ScriptMaker to work better under Jython. Changed the mode setting method to work better under Jython. Changed get_executable() to return a normcased value. Handled multiple-architecture wheel filenames correctly. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.2.1 [2] https://goo.gl/K5Spsp [3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Accessing package data files in wheels
From: Paul Sokolovsky pmis...@gmail.com
Which makes everyone in the audience wonder: how it happens that it's
2015, Python is at 3.5, but pkgutil.get_data() is in stdlib, while
pkg_resources.resource_stream() isn't? An implementation of
pkgutil.get_data() would be based on pkg_resources.resource_stream(),
or would contain just the same code as the latter, so it could easily
be exposed, and yet it isn't.
Perhaps because it's not always that way around -
pkg_resources.get_stream_resource, in relevant cases, returns a BytesIO which
wraps a byte string. If you want a stream, you could just as easily do this
yourself by calling io.BytesIO(pkg_util.get_data('foo.pkg', 'foo_resource')).
In the case of file resources only, pkg_resources.get_stream_resource can open
the file and return the stream directly. But this is an optimisation and is not
always necessarily available for all different loader implementations - which
is perhaps why a `pkgutil.get_data_stream` is not provided.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Fwd: add install_paths.json to .egg-info and .dist-info directory
From: Nick Coghlan ncogh...@gmail.com On 13 May 2015 at 00:44, Daniel Holth dho...@gmail.com wrote: As a prerequisite to adding useful finer-grained installations to setuptools wheel, I propose adding install_paths.json as a metadata +1, this sounds like a good incremental step forward to me. I'm not sure how best to document it, though. Perhaps have this initial increment as an implementation detail, and then formalise it in the next version of the wheel spec. Sounds like a reasonable idea to me, too. But IIUC this file will be written by an installer, but not necessarily present in the wheel - is that right? If not present in the wheel (or perhaps anyway) ISTM it makes sense to specify it in PEP 426 because it's part of the installation metadata. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Fwd: add install_paths.json to .egg-info and .dist-info directory
From: Daniel Holth dho...@gmail.com No, PEP 426 needs to be static metadata that is not modified by the installer. The new json file would make more sense in (a successor to) PEP 376 Database of Installed Python Distributions. Agreed. That's what I meant :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] elements of setuptools/wheel/FHS file categories proposal
From: Daniel Holth dho...@gmail.com pkg_resources without declaring that dependency. That is why I proposed writing the install paths to an importable file in the package's namespace on request without a new API. This would also Not sure what you mean by this, but I hope by importable you don't mean Python source. JSON sounds like a better option. We currently write all the installed files as a simple text file (RECORD), but there's no reason it couldn't be a JSON file which held the mappings of category - path used during installation, as well as the list of files installed. When you say package's namespace, are you referring to the .dist-info directory, or something else? Those two words are fraught with ambiguity. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] elements of setuptools/wheel/FHS file categories proposal
From: Daniel Holth dho...@gmail.com I had suggested writing the mapping next to one of the package's installed .py files, but it sounds like all other commenters would prefer a JSON file inside the .dist-info directory. Since it's installation metadata, .dist-info seems the right place for it. I would prefer to keep the RECORD manifest of all installed files plus hashes separate from the e.g. .dist-info/install_scheme.json, it should not be necessary to parse the former just to figure out where the config directory is. It's just a json.load - no specialised parsing code would seem to be required. Is your preference due to concerns about performance, aesthetics, backward compatibility or something else? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] elements of setuptools/wheel/FHS file categories proposal
From: Daniel Holth dho...@gmail.com orthogonality. If you want to argue it's equally fast, tell me that it's equally fast on a 1st gen Raspberry Pi, not on an 8-core Zeon. No point arguing either way on performance or memory usage without measurements :-) Backward compatibility seems easy enough to cater for by using a name other than RECORD. For some value of easy enough, obviously. Obviously we have to consider backward compatibility any way, since existing installations wouldn't have the category mapping in any form. Most importantly the RECORD just has nothing to do with the paths to each file category, and it doesn't even exist in a development checkout which has not been installed. In a development checkout, there would presumably be no this mapping of categories to paths was used for installation either. There would seem to be at least two such mappings, anyway - a default mapping specifying the publisher's preference, which might live in PEP 426 source metadata, and a logically separate mapping written to .dist-info which might be different, depending on how the installer was invoked (e.g. with command line overrides specified by a user at installation time). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Add additional file categories for distutils, setuptools, wheel
From: Paul Moore p.f.mo...@gmail.com On 19 April 2015 at 11:55, David Cournapeau courn...@gmail.com wrote: I can work on a paperman implementation of this on top of the wheel installer for the end of this week. I think that would both alleviate some concerns for people interested in everything in package directory, and make the discussion more focused. Is paperman a Disney reference, or something else? That sounds good. One thing the wheel install command doesn't support is per-user installs. I'd appreciate seeing some details of how those would be handled - on the assumption that you don't want to deal with pip's source just yet, a rough spec would be fine for now. I presume the way wheel install works is orthogonal to the scheme for handling different categories of data - distil install, for example, does per-user installs by default. Are the proposed implementations just a proof of concept, to validate the usability of the implemented scheme? What is the envisaged timeline for proposing/agreeing specifications for how the file categories will work cross-platform? I'm bearing in mind that there might be other implementations of installers which would need to interoperate with any file category scheme ... Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to sign a exe created with bdist_wininst?
According to this resource:
http://recon.cx/2012/schedule/attachments/54_Signed_executables.pps
it is doable, but tricky, and IIUC may not work on Windows XP SP2/SP3.
Wouldn't it be safer for the stub to work correctly in the presence of a
signature? Presumably it could use a different algorithm to locate the archive
directory, rather than just expecting it to be at the end of the file. Or if it
is less work, just make a temporary copy of the wininst .exe excluding the
appended signature, and use that for the unarchiving operation. (Just my 2
cents, or should I say tuppence ...)
Regards,
Vinay Sajip
From: Steve Dower steve.do...@microsoft.com
To: Paul Moore p.f.mo...@gmail.com; Brian Cole co...@eyesopen.com
Cc: distutils-sig@python.org distutils-sig@python.org
Sent: Saturday, 18 April 2015, 15:46
Subject: Re: [Distutils] How to sign a exe created with bdist_wininst?
#yiv2682230560 #yiv2682230560 -- .yiv2682230560EmailQuote
{margin-left:1pt;padding-left:4pt;border-left:#80 2px solid;}#yiv2682230560
It may be possible to add an empty key container to the stub with signtool so
that it can be filled in after adding the zip without having to extend the
length. I believe the PE header is modified to locate the certificate, so it
doesn't necessarily have to be at the end.
Feel free to investigate this yourself with the wininst stub in
Lib\distutils\command. I'll take a look, but may not be able to get to it for a
while (file an issue and nosy me if you don't get anywhere, or even if you do
and we can support this in newer versions).
Cheers,
Steve
Top-posted from my Windows Phone
From:Paul Moore
Sent:4/18/2015 2:58
To:Brian Cole
Cc:distutils-sig@python.org
Subject:Re: [Distutils] How to sign a exe created with bdist_wininst?
On 17 April 2015 at 16:17, Brian Cole co...@eyesopen.com wrote:
We've recently converted over to using bdist_wininst for creating our
Windows .exe installers for our libraries. Unfortunately, whenever we use
the Windows signtool utility to cryptographically sign our installer it
appears to corrupt the .exe and it can't be run anymore. The error message
thrown by Windows is Setup program invalid or damaged.
My best guess at this point is that bdist_wininst is creating a checksum of
the file somehow and signtool is altering the file in such a way to
invalidate that checksum. The commands we're using at this point is like
this:
python3.4.exe setup.py bdist_wininst --target-version 3.4 --bitmap OurLogo
--title OurTitle-OurVersion
cp DistUtilsSetupFileName.exe OurSetupFileName.exe
call C:\program Files (x86)\Microsoft Visual Studio
9.0\Common7\Tools\vsvars32.bat
signtool sign /n OurCompany /t
http://timestamp.verisign.com/scripts/timstamp.dll /d OurProject /du
OurWebsite OurSetupFileName.exe
Anyone know of a way to cryptographically sign an .exe installer from
bdist_wininst?
The wininst format is a stub Windows executable, with some ini-format
data and a zipfile appended (in that order). I don't know where
signtools adds the signature, but if it's at the end, then that won't
work (as it's necessary for the zip data to be the *last* thing in the
file - zipfile format supports prepending data but not appending it as
the central directory is defined as being at a fixed offset from the
end of the file).
There may also be a length or checksum in the ini data, I'd have to
check the source to confirm that. pause Just checked, no it doesn't
- the full details are here:
https://hg.python.org/cpython/file/bc1a178b3bc8/PC/bdist_wininst/install.c
So basically, I don't think it's possible to sign (or otherwise
modify) wininst executables.
Paul
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Get dependencies of a package without full download
From: Daniel Holth dho...@gmail.com Vinay Sajip was maintaining metadata as described here, I'm sure there are functions in distil to help fetch it. Yes, though there is no need for any special API to access it - the metadata is in JSON files served statically. You just make a standard HTTP request, using the client library of your choice, to get a specific URL. The files are under http://www.red-dove.com/pypi/projects/ And you can just use a browser to browse the metadata from this starting point. The most severe problem with this data is of course that it is not always correct because the environment he evaluated setup.py in to get the data may be different than yours. Indeed, which is why declarative metadata is so important. Down with setup.py! Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [ANN]: distlib 0.2.0 released on PyPI
I've just released version 0.2.0 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Updated match_hostname to use the latest Python implementation. Updates to better support PEP 426 / PEP 440. You can now provide interpreter arguments in shebang lines written by distlib. Removed reference to __PYVENV_LAUNCHER__ (relevant to OS X only). A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.2.0 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Standard packaging API? (was Re: Are there any plans to move to pip/wheels in buildout?)
AFAIK pip does use distlib (it is vendored by pip), but only for some ancillary functions such as pre-release version checks. I'm not sure it's a good idea to use pip's internal API (as it's internal, and I don't believe it's been designed for use as a library by external code). Regards, Vinay Sajip From: Leonardo Rochael Almeida leoroch...@gmail.com To: distutils sig distutils-sig@python.org Sent: Monday, 1 December 2014, 16:23 Subject: Re: [Distutils] Standard packaging API? (was Re: Are there any plans to move to pip/wheels in buildout?) I thought distlib was supposed to be that API... Even though pip doesn't use it. Though that would mean a new major version of buildout that worked on wheels exclusively instead of eggs. Pip itself has an internal API in the `pip.commands` package. From a casual glance it seems usable from other programs. I.e. you could import `pip.commands.install:Install`, instantiate and call `.run()`. On 1 December 2014 at 12:53, Jim Fulton j...@zope.com wrote: On Mon, Dec 1, 2014 at 8:55 AM, Piotr Dobrogost p...@lists-2014.dobrogost.net wrote: Are there any plans to move from easy_install/eggs to pip/wheels in buildout? Buildout doesn't really use easy_install. It uses setuptools. Originally, I tried to use easy_install directly (and do in some special cases where I shouldn't), but I needed a real API, which is why I moved down to setuptools. My hope is that some new API will emerge to replace setuptools. I have an impression that buildout project has stagnated I prefer to say it's stable. :) A great sign is that other folks on the project have driven recent work. which is unfortunate taking into consideration how much python packaging has changed recently. Buildout as it, is is entirely dependent on setuptools to add wheel support, at least until a new API emerges. AFAIK, pip doesn't provide an API for use by other tools. I'd be very happy to find out I'm wrong. I hope there's room for more than one command-line tool for working with packages in the ecosystem. It would be crazy for each tool to implement the low-level packaging machinery separately. We need a library to replace setuptools that pip uses and that other tools can use. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?
Thanks, that's very useful feedback. I agree, the need for RDP is very Windows-specific - I don't know how common RDP tools are for Unix, but Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access Windows machines via RDP, and it seems fairly stable. There are alternative tools available (such as remmina). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
From: Steve Dower steve.do...@microsoft.com Microsoft has released a compiler package for Python 2.7 Great. Thank you very much! Downloading it now :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadata 2.0: Is there a formal spec for a requirement?
From: Paul Moore p.f.mo...@gmail.com One thing that might be worth clarifying somewhere/somehow (not particularly in the specs, though) is where is the best place to find the canonical implementations of the various metadata specs. At one point, distlib seemed to be taking that role, but I'm not sure it is any more. That's more to do with the preferences of, and choices made by, other people. I don't know who decides what's canonical (I suppose Nick, naturally, but I'm not sure if it's *only* Nick) or what criteria they would use, but I've aimed distlib to implement the various PEPs in their various states of completion. Although I have been less active of late than I have previously, that's just down to my current workload: I'm busy with consultancy work :-) Note that I have recently updated distlib to reflect changes in PEP 440, though this functionality has not been officially released (it's available in the public repos, though). On requirements, distlib supports both the setuptools forms foo=X.Y and the PEP forms such as foo (~ X.Y). Generally, distlib and distil work well enough [for my needs] for the most part. Where distributions uses custom code in setup.py or extends distutils/setuptools command classes, I use distil pip to convert sdists to wheels, or distil package to convert .exe installers (like those on Christoph Golhke's site) to wheels. When you pin your dependencies, you don't have to do this dance too often. I feel I'm fairly responsive when issues are raised against either distlib or distil. I'm always open to feedback, try to keep the docs up to date, etc. The coverage docs are a little out of date, and coveralls is on my todo list. Not sure what more would be expected from a canonical implementation, other than an official blessing. While on the topic of specs, I'm curious to know what the specification status is for other elements in the packaging landscape, such as Warehouse or Twine - are there any PEPs specifying anything new that they do over existing PyPI/distutils, or is there nothing new over and above existing code other than (no doubt improved) reimplementation of existing functionality? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadata 2.0: Is there a formal spec for a requirement?
From: Paul Moore p.f.mo...@gmail.com PS This isn't about pressuring anyone to write such modules. If they don't exist and are needed, I'm more than willing to help write them, but only if they are intended as reference implementations, not just as yet another version module. I've no particular wish to set myself up as a competitor to setuptools and distlib :-) Well, the distlib implementation is intended to be faithful to the PEPs, and to act as a reference implementation in the absence of anything better. Some parts of distlib (including the version functionality) were blessed by python-dev to the extent that they were expected to be part of Python 3.3 (in the packaging package), though I've changed the implementation considerably, both to allow conformance with setuptools formats and to implement functionality developed in newer PEPs such as 426, 427, and 440. Is there some particular functionality you need that distlib.version doesn't provide? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadata 2.0: Is there a formal spec for a requirement?
Warehouse is currently focused on reimplementation with the future being standization and spec work for new stuff. Twine uses the same APIs as distutils does on PyPI, but it A) Verifies TLS and B) enables uploading an already built distribution instead of mandating that you build it and upload at the same time. Thanks for the update. Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadata 2.0: Is there a formal spec for a requirement?
From: Donald Stufft don...@stufft.io Technically that was a PEP 426 change. Yes, and I haven't yet changed distlib to remove support for the older foo (=X.Y) form in the earlier version of the PEP. Yea, my “problem” with distlib was always that I think Vinay and I wanted two different things from it. I wanted a reference implementation that only came with the PEP standardized pieces, vinay wanted a library that implemented things he could use for distitil. Not quite - it's the other way around: distil is mainly a test bed for distlib, to verify that the latter's functionality is usable in practice. What I want is a rather more modern packaging system than we presently have - for example having to download archives in order to determine dependencies is, shall we say, sub-optimal. I want to move away from setup.py, towards declarative metadata, while offering a migration path (which 3.3 packaging didn't). While they're not perfect, distlib/distil allow me to install stuff without executing setup.py on target systems a lot of the time, and ISTM that's moving things in the right direction. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] additional paths in wheel
Provide a mechanism for recording the actual paths used during installation (in wheel.json). Well, there is already a list of paths used which is used during uninstallation (RECORD). It would make sense to record all the installed files there, and no reason to duplicate that elsewhere. In general, IMO importable (executable) Python files should be avoided for this kind of use case - JSON files in the .dist-info would be better. We see from the .pth file and setup.py examples that executable Python for these types of usages can be a bit of a double-edged sword. Possibly we should replace RECORD, etc. with JSON equivalents (to be neat and tidy). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] defining requirements on pypi
PyPI won’t show metadata for Wheels either unless you upload them with Twine Why is that, exactly? Is there something alternate tools would need to do to achieve the same thing? Regards, Vinay Sajip From: Donald Stufft don...@stufft.io To: Daniel Holth dho...@gmail.com Cc: Brett Graham brettgra...@gmail.com; DistUtils mailing list distutils-sig@python.org Sent: Thursday, 28 August 2014, 23:11 Subject: Re: [Distutils] defining requirements on pypi PyPI doesn’t parse anything, metadata is passed alongside the file upload. This means that for sdists it would just need modified to communicate that information to PyPI. PyPI won’t show metadata for Wheels either unless you upload them with Twine. On Aug 28, 2014, at 6:02 PM, Daniel Holth dho...@gmail.com wrote: Pypi only parses requirements from wheels for display but this doesn't affect installation. On Aug 28, 2014 7:49 AM, Brett Graham brettgra...@gmail.com wrote: Hi, I feel like these are stupid questions but I cannot seem to find a straight answer. In brief, 1) what is egg-info/requires.txt used for? 2) how do I properly define requirements for pypi? The details are: I'm updating some packages on pypi and am having difficulty defining requirements. One of the packages in question is: pypi.python.org/pypi/jsui I'm initially defining the requirements in a requirements.txt that then gets parsed in setup.py and install_requires gets set to the contents of requirements.txt. Some of the output from python setup.py sdist build is below. The resulting requires.txt in jsui.egg-info is: flask wsrpc However, when I upload this to pypi with python setup.py sdist upload I'm not seeing these requirements listed nor does pip installing the package install the requirements. Thanks for your help. python setup.py sdist build partial output running sdist running egg_info writing requirements to jsui.egg-info/requires.txt writing jsui.egg-info/PKG-INFO writing top-level names to jsui.egg-info/top_level.txt writing dependency_links to jsui.egg-info/dependency_links.txt reading manifest file 'jsui.egg-info/SOURCES.txt' writing manifest file 'jsui.egg-info/SOURCES.txt' running check warning: check: missing required meta-data: url creating jsui-0.0.1 creating jsui-0.0.1/jsui creating jsui-0.0.1/jsui.egg-info making hard links in jsui-0.0.1... hard linking README - jsui-0.0.1 hard linking setup.py - jsui-0.0.1 hard linking jsui/__init__.py - jsui-0.0.1/jsui hard linking jsui/serve.py - jsui-0.0.1/jsui hard linking jsui.egg-info/PKG-INFO - jsui-0.0.1/jsui.egg-info hard linking jsui.egg-info/SOURCES.txt - jsui-0.0.1/jsui.egg-info hard linking jsui.egg-info/dependency_links.txt - jsui-0.0.1/jsui.egg-info hard linking jsui.egg-info/requires.txt - jsui-0.0.1/jsui.egg-info hard linking jsui.egg-info/top_level.txt - jsui-0.0.1/jsui.egg-info Writing jsui-0.0.1/setup.cfg Creating tar archive removing 'jsui-0.0.1' (and everything under it) running build running build_py ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pypi suggestion
- Original Message - From: Nick Coghlan ncogh...@gmail.com Most depended on, but that's currently a pain to extract. Making it easier to do that kind of analysis is actually one of my goals for metadata 2.0. Here's my list of the top 25, based on the PyPI metadata that I maintain (not rigorously tested, but seems plausible): 6265:setuptools 2371:django 2178:requests 1007:zope.interface 1003:numpy 873:pyyaml 860:lxml 784:simplejson 758:zope.component 734:six 679:flask 678:nose 677:argparse 633:sqlalchemy 569:jinja2 551:python-dateutil 471:zope.schema 456:sphinx 433:pytz 427:mock 414:scipy 405:distribute 396:zc.buildout 364:docopt 363:pil The number against each dependency represents the number of distinct PyPI projects which list the dependency as a run-time requirement (:run: or :meta:). It's possible that some projects declare things as run-time dependencies when they really mean build-time: I'm not sure that there are really over 400 projects which are Sphinx plugins / extensions. Just throwing this out there as it may be of interest. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pypi suggestion
- Original Message - Also, how does this measure optional dependencies? I capture extras information where provided, but it all depends on what's declared in the metadata (via install_requires, setup_requires, test_requires). For example, html5lib doesn't declare any dependencies other than six, so that's all my code would know about. Interesting (not surprising, though) that pyyaml is so high, and that docopt is on there. Also notable that pil is in there. (Does your data distinguish pil and pillow? I understood the common view these days was to use pillow in place of pil). The data hasn't been thoroughly validated - PyPI is a vast repository of information. While I've checked the sanity of my metadata for projects that I work on / use, it's too large a body of data to check everything, and it's certainly possible that there are a few bugs lurking in my metadata scraping code. The code doesn't do anything clever like aliasing PIL/Pillow or distribute/setuptools - it's the bare dependencies as declared in zillions of setup.py files. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setup_requires and install_requires
From: Daniel Holth dho...@gmail.com Just build a wheel first. Then numpy is installed twice but only built once. Isn't that just papering over the cracks? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [ANN]: distlib 0.1.9 released on PyPI
I've just released version 0.1.9 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Fixed issue #47: Updated binary launchers to fix double-quoting bug where script executable paths have spaces. Added ``keystore`` keyword argument to signing and verification APIs. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.9 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
Correct me if I'm wrong, but I've a feeling you once said you'd tested distil against all the packages on PyPI (which is a mammoth task, so I could easily be wrong...) Not fully tested in the sense you mean - that *would* be a mammoth task :-) However, I have tried to make declarative metadata from all releases on PyPI :-) and then distil uses that metadata to build / package to wheel, install etc. I use distil wherever possible, and outside of NumPy related work, it has worked as well as pip on the modest selection of PyPI projects that I use, and better than pip in the sense that I can see all dependencies needed before downloading anything. Many packages presented no problems during smoke testing - e.g. Django (lots of package data), Jinja2, hiredis (C extension), Assimulo (Fortran extensions). By no problems I mean that the same files were installed by distil in the same places as was done by pip (IIRC - it was a while ago). But there are a lot of important packages that distil can't handle, such as NumPy, because they e.g. generate sources in their setup.py, which means that a purely declarative approach doesn't currently work. That's not to say that such an approach *couldn't* work - it most definitely could, with a sensible hook mechanism where needed for publisher-defined code, and a better approach for build metadata (still to be addressed) and installation metadata (which we have now got a good handle on). People have gone with setup.py as that's all there was, but that's just circumstance rather than a technically necessary approach. There are migration issues, of course, but I would guess that for the vast majority of packages, the auto-generated metadata approach that distil uses is good enough. Of course, more complex / important packages are exceptions, and would need more work (corresponding to their complexity) to fit into a declarative approach. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I thought there was a general consensus to move *away* from the need to run setup.py when it's practicable to do so. There was no choice about this with distutils and setuptools, but going forwards, I'm not sure about impossible to install as long as *some* mechanism is there for package publisher-defined code to run at installation time. However, setup.py as it is now is a complete free-for-all where anything goes, which I don't think is a good thing. Many PyPI packages are installable with distil (which doesn't use setup.py at installation time), and that includes packages with C extensions, Cython, and even Fortran. The packages distil has problems with are those that do significant things in setup.py, such as moving files around in the source tree, generating new source files, subclassing distutils so you can't see what the actual operations being carried out are, etc. I'm fairly sure that all of these things can be accommodated using installation time-hooks with sensible APIs rather than the ad hoc-ery of setup.py. Of course I'm not suggesting that publishers have to rework existing releases - it's just that the setup.py scheme is a bit archaic and not entirely problem-free. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
From: Paul Moore p.f.mo...@gmail.com PS To my knowledge, neither distil nor easy_install implement PEP 438 I haven't done any work to make distil production-ready in any sense, and PEP 438 compliance would be a part of such activity. I've been more focused on the build/meta-build functionality, as that needs more attention than the installer/uninstaller functionality. I've been meaning to address some items relating to PEP 426, but unfortunately time has been in shorter supply than it is usually :-( Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Comments on PEP 426 and 459
That would be the == in requires: [SoftCushions == 4] which IIUC in the current PEP would be allowed in meta_ but disallowed in run_requirements. Okay, I see it now. Thanks. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Versioned entry points
Does anyone have any thoughts? Note that there is one situation where scripts for multiple Python versions reside in the same directory: per-user site-packages (PEP 370). If you install a package which has scripts and you don't write versioned scripts, a second installation of that package (with a different Python version) will cause the first script to be overwritten. Much of the time this won't matter, but there might be scenarios where it does. It may also give a problem when uninstalling, or when verifying the installation (as the hash of an overwritten script may not match what's expected, as per the original installation). To avoid overwriting, one would need to write versioned scripts *only* :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
Step 3 (what, if anything, should replace the setup.py CLI as the build system abstraction?) [...] research/documentation consolidation project at this point than it is a design project. FYI, I've been experimenting with a design approach for a build system in distil. Currently it only covers what distutils/setuptools already do - i.e. build_py functionality + build_ext functionality covering libraries, extensions, setuptools Features, Cython / Pyrex and Fortran / f2py. Not too shabby, but clearly that's not enough. Instead of compiler classes (the abstractions used in distutils / setuptools), distil's approach focuses on a command line with variable placeholders - this allows just about anything to be plugged in for custom builds. There's still work to be done on it, but the initial approach looks promising. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
1. Formally decouple the setup.py CLI from the distutils command system Agreed - it looks like a compatible CLI needs to be part of the transition to any new system (I assume that's what you meant). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
This strategy does not generally try to eliminate arbitrary code execution during builds - builds are an inherently arbitrary-code process. But once the build has happened most installs should work without arbitrary code execution. I don't think builds should be a *completely* arbitrary-code process. I understand well that user-defined code should be accommodated, but IMO this should be within a declarative framework with well-defined hooks, otherwise it will ultimately lead to the same problems that setup.py has. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [ANN]: distlib 0.1.8 released on PyPI
I've released version 0.1.8 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed issue #45: Improved thread-safety in SimpleScrapingLocator. * Fixed issue #42: Handling of pre-release legacy version numbers now mirrors setuptools logic. * Added exists, verify, update, is_compatible and is_mountable methods to the Wheel class (the update method fixed issue #41). * Added a search method to the PackageIndex class. * Fixed a bug in the Metadata.add_requirements method. * Allowed versions with a single numeric component and a local version component (tracking changes to PEP 440).* Corrected spelling of environment variable used for the stub launcher on OS X. * Avoided using pydist.json in 1.0 wheels (bdist_wheel writes a non- conforming pydist.json). * Improved computation of ABI tags on Python versions where SOABI is not available, and improved computation of compatibility tags on OS X to allow for multiple architectures and older OS X versions. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.8 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
Daniel Holth dholth at gmail.com writes: extensions without using distutils. The problem of invoking the compiler has been solved by many build systems and is not just a unique and mysterious distutils feature. Did someone say it was? Building extensions is something distil does too, and without using distutils or setuptools. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] tourist here, with a dumb RTFM question
Michael Bayer mike_mp at zzzcomputing.com writes: I don’t know the metadata format in pep426 well enough to comment (as I wanted to use it one day and found that it seemed to still be pretty much vapor), but I’ll reiterate: these are source distributions, not binaries or wheels or anything like that. A source distro has to build, and builds need options.A metadata format that purports to represent metadata about a source distribution and does not support flags is a broken metadata format. PEP 426 only covers installation metadata - its scope does not cover source distributions (there will be another PEP covering that). Apart from the most recent changes being not yet implemented and some open issues relating to hooks, PEP 426 seems to cover installation metadata reasonably well. However, sdist metadata is a different kettle of fish. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] tourist here, with a dumb RTFM question
Donald Stufft donald at stufft.io writes: 2. Use older versions of setuptools That's not really an answer, unless downgrading works. For example, a recently created venv would contain the latest setuptools, and perhaps it would be required by other distributions in the venv. How then would e.g. SQLAlchemy specify that an older setuptools version should be used? Would pip downgrade the venv's setuptools to an older version? Even if it did, might that not break other distributions in the venv? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] tourist here, with a dumb RTFM question
From:Michael Bayer mike...@zzzcomputing.com OK so why does PEP-426 compatibility imply removal of command line switches from setup.py files ? As far as I know, it doesn't. My distil tools complies with a fairly recent version of PEP 426 and AFAIK has no problem building / installing SQLAlchemy with / without extensions, but it uses an extended set of metadata which is not specified in any PEP - for example, http://www.red-dove.com/pypi/projects/S/SQLAlchemy/package-0.9.3.json This is just FYI - it's experimental metadata [which has some redundancy for historical reasons, to be tidied up in the future] and is extracted from what is passed to setup.py - it includes the PEP 426 metadata as a subset (index-metadata), and the extra stuff is needed for building. This is used by the distil tool, but there's no support for any specific command-line flags to exclude the building extensions. (That's not because of PEP compliance - it's just that I haven't needed that level of control in the tool.) Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The PEP 426 defined metadata version will be metadata 3.0
but my question was what are you adding, if anything, that warranted it having Metadata-Version: 2.0? I believe that the fields 'Private-Version', 'Obsoleted-By', 'Setup-Requires-Dist', 'Extension' and 'Provides-Extra' were added on top of the 1.2 metadata in an early version of PEP 426, before the move to JSON. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The PEP 426 defined metadata version will be metadata 3.0
There is always going to be multiple files, it’s kind of silly to tie the definition of the dist-info directory to the pydist.json when that’s perhaps not the file you care about how to interpret. How do you interpret the RECORD file? The INSTALLER file? The versioned definition of the dist-info directory would say, “RECORD file is an XYZ file” “pydist.json is an ABC file”. It’s the only sane way to handle the case where you can have an unbounded number of unknown files in the directory. Those files are either implementation specific (in which case, not our responsibility) or they are standardised, in which case there will be a PEP (376 for the files apart from pydist.json). If it is felt that the formats in PEP 376 should be future-proofed, then perhaps they should be coalesced into a single JSON file with its own metadata version and schema definition (/ducks ;-) The PEP 376 files are INSTALLER, RECORD, REQUESTED and RESOURCES (which AFAIK isn't used). I don't know of any reason why they couldn't be merged into a single file. Any implementation specific caches of parts of those files (e.g. exports, for speed of scanning) would be up to individual implementations to maintain (and perhaps belong in implementation-named subdirectories of dist-info, to avoid conflicts with other implementations). Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The PEP 426 defined metadata version will be metadata 3.0
If Vinay is happy with that last option for handling the current bdist_wheel misbehaviour that would mean we could leave the metadata version alone I've changed distlib to ignore pydist.json for wheels with version 1.0, where METADATA is used instead. Note that bdist_wheel also puts a metadata version of 2.0 in the METADATA file. IMO versioning the pydist.json is equivalent to versioning the .dist-info directory - it certainly doesn't seem right to put a metadata version in the directory name itself, as it would effectively be noise almost all of the time. I think it may be premature to promote wheels (as pythonwheels.com is doing with vigour) - nothing wrong with it in principle, of course, it's just the timing which seems wrong. Before such promotion to the community at large, It would seem advisable to get the PyPA house in order by (a) finalising the PEPs which are in limbo because of the focus on getting 3.4 out, (b) ensuring that the 1.1 spec for wheel covers any shortcomings which have been identified in 1.0, as well as more extensive testing for the implementations, and (c) revisiting accepted PEPs such as 425 to ensure that things are tightened up where needed (for example, there are some problems relating to tags, one of which IIRC Brian Wickman brought up a little while ago - about whether tags should follow the PyPy version numbering rather than the Python language version numbering). Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The PEP 426 defined metadata version will be metadata 3.0
It makes perfect sense to version the dist-info directory. You don’t know how to interpret the files inside that directory without it. You have to rely on heuristics and guessing. Not if you specify up front how it will work, which is doable. It's not clear to me if you mean putting the metadata version in the directory name itself - if so, that's a -1 from me. It doesn't actually make things better as far as the PEP 376 database logic goes, given that a site-packages could have any number of dist-info dirs which could have been installed with different metadata versions. If you mean putting some marker inside the directory, then pydist.json is as good a place as any, because as packaging evolves, the chances are that pydist.json will need to contain metadata version information telling how to process it, and there's little point in providing the information in two places. All of the legacy stuff is in separate files purely for speed of processing or through historical accident, but those are implementation details which should be under the hood as far as possible. Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The PEP 426 defined metadata version will be metadata 3.0
PEP 425 explicitly covers that. It says The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead. PyPy should probably use its own versions here pp18, pp19. The probably leaves some room for doubt as to what exactly is meant: both wheel and distlib use py_version_nodot, and the last sentence I quoted looks provisional rather than definitive. Regards, Vinay Sajip___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Cross-platform way to get default directory for binary files like console scripts?
Is there cross-platform way to get default directory for binary files
(console scripts for instance)
Well, there's
$ /tmp/venv/bin/python
Python 3.3.0+ (3.3:c28b0b4e872b, Mar 25 2013, 17:51:34)
[GCC 4.6.1] on linux
Type help, copyright, credits or license for more information.
import sysconfig; print(sysconfig.get_path('scripts'))
/tmp/venv/bin
$ python
Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
[GCC 4.6.1] on linux2
Type help, copyright, credits or license for more information.
import sysconfig; print(sysconfig.get_path('scripts'))
/usr/local/bin
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))
On Tue, 4/2/14, Carl Meyer c...@oddbird.net wrote: Sadly I can't offer that specific problem, because AFAIK we never tracked it down. It was a recurring problem in IRC where people would report using -E and pip would fail to take its action within the indicated virtualenv, instead impacting their global environment. After trying and failing a number of times to isolate and reproduce that problem, I decided not to continue spending time on that, and that it would be more robust to rely on the natural isolation provided by just using pip inside the env, which required no additional code to debug and maintain. (Pip had already been installed inside every virtualenv for quite a while by then.) I regret the way I handled that particular backwards-incompatibility (not providing a deprecation path), but I think the technical choice was a good one. Identifying code that is a bug-report-magnet and choosing a viable approach where that code can be removed entirely is a sound technical choice, not a matter of politics or religion (I had no political or religious motivations in removing pip -E). I haven't made any comment about pip -E and how / why it was removed - until Donald's post about it, I didn't even know about it. To me, the correct solution would have been to isolate the bug and remove it, but I completely understand the pragmatic approach you took. But that leaves a dark corner where people might be afraid to experiment in this area with pip, because it's somehow scary. This illustrates in part why working *on* pip is painful for me - it's not exactly a large code-base, but the way it and its tests are structured makes debugging it needlessly hard. In an environment where this is all time-constrained volunteer activity, it is not at all surprising that you took the action you did. As far as I can see, you haven't presented any technical issues with the current approach, just a feeling that it's inelegant (is this perhaps a religious feeling? :P). My feeling differs; I find it elegant to rely only on the virtualenv's built-in isolation, and inelegant to require extra restart-in-env code. I didn't say that the current approach didn't work - it's just that IMO it's inelegant to require something to be installed into every virtual environment, just to be able to install packages into it. Seemingly we've gotten so used to it, like python setup.py install, that it seems like part of the natural order of the universe :-) It's not especially elegant to have to restart in a venv, but IMO it's the lesser of two evils. It is something that we currently have to do (virtualenv does it too, with -p) but if a more elegant approach were found, I'd certainly use it. The other inelegance (install pip in every venv) is not essential, as I've shown. I don't know exactly what approach pip took to implement -E - was it similar to the approach in my snippet? Certainly, if anyone were to point out any drawbacks with the logic in that snippet, I'd be interested. But if pip took a different approach to implement -E, it's possible that there was a problem with the details of that particular approach, and the problems with pip -E don't provide any evidence to invalidate the restart approach in general. I agree that tastes differ, and technical arguments can perfectly validly be about aesthetics in my view, but I don't regard these sorts of arguments as religious. If religion is to be mentioned in connection with pip, then perhaps it's fair to say that pip is the sacred cow of Python packaging ;-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))
Carl Meyer carl at oddbird.net writes: Sadly I can't offer that specific problem, because AFAIK we never tracked it down. It was a recurring problem in IRC where people would report using -E and pip would fail to take its action within the After posting my first response to your post, I remembered that Donald had provided a link to the commit that removed the -E. From a quick look at that commit, ISTM that distil's approach to restarting is somewhat easier to reason about and debug than the approach taken by pip to implement -E: it takes place before any of the rest of the logic in the restarted program is run, so a question like why didn't restart_in_venv get called, when it should have been? doesn't arise. There are no rabbit-holes to go down. The approach in that snippet could apply to any script - it's not especially distil-specific. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv
On Tue, 4/2/14, Nick Coghlan ncogh...@gmail.com wrote: I've pointed out the technical problem with trying to rely solely on a global install on Linux: it makes it hard to use a newer version of pip than the OS provides. Installing pip into the virtual environments avoids that problem without conflicting with the OS package manager when it comes to the system Python installation. That decoupling from the distro version is a worthwhile technical benefit of the current approach. While it does mean that multiple virtual environments may need to be updated when a new version of pip is released, it *also* makes testing with different versions straightforward. Well, it's not a bad thing IMO that distil leaves no trace of itself in a venv, so no updating of any venv is required, were a new release of distil to come out. In terms of distro version isolation, I'm still not sure I get the practical problem you're talking about, so let's get into specifics. In a parallel universe, let's assume there's a tool that can install packages into venvs, without needing to be installed into them first. (It seems a desirable feature, which prompted the development of pip -E in the first place). Let's say that this tool is shipped as part of a distro and generally kept up to date by the distro's package manager. Let's say that a new version of this tool is available upstream, but not yet integrated into the distro, and that someone wants to use it instead of the distro-supplied version. Where's the problem exactly? It could just be a single file to download and run, and you could have any number of different versions of it co-existing without any particular problem. It needn't be any worse than any other type of tool where a more-recent-than-distro version is desired. I'm probably being dense, so perhaps you could outline any specific problem scenarios along the lines of a user wants to do X, but can't, because of Y where X is some aspect of installation / upgrading / uninstallation of packages in a venv and Y is something which intrinsically arises from the installation tool not being installed in the venv. That would certainly help me understand where you're coming from. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv
On Tue, 4/2/14, Nick Coghlan ncogh...@gmail.com wrote: It's a higher level of isolation than that offered by *not* installing pip into the virtual environments - the two approaches are *not* equivalent, and one is *not* clearly superior to the other, but rather there are pros and cons to both. Obviously the two approaches aren't equivalent, or there wouldn't be anything to discuss. What you haven't yet done is answered my question about what the cons are of the not-in-venv approach in the practical you can't do X, because of Y, sense. What *practical* problems are caused by a lower level of isolation exactly, which make the higher level of isolation necessary? ISTM the pip developers tried to implement -E because it was (independently of me) seen as a desirable feature, and Paul said as much, too. If it didn't work for pip, that could well be because of the specific implementation approach used, as I've said in my response to Carl. Rather than talk in abstractions like higher levels of isolation, which have an implied sense of goodness, it would be helpful to spell out specific scenarios (in the X/Y sense I mentioned before) that illustrate that goodness versus the badness of other approaches. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))
On Tue, 4/2/14, Paul Moore p.f.mo...@gmail.com wrote: The big problem with this type of thing is that pip tends to be used on lots of systems with sometimes *extremely* obscure and odd setups - what some of the Linux distros do to a standard Python install amazes me (I'm sure they have good reasons, of course...). So it's not so much a case of scary dark corners as lots of unanticipated and extremely difficult to reproduce use cases that we have to support but where the only knowledge of what workarounds are needed is encapsulated in the code. You've just restated scary dark corners in a different way which sounds like there's more to it technically. But there' s still no hard data on the table! Surely, with all the convolutions that these distros go through, can one still assume that pip-the-program-that-they-all-run assumes control via some notional main program, which is passed some arguments? If we can't assume this, please tell me why not. If we can, then it would be like this: def main(): # Do stuff with sys.argv Now, distil has a similar entry point (as innumerable other scripts do) and all I did to restart in venv was to look for a -e argument and process it before the main event: def main(): if '-e' found in sys.argv: validate the passed parameter, prepare a new command line using the interpreter from the venv and the sys.argv passed in, but minus the -e bit, since we've dealt with that here call subprocess with that command line (the restart) return - our work here is done # Do stuff with sys.argv It doesn't get much simpler. Of course there could be bugs in the code called in the if suite, but it's not doing a complicated job so it should be easy to identify and eradicate them. Now, are you saying that the bit of code represented by that if suite, which only has to process sys.argv, look for an identifiable venv in the file system and check that it seems to have a runnable Python interpreter, can be bamboozled by something distros do to their Python installations? I'm having trouble buying that as an argument, unless there is more hard data on the table. What unanticipated and extremely difficult to reproduce use cases could interfere with the operations in that if suite? It's hard to argue against voodoo, but anyone on this list could put me right with a particular circumstance that I haven't thought of, but that they have run into, which undermines my argument. It's not uncommon to see the well ... we're not quite sure what's happening in there ... better play it safe, but that seems out of place in an engineering context. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv
On Tue, 4/2/14, Nick Coghlan ncogh...@gmail.com wrote: I'm not sure how I can explain it more clearly, but I'll try. [snip] I understood the scenarios you outlined. I'm not saying what you want to do *can't* be done (it obviously can). I'm saying it adds additional complexity, because you're adding a second isolation mechanism into the mix, with a completelely independent set of ways of interacting with it. There is no separate isolation mechanism in my approach - distil runs completely in the context of the virtualenv, so that's all the isolation mechanism there is, in both cases. The difference is that distil doesn't need to be installed in the venv - but otherwise, its sys.path is the same as for any other tool in the venv, as it is running *with the venv's interpreter*, just as the pip in a venv is. Perhaps I haven't made that explicitly clear (I assumed it was implicit in restart in venv). For example, the distil run with -e has no access to system site-packages if the venv itself is isolated from system site-packages. For an example of that consider that pip install and python -m pip install are essentially equivalent commands. The status quo preserves that equivalence even inside a virtual environment, because there is no separate mechanism to be considered. Well, distil doesn't have a __main__.py and is not designed to be called with -m, but I don't see an essential difference, because the main vehicle of isolation is *the venv machinery* in *both* pip and distil cases. If the system Python site-packages is disabled in an active virtual environment, you don't need to carefully ensure that doesn't affect your install tool, because it's right there in the virtual environment with you. As I said above, distil respects venv isolation: if a venv is isolated from system site-packages, then distil code run with -e can't see system site-packages, automatically, without any careful ensuring being necessary. So from what I can see, your argument is based on an imperfect understanding of how distil works. If you'd like to understand it better I'll provide the support, but if you don't want to or can't do that - well, that's a shame, but such is life :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))
On Tue, 4/2/14, Nick Coghlan ncogh...@gmail.com wrote: That only works if the system site packages is configured to be visible inside the virtual environment - it usually isn't these days. If that were true, distil wouldn't work in such venvs, but it seems to work fine. Can you show otherwise? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))
On Tue, 4/2/14, Nick Coghlan ncogh...@gmail.com wrote: Vinay, please let it drop. You accuse us of FUD, yet have presented no evidence that your approach works flawlessly across multiple versions of Fedora, Debian, openSUSE, RHEL, CentOS, Debian, Ubuntu, Windows, Mac OS X, etc, I merely asked for a hearing, and made no claims I couldn't substantiate, and with actual code in places. But you seem determined not to give me that hearing, so I'll drop it. It's a shame you have to resort to an argument from authority. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))
On Tue, 4/2/14, Nick Coghlan ncogh...@gmail.com wrote: No hearing? You haven't answered *any* of our technical objections, instead dismissing them all as irrelevant. Because you advanced no specific technical objections, instead invoking generalisations. If you prefer the status quo because it's easier to explain (i.e. there's nothing to explain), that's not a technical objection in my book. However you want to look at it, that's fine by me. The example of distil shows that the status quo is not the only way of doing things, and I'll remain interested in any feedback from anyone regarding specific failures of distil in terms of practical workflows that aren't possible. Doesn't need to be installed in a venv or runs as a single file isn't a failure from my POV, so we'll have to agree to disagree. But, as you say, let's drop it. Jonathan Swift's observation applies. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Fri, 31/1/14, Brian Wickman wick...@gmail.com wrote: There are myriad other practical reasons. Here are some: Thanks for taking the time to respond with the details - they are good data points to think about! Lastly, there are social reasons. It's just hard to convince most engineers to use things like pkg_resources or pkgutil to manipulate resources when for them the status quo is just using __file__. Bizarrely the social challenges are just as hard as the abovementioned technical challenges. I agree it's bizarre, but sadly it's not surprising. People get used to certain ways of doing things, and a certain kind of collective myopia develops when it comes to looking at different ways of doing things. Having worked with fairly diverse systems in my time, ISTM that sections of the Python community have this myopia too. For example, the Java hatred and PEP 8 zealotry that you see here and there. One of the things that's puzzled me, for example, is why people think it's reasonable or even necessary to have copies of pip and setuptools in every virtual environment - often the same people who will tell you that your code isn't DRY enough! It's certainly not a technical requirement, yet one of the reasons why PEP 405 venvs aren't that popular is that pip and setuptools aren't automatically put in there. It's a social issue - it's been decided that rather than exploring a technical approach to addressing any issue with installing into venvs, it's better to bundle pip and setuptools with Python 3.4, since that will seemingly be easier for people to swallow :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Nick Coghlan ncogh...@gmail.com wrote: FWIW, installing into a venv from outside it works fine That's what I'm saying :-) However, it's substantially *harder* to explain to people how to use it correctly that way. But distil installs into venvs (both PEP 405 and virtualenv) without needing to be in there. It's as simple as distil -e path/to/venv install XYZ So I'm not sure what you mean by use correctly or what exactly is hard to explain. Distil doesn't do anything special - after all, installing a distribution just means moving certain files into certain locations. If you know that you're installing into one of (a) a specific venv, or (b) user site-packages, or (c) system site-packages, then you know what the installation locations are without doing anything especially clever, don't you? Of course, building and installation are commingled in distutils and setuptools - that's build-from-source-is- all-you need myopia right there - but I don't see how that prevents installation-from-without or forces the usage of installation-from-within, at least at a fundamental level. In theory you could change activation so that it also affected the default install locations, but the advantage of just having them installed per venv is that you're relying more on the builtin Python path machinery rather than adding something new. But virtualenv solves an analogous problem when creating venvs with different interpreters by reinvoking itself with a specific interpreter. Distil does something similar, but that's more to do with the fact that code in setup.py sometimes takes different paths depending on which version of Python is running it. So while it's wasteful of disk space and Disk space is cheap, so that's not the issue, it's the lack of elegance. Avoiding work by leveraging existing capabilities is a time honoured engineering tradition, even when the simple way isn't the most elegant way. I'm pretty pragmatic, so I agree with this point in general, but in this case I think you're side-stepping the issue of technical debt. We're talking about file-system manipulations, after all, not anything more esoteric than that. Any approach which perpetuates the python setup.py install way of doing things is, to me, not a good thing, which I think has been recognised from the good old days when Tarek started taking a fresh look at packaging. I have now realised the much-reviled (for good reason) *.pth files actually have a legitimate use case in allowing API compatible versions of packages to be shared between multiple virtual environments Do you mean ref files, as discussed on import-sig? Sure, but that's a tangential issue in my view. Worthwhile pursuing, certainly, but an incremental improvement over what we have now (and actually less necessary in the parallel universe where wheels are more like jars and not reviled for that just because of Java - ewww). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Noah Kantrowitz n...@coderanger.net wrote: In all but a tiny number of cases, you could use a symlink for this. Much less magic :-) That's POSIX is all there is myopia, right there. While recent versions of Windows have symlinks more like POSIX symlinks, XP only has a stunted version called reparse points or junction points which are not really fit for purpose. I think you'll find that XP environments are found in rather more than a tiny number of cases, and even though Microsoft has end-of-lifed XP in terms of support, I fear it'll be around for a while yet. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Nick Coghlan ncogh...@gmail.com wrote: I'm talking about easing the cognitive burden on developers. The pip/virtualenv model is that once an environment is activated, developers don't need to think about it anymore - they're in that environment until they explicitly switch it off. But distil deals with this:: $ pyvenv-3.3 /tmp/venv $ source /tmp/venv/bin/activate (venv) $ distil --ve [...] Python: 3.3 [...] So, distil is using the venv's Python. Let's deactivate and try again: (venv) $ deactivate $ distil --ve [...] Python: 2.7 [...] So we're now using the system python - so far, so good. By installing pip *into* the environment, then it is automatically affected by the same settings as all other Python components, so pip install library inside an activated virtual environment *just does the right thing*, and it does it in a way that doesn't require any additional custom machinery or testing. Likewise, with distil: $ source /tmp/venv/bin/activate (venv) $ distil list (venv) $ distil install sarge Checking requirements for sarge (0.1.3) ... done. The following new packages will be downloaded and installed: sarge (0.1.3) Downloading sarge-0.1.3.tar.gz to /tmp/tmpn01ls2 [...] Building sarge (0.1.3) ... [...] Installing sarge (0.1.3) ... [...] Installation completed. (venv) $ distil list sarge 0.1.3 (venv) $ So, I am at a loss to see your point. Making any installer automatically respect an activated virtual environment without installing it into that environment is no doubt *feasible*, but just installing the installer *into* the environment has the same effect with requiring any additional engineering or testing effort. Not only feasible, there needing to be no doubt about it ;-), but also not rocket science, and demonstrated above. Distil isn't doing anything especially clever, so I don't understand what you mean by any additional engineering or testing effort. ISTM you're making things seem more scary than they are. Of course, if you find some specific *installation* scenario that distil doesn't cover, I'll be all ears. I emphasised *installation* to distinguish from building - distil eschews running setup.py during installation, since if it did that then it would be no different from pip in that respect, so there'd be no point to it. So, distil install can't deal with cases of substantive logic in setup.py - but that's not to say that distil can't deal with setup.py at all. Instead, that functionality is provided in other distil commands which deal with building wheels from sdists. They could be relatively easily merged into an install command that works like pip, but I thought the point was to separate building from installation, so that's why distil is like it is at the moment. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
, then so be it - nobody ever got fired for recommending pip. The Python community will get the packaging infrastructure that it deserves, just like a populace does with elections :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Nick Coghlan ncogh...@gmail.com wrote: My point is that doing it the way virtualenv/pip did avoided a bunch of design work and associated testing, and reduced the opportunities for bugs - when you're trying to get things done with limited resources, that's a sensible engineering trade-off to make. A bunch of design work makes it seem a lot more complicated than it really is. Your suggestion comes across like an ex post facto rationalisation of a decision which was perhaps more truly based on social concerns than technical ones. Note that I've developed distil as part of my volunteer activities - it doesn't pay any of my bills - and on my own. And you're telling me about how to get the best out of limited resources? :-) That said (and this is a point that hadn't occurred to me earlier), it's also worth noting that not only does the bootstrapping approach work well enough in most cases, but it also solves the problem of being able to easily have a newer (or older!) pip in virtual environments than is provided by the distro package manager on Linux systems Eh? The problem of having a newer or older pip in a venv only exists because pip needs to be in a venv ... so I don't see the relevance of this to our earlier discussion. Since distil doesn't occupy a space in venvs,. the concern of a system version being older or newer than that in a venv doesn't arise. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Donald Stufft don...@stufft.io wrote: So perhaps it isn't that pip is lacking a useful feature, it's that pip tried it, as you did with distil, and due to pip's much larger user base it got a much more thorough real world experience with it and it turned out to be ultimately more trouble than it was worth? It could also have been a problem with how pip wrote it of course, and perhaps you've discovered a better way of writing it. However at the time it's obvious that the pain it was causing was enough to cause the pip developers to break backwards compatibility to remove that feature. Why exactly was it removed? Well I can't find the reasoning beyond that commit so we'd need to get Carl or perhaps Jannis to come and tell us if they still remember. However it's clearly not that pip didn't just try or think of this, it did and it ultimately removed it. Sure, and if they did find some problem which is also present in distil (i.e. due to a fundamental drawback of the approach) then I'll put my thinking cap back on. It'll be worth having that problem on record to avoid any repetition of work in the future. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Donald Stufft don...@stufft.io wrote: Additionally I'd point out that, looking at how you've implemented it, it seems to depend on the fact that distil is a single file and thus doesn't need installed into site-packages. No, I don't believe that's the case - I developed (and continue to develop) distil without running it as a single-file executable, since most of its code is in a zip when deployed as a single file, so not readily debuggable. The deployment of distil as a single file, though seen by some as controversial (because unusual), is merely meant as a convenience. Any other deployment mechanism would mean using pip to install it, which kind of defeats the purpose :-) Finally I haven't touched distil because you've been less then forthcoming about it's own code. I realize it's just using zip tech to bundle it and thus I can unpack it and read it but: Well, it takes less commitment to try it out as a black box and report failures or usability drawbacks than to dig into its source code, but there hasn't been a lot of feedback, so I really doubt if that's a big barrier to people trying it out. I don't believe pip users spend a lot of time looking into pip source code. As you say, the code is inspectable, though not as readily as it would be if it were on GitHub, say. I haven't given particular thought to the license since I haven't published it, but there's no conspiracy to keep anything secret or proprietary, and nothing is obfuscated. I've been looking for usability feedback and some of the features are still experimental (not so much on the installation side, more on the build side), so it's premature to open it up fully: I'm not looking for code contributions, as distil is still a test-bed for distlib and some ideas about how to improve the user experience. If that means some people lose interest, I'm OK with that: I've contributed plenty to open source and I expect to continue to do so. I just don't subscribe to the idea that somehow everything magically gets better when it's put on GitHub. Note that distlib is licensed under a valid OSS license, but it hasn't made much difference in terms of the amount of feedback I've received. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Donald Stufft don...@stufft.io wrote: I never said anything gets magically better on GitHub, I told you the reason why *I* haven’t bothered to try it. And I'm OK with that. Distlib is a significantly different case, it’s a library that is going to be directly useful for a very small population of people. The vast majority of people do not have any need or desire to directly work with distlib. And your point is? This is distutils-sig, not python-list: it's the only reasonable place to try and solicit feedback for something like distlib. I'm not *complaining* that I haven't had much feedback - just stating it. To state the bleedin' obvious, I don't expect to have any say in how other people spend their time, and I hope I didn't give any contrary expectation. I can guarantee you that if distlib wasn’t released under an OSS license that pip wouldn’t have used it Sure, but I assume it wasn't used *just* because of the license. It has to be useful in some way too. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Brett Cannon br...@python.org wrote:
PEP 302 tried to unify this with get_data() and set_data()
on loaders, but prior to Python 3.3 you just didn't have any
guarantee that __loader__ would even be set, let alone
have a loader with those methods. Paul can tell me if my
hunch is off, but I assume the dream was that people would
do `__loader__.get_data('asset.txt')` instead of
`os.path.join(os.path.dirname(__file__), 'asset.txt')` to read
something bundled with your package. But as Brian has
pointed out, people just have habits at this point of assuming
unpacked files and working off of __file__.
Right, and one question is whether we need to educate people
away from this way of doing things. In the distlib.resources module
I've tried to work around things like the absence of __loader__
that you mention, and to provide a uniform API for accessing
package resources whether in file system or zip, using the
PEP 302 machinery whenever possible, and filling in some of
the gaps that pkgutil has and which pkg_resources attempts to
fill.
My pref is [...] and really pushing people to use the loader APIs
for reading intra-package files.
+1
Maybe the Packaging Users Guide could have a
Recommended Deployment Strategies section
[...]
(especially if the instructions work on all platforms, i.e. no
symlink discussions)
+1 again.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Donald Stufft don...@stufft.io wrote: There's also the problem when you need to give a file that you have packaged as part of your thing to an API that only accepts filenames. The standard ssl module is one of these that i've run into recently. Yes, this is one of the pkgutil gaps I referred to in my reply to Brett, which pkg_resources and distlib.resources both address. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Noah Kantrowitz n...@coderanger.net wrote: Junctions on Windows are actually more flexible than POSIX symlinks, and I have in fact used them for Python packages before when doing Django dev on Windows XP. Why do you say they're more flexible? They work as directories/mount points and AFAIK you can't use them for files. Plus, what Paul said about the elevation issues (not a factor for XP, I know, but would need to be taken into account for any solution across multiple Windows versions). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Brett Cannon br...@python.org wrote: Yes, that is definitely a design flaw in the ssl module that should get remedied. Did you file a bug to add a new API (whether new function or new parameters) to accept a file-like object or string instead? While that might improve flexibility in how you use the ssl module, it sadly won't solve the problem in general. Some third party APIs will expect file paths, whereas others will require passing OS-level handles, rather than file-like objects - thus effectively requiring a file which you can open with os.open(). There are other Python APIs which need a pathname - I know imp.load_dynamic() will be familiar to you :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Donald Stufft don...@stufft.io wrote: distil isn't licensed so that I can even legally *use* it as far as I understand the law. Oh really? I hadn't realised. I had thought my announcement in March inviting people to try it out would suffice, but IANAL. Having a single-file deployment means having a comprehensive license statement is tricky, but I will think on it. Not that it makes a difference to you, of course, but thanks for pointing it out. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Donald Stufft don...@stufft.io wrote: distil isn't licensed so that I can even legally *use* it as far as I understand the law. From a quick Google search, I got the following from Daniel J. Bernstein's website at http://cr.yp.to/softwarelaw.html - the page entitled Software user's rights: Free software --- What does all this mean for the free software world? Once you've legally downloaded a program, you can compile it. You can run it. You can modify it. You can distribute your patches for other people to use. If you think you need a license from the copyright holder, you've been bamboozled by Microsoft. As long as you're not distributing the software, you have nothing to worry about. I don't see anything illegal about downloading from the BitBucket page I linked to in my original post. So, it doesn't seem all that clear cut ... I suppose it depends on the jurisdiction, too. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
On Sat, 1/2/14, Brett Cannon br...@python.org wrote: No one is talking about solving this problem overnight. But if we don't want to bother putting the effort in now then the zipimport module might as well get tossed as this is a constant issue that people are not designing APIs to support. Whoa - nobody said that in-Python APIs of this nature shouldn't be improved (as Christian says he's already done with this one). I was pointing out that there are also third-party APIs that we don't control which people will want to use, meaning that the need [for an API] to copy in-zip resources to the file system will be around for quite some time. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))
Paul Moore p.f.moore at gmail.com writes: I *would* like to see the various technical issues and implications in the API documentation, though. The implications and limitations, and in particular the manual cache management requirements, need to be made explicit. I've updated the documentation, see http://distlib.readthedocs.org/en/latest/reference.html The docs on distlib.util.get_cache_base have the info on the cache and cleanup thereof, including handling of no-home-directory, security implications, open files in Windows etc. I've also updated the distlib.resources.Cache documentation to link to get_cache_base. I've added is_mountable and is_compatible methods to Wheel, and added/ updated the docs for them and the mount() method accordingly, and referenced get_cache_base as the place where extensions are extracted to. More detailed suggestions for improvements are welcome. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
Thomas Heller theller at ctypes.org writes: It uses the _memimporter extension which uses code from Joachim Bauch's MemoryModule library. This library emulates the win32 api function LoadLibrary. When this was mentioned on python-dev last March you said it was 32-bit and Python 2.x only, is that still the case? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))
On Fri, 31/1/14, Paul Moore p.f.mo...@gmail.com wrote: Thanks for this. I see no means to set the cache that will be used by wheel.mount. It's not as configurable as the distlib.resources cache (needs a method to be overridden, which isn't ideal), but I'll look at making it follow the same scheme. Are you meant to set the global distlib.resources.cache value to a user- created Cache instance if you want to control the location of the cache? Yes. Is the default cache created on demand? In other words, if I set up my own cache on application startup, will the %LOCALAPPDATA%\.distlib directory still get created? No - currently it's created on module import by a line 'cache = Cache()'. This can easily be changed to defer the cache creation until it's needed, allowing user code to set a custom Cache. When I do this, no .distlib directory will be created in %LOCALAPPDATA% unless no other cache has been set, and it's needed. So I don't have an agenda here, I'm just curious. I hope that answers things! I will update distlib's cache usage logic shortly to (a) allow better control over the wheel mount cache location and (b) create caches on demand rather than on module import. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
On Fri, 31/1/14, Thomas Heller thel...@ctypes.org wrote: This limitation does no longer exist; it works in 2.x and 3.x as well as on 32-bit and 64-bit. That's good to know. Of course, as it's platform specific, it can't be used as a generic C extension import facility, though I suppose that if/when such a capability becomes available on mainstream platforms, the memimporter API could be adapted to cover it. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On Thu, 30/1/14, Ralf Gommers ralf.gomm...@gmail.com wrote: Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying. If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
On Thu, 30/1/14, Paul Moore p.f.mo...@gmail.com wrote: This is the biggest concern I see with promoting wheels being directly importable via zipimport (I say promoting and not specifying deliberately, but I don't want to get back into the process debate). In my view, we can have our cake and eat it. Those who don't want their wheels to be mountable can say so in their wheel metadata. I expect that distlib will honour such metadata (once the details have been worked out) and not mount wheels which aren't supposed to be mounted. However, it's perfectly easy to write code that runs from zip, as long as you know that's a deployment option you want to support. So it doesn't make sense to prevent that useful functionality - even pip is using it, I believe ;-) - just because some people think it's a bad idea, for reasons that are hardly compelling. To say that we should keep packaging separate from importing is in some sense a religious argument, unless sound technical reasons are given for it. Java proves otherwise otherwise - and for those who hate Java, that's a religious viewpoint, too ... The argument that importing wheels will cause problems is trumped by allowing the mountability to be configurable by wheel metadata: the builder of a wheel, by saying it is mountable, is agreeing to all that entails in terms of handling package resources appropriately, etc. So the support burden shifts to them, and not to wheel, distlib or any packaging tool. So much heat over process, but so little light over exactly why *appropriately designed* software deployed in wheels shouldn't be importable. No detailed analysis of any problem with wheels taking into account the differences from eggs (the things Daniel mentioned in the other thread problems with eggs?, plus the fact that there's no sys.path manipulation *magic*). Just blanket statements to the effect that eggs were importable and bad, so wheels must be too smacks of superstition, not engineering. I would like the detractors of importable wheels to put their technical reasoning on the table, not use process debates to try to revert situations they're not happy with. That technical reasoning should address wheels as they are now and avoid referring to eggs as the bogeyman. The only reason I've heard from detractors so far is that software not designed to run from zip can give unexpected results when run from zip, which could give the wheel format a bad name. Given that a wheel publisher can have a say on importability, this issue seems to me to be adequately addressed. No detractor has come up with any other solid reasoning as to why the useful feature of wheel importability is bad. I invite them to put up, or ... we'll still be in the dark the next time this debate comes around ;-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheels on sys.path clarification (reboot)
My one technical issue is with going beyond zipimport behaviour to the point of extracting DLLs to the filesystem. I remain -1 on that feature, and I believe I have explained why I think there are issues (and why I think that any solution should be part of zipimport and not added on in library or user code). But I'm happy to go through the details again, if you like - or just to accept that I don't Yes please, let's get into some details. Of course I understand that you might not want to use the feature, but I don't understand the -1 on the feature per se - whether it is in distlib or in zipimport is a secondary consideration. I agree that zipimport is the logical place for it, but ISTM the reason why it can't go in there just yet is also the reason why one might have some reservations about the feature: binary compatibility. I accept that this not yet a fully resolved issue in general (cf. the parallel discussion about numpy), but if we can isolate these issues, we can perhaps tackle them. But for me, that's the main reason why this part of the distlib API is experimental. Since zipimport makes no prescription about the contents of a zip (beyond describing how importing works), there is no agreed place to place metadata information such as about binary compatibility. Nor is binary compatibility between third-party packages of core concern to python-dev, so I'm not sure discussions there will be fruitful. However, such compatibility is a valid concern here, so I would expect useful input to come from interested parties. Also, the wheel format already caters for a limited set of binary compatibility info to be communicated, but this information is incomplete, which results in reservations about the dangers of extracting C extensions. In a sense, the contentiousness of extraction of these from a wheel is a red herring, because the exact same issues would arise if you installed from the wheel and then tried to use software which purported to be binary compatible with your system, but wasn't. That's why we don't put Linux wheels on PyPI, right? It's to avoid binary wheels compiled on e.g. CentOS ending up on an e.g. Ubuntu system which is not quite binary compatible. But that doesn't solve the problem at source, so much as act as a prophylactic. Much of the Python community works in a POSIX environment where build- from-source is the norm and binary compatibility is a non-issue. In a corporate environment with a homogeneous infrastructure, the same applies; but in a heterogeneous environment, not so much. I think some benefit would accrue to packaging as a whole if we had better definitions of binary compatibility, the ability to express it in more detail, how to test for it at installation time and so on. I think this is one of the areas where we can and should improve WHEEL metadata. Perhaps the NumPy/SciPy readers would care to comment, as we're talking beyond the realms of Python version compatibility. If you have other reasons for your -1, I'd like to hear them. need to use the feature. (Would you be willing to add some sort of never extract C extensions regardless of what the metadata might say option to wheel mount? It's not critical, as mounting random wheels without knowing how they work is clearly bad, but it does add a level of assurance that might be helpful,) Currently, extraction only happens if there is metadata in the wheel to indicate that extraction should occur, and it's up to the wheel builder to put it there. It doesn't make sense in general to prohibit just the C extensions but allow importing pure-Python modules in a wheel: the pure-Python bits mightn't work properly if the C extensions aren't available. So it seems safer in general to have all-or-nothing, or else have an additional flag in the metadata which indicates that the C extensions have to be extracted for the wheel to be useful. End user control of mounting should be at the discretion of the developer of the software which invokes the mount method. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
