Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
Err, slipped my mind that Donald is working on https://github.com/pypa/packaging which also might end up being what I was describing in my previous message On 7 December 2014 at 20:28, Matthew Iversen m...@notevencode.com wrote: Between setuptools / wheel / pip / virtualenv there is actually a good case for a shared common code project to vendor from. The easiest example of functionality contained within this would be version parsing, although there are a number of other features as well (eg installing a wheel). Currently the case is that this is half done by pip vendoring setuptools' pkg_resources.py although I think it would be of great benefit to all projects to formalise this situation into a new discrete codebase in the future. Distlib is a candidate project to fulfill this role but it would need a good bit of vetting beforehand AFAIK. Pip doesn't offer any api mainly because is too messy / large / would take away too much time for the developers to spend time creating a good api that we could stick to. Sort of We'd love to, but have you seen the issue list... :(. In that respect that is actually the use case that distlib aims to provide. Pip currently uses distlib's code to install wheels. That said I'd reiterate Donald's comment that we'd love to see a wanted-functionality list from projects like buildout for such a hypothetical official future utility project. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Are there any plans to move to pip/wheels in buildout?
Between setuptools / wheel / pip / virtualenv there is actually a good case for a shared common code project to vendor from. The easiest example of functionality contained within this would be version parsing, although there are a number of other features as well (eg installing a wheel). Currently the case is that this is half done by pip vendoring setuptools' pkg_resources.py although I think it would be of great benefit to all projects to formalise this situation into a new discrete codebase in the future. Distlib is a candidate project to fulfill this role but it would need a good bit of vetting beforehand AFAIK. Pip doesn't offer any api mainly because is too messy / large / would take away too much time for the developers to spend time creating a good api that we could stick to. Sort of We'd love to, but have you seen the issue list... :(. In that respect that is actually the use case that distlib aims to provide. Pip currently uses distlib's code to install wheels. That said I'd reiterate Donald's comment that we'd love to see a wanted-functionality list from projects like buildout for such a hypothetical official future utility project. On 4 Dec 2014 03:22, Donald Stufft don...@stufft.io wrote: On Dec 3, 2014, at 11:19 AM, Jim Fulton j...@zope.com wrote: On Wed, Dec 3, 2014 at 11:14 AM, Paul Moore p.f.mo...@gmail.com wrote: On 3 December 2014 at 16:05, Daniel Holth dho...@gmail.com wrote: Wheel also has its own installer. It hasn't received as much attention since pip gained support. That's actually a very good point - the wheel project has its own install a wheel API, not just the command line tool. If that's all buildout needs, it may be a simple way of adding wheel support. If buildout needs the other stuff that pip/setuptools provides, around discovery and downloading from PyPI, version/requirements checking, script wrappers, etc, then that's a bigger issue of course. Yup. Buildout uses setuptools to find distributions. I suppose the wheel installer library could facilitate adding wheel support to setuptools. Maybe someone on the buildout team should look at that. My guess is that adding discovery of wheels in indexes and such probably isn't that hard. Installing Wheels in general isn’t that hard either TBH. It probably wouldn’t be a ton of work to add support for that to setuptools. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pypi to enforce wheel, python3 - is it possible?
Wheels can't be required because they can be platform/python version specific, being binary distributions. We can't demand that independent developers support some combination of platforms and versions when their computer might only be running one of them, simply because their wheel might be platform dependant rather than universal. Pypi doesn't even allow Linux binary wheels at the moment because compatibility between different things and distros that all call themselves Linux running on some instruction set is a big wild problem for the moment. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing Statsmodel
On 03/06/14 08:03, Wesley Huang wrote: Hi there, I am using a Mac and trying to install the package called statsmodel. Based on available downloads, it seems i can only get it if i have a PC? Or if not true, can you link me to where I'd be able to download it? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig This is a binary package that depends on numpy, scipy, pandas and cython, at the least. These packages are also binary packages, that are best compiled for you by someone who knows how to make efficient binaries of them. The easiest way to get these packages probably begins by looking at http://www.scipy.org/install.html http://www.scipy.org/scipylib/download.html -- Matt Iversen // m...@notevencode.com PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse XMLRPC
On 06/03/14 20:10, Marius Gedminas wrote: On Wed, Mar 05, 2014 at 07:01:49AM -0500, Donald Stufft wrote: On Mar 5, 2014, at 5:48 AM, anatoly techtonik techto...@gmail.com wrote: JSON-RPC is a better choice for recommended external API, because it doesn't rely on memory hungry and potentially unsafe XML libraries. On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft don...@stufft.io wrote: Just a quick FYI that the last missing piece (search) for XMLRPC was merged in Warehouse today. If you have an application that depends on XMLRPC on PyPI and you'd like to test it, please try against the url: https://pypi-preview.a.ssl.fastly.net/pypi Let me know if you find any issues, or you can file a bug at: https://github.com/pypa/warehouse XMLRPC is used in order to maintain compatibility with what is already there. Something better will replace it eventually and XMLRPC will be deprecated. This is a bit confusing to me now. You're reacting as if anatoly suggested a new JSON interface to replace the legacy XML-RPC one. But the old PyPI codebase already had a JSON API[1]. I'm using it to keep track of Python 3 support status of about 800 packages maintained by the Zope Foundation: http://zope3.pov.lt/py3/ [1] https://wiki.python.org/moin/PyPIJSON I assume at some point Warehouse will add support for the JSON API and you'll issue a call for testing? Marius Gedminas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig A JSON-RPC interface is slightly different from a normal 'RESTful JSON' api. PyPI currently offers some normal JSON REST resources, as well as an XML-RPC interface. AFAIK Anatoly was suggesting to replace XML-RPC, with JSON-RPC (not just a JSON API, which as you mention already exists), however the former needs to continue to exist because it is an existing feature that many things use. Also AFAIK, warehouse is currently in the business of gaining feature parity with the current PyPI, after which it will be looked into if or what new APIs will be useful to add to make better clients with. Of course a prime candidate for that might be expanding the current RESTful JSON api. -- Matt Iversen PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Wheel with zipimport
On the one hand, it's easy to see the allure of a zipimport-able format. Grab a file, import it, use functionality straight away. It has an attraction of self-containedness. On the other hand, python's dynamic nature means that things are not as simple as Java jar's as Donald points out. I agree that wheels shouldn't officially support this feature wholesale. However, perhaps they could do so conditionally? E.g. designate some mechanism by which it is possible to inspect a wheel and determine that it intends to explicitly be zipimport compatible. A prime candidate would be that the wheel is of a purelib in nature, another would be to add a top-level file or metadata indicating such (e.g. '.zipimportable'). But other wise I agree that we should explicitly seek to avoid any language that could lead users to expect this functionality of wheels wholesale in any way, even though they happen to be regular zip files. On 29 Jan 2014 17:09, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 11:14 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format. When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified. Last time this topic came up Jim Fulton did point things out. 1. That unpacking into the home directory is problematic because users that run services often don't have home directories. This you waved away by saying that the home directory isn't a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and generally cause pain. Which I wholly agree with, intact Mitre has assigned CVE's to applications that did switch their Egg Cache to directories other than HOME. 2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a deployment format (which I don't believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him. 3. Zipped imports (In Jim's experience) are slower than unpacked imports, which you replied (quite literally) Caveat emptor. Jim's response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words: It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony. He closes up with, Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs. So you have pip that doesn't use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it). Why do we insist upon repeating the mistakes of the past? Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a mount method to add a wheel to sys.path. This uses the tags to check for compatibility - there's no figuring out that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it. Zipped Eggs have a long long history of causing very weird failures conditions, you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn't using __file__
Re: [Distutils] Using Wheel with zipimport
On 30/01/14 00:59, Nick Coghlan wrote: On 29 January 2014 23:48, Donald Stufft don...@stufft.io wrote: So what did you mean when you said We discussed it extensively before PEP 427 was approved if you're now saying that it wasn't discussed. Explicitly would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification). Cheers, Nick. In My Humble Opinion, that just isn't good enough. I am sure that everyone on this list has a passionate interest in making sure that the decisions for python packaging are as right as possible, and we know there are processes to follow that help us achieve that. One of those is that one should always be explicit about what features and functionality anything supports, which means that all features and functionality can be discussed, agreed upon, improved, etc, before it becomes something that everyone wilfully agrees to support. Wheels are either *documented* to be a binary format that *officially supports* zipimport-ability (fully, conditionally, or whatever), or the default answer is, like any other functionality not explicitly mentioned, no they are not. That's how you get a standard where issues can be discussed without tension. Two people agreeing to themselves that this is an uncontroversial implicit feature is not part of the documentation process, and whether or not they are the creators and acceptors of such standards, it leaves others out of the process until this late stage of the game. The unfortunate disagreements are exactly what can happen when things aren't made explicit from the start and are what are preventable when they are. Docs or it doesn't exist is popular developer mantra for a reason. Even as mostly an kibitzer on this group I hope it can learn from and avoid this sort of discussion into the future. -- Matt Iversen PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip on windows experience
Might I suggest you could upload some wheels (both windows and linux) to testpypi, which afaik is pretty much made for this purpose? https://wiki.python.org/moin/TestPyPI People can easily install then with e.g. `pip install --index-url https://testpypi.python.org/pypi numpy`, and see what tends to break or what doesn't. On 24 January 2014 05:52, Ralf Gommers ralf.gomm...@gmail.com wrote: On Thu, Jan 23, 2014 at 3:42 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On Thu, Jan 23, 2014 at 12:16:02PM +, Paul Moore wrote: The official numpy installer uses some complex magic to select the right binaries based on your CPU, and this means that the official numpy superpack wininst files don't convert (at least I don't think they do, it's a while since I tried). It's probably worth noting that numpy are toying around with wheels and have uploaded a number of them to PyPI for testing: http://sourceforge.net/projects/numpy/files/wheels_to_test/ Currently there are only OSX wheels there (excluding the puer Python ones) and they're not available on PyPI. I assume that they're waiting for a solution for the Windows installer (a post-install script for wheels). That would give a lot more impetus to put wheels up on PyPI. Indeed. We discussed just picking the SSE2 or SSE3 build and putting that up as a wheel, but that was deemed a not so great idea: http://article.gmane.org/gmane.comp.python.numeric.general/56072 The Sourceforge OSX wheels are presumably not getting that much use right now. The OSX-specific numpy wheel has been downloaded 4 times in the last week: twice on Windows and twice on Linux! Some feedback from the people who did try those wheels would help. I asked for that on the numpy list after creating them, but didn't get much. So I haven't been in a hurry to move them over to PyPi. Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Please do not remove dependency_links
On 17/01/14 13:52, Hannes Schmidt wrote: I read through the Removing dependency_links thread [1] and I beg you not follow through with the deprecation and removal of dependency_links and to rethink your approach. The mentioned thread indicates that research was done to gauge the popularity of the dependency_links in publicly hosted Python projects. That approach is fundamentally flawed: Publicly hosted projects are much more likely to also be available on PyPI than private, closed-source projects. Consequently, their dependencies are also more likely to be hosted on PyPI as well. Because of that, they are much less likely to rely on the dependency_links feature. Another misconception seem to be that dependency_links is predominantly used for installing patched or customized versions of dependencies hosted on PyPI. I'm pretty sure the predominant use case for dependency_links is with projects that are hosted privately, e.g. for an organization's internal use. I represent such an organization and removing dependency_links would impact us negatively. We host a set of internal projects and their dependencies on Bitbucket and we rely on dependency_links to install them directly from there. I understand the motivation for this change -- security -- but there must be smarter way to handle it. Could we fallback to dependency_links if a PyPI lookup isn't successful? Could we restrict dependency_links to links that share a prefix with the link from which the package is currently being installed? A combination of the two? [1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) han...@ucsc.edu mailto:han...@ucsc.edu ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Are you aware that you can also provide a solid infrastructure for a proprietary environment of packages by hosting your own private pypi mirror? There are quite a few projects that enable this, such as https://pypi.python.org/pypi/pypiserver https://pypi.python.org/pypi/mypypi https://github.com/steiza/simplepypi http://djangopypi2.readthedocs.org/en/latest/ You can use the -i flag, or a .pypirc config file to tell pip to look at your own private index instead of the official one; or direct it to use the official pypi only after looking at your private pypi. You can also host a network-available folder of wheels for pip to find, or simply a http accessible folder of packages as Donald suggested. -- Matt Iversen PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973 signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
