Re: [Distutils] Role of setuptools and eggs in modern distributing...
On Tue, 23 Dec 2014 09:36:36 -0800 Chris Barker chris.bar...@noaa.gov wrote: So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools. I figure we we'll do one (or both) of: - providing a custom wheel house with our packages and the dependencies that are hard to come by - provide a binstar channel with conda packages of all the same stuff but a totally different set of other packages. At the moment, I'm working on making conda packages, which brings me to my questions. Note you can use pip inside conda environments, which works quite well at least for pure Python packages (start with conda install pip). Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Role of setuptools and eggs in modern distributing...
On Tue, Dec 30, 2014 at 2:21 PM, Reinout van Rees rein...@vanrees.org wrote: Well, we're in a bit of the same boat here. We make django websites, which means pretty much well-behaved setup.py-using pure python stuff. The websites are heavy users of numpy/scipy/pandas/matplotlib and of the geo packages like gdal/mapnik/pyproj. Ouch. yup -- this is pretty much out stack (though pyramid in our case...) Funny, though, as coming from my background, I see it as a scipy stack app with a little web stuff, rather than a web app that needs some scipy stuff ;-) But the core problem here is that the scipy folks have been going to conda and enthought to solve their pacakgeing problems, and the web folks have been doing pip, and maybe buildout -- so you get a bit of mess when you mix them. The combination we use now is to use buildout (instead of pip) in combination with the syseggrecipe (https://pypi.python.org/pypi/ syseggrecipe) buildout add-on. Syseggrecipe allows us to depend explicitly on **system** packages for gdal/matplotlib/mapnik/scipy/numpy. Which takes care of most of the compilation headaches. well, it sounds like you are simple punting -- passing off all the complex stuff to the system, which may work well if the system is up to date linux with the packages you need available, but pretty worthless on a Mac or Windows box. The scipy folks have been doing a pretty good job lately keeping up with wheels, but there's still a big hole there for the geo stuff.(GDAL, Shapely, Fiona) So I've been looking at going the Anaconda route -- it provides the hard stuff, though it turns out it's a bit ugly when using it as a development environment for extensions liked against libs that are both in the system and Anaconda provided. Antoine Pitrou wrote: Note you can use pip inside conda environments, which works quite well at least for pure Python packages (start with conda install pip). True -- though it gets a bit ugly, as then conda doesn't know about the packages, so switching environments is a mess, and conda can't manage the dependencies. So not ideal. I've actually spend the last two days writing a script that auto-grabs packages from PyPI, builds conda packages out of them, and then uploads them to binstar -- so we can have all our dependencies supported by conda. I'd love it if Continuum would build a pip bridge on binstar that would do all that automagically if you request a pip-installable package from binstar. Just throwing this into the mix as a potential solution. Note that you'll get to explain buildout to your users, but... :-) yup -- not sure I want to go there yet, either... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
On 23 December 2014 at 04:15, Vladimir Diaz vladimir.v.d...@gmail.com wrote: On Mon, Dec 22, 2014 at 11:30 AM, Nick Coghlan ncogh...@gmail.com wrote: From my perspective, the split into two PEPs meant most of the areas I have doubts about have been moved to the end-to-end security model in PEP 480, leaving PEP 458 to cover the simpler task of securing the link from PyPI to the end user in such a way that public mirrors of packages can be trusted to accurately reflect the content published by PyPI. I think splitting the proposal into two PEPs was the right decision. We hope working with Donald on the end-to-end security model (PEP 480), and feedback from the community will help to address any remaining questions. Excluding the end-to-end option from the revised version of PEP 458 also made room for an overview of the metadata and framework, which was requested by multiple members of the community. An off-list question from Richard made me realise we should likely retitle the two PEPs slightly. I'd suggest the following names: PEP 458: Surviving a compromise of the PyPI CDN PEP 480: Surviving a compromise of PyPI That encapsulates the difference between the threat model of the two PEPs in a way that the current titles don't quite convey (the reduced scope of PEP 458 in particular means that the current title is actually outright wrong - protecting against a compromise of PyPI itself is the scope that was moved to PEP 480). The reduced scope of PEP 458 also still protects against the compromise of read-only mirrors, but I don't think we need to try to capture that directly in the title. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] New User: depends not appearing in control file
On 23 December 2014 at 02:26, Geoff Clements ge...@electron.me.uk wrote: Hi! I just started using stdeb but have run into a problem which I can't solve. Hi Geoff, I just noticed you didn't get a reply to this or your other question. For distro specific tools like stdeb, you may need to look for assistance through Debian and/or Ubuntu specific channels - while we do see folks like Matthias Klose and Barry Warsaw here on distutils-sig occasionally, I don't believe there's anyone specifically monitoring this list for questions about the distro level integration utilities. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
On Dec 30, 2014, at 9:29 PM, Richard Jones rich...@python.org wrote: Thanks for the clarification, guys. Donald, I'm not sure what you mean by a compromise of the CDN for *uploading*”. PyPI trusts the CDN to give it the correct bits, without a signature from the author that is being verified uploading just relies on TLS again. The other PEP should close that gap though I believe. Note: I have yet to read these PEPs so I’m just going by a casual glance of them. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
Thanks for the clarification, guys. Donald, I'm not sure what you mean by a compromise of the CDN for *uploading*. On Wed Dec 31 2014 at 1:21:18 PM Donald Stufft don...@stufft.io wrote: On Dec 30, 2014, at 8:24 PM, Nick Coghlan ncogh...@gmail.com wrote: On 23 December 2014 at 04:15, Vladimir Diaz vladimir.v.d...@gmail.com wrote: On Mon, Dec 22, 2014 at 11:30 AM, Nick Coghlan ncogh...@gmail.com wrote: From my perspective, the split into two PEPs meant most of the areas I have doubts about have been moved to the end-to-end security model in PEP 480, leaving PEP 458 to cover the simpler task of securing the link from PyPI to the end user in such a way that public mirrors of packages can be trusted to accurately reflect the content published by PyPI. I think splitting the proposal into two PEPs was the right decision. We hope working with Donald on the end-to-end security model (PEP 480), and feedback from the community will help to address any remaining questions. Excluding the end-to-end option from the revised version of PEP 458 also made room for an overview of the metadata and framework, which was requested by multiple members of the community. An off-list question from Richard made me realise we should likely retitle the two PEPs slightly. I'd suggest the following names: PEP 458: Surviving a compromise of the PyPI CDN This isn’t exactly right either, because it won’t survive a compromise of the CDN for *uploading*, but it might be close enough not to matter. Perhaps better would be something about not relying on TLS or something. PEP 480: Surviving a compromise of PyPI That encapsulates the difference between the threat model of the two PEPs in a way that the current titles don't quite convey (the reduced scope of PEP 458 in particular means that the current title is actually outright wrong - protecting against a compromise of PyPI itself is the scope that was moved to PEP 480). The reduced scope of PEP 458 also still protects against the compromise of read-only mirrors, but I don't think we need to try to capture that directly in the title. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Last call for feedback: PEP 440 exclusive ordered comparison fix
Donald is keen to get the updated versions of packaging/pip/setuptools out that fix the regression in handling exclusive ordered comparison, so this is a last call for feedback on those changes before we publish the updated version of the PEP to python.org (and then update the tools accordingly). Given the proximity to New Year's celebrations, I currently plan to publish the updated version some time on Saturday 3rd January (UTC+10). The actual definition of the change is in https://github.com/pypa/interoperability-peps/commit/225de0a41270189dbfd4a038e8c6d56129580f8b The key fix is making X.Y accept X.Y.Z maintenance releases again, which brings PEP 440 into line with the behaviour in pkg_resources prior to the release of setuptools 8.0. The change log also notes that this update includes the revision to the normalised form of release candidates: https://github.com/pypa/interoperability-peps/commit/68ce85faaca1ee715e2a9f46cdd14ed01a2b1d73 There's at least one more clarification to the PEP text still to be made based on community feedback ( https://github.com/pypa/interoperability-peps/issues/6), but that doesn't require any changes to the tools, so I don't mind if it doesn't make it into this initial update. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
On 31 December 2014 at 12:32, Donald Stufft don...@stufft.io wrote: PyPI trusts the CDN to give it the correct bits, without a signature from the author that is being verified uploading just relies on TLS again. The other PEP should close that gap though I believe. I'm actually not sure what going through the CDN is buying us on the upload side of things in the first place, given the main pay-off provided by a CDN is geographically distributed caching of unchanging data for faster downloads. So it seems to me that that particular vulnerability could potentially be fixed more simply by bypassing the CDN entirely for the upload case. That's simplicity in the *conceptual* sense, though - there may be architectural issues in the current implementation of PyPI and the related tools that make it harder in practice than it would be in theory. Either way, I agree that any kind of upload compromise based attack is also out of scope for PEP 458 - that's now entirely about ensuring that the bits delivered to end users are the same bits PyPI published. Making sure that the bits *PyPI* publishes are the same ones that the *developer* published is the domain of PEP 480. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
Now that I think about it, I'm almost certain that Donald and I have had the hey, what about an upload.pypi.python.org conversation in the past, as a way around issues involving the CDN :) Still a good idea, in my opinion. Richard On Wed Dec 31 2014 at 3:01:53 PM Nick Coghlan ncogh...@gmail.com wrote: On 31 December 2014 at 12:32, Donald Stufft don...@stufft.io wrote: PyPI trusts the CDN to give it the correct bits, without a signature from the author that is being verified uploading just relies on TLS again. The other PEP should close that gap though I believe. I'm actually not sure what going through the CDN is buying us on the upload side of things in the first place, given the main pay-off provided by a CDN is geographically distributed caching of unchanging data for faster downloads. So it seems to me that that particular vulnerability could potentially be fixed more simply by bypassing the CDN entirely for the upload case. That's simplicity in the *conceptual* sense, though - there may be architectural issues in the current implementation of PyPI and the related tools that make it harder in practice than it would be in theory. Either way, I agree that any kind of upload compromise based attack is also out of scope for PEP 458 - that's now entirely about ensuring that the bits delivered to end users are the same bits PyPI published. Making sure that the bits *PyPI* publishes are the same ones that the *developer* published is the domain of PEP 480. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Surviving a Compromise of PyPI - PEP 458 and 480
On Dec 10, 2014, at 10:16 PM, Vladimir Diaz vladimir.v.d...@gmail.com wrote: Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, Surviving a Compromise of PyPI. The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: Surviving a Compromise of PyPI proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework https://github.com/theupdateframework/tuf, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 http://legacy.python.org/dev/peps/pep-0458/) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 http://legacy.python.org/dev/peps/pep-0480/) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. I’ve just finished (re)reading the white paper, PEP 450, PEP 480, and some of the supporting documentation on the TUF website. I’m confused about what exactly is contained within the TUF metadata and who signs what in a PEP 480 world. Currently when you do something like ``pip install FooBar``, pip fetches /simple/FooBar/ to look for potential installation candidates, and when it finds one it downloads it and installs it. This all all “signed” by online keys via TLS. 1. In a TUF world, would pip still fetch /simple/FooBar/ to discover things to install or would it fetch some TUF metadata to find things to install? 2. If it’s fetching /simple/FooBar/ is that secured by TUF? 3. If it’s secured by TUF who signs the TUF metadata that talks about /simple/FooBar/ in PEP 480 the author or PyPI? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
