Re: [Distutils] Request to add a trove classifier for pelican plugins
Sorry Alexis! I'm a little confused as to how Pelican deserves to be considered a framework. Generally before we add a Framework classifier we need to see a number of packages in PyPI which would have that classifier applied to them. Can you point to such packages? Richard On 1 August 2013 01:04, Alexis Métaireau wrote: > On 15/07/2013 18:27, Alexis Métaireau wrote: >> Hi, >> >> I hope this is the right place to ask for this. >> >> I would like to have a trove classifier for pelican [0] plugins. We >> plan to release them on PyPI and having a classifier to distinguish >> them from all the other packages sounds the way to go. >> >> We're not really a framework, but following the already established >> pattern, I guess "Framework :: Pelican" makes sense. >> >> Thanks! >> Alexis >> >> [0] http://getpelican.com > > Hi, Just a quick follow-up on this request since I didn't got any > answer. Don't hesitate to point me to the right person if you know more > than I do. > > Cheers, > Alexis > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse Migration Plan
On Jul 31, 2013, at 8:11 AM, holger krekel wrote: > On Wed, Jul 31, 2013 at 00:15 -0400, Donald Stufft wrote: >> So, in the spirit of not treating distutils-sig like an adversary, here's >> the main thing I've been working on lately with regards to PyPI. None >> of this is set in stone but this is the general gist of the plan for moving >> things to be developed in a modern framework, as well as cleaning >> up the code and getting repeatable deployments. > > Is warehouse a re-implementation or did it start from the current code base? Reimplementation. > >> Warehouse Migration Plan >> >> >> Warehouse is currently primarily besides modeling for user accounts. It >> will be deployed alongside pypi-legacy at next.pypi.python.org in the near >> future. Initially it will have zero user facing value. Just an update here, the name was changed to preview.pypi.python.org to follow suite with preview.python.org and to prevent any sort of confusion with last.pypi.python.org from the mirroring protocol. >> >> As time goes on the database tables will be migrated from being "owned" >> by pypi-legacy to being "owned" by Warehouse. This primarily means that >> the schema definition and migration of those tables will be controlled by >> Warehouse. As tables are migrated to Warehouse ownership the PyPI code >> will be updated to reflect any changes in schema (without modification to >> what end users see). > > Am a bit skeptical if sharing databases is a good approach. > Certainly has potential for disrupting pypi.python.org and making > it harder for next. Sharing the database is has two major purposes, first it allows Warehouse to be phased in having people test it with real live data while PyPI itself is largely unaffected. Second it prevents the need to have a big major switch point which will largely be a one way switch with no easy way to revert it. Essentially it's designed to strangle pypi-legacy slowly over time in smaller bite sized chunks instead of needing to do everything at once. > >> Once all tables that are going to be kept have been migrated, we will have >> a shared database that is accessible from both pypi-legacy and Warehouse. >> From this point Warehouse will begin evolving "legacy" views such as the >> simple and other pieces of API. The UX itself will continue to be ignored and >> focus will be on getting feature parity for automated tooling. >> >> Changes in behavior on these legacy views should be minimal and >> discussed on distutils-sig. > > Having a doc/spec of those interactions would indeed help and contribute > to "not defined by implementation" as you state below. > >> Once the legacy views are finished people will be encouraged to test their >> real world workloads against those reimplemented legacy APIs. As changes >> in behaviors, missing functionality, or bugs are found they will be >> rectified or >> declared unsupported. >> >> During this time work on the UI of Warehouse will begin focusing on maintaing >> feature parity but with no promises of no changes to the url structure or UX. >> >> Once Warehouse gains feature parity with PyPI and has gotten enough testing >> against it's APIs then pypi-legacy will be retired and Warehouse will move >> from >> next.pypi.python.org to pypi.python.org. From there it will evolve on it's >> own without >> needing to keep pypi-legacy in sync. >> >> Specification & Acceptance Testing >> >> >> I do not want a packaging index server to be specified by implementation, so >> as >> the legacy API is ported over to Warehouse a specification will be drafted. >> This >> spec will represent the "promise" that PyPI makes about the API it presents >> to be >> consumed by machines. During the migration any behavior not codified inside >> of >> the spec is considered implementation defined behavior and backwards >> compatibility >> for that behavior will not be promised. >> >> In addition to a defined specification A repository of acceptance tests will >> be developed. >> These tests will be part of the test framework for any future changes to >> Warehouse >> but will be maintained separately alongside the specification. They will >> also allow >> any alternative implementation (such as DevPI) to test themselves against >> the spec. > > I'd be happy to discuss if we can collaborate or even merge some of our > efforts. Sure. > > cheers, > holger > >> >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > > > >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail _
Re: [Distutils] Request to add a trove classifier for pelican plugins
On 15/07/2013 18:27, Alexis Métaireau wrote: > Hi, > > I hope this is the right place to ask for this. > > I would like to have a trove classifier for pelican [0] plugins. We > plan to release them on PyPI and having a classifier to distinguish > them from all the other packages sounds the way to go. > > We're not really a framework, but following the already established > pattern, I guess "Framework :: Pelican" makes sense. > > Thanks! > Alexis > > [0] http://getpelican.com Hi, Just a quick follow-up on this request since I didn't got any answer. Don't hesitate to point me to the right person if you know more than I do. Cheers, Alexis ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Status report on PyPI+pip+TUF
Hello Holger, On 07/31/2013 08:13 AM, holger krekel wrote: thanks for the high level overview. Do you have a current web page with more detailed technical info with respect to PyPI/TUF? Good question! I think it is a good idea to put up a "PyPI+pip+TUF current status" page on our web site, but in the meantime, here are a few links which should point you in the right direction: 1. pip+TUF: we use the interposition technique [https://github.com/theupdateframework/tuf/tree/master/tuf/interposition] to minimally modify pip [https://github.com/theupdateframework/pip/compare/tuf] to talk to a TUF-secured PyPI mirror. 2. PyPI+TUF: we use automation to build a testbed for investigating different key management and metadata schemes to secure PyPI [https://github.com/theupdateframework/pypi.updateframework.com]. (Note: at the time of writing, the automation is slightly out-of-date with our work-in-progress.) 3. These two links should give you a good picture, but they will not give you a complete one. We will formally write about what we mean with our upcoming key management as well as metadata generation and download scheme. Let me start a document and get back to you on that. Thanks, Trishank ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse Migration Plan
On Wed, Jul 31, 2013 at 00:15 -0400, Donald Stufft wrote: > So, in the spirit of not treating distutils-sig like an adversary, here's > the main thing I've been working on lately with regards to PyPI. None > of this is set in stone but this is the general gist of the plan for moving > things to be developed in a modern framework, as well as cleaning > up the code and getting repeatable deployments. Is warehouse a re-implementation or did it start from the current code base? > Warehouse Migration Plan > > > Warehouse is currently primarily besides modeling for user accounts. It > will be deployed alongside pypi-legacy at next.pypi.python.org in the near > future. Initially it will have zero user facing value. > > As time goes on the database tables will be migrated from being "owned" > by pypi-legacy to being "owned" by Warehouse. This primarily means that > the schema definition and migration of those tables will be controlled by > Warehouse. As tables are migrated to Warehouse ownership the PyPI code > will be updated to reflect any changes in schema (without modification to > what end users see). Am a bit skeptical if sharing databases is a good approach. Certainly has potential for disrupting pypi.python.org and making it harder for next. > Once all tables that are going to be kept have been migrated, we will have > a shared database that is accessible from both pypi-legacy and Warehouse. > From this point Warehouse will begin evolving "legacy" views such as the > simple and other pieces of API. The UX itself will continue to be ignored and > focus will be on getting feature parity for automated tooling. > > Changes in behavior on these legacy views should be minimal and > discussed on distutils-sig. Having a doc/spec of those interactions would indeed help and contribute to "not defined by implementation" as you state below. > Once the legacy views are finished people will be encouraged to test their > real world workloads against those reimplemented legacy APIs. As changes > in behaviors, missing functionality, or bugs are found they will be rectified > or > declared unsupported. > > During this time work on the UI of Warehouse will begin focusing on maintaing > feature parity but with no promises of no changes to the url structure or UX. > > Once Warehouse gains feature parity with PyPI and has gotten enough testing > against it's APIs then pypi-legacy will be retired and Warehouse will move > from > next.pypi.python.org to pypi.python.org. From there it will evolve on it's > own without > needing to keep pypi-legacy in sync. > > Specification & Acceptance Testing > > > I do not want a packaging index server to be specified by implementation, so > as > the legacy API is ported over to Warehouse a specification will be drafted. > This > spec will represent the "promise" that PyPI makes about the API it presents > to be > consumed by machines. During the migration any behavior not codified inside of > the spec is considered implementation defined behavior and backwards > compatibility > for that behavior will not be promised. > > In addition to a defined specification A repository of acceptance tests will > be developed. > These tests will be part of the test framework for any future changes to > Warehouse > but will be maintained separately alongside the specification. They will also > allow > any alternative implementation (such as DevPI) to test themselves against the > spec. I'd be happy to discuss if we can collaborate or even merge some of our efforts. cheers, holger > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Status report on PyPI+pip+TUF
Hi Trishank, thanks for the high level overview. Do you have a current web page with more detailed technical info with respect to PyPI/TUF? best, holger On Wed, Jul 31, 2013 at 07:27 -0400, Trishank Karthik Kuppusamy wrote: > Hello Nick and the PyPI community, > > This is a brief status report on the integration of PyPI and pip with TUF. > > (A quick reminder: TUF is a general "plug-n-play" update framework > designed to introduce usable security to community software > repositories such as PyPI. If you think of PyPI as HTTP, then TUF is > like adding SSL, and more, to HTTP. More information may be found at > [https://www.updateframework.com/].) > > Firstly, thanks to the generous funding of the National Science > Foundation, we are pleased to introduce the addition of a full-time > developer, Vladimir Diaz, to our team. Vladimir has been > instrumental to the development of TUF, and we are excited to have > him join us full-time. (Now we do not just have one PhD student who > works on TUF when he is not busy working on other projects!) We are > also happy to have a few interns --- Zane Fisher, Tian Tian, John > Ward, and Yuyu Zheng --- on board for the summer. > > Since the security attacks on the Python wiki infrastructure earlier > this year, we have been closely following Distutils-SIG to see what > we could do to help secure PyPI. We use Python heavily in all of our > projects, and would love to help in any way we can. > > Here is what we have done: > == > > 1. At PyCon 2013, we showed that pip needs very little modification > to work with a TUF-enabled PyPI mirror. > > 2. Soon after (during the spring break), we wrote automation to > build a TUF-secured PyPI mirror (which is indistinguishable from any > other PyPI mirror except that it has signed metadata about all of > the files on PyPI). > > 3. At the same time, thanks to efforts of Konstantin Andrianov, we > also wrote a lot of unit and integration tests to show the attacks > that are possible without TUF and impossible with TUF. > > 4. After that, we started investigating the most efficient way to > build TUF metadata for PyPI. We found that requiring a separate key > for every package on PyPI may sound like a good idea, but besides > generating too much metadata, this scheme also makes key management > difficult. > > Here is what we are doing now: > == > > We are designing a usable key management scheme, coupled with > efficient generation and download of metadata, which we think should > make for a smooth integration of PyPI with TUF. We are actively > working on this and think that we are almost there. As a > conservative estimate, we do not believe that this should take > longer than two weeks. > > Here is what we are going to do next: > = > > In about a month, we will present to you a demonstration of a PyPI > mirror and a pip client which are robust against entire classes of > security attacks. We welcome you then to try our demo, be really > critical of it and tell us what you think about what we could do > better. Our goal with TUF is to provide a framework that works with > as many software community repositories as possible and that secures > as many users as possible. > > More details on our development are available at our mailing list: > https://groups.google.com/forum/#!forum/theupdateframework > > We hope this gives you a good idea of the current status of > integrating TUF with PyPI and pip. Let us know if you have > questions. > > Thanks, > The TUF team > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Status report on PyPI+pip+TUF
Hello Nick and the PyPI community, This is a brief status report on the integration of PyPI and pip with TUF. (A quick reminder: TUF is a general "plug-n-play" update framework designed to introduce usable security to community software repositories such as PyPI. If you think of PyPI as HTTP, then TUF is like adding SSL, and more, to HTTP. More information may be found at [https://www.updateframework.com/].) Firstly, thanks to the generous funding of the National Science Foundation, we are pleased to introduce the addition of a full-time developer, Vladimir Diaz, to our team. Vladimir has been instrumental to the development of TUF, and we are excited to have him join us full-time. (Now we do not just have one PhD student who works on TUF when he is not busy working on other projects!) We are also happy to have a few interns --- Zane Fisher, Tian Tian, John Ward, and Yuyu Zheng --- on board for the summer. Since the security attacks on the Python wiki infrastructure earlier this year, we have been closely following Distutils-SIG to see what we could do to help secure PyPI. We use Python heavily in all of our projects, and would love to help in any way we can. Here is what we have done: == 1. At PyCon 2013, we showed that pip needs very little modification to work with a TUF-enabled PyPI mirror. 2. Soon after (during the spring break), we wrote automation to build a TUF-secured PyPI mirror (which is indistinguishable from any other PyPI mirror except that it has signed metadata about all of the files on PyPI). 3. At the same time, thanks to efforts of Konstantin Andrianov, we also wrote a lot of unit and integration tests to show the attacks that are possible without TUF and impossible with TUF. 4. After that, we started investigating the most efficient way to build TUF metadata for PyPI. We found that requiring a separate key for every package on PyPI may sound like a good idea, but besides generating too much metadata, this scheme also makes key management difficult. Here is what we are doing now: == We are designing a usable key management scheme, coupled with efficient generation and download of metadata, which we think should make for a smooth integration of PyPI with TUF. We are actively working on this and think that we are almost there. As a conservative estimate, we do not believe that this should take longer than two weeks. Here is what we are going to do next: = In about a month, we will present to you a demonstration of a PyPI mirror and a pip client which are robust against entire classes of security attacks. We welcome you then to try our demo, be really critical of it and tell us what you think about what we could do better. Our goal with TUF is to provide a framework that works with as many software community repositories as possible and that secures as many users as possible. More details on our development are available at our mailing list: https://groups.google.com/forum/#!forum/theupdateframework We hope this gives you a good idea of the current status of integrating TUF with PyPI and pip. Let us know if you have questions. Thanks, The TUF team ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SSL problem with PyPI?
On Jul 31, 2013, at 5:34 AM, Vinay Sajip wrote: > I've just started getting PyPI failures because the SSL certificate being > presented by pypi.python.org is only valid for addvocate.com. Can anyone shed > some light on this? > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Should be resolved now. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SSL problem with PyPI?
On Jul 31, 2013, at 5:34 AM, Vinay Sajip wrote: > I've just started getting PyPI failures because the SSL certificate being > presented by pypi.python.org is only valid for addvocate.com. Can anyone shed > some light on this? > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig http://status.python.org/incidents/jj8d7xn41hr5 - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] SSL problem with PyPI?
I've just started getting PyPI failures because the SSL certificate being presented by pypi.python.org is only valid for addvocate.com. Can anyone shed some light on this? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse Migration Plan
On 31 July 2013 05:15, Donald Stufft wrote: [...] +1. This sounds like a good plan I do not want a packaging index server to be specified by implementation, > so as > the legacy API is ported over to Warehouse a specification will be > drafted. This > spec will represent the "promise" that PyPI makes about the API it > presents to be > consumed by machines. During the migration any behavior not codified > inside of > the spec is considered implementation defined behavior and backwards > compatibility > for that behavior will not be promised. > And this is something I wholeheartedly support - having a properly specified PyPI API will not only help protect against future changes, it will also ensure that people writing their own index servers have a well-defined spec to work to. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
