Re: the new PyPI, coming next month

[Sumana Harihareswara]

Reminder that the next livechat is in a little under 2 hours, 1500 UTC.

On 04/01/2018 06:54 AM, Sumana Harihareswara wrote:
> On 03/31/2018 10:15 PM, Sumana Harihareswara wrote:
>> Debian-Python experts,
>>
>> I'm writing to you in hopes you will forward this to the right places,
>> and file relevant bugs against uscan/watch, which I don't quite
>> understand enough to do myself. And if you want to follow up on
>> https://github.com/pypa/warehouse/issues/358#issuecomment-337233792 and
>> file a new issue asking for us to support your redirector more cleanly,
>> I'd welcome that.
>>
>> I'm the project manager for the new Python Package Index (Warehouse),
>> which is currently in beta at http://pypi.org/ 
> 
> [snip]
> 
> Because the above was basically a copy of a mail I attempted to have
> posted to this list a few weeks ago, I neglected to add a link to our
> beta announcement
> 
> https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html
> 
> which has an updated list of migration steps, and our IRC/Twitter
> livechat hours. Apologies. The upcoming livechats:
> 
> * Tuesday, April 3rd, 8am-9am PDT, 11am-noon EDT, 17:00-18:00 CEST,
> 8:30pm-9:30pm India, 15:00-16:00 UTC
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=Warehouse/PyPI+beta+livechat=20180403T10=24=1
> 
> * Thursday, April 5th, 5pm-6pm PDT, 8pm-9pm EDT, (April 5th) 8am-9am
> Manila, (April 5th) 10am-11am Melbourne, (April 5th) 0:00-1:00 UTC
> https://www.timeanddate.com/worldclock/fixedtime.html?p1=24=20180405T19=Warehouse/PyPI%20beta%20livechat=1=4
> 
> And please forward
> https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html
> widely; we'd like to get PyPI users to test Warehouse as much as
> possible during the next couple weeks.
> 

Re: the new PyPI, coming next month

[Donald Stufft]

> On Apr 1, 2018, at 2:27 AM, Dominik George  wrote:
> 
> Hi,
> 
>> To be clear, PGP signatures can still be uploaded and they are still
>> available for download, they just don’t appear in the UI anymore.
> 
> So, what does the pypi.debian.net redirector use for uscan?  I imagine it
> used to scrape the website.  Can it be changed to use the JSON API?

The original PoC I wrote used the JSON API, but I don’t think what’s being 
deployed is descendant from my PoC so I have no idea what it uses, but if it’s 
not using the JSON API then yes it can be.

> 
>> Longer term I’d *like* to get rid of PGP signatures, because I think
>> their value here is actually pretty low.
> 
> I partially share this opinion, but that's a question to be discusses with
> the Debian policy people in general.  While checking a GPG signature on the
> source tarball in general is a good idea, I am afraid some developers just
> drop any key they find on first glance into the package and are done with
> it, which actually provides nothing but a false sense of safety.
> 
>> In that case they’d be replaced with TUF, but that’s a longer term
>> project.
> 
> That one?: https://github.com/theupdateframework/tuf 
> 


Yes.


> 
> Well, I can only say *please* do not remove the possibility to upload signed
> source tarballs, but leave that to the developers!
> 
> -nik
> 
> --
> PGP-Fingerprint: 3C9D 54A4 7575 C026 FB17  FD26 B79A 3C16 A0C4 F296
> 
> Dominik George · Hundeshagenstr. 26 · 53225 Bonn
> Phone: +49 228 92934581 · https://www.dominik-george.de/
> 
> Teckids e.V. · FrOSCon e.V. · Debian Developer
> 
> LPIC-3 Linux Enterprise Professional (Security)



signature.asc
Description: Message signed with OpenPGP

Re: the new PyPI, coming next month

[Sumana Harihareswara]

On 03/31/2018 10:15 PM, Sumana Harihareswara wrote:
> Debian-Python experts,
> 
> I'm writing to you in hopes you will forward this to the right places,
> and file relevant bugs against uscan/watch, which I don't quite
> understand enough to do myself. And if you want to follow up on
> https://github.com/pypa/warehouse/issues/358#issuecomment-337233792 and
> file a new issue asking for us to support your redirector more cleanly,
> I'd welcome that.
> 
> I'm the project manager for the new Python Package Index (Warehouse),
> which is currently in beta at http://pypi.org/ 

[snip]

Because the above was basically a copy of a mail I attempted to have
posted to this list a few weeks ago, I neglected to add a link to our
beta announcement

https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html

which has an updated list of migration steps, and our IRC/Twitter
livechat hours. Apologies. The upcoming livechats:

* Tuesday, April 3rd, 8am-9am PDT, 11am-noon EDT, 17:00-18:00 CEST,
8:30pm-9:30pm India, 15:00-16:00 UTC
https://www.timeanddate.com/worldclock/fixedtime.html?msg=Warehouse/PyPI+beta+livechat=20180403T10=24=1

* Thursday, April 5th, 5pm-6pm PDT, 8pm-9pm EDT, (April 5th) 8am-9am
Manila, (April 5th) 10am-11am Melbourne, (April 5th) 0:00-1:00 UTC
https://www.timeanddate.com/worldclock/fixedtime.html?p1=24=20180405T19=Warehouse/PyPI%20beta%20livechat=1=4

And please forward
https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html
widely; we'd like to get PyPI users to test Warehouse as much as
possible during the next couple weeks.

-- 
Sumana Harihareswara
Warehouse project manager
Changeset Consulting
https://changeset.nyc

Re: the new PyPI, coming next month

[Dominik George]

Hi,

> To be clear, PGP signatures can still be uploaded and they are still
> available for download, they just don’t appear in the UI anymore.

So, what does the pypi.debian.net redirector use for uscan?  I imagine it
used to scrape the website.  Can it be changed to use the JSON API?

>  Longer term I’d *like* to get rid of PGP signatures, because I think
> their value here is actually pretty low.

I partially share this opinion, but that's a question to be discusses with
the Debian policy people in general.  While checking a GPG signature on the
source tarball in general is a good idea, I am afraid some developers just
drop any key they find on first glance into the package and are done with
it, which actually provides nothing but a false sense of safety.

> In that case they’d be replaced with TUF, but that’s a longer term
> project.

That one?: https://github.com/theupdateframework/tuf

Well, I can only say *please* do not remove the possibility to upload signed
source tarballs, but leave that to the developers!

-nik

-- 
PGP-Fingerprint: 3C9D 54A4 7575 C026 FB17  FD26 B79A 3C16 A0C4 F296

Dominik George · Hundeshagenstr. 26 · 53225 Bonn
Phone: +49 228 92934581 · https://www.dominik-george.de/

Teckids e.V. · FrOSCon e.V. · Debian Developer

LPIC-3 Linux Enterprise Professional (Security)


signature.asc
Description: PGP signature

Re: the new PyPI, coming next month

[Sumana Harihareswara]

Scott,

Thanks for your reply. I wrote about this at a little more length in
https://mail.python.org/pipermail/python-list/2018-March/732329.html in
response to a related question. But for more discussion on this
particular point, the people you want to talk with are in the Python
distribution/packaging SIG list,
https://mail.python.org/mailman/listinfo/distutils-sig . Sorry to be
pushing you to yet another list, but the in-depth answers you want,
you're more likely to get there.

thanks,
Sumana Harihareswara
-- 
Sumana Harihareswara
Warehouse project manager
Changeset Consulting
https://changeset.nyc

On 03/31/2018 11:23 PM, Scott Kitterman wrote:
> What replaces gpg for ensuring integrity of the uploaded code?
> 
> Scott K
> 
> On April 1, 2018 2:15:54 AM UTC, Sumana Harihareswara  
> wrote:
>> Debian-Python experts,
>>
>> I'm writing to you in hopes you will forward this to the right places,
>> and file relevant bugs against uscan/watch, which I don't quite
>> understand enough to do myself. And if you want to follow up on
>> https://github.com/pypa/warehouse/issues/358#issuecomment-337233792 and
>> file a new issue asking for us to support your redirector more cleanly,
>> I'd welcome that.
>>
>> I'm the project manager for the new Python Package Index (Warehouse),
>> which is currently in beta at http://pypi.org/ . On the Warehouse
>> roadmap[1], it looks like the full switch will happen sometime
>> in April, so here's a heads-up about why we're switching, what's
>> changed, and what to expect. (Much of it won't be directly important to
>> you, but I figure you might want to know anyway!)
>>
>> The legacy PyPI site at https://pypi.python.org started in the early
>> 2000s. In recent years, users faced outages, malicious packages, and
>> spam attacks, and the legacy codebase made it hard to maintain and even
>> harder to develop new features.
>>
>> The new PyPI has a far more modern look, and is up-to-date under the
>> hood as well; a proper web framework (Pyramid), 100% backend test
>> coverage, and a Docker-based development environment, make it easier
>> for
>> current and new developers to maintain it and add features.
>>
>> Thanks to Mozilla's Open Source Support funding[2], developers have
>> added many new features, overhauled infrastructure, and made steady
>> progress towards redirecting traffic to the new site and shutting down
>> the old one. As of the middle of last year, package releases must go
>> through the new PyPI, and as of late February, new user account
>> registration is only available on the new site. The full switch will
>> include redirecting browser and pip install traffic from the old site;
>> then, sometime in late April or early May, the legacy site will be
>> entirely shut down.
>>
>> Thanks to redirects, you may not have to change anything immediately.
>> Here's a migration guide.[3]
>>
>>
>> Some new PyPI features:
>> * mobile-responsive UI
>> * chronological release history for each project (example[4])
>> * easy-to-read project activity journal for project maintainers
>> * better search and filtering
>> * support for multiple project URLs (e.g., for a homepage and a
>>   repo[5])
>> * user-visible Gravatars and email addresses for maintainers
>> * no need to "register" a project before initial upload
>> * far better backend infrastructure, reducing the frequency of outages
>>
>>
>> Things that are going away, or already have (sometimes for policy or
>> spam-fighting reasons), include:
>> * pythonhosted.com documentation hosting (pypa/warehouse#582[6])
>> * download counts visible in the API[7] (instead, use the Google
>>   BigQuery service[8])
>> * GPG/PGP signatures for packages (still visible in the Simple Project
>>   API[9] per PEP 503[10], but no longer visible in the web UI
>> * key management: PyPI no longer has a UI for users to manage their GPG
>>   or SSH public keys
>> * package maintainers being able to upload a new release via the web UI
>>   (instead, the recommended command-line tool is Twine[11])
>> * package maintainers being able to log in and update release
>>  descriptions via the web UI (to update release metadata, they need to
>>   upload a new release; see distutils-sig discussion[12])
>> * OpenID and Google auth login[13]
>> * users being able to upload a package without verifying their email
>>   address with PyPI first
>> * HTTP access to APIs; now it's HTTPS-only[14]
>>
>>
>> And in the works:
>> * PEP 541[15] will enable more timely package takeovers, as people get
>>   package names transferred to them after conflict resolution
>> * Now that PEP 566 has been approved, developers are working to get
>>   Markdown supported for README files on PyPI[16]
>>
>>
>> For future updates, please sign up for the low-traffic PyPI
>> announcements email list[17].
>>
>> Thank you for integrating with PyPI, and please let us know[18] if you
>> have any questions or problems with the new site!
>> --
>> Sumana Harihareswara
>> Changeset 

Re: the new PyPI, coming next month

[Scott Kitterman]

What replaces gpg for ensuring integrity of the uploaded code?

Scott K

On April 1, 2018 2:15:54 AM UTC, Sumana Harihareswara  
wrote:
>Debian-Python experts,
>
>I'm writing to you in hopes you will forward this to the right places,
>and file relevant bugs against uscan/watch, which I don't quite
>understand enough to do myself. And if you want to follow up on
>https://github.com/pypa/warehouse/issues/358#issuecomment-337233792 and
>file a new issue asking for us to support your redirector more cleanly,
>I'd welcome that.
>
>I'm the project manager for the new Python Package Index (Warehouse),
>which is currently in beta at http://pypi.org/ . On the Warehouse
>roadmap[1], it looks like the full switch will happen sometime
>in April, so here's a heads-up about why we're switching, what's
>changed, and what to expect. (Much of it won't be directly important to
>you, but I figure you might want to know anyway!)
>
>The legacy PyPI site at https://pypi.python.org started in the early
>2000s. In recent years, users faced outages, malicious packages, and
>spam attacks, and the legacy codebase made it hard to maintain and even
>harder to develop new features.
>
>The new PyPI has a far more modern look, and is up-to-date under the
>hood as well; a proper web framework (Pyramid), 100% backend test
>coverage, and a Docker-based development environment, make it easier
>for
>current and new developers to maintain it and add features.
>
>Thanks to Mozilla's Open Source Support funding[2], developers have
>added many new features, overhauled infrastructure, and made steady
>progress towards redirecting traffic to the new site and shutting down
>the old one. As of the middle of last year, package releases must go
>through the new PyPI, and as of late February, new user account
>registration is only available on the new site. The full switch will
>include redirecting browser and pip install traffic from the old site;
>then, sometime in late April or early May, the legacy site will be
>entirely shut down.
>
>Thanks to redirects, you may not have to change anything immediately.
>Here's a migration guide.[3]
>
>
>Some new PyPI features:
> * mobile-responsive UI
> * chronological release history for each project (example[4])
> * easy-to-read project activity journal for project maintainers
> * better search and filtering
> * support for multiple project URLs (e.g., for a homepage and a
>   repo[5])
> * user-visible Gravatars and email addresses for maintainers
> * no need to "register" a project before initial upload
> * far better backend infrastructure, reducing the frequency of outages
>
>
>Things that are going away, or already have (sometimes for policy or
>spam-fighting reasons), include:
> * pythonhosted.com documentation hosting (pypa/warehouse#582[6])
> * download counts visible in the API[7] (instead, use the Google
>   BigQuery service[8])
> * GPG/PGP signatures for packages (still visible in the Simple Project
>   API[9] per PEP 503[10], but no longer visible in the web UI
>* key management: PyPI no longer has a UI for users to manage their GPG
>   or SSH public keys
>* package maintainers being able to upload a new release via the web UI
>   (instead, the recommended command-line tool is Twine[11])
> * package maintainers being able to log in and update release
>  descriptions via the web UI (to update release metadata, they need to
>   upload a new release; see distutils-sig discussion[12])
> * OpenID and Google auth login[13]
> * users being able to upload a package without verifying their email
>   address with PyPI first
> * HTTP access to APIs; now it's HTTPS-only[14]
>
>
>And in the works:
> * PEP 541[15] will enable more timely package takeovers, as people get
>   package names transferred to them after conflict resolution
> * Now that PEP 566 has been approved, developers are working to get
>   Markdown supported for README files on PyPI[16]
>
>
>For future updates, please sign up for the low-traffic PyPI
>announcements email list[17].
>
>Thank you for integrating with PyPI, and please let us know[18] if you
>have any questions or problems with the new site!
>--
>Sumana Harihareswara
>Changeset Consulting
>https://changeset.nyc
>
>
>Links:
>
>   1. https://wiki.python.org/psf/WarehouseRoadmap
>   2.
>https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html
>   3.
>https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating-to-the-new-pypi
>   4. https://pypi.org/project/pip/#history
>   5.
>https://packaging.python.org/tutorials/distributing-packages/#project-urls
>   6. https://github.com/pypa/warehouse/issues/582
>   7.
>https://warehouse.readthedocs.io/api-reference/xml-rpc/#changes-to-legacy-api
>8.
>https://packaging.python.org/guides/analyzing-pypi-package-downloads/
>   9.
>https://warehouse.readthedocs.io/api-reference/legacy/#simple-project-api
>  10. https://www.python.org/dev/peps/pep-0503/
>  11. http://twine.readthedocs.io/
>  12.

the new PyPI, coming next month

[Sumana Harihareswara]

Debian-Python experts,

I'm writing to you in hopes you will forward this to the right places,
and file relevant bugs against uscan/watch, which I don't quite
understand enough to do myself. And if you want to follow up on
https://github.com/pypa/warehouse/issues/358#issuecomment-337233792 and
file a new issue asking for us to support your redirector more cleanly,
I'd welcome that.

I'm the project manager for the new Python Package Index (Warehouse),
which is currently in beta at http://pypi.org/ . On the Warehouse
roadmap[1], it looks like the full switch will happen sometime
in April, so here's a heads-up about why we're switching, what's
changed, and what to expect. (Much of it won't be directly important to
you, but I figure you might want to know anyway!)

The legacy PyPI site at https://pypi.python.org started in the early
2000s. In recent years, users faced outages, malicious packages, and
spam attacks, and the legacy codebase made it hard to maintain and even
harder to develop new features.

The new PyPI has a far more modern look, and is up-to-date under the
hood as well; a proper web framework (Pyramid), 100% backend test
coverage, and a Docker-based development environment, make it easier for
current and new developers to maintain it and add features.

Thanks to Mozilla's Open Source Support funding[2], developers have
added many new features, overhauled infrastructure, and made steady
progress towards redirecting traffic to the new site and shutting down
the old one. As of the middle of last year, package releases must go
through the new PyPI, and as of late February, new user account
registration is only available on the new site. The full switch will
include redirecting browser and pip install traffic from the old site;
then, sometime in late April or early May, the legacy site will be
entirely shut down.

Thanks to redirects, you may not have to change anything immediately.
Here's a migration guide.[3]


Some new PyPI features:
 * mobile-responsive UI
 * chronological release history for each project (example[4])
 * easy-to-read project activity journal for project maintainers
 * better search and filtering
 * support for multiple project URLs (e.g., for a homepage and a
   repo[5])
 * user-visible Gravatars and email addresses for maintainers
 * no need to "register" a project before initial upload
 * far better backend infrastructure, reducing the frequency of outages


Things that are going away, or already have (sometimes for policy or
spam-fighting reasons), include:
 * pythonhosted.com documentation hosting (pypa/warehouse#582[6])
 * download counts visible in the API[7] (instead, use the Google
   BigQuery service[8])
 * GPG/PGP signatures for packages (still visible in the Simple Project
   API[9] per PEP 503[10], but no longer visible in the web UI
 * key management: PyPI no longer has a UI for users to manage their GPG
   or SSH public keys
 * package maintainers being able to upload a new release via the web UI
   (instead, the recommended command-line tool is Twine[11])
 * package maintainers being able to log in and update release
   descriptions via the web UI (to update release metadata, they need to
   upload a new release; see distutils-sig discussion[12])
 * OpenID and Google auth login[13]
 * users being able to upload a package without verifying their email
   address with PyPI first
 * HTTP access to APIs; now it's HTTPS-only[14]


And in the works:
 * PEP 541[15] will enable more timely package takeovers, as people get
   package names transferred to them after conflict resolution
 * Now that PEP 566 has been approved, developers are working to get
   Markdown supported for README files on PyPI[16]


For future updates, please sign up for the low-traffic PyPI
announcements email list[17].

Thank you for integrating with PyPI, and please let us know[18] if you
have any questions or problems with the new site!
--
Sumana Harihareswara
Changeset Consulting
https://changeset.nyc


Links:

   1. https://wiki.python.org/psf/WarehouseRoadmap
   2.
https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html
   3.
https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating-to-the-new-pypi
   4. https://pypi.org/project/pip/#history
   5.
https://packaging.python.org/tutorials/distributing-packages/#project-urls
   6. https://github.com/pypa/warehouse/issues/582
   7.
https://warehouse.readthedocs.io/api-reference/xml-rpc/#changes-to-legacy-api
   8. https://packaging.python.org/guides/analyzing-pypi-package-downloads/
   9.
https://warehouse.readthedocs.io/api-reference/legacy/#simple-project-api
  10. https://www.python.org/dev/peps/pep-0503/
  11. http://twine.readthedocs.io/
  12.
https://mail.python.org/pipermail/distutils-sig/2017-December/031826.html
  13.
https://mail.python.org/pipermail/distutils-sig/2018-January/031855.html
  14.
https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html
  15.