Re: [Distutils] What's the use case of testpypi?

[Nick Coghlan]

On 31 October 2017 at 23:53, Nick Coghlan  wrote:

> On 31 October 2017 at 22:49, Jeremy Stanley  wrote:
>
>> On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
>> > Ideally we'd be recommending
>> > https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks
>> looking
>> > to develop a robust pre-release artifact testing workflow.
>> >
>> > While we mention it a couple of times on packaging.python.org [1,2],
>> and
>> > include it in the packaging related Non-PyPA Projects list [3], we don't
>> > really emphasise that it makes a much better platform for pre-release
>> > testing and private experimentation than PyPI itself does (see
>> > https://devpi.net/ for an example of a deployed instance with multiple
>> > distinct user namespaces).
>> >
>> > Given Donald's comment about the current test PyPI not being
>> particularly
>> > good at any of its roles, perhaps it would make sense to aim to replace
>> it
>> > with a periodically purged DevPi instance?
>> [...]
>>
>> Do the two share enough common code for successful uploading to a
>> devpi instance to be indicative of whether PyPI will accept or
>> reject on the grounds of, e.g., invalid trove classifiers (this one
>> in particular has been the most common preventable but otherwise
>> untestable upload failure our community encounters).
>>
>
> Even if they don't currently, that's likely an easier problem to fix than
> working out how to intermingle devpi's "pre-release developer sandbox" user
> experience with PyPI's "authoritative release publishing platform" user
> experience in one service, especially since the Legacy PyPI -> Warehouse
> migration is already encouraging several of those validation features to be
> extracted out to helper libraries.
>

Taking the classifiers list as a specific example: while that's *currently*
stored in the live PyPI database, there isn't anything in particular that
says that that needs to be the authoritative source of the list - it could
be redefined as a cache of a list that PyPI picks up from somewhere else.

It could, for example, be moved into a static data file in the `packaging`
project, and have updates be deployed to the live service by way of new
releases of that project. Tools could then decide for themselves whether to
depend on the static list in the packaging project, or query PyPI to get
the list for whichever version of the library PyPI itself was using.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's the use case of testpypi?

[Nick Coghlan]

On 31 October 2017 at 22:49, Jeremy Stanley  wrote:

> On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
> > Ideally we'd be recommending
> > https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks
> looking
> > to develop a robust pre-release artifact testing workflow.
> >
> > While we mention it a couple of times on packaging.python.org [1,2], and
> > include it in the packaging related Non-PyPA Projects list [3], we don't
> > really emphasise that it makes a much better platform for pre-release
> > testing and private experimentation than PyPI itself does (see
> > https://devpi.net/ for an example of a deployed instance with multiple
> > distinct user namespaces).
> >
> > Given Donald's comment about the current test PyPI not being particularly
> > good at any of its roles, perhaps it would make sense to aim to replace
> it
> > with a periodically purged DevPi instance?
> [...]
>
> Do the two share enough common code for successful uploading to a
> devpi instance to be indicative of whether PyPI will accept or
> reject on the grounds of, e.g., invalid trove classifiers (this one
> in particular has been the most common preventable but otherwise
> untestable upload failure our community encounters).
>

Even if they don't currently, that's likely an easier problem to fix than
working out how to intermingle devpi's "pre-release developer sandbox" user
experience with PyPI's "authoritative release publishing platform" user
experience in one service, especially since the Legacy PyPI -> Warehouse
migration is already encouraging several of those validation features to be
extracted out to helper libraries.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's the use case of testpypi?

[Jeremy Stanley]

On 2017-10-31 13:09:34 + (+), Thomas Kluyver wrote:
[...]
> If anyone wants to adapt the code to use in other tools, it's
> here:
> 
> https://github.com/takluyver/flit/blob/ca08119/flit/inifile.py#L63-L108
[...]

Thanks for the pointer! That would make for a really great
stand-alone linting utility or maybe a flake8 plug-in (particularly
if it vendored a copy of the classifiers list and included an
option/flag to skip downloading).
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's the use case of testpypi?

[Thomas Kluyver]

On Tue, Oct 31, 2017, at 12:49 PM, Jeremy Stanley wrote:
> Do the two share enough common code for successful uploading to a
> devpi instance to be indicative of whether PyPI will accept or
> reject on the grounds of, e.g., invalid trove classifiers (this one
> in particular has been the most common preventable but otherwise
> untestable upload failure our community encounters).

It's feasible for tools to test this before upload. Flit does this. If
anyone wants to adapt the code to use in other tools, it's here:

https://github.com/takluyver/flit/blob/ca081197df229a0df07e98c6dece13e62e2bdcd1/flit/inifile.py#L63-L108

It's definitely useful to have a testing site based on the same code as
PyPI, but I hope that with smarter tools, building invalid packages
should be less common.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's the use case of testpypi?

[Jeremy Stanley]

On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
> Ideally we'd be recommending
> https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking
> to develop a robust pre-release artifact testing workflow.
> 
> While we mention it a couple of times on packaging.python.org [1,2], and
> include it in the packaging related Non-PyPA Projects list [3], we don't
> really emphasise that it makes a much better platform for pre-release
> testing and private experimentation than PyPI itself does (see
> https://devpi.net/ for an example of a deployed instance with multiple
> distinct user namespaces).
> 
> Given Donald's comment about the current test PyPI not being particularly
> good at any of its roles, perhaps it would make sense to aim to replace it
> with a periodically purged DevPi instance?
[...]

Do the two share enough common code for successful uploading to a
devpi instance to be indicative of whether PyPI will accept or
reject on the grounds of, e.g., invalid trove classifiers (this one
in particular has been the most common preventable but otherwise
untestable upload failure our community encounters).
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's the use case of testpypi?

[Nick Coghlan]

On 31 October 2017 at 03:41, Toshio Kuratomi  wrote:

> When we locked down pypi to prevent uploading an sdist to overwrite a
> previous I remember that some people wanted a brief window to check
> for brown paper bag issues and be able to upload a new tarball in that
> window if needed.  IIRC, those people were told to use testpypi for
> that sort of thing.  Upload potential tarball to testpypi.  If it
> works, go ahead and rerun to upload to the real pypi.  If it doesn't,
> fix it and try again.
>

Ideally we'd be recommending
https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking
to develop a robust pre-release artifact testing workflow.

While we mention it a couple of times on packaging.python.org [1,2], and
include it in the packaging related Non-PyPA Projects list [3], we don't
really emphasise that it makes a much better platform for pre-release
testing and private experimentation than PyPI itself does (see
https://devpi.net/ for an example of a deployed instance with multiple
distinct user namespaces).

Given Donald's comment about the current test PyPI not being particularly
good at any of its roles, perhaps it would make sense to aim to replace it
with a periodically purged DevPi instance?

Cheers,
Nick.

[1]
https://packaging.python.org/guides/index-mirrors-and-caches/?highlight=devpi#caching-with-devpi
[2]
https://packaging.python.org/guides/hosting-your-own-index/?highlight=devpi
[3]
https://packaging.python.org/key_projects/?highlight=devpi#non-pypa-projects

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's the use case of testpypi?

[Donald Stufft]

> On Oct 30, 2017, at 1:41 PM, Toshio Kuratomi  wrote:
> 
> I figured that unless I
> asked, I would never know the answer :-)
> 


TestPyPI has a couple of use cases, and it’s not very good at any of them.

One use case is to serve as a staging site for production PyPI to test changes 
before they go to live. If I had to pick any use case as it’s “core” use case, 
it would be this one, it’s currently optimized for it and we’re unlikely to 
make changes that deviate from it. Unfortunately it’s not really good at this 
because staging sites are generally not a great way to prevent regression 
(versus something like automated testing), so with Warehouse TestPyPI is no 
longer a staging site for the production code, and they are deployed at the 
same time.

Another use case is the one you specified, someone wants to test their releases 
prior to uploading to the “real” PyPI. TestPyPI is not great at this use case 
both because, as you noted, it has the same “you can only upload a file once” 
semantics as PyPI does, but also because it doesn’t have a full copy of 
everything on PyPI (and worse, it potentially has broken or insecure or 
malicious versions of your dependencies) so pip might either be blatantly 
broken OR produce subtly broken installs. Long term the solution that I believe 
has merit for this, is the ability to, on regular PyPI, “stage” an upload to a 
temporary location that would automatically be deleted after some period of 
time, and that once you were satisfied with it, could be published/committed 
and thus be available to normal ``pip install`` invocations. This not only 
would allow testing a release beforehand, but it would also allow projects that 
produce many different artifacts across a variety of platforms to collect them 
all and upload them and make them all available immediately instead of one at a 
time. You can see more about this idea/use case at 
https://github.com/pypa/warehouse/issues/726 
.

The third relevant use case of PyPI is people simply messing around or doing 
testing of a generalized workflow tool but not specific to a specific package. 
This could also include people who are learning how to produce a package, but 
using a trivial throw away project (the infamous printer of nested lists is an 
example of this). TestPyPI can be used for this as well, but again it’s not 
really good at it. The key difference here is that ideally these projects are 
either ephemeral or namespaced to the specific user. I don’t have a great idea 
for how best to handle this, the immediate thought is a secondary deployment of 
Warehouse (such as what TestPyPI is now), that just periodically and 
automatically purges any projects > N days old to give it a more of a “sandbox” 
vibe and use case. Given the aforementioned staging feature, we would likely 
otherwise keep the same ruleset as exists on the production PyPI (and ideally, 
we’d be re-using credentials too, instead of it literally being a complete 
secondary deployment). That idea isn’t set in stone and I haven’t done a lot of 
thinking about it yet, but discussion of that particular use case and ideas to 
fix it would take place in https://github.com/pypa/warehouse/issues/2286 
.

So longer term, I hope to shut down TestPyPI and replace it with features 
and/or deployments more suited to whatever specific task/use case we’re hoping 
to have served by it, rather than just sort of being poor at everything.

- dstufft___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's the use case of testpypi?

[Mariatta Wijaya]

TestPyPI can be useful for those who are not package maintainers, but want
to learn about Python Packaging and need a 'playground' to experiment with
it.


Mariatta Wijaya

On Mon, Oct 30, 2017 at 10:41 AM, Toshio Kuratomi 
wrote:

> When we locked down pypi to prevent uploading an sdist to overwrite a
> previous I remember that some people wanted a brief window to check
> for brown paper bag issues and be able to upload a new tarball in that
> window if needed.  IIRC, those people were told to use testpypi for
> that sort of thing.  Upload potential tarball to testpypi.  If it
> works, go ahead and rerun to upload to the real pypi.  If it doesn't,
> fix it and try again.
>
> This past week I decided to try out that workflow for a project that
> I'm managing the releases for and ran into a snag.  testpypi has the
> same behaviour as production pypi wherein you can only upload a given
> sdist once and afterwards it's no longer allowed.  For the use case
> above this is problematic.  It essentially changes the idea of "test
> your release on testpypi before making a release" into "You can have
> to chances to get it right if you use testpypi" which, although better
> than uploading directly to pypi, still leaves a lot of room for error
> (let's face it: I know that if I'm stuck releasing late at night due
> to time constraints and make a mistake, chances are better than normal
> that my fix won't be the perfection that my ordinary code is and could
> have other showstopper bugs that I'd want my testing to catch as well
> ;-)
>
> Is this something that we could change for testpypi?  It could be
> implemented in many ways: straight overwrite, being able to destroy a
> version so that it seems to never have existed, or being able to
> destroy and recreate a package so that it has no uploaded sdists
> recorded.
>
> On the other side of the usefulness of enabling the testing use case
> above, such a change would be a difference between testpypi and
> production pypi meaning that it would no longer be testing exactly the
> same functionality as will be deployed in production.  I'm not sure if
> that's a more important consideration or not.  I figured that unless I
> asked, I would never know the answer :-)
>
> Thanks,
> -Toshio
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] What's the use case of testpypi?

[Toshio Kuratomi]

When we locked down pypi to prevent uploading an sdist to overwrite a
previous I remember that some people wanted a brief window to check
for brown paper bag issues and be able to upload a new tarball in that
window if needed.  IIRC, those people were told to use testpypi for
that sort of thing.  Upload potential tarball to testpypi.  If it
works, go ahead and rerun to upload to the real pypi.  If it doesn't,
fix it and try again.

This past week I decided to try out that workflow for a project that
I'm managing the releases for and ran into a snag.  testpypi has the
same behaviour as production pypi wherein you can only upload a given
sdist once and afterwards it's no longer allowed.  For the use case
above this is problematic.  It essentially changes the idea of "test
your release on testpypi before making a release" into "You can have
to chances to get it right if you use testpypi" which, although better
than uploading directly to pypi, still leaves a lot of room for error
(let's face it: I know that if I'm stuck releasing late at night due
to time constraints and make a mistake, chances are better than normal
that my fix won't be the perfection that my ordinary code is and could
have other showstopper bugs that I'd want my testing to catch as well
;-)

Is this something that we could change for testpypi?  It could be
implemented in many ways: straight overwrite, being able to destroy a
version so that it seems to never have existed, or being able to
destroy and recreate a package so that it has no uploaded sdists
recorded.

On the other side of the usefulness of enabling the testing use case
above, such a change would be a difference between testpypi and
production pypi meaning that it would no longer be testing exactly the
same functionality as will be deployed in production.  I'm not sure if
that's a more important consideration or not.  I figured that unless I
asked, I would never know the answer :-)

Thanks,
-Toshio
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig