Re: [Distutils] PyPi not allowing duplicate filenames

[Paul Moore]

On 14 October 2015 at 19:48, Matthew Brett  wrote:
> I'm guessing that this is what y'all expected?

Yes, that sounds right. 1.10.0.post2 is a new (different) release
number, and so a new release would be created.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Matthew Brett]

Hi,

On Wed, Oct 14, 2015 at 11:25 AM, Nathaniel Smith  wrote:
> On Oct 14, 2015 11:12 AM, "Donald Stufft"  wrote:
>>
> [...]
>>> Apparently some packages were making assumptions about the format of the
>>> numpy.__version__ string, and having .postN in there caused errors when they
>>> tried to process it. (It would be helpful if there were a little
>>> permissively licensed standalone implementation of PEP 440 comparisons,
>>> suitable for the "if pkg.version > ...:" checks that people insist on doing
>>> -- I couldn't find one in some quick searches.)
>>
>> https://github.com/pypa/packaging
>>
>> It’s what both pip and setuptools use (though we embed it, but it’s fine
>> to depend on it too).
>
> That's under Apache 2, so it can't be used by GPLv2 packages, or any package
> that might be used by GPLv2 packages.
>
>>>
>>> IIUC, the specific problems numpy ran into that caused the creation of
>>> .postN releases were:
>>> - oops, didn't sign the uploads, re-upload identical file with proper
>>> signature attached -> not allowed. (I'm not sure if these were embedded or
>>> detached signatures. Either way it'd be nice if pypi allowed it, but for
>>> embedded signatures in particular I can see how this might be a hassle.)
>>
>> I don’t think we allow embedded signatures, it would be reasonable to
>> allow uploading detached signatures after the fact though.
>>>
>>>
>>> - our OS X maintainer tried to use twine to upload OS X wheels for the
>>> existing release; instead it created a new release. Not sure if a bug was
>>> filed on twine, but if not then one probably should be. As a workaround our
>>> release docs now say "always upload wheels by hand using the web interface,
>>> never use setup.py upload or twine".
>>
>> This shouldn’t create a new release unless you’ve changed the version
>> number (including adding post releases). If you can reproduce on Test PyPI I
>> can fix it.
>
> Matthew? Any thoughts?

Sorry - I'm not sure whether y'all are surprised by the behavior of
twine in this situation.

Specifically, the situation was:

* There was a release on pypi was labeled 1.10.0 :
https://pypi.python.org/pypi/numpy/1.10.0;
* The source archives that Chuck uploaded were from code that gave a
version string of '1.10.0.post2';
* I built wheels from this code, that had this version encoded in the
filename - e.g.
http://wheels.scikit-image.org/numpy-1.10.0.post2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
;
* I used `twine upload
http://wheels.scikit-image.org/numpy-1.10.0.post2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl`
etc to upload these.

When I did this, pypi created a new release called
https://pypi.python.org/pypi/numpy/1.10.0.post2 and put the wheels
there.

I'm guessing that this is what y'all expected?

Best,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Nathaniel Smith]

On Wed, Oct 14, 2015 at 3:02 PM, Donald Stufft  wrote:
> On October 14, 2015 at 5:17:42 PM, Nathaniel Smith (n...@pobox.com) wrote:
>> > @Donald: is there somewhere we should be filing these warehouse
>> wish list things so they don't get lost? :-)
>
> https://github.com/pypa/warehouse/issues

https://github.com/pypa/warehouse/issues/726

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Ben Finney]

Ben Finney  writes:

> Apache License 2.0 imposes no additional reastrictions on the
> recipient of a work licensed under GNU GPL v3. So that meets the
> conditions of both.

Apache License 2.0 does impose further restrictions not found in GPL v2
though. Sorry for the confusion.


Nathaniel Smith  writes:

> The FSF believes that Apache 2 is incompatible with GPLv2 (but v3 is
> OK)

Yes, I'd neglected that. 


-- 
 \“My house is made out of balsa wood, so when I want to scare |
  `\ the neighborhood kids I lift it over my head and tell them to |
_o__) get out of my yard or I'll throw it at them.” —Steven Wright |
Ben Finney

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Xavier Fernandez]

FWIW https://github.com/pypa/pip/pull/3146 exists for the wheel cache.

Concerning http cache and https://github.com/ionrock/cachecontrol the main
issue is that the cached request is not present in the cache. We only have
a "hash -> response" mapping.
I plan to make a PR asking to also keep the request for introspection and
move to a "hash -> (request, response)" mapping.

On Wed, Oct 14, 2015 at 11:01 PM, Glyph Lefkowitz 
wrote:

>
> On Oct 14, 2015, at 1:04 PM, Donald Stufft  wrote:
>
> Generally within 60-120 seconds it’s available in mirrors (most of them
> resync once a minute). If anyone has downloaded it then they will have
> pretty much permanently cached the package, first in the download cache and
> then again in the wheel cache (assuming it wasn’t a wheel already, and they
> had that enabled). The original package was NumPy. It had 30,982 downloads
> in the last day, so we can average that out to 1290 downloads an hour or 21
> downloads a minute. If it takes you two minutes to notice it and delete it,
> then there are ~40 people who already have the original version cached and
> who will not notice the updated version.
>
>
> While I don't think PyPI should allow modification of uploaded packages
> necessarily, I do think that Pip's caching is (A) too aggressive and (B)
> too opaque.  For example:
>
> https://github.com/pypa/pip/issues/3127
> https://github.com/pypa/pip/issues/3034
> https://github.com/pypa/pip/issues/3025
> https://github.com/pypa/pip/issues/2908
> https://github.com/pypa/pip/issues/2882
>
> etc, etc.
>
> I know there are some directories platform-specific directories I can
> delete, but almost once a day I want a command like `pip cache show´ which
> can show me what is cached and when/where it was built, `pip cache clear´
> or `pip cache remove twisted´ or `pip cache remove cffi>=1.0´.  I don't
> want to have to care if it's in the HTTP cache or the wheel cache, or how
> it got there; I also don't want to have to bust a ~200 megabyte cache that
> saves me hours a day just because there's one bad entry in there.
>
> -glyph
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Nathaniel Smith]

On Oct 14, 2015 10:04 AM, "Ionel Cristian Mărieș" 
wrote:
>
>
> On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker 
wrote:
>>
>> some packages were unable to work with the postN suffix.
>
>
> ​Can you elaborate a bit more here?​
>

Apparently some packages were making assumptions about the format of the
numpy.__version__ string, and having .postN in there caused errors when
they tried to process it. (It would be helpful if there were a little
permissively licensed standalone implementation of PEP 440 comparisons,
suitable for the "if pkg.version > ...:" checks that people insist on doing
-- I couldn't find one in some quick searches.)

IIUC, the specific problems numpy ran into that caused the creation of
.postN releases were:
- oops, didn't sign the uploads, re-upload identical file with proper
signature attached -> not allowed. (I'm not sure if these were embedded or
detached signatures. Either way it'd be nice if pypi allowed it, but for
embedded signatures in particular I can see how this might be a hassle.)
- our OS X maintainer tried to use twine to upload OS X wheels for the
existing release; instead it created a new release. Not sure if a bug was
filed on twine, but if not then one probably should be. As a workaround our
release docs now say "always upload wheels by hand using the web interface,
never use setup.py upload or twine".

My feeling is that pypi is correct to disallow the mutation of releases
once they become public, but that the ergonomics around this could probably
be improved :-). A more general solution that might be nice to have Someday
would be if you could upload a release in one step, and then get a private
link to poke at what was uploaded and make sure it looks correct, before
making it public in a second step.

CC'ing the release manager and OS X maintainer in question, in case I got
something wrong or more details are wanted...

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (n...@pobox.com) wrote:
> On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote:
> >
> [...]
> >> Apparently some packages were making assumptions about the format of the
> numpy.__version__ string, and having .postN in there caused errors when
> they tried to process it. (It would be helpful if there were a little
> permissively licensed standalone implementation of PEP 440 comparisons,
> suitable for the "if pkg.version > ...:" checks that people insist on doing
> -- I couldn't find one in some quick searches.)
> >
> > https://github.com/pypa/packaging
> >
> > It’s what both pip and setuptools use (though we embed it, but it’s fine
> to depend on it too).
>  
> That's under Apache 2, so it can't be used by GPLv2 packages, or any
> package that might be used by GPLv2 packages.

I suspect it’d be trivial to relicense it. There’s a total of 6 contributors 
and I think I know how to get ahold of all of them.

>  
> >>
> >> IIUC, the specific problems numpy ran into that caused the creation of
> .postN releases were:
> >> - oops, didn't sign the uploads, re-upload identical file with proper
> signature attached -> not allowed. (I'm not sure if these were embedded or
> detached signatures. Either way it'd be nice if pypi allowed it, but for
> embedded signatures in particular I can see how this might be a hassle.)
> >
> > I don’t think we allow embedded signatures, it would be reasonable to
> allow uploading detached signatures after the fact though.
> >>
> >>
> >> - our OS X maintainer tried to use twine to upload OS X wheels for the
> existing release; instead it created a new release. Not sure if a bug was
> filed on twine, but if not then one probably should be. As a workaround our
> release docs now say "always upload wheels by hand using the web interface,
> never use setup.py upload or twine".
> >
> > This shouldn’t create a new release unless you’ve changed the version
> number (including adding post releases). If you can reproduce on Test PyPI
> I can fix it.
>  
> Matthew? Any thoughts?
>  
> -n
>  

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Nathaniel Smith]

On Oct 14, 2015 11:12 AM, "Donald Stufft"  wrote:
>
[...]
>> Apparently some packages were making assumptions about the format of the
numpy.__version__ string, and having .postN in there caused errors when
they tried to process it. (It would be helpful if there were a little
permissively licensed standalone implementation of PEP 440 comparisons,
suitable for the "if pkg.version > ...:" checks that people insist on doing
-- I couldn't find one in some quick searches.)
>
> https://github.com/pypa/packaging
>
> It’s what both pip and setuptools use (though we embed it, but it’s fine
to depend on it too).

That's under Apache 2, so it can't be used by GPLv2 packages, or any
package that might be used by GPLv2 packages.

>>
>> IIUC, the specific problems numpy ran into that caused the creation of
.postN releases were:
>> - oops, didn't sign the uploads, re-upload identical file with proper
signature attached -> not allowed. (I'm not sure if these were embedded or
detached signatures. Either way it'd be nice if pypi allowed it, but for
embedded signatures in particular I can see how this might be a hassle.)
>
> I don’t think we allow embedded signatures, it would be reasonable to
allow uploading detached signatures after the fact though.
>>
>>
>> - our OS X maintainer tried to use twine to upload OS X wheels for the
existing release; instead it created a new release. Not sure if a bug was
filed on twine, but if not then one probably should be. As a workaround our
release docs now say "always upload wheels by hand using the web interface,
never use setup.py upload or twine".
>
> This shouldn’t create a new release unless you’ve changed the version
number (including adding post releases). If you can reproduce on Test PyPI
I can fix it.

Matthew? Any thoughts?

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Jeremy Stanley]

On 2015-10-14 09:43:25 -0700 (-0700), Chris Barker wrote:
[...]
> I propose that PyPi allow projects to replace existing files if
> they REALLY REALLY want to.
> 
> You should have to jump through all sorts of hoops, and make it
> really clear that it is a BAD IDEA in the general case, but it'd
> be good to have it be possible.
[...]

It used to be possible. If that possibility returns, I'm not looking
forward to more of the bugs I used to semi-regularly file explaining
to package authors why their assumptions about the safety of
silently replacing their broken releases were flawed.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Ionel Cristian Mărieș]

On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker  wrote:

> some packages were unable to work with the postN suffix.


​Can you elaborate a bit more here?​



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Paul Moore]

On 14 October 2015 at 19:27, Donald Stufft  wrote:
> I suspect it’d be trivial to relicense it. There’s a total of 6 contributors 
> and I think I know how to get ahold of all of them.

I only contributed a tiny bit, but I'm happy for you to relicense if
you want to.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Robert Collins]

On 15 October 2015 at 07:27, Donald Stufft  wrote:
> On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (n...@pobox.com) wrote:
>> On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote:
>> >
>> [...]
>> >> Apparently some packages were making assumptions about the format of the
>> numpy.__version__ string, and having .postN in there caused errors when
>> they tried to process it. (It would be helpful if there were a little
>> permissively licensed standalone implementation of PEP 440 comparisons,
>> suitable for the "if pkg.version > ...:" checks that people insist on doing
>> -- I couldn't find one in some quick searches.)
>> >
>> > https://github.com/pypa/packaging
>> >
>> > It’s what both pip and setuptools use (though we embed it, but it’s fine
>> to depend on it too).
>>
>> That's under Apache 2, so it can't be used by GPLv2 packages, or any
>> package that might be used by GPLv2 packages.
>
> I suspect it’d be trivial to relicense it. There’s a total of 6 contributors 
> and I think I know how to get ahold of all of them.

Presumably to MIT + ApacheV2 ?

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 3:05:57 PM, Robert Collins (robe...@robertcollins.net) 
wrote:
> On 15 October 2015 at 07:27, Donald Stufft wrote:
> > On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (n...@pobox.com) wrote:
> >> On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote:
> >> >
> >> [...]
> >> >> Apparently some packages were making assumptions about the format of the
> >> numpy.__version__ string, and having .postN in there caused errors when
> >> they tried to process it. (It would be helpful if there were a little
> >> permissively licensed standalone implementation of PEP 440 comparisons,
> >> suitable for the "if pkg.version > ...:" checks that people insist on doing
> >> -- I couldn't find one in some quick searches.)
> >> >
> >> > https://github.com/pypa/packaging
> >> >
> >> > It’s what both pip and setuptools use (though we embed it, but it’s fine
> >> to depend on it too).
> >>
> >> That's under Apache 2, so it can't be used by GPLv2 packages, or any
> >> package that might be used by GPLv2 packages.
> >
> > I suspect it’d be trivial to relicense it. There’s a total of 6 
> > contributors and I think  
> I know how to get ahold of all of them.
>  
> Presumably to MIT + ApacheV2 ?

2 Clause BSD and Apache v2.0.


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Chris Barker]

On Wed, Oct 14, 2015 at 9:56 AM, Dave Forgac 
wrote:

> This was discussed recently here:
> https://github.com/pypa/packaging-problems/issues/74
>

and on this list at other times. Though the above issue was pretty focused
on restoring a deleted file without any changes -- which seems like a
no-brainer to me, as long as someone wants to put the effort into the
infrastructure.

(the soft delete option seems like a good idea to me).

But I'm talking about the cases of "whoops! I really wish I hadn't uploaded
that one". We can improve the tooling (some discussion on this in this
thread right now...), but people are people and some of us are stupid
and/or careless. So this WILL happen.

And it just seems pedantic to say: hey -- you've already put that one there
-- maybe even two minutes ago, so there is NO WAY to fix your mistake. If
it happens quickly, then no one has downloaded it, it hasn't made its way
to the mirrors, etc...

And again -- we are all adults here: if you as the package maintainer want
to do somethign that is sonfusing to your users, is it up to PyPi to never
let you do that? (I think it IS up to PyPi to strongly recommend that you
don't -- i.e. make it hard to do, and impossible to do thoughtlessly)

On Wed, Oct 14, 2015 at 10:00 AM, Jeremy Stanley  wrote:

> > You should have to jump through all sorts of hoops, and make it
> > really clear that it is a BAD IDEA in the general case, but it'd
> > be good to have it be possible.
> [...]
>
> It used to be possible.


Was it also easy to do without very careful consideration? Or were the
hoops I mentioned in place?

I can't find it right now, but I think someone in this thread suggested a
"staging area", so we could essentially do a trail run: upload to PyPi,
tell a few trusted friends about it, have them test it, then, and only
then, push it to the official channel. Maybe the infrastructure for that
would be more work than it's worth, but I like it.

This would fit into the basic principle that you should always be able to
test something in as close the final environment as possible, before you
commit to it.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 3:58:52 PM, Chris Barker (chris.bar...@noaa.gov) wrote:
> On Wed, Oct 14, 2015 at 9:56 AM, Dave Forgac  
> wrote:
>  
> > This was discussed recently here:
> > https://github.com/pypa/packaging-problems/issues/74
> >
>  
> and on this list at other times. Though the above issue was pretty focused
> on restoring a deleted file without any changes -- which seems like a
> no-brainer to me, as long as someone wants to put the effort into the
> infrastructure.
>  
> (the soft delete option seems like a good idea to me).

I plan on implementing a soft delete in Warehouse.

>  
> But I'm talking about the cases of "whoops! I really wish I hadn't uploaded
> that one". We can improve the tooling (some discussion on this in this
> thread right now...), but people are people and some of us are stupid
> and/or careless. So this WILL happen.
>  
> And it just seems pedantic to say: hey -- you've already put that one there
> -- maybe even two minutes ago, so there is NO WAY to fix your mistake. If
> it happens quickly, then no one has downloaded it, it hasn't made its way
> to the mirrors, etc…

Generally within 60-120 seconds it’s available in mirrors (most of them resync 
once a minute). If anyone has downloaded it then they will have pretty much 
permanently cached the package, first in the download cache and then again in 
the wheel cache (assuming it wasn’t a wheel already, and they had that 
enabled). The original package was NumPy. It had 30,982 downloads in the last 
day, so we can average that out to 1290 downloads an hour or 21 downloads a 
minute. If it takes you two minutes to notice it and delete it, then there are 
~40 people who already have the original version cached and who will not notice 
the updated version.

Version numbers are free, use more of them. If you can’t just issue new 
releases quickly, then test your release before you upload it (and then upload 
it with twine) and you can even upload it to Test PyPI to test things earlier 
than that.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Ionel Cristian Mărieș]

On Wed, Oct 14, 2015 at 11:04 PM, Donald Stufft  wrote:

> Generally within 60-120 seconds it’s available in mirrors (most of them
> resync once a minute). If anyone has downloaded it then they will have
> pretty much permanently cached the package, first in the download cache and
> then again in the wheel cache (assuming it wasn’t a wheel already, and they
> had that enabled). The original package was NumPy. It had 30,982 downloads
> in the last day, so we can average that out to 1290 downloads an hour or 21
> downloads a minute. If it takes you two minutes to notice it and delete it,
> then there are ~40 people who already have the original version cached and
> who will not notice the updated version.
>

​This reminds me of Gmail's "unsend" feature where email would be delayed
10 seconds or something, giving a window to press the unsend button. Maybe
something like that could be implemented? Like a minute or two in which you
could unpublish? And a --no-regrets mode of course​, for people that want
to live the moment :-)



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Robert Collins]

On 15 October 2015 at 09:55, Glyph Lefkowitz  wrote:
>
> On Oct 14, 2015, at 10:36 AM, Nathaniel Smith  wrote:
>
> My feeling is that pypi is correct to disallow the mutation of releases once
> they become public, but that the ergonomics around this could probably be
> improved :-). A more general solution that might be nice to have Someday
> would be if you could upload a release in one step, and then get a private
> link to poke at what was uploaded and make sure it looks correct, before
> making it public in a second step.
>
> IMHO it would be really neat if 'pip' could authenticate to PyPI, and the
> authentication could affect package visibility, so that you could end-to-end
> test 'pip install foo' and ensure that you get what you expect before
> pushing the go-live switch.

And if its bust for you?

Surely testpypi is equally good and its an existing solution?

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

> On Oct 14, 2015, at 4:55 PM, Glyph Lefkowitz  wrote:
> 
> 
>> On Oct 14, 2015, at 10:36 AM, Nathaniel Smith  wrote:
>> 
>> My feeling is that pypi is correct to disallow the mutation of releases once 
>> they become public, but that the ergonomics around this could probably be 
>> improved :-). A more general solution that might be nice to have Someday 
>> would be if you could upload a release in one step, and then get a private 
>> link to poke at what was uploaded and make sure it looks correct, before 
>> making it public in a second step.
>> 
> 
> IMHO it would be really neat if 'pip' could authenticate to PyPI, and the 
> authentication could affect package visibility, so that you could end-to-end 
> test 'pip install foo' and ensure that you get what you expect before pushing 
> the go-live switch.

Can't be done without a pretty huge regression in uptime and speed. ___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 5:17:42 PM, Nathaniel Smith (n...@pobox.com) wrote:
> > @Donald: is there somewhere we should be filing these warehouse 
> wish list things so they don't get lost? :-)

https://github.com/pypa/warehouse/issues

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Chris Barker]

On Wed, Oct 14, 2015 at 1:04 PM, Donald Stufft  wrote:

> > And it just seems pedantic to say: hey -- you've already put that one
> there
> > -- maybe even two minutes ago, so there is NO WAY to fix your mistake. If
> > it happens quickly, then no one has downloaded it, it hasn't made its way
> > to the mirrors, etc…
>
> Generally within 60-120 seconds it’s available in mirrors (most of them
> resync once a minute).


interesting -- it doesn't show up in the PyPi search nearly that fast :-)

If anyone has downloaded it


Honestly, I'm really not worried about people having downloaded it -- but
if it gets mirrored almost instantaneously, then there's a problem.


> The original package was NumPy. It had 30,982 downloads in the last day,


well, let's face it, most packages are not nearly as popular as numpy...

But again, if it's been mirrored, we're stuck.

> Version numbers are free, use more of them. If you can’t just issue new
> releases quickly, then test your release before you upload it (and then
> upload it with twine) and you can even upload it to Test PyPI to test
> things earlier than that.


Bingo! I'll check out Test PyPi -- maybe that's just what I"ve been looking
for.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Chris Barker]

On Wed, Oct 14, 2015 at 1:20 PM, Ionel Cristian Mărieș 
wrote:


> ​This reminds me of Gmail's "unsend" feature where email would be delayed
> 10 seconds or something, giving a window to press the unsend button.
>

FWIW, I am a big fan of that -- and use it with remarkably frequency. --
maybe I'm just twitchier than most...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 6:24:55 PM, Chris Barker (chris.bar...@noaa.gov) wrote:
> > another note-- conda has teh concetp of a "build" that's tacked  
> on teh release for conda pacakges.
>  
> So if I updated somethign about how teh packge is buitl, but am  
> using teh same underllying version of teh package, I update teh  
> build number, get a new "version" of the package, but it's clear  
> that the pacakge itself is the same version.

Wheels have a build number concept which is independent of the version and can 
be used to upload a new build without creating a new release… at least in 
theory. I’m not sure if anyone has ever tried to actually use it though. 

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Glyph Lefkowitz]

> On Oct 14, 2015, at 1:04 PM, Donald Stufft  wrote:
> 
> Generally within 60-120 seconds it’s available in mirrors (most of them 
> resync once a minute). If anyone has downloaded it then they will have pretty 
> much permanently cached the package, first in the download cache and then 
> again in the wheel cache (assuming it wasn’t a wheel already, and they had 
> that enabled). The original package was NumPy. It had 30,982 downloads in the 
> last day, so we can average that out to 1290 downloads an hour or 21 
> downloads a minute. If it takes you two minutes to notice it and delete it, 
> then there are ~40 people who already have the original version cached and 
> who will not notice the updated version.

While I don't think PyPI should allow modification of uploaded packages 
necessarily, I do think that Pip's caching is (A) too aggressive and (B) too 
opaque.  For example:

https://github.com/pypa/pip/issues/3127
https://github.com/pypa/pip/issues/3034
https://github.com/pypa/pip/issues/3025
https://github.com/pypa/pip/issues/2908
https://github.com/pypa/pip/issues/2882

etc, etc.

I know there are some directories platform-specific directories I can delete, 
but almost once a day I want a command like `pip cache show´ which can show me 
what is cached and when/where it was built, `pip cache clear´ or `pip cache 
remove twisted´ or `pip cache remove cffi>=1.0´.  I don't want to have to care 
if it's in the HTTP cache or the wheel cache, or how it got there; I also don't 
want to have to bust a ~200 megabyte cache that saves me hours a day just 
because there's one bad entry in there.

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Nathaniel Smith]

On Oct 14, 2015 1:55 PM, "Glyph Lefkowitz"  wrote:
>
>
>> On Oct 14, 2015, at 10:36 AM, Nathaniel Smith  wrote:
>>
>> My feeling is that pypi is correct to disallow the mutation of releases
once they become public, but that the ergonomics around this could probably
be improved :-). A more general solution that might be nice to have Someday
would be if you could upload a release in one step, and then get a private
link to poke at what was uploaded and make sure it looks correct, before
making it public in a second step.
>
> IMHO it would be really neat if 'pip' could authenticate to PyPI, and the
authentication could affect package visibility, so that you could
end-to-end test 'pip install foo' and ensure that you get what you expect
before pushing the go-live switch.

The slick way to do this would be to provide a private index URL that gives
you a view of what pypi would look like after your release goes live, and
could be used like

pip install --index-url https://pypi.python.org/tmp/foo/acd1538afe267/ foo

That way you skip all the issues of adding an authentication system to pip,
plus get a URL that you can pass around to other people ("hey #twisted, can
someone who's on Windows test the new release? Just type: ...")

You'd want to expire these after some time to prevent them being abused as
private package indices for proprietary software, but that's easy enough.

@Donald: is there somewhere we should be filing these warehouse wish list
things so they don't get lost? :-)

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Chris Barker]

On Wed, Oct 14, 2015 at 1:59 PM, David Cournapeau 
wrote:

> But I'm talking about the cases of "whoops! I really wish I hadn't
>> uploaded that one". We can improve the tooling (some discussion on this in
>> this thread right now...), but people are people and some of us are stupid
>> and/or careless. So this WILL happen.
>>
>

> I have not done numpy releases for half a decade now, but it was already
>> automated enough that putting a new version was not very costly then.
>>
>
yeah, I suppose releases are cheap -- though part of the problem is that
your users are likley to think that you actually fixed a bug or something.
And maybe wonder why you went from 1.2.3 to 1.3.5 seemingly all at once...

another note-- conda has teh concetp of a "build" that's tacked on teh
release for conda pacakges.

So if I updated somethign about how teh packge is buitl, but am using teh
same underllying version of teh package, I update teh build number, get a
new "version" of the package, but it's clear that the pacakge itself is the
same version.

for instance, I'm messing around right now with building libgd for conda,
and the latest version I have up on anaconda.org is:

libgd-2.1.1

but the actual file is:

libgd-2.1.1-1.tar.bz2

(and there is a libgd-2.1.1-0.tar.bz2 there too...)

Maybe this would be helpful for PyPi, too?

I am sure there are things we can do to improve numpy's release process to
> avoid this in the future.
>

numpy was just an example -- we are all likely to make mistakes in the
future - it's human nature.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[David Cournapeau]

On Wed, Oct 14, 2015 at 8:52 PM, Chris Barker  wrote:

>
>
> On Wed, Oct 14, 2015 at 9:56 AM, Dave Forgac 
> wrote:
>
>> This was discussed recently here:
>> https://github.com/pypa/packaging-problems/issues/74
>>
>
> and on this list at other times. Though the above issue was pretty focused
> on restoring a deleted file without any changes -- which seems like a
> no-brainer to me, as long as someone wants to put the effort into the
> infrastructure.
>
> (the soft delete option seems like a good idea to me).
>
> But I'm talking about the cases of "whoops! I really wish I hadn't
> uploaded that one". We can improve the tooling (some discussion on this in
> this thread right now...), but people are people and some of us are stupid
> and/or careless. So this WILL happen.
>
> And it just seems pedantic to say: hey -- you've already put that one
> there -- maybe even two minutes ago, so there is NO WAY to fix your
> mistake. If it happens quickly, then no one has downloaded it, it hasn't
> made its way to the mirrors, etc...
>

It is not pendantic, for reasons mentioned by Donald.

I have not done numpy releases for half a decade now, but it was already
automated enough that putting a new version was not very costly then. And
you did not have travis-ci, appveyor, tox, ubiquitous AWS 5 years ago ...

I am sure there are things we can do to improve numpy's release process to
avoid this in the future.

David


> And again -- we are all adults here: if you as the package maintainer want
> to do somethign that is sonfusing to your users, is it up to PyPi to never
> let you do that? (I think it IS up to PyPi to strongly recommend that you
> don't -- i.e. make it hard to do, and impossible to do thoughtlessly)
>
> On Wed, Oct 14, 2015 at 10:00 AM, Jeremy Stanley 
>  wrote:
>
>> > You should have to jump through all sorts of hoops, and make it
>> > really clear that it is a BAD IDEA in the general case, but it'd
>> > be good to have it be possible.
>> [...]
>>
>> It used to be possible.
>
>
> Was it also easy to do without very careful consideration? Or were the
> hoops I mentioned in place?
>
> I can't find it right now, but I think someone in this thread suggested a
> "staging area", so we could essentially do a trail run: upload to PyPi,
> tell a few trusted friends about it, have them test it, then, and only
> then, push it to the official channel. Maybe the infrastructure for that
> would be more work than it's worth, but I like it.
>
> This would fit into the basic principle that you should always be able to
> test something in as close the final environment as possible, before you
> commit to it.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Glyph Lefkowitz]

> On Oct 14, 2015, at 10:36 AM, Nathaniel Smith  wrote:
> 
> My feeling is that pypi is correct to disallow the mutation of releases once 
> they become public, but that the ergonomics around this could probably be 
> improved :-). A more general solution that might be nice to have Someday 
> would be if you could upload a release in one step, and then get a private 
> link to poke at what was uploaded and make sure it looks correct, before 
> making it public in a second step.
> 

IMHO it would be really neat if 'pip' could authenticate to PyPI, and the 
authentication could affect package visibility, so that you could end-to-end 
test 'pip install foo' and ensure that you get what you expect before pushing 
the go-live switch.

-glyph___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 6:13:14 PM, Chris Barker (chris.bar...@noaa.gov) wrote:
> On Wed, Oct 14, 2015 at 1:04 PM, Donald Stufft wrote:
>  
> > > And it just seems pedantic to say: hey -- you've already put that one
> > there
> > > -- maybe even two minutes ago, so there is NO WAY to fix your mistake. If
> > > it happens quickly, then no one has downloaded it, it hasn't made its way
> > > to the mirrors, etc…
> >
> > Generally within 60-120 seconds it’s available in mirrors (most of them
> > resync once a minute).
>  
>  
> interesting -- it doesn't show up in the PyPi search nearly that fast :-)

That’s mostly because search is barely being held together right now. When all 
the new stuff shakes out, that should update within a second or two. For right 
now it’s on a 15 minute cron job.

>  
> If anyone has downloaded it
>  
>  
> Honestly, I'm really not worried about people having downloaded it -- but
> if it gets mirrored almost instantaneously, then there's a problem.
>  
>  
> > The original package was NumPy. It had 30,982 downloads in the last day,
>  
>  
> well, let's face it, most packages are not nearly as popular as numpy...
>  
> But again, if it's been mirrored, we're stuck.
>  
> > Version numbers are free, use more of them. If you can’t just issue new
> > releases quickly, then test your release before you upload it (and then
> > upload it with twine) and you can even upload it to Test PyPI to test
> > things earlier than that.
>  
>  
> Bingo! I'll check out Test PyPi -- maybe that's just what I"ve been looking
> for.
>  
> -CHB
>  
> --
>  
> Christopher Barker, Ph.D.
> Oceanographer
>  
> Emergency Response Division
> NOAA/NOS/OR (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>  
> chris.bar...@noaa.gov
>  

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Chris Barker]

On Wed, Oct 14, 2015 at 3:26 PM, Donald Stufft  wrote:

> On October 14, 2015 at 6:24:55 PM, Chris Barker (chris.bar...@noaa.gov)
> wrote:
> > > another note-- conda has teh concetp of a "build" that's tacked
> > on teh release for conda pacakges.
> >
> > So if I updated somethign about how teh packge is buitl, but am
> > using teh same underllying version of teh package, I update teh
> > build number, get a new "version" of the package, but it's clear
> > that the pacakge itself is the same version.
>
> Wheels have a build number concept which is independent of the version and
> can be used to upload a new build without creating a new release… at least
> in theory.


sounds like exactly what conda is doing (and the same context)


> I’m not sure if anyone has ever tried to actually use it though.


I expect they will -- it's pretty darn useful in conda. And it's all too
easy to build a binary wheel that works on the developer machine, but not
everywhere else... (OK, the CI systems are making that less likely..)

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Robert Collins]

On 15 October 2015 at 10:01, Glyph Lefkowitz  wrote:
>
> On Oct 14, 2015, at 1:04 PM, Donald Stufft  wrote:
>
> Generally within 60-120 seconds it’s available in mirrors (most of them
> resync once a minute). If anyone has downloaded it then they will have
> pretty much permanently cached the package, first in the download cache and
> then again in the wheel cache (assuming it wasn’t a wheel already, and they
> had that enabled). The original package was NumPy. It had 30,982 downloads
> in the last day, so we can average that out to 1290 downloads an hour or 21
> downloads a minute. If it takes you two minutes to notice it and delete it,
> then there are ~40 people who already have the original version cached and
> who will not notice the updated version.
>
>
> While I don't think PyPI should allow modification of uploaded packages
> necessarily, I do think that Pip's caching is (A) too aggressive and (B) too
> opaque.  For example:
>
> https://github.com/pypa/pip/issues/3127
> https://github.com/pypa/pip/issues/3034
> https://github.com/pypa/pip/issues/3025
> https://github.com/pypa/pip/issues/2908
> https://github.com/pypa/pip/issues/2882
>
> etc, etc.
>
> I know there are some directories platform-specific directories I can
> delete, but almost once a day I want a command like `pip cache show´ which
> can show me what is cached and when/where it was built, `pip cache clear´ or
> `pip cache remove twisted´ or `pip cache remove cffi>=1.0´.  I don't want to
> have to care if it's in the HTTP cache or the wheel cache, or how it got
> there; I also don't want to have to bust a ~200 megabyte cache that saves me
> hours a day just because there's one bad entry in there.

So - none of those need PEPs - just dive in and put a patch forward.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 5:01:49 PM, Glyph Lefkowitz (gl...@twistedmatrix.com) 
wrote:
>  
> > On Oct 14, 2015, at 1:04 PM, Donald Stufft wrote:
> >
> > Generally within 60-120 seconds it’s available in mirrors (most of them 
> > resync once  
> a minute). If anyone has downloaded it then they will have pretty much 
> permanently cached  
> the package, first in the download cache and then again in the wheel cache 
> (assuming it  
> wasn’t a wheel already, and they had that enabled). The original package was 
> NumPy. It  
> had 30,982 downloads in the last day, so we can average that out to 1290 
> downloads an hour  
> or 21 downloads a minute. If it takes you two minutes to notice it and delete 
> it, then there  
> are ~40 people who already have the original version cached and who will not 
> notice the  
> updated version.
>  
> While I don't think PyPI should allow modification of uploaded packages 
> necessarily,  
> I do think that Pip's caching is (A) too aggressive and (B) too opaque. For 
> example:
>  
> https://github.com/pypa/pip/issues/3127
> https://github.com/pypa/pip/issues/3034

These two are the same thing, and a bug. Fix the bug and the http cache is 
perfectly fine IMO. It never caches without headers or an Etag saying that it 
can. PyPI uses really long cache-control headers (1y+) on a package, and it can 
do this because it can promise that this URL will either continue to exist, or 
it’ll just go away.

There are not, AFAIK, any problems with the download cache being too aggressive 
other than one bug where it doesn’t correctly identify a failed download.

> https://github.com/pypa/pip/issues/3025

This is a problem with the wheel cache, with a path to fix it on the ticket. 
Just needs someone to implement the PR.

> https://github.com/pypa/pip/issues/2908

I think this is probably just a wontfix issue tbh. The cache isn’t designed to 
be used that way really.

> https://github.com/pypa/pip/issues/2882

Another problem with the wheel cache :/

>  
> etc, etc.
>  
> I know there are some directories platform-specific directories I can delete, 
> but almost  
> once a day I want a command like `pip cache show´ which can show me what is 
> cached and when/where  
> it was built, `pip cache clear´ or `pip cache remove twisted´ or `pip cache 
> remove cffi>=1.0´.  
> I don't want to have to care if it's in the HTTP cache or the wheel cache, or 
> how it got there;  
> I also don't want to have to bust a ~200 megabyte cache that saves me hours a 
> day just because  
> there's one bad entry in there.

Absent that one bug, I’m not aware of any reason why someone would need to ever 
purge their http cache except for if they were using a non PyPI index and they 
sent cache headers when they shouldn’t have (which is not pip’s fault in the 
slightest).

The wheel cache is new and still has some issues to work out. I suspect it will 
need some method to inspect and remove items from the cache, but that work 
won’t get done until someone does it.


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[David Cournapeau]

On Wed, Oct 14, 2015 at 11:23 PM, Chris Barker 
wrote:

> On Wed, Oct 14, 2015 at 1:59 PM, David Cournapeau 
> wrote:
>
>> But I'm talking about the cases of "whoops! I really wish I hadn't
>>> uploaded that one". We can improve the tooling (some discussion on this in
>>> this thread right now...), but people are people and some of us are stupid
>>> and/or careless. So this WILL happen.
>>>
>>
>
>> I have not done numpy releases for half a decade now, but it was already
>>> automated enough that putting a new version was not very costly then.
>>>
>>
> yeah, I suppose releases are cheap -- though part of the problem is that
> your users are likley to think that you actually fixed a bug or something.
> And maybe wonder why you went from 1.2.3 to 1.3.5 seemingly all at once...
>
> another note-- conda has teh concetp of a "build" that's tacked on teh
> release for conda pacakges.
>

Note that build is a feature for binaries: different builds refer to the
same upstream source. Most linux distributions have the notion of
downstream version which is a generalization of that.

David


> So if I updated somethign about how teh packge is buitl, but am using teh
> same underllying version of teh package, I update teh build number, get a
> new "version" of the package, but it's clear that the pacakge itself is the
> same version.
>
> for instance, I'm messing around right now with building libgd for conda,
> and the latest version I have up on anaconda.org is:
>
> libgd-2.1.1
>
> but the actual file is:
>
> libgd-2.1.1-1.tar.bz2
>
> (and there is a libgd-2.1.1-0.tar.bz2 there too...)
>
> Maybe this would be helpful for PyPi, too?
>
> I am sure there are things we can do to improve numpy's release process to
>> avoid this in the future.
>>
>
> numpy was just an example -- we are all likely to make mistakes in the
> future - it's human nature.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Wes Turner]

On Oct 14, 2015 12:05 PM, "Ionel Cristian Mărieș" 
wrote:
>
>
> On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker 
wrote:
>>
>> some packages were unable to work with the postN suffix.
>
>
> ​Can you elaborate a bit more here?​

AFAIU, the community best practice here is to issue a new release with a
bumped version number/pre/post/releasecode.

* https://www.python.org/dev/peps/pep-0440/#version-scheme
  * https://github.com/peritus/bumpversion
  *
http://docs.openstack.org/developer/pbr/semver.html#changes-vs-semver-2-0

MAJOR.MINOR.PATCH

The reasons for this are many:
* HTTP caches
* local pip/buildout pkg caches
* it is actually a different resource, with a different URI
(repo,pkg,version,[pkgtype,platform,buildtags]) because it contains
different files

>
>
>
> Thanks,
> -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 1:37:05 PM, Nathaniel Smith (n...@pobox.com) wrote:
On Oct 14, 2015 10:04 AM, "Ionel Cristian Mărieș"  wrote:
>
>
> On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker  wrote:
>>
>> some packages were unable to work with the postN suffix.
>
>
> ​Can you elaborate a bit more here?​
>

Apparently some packages were making assumptions about the format of the 
numpy.__version__ string, and having .postN in there caused errors when they 
tried to process it. (It would be helpful if there were a little permissively 
licensed standalone implementation of PEP 440 comparisons, suitable for the "if 
pkg.version > ...:" checks that people insist on doing -- I couldn't find one 
in some quick searches.)

https://github.com/pypa/packaging

It’s what both pip and setuptools use (though we embed it, but it’s fine to 
depend on it too).


IIUC, the specific problems numpy ran into that caused the creation of .postN 
releases were:
- oops, didn't sign the uploads, re-upload identical file with proper signature 
attached -> not allowed. (I'm not sure if these were embedded or detached 
signatures. Either way it'd be nice if pypi allowed it, but for embedded 
signatures in particular I can see how this might be a hassle.)

I don’t think we allow embedded signatures, it would be reasonable to allow 
uploading detached signatures after the fact though.


- our OS X maintainer tried to use twine to upload OS X wheels for the existing 
release; instead it created a new release. Not sure if a bug was filed on 
twine, but if not then one probably should be. As a workaround our release docs 
now say "always upload wheels by hand using the web interface, never use 
setup.py upload or twine".

This shouldn’t create a new release unless you’ve changed the version number 
(including adding post releases). If you can reproduce on Test PyPI I can fix 
it.


My feeling is that pypi is correct to disallow the mutation of releases once 
they become public, but that the ergonomics around this could probably be 
improved :-). A more general solution that might be nice to have Someday would 
be if you could upload a release in one step, and then get a private link to 
poke at what was uploaded and make sure it looks correct, before making it 
public in a second step.

The ergonomics of it is pretty bad. Once we kill PyPI legacy and Warehouse is 
the thing we’re using, improving things like that will be a lot simpler.


CC'ing the release manager and OS X maintainer in question, in case I got 
something wrong or more details are wanted...

-n

___ 
Distutils-SIG maillist - Distutils-SIG@python.org 
https://mail.python.org/mailman/listinfo/distutils-sig 



-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Donald Stufft]

On October 14, 2015 at 2:27:40 PM, Donald Stufft (don...@stufft.io) wrote:
> On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (n...@pobox.com) wrote:
> > On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote:
> > >
> > [...]
> > >> Apparently some packages were making assumptions about the format of the
> > numpy.__version__ string, and having .postN in there caused errors when
> > they tried to process it. (It would be helpful if there were a little
> > permissively licensed standalone implementation of PEP 440 comparisons,
> > suitable for the "if pkg.version > ...:" checks that people insist on doing
> > -- I couldn't find one in some quick searches.)
> > >
> > > https://github.com/pypa/packaging
> > >
> > > It’s what both pip and setuptools use (though we embed it, but it’s fine
> > to depend on it too).
> >
> > That's under Apache 2, so it can't be used by GPLv2 packages, or any
> > package that might be used by GPLv2 packages.
>  
> I suspect it’d be trivial to relicense it. There’s a total of 6 contributors 
> and I think  
> I know how to get ahold of all of them.

https://github.com/pypa/packaging/issues/42



-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Ben Finney]

Nathaniel Smith  writes:

> That's under Apache 2, so it can't be used by GPLv2 packages, or any
> package that might be used by GPLv2 packages.

I'm not sure what the claim is here, but it seems false to me.

Apache License 2.0 permits a derived work to be released under a
different license (even a non-free license). See Apache License 2.0 §4.

So a work BAR can be released under GNU GPL (e.g. “version 3 or later”)
that derives from work FOO released under Apache License 2.0.

The GNU GPL version 3 allows combination with other works with different
license conditions, provided those conditions don't impose further
restrictions on recipients. See GNU GPL v3 §7, §10, §12. (GPL v2 has
different wording with similar effect.)

Apache License 2.0 imposes no additional reastrictions on the recipient
of a work licensed under GNU GPL v3. So that meets the conditions of both.

FOO as a distinct work retains its license conditions, and all
recipients of FOO or BAR must comply with its conditions. The work BAR
as a whole carries the GNU GPL version (for example) 3 conditions, and
all recipients of BAR must comply with its conditions.

The two licenses are compatible, as far as I can see. The effective
result is a work BAR, licensed as free software under GPL version (e.g.)
3 or later.

Does that not match your understanding? Have I misunderstood your claim?

-- 
 \   “Corporation, n. An ingenious device for obtaining individual |
  `\   profit without individual responsibility.” —Ambrose Bierce, |
_o__)   _The Devil's Dictionary_, 1906 |
Ben Finney

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Nathaniel Smith]

On Oct 14, 2015 10:02 PM, "Ben Finney"  wrote:
>
> Nathaniel Smith  writes:
>
> > That's under Apache 2, so it can't be used by GPLv2 packages, or any
> > package that might be used by GPLv2 packages.
>
> I'm not sure what the claim is here, but it seems false to me.
>
> Apache License 2.0 permits a derived work to be released under a
> different license (even a non-free license). See Apache License 2.0 §4.

The FSF believes that Apache 2 is incompatible with GPLv2 (but v3 is OK),
IIRC because of some technicalities involving the patent provisions. (I'm
sure you can google up infinite detail if you want -- I'm just the
messenger.) There are others who disagree and think it's fine, but no one
wants murkiness when it comes to licenses, so people with BSD or GPLv2
projects are unlikely to risk incorporating code from a project under
Apache 2.

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig