Re: [Distutils] advice re: packaging tasks

[Donald Stufft]

> On Sep 30, 2014, at 7:44 PM, Chris Jerdonek  wrote:
> 
> Hi,
> 
> I was curious what others do for the following packaging tasks, or if
> you have any recommendations otherwise.  There is also a code
> organization question at the end.
> 
> 1) For starters, it's very easy to make mistakes in one's MANIFEST.in,
> so I hacked the sdist command in my setup.py to list the differences
> between one's project repo and the generated sdist each time you run
> "python setup.py sdist".  Are there better solutions for this out
> there so I don't have to rely on my own hack?

https://warehouse.python.org/project/check-manifest/

> 
> 2)  Secondly, like many, my README files are in markdown, so I hacked
> a command in my setup.py to use Pandoc to convert README.md to a .rst
> file for use as the long_description argument to setup().  I also
> check in the resulting file for troubleshooting purposes, etc.  Are
> there more elegant solutions for this that people know of?

Not that I’m aware of.

> 
> Also, for commands like the latter, is it better to define them in
> one's setup.py, or simply to have separate scripts in one's repo?

I’d probably do it in the setup.py, or as an invoke task, see
https://warehouse.python.org/project/invoke/.

> 
> Lastly, as these setup-related tasks grow larger and more complicated,
> I found it helped to break them out into a separate setup package that
> sits alongside my project's main package library (and even adding
> tests in some cases).  Is this normal?  Have other people run into
> this?

This is especially the case when I start using invoke.

> 
> Thanks,
> --Chris
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] advice re: packaging tasks

[Chris Jerdonek]

Hi,

I was curious what others do for the following packaging tasks, or if
you have any recommendations otherwise.  There is also a code
organization question at the end.

1) For starters, it's very easy to make mistakes in one's MANIFEST.in,
so I hacked the sdist command in my setup.py to list the differences
between one's project repo and the generated sdist each time you run
"python setup.py sdist".  Are there better solutions for this out
there so I don't have to rely on my own hack?

2)  Secondly, like many, my README files are in markdown, so I hacked
a command in my setup.py to use Pandoc to convert README.md to a .rst
file for use as the long_description argument to setup().  I also
check in the resulting file for troubleshooting purposes, etc.  Are
there more elegant solutions for this that people know of?

Also, for commands like the latter, is it better to define them in
one's setup.py, or simply to have separate scripts in one's repo?

Lastly, as these setup-related tasks grow larger and more complicated,
I found it helped to break them out into a separate setup package that
sits alongside my project's main package library (and even adding
tests in some cases).  Is this normal?  Have other people run into
this?

Thanks,
--Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[David Cournapeau]

On Tue, Sep 30, 2014 at 3:44 PM, Nick Coghlan  wrote:

> On 1 October 2014 00:37, Paul Moore  wrote:
> > On 30 September 2014 15:31, David Genest 
> wrote:
> >> Ok, so what if the dll is shared in a given environment (multiple
> extensions use it)?,  the shared dll should be copied to every package?
> Won't that cause multiple loads by the system?
> >
> > I honestly don't know in that case, sorry. You might get a better
> > answer on python-list for that, if no-one here can help.
> >
> > Presumably the usage is all within one distribution, otherwise the
> > question would have to be, which distribution ships the DLL? But that
> > question ends up leading onto the sort of discussion that starts
> > "well, I wouldn't design your system the way you have", which isn't
> > likely to be of much help to you :-(
> >
> > Sorry I can't offer any more help.
>
> Note that this is the external binary dependency problem that the
> scientific folks are currently using conda to address. It's basically
> the point where you cross the line from "language specific packaging
> system" to "multi-language cross-platform platform".
>

Conda is one such solution, not the solution ;)

I don't know any "sumo" distribution which solves this problem correctly
ATM, and windows makes this rather difficult to solve.

David


> That said, pip/wheel *may* get some capabilities along these lines in
> the future, it just isn't a high priority at this point.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[David Cournapeau]

On Tue, Sep 30, 2014 at 3:31 PM, David Genest 
wrote:

>
> > This is not true. Python loads DLLs with LOAD_WITH_ALTERED_SEARCH_PATH,
> to allow them to be located alongside the pyd file. You should therefore be
> able to ship the
> > dependent dll in the package directory (which wheels support fine).
>
> > Paul
>
> Ok, so what if the dll is shared in a given environment (multiple
> extensions use it)?,  the shared dll should be copied to every package?
> Won't that cause multiple loads by the system?
>

Yes it will, and it is indeed problematic. There are no great solutions:

 - bundle it in your package
 - have a separate wheel and then put it in PATH
 - have a separate wheel but use preload tricks (e.g. using ctypes) to
avoid using PATH

There are better solutions if one can patch python itself, though that's
obviously not practical in most cases.

David


> Thanks for your response,
>
> D.
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Steve Dower]

David Genest wrote:
> Subject: Re: [Distutils] Wheels and dependent third party dlls on windows
> 
> 
>> This is not true. Python loads DLLs with
>> LOAD_WITH_ALTERED_SEARCH_PATH, to allow them to be located alongside the pyd
> file. You should therefore be able to ship the dependent dll in the package
> directory (which wheels support fine).
> 
>> Paul
> 
> Ok, so what if the dll is shared in a given environment (multiple extensions 
> use
> it)?, the shared dll should be copied to every package? Won't that cause
> multiple loads by the system?

A DLL can only be loaded once per process (python.exe, in this case) and it 
will be loaded based on its file name (not the full path). Whoever loads first 
will win every future load for the same filename.

If you're loading it directly, it's fairly easy to rename a DLL to something 
likely to be unique to your project (or at least to put a version number in it) 
so that other packages won't use it. There are more complicated approaches 
using manifests and activation contexts (this is how different .pyd files with 
the same name can be correctly loaded), but ensuring a unique name is much 
easier.

If the DLL is loaded implicitly by a .pyd, then as Paul says it should be 
loaded correctly if it is alongside the .pyd.

Dependency Walker from www.dependencywalker.com is a great tool for checking 
what DLLs will be loaded by an executable or DLL. I recommend enabling 
profiling of your python.exe process when you try and import your packages to 
see where it is looking for its dependencies.

Hope that helps,
Steve

> Thanks for your response,
> 
> D.
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling Case/Normalization Differences

[Donald Stufft]

> On Sep 30, 2014, at 11:14 AM, Paul Moore  wrote:
> 
> On 30 September 2014 15:25, Paul Moore  wrote:
>> On 28 August 2014 19:58, Donald Stufft  wrote:
>>> To fix this I'm going to modify PyPI so that it uses the normalized name in
>>> the /simple/ URL and redirects everything else to the non-normalized name.
>>> I'm  also going to submit a PR to bandersnatch so that it will use 
>>> normalized
>>> names for it's directories and such as well. These two changes will make it 
>>> so
>>> that the client side will know ahead of time exactly what form the server 
>>> expects
>>> any given name to be in. This will allow a change in pip to happen which
>>> will pre-normalize all names which will make the interaction with mirrors
>>> better and will reduce the number of HTTP requests that a single ``pip 
>>> install``
>>> needs to make.
>> 
>> Just to clarify, this means that if I want to find the simple index
>> page for a distribution, without hitting redirects, I should first
>> normalise the project name (so "Django" becomes "django") and then
>> request https://pypi.python.org/simple// (with a
>> slash on the end). Is that correct? It seems to match what I see in
>> practice (in particular, the version without a terminating slash
>> redirects to the version with a terminating slash).
>> 
>> The JSON API has the opposite behaviour -
>> https://pypi.python.org/pypi/Django/json redirects to
>> https://pypi.python.org/pypi/django/json. Should that not be changed
>> to match? Will it be?
> 
> One further thought. Where is the definition of how to normalise a
> name? I could probably dig through the pip sources and find it, but it
> would be nice if it were documented somewhere. From experiment, it
> seems like lowercase, and with hyphens rather than underscores, is the
> definition. Does PyPI allow names not allowed by
> http://legacy.python.org/dev/peps/pep-0426/#name and if it does, how
> are they normalised?
> 
> In case it's not obvious, I'm writing a client for the PyPI API, and
> these questions are coming out of that process.
> 
> Paul.
> 
> PS The Python wiki has pages for the XMLRPC and JSON API. Any
> objections to me adding a page for the simple API? (The obvious
> objection being that it's documented somewhere else, and I should just
> put a pointer to the real documentation...)
> 
> Paul

PyPI follows PEP 426, I think we even include the confusables support.

Generally the normalization is done with pkg_resources.safe_name(…).lower().

I don’t think there’s any reason not to document it, setuptools has it’s
routine documented but that does’t have everything that the /simple/
API supports documented since it’s really documentation for what setuptools
does.

The URL redirect for the json endpoint was made to match what happens with
/pypi/django/. Lately I’ve been thinking that maybe we should just use the
normalized form in URLs always and use the author provided name for display
purposes.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Chris Barker]

On Tue, Sep 30, 2014 at 7:45 AM, Daniel Holth  wrote:

> Or you could just create a Python package that only contains the dll,
> and depend on it from your others.


but it won't be on the dll search path.

Paul Moore wrote:


> What you could do is distribute a package containing:
> 1. The dll
> 2. An __init__.py that adds the package directory (where the DLL is)
> to os.environ['PATH'].
> If you import this package before any of the ones that depend on the
> DLL (or even in the __init__ of those packages) then you should have
> PATH set up correctly, and things will work as you need.
> I think this works, although I will admit it feels like a bit of a hack to
> me.
> --


I'm not sure it will -- I tried somethign similar -- where I compiled some
code into an extension, hoping that importing that extension would make
that code available to other extensions -- works fine on OS-X and Linux,
but no dice on Windows.

So we build a dll of the code we need to share, and link all the extensions
that need it to it. In our case, all the extensions are part of the same
package, so we can put the dll in with the extensions and we're set.

It seems the "right" thing to do here is put the dll in with the dlls
provided by python (can't remember that path right now -- no Windows box
running) -- but I don't know that you can do that with wheel -- and it
would make it easy for different packages to stomp on each-other.

I actually think the thing to do here is either statically link it to each
extension that needs it, or deliver it with each of them, in the package
dir. Or, if you don't want duplicates, then use conda -- it's designed just
for this.

I'd also look at what Chris Gohlke does with his MSI installers:

http://www.lfd.uci.edu/~gohlke/pythonlibs/

maybe he's doing something you can do with MSI that you can't do with
wheels, but worth a look.

-Chris







Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Robert Collins]

On 1 October 2014 04:40, Wichert Akkerman  wrote:
>
>> On 30 Sep 2014, at 17:35, Barry Warsaw  wrote:
>>
>> On Sep 30, 2014, at 11:06 AM, M.-A. Lemburg wrote:
>>
>>> Installers and PyPI would then regard "3.1.4-1" as belonging to
>>> release "3.1.4", but being a more current build as a distribution
>>> file carrying "3.1.4" in its file name.
>>
>> Please don't literally use "3.1.4-1".  That will cause all kinds of havoc 
>> with
>> the Debian ecosystem.  There we use a dash to separate upstream version
>> numbers from Debian version numbers.  Thus 3.1.4-1 means the first Debian
>> upload of upstream's 3.1.4.  3.1.4-2 is the second, etc.
>
> Debian does allow 3.1.4-1-1. I forgot the exact rules, but I seem to remember 
> the package version is considered to start after the last dash. Debian will 
> also sort 3.1.4a after 3.1.4 unlike Python rules, so version “massaging” 
> might be necessary in other situations as well.

It's all in policy :)

PyPI 3.1.4a should be 3.1.4~a in Debian.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 04:25 PM, Jeremy Stanley wrote:

>Good to know. The Debian Developer packaging the majority of the
>projects I work on must be in that minority.

IIRC, the OpenStack packagers were probably the most prominent proponent of
release-from-tag.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Paul Moore]

On 30 September 2014 17:07, Steve Dower  wrote:
> The answer is basically no chance - the slippery slope was considered and 
> shut down.

Fair enough. Actually, it's good to know that this sort of thing was
thought through.

> If VC14 slips significantly and we have to stick with VC10 for Python 3.5, 
> then I'll make the case again and see what we get, but for now the future 
> story is to upgrade. Luckily, 3.3->3.5 is not going to be as hard as 2.7->3.5.

Agreed - and given that VC14 Express will include 32- and 64-bit
compilers, the whole SDK rigmarole is avoided which was the key pain
point (well, that and the fact that things went out of support, but
the forward compatibility of VC14 addresses that one too).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Jeremy Stanley]

On 2014-09-30 12:07:23 -0400 (-0400), Barry Warsaw wrote:
> We had a discussion about this at the recently concluded Debian
> conference. There are folks who only want to use git tags as the
> consumption point for Debian packages, but this opinion was not
> the majority opinion.

Good to know. The Debian Developer packaging the majority of the
projects I work on must be in that minority.

> There's no guarantee that what you get from a tagged upstream
> source revision will match what comes in the sdist tarball.
[...]

Indeed, we've implemented quite a few workarounds specifically
requested by distro packagers who want to be able to ignore our
tarballs and use their own tools/workflow to generate them without
ever even running sdist. It seems backwards to me, but I'm not the
one doing their packaging work.

> Thus, in the Debian Python team our policy is that if upstream produces
> tarballs (as is the case for the vast majority of our packages, which are
> sourced from PyPI), then we want the Debian package to use tarballs.
[...]

Refreshing to hear!
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

Paul Moore wrote:
> On 30 September 2014 16:56, Olivier Grisel  wrote:
>> What is the story for project maintainers who want to also support 
>> Python 3.3+ (for 32 bit and 64 bit python) for their project with 
>> binary wheels for windows?
>
> It would be so easy at this point to ask "What's the chance of a similarly 
> packaged
> version of VS2010 for Python 3.2/3.3/3.4?" But I really don't want Steve to 
> get into
> any trouble with people saying "now look what you've started" :-)

:-)

The answer is basically no chance - the slippery slope was considered and shut 
down.

If VC14 slips significantly and we have to stick with VC10 for Python 3.5, then 
I'll make the case again and see what we get, but for now the future story is 
to upgrade. Luckily, 3.3->3.5 is not going to be as hard as 2.7->3.5.

Cheers,
Steve

>
>Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 05:40 PM, Wichert Akkerman wrote:

>Debian does allow 3.1.4-1-1. I forgot the exact rules, but I seem to remember
>the package version is considered to start after the last dash. Debian will
>also sort 3.1.4a after 3.1.4 unlike Python rules, so version “massaging”
>might be necessary in other situations as well.

Yeah, but it's maddeningly confusing.  The havoc I mention is human based[*]
if not tools based. ;)

-Barry

[*] at least to *this* human.


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 02:34 PM, Jeremy Stanley wrote:

>I'm becoming less and less convinced it actually *is* a source
>distribution any more. My constant interaction with downstream Linux
>distro packagers shows a growing disinterest in consuming release
>"tarballs" of software, that they would generally prefer to pull
>releases directly from tags in the project's revision control
>systems instead.

This is not a universally held consensus.

We had a discussion about this at the recently concluded Debian conference.
There are folks who only want to use git tags as the consumption point for
Debian packages, but this opinion was not the majority opinion.  There's no
guarantee that what you get from a tagged upstream source revision will match
what comes in the sdist tarball.  Plus, the greater Debian ecosystem is firmly
camped in the tarball world, so even if you do checkout from a tag, you still
have to build a tarball for uploads, *and* you have to do it in a binary exact
copy reproducible way.

Thus, in the Debian Python team our policy is that if upstream produces
tarballs (as is the case for the vast majority of our packages, which are
sourced from PyPI), then we want the Debian package to use tarballs.  There
can be exceptions to the rule, but still today they are exceptions.

I don't think the tarball format is dead yet.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

Olivier Grisel wrote:
> Thank you very Steve for pushing that installer out, this is very appreciated.
> 
> What is the story for project maintainers who want to also support Python 3.3+
> (for 32 bit and 64 bit python) for their project with binary wheels for 
> windows?
> At the moment it's possible to use the Windows SDK as documented here:
> 
> http://scikit-learn.org/dev/install.html#building-on-windows
> 
> However getting VC Express + Windows SDK is hard and slow to setup and cannot 
> be
> scripted in a CI environment.

It can be, but there are a few tricks involved...
 
> In the mean time, it's possible to use CI environments that already feature 
> all
> the necessary versions of the VC compilers and libraries such as appveyor.com,
> see this demo project:
> 
> https://github.com/ogrisel/python-appveyor-demo
> https://ci.appveyor.com/project/ogrisel/python-appveyor-demo

This is the best way to have it set up - create a base VM image for your CI 
environment manually and clone it. I believe all the major cloud providers 
support this, though using a CI specialist like Appveyor makes it even easier.

As far as the future story, it will probably be "move to 3.5 on VC14 as soon as 
possible". Internally, I'll be pushing for a CI-compatible installer for our 
build tools, which I expect will actually get quite a bit of traction right now.

Unfortunately, going back in time to do it for both VC9 and VC10 was not an 
option. We chose VC9 because 2.7 is where people are stuck, while migrating 
from 3.3->3.5 should not be as big an issue.

Cheers,
Steve

> --
> Olivier
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Paul Moore]

On 30 September 2014 16:56, Olivier Grisel  wrote:
> What is the story for project maintainers who want to also support
> Python 3.3+ (for 32 bit and 64 bit python) for their project with
> binary wheels for windows?

It would be so easy at this point to ask "What's the chance of a
similarly packaged version of VS2010 for Python 3.2/3.3/3.4?" But I
really don't want Steve to get into any trouble with people saying
"now look what you've started" :-)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Olivier Grisel]

Thank you very Steve for pushing that installer out, this is very appreciated.

What is the story for project maintainers who want to also support
Python 3.3+ (for 32 bit and 64 bit python) for their project with
binary wheels for windows? At the moment it's possible to use the
Windows SDK as documented here:

  http://scikit-learn.org/dev/install.html#building-on-windows

However getting VC Express + Windows SDK is hard and slow to setup and
cannot be scripted in a CI environment.

In the mean time, it's possible to use CI environments that already
feature all the necessary versions of the VC compilers and libraries
such as appveyor.com, see this demo project:

https://github.com/ogrisel/python-appveyor-demo
https://ci.appveyor.com/project/ogrisel/python-appveyor-demo

-- 
Olivier
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Jeremy Stanley]

On 2014-10-01 00:41:38 +1000 (+1000), Nick Coghlan wrote:
[...]
> Which distro packagers? For Fedora, even if we pull from an
> upstream source control system, we'll still wrap it as a tarball
> inside an SRPM in order to feed it into the buld system.
[...]

Precisely that. Also lot of Debian, Ubuntu, SuSE, et cetera
packagers are following that same pattern. Even when there is an
upstream release tarball available, many prefer to create one from
the VCS themselves and use that as the basis for their packaging.

What I've seen suggests the increase in (not necessarily
Python-based) projects who don't bother to create tarballs and
simply "release" from their version control systems has resulted in
a proliferation of distro packager countermeasures/immune responses.
They're beginning to rely on tools which automate dealing with the
fact that there may be no initial tarball (Debian's git-buildpackage
for example), and in the end it becomes easier for them to just
assume there is never an initial tarball and always create their own
anyway. So while their packaging formats use tarballs internally, a
lot of them are no longer using *upstream-provided* tarballs in
source packages and the existence of tarballs in their source
packages has instead become a mere implementation detail.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Jeremy Stanley]

On 2014-09-30 09:22:29 -0600 (-0600), Carl Meyer wrote:
> On 09/30/2014 08:41 AM, Nick Coghlan wrote:
> > Why is your setup.py sdist including autogenerated content? It
> > shouldn't be doing that.
> 
> Don't almost all sdists? At the very least egg-info is
> autogenerated, MANIFEST usually is too.

Bingo. Also in some cases I see files autogenerated from VCS
metadata to avoid double-entry... things like change histories,
authors lists, documentation indices, and even version numbers.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Wichert Akkerman]

> On 30 Sep 2014, at 17:35, Barry Warsaw  wrote:
> 
> On Sep 30, 2014, at 11:06 AM, M.-A. Lemburg wrote:
> 
>> Installers and PyPI would then regard "3.1.4-1" as belonging to
>> release "3.1.4", but being a more current build as a distribution
>> file carrying "3.1.4" in its file name.
> 
> Please don't literally use "3.1.4-1".  That will cause all kinds of havoc with
> the Debian ecosystem.  There we use a dash to separate upstream version
> numbers from Debian version numbers.  Thus 3.1.4-1 means the first Debian
> upload of upstream's 3.1.4.  3.1.4-2 is the second, etc.

Debian does allow 3.1.4-1-1. I forgot the exact rules, but I seem to remember 
the package version is considered to start after the last dash. Debian will 
also sort 3.1.4a after 3.1.4 unlike Python rules, so version “massaging” might 
be necessary in other situations as well.

Wichert.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Barry Warsaw]

On Sep 30, 2014, at 11:06 AM, M.-A. Lemburg wrote:

>Installers and PyPI would then regard "3.1.4-1" as belonging to
>release "3.1.4", but being a more current build as a distribution
>file carrying "3.1.4" in its file name.

Please don't literally use "3.1.4-1".  That will cause all kinds of havoc with
the Debian ecosystem.  There we use a dash to separate upstream version
numbers from Debian version numbers.  Thus 3.1.4-1 means the first Debian
upload of upstream's 3.1.4.  3.1.4-2 is the second, etc.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Carl Meyer]

On 09/30/2014 08:41 AM, Nick Coghlan wrote:
> Why is your setup.py sdist including autogenerated content? It
> shouldn't be doing that.

Don't almost all sdists? At the very least egg-info is autogenerated,
MANIFEST usually is too.

Carl
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling Case/Normalization Differences

[Paul Moore]

On 30 September 2014 15:25, Paul Moore  wrote:
> On 28 August 2014 19:58, Donald Stufft  wrote:
>> To fix this I'm going to modify PyPI so that it uses the normalized name in
>> the /simple/ URL and redirects everything else to the non-normalized name.
>> I'm  also going to submit a PR to bandersnatch so that it will use normalized
>> names for it's directories and such as well. These two changes will make it 
>> so
>> that the client side will know ahead of time exactly what form the server 
>> expects
>> any given name to be in. This will allow a change in pip to happen which
>> will pre-normalize all names which will make the interaction with mirrors
>> better and will reduce the number of HTTP requests that a single ``pip 
>> install``
>> needs to make.
>
> Just to clarify, this means that if I want to find the simple index
> page for a distribution, without hitting redirects, I should first
> normalise the project name (so "Django" becomes "django") and then
> request https://pypi.python.org/simple// (with a
> slash on the end). Is that correct? It seems to match what I see in
> practice (in particular, the version without a terminating slash
> redirects to the version with a terminating slash).
>
> The JSON API has the opposite behaviour -
> https://pypi.python.org/pypi/Django/json redirects to
> https://pypi.python.org/pypi/django/json. Should that not be changed
> to match? Will it be?

One further thought. Where is the definition of how to normalise a
name? I could probably dig through the pip sources and find it, but it
would be nice if it were documented somewhere. From experiment, it
seems like lowercase, and with hyphens rather than underscores, is the
definition. Does PyPI allow names not allowed by
http://legacy.python.org/dev/peps/pep-0426/#name and if it does, how
are they normalised?

In case it's not obvious, I'm writing a client for the PyPI API, and
these questions are coming out of that process.

Paul.

PS The Python wiki has pages for the XMLRPC and JSON API. Any
objections to me adding a page for the simple API? (The obvious
objection being that it's documented somewhere else, and I should just
put a pointer to the real documentation...)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Joe Smith]

To add slightly to the bikeshed (sorry, since I'm not going to implement
this and have no horse in the race) you could also take a look at spinning
up a vm using Vagrant- it's worked out pretty well for Apache Aurora as a
testing environment
.
Although you have the high upfront-cost of downloading the image, it's
slightly offset by not requiring an internet connection for subsequent
testing.

On Mon, Sep 29, 2014 at 3:26 PM, Paul Moore  wrote:

> On 29 September 2014 23:16, Donald Stufft  wrote:
> >> It occurs to me that a devpi quickstart for OpenShift (or another
> PaaS's)
> >> free tier could be useful - if a devpi instance is just for pre-release
> >> testing of packages, then the free tier should accommodate it
> comfortably.
> >>
> >>
> >
> > I'm not familiar with with what is available on OpenShift, however this
> sounds
> > like something that would be great for Heroku's Deploy button[1].
> >
> > [1] https://devcenter.heroku.com/articles/heroku-button
>
> I've no experience with either of those services, but something like
> that sounds awesome! (I've toyed with setting up my own local devpi
> instance, but never got round to it...
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Robin Becker]

On 29/09/2014 10:50, Nick Coghlan wrote:

On 29 Sep 2014 19:04, "M.-A. Lemburg"  wrote:


Do you seriously want to force package authors to cut a new release
just because a single uploaded distribution file is broken for
some reason and then ask all users who have already installed one
of the non-broken ones to upgrade again, even though they are not
affected ?


Yes, I do. Silently changing released artefacts is actively user hostile.
It breaks mirroring, it breaks redistribution, it breaks security audits,
and it can even break installation for security conscious users that are
using peep rather than pip.








...
What would be the objection to removing or nulling a release package that had 
actual malware embedded in it some how. It seems reasonable to have some last 
resort take down mechanism.

--
Robin Becker
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Paul Moore]

On 30 September 2014 11:35, Robin Becker  wrote:
> What would be the objection to removing or nulling a release package that
> had actual malware embedded in it some how. It seems reasonable to have some
> last resort take down mechanism.

None at all. Removal is specifically still allowed.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Paul Moore]

On 30 September 2014 15:45, Daniel Holth  wrote:
> Or you could just create a Python package that only contains the dll,
> and depend on it from your others.

The problem is getting the DLL on PATH.

What you could do is distribute a package containing:

1. The dll
2. An __init__.py that adds the package directory (where the DLL is)
to os.environ['PATH'].

If you import this package before any of the ones that depend on the
DLL (or even in the __init__ of those packages) then you should have
PATH set up correctly, and things will work as you need.

I think this works, although I will admit it feels like a bit of a hack to me.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Daniel Holth]

Or you could just create a Python package that only contains the dll,
and depend on it from your others.

On Tue, Sep 30, 2014 at 10:44 AM, Nick Coghlan  wrote:
> On 1 October 2014 00:37, Paul Moore  wrote:
>> On 30 September 2014 15:31, David Genest  wrote:
>>> Ok, so what if the dll is shared in a given environment (multiple 
>>> extensions use it)?,  the shared dll should be copied to every package? 
>>> Won't that cause multiple loads by the system?
>>
>> I honestly don't know in that case, sorry. You might get a better
>> answer on python-list for that, if no-one here can help.
>>
>> Presumably the usage is all within one distribution, otherwise the
>> question would have to be, which distribution ships the DLL? But that
>> question ends up leading onto the sort of discussion that starts
>> "well, I wouldn't design your system the way you have", which isn't
>> likely to be of much help to you :-(
>>
>> Sorry I can't offer any more help.
>
> Note that this is the external binary dependency problem that the
> scientific folks are currently using conda to address. It's basically
> the point where you cross the line from "language specific packaging
> system" to "multi-language cross-platform platform".
>
> That said, pip/wheel *may* get some capabilities along these lines in
> the future, it just isn't a high priority at this point.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Nick Coghlan]

On 1 October 2014 00:37, Paul Moore  wrote:
> On 30 September 2014 15:31, David Genest  wrote:
>> Ok, so what if the dll is shared in a given environment (multiple extensions 
>> use it)?,  the shared dll should be copied to every package? Won't that 
>> cause multiple loads by the system?
>
> I honestly don't know in that case, sorry. You might get a better
> answer on python-list for that, if no-one here can help.
>
> Presumably the usage is all within one distribution, otherwise the
> question would have to be, which distribution ships the DLL? But that
> question ends up leading onto the sort of discussion that starts
> "well, I wouldn't design your system the way you have", which isn't
> likely to be of much help to you :-(
>
> Sorry I can't offer any more help.

Note that this is the external binary dependency problem that the
scientific folks are currently using conda to address. It's basically
the point where you cross the line from "language specific packaging
system" to "multi-language cross-platform platform".

That said, pip/wheel *may* get some capabilities along these lines in
the future, it just isn't a high priority at this point.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Jeremy Stanley]

On 2014-09-30 11:06:40 +0200 (+0200), M.-A. Lemburg wrote:
[...]
> You're regularly bringing up this argument.
> 
> Let's just be fair here: external hosting of packages has been
> made so user unfriendly in recent pip releases, that this has
> pretty much become a non-option for anyone who wants to create a
> user friendly package installation environment.
[...]

And I'm seeing this argument regularly brought up as well. As a
heavy user of Python packages and someone who maintains very large
systems depending on them, I had a hard time trusting pip back when
it still would spontaneously grab software from wherever the
upstream author had decided to stick it that day (with whatever
hosting stability issues that implied). Our projects would regularly
audit our hundreds of requirements just to make sure nobody
*accidentally* added one which was hosted off PyPI, and that our
dependencies hadn't suddenly decided to start sticking new releases
off PyPI.

The suggestion that some developers want to control their release
process *so* tightly that they host their software in random places
without uploading them to the community package system or quietly
replace broken releases with new packages using the same version
numbers is a non-argument as far as I'm concerned. The software
authors I've talked to in these cases are pretty much always easily
convinced that those are troublesome behaviors (once it's pointed
out) and readily adjust their release processes to a more
user-friendly end result.

If there are a few who are so completely insistent on continuing in
this manner the projects I work on will not use them (for our own
sanity), and if pip and PyPI implement assurances against these
which have a side effect of driving *those particular* development
teams off of the community packaging channel then that can only be a
positive net effect in my opinion.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Nick Coghlan]

On 1 October 2014 00:34, Jeremy Stanley  wrote:
> On 2014-09-30 07:26:32 -0400 (-0400), Donald Stufft wrote:
> [...]
>> I don’t personally believe it makes sense for a source
>> distribution to have a build number.
> [...]
>
> I'm becoming less and less convinced it actually *is* a source
> distribution any more. My constant interaction with downstream Linux
> distro packagers shows a growing disinterest in consuming release
> "tarballs" of software, that they would generally prefer to pull
> releases directly from tags in the project's revision control
> systems instead.

Which distro packagers? For Fedora, even if we pull from an upstream
source control system, we'll still wrap it as a tarball inside an SRPM
in order to feed it into the buld system.

> Couple this with the fact that setup.py sdist can
> (and often does) include autogenerated content in its output which
> the packagers would rather strip or regenerate themselves, and I'm
> of the opinion that the tarballs I create are only for PyPI/pip
> consumption any longer. This effectively makes them a
> channel-specific packaging format rather than a generally reusable
> release source artifact.

Why is your setup.py sdist including autogenerated content? It
shouldn't be doing that.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Paul Moore]

On 30 September 2014 15:31, David Genest  wrote:
> Ok, so what if the dll is shared in a given environment (multiple extensions 
> use it)?,  the shared dll should be copied to every package? Won't that cause 
> multiple loads by the system?

I honestly don't know in that case, sorry. You might get a better
answer on python-list for that, if no-one here can help.

Presumably the usage is all within one distribution, otherwise the
question would have to be, which distribution ships the DLL? But that
question ends up leading onto the sort of discussion that starts
"well, I wouldn't design your system the way you have", which isn't
likely to be of much help to you :-(

Sorry I can't offer any more help.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Jeremy Stanley]

On 2014-09-30 07:26:32 -0400 (-0400), Donald Stufft wrote:
[...]
> I don’t personally believe it makes sense for a source
> distribution to have a build number.
[...]

I'm becoming less and less convinced it actually *is* a source
distribution any more. My constant interaction with downstream Linux
distro packagers shows a growing disinterest in consuming release
"tarballs" of software, that they would generally prefer to pull
releases directly from tags in the project's revision control
systems instead. Couple this with the fact that setup.py sdist can
(and often does) include autogenerated content in its output which
the packagers would rather strip or regenerate themselves, and I'm
of the opinion that the tarballs I create are only for PyPI/pip
consumption any longer. This effectively makes them a
channel-specific packaging format rather than a generally reusable
release source artifact.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[David Genest]

> This is not true. Python loads DLLs with LOAD_WITH_ALTERED_SEARCH_PATH, to 
> allow them to be located alongside the pyd file. You should therefore be able 
> to ship the 
> dependent dll in the package directory (which wheels support fine).

> Paul

Ok, so what if the dll is shared in a given environment (multiple extensions 
use it)?,  the shared dll should be copied to every package? Won't that cause 
multiple loads by the system?

Thanks for your response,

D.



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling Case/Normalization Differences

[Paul Moore]

On 28 August 2014 19:58, Donald Stufft  wrote:
> To fix this I'm going to modify PyPI so that it uses the normalized name in
> the /simple/ URL and redirects everything else to the non-normalized name.
> I'm  also going to submit a PR to bandersnatch so that it will use normalized
> names for it's directories and such as well. These two changes will make it so
> that the client side will know ahead of time exactly what form the server 
> expects
> any given name to be in. This will allow a change in pip to happen which
> will pre-normalize all names which will make the interaction with mirrors
> better and will reduce the number of HTTP requests that a single ``pip 
> install``
> needs to make.

Just to clarify, this means that if I want to find the simple index
page for a distribution, without hitting redirects, I should first
normalise the project name (so "Django" becomes "django") and then
request https://pypi.python.org/simple// (with a
slash on the end). Is that correct? It seems to match what I see in
practice (in particular, the version without a terminating slash
redirects to the version with a terminating slash).

The JSON API has the opposite behaviour -
https://pypi.python.org/pypi/Django/json redirects to
https://pypi.python.org/pypi/django/json. Should that not be changed
to match? Will it be?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Paul Moore]

On 30 September 2014 14:32, David Genest  wrote:
> But the only way to get a dependent dll found on windows is to have it on 
> PATH, and the scripts directory on
> windows is on path when a virtualenv is activated.

This is not true. Python loads DLLs with
LOAD_WITH_ALTERED_SEARCH_PATH, to allow them to be located alongside
the pyd file. You should therefore be able to ship the dependent dll
in the package directory (which wheels support fine).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Wheels and dependent third party dlls on windows

[David Genest]

Hi, 

I was wondering what is the recommended approach to bundling runtime dll 
dependencies when using wheels. 

We are migrating from egg to wheels for environment installation and of various 
python dependencies. 
Some of those have extension modules, and some have extension modules that 
depend on the presence 
of a third party dll (in our situation, libzmq-v100-mt-4_0_3.dll).

Up to now, these dlls have been installed by the use of the scripts parameter 
in the setup command of setup.py, but 
https://mail.python.org/pipermail/distutils-sig/2014-July/024554.html

points to it as not being a good idea.

But the only way to get a dependent dll found on windows is to have it on PATH, 
and the scripts directory on 
windows is on path when a virtualenv is activated.

I have observed two situations:

1) If we use pip wheel to build the wheel, the scripts parameter is ignored and 
the dlls do not even get to the archive.
2) If we use setup.py bdist_wheel, the dll gets into the archive, but this 
relies on the non-documented feature of packaging scripts-as-data of dlls.

What is the correct approach at this time ?

Thanks,

David

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Nick Coghlan]

On 30 September 2014 21:30, Donald Stufft  wrote:
> In pip 1.5.x (current latest release) people can add additional indexes (or
> replace PyPI all together) on a per user basis. In pip 6.0 (next release) that
> still exists, and in addition it also includes a global configuration file
> for the entire machine and a per virtual env configuration file.
>
> About the only thing it doesn't support is a directory of files ala
> apt.sources.d/*.list.

Donald & I chatted about this off-list, and the base URL is currently
the only configuration setting pip retains per index. While we may
still want to explore named indices at some point in the future, at
this point in time, they wouldn't add a lot over just using the index
URL directly.

So I think this is basically already covered on the pip side (once 6.0
goes out), and then PEP 470 aims to improve the user experience for
external index discovery.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Donald Stufft]

On September 30, 2014 at 5:19:50 AM, Nick Coghlan (ncogh...@gmail.com) wrote:
> On 30 Sep 2014 19:06, "M.-A. Lemburg" wrote:
> > You're regularly bringing up this argument.
> >
> > Let's just be fair here: external hosting of packages has been made so
> > user unfriendly in recent pip releases, that this has pretty much
> > become a non-option for anyone who wants to create a user friendly
> > package installation environment.
> >
> > pip is unfortunately using the same kind of
> > --no-one-will-want-to-use-this-option-because-its-too-long
> > approach as setuptools/easy_install has done in the past to force
> > people into installing packages as eggs rather than installing
> > the packages in the standard write to site-packages dir way.
> >
> > The end result is the same: users will not want to go
> > through those extra hoops and thus packages not hosted on PyPI
> > itself will be regarded as broken (because they don't install using
> > the standard method; not because they are really broken).
> 
> I personally think pip needs a virtualenv friendly equivalent to
> yum.repos.d or conda channels to improve the multi-index experience.
> 
> This is a solved problem from my perspective, we just need folks willing to
> spend the time making the case for the proven solution, and adapting it to
> the PyPI ecosystem.

In pip 1.5.x (current latest release) people can add additional indexes (or
replace PyPI all together) on a per user basis. In pip 6.0 (next release) that
still exists, and in addition it also includes a global configuration file
for the entire machine and a per virtual env configuration file.

About the only thing it doesn't support is a directory of files ala
apt.sources.d/*.list.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Donald Stufft]

On September 30, 2014 at 5:07:06 AM, M.-A. Lemburg (m...@egenix.com) wrote:
> Thanks for the confirmation that my interpretation was wrong. This
> makes things a lot better :-)
>  
> More below...
>  
> On 29.09.2014 11:39, Nick Coghlan wrote:
> > On 29 Sep 2014 18:49, "M.-A. Lemburg" wrote:
> >>
> >> You are missing out on cases, where the release process causes files to
> >> be omitted, human errors where packagers forget to apply changes to
> >> e.g. documentation files, version files, change logs, etc., where
> >> packagers want to add information that doesn't affect the software
> >> itself, but meta information included in the distribution files.
> >>
> >> Such changes often do not affect the software itself, and so are not
> >> detected by software tests.
> >
> > Fixing such packaging errors is the primary intended use case of the "post"
> > field in PEP 440. Alternatively, many projects will just spin a new release
> > that addresses those issues, just as they would for any other bug.
> >
> > Both of those approaches have the advantage of letting users (especially
> > those operating mirrors, or downloading tarballs and feeding them into a
> > separate redistribution system) easily tell whether or not they have the
> > fixed version.
>  
> I don't see how that would help. AFAIU, PyPI would regard a "3.1.4.post1"
> as new release and so invalidate all other already uploaded distribution
> files rather than just regard the fixed upload as update
> to the "3.1.4" release.
>  
> If we could get a widely adopted notion of build numbers into the
> tools that would help a lot.
>  
> Installers and PyPI would then regard "3.1.4-1" as belonging to
> release "3.1.4", but being a more current build as a distribution
> file carrying "3.1.4" in its file name.
>  
> This would also have to work for all files uploaded for a release,
> not only for some distribution formats, of course, including source
> release files, Windows installers, egg files, etc.
>  
> I'd have to run some experiments, but don't believe that setuptools
> and associated tools support this at the moment.

I don’t personally believe it makes sense for a source distribution to have
a build number. Yes it would invalidate all other uploads because it should,
They take the source distribution as an input, if you change the input then
in all likelihood you change the output. This won’t be true in every case but
to be able to determine when and where it won’t be true requires intimate
knowledge of the package formats as well as the project in question.

On the other hand Wheel files already support the concept of build numbers,
I don’t see any technical reason why Windows installers can’t support them
given that nothing automated pulls them down. Getting Egg files to support
them may be a lost cause though since I’m not sure there’s a way to add that
information to the Egg spec without mandating a new setuptools.

>  
> >> If I understand you correctly, you are essentially suggesting that it
> >> becomes impossible to ever delete anything uploaded to PyPI, i.e.
> >> turning PyPI into a WORM.
> >
> > No, deletion remains supported. The only capability being removed is silent
> > substitution of hosted files with different ones bearing the same name.
> >
> > So if, for example, release "3.1.4" had a packaging error, then deleting it
> > would still be possible, but the replacement would need to be something
> > like "3.1.4.post1" or "3.1.5", rather than being permitted to reuse the
> > "3.1.4" name.
>  
> So just to summarize: the proposal is to turn PyPI into a WORM,
> but at least it's still possible to remove distribution files.
>  
> > The external hosting support is then the mechanism by which authors can
> > retain complete and total control over their release process. That approach
> > avoids all licensing concerns (including those related to US export
> > controls), as well as ensuring they have the ability to silently change the
> > contents of previously released files if they choose to do so (although, as
> > noted above, actually doing so may break installation for anyone installing
> > with peep, which checks file hashes based on the first version downloaded).
>  
> You're regularly bringing up this argument.
>  
> Let's just be fair here: external hosting of packages has been made so
> user unfriendly in recent pip releases, that this has pretty much
> become a non-option for anyone who wants to create a user friendly
> package installation environment.
>  
> pip is unfortunately using the same kind of
> --no-one-will-want-to-use-this-option-because-its-too-long
> approach as setuptools/easy_install has done in the past to force
> people into installing packages as eggs rather than installing
> the packages in the standard write to site-packages dir way.
>  
> The end result is the same: users will not want to go
> through those extra hoops and thus packages not hosted on PyPI
> itself will be regarded as broken (because they don't ins

Re: [Distutils] Immutable Files on PyPI

[Paul Moore]

On 30 September 2014 10:06, M.-A. Lemburg  wrote:
> Let's just be fair here: external hosting of packages has been made so
> user unfriendly in recent pip releases, that this has pretty much
> become a non-option for anyone who wants to create a user friendly
> package installation environment.

This is a fair point. But it is an acknowledged problem in pip and
we're working on resolving it. See
http://legacy.python.org/dev/peps/pep-0470/. The problem (as with
anything else) is that someone needs to write the code, and that
hasn't happened yet. We'll get to it, but the usual "PRs welcome"
comment applies here :-)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[Nick Coghlan]

On 30 Sep 2014 19:06, "M.-A. Lemburg"  wrote:
> You're regularly bringing up this argument.
>
> Let's just be fair here: external hosting of packages has been made so
> user unfriendly in recent pip releases, that this has pretty much
> become a non-option for anyone who wants to create a user friendly
> package installation environment.
>
> pip is unfortunately using the same kind of
> --no-one-will-want-to-use-this-option-because-its-too-long
> approach as setuptools/easy_install has done in the past to force
> people into installing packages as eggs rather than installing
> the packages in the standard write to site-packages dir way.
>
> The end result is the same: users will not want to go
> through those extra hoops and thus packages not hosted on PyPI
> itself will be regarded as broken (because they don't install using
> the standard method; not because they are really broken).

I personally think pip needs a virtualenv friendly equivalent to
yum.repos.d or conda channels to improve the multi-index experience.

This is a solved problem from my perspective, we just need folks willing to
spend the time making the case for the proven solution, and adapting it to
the PyPI ecosystem.

> This is what I'm trying to address in discussions like these all
> along. PyPI has a responsibility not only for the part consuming
> part of the Python community, but also for the creating part of it.
>
> While PyPI is great for indexing packages, it's not necessarily
> the best answer for hosting the distribution files and I believe
> we should open up some more to allow for making it possible to
> offer the same kind of user experience while not making pypi.python.org
> the only source of distribution files.
>
> In the Linux world, this already works great by having multiple
> repos which you can switch on/off easily.

And I've said all along that's the experience I would like to see in
upstream Python as well. But the code isn't going to write itself, and
there's also the political work needed to persuade folks that it's the
right path to go down.

> For eGenix, we've found a way to work around the PyPI constraints:
>
>
http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/
>
> which addresses our user's problems. Still, we'd much rather use
> standard ways of working *with* PyPI rather than work around it.

Sure, I'd like to see that to. It's just only one problem amongst a great
many of them, and people's upstream work is driven by their own balancing
of a complex set of priorities.

Cheers,
Nick.

>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Sep 30 2014)
> >>> Python Projects, Consulting and Support ...   http://www.egenix.com/
> >>> mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
> 
> 2014-09-30: Python Meeting Duesseldorf ... today
>
> : Try our mxODBC.Connect Python Database Interface for free ! ::
>
>eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>Registered at Amtsgericht Duesseldorf: HRB 46611
>http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

[M.-A. Lemburg]

Thanks for the confirmation that my interpretation was wrong. This
makes things a lot better :-)

More below...

On 29.09.2014 11:39, Nick Coghlan wrote:
> On 29 Sep 2014 18:49, "M.-A. Lemburg"  wrote:
>>
>> You are missing out on cases, where the release process causes files to
>> be omitted, human errors where packagers forget to apply changes to
>> e.g. documentation files, version files, change logs, etc., where
>> packagers want to add information that doesn't affect the software
>> itself, but meta information included in the distribution files.
>>
>> Such changes often do not affect the software itself, and so are not
>> detected by software tests.
> 
> Fixing such packaging errors is the primary intended use case of the "post"
> field in PEP 440. Alternatively, many projects will just spin a new release
> that addresses those issues, just as they would for any other bug.
> 
> Both of those approaches have the advantage of letting users (especially
> those operating mirrors, or downloading tarballs and feeding them into a
> separate redistribution system) easily tell whether or not they have the
> fixed version.

I don't see how that would help. AFAIU, PyPI would regard a "3.1.4.post1"
as new release and so invalidate all other already uploaded distribution
files rather than just regard the fixed upload as update
to the "3.1.4" release.

If we could get a widely adopted notion of build numbers into the
tools that would help a lot.

Installers and PyPI would then regard "3.1.4-1" as belonging to
release "3.1.4", but being a more current build as a distribution
file carrying "3.1.4" in its file name.

This would also have to work for all files uploaded for a release,
not only for some distribution formats, of course, including source
release files, Windows installers, egg files, etc.

I'd have to run some experiments, but don't believe that setuptools
and associated tools support this at the moment.

>> If I understand you correctly, you are essentially suggesting that it
>> becomes impossible to ever delete anything uploaded to PyPI, i.e.
>> turning PyPI into a WORM.
> 
> No, deletion remains supported. The only capability being removed is silent
> substitution of hosted files with different ones bearing the same name.
> 
> So if, for example, release "3.1.4" had a packaging error, then deleting it
> would still be possible, but the replacement would need to be something
> like "3.1.4.post1" or "3.1.5", rather than being permitted to reuse the
> "3.1.4" name.

So just to summarize: the proposal is to turn PyPI into a WORM,
but at least it's still possible to remove distribution files.

> The external hosting support is then the mechanism by which authors can
> retain complete and total control over their release process. That approach
> avoids all licensing concerns (including those related to US export
> controls), as well as ensuring they have the ability to silently change the
> contents of previously released files if they choose to do so (although, as
> noted above, actually doing so may break installation for anyone installing
> with peep, which checks file hashes based on the first version downloaded).

You're regularly bringing up this argument.

Let's just be fair here: external hosting of packages has been made so
user unfriendly in recent pip releases, that this has pretty much
become a non-option for anyone who wants to create a user friendly
package installation environment.

pip is unfortunately using the same kind of
--no-one-will-want-to-use-this-option-because-its-too-long
approach as setuptools/easy_install has done in the past to force
people into installing packages as eggs rather than installing
the packages in the standard write to site-packages dir way.

The end result is the same: users will not want to go
through those extra hoops and thus packages not hosted on PyPI
itself will be regarded as broken (because they don't install using
the standard method; not because they are really broken).

This is what I'm trying to address in discussions like these all
along. PyPI has a responsibility not only for the part consuming
part of the Python community, but also for the creating part of it.

While PyPI is great for indexing packages, it's not necessarily
the best answer for hosting the distribution files and I believe
we should open up some more to allow for making it possible to
offer the same kind of user experience while not making pypi.python.org
the only source of distribution files.

In the Linux world, this already works great by having multiple
repos which you can switch on/off easily.

For eGenix, we've found a way to work around the PyPI constraints:

http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/

which addresses our user's problems. Still, we'd much rather use
standard ways of working *with* PyPI rather than work around it.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 30 2014)
>>> P