from:"M.\-A. Lemburg"

On 10.02.2016 14:00, Oscar Benjamin wrote:
> On 10 February 2016 at 12:21, M.-A. Lemburg <m...@egenix.com> wrote:
>>> So "easy to achieve" still needs someone to take the time to deal with
>>> these sorts of issue. It's the usual process of the people willing to
>>> put in the effort get to choose the direction (which is also why I
>>> just provide feedback, and don't tend to offer my own proposals,
>>> because I'm not able to commit that sort of time).
>>
>> Wait. You are missing the point that the setup.py interface
>> already does work, so no extra effort is needed. All that's
>> needed is some documentation of what's currently being used,
>> so that other tools can support the interface going forward.
> 
> You can see an example of a minimal setup.py file here:
> 
> https://github.com/oscarbenjamin/setuppytest/blob/master/setuppytest/setup.py
> 
> I wrote that some time ago and don't know if it still works (that's
> the problem with just having a de facto standard).

Agreed, and something that we should address in a PEP.

>> At the moment, pip this interface is only defined by
>> "what pip uses" and that's a moving target.
> 
> The setup.py interface is a terrible interface for tools like pip to
> use and for tools like flit to emulate. 

I'm not saying that it's a great interface, but it's one that
by far most sdists out there support.

> Currently what pip does is to
> invoke
> 
> $ python setup.py egg_info --egg-base $TEMPDIR
> 
> to get the metadata. It is not possible to get the metadata without
> executing the setup.py which is problematic for many applications.
> Providing a static pypa.json file is much better: tools can read a
> static file to get the metadata.

Depends on which kind of meta data you're after. sdist packages
do include the static PKG-INFO file which has the version 1.0
meta data. This doesn't include dependencies or namespace
details, but it does have important data such as version,
package name, description, etc.

> To install a distribution pip runs:
> 
> $ python setup.py install --record $RECORD_FILE \
> --single-version-externally-managed
> 
> So the setup.py is entirely responsible not just for building but also
> for installing everything. This makes it very difficult to develop a
> system where different installer tools and different build tools can
> cooperate to allow end users to specify installation options. It also
> means that the installer has no direct control over where any of the
> files are installed.

Why is that ? The install command is very flexible in allowing
you to define where the various parts are installed.

When defining a minimal set of supported options, the
various --install-* options should be part of this.

It would also be possible to separate the build and install
steps, since distutils is well capable to do this.

However, I'm not sure where this aspect fits in relation to the
proposed PEP, since it is targeting the operation of building
the package and wrapping it into a wheel file, so the
bdist_wheel command would have to be used instead.

pip wheel pkg runs this command:

python setup.py bdist_wheel -d targetdir

> If you were designing this from scratch then there are some obvious
> things that you would want to do differently here. The setup.py
> interface also has so much legacy usage that it's difficult for
> setuptools and pip to evolve. The idea with this proposal is to
> decouple things by introducing a new interface with well defined and
> sensible behaviour.

In the end, you'll just be defining a different standard
to express the same thing in different ways.

The setup.py interface was never designed with integration in mind
(only with the idea to provide an extensible interface; and I'm not
going get into the merits of setuptools additions such as
--single-version-externally-managed :-)) but it's still
quite usable for the intended purpose.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

> Paul Moore  writes:
> 
>> But as I said in my response to Nathaniel, it may be that all that is
>> needed is some context in the PEP explaining how we require[1] people
>> to upload source to PyPI in the new world where we support build
>> systems which don't have a "sdist" command like setuptools does.
>>
>> Paul
>>
>> [1] I say "require" in the sense of "you have to follow these rules if
>> pip is to be able to use your source", not "you must upload source" -
>> although I hope that the number of people actually preferring to *not*
>> include source in their PyPI uploads is vanishingly small...

I'm not sure I'm parsing your comment correctly, but if you are
suggesting that PyPI should no longer allow supporting
non-open-source packages, this is definitely not going to
happen.

Python is free for everyone to use without any GPL-like
restrictions, which is part of our big success, and our packaging
environment has to follow the same principle.

The attitude that some people in this discussion are showing
does not align with those principles, which I find increasingly
worrying. When discussing technicalities in this space, you always
have to take the political implications into account as well.

Back on topic:

I don't think that the build system abstraction is
moving in the right direction. Instead of coming up with
yet another standard for build interfacing, we should simply
pin down the commands and options that pip and other installers
will want to see working with the standard setup.py command
line interface we have.

There aren't all that many - simply take what pip does now
as minimal standard.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

On 10.02.2016 11:08, Paul Moore wrote:
> On 10 February 2016 at 09:34, M.-A. Lemburg <m...@egenix.com> wrote:
>> I'm not sure I'm parsing your comment correctly, but if you are
>> suggesting that PyPI should no longer allow supporting
>> non-open-source packages, this is definitely not going to
>> happen.
> 
> Not at all.

That's good to know :-)

> Although as far as I know the number of closed-source packages
> on PyPI is vanishingly small...
>
> My concern is that we seem to be opening up the option of using
> non-setuptools build systems without having a good solution for people
> *wishing* to upload sources. It's more a matter of timing - if we
> allow people to use (say) flit for their builds then presumably a
> proportion of people will, because it's easier to use than setuptools,
> *for builds*. But those people will then find that distributing their
> sources isn't something that flit covers, so they'll make up their own
> approach (if it were me, I'd probably just point people at the
> project's github account).
> 
> Once people get set up with a workflow that goes like this (build
> wheels and point people to github for source) it'll be harder to
> encourage them later to switch *back* to a process of uploading
> sources to PyPI.
> 
> And that I do think is bad - that we end up pushing people who would
> otherwise happily use PyPI for source and binary hosting, to end up
> with a solution where they host binaries only on PyPI and make the
> source available via another (non-standardised) means.

That's a fair argument indeed.

> In no way though am I proposing that we stop people making deliberate
> choices on how they distribute their packages. Just that we make
> hosting both source and binaries on PyPI the "no friction" easy option
> for (the majority of?) people who don't really mind, and just want to
> make their work publicly available.

Well, you know, there's an important difference between making
work publicly available and giving away the source code :-)

But I get your point and do support it: It should be possible
for package authors to use different build tools than distutils.

IMO, that's easy to achieve, though, with the existing de-facto
standard interface we already have: the setup.py command line API.
We'd just need to publish the minimal set of commands and options,
installer will want to see implemented in order to initiate
the builds.

> PS This has gone a long way off the topic of the build interface
> proposal, so I'm glad it's been spun off into its own thread. I'm now
> of the view that this relates at best peripherally to the build
> interface proposal, which I'll comment on in the other thread. 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

On 10.02.2016 12:10, Paul Moore wrote:
> On 10 February 2016 at 10:23, M.-A. Lemburg <m...@egenix.com> wrote:
>> IMO, that's easy to achieve, though, with the existing de-facto
>> standard interface we already have: the setup.py command line API.
>> We'd just need to publish the minimal set of commands and options,
>> installer will want to see implemented in order to initiate
>> the builds.
> 
> No-one who's investing time in writing PEPs is willing to thrash out
> the details of how to use the setup.py interface in a formal proposal
> that sticks to the sort of "minimum required" spec that alternative
> tools would be willing to support. And there's no indication that tool
> developers are willing to implement a setup.py compatible interface
> format as you suggest. And finally, you'd need a way to declare that
> pip installs tool X before trying to run setup.py.

I don't think that installing 3rd party tools is within the scope
of such a proposal. The setup.py of packages using such tools would
have to either define a dependency to have the installer get the
extra tool, download and install it directly when needed, or tell
the user how to install the tool.

Alternatively, the package distro could simply ship the tool
embedded in the package. That's what we're doing with
mxSetup.py.

> So "easy to achieve" still needs someone to take the time to deal with
> these sorts of issue. It's the usual process of the people willing to
> put in the effort get to choose the direction (which is also why I
> just provide feedback, and don't tend to offer my own proposals,
> because I'm not able to commit that sort of time).

Wait. You are missing the point that the setup.py interface
already does work, so no extra effort is needed. All that's
needed is some documentation of what's currently being used,
so that other tools can support the interface going forward.

At the moment, pip this interface is only defined by
"what pip uses" and that's a moving target.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 10 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

2016-01-23 Thread M.-A. Lemburg

On 23.01.2016 04:26, Nick Coghlan wrote:
> On 22 January 2016 at 22:07, M.-A. Lemburg <m...@egenix.com> wrote:
>> However, system vendors will often be a lot faster with updates
>> than package authors, simply because it's their business model,
>> so as user you will want to benefit from those updates and
>> not have to rely on the package author to ship new wheel files.
> 
> This is true for the subset of packages monitored by distro security
> response teams, but there's a *lot* of software not currently packaged
> for Linux distros that never will be as more attention is given to the
> "rebuild the world on demand" model that elastic cloud computing and
> fast internet connections enable.
> 
> My fundamental concern is that if a package author publishes a distro
> dependent wheel file, pip attempts to install it, and it doesn't work,
> the reaction for many users is going to be "Python packaging is
> broken", not "the packaging of this particular package is broken".

I think helping the user to identify where the problem
originates is certainly possible.

This can be done in a generic way by e.g. having pip or the wheel
package scan the wheel file for shared libraries using ldd and
finding potentially missing libs.

Or we could define a special package post install entry point which
pip calls to have the wheel itself check the system it was installed
on for missing system packages or any other post install actions
that need to take place before the wheel file can be used, e.g.
setting up initial config files.

> However, moving the "generic linux wheels are ignored by default"
> behaviour to pip-the-client, rather than enforcing it as a restriction
> on PyPI uploads could definitely be a reasonable alternative way of
> addressing that concern.

I don't think that's the right strategy. There are certainly ways
to improve error reporting for Python packaging (see above), but
outright rejecting generic wheels is not a good approach, IMO.

The wheel system is not yet complete, but until it is, using
a "we want to protect the user from failing wheels" approach
is not going to help much, since we're just replacing this
with a "we'll let the user handle failing source installations"
approach instead - with the main reason apparently being that
we want to avoid putting the user blame for failures on PyPI,
pip or wheels.

This ignores that fact that generic wheels have a much
better chance of success than source installations of the same
package on the same system (it's rather unlikely that the user
will have the foo-dev package installed with the corresponding
foo binary package).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 23 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

2016-01-22 Thread M.-A. Lemburg

On 22.01.2016 11:03, Nathaniel Smith wrote:
> On Fri, Jan 22, 2016 at 1:33 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>> On 21.01.2016 20:05, Matthew Brett wrote:
>>> Hi,
>>>
>>> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>>>> On 21 January 2016 at 19:03, M.-A. Lemburg <m...@egenix.com> wrote:
>>>>>> By using the version based approach, we'd not run into this
>>>>>> problem and gain a lot more.
>>>>>
>>>>> I think it's better to start with a small core that we *know* works,
>>>>> then expand later, rather than trying to make the first iteration too
>>>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>>>> than newer ones.
>>>>
>>>> My argument is that the file based approach taken by the PEP
>>>> is too limiting to actually make things work for a large
>>>> set of Python packages.
>>>>
>>>> It will basically only work for packages that do not interface
>>>> to other external libraries (except for the few cases listed in
>>>> the PEP, e.g. X11, GL, which aren't always installed or
>>>> available either).
>>>>
>>>> IMO, testing the versions of a set of libraries is a safer
>>>> approach. It's perfectly fine to have a few dependencies
>>>> not work in a module because an optional system package is not
>>>> installed, e.g. say a package comes with UIs written in
>>>> Qt and one in GTK.
>>>
>>> Please forgive my slowness, but I don't understand exactly what you
>>> mean.  Can you give a specific example?
>>>
>>> Say my package depends on libpng.
>>>
>>> Call the machine I'm installing on the client machine.
>>>
>>> Are you saying that, when I build a wheel, I should specify to the
>>> wheel what versions of libpng I can tolerate on the the client
>>> machine, and if if the client does have a compatible version, then pip
>>> should raise an error, perhaps with a useful message about how to get
>>> libpng?
>>>
>>> If you do mean that, how do you want the PEP changed?
>>
>> I already posted a change proposal earlier on in the thread.
>> I'll repeat it here (with a minor enhancements):
> 
> Okay, I think I get it now. I'll try to repeat back to summarize and
> see if I have understood your proposal correctly:
> 
> In the PEP 513 "manylinux1" approach, when users do 'pip install foo',
> then one of three things happens:
> 1) they get a working foo and are immediately good-to-go, or
> 2) pip says "I'm sorry, there's no compatible wheel", or
> 3) something else happens, in which case this is a bug, and the spec
> provides some framework to help us determine whether this is a bug in
> the wheel, a bug in pip, or a bug in the spec.
> 
> In your approach, users do 'pip install foo', and then pip installs
> the wheel, and then when they try to use the wheel they get an error
> message from the dynamic linker about missing libraries, and then the
> user has to read the docs or squint at these error messages in order
> to figure out what set of apt-get / yum / pacman / ... commands they
> need to run in order to make foo work. (And possibly there is no such
> combination of commands that will actually work, because e.g. the
> wheel was linked against Debian's version of libbar.so.7 and Fedora's
> version of libbar.so.7 turns out to have an incompatible ABI, or
> Fedora simply doesn't provide a libbar.so.7 package at all.)

pip could be made to check the wheel for missing library
dependencies in order to provide help with cases where
additional packages are needed, but overall, yes, that's
the way it should work, IMO.

It's better to have wheels than not to have them, since
installing an additional system package is by far easier
than trying to compile packages from source (this will
usually also require additional -dev packages to be
installed).

>>  * no lock-out of package authors who would like to push
>>wheel files for their packages to PyPI, but happen to
>>use libraries not in the predefined list of the original
>>draft PEP
>
> https://mail.python.org/pipermail/distutils-sig/2016-January/028050.html

Embedding additional libraries in the wheels files to overcome
deficiencies in the PEP design simply doesn't feel right
to me.

People who rely on Linux di

Re: [Distutils] draft PEP: manylinux1

2016-01-22 Thread M.-A. Lemburg

On 21.01.2016 20:05, Matthew Brett wrote:
> Hi,
> 
> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <m...@egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach. It's perfectly fine to have a few dependencies
>> not work in a module because an optional system package is not
>> installed, e.g. say a package comes with UIs written in
>> Qt and one in GTK.
> 
> Please forgive my slowness, but I don't understand exactly what you
> mean.  Can you give a specific example?
> 
> Say my package depends on libpng.
> 
> Call the machine I'm installing on the client machine.
> 
> Are you saying that, when I build a wheel, I should specify to the
> wheel what versions of libpng I can tolerate on the the client
> machine, and if if the client does have a compatible version, then pip
> should raise an error, perhaps with a useful message about how to get
> libpng?
> 
> If you do mean that, how do you want the PEP changed?

I already posted a change proposal earlier on in the thread.
I'll repeat it here (with a minor enhancements):

"""
The ``manylinux1`` policy
=

For these reasons, we define a set of library versions that
are supported by a wide range of Linux distributions. We therefore
pick library versions which have been around for at least 5 years.

When using these external libraries, Python wheels should
only depend on library versions listed in the section below.

Python wheels are free to depend on additional libraries
not included in this set, however, care should be taken
that these additional libraries do not depend on later
versions of the listed libraries, e.g. OpenSSL libraries
compiled against the C library versions listed below.

The ``manylinux1`` policy thus encompasses a standard for which
versions of these external shared libraries a wheel may depend on,
and the maximum depended-upon symbol versions therein.

Future versions of the manylinux policy may include more
libraries, or move on to later versions.

The permitted external shared libraries versions for
``manylinux1``are: ::

libgcc_s.so.1
libstdc++.so.6
... only include the basic set of libs, no GUI or curses ...

"""

The idea is to not pin down the set of usable external libraries,
but instead pin down a set of versions for the most important
libraries wheels will depend on and then let the wheels use
other external libraries as necessary without version checks.

In more details:

If you want a wheel to run on many Linux distributions, you have
to make sure that the basic lib C and a few other utility
libraries are available and compatible with the ones you used
to build the wheel.

This can be addressed by defining a set of important libraries
and corresponding versions. You do not have to limit the overall
set of usable libraries for this, since less commonly used
libraries will usually have to be installed separately anyway.

For example, if a package needs a specific version of libpng,
the package author can document this and the user can then make
sure to install that particular version.

The PEP should only be concerned with the basic set of
libraries you typically need for a wheel, not any of the
less common ones. The X11 libs for example do not have to
be version pinned for the manylinux tag, since they are not
essential for the vast majority of Python packages (and here
I'm talking about the thousands of packages on PyPI, not the
few hundred mentioned earlier in the thread, which are
covered by Anaconda and Canopy).

By defining "manylinux1" in such a way you get:

 * broad compatibility of wheel files on Linux

 * full flexibility of wheels interfacing or wrapping
   to other external libraries not covered in the PEP

 * no lock-out of package authors wh

Re: [Distutils] draft PEP: manylinux1

2016-01-22 Thread M.-A. Lemburg

On 22.01.2016 12:25, Donald Stufft wrote:
> 
>> On Jan 22, 2016, at 5:48 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>
>> Embedding additional libraries in the wheels files to overcome
>> deficiencies in the PEP design simply doesn't feel right
>> to me.
>>
>> People who rely on Linux distributions want to continue
>> to do so and get regular updates for system packages from
>> their system vendor. Having wheel files override these
>> system packages by including libs directly in the wheel
>> silently breaks this expectation, potentially opening
>> up the installations for security holes, difficult to
>> track bugs and possible version conflicts with already
>> loaded versions of the shared libs.
>>
>> IMO, that's much worse than having to install additional
>> system packages to make a Python wheel work.
>>
>> The embedding approach also creates licensing problems,
>> since those libs may be under different licenses than the
>> package itself. And of course, it increases the size of the
>> wheel files, causing more bandwidth to be necessary,
>> more disk space to be used for wheel caches, etc.
> 
> I think there are a few things here, but this is not my area of expertise so I
> could be wrong. As I understand it, The manylinux platform definition is
> largely going to be a documentation effort and there isn't going to be much in
> the way of enforcement. That means that people who build wheels against the
> manylinux platform tag are free to really do whatever they want even if it
> doesn't strictly match the definition of the manylinux platform. The 
> difference
> here is that if you link against something that isn't included in the set of
> libraries, and that subsequently breaks due to an ABI incompatability, that's
> not a pip bug or a manylinux bug, that's a packaging bug with that particular
> library and they'll have to decide how they want to resolve it (if they want
> to resolve it). So you'll be free to link to anything you want, but you get to
> keep both pieces if it breaks and it's outside this defined set of libraries.

Hmm, if that were the reading, things would look a lot brighter,
but if PyPI will start to only support uploading manylinux wheels
for Linux platforms, you essentially have the effect that the
PEP ends up defining the set of allowed external libraries and forces
package authors to embed any other external libraries into the
wheel file - or not be able to upload wheel files for Linux at all.

This can hardly be in the interest of Python users who don't want
to use wheel embedded system libraries on their Linux system and
most likely also don't expect wheel files to ship alternative
versions with them in the first place.

If we'd lift the ban of "linux_*" tagged wheels on PyPI at
the same time we allow "manylinux" wheels, that'd remove a lot
of my concerns.

In that case, I'd just like to see a way to tell pip not to install
manylinux wheels with embedded system libraries, or simply outright
reject embedded system libraries in manylinux wheel files.

> I also agree that it's OK for users to have to ``apt-get`` (or whatever) a
> particular library to use something and we don't have to *only* rely on items
> that are installed as part of a "standard" linux base system. However, what is
> not OK (IMO) is for the PEP to bless something that has a high chance of 
> ending
> up with ABI issues rather than "need to apt-get install" issues. For instance,
> even if you compile against a sufficiently old copy of OpenSSL, OpenSSL (to my
> understanding) does not have a stable ABI and you cannot take something
> compiled against OpenSSL on CentOS 5.reallyold and expect it to work on say
> Arch Linux.

True. There will always be incompatibilities out there which
cannot be addressed with a one-fits-all approach. For those
cases, vendor specific wheels would need to be created.

> So I think there's an explicit list of packages that we know will generally
> work as long as you build against a sufficiently old copy of them and outside
> of that it's really a big unknown in general if a particular library can be
> used in this way or not. We obviously can't enumerate the list of every
> possible C library that has a stable ABI that can sanely be used cross distro
> but I think it's reasonable to list some sort of base minimum here, and if
> people experiment with stepping outside the standard list and can come to us
> and show "hey, I tried it with xyz library, we've gotten X installs and no
> complaints" we can then possibly expand the definition of the manylinux
> platform to include that library and move that project from depending on
> undefined behavior to defined behavior.
> 
> Th

Re: [Distutils] draft PEP: manylinux1

On 21.01.2016 11:11, Nick Coghlan wrote:
> On 21 January 2016 at 20:05, M.-A. Lemburg <m...@egenix.com> wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <m...@egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach.
> 
> I still don't really understand what you mean by "testing the versions
> of a set of libraries", but if you have the time available to propose
> a competing PEP, that always leads to a stronger result than when we
> only have only proposed approach to consider.

I think the PEP is fine as it is, just the restriction to test
for the library file names is something that would need to be
changed to implement the different approach:

"""
For these reasons, we define a set of library versions that
are supported by a wide range of Linux distributions. We therefore
pick library versions which have been around for at least 5 years.

When using these external libraries, Python wheels should
only depend on library versions listed in the section below.

Python wheels are free to depend on additional libraries
not included in this set, however, care should be taken
that these additional libraries do not depend on later
versions of the listed libraries, e.g. OpenSSL libraries
compiled against the C library versions listed below.

The ``manylinux1`` policy thus encompasses a standard for which
versions of these external shared libraries a wheel may depend on,
and the maximum depended-upon symbol versions therein.

Future versions of the manylinux policy may include more
libraries, or move on to later versions.

The permitted external shared libraries and versions for
``manylinux1``are: ::

libpanelw.so.5
libncursesw.so.5
libgcc_s.so.1
libstdc++.so.6
...
"""

This will still lead to cases where a package doesn't work
because of missing system packages, but at least they won't
fail due to mismatch in basic C library versions, which is the
most problematic case for users.

The PEP will also have to address to problems introduced
by versioned symbols in more recent Linux shared libs:
even though the library file names have not changed,
they may well include different support levels for the
various APIs, e.g. glibc 2.1, 2.2, 2.3, etc.

For our binaries, we have chosen to use a system where
this versioning has not yet been enabled for system libs.
We did this because we found using a library compiled
against a versioned lib on a system which comes with
an unversioned lib causes warnings to be issued.

For openSUSE the change was applied between the 11.3 and
11.4 releases.

Some references which show case the problem:
 - 
http://stackoverflow.com/questions/137773/what-does-the-no-version-information-available-error-from-linux-dynamic-linker/156387#156387
 - 
http://superuser.com/questions/305055/how-to-diagnosis-and-resolve-usr-lib64-libz-so-1-no-version-information-avail
 - 
http://forums.opensuse.org/english/get-technical-help-here/applications/466547-google-chrome-issues.html

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 21 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

On 21.01.2016 10:31, Nick Coghlan wrote:
> On 21 January 2016 at 19:03, M.-A. Lemburg <m...@egenix.com> wrote:
>> By using the version based approach, we'd not run into this
>> problem and gain a lot more.
> 
> I think it's better to start with a small core that we *know* works,
> then expand later, rather than trying to make the first iteration too
> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
> end), so "manylinux2" may simply have *more* libraries defined, rather
> than newer ones.

My argument is that the file based approach taken by the PEP
is too limiting to actually make things work for a large
set of Python packages.

It will basically only work for packages that do not interface
to other external libraries (except for the few cases listed in
the PEP, e.g. X11, GL, which aren't always installed or
available either).

IMO, testing the versions of a set of libraries is a safer
approach. It's perfectly fine to have a few dependencies
not work in a module because an optional system package is not
installed, e.g. say a package comes with UIs written in
Qt and one in GTK.

pip could then warn about missing dependencies in the
installed packages.

Another detail we have found when external dependencies
is that some platforms use different names for the libraries,
e.g. RedHat has a tendency to use non-standard OpenSSL
library names (/lib64/libssl.so.10 instead of the more
common libssl.so.1.0.0).

> The key is that we only have one chance to make a good first
> impression with binary Linux wheel support on PyPI, and we want that
> to be positive for everyone:

Sure, but if we get the concept wrong, it'll be difficult
to switch later on and since there will always be libs not in the
set, we'll need to address this in some way.

> * for publishers, going from "no Linux wheels" to "Linux wheels if you
> have few external dependencies beyond glibc" is a big step up (it's
> enough for a Cython accelerator module, for example, or a cffi wrapper
> around a bundled library)
> * for end users, we need to nigh certain that wheels built this way
> will *just work*
> 
> Even with a small starting list of libraries defined, we're going to
> end up with cases where the installed extension module will fail to
> load, and end users will have to figure out what dependencies are
> missing. The "external dependency specification" at
> https://github.com/pypa/interoperability-peps/pull/30 would let pip
> detect that at install time (rather the user finding out at runtime
> when the module fails to load), but that will still leave the end user
> to figure out how to get the external dependencies installed.
> 
> If Donald can provide the list of "most downloaded wheel files" for
> other platforms, that could also be a useful guide as to how many
> source builds may potentially already be avoided through the draft
> "manylinux1" definition.

I still believe that installers are in a better position to
decide which binary files to install based on the findings
on the installation system. This is why we are using a more
flexible tag system in our prebuilt format:

http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/

In essence, the installer knows which files are available
and can then analyze the installation system to pick the
right binary. Tags can be added as necessary to address all
the different dimensions that need testing, e.g. whether the
binary runs on a Raspi2 or only a Raspi1.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 21 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

Robert McGibbon: Thanks for writing up this PEP :-)

Some comments below...

On 21.01.2016 04:55, Nathaniel Smith wrote:
> The ``manylinux1`` policy
> =
> 
> For these reasons, to achieve broad portability, Python wheels
> 
>  * should depend only on an extremely limited set of external shared
>libraries; and
>  * should depend only on ``old`` symbol versions in those external shared
>libraries.
> 
> The ``manylinux1`` policy thus encompasses a standard for what the
> permitted external shared libraries a wheel may depend on, and the maximum
> depended-upon symbol versions therein.
> 
> The permitted external shared libraries are: ::
> 
> libpanelw.so.5
> libncursesw.so.5
> libgcc_s.so.1
> libstdc++.so.6
> libm.so.6
> libdl.so.2
> librt.so.1
> libcrypt.so.1
> libc.so.6
> libnsl.so.1
> libutil.so.1
> libpthread.so.0
> libX11.so.6
> libXext.so.6
> libXrender.so.1
> libICE.so.6
> libSM.so.6
> libGL.so.1
> libgobject-2.0.so.0
> libgthread-2.0.so.0
> libglib-2.0.so.0

The list is good start, but limiting the possible external
references to only these libraries will make it impossible
to have manylinux1 wheels which link against other, similarly
old, but very common libraries, or alternatively rather
new ones, which are then optionally included via subpackage
instead of being mandatory.

At eGenix we have been tackling this problem for years with
our extensions and the approach that's been the most
successful was to simply use Linux build systems which
are at least 5 years old. In our case, that's openSUSE 11.3.

I think a better approach is to use the above list to
test for used library *versions* and then apply the tag
based on the findings.

If a package includes binaries which link to e.g.
later libc.so versions, it would be rejected. If it
includes other libraries not listed in the above listing,
that's fine, as long as these libraries also comply to
the version limitation.

What I'm getting at here is that incompatibilities are
not caused by libraries being absent on the system
(the package simply won't load, but that's not due to the
the package being incompatible to the platform, only due
to the system lacking a few packages), but instead by
having the packages use more recent versions of these
system libraries.

> Compilation and Tooling
> ===
> 
> To support the compilation of wheels meeting the ``manylinux1`` standard, we
> provide initial drafts of two tools.
> 
> The first is a Docker image based on CentOS 5.11, which is recommended as an
> easy to use self-contained build box for compiling ``manylinux1`` wheels [4]_.
> Compiling on a more recently-released linux distribution will generally
> introduce dependencies on too-new versioned symbols. The image comes with a
> full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as
> well as the latest releases of Python and pip.
> 
> The second tool is a command line executable called ``auditwheel`` [5]_. 
> First,
> it inspects all of the ELF files inside a wheel to check for dependencies on
> versioned symbols or external shared libraries, and verifies conformance with
> the ``manylinux1`` policy. This includes the ability to add the new platform
> tag to conforming wheels.
> 
> In addition, ``auditwheel`` has the ability to automatically modify wheels 
> that
> depend on external shared libraries by copying those shared libraries from
> the system into the wheel itself, and modifying the appropriate RPATH entries
> such that these libraries will be picked up at runtime. This accomplishes a
> similar result as if the libraries had been statically linked without 
> requiring
> changes to the build system.

This approach has a few problems:

* Libraries typically depend on a lot more context than just
  the code that is provided in the libraries file, e.g. config
  files, external resources, other libraries which are loaded
  on demand, etc.

* By including the libraries in the wheel you are distributing
  the binary, which can lead to licensing problems, esp. with
  GPLed or LGPLed code.

> Neither of these tools are necessary to build wheels which conform with the
> ``manylinux1`` policy. Similar results can usually be achieved by statically
> linking external dependencies and/or using certain inline assembly constructs
> to instruct the linker to prefer older symbol versions, however these tricks
> can be quite esoteric.

Static linking only helps in very few cases, where the context
needed for the external library to work is minimal.

> Platform Detection for Installers
> =
> 
> Because the ``manylinux1`` profile is already known to work for the many
> thousands of users of popular commercial Python distributions, we suggest that
> installation tools like ``pip`` should error on the side of assuming that a
> system *is* compatible, unless there is specific reason to

Re: [Distutils] draft PEP: manylinux1

On 21.01.2016 17:13, Nathaniel Smith wrote:
> On Jan 21, 2016 2:07 AM, "M.-A. Lemburg" <m...@egenix.com> wrote:
>>
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <m...@egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
> 
> The list of allowed libraries is exactly the same list of libraries as are
> required by the Anaconda python distribution, so we *know* that it works
> for about a hundred different python packages, including lots of tricky
> ones (the whole scientific stack), and had been tested by tens or hundreds
> of thousands of users. (I posted a link above to some actual working,
> compliant pyside and numpy packages, which we picked for testing because
> they're particular interesting/difficult packages that need to interface to
> external libraries.) Yes, various extra tricks are needed to get things
> working on top of this base, including strategies for shipping libraries
> that are not in the baseline set, but these are problems that can be solved
> on a project-by-project basis, and don't need a PEP.

And that's the problem: The set is limited to the needs
of the scientific community and there to the users of
one or two distributions only.

It doesn't address needs of others that e.g. use Qt or GTK as
basis for GUIs, people using OpenSSL for networking, people
using ImageMagick for processing images, or type libs for
type setting, or sound libs for doing sound processing,
codec libs for video processing, etc.

The idea to include the needed share libs in the wheel
goes completely against the idea of relying on a system
vendor to provide updates and security fixes. In some cases,
this may be reasonable, but as design approach, it's
not a good idea.

> [...]
>> Another detail we have found when external dependencies
>> is that some platforms use different names for the libraries,
>> e.g. RedHat has a tendency to use non-standard OpenSSL
>> library names (/lib64/libssl.so.10 instead of the more
>> common libssl.so.1.0.0).
>>
>>> The key is that we only have one chance to make a good first
>>> impression with binary Linux wheel support on PyPI, and we want that
>>> to be positive for everyone:
>>
>> Sure, but if we get the concept wrong, it'll be difficult
>> to switch later on and since there will always be libs not in the
>> set, we'll need to address this in some way.
> 
> There's no lock-in here -- any alternative approach just needs its own
> platform tag. Pypi and pip can easily support multiple such tags at the
> same time, if more sophisticated proposals arise in the future. In the mean
> time, for packagers targeting manylinux is at least as easy as targeting
> windows (which also provides very few libraries "for free").

Yes, there's no lock-in, there's lock-out :-)

We'd simply not allow people who have other requirements to
upload Linux wheels to PyPI and that's not really acceptable.

Right now, no one can upload Linux wheels, so that a fair
setup.

Using an approach where every single group first has to write
a PEP, get it accepted and have PyPI and pip patched before
they can upload wheels to PyPI does not read like a community
friendly approach.

I also can't imagine that we really want proliferation of
"linux" tags for various purposes or even for various
versions of library catalogs.

What we need is a system that provides a few dimensions
for various system specific differences (e.g. bitness,
architecture) and a recommendation for library
versions of a few very basic libraries to use when
compiling for those systems.

I believe the PEP is almost there, it just needs to use a
slightly different concept, since limiting the set of
allowed libraries does not provide the basis of an open
system which PyPI/pip/wheels need to be.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Ja

Re: [Distutils] build system abstraction PEP, take #2

2015-12-10 Thread M.-A. Lemburg

In all this discussion, please don't forget that distutils and
setuptools differentiate between building the code and creating
a distribution file.

The later is not restricted to just the sdist and wheel formats.
distutils can create a wide variety of other distribution formats
such as MSI files, Windows .exe installers, binary in-place
distributions. With extensions it's possible to create the
ActiveState package format, RPMs, DEBs, Solaris PKG files and
other formats such as our eGenix prebuilt format or web
installers.

Apart from that it's also possible to have distutils
completely drop the distribution file generation and go
straight to installation after the build step, e.g.
when not using a package manager at all.

IMO, it would be better to use the existing "setup.py build"
and "setup.py bdist_wheel" APIs to create a build system
abstraction, since that'll keep the other existing distribution
methods working in the same context, which is important since
pip is not the only way to distribution Python packages.

The PEP's ideas as well as many other approaches to building
packages can be implemented using a "setup.py build"
and "setup.py bdist_wheel" interface ("bdist_wheel" will
implicitly run "build").

To specifically use the PEP's mechanism, a package could be
created which overrides and implements the PEP's build strategy,
e.g. "pep778build" (assuming the PEP number is 778 as example).

The dependency of a package on this pep778build package
would then signal the compatibility of the package with
the PEP's build mechanism.

In summary: As long as the final result of a call to
"setup.py bdist_wheel" is a .whl file in dist/, pip should be
able to continue working as normal (and as it does today),
regardless of what "setup.py build" does under the hood.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Dec 10 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Please don't impose additional barriers to participation

2015-10-28 Thread M.-A. Lemburg

On 28.10.2015 06:02, Ben Finney wrote:
> Marcus Smith  writes:
> 
>> 1) *Please*, *please*, *please* let's start doing PEP conversations as
>> PRs to pypa/interoperability-peps : )
> 
> Please keep the conversation on a mailing list where one can participate
> without needing to sign up with any particular service provider.
> 
> Your proposal would have the effect of excluding people from the
> conversation if they don't agree to have a GitHub account. I think it's
> valuable to avoid that barrier to entry, needing only an email account.

I agree with Ben. Discussions on PEPs need to happen on mailing lists,
not hidden away on some issue tracker or PR ticket.

PEP generally affect a large number of Python users, so it's vital
to get as much feedback as possible. This is not possible using
a PR approach.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 28 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Please don't impose additional barriers to participation

2015-10-28 Thread M.-A. Lemburg

On 28.10.2015 19:33, Ian Cordasco wrote:
> On Wed, Oct 28, 2015 at 3:31 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>> On 28.10.2015 06:02, Ben Finney wrote:
>>> Marcus Smith <qwc...@gmail.com> writes:
>>>
>>>> 1) *Please*, *please*, *please* let's start doing PEP conversations as
>>>> PRs to pypa/interoperability-peps : )
>>>
>>> Please keep the conversation on a mailing list where one can participate
>>> without needing to sign up with any particular service provider.
>>>
>>> Your proposal would have the effect of excluding people from the
>>> conversation if they don't agree to have a GitHub account. I think it's
>>> valuable to avoid that barrier to entry, needing only an email account.
>>
>> I agree with Ben. Discussions on PEPs need to happen on mailing lists,
>> not hidden away on some issue tracker or PR ticket.
> 
> Others may be willing to tolerate your FUD, but without concrete
> reasons against GitHub (other than zomg it's a proprietary service) I
> don't see a reason to not use the pull request flow on an open
> repository that is free for people to clone, fork, contribute to, etc.
> 
> GitHub isn't my preferred hosting platform for git but it is the
> defacto standard and it's workflow ubiquitous, documented, and far
> more user-friendly than mailing list threads (especially when they
> devolve into ideology wars).
> 
> Also nothing precludes mailing list discussions, so without details
> about your objections, I don't see why this should be held up.

Ok, as first step, please change your tone and reread my
reply. I'm not willing to tolerate your tone. Thanks.

The argument here is not about Github or not, and it's not FUD.
It's the same argument as is used when starting into discussions
on the Python issue tracker and the reason for often moving those to
the python-dev or -ideas mailing lists.

Hiding away discussions is not the way to go about when discussing
PEPs. Using PRs is fine when it comes to improving the language
of the PEP or adding new aspects, but the reasoning for applying
these changes should be based on the results of discussions on
the relevant lists, distutils-sig in this case.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 28 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 503 - Simple Repository API

2015-09-07 Thread M.-A. Lemburg

On 05.09.2015 18:12, Donald Stufft wrote:
> On September 5, 2015 at 5:43:58 AM, M.-A. Lemburg (m...@egenix.com) wrote:
>>  
>> Hmm, if the installer will build the URL itself, why is there even
>> a need for a top-level index page ?
>>  
>> I mean for the occasional human reading the page it will certainly
>> make sense to have such a page, but for the API this doesn't
>> appear to be essentially needed.
>>  
>> Or is the idea to have the package manager scan the index for package
>> hosted on that index prior to asking for the package it would like
>> to install ?
> 
> The latest versions of pip won't use it, setuptools and older versions of pip
> will use it though. The versions of pip/setuptools that would use it, use it 
> as
> a fallback. They don't pre-normalize the name before requesting the URL so 
> they
> just used whatever the user typed. This comes from when a project like 
> "Django"
> was at /simple/Django/ on PyPI but if a user typed ``pip install django`` it
> would first fetch /simple/django/ and if that 404'd it would fall back onto
> /simple/ and look for these links. On PyPI this rarely happened because PyPI
> redirects /simple/anything-that-normalizes-to-the-name/ to the correct URL but
> it's useful for static repositories that don't have something to redirect it
> in front.
> 
> I've tried to make it so that all of the SHOULD and MUST directives can be
> implement by a standard Nginx/Apache/whatever web server with static files
> while maintaining compatability with older installers.

Yes, understood, and that's good.

Perhaps having an index page is a good thing if we want
package managers to implement search functionality.

>> Would it help the package manager to more easily detect the links
>> that point to distribution files instead of e.g. documentation or
>> other resources ?
>>  
>> setuptools uses rel="download" for this:
>>  
>> https://pythonhosted.org/setuptools/easy_install.html#package-index-api 
> 
> This is actually for the link spidering that PEP 470 removed, links marked 
> with
> either rel="download" or rel="homepage" would be fetched (unless they looked
> installable) and searched for additional links before PEP 438/470 started to
> deprecate/remove them. Both setuptools and pip only need a simple page that 
> has
> links that point to files on the, see for example the /simple/ page for
> requests: https://pypi.python.org/simple/requests/

Right. Perhaps I should have made the use case I'm thinking of
more obvious:

If you set up a page with links to projects and distribution files,
you will likely not make completely unstyled but instead integrate
it into some website which also has lots of other links to e.g.
other parts of the website, images, documentation, etc.

In such a setup, the package manager would see lots and lots of
links which are not relevant for the task. With the rel attributes,
the package manager can focus on those links which are relevant.
That's also the main reason for having those rel links in setuptools.

>> Could we perhaps also add optional features like:
>>  
>> * Distribution link elements MAY include a data-gpg-sig=""
>> attribute to provide a GPG signature of the linked file
>>  
>> This could later be extended to more meta data, such as platform
>> tags, distribution file types, license info, mirror locations,
>> documentation, help strings, etc.
> 
> I actually forgot to mention the GPG signatures, currently the assumption is
> that if a GPG signature exists it will live at the same location as the file
> with a .asc on the end, so if the file is /packages/Django-1.0.tar.gz then the
> GPG signature will be located at /packages/Django-1.0.tar.gz.asc. I'll add 
> this
> to the PEP.

Hmm, that's convention based and does not allow detecting
the presence of such signatures without actually trying a download.

I think it would be better to make the availability explicit
by adding an attribute to the link element (just like for other
such features).

> I don't want to add more features to the API, particularly not in this PEP. My
> longer term plan is to work on a a new API for installers to use which will be
> easier to work with. The current API is great for it's simplicity and the fact
> it can be implemented on the server side with nothing more than a directory
> structure full of files and python -m http.server. The plan in my head is to
> add a new repository API which can handle the more complex cases AND which 
> will
> most likely be JSON based to simplify parsing of it. The simple API would not
> be deprecated, it would just be up to the repository which "version" of the 
> API
&g

Re: [Distutils] PEP 503 - Simple Repository API

2015-09-05 Thread M.-A. Lemburg

On 05.09.2015 03:17, Donald Stufft wrote:
> You can see this PEP online at https://www.python.org/dev/peps/pep-0503/ or I
> have reproduced it inline below.

Thanks for writing this up, Donald.

Some comments below...

> ---
> 
> Abstract
> 
> 
> There are many implementations of a Python package repository and many tools
> that consume them. Of these, the cannonical implementation that defines what

s/cannonical/canonical/

> the "simple" repository API looks like is the implementation that powers
> PyPI. This document will specify that API, documenting what the correct
> behavior for any implementation of the simple repository API.
> 
> Specification
> =
> 
> A repository that implements the simple API is defined by its base url, this 
> is
> the top level URL that all additional URLS are below. The API is named the
> "simple" repository due to fact that PyPI's base URL is
> ``https://pypi.python.org/simple/``.
> 
> .. note:: All subsequent URLs in this document will be relative to this base
>   URL (so given PyPI's URL, an URL of ``/foo/`` would be
>   ``https://pypi.python.org/simple/foo/``).
> 
> 
> Within a repository, the root URL (``/``) **MUST** be a valid HTML5 page with 
> a
> single anchor element per project in the repository. The text of the anchor 
> tag
> **MUST** be the normalized name of the project and the href attribute **MUST**
> link to the URL for that particular project. As an example::
> 
>
>
>  
>frob
>spamspamspam
>  
>
> 
> Below the root URL is another URL for each individual project contained within
> a repository. The format of this URL is ``//`` where the 
> 
> is replaced by the normalized name for that project, so a project named
> "HolyGrail" would have an URL like ``/holygrail/``. 

Hmm, if the installer will build the URL itself, why is there even
a need for a top-level index page ?

I mean for the occasional human reading the page it will certainly
make sense to have such a page, but for the API this doesn't
appear to be essentially needed.

Or is the idea to have the package manager scan the index for package
hosted on that index prior to asking for the package it would like
to install ?

> This URL must response with
> a valid HTML5 page with a single anchor element per file for the project. The
> text of the anchor tag **MUST** be the filename of the file and the href
> attribute **MUST** be an URL that links to the location of the file for
> download. The URL **SHOULD** include a hash in the form of an URL fragment 
> with
> the following syntax: ``#=``, where  is the
> lowercase name of the hash function (such as ``sha256``) and  
> is
> the hex encoded digest.
> 
> In addition to the above, the following constraints are placed on the API:
> 
> * All URLs **MUST** end with a ``/`` and the repository **SHOULD** redirect 
> the
>   URLs without a ``/`` to add a ``/`` to the end.

I think you only meant this for URLs that point to index pages,
since doing this for filenames would not be such a good idea
(confuses the MIME content type logic).

For site navigation, pages will typically also include relative links
such as "..". The spec should not disallow these.

> * There is no constraints on where the files must be hosted relative to the
>   repository.
> 
> * There may be any other HTML elements on the API pages as long as the 
> required
>   anchor elements exist.

Would it help the package manager to more easily detect the links
that point to distribution files instead of e.g. documentation or
other resources ?

setuptools uses rel="download" for this:

https://pythonhosted.org/setuptools/easy_install.html#package-index-api

The downside here is that a simple web server directory listing would
no longer be compatible with the spec, so perhaps just make this optional
to optimize the link scanning:

* Project pages SHOULD add a rel="homepage" attribute to link
  elements of distribution file.

The same could then be done for index page links to project pages:

* Index pages SHOULD add a rel="download" attribute to link
  elements of distribution file.

The rel attributes used here are the ones that setuptools requires,
in order to be able to build indexes which are compatible to
setuptools as well.

> * Repositories **MAY** redirect unnormalized URLs to the cannonical normalized
>   URL (e.g. ``/Foobar/`` may redirect to ``/foobar/``), however clients
>   **MUST NOT** rely on this redirection and **MUST** request the normalized
>   URL.
> 
> * Repositories **SHOULD** choose a hash function from one of the ones
>   guarenteed to be available via the ``hashlib`` module in the Python standard

s/guarenteed/guaranteed/

>   library (currently ``md5``, ``sha1``, ``sha224``, ``sha256``, ``sha384``,
>   ``sha512``). The current recommendation is to use ``sha256``.

Could we perhaps also add optional features like:

* Distribution link elements

Re: [Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

2015-08-31 Thread M.-A. Lemburg

On 29.08.2015 15:57, Nick Coghlan wrote:
> On 28 Aug 2015 07:31, "Robert Collins" <robe...@robertcollins.net> wrote:
>>
>>
>> On 28 Aug 2015 9:00 am, "M.-A. Lemburg" <m...@egenix.com> wrote:
>>>
>>
>>> All Linux distros I know and use have repositories distributed all
>>> over the planet, and many also provide official and less official
>>> ones, for the users to choose from, so there is more than enough
> evidence
>>> that a federated system for software distribution works better than a
>>> centralized one.
>>
>> None of them provide cross repository discovery except Conary ttbomk. And
> its is inherited so a different ux.
>>
>> So that's a difference.
> 
> Right, the distro model is essentially the one Donald is proposing -
> centrally controlled default repos, ability to enable additional repos on
> client systems. Geographically distributed mirrors are different, as those
> are just redistributing signed content from the main repos.
> 
> Hosting in multiple regions and/or avoiding selected regions would
> definitely be a nice service to offer, and it would be good to have a
> straightforward way to deploy and run an external repo (e.g. a devpi Docker
> image), but the proposed core model is itself a tried and tested one.
> Reducing back to that, and restarting the exploration of multi-index
> support from there with a clear statement of objectives would be a good way
> to go.
> 
> If we need to manually whitelist some external repos for transition
> management purposes, then that's likely a better option than settling for a
> nominally general purpose feature we'd prefer people didn't actually use.

There are quite a few systems out there that let you search for
repos with the packages you need, but they are usually web based and
not integrated into the package managers, e.g. rpmfind, various PPA
search tools (e.g. Launchpad or open build service), etc.

There's also another difference: Linux repos are usually managed
by a single entity owning the packages, very much unlike PyPI which
is merely a hosting platform and index to point to packages owned
by the authors.

So it's natural for PyPI to let package manager users know about
where to find packages which are not hosted on PyPI and the user
experience (which people always bring up as the number one argument
for all sorts of things on this list ;-)), is much better when
providing this information to the user directly, rather than saying
"I couldn't find any distribution files for you - go look on PyPI
for instructions where to find them...".

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 31 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-08-27: Released eGenix mx Base 3.2.9 ... http://egenix.com/go83
2015-08-19: Released mxODBC 3.3.5 ... http://egenix.com/go82

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

2015-08-31 Thread M.-A. Lemburg

On 31.08.2015 11:05, Wichert Akkerman wrote:
> On 31 Aug 2015, at 10:44, M.-A. Lemburg <m...@egenix.com> wrote:
>>
>> There's also another difference: Linux repos are usually managed
>> by a single entity owning the packages, very much unlike PyPI which
>> is merely a hosting platform and index to point to packages owned
>> by the authors.
> 
> That is probably true for public repositories. However, there are also a huge 
> number of organisations who have internal repositories for deb/rpm packages, 
> and many of those contain third party packages. I have a couple, and most of 
> them contain a combination of our own packages as well as collection of 
> backports and custom packages for software that hasn’t been packaged by 
> anyone else.

True, but for those, I think explicitly adding the index URL to
the package installer search path is the better approach.

Or perhaps I misunderstood and you meant something like:

"If the package is not in my internal repo, I don't want
pip to look it up on PyPI or anywhere else."

That's a valid use case, but it seems orthogonal to the
question of making public repositories for specific packages
more easily configurable for package manager users.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 31 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2015-08-27: Released eGenix mx Base 3.2.9 ... http://egenix.com/go83
2015-08-19: Released mxODBC 3.3.5 ... http://egenix.com/go82

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

2015-08-27 Thread M.-A. Lemburg

On 27.08.2015 13:51, Donald Stufft wrote:
 On August 27, 2015 at 6:57:17 AM, M.-A. Lemburg (m...@egenix.com) wrote:
 This feature was part of a compromise to reach consensus on the removal
 of external hosting. While I don't think the details of the repository 
 discovery
 need to be part of PEP 470, I do believe that the PEP should continue
 to support the idea of having a way for package managers to easily
 find external indexes for a particular package and not outright reject it.
  
 Instead, the PEP should highlight this compromise and defer it to a
 separate PEP.
 
 I’ve never thought that particular API was actually a good idea, I think it’s 
 a poor end user experience because it invokes the same sort of “well if you 
 knew what I needed to type to make it work, why didn’t you just do it for me” 
 reaction as PEP 438 does. The user experience will be something like:
 
 $ pip install something-not-hosted-on-pypi
 ...
 ERROR: Can not find something-not-hosted-on-pypi, it is not hosted on PyPI, 
 it's author has indicated that it found at:
 
 * https://pypi.example.com/all/ : All Platforms
 * https://pypi.example.com/ubuntu-trust/ : Ubuntu Trusty
 
 To enable, please invoke pip by added --extra-index-url An URL from Above

Uhm, no :-) This would be a more user friendly way of doing it:

$ pip install something-not-hosted-on-pypi
...
I'm sorry, but we cannot find something-not-hosted-on-pypi on the available
configured trusted indexes:

* https://pypi.python.org/

However, the author has indicated that it can be found at:

* https://pypi.example.com/

Should we add this PyPI index to the list of trusted indexes ? (y/n)
 y

Thank you. We added https://pypi.example.com/ to the list of indexes. You
are currently using these indexes as trusted indexes:

* https://pypi.python.org/
* https://pypi.example.com/

We will now retry the installation.

...

something-not-hosted-on-pypi installed successfully.
$

 $ pip install --extra-index-url https://pypi.example.com/all/ 
 something-not-hosted-on-pypi
 
 This leaves the user feeling annoyed that we didn’t just search those 
 locations by default. I truly think it is a bad experience and I only ever 
 added it because I wanted the discussion to be over with and I was trying to 
 placate people by giving them a bad feature.

There's nothing bad in the above UI. A user will go through the discovery
process once and the next time around, everything will just work.

 Instead, I think that we can design a solution that works by default and will 
 work without the end user needing to do anything at all. However, I’m not an 
 expert in the US laws (the country I live in and have lived in all my life) 
 and I’m especially not an expert in the laws of countries other than the US. 
 This means I don’t fully understand the issue that needs to be solved. In 
 addition to that, I only have so many hours in the day and I need to find a 
 way to prioritize what I’m working on, the data sovereignty issue may only 
 affect people who do not live in the US, however it does not affect everyone 
 who is outside of the US. Most projects from authors outside of the US are 
 perfectly fine with hosting their projects within the US and it is a minority 
 of projects who cannot or will not for one reason or another.

All Linux distros I know and use have repositories distributed all
over the planet, and many also provide official and less official
ones, for the users to choose from, so there is more than enough evidence
that a federated system for software distribution works better than a
centralized one.

I wonder why we can't we agree on this ?

 I am happy to work with someone impacted by the removal of offsite to design 
 and implement a solution to these issues that provides an experience to those 
 people that matches the experience for people willing or able to host in the 
 US. If the PSF wants to hire someone to do this instead of relying on someone 
 affected to volunteer, I’m also happy to work with them. However, I do not 
 think it’s fair to everyone else, inside and outside of the US, to continue 
 to confuse and infuriate them while we wait for someone to step forward. I’m 
 one person and I’m the only person who gets paid dedicated time to work on 
 Python’s packaging, but I’m spread thin and I have a backlog a mile long, if 
 I don’t prioritize the things that affect most people over the things that 
 affect a small number of people, and leave it up to the people who need an 
 edge case feature things that are already blocked on me are going to languish 
 even further.

I'm happy to help write a PEP for the discovery feature and I'd also
love to help with the implementation. My problem is that no one is
paying me to work on this and so my time investment into this has
to stay way behind of what I'd like to invest.

 Finally, I wasn’t sure if this should be a new PEP or if it should continue 
 as PEP 470, I talked to Nick and he suggested it would be fine to just

Re: [Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

2015-08-27 Thread M.-A. Lemburg

On 27.08.2015 03:24, Donald Stufft wrote:
 While developing Warehouse, one of the things I wanted to get done was a 
 final ruling on PEP 470. With that in mind I’d like to bring it back up for 
 discussion and hopefully ultimately a ruling.
 
 Their are two major differences in this version of PEP 470, and I’d like to 
 point them out explicitly.
 
 Removal of the “External Repository Discover” feature. I’ve been thinking 
 about this for awhile, and I finally removed it. I’ve always been 
 uncomfortable with this feature and I finally realized why it was. 
 Essentially, the major use case for not hosting things on PyPI that I think 
 PyPI can reasonably be expected to accommodate is people who cannot publish 
 their software to the US for various reasons. At the time I came up with the 
 solution I did, It was an attempt to placate the folks who were against PEP 
 470 while assuming very few people would ever actually use it, essentially a 
 junk feature to push the PEP through. I think that the feature itself is a 
 bad feature and I think it presents a poor experience for people who want to 
 use it, so I’ve removed it from the PEP and instead focused the PEP on 
 explicitly recommending that all installers should implement the ability to 
 specify multiple repositories and deprecating and removing the ability for 
 finding anything but file
 s hoste
d by the repository itself on /simple/.

This feature was part of a compromise to reach consensus on the removal
of external hosting. While I don't think the details of the repository discovery
need to be part of PEP 470, I do believe that the PEP should continue
to support the idea of having a way for package managers to easily
find external indexes for a particular package and not outright reject it.

Instead, the PEP should highlight this compromise and defer it to a
separate PEP.

More comments:

* The user experience argument you give in the PEP 470 for rejecting the
idea is not really sound: the purpose of the discovery feature is to
provide a *better user experience* than an error message saying that a package
cannot be found and requiring the user to do research on the web to find
the right URLs. Package managers can use the information about the
other indexes they receive from PyPI to either present them to the user
to use/install them or to even directly go there to find the packages.

* The section on data sovereignty should really be removed or reworded.
PEPs should be neutral and not imply political views, in particular not
make it look like the needs of US users of PyPI are more important then
those of non-US users. Using poor user experience as argument here is
really not appropriate.

  PyPI is a central part of the Python community infrastructure and
should be regarded as a resource for the world-wide community.
There is no reason to assume that we cannot have several PyPI
installations around the world to address any such issues.

 * It is rather unusual to have a PEP switch from a compromise solution
to a rejection of the compromise this late in the process.

I will soon be starting a PSF working group to address some of
the reasons why people cannot upload packages to PyPI. The intent
is to work on the PyPI terms to make them more package author
friendly. Anyone interested to join ?

 I recognize this is a regression for anyone who *does* have concerns with 
 uploading their projects to a server hosted in the US. If there is someone 
 that has this concern, and is also willing to put in the effort and legwork 
 required, I will happily collaborate with them to design a solution that both 
 follows whatever legal requirements they might have, as well as provides a 
 good experience for people using PyPI and pip. I have some rough ideas on 
 what this could look like, but I think it’s really a separate discussion 
 since I believe externally hosted files like we were is an overall bad 
 experience for people and is largely a historic accident from how PyPI and 
 Python packaging has evolved. I don’t want to derail this thread or PEP 
 exploring these ideas (some of which I don’t even know if they would satisfy 
 the requirements since it’s all dealing with legal jurisdictions other than 
 my own), but i wanted to make explicit that someone who knows the legalities 
 and is willing 
 to put 
in the work can reach out to me.

We can start a separate thread about discovery, using a separate PEP
to formalize it.

This could be as simply as having a flag saying use the download URL
as index and offer this to the user trying to find the package distribution
files.

 The other major difference is that I’ve shortened the time schedule from 6 
 months to 3 months. Given that authors are either going to upload their 
 projects to PyPI or not and there is no longer a need to setup an external 
 index I think a shorter time schedule is fine, especially since they will be 
 given a script they can run that will spider their projects for any

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

2015-04-22 Thread M.-A. Lemburg

[adding list back on CC]

On 22.04.2015 16:11, Christoph Schmitt wrote:
 Am 2015-04-22 12:59, schrieb M.-A. Lemburg:
 On 22.04.2015 12:38, Robert Collins wrote:
 On 22 April 2015 at 22:13, Christoph Schmitt dev-maili...@gongy.de wrote:
 Hello again,

 since I haven't got any replies yet I'm trying to make myself a bit more
 precise now. I consider the behaviour described in my original posting a
 bug. I posted to this list because the setuptools docs say Please use the
 distutils-sig mailing list [3] for questions and discussion about

 Test 3)
 DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project
 Result: all modules can be imported

 This is correct AFAICT.

 the setuptools namespace_packages thing predates PEP-420, and because
 PEP-420 namespaces don't interoperate with .pth file based packages
 (expecially when you get into interactions between system non-PEP-420
 + virtualenv PEP-420 packages!) changing this is super hard: you'll
 guarantee to break many existing installs.

 Perhaps there should be a new keyword, but since nothing is needed to
 make things work, it seems like it would be rather redundant.

 You can make use of the namespace_packages keyword argument to setup()
 optional depending on which Python version is running it.

 I guess that's the only way forward unless you want to break
 the package for previous Python versions.

 However, doing so may be hard for namespaces which are used
 by a lot of packages.

 Perhaps setuptools could simply ignore the keyword for
 Python 3.3+ and then rely on PEP 420 to get things working
 in more or less the same way:

 https://www.python.org/dev/peps/pep-0420/
 
 I would be fine with declaring namespace_packages conditionally. But doesn't 
 this affect sdist in
 another way than install (or pip install)? If an sdist intended for use with 
 Python  3.3 is created
 with Python = 3.3, the included metadata (egg-info) would look different (I 
 don't know if pip
 relies on egg-info or setup.py).

The egg-info generated by setup.py at install time :-)

 That would apply also if namespace_packages would be ignored
 automatically for Python = 3.3 as you proposed.
 
 As a consequence, two distributions were neccessary. One with 
 namespace_packages declared and
 containing an __init__.py (with pkg_resources.declare_namespace) and another 
 one without those
 additions. But how does setuptools figure out to leave out the __init__.py 
 for non-declard namespace
 packages in the latter case?

Like I mentioned above: it's probably better for setuptools to handle
this in a consistent way rather than changing all setup.pys.

One detail I'm not sure about is how compatible the two namespace
package techniques really are. The PEP 420 hand waves over possible
differences and only addresses pkgutil, not the setuptools
approach, which is by far more common. For most simply applications
not relying on any of the advanced features, they will most likely
be compatible.

I guess some experiments are necessary to figure that out.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

2015-04-22 Thread M.-A. Lemburg

On 22.04.2015 21:08, Robert Collins wrote:
 On 23 April 2015 at 03:09, M.-A. Lemburg m...@egenix.com wrote:
 [adding list back on CC]

 On 22.04.2015 16:11, Christoph Schmitt wrote:
 Am 2015-04-22 12:59, schrieb M.-A. Lemburg:
 On 22.04.2015 12:38, Robert Collins wrote:
 
 
 That would apply also if namespace_packages would be ignored
 automatically for Python = 3.3 as you proposed.

 As a consequence, two distributions were neccessary. One with 
 namespace_packages declared and
 containing an __init__.py (with pkg_resources.declare_namespace) and 
 another one without those
 additions. But how does setuptools figure out to leave out the __init__.py 
 for non-declard namespace
 packages in the latter case?

 Like I mentioned above: it's probably better for setuptools to handle
 this in a consistent way rather than changing all setup.pys.
 
 I agree, but consider this situation - on any PEP-420 supporting python
 
 Two packages: name.A and name.B. name.A already installed on the
 machine systemwide using old-style namespace path hacks, and then do a
 wheel install of name.B.
 
 If the wheel for name.B was built expecting PEP-420, it won't be
 importable after install (because the path manipulation that sets up
 name as an old-style namespace happens in site.py).
 
 If the wheel was built expecting old-style namespaces, but A was
 installed using PEP-420, then A won't be installable after B is
 installed (same reason).
 
 The point of splitting the place the two are installed is to show that
 the user may not be able to fix the existing install.
 
 So - pip would have to a) detect both styles of package, b)
 automatically install all installed-but-wrong-style versions to match
 the site installed ones. And if any of the packages in the namespace
 only support legacy, everything would be clamped down to legacy.

I don't think support mixed setups is really a practical option.

Either the namespace package is legacy all the way, or it
isn't and uses PEP 420.

Wouldn't it be possible for setuptools or pip to work this out
depending on the Python version ?

Python  3.3: use legacy system for everything
Python = 3.3: use PEP 420 for everything (and remove __init__.py
  files at install time as necessary)

 One detail I'm not sure about is how compatible the two namespace
 package techniques really are. The PEP 420 hand waves over possible
 differences and only addresses pkgutil, not the setuptools
 approach, which is by far more common. For most simply applications
 not relying on any of the advanced features, they will most likely
 be compatible.
 
 They are entirely incompatible.

Oh, I meant from a functionality point of view, not the
technology side.

They both allow installing packages in different directory
trees and that's the only feature most namespace packages
use.

 I guess some experiments are necessary to figure that out.
 
 I spent a week last year debugging issues within openstack with
 namespace packages, local tree imports of same, and both pure venv and
 split system and venv environments.
 
 Some key interesting things:
  - the setuptools pth files aren't fully venv aware today -
 potentially fixable but the resulting pth file is fugly, so I decided
 not worth it.
  - local tree imports work a heck of a lot better in tox etc with
 PEP-420 namespaces
  - PEP-420 namespaces can work on older pythons with importlib, but
  - PEP-420 and legacy packages being mixed for one namespace doesn't
 work at all today - in principle fixable via changes to both
 setuptools and importlib - but it was about here that the other
 openstack folk said 'ok wow, lets just stop using namespace packages
 :)

It's certainly easier to not use namespace packages and simply
install packages into the same tree. The main reason for
namespace packages in setuptools was the desire to stuff everything
into ZIP files (the eggs) or egg directories, even when installed,
to simplify uninstalls.

As soon as you drop that requirement you can have the package manager
deal with the complexities of having multiple packages share the
same directory in site-packages. That's solvable as pip, RPM,
apt, et al. show :-)

But ok, that doesn't solve the issue of support namespace
packages if the developers want to use them ;-)

 I think its a -lot- easier to reason about these things as two
 entirely separate features.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

2015-04-22 Thread M.-A. Lemburg

On 22.04.2015 12:38, Robert Collins wrote:
 On 22 April 2015 at 22:13, Christoph Schmitt dev-maili...@gongy.de wrote:
 Hello again,

 since I haven't got any replies yet I'm trying to make myself a bit more
 precise now. I consider the behaviour described in my original posting a
 bug. I posted to this list because the setuptools docs say Please use the
 distutils-sig mailing list [3] for questions and discussion about
 
 Test 3)
 DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project
 Result: all modules can be imported
 
 This is correct AFAICT.
 
 the setuptools namespace_packages thing predates PEP-420, and because
 PEP-420 namespaces don't interoperate with .pth file based packages
 (expecially when you get into interactions between system non-PEP-420
 + virtualenv PEP-420 packages!) changing this is super hard: you'll
 guarantee to break many existing installs.
 
 Perhaps there should be a new keyword, but since nothing is needed to
 make things work, it seems like it would be rather redundant.

You can make use of the namespace_packages keyword argument to setup()
optional depending on which Python version is running it.

I guess that's the only way forward unless you want to break
the package for previous Python versions.

However, doing so may be hard for namespaces which are used
by a lot of packages.

Perhaps setuptools could simply ignore the keyword for
Python 3.3+ and then rely on PEP 420 to get things working
in more or less the same way:

https://www.python.org/dev/peps/pep-0420/

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 440 on PyPI

2015-01-25 Thread M.-A. Lemburg

On 25.01.2015 17:46, Ian Cordasco wrote:
 On Sun, Jan 25, 2015 at 10:43 AM, M.-A. Lemburg m...@egenix.com wrote:
 On 25.01.2015 16:34, Donald Stufft wrote:

 On Jan 25, 2015, at 9:32 AM, M.-A. Lemburg m...@egenix.com wrote:

 I think you ought to make a more prominent announcement on
 c.l.p, c.l.p.a and perhaps a distutils blog (if there is one).

 What is c.l.p and c.l.p.a?

 That's short for comp.lang.python and comp.lang.python.announce,
 which are gateway'ed to mailing lists:

 https://mail.python.org/mailman/listinfo/python-list
 https://mail.python.org/mailman/listinfo/python-announce-list

 There is no distutils blog.

 Given how quickly things change, I think this would be a good
 way of getting the word out to a wider audience than the
 distutils mailing list.
 
 I've only been on this list for about a year. This PEP (and others
 like it) has been in motion for quite a while. I think the blog would
 miss far more people than the mailing list would and I'm not sure I
 agree with how quickly things change. There doesn't seem to be much
 content for the blog and it seems like something that would just
 become neglected. What topics do you really think would be better
 suited for a blog post than a message to here and related announcement
 lists?

The blog posts could automatically get copied over to
Twitter and mailing lists using e.g. IFTTT, and it would
be possible to subscribe using RSS/Atom feeds.

The advantage of a blog is having news entries persist and be
easy to find, while at the same time simplifying the whole
publishing process.

Anyway, just a suggestion.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 25 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 440 on PyPI

2015-01-25 Thread M.-A. Lemburg

I think you ought to make a more prominent announcement on
c.l.p, c.l.p.a and perhaps a distutils blog (if there is one).

On 24.01.2015 19:40, Donald Stufft wrote:
 I've just started enforcing parts of PEP 440 on PyPI.
 
 In particular:
 
 * All new versions must be a valid PEP 440 version (including normalization).
 * All new versions must not have leading or trailing white space.
 * All new versions must not have a local version.
 * Going forward sort order will be determined by PEP 440 sorting.
 
 ---
 Donald Stufft
 PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
 
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 25 2015)
 Python Projects, Coaching and Consulting ...  http://www.egenix.com/
 mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP425 - Wheel Binary Package Compatibility

2014-10-28 Thread M.-A. Lemburg

On 28.10.2014 11:13, Matthias Hafner wrote:
 Hi there,
 
 Following up on
 https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility.
 
 How should we deal with incompatibility of dynamically linked libraries?
 Doing pip wheel numpy on a Linux 64bit machine results in , which is
 linked dynamically against the GLIBC version installed on the build
 machine.
 
 So should the wheel be shipped with GLIBC, or the GLIBC version be
 specified in the wheel name (means to build a new wheel for each GLIBC
 version?). Also, maybe this is not the only library it is dynamically
 linked against?
 
 Why does that work on MacOS, btw? Are all library versions fixed there for
 one version of OSX?
 
 Thanks for putting some light into this issue.

Since OSes typically ship with older libc versions, your best bet
is to build the package on a OS release that's a few years older
than the latest version, e.g. for Ubuntu you'd use 12.04 or even 10.04
instead of 14.04.

On Linux, you can check the min required GLIBC version by looking
at the nm output of the binaries. They will have a @@GLIBC_x.x.x
modifier attached to the symbols loaded from the GLIBC. The
platform module has a helper function which does this for you:
https://docs.python.org/2.7/library/platform.html#platform.libc_ver

That way you can avoid many incompatibilities with libc versions.

This works on Mac OS X and other Unix-based systems as well.

You may run into library version info problems, though:
http://stackoverflow.com/questions/137773/what-does-the-no-version-information-available-error-from-linux-dynamic-linker/156387#156387
A way around this is to build on systems that don't yet include
this version information.

The alternative is static linking, but this is often not possible
or desired.

On Windows you have to use the libc versions that were used to
compile Python itself, so things are easier.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 28 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-10-24: Released eGenix pyOpenSSL 0.13.5 ...  http://egenix.com/go63

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] cdecimal licensing/hosting (was: some questions about PEP470)

2014-10-14 Thread M.-A. Lemburg

Gentlemen,

could you please stop this and show some more respect in these
discussions ?

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/

On 12.10.2014 21:26, Ian Cordasco wrote:
 On Sun, Oct 12, 2014 at 1:44 PM, Alex Gaynor alex.gay...@gmail.com wrote:
 Stefan Krah stefankrah at freenet.de writes:



 (for example right now bytereef.org is down, so
 we’d not discover any files there).

 Indeed.  It was up reliably since 2005, down for maintenance on
 September 23rd (before ShellShock ...).  Then I discovered that
 someone had put up m3-cdecimal on PyPI (presumably abusing PyPI
 as their private repo --- there are several m3-* packages now).

 This triggered some reflection on whether I would make a significant
 effort in the future to keep things running smoothly for an open source
 community where authors are largely viewed as expendable.

 I don't know what it means for authors to be largely viewed as expendable,
 but half the point of hosting things on PyPI is that you *don't* need to do 
 any
 work at all as an author for reliable delivery of your package.


 Subsequently the downtime (again, the first one since 2005) was picked
 up for propagandistic purposes on Twitter and Reddit.

 Ok, but you seem to be doing the other side's propaganda. Every single person
 I've spoken to agrees that this just underscores the need to encourage 
 packages
 to be on PyPI.


 Last year I would have felt an obligation to minimize the downtime
 to an hour at most.  I no longer feel any such obligations and I'll
 do it when I have time.


 Ok. The PyPI administrators still feel an obligation to their users, so I'll
 prefer packages under their care.

 Stefan Krah


 Cheers,
 Alex
 
 Perhaps Stefan's referring to my tweets about the inability to reach
 bytereef but those weren't propaganda tweets. Those were tweets born
 out of utter frustration. Further, I'm rather shocked that you've
 decided to allow the site to remain unreachable because someone did
 what your license allowed them to do (redistribute the software while
 retaining the required information: copyright, license, etc). If you
 think that makes you expendable, you're half right. Users can
 redistribute your software, that's the nature of the license you chose
 to use. You're wrong because you, the author, are still very valuable
 to those very users who may encounter a bug in the future. I don't see
 how intentionally keeping your site unreachable does anything but hurt
 your users (unless of course you want them to redistribute it
 themselves or switch to Python 3.4).
 
 Does this mean that companies using devpi to keep an internal index
 that also have copies of cdecimal are somehow violating your rights?
 They're doing exactly what your license allows them to do. Or is it
 just that some group has decided to redistribute it directly through
 PyPI? I'm thoroughly confused here.
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP470 installation security problems

On 08.10.2014 14:30, Nick Coghlan wrote:
 On 8 October 2014 22:22, Donald Stufft don...@stufft.io wrote:

 On Oct 8, 2014, at 8:17 AM, holger krekel hol...@merlinux.eu wrote:

 Also, i am worried on principle grounds if pip maintainers are putting
 themselves outside PEP reach, yet pip is distributed along with Python.

 We’re not “putting ourselves outside of PEP reach”. We are an external
 project and we are not bound by the PEP process. Devpi, py.test, Django,
 requests, etc are also not bound by the PEP process.
 
 Note also that even for CPython itself, it is *up to us as core
 developers* to decide when something needs to be escalated through the
 PEP process. The vast majority of CPython changes are handled directly
 through the issue tracker, and there's still the occasional change
 that doesn't even make it that far (e.g. if we notice a problem while
 working on something else, we have the option of just committing the
 fix directly).
 
 PEPs are primarily for changes which have broad ecosystem implications
 where the additional overhead is justified. We don't write PEPs for
 every change to the CPython command line interface (e.g. there's no
 PEP for isolated mode), and the same kind of assessment of external
 impact applies to pip and the PyPA in general when decided whether a
 change can be handled within the scope of an individual project, or if
 it needs to be escalated for broader discussion.

I don't follow Donald's reasoning and I'm not sure I understand
whether your comments are meant as clarification of pip being
subject to the PEP process or support for Donald's reasoning :-)

Changes to pip and PyPI *do* have a global effect on the Python
ecosystem and thus need to be covered by the PEP process.

If pip decides to go with a strategy that ignores this, I think we
have a problem. The core developers put trust into pip when allowing
it to (effectively) get distributed with Python and making it the
default Python packaging manager. Please use that trust with the
appropriate care and respect.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP470 installation security problems

On 08.10.2014 15:05, Donald Stufft wrote:
 
 On Oct 8, 2014, at 8:55 AM, M.-A. Lemburg m...@egenix.com wrote:

 On 08.10.2014 14:30, Nick Coghlan wrote:
 On 8 October 2014 22:22, Donald Stufft don...@stufft.io wrote:

 On Oct 8, 2014, at 8:17 AM, holger krekel hol...@merlinux.eu wrote:

 Also, i am worried on principle grounds if pip maintainers are putting
 themselves outside PEP reach, yet pip is distributed along with Python.

 We’re not “putting ourselves outside of PEP reach”. We are an external
 project and we are not bound by the PEP process. Devpi, py.test, Django,
 requests, etc are also not bound by the PEP process.

 Note also that even for CPython itself, it is *up to us as core
 developers* to decide when something needs to be escalated through the
 PEP process. The vast majority of CPython changes are handled directly
 through the issue tracker, and there's still the occasional change
 that doesn't even make it that far (e.g. if we notice a problem while
 working on something else, we have the option of just committing the
 fix directly).

 PEPs are primarily for changes which have broad ecosystem implications
 where the additional overhead is justified. We don't write PEPs for
 every change to the CPython command line interface (e.g. there's no
 PEP for isolated mode), and the same kind of assessment of external
 impact applies to pip and the PyPA in general when decided whether a
 change can be handled within the scope of an individual project, or if
 it needs to be escalated for broader discussion.

 I don't follow Donald's reasoning and I'm not sure I understand
 whether your comments are meant as clarification of pip being
 subject to the PEP process or support for Donald's reasoning :-)

 Changes to pip and PyPI *do* have a global effect on the Python
 ecosystem and thus need to be covered by the PEP process.

 If pip decides to go with a strategy that ignores this, I think we
 have a problem. The core developers put trust into pip when allowing
 it to (effectively) get distributed with Python and making it the
 default Python packaging manager. Please use that trust with the
 appropriate care and respect.
 
 I don’t think we’ve *ever* not used that trust with care and respect and
 we’ve been trusted by the Python community for far longer than PEP 453
 has existed. We attempt to follow PEPs where we can and where they make
 good sense. Nobody on the pip team is saying we’re going to flat out
 ignore PEPs or whatever.
 
 We (or at least I am) are saying that dictating UX via PEP process has
 been shown to us *not* to work and that we are not obligated to implement
 or listen to a PEP. This was explicitly spelled out in PEP 453 that we
 remain an external project even with the fact we’re now bundled with
 Python. This does not mean we won’t generally try to use the PEP process
 where our changes have cross cutting concerns between different projects
 but it does mean that we implement or follow PEPs at our discretion. This
 isn’t up for debate, it was an explicit inclusion in PEP 453 and if there
 was a problem with pip maintaining it’s own project the time to bring that
 up was a year ago. 

The intention of PEP 435 was to enable pip to evolve independent
of the Python release process, which is a good thing.

However, your comment that We are an external project and we are not
bound by the PEP process. doesn't really pan out in the light of the PEP's
requirement that The maintainers of the bootstrapped software and the
CPython core team will work together in order to address the needs of both.

If pip maintainers don't feel they are bound by PEPs, you could argue
that you are also not bound by PEP 435, which would result in a
rather pointless cooperation setup :-)

Note that I'm not trying to say that you are actually not respecting the
PEP process. I'm just concerned about comments like the above causing
unnecessary heat in discussions.

I'd also like to request that you take Holger's concerns more
seriously, perhaps add him as PEP author and let him participate
in clarifying it (if he still feels like investing time in this).

PEPs are never perfect and there's always room for improvement.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org

Re: [Distutils] PEP470 installation security problems

On 08.10.2014 15:59, Nick Coghlan wrote:
 On 8 Oct 2014 23:40, M.-A. Lemburg m...@egenix.com wrote:
 The intention of PEP 435 was to enable pip to evolve independent
 of the Python release process, which is a good thing.

 However, your comment that We are an external project and we are not
 bound by the PEP process. doesn't really pan out in the light of the PEP's
 requirement that The maintainers of the bootstrapped software and the
 CPython core team will work together in order to address the needs of
 both.

 If pip maintainers don't feel they are bound by PEPs, you could argue
 that you are also not bound by PEP 435, which would result in a
 rather pointless cooperation setup :-)

 Note that I'm not trying to say that you are actually not respecting the
 PEP process. I'm just concerned about comments like the above causing
 unnecessary heat in discussions.
 
 pip's UX decisions aren't likely to ever be put through the PEP process
 again - the PEP 426 (and now PEP 470) model of providing functional
 requirements and recommendations in the form of MUST and SHOULD statements
 is a cleaner process, since they provide guidance for all clients, not just
 pip, and leave the *details* of the UX to the normal pip development cycle
 (so if user feedback indicates a problem with the specifics of the initial
 approach, they can address that while remaining compliant with the
 specification).
 
 Decoupling functional specifications from UX details of individual tools is
 a good idea in general, this is just applying that model to pip and the PEP
 process in particular.

IMO, specific user interface questions are PEP relevant if they affect
the way users interact with the Python ecosystem.

This doesn't mean mandating specific option names, but e.g.
--using-silly-long-options-that-scare-away-users does have PEP relevance.
A PEP would have to address such user interface designs by defining whether
or not to encourage or discourage certain uses.

And, of course, pip as officially sanctioned Python installer
would need to implement these requirements.

 PyPI needs to be covered in more detail, however, as these PEPs also serve
 as the *interface* specification for both clients and servers, and those
 need concrete API definitions to work with.
 
 PEP 438 was the main case so far where the PEP included specific UX design
 details for pip, and that's the aspect that *won't* be repeated.

PEPs are not set in stone. They can be updated and replaced with
new ones. That's why they are called
Python Enhancement *Proposals* :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP470 installation security problems

On 08.10.2014 15:15, Paul Moore wrote:
 On 8 October 2014 13:55, M.-A. Lemburg m...@egenix.com wrote:
 If pip decides to go with a strategy that ignores this, I think we
 have a problem. The core developers put trust into pip when allowing
 it to (effectively) get distributed with Python and making it the
 default Python packaging manager. Please use that trust with the
 appropriate care and respect.
 
 Just to clarify - the pip team (I hope I speak for all of us) fully
 understand the implications of being the de facto standard package
 manager. And we appreciate the trust placed in us by the fact that pip
 is distributed with Python. But at the same time, that trust was given
 on the basis that (presumably) we have a track record of doing things
 right, in an area that is notoriously full of heated discussions and
 conflicting opinions. So what we'd like to do is to continue handling
 things in the same way as always, working with the packaging
 community.
 
 In particular, that means that we did not align ourselves to the
 CPython development model (as it is designed for a very different
 community and set of problems). But we do want to adopt their good
 practices where possible and appropriate. One of those is the PEP
 process - but it's not entirely suitable (see the trail of PEPs from
 the distribute/packaging/distutils2 era, for why). So we're trying to
 get things right, and in the process we're learning - for example, the
 failure of PEP 438 taught us that specifying installer behaviour too
 closely in a PEP means we can't fix problems that are completely
 messing up our users. But we still believe in the PEP process (anyone
 who thinks otherwise hasn't noticed the amount of effort Donald, in
 particular, is putting into all the PEPs in progress). It doesn't mean
 that it can be treated as a way of forcing us not to do what we think
 is right for the pip user base, though.

Thanks for your clarification, Paul.

I just want to remind everyone that PEPs can be augments and mistakes
can be fixed by superseding one PEP with another. It's a well
working process, one that is accepted in Python land and in line
with the core development process.

Since pip now is part of the Python stdlib (even though not bound
by its release process), and the pip user base is identical with
the CPython user base, the PEP process also applies to pip.

That's the consequence of playing the role of an officially
sanctioned part of the ecosystem and comes as part of the
responsibility resulting from PEP 435.

So far this has worked out well, which is why I'm surprised
by some statements in this discussion.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP470 installation security problems

On 08.10.2014 16:04, Donald Stufft wrote:
 
 I'd also like to request that you take Holger's concerns more
 seriously, perhaps add him as PEP author and let him participate
 in clarifying it (if he still feels like investing time in this).
 
 I take all concerns and feedback seriously else I wouldn’t spend the many
 hours I’ve spent just this morning responding to them. I don’t grok what
 Holger’s actual concern is so it’s hard to turn those concerns into anything
 actionable I can actually do on the PEP.

Holger has made his points very clear in his emails.

If you don't follow/grok his reasoning it may indeed be better to
have him edit the PEP to add his improvements/changes.

I share his view that it is not necessary to break existing
setups to add multi-index support. This can be implemented as
simple extension to what we already have:


Simply add the possibility for authors to register external indexes,
have pip, setuptools, et al. crawl these in addition to what's
up on the PyPI package page (using the logic that has existed in
these tools for years) and then let the author decide whether they
want to remove existing downloads from PyPI or not.

This allows for older installations to continue working, while
also (optionally) supporting a setup which does not use PyPI for
hosting at all.


BTW: For eGenix we've chosen to use a different approach, one
that is based on a Python web installer. I gave a talk about this at
PyCon UK, in case you're interested:
https://downloads.egenix.com/python/PyCon-UK-2014-Python-Web-Installer-Talk.pdf
(talk video here:
http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/)
This solves the issues with the pip user experience for our packages,
solves the download selection issues for the binaries, works with
all Python versions we support and assures that the downloads
are safe. It's still work in progress, but already quite usable.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting

2014-10-07 Thread M.-A. Lemburg

FWIW: I support Holger's request to introduce multi repository
support *without* breaking existing setups.

Simply add the possibility for authors to register external indexes,
have pip, setuptools, et al. crawl these in addition to what's
up on the PyPI package page (using the logic that has existed in
these tools for years) and then let the author decide whether they
want to remove existing downloads from PyPI or not.

This allows for older installations to continue working, while
also (optionally) supporting a setup which does not use PyPI for
hosting at all.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

2014-09-30 Thread M.-A. Lemburg

Thanks for the confirmation that my interpretation was wrong. This
makes things a lot better :-)

More below...

On 29.09.2014 11:39, Nick Coghlan wrote:
 On 29 Sep 2014 18:49, M.-A. Lemburg m...@egenix.com wrote:

 You are missing out on cases, where the release process causes files to
 be omitted, human errors where packagers forget to apply changes to
 e.g. documentation files, version files, change logs, etc., where
 packagers want to add information that doesn't affect the software
 itself, but meta information included in the distribution files.

 Such changes often do not affect the software itself, and so are not
 detected by software tests.
 
 Fixing such packaging errors is the primary intended use case of the post
 field in PEP 440. Alternatively, many projects will just spin a new release
 that addresses those issues, just as they would for any other bug.
 
 Both of those approaches have the advantage of letting users (especially
 those operating mirrors, or downloading tarballs and feeding them into a
 separate redistribution system) easily tell whether or not they have the
 fixed version.

I don't see how that would help. AFAIU, PyPI would regard a 3.1.4.post1
as new release and so invalidate all other already uploaded distribution
files rather than just regard the fixed upload as update
to the 3.1.4 release.

If we could get a widely adopted notion of build numbers into the
tools that would help a lot.

Installers and PyPI would then regard 3.1.4-1 as belonging to
release 3.1.4, but being a more current build as a distribution
file carrying 3.1.4 in its file name.

This would also have to work for all files uploaded for a release,
not only for some distribution formats, of course, including source
release files, Windows installers, egg files, etc.

I'd have to run some experiments, but don't believe that setuptools
and associated tools support this at the moment.

 If I understand you correctly, you are essentially suggesting that it
 becomes impossible to ever delete anything uploaded to PyPI, i.e.
 turning PyPI into a WORM.
 
 No, deletion remains supported. The only capability being removed is silent
 substitution of hosted files with different ones bearing the same name.
 
 So if, for example, release 3.1.4 had a packaging error, then deleting it
 would still be possible, but the replacement would need to be something
 like 3.1.4.post1 or 3.1.5, rather than being permitted to reuse the
 3.1.4 name.

So just to summarize: the proposal is to turn PyPI into a WORM,
but at least it's still possible to remove distribution files.

 The external hosting support is then the mechanism by which authors can
 retain complete and total control over their release process. That approach
 avoids all licensing concerns (including those related to US export
 controls), as well as ensuring they have the ability to silently change the
 contents of previously released files if they choose to do so (although, as
 noted above, actually doing so may break installation for anyone installing
 with peep, which checks file hashes based on the first version downloaded).

You're regularly bringing up this argument.

Let's just be fair here: external hosting of packages has been made so
user unfriendly in recent pip releases, that this has pretty much
become a non-option for anyone who wants to create a user friendly
package installation environment.

pip is unfortunately using the same kind of
--no-one-will-want-to-use-this-option-because-its-too-long
approach as setuptools/easy_install has done in the past to force
people into installing packages as eggs rather than installing
the packages in the standard write to site-packages dir way.

The end result is the same: users will not want to go
through those extra hoops and thus packages not hosted on PyPI
itself will be regarded as broken (because they don't install using
the standard method; not because they are really broken).

This is what I'm trying to address in discussions like these all
along. PyPI has a responsibility not only for the part consuming
part of the Python community, but also for the creating part of it.

While PyPI is great for indexing packages, it's not necessarily
the best answer for hosting the distribution files and I believe
we should open up some more to allow for making it possible to
offer the same kind of user experience while not making pypi.python.org
the only source of distribution files.

In the Linux world, this already works great by having multiple
repos which you can switch on/off easily.

For eGenix, we've found a way to work around the PyPI constraints:

http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/

which addresses our user's problems. Still, we'd much rather use
standard ways of working *with* PyPI rather than work around it.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 30 2014)
 Python Projects, Consulting and Support ...   http

Re: [Distutils] Immutable Files on PyPI

2014-09-29 Thread M.-A. Lemburg

On 28.09.2014 23:59, Donald Stufft wrote:
 
 On Sep 28, 2014, at 5:36 PM, M.-A. Lemburg m...@egenix.com 
 mailto:m...@egenix.com wrote:

 On 28.09.2014 21:31, Donald Stufft wrote:
 Hello All!

 I'd like to discuss the idea of moving PyPI to having immutable files. This
 would mean that once you publish a particular file you can never reupload 
 that
 file again with different contents. This would still allow deleting the 
 file or
 reuploading it if the checksums match what was there prior.

 This would be good for a few reasons:

 * It represents best practices for version numbers. Ideally if two people
  have version 2.1 of a project, they'll have the same code, however as it
  stands two people installing at two different times could have two very
  different versions.

 * This will make improving the PyPI infrastructure easier, in particular it
  will make it simpler to move away from using a glusterfs storage array and
  switch to a redudant set of cloud object stores.


 In the past this was brought up and a few points were brought against it, 
 those
 were:

 1. That authors could simply change files that were hosted on not PyPI 
 anyways
   so it didn't really do much.

 2. That it was too hard to test a release prior to uploading it due to the
   nature of distutils requiring you to build the release in the same command
   as the upload.

 With the fact that pip no longer hits external URLs by default, I believe 
 that
 the first item is no longer that large of a factor. People can do whatever 
 they
 want on external URLs of course, however if something is coming from PyPI as
 end users should now be aware of, they can know it is immutable.

 Now that there is twine, which allows uploading already created packages, I
 also believe that the second item is no longer a concern. People can easily
 create a distribution using ``setup.py sdist``, test it, and then upload 
 that
 exact thing they tested using ``twine upload path to sdist``.

 -1.

 It does happen that files need to be reuploaded because of a bug
 in the release process and how people manage their code is really
 *their* business, not that of PyPI.
 
 Can you describe a reasonable hypothetical situation where this would occur
 often enough as to be something that is likely to happen on a consistent
 basis? Originally the problem was there was little ability to easily upload
 pre-created files so there was a reasonable chance that there may be a
 packaging bug that didn’t get exposed until you actually packaged + released.

 With the advent of twine though it’s now possible to test the exact bits that
 get uploaded to PyPI making that particular issue no longer a problem.

 However, the fact that the files are not immutable *do* cause a number of
 problems that need to be worked around in the mirroring infrastructure, the
 CDN, and for scaling PyPI out and removing the glusterfs component.

You are missing out on cases, where the release process causes files to
be omitted, human errors where packagers forget to apply changes to
e.g. documentation files, version files, change logs, etc., where
packagers want to add information that doesn't affect the software
itself, but meta information included in the distribution files.

Such changes often do not affect the software itself, and so are not
detected by software tests.

If I understand you correctly, you are essentially suggesting that it
becomes impossible to ever delete anything uploaded to PyPI, i.e.
turning PyPI into a WORM.

This would mean that package authors could never correct mistakes,
remove broken packages distribution files, ones which they may be
forced to remove for legal reasons, ones which they find are infected
with a virus or trojan, ones which they uploaded for fun or
by mistake.

This doesn't have anything to do with making the user experience
a better one. It is ignorant to assume that package authors who
sometimes delete distribution files, or at least want to have the
possibility to do so, don't care for their users. We are in
Python land, so most authors will know what they are doing and
do care for their users.

After all: Why do you think I'm arguing against this proposal ?
Because I want users of our packages to get the best experience
they can get, by downloading complete, correct and working
distribution files.

This whole idea also has another angle, namely a legal one:
the PSF doesn't own the distribution files it hosts on PyPI.

So far, the argument to not fix the much too broad license on PyPI
was that authors were able to delete files on PyPI to work around
the unneeded irrevocable part of that license.

With the suggested change, authors would have to give up complete
control over their distribution files to the PSF in order for their
packages to be installable by pip using its default settings.

This kind of lock-in and removal of author rights is not something
I can support as PSF director. Those authors are the ones that have
created a large part

Re: [Distutils] Immutable Files on PyPI

2014-09-29 Thread M.-A. Lemburg

On 29.09.2014 00:51, Nick Coghlan wrote:
 On 29 Sep 2014 07:37, M.-A. Lemburg m...@egenix.com wrote:

 -1.

 It does happen that files need to be reuploaded because of a bug
 in the release process and how people manage their code is really
 *their* business, not that of PyPI.

 FWIW, I am getting increasingly annoyed how PyPI and pip try to dictate
 the way package authors are supposed to build, manage and host their
 Python packages and release process. Can we please stop this ?
 
 As others have noted, these changes represent the PyPA being opinionated on
 behalf of the user community, to provide the best possible user experience
 for the overall Python ecosystem.

See my reply to Donald. I find this wrong on several different levels.

PyPI is run by the PSF, it's a community resource we provide for
package authors and downloaders. We (the PSF) don't take sides.
Instead, we want to help everyone feel at home: the package authors who
provide the Python eco system with fresh software, as well as the
users who greatly benefit from this software.

The PyPA takes care of the technical aspects of this, but not
the ethical and community building aspects.

 We'll accommodate the existing publisher community as far as is feasible
 (that's why PEP 440 is as complicated as it is, for example), but there are
 going to be times where improving the end user experience means adding new
 constraints on publishers.
 
 External hosting (using PyPI as an index, without also using it for release
 file hosting) is the primary escape clause for software publishers that
 prefer to do things differently from the way PyPI does them. That's a user
 experience we'll also continue to work to improve, to ensure it is clear
 that it's a fully supported part of the distribution model.

Right, so authors will move away from PyPI and put their stuff
up elsewhere. Now, how does this help our community ?

What if people find that they can only get packages using
conda instead of pip, or only by cloning from github, because
package authors don't want to bother cutting distribution
files anymore ?

Do you seriously want to force package authors to cut a new release
just because a single uploaded distribution file is broken for
some reason and then ask all users who have already installed one
of the non-broken ones to upgrade again, even though they are not
affected ?

Please repeat with me: Package authors care for their users :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 29 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-09-30: Python Meeting Duesseldorf ...  tomorrow

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Immutable Files on PyPI

2014-09-28 Thread M.-A. Lemburg

On 28.09.2014 21:31, Donald Stufft wrote:
 Hello All!
 
 I'd like to discuss the idea of moving PyPI to having immutable files. This
 would mean that once you publish a particular file you can never reupload that
 file again with different contents. This would still allow deleting the file 
 or
 reuploading it if the checksums match what was there prior.
 
 This would be good for a few reasons:
 
 * It represents best practices for version numbers. Ideally if two people
   have version 2.1 of a project, they'll have the same code, however as it
   stands two people installing at two different times could have two very
   different versions.
 
 * This will make improving the PyPI infrastructure easier, in particular it
   will make it simpler to move away from using a glusterfs storage array and
   switch to a redudant set of cloud object stores.
 
 
 In the past this was brought up and a few points were brought against it, 
 those
 were:
 
 1. That authors could simply change files that were hosted on not PyPI anyways
so it didn't really do much.
 
 2. That it was too hard to test a release prior to uploading it due to the
nature of distutils requiring you to build the release in the same command
as the upload.
 
 With the fact that pip no longer hits external URLs by default, I believe that
 the first item is no longer that large of a factor. People can do whatever 
 they
 want on external URLs of course, however if something is coming from PyPI as
 end users should now be aware of, they can know it is immutable.
 
 Now that there is twine, which allows uploading already created packages, I
 also believe that the second item is no longer a concern. People can easily
 create a distribution using ``setup.py sdist``, test it, and then upload that
 exact thing they tested using ``twine upload path to sdist``.

-1.

It does happen that files need to be reuploaded because of a bug
in the release process and how people manage their code is really
*their* business, not that of PyPI.

FWIW, I am getting increasingly annoyed how PyPI and pip try to dictate
the way package authors are supposed to build, manage and host their
Python packages and release process. Can we please stop this ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 28 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-09-30: Python Meeting Duesseldorf ...  2 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP draft on PyPI/pip package signing

2014-07-28 Thread M.-A. Lemburg

Hi Giovanni,

this sounds like a good proposal. I only have one nit with it:
GPG signing should not be made mandatory for package authors,
but instead just be encouraged.

Not everyone is happy with the way GPG works, or want to maintain
their private keys and it's illegal to use / can cause problems in
some countries:

http://www.cryptolaw.org/cls-sum.htm

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 28 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


On 28.07.2014 17:01, Giovanni Bajo wrote:
 Hello,
 
 on March 2013, on the now-closed catalog-sig mailing-list, I submitted a 
 proposal for fixing several security problems in PyPI, pip and distutils[1]. 
 Some of my proposals were obvious things like downloading packages through 
 SSL, which was already in progress of being designed and implemented. Others, 
 like GPG package signing, were discussed for several days/weeks, but ended up 
 in discussion paralysis because of the upcoming TUF framework.
 
 16 months later, we still don’t have a deployed solution for letting people 
 install signed packages. I see that TUF is evolving, and there is now a 
 GitHub project with documentation, but I am very worried about the 
 implementation timeline.
 
 I was also pointed to PEP458, which I tried to read and found it very 
 confusing; the PEP assumes that the reader must be familiar with the TUF 
 academic paper (which I always found quite convoluted per-se), and goes with 
 an analysis of integration of TUF with PyPI; to the best of my understanding, 
 the PEP does not provide a clear answer to practical questions like: 
 
  * what a maintainer is supposed to do to submit a new signed package
  * how can differ maintainers signal that they both maintain the same package
  * how the user interface of PyPI will change
  * what are the required security maintenance that will need to be regularly 
 performed by the PyPI ops
 
 I’m not saying that the TUF team has no answers to these questions (in fact, 
 I’m 100% sure of the opposite); I’m saying that the PEP doesn’t clearly 
 provide such answers. I think the PEP is very complicated to read as it goes 
 into integration details between the TUF architecture and PyPI, and thus it 
 is very complicated to review and accept. I would love the PEP to be updated 
 to provide an overview on the *practical* effects of the integration of TUF 
 within PyPI/pip, that must be fully readable to somebody with zero previous 
 knowledge of TUF.
 
 As suggested by Richard Jones during EuroPython, I isolated the package 
 signing sections from my original document, evolved them a little bit, and 
 rewritten them in PEP format:
 https://gist.github.com/rasky/bd91cf01f72bcc931000
 
 To the best of my recollection, in the previous review round, there were no 
 critical issues found in the design. It might well be that TUF provides more 
 security in some of the described attack scenarios; on the other hand, my 
 proposal:
 
   * is in line with the security of (e.g..) existing Linux distros
   * is very simple to review, analyze and discuss for anybody with even a 
 basic understanding of security
   * is much simpler than TUF
   * is a clear step forward from the current situation
   * cover areas not covered by PEP458 (e.g.: increasing security of account 
 management on PyPI)
   * can be executed in 2-3 months (to the alpha / pre-review stage), and I 
 volunteer for the execution.
 
 I thus solicit a second round of review of my proposal; if you want me to 
 upload to Google Docs for easier of commenting, I can do that as well. I 
 would love to get the PEP to its final form and then ask for a pronouncement.
 
 I apologize in advance if I made technical mistakes in the PEP 
 format/structure; it is my first PEP.
 
 [1] See here: 
 https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit#
  
 
 
 
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
 


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PyPI XMLRPC interface no longer works with Python 2.6

I just tried the documented interfaces for PyPI
(https://wiki.python.org/moin/PyPIXmlRpc) with Python 2.6 and
it fails with an error:

 python pypirpc.py
Traceback (most recent call last):
  File pypirpc.py, line 7, in module
pprint.pprint(client.package_releases('egenix-web-installer-test'))
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1199, in 
__call__
return self.__send(self.__name, args)
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1489, in 
__request
verbose=self.__verbose
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1253, in 
request
return self._parse_response(h.getfile(), sock)
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1390, in 
_parse_response
p.close()
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 604, in 
close
self._parser.Parse(, 1) # end of data
xml.parsers.expat.ExpatError: no element found: line 1, column 0

The call works with Python 2.7. It appears that xmlrpclib
is not receiving any body data from the server.

The returned data in 2.7 looks completely harmless:

send: POST /pypi HTTP/1.1\r\nHost: pypi.python.org\r\nAccept-Encoding: 
gzip\r\nUser-Agent:
xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: 
text/xml\r\nContent-Length:
185\r\n\r\n?xml
version='1.0'?\nmethodCall\nmethodNamepackage_releases/methodName\nparams\nparam\nvaluestringegenix-web-installer-test/string/value\n/param\n/params\n/methodCall\n
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx/1.6.0
header: Content-Type: text/xml
header: charset: UTF-8
header: Strict-Transport-Security: max-age=31536000; includeSubDomains
header: Transfer-Encoding: chunked
header: Accept-Ranges: bytes
header: Date: Tue, 08 Jul 2014 14:19:41 GMT
header: Via: 1.1 varnish
header: Connection: keep-alive
header: X-Served-By: cache-fra1229-FRA
header: X-Cache: MISS
header: X-Cache-Hits: 0
header: X-Timer: S1404829181.210045,VS0,VE325
body: ?xml
version='1.0'?\nmethodResponse\nparams\nparam\nvaluearraydata\nvaluestring0.2.0/string/value\n/data/array/value\n/param\n/params\n/methodResponse\n

Could this be a network error rather than a program one ?

The code in 2.7 does a retry in case of a connection reset or abort,
which code in 2.6 and earlier does not apply.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-07-21: EuroPython 2014, Berlin, Germany ...   13 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI XMLRPC interface no longer works with Python 2.6

I opened an issue for this:
https://bitbucket.org/pypa/pypi/issue/157/pypi-xmlrpc-interface-no-longer-works-with

On 08.07.2014 16:32, M.-A. Lemburg wrote:
 I just tried the documented interfaces for PyPI
 (https://wiki.python.org/moin/PyPIXmlRpc) with Python 2.6 and
 it fails with an error:
 
 python pypirpc.py
 Traceback (most recent call last):
   File pypirpc.py, line 7, in module
 pprint.pprint(client.package_releases('egenix-web-installer-test'))
   File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1199, in 
 __call__
 return self.__send(self.__name, args)
   File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1489, in 
 __request
 verbose=self.__verbose
   File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1253, in 
 request
 return self._parse_response(h.getfile(), sock)
   File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1390, in 
 _parse_response
 p.close()
   File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 604, in 
 close
 self._parser.Parse(, 1) # end of data
 xml.parsers.expat.ExpatError: no element found: line 1, column 0
 
 The call works with Python 2.7. It appears that xmlrpclib
 is not receiving any body data from the server.
 
 The returned data in 2.7 looks completely harmless:
 
 send: POST /pypi HTTP/1.1\r\nHost: pypi.python.org\r\nAccept-Encoding: 
 gzip\r\nUser-Agent:
 xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: 
 text/xml\r\nContent-Length:
 185\r\n\r\n?xml
 version='1.0'?\nmethodCall\nmethodNamepackage_releases/methodName\nparams\nparam\nvaluestringegenix-web-installer-test/string/value\n/param\n/params\n/methodCall\n
 reply: 'HTTP/1.1 200 OK\r\n'
 header: Server: nginx/1.6.0
 header: Content-Type: text/xml
 header: charset: UTF-8
 header: Strict-Transport-Security: max-age=31536000; includeSubDomains
 header: Transfer-Encoding: chunked
 header: Accept-Ranges: bytes
 header: Date: Tue, 08 Jul 2014 14:19:41 GMT
 header: Via: 1.1 varnish
 header: Connection: keep-alive
 header: X-Served-By: cache-fra1229-FRA
 header: X-Cache: MISS
 header: X-Cache-Hits: 0
 header: X-Timer: S1404829181.210045,VS0,VE325
 body: ?xml
 version='1.0'?\nmethodResponse\nparams\nparam\nvaluearraydata\nvaluestring0.2.0/string/value\n/data/array/value\n/param\n/params\n/methodResponse\n
 
 Could this be a network error rather than a program one ?
 
 The code in 2.7 does a retry in case of a connection reset or abort,
 which code in 2.6 and earlier does not apply.
 
 Thanks,
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-07-21: EuroPython 2014, Berlin, Germany ...   13 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI XMLRPC interface no longer works with Python 2.6

On 08.07.2014 17:09, Donald Stufft wrote:
 You’re using http:// rather than https:// yea?

Yes. xmlrpclib in Python 2.6 doesn't understand HTTPS. Support for
it was added in Python 2.7.

 On Jul 8, 2014, at 10:49 AM, M.-A. Lemburg m...@egenix.com wrote:
 
 I opened an issue for this:
 https://bitbucket.org/pypa/pypi/issue/157/pypi-xmlrpc-interface-no-longer-works-with

 On 08.07.2014 16:32, M.-A. Lemburg wrote:
 I just tried the documented interfaces for PyPI
 (https://wiki.python.org/moin/PyPIXmlRpc) with Python 2.6 and
 it fails with an error:

 python pypirpc.py
 Traceback (most recent call last):
  File pypirpc.py, line 7, in module
pprint.pprint(client.package_releases('egenix-web-installer-test'))
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1199, 
 in __call__
return self.__send(self.__name, args)
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1489, 
 in __request
verbose=self.__verbose
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1253, 
 in request
return self._parse_response(h.getfile(), sock)
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1390, 
 in _parse_response
p.close()
  File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 604, in 
 close
self._parser.Parse(, 1) # end of data
 xml.parsers.expat.ExpatError: no element found: line 1, column 0

 The call works with Python 2.7. It appears that xmlrpclib
 is not receiving any body data from the server.

 The returned data in 2.7 looks completely harmless:

 send: POST /pypi HTTP/1.1\r\nHost: pypi.python.org\r\nAccept-Encoding: 
 gzip\r\nUser-Agent:
 xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: 
 text/xml\r\nContent-Length:
 185\r\n\r\n?xml
 version='1.0'?\nmethodCall\nmethodNamepackage_releases/methodName\nparams\nparam\nvaluestringegenix-web-installer-test/string/value\n/param\n/params\n/methodCall\n
 reply: 'HTTP/1.1 200 OK\r\n'
 header: Server: nginx/1.6.0
 header: Content-Type: text/xml
 header: charset: UTF-8
 header: Strict-Transport-Security: max-age=31536000; includeSubDomains
 header: Transfer-Encoding: chunked
 header: Accept-Ranges: bytes
 header: Date: Tue, 08 Jul 2014 14:19:41 GMT
 header: Via: 1.1 varnish
 header: Connection: keep-alive
 header: X-Served-By: cache-fra1229-FRA
 header: X-Cache: MISS
 header: X-Cache-Hits: 0
 header: X-Timer: S1404829181.210045,VS0,VE325
 body: ?xml
 version='1.0'?\nmethodResponse\nparams\nparam\nvaluearraydata\nvaluestring0.2.0/string/value\n/data/array/value\n/param\n/params\n/methodResponse\n

 Could this be a network error rather than a program one ?

 The code in 2.7 does a retry in case of a connection reset or abort,
 which code in 2.6 and earlier does not apply.

 Thanks,


 -- 
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source  (#1, Jul 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 
 2014-07-21: EuroPython 2014, Berlin, Germany ...   13 days to go

 : Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
 
 
 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-07-21: EuroPython 2014, Berlin, Germany ...   13 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI XMLRPC interface no longer works with Python 2.6

On 08.07.2014 17:15, Donald Stufft wrote:
 Uh what? HTTPS works fine on my copy of 2.6? If I recall this problem
 only happens if you use http:// on 2.6.

Ah, sorry. You're right. I had looked at a diff of the module
between 2.6 and 2.7 and saw lots of HTTPS related changes.

With https:// it does work in 2.6 as well.

Thanks. I'll close the bug report a fix the wiki page.

 On Jul 8, 2014, at 11:13 AM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 08.07.2014 17:09, Donald Stufft wrote:
 You’re using http:// rather than https:// yea?

 Yes. xmlrpclib in Python 2.6 doesn't understand HTTPS. Support for
 it was added in Python 2.7.

 On Jul 8, 2014, at 10:49 AM, M.-A. Lemburg m...@egenix.com wrote:

 I opened an issue for this:
 https://bitbucket.org/pypa/pypi/issue/157/pypi-xmlrpc-interface-no-longer-works-with

 On 08.07.2014 16:32, M.-A. Lemburg wrote:
 I just tried the documented interfaces for PyPI
 (https://wiki.python.org/moin/PyPIXmlRpc) with Python 2.6 and
 it fails with an error:

 python pypirpc.py
 Traceback (most recent call last):
 File pypirpc.py, line 7, in module
   pprint.pprint(client.package_releases('egenix-web-installer-test'))
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1199, 
 in __call__
   return self.__send(self.__name, args)
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1489, 
 in __request
   verbose=self.__verbose
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1253, 
 in request
   return self._parse_response(h.getfile(), sock)
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 1390, 
 in _parse_response
   p.close()
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 604, 
 in close
   self._parser.Parse(, 1) # end of data
 xml.parsers.expat.ExpatError: no element found: line 1, column 0

 The call works with Python 2.7. It appears that xmlrpclib
 is not receiving any body data from the server.

 The returned data in 2.7 looks completely harmless:

 send: POST /pypi HTTP/1.1\r\nHost: pypi.python.org\r\nAccept-Encoding: 
 gzip\r\nUser-Agent:
 xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: 
 text/xml\r\nContent-Length:
 185\r\n\r\n?xml
 version='1.0'?\nmethodCall\nmethodNamepackage_releases/methodName\nparams\nparam\nvaluestringegenix-web-installer-test/string/value\n/param\n/params\n/methodCall\n
 reply: 'HTTP/1.1 200 OK\r\n'
 header: Server: nginx/1.6.0
 header: Content-Type: text/xml
 header: charset: UTF-8
 header: Strict-Transport-Security: max-age=31536000; includeSubDomains
 header: Transfer-Encoding: chunked
 header: Accept-Ranges: bytes
 header: Date: Tue, 08 Jul 2014 14:19:41 GMT
 header: Via: 1.1 varnish
 header: Connection: keep-alive
 header: X-Served-By: cache-fra1229-FRA
 header: X-Cache: MISS
 header: X-Cache-Hits: 0
 header: X-Timer: S1404829181.210045,VS0,VE325
 body: ?xml
 version='1.0'?\nmethodResponse\nparams\nparam\nvaluearraydata\nvaluestring0.2.0/string/value\n/data/array/value\n/param\n/params\n/methodResponse\n

 Could this be a network error rather than a program one ?

 The code in 2.7 does a retry in case of a connection reset or abort,
 which code in 2.6 and earlier does not apply.

 Thanks,


 -- 
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source  (#1, Jul 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 
 2014-07-21: EuroPython 2014, Berlin, Germany ...   13 days to go

 : Try our mxODBC.Connect Python Database Interface for free ! ::

  eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
   D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
  Registered at Amtsgericht Duesseldorf: HRB 46611
  http://www.egenix.com/company/contact/
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


 -- 
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source  (#1, Jul 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
 
 2014-07-21: EuroPython 2014, Berlin, Germany ...   13 days to go

 : Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht

Re: [Distutils] PyPI XMLRPC interface no longer works with Python 2.6

On 08.07.2014 17:18, Donald Stufft wrote:
 Oh Good. I thought I was going crazy there for a minute :) I’m pretty sure
 it happens because of the redirect from HTTP to HTTPS.

That's probably the cause, even though I don't get a redirect header
in the XMLRPC reply. The server appears to return a simple HTTP/1.1 200 OK
and then stops talking to Python 2.6.

Here's a telnet transcript:

 telnet pypi.python.org 80
Trying 185.31.17.175...
Connected to pypi.python.org.
Escape character is '^]'.
POST /pypi HTTP/1.0
Host: pypi.python.org
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)
Content-Type: text/xml
Content-Length: 185


?xml version='1.0'?
methodCall
methodNamepackage_releases/methodName
params
param
valuestringegenix-web-installer-test/string/value
/param
/params
/methodCall
HTTP/1.1 200 OK
Server: nginx/1.6.0
Content-Type: text/xml
charset: UTF-8
Strict-Transport-Security: max-age=31536000; includeSubDomains
Accept-Ranges: bytes
Date: Tue, 08 Jul 2014 15:24:44 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-fra1224-FRA
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1404833078.851717,VS0,VE5364

Connection closed by foreign host.


I guess the HTTP/1.0 in the POST request causes Varnish to close the
connection without a redirect. A bit weird. Python 2.7 sends
an HTTP/1.1.

 On Jul 8, 2014, at 11:17 AM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 08.07.2014 17:15, Donald Stufft wrote:
 Uh what? HTTPS works fine on my copy of 2.6? If I recall this problem
 only happens if you use http:// on 2.6.

 Ah, sorry. You're right. I had looked at a diff of the module
 between 2.6 and 2.7 and saw lots of HTTPS related changes.

 With https:// it does work in 2.6 as well.

 Thanks. I'll close the bug report a fix the wiki page.

Looks like closing the ticket has to be done by someone else.

 On Jul 8, 2014, at 11:13 AM, M.-A. Lemburg m...@egenix.com wrote:

 On 08.07.2014 17:09, Donald Stufft wrote:
 You’re using http:// rather than https:// yea?

 Yes. xmlrpclib in Python 2.6 doesn't understand HTTPS. Support for
 it was added in Python 2.7.

 On Jul 8, 2014, at 10:49 AM, M.-A. Lemburg m...@egenix.com wrote:

 I opened an issue for this:
 https://bitbucket.org/pypa/pypi/issue/157/pypi-xmlrpc-interface-no-longer-works-with

 On 08.07.2014 16:32, M.-A. Lemburg wrote:
 I just tried the documented interfaces for PyPI
 (https://wiki.python.org/moin/PyPIXmlRpc) with Python 2.6 and
 it fails with an error:

 python pypirpc.py
 Traceback (most recent call last):
 File pypirpc.py, line 7, in module
  pprint.pprint(client.package_releases('egenix-web-installer-test'))
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 
 1199, in __call__
  return self.__send(self.__name, args)
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 
 1489, in __request
  verbose=self.__verbose
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 
 1253, in request
  return self._parse_response(h.getfile(), sock)
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 
 1390, in _parse_response
  p.close()
 File /usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py, line 604, 
 in close
  self._parser.Parse(, 1) # end of data
 xml.parsers.expat.ExpatError: no element found: line 1, column 0

 The call works with Python 2.7. It appears that xmlrpclib
 is not receiving any body data from the server.

 The returned data in 2.7 looks completely harmless:

 send: POST /pypi HTTP/1.1\r\nHost: pypi.python.org\r\nAccept-Encoding: 
 gzip\r\nUser-Agent:
 xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: 
 text/xml\r\nContent-Length:
 185\r\n\r\n?xml
 version='1.0'?\nmethodCall\nmethodNamepackage_releases/methodName\nparams\nparam\nvaluestringegenix-web-installer-test/string/value\n/param\n/params\n/methodCall\n
 reply: 'HTTP/1.1 200 OK\r\n'
 header: Server: nginx/1.6.0
 header: Content-Type: text/xml
 header: charset: UTF-8
 header: Strict-Transport-Security: max-age=31536000; includeSubDomains
 header: Transfer-Encoding: chunked
 header: Accept-Ranges: bytes
 header: Date: Tue, 08 Jul 2014 14:19:41 GMT
 header: Via: 1.1 varnish
 header: Connection: keep-alive
 header: X-Served-By: cache-fra1229-FRA
 header: X-Cache: MISS
 header: X-Cache-Hits: 0
 header: X-Timer: S1404829181.210045,VS0,VE325
 body: ?xml
 version='1.0'?\nmethodResponse\nparams\nparam\nvaluearraydata\nvaluestring0.2.0/string/value\n/data/array/value\n/param\n/params\n/methodResponse\n

 Could this be a network error rather than a program one ?

 The code in 2.7 does a retry in case of a connection reset or abort,
 which code in 2.6 and earlier does not apply.

 Thanks,


 -- 
 Marc-Andre Lemburg
 eGenix.com

 Professional Python Services directly from the Source  (#1, Jul 08 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com

[Distutils] pip, virtualenv, setuptools on Windows

2014-07-07 Thread M.-A. Lemburg

I wonder what our story is for the pip, virtualenv, setuptools packaging
setup on Windows.

At the moment, the only way to get this setup seems to be following
guides such as these:

http://www.tylerbutler.com/2012/05/how-to-install-python-pip-and-virtualenv-on-windows-with-powershell/

But that's not really how Windows users expect things to work. They
want to click on an installer, preferably a MSI one, since that
integrates well with managed corporate Windows environments, and then
expect everything to just work. Much in the same way they install Python
itself.

Is there anything planned in this area ?

Hint: If someone were to send in a grant request to build such an
installer for Windows, I'm sure the PSF board would love to fund work
such work.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jul 07 2014)
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-07-21: EuroPython 2014, Berlin, Germany ... 14 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip, virtualenv, setuptools on Windows

2014-07-07 Thread M.-A. Lemburg

On 07.07.2014 15:20, Vinay Sajip wrote:
ISTM the click-on-an-MSI-and-everything-works may be OK for a system-wide
installation for a given version of Python, but I don't see how that can work
with venvs (it's the same problem as for bdist_wininst installers). How would
one pass the venv (to install into) to a point-and-click installer, without
using a command line? Drag-and-drop onto a venv folder is a possibility, but
that's not exactly the conventional usage idiom for MSIs.

Perhaps I wasn't clear enough: I am talking about bootstrapping
a Windows system Python installation with pip, setuptools and
virtualenv.

Once this is done, virtualenvs can be setup as usual from the
command line. The problem is getting to that point easily :-)

Regards,

Vinay Sajip

From: M.-A. Lemburg m...@egenix.com
To: Python Distutils SIG distutils-sig@python.org
Sent: Monday, 7 July 2014, 14:02
Subject: [Distutils] pip, virtualenv, setuptools on Windows

I wonder what our story is for the pip, virtualenv, setuptools packaging
setup on Windows.

At the moment, the only way to get this setup seems to be following
guides such as these:

http://www.tylerbutler.com/2012/05/how-to-install-python-pip-and-virtualenv-on-windows-with-powershell/

Is there anything planned in this area ?

Hint: If someone were to send in a grant request to build such an
installer for Windows, I'm sure the PSF board would love to fund work
such work.

--
Marc-Andre Lemburg
eGenix.com

2014-07-21: EuroPython 2014, Berlin, Germany ... 14 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

Re: [Distutils] pip, virtualenv, setuptools on Windows

2014-07-07 Thread M.-A. Lemburg

On 07.07.2014 15:29, Paul Moore wrote:
 On 7 July 2014 14:25, M.-A. Lemburg m...@egenix.com wrote:
 On 07.07.2014 15:20, Vinay Sajip wrote:
 ISTM the click-on-an-MSI-and-everything-works may be OK for a system-wide 
 installation for a given version of Python, but I don't see how that can 
 work with venvs (it's the same problem as for bdist_wininst installers). 
 How would one pass the venv (to install into) to a point-and-click 
 installer, without using a command line? Drag-and-drop onto a venv folder 
 is a possibility, but that's not exactly the conventional usage idiom for 
 MSIs.

 Perhaps I wasn't clear enough: I am talking about bootstrapping
 a Windows system Python installation with pip, setuptools and
 virtualenv.

 Once this is done, virtualenvs can be setup as usual from the
 command line. The problem is getting to that point easily :-)
 
 The MSI for Python 3.4 includes both pip and venv by default, which
 pretty much covers this scenario. Setuptools is also included but
 that's officially an implementation detail (as the core devs don't
 want to bless setuptools to the extent that distributing it with
 Python would imply). Regardless, pip install setuptools does the
 job, and you have the environment you're describing.
 
 If you want virtualenv rather than venv, pip install virtualenv gets that.
 
 Am I missing something?

Yes: Installers for Python 2.6, 2.7 and 3.3 :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 07 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2014-07-21: EuroPython 2014, Berlin, Germany ...   14 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 470 - Using Multi Index Support for External to PyPI Package File Hosting

2014-05-15 Thread M.-A. Lemburg

On 15.05.2014 01:03, Nick Coghlan wrote:
 On 15 May 2014 01:07, Donald Stufft don...@stufft.io wrote:

 I’ve just published a draft of PEP 470 - Using Multi Index Support for
 External to PyPI Package File Hosting

 You can see this online at http://legacy.python.org/dev/peps/pep-0470/ or
 read below

 
 For the record: I reviewed Donald's initial draft of this PEP and did some
 light copy editing when adding it to the PEPs repo, so this PEP can be
 taken as reflecting my perspective as well.
 
 Of the two currently implemented external hosting mechanisms, the multiple
 indexes model is more general, more explicit, easier to implement, much
 easier to explain and has much cleaner failure modes than the link
 spidering mechanism. The only missing piece is automated external index
 discovery, so adding that is included as part of this PEP. That federated
 discovery mechanism then clears the way for eventually dropping the link
 spidering mechanism (and all its associated complexity and complications)
 entirely.

+1 on the general idea. I do have some issue with some aspects of
the PEP and will respond in more detail in a separate email.

With regard to the index discovery, I'd like to suggest that we
use the existing download_url for this - along the lines I had
suggested last year and what eGenix is already using for such
index pages already (we switched to it as proof of concept
last year).

I'll work out a proposal and send it here later this week.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 15 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Need for respect

2014-05-15 Thread M.-A. Lemburg

On 15.05.2014 00:32, Mark Young wrote:
 I know I'm just an anecdote, but as just a regular user of pypi, pip, and
 friends (I've never even posted something to pypi), when I say pip install
 spam, I don't really care where it comes from and I have no expectation
 that pip has to get it from pypi.

Thank you for your input, Mark. Good to know that I'm not completely
off with my impression of user expectations :-)

FWIW: I don't know anyone who ever bothered researching where their Ubuntu,
Cent OS or SuSE packages came from; and most of the complaints I've heard
about Python installers not working throughout the years were actually due
to PyPI itself not working (in the past), rather than some external hosting
platform being down.


Donald: I understand where you are coming from, hear what you are saying
and know that you want to push for the new PyPI hosting platform.

The hosting platform is brilliant and serves a need for many Python
users, both developers and authors.

I'm just trying to make you see that conceptually separating the
hosting from the index part is a good thing.

This lets us stay more flexible, define clear APIs and remain open
for inclusion of other Python package distribution platforms in
the future, e.g. the binstar platform used by conda/Anaconda or
the PyPM index used by pypm/ActiveState.

I'll comment on the PEP in a separate email.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 15 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


 2014-05-14 18:24 GMT-04:00 Donald Stufft don...@stufft.io:
 

 On May 14, 2014, at 5:33 PM, M.-A. Lemburg m...@egenix.com wrote:

 On 14.05.2014 22:56, Donald Stufft wrote:
 On my phone so I can't respond to everything here but I just want to
 say I don't think a discussion where we can't challenge each other's
 conclusions isn't going to go anywhere. Hopefully we are adults and can
 handle disagreement.

 There's nothing wrong with disagreeing on conclusions and
 agreeing on this disagreement, but trying to shut someone down is
 not the right way forward.

 What I'm trying to get back into the discussion is the view on
 PyPI as a community resource, which does not only need to address
 the needs of users of installers, but also those of developers who
 register their packages with it and make the whole thing come to
 life.

 PyPI is used by people through the browser, it's used by people
 with installer tools such as pip, easy_install, zc.buildout, conda, etc.
 and it's used by developers using distutils, setuptools and
 other build tools that can talk to the PyPI API.

 All of these people have needs and requirements, so we need to
 find ways to make most of them happy. In discussions on this list,
 I often get the feeling that the package authors are underrepresented,
 and that's why I try to add some of package author views to
 the discussion and also views as user of other tools than pip.

 On May 14, 2014, at 4:26 PM, M.-A. Lemburg m...@egenix.com wrote:

 Noah, please reread the subject line and the message that started
 this thread. If we want to have a useful discussion, calling someone's
 conclusion incorrect is not helpful.


 Of course PyPI is a community resource and package authors are important. I
 know that intimately since I publish several things through PyPI that I
 wrote
 and I'm involved with several other projects. I suspect most people on this
 list have published at least one package. If anything we are top heavy in
 the viewpoints of authors with practically no representation from people
 who
 just want to install some stuff.

 I *know* that users have the expectation that installing something from
 PyPI
 means they are downloading it from PyPI. I know this because they tell me
 this,
 constantly. I've dealt with users every single day who are struggling to
 deal
 with the concepts introduced in PEP 438. Whenever I explain to them that
 pip
 will scan PyPI for links to places that are not PyPI they think that is
 just
 crazy. It completely violates their expectations. The *only* people I have
 ever
 seen *not* surprised by that, are people who happen to already know how
 pip/setuptools/etc works. Most of those people that also think it's crazy
 that
 they do that but they aren't surprised by it.

 Framing the hosting of files on PyPI as an extra feature makes it appear
 to
 be something added that isn't part

Re: [Distutils] Need for respect (was: PEP 438, pip and --allow-external)

2014-05-14 Thread M.-A. Lemburg

On 13.05.2014 13:46, Donald Stufft wrote:
 
 On May 13, 2014, at 7:16 AM, Stefan Krah stefan-use...@bytereef.org wrote:
 
 FreeBSD ports have been using the download-from-many-but-verify strategy
 for a long time.  I don't see why users should find this surprising.
 
 The difference is in expectations which is a function of what the “normal” is.
 
 For FreeBSD ports it is normal for those ports to use the 
 download-from-many-but-verify
 strategy. That is the primary mode of operation and for anyone using FreeBSD 
 you know
 that going into it.
 
 However for PyPI it is normal for projects to be hosted on PyPI and the 
 projects which
 are not hosted on PyPI are the outliers which break user expectations. 

I don't think that users generally have such expectations. They just
want to run their favorite installer using myinstaller install package
and get the package installed - the same expectation they have on
Linux distributions, on FreeBSD and other systems with installation
managers.

For most people, it is not important where the installers get their
packages from. They trust the installers to do the right thing.

So from that perspective, we as Python developers need to make sure
that users can trust the installers and infrastructure used
by these (module some definition of trust).

And with Python developers I'm not only talking about PyPI and
installer developers. I'm talking about all Python package
developers as well.

This is a discussions that needs to be had between more people
than just the few people participating in this thread, since
it affects far more people and the whole Python eco system.

Taking a step back:

PyPI is still mainly the Python registry for mapping package
names to URLs and descriptions.

Hosting of distribution files and (less so) documentation
has become an important extra feature (and is now, after all
the hard work you put into it, finally in a usable state), but it's
not the core of what PyPI, the Python Package Index, is all about.

What the team PyPI + installer should thus address is to
satisfy the user trust is to provide a safe way to
get packages installed.

This does not out-rule packages that have to be downloaded
from other indexes or URLs. PyPI and the installers should
make it easy for the users to install all packages in Python
Land, not only the ones hosted on PyPI.

In that context, I find the language being used in these
discussions referring to internal and external packages
somewhat misleading. Such a distinction is not needed,
since packages hosted on PyPI are in no way better, or
of higher value, than packages hosted elsewhere.

In other systems, PyPI hosting would just be called a
repository, one of many which can be used to download
packages from. And it's not uncommon at all to have projects
use their own repositories for their packages on such systems.

 Further more, far more of the installs on PyPI come from linux than come from 
 FreeBSD
 and it stands to reason that we can infer that at least some significant 
 portion of those
 users are incredibly more familiar with Linux systems than FreeBSD. For Linux 
 distros
 it is much more common for them to use a dedicate repository model where 
 packages
 are downloaded from specific locations instead of from wherever the packages 
 might be
 originally hosted at. This further strengthens the idea that a user is 
 expecting PyPI to
 act as a repository and not an index.
 
 You can see some stats I compiled a few months ago based on PyPI’s logs here
 https://gist.github.com/dstufft/8455306#downloads-by-operating-system.

I don't understand what OS choices have to do with this discussion.

Users of apt-get, rpm or FreeBSD's ports are usually not bothered
with having to edit config files for their installers to find
packages.

I think that's the main point here.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 14 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Need for respect (was: PEP 438, pip and --allow-external)

2014-05-14 Thread M.-A. Lemburg

On 14.05.2014 21:48, Noah Kantrowitz wrote:
 
 On May 14, 2014, at 12:44 PM, M.-A. Lemburg m...@egenix.com wrote:
 
 PyPI is still mainly the Python registry for mapping package
 names to URLs and descriptions.
 
 Sorry, going to have to stop you here. This, and all your conclusions based 
 on this assumption, are flat out incorrect. You are far far far in the 
 minority of people that think this is what PyPI is. It was this at one point, 
 but few old-timers are still around to remember those days and new users have 
 very different expectations driven by the cites linux package servers/systems 
 as well as tools like rubygems and cpan.

Noah, please reread the subject line and the message that started
this thread. If we want to have a useful discussion, calling someone's
conclusion incorrect is not helpful.

If you'd read my reply to the end, you'd have noticed that my main
point is that the users want easy installation of packages and don't
care where these are hosted.

This is why installers are attractive to users and this is also why
so many people enjoy using them.

However, such a requirement does not imply that all packages
have to be hosted in a single place, with all the implications
that arise from such a setup.

Coming back to PyPI: Its main purpose is having a central place to
register, search for and find packages. It doesn't matter where the
distribution files are hosted, as long as the installers can find them.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 14 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Need for respect

2014-05-14 Thread M.-A. Lemburg

On 14.05.2014 22:56, Donald Stufft wrote:
 On my phone so I can't respond to everything here but I just want to say I 
 don't think a discussion where we can't challenge each other's conclusions 
 isn't going to go anywhere. Hopefully we are adults and can handle 
 disagreement. 

There's nothing wrong with disagreeing on conclusions and
agreeing on this disagreement, but trying to shut someone down is
not the right way forward.

What I'm trying to get back into the discussion is the view on
PyPI as a community resource, which does not only need to address
the needs of users of installers, but also those of developers who
register their packages with it and make the whole thing come to
life.

PyPI is used by people through the browser, it's used by people
with installer tools such as pip, easy_install, zc.buildout, conda, etc.
and it's used by developers using distutils, setuptools and
other build tools that can talk to the PyPI API.

All of these people have needs and requirements, so we need to
find ways to make most of them happy. In discussions on this list,
I often get the feeling that the package authors are underrepresented,
and that's why I try to add some of package author views to
the discussion and also views as user of other tools than pip.

 On May 14, 2014, at 4:26 PM, M.-A. Lemburg m...@egenix.com wrote:

 Noah, please reread the subject line and the message that started
 this thread. If we want to have a useful discussion, calling someone's
 conclusion incorrect is not helpful.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 14 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)

On 09.05.2014 12:16, Paul Moore wrote:
 So there's an ongoing debate over pip's behaviour around disallowing
 external hosting by default (see thread pip: cdecimal an externally
 hosted file and may be unreliable over on python-dev for the latest
 round).
 
 It appears that the reason for disallowing external hosting (as
 opposed to unverifiable downloads) is purely about reliability - we
 can't be sure that an external host provides the same level of uptime
 as PyPI[1]. Given that, it seems to me that the situation is, for an
 externally hosted package foo:
 
 `pip install foo` - fails immediately, 100% of the time
 `pip install --allow-external foo foo` - works in all but a few
 cases where foo's host is down[2]
 
 I cannot understand how guaranteed failure is ever better than
 occasional but rare failure.
 
 For situations where it is critical to minimise the risk of an
 external host outage causing a deployment to fail, the only answer is
 to not use foo, or to host foo on your own private index. In both
 cases, all you need is to know that foo is externally hosted to do
 that - you certainly don't need pip to fail.
 
 As a concrete proposal:
 
 1. Remove --allow-external/--allow-all-external and make it the
 default behaviour.

+1

 2. Add a new command to pip, maybe something like `pip check-external`
 which checks a set of reuirements, and reports the requirements that
 are externally hosted and which hosts they rely on. That gives users
 who need 100% reliability the information they need to implement the
 appropriate solution. Without causing pain for users who don't.

+1

 Note that the above is based on the fact[3] that *there are no
 security risks to --allow-external*. I am not arguing for a reduction
 in security, or a change to any sort of threat model.

 Comments?

I think that's a good proposal. The main argument we addressed
in the PEP 438 discussion was security, which was addressed by having
tools check the checksums of downloaded packages.

This also clears up the current effect of making externally
hosted packages second class citizens in Python land.

 Paul
 
 [1] Donald explicitly stated that this is the case in the earlier
 thread (https://mail.python.org/pipermail/python-dev/2014-May/134454.html).
 I think Nick confirmed this (although I don't have a reference to
 hand). If it's not true then we need to be a lot clearer and more
 explicit about *why* ignoring external hosting by default is needed,
 because it seems nobody knows :-(
 [2] BTW, the syntax of `--allow-external` is hideous, but that's a
 side issue I don't want to debate right now.
 [3] See the first note.
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Need for respect (was: PEP 438, pip and --allow-external)

Given the thread on python-dev and comments I have read elsewhere,
I would like to remind everyone in this discussion to come back to
a respectful attitude towards the issues being discussed and the
people involved.

I am writing this as Python core developer and as PSF board member.
PyPI is run by the PSF and both the PSF as well as the core developers
have a responsibility to keep the Python eco system a friendly,
open and inviting place to be.

This includes accepting that package authors and companies do have
reasons for not using PyPI to host packages, which some developers
may not follow or find reasonable.

PyPI as community resource still has to make sure that these
package authors are not singled out and can publish their packages
just like other authors who can host their packages on PyPI.

What we can do is mandate security requirements, to make sure
that the users downloading packages via the PyPI index get the
packages that the package author registered, and I'm sure many
authors will understand and respect such added requirements, but
we may not marginalize these authors for not wanting to host on the
PSF infrastructure.

Think about it: PyPI has become a great hosting platform in the last
year, it's attractive to host packages on the platform and this also
shows in the number of package authors that have decided to switch
over to PyPI for hosting.

The ones that don't, will have good reasons not to host on PyPI.
We need to respect those reasons, not question them, and,
if possible, improve the infrastructure to make it more inviting
for them to join in.

We should also reach out to the package authors that currently
do not host on PyPI to find out what their reasons are and
whether we can do something to help convince them.

Note: I haven't yet read the whole thread on the distutils list,
but do want everyone to keep the above in mind when discussing
the topic.

Thank you,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)

On 11.05.2014 16:48, Paul Moore wrote:
 On 11 May 2014 13:47, Donald Stufft don...@stufft.io wrote:
 https://pypi.python.org/simple/egenix-mx-base/ has verifiable external
 links. I'm pretty surprised that Donald hasn't heard of mx-base.

 egenix-mx-base does not have verifiable external links.Verifiable external
 links must be both directly linked to from the /simple/ index page and
 must include a hash. egenix-mx-base does not do this.
 
 OK, that clarifies that, and also makes it clear that what constitutes
 safe is not immediately obvious (something you've been saying a lot,
 but which never eally hit home to me before).
 
 So, some questions:
 
 1. Is MAL aware that egenix-mx-base is not verifiably externally
 hosted[1], and if so, what is he asking for? Automatic download with
 no need for opt-in of unverifiable external downloads? That seems
 pretty much in conflict with the whole intent of PEP 438.

What we are implementing is a proposal that I brought up before
PEP 438 was put in place:

Instead of linking directly to all packages, we put up a verifiable
link to an index page with verifiable links, with the net effect
being that tools can verify the whole chain.

Note that we also provide MD5, SHA1 hashes and GPG signature for
all packages, so users get more security, not less :-)

We had wanted to register links to the download files directly
using the PyPI API and may still implement this (even though it
gets difficult to admin with so many links per release), but have
since shifted focus to working on a web installer which solves
multiple problems at once:

* solving the problem of choosing the right file to download
* making sure downloads are verified for all Python versions
  we support
* adding other features like automatically requesting and
  installing evaluation licenses which we would like to have
  for our commercial products
* making all of the above possible with multiple installers
  such as pip, easy_install, conda, etc. including older
  versions of those installers

With the web installer, we'd just have to upload one file
per release.

PS: Thanks for pointing the broken link on the download page.
This is caused by copying the index page from our normal
PyPI-style simple index to a fixed URL at release, which is done
to make sure that the registered page content hash doesn't change
when we recreate our index.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)

On 12.05.2014 15:58, Paul Moore wrote:
 On 12 May 2014 13:28, M.-A. Lemburg m...@egenix.com wrote:
 So, some questions:

 1. Is MAL aware that egenix-mx-base is not verifiably externally
 hosted[1], and if so, what is he asking for? Automatic download with
 no need for opt-in of unverifiable external downloads? That seems
 pretty much in conflict with the whole intent of PEP 438.

 What we are implementing is a proposal that I brought up before
 PEP 438 was put in place:

 Instead of linking directly to all packages, we put up a verifiable
 link to an index page with verifiable links, with the net effect
 being that tools can verify the whole chain.
 
 Thanks for clarifying. My main concern is that people do actually
 understand the requirements for safe external hosting (I discovered
 that I certainly didn't - it's quite complex!). From what you say, you
 do understand the requirements but have chosen not to follow them at
 this time. That's fine, I'm not trying to debate the validity of your
 choice. But in the context of PEP 438, that means that users of the
 egenix downloads will have to opt into unverifiable downloads in order
 to install using pip. We're working towards improving the user
 experience for end users who need to install unverifiable downloads -
 I'd expect that one consequence of this will be that we'll be better
 able to communicate the situation to those users, which will reduce
 the confusion.

If it helps convince you that allowing verifiable external
links per default is a good thing for user experience, we will
register the distribution file download URLs with the PyPI
web API.

 I don't know if you're suggesting that your proposal is still
 something that we should be looking at implementing. I'm happy to
 discuss that, but it should probably be a separate thread.

You can think of it as per-package PyPI-style simple index
that's registered with PyPI in a secure way (via the check sum
hash).

There's most certainly room for improvement and I'm open for
discussions.

 since shifted focus to working on a web installer which solves
 multiple problems at once:
 [...]
 With the web installer, we'd just have to upload one file
 per release.
 
 I'm not quite sure how you expect this will work, but it's probably
 important that you get involved with the various packaging PEPs. The
 only way I can see such a solution working with pip would be if you
 have a customised setup.py.

Yep, since that's the way installers (not only pip) work when
they see a source distribution.

 As the general trend is towards binary
 installs using wheels, with pip eventually deprecating sdist installs
 and running setup.py directly, we will likely miss your use case
 unless you get involved in those discussions (they are on the back
 burner at the moment, but will likely resurface at some point -
 there's currently a work-in-progress PR for pip to use a wheel
 cache, where the normal install route will be pip wheel; pip install
 the generated wheel, bypassing setup.py install totally).

Binary installs are nice, but they are not the answer to everything
and no matter how much meta data you put into static files,
there will always be cases where that meta data description just
doesn't include those bits you would need to complete the setup,
esp. for packages with C extensions and more complex external
dependencies or setup requirements. (*)

The setup.py interface makes all this possible, which is why so
many Python packages use it to configure themselves automatically.

Deprecating this interface would make some distributions impossible
to install without manual user intervention and we'd be back to the
Makefile.pre.in days.

I don't think that's a good idea. It still is a very good idea
to make more meta data available in static non-executable form
in order to simplify finding packages, determining
dependencies, features, enhancing the PyPI UI, and for
deciding which distribution file to download and install.

This can be generated by setup.py as part of the build process
and made available to PyPI during package release registration
(much like it is now, only in extended form).

(*) This does work if you are only targeting a few select systems and
system versions, but the Python user base out just has too many
diverse setups to be able to address them all to be able to
completely drop setup.py.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 12 2014)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg

Re: [Distutils] PEP 438, pip and --allow-external

On 12.05.2014 22:31, Donald Stufft wrote:
 
 On May 12, 2014, at 3:58 PM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 12.05.2014 15:58, Paul Moore wrote:
 On 12 May 2014 13:28, M.-A. Lemburg m...@egenix.com wrote:
 So, some questions:

 1. Is MAL aware that egenix-mx-base is not verifiably externally
 hosted[1], and if so, what is he asking for? Automatic download with
 no need for opt-in of unverifiable external downloads? That seems
 pretty much in conflict with the whole intent of PEP 438.

 What we are implementing is a proposal that I brought up before
 PEP 438 was put in place:

 Instead of linking directly to all packages, we put up a verifiable
 link to an index page with verifiable links, with the net effect
 being that tools can verify the whole chain.

 Thanks for clarifying. My main concern is that people do actually
 understand the requirements for safe external hosting (I discovered
 that I certainly didn't - it's quite complex!). From what you say, you
 do understand the requirements but have chosen not to follow them at
 this time. That's fine, I'm not trying to debate the validity of your
 choice. But in the context of PEP 438, that means that users of the
 egenix downloads will have to opt into unverifiable downloads in order
 to install using pip. We're working towards improving the user
 experience for end users who need to install unverifiable downloads -
 I'd expect that one consequence of this will be that we'll be better
 able to communicate the situation to those users, which will reduce
 the confusion.

 If it helps convince you that allowing verifiable external
 links per default is a good thing for user experience, we will
 register the distribution file download URLs with the PyPI
 web API.

 I don't know if you're suggesting that your proposal is still
 something that we should be looking at implementing. I'm happy to
 discuss that, but it should probably be a separate thread.

 You can think of it as per-package PyPI-style simple index
 that's registered with PyPI in a secure way (via the check sum
 hash).

 There's most certainly room for improvement and I'm open for
 discussions.

 since shifted focus to working on a web installer which solves
 multiple problems at once:
 [...]
 With the web installer, we'd just have to upload one file
 per release.

 I'm not quite sure how you expect this will work, but it's probably
 important that you get involved with the various packaging PEPs. The
 only way I can see such a solution working with pip would be if you
 have a customised setup.py.

 Yep, since that's the way installers (not only pip) work when
 they see a source distribution.

 As the general trend is towards binary
 installs using wheels, with pip eventually deprecating sdist installs
 and running setup.py directly, we will likely miss your use case
 unless you get involved in those discussions (they are on the back
 burner at the moment, but will likely resurface at some point -
 there's currently a work-in-progress PR for pip to use a wheel
 cache, where the normal install route will be pip wheel; pip install
 the generated wheel, bypassing setup.py install totally).

 Binary installs are nice, but they are not the answer to everything
 and no matter how much meta data you put into static files,
 there will always be cases where that meta data description just
 doesn't include those bits you would need to complete the setup,
 esp. for packages with C extensions and more complex external
 dependencies or setup requirements. (*)

 The setup.py interface makes all this possible, which is why so
 many Python packages use it to configure themselves automatically.

 Deprecating this interface would make some distributions impossible
 to install without manual user intervention and we'd be back to the
 Makefile.pre.in days.

 I don't think that's a good idea. It still is a very good idea
 to make more meta data available in static non-executable form
 in order to simplify finding packages, determining
 dependencies, features, enhancing the PyPI UI, and for
 deciding which distribution file to download and install.

 This can be generated by setup.py as part of the build process
 and made available to PyPI during package release registration
 (much like it is now, only in extended form).

 (*) This does work if you are only targeting a few select systems and
 system versions, but the Python user base out just has too many
 diverse setups to be able to address them all to be able to
 completely drop setup.py.
 
 This is slightly confusing but pip will always be able to go from an sdist to
 an installed system. It'll just build a Wheel first and then install the Wheel
 (at least that's the idea). This is a sort of vague idea right now but it's 
 the
 direction we want to go in.

Ah, so this is just a misunderstanding on my part then. I thought
Paul was saying that having pip run setup.py will go away.

Thanks for the clarification,
-- 
Marc-Andre Lemburg
eGenix.com

Professional

Re: [Distutils] PEP 438, pip and --allow-external

On 12.05.2014 22:37, Donald Stufft wrote:
 
 On May 12, 2014, at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote:
 Binary installs are nice, but they are not the answer to everything
 and no matter how much meta data you put into static files,
 there will always be cases where that meta data description just
 doesn't include those bits you would need to complete the setup,
 esp. for packages with C extensions and more complex external
 dependencies or setup requirements. (*)

 The setup.py interface makes all this possible, which is why so
 many Python packages use it to configure themselves automatically.

 Deprecating this interface would make some distributions impossible
 to install without manual user intervention and we'd be back to the
 Makefile.pre.in days.

 I don't think that's a good idea. It still is a very good idea
 to make more meta data available in static non-executable form
 in order to simplify finding packages, determining
 dependencies, features, enhancing the PyPI UI, and for
 deciding which distribution file to download and install.

 This can be generated by setup.py as part of the build process
 and made available to PyPI during package release registration
 (much like it is now, only in extended form).

 (*) This does work if you are only targeting a few select systems and
 system versions, but the Python user base out just has too many
 diverse setups to be able to address them all to be able to
 completely drop setup.py.

 This is slightly confusing but pip will always be able to go from an sdist 
 to
 an installed system. It'll just build a Wheel first and then install the 
 Wheel
 (at least that's the idea). This is a sort of vague idea right now but it's 
 the
 direction we want to go in.

 Ah, so this is just a misunderstanding on my part then. I thought
 Paul was saying that having pip run setup.py will go away.

 Thanks for the clarification,

 
 I should expand on that answer, that sdist 2.0 will ideally include static
 metadata but it won't be a static build system. Things like name, version,
 dependencies etc will be static, but building will be done by executing some
 code.

Now, you've got me really confused. Is this something I can read up
somewhere ?

sdists already includes static package information in the PKG-INFO file
(format 1.0, but that could be changed to e.g. 2.0).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Bootstrapping pip and setuptools

2013-09-18 Thread M.-A. Lemburg

Not sure whether this is interesting for anyone, but since I saw some
threads about bootstrapping pip and setuptools, I though I might
throw in a tool which does this.

For a while now, we've been making eGenix PyRun available, a Python
run time that fits into a single file on Unix:

http://www.egenix.com/products/python/PyRun/

It's a great virtualenv replacement and doesn't rely on the system
provided Python installation.

Now, to make installation of PyRun even easier, we added an install
script called install-pyrun:

https://downloads.egenix.com/python/install-pyrun

This script downloads the correct PyRun for the platform and
then goes on to bootstrap pip and setuptools fully automatically.
It does this by fetching the most recent versions of these tools
straight from PyPI, without relying on Python (or PyRun).

The script is a simple bash script and uses curl or wget for
the fetch operation, so you get certificate checking, HTTPS,
etc. for free.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 18 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-09-11: Released eGenix PyRun 1.3.0 ...   http://egenix.com/go49
2013-09-20: PyCon UK 2013, Coventry, UK ... 2 days to go
2013-09-28: PyDDF Sprint ...   10 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Bootstrapping pip and setuptools

2013-09-18 Thread M.-A. Lemburg

On 18.09.2013 13:09, Donald Stufft wrote:
 
 On Sep 18, 2013, at 3:54 AM, M.-A. Lemburg m...@egenix.com wrote:
 
 Not sure whether this is interesting for anyone, but since I saw some
 threads about bootstrapping pip and setuptools, I though I might
 throw in a tool which does this.

 For a while now, we've been making eGenix PyRun available, a Python
 run time that fits into a single file on Unix:

http://www.egenix.com/products/python/PyRun/

 It's a great virtualenv replacement and doesn't rely on the system
 provided Python installation.

 Now, to make installation of PyRun even easier, we added an install
 script called install-pyrun:

https://downloads.egenix.com/python/install-pyrun

 This script downloads the correct PyRun for the platform and
 then goes on to bootstrap pip and setuptools fully automatically.
 It does this by fetching the most recent versions of these tools
 straight from PyPI, without relying on Python (or PyRun).

 The script is a simple bash script and uses curl or wget for
 the fetch operation, so you get certificate checking, HTTPS,
 etc. for free.
 
 Are you suggesting this as an alternative to PEP453 or just advertising
 it exists?

Not sure. It works today and is so easy to use, you simply
forget about all the complications it solves :-)

Upside: It works with Python 2.x and 3.x.
Downside: Unix only.

For Windows, using Python would be easier, perhaps using
curl and pycurl to do the SSL heavy lifting:
http://curl.haxx.se/download.html
http://pycurl.sourceforge.net/
https://github.com/pycurl-devs/pycurl/blob/master/examples/retriever.py

It should be possible to wrap all that into an .exe using py2exe
for Python 2.x and cx_Freeze for Python 3.x.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 18 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-09-11: Released eGenix PyRun 1.3.0 ...   http://egenix.com/go49
2013-09-20: PyCon UK 2013, Coventry, UK ... 2 days to go
2013-09-28: PyDDF Sprint ...   10 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Bootstrapping pip and setuptools

2013-09-18 Thread M.-A. Lemburg

On 18.09.2013 14:09, Paul Moore wrote:
 On 18 September 2013 12:49, M.-A. Lemburg m...@egenix.com wrote:
 Upside: It works with Python 2.x and 3.x.
 Downside: Unix only.
 
From your description, it's not clear - is it just the installer
 that's Unix-only, or the runtime as well? I just looked at the link,
 apparently the runtime is as well, but I'm not sure why (it wouldn't
 need curl, for example).

The script is a bash script and the bootstrapping is part of it.

The script downloads setuptools and pip from PyPI using curl
(or wget).

 IMO, anything that doesn't support Windows is at best peripheral to
 this discussion (although PyRun itself sounds awesome, and I'd love to
 see a Windows version!)

Yeah, some rainy day maybe :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 18 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-09-11: Released eGenix PyRun 1.3.0 ...   http://egenix.com/go49
2013-09-20: PyCon UK 2013, Coventry, UK ... 2 days to go
2013-09-28: PyDDF Sprint ...   10 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

On 04.09.2013 01:51, Nick Coghlan wrote:
 On 4 Sep 2013 07:20, M.-A. Lemburg m...@egenix.com wrote:

 On 31.08.2013 17:56, Nick Coghlan wrote:
 setuptools definitely has its issues, but it's still substantially
 superior
 to distutils, and has the critical virtue of behaving the *same* in all
 currently supported versions of Python. Consistency across platform
 versions is something you really want in a build tool, and is something
 a
 standard library module like distutils can never provide. As one of the
 most conservative Linux vendors, even Red Hat has acknowledged this key
 point by creating the Red Hat Developer Toolset to provide a more
 consistent build experience across different RHEL versions. Microsoft
 (with
 Visual Studio) and Apple (with XCode) have long worked the same way.

 I think you're overestimating the usefulness of setuptools here.

 setuptools only extends distutils in various ways, it doesn't
 replace it. And it doesn't do a good job at extending it, since
 it monkey patches distutils in areas where monkey patching
 is not necessary (*).

 distutils does provide a pretty straight forward way to extend it,
 adding new commands to it and new options to existing commands.
 I've been extending distutils for many many years in mxSetup.py
 (which is part of egenix-mx-base). It's been working great and
 I only rarely had to revert to monkey patching in order to get
 something implemented.

 IMO, a much better way forward would be to integrate useful setuptools
 changes right back into distutils, so that the monkey patching
 no longer happens and python-dev can officially bless those
 set of changes.

 BTW: I'm not sure where you get the idea from that setuptools
 behaves the same across Python versions or platforms. It simply
 inherits the distutils changes in each version and thus exhibits
 the same problems (if any) that you see with distutils itself.

 (*) Monkey patching is necessary in a few places, but most of those
 could be fixed by splitting out method code into new methods which can
 then be overridden to provide the new functionality. Note that
 this is a classical problem with OO code, there's nothing special
 here w/r to distutils.
 
 Sure, it's *possible* someone could publish a better setuptools that
 avoids unnecessary monkey patching, leaves out easy_install and separates
 pkg_resources out into its own distribution.
 
 However, setuptools, for all its flaws, already clears the good enough
 bar in most cases, at least as far back as 2.6 (and likely earlier). Any
 new standards are going to have to be supported in setuptools *anyway* due
 to the number of projects that rely on it explicitly for the opt-in
 features like dependency declarations and entry points, so it seems
 expedient to me to avoid duplicating that effort.
 
 There are also existing projects like bento with a setup.py that can cope
 with vanilla distutils *and* with setuptools, but may not cope with a new
 setuptools replacement that does things a bit differently again.
 
 The recommendation to use setuptools and the requirement for all
 distributions to at least *tolerate* setuptools being imported into the
 same process as their setup.py is an entirely pragmatic one in order to
 bootstrap the migration to the next generation dependency management system
 by using the *current* dependency declaration system.

The idea is to integrate setuptools extensions to distutils
back into vanilla distutils, so that you don't need to do
monkey patching and change distutils in wasy that break the
regular distutils extension mechanisms.

This may break a few extensions for Python 3.4, but not many.
Most distributions use plain distutils or plain setuptools
without any changes. Of the ones that use setuptools the most
dominantly used features are being able to build eggs and
requirement tracking.

The ones extending distutils (with or without setuptools
monkey patching enabled) will most likely already support
most of the setuptools hacks, simply because it's been
difficult to ignore setuptools for at least the last
5 years. The few that don't will have to get updated
after such a change.

We need to get rid off hacks like setuptools if we ever
want to see light at the end of the packaging tunnel.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 04 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact

Re: [Distutils] Comments on PEP 426

On 03.09.2013 23:57, Paul Moore wrote:
 On 3 September 2013 22:20, M.-A. Lemburg m...@egenix.com wrote:
 
 IMO, a much better way forward would be to integrate useful setuptools
 changes right back into distutils, so that the monkey patching
 no longer happens and python-dev can officially bless those
 set of changes.

 
 I'm curious about this possibility. One thing that would ease
 experimentation in this area, and which has certainly been discussed
 elsewhere in this thread, would be if there was a clearly defined set of
 setuptools facilities that are missing from distutils, and which the
 modern infrastructure relies on. Off the top of my head, the following
 items would definitely be needed:
 
 1. An extension to the install command to supply a list of the installed
 files (the --record feature of setuptools)

This already exists in distutils' install command.

 2. A bdist_wheel command

This is not part of setuptools, but yes, this should be added
as well.

 3. Updates to the install (or maybe bdist_egg) commands to emit Metadata
 2.0 and dist-info metadata

Should be easy to do (mostly copypaste).

 4. An install_scripts command that created script wrappers from metadata
 (may not be needed if the only supported route to install is via wheels)

Adding new commands is really easy in distutils, so that's
not much of a problem either: distutils has a cmdclass mechanism
which allows to easily register new commands or override existing
command classes.

We've been using this in mxSetup.py to customize distutils
commands and add new commands such as an autoconf mechanism
in the build process, a C library build process (implementing
the ./configure; make dance), bdist_prebuilt or uninstall.

 To be honest, these extensions do seem relatively achievable. But I don't
 know if they are complete - can the advocates of setuptools clarify what
 facilities of setuptools are needed beyond these (with an indication of
 where they are used in the build toolchain).
 
 There are other features of setuptools that I can see would be relevant
 (for instance, version parsing and requirement checking) but these seem to
 me to be more in the nature of utility library features than distutils
 enhancements per se. I'd fully expect that such code could easily be
 extracted from setuptools (indeed, it may be that all of that code is
 isolated in pkg_resources already) without needing any of the distutils
 monkeypatching/extensions.
 
 It may be that such an approach is reinventing the existing wheel that is
 setuptools, and indeed it may never go anywhere - but it is an interesting
 alternative train of thought.

It's not about reinventing the wheel, it's taking the good bits
from setuptools and moving them into distutils to make them
standard for Python 3.4+, allowing setuptools to stop monkey patching
distutils and extensions to stop relying on a setuptools
patched distutils.

I guess that's what the suggestion is all about: avoiding
reinventing the wheel, endless discussions and instead going
for standard software refactoring techniques.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 04 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

On 04.09.2013 09:27, Paul Moore wrote:
 On 4 September 2013 08:13, M.-A. Lemburg m...@egenix.com wrote:
 It's not about reinventing the wheel, it's taking the good bits
 from setuptools and moving them into distutils to make them
 standard for Python 3.4+, allowing setuptools to stop monkey patching
 distutils and extensions to stop relying on a setuptools
 patched distutils.
 
 Personally, I was thinking about externally packaged extensions to
 distutils - a sort of setuptools lite if you like. I think Antoine
 has a point in that people do tend to keep suggesting updating
 distutils forgetting that there is no way that is ever going to
 happen. To my knowledge, no core committer is willing to apply patches
 to distutils[1] and the track record of someone external getting core
 privs and trying to apply distutils changes consists of Tarek, who
 despite all his efforts didn't manage to get anywhere.

The reason for this is not that no one wants to apply such
patches, it's that distutils has been put on a moratorium for
quite some time now - mostly due to the distutils2/packaging
efforts and the concerns about backwards compatibility.

We can lift that moratorium and start to move forward again.

Note that just moving features from setuptools into distutils
will not cause major breakage. If we are serious about this,
we do have to accept some breakage, but it will be minimal.

Related to the suggestion to have distutils extensions,
we could also add a mechanism to make the cmdclass feature
in distutils, I mentioned, easier to use and provide a
way which allows distutils extensions to easily register
themselves.

 I guess that's what the suggestion is all about: avoiding
 reinventing the wheel, endless discussions and instead going
 for standard software refactoring techniques.
 
 To my mind the biggest issue (and again, I'm with Antoine here -
 people keep forgetting this) is that there are no defined API specs to
 work to. You can't implement just the important bits of setuptools
 without knowing what those bits are, and what the interface to them
 is.

I don't quite follow you there. setuptools does come with
documentation and the code is available to be read and reused.

The situation is similar for distutils itself. Most of the
details are laid out in the code. You just need to take
a bit of time and learn the concepts - which is not all
that hard.

 I do suspect that the issue is not so much forgetfulness as optimism -
 people keep hoping that we can find a clean and simple solution, in
 spite of the overwhelming evidence that it'll never happen. But maybe
 that's just me being optimistic as well :-)

I think we've walked down too many dead ends by now to
keep thinking that someone somewhere will come up with
the one and only solution to it all.

distutils itself was an effort in that direction. Before
distutils, the only mechanism we had was Makefile.pre.in.
distutils isn't prefect, but it works mostly and has been
working for quite some time now.

Given that its in the standard library it's hard to
evolve - just like any code that makes it into the stdlib.

I think it's about time that we allow some minimal
breakage to get rid off hacks like setuptools and
first class some, or even all, new commands and
features setuptools adds to distutils.

 [1] For what are probably very good reasons - it seems like it's a
 sure-fire route to instant burnout :-(

True.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 04 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

On 04.09.2013 11:49, Oscar Benjamin wrote:
On 4 September 2013 08:51, M.-A. Lemburg m...@egenix.com wrote:
On 04.09.2013 09:27, Paul Moore wrote:
On 4 September 2013 08:13, M.-A. Lemburg m...@egenix.com wrote:

I guess that's what the suggestion is all about: avoiding
reinventing the wheel, endless discussions and instead going
for standard software refactoring techniques.

To my mind the biggest issue (and again, I'm with Antoine here -
people keep forgetting this) is that there are no defined API specs to
work to. You can't implement just the important bits of setuptools
without knowing what those bits are, and what the interface to them
is.

I don't quite follow you there. setuptools does come with
documentation and the code is available to be read and reused.

The situation is similar for distutils itself. Most of the
details are laid out in the code. You just need to take
a bit of time and learn the concepts - which is not all
that hard.

An implementation is not a defined API spec even if it does come with
some documentation. Tools like pip will need to install projects whose
setup.py uses distutils, or setuptools, or monkey-patched versions of
distutils/setuptools or a reimplementation of some version of
distutils, or with a setup.py that uses neither distutils nor
setuptools. What is needed is an independent specification of the
minimal command-line interface that a setup.py should provide that is
consistent with how things work now for each of those types of
setup.py and sufficient for the needs of past, present and future
packaging tools.

There is documentation for e.g. the egg_info command:
https://pythonhosted.org/setuptools/setuptools.html#egg-info-create-egg-metadata-and-set-build-tags
However this is really just a description of how to use the command
rather than a specification of what expectations can be made of it by
other tools and libraries.

The problem with implementation defined specifications is that there's
no way to reasonably fix or improve the implementation. It is never
possible to say that an implementation conforms or contravenes a
standard if the implementation *is* the standard. When pip fails to
install a project X from PyPI it is not possible to say which of X or
pip (or distutils/setuptools if used) is buggy since there is no
explicitly documented required behaviour anywhere apart from the
general expectation that 'pip install X' should work.

As a case in point 'pip install bento' does not work (fails to find
the egg info directory). I haven't discovered the reason for this yet
but I wouldn't be surprised if the reason is that bento's setup.py
differs in its behaviour in some way that isn't specified in any API
docs. If the answer is that the bento authors should read the whole of
the setuptools codebase and ensure that what they produce is exactly
the same then they may as well use setuptools and there's basically no
way for anyone to distribute sdists that don't use
distutils/setuptools.

If the expected minimal behaviour of setup.py including the relevant
parts that currently *need* to come from setuptools (i.e. the reasons
for pip to monkeypatch distutils with setuptools) were independently
specified in a PEP then those features could be incorporated into
future versions of distutils without propagating the
implementation-defined problems of the past. It would be possible for
pip and other tools to make assumptions in a backward- and
forward-compatible way and to fix bugs in all tools and libraries
since it would be clear what is a bug and what is not.

It would certainly be a good idea to document a minimal
standard setup.py command line interface as PEP.

I quite like the idea of using setup.py as high level
interface to a package for installers to use, since that
avoids having to dig into the details built into the
setup.py code (and whether it uses setuptools, bento,
custom code, etc.).

Package authors could then make sure that they adhere to this
interface spec to avoid breakage such as the one you are
describing.

Could the pip developer perhaps come up with such a minimal
set of requirements they have for the setup.py interface ?

That said, it does, of course, make sense to also document the
distutils internals some more. I was just pointing out that
in order to work on or extend distutils you currently have
to look at the code, since there is only very little developer
documentation available for extending distutils.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Sep 04 2013)
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

: Try our mxODBC.Connect Python Database Interface for free ! ::

eGenix.com

Re: [Distutils] Comments on PEP 426

2013-09-03 Thread M.-A. Lemburg

On 31.08.2013 17:56, Nick Coghlan wrote:
 setuptools definitely has its issues, but it's still substantially superior
 to distutils, and has the critical virtue of behaving the *same* in all
 currently supported versions of Python. Consistency across platform
 versions is something you really want in a build tool, and is something a
 standard library module like distutils can never provide. As one of the
 most conservative Linux vendors, even Red Hat has acknowledged this key
 point by creating the Red Hat Developer Toolset to provide a more
 consistent build experience across different RHEL versions. Microsoft (with
 Visual Studio) and Apple (with XCode) have long worked the same way.

I think you're overestimating the usefulness of setuptools here.

setuptools only extends distutils in various ways, it doesn't
replace it. And it doesn't do a good job at extending it, since
it monkey patches distutils in areas where monkey patching
is not necessary (*).

distutils does provide a pretty straight forward way to extend it,
adding new commands to it and new options to existing commands.
I've been extending distutils for many many years in mxSetup.py
(which is part of egenix-mx-base). It's been working great and
I only rarely had to revert to monkey patching in order to get
something implemented.

IMO, a much better way forward would be to integrate useful setuptools
changes right back into distutils, so that the monkey patching
no longer happens and python-dev can officially bless those
set of changes.

BTW: I'm not sure where you get the idea from that setuptools
behaves the same across Python versions or platforms. It simply
inherits the distutils changes in each version and thus exhibits
the same problems (if any) that you see with distutils itself.

(*) Monkey patching is necessary in a few places, but most of those
could be fixed by splitting out method code into new methods which can
then be overridden to provide the new functionality. Note that
this is a classical problem with OO code, there's nothing special
here w/r to distutils.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 03 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] API for registering/managing URLs for a package

2013-07-19 Thread M.-A. Lemburg

On 18.07.2013 21:00, Donald Stufft wrote:
 Noah,
 External urls are still supported (Although discouraged).

 Marc-Andre,
 There is documentation in the PEP, however I have another PEP
 coming up for a more streamlined upload process that also contains
 a much nicer method of sending external urls as well. So you might
 want to wait for that.

Thanks for the update, Donald.

I found this section:

http://www.python.org/dev/peps/pep-0438/#api-for-submitting-external-distribution-urls

I'll play around with that API a bit and then migrate to your new API.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 19 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] API for registering/managing URLs for a package

2013-07-18 Thread M.-A. Lemburg

I would like to write a script to automatically register release URLs
for PyPI packages.

Is the REST API documented somewhere, or is the implementation the
spec ? ;-)

And related to this:

Will there be an option to tell PyPI's CDN to cache the release
URL's contents ?

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary installer on Windows can't install extension

2013-06-19 Thread M.-A. Lemburg

On 19.06.2013 10:01, Malte Forkel wrote:
 Am 19.06.2013 00:19, schrieb M.-A. Lemburg:

 Are you using our prebuilt or egg distribution of eGenix mx Base,
 or compiling it from source ?

 
 I'm using the prebuilt distribution
 (egenix-mx-base-3.2.6.win-amd64-py2.7.msi).
 
 Thanks to the kind advice of Steve Dower, I have since updated OpenVPN,
 the application had installed the 32-bit version of MSVCR90.DLL in its
 own directory, to a newer 64-bit version which does not install local CRTs.
 
 According to Dependency Walker, it had been OpenVPN's copy of
 MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the
 OpenVPN's entry in the user's search path.

Interesting. I just tried that on my system and indeed, see the
reference to OpenVPN as well. The same happens with all other .pyd
files in the Python installation.

 Now, according to Dependency
 Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved.
 But there are no problems when I run the install script or import
 mx.DateTime myself.

We'll investigate that some more. It may have something to
do with the way distutils handles manifests for Python extensions
at compile/link time. The extensions should get all the MSVCR90.DLL
symbols from the DLL loaded into the PYTHON27.DLL process.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 19 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-06-18: Released mxODBC Django DE 1.2.0 ...   http://egenix.com/go47
2013-07-01: EuroPython 2013, Florence, Italy ...   12 days to go
2013-07-16: Python Meeting Duesseldorf ... 27 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary installer on Windows can't install extension

2013-06-19 Thread M.-A. Lemburg

On 19.06.2013 12:51, M.-A. Lemburg wrote:
 On 19.06.2013 10:01, Malte Forkel wrote:
 Am 19.06.2013 00:19, schrieb M.-A. Lemburg:

 Are you using our prebuilt or egg distribution of eGenix mx Base,
 or compiling it from source ?


 I'm using the prebuilt distribution
 (egenix-mx-base-3.2.6.win-amd64-py2.7.msi).

 Thanks to the kind advice of Steve Dower, I have since updated OpenVPN,
 the application had installed the 32-bit version of MSVCR90.DLL in its
 own directory, to a newer 64-bit version which does not install local CRTs.

 According to Dependency Walker, it had been OpenVPN's copy of
 MSVCR90.DLL that mxDateTime.pyd referenced, probably due to the
 OpenVPN's entry in the user's search path.
 
 Interesting. I just tried that on my system and indeed, see the
 reference to OpenVPN as well. The same happens with all other .pyd
 files in the Python installation.
 
 Now, according to Dependency
 Walker, mxDateTime.pyd refeference to MSVCR90.DLL remains unresolved.
 But there are no problems when I run the install script or import
 mx.DateTime myself.
 
 We'll investigate that some more. It may have something to
 do with the way distutils handles manifests for Python extensions
 at compile/link time. The extensions should get all the MSVCR90.DLL
 symbols from the DLL loaded into the PYTHON27.DLL process.

The unresolved reference to MSVCR90.DLL is shown for all distutils
compiled extensions, so this is most likely the result of distutils
removing the manifest from compiled C extensions to force the linker
to use the Python DLL's CRT libs - this in return avoids problem with
having to place the CRT manifests into each extension directory.

I have no idea why the Windows linker attempts to load a x86 DLL into
a x64 process. The manifest included with OpenVPN clearly shows that
the DLL is for a different platform, even though it uses the same
version as the one listed in the python.exe manifest.

-- 
Marc-Andre Lemburg
Director
Python Software Foundation
http://www.python.org/psf/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary installer on Windows can't install extension

2013-06-18 Thread M.-A. Lemburg

Malte Forkel wrote:
 Am 14.06.2013 00:05, schrieb Steve Dower:
 You will need a 64-bit version of mxDateTime.pyd when used with 64-bit 
 Python.
 The DLL load fails because a 64-bit process can only load 64-bit DLLs.

 If you can't get a 64-bit version, it should not be a problem to install and 
 use
 a 32-bit version of Python on 64-bit Windows (and I generally
 recommend it -  the only thing you gain with 64-bit Python is a
 higher memory limit).

 
 I'm afraid its not that easy. Everything I installed is 64-bit, both
 Python itself and the eGenix mx Base Distribution which includes
 mx.DateTime.
 
 According to Dependency Walker, MSVCR90.DLL is the only 32-bit DLL
 referenced by mxDateTime.pyd. All others DLLs (inluding the version of
 MSVCR90.DLL referenced by PYTHON27.DLL) are 64-bit.

Are you using our prebuilt or egg distribution of eGenix mx Base,
or compiling it from source ?

 When imported from a regular Python process, mxDateTime loads fine. The
 problem only occurred when mxDateTime was imported in the context of the
 installer's (post)install script.
 
 Note the past tense, though. When I ran my tests today, there was no
 error anymore. I guess if I knew what change has caused that, I knew
 what caused the initial problem. I started my tests with Python 2.7.3
 and mx 2.3.1, then upgraded to 2.7.5 and 2.3.6 respectively. The problem
 still occurred. Since, I have rebooted the computer and installed
 another Python package of my own. The paths (user, system) have not changed.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] does pypi or red-dove have a better firehose API than download all the packages?

2013-05-25 Thread M.-A. Lemburg

On 24.05.2013 18:18, Donald Stufft wrote:

On May 24, 2013, at 12:14 PM, M.-A. Lemburg m...@egenix.com wrote:

On 24.05.2013 17:21, Vinay Sajip wrote:
Donald Stufft donald at stufft.io writes:

Most packages also have an egg-info inside of them you can parse.

I don't know how accurate that information is - IIRC pip always runs
egg-info on downloaded archives. I presume this is to get the correct
dependencies which are relevant to the installation system.

It would be nice to have the PKG-INFO readily available on the /simple/
index pages.

This contains the Requires header as well.

The requires headers in the PKG-INFO are practically worthless.

At the moment, PKG-INFO is only available on the PyPI edit pages
for packages you own, even though access is possible without
login:

https://pypi.python.org/pypi?name=egenix-mx-baseversion=3.2.6:action=display_pkginfo

Leaving aside the unusable Requires header, could we still get
PKG-INFO exposed in the /simple/ index ? This would then make
it possible to access the basic meta-data cached via the CDN
and without having to download the package.

Something like:

a
href=https://pypi.python.org/pypi?name=egenix-mx-baseversion=3.2.6:action=display_pkginfo;3.2.6
PKG-INFO/abr/

for each release.

Thanks,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, May 26 2013)
Python Projects, Consulting and Support ... http://www.egenix.com/
mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-07-01: EuroPython 2013, Florence, Italy ... 36 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

Re: [Distutils] does pypi or red-dove have a better firehose API than download all the packages?

2013-05-24 Thread M.-A. Lemburg

On 24.05.2013 17:21, Vinay Sajip wrote:
 Donald Stufft donald at stufft.io writes:
 
 Most packages also have an egg-info inside of them you can parse.
 
 I don't know how accurate that information is - IIRC pip always runs
 egg-info on downloaded archives. I presume this is to get the correct
 dependencies which are relevant to the installation system.

It would be nice to have the PKG-INFO readily available on the /simple/
index pages.

This contains the Requires header as well.

At the moment, PKG-INFO is only available on the PyPI edit pages
for packages you own, even though access is possible without
login:

https://pypi.python.org/pypi?name=egenix-mx-baseversion=3.2.6:action=display_pkginfo

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 24 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-07-01: EuroPython 2013, Florence, Italy ...   38 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] does pypi or red-dove have a better firehose API than download all the packages?

2013-05-24 Thread M.-A. Lemburg

On 24.05.2013 18:18, Donald Stufft wrote:
 
 On May 24, 2013, at 12:14 PM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 24.05.2013 17:21, Vinay Sajip wrote:
 Donald Stufft donald at stufft.io writes:

 Most packages also have an egg-info inside of them you can parse.

 I don't know how accurate that information is - IIRC pip always runs
 egg-info on downloaded archives. I presume this is to get the correct
 dependencies which are relevant to the installation system.

 It would be nice to have the PKG-INFO readily available on the /simple/
 index pages.

 This contains the Requires header as well.
 
 The requires headers in the PKG-INFO are practically worthless.

Hmm, looking at a few PKG-INFO entries for Zope packages, I guess you're
right. Looks like setuptools forgets to add the header to the PKG-INFO.

 At the moment, PKG-INFO is only available on the PyPI edit pages
 for packages you own, even though access is possible without
 login:

 https://pypi.python.org/pypi?name=egenix-mx-baseversion=3.2.6:action=display_pkginfo

It does appear in our PKG-INFO entries:

https://pypi.python.org/pypi?name=egenix-mxodbcversion=3.2.3:action=display_pkginfo

so I guess setuptools isn't behaving quite right in this respect.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 24 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-07-01: EuroPython 2013, Florence, Italy ...   38 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] does pypi or red-dove have a better firehose API than download all the packages?

2013-05-24 Thread M.-A. Lemburg

On 24.05.2013 18:33, Donald Stufft wrote:
 
 On May 24, 2013, at 12:31 PM, M.-A. Lemburg m...@egenix.com wrote:
 
 On 24.05.2013 18:18, Donald Stufft wrote:

 On May 24, 2013, at 12:14 PM, M.-A. Lemburg m...@egenix.com wrote:

 On 24.05.2013 17:21, Vinay Sajip wrote:
 Donald Stufft donald at stufft.io writes:

 Most packages also have an egg-info inside of them you can parse.

 I don't know how accurate that information is - IIRC pip always runs
 egg-info on downloaded archives. I presume this is to get the correct
 dependencies which are relevant to the installation system.

 It would be nice to have the PKG-INFO readily available on the /simple/
 index pages.

 This contains the Requires header as well.

 The requires headers in the PKG-INFO are practically worthless.

 Hmm, looking at a few PKG-INFO entries for Zope packages, I guess you're
 right. Looks like setuptools forgets to add the header to the PKG-INFO.

 At the moment, PKG-INFO is only available on the PyPI edit pages
 for packages you own, even though access is possible without
 login:

 https://pypi.python.org/pypi?name=egenix-mx-baseversion=3.2.6:action=display_pkginfo

 It does appear in our PKG-INFO entries:

 https://pypi.python.org/pypi?name=egenix-mxodbcversion=3.2.3:action=display_pkginfo

 so I guess setuptools isn't behaving quite right in this respect.
 
 setuptools is behaving correctly. the Requires field in the old metadata is 
 for requiring python modules (e.g. things you can directly import). Whereas 
 install_requires is for requiring python distributions (e.g. something you 
 can download from PyPI). The old metadata has no provisions for python 
 distributions.
 
 That's why the Requires field is practically worthless.

That's an unfortunate interpretation, since the original intention
was to describe package dependencies with those fields, but I can
see where this is coming from:

http://www.python.org/dev/peps/pep-0314/

It mentions the module import names, which often does not correspond
to the PyPI package name.

PEP 345 clarifies this (by using a new field), but unfortunately
never made its was into distutils.

Too bad.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 24 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-07-01: EuroPython 2013, Florence, Italy ...   38 days to go

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] HTTPS and certificate check update for distribute ?

2013-05-05 Thread M.-A. Lemburg

On 05.05.2013 00:28, PJ Eby wrote:
 On Thu, May 2, 2013 at 1:41 PM, M.-A. Lemburg m...@egenix.com wrote:
 On 25.04.2013 16:42, M.-A. Lemburg wrote:
 The latest pip supports HTTPS URLs and certificate checks
 (according to the change log).

 Will there be a release of distribute that implements the
 same changes ?

 The current 0.6.36 still defaults to the HTTP PyPI address
 and doesn't do certificate checks.
 
 FWIW, I've just checked in the first phase of my SSL implementation
 for setuptools, to the repository that Jason is doing merges from.
 The current implementation silently uses system-wide root certs from
 the Windows registry or from *nixes that have a well-known root bundle
 location.  (But won't find anything on OS X by default).  It also
 doesn't have any command-line options yet to explicitly select the
 certs used or to control SSL verification.  But it does offer the
 ability to easy_install setuptools[ssl] to download verified copies
 of all the dependencies needed to get SSL support in earlier Pythons,
 including win32 binaries where applicable, without needing anything
 but the original setuptools distribution needing to have been
 downloaded manually via SSL.
 
 There is still more that needs to be done besides command-line
 options, warnings, and docs; providing default root certs for OS X,
 for example.  I've got a couple different ideas on that, from bundling
 the StartCom root cert that python.org uses, to creating a separate
 ca_bundle distribution that contains the files.   There's another
 interesting gotcha with OS X certs, which is that the
 platform-provided openssl may check its built-in cert store in
 addition to what you give it explicitly, which could be a problem.
 
 In short: providing practical, cross-platform,
 cross-wide-array-of-python-versions SSL support is *hard*.  I'm not
 too surprised you haven't heard from anybody yet.  ;-)

http://www.egenix.com/products/python/pyOpenSSL/

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 05 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-04-30: Released eGenix PyRun 1.2.0 ...   http://egenix.com/go44

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] HTTPS and certificate check update for distribute ?

2013-05-02 Thread M.-A. Lemburg

On 25.04.2013 16:42, M.-A. Lemburg wrote:
 The latest pip supports HTTPS URLs and certificate checks
 (according to the change log).
 
 Will there be a release of distribute that implements the
 same changes ?
 
 The current 0.6.36 still defaults to the HTTP PyPI address
 and doesn't do certificate checks.

Hmm, given the lack of response, I guess this will take
a little longer ;-)

I've put Tarek on CC. Perhaps he can chime in...

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 02 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-04-30: Released eGenix PyRun 1.2.0 ...   http://egenix.com/go44

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] HTTPS and certificate check update for distribute ?

2013-04-25 Thread M.-A. Lemburg

The latest pip supports HTTPS URLs and certificate checks
(according to the change log).

Will there be a release of distribute that implements the
same changes ?

The current 0.6.36 still defaults to the HTTP PyPI address
and doesn't do certificate checks.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 25 2013)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-04-17: Released eGenix mx Base 3.2.6 ... http://egenix.com/go43

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427

2012-11-13 Thread M.-A. Lemburg

On 13.11.2012 10:51, Martin v. Löwis wrote:
 Am 13.11.12 03:04, schrieb Nick Coghlan:
 On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth dho...@gmail.com
 mailto:dho...@gmail.com wrote:

 I think Metadata 1.3 is done. Who would like to czar?

 (Apologies for the belated reply, it's been a busy few weeks)

 I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated
 with some additional rationale based on Ronald's comments later in this
 thread, though.
 
 For the record, I'm still -1 on PEP 427, because of the signature issues.
 
 The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot
 readily be used to verify the integrity of an archive - the whole
 point of these technologies is to do exactly that.
 
 The FAQ is entirely silent on why it is not using a more standard
 signature algorithm such as ECDSA. It explains why it uses Ed25519,
 but ignores that the very same rationale would apply to ECDSA as well;
 plus that would be one of the standard JWS algorithms.
 
 In addition, the FAQ claims that the format is designed to introduce
 cryptopgraphy that is actually used, yet leaves the issue of key
 distribution alone (except that pointing out that you can put them
 into requires.txt - a file that doesn't seem to be specified anywhere).

I agree with Martin. If the point is to to protect against cryptography
that is not used, then not using the de-facto standard in signing
open source distribution files, which today is PGP/GPG, misses that
point :-)

Note that signing such distribution files can be handled outside
of the wheel format PEP. It just way to complex and out of scope
for the wheel format itself. Also note that PGP/GPG and the other
signing tools work well on any distribution file. There's really no
need to build these into the format itself.

It's a good idea to check integrity, but that can be done using
hashes.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 13 2012)
 Python Projects, Consulting and Support ...   http://www.egenix.com/
 mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] the 'wheel' binary package format

2012-08-16 Thread M.-A. Lemburg

Daniel Holth wrote:
 On Wed, Aug 15, 2012 at 11:55 AM, M.-A. Lemburg m...@egenix.com wrote:
 You might also want to take a look at the prebuilt binary
 format which we have been using for several years now, e.g.

 http://www.egenix.com/products/python/mxBase/

 The idea is a little different from what you describe, but works
 well: we essentially take a snapshot of the package after it was
 built and then put everything into a ZIP file.

 As a result, you can pick up where distutils left off after the
 build and continue the installation on the target machine.
 
 Thanks, that is fascinating. Wheel goes a little further, and zips up
 an installation of the package (with a particular set of installation
 paths), and it doesn't include setup.py. This is important because I
 want to be able to install into a Python that has no distutils at all.

Ok, that's a different set of requirements then. We wanted
to have most of the distutils commands and options working
in order to stay flexible during the actual installation process.

Since the prebuilt packages are also compatible to the standard
python setup.py install dance after you've unzipped them,
any installer using this approach will just work as well.

We've tested pip and both the install and uninstall commands
work fine out of the box as a result.

 The prebuilt files can also be installed and uninstalled with pip
 if you reference the files directly.

 The only feature missing is support in pip for finding and downloading
 the right prebuilt archive from an index server. pip currently defaults
 to downloading the Windows builds, because it checks for the highest
 version available (for some meaning of high ;-)).

 Which makes me think: would it be possible to make pip more clever
 with respect to platform version strings ?
 
 I do plan to implement this with some help from my friends, at least
 as defined by PEP 425 (in progress;
 http://hg.python.org/peps/file/tip/pep-0425.txt ). I think it will be
 necessary to introduce the concept of picking the best package from a
 set of candidates with the same version, instead of just picking the
 highest version as pip does now.

Great. Please let me know when there's something available to
test. Since we currently only use the version strings for human
consumption, their are easy to change to whatever standard format
will get adopted.

The only requirements that we'd have for such a version tag format
(based on our experience with the prebuilt formats) is that it
includes:

 * a platform string with enough information to determine
   basic binary compatibility (ie. OS, architecture, perhaps
   also OS version depending on OS)
 * Python version, since the byte code changes with every release
 * Unicode variant; at least for those Python versions that need
   it, e.g. non-Windows builds, Python 2.x, 3.0-3.2.

There's an interesting problem we faced with pure-Python builds: on
Windows, many distutils commands store their internal path data in
the OS format, not in the standard distutils format (which uses
the Unix variant with os.sep == '/'). As a result, pure-Python builds
only work cross-platform with the prebuilt format if they are
created on a Unix host, since Windows works fine with the forward
slash os.sep at the API level. As a result, we had to put the
platform string on pure-Python builds created on Windows platforms.

 Here's are some examples of such a version strings as detected by
 pip:

 3.2.4.win32-py2.7.prebuilt
 3.2.4.win-amd64-py2.7.prebuilt
 3.2.4.linux-x86_64-py2.7_ucs4.prebuilt
 3.2.4.linux-x86_64-py2.7_ucs2.prebuilt
 3.2.4.linux-i686-py2.7_ucs4.prebuilt
 3.2.4.linux-i686-py2.7_ucs2.prebuilt

 The code for bdist_prebuilt is in the mxSetup.py that comes with
 egenix-mx-base, in case you want to take a look.
 
 I will take a look.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 16 2012)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2012-08-25: FrOSCon, St. Augustin, Germany ...  9 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] the 'wheel' binary package format

2012-08-15 Thread M.-A. Lemburg

You might also want to take a look at the prebuilt binary
format which we have been using for several years now, e.g.

http://www.egenix.com/products/python/mxBase/

The idea is a little different from what you describe, but works
well: we essentially take a snapshot of the package after it was
built and then put everything into a ZIP file.

As a result, you can pick up where distutils left off after the
build and continue the installation on the target machine.

The prebuilt files can also be installed and uninstalled with pip
if you reference the files directly.

The only feature missing is support in pip for finding and downloading
the right prebuilt archive from an index server. pip currently defaults
to downloading the Windows builds, because it checks for the highest
version available (for some meaning of high ;-)).

Which makes me think: would it be possible to make pip more clever
with respect to platform version strings ?

Here's are some examples of such a version strings as detected by
pip:

3.2.4.win32-py2.7.prebuilt
3.2.4.win-amd64-py2.7.prebuilt
3.2.4.linux-x86_64-py2.7_ucs4.prebuilt
3.2.4.linux-x86_64-py2.7_ucs2.prebuilt
3.2.4.linux-i686-py2.7_ucs4.prebuilt
3.2.4.linux-i686-py2.7_ucs2.prebuilt

The code for bdist_prebuilt is in the mxSetup.py that comes with
egenix-mx-base, in case you want to take a look.

Daniel Holth wrote:
 I got tired of waiting for lxml to compile over and over again, so I
 invented a binary packaging format called 'wheel' (of cheese) that
 uses Metadata 1.2 and .dist-info directories instead of .egg-info,
 patched pkg_resources.py to be able to load .dist-info directories,
 implemented python setup.py bdist_wheel, and patched pip to be able
 to install .whl files.
 
 The gist of the spec is that it is a zip file with the .whl extension
 containing the contents of 'purelib' for a distribution, plus a
 Name-1.0.dist-info/ directory with the metadata files, plus
 Name-1.0.data/subdir directories for scripts, platlib, packaging's
 categories, ...
 
 My specification so far is at
 https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJWABI/edit
 and an lxml compiled for linux-x86-64 is at
 https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs
 
 http://bitbucket.org/dholth/distribute
 http://github.com/dholth/pip
 http://pypi.python.org/pypi/wheel
 
 Perhaps it will be useful. The implementation is still pretty rough,
 for example it does not check the architecture while installing, but
 it could be a handy way to speed up repeated virtualenv builds.
 
 Daniel Holth
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig
 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 15 2012)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2012-08-25: FrOSCon, St. Augustin, Germany ... 10 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Catalog-sig] Fwd: The state of PyPI

2011-09-27 Thread M.-A. Lemburg

Tarek Ziadé wrote:
 = stability and high availability =
 
 we went in two directions to improve PyPI :
 
 1/ add the mirroring protocol
 2/ make the PyPI server more reliable by pushing its storage in a
 redundant cloud.
 ...
 == better infra ==
 
 I think the project is staled right now.

True, I don't have the needed cycles to make progress and neither
do the team members. We do have the Amazon infrastructure setup
(EC2, CloudFront, S3), and some progress has been made to sync
the S3 storage with the local copies of the packages.

The next steps would be to implement a trigger based sync scheme
for packages and the simple/ index, i.e. new uploads should trigger
an S3 copy process and update the simple/ index on S3 as well.

For more info, please have a look at the wiki page:

http://wiki.python.org/moin/CloudPyPI

Here's the original proposal:

http://wiki.python.org/moin/CloudPyPI/Proposal

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 27 2011)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2011-10-04: PyCon DE 2011, Leipzig, Germany 7 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Catalog-sig] pypi/packages/docs.python.org

2011-03-06 Thread M.-A. Lemburg

Martin v. Löwis wrote:
 As for a big link: if you think your page should have one, you are free
 to make it yourself already.

 Sure but,

 1/ I have never asked for the Downloads ↓ link either, but the UI
 did add it, and it's really more ergonomic.

 2/ I have never asked for Latest Version: 0.6.14 on this page : hit
 ttp://pypi.python.org/pypi/distribute/0.6.9
 [...]
 And I think a link to packages.python.org/distribute is at the same level
 
 I can sympathize with that view. Cc'ing catalog-sig here:
 
 if anybody would *not* want to have a Documentation link automatically
 generated that points to packages.python.org/project, please speak up.

That would only make sense if there's something uploaded to
the PyPI docs dir.

If PyPI can detect that, +1. Otherwise, I think it's better not
adding such an automatic link, since no link is better than one
going nowhere.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 06 2011)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Platform naming standardization

2010-01-18 Thread M.-A. Lemburg

Ronald Oussoren wrote:
 
 [fat binaries on Mac OS X]

 It's better to make the included architectures explicit and use
 this logic for all platforms, not just Mac OS X.

 I would probably have done that, knowing what I know now.   

 Hashing out the details on what combinations of architectures are valid 
 during installation will be fun though ;-). That is, if my python says its 
 machine is i386,x86_64 is it then acceptable to install an i386 binary, 
 an i386,x86_64 binary, and i386,ppc, x86_64 binary?

 The point is that even though your Python binary may say it's
 i386,x86_64, the version you run your application with
 will either be i386 or x86_64 (depending on the OS environment
 settings).

 Now let's say you're running the i386 version. As long as all
 installed components provide the i386 part you should be fine.
 In your example all components provide the i386 part, so all
 of them can be installed.
 
 I don't agree, easy_install somepackage should install a component that 
 supports at least all architectures supported by the Python binary.   
 Otherwise you might install a package and have problems later when you try to 
 use it. 
 
 An example of this is a recent 64-bit capable machine with older versions of 
 Tkinter or wxPython:  on those systems python will run as a 64-bit binary by 
 default, but you sometimes have to run python in 32-bit mode to be able to 
 use Tkinter. It would be very annoying and possibly confusing when I install 
 a library and end up not being able to use it sometimes.
 
 I also regularly build standalone app bundles on one machine and run them on 
 other machines, those should also included all components in all supported 
 architectures. 

A package manager should of course try to install the best match
per default and best match should probably mean: use the binary
that supports all architecture parts supported by the currently
running Python binary.

However, it is well possible that some binary packages do not
come in all combinations supported by the Python binary, but do
come in the variant currently running. In such a case, the
user should still be able to install the binary package - perhaps
with a warning that the installed version won't run on some other
supported variants.

With the current aliasing of architecture combinations this
won't be possible, so the user won't be able to install an
ppc/i386 egg, even if she only ever uses the i386 part of a
ppc/i386/x64_86 Python binary.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 18 2010)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Platform naming standardization

2010-01-14 Thread M.-A. Lemburg

Ronald Oussoren wrote:
 
 On 13 Jan, 2010, at 23:44, M.-A. Lemburg wrote:
 
 Ronald Oussoren wrote:

 On 13 Jan, 2010, at 18:41, M.-A. Lemburg wrote:

 Ronald Oussoren wrote:

 On Wednesday, January 13, 2010, at 04:15PM, M.-A. Lemburg 
 m...@egenix.com wrote:


 2) On OS X, the modification to the value returned by
 pkg_resources.get_platform() isn't correct for fat version of Python
 2.5, as detailed in setuptools issue 19.  To solve that, we're using the
 patch I submitted to the issue (with a couple recent changes).

 I think that using fat in the version string is wrong for
 Mac OS X, since there are many ways to build fat binaries.

 Instead, the version string should include the details of
 all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.

 Maybe in the long run, but for now fat has a well-defined meaning for 
 distutils: fat == ppc + x86_64. There is also a number of other variants, 
 as described in the documentation for distutils.

 I think you meant: fat == ppc + i386.

 Thats right.

 However, it's also possible to build binaries with ppc, i386 and
 x86_64 - as are shipped with Mac OS X 10.6, so fat is not really
 well-defined and could lead to trying to install 32-bit software for
 a 64-bit build of Python.

 fat is well-defined for distutils, see the definition of get_platform at 
 http://docs.python.org/distutils/apiref.html. 

 For distutils fat is always a universal binary with architectures i386 
 and ppc, with alternate names for other variants.

 Thanks for pointing that out, however, I don't think that creating
 aliases for combinations of various different architectures
 is a good idea.

 It's better to make the included architectures explicit and use
 this logic for all platforms, not just Mac OS X.
 
 I would probably have done that, knowing what I know now.   
 
 Hashing out the details on what combinations of architectures are valid 
 during installation will be fun though ;-). That is, if my python says its 
 machine is i386,x86_64 is it then acceptable to install an i386 binary, 
 an i386,x86_64 binary, and i386,ppc, x86_64 binary?

The point is that even though your Python binary may say it's
i386,x86_64, the version you run your application with
will either be i386 or x86_64 (depending on the OS environment
settings).

Now let's say you're running the i386 version. As long as all
installed components provide the i386 part you should be fine.
In your example all components provide the i386 part, so all
of them can be installed.

With the aliases, this kind of detection is also possible, but
only after mapping the aliases back to the combination of included
architecture names.

In a few years, we'll probably only see x86_64 binaries for
Mac OS, but until then, package installers will have to
be able to work out the problem of finding installable
distribution files among the available ones.

BTW: With Python 2.6, if you build using the x86_64 version of
Python, distutils will still use the macosx-10.5-i386
platform identifier. Should I file a bug for this ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 14 2010)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Platform naming standardization

2010-01-13 Thread M.-A. Lemburg

Ronald Oussoren wrote:
  
 On Wednesday, January 13, 2010, at 04:15PM, M.-A. Lemburg m...@egenix.com 
 wrote:
 

 2) On OS X, the modification to the value returned by
 pkg_resources.get_platform() isn't correct for fat version of Python
 2.5, as detailed in setuptools issue 19.  To solve that, we're using the
 patch I submitted to the issue (with a couple recent changes).

 I think that using fat in the version string is wrong for
 Mac OS X, since there are many ways to build fat binaries.

 Instead, the version string should include the details of
 all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
 
 Maybe in the long run, but for now fat has a well-defined meaning for 
 distutils: fat == ppc + x86_64. There is also a number of other variants, as 
 described in the documentation for distutils.

I think you meant: fat == ppc + i386.

However, it's also possible to build binaries with ppc, i386 and
x86_64 - as are shipped with Mac OS X 10.6, so fat is not really
well-defined and could lead to trying to install 32-bit software for
a 64-bit build of Python.

IMHO, it's better to list the actually supported architectures
as e.g. darwin-i386-ppc or darwin-i386-ppc-x86_64.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 13 2010)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Platform naming standardization

2010-01-13 Thread M.-A. Lemburg

Ronald Oussoren wrote:
 
 On 13 Jan, 2010, at 18:41, M.-A. Lemburg wrote:
 
 Ronald Oussoren wrote:

 On Wednesday, January 13, 2010, at 04:15PM, M.-A. Lemburg 
 m...@egenix.com wrote:


 2) On OS X, the modification to the value returned by
 pkg_resources.get_platform() isn't correct for fat version of Python
 2.5, as detailed in setuptools issue 19.  To solve that, we're using the
 patch I submitted to the issue (with a couple recent changes).

 I think that using fat in the version string is wrong for
 Mac OS X, since there are many ways to build fat binaries.

 Instead, the version string should include the details of
 all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.

 Maybe in the long run, but for now fat has a well-defined meaning for 
 distutils: fat == ppc + x86_64. There is also a number of other variants, 
 as described in the documentation for distutils.

 I think you meant: fat == ppc + i386.
 
 Thats right.

 However, it's also possible to build binaries with ppc, i386 and
 x86_64 - as are shipped with Mac OS X 10.6, so fat is not really
 well-defined and could lead to trying to install 32-bit software for
 a 64-bit build of Python.
 
 fat is well-defined for distutils, see the definition of get_platform at 
 http://docs.python.org/distutils/apiref.html. 
 
 For distutils fat is always a universal binary with architectures i386 and 
 ppc, with alternate names for other variants.

Thanks for pointing that out, however, I don't think that creating
aliases for combinations of various different architectures
is a good idea.

It's better to make the included architectures explicit and use
this logic for all platforms, not just Mac OS X.

 IMHO, it's better to list the actually supported architectures
 as e.g. darwin-i386-ppc or darwin-i386-ppc-x86_64.
 
 That is not how distutils currently works. I also object to darwin as a 
 prefix, the platform is named macosx. Darwin is the opensource unix variant 
 used in OSX and that name shouldn't be interchangeably with macosx.  I'm also 
 unhappy that sys.platform is darwin on OSX, but it's probably too late to 
 change that.

Darwin is what Mac OS X itself returns as uname -s and that's
generally what's being used for sys.platform on Unix systems (configure
sets MACHDEP which then gets transmogrified into PLATFORM which then
is fed to Py_GetPlatform() which then gets exposed as sys.platform
- just wrote that down here, since I just spent half an hour trying
to find the definition of PLATFORM...).

I agree, though, that the marketing names of the OSes are
somewhat more intuitive :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 13 2010)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Catalog-sig] packaging terminology confusion

2010-01-07 Thread M.-A. Lemburg

Tarek Ziadé wrote:
 On Thu, Jan 7, 2010 at 10:29 PM, Brad Allen bradallen...@gmail.com wrote:
 On Thu, Jan 7, 2010 at 2:40 PM, P.J. Eby p...@telecommunity.com wrote:

 As for projects: fine with me; PyPI would then be the Python Project
 Index.

 +1

 If this gets general agreement, there are probably some places where
 the word 'package' should be replaced with the word 'project', right?
 For example, should PEP-345 be retitled as Metadata for Python
 Software Projects 1.2 ?
 
 +1, we should reflect this terminology in all new PEPs and in
 Distutils doc as well

I don't think we need to change anything - most Python software
components come as Python packages nowadays, so the terminology
'package' we've used all these years is correct.

OTOH, sets of components like Zope are indeed projects. However,
the number of PyPI packages which could reasonably be called a project
is relatively small and even those use Python packages to
separate their namespaces.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 07 2010)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Catalog-sig] packaging terminology confusion

2010-01-07 Thread M.-A. Lemburg

Brad Allen wrote:
 On Thu, Jan 7, 2010 at 3:51 PM, M.-A. Lemburg m...@egenix.com wrote:
 
 I don't think we need to change anything - most Python software
 components come as Python packages nowadays, so the terminology
 'package' we've used all these years is correct.
 
 Do you mean only 'package' in the sense of an __init__.py container,
 or in the sense of a setup.py container? Both usages have been in use
 for years, but only the __init__.py package was formally recognized.

We have used the term 'package' for both a Python
software component as well as the Python module directories on
sys.path with a __init__.py marker.

I usually refer to the latter as 'Python package' and the former
as 'package'.

What you download is a 'distribution file' which could be an exe,
a tarball, an egg, etc. There are many forms such a package
distribution can take and just as many ways to install them.

Once installed a 'package' usually creates a single 'Python package'
(at top-level or some lower level).

As a result, after installation there is little difference between
a Python software component called 'package' and the module
directory called 'Python package'. qed :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 07 2010)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 386 status - last round here ?

Tarek Ziadé wrote:
 Last, as I said in a previous mail, I tend to agree with the people
 who said that we should stick with only one way to write the version
 scheme for the sake of clarity. e.g. dropping aliases and picking
 *one* way to write the markers after major.minor.micro.
 
 I would tend to pick the same scheme than Python for the pre-releases
 (and c + rc):
 
 N.N[.N][(a|b|c|rc)N]
 
 And, for the post/dev markers I think dots are OK for readability,

Sure, but readability and clarity means different things for
different people.

The reason I proposed aliases and underscores is to give package
authors the choice of using terse forms or more verbose ones, as
well as making the whole scheme more compatible to existing
version strings already in use.

Regarding post/dev markers:

IMO, it's not really obvious that a 1.0a1.dev123 release refers to a
snaphost *before* the 1.0a1 release. The string pre is more commonly
used for such pre-release snapshots.

For the .post123 tag, I don't see a need for a post string at all,
1.0a1.123 is clearly a release or build *after* the 1.0a1 release
and since the 1.123 is being treated as alpha version number,
the post part processing can be dropped altogether.

For the .dev part the situation is similar: you can always
choose a pre-release version that is not actually released and then
issue follow up snapshots to this, e.g.

1.0a0.20091203
1.0a0.20091204
1.0a0.20091205

and so on for nightly builds during the development phase.

Instead of writing:

1.0a1.dev20091205

you'd then write

1.0a0.20091205

This is how Python itself is organizing the versions during
development, BTW.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 03 2009)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 386 status - last round here ?

Tarek Ziadé wrote:
 On Thu, Dec 3, 2009 at 1:55 PM, M.-A. Lemburg m...@egenix.com wrote:
 Tarek Ziadé wrote:
 Last, as I said in a previous mail, I tend to agree with the people
 who said that we should stick with only one way to write the version
 scheme for the sake of clarity. e.g. dropping aliases and picking
 *one* way to write the markers after major.minor.micro.

 I would tend to pick the same scheme than Python for the pre-releases
 (and c + rc):

 N.N[.N][(a|b|c|rc)N]

 And, for the post/dev markers I think dots are OK for readability,

 Sure, but readability and clarity means different things for
 different people.

 The reason I proposed aliases and underscores is to give package
 authors the choice of using terse forms or more verbose ones, as
 well as making the whole scheme more compatible to existing
 version strings already in use.

 Regarding post/dev markers:

 IMO, it's not really obvious that a 1.0a1.dev123 release refers to a
 snaphost *before* the 1.0a1 release. The string pre is more commonly
 used for such pre-release snapshots.

 For the .post123 tag, I don't see a need for a post string at all,
 1.0a1.123 is clearly a release or build *after* the 1.0a1 release
 and since the 1.123 is being treated as alpha version number,
 the post part processing can be dropped altogether.

 For the .dev part the situation is similar: you can always
 choose a pre-release version that is not actually released and then
 issue follow up snapshots to this, e.g.

1.0a0.20091203
1.0a0.20091204
1.0a0.20091205

 and so on for nightly builds during the development phase.

 Instead of writing:

1.0a1.dev20091205

 you'd then write

1.0a0.20091205

 This is how Python itself is organizing the versions during
 development, BTW.
 
 So IOW, are you suggesting that a suffix marker is always a post
 release marker ?

In a way, yes.

For pre-releases, it's actually a minor revision of the pre-release
version, e.g. in 1.0a0.123 the 0.123 part is the pre-release
version.

For releases, it's a normal part of the version number, e.g.
in 1.0.123 the .123 marks a patch level release.

 so we have :
 
 1.0a0
  1.0a0.124
  1.0a0.245
  1.0a1
  1.0a1.346
  1.0a2
  1.0
  1.0.567  --- dev marker

That's not a dev-marker, it's actually part of the release version:
the patch level release number.

 The problem in that case is that we would be unable to differenciate
 dev marker with micro markers.

The dev markers introduce an extra level of confusion, which
IMHO is not necessary.

Let's take 1.0a0.dev123 as example, reading it from the left:

1.0  - ok, so this is part of a 1.0 release
1.0a0- oops, no, it's not actually part of a release, it's part
   of a pre-release, so move back
1.0a0.dev123 - hmm, not even that, it's part of what's going
   to become a pre-release, so move even further back

As developer you typically want to use the version number to
tell the user a few things about your package:

1. whether it's release quality code

1.0.0

2. whether it's a development snapshot

1.0.0a0.20091202

3. whether it's working code, but still under development

1.0.0a1

4. whether it fixes some bug that was found after a release

1.0.1

 That's why we added these dev markers in the first place. So
 following your ordering:
 
 - 1.0a0
  1.0a0.dev124
  1.0a0.dev245
  1.0a1
  1.0a1.dev346
  1.0a2
  1.0
  1.0.dev567  --- dev marker
 
 Now about making the dev a post release tag, AFAIK people always use
 dev markers as pre-release tags:
 
 - 1.0a0
  1.0a1.dev124
  1.0a1.dev245
  1.0a1
  1.0a2.dev346
  1.0a2
  1.0.dev567  --- dev marker
  1.0
 
 Last, if we want to do a post release for 1.0 for example, you would
 also need a post- marker, as we added

Sorry, I probably wasn't clear enough. What I proposed looks like this:

  1.0a0 (which is not released and only used to mark the start of 1.0
 development, just like we do for Python)
 1.0a0.124
 1.0a0.245
 1.0a1 (pre-release)
 1.0a1.346
 1.0a2 (pre-release)
 1.0   (release)
 1.0.567   (release)

and the 1.0.567 is not a dev-marker (since the proposal is about
removing these ;-), but instead a post-marker without the
post. In reality, such a post release would be a patch-level
release.

If you want to then work on release 1.1, you'd continue with:

- 1.0.567
 1.1a0 (which is not released and only used to mark the start of 1.1
 development, just like we do for Python)
 1.1a0.124
 1.1a0.245
 1.1a1 (pre-release)
 1.1a1.346
 1.1a2 (pre-release)
 1.1   (release)

As you can see, you don't need any dev-markers and post-markers
turn into pre-release minor version numbers... less noise,
more clarity, well at least IMHO.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 03 2009)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter

Re: [Distutils] PEP 386 status - last round here ?

Toshio Kuratomi wrote:
 On Thu, Dec 03, 2009 at 01:55:53PM +0100, M.-A. Lemburg wrote:
 Tarek Ziadé wrote:
 Last, as I said in a previous mail, I tend to agree with the people
 who said that we should stick with only one way to write the version
 scheme for the sake of clarity. e.g. dropping aliases and picking
 *one* way to write the markers after major.minor.micro.

 I would tend to pick the same scheme than Python for the pre-releases
 (and c + rc):

 N.N[.N][(a|b|c|rc)N]

 And, for the post/dev markers I think dots are OK for readability,

 Sure, but readability and clarity means different things for
 different people.

 The reason I proposed aliases and underscores is to give package
 authors the choice of using terse forms or more verbose ones, as
 well as making the whole scheme more compatible to existing
 version strings already in use.

 I'm not a big fan of underscores -- having multiple separators doesn't seem
 very useful.

I'm not tied to those underscores, just think they'd help in
making the strings more readable.

 I don'tlike aliases but seeing as I like the long forms, having both short
 and long but giving them a distinct ordering would be okay to me (ie:
 
 a1  alpha1  a2  b1  beta1  c1  rc1

That would also work.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 03 2009)
 Python/Zope Consulting and Support ...http://www.egenix.com/
 mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
 mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 386 status - last round here ?