Re: [Distutils] Disabling non HTTPS access to APIs on PyPI

[Erik Bray]

On Thu, Oct 26, 2017 at 5:11 PM, Donald Stufft  wrote:
> Historically PyPI was only available over either HTTP or unvalidated HTTPS,
> and over time we’ve been pushing more and more traffic onto HTTPS. In
> Warehouse the decision was made to *not* redirect “API” URLs from HTTP to
> HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would silently
> get redirects to https and appear to “work”, but you wouldn’t get any of the
> security properties of TLS because an attacker would just intercept the
> request prior to the redirect happening.
>
> Today I’ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/ over
> HTTP and you will have to directly go to HTTPS. For most people this should
> have no effect, because most tooling should be defaulting to HTTPS anyways,
> however if you’re using a significantly old version of tooling, it may still
> be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.

+1

This will probably (unfortunately) break some things for some people,
which is worrying.  But it is the right thing to do and good advice in
general.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires: the obvious option(?)

[Erik Bray]

On Mon, Aug 29, 2016 at 7:29 PM, Antony Lee  wrote:
> Hi all,
>
> The `setup_requires` option to `setup()` is well-known to suffer from
> multiple issues.  Most importantly, as it is a keyword argument to
> `setup()`, it appears too late for modules that may need to be imported for
> the build to occur (e.g., Cython, for which support must explicitly provided
> by setuptools itself rather than by letting Cython hook into it);
> additionally, there are various contorsions that people go to to avoid some
> `setup_requires` when not building the package (such as checking the value
> of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
> `pip`, but I do not see why this could not be fixed; let's focus on the
> first issue instead.
>
> If `setup_requires` appears too late to be useful, the obvious(?) option is
> to move it earlier: provide a function, say, `setuptools.setup_requires()`,
> that should be called *before* `setup()` itself, e.g.:
>
> from setuptools import setup, setup_requires
> setup_requires("numpy", needed_for=["build_ext"])
> try:
> import numpy as np
> except ImportError:
> np = None
> setup(..., include_dirs=[np.get_include()] if np else [])

I mean this sort of already exists but it's spelled:

from setuptools import Distribution
Distribution({'setup_requires': ['numpy'])

Granted it's non-obvious and doesn't have the needed_for flag, which I
like.  It's not entirely clear how needed_for would work though.  For
example, what if the package you're requiring provides the command
that you need that package to run?

The same can be done by subclassing commands, and there can be some
corner cases where that gets extra tricky (Cython comes to mind).

> When `setup.py` is invoked, either directly or by pip, upon the call to
> `setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
> least one requirement is missing, `setup_requires()` calls asks pip to
> install the required packages (similarly to
> `https://bitbucket.org/dholth/setup-requires`) in a temporary directory, and
> the whole Python process replaces itself (in the `os.execv()` sense) by a
> new call to `python setup.py` with this temporary directory prepended to the
> PYTHONPATH.  In this new process, the arguments to `setup_requires()` are
> now available and we can proceed to the rest of `setup.py`.
>
> I feel like this idea is natural enough that someone must already have come
> up with it... but I may be missing something :-)

I'm glad you mentioned Daniel Holth's setup-requires hack.  Although I
haven't used it myself directly I generally like the concept.

Yeah, setup_requires is a mess, but I'd be skeptical of solving the
problem by depending on any new features in setuptools :/

Best,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 516 nitpicks

[Erik Bray]

Hi all,

I was just getting around to reading the latest published draft of PEP
516 and had a few minor nitpicks.  Sorry if this unhelpful, as I know
it's still being drafted and may change anyways.

"The programmatic interface allows decoupling of pip from its current
hard dependency on setuptools [2] able for two key reasons:"

The "able" at the end of this sentence seems out of place.

***

"The file pypa.json acts as a neutral configuration file for pip and
other tools..."

It's not clear to me what "neutral" means here?  From context I'm
pretty sure it means not specific to any build tool, but I think it's
distracting.  I also don't think it's entirely neutral since its
*absence* implies a specific build tool.

***

"The JSON has the following schema. Extra keys are ignored, which
permits the use of pypa.json as a configuration file for other related
tools. If doing that the chosen keys must be namespaced under tools"

This last part seems inconsistent.  It implies that additional keys
may be used for other tools, but that they must be under a "tools"
key.  Okay, but what does that imply for top-level keys not specified
by the schema?  Will they be ignored? Or should they be explicitly
disallowed in order to enable expansion of the format?

***

"As usual with processes, a non-zero exit status indicates an error."

The "As usual with processes" is somewhat *NIX-biased.  I think simply
stating the expectation is better: "Non-zero exit statuses are assumed
to indicate an error."  Of course, any program for which that does not
apply for some reason would have to have a wrapper around itself.

***

"Note that doing so will cause use operations like pip install -e foo to fail."

"use" -> "use of"?  Also, how will they fail?  Will pip still install
dependencies or roll those back too?

***

"develop [--prefix PREFIX]"
...

"flit develop --root /tmp/ --prefix /usr/local"


So does develop have a "--root" option or not?  (Maybe this is a moot
point if the interface is going to switch back to a Python API? It's
not clear to me what the last word was on that.)


That's all for now.  Thanks!
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Idea: Positive/negative extras in general and as replacement for features

[Erik Bray]

On Thu, Mar 3, 2016 at 8:30 PM, Ronny Pfannschmidt
 wrote:
>
>
> Am 09.02.2016 um 22:24 schrieb Robert Collins:
>>
>> Sorry for not replying for so long.
>>
>> On 16 December 2015 at 07:54, Ronny Pfannschmidt
>>  wrote:
>>>
>>> Hello everyone,
>>>
>>> the talk about the sqlalchemy feature extra got me thinking
>>>
>>> what if i could specify extras that where installed by default, but
>>> users could opt out
>>>
>>> a concrete example i have in mind is the default set of pytest plugins
>>> i would like to be able to externalize legacy support details of pytest
>>> without needing people to suddenly depend on pytest[recommended] instead
>>> of pytest to have what they know function as is
>>>
>>> instead i would like to be able to do something like a dependency on
>>> pytest[-nose,-unittest,-cache] to subtract the items
>>
>>
>> So the challenge here will be defining good, useful, and predictable
>> behaviour when the dependency graph is non-trivial.
>>
>> Using your example, what should pip do when told
>>
>> pip install pytest[nose, -unittest] proj2
>>
>> and proj2 depends on pytest[unittest]
>>
>> ?
>
> those should be additive,
> negative extras should be used purely to leave out dependencies for a single
> requirement
> if something else needs them, then they should of course be included
>
> even a dependency on pytest itself should include the unittest dependency
>
> negative requirements should be absolutely local and only effect the
> dependency graph of a single direct dependency
>
> so if there is another dependency without that negation, then it should
> consider the full graph in any case

I only just saw this thread because Ronny's recent(-ish) reply, and I
agree with him.  In my mind the use case for something like this is
pretty independent of "more complex" interactions like Robert's
example.  I think the idea here is that if I run `pip install
pytest[-unittest]` I'm not saying the `pytest[-unittest]` must never
ever be installed on the system.  Rather, it's useful in a far more
limited cases where I need more fine-grained control over what's
installed--think for example application bundles or running on
embedded systems.

A case that would be useful to me would be to have an `astropy[tests]`
extra that is installed by default.  The astropy test suite is a bit
weighty, especially due to the size of test files.  I've proposed
making the tests an extra before but that was shot down (I think
rightfully) since it would be preferable to have it installed by
default.  However, for cases where the tests take up too much space a
`pip install astropy[-tests]` would be nice--there's not likely to be
an additional package that depends on `astropy[tests]` and even if
there were I'd know in building my system whether or not I really need
it.

The alternative is post-install stripping down of packages, which can
be unpredictable and error-prone in my experience.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Can't build C++11 libraries with distutils.command.build_clib because it lacks extra_compile_args

[Erik Bray]

On Tue, Feb 23, 2016 at 1:53 AM, Matthias Goerner  wrote:
> Hi!
>
> I am trying to use distutils.command's build_clib to compile C++11 code but
> had to monkey patch build_clib to support the extra_compile_args
> "-std=c++11", see my_build_libraries in
> http://sageregina.unhyperbolic.org/setup.py .
>
> As you can see, it was only adding the line
>
> extra_postargs = build_info.get('extra_compile_args'))
>
> to
>
> objects = self.compiler.compile(
> sources,
> output_dir=self.build_temp,
> macros=macros,
> include_dirs=include_dirs,
> ...)
>
>
> in build_clib.py. Could you make this change to distutils so that is
> available for everyone and I can eventually stop monkey-patching?

Hi,

I don't believe build_clib is really supported.  You can't even find
much reference to it or how to use it in the documentation.  I believe
it's a legacy of distutils' earlier naive attempts to claim to support
compilation of C/C++ libraries, but that's too lacking in flexibility
to really be useful beyond the most trivial cases.  Hence a decade+ of
frustration with using distutils/setuptools to build projects that
rely on non-Python libraries and various outcroppings of third-party
projects intending to improve on that and the like.

You probably won't get distutils patched, but what you can always do
is subclass commands to add additional functionality.  This is less
hacky the monkey-patching and is generally well supported.  The long
tradition of this is why distutils is nearly impossible to change :)

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Unable to build language-check

[Erik Bray]

On Wed, Jan 13, 2016 at 9:30 AM, Sijo Jose  wrote:
> The language-check (https://pypi.python.org/pypi/language-check)library has
> the dependency of '3to2' in python 2.7,
> In a virtualenv it works fine with pip,that is
>
> pip install  3to2
> pip install language-check
>
>
> But its failing with easy_install in venv
>
> easy_install 3to2
> easy_install language-check (It fails)
>
> Due to this I'm not able to build my project using setuptools.

Why do you need to install it with easy_install?  pip is the
recommended installer for Python packages.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Reinstall with pip

[Erik Bray]

On Wed, Dec 23, 2015 at 5:23 PM, Carlos Barera  wrote:
> Hi,
>
> I have an automation task that installs a python package from within the
> project dir using: pip install .
>
> What's the right way to reinstall the same package (same version)?
> --upgrade + --force-reinstall ?
> Is --no-deps recommended ?
> How about uninstalling and reinstalling ?

I've generally had luck with

pip install --upgrade --force-reinstall --no-deps

This should automatically take care of uninstalling before
reinstalling too, so you shouldn't wind up with a broken package or
outdated files hanging around.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py install using pip

[Erik Bray]

On Thu, Dec 10, 2015 at 5:12 PM, Robert Collins
<robe...@robertcollins.net> wrote:
> On 8 December 2015 at 09:14, Erik Bray <erik.m.b...@gmail.com> wrote:
>> On Mon, Dec 7, 2015 at 2:40 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
>>> On 7 December 2015 at 18:58, Erik Bray <erik.m.b...@gmail.com> wrote:
>>>> I wasn't able to produce this problem.  Even with --no-binary
>>>> specified pip installs (by default) with
>>>> --single-version-externally-managed.  My prototype implicitly disables
>>>> the --pip flag if --single-version-externally-managed was specified
>>>> (true to the purpose of that flag).
>>>
>>> Ah - that was the bit I was missing, the
>>> --single-version-externally-managed flag can be used to trigger
>>> ignoring --pip.
>>>
>>>> What *is* a problem is if --pip is in setup.cfg, and one invokes `pip
>>>> install --egg .`.  I wasn't quite able to make that go into an
>>>> infinite loop, but it did invoke pip.main recursively, and stuff broke
>>>> on the second invocation for reasons not clear to me.
>>>
>>> Yeah, but honestly I don't think pip install --egg is that important a
>>> use case. I may be wrong (there's lots of ways people use pip that I
>>> know nothing of :-)) but as a starting point it might be OK just to
>>> say that at the same time as the --pip flag was introduced, "pip
>>> install --egg" was deprecated (and we explicitly document that pip
>>> install --egg is known to interact badly with setup.py --pip).
>>
>> I'd be fine with that too.  IIRC pip install --egg was introduced in
>> part to work around problems with namespace packages.  This doesn't
>> completely eliminate the need for that workaround, but it does reduce
>> it.
>
> Huh? No, my understanding was that it was introduced solely to support
> interop with folk using 'easy-install', and its considered deprecated
> and delete-as-soon-as-practical.

The original issue that motivated it did have to do with (lack of)
interoperability of different ways namespace packages are implemented:

https://github.com/pypa/pip/issues/3

The fact that it introduced general backwards-compat for
easy-install-like installation was a side "benefit", useful I'm sure
to a few people.  But otherwise as you say, was intended to be deleted
as soon as practical.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Partial installs for in-process distribution upgrades (was: Installing packages using pip)

[Erik Bray]

Apologies for resending this--my original message got buried in
unrelated commentary about how *nix filesystems work.  Resending with
new subject line...

On Fri, Nov 13, 2015 at 3:09 PM, Nathaniel Smith  wrote:
> On Nov 13, 2015 12:00 PM, "Alexander Walters" 
> wrote:
>>
>> import pip
>> pip.install(PACKAGESPEC)
>>
>> something like that?
>
> This would be extremely handy if it could be made to work reliably... But
> I'm skeptical about whether it can be made to work reliably. Consider all
> the fun things that could happen once you start upgrading packages while
> python is running, and might e.g. have half of an upgraded package already
> loaded into memory. It's like the reloading problem but even more so.

Sorry to resurrect an old thread, but I have an idea about how to do
this somewhat safely, at least insofar as the running interpreter is
concerned.  It's still a terrible idea.  Not such a terrible idea in
principle, but as a practical matter in the context of Python it's
probably a bad idea because it uses yet-another-.pth-hack.

Consider a "partial install", wherein pip installs all files into a
non-imported subdirectory of the target site-packages, along with a
.pth file.  This distribution is then considered "partially installed"
in that the files are their (whether extracted from a wheel, or
installed via distutils and the appropriate --root option or similar).
For example, consider running

>>> pip.install('requests')

It would be up to the pip.install() command to determine whether or
not the requests distribution was already installed.  If it's not
install it would proceed as normal.  For now I'm assuming the user
would still have to manually run `import requests` after this.
Auto-import would be nice, but is a separate issue.

Now, if requests were already installed and imported we don't want to
clobber the existing requests running in the interpreter.  pip would
install install into the relevant site-packages:

<...>/site-packages/requests-2.8.1.part/
requests/
requests-2.8.1.dist-info/
<...>/site-packages/requests-2.8.1.part.pth

The .part/ directory contains the results of the partial installation
(for example the contents of the wheel, for wheel installs).  The
.part.pth file is trickier, but could be something like this:

$ cat requests-2.8.1.part.pth
import inspect, shutil, sys, os, atexit;p =
inspect.currentframe().f_locals['sitedir'];part = os.path.join(p,
'requests-2.8.1.part');files = os.path.isdir(part) and
os.listdir(part);files and list(map(lambda s, d, f:
(sys.modules['shutil'].rmtree(os.path.join(d, f),
sys.modules['shutil'].move(os.path.join(s, f), os.path.join(d, f))),
[part] * len(files), [p] * len(files), files));os.rmdir(part);pth =
part + '.pth';os.path.isfile(pth) and atexit.register(os.unlink,
os.path.abspath(pth))

This rifles through the contents of requests.2.8.1.part, deletes any
existing directories in the parent site-packages of the same name,
completes the install by moving the contents of the .part/ directory
into the correct location and then deletes the .part/ directory.  The
.part.pth later deletes itself.

By the time the user restarts the interpreter and runs `import
requests` this will be completed.  Obvious it would have to be
communicated to the user that to upgrade an existing package they will
have to restart the interpreter which is less than ideal, but relates
to a deeper limitation of Python that they should get used to anyways.
At least this would enable in-process installs/upgrades.

There are of course all kinds of problems with this solution too.  It
should perhaps only work in a virtualenv and/or .local site-packages
(or at least somewhere that the user will have write permissions on
the next interpreter run), and probably other error handling too.  The
above .pth file could also be simplified by invoking a function in pip
to complete any partial installs.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py install using pip

[Erik Bray]

On Mon, Dec 7, 2015 at 4:34 AM, Nick Coghlan  wrote:
> On 7 December 2015 at 17:07, Ronny Pfannschmidt
>  wrote:
>> That's a straw man, this has enough inconsistency potential to break many
>> edge cases in ugly ways,
>> So global setup is out.
>
> No, global set up *isn't* out - the inevitable edge cases won't matter
> to an application integrator if none of the components they're using
> hit them, and installation related problems have the virtue of being
> relatively straightforward to pick up in a continuous integration
> system.
>
> Using such a switch wouldn't be the right fit for everyone, but that's
> not the same as it being entirely useless.

Exactly--as both a library developer / maintainer and system
integrator I would find such a flag very useful (especially since I
can just set it in a config file and forget it).  It would be right
for me.  But wouldn't break anything for anyone else.

Ironically, the default behavior of `setup.py install`, on projects
that use setuptools, is to install an egg directory which is
*definitely* not for everybody, especially not anymore.  That's why
--single-version-externally-managed exists.  A --pip flag would be
very much like --single-version-externally-managed (sort of a
specialized extension of it) that also includes "do everything pip
does" which includes installing dependencies and copying
.egg-info/.dist-info to the appropriate location, which is what I want
to replace all instances of `setup.py install` with.  That includes
users running `setup.py install`, who have a hard enough time as it is
keeping up with Python build/installation "best practices" as it is.

Anyways, it has been frequently requested that setuptools change the
default behavior of the "install" command, so I wouldn't discount it
as a valid use case.  So far it hasn't been changed out of
backward-compat concerns, so making it loosely opt-in represents a
possible middle ground.

>> Projects themselves can really just switch to pip commands, same goes for
>> packagers and other tool makers
>>
>> Explicit is better than implicit, and in this case it also won't cost
>> additional maintenance on setuptools.
>> Please keep in mind, that setuptools is completely on volunteer time, and
>> the time given to it is scarce.
>
> Sure, that's why any decision on the desirability of this feature
> would be up to Jason as the setuptools lead. However, there's a
> trade-off to consider here, which is that offering this kind of global
> installer switch may help to lower the priority of some other
> easy_install enhancement requests.

Plus I've contributed to setuptools many times in the past (used to
have commit access on distribute too).  I'm offering to implement and
maintain this feature if it's decided desirable.  I think it *is*
desirable by definition--I desire it, and I suspect others would as
well.  I'd be more interested in technical reasons why it's a bad idea
but I haven't found any yet.

> That's a risk assessment trade-off on future bug reports against
> attempted pip support vs future RFEs against easy_install itself, as
> well as a priority assessment against other open changes proposed for
> setuptools. Those assessments may well come down on the side of "not
> worth the hassle", but the scope of the proposed change still falls a
> long way short of being a "maintenance horror".

I think the only maintenance horror right now is easy_install :)

Thanks,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py install using pip

[Erik Bray]

On Mon, Dec 7, 2015 at 2:40 PM, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 7 December 2015 at 18:58, Erik Bray <erik.m.b...@gmail.com> wrote:
>> I wasn't able to produce this problem.  Even with --no-binary
>> specified pip installs (by default) with
>> --single-version-externally-managed.  My prototype implicitly disables
>> the --pip flag if --single-version-externally-managed was specified
>> (true to the purpose of that flag).
>
> Ah - that was the bit I was missing, the
> --single-version-externally-managed flag can be used to trigger
> ignoring --pip.
>
>> What *is* a problem is if --pip is in setup.cfg, and one invokes `pip
>> install --egg .`.  I wasn't quite able to make that go into an
>> infinite loop, but it did invoke pip.main recursively, and stuff broke
>> on the second invocation for reasons not clear to me.
>
> Yeah, but honestly I don't think pip install --egg is that important a
> use case. I may be wrong (there's lots of ways people use pip that I
> know nothing of :-)) but as a starting point it might be OK just to
> say that at the same time as the --pip flag was introduced, "pip
> install --egg" was deprecated (and we explicitly document that pip
> install --egg is known to interact badly with setup.py --pip).

I'd be fine with that too.  IIRC pip install --egg was introduced in
part to work around problems with namespace packages.  This doesn't
completely eliminate the need for that workaround, but it does reduce
it.

In either case, if the --pip feature is implemented in setuptools it
would still be good to at least have some workaround for installing
with pip install --egg, at least temporarily.  My goal here was to not
break any existing functionality when --pip is specified (either via
command line or setup.cfg).

> Anyway, thanks for the explanation, I see what you're intending now.

Great!
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Installing packages using pip

[Erik Bray]

On Fri, Nov 13, 2015 at 3:09 PM, Nathaniel Smith  wrote:
> On Nov 13, 2015 12:00 PM, "Alexander Walters" 
> wrote:
>>
>> import pip
>> pip.install(PACKAGESPEC)
>>
>> something like that?
>
> This would be extremely handy if it could be made to work reliably... But
> I'm skeptical about whether it can be made to work reliably. Consider all
> the fun things that could happen once you start upgrading packages while
> python is running, and might e.g. have half of an upgraded package already
> loaded into memory. It's like the reloading problem but even more so.

Sorry to resurrect an old thread, but I have an idea about how to do
this somewhat safely, at least insofar as the running interpreter is
concerned.  It's still a terrible idea.  Not such a terrible idea in
principle, but as a practical matter in the context of Python it's
probably a bad idea because it uses yet-another-.pth-hack.

Consider a "partial install", wherein pip installs all files into a
non-imported subdirectory of the target site-packages, along with a
.pth file.  This distribution is then considered "partially installed"
in that the files are their (whether extracted from a wheel, or
installed via distutils and the appropriate --root option or similar).
For example, consider running

>>> pip.install('requests')

It would be up to the pip.install() command to determine whether or
not the requests distribution was already installed.  If it's not
install it would proceed as normal.  For now I'm assuming the user
would still have to manually run `import requests` after this.
Auto-import would be nice, but is a separate issue.

Now, if requests were already installed and imported we don't want to
clobber the existing requests running in the interpreter.  pip would
install install into the relevant site-packages:

<...>/site-packages/requests-2.8.1.part/
requests/
requests-2.8.1.dist-info/
<...>/site-packages/requests-2.8.1.part.pth

The .part/ directory contains the results of the partial installation
(for example the contents of the wheel, for wheel installs).  The
.part.pth file is trickier, but could be something like this:

$ cat requests-2.8.1.part.pth
import inspect, shutil, sys, os, atexit;p =
inspect.currentframe().f_locals['sitedir'];part = os.path.join(p,
'requests-2.8.1.part');files = os.path.isdir(part) and
os.listdir(part);files and list(map(lambda s, d, f:
(sys.modules['shutil'].rmtree(os.path.join(d, f),
sys.modules['shutil'].move(os.path.join(s, f), os.path.join(d, f))),
[part] * len(files), [p] * len(files), files));os.rmdir(part);pth =
part + '.pth';os.path.isfile(pth) and atexit.register(os.unlink,
os.path.abspath(pth))

This rifles through the contents of requests.2.8.1.part, deletes any
existing directories in the parent site-packages of the same name,
completes the install by moving the contents of the .part/ directory
into the correct location and then deletes the .part/ directory.  The
.part.pth later deletes itself.

By the time the user restarts the interpreter and runs `import
requests` this will be completed.  Obvious it would have to be
communicated to the user that to upgrade an existing package they will
have to restart the interpreter which is less than ideal, but relates
to a deeper limitation of Python that they should get used to anyways.
At least this would enable in-process installs/upgrades.

There are of course all kinds of problems with this solution too.  It
should perhaps only work in a virtualenv and/or .local site-packages
(or at least somewhere that the user will have write permissions on
the next interpreter run), and probably other error handling too.  The
above .pth file could also be simplified by invoking a function in pip
to complete any partial installs.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py install using pip

[Erik Bray]

On Mon, Dec 7, 2015 at 11:45 AM, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 7 December 2015 at 16:21, Erik Bray <erik.m.b...@gmail.com> wrote:
>> Exactly--as both a library developer / maintainer and system
>> integrator I would find such a flag very useful (especially since I
>> can just set it in a config file and forget it).  It would be right
>> for me.  But wouldn't break anything for anyone else.
>>
>> Ironically, the default behavior of `setup.py install`, on projects
>> that use setuptools, is to install an egg directory which is
>> *definitely* not for everybody, especially not anymore.  That's why
>> --single-version-externally-managed exists.  A --pip flag would be
>> very much like --single-version-externally-managed (sort of a
>> specialized extension of it) that also includes "do everything pip
>> does" which includes installing dependencies and copying
>> .egg-info/.dist-info to the appropriate location, which is what I want
>> to replace all instances of `setup.py install` with.  That includes
>> users running `setup.py install`, who have a hard enough time as it is
>> keeping up with Python build/installation "best practices" as it is.
>
> One thing that bothers me about this proposal. If someone does "pip
> install --no-binary" for your package, and you have the "--pip" flag
> in your setup.cfg, pip will use "setup.py install" to do the install.
> Which, if I understand this proposal correctly, will attempt to "fall
> back" to pip because "--pip" is in setup.cfg. Which results in an
> infinite loop of pip and setup.py invoking each other.

I wasn't able to produce this problem.  Even with --no-binary
specified pip installs (by default) with
--single-version-externally-managed.  My prototype implicitly disables
the --pip flag if --single-version-externally-managed was specified
(true to the purpose of that flag).

What *is* a problem is if --pip is in setup.cfg, and one invokes `pip
install --egg .`.  I wasn't quite able to make that go into an
infinite loop, but it did invoke pip.main recursively, and stuff broke
on the second invocation for reasons not clear to me.

This is easily worked around, however, by detecting, from the install
command, if we're already using pip.  As a quick hack I added to
finalize_options:

if 'pip' in sys.modules:
self.pip = False

This did the trick.  pip ran fine and installed the package as an egg
using the standard setup.py install.  I don't think a more robust
solution would be hard.  pip could set an environment variable or even
a variable in the pip module itself to indicate that pip is already
being invoked to install this package.  There may even be something
like that already in pip that I'm not aware of.

> I'm not sure how pip could detect a situation like this, so there's a
> risk of some *very* obscure corner cases, which I'm sure people will
> end up hitting.

As mentioned above I don't think it should be pip's job to detect this
situation.  But if the setuptools install command can detect that
we're already in pip then the job is done.

> As a user level command line flag, "setup.py install --pip" isn't much
> better than "pip install ."
> As a project config, we get the issue noted above, and the user has to
> edit the project code to fix it.
> As a per-user or global config, we get the issue above, but we could
> reasonably say it's the user's mistake and the user has the means to
> fix it. But it's still not a great UX.

I don't think so either.  But it's also not great UX that we've told
users for decades that the way to install a Python package is to run
`python setup.py install`, but now the default behavior of that (for
packages using setuptools) which is to install a .egg, is old and bad.
I get confusion about this from users *frequently*.  It's only worse
that egg installs and flat installs are incompatible with each other
with respect to namespace packages.

If it would help, `setup.py install --pip` could also display a
warning that users should run `pip install .` instead.  To cut down on
noise I might only do this if the --pip option came from setup.cfg,
rather than when it's explicitly asked for via the command line.  This
would serve to inform users who don't know any better.

> It's quite possible I'm missing something here (at a minimum, I'm
> making huge assumptions about how the feature would be implemented)
> but I think the behaviour needs to be thought through in a bit more
> detail.

Completely agree!  I know there are corner cases I haven't thought of.
I'm also undecided on whether --pip should invoke pip.main() within
the same process, or run pip from a subprocess.  The former seems
sufficient but I don't know all the cases.

Thanks,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] setup.py install using pip

[Erik Bray]

Hi all,

I've been on vacation for a bit in general, and on vacation from this
mailing list even longer.  I'm not entirely caught up yet on the
latest developments so apologies if something like this is entirely
moot by now.

But I have seen some discussions here and in other lists related to
using pip for all installations, and phasing out the old distutils
`./setup.py install` (eg. [1]).  This is not a new discussion, and
there are many related discussions, for example, about changing
setuptools not to default to egg installs anymore (see [2]).

I'm definitely all for this in general.  Nowadays whenever I install a
package from source I run `pip install .`  But of course there are a
lot of existing tools, not to mention folk wisdom, assuming
`./setup.py install`.  We also don't want to change the long-existing
behavior in setuptools.

I have a modest proposal for a small addition to setuptools that might
be helpful in a transition away from using setuptools+distutils for
installation.  This would be to add a `--pip` flag to setuptools'
install command (or possibly straight in distutils too, but might as
well start with setuptools).

Therefore, running

$ ./setup.py install --pip

would be equivalent to running

$ pip install .

By extension, running

$ ./setup.py install --pip --arg1 --arg2=foo

would be equivalent to

$ pip install --install-option="--arg1" --install-option="--arg2=foo" .

and so on.

By making `--pip` opt-in, it does not automatically break backward
compatibility for users expecting `./setup.py install` to use
easy_install.  However, individual users may opt into it globally by
adding

[install]
pip = True

to their .pydistutils.cfg.  Similarly, package authors who are
confident that none of their users are ever going to care about egg
installs (e.g. me) can add the same to their project's setup.cfg.

Does something like this have any merit?  I hacked together a
prototype which follows the sig line.  It's just a proof of concept,
but seems to work in the most basic cases.  I'd like to add it to my
own projects too, but would appreciate some peer review.

Thanks,
Erik



[1] https://mail.scipy.org/pipermail/numpy-discussion/2015-November/074142.html
[2] 
https://bitbucket.org/pypa/setuptools/issues/371/setuptools-and-state-of-pep-376


$ cat pipinstall.py
from distutils.errors import DistutilsArgError
from setuptools.command.install import install as SetuptoolsInstall


class PipInstall(SetuptoolsInstall):
command_name = 'install'
user_options = SetuptoolsInstall.user_options + [
('pip', None, 'install using pip; ignored when also using '
  '--single-version-externally-managed')
]
boolean_options = SetuptoolsInstall.boolean_options + ['pip']

def initialize_options(self):
SetuptoolsInstall.initialize_options(self)
self.pip = False

def finalize_options(self):
SetuptoolsInstall.finalize_options(self)

if self.single_version_externally_managed:
self.pip = False

if self.pip:
try:
import pip
except ImportError:
raise DistutilsArgError(
'pip must be installed in order to install with the '
'--pip option')

def run(self):
if self.pip:
import pip
opts = (['install', '--ignore-installed'] +
['--install-option="{0}"'.format(opt)
 for opt in self._get_command_line_opts()])
pip.main(opts + ['.'])
else:
SetuptoolsInstall.run(self)

def _get_command_line_opts(self):
# Generate a mapping from the attribute name associated with a
# command-line option to the name of the command-line option (including
# an = if the option takes an argument)
attr_to_opt = dict((opt[0].rstrip('=').replace('-', '_'), opt[0])
   for opt in self.user_options)

opt_dict = self.distribution.get_option_dict(self.get_command_name())
opts = []

for attr, value in opt_dict.items():
if value[0] != 'command line' or attr == 'pip':
# Only look at options passed in on the command line (ignoring
# the pip option itself)
continue

opt = attr_to_opt[attr]

if opt in self.boolean_options:
opts.append('--' + opt)
else:
opts.append('--{0}{1}'.format(opt, value[1]))

return opts

@staticmethod
def _called_from_setup(run_frame):
# A hack to work around a setuptools hack
return SetuptoolsInstall._called_from_setup(run_frame.f_back)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distutils.command.build_clib: How to add additional compiler flags for cl.exe?

[Erik Bray]

On Sun, Nov 29, 2015 at 10:44 AM, Kim Walisch  wrote:
> Hi,
>
> For distutils.command.build_clib the commonly used code below does not
> work for adding additional compiler flags (it works using
> distutils.command.build_ext):
>
> extra_compile_args = '-fopenmp'
>
> On Unix-like system I found a workaround which allows to specify
> additional compiler flags for distutils.command.build_clib:
>
> cflags = distutils.sysconfig.get_config_var('CFLAGS')
> distutils.sysconfig._config_vars['CFLAGS'] = cflags + " -fopenmp"
>
> Unfortunately this does not work with Microsoft's C/C++ compiler
> cl.exe.
>
> Does anybody know how I can add additional compiler flags for cl.exe
> and distutils.command.build_clib?
>
> Thanks and best regards!

Really what I think you want is a way to determine what compiler will
be used, and to add compiler arguments accordingly.  Unfortunately
distutils does not provide one easy way to determine what compiler it
will use--this is among the myriad ways it's not great for use as a
build system.

And yet there are several workarounds.  For starters you can try:

from distutils import ccompiler

compiler = ccompiler.get_default_compiler()

if compiler == 'msvc':
# Add option for MS compiler
elif compiler == 'unix':
# Add option for gcc
elif # etc

This does not actually tell you what compiler executable will be
used--for that you have to dig around more.  This is telling you the
name of the compiler class in distutils that will be used.  But if you
want to pass arguments for cl.exe, it will be 'msvc'.

This also does not guarantee which compiler will actually be used--a
user can override that via a command-line argument or setup.cfg.  To
get around that I usually use a dummy 'Distribution' object to parse
any user arguments.  This isn't perfect either, but it's almost
exactly what happens when the setup() function is called--it just
stops before actually running any of the commands:

from distutils.dist import Distribution
from distutils import ccompiler

def get_compiler():
dist = Distribution({'script_name': os.path.basename(sys.argv[0]),
'script_args': sys.argv[1:]})
dist.parse_config_files()
dist.parse_command_line()
compiler = None

for cmd in ['build', 'build_ext', 'build_clib']:
compiler = dist.command_options.get(cmd, {}).get('compiler',
('', None))[1]
if compiler is not None:
break

if compiler is None:
return ccompiler.get_default_compiler()
else:
return compiler


I'd love to know if there is a better way in general to do this myself...


Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.py install using pip

[Erik Bray]

On Thu, Dec 3, 2015 at 3:37 PM, Ronny Pfannschmidt
<opensou...@ronnypfannschmidt.de> wrote:
> Lets avoid getting setuptools even more complex in that way

It's not deeply complex--it's just bypassing the normal behavior of
using easy_install.  In my example code I subclassed
the existing command, but an easier approach would be to build it into
setuptools.

The way the 'install' command works in setuptools is to hand
installation off to the easy_install command unless
--single-version-externally-managed is specified (as well as
--record).  Otherwise it hands installation off to the default
base 'install' command of distutils.

This proposal just adds a third option for what actual installer the
'install' command should
use.  But it's opt-in instead of forced on any package by default (as
easy_install is forced on us by setuptools).

> Putting pip-ish-ness on top of easy install is a maintenance horror and I 
> don't think setuptools does has enough consistent developer resources to 
> handle something like that

It doesn't put anything "on top of" easy_install; it ignores easy_install.

> Instead let's just give options to destroy the normal install command from 
> setup.py so projects can phase out easy install forcefully making downstream 
> require patches or pip usage

That's the goal, yes.  This is just offering a transitional tool.


Erik

>
> Am 3. Dezember 2015 21:06:06 MEZ, schrieb Erik Bray <erik.m.b...@gmail.com>:
>>Hi all,
>>
>>I've been on vacation for a bit in general, and on vacation from this
>>mailing list even longer.  I'm not entirely caught up yet on the
>>latest developments so apologies if something like this is entirely
>>moot by now.
>>
>>But I have seen some discussions here and in other lists related to
>>using pip for all installations, and phasing out the old distutils
>>`./setup.py install` (eg. [1]).  This is not a new discussion, and
>>there are many related discussions, for example, about changing
>>setuptools not to default to egg installs anymore (see [2]).
>>
>>I'm definitely all for this in general.  Nowadays whenever I install a
>>package from source I run `pip install .`  But of course there are a
>>lot of existing tools, not to mention folk wisdom, assuming
>>`./setup.py install`.  We also don't want to change the long-existing
>>behavior in setuptools.
>>
>>I have a modest proposal for a small addition to setuptools that might
>>be helpful in a transition away from using setuptools+distutils for
>>installation.  This would be to add a `--pip` flag to setuptools'
>>install command (or possibly straight in distutils too, but might as
>>well start with setuptools).
>>
>>Therefore, running
>>
>>$ ./setup.py install --pip
>>
>>would be equivalent to running
>>
>>$ pip install .
>>
>>By extension, running
>>
>>$ ./setup.py install --pip --arg1 --arg2=foo
>>
>>would be equivalent to
>>
>>$ pip install --install-option="--arg1" --install-option="--arg2=foo" .
>>
>>and so on.
>>
>>By making `--pip` opt-in, it does not automatically break backward
>>compatibility for users expecting `./setup.py install` to use
>>easy_install.  However, individual users may opt into it globally by
>>adding
>>
>>[install]
>>pip = True
>>
>>to their .pydistutils.cfg.  Similarly, package authors who are
>>confident that none of their users are ever going to care about egg
>>installs (e.g. me) can add the same to their project's setup.cfg.
>>
>>Does something like this have any merit?  I hacked together a
>>prototype which follows the sig line.  It's just a proof of concept,
>>but seems to work in the most basic cases.  I'd like to add it to my
>>own projects too, but would appreciate some peer review.
>>
>>Thanks,
>>Erik
>>
>>
>>
>>[1]
>>https://mail.scipy.org/pipermail/numpy-discussion/2015-November/074142.html
>>[2]
>>https://bitbucket.org/pypa/setuptools/issues/371/setuptools-and-state-of-pep-376
>>
>>
>>$ cat pipinstall.py
>>from distutils.errors import DistutilsArgError
>>from setuptools.command.install import install as SetuptoolsInstall
>>
>>
>>class PipInstall(SetuptoolsInstall):
>>command_name = 'install'
>>user_options = SetuptoolsInstall.user_options + [
>>('pip', None, 'install using pip; ignored when also using '
>>  '--single-version-externally-managed')
>>]
>>boolean_options = SetuptoolsInstall.boolean_options + ['pip']
>>
>>def initialize_options(self):
>>  

Re: [Distutils] Packaging shared objects with setuptools

[Erik Bray]

On Fri, Oct 30, 2015 at 12:47 PM, Mario Pezzoni  wrote:
> Hello,
>
> I am wrapping a c++ library with cython. I compile the pyxs and the c++ code
> with cmake using the cython-cmake-example.
> If I import the package from the build directory it works flawlessly, if I
> install through "python setup.py install" into a virtual environment it
> breaks because the shared objects are not installed (but all the pure .py
> modules are installed).
>
> How do I tell setuptools to install the shared objects?

For starters, what does your setup.py look like?  You should make sure
all extension modules are registered with setup() as described here:

https://docs.python.org/2/distutils/examples.html#single-extension-module

I'm not sure what cython-cmake-example is though, or how whatever it
does would interact with distutils.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] tests location: Use case: new comers and docs.

[Erik Bray]

On Wed, Oct 7, 2015 at 2:11 AM, Chris Barker  wrote:
> On Tue, Oct 6, 2015 at 10:38 PM, Thomas Güttler
>  wrote:
>>
>> Yes, there is not generic "one right way here".
>>
>> Yes, let's consider individual use cases.
>>
>> My use case are the docs for new comers:
>>
>>  - https://github.com/pypa/sampleproject
>>  - https://packaging.python.org/en/latest/distributing/
>>
>> That's why started the thread.
>
>
> unfortunately, that isn't  a use-case -- every newcomer has a different use
> case.

Indeed--I've helped newcomers whose very first attempt at packaging
Python code includes Cython code for simulations.  I think even for
newcomers what we should be providing is not a "this is the way to do
it" because then they get confused when that way doesn't work for
them.

Better, in the long term (and I'm happy to contribute to such an
effort if it will help) is to provide a sort of Choose Your Own
Adventure story.  It can't all go on one page because that would be a
mess, but a sort of "If you need to do this, read this.  If you need
to do this, read this.  Now if you need to include some data files
that are installed in your package read on, because there's really
only one right way to do that.  But now you have some options if you
want to include tests: ..."

> I was happy to see this thread, because I thought maybe I"d learn what i
> should teach my students - new to python.
>
> But alas - there clearly really is no consensus.
>
> What i've told newbies in the past is somethig like:
>
> """
> if you want your user to be able to install you package, and then run
> something like:
>
> import my_package
> my_package.test()
>
> then put your tests inside the package.
>
> If you are fine with only being able to run the tests from the source tree
> -- then put your tests outside the package.
> """
>
> but really, newbies have no idea how to make this decsion.
>
> Maybe we could come up with a decision tree for this -- some guidance for
> knowing what to do, when?

Exactly.  I think it could even be fun :)

How could we get started to add something like this to the packaging docs?

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?)

[Erik Bray]

On Tue, Oct 6, 2015 at 7:46 PM, Ionel Cristian Mărieș
 wrote:
>
> On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins 
> wrote:
>>
>>
>> Hangon, there's clearly a *huge* gap in understanding here.
>>
>> pbr does *not* modify *anyones* setup.py output unless its enabled.
>
>
> Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr
> doing weird stuff to distributions by abusing said entrypoints.
>
> For reference: https://bugs.launchpad.net/pbr/+bug/1483067
>
> There's nothing special about pbr here. It's not like it's the first package
> doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like
> pdbpp, but you don't put that in production.

Starting a sub-thread since issues with pbr are orthogonal to the
original disucssion.

But one point I'd like to raise about this is that when I originally
designed d2to1, on which a chunk of pbr is based, it was *explicitly*
designed to never be installed in site-packages (with the exception of
downstream packaging systems which can do what they want and are more
controlled).  This is exactly because I knew different packages might
have dependencies on different versions of d2to1 as features are
added, and that if some version is installed in site-packages it can
lead to VersionConflict issues (this is in part exacerbated by a
bug/misfeature in setuptools--I fixed that bug a while ago but the fix
had to be rolled back due to a regression [1]).

So TL;DR unless you know what you're doing, d2to1 should *never* be
"installed"--it was only meant to be used with setup_requires, where
the appropriate d2to1 used in building/installing a package is only
temporarily enabled on sys.path via a temporary egg install.  If some
project is making it a *runtime* requirement that's a mistake.

I don't know what features pbr has grown that might make someone want
it to be a runtime dependency (the only entry-points I noticed were
for adding egg-info writers but that should only be needed at
build-time too...), but maybe something like that should be split off
as a separate module or something...

Best,
Erik


[1] 
https://bitbucket.org/tarek/distribute/pull-requests/20/fixes-and-adds-a-regression-test-for-323/diff
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Erik Bray]

On Tue, Oct 6, 2015 at 6:08 PM, Ionel Cristian Mărieș
 wrote:
>
> On Wed, Oct 7, 2015 at 12:51 AM, Ben Finney 
> wrote:
>>
>> I think the above describes the standard way of declaring the test
>> runner: The ‘setup.py test’ command.
>>
>> Now, I lament that more Python projects don't *conform to* that
>> standard, but at least it exists.
>
>
> There's a very simple answer to that: easy_install (that's what `setup.py
> test` will use to install deps). It has several design issue wrt how
> packages are installed and how dependencies are managed.
>
> Lets not use `setup.py test`. It's either bad or useless.

Says who?  Many of the projects I'm involved in use `setup.py test`
exclusively and for good reason--they all have C and/or Cython
extension modules that need to be built for the tests to even run.
Only setup.py knows about those extension modules and how to find and
build them.  Using `setup.py test` ensures that everything required to
run the package (including runtime dependencies) is built and ready,
and then the tests can start.  Without it, we would have to tell
developers to go through a build process first and then make sure
they're running the tests on the built code.  `setup.py test` makes it
a no-brainer.

For pure Python packages I think it's less important and can usually
rely on "just run 'nose', or 'py.test'"  (or "tox" but that's true
regardless of how the tests are invoked outside of tox).

Best,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Erik Bray]

On Wed, Oct 7, 2015 at 11:31 AM, Ionel Cristian Mărieș
<cont...@ionelmc.ro> wrote:
>
> On Wed, Oct 7, 2015 at 6:13 PM, Erik Bray <erik.m.b...@gmail.com> wrote:
>>
>> > Lets not use `setup.py test`. It's either bad or useless.
>>
>> Says who?  Many of the projects I'm involved in use `setup.py test`
>> exclusively and for good reason--they all have C and/or Cython
>> extension modules that need to be built for the tests to even run.
>> Only setup.py knows about those extension modules and how to find and
>> build them.  Using `setup.py test` ensures that everything required to
>> run the package (including runtime dependencies) is built and ready,
>
>
> Well ok, then it's not useless. :-)
>
>> For pure Python packages I think it's less important and can usually
>> rely on "just run 'nose', or 'py.test'"  (or "tox" but that's true
>> regardless of how the tests are invoked outside of tox).
>
>
> That implies you would be testing code that you didn't install. That allows
> preventable mistakes, like publishing releases on PyPI that don't actually
> work, or do not even install at all (because you didn't test that).
> `setup.py test` doesn't really allow you to fully test that part, but Tox
> does.

Which, incidentally, is a great reason for installable tests :)
Running in the source tree is great for development.  But when
preparing a release it's great to be able to create an sdist, install
that into a virtualenv, and run `package.test()` or `python -m
package.tests` or whatever.  Occasionally catches problems with the
source dist if nothing else.

Best,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Erik Bray]

On Tue, Oct 6, 2015 at 8:34 AM, Ionel Cristian Mărieș
 wrote:
> On Tue, Oct 6, 2015 at 3:13 PM, Antoine Pitrou  wrote:
>>
>> On Tue, 6 Oct 2015 07:07:31 -0400
>> Donald Stufft  wrote:
>> >
>> > I've never, in my entire life [...]
>>
>> Can I suggest your entire life is an anecdotal data point here?
>
>
> Make that two anecdotal data points :-)
>
>> Any number of things can be described as trivial depending on the
>> skillset and patience of the user.  When users report a bug, they are
>> not expecting to be asked to download and "untar" stuff.  Not every
>> user is a programmer.
>
> But seriously now, your arguments are also anecdotal. Lets not pretend we're
> objective here. That sort of attitude is disingenuous and will quickly
> devolve this discussion to mere ad hominems.

Okay, though, so maybe if there is nothing to offer here but anecdata
then maybe we should stop acting like there's "one right way here".  I
have projects that install their test suite and test dependencies
because it is frequently useful to ask users to run a self-test (and
users themselves want to be able to do it for a variety of reasons).

There are other projects where it doesn't make sense, and those don't
have to install the tests (I still think in those cases that the tests
should live in the package instead of outside it, but simply not
installed).

In any case, let's not get trapped into endless circular discussions
about what is correct, period, and instead consider individual use
cases--not dismissing individual projects' or peoples' experiences and
needs--and discuss what the most appropriate action is for those uses
cases.  Python projects are not monolithic in their audiences (and
that includes developer audiences and user audiences).

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Erik Bray]

On Tue, Oct 6, 2015 at 12:04 PM, Ionel Cristian Mărieș
<cont...@ionelmc.ro> wrote:
>
> On Tue, Oct 6, 2015 at 6:33 PM, Erik Bray <erik.m.b...@gmail.com> wrote:
>>
>> Okay, though, so maybe if there is nothing to offer here but anecdata
>> then maybe we should stop acting like there's "one right way here".  I
>> have projects that install their test suite and test dependencies
>> because it is frequently useful to ask users to run a self-test (and
>> users themselves want to be able to do it for a variety of reasons).
>>
>> There are other projects where it doesn't make sense, and those don't
>> have to install the tests (I still think in those cases that the tests
>> should live in the package instead of outside it, but simply not
>> installed).
>
>
> To be honest, I like the idea of providing the tests in the installed
> package. Using the test suite as a debugging/diagnostic tool is obviously
> desirable. It's just that it's unpractical for general use. Another anecdote
> I know, but humour these two concerns:

Those are all fair questions, and I can't say I have great answers for
all of them.  But that's a fair point that if installable tests *are*
recommended as a practice then they should be addressed.

Before continuing on there are really three options being discussed here:

1) Tests totally outside the package.
2) Tests in the package but not installed (only in source dists / VCS checkout)
3) Tests in the package and some or all installed.

Skimming back through the rest of the thread I don't see too much
support for 1).  The only argument against it is the need for
specifying dependencies, etc., which really only impacts developers so
long as the tests aren't *installed*, I think.  But there's also the
question of what kinds of tests we're talking about.  I think unit
tests should live in the .tests for a library.  Other
kinds of tests I don't have a strong opinion about.

So in any case if we did recommend putting tests in a subpackage we
could still do so without making a decree as to whether or not they
should be installed.  Maybe by default suggest that they not be
installed, and only have them installable if the developer has a plan
for addressing the questions below.  I think each of these questions
may have different answers depending on the needs of different
projects.

> * How to deal with dependencies?

What dependencies exactly?  Dependencies just needed for running the
tests (as opposed to running the code being tested?  Some of my code
has optional dependencies, and the tests for that code are only run if
those optional dependencies are satisfied, but that's not what we're
talking about right?)

> ** Should we use extras? Installing test deps has the disadvantage of
> version conflicts. Unless we make some sort of "virtualenv-for-tests" tool?

I'm rather fond of mktmpenv in virtualenv-wrapper, but I admit that
may be out of scope here...

Not to point to using easy_install as best practice, but something
resembling the way setup_requires could actually work
here--dependencies are downloaded into a temp dir and installed there
as eggs; added to sys.path.  No need for something as heavy-weight as
a virtualenv.  This still has some potential for VersionConflict
errors if not handled carefully though.  But I might try out something
like this and see where I get with it, because I think it would be
useful.

> ** Should we vendor the deps? But isn't that maybe too hard to do (or plain
> wrong for other reasons)?
> ** Should we avoid having deps? That can be too limiting in some situations,
> unittest is very bare compared to pytest or nose.

py.test comes with a command to generate a bundled copy of py.test for
vendoring: 
https://pytest.org/latest/goodpractises.html#deprecated-create-a-pytest-standalone-script
 (for reasons I've lost track of it's apparently deprecated now
though?)  I've used this with quite a bit of success to support
installed tests.  It allows me to ship a version of py.test that works
with my tests along with the package, and there is no issue with
version conflicts or anything.  In principle the same trick could be
used to bundle other dependencies.

In practice it's a little more complicated because downstream
packagers don't like this, but it's easy to remove the bundled py.test
and use the one provided by the system instead (as in on Debian).  In
that case we do have to work with the downstream packagers to make
sure all the tests are working for them.  I don't think this will be
be much of an issue for the average pure-Python package.

> * What sort of tests should be included? Integration tests are special case,
> if they need external services, temporary storage or what not. Would we then
> need to have clear separation for different types of tests?

I try to separate out tests like this and disable them by defa

Re: [Distutils] intelcompiler class in distutils

[Erik Bray]

On Wed, Sep 30, 2015 at 3:17 PM, Hogan, Christopher
 wrote:
> Hello,
>
>
>
> We noticed there are classes in distutils for specific compilers like
> Borland, Cygwin, MSVC, etc., and are interested in creating a class for the
> Intel compiler.  Is this something that the python developers/community
> would like to see, or would such a patch have a low chance for acceptance?

Hi,

I don't have any weight to add one way or the other, but I have
encountered a need for this in the past (though it was a long time ago
and I can't recall the context).  It might be tricky to get it
accepted into distutils in the stdlib, if only because it would have
to come with some promise of continued maintenance.  And also getting
anything added to distutils these days is hard (though last I recall
the freeze may be liftable for non-backwards-compat-breaking new
features where needed?)

That said, you might have better luck adding it to setuptools instead,
or a completely separate third-party package.  Adding support for a
new compiler that is *not* directly in the distutils package requires
a bit of hackery though.  Incidentally, d2to1 [1] actually supports
packages that include their own compiler class, and allows them to use
that compiler for builds.  I don't know that this feature has ever
actually been used, but my point in bringing it up is just to
demonstrate that it can be done.  And you can look at that code for an
example of how to add a new compiler without it being directly in the
distutils module (even if it's a bit of a hack).

Best,

Erik

[1] https://github.com/embray/d2to1/blob/master/d2to1/util.py#L330
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip install from source dir??

[Erik Bray]

On Fri, Oct 2, 2015 at 5:39 PM, Donald Stufft  wrote:
> ``pip install path/to/directory/with/setup.py/in/it/``

In particular I install most of my packages from source these days by
cd'ing into the source and

$ pip install .

That's all it takes.  Works with -e too.

Erik

> On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.bar...@noaa.gov) wrote:
>> I can't seem to find a way to pip install from a source dir -- not a
>> tarball.
>>
>> Why would I want to do that?
>>
>> In this case, it's because I'm trying to build conda packages for python
>> packages that have compiled C code, and are not using setuptools.
>>
>> On Windows, you need to use setuptools in order to use the "MS compiler for
>> Python 2.7". So a straight "setup.py build" fails.
>>
>> But pip injects setuptools, so that "pip install" does work. But conda has
>> already unpacked the source distribution, so I'd like to point pip at that,
>> and can't find a way to do that (I could have sworn it was possible...)
>>
>> Do I have to build an sdist, and then point pip at that??
>>
>> -CHB
>>
>>
>>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Problem Report

[Erik Bray]

On Thu, Aug 13, 2015 at 4:42 AM, 俞博文 steven...@hotmail.com wrote:
 Dear Maintainers:

 This problem occurred when
 1. Windows platform
 2. Python is installed on non-Latin path (for example: path contains Chinese
 character).
 3. try to pip install theano

 And I found the problem is in distutils.command.build_scripts module's
 copy_scripts function, on line 106

 executable = os.fsencode(executable)
 shebang = b#! + executable + post_interp + b\n
 try:
 shebang.decode('utf-8')

 actually os.fsencode will encode the path into GBK encoding on windows, it's
 certainly that will fail to decode via utf-8.

 Solution:

 #executable = os.fsencode(executable) (delete this line)
 executable = executable.encode('utf-8')

 Theano successfully installed after this patch.

Hi,

This is a bit tricky--I think, from the *nix perspective, using
os.fsencode() looks like the correct approach here.  However, if
sys.getfilesystemencoding() != 'utf-8', and if the result of
os.fsencode(executable) is not decodable as utf-8, then that's going
to be a problem for the Python interpreter which begins reading a file
as utf-8 until it gets to the coding token.

Unfortunately this is a bit contradictory--if the path to the
interpreter in the local filesystem encoding is not UTF-8 it is
impossible to parse that file in Python.  On Windows this shouldn't
matter--I agree with your patch, that it should just write the shebang
line in UTF-8.  However, on *nix systems it really should be using
os.fsencode, I think.

I wonder if this was brought up in the discussion around PEP-263.  I
feel like as long as the file encoding is declared to be the same as
whatever encoding was used the write the shebang line, that this
should be valid.  However, the Python interpreter still tries to
interpret the shebang line as UTF-8, and hence falls over in your
case.  This is unfortunate...

Best,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Setuptools 18.0b1 available for testing (improved Cython build support).

[Erik Bray]

On Wed, Jun 10, 2015 at 1:25 PM, Jason R. Coombs jar...@jaraco.com wrote:
 I’m pleased to announce a fix for improved Cython support per Setuptools
 #288 (https://bitbucket.org/pypa/setuptools/issue/288). I’ve published a
 beta release 18.0b1 in the Setuptools project downloads
 (https://bitbucket.org/pypa/setuptools/downloads/setuptools-18.0b1.zip) for
 pre-release testing. Release notes are here
 (https://bitbucket.org/pypa/setuptools/src/4249669040239bff20196b17ccd4d73870d37343/CHANGES.txt#cl-10).

 Please, if you use any projects with Cython modules, give this install a
 trial run and report any issues in the Setuptools bug tracker. I hope to
 release 18.0 in the next few weeks if there aren’t any emergent blockers.

Thanks for this, I'll try it out when I get a chance.  Although I
don't think I've run into this issue often, as I don't usually put
Cython in setup_requires, as I prefer to package sources generated by
Cython in the source dist (so users installing from the source dist
don't need Cython themselves, and I also get more control over the
Cython version used).

That said, I've wanted to work out a schema for adding Cython to
setup_requires only for source checkouts (but remove it when making a
source dist), just as a development convenience.  I guess I've just
never tried it though...or maybe did and realized it was buggy and
forgot about it.  Good to know that it *should* work now.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine

[Erik Bray]

On Tue, Jun 2, 2015 at 3:41 AM, Robert Collins
robe...@robertcollins.net wrote:
 On 2 June 2015 at 02:51, Erik Bray erik.m.b...@gmail.com wrote:
 On Sun, May 31, 2015 at 5:07 PM, AJ Friend ajfri...@gmail.com wrote:
 Hi,

 I'm trying to write a new `setup.py` file for an extension module to
 wrap a C library (https://github.com/cvxgrp/scs).

 The current `setup.py` file imports numpy. I'm trying to delay that
 import statement until setuptools has a chance to install numpy if
 it's not already installed. I'm trying to do that with this bit of
 code:

 from setuptools.command.build_ext import build_ext as _build_ext
 class build_ext(_build_ext):
 def finalize_options(self):
 _build_ext.finalize_options(self)
 # Prevent numpy from thinking it is still in its setup process:
 __builtins__.__NUMPY_SETUP__ = False
 import numpy
 self.include_dirs += ext['include_dirs'] + [numpy.get_include()]

 Running `python setup.py install` seems to work fine on my OSX
 machine, but when I run `pip install .` in the directory with
 `setup.py`, I get a clang error that it can't find one of the header
 files.

 Any idea why that would be happening? Could it have anything to do
 with the relative path I'm giving for the include directories?

 Also, I had trouble finding good documentation on subclassing
 build_ext. Does anyone know if setting self.include_dirs overwrites or
 appends to the include_dirs attribute of an Extension object defined
 later in setup.py?

 For the curious, my current attempt at setup.py is
 athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The
 original can be found in the same directory.

 More generally, since I'm new to python packaging, I'm not sure how
 well or correctly I've written my `setup.py` file. Any feedback on
 doing things correctly would be appreciated.

 Hi AJ,

 For a lot of things in Python packaging there is not, sadly, One Right
 Way to Do It.  Your setup.py looks okay though.

 You may want to have a look at the get_numpy_include_path utility here:
 https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66

 It's similar to what you're already doing, but maybe a little more
 'robust'.  In particular, I think the reload of the numpy module may
 be important.

 It is, because what numpy is doing is importing from its own tree
 before setup has completed - which is in general unsafe.

 And numpy may be in process depending on exactly what setuptools
 setup_requires easy-install code has done, which can lead to the
 symptoms you see. This should be better once we get pip interpreting
 setup_requires for you, but a perhaps shorter term fix would be to
 factor out the distutils support glue from numpy (which seems pretty
 self contained) into a out of package file - either adjacent to
 setup.py, or, if numpy is willing to make the dependency on setuptools
 a hard dep (which I recommend), then as a separate package which can
 be setup_require'd.

 That would avoid the bootstrapping complexity, resulting in numpy not
 being imported until it has actually been installed, and things should
 be rosy for folk downstream of numpy from that point on.

That might not be a bad idea.  This is more or less what we did for
Astropy with astropy-helpers (this was done also so that other
software distributions in the Astropy affiliated package ecosystem
could take advantage of it).

I think maybe splitting out much of the stuff in numpy.distutils into
its own package (though unfortunately it would need a new name since I
don't think we could make numpy into a namespace package) might help.
Though I'm not sure how that would work for numpy.get_includes().
Would it just figure out where the include files *would* be installed
to if one were installing Numpy into the currently active environment?

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine

[Erik Bray]

On Sun, May 31, 2015 at 5:07 PM, AJ Friend ajfri...@gmail.com wrote:
 Hi,

 I'm trying to write a new `setup.py` file for an extension module to
 wrap a C library (https://github.com/cvxgrp/scs).

 The current `setup.py` file imports numpy. I'm trying to delay that
 import statement until setuptools has a chance to install numpy if
 it's not already installed. I'm trying to do that with this bit of
 code:

 from setuptools.command.build_ext import build_ext as _build_ext
 class build_ext(_build_ext):
 def finalize_options(self):
 _build_ext.finalize_options(self)
 # Prevent numpy from thinking it is still in its setup process:
 __builtins__.__NUMPY_SETUP__ = False
 import numpy
 self.include_dirs += ext['include_dirs'] + [numpy.get_include()]

 Running `python setup.py install` seems to work fine on my OSX
 machine, but when I run `pip install .` in the directory with
 `setup.py`, I get a clang error that it can't find one of the header
 files.

 Any idea why that would be happening? Could it have anything to do
 with the relative path I'm giving for the include directories?

 Also, I had trouble finding good documentation on subclassing
 build_ext. Does anyone know if setting self.include_dirs overwrites or
 appends to the include_dirs attribute of an Extension object defined
 later in setup.py?

 For the curious, my current attempt at setup.py is
 athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The
 original can be found in the same directory.

 More generally, since I'm new to python packaging, I'm not sure how
 well or correctly I've written my `setup.py` file. Any feedback on
 doing things correctly would be appreciated.

Hi AJ,

For a lot of things in Python packaging there is not, sadly, One Right
Way to Do It.  Your setup.py looks okay though.

You may want to have a look at the get_numpy_include_path utility here:
https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66

It's similar to what you're already doing, but maybe a little more
'robust'.  In particular, I think the reload of the numpy module may
be important.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI and Uploading Documentation

[Erik Bray]

On Fri, May 15, 2015 at 9:48 AM, Donald Stufft don...@stufft.io wrote:
 Hey!

 First, for anyone who isn't aware we recently migrated PyPI and TestPyPI so
 that instead of storing files and documentation locally (really in a glusterfs
 cluster) it will store them inside of S3. This will reduce maintenance 
 overhead
 of running PyPI by two servers since we'll no longer need to run our own
 glusterfs cluster as well as improve the reliaiblity and scalability of the
 PyPI service as a whole since we've had nothing but problems from glusterfs in
 this regard.

 One of the things that this brought to light was that the documentation
 upload ability in PyPI is something that is not often used* however it
 represents something which is one of our slowest routes. It's not a well
 supported feature and I feel that it's going outside of the core competancy 
 for
 PyPI itself and instead PyPI should be focused on the files themselves. In
 addition since the time this was added to PyPI a number of free services or
 cheap services have came about that allow people to sanely upload raw document
 without a reliance on any particular documentation system and we've also had
 the rise of ReadTheDocs for when someone is using Sphinx as their 
 documentation
 system.

 I think that it's time to retire this aspect of PyPI which has never been well
 supported and instead focus on just the things that are core to PyPI. I don't
 have a fully concrete proposal for doing this, but I wanted to reach out here
 and figure out if anyone had any ideas. The rough idea I have currently is to
 simply disable new documentation uploads and add two new small features. One
 will allow users to delete their existing documentation from PyPI and the 
 other
 would allow them to register a redirect which would take them from the current
 location to wherever they move their documentation too. In order to prevent
 breaking documentation for projects which are defunct or not actively
 maintained we would maintain the archived documentation (sans what anyone has
 deleted) indefinetely.

 Ideally I hope people start to use ReadTheDocs instead of PyPI itself. I think
 that ReadTheDocs is a great service with heavy ties to the Python community.
 They will do a better job at hosting documentation than PyPI ever could since
 that is their core goal. In addition there is a dialog between ReadTheDocs and
 PyPI where there is an opportunity to add integration between the two sites as
 well as features to ReadTheDocs that it currently lacks that people feel are a
 requirement before we move PyPI's documentation to read-only.

 Thoughts?

 * Out of ~60k projects only ~2.8k have ever uploaded documentation. It's not
   easy to tell if all of them are still using it as their primary source of
   documentation though or if it's old documentation that they just can't
   delete.

+1 for all the stated reasons.

I have a few docs hosted on pythonhosted.org, but it's become a
nuisance to maintain since it does not support multiple doc versions
like ReadTheDocs, so now I've wound up with documentation for the same
projects on both sites.  The nuisance comes not so much in the process
(like Barry wrote, I've enjoyed the simplicity of `setup.py
upload_docs`), but because more often than not I've had to redirect
users to the Readthedocs docs to make sure they're using the correct
version of the docs.  So I wish I were not locked into updating the
pythonhosted.org docs and would be happy to retire them altogether
(much as I appreciated the service).

One question is how this would be handled at the tooling end.
setup.py upload_docs would have to be retired somehow.  Though it
might also be nice if some simple tools were added to make it just as
easy to add docs to ReadTheDocs.  I know something like upload_docs
doesn't really make sense, since RTD handles the checkout and build of
the docs.  But there's still a manual step of enabling new versions of
the docs that it would be nice to make as effortless as `setup.py
upload_docs`.  I gues that's off-topic for the PyPI end of things
though.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] d2to1 setup.cfg schema

[Erik Bray]

On Tue, Mar 24, 2015 at 6:51 PM, Nick Coghlan ncogh...@gmail.com wrote:

 On 25 Mar 2015 07:35, Robert Collins robe...@robertcollins.net wrote:

 This is a break-out thread from the centi-thread that spawned about
 setup-requires.

 d2to1 defined some metadata keys in setup.cfg,in particular 'name' and
 'requires-dist'. Confusing 'requires-dist' contains the
 'install_requires' one might use with setuptools' setup() function.

 That particular name comes from PEP 345:
 https://www.python.org/dev/peps/pep-0345/

 Extending d2to1 to accept install-requires as meaning the same thing as
 the existing requires-dist (and complaining if a setup.cfg file contains
 both) would make sense to me, as it provides a more obvious migration path
 from setuptools, and pairs up nicely with a new setup-requires section for
 setup.py dependencies.

I would be fine with that, and other similar changes.  Better
documentation of the format is needed too--it used to just rely on the
distutils2 documentation, but since distutils2 is dead d2to1 deserves
documentation in its own right.

 (It also occurs to me that we should probably ask the d2to1 folks if they'd
 be interested in bringing the project under the PyPA banner as happened with
 setuptools, distlib, etc. It's emerged as a key piece of the transition from
 Turing complete build process customisation to static build metadata
 configuration)

As the d2to1 folks, more or less, (not counting the pbr folks who've
done their own thing and might have opinions), I would be fine with
this.  It has been on my agenda for over a year to release an update
to d2to1 under a new name--something less tied to the failed
distutils2 project (along with its own documentation, see above).

The new name I've been working under cheekily called setup.cfg, that
is, the actual Python package is named setup.cfg.  You import
setup.cfg in your setup.py and it basically does the rest.  But if
that ends up being deemed too confusing/silly that would be
understandable--I'm open to other ideas.

 Since the declarative setup-requires concept also involves putting
 dependencies in setup.cfg (but setup_requires rather than
 install_requires), I followed the naming convention d2to1 had started.
 But - all the reviewers (and I agree) think this is confusing and
 non-obvious.

 Since d2to1 is strictly a build-time thing - it reflects the keys into
 the metadata and thus your egg-info/requires.txt is unaltered in
 output, I think its reasonable to argue that we don't need to be
 compatible with it.

 OTOH folk using d2to1 would not gain the benefits that declarative
 setup-requires may offer in setuptools // pip.

I haven't followed this whole discussion (I started to in the
beginning, but haven't kept up), but I'm not really sure what's being
said here.  d2to1 *does* support declaring setup-requires dependencies
in setup.cfg, and those dependencies should be loaded before any hook
scripts are used.  Everything in d2to1 is done via various hook
points, and the hook functions can be either shipped with the package,
or come from external requirements installed via setup-requires.  It
works pretty well in most cases.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] d2to1 setup.cfg schema

[Erik Bray]

On Mon, Mar 30, 2015 at 7:07 PM, Robert Collins
robe...@robertcollins.net wrote:
 On 31 March 2015 at 12:03, Erik Bray erik.m.b...@gmail.com wrote:

 I haven't followed this whole discussion (I started to in the
 beginning, but haven't kept up), but I'm not really sure what's being
 said here.  d2to1 *does* support declaring setup-requires dependencies
 in setup.cfg, and those dependencies should be loaded before any hook
 scripts are used.  Everything in d2to1 is done via various hook
 points, and the hook functions can be either shipped with the package,
 or come from external requirements installed via setup-requires.  It
 works pretty well in most cases.

 Oh, it does!? I was looking through the source and couldn't figure it
 out. What key is looked for for setup-requires? Also does it define a
 schema for extra-requires?

Yeah, sorry about that.  That's one of those things that was never
actually supported in distutils2 by the time it went poof, and that I
added later.

You can use:

[metadata]
setup-requires-dist = foo

So say, for example you have some package called versionutils that's
used to generate the package's version number (by reading it from
another file, tacking on VCS info, etc.)  You can use:

[metadata]
setup-requires-dist = versionutils

[global]
setup-hooks = versionutils.version_hook

or something to that effect.  It will ensure versionutils is
importable (this uses easy_install just like the normal setup_requires
feature in setuptools; I would like to change this one day to instead
use something like Daniel's setup-requires [1] trick).
It will then, fairly early in the setup process, hand the package
metadata over to versionutils.version_hook, and let it insert a
version string.

Erik

[1] https://bitbucket.org/dholth/setup-requires
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Installing a file into sitepackages

[Erik Bray]

On Fri, Feb 20, 2015 at 1:49 PM, Stuart Axon
stua...@yahoo.com.dmarc.invalid wrote:
 Hi,
In my project, I install a .pth file into site-packages, I use the 
 data_files...  in Ubuntu this seems to work OK, but in a Windows VM the file 
 doesn't seem to be being installed:

 setup(

# Install the import hook
 data_files=[
 (site_packages_path, [vext_importer.pth] if 
 environ.get('VIRTUAL_ENV') else []),
 ],
 )


 - Is there a better way to do this ?



 I realise it's a bit odd installing a .pth - my project is to allow certain 
 packages to use the system site packages from a virtualenv -
 https://github.com/stuaxo/vext

Hi Stuart,

I know this is old so sorry to anyone else.  But since no one
replied--I haven't looked too closely at what it is you're trying to
accomplish.  But whatever the reason, that seems like a
reasonable-enough way to me if you need to get a .pth file installed
into site-packages.  I'm not sure why it isn't working for you in
WIndows but it wasn't clear what the problem was.  I just tried this
in a Windows VM and it worked fine?

Best,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Closing the Delete File + Re-upload File Loophole.

[Erik Bray]

On Sat, Jan 24, 2015 at 1:53 PM, Marc Abramowitz msabr...@gmail.com wrote:
 You can re-run register as many times as you want which is all you need to 
 adjust the README.

 Maybe true but it would be pretty awesome to solve 
 https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way
  because repeatedly registering and doing trial and error is also not a great 
 experience and it wastes PyPI resources.

I usually register on testpypi.python.org to assuage this fear.
Granted it's a little annoying to do, because last I checked the only
way to do this (may this is has been fixed--someone can correct me) is
to change your .pypirc with credentials to testpypi.

Has .pypirc been done away with yet?

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Python module for use in ‘setup.py’ but not to install

[Erik Bray]

On Sat, Jan 31, 2015 at 3:19 PM, Ethan Furman et...@stoneleaf.us wrote:
 On 01/29/2015 08:58 PM, Ben Finney wrote:
 Ethan Furman writes:

 However, I feel that requiring a dependency simply for the
 installation of the main package (the main package doesn't actually
 use this install-dependency, correct?) is heavy-handed and should be
 avoided.

 It is ideally a build-time dependency and not an install-time
 dependency, but I'm having a difficult time figuring out how to
 distinguish those so Setuptools will pay attention.

 Ah, so it's needed with the sdist command and not the install command?  Yeah, 
 you definitely have my sympathies with
 that one.  Too bad there's no easy way to specify behavior based on the 
 command used... (hopefully someone will now tell
 how I am wrong about that ;) .

We just subclass the command in question to do the relevant tasks
(like generate some source files when running sdist).  The subclassed
command can just try to import the dependencies
and raise a (useful) exception message if they are not available.  For
example Astropy recently grew Jinja2 is a dependency to build some
source files.  The built files are included in the sdist so casual
users don't have to worry about it.

Developers on the other hand will be nagged that they need Jinja2, and
it isn't automatically installed, which sucks.  But for the most part
the only people who will need to worry about that can figure out how
to install build dependencies.  I'm less worried about it for them
than I am for the casual user trying to run `python setup.py install`
(or pip install for that matter).

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup.cfg

[Erik Bray]

On Sat, Nov 22, 2014 at 10:49 AM, Donaldo Fastoso
donquest...@rocketmail.com wrote:
 Hi,

 is there a way for python 2.7 to write something like:

 setup.py:

 from distutils.core import setup
 setup()

 setup.cfg:

 [metadata]
 name=foo
 version=0.1


 and get on the cmdline:

 python setup.py --name
 foo


 in distutils2 this was possible. Has python 2.7 stdlib distutils similar
 capabilities? May i use the [global] section? (as for now i only get
 errors trying that.) Or should i use the [metadata] command to implement
 it myself?

You might consider giving d2to1 https://pypi.python.org/pypi/d2to1 a
whirl, if it suits your needs otherwise.

It's long been on my todo list to release a successor to d2to1 that is
better documented and not as tied to the ill-fated distutils2 project.
But in the meantime d2to1 does work and pretty well at that.
Unfortunately other priorities have kept updates to it on the
backburner.  You might also consider giving pbr a look:

http://docs.openstack.org/developer/pbr/

This is basically a fork of d2to1 made for OpenStack that I think may
be more actively developed.


Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where is distribute at pypi?

[Erik Bray]

On Thu, Oct 23, 2014 at 11:01 AM, Rupert Kolb
rupert.k...@uni-tuebingen.de wrote:
 Hi,

 where is the distribute package at pypi?
 There is software (for instance pydicom) which need the
 distribute-0.6.xxx package. It doesn't work with the setuptools
 replacement out of the box!
 You can not just remove it!?

Well distribute is right here: https://pypi.python.org/pypi/distribute/0.7.3

But nothing should be explicitly depending on it (particularly in
favor of setuptools).  Looking at pydicom there's nothing remarkable
about it that it should make such a requirement, so if they do they're
wrong and its developers should fix that.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Installing namespace packages with pip inserts strange modules into sys.modules

[Erik Bray]

On Fri, Sep 12, 2014 at 8:38 AM, Thomas Heller thel...@ctypes.org wrote:
 While trying to add support for PEP420 implicit namespace packages
 support to py2exe's modulefinder, I noticed something strange.

 I'm using the wheezy.template and zope.interface packages for my tests,
 they are installed with pip.

 pip creates wheezy.template-0.1.155-py3.4-nspkg.pth and
 zope.interface-4.1.1-py3.4-nspkg.pth files that are not present in the
 package sources.  The contents is like this:

 
 import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
 *('wheezy',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not
 ie and sys.modules.setdefault('wheezy',types.ModuleType('wheezy')); mp = (m
 or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and
 mp.append(p)
 

 What is the purpose of these files?  (The packages seem to work also
 when I delete these files; which is what I expected from implicit
 namespace packages...)


 These .pth-files have the effect that they insert the modules 'wheezy'
 and 'zope' into sys.modules, even when I don't import anything
 from the packages.  However, these modules are IMO damaged, they
 only have __doc__, __name__, and __path__ attributes (plus __loader__
 and __spec__ in Python 3.4, both set to None).

 Reloading the package in Python 3.3 raises an error, and
 importlib.find_loader(zope) also:

 sys.modules[zope]
 module 'zope'
 import zope
 zope
 module 'zope'

 imp.reload(zope)
 Traceback (most recent call last):
   File stdin, line 1, in module
   File C:\Python33\lib\imp.py, line 276, in reload
 module.__loader__.load_module(name)
 AttributeError: 'module' object has no attribute '__loader__'

 importlib.find_loader(zope, None)
 Traceback (most recent call last):
   File stdin, line 1, in module
   File C:\Python33\lib\importlib\__init__.py, line 64, in find_loader
 loader = sys.modules[name].__loader__
 AttributeError: 'module' object has no attribute '__loader__'


 Trying the same in Python 3.4, behaviour is quite similar although
 imp.reload() 'fixes' that package:

 import sys
 sys.modules[zope]
 module 'zope'
 dir(sys.modules[zope])
 ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
 sys.modules[zope].__spec__

 import zope
 zope
 module 'zope'
 import importlib.util
 importlib.util.find_spec(zope, None)
 Traceback (most recent call last):
   File stdin, line 1, in module
   File C:\Python34\lib\importlib\util.py, line 100, in find_spec
 raise ValueError('{}.__spec__ is None'.format(name))
 ValueError: zope.__spec__ is None
 import imp
 imp.reload(zope)
 module 'zope' (namespace)
 importlib.util.find_spec(zope, None)
 ModuleSpec(name='zope', loader=None, origin='namespace',
 submodule_search_locations=_NamespacePath(['C:\\Python34\\lib\\site-packages\\zope']))


Hi Thomas,

I've dealt with this issue myself extensively in the past, but it's
been a while so I might have to give it a bit of thought before I
comment more on detail (if no one else does first).

But I don't think (unless there's something new I don't know about)
those come directly from pip.  Rather, they are created by setuptools.

The issue here is that before Python had any built-in notion of
namespace packages, namespace packages were a feature in setuptools,
and this was the hack, I suppose, that allowed it to work.
Unfortunately as different versions of Python have grown different
levels of support for namespace packages over the years, the
setuptools hack is still the implementation of the concept that will
work on the broadest range of Python versions (and I definitely know
zope.interface has been using it going far back).

That said, I've been meaning to try to figure out some way of
supporting all three forms of namespace packages in a way that is
interfering.  As I said, I've had problems with this myself :/

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Installing namespace packages with pip inserts strange modules into sys.modules

[Erik Bray]

On Fri, Sep 12, 2014 at 3:24 PM, Eric V. Smith e...@trueblade.com wrote:
 [Oops, replying to the list this time. Sorry for the dupe, Thomas.]

 On 9/12/2014 3:14 PM, Thomas Heller wrote:
 So it seems that it is a bug in setuptools:  It must not create or
 install these pth files when installing in Python 3.3 or newer (which
 implement PEP 420).

 PEP 420 goes out of its way to support pkgutil.extend_path():
 http://legacy.python.org/dev/peps/pep-0420/#migrating-from-legacy-namespace-packages

 So it should be possible for some cross-version code to work.

The pkgutil.extend_path() way can be made to work with PEP 420
reasonably well, but *not* with the older setuptools approach.

Unfortunately there are some dark corners of setuptools I've
encountered where namespace packages don't work properly during
installation *unless* they were installed in the old-fashioned
setuptools way.  I'll have to see if I can dig up what those cases
are, because they should be fixed.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Undefined symbol when loading a library using dlopen

[Erik Bray]

On Tue, Aug 26, 2014 at 5:54 PM, Jerome Fuselier
jerome.fusel...@free.fr wrote:
 Hello,

 I'm trying to wrap a C++ library which uses dlopen to load a dynamic library
 and I don't manage to get rid of an undefined symbol error.

 I tried to simulate my problem with a simpler example so hopefully someone
 may be able to tell me what I'm doing wrong.

 I have a file func.cpp which provides 2 functions :

 func.cpp:

 int simple(void) {
 return 12;
 }

 int call_lib(void) {
 int res;
 void *handle = dlopen(/home/jerome/Test/libDyn.so, RTLD_NOW);

 if (!handle) {
 return -1;
 }
 dlerror();
 char* err = 0;
 int (*fun_dyn)() = (int (*)())dlsym(handle, fun_dyn);
 if ( (err = dlerror()) != 0 ) {
 dlclose(handle);
 return -1;
 }
 res = (*fun_dyn)();
 dlclose(handle);
 return res;
 }

 It is created like that:
 $ g++ -Wall -g -fPIC -c -o func.o func.cpp
 $ ar rcs libFun.a func.o

 I have a dynamic library which is called from call_lib() which depends on
 libFun.a:

 libDyn.cpp:

 #include func.hpp

 int fun_dyn(void) {
 return simple();
 }

 It is created like that:
 $ g++ -Wall -g -fPIC -c -o libDyn.o libDyn.cpp
 $ g++ -Wall -g -fPIC -shared -o libDyn.so libDyn.o

 I'm using swig to wrap the libFun.a library with this mylib.i interface
 file:

 %module mylib

 int simple(void);
 int call_lib(void);

 Then I'm using this setup.py file to create the Python module:

 from distutils.core import setup, Extension

 mylib = Extension(_mylib,
   sources=['mylib.i'],
   language=c++,
   extra_link_args=['-rdynamic'],
   libraries=[Fun, dl],
  )

 setup(name=MyLib,
   platforms=[Linux],
   ext_modules=[mylib],
   py_modules = [mylib])


 It's a simplified example but when I compile everything I get this error in
 Python:
 import _mylib
 _mylib.call_lib()
 failed to open shared object file
 /home/jerome/Test/libDyn.so: undefined symbol: simple

 If I do a nm _mylib.so I see that the symbol simple is present in the
 library so I'm wondering why dlopen doesn't see it when it tries to load the
 library.

 I know I can recompile libDyn.so to link libFun.a but this would means I
 would need to recompile all the plugins provided for the library I'm trying
 to wrap and that's not really a solution for my project.

I don't think this question really has to do with distutils' role in
this.  But have you tried declaring

extern C {
int simple(void);
}

in func.cpp?  I think that should do it (to get around C++ name mangling).

Best of luck,

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI changelog support, releases.json / common NEWS.rst format?

[Erik Bray]

On Fri, Jul 11, 2014 at 5:24 PM,  ma...@include-once.org wrote:
 I'm currently tinkering on a freshmeat substitute. And for automating
 release announcements been looking around for common package meta
 data schemes. PyPIs /pypi/pkgname/json (or the xmlrpc interface) looks
 quite interesting. It obviously mostly targets dependency management
 and systemic categorization.

 So from googling around this never came up: But would it be feasible
 to include a version changelog / release summary via du.register?

 Of course, I'm referring to a human-readable This version adds and
 fixes... changelog, not the (name, version, timestamp) journal tuple.

 The releases{} per-URL `comment_text` seems widely unused.
 Was that its purpose?


 I hope this isn't getting too off-topic, this is -just for comparison
 and context- what I'm intending to eventually map PyPI release streams
 onto:
http://fossil.include-once.org/freshcode/wiki/releases.json


 There are probably other priorities for distutils / warehouse
 currently. So alternatively, is there a semi-standard for NEWS.txt
 or CHANGELOG etc. files within Python packages?

 Cheeseshop project homepages which also include a release notes list
 via `long_description` seem far and few between. I actually found
 just one:
https://pypi.python.org/pypi/py-translate
 Which seems to share a reStructuredText source for documentation and
 pypi homepage:

 https://raw.githubusercontent.com/jjangsangy/py-translate/master/HISTORY.rst

 (I'd have presumed Markdown-style release notes to be favoured.)
 Anyway, is there an estimate on how many packages include release
 notes at all?

Interesting question--I've long struggled over the question of how
best to maintain a readable human-readable changelog/release notes.
Part of it is social, and getting all contributors to enter the
correct changelog entry for any changes they submit.  I've mostly
gotten that hammered out for my main projects.

But managing a changelog for a project with multiple release branches
is also a bit tricky and something I've been wanting to find a better
way to automate.

As for actual formats I long ago adopted the format used by
zest.releaser [1] even though I'm not using it as much any more to
make my releases.  I think the format it recognizes by default is all
valid ReST, and I've preferred to stick with that.  I also try to link
every changelog entry to a bugtracker issue number in square brackets.
So for the most part the format has been kept machine-parseable (in
fact we have a Sphinx plugin that automatically converts the [#]
issue number markers to links).  See for example the Astropy changelog
[2].

Still, if anyone else has further thoughts on this topic I'd be interested.

Erik


[1] https://github.com/zestsoftware/zest.releaser/blob/master/CHANGES.rst
[2] https://github.com/astropy/astropy/blob/master/CHANGES.rst
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] fix setup_requires by allowing it in setup.cfg or a setup_requires.txt

[Erik Bray]

On Thu, May 15, 2014 at 6:31 PM, Daniel Holth dho...@gmail.com wrote:
 On Thu, May 15, 2014 at 6:04 PM, Marcus Smith qwc...@gmail.com wrote:
 I'm of course hoping that someone uses the feature to do a
 setup.py-command-line-interface-compatible distutils replacement
 (rather than a distutils extension) like what Bento does with its
 pip-compatible setup.py replacement.


 ah, I see.  interesting.
 I admit my initial reaction to quack like a setup.py alternative build
 systems is that it sounds invasive and sneaky, and I want it to be explicit
 and overt,  like it will be in metadata 2.X.
 but on the other hand,  I hear you about wanting to see alternatives have
 some way to get off the ground now.

 There's a bit of a chicken-egg problem obviously.

 For the moment I am more interested in just giving people a usable
 setup_requires.

Just now catching up on this, but I'm strongly in favor of fixing this
issue.  It's a simple addition to setuptools.

I actually am about to release the first version of a package called
astropy_helpers, which basically is a bundle of all the build tools
used by the Astropy project so that other related projects can take
advantage of them as well.  This has the same chicken-egg problem--you
want to be able to use astropy_helpers in setup.py, but you can't get
to it without first calling setup(setup_requires=['astropy_helpers']).

This can actually be gotten around by simply creating a dummy
Distribution object and calling it like:

from setuptools.dist import Distribution
Distribution({'setup_requires': ['astropy_helpers']})

But that's an ugly hack.  To that end I added bootstrap script that's
imported at the top of setup.py that knows how to do this (it also has
code for finding astropy_helpers as a git submodule if the project has
one in its repo--useful for development).

d2to1 takes a slightly different approach in that one doesn't pass
anything to the normal setup() except for setup_requires=['d2to1'].
Once d2to1 is bootstrapped it takes over the entire setup process,
including reading additional setup_requires from setup.cfg as Daniel
suggested.  I'm actually planning on relaunching d2to1 with a new name
(less tied to the defunct distutils2) because I still think it's a
useful alternative to setup.py, while still working within the
existing framework (and easily transferable to any new framework).

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A setup-requires implementation

[Erik Bray]

On Mon, May 19, 2014 at 9:24 PM, Daniel Holth dho...@gmail.com wrote:
 Here's a short setup.py replacement that makes setup-requires work:
 https://bitbucket.org/dholth/setup-requires/src/ . I'd appreciate a
 code review.

 Use by renaming your own package's setup.py to real-setup.py and
 copying this setup.py in its place.

 List only the requirements setup.py itself needs to run in the
 `setup-requires =` key of the `[metadata]` section of setup.cfg,
 one per line::

 [metadata]
 setup-requires = cffi
 pip
 pycparser = 2.10

 (Only the name and required versions are allowed, not the full pip
 syntax of URLs to specific repositories. Instead, install internal
 setup-requires dependencies manually or set PIP_FIND_LINKS=... to point
 to the necessary repositories.)

 When run, setup-requires' setup.py checks that each distribution
 listed in setup-requires is installed; if not, it installs them into
 the ./setup-requires directory in a pip subprocess. Then real-setup.py
 continues to execute with the same arguments.

 Why a custom section in setup.cfg? Users are accustomed to editing
 setup.cfg to configure random things like unit tests, bdist_wheel
 etc.; this just adds a field instead of a new file. Unlike a .txt file
 it should be more intuitive that setup.cfg does not support the full
 pip requirements syntax.

 Please note that not every package installs correctly with pip -t.

 Now let's see some setup.py helper packages.

Thanks, I like the approach this takes of using pip instead of
easy-install, and caching all the setup_requires packages in the
setup-requires directory without cluttering the source root with eggs.
 I'm not crazy about having to launch a separate process to do
this--do you think it can be done without that, by just importing pip
and using its API?  Or did you find that to be too problematic?

I'm also not crazy about putting the real setup.py in a separate file,
but only because, in my case, I think it's likely to confuse some of
my co-developers.  That said, I think that this approach could be
adapted to suit my needs.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] tourist here, with a dumb RTFM question

[Erik Bray]

On Sat, Mar 8, 2014 at 10:49 AM, Daniel Holth dho...@gmail.com wrote:
 Some packaging systems do not have the same 1:1 source : distribution
 relationship that is so prominent in the distutils model. This should
 probably change long term.

A common (I think?) example of this, which comes up in Astropy, is
pacakges containing Cython modules.  We don't keep the generated C
files in version control.  But we *do* make sure that when running
sdist that the Cython modules are collected and the C modules
generated and added to the manifest so that end users do not need
Cython to `pip install astropy`.  As long as their C compiler works it
can build the generated C modules without Cython.

Right now that's easy enough to do by extending the relevant distutils
commands.  Definitely a distinction to keep in mind in the future
though.

Erik

 On Sat, Mar 8, 2014 at 9:53 AM, Michael Bayer mike...@zzzcomputing.com 
 wrote:

 On Mar 8, 2014, at 9:19 AM, Jason R. Coombs jar...@jaraco.com wrote:



 From: Distutils-SIG
 [mailto:distutils-sig-bounces+jaraco=jaraco@python.org] On Behalf Of
 Marcus Smith
 Sent: Thursday, 06 March, 2014 16:16
 To: DistUtils mailing list
 Subject: Re: [Distutils] tourist here, with a dumb RTFM question





 On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer mike...@zzzcomputing.com
 wrote:


 On Mar 6, 2014, at 1:07 PM, Daniel Holth dho...@gmail.com wrote:

 pje said:

 The Feature() facility was never completely implemented or
 supported, and even if it were, it should be deprecated now, as it
 will not be compatible with the coming packaging systems based on PEP
 426. If you need separate features, use separate distributions and
 extras instead.

 wait, ok this is that thing.  separate distributions means


 SQLAlchemy-0.9.3-nocext.tar.gz
 SQLAlchemy-0.9.3-cext.tar.gz


 I'm new to understanding setuptools Feature myself, so don't crank too
 much, if I get anything wrong.

 afaik, it means something like:

 SQLAlchemy-X.Y.tar.gz
 SQLAlchemy-cext-X.Y.tar.gz

 i.e. isolating the feature into a separate project.

 and then instead of having the handy  --with-cext option,  it has to be
 pip install SQLAlchemy[SQLAlchemy-cext]

 PJ, Jason: can you clarify what the alternative is supposed to look like?

 I'm new to understanding Features as well, but based on PJ's post, I would
 agree with Marcus here. This is how I interpreted the use of extras to
 supersede Features. I see a lot of advantages to this approach over
 build-time selection. Because it decouples the main project from the
 C-extension speedups, it gives the installer control over what behavior is
 present. It also gives deployment tools visibility over what capability is
 present (with features, it's harder to tell if the C-extension speedups are
 present; with extras, one can query pkg_resources).



 If there's an easy way to do this from setup.py it wouldn't be so bad:

 python setup.py --without-cext sdist upload

 python setup.py sdist upload

 but it still seems kind of strange to deliver the exact same source files
 with some flag somewhere different?   I'd assume that flag is in setup.cfg,
 which means, it's a setup.py flag in any case!   not to mention its still
 needed as part of the sdist and other commands to build dists.

 I think the issue of you should have individual projects on pypi with
 different options to make life easier for builders is orthogonal to
 setup.py should or should not support flags.











 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig



 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] errors trying to get setuptools 2.0.2

[Erik Bray]

I'm not sure why your powershell is blowing up, but try running

python ez_setup.py --insecure

instead.

On Sun, Jan 5, 2014 at 7:38 AM, Malik Rumi malik.a.r...@gmail.com wrote:
 Here is what I did and the response:

 c:\

 c:\ez_setup.py

 Downloading


 https://pypi.python.org/packages/source/s/setuptools/setuptools-2.0.

 2.tar.gz

 Exception calling DownloadFile with 2 argument(s): An exception


 occurred

 during a WebClient request.

 At line:1 char:1

 + (new-object

 System.Net.WebClient).DownloadFile('https://pypi.python.org/packages


 ...

 +


 ~~~


 ~~

 ~~~

 + CategoryInfo  : NotSpecified: (:) [],


 MethodInvocationException

 + FullyQualifiedErrorId : WebException


 Traceback (most recent call last):

   File C:\Python27\Scripts\ez_setup.py, line 361, in module

 sys.exit(main())

   File C:\Python27\Scripts\ez_setup.py, line 357, in main

 downloader_factory=options.downloader_factory)

   File C:\Python27\Scripts\ez_setup.py, line 282, in


 download_setuptools

 downloader(url, saveto)

   File C:\Python27\Scripts\ez_setup.py, line 169, in


 download_file_powershell

 _clean_check(cmd, target)

   File C:\Python27\Scripts\ez_setup.py, line 152, in _clean_check

 subprocess.check_call(cmd)

   File C:\Python27\lib\subprocess.py, line 511, in check_call

 raise CalledProcessError(retcode, cmd)

 subprocess.CalledProcessError: Command '['powershell', '-Command',


 (new-object

 System.Net.WebClient).DownloadFile


 ('https://pypi.python.org/packages/source/s/se

 tuptools/setuptools-2.0.2.tar.gz', 'c:setuptools-2.0.2.tar.gz')]'


 returned

 non-zero exit status 1


 c:\

 The parts in red were in red in my terminal window. I am on windows 8.1 and
 I am brand new. Less than a week ago, I installed Distribute because someone
 on Django Google Groups recommended it to me, along with virtualenv and pip.
 Then today - well, last night since it was a few hours ago - I found out
 Distribute is deprecated. So I am trying to follow the uninstall
 instructions but this is what I got. Just for kicks I ran it again just now
 - same result. Please help. I took time off from work Thursday and Friday to
 get all this set up have been at this all weekend only to be stumped again.
 And no, I do not have Django up and running yet. Thank you.

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Setuptools 2.0 Released

[Erik Bray]

On Sat, Dec 7, 2013 at 1:29 PM, Jason R. Coombs jar...@jaraco.com wrote:
 The PyPA is pleased to announce the Setuptools 2.0 release.



 This backward-incompatible release drops support for Python 2.4 and Python
 2.5, but is otherwise compatible with 1.x releases.



 The 1.x series will continue to be supported for bug and security fixes for
 the foreseeable future. This means that any systems with older versions of
 Python should install “setuptools2dev” or use the legacy bootstrap script,
 available at
 https://bitbucket.org/pypa/setuptools/raw/bootstrap-py24/ez_setup.py for
 Python 2.4 and Python 2.5 users. That link is also published in the README
 as install instructions.



 Please report any emergent issues at
 https://bitbucket.org/pypa/setuptools/issues.

Thanks for the awesome work!  I still support 2.5 for a few projects
so it's helpful to know that there are still options available for
such cases.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How setuptools reads requirements

[Erik Bray]

On Tue, Nov 19, 2013 at 1:56 PM, Kura k...@kura.io wrote:
 Hey guys

 I'm trying to dig in to how setuptools/distutils reads the
 [install_]requires keyword and how requirements.txt is handled.

 I've tried running setup.py --requires but get an empty response back, this
 also fails when using =x.x,=x.y and causes a Python stack trace.

 Was wondering if someone on here could shed some light on the topic for me.

`setup.py --requires` is mostly useless.  It only reports back entries
from the `requires=` argument to setup() supported by distutils.  This
only tracks the names of other modules that a module needs to import,
and is being phased out as not particularly useful.  It's not in any
way tied to `install_requires` which is a feature added by setuptools.

As for requirements.txt, you might mean requires.txt which is a
file added to the .egg_info directory when you run the `setup.py
egg_info` command on a setuptools-enabled package.  Pip has its own
code for reading information from files out of .egg_info.  I believe
there are other libraries like pkginfo [1] that do this.

 It's for a Python dependency management/monitoring system I am writing so
 hopefully would benefit the overall community.

Sounds intriguing!

[1] https://pypi.python.org/pypi/pkginfo
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecate and Block requires/provides

[Erik Bray]

On Thu, Oct 17, 2013 at 9:53 AM, Nick Coghlan ncogh...@gmail.com wrote:
 Yes, but breaking these users' uploads is not the way to do that.
 That's an incredibly user hostile step to take, and the problem
 doesn't even come close to justifying breaking even a
 not-really-working process (since you can work around the current
 breakage by manually installing the dependencies - you can't work
 around an inability to upload without making changes to your code).

+1  I can imagine that from many users' perspectives you're doing it
wrong isn't even true.  The docs for distutils *still* read:

Dependencies on other Python modules and packages can be specified by
supplying the requires keyword argument to setup()

and

A package can declare that it obsoletes other packages using the
obsoletes keyword argument

and nowhere does it read but these keywords are broken and obsolete
as of some obscure PEP and you're wrong to have used them.

 If you really wanted to push them towards fixing their releases, you
 could always have PyPI send them an email saying something like:

 Hi, an automated scan of PyPI has shown that you're currently
 uploading metadata that uses the obsolete module dependency fields
 defined by distutils. These fields aren't actually checked by
 automated installation tools like pip, so if you're attempting to
 indicate that your project depends on other PyPI projects, this
 mechanism doesn't actually work to achieve that.

 At some point in the future, PyPI may disallow use of these fields
 entirely, so if you'd like to declare your dependencies in a way that
 automated tools can process, you may want to look at using the
 dependency declaration features in setuptools rather than using plain
 distutils.

I was just going to ask--why not at least try to e-mail the
maintainers of those packages?  If they're unreachable to begin with
then I'm a little less concerned.  But at least give those who might
care a direct heads up about where things are going.  Then when the
inevitable complaints do come in at least you've covered your ass :)

Best,

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Changing the install hooks mechanism for PEP 426

[Erik Bray]

On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan ncogh...@gmail.com wrote:
 I spent last weekend at Flock to Fedora, mostly due to my day job
 working on the Beaker integration testing system
 (http://beaker-project.org) for Red Hat, but also to talk to folks
 about Fedora and Python interactions.

 A completely unexpected discovery over the weekend, was that some of
 the RPM folks are exploring the idea of switching the *user* facing
 format for the packaging system away from spec files and towards
 directly executable Python code. Thus, you'd get away from the painful
 mess that is RPM conditionals and macros and have a real programming
 language to define what your built packages *should* look like, while
 still *producing* static metadata for consumption by installers and
 other software distribution tools.

 Hmm, does that approach sound familiar to anyone? :)

 Anyway, we were talking about how they're considering approaching the
 install hook problem, and their approach gave me an idea for a better
 solution in PEP 426.

 Currently, PEP 426 allows a distribution to define install hooks:
 hooks that will execute after the distribution is installed and before
 it is uninstalled.

 I'm now planning to change that to allowing distributions to define
 export hooks, based on the cleaned up notion of export groups in
 the latest version of PEP 426. An export hook definition consists of
 the following fields:

 * group - name of the export group to hook
 * preupdate - export to call prior to installing/updating/removing a
 distribution that exports this export group
 * postupdate - export to call after installing/updating/removing a
 distribution that exports this export group
 * refresh - export to call to resynchronise any caches with the system
 state. This will be invoked for every distribution on the system that
 exports this export group any time the distribution defining the
 export hook is itself installed or upgraded

 If a distribution exports groups that it also defines hooks for, it
 will exhibit the following behaviours:

 Fresh install:
 * preupdate NOT called (hook not yet registered)
 * postupdate called
 * refresh called

 Upgrade:
 * preupdate called (old version)
 * postupdate called (new version)
 * refresh called (new version)

 Complete removal:
 * preupdate called
 * postupdate NOT called (hook no longer registered)
 * refresh NOT called (hook no longer registered)

 This behaviour follows naturally from *not* special casing
 self-exports: prior to installation, the export hooks won't be
 registered, so they won't be called, and the same applies following
 complete removal.

 The hooks would have the following signatures:

 def preupdate(current_meta, next_meta):
# current_meta==None indicates fresh install
# next_meta==None indicates complete removal

def postupdate(previous_meta, current_meta):
# previous_meta==None indicates fresh install
# current_meta==None indicates complete removal

 def refresh(current_meta):
 # Used to ensure any caches are consistent with system state
 # Allows handling of previously installed distributions


I think I'm okay with this so long as it remains optional.  I'm not
crazy about executable build specs where they're not necessary.  For
most cases, especially in pure Python packages, it's frequently
overkill and asking for trouble.  So I would still want to see a
well-accepted static build spec for Python packages too (sort of a la
setup.cfg as parsed by d2to1, only better), though I realize that's a
separate issue from PEP 426.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to disable PYTHONPATH checking when installing packages using distribute

[Erik Bray]

On Fri, Aug 2, 2013 at 8:53 AM,  lukshun...@gmail.com wrote:
 Hi,

 During installing a package which uses distribute (matplotlib in this case),
 it refuses to work with this message

 running install
 Checking .pth file support in
 /usr/local/stow/matplotlib-1.3.0/lib/python2.7/site-packages/
 /usr/bin/python -E -c pass
 TEST FAILED: /usr/local/stow/matplotlib-1.3.0/lib/python2.7/site-packages/
 does NOT support .pth files
 error: bad install directory or PYTHONPATH
 ...
 Please make the appropriate changes for your system and try again.

 I install local packages using the stow approach, which installs each
 package under its own sub-directory and later stowed
 (https://www.gnu.org/software/stow/). Such error becomes a nuisance as a
 different PYTHONPATH has to be set for each installation of a package.

 How can the checking be disable? I don't seem to be able to find anything in
 the documentation and would be grateful for any pointer.

 Or maybe it's better turned into a warning and users be reminded to add the
 install directory to PYTHONPATH.

By default setuptools (distribute) installs packages as eggs, and
loading eggs requires the ability to write a .pth file to a directory
that will be checked for .pth files at start up (i.e. is in PYTHONPATH
or otherwise on sys.path by default).  You can avoid doing an
egg-based install by instead running:

python setup.py install --single-version-externally-managed --prefix
/usr/local/stow/matplotlib-1.3.0

or something to that effect.  I think if you do this you also need to
make sure to manually add the .egg-info directory as well.  I think
you can do this with

python setup.py install_egg_info --install-dir
/usr/local/stow/matplotlib-1.3.0/lib/python2.7/site-packages/

but YMMV.  You might also try just installing with pip since it will
basically install the package in the same way by default.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 439 and pip bootstrap updated

[Erik Bray]

On Wed, Jul 10, 2013 at 12:37 AM, Richard Jones r1chardj0...@gmail.com wrote:
 pip without virtualenv in python 2 contexts is pretty rare (or at
 least *should* be wink) so I think I'll retain it in that bootstrap
 code.

I agree it *should* be rare in most cases but it most assuredly is
not.  I can tell you from experience that a lot of people in the
scientific community, for example, do not use virtualenv (sometimes
with good reasons, but more often not).

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] complicated setup

[Erik Bray]

On Wed, Jun 26, 2013 at 4:21 PM, Erik Bray erik.m.b...@gmail.com wrote:
 On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman et...@stoneleaf.us wrote:
 Here's my file layout:

 root /
|- setup.py
|
|- enum /
|- __init__.py
|
|- py2_enum.py
|
|- py3_enum.py
|
|- test /
|- test_enum.py
|
|- py2_test_enum.py
|
|- py3_test_enum.py

 __init__ and test_enum are both smart enough to pull in the correct code
 when imported.  The issue I am having is this:

 --8--
 ethan@hydra:~$ sudo easy_install enum34
 [sudo] password for ethan:
 Searching for enum34
 Reading http://pypi.python.org/simple/enum34/
 Best match: enum34 0.9
 Downloading
 http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8
 Processing enum34-0.9.zip
 Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg
 Running enum34-0.9/setup.py -q bdist_egg --dist-dir
 /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5
 SyntaxError: ('invalid syntax',
 ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, 'def
 __call__(cls, value, names=None, *, module=None, type=None):\n'))

 SyntaxError: ('invalid syntax',
 ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, '
 class AutoNumberedEnum(Enum, metaclass=auto_enum):\n'))

 zip_safe flag not set; analyzing archive contents...
 SyntaxError: ('invalid syntax',
 ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py',
 211, 43, 'def __call__(cls, value, names=None, *, module=None,
 type=None):\n'))

 SyntaxError: ('invalid syntax',
 ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py',
 630, 47, 'class AutoNumberedEnum(Enum, metaclass=auto_enum):\n'))

 Adding enum34 0.9 to easy-install.pth file

 Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg
 Processing dependencies for enum34
 Finished processing dependencies for enum34
 --8--

 distutils is trying to load the py3 versions, which of course fails on a py2
 install.  The package installs successfully anyway, but if I were a user I
 would be wondering if the install was trustworthy.

 It seems to me that I need to either have distutils only install the version
 appropriate files, or to not try to scan the version inappropriate files,
 but at this point I do not know how to do either.

 Any pointers would be greatly appreciated.

 That's odd.  I work on a package that ships Python 2 and Python 3
 versions of some modules and I have never seen this problem before.
 Perhaps you could post your setup.py?

It occur to me now that the reason I don't see this kind of error on
my project is because in the setup.py I am just excluding the
version-specific files based on what Python version the user has.
Perhaps you should do the same--only install the Python 2 tests on
Python 2 and so on.  Just make sure that the MANIFEST.in is set up to
include both versions in the source distribution.  As long as you
don't install the Py2 files in Py3 or vice versa it shoudn't try to
compile the bytecode for those files.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] complicated setup

[Erik Bray]

On Sun, Jun 16, 2013 at 3:13 AM, Ethan Furman et...@stoneleaf.us wrote:
 Here's my file layout:

 root /
|- setup.py
|
|- enum /
|- __init__.py
|
|- py2_enum.py
|
|- py3_enum.py
|
|- test /
|- test_enum.py
|
|- py2_test_enum.py
|
|- py3_test_enum.py

 __init__ and test_enum are both smart enough to pull in the correct code
 when imported.  The issue I am having is this:

 --8--
 ethan@hydra:~$ sudo easy_install enum34
 [sudo] password for ethan:
 Searching for enum34
 Reading http://pypi.python.org/simple/enum34/
 Best match: enum34 0.9
 Downloading
 http://pypi.python.org/packages/source/e/enum34/enum34-0.9.zip#md5=4717b8c328083d816b3b987f24446ad8
 Processing enum34-0.9.zip
 Writing /tmp/easy_install-sB55B5/enum34-0.9/setup.cfg
 Running enum34-0.9/setup.py -q bdist_egg --dist-dir
 /tmp/easy_install-sB55B5/enum34-0.9/egg-dist-tmp-qUYAv5
 SyntaxError: ('invalid syntax',
 ('build/bdist.linux-x86_64/egg/enum/py3_enum.py', 211, 43, 'def
 __call__(cls, value, names=None, *, module=None, type=None):\n'))

 SyntaxError: ('invalid syntax',
 ('build/bdist.linux-x86_64/egg/enum/test/py3_test_enum.py', 630, 47, '
 class AutoNumberedEnum(Enum, metaclass=auto_enum):\n'))

 zip_safe flag not set; analyzing archive contents...
 SyntaxError: ('invalid syntax',
 ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/py3_enum.py',
 211, 43, 'def __call__(cls, value, names=None, *, module=None,
 type=None):\n'))

 SyntaxError: ('invalid syntax',
 ('/usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg/enum/test/py3_test_enum.py',
 630, 47, 'class AutoNumberedEnum(Enum, metaclass=auto_enum):\n'))

 Adding enum34 0.9 to easy-install.pth file

 Installed /usr/local/lib/python2.7/dist-packages/enum34-0.9-py2.7.egg
 Processing dependencies for enum34
 Finished processing dependencies for enum34
 --8--

 distutils is trying to load the py3 versions, which of course fails on a py2
 install.  The package installs successfully anyway, but if I were a user I
 would be wondering if the install was trustworthy.

 It seems to me that I need to either have distutils only install the version
 appropriate files, or to not try to scan the version inappropriate files,
 but at this point I do not know how to do either.

 Any pointers would be greatly appreciated.

That's odd.  I work on a package that ships Python 2 and Python 3
versions of some modules and I have never seen this problem before.
Perhaps you could post your setup.py?

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Setuptools 0.7 final available for download

[Erik Bray]

On Sun, Jun 2, 2013 at 11:42 AM, Jason R. Coombs jar...@jaraco.com wrote:
 I’m pleased to announce the official release of Setuptools 0.7, now
 available for download from the project page.



 Nothing has changed from the 0.7b4 pre-release. This is the version that
 will be uploaded to PyPI after we work out the technique to deploy to PyPI
 without interfering with the setuptools 0.6 releases.



 For convenience, I’ve also added experimental .msi installers for Windows
 for Python 3.3 and Python 2.7. Work may continue on these in the future, but
 as the documentation states, the recommended installation procedure is to
 use ez_setup.py.



 To install this latest release, follow the canonical install instructions
 (using ez_setup.py), but direct the script to use bitbucket instead of PyPI:



 python ./ez_setup.py
 --download-base=https://bitbucket.org/pypa/setuptools/downloads/



 I’ll send another announcement when the official release has been uploaded
 to PyPI and the ez_setup.py script can be used directly.


Thank you Jason and PJ for all your work on this.

It's maybe a little late to bring this up now, but I wonder if, in a
future release, we could deprecate 'ez_setup.py' (venerable though it
is) and name it something a little more obvious, like
'setuptools_setup.py'.  Despite being longer and including the word
'setup' twice I think it is a great deal more clear what it does.

I remember in the past, before I switched to distribute, I got reports
from users who were running the ez_setup.py that I included in
projects thinking that it was the easy way to install my software.
Yes, they should have just read the README that says python setup.py
install and makes no mention of ez_setup.py, but nevertheless I can't
blame anyone for finding it vague (and maybe even a bit enticing to
run).

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Small Usability/UX Change to PyPI

[Erik Bray]

On Wed, May 29, 2013 at 9:28 AM, Donald Stufft don...@stufft.io wrote:
 Just deployed a small update to PyPI that sets autofocus on the search
 form. This'll mean that you don't need to manually move to the search
 form upon page load and should be able to start typing immediately.

That is so trivial yet so awesome.  Thank you!
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Draft PEP for JSON based metadata published

[Erik Bray]

On Mon, May 27, 2013 at 7:36 AM, Nick Coghlan ncogh...@gmail.com wrote:
 After preliminary reviews by Donald and Daniel, I have now pushed the
 first complete draft of the JSON-based metadata 2.0 proposal to
 python.org

 PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/
 PEP 440 (versioning): http://www.python.org/dev/peps/pep-0440/

 With the rationale and commentary, they're over 3000 lines between
 them, so I'm not attaching them here.

 The rationale for many of the changes is at the end of each PEP, along
 with some comments on features that I have either rejected or
 deliberately chosen to defer to the next revision of the metadata (at
 the earliest).

 Those with BitBucket accounts may also comment inline on the drafts here:

 PEP 426: 
 https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0426.txt?at=default
 PEP 440: 
 https://bitbucket.org/ncoghlan/misc/src/05d3586464b10d6a04a35409468269d7c89a87ba/pep_drafts/pep-0440.txt?at=default

This is looking fantastic so far--thanks to Nick, Daniel, and Donald
for their continued work on this.  For now I just have a handful of
minor notes on the latest draft of PEP 426:

Typos:

Under Essential dependency resolution metadata the may_require and
related metadata keywords are spelled with hyphens instead of
underscores.

Under Metabuild system in the first example I think
some_test_harness.metabuild_hook was meant to read
some_test_harness:metabuild_hook


Under Development, build and deployment dependencies:  allow - allows

Under Support for metabuild hooks:  by allows projects - by
allowing projects

Comment:

I'm not sure if this PEP is the best place for this, but I wonder if
the description of the Keywords format could provide some
clarification on how that field should be formatted in older metadata
versions (specifically when including version 1.x metadata for
backwards compatibility).  In the past its format has never been
specified.  Some tools treat it as a space-separated fields.  Others
have treated it as a comma-separated field.  Sometimes one or the
other depending on whether commas are present.  It's a very annoying
field.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Good news everyone, PyPI is behind a CDN

[Erik Bray]

On Mon, May 27, 2013 at 1:19 AM, Lennart Regebro rege...@gmail.com wrote:
 On Sun, May 26, 2013 at 7:34 PM, Noah Kantrowitz n...@coderanger.net wrote:
 /farnsworth

 but seriously, at long last today it was my honor to throw the DNS switch to 
 move PyPI to the Fastly caching CDN. I would like to thank Donald Stufft for 
 doing much of the heavy lifting on the PyPI side, and to Fastly for 
 graciously offering to host us. What does this mean for everyone? Well the 
 biggest change is PyPI should get a whole lot faster. There are two major 
 downsides however. There will now be a delay of several minutes in some 
 cases between updating a package and having it be installable, and download 
 counts will now be even more incorrect than they were before. The PyPI 
 admins are discussing what to do about download counts long-term, but for 
 now we all feel that the performance and availability benefits outweigh the 
 loss. If anyone has any questions, or hears anything about issues with PyPI 
 please don't hesitate to contact me.

 This is going to spell disaster for the coffee industry, as you no
 longer have to take a coffee break when re-running a buildout.

 Thanks!

I always test pip installation from PyPI just in case after
uploading a new package, so the new cache delay still leaves some time
for a coffee break (until Daniel gets the cache invalidation
integrated :/).  But yes, so many hoorays for this \o/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] User Viewpoint: Packaging History, Confusion and User Advice

[Erik Bray]

Hi, I just wanted to point out a couple small corrections to an
otherwise good summary of the situation.

On Wed, Apr 24, 2013 at 10:55 AM, Erik Bernoth erik.bern...@gmail.com wrote:
 distutils (classic) - the old school, deprecated, pretty much unmaintained
 packaging tooling; still the official standard [4] (Note: reasons [13])

distutils isn't deprecated--almost all of the other mentioned projects
rely on distutils on some level, though there seems to be some effort
to undo that reliance and relegate distutils to just a build tool that
can be swapped out for others.  The problem with distutils is not so
much that it's deprecated, as that there are so many other projects
built on top of it, many of them making use of poorly documented
internal interfaces, that it's very very difficult to change or
improve distutils without breaking lots of other projects.

 setuptools - fork of distutils, which wasn't maintained for a long time but
 will now be merged with it's successor [8].

Setuptools is not a fork of distutils. It's one of the aforementioned
products built on top of distutils that are very volatile to changes
in distutils.  Much of the functionality of setuptools is based on
subclasses of some of the class interfaces in distutils.  But it would
not be accurate to describe it as a fork.  (distribute however *is* a
fork of setuptools).

Thanks,
Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] special compiler options for only one file

[Erik Bray]

On Tue, Mar 19, 2013 at 7:54 PM, Andrew Dalke da...@dalkescientific.com wrote:
 Hi all,

   I have a Python extension which uses CPU-specific features,
 if available. This is done through a run-time check. If the
 hardware supports the POPCNT instruction then it selects one
 implementation of my inner loop, if SSSE3 is available then
 it selects another, otherwise it falls back to generic versions
 of my performance critical kernel. (Some 95%+ of the time is
 spent in this kernel.)

 Unfortunately, there's a failure mode I didn't expect. I
 use -mssse3 and -O3 to compile all of the C code, even though
 only one file needs that -mssse3 option.

 As a result, the other files are compiled with the expectation
 that SSSE3 will exist. This causes a segfault for the line

  start_target_popcount = (int)(query_popcount * threshold);

 because the compiler used fisttpl, which is an SSSE-3 instruction.
 After all, I told it to assume that ssse3 exists.

 The Debian packager for my package recently ran into this problem,
 because the test machine has a gcc which understands -mssse3 but
 the machine itself has an older CPU without those instructions.

 I'm trying to come up with a solution that can be automated for
 the Debian distribution. I want a solution where the same binary
 can work on older machines and on newer ones

 Ideally I would like to say that only one file is compiled
 with the -mssse3 option. Since my selector code isn't part of this
 file, SSSE-3 code will never be executed unless the CPU supports is.

 However, I can't figure out any way to tell distutils that
 a set of compiler options are specific to a single file.

 Is that even possible?


One possible solution, albeit more complex code-wise than compiling
that single file as a static lib, would be to subclass the compiler
class you want to use and override its _compile() method, which is the
one responsible for compiling a single file.  You can then override
the arguments in there.  Look, for example, at
distutils.unixcompiler.UnixCCompiler._compile.

If you want to support different compiler implementations, you could
even detect at runtime which compiler was selected with the --compiler
option and subclass the appropriate implementation.  Also getting
distutils to accept a compiler subclass requires a little bit of
hackery, but it's not undoable.  If that optioni sounds viable to you
I can delve into more details about how to actually do it, as I've had
to do this sort of thing before myself.  So yes, it can be done.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP439 and backward compat / easy_install / distlib

[Erik Bray]

On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth dho...@gmail.com wrote:
 My vision for the setuptools deprecation process is that distutils
 rides into the sunset with it. In this future eventually bugs in
 setuptools will be solved by porting setup.py to one of (X, Y, Z)
 which haven't necessarily been invented yet.

That would be nice (really!) but what are you proposing replace it for
building packages with heavy reliance on C extensions?  Because for
that one use case (and perhaps that alone) it works pretty okay for
most cases.  I don't want to start seeing an infinite number of ways
to configure and build extension modules.  The great thing about using
distutils (or some variant) for this is that if I had the source for a
package I could just `./setup.py build` and it would just work for
all but the most complex cases (SciPy for example).

I don't want to have a situation where some projects are using bento
and others are using scons and some are using waf and others are using
autoconf, etc, etc.  It's fine if a few projects have their own
special needs for build toolchains and I've been saying all along that
building should be separate from installing, and it should be easier
to drop in one's own build system.

Another thing that setuptools provides that currently works pretty
well with extension modules is `./setup.py develop`.  It calls
`setup.py build_ext --inplace` to make extension modules importable.
Any build system for extension modules needs to be able to do
something similar to support in-place install functionality like
`setup.py develop`, `pip install -e`, etc.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP439 and backward compat / easy_install / distlib

[Erik Bray]

On Tue, Mar 26, 2013 at 11:54 AM, Erik Bray erik.m.b...@gmail.com wrote:
 I don't think downloading the installer should be a side effect of
 running the installation either, but until this mess is cleaned up
 it's a necessary evil.  Yes, making things easier for users who don't
 know what they're doing is a legitimate use case.

I should clarify--when I write until this mess is cleaned up what I
really mean is, as soon as most packages have wheels built for them
for a wide range of platforms.  Then I don't really see it as an
issue :)

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP439 and backward compat / easy_install / distlib

[Erik Bray]

On Tue, Mar 26, 2013 at 11:21 AM, Tres Seaver tsea...@palladion.com wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 On 03/26/2013 05:28 AM, Ronald Oussoren wrote:

 On 25 Mar, 2013, at 19:16, PJ Eby p...@telecommunity.com wrote:


 Also, as far as detecting the need for setuptools, I think that can
 be done just by noticing whether the PKG-INFO included in an sdist
 is metadata 2.0 or not.  If it is, then setuptools should be
 explicitly declared as a build-time dependency, otherwise it's not
 needed.  If it's an older metadata version, then you probably need
 setuptools.

 Is it even necessary to automaticly install setuptools?
 Setuptools-using package are supposed to use  ez_setup.py, or
 distribute_setup.py for distribute, to ensure that the setuptools
 package is available during setup.

 No, they are not.  That usage was for bootstrapping in an era when
 setuptools was not widely presetn.  Most packages have *removed* those
 files today.

I still use distribute_setup.py very regularly.  I'm dealing with
scientific users, mostly on Macs or Windows who barely even know what
version of Python they have installed (or even what distribution of
Python--python.org/macports/homebrew/etc.) much less that they need
some variant of setuptools to install a large percentage of packages
out there.  Sometimes they do have setuptools installed but it's an
outdated version, or they didn't install it properly, or something to
that effect.

I don't think downloading the installer should be a side effect of
running the installation either, but until this mess is cleaned up
it's a necessary evil.  Yes, making things easier for users who don't
know what they're doing is a legitimate use case.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP439 and backward compat / easy_install / distlib

[Erik Bray]

On Tue, Mar 26, 2013 at 12:46 PM, Daniel Holth dho...@gmail.com wrote:
 On Tue, Mar 26, 2013 at 11:45 AM, Erik Bray erik.m.b...@gmail.com wrote:
 On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth dho...@gmail.com wrote:
 My vision for the setuptools deprecation process is that distutils
 rides into the sunset with it. In this future eventually bugs in
 setuptools will be solved by porting setup.py to one of (X, Y, Z)
 which haven't necessarily been invented yet.

 That would be nice (really!) but what are you proposing replace it for
 building packages with heavy reliance on C extensions?  Because for
 that one use case (and perhaps that alone) it works pretty okay for
 most cases.  I don't want to start seeing an infinite number of ways
 to configure and build extension modules.  The great thing about using
 distutils (or some variant) for this is that if I had the source for a
 package I could just `./setup.py build` and it would just work for
 all but the most complex cases (SciPy for example).

 I don't want to have a situation where some projects are using bento
 and others are using scons and some are using waf and others are using
 autoconf, etc, etc.  It's fine if a few projects have their own
 special needs for build toolchains and I've been saying all along that
 building should be separate from installing, and it should be easier
 to drop in one's own build system.

 Another thing that setuptools provides that currently works pretty
 well with extension modules is `./setup.py develop`.  It calls
 `setup.py build_ext --inplace` to make extension modules importable.
 Any build system for extension modules needs to be able to do
 something similar to support in-place install functionality like
 `setup.py develop`, `pip install -e`, etc.

 It's true that de-standardization of the build process has its own
 problems. As a consolation you don't have to do it as often because we
 have binary packages. It's hard to say how much fragmentation will
 happen, but distutils Extension() is an awful way to compile things!
 It has very little to recommend it apart from that it's there. For
 example, no parallel builds and no partial re-compiles based on what's
 changed. The trouble is that we know the packages on the cheeseshop
 mostly work but it's harder to count the stuff that avoids the
 cheeseshop because setuptools and setup.py wasn't a good solution.

 The example I've had to deal with recently is pycairo. They already
 use waf to compile their Python extension and don't use setup.py at
 all, so I argue that de-standardization has already happened. Whatever
 very easy (for pycairo) thing we can do make them pip install-able
 again is a plus.

 Even my own simple bcrypt wrapper cryptacular would compile an
 assembler file if I knew how to make that work with distutils. It
 doesn't and it's a little slower than it would have been if I had a
 decent build tool.

 I suspect at least 80% of packages will use some simple thing that
 comes with Python, two third party build tools will dominate, and we
 will discover interesting things that just weren't possible before. At
 least if someone wants to improve packaging we can make it easy for
 them to try without having to ask distutils-sig.

I pretty much agree with you on all of this, but I don't think the
question should be ignored either--avoiding this question is one of
the things that got previous packaging reform efforts into trouble.
Though the agreement to treat build and installation as two
different stories mitigates the issue this time around.  In any case
it's sort of off topic for this thread so I'll bring it up again
elsewhen.  One thing I see as a possible short-term solution is to
still rely on some version of distutils as a build tool *only*.  But
it would still be nice to have some easy way to standardize in-place
installation regardless of how extension modules get built.

Erik

P.S. pycairo does have a setup.py which worked for me, but the
installation instructions say it's unsupported, though I don't see
the waf script doing anything enormously different from it.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Builders vs Installers

[Erik Bray]

On Tue, Mar 26, 2013 at 2:48 PM, Nick Coghlan ncogh...@gmail.com wrote:
 On Tue, Mar 26, 2013 at 3:15 PM, PJ Eby p...@telecommunity.com wrote:
 More specifically, I was hoping to move the discussion forward on
 nailing down some of the details that still need specifying in a PEP
 somewhere, to finish out what the new world of packaging will look
 like.

 I'm deliberately trying to postpone some of those decisions - one of
 the reasons distutils2 foundered is because it tried to solve
 everything at once, and that's just too big a topic.

 So, *right now*, my focus is on making it possible to systematically
 decouple building from installing, so that running setup.py install
 on a production system becomes as bizarre an idea as running make
 install.

 As we move further back in the tool chain, I want to follow the lead
 of the most widely deployed package management systems (i.e. Debian
 control files and RPM SPEC files) and provide appropriate configurable
 hooks for *invoking* archivers and builders, allowing developers to
 choose their own tools, so long as those tools correctly emit
 standardised formats understood by the rest of the Python packaging
 ecosystem.

Right--what we really need here is something akin to the debian/rules
file, only not a shell script :)  I like the hook idea.  It's the so
long as those tools correctly emit standardised formats that's the
problem.

 In the near term, however, these hooks will still be based on setup.py
 (specifically setuptools rather than raw distutils, so we can update
 older versions of Python).

That pretty much eases the concerns I brought up in the backwards
compat thread.

 But, if we support either you have a setup.cfg specifying your
 archiver, or a PKG-INFO so an archiver isn't needed, then that would
 probably cover all the bases, actually.

 Yeah, you're probably right that we will need to support something
 else in addition to the PKG-INFO file. A PKG-INFO.in could work,
 though, rather than a completely independent format like setup.cfg.
 That way we could easily separate a source checkout/tarball (with
 PKG-INFO.in) from an sdist (with PKG-INFO) from a wheel (with a named
 .dist-info directory).

I'm partly in favor of just saying, there should be a PKG-INFO in
your version control to be considered a valid python distribution.
Intermediate formats like a setup.cfg or Nick's JSON format seem kind
of unnecessary to me--why have two different formats to describe the
same thing?  In cases where the metadata needs to be mutated
somehow--such as attaching revision numbers to the version--some sort
of PKG-INFO.in like you suggest would be great. But I don't see why it
should have a different format from PKG-INFO itself.  I'd think it
would just be a superset of the metadata format but with support for
hooks.  Basically akin to what d2to1 does with setup.cfg, but without
the unnecessarily different-looking intermediate format

(I do agree that a JSON format would allow a much greater degree of
expression and flexibility, but I'm not entirely sure how I feel about
having one file format that generates an entirely different file
format).

 (For consistency, we may want to rename PKG-INFO to DIST-INFO in sdist
 2.0, though)

+1

Thanks,

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Builders vs Installers

[Erik Bray]

On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby p...@telecommunity.com wrote:
 On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth dho...@gmail.com wrote:
 I think PKG-INFO is a highly human-editable format.

 That doesn't mean you necessarily want to edit it yourself; notably,
 there will likely be some redundancy between the description in the
 file and other files like the README.

 Also, today one of the key use cases people have for custom code in
 setup.py is to pull the package version from a __version__ attribute
 in a module.  (Which is evil, of course, but people do it anyway.)

 But it might be worth adding a setuptools feature to pull metadata
 from PKG-INFO (or DIST-INFO) instead of generating a new one, to see
 what people think of using PKG-INFO first, other files second.  In
 principle, one could reduce a setup.py to just from setuptools import
 setup_distinfo; setup_distinfo() or some such.

In other words, using d2to1 and only for `setup.py egg_info` (only not
egg_info but whatever we're doing instead to generate the metadata ;)

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Builders vs Installers

[Erik Bray]

On Tue, Mar 26, 2013 at 5:21 PM, Daniel Holth dho...@gmail.com wrote:
 I want to poke myself in the eye every time I have to edit json by hand.
 Especially the description field.

I'm with you on that--I much prefer YAML (which is a superset of
JSON!) but we don't even have that in the stdlib and it's not worth
bikeshedding over to me.

 On Mar 26, 2013 5:17 PM, Donald Stufft don...@stufft.io wrote:


 On Mar 26, 2013, at 5:12 PM, Daniel Holth dho...@gmail.com wrote:

  I am -1 on renaming anything unless it solves a technical problem.
  Forever after we will have to explain well, it used to be called X,
  now it's called Y...
 
  On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray erik.m.b...@gmail.com
  wrote:
  On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby p...@telecommunity.com wrote:
  On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth dho...@gmail.com
  wrote:
  I think PKG-INFO is a highly human-editable format.
 
  That doesn't mean you necessarily want to edit it yourself; notably,
  there will likely be some redundancy between the description in the
  file and other files like the README.
 
  Also, today one of the key use cases people have for custom code in
  setup.py is to pull the package version from a __version__ attribute
  in a module.  (Which is evil, of course, but people do it anyway.)
 
  But it might be worth adding a setuptools feature to pull metadata
  from PKG-INFO (or DIST-INFO) instead of generating a new one, to see
  what people think of using PKG-INFO first, other files second.  In
  principle, one could reduce a setup.py to just from setuptools import
  setup_distinfo; setup_distinfo() or some such.
 
  In other words, using d2to1 and only for `setup.py egg_info` (only not
  egg_info but whatever we're doing instead to generate the metadata ;)
 
  Erik
  ___
  Distutils-SIG maillist  -  Distutils-SIG@python.org
  http://mail.python.org/mailman/listinfo/distutils-sig


 Rename it and make it JSON instead of the homebrew* format!

 * Yes techincally it's based on a real format, but that format doesn't
 support all the things it needs so there are hackishly added extensions
 added to it.

 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
 DCFA


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Setuptools-Distribute merge announcement

[Erik Bray]

On Wed, Mar 13, 2013 at 8:54 PM, PJ Eby p...@telecommunity.com wrote:
 Jason Coombs (head of the Distribute project) and I are working on
 merging the bulk of the improvements distribute made into the
 setuptools code base.  He has volunteered to take over maintenance of
 setuptools, and I welcome his assistance.  I appreciate the
 contributions made by the distribute maintainers over the years, and
 am glad to have Jason's help in getting those contributions into
 setuptools as well.  Continuing to keep the code bases separate isn't
 helping anybody, and as setuptools moves once again into active
 development to deal with the upcoming shifts in the Python-wide
 packaging infrastructure (the new PEPs, formats, SSL, TUF, etc.), it
 makes sense to combine efforts.

 Aside from the problems experienced by people with one package that
 are fixed in the other, the biggest difficulties with the fork right
 now are faced by the maintainers of setuptools-driven projects like
 pip, virtualenv, and buildout, who have to either take sides in a
 conflict, or spend additional time and effort testing and integrating
 with both setuptools and distribute.  We'd like to end that pain and
 simplify matters for end users by bringing distribute enhancements to
 setuptools and phasing out the distribute fork as soon as is
 practical.

 In the short term, our goal is to consolidate the projects to prevent
 duplication, wasted effort, and incompatibility, so that we can start
 moving forward. This merge will allow us to combine resources and
 teams, so that we may focus on a stable but actively-maintained
 toolset.  In the longer term, the goal is for setuptools as a concept
 to become obsolete.  For the first time, the Python packaging world
 has gotten to a point where there are PEPs *and implementations* for
 key parts of the packaging infrastructure that offer the potential to
 get rid of setuptools entirely.  (Vinay Sajip's work on distlib,
 Daniel Holth's work on the wheel format, and Nick Coghlan's taking
 up the reins of the packaging PEPs and providing a clear vision for a
 new way of doing things -- these are just a few of the developments in
 recent play.)

 Obsolete, however, doesn't mean unmaintained or undeveloped.  In
 fact, for the new way of doing things to succeed, setuptools will
 need a lot of new features -- some small, some large -- to provide a
 migration path.

 At the moment, the merge is not yet complete.  We are working on a
 common repository where the two projects' history has been spliced
 together, and are cleaning up the branch heads to facilitate
 re-merging them.  We'd hoped to have this done by PyCon, but there
 have been a host of personal, health, and community issues consuming
 much of our available work time.  But we decided to go ahead and make
 an announcement *now*, because with the big shifts taking place in the
 packaging world, there are people who need to know about the upcoming
 merge in order to make the best decisions about their own projects
 (e.g. pip, buildout, etc.) and to better support their own users.

 Thank you once again to all the distribute contributors, for the many
 fine improvements you've made to the setuptools package over the
 years, and I hope that you'll continue to make them in the future.
 (Especially as I begin to phase myself out of an active role in the
 project!)

 I now want to turn the floor over to Jason, who's put together a
 Roadmap/FAQ for what's going to be happening with the project going
 forward.  We'll then both be here in the thread to address any
 questions or concerns you might have.

Quick question regarding open issues on Distribute (of which I have a
handful assigned to me, and of which I intend to tackle a few others):
 Would it it make sense to just hold off on those until the merge is
completed?  Also is there anything I can do to help with the merge?
How is that coming along?

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distlib - installer support vs runtime support

[Erik Bray]

On Sat, Mar 9, 2013 at 9:33 PM, PJ Eby p...@telecommunity.com wrote:
 On Sat, Mar 9, 2013 at 8:14 PM, Nick Coghlan ncogh...@gmail.com wrote:
 Longer term, something like the import engine PEP may let us implement a
 cleaner solution.

 I've been giving the bootstrapping issue a bit more thought, though,
 and I think there's a way to take the pain out of multi-file
 bootstraps of things like pip and setuptools and whatnot.

 Suppose you make a .py file that contains (much like ez_setup.py) a
 list of files to download, along with their checksums.  And suppose
 this .py file basically just has some code that checks a local cache
 directory for those files, adds them to sys.path if they're found,
 downloads them if they're not (validating the checksum), and then
 proceeds to import a specific entry point and run it?

I've been thinking about something like this lately too. A simple
script like your proposed pystart could be used to generate these
files.  I've gone one further and considered a format that includes
dependencies (or at the very least install-time dependencies right in
the .py file itself as a base64 string.  This would work fine at least
for small dependencies--no downloads necessary and you're guaranteed
to get the right version that actually installs the damn package.

For packages using setuptools it's also possible to just include a
dependency as an egg or tar.gz right in your source dist and add
something like [easy_install]\nfind_links = . to your setup.cfg and
it'll go.  But I'm not sure that's exactly the use case you're after
here.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 426, round 733 ;)

[Erik Bray]

On Tue, Feb 5, 2013 at 9:08 AM, Daniel Holth dho...@gmail.com wrote:
 On Tue, Feb 5, 2013 at 7:54 AM, Donald Stufft donald.stu...@gmail.com
 wrote:

 On Tuesday, February 5, 2013 at 5:35 AM, a.cava...@cavallinux.eu wrote:

 Ideally it would be something that connects to the source revision number
 (as in
 subversion) or the integral id or even a timestamp (that's what an
 exported
 version must be).

 Timestamp or reversion number is not overall useful number for a version
 and
 here's why: Django (for example) maintains older versions for a set period
 of time,
 If you do it via timestamps than 1.1.1 which happens to be released after
 1.2.0 (because
 of a security issue or glaring bug) will be considered newer. Handwaving
 the problem
 away with a source revision number or timestamp ignores a fundamental
 property
 of a version


 What pkg_resources does for sorting is to append final for sorting. So the
 sorting is really just a, b, c, f. This scheme has been important for 8+
 years now. PEP 386, from 2009, narrows the allowed versions. Now the salient
 part of PEP 386 has been incorporated into Metadata 1.3.

 1.0.0a0
 1.0.0b0
 1.0.0c0 ~= 1.0.0rc0
 1.0.0 == 1.0.0f

 The marketing pre-release feature exists to allow publishers to put
 immature versions of their software on pypi where they can be easily
 downloaded. Recently SQLAlchemy did this but had to delete the beta release
 from pypi because too many deployments upgraded to an unstable version
 without realizing it. Once the tools are updated it will be easy to install
 a beta release with pip if and only if you specifically ask for it.

Yes, this is incredibly useful and I'm glad it was incorporated into
this PEP.  For the Astropy project for example we recently release
version 0.2b2 and we absolutely want to be able to put it on PyPI for
users to test out.  But as the current situation stands we can't do
that because 0.2b2 then becomes the latest version, and the one the
gets installed when users expecting the latest stable release do `pip
install astropy`.

So instead we've just been adding the pre-final release to testpypi
and pointing people there. And that's not really meant to be a
reliable service.  So...yeah.  A++ for adding this requirement.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 426, round 733 ;)

[Erik Bray]

On Tue, Feb 5, 2013 at 10:53 AM, Donald Stufft donald.stu...@gmail.com wrote:
 On Tuesday, February 5, 2013 at 10:49 AM, a.cava...@cavallinux.eu wrote:

 Or until when somebody will explain if X.Y.devN comes before of after
 X.Y.N...

 Easy, before.

Which is explained in the PEP pretty clearly I thought. But if you're
still not sure there were lengthy discussions on this very mailing
list that led to that decision.  For what it's worth, there were
people who argued the opposite too, but eventually it seems like a
consensus was reached.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Differences in PEP386 and setuptools

[Erik Bray]

On Thu, Sep 27, 2012 at 7:15 PM, Toshio Kuratomi a.bad...@gmail.com wrote:
 So to be more complete, the proposal is to do this::

 0.1
0.1.post1
1.0.dev1# The .dev that's changing position
1.0a1.dev1
1.0a1
1.0a1.post1
1.0b1.dev1
1.0b1
1.0b1.post1
1.0c1.dev1
1.0c1
1.0c1.post1
 # The current PEP386 position of 1.0.dev1
1.0
1.0.post1


This looks perfect to me, and very unambiguous.  Allowing the .dev and
.post modifiers at every release stage I think clears up the issue
here better than any other proposal.  +1.

(This is also semantically the same as semver--just not syntactically
the same. I like semver but I think we're stuck with the existing
syntax which is fine.)

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Differences in PEP386 and setuptools

[Erik Bray]

On Wed, Sep 26, 2012 at 4:24 PM, Donald Stufft donald.stu...@gmail.com wrote:
 On Wednesday, September 26, 2012 at 4:00 PM, Toshio Kuratomi wrote:

 Note that this was an intentional difference with setuptools.

 Do you have any idea _why_? It seems from my perspective (of someone
 who wasn't really paying attention back then) to be a fairly arbitrary
 difference
 in both prior art and what I've personally seen as the common usage that has
 a good chance of causing confusion (as it already has).


My guess--and this is pure speculation as I wasn't following the
original discussions either--is that it fell out of the desire to
require versions to be lexicographically ordered.  So dev would
*have* to come after a, b, and c whether we like it or not.  But
at some point the convention to use .dev was proposed, which solves
this issue--this puts it before a lexicographically and thus makes
more sense in general.  Although .dev made it into PEP 386 it seems
the rest of the PEP wasn't changed to reflect the fact that .dev now
comes first.

Again--just speculation.  But I don't see any way that the PEP as
written makes sense without assuming some kind of editorial oversight.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

[Erik Bray]

On Thu, Sep 20, 2012 at 4:24 PM, Daniel Holth dho...@gmail.com wrote:
 And they answer most of your questions.  A few call-outs below:

 dependency_links.txt: url's of the package's dependencies. Speak up if
 you use this; it is very surprising, and has a much better replacement
 with pip --index options and requirements files.

 I will check to see how often this is used.

I definitely use this. This way all my packages can get their
dependencies from my own package index instead of PyPI. This gives a
greater deal of control in dealing with users who are angry/frustrated
when they can't reach PyPI for some reason. Though I think this will
be less of an issue with PyPI mirror support.  Though it's also nice
to be able to release development packages that have dev versions of
other packages as dependencies without having to put all my dev
releases on PyPI.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

[Erik Bray]

On Fri, Sep 21, 2012 at 12:57 AM, PJ Eby p...@telecommunity.com wrote:
 I'm not seeing how a documented file discovery protocol is
 guesswork.  Perhaps you've not read the documentation?  e.g.:

 http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-in-pypi
 http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall

Exactly.  This is the opposite of guesswork, in fact.  I have use
cases where I want to know/specify exactly where my dependencies are
coming from, and not just put out an abstract name.  I agree this
information shouldn't be a part of the package's Metadata.  I think
the existing solution in setuptools of including it in a
dependency_links.txt file is fine.  This wouldn't be an option though
with Nick's one JSON file to rule them all proposal (which I'm not
against--it just precludes keeping dependency-links-like info out of
band).

 As far as the practicality vs. purity question, Python has already had
 Provides/Requires in the metadata format for several years, and it
 contained all the features that were needed for a pure dependency
 resolution system.  In *theory* you could meet everyone's use cases
 with that, if you're going to assume that all packages live in PyPI
 and everyone has a compiler.  And then there wouldn't be any need for
 any of the new metadata specs.

 In practice, however, not everything's in PyPI and not everybody has
 compilers, and not every package is instantly updated or maintained
 indefinitely.  If you don't like how dependency links solve the
 problem, perhaps you could propose some other way of handling those
 use cases, such that developer B's dependency maintenance burdens are
 not propagated to developer A and his peer/parent dependers?

And not everybody wants to put everything on PyPI. In fact, I once had
a use case to build a system on top of pkg_resources that could locate
dependencies in SVN repository checkouts by adding their paths to
dependency_links/find_links.  There are all kinds of uses for this
kind of thing that I don't think should be ignored, and pkg_resources
has already provided the means to implement most of them (the
aforementioned example also required subclassing easy_install to work
properly, but that wasn't too big a deal).

I think a broader point is that package distribution and dependency
resolution has some ugly and messy corners and there's no way around
that.  Just saying, Well all the dependencies should be clearly
documented and it's up to the user to figure how to assemble them all
isn't going to fly either.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

[Erik Bray]

On Fri, Sep 21, 2012 at 3:06 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On Fri, Sep 21, 2012 at 2:57 PM, PJ Eby p...@telecommunity.com wrote:
 This is no more guesswork than the PyPI /simple index discovery protocol is.

 You have zero idea what's at the end of a URL link. You're just hoping
 it's the file you expect.

That's not really something the average science user is going to care
about.  All they care is that when they run `setup.py install` like I
told them to it installs the package and gets all the right versions
of the dependencies, some of which may not be on PyPI, or have been
repackaged for some reason because the official package that is on
PyPI was created by someone who doesn't know what they're doing and
isn't inclined to fix it.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] environment markers in setup.cfg -- vs ;

[Erik Bray]

On Wed, Sep 19, 2012 at 9:26 PM, Daniel Holth dho...@gmail.com wrote:
 I am implementing requirements-in-setup.cfg for wheel because there is
 no place to put environment markers in setup.py

 The Requires-Dist: lines in PKG-INFO use ; to separate environment
 markers from the dep name. Is setup.cfg supposed to use -- ? Why?

 requires-dist = keyring; python_version == '2.6'

 versus

 requires-dist = keyring -- python_version == '2.6'

(Sorry, meant to reply-all; hate how that's not a default on
python.org mailing lists.)

Where does it say that?  I see to recall it always using ; both in the
PKG-INFO and in setup.cfg.  I'm probably mistaken, but I'm just
wondering where you got this from?  And if so I wonder if it could be
changed. The -- syntax is just confusing IMO.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] zip imports ftw

[Erik Bray]

On Sun, Sep 16, 2012 at 3:33 PM, Daniel Holth dho...@gmail.com wrote:
 It is because the file has a controversial coding style, with up to 25
 lines of whitespace between classes, and includes functionality for
 eggs and some other stuff that was unwanted on top of the resources
 stuff that partly made its way into pkgutil.

I don't see why that couldn't be fixed though.  As it is, almost every
time I make a patch to distribute I clean things up a bit, in part
because I have vim configured by default to run autopep8 and some
other scripts on every Python file I open so I don't have to look at
*quite* as much ugly code (this has proven very useful at work).

 I am really impressed with the file after seeing how easy it was to
 add PEP-345 and PEP-376 support to pkg_resources.py without having to
 present a different API to the user, and seeing how it provides a lot
 of what an installer needs especially being able to iterate over all
 the distributions in a particular folder, but I am glad it is not in
 the stdlib because I wouldn't have been able to do that if it was.

Yes--I frequently have found myself wishing pkg_resources'
availability could be assumed. Especially for getting handles to
non-code resources, and for things like get_distribution().

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP386 Pre releases

[Erik Bray]

On Tue, Sep 11, 2012 at 5:04 PM, Donald Stufft donald.stu...@gmail.com wrote:
 Why does 1.0a1 sort before 1.0.dev1? It appears to me that common
 usage in the wild of dev1 releases is that they are used for the
 development version before any sort of alpha, beta, rc, or final has
 been cut?

If I had to guess, the rationale might be that when developing a new
version you're developing toward the first alpha release, so
something like .dev1 would be a development preview of a final release
after all a/b/rc pre-releases have been released.

That said, this doesn't match my workflow at all.  After releasing
1.0 the next version is going to be 1.1, and any development
pre-release will be 1.1.devX.  1.1a might not ever even exist.  I
think others brought up this critique at the time PEP 386 was being
discussed, but then nothing was ever done about it _

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP386 Pre releases

[Erik Bray]

On Wed, Sep 12, 2012 at 1:46 PM, Daniel Holth dho...@gmail.com wrote:
 Isn't the pep carefully constructed to respect lexicographical sorting?

Yes.  That's why it's .dev and not just dev.  Lexicographically,
1.0.dev  1.0a.

 On Sep 12, 2012 1:44 PM, Erik Bray erik.m.b...@gmail.com wrote:

 On Tue, Sep 11, 2012 at 5:04 PM, Donald Stufft donald.stu...@gmail.com
 wrote:
  Why does 1.0a1 sort before 1.0.dev1? It appears to me that common
  usage in the wild of dev1 releases is that they are used for the
  development version before any sort of alpha, beta, rc, or final has
  been cut?

 If I had to guess, the rationale might be that when developing a new
 version you're developing toward the first alpha release, so
 something like .dev1 would be a development preview of a final release
 after all a/b/rc pre-releases have been released.

 That said, this doesn't match my workflow at all.  After releasing
 1.0 the next version is going to be 1.1, and any development
 pre-release will be 1.1.devX.  1.1a might not ever even exist.  I
 think others brought up this critique at the time PEP 386 was being
 discussed, but then nothing was ever done about it _

 Erik
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP345 - Distutils2 - Cannot pin to exact version

[Erik Bray]

On Tue, Sep 11, 2012 at 2:35 PM, Donald Stufft donald.stu...@gmail.com wrote:
 I was digging through PEP386  PEP345 today, and I noticed something odd
 about the wording of PEP345.

 It states:

 When a version is provided, it always includes all versions that starts
 with the same value. For
 example the 2.5 version of Python will include versions like 2.5.2
 or 2.5.3. Pre and post
 releases in that case are excluded. So in our example, versions like
 2.5a1 are not included
 when 2.5 is used. If the first version of the range is required, it
 has to be explicitly given. In
 our example, it will be 2.5.0.

 It also states:

 In that case, 2.5.0 will have to be explicitly used to avoid any
 confusion between the 2.5
 notation that represents the full range. It is a recommended practice to
 use schemes of the
 same length for a series to completely avoid this problem.

 This effectively translates to an inability to pin to an exact version. Even
 in the case of specifying
 == it checks that the version starts with the value you selected. So if
 you pin to 2.5, and the
 author then releases 2.5.1, that will count as ==2.5. If you try to then
 pin to 2.5.0, and the
 author releases 2.5.0.1, then that will count as ==2.5.0.

 Essentially this translates to:

 ==2.5   - =2.52.6
 ==2.5.0- =2.5.02.5.1
 ==2.5.0.0 - =2.5.0.02.5.0.1

 Which means that version specifiers are _always_ ranges and are never exact
 versions. The PEP
 as written relies on authors to decide beforehand how many digits they are
 going to use in their
 versions, and for them to never increase or decrease that number.

 I also checked to see if Distutils2/packaging implemented VersionPredicates
 that way or if they
 allowed specifying an exact version. It turned out that it implements the
 PEP as written:

 from distutils2 import version
 predicate = version.VersionPredicate(foo (==2.5))
 print predicate
 foo (==2.5)
 predicate.match(2.5)
 True
 predicate.match(2.5.0)
 True
 predicate.match(2.5.0.0)
 True
 predicate.mach(2.5.0.5)
 True

That's kind of annoying.  Does anyone know if this is by design?

FWIW there is a workaround. For example if you want to pin to exactly 2.5.1:

 predicate = version.VersionPredicate(foo (==2.5.1,2.5.1.1))
 predicate.match('2.5.1')
True
 predicate.match('2.5.2')
False
 predicate.match('2.5.1.0')
True
 predicate.match('2.5.1.1')


Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP345 - Distutils2 - Cannot pin to exact version

[Erik Bray]

On Tue, Sep 11, 2012 at 3:19 PM, Daniel Holth dho...@gmail.com wrote:
 On Tue, Sep 11, 2012 at 3:11 PM, Erik Bray erik.m.b...@gmail.com wrote:
 On Tue, Sep 11, 2012 at 2:35 PM, Donald Stufft donald.stu...@gmail.com 
 wrote:
 I was digging through PEP386  PEP345 today, and I noticed something odd
 about the wording of PEP345.

 It states:

 When a version is provided, it always includes all versions that starts
 with the same value. For
 example the 2.5 version of Python will include versions like 2.5.2
 or 2.5.3. Pre and post
 releases in that case are excluded. So in our example, versions like
 2.5a1 are not included
 when 2.5 is used. If the first version of the range is required, it
 has to be explicitly given. In
 our example, it will be 2.5.0.

 It also states:

 In that case, 2.5.0 will have to be explicitly used to avoid any
 confusion between the 2.5
 notation that represents the full range. It is a recommended practice to
 use schemes of the
 same length for a series to completely avoid this problem.

 This effectively translates to an inability to pin to an exact version. Even
 in the case of specifying
 == it checks that the version starts with the value you selected. So if
 you pin to 2.5, and the
 author then releases 2.5.1, that will count as ==2.5. If you try to then
 pin to 2.5.0, and the
 author releases 2.5.0.1, then that will count as ==2.5.0.

 Essentially this translates to:

 ==2.5   - =2.52.6
 ==2.5.0- =2.5.02.5.1
 ==2.5.0.0 - =2.5.0.02.5.0.1

 Which means that version specifiers are _always_ ranges and are never exact
 versions. The PEP
 as written relies on authors to decide beforehand how many digits they are
 going to use in their
 versions, and for them to never increase or decrease that number.

 I also checked to see if Distutils2/packaging implemented VersionPredicates
 that way or if they
 allowed specifying an exact version. It turned out that it implements the
 PEP as written:

 from distutils2 import version
 predicate = version.VersionPredicate(foo (==2.5))
 print predicate
 foo (==2.5)
 predicate.match(2.5)
 True
 predicate.match(2.5.0)
 True
 predicate.match(2.5.0.0)
 True
 predicate.mach(2.5.0.5)
 True

 That's kind of annoying.  Does anyone know if this is by design?

 FWIW there is a workaround. For example if you want to pin to exactly 2.5.1:

 predicate = version.VersionPredicate(foo (==2.5.1,2.5.1.1))
 predicate.match('2.5.1')
 True
 predicate.match('2.5.2')
 False
 predicate.match('2.5.1.0')
 True
 predicate.match('2.5.1.1')

 But you could still release 2.5.1.0.0? I suppose we limit the number
 of version parts these days.

 Why don't we update the spec so that (2.0) means (2.0) the range, and
 (==2.0) means 2.0 (exactly).

 Daniel

The PEP is ambiguous on this, but you could get away with reading it
as When a version is provided (without a conditional operator) it
always includes all versions that start with the same value.
Although it's unwritten in the PEP exactly how the operators are meant
to be interpreted, I would say they should be interpreted strictly.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding Provides-Extra as an edit to PEP 345

[Erik Bray]

On Mon, Jul 9, 2012 at 10:28 PM, Daniel Holth dho...@gmail.com wrote:
 New draft up at 
 https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068

 Adds Provides-Extra: (with longer explanation), Setup-Requires-Dist:
 (full text below)

 Setup-Requires-Dist lives because it makes sense to list packages only
 needed for setup.py to run before install is possible.

 The reserved extra names test and doc live. No one should mind
 installing a distribution and its dependencies in order to run the
 unit tests or generate the documentation.


 Setup-Requires-Dist (multiple use)
 ::

 Like Requires-Dist, but names dependencies needed while the
 distributions's distutils / packaging `setup.py` / `setup.cfg` is run.
 Commonly used to generate a manifest from version control.

 Examples::

 Setup-Requires-Dist: custom_setup_command

 Dependencies mentioned in `Setup-Requires-Dist` may be installed
 exclusively for setup and are not guaranteed to be available at run
 time.

+1 To this.  I agree Setup-Requires and Provides-Extra are different
use cases and should both be added to the spec.  I think that filling
in these two missing features will benefit packaging in the long run.
If this idea gains some amount of acceptance I'm happy to work on an
implementation of the setup-requires end of things for packaging.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding Provides-Extra as an edit to PEP 345

[Erik Bray]

On Fri, Jul 6, 2012 at 11:20 AM, Daniel Holth dho...@gmail.com wrote:
 It is exactly the same as setuptools' extras feature, and I didn't
 quote the entire edit on-list. Also, there is a reference
 implementation at https://bitbucket.org/dholth/distribute/ in the
 DistInfoDistribution class.

 The 'test' and 'doc' reserved names are not an essential part of the
 Provides-Extra feature. Provides-Extra: is primarily for optional
 run-time (not build-time) dependencies.

Though in the interest of not having too many different ways to do a
similar thing, could this not also be used for tests/docs?  Perhaps
even Provides-Extra: tests, and Provides-Extra: docs should be
reserved names as you suggested.  I envision something like:


Provides-Extra: tests
Requires-Dist: nose; extra == 'tests'
Provides-Extra: docs
Requires-Dist: sphinx; extra == 'docs'


$ pysetup test
Tests require the 'nose' package. Please run `pysetup install
mypackage[tests]` to install the test dependencies.

Ditto for building the docs.  In a sense, being able to run the tests,
or build the docs could be considered 'extra' features.  One could
even have:

Provides-Extra: setup
Requires-Dist: mysetuphooks; extra == 'setup'

for build-time dependencies.  Although in that case, pysetup (or any
other installer) would explicitly have to check for this in the
metadata before doing anything else.  So maybe this example stretches
this syntax too far?  Plus build-time dependencies are not exactly
extras.  It's very much required to do anything with the package.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding Provides-Extra as an edit to PEP 345

[Erik Bray]

On Mon, Jul 9, 2012 at 1:27 PM, Daniel Holth dho...@gmail.com wrote:
 Though in the interest of not having too many different ways to do a
 similar thing, could this not also be used for tests/docs?  Perhaps
 even Provides-Extra: tests, and Provides-Extra: docs should be
 reserved names as you suggested.  I envision something like:

 ...

 Provides-Extra: setup
 Requires-Dist: mysetuphooks; extra == 'setup'

 for build-time dependencies.  Although in that case, pysetup (or any
 other installer) would explicitly have to check for this in the
 metadata before doing anything else.  So maybe this example stretches
 this syntax too far?  Plus build-time dependencies are not exactly
 extras.  It's very much required to do anything with the package.

 Pre-built or binary packages won't want to install the build-time
 dependencies. The check for a particular Provides-Extra is trivial,
 just a list membership test.

 I expect to see a lot of bugs in packages that require some of their
 build-time dependencies at run time.

Maybe, but wouldn't that be the developer's problem?  setup_requires
has the same issue--it does not install the setup requirements into
site-packages.  Instead it just does an egg install into the source
directory and activates it on the path.  Outside of future setup.py
runs, the setup_requires distributions are not available (you would
have to add them to install_requires too if you need them at runtime).

Under a scheme like this one would have to list that dependency under
Requires-Dist twice: with and without the 'extra' marker.  I might
still prefer extending the metadata format to add a
Setup-Requires-Dist or the like.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Adding Provides-Extra as an edit to PEP 345

[Erik Bray]

On Mon, Jul 9, 2012 at 4:37 PM, Daniel Holth dho...@gmail.com wrote:
 I expect to see a lot of bugs in packages that require some of their
 build-time dependencies at run time.

 Maybe, but wouldn't that be the developer's problem?  setup_requires
 has the same issue--it does not install the setup requirements into
 site-packages.  Instead it just does an egg install into the source
 directory and activates it on the path.  Outside of future setup.py
 runs, the setup_requires distributions are not available (you would
 have to add them to install_requires too if you need them at runtime).

 OK, it sounds like this bug has been mostly anticipated and taken
 care of already.

 Under a scheme like this one would have to list that dependency under
 Requires-Dist twice: with and without the 'extra' marker.  I might
 still prefer extending the metadata format to add a
 Setup-Requires-Dist or the like.

 I don't follow. Does anyone build/install packages without also first
 installing the run-time requirements? Surely Setup-Requires-Dist would
 have exactly the same problem. It seems like two ways to say exactly
 the same thing.

The difference is that setup/build dependencies generally should be
downloaded, extracted, and added to the path before another commands
are run, because the packaging framework allows for the behavior of
built-in commands themselves to be modified, as well as certain
metadata.  The usefulness of this has been discussed in past threads.
See

http://mail.python.org/pipermail/distutils-sig/2012-May/018627.html

and the thread Eric Araujo pointed out at:

https://groups.google.com/forum/?fromgroups#!searchin/the-fellowship-of-the-packaging/setup-requires/the-fellowship-of-the-packaging/KVsDQawfEno/M-ODKx9Ne5YJ

 Let's just try to focus on getting Metadata 1.2 out the door and
 implemented before its 8th birthday, preferably during Julython... The
 only thing I need is to be able to represent the wildly popular
 extras feature in METADATA to benefit from PEP 345 and 376 without
 rewriting thousands of Python packages. That is all. Save the debate
 about reserved extra names for setup / build / test / doc
 dependencies for Metadata 1.3.

I don't have data on the popularity of setup_requires--is there some
easy way to find that out without downloading an entire PyPI mirror?

That said, I'm anticipating that it would be very useful for
packaging.  As it is, setup.py files tend to contain a lot of
repetitive boilerplate, especially for non-pure-Python packages.  With
the elimination of setup.py, more packages are going to require setup
hook-script modules to be shipped with their code, and I could see
some of these being highly reusable.

I'm certain this is a useful feature--just not so certain whether it's
desirable needed to modify the Metadata spec to account for it.

In the worst case, I could imagine writing some kind of bootstrap
module that can be dropped into a project and used via a setup-hook
that is then possible of downloading other build-time dependencies.
But that's exactly the sort of thing I'm trying to get away from--if I
have to maintain a bootstrap script (or another module containing
setup-hooks, custom commands, etc.) I have to update every project
that uses it every time those scripts are updated.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Packaging dependencies

[Erik Bray]

On Tue, Jun 12, 2012 at 4:40 PM, Keshav Kini keshav.k...@gmail.com wrote:
 Hi,

 We're using distribute to package our software, but we'd like to do
 something different with the sdist command. We would like to
 recursively find all packages necessary to build our package from a base
 Python install, download these packages, and store them all in a
 directory inside our source distribution tarball; then, when the tarball
 is unpacked and setup.py is run, we want install to use those cached
 packages. (This is to support installation on machines that can't or
 shouldn't access the network.)

 One solution I found in the distribute docs was to use `easy_install
 -zmaxd dirname package` on our package to generate and store dependency
 eggs, and then `easy_install -H none -f dirname` when installing the
 package. Unfortunately this actually builds packages which have C
 extensions and stuff like that in them, whereas we would like to only
 ship the source and build the dependencies when our package is built (to
 support multiple platforms). Also I guess we'd like `setup.py install`
 to just work in the directory without the user having to care about the
 fact that these source packages were shipped inside the tarball they
 just extracted.

 So I'm wondering if I can actually do this by importing something from
 distribute itself and writing some nontrivial code in setup.py. Can
 someone point me to what would be the best way to do this?

 I'm guessing I'll need to start by subclassing Environment to create a
 fake default environment which looks like how a bare Python 2.7.3
 (say) install would. Then I'd need to somehow modify sdist somewhere
 to make it download the dependency tarballs that would be necessary if
 you were installing our packages into that Environment, and also need to
 modify install to make it look in the correct directory. Is that about
 right?

 Thanks and sorry for the simple question.

I've found that the find_links option works fine for this sort of
thing.  For example, you can put source tarballs of your dependencies
in a directory under your source tree like `dependencies/`, and then
in setup.cfg add:

[easy_install]
find_links = dependencies

Something like this has worked for me in the past, I think.

As for automatically downloading the source, adding a custom subclass
of sdist seems like the way to go.  The setuptools package contains
the required machinery to download dependencies from PyPI (or another
package index) with or without installing them.

Not sure I fully understand what the issue is regarding building C extensions.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Requires-Dist: unittest2; 'test' in extras

[Erik Bray]

On Wed, May 30, 2012 at 5:35 PM, PJ Eby p...@telecommunity.com wrote:
 On Wed, May 30, 2012 at 3:09 PM, Daniel Holth dho...@gmail.com wrote:

 It looks like you can only install one extra at a time,


 Actually, you can specify more than one, using commas.  e.g. easy_install
 foo[testing,c-extensions,celery-support,...].

Would we want this to look the same way for pysetup?  Something like
`pysetup install foo[tests,docs]`?  That would be a pretty nice way to
handle docs, tests, and other miscellaneous extra requirements.  I
like the idea of using environment markers for that and not having to
add any new metadata fields.

Using an environment marker might also work for some kind of
build-requires, though support for that would still require some
special machinery.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command hooks...

[Erik Bray]

On Tue, May 29, 2012 at 7:41 PM, PJ Eby p...@telecommunity.com wrote:
  I might be confused; I haven't been following the goings-on of late with
  distutils2.  At one point, I thought the plan was not to bless or
  include dependency-managing installers with the stdlib, or something
  like that.  i.e., I thought the plan wasn't to support or bless
  full-service tools like buildout, easy_install, or pip, or anything
  comparable to them.

 Right, yeah, the plans in this area were fluid for awhile, but the
 eventual conclusion was that the stdlib should have a command-line
 utility capable of installing packages with dependencies. That exists in
 default branch now; it's called pysetup. It doesn't have nearly all the
 features of easy_install, buildout, or pip, but it can install packages
 from an index with deps.


 In any case, it still doesn't change the part where it's a good idea to ship
 a static setup.cfg, with hooks only needing to run on the sdist-building
 machine, unless they are actually part of the build process.  There are use
 cases for calculated data to be in the initial setup.cfg, where the
 calculation machinery doesn't need to be on the target (like generating the
 file list or version from revision control info).  So, a setup_requires (or
 maybe better named build_requires) would still be helpful, but probably
 shouldn't be used for setup.cfg stability.

That's not a bad idea for certain kinds of metadata--version/vcs info
for example.  I like the idea of including a generated static
setup.cfg in a source dist as a solution to that kind of problem.  But
that doesn't eliminate the need for setup_hooks (or even more
complicated objects like custom commands) in an sdist.

For example, the majority of projects I work on require Numpy to build
one or two extension modules.  They require hooks to check that the
numpy package is importable, and then to use numpy's API to get the
paths to the numpy headers and and them to the include_dirs for each
extension module that requires them.  That's not the only one
though--one could have a whole suite of setup_hooks common to a bunch
of projects.  Custom Compiler classes are a possibility now too.

One could ship a copy of those dependencies with each project, or have
some kind of bootstrap script.  But to be able to automatically
download and add build dependencies to the path (a la setup_requires)
would be much nicer.  And packaging will have pysetup, so it should be
doable.  (Having the same capability for test dependencies and doc
dependencies would be nice too, but not nearly as important).

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command hooks...

[Erik Bray]

On Wed, May 30, 2012 at 12:19 PM, PJ Eby p...@telecommunity.com wrote:


 On Wed, May 30, 2012 at 11:54 AM, Erik Bray erik.m.b...@gmail.com wrote:

 On Tue, May 29, 2012 at 7:41 PM, PJ Eby p...@telecommunity.com wrote:
   I might be confused; I haven't been following the goings-on of late
   with
   distutils2.  At one point, I thought the plan was not to bless or
   include dependency-managing installers with the stdlib, or something
   like that.  i.e., I thought the plan wasn't to support or bless
   full-service tools like buildout, easy_install, or pip, or anything
   comparable to them.
 
  Right, yeah, the plans in this area were fluid for awhile, but the
  eventual conclusion was that the stdlib should have a command-line
  utility capable of installing packages with dependencies. That exists
  in
  default branch now; it's called pysetup. It doesn't have nearly all the
  features of easy_install, buildout, or pip, but it can install packages
  from an index with deps.
 
 
  In any case, it still doesn't change the part where it's a good idea to
  ship
  a static setup.cfg, with hooks only needing to run on the sdist-building
  machine, unless they are actually part of the build process.  There are
  use
  cases for calculated data to be in the initial setup.cfg, where the
  calculation machinery doesn't need to be on the target (like generating
  the
  file list or version from revision control info).  So, a setup_requires
  (or
  maybe better named build_requires) would still be helpful, but
  probably
  shouldn't be used for setup.cfg stability.

 That's not a bad idea for certain kinds of metadata--version/vcs info
 for example.  I like the idea of including a generated static
 setup.cfg in a source dist as a solution to that kind of problem.  But
 that doesn't eliminate the need for setup_hooks (or even more
 complicated objects like custom commands) in an sdist.

 For example, the majority of projects I work on require Numpy to build
 one or two extension modules.  They require hooks to check that the
 numpy package is importable, and then to use numpy's API to get the
 paths to the numpy headers and and them to the include_dirs for each
 extension module that requires them.  That's not the only one
 though--one could have a whole suite of setup_hooks common to a bunch
 of projects.  Custom Compiler classes are a possibility now too.

 One could ship a copy of those dependencies with each project, or have
 some kind of bootstrap script.  But to be able to automatically
 download and add build dependencies to the path (a la setup_requires)
 would be much nicer.  And packaging will have pysetup, so it should be
 doable.  (Having the same capability for test dependencies and doc
 dependencies would be nice too, but not nearly as important).


 Certainly.  I was just saying that the generated-metadata cases need
 handling, too, and that people should also be informed that they don't need
 (and shouldn't use) setup_requires for simple metadata generation, and that
 it might be a good idea to call the feature build_requires, and perhaps
 distinguish between setup hooks (for developers to have nice things) and
 build hooks (for stuff that absolutely has to run on the install machine).

I think we're on the same page then :)  packaging/d2 also supports
pre/post-command hooks which might, in most cases, be more appropriate
for the build hooks that absolutely have to run case.  In fact, I
had completely forgotten this, but my aforementioned Numpy extension
module hook is a pre-build_ext hook.  That makes it absolutely clear
that this is something we have to do before we can build an extension
module, and that it doesn't have any purpose outside that context and
shouldn't be executed ever time I run `pysetup something`.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distutils2: Does uploading to pypi work yet?

[Erik Bray]

On Sun, May 27, 2012 at 11:19 AM, Todd DeLuca todddel...@gmail.com wrote:
 I recently failed to upload a project to pypi using distutils2 and failed to
 install the project using pip.  Are distutils2 and pip not yet supporting
 'setup.cfg' only projects yet?

 The upload failed to upload the tarball to pypi because of what looks like a
 problem encoding a multipart form message with
 distutils2.util.encode_multipart(). I tried to fix that problem (using a
 base64 encoding) but ran into 500 http return code from the pypi server.
 The uploading code seems pretty 'alpha', so I was wondering if anyone is
 successfully using 'pysetup' to upload projects to pypi.

 After failing to upload the distribtion, I uploaded it to github and then
 registered my project with a github download url.  Then I tried installing
 my project using pip and that failed too, since I did not have a setup.py,
 only a setup.cfg, in the project.

 I'm using python 2.7.3, pip 1.1, and distutils2 from
 hg.python.org/distutils2 (pulled and updated) on Mac OS X 10.6.8.

I can't speak for PyPI or pip, but not a lot of stuff works with
setup.cfg-only distutils2/packaging projects.  Though I'm fairly
certain they shouldn't be expected to work yet.

Shameless plug, but have you tried d2to1
(http://pypi.python.org/pypi/d2to1)?  It allows you to use a setup.cfg
with existing tools like pip by way of a simple stub setup.py.  It
might not suit all your needs, but it also might :)

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] one project into several sub projects???

[Erik Bray]

On Tue, May 29, 2012 at 12:04 AM, Rob Healey robheal...@gmail.com wrote:
 Greetings:

 I would like to know what would be the best way to handle this situation???

 I have a project that needs to be split into several smaller pieces/
 projects:
 1) core
 2) gui
 3) cli
 4) web

 Could you give me the best case scenario to split up this one project?  It
 would be best if I used DU2 or Packaging...  I need something that will be
 supported for as while to come...

 Our project will be moving into Python3 compatibility...

Rob,

That's a big question with no simple answer, and might be better
addressed to comp.lang.python or the like.  How you do this isn't tied
to the packaging framework you use (thought it might be best to keep
the packaging aspect as simple as possible).

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command hooks...

[Erik Bray]

On Thu, May 24, 2012 at 8:27 PM, PJ Eby p...@telecommunity.com wrote:
 On Thu, May 24, 2012 at 6:56 PM, Daniel Holth dho...@gmail.com wrote:

  For something like Pyramid the main piece that's missing is support
  for an entry_points like system.  As Tarek and other have pointed out
  the past, such a system could still be supported through a third-party
  plugin system that works via setup hooks and custom metadata.  I think
  there was even a prototype for something like that at one point...

 'packaging' will be powerful enough to copy entry_points.txt into the
 .dist-info directory as a resource. Python bug #11880.

 The existing entry points code has a plugin mechanism for itself that
 will make it easy to iterate over entry_points.txt in dist-info as
 well as egg-info directories. Just write that plugin and you should be
 good to go. Though you might have to install the plugin in an egg-info
 directory at first...


 What plugin mechanism are you talking about here, specifically?

 For that matter, which existing entry points code are you talking about,
 too?  ;-)

Yeah, I'm a little confused by this too.  Sure you could include an
entry_points.txt in the dist-info. Though it's not clear to me that
there's an entirely straightforward way to do this yet.  I think some
ideas have been tossed about for how to make it easier to include
custom files in dist-info, but that there were no conclusions on that.
 Or maybe it already is easy and I'm just missing something.

Then there would need to be some interface like
pkg_resources.iter_entry_points() available to all software that
relies on entry_points.  This could be provided by the same plugin
that adds entry_points.txt to dist-info.  That plugin would have to be
an installation requirement for any software that relies on it (as is
setuptools for any package that currently uses entry_points, so I
don't think that's such a hardship...)

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command hooks...

[Erik Bray]

On Fri, May 25, 2012 at 2:24 PM, Daniel Holth dho...@gmail.com wrote:
 This discussion is confusing because we are either talking about a
 general plugin discovery mechanism for Python, or a plugin mechanism
 that packaging itself uses.

 After the patch, entry_points.txt (which you are encouraged to write
 yourself) is copied by means of a new {dist-info} category using the
 resources = mechanism in packaging. This is a straightforward way to
 copy anything into the .dist-info directory. Apart from copying the
 file, 'packaging' has no role.

 setup.cfg:

 [files]
 packages = mypackage
 resources =
    entry_points.txt={dist-info}

 That's all there is to it.

Neat! I didn't realize there was a {dist-info} resource category.
That, plus the register_finder() approach mentioned in the other
thread could be a way to go for interoperability with packages using
setuptools.  It would still require a bit of hackery though...  I
suspect there would be resistance to adding support for this directly
in packaging.  But a setup_hook could set this up.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command hooks...

[Erik Bray]

On Wed, May 23, 2012 at 11:59 PM, Chris McDonough chr...@plope.com wrote:
 It sounds like a reasonable task for me to try out.  In the meantime we've
 had a bit of a family emergency which will take some time to overcome and I
 won't be able to dedicate much energy to this.  As a result, I have to
 declare bankruptcy here, so I'll have to live with whatever I get.

Sorry to hear about that.

 I'm hoping though that someone else will step up here and do an evaluation
 and try to get things like Pyramid and popular Zope packages installed in a
 way that makes sense for straddling Python 2 and Python 3.  I suspect if no
 one does this, it's going to be rough going.


 - C

For something like Pyramid I don't actually think it would be too
difficult to get it *mostly* working with packaging/distutils2 in its
present state.  The word is mostly.  And that's to say nothing of
all its dependencies, though they don't all necessarily need to work
with the same installer.

For something like Pyramid the main piece that's missing is support
for an entry_points like system.  As Tarek and other have pointed out
the past, such a system could still be supported through a third-party
plugin system that works via setup hooks and custom metadata.  I think
there was even a prototype for something like that at one point...

But that raises another issue:  If packaging is going to be like a
framework that additional features like plugin systems or console
script generators can be tacked on to, there needs to be a way to
specify build dependencies in setup.cfg, and to have those
dependencies automatically downloaded and added to sys.path during
setup, a la setup_requires in setuptools.

I know there have been discussions and proposals in the past to add
something like this to packaging, and to amend PEP 345 to include it.
Metadata for test dependencies and docs dependencies have also been
discussed in the past.  But I think a way of adding build/setup
dependencies is critical if packaging is going to be useful as a
framework.

I have a few packages that use d2to1, which supports setup_requires
through setuptools.  Thanks to that, I'm able to make those packages
have an additional setup_requirement of a package that contains
several pre-canned setup_hooks that I use in all my projects.  I would
point out specific examples, but they're a bit obscure...

Anyways, I'm working right now to finish a software release.  But then
I want to spend some time working on this issue of adding build
requirements support to packaging and to PEP 345.  I might also work
on a plugin system that can work in packaging as a replacement for
entry_points, if no one else does.  Without these features I don't see
packaging being very successful as an installation framework, IMO.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command hooks...

[Erik Bray]

On Mon, May 21, 2012 at 4:42 PM, Tarek Ziadé ta...@ziade.org wrote:
 My understanding is that the legacy tools already work, for some vague
 definition of work.  Your task is to convince me that packaging is
 better -- which I think will be hard to do if packaging is not better.

 It really depends what you put behind the word better. And no, My task is
 not to convince you that packaging is better.

 My tasks were :

 1 - try to define standards which every packaging tool can implement, for
 the sake of interoperability, have a consensus on them, have them accepted
 2 - allow more static definitions of metadata
 3 - add in the standard library
  a/ a reference implementation for all the standards
  b/ a minimal installer.

 3-b is not ready for prime time obviously - but this should not take too
 long.

I don't usually do this, and I think most people on this list are
already aware of it, but this seems like a decent place to insert a
shameless plug for d2to1 [1].  d2to1 allows Python developers to take
advantage of many of the features and advantages of packaging
today--in particular static metadata and hook functions--while still
supporting existing installers like easy_install and pip.  It's a
total hack, and only meant to aid with transition, but it actually
works quite well in practice.

Unfortunately I haven't found the time to work on it in a little while
so I need to find that time.  In particular it would be useful for it
to add resources support.

Erik

[1] http://pypi.python.org/pypi/d2to1
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distutils2 Documentation...

[Erik Bray]

On Wed, Apr 11, 2012 at 1:44 AM, Rob Healey robheal...@gmail.com wrote:
 Greetings:

Hi there.

 From this web page,
 http://packages.python.org/Distutils2/distutils/setupscript.html,

 #!/usr/bin/env python

 from distutils2.core import setup, find_packages

 setup(name='Distutils2',
      version='1.0',
      summary='Python Distribution Utilities',
      keywords=['packaging', 'distutils2'],
      author=u'Tarek Ziadé',
      author_email='ta...@ziade.org',
      home_page='http://bitbucket.org/tarek/distutils2/wiki/Home',
      license='PSF',
      packages=find_packages())

 Can anyone tell me what I need to do to import setup using Distutils2???

You don't need or want (or have the ability) to do that.  there is no
setup() in distutils2.  I would refer you to the last time you asked
this question: 
http://mail.python.org/pipermail/distutils-sig/2012-March/018408.html
 Eric's answer stands: there is no setup.py.  Make a setup.cfg.  Read
the docs on docs.python.org/dev.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distutils2 1.0a4 and simple package

[Erik Bray]

On Wed, Mar 21, 2012 at 11:27 PM, Éric Araujo mer...@netwok.org wrote:
 Thanks for the pointer, is there anyway to use another filename other
 than setup.cfg?
 I am trying to build several related package from a common root?
 In the current code the filename must be setup.cfg.  I think the rise of
 distributed version control systems and usable packaging tools like pip
 and buildout (and setuptools’ namespace package feature) have encouraged
 the separation of projects into separate libraries, each one with its
 repository and setup.py file.  That may be the reason why setup.cfg is
 hard-coded in distutils2.

 What do other people think about this?  Is the ability to create more
 than one Python distribution from one directory something we really need?

This is a use case I've had twice in the past, if I understand
correctly at least.  In both cases a fair bit of evil hackery was
employed to make it possible in distutils (actually setuptools).  So
it can be done, but it gets ugly, and can cause some surprisingly
problems, most of which I can't even think of at the moment.  What I
can say is that the take away lesson of that experiment was to not do
that, and to just break everything into separately distributed
packages.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Installation

[Erik Bray]

Hi Vikash,

On Thu, Feb 9, 2012 at 1:02 AM, vikash agrawal
vikashagrawal1...@gmail.com wrote:
 Ok, I will try installing it tonight from tarek's, and will let you know.
 Also some time ago, I was trying to package it for Fedora 16 and I faced
 similar problem with that, pysetup not getting installed.

 Also @tarek, if possible Please update python-side for the package

 And why do I need latest cython for distutils? I would be using 2.X and not
 3.X

Distutils2 is included in Python 3 under the name 'packaging' which is
where, I believe, most of its development is focused.  The
distutils2 name is used for the backport of packaging to Python 2.x
as well as Python 3.x prior to the availability of packaging in the
standard library.

Confused yet?  There are some historical reasons for all that, but the
main point is that if you're interested in working on distutils2, you
really want to up and running on the main Python development branch
and work on *packaging*.  I think that most of the work is being done
there, and then backported to distutils2.  This post Tarek made a
while back should help you get up and running:
http://tarekziade.wordpress.com/2011/06/02/help-us-ironing-packaging/

Erik

 [...]

  Also, I had previously come in contact with merwok and I would be
  applying
  in GSoC this year under PSF for distutils2 project (as I believe PSF
  will be
  under GSoC and distutils will surely be accepted), so I am writing to
  seek
  help on what all things should be my target, and how do I move ahead.
  Currently, I am trying to work on the ]issue 12944 but due to certain
  constraints things are not geared up, though I hope they would soon be
  :)
  Please guide me and drop-in all your suggestions
 
  Regards
 
  Vikash Agrawal
 

 [...]

 Waiting for guidance


 Regards
 Vikash Agrawal


 --
 sent via HTC Sensation


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Installation

[Erik Bray]

On Wed, Feb 8, 2012 at 12:32 PM, vikash agrawal
vikashagrawal1...@gmail.com wrote:
 hi Everyone,

 I am using Python 2.7, so how do I install pysetup. though when I install
 distutils1.0a3 I do see distutils module in my interactive session but I
 dont see any pysetup being installed

That release is pretty old, and I'm not sure it even includes the
pysetup script.  If you're going to be doing d2 development you're
best off getting the latest source from hg:
https://bitbucket.org/tarek/distutils2

You should also make sure to get the latest cpython repo from
http://hg.python.org/cpython to work on packaging.  Though I don't
know exactly what the current policy is on merging from packaging to
distutils2 or vice-versa, so I'm not the best person to comment on
that.

Erik

 Also, I had previously come in contact with merwok and I would be applying
 in GSoC this year under PSF for distutils2 project (as I believe PSF will be
 under GSoC and distutils will surely be accepted), so I am writing to seek
 help on what all things should be my target, and how do I move ahead.
 Currently, I am trying to work on the ]issue 12944 but due to certain
 constraints things are not geared up, though I hope they would soon be :)
 Please guide me and drop-in all your suggestions

 Regards

 Vikash Agrawal

 --
 sent via HTC Sensation



 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distribute instead of setuptools for virtualenv?

[Erik Bray]

2012/1/16 Lennart Regebro rege...@gmail.com:
 2012/1/16 Christian Ştefănescu st.ch...@gmail.com:
 Also, if this works just fine, wouldn't it make sense to make Distribute the
 default for virtualenv at some point?

 In my opinion, yes.

FWIW you can also set VIRTUALENV_USE_DISTRIBUTE=1 in your environment.
 Or, in the latest version of virtualenv I think it's just
VIRTUALENV_DISTRIBUTE.  You can also set it in a config file now; I
forget the details but it's explained in the virtualenv docs.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig