Re: [Distutils] .egg-info metadata

2012-09-21 Thread Nick Coghlan 
On Fri, Sep 21, 2012 at 2:57 PM, PJ Eby p...@telecommunity.com wrote:
 This is no more guesswork than the PyPI /simple index discovery protocol is.

You have zero idea what's at the end of a URL link. You're just hoping
it's the file you expect.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread PJ Eby 
On Fri, Sep 21, 2012 at 3:06 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On Fri, Sep 21, 2012 at 2:57 PM, PJ Eby p...@telecommunity.com wrote:
 This is no more guesswork than the PyPI /simple index discovery protocol is.

 You have zero idea what's at the end of a URL link. You're just hoping
 it's the file you expect.

If that's really a problem, you can include an #md5 tag in the link,
and easy_install will refuse to install it if it doesn't match.

However, in a major common use case for dependency links, the person
who put the link in knows *exactly* what's at the end of that link,
because it's a URL they control with contents they produced.

I think, though, that maybe you're confused about how these links
work.  A dependency link has to be one of two types:

1. A direct link (a link that's recognizable as a distribution,
either because its final path component is a distribution filename or
because it's tagged with an #egg identifier), or

2. An indirect link, which will be treated as an HTML page and
scanned for direct links.

This is the *same* protocol used with the PyPI /simple index, or more
precisely, it's a *subset* of that protocol.  (Index pages are scanned
for indirect links tagged as a home page or download URL; this
extension is not supported for dependency links.)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Brett Cannon 
On Fri, Sep 21, 2012 at 1:28 AM, Donald Stufft donald.stu...@gmail.comwrote:

  On Friday, September 21, 2012 at 12:57 AM, PJ Eby wrote:

 As far as the practicality vs. purity question, Python has already had
 Provides/Requires in the metadata format for several years, and it
 contained all the features that were needed for a pure dependency
 resolution system. In *theory* you could meet everyone's use cases
 with that, if you're going to assume that all packages live in PyPI
 and everyone has a compiler. And then there wouldn't be any need for
 any of the new metadata specs.

 That field only allows importable names which many distributions do not
 have (e.g. you can't put in `django-tastypie` into requires).

 The exact wording is:
  The format of a requirement string is identical to that of a
   module or package name usable with the 'import' statement,
   optionally followed by a version declaration within parentheses.

 These fields were _not_ for saying that it required a particular
 distribution/project
 and _were_ for saying it requires a particular module or package (in the
 import sense).


 In practice, however, not everything's in PyPI and not everybody has
 compilers, and not every package is instantly updated or maintained
 indefinitely. If you don't like how dependency links solve the
 problem, perhaps you could propose some other way of handling those
 use cases, such that developer B's dependency maintenance burdens are
 not propagated to developer A and his peer/parent dependers?

 I don't see why pushing the maintenance burden down the stack isn't an ok
 thing
 to do. We already do it for External requirements that either aren't
 python packages
 or are packaged in a way that standard tools cannot handle them. Obviously
 you
 have to draw the line somewhere between packages that you can depend on
 automatically and ones you can't.

 My problems with dependency_links is less to do with guessing and more
 about
 the fact it ties dependency installation to a particular host. The fact
 that dependencies
 are abstract requirements (a distribution named foo at version 2.0) and
 not a concrete
 location (distribution foo version 2.0 found at http://example.com/dists/)
 means that
 you get to choose at install time where your requirements come from. So a
 user could
 easily host their own intranet PyPI style server that they installed their
 dependencies
 from. A dependency that comes from dependency_links can't be installed
 from your
 own internal PyPI (or another public one) without rewriting the setup.py
 scripts (Unless
 my understanding of dependency links is wrong).


I have tangentially following this discussion, but Donald's point about
keeping dependencies as abstract against a project and version and not
concrete as to where to grab it from seems very reasonable for metadata.
While PyPI might be considered a special-case as it's essentially the
canonical place to find packages and it is under our control (and thus a
place that we might provide a basic tool to pull from), past that I think
it's the purview of pip to decide how it wants to find places to resolve
things. This very much ties into the library vs. tool dividing line.



 Additionally by tying a dependency to an external system you decrease the
 overall
 installablity of a set of requirements* and you increase the number of
 locations that
 someone can use to exploit developers installing Python packages. These
 are important
 considerations when deciding to install or depend on a package and the
 dependency_links
 system hides it and makes it non obvious to end users what all is
 entailed in
 installing their set of requirements. (Note I have the same basic problem
 with external
 links on PyPI).

 I don't think it's entirely unreasonable for a packaging tool to have the
 ability to install
 from other locations that PyPI or a PyPI like server, I think that a
 packaging tool
 probably should have a dependency_links like system, however I think that
 the choice
 to use them should be in the hands of the person _installing_ the package,
 not the
 person creating the package.


Agreed. While I might have a suggestion to grab some project from some
place on e.g. bitbucket, that doesn't mean it is the best for someone else
(e.g. the NSA probably wants to pull certain projects form their own
super-secret, vetted project repo rather than from joe-schmoe.com). It
really shouldn't be a packager's job to figure out where to get something,
just that something is needed (e.g. I don't tell you where to go buy your
milk to make waffles, just that you need milk); it is a recipe and thus
should be kept abstract as such.

-Brett



 * If You depend on 5 different hosts for installing any particular set of
 requirements,
   and each of those 5 servers have a 99% uptime then the total combined
 uptime
   would be (0.99 * 0.99 * 0.99 * 0.99 * 0.99) = ~95%. In an average month
 you'd
   go from ~7 hours of uninstallable time per

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Daniel Holth 
Let's agree that there is/should be an --ignore flag for
dependency_links.txt and argue about whether it is on by default.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread PJ Eby 
On Fri, Sep 21, 2012 at 1:28 AM, Donald Stufft donald.stu...@gmail.com wrote:
 These fields were _not_ for saying that it required a particular
 distribution/project
 and _were_ for saying it requires a particular module or package (in the
 import sense).

Yes, but that was still sufficient information to implement a
dependency system, in theory.  Specifically, it would have worked for
the case where all projects are on PyPI and have correct metadata.

If you assume that condition, you can trivially solve all
dependencies; but if you don't assume that condition, you need
something like dependency links.


 I don't see why pushing the maintenance burden down the stack isn't an ok 
 thing to do.

For the same reason that requiring someone to read the source code of
every function called by every function called by every function
called by code they use isn't an ok thing to do.


 A dependency that comes from dependency_links can't be installed from
 your  own internal PyPI (or another public one) without rewriting the setup.py
 scripts (Unless my understanding of dependency links is wrong).

This is indeed wrong.  Dependency links are a last resort, used after
all other provided dependency resolution sources have failed.

I think I see where Nick is coming from now - when he said you don't
know what's at the end of the URL, my first reaction was WTF?, only
much more elaborate than WTF, because I couldn't conceive of how
that's even remotely a meaningful comment.

However, if he, like you, thought that a dependency link means
download and install this along with the package, then that's a
perfectly understandable further misconception.

That isn't how they work at all: they are simply links that
easy_install is *allowed* to use to resolve dependencies.  It doesn't
just download those URLs and install them.

If you depend on Foo and Bar, your standard requirements list *still
has to include Foo and Bar*!

The dependency links only say, here are some places to *look* for Foo
and Bar, *if* you can't find them through the normal channels (e.g.
PyPI, your local index server, already-installed files, links
specified on the command line, etc.).

In other words, they *supplement* the requirements metadata and index
system, they don't *subvert* it.


 I think that the choice to use them should be in the hands of the person
 _installing_ the package, not the person creating the package.

That's what --allow-hosts is for: easy_install can be locked down to
not install except from your local intranet or PyPI or whatever, and
it can be done on a sitewide basis by default (by settings in the
site's distutils.cfg).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread PJ Eby 
On Fri, Sep 21, 2012 at 11:04 AM, Brett Cannon br...@python.org wrote:
 Agreed. While I might have a suggestion to grab some project from some place
 on e.g. bitbucket,

This might've been buried in the thread, but that's *precisely* what
dependency links are: a *suggestion* as to where a *normally declared
dependency* can be filled.

Tools have the option to consume these suggestions, or not.  And
easy_install *only* uses these suggestions after all other available
options for finding the dependency have been exhausted.  (E.g. PyPI,
already-installed packages, links or indexes supplied on the command
line, etc.)

So there are two ways to block the use of dependency links:

1. Host restrictions (which can come from the command-line, personal
config, or site config)
2. Alternate indexes or command-line overrides for dependency links
(also configurable in the above-mentioned places) that satisfy all the
needed dependencies (so that easy_install doesn't bother with the
suggested links)

And you can easily combine the two, even if you're not the NSA.  ;-)

(You can also ask easy_install not to install dependencies at all, and
handle it all yourself.)

My point here, btw, isn't to advocate for the exact implementation of
dependency links; I just want to make sure everybody knows what they
*are*, because AFAICT there has been some huge misunderstanding and
FUD going on in this thread, and people making judgments based on
amplifying other people's misinterpretations of other people's
misunderstandings.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Donald Stufft 
On Friday, September 21, 2012 at 11:14 AM, PJ Eby wrote:
 Yes, but that was still sufficient information to implement a
 dependency system, in theory. Specifically, it would have worked for
 the case where all projects are on PyPI and have correct metadata.
 
 If you assume that condition, you can trivially solve all
 dependencies; but if you don't assume that condition, you need
 something like dependency links.
 
 

How would you take ``requires: tastypie`` and turn it into `django-tastypie`? 
Remebering
that PyPI enforces a unique distribution name, not a unique python package (in 
the import
sense) name. In tastypie's case there are 2 libraries that both provide a top 
level ``tastypie``
import, one is on PyPI as just ``tastypie`` and the other is 
``django-tastypie``.

That also sidesteps the issue that those fields were already in use and if they 
were
hijacked for this new purpose all that old data would (could?) have created 
issues. The
new meta-data sidestepped all these issues.
 For the same reason that requiring someone to read the source code of
 every function called by every function called by every function
 called by code they use isn't an ok thing to do.

As I said before, obviously you must draw the line somewhere. setuptools
choose to draw it at a different place than I believe it should have. I don't
believe saying you must be using some combination of indexes that
include all the dependencies or else you'll need to manually fetch them is
that large of a burden or requirement to make, obviously you disagree.
 This is indeed wrong. Dependency links are a last resort, used after
 all other provided dependency resolution sources have failed.

This makes them better than I thought, but still have the installability and
security. The question this raises in my head, can they be used for any
dependency, or only dependencies that the distribution defining them has?
e.g. if I install Foo, and it depends on Bar, and Wat, and Bar has a
dependency_links, will it only work for Bar and it's dependencies
or will it work for Wat's dependencies as well? 
 That's what --allow-hosts is for: easy_install can be locked down to
 not install except from your local intranet or PyPI or whatever, and
 it can be done on a sitewide basis by default (by settings in the
 site's distutils.cfg).

I'd have been less uneasy about them if they were opt in instead of 
opt out, but that'd require a time machine. I'm mostly trying to stress
why I'm against them to make sure we don't repeat (in my opinion)
the mistakes of the past.

I should probably mention here that I don't particularly blame setuptools
for their invention, I think it's a side effect of one tool owning the entire
packaging chain from top to bottom and not any fault with the implementer(s)
and because easy_install (to my knowledge) does not have anything like
a requirements file from pip so without that the obvious place for these
extra locations is a setup.py. I just think it's better to split them out and
keep the packaging metadata abstract, and them have the concrete meta-data
be inferred by taking that abstract data and combining it with a 
requirements.txt
like file.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread PJ Eby 
On Fri, Sep 21, 2012 at 12:59 PM, Donald Stufft donald.stu...@gmail.com wrote:
 How would you take ``requires: tastypie`` and turn it into
 `django-tastypie`?

By matching Requires with Provides at the index level.  That is, by
having PyPI index packages' Provides so that you can search for
packages that match a given Requires.

 As I said before, obviously you must draw the line somewhere. setuptools
 choose to draw it at a different place than I believe it should have. I
 don't
 believe saying you must be using some combination of indexes that
 include all the dependencies or else you'll need to manually fetch them is
 that large of a burden or requirement to make, obviously you disagree.

This wasn't added because I personally disagreed, it's the managers of
projects that use large dependency sets while also being relied upon
as part of other projects' large dependency sets.


 This makes them better than I thought, but still have the installability
 and
 security.

How is this any different than any other part of PyPI?

 The question this raises in my head, can they be used for any
 dependency, or only dependencies that the distribution defining them has?
 e.g. if I install Foo, and it depends on Bar, and Wat, and Bar has a
 dependency_links, will it only work for Bar and it's dependencies
 or will it work for Wat's dependencies as well?

I would have to dig into the code to be sure, actually.  But I don't
see any problem with restricting them to the dependency subtree being
resolved.  It's possible that the current easy_install implementation
leaks dependencies to sibling installs, but if it does, it's probably
not hard to fix.

 and because easy_install (to my knowledge) does not have anything like
 a requirements file from pip

I just googled requirements.txt, so now I understand why people are so
confused about dependency_links.  Dependency links are *not*
requirements.txt.  A pip requirements.txt is basically equivalent to a
set of easy_install command line options.

So, technically, you can do something  easy_install `cat
requirements.txt` and use a requirements.txt with it.  ;-)

Dependency links, OTOH, are only parameters to the easy_install
--find-links option: a *much* more limited functionality.  Not only is
it not the same functionality, the *use case* is quite different as
well.  Requirements.txt is for people installing packages, dependency
links are for people *providing* packages.  Shipping it in-band is the
whole point of the operation.


 so without that the obvious place for these
 extra locations is a setup.py.

Again: the entire point of dependency_links is to have an in-band,
automated way to ship third-party indexing information that would
otherwise require manual, out-of-band, *recursive* distribution.
Dependency links are download locations, and are independent of
version specifications.


 I just think it's better to split them out

They *are* split out - that's why they're not in requires.txt or PKG-INFO

 and keep the packaging metadata abstract, and them have the concrete meta-data

I agree that dependency links should not be part of the PKG-INFO
format.  They are indeed a concrete, practical matter for installation
tools.


 be inferred by taking that abstract data and combining it with a 
 requirements.txt like file.

That is basically how dependency links work now, *except* that they
are not as powerful as a requirements.txt file, and they are shipped
with the distribution.

As with entry points, though, there's no need for it to be in
PKG-INFO, since it is for the convenience of installation tools and
end users; it isn't really about the project itself.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Donald Stufft 
On Friday, September 21, 2012 at 3:09 PM, PJ Eby wrote:
 On Fri, Sep 21, 2012 at 12:59 PM, Donald Stufft donald.stu...@gmail.com 
 (mailto:donald.stu...@gmail.com) wrote:
  How would you take ``requires: tastypie`` and turn it into
  `django-tastypie`?
  
 
 
 By matching Requires with Provides at the index level. That is, by
 having PyPI index packages' Provides so that you can search for
 packages that match a given Requires.
 
 

And when you have 2 packages that both provides: tastypie? Which is a real
world occurence. I actually got my projects switched up, it's haystack and
django-haystack which have different names on PyPI (and entirely different 
functions)
but are both ``import haystack`` and thus would both be Provides: haystack.

https://crate.io/packages/haystack/
https://crate.io/packages/django-haystack/
  This makes them better than I thought, but still have the installability
  and
  security.
  
 
 
 How is this any different than any other part of PyPI?
I hate it there too and I've done a little bit to try and make it more obvious
to people when people rely on dependencies that aren't hosted on PyPI and
I plan to do more in the future. (I realize the historical significance of it, 
I just
don't think it should bind us to be stuck with it forever). But that's a 
separate
argument for a different day.
 I would have to dig into the code to be sure, actually. But I don't
 see any problem with restricting them to the dependency subtree being
 resolved. It's possible that the current easy_install implementation
 leaks dependencies to sibling installs, but if it does, it's probably
 not hard to fix.

I only asked this because if it works for siblings I could imagine a case
where someones install set worked, but then they remove a dependency
and suddenly unrelated installs stopped working. Not really pertinent to the
discussion at hand, just a possible bug (or surprising behavior) that popped
into my head. 
 Again: the entire point of dependency_links is to have an in-band,
 automated way to ship third-party indexing information that would
 otherwise require manual, out-of-band, *recursive* distribution.
 Dependency links are download locations, and are independent of
 version specifications.

Yea I get that, I think at this point it's just a differing of opinion about
wether that is a desirable trait for a system to have or not. Obviously
I think not, and you I believe think it is. Setuptools isn't (and shouldn't)
be removing them and as you said they don't belong in PKG-INFO so I
I think we are in agreement about that :)

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread PJ Eby 
On Fri, Sep 21, 2012 at 3:25 PM, Donald Stufft donald.stu...@gmail.com wrote:
 And when you have 2 packages that both provides: tastypie? Which is a real
 world occurence.

Hey, I didn't say it would work in practice, I said it would work in
*theory*.  ;-)  The point was, if you only have to have something work
in theory, you can make much simpler design choices.


 I think at this point it's just a differing of opinion about
 wether that is a desirable trait for a system to have or not. Obviously
 I think not, and you I believe think it is.

It's not a matter of opinion whether the trait is desirable: I added
it because people actually desired it, not because I personally have
an opinion about it.  (I'm certainly not a frequent user of the
feature, myself.)  And the use case exists, even if you personally
don't have that use case.

IOW, people have in fact had the use case and desired the feature, so
framing it as a matter of opinion is wrong as a matter of fact.  ;-)

Now, I am not arguing that any particular metadata scheme *must*
support that use case.  I am simply pointing out that the use case
exists and hand-waving it away as a non-existent requirement is
pointless.  If you want to investigate who's using it and decide
whether you are okay with telling those people using it to find
another way to meet the use case, that's fine by me.

However, bear in mind that potentially every download of a package
using dependency links represents a user who would have to manually
handle some dependency downloads, and that the burden propagates to
packages *depending* on that package, not just the direct package.

(I suspect you will also find that dependency links are more likely to
be used in projects that are frequently downloaded; the whole point is
to save a lot of people time hunting down dependencies that would
otherwise be difficult to find.)

Anyway, the point is that arguing whether the use case is desirable
from a purity perspective, or necessary in an ideal world, is
irrelevant to the practical question of whether there should be an
official (or semi-official) way to support the use case in practice.

IOW, in theory, the use case is unnecessary and undesirable.  In
practice, it's not a theoretical question.  ;-)

You can say We understand the use case, but no, we won't support it.
 I am only trying to point out that we think this use case doesn't or
shouldn't exist is not the same thing.

(And at the beginning, I was only trying to answer Daniel's questions
about the specs of existing .egg-info metadata.)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Daniel Holth 
FWIW bdist_wheel knows about dependency_links.txt. It deletes it if it
is only whitespace, otherwise it keeps it.

As of last May when I downloaded pypi,

16984 of the latest sdists

13028 have dependency_links.txt

The 211 in the postscript have a non-empty dependency_links.txt

These point at peak.telecomunity.com (pardon my dict):

{'HTConsole-0.2.tar.gz':
'http://peak.telecommunity.com/snapshots/\nhttp://prdownloads.sourceforge.net/cheetahtemplate/Cheetah-2.0rc6.tar.gz?download\n',
 'plone.checksum-0.1.tar.gz': 'http://peak.telecommunity.com/snapshots/\n',
 'tagger-0.3.1.tar.gz':
'http://www.openplans.org/projects/yucca/install-tagger\nhttp://svn.rdflib.net/branches/michel-events#egg=rdflib-2.3.1events\nhttp://peak.telecommunity.com/snapshots/\n',
 'tw.starrating-1.0.2.tar.gz': 'http://peak.telecommunity.com/snapshots/\n'}

-- Daniel

p.s.

['DataPipeline-1.0.tar.gz',
 'DjangoSkel-0.1.3.tar.gz',
 'Dozer-0.1.tar.gz',
 'Flask-Security-1.2.2.tar.gz',
 'Flask-Social-1.0.0.tar.gz',
 'HTConsole-0.2.tar.gz',
 'Ibid-0.1.1.tar.gz',
 'MathBench-1.1.0.tar.gz',
 'Mtrax-2.2.07.zip',
 'OpenGLContext-full-2.1.0a9.tar.gz',
 'OpencoreRedirect-0.5.1.tar.gz',
 'Orange-Bioinformatics-2.5a4.tar.gz',
 'Pylons-1.0.1rc1.tar.gz',
 'SatchmoBeanstream-0.2.tar.gz',
 'Schevo-3.0.zip',
 'SchevoGears-1.0a1.zip',
 'SchevoPolicy-1.0a1.zip',
 'SchevoSql-1.0a1.zip',
 'SchevoWeb-1.0a2.zip',
 'Shabti-0.4.4.tar.gz',
 'Sunflower-1.1.0.tar.gz',
 'TaskTracker-0.4.tar.gz',
 'TurboGears-1.5.1.tar.gz',
 'TurboGears2-2.1.5.tar.gz',
 'Twirlip-0.6.tar.gz',
 'Twitabit-1.0a1.zip',
 'ZSI-2.0-rc3.tar.gz',
 'aipy-1.1.1.tar.gz',
 'airi-1.0.c11.tar.gz',
 'android2po-1.2.0.tar.gz',
 'asilib-1.0.2.tar.gz',
 'bda.bfg.tile-1.0.tar.gz',
 'bitsyauth-0.1.2.tar.gz',
 'bitsyblog-2.4.1.tar.gz',
 'boprox_client-0.2.1.tar.gz',
 'boprox_server-0.3.0.tar.gz',
 'buildout.python_libevent-0.9.1.tar.gz',
 'causal-1.0.21.tar.gz',
 'cc.licenserdf-0.2.29.tar.gz',
 'cdat_lite-5.2rc1.tar.gz',
 'chimera-python-0.1.zip',
 'cocograph-0.1.0.tar.gz',
 'cocos2d-0.5.0.zip',
 'collective.recipe.seleniumrc-0.6.1.1.tar.gz',
 'compoze-0.4.1.1.tar.gz',
 'configglue-1.0.3.tar.gz',
 'daversy-0.1.0.dev-r330.tar.gz',
 'dbsh-0.1.6.tar.gz',
 'decoupage-0.10.3.tar.gz',
 'dexy-0.5.7.tar.gz',
 'django-apps-0.1.0.tar.gz',
 'django-chronograph-0.3.1.tar.gz',
 'django-ckeditor-3.6.2.tar.gz',
 'django-dynamic-rules-0.3.2.tar.gz',
 'django-dynamic-validation-0.2.5.tar.gz',
 'django-dzenlog-0.3.1.tar.gz',
 'django-faces-0.2.1.tar.gz',
 'django-fiber-0.9.6.6.tar.gz',
 'django-lfs-0.7.4.tar.gz',
 'django-markdown2-0.1.4.tar.gz',
 'django-modeldict-1.3.1.tar.gz',
 'django-mutant-0.0.2.tar.gz',
 'django-profile-0.0.3.tar.gz',
 'django-pushserver-0.1.8.tar.gz',
 'django-star-0.999.tar.gz',
 'django-stories-0.6.6.tar.gz',
 'django-twitter-tag-0.4.0.tar.gz',
 'django-userprofile-0.0.5.tar.gz',
 'dzenlog-link-0.1.2.tar.gz',
 'ebpub-1.2.tar.gz',
 'elaphe-0.5.3.tar.gz',
 'evogrid-0.1.0.tar.gz',
 'externalator-0.7.0.tar.gz',
 'eyefi-0.2.tar.gz',
 'fassembler-0.6.tar.gz',
 'fetcher-0.9.2.6.zip',
 'finch-cms-0.8.3.tar.gz',
 'fiveruns.dash.django-0.2.1.tar.gz',
 'flunc-0.8.tar.gz',
 'getpaid.report-0.1.1.tar.gz',
 'getpaid.wizard-0.4.zip',
 'gocept.zope3instance-2.0a2.tar.gz',
 'googl-0.1dev.tar.gz',
 'graphite-0.71.tar.gz',
 'graphtool-0.5.2.tar.gz',
 'gtwpy-1.3.9.tar.gz',
 'hapicake-0.1.1.tar.gz',
 'hiringpond-0.1dev.tar.gz',
 'huImages-2.2.tar.gz',
 'huSoftM-0.65p2.tar.gz',
 'hyde-0.8.4.tar.gz',
 'ice.adverlet-0.2.3.tar.gz',
 'ice.template-0.1.0.tar.gz',
 'isk-daemon-0.9.3.zip',
 'isotoma.zope.testpythonscript-1.0.0.tar.gz',
 'iw.subscriber-0.1.1.tar.gz',
 'jmbo-0.2.5.tar.gz',
 'jmbo-social-0.0.6.tar.gz',
 'jquery.jsonform-0.1.0c1.tar.gz',
 'jquery.widget-0.1.0c1.tar.gz',
 'kegbot-0.8.5.tar.gz',
 'korovic-1.0.5.tar.gz',
 'kpwrapper-1.0.4.tar.gz',
 'ks.smartimage-1.2.2.tar.gz',
 'kytten-6.0.0.tar.gz',
 'labjacku12-0.4.tar.gz',
 'lfs-contact-1.0.tar.gz',
 'lfs-order-numbers-1.0b1.tar.gz',
 'll-orasql-1.27.1.zip',
 'll-toxic-0.10.zip',
 'll-xist-3.25.zip',
 'localwiki-0.3.1.tar.gz',
 'logfollow-server-0.0.9.tar.gz',
 'lojbansuggest-0.2.2.tar.gz',
 'massmedia-0.6.4.tar.gz',
 'matter-0.8-r960-20110308.tar.gz',
 'mdemos.server-0.1.3dev.tar.gz',
 'mobi.devices-1.2.tar.gz',
 'moxie-8.12.1.tar.gz',
 'nagare-0.4.1.tar.gz',
 'nagare.examples-0.3.0.tar.gz',
 'ndg_xacml-0.5.0.tar.gz',
 'neo4django-0.1.5.tar.gz',
 'numericube-twistranet-1.1.4.zip',
 'nydus-0.9.0.tar.gz',
 'opencore-0.9a4.tar.gz',
 'opencore-listen-0.8.0.tar.gz',
 'ore.viewlet-0.2.1.tar.gz',
 'panya-0.1.6.tar.gz',
 'panya-calendar-0.0.2.tar.gz',
 'panya-contact-0.0.6.tar.gz',
 'panya-social-0.0.4.tar.gz',
 'panya-tube-0.0.3.tar.gz',
 'pbp.recipe.trac-0.4.3.tar.gz',
 'pdfserver-0.6.tar.gz',
 'peer-0.10.0.tar.gz',
 'pg_boss-1.0dev2.tar.gz',
 'pg_foundation-1.1.1.tar.gz',
 'pg_greentrunk-1.0.1.tar.gz',
 'pg_pqueue-1.0.2.tar.gz',
 'pg_proboscis-1.0.5.tar.gz',
 'pg_tin-1.0dev.tar.gz',
 'pg_typical-1.2.tar.gz',
 'philo-0.9.2.tar.gz',
 'plone.checksum-0.1.tar.gz',

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Erik Bray 
On Thu, Sep 20, 2012 at 4:24 PM, Daniel Holth dho...@gmail.com wrote:
 And they answer most of your questions.  A few call-outs below:

 dependency_links.txt: url's of the package's dependencies. Speak up if
 you use this; it is very surprising, and has a much better replacement
 with pip --index options and requirements files.

 I will check to see how often this is used.

I definitely use this. This way all my packages can get their
dependencies from my own package index instead of PyPI. This gives a
greater deal of control in dealing with users who are angry/frustrated
when they can't reach PyPI for some reason. Though I think this will
be less of an issue with PyPI mirror support.  Though it's also nice
to be able to release development packages that have dev versions of
other packages as dependencies without having to put all my dev
releases on PyPI.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Erik Bray 
On Fri, Sep 21, 2012 at 12:57 AM, PJ Eby p...@telecommunity.com wrote:
 I'm not seeing how a documented file discovery protocol is
 guesswork.  Perhaps you've not read the documentation?  e.g.:

 http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-in-pypi
 http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall

Exactly.  This is the opposite of guesswork, in fact.  I have use
cases where I want to know/specify exactly where my dependencies are
coming from, and not just put out an abstract name.  I agree this
information shouldn't be a part of the package's Metadata.  I think
the existing solution in setuptools of including it in a
dependency_links.txt file is fine.  This wouldn't be an option though
with Nick's one JSON file to rule them all proposal (which I'm not
against--it just precludes keeping dependency-links-like info out of
band).

 As far as the practicality vs. purity question, Python has already had
 Provides/Requires in the metadata format for several years, and it
 contained all the features that were needed for a pure dependency
 resolution system.  In *theory* you could meet everyone's use cases
 with that, if you're going to assume that all packages live in PyPI
 and everyone has a compiler.  And then there wouldn't be any need for
 any of the new metadata specs.

 In practice, however, not everything's in PyPI and not everybody has
 compilers, and not every package is instantly updated or maintained
 indefinitely.  If you don't like how dependency links solve the
 problem, perhaps you could propose some other way of handling those
 use cases, such that developer B's dependency maintenance burdens are
 not propagated to developer A and his peer/parent dependers?

And not everybody wants to put everything on PyPI. In fact, I once had
a use case to build a system on top of pkg_resources that could locate
dependencies in SVN repository checkouts by adding their paths to
dependency_links/find_links.  There are all kinds of uses for this
kind of thing that I don't think should be ignored, and pkg_resources
has already provided the means to implement most of them (the
aforementioned example also required subclassing easy_install to work
properly, but that wasn't too big a deal).

I think a broader point is that package distribution and dependency
resolution has some ugly and messy corners and there's no way around
that.  Just saying, Well all the dependencies should be clearly
documented and it's up to the user to figure how to assemble them all
isn't going to fly either.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-21 Thread Erik Bray 
On Fri, Sep 21, 2012 at 3:06 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On Fri, Sep 21, 2012 at 2:57 PM, PJ Eby p...@telecommunity.com wrote:
 This is no more guesswork than the PyPI /simple index discovery protocol is.

 You have zero idea what's at the end of a URL link. You're just hoping
 it's the file you expect.

That's not really something the average science user is going to care
about.  All they care is that when they run `setup.py install` like I
told them to it installs the package and gets all the right versions
of the dependencies, some of which may not be on PyPI, or have been
repackaged for some reason because the official package that is on
PyPI was created by someone who doesn't know what they're doing and
isn't inclined to fix it.

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread PJ Eby 
On Sun, Sep 16, 2012 at 9:40 PM, Daniel Holth dho...@gmail.com wrote:
 I've been chatting to Chris McDonough a bit as well, and one
 potentially useful thing would be to clearly document the
 setuptools/distribute metadata precisely as it is generated today.

The official docs for all egg formats can be found at:

  http://peak.telecommunity.com/DevCenter/EggFormats

And they answer most of your questions.  A few call-outs below:

 dependency_links.txt: url's of the package's dependencies. Speak up if
 you use this; it is very surprising, and has a much better replacement
 with pip --index options and requirements files.

No, actually, those are not replacements at all, let alone better
ones.  They're for a package's supplier to offer third-party builds to
fulfill requirements not available via PyPI.  E.g. the author of
package Foo offers unofficial Windows builds of package Bar, and
includes dependency links so that those dependencies can be fulfilled
even if not present on PyPI.

 Provides works the same way in setuptools, it is in PKG-INFO.

Setuptools doesn't actually use Provides.

Lexical formats of the files are described in terms of pkg_resources
API calls -- these calls (and more precise syntax documentation) are
documented here:

  http://peak.telecommunity.com/DevCenter/PkgResources

Mainly in the Parsing Utilities section:

  http://peak.telecommunity.com/DevCenter/PkgResources#parsing-utilities
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread Daniel Holth 
 And they answer most of your questions.  A few call-outs below:

 dependency_links.txt: url's of the package's dependencies. Speak up if
 you use this; it is very surprising, and has a much better replacement
 with pip --index options and requirements files.

I will check to see how often this is used.

 No, actually, those are not replacements at all, let alone better
 ones.  They're for a package's supplier to offer third-party builds to
 fulfill requirements not available via PyPI.  E.g. the author of
 package Foo offers unofficial Windows builds of package Bar, and
 includes dependency links so that those dependencies can be fulfilled
 even if not present on PyPI.

It's probably a style thing, but I like to distribute this information
out-of-band. So as the author of foopackage (an application) I list
all the requirements and their requirements and where they can be
found, including git+http://github urls and so forth. I'm sure this
has all been discussed to death over the last 6+ years.

 Provides works the same way in setuptools, it is in PKG-INFO.

 Setuptools doesn't actually use Provides.

I noticed that. An installer could generate a file
provided-name.egg-info based on this information. Ironically
distribute would probably be the main client of this feature.

 Lexical formats of the files are described in terms of pkg_resources
 API calls -- these calls (and more precise syntax documentation) are
 documented here:

   http://peak.telecommunity.com/DevCenter/PkgResources

 Mainly in the Parsing Utilities section:

   http://peak.telecommunity.com/DevCenter/PkgResources#parsing-utilities

I am impressed by pkg_resources.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread Donald Stufft 
On Thursday, September 20, 2012 at 4:24 PM, Daniel Holth wrote:
 It's probably a style thing, but I like to distribute this information
 out-of-band. So as the author of foopackage (an application) I list
 all the requirements and their requirements and where they can be
 found, including git+http://github urls and so forth. I'm sure this
 has all been discussed to death over the last 6+ years.

For what it's worth I agree with you here. I think that setup.(py|cfg) should 
point towards abstract requirements. (I need a distribution named foo) not
concrete requirements (I need the distribution named foo from example.com).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread PJ Eby 
On Thu, Sep 20, 2012 at 4:53 PM, Donald Stufft donald.stu...@gmail.com wrote:
 On Thursday, September 20, 2012 at 4:24 PM, Daniel Holth wrote:

 It's probably a style thing, but I like to distribute this information
 out-of-band. So as the author of foopackage (an application) I list
 all the requirements and their requirements and where they can be
 found, including git+http://github urls and so forth. I'm sure this
 has all been discussed to death over the last 6+ years.

 For what it's worth I agree with you here.

As a practical matter (and practicality beats purity), it's sometimes
necessary to provide this information in order to depend on packages
that aren't cataloged in PyPI, or when third-party build support is
required, and your goal is to offer a library or application which
itself is used as a dependency.

That is, if project A depends on project B, which depends on C, which
is not on PyPI, then A cannot simply depend on B without *copying*
this out of band information, propagating a maintenance burden.
Dependency links work around this problem (among others).

So, IMO, this is actually a crucial part of making a composable
dependency system in *practice*, even if it is not required in
*theory*.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread Donald Stufft 
On Thursday, September 20, 2012 at 9:24 PM, PJ Eby wrote:
 As a practical matter (and practicality beats purity), it's sometimes
 necessary to provide this information in order to depend on packages
 that aren't cataloged in PyPI, or when third-party build support is
 required, and your goal is to offer a library or application which
 itself is used as a dependency.

This is likely the cause of the difference in opinion. I don't think that 
packaging metadata should contain information other than informative
for dependencies external to the ecosystem.

In this case If something isn't on PyPI then you don't get to have it
automatically installed. If I depend on lxml (or somewhere in my
dependency chain something does) then I have to convey that
requirements for libxml2 and libxslt. I don't see things external to PyPI
as any different. In metadata 1.2+ this could be expressed via
requires-external. It's a dependency external to that of the Python
packaging ecosystem.

Maybe that's just me though.___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread Nick Coghlan 
On Fri, Sep 21, 2012 at 12:08 PM, Donald Stufft donald.stu...@gmail.com wrote:
 Maybe that's just me though.

Nope, it's In the face of ambiguity, refuse the temptation to guess
applied to dependency metadata. It's one of the core philosophical
differences between the stdlib and systems like setuptools.

If *others* want to guess in their automated systems, that's their
prerogative. We prefer to put up a barrier and say be explicit about
what you want. You see the same difference in perspective between
datetime and dateutil - even though we agree there are contexts where
guessing is useful, we will generally consider such tools a poor fit
for the standard library, instead leaving them to the third party
ecosystem.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread PJ Eby 
On Thu, Sep 20, 2012 at 10:51 PM, Nick Coghlan ncogh...@gmail.com wrote:
 On Fri, Sep 21, 2012 at 12:08 PM, Donald Stufft donald.stu...@gmail.com 
 wrote:
 Maybe that's just me though.

 Nope, it's In the face of ambiguity, refuse the temptation to guess
 applied to dependency metadata.

How are dependency links guessing anything?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread PJ Eby 
On Fri, Sep 21, 2012 at 12:24 AM, Nick Coghlan ncogh...@gmail.com wrote:
 On Fri, Sep 21, 2012 at 1:58 PM, PJ Eby p...@telecommunity.com wrote:
 On Thu, Sep 20, 2012 at 10:51 PM, Nick Coghlan ncogh...@gmail.com wrote:
 On Fri, Sep 21, 2012 at 12:08 PM, Donald Stufft donald.stu...@gmail.com 
 wrote:
 Maybe that's just me though.

 Nope, it's In the face of ambiguity, refuse the temptation to guess
 applied to dependency metadata.

 How are dependency links guessing anything?

 What are they a link to? What OSes will it run on? How do you install
 them? As you soon as you step outside the defined dependency
 framework, it's all guesswork.

I'm not seeing how a documented file discovery protocol is
guesswork.  Perhaps you've not read the documentation?  e.g.:

http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-in-pypi
http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall

This is no more guesswork than the PyPI /simple index discovery protocol is.

The only part I can think of in the whole process that is actually
guesswork is interpreting distutils-generated (as opposed to
setuptools-generated) distribution filenames -- a necessary nod to
backward-compatibility.

As far as the practicality vs. purity question, Python has already had
Provides/Requires in the metadata format for several years, and it
contained all the features that were needed for a pure dependency
resolution system.  In *theory* you could meet everyone's use cases
with that, if you're going to assume that all packages live in PyPI
and everyone has a compiler.  And then there wouldn't be any need for
any of the new metadata specs.

In practice, however, not everything's in PyPI and not everybody has
compilers, and not every package is instantly updated or maintained
indefinitely.  If you don't like how dependency links solve the
problem, perhaps you could propose some other way of handling those
use cases, such that developer B's dependency maintenance burdens are
not propagated to developer A and his peer/parent dependers?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-20 Thread Donald Stufft 
On Friday, September 21, 2012 at 12:57 AM, PJ Eby wrote:
 As far as the practicality vs. purity question, Python has already had
 Provides/Requires in the metadata format for several years, and it
 contained all the features that were needed for a pure dependency
 resolution system. In *theory* you could meet everyone's use cases
 with that, if you're going to assume that all packages live in PyPI
 and everyone has a compiler. And then there wouldn't be any need for
 any of the new metadata specs.

That field only allows importable names which many distributions do not
have (e.g. you can't put in `django-tastypie` into requires).

The exact wording is:
 The format of a requirement string is identical to that of a
  module or package name usable with the 'import' statement,
  optionally followed by a version declaration within parentheses.


These fields were _not_ for saying that it required a particular 
distribution/project
and _were_ for saying it requires a particular module or package (in the import 
sense). 
 
 In practice, however, not everything's in PyPI and not everybody has
 compilers, and not every package is instantly updated or maintained
 indefinitely. If you don't like how dependency links solve the
 problem, perhaps you could propose some other way of handling those
 use cases, such that developer B's dependency maintenance burdens are
 not propagated to developer A and his peer/parent dependers?

I don't see why pushing the maintenance burden down the stack isn't an ok thing
to do. We already do it for External requirements that either aren't python 
packages
or are packaged in a way that standard tools cannot handle them. Obviously you
have to draw the line somewhere between packages that you can depend on
automatically and ones you can't.

My problems with dependency_links is less to do with guessing and more about
the fact it ties dependency installation to a particular host. The fact that 
dependencies
are abstract requirements (a distribution named foo at version 2.0) and not a 
concrete
location (distribution foo version 2.0 found at http://example.com/dists/) 
means that
you get to choose at install time where your requirements come from. So a user 
could
easily host their own intranet PyPI style server that they installed their 
dependencies
from. A dependency that comes from dependency_links can't be installed from your
own internal PyPI (or another public one) without rewriting the setup.py 
scripts (Unless
my understanding of dependency links is wrong).

Additionally by tying a dependency to an external system you decrease the 
overall
installablity of a set of requirements* and you increase the number of 
locations that
someone can use to exploit developers installing Python packages. These are 
important
considerations when deciding to install or depend on a package and the 
dependency_links
system hides it and makes it non obvious to end users what all is entailed in
installing their set of requirements. (Note I have the same basic problem with 
external
links on PyPI).

I don't think it's entirely unreasonable for a packaging tool to have the 
ability to install
from other locations that PyPI or a PyPI like server, I think that a packaging 
tool
probably should have a dependency_links like system, however I think that the 
choice
to use them should be in the hands of the person _installing_ the package, not 
the
person creating the package.

* If You depend on 5 different hosts for installing any particular set of 
requirements,
  and each of those 5 servers have a 99% uptime then the total combined uptime
  would be (0.99 * 0.99 * 0.99 * 0.99 * 0.99) = ~95%. In an average month you'd 
  go from ~7 hours of uninstallable time per month (for one server at 99% 
uptime)
  to over a day of uninstallable time per month (for 5 servers at 99% uptime 
each).
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] .egg-info metadata

2012-09-16 Thread Daniel Holth 
 I've been chatting to Chris McDonough a bit as well, and one
 potentially useful thing would be to clearly document the
 setuptools/distribute metadata precisely as it is generated today.
 Currently these formats are entirely implicit in the implementation of
 the code that reads and writes them, as far as I can tell anyway. The
 distribute docs seem to do a decent job of explaining setup.py and the
 various setuptools specific arguments, but *not* what the file formats
 will look like inside the metadata directory when installed.

 The main advantages of this would be to make it clear:
 1. What can setuptools metadata describe, that v1.2 of the official
 metadata standard cannot?
 2. Does v1.3 allow everything that setuptools can currently describe
 (either directly, or as an extension)?
 3. Does v1.3 allow some things to be expressed more clearly than they
 can be with setuptools?

The big new feature with Metadata 1.2 and up that has no
representation in setuptools is the environment markers specification,
which for common cases like 'require x when Python version is y'
allows the generated requirements file to be the same no matter which
version of Python was used to produce the dist:

Requires-Dist: argparse; python_version  '2.7'

would be a common example. The only thing I don't like about it is
that I cannot remember whether to use _ or . to separate the words.
Metadata 1.3 restores extras from setuptools by adding 'extra' as a
variable in environment markers.

In setuptools:

requires.txt:
normal
requirements
not-in-a-section

[extra_name]
extra_requirement

Nothing prevents an installer from generating requires.txt from
Metadata 1.3, but you can't go the other way due to the environment
markers.


entry_points.txt is the plugin mechanism. I am leaving this unchanged
in my implementation; I'm not motivated to put them into PKG-INFO and
there's no benefit. Each section is a kind of entry point, and each
entry is

key = module.name:callable

Split on the colon :, import the first part, return the second part.
console_scripts is the most widely used by far, but thousands of
packages provide other kinds of entry points. I wish pypi was smart
enough to index this file.

entry_points.txt:
[console_scripts]
wininst2wheel = wheel.wininst2wheel:main
egg2wheel = wheel.egg2wheel:main
wheel = wheel.__main__:main

[distutils.commands]
bdist_wheel = wheel.bdist_wheel:bdist_wheel


top_level.txt is just a list of top-level packages defined by the
dist, one per line. Is this still used?

namespace_packages.txt - same format as top_level.txt. Is used.

SOURCES.txt - list of source files, one per line. Not used.

not-zip-safe: present if package should not be zipped up. empty.

dependency_links.txt: url's of the package's dependencies. Speak up if
you use this; it is very surprising, and has a much better replacement
with pip --index options and requirements files.


Provides works the same way in setuptools, it is in PKG-INFO.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig