Re: [Distutils] URL Structure of Packages URLs
On Tue, Oct 08, 2013 at 05:05:54PM +0300, Marius Gedminas wrote: > On Tue, Oct 08, 2013 at 09:19:37AM -0400, Donald Stufft wrote: > > I was wondering if anyone was aware of anything that relies on > > the current structure of package urls, namely: > > > > /packages/source|any|etc/D/Django-1.0.tar.gz? > > > > I would like to change this but before I work up a concrete plan for > > people to comment on and discuss I'm trying to figure out what, if > > anything, depends on that current structure. > > Some Debian packages have debian/watch files using similar URL patterns Perhaps "Some" is an understatement ;) http://qa.debian.org/developer.php?login=python-modules-t...@lists.alioth.debian.org lists over 500 packages, almost all of which have watch files (last 2 columns). There are probably quite a few more Debian packages with watch files not maintained by the Python Modules Team. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Buildout 2.2.0b1 is available fort testing
I've been using this the last few days under python 2.7 and virtualenv 1.9.1. No problems whatsoever, apart from having to upgrade setuptools before running the buildout's bootstrap. On Mon, Jun 17, 2013 at 08:44:30AM -0400, Jim Fulton wrote: > Changes: > > - Switch dependency from ``distribute 0.6.x`` to ``setuptools 0.7.x``. > > - Handle both addition and subtraction of elements (+= and -=) on the same key > in the same section. Forward-ported from buildout 1.6. > > - Suppress the useless ``Link to ***BLOCKED*** by --allow-hosts`` > error message being emitted by distribute / setuptools. > > - Extend distutils script generation to support module docstrings and > __future__ imports. > > - Refactored picked versions logic to make it easier to use for plugins. > > - Use ``get_win_launcher`` API to find Windows launcher (falling back to > ``resource_string`` for ``cli.exe``). > > - Remove ``data_files`` from ``setup.py``: it was installing ``README.txt`` > in current directory during installation (merged from 1.x branch). > > To try it out, copy the new bootstrap file from: > > http://downloads.buildout.org/2.2.0b1/bootstrap.py > > Then run the bootstrap, making sure to include the -t option: > > python bootstrap.py -t > > The -t option is necessary to use the non-final buildout release. > > We'd like to release 2.2 final soon, so your testing is really > helpful. > > Special thanks to Tres Seaver for updating buildout to work with > setuptools 0.7 and for reviewing and merging a number of outstanding > pull requests. Of course, thanks to the folks to contributed the > many other changes in this release. > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton > _______ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI Download Counts
On Sun, May 26, 2013 at 08:08:35PM -0400, Donald Stufft wrote: > Hello! > > As you have have noticed the download counts on PyPI are no longer updating. > Originally this was due to an issue with the script that processes these > download counts. However I have now removed the download counts from the PyPI > webui and their use via the API is considered deprecated. > > There are numerous reasons for their removal/deprecation some of which are: > - Technically hard to make work with the new CDN > - The CDN is being donated to the PSF, and the donated tier does not > offer any form of log access > - The work around for not having log access would greatly reduce the > utility of the CDN > - Highly inaccurate > - A number of things prevent the download counts from being > inaccurate, some of which include: > - pip download cache > - Internal or unofficial mirrors > - Packages not hosted on PyPI (for comparisons sake) > - Mirrors or unofficial grab scripts causing inflated counts > (Last I looked 25% of the downloads were from a known mirroring script). > - Not particularly useful > - Just because a project has been downloaded a lot doesn't mean it's > good > - Similarly just because a project hasn't been downloaded a lot > doesn't mean it's bad > > In short because it's value is low for various reasons, and the tradeoffs > required to make it work are high It has been not an effective use of > resources. > > The API will continue to return values for it in order to not break scripts, > however in the future all these values will be set to 0. The Web UI has been > modified to no longer display it. There are other ways to get usage statistics. Debian has Popularity Contest (http://popcon.debian.org) statistics for python libraries. e.g. babel and twisted: http://qa.debian.org/popcon.php?package=python-babel http://qa.debian.org/popcon.php?package=twisted Of course, the these are probably still "Highly inaccurate", just in a different way;) -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding Provides-Extra as an edit to PEP 345
On Fri, Jul 06, 2012 at 11:01:33AM -0400, Éric Araujo wrote: > Hi, > > >Hopefully this will be a more useful spec. > Thanks for the proposal, but it is not really a spec. You’re > suggesting name and syntax for a new field, but you don’t describe > what it means and what it is used for. From what I understand, the > goal is similar to build dependencies and optional dependencies in > systems like Debian’s dpkg; You can also think of extras as similar to Debian's metapackages. For example, looking at the zope.component packaging: http://packages.debian.org/source/sid/zope.component The zope.component [zcml] extra is translated to metapackage (python-zope.component-zcml) that depends on python-zope.component and python-zope.configuration. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout fails to use system-installed dep?
On Fri, Dec 10, 2010 at 08:19:32AM -0500, Barry Warsaw wrote:
> On Dec 10, 2010, at 01:42 PM, Brian Sutherland wrote:
>
> >On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote:
>
> >> The way zope.interface "owns" the zope namespace is not correct. Ideally,
> >> there would be a separate binary package that owns zope/__init__.py and on
> >> which all other zope subpackages depends. This could of course be built
> >> from the zope.interface source package.
> >
> >I originally had it that way, but was strongly advised to change it to
> >the current method to be able to pass the Debian FTP Master gauntlet.
>
> Just goes to illustrate the myth of TOOWTDI. :) AFAICT, from my discussions
> with various folks on debian-python, it should now[*] be done with the
> separate binary package.
>
> [*] At least until dh_python2, if we can ensure it always DTRT.
I'll just till dh_python2 then before changing anything ;)
hmm, perhaps I should join debina-python again...
> >> Second, we're going to make a big push after Squeeze is released to convert
> >> packaging to use dh_python2, the new goodness in Debian Python packaging.
> >
> >I've had a brief look at it already and will have a much deeper look
> >once squeeze is released. AFAIKR the only feature it was missing to
> >completely cover my usecase was good handling of setuptools extras.
>
> Can you be more specific? Before I got swamped with the Python 2.7 transition
> (it will be the default in Ubuntu 11.04), I began looking at dh_python2,
> adding unit tests, etc. I do plan to get back to it once we have the archive
> more happily on Python 2.7[*].
>
> [*] https://bugs.launchpad.net/ubuntu/+bugs?field.tag=python27
I always use zope.component as an example because it has the worst case
use of setuptools extras which a lot of other packages depend on.
In 3.9.1 this is:
extras_require = dict(
hook = ['zope.hookable'],
persistentregistry = ['ZODB3'],
zcml = ['zope.configuration',
'zope.i18nmessageid',
],
test = ['ZODB3',
'zope.testing',
'zope.hookable',
'zope.location',
'zope.proxy',
'zope.security',
],
docs = ['z3c.recipe.sphinxdoc'],
),
The way we treat these extras in the python-zope.* packages is as if
they were extra binary packages with names like
python-${distribution_name}-${extra_name}. So zope.component is actually
6 binary packages:
python-zope.component
python-zope.component-hook
python-zope.component-persistentregistry
python-zope.component-zcml
python-zope.component-test
python-zope.component-docs
But, we can't make 6 binary packages or the FTP Masters will kill us. So
we do a few things with extras:
* Have Ignore them and their dependencies.
* Provide the extra package from the main package and depend on it's
dependencies.
* Have their dependencies as "suggests" on the main package
* Break them out into a binary metapackage.
python-zope.component does all of these for it's extras as appropriate
in each case.
I wrote a plugin to debhelper 7 so this can be specified in an easy way
using environment variables in the rules file:
http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?revision=2053&view=markup
The code to do the work is in the python-van.pydeb package. One of the
things I had hoped to do after squeeze was port the van.pydep tests to
dh_python2 and then bother someone else to fix them.
python-zope.security is also another tough case. In that case, Ubuntu
needs to carry a diff against Debian to cope with the "untrustedpython"
extra.
> >> This should make things more consistent, and, while I have not yet tracked
> >> down specific details yet, it should make handling namespace packages much
> >> more robust. For example, with my flufl packages, there is no binary
> >> package owning flufl/__init__.py. That file does not show up in the binary
> >> package and yet it still gets created properly when any of the subpackages
> >> get installed.
> >
> >Any idea as to the mechanism?
>
> Hints, but nothing definite. I'm not sure which part of the tool chain is
> suppressing the neamespace package's __init__.py, and which part is laying it
> down when the package gets installed. It *is* nice that the packager doesn't
> have to worry about it though. I don't think you should have to do the tricks
> zope.interface is doing, or even do the
Re: [Distutils] zc.buildout fails to use system-installed dep?
On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote: > On Dec 10, 2010, at 09:36 AM, Brian Sutherland wrote: > > >Please don't, this has been reported various times already. > > > >Removing the zope/__init__.py causes breakage in other places, as I > >detail in my response to your bug. > > > >I would guess that the real bug is at a deeper level. > > Just a couple of things: > > The way zope.interface "owns" the zope namespace is not correct. Ideally, > there would be a separate binary package that owns zope/__init__.py and on > which all other zope subpackages depends. This could of course be built from > the zope.interface source package. I originally had it that way, but was strongly advised to change it to the current method to be able to pass the Debian FTP Master gauntlet. The current method using dpkg-divert is not too bad, more packages than python-zope.interface could include that file as well. So you don't force installation of python-zope.interface. It also uses standard dpkg functionality, which is a robustness bonus. > Second, we're going to make a big push after Squeeze is released to convert > packaging to use dh_python2, the new goodness in Debian Python packaging. I've had a brief look at it already and will have a much deeper look once squeeze is released. AFAIKR the only feature it was missing to completely cover my usecase was good handling of setuptools extras. I'm hoping to be able to completely replace the custom tools we use in python-zope.* packages with it at some point. > This should make things more consistent, and, while I have not yet tracked > down specific details yet, it should make handling namespace packages much > more robust. For example, with my flufl packages, there is no binary package > owning flufl/__init__.py. That file does not show up in the binary package > and yet it still gets created properly when any of the subpackages get > installed. Any idea as to the mechanism? What about 2nd level namespace packages (horror: zope.app.foo)? > Sadly PEP 382 was not complete in time for Python 3.2. Yeah, there are a lot of packaging related PEPs coming out lately. It's really great to see attention being paid to these dusty corners:) -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout fails to use system-installed dep?
On Fri, Dec 10, 2010 at 10:34:57AM +0100, Alan Franzoni wrote: > On Fri, Dec 10, 2010 at 9:36 AM, Brian Sutherland > wrote: > > > > Please don't, this has been reported various times already. > > > > Removing the zope/__init__.py causes breakage in other places, as I > > detail in my response to your bug. > > Sigh, I guess we'd need a Ubuntu/Debian python packaging FAQ? > > If it has been reported many times but keeps popping up, it's probably > unpredictable enough. It took me 3 hours just on yesterday evening to > find a workaround to that, and I'd have gladly spared that :-( Sorry, if I'd realized sooner that your issues with buildout were related, I would have spoken up sooner. I've also lost many hours of my life due to this one :( I'll mark your launchpad bug for now as Wontfix and hope that the deeper bug gets resolved sometime. Unfortunately even figuring out what that is (or what causes it) is beyond my ken. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout fails to use system-installed dep?
On Fri, Dec 10, 2010 at 12:17:02AM +0100, Alan Franzoni wrote: > On Thu, Dec 9, 2010 at 11:25 PM, Alan Franzoni wrote: > [cut] > > It appears that the zope.interface mantainer from Debian manually > copies the unnecessary - and confusing - __init__.py files: > > http://svn.debian.org/viewsvn/pkg-zope/zope.interface/trunk/debian/rules?revision=2074&view=markup > > It seems the only package in the python-zope.* hierarchy doing that; I > don't know why. > > I have opened a ticket in Ubuntu: > > https://bugs.launchpad.net/ubuntu/+source/zope.interface/+bug/688335 > > I'm unable to do the same in Debian although it's probably there, > because they seem to have no web interface and I don't have a Debian > machine available right now. If anybody could check it and do so it > would be great. Please don't, this has been reported various times already. Removing the zope/__init__.py causes breakage in other places, as I detail in my response to your bug. I would guess that the real bug is at a deeper level. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Low Level API for translating distutils/setuptools metatdata to Debian metadata
Hi Zooko,
I just implemented what I was talking about below on the van.pydeb
trunk. Running the script against unstable I get a list of 565 packages
that don't match the default mapping, I've added them to van.pydeb's
database.
The checkin is here:
http://mail.zope.org/pipermail/checkins/2009-July/036397.html
If you have the time, please have a look over the list to see that it
contains the packages you need correctly. At least pyOpenSSL you mention
below is handled correctly.
I probably will make a new release of van.pydeb in a week or so.
On Wed, Jul 22, 2009 at 05:52:33PM +0200, Brian Sutherland wrote:
> On Wed, Jul 22, 2009 at 08:41:38AM -0600, Zooko Wilcox-O'Hearn wrote:
> >> * Provides a mapping between python project names and Debian
> >> binary/source package names
> >> * Converts setuptools versions to Debian versions while
> >> maintaining
> >> sort order
> >> * Can introspect an .egg-info directory to figure out the Debian
> >> dependencies for use in the debian/control file. It can also
> >> handle/understand extras (I Hope!)
> >
> > I looked briefly at this code, and it appears that it is doing purely
> > syntactic mapping between Debian package names and Python distribution
> > names, for example:
> >
> > def py_to_bin(setuptools_project):
> > """Convert a setuptools project name to a debian binary package
> > name"""
> > return _PY_TO_BIN.get(setuptools_project, 'python-%s' %
> > setuptools_project.lower())
> >
> > This works most of the time, but it isn't reliable. For example, the
> > module name is "OpenSSL", the distribution name is "pyOpenSSL", and the
> > Debian package name is "python-openssl":
> >
> > http://packages.debian.org/sid/amd64/python-openssl/filelist
> >
> > That's why I contributed a patch to stdeb which uses the Debian database
> > of which files are included in which packages (the same database that
> > generates the web page linked above):
> >
> > http://github.com/astraw/stdeb/blob/647dd441a1712f8df37b5f7f5ba22ab6aeb2c3e7/stdeb/util.py#L135
> >
> > The way stdeb does it looks in the database for files named
> > "$DISTRIBUTION-$VERSION-py$PYTHONVERSION.egg-info". The Debian package
> > that includes such a file is the Debian package that you need to install
> > in order to satisfy a dependency on $DISTRIBUTION, $VERSION. This works
> > regardless of whether the Python package is built with distutils or
> > setuptools, and indeed it works for all packages that I know of. (There
> > is actually one exception: the Debian package for setuptools itself
> > doesn't include a version number in its .egg-info filename:
> >
> > http://packages.debian.org/sid/amd64/python-setuptools/filelist
> >
> > It has a file named:
> >
> > /usr/share/pyshared/setuptools.egg-info/
> >
> > I guess we should add a fall-back-with-warning behavior to stdeb that if
> > it can't find "$DISTRIBUTION-$VERSION.egg-info", but it can find
> > "$DISTRIBUTION.egg-info", then it should (optionally) assume that the
> > Debian package that has that file will satisfy the requirement,
> > regardless of the version number in the requirement. That, or someone
> > should open a ticket asking Debian to add a version number to that
> > filename.
> >
> > The exact regexp is currently:
> >
> > egginfore=("(/(%s)(?:-[^/]+)?(?:-py[0-9]\.[0-9.]+)?\.egg-info)"
> >% '|'.join(req.project_name for req in requirements))
> >
> > If you would be interested in including this mechanism to query the
> > database in van.pydeb, I would be happy to advise you.
>
> Hi Zooko,
>
> van.pydeb is designed to be run at package build time, rather than at
> the time you create the source package. I assume that's when stdeb's
> code runs? Using apt-file during package build time on one of the Debian
> project's auto-builders will not be acceptable (I assume).
>
> Also, "isn't reliable" has very different meanings :) van.pydeb is
> reliable in that it produces the same answer independently of the
> machine it's run on. But it's not reliable in that often the answer is
> just plain wrong;)
>
> The results of stdeb's mechanism depend on the configuration of the
> machine where you run it (or even if you havn't run apt-file update
> recently).
>
> So, I don't think van.pydeb
Re: [Distutils] Low Level API for translating distutils/setuptools metatdata to Debian metadata
On Wed, Jul 22, 2009 at 08:41:38AM -0600, Zooko Wilcox-O'Hearn wrote:
>> * Provides a mapping between python project names and Debian
>> binary/source package names
>> * Converts setuptools versions to Debian versions while
>> maintaining
>> sort order
>> * Can introspect an .egg-info directory to figure out the Debian
>> dependencies for use in the debian/control file. It can also
>> handle/understand extras (I Hope!)
>
> I looked briefly at this code, and it appears that it is doing purely
> syntactic mapping between Debian package names and Python distribution
> names, for example:
>
> def py_to_bin(setuptools_project):
> """Convert a setuptools project name to a debian binary package
> name"""
> return _PY_TO_BIN.get(setuptools_project, 'python-%s' %
> setuptools_project.lower())
>
> This works most of the time, but it isn't reliable. For example, the
> module name is "OpenSSL", the distribution name is "pyOpenSSL", and the
> Debian package name is "python-openssl":
>
> http://packages.debian.org/sid/amd64/python-openssl/filelist
>
> That's why I contributed a patch to stdeb which uses the Debian database
> of which files are included in which packages (the same database that
> generates the web page linked above):
>
> http://github.com/astraw/stdeb/blob/647dd441a1712f8df37b5f7f5ba22ab6aeb2c3e7/stdeb/util.py#L135
>
> The way stdeb does it looks in the database for files named
> "$DISTRIBUTION-$VERSION-py$PYTHONVERSION.egg-info". The Debian package
> that includes such a file is the Debian package that you need to install
> in order to satisfy a dependency on $DISTRIBUTION, $VERSION. This works
> regardless of whether the Python package is built with distutils or
> setuptools, and indeed it works for all packages that I know of. (There
> is actually one exception: the Debian package for setuptools itself
> doesn't include a version number in its .egg-info filename:
>
> http://packages.debian.org/sid/amd64/python-setuptools/filelist
>
> It has a file named:
>
> /usr/share/pyshared/setuptools.egg-info/
>
> I guess we should add a fall-back-with-warning behavior to stdeb that if
> it can't find "$DISTRIBUTION-$VERSION.egg-info", but it can find
> "$DISTRIBUTION.egg-info", then it should (optionally) assume that the
> Debian package that has that file will satisfy the requirement,
> regardless of the version number in the requirement. That, or someone
> should open a ticket asking Debian to add a version number to that
> filename.
>
> The exact regexp is currently:
>
> egginfore=("(/(%s)(?:-[^/]+)?(?:-py[0-9]\.[0-9.]+)?\.egg-info)"
>% '|'.join(req.project_name for req in requirements))
>
> If you would be interested in including this mechanism to query the
> database in van.pydeb, I would be happy to advise you.
Hi Zooko,
van.pydeb is designed to be run at package build time, rather than at
the time you create the source package. I assume that's when stdeb's
code runs? Using apt-file during package build time on one of the Debian
project's auto-builders will not be acceptable (I assume).
Also, "isn't reliable" has very different meanings :) van.pydeb is
reliable in that it produces the same answer independently of the
machine it's run on. But it's not reliable in that often the answer is
just plain wrong;)
The results of stdeb's mechanism depend on the configuration of the
machine where you run it (or even if you havn't run apt-file update
recently).
So, I don't think van.pydeb can use stdeb's mechanism as-is, but I'd
love to see a patch (with tests!) for a variation of it. I was thinking
of writing a script/function that could use apt-file to generate a list
of python->debian package mappings that don't fit the heuristic.
So py_to_bin could be re-written as:
def py_to_bin(setuptools_project):
"""Convert a setuptools project name to a debian binary package name"""
result = _HANDWRITTEN_PY_TO_BIN.get(setuptools_project)
if result is None:
result = _APT_FILE_GENERATED_PY_TO_BIN.get(setuptools_project)
if result is None:
result = 'python-%s' % setuptools_project.lower())
return result
The _APT_FILE_GENERATED_PY_TO_BIN could be re-generated periodically.
I'm hoping that'll be enough given that:
* packages don't change names that frequently
* there should be few that don't fit the heuristic (*cough* policy *cough*)
> Regards,
>
> Zooko
--
Brian Sutherland
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Utility to mirror an egg index from a Debian distribution
On Mon, Jun 15, 2009 at 11:08:22AM -0400, P.J. Eby wrote: > At 01:10 PM 6/15/2009 +0200, Brian Sutherland wrote: >> Thanks, I've just released a prototype: >> >> http://pypi.python.org/pypi/van.reposync >> >> However, as I did not want to actually execute code contained in the >> tarball, I did something like this: >> >> basedir = os.path.dirname(egg_info) >> metadata = PathMetadata(basedir, egg_info) >> dist_name = os.path.splitext(os.path.basename(egg_info))[0] >> dist = Distribution(basedir, project_name=dist_name, metadata=metadata) >> >> Unfortunately I had to play some guessing games to find where the >> .egg-info directory was. > > Actually, if all you want is the name and version, I suppose you could > parse the PKG-INFO, which is always in a set location in an sdist tarball > or zip. Yep, I changed to that method and managed to remove the guessing games. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Utility to mirror an egg index from a Debian distribution
On Fri, May 15, 2009 at 12:25:31PM -0400, P.J. Eby wrote: > At 01:00 PM 5/15/2009 +0200, Brian Sutherland wrote: >> Hi, >> >> It may seem like a backwards way of doing things, but I have a need for >> a utility that can maintain a python package index mirror of a Debian >> repository. >> >> The basic idea is to extract the tarballs of Python packages from the >> Debian repository and rename them to the original setuptools name. It >> should also create a buildout-compatible versions file of the versions >> in the repository. >> >> My current implementation idea is to unpack the tarball and use the >> egg-metadata to figure out what the "egg" name of the tarball should be. > > Running "setup.py --name --version" will dump out the name and version, > whether you use distutils or setuptools. If you want a > setuptools-compatible name and version, you'll need to postprocess those > strings with pkg_resources.safe_name() and safe_version(), then escape > them with to_filename() if you're using them as components of a sdist or > egg filename. Thanks, I've just released a prototype: http://pypi.python.org/pypi/van.reposync However, as I did not want to actually execute code contained in the tarball, I did something like this: basedir = os.path.dirname(egg_info) metadata = PathMetadata(basedir, egg_info) dist_name = os.path.splitext(os.path.basename(egg_info))[0] dist = Distribution(basedir, project_name=dist_name, metadata=metadata) Unfortunately I had to play some guessing games to find where the .egg-info directory was. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345, PEP 376, PEP 386
2009/6/4 David Cournapeau : > Brian Sutherland wrote: >> Yep. Though I think that nowdays dpkg installs to a different directory than >> Distutils' default. So users have to specify extra options to break >> their systems. >> > > That's unfortunately not true: by default (wo any --prefix option), > distutils install into /usr. So if a user just uses sudo with a > distutils-based package, he is very likely to mess up his system. I believe that changes from 2.6 onwards: https://lists.ubuntu.com/archives/ubuntu-devel/2009-February/027439.html ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345, PEP 376, PEP 386
2009/6/4 Tarek Ziadé : > On Thu, Jun 4, 2009 at 3:41 PM, Brian Sutherland > wrote: >>> And easy_install would have its own marker maybe, if the way it >>> installs stuff slighlty differs >> >> ok, seems reasonable:) > > I'll work on that. I guess it's time to code the uninstall prototype, > >> >> Perhaps the --installer option take a string (like "dpkg") and convert >> that to an md5 sum? >> > > Mmm, or maybe drop the md5 completely, and go for a human readable name. > > (distutils, dpkg, etc..) > > The api would raise an error if the name is not in [A-Za-z0-9_\-] I just remembered that setuptools implicitly sets --single-version-externally-managed when the --root option is set. Unfortunately my packaging-fu is not good enough to tell if it's an option in this case as well, i.e. to set --installer to "external". http://peak.telecommunity.com/DevCenter/setuptools#install-run-easy-install-or-old-style-installation ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345, PEP 376, PEP 386
2009/6/4 Tarek Ziadé : > On Thu, Jun 4, 2009 at 3:23 PM, Brian Sutherland > wrote: >> >> So, with setuptools I was running this while building a package: >> >> python2.X setup,py install --single-version-externally-managed >> --root=debian/$(package) --install-data=usr/lib/$(package) >> >> So, now I would need to run this: >> >> python2.X setup,py install --root=debian/$(package) >> --install-data=usr/lib/$(package) --installer >> 7e2fccc88b1f74aeee3d37340e8183ed >> >> And that would cause distutils to refuse to uninstall that package by >> default? > > something like that, > > but it would be even simpler : the installer option would be > automatically added by setuptools > when the 'install' command is called, with setuptools md5 marker. > (since setuptools has its own install command) > > That said, maybe setuptools would decide to rely on distutils for > uninstallation if its installation turns to be compatible > with distutils'one , and not provide its own marker when you manually > call 'install' like you showed. Ok, so then in this situation the --single-version-externally-managed setuptools option could tell distutils to use a non-distutils marker while the default distutils marker is used when this option is not present. > And easy_install would have its own marker maybe, if the way it > installs stuff slighlty differs ok, seems reasonable:) Perhaps the --installer option take a string (like "dpkg") and convert that to an md5 sum? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345, PEP 376, PEP 386
2009/6/4 Tarek Ziadé : > Paul >> I'd say that it's distutils' responsibility not to offer to uninstall >> anything it didn't install. > > Brian >> Yep. Though I think that nowdays dpkg installs to a different directory than >> Distutils' default. So users have to specify extra options to break >> their systems. > > But how does those third-party install the projects ? > > Don't they use Distutils' install command under the hood ? Yes, most debian packages do, during the build process. However, the actual install/uninstall on the user's machine is handled by dpkg (which maintains it's own database of installed files). > If so, the new PEP 376 APIs are ignoring previous versions of > egg-infos and work only with the ones that complies with the new > standard (it's a quick control right now, look at the "is_egg_info" > API in http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py) > > So, in any case the uninstall command will not work with project that > are filtered out by is_egg_info(). > > So maybe we could add a "INSTALLER" file with a unique md5 key > provided by the project that installed the package, > and ask for the key when calling this API ? If not provided, it would > use Distutils's md5 key So, with setuptools I was running this while building a package: python2.X setup,py install --single-version-externally-managed --root=debian/$(package) --install-data=usr/lib/$(package) So, now I would need to run this: python2.X setup,py install --root=debian/$(package) --install-data=usr/lib/$(package) --installer 7e2fccc88b1f74aeee3d37340e8183ed And that would cause distutils to refuse to uninstall that package by default? > > > Tarek > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345, PEP 376, PEP 386
2009/6/4 Tarek Ziadé :
> Oups messed up sorry -> resending my answer
>
> On Thu, Jun 4, 2009 at 11:52 AM, Brian Sutherland
> wrote:
>>> - http://svn.python.org/projects/peps/trunk/pep-0386.txt
>>
>> ... < V('1.0.dev456')
>> ... < V('1.0')
>> ... < V('1.0.dev456post623')
>>
>> Looks like a typo or very un-intuitive. It doesn't seem right that a
>> "dev" version sorts after a full release.
>
> This is a dev version of a post-release version. Which is an edge case
> submitted by Phillip.
>
> How would you write it ?
... < V('1.0.dev456')
... < V('1.0.dev456post123') # post-release of a dev version
... < V('1.0')
... < V('1.0.post456dev623') # dev version of a post-release
... < V('1.0.post456')# post release
i.e. that a postfix of "dev" means "before" and a postfix of "post" means after.
>>
>>>
>>> - PEP 376 | status : waiting for Phillip complementary feedback (and
>>> anyone else of course)
>>
>>
>> I can imagine distutils uninstalling files previously installed by dpkg
>> as a shortcut to breaking a machine. Though I'm not sure what will
>> actually happen in practice.
>
> Distutils defines a standard for an EGG-INFO structure, and provides a
> API for the uninstallation that is more likely to be a reference
> implementation.
>
> Although, It is already providing an install feature.
>
> If uninstalling a package with Distutils, while it was installed by dpkg
> breaks,
>
> I can imagine that in the very same system, you can also break it
> if you *install* packages with Distutils, easy_install, pip because
> you shortcut dpkg
> as well.
Yep. Though I think that nowdays dpkg installs to a different directory than
Distutils' default. So users have to specify extra options to break
their systems.
> I think this is the responsability of such a system to make sure it handles
> all installation and uninstallation.
How should such a system stop distutils (acting as root) from doing an
uninstallation?
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345, PEP 376, PEP 386
On Thu, Jun 04, 2009 at 11:17:38AM +0200, Tarek Ziad? wrote:
> Hello
>
> Here's a status of the current work waiting to be included in
> Distutils. (target: Python 2.7 and Python 3.2)
>
> I have created PEP 386 for the version comparison work, and gathered
> in it all the work related to version comparison,
> I am not an Fedora, Ubuntu, [put your os here] specialist and the PEP
> is in its early stage, so it needs your feedback
> of you see false statements, incomplete info or errrors.
>
> STATUS:
>
> - new PEP 386 | waiting for your feedback
>
> - http://svn.python.org/projects/peps/trunk/pep-0386.txt
... < V('1.0.dev456')
... < V('1.0')
... < V('1.0.dev456post623')
Looks like a typo or very un-intuitive. It doesn't seem right that a
"dev" version sorts after a full release.
>
> - PEP 376 | status : waiting for Phillip complementary feedback (and
> anyone else of course)
>
>- up-to-date PEP proposal :
> http://bitbucket.org/tarek/pep376/src/tip/docs/pep-0376
Will there be a way to tell distutils that it is not responsible for a
package and should error when trying to uninstall it?
Perhaps not installing the RECORD file at all in this case.
I can imagine distutils uninstalling files previously installed by dpkg
as a shortcut to breaking a machine. Though I'm not sure what will
actually happen in practice.
>- up-to-date prototype code for pkgutil :
> http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py
>
> - PEP 345 | status : waiting for PEP 386 validation
>
>
> Regards
> Tarek
>
> --
> Tarek Ziadé | http://ziade.org
> _______
> Distutils-SIG maillist - Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
--
Brian Sutherland
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Low Level API for translating distutils/setuptools metatdata to?Debian metadata
On Sat, May 30, 2009 at 07:27:06PM -0400, David Lyon wrote: > On Fri, 29 May 2009 08:35:16 -0700, Andrew Straw > wrote: > > Brian Sutherland wrote: > >> I've just published a very small library that does three things so far: > >> > >> * Provides a mapping between python project names and Debian > >> binary/source package names > >> * Converts setuptools versions to Debian versions while maintaining > >> sort order > >> * Can introspect an .egg-info directory to figure out the Debian > >> dependencies for use in the debian/control file. It can also > >> handle/understand extras (I Hope!) > >> > >> It's basically an attempt to find a common ground between all the > >> projects doing automated Python->Debian packaging. I have a feeling > >> everyone is re-implementing this code all the time. > > Maybe not everybody - but I am sure there are a few... > > at least there are many who would like to do it.. > > Recently I posted a suggestion to Andrea Gavana about py2exe on the > wxPython list about his excellent gui interface for py2exe. > > My use case is this... I'm writing a package manager for python.. > I want it to be deployed to every python platform... I dont have > all those platforms > > I want p2exe or distutils (Distribution Utils?? is that what it > stands for?) to take my program and make it for every platform... > > - windows > - mac > - linux (ubuntu,debian,suse,redhat) > > It's actually not so many.. > > No, I am not talking jibberish or pie in the sky here.. > > Here's specifically what needs doing. > > 1) create a three page "wizard" style app in wxpython > > 2) the app has checkboxes to allow the user to select the > operating systems that they want to deploy to > > 3) p2app for mac, py2exe for windows.. whichever one > for linux is driven.. much like gui2exe > > 4) Cheetah templates are used to create the necessary > output scripts from the meta information. > > (building scripts in code is so 1970s.. we don't > want to do it that way again) > > 5) The scripts get run and the user ends up with > .debs .rpm... windows installer > > basic structure... > > for plat in user_selected_platforms: > build_for_platform(plat) > > def build_for_platform(platform): > if platform == "mac": > .. > elif > > I think distutils has been in a "frozen" state for the > last few years. With no notable improvements apart > from bug fixes and internal work. > > Given that the general consensus is largely to leave > package/application management to the operating > system, what I describe above is a more modern way to > accomplish what people are trying to do with less > effort on the part of all... > > If a user makes a python app... why shouldn't they > be able to run a distutils script to "distribute" > to a platform they don't have > > Building ubuntu or debian packages is no longer > a black art To be honest, it still is;) > It's 2009 now nearly 2010... > > When was the last time distutils ever got a > major upgrade? > > Is this not what "disutils" should be doing? I would personally be very happy if distutils kept out of the business of making packages for anything but python to consume. Making OS level packages is, in my view, something that should be built on top of, not inside distutils. So the current distutils development is going in the right direction for me: Specifying common metadata formats and making APIs to access that. > in this day and age? > > David > > -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Low Level API for translating distutils/setuptools metatdata to Debian metadata
On Fri, May 29, 2009 at 08:35:16AM -0700, Andrew Straw wrote: > Brian Sutherland wrote: > > Hi, > > > > I've just published a very small library that does three things so far: > > > > * Provides a mapping between python project names and Debian > > binary/source package names > > * Converts setuptools versions to Debian versions while maintaining > > sort order > > * Can introspect an .egg-info directory to figure out the Debian > > dependencies for use in the debian/control file. It can also > > handle/understand extras (I Hope!) > > > > It's basically an attempt to find a common ground between all the > > projects doing automated Python->Debian packaging. I have a feeling > > everyone is re-implementing this code all the time. > > The stdeb package certainly needs most of these things, so it would be > nice to consolidate the implementation. It looks like there's some very > useful stuff in van.pydeb along with good tests. I'll attempt to convert > stdeb to use it. Do you have a public source code repository for > van.pydeb in case I want to start making patches? I added a link to the svn repository to the web page, and am willing to accept patches! > > And an example of direct use in a packaging situation with complex > > dependencies is here: > > > > > > http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?view=markup > > As an aside, now that I've been looking at debian/rules files made for > debhelper 7 and python-support, that file looks bloated to my eye. (Of > course, if you're targeting older Debians without dh7, there's not much > you can do.) FWIW stdeb is growing dh7 support in the "dh7" branch. For > example, stdeb just generated this debian/rules file, which is > completely functional: > > #!/usr/bin/make -f > > # This file was automatically generated by stdeb 0.3+git+dh7 at > # Thu, 28 May 2009 15:38:44 -0700 > > %: > dh $@ > Yes we were thinking of using something similar; i.e. an includable makefile. But this does look very interesting, I will definitely investigate. But just to point it out, it's way beyond the scope of van.pydeb to define the rules file, that's for things like stdeb to do. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Low Level API for translating distutils/setuptools metatdata to Debian metadata
Hi, I've just published a very small library that does three things so far: * Provides a mapping between python project names and Debian binary/source package names * Converts setuptools versions to Debian versions while maintaining sort order * Can introspect an .egg-info directory to figure out the Debian dependencies for use in the debian/control file. It can also handle/understand extras (I Hope!) It's basically an attempt to find a common ground between all the projects doing automated Python->Debian packaging. I have a feeling everyone is re-implementing this code all the time. The code is here: http://pypi.python.org/pypi/van.pydeb/ And an example of direct use in a packaging situation with complex dependencies is here: http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?view=markup -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Utility to mirror an egg index from a Debian distribution
On Sat, May 16, 2009 at 07:13:43AM -0400, David Lyon wrote: > > Hi Brian, > Hi David, > It sounds interesting. I might be interested in helping. Great :) > What I would like to do is make a test script to download all the > packages off pypi and build them under multiple platforms. > > Basically, I want to make some tests that will try to install > "everything" and then deinstall "everything" and see what happens. > > I'm not sure if this has ever been done.. but it might provide > some interesting test results.. It think that could give some good information, especially about dependency issues, and perhaps file conflicts. > Maybe it's similar... Unfortunately I don't think it's similar enough to fold these 2 projects into one. I'm basically trying to re-make pypi (or a list of links to pypi) from a Debian repository. I.e something that looks like this: http://ftp.us.debian.org/debian/ Also, while I will probably be downloading most of the tarballs, I definitely won't be installing anything. > > > On Fri, 15 May 2009 13:00:41 +0200, Brian Sutherland > wrote: > > Hi, > > > > It may seem like a backwards way of doing things, but I have a need for > > a utility that can maintain a python package index mirror of a Debian > > repository. > > > > The basic idea is to extract the tarballs of Python packages from the > > Debian repository and rename them to the original setuptools name. It > > should also create a buildout-compatible versions file of the versions > > in the repository. > > > > My current implementation idea is to unpack the tarball and use the > > egg-metadata to figure out what the "egg" name of the tarball should be. > > > > Does such a tool exists? If not, I'll probably start working on one on > > svn.zope.org Real Soon Now (TM). Comments, suggestions much > > appreciated! > -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Utility to mirror an egg index from a Debian distribution
Hi, It may seem like a backwards way of doing things, but I have a need for a utility that can maintain a python package index mirror of a Debian repository. The basic idea is to extract the tarballs of Python packages from the Debian repository and rename them to the original setuptools name. It should also create a buildout-compatible versions file of the versions in the repository. My current implementation idea is to unpack the tarball and use the egg-metadata to figure out what the "egg" name of the tarball should be. Does such a tool exists? If not, I'll probably start working on one on svn.zope.org Real Soon Now (TM). Comments, suggestions much appreciated! -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Adding a sub-command to the setup.py "build" and "develop" commands
Hi,
I've just released a setuptools/distutils extension that makes the
process of compiling translations (i.e. .po -> .mo files) quite
automatic.
http://pypi.python.org/pypi/van.potomo/
However, one major problem is that to modify the function of the
setup.py "build" and "develop" commands one needs to do this in the
setup.py:
from setuptools import setup, find_packages
from van.potomo import develop, build
setup(
name = "HelloWorld",
cmdclass = {'build': build,
'develop': develop},
setup_requires = ["van.potomo"],
version = "0.1",
packages = find_packages(),
)
Meaning that you have to manually install van.potomo before tools like
buildout can run the setup.py to figure out the dependencies. Is there
any way to make that more automatic, especially so that buildout can
have a chance?
>From my experience with Debian, there's a "Build Dependencies" field in
the control file where one can specify such things. I'm kindof hoping
there's a setuptools equivalent?
I've searched for a while and most solutions involve monkey patching or
too much code in the setup.py.
--
Brian Sutherland
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and beyond
On Sat, Mar 22, 2008 at 02:31:34AM +0100, "Martin v. L?wis" wrote: > > Tools which will need this data, in order to do their work. Hence, > > the reason for standardizing the data, instead of the tool(s). > > If there was a chance that the infrastructure being developed > actually helps these tools, *that* would be a reasonable goal, > IMO. > > However, I'm extremely skeptical that this can ever succeed > to the degree that whoever provides RPMs, .debs, or MSI > files will actually use such data, as they will find that > the data are incomplete, and they have to redo all of it, > anyway. I've found it extremely useful to have access to dependency information when making Debian packages automagically out of setuptools tarballs. It's not easy or robust to access/use it, but for my simple pure python packages, it's been working wonderfully. -- Brian Sutherland ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
