Re: new dh_python proposal
[Josselin Mouette, 2010-01-22] Le mardi 19 janvier 2010 à 20:45 +0100, Piotr Ożarowski a écrit : [Josselin Mouette, 2010-01-19] Le vendredi 15 janvier 2010 à 11:58 +0100, Piotr Ożarowski a écrit : - starting/stopping daemons problems (very long downtimes due to byte compilation via triggers) in pysupport, This is only a problem for daemons using namespace packages, which you can count with one hand’s fingers. it takes too long even when I run `dpkg -i foo.deb`, IMHO What takes too long? I just run `dpkg -i python-routes_1.11-1_all.deb` 4 times and pysupport's triggers needed 6, 7, 6 and 5 seconds to byte compile 8 .py files on Core 2 Duo @ 2.33GHz. compileall.py needed 0m0.048s to do the same. I will try to debug ( fix) it after cleaning my TODO a little bit. This is another big regression for packages that don’t use distutils / setuptools / whatever crap upstreams invent. Currently, for architecture-independent packages, you can use whatever build system is shipped, close your eyes, run dh_py* and get done with it. With your proposal, it means that *all* archirecture-independent packages would require adaptation to loop over supported Python versions. Since binary packages will have hardcoded list of supported Python versions, why not additionally test these versions at build time instead of install time? If it really is a problem to install for all supported Python versions, I can add something like -V 2.5+ in order to create symlinks for missing .py files and bytecompile for installed versions only, but I'd strongly encourage developers to add python-all to Build-Depends-Indep anyway (so that I could test them and FTBFS if needed). I don’t think you understand the problem here. For example, if a package uses the autotools instead of distutils, it will require lots of changes in debian/rules. Multiply them by all packages not using distutils and have fun. I already wrote that I will add -V option (I would have to do it anyway, as I want to also prepare dh_pycentral replacement) This is a huge task, and a complete waste of time. Don’t forget that most people that were motivated to work on Python packages are tired of endless changes that had to be done in hundreds of packages because of thoughtless changes in core Python packages. I'll do my best to make it work even without changes in packages that currently use pycentral or pysupport. How would it work? Currently the proposal doesn’t describe that. by replacing dh_pycentral with dh_python wrapper (note that dh_pycentral is invoked only at build time) I see it the other way around: short scripts will allow me to fix things in /usr/bin/py* commands instead of requiring to rebuild all packages (problem we currently have with Lenny's pycentral based packages). How do you fix autogenerated .rtupdate scripts without rebuilding all packages? if all what this autogenerated .rtupdate script does is invoking another script (with a set of args) provided by python package, in most cases it should be enough to fix python package's script. That's what you do f.e. in maintainer scripts generated by python-support, no? You fix update-python-modules instead of regenerating packages. Please note that even if something will fail in maintainer script, the package will still be usable (.py files are already in the final location). That’s until you discover a new way for the scripts to fail - although I agree that in theory, there should be less problems. I can use: `pycompile ... || true` for public modules (and I consider it a great advantage over pysupport/pycentral) Or even better, always pycompile mypackage, and ship the list of directories to byte-compile in a file. I want to avoid shipping additional files or hardcoding list of files in maintainer scripts. I see no problem in merging package contents with pycompile command line options. The problem will come if you ever change the exclusion lists of what needs and needs not to be byte-compiled. All my exclusions will be either hardcoded in pycompile argv (private dirs and few broken public modules) or in pycompile sources (i.e. I can use package contents). Only the first one requires rebuilding a package if something will not work correctly. -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 signature.asc Description: Digital signature
Re: new dh_python proposal
Le mardi 19 janvier 2010 à 20:45 +0100, Piotr Ożarowski a écrit : [Josselin Mouette, 2010-01-19] Le vendredi 15 janvier 2010 à 11:58 +0100, Piotr Ożarowski a écrit : - starting/stopping daemons problems (very long downtimes due to byte compilation via triggers) in pysupport, This is only a problem for daemons using namespace packages, which you can count with one hand’s fingers. it takes too long even when I run `dpkg -i foo.deb`, IMHO What takes too long? The time between stopping and restarting the daemon? In this case, the package needs to be adapted so that the daemon is restarted - or even better, told to restart itself through a signal - in the postinst without being stopped in the prerm. This is another big regression for packages that don’t use distutils / setuptools / whatever crap upstreams invent. Currently, for architecture-independent packages, you can use whatever build system is shipped, close your eyes, run dh_py* and get done with it. With your proposal, it means that *all* archirecture-independent packages would require adaptation to loop over supported Python versions. Since binary packages will have hardcoded list of supported Python versions, why not additionally test these versions at build time instead of install time? If it really is a problem to install for all supported Python versions, I can add something like -V 2.5+ in order to create symlinks for missing .py files and bytecompile for installed versions only, but I'd strongly encourage developers to add python-all to Build-Depends-Indep anyway (so that I could test them and FTBFS if needed). I don’t think you understand the problem here. For example, if a package uses the autotools instead of distutils, it will require lots of changes in debian/rules. Multiply them by all packages not using distutils and have fun. This is a huge task, and a complete waste of time. Don’t forget that most people that were motivated to work on Python packages are tired of endless changes that had to be done in hundreds of packages because of thoughtless changes in core Python packages. I'll do my best to make it work even without changes in packages that currently use pycentral or pysupport. How would it work? Currently the proposal doesn’t describe that. - create simple maintainer scripts with pycompile and pyclean commands and (if needed) rtupdate script (for private directories that use default Python version and will most probably work with next default version). Private modules that cannot be used with default Python version will get additional pycompile command with all needed arguments, like minimum required Python version or hardcoded version in both, maintainer and rtupdate scripts, This part looks worrysome to me. Putting the complexity of the installation/cleanup actions in auto-generated maintainer scripts (and now auto-generated rtupdate scripts) sounds like reproducing one of the core design errors of python-central. The more complexity you put in auto-generated scripts, the less chances you leave for the system to clean itself up by just upgrading the helper scripts in case of a bug. I see it the other way around: short scripts will allow me to fix things in /usr/bin/py* commands instead of requiring to rebuild all packages (problem we currently have with Lenny's pycentral based packages). How do you fix autogenerated .rtupdate scripts without rebuilding all packages? Please note that even if something will fail in maintainer script, the package will still be usable (.py files are already in the final location). That’s until you discover a new way for the scripts to fail - although I agree that in theory, there should be less problems. ok, `pycompile -p python-foo` and `pycompile -p mypackage \ /usr/share/privatedir1` it is. Much better indeed. Or even better, always pycompile mypackage, and ship the list of directories to byte-compile in a file. I want to avoid shipping additional files or hardcoding list of files in maintainer scripts. I see no problem in merging package contents with pycompile command line options. The problem will come if you ever change the exclusion lists of what needs and needs not to be byte-compiled. For architecture: all packages, I’m still not convinced. Having to rebuild these Python packages means roughly 3 times as many packages to rebuild during transitions as there are now. This is going to make transitions a *lot* more complicated. This means much more work for everyone, including the RT, at times of changing the list of supported Python versions. We also lose the current potential to have a large subset of the archive immediately available for a new Python version when it becomes ready. that's the main disadvantage, I agree. It's not that we change the list of supported Python versions that often, though...
Re: new dh_python proposal
Le vendredi 15 janvier 2010 à 11:58 +0100, Piotr Ożarowski a écrit :
Here's updated dh_python proposal. Main change since my last mail:
pycompile/pyclean will no longer search for files to compile/remove
in public directories as Matthias will fix #552595 using triggers so I don't
have to care about pycentral bugs anymore and dpkg -L seems to be fast enough
to be used in maintainer scripts (BTW: my previous proposal had a typo:
prerm
should be replaced with postrm, I guess everyone figured that out from the
context…).
There is not much change since the first proposal, so I’m just sending
an updated version of the comments I already sent privately.
Again, the main idea is to ship files in official site/dist-packages location
or symlink files from /usr/share/py{,3}shared directory there (if files can be
shared).
This means Architecture:all packages will need sourceful uploads once list of
supported Python versions will change (binNMUs do not touch architecture
independent packages and even if they'd do, that still leaves the
=${source:Version} problem).
This is an *extremely* bad idea. It feels like a huge regression in
terms of testing migration. It means a lot of unnecessary work,
precisely the kind of work we want to avoid.
Advantages:
* dpkg is aware of all .py files (including symlinks) and their locations (it
isn't true for pysupport/pycentral),
* no need for helper in Depends and Build-Depends - I want dh_python and
pycompile/pyclean to be shipped in python packages,
* broken modules that use __file__ incorrectly will work without problems,
True, but the right thing to do in the long term is to fix them:
http://malsain.org/~joss/python.jpg
* less opportunities to break a system while installing / upgrading Python
packages:
- problems with creating/removing symlinks in pycentral (see f.e. #552595),
And you will run into new problems with a new tool. Although this can be
better if the maintainer is simply responsive enough.
- starting/stopping daemons problems (very long downtimes due to byte
compilation via triggers) in pysupport,
This is only a problem for daemons using namespace packages, which you
can count with one hand’s fingers.
* no more compile errors at install time in public directories, all Python
versions tested at build time,
* Python modules available out of the box (think about daemons and upgrades)
Short overview of how things will work with new dh_python:
==
build time:
^^^
* files installed into the standard location[1] or into private directory.
It's up to maintainer which files will be installed for which Python
version,
by default every package providing public module should build/install for
all
`pyversions -vs` versions.
* dh_python will:
- move files to /usr/lib/pythonX.Y/*-packages if /usr/local or
/usr/lib/python2.X/site-packages (for X = 6) is used
(i.e. --install-layout or --prefix can be skipped in most cases),
- copy all files that can be shared to /usr/share/py{,3}shared and replace
the ones in *-packages with a symlink (if package supports only one Python
version, there's no need to use pyshared),
The way you describe it makes it sound like architecture: all packages
will now need to depend on python-all-dev and install their files for
all supported Python versions so that symlinks are created this way.
This is another big regression for packages that don’t use distutils /
setuptools / whatever crap upstreams invent. Currently, for
architecture-independent packages, you can use whatever build system is
shipped, close your eyes, run dh_py* and get done with it. With your
proposal, it means that *all* archirecture-independent packages would
require adaptation to loop over supported Python versions.
This is a huge task, and a complete waste of time. Don’t forget that
most people that were motivated to work on Python packages are tired of
endless changes that had to be done in hundreds of packages because of
thoughtless changes in core Python packages.
- create simple maintainer scripts with pycompile and pyclean
commands and (if needed) rtupdate script (for private directories that
use default Python version and will most probably work with next default
version). Private modules that cannot be used with default Python version
will get additional pycompile command with all needed arguments, like
minimum required Python version or hardcoded version in both, maintainer
and rtupdate scripts,
This part looks worrysome to me. Putting the complexity of the
installation/cleanup actions in auto-generated maintainer scripts (and
now auto-generated rtupdate scripts) sounds like reproducing one of the
core design errors of python-central. The more complexity you put in
auto-generated scripts, the less chances you leave for the system to
clean itself up by just upgrading the
Re: new dh_python proposal
[Josselin Mouette, 2010-01-19] Le vendredi 15 janvier 2010 à 11:58 +0100, Piotr Ożarowski a écrit : - starting/stopping daemons problems (very long downtimes due to byte compilation via triggers) in pysupport, This is only a problem for daemons using namespace packages, which you can count with one hand’s fingers. it takes too long even when I run `dpkg -i foo.deb`, IMHO [...] The way you describe it makes it sound like architecture: all packages will now need to depend on python-all-dev and install their files for all supported Python versions so that symlinks are created this way. python-all, not python-all-dev, but that's correct. This is another big regression for packages that don’t use distutils / setuptools / whatever crap upstreams invent. Currently, for architecture-independent packages, you can use whatever build system is shipped, close your eyes, run dh_py* and get done with it. With your proposal, it means that *all* archirecture-independent packages would require adaptation to loop over supported Python versions. Since binary packages will have hardcoded list of supported Python versions, why not additionally test these versions at build time instead of install time? If it really is a problem to install for all supported Python versions, I can add something like -V 2.5+ in order to create symlinks for missing .py files and bytecompile for installed versions only, but I'd strongly encourage developers to add python-all to Build-Depends-Indep anyway (so that I could test them and FTBFS if needed). This is a huge task, and a complete waste of time. Don’t forget that most people that were motivated to work on Python packages are tired of endless changes that had to be done in hundreds of packages because of thoughtless changes in core Python packages. I'll do my best to make it work even without changes in packages that currently use pycentral or pysupport. - create simple maintainer scripts with pycompile and pyclean commands and (if needed) rtupdate script (for private directories that use default Python version and will most probably work with next default version). Private modules that cannot be used with default Python version will get additional pycompile command with all needed arguments, like minimum required Python version or hardcoded version in both, maintainer and rtupdate scripts, This part looks worrysome to me. Putting the complexity of the installation/cleanup actions in auto-generated maintainer scripts (and now auto-generated rtupdate scripts) sounds like reproducing one of the core design errors of python-central. The more complexity you put in auto-generated scripts, the less chances you leave for the system to clean itself up by just upgrading the helper scripts in case of a bug. I see it the other way around: short scripts will allow me to fix things in /usr/bin/py* commands instead of requiring to rebuild all packages (problem we currently have with Lenny's pycentral based packages). Please note that even if something will fail in maintainer script, the package will still be usable (.py files are already in the final location). [...] $ grep foo debian/python-foo.prerm dpkg -L python-foo | pyclean Ugh. Use 'pycompile python-foo', pretty please. [...] $ grep privatedir debian/mypackage.postinst dpkg -L mypackage | pycompile /usr/share/privatedir1 dpkg -L mypackage | pycompile /usr/share/privatedir2 -V 2.4 Ugh again. How about that? pycompile -p mypackage /usr/share/privatedir1 ok, `pycompile -p python-foo` and `pycompile -p mypackage \ /usr/share/privatedir1` it is. Or even better, always pycompile mypackage, and ship the list of directories to byte-compile in a file. I want to avoid shipping additional files or hardcoding list of files in maintainer scripts. I see no problem in merging package contents with pycompile command line options. Overall, I’m still dissatisfied with this proposal. First of all, for private modules, it has absolutely zero interest, and only consists in regressions. I don’t see the point in the proposed changes for private modules, apart from introducing a third helper tool with less features than both existing ones. maintainers of packages with private modules only will not see many advantages over pysupport or pycentral, agreed. [...] For architecture: all packages, I’m still not convinced. Having to rebuild these Python packages means roughly 3 times as many packages to rebuild during transitions as there are now. This is going to make transitions a *lot* more complicated. This means much more work for everyone, including the RT, at times of changing the list of supported Python versions. We also lose the current potential to have a large subset of the archive immediately available for a new Python version when it becomes ready. that's the main disadvantage, I agree. It's not that we change the list
Re: new dh_python proposal
Here's updated dh_python proposal. Main change since my last mail:
pycompile/pyclean will no longer search for files to compile/remove
in public directories as Matthias will fix #552595 using triggers so I don't
have to care about pycentral bugs anymore and dpkg -L seems to be fast enough
to be used in maintainer scripts (BTW: my previous proposal had a typo: prerm
should be replaced with postrm, I guess everyone figured that out from the
context…).
Again, the main idea is to ship files in official site/dist-packages location
or symlink files from /usr/share/py{,3}shared directory there (if files can be
shared).
This means Architecture:all packages will need sourceful uploads once list of
supported Python versions will change (binNMUs do not touch architecture
independent packages and even if they'd do, that still leaves the
=${source:Version} problem).
Advantages:
* dpkg is aware of all .py files (including symlinks) and their locations (it
isn't true for pysupport/pycentral),
* no need for helper in Depends and Build-Depends - I want dh_python and
pycompile/pyclean to be shipped in python packages,
* broken modules that use __file__ incorrectly will work without problems,
* less opportunities to break a system while installing / upgrading Python
packages:
- problems with creating/removing symlinks in pycentral (see f.e. #552595),
- starting/stopping daemons problems (very long downtimes due to byte
compilation via triggers) in pysupport,
* no more compile errors at install time in public directories, all Python
versions tested at build time,
* Python modules available out of the box (think about daemons and upgrades)
Short overview of how things will work with new dh_python:
==
build time:
^^^
* files installed into the standard location[1] or into private directory.
It's up to maintainer which files will be installed for which Python version,
by default every package providing public module should build/install for all
`pyversions -vs` versions.
* dh_python will:
- move files to /usr/lib/pythonX.Y/*-packages if /usr/local or
/usr/lib/python2.X/site-packages (for X = 6) is used
(i.e. --install-layout or --prefix can be skipped in most cases),
- copy all files that can be shared to /usr/share/py{,3}shared and replace
the ones in *-packages with a symlink (if package supports only one Python
version, there's no need to use pyshared),
- create simple maintainer scripts with pycompile and pyclean
commands and (if needed) rtupdate script (for private directories that
use default Python version and will most probably work with next default
version). Private modules that cannot be used with default Python version
will get additional pycompile command with all needed arguments, like
minimum required Python version or hardcoded version in both, maintainer
and rtupdate scripts,
- add substvar for ${python:Versions} (used in XB-Python-Version, i.e. the
make release managers happy field (this field will be used by RMs to
track transitions)
- add substvar for ${python:Provides}
[1] SL = standard distutils/setuptools/distribute location:
/usr/{,local}/lib/pythonX.Y/{site,dist}-packages/
installation time:
^^
* maintainer script will byte compile .pyc files for all provided
symlinks / private directories if given Python version is installed
(dpkg -L output will be used to detect which files need byte compilation,
directories without __init__.py file will be skipped),
* user installs new pythonX.Y package:
- bytecompile related symlinks (pycompile -V X.Y) - no need for a list of
files to compile at this point - all .py files in
/usr/lib/pythonX.Y/{site,dist}-packages will have to be byte-compiled,
byte compilation will not fail as it was already tested at build time,
* user removes pythonX.Y package:
- remove all pythonX.Y's .pyc files (pyclean -V X.Y)
- all packages with private directories that use this Python version will be
removed by dpkg (Depends field will do its job) so prerm will remove all
remaining .pyc files in private directories
* default Python version changes:
- rtupdate scripts for packages with private modules that support it (i.e.
the ones without hardcoded Python version and without private extensions)
will be invoked
examples:
=
example 1 - public modules/extensions only
^^
$ python2.4 ./setup.py install --root=debian/python-foo/
debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py
debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py
debian/python-foo/usr/lib/python2.4/site-packages/spam.py
$ python2.5 ./setup.py install --root=debian/python-foo/
debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py
Re: new dh_python proposal
Piotr Ożarowski ha scritto:
* no need for helper in Depends and Build-Depends - I want dh_python and
pycompile/pyclean to be shipped in python packages,
It is not clear to me how to achieve this.
* broken modules that use __file__ incorrectly will work without problems,
OK, I'd still warn loudly if that happens, though.
* user installs new pythonX.Y package:
- bytecompile related symlinks (pycompile -V X.Y) - no need for a list of
files to compile at this point - all .py files in
/usr/lib/pythonX.Y/{site,dist}-packages will have to be byte-compiled,
Suppose I have foo package installed, then I install pythonX.Y, so
symlinks will be byte-compiled for X.Y. What happens if I remove foo
package? Will byte-compiled files for X.Y be removed as well?
byte compilation will not fail as it was already tested at build time,
What about cases when code is no longer supported in a given Python
release? I think of except YourFavouriteException, e: code with 3.1
(but feel free to use a better example), byte-compilation will fail.
$ python3.1 ./3.x/setup.py install --root=debian/python3-foo/
debian/python3-foo/usr/local/lib/python3.1/dist-packages/foo/__init__.py
debian/python3-foo/usr/local/lib/python3.1/dist-packages/foo/_foo.so
debian/python3-foo/usr/local/lib/python3.1/dist-packages/foo/bar.py
debian/python3-foo/usr/local/lib/python3.1/dist-packages/spam.py (?)
# egg uses Python = 2.6
$ python ./setup.py install --root=debian/mypackage/
--install-lib=/usr/share/privatedir3/ --install-scripts=/usr/share/privatedir3
debian/mypackage/usr/lib/privatedir3/egg/__init__.py
debian/mypackage/usr/lib/privatedir3/run.py
Are files installed under $(CURDIR)/usr/*lib* ?
--
.''`.
: :' : Luca Falavigna dktrkr...@debian.org
`. `'
`-
signature.asc
Description: OpenPGP digital signature
Re: new dh_python proposal
* Piotr Ożarowski pi...@debian.org, 2010-01-15, 11:58: * maintainer script will byte compile .pyc files for all provided symlinks / private directories if given Python version is installed (dpkg -L output will be used to detect which files need byte compilation, directories without __init__.py file will be skipped), IOW, you want to skip modules that are not a part of a package? Why? -- Jakub Wilk signature.asc Description: Digital signature
Re: new dh_python proposal
[Jakub Wilk, 2010-01-15] * Piotr Ożarowski pi...@debian.org, 2010-01-15, 11:58: * maintainer script will byte compile .pyc files for all provided symlinks / private directories if given Python version is installed (dpkg -L output will be used to detect which files need byte compilation, directories without __init__.py file will be skipped), IOW, you want to skip modules that are not a part of a package? Why? because both Joss and Matthias didn't like touching files from other packages (although it would happen very rare). Without dpkg -L, I'd also have to use postrm instead of prerm and thus I'd have to take care of directories that dpkg refused to remove due to __init__.pyc file. -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
On 15.01.2010 14:19, Luca Falavigna wrote:
* broken modules that use __file__ incorrectly will work without problems,
OK, I'd still warn loudly if that happens, though.
there are other cases where some modules/extensions do encode paths in generated
config files, like sip and the qt/kde bindings. to avoid such problems it's
safe to call setup.py install with --install-layout=deb to have the files in the
final installation location.
* user installs new pythonX.Y package:
- bytecompile related symlinks (pycompile -V X.Y) - no need for a list of
files to compile at this point - all .py files in
/usr/lib/pythonX.Y/{site,dist}-packages will have to be byte-compiled,
Suppose I have foo package installed, then I install pythonX.Y, so
symlinks will be byte-compiled for X.Y. What happens if I remove foo
package? Will byte-compiled files for X.Y be removed as well?
that's my understanding. byte-compiled files shouldn't exist for a python
runtime which is currently not installed on the system, and there shouldn't
exist byte-compiled files on the system without a corresponding source file (at
least in the public site directory).
byte compilation will not fail as it was already tested at build time,
What about cases when code is no longer supported in a given Python
release? I think of except YourFavouriteException, e: code with 3.1
(but feel free to use a better example), byte-compilation will fail.
the source code for python3 is almost always different compared to python2.x,
although the transformation can be done automatically by the new distribute
tools on installation time. Adding another copy of python (source) code into the
same package would enlarge the package, so it would be the best to have a
separate binary package python3-* (which could be built from the same source
package).
Matthias
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
On 15.01.2010 14:27, Jakub Wilk wrote: * Piotr Ożarowski pi...@debian.org, 2010-01-15, 11:58: * maintainer script will byte compile .pyc files for all provided symlinks / private directories if given Python version is installed (dpkg -L output will be used to detect which files need byte compilation, directories without __init__.py file will be skipped), IOW, you want to skip modules that are not a part of a package? Why? which files would that be? Matthias -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
[Piotr Ożarowski, 2010-01-15] [Luca Falavigna, 2010-01-15] Piotr Ożarowski ha scritto: byte compilation will not fail as it was already tested at build time, What about cases when code is no longer supported in a given Python release? I think of except YourFavouriteException, e: code with 3.1 (but feel free to use a better example), byte-compilation will fail. then package will FTBFS and maintainer will have to skip this version in debian/rules. After a while RMs will remove such package from Debian archive (they'll watch how XB-Python-Version is changing after rebuilds) oh, you mean runtime problems that weren't detected at build time (byte compilation and tests didn't fail) - well, maintainer will have to detect this and fix debian/rules -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
On 15.01.2010 11:58, Piotr Ożarowski wrote: * no need for helper in Depends and Build-Depends - I want dh_python and pycompile/pyclean to be shipped in python packages, agreed. once these helper tools are mature for unstable/experimental, please add yourself as an uploader to python-defaults. I assume this is work which will land in squeeze. I'll see to change dh_pycentral to generate the installation scripts based on pycompile/pyclean such that no more runtime dependency on python-central is needed anymore. It would help in having to touch each source for the dh_pycentral/dh_python replacement. Matthias -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
[Jakub Wilk, 2010-01-15] * Matthias Klose d...@debian.org, 2010-01-15, 14:58: symlinks / private directories if given Python version is installed (dpkg -L output will be used to detect which files need byte compilation, directories without __init__.py file will be skipped), IOW, you want to skip modules that are not a part of a package? Why? which files would that be? $ apt-file search -x '/usr/share/pyshared/[^/]*.py$' | wc -l 554 Plus some modules directly under /usr/lib/python2.*/site-packages/, plus presumably many private modules. You mean package as in Python package (i.e. not .deb file)? These files will be byte compiled of course. By directories without __init__.py I meant something like site-packages/pylons/templates where templates contains .py files but there's no templates/__init__.py -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
* Piotr Ożarowski pi...@debian.org, 2010-01-15, 16:08: symlinks / private directories if given Python version is installed (dpkg -L output will be used to detect which files need byte compilation, directories without __init__.py file will be skipped), IOW, you want to skip modules that are not a part of a package? Why? which files would that be? $ apt-file search -x '/usr/share/pyshared/[^/]*.py$' | wc -l 554 Plus some modules directly under /usr/lib/python2.*/site-packages/, plus presumably many private modules. You mean package as in Python package (i.e. not .deb file)? Yes. These files will be byte compiled of course. By directories without __init__.py I meant something like site-packages/pylons/templates where templates contains .py files but there's no templates/__init__.py You want to skip only directories without __init__.py and under /usr/lib/python*/*-packages/, right? That's fine. -- Jakub Wilk signature.asc Description: Digital signature
Re: new dh_python proposal
Here's updated dh_python proposal. Main change since my last mail:
pycompile/pyclean will no longer search for files to compile/remove
in public directories as Matthias will fix #552595 using triggers so I
don't
have to care about pycentral bugs anymore and dpkg -L seems to be fast
enough
to be used in maintainer scripts (BTW: my previous proposal had a typo:
prerm
should be replaced with postrm, I guess everyone figured that out from
the
contextâ¦).
Again, the main idea is to ship files in official site/dist-packages
location
or symlink files from /usr/share/py{,3}shared directory there (if files
can be
shared).
This means Architecture:all packages will need sourceful uploads once list
of
supported Python versions will change (binNMUs do not touch architecture
independent packages and even if they'd do, that still leaves the
=${source:Version} problem).
In line with this, I have been working on Python policy updates to
encompass Python 3 and make sure that this proposal and policy were
aligned. Attached is a draft. This is mostly my work with only some
minimal review by a few people. It is definitely a work in progress (I
ran out of time and have not verified I successfully addressed what
comments I have gotten).
I've attached both the full text of my current draft and a diff from
what's in Testing/Unstable now.
Comments please.
Scott K
Debian Python Policy
Neil Schemenauer n...@debian.org
Matthias Klose d...@debian.org
Gregor Hoffleit fli...@debian.org
Josselin Mouette j...@debian.org
Joe Wreschnig pi...@debian.org
Loic Minier l...@debian.org
Scott Kitterman sc...@kitterman.com
version 0.9.1.0
---
Abstract
This document describes the packaging of Python within the Debian
GNU/Linux distribution and the policy requirements for packaged Python
programs and modules.
Copyright Notice
Copyright (C) 1999, 2001, 2003, 2006, 2009 Software in the Public
Interest
This manual is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2 of the License, or (at your
option) any later version.
This is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
A copy of the GNU General Public License is available as
`/usr/share/common-licences/GPL' in the Debian GNU/Linux distribution
or on the World Wide Web at The GNU General Public License
(http://www.gnu.org/copyleft/gpl.html).
You can also obtain it by writing to the Free Software Foundation,
Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA.
---
Contents
1.Python Packaging
1.1. Versions
1.2. Main packages
1.3. Minimal packages
1.4. Python Interpreter
1.4.1. Interpreter Name
1.4.2. Interpreter Location
1.5. Module Path
1.6. Hooks for updates to installed runtimes
1.7. Documentation
2.Packaged Modules
2.1. Types of Python Modules
2.2. Module Package Names
2.3. Specifying Supported Versions
2.3.1. Build time determination for python2.X versions
2.3.2. Build time determinations for python3.X versions
2.3.3. Run Time/Binary Package Versions
2.4. Dependencies
2.5. Provides
2.6. Modules Byte-Compilation
3.Python Programs
3.1. Programs using the default Python
3.1.1. Programs Shipping Private Modules
3.2. Programs Using a Particular Python Version
4.Programs Embedding Python
4.1. Building Embedded Programs
4.2. Embedded Python Dependencies
5.Interaction with Locally Installed Python Versions
A.Build Dependencies
B.Packaging Tools
B.1. distutils
B.2. python-support
B.3. python-central
B.4. CDBS
C.Upgrade Procedure
---
1. Python Packaging
---
1.1. Versions
-
At any given time, the binary package `python' will represent the
current default Debian Python version.
The default Debian Python version should
Re: new dh_python proposal
[Jakub Wilk, 2010-01-15] You want to skip only directories without __init__.py and under /usr/lib/python*/*-packages/, right? That's fine. yes (at least by default) -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 signature.asc Description: Digital signature
Re: new dh_python proposal
Kumar Appaiah wrote: On Tue, Aug 04, 2009 at 12:00:09PM -0400, anatoly techtonik wrote: Now about the proposal (from newcomer's point of view): dh_python is a shell script -- I have a strong belief that Python package automation scripts should be written in Python, there is no need to learn bashes when you program Python - do not expect that package maintainers will be able to debug their scripts in shell language. [skipping the rest of your mail] I got the impression that dh_python will be written in Piotr's favourite language, Python, but may need to be rewritten in Perl for inclusion in debhelper: http://lists.debian.org/debian-python/2009/08/msg6.html Time for a python binding to debhelper then? ;) But indeed the only proper way to write a debhelper tool is in perl. -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprints: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79 ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
FTR: Joss and few other maintainers (whose opinion I care about) didn't like my proposal (mainly due to binNMUs for arch:all packages) so I'm not working on this new tool. I planed to start working on it once we'll agree how it should look like. There's no consensus so I'm focusing on preparing all current tools for python2.6 upload to unstable. Today I uploaded cdbs NMU to delayed/7 (that fixes #537373) and reported a bug against python-central (#547565) which I plan to fix in NMU this week as well. -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 signature.asc Description: Digital signature
Re: new dh_python proposal
On 03.08.2009 19:16, Piotr Ożarowski wrote:
Just a follow-up the clear some things up:
* about the -common package (i.e. pysupport namespace issue):
- it's not a must, if one package can act as namespace provider,
there's no need to provide another one, of course,
- being able to list all files provided by packages is something we
really want to have
packaging the zope tree by choosing existing packages to include the __init__.py
files did work well. Afaik there's no other package in debian not shipping
files, and then creating these.
* dh_python[1] will try to avoid moving files to pyshared if
package supports only one Python version (f.e. the latest one)
* py{,3}compile will support -X/--exclude option
I maintain one module that uses site-packages to keep templates with
.py files inside and although I patched it to move these files outside
site-packages, I know that not every maintainer will want to do that,
so -X will disable byte compilation of files for given directory/file
* about lack of XS-Python-Version and debian/pyversions
- if available, both previous places will be used to get
minimum/maximum required Python version, it will complicate
detection of packages that need binNMU, so I want to drop both of
them at some point,
- dh/cdbs/dh_python will get minimum/maximum required Python versions
from Build-Depends{,-Indep} and/or python-all's Depends.
If one will need to build depend on newer version of
python{,-all,-dev} (due to some Debian specific changes) - tools
will ignore versioned dependencies that include dash sign or use the
lower one if two different dependencies are provided
(f.e. Build-Depends: python-all-dev (= 2.5.8), python-all-dev (= 2.4)
will be equivalent of XS-Python-Version:=2.4)
Why move this attribute from an explicit place to an implicit one? Encoding all
stuff in dependencies isn't that obvious. Indeed we do create new attributes
like Homepage, which were included before in the description.
* how to detect which package needs binNMU?
I think parsing Build-Depends{,-Indep}, Contents file and Depends
header will be enough to detect such packages, i.e. packages that:
- Build-Depends{,-Indep} on python-all{,-dev}
AND there's noX.Y build dependency (where X.Y is the one to be
introduced)
- Build-Depends on python-dev (python-dev (=X.Y) | pythonX.Y-dev or
python-dev (X.Y) might help to detect some false positives)
AND provide private extension (/usr/lib/*/*.so)
- Build-Depends{,-Indep} on python (=X.Y) | pythonX.Y and Depend on
pythonX.Y
(i.e. python (=X.Y) | pythonX.Y will NOT be in Depends)
[arch:all packages with hardcoded shebang due to default python not
being good enough]
- Build-Depends{,-Indep} on python or python (=X.Y) | pythonX.Y
and there's no rtupdate script in binary package
[private modules without hardcoded shebang]
will need binNMU once new Python version will be added to the
supported ones
Is it really worth adding semantics to the build dependency/dependency fields
and instead removing the explicit information about version information?
Advantages:
* a lot less opportunities to break a system while installing / upgrading
Python packages,
* no more problems with packages that provide the same namespace and use
different helper tool,
* Python modules available out of the box (think about daemons),
I appreciate these goals and I'm fine to provide a wrapper for dh_pycentral if
the new dh_python implements these goals.
Matthias
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
[Matthias Klose, 2009-08-20] On 03.08.2009 19:16, Piotr Ożarowski wrote: * about lack of XS-Python-Version and debian/pyversions - if available, both previous places will be used to get minimum/maximum required Python version, it will complicate detection of packages that need binNMU, so I want to drop both of them at some point, [...] Why move this attribute from an explicit place to an implicit one? Encoding all stuff in dependencies isn't that obvious. Indeed we do create new attributes like Homepage, which were included before in the description. [...] Is it really worth adding semantics to the build dependency/dependency fields and instead removing the explicit information about version information? * to ease detecting packages that need binNMU? * to make it simpler for package maintainers (my module needs Python =2.4 - lets build depend on it instead of where the heck is XS-P-V / debian/pyversion's syntax described?) * to make it easier for dh/cdbs maintainers? Anyway, I don't care about it that much, it will not make any difference if we decide to support all 3 approaches or only existing two (both previous ones has to be supported at the beginning anyway). -- -=[ Piotr Ożarowski ]=- -=[ http://www.ozarowski.pl ]=- signature.asc Description: Digital signature
Re: new dh_python proposal
Hello, Piotr. In short - your post is long. Brain is limited. Better place an overview in wiki and discuss in parts. Before discussing your proposal I would really appreciate if somebody from insiders could describe situation in Debian Python in all possible detail including the history of development. I believe that this piece of work is absolutely necessary, because the problem of repackaging Python packages for Debian and maintaining integrity between installed packets, packages and Python versions is too complex to keep every detail in mind unless you can devote your full time to the problem. One way to make the status more obvious is to answer to emails in this group, and merge answers into wiki, so that the next time you can point a person to the exact place of documentation. You say that attempt to merge -cental and -support failed, but didn't mention why. You say that current tools have problems that occur at install/upgrade time, but do not mention these problems explicitly. So it is impossible to say whatever your ideas solve current problems and won't add new. Referencing rtinstall/rtupdate/rtremove scripts is cool for a seasoned Debian developer, but for a Python developer it means nothing. In other words - not only Python things are obscure for Debian developers, but Debian stuff is also a mystery for Python masters, so these things should be explained symmetrically for both communities. The chances for Debian Python Packaging experts to pop up are magnitude less if we won't explain the situation in detail. Now about the proposal (from newcomer's point of view): dh_python is a shell script -- I have a strong belief that Python package automation scripts should be written in Python, there is no need to learn bashes when you program Python - do not expect that package maintainers will be able to debug their scripts in shell language. The description that dh_python will do at a build time looks like a solution, but again - it doesn't specify what problem is being solved. Perhaps I expected to see more high-level definition. May I ask a question - is there a difference between installation of Python Modules and Python Applications? Does the problem set you are trying to resolve with this proposal include the difference? Have you considered distributing Python Applications via virtualenv? How this proposal handles virtualenv installations? In conclusion my opinion is that problem set is not defined well enough to propose a solution or estimate if it will be effective both for current flow and for future ideas. I would start from wiki. --anatoly t. On Sun, Aug 2, 2009 at 6:53 AM, Piotr Ożarowskipi...@debian.org wrote: My attempt to merge python-central and python-support failed few months ago, so here's another proposal to improve the Python situation in Debian. It's nothing new actually, just a compilation of previous Matthias' and Joss' ideas and few of my own. The main idea is to ship symlinks (not .pyc files!) in binary packages as Matthias proposed back in February and thus get rid of both helper tools (most problems with current tools occur at install / upgrade time). I want to keep it as simple as possible and let maintainers customize it in maintainer or rtinstall/rtupdate/rtremove scripts (if really needed). Advantages: * a lot less opportunities to break a system while installing / upgrading Python packages, * no more problems with packages that provide the same namespace and use different helper tool, * Python modules available out of the box (think about daemons), * no more which tool should I use? or how do they differ? questions ;-) The main disadvantage of this approach is that architecture independent packages will have to be rebuilt once new Python version will be added to the supported ones (I think we can avoid a rebuild if a version is removed from the supported ones). Luk (our release manager) told me that binNMUs for arch:all packages will be possible in Debian at some point, but it's a matter or months or even years, so for now - manual NMUs will be needed. I think it will be easy to detect which packages need one basing on Contents file and Depends field. Another disadvantage is the fact that pysupport's namespace feature will not be supported. If we want to keep it simple and let dpkg know about all the files - new *-common package with all common __init__.py (and other if needed) files will have to be provided, just like we did before pysupport 0.7. Otherwise removing .pyc files will not be a trivial thing to do. Note that there are lots of problems with this feature anyway (upstreams tend to use site-packages for things that don't belong there), so removing it will improve the situation at the end, IMHO. I'll try to make dh_python a drop in replacement for dh_pycentral and dh_pysupport (i.e. tools used at build time) for most packages, but I don't consider it to be a main goal. Some packages
Re: new dh_python proposal
Just a follow-up the clear some things up:
* about the -common package (i.e. pysupport namespace issue):
- it's not a must, if one package can act as namespace provider,
there's no need to provide another one, of course,
- being able to list all files provided by packages is something we
really want to have
* dh_python[1] will try to avoid moving files to pyshared if
package supports only one Python version (f.e. the latest one)
* py{,3}compile will support -X/--exclude option
I maintain one module that uses site-packages to keep templates with
.py files inside and although I patched it to move these files outside
site-packages, I know that not every maintainer will want to do that,
so -X will disable byte compilation of files for given directory/file
* about lack of XS-Python-Version and debian/pyversions
- if available, both previous places will be used to get
minimum/maximum required Python version, it will complicate
detection of packages that need binNMU, so I want to drop both of
them at some point,
- dh/cdbs/dh_python will get minimum/maximum required Python versions
from Build-Depends{,-Indep} and/or python-all's Depends.
If one will need to build depend on newer version of
python{,-all,-dev} (due to some Debian specific changes) - tools
will ignore versioned dependencies that include dash sign or use the
lower one if two different dependencies are provided
(f.e. Build-Depends: python-all-dev (= 2.5.8), python-all-dev (= 2.4)
will be equivalent of XS-Python-Version: =2.4)
* how to detect which package needs binNMU?
I think parsing Build-Depends{,-Indep}, Contents file and Depends
header will be enough to detect such packages, i.e. packages that:
- Build-Depends{,-Indep} on python-all{,-dev}
AND there's no X.Y build dependency (where X.Y is the one to be
introduced)
- Build-Depends on python-dev (python-dev (=X.Y) | pythonX.Y-dev or
python-dev (X.Y) might help to detect some false positives)
AND provide private extension (/usr/lib/*/*.so)
- Build-Depends{,-Indep} on python (=X.Y) | pythonX.Y and Depend on
pythonX.Y
(i.e. python (=X.Y) | pythonX.Y will NOT be in Depends)
[arch:all packages with hardcoded shebang due to default python not
being good enough]
- Build-Depends{,-Indep} on python or python (=X.Y) | pythonX.Y
and there's no rtupdate script in binary package
[private modules without hardcoded shebang]
will need binNMU once new Python version will be added to the
supported ones
[1] or dh_pypython - I'm still not sure if using the same name is
a good idea as I want to write it in Python and maybe at some point
let someone rewrite it in Perl so that it could be included in
debhelper package.
[Piotr Ożarowski, 2009-08-02]
$ python3.1 ./3.x/setup.py install --root=debian/python3-foo/
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/__init__.py
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/_foo.so
debian/python3-foo/usr/lib/python3.1/dist-packages/foo/bar.py
will be debian/python3-foo/usr/local/lib... here by default, of course
[...]
# baz uses Python = 2.5 and has private extension
$ python ./setup.py install --root=debian/foo/
--install-lib=/usr/share/privatedir4/ --install-scripts=/usr/share/privatedir4
s,/usr/share/,/usr/lib/,g
debian/foo/usr/lib/privatedir4/baz/__init__.py
debian/foo/usr/lib/privatedir4/baz/baz.so
debian/foo/usr/lib/privatedir4/run.py
--
-=[ Piotr Ożarowski ]=-
-=[ http://www.ozarowski.pl ]=-
signature.asc
Description: Digital signature
new dh_python proposal
My attempt to merge python-central and python-support failed few months ago, so
here's another proposal to improve the Python situation in Debian. It's nothing
new
actually, just a compilation of previous Matthias' and Joss' ideas and few of my
own.
The main idea is to ship symlinks (not .pyc files!) in binary packages as
Matthias proposed back in February and thus get rid of both helper tools (most
problems with current tools occur at install / upgrade time). I want to keep it
as simple as possible and let maintainers customize it in maintainer or
rtinstall/rtupdate/rtremove scripts (if really needed).
Advantages:
* a lot less opportunities to break a system while installing / upgrading
Python packages,
* no more problems with packages that provide the same namespace and use
different helper tool,
* Python modules available out of the box (think about daemons),
* no more which tool should I use? or how do they differ? questions ;-)
The main disadvantage of this approach is that architecture independent packages
will have to be rebuilt once new Python version will be added to the supported
ones
(I think we can avoid a rebuild if a version is removed from the supported
ones).
Luk (our release manager) told me that binNMUs for arch:all packages will be
possible in Debian at some point, but it's a matter or months or even years, so
for now - manual NMUs will be needed. I think it will be easy to detect which
packages need one basing on Contents file and Depends field.
Another disadvantage is the fact that pysupport's namespace feature will not be
supported. If we want to keep it simple and let dpkg know about all the files -
new *-common package with all common __init__.py (and other if needed) files
will have to be provided, just like we did before pysupport 0.7. Otherwise
removing .pyc files will not be a trivial thing to do. Note that there are lots
of problems with this feature anyway (upstreams tend to use site-packages for
things that don't belong there), so removing it will improve the situation at
the end, IMHO.
I'll try to make dh_python a drop in replacement for dh_pycentral and
dh_pysupport (i.e. tools used at build time) for most packages, but I don't
consider it to be a main goal. Some packages will have to be manually updated
(f.e. the ones that use pysupport's namespace feature).
I want the tool to be team maintained, I didn't decide yet if new Alioth
project should be created or if we should use DPMT or PAPT repo for this,
though.
Short overview of how things will work with new dh_python:
==
build time:
^^^
* files installed into the standard location[1] or into private directory
* dh_python will:
- move .py files from SL[1] into /usr/share/py{,3}shared,
- move .so files to /usr/lib/pythonX.Y/*-packages if /usr/local was used,
- create symlinks for all py{,3}shared content in
/usr/lib/pythonX.Y/*-packages
*if* files are the same for all Python versions in SL[1] (if not, leave
original files) where X.Y are requested Python versions,
- create simple maintainer scripts with pycompile and pyclean commands
and (if needed) rtupdate script (for private directories that use default
Python version). Private modules that cannot be used with default Python
version will get additional pycompile command with all needed arguments,
like
minimum required Python version or hardcoded one in both, maintainer script
and rtupdate one
[1] SL = standard distutils/setuptools location:
/usr/{,local}/lib/pythonX.Y/{site,dist}-packages/
installation time:
^^
* maintainer script should bytecompile pyc files for all provided
symlinks / private directories if given Python version is installed
* user installs new pythonX.Y package:
- bytecompile related symlinks (pycompile -V X.Y)
* user removes pythonX.Y package:
- remove all pythonX.Y's .pyc files (pyclean -V X.Y)
- all packages with private directories that use this Python version will be
removed
by dpkg (Depends field will do its job)
* default Python version changes:
- rtupdate scripts for packages with private modules that support it (i.e.
the ones without hardcoded Python version and without private extensions)
invoked
examples:
=
example 1 - public modules/extensions only
^^
$ python2.4 ./setup.py install --root=debian/python-foo/
debian/python-foo/usr/lib/python2.4/site-packages/foo/__init__.py
debian/python-foo/usr/lib/python2.4/site-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.4/site-packages/foo/bar.py
debian/python-foo/usr/lib/python2.4/site-packages/spam.py
$ python2.5 ./setup.py install --root=debian/python-foo/
debian/python-foo/usr/lib/python2.5/site-packages/foo/__init__.py
debian/python-foo/usr/lib/python2.5/site-packages/foo/_foo.so
debian/python-foo/usr/lib/python2.5/site-packages/foo/bar.py
Re: Bug#370833: New dh_python proposal
Le mardi 13 juin 2006 à 15:23 -0400, Joey Hess a écrit : (FWIW, I began ignoring this issue when the politics and constant advocacy and pressure became too annoying to bother with, and I expect to ignore it for at least another couple of weeks. Unfortunatly that means that to avoid either trampeling over or blessing your NMU, I won't be able to release any other debhelper fixes in that timeframe either.) If it is really slowing things down this way, I will ask for the removal of python-support. Although it is a better and simpler solution than python-central, I don't want to delay the move to python2.4 several more month. We've already waited too much for Matthias to start moving. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: Bug#370833: New dh_python proposal
Joey Hess writes: I don't particularly mind that you've chosen to NMU debhelper. However, I can't guarantee that I will preserve the interfaces that you've added to dh_python in a backwards compatible way when I get around to looking at it. (FWIW, I began ignoring this issue when the politics and constant advocacy and pressure became too annoying to bother with, and I expect to ignore it for at least another couple of weeks. Unfortunatly that means that to avoid either trampeling over or blessing your NMU, I won't be able to release any other debhelper fixes in that timeframe either.) could you agree in temporarily/permanently splitting out dh_python from debhelper? Matthias -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Bug#370833: New dh_python proposal
On Mon, 12 Jun 2006, Raphael Hertzog wrote: I would love to have this fixed by today's dinstall. That's why I prepared the attached NMU, please just send an OK and I'll upload it (of course, feel free to do it yourself in a maintainer upload). Without any answer from you, I might upload it right before dinstall (depending on the opinion of other involved people). Just to avoid confusion, this would mean uploading between 18:00 and 19:00 UTC today (in about 9.5 hours). Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Bug#370833: New dh_python proposal
On Mon, 12 Jun 2006, Raphael Hertzog wrote: On Mon, 12 Jun 2006, Raphael Hertzog wrote: I would love to have this fixed by today's dinstall. That's why I prepared the attached NMU, please just send an OK and I'll upload it (of course, feel free to do it yourself in a maintainer upload). Without any answer from you, I might upload it right before dinstall (depending on the opinion of other involved people). Just to avoid confusion, this would mean uploading between 18:00 and 19:00 UTC today (in about 9.5 hours). And for extra-care, here's the package for testing: http://people.debian.org/~hertzog/python/ http://people.debian.org/~hertzog/python/debhelper_5.0.37.1_all.deb Please test and report me ASAP any problem with it. Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Bug#370833: New dh_python proposal
Coin, Raphael Hertzog [EMAIL PROTECTED] writes: Without any answer from you, I might upload it right before dinstall (depending on the opinion of other involved people). This is perfectly reasonnable. -- Marc Dequènes (Duck) pgpU3t0AM8OF7.pgp Description: PGP signature
Re: Bug#370833: New dh_python proposal
On Sun, 11 Jun 2006, Dafydd Harries wrote: Ar 10/06/2006 am 21:50, ysgrifennodd Raphael Hertzog: I have written dh_pycentral and dh_pysupport which will take care of the byte-compilation of the modules and integration with the respective tools. Those should be integrated in their respective packages (after review by their maintainer). Please no. Let's not have multiple solutions to the same problem. This wastes the time of each maintainer of a Python package who needs to work out whether they need to use dh_python, dh_pycentral, dh_pysupport or some combination of the three. That's why we have a policy document, to say what they need to do and how they can do it. If you want your modules to be byte-compiled you need to use either dh_pycentral or dh_pysupport. Then, you will always have to use dh_python to generate the right dependencies/provides/Python-Version field. It really isn't *that* complicated. We are on the verge of having a greatly improved Python policy, having that policy implemented, and finally having Python 2.4 as default in Debian. This reluctance to settle on a standard way of installing modules is slowing our progress on all three fronts, and release time draws ever nearer. What is slowing our progress is the refusal to let the time decide between python-central and/or python-support. There's no consensus here and if you need to make a choice, your only possibility is to request the technical committee to make a choice. And the technical committee needs time as well. Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Bug#370833: New dh_python proposal
* Raphael Hertzog ([EMAIL PROTECTED]) [060611 10:00]: What is slowing our progress is the refusal to let the time decide between python-central and/or python-support. There's no consensus here and if you need to make a choice, your only possibility is to request the technical committee to make a choice. And the technical committee needs time as well. Especially as I don't see the good reason why we cannot have two competing implementations. Cheers, Andi -- http://home.arcor.de/andreas-barth/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Bug#370833: New dh_python proposal
On Fri, 09 Jun 2006, Joey Hess wrote: Raphael Hertzog wrote: Because Matthias and Josselin do not (or can't or don't want to) work together. I tried my best to fill the gap. :-/ I appreciate your work, but I am not comfortable supporting two competing implementations in debhelper. Please reconsider. If you don't integrate that version now, we'll most probably do the switch to python2.4 in the next days and we'll replace dh_python calls by dh_python2 calls (integrated for example in the python package) or we'll remove call to dh_python in favor of dh_pycentral (existing and in python-central) / dh_pythonsupport (not existing but quick to create). And if we go that way, then we loose the possibility to move easily to a single unified helper tool in the future since all packages would have again to be modified to reuse dh_python. Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Bug#370833: New dh_python proposal
On Sat, 2006-06-10 at 09:08 +0200, Raphael Hertzog wrote: On Fri, 09 Jun 2006, Joey Hess wrote: Raphael Hertzog wrote: Because Matthias and Josselin do not (or can't or don't want to) work together. I tried my best to fill the gap. :-/ I appreciate your work, but I am not comfortable supporting two competing implementations in debhelper. Please reconsider. If you don't integrate that version now, we'll most probably do the switch to python2.4 in the next days and we'll replace dh_python calls by dh_python2 calls (integrated for example in the python package) or we'll remove call to dh_python in favor of dh_pycentral (existing and in python-central) / dh_pythonsupport (not existing but quick to create). Please don't prefix non-debhelper commands with dh_ if you do this. Call it deb_pysupport/deb_pycentral or something, and likewise please don't mess with #DEBHELPER#. Stomping all over the namespace because you were rejected from the package per se seems to miss Joey's point. -- Joe Wreschnig [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: Bug#370833: New dh_python proposal
On Sat, 10 Jun 2006, Raphael Hertzog wrote:
I appreciate your work, but I am not comfortable supporting two
competing implementations in debhelper.
Please reconsider. If you don't integrate that version now, we'll most
probably do the switch to python2.4 in the next days and we'll replace
dh_python calls by dh_python2 calls (integrated for example in the python
package) or we'll remove call to dh_python in favor of dh_pycentral (existing
and in python-central) / dh_pythonsupport (not existing but quick to
create).
After discussing with doko and vorlon, I decided to change dh_python to
respect your wish (it's understandable).
Please find a new dh_python attached. This one supports neither
python-central nor python-support. It does however all the substvars
handling required by the new Python policy and is still compatible with
the old policy.
Please integrate that one.
I have written dh_pycentral and dh_pysupport which will take care of the
byte-compilation of the modules and integration with the respective tools.
Those should be integrated in their respective packages (after review by
their maintainer).
Cheers,
--
Raphaël Hertzog
Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/
#!/usr/bin/perl -w
=head1 NAME
dh_python - calculates python dependencies and adds postinst and prerm python
scripts
=cut
use strict;
use File::Find;
use Debian::Debhelper::Dh_Lib;
=head1 SYNOPSIS
Bdh_python [SIdebhelper options] [B-n] [B-V Iversion] [SImodule
dirs ...]
=head1 DESCRIPTION
dh_python is a debhelper program that is responsible for generating the
${python:Depends} substitutions and adding them to substvars files. It
will also add a postinst and a prerm script if required.
The program will look at python scripts and modules in your package, and
will use this information to generate adequate dependencies. There is two
scenarios: if the package uses the XS-Python-Version field then
the new policy will be applied, otherwise the old policy will be used.
=head2 New policy
The XS-Python-Version field (on the source package) defines which Python
versions are supported by the package. It can be all, current,
current, = X.Y, a single version (2.3) or a list of versions with
optional comparison operators (ex: 2.3, 2.4 or = 2.2, 2.5 or
= 2.4).
The binary packages should have a XB-Python-Version: ${python:Versions}
field and dh_python will generate the right substvar for that. The
resulting value can be all for public modules which work with all python
versions, current for private modules which are always byte-compiled
with the current python version or a list of of all versions for which the
extensions have been compiled (ex: 2.3, 2.4). The dependencies are
adjusted accordingly as well.
Packages with public extensions should also have a Provides:
${python:Provides} field. The corresponding substvar will indicate
pythonX.Y-foo, pythonX.Z-foo according to all the extensions
effectively available in the package.
=head2 Old policy
It looks at scripts and modules in your package and adds a dependency on
python, with the
current major version, or on pythonX.Y if your scripts or modules need a
specific python version. The dependency will be substituted into your
package's control file wherever you place the token ${python:Depends}.
If some modules need to be byte-compiled at install time, appropriate
postinst and prerm scripts will be generated. If already byte-compiled
modules are found, they are removed.
If you use this program, your package should build-depend on python.
Note: in the old policy, /usr/lib/site-python is also scanned for modules.
=head1 OPTIONS
=over 4
=item Imodule dirs
If your package installs python modules in non-standard directories, you
can make dh_python check those directories by passing their names on the
command line. By default, it will check /usr/lib/$PACKAGE, /usr/share/$PACKAGE,
/usr/lib/games/$PACKAGE,
/usr/share/games/$PACKAGE and /usr/lib/python?.?/site-packages.
Note: only /usr/lib/python?.?/site-packages and the
extra names on the command line are searched for binary (.so) modules.
=item B-V Iversion
If the .py files your package ships are meant to be used by a specific
pythonX.Y version, you can use this option to specify the desired version,
such as 2.3. Do not use if you ship modules in /usr/lib/site-python.
With the new policy, this option is mostly deprecated. Use the
XS-Python-Field to indicate that you're using a specific python version.
=item B-n, B--noscripts
Do not modify postinst/postrm scripts.
=back
=head1 CONFORMS TO
Debian policy, version 3.5.7
Python policy, version 0.3.7
=cut
init();
my $python = 'python';
# The current python major version
my $python_major;
my $python_version = `$python -V 21`;
if (! defined $python_version || $python_version eq ) {
error(Python is not installed, aborting. (Probably forgot to
Build-Depend on python.));
}
elsif ($python_version =~
Re: Bug#370833: New dh_python proposal
Ar 10/06/2006 am 21:50, ysgrifennodd Raphael Hertzog: I have written dh_pycentral and dh_pysupport which will take care of the byte-compilation of the modules and integration with the respective tools. Those should be integrated in their respective packages (after review by their maintainer). Please no. Let's not have multiple solutions to the same problem. This wastes the time of each maintainer of a Python package who needs to work out whether they need to use dh_python, dh_pycentral, dh_pysupport or some combination of the three. We are on the verge of having a greatly improved Python policy, having that policy implemented, and finally having Python 2.4 as default in Debian. This reluctance to settle on a standard way of installing modules is slowing our progress on all three fronts, and release time draws ever nearer. -- Dafydd -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: New dh_python proposal
On Fri, 09 Jun 2006, Raphael Hertzog wrote:
This dh_python depends neither on python-central nor on python-support but it
can work with both.
It will use python-central if it appears in a build-dependency.
It will use python-support if it detects /usr/share/python-support/$package.
If the XS-Python-Version field is present, it will work following the new
policy otherwise it should follow the old policy.
At least that's the theory. I did some tests on two recent packages with
python-central and python-support. I'd like people to try to recompile
some actual python-related packages modules with that dh_python and check
if it's really enough backwards compatible.
OK, I did test it myself finally this morning (with regular feedback from
Matthias as well). It leads me to some corrections so please find an
updated (and final!) dh_python in attachment.
Matthias and Josselin, please say clearly here in the bug report that
you are OK with this dh_python implementation so that Joey can confidently
integrate it in debhelper RSN.
[ Joey Hess ]
So I'm confused, why do we have competing implementations here?
Because Matthias and Josselin do not (or can't or don't want to) work
together. I tried my best to fill the gap. :-/
FWIW, my opinion is that the BoF at Debconf clearly decided of a new
python policy but didn't explicitely choose an implementation. At that
time, only python-support was working and integrated in unstable.
We all knew doko's ambitious plan with python-central and since we didn't
discuss the implementation, there was a kind of implicit agreement that we
would use whatever would be ready in time for the python2.4 switch
(with many people convincend that python-central wouldn't be ready in
time).
Now, we have two working python helper tool but python-central has
not yet been widely tested.
If you check the code you'll see that the python-(support|central)
specific code is very limited and mainly consists on adding postinst
script snippet and dependencies when needed. I tried my best to have a
dh_python which is agnostic but still effective. :-)
I hope you understand better the situation and I really hope that you'll
accept this dh_python so that we can effectively start moving to python2.4
by default RSN.
Cheers,
--
Raphaël Hertzog
Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/
--- dh_python.orig 2006-06-08 10:57:59.0 +0200
+++ dh_python 2006-06-09 12:45:52.0 +0200
@@ -21,7 +21,42 @@
will also add a postinst and a prerm script if required.
The program will look at python scripts and modules in your package, and
-will use this information to generate a dependency on python, with the
+will use this information to generate adequate dependencies. There is two
+scenarios: if the package uses the XS-Python-Version field then
+the new policy will be applied, otherwise the old policy will be used.
+
+If dh_python detects python-central in one of the build-dependencies,
+then it will automatically use it: it will be added in the dependencies
+and corresponding postinst/postrm scripts are generated.
+
+If dh_python detects that you have a directory
+/usr/share/python-support/$PACKAGE, then it will automatically generate the
+required postinst/postrm scripts.
+
+=head2 New policy
+
+The XS-Python-Version field (on the source package) defines which Python
+versions are supported by the package. It can be all, current,
+current, = X.Y, a single version (2.3) or a list of versions with
+optional comparison operators (ex: 2.3, 2.4 or = 2.2, 2.5 or
+= 2.4).
+
+The binary packages should have a XB-Python-Version: ${python:Versions}
+field and dh_python will generate the right substvar for that. The
+resulting value can be all for public modules which work with all python
+versions, current for private modules which are always byte-compiled
+with the current python version or a list of of all versions for which the
+extensions have been compiled (ex: 2.3, 2.4). The dependencies are
+adjusted accordingly as well.
+
+Packages with public extensions should also have a Provides:
+${python:Provides} field. The corresponding substvar will indicate
+pythonX.Y-foo, pythonX.Z-foo according to all the extensions
+effectively available in the package.
+
+=head2 Old policy
+
+It looks at scripts and modules in your package and adds a dependency on
python, with the
current major version, or on pythonX.Y if your scripts or modules need a
specific python version. The dependency will be substituted into your
package's control file wherever you place the token ${python:Depends}.
@@ -32,6 +67,8 @@
If you use this program, your package should build-depend on python.
+Note: in the old policy, /usr/lib/site-python is also scanned for modules.
+
=head1 OPTIONS
=over 4
@@ -40,11 +77,10 @@
If your package installs python modules in non-standard directories, you
can make dh_python check those directories by passing their names on
Re: New dh_python proposal
Raphael Hertzog writes: On Fri, 09 Jun 2006, Raphael Hertzog wrote: This dh_python depends neither on python-central nor on python-support but it can work with both. It will use python-central if it appears in a build-dependency. It will use python-support if it detects /usr/share/python-support/$package. If the XS-Python-Version field is present, it will work following the new policy otherwise it should follow the old policy. At least that's the theory. I did some tests on two recent packages with python-central and python-support. I'd like people to try to recompile some actual python-related packages modules with that dh_python and check if it's really enough backwards compatible. OK, I did test it myself finally this morning (with regular feedback from Matthias as well). It leads me to some corrections so please find an updated (and final!) dh_python in attachment. Matthias and Josselin, please say clearly here in the bug report that you are OK with this dh_python implementation so that Joey can confidently integrate it in debhelper RSN. checked and tested together with Raphael. Looks OK for me. Matthias -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: New dh_python proposal
Le vendredi 09 juin 2006 à 14:07 +0200, Raphael Hertzog a écrit : OK, I did test it myself finally this morning (with regular feedback from Matthias as well). It leads me to some corrections so please find an updated (and final!) dh_python in attachment. Matthias and Josselin, please say clearly here in the bug report that you are OK with this dh_python implementation so that Joey can confidently integrate it in debhelper RSN. I haven't had the time to test this implementation in complicated cases, but I have read it extensively and I believe it does what it needs to do. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom
Re: Bug#370833: New dh_python proposal
Raphael Hertzog wrote: Because Matthias and Josselin do not (or can't or don't want to) work together. I tried my best to fill the gap. :-/ I appreciate your work, but I am not comfortable supporting two competing implementations in debhelper. Sorry. -- see shy jo signature.asc Description: Digital signature
New dh_python proposal
On Thu, 08 Jun 2006, Raphael Hertzog wrote:
This dh_python will use python-central for any binary package with a
XB-Python-Version: field. There's no optional python-support either.
So anyone unhappy with python-central should really come up with another
dh_python implementation RSN.
OK I discussed with Josselin and Matthias and came up with the attached
dh_python (patch attached as well).
This dh_python depends neither on python-central nor on python-support but it
can work with both.
It will use python-central if it appears in a build-dependency.
It will use python-support if it detects /usr/share/python-support/$package.
If the XS-Python-Version field is present, it will work following the new
policy otherwise it should follow the old policy.
At least that's the theory. I did some tests on two recent packages with
python-central and python-support. I'd like people to try to recompile
some actual python-related packages modules with that dh_python and check
if it's really enough backwards compatible.
I think we'll be able to move forward when this dh_python is integrated in
debhelper.
Cheers,
--
Raphaël Hertzog
Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/
--- dh_python.orig 2006-06-08 10:57:59.0 +0200
+++ dh_python 2006-06-09 04:07:56.0 +0200
@@ -21,7 +21,42 @@
will also add a postinst and a prerm script if required.
The program will look at python scripts and modules in your package, and
-will use this information to generate a dependency on python, with the
+will use this information to generate adequate dependencies. There is two
+scenarios: if the package uses the XS-Python-Version field then
+the new policy will be applied, otherwise the old policy will be used.
+
+If dh_python detects python-central in one of the build-dependencies,
+then it will automatically use it: it will be added in the dependencies
+and corresponding postinst/postrm scripts are generated.
+
+If dh_python detects that you have a directory
+/usr/share/python-support/$PACKAGE, then it will automatically generate the
+required postinst/postrm scripts.
+
+=head2 New policy
+
+The XS-Python-Version field (on the source package) defines which Python
+versions are supported by the package. It can be all, current,
+current, = X.Y, a single version (2.3) or a list of versions with
+optional comparison operators (ex: 2.3, 2.4 or = 2.2, 2.5 or
+= 2.4).
+
+The binary packages should have a XB-Python-Version: ${python:Versions}
+field and dh_python will generate the right substvar for that. The
+resulting value can be all for public modules which work with all python
+versions, current for private modules which are always byte-compiled
+with the current python version or a list of of all versions for which the
+extensions have been compiled (ex: 2.3, 2.4). The dependencies are
+adjusted accordingly as well.
+
+Packages with public extensions should also have a Provides:
+${python:Provides} field. The corresponding substvar will indicate
+pythonX.Y-foo, pythonX.Z-foo according to all the extensions
+effectively available in the package.
+
+=head2 Old policy
+
+It looks at scripts and modules in your package and adds a dependency on
python, with the
current major version, or on pythonX.Y if your scripts or modules need a
specific python version. The dependency will be substituted into your
package's control file wherever you place the token ${python:Depends}.
@@ -32,6 +67,8 @@
If you use this program, your package should build-depend on python.
+Note: in the old policy, /usr/lib/site-python is also scanned for modules.
+
=head1 OPTIONS
=over 4
@@ -40,11 +77,10 @@
If your package installs python modules in non-standard directories, you
can make dh_python check those directories by passing their names on the
-command line. By default, it will check /usr/lib/site-python,
-/usr/lib/$PACKAGE, /usr/share/$PACKAGE, /usr/lib/games/$PACKAGE,
+command line. By default, it will check /usr/lib/$PACKAGE,
/usr/share/$PACKAGE, /usr/lib/games/$PACKAGE,
/usr/share/games/$PACKAGE and /usr/lib/python?.?/site-packages.
-Note: only /usr/lib/site-python, /usr/lib/python?.?/site-packages and the
+Note: only /usr/lib/python?.?/site-packages and the
extra names on the command line are searched for binary (.so) modules.
=item B-V Iversion
@@ -53,6 +89,9 @@
pythonX.Y version, you can use this option to specify the desired version,
such as 2.3. Do not use if you ship modules in /usr/lib/site-python.
+With the new policy, this option is mostly deprecated. Use the
+XS-Python-Field to indicate that you're using a specific python version.
+
=item B-n, B--noscripts
Do not modify postinst/postrm scripts.
@@ -85,10 +124,10 @@
}
# The next python version
-my $python_nextversion = $python_version + 0.1;
+my $python_nextversion = next_minor_version($python_version);
my $python_nextmajor = $python_major + 1;
-my @python_allversions =
