On Sat, 17 Jun 2006, Raphael Hertzog wrote: > > Please find it attached: > > - it's vastly refactored, it's easier to understand, and the difference > > between old policy and new policy is easier to see > > - it supports an -X option which can be used to not scan some files > > - it fixes the bug 373853 (uses XS-Python-Version: value instead of "all" > > when the modules are shared) > > I forgot: > - it correctly puts a python dependency if the package has only shared > python modules (via python-central / python-support) without shebang
And while doing NMU of various python packages I found another little problem. A package providing modules which work only with python2.4 would get a dependency "python (>= 2.4)" even if the current version is still 2.3, resulting in an uninstallable package. An updated dh_python is attached and I also updated the debhelper package on my site: http://people.debian.org/~hertzog/python/debhelper_5.0.37.2_all.deb BTW, since I worked quite a lot in dh_python recently, there's a bzr branch available with my successive changes: bzr branch http://ouaza.com/~rhertzog/dh-python/ (Not always very useful as I did rather big commits) Cheers, -- Raphaël Hertzog Premier livre français sur Debian GNU/Linux : http://www.ouaza.com/livre/admin-debian/
#!/usr/bin/perl -w =head1 NAME dh_python - calculates python dependencies and adds postinst and prerm python scripts =cut use strict; use File::Find; use Debian::Debhelper::Dh_Lib; =head1 SYNOPSIS B<dh_python> [S<I<debhelper options>>] [B<-n>] [B<-X>I<item>] [B<-V> I<version>] [S<I<module dirs ...>>] =head1 DESCRIPTION dh_python is a debhelper program that is responsible for generating the ${python:Depends} substitutions and adding them to substvars files. It will also add a postinst and a prerm script if required. The program will look at python scripts and modules in your package, and will use this information to generate adequate dependencies. There is two scenarios: if the package uses the XS-Python-Version field then the new policy will be applied, otherwise the old policy will be used. =head2 New policy The XS-Python-Version field (on the source package) defines which Python versions are supported by the package. It can be "all", "current", "current, >= X.Y", a single version ("2.3") or a list of versions with optional comparison operators (ex: "2.3, 2.4" or ">= 2.2, << 2.5" or ">= 2.4"). The binary packages should have a "XB-Python-Version: ${python:Versions}" field and dh_python will generate the right substvar for that. The resulting value can be "all" for public modules which work with all python versions, "current" for private modules which are always byte-compiled with the current python version or a list of of all versions for which the extensions have been compiled (ex: "2.3, 2.4"). The dependencies are adjusted accordingly as well. Packages with public extensions should also have a "Provides: ${python:Provides}" field. The corresponding substvar will indicate "pythonX.Y-foo, pythonX.Z-foo" according to all the extensions effectively available in the package. =head2 Old policy It looks at scripts and modules in your package and adds a dependency on python, with the current major version, or on pythonX.Y if your scripts or modules need a specific python version. The dependency will be substituted into your package's control file wherever you place the token "${python:Depends}". If some modules need to be byte-compiled at install time, appropriate postinst and prerm scripts will be generated. If already byte-compiled modules are found, they are removed. If you use this program, your package should build-depend on python. Note: in the old policy, /usr/lib/site-python is also scanned for modules. =head1 OPTIONS =over 4 =item I<module dirs> If your package installs python modules in non-standard directories, you can make dh_python check those directories by passing their names on the command line. By default, it will check /usr/lib/$PACKAGE, /usr/share/$PACKAGE, /usr/lib/games/$PACKAGE, /usr/share/games/$PACKAGE and /usr/lib/python?.?/site-packages. Note: only /usr/lib/python?.?/site-packages and the extra names on the command line are searched for binary (.so) modules. =item B<-V> I<version> If the .py files your package ships are meant to be used by a specific pythonX.Y version, you can use this option to specify the desired version, such as 2.3. Do not use if you ship modules in /usr/lib/site-python. With the new policy, this option is mostly deprecated. Use the XS-Python-Field to indicate that you're using a specific python version. =item B<-n>, B<--noscripts> Do not modify postinst/postrm scripts. =item B<-X>I<item>, B<--exclude=>I<item> Exclude files that contain "item" anywhere in their filename from being taken into accout to generate the python dependency. You may use this option multiple times to build up a list of things to exclude. =back =head1 CONFORMS TO Debian policy, version 3.5.7 Python policy, version 0.3.7 =cut init(); my $python = 'python'; # The current python major version my $python_major; my $python_version = `$python -V 2>&1`; if (! defined $python_version || $python_version eq "") { error("Python is not installed, aborting. (Probably forgot to Build-Depend on python.)"); } elsif ($python_version =~ m/^Python\s+(\d+)\.(\d+)(\.\d+)*/) { $python_version = "$1.$2" ; $python_major = $1 ; } else { error("Unable to parse python version out of \"$python_version\"."); } # The next python version my $python_nextversion = next_minor_version($python_version); my $python_nextmajor = $python_major + 1; my @python_allversions = ('1.5','2.1','2.2','2.3','2.4','2.5'); foreach (@python_allversions) { s/^/python/; } # Check for -V my $usepython = "python$python_version"; if($dh{V_FLAG_SET}) { $usepython = $dh{V_FLAG}; $usepython =~ s/^/python/; if (! grep { $_ eq $usepython } @python_allversions) { error("Unknown python version $dh{V_FLAG}"); } } # Cleaning the paths given on the command line foreach (@ARGV) { s#/$##; s#^/##; } # Check if we have an XS-Python-Version and extract it from the control # file, also check if we use python-central my $python_header = ""; { local $/ = ""; # Grab until empty line open(CONTROL, "debian/control"); # Can't fail, dh_testdir has already been called my $source = <CONTROL>; $python_header = $1 if ($source =~ m/^XS-Python-Version: \s*(.*)$/m); chomp($python_header); } # dependency types use constant PROGRAM => 1; use constant PY_PRIVATE_MODULE => 2; use constant PY_PUBLIC_MODULE => 4; use constant PY_OFFICIAL_MODULE => 8; use constant PY_UNKNOWN_MODULE => 16; use constant SO_PRIVATE_MODULE => 32; use constant SO_PUBLIC_MODULE => 64; use constant SO_OFFICIAL_MODULE => 128; use constant SO_UNKNOWN_MODULE => 256; foreach my $package (@{$dh{DOPACKAGES}}) { my $tmp = tmpdir($package); my @dirs = ("usr/lib/$package", "usr/share/$package", "usr/lib/games/$package", "usr/share/games/$package", @ARGV ); my @dirs_so = (@ARGV); my $dep_on_python = 0; my $strong_dep = 0; # Fail early if the package use usr/lib/site-python if (-d "$tmp/usr/lib/site-python") { if ($python_header) { # Error only on new policy error("The package $package puts files in /usr/lib/site-python: forbidden by policy"); } else { # Old policy allowed that directory, so scan it push @dirs, "usr/lib/site-python"; push @dirs_so, "usr/lib/site-python"; } } @dirs = grep -d, map "$tmp/$_", @dirs; @dirs_so = grep -d, map "$tmp/$_", @dirs_so; my $deps = 0; my %verdeps = (); foreach (@python_allversions) { $verdeps{$_} = 0; } # Global scan my $private_pydirs_regex = join('|', map { "\Q$_\E" } @dirs); my $private_sodirs_regex = join('|', map { "\Q$_\E" } @dirs_so); my %private_dirs_list; my %pyversions_found; find sub { return unless -f; # See if we were asked to exclude this file. # Note that we have to test on the full filename, including directory. my $fn="$File::Find::dir/$_"; if (excludefile($fn)) { verbose_print("Ignoring $fn"); return; } # Find scripts if (-x or /\.py$/) { local *F; return unless open F, $_; if (read F, local $_, 32 and m%^#!\s*/usr/bin/(env\s+)?(python(\d+\.\d+)?)\s%) { if ( "python" eq $2 ) { $deps |= PROGRAM; } elsif(defined $verdeps{$2}) { $verdeps{$2} |= PROGRAM; } } close F; } # Continue only with .py or .so return unless (/\.py$/ or /\.so$/); # Remove any byte-compiled file doit(("rm","-f",$_."c",$_."o")) if /\.py$/; # Classify the file in the right category if (/\.py$/ and $private_pydirs_regex and $fn =~ m/(?:$private_pydirs_regex)/) { # Private python module verbose_print("Found private module: $fn"); my $dir; foreach $dir (@dirs) { if ($fn =~ m/\Q$dir\E/) { $dir =~ s/^$tmp//; verbose_print("Memorizing dir to byte-compile: $dir"); $private_dirs_list{"$dir"} = 1; } } if ($dh{V_FLAG_SET}) { $verdeps{$usepython} |= PY_PRIVATE_MODULE; } else { $deps |= PY_PRIVATE_MODULE; } } elsif (/\.so$/ and $private_sodirs_regex and $fn =~ m/(?:$private_sodirs_regex)/) { # Private python extension verbose_print("Found private extension: $fn"); if ($dh{V_FLAG_SET}) { $verdeps{$usepython} |= SO_PRIVATE_MODULE; } else { $deps |= SO_PRIVATE_MODULE; } } elsif ($fn =~ m|$tmp/usr/lib/python([\d\.]+)/site-packages/|) { $pyversions_found{$1} = 1; my $v = $1; if (/\.py$/) { verbose_print("Found public module: $fn"); $verdeps{"python$v"} |= PY_PUBLIC_MODULE; } else { verbose_print("Found public extension: $fn"); $verdeps{"python$v"} |= SO_PUBLIC_MODULE; } } elsif ($fn =~ m|$tmp/usr/lib/python([\d\.]+)/|) { $pyversions_found{$1} = 1; my $v = $1; if (/\.py$/) { verbose_print("Found official module: $fn"); $verdeps{"python$v"} |= PY_OFFICIAL_MODULE; } else { verbose_print("Found official extension: $fn"); $verdeps{"python$v"} |= SO_OFFICIAL_MODULE; } } elsif ($fn =~ m{$tmp(?:/usr/share/pycentral/|/usr/share/python-support/)}) { if (/\.py$/) { verbose_print("Found public module: $fn"); $deps |= PY_PUBLIC_MODULE; } # No extensions here } elsif ($fn =~ m|$tmp/usr/share/doc/|) { # Ignore .py files in doc directory } else { # Unknown pyfiles if (/\.py$/) { verbose_print("Found unclassified module: $fn"); if ($dh{V_FLAG_SET}) { $verdeps{$usepython} |= PY_UNKNOWN_MODULE; } else { $deps |= PY_UNKNOWN_MODULE; } } else { verbose_print("Found unclassified extension: $fn"); if ($dh{V_FLAG_SET}) { $verdeps{$usepython} |= SO_UNKNOWN_MODULE; } else { $deps |= SO_UNKNOWN_MODULE; } } } }, $tmp; # Common dependency handling foreach my $pyver (keys %verdeps) { # Always add pythonX.Y dependency if a script uses that interpreter if ($verdeps{$pyver} & PROGRAM) { addsubstvar($package, "python:Depends", $pyver); } # Always add pythonX.Y dependency if some private modules are # byte-compiled with it (or if extensions are # byte-compiled with it) if ($verdeps{$pyver} & (PY_PRIVATE_MODULE|SO_PRIVATE_MODULE)) { addsubstvar($package, "python:Depends", $pyver); } } if ($python_header) { # # NEW POLICY # # Generate the depends to accept all python # versions that this package effectively provides my $min_version = ""; my $stop_version = ""; my $versions_field = ""; # Reset again, analysis using new policy follows $dep_on_python = 0; # Extracting min version from XS-Python-version if ($python_header =~ /^current(?:,\s*>=\s*([\d\.]+))?/) { if (defined $1) { $min_version = $1; } $versions_field = "current"; } elsif ($python_header =~ />=\s*([\d\.]+)/) { $min_version = $1; } if ($python_header =~ /<<\s*([\d\.]+)/) { $stop_version = $1; } # Private extensions, must be rebuilt for each python version if ($deps & SO_PRIVATE_MODULE) { $dep_on_python++; $stop_version = next_minor_version($python_version); # Packages using a private extension can only # support one version and they indicate which one # in XS-Python-Version if ($versions_field eq "current") { $versions_field = $python_version; } else { $versions_field = $python_header; } } # Private modules if ($deps & PY_PRIVATE_MODULE) { # Package with private modules can only support one version at a time # (even if the modules can be automatically byte-compiled for any new # python version). unless ($versions_field) { # If XS-Python-Version indicated "current" we would already have a value, # since we have not, XS-Python-Version must contain a hardcoded version. if ($python_header =~ /^[\d\.]+$/) { $versions_field = $python_header; } else { warning("The package contains private modules. It should indicate 'current'\n". "or a harcoded python version (ex: '2.4') in the XS-Python-Version field.\n") } } $versions_field = "current" unless $versions_field; } # Python scripts & public modules if ($deps & (PROGRAM|PY_PUBLIC_MODULE)) { $dep_on_python++; } # Public extensions if (scalar keys %pyversions_found) { # Extensions will always trigger this (as well as public # modules not handled by python-support/python-central) $dep_on_python++; if (scalar grep { $python_version eq $_ } keys %pyversions_found) { # Current versions is supported by the package # It's safe to depends on the corresponding # interval of python versions $min_version = min(keys %pyversions_found); unless ($stop_version) { my $max_version = max(keys %pyversions_found); $stop_version = next_minor_version($max_version); } } # Generate the Python-Version field foreach (keys %pyversions_found) { addsubstvar($package, "python:Versions", $_); } # Generate provides for the python2.X-foo packages that we emulate if ($package =~ /^python-/) { foreach (keys %pyversions_found) { my $virtual = $package; $virtual =~ s/^python-/$python$_-/; addsubstvar($package, "python:Provides", $virtual); } } } else { # Still try to describe the versions that the package support $versions_field = $python_header unless $versions_field; addsubstvar($package, "python:Versions", $versions_field); } if ($dep_on_python) { # At least a script has been detected if ($min_version) { if (compare_version($min_version, $python_version) <= 0 ) { # Min-version is less or equal to current version addsubstvar($package, "python:Depends", $python, ">= $min_version"); } else { # Min version is greater to current version # Supposition: new stuff working only with the latest python, # depend on that specific python version addsubstvar($package, "python:Depends", "python$min_version"); } } # If a stronger dependency is needed if ($stop_version) { if (compare_version($stop_version, $python_version) > 0 ) { # Stop version is strictly bigger than current version addsubstvar($package, "python:Depends", $python, "<< $stop_version"); } else { # Only works with old versions, # package is mostly deprecated, # no need for a python dependency } } # Let's depend on python anyway addsubstvar($package, "python:Depends", $python) unless ($min_version or $stop_version); } } else { # # OLD POLICY # # First, the case of python-foo and pythonX.Y-foo if ($package =~ /^python-/) { $dep_on_python = 1; $strong_dep = 1; # This adds a dependency to python<current>-foo in python-foo my $pack = $package; $pack =~ s/^python/python$python_version/; if (grep { "$_" eq "$pack" } getpackages()) { addsubstvar($package, "python:Depends", $pack); } } # Dependencies on current python $dep_on_python = 1 if ($deps & (PROGRAM|PY_PUBLIC_MODULE|SO_PUBLIC_MODULE|PY_PRIVATE_MODULE|SO_PRIVATE_MODULE)); $strong_dep = 1 if ($deps & (PY_PUBLIC_MODULE|SO_PUBLIC_MODULE)); # Add python dependencies if ($dep_on_python) { addsubstvar($package, "python:Depends", $python, ">= $python_version"); if ($strong_dep) { addsubstvar($package, "python:Depends", $python, "<< $python_nextversion"); } else { addsubstvar($package, "python:Depends", $python, "<< $python_nextmajor"); } } # Add postinst/preinst my $need_prerm = 0; foreach my $pyver (@python_allversions) { my $pydir="/usr/lib/$pyver/site-packages"; my @dirlist = (); # List of directories to byte-compile # If we have public modules, byte-compile /usr/lib/pythonX.Y/site-packages/ if ($verdeps{$pyver} & PY_PUBLIC_MODULE) { push @dirlist, $pydir; } # Byte-compile private modules with current python # (or the one indicated by -V) if (($pyver eq "$usepython") and scalar(keys %private_dirs_list)) { push @dirlist, keys %private_dirs_list; } # If we have something to byte-compile, add the # corresponding postinst snippet if (scalar(@dirlist) && ! $dh{NOSCRIPTS}) { autoscript($package,"postinst","postinst-python","s%#PYVER#%$pyver%;[EMAIL PROTECTED]"); $need_prerm = 1; } } if ($need_prerm && ! $dh{NOSCRIPTS}) { autoscript($package,"prerm","prerm-python","s%#PACKAGE#%$package%"); } } } sub next_minor_version { my $version = shift; # Handles 2.10 -> 2.11 gracefully my @items = split(/\./, $version); $items[1] += 1; $version = join(".", @items); return $version; } sub compare_version { my ($a, $b) = @_; my @A = split(/\./, $a); my @B = split(/\./, $b); my $diff = 0; for (my $i = 0; $i <= $#A; $i++) { $diff = $A[$i] - $B[$i]; return $diff if $diff; } # They are the same return 0; } sub max { my $max = shift; foreach (@_) { $max = $_ if (compare_version($_, $max) > 0); } return $max; } sub min { my $min = shift; foreach (@_) { $min = $_ if (compare_version($_, $min) < 0); } return $min; } =head1 SEE ALSO L<debhelper(7)> This program is a part of debhelper. =head1 AUTHORS Josselin Mouette <[EMAIL PROTECTED]> Raphael Hertzog <[EMAIL PROTECTED]> most ideas stolen from Brendan O'Dea <[EMAIL PROTECTED]> =cut