Ian Jackson writes ("dgit cannot cope with some patches because it uses gbp-pq"): > gbp pq cannot cope with all patches that dpkg-source can, because it > uses git-apply and git-apply is not very compatible. > > At the moment, this means that `dgit clone glibc' fails: ... > I think I need to switch to using repeated dpkg-source --before-build. > This means I need to reimplement gbp pq's metadata processing > algorithm :-/.
You are not going to believe what I have done instead. I suggest you sit down before you read what follows. Ian. >From 05db7ab051736bb106467e7f83b9e45684dd227c Mon Sep 17 00:00:00 2001 From: Ian Jackson <ijack...@chiark.greenend.org.uk> Date: Mon, 24 Oct 2016 01:06:29 +0100 Subject: [PATCH] Import: Use absurd `git apply' emulation if gbp pq import fails gbp import can fail due to git apply not understanding patches. This is #841867 (against dgit). The underlying problem is #841866 (in gbp pq) which exposes things like #841865 and #829067 (in git). I imagine there are other lurking incompatibilities between git-apply and dpkg-source. We could in principle reimplement the gbp patch metadata extraction. But that would be quite tiresome and have its own compatibility problems. The real problem is just `git apply'. (Indeed gbp already tries git apply without, and then with, a whitespace fix option.) We work around the trouble by providing our own implementation of `git apply'. Specifically: We try to do things the sane way (just running gbp pq import) first. If that works, great. If it doesn't, we put /usr/share/dgit/absurd on the PATH. That contains only a sh script called `git'. This sh script figures out whether the caller is trying to invoke `git apply'. If not, it runs the real git. If the caller wanted git-apply, the absurd git script emulates it using dpkg-source --before-build. Conveniently, we know that the series file will not be touched by patches. So we can write just the patch we care about into the series file, and run --before-build, which applies just that one patch. The results are committed (minus the .pc), and for the next patch, dpkg-source sees again a tree with simply a single patch to apply. We try ordinary gbp pq first because our absurd approach is very slow on a big tree. Also we would like to maximise our chances of the import working. If git and/or gbp ever work better by themselves, all of this craziness will simply not happen. Signed-off-by: Ian Jackson <ijack...@chiark.greenend.org.uk> --- Makefile | 3 +++ absurd/git | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ dgit | 49 +++++++++++++++++++++++++++++++------------ 3 files changed, 109 insertions(+), 13 deletions(-) create mode 100755 absurd/git diff --git a/Makefile b/Makefile index b72a1e9..9491e4d 100644 --- a/Makefile +++ b/Makefile @@ -37,6 +37,7 @@ MAN1PAGES=dgit.1 MAN7PAGES=dgit.7 dgit-maint-merge.7 TXTDOCS=README.dsc-import PERLMODULES=Debian/Dgit.pm +ABSURDITIES=git INFRA_PROGRAMS=dgit-repos-server dgit-ssh-dispatch \ dgit-repos-policy-debian dgit-repos-admin-debian \ @@ -57,6 +58,8 @@ substituted/%: % install: installdirs all $(INSTALL_PROGRAM) $(addprefix substituted/,$(PROGRAMS)) \ $(DESTDIR)$(bindir) + $(INSTALL_PROGRAM) $(addprefix absurd/,$(ABSURDITIES)) \ + $(DESTDIR)$(absurddir) $(INSTALL_DATA) $(MAN1PAGES) $(DESTDIR)$(man1dir) $(INSTALL_DATA) $(MAN7PAGES) $(DESTDIR)$(man7dir) $(INSTALL_DATA) $(TXTDOCS) $(DESTDIR)$(txtdocdir) diff --git a/absurd/git b/absurd/git new file mode 100755 index 0000000..039bbf0 --- /dev/null +++ b/absurd/git @@ -0,0 +1,70 @@ +#!/bin/sh +set -e + +fail () { + echo >&2 "DGIT ABSURD GIT APPLY - FAILED: $*" + exit 127 +} + +self=${0%/*} +npath=${PATH#$self:} +if test "x$PATH" = "x$npath"; then + fail "PATH FILTER FAIL ($0 $self $PATH)" +fi + +bypass=true +for arg in "$@"; do + case "$arg" in + apply) bypass=false; break ;; + -*) ;; + *) bypass=true; break ;; + esac +done + +if $bypass; then + PATH=$npath + exec git "$@" +fi + +echo >&2 "DGIT ABSURD GIT APPLY - NO BYPASS: $*" + +#exec >/dev/tty 2>&1 + +index=0 +noo=0 + +for arg in "$@"; do + case "$noo.$arg" in + 1.--index) + index=1 + continue + ;; + ?.-*) + fail "UNKNOWN OPTION $arg ($*)" + ;; + 0.apply) + ;; + 1.*) patch="$arg" + ;; + *) + fail "BAD USAGE $arg ($noo $*)" + esac + noo=$(( $noo + 1 )) +done + +if [ $noo != 2 ]; then + fail "NO PATCH ($*)" +fi + +pwd=`pwd` +patch=${patch#$pwd/debian/patches/} +printf "%s\n" "$patch" >debian/patches/series + +dpkg-source --before-build . + +rm -rf .pc +git checkout debian/patches/series +git add -Af . + +echo >&2 "DGIT ABSURD GIT APPLY - APPLIED $patch" +#printf 'APPLIED '; date --iso-8601=ns diff --git a/dgit b/dgit index 344d006..f821299 100755 --- a/dgit +++ b/dgit @@ -1954,25 +1954,48 @@ END local $ENV{GIT_AUTHOR_EMAIL} = $authline[1]; local $ENV{GIT_AUTHOR_DATE} = $authline[2]; - eval { - runcmd shell_cmd 'exec >/dev/null 2>../../gbp-pq-output', - gbp_pq, qw(import); - }; - if ($@) { - { local $@; eval { runcmd qw(cat ../../gbp-pq-output); }; } - die $@; - } + my $path = $ENV{PATH} or die; + + foreach my $use_absurd (qw(0 1)) { + local $ENV{PATH} = $path; + if ($use_absurd) { + chomp $@; + progress "warning: $@"; + $path = "$absurdity:$path"; + progress "$us: trying slow absurd-git-apply..."; + rename "../../gbp-pq-output","../../gbp-pq-output.0" + or die $!; + } + eval { + local $ENV{PATH} = $path if $use_absurd; + + my @showcmd = (gbp_pq, qw(import)); + my @realcmd = shell_cmd + 'exec >/dev/null 2>../../gbp-pq-output', @showcmd; + debugcmd "+",@realcmd; + if (system @realcmd) { + die +(shellquote @showcmd). + " failed: ". + failedcmd_waitstatus()."\n"; + } - my $gapplied = git_rev_parse('HEAD'); - my $gappliedtree = cmdoutput @git, qw(rev-parse HEAD:); - $gappliedtree eq $dappliedtree or - fail <<END; + my $gapplied = git_rev_parse('HEAD'); + my $gappliedtree = cmdoutput @git, qw(rev-parse HEAD:); + $gappliedtree eq $dappliedtree or + fail <<END; gbp-pq import and dpkg-source disagree! gbp-pq import gave commit $gapplied gbp-pq import gave tree $gappliedtree dpkg-source --before-build gave tree $dappliedtree END - $rawimport_hash = $gapplied; + $rawimport_hash = $gapplied; + }; + last unless $@; + } + if ($@) { + { local $@; eval { runcmd qw(cat ../../gbp-pq-output); }; } + die $@; + } } progress "synthesised git commit from .dsc $cversion"; -- 2.9.3