On 19/01/17 07:00, Robie Basak wrote:
Hi Peter,
On Thu, Jan 19, 2017 at 02:16:27AM +0000, peter green wrote:
Some time ago I put together what I call the "autoforwardporter". The
aim of this is to take downstream changes and apply them on top of new
debian uploads.
Can you expand on this?
Here is my attempt at describing how the system works.
Throughout the process "local" versions are distinguished from "debian"
versions by looking for tags (in our case +rpi and +rvt) in the version number. The +rpi tag is
used where we have local modifications, the +rvt tag is used where we previously had local
modifications, have since removed them but for version numbering reasons we can't just use the
Debian version directly.
The tag is normally located at the end of the version number, but occasionally
it is located earlier in the version number for version ordering reasons.
The system relies on a whitelist of source package names that it should
consider processing (some local changes are short term hacks that should
probbablly not be carried forward)
When the system tries to make a forwardport it creates a logfile. It will only
try to make a forwardport with a given version number once unless the logfile
is manually removed.
Stage1: local repo on autoforwardporter box
We use reprepro to maintain local copies of both our own suites and the corresponding suite from Debian on the box that runs the
autoforwardporter. On our own side we have a "main" suite and a "staging suite" (similar in concept at least to
ubuntu's "proposed"). For stretch these suites are called "stretch" (our main suite) "stretch-staging" (our
staging suite) and "stretch-deb" (debian stretch)
Stage1: import.
I am using dgit import-dsc for this with a custom driver script that takes a directory
full of dscs, imports and tags them. If the package has a "dgit:" header we
just let dgit import it with whatever history is in the git repo.
If the dsc does not use dgit then the import script figures out the history
shape it should use from debian/changelog. For locally modified packages the
immediate parent must be present or the import fails. For non locally modified
packages we are far more lenient allowing version skipping and even orphan
histories.
Stage2: decide whether to forward port and figure out version numbers
For each package in the whitelist we check the version numbers. Figure out whether
there are local changes in the staging or main suites that should be brought
forward to a new version from Debian and if-so what the new version number should
be. The code is quite tricky because it has to handle some weird corner cases (for
example if we have a version 1.0+rpi1 and debian introduces 1.0+deb9u1 then the
new version needs to be numbered 1.0+rpi1+deb9u1 since 1.0+deb9u1+rpi1 <
1.0+rpi1 )
Stage3: potential psuedomerge.
First we grab the changelog from our old local version and look for the first
entry in the changelog that does not have one of our local tags on it. This
tells us the version from Debian that it was based on.
In most cases the old version from Debian will be an ancestor of the new
version from Debian but sometimes it isn't. In this case we do a psuedomerge to
create a commit that has the tree from the new debian version and has the old
debian version as an ancestor. This avoids trying to merge changes from the old
Debian version into the new Debian version when we do the main merge.
Stage 4: main merge
This bit is a simple git merge. The "ours" side of the merge is our old version and the
"theirs side" is the new version from Debian with the psudomerge if needed.
We use the "diff3" conflictstyle so our later fixup tools can get the context
they need.
Stage 5: post merge fixup
It is necessary to fix up debian/changelog so it doesn't contain conflict markers and so
it has the right versoin number. Exactly how this is done is a matter of taste,
personally I squash together all the "local" changes into a single changelog
entry on top of the Debian changelog. Ubuntu seem to be a fan of mergechangelogs
(personally I find the result of mergechangelogs to be a confusing mess).
It is also frequently needed to fix up debian/patches/series . The script tries
to retain order with local patches sorted after debian ones.
I expect further fixup will be needed, in particular I am expecting to have to
add some further quilt-related hackery. I also want to take a look at
debian/control merge failures, at least the simple cases.
Stage 6: commit results to working branch for archival
We commit the results to a working branch and push them to a suitable (local)
repo to preserve them. If there were merge failures that could not be
auto-resolved we commit the files with conflict markers still in them and put a
list at the top of debian/changelog. This allows a human to clone the working
branch, finish the fixup manually and amend the merge commit.
Stage 7: build
Assuming we were able to automatically resolve the merge conflicts we build a
source package with dgit and then call sbuild to build that source package.
Assuming it built succesfully we copy it to our "output" directory.
Stage 8: dgit push
Assuming the source package builds successfully we do a dgit push to add the
dgit: tag to the dsc and push the git stuff to our dgit server. We suppress the
actual uploading for the moment as I like to make some final manual checks
before uploading.
If the dgit push succeeds we re-copy the package with the dgit: header added to our
"output" directory.
_______________________________________________
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
