Re: extracting upstream source.
Package: dgit Version: 3.12 Peter Green writes ("extracting upstream source."): > As part of my dgit based autoforwardporter I want to add a script that will > de-fuzz patches with fuzz and remove patches that cannot be applied. > > To do this I need the "upstream" source tree. However the process of > extracting the upstream source is quite fiddly, there may be multiple > tarballs to deal with, tarballs may or may not have a top-level directory > that needs to be removed from the paths etc. > > Both dpkg-source ("commit" and "build")and dgit (quilt fixup) clearly extract > the upstream source tree as part of their processing, so the code to do it > already exists but i'm not sure if there is a convenient way to access it. dgit often has "something like" the upstream source tree as a git tree object. dgit should provide a way for you to get at it. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: extracting upstream source.
Ian Jackson writes ("Re: extracting upstream source."): > dgit often has "something like" the upstream source tree as a git tree > object. dgit should provide a way for you to get at it. This is now #869675. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]
Robie Basak writes ("Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]"): > I disagree here. We don't need to "hope". I don't expect the build to > see the importer's git history. It should be invisible to the build > process, and I intend to make our "git ubuntu build" wrapper make it > invisible. I think this is the key difference in our points of view. I look forward to a future where users routinely build Debian-format source packages from their distro's provided git history. For a user, in the most usual and simplest case, these will usually be in-tree builds run from the git working directory using dpkg-buildpackage. In this future world, the "distro's" .git (ie, in your case, your importer's) is present in the user's working tree. I doubt that there are a significant number of package build systems which rely on the presence or absence (or specific contents) of .git, (or for that matter of ..git). If such things exist right now they need to be fixed ASAP - but I very much doubt they do. Building historical versions ad-hoc may need an ad-hoc "detransform" step; but in general, building historical versions is going to be quite hard anyway for a whole host of reasons. Building historical versions in bulk may need the detransform to be automated, but this should be confined to this archaeology exercise. Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]
Robie Basak writes ("Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]"): > I don't really follow what you're saying. Are you saying that wrapping > these types of things in general is bad, or that having a wrapper for > the specific case of unescaping .git is bad? Yes, the latter. > For the latter, I think that safe round-tripping is an important > property of an importer and that ditching this property shouldn't be > taken lightly. I'm applying this principle in making design decisions > that we can't easily change later, such as the imported "format". There is some logic to this, indeed. I think your model requires a higher degree of fidelity. So I guess it's worthwhile thinking about how to make any transformations you do reversible, from which pov ..git is not too bad an idea. Before we adopt this I think we should consult more widely, though. And: even if the transformation is reversible, I think this should be for archeaological purposes, not for operational ones. Ie you should be able to inspect what's there, but any work based on the old branch should probably either preserve it or discard it. > OTOH, I wouldn't consider it to be a high priority to actually > implement, and a default of failing if ..git exists would be perfectly > acceptable for this extreme edge case. We could require the user to tell > us exactly what is required (drop or unescape) when rebuilding the > source package. > > Though then we'd probably need a "batch mode" that would probably > default to unescape to avoid creating a minefield of edge cases for > script writers. I'm not sure what you mean. Do you mean something which contradicts my previous paragraph. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]
Robie Basak writes (""git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]"): > Some examples: > > Upstreams that ship .gitignore confuses distribution developers (IMHO) > who want to see *everything* that has changed. I'd like to work with git > upstream in adding a repository level configuration option (or similar) > to completely ignore all dotfiles in the worktree that affect git's > behaviour. As a distribution developer, git's behaviour should come from > me, not some random upstream whose package I happen to be working on. In > the meantime, our wrapper warns you if these are present. AIUI .gitignore does not affect whether git shows you diffs to files that have changed. It simply suppresses listing of _added_ files. > Similarly, there are upstreams that use $Id$ and similar abominations. > The only sensible way to handle these and other corruptions is to turn > off ident, text and eol in .git/info/attributes, which our wrapper adds > on "git ubuntu clone". `dgit clone' disables these .gitattributes; provides a separate verb for disabling them in other trees; and `dgit fetch' warns about them if it finds them. > Having sensible default refspecs and dealing with sharing repositories > when you can't push to the main importer repository is a pain. This is > amplified when few developers work across many packages and don't keep > persistent local git repositories for them. So "git ubuntu clone" and > "git ubuntu remote add" deal with setting up reasonable default > remotes and refspecs. Obviously having sensible remotes is nice, but not IMO essential. > In Ubuntu, the importer must maintain separate pristine-tar branches for > Debian and Ubuntu because orig tarballs may not necessarily match. We > try to make them match, but the importer must be able to represent > everything that happened, including past mistakes. But "git clone" > followed by "dpkg-buildpackage" won't be able to find the upstream > tarball, and nor can gbp without adjusting the local pristine-tar > branch. This is solved by "git ubuntu build-source", which takes care of > getting the upstream tarball for you first. dpkg-buildpackage -B does not need an upstream tarball. It seemed obvious to me when writing dgit that it would be too hard to to make a uniform data model and workflows that could be used to generate source packages from git. Anyway, I don't think a wrapper that does "unescaping" of .git is a good idea. Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: What to do with .git directories in source package uploads?
Nish Aravamudan writes ("What to do with .git directories in source package uploads?"): > I noticed today that our Ubuntu git importer failed to import > src:nplan (which is only in Ubuntu). This is because the upload > (incorrectly) includes a .git directory. I see that dgit's behavior > for the same DSC file with `dgit import-dsc` is to delete the .git > directory from the source package before import. Is that a reflection > of Debian policy? Debian ftpmasters REJECT out of NEW packages that contain .git directories. But there is a loophole, which I think is accidental, that lets later uplods containing .git directories slip through. > 1) What dgit currently does, which is remove .git directories from > source packages, with a warning. This does mean that an imported > object does not match an uploaded exactly. Perhaps this is OK, but for > instance, in the case of nplan (purely as an example), if a developer > were to then use our git repository to develop a follow-on change, the > resulting debdiff would include (surprisingly to the end-user) a > removal of .git/) I think that is actually correct. If Debian ftpmasters fix the bug in their system, so that uploads containing .git are properly rejected, it would even be necessary. Furthermore many Debian package-handling tools will delete the .git anyway. > 2) In order from longest to shortest: > Change any file in the source package matching ^\(\.+git\)$ to > \.\1 > [essentially prefixing a '.' to any file consisting of only > leading '.' and the string 'git' ] ... > I'm wondering, are there are any explicit policy statements that say a > build system cannot rely on a git directory being present? Say a > debian/rules that does a `git describe`? I don't want to break such a > source package if it does exist. I'm almost certain that all the .git directories found in actual package uploads are accidents caused my buggy or misoperated packaging tools. > I'm planning on implementing 2), as it leaves us the most wiggle-room > and we haven't asserted anything about hash stability yet in our > imports. But I'd love for some input on why 1) was chosen, and, as I > mentioned earlier, if it's considered 'policy' or not. Your scheme is a possible alternative approach, indeed. It might result in slightly confusing git trees. It doesn't lose information, it's true, which is a slight advantage. But if there are packages where the .git directory is somehow actually used in the build, your scheme won't help because a build from the imported git tree will use your history, not the history found in the .dsc's .git. Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: git based autoforwardporter.
peter green writes ("git based autoforwardporter."): > I can't imagine raspbian is the only project who would find such a > tool useful. OTOH it would be a fair bit of work to clean up the > tool to seperate local raspbian assumptions from more generally > applicable functionality. > > Are there people who would be interested in using and collaborating > on such a tool? I think such a tool would be very useful in general, if it could be used for individual users. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Nish Aravamudan writes ("patches-applied historical imports (usd import)"): > 1) Some source packages (bouncycastle, php7.0 are the ones I can think > of off the top of my head) upstream tarballs contain .gitattributes I investigted this. After looking at the docs and playing about, I concluded that the only sane approach is to record as the actual tree object (in the git history) the contents of the Debian source package. Otherwise it will become impossible to represent certain source packages and all sorts of unanticipated madness could occur. So I'm currently testing an enhancement to dgit which causes it to * unconditionally suprress transforming gitattributes when working behind the scenes to import a .dsc * by default, configure suppression of these attributes, when creating a fresh tree with `dgit clone' (and also in `dgit setup-new-tree') The only other possiblity would be to apply a lossless rename operation to all .gitattributes, which would be worse. Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Ian Jackson writes ("Re: patches-applied historical imports (usd import)"): > I suggest you read "Intent to commit craziness - source package > unpacking" et seq, which were on this list in September. Here's the > archive: > > > http://lists.alioth.debian.org/pipermail/vcs-pkg-discuss/2016-September/thread.html And after you've read that, you can see that what I actually did was even worse. https://browse.dgit.debian.org/dgit.git/commit/?id=05db7ab051736bb106467e7f83b9e45684dd227c You will want to look at the most recent dgit HEAD for the full algorith, because there are other workarounds for bugs in dpkg-source etc. Near here: https://browse.dgit.debian.org/dgit.git/tree/dgit#n2365 -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Ian Jackson writes ("Re: patches-applied historical imports (usd import)"): > Urk! I have just filed: ... > https://bugs.debian.org/850845 > dpkg-source fails to extract samba_3.6.5-2.dsc but exits status 0 - > https://bugs.debian.org/850845 This was actually in dget, not dpkg-source. dpkg-source exits 2. Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Nish Aravamudan writes ("patches-applied historical imports (usd import)"): > [stuff] It occurs to me that it would be a good idea for you to check that your importer produces identical answers to dgit. If nothing else, from your point of view dgit can help you test that your importer DTRT. At the very least, for every .dsc which is successfully imported by both dgit and your tool, the resulting imported tree object should be identical. Since I think dgit should be able to import every .dsc in the whole of history (given new enough underlying tools), I would encourage you to report every import failure as a bug against dgit. When considering such bugs: * I do not mind if you do not repro the bug on Debian * Point me at the exact .dsc * Run dgit with -D and include the output in the bug report * Use Severity: important Ideally you and I could agree on a commit structure for the imported packages, and metadata processing rules, so that the commits are identical (not just the trees). Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Nish Aravamudan writes ("patches-applied historical imports (usd import)"): > 1) Some source packages (bouncycastle, php7.0 are the ones I can think > of off the top of my head) upstream tarballs contain .gitattributes > files, which will change the behavior of git itself when checking out a > branch. I need to do some tests to see exactly how to work around this. Can you point me at an affected source package version ? > 2) How do we determine if a source package is 1.0 vs. 3.0? I am > currently using `dpkg-source --print-format`, but have found one source > package (util-linux 2.13~rc3-5), where dpkg-source emits: > > syntax error in /tmp/tmp3y515osf/util-linux-2.13~rc3/debian/control at > line 14: duplicate field Depends found > > and thus we error out. How exciting. dgit looks at debian/source/format, which is what dpkg-source reads when generating the package. > 3) Imagine the following graph in the git repository: > >A D > ->o.f>o-> >^ . . ^ >| c e | >o . . o >^. .^ >| B | >o o o >^ ^ ^ >| | | > ->o>o>o-> >a b d ... > Let's assert that there is a problem with obtaining the patches-applied > version of b. This can occur for (at least) the following reasons: > > i) as in 2), we might not be able to determine the source package format > (implementation detail, to some degree), so are unable to correctly > derive if there is a patches-applied state that is distinct from b. You can use dpkg-source -x to extract the patches-applied state corresponding to any .dsc, I think. Any case where you can't is a bug in dpkg-source. > ii) some patches may fail to apply with a trivial `quilt push`. This > occurs with, at least, a historical publish of samba. Do you have an example source package ? > In theory, there are other reasons/cases where this might happen and the > importer needs to never fail (so it is of some use to run automatically > :) I haven't run dgit's dsc importer on a whole historical archive but Peter Green of Raspian has been running it and filing bugs. I haven't seen such a bug recently so I hope it has been working for all the packages he's seen. > The questions I have relate to what to do when we encounter this > situation, which in turn is divided into two parts: I think this situation must be excluded. This kind of difficulty is one of the main reasons for the failure of previous efforts to transition universally to git. It is why dgit does not currently work, in Debian, with a full history of each package. Instead there's a bit of a bodge, with a locally generated history. Joey Hess and I came up with this approach at the Vaumarcus Debconf. Eventually I intend to import the whole history but not yet. If you intend to go forward without fully solving this problem, your choices are not good. > Let's presume that this failure is not persistent and that D is able to > be imported successfully, we again have to make decisions about > parenting. I think it only makes sense for one of c or f to exist, based > upon what we decide is the right policy above. This is particularly troublesome. When you can generate the missing package, you will not be able to make it an ancestor of your existing history. You could rewrite the existing history and then pseudomerge the old and new histories together, but that means duplicating most of the history metadata. If you wait with making an official import until it is suitable for use with dgit, then I might consider using the relevant parts of your imports history as the official Debian dgit history. That would save you from the problem which I foresee in our future, where Debian and Ubuntu have disjoint imports of the same history. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
~Johannes Schauer writes ("Re: Intent to commit craziness - source package unpacking"): > sbuild supports multiple backends. The default backend is schroot. How to get > from "apt-get install sbuild" to actually building packages depends on the > backend used. For the default schroot backend, sbuild provides the > sbuild-createchroot tool which runs debootstrap and creates a schroot config > file. So if you stay with the defaults, then you run: > > $ apt-get install sbuild > $ sbuild-createchroot unstable /srv/chroot/unstable-amd64 WIBNI the final argument had a default ? :-) Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
I forgot one: Ian Jackson writes ("Re: DEP14 policy for two dots"): > A patches-unapplied tree: > > * produces confusing and sometimes misleading output from >git grep, or (even if appropriate history is available) >with git blame; > > * cannot be used with `git cherry pick '; > > * cannot be used as a basis for `git merge upstream/'; > > * requires that the user not say `git diff upstream/master' >but rather that they read patches in debian/patches; > > * cannot be directly edited by the user; > > * leaves the git tree dirty after every build with dpkg-buildpackage >no matter how careful or tidy the package's build system. * when built with the upstream build system (eg, for a GNU package, ./configure && make), silently and successfuly produces wrong output - perhaps dangerously wrong output, such as binaries lacking important security patches. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Raphael Hertzog writes ("Re: DEP14 policy for two dots"): > Ok, can you prepare a patch for DEP-14 then? I'll apply it as it looks like > a reasonable extension. Attached. FYI I intend to implement this in dgit. Thanks, Ian. >From 5c63400e9be8cb1532515764a1179730aed550fb Mon Sep 17 00:00:00 2001 From: Ian Jackson <ijack...@chiark.greenend.org.uk> Date: Wed, 9 Nov 2016 18:36:23 + Subject: [PATCH] DEP-14: Version -> refname mangling: Escape dots Signed-off-by: Ian Jackson <ijack...@chiark.greenend.org.uk> --- deps/dep14.mdwn | 25 ++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/deps/dep14.mdwn b/deps/dep14.mdwn index 4c7ce63..a7328a4 100644 --- a/deps/dep14.mdwn +++ b/deps/dep14.mdwn @@ -3,7 +3,7 @@ Title: Recommended layout for Git packaging repositories DEP: 14 State: DRAFT -Date: 2014-11-04 +Date: 2016-11-09 Drivers: Raphael Hertzog <hert...@debian.org> URL: http://dep.debian.net/deps/dep14 Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep14.mdwn @@ -60,8 +60,26 @@ Version mangling When a Git tag needs to refer to a specific version of a Debian package, the Debian version needs to be mangled to cope with Git's restrictions. -The colon (`:`) needs to be replaced with a percent (`%`), and the tilde -(`~`) needs to be replaced with an underscore (`_`). +This mangling is deterministic and reversible: + + * Each colon (`:`) is replaced with a percent (`%`) + * Each tilde (`~`) is replaced with an underscore (`_`) + * A hash (`#`) is inserted between each pair of adjacent dots (`..`) + * A hash (`#`) is appended if the last character is a dot (`.`) + * If the version ends in precisely `.lock` + (dot `l` `o` `c` `k`, lowercase, at the end of the version), + a hash (`#`) is inserted after the dot, giving `.#lock`. + +This can be expressed concisely in the following Perl5 statements: + + y/:~/%_/; + s/\.(?=\.|$|lock$)/.#/g; + +The reverse transformation is: + + * Each percent (`%`) is replaced with a colon (`:`) + * Each underscore (`_`) is replaced with a tilde (`~`) + * Each hash (`#`) is deleted Packaging branches and tags === @@ -274,3 +292,4 @@ Changes === * 2014-11-05: Initial draft by Raphaël Hertzog. +* 2016-11-09: Extended version mangling to troublesome dots - Ian Jackson. -- 2.10.2 -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Ian Jackson writes ("Re: DEP14 policy for two dots"): > Raphael Hertzog writes ("Re: DEP14 policy for two dots"): > > On Fri, 04 Nov 2016, Ian Jackson wrote: > > > My proposal is reversible. It does not need to be extensible. > > > > So what about "..."? Would it give ".#.#."? > > Yes. I said (fixing my bug): > > > Insert "#": > >- between each pair of adjacent dots > >- after any trailing dot > >- before any leading dot > - after the `.' of a trailing `.lock' > > The latter is necessary because git reserves .lock. (!) > The summary is `add # after any troublesome dot' (discounting leading > dots which you say are now illegal in Debian). > > I'm running some exhaustive tests to check that this rule is > sufficient, because I'm not sure I trust the git docs. I have now: * Read the code in git upstream master. It's not particularly easy to analyse conclusively, but I'm pretty sure it doesn't have any special cases which involve longer strings than ".lock". I felt I was able to identify the manpage rule corresponding to each element of the logic. * Conducted an exhaustive search of all strings of length 6 or less. Specifically, I generated all strings of between zero and 6 characters from this set (in C notation): "0123456789" "abcdefghijklmnopqrstuvwxyz" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" ".-+:~" filtered them by whether the `parseversion' function in dpkg likes them, escaped them with the following perl script #!/usr/bin/perl -pw use strict; y/:~/%_/; s/\.(?=\.|$|lock$)/.#/g; prepended "refs/tags/" to each one and fed them to git-check-ref-format (modified to run in a pipe mode). I also verified that when I don't escape ".lock", my exhaustive search correctly detects the illegal ref name "refs/tags/1.lock" genrated by version "1.lock" (and similar). > The reverse rule is to convert _ and % and delete all #. Quoted for completeness. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Raphael Hertzog writes ("Re: DEP14 policy for two dots"): > On Fri, 04 Nov 2016, Ian Jackson wrote: > > My proposal is reversible. It does not need to be extensible. > > So what about "..."? Would it give ".#.#."? Yes. I said (fixing my bug): > Insert "#": >- between each pair of adjacent dots >- after any trailing dot >- before any leading dot - after the `.' of a trailing `.lock' The latter is necessary because git reserves .lock. (!) The summary is `add # after any troublesome dot' (discounting leading dots which you say are now illegal in Debian). I'm running some exhaustive tests to check that this rule is sufficient, because I'm not sure I trust the git docs. > What's the rule to apply? if it's just to drop the "#", then yes > it's reversible in a single step. If it's "s/\.#\./../g" then you need > to do it multiple times until you no longer find ".#.". The reverse rule is to convert _ and % and delete all #. > > > My suggestion would be to allow "##". > > > Thus my personal preference would be to replace ".." with ".#2e#". > > > > This is a bad idea because it (implicitly) makes the conversion > > nondeterministic. > > We define the conversion rule in DEP-14. We can define it in a > deterministic way. If you define it in a deterministic way then it is by definition not extensible, because all valid version strings have a definitive git tag representation. Unless by `extensible' you mean `we can update the rule if we discover that some of the specified git tag representations are not accepted by git', or `we can update the rule if the set of valid Debian version strings is extended'. But this is true of any proposal, no matter what the syntax is. > I wanted something extensible because what's allowed in git ref names > might evolve. It would not be the first time that a special syntax > is introduced with a new feature. I think the git folks are going to try not to further restrict the git ref name syntax. After all, if they do restrict it, what about existing tags with the now-forbidden names ? > Which of # or = is more likely to be used for a new syntax/feature in git? > My bet would go for "#" so that "=" is an even better choice. I think = is more likely to be used for other things (both by git and by others). Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Nish Aravamudan writes ("DEP14 policy for two dots"): > [ Raphael, apologies for sending twice, had a error in the headers in > the prior one ] > > Not sure exactly where to ask this better than debian-devel, but I am > working on an importer for the Ubuntu Server team which parses published > versions of source packages in Debian and Ubuntu. I ran into an issue > today where there is a published version of src:pcre3 with version > '8.30..2'. `man git-check-ref-format` says that reference names "cannot > have two consecutive dots .. anywhere." DEP14 specifies appropriate > substitutions for : and ~, but it seems like .. should also be accounted > for so I can correctly tag historic versions? Urk. How exciting. I think we may need a more general escaping scheme for these and other weirdnesses. I have an interest as dgit uses DEP-14 tag escaping. I have CC'd the vcs-pkg list. tl;dr: I think we should insert `#' characters as needed. Looking at git-check-ref-format(1) and https://wiki.debian.org/Punctuation: .special to git, generally permitted in versions, and we want it usually to be literal - this is our problem ~special to git, permitted in versions, handled by DEP-14 as _ :special to git, epoch in versions, handled by DEP-14 as % @special to git (although sometimes allowed), forbidden in versions % _ not special to git but already used by DEP-14 # , = not mentioned in the git manual as special, forbidden in versions ]not special to git, although [ is so let's not, eh ? + - not special to git, permitted in versions " ' $ & ( ) * ; < > ? ` not mentioned in the git manual but troublesome shell metacharacters which we would be insane to use here [ / { } interpreted specially by git some of the time, forbidden in versions - not really useful ^ ? * \ all of these are forbiden by git, not permitted in versions So I think in fact the only thing we have a problem with is multiple dots. Looking at the summary above, we have the choice of one of these: # Its use as a shell comment character is fine, because when inside a version tag it is always preceded by some string like "debian/" or "upstream/". We would almost never need to put it at the start of the encoded version string anyway, and we have already tolerated a similar situation with ~. There is possible confusion with HTML fragment identifiers, and possibly in languages other than shell which use # for comments (athough hopefuly they aren't dealing with our versions as literals anyway). Proposed rule: Insert "#": - between each pair of adjacent dots - after any trailing dot - before any leading dot Examples: 8.30..2 => 8.30.#.2 8.30. => 8.30.# .42 => #.42 , I would like to avoid this because lots of people are probably using it as a list separator in ways that are difficult for us to predict. If we used it, I would suggest the same as for #. = In principle we could use this. I don't like it for a similar reason to above. If we did use it it might look a bit like Q-P encoding in some contexts. @ We could use this although I wouldn't like to rely on the fact that git dislikes `@{' and `@' but not @ followed by other things. % Reusing this is tempting because an epoch separator can never follow `.', so any `%' after any `.' would unambiguously mean `escape for dot rather than colon'. But in principle `.' can occur at the start of the version, so `:3' and `.3' both => `%3'. There would have to be some horror of an exception rule. (Although `:3' and `3' compare equal as Debian versions, they are different textual strings and the tag needs to convey the whole string.) Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Russ Allbery writes ("Re: Intent to commit craziness - source package unpacking"): > Ian Jackson <ijack...@chiark.greenend.org.uk> writes: > > I'm not sure of the logic behind that. I don't think dgit helps much > > with the kind of tasks that pristine-tar helps with. > > The main benefit of pristine-tar is that you can clone the development > repository on any random host and do package development, build local > packages for testing, and do a package upload without having to locate any > additional pieces. I haven't been following dgit development closely, but > it did sound like you were addressing the same use case. dgit will fetch the orig tarball for you. So you don't have to "locate" the pieces, but you do have to pay the download cost. > If sbuild is now at the point where you can just apt-get install sbuild, > run a single setup command, and be ready to build packages (which is where > cowbuilder is right now), I personally would be happy to use something > that's a bit closer to what the buildds are doing. There is sbuild-setupchroot or something. I do find that I don't want it because I like fiddling with the config, so I just use lower level commands myself. > However, cowbuilder does just work, in my experience, and it's nice to not > have to change. Of course. I'm sorry I stepped into the argument about which chroot build tool is best :-). Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Guido Günther writes ("Re: Intent to commit craziness - source package unpacking"): > On Mon, Oct 03, 2016 at 04:15:08PM +0100, Ian Jackson wrote: > [..snip..] > > Recommends: pristine-tar (>= 0.5) ...> > > > pristine-tar has been declared unmaintainable by its original > > author and abandoned. ... > > Certainly dgit users do not need pristine-tar. But our dependency > > system does not allow us to honour only direct Recommends and not > > transitive ones. > > Looking at git.debian.org I found plenty of users. I did an archive > import of sid during Debconf and was only ran into 20 pristine-tar > failures (bugs yet to be filed). Interesting. > From the discussions at DC16 we're on our way to make this a hard > dependency: > > http://lists.sigxcpu.org/pipermail/git-buildpackage/2016-July/000143.html > > The only thing I can think of (since we will keep support for not using > pristine-tar nevertheless) is using: > > Recommends: pristine-tar | dgit I'm not sure of the logic behind that. I don't think dgit helps much with the kind of tasks that pristine-tar helps with. > > Recommends: cowbuilder<= jessie > > Recommends: cowbuilder | pbuilder | sbuild<= sid ... > gbp buildpackage has integration with pbuilder/cowbuilder (via > git-builder) and I know people are using it since its better integrated > into gbp since you don't need additional and it's documented in the > manual. The sbuild dependency is there to have people not pull in > cowbuilder/pbuilder so they can use --git-builder=sbuild. Ah. > Not sure what can be done here. It sounds like it should be left as-is, TBH. > > Depends: devscripts > > > > devscripts is very full of commands with poor namespacing. It > > also has an enormous dependency chain. > > > > Unfortunately dgit has a dependency on devscripts too. Maybe we > > should work to take the pieces of devscripts that we really need > > and put them in something else, or something. > > We're mostly using dch with "gbp dch" and I would also be happy to have > the dependency chain shortened. If it were my package, and that was all I depended on devscripts for, I would drop it entirely. I think it's fair to expect someone who uses `gbp dch' to install the package containing dch. But this is a matter of taste. dgit has a much harder dependency because dgit push uses dput. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package unpacking"): > Guido Günther writes ("Re: Intent to commit craziness - source package > unpacking"): > > 'gbp pq import' does have 'debian/patches' since it just puts the > > patches that are in debian/patches on top of the unpatched source > > tree. In contrast to your solution it doesn't try to be able to > > roundtrip without changes for any given series on "gbp pq export". This > > is only true if the series was also created by "gbp pq" (or adheres to > > what git-format-patch does). > > Currently the output of dpkg-source --commit doesn't look much like > the output of git-format-patch. > > I have tried to make dgit produce patches (when it needs to produce > patches) that look like dpkg-source --commit. But maybe it should > produce patches using git-format-patch (or that look like > git-format-patch). I have looked at the specs (including DEP-3) and the behaviour of the various tools a bit more. I have decided that when I should replace dgit's adhoc algorithm for generating a DEP-3 compliant commit message, with a call to git-format-patch. (The actual patch body has to come from dpkg-soure, because of the special handling of debian/, etc. This code path is used when a dgit user is pushing and needs to convert raw git commits into `3.0 (quilt)' patches.) And, after having half-written a DEP-3 importer (ie a thing that turns a DEP-3 patch message into a git commit), and investigating the behaviour of various tools, I have decided I should just use gbp pq. (This will replace the repeated use of dpkg-source --before-build as a described in my previous mail.) This will mean that dgit will grow a hard dependency on git-buildpackage. Because I'm a pernickety type of person I reviewed the dependencies of git-buildpackage. I have some qualms about the following dependencies: Recommends: pristine-tar (>= 0.5) pristine-tar has been declared unmaintainable by its original author and abandoned. IDK know what proportion of actual git trees that gbp users will encounter would break without pristine-tar. Perhaps this dependency can be demoted to Suggests, but I don't really know. Certainly dgit users do not need pristine-tar. But our dependency system does not allow us to honour only direct Recommends and not transitive ones. Recommends: cowbuilder<= jessie Recommends: cowbuilder | pbuilder | sbuild<= sid Many users of dgit will never want to build a package for upload. This is probably true of gbp users too. I'm not sure why it's valuable to have this as a Recommends dependency for gbp. I think more people now are using sbuild. I'm not sure that pulling in cowbuilder on systems where the user has not yet installed such a tool is right. Depends: devscripts devscripts is very full of commands with poor namespacing. It also has an enormous dependency chain. Unfortunately dgit has a dependency on devscripts too. Maybe we should work to take the pieces of devscripts that we really need and put them in something else, or something. Overall I don't think these are an impediment. But since I had done the review, I thought I'd share my thoughts. Regards, Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Guido Günther writes ("Re: Intent to commit craziness - source package unpacking"): > The authorship and subject information is taken from the patch if > available or made up from the filename and committer when not. > > If patches in the series file are in subdirs of debian/patches we store > that in "Gbp-Pq: topic" in the commit message. The patch name itself > is stored in "Gbp-Pq: Name" we can reproduce it on "gbp pq export" > independent from the patch's subject. Right. > > Bear in mind that because the output of gbp-pq import doesn't > > contain debian/patches, I would need to rewrite its output (perhaps > > with git-filter-branch). > > 'gbp pq import' does have 'debian/patches' since it just puts the > patches that are in debian/patches on top of the unpatched source > tree. In contrast to your solution it doesn't try to be able to > roundtrip without changes for any given series on "gbp pq export". This > is only true if the series was also created by "gbp pq" (or adheres to > what git-format-patch does). Currently the output of dpkg-source --commit doesn't look much like the output of git-format-patch. I have tried to make dgit produce patches (when it needs to produce patches) that look like dpkg-source --commit. But maybe it should produce patches using git-format-patch (or that look like git-format-patch). Thanks for suggesting libvirt as an example. I will play about with it. Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package unpacking"): > Guido Günther writes ("Re: Intent to commit craziness - source package > unpacking"): > > Hi Ian, > > Hi, thanks for your attention. Please disregard this email. I hadn't finished editing it! Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package unpacking"): > Thanks for your comments. I feel unblocked :-). So, I now intend to go and implement my plan. There will be a little while (perhaps a few weeks) before I am in a postion to release this in dgit 2.0. But after I do that, I will not want to change the import algorithm again: it is important that the imports be as stable as possible. So now would be a good time for maintainers of git packaging tools (eg git-dpm and gpb) to have an opinion about the detail of the generated pseudohistory - in particular, the detail of the commits generated from the `3.0 (quilt)' dpkg-source patches. Also, I would welcome suggestions for what kind of compatibility test I could perform on such a series of commits. dgit has an extensive test suite (advertised via autopkgtest) which would be well-suited to such a compatibility test. An example of such a tree might be, "split out the patch queue part of the git pseudohistory and feed it to gbp-pq, asking gbp-pq to regenerate the source package, and expect the output to be identical to the original input source package". Guido, if I get the preconditions right, should I expect such a test to pass ? Is there a risk it would break in the future due to changes in gbp-pq's conversion algorithm ? I confess that I am less familiar with git-dpm. I don't know what I should be thinking about to try to make the output most useful to git-dpm users. (I also don't know whether the goals of helping git-dpm users and gbp-pq users, and potentially users of any other tools, are in conflict. It would be annoying if these tools would disagree about the best form of import of a particular patch queue: the import algorithm should be the same for different dgit users, so I wouldn't be able to make this a per-user configuration option and would have to choose..) Thanks, Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Intent to commit craziness - source package unpacking
.) * No back doors into the innards of dpkg-source (nor changes to dpkg-dev) are required. * dgit does grow a dependency on Dpkg::Compression. * Knowledge of the source format embedded in dgit is is restricted to iterating over tarballs and manipulating debian/patches/series, which dgit already does. * dgit now depends on dpkg-source --before-build idempotently applying patches as they successively appear on debian/patches/series. * Perhaps the git commits generated by dgit to represent patches can be made to round-trip nicely into tools like git-dpm and git-buildpackage. I have found the information about tags in gbp-dch(1), but that doesn't seem like it's applicable. I have also found the information about tags in gbp-pq(1). From that it looks like I ought to generate "Gbp-Pq: Name" and "Gbp-Pq: Topic". * The scheme I describe avoids introducing a dependency from dgit to git-buildpackage. I might be able to replace the successive-patch-application part with an appropriate invocation of gbp-pq. Would that be better ? Bear in mind that because the output of gbp-pq import doesn't contain debian/patches, I would need to rewrite its output (perhaps with git-filter-branch). Comments welcome. Please be quick - this is very close to the top of my dgit todo list. Thanks, Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: git, debian/ tags, dgit - namespace proposal [and 1 more messages]
Ian Jackson writes ("Re: git, debian/ tags, dgit - namespace proposal [and 1 more messages]"): > Colin Watson suggested (in pers.comm) >pkg/debian/ > This is better but it still has a problem with collate order. > > It may seem a petty thing to worry about, but for the reasons I > explain above I want something that sorts before `debian/'. A conversation with Owen Dunn produced suggestions of `actual' and `archive' as name components. I'm considering: archive/{debian,ubuntu}/ {debian,ubuntu}/archive/ I'm still considering the capitalisation idea. Other suggestions welcome. Thanks, Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
git, debian/ tags, dgit - namespace proposal
Currently, the debian/ git tag namespace is used by a number of different tools, and can mean different things in different packages and sometimes even different things for the same package in different repos or trees. dgit produces, and the dgit git server requires, tags of this form, which refer to the dgit branch, which is roughly what some people call a `patches-applied packaging branch'. So far, dgit has been using debian/ for this. But this is a problem because this tag namespace is used for other purposes and has no clear meaning. I'm currently working on dgit native push support for users of 3.0-quilt+git-buildpackage, where this is particularly bad because it would involve dgit making two tags with the same name (!) Having considered things, I think it would be best for dgit to concede this area of namespace for all the existing uses. Declaring those existing uses wrong, just because they're varied and and mutually inconsistent, is not very helpful. Consequently I need a new namespace. I don't want to put `dgit' in the name because the format is not specific to dgit. It is simply the result of `dpkg-source -x' (only without the .pc directory which dpkg-source sometimes produces). I am currently thinking that dgit would start to use the tag namespace Debian/ instead. These tags would always refer to a Patches-applied packaging branch (where applicable containing a patch to add or update .gitignore). For other distros, I would likewise capitalise just the first character of the distro name (with `ucfirst' in Perl). So `Ubuntu/'. is translated as specified in DEP-14. So this message is: * A request for anyone to say if they know of a reason I shouldn't do this. * A declaration (currently, tentative) of intent to claim this part of the git tag namespace. * A proposal to update DEP-14 to declare that "vendor" in DEP-14 should start with a lowercase letter, and ideally, to reserve the corresponding starts-with-uppercase tags for dgit. Thanks, Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: git-buildpackage / dgit integration
Guido Günther writes (Re: git-buildpackage / dgit integration): On Sun, Aug 16, 2015 at 03:46:46PM +0100, Ian Jackson wrote: * .gitignore: We agree that the source package should contain .gitignore if the git tree does. (dgit requires this.) If the way that gbp does builds currently ends up with .gitignore missing from source packages, this needs to be fixed. I do wonder if this is an open issue? I looked at git trees containing .gitignore and they ended up in the source package. In case the Debian maintainer modifies .gitignore this will then end up as a quilt patch of course. Thanks for looking at this. My summary was inaccurate. If _upstream_ have a gitignore then it does indeed end up in the package. But if the gitignore is created or changed by the maintainer, the default -I omits it, I think. I haven't tried this with gbp yet. (I'm currently wrestling with some dgit misbehaviour with 3.0 single-debian-patch.) If I'm wrong then excellent. Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss