Re: extracting upstream source.
Package: dgit
Version: 3.12
Peter Green writes ("extracting upstream source."):
> As part of my dgit based autoforwardporter I want to add a script that will
> de-fuzz patches with fuzz and remove patches that cannot be applied.
>
> To do this I need the "upstream" source tree. However the process of
> extracting the upstream source is quite fiddly, there may be multiple
> tarballs to deal with, tarballs may or may not have a top-level directory
> that needs to be removed from the paths etc.
>
> Both dpkg-source ("commit" and "build")and dgit (quilt fixup) clearly extract
> the upstream source tree as part of their processing, so the code to do it
> already exists but i'm not sure if there is a convenient way to access it.
dgit often has "something like" the upstream source tree as a git tree
object. dgit should provide a way for you to get at it.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: extracting upstream source.
Ian Jackson writes ("Re: extracting upstream source."):
> dgit often has "something like" the upstream source tree as a git tree
> object. dgit should provide a way for you to get at it.
This is now #869675.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]
Robie Basak writes ("Re: "git ubuntu" wrappers [was: What to do with .git
directories in source package uploads?]"):
> I disagree here. We don't need to "hope". I don't expect the build to
> see the importer's git history. It should be invisible to the build
> process, and I intend to make our "git ubuntu build" wrapper make it
> invisible.
I think this is the key difference in our points of view.
I look forward to a future where users routinely build Debian-format
source packages from their distro's provided git history. For a user,
in the most usual and simplest case, these will usually be in-tree
builds run from the git working directory using dpkg-buildpackage.
In this future world, the "distro's" .git (ie, in your case, your
importer's) is present in the user's working tree.
I doubt that there are a significant number of package build systems
which rely on the presence or absence (or specific contents) of .git,
(or for that matter of ..git). If such things exist right now they
need to be fixed ASAP - but I very much doubt they do.
Building historical versions ad-hoc may need an ad-hoc "detransform"
step; but in general, building historical versions is going to be
quite hard anyway for a whole host of reasons. Building historical
versions in bulk may need the detransform to be automated, but this
should be confined to this archaeology exercise.
Ian.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]
Robie Basak writes ("Re: "git ubuntu" wrappers [was: What to do with .git
directories in source package uploads?]"):
> I don't really follow what you're saying. Are you saying that wrapping
> these types of things in general is bad, or that having a wrapper for
> the specific case of unescaping .git is bad?
Yes, the latter.
> For the latter, I think that safe round-tripping is an important
> property of an importer and that ditching this property shouldn't be
> taken lightly. I'm applying this principle in making design decisions
> that we can't easily change later, such as the imported "format".
There is some logic to this, indeed. I think your model requires a
higher degree of fidelity.
So I guess it's worthwhile thinking about how to make any
transformations you do reversible, from which pov ..git is not too
bad an idea.
Before we adopt this I think we should consult more widely, though.
And: even if the transformation is reversible, I think this should be
for archeaological purposes, not for operational ones. Ie you should
be able to inspect what's there, but any work based on the old branch
should probably either preserve it or discard it.
> OTOH, I wouldn't consider it to be a high priority to actually
> implement, and a default of failing if ..git exists would be perfectly
> acceptable for this extreme edge case. We could require the user to tell
> us exactly what is required (drop or unescape) when rebuilding the
> source package.
>
> Though then we'd probably need a "batch mode" that would probably
> default to unescape to avoid creating a minefield of edge cases for
> script writers.
I'm not sure what you mean. Do you mean something which contradicts
my previous paragraph.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: "git ubuntu" wrappers [was: What to do with .git directories in source package uploads?]
Robie Basak writes (""git ubuntu" wrappers [was: What to do with .git
directories in source package uploads?]"):
> Some examples:
>
> Upstreams that ship .gitignore confuses distribution developers (IMHO)
> who want to see *everything* that has changed. I'd like to work with git
> upstream in adding a repository level configuration option (or similar)
> to completely ignore all dotfiles in the worktree that affect git's
> behaviour. As a distribution developer, git's behaviour should come from
> me, not some random upstream whose package I happen to be working on. In
> the meantime, our wrapper warns you if these are present.
AIUI .gitignore does not affect whether git shows you diffs to files
that have changed. It simply suppresses listing of _added_ files.
> Similarly, there are upstreams that use $Id$ and similar abominations.
> The only sensible way to handle these and other corruptions is to turn
> off ident, text and eol in .git/info/attributes, which our wrapper adds
> on "git ubuntu clone".
`dgit clone' disables these .gitattributes; provides a separate verb
for disabling them in other trees; and `dgit fetch' warns about them
if it finds them.
> Having sensible default refspecs and dealing with sharing repositories
> when you can't push to the main importer repository is a pain. This is
> amplified when few developers work across many packages and don't keep
> persistent local git repositories for them. So "git ubuntu clone" and
> "git ubuntu remote add" deal with setting up reasonable default
> remotes and refspecs.
Obviously having sensible remotes is nice, but not IMO essential.
> In Ubuntu, the importer must maintain separate pristine-tar branches for
> Debian and Ubuntu because orig tarballs may not necessarily match. We
> try to make them match, but the importer must be able to represent
> everything that happened, including past mistakes. But "git clone"
> followed by "dpkg-buildpackage" won't be able to find the upstream
> tarball, and nor can gbp without adjusting the local pristine-tar
> branch. This is solved by "git ubuntu build-source", which takes care of
> getting the upstream tarball for you first.
dpkg-buildpackage -B does not need an upstream tarball.
It seemed obvious to me when writing dgit that it would be too hard to
to make a uniform data model and workflows that could be used to
generate source packages from git.
Anyway, I don't think a wrapper that does "unescaping" of .git is a
good idea.
Ian.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: What to do with .git directories in source package uploads?
Nish Aravamudan writes ("What to do with .git directories in source package
uploads?"):
> I noticed today that our Ubuntu git importer failed to import
> src:nplan (which is only in Ubuntu). This is because the upload
> (incorrectly) includes a .git directory. I see that dgit's behavior
> for the same DSC file with `dgit import-dsc` is to delete the .git
> directory from the source package before import. Is that a reflection
> of Debian policy?
Debian ftpmasters REJECT out of NEW packages that contain .git
directories. But there is a loophole, which I think is accidental,
that lets later uplods containing .git directories slip through.
> 1) What dgit currently does, which is remove .git directories from
> source packages, with a warning. This does mean that an imported
> object does not match an uploaded exactly. Perhaps this is OK, but for
> instance, in the case of nplan (purely as an example), if a developer
> were to then use our git repository to develop a follow-on change, the
> resulting debdiff would include (surprisingly to the end-user) a
> removal of .git/)
I think that is actually correct. If Debian ftpmasters fix the bug in
their system, so that uploads containing .git are properly rejected,
it would even be necessary.
Furthermore many Debian package-handling tools will delete the .git
anyway.
> 2) In order from longest to shortest:
> Change any file in the source package matching ^\(\.+git\)$ to
> \.\1
> [essentially prefixing a '.' to any file consisting of only
> leading '.' and the string 'git' ]
...
> I'm wondering, are there are any explicit policy statements that say a
> build system cannot rely on a git directory being present? Say a
> debian/rules that does a `git describe`? I don't want to break such a
> source package if it does exist.
I'm almost certain that all the .git directories found in actual
package uploads are accidents caused my buggy or misoperated packaging
tools.
> I'm planning on implementing 2), as it leaves us the most wiggle-room
> and we haven't asserted anything about hash stability yet in our
> imports. But I'd love for some input on why 1) was chosen, and, as I
> mentioned earlier, if it's considered 'policy' or not.
Your scheme is a possible alternative approach, indeed. It might
result in slightly confusing git trees. It doesn't lose information,
it's true, which is a slight advantage.
But if there are packages where the .git directory is somehow actually
used in the build, your scheme won't help because a build from the
imported git tree will use your history, not the history found in the
.dsc's .git.
Ian.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: git based autoforwardporter.
peter green writes ("git based autoforwardporter."):
> I can't imagine raspbian is the only project who would find such a
> tool useful. OTOH it would be a fair bit of work to clean up the
> tool to seperate local raspbian assumptions from more generally
> applicable functionality.
>
> Are there people who would be interested in using and collaborating
> on such a tool?
I think such a tool would be very useful in general, if it could be
used for individual users.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Nish Aravamudan writes ("patches-applied historical imports (usd import)"):
> 1) Some source packages (bouncycastle, php7.0 are the ones I can think
> of off the top of my head) upstream tarballs contain .gitattributes
I investigted this. After looking at the docs and playing about, I
concluded that the only sane approach is to record as the actual tree
object (in the git history) the contents of the Debian source package.
Otherwise it will become impossible to represent certain source
packages and all sorts of unanticipated madness could occur.
So I'm currently testing an enhancement to dgit which causes it to
* unconditionally suprress transforming gitattributes when working
behind the scenes to import a .dsc
* by default, configure suppression of these attributes, when
creating a fresh tree with `dgit clone' (and also in `dgit
setup-new-tree')
The only other possiblity would be to apply a lossless rename
operation to all .gitattributes, which would be worse.
Ian.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Ian Jackson writes ("Re: patches-applied historical imports (usd import)"):
> I suggest you read "Intent to commit craziness - source package
> unpacking" et seq, which were on this list in September. Here's the
> archive:
>
>
> http://lists.alioth.debian.org/pipermail/vcs-pkg-discuss/2016-September/thread.html
And after you've read that, you can see that what I actually did was
even worse.
https://browse.dgit.debian.org/dgit.git/commit/?id=05db7ab051736bb106467e7f83b9e45684dd227c
You will want to look at the most recent dgit HEAD for the full
algorith, because there are other workarounds for bugs in dpkg-source
etc. Near here:
https://browse.dgit.debian.org/dgit.git/tree/dgit#n2365
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Ian Jackson writes ("Re: patches-applied historical imports (usd import)"):
> Urk! I have just filed:
...
> https://bugs.debian.org/850845
> dpkg-source fails to extract samba_3.6.5-2.dsc but exits status 0 -
> https://bugs.debian.org/850845
This was actually in dget, not dpkg-source. dpkg-source exits 2.
Ian.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Nish Aravamudan writes ("patches-applied historical imports (usd import)"):
> [stuff]
It occurs to me that it would be a good idea for you to check that
your importer produces identical answers to dgit. If nothing else,
from your point of view dgit can help you test that your importer
DTRT.
At the very least, for every .dsc which is successfully imported by
both dgit and your tool, the resulting imported tree object should be
identical.
Since I think dgit should be able to import every .dsc in the whole of
history (given new enough underlying tools), I would encourage you to
report every import failure as a bug against dgit. When considering
such bugs:
* I do not mind if you do not repro the bug on Debian
* Point me at the exact .dsc
* Run dgit with -D and include the output in the bug report
* Use Severity: important
Ideally you and I could agree on a commit structure for the imported
packages, and metadata processing rules, so that the commits are
identical (not just the trees).
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
Nish Aravamudan writes ("patches-applied historical imports (usd import)"):
> 1) Some source packages (bouncycastle, php7.0 are the ones I can think
> of off the top of my head) upstream tarballs contain .gitattributes
> files, which will change the behavior of git itself when checking out a
> branch.
I need to do some tests to see exactly how to work around this.
Can you point me at an affected source package version ?
> 2) How do we determine if a source package is 1.0 vs. 3.0? I am
> currently using `dpkg-source --print-format`, but have found one source
> package (util-linux 2.13~rc3-5), where dpkg-source emits:
>
> syntax error in /tmp/tmp3y515osf/util-linux-2.13~rc3/debian/control at
> line 14: duplicate field Depends found
>
> and thus we error out.
How exciting. dgit looks at debian/source/format, which is what
dpkg-source reads when generating the package.
> 3) Imagine the following graph in the git repository:
>
>A D
> ->o.f>o->
>^ . . ^
>| c e |
>o . . o
>^. .^
>| B |
>o o o
>^ ^ ^
>| | |
> ->o>o>o->
>a b d
...
> Let's assert that there is a problem with obtaining the patches-applied
> version of b. This can occur for (at least) the following reasons:
>
> i) as in 2), we might not be able to determine the source package format
> (implementation detail, to some degree), so are unable to correctly
> derive if there is a patches-applied state that is distinct from b.
You can use dpkg-source -x to extract the patches-applied state
corresponding to any .dsc, I think. Any case where you can't is a bug
in dpkg-source.
> ii) some patches may fail to apply with a trivial `quilt push`. This
> occurs with, at least, a historical publish of samba.
Do you have an example source package ?
> In theory, there are other reasons/cases where this might happen and the
> importer needs to never fail (so it is of some use to run automatically
> :)
I haven't run dgit's dsc importer on a whole historical archive but
Peter Green of Raspian has been running it and filing bugs. I haven't
seen such a bug recently so I hope it has been working for all the
packages he's seen.
> The questions I have relate to what to do when we encounter this
> situation, which in turn is divided into two parts:
I think this situation must be excluded.
This kind of difficulty is one of the main reasons for the failure of
previous efforts to transition universally to git. It is why dgit
does not currently work, in Debian, with a full history of each
package. Instead there's a bit of a bodge, with a locally generated
history. Joey Hess and I came up with this approach at the Vaumarcus
Debconf.
Eventually I intend to import the whole history but not yet.
If you intend to go forward without fully solving this problem, your
choices are not good.
> Let's presume that this failure is not persistent and that D is able to
> be imported successfully, we again have to make decisions about
> parenting. I think it only makes sense for one of c or f to exist, based
> upon what we decide is the right policy above.
This is particularly troublesome. When you can generate the missing
package, you will not be able to make it an ancestor of your existing
history.
You could rewrite the existing history and then pseudomerge the old
and new histories together, but that means duplicating most of the
history metadata.
If you wait with making an official import until it is suitable for
use with dgit, then I might consider using the relevant parts of your
imports history as the official Debian dgit history.
That would save you from the problem which I foresee in our future,
where Debian and Ubuntu have disjoint imports of the same history.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
~Johannes Schauer writes ("Re: Intent to commit craziness - source package
unpacking"):
> sbuild supports multiple backends. The default backend is schroot. How to get
> from "apt-get install sbuild" to actually building packages depends on the
> backend used. For the default schroot backend, sbuild provides the
> sbuild-createchroot tool which runs debootstrap and creates a schroot config
> file. So if you stay with the defaults, then you run:
>
> $ apt-get install sbuild
> $ sbuild-createchroot unstable /srv/chroot/unstable-amd64
WIBNI the final argument had a default ? :-)
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
I forgot one:
Ian Jackson writes ("Re: DEP14 policy for two dots"):
> A patches-unapplied tree:
>
> * produces confusing and sometimes misleading output from
>git grep, or (even if appropriate history is available)
>with git blame;
>
> * cannot be used with `git cherry pick ';
>
> * cannot be used as a basis for `git merge upstream/';
>
> * requires that the user not say `git diff upstream/master'
>but rather that they read patches in debian/patches;
>
> * cannot be directly edited by the user;
>
> * leaves the git tree dirty after every build with dpkg-buildpackage
>no matter how careful or tidy the package's build system.
* when built with the upstream build system (eg, for a GNU package,
./configure && make), silently and successfuly produces wrong
output - perhaps dangerously wrong output, such as binaries
lacking important security patches.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Raphael Hertzog writes ("Re: DEP14 policy for two dots"):
> Ok, can you prepare a patch for DEP-14 then? I'll apply it as it looks like
> a reasonable extension.
Attached. FYI I intend to implement this in dgit.
Thanks,
Ian.
>From 5c63400e9be8cb1532515764a1179730aed550fb Mon Sep 17 00:00:00 2001
From: Ian Jackson <ijack...@chiark.greenend.org.uk>
Date: Wed, 9 Nov 2016 18:36:23 +
Subject: [PATCH] DEP-14: Version -> refname mangling: Escape dots
Signed-off-by: Ian Jackson <ijack...@chiark.greenend.org.uk>
---
deps/dep14.mdwn | 25 ++---
1 file changed, 22 insertions(+), 3 deletions(-)
diff --git a/deps/dep14.mdwn b/deps/dep14.mdwn
index 4c7ce63..a7328a4 100644
--- a/deps/dep14.mdwn
+++ b/deps/dep14.mdwn
@@ -3,7 +3,7 @@
Title: Recommended layout for Git packaging repositories
DEP: 14
State: DRAFT
-Date: 2014-11-04
+Date: 2016-11-09
Drivers: Raphael Hertzog <hert...@debian.org>
URL: http://dep.debian.net/deps/dep14
Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep14.mdwn
@@ -60,8 +60,26 @@ Version mangling
When a Git tag needs to refer to a specific version of a Debian package,
the Debian version needs to be mangled to cope with Git's restrictions.
-The colon (`:`) needs to be replaced with a percent (`%`), and the tilde
-(`~`) needs to be replaced with an underscore (`_`).
+This mangling is deterministic and reversible:
+
+ * Each colon (`:`) is replaced with a percent (`%`)
+ * Each tilde (`~`) is replaced with an underscore (`_`)
+ * A hash (`#`) is inserted between each pair of adjacent dots (`..`)
+ * A hash (`#`) is appended if the last character is a dot (`.`)
+ * If the version ends in precisely `.lock`
+ (dot `l` `o` `c` `k`, lowercase, at the end of the version),
+ a hash (`#`) is inserted after the dot, giving `.#lock`.
+
+This can be expressed concisely in the following Perl5 statements:
+
+ y/:~/%_/;
+ s/\.(?=\.|$|lock$)/.#/g;
+
+The reverse transformation is:
+
+ * Each percent (`%`) is replaced with a colon (`:`)
+ * Each underscore (`_`) is replaced with a tilde (`~`)
+ * Each hash (`#`) is deleted
Packaging branches and tags
===
@@ -274,3 +292,4 @@ Changes
===
* 2014-11-05: Initial draft by Raphaël Hertzog.
+* 2016-11-09: Extended version mangling to troublesome dots - Ian Jackson.
--
2.10.2
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Ian Jackson writes ("Re: DEP14 policy for two dots"):
> Raphael Hertzog writes ("Re: DEP14 policy for two dots"):
> > On Fri, 04 Nov 2016, Ian Jackson wrote:
> > > My proposal is reversible. It does not need to be extensible.
> >
> > So what about "..."? Would it give ".#.#."?
>
> Yes. I said (fixing my bug):
>
> > Insert "#":
> >- between each pair of adjacent dots
> >- after any trailing dot
> >- before any leading dot
> - after the `.' of a trailing `.lock'
>
> The latter is necessary because git reserves .lock. (!)
> The summary is `add # after any troublesome dot' (discounting leading
> dots which you say are now illegal in Debian).
>
> I'm running some exhaustive tests to check that this rule is
> sufficient, because I'm not sure I trust the git docs.
I have now:
* Read the code in git upstream master. It's not particularly easy
to analyse conclusively, but I'm pretty sure it doesn't have any
special cases which involve longer strings than ".lock". I felt I
was able to identify the manpage rule corresponding to each element
of the logic.
* Conducted an exhaustive search of all strings of length 6
or less. Specifically, I generated all strings of between
zero and 6 characters from this set (in C notation):
"0123456789"
"abcdefghijklmnopqrstuvwxyz"
"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
".-+:~"
filtered them by whether the `parseversion' function in
dpkg likes them, escaped them with the following perl
script
#!/usr/bin/perl -pw
use strict;
y/:~/%_/;
s/\.(?=\.|$|lock$)/.#/g;
prepended "refs/tags/" to each one and fed them to
git-check-ref-format (modified to run in a pipe mode).
I also verified that when I don't escape ".lock", my exhaustive
search correctly detects the illegal ref name "refs/tags/1.lock"
genrated by version "1.lock" (and similar).
> The reverse rule is to convert _ and % and delete all #.
Quoted for completeness.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Raphael Hertzog writes ("Re: DEP14 policy for two dots"):
> On Fri, 04 Nov 2016, Ian Jackson wrote:
> > My proposal is reversible. It does not need to be extensible.
>
> So what about "..."? Would it give ".#.#."?
Yes. I said (fixing my bug):
> Insert "#":
>- between each pair of adjacent dots
>- after any trailing dot
>- before any leading dot
- after the `.' of a trailing `.lock'
The latter is necessary because git reserves .lock. (!)
The summary is `add # after any troublesome dot' (discounting leading
dots which you say are now illegal in Debian).
I'm running some exhaustive tests to check that this rule is
sufficient, because I'm not sure I trust the git docs.
> What's the rule to apply? if it's just to drop the "#", then yes
> it's reversible in a single step. If it's "s/\.#\./../g" then you need
> to do it multiple times until you no longer find ".#.".
The reverse rule is to convert _ and % and delete all #.
> > > My suggestion would be to allow "##".
> > > Thus my personal preference would be to replace ".." with ".#2e#".
> >
> > This is a bad idea because it (implicitly) makes the conversion
> > nondeterministic.
>
> We define the conversion rule in DEP-14. We can define it in a
> deterministic way.
If you define it in a deterministic way then it is by definition not
extensible, because all valid version strings have a definitive git
tag representation.
Unless by `extensible' you mean `we can update the rule if we discover
that some of the specified git tag representations are not accepted by
git', or `we can update the rule if the set of valid Debian version
strings is extended'. But this is true of any proposal, no matter
what the syntax is.
> I wanted something extensible because what's allowed in git ref names
> might evolve. It would not be the first time that a special syntax
> is introduced with a new feature.
I think the git folks are going to try not to further restrict the git
ref name syntax. After all, if they do restrict it, what about
existing tags with the now-forbidden names ?
> Which of # or = is more likely to be used for a new syntax/feature in git?
> My bet would go for "#" so that "=" is an even better choice.
I think = is more likely to be used for other things (both by git and
by others).
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: DEP14 policy for two dots
Nish Aravamudan writes ("DEP14 policy for two dots"):
> [ Raphael, apologies for sending twice, had a error in the headers in
> the prior one ]
>
> Not sure exactly where to ask this better than debian-devel, but I am
> working on an importer for the Ubuntu Server team which parses published
> versions of source packages in Debian and Ubuntu. I ran into an issue
> today where there is a published version of src:pcre3 with version
> '8.30..2'. `man git-check-ref-format` says that reference names "cannot
> have two consecutive dots .. anywhere." DEP14 specifies appropriate
> substitutions for : and ~, but it seems like .. should also be accounted
> for so I can correctly tag historic versions?
Urk. How exciting. I think we may need a more general escaping
scheme for these and other weirdnesses.
I have an interest as dgit uses DEP-14 tag escaping. I have CC'd the
vcs-pkg list.
tl;dr: I think we should insert `#' characters as needed.
Looking at git-check-ref-format(1) and
https://wiki.debian.org/Punctuation:
.special to git, generally permitted in versions,
and we want it usually to be literal - this is our problem
~special to git, permitted in versions, handled by DEP-14 as _
:special to git, epoch in versions, handled by DEP-14 as %
@special to git (although sometimes allowed), forbidden in versions
% _ not special to git but already used by DEP-14
# , =
not mentioned in the git manual as special, forbidden in versions
]not special to git, although [ is so let's not, eh ?
+ - not special to git, permitted in versions
" ' $ & ( ) * ; < > ? `
not mentioned in the git manual but troublesome shell
metacharacters which we would be insane to use here
[ / { }
interpreted specially by git some of the time,
forbidden in versions - not really useful
^ ? * \
all of these are forbiden by git, not permitted in versions
So I think in fact the only thing we have a problem with is multiple
dots. Looking at the summary above, we have the choice of one of
these:
# Its use as a shell comment character is fine, because when inside
a version tag it is always preceded by some string like
"debian/" or "upstream/". We would almost never need to put it
at the start of the encoded version string anyway, and we have
already tolerated a similar situation with ~.
There is possible confusion with HTML fragment identifiers, and
possibly in languages other than shell which use # for
comments (athough hopefuly they aren't dealing with our versions
as literals anyway).
Proposed rule:
Insert "#":
- between each pair of adjacent dots
- after any trailing dot
- before any leading dot
Examples:
8.30..2 => 8.30.#.2
8.30. => 8.30.#
.42 => #.42
, I would like to avoid this because lots of people are probably
using it as a list separator in ways that are difficult for us
to predict. If we used it, I would suggest the same as for #.
= In principle we could use this. I don't like it for a similar
reason to above. If we did use it it might look a bit like
Q-P encoding in some contexts.
@ We could use this although I wouldn't like to rely on the fact
that git dislikes `@{' and `@' but not @ followed by other
things.
% Reusing this is tempting because an epoch separator can never
follow `.', so any `%' after any `.' would unambiguously mean
`escape for dot rather than colon'. But in principle `.' can
occur at the start of the version, so `:3' and `.3' both =>
`%3'. There would have to be some horror of an exception rule.
(Although `:3' and `3' compare equal as Debian versions, they
are different textual strings and the tag needs to convey the
whole string.)
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Russ Allbery writes ("Re: Intent to commit craziness - source package
unpacking"):
> Ian Jackson <ijack...@chiark.greenend.org.uk> writes:
> > I'm not sure of the logic behind that. I don't think dgit helps much
> > with the kind of tasks that pristine-tar helps with.
>
> The main benefit of pristine-tar is that you can clone the development
> repository on any random host and do package development, build local
> packages for testing, and do a package upload without having to locate any
> additional pieces. I haven't been following dgit development closely, but
> it did sound like you were addressing the same use case.
dgit will fetch the orig tarball for you. So you don't have to
"locate" the pieces, but you do have to pay the download cost.
> If sbuild is now at the point where you can just apt-get install sbuild,
> run a single setup command, and be ready to build packages (which is where
> cowbuilder is right now), I personally would be happy to use something
> that's a bit closer to what the buildds are doing.
There is sbuild-setupchroot or something. I do find that I don't want
it because I like fiddling with the config, so I just use lower level
commands myself.
> However, cowbuilder does just work, in my experience, and it's nice to not
> have to change.
Of course.
I'm sorry I stepped into the argument about which chroot build tool is
best :-).
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Guido Günther writes ("Re: Intent to commit craziness - source package
unpacking"):
> On Mon, Oct 03, 2016 at 04:15:08PM +0100, Ian Jackson wrote:
> [..snip..]
> > Recommends: pristine-tar (>= 0.5)
...> >
> > pristine-tar has been declared unmaintainable by its original
> > author and abandoned.
...
> > Certainly dgit users do not need pristine-tar. But our dependency
> > system does not allow us to honour only direct Recommends and not
> > transitive ones.
>
> Looking at git.debian.org I found plenty of users. I did an archive
> import of sid during Debconf and was only ran into 20 pristine-tar
> failures (bugs yet to be filed).
Interesting.
> From the discussions at DC16 we're on our way to make this a hard
> dependency:
>
> http://lists.sigxcpu.org/pipermail/git-buildpackage/2016-July/000143.html
>
> The only thing I can think of (since we will keep support for not using
> pristine-tar nevertheless) is using:
>
> Recommends: pristine-tar | dgit
I'm not sure of the logic behind that. I don't think dgit helps much
with the kind of tasks that pristine-tar helps with.
> > Recommends: cowbuilder<= jessie
> > Recommends: cowbuilder | pbuilder | sbuild<= sid
...
> gbp buildpackage has integration with pbuilder/cowbuilder (via
> git-builder) and I know people are using it since its better integrated
> into gbp since you don't need additional and it's documented in the
> manual. The sbuild dependency is there to have people not pull in
> cowbuilder/pbuilder so they can use --git-builder=sbuild.
Ah.
> Not sure what can be done here.
It sounds like it should be left as-is, TBH.
> > Depends: devscripts
> >
> > devscripts is very full of commands with poor namespacing. It
> > also has an enormous dependency chain.
> >
> > Unfortunately dgit has a dependency on devscripts too. Maybe we
> > should work to take the pieces of devscripts that we really need
> > and put them in something else, or something.
>
> We're mostly using dch with "gbp dch" and I would also be happy to have
> the dependency chain shortened.
If it were my package, and that was all I depended on devscripts for,
I would drop it entirely. I think it's fair to expect someone who
uses `gbp dch' to install the package containing dch. But this is a
matter of taste.
dgit has a much harder dependency because dgit push uses dput.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package
unpacking"):
> Guido Günther writes ("Re: Intent to commit craziness - source package
> unpacking"):
> > 'gbp pq import' does have 'debian/patches' since it just puts the
> > patches that are in debian/patches on top of the unpatched source
> > tree. In contrast to your solution it doesn't try to be able to
> > roundtrip without changes for any given series on "gbp pq export". This
> > is only true if the series was also created by "gbp pq" (or adheres to
> > what git-format-patch does).
>
> Currently the output of dpkg-source --commit doesn't look much like
> the output of git-format-patch.
>
> I have tried to make dgit produce patches (when it needs to produce
> patches) that look like dpkg-source --commit. But maybe it should
> produce patches using git-format-patch (or that look like
> git-format-patch).
I have looked at the specs (including DEP-3) and the behaviour of the
various tools a bit more. I have decided that when I should replace
dgit's adhoc algorithm for generating a DEP-3 compliant commit
message, with a call to git-format-patch. (The actual patch body has
to come from dpkg-soure, because of the special handling of debian/,
etc. This code path is used when a dgit user is pushing and needs to
convert raw git commits into `3.0 (quilt)' patches.)
And, after having half-written a DEP-3 importer (ie a thing that turns
a DEP-3 patch message into a git commit), and investigating the
behaviour of various tools, I have decided I should just use gbp pq.
(This will replace the repeated use of dpkg-source --before-build as a
described in my previous mail.)
This will mean that dgit will grow a hard dependency on
git-buildpackage.
Because I'm a pernickety type of person I reviewed the dependencies of
git-buildpackage. I have some qualms about the following
dependencies:
Recommends: pristine-tar (>= 0.5)
pristine-tar has been declared unmaintainable by its original
author and abandoned.
IDK know what proportion of actual git trees that gbp users will
encounter would break without pristine-tar. Perhaps this
dependency can be demoted to Suggests, but I don't really know.
Certainly dgit users do not need pristine-tar. But our dependency
system does not allow us to honour only direct Recommends and not
transitive ones.
Recommends: cowbuilder<= jessie
Recommends: cowbuilder | pbuilder | sbuild<= sid
Many users of dgit will never want to build a package for upload.
This is probably true of gbp users too. I'm not sure why it's
valuable to have this as a Recommends dependency for gbp.
I think more people now are using sbuild. I'm not sure that
pulling in cowbuilder on systems where the user has not yet
installed such a tool is right.
Depends: devscripts
devscripts is very full of commands with poor namespacing. It
also has an enormous dependency chain.
Unfortunately dgit has a dependency on devscripts too. Maybe we
should work to take the pieces of devscripts that we really need
and put them in something else, or something.
Overall I don't think these are an impediment. But since I had done
the review, I thought I'd share my thoughts.
Regards,
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Guido Günther writes ("Re: Intent to commit craziness - source package
unpacking"):
> The authorship and subject information is taken from the patch if
> available or made up from the filename and committer when not.
>
> If patches in the series file are in subdirs of debian/patches we store
> that in "Gbp-Pq: topic" in the commit message. The patch name itself
> is stored in "Gbp-Pq: Name" we can reproduce it on "gbp pq export"
> independent from the patch's subject.
Right.
> > Bear in mind that because the output of gbp-pq import doesn't
> > contain debian/patches, I would need to rewrite its output (perhaps
> > with git-filter-branch).
>
> 'gbp pq import' does have 'debian/patches' since it just puts the
> patches that are in debian/patches on top of the unpatched source
> tree. In contrast to your solution it doesn't try to be able to
> roundtrip without changes for any given series on "gbp pq export". This
> is only true if the series was also created by "gbp pq" (or adheres to
> what git-format-patch does).
Currently the output of dpkg-source --commit doesn't look much like
the output of git-format-patch.
I have tried to make dgit produce patches (when it needs to produce
patches) that look like dpkg-source --commit. But maybe it should
produce patches using git-format-patch (or that look like
git-format-patch).
Thanks for suggesting libvirt as an example. I will play about with
it.
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package
unpacking"):
> Guido Günther writes ("Re: Intent to commit craziness - source package
> unpacking"):
> > Hi Ian,
>
> Hi, thanks for your attention.
Please disregard this email. I hadn't finished editing it!
Ian.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: Intent to commit craziness - source package unpacking
Ian Jackson writes ("Re: Intent to commit craziness - source package
unpacking"):
> Thanks for your comments. I feel unblocked :-).
So, I now intend to go and implement my plan.
There will be a little while (perhaps a few weeks) before I am in a
postion to release this in dgit 2.0. But after I do that, I will not
want to change the import algorithm again: it is important that the
imports be as stable as possible.
So now would be a good time for maintainers of git packaging tools (eg
git-dpm and gpb) to have an opinion about the detail of the generated
pseudohistory - in particular, the detail of the commits generated
from the `3.0 (quilt)' dpkg-source patches.
Also, I would welcome suggestions for what kind of compatibility test
I could perform on such a series of commits. dgit has an extensive
test suite (advertised via autopkgtest) which would be well-suited to
such a compatibility test.
An example of such a tree might be, "split out the patch queue part of
the git pseudohistory and feed it to gbp-pq, asking gbp-pq to
regenerate the source package, and expect the output to be identical
to the original input source package". Guido, if I get the
preconditions right, should I expect such a test to pass ? Is there a
risk it would break in the future due to changes in gbp-pq's
conversion algorithm ?
I confess that I am less familiar with git-dpm. I don't know what I
should be thinking about to try to make the output most useful to
git-dpm users.
(I also don't know whether the goals of helping git-dpm users and
gbp-pq users, and potentially users of any other tools, are in
conflict.
It would be annoying if these tools would disagree about the best form
of import of a particular patch queue: the import algorithm should be
the same for different dgit users, so I wouldn't be able to make this
a per-user configuration option and would have to choose..)
Thanks,
Ian.
--
Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Intent to commit craziness - source package unpacking
.) * No back doors into the innards of dpkg-source (nor changes to dpkg-dev) are required. * dgit does grow a dependency on Dpkg::Compression. * Knowledge of the source format embedded in dgit is is restricted to iterating over tarballs and manipulating debian/patches/series, which dgit already does. * dgit now depends on dpkg-source --before-build idempotently applying patches as they successively appear on debian/patches/series. * Perhaps the git commits generated by dgit to represent patches can be made to round-trip nicely into tools like git-dpm and git-buildpackage. I have found the information about tags in gbp-dch(1), but that doesn't seem like it's applicable. I have also found the information about tags in gbp-pq(1). From that it looks like I ought to generate "Gbp-Pq: Name" and "Gbp-Pq: Topic". * The scheme I describe avoids introducing a dependency from dgit to git-buildpackage. I might be able to replace the successive-patch-application part with an appropriate invocation of gbp-pq. Would that be better ? Bear in mind that because the output of gbp-pq import doesn't contain debian/patches, I would need to rewrite its output (perhaps with git-filter-branch). Comments welcome. Please be quick - this is very close to the top of my dgit todo list. Thanks, Ian. -- Ian Jackson <ijack...@chiark.greenend.org.uk> These opinions are my own. If I emailed you from an address @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: git, debian/ tags, dgit - namespace proposal [and 1 more messages]
Ian Jackson writes ("Re: git, debian/ tags, dgit - namespace proposal
[and 1 more messages]"):
> Colin Watson suggested (in pers.comm)
>pkg/debian/
> This is better but it still has a problem with collate order.
>
> It may seem a petty thing to worry about, but for the reasons I
> explain above I want something that sorts before `debian/'.
A conversation with Owen Dunn produced suggestions of `actual' and
`archive' as name components.
I'm considering:
archive/{debian,ubuntu}/
{debian,ubuntu}/archive/
I'm still considering the capitalisation idea.
Other suggestions welcome.
Thanks,
Ian.
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
git, debian/ tags, dgit - namespace proposal
Currently, the debian/ git tag namespace is used by a number of different tools, and can mean different things in different packages and sometimes even different things for the same package in different repos or trees. dgit produces, and the dgit git server requires, tags of this form, which refer to the dgit branch, which is roughly what some people call a `patches-applied packaging branch'. So far, dgit has been using debian/ for this. But this is a problem because this tag namespace is used for other purposes and has no clear meaning. I'm currently working on dgit native push support for users of 3.0-quilt+git-buildpackage, where this is particularly bad because it would involve dgit making two tags with the same name (!) Having considered things, I think it would be best for dgit to concede this area of namespace for all the existing uses. Declaring those existing uses wrong, just because they're varied and and mutually inconsistent, is not very helpful. Consequently I need a new namespace. I don't want to put `dgit' in the name because the format is not specific to dgit. It is simply the result of `dpkg-source -x' (only without the .pc directory which dpkg-source sometimes produces). I am currently thinking that dgit would start to use the tag namespace Debian/ instead. These tags would always refer to a Patches-applied packaging branch (where applicable containing a patch to add or update .gitignore). For other distros, I would likewise capitalise just the first character of the distro name (with `ucfirst' in Perl). So `Ubuntu/'. is translated as specified in DEP-14. So this message is: * A request for anyone to say if they know of a reason I shouldn't do this. * A declaration (currently, tentative) of intent to claim this part of the git tag namespace. * A proposal to update DEP-14 to declare that "vendor" in DEP-14 should start with a lowercase letter, and ideally, to reserve the corresponding starts-with-uppercase tags for dgit. Thanks, Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: git-buildpackage / dgit integration
Guido Günther writes (Re: git-buildpackage / dgit integration): On Sun, Aug 16, 2015 at 03:46:46PM +0100, Ian Jackson wrote: * .gitignore: We agree that the source package should contain .gitignore if the git tree does. (dgit requires this.) If the way that gbp does builds currently ends up with .gitignore missing from source packages, this needs to be fixed. I do wonder if this is an open issue? I looked at git trees containing .gitignore and they ended up in the source package. In case the Debian maintainer modifies .gitignore this will then end up as a quilt patch of course. Thanks for looking at this. My summary was inaccurate. If _upstream_ have a gitignore then it does indeed end up in the package. But if the gitignore is created or changed by the maintainer, the default -I omits it, I think. I haven't tried this with gbp yet. (I'm currently wrestling with some dgit misbehaviour with 3.0 single-debian-patch.) If I'm wrong then excellent. Ian. ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
