What to do with .git directories in source package uploads?
Ian, I noticed today that our Ubuntu git importer failed to import src:nplan (which is only in Ubuntu). This is because the upload (incorrectly) includes a .git directory. I see that dgit's behavior for the same DSC file with `dgit import-dsc` is to delete the .git directory from the source package before import. Is that a reflection of Debian policy? What I am hoping we can agree on is one of the two (it seems like) options for handling .git directories. While it does seem that one can trick git into representing .git in the treeish(s), `git push` will fail and it just sort of conflicts with the notion of git-based tooling, so I think actually having them in the source package treeish is a non-starter. The alternatives Robie and I devised are: 1) What dgit currently does, which is remove .git directories from source packages, with a warning. This does mean that an imported object does not match an uploaded exactly. Perhaps this is OK, but for instance, in the case of nplan (purely as an example), if a developer were to then use our git repository to develop a follow-on change, the resulting debdiff would include (surprisingly to the end-user) a removal of .git/) 2) In order from longest to shortest: Change any file in the source package matching ^\(\.+git\)$ to \.\1 [essentially prefixing a '.' to any file consisting of only leading '.' and the string 'git' ] The idea here is that while we can't save the .git directory as-is, we can losslessly convert the name (which might collide with other files, so we must rename anything in that 'namespace'). This won't help `dpkg-buildpackage` users, but if using our tooling, we can catch this and rename files back before building with a conversion, in order from shortest to longest: ^\.\(.+git\)$ -> \1 I'm wondering, are there are any explicit policy statements that say a build system cannot rely on a git directory being present? Say a debian/rules that does a `git describe`? I don't want to break such a source package if it does exist. I'm planning on implementing 2), as it leaves us the most wiggle-room and we haven't asserted anything about hash stability yet in our imports. But I'd love for some input on why 1) was chosen, and, as I mentioned earlier, if it's considered 'policy' or not. Let me know if I can clarify anything here! -Nish ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
On 09.01.2017 [14:33:29 -0800], Nish Aravamudan wrote:
> On 09.01.2017 [13:40:16 +], Ian Jackson wrote:
> > Nish Aravamudan writes ("patches-applied historical imports (usd import)"):
> > > ii) some patches may fail to apply with a trivial `quilt push`. This
> > > occurs with, at least, a historical publish of samba.
> >
> > Do you have an example source package ?
>
> I will re-run the import today and get that info for you.
src:samba 2:3.6.5-2 fails via `pull-debian-source` and (which the former
calls) `dpkg-source -x` and `quilt push`:
dpkg-source: info: the patch has fuzz which is not allowed, or is malformed
dpkg-source: info: if patch 'waf-as-source.patch' is correctly applied by
quilt, use 'quilt refresh' to update it
dpkg-source: info: restoring quilt backup files for waf-as-source.patch
dpkg-source: error: LC_ALL=C patch -t -F 0 -N -p1 -u -V never -E -b -B
.pc/waf-as-source.patch/ --reject-file=- <
samba-3.6.5/debian/patches/waf-as-source.patch gave error exit status 1
pull-debian-source: Error: Source unpack failed.
Thanks,
Nish
--
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
Re: patches-applied historical imports (usd import)
On 09.01.2017 [13:40:16 +], Ian Jackson wrote:
> Nish Aravamudan writes ("patches-applied historical imports (usd import)"):
> > 1) Some source packages (bouncycastle, php7.0 are the ones I can think
> > of off the top of my head) upstream tarballs contain .gitattributes
> > files, which will change the behavior of git itself when checking out a
> > branch.
>
> I need to do some tests to see exactly how to work around this.
> Can you point me at an affected source package version ?
I believe all php7.0 source packages will have a .gitattributes entry
containing 'ident' attributes amongst others.
> > 2) How do we determine if a source package is 1.0 vs. 3.0? I am
> > currently using `dpkg-source --print-format`, but have found one source
> > package (util-linux 2.13~rc3-5), where dpkg-source emits:
> >
> > syntax error in /tmp/tmp3y515osf/util-linux-2.13~rc3/debian/control at
> > line 14: duplicate field Depends found
> >
> > and thus we error out.
>
> How exciting. dgit looks at debian/source/format, which is what
> dpkg-source reads when generating the package.
Yep, based upon Sean's advice, I will switch to using that as well.
> > 3) Imagine the following graph in the git repository:
> >
> >A D
> > ->o.f>o->
> >^ . . ^
> >| c e |
> >o . . o
> >^. .^
> >| B |
> >o o o
> >^ ^ ^
> >| | |
> > ->o>o>o->
> >a b d
> ...
> > Let's assert that there is a problem with obtaining the patches-applied
> > version of b. This can occur for (at least) the following reasons:
> >
> > i) as in 2), we might not be able to determine the source package format
> > (implementation detail, to some degree), so are unable to correctly
> > derive if there is a patches-applied state that is distinct from b.
>
> You can use dpkg-source -x to extract the patches-applied state
> corresponding to any .dsc, I think. Any case where you can't is a bug
> in dpkg-source.
Right, that will get me to the fully patches-applied state, but what
about the intervening ones?
> > ii) some patches may fail to apply with a trivial `quilt push`. This
> > occurs with, at least, a historical publish of samba.
>
> Do you have an example source package ?
I will re-run the import today and get that info for you.
> > In theory, there are other reasons/cases where this might happen and the
> > importer needs to never fail (so it is of some use to run automatically
> > :)
>
> I haven't run dgit's dsc importer on a whole historical archive but
> Peter Green of Raspian has been running it and filing bugs. I haven't
> seen such a bug recently so I hope it has been working for all the
> packages he's seen.
And it's also very possible that it may not be a Debian publish, but an
Ubuntu publish where this occurs :)
> > The questions I have relate to what to do when we encounter this
> > situation, which in turn is divided into two parts:
>
> I think this situation must be excluded.
>
> This kind of difficulty is one of the main reasons for the failure of
> previous efforts to transition universally to git. It is why dgit
> does not currently work, in Debian, with a full history of each
> package. Instead there's a bit of a bodge, with a locally generated
> history. Joey Hess and I came up with this approach at the Vaumarcus
> Debconf.
>
> Eventually I intend to import the whole history but not yet.
>
> If you intend to go forward without fully solving this problem, your
> choices are not good.
Agreed.
> > Let's presume that this failure is not persistent and that D is able to
> > be imported successfully, we again have to make decisions about
> > parenting. I think it only makes sense for one of c or f to exist, based
> > upon what we decide is the right policy above.
>
> This is particularly troublesome. When you can generate the missing
> package, you will not be able to make it an ancestor of your existing
> history.
>
> You could rewrite the existing history and then pseudomerge the old
> and new histories together, but that means duplicating most of the
> history metadata.
>
>
> If you wait with making an official import until it is suitable for
> use with dgit, then I might consider using the relevant parts of your
> imports history as the official Debian dgit history.
>
> That would save you from the problem which I foresee in our future,
> where Debian and Ubuntu have disjoint imports of the same history.
Right, ideally, we'd be able (at some point in the fut
Re: patches-applied historical imports (usd import)
Hi Sean, On 08.01.2017 [21:34:50 -0700], Sean Whitton wrote: > Hello Nish, > > On Fri, Jan 06, 2017 at 02:26:29PM -0800, Nish Aravamudan wrote: > > 2) How do we determine if a source package is 1.0 vs. 3.0? I am > > currently using `dpkg-source --print-format`, but have found one source > > package (util-linux 2.13~rc3-5), where dpkg-source emits: > > I would just introspect for the debian/source/format file. If it has a > line "3.0 (.*)" then it's 3.0, otherwise it's 1.0 (for all packages that > were ACCEPTed to ftp-master). Ok, is this documented in the manual or anything (for my own edification). I spent some time searching, but didn't find anything definitive except for the hint provided by `dpkg-source --print-format`. And, presumably, if a historical publish exists without any debian/source/format file, it should be treated as 1.0? Ah, yep, I see that documented in `man dpkg-source` under DIAGNOSTICS. > > ii) some patches may fail to apply with a trivial `quilt push`. This > > occurs with, at least, a historical publish of samba. > > Have you considered obtaining the patches-applied tree using > `dpkg-source -x`? That applies the patches without using quilt(1), so > might workaround this sort of bug (if it didn't, the package would > FTBFS). Well, I would do that, but afaict, there is no way to tell dpkg-source to only apply one patch at a time? This is to go from a patches-unapplied state to the fully patches-applied state, commiting each patch application into the repository. Perhaps I missed a flag in the manpage, though? Thanks, Nish -- Nishanth Aravamudan Ubuntu Server Canonical Ltd ___ vcs-pkg-discuss mailing list vcs-pkg-discuss@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
patches-applied historical imports (usd import)
Hi Ian, I've got a few questions/comments about git-based publishing imports and history for patches-applied imports that I was hoping to bounce off you and other VCS folks. I apologize for the very long e-mail to follow! For context for everyone: I (along with Robie Basak and others) have been developing an importer that will take the publishing history of a source package and import the source package contents into a git tree, with tags for each publish and branches for each 'series'. That's a broad gloss of the details, of course, but probably sufficient. This could be, I think, of use to both Ubuntu and Debian. If you're not interested in this, though, please feel free to disregard :) On to the questions/comments: 1) Some source packages (bouncycastle, php7.0 are the ones I can think of off the top of my head) upstream tarballs contain .gitattributes files, which will change the behavior of git itself when checking out a branch. This is not by definition a problem, except that to get to a fully patches-applied state, I believe you must be checkout an appropriate commit to be at (meaning you're adjusting the working directory's contents) -- which may then differ from what is shipped by the upstream tarball. I have seen this with eol adjustments, and much more annoyingly (because git knows it is doing it, while vi/emacs do not) the special ident handling. For now, we've added the following to our git repository's .git/info/attributes at checkout time (using our wrapper to `git clone`): * -ident * -text * -eol In other words, the underlying issue is that the upstream uses git as well, and their git 'configuration' (not necessarily just .gitconfig) will interfere with the behavior of any git using the upstream contents in the working tree. Does the above seem reasonable? Afaict, there is no way for me to really enforce the above in the repository itself, without patching the upstream source. 2) How do we determine if a source package is 1.0 vs. 3.0? I am currently using `dpkg-source --print-format`, but have found one source package (util-linux 2.13~rc3-5), where dpkg-source emits: syntax error in /tmp/tmp3y515osf/util-linux-2.13~rc3/debian/control at line 14: duplicate field Depends found and thus we error out. 3) Imagine the following graph in the git repository: A D ->o.f>o-> ^ . . ^ | c e | o . . o ^. .^ | B | o o o ^ ^ ^ | | | ->o>o>o-> a b d Each o is a commit in the repository a, b, d are patches-unapplied imports of publishes for a given release, which are on a fast-forwarding, branch-unnapplied branch. A is the corresponding patches-applied import of a (with each o reflecting one patch application from the source package). D is the corresponding patches-applied import of d (with each o reflecting one patch application from the source package). c, e and f are for demonstration purposes and do not necessarily exist (except as discussed below). Ideally, we'd have a fast-forwarding branch for the patches-applied imports, as well. Let's assert that there is a problem with obtaining the patches-applied version of b. This can occur for (at least) the following reasons: i) as in 2), we might not be able to determine the source package format (implementation detail, to some degree), so are unable to correctly derive if there is a patches-applied state that is distinct from b. ii) some patches may fail to apply with a trivial `quilt push`. This occurs with, at least, a historical publish of samba. In theory, there are other reasons/cases where this might happen and the importer needs to never fail (so it is of some use to run automatically :) The questions I have relate to what to do when we encounter this situation, which in turn is divided into two parts: i) Do we want to 'tag' this failed-to-apply patches-applied import in the repository (currently, every successful patches-applied import is tagged as 'applied/')? This is important for semantics for end-users (and the importer itself). - We assert, currently that tagged objects in the tree correspond to the source package as published in Launchpad/archive. Tagging this failed-to-apply state as that, would violate that assertion. - We also rely (implementation detail) on being able to find 'nearest' publishes by tag name, which sort of leads into the next issue... ii) What should happen to the branch? - if the branch is left at A, then (even if only momentarily), upon finishing the import at B, the branch does not reflect the latest state in Launchpad relative to the importer's progress. - if the branch is left at B, then it is no longer fast-forwarding, as there is no connection between A and B. - if the branch is left at B, and we add the connection c to make it fast-forwarding, we violate a different semantic we assert about parenting relationships in our
Re: DEP14 policy for two dots
On 09.11.2016 [21:27:14 +], Ian Jackson wrote:
> Raphael Hertzog writes ("Re: DEP14 policy for two dots"):
> > Ok, can you prepare a patch for DEP-14 then? I'll apply it as it looks like
> > a reasonable extension.
>
> Attached. FYI I intend to implement this in dgit.
Thank you! We will follow the same in the Ubuntu tooling used by the Server
Team.
--
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd
___
vcs-pkg-discuss mailing list
vcs-pkg-discuss@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/vcs-pkg-discuss
