Re: Licensing of old Debconf::Gettext module in dpkg

[Joey Hess]

Guillem Jover wrote:
> Hi Joey and Nicolas!
> 
> Long time ago the Debconf::Gettext [O] module got imported into the dpkg
> code base [I] with some modifications from Nicolas. The current module
> in the dpkg code base is [C], which has seen substantial modifications
> by me since its import.
> 
> This is the only BSD-2-clause file in the whole code base, so it would
> be nice if it could get relicensed into GPL-2+ like almost all other
> code in dpkg, to simplify the licensing. Would both of you be fine
> with such relicense?
> 
> Thanks,
> Guillem
> 
> [O] 
> 
> [I] 
> 
> [C] 

I'm not sure if there is enough of my code left here, or enough
expressiveness, for me to hold a legitimate copyright on it. But I
am fine with relicensing it GPL-2+.

- 
see shy jo


signature.asc
Description: PGP signature

Re: Bug#714409: libgtk-3-0: "triggers ci file contains unknown directive `interest-noawait'" on install (needs newer dpkg)

[Joey Hess]

Michael Biebl wrote:
> Couldn't debhelper/dh_installdeb generate that Pre-Depends via
> ${misc:Pre-Depends} if debian/*.triggers contains noawait?
> That sounds better to me then hard-coding the dependency.
> 
> This would have the additional benefit, that in jessie+1, a simple
> rebuild would be sufficient to get rid of the dependency again without
> having to touch the source packages.
> 
> Joey, dpkg maintainers, what do you think?

Fine by me, a patch would be applied.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Bug#560317: dpkg-reconfigure does not set DPKG_MAINTSCRIPT_PACKAGE (et al)

[Joey Hess]

Colin Watson wrote:
> + my $templates=`dpkg-query --control-path $pkg templates`;

> + my $path_script=`dpkg-query --control-path $pkg $script`;

> + open (QUERY, '-|', 'dpkg-query', '-W',
> + '-f', '${Package} ${Triggers-Pending}\n');

It's a shame that dpkg-query is so slow. Cold cache (which will be
typical), it takes something like a second to run; warm cache around .3
seconds. Overhead includes reading the entire status file. The above
code runs it 5 times in all.

So, it would be worth only running it once and picking out the
appropriate filenames from its output. At least this would reduce the
number of runs to 3, and dpkg-reconfigure only gets 1.6 seconds slower,
rather than 2.2 seconds slower.


PS, cdebconf includes a dpkg-reconfigure now and will need the same
changes. Probably we need to make sure to communicate substantial
changes like this with Regis at least until cdebconf takes over from
debconf.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Translated manual pages & dh_installman behaviour

[Joey Hess]

Raphael Hertzog wrote:
> Joey, would it be possible to also extract the language code from the
> path when the dirname matches m{/man/([a-z][a-z](?:_[A-Z][A-Z])?)/man\d$} ?
> 
> It's specific enough to avoid wrong guesses and it seems to make sense
> when you want to use dh_installman to install manual pages taken out of
> debian/tmp as is the case here:
> $ head -n 4 debian/dpkg.manpages 
> debian/tmp/usr/share/man/*/*/dpkg-deb.1
> debian/tmp/usr/share/man/*/dpkg-deb.1
> debian/tmp/usr/share/man/*/*/dpkg-divert.8
> debian/tmp/usr/share/man/*/dpkg-divert.8

If your manual pages are already being installed to the correct
directories, you don't need to use dh_installman to install them,
you can just use dh_install. dh_installman will find man pages installed
by dh_install and do utf-8 fixups etc to them as needed.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Multiarch file overlap summary and proposal

[Joey Hess]

Goswin von Brederlow wrote:
> pkg:arch will still be unique and the dpkg/apt output will use the
> architecture where required for uniqueness. So I think that after some
> getting used to it it will be clear enough again.

Here are a few examples of the problems I worry about. I have not
verified any of them, and they're clearly biased toward code I am
familiar with, which suggests there are many other similar problems.

* Puppet not only installs packages, it may remove them. A puppet config
  that does dpkg --purge foo will fail if multiarch is enabled, now
  it needs to find and remove foo:*

* dpkg-repack pkg:arch will create a package with that literal name (or fail)

* dpkg-reconfigure probably can't be used with M-A same packages.
  debconf probably generally needs porting to multiarch.

* tasksel uses dpkg --query to work out if a task's dependencies are
  installed. In the event that a library is directly part of a task,
  this will fail when multiarch is enabled.

* Every piece of documentation that gives commands lines manipulating
  library packages is potentially broken.

Seems like we need a release where multiarch is classed as an
experimental feature, which when enabled can break the system. But the
sort of problems above are the easy to anticipate ones; my real worry is
the unanticipated classes of problems. Especially if we find intractable
problems or levels of complexity introduced by dropping the unique
package name invariant.

My nightmare scenario is that we release with multiarch, discover that
it's a net negative for our users (proprietary software on amd64 aside,
nearly all the benefits are to developers AFAICS), and are stuck with it.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Multiarch file overlap summary and proposal (was: Summary: dpkg shared / reference counted files and version match)

[Joey Hess]

Guillem Jover wrote:
> Aside from what I said on my other reply, I just wanted to note that
> this seems to be a recurring point of tension in the project when it
> comes to archive wide source package changes, where supposed short
> term convenience (with its usually long term harmful effects) appears
> to initially seduce people over what seems to be the cleaner although
> slightly a bit more laborious solution.
> 
> Other recent-ish incarnations of this tension could be the build-arch
> build-indep targets, or the build flag settings; where the former got
> recently resolved so that the right thing to do is for *all* packages
> needing to eventually support those targets, or for the latter which
> got switched from the seemingly more convenient to the more laborious
> but correct solution, that is, *all* packages need to set those build
> flags by themselves.
> 
> This is a fundamental issue with how our source packages are handled,
> and the freedom and power it gives to experiment and implement them
> whatever way the maintainer wants, has the price that doing some
> archive wide changes is sometimes more costly, than changing something
> centrally and be done with it. But trying to workaround this by coming
> up with stacks of hacked up solutions will not solve that fundamental
> issue, and this kind of tension will keep coming up again and again,
> as long as the foundation is not reworked. Either that, or the project
> needs to accept that fact and learn to live with this kind of changes,
> with patience.

Very interesting mail. While I certianly agree with your examples, it's
worth remembering the counterexample of the /usr/doc transition which
took approximately 5 years to complete[1], and probably could have been
accomplished quickly and without pain with a simple hack to dpkg.

Anyway, my worry about the refcounting approach (or perhaps M-A: same in
general) is not the details of the implementation in dpkg, but the added
mental complexity of dpkg now being able to have multiple distinct
packages installed under the same name. I had a brief exposure to rpm,
which can install multiple versions of the same package, and that was
the main cause of much confusing behavior in rpm. While dpkg's invariant
that all co-installable package names be unique (and have unique files)
has certianly led to lots of ugly package names, it's kept the users'
and developers' mental models quite simple.

I worry that we have barely begun to scratch the surface of the added
complexity of losing this invariant.

-- 
see shy jo

[1] To the extent it was ever completed.. master.debian.org still has
a vestigial /usr/doc/


signature.asc
Description: Digital signature

Re: dpkg-reconfigure does not set DPKG_MAINTSCRIPT_PACKAGE (et al)

[Joey Hess]

I keep seeing people complain that this bug is not fixed, but every
time I look at it, I find myself unable to fix it, and with issues like
these:

* Where are these variables documented?
  (Appears that they're basically not, which makes it sorta hard to
  know that they are being set, or used, as intended.)

* How is debconf supposed to set DPKG_MAINTSCRIPT_ARCH? If it has to call
  dpkg --print-architecture every time, that just makes every run slower,
  which would seem to be the opposite of the point of having such a
  variable in the first place. But it cannot be that simple anyway, with
  multiarch. What uses DPKG_MAINTSCRIPT_ARCH anyway?

* What should DPKG_MAINTSCRIPT_NAME be set to when the config script
  is being run?

* How is it appropriate for dpkg-maintscript-helper etc to be already using
  these variables when debconf is not yet setting them? Would it make
  more sense for dpkg-reconfigure to not set them, and
  dpkg-maintscript-helper etc to be a no-op when a package is being
  reconfigured?

* Nobody has ever addressed my concern that, if dpkg-reconfigure runs
  dpkg --configure --pending, this will result in it confusingly doing
  other things than configuring the specified package.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Semantic change for dpkg triggers?

[Joey Hess]

Raphael Hertzog wrote:
> I don't know anything about Haskell. What does the trigger do? Is it some
> sort of non-optional byte-compilation?

It makes the cabal package manager aware of the package installed by the
dpkg package manager, iirc.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Semantic change for dpkg triggers?

[Joey Hess]

Raphael Hertzog wrote:
> My question is thus: are there triggers currently in use where this
> relaxed behaviour would be wrong? Or more simply are there packages which
> are really not working before the processing of their awaited triggers?

python-support seems to need that; python does not see files in
/usr/share/pyshared/, so until update-python-modules has run, a python
module cannot be used.

ghc libraries could need it, if any package depends on such a library
being available to be compiled against at installation time. I don't
know if any packages use ghc libraries like this currently, but I have
considered writing one that does, if I ever found the time to rewrite
ikiwiki in haskell.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: trigger processing

[Joey Hess]

Phillip Susi wrote:
> I've noticed triggers being invoked repeatedly during upgrades rather
> than once at the end, as they are supposed to.  I started looking at
> /var/log/dpkg.log to try and figure out why.

Becaue apt has not been changed to tell dpkg to defer trigger processing,
and to them run them all at the end. Or rather, it has, see
http://bugs.debian.org/473461 -- but they do not have the options
enabled by default.

DPkg
{
   // let apt aggressivly use dpkg triggers
   NoTriggers "true";
   NoConfigure "true";
   ConfigurePending "true";

-- 
see shy jo


signature.asc
Description: Digital signature

[PATCH] improve dpkg-source documentation of git format

[Joey Hess]

Based on a patch by Tanguy Ortolo.

---
 man/dpkg-source.1 |   12 +---
 1 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/man/dpkg-source.1 b/man/dpkg-source.1
index 3d87bc5..69b84fe 100644
--- a/man/dpkg-source.1
+++ b/man/dpkg-source.1
@@ -555,12 +555,18 @@ The generated .dsc file will contain this value in its 
\fIFormat\fP field
 and not "3.0 (custom)".
 .
 .SS Format: 3.0 (git)
-This format is experimental. It uses a bundle of a git repository to hold
-the source of a package.
+This format is experimental.
+.PP
+A source package in this format consists of a
+single bundle of a git repository \fB.git\fP to hold the source of a package.
+There may also be a \fB.gitshallow\fP file listing revisions for a shallow
+git clone.
 .PP
 .B Extracting
 .PP
-The bundle is cloned to a new git repository.
+The bundle is cloned as a git repository to the target directory.
+If there is a gitshallow file, it is installed as `.git/shallow` inside
+the cloned git repository.
 .PP
 Note that by default the new repository will have the same branch checked
 out that was checked out in the original source. (Typically "master", but
-- 
1.7.1


-- 
see shy jo


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100817210044.ga17...@gnu.kitenet.net

[PATCH] fix path to gitshallow file

[Joey Hess]

It was looking in the current directory, which works most of the time,
but not always.
---
 scripts/Dpkg/Source/Package/V3/git.pm |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/scripts/Dpkg/Source/Package/V3/git.pm 
b/scripts/Dpkg/Source/Package/V3/git.pm
index c7429c1..de80b4e 100644
--- a/scripts/Dpkg/Source/Package/V3/git.pm
+++ b/scripts/Dpkg/Source/Package/V3/git.pm
@@ -225,7 +225,7 @@ sub do_extract {
 # Move shallow info file into place, so git does not
 # try to follow parents of shallow refs.
 info(_g("setting up shallow clone"));
-system("cp", "-f",  $shallow, "$newdirectory/.git/shallow");
+system("cp", "-f",  $dscdir.$shallow, "$newdirectory/.git/shallow");
 $? && subprocerr("cp");
 }
 
-- 
1.7.1


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100725182041.ga21...@gnu.kitenet.net

[PATCH] add missing paragraph break to man page

[Joey Hess]

---
 man/dpkg-source.1 |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/man/dpkg-source.1 b/man/dpkg-source.1
index ac3ff82..3d87bc5 100644
--- a/man/dpkg-source.1
+++ b/man/dpkg-source.1
@@ -587,6 +587,7 @@ include. It may also be any parameter that can be passed to
 the master branch, use \-\-git\-ref=master. To include all tags and
 branches, except for the private branch, use \-\-git\-ref=\-\-all
 \-\-git\-ref=^private
+.TP
 .BI \-\-git\-depth= number
 Creates a shallow clone with a history truncated to the specified number of
 revisions.
-- 
1.7.1


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100725181746.ga21...@gnu.kitenet.net

dpkg-buildflags

[Joey Hess]

Some questions I have..

* Do you anticipate adding more flags later? [1]

* Do you think it would be a good idea for packages to set all flags
  dpkg-buildflags supports?

* If packages should set all the flags, have you considered having a
  mode where it lists them all (like dpkg-architecture -l), perhaps
  with export statements?


[1] automake documentation mentions "CCASFLAGS, CFLAGS, CPPFLAGS,
CXXFLAGS, FCFLAGS, FFLAGS, GCJFLAGS, LDFLAGS, LFLAGS, LIBTOOLFLAGS,
OBJCFLAGS, RFLAGS, UPCFLAGS, and YFLAGS"

-- 
see shy jo


signature.asc
Description: Digital signature

Re: RFC: 3.0 (git), now with bundles

[Joey Hess]

Attached updated patch adds the ability to limit the history depth that
is included in the git bundle (via --git-depth), as well as fully
control which tags/branches/objects are included/excluded in the bundle
(via --git-ref).

A new .gitshallow file is added to the source package to record info
git needs for shallow clones.


j...@gnu:~/src>cat ttyrec/debian/source/format 
3.0 (git)
j...@gnu:~/src>cat ttyrec/debian/source/options 
git-ref=debian
git-ref=mytag
j...@gnu:~/src>dpkg-source --git-depth=3 -b ttyrec
dpkg-source: info: using options from ttyrec/debian/source/options:
--git-ref=debian --git-ref=mytag
dpkg-source: info: using source format `3.0 (git)'
dpkg-source: info: creating shallow clone with depth 3
dpkg-source: info: bundling: debian mytag
Counting objects: 38, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (26/26), done.
Writing objects: 100% (38/38), 49.49 KiB, done.
Total 38 (delta 5), reused 35 (delta 4)
dpkg-source: info: building ttyrec in ttyrec_1.0.8-4.dsc
j...@gnu:~/src>cat ttyrec_1.0.8-4.gitshallow
2387fc0d9c95c8de1b9afc60897fb3573761a766
79b1f1aff6d21b6028a3948b223d200925e56fa1
c15672934bfa53a33d4c329c641d8d3943194715
d8366ac884b62a90e62e506016c5b1796b7de3d2
j...@gnu:~/src>dpkg-source -x ttyrec_1.0.8-4.dsc 
dpkg-source: warning: extracting unsigned source package (ttyrec_1.0.8-4.dsc)
dpkg-source: info: extracting ttyrec in ttyrec-1.0.8
dpkg-source: info: cloning ttyrec_1.0.8-4.git
dpkg-source: info: setting up shallow clone
j...@gnu:~/src>cd ttyrec-1.0.8 
j...@gnu:~/src/ttyrec-1.0.8>git log --pretty=oneline
j...@gnu:~/src/ttyrec-1.0.8>git log mytag --pretty=oneline
c54af7a7a1797e41fabfa9625db9c67617d89bd3 update
7f73eb6ce9522d99ee108dd89794198c7f095d60 update
135bef2b5363925a2b1a444970b3f109e5bee5d8 add
c15672934bfa53a33d4c329c641d8d3943194715 add
j...@gnu:~/src/ttyrec-1.0.8>git log --pretty=oneline
807f48d0db54f314d4397340467d8de7976940de update
c54af7a7a1797e41fabfa9625db9c67617d89bd3 update
7f73eb6ce9522d99ee108dd89794198c7f095d60 update
135bef2b5363925a2b1a444970b3f109e5bee5d8 add
c15672934bfa53a33d4c329c641d8d3943194715 add

Note that in this example, 4 revs of mytag were stored, because git
clone --depth always stores one more rev than requested. 5 revs of the
debian branch were stored, because it shares history with mytag. The
debian/source/options caused all other tags and branches to be omitted,
so their history is not included.

-- 
see shy jo
From 1d1da3324366280b1cfd79bd0508377dccc3bbfb Mon Sep 17 00:00:00 2001
From: Joey Hess 
Date: Tue, 1 Jun 2010 16:01:35 -0400
Subject: [PATCH] modify 3.0 (git) to use git bundle

Much better than the old approach of a tarball of the .git repository,
the git bundle format is simple to understand and work with, and
doesn't need to be sanitized for security. Much code went away.

Supports limiting history depth by creating a shallow clone.
---
 man/dpkg-source.1 |   44 +-
 scripts/Dpkg/Source/Package/V3/git.pm |  277 +++--
 2 files changed, 131 insertions(+), 190 deletions(-)

diff --git a/man/dpkg-source.1 b/man/dpkg-source.1
index 5e4d30b..4a2c01d 100644
--- a/man/dpkg-source.1
+++ b/man/dpkg-source.1
@@ -531,13 +531,49 @@ in the current directory. At least one file must be given.
 The generated .dsc file will contain this value in its \fIFormat\fP field
 and not "3.0 (custom)".
 .
-.SS Format: 3.0 (git) and 3.0 (bzr)
-Those formats are experimental. They generate a single tarball
-containing the corresponding VCS repository.
+.SS Format: 3.0 (git)
+This format is experimental. It uses a bundle of a git repository to hold
+the source of a package.
 .PP
 .B Extracting
 .PP
-The tarball is unpacked and then the VCS is used to checkout the current
+The bundle is cloned to a new git repository.
+.PP
+Note that by default the new repository will have the same branch checked
+out that was checked out in the original source. (Typically "master", but
+it could be anything.) Any other branches will be available, under as
+`remotes/origin/`
+.PP
+.B Building
+.PP
+Before going any further, some checks are done to ensure that we
+don't have any non-ignored uncommitted changes.
+.PP
+\fBgit-bundle\fP(1) is used to generate a bundle of the git repository.
+By default, all branches and tags in the repository are included in the
+bundle.
+.PP
+.B Build options
+.TP
+.BI \-\-git\-ref= ref
+Allows specifying a git ref to include in the git bundle. Use disables
+the default behavior of including all branches and tags. May be specified
+multiple times. The \fIref\fP can be the name of a branch or tag to
+include. It may also be any parameter that can be passed to
+\fBgit-rev-list\fP(1). For example, to include only
+the master branch, use \-\-git\-ref=master. To include all tags and
+branches, except for the private branch, use \-\-git\-ref=\-\-all
+\-\-git\-ref=^private
+.BI \-\-git\-depth= number
+Cre

Re: RFC: 3.0 (git), now with bundles

[Joey Hess]

Tollef Fog Heen wrote:
> | * All history is currently included (via the --all switch to git-bundle),
> |   but I plan to add a format-specific dpkg-source option, to allow
> |   filtering of what is included in the bundle.
> 
> Maybe allow a filter to be specified in debian/source somewhere?

It should be possible to use debian/source/options to specify such
options. (Or debian/source/local-options)

-- 
see shy jo


signature.asc
Description: Digital signature

Re: RFC: 3.0 (git), now with bundles

[Joey Hess]

Peter Krefting wrote:
> How does this interact with the actual Git repository, when it comes
> to detecting patches to upstream and such? I haven't really read up
> on how the format is specified, so please point me in that direction
> if what I am asking is obvious.

It *is* the actual git repository, in basically all respects.

> I have some software packaged for Debian, and I usually have the
> upstream master branch as master, and a debian branch with Debian
> changes (merging the upstream release as they come). Or, if the
> upstream source is not in Git already, a branch of imported
> tarballs. Or, when I am the upstream, I have everything (including
> the debianization) on the master branch.
> 
> When I want to package such software, do I just clone the repository
> on the correct commit and have dpkg handle it? Does it still need
> the upstream tarball, or will the entire source be reconstructed
> from the git archive?

You checkout the debian branch or whatever branch you have the
debianization on, and build the package from there.

(You may use pristine-tar to store the upstream tarball in the git repo
if desired, but this is not strictly necessary.)

-- 
see shy jo


signature.asc
Description: Digital signature

RFC: 3.0 (git), now with bundles

[Joey Hess]

A while ago there was discussion onlist about making 3.0 (git) use
git-bundle as its repository serialization format. I think that is a
great improvement from the old ad-hoc format. Attached patch implements
it.

Notes:

* I have not tried to keep compatability with old 3.0 (git) packages.
  The format is experimental..
* The bundle file has a .git extension. There is no real standard for
  extensions of git bundles, though .bundle is sometimes used. I choose .git
  for namespace reasons. Also, a .git bundle file can be cloned just like
  a .git bare repo directory.
* All tags and local branches are included in the bundle, and are restored
  from the bundle. Whatever branch was checked out when the source package
  was built will be the one checked out when the source package is unpacked.
* Since it clones the bundle to unpack, the bundle is set as the origin
  repository. The remote git repo configuration is not preserved. (Maybe
  enhance it so Vcs-Git is automatically set up as a remote?)
* All history is currently included (via the --all switch to git-bundle),
  but I plan to add a format-specific dpkg-source option, to allow
  filtering of what is included in the bundle.

Demo:

j...@gnu:~/src/ttyrec>cat debian/source/format
3.0 (git)
j...@gnu:~/src/ttyrec>git branch
* debian
  master
  upstream
j...@gnu:~/src/ttyrec>dpkg-source -b .
dpkg-source: info: using source format `3.0 (git)'
Counting objects: 204, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (99/99), done.
Writing objects: 100% (204/204), 79.94 KiB, done.
Total 204 (delta 102), reused 160 (delta 95)
dpkg-source: info: building ttyrec in ttyrec_1.0.8-4.dsc

j...@gnu:~/src/ttyrec>cat ../ttyrec_1.0.8-4.dsc
Format: 3.0 (git)
Source: ttyrec
Binary: ttyrec
Architecture: any
Version: 1:1.0.8-4
Maintainer: NIIBE Yutaka 
Standards-Version: 3.8.0
Build-Depends: debhelper (>= 5), sharutils
Checksums-Sha1: 
 d1e2f70b56c041a4360ebb8f2a9d45c8d28faf0f 83369 ttyrec_1.0.8-4.git
Checksums-Sha256: 
 3f489ba4f49606369adb8c17ed45885bb7ead9e151b174ec955198ced1867f8b 83369 
ttyrec_1.0.8-4.git
Files: 
 ec5322b0da83278f42471cc35d8bd41c 83369 ttyrec_1.0.8-4.git
j...@gnu:~/src/ttyrec>file ../ttyrec_1.0.8-4.git
../ttyrec_1.0.8-4.git: Git bundle

j...@gnu:~/src/tmp>dpkg-source -x ../ttyrec_1.0.8-4.dsc 
dpkg-source: warning: extracting unsigned source package (../ttyrec_1.0.8-4.dsc)
dpkg-source: info: extracting ttyrec in ttyrec-1.0.8
dpkg-source: info: cloning ttyrec_1.0.8-4.git
Initialized empty Git repository in /home/joey/src/tmp/ttyrec-1.0.8/.git/
j...@gnu:~/src/tmp>cd ttyrec-1.0.8 
j...@gnu:~/src/tmp/ttyrec-1.0.8>ls
Makefile  README  debian/  io.c  io.h  ttyplay.1  ttyplay.c  ttyrec.1  ttyrec.c 
 ttyrec.h  ttytime.1  ttytime.c
j...@gnu:~/src/tmp/ttyrec-1.0.8>git branch -a
* debian
  remotes/origin/HEAD -> origin/debian
  remotes/origin/debian
  remotes/origin/master
  remotes/origin/upstream
j...@gnu:~/src/tmp/ttyrec-1.0.8>git tag
debian/1.0.8-5
upstream/1.0.8

-- 
see shy jo
From 4ea8342f75f1fd5119b1a0732fb0227b0ab06382 Mon Sep 17 00:00:00 2001
From: Joey Hess 
Date: Tue, 1 Jun 2010 16:01:35 -0400
Subject: [PATCH] modify 3.0 (git) to use git bundle

Much better than the old approach of a tarball of the .git repository,
the git bundle format is simple to understand and work with, and
doesn't need to be sanitized for security. Much code went away.
---
 man/dpkg-source.1 |   28 -
 scripts/Dpkg/Source/Package/V3/git.pm |  205 +++--
 2 files changed, 40 insertions(+), 193 deletions(-)

diff --git a/man/dpkg-source.1 b/man/dpkg-source.1
index 5e4d30b..8d52737 100644
--- a/man/dpkg-source.1
+++ b/man/dpkg-source.1
@@ -531,13 +531,33 @@ in the current directory. At least one file must be given.
 The generated .dsc file will contain this value in its \fIFormat\fP field
 and not "3.0 (custom)".
 .
-.SS Format: 3.0 (git) and 3.0 (bzr)
-Those formats are experimental. They generate a single tarball
-containing the corresponding VCS repository.
+.SS Format: 3.0 (git)
+This format is experimental. It uses a bundle of a git repository to hold
+the source of a package.
 .PP
 .B Extracting
 .PP
-The tarball is unpacked and then the VCS is used to checkout the current
+The bundle is cloned to a new git repository.
+.PP
+Note that by default the new repository will have the same branch checked
+out that was checked out in the original source. (Typically "master", but
+it could be anything.) Any other branches will be available, under as
+`remotes/origin/`
+.PP
+.B Building
+.PP
+Before going any further, some checks are done to ensure that we
+don't have any non-ignored uncommitted changes.
+.PP
+\fBgit-bundle\fP(1) is used to generate a bundle of the git repository.
+All branches and tags in the repository are included in the bundle.
+.SS Format: 3.0 (bzr)
+This format is experimental. It generates a single tarball
+containin

Re: [PATCH/RFC] dpkg-shlibdeps: ignore shell scripts

[Joey Hess]

Raphael Hertzog wrote:
> Why? I mean dh_shlibdeps already does the right thing and is passing
> binaries only to dpkg-shlibdeps.

At the expense of wasting time running file on everything to determine
which are binaries, which has been marked "TODO:slow" in the source
forever.

And then, file is probably not the best test, the best test would be an
identical one to that used by dpkg-shlibdeps. There have been some bugs
because of this; not all of them are even fixed. #133092 is the best
one.

And too, the more effort that needs to be spent to make dh_shlibdeps be
able to drive dpkg-shlibdeps, the harder it is for anyone to make a
credible alternative to it. Oh, oops, I'm supposed to consider that an
advantage, right? :-P

-- 
see shy jo


--
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100421161640.ga24...@kitenet.net

Re: Bug#574443: debhelper: New helper proposal: dh_oldconffiles (or similar name)

[Joey Hess]

Didier Raboud wrote:
> while updating one of my packages (which again dropped a conffile), I had to
> modify its preinst, according to http://wiki.debian.org/DpkgConffileHandling .
> 
> This works but is IMHO suboptimal: the code referenced on the wiki has to be
> hand copied in each and every package that has to move or delete conffiles.
> 
> This could maybe be done through a dh_oldconffiles (for removing) and a
> dh_moveconffiles (for moving). Those helpers would read 
> ${package}.oldconffiles
> in debian/ containing e.g.:

I prefer to keep large blocks of code out of maintainer scripts
entirely, by putting them in utilities that debhelper can then arrange
maintainer scripts to call. Or that maintainers can call directly.

So, I wrote a patch to dpkg in 2008 to add a dpkg-conffile command,
that handles renaming and removal of conffiles.

http://lists.debian.org/debian-dpkg/2008/01/msg00143.html

There was a little bit of followup, but dpkg's maintainers never
integrated it, nor do they seem to have given a reason not to integrate it.

The patch still seems good. From the history of
http://wiki.debian.org/DpkgConffileHandling , I see only one bugfix has
been made to that page since, that should be folded into the patch. It is
described here: http://lists.debian.org/debian-devel/2008/03/msg00795.html

So, I encourage dpkg's maintainers to review and accept that patch.

(Then debhelper can remove the conffile moving code that it already
uses in dh_installmodules and dh_installudev, and I can think about
perhaps providing a dh_oldconffiles command.)

-- 
see shy jo


signature.asc
Description: Digital signature

Re: quilt 3.0 source format and dpkg-source/dpkg-buildpackage

[Joey Hess]

Joachim Wiedorn wrote:
> I still use CDBS and I use "simple-patches" - but now without CDBS
> support. My minor change is the file "patches/series" which let
> dpkg-buildpackages know that there are patches. This seems very simple,
> too. To get the old manner, I must only delete the series file and add
> the CDBS line into debian/rules. But remaining in format 3.0. 

> Because I never used quilt, the syntax for quilt is a little bit stupid
> for me. So also with the new format I don't use quilt.

If I understand correctly what you're doing, you have a 3.0 (quilt)
package with no quilt managed patches, and using a debian/rules based
patch system on top of it. That perversion of the 3.0 format threatens
to eliminate the main archive-wide gain of that format: That any 3.0
source package can be unpacked and the pre-patched source accessed
without needing to manually follow a README.source file.

I would prefer not to see such packages in the archive using source
format 3.0. We've been down that road with 1.0, and it was not pretty.
Above all other goals, my goal with putting the framework of source 3.0
in place was to allow the flexability that that mess never need to
happen again. Please try to respect that; the benefits of getting this
consistently right, are widespread, diffuse, but very real.

I think you should be using a dedicated source format for your patch
system, preferably one that preserves the pre-patched source on unpack
invariant. Either the existing 3.0 (custom), or a new 3.0 subformat.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Enhancing 3.0 (git) source package format

[Joey Hess]

I support changing 3.0 (git) to use a bundle. Besides closing the bugs
mentioned in this thread, a bundle consists of a simple header + a standard
git pack. Since git packs are used as a wire format, this provides better
assurance that future versions of git will retain compatability with
the source packages. Also, being able to directly clone a bundle file
is a great feature.

The only thing I would miss from this version of the format is that when
unpacked, it would not include remotes information, and so remotes would need
to be manually added if you wanted to 'git pull' to get newer commits etc. I
suppose that the remotes info could be added into the source format in some
other way however.

Johan Kiviniemi wrote:
> git for-each-ref --format='%(refname)' | \
>  grep -Ev '^refs/(remotes/|stash$)' | \
>  xargs -d'\n' git bundle create ../PKG_VER.git-bundle HEAD

You forgot to drop bisect refs. Also, if there are a *lot* of refs, xargs may
run git-bundle repeatedly; subsequent runs will happily overwrite the bundle. 
So I suggest:

git bundle create ../bundle $(git for-each-ref --format='%(refname)' | egrep -v 
'^refs/(remotes/|bisect/|stash$)')

This still has the potential problem of failing if there are >20kb worth of
ref information. Probably the right fix for that limit is to patch git-bundle
so it can read the refs from stdin, as git-pack-objects already does.

Russ Allbery wrote:
> It is when package maintainers orphan packages and rm -rf all their Git
> repositories, such as just happened recently with a pile of Debian
> packages.  If those packages had been 3.0 (git) packages, a new maintainer
> would be in much better shape.

Even when package orphaners are careful to preserve their git repos,
it can be easy for them to be dropped in transition to the new maintainer.
(See for example: uqm.)

-- 
see shy jo


signature.asc
Description: Digital signature

reassign 503954 to dpkg-dev

[Joey Hess]

reassign 503954 dpkg-dev 


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: New Build-Options field and build-arch option, please review

[Joey Hess]

Raphael Hertzog wrote:
> Even if there's only two things, the fact is that the package maintainer
> wants not only to decide what is supported but he might also want to
> enable some features...

Did you think about having two fields, one to specify the set of
supported options, and one to allow setting defaults?

FWIW, Manoj, Steve, Yuri[1] and I had a good chat about this on the
train across Scotland last summer. 

For some types of options, it makes sense to not just declare that
they're supported, but that some particular combinations of options is
supported, while declaring other combinations as unsupported. This would
be particularly useful when setting compile options (including librarary
link combinations).

Hmm, my notebook[2] from that trip suggests the following syntax:

Build-Options: strip, debug, bar, foo, !foo+bar

Indicating that foo and bar cannot be combined.

Also, I think it would be a good idea to explicitly make "x-foo" be
reserved for non-standard options.

-- 
see shy jo

[1] A gentoo guy. Who better to discuss build options. ;-)
[2] http://farm2.static.flickr.com/1121/639896787_dfe8d0f8c2.jpg


signature.asc
Description: Digital signature

Re: git-style file storage for .deb

[Joey Hess]

Phillip Susi wrote:
> I'm afraid you have it backwards.  In CVS when you move a file it thinks  
> you removed one file and added a totally new file somewhere else.  In  
> SVN, it thinks that you copied the old file to a new location, and then  
> removed it.  In git, the diff clearly shows fileA -> fileB.

"Git does not explicitly record file revision relationships at any level
below the source code tree; content copying and moving between files (so
called "rename detection") is done at runtime." 
--  http://git.or.cz/gitwiki/Git

HTH

-- 
see shy jo


signature.asc
Description: Digital signature

Re: git-style file storage for .deb

[Joey Hess]

Phillip Susi wrote:
> You tell git when you move a file and it records the fact in the change  
> record.

No, that's how every VCS *except* git works.

-- 
see shy jo


signature.asc
Description: Digital signature

reassign 484469 to dpkg

[Joey Hess]

# Automatically generated email from bts, devscripts version 2.10.28
reassign 484469 dpkg 


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: Bug#471060: debhelper, start-stop-daemon -exec, and restarting daemons during upgrade

[Joey Hess]

Guillem Jover wrote:
> Just to clarify, for upgrades the original file is not directly deleted,
> but in shell examples that's the closest we can do to test the
> behaviour. Something like this roughly simulates what dpkg would do:
> 
>   $ s-s-d start dictd
>   $ cd /usr/sbin
>   $ cp /bin/ls dictd.new
>   $ ln dictd dictd.old
>   $ mv dictd.new dictd
>   $ rm dict.old
>   $ s-s-d stop dictd
> 
> and this should work on the shell and when done by dpkg.

If you actually try to make a package that uses -exec and restarts its
deamon in the postinst, it doesn't work. Example:

http://kitenet.net/~joey/tmp/dictd_1.10.11.dfsg-2_i386.deb

[EMAIL PROTECTED]:/home/joey/tmp>dpkg -i dictd_1.10.11.dfsg-2_i386.deb 
Selecting previously deselected package dictd.
(Reading database ... 207015 files and directories currently installed.)
Unpacking dictd (from dictd_1.10.11.dfsg-2_i386.deb) ...
usermod: no changes
Setting up dictd (1.10.11.dfsg-2) ...
Starting dictionary server: dictd.
Processing triggers for man-db ...
[EMAIL PROTECTED]:/home/joey/tmp>ps ax |grep dictd
27024 ?Ss 0:00 dictd 1.10.11: 0/0
[EMAIL PROTECTED]:/home/joey/tmp>/etc/init.d/dictd restart
Restarting dictionary server: dictd.
[EMAIL PROTECTED]:/home/joey/tmp>ps ax |grep dictd
27046 ?Ss 0:00 dictd 1.10.11: 0/0
[EMAIL PROTECTED]:/home/joey/tmp>/var/lib/dpkg/info/dictd.postinst configure 
1.10.11.dfsg-2
Restarting dictionary server: dictd.
[EMAIL PROTECTED]:/home/joey/tmp>ps ax |grep dictd
27131 ?Ss 0:00 dictd 1.10.11: 0/0

All good so far, but now try reinstalling:

[EMAIL PROTECTED]:/home/joey/tmp>dpkg -i dictd_1.10.11.dfsg-2_i386.deb 
(Reading database ... 207054 files and directories currently installed.)
Preparing to replace dictd 1.10.11.dfsg-2 (using dictd_1.10.11.dfsg-2_i386.deb) 
...
usermod: no changes
Unpacking replacement dictd ...
Setting up dictd (1.10.11.dfsg-2) ...
Restarting dictionary server: dictd.
Processing triggers for man-db ...
[EMAIL PROTECTED]:/home/joey/tmp>ps ax |grep dictd
27131 ?Ss 0:00 dictd 1.10.11: 0/0

pid did not change despite it "restarting" and now it's hosed:

[EMAIL PROTECTED]:/home/joey/tmp>/etc/init.d/dictd restart
Restarting dictionary server: dictd.
[EMAIL PROTECTED]:/home/joey/tmp>/etc/init.d/dictd stop   
Stopping dictionary server: dictd (not running).
[EMAIL PROTECTED]:/home/joey/tmp>ps ax |grep dictd
27131 ?Ss 0:00 dictd 1.10.11: 0/0

The init script uses this to test if it's running, and after dpkg replaces
the binary it fails.

[EMAIL PROTECTED]:/home/joey/tmp>start-stop-daemon --start --quiet --pidfile 
/var/run/dictd.pid --exec /usr/sbin/dictd --test -v
Would start /usr/sbin/dictd .

-- 
see shy jo


signature.asc
Description: Digital signature

Re: debhelper, start-stop-daemon -exec, and restarting daemons during upgrade

[Joey Hess]

Guillem Jover wrote:
> The test on 
> seems bogus to me, as the original file would have been removed at
> "postinst configure" time, and s-s-d should work then.

It doesn't matter if dictd is copied to dictd.orig as in my example, or
deleted entirely.

start-stop-daemon -exec checks to see if the process it is stopping has
an executable that is the same as the specified filename. On upgrade,
dpkg replaces one dictd executable with the other, so by postinst
configure time it will behave as I describe.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Triggers Pending

[Joey Hess]

Sven Joachim wrote:
> This is because initramfs-tools already uses triggers, see #447611¹.
> I'm not convinced that it is a very good idea to do this in Lenny
> packages, since the Etch versions of apt and aptitude lack support for
> the new trigger states.   While dpkg 1.14.18 conflicts with these
> versions, a dist-upgrade from Etch to Lenny will still be done by the
> old versions.

When an old version of dpkg is installed, the code in initramfs-tools
will behave exactly as it did before trigger support was added.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Git internal format and compatibilty

[Joey Hess]

Raphael Hertzog wrote:
> > By contrast to old formats like tar, the git pack (and idx) formats _do_
> > contain a version number. Three versions exist: Version 1 was never
> > broadly used. ("Throw one away".) Version 3 is not generated by current
> > versions of git. Git might start generating it at some point in the future,
> > but has already supported reading it since 2006 -- a positively
> > Debianish update schedule. Version 2 is what everyone uses now.
> 
> Is there a way to extract that version number with a git command line?

It should be possible to teach file to do it, since it's just a matter
of reading a C struct header.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Git internal format and compatibilty

[Joey Hess]

Raphael Hertzog wrote:
> The version of the software called "git" (we have nothing better to
> identify the internal format AFAIK).
> 
> If the internal format changes, I expect that git will upgrade it in place
> or something similar. However a source package published in a given
> release is a git repository that nobody use and will thus never be
> upgraded to a newer format. And after some time, we'll have lost the
> ability to extract older source packages.

In several ways, that doesn't make much sense. There's also some
inaverdant FUD going on here. Here is the actual situation, as I
understand it.

Neither gzip nor tar files include an embedded version number and yet
the tar format has several times been extended without breaking
backwards compatability. The family of zip/compress formats is a mess
and yet current gzip can extract every zip-like thing ever created. In a
sense that's what "file format" means today -- even if the format
changes, old versions are still supported[1].

By contrast to old formats like tar, the git pack (and idx) formats _do_
contain a version number. Three versions exist: Version 1 was never
broadly used. ("Throw one away".) Version 3 is not generated by current
versions of git. Git might start generating it at some point in the future,
but has already supported reading it since 2006 -- a positively
Debianish update schedule. Version 2 is what everyone uses now.

I'm confident that the current git object format will be supported as
long as git exists. Changing that format and doing the "upgrade in
place" you suggest git might do would change every sha1 in the git
repository, breaking all references to the sha1s, including all signed
tags, and all cloned repositories. New object file formats might be
developed, but existing ones will always have to be supported going
forward.

Just because the git pack, idx, and object are relatively new file
formats to tar or gz is not a reason to fear them changing in
incompatible ways.

-- 
see shy jo

[1] Doesn't dpkg even support all the old variants of the deb format?


signature.asc
Description: Digital signature

3.0 (git) "experimental"

[Joey Hess]

Could the maintainers clarify what criteria are used to mark a given source
format such as 3.0 (git) as "experimental"?

It doesn't seem to be when the format was implemented or merged, or the
amount of testing the format has had, since the git format seems as good
or better than other non-experimental formats in those regards.

I hope it doesn't come down to one member of the dpkg team's personal
preference.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: triggers question

[Joey Hess]

Ian Jackson wrote:
> (Also, while I'm here, I note that #438547 is still open.  I hope apt
> has been updated already; if not it should be done forthwith.)

Apt was updated quite a while ago, this bug seems to remain open in
error since your changelog line shows up in version 0.7.7.

> >A file trigger is guaranteed to be activated before the file in
> > ^^
> >question is modified by dpkg; on the other hand, a file trigger might
> >be activated even though no file was actually modified.
> > 
> > AFAICS, the file trigger runs AFTER the file is modified. It would
> > not be very useful to run it before the file was modified!
> 
> The trigger is _activated_ before the file is modified but _run_ much
> later.

Ok, I was reading "activated" to mean run, so it's a little unclear.

-- 
see shy jo


signature.asc
Description: Digital signature

triggers question

[Joey Hess]

So I've modified update-menus as the triggers docs suggest, so that
when it's run by a postinst and --real is not passed, it runs
dpkg-trigger /usr/share/menu and then exits. If --real is passed, it
does a menu update (in the foreground, no more forking to background).
I made menu declare interest in /usr/share/menu, and I added a triggered
action to its postinst, that calls update-menus --real.

Here's what I'm seeing with the new dpkg when I install packages that
contain menu files and that also run update-menus in their postinsts:

Unpacking xaos (from .../archives/xaos_3.3-1_i386.deb) ...
Selecting previously deselected package xemeraldia.
Unpacking xemeraldia (from .../xemeraldia_0.4.1-4_i386.deb) ...
Processing triggers for menu ...
Setting up xaos (3.3-1) ...
Setting up xemeraldia (0.4.1-4) ...
Processing triggers for menu ...

Note the dual trigger run. The first time occurs after the packages are
unpacked, when the /usr/share/menu files are updated. The second time
occurs after the postinsts run, since they trigger again by running
update-menus.

Of course once all packages are updated to drop the update-menus calls,
the trigger will only run once per dpkg invocation of a menu-providing
package. So this is temporary. It's also probably fine to run
update-menus trice for a while. Just wanted to make sure this is
as-indended, since I don't remember realizing this would happen when I
read the original triggers documentation.


BTW, there seems to be a typo in triggers.txt:

   A file trigger is guaranteed to be activated before the file in
^^
   question is modified by dpkg; on the other hand, a file trigger might
   be activated even though no file was actually modified.

AFAICS, the file trigger runs AFTER the file is modified. It would
not be very useful to run it before the file was modified!

-- 
see shy jo


signature.asc
Description: Digital signature

Re: dpkg semi-hijack - an announcement (also, triggers)

[Joey Hess]

William Pitcock wrote:
> I think you mean package install-time improvements, due to postponing
> ldconfig until the end of the installation. However, I am not sure how
> useful this is because many maintainer scripts not generated by
> debhelper call ldconfig locally.
> 
> Obviously the maintainer scripts autogenerated by debhelper would be ok
> though.

Triggers, if they ever reach the archive, should allow a vast improvement
in the majority of maintainer script fragments inserted by debhelper,
and in the simplicity, correctness, and speed of package installations.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Bug#452806: [debchange] dch -a shouldn't modify existing entries

[Joey Hess]

Adam D. Barratt wrote:
> reassign 452806 dpkg
> retitle 452806 [dpkg-parsechangelog] Please maintain trailing whitespace
> kthxbye
> 
> Hi,
> 
> On Sun, 2007-11-25 at 11:41 +0100, Josselin Mouette wrote:
> > Package: devscripts
> > Version: 2.10.10
> > Severity: normal
> > File: /usr/bin/dch
> > 
> > When "dch -a" is called, it strips any whitespace at the end of lines in 
> > the existing changelog entries.
> > 
> > This is very annoying for team-maintained packages, especially combined 
> > to debcommit which then considers these lines as having changed and adds 
> > them to the commit message.
> 
> debchange simply parses the output of dpkg-parsechangelog(1) in order to
> derive the changes. dpkg-pc is stripping the whitespace before debchange
> begins processing it; I'm therefore reassigning the bug.

Fine, but debcommit should use diff -w ..

-- 
see shy jo


signature.asc
Description: Digital signature

Re: VCS-based source packages and helper tools

[Joey Hess]

Raphael Hertzog wrote:
> among people using VCS to maintain their packages, we have a bunch of
> people using helper packages (like git-buildpackage, gitpkg, bzr-builddeb,
> svn-buildpackage, etc.). 
> 
> I'd like to know if VCS-based source packages would obsolete (part of?)
> those tools, and in general, how much of the workflow will be imposed by
> the usage of a VCS based source package.

Looking at just the git tools, some might need slight changes, but most
would be usable as-is, while a few might need significant changes. For
example, git-buildpackage(1) adds .git to ignores, which wouldn't be
appropriate if building a git package. The other stuff it does might
still be useful, although some of it would be redundant to dpkg (such as
checking for uncommitted changes). On the other hand gitpkg(1) uses
git-archive to export the sources, which wouldn't allow building a git
package. Tools like git-import-orig would continue to be useful as-is.

There's potential to subsume the *-buildpackage type tools into dpkg, but
it's not really a requirement, and git-buildpackage vs gitpkg show that
there can be two very different approaches to doing the same thing, so
there will probably always be such wrappers.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: dpkg-source's future and relation with VCS

[Joey Hess]

Russ Allbery wrote:
> (it's not yet clear to me that Git can usefully represent changesets
> via feature branches, but that's another argument that is already
> ongoing elsewhere).

People are arguing about that because bikeshedding and random discussion
of lattices, is, apparently, fun. 

apt-cache show stgit

-- 
see shy jo


signature.asc
Description: Digital signature

Re: dpkg-source's future and relation with VCS

[Joey Hess]

Anthony Towns wrote:
> Are you able to use the .git source package format right now? Yes, you grab
> Joey's sources, create a package with git, and you're done. You can't
> upload it to the archive, because it's not supported by dpkg in stable, but
> that's it. 

It's also not supported by the many hardcoded filename bits in dak,
FWIW.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Anthony Towns wrote:
> > 1) I don't really like the current behaviour when there are uncommitted
> > changes in the package directory. I would suggest as default behaviour
> > creating a commit containing these changes. This would eliminate the
> > need for people having to commit changes if they don't really care.
> > The most elegant solution would probably to create the commit, clone it
> > and then do a "git reset HEAD^" in the package directory. Don't know if
> > that is robust enough, though.
> 
> Maybe you could do a "git branch dpkg-build.$UUID", commit in the branch,
> generate the .git.tgz, then drop the branch?

Ah, now I understand what Frank was getting at. Only problem I see with
a temp branch is the source package would be shipped using this funky
branch name.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Frank Lichtenheld wrote:
> I've now added this branch to the "official" dpkg repository on alioth
> with the intention to work on it. I've at least fixed it up so that
> it works with the current code base.

Excellent. I had kept it merged to master, but haven't checked that it's
not bit-rotted lately.

> After thinking a bit about this proposal I have the following
> suggestions for changes that I would like to put up for discussion:
> 
> 1) I don't really like the current behaviour when there are uncommitted
> changes in the package directory. I would suggest as default behaviour
> creating a commit containing these changes. This would eliminate the
> need for people having to commit changes if they don't really care.
> 
> The most elegant solution would probably to create the commit, clone it
> and then do a "git reset HEAD^" in the package directory. Don't know if
> that is robust enough, though.

Sounds like git stash?

However, stashing away uncommitted changes and not including them in the
build violates least suprise. I'd except to see them either commited
automatically, or the current error forcing me to resolve them before
building. The advantage to auto-committing, of course, is that you don't
have to know how to use git (or debcommit) to build a package that uses it.

> Prompting the user for the commit message would probably be best but
> would break if people try to run the program non-interactivly.

I don't think it's a good idea to prompt for a commit message.

> 2) Independently from the default behaviour on pack we should definetly add
> a command-line option for the user to choose between the three
> possibilities 1) error out, 2) create a commit, 3) create a commit
> interactivly

Not sure sure what you mean here?

> 3) About the plugin interface: I was considering whether it would be
> better to move the tar generation into the plugin itself. This would
> allow other plugins more flexibility (e.g. generating more than one
> file). My masterplan includes making source formats 1.0 and 2.0 plugins
> internally ;)
> 
> This would of course require to move the tar generating and compressing
> code to a module that can then be used by the plugins.

That would of course be fine. I didn't want to touch doing that in my
branch for obvious reasons. :-)

> 4) aj suggested in this thread to add a Source-Depends field which could
> be used to specify the dependencies needed to unpack the package. I
> guess that could prove useful, but I really would like to avoid that
> all packages need to specify it (even though that might be solvable with
> substvars defined by the plugin). OTOH if dpkg uses an internal
> mechanism to map format to dependencies it would be more difficult for
> other programs like apt to get to this information. Or is this all
> over-engineering and the plugin should check its pre-requisites itself
> and note the dependencies in the error message like the current code
> does.

One appoach would be for dpkg to build a dpkg-dev-git package that
includes the git format (and depends on git-core), and so on,
then "Format: 3.0 (foo)" could be converted to dpkg-dev-foo.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] add dpkg-conffile, based upon http://wiki.debian.org/DpkgConffileHandling

[Joey Hess]

Michael Biebl wrote:
> Interesting. When I rename an obsolete, modified conffile in preinst to  
> .dpkg-bak, I can't find a reference anymore in /var/lib/dpkg/status or  
> /var/lib/dpkg/info/$pkg.*
>
> Where does dpkg store the information about conffile then?

It must drop it from the status file during configuration if the
conffile is removed and keep it otherwise.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] add dpkg-conffile, based upon http://wiki.debian.org/DpkgConffileHandling

[Joey Hess]

Michael Biebl wrote:
> Hm, when you rename the conffile to *.dpkg-bak in pre-inst, dpkg does no  
> longer track the conffile (as the original conffile does no longer  
> exist). So how do you think, dpkg should handle that?

dpkg does continue to remember that there was an obsolete conffile, even
after the conffile moved, until the package is purged.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] add dpkg-conffile, based upon http://wiki.debian.org/DpkgConffileHandling

[Joey Hess]

Michael Biebl wrote:
> It was discussed briefly on debian-devel, but there is also the case  
> that the backup files should be cleaned up on purge.

dpkg tracks obsolete conffiles, but on removal/purge, it does not remove the
obsolete conffile. If it did, it would make sense for it to also remove
any .dpkg-old file, and that would seem to be a better way to handle the
purge case.

-- 
see shy jo


signature.asc
Description: Digital signature

[PATCH] add dpkg-conffile, based upon http://wiki.debian.org/DpkgConffileHandling

[Joey Hess]

As discussed on debian-devel, different versions of these conffile handling
shell functions on the wiki are being copied into maintainer scripts
repeatedly, and adding a program to dpkg that handles things is a more
maintainable approach.

Note that in two places, parameters are required by dpkg-conffile that are
not currently used. I did this to future-proof calls to the program.
Perhaps in the future it will want to know the new conffile name in
`prep_mv`, or the package name in `mv`.

There is no copyright statement on the code in the wiki. (Which is being
copied into many packages .. without a copyright statment!) I looked at
the edit histories of both
http://wiki.debian.org/DpkgConffileHandling and
http://www.dpkg.org/dpkg/ConffileHandling, and it was originally written
by Scott, and was then modified by Guillem. This patch assumes that both
of them are willing to license it under GPL2+.

Signed-off-by: Joey Hess <[EMAIL PROTECTED]>
---
 debian/changelog |3 +
 debian/copyright |3 +-
 debian/dpkg.install  |3 +
 man/Makefile.am  |1 +
 man/dpkg-conffile.8  |  132 ++
 scripts/Makefile.am  |2 +
 scripts/dpkg-conffile.sh |   94 
 7 files changed, 237 insertions(+), 1 deletions(-)
 create mode 100644 man/dpkg-conffile.8
 create mode 100755 scripts/dpkg-conffile.sh

diff --git a/debian/changelog b/debian/changelog
index 1eb6246..86c51e1 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -22,6 +22,9 @@ dpkg (1.14.17) UNRELEASED; urgency=low
   * German (Helge Kreutzmann).
   * Swedish (Peter Karlsson).
 
+  [ Joey Hess ]
+  * Add dpkg-conffile, based upon http://wiki.debian.org/DpkgConffileHandling
+
  -- Guillem Jover <[EMAIL PROTECTED]>  Mon, 21 Jan 2008 10:11:55 +0200
 
 dpkg (1.14.16.5) unstable; urgency=low
diff --git a/debian/copyright b/debian/copyright
index 8ea1e6d..ac15365 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -1,11 +1,12 @@
 This is Debian's package maintenance system.
 
+Copyright © 2007 Guillem Jover <[EMAIL PROTECTED]>
 Copyright © 2006-2007 Frank Lichtenheld <[EMAIL PROTECTED]>
 Copyright © 2004-2005 Scott James Remnant <[EMAIL PROTECTED]>
 Copyright © 2004-2005 Canonical Ltd.
 Copyright © 1999-2002 Wichert Akkerman <[EMAIL PROTECTED]>
 Copyright © 1999-2001 Marcus Brinkmann <[EMAIL PROTECTED]>
-Copyright © 2001 Joey Hess <[EMAIL PROTECTED]>
+Copyright © 2001,2008 Joey Hess <[EMAIL PROTECTED]>
 Copyright © 1994-1999 Ian Jackson <[EMAIL PROTECTED]>
 Copyright © 1999 Richard Kettlewell <[EMAIL PROTECTED]>
 Copyright © 1999 Ben Collins <[EMAIL PROTECTED]>
diff --git a/debian/dpkg.install b/debian/dpkg.install
index 2f54618..bd33273 100644
--- a/debian/dpkg.install
+++ b/debian/dpkg.install
@@ -7,6 +7,7 @@ usr/bin/dpkg
 usr/bin/dpkg-deb
 usr/bin/dpkg-query
 usr/bin/dpkg-split
+usr/bin/dpkg-conffile
 usr/lib/dpkg/mksplit
 usr/sbin
 usr/share/dpkg
@@ -21,6 +22,8 @@ usr/share/man/*/*/dpkg-query.1
 usr/share/man/*/dpkg-query.1
 usr/share/man/*/*/dpkg-split.1
 usr/share/man/*/dpkg-split.1
+usr/share/man/*/*/dpkg-conffile.8
+usr/share/man/*/dpkg-conffile.8
 usr/share/man/*/*/dpkg-statoverride.8
 usr/share/man/*/dpkg-statoverride.8
 usr/share/man/*/*/dpkg.cfg.5
diff --git a/man/Makefile.am b/man/Makefile.am
index 42b17af..c975329 100644
--- a/man/Makefile.am
+++ b/man/Makefile.am
@@ -78,6 +78,7 @@ dist_man_MANS = \
dpkg-architecture.1 \
dpkg-buildpackage.1 \
dpkg-checkbuilddeps.1 \
+   dpkg-conffile.8 \
dpkg-deb.1 \
dpkg-distaddfile.1 \
dpkg-divert.8 \
diff --git a/man/dpkg-conffile.8 b/man/dpkg-conffile.8
new file mode 100644
index 000..6bd8bf6
--- /dev/null
+++ b/man/dpkg-conffile.8
@@ -0,0 +1,132 @@
+.\" This is an -*- nroff -*- source file.
+.\" dpkg-conffile and this manpage are Copyright 2008 by Joey Hess.
+.\"
+.\" This is free software; see the GNU General Public Licence version 2
+.\" or later for copying conditions.  There is NO warranty.
+.TH dpkg\-conffile 8 "2008-1-25" "Debian Project" "dpkg utilities"
+.SH NAME
+dpkg\-conffile \- graceful deletion or moving of conffiles
+.
+.SH SYNOPSIS
+.B dpkg\-conffile
+.I rm  
+.P
+.B dpkg\-conffile
+.I prep_mv   
+.P
+.B dpkg\-conffile
+.I mv   
+.
+.SH DESCRIPTION
+When upgrading a package, dpkg will not automatically remove a conffile (a
+configuration filefor which dpkg should preserve user changes) if it is not
+present in the newer version. There are two principal reasons for this; the
+first is that the conffile could've been dropped by accident and the next
+version could restore it, users wouldn't want their changes thrown away.
+The second is to allow packages to transition files from a dpkg-maintained
+conffile to a file maintained by the package's maintainer scripts, usually
+with a to

Re: dpkg patch

[Joey Hess]

Ian Jackson wrote:
> Many packages nowadays are maintained by teams sharing a revision
> control system.  When this happens many others who work with the
> package end up using the same RCS to prepare NMUs, local versions,
> etc.
> 
> So I think in these cases the fact that a particular RCS is being used
> is a fact about the package and not about the person doing the work.
> It should be stored with the package.

Well, we do store such info in the package, via the new Vcs- fields. I
probably wouldn't mind if dpkg-source looked at those and turned on
appropriate ignores.. Some people might argue that such a behavior would
need to be disabled in some cases.

> I agree with this, I think.  That's not to say that Ubuntu's ought to
> be identical.

No more than it's important that the dpkg in debian stable and unstable
be identical. However, non-bugfix divergences that change behavior, and
that one distro can come to rely on, while another decides not to
support or use, can be a big problem.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: dpkg patch

[Joey Hess]

Ryan Lortie wrote:
> The reason for these files is obviously to avoid the need to remember to
> type -I or -i every time.

>   3) debuild

The obvious way to solve needing to type -I and -i every time is to use
a wrapper, such as debuild, to do it. 

I've been using a personal "build" wrapper[1] for 9 years, to do just
that, and AFAIK I have never accidentially forgotten to exclude files,
thanks to having a wrapper to remember it for me.

I quite like Raphael's idea of a configuration file (or environment
variables?) to add default options for dpkg-buildpackage and related
tools. It's much nicer to have the functionality in the tools, rather
than in a wrapper, as you then don't have to worry about making sure
that the wrapper is always used.

That said, if you're already using debuild as a wrapper, it could either
be extended, or a wrapper put around it. Either alternative seems much
cleaner than adding these files to dpkg-source.

> 1) In almost any sane situation, building the source package involves
> running the clean rule first.

I think everyone is agreed that a tool like dpkg-source (or rpm, or
tar..) should not execute code from a source package when unpacking it.
It seems counterintuitive for such a tool to execute code from a source
package when doing the inverse operation, and dpkg-source would be the
only such low-level tool that I know of to do that. Combined with it not
explicitly doing it, but only allowing code execution via a regexp, this
has all the earmarks of either being a security hole, or contributing to
one later.

> 2) Someone with slightly more perl-fu than myself could surely write
> some regexp to detect exploit attempts.

Doing so securely requires not just perl-fu, but prescience, since
a) perl tends to add new constructs from time to time
b) new security holes tend to be found in regexp engines from time to time

I believe the following transformation of globs into regexps is
reasonably secure. (I use this in ikiwiki.) I'd be wary of anything more
powerful.

$regexp=quotemeta($glob);
$regexp=~s/\\\*/.*/g;
$regexp=~s/\\\?/./g;

-- 
see shy jo

[1] http://git.kitenet.net/?p=joey/home;a=blob;f=bin/build


signature.asc
Description: Digital signature

Re: dpkg patch

[Joey Hess]

Ryan Lortie wrote:
> +  * Add 'tarignore' and 'diffignore' functionality
> +- list files (eg: '.git' or '.bzr') one per line to debian/tarignore
> +  for dpkg-source -b to exclude them from the tarball
> +- list regexps one per line to debian/diffignore for dpkg-source -b
> +  to exclude them from the diff
> +  - lines starting with '#' are comments
> +  - blank lines ignored
> +  - empty file means to use the default (same as '-i' on command line)

dpkg-source options -I and -i already solve the problem of excluding
files from a tarball and diff. That works well for excluding revision
control system files. Why would you want to put this information about
the revision control system you're using into the source package?

I suppose that diffignore could be used as an alternative to a proper
debian/rules clean target, if the build modifies some files and it's
too much bother to undo the modifications in clean. But is there really
a benefit from making it easier to write clean rules that don't fully
clean the build tree? What if the someone later needs to modify the
diffignored files to fix an issue in the package? They then build a
source package, and it silently fails to include their modifications!

Also, as implemented, dpkg-source -i overules the diffignore file. So if
you put a file in diffignore, everyone who builds the package has to be
very careful not to use dpkg-source -i. Inconsistently, dpkg-source -I
doesn't override the tarignore file. (Nor does an empty tarignore file
lead to the default -I behavior, like an empty diffignore does.)

Finally, the regexps in debian/diffignore can be used to cause
dpkg-source to run arbitrary code (embedded in the regexp via (?{ code })
when building a source package. This is not as dire as if it ran arbitrary 
code when unpacking a source package, but it is still a new security issue,
and I think dpkg-source should be safe to use to build source packages
without worrying about code injection.

(BTW, your YY_TEXTPOINTER change seems to have nothing to do with
rest of the patch.)

> Pending further review this will be in Ubuntu's dpkg soon.

I have the same opinion of unnecessarily diverging ubuntu and debian's
dpkg as I have of diverging debhelper. It will come back to haunt all of
us. If you think it won't, just look at the gawdawful mess rpm has been
in over the past few years, with each vendor maintaining slightly
incompatible versions.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Triggers status?

[Joey Hess]

Raphael Hertzog wrote:
> I just gave a quick look to your branch:

I can understand why all these might be best practices for submitting
patches via git for a project like the linux kernel, but dpkg is not
exactly drowning in perfectly-presented git patchsets, and IMHO there are
good reasons to ignore every suggestion you made. 

> - please rebase it on the current master branch (that way you're sure that
>   there are no conflicts)

I have yet to see a use of git rebase that I am comfortable with.
History is very important to me, even the ugly bits. Being able to see
every misttep that was committed actually helps understanding the end
result.

Merging it with the current upstream master would have the same effect
with a more "ugly" history, surely?

> - please update your git identity, I see "Ian Jackson
>   <[EMAIL PROTECTED]>". You should standardize
>   your contributions to a single email. FYI, your previous patches have been
>   applied as "Ian Jackson <[EMAIL PROTECTED]>". You can use git-filter-branch
>   to rewrite that field automatically.

Again, history is important to me, in this case I quite like that git
preserves a history of where I did my work, so I have no interest in
using a single email address.

(AFAIK, this isn't even a general best practice. For example Linus
commits from [EMAIL PROTECTED] and signs off patches from
[EMAIL PROTECTED] Earlier in the kernel history, it's
[EMAIL PROTECTED])

> - please update the commit logs, a copy/paste of the ubuntu changelog
>   doesn't look good to me, follow the recommandation of using
>   a summary in the first line, then an empty line, then a long
>   description.
>   see http://wiki.debian.org/Teams/Dpkg/GitUsage
>   (this can be done during the interactive rebase process, see man
>   git-rebase and its -i option)

Again, history is important to me, moreover, this is a lot of work for
essentially zero gain. Yes, I'm lazy. If I have a perfectly ok changelog
entry in debian/changelog, I will debcommit it, and not reiterate the
same changes in a different style.

> - you can also restructure the serie of changes with "git rebase -i 
> origin/master"
>   (provided that origin/master is an up-to-date copy of the master branch
>   of the main dpkg repository)
>   (merge several commits in a single, in particular bug fixes with initial
>   implementation so that while reviewing diff you don't find bugs which
>   are in fact fixed by a subsequent commit)

This loses history, is highly complex, requires a significant knowledge
of git (more than I am comfortable with after using it for a month), and
AFAICS is an anourmous time sink for no benefit. I'd rather be coding.

> I can't judge the content but following the "right form" is more likely to
> lead to a quicker review.

If that's actually true of dpkg, I'll be sure to limit the information I
publish about dpkg patches to unified diffs and avoid all this nonsense.

As someone who needs this patch merged as soon as possible so we can
start *using* it, your comments came off as somewhat tinged with
obstructionism and git fanboyism. I'm sure you didn't intend that, but
that was my impression. :-/

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Frank Lichtenheld wrote:
> Note that he was talking about the size of the working tree, not the git
> repository.
> 
> The distribution tarball contains more than a checkout from dpkg's git
> repository, though, since it does contains the files copied and/or
> generated by autoreconf. However, this shouldn't really make a huge difference
> in size.

Ah yes, I didn't include such files since I did work from dpkg's git
repo.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Phillip Susi wrote:
> Joey Hess wrote:
>> A sample dpkg source package built using this is at
>> <http://kitenet.net/~joey/tmp/git-demo/>. This demo package includes only
>> the last 200 commits to the dpkg git repo, so it's more than 1 mb 
>> *smaller*
>> than dpkg's normal .tar.gz!
>
> What was removed from the source tree when importing it into git to save 
> this space?

Like I said, I included only the last 200 commits in the git repo.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Phillip Susi wrote:
> Why is that a bad thing?  What good does it do to have the git repo packed 
> inside the source archive?

http://kitenet.net/~joey/blog/entry/an_evolutionary_change_to_the_Debian_source_package_format/

-- 
see shy jo, over and over, and out


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

FWIW, I listed my goals and reasons for working on this in the blog post
linked to in the head of this thread.

I feel that I should bow out of this thread here. I've presented an
idea, a working implementation, and addressed the issues with it to the
best of my ability. Far too many times in this project I've seen a good
idea be indefinitely delayed or killed when everyone piles on and
nitpicks it to death. This idea is in danger of that happening.

If the dpkg maintainers decide to add support to this format to dpkg,
I'll be happy to work with them to make any further fixes needed to my
patch. (My git repo has a couple more fixes in it BTW.)

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Ian Jackson wrote:
> How about we ship the .orig.tar.gz, plus an rsync batched update (with
> a suitably early rsync version) which turns the unpacked source into
> working tree plus revision history ?

I'm afraid that due to consisting of many small gzipped compontents,
.git is not ameanable to being efficiently binary deltaed, so, you'll
still end up with approximatly 2x doubled data. This is probably true of
many revision control backends, though not all .. you might be able to
do it with CVS.

It might be possible to start with the pristine source, check it into
git, and apply a set of git packs that merges the resulting repository
forward to match the maintainer's git repository. However, I think this
could only work if the maintainer's git repository began with importing
that same pristine source[1]. Which means throwing away your git repo for
each new upstream version and starting afresh, which doesn't seem very
practical.

-- 
see shy jo

[1] git's sha1sums are AIUI based on the entire history of the repo,
so you can't go back and change history


signature.asc
Description: Digital signature

Re: Triggers status?

[Joey Hess]

Colin Watson wrote:
> I have a change to man-db that uses triggers to update the manual page
> database automatically, fixing my second oldest remaining bug. I'd love
> to upload this. While it doesn't break with a non-triggers-supporting
> dpkg, I'd rather wait until triggers are merged in case their
> implementation changes incompatibly, so I'm curious as to how long I
> might need to wait.

I have a fairly vast number of things that I want to use triggers for in
debhelper..

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Colin Watson wrote:
> FWIW, I was thinking much more of native packages here; non-native
> packages already tend to just import the upstream tarball which usually
> contains generated files, which is probably why this hasn't been a
> problem for things like git-buildpackage. If nothing else, there are
> several native packages in the d-i tree alone that don't have configure
> et al in Subversion.

Or these files could be checked into a copy of the repo that is used to
build the source package, and not checked into the "main" vcs. This is
not unlike those same packages in d-i shipping the generated files in
their .diff.gz, if you look at diff as just another vcs..

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Florian Weimer wrote:
> What about empty directories?

Do you mean empty directories under .git or empty directories stored
*in* git (can't be done, use a .gitignore in the directory)

> I really think you need to work off a clone (or a cleaned-up cp -al'ed
> copy).  For instance, you do not necessary want to upload the reflog, or
> unreachable objects.  The GIT configuration stored inside .git is
> probably uninteresting, too.

I think if you read my code you'll see that I've dealt with these
problems (Frank pointed out the reflog issue earlier in this thread),
and of course it *does* build from a cleaned, cp'd copy, and run git-gc,
and sanitise the .git/config, and...

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Anthony Towns wrote:
> Oh, one question that comes to mind: how does this affect checking for
> non-free stuff in past revisions? If 3.1-4 had some non-free files that
> get reimplemented for 3.2-1, do we (a) expect the maintainer to do a
> no-history upload for 3.2-1; (b) check that this happens somehow; (c) not
> worry about it as long as it's only in the history; (d) something else?
> 
> Verifying that not just the current tree is DFSG-free, but all the history
> is too seems potentially difficult.

Yes, the faq discusses this problem. This is why shallow repos are IMHO
important and non-shallow repos should only be uploaded with caution.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Anthony Towns wrote:
> > > Maybe providing a feature on packages.debian.org (or similar) to download
> > > sources in simple, non-VC, tarball format would make this a complete
> > > non-issue though?
> > pristine-tar could be used for this, it would just need source packages
> > to put the delta somewhere standaised (under debian/), and would need 
> > some standarised way to get to the upstream source branch in git.
> 
> So the logic there would be:
> 
>   if there's an upstream tag, then
>   generate an .orig.tgz
>   if there's a pristine-tar info,
>   hax0r it to be pristine
>   generate a .diff.gz
>   if the .diff failed goto bailout
>   generate a .dsc containing the orig and diff

It's not generally possible to generate a .diff.gz that expresses all
the changes that might be in a git repository.

> Repo formats that bzr in etch can unpack could be denoted by
> 
>   Source-Depends: dpkg-bzr (>= 0.11)
> 
> while repo formats that require bzr from lenny or later could be
> denoted by:
> 
>   Source-Depends: dpkg-bzr (>= 0.18)

I was thinking about Source-Depends too, the main problem seems to be
that it would need to be supported in apt-get source too. I wonder if
we could just use build-depends.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Frank Lichtenheld wrote:
> I guess if we use Joey's idea at all we will not be able to avoid
> shipping such a module for each distributed VCS, and I didn't get
> the impression that Joey thought otherwise.

I do think otherwise. If the distributed (or other) VCS does not meet
our criteria for security and backwards compatability, then we should
not ship it.

And yes, it'll be up to the dpkg maintainers to enforce those criteria
if you crack open the floodgates..

> Is the last one really such a big problem in Debian? I know that many upstream
> VCS don't contain autogenerated files but most .orig.tar.gz's already
> contain them today, so I would have guessed people either only have
> their debian/ in their Debian VCS or all upstream files from the
> .orig.tar.gz.

So would I, and most of the tools like git-buildpackage seem to assume
it too and not try to support this case AFAICS. Colin's probably right
that it's an issue religious wars can be fought over, but if they're
being fought in the context of keeping package source in revision
control it's happening quietly..

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Colin Watson wrote:
> So, I can't stand git's user interface. I generally try to avoid making
> a huge issue of this since it seems to be massively political on places
> like Planet at the moment, there seems to be a certain amount of
> confusion of people's personal opinions with that of their employers
> going on, and in any case I normally find that revision control
> flamewars have negative utility. (I don't think it's terribly relevant
> to this discussion why I prefer not to use git, and I don't want to
> sidetrack the thread with that; I just wanted to present an existence
> case of somebody who doesn't want to switch to .git.tar.gz and yet
> doesn't want to stay with .orig.tar.gz and .diff.gz forever.)

(So, FWIW, I'm not sold on git. Not sold at all yet. But it was a good
choice for this implementation for several reasons.)

> Still, this work looks pretty cool, and I'd like to be able to make use
> of it despite avoiding git whenever I can. I noticed that you'd
> helpfully structured your changes such that it would be possible to plug
> in a different revision control system, so I wrote a module to support
> bzr.

Nice. The FAQ has some questions aimed at adding other revision control
systems, could you try to answer those in the context of bzr? In
particular, is the data that would be shipped in the source package the
same data that bzr normally reads from untrusted sources, thus ensuring
that using it this way is equally (in)secure as using bzr to pull data
over the network? (Note that this wasn't 100% true for git and I have
had to put in several workarounds.) And is the data format stable and/or
one that bzr has a history of supporting old versions of in a way that
ensures backwards compatability?

Also, will the bzr repos always contain the full history, or is there
an equivilant to git shallow clones? How big do they tend to be?

> It's true that this is no worse than dbs or dpatch or whatever, and
> in fact it's better because dpkg-source will take care of the
> unpacking step automatically. Still, I do think it is a downside; we
> do still ship /usr/share/doc/debian/source-unpack.txt

BTW, source-unpack.txt fails for both packages containing
debian/subdirs/ and of course for wig-n-pen..

>   * Buildds will need to have the VCS installed in their base system.

This seems easily solved by recommends (installed by default).

>   * Some source packages want to ship non-VCS-managed files.
> 
> It's very common for source packages to include autogenerated
> objects like configure, Makefile.in, etc. Whether to check these
> into a VCS is a somewhat religious matter (as acknowledged by the
> gettext info documentation, for instance), and personally I lean
> towards checking them in (with a few exceptions) just because it
> makes it easier to see when they change and keep an eye out for
> oddities, but I know that a lot of developers prefer to keep these
> outside their VCS. Shipping a working tree would make it easier to
> handle cases like this.

Hmm, I hadn't considered that this might be a problem.

I don't know if I'd want to write the code to do this, but shipping a
partial working tree consisting of just those files would be enough to
solve this.

>   * Space-constrained mirrors could conceivably exclude the VCS data if
> they had to, though we probably wouldn't encourage this.
> 
> These seem to me to be non-trivial advantages that outweigh the space
> costs of shipping around the working tree.

The space constraints seem pretty hard to me. Specifically, I don't want
to piss the ftpmasters off and get vcs source packages banned from the
archive.. The only saving grace really seems to be that shipping both
vcs and upstream tar will only double the size of the archive once most
everything uses the new format, and the archive will have probably
doubled in size several times over due to other factors before then.


I've eyeballed the code, it looks ok though so close to code I've been
looking at all week that I may be missing trees for the forest. :-)

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Frank Lichtenheld wrote:
> I think there is a mechanism in git to disallow replacing old pack
> files (i.e. forcing to create additional ones with only new objects),
> however, I haven't used that myself, yet.

The packs in the diff package would be basically the same packs that
git-send-pack generates when git is pushing objects to a remote
repository. Where the "remote" repo would be the contents of
foo_1.0-1.git.gz, and the "local" repo would be foo-1.0-2. Intercept
those packs in transit (how?), and then you can take the 1.0-1 repo
and later apply them to it to regenerate the 1.0-2 repo.

> On a general note: I think we definetly could need the better tarball
> compression support _before_ adding huge amount of history into the
> archive...

This would mostly be an optimisation for upload size, total archive size
is only affected if foo 1.0-1 is in testing and 1.0-2 in unstable.

It's actually much more significant to both upload and total archive
size that all 61mb of dpkg's .git not be put into its .git.tar.gz. Thus
the shallow clones with only a few hundred repos or so.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Joey Hess wrote:
> > Bonus points: rather than "debian/rules clean, create a diff, build",
> > have dpkg do "debian/rules clean, commit any uncommitted changes with the
> > commit message being the changes from the changelog, create a .git.tgz,
> > build" for git-source-format packages.
> 
> I have a feeling that any auto-commit stuff should be controlled by an
> option. I'm *sure* that it would annoy some developers. No strong
> feelings about whether it should default on or off, though least suprise
> suggests *off*.

One problem with auto-committing is tags. Developers will probably
want to tag their release before doing the final release build, and
if dpkg-source then found and auto-committed a further change, the tag
wouldn't accurately match the release.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Joey Hess wrote:
> > Maybe providing a feature on packages.debian.org (or similar) to download
> > sources in simple, non-VC, tarball format would make this a complete
> > non-issue though?
> 
> pristine-tar could be used for this, it would just need source packages
> to put the delta somewhere standaised (under debian/), and would need 
> some standarised way to get to the upstream source branch in git.

BTW, if that were standardised, the other option would be for
dpkg-source -x to regenerate the pristine upstream tarball.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Anthony Towns wrote:
> Is a .gitdiff.tar.gz possible, so the archive doesn't need to have the
> full git repo replaced by each upload? ie, something like
> 
>   Files:
> foo_1.0-1.git.tar.gz
> foo_1.0-2.gitdiff.tar.gz
> 
> so that a small patch only adds a small file to the archive rather than
> replacing a large one?

I think it's possible, the gitdiff might use git packs against a prior
repo. That would be a nice enhancement to what I have done.

> This means you can't build the package by hand with standard unix tools
> -- at the very least you need git installed, and if other VC systems
> are to be supported, you need them too.

Yes, as I mention in the faq I think this is an acceptable tradeoff to
get away from having to use diff.

> Changes in repository formats will presumably result in versioned
> dependencies too.

I don't think that dpkg should add vcs formats that we don't have a good
expectation of remaining supported by newer versions of the tools going
forward (so svn repos are out). There's a bit of discussion of this in
the faq. I think that git has a pretty good track record and has
incentive to keep compatibility support since this format is
used over the wire by git (eg, with http urls).

If the format changes in a non-backwards compatible way, we could have
source packages built on unstable that cannot be extracted on stable,
which I also think is suboptimal, but hard to completly avoid.

> This is slightly worse than the case for existing patch management tools
> in that most of those can be dealt with by hand; though cdbs and to a
> lesser extent debhelper can't be quite as easily replicated I guess.

Neither could packages using quilt before it was available in
stable or dbs before it was.

> Once the unpack is done, I don't see any reason why you can't do an NMU
> in the traditional way, so presuming "dpkg-source -x" or "apt-get source"
> handles the unpack automatically, I don't think it necessarily imposes
> any new requirements on NMUers.

Basically, you have to know how to git commit your changes before building
the NMU, and that's all. As a bonus, it's rather easier to generate NMU
patchsets. :-)

> Maybe providing a feature on packages.debian.org (or similar) to download
> sources in simple, non-VC, tarball format would make this a complete
> non-issue though?

pristine-tar could be used for this, it would just need source packages
to put the delta somewhere standaised (under debian/), and would need 
some standarised way to get to the upstream source branch in git.

> Would it make sense to have the source format look more like:
> 
>   Format: 3.0
>   Source: dpkg
>   ...
>   Source-Depends: git-dpkg (>= 3.14159)
>   Source-Hooks: /usr/bin/git-dpkg
>   ...
>   Files:
>... foo_1.2.git.tar.gz
> 
> and have the git specific functionality be provided by a /usr/bin/git-dpkg
> binary (with standardised arguments) from the git-dpkg package? That
> would let you smoothly deal with repository changes and implementing new
> interfaces, and also let us limit the allowable formats for the archive
> reasonably simply.
> 
> You could drop the Source-Hooks: line, and just have dpkg-source know
> to associate *.git.tar.gz with /usr/lib/dpkg/source/git, and trust the
> package will provide it.
 
Not sure if this buys anything that using perl modules for the vcses
can't do, really. How do you envision this helping deal with repository
format changes?

> Bonus points: rather than "debian/rules clean, create a diff, build",
> have dpkg do "debian/rules clean, commit any uncommitted changes with the
> commit message being the changes from the changelog, create a .git.tgz,
> build" for git-source-format packages.

I have a feeling that any auto-commit stuff should be controlled by an
option. I'm *sure* that it would annoy some developers. No strong
feelings about whether it should default on or off, though least suprise
suggests *off*.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Russ Allbery wrote:
> It's a little disturbing to have content in parentheses be significant in
> a format based on RFC 822, although we have broken this "rule" elsewhere
> (most notably in dependency fields, of course).

If it helps, the "(git)" comment is only used in debian/control, it's
not put in the dsc files.

I'd be just as happy to use "[git]" or even "Foo: git".

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Here's an updated patch, full diff from head again, with:

- use git-config --null
- git-config --filename only needs a full path if not run from a git WC
- import the VCS module so it can check if the VCS is available
- fix all commands that spawn a subshell
- delete the reflog

-- 
see shy jo
diff --git a/debian/dpkg-dev.install b/debian/dpkg-dev.install
index 49e3835..ee65dbf 100644
--- a/debian/dpkg-dev.install
+++ b/debian/dpkg-dev.install
@@ -56,3 +56,4 @@ usr/share/man/*/dpkg-shlibdeps.1
 usr/share/man/*/*/dpkg-source.1
 usr/share/man/*/dpkg-source.1
 usr/share/perl5/Dpkg/BuildOptions.pm
+usr/share/perl5/Dpkg/Source
diff --git a/man/dpkg-source.1 b/man/dpkg-source.1
index 9bf9ff3..14c17c3 100644
--- a/man/dpkg-source.1
+++ b/man/dpkg-source.1
@@ -55,6 +55,10 @@ will look for the original source tarfile
 or the original source directory
 .IB directory .orig
 depending on the \fB\-sX\fP arguments.
+
+
+If the source package is being built as a version 3 source package using
+a VCS, no upstream tarball or original source directory is needed.
 .TP
 .BR \-h ", " \-\-help
 Show the usage message and exit.
@@ -109,7 +113,9 @@ This option negates a previously set
 .BR \-i [\fIregexp\fP]
 You may specify a perl regular expression to match files you want
 filtered out of the list of files for the diff. (This list is
-generated by a find command.) \fB\-i\fR by itself enables the option,
+generated by a find command.) (If the source package is being built as a
+version 3 source package using a VCS, this is instead used to
+ignore uncommitted files.) \fB\-i\fR by itself enables the option,
 with a default that will filter out control files and directories of the
 most common revision control systems, backup and swap files and Libtool
 build output directories. There can only be one active regexp, of multiple
@@ -162,6 +168,9 @@ will not overwrite existing tarfiles or directories. If this is
 desired then
 .BR \-sA ", " \-sP ", " \-sK ", " \-sU " and " \-sR
 should be used instead.
+.PP
+If the source package is being built as a version 3 source package using
+a VCS, these options do not make sense, and will be ignored.
 .TP
 .BR \-sk
 Specifies to expect the original source as a tarfile, by default
diff --git a/scripts/Dpkg/Source/VCS/git.pm b/scripts/Dpkg/Source/VCS/git.pm
new file mode 100644
index 000..431fab3
--- /dev/null
+++ b/scripts/Dpkg/Source/VCS/git.pm
@@ -0,0 +1,257 @@
+#!/usr/bin/perl
+#
+# git support for dpkg-source
+#
+# Copyright © 2007 Joey Hess <[EMAIL PROTECTED]>.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
+package Dpkg::Source::VCS::git;
+
+use strict;
+use warnings;
+use Cwd;
+use File::Find;
+use Dpkg;
+use Dpkg::Gettext;
+
+push (@INC, $dpkglibdir);
+require 'controllib.pl';
+
+# Remove variables from the environment that might cause git to do
+# something unexpected.
+delete $ENV{GIT_DIR};
+delete $ENV{GIT_INDEX_FILE};
+delete $ENV{GIT_OBJECT_DIRECTORY};
+delete $ENV{GIT_ALTERNATE_OBJECT_DIRECTORIES};
+delete $ENV{GIT_WORK_TREE};
+
+sub import {
+	foreach my $dir (split(/:/, $ENV{PATH})) {
+		if (-x "$dir/git") {
+			return 1;
+		}
+	}
+	main::error(sprintf(_g("This source package can only be unpacked using git, which is not in the PATH.")));
+}
+
+sub sanity_check {
+	my $srcdir=shift;
+
+	if (! -d "$srcdir/.git") {
+		main::error(sprintf(_g("source directory is not the top directory of a git repository (%s/.git not present), but Format git was specified"), $srcdir));
+	}
+	if (-s "$srcdir/.gitmodules") {
+		main::error(sprintf(_g("git repository %s uses submodules. This is not yet supported."), $srcdir));
+	}
+
+	# Symlinks from .git to outside could cause unpack failures, or
+	# point to files they shouldn't, so check for and don't allow.
+	if (-l "$srcdir/.git") {
+		main::error(sprintf(_g("%s is a symlink"), "$srcdir/.git"));
+	}
+	my $abs_srcdir=Cwd::abs_path($srcdir);
+	find(sub {
+		if (-l $_) {
+			if (Cwd::abs_path(readlink($_)) !~ /^\Q$abs_srcdir\E(\/|$)/) {
+main::error(sprintf(_g("%s is a symlink to outside %s"), $File::Find::name, $srcdir));
+			}
+		}
+	}, "$srcdir/.git");
+
+	return 1;
+}
+
+# Returns a hash o

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Frank Lichtenheld wrote:
> One thing I forgot:
> 
> On Fri, Oct 05, 2007 at 07:16:13PM -0400, Joey Hess wrote:
> > @@ -825,14 +881,17 @@ if ($opmode eq 'build') {
> >  if ($native) {
> > warning(_g("multiple tarfiles in native package")) if @tarfiles > 1;
> > warning(_g("native package with .orig.tar"))
> > -   unless $seen{'.tar'} or $seen{"-$revision.tar"};
> > +   unless $seen{'.tar'} or $seen{"-$revision.tar"} or %vcsfiles;
> >  } else {
> > -   warning(_g("no upstream tarfile in Files field")) unless 
> > $seen{'.orig.tar'};
> > +   warning(_g("no upstream tarfile in Files field")) unless 
> > $seen{'.orig.tar'} or %vcsfiles;
> 
> This should probably error out. Aren't v3 packages always native in the
> sense tested here?

Not necessarily. I wanted to leave the option open to use wig-n-pen to
constuct mixed source packages that maybe use vcs for debian/ and
pristine source for the rest + a diff.gz, or something like that.

I think the code will basically handle unpacking such a mongrel,
although there are no tools to create one.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

Thanks a lot for the code review. Any comments on the big picture or design?

Frank Lichtenheld wrote:
> On Fri, Oct 05, 2007 at 07:16:13PM -0400, Joey Hess wrote:
> > I have a sourcev3 branch with my changes at ,
> > and have also attached a diff to this mail. I feel that this is ready
> > for review and hopefully merging into dpkg now. Looking forward to your
> > comments.
> 
> A little code review follows.
> 
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write to the Free Software
> > +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
> 
> old FSF address (not really important, but while we're at it ;)

Copied from elsewhere in dpkg source. :-)

> > +sub sanity_check {
> > +   my $srcdir=shift;
> > +
> > +   if (! -s "$srcdir/.git") {
> > +   main::error(sprintf(_g("%s is not the top directory of a git 
> > repository (%s/.git not present), but Format git was specified"), $srcdir, 
> > $srcdir));
> 
> you probably mean -e or -d here? -s on a directory is kinda strange.
> printing $srcdir twice might bloat the error message.

Yes, I meant -d, the -s snuck in from the other test.

ACK on the duplication.

> > +   }
> > +   if (-s "$srcdir/.gitmodules") {
> > +   main::error(sprintf(_g("git repository %s uses submodules. This 
> > is not yet supported."), $srcdir));
> > +   }
> > +
> > +   # Symlinks from .git to outside could cause unpack failures, or
> > +   # point to files they shouldn't, so check for and don't allow.
> > +   if (-l "$srcdir/.git") {
> > +   main::error(sprintf(_g("%s is a symlink"), "$srcdir/.git"));
> > +   }
> > +   my $abs_srcdir=Cwd::abs_path($srcdir);
> > +   find(sub {
> > +   if (-l $_) {
> > +   if (Cwd::abs_path(readlink($_)) !~ 
> > /^\Q$abs_srcdir\E(\/|$)/) {
> > +   main::error(sprintf(_g("%s is a symlink to 
> > outside %s"), $File::Find::name, $srcdir));
> > +   }
> > +   }
> > +   }, "$srcdir/.git");
> 
> Maybe it would be easier to just disallow symlinks completly? Or are
> there important use cases for that?

I've tried to not make dpkg have to know too much about git internals.
(As you can see I've not been 100% successful, but have kept it to about
the level someone with a week's knowledge of git would be comfortable
with.) So while I don't see any symlinks in my git repos, if git decides
to use symlinks, I don't want dpkg to have to be updated. (I think git
did historically use symlinks in the repo).

There are probably semi-valid reasons to manually add symlinks inside a .git
directory today, too.

> > +}
> > +
> > +# Called before a tarball is created, to prepare the tar directory.
> > +sub prep_tar {
> > +   my $srcdir=shift;
> > +   my $tardir=shift;
> > +   
> > +   sanity_check($srcdir);
> > +
> > +   if (! -e "$srcdir/.git") {
> > +   main::error(sprintf(_g("%s is not a git repository, but Format 
> > git was specified"), $srcdir));
> > +   }
> > +   if (-e "$srcdir/.gitmodules") {
> > +   main::error(sprintf(_g("git repository %s uses submodules. This 
> > is not yet supported."), $srcdir));
> > +   }
> 
> Duplicated code from sanity_check

Doh!

> > +
> > +   # Check for uncommitted files.
> > +   open(GIT_STATUS, "LANG=C cd $srcdir && git-status |") ||
> > +   main::subprocerr("cd $srcdir && git-status");
> 
> you make a lot "cd $srcdir". Maybe you should just chdir() in the parent
> process?

I could make it do that, I suppose it would be safe as long as I cd back
(dpkg-source in general assumes it's in the parent dir of the source
tree).

> This would also take care of funny things in $srcdir like
> whitespaces...

> If you get rid of the cd you could use the '-|', @array form of open
> here which would be preferable imho.

Wow, you've taught me something new, I only knew about the much more
clumsy manual fork and open("-|") approach. I'll do this, but it will
take a little while.

> > +   my $clean=0;
> > +   my $status="";
> > +   while () {
> > +   if (/^\Qnothing to commit (working directory clean)\E$/) {
> > +   $clean=1;
> > +   }
> > +   else {
> > +   $status.=

[PATCH] proposed v3 source format using .git.tar.gz

[Joey Hess]

I've been working on making dpkg-source support a new source package format
based upon git. The idea is that a source package has only a .dsc and a
.git.tar.gz, which is just a git repo.

I've blogged[1] about some of what led me to this idea, and I've also written
a short FAQ[2]. Suggest reading both to understand where I'm coming from with
this.

[1] 
http://kitenet.net/~joey/blog/entry/an_evolutionary_change_to_the_Debian_source_package_format/
[2] http://wiki.debian.org/GitSrc

My implementation adds a new 3.0 version source format. A 3.0 format debian
source package can consist of any files allowed by formats 1 and 2, but
may also contain .$VCS.tar.gz files. To build a version 3 source package,
a new field is needed in debian/control:

Format: 3.0 (git)

The bit in parens specifies that it should use the git backend, which
is currently the only one available. That backend is in the
Dpkg::Source::VCS::git perl module.

I have a sourcev3 branch with my changes at ,
and have also attached a diff to this mail. I feel that this is ready
for review and hopefully merging into dpkg now. Looking forward to your
comments.

A sample dpkg source package built using this is at
<http://kitenet.net/~joey/tmp/git-demo/>. This demo package includes only
the last 200 commits to the dpkg git repo, so it's more than 1 mb *smaller*
than dpkg's normal .tar.gz!

-- 
see shy jo
diff --git a/debian/dpkg-dev.install b/debian/dpkg-dev.install
index 49e3835..ee65dbf 100644
--- a/debian/dpkg-dev.install
+++ b/debian/dpkg-dev.install
@@ -56,3 +56,4 @@ usr/share/man/*/dpkg-shlibdeps.1
 usr/share/man/*/*/dpkg-source.1
 usr/share/man/*/dpkg-source.1
 usr/share/perl5/Dpkg/BuildOptions.pm
+usr/share/perl5/Dpkg/Source
diff --git a/man/dpkg-source.1 b/man/dpkg-source.1
index 9bf9ff3..14c17c3 100644
--- a/man/dpkg-source.1
+++ b/man/dpkg-source.1
@@ -55,6 +55,10 @@ will look for the original source tarfile
 or the original source directory
 .IB directory .orig
 depending on the \fB\-sX\fP arguments.
+
+
+If the source package is being built as a version 3 source package using
+a VCS, no upstream tarball or original source directory is needed.
 .TP
 .BR \-h ", " \-\-help
 Show the usage message and exit.
@@ -109,7 +113,9 @@ This option negates a previously set
 .BR \-i [\fIregexp\fP]
 You may specify a perl regular expression to match files you want
 filtered out of the list of files for the diff. (This list is
-generated by a find command.) \fB\-i\fR by itself enables the option,
+generated by a find command.) (If the source package is being built as a
+version 3 source package using a VCS, this is instead used to
+ignore uncommitted files.) \fB\-i\fR by itself enables the option,
 with a default that will filter out control files and directories of the
 most common revision control systems, backup and swap files and Libtool
 build output directories. There can only be one active regexp, of multiple
@@ -162,6 +168,9 @@ will not overwrite existing tarfiles or directories. If this is
 desired then
 .BR \-sA ", " \-sP ", " \-sK ", " \-sU " and " \-sR
 should be used instead.
+.PP
+If the source package is being built as a version 3 source package using
+a VCS, these options do not make sense, and will be ignored.
 .TP
 .BR \-sk
 Specifies to expect the original source as a tarfile, by default
diff --git a/scripts/Dpkg/Source/VCS/git.pm b/scripts/Dpkg/Source/VCS/git.pm
new file mode 100644
index 000..cac7d05
--- /dev/null
+++ b/scripts/Dpkg/Source/VCS/git.pm
@@ -0,0 +1,226 @@
+#!/usr/bin/perl
+#
+# git support for dpkg-source
+#
+# Copyright © 2007 Joey Hess <[EMAIL PROTECTED]>.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
+package Dpkg::Source::VCS::git;
+
+use strict;
+use warnings;
+use Cwd;
+use File::Find;
+use Dpkg;
+use Dpkg::Gettext;
+
+push (@INC, $dpkglibdir);
+require 'controllib.pl';
+
+# Remove variables from the environment that might cause git to do
+# something unexpected.
+delete $ENV{GIT_DIR};
+delete $ENV{GIT_INDEX_FILE};
+delete $ENV{GIT_OBJECT_DIRECTORY};
+delete $ENV{GIT_ALTERNATE_OBJECT_DIRECTORIES};
+delete $ENV{GIT_WORK_TREE};
+
+sub sanity_check {
+	my $srcdir=shift;
+
+	if (! -s "$srcdir/.git") {
+		main::error(sprintf(_g("%s is not the 

Re: Draft spec for new dpkg "triggers" feature (v2, repost)

[Joey Hess]

Fabian Pietsch wrote:
> > However, dh_scrollkeeper doesn't generate the scrollkeeper dependency
> > in the control file, which makes it excessively hard to get the
> > dependency up to date.  (This was a mistake in the design of
> > dh_scrollkeeper.)  The bad consequences of the inaccurate dependencies
> > are less severe than the contortions which would be required to deal
> > with the problem.
> 
> Why can't an updated dh_scrollkeeper set a versioned Conflicts of the
> triggers unaware scrollkeeper packages on new gnome packages? Would it
> be a problem if there is a "Conflicts: scrollkeeper (< N)" and at the
> same time a "Depends: scrollkeeper (>= 3.8)" with 3.8 being less than N?

I already discussed this with Ian in the first thread. I don't know why
he left in the text about dh_scrollkeeper being "misdesigned", which it
isn't.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Draft spec for new dpkg "triggers" feature

[Joey Hess]

Ian Jackson wrote:
> TRANSITION PLAN
> ===
> 
> 
> >From old dpkg to new dpkg
> -
> 
> The first time a trigger-supporting dpkg is run on any system, it will
> activate all triggers in which anyone is interested, immediately.
> 
> These trigger activations will not be processed in the same dpkg run,
> to avoid unexpectedly processing triggers while attempting an
> unrelated operation.  dpkg --configure --pending (and not other dpkg
> operations) will run the triggers, and the dpkg postinst will warn the
> user about the need to run it.
>
> To use this correctly:
>  * Packages which are interested in triggers, or which want to
> explicitly activate triggers, should Depend on the
> triggers-supporting version of dpkg.
>  * Update instructions and tools should arrange to run
> dpkg --configure --pending
>after the install; this will process the pending triggers.

I wonder if you could expand a bit on how this will look if it happens
in the middle of an apt run performing a major upgrade.

Probably apt will tend to upgrade to the new dpkg early in the run,
since many packages will depend on it for triggers. This will probably
happen in a separate dpkg invocation than most of the rest of the
upgrade, given the way apt works. Some other packages might be upgraded
first, and some of those might activate triggers installed later. Will
those be the ones that will need the --configure --pending to be
handled? Or would it only be limited to trigger-defining packages that
are installed in the same dpkg run as the new dpkg?

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Draft spec for new dpkg "triggers" feature

[Joey Hess]

Ian Jackson wrote:
> I haven't looked at rpm's approach at all, I confess.

Well, you didn't miss much. 

http://www.rpm.org/support/RPM-Changes-6.html

To quickly summarise, it's backwards to your design in that the
triggered package defines the trigger, which can be something like
"installation of package X, Y, or Z". There are some complications like
versioned package dependencies in triggers.

> The correct answer would have been for dh_scrollkeeper to have written
> ${dh_scrollkeeper:Depends}.  Then each package could have decided
> whether to put the dependency in Depends, Recommends or Suggests.

That doesn't scale very well, requiring users to edit debian/control as
well when editing debian/rules. I've considered making the dependency
addition configurable (on a per subpackage basis of course) with a
switch. It looks like with triggers, I'll instead be able to remove most
of it.

> As I say above, I think the right design is for the updated packages
> not to relate to scrollkeeper at all.

Aside potentially from the upgrade issue..

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Draft spec for new dpkg "triggers" feature

[Joey Hess]

This looks like a very good design, although I will need to think about
it some more (and revisit how rpm handles triggers) to post anything
general about it. For now I'm limiting my comments to debhelper-related 
aspects.

Ian Jackson wrote:
>  3. Update debhelper's dh_scrollkeeper not to edit maintainer
> scripts.  dh_scrollkeeper should also issue a warning if it finds
> scrollkeeper (>= 3.8) in the Depends control file line.

Actually, lintian should issue such a warning. Debhelper is not intended
to find packaging problems.

> However, dh_scrollkeeper doesn't generate the scrollkeeper dependency
> in the control file, which makes it excessively hard to get the
> dependency up to date.  (This was a mistake in the design of
> dh_scrollkeeper.)  The bad consequences of these missing dependencies
> are less severe than the contortions which would be required to deal
> with the problem.

This is not a design problem in dh_scrollkeeper. It's not always
appropriate for a package that ships gnome documentation to depend on
scrollkeeper, and things will work without the dependency, with
scrollkeeper noticing the previously installed documentaton if/when it
is installed.

If the new design called for all packages that trigger scrollkeeper to
depend on the new version of scrollkeeper, dh_scrollkeeper could easily
be changed to add that dependency through misc:Depends. Some packages
that use dh_scrollkeeper will probably not have ${misc:Depends} in their
control file, since some developers remove it due to the ugly messages
that dpkg-gencontrol generates if it's empty. Such packages could be
automatically detected and fixed.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Draft spec for new dpkg "triggers" feature

[Joey Hess]

Ian Jackson wrote:
> 1. There will obviously be a limit to how many trigger names can be
> passed in this way.  If a package comes too close to this limit we may
> have to consider offering a different way to pass this information, or
> perhaps just truncating it.  As a workaround, some distinct triggers
> could be merged.
> 
> 2. That's irrelevant.  dpkg does not need to run on every
> POSIX-confirming system.  It only has to run on reasonable free
> systems.  If the command-line length limit is too severe then that is
> a bug in the underlying system.
> 
> Other possibilities for passing this information seem to include:
>  * writing it to a temporary file (what a palaver both for dpkg
>and for the maintainer script)
>  * environment variables (messy and still subject to possibly small
>implementation limits)
>  * not at all

Couldn't dpkg just run the trigger muliple times (like xargs) if it came
to that?

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Draft spec for new dpkg "triggers" feature

[Joey Hess]

Ian Jackson wrote:
> Because dh_scrollkeeper doesn't generate the dependency, making sure
> that gnome binary packages declare some relationship to the
> triggers-interested scrollkeeper would involve (at least) all of the
> updated gnome packages having a Build-Depends on the new debhelper.

If dh_scrollkeeper had always behaved like you wish it had, then it
would currently be adding a dependency on scrollkeeper (>= 0.3.8). Once
debhelper is updated to add a dependency on the new scrollkeeper that uses
triggers, all the packages using dh_scrollkeeper will need to be rebuilt
to get the new dependency. The only way to gurantee that the right
dependency is added will be to build-depend on the relevant version of
debhelper. The current and past behavior of dh_scrollkeeper has no
bearing on this.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: How are things going?

[Joey Hess]

Steve Langasek wrote:
> Since "Breaks field" here means "doesn't complain about the Breaks field",
> rather than "honors the Breaks field", these changes look ok.
> 
> As far as *implementing* Breaks, I don't think a new feature of that level
> should be introduced during a freeze.

Couldn't it be potentially dangerous to have a dpkg in a released
version of Debian that silently ignores Breaks? It seems it would both
allow for much foot-shooting by anyone who tries to install packages
from another source that use Breaks, as well as prevent us from using
Breaks in any packages in etch+1, since upgrades won't work.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: How are things going?

[Joey Hess]

Christian Perrier wrote:
> There has also been, IIRC, some exchanges with Ian Jackson. Probably
> more could be done here.

I'm rather worried that there's been no apparent progress on getting
Ian's Breaks support merged in, especially since in my limited
understanding it's already present in Ubuntu and presumably being used
there. That doesn't seem very good for cross-distro package
compatability if our dpkg ignores the field.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Using GNU's install-info in Debian instead of dpkg's install-info

[Joey Hess]

Guillem Jover wrote:
> The nice property of this one, is that then we can conditionally
> call install-info on maintainer scripts, and make the new package
> where install-info is going to be move to, non-essential (after a
> transition phase). I'd go for this one.

It also allows for competing install-info implementations in Debian,
to explore all kinds of interesting[1] ways to better organise and lay out 
the info dir file.

-- 
see shy jo

[1] If anyone still finds info interesting. ;-)


signature.asc
Description: Digital signature

Re: Using GNU's install-info in Debian instead of dpkg's install-info

[Joey Hess]

Ian Zimmerman wrote:
> It's wrong on two counts:
> 1/ The visual format of the dir file is used for collecting the sections.
> This has all the robustness of, say, grepping for text in a Postscript file.
> 
> 2/ Since no record is kept of the individual additions by packages,
> there's no way to recreate the dir file from scratch.  So, if a package
> does screw up (often due to 1 above), all one can do is manually edit
> the dirfile.  This happens maddeningly often, at least to me:
> see #367255, #367254, #367251.

I agree, there's also all the fun of bugs in packages that have to deal
with the current complex install-info. (Anyone want to fix #337215 or
#357434 for me?)

> There are two proper fixes.  One is in the realm of policy: require
> every info file passed to install-info to contain the full metadata,
> including INFO-DIR-SECTION.  Then we can dispense with the --section
> option completely and 1 magically goes away, while 2 is solved by
> rescanning the metadata in the top-level installed files (as Joey
> suggests in the other thread).
> 
> The other way is to keep some kind of simple database on the side which
> install-info first updates, then uses to generate the dirfile.  dpkg is
> a required package, so if install-info stays in dpkg the database library
> must also be required.  I thought of asking the Perl maintainer to move
> the venerable Data::Dumper module into perl-base, that would be enough
> for me.

My experience with using Data::Dumper for long-term data storage is
abysmal, it's on the order of dumping C structs to disk. However, my
benchmarks of parsing /usr/share/info/*.info* for all the metadata I think
install-info should need to use the first method suggests it can be done
without perceivable delay on any modern machine (and even reasonably
quickly on my less than modern one).

When I discussed this IRL at DebConf, one problem raised is that this
still leaves Debian w/o an install-info that behaves like GNU expects,
so we still have the portability problems we have now. I wonder if
there's any way to provide an install-info that ignores most/all
parameters and manages to be effectively compatible with programs
expecting the GNU or Debian install-infos while still avoiding
parsing the info dir file.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Status of install-info

[Joey Hess]

Ian Zimmerman wrote:
> Another possiblity to hold the structure is a XML file.
> Unfortunately perl is only standard now.  Hmm, I heard somebody
> was on a campaign to make python important or better?  That would
> help a lot, python has great XML support.

dh_installinfo is able to determine the section of an info page by
looking at the content of the info file, and is able to determine which
file is the main file in a collection of info files by looking at the
filenames. This is all the info it needs to pass to install-info, and it
can all be found by just examining the files.

So why not just let install-info look at all the installed info files,
find the above data on its own, and generate the info dir based on that?
This would eliminate a lot of the possible problems with removing
entries correctly, and would be very simple as it would need to store no
external state, and could simply regenerate the info dir from scratch
each time instead of the error-prone editing it in place that install-info
does now.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: The way to the next dpkg release

[Joey Hess]

Christian Perrier wrote:
> About usertags, we should probably agree on one e-mail address to use,
> probably "[EMAIL PROTECTED]".

Best thing is to use whatever address is listed in the Maintainer field
then the usertags will be visible by default.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: bug maintenance, last comment.

[Joey Hess]

Nathanael Nerode wrote:
> My analysis was that it's unreasonable to ask a program
> written in perl to use fcntl

That's weird, there's 

a) POSIX::fcntl
b) strace perl -e 'flock(1, "/etc/passwd")' 2>&1 |egrep '^flock|^fcntl'
   fcntl64(3, F_SETFD, FD_CLOEXEC) = 0

   perl has used fcntl internally for its flock builtin for some time

-- 
see shy jo


signature.asc
Description: Digital signature

Re: [proposal] udeb shlibdeps files

[Joey Hess]

Adam Heath wrote:
> I don't see any problems with it.  But I haven't thought it thru deeply.

I started implementing it and found one problem. Once a library package
switches over to including udeb shlibdeps lines, versions of
dpkg-shlibdeps that don't know about those lines will skip them with a
slightly ugly warning message:

shared libs info file `foo.shlibs' line 2: bad line `udeb: foo 1 foo-udeb'

Still backwards compatability will be preserved, so this isn't a very
large problem. It can also be mostly avoided by putting the tagged lines
at the end of a substvars file.

Here's a patch adding -t to dpkg-shlibdeps.

diff -ur old/dpkg-1.13.9/man/C/dpkg-source.1 dpkg-1.13.9/man/C/dpkg-source.1
--- old/dpkg-1.13.9/man/C/dpkg-source.1 2005-06-11 14:30:46.0 -0400
+++ dpkg-1.13.9/man/C/dpkg-source.1 2005-06-25 10:54:42.0 -0400
@@ -516,7 +516,7 @@
 .TP
 .BI \-L localshlibsfile
 Causes
-.B dpkg\-shlibs
+.B dpkg\-shlibdeps
 to read overriding shared library dependency information from
 .I localshlibsfile
 instead of
@@ -527,6 +527,15 @@
 output, rather than being added to the substitution variables file
 .RB ( debian/substvars
 by default).
+.TP
+.BI \-t type
+Causes
+.B dpkg\-shlibdeps
+to prefer shared library dependency information tagged for the given
+package type. If no tagged information is available, falls back to untagged
+information. The default package type is "deb". Shared library dependency
+information is tagged for a given type by prefixing it with the name of the
+type, a colon, and whitespace.
 .SH dpkg\-GENCHANGES OPTIONS
 .B dpkg\-genchanges
 does not take any non-option arguments.
Only in dpkg-1.13.9/scripts: debian
diff -ur old/dpkg-1.13.9/scripts/dpkg-shlibdeps.pl 
dpkg-1.13.9/scripts/dpkg-shlibdeps.pl
--- old/dpkg-1.13.9/scripts/dpkg-shlibdeps.pl   2005-06-06 00:07:12.0 
-0400
+++ dpkg-1.13.9/scripts/dpkg-shlibdeps.pl   2005-06-25 11:36:45.0 
-0400
@@ -17,6 +17,7 @@
 $varnameprefix= 'shlibs';
 $dependencyfield= 'Depends';
 $varlistfile= 'debian/substvars';
+$packagetype= 'deb';
 
 @depfields= qw(Suggests Recommends Depends Pre-Depends);
 
@@ -42,6 +43,7 @@
-O print variable settings to stdout
-Lshlibs override file, not debian/shlibs.local
-Tupdate variables here, not debian/substvars
+   -t   set package type (default is deb)
 Dependency fields recognised are ".join("/",@depfields)."
 ";
 }
@@ -66,6 +68,8 @@
 &warn("unrecognised dependency field \`$dependencyfield'");
 } elsif (m/^-e/) {
 push(@exec,$'); push(@execf,$dependencyfield);
+} elsif (m/^-t/) {
+$packagetype= $';
 } elsif (m/^-/) {
 usageerr("unknown option \`$_'");
 } else {
@@ -237,33 +241,37 @@
 
 while () {
 s/\s*\n$//; next if m/^\#/;
-if (!m/^\s*(\S+)\s+(\S+)/) {
+if (!m/^\s*(?:(\S+):\s+)?(\S+)\s+(\S+)/) {
 &warn("shared libs info file \`$fn' line $.: bad line \`$_'");
 next;
 }
-next if $1 ne $ln || $2 ne $lsn;
+next if defined $1 && $1 ne $packagetype;
+next if $2 ne $ln || $3 ne $lsn;
 return 1 if $fn eq "$curpackdir/DEBIAN/shlibs";
 $da= $';
-for $dv (split(/,/,$da)) {
-$dv =~ s/^\s+//; $dv =~ s/\s+$//;
-if (defined($depstrength{$lf})) {
-if (!defined($predefdepfdep{$dv}) ||
-$depstrength{$predefdepfdep{$dv}} < $depstrength{$lf}) {
-$predefdepfdep{$dv}= $lf;
-}
-} else {
-$dk= "$lf: $dv";
-if (!defined($unkdepfdone{$dk})) {
-$unkdepfdone{$dk}= 1;
-$unkdepf{$lf}.= ', ' if length($unkdepf{$lf});
-$unkdepf{$lf}.= $dv;
-}
+last if defined $1; # exact match, otherwise keep looking
+}
+close(SLF);
+
+return 0 unless defined $da;
+
+for $dv (split(/,/,$da)) {
+$dv =~ s/^\s+//; $dv =~ s/\s+$//;
+if (defined($depstrength{$lf})) {
+if (!defined($predefdepfdep{$dv}) ||
+$depstrength{$predefdepfdep{$dv}} < $depstrength{$lf}) {
+$predefdepfdep{$dv}= $lf;
+}
+} else {
+$dk= "$lf: $dv";
+if (!defined($unkdepfdone{$dk})) {
+$unkdepfdone{$dk}= 1;
+$unkdepf{$lf}.= ', ' if length($unkdepf{$lf});
+$unkdepf{$lf}.= $dv;
 }
 }
-return 1;
 }
-close(SLF);
-return 0;
+return 1;
 }
 
 if (!$stdout) {

-- 
see shy jo


signature.asc
Description: Digital signature

[proposal] udeb shlibdeps files

[Joey Hess]

d-i currently suffers from a mess involving the library dependencies of
udebs. Most library udebs have different names than their corresponding
deb, for example, libc6-udeb, but since the udebs that should depend on
them are built on regular Debian systems, their dependencies are either
calculated with dpkg-shlibdeps, and thus are wrong, or are hardcoded,
and thus break, or are noneixistant.

This problem makes d-i less robust at runtime since some needed
libraries are only pulled in by luck. It makes d-i builds more fragile
since the build system requires udeb dependencies to be satisfied, and
so we have to build-depend on a bunch of libraries we do not really use
for the build, and track changes to them. It makes the build hard to
understand as well, since some other libraries enter the d-i images due
to library reduction rather than as udebs, and it's hard to tell which
libraries are intended to work which way.

I propose that we fix this by making dpkg-shlibdeps generate correct
dependencies on library udebs when building udebs. This will mean making
it change its behavior when it is building a udeb, but rather than
hardcode the concept of a udeb at such a low level, I think we can make
it generic and possibly even useful for something else. So let's add a
new switch to dpkg-shlibdeps, -t (for type), which takes a string parameter
such as "udeb". The default type would be "deb".

Then extend the shlibs file format, so some lines start with a marker
indicating they only apply to packages of a given type. For example:

libiw 28 libiw28 (>= 27+28pre8)
udeb: libiw 28 libiw28-udeb (>= 27+28pre8)

If no matching line of the right type are found in a shlibs file, it will
fall back to the untyped lines.

I think that with this small change to dpkg-shlibdeps, we can get the
dependencies striaghtened out, and probably fully automate it all in
debhelper. I would hate to try to implement something similar without
changing dpkg-shlibdeps, so I hope that this meets your approval.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Bug #179320 continues to haunt people.

[Joey Hess]

Kipp Cannon wrote:
> I think dselect should not crash no matter how out of spec a package might
> be, and it certainly should not be crashed by a package that is
> automatically generated by another Debian tool;  so it would be nice if
> either dselect could be fixed or alien modified so as to not produce
> description texts that trigger the bug.

It's really not fair to say that alien produces descriptions that
trigger the bug; it's not as if I'm adding very long lines to package
descriptions in alien. Alien simply uses whatever package description is
in the rpm, and any long lines are preserved.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: dpkg musings

[Joey Hess]

Wichert Akkerman wrote:
> * beter conffile handling
> 
>   I already have this pretty much all implemented based on the proposal
>   on www.dpkg.org. It needs a better 3-way diff function (it calls
>   diff3 in a bad manner right now) and needs interpreter bindings
>   so it can replace ucf, but the basis is pretty much there.

While your API will allow for things like ucf (or aptitude) to use the
same DB as dpkg does for managing config files, it does not allow dpkg's
conffile prompting interface to be replaced. I'm thinking of a debconf
frontend or similar. It also doesn't let a standalone tool like ucf use
the same UI as does dpkg for the prompting.

I suggest having dpkg do the prompting in a standalone binary. dpkg
could provide a basic version and alternatives/diversions be used to 
replace it with other versions. Of course this standalone program would
use the API.

(No comment on the XML foo.)

> + since we can now have an arbitrary number of files for a package
>   we can longer distribute it as seperate .orig.tar.gz, .diff.gz and
>   .dsc but we will have to put it in real package format. This
>   should be either files in a ar archive like the deb format, or
>   (my preference) a tar file.

This has the potential to make it impossible for some of us to continue
maintaining certian source packages, because we do not have enormous
amounts of bandwidth. 

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Problem unravaling mixed system

[Joey Hess]

Colin Watson wrote:
> > strace -o /tmp/trace dpkg -i doc-linux-text_2003.10-1_all.deb
> > 
> > and attach output.  There are some errors at the end but I am afraid
> > this is way beyond my level of expertise (this is actualy the first time
> > I have used strace).
> [...]
> > write(8, "\37\213\10\10]\307u?\2\3Encrypted-Root-Filesys"..., 4302) = -1 
> > ENOSPC (No space left on device)
> 
> Yup, ENOSPC == out of disk space, as you said in a separate mail.
> 
> By the way, please keep all e-mails on the list; others may be
> interested in helping and I don't always have time to see a problem
> through from beginning to end.

I have also seen dpkg segfault when out of space during d-i install
testing.

-- 
see shy jo


signature.asc
Description: Digital signature

Bug#218787: cannot handle swapping 2 files between 2 packages

[Joey Hess]

Package: dpkg
Version: 1.10.18
Severity: normal

debconf has a file that should be in debconf-utils, and debconf-utils has a
file that should be in debconf. But if I swap the files and make debconf-utils
conflict and replace debconf, and debconf conflict and replace debconf-utils
(which also depends on debconf..), dpkg gets confused and wants to remove
debconf-utils entirely:

[EMAIL PROTECTED]:/home/joey/src/debconf>dpkg -i ../debconf_1.3.20_all.deb 
../debconf-utils_1.3.20_all.deb
dpkg: considering removing debconf-utils in favour of debconf ...
dpkg: no, cannot remove debconf-utils (--auto-deconfigure will help):
 debhelper depends on debconf-utils (>= 1.1.1)
  debconf-utils is to be removed.
dpkg: regarding ../debconf_1.3.20_all.deb containing debconf:
 debconf conflicts with debconf-utils (= 1.3.19)
  debconf-utils (version 1.3.19) is installed.
dpkg: error processing ../debconf_1.3.20_all.deb (--install):
 conflicting packages - not installing debconf
dpkg: considering removing debconf in favour of debconf-utils ...
dpkg: no, cannot remove debconf (--auto-deconfigure will help):
 ipmasq pre-depends on debconf (>= 0.5.00)
  debconf is to be removed.
dpkg: regarding .../debconf-utils_1.3.20_all.deb containing debconf-utils:
 debconf-utils conflicts with debconf (= 1.3.19)
  debconf (version 1.3.19) is installed.
dpkg: error processing ../debconf-utils_1.3.20_all.deb (--install):
 conflicting packages - not installing debconf-utils
Errors were encountered while processing:
 ../debconf_1.3.20_all.deb
 ../debconf-utils_1.3.20_all.deb
zsh: exit 1 dpkg -i ../debconf_1.3.20_all.deb 
../debconf-utils_1.3.20_all.deb

My conflicts and replaces are versioned, and I think dpkg should be able
to figure out that it can break the sycle by doing both operations at the
same time.

Any advice to deal with this would be appreciated.

-- System Information:
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux dragon 2.4.22 #1 Sun Oct 12 15:11:10 EDT 2003 i686
Locale: LANG=en_US, LC_CTYPE=en_US

Versions of packages dpkg depends on:
ii  dselect  1.10.18 a user tool to manage Debian packa
ii  libc62.3.2.ds1-8 GNU C Library: Shared libraries an

-- no debconf information

-- 
see shy jo


signature.asc
Description: Digital signature

Bug#216606: dpkg: update-alternatives should be in /usr/bin, not /usr/sbin

[Joey Hess]

Josip Rodin wrote:
> Well, /usr/sbin is already in PATH because dpkg insists on it, so there
> really shouldn't be (m)any of those.

5 maintainer scripts on my laptop do this. Mostly those that do a test
-x for some reason.

-- 
see shy jo, who has given up on trying to teach people that hardcoding
paths is bad. :-(


signature.asc
Description: Digital signature

Bug#212796: kdebase build-depends issue as well

[Joey Hess]

Martin Michlmayr wrote:
> * Joey Hess <[EMAIL PROTECTED]> [2003-09-30 11:16]:
> > > The priority of this bug should probably be RC.
> > 
> > I will probably NMU dpkg to fix this bug in a few days if I don't hear
> > from its maintainers.
> 
> Did you hear from them?

Some vague promise of "soon" from doogie. Unfortunatly, I'm on another
trip..

-- 
see shy jo

Bug#212796: kdebase build-depends issue as well

[Joey Hess]

Chris Cheney wrote:
> The priority of this bug should probably be RC.

I will probably NMU dpkg to fix this bug in a few days if I don't hear
from its maintainers.

-- 
see shy jo


signature.asc
Description: Digital signature

Bug#212796: dpg-builddeps fails for complex deps

[Joey Hess]

Package: dpkg-dev
Version: 1.10.15
Severity: important

The debian-installer build directory has a debian/control as follows:

Source: build-installer
Section: devel
Priority: extra
Build-Depends: apt, bc, slang1-utf8-pic, libnewt-pic, syslinux (>= 2.04-1) 
[i386], sysutils [i386], dosfstools [i386], libdiscover1-pic [!s390 !s390x 
!mips !mipsel], libdiscover1 [!s390 !s390x !mips !mipsel], ncurses-base, 
genext2fs, mklibs (>= 0.1.11), libdebconfclient0, libdebian-installer3, 
modutils, glibc-pic, mtools [i386], bf-utf-source, upx-ucl-beta [i386], 
libbogl-dev, sed (>= 4.0.7-1), libtextwrap1, debhelper
Maintainer: Debian Install System Team 
Uploaders: Joey Hess <[EMAIL PROTECTED]>, Alastair McKinstry <[EMAIL PROTECTED]>
Standards-Version: 3.6.0

dpkg-checkbuilddeps on my system does not notice that I do not have eg,
glibc-pic installed. In fact it misses every build-dep listed after
syslinux. I think the bug is in parsedep, it stops finding new
$dep_and's after syslinux.

Ok, I see.. $use_arch is not set, so it never removes the [i386] from
$build_dep, so it is set after the loop, so it bails.

I won[t even comment on how gross it is that your "improved" dependency
parser exits w/o an error if it has a parse failure. Er, maybe I just
did.

-- System Information:
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux dragon 2.4.22 #1 Tue Aug 26 17:27:18 EDT 2003 i686
Locale: LANG=C, LC_CTYPE=C

Versions of packages dpkg-dev depends on:
ii  binutils   2.14.90.0.6-3 The GNU assembler, linker and bina
ii  cpio   2.5-1.1   GNU cpio -- a program to manage ar
ii  make   3.80-4The GNU version of the "make" util
ii  patch  2.5.9-1   Apply a diff file to an original
ii  perl [perl5]   5.8.0-21  Larry Wall's Practical Extraction 
ii  perl-modules   5.8.0-21  Core Perl modules.

-- no debconf information

-- 
see shy jo


signature.asc
Description: Digital signature

Re: logcheck: Dangerous usage of /var/tmp ?

[Joey Hess]

Adam Heath wrote:
> Consider the case of /usr.  Almost all packages have that in their deb.  And
> if just one package has odd perms on that dir, then very odd things could
> occur, when it is unpacked.

This is the argument you gave last time, and I still hold that it would
be better to find out immediatly that a broken package has made /usr
mode 400 than it would be to wait for a bug report from some poor user
two happens to install that package before any others and gets a broken
system two years down the road.

Of course /usr is a poor example; something like /etc/X11 is a much
better example.

> Also, it could be argued, that /var/tmp should exist in any deb.  It's
> equivalent to /tmp.  Maybe this should be mentioned in policy.

Agreed.

-- 
see shy jo


pgpt13kZ7oyiv.pgp
Description: PGP signature

Bug#203782: dpkg-statoverride alwats prints "stripping trailing /"

[Joey Hess]

Package: dpkg
Version: 1.10.10
Severity: normal

dpkg-statoverride contains three copies of this:

s,/*$,, && print STDERR "stripping trailing /\n";

Problem is, /* matches "", and this will in fact happily substitute
nothing for nothing, return 1, and print the message, when no slashes
are involved at all:

[EMAIL PROTECTED]:~>perl -le '$_="foo"; print s,/*$,,'
1
[EMAIL PROTECTED]:~>dpkg-statoverride --list foo
stripping trailing /

So every maintainer script that uses dpkg-statoverride prints out these
useless messages.

Fix is to make sure your pattern actually matches something:

s,/+$,, && print STDERR "stripping trailing /\n";

-- System Information:
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux dragon 2.4.21 #1 Fri Jun 13 21:33:56 EDT 2003 i686
Locale: LANG=C, LC_CTYPE=C

Versions of packages dpkg depends on:
ii  dselect   1.10.10a user tool to manage Debian packa
ii  libc6 2.3.1-17   GNU C Library: Shared libraries an

-- no debconf information

-- 
see shy jo


pgp6vfeTHvdO6.pgp
Description: PGP signature

Re: dpkg and debhelper patches for lib64 support

[Joey Hess]

I like this idea. It has reach.

Adam Heath wrote:
> Package: xlibs
> Package.athlon: xlibs-athlon
> Depends.any: coreutils
> Depends: libgcc1
> Depends.athlon: libathlon_gcc1
> Architecture: any
> Architecture.athlon: x86-64

So then it would look something like this for a generic lib package:

Package: libfoo1
Package.64: lib64foo1
Depends: ${shlib:Depends}
Architecture: any
Description: the foo library
 The foo library for all your bar needs.
Description.64: the 64 bit foo library
 The foo library for all your 64 bit bar needs.
 If you need to run 32 bit foo programs, install libfoo1.

Package: libfoo-dev
Package.64: lib64foo-dev
Architecture: any
Depends: libfoo1
Depends.64: lib64foo1
Conflicts.64: libfoo-dev
Description: the foo library development files
 Development files for the foo library.
Description.64: the 64 bit foo library development files
 Development files for the 64 bit foo library.
 .
 If you need to build 32 bit programs using foo, install libfoo-dev
 instead. Currently that package cannot be installed at the same time as
 this one.

> A user compiling the package can choose the default, or can choose to compile
> for athlon.

There will need to be a way to set the default subfield[1] on a per arch
basis, because I think the x86-64 arch will want to default to using the
.64 fields.

> When generating an output field, the variant of the field is choosen.  If one
> isn't found, then the standard field is used.  Additionally, .any fields are
> merged in with whatever field is choosen.

I'm ambivilant about this any thing, and I note it has to magically add
the right delimiters when merging fields -- ", " for relationship and
architecture fields, " " for descriptions, and presumably no delimiters
for other fields.

That said, could the .any field merge short and extended descriptions?

Package: libfoo1
Package.64 lib64foo1
Description.any: the foo library
 The foo library for all your bar needs.
Description.64: (64 bit version)
 .
 This is a 64 bit version of the library. If you need to run 32 bit
 foo programs, install libfoo1.

Yeilding:

Package: libfoo1
Descrption: the foo library
 The foo library for all your bar needs.

Package: lib64foo1
Description: the foo library (64 bit version)
 The foo library for all your bar needs.
 .
 This is a 64 bit version of the library. If you need to run 32 bit
 foo programs, install libfoo1.

This might be asking too much.

> Additionally, this support is really simple to do, and could be done with
> dpkg-gencontrol, or even some preprocessing magic that reads control.in.

Remember that dpkg-genchanges will also need to take this into account in
its debian/cpntrol parsing. See bug #150597.

I'd hope we could avoid templates once we get past prototyping[2], as they're
just kinda gross. Anyway, I hope this new mechanism will do away with
a whole class of existing control file templates: the ones used for kernel
module packages.

Package: linux-wlan-ng
Architecture: i386 powerpc arm alpha hppa
Description: utilities for wireless prism2 cards

Package.modules: linux-wlan-ng-modules-${kvers}
Architecture.modules: i386 powerpc arm alpha hppa
Provides.modules: linux-wlan-ng-0.2.0-modules
Description.modules: drivers for wireless prism2 cards

Here if it's building the kernel modules it sets the subfield to
"modules", and sets kvers in substvars. For a normal build it doesn't
build modules, but just the linux-wlan-ng package. I'm assuming that the
dpkg tools would be smart enough to ignore the second stanza if the
subfield is not set. If not I'd have to remove the blank line between
the stanzas, which would be uglier. 

I have not figured out what would keep it from building the
linux-wlan-ng package when the subfield is set. Maybe there would be a
way to set the subfield from inside the rules file, so I use them in
both stanzas: 

Package.normal: linux-wlan-ng
Architecture.normal: i386 powerpc arm alpha hppa
Description.normal: utilities for wireless prism2 cards

Package.modules: linux-wlan-ng-modules-${kvers}
Architecture.modules: i386 powerpc arm alpha hppa
Provides.modules: linux-wlan-ng-0.2.0-modules
Description.modules: drivers for wireless prism2 cards

binary:
if $(BUILDING_KERNEL_MODULES); then
dpkg-setsubfield modules
else
dpkg-setsubfield normal
fi

But this would not work unless dpkg-genchanges stopped parsing the
control file too -- otherwise it would find no packages to put in the
changes file!

-- 
see shy jo

[1] It needs a better name.
[2] Now I can't stop thinking about C++.


pgpIbxyIIm482.pgp
Description: PGP signature

Re: dpkg and debhelper patches for lib64 support

[Joey Hess]

Bart Trojanowski wrote:
> On a 64 bit kernel with 32 bit binary emulation support, you can run
> aything.  However, when using a 32 bit kernel, one must have at least a
> 32 bit modutils.  Running a 64 bit insmod, which I had the displeasure
> of doing once, is disastrous on a 32 bit kernel.
> 
> There may be other instances where a package (ex: the i386 kernel) would
> require another to provide a particular binary variant (ex: the i386
> modutils).
> 
> Granted this is a small problem.

This kind of thing is why I think that when dpkg gets support for
installing multiple architectures on a single machine, dependencies
should also get support for architectures added to them in some form or
another. Possibly like this:

Depends: package (version) [arch]

There are other more flexible solutions involving virtual packages,
which have been proposed by Marcus Brinkmann before.

-- 
see shy jo


pgpSUQSFpOnGL.pgp
Description: PGP signature

Re: dpkg and debhelper patches for lib64 support

[Joey Hess]

Adam Heath wrote:
> No to ${field}, ${field}32, ${field}64.  Yes to ${field}[.${variant}].
> 
> This allows for completely abstract package field variants, and targets.
> It's not hard-coded to 32/64, and can be extended for athlon, ppc64, sse,
> whatever.

Could you write out an example?

-- 
see shy jo


pgpZXhmj2D68Z.pgp
Description: PGP signature

Re: dpkg and debhelper patches for lib64 support

[Joey Hess]

Adam Heath wrote:
> On Tue, 17 Jun 2003, Gerhard Tonn wrote:
> 
> > >  Package: aalib1
> > > +Package64: lib64aalib1
> > >  Architecture: any
> > >  Depends: ${shlibs:Depends}, ${misc:Depends}
> > >  Description: ascii art library
> > >
> > > I would rather maintain a separate copy of this stanza for the 64 bit
> > > package. This would also let me give it a useful description.
> > >
> >
> > Again, I think, this creates a lot of redundancy, since 99 per cent of the
> > information of both packages are identical. If we add another copy for the 
> > 64
> > bit package, it is open how to name the Architecture for the 64 bit as well
> > as for the 32 bit variant.
> 
> Package: libfoo1
> Description: ${description}
>  This package contains the 32 bit version of the foo library.
> 
> Package: lib64foo1
> Description: ${description}
>  This package contains the 64 bit version of the foo library.

That was already more or less proposed, and shot down. What do you
suggest goes in the Architecture field of lib64foo1?

-- 
see shy jo


pgpzIvwZdqaBJ.pgp
Description: PGP signature