Re: Temporary solution for changelog problem in binNMUs

[Goswin von Brederlow]

On Mon, May 13, 2013 at 11:09:53PM +0200, Ansgar Burchardt wrote:
> Hi,
> 
> [ dropped -release and -wb-team ]
> 
> Jonathan Nieder  writes:
> > One problem that that doesn't solve is what to do when a package would
> > be able to borrow its /doc/ directory from another package
> > (using a symlink) but for the changelog and copyright (which gets even
> > harder when binnmus are involved).  See http://bugs.debian.org/556015
> 
> Right, that would need looking into. A random idea: if doc/ is
> a symlink to doc/, then copy the binNMU part of the
> changelog to doc//changelog.Debian..gz, but do not
> generate doc//changelog.Debian.gz.
> 
> Ansgar

Huh?

If the changelog is symlinked and the package is binNMUed don't you
end up with something like in the deb?

foo-common:
/usr/share/doc//changelog.Debian.gz

libfoo (binNMU):
/usr/share/doc/ -> 
/usr/share/doc//changelog.Debian.binNMU-.gz (binNMU stanza)

The split off binNMU part needs to be arch qualified for multiarch or
you have the same problem as before in case binNMUs for multiple archs
are done independently (e.g. not the same reason).

I wouldn't want the original changelog to be modified at all at
install time and the other file causes no problem if it is arch
qualified.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130514142258.GJ27079@frosties

Re: architecture-specific dependencies on virtual packages

[Goswin von Brederlow]

On Mon, Jul 09, 2012 at 02:00:14PM +0200, David Kalnischkies wrote:
> Hi,
> 
> while playing around with the APT code regarding architecture-specific
> dependencies I stumbled over the handling of Provides in that context:
> 
> Package: pkga
> Status: install ok installed
> Architecture: i386
> Provides: foo
> 
> Package pkgb
> Architecture: amd64
> Conflicts: foo:amd64
> 
> 
> The result with dpkg (1.16.7):
> dpkg: regarding .../pool/pkgb_1_amd64.deb containing pkgb:
>  pkgb conflicts with foo:amd64
>   pkga provides foo and is present and installed.
> 
> 
> Even more interesting, if I change this conflict to a depends dpkg happily
> installs the package and dpkg --audit gives no indication that a dependency
> is unsatisfied (same problem with depends if no architecture is specified).
> 
> Shouldn't provides be limited by the architecture their provider can work on?
> Or did I miss something here?
> 
> 
> Best regards
> 
> David Kalnischkies

Imho a Povides creates an alias for the package. So

Package: pkga
Status: install ok installed
Architecture: i386
Provides: foo

should behave identical to

Package: foo
Status: install ok installed
Architecture: i386


Note that the behaviour of a real package changes depending on the
Multi-Arch setting and the same should be true for a provided package.

The only difference with a provides should be that provides don't have
a version.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120718103526.GA5976@frosties

Re: BinNMU changelog handling for Multi-Arch: same packages

[Goswin von Brederlow]

On Thu, Jul 12, 2012 at 08:59:38AM +0200, Raphael Hertzog wrote:
> On Thu, 12 Jul 2012, Guillem Jover wrote:
> > On Wed, 2012-07-11 at 09:23:05 +0200, Raphael Hertzog wrote:
> > > 2/ we modify dpkg to allow co-installation of M-A: same packages which 
> > > share the
> > >same source version regardless of the binary version
> > 
> > As I've said before, this right here seems unacceptable. This implies
> > at least:
> > 
> >  * loosing the binNMU changelog entry, with a version in the changelog
> >not matching the one on the dpkg db (in possibly both directions).
> >  * making installed file contents flip-flop depending on what package
> >got installed last.
> >  * making dpkg unable to detect different generated file contents on
> >different binary rebuilds.
> 
> There's a mis-understanding here. I was not telling to drop the check
> that ensures that the files are identical between the various arches.
> Instead I'm just saying that we must change the check that ensures that
> all package are at the same version to use the source version and not the
> binary version.
> 
> And this is required even if we move the changelog to control.tar.
> Otherwise the release team will still have to bin-nmu all arches at the
> same time, which is what they would like to avoid.

In the interim till this is solved properly, and this might be from
till wheezy up to as long as wheezy+1 if things go bad, may I suggest
the following:

* For the instability of a package from different architectures the
  source version counts. A bin-NMU version does not conflict with
  a normal version.
* Frontends put a high value on trying to match versions. If arch1
  is updated to a bin-NMU then arch2 will be updated to the same bin-NMU
  too where available.

Those two will have to happen, as Raphael says, no matter what the
changelog solution will eventually be to allow the release team to
only bin-NMU some archs for MA packages. So we might as well do that
now.

* File conflicts in /usr/share/doc// between packages of the same
  source version are ignored. This could be even further limited to
  changelogs.

Yes, this is huge hack, a violation of layering, yada yada yada. But it
is simply the easiest quick fix. And since programs must not rely on
/usr/share/doc/ to be available any inconsistencies there will
be harmless.

It also preserves the changelog entries in packages so apt-listchanges
can display it and has no efect on non-multiarch systems.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120712084455.GB25594@frosties

Re: RFC: Proposal to combine diversions and Replaces into DEBIAN/replaces

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Sato Mayamoto wrote:
>
>> Is it too complicated to have a kind of automatization for divert as
>> a feature in deb-packages?
>
> It sounds like a good feature if someone has time to work on it.  In
> fact, I think someone did a Google summer of code project on it last
> summer, though they were not very active on this list proposing design
> or initial patches.
>
> I imagine that the people on this list would be happy to help out with
> anyone working on that.  As a first step in reviving the effort, maybe
> it would make sense to describe what effect a declared diversion
> should have at each stage of the package installation and removal
> process.  What happens at preinst, postinst, prerm, and postrm time?
> Especially: what happens during upgrades to a version without the
> diversion?  What happens if two packages declare a diversion for the
> same file?
>
> Jonathan

The behaviour would be what is currently done with dpkg-divert in
maintainer script for the usual use case of dpkg-divert. Just that it
would do it automatically for each file listed (or no longer listed :).

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87mx3xkoc7.fsf@frosties.localnet

Re: Handling of changelogs and bin-nmus

[Goswin von Brederlow]

Guillem Jover  writes:

> On Sun, 2012-06-10 at 14:40:28 +0200, Raphael Hertzog wrote:
>> On Sun, 10 Jun 2012, Guillem Jover wrote:
>> > As I mentioned in the long ref-counting thread, I strongly disagree this
>> > is a correct solution, it just seems like a hack to me. Instead I
>> > think we should consider changelog (and copyright as long as it's in
>> > machine parseable format) as dpkg metadata (something dpkg misses
>> > compared to rpm or other package managers for example) and as such they
>> > should go into the .deb control member, which would end up in the dpkg
>> > database w/o any kind of file conflict, and very minor packaging effort
>> > as for most that would be handled by helpers.
>> 
>> I agree that we should move into this direction. Still I believe it's
>> important to distinguish "source changelog" and "binary changelog".
>> And while we might not want to keep this distinction in the generated
>> package, we should have it at the source package level.
>
>> As such, I suggest that we handle "binary rebuild" differently:
>> - debian/changelog is left unmodified since it's the source changelog
>>   => it defines the ${source:Version} substvar
>> - debian/changelog.binary-rebuild (or any other better name) is created
>>   when we want to do a bin-nmu
>>   => it defines the ${binary:Version} and it's not included in
>>   the generated source package
>
> By definition a binNMU cannot produce a source package anyway, so I
> fail to see the point in this artifical need to distinguish “source”
> and “binary” changelogs through different files, AFAIR I already
> mentioned reasons against this in the ref-counting thread, and now
> again in reply to Ian's mail.

But if a binNMU changes debian/changelog then that change ends up in the
binary package. And that means you get a file collision for multiarch
where 2 packages contain the same file name but different contents.

So what is your solution there?

And no, moving the changelog to a -common deb does not solve the problem
since the binNMU would require a new -common deb while the original
wants the old one.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ehpjo8u1.fsf@frosties.localnet

Re: dpkg fails to install 32-bit adobe acrobat reader on 64-bit wheezy system: depends issue

[Goswin von Brederlow]

Guillem Jover  writes:

> On Fri, 2012-04-20 at 14:58:33 -0700, Chris Hiestand wrote:
>> Actually now that I'm trying to install from scratch instead of upgrade
>> the message seems to be different (libgtk2.0-0:any instead of libgtk2.0-0):
>
>> > chiestand@wheezy:/tmp$ sudo dpkg -i --force-architecture 
>> > adobereader-enu_9.5.1_i386.deb
>> > dpkg: warning: overriding problem because --force enabled:
>> >  package architecture (i386) does not match system (amd64)
>> > Selecting previously unselected package adobereader-enu.
>> > (Reading database ... 782797 files and directories currently installed.)
>> > Unpacking adobereader-enu (from adobereader-enu_9.5.1_i386.deb) ...
>> > dpkg: dependency problems prevent configuration of adobereader-enu:
>> >  adobereader-enu depends on libgtk2.0-0:any.
>> > dpkg: error processing adobereader-enu (--install):
>> >  dependency problems - leaving unconfigured
>> > Processing triggers for man-db ...
>> > Errors were encountered while processing:
>> >  adobereader-enu
>> 
>> Perhaps the upgrade message should say libgtk2.0-0:i386 instead?
>
> Well, certainly at least something clearer than libgtk2.0-0:any. I've
> put that on my list of messages to clarify for 1.16.3 or 1.16.4.
>
> thanks,
> guillem

How would it know that? If libgtk2.0-0 is Multi-Arch:foreign then
libgtk2.0-0:amd64 will do just fine.

I guess if it knows about libgtk2.0-0:amd64 not bein Multi-Arch:foreign
it can say :i386 but carefull not to assume the same for completly
unknown packages.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762co2kxp.fsf@frosties.localnet

Re: dep-trace v. tsort (dpkg, source, bsd, reason)

[Goswin von Brederlow]

"John D. Hendrickson and Sara Darnell"  writes:

> Thanks !  I'm sure someone "is aware" :)  Was not sure it posted.
>
> Correct and yes your right, but not "optimal".  Let's say apt selected this 
> depends list:
>
> A
> B C
> D
>
> All satisfied as a list to install, yes.  Should work, the book says.
>
> But if installed in order "C B D A" it may cause headaches (see below).

If it causes headaches then the packages are bugy because they did not
set the right dependencies. And if they did not set the right
dependencies no sorting tool will be able to sort them magically right.
Apt and dpkg already use all the dependency informations to sort
packages correctly. The order they find is not unique but is no better
or worse than any other order that doesn't break the dependency
restrictions.

> If optionally applied in order "A B C D", sorted by deps, success is more 
> likely.

The dependencies of packages define a partial order and any order that
does not violate that partial order has to succeed. Otherwise the
packages dependencies are buggy.

And since people don't always install "A B C D" as set but frequently
just update a subset it is pretty much garantied that over time and many
many users any order of that set that is allowed will be tried. So this
really is a must work. There is no "is more likely", it has to work in
any order.

> # in a 90's release, "buzz", install guide called lack of order "a bit 
> loopy"

Dpkg and its frontends have come a long way sinze buzz.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4x364q9.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Guillem Jover  writes:

> On Wed, 2012-02-15 at 16:32:38 -0800, Russ Allbery wrote:
>> Guillem Jover  writes:
>> > If packages have to be split anyway to cope with the other cases, then
>> > the number of new packages which might not be needed otherwise will be
>> > even smaller than the predicted amount, at which point it makes even
>> > less sense to support refcnt'ing.
>> 
>> I don't think the package count is really the interesting metric here,
>> unless the number of introduced packages is very large (see below about
>> -dev packages).  I'm more concerned with maintainer time and with
>> dependency complexity, and with the known problems that we introduce
>> whenever we take tightly-coupled files and separate them into independent
>> packages.
>
> Well, people have been using the amount of packages as a metric, I've
> just been trying to counter it. It also in a way represents the amount
> of work needed.
>
> About tightly-coupled files, they can cause serious issues also with
> refcounting, consider that there's always going to be a point when
> unpacking one of the new instances will have a completely different
> vesion than the other already unpacked instance(s). So packages could
> stop working for a long time if say unpacked libfoo0:i386 1.0 has
> file-format-0, but new libfoo0:amd64 4.0 has file-format-2, and the
> file didn't change name (arguably this could be considered an upstream
> problem, depending on the situation), this would be particularly
> problematic for pseudo-essential packages.

That is not an argument for or against refcounting. If at all it would
be marginally for refcounting:

The same situation would occur with libfoo0:i386 1.0, libfoo0:amd64 4.0
and libfoo0-common:all 2.0. But now even worse because you have 3
versions that can be out-of-sync.

Actualy if the file is shipped in the package then ref counting would
automatically detect the difference in contents and fail to install the
new libfoo0:amd64 4.0. And if the file is not shipped in the package
then ref counting has no effect on it. Again ref counting comes out
better.

Ref counting would catch some of those cases but not all and it never
makes it worse. What solves this problem is the same version requirement
or simply adding Breaks: libfoo0 (<< 4.0~) to libfoo0:* 4.0. The only
point you've made is that ref counting isn't a magic bullet that brings
us world peace.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762ekkh3g.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

m...@linux.it (Marco d'Itri) writes:

> On Mar 01, Russ Allbery  wrote:
>
>> The situation with refcounting seems much less fragile than the situation
>> without refcounting to me.
> I totally agree.
>
> Also, why does refcounting have to be "perfect"?
> What would break if it did not actually check that the two files 
> provided by the same package for different architectures are identical?

Everything that can go wrong when splitting packages. You would loose
the stability advantage.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87aa3wkhpv.fsf@frosties.localnet

Re: Upcoming dpkg 1.16.2 upload

[Goswin von Brederlow]

Guillem Jover  writes:

> On the other hand I've also become disappointed after possibly realizing
> that Debian's culture seems to have been shifting into something different
> than what it was when I joined, where technical excellence seems to be
> less important than rushing things out, patience is not considered an
> important virtue anymore, but being pushy and demanding are, etc.
> Which means I've been and will be reconsidering what my involvment in
> the project should be.

You are kind of implying that you are the only one with technical
excellence and that all the people that designed, wrote and send you
multiarch stuff do not.

It saddens me too that Debian's culture seem to have changed to where a
single person things he has the only technical excellence on a subject.


Please do not quit but maybe stop putting it all on just your
shoulders. There must be someone else out there that you consider to
have sufficient technical experteese to work with you. And if not then
it would be even more important to bring someone else up to code. Having
such a central component as dpkg rely on just one person is a disaster
and the 10 month delay to review the multiarch patches will be the least
of the problems over time. Say next year you do quit or something
happens to you where would dpkg and Debian stand then.

MfG
Goswin

PS: having someone else on the team can also mean you can offload more
stuff you don't like on them and do more of the stuff you do like in
dpkg.



-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87eht8khu4.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

David Kalnischkies  writes:

> On Thu, Feb 16, 2012 at 23:10, Carsten Hey  wrote:
>> * David Kalnischkies [2012-02-16 03:59 +0100]:
>>> On Thu, Feb 16, 2012 at 00:39, Russ Allbery  wrote:
>>> (the only problem i see is that i don't have ${source:Version} available
>>>  currently in the version structure, but we haven't even tried pushing
>>>  apt's abibreak to sid specifically as i feared "last-minute" changes…)
>>
>> I'm not sure if you meant this with "Source tag", anyway, the 'Packages'
>> files miss the source version too, but this could be added as optional
>> field that would be used if it differs from the 'Version:' field.
>
> It's already in for quiet some time ('current' sid amd64, first hit):
> Package: 3depict
> Source: 3depict (0.0.9-1)
> Version: 0.0.9-1+b1
> […]
>
> It's used in other places in APT, e.g. 'apt-get source', which just looks
> at the Packages file stanza. That's fine as this isn't a speed critical
> operation - but if we want it for the lock-step operation apt needs that
> piece of information in its internal structures for fast access to it and
> adding new fields in these structures will require an abibreak.
> That's the intended meaning of the quoted sentence.

Except that doesn't have to work (sorry for the ubuntu part):

Package: gcc
Source: gcc-defaults (1.93ubuntu1)
Version: 4:4.4.3-1ubuntu1

What would the version be for a binNMU of gcc-defaults? I think it would
be

Package: gcc
Source: gcc-defaults (1.93ubuntu1)
Version: 4:4.4.3-1ubuntu1+b1

What we want is for apt/dpkg to consider this to be compatible with
"4:4.4.3-1ubuntu1".

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4xly64e.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Russ Allbery  writes:

> I think it would be better to have a world in which all the architectures
> of the foo package on the system have to have the same version, because
> then you don't have to treat foo:i386 and foo:amd64 like they're separate
> packages.  The list of installed architectures is an *attribute* of the
> package.  A package is still either installed or not installed, but when
> it's installed, it can be installed for one or more architectures.  But if
> it's installed for multiple architectures, those architectures are always
> upgraded together and always remain consistent.  That avoids all weirdness
> of having a package work differently because the version varies depending
> on the ABI, and it significantly simplifies the mental model behind the
> package.

In such a world architecture all could also be considered another
architecture. And then foo:i386, foo:amd64 and foo:all could be
coinstallable. That would mean that files shared between architectures
could be moved into foo:all and foo:any could implicitly depend on
foo:all. The benefit of this over foo-common would be that apt-cache
search, apt-cache policy, aptitude, dpkg --remove, ... would only have
one package (foo) instead of 2 (foo + foo-common).

This has been previously suggested too but has been droped because it
would be incompatible with existing systems (i.e. monoarch dpkg couldn't
install packages from such a world).

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vcmxy6dk.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Joey Hess  writes:

> Goswin von Brederlow wrote:
>> pkg:arch will still be unique and the dpkg/apt output will use the
>> architecture where required for uniqueness. So I think that after some
>> getting used to it it will be clear enough again.
>
> Here are a few examples of the problems I worry about. I have not
> verified any of them, and they're clearly biased toward code I am
> familiar with, which suggests there are many other similar problems.
>
> * Puppet not only installs packages, it may remove them. A puppet config
>   that does dpkg --purge foo will fail if multiarch is enabled, now
>   it needs to find and remove foo:*
>
> * dpkg-repack pkg:arch will create a package with that literal name (or fail)
>
> * dpkg-reconfigure probably can't be used with M-A same packages.
>   debconf probably generally needs porting to multiarch.
>
> * tasksel uses dpkg --query to work out if a task's dependencies are
>   installed. In the event that a library is directly part of a task,
>   this will fail when multiarch is enabled.
>
> * Every piece of documentation that gives commands lines manipulating
>   library packages is potentially broken.
>
> Seems like we need a release where multiarch is classed as an
> experimental feature, which when enabled can break the system. But the
> sort of problems above are the easy to anticipate ones; my real worry is
> the unanticipated classes of problems. Especially if we find intractable
> problems or levels of complexity introduced by dropping the unique
> package name invariant.
>
> My nightmare scenario is that we release with multiarch, discover that
> it's a net negative for our users (proprietary software on amd64 aside,
> nearly all the benefits are to developers AFAICS), and are stuck with it.

The specs were initialy written in such a way that single arch systems
would not change, that multiarch packages would keep functioning with a
mono-arch apt/dpkg and I think this was preserved so far.

If all interface changes foolow that idea then worst case tools will not
work in a multiarch configuration but still work in a monoarch
configuration. So let multiarch be experimental and only for developers
and risk takers. That is already a huge number of people.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkc9y6p4.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Josselin Mouette  writes:

> Le lundi 13 février 2012 à 22:43 -0800, Russ Allbery a écrit : 
>> There's been a lot of discussion of this, but it seems to have been fairly
>> inconclusive.  We need to decide what we're doing, if anything, for wheezy
>> fairly soon, so I think we need to try to drive this discussion to some
>> concrete conclusions.
>
> Thank you very much for your constructive work.
>
>> 3. Generated documentation.  Here's where I think refcounting starts
>>failing.
>
> So we need to move a lot of documentation generated with gtk-doc or
> doxygen from -dev packages to -doc packages. But it really seems an
> acceptable tradeoff between the amount of work required and the
> cleanness of the solution.
>
>> Does this seem comprehensive to everyone?  Am I missing any cases?
>
> Are there any cases of configuration files in /etc that vary across
> architectures? Think of stuff like ld.so.conf, where some plugins or
> library path is coded in a configuration file.

Generally conffiles in library packages is a violation of policy
8.2. They would create a file overwrite conflict if the SONAME
changes. Putting the conffile into the -common package is a good
solution.

There are some exceptions where the conffiles are version qualified.
E.g.: libgtk2.0-0: /etc/gtk-2.0/im-multipress.conf

For conffiles that vary across architectures the path or name must
include the multiarch triplet.
E.g: libc6: /etc/ld.so.conf.d/x86_64-linux-gnu.conf

(Note: this is actualy a violation of policy 8.2 and needs to be fixed
when we get a libc7).

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nuhzm2m.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Russ Allbery  writes:

> Carsten Hey  writes:
>> * Russ Allbery [2012-02-16 14:55 -0800]:
>
>>> Every file that differs has to be fixed in the current multi-arch plan.
>>> Documentation that contains its build date is going to need to be split
>>> out into a separate -docs package.
>
>> I doubt that ftpmaster would be happy about -doc packages that contain
>> just a few small man pages.
>
> I think they'll be okay with it when it's the cost of reasonable
> multiarch.
>
> I feel fairly strongly that it isn't sane to have a file overlap when the
> file doesn't match.  You then lose the error detection when there are real
> problems, and I don't trust any of us, myself included, to correctly tag
> files where it doesn't matter.
>
> On this front, I agree with Guillem: some amount of package splitting is
> fine, and package splitting instead of additional complexity, like tagging
> files that are allowed to vary, looks like a good tradeoff to me.  The
> splitting that I'm worried about is where the files are tightly coupled,
> which is not the case for development man pages that are now in -dev
> packages.

+1.

I find the argument about past experience with splitting packages and
upstream later changing files so they become arch dependent convincing
in this regard. Refcounting and no exception for file differences seems
to be the best way.

>> debianutils uses a special make target 'prebuild' in debian/rules to
>> update build system related files and PO files before the actual source
>> package is built.
>
>> This basic idea also could be used to build problematic documentation
>> files on the maintainers computer before he/she builds the package.  The
>> other targets would then install the prebuilt documentation into the
>> package without the need to build it first.  A proper dependency on
>> debian/$prebuilt_doc could ensure that maintainers do not forget to run
>> debian/rules prebuild.
>
>> If maintainers choose to use such a target, suggesting a common name for
>> it in the developers reference could be reasonable.
>
> That's an interesting idea.  That's very similar to what I already do as
> upstream (I build POD-generated man pages from my autogen script, and in
> Debian packaging don't bother to regenerate them).

Indeed. Another +1.

You are probably not the only one doing something like this. Who else
does this? What automatism do you have in your debian/rules to help?
Lets see if we can get a good set of examples to work out some
recommendations.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/878vjtzmjj.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Russ Allbery  writes:

> If this is comprehensive, then I propose the following path forward, which
> is a mix of the various solutions that have been discussed:
>
> * dpkg re-adds the refcounting implementation for multiarch, but along
>   with a Policy requirement that packages that are multiarch must only
>   contain files in classes 1 and 2 above.
>
> * All packages that want to be multiarch: same have to move all generated
>   documentation into a separate package unless the maintainer has very
>   carefully checked that the generated documentation will be byte-for-byte
>   identical even across minor updates of the documentation generation
>   tools and when run at different times.
>
> * Lintian should recognize arch-qualified override files, and multiarch:
>   same packages must arch-qualify their override files.  debhelper
>   assistance is desired for this.

I think that, provided the files are byte for byte identical across
architectures they need not be arch qualified. So they should be
refcounted and having non-identical files across archs should be
forbidden by policy. The maintainer must then resolve this by 1) making
the file identical across archs or 2) arch qualifying the name.

So lintian should support arch qualified names but policy should not
needlessly require them.

> * Policy prohibits arch-varying data files in multiarch: same packages
>   except in arch-qualified paths.
>
> * The binNMU process is changed to add the binNMU changelog entry to an
>   arch-qualified file (changelog.Debian.arch, probably).  We need to
>   figure out what this means if the package being binNMU'd has a
>   /usr/share/doc/ symlink to another package, though; it's not
>   obvious what to do here.
>
> Please note that this is a bunch of work.  I think the Lintian work is a
> good idea regardless, and it can start independently.  I think the same is
> true of the binNMU changelog work, since this will address some
> long-standing issues with changelog handling in some situations, including
> resolving just how we're supposed to handle /usr/share/doc symlinks.  But
> even with those aside, this is a lot of stuff that we need to agree on,
> and in some cases implement, in a fairly short timeline if this is going
> to make wheezy.

In case /usr/share/doc/pkg is a symlink the binNMU changelog should be
stored in the destination of the symlink. For this to work the (binNMU)
changelog should be both arch and package qualified
(changelog.Debain.bin-pkg.arch). This would allow libfoo:any,
foo-bin:any + foo-common:all to be binNMUed. Without the package
qualifier the libfoo and foo-bin package would both contain
/usr/share/doc/foo-common/changelog.Debian.arch and produce a file
overwrite error.

After a binNMU (on amd64) the following files would exist:

foo-bin:/usr/share/doc/foo-bin -> foo-common
foo-bin:/usr/share/doc/foo-common/changelog.Debian.foo-bin.amd64
foo-common: /usr/share/doc/foo-common/changelog.Debian
libfoo: /usr/share/doc/foo-common/changelog.Debian.libfoo.amd64
libfoo: /usr/share/doc/libfoo -> foo-common

The binNMU changelogs would be identical wasting a little disk and
mirror space and bandwidth but that is probably unavoidable.

One way to reduce the overhead would be to split the binNMU entry into a
separate changelog:

foo-bin:/usr/share/doc/foo-bin -> foo-common
foo-bin:/usr/share/doc/foo-common/changelog.binNMU.foo-bin.amd64
foo-common: /usr/share/doc/foo-common/changelog.Debian
libfoo: /usr/share/doc/foo-common/changelog.binNMU.libfoo.amd64
libfoo: /usr/share/doc/libfoo -> foo-common

The binNMU changelogs would be just one entry each, the reason for the
binNMU.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d395zn6m.fsf@frosties.localnet

Re: dpkg behavior while changing a foreign package from arch:any to arch:all (and v.v.)

[Goswin von Brederlow]

Raphael Hertzog  writes:

> On Fri, 10 Feb 2012, David Kalnischkies wrote:
>> I am not sure through if this is really a bug in dpkg, given that arch:all is
>> to be interpreted as arch:native in multiarch, so on an amd64 system a
>> flip from arch:amd64 to arch:all is not really a flip (and therefore works)
>> while arch:i386 to arch:all is not (and therefore doesn't work).
>> So this might be very well the intended behavior and not a bug.
>
> At the very least it was coded that way and the test suite ensures that
> this is the case:
> # M-A: $* / Upgrade from arch: any-foreign -> all
> $(DPKG_INSTALL) pkg-ma-$*_1.0_$(FOREIGN_ARCH).deb
> ! $(DPKG_INSTALL) pkg-ma-$*_1.0_all.deb
> $(call pkg_is_installed,pkg-ma-$*:$(FOREIGN_ARCH))
> $(call pkg_is_not_installed,pkg-ma-$*)
> $(DPKG_PURGE) pkg-ma-$*:$(FOREIGN_ARCH)
> [...]
> # M-A: $* / Upgrade from arch: all -> any-foreign
> $(DPKG_INSTALL) pkg-ma-$*_1.0_all.deb
> ! $(DPKG_INSTALL) pkg-ma-$*_1.0_$(FOREIGN_ARCH).deb
> $(call pkg_is_not_installed,pkg-ma-$*:$(FOREIGN_ARCH))
> $(call pkg_is_installed,pkg-ma-$*)
> $(DPKG_PURGE) pkg-ma-$*
>
> The reasoning is precisely the one you gave. It's also coherent
> with the current impossibility to cross-grade an "Architecture: any /
> Multi-Arch: foreign/no" package from native to foreign and vice-versa.
>
> This is just one reason more that we have to change this at some point
> to properly support cross-grades from one arch to the other.
>
>> Somehow I get the impression it would be good to have an option in dpkg
>> to tell dpkg what it should consider as "native" architecture similar to
>> APT::Architecture to easily solve these kind of communication problems.
>
> Hum, I had not thought of that. It could also be useful to use the host's
> dpkg in a chroot of a foreign architecture. Or to use a foreign dpkg for
> some reasons (perf? availability?) and still assume the system is running
> some other arch by default.
>
> Cheers,

Don't forget the case of a package rename:

A:native -> B_old:all (metapackage) -> B_new:native (M-A:same)

Without the assumption of all == native dpkg could not ensure such a
renamed package installs the right version.


The multiarch specs [1] say the following:

Dependencies involving Architecture: all packages

Pre-multiarch, architecture-dependent packages may depend on
Architecture: all packages and assume that the transitive
dependencies will be resolved using packages of the same
architecture or other packages that are Architecture: all. To avoid
breaking this assumption, Architecture: all packages will, at least
initially, be treated as equivalent to packages of the native
architecture for all dependency resolution. This means that for an
Architecture: all package to satisfy the dependencies of a
foreign-architecture package, it must be marked Multi-Arch: foreign
or Multi-Arch: allowed.

Imho this should also affect the implicit foo:foreign <-> foo:all
relationships. A foo:all (M-A:foreign/allowed) package should be
considered an upgrade to foo:foreign while (M-A:none) should not to
preserve existing behaviour. That way maintainers could declare wether a
package has become architecture independent or is a metapackage for a
rename.

So my question is: Does the arch:all package in question have a
Multi-Arch field? And does dpkg take this into account?

MfG
Goswin

1) https://wiki.ubuntu.com/MultiarchSpec


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87hayhzplu.fsf@frosties.localnet

Re: Please test gzip -9n - related to dpkg with multiarch support

[Goswin von Brederlow]

Russ Allbery  writes:

> Goswin von Brederlow  writes:
>
>> Changing the name in the package would break tools that rely on the name
>> (like packages.debian.org extracting the Changelog). Also ugly.
>
> We control the tools; we can change the tools.  Multiarch is a big deal.
> We weren't going to get through this without changing some tools.  (One
> should not read that as my support of this specific alternative, as I've
> not decided there yet, but in general I think it's fair game to change our
> tools to support multiarch.)

One should not read that as my rejection of this specific alternative, as
I've not decided there yet, but in general I think it's fair game to
change our tools to support multiarch.


Problem I have is with the timeframe of such a change. While we can
change packages.d.o quite quickly given somebody willing to write the
patch we can not quickly change the tools in stable. And I really really
really do not want to have to wait yet another release cycle.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87lintzq9k.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Jonathan Nieder wrote:
>> David Kalnischkies wrote:
>
>>> Why would it be intuitive to add a specific value for the arch attribute 
>>> with
>>> apt-get install foo   # arch |= native
>>> but remove all values of the attribute with
>>> apt-get remove foo# arch &= ~all-architectures
>>> ?
> [...]
>> But I really think this is something anyone can get used to.  In the
>> examples you listed above:
>>
>>  apt-get install foo;# install foo with default arch-list (native)
>>  apt-get remove foo; # remove foo
>>
>> If foo is installed for no architectures, that does not mean it is
>> installed with an empty architecture list.  It means it is simply not
>> installed.
>
> Ok, now I think I figured out the inconsistency you are pointing to.
> If i386 is the native architecture, what would you expect the
> following sequence of commands to do?
>
>   apt-get install linux-image-3.2.0-1-amd64:amd64
>
>   ... wait a few weeks ...
>
>   apt-get install linux-image-3.2.0-1-amd64
>
> I would expect it to install the kernel with 'Architecture: amd64' and
> then to upgrade it.
>
> So the proposed semantics are not quite 'arch |= native'.  They are
> more like 'arch defaults to native for non-installed packages'.
>
> Jonathan

Assuming linux-image-3.2.0-1-amd64:i386 still exists I would expect apt
to install that if it has a equal or greater version than the installed
linux-image-3.2.0-1-amd64:amd64.

Current apt behaviour is a bit strange there though:

mrvn@frosties:~% apt-cache policy acl
acl:
  Installed: 2.2.51-4
  Candidate: 2.2.51-5
  Version table:
 2.2.51-5 0
500 http://ftp.de.debian.org/debian/ sid/main amd64 Packages
 *** 2.2.51-4 0
100 /var/lib/dpkg/status

mrvn@frosties:~% apt-cache policy acl:i386
acl:i386:
  Installed: (none)
  Candidate: 2.2.51-5
  Version table:
 2.2.51-5 0
500 http://ftp.de.debian.org/debian/ sid/main i386 Packages

I would expect something like:

mrvn@frosties:~% apt-cache policy acl
acl:
  Installed: 2.2.51-4 amd64
  Candidate: 2.2.51-5 amd64
  Version table:
 2.2.51-5 0
500 http://ftp.de.debian.org/debian/ sid/main amd64 Packages
499 http://ftp.de.debian.org/debian/ sid/main i386 Packages
 *** 2.2.51-4 0
100 /var/lib/dpkg/status amd64


But it seems my patch to reduce the pin of non-native architectures is
not in current apt and policy doesn't list all archs for M-A:foreign
packages.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4xn4171.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Russ Allbery  writes:

> Joey Hess  writes:
>
>> Anyway, my worry about the refcounting approach (or perhaps M-A: same in
>> general) is not the details of the implementation in dpkg, but the added
>> mental complexity of dpkg now being able to have multiple distinct
>> packages installed under the same name. I had a brief exposure to rpm,
>> which can install multiple versions of the same package, and that was
>> the main cause of much confusing behavior in rpm. While dpkg's invariant
>> that all co-installable package names be unique (and have unique files)
>> has certianly led to lots of ugly package names, it's kept the users'
>> and developers' mental models quite simple.
>
>> I worry that we have barely begun to scratch the surface of the added
>> complexity of losing this invariant.
>
> This does seem to be more M-A: same in general, to me, since whether we
> have file overlaps or not we still have multiple packages with the same
> name.  Which will force changes in everything that deals with packages,
> like Puppet, to be able to specify packages with particular architectures.
>
> I definitely agree on the complexity this adds.  But I don't think there's
> an alternative to that complexity without using something like --sysroot
> or mini-chroots, and I don't think those are satisfying solutions to the
> set of problems we're trying to solve.

pkg:arch will still be unique and the dpkg/apt output will use the
architecture where required for uniqueness. So I think that after some
getting used to it it will be clear enough again.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4xwrvw2.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Ian Jackson  writes:

> Guillem Jover writes ("Re: Multiarch file overlap summary and proposal (was: 
> Summary: dpkg shared / reference counted files and version match)"):
>> This still does not solve the other issues I listed, namely binNMUs
>> have to be performed in lock-step, more complicated transitions /
>> upgrades.
>
> I don't think I see where this is coming from.  Are you talking about
> variation in gzip output ?  Given the evidence we've seen here, in
> practice I think that is not going to be a problem.  Certainly it
> won't demand that binNMUs be performed in lock-step.

Note that splitting files (specifically changelog) into -common package
would require an explicit versioned dependency on the -common package and
produce the same (or similar) lock-step problem for upgrades and
binNMUs. Arch qualifying the files on the other hand would avoid that.

Splitting data files into -common packages will also often need a close
versioned dependency forcing a lock-step of packages. But probably not
so terse that binNMUs would have to be lock-steped.

Overall I think the lock-step being required for reference counted files
won't have such a large effect as you might think.



I think the idea of splitting the binNMU changelog into an extra file is
a great idea as that would allow putting the changelog into -common:all
and depend on the source version and then have the binNMU changelog in
the foo:any package in the symlinked directory. For this to work the
binNMU changelog should be arch and pkg qualified, e.g.

/usr/share/doc/foo-common/Changelog
/usr/share/doc/foo-common/Changelog.binNMU-foo-amd64
/usr/share/doc/foo-common/Changelog.binNMU-bar-i386
/usr/share/doc/foo -> foo-common
/usr/share/doc/bar -> foo-common 

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vcn8rw1z.fsf@frosties.localnet

Re: Multiarch file overlap summary and proposal

[Goswin von Brederlow]

Ian Jackson  writes:

> Russ Allbery writes ("Multiarch file overlap summary and proposal (was: 
> Summary: dpkg shared / reference counted files and version match)"):
>> 5. Data files that vary by architecture.  This includes big-endian
>>vs. little-endian issues.  These are simply incompatible with
>>multiarch as currently designed, and incompatible with the obvious
>>variations that I can think of, and will have to either be moved
>>into arch-qualified directories (with corresponding patches to the
>>paths from which the libraries load the data) or these packages
>>can't be made multiarch.
>
> Yes.  Of these, arch-qualifying the path seem to be to be obviously
> the right answer.  Of course eg if the data files just come in big-
> and little-endian, you can qualify the path with only the endianness
> and use refcounting to allow the equal-endianness packages to share.
>
> Ian.

Preferably -data-be:all and -data-le:all packages if they can be build
irespective of the buildds endianness.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkckrwnl.fsf@frosties.localnet

Re: Endianness of data files in MultiArch

[Goswin von Brederlow]

Guillem Jover  writes:

> On Thu, 2012-02-09 at 13:52:34 +0100, Goswin von Brederlow wrote:
>> Aron Xu  writes:
>> > This looks not very nice, because we need to maintain a list of
>> > architectures in debian/control, and when new architectures are added
>> > the package is potentially broken.
>> 
>> If endian dependend data is really a larger issue then introduce a
>> 
>>dpkg-architecture -qDEB_HOST_ENDIANESS
>
> This already exists: DEB_BUILD_ARCH_ENDIAN and DEB_HOST_ARCH_ENDIAN
>
> regards,
> guillem

Even better. Should have tested in a sid chroot.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nv0oxo4.fsf@frosties.localnet

Re: Please test gzip -9n - related to dpkg with multiarch support

[Goswin von Brederlow]

Ian Jackson  writes:

> Goswin von Brederlow writes ("Re: Please test gzip -9n - related to dpkg with 
> multiarch support"):
>> Ian Jackson  writes:
>> > One thing which no-one yet seems to have suggested is to have
>> > multiarch:same packages put the changelog in a filename which is
>> > distinct for each architecture.  (It wouldn't have to be the triplet;
>> > the shorter Debian arch would do.)  Perhaps there are obvious reasons
>> > (which I have missed) why this is a terrible idea, but it seems to me
>> > that it's something we should consider.
>> 
>> Or dpkg could do that for you. At least for files in /usr/share/doc when
>> there is a collision.
>
> Urgh, I think this is a really ugly idea, compared to just having the
> packages contain the arch-specific filenames.  After all a
> multiarch:same package knows that it is, and can DTRT.
>
> Ian.

Changing the name in the package would break tools that rely on the
name (like packages.debian.org extracting the Changelog). Also ugly.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ty30p0re.fsf@frosties.localnet

Re: Endianness of data files in MultiArch

[Goswin von Brederlow]

Aron Xu  writes:

> On Thu, Feb 9, 2012 at 01:35, Simon McVittie  wrote:
>> On 08/02/12 17:22, Aron Xu wrote:
>>> Some packages come with data files that endianness matters, and many
>>> of them are large enough to split into a separate arch:all package if
>>> endianness were not something to care about. AFAIK some maintainers
>>> are not aware of endianness issues in their packages and then just
>>> ignored it (not sure how many, but if any of them are discovered it
>>> should lead to RC bug).
>>
>> Hopefully Jakub Wilk's automatic checks for conflicting files
>>  will already be picking
>> this up, in cases where the less-used-endianness architectures aren't
>> broken already.
>>
>> If the less-used-endianness architectures are already broken, that's
>> also a bug (potentially an RC one), just like code that compiles but
>> doesn't work on a particular endianness due to other assumptions - and
>> if nobody has noticed it yet, presumably the package doesn't have any
>> users (or regression tests) on those architectures.
>>
>
> Or some of them just gave up because it is "less-used" architecture.
>
>>> It would be great to have some mechanism to
>>> handle such kind of problems in Debian, to avoid forcing those data to
>>> be placed into arch:any package.
>>
>> If the right endianness is critical: libfoo:i386 Depends:
>> libfoo-data-le, libfoo:powerpc Depends: libfoo-data-be, both data
>> packages arch:all, data files in /usr/share/foo/le and /usr/share/foo/be
>> respectively?

I would have them conflict and use the same directory. Otherwise you
need to use different paths in the binary, docs and maybe even
conffiles (which would then be architecture dependend too).

> This looks not very nice, because we need to maintain a list of
> architectures in debian/control, and when new architectures are added
> the package is potentially broken.

If endian dependend data is really a larger issue then introduce a

   dpkg-architecture -qDEB_HOST_ENDIANESS

> Also, arch:all packages are usually generated by the uploading DD on
> one architecture, mostly amd64 and i386 today, how can he managed to
> generate be data files if he doesn't have access to such a machine?
> Adding an option to the data generator/parser and make it able to
> generate be/le data on any architecture seems not to be a reasonable
> approach.

That is indeed the biggest problem currently. Also a problem for the
idea of building arch:all packages on buildds. They might not build on
all archs.

>> Or just make sure the data has an endianness marker, and enhance the
>> reading package to do the right byteswapping based on the endianness
>> marker - e.g. this has been discussed for gettext, which ended up just
>> writing out the same endianness on all platforms. Many formats
>> (particularly those that originated on Windows) are always
>> little-endian, and big-endian platforms reading them just take the minor
>> performance hit; formats that respect "network byte order" have the
>> opposite situation.
>>
>
> This is valid for most-used applications/formats like gettext, images
> that are designed to behave in this way, but on the contrary there are
> upstream that don't like to see such impact, especially due to the
> complexity and performance impact.
>
> Currently I am using arch:any for data files which aren't be affected
> with multiarch, i.e. not "same" or "foreign". For endianness-critical
> data that is required to make a library working, I have to force them
> to be installed into /usr/lib//$package/data/ and mark them
> as "Multiarch: same", this is sufficient to avoid breakage, but again
> it consumes a lot of space on mirror.
>
> I thought about something like /usr/share/$package/data/{be,le} in
> arch:all, but appears to be not a reasonable solution because we need
> to modify the data generator/parser.

It should be possible to build a converter or generator that can output
either endianess. So you could have a single arch:all package with both
/usr/share/$package/data/{be,le} in it or to generate the right
endianness on install. That way the "performance impact" argument is non
existant.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqdoqict.fsf@frosties.localnet

Re: Please test gzip -9n - related to dpkg with multiarch support

[Goswin von Brederlow]

Ian Jackson  writes:

> Russ Allbery writes ("Re: Please test gzip -9n - related to dpkg with 
> multiarch support"):
>> Another possible solution is to just give any package an implicit Replaces
>> (possibly constrained to /usr/share/doc) on any other package with the
>> same name and version and a different architecture.  This isn't as
>> defensive, in that it doesn't catch legitimate bugs where someone has made
>> a mistake and the packages contain different contents, but it also solves
>> the binNMU issue (well, "solves"; the changelog will randomly swap back
>> and forth between the packages, but I'm having a hard time being convinced
>> this is a huge problem).
>
> Well, it does mean that you might be lacking important information
> because the other changelog wouldn't be present on the system.
>
> One thing which no-one yet seems to have suggested is to have
> multiarch:same packages put the changelog in a filename which is
> distinct for each architecture.  (It wouldn't have to be the triplet;
> the shorter Debian arch would do.)  Perhaps there are obvious reasons
> (which I have missed) why this is a terrible idea, but it seems to me
> that it's something we should consider.
>
> Ian.

Or dpkg could do that for you. At least for files in /usr/share/doc when
there is a collision.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871uq4rxzq.fsf@frosties.localnet

Re: authbind (LD_PRELOAD) and multiarch

[Goswin von Brederlow]

Ian Jackson  writes:

> Goswin von Brederlow writes ("Re: authbind (LD_PRELOAD) and multiarch"):
>> Raphael Hertzog  writes:
>> > What does fakeroot do? My first idea would be to fail early and provide a
>> > useful error message.
>> 
>> fake(ch)root sets LD_PRELOAD=lib.so and
>> LD_LIBRARY_PATH="/usr/lib/fakeroot:/usr/lib32/fakeroot:/usr/lib64/fakeroot:$LD_LIBRARY_PATH"
>> or nowadays the same using the multiarch dirs. So the library remains a
>> private library but ld.so is tasked with searching for the one mathing
>> the elf format it is loading.
>
> This is addressing the question of where to put the library and still
> keep it private.  But TBH I'm not sure that there's actually a good
> reason for the .so not to go in /usr/lib.  That would obviate the need
> for the kind of helper you are suggesting here:
>
>> The tool would then contain the know-how to figure out which multiarch
>> directories to include in a single point instead of eveyone duplicating
>> that code.
>
> An alternative would be to have a single subdirectory of /usr/lib, say
> /usr/lib/preload-weird-shit/, which was always searched by
> the linker.
>
>
> I think the harder problem is how to get the right libraries
> installed.  If you want to run fakeroot you need the library for every
> configured architecture.
>
> Ian.

This came up before for plugins, e.g. support for inputing japanese or
chinese characters. If the input plugin is installed for one
architecture it should be installed for all architectures (the plugin
using library is installed for). I suggested to have "Depends: foo:*",
an architecture specific dependency with wildcard.

This is something that could use some native english speaker to write a
proposal/specs for with some examples where it is needed. Any takers?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqfg77nf.fsf@frosties.localnet

Re: authbind (LD_PRELOAD) and multiarch

[Goswin von Brederlow]

Raphael Hertzog  writes:

> On Mon, 12 Dec 2011, Ian Jackson wrote:
>>  * I will need to arrange for the same LD_PRELOAD setting to load the
>>correct libauthbind for each arch.  So I guess I do
>>LD_PRELOAD=libauthbind.so.1 rather than supplying an absolute path,
>>and trust ld.so to get the right one out of /usr/lib/.
>>Is that right ?
>
> I don't know, ld.so(8) does not document this. (But it doesn't look wrong
> at least)
>
>>  * AFAICT there is no way on a multiarch system to say in my
>>dependencies "I need this package on all architectures supported on
>>this system".  I went and looked at testing's fakeroot but I'm not
>>sure I should be using what it does as an example.  How should an
>>LD_PRELOAD hack approach this problem ?
>
> What does fakeroot do? My first idea would be to fail early and provide a
> useful error message.

fake(ch)root sets LD_PRELOAD=lib.so and
LD_LIBRARY_PATH="/usr/lib/fakeroot:/usr/lib32/fakeroot:/usr/lib64/fakeroot:$LD_LIBRARY_PATH"
or nowadays the same using the multiarch dirs. So the library remains a
private library but ld.so is tasked with searching for the one mathing
the elf format it is loading.

> Find out the arch of the executable, verify if the corresponding preload
> library is available. If it's not, then fail and instruct the user to
> install the missing library.
>
> But this means that you have done (part of) the work that you were
> intending to offload to ld.so.
>
>> To best avoid transitional problems I guess piece 2 should go into
>> "authbind" (Multi-arch: same; Depends: authbind-support) and pieces 1
>> and 3 would in "authbind-support" (Multi-arch: foreign; no
>> dependency).  But I'm not sure.
>
> Yes.
>
> Cheers,

The hard part comes with extending this past the simple biarch cases and
support for example arm binaries run via binfmt-misc on amd64. The list
of directories to add to LD_LIBRARY_PATH needs to be dynamic depending
on what architectures are configured for the system.

Since we have a few packages that do LD_PRELOAD tricks (eatmydata is
another one) maybe we should have a common tool that extends the
LD_LIBRARY_PATH, e.g.

LD_LIBRARY_PATH="$(extend-library-path --pkg fakeroot)"
LD_LIBRARY_PATH="$(extend-library-path --pkg fakechroot --sysroot /srv/chroot)"

The tool would then contain the know-how to figure out which multiarch
directories to include in a single point instead of eveyone duplicating
that code.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sjket1lw.fsf@frosties.localnet

Re: Multiarch support in dpkg really in time for wheezy?

[Goswin von Brederlow]

Raphael Hertzog  writes:

> Hi,
>
> I'll try to share some news with the release team.
>
> On Sat, 22 Oct 2011, Guillem Jover wrote:
>> What I'll do though, when I get back home tomorrow from my current
>> trip, is to push already reviewed stuff and keep pushing incrementally,
>> instead of my usual big pushes, so that the progress is more visible.
>
> This has been the case so far with regular progress every week. The last
> changes have not been pushed on the master branch but on guillem's private
> repository.
>
> http://git.hadrons.org/?p=debian/dpkg.git;a=shortlog;h=refs/heads/pu/multiarch/master
>
> The latest changes have been made today. About half of the multiarch
> branch has already been merged on master.
>
>> The 1.16.2 will still happen, as planned, quicker than our usual 2
>> months between micro releases.
>
> According to this sentence, the last possible date for the 1.16.2 release
> is today.
>
> Guillem, given that you're not yet over, can you try to set a new target?
>
> Cheers,

Hi Raphael,

could you maybe please just upload a multiarch dpkg to experimental
without waiting for Guillem?

If he wants to audit the changes before they go into unstable that is
fine but it makes no sense not to test the code by more people already
or to have them all search around for what git repository to use this
week for the latest changes.

Lets find and fix the bugs before Guillem has to audit them.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r50slgaj.fsf_-_@frosties.localnet

Re: Multiarch interfaces

[Goswin von Brederlow]

Steve Langasek  writes:

> On Mon, Nov 28, 2011 at 12:38:01AM +0100, Guillem Jover wrote:
>> > > Except for things like possibly firmware, which AFAIR there seemed to
>> > > be consensus should be kept as arch:all for now (otherwise they'd
>> > > require Packages file for lots of architectures), the rest require
>> > > run-time support from libc. The problem with adding the arch from
>> > > the libc package is that's a chicken and egg situation.
>
>> > What libc support do you mean?  All per-architecture executables should 
>> > have
>> > dependencies on the libc package for their arch anyway, so I don't see how
>> > libc support really enters into this.
>
>> Yes, I meant them needing the dynamic linker and libc.so. So if we are
>> on arch:amd64, want to install pkga:i386, and libc:i386 is the one
>> doing the dpkg --add-architecture i386, then apt will not be able to
>> nicely present the package to the user before the user has installed
>> it. Hope this clarifies.
>
> Oh, ok.  I had understood "the rest of the architectures" rather than "the
> rest of the packages" - thanks for the clarification.  Yes, it would be a
> bootstrapping problem to have libc registering the architecture with dpkg.

Wouldn't it make more sense to highjack the multiarch-support (the
native one) package and have it ask a debconf question wether to
activate i386 on amd64 and amd64 on i386 and so on?

And qemu-armel would add armel support, not the libc:armel.

I don't see how the libc would be the right place for this. That just
gives you a chicken&egg problem.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkfglhgm.fsf@frosties.localnet

Re: Multiarch interfaces

[Goswin von Brederlow]

Guillem Jover  writes:

> The other interface change is related to the --foreign-architecture
> option, the problem with it comes from the fact that the architectures
> supported by the database are not configuration, they are state. This
> shows up in several ways. When a front-end needs to load the list of
> architectures, it needs to get someone to parse dpkg.cfg files, this
> is currently done by dpkg itself, and the list can be retrieved with
> --print-foreign-architectures, the problem comes when wanting a
> front-end to load them through libdpkg. Making the latter have to
> execute «dpkg --print-foreign-architectures» would be suboptimal, and
> making libdpkg have to load dpkg.cfg would be distasteful. Another
> issue is that if the list of foreign architectures is on the
> configuration files it makes it slightly more tricky to cross-grade
> dpkg, and it makes it fairly easy to accidentally remove architectures
> required by the database. I've fixed all this by replacing the
> --foreign-architecture option with two new commands --add-architecture
> and --del-architecture which will perform sanity checks and store and
> load the architecture list (including the native arch) in an internal
> db file under /var/lib/dpkg. (I'll probably rename --del-architecture
> to something like --remove-architecture before pushing, to match the
> convention in the rest of the dpkg CLI).
>
> regards,
> guillem

Please add support for more than just an on/off switch. An architecture
can have different "class":

1) first class architecture

The hardware can natively run this architecture. Anything can be
installed for it.

2) second class architecture

The hardware can not natively run this but there is an emulator
(e.g. qemu-armel) installed. Some things can not be installed for this
architecture and a first class architecture should be used by frontends
whenever possible. Worst case would be installing an armel /bin/sh on
amd64.

This probably is nothing dpkg has to care about but it should have the
information so frontends can get to it and do the right thing
(e.g. prefer first class archs, ask really loudly before replacing
anythign essential with a second class arch).

3) cross-compile architecture

The hardware may or may not be able to run this at all, natively or
emulated. Only Multi-Arch: foreign packages should be installable for
this architecture. 

This is something dpkg could ensure but might also just be left to
frontends. But again there should be a central place to keep the
information.

MfG
Goswin



-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87aa7l5xm0.fsf@frosties.localnet

Re: Multiarch support in dpkg really in time for wheezy?

[Goswin von Brederlow]

Michael Gilbert  writes:

> On Sat, Oct 29, 2011 at 7:10 AM, Stefano Zacchiroli wrote:
>> What worries me is that there is multi-arch work in dpkg, work that has
>> its origins in Debian. That work is ready enough to be deployed in
>> popular Debian derivatives such as Ubuntu, but is not in Debian proper
>> yet. That is bad for Debian morale and should be avoided. Moreover, that
>> work is also considered ready enough by other dpkg co-maintainers, by
>> the Release Team, and by various porters, which have all asked multiple
>> times to have that work in the Debian archive.
>
> You could also make a case from a terminological perspective as well.
> Unstable is where development in Debian is supposed to happen, so it's
> perfectly acceptable to upload unfinished/unstable changes, and if you
> happen to break something (at least with dpkg) you'll have hundreds of
> eyes looking at what you broke and trying to figure out how to fix it.
>  So anyway, don't worry so much about breaking unstable.  That's what
> its there for.
>
> Best wishes,
> Mike

s/unstable/experimental/

Screwing up dpkg in unstable is too anoying to too many users and really
not neccessary.

But yes, multi-arch dpkg is way overdue.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ehwx5yi1.fsf_-_@frosties.localnet

Re: [Soc-coordination] Declarative Diversions - Report 1

[Goswin von Brederlow]

Steve Langasek  writes:

> On Sat, Jun 04, 2011 at 07:29:02AM +0200, Luk Claes wrote:
>> > -
>> > Details - Control File Syntax
>> > -
>> > It will conform to RFC2822 style with the following format:
>> >  * Divert-From:
>> >  * Divert-To:
>> >  * Blank lines and lines beginning with '#' will be comments
>> >  
>> > 'Divert-To' will be optional and if it is ommitted then files being
>> > diverted 
>> > will have their filename changed to 'file.distrib'
>
>> Would it not be better to have the filename changed to
>> 'file.' if 'Divert-To' is not specified, so it's possible
>> to support more packages diverting the same file?
>
> If you do that, how do you keep track of which package's file was diverted
> where, so that on *removal*, the files are put where they belong?

Because dpkg would record the name in its database in the "Divert-To"
slot. It already needs to know the name in the case Divert-To was
specified so this doesn't make it more complex.

> Why do you *want* to have parallel diversions of the same file by more than
> one package?  It may seem the answer is obvious, but if you think about it I
> believe you'll find those semantics aren't actually useful.  *Nested*
> diversions can be useful (one package diverts foo to foo.distrib and wraps
> it; another package diverts foo.distrib to foo.distrib.distrib and wraps it
> again), but having two diversions happen in parallel, where the unpack order
> determines which package ends up on top, isn't useful at all.

One use for this would be to later merge in alternatives.

Think about it. What are alternatives? Some packages all provide the
same file and based on priority or users choice one of them wins out and
gets to use the name. One minor change that would probably be usefull
for alternatives would be to allways store the "file" as
"file." even for the package that gets picked to provide
"file".

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d3ikx32l.fsf@frosties.localnet

Re: Declarative Diversions - GSoC Project Update 1

[Goswin von Brederlow]

Steve Langasek  writes:

> On Tue, May 31, 2011 at 09:49:26AM +0200, Goswin von Brederlow wrote:
>> Raphael Hertzog  writes:
>
>> > On Mon, 30 May 2011, Goswin von Brederlow wrote:
>> >> Sam Dunne  writes:
>
>> >> > This project will infer --add, --remove and --package and will not
>> >> > allow you to specify them.
>
>> >> > The same is goes for --rename, it is being made default and not 
>> >> > optional (for
>> >> > now, I am aware of some special cases).
>
>> >> > --divert will not be specifiable and will take the default format
>> >> > "file.divert-$package".
>
>> >> What about moving a diversion from package A to B?
>
>> > That's why we have "Conflicts" to ensure a package is removed before the
>> > other is installed.
>
>> But we only need breaks+replaces there.
>
> In theory.  Do you have an idea for a non-brittle mechanism that would let
> this work for diversions in practice?

That would be for him to figure out an implement.

We have the following case:

old A: has diversion
old B: does not have diversion

new A: does not have diversion
new B: does have diversion
new B: Replaces: A (<< new), Breaks: A (<< new)

A and B can be unpacked in any order as long as A gets deconfigured
first. The problematic case is B getting unpacked first.

I would make dpkg be able to track multiple diversions but only one
of the diverting packages may be configured at any one time. Dpkg would
have to track which packages diversion was last active (what the current
diverted name of the file is) and update that when configuring a
package.

I never liked the breaks+replaces because it breaks downgrades (or
even just reverting all failed upgrade) by loosing files. So maybe
moving a file from A to B should go back to using Conflicts? At least
for the case of diversion.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d3ixg12l.fsf@frosties.localnet

Re: Declarative Diversions - GSoC Project Update 1

[Goswin von Brederlow]

Tollef Fog Heen  writes:

> ]] Raphael Hertzog 
>
> | On Tue, 31 May 2011, sean finney wrote:
> | > Does this mean it will not be possible to divert a file when the source
> | > or destination has whitespace in it?  If so I'd suggest you also have
> | > support for some backslash-type escaping.
> | 
> | Or to use two lines instead of one. And to support multiple entries make
> | the different entries separated by an empty line (so that it's possible to
> | add supplementary fields in the future shall that be required).
>
> I'd like us to go with the standard that's used in most other similar
> files: RFC2822 style, so something like:
>
> Divert-From: /usr/share/foo
> Divert-To: /usr/share/bar
>
> Divert-From: /usr/share/baz
> Divert-To: /usr/share/coo
>
> This makes it easier to figure out what the contents of the file means
> when you're tired or don't care to check the documentation.
>
> Regards,

Do we even need a "Divert-To"? In most cases the new name doesn't matter
or it is enough to know that the new name will be old name +.dpkg-divert
or +. choosen automatically.

With RFC2822 style it would be possible to make the Divert-To optional
without being confusing to read or parse.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r57f6klc.fsf@frosties.localnet

Re: Declarative Diversions - GSoC Project Update 1

[Goswin von Brederlow]

Raphael Hertzog  writes:

> On Mon, 30 May 2011, Goswin von Brederlow wrote:
>> Sam Dunne  writes:
>> 
>> > This project will infer --add, --remove and --package and will not
>> > allow you to specify them.
>> >
>> > The same is goes for --rename, it is being made default and not optional 
>> > (for
>> > now, I am aware of some special cases).
>> >
>> > --divert will not be specifiable and will take the default format
>> > "file.divert-$package".
>> 
>> What about moving a diversion from package A to B?
>
> That's why we have "Conflicts" to ensure a package is removed before the
> other is installed.

But we only need breaks+replaces there.

> I mean moving a diversion (which implies moving a file) without this
> precaution would fail because dpkg-trigger is run in the preinst. Thus
> it's not a new problem.
>
> Cheers,

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87oc2j8g3d.fsf@frosties.localnet

Re: Declarative Diversions - GSoC Project Update 1

[Goswin von Brederlow]

Sam Dunne  writes:

> This project will infer --add, --remove and --package and will not
> allow you to specify them.
>
> The same is goes for --rename, it is being made default and not optional (for
> now, I am aware of some special cases).
>
> --divert will not be specifiable and will take the default format
> "file.divert-$package".

What about moving a diversion from package A to B?

This should not run into the error that 2 packages are trying to divert
the same file.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ipsstaht.fsf@frosties.localnet

Re: RFC: a new (meta data) field in .DEB packages

[Goswin von Brederlow]

Gerold Jens Wucherpfennig  writes:

> Hi all,
>
> I propose a new field in the control file of .DEB packages
> and want to know what you think about it.
>
> In the "control.tar.gz" file of every .DEB archive is the file
> "control". There are fields for the package name, the dependencies
> etc.
>
> My REQUEST is to add a field called "Development-Package".
>
> Every library package should have the field
> "Development-Package" set to the corresponding package name.
>
> For example package "libc6" will have set:
>
> "Development-Package: libc6-dev"
>
> So the package manager can examine all installed packages and
> will collect the data of the field: "Development-Package"
>
> Then the package manager may ask the user if he wants to
> install all these packages in one turn ...
>
> ... quite useful for all software
> developers and from-source-installing geeks :-)
>
>
> I'm looking forward to reading your reply
>
> Regards
> Gerold

I think the field will be quite useless. You don't want to blindly
install all -dev packages for all your installed libraries. That would
be a huge waste of space and bandwidth. You only need the -dev packages
used by the source you want to compile and for that we have
Build-Depends already.

Further -dev packages aren't the only thing you need to compile
sources. You often also need extra binary packages, which your
"Development-Package" field would not find. So again you need to install
Build-Depends. And if you need to install Build-Depends anyway then what
is the point of installing Development-Package for all libs first?


If you want to do a from-source-install then I recommend just running
apt-get build-dep . Problem solved.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762p03h70.fsf@frosties.localnet

Re: [RFC] Trigger directives that do not put packages in triggers-awaited

[Goswin von Brederlow]

Raphael Hertzog  writes:

> Hi,
>
> please find attached a patch that implements 2 new directives
> "interest-noawait" and "activate-noawait" for the triggers control file.
>
> They work exactly like "interest" and "activate" but the package
> that activate the triggers are not put in triggers-awaited status instead
> they are directly configured. Up to now, this was only possible if you
> manually activated triggers with dpkg-trigger. With this patch, the
> package implementing the trigger can also specify that the trigger
> is not important enough to consider that the triggering package is
> not configured.
>
> My goal is to simplify the configuration process because most
> triggers are purely convenience triggers and are not required by
> the triggering packages... (see man-db, bash-completion, etc.)
> This will also make it more likely that the trigger is really only
> executed once at the end instead of multiple times because some
> package had to be configured earlier.
>
> My main problem is how to introduce this new feature. For now, I
> have added "-noawait" variants but given they are the most common case, I
> wonder if I shall not change the current directive to not
> put the packages in triggers-awaited and instead to add new directives
> "interest-await" and "activate-await" (or "interest-crucial",
> "activate-crucial") with the current behaviour. That would also reduce the
> number of packages to modify to get the expected benefits... very few
> packages really need to put the triggering package in triggers-awaited.
>
> What do you think?

Not all trigger are convenience triggers and switching the default to
not wait would break those. So I think that is not an option in the
short run.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/878vu2zybn.fsf@frosties.localnet

Re: dpkg and apt status files.

[Goswin von Brederlow]

Raphael Hertzog  writes:

> On Mon, 09 May 2011, Ciaran Carthy wrote:
>> quickly rebuilds it to its full size. Does it rebuild it by trawling
>> through the repository again?
>
> AFAIK yes.

Actually apt-get, apt-cache, ... use them if they are current, rebuild
them if they have write access and otherwise just rebuild them in
memory. It is purely an optimization to avoid having to parse the
Packages/Sources files on every invocation.

>> What is it used for/what does it contain? It would seem to be
>> redundant based on what the next two files are used for.
>
> It's the list of available packages known to APT.
>
>> In the /var/lib/dpkg directory there are two important files:
>> *available *and *status*.
> [...]
>> This seems to be the definitive source for determining that a
>> package is installed.
>
> Yes.
>
>> What is "available" for and how is it built up?
>> My understanding is that dpkg does not have the concept of a
>> configured remote repository.
>> So in the world of dpkg, what does it mean to say that a package is
>> "available"?
>
> The available file is mainly useful for dselect. It's maintained by dpkg

Is anything but dselect using this (not just filling it like dpkg but
actually using it)?

> but fed by external programs/scripts via "dpkg --update-avail" (and/or
> --merge-avail). For historical reasons, it also keeps the information of
> packages that have been installed in the past (dpkg -i adds the info both
> to the available file and to the status file).
>
> This file is not used by APT.

If you do not use dselect you can save some disk space by running dpkg
--clear-avail. One can even add this as hook to apt-conf.d so it doesn't
kepp growing back.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sjsjsonh.fsf@frosties.localnet

Re: Synchronizing local debian repository

[Goswin von Brederlow]

jarek  writes:

> Hello!
>
>   I don't know it this list i suitable from my question, so if there is a
> better list, let me know.

Wrong list. Not sure where it should go best (debian-user?) so i will
just answere here.

>   For a long time, I'm looking for solution to synchronize local debian
> repository which is not connected to internet.
>   At the moment I'm doing this the following way:
>   1. With apt-mirror synchronize local repository on computer connected
> to internet.
>   2. From apt-mirror log get list of changed packages.
>   3. Transfer changed packages to DVD-R.
>   4. Merge changed files with second repository not connect to internet.
> It works, but is complicated and prone to various errors, i.e. if
> apt-mirror will fail in middle of work.
>   Now I have idea, to create a script, which will fetch packages with
> MTIME greater then last fetch (with lets say 1 day gap).
>   Is this a good idea ? I've compared dates from few mirrors and it looks
> promising, but I didn't found any official info about this.
>   If there is other, known way to do this, I'll be happy to hear about
> this. 
>   From my point of view it would be great if there will be some number
> telling in which order packages were added to repository, i.e:
> Package: libc-bin
> Priority: required
> Section: libs
> Installed-Size: 1636
> Maintainer: GNU Libc Maintainers 
> Architecture: amd64
> Source: eglibc
> Version: 2.11.2-10
> Repository-revision: 23423
> 
>
>  It should work similar to revision number in svn. The other solution is
> to have build date in each package, i.e.:
>
> Package: libc-bin
> Priority: required
> Section: libs
> Installed-Size: 1636
> Maintainer: GNU Libc Maintainers 
> Architecture: amd64
> Source: eglibc
> Version: 2.11.2-10
> Created: 23.01.2011 21:17:00
> 
>
> best regards
> Jarek

Each package has a version and name+version(+arch) is unique. The
version is also monotone rising but that is irelevant. The packages
filename is also unqiue. So you really don't need a revision or
timestamp.

I think what you are looking for is apt-zip. Or at least apt-zip should
be a good start. You should not rely on the MTIME of files. Use the
Packages file to compare what you non-networks system has with what it
should have and fetch the difference.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkmrsp7x.fsf@frosties.localnet

Re: Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

[Goswin von Brederlow]

Bill Allombert  writes:

> On Wed, Apr 20, 2011 at 09:11:59PM +0200, Raphael Hertzog wrote:
>> (BTW, popcon is mainly run from cron so interactive performance is not so
>> critical.)
>  
> The issue is not interactive performances but waste of system resource. Users
> will complain.
>
> Cheers,

Please do get real. This runs once a week at an ungodly hour. 2s or 30s
is not going to make a collosal difference. 30s is ~0.005% of the weekly
cpu time. Other cron jobs take far longer. And 30s is better than not
working.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zknhdzls.fsf@frosties.localnet

Re: file trigger and dpkg aborting due to dependency errors

[Goswin von Brederlow]

Raphael Hertzog  writes:

> On Wed, 30 Mar 2011, Goswin von Brederlow wrote:
>> > Note also that the trigger activation is a no-op if the 
>> > qlustar-image-common is
>> > not configured which means that you have to regenerate your image in the
>> > "configure" of qlustar-image-common as well as in the "triggered" case.
>> 
>> Is a package allowed to trigger itself? Or is that already considered a
>> cycle?
>
> I don't see how a package can trigger itself. If you run dpkg-trigger from
> the postinst, the package is not configured yet and the trigger is a
> no-op. It's the same for a file trigger which is executed during unpack.
>
> Cheers,

Looking closer at /usr/share/doc/dpkg-dev/triggers.txt.gz:

Packages in `config-failed' or worse are never considered to have
lists of pending triggers.  A package whose postinst is being run
can however acquire pending triggers during that run (ie, a package
can trigger itself).

The docs seem to support triggering oneself is valid and it works not
just for pakages in `config-failed' state.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zko7ilyt.fsf@frosties.localnet

Re: file trigger and dpkg aborting due to dependency errors

[Goswin von Brederlow]

Raphael Hertzog  writes:

> Hi,
>
> On Wed, 30 Mar 2011, Goswin von Brederlow wrote:
>> Unpacking replacement qlustar-module-ha-lucid-amd64-7.4.1 ...
>> dpkg: dependency problems prevent configuration of
>> qlustar-module-ha-lucid-amd64-7.4.1:
>>  qlustar-module-ha-lucid-amd64-7.4.1 depends on 
>> qlustar-module-core-lucid-amd64-7.4.1 (= 7.4.1~exp+2011-03-30+15.27); 
>> however:
>>   Version of qlustar-module-core-lucid-amd64-7.4.1 on system is 
>> 7.4.1~exp+2011-03-30+14.37.
>> dpkg: error processing qlustar-module-ha-lucid-amd64-7.4.1 (--install):
>>  dependency problems - leaving unconfigured
>> Processing triggers for qlustar-image-common ...
>> qlustar-image-common: triggered /usr/lib/qlustar/modules
>> 
>> 
>> As you can see the the ha module can not be configure due to
>> dependencies. But the file trigger is run none the less. This causes
>> images to be build with a version skew between the core and ha
>> module. Can this be avoided?
>
> I don't see how it could be avoided. The file trigger is triggered by the 
> unpack
> which succeeds.
>
> It's execute because qlustar-image-common is in configured state at the
> start and it's required to come back to the configured state (and not stay
> in triggers-pending).

Which is a bit surprising but on closer examination it makes sense.

> What you can do is code your trigger to do nothing if it detects a version
> skew...
>
>> % sudo dpkg --configure qlustar-module-ha-lucid-amd64-7.4.1
>> Setting up qlustar-module-ha-lucid-amd64-7.4.1 (7.4.1~exp+2011-03-30+15.27) 
>> ...
>> 
>> The file trigger is not run again. For me this is not a problem right
>> now. But what if the postinst of the ha module needs to be run before
>> the file trigger is to be run?
>
> At some point you must decide that the file trigger is definitely not
> the right interface for what you're trying to do instead of trying to
> paper over poor design choices.

It does look that way. So file triggers are definetly out of the picture
when the trigger has to run after configuring the triggering package.

>> Or actualy it is a problem right now because I need the depends of my
>> package to be satisfied before the file trigger is run.
>
> You can only be sure that the depends of the package that runs the triggers
> are satisfied, those of the triggering packages might not be since the
> file trigger is activated at unpack time.

Ok, so the observed behaviour is the intended behaviour for file
trigger. Knowing that I can work with it.

>> What are my options here? Do I need to Pre-Depend on the core module or
>> do I need to use an explicit trigger in the modules postinst scripts?
>
> Obviously a Pre-Depend can ensure you that you have the right dependencies
> when you unpack and thus activate the file trigger but that's ugly IMO.

It also wouldn't help because core would be unpacked and configured and
trigger the rebuild and only then in a second dpkg call ha would be
unpacked and trigger again. There would still be a window where an image
with version skew is build.

> An explicit trigger in the postinst(s) is probably better but it won't
> solve your problem of version skew... since the core will be configured
> first and the image might be regenerated before the "ha" part gets
> configured.

You are right, there is still the chance of a version skew and explicit
trigger (alone) won't solve the problem.


In all cases the trigger seems to need to handle versions skews gracefully.
And given that any option will work and a file trigger looks like the
best solution. That way module packages don't need a (identical)
postinst and prerm script.

> Note also that the trigger activation is a no-op if the qlustar-image-common 
> is
> not configured which means that you have to regenerate your image in the
> "configure" of qlustar-image-common as well as in the "triggered" case.

Is a package allowed to trigger itself? Or is that already considered a
cycle?

It seems to work to manually trigger /usr/lib/qlustar/modules during
"configure". That way images are build at least once and only once.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87hbaks32f.fsf@frosties.localnet

file trigger and dpkg aborting due to dependency errors

[Goswin von Brederlow]

Hi,

I have a little problem with file triggers.

I have a tool (qlustar-image-common) that builds a root filesystem image
from a set of modules. The tool has a file trigger on
/usr/lib/qlustar/modules. Now I have 2 modules: core and ha. The core
module contains a minimal bootable system and the ha modules adds high
availability stuff to it. The ha module depends on the core module.

Now here is what happens:

% sudo dpkg -i 
../qlustar-module-ha-lucid-amd64-7.4.1_7.4.1~exp+2011-03-30+15.27_all.deb

(Reading database ... 94836 files and directories currently installed.)
Preparing to replace qlustar-module-ha-lucid-amd64-7.4.1
7.4.1~exp+2011-03-30+15.14 (using
.../qlustar-module-ha-lucid-amd64-7.4.1_7.4.1~exp+2011-03-30+15.27_all.deb)
...
Unpacking replacement qlustar-module-ha-lucid-amd64-7.4.1 ...
dpkg: dependency problems prevent configuration of
qlustar-module-ha-lucid-amd64-7.4.1:
 qlustar-module-ha-lucid-amd64-7.4.1 depends on 
qlustar-module-core-lucid-amd64-7.4.1 (= 7.4.1~exp+2011-03-30+15.27); however:
  Version of qlustar-module-core-lucid-amd64-7.4.1 on system is 
7.4.1~exp+2011-03-30+14.37.
dpkg: error processing qlustar-module-ha-lucid-amd64-7.4.1 (--install):
 dependency problems - leaving unconfigured
Processing triggers for qlustar-image-common ...
qlustar-image-common: triggered /usr/lib/qlustar/modules



As you can see the the ha module can not be configure due to
dependencies. But the file trigger is run none the less. This causes
images to be build with a version skew between the core and ha
module. Can this be avoided?


% sudo dpkg -i 
../qlustar-module-core-lucid-amd64-7.4.1_7.4.1~exp+2011-03-30+15.27_all.deb 

(Reading database ... 94836 files and directories currently installed.)
Preparing to replace qlustar-module-core-lucid-amd64-7.4.1 
7.4.1~exp+2011-03-30+14.37 (using 
.../qlustar-module-core-lucid-amd64-7.4.1_7.4.1~exp+2011-03-30+15.27_all.deb) 
...
/var/lib/dpkg/info/qlustar-module-core-lucid-amd64-7.4.1.prerm upgrade 
7.4.1~exp+2011-03-30+15.27
Unpacking replacement qlustar-module-core-lucid-amd64-7.4.1 ...
Setting up qlustar-module-core-lucid-amd64-7.4.1 (7.4.1~exp+2011-03-30+15.27) 
...
Processing triggers for qlustar-image-common ...
qlustar-image-common: triggered /usr/lib/qlustar/modules
...

Installing the core module works fine and the file trigger is run as expected.
This luckily fixes the images as they are generated again.

And last:

% sudo dpkg --configure qlustar-module-ha-lucid-amd64-7.4.1
Setting up qlustar-module-ha-lucid-amd64-7.4.1 (7.4.1~exp+2011-03-30+15.27) ...

The file trigger is not run again. For me this is not a problem right
now. But what if the postinst of the ha module needs to be run before
the file trigger is to be run?

Or actualy it is a problem right now because I need the depends of my
package to be satisfied before the file trigger is run.

What are my options here? Do I need to Pre-Depend on the core module or
do I need to use an explicit trigger in the modules postinst scripts?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqp8itnj.fsf@frosties.localnet

Re: dpkg fails on low-memory-hardware

[Goswin von Brederlow]

Martin Klaiber  writes:

> Raphael Hertzog  wrote:
>
>> How do you know that they are not using swap ?
>
> When I tried to upgrade (btw the currently installed Debian is Sarge
> not Etch, sorry), I could see in top how memory consumption of apt-get
> or aptitude and dpkg increased but swap wasn't used.
>
>> I don't know of any reason why dpkg/apt-get would not use swap.
>
> ACK. Today I did another test. I reduced the amount of RAM to 8 MB and
> installed some programs with aptitude and apt-get. The laptop became
> terribly slow and was heavily swapping but everything was running fine.
>
> So, I will try to upgrade to Lenny in the next days and will tell you,
> if it worked or not. Perhaps it was only a temporary problem with swap.
>
> Thanks for your help,
> Martin

I've used apt/dpkg with 64MB ram just fine. But something as trivial
looking as installing locales already exceeds the 64MB with just
dpkg. You will need lots of swap for apt/aptitude.

I recommend creating a local partial mirror with reprepro or debmirror
with just the packages you need. As others have pointed out apt/aptitude
consume ram proportional to the amount of available packages so reducing
that saves ram. Also makes it run faster and if it prevents (some)
swapping it will be magnitudes faster.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87wrkv1yiq.fsf@frosties.localnet

Re: Cross-upgrading packages with multiarch packages

[Goswin von Brederlow]

Steve Langasek  writes:

> On Fri, Feb 18, 2011 at 12:31:18PM +0100, Raphael Hertzog wrote:
>
>> I'd like to seek feedback about what kind of upgrades dpkg should support
>> with multi-arch packages.
>
>> dpkg treats foo:native and foo:all like the same package and it's thus
>> possible to upgrade foo_1.0_all to foo_2.0_ and vice-versa.
>
>> However if you have installed foo_1.0_, you can't upgrade that
>> package to foo_2.0_all (and vice-versa). Same goes for foo_1.0_ to
>> foo_1.0_ (provided they are not multi-arch: same, otherwise they
>> could coexist). You have to remove the conflicting package first and
>> reinstall afterwards.

I hope you mean that one has to remove the foo_1.0_ /
foo_1.0_ in the same dpkg call as adding foo_2.0_all /
foo_1.0_. If this needs 2 dpkg calls, one for removing, one
for reinstalling then I'm flat out against it. That would harm all the
reverse depends and cause significant blockage on upgrades.

Maybe this could be done like breaks instead of like conflicts. The
frontend has to deconfigure the old packages before installing the new
ones and eventually it has to remove the old packages. That might make
upgrades easier and prevent deadlocks where packages need to be
temporarily removed.

>> In a similar vein, if you have several instances of the same package
>> (Multi-Arch: same, say foo_1.0_ and foo_1.0_), you can't
>> install foo_2.0_all because it's in conflict with foo_1.0_.
>
>> Is this behaviour what we want?

I wouldn't say it is what we wan't. But it is something we can live
with. Frontends can handle the finer details.

> I think that actually sounds entirely reasonable.  If you have
> foo_1.0_native, foo_1.0_foreign installed, and you try to install
> foo_2.0_all, which package is it an upgrade of?  I don't think it makes
> sense from dpkg's POV to be a simultaneous upgrade of *both* packages.  So
> it's an upgrade of foo_1.0_native, and should be handled the same way as you
> would handle an attempt to upgrade to foo_2.0_native without also upgrading
> to foo_2.0_foreign.

A foo_2.0_all Multi-Arch:foreign package fullfills the depends for any
architectures. So logically it does represent an upgrade of *both*
foo_1.0_native and foo_1.0_foreign. I think dpkg should handle it that
way, extending the existing behaviour with nativ -> all upgrades to
mutiarch.

On the other hand foo_2.0_all Multi-Arch:none only fullfills the depends
for the native architecture and therefor should only upgrade
foo_1.0_native and complain about foo_1.0_foreign.

Similary with Multi-Arch: foreign the foo_1.0_ and
foo_1.0_ are equivalent. It would make sense for dpkg to allow
a plain "dpkg -i foo_1.0_.deb" to work and replace
foo_1.0_.

But it isn't essential to getting multiarch working. If it is simpler to
not do this now then lets leave it to the frontends for now.


Futher there is one case you haven't considered. Say you have only
foo_1.0_foreign installed. Should foo_2.0_all Multi-Arch:foreign be
considered an upgrade? I think so.

MfG

Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871v333dby.fsf@frosties.localnet

Re: Multi-arch and dependencies on arch: all packages

[Goswin von Brederlow]

Steve Langasek  writes:

> On Wed, Feb 16, 2011 at 05:12:03PM +0100, Goswin von Brederlow wrote:
>> It actually makes less sense.
>
>> You have identified the 2 cases of arch:all apckages:
>
>> 1) truely architecture independent and satisfies dependencies for any
>> arch
>> 2) meta packages that are placeholder for an arch:any package
>
>> So why then forbid the use of the Multi-Arch field to let maintainers
>> choose between the 2 types?
>
> What?  We didn't do that at all.  This thread is about making it clear that
> the Multi-Arch field *can* be set on Architecture: all packages!  Case 1 is
> Multi-Arch: foreign; case 2 is Multi-Arch: none.

Yet Multi-Arch: same is forbidden in the specs. Which you would need for
a meta package that depends on a library.

>> The other solution is to let arch:all packages fullfill depends for any
>> arch but consider them as if they were arch:any when computing the
>> closure of dependencies. That means things the arch:all package depends
>> on must be the same arch as the package depending on the arch:all
>> package or Multi-Arch: foreign.
>
>> Dependency chains could then look like this:
>> foo:i386 -> bla-plugins-meta:all -> bla-plugin:i386
>> foo:amd64 -> bla-plugins-meta:all -> bla-plugin:amd64
>
> Which requires the package manager to keep track of state about *transitive*
> dependencies.  Which is what I think we should avoid, especially for dpkg.
>
> Raphaël seems to agree with me, and has implemented my request in his
> branch.
>
>> "Pre-multiarch, architecture-dependent packages may depend on
>> Architecture: all packages and assume that the transitive dependencies
>> will be resolved using packages of the same architecture or other
>> packages that are Architecture: all."
>
>> Under multiarch the "same architecture" would mean the same architecture
>> as the package depending on the arch:all package.
>
>> Now you seem to say that the second chain would be illegal. Which
>> matches the next sentence in the specs:
>
>> "To avoid breaking this assumption, Architecture: all packages will, at
>> least initially, be treated as equivalent to packages of the native
>> architecture for all dependency resolution."
>
>> But that seems unneccessary. You only need to use the same, not the
>> native architecture.
>
> There is no way to know that it's the same architecture without having to
> fully traverse the dependency tree.  That's the complexity that should be
> avoided in dpkg.  By smashing these to :native instead of :same, we have
> enough information to decide *locally* whether a given package's
> dependencies are satisfied.  Robust, efficient, and avoids getting hung up
> on hypothetical corner cases.

You don't have to traverse the whole tree, only until you hit a package
that isn't arch:all. Also you can work around that by simply duplicating
the package internally with different Architecture strings as needed.
I totaly agree that this is a step up in complexity. It would be nice to
have though.

By smashing arch:all packages to :native you greatly reduce the
installability of packages. Lets take a simple game (freeciv) as example
and pretend this were a non-free package available for i386 only:

Package: freeciv-client-gtk
Architecture: i386
Depends: libatk1.0-0 (>= 1.29.3), ..., freeciv-data (>= 2.2.0), ggzcore-bin

Package: freeciv-data
Architecture: all
Version: 2.2.4-1

That means freeciv will be uninstallable on amd64 for no good reason.


Could we default arch:all leaf packages (no further Depends) to
Multi-Arch: foreign? That would cover data packages like the above
example. That should be local enough to check.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762sf3iw2.fsf@frosties.localnet

Re: Multi-arch and dependencies on arch: all packages

[Goswin von Brederlow]

Steve Langasek  writes:

> On Thu, Feb 10, 2011 at 05:00:13PM +0100, Raphael Hertzog wrote:
>> Steve reported me this problem concerning the current implementation of
>> the multiarch spec (he uses my latest pu/multiarch/snapshot/* branch).
>
>> Le mercredi 09 févr. 2011, Steve Langasek a écrit :
>> >  - I've just marked tzdata (an Architecture: all package) as Multi-Arch:
>> >foreign in the Ubuntu archive because I noticed my test libc6 package
>> >(libc6 depends on tzdata in Ubuntu but not in Debian) failed to install,
>> >listing this dependency as one of the reasons.  At first I was going to
>> >report this as a bug, but then I remembered this:
>> >
>> > https://wiki.ubuntu.com/MultiarchSpec#Dependencies%20involving%20Architecture:%20all%20packages
>> > 
>> >Now dpkg fails to install this package at all, with this error:
>> > 
>> >  parsing file '/var/lib/dpkg/tmp.ci/control' near line 18 package 'tzdata':
>> >  package has multiarch field but is architecture all
>> > 
>> >And doing so complies with this requirement from the spec:
>> > 
>> >"Setting the Multi-Arch field on a package which is Architecture: all is
>> >considered an error. dpkg-deb must refuse to generate a .deb with this
>> >combination of values. Behavior when trying to install such a package is
>> >undefined."
>> > 
>> >
>> > (https://wiki.ubuntu.com/MultiarchSpec#Binary%20package%20control%20fields)
>> > 
>> >These two requirements are clearly in conflict.  I've updated the wiki
>> >page to make it clear that only Multi-Arch: same is disallowed for
>> >Architecture: all packages.  Please update the dpkg implementation to
>> >match when you have a chance.
>
>> I think it's wrong to (have to) add the Multi-Arch field to architecture
>> all packages. I would rather suggest that we consider them as
>> automatically satisfying any dependency (i.e. the Multi-Arch: foreign
>> would be implicit).
>
>> What do other people think?
>
>> An architecture all package that provides something arch specific is
>> very rare and it's often meant to be used in situation where you precisely
>> want to make this available to other architectures (i.e. syslinux-common).
>
>> So I don't think that we have to go through all the trouble to whitelist
>> them one by one.
>
> Sorry, I should have included in my mail the rationale for this requirement
> - which I remembered from the drafting sessions but didn't add when updating
> the wiki page.
>
> The reason we determined while drafting that Architecture: all packages
> should not automatically satisfy dependencies as if they were Multi-Arch:
> foreign is that in the current scheme, an Architecture: all package can very
> well be used as a metapackage to bridge a dependency between two
> Architecture: any packages that need to be of the same architecture.
>
> As a practical example: the python package is Architecture: all.  Every
> python C extension package is Architecture: any and depends on python as an
> /interface/ to the python2.x Architecture: any packages which *must* be of
> the same architecture.
>
> I believe the consensus when drafting was that, if we didn't treat
> Architecture: all packages as being equivalent to the native arch for
> dependency solving, there would be cases when we would have insufficient
> information to properly verify dependencies due to the masking effect.
>
> With that rationale, does this requirement make more sense to you?  Or do
> you see some other way to make this work?  (Bearing in mind that both dpkg
> and higher-level package managers like apt need to be able to sort through
> the deps correctly)

It actually makes less sense.

You have identified the 2 cases of arch:all apckages:

1) truely architecture independent and satisfies dependencies for any
arch
2) meta packages that are placeholder for an arch:any package

So why then forbid the use of the Multi-Arch field to let maintainers
choose between the 2 types?


The other solution is to let arch:all packages fullfill depends for any
arch but consider them as if they were arch:any when computing the
closure of dependencies. That means things the arch:all package depends
on must be the same arch as the package depending on the arch:all
package or Multi-Arch: foreign.

Dependency chains could then look like this:
foo:i386 -> bla-plugins-meta:all -> bla-plugin:i386
foo:amd64 -> bla-plugins-meta:all -> bla-plugin:amd64

This is how I read the specs:

"Pre-multiarch, architecture-dependent packages may depend on
Architecture: all packages and assume that the transitive dependencies
will be resolved using packages of the same architecture or other
packages that are Architecture: all."

Under multiarch the "same architecture" would mean the same architecture
as the package depending on the arch:all package.

Now you seem to say that the second chain would be illegal. Which
matches the next sentence in the specs:

"To avoid breaking this assumption, Architecture

Re: Need feedback on dpkg's behaviour with multiarch packages

[Goswin von Brederlow]

Raphael Hertzog  writes:

> Hi,
>
> with Guillem we're wondering how we should "interpret" an unqualified
> "foo" when foo is a multi-arch package. For the sake of the example,
> we'll use "libsame" a package which is Multi-Arch: same (and thus
> coinstallable) and "pkg-foreign" a package which is Multi-Arch:
> foreign/allowed (and we assume we actually have installed a foreign
> instance on this package on our system).
>
> How to interpret the input
> --
>
> Consider that we have installed libsame:i386 and libsame:amd64 on the
> system (which is an i386 system).
>
> The user types "dpkg -r libsame". What do you expect dpkg to do?
>
> 1/ Assume libsame == libsame:native and remove libsame:i386 only.
> 2/ Assume he meant all instances of libsame, and remove libsame:i386 and
> libsame:amd64.
> 3/ Fail because the user has not been specific enough.

I think the behaviour should be to match how you print package names. So
lets skip to that.

> How to print the package names
> --
>
> In the same vein, we should decide how dpkg is going to print package
> names in various contexts. Obviously the decision above will have an
> impact here... because scripts might retrieve package names from
> the output of dpkg  and feed it back in the input of other dpkg commands.
>
> My suggestion is along those lines:
> - any package of foreign arch should always have its architecture appended
>   to its package name
> - package of native arch which are multi-arch same should have the
>   architecture appended only if there are other instances of the same
>   package installed
> - other packages do not need the arch qualifier
>
> Does that seem reasonable enough?

Lets first consider an existing systems. The output of packages is
without the architecture. In the future many systems will still have
just one architecture installed despite having multiarch packages
available. The conservative approach is to sill not add the architecture
to the output.

On the other hand for multiarch systems there must be a way to
differentiate the library packages from different architectures. There
can be many architectures but only one native one. Since we can only
allow one architecture to not add the architecture to the name and still
be unique the foreign archs must add the architecture.

If we show native packages without the architecture added at all times
then the output is the same for old systems, new systems with only
native packages and new systems with foreign packages. Installing
libfoo:amd64 does not change that libfoo means libfoo:i386 and that
libfoo:i386 is displayed as libfoo. I think that is the most consistent
way. Having the name change depending on what else is installed would be
confusing.

As a special case I would allow foo to match foo:amd64 if foo is not
multi-arch same package for the purpose of specifying packages but still
display the architecture on output.

> One more question:
> - shall we invent a "foo:native" syntax that we can output instead of
>   "foo:i386" in the output of dpkg --get-selections so that transferring
>   the selection over to another computer running another architecture
>   will still work ?

Using plain "foo" avoids this issue and keeps things compatible between
old and new systems. You can then transfere your selections from squeeze
to wheezy and back.

> I have included de...@lists.debian.org because APT developers have
> certainly taken some decision on this topic when they have implemented
> multi-arch support in APT. Dear APT developers, your input as frontend
> developer is very welcome on those topics.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkpwc6u2.fsf@frosties.localnet

Re: The fsync issue

[Goswin von Brederlow]

Guillem Jover  writes:

> On Sat, 2010-11-27 at 01:41:19 -0600, Jonathan Nieder wrote:
>> Guillem Jover wrote:
>> > Unfortunately that patch does not seem much appealing, it's Linux only,
>> > not even in mainline, and it would need for dpkg to track on which file
>> > system each file is located and issue such ioctl once per file system.

What if you issue one ioctl per file? Won't the duplicates just return
provided there is nothing else writing fresh data to the FS?

>> > I'd rather not complicate the dpkg source code even more for something
>> > that seems to me to be a bug or missfeature in the file system. More so
>> > when there's a clear fix (nodelalloc) that solves both the performance
>> > and data safety issues in general.
>> 
>> I don't really understand this point of view: isn't the fsync storm
>> going to cause seeky I/O on just about all file systems?
>
> Well sure it might, but then some seem to be able to cope just fine, even
> ext4 with nodelalloc. Also seeks might stop being that relevant (in the
> mid/long term) once SSD becomes more widespread.
>
>> So the POSIX primitives are not rich enough to express what we want to
>> happen.  Delayed allocation is pretty much essential for the use case
>> ubifs targets, so it doesn't make much sense to me to pretend it
>> doesn't exist.
>
> As long as delayed allocation is a synonym for zero-length files, then
> I personally consider it a misfeature. This is data loss we are talking
> about, and while data coming from packages is easily recoverable
> although cumbersome, user data might not. We got fsck, journals and
> similar to recover from system crashes, and now we get zero-length
> files in the name of performance, it seems clear to me that's a
> regression.

What if you use data journaling? Shouldn't that replay the data after a
crash and thus not suffer from 0 byte files? Or does delalloc prevent
the data to be written to journal until the time it allocates a block
for it?

> Anyway my thinking process goes a bit like this: There's currently a
> handful of programs doing the complete write+fsync+rename dance, with
> the file systems which need it penalize heavily. If more programs start
> to get "fixed" to do the fsyncs then the situation overall will just
> worsen. And then at that point I think it's completely unreasonable
> to expect every userland program to add such complexity and unportable
> hack over hack to workaround the file system problems.

Usualy one does this on ONE file and everything is fine.

The problem only arises because dpkg is doing this on a million files
and if I understood the problem correctly in ext4 each one of them
causes a lengthy data + metadata + super sync again and again.

I think one long term solution to this might be to invent an async
fsync() call. A way to tell the FS that the file should be synced
soonest and report back when it is done. This should make the FS collect
multiple files into a single sync. One possible way to implement this
would be to mmap each file and msync() it with MS_ASYNC. But as that
doesn't cover the metadata part I'm not to sure it would completly solve
the bottleneck.

Anyone with ext4 feel up to implementing this in dpkg and measuring it?

> For non-technical users, data safety should be way more important than
> performance, having to recover a hosed system might mean they'd just
> reinstall it. For technical users I see the options as follows: help
> fix the file system to perform reasonably with fsync() or not lose
> data w/o fsync(), use another file system, use other better mount
> options, use dpkg --force-unsafe-io and cope with data loss.
>
> But then I think I've said most of this elsewhere already.
>
>> I'll look into a (Linux-specific, obviously) patch to add a function
>> that takes an array of paths and performs the relevant syncs of
>> filesystems where that ioctl exists tomorrow.  I would rather see a
>> system call that just takes an array of paths, since I imagine
>> filesystems like btrfs could do something good with that, but since
>> there are no VFS primitives for it I can see why that wasn't proposed.
>
> Tracking fds is going to be easier, at that point dpkg already has
> the stat information, so it could queue an fd per unique st_dev for
> example.

That sounds like a good plan. How hard would it be to implement this
based on FDs instead of path? Would the ioctl patch need changes to work
on an FD instead of path? (sorry, haven't read the patch)

> regards,
> guillem

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqtp58tu@frosties.localnet

Re: Pre-approval request for dpkg sync() changes for squeeze

[Goswin von Brederlow]

Phillip Susi  writes:

> On 10/24/2010 07:16 AM, Goswin von Brederlow wrote:
>> Or 5 minutes because sync() also needs to write out the 16GB cache data
>> to my usb 1.0 drive that is not involved with dpkg at all.
>
> True, but that seems a bit of a contrived corner case.  Most of the
> time when people are upgrading, I'd wager that they don't have many gb
> of dirty cache buffers already sitting in the cache.  Though that does
> make me wonder why there isn't a sync() variant that applies only to a
> specific mount point.  That would at least keep other unrelated mounts
> out of the picture.
>
> If you are going to use fsync() to make sure you don't flush any
> unrelated data, then maybe you should at least use aio and issue all
> of the fsync calls overlapped at once, so you can hopefully have the
> writes combined into fewer, larger writes, with fewer seeks between
> them.
>
> Then again, you could probably do better by using O_DIRECT when
> writing the file in the first place and avoid having to latter flush
> at all.

Except that the fsync() patches from RedHat were never added to the AIO
code in the kernel. The portable way for this is to mmap() the files and
msync() them with MS_ASYNC. But that makes catching write errors a
nightmare.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874ocallqc@frosties.localdomain

Re: Pre-approval request for dpkg sync() changes for squeeze

[Goswin von Brederlow]

Phillip Susi  writes:

> On 10/22/2010 5:35 AM, Guillem Jover wrote:
>>   1) Switch back from sync() to fsync() before rename() (while keeping
>
> Don't you WANT to use sync?  If you fsync every file that is going to be
> rather slow since it forces a disk write for every file, rather than
> allowing writes between each file to be combined and batched.  Ideally
> you want to unpack all files from all packages being installed/upgraded,
> then issue one big sync(), and finally all of the rename()s, with one
> final sync(), don't you?
>
> To throw out some numbers, it seems like it can easily take 60ms or more
> for an fsync(), between writing the file data, writing the journal,
> writing inode, writing the directory entry, then finally the journal
> again.  If you are replacing 1000 files then you are spending 60 seconds
> just on the fsync calls.  If you only use a single sync, then the whole
> thing could easily write all of the file data, adjacent inodes, journal
> commits, etc in 1-2 seconds with only 2-3 seeks.

Or 5 minutes because sync() also needs to write out the 16GB cache data
to my usb 1.0 drive that is not involved with dpkg at all.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87eibfet8q@frosties.localdomain

Re: reassign to dpkg-dev with request to support -sn for all formats?

[Goswin von Brederlow]

Yaroslav Halchenko  writes:

> CCing Dpkg Developers -- please reassign to dpkg-dev (dpkg-source)
> if you consider my concern valid
>
> I got the same warning while playing with reprepro's changelogs.example.
> It also relies on dpkg-source -sn -x to extract packaging only directory
> from the sources to obtain copyright and Debian changelog files.
>
> Although -sn is listed as a "Format: 1.0" specific, it makes complete
> sense to have it supported for any other format to do precisely that --
> "Ensures that the original source is neither copied to the current
> directory nor unpacked".
>
> As a result of the absent V3 support for -sn, dpkg-source does a copy of
> the original sources, which in our case 1.7GB data package, thus not
> acceptable situation and demolishing the benefits for V3 having a
> neat .debian.tar.g.
>
> Since there is no other cross-format option for dpkg-source to extract
> Debian packaging only, I do not think it is viable to provide
> format-specific handling in every dependent tool (e.g. sbuild,
> reprepro) instead of fixing dpkg-source to provide such generic
> functionality across all (or major) formats.

The same could be said for -sp, -su and --skip-debianization. The last
is specified for 3.0 (quilt) but would make sense for 3.0 (git) and 3.0
(bzr) too [skip the checkout].

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d3retbxo@frosties.localdomain

Re: Use of dpkg to install powerpc .deb on x86 host

[Goswin von Brederlow]

Sergei Poselenov  writes:

> Hello all,
>
> We are developing a design for a Debian-based SDK distribution,
> targetting embedded systems.
>
> The SDK will be installed on a x86 host, using the dpkg utility.
> to install a set of binary packages, part of them for different
> architecture (powerpc) (the so-called target packages).
> By design, the SDK should be able to be installed to arbitrary
> directory, and should not interact with the host' /var/lib/dpkg
> database.
> We are thinking of "dpkg --root=", to install target-depended
> packages to. Will this work?
>
> I guess, dpkg will complain about wrong architecture of the powerpc
> packages? Can this be worked around with --force-architecture? Any
> other ways?
>
> Another question - when installing/removig the packages for different
> architecture, the scripts should not be allowed to be executed - 
> being chrooted to dir pointed to with "--root=" the execution would try
> to execute powerpc binaries on x86 host.
> Is it possible to disable scripts processing in dpkg?
>
> Thanks in advance.
> (Could you please reply me directly - I'm a Debian newbie and not
> (yet) subscribed to this mailing list.)
>
> Regards,
> Sergei

You might want to look at dpkg-cross and ask on the debian-embedded
list.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8739t27ypd@frosties.localdomain

Re: dpkg armhf patch acceptance status?

[Goswin von Brederlow]

Steve Langasek  writes:

> On Wed, Sep 08, 2010 at 11:40:13AM +0200, Guillem Jover wrote:
>> This also causes issue with not being able to have installed two
>> cross-toolchains for say armel and armhf as they share triplet,
>> although you can use the armel toolchain with few options to build for
>> armhf, but then you'd need to specify those as part of the CC variable.
>> Also that also happens with biarch, you can produce i386 binaries from
>> an x86_64 toolchain, yet they both have their own triplet.
>
>> I'm just wondering if this is also the case for mips triarch, or do
>> those have each their own triplet, and is only arm that has this
>> issue?
>
> mips have distinct triplets for each of their archs, yes.
>
>> Anyway, ideally I'd rather see this addressed by giving armhf a real
>> triplet, which would also make multiarch life way easier as there'd not
>> be any need to define an artificial set of neutral architecture names
>> to be able to place objects in the file system.
>
> I realize this is ideal, but:
>
>  - there's been very strong upstream pushback on this, asserting that this
>is the correct triplet to use for both arm calling conventions, so if we
>require a distinct triplet for armhf (instead of using the vendor field),
>that's going to block any armhf port for quite a while (possibly
>indefinitely)
>
>  - armhf was not the sole motivation for the proposal to define neutral
>architecture names; x86 was already a problem because of the changing
>triplets depending on which level of instruction set compatibility is
>targeted.  *Both* of these examples show that GNU triplets are not
>defined in a way that they map directly to what we need for multiarch, so
>it's best to explicitly define our mapping externally.
>
> So even if you persuaded the upstream toolchain folks to specify a new
> triplet for armhf after all, I think we should still go ahead with a
> separate name mapping table for multiarch.
>
> Cheers,

Note that uclibc also suffers this problems. x86_64-linux-uclibc is in
no way unique as different uclibc compile options create different ABIs
all with the same tripplet.

I filed a wishlist bug against dpkg-architecture to please provide
DEB_HOST_ABI / DEB_BUILD_ABI and to start giving that as
DEB_HOST_GNU_TYPE / DEB_BUILD_GNU_TYPE initialy. Ports where this is
insufficient can then extend the table to give a unique value.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87tylyoxtv@frosties.localdomain

Could we add last minute minimal support for pkg:arch?

[Goswin von Brederlow]

Hi,

I'm wondering if it would be possible to add minimal support for
pkg:arch, as needed for full multiarch later on, to dpkg in squeeze.

By minimal support I mean the following changes:

- ':' is allowed in package names. Might be restricted to package names
in dependencies. Naming a package foo:i386 is invalid but Depends:
foo:i386 is valid.

- pkg:arch is preserved in the status and available file.

- pkg: is treated as just pkg internally.

- pkg: is left as is and thus won't ever be fullfilled. No
such package will exist in squeeze dpkg.

Using the simple changes above would make arch:pkg work for uni-arch
configurations. Meaning squeeze+1 could use the arch:pkg syntax without
the packages becoming uninstallable with stable dpkg allowing for better
multiarch support without another release cycle.

If you are willing to consider adding this to squeeze at this late a
date then I would be willing to write a patch along the 4 points above.
I believe the change can be done with minimal changes to the parser
only. The dependency resolver wouldn't be touched.

Note that apt 0.8.0 has added (some) support for pkg:arch already.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87fwy3c8nm@frosties.localdomain

Re: RFC: Proposal to combine diversions and Replaces into DEBIAN/replaces

[Goswin von Brederlow]

Steve Langasek  writes:

> On Fri, Aug 13, 2010 at 11:18:32PM +0200, Goswin von Brederlow wrote:
>> The only real fix for this problem seems to be to preserve the replaced
>> files under another name and restore them when the replacing package is
>> removed. That starts to sound a lot like diversions doesn't it. The
>> only difference is that Replaces is usualy versioned.
>
> I think you are proposing a cure that is worse than the disease.  Installing
> a Replacing package and then removing it, leaving the Replaced package
> broken, is an increasingly obscure corner case, precisely because
> Breaks+Replaces works smoothly enough that there's no reason not to use it,
> and under apt's guidance the Breaks will trigger an upgrade of the affected
> package ASAP.

It might not be too common but it happens often enough. Either due to
downgrades or broken packages that abort an upgrade. It makes removing a
package dangerous since dpkg offers absolutely no protection against it.

If dpkg doesn't keep track of the files then it should at least keep
track of the fact that files were replaced and mark the package as
incomplete and warn the user about the fact when the replacing package
is removed.

> In the rare event that a package is /not/ listed in Breaks, or is kept
> around anyway in a broken state, I don't agree that we should keep the
> replaced files around indefinitely to support this narrow use case.

Never thought about a Replaces without Breaks or Conflicts. Is there
even one such case in Debian?

>> Syntax of the "replaces" file:
>
>> The "replaces" file usualy contains one entry per line. Except that
>> lines starting with a single space are concatenated to the line before
>> with the space, but not the newline, removed. This is solely to support
>> filenames containing newlines.
>
>> Each entry has one of the following forms:
>
>> pkg /absolute/path
>> pkg (<< ver) /absolute/path
>> pkg (<= ver) /absolute/path
>
>> This means that the file  from the package  is to
>> be diverted or replaced. If a version is given the entry only applies if
>>  has version << 'ver' or <= 'ver'. Dpkg will rename the
>>  file from  to > the same
>> way diversions function now.
>
>> The first form provides what dpkg-divert does now while the other two
>> provide what Replaces does.
>
>> To make replacing multiple files simpler the  could
>> contain wildcards like e.g. package foo-data containing
>
>> foo (<< 1.2-3) /usr/share/foo/*
>
> This is insufficient to handle the incompatible requirements of diversions
> and Replaces.  The semantics of diversions are that only the *named* files

And a diversion should not use a wildcard.

> should be moved aside.  The semantics of Replaces are that *any* files of
> the same name belonging to both packages should be replaced.  You don't want
> to move aside every file matching a glob for Replaces handling, only those
> that are actually replaced; and for diversions handling, you don't want to

Which 'foo (<< 1.2-3) /*' could do. Only requirement is that wildcards
match files present in both packages and not all files in foo. Then you
get the behaviour you want.

> automatically assume a diversion is meant for each file collision.

As said above a diversion shouldn't be using a wildcard. Shouldn't need
to since the number of diversions is usualy verry small anyway. They
certainly shouldn't be using 'foo /*' and that would be the only way to
have a diversion for every file conflict.


The syntax for diversions and Replaces is distinct so dpkg can perfectly
know which of the two cases is ment and behave acordingly. But there
isn't even a difference in behaviour needed. A specifically named file
is always renamed and a wildcard matches files present in both packages.

> Declarative diversions are long overdue; but I see no value in conflating
> them with Replaces like this.

- allowing downgrades of replacing packages
- allowing abort of installs
- simplifying the dpkg logic by merging replaces and diversions into a
  single case.

Even if you consider the first 2 as obscure cases the third should be
beneficial.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762z8dsyj@frosties.localdomain

Re: RFC: Proposal to combine diversions and Replaces into DEBIAN/replaces

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Goswin von Brederlow wrote:
>
>> Also converting from old style dpkg-divert to
>> the "replaces" file is simply done by removing all dpkg-divert calls and
>> listing the respective files in "replaces".
>
> This part is hazy to me, but I’m willing to give it the benefit of the
> doubt.

Say you have a package that has diverted /usr/bin/tool and now you want
to change the package to use the "replaces" file instead. How would you
manage the transition?

Assuming dpkg wouldn't take care of this automatically you would have to
first remove the old style diversion in preinst before dpkg adds the new
style transition, which you can't because the diverted file conflicts
with the one you shipped in the old version. Or dpkg would create the
new style transition by probably renaming the wrong file and you would
have to shuffel files around and remove the old style conversion in
postinst in some hackish way to repair things again.

Overall that would create a huge mess risking the maintainers screwing
it up in every diversion using package. So I think it is essential that
dpkg automatically handles transitioning the diversions already in its
DB to the new "replaces" file syntax cleanly so that individual packages
can't screw it up. The information in the diversions DB of dpkg is
basically the same as packages will have in their "replaces" file so
that should not be a big deal to implement. It just moves the entries
from /var/lib/dpkg/diversions to /var/lib/dpkg/info/.replaces.

>> The idea is that no effort has to be taken in the maintainer script to
>> keep the diversions in line. Simply listing them in the "replaces" file
>> is all that is needed.
> [...]
>> All the dangers and
>> dificulties of using diversions right is eliminated.
>>
>> Further replacing files becomes more specific preventing accidentally
>> replacing the wrong files and reversible. This would make the use of
>> Breaks + "replaces" file a safe operation unlike Breaks + Replaces is
>> now, not just when aborting an upgrade but even when downgrading.
>>
>> Comments?
>
> So where’s the patch? :)
>
> Seriously, this sounds like an excellent thing to do, and best of all,
> it would be perfectly safe to include something like this in dpkg as
> an experiment before permitting it in the archive/policy.
>
> Jonathan

So far it is just an idea, unfortunately.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqxlrjqh@frosties.localdomain

RFC: Proposal to combine diversions and Replaces into DEBIAN/replaces

[Goswin von Brederlow]

Hi,

recently on irc we discussed the problem of using Breaks + Replaces when
moving files from one package to another. The problem is that Replaces
is a one way street. Once the replacing package gets unpacked you can not
abort or later remove the the replacing package without loosing files in
the replaced package, which usualy breaks the replaced package.

The only real fix for this problem seems to be to preserve the replaced
files under another name and restore them when the replacing package is
removed. That starts to sound a lot like diversions doesn't it. The
only difference is that Replaces is usualy versioned.

On the other hand diversions have problems of their own. While the use
case is nearly always the same their actual use is verry fragile. One
has to get the preinst and postrm scripts just right and obsoleting a
diversion leaves cruft in the maintainer scripts for years to
come. Worse moving a diversion from one package to another is nearly
impossible.


So here is my proposal:

1) dpkg-divert is deprecated for use in packages. It should only be used
for diversions by the local administrator. The functionality remains
though for backward compatibility.

2) the Replaces field is deprecated for moving files from one package to
another. Its use to replaces whole packages could remain (used with
Conflicts). The functionality remains though for backward compatibility.

3) A new file is introduced in the control.tar.gz called "replaces" that
takes over the job of diversions and file replacements.


Syntax of the "replaces" file:

The "replaces" file usualy contains one entry per line. Except that
lines starting with a single space are concatenated to the line before
with the space, but not the newline, removed. This is solely to support
filenames containing newlines.

Each entry has one of the following forms:

pkg /absolute/path
pkg (<< ver) /absolute/path
pkg (<= ver) /absolute/path

This means that the file  from the package  is to
be diverted or replaced. If a version is given the entry only applies if
 has version << 'ver' or <= 'ver'. Dpkg will rename the
 file from  to > the same
way diversions function now.

The first form provides what dpkg-divert does now while the other two
provide what Replaces does.

To make replacing multiple files simpler the  could
contain wildcards like e.g. package foo-data containing

foo (<< 1.2-3) /usr/share/foo/*



Implementation wise dpkg will handle additions or removals to the
"replaces" file automatically. That means that if on upgrade a new entry
appears in "replaces" then dpkg will rename the existing file before
unpacking the new package. If an old-style diversion for this file
already exists by the same package that diversion is taken over by dpkg
and now automatically handled. If an entry is removed from "replaces"
on upgrade then dpkg will restore the replaced file after removing the
files from the old package.

Dpkg will also have to handle the case of multiple packages diverting
or replacing the same file. I see 2 possible reasons why this could
happen legally:

1) A file conflict is turned into an alternative

Say package A, B and C all contain /usr/bin/tool. To turn that into an
alternative each of these packages have to divert /usr/bin/tool from the
other too prior to the version with alternatives. That ensures that at
the time the alternative is created in postinst /usr/bin/tool will not
exist. It might be possible to use a virtual package name in the
"replaces" file to capture all packages providing /usr/bin/tool with a
single entry instead of listing each individually. [Which then adds the
exception that a package never diverts/repalces itself.]

2) A file is moved multiple times

Say a file moves from A to B and later from B to C. Then B will replace
the file from A and C will replace the same file from B. But dpkg will
know which is ment in each case and only the file from C is left with
the original name. The other 2 files will be names .dpkg-A and .dpkg-B
if they exist.



The idea is that no effort has to be taken in the maintainer script to
keep the diversions in line. Simply listing them in the "replaces" file
is all that is needed. Also converting from old style dpkg-divert to
the "replaces" file is simply done by removing all dpkg-divert calls and
listing the respective files in "replaces". All the dangers and
dificulties of using diversions right is eliminated.

Further replacing files becomes more specific preventing accidentally
replacing the wrong files and reversible. This would make the use of
Breaks + "replaces" file a safe operation unlike Breaks + Replaces is
now, not just when aborting an upgrade but even when downgrading.

Comments?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/877hjuky8n@frosties.localdomain

Re: Multiarch and ABI support

[Goswin von Brederlow]

Hector Oron  writes:

> Hi Goswin,
>
 It wouldn't.  I don't see a compelling reason for dpkg to do this at all.
 Your quote shows that dpkg *does* do this today, which I didn't remember
 before this conversation, but that's not an explanation for *why* it does
 -
 as opposed to dpkg directly recording what its current architecture is.
>>>
>>> I am not `dpkg' maintainer, nor I did such change, so *why* it does
>>> that I do not know, but surely there is a reason which stands valid or
>>> it is outdated.
>>
>> Compiling dpkg on a system that has no dpkg. No dpkg-architecture to get
>> the current architecture from.
>
> Sounds like a reason. Thanks.
>
>> /sysroot/usr/lib/ is the normal directory.
>> /sysroot/usr/lib/ is the multiarch directory under a sysroot
>>
>> Are you thinking of the special case of multiarch with sysroot=/?
>
> Yes, I was thinking that but later I realized that does not fit. Where
> do you place cross built binaries? It could work as build this armel
> binary and upload it to target device, but it will not work for cross
> building a small rootfs (uclibc+busybox) with in host system and then
> export it to target device or roll a tarball.

Obviously you won't be able cross-build a bash and install it in /. But
you can build all the debs for a rootfs and then debootstrap
--first-stage them, boot that as NFS root and run the second stage on
the target. Or use qemu to emulate the target architecture.

But that is the standard problem of not being able to run maintainer
script when cross creating a rootfs. I don't see where sysroot comes
into this.

>> What is wrong with the path is that the triplet is not unique and
>> sometimes differs when it shouldn't. So instead of the triplet use a
>> different string that uniquely identifies the abi. In the past the
>> triplet has just been the standard for that.
>>
>> Could I suggest dpkg-architecture -qDEB_HOST_ABI? The string should only
>> be hardcoded in one single spot.
>
> I think there is yet another proposal waiting on the pipeline to be
> proposed publicly.
>
> Cheers,
> -- 
>  Héctor Orón

There is. The hints dropped so far though indicate that its only to fix
the  into something better fitting. As said before in this
thread it really doesn't matter what you call it or what is used in the
end. The idea of /usr/lib// remains and the actual
string used doesn't change the implementation needed, just the string.

It might be better to call it a refinement of the existing proposal than
to risk people thinking they are competing proposals. At least until
there actualy is one.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87wrsielly@frosties.localdomain

Re: Multiarch and ABI support

[Goswin von Brederlow]

Hector Oron  writes:

> Dear Steve,
>
> 2010/7/24, Steve Langasek :
>> On Mon, Jul 19, 2010 at 07:02:32PM +0100, Hector Oron wrote:
>>> 2010/7/18 Steve Langasek :
>
>>> AFAICS `dpkg' relais on -dumpmachine from `gcc'
>>> scripts/Dpkg/Arch.pm:68:my $gcc_host_gnu_type = `\${CC:-gcc}
>>> -dumpmachine`;
>
>> It wouldn't.  I don't see a compelling reason for dpkg to do this at all.
>> Your quote shows that dpkg *does* do this today, which I didn't remember
>> before this conversation, but that's not an explanation for *why* it does -
>> as opposed to dpkg directly recording what its current architecture is.
>
> I am not `dpkg' maintainer, nor I did such change, so *why* it does
> that I do not know, but surely there is a reason which stands valid or
> it is outdated.

Compiling dpkg on a system that has no dpkg. No dpkg-architecture to get
the current architecture from.

>>> Using sysroot approach you might be able to co-install foreign (other
>>> arches) libraries into the sysroot for case a) - it even lets the user
>>> have multiple systems with same ABI and architecture, but with
>>> different libraries (in the case you want to cross build non Debian
>>> systems).
>
>> sysroot is irrelevant to what multiarch aims to achieve.
>
> Sorry, I think I have not yet understand you. Would you mind to expand
> your point here?
>
> While sysroot is just a tool which it is probably very helpful on
> multiarch design. What is orthogonal on sysroot and multiarch are the
> hardcoded paths (/usr/lib/) which we had discussed above that
> are wrong.

/sysroot/usr/lib/ is the normal directory.
/sysroot/usr/lib/ is the multiarch directory under a sysroot

Are you thinking of the special case of multiarch with sysroot=/?

What is wrong with the path is that the triplet is not unique and
sometimes differs when it shouldn't. So instead of the triplet use a
different string that uniquely identifies the abi. In the past the
triplet has just been the standard for that.

Could I suggest dpkg-architecture -qDEB_HOST_ABI? The string should only
be hardcoded in one single spot.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d3ubuv1b@frosties.localdomain

Re: Multiarch and ABI support

[Goswin von Brederlow]

Hector Oron  writes:

> (CC: debian-dpkg as it is `dpkg' and multiarch related)
> (Reset and rename subject from `Re: cortex /
> arm-hardfloat-linux-gnueabi (was Re: armelfp: new architecture name
> for an armel variant)' to current)
> (Please follow-up discussion on debian-dpkg@ for multiarch and `dpkg'
> related topics)
>
> Dear Steve,
>
> 2010/7/18 Steve Langasek :
>> I'm puzzled why dpkg needs a unique triplet for a port.  dpkg needs to map
>> port names to triplets, but why does it need to do the inverse?  And if it
>> doesn't need to map triplet->port, why would the triplet need to be unique?

Another reason is that packages build for one triplet should not be
accidentally installable on another triplet. What is important is that
the ABI is compatible and so far only the debian port name is used to
ensure that, which is based purely on the triplet. So currently the
assumption is that equal (compatible, think i*86-linux-gnu) triplet
means compatible ABI.

> AFAICS `dpkg' relais on -dumpmachine from `gcc'
> scripts/Dpkg/Arch.pm:68:my $gcc_host_gnu_type = `\${CC:-gcc}
> -dumpmachine`;
>
> Then all the magic starts...
> How would `dpkg' can then map GCC tuplets into Debian triplets being
> the same with different Debian architecture names?
>
> $ gcc -dumpmachine
> arm-linux-gnueabi
> $ dpkg --print-architecture
> armel
> which maps to 'armel' on the triplet tables.
> (It seems that those mappings are perl hash maps one-to-one)
>
> In 'armhf' case $ gcc -dumpmachine spits the same GCC tuplet (unless
> we use GCC vendor tag as an ABI tag)
> But `dpkg' do not mach quadruplet names, not yet... ;-)
>
>> There is a need for a distinct port name; I think this is also not a
>> Debian-specific problem, it's a problem common to any distributions that
>> want to do what's described here.
>
> There is no need of distinct port name for 'any' distro. There is need
> of ABI distinction within Debian. How? Adding an ABI control field (as
> I said previously on this conversation)? Codify vendor tag as ABI tag?
> else?

It could be nice to have a full debian i386 port (as we have now) and
partial ports for i586, i686 with just a few relevant packages. All of
them setting ABI: ia32. I know that it is actualy the reverse of what
you suggest but it would work with the same code.

>> Paul commented that:
>>
>>> Anything that relies on the target triplet is going to break as soon as you
>>> move outside a pure Debian system. i.e. any patches relying on a particular
>>> target triplet are inherently Debian specific, and IMO should never be
>>> pushed upstream.
>>
>> But, er, relying on the target triplet for this shouldn't be done in Debian,
>> either!
>
> We have same GCC tuplet, with different ABI that needs mapping into
> Debian naming scheme, how do we workout that (without hacks)? (again,
> same questions as above).
>
> This problem can be also seen on MIPS triarch, where Anthony Fok tried
> to bootstrap some MIPS variant during a GSoC [1] (MIPS 3 N32 ABI
> Port), but again they had name clashing.
>
>> FWIW, this discussion ties in with one of the known outstanding issues with
>> multiarch, namely we want to support coinstallation of libraries optimized
>> for multiple /compatible/ instruction sets.  And we don't want to have to
>> add /lib/i386-linux-gnu, /lib/i486-linux-gnu, /lib/i586-linux-gnu [,...] to
>> the path one by one to accomplish that, especially when we already have
>> hwcaps to do the job for us.  So perhaps triplets aren't the right level of
>> abstraction to encode in the library paths after all?  I mean, it's ok to
>> have libraries in /lib/i486-linux-gnu/686/cmov, but we definitely don't want
>> to *change* the library path if and when Debian's base compatibility level
>> moves from i486 to i586 (or from armv4t to armv5t, to keep this on topic ;)
>> and have to deal with path transitions.  Is there a better identifier we
>> should be using to qualify the library paths, instead of these triplets?
>> I.e., instead of the cpu from the triplet, perhaps there are official names
>> for the ELF architectures that can be used - that can be determined without
>> too much hard-coding all over the place?

The transition should be relatively painless. The triplet comes from
dpkg-architecture and adding multiple triplet paths to
/etc/ld.so.conf.d/ is trivial. So nothing is hardcoded in 99% of
packages. The problem cases are plugins and there the library/app using
the plugin and the plugins themself have to be compiled using the same
triplet. Often a binNMU is enough but sometimes it will require a
versioned Depends.

The ELF architecture is not specific enough. For example both glibc and
uclibc linked binaries are elf_x86-64 but they certainly are not
compatible.

> As I had stated previously my though on another thread [2], there are
> two different cases mixed on `multiarch':
> a) Co-installing libraries (in x86 world, with compatible ABI) which
> main purpose it is to be able to run `non-free' softwa

Re: dpkg-buildflags

[Goswin von Brederlow]

Pierre Habouzit  writes:

> On Thu, Jul 01, 2010 at 10:04:02AM +0200, Goswin von Brederlow wrote:
>> Pierre Habouzit  writes:
>> 
>> > On Wed, Jun 30, 2010 at 03:05:10PM +0200, Goswin von Brederlow wrote:
>> >> m...@book:~% cat Makefile
>> >> $(eval $(shell echo FOO=1; echo BAR=2))
>> >> $(info FOO=$(FOO))
>> >> $(info BAR=$(BAR))
>> >> all:
>> >> 
>> >> m...@book:~% make
>> >> FOO=1 BAR=2
>> >> BAR=
>> >> make: Nothing to be done for `all'.
>> >> 
>> >> 
>> >> How do you get make to honor line breaks in $(eval ...)?
>> >
>> > gah, make is annoying. Well there is always the possibility:
>> >
>> > -include dpkg-buildflags.mk
>> > dpkg-buildflags.mk:
>> >$(shell dpkg-buildflags --export=make) > $@
>> >
>> > Which is probably a hell of a lot simpler.
>> > Stupid make
>> 
>> Indeed.
>> 
>> > (for those not very comfortable with make, the -include
>> > dpkg-buildflags.mk will check if the file exist, if not won't complain
>> > but will find if there is a target named 'dpkg-buildflags.mk', if yes
>> > will do its rules and retry to include the file.
>> >
>> > Which in the end does what is meant.
>> 
>> Except it won't make the file ever again even if the dpkg-buildflags
>> output changes. So if you want to preserve timestamps unless the output
>> changes you need something like this. If the timestamp doesn't matter
>> you just need the .PHONY.
>> 
>> -include dpkg-buildflags.mk
>> dpkg-buildflags.mk:
>>  $(shell dpkg-buildflags --export=make) > $...@.new
>> @if ! cmp 2>/dev/null --quiet $...@.new $@; then \
>>   echo Updating $@; \
>>   mv $...@.new $@; \
>> else \
>>   rm $...@.new; \
>> fi
>
> You don't need to protect the regeneration with your
> update-only-if-changed trick because no rule will ever depend upon the
> dpkg-buildflags.mk freshness, so the sole need you have is to mark the
> rule PHONY if you need such a thing.

Since the file should be removed in clean it probably doesn't need to be
PHONY. If any of the flags are changed you need to remove all object
files anyway so debian/rules clean would be in order then.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkybh8ms@frosties.localdomain

Re: RFC: Building a source package directly from a RCS

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Hi Goswin,
>
> Goswin von Brederlow wrote:
>
>> Now to build a source I do the following:
>> 
>> 1) if missing: create the orig.tar.ext using pristine-tar
>> 2) if needed: update the auto-patched branch
>> 3) Check out debian dir
>> 4) if used: Create debian/patches/* from branches, stacked git,
>>mercurial patch queue
>> 5) generate debian/patches/debian-changes[-version] by "diffing"
>>auto-patched against master
>> 6) Create debian.tar.ext from 3+4+5
>> 7) Create a .dsc file from 1+6
>> 
>> Overall this takes full benefit of the RCS and avoids for every (source)
>> build a costly unpacking of orig.tar.ext, patching and diffing the
>> source by using the superior features of the RCS to do the same job.
>> 
>> Curently I've put together a little wrapper, call it a proof-of-concept,
>> that does exactly that (except step 2). I use the 3.0 (custom) format
>> and in step 7 I run
>>   dpkg-source --target-format="3.0 (quilt)" -b foo-1.0 foo_1.0.orig.tar.gz 
>> foo_1.0-1.debian.tar.gz
>
> Sounds interesting.
>
> What is the user experience like?  With separate debian dir, sometimes
> making changes can be a pain:
>
>  checkout patched-upstream
>   fix a build system bug
>  checkout debian
>   update rules
>  checkout master
>   merge patched-upstream
>   merge debian
>   test
>  ... go back and fix things ...
>
> while if the patched upstream + debian dir is tracked with a single tree,
> it can be easier:
>
>  checkout master
>   fix a build system bug
>   update rules
>   test
>   ... fix things ...
>
> with the main downside being that it can be hard to generate
> debian/patches/* from a history with debian dir and upstream changes
> mixed.

With the above the debian/patches/* is generated from the feature
branches and the working dir. So if you just edit master the changes
collect in debian/patches/debian-changes[-version]. Thenm when you are
finished testing and everything works, the right thing to do would be to
create a new feature branch and cherry pick the commits for that
feature.

> Where does your tool fall in this spectrum?  What branch is checked
> out when the wrapper is called?

Currently my script does check out the debian branch. But that is just a
proof-of-concept implementation. You could just as well checkout the
debian subdir from the master branch, provided your RCS allows that. Or
just plain "cp -a $WORKDIR/debian $TMPDIR/". With uncommited changes (to
the debian dir) the last would be neccessary anyway. Often I like to
test build something before commiting it so that is something to
implement if the overall idea is sound.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d3v7ioo3@frosties.localdomain

Re: dpkg-buildflags

[Goswin von Brederlow]

Pierre Habouzit  writes:

> On Wed, Jun 30, 2010 at 03:05:10PM +0200, Goswin von Brederlow wrote:
>> m...@book:~% cat Makefile
>> $(eval $(shell echo FOO=1; echo BAR=2))
>> $(info FOO=$(FOO))
>> $(info BAR=$(BAR))
>> all:
>> 
>> m...@book:~% make
>> FOO=1 BAR=2
>> BAR=
>> make: Nothing to be done for `all'.
>> 
>> 
>> How do you get make to honor line breaks in $(eval ...)?
>
> gah, make is annoying. Well there is always the possibility:
>
> -include dpkg-buildflags.mk
> dpkg-buildflags.mk:
>   $(shell dpkg-buildflags --export=make) > $@
>
> Which is probably a hell of a lot simpler.
> Stupid make

Indeed.

> (for those not very comfortable with make, the -include
> dpkg-buildflags.mk will check if the file exist, if not won't complain
> but will find if there is a target named 'dpkg-buildflags.mk', if yes
> will do its rules and retry to include the file.
>
> Which in the end does what is meant.

Except it won't make the file ever again even if the dpkg-buildflags
output changes. So if you want to preserve timestamps unless the output
changes you need something like this. If the timestamp doesn't matter
you just need the .PHONY.

-include dpkg-buildflags.mk
dpkg-buildflags.mk:
$(shell dpkg-buildflags --export=make) > $...@.new
@if ! cmp 2>/dev/null --quiet $...@.new $@; then \
  echo Updating $@; \
  mv $...@.new $@; \
else \
  rm $...@.new; \
fi

.PHONY: dpkg-buildflags.mk

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87hbkjiq6l@frosties.localdomain

RFC: Building a source package directly from a RCS

[Goswin von Brederlow]

Hi,

the other day Ron Lee mentioned on irc how he builds a debian source
package from git with a wrapper script and there was some discussion of
the use and danger of the various -sX flags to dpkg-source.

His method creates both the .orig.tar.gz and .orig directory from his
git repository and tells dpkg to use them in an effort to save
extracting the orig.tar.gz needlessly. But that still means dpkg will
diff the orig against the working dir to create the diff.gz and risks
diverging orig.tar.gz and .orig dirs. And there are some problems
adapting this to the 3.0 formats. So the method isn't really optimal.


So lets look at this fresh. I'm assuming the following workflow and
repository layout:

upstream - upstream source from the orig.tar.ext
pristine-tar - delta + id to recreate orig.tar.ext
debian   - debian directly only (without patches if patch-* is used)
master   - working dir

Optional (or stacked git ot mercurial patch queue):

patch-foo- upstream + foo patched in
patch-bar- upstream + bar patched in
patch-baz- upstream + baz patched in

Automatic branch:

auto-patched - upstream with all patch-* applied


Resolving merge conflicts for patch-* in some way is left to the
implementation. For simplicity lets assuming the patch-* branches are
already serialized and merge conflict free.

The above is just an example. What is required is that the orig source
is there, the debian dir is there and a branch with debian/patches/*
applied can be maintained (auto-patched).


Now to build a source I do the following:

1) if missing: create the orig.tar.ext using pristine-tar
2) if needed: update the auto-patched branch
3) Check out debian dir
4) if used: Create debian/patches/* from branches, stacked git,
   mercurial patch queue
5) generate debian/patches/debian-changes[-version] by "diffing"
   auto-patched against master
6) Create debian.tar.ext from 3+4+5
7) Create a .dsc file from 1+6

Overall this takes full benefit of the RCS and avoids for every (source)
build a costly unpacking of orig.tar.ext, patching and diffing the
source by using the superior features of the RCS to do the same job.

Curently I've put together a little wrapper, call it a proof-of-concept,
that does exactly that (except step 2). I use the 3.0 (custom) format
and in step 7 I run
  dpkg-source --target-format="3.0 (quilt)" -b foo-1.0 foo_1.0.orig.tar.gz 
foo_1.0-1.debian.tar.gz



So what do you think about this approach?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87iq50k44b@frosties.localdomain

Re: dpkg-buildflags

[Goswin von Brederlow]

Pierre Habouzit  writes:

> On Mon, Jun 07, 2010 at 08:53:03AM +0200, Raphael Hertzog wrote:
>> > * If packages should set all the flags, have you considered having a
>> >   mode where it lists them all (like dpkg-architecture -l), perhaps
>> >   with export statements?
>> 
>> Would you like this? I have no problem adding this feature to
>> dpkg-buildflags.
>
> That would be very useful, and you want it to be suitable for $(eval).
>
> It's likely that you will have to support various escapings for it to be
> suitable for sh eval, make eval, and maybe other languages.
>
> something like:
>
> dpkg-buildflags --export=[sh|make]
>
>
> One would use it this way with the shell:
>
> eval $(dpkg-buildflags --export=sh || echo exit $?)
>
> and in make:
>
> $(eval $(shell dpkg-buildflags --export=make || echo '$$(error 
> dpkg-buildflags failed with code '$$?')')))
>
> It's likely to be wise that the output "exports" the variables (as in
> shell or make "export" feature to make this bit of the environment
> available to sub-shells).

m...@book:~% cat Makefile
$(eval $(shell echo FOO=1; echo BAR=2))
$(info FOO=$(FOO))
$(info BAR=$(BAR))
all:

m...@book:~% make
FOO=1 BAR=2
BAR=
make: Nothing to be done for `all'.


How do you get make to honor line breaks in $(eval ...)?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87mxuck6wp@frosties.localdomain

Re: dpkg-buildflags

[Goswin von Brederlow]

Raphael Hertzog  writes:

> Hi Joey,
>
> On Sun, 06 Jun 2010, Joey Hess wrote:
>> Some questions I have..
>> 
>> * Do you anticipate adding more flags later? [1]
>
> Possibly yes, although they might not necessarily be among the flags you
> listed.
>
> Ubuntu uses dpkg-buildflags to export supplementary variables related to
> their hardened gcc wrapper (see Dpkg::Vendor::Ubuntu) so in essence
> we already have the case where dpkg-buildflags knows about more flags.
>
> I also wondered if we could not (ab)use dpkg-buildflags to offer a simple
> standardized way to forward supplementary options to ./configure:
>
> ./configure [...] $(dpkg-buildflags --get configure_opts)

Will there ever be any flags that are not configure_opts?

> Maybe it could even provide the --build and --host options as required
> by default (or maybe not if we want to a have a single flag name that can be
> used whatever the build system currently is).
>
> We could define a rule that all caps flags are to be exported while others
> are not.
>
>> * Do you think it would be a good idea for packages to set all flags
>>   dpkg-buildflags supports?
>
> At first I hardcoded a list of flags in dpkg-buildpackage but then
> we dropped that so that Ubuntu can export more flags than the initial
> list.
>
> So I think that it's probably best (in terms of consistency) to have
> all flags exported.
>
>> * If packages should set all the flags, have you considered having a
>>   mode where it lists them all (like dpkg-architecture -l), perhaps
>>   with export statements?
>
> Would you like this? I have no problem adding this feature to
> dpkg-buildflags.
>
> Cheers,

An equivalent to

.buildflags:
for i in $(shell dpkg-buildflags --list); do \
  echo "$$i := $$(dpkg-buildflags --get $$i)"; echo "export $$i"; \
done >.buildflags

#include .buildflags
PHONY: .buildflags

would be nice. I.e. set and export all variables dpkg-buildflags knows
about.

MfG
Goswin



-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sk4dbwrm@frosties.localdomain

Re: Bug#582423: tech-ctte: reaffirm that violating Debian Policy deserves RC bug [and 1 more messages]

[Goswin von Brederlow]

Ian Jackson  writes:

>> >> +---+-
>> >> Rename A to B   | optional make A   | Conflicts: A
>> >> | dummy/transitional| Replaces: A
>> >> |   Depends: B  | Provides: A optional
> ...
>> Going the way of a dummy/transitional package is probably better than
>> the provides way. But I'm not sure there too. But one or the other can
>> be needed:
>
> Yes.  I think the transitional packages are usually better, but
> it seems that Provides is acceptable.  My proposed policy wording
> needs to be fixed.
>
>> Note: With a dummy/transitional package the Conflicts/Replaces in B must
>> be (<<) versioned I believe. Or should that be Breaks/Replaces then?
>
> Yes, it should be versioned Conflicts/Replaces as Jonathan points out.

Policy says Conflicts should always never be versioned. If
Conflicts/Replaces should be versioned then I think that should be
clarified more saying a lonely Conflicts shouldn't be versioned but a
Conflicts/Replaces combo should.

>> >> +---+-
>> >> Makes older A not work  | Breaks: B (<<)| Breaks: A (<<)
>> >> but newer A is ok   | if it conflicted  |
>> >
>> > I'm not sure I follow your suggestion for A flags.  If what conflicted
>> > with what ?I don't think anything should be needed in A.
>> 
>> Package: A
>> Versions: 1
>> Conflicts: B
>> 
>> Package: B
>> Versions: 1
>> Conflicts: A
>> 
>> ===>
>> 
>> Package: A
>> Versions: 2
>> Breaks: B (<< 2)
>> 
>> Package: B
>> Versions: 2
>> Breaks: A (<< 2)
>> 
>> If the old A conflicts with B then there is no way around updating A
>> along with B to make them coinstallable.
>
> How does the Breaks help here ?  It doesn't make the Conflict cease to
> have effect so yes if you have a system which has one of the old A or
> B you cannot install the other, even the new one, until you have first
> upgraded the one you have.
>
> Breaks relationships do not have to be symmetric, unlike Conflicts
> which are implicitly symmetric even though they are declared by only
> one of the packages.

The Breaks prevent the new A to be configured with the old B or new B
with old A. It also allows (hints even) any old A or B to be
deconfigured and then A and B can be upgraded/installed at the same
time. Unlike Conflicts it doesn't put such a constraint on the order the
packages are upgraded/installed.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8739xk1yf9@frosties.localdomain

Re: Bug#582423: tech-ctte: reaffirm that violating Debian Policy deserves RC bug

[Goswin von Brederlow]

Ian Jackson  writes:

> Goswin von Brederlow writes ("Re: Bug#582423: tech-ctte: reaffirm that 
> violating Debian Policy deserves RC bug"):
>> Just to make sure that means now the following is to be used, right?
>
> Thanks for the helpful presentation of the questions.
>
> Typically the situation involves an old package and a newer one, and
> the idea is that the old package does not need to have anything put in
> its dependencies (since after all the purpose is to deal with it).
>
> So, with the proviso that "A Flags" and "B Flags" refer in each case
> to the new version of each package:

I was looking at it from the point of A existing in the archive while
you prepare a new version of B. But then I found that for some cases A
is required to already have a flag or needs to be updated as well to do
have it. But yeah, it would be nice if only B need to be uploaded.

>> Action  | A Flags   | B Flags
>> +---+-
>> Move file from A to B   | Depends: B (>=)   | Replaces: A (<<)
>> | or Breaks: B (<<) |
>> | as needed |
>
> Yes, but only in theYes.
>   new A without the file.

If there is no new A then installing A, installing B, removing B will
leave A with missing files. By updating A to be without the file that at
least gets avoided when you upgrade all packages to the same
release. While not required I think it would be good to recommend
updating A as soon as B has entered the archive or the forseeable
future. If A and B come from the same source anyway the point is mood.

>> +---+-
>> Rename A to B   | optional make A   | Conflicts: A
>> | dummy/transitional| Replaces: A
>> |   Depends: B  | Provides: A optional
>
> I think this is right but I'd like Raphael or someone to confirm.
> This contradicts what I wrote in my proposed policy fragment about a
> package not conflicting/replacing/providing a single virtual package.

Going the way of a dummy/transitional package is probably better than
the provides way. But I'm not sure there too. But one or the other can
be needed:

Package: C
Depends: A

Without either a dummy/transitional package or a provides C becomes
uninstallable.

Note: With a dummy/transitional package the Conflicts/Replaces in B must
be (<<) versioned I believe. Or should that be Breaks/Replaces then?

Note2: So far only dummy/transitional package work on versioned depends.

>> +---+-
>> Makes older A not work  | Breaks: B (<<)| Breaks: A (<<)
>> but newer A is ok   | if it conflicted  |
>
> I'm not sure I follow your suggestion for A flags.  If what conflicted
> with what ?I don't think anything should be needed in A.

Package: A
Versions: 1
Conflicts: B

Package: B
Versions: 1
Conflicts: A

===>

Package: A
Versions: 2
Breaks: B (<< 2)

Package: B
Versions: 2
Breaks: A (<< 2)

If the old A conflicts with B then there is no way around updating A
along with B to make them coinstallable.

>> +---+-
>> Implements virtual pkg  | Conflicts: virt   | Conflicts: virt
>> | Provides: virt| Provides: virt
>
> Often two implementations of the same virtual package can coexist (eg
> because they use update-alternatives or some similar mechanism) so
> Conflicts is optional.

Right. Forgot about that.

>> PS: If this is all correct maybe this overview could be added to the
>> Debian Developer's Reference 6.7. Common packaging situations.
>
> Yes, that would be useful.
>
> Ian.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87632hrrk2@frosties.localdomain

Re: Bug#582423: tech-ctte: reaffirm that violating Debian Policy deserves RC bug

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Ian Jackson wrote:
>
>>  A. Open a new bug against policy, asking that 7.6.2 be replaced with
>> something like this (exercising our power to specify what should
>> be in policy):
>>   
>>   Secondly, Replaces allows the packaging system to resolve which
>>   package should be removed when there is a conflict - see
>>   Conflicting binary packages - Conflicts, Section 7.4.  This
>>   usage only takes effect when the two packages do conflict, so
>>   that the two usages of this field do not interfere with each
>>   other.
>> 
>>   Conflicts+Replaces should be used only to ensure that obsolete
>>   packages are removed in favour of new packages.  A package
>>   should not declare Replaces against any non-obsolete package,
>>   and it should not declare Replaces against any virtual package
>>   it itself provides.
>> 
>>   (In previous versions of policy there was a recommendation to do
>>   the opposite.  That was incorrect and inconsistent with the
>>   purpose of Replaces, and was not fully honoured by dpkg.
>
> For what it’s worth, I like this proposal and think it’s the right thing
> to do.  Thank you for making it.
>
> Jonathan

Just to make sure that means now the following is to be used, right?

Action  | A Flags   | B Flags
+---+-
Move file from A to B   | Depends: B (>=)   | Replaces: A (<<)
| or Breaks: B (<<) |
| as needed |
+---+-
Rename A to B   | optional make A   | Conflicts: A
| dummy/transitional| Replaces: A
|   Depends: B  | Provides: A optional
+---+-
Never work together | Conflicts: B  | Conflicts: A
+---+-
Makes older A not work  | Breaks: B (<<)| Breaks: A (<<)
but newer A is ok   | if it conflicted  |
+---+-
Implements virtual pkg  | Conflicts: virt   | Conflicts: virt
| Provides: virt| Provides: virt
+---+-

MfG
Goswin

PS: If this is all correct maybe this overview could be added to the
Debian Developer's Reference 6.7. Common packaging situations.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/877hmyovka@frosties.localdomain

Re: [PATCH] dpkg-source: do not test source package for --before-build

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Raphael Hertzog wrote:
>
>> Pushed a slightly modified patch, thanks.
>
> Looks good.  Thank you.
>
>> There's a changelog entry though:
>
> Yes, sorry for the lack of clarity.  No additional changelog entry
> needed is what I meant.
>
> Good night,
> Jonathan

Thanks Raphael for writing this and thanks Jonathan for testing. I
eagerly await this featurein sid. A glas of milk for you both.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87bpcaoxdu@frosties.localdomain

Re: "dpkg-buildpackage -b" and "dpkg-source --before-build"

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Hi Raphaël,
>
> Trying to build a package of format 3.0 (quilt) with the
> upstream tarball missing, I get errors:
>
>  $ debuild -I -i -b
>   dpkg-buildpackage -rfakeroot -D -us -uc -I -i -b
>  [...]
>   dpkg-source -I -i --before-build csound

m...@frosties:/tmp/mdadm/mdadm-3.1.1% dpkg-source -I -i --before-build mdadm
dpkg-source: need -x or -b

What version of dpkg-dev is that? My 1.15.7.2 doesn't have that option.

>  dpkg-buildpackage: host architecture i386
>  dpkg-source: error: can't build with source format '3.0 (quilt)': no 
> orig.tar file found
>
> As I understand it, the problem is this:
>
>  - before building, people want dpkg-buildpackage to make sure
>all patches are applied for them
>
>  - this requires looking at the source package
>
>  - there is no source package
>
> Personally, I am not sure about that first step: shouldn’t
> dpkg-buildpackage just build whatever it is presented with, leaving
> the unpack and build processes entirely independent?  In other words,
> I wish the facility to call --before-build were optional.

Definetly not. It is too easy to quilt pop a patch and forget to push it
again. And then you build the package without the patch.

I don't see why you need the orig.tar.gz for this though. Look at the
.pc dir and debian/patches/series and figure it out from there without
this error. All patches need to be applied except the last one if it is
debian-changes (unversioned case) or debian-changes-.
I think the debian-changes patch should be optional as a source build
would recreate it from the current working dir. Building with and
without source building should behave the same in regards to applying
patches.

> But I am open to other solutions, too.  Thoughts?
> Jonathan

Fix dpkg-source --before-build to work without orig.tar.gz and maybe add
an option to build an unpatched working dir for when you really want to
do that.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r5l7magg@frosties.localdomain

Re: file filtering in dpkg -i

[Goswin von Brederlow]

Martin Pitt  writes:

> Hello dpkg team,
>
> for a project of mine (embedded/thin client system) I need to bring up
> a reasonably useful system in as little disk space as possible. This
> involves careful selection of packages, file system, etc., but one
> part of this is also to reduce the footprint of packages themselves,
> such as removing documentation.

You might want to join the Debian-embedded ML.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87632mmzq8@frosties.localdomain

Re: Bug#574956: dpkg drops zero-epoch in status file

[Goswin von Brederlow]

Florian Weimer  writes:

> * Integer overflow in epoch handling
>
> (i386)$ dpkg --compare-versions 4294967296:1 '>>' 4294967295:1 ; echo $?
> 1
> (amd64)$ dpkg --compare-versions 4294967296:1 '>>' 4294967295:1 ; echo $?
> 0

Well, this is wrong if one is to take the wording of policy to mean a C
type. An "unsigned integer" has the same size on i386 and amd64.

> The problem is that the size of long is archtecture-specific, and that
> there is no proper error handling.  apt is not affected by this.
>
> This appears to be a dpkg bug, filed as #580038.

An epoch is defined as

epoch
This is a single (generally small) unsigned integer. It may be
omitted, in which case zero is assumed. If it is omitted then the
upstream_version may not contain any colons.

Lets remove the "generally" from policy so we can truely declare this
case as insane.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8739y8bce8@frosties.localdomain

Re: Bug#574956: dpkg drops zero-epoch in status file

[Goswin von Brederlow]

Guillem Jover  writes:

> Hi!
>
> On Fri, 2010-04-30 at 12:56:08 +0200, David Kalnischkies wrote:
>> The problem:
>> To compare versions with the same version number apt generates
>> a hash over a few informations which are available online and
>> in dpkgs status file: all dependencies and the installation size.
>> (as these are likely to change if this is another version)
>
> [...]
>
>> we can see that dpkg in his status file drops the zero-epoch.
>> This doesn't change the semantic as zero-epoch and no epoch are
>> considered equal, but as apt uses the string without deeper
>> parsing at this stage this is a big difference in the hash.
>> ( can be seen in apt-pkg/deb/deblistparser.cc VersionHash() )
>
> The same problem arises with non-significant zeros before digits, for
> example:
>
>   0.001 == 0.1 == 00:000.1
>
> Although this might be trickier to see in the wild, as dpkg itself
> would not normalize these versions, but an unknowing packager could
> generate those (somehow) thinking they are different.

I don't think 0.001 and 0.1 are identical. They should only compare as
equal. If you have 2 packages with those versions then that means you
have 2 different packages. So apt is totaly right in treating them
differently. But for apt to do that dpkg has to keep the version string
as it is even if it contains non-significant zeroes. The zeroes might
also be part of the upstream version so maintainers might not have much
control over it and the debian version should not differ from upstream.

>> Quick&Dirty workaround is to drop the zero-epoch in the package
>> so the Packages and status file are equal again. (cc'ed perl group
>> and last uploader, maybe they want to do that)
>>
>> The real solution is to either tell apt to strip out the zero-epoch for
>> the hash or to tell dpkg to not remove the zero-epoch in status.
>
>> It is relatively easy for apt to ignore the epoch for the hash as it is
>> a simple hash and we don't need to care about false positive removes
>> so apt still doesn't need to do expensive parsing here, but i want
>> to ask dpkg maintainers (cc'ed and titled to grep their attension)
>> for their opinion as this is maybe something they want to change
>> instead in their logic for consistence…
>> (through no zero epoch could be a change to be consistence…).
>
> I don't think dpkg should stop removing the zero-epoch, it would be
> redundant and confusing information. But it might be a good idea to
> make dpkg-gencontrol for example drop it on output. And it seems that
> apt might need to consider the other equivalent versions too.
>
> regards,
> guillem


On the other hand the epoch is 100% maintainer controlled. So Debian can
make a policy how they are to be used and treated. Since a zero epoch
indeed is redundant and confusing, to both users and application it
turns out :), I agree with you there. They should not end up in
debs. But I would go one step further. I would give an error if a
explicit zero epoch is being used. If they are not allowed in debs then
why allow them in source? They are just as confusing there and (bad)
maintainer might even wonder where their zero epoch disapeared to. If
you don't push the maintainers nose into it how will they ever learn?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874oirj46c@frosties.localdomain

Re: [PATCH/RFC] dpkg-shlibdeps: ignore shell scripts

[Goswin von Brederlow]

Raphael Hertzog  writes:

> Hi,
>
> On Fri, 23 Apr 2010, Jonathan Nieder wrote:
>> Goswin von Brederlow wrote:
>> 
>> > What is wrong with excluding only scripts, those files begining with a
>> > shebang token? The test for this is easy and you still get errors for
>> > other files that objdump doesn't understand.
>> 
>> Yes, I wonder the same thing.  What would be the harm in making
>> dpkg-shlibdeps always ignore files starting with a #! line?
>
> If we go down that path, then I would have no reason to refuse patches to
> ignore images, known document formats, any textual file, etc. This is
> useless bloat.

Images, doc and text files are not executables and dpkg-shlibdeps only
expects executables. There is your reason. :)

> I'm all for smartness but reasoned smartness. :-) In this case, we know
> what we're looking after, no need to use a blacklist approach.

If you only check ELF files then you don't get an error if there is an
image in usr/bin. It also becomes more complicated if/when other formats
get added.

The idea of blacklisting #! is that you only exclude known safe files
and still get errors on unknown files. You could combine the two:

if (has_shebang(file)) {
  ignore;
} else if (is_elf(file)) {
  process;
} else unknown_format_error;

I like the idea of supporting 'dpkg-shlibdeps .../bin/' but that could
be combined with the filtering.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87aasutsoj@frosties.localdomain

Re: [PATCH/RFC] dpkg-shlibdeps: ignore shell scripts

[Goswin von Brederlow]

Jonathan Nieder  writes:

> Hi again,
>
> Raphael Hertzog wrote:
>
>> That would be the wrong thing to check. We want to verify if it's a
>> ELF object and if not then we skip it. And we should not skip it silently
>> IMO as it was explicitly passed in a list of stuff to analyze.
>> 
>> We already have the required Dpkg::Shlibs::Objdump::is_elf().
>
> I went back to make this change but I decided I do not want it.  It
> would have brought on two regressions:
>
> 1. dpkg-shlibdeps currently supports non-ELF files as long as objdump
>does.  Of course all Debian architectures use ELF by default, but
>there are users for dpkg outside of Debian (fink on Mac OS X, for
>example).  Can we assume that all users of dpkg-shlibdeps use ELF
>objects exclusively?
>
> If we can, dpkg-shlibdeps could be simplified a lot by using readelf
> instead of objdump.  That’s something I would enjoy doing.
>
> 2. dpkg-shlibdeps currently complains if you pass it some random
>garbage.  Maybe it would be nice to have a separate accept-anything
>mode so you can throw your entire debian/tmp at it.  That’s not my
>itch.
>
> I was looking to ignore interpreted files because they are the only
> files other than object files that are supposed to be marked
> executable and placed in /usr/bin or /usr/lib/package.  This is better
> than suppressing all errors because errors are useful.
>
> Meanwhile I do not want to break other people’s workflows.  So I
> would be very interested in hearing concrete problems this imposes, so
> I can find a way to avoid breakage without unnecessary complication.
>
> Jonathan

What is wrong with excluding only scripts, those files begining with a
shebang token? The test for this is easy and you still get errors for
other files that objdump doesn't understand.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87633ky6ty@frosties.localdomain

Re: Bug#540215: Introduce dh_checksums

[Goswin von Brederlow]

Raphael Hertzog  writes:

> On Fri, 16 Apr 2010, Harald Braumann wrote:
>> On Thu, Apr 15, 2010 at 05:03:44PM +0200, Raphael Hertzog wrote:
>> 
>> > Even if it creates a checksum file, someone could always hand-edit the
>> > package to add files not listed in the checksum files and we need to
>> > decide whether that's something that needs to be catched and if yes by
>> > whom and at what point.
>> 
>> Do you mean a maintainer, who hand-edits a package after it was
>> built, or do you mean an adversery who has evil intentions? If the
>
> The latter.
>
>> former, then this should just be forbidden. If the latter, than this
>> can be solved by package signatures.
>
> Which one? We are discussing something that is a signature of the (content
> of the) package. And there's the signature on Release/Package which can
> authenticate the .deb in its entirety.
>
> I'm discussing the case where the signature of the "checksums" file is valid
> but that checksums file does not list all the files present in
> data.tar.gz or control.tar.gz.

The checksum file can be altered prior to the signature being added. But
so can any other part of the .deb file. We have to assume that no
adversery with evil intentions has access to the .deb prior to it being
signed. So it comes down to the maintainer not screwing up the package
prior to uploading.

The DAK can verify the validity of the signature and the completness of
the checksum file during upload if that is considered neccessary. I do
not think every user should have to do so during install. But it could
be optional with default off.

> Cheers,

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87fx2vl99k@frosties.localdomain

Re: Bug#540215: Introduce dh_checksums

[Goswin von Brederlow]

Harald Braumann  writes:

> On Thu, Apr 15, 2010 at 04:04:51PM +0200, Goswin von Brederlow wrote:
>
>> The checksum file could be attached as additional member in the
>> .deb. And a signature could be a signed file containing the checksum
>> size and name of all members of a .deb preceeding the signature. That
>> way the signature can verify the deb itself or individual members, like
>> the checksum file, in the .deb. Just a thought.
>
> I'm not sure, how you mean that exactly. But the signature must be
> over the checksum file, nothing more and nothing less. Otherwise
> you won't be able to verify the checksum file.

A signature could look like this:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

90d462d27ac404ecabfc9ca7f306dec0b81d3576 3456 control.tar.gz
ed43cc24b4f5472d25fc9c82a67daed317c8d415 3573458 data.tar.gz
90d462d27ac404ecab247a82a67daed317c8d415 971 checksum_control
ed43cc24b4f5472d25fc9ca7f306dec0b81d3576 1234 checksum_data
9528348234958345473658358238452836482685 3536 signature_01

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)

iD8DBQFLyHvbH8SBz+0NfPoRAofQAJoDlO38O3UqfcSyN6xj92s/LQlAzwCgweC2
BiK6lI0aABtTwvXVIEiqXNg=
=cOUY
-END PGP SIGNATURE-

> Also I think it's really a very bad idea in general to mix multiple
> different things into one signature. The one thing is a signature over
> installed files (via the checksum file). The other is a signature over
> a package. The two are completely orthogonal and serve different
> purposes.

It would be a signature over members of the .deb file. The meaning of
each member doesn't matter.

> harry

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87k4s7l9pj@frosties.localdomain

Re: Bug#540215: Introduce dh_checksums

[Goswin von Brederlow]

Raphael Hertzog  writes:

> Hi Wouter,
>
> I followed somewhat the discussion and I'm sorry for the delay
> answering but package signatures are quite low in priority
> in our roadmap currently:
> http://wiki.debian.org/Teams/Dpkg/RoadMap
>
> We would be glad however if someone would lead this project. I think this
> project would benefit from a DEP so that we have a spec to refer to and so
> that we can get buy-in from all other involved parties before going too far
> in the implementation.
>
> Otherwise it will be one more thread without clear outcome...
>
> On Tue, 23 Mar 2010, Wouter Verhelst wrote:
>> The idea would be to provide a path from a binary on disk to a GPG
>> signature for installed packages of which the user no longer has the
>> .deb file from which it was originally installed, nor the Packages
>> and/or Release.gpg file that was used to download it.
>
> Ok, it looks like a good goal.
>
>> The proposal as it currently stands would be that:
>> - md5sums are phased out in favour of a more generic 'checksums' file
>>   that would use a more secure hash algorithm than MD5 (one of the SHA*
>>   family of hashes would probably do at the moment).
>> - When asked to sign a .deb file, dpkg(-deb) would extract the checksums
>>   data member from control.tar.gz, create a detached signature for that
>>   file, and store it as an additional data member in that .deb.
>
> OK, but additional data members in .deb are ignored by dpkg like you
> noted, this also means that it doesn't end up on the disk after unpack.
>
> I assume your proposal is to modify dpkg to extract those and store them
> somewhere. Is that true? In that case, where and under which format?
>
> You have mentioned multiple signatures per package, would they be stored
> in multiple ar member or in a single one?
>
> My only wish at this point is to avoid exploding the number of
> small administrative files in /var/lib/dpkg/info/ due to this new feature.

The introduction of multiarch will need to change the way metadata is
stored there. Since some change is needed anyway it might be a good time
to adapt to the increase in files stored there and use some subdirs.
Maybe even have one dir per package.

> The biggest downside in your approach is that it's somewhat painful to
> ensure that all the content of the package is signed. If the checksums
> files is incomplete, what is supposed to happen? Is that something that
> dpkg should take care of or should that be outside the scope of dpkg?

Yes, dpkg should create the checksum file.

The checksum file could be attached as additional member in the
.deb. And a signature could be a signed file containing the checksum
size and name of all members of a .deb preceeding the signature. That
way the signature can verify the deb itself or individual members, like
the checksum file, in the .deb. Just a thought.

>> The benefits of this method as opposed to storing the signature in the
>> control.tar.gz file would be:
>> - Autobuilders would not need to be modified to support signed packages.
>> - Adding a member to an ar file and removing it again later can be done
>>   in a way which is idempotent; the same is not true for modifying a tar
>>   file. As such, it is trivially possible to remove signatures from a
>>   .deb file to allow its verification against the original .changes file
>>   that was used for its upload, should this at any point be necessary
>> - If adding multiple signatures is possible in a way which does not
>>   modify the contents of the .deb file itself, then the archive-wide key
>>   could in theory be used to sign individual .deb files. While a massive
>>   increase of CPU power on ftp-master.d.o would be needed to support
>>   this, it would allow for easier key management on the end-user end.
>> - It would not break backwards compatibility. I tried this, by manually
>>   adding a file "signature_01.asc" to a .deb file; dpkg was still able
>>   to install this package, it just ignored the unknown file.
>
> The biggest concern I have with modifying the .deb multiple times along
> the path is the fact that we're changing the checksum of the whole file
> and we're not changing its name. Changing this invariant will lead to
> problems. On the other hand, adding a signature should not require
> changing the filename either.
>
> Did ftpmaster (and buildd maintainers) comment on this problem?
>
>> the metadata and maintainer scripts in the control.tar.gz file. Doing
>> this would require some way to clarify the difference between data in
>> control.tar.gz and data in data.tar.gz; current suggestions are to
>> either use a prefix (something like CONTROL:preinst, for instance) or to
>> use the path of the binary-as-installed (wherein the above would become
>> "/var/lib/dpkg/info/.preinst"). There aren't any strong
>> feelings towards one or the other, however.
>
> I prefer the prefix approach because the other one is hardcoding internal
> information that is not guaranteed to be stable bet

Re: md5sums files

[Goswin von Brederlow]

Bernd Zeimetz  writes:

> Philipp Kern wrote:
>> On 2010-03-03, Wouter Verhelst  wrote:
>>> This is where I disagree. When a checksum algorithm is compromised (and
>>> MD5 *is* compromised), things only ever get worse, not better. Indeed,
>>> MD5 preimage attacks are pretty hard *today*. But switching to something
>>> more secure in preparation for the day when MD5 will be easily cracked
>>> by every script kiddo around is *not* overkill.
>> 
>> Sure, but to be honest, not even all packages managed to generate md5sums
>> 'till now (with some quite core, omnipresent packages missing) so it seems 
>> out
>> of scope for squeeze.  Maybe squeeze+1.
>
> I think its about time to require to generate checksums for packages and make
> all packages which do not do so RC buggy.

If a checksum file becomes required then it really is not the job of the
package to build it. Instead dpkg should generate one and include it
automatically. And given the widespread generation of md5sum files that
really should be automated anyway.

This would go nicely with changing the checksum algorithm. Just prepare
a patch for dpkg to generate a sha256sum file automatically when it
builds a deb and then packages can stop generating md5sum files over
time.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87aaumzjiy@frosties.localdomain

Re: Can't externally acquire lock on /var/lib/apt/lists/lock

[Goswin von Brederlow]

Francesco Pretto  writes:

> Hi all, newbie here.
>
> What I'm trying to accomplish is offline updating packages lists in
> /var/lib/apt/lists. As far as I know, there's no supported way of
> doing this with apt tools, so I'm trying to do so with a bash script.
> Assuming that no synaptic, aptitude, apt-get, etc. process is running,
> doing the following will just works (assuming you have saved
> /var/lib/apt/lists folder in lists.tar.bz2):
>
> rm -r /var/lib/apt/lists
> tar xvjpf lists.tar.bz2 -C /var/lib/apt
>
> I realize this is ugly and unsafe, so I'd like to write the same in a
> cleaner way, taking the lock on /var/lib/apt/lists/lock first,
> similarly to what apt-get update and other tools do. Surprisingly, it
> wasn't so easy as I tough, and the following bash script doesn't work
> as I expected:
>
> --
> #!/bin/bash
> set -e
> (
> flock --exclusive --nonblock 200
> # do something inside /var/lib/apt/lists
> sleep 50 # Taking some time to test if the lock works
> ) 200>/var/lib/apt/lists/lock
> --
>
> It doesn't work in the sense that trying to launch apt-get update
> during the sleep time will simply works, like no lock was set. What am
> I doing wrong?
>
> I hope apt-get is not actually checking which process is taking the
> lock and failing just if the lock was taken by apt-get: I think it
> would not be a good design and would mean that my task, offline
> updating apt packages lists with a provided archive, can't be safely
> implemented.
>
> Any idea, workaround? Thanks a lot.
>
> Greetings,
> Francesco

Not tested and off the top of my head:

apt-get --no-download -o APT::Update::Pre-Invoke::="find /var/lib/apt/lists 
-type f -a ! -name lock && tar xvjpf lists.tar.bz2 -C /var/lib/apt" update


- Invoking apt-get takes the lock so that is out of the way.

- --no-download prevents apt-get from going online and downloading Index files.

- APT::Update::Pre-Invoke is run before updating the index
  files. Although since you download nothing Post-Invoke would work
  too. But maybe something else wants to run then already.

MfG
Goswin

PS: Why delete files prior to unpacking? Just unpack the new ones and
let apt clean up any extra files like it already does. That also allows for 
partial updates.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: webkitkde_0.9svn1080813-1_i386.changes REJECTED

[Goswin von Brederlow]

Adrian von Bidder  writes:

> On Thursday 04 February 2010 18.38:45 Goswin von Brederlow wrote:
>
>
>> Are you using pristine-tar? mercurial-buildpackage?
>
> Very funny ...
>
> I used to use hg-buildpackage, but as that was discontinued, I wrote my own 
> script for the few operations I did.  Obviously, including bugs that hg-
> buildpackage probably didn't have...

Sounded like it so I thought to mention it. :)

> It's only now that I discover that hg-buildpackage wasn't really 
> discontinued, but renamed.  (Or re-implemented?)
>
> Grrr.
>
> cheers
> -- vbi

Afaik someone else missed the discontinued hg-buildpackage and uploaded
something equivalent as mercurial-buildpackage. The changelog says:

  * Initial package.  It is a redesign of John Goerzen's hg-buildpackage.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: webkitkde_0.9svn1080813-1_i386.changes REJECTED

[Goswin von Brederlow]

Adrian von Bidder  writes:

> On Thursday 04 February 2010 09.40:54 Raphael Hertzog wrote:
>> On Wed, 03 Feb 2010, Adrian von Bidder wrote:
>> > Ah.  I need to modify my script to generate the orig tarball, then. 
>> > Thanks, I've not noticed this.
>> >
>> > Q: why doesn't dpg-source complain when I try this here?  Why didn't I
>> > notice when I've run the package through pbuilder?
>> 
>> Because quilt push returning 2 should not be considerer an error. You can
>> consider it an error from the older dpkg-source.
>
> Ok.
>
>> Still, an upstream
>>  tarball should not contain the debian patches pre-applied.
>
> Yes, as I said, this is clearly my error (corrected and re-uploaded now, and 
> corrected my script to import new verisions to hg and generate orig tarball, 
> so it shouldn't happen again.)
>
> Anyway, thanks for your help
>
> cheers
> -- vbi

Are you using pristine-tar? mercurial-buildpackage?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: webkitkde_0.9svn1080813-1_i386.changes REJECTED

[Goswin von Brederlow]

Adrian von Bidder  writes:

> Hi all,
>
> I got a reject on a webkit upload, ftpmasters suggested I should ask you:
>
> On Thursday 28 January 2010 23.01:04 Torsten Werner wrote:
> [... me pointing out that dpkg-source from lenny worksforme™ ...]
>> but it does not work on ftp-master.debian.org for unknown reasons. You
>> might want to ask the dpkg developers.
>
> Original reject attached, the pertinent part is:
>
> 'dpkg-source -x' failed for webkitkde_0.9svn1080813-1.dsc [return code: 2].
>  [dpkg-source output:] dpkg-source: warning: -sn is not a valid option for 
> Dpkg::Source::Package::V3::quilt

The actual failure is not the -sn option, but:

>>>  [dpkg-source output:] dpkg-source: failure: quilt --quiltrc
>>>  /dev/null push debian-changes gave error exit status 2

So the question is why did the patch not apply?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Odd patch behaviour [Was: quilt 3.0 source format and dpkg-source/dpkg-buildpackage]

[Goswin von Brederlow]

tangke  writes:

> On 2010年01月20日 23:39, Goswin von Brederlow wrote:
>> Norbert Preining  writes:
>>
>>
>>> On Mo, 28 Dez 2009, Raphael Hertzog wrote:
>>>  
>>>>> Mind that git-buildpackage with normal 1.0 source format does NOT pollute
>>>>> the git repository, so my expectation is that the 3.0 format does the
>>>>> same, but alas, it doesn't.
>>>>>  
>>>> Well, if you have the usual quilt rules, you working copy is also modified
>>>> until the next debian/rules clean. Here the clean doesn't unapply the
>>>>
>>> I disagree:
>>>
>>> $ ls ~/.gbp.conf
>>> ls: cannot access /home/norbert/.gbp.conf: No such file or directory
>>> $ git status
>>> # On branch master
>>> nothing to commit (working directory clean)
>>> $ git-buildpackage -us -uc -rfakeroot -S
>>> ...
>>> dpkg-buildpackage: source only, diff-only upload (original source NOT 
>>> included)
>>> Now running lintian...
>>> Finished running lintian.
>>> $ git status
>>> # On branch master
>>> nothing to commit (working directory clean)
>>> $
>>>
>>> So please tell me *what* has changed?
>>>
>>> Doing the same with source format 3.0 git status reports hundreds
>>> of differences.
>>>
>>> So please do not spread things that are not correct!
>>>  
>> He did say that with 3.0 (quilt) format the "clean" state is with
>> patches applied, like you get after dpkg-source -x. That part is
>> different to 1.0 format where "clean" would have patches unapplied.
>>
>> So when you go from 1.0 to 3.0 you will get a ton of files changed
>> after the first build + clean because now patches are applied.
>>
>> MfG
>>  Goswin
>>
> why not apply the patches when build automatically, and make clean to 
> unapply the patches?
> when I build the package, I must check quilt applied and quilt 
> unapplied, It's very trouble.
>
> thanks
>
> best regards.
>
> ta...@lemote.com

I was under the impression that build would always apply all missing
patches now that dpkg-source always creates a .pc dir. It certainly
does apply patches when none are applied.

% apt-get source mdadm
Reading package lists... Done
Building dependency tree   
Reading state information... Done
Need to get 469kB of source archives.
Get:1 http://chocos sid/main mdadm 3.0.3-2 (dsc) [1261B]
Get:2 http://chocos sid/main mdadm 3.0.3-2 (tar) [337kB]
Get:3 http://chocos sid/main mdadm 3.0.3-2 (tar) [131kB]
Fetched 469kB in 0s (1831kB/s)
dpkg-source: info: extracting mdadm in mdadm-3.0.3
dpkg-source: info: unpacking mdadm_3.0.3.orig.tar.gz
dpkg-source: info: unpacking mdadm_3.0.3-2.debian.tar.gz
dpkg-source: info: applying all patches with quilt push -q 
debian/conffile-location.diff
Applying patch fixes/udev-blkid.diff
Applying patch contrib/docs/raid5-vs-raid10.diff
Applying patch contrib/docs/superblock_formats.diff
Applying patch contrib/docs/md.txt.diff
Applying patch contrib/docs/jd-rebuilding-raid.diff
Applying patch debian/conffile-location.diff
Now at patch debian/conffile-location.diff

% cd mdadm-3.0.3 
% quilt pop 
Now at patch contrib/docs/jd-rebuilding-raid.diff
% debuild -us -uc
 dpkg-buildpackage -rfakeroot -D -us -uc -ICVS
dpkg-buildpackage: set CFLAGS to default value: -g -O2
dpkg-buildpackage: set CPPFLAGS to default value: 
dpkg-buildpackage: set LDFLAGS to default value: 
dpkg-buildpackage: set FFLAGS to default value: -g -O2
dpkg-buildpackage: set CXXFLAGS to default value: -g -O2
dpkg-buildpackage: source package mdadm
dpkg-buildpackage: source version 3.0.3-2
dpkg-buildpackage: source changed by martin f. krafft 
dpkg-buildpackage: host architecture amd64
 fakeroot debian/rules clean
dh_testdir
dh_testroot
rm -f debian/bugscript debian/mkconf
rm -f build-stamp
[ ! -f Makefile ] || /usr/bin/make clean
make[1]: Entering directory `/tmp/mdadm/mdadm-3.0.3'
rm -f mdadm mdmon mdadm.o config.o mdstat.o  ReadMe.o util.o Manage.o 
Assemble.o Build.o Create.o Detail.o Examine.o Grow.o Monitor.o dlink.o Kill.o 
Query.o Incremental.o mdopen.o super0.o super1.o super-ddf.o super-intel.o 
bitmap.o restripe.o sysfs.o sha1.o mapfile.o crc32.o sg_io.o msg.o 
platform-intel.o probe_roms.o mdmon.o monitor.o managemon.o util.o mdstat.o 
sysfs.o config.o Kill.o sg_io.o dlink.o ReadMe.o super0.o super1.o 
super-intel.o super-ddf.o sha1.o crc32.o msg.o bitmap.o platform-intel.o 
probe_roms.o pwgr.o core *.man \
mdadm.tcc mdadm.uclibc mdadm.static *.orig *.porig *.rej *.alt \
mdadm.Os mdadm.O2 \
mdassemble mdassemble.static mdassemble.auto mdassemble.uclibc \

Re: dpkg feature implementation

[Goswin von Brederlow]

"dE ."  writes:

> On Wed, Jan 6, 2010 at 6:31 AM, Steve Langasek  wrote:
>> On Tue, Jan 05, 2010 at 06:59:40PM +0530, dE . wrote:
>>> Problem is you have to make these DVD/CD, or in general storage media.
>>> Windows people are not willing to do that...they just want click and
>>> install.
>>
>> You want people to be able to install packages without a network, and
>> without using DVDs/CDs.
>>
>> What, pray, are you intending them to click on if it's neither media nor a
>> network resource?
>>
>
> What about a single package which was brought on a pendrive and made
> to install on just clicking it?...gdebi style.
>
> I've repeated this many times now.

That pendrive contains the install button and an iso image with the
repository. When you click at the install button it mounts the iso,
runs apt-cdrom, and apt-get install and cleans up again.

Problem solved.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Enhancing 3.0 (git) source package format

[Goswin von Brederlow]

Tollef Fog Heen  writes:

> ]] Goswin von Brederlow 
>
> | Tollef Fog Heen  writes:
> | 
> | > ]] Goswin von Brederlow 
> | > | My feeling is that 3.0 (git) format adds bloat to the source packages
> | > | that hardly anyone ever uses, makes it that much harder for any
> | > | non-git user to edit the source and is of little extra value when the
> | > | maintainers git is month or years further along.
> | >
> | > Even if the upstream VCS has moved on, you save a bit of bandwidth by
> | > having something that comes with half the history, even if you don't
> | > have all of it.
> | 
> | Weigh that against the bandwidth spend for mirrors and for people that
> | do not need or want the history and the extra cost in terms of needing
> | more CD/DVD images to contain a source snapshot. Also the cost for
> | snapshot.debian.org having to have the extra bloat for every single
> | version uploaded. For a worst case take linux-2.6 as example.
>
> If we (as I do) consider history part of the source, that size increase
> is irrelevant.
>
> | Also why would you download the source package in the first place if
> | what you really want is a git checkout. The extra bandwidth for a git
> | checkout would only be as much as the 3.0 (git) format would lack in
> | history.
>
> Because I want to add a patch that changes a behaviour in a stable
> package, and I want to add that patch in a way that gives me the least
> work, both when writing it, but also when bringing it forward.  Also, my
> mirror might be local; git.d.o and random upstream repositories
> certainly are not.
>
> | My expectation is also that I can "apt-get source foo", edit some
> | files and debuild without having to learn a new tool and completly
> | foreign workflow. The various patch systems used with 1.0 packages
> | destroy that somewhat but 3.0 (quilt) restores that feature again. 3.0
> | (git) on the other hand goes in the wrong direction as it makes the
> | package even more special.
>
> I'm trying to come up with a reasonable workflow rather than getting
> entangled in what intricasies of the different source package formats.
> At the moment, what I want is best done with a bundle in debian/ and a
> 3.0 (native) package and a README.source.
>
> | But in the end it comes down to taste I guess. Do you want to force
> | people to use git or are you friendly to those that don't use it?
> | So I will shut up now before we go around the circle again.
>
> I don't see why you think «ship the history with the package» (which is
> what I want to do) implies that you can't do apt-get source foo ; cd
> foo-* ; hack.

It currently implies that. The 3.0 (git) format, last I tried, is not
transparent to the user like 3.0 (quilt) is.

I'm not saying it can't be made equaliy transparent but you will have
your work cut out there.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Enhancing 3.0 (git) source package format

[Goswin von Brederlow]

Tollef Fog Heen  writes:

> ]] Goswin von Brederlow 
> | My feeling is that 3.0 (git) format adds bloat to the source packages
> | that hardly anyone ever uses, makes it that much harder for any
> | non-git user to edit the source and is of little extra value when the
> | maintainers git is month or years further along.
>
> Even if the upstream VCS has moved on, you save a bit of bandwidth by
> having something that comes with half the history, even if you don't
> have all of it.

Weigh that against the bandwidth spend for mirrors and for people that
do not need or want the history and the extra cost in terms of needing
more CD/DVD images to contain a source snapshot. Also the cost for
snapshot.debian.org having to have the extra bloat for every single
version uploaded. For a worst case take linux-2.6 as example.

Also why would you download the source package in the first place if
what you really want is a git checkout. The extra bandwidth for a git
checkout would only be as much as the 3.0 (git) format would lack in
history.

For me a debian source package is a snapshot of a particular version
of the source that exactly coresponds to the binary packages build
from it. For history there is the changelog, old source packages on
snapshot.debian.org (soon again) and cvs/svn/hg/git/arch repositories.

My expectation is also that I can "apt-get source foo", edit some
files and debuild without having to learn a new tool and completly
foreign workflow. The various patch systems used with 1.0 packages
destroy that somewhat but 3.0 (quilt) restores that feature again. 3.0
(git) on the other hand goes in the wrong direction as it makes the
package even more special.


But in the end it comes down to taste I guess. Do you want to force
people to use git or are you friendly to those that don't use it?
So I will shut up now before we go around the circle again.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Enhancing 3.0 (git) source package format

[Goswin von Brederlow]

Tollef Fog Heen  writes:

> ]] Goswin von Brederlow 
>
> | Remove 3.0 (qit) format completly. A full history bloats the source
> | and a cut down history has no advantage over a plain source.
>
> I think this would be a mistake.  I'm increasingly considering a full
> git clone the source of my software (in the «preferred form of
> modification») and so having dpkg able to natively consume and produce
> that preferred form would be very useful.

Isn't your prefered form to git clone/uppdate/merge from the actual
repository used by the maintainer?

A copy of the maintainers repository as it was 2 years ago (think
stable) is not as usefull as a fresh clone.

> | It is far more usefull to put the url for the real git repository into
> | the control file. The number of people wanting to use git (instead of
> | just a snapshot of the source) but unable to git clone from the
> | internet is really miniscule and does not warrant the bloat for
> | everybody else.
>
> I don't see why you would do one or the otherrather than both?  Being
> able to do git log and such on the result of an apt-get-ed package is
> quite useful, without having to wait for a random upstream git
> repository that might not be available.

Is it? Do you actualy do that?

And we do have git.debian.org with high availability. It should not be
a random upstream git that might or might not be there. Make it a
reliable repository that will be there.

My feeling is that 3.0 (git) format adds bloat to the source packages
that hardly anyone ever uses, makes it that much harder for any
non-git user to edit the source and is of little extra value when the
maintainers git is month or years further along.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Enhancing 3.0 (git) source package format

[Goswin von Brederlow]

Johan Kiviniemi  writes:

> Hi,
>
> It seems the current format of the 3.0 (git) style source packages
> could be improved somewhat.
>
> Currently, dpkg-source seems to create a tarball of the .git tree
> under the working directory (with some exclusions).
>
> This leads to the following issues:

Artifically bloated snapshots of the developers git repository in the
debian source that are 99.99% useless.

> • Git keeps unreferenced objects around in the local repository for a
> while in case you want to undo an accidental deletion of e.g. a commit
> or a branch. These deleted objects end up in the generated source
> tarball (I checked).
>
> • Git is good at creating an efficiently packed archive of objects. By
> just creating a tarball of objects in whatever order they appear under
> .git/objects and gzipping it, you lose the advantage of having e.g.
> various revisions of the same file next to each other when
> compressing. In addition, Git does delta encoding between file
> revisions.
>
> • Extra files, such as COMMIT_EDITMSG, hooks and config are added to
> the source package. It appears this may be intentional, since they
> aren’t excluded by dpkg-source -b, but should they *really* go into
> the source package? Whenever your real Git repository is cloned, they
> aren’t retained anyway.
>
> In addition, dpkg-source explicitly has to make the hooks
> non-executable for security reasons. Might as well not include them in
> the first place in my *humble* opinion.
>
> My proposal:

Remove 3.0 (qit) format completly. A full history bloats the source
and a cut down history has no advantage over a plain source.

It is far more usefull to put the url for the real git repository into
the control file. The number of people wanting to use git (instead of
just a snapshot of the source) but unable to git clone from the
internet is really miniscule and does not warrant the bloat for
everybody else.

Just my 2c,
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: How to update Debian repository not connected to internet ?

[Goswin von Brederlow]

Jarek  writes:

> Hi all!
>
>   Is there any way to transfer repository updates to system not connected
> to internet ?
>   I need to keep up to date repository in my system, but I can transfer
> updates only with CD-R/DVD-R.

If you know a little bit perl you can modify debmirror for this (or
file a wishlist bug). Instead of downloading make it print what it
wants to download into a script and then wget/curl those files where
you do have access.

Alternatively you could modify reprepro.

Both would work best if you have a little internet access to at least
update the Packages.gz files. Otherwise you will allways need 2 round
trips, one for Packages.gz, the other for debs.


If you only want to update a system there is also apt-zip. You can
probably run apt-zip on multiple systems and combine the output too.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: [PATCH 2/2] Reimplement dpkg-divert in C

[Goswin von Brederlow]

Mikhail Gusarov  writes:

> Twas brillig at 02:05:47 17.10.2009 UTC+02 when guil...@debian.org did gyre 
> and gimble:
>
>  >> +varbufaddstr(ret, "any ");
>  >> +
>  >> +varbufaddstr(ret, "diversion of ");
>  >> +varbufaddstr(ret, file);
>  >> +
>  >> +if (divertto) {
>  >> +varbufaddstr(ret, " to ");
>  >> +varbufaddstr(ret, divertto);
>  >> +}
>  >> +
>  >> +if (package && strcmp(package, ":") != 0) {
>  >> +varbufaddstr(ret, " by ");
>  >> +varbufaddstr(ret, package);
>  >> +}
>
>  GJ> Hmm, this makes translators work pretty hard. Also carried over from
>  GJ> the perl code, but as this is getting rewritten let's get it right.
>
> Is --list supposed to be only human-readable? Then we can change output
> format a bit to make it more localizable (12 similar strings like "local
> diversion of %s to %s by %s", "diversion of %s", "any diversion of %s by
> %s"... are localizable, but insane).

I think some maintainer scripts use --list. But I think they only care
wether there is a diversion. Maybe a --test for testing if something
is already diverted by a previous version and --list for users would
be better.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: [PATCH] hrmib cache for dpkg

[Goswin von Brederlow]

Anders F Björklund  writes:

> Goswin von Brederlow wrote:
>
>>> I made a quick port of the "HR-MIB cache" for net-snmp of rpm,
>>> so that it can be used with the dpkg packaging system as well.
> ...
>>> PS. It can also be used for doing bash-completion of packages.
>>> There are patches for this available elsewhere on the net.
>>
>> A few points:
>>
>> 2) If versions are taken without epochs then versions can collide and
>> they won't work well.
>
> Is it normal to have more than one package of the same name installed ?
> I thought it would error on the versions already, let alone the epochs.
>
> But if you do have the same name-version installed more than once then
> it would only show up once and removing one might delete the file,
> right.

I was thinking more along the line of 2 packages with the same version
but different epoch and upgrading from one to the other. If removing a
package removes the cache file then the old package would remove the
new packages file.

>> 3) Each new version creates a new file. Who cleans up obsolete files?
>> Or is the "cache" supposed to grow and grow and grow?
>
> If you remove the package, the file is supposed to get deleted too...
> (the files are 0 bytes in size, so it doesn't grow by all that much)
>
> It's of course easy enough to just delete the entire directory and
> recreate it from the current dpkg package list if anything goes wrong.
>
>> 4) /var/log/dpkg seems to also alraedy have the info.
>
> It was missing for some packages, and old log files were rotated away ?
> But it wasn't a log, more of a exported information directory (cache)
>
> Grepping through the log output is probably worse than grepping status,
> which is already twice as slow for bash-completion than checking a file.
>
> --anders

Not sure what my zsh does but the first time I  apt-get it takes
a second or two and then it has it cached.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Rewrite of dpkg-divert to C

[Goswin von Brederlow]

Mikhail Gusarov  writes:

> This patchset adds testsuite for dpkg-divert which is passed by
> Perl implementation of dpkg-divert and then replaces Perl
> implementation by C implementation which passes the same
> testsuite.

Since you seem to be interested in diversion handling let me bring up
an old idea of mine. Hopefully you will implement this.

I would like diversions (and alternatives actualy) to be handled
automagically by dpkg directly instead of in maintainer scripts. The
handling of diversions is nearly allways the same in all packages and
easy to screw up.

Automagic diversions in dpkg


1) dpkg-divert remains for old packages and cases where the automagic
way is insufficient.

2) new file DEBIAN/diversions

This file lists the diversions of a package, one file per line.
(optionaly 'file new-name'?)

3) dpkg automagic diversion handling

During preinst phase dpkg parses DEBIAN/diversions and runs
  dpkg-divert --package  --rename --add 
for every entry (optionally --divert new-name).

During postrm phase dpkg parses DEBIAN/diversions and runs 
  dpkg-divert --package  --rename --remove  
for every entry (optionally --divert new-name). 



Similary alternatives should be listed in DEBIAN/alternatives or a
common file and syntax could be used.

MfG
Goswin

PS: "runs dpkg-divert" preferably means does the equivalent directly.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Bug#274747: dependency on dpkg-dev is not required for provides

[Goswin von Brederlow]

Aurelien Jarno  writes:

> On Mon, Oct 12, 2009 at 01:05:18PM +0200, Guillem Jover wrote:
> Another option is that all the Provides: part of type-handling is done
> on dpkg. The current problem with the implementation:
> - A rebuild of type-handling is needed each time a new architecture is
>   added to dpkg.
> - Provides do not always work as an alternative. This require for
>   example type-handling to be installed on the build daemons 
>   (technically it should be an Essential package for this to work).
>
> That's why I am not really for more usage of type-handling, which IMHO
> should simply disappear.

This gets even worse with multiarch.

Depending on the dpkg config (what archs to allow, and any multiarch
config will break something) the type-handling will break down.

Examples:

amd64 + i386: amd64 has not-i386 and i386 has not-amd64. It is kind of
strange that you are both i386 and not-i386 but that is multiarch for
you.

amd64 + kfreebsd-amd64: Now you have linux and not-linux.

I already saw some nice confusing effects of this with ia32-apt-get
and had to blacklist type-handling from the non-native archs.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: [PATCH] hrmib cache for dpkg

[Goswin von Brederlow]

Anders F Björklund  writes:

> Hi all,
>
> I made a quick port of the "HR-MIB cache" for net-snmp of rpm,
> so that it can be used with the dpkg packaging system as well.
>
> What it does is that it keeps a simple directory of files in
> /var/cache/hrmib with one entry for each package, that has a
> timestamp of when the package was installed. So this directory
> can be used to get a package list, without needing db or grep.
>
> This is different from a normal cache dumped from cron or so,
> in that the package manager itself keeps it up-to-date at all
> times by adding and removing files as packages are modified.
> The initial cache can easilly be populated by e.g. touch(1).
>
>
> The files are named in a simple N-V-R.A style, without epochs.
> And for rpm, those "gpg-pubkey" packages aren't listed either.
>
> That is, the file list of the hrmib cache directory should be
> the same as the output of one of the following shell scriptlets:
>
> rpm -qa --queryformat="%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n" |
> grep -v ^gpg-pubkey- # remove public keys
>
> dpkg-query -W --showformat='${Package}-${Version}.${Architecture}\n' |
> sed -e 's/[0-9]\{1,\}://' # remove epochs
>
> The cache must first be populated with the currently installed
> packages, the code only makes sure that it is kept up-to-date.
>
> To get the installation date of each package, one can loop over
> the installed package names and touch the files with a date of:
>
> rpm -q --queryformat="%{INSTALLTIME:date}" $package
>
> ls -l --time-style="+%a %b %e %H:%M:%S %Y" /var/lib/dpkg/info/
> $package.list | cut -d\  -f6-10
>
>
> Attaching patch from the git repository, tried with a similar
> version for the dpkg 1.14 version present in Ubuntu 9.04...
>
> It is available in rpm since version 4.4.7, by uncommenting
> the %_hrmib_path macro in the /usr/lib/rpm/macros config...
>
> --anders
>
>
> PS. It can also be used for doing bash-completion of packages.
> There are patches for this available elsewhere on the net.

A few points:

1) If the $package.list file already contains all that information for
the current packages in its timestamp then what exactly is the use of
this? Seems more like a log than a cache.

2) If versions are taken without epochs then versions can collide and
they won't work well.

3) Each new version creates a new file. Who cleans up obsolete files?
Or is the "cache" supposed to grow and grow and grow?

4) /var/log/dpkg seems to also alraedy have the info.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Installing packages to a foreign system

[Goswin von Brederlow]

Phillip Susi  writes:

> On 10/7/2009 12:42 AM, Goswin von Brederlow wrote:
>> I use xen or kvm. With a chroot you can never be sure some maintainer
>> doesn't do something like this.
>>
>> But if you do see something of this kind I would report it as bug.
>
> Well, you can be sure it won't happen if you don't bind mount /sys,
> /proc, and /dev into the chroot.  But I guess that answers my question
> then; if a package fails to install because there is no /sys, /proc,
> and /dev, then it is broken and should be bug reported.

Nah, you can need those.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org