Re: misleading timestamps in binnmus

[Aurelien Jarno]

On 2016-11-10 11:33, Ian Jackson wrote:
> Cyril Brulebois writes ("Re: misleading timestamps in binnmus"):
> > Ian Jackson <ijack...@chiark.greenend.org.uk> (2016-11-09):
> > > What version of sbuild do buildds run ?  Ie, supposing that this is
> > > fixed in sbuild in stretch, will this be fixed on the buildds ?  Or do
> > > we need to update jessie, or what ?
> > 
> > sbuild on buildds uses a specific version of sbuild, for reasons I'm not
> > going to summarize. The base version is close to what's in jessie (see the
> > first lines of any build log which has “sbuild (Debian sbuild) 0.65.2”).
> ...
> > Repository seems to live under:
> >   https://apt.buildd.debian.org/
> 
> Thanks.  When Johannes has decided exactly what the sbuild patch looks
> like in sid, I will take a look at the repo there and backport the
> change.  In what form should I supply mhy update ?  As an source+all

When it's done, just ping us with the commit number, we will backport it
in our branch and we will deploy it on the build daemons.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Niko Tyni]

On Thu, Nov 10, 2016 at 10:34:33AM +, Holger Levsen wrote:
> On Thu, Nov 10, 2016 at 08:24:38AM -0200, Johannes Schauer wrote:
> > > I certainly hope it's part of the .buildinfo file as well, else, for
> > > reproducing binNMUs we would also need to store the .changes files in an
> > > easily accessable manner… (which we plan to do for .buildinfo files, but
> > > not for .changes files atm…)
> > 
> > for binary-only uploads, dpkg-genbuildinfo will add the Binary-Only-Changes
> > field to the .buildinfo file which contains the text of the last changelog
> > entry together with the maintainer name and date.
> 
> can someone please point at a real life/archive example of such a file?
> (a binNMU .changes file with Binary-Only-Changes field…)

That's in the .buildinfo file (not .changes), and I don't think they are
stored in the archive yet? But just try building a binNMU with sbuild
and look at the resulting .buildinfo. Something like

  sbuild --make-binNMU="test rebuild" -m"Niko Tyni " 
--binNMU=2 libxml-parser-perl_2.44-2

results in a .buildinfo file with

  Format: 0.1
  Source: libxml-parser-perl (2.44-2)
  Binary: libxml-parser-perl
  Architecture: amd64
  Version: 2.44-2+b2
  Binary-Only-Changes:
   libxml-parser-perl (2.44-2+b2) unstable; urgency=low, binary-only=yes
   .
 * Binary-only non-maintainer upload for amd64; no source changes.
 * test rebuild
   .
-- Niko Tyni   Tue, 05 Jul 2016 21:55:41 +0200
  Checksums-Md5:
 [...]

> I'm still confused, thinking that this Binary-Only-Changes field needs
> to be assembled into a file, called changelog.$arch, which is then put
> into the debian directory of the unpacked source package. (And which is
> then not included in the resulting binary packages…)

When asked to make a binNMU, sbuild will append an entry to
debian/changelog in the source, containing "binary-only=yes". During
package build, dh_installchangelogs (so debhelper not dpkg!) will
then extract that debian/changelog entry and install that in the
binary package as /usr/share/doc//changelog.Debian.,
separately from the rest of the changelog which goes to
/usr/share/doc//changelog.Debian. (This is done to not break
M-A:same coinstallability.)

As Johannes wrote, dpkg-genbuildinfo will also read debian/changelog in
the source and write out a corresponding Binary-Only-Changes field in the
resulting .buildinfo if the changelog entry contains "binary-only=yes".

To reproduce a binNMU from a .buildinfo file, one would need to parse
the Binary-Only-Changes field and extract the parts that needs to be
passed to sbuild. This currently seems rather fragile as noted by Ian
in #843773: the binNMU version needs to be parsed from the +bX notation,
and the message needs to be separated from the "Binary-only [...]" text
that's hardcoded in sbuild and might even change in the future. And then
there's the timestamp issue where I'll defer to others :)

But all that seems fixable on the sbuild side, and I think Johannes is
actively working on this stuff (thanks!)

Hope this clarifies a bit,
-- 
Niko Tyni   nt...@debian.org

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Ian Jackson]

Cyril Brulebois writes ("Re: misleading timestamps in binnmus"):
> Ian Jackson <ijack...@chiark.greenend.org.uk> (2016-11-09):
> > What version of sbuild do buildds run ?  Ie, supposing that this is
> > fixed in sbuild in stretch, will this be fixed on the buildds ?  Or do
> > we need to update jessie, or what ?
> 
> sbuild on buildds uses a specific version of sbuild, for reasons I'm not
> going to summarize. The base version is close to what's in jessie (see the
> first lines of any build log which has “sbuild (Debian sbuild) 0.65.2”).
...
> Repository seems to live under:
>   https://apt.buildd.debian.org/

Thanks.  When Johannes has decided exactly what the sbuild patch looks
like in sid, I will take a look at the repo there and backport the
change.  In what form should I supply mhy update ?  As an source+all
upload of sbuild, as if I were being sponsored ?  As a
git-format-patch against a git import of what I find there (or a
dgitish git branch) ?

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Holger Levsen]

On Thu, Nov 10, 2016 at 08:24:38AM -0200, Johannes Schauer wrote:
> > I certainly hope it's part of the .buildinfo file as well, else, for
> > reproducing binNMUs we would also need to store the .changes files in an
> > easily accessable manner… (which we plan to do for .buildinfo files, but
> > not for .changes files atm…)
> 
> for binary-only uploads, dpkg-genbuildinfo will add the Binary-Only-Changes
> field to the .buildinfo file which contains the text of the last changelog
> entry together with the maintainer name and date.

can someone please point at a real life/archive example of such a file?
(a binNMU .changes file with Binary-Only-Changes field…)

I'm still confused, thinking that this Binary-Only-Changes field needs
to be assembled into a file, called changelog.$arch, which is then put
into the debian directory of the unpacked source package. (And which is
then not included in the resulting binary packages…)

Maybe I'm just confused because this is news to me and I've never seen
nor heart about it.


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Johannes Schauer]

Hi,

Quoiting Holger Levsen (2016-11-10 07:48:33)
> On Thu, Nov 10, 2016 at 10:38:45AM +0100, Emilio Pozuelo Monfort wrote:
> > > I see. And this changelog.$arch is neither part of the source package,
> > > the .changes file nor the .buildinfo file, it's just included in the
> > > binary packages? Or is it also part of the .changes file?
> > It's in .changes as well. No idea if (any of it) is in .buildinfo
>  
> I certainly hope it's part of the .buildinfo file as well, else, for
> reproducing binNMUs we would also need to store the .changes files in an
> easily accessable manner… (which we plan to do for .buildinfo files, but
> not for .changes files atm…)

for binary-only uploads, dpkg-genbuildinfo will add the Binary-Only-Changes
field to the .buildinfo file which contains the text of the last changelog
entry together with the maintainer name and date.

cheers, josch


signature.asc
Description: signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Holger Levsen]

On Thu, Nov 10, 2016 at 10:38:45AM +0100, Emilio Pozuelo Monfort wrote:
> > I see. And this changelog.$arch is neither part of the source package,
> > the .changes file nor the .buildinfo file, it's just included in the
> > binary packages? Or is it also part of the .changes file?
> It's in .changes as well. No idea if (any of it) is in .buildinfo
 
I certainly hope it's part of the .buildinfo file as well, else, for
reproducing binNMUs we would also need to store the .changes files in an
easily accessable manner… (which we plan to do for .buildinfo files, but
not for .changes files atm…)


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Emilio Pozuelo Monfort]

On 10/11/16 10:33, Holger Levsen wrote:
> On Thu, Nov 10, 2016 at 10:01:55AM +0100, Emilio Pozuelo Monfort wrote:
>> These days, a changelog entry is added to a changelog.$arch. This is to avoid
>> problems when co-installing ma:same packages, as the changelog entries 
>> will/may
>> differ between different architectures.
> 
> I see. And this changelog.$arch is neither part of the source package,
> the .changes file nor the .buildinfo file, it's just included in the
> binary packages? Or is it also part of the .changes file?

It's in .changes as well. No idea if (any of it) is in .buildinfo

Cheers,
Emilio

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Holger Levsen]

On Thu, Nov 10, 2016 at 10:01:55AM +0100, Emilio Pozuelo Monfort wrote:
> These days, a changelog entry is added to a changelog.$arch. This is to avoid
> problems when co-installing ma:same packages, as the changelog entries 
> will/may
> differ between different architectures.

I see. And this changelog.$arch is neither part of the source package,
the .changes file nor the .buildinfo file, it's just included in the
binary packages? Or is it also part of the .changes file?


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Emilio Pozuelo Monfort]

On 10/11/16 00:53, Wouter Verhelst wrote:
> On Tue, Nov 08, 2016 at 10:41:09PM +, Ian Jackson wrote:
>> Is this a recommended recipe ?  AIUI a buildd doing a binnmu will not
>> modify the debian/changelog file.
> 
> Are you sure? When last I checked, this was not true (it may have
> changed since, however).

These days, a changelog entry is added to a changelog.$arch. This is to avoid
problems when co-installing ma:same packages, as the changelog entries will/may
differ between different architectures.

Cheers,
Emilio

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Cyril Brulebois]

Ian Jackson  (2016-11-09):
> What version of sbuild do buildds run ?  Ie, supposing that this is
> fixed in sbuild in stretch, will this be fixed on the buildds ?  Or do
> we need to update jessie, or what ?

sbuild on buildds uses a specific version of sbuild, for reasons I'm not
going to summarize. The base version is close to what's in jessie (see the
first lines of any build log which has “sbuild (Debian sbuild) 0.65.2”).

dsa-puppet.git has:
,---[ modules/debian-org/files/apt.preferences ]---
| …
| Package: sbuild
| Pin: release o=buildd.debian.org
| Pin-Priority: 500
| 
| Package: buildd
| Pin: release o=buildd.debian.org
| Pin-Priority: 500
| 
| Package: libsbuild-perl
| Pin: release o=buildd.debian.org
| Pin-Priority: 500
`---

Repository seems to live under:
  https://apt.buildd.debian.org/


KiBi.


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Wouter Verhelst]

On Tue, Nov 08, 2016 at 10:41:09PM +, Ian Jackson wrote:
> Is this a recommended recipe ?  AIUI a buildd doing a binnmu will not
> modify the debian/changelog file.

Are you sure? When last I checked, this was not true (it may have
changed since, however).

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Ian Jackson]

Thanks to everyone who has provided information.  I have summarised
it in #843773, against sbuild.

What version of sbuild do buildds run ?  Ie, supposing that this is
fixed in sbuild in stretch, will this be fixed on the buildds ?  Or do
we need to update jessie, or what ?

Ian.

-- 
Ian Jackson    These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Guillem Jover]

Hi!

On Wed, 2016-11-09 at 11:16:09 +, Ian Jackson wrote:
> Sven Joachim writes ("Re: misleading timestamps in binnmus"):
> > I'm afraid I don't really have a good suggestion.  Using current date
> > would work but obviously break reproducibility, and any other date seems
> > arbitrary.
> 
> I don't understand why using the current date would break
> reproducibility.
> 
> How does one reproduce a binnmu ?

As Mattia replied, by taking the data from the .buildinfo file:

> Clearly one needs a copy of the binnmu changelog stanza, because that
> (a) determines the version numbers of the generated .debs and (b) is
> actually included in the generated .debs.

man deb-buildinfo, Binary-Only-Changes field. :)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Ian Jackson]

(CCing reproducible-builds again:)

Sven Joachim writes ("Re: misleading timestamps in binnmus"):
> I'm afraid I don't really have a good suggestion.  Using current date
> would work but obviously break reproducibility, and any other date seems
> arbitrary.

I don't understand why using the current date would break
reproducibility.

How does one reproduce a binnmu ?

Clearly one needs a copy of the binnmu changelog stanza, because that
(a) determines the version numbers of the generated .debs and (b) is
actually included in the generated .debs.

There is no updated .dsc in the archive, but the .deb one is trying to
reproduce contains the very changelog fragment in question.  So the
procedure has to be to combine the archive's .dsc with the .deb's
binnmu changelog stanza into a new source package.

This ought to be be reproducible regardless of what date is in the
.deb's binnmu changelog stanza.  So I think the .deb's binnmu
changelog stanza can be the date of the build (or the date of the
binnmu request, or whatever is convenient).

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Guillem Jover]

Hi!

On Tue, 2016-11-08 at 22:41:09 +, Ian Jackson wrote:
> I see the python2.7 source package does this:
> 
>  LAST_CHANGE := $(shell dpkg-parsechangelog -S Date)
>  export BUILD_DATE := $(shell LC_ALL=C date -u +'%b %e %Y' -d 
> '$(LAST_CHANGE)')
>  export BUILD_TIME := $(shell LC_ALL=C date -u +'%H:%M:%S' -d 
> '$(LAST_CHANGE)')
> 
> Is this a recommended recipe ?

I think this should be fine. There's also SOURCE_DATE_EPOCH, that
dpkg-buildpackage honors and otherwise sets now, which can be also
retrieved with «dpkg-parsechangelog -STimestamp», but that should not
be needed here anyway, because…

> AIUI a buildd doing a binnmu will not
> modify the debian/changelog file.

… the changelogs do get a new entry, otherwise everything would fall
apart. Something like this:

,---
zlib (1:1.2.8.dfsg-2+b3) sid; urgency=low, binary-only=yes

  * Binary-only non-maintainer upload for amd64; no source changes.
  * Recompile static libraries with PIE

 -- amd64 / i386 Build Daemon (x86-grnet-01) 
  Sat, 16 Aug 2014 15:12:11 +0100
`---

But I just noticed that the timestamp in this entry seemed suspect,
and it indeed is, as the last maintainer upload was this:

,---
zlib (1:1.2.8.dfsg-2) unstable; urgency=low

  * Drop zlib-bin package as minizip has now been packaged separately,
delay due to lack of notice regarding upload (closes: #753070).

 -- Mark Brown   Sat, 16 Aug 2014 15:12:11 +0100
`---

> The result is that the file timestamps will be lies after a binnmu.  I
> think this is quite undesirable.  Less careful backup programs than
> mine wouldn't notice.  The results would then be a corrupted backup,
> which would break when restored.  There are probably other bad
> consequences.

So the actual problem is that the last timestamp gets reused for the
binNMUs, which seems totally bogus to me. This needs to be fixed in
whatever is injecting the binNMU entries on the buildds.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Ximin Luo]

Ian Jackson:
> I have just had my backups fail due to the following chain of events:
> 
>  * I had python2.7-stdlib:amd64 2.7.12-3 installed
>  * I did a backup
>  * I upgraded to python2.7-stdlib:amd64 2.7.12-3+b1 was built
>  * I did another backup, during which:
>  * Many files were not copied to the backup volume because
>their mtimes had not changed
>  * However, the contents of these files had changed because of the
>rebuild
>  * My backup system is clever enough to spot that there is a problem
> 
> I see the python2.7 source package does this:
> 
>  LAST_CHANGE := $(shell dpkg-parsechangelog -S Date)
>  export BUILD_DATE := $(shell LC_ALL=C date -u +'%b %e %Y' -d 
> '$(LAST_CHANGE)')
>  export BUILD_TIME := $(shell LC_ALL=C date -u +'%H:%M:%S' -d 
> '$(LAST_CHANGE)')
> 
> Is this a recommended recipe ?  AIUI a buildd doing a binnmu will not
> modify the debian/changelog file.
> 
> The result is that the file timestamps will be lies after a binnmu.  I
> think this is quite undesirable.  Less careful backup programs than
> mine wouldn't notice.  The results would then be a corrupted backup,
> which would break when restored.  There are probably other bad
> consequences.
> 
> Can we come up with a better recipe ?  For example perhaps
> 
>  DEBIAN_LAST_CHANGE ?= $(shell dpkg-parsechangelog -S Date)
> 
> which would make it possible for a buildd to override the value.
> 
> Thanks for your attention.
> 

Hi Ian, we have SOURCE_DATE_EPOCH for this. debhelper, CDBS, dpkg-buildpackage 
and /usr/share/dpkg/pkg-info.mk set this, and possibly others. The code you 
quote above might have been written before we introduced this though. Yes, it 
should be changed.

In the spec we explicitly state that processes should not set this variable if 
it is already set, so buildds doing binNMUs are free to override it with a 
later value.

https://reproducible-builds.org/specs/source-date-epoch/

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds