Re: Bug#862112: #862112: r-base-dev: Generate reproducible output independently of the build path
Sorry, was out traveling / running a relay. Now back. On 15 May 2017 at 17:12, Ximin Luo wrote: | Holger Levsen: | > control: found -1 3.4.0-1 | > control: notfound -1 3.4.0-1.0~reproducible2 | > | > Hi Dirk, | > | > I've asked Ximin to file this bug so that we have something to track and to | > refer in discussions… | > | > you wrote: | > | >> At work now with limited time, but I think I already told you twice that | >> there were two of the three parts of the patch you proposed to upstream that | >> I would not take, were I upstream (which I am not). | > | > Dirk, could you please point out (here in this bug report) which of the two parts | > you consider unsuitable? | > | | We talked about this over private email, this refers to the patch hunks in: | | - src/library/tools/R/admin.R | - src/library/tools/R/parseRd.R | | The suggestion was to guard them behind a command-line option using getOption, so presumably Debian could set this whilst upstream / other distros do not. Corret. As posted, the patch changes existing behaviour _unconditionally_ so I don't not think I can say with a straight face to upstream that they should take this. And also: you can turn them on for your builds, I won't at first, but advertise so that more users can test it. "Eventually" I may turn them on as well. The R "universe" is _at least_ the (currently around) 10600+ packages (yes, really, 10 thousand) on CRAN. The fact that we successfully rebuilt our 400+ is important aspect but nowhere near comprehensive enough. | However, since I'm not familiar with the full R codebase I was waiting to see if upstream had any further comments, because even with the getOption() change there might be other consequences that I didn't foresee. In this case it would be beneficial for Debian's behaviour to exactly match upstream, so we get the same bugs (if any). | | > Ximin, looking at https://anonscm.debian.org/cgit/reproducible/r-base.git/log/?h=pu/reproducible_builds | > I believe it would be best to merge those two (top most) commits into one? | > | >> A decent longer-term strategy may well to stress-test the patch by including | >> it in our builds for a while. We can work on that. | > | > actually we've been using Ximin's patches on tests.reproducible-builds.org | > since the 2nd of May on amd64 and since the 3rd on arm64, armhf and i386. | > | > According to https://tests.reproducible-builds.org/debian/index_performance.html | > (top row labeled "oldest build result) in the first table on the right) this | > means we've almost done a full rebuild with the patch on amd64+arm64 (probably | > 85% done now) and pretty close to that on i386… | > | > According to this, this patch seems to work well… | > | | The patch works well for getting stuff built, but I haven't tested all of | the *behaviour* of these packages. (Some probably have unit tests, but these | don't cover everything at any rate.) Exactly. | That is why I wanted to wait for upstreams' comments first, before adding the getOption guards - which are relatively minor IMO, compared to what the full patch does. Correct if I am wrong but you have not heard back from upstream, have you? [ That is not uncommon for posts on r-devel. Sometimes one needs to be persistent, and/or ally with an R Core maintainer. Which is pretty much what I suggested early one. ] | I also haven't tested other potential tools that might process R rdb files, who might for some reason expect absolute build paths. Correct as well. We are moving in the right direction, but we should not rush this. Upstream *is* sympathetic, they have taken an earlier 'repro build' patch. Dirk | X | | -- | GPG: ed25519/56034877E1F87C35 | GPG: rsa4096/1318EFAC5FBBDBCE | https://github.com/infinity0/pubkeys.git -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Repro Build report I do not understand
On 7 November 2015 at 00:11, Jérémy Bobbio wrote:
| Dirk Eddelbuettel:
| > | From the quick look I had, it seems symbols are sorted in a different
| > | order depending on the locale:
| > |
https://reproducible.debian.net/dbd/unstable/amd64/littler_0.3.0-2.debbindiff.html#r-cran-littler_0.3.0-2_amd64.deb/data.tar.xz/data.tar/./usr/bin/r/objdump%20--disassemble%20--full-contents%20{}
| > | (It's my guess because both are sorted by the second build has lowercase
| > | 'a' grouped together with uppercase 'A'.)
| > |
| > | The symbols match the ones in
| > | https://sources.debian.net/src/r-base/3.2.2-1/src/library/datasets/data/
| > |
| > | Hope that helps,
| >
| > That is very good too. Should I ensure a locale during the build? Any
other
| > heavy hand?
|
| You can try to set LC_ALL (or set LC_COLLATE and unset LC_ALL). But you
| might want to identify where the sorting happens first.
It is something else. Both littler (r-cran-littler) and RInside
(r-cran-inside) embed R, and both are irreproducible. I had overlooked how
RInside had failed this same issue and focused on the fact that I added
compilation-time timestamps (which I changed).
There is something else going on having to do with symbols from R. I did five
builds in quick succession, saving the deb file under different names. They
all ended up with slightly different file sizes (!!) which is REALLY weird.
Not sure where to go from here.
Dirk (who knows the R build system pretty well)
--
http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Repro Build report I do not understand
On 6 November 2015 at 23:44, Jérémy Bobbio wrote:
| Dirk Eddelbuettel:
| > One of my packages which still didn't build reproduciby is littler -- for
| > which I am upstream. I rewrote the build process, and even though it
produces
| > a small binary (which embeds R for use in #! scripts etc) it now ships as an
| > R package on CRAN. Which all build reproducibly.
| >
| > Yet I have this:
| >https://reproducible.debian.net/rb-pkg/unstable/amd64/littler.html
| >
| > And I don't understand the 'dbgsym' part. What turns that on? How can I
turn
| > it off?
|
| You might want to read the latest status update to learn about them:
| https://lists.debian.org/debian-devel/2015/08/msg00443.html
|
| But they are not the source of reproducibility, just a symptom.
|
| From the quick look I had, it seems symbols are sorted in a different
| order depending on the locale:
|
https://reproducible.debian.net/dbd/unstable/amd64/littler_0.3.0-2.debbindiff.html#r-cran-littler_0.3.0-2_amd64.deb/data.tar.xz/data.tar/./usr/bin/r/objdump%20--disassemble%20--full-contents%20{}
| (It's my guess because both are sorted by the second build has lowercase
| 'a' grouped together with uppercase 'A'.)
|
| The symbols match the ones in
| https://sources.debian.net/src/r-base/3.2.2-1/src/library/datasets/data/
|
| Hope that helps,
That is very good too. Should I ensure a locale during the build? Any other
heavy hand?
Dirk
--
http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Repro Build report I do not understand
On 6 November 2015 at 23:35, Vincent Bernat wrote: | ❦ 6 novembre 2015 14:03 -0600, Dirk Eddelbuettel <e...@debian.org> : | | > One of my packages which still didn't build reproduciby is littler -- for | > which I am upstream. I rewrote the build process, and even though it produces | > a small binary (which embeds R for use in #! scripts etc) it now ships as an | > R package on CRAN. Which all build reproducibly. | > | > Yet I have this: | >https://reproducible.debian.net/rb-pkg/unstable/amd64/littler.html | > | > And I don't understand the 'dbgsym' part. What turns that on? How can I turn | > it off? | | See man dh_strip. However, the correct action would be to find why there | is a difference in the debug symbols. The difference seems to stem from I was thinking that too but didn't test. As an R package, it installs into a 'tree' below /usr/lib/R/site-library/$PACKAGE but I also copy the resulting binary to /usr/bin/r. I think I need to strip that. Will the weird dbgsym then be gone? | the way some strings are sorted (see the end of the differences). They | come from the r-base package. I don't know why they are embedded into | debug symbols. Right. Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Repro Build report I do not understand
Howdy, One of my packages which still didn't build reproduciby is littler -- for which I am upstream. I rewrote the build process, and even though it produces a small binary (which embeds R for use in #! scripts etc) it now ships as an R package on CRAN. Which all build reproducibly. Yet I have this: https://reproducible.debian.net/rb-pkg/unstable/amd64/littler.html And I don't understand the 'dbgsym' part. What turns that on? How can I turn it off? Help much was appreciated, Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Timestamps from pdflatex -- almost but not quite there (?)
Gents,
I still have a few packages where the included pdf has a creation/mod time
stamp that differs. I am __almost__ at the point where I can override the
created pdf file:
- it works in a normal shell, ie the sequence
pdfinfo somefile.pdf
fixTimestamps.sh somefile.pdf 'timestamp_from_debian/changelog'
pdfinfo somefile.pdf
works: the file remains valid, and pdfinfo shows the before/after
timestamps as desired
- it does NOT work in pbuilder and I cannot narrow down why -- the second
pdfinfo call (added for debugging) reports a corrupted pdf file.
To be clear, I use the same exact helper script: it works 'on the shell' but
not in pbuilder. What am I missing?
The script-let (in its current form) is below.
Dirk
#!/bin/bash
##
## Override CreationDate and ModDate timestamps in pdf files
##
## Based in part on
##
https://wiki.debian.org/ReproducibleBuilds/TimestampsInPDFGeneratedByLaTeX
## as well as a fair amount of trial and error
##
## Dirk Eddelbuettel, September 2015
## Released under GPL-2 or later
set -u
set -e
if [ $# -lt 2 ]; then
echo "Usage: $0 file datestr"
exit 1
fi
FILE=$1
DEB_DATE=$2
#echo "Using arguments '${FILE}' and '${DEB_DATE}'"
## Meant for debian/ subdirectories containing file debian/changelog
#DEB_DATE=$(dpkg-parsechangelog -S date)
PDF_DATE=$(date -ud "${DEB_DATE}" +D:%Y%m%d%H%M%S-00\'00\')
#echo "Using PDF_DATE ${PDF_DATE}"
sed -i -e "s|^/CreationDate (.*)$|/CreationDate (${PDF_DATE})|" \
-e "s|^/ModDate (.*)$|/ModDate (${PDF_DATE})|" ${FILE}
MD5=$((echo "${DEB_DATE}" && du -b ${FILE}) | md5sum | cut -c -32)
sed -i "s|^/ID \\[\\(<[0-9A-F]\\{32\\}>\\) \\1]$|/ID [<${MD5}> <${MD5}>]|"
${FILE}
--
http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
