Bug#1069329: fixed in diffoscope 266

[Paul Wise]

On Wed, 2024-05-15 at 10:58 +0100, Chris Lamb wrote:

> Ah, I foolishly didn't check back with the original test case. The
> root cause here, if I can call it that, is that we were calling "xz
> --list --verbose" and not specifying a second "--verbose".

That would explain it indeed.

> This has now been remedied in Git, which will most likely be released
> on Friday 17th May. I've used your original test case as a literal
> test fixture, and can confirm it now shows a difference.

Excellent, thanks!

> Given the extra verbose information, I did alas make a related
> change so that it will not show the "--verbose --verbose" output if
> there are differences between the files contained within the .xz.
> Otherwise every single .deb package would show a very lengthy and
> distracting output.

That seems like the right thing to do indeed.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#1069329: fixed in diffoscope 266

[Paul Wise]

On Fri, 2024-05-10 at 14:49 +, Chris Lamb wrote:

>    * Use "xz --list" to supplement the output when comparing .xz archives;
>  essential when some underlying metadata differs. (Closes: #1069329)
>    * Actually append the xz --list after the container differences, as it
>  simplifies tests and the output.

Hmm, I still get a hex diff with the test case I posted in the bug:

$ echo foo > foo
$ xz -0 < foo > foo.0.xz
$ xz -9 < foo > foo.9.xz
$ diffoscope foo.0.xz foo.9.xz
--- foo.0.xz
+++ foo.9.xz
│┄ Format-specific differences are supported for XZ compressed files but no 
file-specific differences were detected; falling back to a binary diff. file(1) 
reports: XZ compressed data, checksum CRC64
@@ -1,4 +1,4 @@
 : fd37 7a58 5a00 0004 e6d6 b446 0200 2101  .7zXZ..F..!.
-0010: 0c00  8f98 419c 0100 0366 6f6f 0a00  ..Afoo..
+0010: 1c00  10cf 58cc 0100 0366 6f6f 0a00  ..Xfoo..
 0020: ffd7 ac5a 3031 9cf2 0001 1c04 6f2c 9cc1  ...Z01..o,..
 0030: 1fb6 f37d 0100  0004 595a...}..YZ

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#1069329: diffoscope: XZ: compare metadata before comparing compressed data

[Paul Wise]

Package: diffoscope
Severity: wishlist
X-Debbugs-Cc: Sam James 

When comparing two XZ compressed files that decompress to identical
data, please compare the metadata before comparing compressed data.

The xz --list option can be used for this, from the manual page:

   Print information about compressed files. No uncompressed output is
   produced, and no files are created or removed. In list mode, the 
   program cannot read the compressed data from standard input or from
   other unseekable sources.
   
   The default listing shows basic information about files, one file
   per line. To get more detailed information, use also the --verbose
   option. For even more information, use --verbose twice, but note
   that this may be slow, because getting all the extra information
   requires many seeks. The width of verbose output exceeds 80
   characters, so piping the output to, for example, less -S may be
   convenient if the terminal isn't wide enough.
   
   The exact output may vary between xz versions and different locales.
   For machine-readable output, --robot --list should be used.

In addition, the filename printed is the one from the command-line,
there does not appear to be filename metadata embedded in XZ files,
so the filename could be stripped before comparison where possible,
since diffoscope already compares filenames at a different layer.

The human-readable output may change between xz versions, so the
filename stripping will be brittle and could break on xz updates,
but comparing human-readable output is more user-friendly than
comparing the robot output, so any stripping would have to be
best-effort, but since any tests for it could break on xz upgrades,
and probably many comparisons will be between the same filenames within
different dirs, it might not be worth adding filename stripping yet,
until xz itself offers an option for hiding the filenames.

In addition, currently the xz --list option only supports the xz file
format and does not support the lzma and lzip file formats.

   $ echo foo > foo
   $ xz -0 < foo > foo.0.xz
   $ xz -9 < foo > foo.9.xz
   
   $ diffoscope foo.0.xz foo.9.xz 
   --- foo.0.xz
   +++ foo.9.xz
   │┄ Format-specific differences are supported for XZ compressed files but no 
file-specific differences were detected; falling back to a binary diff. file(1) 
reports: XZ compressed data, checksum CRC64
   @@ -1,4 +1,4 @@
: fd37 7a58 5a00 0004 e6d6 b446 0200 2101  .7zXZ..F..!.
   -0010: 0c00  8f98 419c 0100 0366 6f6f 0a00  ..Afoo..
   +0010: 1c00  10cf 58cc 0100 0366 6f6f 0a00  ..Xfoo..
0020: ffd7 ac5a 3031 9cf2 0001 1c04 6f2c 9cc1  ...Z01..o,..
0030: 1fb6 f37d 0100  0004 595a...}..YZ
   
   $ diff -u <(xz --list foo.0.xz) <(xz --list foo.9.xz)
   --- /dev/fd/63   2024-04-20 08:26:27.769377608 +0800
   +++ /dev/fd/62   2024-04-20 08:26:27.769377608 +0800
   @@ -1,2 +1,2 @@
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
   -1   1 60 B  4 B---  CRC64   foo.0.xz
   +1   1 60 B  4 B---  CRC64   foo.9.xz
   
   $ diff -u <(xz --list --verbose foo.0.xz) <(xz --list --verbose foo.9.xz)
   --- /dev/fd/63   2024-04-20 08:26:43.701196927 +0800
   +++ /dev/fd/62   2024-04-20 08:26:43.705196881 +0800
   @@ -1,4 +1,4 @@
   -foo.0.xz (1/1)
   +foo.9.xz (1/1)
  Streams:   1
  Blocks:1
  Compressed size:   60 B
   
   $ diff -u <(xz --list --verbose --verbose foo.0.xz) <(xz --list --verbose 
--verbose foo.9.xz)
   --- /dev/fd/63   2024-04-20 08:26:56.029056126 +0800
   +++ /dev/fd/62   2024-04-20 08:26:56.029056126 +0800
   @@ -1,4 +1,4 @@
   -foo.0.xz (1/1)
   +foo.9.xz (1/1)
  Streams:   1
  Blocks:1
  Compressed size:   60 B
   @@ -11,7 +11,7 @@
 1 1   0   0  60
   4---  CRC640
  Blocks:
Stream Block  CompOffsetUncompOffset   TotalSize  
UncompSize  Ratio  Check  CheckVal  Header  FlagsCompSize   
 MemUsage  Filters
   - 1 1  12   0  28
   4  7.000  CRC64  f29c31305aacd7ff  12  --  8 
  1 MiB  --lzma2=dict=256KiB
   -  Memory needed: 1 MiB
   + 1 1  12   0  28
   4  7.000  CRC64  f29c31305aacd7ff  12  --  8 
 65 MiB  --lzma2=dict=64MiB
   +  Memory needed: 65 MiB
  Sizes in headers:  No
  Minimum XZ Utils version: 5.0.0
   
   $ diff -u <(xz --robot --list --verbose --verbose foo.0.xz) <(xz --robot 
--list --verbose --verbose foo.9.xz)
   --- /dev/fd/63   2024-04-20 08:31:42.445584805 +0800
   +++ /dev/fd/62   2024-04-20 08:31:42.449584755 +0800
   @@ -1,6 +1,6 @@
   -namefoo.0.xz
   

Bug#1022210: diffoscope: highlight whitespace-only differences in text data

[Paul Wise]

Package: diffoscope
Version: 224
Severity: wishlist

It would be nice if diffoscope could help highlight that text data
differ only in the whitespace or if they differ in the text too.

The proposal would be that by default diffoscope would use wdiff (or
similar) to compare all line based text buffers (including those
converted from other formats) in order to check for whitespace-only
differences. When there are non-whitespace differences then display the
wdiff (or similar) output, with a comment saying these are differences
in the text after ignoring whitespace. In situations where the text
buffer differs only by whitespace, diffoscope would do a line diff of
the text buffer itself, with a comment saying the text of the two text
buffers was not different and only whitespace changes were present.

Since this check probably won't be useful for all diffs of line based
text buffers, probably there will need to be various heuristics for
when to apply it and when not to apply it, perhaps starting with
applying it to everything and then adding exceptions over time.

This would be useful in some situations like when comparing old
versions of a document with newer versions of a document or similar.
In particular it would have been useful when preparing this mail:

https://lists.debian.org/msgid-search/197a4671e7694c24424b91b4d7288867c0c85d9b.ca...@debian.org

-- System Information:
Debian Release: bookworm/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
merged-usr: no
Architecture: amd64 (x86_64)

Kernel: Linux 6.0.0-1-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  224

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.3.1-1
ii  androguard   3.4.0~a1-5
ii  apksigner    31.0.2-1
ii  apktool  2.6.1+dfsg.1-2
ii  binutils-multiarch   2.39-8
ii  bzip2    1.0.8-5+b1
ii  caca-utils   0.99.beta20-3
ii  colord   1.4.6-1
ii  coreboot-utils   4.15~dfsg-2
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs    1.46.6~rc1-1+b1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:5.1.2-1
ii  fontforge-extras 1:20220308~dfsg-1
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.4
ii  gettext  0.21-9
ii  ghc  9.0.2-4
ii  ghostscript  9.56.1~dfsg-1
ii  giflib-tools 5.2.1-2.5
ii  gnumeric 1.12.52-1
ii  gnupg    2.2.39-1
ii  gnupg-utils  2.2.39-1+b1
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3+b3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3+b3
ii  jsbeautifier 1.14.4-1
ii  libarchive-tools 3.6.0-1
pn  libxmlb-dev  
ii  llvm 1:14.0-55.2+b1
ii  lz4 [liblz4-tool]    1.9.4-1
pn  mono-utils   
ii  ocaml-nox    4.13.1-3
pn  odt2txt  
pn  oggvideotools    
ii  openjdk-11-jdk [java-sdk]    11.0.17+8-2
ii  openssh-client   1:9.0p1-1+b2
ii  openssl  3.0.5-4
ii  pgpdump  0.34-1
ii  poppler-utils    22.08.0-2.1
pn  procyon-decompiler   
ii  python3-argcomplete  2.0.0-1
ii  python3-binwalk  2.3.3+dfsg1-2
ii  python3-debian   0.1.48
ii  python3-defusedxml   0.7.1-2
ii  python3-guestfs  1:1.48.4-2+b1
hi  python3-jsondiff 1.1.1-4
ii  python3-pdfminer 20220319+dfsg-1
ii  python3-progressbar  2.5-3
ii  python3-pypdf2   2.11.0-1
ii  python3-pyxattr  0.7.2-2+b1
ii  python3-rpm  4.17.1.1+dfsg-1
ii  python3-tlsh 3.4.4+20151206-1.4+b2
pn  r-base-core  
pn  radare2  
ii  rpm2cpio 4.17.1.1+dfsg-1
ii  sng 

Bug#1022209: diffoscope: highlight text-only differences in HTML files

[Paul Wise]

Package: diffoscope
Version: 224
Severity: wishlist

It would be nice if diffoscope could help highlight that HTML files
differ in the text output or if they differ only in the non-text HTML
bytes like the page title, the stylesheet etc.

The proposal would be that by default diffoscope would convert HTML
files to text, diff that and if there were text differences then
display them, with a comment saying these are differences in the text.
In situations where the text does not differ, diffoscope would do a
line diff of the HTML file itself, with a comment saying the text of
the two files was not different.

This is useful in some situations like when comparing old versions of a
document with newer versions of a document or similar. In particular it
would have been useful when preparing this mail to debian-mentors:

https://lists.debian.org/msgid-search/197a4671e7694c24424b91b4d7288867c0c85d9b.ca...@debian.org

Since there are many different tools for conversion of HTML to text and
each of them have different bugs and features, probably this feature
should allow the user to choose the tool they want to use for this.

   $ head -vn-0 *.html
   ==> bar.html <==
   
   
   bar
   
   
   
   
   
   
   bar
   
   
   
   
   ==> foo.html <==
   
   
   foo
   
   
   
   
   
   
   foo
   
   
   
   
   $ diffoscope foo.html bar.html 
   --- foo.html
   +++ bar.html
   @@ -1,17 +1,17 @@


   -foo
   +bar






   -foo
   +bar



   
   $ diff -u <(w3m -dump foo.html) <(w3m -dump bar.html)
   --- /dev/fd/63   2022-10-22 08:52:33.581676470 +0800
   +++ /dev/fd/62   2022-10-22 08:52:33.585676477 +0800
   @@ -1,2 +1,2 @@
   -foo
   +bar
   
   $ diff -u <(html2text foo.html) <(html2text bar.html)
   --- /dev/fd/63   2022-10-22 08:54:43.793859066 +0800
   +++ /dev/fd/62   2022-10-22 08:54:43.781859049 +0800
   @@ -1 +1 @@
   -foo
   +bar

-- System Information:
Debian Release: bookworm/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
merged-usr: no
Architecture: amd64 (x86_64)

Kernel: Linux 6.0.0-1-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  224

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.3.1-1
ii  androguard   3.4.0~a1-5
ii  apksigner    31.0.2-1
ii  apktool  2.6.1+dfsg.1-2
ii  binutils-multiarch   2.39-8
ii  bzip2    1.0.8-5+b1
ii  caca-utils   0.99.beta20-3
ii  colord   1.4.6-1
ii  coreboot-utils   4.15~dfsg-2
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs    1.46.6~rc1-1+b1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:5.1.2-1
ii  fontforge-extras 1:20220308~dfsg-1
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.4
ii  gettext  0.21-9
ii  ghc  9.0.2-4
ii  ghostscript  9.56.1~dfsg-1
ii  giflib-tools 5.2.1-2.5
ii  gnumeric 1.12.52-1
ii  gnupg    2.2.39-1
ii  gnupg-utils  2.2.39-1+b1
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3+b3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3+b3
ii  jsbeautifier 1.14.4-1
ii  libarchive-tools 3.6.0-1
pn  libxmlb-dev  
ii  llvm 1:14.0-55.2+b1
ii  lz4 [liblz4-tool]    1.9.4-1
pn  mono-utils   
ii  ocaml-nox    4.13.1-3
pn  odt2txt  
pn  oggvideotools    
ii  openjdk-11-jdk [java-sdk]    11.0.17+8-2
ii  openssh-client   1:9.0p1-1+b2
ii  openssl  3.0.5-4
ii  pgpdump  0.34-1
ii  poppler-utils    22.

Bug#1022146: diffoscope: detect ordering-only differences in XML files

[Paul Wise]

Package: diffoscope
Version: 224
Severity: wishlist

It would be nice if diffoscope could detect that XML files are only
different because of ordering differences, currently it cannot:

   $ head -vn-0 *.xml
   ==> bar.xml <==
   
   


   
   
   ==> foo.xml <==
   
   


   
   
   $ diffoscope *.xml
   --- bar.xml
   +++ foo.xml
   │   --- bar.xml
   ├── +++ foo.xml
   │ @@ -1,5 +1,5 @@
   │  
   │  
   │ -  
   │
   │ +  
   │  
   
   $ echo $?
   1
   
-- System Information:
Debian Release: bookworm/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
merged-usr: no
Architecture: amd64 (x86_64)

Kernel: Linux 6.0.0-1-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  224

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.3.1-1
ii  androguard   3.4.0~a1-5
ii  apksigner    31.0.2-1
ii  apktool  2.6.1+dfsg.1-2
ii  binutils-multiarch   2.39-8
ii  bzip2    1.0.8-5+b1
ii  caca-utils   0.99.beta20-3
ii  colord   1.4.6-1
ii  coreboot-utils   4.15~dfsg-2
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs    1.46.6~rc1-1+b1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:5.1.2-1
ii  fontforge-extras 1:20220308~dfsg-1
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.4
ii  gettext  0.21-9
ii  ghc  9.0.2-4
ii  ghostscript  9.56.1~dfsg-1
ii  giflib-tools 5.2.1-2.5
ii  gnumeric 1.12.52-1
ii  gnupg    2.2.39-1
ii  gnupg-utils  2.2.39-1+b1
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3+b3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3+b3
ii  jsbeautifier 1.14.4-1
ii  libarchive-tools 3.6.0-1
pn  libxmlb-dev  
ii  llvm 1:14.0-55.2+b1
ii  lz4 [liblz4-tool]    1.9.4-1
pn  mono-utils   
ii  ocaml-nox    4.13.1-3
pn  odt2txt  
pn  oggvideotools    
ii  openjdk-11-jdk [java-sdk]    11.0.17+8-2
ii  openssh-client   1:9.0p1-1+b2
ii  openssl  3.0.5-4
ii  pgpdump  0.34-1
ii  poppler-utils    22.08.0-2.1
pn  procyon-decompiler   
ii  python3-argcomplete  2.0.0-1
ii  python3-binwalk  2.3.3+dfsg1-2
ii  python3-debian   0.1.48
ii  python3-defusedxml   0.7.1-2
ii  python3-guestfs  1:1.48.4-2+b1
hi  python3-jsondiff 1.1.1-4
ii  python3-pdfminer 20220319+dfsg-1
ii  python3-progressbar  2.5-3
ii  python3-pypdf2   2.11.0-1
ii  python3-pyxattr  0.7.2-2+b1
ii  python3-rpm  4.17.1.1+dfsg-1
ii  python3-tlsh 3.4.4+20151206-1.4+b2
pn  r-base-core  
pn  radare2  
ii  rpm2cpio 4.17.1.1+dfsg-1
ii  sng  1.1.0-4
ii  sqlite3  3.39.4-1
ii  squashfs-tools   1:4.5.1-1
ii  tcpdump  4.99.1-4+b1
ii  u-boot-tools 2022.10+dfsg-1
ii  unzip    6.0-27
pn  wabt 
pn  xmlbeans 
ii  xxd  2:9.0.0626-1
ii  xz-utils 5.2.5-2.1
ii  zip  3.0-12
ii  zstd 1.5.2+dfsg-1

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.6.1+dfsg+~3.5.14-1

-- no debconf information

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https

Bug#1022145: diffoscope: detect ordering-only differences in text files

[Paul Wise]

Package: diffoscope
Version: 224
Severity: wishlist

It would be nice if diffoscope could detect that text files are only
different because of ordering differences, currently it cannot:

   $ (echo foo ; echo bar) > foo
   $ (echo bar ; echo foo) > bar
   
   $ diffoscope foo bar
   --- foo
   +++ bar
   @@ -1,2 +1,2 @@
   -foo
bar
   +foo
   
   $ echo $?
   1
   
   $ (echo foo ; echo bar ; echo baz ; echo biz ; echo buz) > foo
   $ (echo bar ; echo foo ; echo baz ; echo biz ; echo buz) > bar
   
   $ diffoscope foo bar
   --- foo
   +++ bar
   @@ -1,5 +1,5 @@
   -foo
bar
   +foo
baz
biz
buz
   
   $ echo $?
   1

-- System Information:
Debian Release: bookworm/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
merged-usr: no
Architecture: amd64 (x86_64)

Kernel: Linux 6.0.0-1-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  224

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.3.1-1
ii  androguard   3.4.0~a1-5
ii  apksigner31.0.2-1
ii  apktool  2.6.1+dfsg.1-2
ii  binutils-multiarch   2.39-8
ii  bzip21.0.8-5+b1
ii  caca-utils   0.99.beta20-3
ii  colord   1.4.6-1
ii  coreboot-utils   4.15~dfsg-2
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.46.6~rc1-1+b1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:5.1.2-1
ii  fontforge-extras 1:20220308~dfsg-1
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.4
ii  gettext  0.21-9
ii  ghc  9.0.2-4
ii  ghostscript  9.56.1~dfsg-1
ii  giflib-tools 5.2.1-2.5
ii  gnumeric 1.12.52-1
ii  gnupg2.2.39-1
ii  gnupg-utils  2.2.39-1+b1
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3+b3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3+b3
ii  jsbeautifier 1.14.4-1
ii  libarchive-tools 3.6.0-1
pn  libxmlb-dev  
ii  llvm 1:14.0-55.2+b1
ii  lz4 [liblz4-tool]1.9.4-1
pn  mono-utils   
ii  ocaml-nox4.13.1-3
pn  odt2txt  
pn  oggvideotools
ii  openjdk-11-jdk [java-sdk]11.0.17+8-2
ii  openssh-client   1:9.0p1-1+b2
ii  openssl  3.0.5-4
ii  pgpdump  0.34-1
ii  poppler-utils22.08.0-2.1
pn  procyon-decompiler   
ii  python3-argcomplete  2.0.0-1
ii  python3-binwalk  2.3.3+dfsg1-2
ii  python3-debian   0.1.48
ii  python3-defusedxml   0.7.1-2
ii  python3-guestfs  1:1.48.4-2+b1
hi  python3-jsondiff 1.1.1-4
ii  python3-pdfminer 20220319+dfsg-1
ii  python3-progressbar  2.5-3
ii  python3-pypdf2   2.11.0-1
ii  python3-pyxattr  0.7.2-2+b1
ii  python3-rpm  4.17.1.1+dfsg-1
ii  python3-tlsh 3.4.4+20151206-1.4+b2
pn  r-base-core  
pn  radare2  
ii  rpm2cpio 4.17.1.1+dfsg-1
ii  sng  1.1.0-4
ii  sqlite3  3.39.4-1
ii  squashfs-tools   1:4.5.1-1
ii  tcpdump  4.99.1-4+b1
ii  u-boot-tools 2022.10+dfsg-1
ii  unzip6.0-27
pn  wabt 
pn  xmlbeans 
ii  xxd  2:9.0.0626-1
ii  xz-utils 5.2.5-2.1
ii  zip  3.0-12
ii  zstd 1.5.2+dfsg-1

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.6.1+dfsg+~3.5.14-1

-- no debconf information

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signe

Re: hope help reproducible-build to support Debian riscv64 arch

[Paul Wise]

On Thu, 2022-06-09 at 13:25 +0200, Mattia Rizzolo wrote:

> Oh, interesting.  TIL.

The docs are relatively new, at one point we had a few folks who had
never contributed to Debian show up wanting to convert their company's
internal port of Debian for their company's CPU arch into an official
Debian port.

I figured it was time to document the process and so helmut, josch,
guillem, aurel32, other folks and myself translated the PortTemplate
page (which I created as a more organised alternative to current port
pages) into a proper checklist that ideally every port could comply
with. I consider reproducible builds (and bootstrappable builds) an
essential part of any ideal port, so I of course mentioned both.

https://wiki.debian.org/PortTempate

The architectures they were interested in are ARC from Synopsis and
LoongArch from Loongson, who used to make MIPS chips. Later we had some
people new to Debian wanting to improve riscv64, so the new port docs
proved useful for guiding them too.

https://wiki.debian.org/Ports/arc
https://wiki.debian.org/Ports/loongarch64

> > Since repro builds […] is a blocker for testing migration,
> 
> Ahem, I'd really love if that was the case, but it really isn't at
> the moment.

Hmm, I wonder how I got this so wrong. I thought britney was taking
into account repro builds tests in some way, maybe blocking regressions
or reducing migration time, I guess I was remembering the 2019 thread
about this but completely misremembering the outcome, wooops!

https://lists.debian.org/msgid-search/f6a50af4-9857-d491-0d7b-bf4cadd79...@debian.org

If you'd prefer to move repro builds to a different section, please
edit the docs or let me know which stage is most appropriate for it.
That can of course change as circumstances in Debian change.

https://wiki.debian.org/PortsDocs/New

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: hope help reproducible-build to support Debian riscv64 arch

[Paul Wise]

On Thu, 2022-06-09 at 11:00 +0100, Chris Lamb wrote:

> For some unknown reason, I thought that riscv64 *was* an official
> Debian architecture, and the tone of my previous emails should be
> read with that (incorrect) assumption in mind.

riscv64 is in the process of becoming an official port and then 
hopefully a released port for bookworm, but isn't quite there yet, the
folks who are involved are looking at the remaining steps from the new
port docs. Repro-builds is in the official port section, after the step
of converting from unofficial to official and before the step of
applying to become a released port. Since repro builds doesn't test
unofficial ports and is a blocker for testing migration, I think this
is the right section of the new port docs, feel free to adjust the
advice if you disagree. Probably this thread was started prematurely
due to the enthusiasm of the port advocates for making the port
available in bookworm and are perhaps looking a few steps too far ahead
in the steps of the process.

https://wiki.debian.org/PortsDocs/New#Unofficial_port
https://wiki.debian.org/PortsDocs/New#Official_port
https://wiki.debian.org/PortsDocs/New#Released_port

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: hope help reproducible-build to support Debian riscv64 arch

[Paul Wise]

On Thu, 2022-05-26 at 14:00 +0100, Chris Lamb wrote:

> What I was trying to say is that we would need enough riscv64
> computing power to build all of Debian in good time... where "good
> time" is loosely defined as being able to keep up with the current
> architectures.

Here are the statistics on performance of the current architectures:

https://tests.reproducible-builds.org/debian/index_performance.html

So it sounds like enough RISC-V hardware to build between 2-3 thousand
packages per day on average will be needed.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: Including build metadata in packages

[Paul Wise]

Simon McVittie wrote:

> Relatedly, I would like to be able to capture some information about
> builds even if (perhaps especially if) the build fails.

That is a good point that I hadn't considered.

> so that failing builds can also produce artifacts, to help the
> maintainer and/or porters to figure out why the build failed.

Agreed that this is useful.

> handling build logs is not dak's job (and I don't think handling
> things like the binutils test results should be dak's job either).

It has always felt weird to me that build logs are entirely separate to
the archive off in a side service rather than first-class artefacts
that people occasionally need to look at. Also that the maintainer
build logs don't end up anywhere and are probably just deleted. I think
the same applies to the buildinfo files and also these tests results
and other artefacts that are mentioned in this thread.

> Here's a straw-man spec, which I have already prototyped in
> :

This seems better than my proposal, modulo the above and also the repro
builds need for a way to distribute buildinfo files somehow.

IIRC last time the build artefact discussion came up I was cycling
between having the artefact handling in the sbuild configs on the
buildds for quick implementation vs having it in debian/ dirs for
distributed maintenance by maintainers.

I think there is a fundamental question here that needs answering
definitively: who is the audience for the artefact feature?

 * Is it individual package maintainers who want test result details?
 * Is it build tool maintainers who want data on tool use/failures?
 * Is it porters who want more detailed logs in case of failure?
 * Is it buildd maintainers for some reason?
 * Is it RC bug fixers?
 * Is it all of the above?

Once that is answered, then we can think about how to accommodate how
and where the list(s?) of files are to be maintained?

 * in debian/
 * in build tools (meson, gcc etc)
 * in debhelper extensions
 * in debhelper
 * in wanna-build
 * in sbuild
 * in sbuild.conf in dsa-puppet
 * in sbuild overrides on buildds

Some of the above will be faster to implement and some will be slower.
The faster parts can possibly even make up for the slower parts, by for
example doing the sbuild proposal in hooks until it is done in stable.

Then there is the question of how the files get off the systems where
builds happen (buildds, maintainer systems). Again, the faster/slower
implementation implications exist here too.

Then there is the question of how the files are further distributed
from there and the question of how people access them.

Then there is the question of whether any of the above will be
implemented in a way that is useful solely to Debian, or in a more
general way to all Debian or apt repository based distributions. Being
able to publish build logs/artifacts seems like something other distros
would be interested in. It sounds like at least the GCC maintainers
want that for too Ubuntu at minimum.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: Including build metadata in packages

[Paul Wise]

On Sun, 2022-02-13 at 14:13 -0800, Vagrant Cascadian wrote:

> * Split build metadata into a separate file or archive
> 
> Some of the debian-installer packages generate tarballs that are not
> .deb files and are included in the .changes files when uploading to
> the archive; making a similar generalized option for other packages to
> put build metadata into a separate artifact might be workable approach,
> although this would presumably require toolchain changes in dpkg and
> dak at the very least, and might take a couple release cycles, which
> is... well, debian.

I already sent a mail like this in the past, but...

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=950585#30

This approach is already in use in the archive, but not
yet for the kind of artefacts that you are talking about:

https://codesearch.debian.net/search?perpkg=1&q=dpkg-distaddfile
https://salsa.debian.org/ftp-team/dak/raw/master/config/debian/dak.conf (search 
for AutomaticByHandPackages)
https://salsa.debian.org/ftp-team/dak/raw/master/daklib/upload.py (search for 
byhand_files)
https://salsa.debian.org/ftp-team/dak/tree/master/scripts/debian/

I think this would not require anything except a new dak config stanza
for AutomaticByHandPackage and potentially a patch to dak code or a
script. Seems unlikely it would require changes to anything other than
dak plus the packages that want to opt in to using it, so should be
completely doable within the bookworm release cycle.

If you want to have some way to automatically download the data, then
something like apt-file and Contents files could be done, I expect that
would also be able to be done for the bookworm release and also be
possible to put in bullseye-backports.

You could even include all the build logs and build info in the same
data set, and potentially modify the package build process so that
build logs for maintainer built binaries end up there too.

Something like this would be my suggested archive structure:

Release -> Builds-amd64 -> foo_amd64.build
 \ \-> foo_amd64.buildinfo
  \--> foo_amd64.buildmeta.tar.xz

Or since the buildinfo files are similar to Packages/Sources stanzas:

Release -> BuildLogs-amd64 -> foo_amd64.build.xz
 \ \-> BuildInfo-amd64
  \--> BuildMeta-amd64 -> foo_amd64.buildmeta.tar.xz

This could be in the main archive, or a separate debian-builds archive.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: Embedded buildpath via rpath using cmake

[Paul Wise]

Vagrant Cascadian wrote:

> Over the last several months, I and others have found quite a few
> packages that embed build paths via rpath when building with cmake.

This seems like the sort of thing that will be an ongoing problem, so
if it is detectable statically then a lintian warning might be good.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#1002002: diffoscope: note order-only changes in .dsc fields

[Paul Wise]

Package: diffoscope
Version: 197
Severity: wishlist

I recently sponsored memtest86+ and used diffoscope to compare the
Debian source packages before uploading. The debian/ directory had been
subject to wrap-and-sort, which also affects the Architecture field.
The architecture list hadn't been altered, just sorted. Please add an
"Ordering differences only" comment for all the .dsc fields that
contain multiple values where the order has no effect. That includes at
minimum Binary, Architecture, Uploaders, Build-Depends, Package-List
(including the embedded architecture lists) and the checksums fields.

   $ diffoscope memtest86+_5.01-3.1.dsc memtest86+_5.01-4.dsc 
   --- memtest86+_5.01-3.1.dsc
   +++ memtest86+_5.01-4.dsc
   ...
   ├── Architecture
   │ @@ -1 +1 @@
   │ -i386 amd64 kfreebsd-i386 kfreebsd-amd64 lpia hurd-i386
   │ +amd64 hurd-i386 i386 kfreebsd-amd64 kfreebsd-i386 lpia
   ├── Package-List
   │ @@ -1 +1 @@
   │ -memtest86+ deb misc optional 
arch=i386,amd64,kfreebsd-i386,kfreebsd-amd64,lpia,hurd-i386
   │ +memtest86+ deb misc optional 
arch=amd64,hurd-i386,i386,kfreebsd-amd64,kfreebsd-i386,lpia
   ...

-- System Information:
Debian Release: bookworm/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (860, 
'testing-proposed-updates-debug'), (860, 'testing-proposed-updates'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.15.0-2-amd64 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  197

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.3.1-1
ii  androguard   3.4.0~a1-1
ii  apksigner31.0.2-1
ii  apktool  2.5.0+dfsg.1-2
ii  binutils-multiarch   2.37-7
ii  bzip21.0.8-5
ii  caca-utils   0.99.beta19-2.2
ii  colord   1.4.5-3
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.46.4-1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:4.4.1-2+b1
ii  fontforge-extras 1:20201107~dfsg-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.2
ii  gettext  0.21-4
ii  ghc  8.8.4-3
ii  ghostscript  9.55.0~dfsg-3
ii  giflib-tools 5.1.9-2
ii  gnumeric 1.12.50-1
ii  gnupg2.2.27-2
ii  gnupg-utils  2.2.27-2
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3
ii  jsbeautifier 1.14.0-1
ii  libarchive-tools 3.4.3-2+b1
ii  llvm 1:11.0-51+nmu5
ii  lz4 [liblz4-tool]1.9.3-2
pn  mono-utils   
ii  ocaml-nox4.11.1-5
pn  odt2txt  
pn  oggvideotools
ii  openjdk-11-jdk [java-sdk]11.0.13+8-1
ii  openssh-client   1:8.7p1-2
ii  openssl  1.1.1l-1
ii  pgpdump  0.33-2
ii  poppler-utils20.09.0-3.1
pn  procyon-decompiler   
ii  python3-argcomplete  1.12.3-0.1
ii  python3-binwalk  2.3.2+dfsg1-1
ii  python3-debian   0.1.42
ii  python3-defusedxml   0.7.1-1
ii  python3-guestfs  1:1.44.2-1.1
ii  python3-jsondiff 1.1.1-4
ii  python3-pdfminer 20201018+dfsg-1
ii  python3-progressbar  2.5-2
ii  python3-pypdf2   1.26.0-4
ii  python3-pyxattr  0.7.2-2
ii  python3-rpm  4.17.0+dfsg1-4
ii  python3-tlsh 3.4.4+20151206-1.4+b1
pn  r-base-core  
pn  radare2  
ii  rpm2cpio 4.17.0+dfsg1-4
ii  sng  1.1.0-4
ii  sqlite3  3.36.0-2
ii  squashfs-tools   1:4.5-3
ii  tcpdump  4.99.1-3
ii  u-boot-tools 2021.10+dfsg-1
ii  unzip6.0-26
ii  vim-common   2:8.2.35

Bug#999438: diffoscope: issues with XML files not named *.xml

[Paul Wise]

Package: diffoscope
Version: 190
Severity: normal

There are two issues with XML files not named *.xml:

They don't get reformatted before comparison, resulting in a diff of
the plain text, instead of a diff of the reformatted XML.

When comparing them with XML files named *.xml, a comparison of the
bytes is done, resulting in a diff of two hex dumps, instead of a diff
of the reformatted XML or a diff of the plain text. The reformatted XML
would be the best thing to diff, but plain text should be a fallback.

The xmllint tool can reformat them just fine and the file tool can
detect them as XML and detect their MIME type, so this issue is likely
to be a problem in the diffoscope code.

   $ head -vn-0 test-{old,new}.xml 
   ==> test-old.xml <==
   
   
   
   
   
   
   
   
   ==> test-new.xml <==
   
   
   
   
   
   
   
   
   
   
   $ diffoscope test-{old,new}.xml 
   --- test-old.xml
   +++ test-new.xml
   │   --- test-old.xml
   ├── +++ test-new.xml
   │ @@ -1,6 +1,8 @@
   │  
   │  
   │    
   │ -    
   │ +    
   │ +  
   │ +    
   │    
   │  
   
   $ cp test-new.xml test-new.not-xml
   
   $ cp test-old.xml test-old.not-xml
   
   $ diffoscope test-{old,new}.not-xml 
   --- test-old.not-xml
   +++ test-new.not-xml
   @@ -1,7 +1,9 @@
    
    
    
    
   +
   +
    
    
    
   
   $ diffoscope test-old.xml test-new.not-xml 
   --- test-old.xml
   +++ test-new.not-xml
   @@ -1,5 +1,6 @@
    : 3c3f 786d 6c20 7665 7273 696f 6e3d 2231  
   -0040: 0a3c 2f66 6f6f 3e0a 3c2f 7465 7374 3e0a  ...
   +0030: 6f6f 3e0a 3c62 6172 3e0a 3c62 617a 3e0a  oo>...
   +0040: 3c2f 6261 7a3e 0a3c 2f62 6172 3e0a 3c2f  
   
   $ diffoscope test-old.not-xml test-new.xml 
   --- test-old.not-xml
   +++ test-new.xml
   @@ -1,5 +1,6 @@
    : 3c3f 786d 6c20 7665 7273 696f 6e3d 2231  
   -0040: 0a3c 2f66 6f6f 3e0a 3c2f 7465 7374 3e0a  ...
   +0030: 6f6f 3e0a 3c62 6172 3e0a 3c62 617a 3e0a  oo>...
   +0040: 3c2f 6261 7a3e 0a3c 2f62 6172 3e0a 3c2f  
   
   $ xmllint --format test-old.xml
   
   
 
   
   
 
   
   
   $ xmllint --format test-new.xml
   
   
 
   
 
   
   
 
   
   
   $ xmllint --format test-old.not-xml
   
   
 
   
   
 
   
   
   $ xmllint --format test-new.not-xml
   
   
 
   
 
   
   
 
   

   $ file test-*
   test-new.not-xml: XML 1.0 document, ASCII text
   test-new.xml: XML 1.0 document, ASCII text
   test-old.not-xml: XML 1.0 document, ASCII text
   test-old.xml: XML 1.0 document, ASCII text
   
   $ file --mime test-*
   test-new.not-xml: text/xml; charset=us-ascii
   test-new.xml: text/xml; charset=us-ascii
   test-old.not-xml: text/xml; charset=us-ascii
   test-old.xml: text/xml; charset=us-ascii
   
-- System Information:
Debian Release: bookworm/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (860, 
'testing-proposed-updates-debug'), (860, 'testing-proposed-updates'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.14.0-4-amd64 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  190

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.3.1-1
ii  androguard   3.4.0~a1-1
ii  apksigner    30.0.3-4
ii  apktool  2.5.0+dfsg.1-2
ii  binutils-multiarch   2.37-7
ii  bzip2    1.0.8-4
ii  caca-utils   0.99.beta19-2.2
ii  colord   1.4.5-3
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs    1.46.4-1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:4.4.1-1+b1
ii  fontforge-extras 1:20201107~dfsg-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.2
ii  gettext  0.21-4
ii  ghc  8.8.4-3
ii  ghostscript  9.54.0~dfsg-5
ii  giflib-tools 5.1.9-2
ii  gnumeric 1.12.50-1
ii  gnupg    2.2.27-2
ii  gnupg-utils  2.2.27-2
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3
ii  imagemagick-6.q16 [image

Bug#989626: diffoscope: when comparing fonts with .ttx files, convert the font to .ttx first

[Paul Wise]

Package: diffoscope
Version: 177
Severity: wishlist

diffoscope currently prints a hex dump when comparing a TTF font with a
.ttx XML dump of a TTF font. If it used ttx (from fonttools) to convert
that TTF font to .ttx instead, then the comparison would become an XML
diff, which diffoscope treats as a text diff, which is better than hex.

   $ git clone -q https://gitlab.gnome.org/GNOME/gtksourceview
   
   $ diffoscope gtksourceview/data/fonts/BuilderBlocks.tt{f,x}
   --- gtksourceview/data/fonts/BuilderBlocks.ttf
   +++ gtksourceview/data/fonts/BuilderBlocks.ttx
   @@ -1,28 +1,259 @@
   -: 0001  0008 0080 0003  636d 6170  cmap
   -0010: 0021 0258  0114  004c 676c 7966  .!.X...Lglyf
   -0020: 2644 7e48  016c  0030 6865 6164  &D~H...l...0head
   -0030: 0733 9950  008c  0036 6868 6561  .3.P...6hhea
   -0040: 0c01 0402  00c4  0024 686d 7478  ...$hmtx
   -0050: 0400   0108  000a 6c6f 6361  loca
   -0060: 0024 000c  0160  000a 6d61 7870  .$.`maxp
   -0070: 0006 0005  00e8  0020 6e61 6d65  ... name
   ...
   +: 3c3f 786d 6c20 7665 7273 696f 6e3d 2231  ...  .

   -
   -
   -
   +
   +
   +
  

  



   -
   +




   -
   +


   -
   -
   +
   +





  
   ...

-- System Information:
Debian Release: 11.0
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental'), 
(500, 'testing-security')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-7-amd64 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  177

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.53-10
ii  apksigner30.0.3-4
ii  apktool  2.5.0+dfsg.1-2
ii  binutils-multiarch   2.35.2-2
ii  bzip21.0.8-4
ii  caca-utils   0.99.beta19-2.2
ii  colord   1.4.5-3
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.46.2-1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:4.3.2-0+deb11u2
ii  fontforge-extras 1:20201107~dfsg-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.2
ii  gettext  0.21-4
ii  ghc  8.8.4-2
ii  ghostscript  9.53.3~dfsg-7
ii  giflib-tools 5.1.9-2
ii  gnumeric 1.12.48-1+b2
ii  gnupg2.2.27-2
ii  gnupg-utils  2.2.27-2
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3
ii  jsbeautifier 1.13.0-1
ii  libarchive-tools 3.4.3-2+b1
ii  llvm 1:11.0-51+nmu5
ii  lz4 [liblz4-tool]1.9.3-2
pn  mono-utils   
ii  ocaml-nox4.11.1-4
pn  odt2txt  
pn  oggvideotools
ii  openjdk-11-jdk [java-sdk]11.0.11+9-1
ii  openssh-client   1:8.4p1-5
ii  openssl  1.1.1k-1
ii  pgpdump  0.33-2
ii  poppler-utils20.09.0-3.1
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1.5
ii  python3-binwalk  2.3.1+dfsg1-1
ii  python3-debian   0.1.39
ii  python3-defusedxml   0.6.0-2
ii  python3-guestfs  1:1.44.0-2
ii  python3-jsondiff 1.1.1-4
ii  python3-pdfminer 20200726-1
ii  python3-progressbar  2.5-2
ii  python3-pypdf2   1.26.0-4
ii  python3-pyxattr  0.7.2-1+b1
ii  python3-rpm  4.16.1.2+dfsg1-0.4
ii  python3-tlsh 3.4.4+20151206-1.3+b4
pn  r-base-core  
pn  radare2  
ii  rpm2cpio 4.16.1.2+dfsg1-0.4
ii  sng   

Bug#988488: diffoscope: allow comparing directories with container formats (tar/ZIP/ISO etc)

[Paul Wise]

Package: diffoscope
Version: 174
Severity: wishlist

When comparing directories and files that are essentially directories
serialised into a file (such as tar, ZIP, ISO etc), currently
diffoscope just says one is a file while one is a directory.

   $ rm -rf ./* ; mkdir foo ; echo foo > foo/bar ; tar acf foo.tar foo ; echo 
bar >> foo/bar
   $ diffoscope foo foo.tar
   --- foo
   +++ foo.tar
   @@ -1 +1 @@
   -type: directory
   +type: file

diffoscope can already compare two directories, or two tar files or a
zip file and a tar file, so it would be useful if it could compare a
directory with a file in one of the many types of container formats.

This would be useful in many situations but especially for the
situation where dpkg-source reports a change to a file in the working
directory compared to the upstream tarball, but the change isn't
obvious from the output and should not be present in theory.

   dpkg-source: error: cannot represent change to foo.exe: binary file contents 
changed

-- System Information:
Debian Release: bullseye/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental'), 
(500, 'testing-security')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-6-amd64 (SMP w/8 CPU threads)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  diffoscope-minimal  174

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.53-10
ii  apksigner30.0.3-4
ii  apktool  2.5.0+dfsg.1-2
ii  binutils-multiarch   2.35.2-2
ii  bzip21.0.8-4
ii  caca-utils   0.99.beta19-2.2
ii  colord   1.4.5-3
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.46.2-1
ii  enjarify 1:1.0.3-5
ii  ffmpeg   7:4.3.2-0+deb11u1
ii  fontforge-extras 1:20201107~dfsg-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.2
ii  gettext  0.21-4
ii  ghc  8.8.4-2
ii  ghostscript  9.53.3~dfsg-7
ii  giflib-tools 5.1.9-2
ii  gnumeric 1.12.48-1+b2
ii  gnupg2.2.27-2
ii  gnupg-utils  2.2.27-2
pn  hdf5-tools   
ii  imagemagick  8:6.9.11.60+dfsg-1.3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3
ii  jsbeautifier 1.13.0-1
ii  libarchive-tools 3.4.3-2+b1
ii  llvm 1:11.0-51+nmu4
ii  lz4 [liblz4-tool]1.9.3-2
pn  mono-utils   
ii  ocaml-nox4.11.1-4
pn  odt2txt  
pn  oggvideotools
ii  openjdk-11-jdk [java-sdk]11.0.11+9-1
ii  openssh-client   1:8.4p1-5
ii  openssl  1.1.1k-1
ii  pgpdump  0.33-2
ii  poppler-utils20.09.0-3.1
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1.5
ii  python3-binwalk  2.3.1+dfsg1-1
ii  python3-debian   0.1.39
ii  python3-defusedxml   0.6.0-2
ii  python3-guestfs  1:1.44.0-2
ii  python3-jsondiff 1.1.1-4
ii  python3-pdfminer 20200726-1
ii  python3-progressbar  2.5-2
ii  python3-pypdf2   1.26.0-4
ii  python3-pyxattr  0.7.2-1+b1
ii  python3-rpm  4.16.1.2+dfsg1-0.4
ii  python3-tlsh 3.4.4+20151206-1.3+b4
pn  r-base-core  
pn  radare2  
ii  rpm2cpio 4.16.1.2+dfsg1-0.4
ii  sng  1.1.0-4
ii  sqlite3  3.34.1-3
ii  squashfs-tools   1:4.4-2
ii  tcpdump  4.99.0-2
ii  u-boot-tools 2021.01+dfsg-4
ii  unzip6.0-26
ii  vim-common   2:8.2.2434-3
pn  wabt 
pn  xmlbeans 
ii  xxd  2:8.2.2434-3
ii  xz-utils 5.2.5-2
ii  zip  3.0-12
ii  zstd 1.4.8+dfsg-2.1

Re: partial snapshot mirror amd64/bullseye/bookworm

[Paul Wise]

On Fri, Mar 5, 2021 at 8:26 AM Frédéric Pierret wrote:

> Is it possible to have an exported version somewhere for starting working on 
> the mirroring? To not bothering you currently, just a one shot export until 
> we are sure that would be absolutely necessary to have this daily export.

Please file a ticket with the Debian sysadmins.

https://wiki.debian.org/rt.debian.org#Debian_System_Administration

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: partial snapshot mirror amd64/bullseye/bookworm

[Paul Wise]

On Thu, Mar 4, 2021 at 8:57 AM Frédéric Pierret wrote:

> Following discussion on #debian-reproducible, would this be feasible from the 
> snapshot team to allow something like a daily export of the database on a 
> public exposed server?

Looking on the database server, the raw PostgreSQL tables are 38.7 GiB
total. For UDD the raw tables are 21.8 GiB while the export is 1.6 GB.

> Not necessary the whole, but only restricted suites and arches.

The snapshot database doesn't have information about suites/codenames,
that is why metasnap.d.n exists. Also architecture information is only
stored for .deb files.

https://salsa.debian.org/snapshot-team/snapshot/blob/master/db/db-create.sql

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: partial snapshot mirror amd64/bullseye/bookworm

[Paul Wise]

On Tue, 2021-03-02 at 10:58 +0100, Lucas Nussbaum wrote:

> - queries the snapshot DB to identify the files and hashes for each
>   source package
> - fetches and analyses Sources files to identify (source,version) of
>   interest, and thus hashes to transfer

You can probably replace this with the snapshot.d.o API and or metasnap.d.n.

> - transfers those hashes from snapshot.d.o to my own machine using rsync

and replace this with the snapshot.d.o API.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: partial snapshot mirror amd64/bullseye/bookworm

[Paul Wise]

On Mon, Mar 1, 2021 at 5:25 PM Holger Levsen wrote:

> > How would the mirroring work?
>
> to be discussed, but my raw idea would be to use rsync with excluding the 
> years
> before 2015 or 2017. or can't this work? 8-)

That won't work, since the filesystem storing the data is hash (SHA1)
based, so you need to look up hashes for the relevant data in the
database and then copy only those files.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: reproducible builds Debian usertags?

[Paul Wise]

On Sun, 2020-12-06 at 11:51 +0100, Mattia Rizzolo wrote: 

> Note that the formal explenation for those usertags is at
> https://reproducible-builds.org/contribute/#how-to-report-bugs-in-debian
> and https://wiki.debian.org/ReproducibleBuilds/UserCategory (though
> I'm not sure anymore if they are properly in sync, plus the latter is
> not linked anywhere)

Thanks for the info.

> depends on what that padding is about.  If it's due to different
> lenght of the build paths, then it should be "buildpath", IMHO.

Reading the bug/patch, it was about random unallocated memory from a
the padding of a struct ending up on disk.

> umask it is.

I'm not sure, the patch just normalised everything to 644?

> well, depends.  fileordering is specifically about readdir().  if
> it's, say, perl hashes not being ordered, that would be randomeness.

I suggested fileordering because of the patches on the bugs, the
ordering tags are definitely about readdir().


> not quite IMHO.  "kernel" is already tracked under "uname", whereas
> "timezone" likely falls into "timestamps"

I suggest reviewing the list of bugs for each to confirm this, which I
hadn't yet done before starting the thread (but did for other tags).

> these should all be "fileordering", the later "filesystem" is bogus.
> did you find it somewhere?

filesystem is an existing separate tag on index_bugs.html and h01ger
suggested on IRC that fileordering is not the same as filesystem but I
note that the filesystem tag isn't on the pages that you linked to.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

reproducible builds Debian usertags?

[Paul Wise]

Hi all,

[Please CC me on any replies you would like me to read]

As part of my efforts to do some QA on Debian usertags, I noticed a few
typos and other issues with the reproducible builds Debian usertags,
which I fixed using a script and the attached commands to the bts bot.

https://salsa.debian.org/pabs/qa/-/compare?from=master&to=usertags

I also noticed that the index of bugs is missing several usertags, I've
included below the missing usertags and bugs/counts for each of them.

https://tests.reproducible-builds.org/debian/index_bugs.html

   padding 1: #881231
   ASLR 1: #881231
   permissions 1: #858333
   ordering 2: #825406 #835263
   kernel 5: #849968 #858152 #884256 #970130 #976071
   timezones 1: #831583
   timezone 11: #834537 #835143 #840847 #841532 #841784 #843469 #855909 #872460 
#968627 #974911 #976359

It seems to me that:

 * ASLR tag should be removed since the bug is about padding/randomness
 * padding bug should maybe be removed in favour of randomness?
 * permissions bug should either stay or maybe move to umask?
 * timezones bugs should move to timezone
 * ordering bugs should move to fileordering
 * the kernel and timezone bugs should become official

Any thoughts on these suggestions?

PS: the typo or other fixes can be done by the usertags-fixes cron job
automatically if wanted, for now I elected to do them just the once,
but if they recur then I will notice them via another cron job.

PS: the current usertags can be obtained using this command:

rsync -a rsync://bugs-mirror.debian.org/bts-spool-index/user/*/*reprod* 
repro-usertags/

-- 
bye, pabs https://wiki.debian.org/PaulWise
user reproducible-bui...@lists.alioth.debian.org
usertags 848136 - fileorder
usertags 848136 + fileordering
usertags 872729 - filesystemordering
usertags 875700 - filesystemordering
usertags 875704 - filesystemordering
usertags 872729 + filesystem
usertags 875700 + filesystem
usertags 875704 + filesystem
thanks

user reproducible-bui...@lists.alioth.debian.org
usertags 835447 - build-path
usertags 835447 + buildpath
usertags 897902 - buildpaths
usertags 899358 - buildpaths
usertags 931102 - buildpaths
usertags 976449 - buildpaths
usertags 897902 + buildpath
usertags 899358 + buildpath
usertags 931102 + buildpath
usertags 976449 + buildpath
usertags 846842 - builpath
usertags 846842 + buildpath
usertags 942342 - toolchian
usertags 942342 + toolchain
usertags 788122 - timestamp
usertags 788122 + timestamps
usertags 829296 - randomeness
usertags 829296 + randomness
usertags 886523 - randoness
usertags 928288 - randoness
usertags 886523 + randomness
usertags 928288 + randomness
usertags 793469 - locales
usertags 793469 + locale
thanks

user reproducible-b...@lists.alioth.debian.org
usertags 914090 - uname
user reproducible-bui...@lists.alioth.debian.org
usertags 914090 + uname
user reproducible-b...@lists.alioth.debian.org
usertags 765277 - ftbfs
usertags 811766 - ftbfs
usertags 841360 - ftbfs
usertags 841556 - ftbfs
usertags 853532 - ftbfs
usertags 888192 - ftbfs
usertags 888194 - ftbfs
usertags 888422 - ftbfs
usertags 890257 - ftbfs
usertags 890905 - ftbfs
usertags 892156 - ftbfs
usertags 893547 - ftbfs
user reproducible-bui...@lists.alioth.debian.org
usertags 765277 + ftbfs
usertags 811766 + ftbfs
usertags 841360 + ftbfs
usertags 841556 + ftbfs
usertags 853532 + ftbfs
usertags 888192 + ftbfs
usertags 888194 + ftbfs
usertags 888422 + ftbfs
usertags 890257 + ftbfs
usertags 890905 + ftbfs
usertags 892156 + ftbfs
usertags 893547 + ftbfs
thanks

user reproduciblec-bui...@lists.alioth.debian.org
usertags 956120 - buildpath
user reproducible-bui...@lists.alioth.debian.org
usertags 956120 + buildpath
thanks

signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#964365: diffoscope: comparing squashfs filesystems: Could not open foo.squashfs, because No such file or directory

[Paul Wise]

Package: diffoscope
Version: 150
Severity: normal

When comparing squashfs filesystems I get an error saying the first
filesystem could not be found, even though I created it just before. 

$ mkdir foo bar
$ touch foo/foo bar/foo
$ chronic mksquashfs foo foo.squashfs
$ chronic mksquashfs bar bar.squashfs
$ diffoscope foo.squashfs bar.squashfs 
--- foo.squashfs
+++ bar.squashfs
│┄ Command `unsquashfs -n -f -no -li -d . foo.squashfs` failed with exit code 
1. Standard error:
│┄ Could not open foo.squashfs, because No such file or directory
@@ -1,8 +1,8 @@
-: 6873 7173 0200  eeac 025f  0200  hsqs..._
+: 6873 7173 0200  f1ac 025f  0200  hsqs..._
 0010:   0100 1100 c000 0100 0400   
 0020: 2000    c700      ...
 0030: bf00         
 0040: 6000    8c00     `...
 0050: a100    b100     
 0060: 2a00 78da 6362 58c0 c800 044f d730 c583  *.x.cbXO.0..
 0070: 1940 f01f 08a0 4c06 4686 1770 7926 a818  .@L.F..py&..

-- System Information:
Debian Release: bullseye/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.7.0-1-nouveau-extra-crash-debug-info-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND, TAINT_UNSIGNED_MODULE
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en 
(charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  python33.8.2-3
ii  python3-distro 1.5.0-1
ii  python3-distutils  3.8.3-2
ii  python3-libarchive-c   2.9-0.1
ii  python3-magic  2:0.4.15-4
ii  python3-pkg-resources  46.1.3-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.53-8
ii  apksigner0.9-1
ii  apktool  2.4.1-1
ii  binutils-multiarch   2.34-8
ii  bzip21.0.8-3
ii  caca-utils   0.99.beta19-2.1
ii  colord   1.4.4-2
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.45.6-1
ii  enjarify 1:1.0.3-4
ii  ffmpeg   7:4.3-2
ii  fontforge-extras 1:20190801~dfsg-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.1
ii  gettext  0.19.8.1-10
ii  ghc  8.8.1+dfsg1+is+8.6.5+dfsg1-3
ii  ghostscript  9.52~dfsg-1
ii  giflib-tools 5.1.9-1
ii  gnumeric 1.12.47-1
ii  gnupg2.2.20-1
ii  gnupg-utils  2.2.20-1
pn  hdf5-tools   
ii  imagemagick  8:6.9.10.23+dfsg-2.1+b2
ii  imagemagick-6.q16 [imagemagick]  8:6.9.10.23+dfsg-2.1+b2
ii  jsbeautifier 1.11.0-1
ii  libarchive-tools 3.4.3-1
ii  llvm 1:9.0-49.1
ii  lz4 [liblz4-tool]1.9.2-2
pn  mono-utils   
ii  ocaml-nox4.08.1-8
pn  odt2txt  
pn  oggvideotools
ii  openjdk-11-jdk [java-sdk]11.0.7+10-3
ii  openssh-client   1:8.3p1-1
ii  openssl  1.1.1g-1
ii  pgpdump  0.33-2
ii  poppler-utils0.71.0-6
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1.3
ii  python3-binwalk  2.2.0+dfsg1-1
ii  python3-debian   0.1.37
ii  python3-defusedxml   0.6.0-2
ii  python3-guestfs  1:1.42.0-6
ii  python3-jsondiff 1.1.1-4
pn  python3-pdfminer 
ii  python3-progressbar  2.5-2
pn  python3-pypdf2   
ii  python3-pyxattr  0.6.1-2+b1
ii  python3-rpm  4.14.2.1+dfsg1-1.1+b1
ii  python3-tlsh 3.4.4+20151206-1.3+b2
pn  r-base-core  
ii  rpm2cpio 4.14.2.1+dfsg1-1.1+b1
ii  sng  1.1.0-4
ii  sqlite3  3.32.3-1
ii  squashfs-tools   1:4.4-2
ii  tcpdump  4.9.3-6
ii  unzip   

Re: Storing build logs outside of .deb packages

[Paul Wise]

On Mon, 2020-06-01 at 20:53 +0200, Mattia Rizzolo wrote:

> But so, do you think the first step lies in pbuilder/sbuild to decide an
> interface for the package maintainers to use, or for the buildd managers
> (wanna-build/buildd and launchpad) to decide on an interface that
> sbuild needs to provide (and I would guess pbuilder will try to copy it
> for consistency).

My initial thought was that none of the existing codebases should be
doing this. Instead dsa-puppet should get sbuild post-build hooks that
tar up a configurable set of build artefacts and transfer those to
buildd.d.o to be displayed alongside build logs. The list of build
artefact path patterns would be maintained in wanna-build/launchpad,
transferred to each buildd before each build, matched after each build,
the matches passed to tar and the tarball uploaded alongside the log.

Since then I talked about this with aurel32/jcristau on #debian-buildd
and they reached the conclusion that this should be a per-package thing
left up to maintainers of packages rather than a global thing done by
buildd management systems like wanna-build/launchpad. They thought the
audience for this was mainly individual package maintainers rather than the 
porters, maintainers, RC bug fixers, NMUers and everyone else.

All that said, I am not really one of the stakeholders in this
discussion so I'll just follow along as a lurker from now on.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: Storing build logs outside of .deb packages

[Paul Wise]

On Mon, 2020-06-01 at 11:12 +0200, Mattia Rizzolo wrote:

> I know this has been linked in multiple places, but I'll link it
> again:
> https://bugs.launchpad.net/launchpad/+bug/1845159
> there has also been a relevant thread in debian-devel a few months
> back, spawned off the debian/.build thread started by Niels Thykier.

Here is the thread you have mentioned:

https://lists.debian.org/msgid-search/5f47b00f-50f0-98b2-776a-28ead95ad...@thykier.net

> I'm not quite clear what's the best way forward: I could go and
> implement whatever feels best to me in pbuilder, and then hope the
> rest of the tooling with catch up, copy my idea and adapt, but that
> feels unlikely…

Re-reading the above thread, I now think that debhelper/dpkg-dev is the
wrong layer to be implementing this and that we want it in the hooks or
config for the build system in use; launchpad, sbuild, pbuilder etc.

The reason is that it needs to be able to evolve more rapidly than the
codebases of any of the tools that the builds use, especially in
response to maintainer requests after they notice new build failures.

So, I would suggest buildd.d.o folks and launchpad folks would be the
maintainers of the rules about artefacts to preserve and that the
artefacts be published in the same place as the build logs. Probably
for Debian, DSA would need to add to the buildd puppet module a way for
the buildd admins to specify a set of globs for files to preserve.

Sorry for getting side-tracked by BYHAND stuff.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: binutils-dev: included log files introduce reproducibility issues

[Paul Wise]

On Sun, 2020-05-17 at 13:12 +0200, Matthias Klose wrote: 

> Any idea which package this should be filed for?

I think this is going to require a bunch of different changes in
various places including dpkg-dev, debhelper, reprepro, dak, launchpad
and any other apt repos that have an incoming system. As long as the
files are mentioned in the .changes files dupload/dput won't change. So
probably there needs to be a discussion on debian-devel CCed to the
debian-dpkg list and to the ftp-masters and other folks.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#960814: diffoscope: comparing an unsigned and a signed buildinfo file gives a hex diff

[Paul Wise]

Package: diffoscope
Version: 144
Severity: minor

When I compare the attached files (one unsigned buildinfo file, one
signed buildinfo file) I get a hex diff. When I strip away the
signature, the comparison is a text diff as normal. The optimal
comparison would be to compare the missing container with the signed
container and then within that compare the two buildinfo files.

The same issue applies to comparing a text file with a tarball
containing the same text file, so I'm assuming diffoscope cannot yet
compare files across container boundaries?

-- System Information:
Debian Release: bullseye/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.6.0-1-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), LANGUAGE=en_AU:en 
(charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  python33.8.2-3
ii  python3-distro 1.5.0-1
ii  python3-distutils  3.8.2-2
ii  python3-libarchive-c   2.9-0.1
ii  python3-magic  2:0.4.15-4
ii  python3-pkg-resources  46.1.3-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.53-8
ii  apktool  2.4.1-1
ii  binutils-multiarch   2.34-5
ii  bzip21.0.8-2
ii  caca-utils   0.99.beta19-2.1
ii  colord   1.4.4-2
ii  db-util  5.3.1+nmu1
ii  default-jdk [java-sdk]   2:1.11-72
ii  default-jdk-headless 2:1.11-72
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.45.6-1
ii  enjarify 1:1.0.3-4
ii  ffmpeg   7:4.2.2-1+b1
ii  fontforge-extras 1:20190801~dfsg-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3.1
ii  gettext  0.19.8.1-10
ii  ghc  8.8.1+dfsg1+is+8.6.5+dfsg1-3
ii  ghostscript  9.52~dfsg-1
ii  giflib-tools 5.1.9-1
ii  gnumeric 1.12.46-1
ii  gnupg2.2.20-1
ii  gnupg-utils  2.2.20-1
pn  hdf5-tools   
ii  imagemagick  8:6.9.10.23+dfsg-2.1+b2
ii  imagemagick-6.q16 [imagemagick]  8:6.9.10.23+dfsg-2.1+b2
ii  jsbeautifier 1.11.0-1
ii  libarchive-tools 3.4.2-1
ii  llvm 1:9.0-49.1
ii  lz4 [liblz4-tool]1.9.2-2
pn  mono-utils   
ii  ocaml-nox4.08.1-8
pn  odt2txt  
pn  oggvideotools
ii  openjdk-11-jdk [java-sdk]11.0.7+10-3
ii  openssh-client   1:8.2p1-4
ii  openssl  1.1.1g-1
ii  pgpdump  0.33-2
ii  poppler-utils0.71.0-6
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1.3
ii  python3-binwalk  2.2.0+dfsg1-1
ii  python3-debian   0.1.37
ii  python3-defusedxml   0.6.0-2
ii  python3-guestfs  1:1.42.0-1
ii  python3-jsondiff 1.1.1-4
pn  python3-pdfminer 
ii  python3-progressbar  2.5-2
pn  python3-pypdf2   
ii  python3-pyxattr  0.6.1-2+b1
ii  python3-tlsh 3.4.4+20151206-1.3+b2
pn  r-base-core  
ii  rpm2cpio 4.14.2.1+dfsg1-1.1+b1
ii  sng  1.1.0-3
ii  sqlite3  3.31.1-5
ii  squashfs-tools   1:4.4-2
ii  tcpdump  4.9.3-5
ii  unzip6.0-25
ii  vim-common   2:8.2.0716-3
pn  wabt 
pn  xmlbeans 
ii  xxd  2:8.2.0716-3
ii  xz-utils 5.2.4-1+b1
ii  zip  3.0-11+b1
ii  zstd 1.4.4+dfsg-3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.5.1+dfsg-3

-- no debconf information

-- 
bye,
pabs

https://wiki.debian.org/PaulWise
Format: 1.0
Source: whowatch
Binary: whowatch
Architecture: amd64 source
Version: 1.8.6-1
Checksums-Md5:
 f7227fdeb981ecfc22bcf9b0393cc7a3 1089 whowatch_1.8.6-1.dsc
 0eb939de8b126cae7670a6a9ea80a6b7 80600 whowatch-dbgsym_1.8.6-1_amd64.deb
 b6d241b59dba924eb944a1a15accc5dd 17335 whow

Re: binutils-dev: included log files introduce reproducibility issues

[Paul Wise]

On Sat, 2020-05-16 at 22:58 +, Chris Lamb wrote:

> My gut feeling is that this is the avenue we want to explore. Having a
> separate mechanism to capture this build-specific metadata would be an
> elegant solution and, as you imply, having the logs would have QA
> advantages as well as permit reproducible builds. The system could be
> generic enough for future use-cases that we cannot envisage too.

Agreed that this is the best option. In order to standardise the
naming, structure and organisation of the data in order to make it work
across all Debian derivatives, probably a new conversation needs to be
started between ftp-master and the dpkg maintainer.

> We have taken great pains over the years that no special knowledge,
> tools or tricks are required to compare the artifacts of a Debian
> build with respect to reproducibility.

I'm not involved in repro builds enough so your statement leads me to
wonder how you deal with ignoring the inevitable differences between
the buildinfo files, which would record the inevitable differences in
build environment between different builders. I'm guessing you just
ignore all differences in buildinfo files and would have to add to that
ignoring differences between build logs and other build metadata?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Re: binutils-dev: included log files introduce reproducibility issues

[Paul Wise]

On Mon, 24 Feb 2020 14:09:23 -0800 Vagrant Cascadian wrote:

> Exploring avenues to put files like this into some separate artifact for
> things that are not reproducible might be one avenue

There is already the BYHAND (and automatic BYHAND) mechanisms for files
that get installed outside of pool/ in the Debian apt repository. Each
one needs support from dak too though.

https://salsa.debian.org/ftp-team/dak/-/tree/master/scripts/debian/
https://codesearch.debian.net/search?q=byhand+path%3Adebian&literal=0

The other option would be to put the files in a test results .deb file,
but that would still mean repro-builds folks would compare them, unless
there were a naming convention that could be used to ignore them.

It strikes me that these files are most similar to .buildinfo or the
build logs in that they are data *about* the builds. I've wanted
maintainers to be able to also upload build logs with their binary
builds and started a WIP patch for that.

https://salsa.debian.org/pabs/dak/-/commits/maintainer-build-logs

It looks like dpkg-genchanges can already add more files through the
debian/files input file. I managed to get my build log included in my
.changes file using this mechanism. I then attempted to upload it to
the Debian archive, queued gladly accepted the upload but dak rejected
it saying that it looks like a BYHAND package:

   whowatch_1.8.6-2_amd64.changes: whowatch_1.8.6-1_amd64.build.log looks like 
a byhand package, but is in section build

I suggest that the dpkg-dev maintainer and the ftp-masters should be
talked to about this topic. Probably the right mechanism is to have a
convention in debian/files similar to how dbgsyms are represented and
similar to byhand but the files go into the pool like .deb files do.

   whowatch_1.8.6-1_amd64.build build optional automatic=yes
   whowatch-dbgsym_1.8.6-2_amd64.deb debug optional automatic=yes

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#908994: diffoscope: extract information from OpenPGP keys and signatures

[Paul Wise]

On Tue, 2018-11-27 at 03:11 -0500, Daniel Kahn Gillmor wrote:

> for machine-parseable stuff or automated processing, you should also
> probably supply --with-colons.

In this case we want human-readable output since this is for diffs.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#908994: diffoscope: extract information from OpenPGP keys and signatures

[Paul Wise]

On Mon, 2018-11-26 at 16:31 -0500, Chris Lamb wrote:

> I added support for pgpdump in diffoscope version 82 via #859034.
> Had you seen this?

I hadn't noticed that yet, thanks for that.

> I do note that your bug mentions some other tools (name gpg --list-
> packets, hot dump and hot dearmor) however. Are there any of these you
> would suggest we prioritise?

I don't recall if any of the tools are more useful than the others but
playing around just now I see only minor differences. I'd suggest
prioritising supporting gpg because that is more likely to be already
installed on more systems than pgpdump is. The hopenpgp-tools package
is more obscure and less likely to be installed as it is Haskell so
might not be available on some platforms that gpg/pgpdump are and it
also seems a bit less capable than gpg/pgpdump so maybe ignore it.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#908992: diffoscope: side-by-side diff for terminal output

[Paul Wise]

On Mon, 2018-11-26 at 16:58 -0500, Chris Lamb  wrote:

> I think this could be made to work.
> As a demo, I just knocked together:
> ... which resulted in the attached.

Looks good.

> One difficulty however will be in passing in a sane value of `cols` to
> ConsoleDiff; not because it's difficult to obtain (see icdiff.py for
> this terminal foo) but rather we need to subtract 2 or 3 for each
> indentation/recursion level and we may not know that at the time we
> run icdiff.

Hmm, I would have thought the recursion level could be determined at
the time the data is being extracted from the data structure? How is
the recursion level determined right now?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#908994: diffoscope: extract information from OpenPGP keys and signatures

[Paul Wise]

Package: diffoscope
Version: 101
Severity: wishlist

When comparing OpenPGP keys or signatures (binary and ASCII-armoured),
either as raw files or embedded in other formats (like .dsc files),
it would be nice to compare the internal data of those keys and sigs.

There are some different ways to output textual data from these formats:

pgpdump:

pgpdump file
pgpdump < data

gnupg:

gpg --list-packets file
gpg --list-packets < data

For keys only:

gpg --home $(mktemp -d) file

hopenpgp-tools:

hot dump < data
hot dearmor < data | hot dump

For keys only:

hot dearmor < data | hokey lint

-- System Information:
Debian Release: buster/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.18.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), 
LANGUAGE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib3.6.6-1
ii  python33.6.5-3
ii  python3-distro 1.3.0-1
ii  python3-distutils  3.6.6-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  2:0.4.15-2
ii  python3-pkg-resources  40.2.0-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.52-3+b1
ii  apktool  2.3.3-2
ii  binutils-multiarch   2.31.1-5
ii  bzip21.0.6-9
ii  caca-utils   0.99.beta19-2+b3
ii  colord   1.3.3-2
ii  db-util  5.3.1
ii  default-jdk [java-sdk]   2:1.10-68
ii  default-jdk-headless 2:1.10-68
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.44.4-2
ii  enjarify 1:1.0.3-4
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-7
ii  ghc  8.2.2-4
ii  ghostscript  9.25~dfsg-2
ii  giflib-tools 5.1.4-3
ii  gnumeric 1.12.41-1
ii  gnupg2.2.10-1
ii  imagemagick  8:6.9.10.8+dfsg-1
ii  imagemagick-6.q16 [imagemagick]  8:6.9.10.8+dfsg-1
ii  jsbeautifier 1.6.4-7
pn  libarchive-tools 
ii  llvm 1:6.0-43
ii  lz4  1.8.2-1
pn  mono-utils   
pn  odt2txt  
pn  oggvideotools
ii  openjdk-10-jdk [java-sdk]10.0.2+13-1
ii  openssh-client   1:7.8p1-1
ii  pgpdump  0.33-1
ii  poppler-utils0.63.0-2
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1
ii  python3-binwalk  2.1.2~git20180830+dfsg1-1
ii  python3-debian   0.1.33
pn  python3-defusedxml   
pn  python3-guestfs  
ii  python3-jsondiff 1.1.1-2
ii  python3-progressbar  2.3-4
ii  python3-pyxattr  0.6.0-2+b2
ii  python3-tlsh 3.4.4+20151206-1+b4
pn  r-base-core  
ii  rpm2cpio 4.14.1+dfsg1-4
ii  sng  1.1.0-1+b1
ii  sqlite3  3.24.0-1
ii  squashfs-tools   1:4.3-6
ii  tcpdump  4.9.2-3
ii  unzip6.0-21
ii  vim-common   2:8.1.0320-1
pn  xmlbeans 
ii  xxd  2:8.1.0320-1
ii  xz-utils 5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

-- no debconf information

-- 
bye,
pabs

https://wiki.debian.org/PaulWise



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#908993: diffoscope: use colour to highlight intra-line differences in the terminal output too

[Paul Wise]

Package: diffoscope
Version: 101
Severity: wishlist

diffoscope uses colour to highlight intra-line differences in the HTML
output but the terminal output only does a colordiff-style output. It
would be nice if the terminal mode could highlight intra-line changes.

The package ccdiff contains a tool that can do intra-line difference
highlighting with `diff -u` style output, but it is written in Perl.

For example, diffoscope only highlights full lines:

$ diffoscope gtranscribe_0.7.1-2.dsc gtranscribe_0.8-1.dsc
--- gtranscribe_0.7.1-2.dsc
+++ gtranscribe_0.8-1.dsc
├── Build-Depends
│ @@ -1 +1 @@
│ -debhelper (>= 10), dh-python, python3-all, python3-dbus, 
python3-distutils-extra, python3-gi
│ +debhelper (>= 11), dh-python, python3-all, python3-dbus, 
python3-distutils-extra, python3-gi

$ ccdiff -u gtranscribe_0.7.1-2.dsc gtranscribe_0.8-1.dsc
...
-Build-Depends: debhelper (>= 10), dh-python, python3-all, python3-dbus, 
python3-distutils-extra, python3-gi
+Build-Depends: debhelper (>= 11), dh-python, python3-all, python3-dbus, 
python3-distutils-extra, python3-gi
   ^
   \ Only this character is highlighted

-- System Information:
Debian Release: buster/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.18.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), 
LANGUAGE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib3.6.6-1
ii  python33.6.5-3
ii  python3-distro 1.3.0-1
ii  python3-distutils  3.6.6-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  2:0.4.15-2
ii  python3-pkg-resources  40.2.0-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.52-3+b1
ii  apktool  2.3.3-2
ii  binutils-multiarch   2.31.1-5
ii  bzip21.0.6-9
ii  caca-utils   0.99.beta19-2+b3
ii  colord   1.3.3-2
ii  db-util  5.3.1
ii  default-jdk [java-sdk]   2:1.10-68
ii  default-jdk-headless 2:1.10-68
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.44.4-2
ii  enjarify 1:1.0.3-4
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-7
ii  ghc  8.2.2-4
ii  ghostscript  9.25~dfsg-2
ii  giflib-tools 5.1.4-3
ii  gnumeric 1.12.41-1
ii  gnupg2.2.10-1
ii  imagemagick  8:6.9.10.8+dfsg-1
ii  imagemagick-6.q16 [imagemagick]  8:6.9.10.8+dfsg-1
ii  jsbeautifier 1.6.4-7
pn  libarchive-tools 
ii  llvm 1:6.0-43
ii  lz4  1.8.2-1
pn  mono-utils   
pn  odt2txt  
pn  oggvideotools
ii  openjdk-10-jdk [java-sdk]10.0.2+13-1
ii  openssh-client   1:7.8p1-1
ii  pgpdump  0.33-1
ii  poppler-utils0.63.0-2
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1
ii  python3-binwalk  2.1.2~git20180830+dfsg1-1
ii  python3-debian   0.1.33
pn  python3-defusedxml   
pn  python3-guestfs  
ii  python3-jsondiff 1.1.1-2
ii  python3-progressbar  2.3-4
ii  python3-pyxattr  0.6.0-2+b2
ii  python3-tlsh 3.4.4+20151206-1+b4
pn  r-base-core  
ii  rpm2cpio 4.14.1+dfsg1-4
ii  sng  1.1.0-1+b1
ii  sqlite3  3.24.0-1
ii  squashfs-tools   1:4.3-6
ii  tcpdump  4.9.2-3
ii  unzip6.0-21
ii  vim-common   2:8.1.0320-1
pn  xmlbeans 
ii  xxd  2:8.1.0320-1
ii  xz-utils 5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

-- no debconf information


-- 
bye,
pabs

https://wiki.debian.org/PaulWise



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alio

Bug#908992: diffoscope: side-by-side diff for terminal output

[Paul Wise]

Package: diffoscope
Version: 101
Severity: wishlist

The HTML output uses a side-by-side diff but the terminal output uses
`diff -u` style output. It would be nice to have an option for
side-by-side diffs with the terminal output. The icdiff package
contains a Python 3 module that could be used for this.

-- System Information:
Debian Release: buster/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.18.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), 
LANGUAGE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib3.6.6-1
ii  python33.6.5-3
ii  python3-distro 1.3.0-1
ii  python3-distutils  3.6.6-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  2:0.4.15-2
ii  python3-pkg-resources  40.2.0-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.52-3+b1
ii  apktool  2.3.3-2
ii  binutils-multiarch   2.31.1-5
ii  bzip21.0.6-9
ii  caca-utils   0.99.beta19-2+b3
ii  colord   1.3.3-2
ii  db-util  5.3.1
ii  default-jdk [java-sdk]   2:1.10-68
ii  default-jdk-headless 2:1.10-68
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.44.4-2
ii  enjarify 1:1.0.3-4
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-7
ii  ghc  8.2.2-4
ii  ghostscript  9.25~dfsg-2
ii  giflib-tools 5.1.4-3
ii  gnumeric 1.12.41-1
ii  gnupg2.2.10-1
ii  imagemagick  8:6.9.10.8+dfsg-1
ii  imagemagick-6.q16 [imagemagick]  8:6.9.10.8+dfsg-1
ii  jsbeautifier 1.6.4-7
pn  libarchive-tools 
ii  llvm 1:6.0-43
ii  lz4  1.8.2-1
pn  mono-utils   
pn  odt2txt  
pn  oggvideotools
ii  openjdk-10-jdk [java-sdk]10.0.2+13-1
ii  openssh-client   1:7.8p1-1
ii  pgpdump  0.33-1
ii  poppler-utils0.63.0-2
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1
ii  python3-binwalk  2.1.2~git20180830+dfsg1-1
ii  python3-debian   0.1.33
pn  python3-defusedxml   
pn  python3-guestfs  
ii  python3-jsondiff 1.1.1-2
ii  python3-progressbar  2.3-4
ii  python3-pyxattr  0.6.0-2+b2
ii  python3-tlsh 3.4.4+20151206-1+b4
pn  r-base-core  
ii  rpm2cpio 4.14.1+dfsg1-4
ii  sng  1.1.0-1+b1
ii  sqlite3  3.24.0-1
ii  squashfs-tools   1:4.3-6
ii  tcpdump  4.9.2-3
ii  unzip6.0-21
ii  vim-common   2:8.1.0320-1
pn  xmlbeans 
ii  xxd  2:8.1.0320-1
ii  xz-utils 5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

-- no debconf information

-- 
bye,
pabs

https://wiki.debian.org/PaulWise



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#908991: diffoscope: diffs hex dump of ASCII-armoured OpenPGP upstream signature files instead of plain text

[Paul Wise]

Package: diffoscope
Version: 101
Severity: normal

diffoscope currently diffs a hex dump of ASCII-armoured OpenPGP
upstream signature files instead of comparing the plain text.

The ASCII-armoured OpenPGP signature files are plain text containing
base64-encoded signatures so it makes more sense to diff the plain text
than a hex dump of the plain text.

For example:

$ dget -dq 
https://snapshot.debian.org/archive/debian/20161231T091558Z/pool/main/g/gtranscribe/gtranscribe_0.7.1-1.dsc
$ dget -dq 
https://snapshot.debian.org/archive/debian/20161206T091948Z/pool/main/g/gtranscribe/gtranscribe_0.6-1.dsc

$ diffoscope gtranscribe_0.6-1.dsc gtranscribe_0.7.1-1.dsc
...
│   --- gtranscribe_0.6.orig.tar.gz.asc
├── +++ gtranscribe_0.7.1.orig.tar.gz.asc
│ @@ -1,53 +1,53 @@
│  : 2d2d 2d2d 2d42 4547 494e 2050 4750 2053  -BEGIN PGP S
│  0010: 4947 4e41 5455 5245 2d2d 2d2d 2d0a 0a69  IGNATURE-..i
│  0020: 5149 7a42 4141 4243 6741 6446 6945 454b  QIzBAABCgAdFiEEK
│  0030: 396a 5534 3565 5658 3364 4732 7a75 4a72  9jU45eVX3dG2zuJr
│ -0040: 576b 576c 6e4f 546d 4373 4641 6c68 4556  WkWlnOTmCsFAlhEV
│ -0050: 6e49 4143 676b 5172 576b 576c 6e4f 540a  nIACgkQrWkWlnOT.
│ -0060: 6d43 765a 6577 2f2b 4e7a 3434 3034 4e74  mCvZew/+Nz4404Nt
│ -0070: 6458 4253 584b 394a 6668 4c43 386e 787a  dXBSXK9JfhLC8nxz
│ -0080: 7754 4c51 4246 5778 5438 4653 774f 5054  wTLQBFWxT8FSwOPT
│ -0090: 4879 5335 5934 306f 5763 497a 7544 316a  HyS5Y40oWcIzuD1j
│ -00a0: 0a7a 4763 714e 6747 7947 5044 4a5a 3134  .zGcqNgGyGPDJZ14
│ -00b0: 6262 6536 716d 6f49 4166 3657 2b4b 4d5a  bbe6qmoIAf6W+KMZ
│ -00c0: 4230 5044 6c32 4337 6853 6951 522b 7975  B0PDl2C7hSiQR+yu
│ -00d0: 4b36 396d 4530 5858 564f 4e50 4835 5538  K69mE0XXVONPH5U8
│ -00e0: 480a 2f74 7677 5a2f 7242 764f 5370 3371  H./tvwZ/rBvOSp3q
│ -00f0: 4930 6942 5877 4450 4445 4d48 7a67 2f72  I0iBXwDPDEMHzg/r
│ -0100: 3572 6c54 6732 442b 5976 7255 3236 6b49  5rlTg2D+YvrU26kI
│ -0110: 794b 6c68 7734 3063 7050 4f43 3730 6754  yKlhw40cpPOC70gT
│ -0120: 4170 0a6c 6c45 2b51 4a4b 6a44 4359 6933  Ap.llE+QJKjDCYi3
│ -0130: 3957 4438 5a67 4c56 4a2f 7853 646b 4630  9WD8ZgLVJ/xSdkF0
│ -0140: 3549 4330 7142 6872 4346 6563 6370 5370  5IC0qBhrCFeccpSp
│ -0150: 7372 4f70 4d62 5247 5262 2f68 3564 6c6d  srOpMbRGRb/h5dlm
│ -0160: 534e 620a 6b72 7148 2f30 4e77 7133 7546  SNb.krqH/0Nwq3uF
│ -0170: 4d69 7768 5438 5759 3064 474b 384e 5453  MiwhT8WY0dGK8NTS
│ -0180: 7356 4578 6377 3475 3235 5450 6565 7956  sVExcw4u25TPeeyV
│ -0190: 6834 377a 4f34 2f75 7247 6d4d 3762 582f  h47zO4/urGmM7bX/
│ -01a0: 5844 6850 0a44 4c6f 6834 4570 4e54 327a  XDhP.DLoh4EpNT2z
│ -01b0: 3941 456e 6862 4867 766b 4579 5268 4d56  9AEnhbHgvkEyRhMV
│ -01c0: 756a 326f 4d62 3677 364a 4e6a 6638 6659  uj2oMb6w6JNjf8fY
│ -01d0: 6741 5772 652b 6552 5557 6a63 6246 7733  gAWre+eRUWjcbFw3
│ -01e0: 5579 6130 390a 652f 4645 702b 5276 4642  Uya09.e/FEp+RvFB
│ -01f0: 6877 457a 6b45 3959 6b65 504c 3464 347a  hwEzkE9YkePL4d4z
│ -0200: 5543 566a 776c 346a 7675 5438 474a 6b4e  UCVjwl4jvuT8GJkN
│ -0210: 4173 4d34 584a 5537 4d57 6b65 5963 622f  AsM4XJU7MWkeYcb/
│ -0220: 7a5a 7a2f 6570 0a72 7767 6d36 4454 3146  zZz/ep.rwgm6DT1F
│ -0230: 496c 7756 5654 7256 5950 2b51 636d 7935  IlwVVTrVYP+Qcmy5
│ -0240: 446c 3870 746e 3972 7667 6135 7759 4e6b  Dl8ptn9rvga5wYNk
│ -0250: 6977 4249 566a 5146 5a76 4d6a 6364 4d44  iwBIVjQFZvMjcdMD
│ -0260: 6a65 7551 4937 730a 4871 4e4f 4978 5348  jeuQI7s.HqNOIxSH
│ -0270: 356a 5770 5478 7439 6b73 6932 6d4f 7177  5jWpTxt9ksi2mOqw
│ -0280: 6f6b 2f76 5359 6d35 697a 4e78 5378 5565  ok/vSYm5izNxSxUe
│ -0290: 4d44 5251 4a49 6154 7459 4347 4477 6c37  MDRQJIaTtYCGDwl7
│ -02a0: 7563 6156 774b 5235 0a2b 6756 4f35 6367  ucaVwKR5.+gVO5cg
│ -02b0: 7542 6866 686a 2f44 6e4f 5078 4551 614b  uBhfhj/DnOPxEQaK
│ -02c0: 6742 4f65 7430 6966 4f4d 314e 5a51 3459  gBOet0ifOM1NZQ4Y
│ -02d0: 704c 6367 5445 7a36 4a76 7132 7a4b 546d  pLcgTEz6Jvq2zKTm
│ -02e0: 3139 3068 7945 6f4b 690a 7951 6849 6a4e  190hyEoKi.yQhIjN
│ -02f0: 5834 6d6a 792f 516b 7931 7639 4955 6a5a  X4mjy/Qky1v9IUjZ
│ -0300: 5973 3879 6b38 4141 5274 2f63 6279 3831  Ys8yk8AARt/cby81
│ -0310: 7166 3775 314d 6a77 6849 3457 413d 0a3d  qf7u1MjwhI4WA=.=
│ -0320: 5157 3471 0a2d 2d2d 2d2d 454e 4420 5047  QW4q.-END PG
│ +0040: 576b 576c 6e4f 546d 4373 4641 6c68 6d68  WkWlnOTmCsFAlhmh
│ +0050: 4130 4143 676b 5172 576b 576c 6e4f 540a  A0ACgkQrWkWlnOT.
│ +0060: 6d43 755a 6668 4141 6a41 6b6d 4151 3056  mCuZfhAAjAkmAQ0V
│ +0070: 6877 654f 4d38 4446 4950 5366 4b64 5043  hweOM8DFIPSfKdPC
│ +0080: 3637 5a2f 7477 6235 3468 6652 365a 5143  67Z/twb54hfR6ZQC
│ +0090: 5a6b 3941 4e4b 344d 5750 6776 5178 7733  Zk9ANK4MWPgvQxw3
│ +00a0: 0a67 4469 3064 4766 6b41 654d 6671 5a4d  .gDi0dGfkAeMfqZM
│ +00b0: 7072 5246 3366 7264 7362 656b 4153 4c33  prRF3frdsbekASL3
│ +00c0: 6c33 6246 7078 3274 7463

Bug#903535: bls: warn about -mtune=native in build logs

[Paul Wise]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: bls
X-Debbugs-CC: reproducible-bui...@lists.alioth.debian.org, brl...@debian.org
Control: user reproducible-bui...@lists.alioth.debian.org
Control: usertags -1 + infrastructure
Control: clone -1 -2
Control: reassign -2 bls-standalone

The Debian build log scanner (bls)[1] already detects[2] packages that
use -march=native in their build logs. This flag was added because it
breaks packages for users with older CPUs.

It is also important to detect for reproducible builds[3] purposes,
since it means the build can differ depending on the features supported
by the build CPU. 

Additionally, the -mtune=native flag can vary the build based on the
build CPU but not break it for users with older CPUs. This makes it an
important flag to detect for reproducible builds purposes too.

   1. https://qa.debian.org/bls/
   2. https://qa.debian.org/bls/bytag/E-march-native.html
   3. https://codesearch.debian.net/search?perpkg=1&q=-mtune=native
   4. https://reproducible-builds.org/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#901758: diffoscope: leaves progress bar on terminal after completion

[Paul Wise]

On Sat, 2018-07-07 at 11:26 +0100, Chris Lamb wrote:

> Thanks, applied.

Attached a couple more patches.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise
From 2c95bd3ec8e17c54fc2f8a1c4a1735d17df7ab55 Mon Sep 17 00:00:00 2001
From: Paul Wise 
Date: Sat, 7 Jul 2018 16:31:24 +0800
Subject: [PATCH 1/2] Use the ProgressBar's idea of the terminal width instead

More likely to overwrite the correct number of characters.

Fixes: commit dfb0c190173dff953fe1481797486d0b3e09306e
---
 diffoscope/progress.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/diffoscope/progress.py b/diffoscope/progress.py
index 286f9d9..5edfe84 100644
--- a/diffoscope/progress.py
+++ b/diffoscope/progress.py
@@ -232,10 +232,8 @@ class ProgressBar(object):
 if self.erase_to_eol:
 self.fd.buffer.write(self.erase_to_eol)
 elif self.fd.isatty():
-from shutil import get_terminal_size
-width = get_terminal_size().columns
 print(end='\r', file=self.fd)
-print(' ' * width, end='', file=self.fd)
+print(' ' * self.term_width, end='', file=self.fd)
 else:
 # Do not flush if nothing was written
 return
-- 
2.18.0

From ed9cfc95bce9fc08aa5d0ac0d2ad1b709e9cb108 Mon Sep 17 00:00:00 2001
From: Paul Wise 
Date: Sat, 7 Jul 2018 16:13:35 +0800
Subject: [PATCH 2/2] Do not delete the current terminal line for every
 progress bar update

The progress bar already overwrites the entire line
so erasing it before that serves no purpose.

The erasure was also causing the progress bar to flicker
and wasn't portable to different types of terminals.
---
 diffoscope/progress.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/diffoscope/progress.py b/diffoscope/progress.py
index 5edfe84..c0e140c 100644
--- a/diffoscope/progress.py
+++ b/diffoscope/progress.py
@@ -34,9 +34,6 @@ class ProgressLoggingHandler(logging.StreamHandler):
 
 def emit(self, record):
 try:
-# Delete the current line (i.e. the progress bar)
-self.stream.write("\r\033[K")
-self.flush()
 super().emit(record)
 if not self.progressbar.bar.finished:
 self.progressbar.bar.update()
-- 
2.18.0



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#901758: diffoscope: leaves progress bar on terminal after completion

[Paul Wise]

On Sat, 2018-07-07 at 07:45 +0100, Chris Lamb wrote:

> I wasn't aware that was a possibility.

Yeah, different terminals have different capabilities.
Most modern ones of course can erase to EOL though.

> If you mean via --text=filename then the progress bar does not
> appear there.

I was thinking diffoscope &> output shell redirection.

> In that case, please could you provide a merge request?

I'm not interested in touching gitlab, patches attached.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise
From 9fa755c81033148cc1a5a49bc4d2fe865ec6e0c8 Mon Sep 17 00:00:00 2001
From: Paul Wise 
Date: Sat, 7 Jul 2018 16:13:35 +0800
Subject: [PATCH 1/3] Do not delete the current terminal line for every
 progress bar update

The progress bar already overwrites the entire line
so erasing it before that serves no purpose.

The erasure was also causing the progress bar to flicker.
---
 diffoscope/progress.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/diffoscope/progress.py b/diffoscope/progress.py
index e60188c..a584ef7 100644
--- a/diffoscope/progress.py
+++ b/diffoscope/progress.py
@@ -32,9 +32,6 @@ class ProgressLoggingHandler(logging.StreamHandler):
 
 def emit(self, record):
 try:
-# Delete the current line (i.e. the progress bar)
-self.stream.write("\r\033[K")
-self.flush()
 super().emit(record)
 if not self.progressbar.bar.finished:
 self.progressbar.bar.update()
-- 
2.18.0

From f386b5e3dc5a5ebbc0dc06720b56f8af33ea7c20 Mon Sep 17 00:00:00 2001
From: Paul Wise 
Date: Sat, 7 Jul 2018 16:31:24 +0800
Subject: [PATCH 2/3] Clear the progress bar after completion. (Closes:
 #901758)

Handle terminals that do not support erasing the line by
filling the terminal with spaces.

Ignore output devices that are not terminals.

Implements: https://bugs.debian.org/901758
Requires: Python 3.2 for shutil.get_terminal_size
---
 diffoscope/progress.py | 30 ++
 1 file changed, 30 insertions(+)

diff --git a/diffoscope/progress.py b/diffoscope/progress.py
index a584ef7..215bb23 100644
--- a/diffoscope/progress.py
+++ b/diffoscope/progress.py
@@ -3,6 +3,7 @@
 # diffoscope: in-depth comparison of files, archives, and directories
 #
 # Copyright © 2016 Chris Lamb 
+# Copyright © 2018 Paul Wise 
 #
 # diffoscope is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -20,6 +21,7 @@
 import os
 import sys
 import json
+import signal
 import logging
 
 logger = logging.getLogger(__name__)
@@ -212,10 +214,38 @@ class ProgressBar(object):
 # Remove after https://github.com/niltonvolpato/python-progressbar/pull/57 is fixed.
 kwargs.setdefault('fd', sys.stderr)
 super().__init__(*args, **kwargs)
+# Terminal handling after parent init since that sets self.fd
+if self.fd.isatty():
+from curses import tigetstr, setupterm
+setupterm()
+self.erase_to_eol = tigetstr('el')
+else:
+self.erase_to_eol = None
 
 def _need_update(self):
 return True
 
+def erase_line(self):
+if self.erase_to_eol:
+self.fd.buffer.write(self.erase_to_eol)
+elif self.fd.isatty():
+from shutil import get_terminal_size
+width = get_terminal_size().columns
+print(end='\r', file=self.fd)
+print(' ' * width, end='', file=self.fd)
+else:
+# Do not flush if nothing was written
+return
+self.fd.flush()
+
+def finish(self):
+self.finished = True
+self.update(self.maxval)
+# Clear the progress bar after completion
+self.erase_line()
+if self.signal_set:
+signal.signal(signal.SIGWINCH, signal.SIG_DFL)
+
 self.bar = OurProgressBar(widgets=(
 ' ',
 progressbar.Bar(),
-- 
2.18.0

From ae7c7acc52bc45a452555dd16932fc21419426f8 Mon Sep 17 00:00:00 2001
From: Paul Wise 
Date: Sat, 7 Jul 2018 17:22:19 +0800
Subject: [PATCH 3/3] Erase the progress bar line when diffoscope is
 interrupted

Otherwise cruft is left on the terminal after exit.
---
 diffoscope/main.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/diffoscope/main.py b/diffoscope/main.py
index 375cdc2..e2fb945 100644
--- a/diffoscope/main.py
+++ b/diffoscope/main.py
@@ -459,6 +459,8 @@ def main(args=None):
 post_parse(parsed_args)
 sys.exit(run_diffoscope(parsed_args))
 except KeyboardInterrupt:
+if log_handler:
+log_

Bug#901758: diffoscope: leaves progress bar on terminal after completion

[Paul Wise]

On Fri, 06 Jul 2018 18:50:44 +0100 Chris Lamb wrote:

> https://salsa.debian.org/reproducible-builds/diffoscope/commit/68c8263a430aca779b0c44fd31e729ae9bd59e0b.patch

> +# Clear the progress bar after completion (#901758)

I don't think mentioning the bug in the comment is useful here,
since the comment is self explanatory without the bug link.

> +self.fd.write("\033[K")

This isn't going to work if the terminal being used doesn't support
that escape sequence or if the user requested a progress bar but
directed output to a file. I'd suggest the approach I mentioned in the
initial bug report instead. Detect the erase-to-eol escape sequence
for the current terminal, use spaces if the current terminal doesn't
support it and don't print anything if the output is not a terminal.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#901982: diffoscope: crash comparing ext4 filesystems: AttributeError: 'FsImageContainer' object has no attribute 'g'

[Paul Wise]

Package: diffoscope
Version: 96
Severity: normal
Usertags: crash

diffoscope crashes when comparing ext4 filesystems:

$ truncate -s 1M foo.ext4.initial

$ /sbin/mkfs.ext4 foo.ext4.initial
mke2fs 1.44.2 (14-May-2018)

Filesystem too small for a journal
Discarding device blocks: done
Creating filesystem with 1024 1k blocks and 128 inodes

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

$ cp foo.ext4.initial foo.ext4.mount

$ mkdir foo.ext4.mountdir

$ sudo mount foo.ext4.mount foo.ext4.mountdir

$ sudo umount foo.ext4.mountdir

$ diffoscope foo.ext4.initial foo.ext4.mount
Traceback (most recent call 
last):#|
  100% ETA:  0:00:00 
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 460, in main
sys.exit(run_diffoscope(parsed_args))
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 432, in 
run_diffoscope
difference = compare_root_paths(path1, path2)
  File 
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 
68, in compare_root_paths
difference = compare_files(file1, file2)
  File 
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 
118, in compare_files
return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", 
line 366, in compare
difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", 
line 321, in _compare_using_details
other.as_container, no_recurse=no_recurse))
  File 
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", 
line 131, in comparisons
my_members = OrderedDict(self.get_adjusted_members_sizes())
  File 
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", 
line 127, in get_adjusted_members_sizes
size = path_apparent_size(member.path)
  File 
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/archive.py", line 
99, in path
self._name, self._temp_dir.name)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/fsimage.py", line 
74, in extract
self.g.tar_out('/', dest_path)
AttributeError: 'FsImageContainer' object has no attribute 'g'

-- System Information:
Debian Release: buster/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.16.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), 
LANGUAGE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib3.6.6~rc1-1
ii  python33.6.5-3
ii  python3-distro 1.0.1-2
ii  python3-distutils  3.6.5-4
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  2:0.4.15-1
ii  python3-pkg-resources  39.1.0-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.52-3+b1
ii  apktool  2.3.3-1
ii  binutils-multiarch   2.30-21
ii  bzip21.0.6-8.1
ii  caca-utils   0.99.beta19-2+b3
ii  colord   1.3.3-2
ii  db-util  5.3.1
ii  default-jdk [java-sdk]   2:1.10-67
ii  default-jdk-headless 2:1.10-67
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.44.2-1
ii  enjarify 1:1.0.3-4
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-6+b1
ii  ghc  8.0.2-11
ii  ghostscript  9.22~dfsg-2.1
ii  giflib-tools 5.1.4-3
ii  gnumeric 1.12.39-1
ii  imagemagick  8:6.9.9.39+dfsg-1
ii  imagemagick-6.q16 [imagemagick]  8:6.9.9.39+dfsg-1
ii  jsbeautifier 1.6.4-7
pn  libarchive-tools 
ii  llvm 1:4.0-40
pn  mono-utils   
pn  odt2txt  
pn  oggvideotools
ii  openjdk-10-jdk [java-sdk]10.0.1+10-4
ii  openssh-client   1:7.7p1-2
ii  pgpdump  0.31-0.2
ii  poppler-utils0.63.0-2
pn  procyon-decompiler   
ii  python3-argcomplete  1.

Bug#901758: diffoscope: leaves progress bar on terminal after completion

[Paul Wise]

Package: diffoscope
Version: 95
Severity: minor

diffoscope leaves a giant progress bar on the terminal after it
completes. It would be nice if the progress bar could disappear after
it has served its purpose. In case the progress bar code you use
doesn't support this, I've included code below from another project
that clears to the end of the line if the terminal supports that and
falls back to overwriting the line with spaces if it does not.

$ touch foo bar

$ diffoscope foo bar
 
|#|
  100% Time: 0:00:00 

try:
from shutil import get_terminal_size

def get_columns():
return get_terminal_size().columns
except ImportError:
# Python 3.2 compatibility:
from fcntl import ioctl
from termios import TIOCGWINSZ
from struct import unpack

def get_columns():
try:
buf = ioctl(sys.stdout.fileno(), TIOCGWINSZ, ' ' * 4)
return unpack('hh', buf)[1]
except IOError:
return 80

if sys.stdout.isatty():
from curses import tigetstr, setupterm
setupterm()
erase_line = tigetstr('el')

def erase_to_eol_cr(size=0):
if erase_line:
sys.stdout.buffer.write(erase_line)
else:
width = get_columns()
print(' ' * (width - size), end='')
print(end='\r')
sys.stdout.flush()

-- System Information:
Debian Release: buster/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.16.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_AU:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib3.6.6~rc1-1
ii  python33.6.5-3
ii  python3-distro 1.0.1-2
ii  python3-distutils  3.6.5-4
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  2:0.4.15-1
ii  python3-pkg-resources  39.1.0-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.52-3+b1
ii  apktool  2.3.3-1
ii  binutils-multiarch   2.30-21
ii  bzip21.0.6-8.1
ii  caca-utils   0.99.beta19-2+b3
ii  colord   1.3.3-2
ii  db-util  5.3.1
ii  default-jdk [java-sdk]   2:1.10-67
ii  default-jdk-headless 2:1.10-67
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.44.2-1
ii  enjarify 1:1.0.3-4
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-6+b1
ii  ghc  8.0.2-11
ii  ghostscript  9.22~dfsg-2.1
ii  giflib-tools 5.1.4-3
ii  gnumeric 1.12.39-1
ii  gnupg2.2.8-1
ii  imagemagick  8:6.9.9.39+dfsg-1
ii  imagemagick-6.q16 [imagemagick]  8:6.9.9.39+dfsg-1
ii  jsbeautifier 1.6.4-7
pn  libarchive-tools 
ii  llvm 1:4.0-40
pn  mono-utils   
pn  odt2txt  
pn  oggvideotools
ii  openjdk-10-jdk [java-sdk]10.0.1+10-4
ii  openssh-client   1:7.7p1-2
ii  pdftk2.02-4+b2
ii  pgpdump  0.31-0.2
ii  poppler-utils0.63.0-2
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1
ii  python3-binwalk  2.1.1-16
ii  python3-debian   0.1.32
pn  python3-defusedxml   
pn  python3-guestfs  
ii  python3-jsondiff 1.1.1-2
ii  python3-progressbar  2.3-4
ii  python3-pyxattr  0.6.0-2+b1
ii  python3-tlsh 3.4.4+20151206-1+b3
pn  r-base-core  
ii  rpm2cpio 4.14.1+dfsg1-3
ii  sng  1.1.0-1+b1
ii  sqlite3  3.24.0-1
ii  squashfs-tools   1:4.3-6
ii  tcpdump  4.9.2-3
ii  unzip6.0-21
ii  vim-common   2:8.0.1766-1
pn  xmlutils 
ii  xxd  2:8.0.1766-1
ii  xz-utils 5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

Bug#901757: diffoscope: ZIP files: compare comments with zipnote or zipdetails output

[Paul Wise]

Package: diffoscope
Version: 95
Severity: wishlist

It would be nice if comparing ZIP files with diffoscope could compare
the comments embedded in the CDFH and EOCD structures:

https://en.wikipedia.org/wiki/Zip_(file_format)#Central_directory_file_header

This can be done using both zipnote or zipdetails but zipinfo (used by
diffoscope) does not print anything about zip file comments.

I'm not sure which tool to use but maybe use zipnote first and if that
produces no differences then use zipdetails, because the latter also
allows for comparing the compression level of individual files. Or
maybe just use zipdetails if the zipinfo output only differs by the
file size information, since scanning the whole zip may be expensive.

I've illustrated this with the command sequence below.

$ touch foo

$ zip foo.zip foo
  adding: foo (stored 0%)

$ cp foo.zip bar.zip

$ zipnote -w bar.zip <(zipnote foo.zip | sed -e '/@ foo/a hello' -e '$a 
goodbye')

$ diffoscope foo.zip bar.zip
 
|#|
  100% Time: 0:00:00 
--- foo.zip
+++ bar.zip
├── zipinfo /dev/stdin
│ @@ -1,3 +1,3 @@
│ -Zip file size: 156 bytes, number of entries: 1
│ +Zip file size: 168 bytes, number of entries: 1
│  -rw-r-  3.0 unx0 bx stor 18-Jun-18 01:29 foo
│  1 file, 0 bytes uncompressed, 0 bytes compressed:  0.0%

$ diff -u <(zipnote foo.zip) <(zipnote bar.zip)
--- /dev/fd/63  2018-06-18 09:54:26.288808196 +0800
+++ /dev/fd/62  2018-06-18 09:54:26.288808196 +0800
@@ -1,3 +1,5 @@
 @ foo
+hello
 @ (comment above this line)
 @ (zip file comment below this line)
+goodbye

$ diff -u <(zipdetails foo.zip) <(zipdetails bar.zip)
--- /dev/fd/63  2018-06-18 09:54:42.860925054 +0800
+++ /dev/fd/62  2018-06-18 09:54:42.860925054 +0800
@@ -37,7 +37,7 @@
 0055 Uncompressed Length   
 0059 Filename Length   0003
 005B Extra Length  0018
-005D Comment Length
+005D Comment Length0005
 005F Disk Start
 0061 Int File Attributes   
  [Bit 0]   0 'Binary Data'
@@ -55,13 +55,15 @@
 007D   UID 03E8
 0081   GID Size04
 0082   GID 03E8
+0086 Comment   'hello'
 
-0086 END CENTRAL HEADER06054B50
-008A Number of this disk   
-008C Central Dir Disk no   
-008E Entries in this disk  0001
-0090 Total Entries 0001
-0092 Size of Central Dir   0049
-0096 Offset to Central Dir 003D
-009A Comment Length
+008B END CENTRAL HEADER06054B50
+008F Number of this disk   
+0091 Central Dir Disk no   
+0093 Entries in this disk  0001
+0095 Total Entries 0001
+0097 Size of Central Dir   004E
+009B Offset to Central Dir 003D
+009F Comment Length0007
+00A1 Comment   'goodbye'
 Done

-- System Information:
Debian Release: buster/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.16.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_AU:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib3.6.6~rc1-1
ii  python33.6.5-3
ii  python3-distro 1.0.1-2
ii  python3-distutils  3.6.5-4
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  2:0.4.15-1
ii  python3-pkg-resources  39.1.0-1

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.52-3+b1
ii  apktool  2.3.3-1
ii  binutils-multiarch   2.30-21
ii  bzip21.0.6-8.1
ii  caca-utils   0.99.beta19-2+b3
ii  colord   1.3.3-2
ii  db-util  5.3.1
ii  default-jdk [java-sdk]   2:1.10-67
ii  default-jdk-headless 2:1.10-67
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.44.2-1
ii  enjarify 1:1.0.3-4
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-6+b1
ii  ghc  8.0.2-11
ii  ghostscript  9.22~dfsg-2.1
ii  giflib-tools 5.1.4-3
ii  gnumeric 1.12.39-1
ii  gnupg2.2.8-1
ii  imagemagick  8:6.9.9.39+dfsg-1
ii  imagemagick-6.q16 [imagemagick]  8:6.9.9.39+dfsg-1
ii  jsbeautifier

Bug#901300: bls: warn about strip-nondeterminism output in build logs

[Paul Wise]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: bls
X-Debbugs-CC: reproducible-bui...@lists.alioth.debian.org, brl...@debian.org
Control: user reproducible-bui...@lists.alioth.debian.org
Control: usertags -1 + infrastructure
Control: clone -1 -2
Control: reassign -2 bls-standalone

It would be nice if the Debian build log scanner (bls)[1] could detect
when strip-nondeterminism finds some nondeterminism and warns about
that in the build logs. The strip-nondeterminism tool is a temporary
workaround for nondeterminism but packages should build reproducibly
even without it enabled. If we are to eventually drop this tool, we
need ways of measuring how often it is needed and encouraging people to
reduce the need for the tool. Since it is used at build time and
outputs to build logs and the bls results are communicated to
maintainers via the package tracker, bls could help achieve this.

Here is an example of the build log output from strip-nondeterminism:

   dh_strip_nondeterminism
Normalized debian/sysstat/usr/share/locale/hu/LC_MESSAGES/sysstat.mo
Normalized debian/sysstat/usr/share/locale/nn/LC_MESSAGES/sysstat.mo
Normalized debian/sysstat/usr/share/locale/zh_CN/LC_MESSAGES/sysstat.mo
Normalized debian/sysstat/usr/share/locale/nb/LC_MESSAGES/sysstat.mo
Normalized debian/sysstat/usr/share/locale/ja/LC_MESSAGES/sysstat.mo
Normalized debian/sysstat/usr/share/locale/ko/LC_MESSAGES/sysstat.mo
Normalized debian/sysstat/usr/share/locale/af/LC_MESSAGES/sysstat.mo
Normalized debian/sysstat/usr/share/locale/mt/LC_MESSAGES/sysstat.mo

So BLS should detect a line with "dh_strip_nondeterminism" followed by
lines beginning with "Normalized ". Initial whitespace can be ignored.

   1. https://qa.debian.org/bls/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#896934: diffoscope: decode monitor EDID data to text

[Paul Wise]

On Sat, 2018-05-26 at 08:37 +0100, Chris Lamb wrote:

> Is this something you think you can drive or should someone take
> over..?

If someone wants to take over, please feel free.
Otherwise I'll get to it at some point.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#896934: diffoscope: decode monitor EDID data to text

[Paul Wise]

On Thu, 2018-04-26 at 09:08 +0200, Mattia Rizzolo wrote:

>  * add a test

EDID data is almost always proprietary so I'm not sure how to do that.

>  * expand that TODO item, I can't understand what it is about

diffoscope does not compare the output of the commands when one of them
returns an exit code other than zero. As far as I can tell this is
hard-coded in the File class, so I assumed this was a design choice?

$ git grep -p -B9 -A1 'exited with'

>  * explain why the external_tools.py's dict lists both edid-decode and
>read-edid, but in the build-dep are made as alternatives

Both are optional and only run if installed, either one is useful for
comparing the files but edid-decode shows more details.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Bug#896934: diffoscope: decode monitor EDID data to text

[Paul Wise]

Package: diffoscope
Version: 93
Severity: wishlist
Tags: patch

I recently had occasion to compare two EDID files. I initially tried
diffoscope, saw only a hexdump and then manually compared the files
using a diff of pipes through edid-decode. Subsequently I looked up how
to add new formats to diffoscope and found it was fairly easy.

If the attached patch is acceptable, let me know and I'll push the
OpenPGP-signed version of the commit.

diff -u <(edid-decode /lib/firmware/edid/VGA-1) <(edid-decode 
/sys/devices/pci:00/:00:03.0/:01:00.0/drm/card0/card0-DVI-D-1/edid )

https://en.wikipedia.org/wiki/Extended_Display_Identification_Data

-- System Information:
Debian Release: buster/sid
  APT prefers testing-debug
  APT policy: (900, 'testing-debug'), (900, 'testing'), (800, 
'unstable-debug'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 
'experimental-debug'), (700, 'experimental'), (690, 'buildd-experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.15.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8), 
LANGUAGE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  python33.6.4-1
ii  python3-distro 1.0.1-2
ii  python3-distutils  3.6.5~rc1-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  2:0.4.15-1
ii  python3-pkg-resources  39.0.1-2

Versions of packages diffoscope recommends:
ii  abootimg 0.6-1+b2
ii  acl  2.2.52-3+b1
ii  apktool  2.3.2-1
ii  binutils-multiarch   2.30-15
ii  bzip21.0.6-8.1
ii  caca-utils   0.99.beta19-2+b2
ii  colord   1.3.3-2
ii  db-util  5.3.1
ii  default-jdk [java-sdk]   2:1.9-63
ii  default-jdk-headless 2:1.9-63
pn  device-tree-compiler 
pn  docx2txt 
ii  e2fsprogs1.44.1-2
ii  enjarify 1:1.0.3-4
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-6
ii  ghc  8.0.2-11
ii  ghostscript  9.22~dfsg-2.1
ii  giflib-tools 5.1.4-2
ii  gnumeric 1.12.35-1
ii  gnupg2.2.5-1
ii  imagemagick  8:6.9.9.39+dfsg-1
ii  imagemagick-6.q16 [imagemagick]  8:6.9.9.39+dfsg-1
ii  jsbeautifier 1.6.4-7
pn  libarchive-tools 
ii  llvm 1:4.0-40
pn  mono-utils   
pn  odt2txt  
pn  oggvideotools
ii  openjdk-8-jdk [java-sdk] 8u162-b12-1
ii  openjdk-9-jdk [java-sdk] 9.0.4+12-4
ii  openssh-client   1:7.7p1-2
ii  pdftk2.02-4+b2
ii  pgpdump  0.31-0.2
ii  poppler-utils0.62.0-2
pn  procyon-decompiler   
ii  python3-argcomplete  1.8.1-1
ii  python3-binwalk  2.1.1-16
ii  python3-debian   0.1.32
pn  python3-defusedxml   
pn  python3-guestfs  
ii  python3-jsondiff 1.1.1-2
ii  python3-progressbar  2.3-4
ii  python3-pyxattr  0.6.0-2+b1
ii  python3-tlsh 3.4.4+20151206-1+b3
pn  r-base-core  
ii  rpm2cpio 4.14.1+dfsg1-2
ii  sng  1.1.0-1+b1
ii  sqlite3  3.23.1-1
ii  squashfs-tools   1:4.3-6
ii  tcpdump  4.9.2-3
ii  unzip6.0-21
ii  vim-common   2:8.0.1453-1
pn  xmlutils 
ii  xxd  2:8.0.1453-1+b1
ii  xz-utils 5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

-- no debconf information

-- 
bye,
pabs

https://wiki.debian.org/PaulWise
From 526a76729430897b4d5115f8b7d85efd26b094af Mon Sep 17 00:00:00 2001
From: Paul Wise 
Date: Mon, 23 Apr 2018 13:54:35 +0800
Subject: [PATCH] Add support for comparing monitor EDID information

Uses edid-decode and parse-edid to convert to text.
---
 debian/control |  1 +
 diffoscope/comparators/__init__.py |  1 +
 diffoscope/comparators/edid.py | 60 ++
 diffoscope/external_tools.py   |  6 +++
 4 files changed, 68 insertions(+)
 create mode 100644 diffoscope/comparators/edid.py

diff --git a/debian/control b/debian/control
index 23a7265..e607c95 100644
--- a/d