Re: diffoscope timing out.

Wed, 15 Nov 2023 06:15:33 -0800 

On 2023-11-15 4:29 am, Holger Levsen wrote:

On Mon, Nov 06, 2023 at 10:42:05AM +0000, Chris Lamb wrote:

Christos Zoulas wrote:

> For weeks now there is no useful output from diffoscope. Is there anything I
> can do to help debug the issue?
>
> tests.reproducible-builds.org        [eeB]
> Wed 1 Nov 23:35:09 UTC 2023 - diffoscope 251 produced no output for
> x86_64-amd64/amd64/binary/sets/comp.tar.xz and was killed after running into
> timeout after 30m...


these issues seem to have been resolved,
https://jenkins.debian.net/job/reproducible_netbsd/1029/console
contains no such timeouts.


We'd actually love some help debugging this issue. As you can see from
the list archives, we have indeed been seeing this for a little while:
https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20230828/thread.html 
… as well as some threads in later weeks.


I'm not sure those issues are related, maybe or maybe not.


However, are unsure whether this an issue with:

a) The system running tests.reproducible-builds.org (which, I believe,
was upgraded around the time this issue appeared?)

b) The code in diffoscope itself (which is somewhat unlikely, as there
have not been meaningful changes that might cause crashes or
timeouts.)


https://jenkins.debian.net/job/reproducible_netbsd/1029/console
contains this, which IMO should not happen and is an issue in
diffoscope:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 767, in main 
    sys.exit(run_diffoscope(parsed_args))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 718,
in run_diffoscope
    difference = compare_root_paths(path1, path2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 69, in compare_root_paths
    difference = compare_files(file1, file2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 149, in compare_files
    return file1.compare(file2, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 532, in compare
    difference = self._compare_using_details(other, source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 467, in _compare_using_details
    details.extend(
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py",
line 197, in compare_pair
    difference = compare_files(
                 ^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 149, in compare_files
    return file1.compare(file2, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 616, in compare
    return self.compare_bytes(other, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 412, in compare_bytes
    return compare_binary_files(self, other, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 176, in compare_binary_files
    return Difference.from_operation(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
267, in from_operation
    return Difference.from_operation_exc(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
288, in from_operation_exc
    feeder1, operation1, excluded1 = operation_and_feeder(path1)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
285, in operation_and_feeder
    operation.start()
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/command.py",
line 45, in start
    self._process = subprocess.run(
                    ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 2128, in _communicate
    data = os.read(key.fd, 32768)
           ^^^^^^^^^^^^^^^^^^^^^^
MemoryError
Fri 10 Nov 22:41:22 UTC 2023 - diffoscope 251 had trouble comparing
the two builds. Please investigate
x86_64-amd64/images/NetBSD-10.99.10-amd64-bios-install.img.gz
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 767, in main 
    sys.exit(run_diffoscope(parsed_args))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 718,
in run_diffoscope
    difference = compare_root_paths(path1, path2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 69, in compare_root_paths
    difference = compare_files(file1, file2)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 149, in compare_files
    return file1.compare(file2, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 532, in compare
    difference = self._compare_using_details(other, source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 467, in _compare_using_details
    details.extend(
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py",
line 197, in compare_pair
    difference = compare_files(
                 ^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 149, in compare_files
    return file1.compare(file2, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 616, in compare
    return self.compare_bytes(other, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
line 412, in compare_bytes
    return compare_binary_files(self, other, source)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
line 176, in compare_binary_files
    return Difference.from_operation(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
267, in from_operation
    return Difference.from_operation_exc(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
288, in from_operation_exc
    feeder1, operation1, excluded1 = operation_and_feeder(path1)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
285, in operation_and_feeder
    operation.start()
  File
"/usr/lib/python3/dist-packages/diffoscope/comparators/utils/command.py",
line 45, in start
    self._process = subprocess.run(
                    ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 2128, in _communicate
    data = os.read(key.fd, 32768)
           ^^^^^^^^^^^^^^^^^^^^^^
MemoryError
Fri 10 Nov 22:43:45 UTC 2023 - diffoscope 251 had trouble comparing
the two builds. Please investigate
x86_64-amd64/images/NetBSD-10.99.10-amd64-install.img.gz
Thanks for looking into this Holger. It seems to me that some of the issues were resolved because we fixed the non-reproducible parts of the build for 
those files. From the trace above, it seems that the subprocess that
diffoscope was running became unresponsive which is probably the original 
reason we got timeouts.

Best,

christos

_______________________________________________
Reproducible-builds mailing list
Reproducible-builds@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

Reply via email to