Hello Vagrant,
This you for this insight. I will take care of the -DLINUX issue with
condor. I can work directly with the developers to see what all the
ramifications are.
...Tim
On 5/24/26 19:03, Vagrant Cascadian wrote:
On 2026-05-24, Vagrant Cascadian wrote:
On 2026-05-23, Vagrant Cascadian wrote:
On 2026-05-22, Tim Theisen wrote:
Now that reproducible builds are mandatory, I am working on getting
condor to meet those requirements.
...
It claims that there was a regression in reproducibility on i386 it that
is blocking promotion. None of the recent changes should caused this.
It is entirely possible that there is some non-deterministic
issue... such as an ordering issue with a reasonable chance to come out
reproducible, but some N out of 100 times comes out different.
A hunch would be to check the version of the kernel that was used in
both builds, as there have been a lot of kernel security updates
lately... and those versions can sometimes get embedded in builds. That
is not something that is intentionally varied, but due to security
updates, sometimes that can cause issues.
Ok, my hunch has a little more background to go on that this is at least
part of the issue...
During a local build I see various build arguments passing the kernel
version:
/usr/bin/c++ -DCONDOR_VERSION=\"25.10.1\" -DHAVE_CONFIG_H
-DLINUX=\"LINUX_6.12.90+DEB13-AMD64\" ...
This is probably a permutation of this issue:
https://tests.reproducible-builds.org/debian/issues/unstable/captures_kernel_version_via_CMAKE_SYSTEM_issue.html
Probably caused by the use of CMAKE_SYSTEM_VERSION:
CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_NAME}" OS_NAME )
CMakeLists.txt:string( TOUPPER "${CMAKE_SYSTEM_VERSION}" OS_VER )
CMakeLists.txt-string( TOUPPER "${CMAKE_SYSTEM_PROCESSOR}" SYS_ARCH )
Not sure if just removing that line entirely, or providing a "REDACTED"
string would be an option here... we have the buildd.debian.org logs to
get information like that if needed, rather than embedding in the
binaries.
It seems to build by replacing OS_VER with "REDACTED" ... although
passes:
-DLINUX=\"LINUX_\"
And it dramatically reduced the diffoscope output (from hundreds of
megabytes to ~10MB)! Checking another build to see if I can figure out
what is left...
buildd.debian.org and reproduce.debian.net might end up with a different
version depending on the timing and frequency of kernel updates being
available and applied, which might lead to inconsistant reproducibility
results.
Yes, this is exactly what happened on i386:
https://buildd.debian.org/status/fetch.php?pkg=condor&arch=i386&ver=25.10.1%2Bdfsg-1&stamp=1778880751&raw=0
-DLINUX=\"LINUX_6.12.86+DEB13-AMD64\"
And on reproduce.debian.net:
https://reproduce.debian.net/i386/api/v1/builds/208996/log
-DLINUX=\"LINUX_6.12.88+DEB13-AMD64\"
For the corresponding amd64 logs, they were running the same
kernel. Probably the same for other architectures too...
I think tests.reproducible-builds.org may not actually be varying the
kernel version, though at least historically the intention is to vary
the kernel version (e.g. use a backports kernel and a stable kernel
across two builds).
We should really ensure tests.r-b.o is systematically testing different
kernel versions... since buildd.debian.org/reproduce.debian.net may run
arbitrary kernel versions that will be out of sync on occasion, we want
some way to catch that.
I do not think it is reasonable for reproduce.debian.net to run the
original running kernel version (E.g. running a security vulnerable
kernel) in the build environment just to get reproducible builds...
live well,
vagrant
--
Tim Theisen (he, him, his)
Release Manager
Center for High Throughput Computing
University of Wisconsin - Madison
3695 Morgridge Hall
1205 University Ave
Madison, WI 53706
_______________________________________________
Reproducible-builds mailing list
[email protected]
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds
