Hello, A few hundred packages that use reprotest in Salsa-CI appear to be misconfigured; the remainder of this message explains the problem, and asks for help figuring out what to do.
Context ------- The reprotest[1] utility tests reproducibility of .deb package builds by performing two comparative builds with selective differences in the environment. As documented[2], the extent of build-env difference can be customized using the 'variations' command-line argument, that has a default value of 'all', or similarly the 'vary' argument. These arguments can be used together, and they support plus-or-minus symbols as value prefixes (+/-) to indicate whether a variance factor is being added or removed. The reprotest commandline is parsed in sequence from left-to-right, with each 'vary' argument applied like a patch -- amending existing settings -- while in contrast each 'variations' argument performs a complete reset of the variance context. To examine/confirm reprotest's behaviour locally I can recommend its '--dry-run' argument, instructing it to print what it would do without performing any build actions. Problem: misconfiguration case ------------------------------ Although the single argument '--variations=-timezone' could reasonably be expected to disable a single form of variance (timezone) during a test, in fact it resets the variance context to empty (it does not contain 'all', begins with an empty context, and then attempts performs a no-op removal of timezone from that). This could allow packages to succeed when they would otherwise fail if the intended level of build variation was enabled. This misconfiguration has occurred in practice, and based on some code searches (example[3]) I believe that around 400 Debian packages are affected by this. Resolution ---------- My working assumption is that packages that have a single negative-variations entry (like the -timezone example above) intended to disable solely the named factor during reprotest testing. To resolve this it seems that we could: * Update the salsa-ci.yml files in each affected case to replace '--variations=-' with '--vary=-'. * Update reprotest to handle a single-disabled-varations-value as a special case - treating it as vary and/or emitting a warning. * Treat removal of a variance factor from an already-empty-context as an error. * Radically, remove the ability for packages to customize their reprotest arguments at all. To readers of these lists: does this analysis and set of assumptions make sense, and if so: do you prefer/recommend any of the suggested approaches, or have alternative suggestions of your own? Thank you, James [1] - https://salsa.debian.org/reproducible-builds/reprotest [2] - https://salsa.debian.org/reproducible-builds/reprotest/-/blob/6cb0328ea422e12d115737714627850745f93a71/README.rst?plain=1#L299-311 [3] - https://codesearch.debian.net/search?q=path%3Asalsa-ci.yml+SALSA_CI_REPROTEST_ARGS%3A+%27--variations%3D-build-path%27&literal=1