Bug#972294: openorienteering-mapper: FTBFS with DEB_BUILD_OPTIONS=reproducible=+fixfilepath

[Kai Pastor, DG0YT]

Am 10.11.20 um 11:35 schrieb Simon McVittie:

On Fri, 16 Oct 2020 at 08:29:48 +0200, Kai Pastor, DG0YT wrote:

So far, all cases in openorienteering-mapper were tests which were expected
to be run in the build environment and indeed access the pristine test data
in the source directory.

The current issue comes from using Qt's QFINDTESTDATA(), which relies on a
cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory from
which the compiler is invoked" in order to "the absolute path of the source
directory" [!], https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA .
QT_TESTCASE_BUILDDIR is defined by Qt's cmake file.

One way this is often done, particularly in the GNOME ecosystem, is to
check for an environment variable that is set by the build system while
running tests. There are often two environment variables - one for the
source directory and one for the build directory - so that tests can
either ask for data files that are distributed with the source code, or
data files that were compiled along with the test itself and placed in
the build directory, if those directories are different.

If the environment variable is not set, there's a fallback that would
be reasonable to use if the test has been installed system-wide,
typically into /usr/libexec/installed-tests (which is something
that various GNOME and GNOME-adjacent packages support doing, for
"as-installed" testing like Debian's autopkgtest: see
).
In GLib the fallback is to use dirname(argv[0]), but hard-coding an
installation path would also be reasonable here.

This avoids having to hard-code the path to either the source or build
directory into any binaries, even if test executables and data are going
to be installed into /usr/libexec/installed-tests for "as-installed"
testing.

Some packages need to set environment variables for build-time testing
*anyway*, so that third-party components will find their required data
in the source or build tree: for example, you might need to set PATH,
LD_LIBRARY_PATH, XDG_DATA_DIRS and similar variables. The variables used
by this particular package's tests can be set in the same way.

The implementation that is normally used in GNOME is GLib's
g_test_build_filename(), which works something like this pseudocode:

 if file_type == G_TEST_DIST:
 dir = getenv("G_TEST_SRCDIR")
 elif file_type == G_TEST_BUILT:
 # this branch is the closest equivalent of QFINDTESTDATA
 dir = getenv("G_TEST_BUILDDIR")
 else:
 fatal error

 if dir is null:
 dir = directory containing the running executable

 return join_paths(dir, first_path, ...)

An implementation of the build-system side of this in a simple
Makefile-based build system would look something like:

 export G_TEST_BUILDDIR = $(CURDIR)
 export G_TEST_SRCDIR = $(srcdir)

 check:
 ./test-foo
 ./test-bar

Obviously the code would look a bit different for Autotools, CMake or
Meson, but all are capable of doing this (Autotools uses
AM_TESTS_ENVIRONMENT, CMake uses
set_tests_properties(... PROPERTIES ENVIRONMENT ...), Meson uses
test(..., env : ...)).

I assume qmake would also be able to do this, but I don't know how.

 smcv

I did consider these options, but I am not convinced. One perspective I 
am looking at is onboarding of new contributors (upstream). 
Reproducibility helps with that, no doubt. But with your favourite IDE 
as a development tool, or at the command line, why to make it hard to 
just run a particular test executable? Why add complexity to each and 
every package's source code or build system?


... or how to fix QFINDTESTDATA? This macro is the canonical way with 
Qt. It should be made compatible with reproducible builds. The solution 
might include environment variables, but implemented by Qt Test, not 
implemented in each and every package.


And then one more step, and standardize the variables' names (like 
SOURCE_DATE_EPOCH).

Bug#972294: openorienteering-mapper: FTBFS with DEB_BUILD_OPTIONS=reproducible=+fixfilepath

[Simon McVittie]

On Fri, 16 Oct 2020 at 08:29:48 +0200, Kai Pastor, DG0YT wrote:
> So far, all cases in openorienteering-mapper were tests which were expected
> to be run in the build environment and indeed access the pristine test data
> in the source directory.
> 
> The current issue comes from using Qt's QFINDTESTDATA(), which relies on a
> cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory from
> which the compiler is invoked" in order to "the absolute path of the source
> directory" [!], https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA .
> QT_TESTCASE_BUILDDIR is defined by Qt's cmake file.

One way this is often done, particularly in the GNOME ecosystem, is to
check for an environment variable that is set by the build system while
running tests. There are often two environment variables - one for the
source directory and one for the build directory - so that tests can
either ask for data files that are distributed with the source code, or
data files that were compiled along with the test itself and placed in
the build directory, if those directories are different.

If the environment variable is not set, there's a fallback that would
be reasonable to use if the test has been installed system-wide,
typically into /usr/libexec/installed-tests (which is something
that various GNOME and GNOME-adjacent packages support doing, for
"as-installed" testing like Debian's autopkgtest: see
).
In GLib the fallback is to use dirname(argv[0]), but hard-coding an
installation path would also be reasonable here.

This avoids having to hard-code the path to either the source or build
directory into any binaries, even if test executables and data are going
to be installed into /usr/libexec/installed-tests for "as-installed"
testing.

Some packages need to set environment variables for build-time testing
*anyway*, so that third-party components will find their required data
in the source or build tree: for example, you might need to set PATH,
LD_LIBRARY_PATH, XDG_DATA_DIRS and similar variables. The variables used
by this particular package's tests can be set in the same way.

The implementation that is normally used in GNOME is GLib's
g_test_build_filename(), which works something like this pseudocode:

if file_type == G_TEST_DIST:
dir = getenv("G_TEST_SRCDIR")
elif file_type == G_TEST_BUILT:
# this branch is the closest equivalent of QFINDTESTDATA
dir = getenv("G_TEST_BUILDDIR")
else:
fatal error

if dir is null:
dir = directory containing the running executable

return join_paths(dir, first_path, ...)

An implementation of the build-system side of this in a simple
Makefile-based build system would look something like:

export G_TEST_BUILDDIR = $(CURDIR)
export G_TEST_SRCDIR = $(srcdir)

check:
./test-foo
./test-bar

Obviously the code would look a bit different for Autotools, CMake or
Meson, but all are capable of doing this (Autotools uses
AM_TESTS_ENVIRONMENT, CMake uses
set_tests_properties(... PROPERTIES ENVIRONMENT ...), Meson uses
test(..., env : ...)).

I assume qmake would also be able to do this, but I don't know how.

smcv

Bug#972294: openorienteering-mapper: FTBFS with DEB_BUILD_OPTIONS=reproducible=+fixfilepath

[Vagrant Cascadian]

On 2020-11-10, Kai Pastor, DG0YT wrote:
> Am 10.11.20 um 01:00 schrieb Vagrant Cascadian:
>> On 2020-10-16, Kai Pastor, DG0YT wrote:
>>> Am 16.10.20 um 01:19 schrieb Vagrant Cascadian:
 When the reproducible=+fixfilepath feature is enabled (either through
 DEB_BUILD_OPTIONS, or using a dpkg that enables this by default),
 openorienteering-mapper fails to build from source:

 
 http://qa-logs.debian.net/2020/09/26.fixfilepath/openorienteering-mapper_0.9.3-1_unstable_fixfilepath.log

 While the "fixfilepath" feature is not currently enabled by
 dpkg-buildflags by default, it may become the default at some point in
 the future, and can by triggered manually by setting
 DEB_BUILD_OPTIONS=reproducible=+fixfilepath in the build environment.

 More information about this issue is available at:

 
 https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

 I have not identified the exact cause of this issue for
 openorienteering-mapper, but a common triggering issue is test suites
 expectinfg __FILE__ to resolve to an absolute path.

 The attached patch works around this issue by disabling the fixfilepath
 feature in debian/rules using DEB_BUILD_MAINT_OPTIONS=-fixfilepath.
>> ... sorry for the delay...
>>
>>> So far, all cases in openorienteering-mapper were tests which were
>>> expected to be run in the build environment and indeed access the
>>> pristine test data in the source directory.
>>> The current issue comes from using Qt's QFINDTESTDATA(), which relies on
>>> a cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory
>>> from which the compiler is invoked" in order to "the absolute path of
>>> the source directory" [!],
>>> https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA . QT_TESTCASE_BUILDDIR
>>> is defined by Qt's cmake file. I do see some inconsistency there, but
>>> this is a different story.
>> Yes, this is part of the known issues with test data mentioned; the
>> majority of known build failures triggered by fixfilepath are these
>> use-cases with QFINDTESTDATA.
>>
>>
>>> In previous cases I "solved" the failing reproducible builds by: using
>>> another macro, carrying the source directory. But I'm not sure if this
>>> is what is intented. While I do have ideas how to workaround this in
>>> other ways, I would appreciate a clear recommendation how test data in
>>> the source dir should be handled.
>> If the package in question only uses such features in test suites that
>> are not shipped in the binary packages, it's perfectly reasonable to
>> disable the fixfilepath feature; it will likely have no effect on the
>> resulting packages either way.
>>
>> If the package embeds file paths in files shipped in the binary
>> packages, then it might be worth some of the workarounds you suggested
>> with the test suites, and further debugging what exactly is embedding
>> the build paths; enabling fixfilepath only catches some of the issues of
>> embedded build paths, so it may be a moot point for any particular
>> package.
>>
>> Disabling fixfilepath in your package will allow
>> tests.reproducible-builds.org to be able to test builds of the
>> openorienteering-mapper package in unstable and experimental again,
>> where the build path is varied, and possibly help identify weather
>> further exploration is needed.
>>
>> At this point, I'd suggest disabling fixfilepath for this particular
>> package. But I submitted the patch to do that, so maybe I am biased? :)
>
> Trying to express my concerns in terms of reproducible-builds.org 
> terminology definitions:
>
> As the author of the software, I define the "expected reproducible 
> artifacts" to be the files created by "make install".
> With disabling fixfilepath, is there another test which not only 
> verifies "bit-by-bit identical copies of all specified artifacts", but 
> would also offer the diagnosis that the build path ended up in the 
> installed artifacts?

I'm not sure what exactly you are asking, but here is my attempt to
answer... :)


In this case, DEB_BUILD_OPTIONS or
DEB_BUILD_MAINT_OPTIONS=reproducible=+fixfilepath merely tells
dpkg-buildflags to set various flags (e.g. CFLAGS) to include
-ffile-prefix-map=BUILDPATH=. It does not verify reproducibility of
resulting artifacts. It can lead to artifacts which are easier to build
reproducibly in some cases.


Determining weather the artifacts are reproducible is done with
mechanisms such as comparing checksums of the artifacts, and diagnosis
of reproducibility issues when the checksums do not match is done by
comparing the resulting artifacts of multiple builds using tools such as
diffoscope... or tools such as reprotest which perform multiple builds
varying a variety of things, and then compares the results with diff or
diffoscope.

Lintian appears to be able to detect build path in some cases:

  

Bug#972294: openorienteering-mapper: FTBFS with DEB_BUILD_OPTIONS=reproducible=+fixfilepath

[Kai Pastor, DG0YT]

Am 10.11.20 um 01:00 schrieb Vagrant Cascadian:

On 2020-10-16, Kai Pastor, DG0YT wrote:

Am 16.10.20 um 01:19 schrieb Vagrant Cascadian:

When the reproducible=+fixfilepath feature is enabled (either through
DEB_BUILD_OPTIONS, or using a dpkg that enables this by default),
openorienteering-mapper fails to build from source:


http://qa-logs.debian.net/2020/09/26.fixfilepath/openorienteering-mapper_0.9.3-1_unstable_fixfilepath.log

While the "fixfilepath" feature is not currently enabled by
dpkg-buildflags by default, it may become the default at some point in
the future, and can by triggered manually by setting
DEB_BUILD_OPTIONS=reproducible=+fixfilepath in the build environment.

More information about this issue is available at:


https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

I have not identified the exact cause of this issue for
openorienteering-mapper, but a common triggering issue is test suites
expectinfg __FILE__ to resolve to an absolute path.

The attached patch works around this issue by disabling the fixfilepath
feature in debian/rules using DEB_BUILD_MAINT_OPTIONS=-fixfilepath.

... sorry for the delay...


So far, all cases in openorienteering-mapper were tests which were
expected to be run in the build environment and indeed access the
pristine test data in the source directory.
The current issue comes from using Qt's QFINDTESTDATA(), which relies on
a cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory
from which the compiler is invoked" in order to "the absolute path of
the source directory" [!],
https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA . QT_TESTCASE_BUILDDIR
is defined by Qt's cmake file. I do see some inconsistency there, but
this is a different story.

Yes, this is part of the known issues with test data mentioned; the
majority of known build failures triggered by fixfilepath are these
use-cases with QFINDTESTDATA.



In previous cases I "solved" the failing reproducible builds by: using
another macro, carrying the source directory. But I'm not sure if this
is what is intented. While I do have ideas how to workaround this in
other ways, I would appreciate a clear recommendation how test data in
the source dir should be handled.

If the package in question only uses such features in test suites that
are not shipped in the binary packages, it's perfectly reasonable to
disable the fixfilepath feature; it will likely have no effect on the
resulting packages either way.

If the package embeds file paths in files shipped in the binary
packages, then it might be worth some of the workarounds you suggested
with the test suites, and further debugging what exactly is embedding
the build paths; enabling fixfilepath only catches some of the issues of
embedded build paths, so it may be a moot point for any particular
package.

Disabling fixfilepath in your package will allow
tests.reproducible-builds.org to be able to test builds of the
openorienteering-mapper package in unstable and experimental again,
where the build path is varied, and possibly help identify weather
further exploration is needed.

At this point, I'd suggest disabling fixfilepath for this particular
package. But I submitted the patch to do that, so maybe I am biased? :)


Trying to express my concerns in terms of reproducible-builds.org 
terminology definitions:


As the author of the software, I define the "expected reproducible 
artifacts" to be the files created by "make install".
With disabling fixfilepath, is there another test which not only 
verifies "bit-by-bit identical copies of all specified artifacts", but 
would also offer the diagnosis that the build path ended up in the 
installed artifacts?


Last not least, "make install" is the canonical way of creating the set 
of artifacts meant for distribution.




Hope that is a clear enough recommendation?


Oh, and, sorry for not getting back to you till now! Not sure how I lost
track of this...


live well,
   vagrant


Thanks!
Kai

Bug#972294: openorienteering-mapper: FTBFS with DEB_BUILD_OPTIONS=reproducible=+fixfilepath

[Vagrant Cascadian]

On 2020-10-16, Kai Pastor, DG0YT wrote:
> Am 16.10.20 um 01:19 schrieb Vagrant Cascadian:
>> When the reproducible=+fixfilepath feature is enabled (either through
>> DEB_BUILD_OPTIONS, or using a dpkg that enables this by default),
>> openorienteering-mapper fails to build from source:
>>
>>
>> http://qa-logs.debian.net/2020/09/26.fixfilepath/openorienteering-mapper_0.9.3-1_unstable_fixfilepath.log
>>
>> While the "fixfilepath" feature is not currently enabled by
>> dpkg-buildflags by default, it may become the default at some point in
>> the future, and can by triggered manually by setting
>> DEB_BUILD_OPTIONS=reproducible=+fixfilepath in the build environment.
>>
>> More information about this issue is available at:
>>
>>
>> https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
>>
>> I have not identified the exact cause of this issue for
>> openorienteering-mapper, but a common triggering issue is test suites
>> expectinfg __FILE__ to resolve to an absolute path.
>>
>> The attached patch works around this issue by disabling the fixfilepath
>> feature in debian/rules using DEB_BUILD_MAINT_OPTIONS=-fixfilepath.

... sorry for the delay...

> So far, all cases in openorienteering-mapper were tests which were 
> expected to be run in the build environment and indeed access the 
> pristine test data in the source directory.

> The current issue comes from using Qt's QFINDTESTDATA(), which relies on 
> a cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory 
> from which the compiler is invoked" in order to "the absolute path of 
> the source directory" [!], 
> https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA . QT_TESTCASE_BUILDDIR 
> is defined by Qt's cmake file. I do see some inconsistency there, but 
> this is a different story.

Yes, this is part of the known issues with test data mentioned; the
majority of known build failures triggered by fixfilepath are these
use-cases with QFINDTESTDATA.


> In previous cases I "solved" the failing reproducible builds by: using 
> another macro, carrying the source directory. But I'm not sure if this 
> is what is intented. While I do have ideas how to workaround this in 
> other ways, I would appreciate a clear recommendation how test data in 
> the source dir should be handled.

If the package in question only uses such features in test suites that
are not shipped in the binary packages, it's perfectly reasonable to
disable the fixfilepath feature; it will likely have no effect on the
resulting packages either way.

If the package embeds file paths in files shipped in the binary
packages, then it might be worth some of the workarounds you suggested
with the test suites, and further debugging what exactly is embedding
the build paths; enabling fixfilepath only catches some of the issues of
embedded build paths, so it may be a moot point for any particular
package.

Disabling fixfilepath in your package will allow
tests.reproducible-builds.org to be able to test builds of the
openorienteering-mapper package in unstable and experimental again,
where the build path is varied, and possibly help identify weather
further exploration is needed.

At this point, I'd suggest disabling fixfilepath for this particular
package. But I submitted the patch to do that, so maybe I am biased? :)

Hope that is a clear enough recommendation?


Oh, and, sorry for not getting back to you till now! Not sure how I lost
track of this...


live well,
  vagrant


signature.asc
Description: PGP signature

Bug#972294: openorienteering-mapper: FTBFS with DEB_BUILD_OPTIONS=reproducible=+fixfilepath

[Kai Pastor, DG0YT]

Am 16.10.20 um 01:19 schrieb Vagrant Cascadian:

Package: openorienteering-mapper
Severity: normal
Tags: patch
User: reproducible-bui...@lists.alioth.debian.org
Usertags: fixfilepath ftbfs
X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org

When the reproducible=+fixfilepath feature is enabled (either through
DEB_BUILD_OPTIONS, or using a dpkg that enables this by default),
openorienteering-mapper fails to build from source:

   
http://qa-logs.debian.net/2020/09/26.fixfilepath/openorienteering-mapper_0.9.3-1_unstable_fixfilepath.log

While the "fixfilepath" feature is not currently enabled by
dpkg-buildflags by default, it may become the default at some point in
the future, and can by triggered manually by setting
DEB_BUILD_OPTIONS=reproducible=+fixfilepath in the build environment.

More information about this issue is available at:

   
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

I have not identified the exact cause of this issue for
openorienteering-mapper, but a common triggering issue is test suites
expectinfg __FILE__ to resolve to an absolute path.

The attached patch works around this issue by disabling the fixfilepath
feature in debian/rules using DEB_BUILD_MAINT_OPTIONS=-fixfilepath.

Thanks for maintaining openorienteering-mapper!


live well,
   vagrant

So far, all cases in openorienteering-mapper were tests which were 
expected to be run in the build environment and indeed access the 
pristine test data in the source directory.


The current issue comes from using Qt's QFINDTESTDATA(), which relies on 
a cpp macro (QT_TESTCASE_BUILDDIR) pointing "to the working directory 
from which the compiler is invoked" in order to "the absolute path of 
the source directory" [!], 
https://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA . QT_TESTCASE_BUILDDIR 
is defined by Qt's cmake file. I do see some inconsistency there, but 
this is a different story.


In previous cases I "solved" the failing reproducible builds by: using 
another macro, carrying the source directory. But I'm not sure if this 
is what is intented. While I do have ideas how to workaround this in 
other ways, I would appreciate a clear recommendation how test data in 
the source dir should be handled.


Thanks,
Kai

Bug#972294: openorienteering-mapper: FTBFS with DEB_BUILD_OPTIONS=reproducible=+fixfilepath

[Vagrant Cascadian]

Package: openorienteering-mapper
Severity: normal
Tags: patch
User: reproducible-bui...@lists.alioth.debian.org
Usertags: fixfilepath ftbfs
X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org

When the reproducible=+fixfilepath feature is enabled (either through
DEB_BUILD_OPTIONS, or using a dpkg that enables this by default),
openorienteering-mapper fails to build from source:

  
http://qa-logs.debian.net/2020/09/26.fixfilepath/openorienteering-mapper_0.9.3-1_unstable_fixfilepath.log

While the "fixfilepath" feature is not currently enabled by
dpkg-buildflags by default, it may become the default at some point in
the future, and can by triggered manually by setting
DEB_BUILD_OPTIONS=reproducible=+fixfilepath in the build environment.

More information about this issue is available at:

  
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

I have not identified the exact cause of this issue for
openorienteering-mapper, but a common triggering issue is test suites
expectinfg __FILE__ to resolve to an absolute path.

The attached patch works around this issue by disabling the fixfilepath
feature in debian/rules using DEB_BUILD_MAINT_OPTIONS=-fixfilepath.

Thanks for maintaining openorienteering-mapper!


live well,
  vagrant

From f25c2a9a1ebb440eaa6a470f193ed9acca5a90a2 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian 
Date: Thu, 15 Oct 2020 23:06:08 +
Subject: [PATCH] debian/rules: Disable fixfilepath feature, as it triggers
 build failures when enabled.

https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
---
 debian/rules | 1 +
 1 file changed, 1 insertion(+)

diff --git a/debian/rules b/debian/rules
index f1e73000..9c412c76 100755
--- a/debian/rules
+++ b/debian/rules
@@ -9,6 +9,7 @@ include /usr/share/dpkg/default.mk
 
 # see FEATURE AREAS in dpkg-buildflags(1)
 #export DEB_BUILD_MAINT_OPTIONS = hardening=+all
+export DEB_BUILD_MAINT_OPTIONS = reproducible=-fixfilepath
 
 # see ENVIRONMENT in dpkg-buildflags(1)
 # package maintainers to append CFLAGS
-- 
2.20.1



signature.asc
Description: PGP signature