Re: [PATCH v4 4/5] test: support testing notmuch as installed
Tomi Ollila writes:
>> fi
>>
>> +if [ -n "${LD_PRELOAD-}" ]; then
>
> Is it problem to chain LD_PRELOADs here ? I see later in this message
> there is LD_PRELOAD's concatenated with :'s (in case non-empty) ?
>
It is not a problem in general, but ASAN insists on being first in
LD_PRELOAD, so it's easier just to skip the tests for now. I updated
the printed message to
printf "Skipping due to ASAN LD_PRELOAD restrictions\n"
Since I also could not remember why that was there :P.
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v4 4/5] test: support testing notmuch as installed
Tomi Ollila writes:
>> -LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} notmuch-shared
>> "$@"
>> +LD_PRELOAD=${LD_PRELOAD:+:$LD_PRELOAD}:./${shim_file} $notmuch_cmd "$@"
>
> hmm, should the "chaining" be done on other order (i.e. the shim_file
> before anything if there... (that could make the case where there is
> value in LD_PRELOAD to work (commented above))
That seems like an orthogonal change, and as it turns out I don't think
it would directly help with ASAN.
d
___
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-le...@notmuchmail.org
Re: [PATCH v4 4/5] test: support testing notmuch as installed
On Sun, Jul 09 2023, David Bremner wrote:
Note: I almost commented cases where it was not ensured that variable
is defined, to notice later that those were actually defined for sure
-- so in some previous mails in this series I may have missed
same cases.
The comments for this message may be more accurate ;D
Tomi
> We put some effort into testing the built copy rather than some
> installed copy. On the other hand for people like packagers, testing
> the installed copy is also of interest.
>
> When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
> require a built notmuch tree or running configure.
>
> Some of the tests marked as broken when running against installed
> notmuch are probably fixable.
> ---
> test/README| 17 +
> test/T000-basic.sh | 1 +
> test/T010-help-test.sh | 3 +++
> test/T160-json.sh | 2 ++
> test/T240-dump-restore.sh | 2 ++
> test/T310-emacs.sh | 2 ++
> test/T350-crypto.sh| 1 +
> test/T355-smime.sh | 2 ++
> test/T360-symbol-hiding.sh | 4
> test/T390-python.sh| 4
> test/T391-python-cffi.sh | 4
> test/T395-ruby.sh | 8 ++--
> test/T410-argument-parsing.sh | 4
> test/T480-hex-escaping.sh | 4
> test/T490-parse-time-string.sh | 4
> test/T550-db-features.sh | 4
> test/T566-lib-message.sh | 4
> test/T592-thread-breakage.sh | 4
> test/T710-message-id.sh| 4
> test/T800-asan.sh | 5 +
> test/T850-git.sh | 6 ++
> test/export-dirs.sh| 2 +-
> test/notmuch-test | 12 ++--
> test/test-lib-common.sh| 13 ++---
> test/test-lib-emacs.sh | 12 ++--
> test/test-lib.sh | 27 +--
> 26 files changed, 139 insertions(+), 16 deletions(-)
>
> diff --git a/test/README b/test/README
> index 10f127cb..a81808b1 100644
> --- a/test/README
> +++ b/test/README
> @@ -137,6 +137,23 @@ detection of missing prerequisites. In the future we may
> treat tests
> unable to run because of missing prerequisites, but not explicitly
> skipped by the user, as failures.
>
> +Testing installed notmuch
> +-
> +
> +Systems integrators (e.g. Linux distros) may wish to test an installed
> +version of notmuch. This can be done be running
> +
> + $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
> +
> +In this scenario the test suite does not assume a built tree, and in
> +particular cannot rely on the output of 'configure'. You may want to
> +set certain feature environment variables ('NOTMUCH_HAVE_*') directly
> +if you know those apply to your installed notmuch). Consider also
> +setting TERM=dumb if the value of TERM cannot be used (e.g. in a
> +chroot with missing terminfo). Note that having a built tree may cause
> +surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
> +cleaning first.
> +
> Writing Tests
> -
> The test script is written as a shell script. It is to be named as
> diff --git a/test/T000-basic.sh b/test/T000-basic.sh
> index a2f4d93f..642f918d 100755
> --- a/test/T000-basic.sh
> +++ b/test/T000-basic.sh
> @@ -66,6 +66,7 @@ test_begin_subtest 'NOTMUCH_CONFIG is set and points to an
> existing file'
> test_expect_success 'test -f "${NOTMUCH_CONFIG}"'
>
> test_begin_subtest 'PATH is set to build directory'
> +test_subtest_broken_for_installed
> test_expect_equal \
> "$(dirname ${TEST_DIRECTORY})" \
> "$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
> diff --git a/test/T010-help-test.sh b/test/T010-help-test.sh
> index 1fbc46a2..827edc14 100755
> --- a/test/T010-help-test.sh
> +++ b/test/T010-help-test.sh
> @@ -19,6 +19,9 @@ if [ "${NOTMUCH_HAVE_MAN-0}" = "1" ]; then
> test_begin_subtest 'notmuch help tag'
> test_expect_success 'notmuch help tag'
> else
> +if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
> + test_done
> +fi
> test_begin_subtest 'notmuch --help tag (man pages not available)'
> test_expect_success 'test_must_fail notmuch --help tag >/dev/null'
>
> diff --git a/test/T160-json.sh b/test/T160-json.sh
> index f61c3f2d..318c9788 100755
> --- a/test/T160-json.sh
> +++ b/test/T160-json.sh
> @@ -65,6 +65,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
> \"tags\": [\"inbox\",
> \"unread\"]}]"
>
> +if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
> test_begin_subtest "Search message: json, 64-bit timestamp"
> if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
> test_subtest_known_broken
> @@ -81,6 +82,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
> \"query\": [\"id:$gen_msg_id\", null],
> \"tags\": [\"inbox\",
> \"unread\"]}]"
> +fi # NOTMUCH_TEST_INSTALLED undefined / empty
>
> test_begin_subtest "Format version: too low"
> test_expe
[PATCH v4 4/5] test: support testing notmuch as installed
We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.
When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.
Some of the tests marked as broken when running against installed
notmuch are probably fixable.
---
test/README| 17 +
test/T000-basic.sh | 1 +
test/T010-help-test.sh | 3 +++
test/T160-json.sh | 2 ++
test/T240-dump-restore.sh | 2 ++
test/T310-emacs.sh | 2 ++
test/T350-crypto.sh| 1 +
test/T355-smime.sh | 2 ++
test/T360-symbol-hiding.sh | 4
test/T390-python.sh| 4
test/T391-python-cffi.sh | 4
test/T395-ruby.sh | 8 ++--
test/T410-argument-parsing.sh | 4
test/T480-hex-escaping.sh | 4
test/T490-parse-time-string.sh | 4
test/T550-db-features.sh | 4
test/T566-lib-message.sh | 4
test/T592-thread-breakage.sh | 4
test/T710-message-id.sh| 4
test/T800-asan.sh | 5 +
test/T850-git.sh | 6 ++
test/export-dirs.sh| 2 +-
test/notmuch-test | 12 ++--
test/test-lib-common.sh| 13 ++---
test/test-lib-emacs.sh | 12 ++--
test/test-lib.sh | 27 +--
26 files changed, 139 insertions(+), 16 deletions(-)
diff --git a/test/README b/test/README
index 10f127cb..a81808b1 100644
--- a/test/README
+++ b/test/README
@@ -137,6 +137,23 @@ detection of missing prerequisites. In the future we may
treat tests
unable to run because of missing prerequisites, but not explicitly
skipped by the user, as failures.
+Testing installed notmuch
+-
+
+Systems integrators (e.g. Linux distros) may wish to test an installed
+version of notmuch. This can be done be running
+
+ $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
+
+In this scenario the test suite does not assume a built tree, and in
+particular cannot rely on the output of 'configure'. You may want to
+set certain feature environment variables ('NOTMUCH_HAVE_*') directly
+if you know those apply to your installed notmuch). Consider also
+setting TERM=dumb if the value of TERM cannot be used (e.g. in a
+chroot with missing terminfo). Note that having a built tree may cause
+surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
+cleaning first.
+
Writing Tests
-
The test script is written as a shell script. It is to be named as
diff --git a/test/T000-basic.sh b/test/T000-basic.sh
index a2f4d93f..642f918d 100755
--- a/test/T000-basic.sh
+++ b/test/T000-basic.sh
@@ -66,6 +66,7 @@ test_begin_subtest 'NOTMUCH_CONFIG is set and points to an
existing file'
test_expect_success 'test -f "${NOTMUCH_CONFIG}"'
test_begin_subtest 'PATH is set to build directory'
+test_subtest_broken_for_installed
test_expect_equal \
"$(dirname ${TEST_DIRECTORY})" \
"$(echo $PATH|cut -f1 -d: | sed -e 's,/test/valgrind/bin$,,')"
diff --git a/test/T010-help-test.sh b/test/T010-help-test.sh
index 1fbc46a2..827edc14 100755
--- a/test/T010-help-test.sh
+++ b/test/T010-help-test.sh
@@ -19,6 +19,9 @@ if [ "${NOTMUCH_HAVE_MAN-0}" = "1" ]; then
test_begin_subtest 'notmuch help tag'
test_expect_success 'notmuch help tag'
else
+if [ -n "${NOTMUCH_TEST_INSTALLED-}" ]; then
+ test_done
+fi
test_begin_subtest 'notmuch --help tag (man pages not available)'
test_expect_success 'test_must_fail notmuch --help tag >/dev/null'
diff --git a/test/T160-json.sh b/test/T160-json.sh
index f61c3f2d..318c9788 100755
--- a/test/T160-json.sh
+++ b/test/T160-json.sh
@@ -65,6 +65,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"tags\": [\"inbox\",
\"unread\"]}]"
+if [ -z "${NOTMUCH_TEST_INSTALLED-}" ]; then
test_begin_subtest "Search message: json, 64-bit timestamp"
if [ "${NOTMUCH_HAVE_64BIT_TIME_T-0}" != "1" ]; then
test_subtest_known_broken
@@ -81,6 +82,7 @@ test_expect_equal_json "$output" "[{\"thread\": \"XXX\",
\"query\": [\"id:$gen_msg_id\", null],
\"tags\": [\"inbox\",
\"unread\"]}]"
+fi # NOTMUCH_TEST_INSTALLED undefined / empty
test_begin_subtest "Format version: too low"
test_expect_code 20 "notmuch search --format-version=0 \\*"
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index 80939030..c3f18839 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -139,6 +139,7 @@ notmuch dump --output=dump-outfile-dash-inbox.actual --
from:cworth
test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
test_begin_subtest "Check for a safe set of message-ids"
+test_subtest_broken_for_installed
notmuch search --output=messages from:cworth | sed s/^id:// > EXPECT
