Re: Bug#842291: notmuch processes frequently stuck in select()
Daniel Kahn Gillmor writes: > > 0) turn off CRL updates entirely during s/mime signature verification > > 1) do s/mime signature verification without CRL updates, but schedule > CRL checks to happen in the background for dirmngr, so that future > verifications will reflect the cert validity > > 2) have dirmngr avoid checking CRLs that it knows it has already > updated recently > > 3) tell dirmngr to use much shorter CRL fetch timeouts > > > Any thoughts on the best way to pursue this? > > --dkg Maybe the issue is in gmime's usage of gpgme. If I understand correctly (which is far from a sure thing), pkcs7_verify calls gpgme_op_verify which is synchronous, and (apparently) does not support timeouts. An alternate strategy would be to call gpgme_op_verify_start, and then call gpgme_wait, which has a nonblocking mode. I don't really understand the S/MIME model, but naively it seems OK for signature verification to fail if the CRL check doesn't finish quickly. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] tests: account for varying-size OpenPGP signatures
On Wed 2016-11-23 11:59:38 -0500, David Bremner wrote: > there are a bunch of sed oneliners (or perl equivalent) collected in > test-lib.sh as functions test_*_sanitize. I wonder if that would be > worthwhile here, to have one place to update regexps etc... Something > in the style of notmuch_show_sanitize wrapping a call to > notmuch_json_show_sanitize Thanks for the suggestion, please see v2 of this patch. --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v2] tests: account for varying-size cryptographic signatures
GnuPG 2.1.16 is now injecting the full issuer fingerprint in its signatures, which makes them about 32 octets larger when ascii-armored. This change in size means that the size of the MIME parts will vary depending on the version of gpg that the user has installed. at any rate, the signature part should be non-zero (this is true for basically any MIME part), so we just test for that instead of an exact size. --- test/T350-crypto.sh | 22 ++ test/T355-smime.sh | 2 +- test/test-lib.sh| 3 ++- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh index df2dc74..a1e5e20 100755 --- a/test/T350-crypto.sh +++ b/test/T350-crypto.sh @@ -59,7 +59,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ @@ -94,7 +94,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ @@ -127,7 +127,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ @@ -197,7 +197,7 @@ expected='[[[{"id": "X", "content-type": "multipart/encrypted", "content": [{"id": 2, "content-type": "application/pgp-encrypted", - "content-length": 11}, + "content-length": "NONZERO"}, {"id": 3, "content-type": "multipart/mixed", "content": [{"id": 4, @@ -205,7 +205,7 @@ expected='[[[{"id": "X", "content": "This is a test encrypted message.\n"}, {"id": 5, "content-type": "application/octet-stream", - "content-length": 28, + "content-length": "NONZERO", "content-transfer-encoding": "base64", "filename": "TESTATTACHMENT"}]}]}]}, [' @@ -234,11 +234,9 @@ test_expect_equal_file OUTPUT TESTATTACHMENT test_begin_subtest "decryption failure with missing key" mv "${GNUPGHOME}"{,.bak} -# The length of the encrypted attachment varies so must be normalized. output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \ | notmuch_json_show_sanitize \ -| sed -e 's|"created": [1234567890]*|"created": 946728000|' \ -| sed -e 's|"content-length": 6[1234567890]*|"content-length": 652|') +| sed -e 's|"created": [1234567890]*|"created": 946728000|') expected='[[[{"id": "X", "match": true, "excluded": false, @@ -255,10 +253,10 @@ expected='[[[{"id": "X", "content-type": "multipart/encrypted", "content": [{"id": 2, "content-type": "application/pgp-encrypted", - "content-length": 11}, + "content-length": "NONZERO"}, {"id": 3, "content-type": "application/octet-stream", - "content-length": 652}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ @@ -295,7 +293,7 @@ expected='[[[{"id": "X", "content-type": "multipart/encrypted", "content": [{"id": 2, "content-type": "application/pgp-encrypted", - "content-length": 11}, + "content-length": "NONZERO"}, {"id": 3, "content-type": "text/plain", "content": "This is another test encrypted message.\n"}]}]}, @@ -370,7 +368,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ diff --git a/test/T355-smime.sh b/test/T355-smime.sh index d942412..a8be45e 100755 --- a/test/T355-smime.sh +++ b/test/T355-smime.sh @@ -69,7 +69,7 @@ expected='[[[{"id": "X", "content-type": "text/plain", "content": "This is a test signed message.\n"}, {"id": 3, - "content-length": 1922, + "content-length": "NONZERO", "content-transfer-encoding": "base64", "content-type": "application/x-pkcs7-signature", "filename": "smime.p7s"}]}]}, diff --git a/test/test-lib.sh b/test/test-lib.sh index a12c6d0..f55d2c6 100644 --- a/test/test-lib.sh +++ b/test/test-lib.sh @@ -736,7 +736,8 @@ notmuch_json_show_sanitize () -e 's|"Date": "Fri, 05 Jan 2001 [^"]*"|"Date": "GENERATED_DATE"|g' \ -e 's|"filename": "signature.asc",||g' \ -e 's|"filename": "/[^"]*",|"filename": "Y",|g' \ - -e 's|"timestamp": 97...|"timestamp": 42|g' + -e 's|"timestamp": 97...|"timestamp": 42|g' \ +-e 's|"content-length": [1-9][0-9]*|"content-length": "NONZERO"|g' } notmuch_emacs_error_sanitize () -- 2.10.2 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Bug#842291: notmuch processes frequently stuck in select()
Control: affects 842291 + gpgsm dirmngr On Wed 2016-11-23 03:50:40 -0500, David Bremner wrote: > David Bremner writes: > >> Brian May writes: >>> strace shows notmuch looping in select. >>> >>> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >>> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >>> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >>> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >>> etc >>> >> >> a backtrace would be helpful. >> >> d > > Nevermind, I managed to download the list archive for debian-devel and > replicate the bug. > > The bug seems to be related to smime signature verification. After > adding the attached mail message (and running notmuch-new), to replicate > the hang it suffices to run > > % notmuch show --decrypt id:20161116t143809.ga.21721.s...@fsing.rootsland.net > > > As far as workarounds, turning off decryption / signature verification > should allow you to at least view the thread. I've noticed similar behavior, and it seems to correlate with gpgsm asking dirmngr for an update to CRLs related to S/MIME certs. some CRLs simply do not respond at all (resulting in a timeout), and some do not respond, or are laggy, when accessed over tor specifically. I see a couple possible ways to consider resolving this, none of them great, and i don't know exactly how to implement any of them: 0) turn off CRL updates entirely during s/mime signature verification 1) do s/mime signature verification without CRL updates, but schedule CRL checks to happen in the background for dirmngr, so that future verifications will reflect the cert validity 2) have dirmngr avoid checking CRLs that it knows it has already updated recently 3) tell dirmngr to use much shorter CRL fetch timeouts Some example traffic from my dirmngr that uses tor (complete with timestamps indicating just how bad the delays can be): Nov 22 14:08:24 alice dirmngr[11976]: no CRL available for issuer id 770B4DA5913F2572B9F679AE0819FB7D77572689 Nov 22 14:08:24 alice dirmngr[11976]: fetching CRL from 'http://crl.ll.mit.edu/getcrl/LLCA3' Nov 22 14:08:44 alice dirmngr[11976]: resolving 'crl.ll.mit.edu' failed: No data Nov 22 14:08:44 alice dirmngr[11976]: can't connect to 'crl.ll.mit.edu': host not found Nov 22 14:08:44 alice dirmngr[11976]: error retrieving 'http://crl.ll.mit.edu/getcrl/LLCA3': Unknown host Nov 22 14:08:44 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host Nov 22 14:08:45 alice dirmngr[11976]: no CRL available for issuer id 770B4DA5913F2572B9F679AE0819FB7D77572689 Nov 22 14:08:45 alice dirmngr[11976]: fetching CRL from 'http://crl.ll.mit.edu/getcrl/LLCA3' Nov 22 14:09:05 alice dirmngr[11976]: resolving 'crl.ll.mit.edu' failed: No data Nov 22 14:09:05 alice dirmngr[11976]: can't connect to 'crl.ll.mit.edu': host not found Nov 22 14:09:05 alice dirmngr[11976]: error retrieving 'http://crl.ll.mit.edu/getcrl/LLCA3': Unknown host Nov 22 14:09:05 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host Nov 22 14:09:05 alice dirmngr[11976]: no CRL available for issuer id 26FD002905277B015EE9B2A3C092A348F28A4C6B Nov 22 14:09:05 alice dirmngr[11976]: fetching CRL from 'http://crl.startssl.com/sca-client1.crl' Nov 22 14:09:25 alice dirmngr[11976]: resolving 'crl.startssl.com' failed: No data Nov 22 14:09:25 alice dirmngr[11976]: can't connect to 'crl.startssl.com': host not found Nov 22 14:09:25 alice dirmngr[11976]: error retrieving 'http://crl.startssl.com/sca-client1.crl': Unknown host Nov 22 14:09:25 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host Nov 22 14:09:25 alice dirmngr[11976]: no CRL available for issuer id 26FD002905277B015EE9B2A3C092A348F28A4C6B Nov 22 14:09:25 alice dirmngr[11976]: fetching CRL from 'http://crl.startssl.com/sca-client1.crl' Nov 22 14:09:45 alice dirmngr[11976]: resolving 'crl.startssl.com' failed: No data Nov 22 14:09:45 alice dirmngr[11976]: can't connect to 'crl.startssl.com': host not found Nov 22 14:09:45 alice dirmngr[11976]: error retrieving 'http://crl.startssl.com/sca-client1.crl': Unknown host Nov 22 14:09:45 alice dirmngr[11976]: crl_fetch via DP failed: Unknown host that's a 20-second lag between each failed check, adding up to 80 seconds delay in rendering a single thread where 4 messages were signed by S/MIME keys signed by two different authorities. Fwiw, crl.ll.mit.edu doesn't seem to respond over tor on port 80 at all in some cases, and in other cases takes nearly a minute to reply: 0 dkg@alice:/tmp/cdtemp.Ue45bu$ time wget -q 'http://crl.ll.mit.edu/getcrl/LLCA3' real0m0.694s user0m0.008s sys 0m0.008s 0 dkg@alice:/tmp/cdtemp.Ue45bu$ time torsocks wget -q 'http://crl.ll.mit.edu/getcrl/LLCA3' real0m58.828s user0m0.008s sys 0m0.008s 0 dkg@alice:/tmp/cdtemp.Ue45bu$ Any thoughts on the best way to pursue this? --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.o
Re: [PATCH] tests: account for varying-size OpenPGP signatures
Daniel Kahn Gillmor writes: > @@ -37,7 +37,8 @@ test_expect_success 'emacs delivery of signed message' \ > test_begin_subtest "signature verification" > output=$(notmuch show --format=json --verify subject:"test signed message > 001" \ > | notmuch_json_show_sanitize \ > -| sed -e 's|"created": [1234567890]*|"created": 946728000|') > +| sed -e 's|"created": [1234567890]*|"created": 946728000|' \ > + -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/') there are a bunch of sed oneliners (or perl equivalent) collected in test-lib.sh as functions test_*_sanitize. I wonder if that would be worthwhile here, to have one place to update regexps etc... Something in the style of notmuch_show_sanitize wrapping a call to notmuch_json_show_sanitize ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] tests: account for varying-size OpenPGP signatures
GnuPG 2.1.16 is now injecting the full issuer fingerprint in its signatures, which makes them about 32 octets larger when ascii-armored. This change in size means that the size of the MIME parts will vary depending on the version of gpg that the user has installed. at any rate, the signature part should be non-zero, so we just test for that instead of an exact size. --- test/T350-crypto.sh | 20 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/test/T350-crypto.sh b/test/T350-crypto.sh index df2dc74..38f984f 100755 --- a/test/T350-crypto.sh +++ b/test/T350-crypto.sh @@ -37,7 +37,8 @@ test_expect_success 'emacs delivery of signed message' \ test_begin_subtest "signature verification" output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ -| sed -e 's|"created": [1234567890]*|"created": 946728000|') +| sed -e 's|"created": [1234567890]*|"created": 946728000|' \ + -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/') expected='[[[{"id": "X", "match": true, "excluded": false, @@ -59,7 +60,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ @@ -71,7 +72,8 @@ echo "${FINGERPRINT}:6:" | gpg --no-tty --import-ownertrust >>"$GNUPGHOME"/trust gpg --no-tty --check-trustdb >>"$GNUPGHOME"/trust.log 2>&1 output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ -| sed -e 's|"created": [1234567890]*|"created": 946728000|') +| sed -e 's|"created": [1234567890]*|"created": 946728000|'\ + -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/') expected='[[[{"id": "X", "match": true, "excluded": false, @@ -94,7 +96,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ @@ -105,7 +107,8 @@ test_begin_subtest "signature verification with signer key unavailable" mv "${GNUPGHOME}"{,.bak} output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ -| sed -e 's|"created": [1234567890]*|"created": 946728000|') +| sed -e 's|"created": [1234567890]*|"created": 946728000|' \ + -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/') expected='[[[{"id": "X", "match": true, "excluded": false, @@ -127,7 +130,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ @@ -348,7 +351,8 @@ y | gpg --no-tty --quiet --import output=$(notmuch show --format=json --verify subject:"test signed message 001" \ | notmuch_json_show_sanitize \ -| sed -e 's|"created": [1234567890]*|"created": 946728000|') +| sed -e 's|"created": [1234567890]*|"created": 946728000|' \ + -e 's/"content-length": [1-9][0-9]*/"content-length": "NONZERO"/') expected='[[[{"id": "X", "match": true, "excluded": false, @@ -370,7 +374,7 @@ expected='[[[{"id": "X", "content": "This is a test signed message.\n"}, {"id": 3, "content-type": "application/pgp-signature", - "content-length": 280}]}]}, + "content-length": "NONZERO"}]}]}, [' test_expect_equal_json \ "$output" \ -- 2.10.2 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: some issues with emacs 25
On 23/11/16 11:57, David Bremner wrote: > In a fresh emacs, can you generate an error using the recipe in > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24950 ? No. I can evaluate the recipe ok and eval of the last line yields: 10 (#o12, #xa, ?\C-j) ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: some issues with emacs 25
Matthew Lear writes: > I'd happily post some debug info to help analyse this if somebody could > give me a few pointers about what would be needed and the emacs-fu > needed to generate this. > > Cheers, > -- Matt In a fresh emacs, can you generate an error using the recipe in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24950 ? d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: some issues with emacs 25
On 22/11/16 14:24, Matthew Lear wrote: > I can confirm that it is indeed fixed. I'm not able to reproduce it with > the latest emacs 25 from git. Unfortunately, it seems I was too hasty in confirming this. For some reason unknown to me, replying to the html in some multipart/alternative works fine, but others cause the same error to be thrown. I've also seen the error occur when replying to a multipart/mixed message, too. Even more bizarrely, I'm currently looking at a thread with two emails where both are multipart/alternative and contain text/plain and text/html parts, and replying to the first message triggers the error/bug but replying to the second message does not. I can't see any formatting difference between the two. Sorry, but it would appear as though there is still some work needed to resolve this in it's entirety. I suppose it's likely that this could be an emacs-25 issue, but what I mentioned above makes no sense to me so I can't say for sure. I'd happily post some debug info to help analyse this if somebody could give me a few pointers about what would be needed and the emacs-fu needed to generate this. Cheers, -- Matt ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Bug#842291: notmuch processes frequently stuck in select()
David Bremner writes: > Brian May writes: >> strace shows notmuch looping in select. >> >> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >> select(10, [9], [], NULL, {1, 0}) = 0 (Timeout) >> etc >> > > a backtrace would be helpful. > > d Nevermind, I managed to download the list archive for debian-devel and replicate the bug. The bug seems to be related to smime signature verification. After adding the attached mail message (and running notmuch-new), to replicate the hang it suffices to run % notmuch show --decrypt id:20161116t143809.ga.21721.s...@fsing.rootsland.net As far as workarounds, turning off decryption / signature verification should allow you to at least view the thread. 20161116T143809.GA.21721.stse@fsing.rootsland.net.eml:2, Description: Binary data ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch