Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys
On Thu 2017-12-07 00:20:12 -0800, Jameson Graef Rollins wrote: > On Mon, Dec 04 2017, David Bremner wrote: >> Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed >> concerns/suggestions Jamie had, so I'm hoping he can have a quick look >> at those. > > Yes, this new series is great and definitely addresses all my concerns. > I'm stoked to see that the first part of it has been pushed, and looking > forward to the full series! > > This is really great progress, Daniel. Thanks for pushing on this. Thanks for the review! I've just pushed v3 of what remains of this series, which is basically the same as the remaining patches here, with a couple minor cleanups. you can find it starting at id:20171208062404.17269-1-...@fifthhorseman.net --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v3 11/15] cli/reindex: destroy stashed session keys when --decrypt=false
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --decrypt=false id:whate...@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
---
doc/man1/notmuch-reindex.rst | 3 +++
lib/message.cc| 5 +
test/T357-index-decryption.sh | 17 +
3 files changed, 25 insertions(+)
diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst
index d87e9d85..e8174f39 100644
--- a/doc/man1/notmuch-reindex.rst
+++ b/doc/man1/notmuch-reindex.rst
@@ -30,6 +30,9 @@ Supported options for **reindex** include
the user's secret keys. If decryption is successful, index
the cleartext itself.
+If ``false``, notmuch reindex will also delete any stashed
+session keys for all messages matching the search terms.
+
Be aware that the index is likely sufficient to reconstruct
the cleartext of the message itself, so please ensure that the
notmuch message index is adequately protected. DO NOT USE
diff --git a/lib/message.cc b/lib/message.cc
index 12743460..d5db89b6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -2002,6 +2002,11 @@ notmuch_message_reindex (notmuch_message_t *message,
ret = notmuch_message_remove_all_properties_with_prefix (message,
"index.");
if (ret)
goto DONE; /* XXX TODO: distinguish from other error returns above? */
+if (indexopts && notmuch_indexopts_get_decrypt_policy (indexopts) ==
NOTMUCH_DECRYPT_FALSE) {
+ ret = notmuch_message_remove_all_properties (message, "session-key");
+ if (ret)
+ goto DONE;
+}
/* re-add the filenames with the associated indexopts */
for (; notmuch_filenames_valid (orig_filenames);
diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index bd213415..9f46a01b 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -227,6 +227,23 @@ test_expect_equal \
"$output" \
"$expected"
+test_begin_subtest "purging stashed session keys should lose access to the
cleartext"
+notmuch reindex --decrypt=false id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "and cleartext should be unrecoverable now that there are
no stashed session keys"
+notmuch dump
+notmuch reindex --decrypt=true id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
# TODO: test removal of a message from the message store between
# indexing and reindexing.
--
2.15.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v3 07/15] cli/show, reply: document use of stashed session keys in notmuch-properties
The stashed session keys are stored internally as notmuch properties. So a user or developer who is reading about those properties might want to understand how they fit into the bigger picture. Note here that decrypting with a stored session key no longer needs -decrypt for "notmuch show" and "notmuch reply". --- doc/man7/notmuch-properties.rst | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst index 05444f6c..1a3f690e 100644 --- a/doc/man7/notmuch-properties.rst +++ b/doc/man7/notmuch-properties.rst @@ -77,9 +77,12 @@ of its normal activity. **session-key** When **notmuch-show(1)** or **nomtuch-reply** encounters a message -with an encrypted part and ``--decrypt`` is set, if notmuch finds a -``session-key`` property associated with the message, it will try -that stashed session key for decryption. +with an encrypted part, if notmuch finds a ``session-key`` +property associated with the message, it will try that stashed +session key for decryption. + +If you do not want to use any stashed session keys that might be +present, you should pass those programs ``--decrypt=false``. Using a stashed session key with "notmuch show" will speed up rendering of long encrypted threads. It also allows the user to -- 2.15.0 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys
On Mon, Dec 04 2017, David Bremner wrote: > Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed > concerns/suggestions Jamie had, so I'm hoping he can have a quick look > at those. Yes, this new series is great and definitely addresses all my concerns. I'm stoked to see that the first part of it has been pushed, and looking forward to the full series! This is really great progress, Daniel. Thanks for pushing on this. jamie. signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys
On Mon 2017-12-04 21:59:18 -0400, David Bremner wrote: > Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed > concerns/suggestions Jamie had, so I'm hoping he can have a quick look > at those. to be fair, i thought Jamie's concerns were correct -- the normalized interface is better. i was only a bit leery about the change introducing an extra delay. But Jamie's concerns were also corroborated by id:87r2szgvik@tethera.net -- maybe you could ask the author of that message to review? :P I think Jamie is currently slammed by non-notmuch work, unfortunately. --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys
Pushed patches 1 to 6. I seem to recall 7 and 8 basically adressed concerns/suggestions Jamie had, so I'm hoping he can have a quick look at those. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v2 13/21] cli/show, reply: document use of stashed session keys in notmuch-properties
The stashed session keys are stored internally as notmuch properties. So a user or developer who is reading about those properties might want to understand how they fit into the bigger picture. Note here that decrypting with a stored session key no longer needs -decrypt for "notmuch show" and "notmuch reply". --- doc/man7/notmuch-properties.rst | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst index f27dd607..1bb7b1ca 100644 --- a/doc/man7/notmuch-properties.rst +++ b/doc/man7/notmuch-properties.rst @@ -77,9 +77,12 @@ of its normal activity. **session-key** When **notmuch-show(1)** or **nomtuch-reply** encounters a message -with an encrypted part and ``--decrypt`` is set, if notmuch finds a -``session-key`` property associated with the message, it will try -that stashed session key for decryption. +with an encrypted part, if notmuch finds a ``session-key`` +property associated with the message, it will try that stashed +session key for decryption. + +If you do not want to use any stashed session keys that might be +present, you should pass those programs ``--decrypt=false``. Using a stashed session key with "notmuch show" will speed up rendering of long encrypted threads. It also allows the user to -- 2.15.0 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v2 06/21] crypto: Test restore of cleartext index from stashed session keys
If you've got a notmuch dump that includes stashed session keys for every decrypted message, and you've got your message archive, you should be able to get back to the same index that you had before. Here we add a simple test that give some flavor of how that works. --- test/T357-index-decryption.sh | 31 +++ 1 file changed, 31 insertions(+) diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh index 22e716c6..11ea2074 100755 --- a/test/T357-index-decryption.sh +++ b/test/T357-index-decryption.sh @@ -156,6 +156,37 @@ test_expect_equal \ "$output" \ "$expected" +add_email_corpus crypto + +test_begin_subtest "indexing message fails when secret key not available" +notmuch reindex --try-decrypt id:simple-encryp...@crypto.notmuchmail.org +output=$(notmuch dump ) +expected='#notmuch-dump batch-tag:3 config,properties,tags ++encrypted +inbox +unread -- id:simple-encryp...@crypto.notmuchmail.org +#= simple-encryp...@crypto.notmuchmail.org index.decryption=failure' +test_expect_equal \ +"$output" \ +"$expected" + +test_begin_subtest "cannot find cleartext index" +output=$(notmuch search sekrit) +expected='' +test_expect_equal \ +"$output" \ +"$expected" + +test_begin_subtest "cleartext index recovery on reindexing with stashed session keys" +notmuch restore <https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v2 17/21] cli/reindex: destroy stashed session keys when --decrypt=false
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --decrypt=false id:whate...@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
---
doc/man1/notmuch-reindex.rst | 3 +++
lib/message.cc| 5 +
test/T357-index-decryption.sh | 17 +
3 files changed, 25 insertions(+)
diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst
index d87e9d85..e8174f39 100644
--- a/doc/man1/notmuch-reindex.rst
+++ b/doc/man1/notmuch-reindex.rst
@@ -30,6 +30,9 @@ Supported options for **reindex** include
the user's secret keys. If decryption is successful, index
the cleartext itself.
+If ``false``, notmuch reindex will also delete any stashed
+session keys for all messages matching the search terms.
+
Be aware that the index is likely sufficient to reconstruct
the cleartext of the message itself, so please ensure that the
notmuch message index is adequately protected. DO NOT USE
diff --git a/lib/message.cc b/lib/message.cc
index 12743460..d5db89b6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -2002,6 +2002,11 @@ notmuch_message_reindex (notmuch_message_t *message,
ret = notmuch_message_remove_all_properties_with_prefix (message,
"index.");
if (ret)
goto DONE; /* XXX TODO: distinguish from other error returns above? */
+if (indexopts && notmuch_indexopts_get_decrypt_policy (indexopts) ==
NOTMUCH_DECRYPT_FALSE) {
+ ret = notmuch_message_remove_all_properties (message, "session-key");
+ if (ret)
+ goto DONE;
+}
/* re-add the filenames with the associated indexopts */
for (; notmuch_filenames_valid (orig_filenames);
diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index b2717a7a..64317c64 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -218,6 +218,23 @@ test_expect_equal \
"$output" \
"$expected"
+test_begin_subtest "purging stashed session keys should lose access to the
cleartext"
+notmuch reindex --decrypt=false id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "and cleartext should be unrecoverable now that there are
no stashed session keys"
+notmuch dump
+notmuch reindex --decrypt=true id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
# TODO: test removal of a message from the message store between
# indexing and reindexing.
--
2.15.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: Stashed session keys
Hi meskio-- On Wed 2017-11-15 23:41:28 +0100, meskio wrote: > Nice feature. I'm using it and it works fine. I notice some speed up, > improving > the painfulness of reading long encrypted threads in alot. And I like to > don't > be able to have around my old private keys. cool, i'm glad it's working for you! > I implemented some support for it in alot (using the patch I just sent adding > notmuch_message_get_property to the python binding): > https://github.com/meskio/alot/tree/session-key very nice :) --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Stashed session keys
Quoting Daniel Kahn Gillmor (2017-10-25 08:51:45) > Now that cleartext indexing is merged, let's add the ability to stash > session keys! Nice feature. I'm using it and it works fine. I notice some speed up, improving the painfulness of reading long encrypted threads in alot. And I like to don't be able to have around my old private keys. I implemented some support for it in alot (using the patch I just sent adding notmuch_message_get_property to the python binding): https://github.com/meskio/alot/tree/session-key Thanks for the work. -- meskio | http://meskio.net/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- My contact info: http://meskio.net/crypto.txt -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Nos vamos a Croatan. signature.asc Description: signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys
Daniel Kahn Gillmor writes: > On Tue 2017-11-14 09:13:52 -0400, David Bremner wrote: >> Daniel Kahn Gillmor writes: >> >>> If you've got a notmuch dump that includes stashed session keys for >>> every decrypted message, and you've got your message archive, you >>> should be able to get back to the same index that you had before. >> >> Out of curiousity, have you given any thought to what happens when >> someone sends a message with the same message-id but a different >> session-key? it seems like the user can potentially lose access to the >> encrypted message. > > yep! I even have that case in my own mailbox due to messages i've sent > to schleuder encrypted mailing lists to which i'm also subscribed. > > It works fine. notmuch stashes both session keys against the message-id > (you can have multiple properties with the same name as long as they > have different values). And upon decryption, it tries each session-key > in succession. This is a little bit sloppy (maybe it would be less > sloppy to associate each message key with each version of the message > somehow?), but it's significantly simpler and basically unnoticeable > compared to the speedup gains provided by the rest of the series. > > --dkg Great! d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys
On Tue 2017-11-14 09:13:52 -0400, David Bremner wrote: > Daniel Kahn Gillmor writes: > >> If you've got a notmuch dump that includes stashed session keys for >> every decrypted message, and you've got your message archive, you >> should be able to get back to the same index that you had before. > > Out of curiousity, have you given any thought to what happens when > someone sends a message with the same message-id but a different > session-key? it seems like the user can potentially lose access to the > encrypted message. yep! I even have that case in my own mailbox due to messages i've sent to schleuder encrypted mailing lists to which i'm also subscribed. It works fine. notmuch stashes both session keys against the message-id (you can have multiple properties with the same name as long as they have different values). And upon decryption, it tries each session-key in succession. This is a little bit sloppy (maybe it would be less sloppy to associate each message key with each version of the message somehow?), but it's significantly simpler and basically unnoticeable compared to the speedup gains provided by the rest of the series. --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys
Daniel Kahn Gillmor writes: > If you've got a notmuch dump that includes stashed session keys for > every decrypted message, and you've got your message archive, you > should be able to get back to the same index that you had before. > Out of curiousity, have you given any thought to what happens when someone sends a message with the same message-id but a different session-key? it seems like the user can potentially lose access to the encrypted message. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Stashed session keys
On Sun 2017-11-12 11:51:11 +0800, Daniel Kahn Gillmor wrote: > On Sat 2017-11-11 15:31:36 -0800, Jameson Graef Rollins wrote: >> I haven't decided what's the best way to do that yet, but something >> like the following happening automatically at inbox view might do the >> trick: >> >> notmuch reindex --try-decrypt=true (tag:inbox AND tag:encrypted) > > This seems like a reasonable way to ensure that your long-term, personal > secret keys only get accessed when you are interactively working with > your mail user agent. > > You might be able to target the reindex even more narrowly by adding > something like "AND not property:index-decryption=success" Sorry, this should be "AND not property:index.decryption=success" --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Stashed session keys
On Sun, Nov 12 2017, Daniel Kahn Gillmor wrote: >> Finally, I think it would be worthwhile to resolve the disparity between >> the usage of "decrypt" and "try-decrypt" in the CLI and config options. >> I'm not sure why we're using different terms in different contexts, even >> though the meanings are essentially the same. A follow-up patch series >> changing "try-decrypt" -> "decrypt" would probably be in order. > > If this series lands, i'd be happy to supply such an term-normalizing > series for subsequent consideration. > > If people feel that this term normalization is the main blocker to > landing this series, i could try to rebase it with a different UI terms, > but rebasing the series for this change feels like busy-work to me (and > would be more effort than a simple normalization patch on top). i'd > rather spend my limited notmuch hacking+reviewing time providing useful > features. I don't think it's a blocker at all, but I do think the term normalization should happen before the next release, since I think we should change "try-decrypt" -> "decrypt", which would affect things already in master. But I very much hope this series will land before the next release, in which case I think it's fine to wait to do the normalization after this series lands. jamie. signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Stashed session keys
On Sat 2017-11-11 15:31:36 -0800, Jameson Graef Rollins wrote: > I have reviewed and tested this series, and it seems solidly > implemented and very well motivated. I have been using it regularly > for a couple weeks now and have found no issues with it's usage, and > have noticed the considerable speed up when viewing encrypted threads > (as much as x8 for show on a thread of just 8 encrypted messages). I > fully support it's integration unconditionally. thanks for the review, the testing, and the reportback. I'm glad to hear that it's giving you the same sort of speedups that it gives me. > Daniel likes to think of this in terms of being able to "delete" > encrypted messages in the wild (via deletion of the original > encryption key) whereas I like to think of it in terms of preserving > access to received encrypted messages after key rotation. Both > benefits hold, though, obviously. yes, these are different ways of looking at the same key management strategy. > I think these policies cover all potential use cases that I can see. > However, there will need to be further work on the UX to make things > flow more smoothly. Agreed. The goal of this series is to provide the framework that can be used to build smoother UX, but it doesn't get all the way to providing the smoothest possible UX. Such is the nature of toolkit development. > I haven't decided what's the best way to do that yet, but something > like the following happening automatically at inbox view might do the > trick: > > notmuch reindex --try-decrypt=true (tag:inbox AND tag:encrypted) This seems like a reasonable way to ensure that your long-term, personal secret keys only get accessed when you are interactively working with your mail user agent. You might be able to target the reindex even more narrowly by adding something like "AND not property:index-decryption=success" > Finally, I think it would be worthwhile to resolve the disparity between > the usage of "decrypt" and "try-decrypt" in the CLI and config options. > I'm not sure why we're using different terms in different contexts, even > though the meanings are essentially the same. A follow-up patch series > changing "try-decrypt" -> "decrypt" would probably be in order. If this series lands, i'd be happy to supply such an term-normalizing series for subsequent consideration. If people feel that this term normalization is the main blocker to landing this series, i could try to rebase it with a different UI terms, but rebasing the series for this change feels like busy-work to me (and would be more effort than a simple normalization patch on top). i'd rather spend my limited notmuch hacking+reviewing time providing useful features. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Stashed session keys
On Wed, Oct 25 2017, Daniel Kahn Gillmor wrote: > Now that cleartext indexing is merged, let's add the ability to stash > session keys! I have reviewed and tested this series, and it seems solidly implemented and very well motivated. I have been using it regularly for a couple weeks now and have found no issues with it's usage, and have noticed the considerable speed up when viewing encrypted threads (as much as x8 for show on a thread of just 8 encrypted messages). I fully support it's integration unconditionally. It should be emphasized that this series is actually fairly critical for good support of message encryption. Without this series it's actually possible to completely lose access to encrypted mail if one were to rotate/replace their encryption key, which one might reasonably be expected to do. Without access to the original encryption key or the message session key, there is no way to access the contents of an encrypted message. If, however, the session key is stashed in the index, the original encryption key can be destroyed and the message can still be accessed. Daniel likes to think of this in terms of being able to "delete" encrypted messages in the wild (via deletion of the original encryption key) whereas I like to think of it in terms of preserving access to received encrypted messages after key rotation. Both benefits hold, though, obviously. >++---+--+-+--+ >|| false | auto | nostash | true | >++===+==+=+==+ >| Index cleartext using | | X |X| X | >| stashed session keys | | | | | >++---+--+-+--+ >| Index cleartext| | |X| X | >| using secret keys | | | | | >++---+--+-+--+ >| Stash session keys | | | | X | >++---+--+-+--+ >| Delete stashed session | X | | | | >| keys on reindex| | | | | >++---+--+-+--+ I think these policies cover all potential use cases that I can see. However, there will need to be further work on the UX to make things flow more smoothly. I've been using this series with index.try_decrypt set to 'true', which causes encrypted messages to be indexed on new. I do this because I don't want to be bothered to manually initiate indexing of encrypted messages. However, since my mail retrieval and indexing happen in the background, this has the unfortunate side effect that I am occasionally presented with a gpg agent prompt at random unexpected times. Ideally, one would leave index.try_decrypt set to 'auto', and there would be an easy/automatic way to prompt reindexing when the user is interacting directly with their MUA. I haven't decided what's the best way to do that yet, but something like the following happening automatically at inbox view might do the trick: notmuch reindex --try-decrypt=true (tag:inbox AND tag:encrypted) Finally, I think it would be worthwhile to resolve the disparity between the usage of "decrypt" and "try-decrypt" in the CLI and config options. I'm not sure why we're using different terms in different contexts, even though the meanings are essentially the same. A follow-up patch series changing "try-decrypt" -> "decrypt" would probably be in order. But these are next steps. The series in question here is absolutely ready, and needed, as is. jamie. signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: Stashed session keys
On Wed 2017-10-25 02:51:45 -0400, Daniel Kahn Gillmor wrote: > Now that cleartext indexing is merged, let's add the ability to stash > session keys! I'd really appreciate feedback on this series. It works for me, and i'm using it. I don't want this to take another two years to land, if possible. And it blocks additional improvements that i would like to eventually land as well. So any review of any part of this series would be appreciated. I want to make notmuch a mail client that provides actually usable encrypted mail. Regards, --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 05/18] crypto: Test restore of cleartext index from stashed session keys
If you've got a notmuch dump that includes stashed session keys for every decrypted message, and you've got your message archive, you should be able to get back to the same index that you had before. Here we add a simple test that give some flavor of how that works. --- test/T357-index-decryption.sh | 31 +++ 1 file changed, 31 insertions(+) diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh index 22e716c6..11ea2074 100755 --- a/test/T357-index-decryption.sh +++ b/test/T357-index-decryption.sh @@ -156,6 +156,37 @@ test_expect_equal \ "$output" \ "$expected" +add_email_corpus crypto + +test_begin_subtest "indexing message fails when secret key not available" +notmuch reindex --try-decrypt id:simple-encryp...@crypto.notmuchmail.org +output=$(notmuch dump ) +expected='#notmuch-dump batch-tag:3 config,properties,tags ++encrypted +inbox +unread -- id:simple-encryp...@crypto.notmuchmail.org +#= simple-encryp...@crypto.notmuchmail.org index.decryption=failure' +test_expect_equal \ +"$output" \ +"$expected" + +test_begin_subtest "cannot find cleartext index" +output=$(notmuch search sekrit) +expected='' +test_expect_equal \ +"$output" \ +"$expected" + +test_begin_subtest "cleartext index recovery on reindexing with stashed session keys" +notmuch restore <https://notmuchmail.org/mailman/listinfo/notmuch
Stashed session keys
Now that cleartext indexing is merged, let's add the ability to stash
session keys!
Background
==
Encrypted e-mail messages are "hybrid" encryption. The message body
is encrypted with an ephemeral session key, and then that session key
is itself encrypted to the user's public key.
If an MUA retains (or obtains) a copy of the session key for a given
message, it can access the cleartext of that message without needing
any access to the user's private key material.
This offers possible wins in efficiency, usability, convenience *and*
security, as the series hopefully makes clear.
Decryption Policies
===
At the end of the series, there are four sensible policies defined for
message decryption and stashing of session keys. There are only two i
expect to see any widespread regular use: "auto", and "true". But
hopefully the reasons for including the other two policies ("false"
and "nostash") are made clear by the series itself.
I'll replicate here the table this series adds to notmuch-config(1),
in describing the available values for index.try_decrypt:
++---+--+-+--+
|| false | auto | nostash | true |
++===+==+=+==+
| Index cleartext using | | X |X| X |
| stashed session keys | | | | |
++---+--+-+--+
| Index cleartext| | |X| X |
| using secret keys | | | | |
++---+--+-+--+
| Stash session keys | | | | X |
++---+--+-+--+
| Delete stashed session | X | | | |
| keys on reindex| | | | |
++---+--+-+--+
Please let me know what you think! I'd love feedback and critique.
Happy hacking,
--dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 14/18] cli/reindex: destroy stashed session keys when --try-decrypt=false
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --try-decrypt=false id:whate...@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
---
doc/man1/notmuch-reindex.rst | 3 +++
lib/message.cc| 5 +
test/T357-index-decryption.sh | 17 +
3 files changed, 25 insertions(+)
diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst
index b15981a2..6f5e48a9 100644
--- a/doc/man1/notmuch-reindex.rst
+++ b/doc/man1/notmuch-reindex.rst
@@ -30,6 +30,9 @@ Supported options for **reindex** include
the user's secret keys. If decryption is successful, index
the cleartext itself.
+If ``false``, notmuch reindex will also delete any stashed
+session keys for all messages matching the search terms.
+
Be aware that the index is likely sufficient to reconstruct
the cleartext of the message itself, so please ensure that the
notmuch message index is adequately protected. DO NOT USE
diff --git a/lib/message.cc b/lib/message.cc
index 12743460..cfd99130 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -2002,6 +2002,11 @@ notmuch_message_reindex (notmuch_message_t *message,
ret = notmuch_message_remove_all_properties_with_prefix (message,
"index.");
if (ret)
goto DONE; /* XXX TODO: distinguish from other error returns above? */
+if (indexopts && notmuch_indexopts_get_try_decrypt (indexopts) ==
NOTMUCH_DECRYPT_FALSE) {
+ ret = notmuch_message_remove_all_properties (message, "session-key");
+ if (ret)
+ goto DONE;
+}
/* re-add the filenames with the associated indexopts */
for (; notmuch_filenames_valid (orig_filenames);
diff --git a/test/T357-index-decryption.sh b/test/T357-index-decryption.sh
index 7fc59f1e..4f5a501a 100755
--- a/test/T357-index-decryption.sh
+++ b/test/T357-index-decryption.sh
@@ -218,6 +218,23 @@ test_expect_equal \
"$output" \
"$expected"
+test_begin_subtest "purging stashed session keys should lose access to the
cleartext"
+notmuch reindex --try-decrypt=false id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
+test_begin_subtest "and cleartext should be unrecoverable now that there are
no stashed session keys"
+notmuch dump
+notmuch reindex --try-decrypt=true id:simple-encryp...@crypto.notmuchmail.org
+output=$(notmuch search sekrit)
+expected=''
+test_expect_equal \
+"$output" \
+"$expected"
+
# TODO: test removal of a message from the message store between
# indexing and reindexing.
--
2.14.2
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 10/18] cli/show, reply: document use of stashed session keys in notmuch-properties
The stashed session keys are stored internally as notmuch properties. So a user or developer who is reading about those properties might want to understand how they fit into the bigger picture. Note here that decrypting with a stored session key no longer needs -decrypt for "notmuch show" and "notmuch reply". --- doc/man7/notmuch-properties.rst | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst index 31d7a104..e2ab43be 100644 --- a/doc/man7/notmuch-properties.rst +++ b/doc/man7/notmuch-properties.rst @@ -77,9 +77,12 @@ of its normal activity. **session-key** When **notmuch-show(1)** or **nomtuch-reply** encounters a message -with an encrypted part and ``--decrypt`` is set, if notmuch finds a -``session-key=`` property associated with the message, it will try -that stashed session key for decryption. +with an encrypted part, if notmuch finds a ``session-key=`` +property associated with the message, it will try that stashed +session key for decryption. + +If you do not want to use any stashed session keys that might be +present, you should pass those programs ``--decrypt=false``. Using a stashed session key with "notmuch show" will speed up rendering of long encrypted threads. It also allows the user to -- 2.14.2 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
