[PATCH 0/3] Outline fix for emacs tagging race
On Fri, Nov 09 2012, Mark Walters wrote:
>
> This patch series implements the above mentioned solution and seems to
> work except for one problem.
>
> Since emacs now tags each message in a thread in the search buffer it
> is easy to ask it to tag a lot of messages. This could be done
> individually which would be ridiculously slow so instead they are all
> done in one batch. But now it is relatively easy to take notmuch over
> the threshold for ARG_MAX.
>
> In [3] Tomi did some experiments and found on a standard Debian system
> with getconf ARG_MAX =131072 that command lines with 1 200 byte
> arguments worked. I am a little puzzled by that as I get the same
> results and I getconf ARG_MAX gives 2097152 for me.
>
> More importantly though, when trying to execute a command from emacs I
> am finding that 131072 is the limit on the command length in bytes and
> we can hit this with something around 1500 messages (see end for a
> very hacky emacs test script). This is probably more than we can
> expect in a single thread so tagging from show is probably safe but it
> does cause a problem when tagging from search.
That's because your test script below constructs one very lng
string (and used just one arg). Like you said, [3] used 1 200-char args.
This test script is more like what the perl(1) script did in [3]
(progn
(setq arglist
'("aaa"))
(setq n 1)
(while (> n 0)
(setq n (1- n))
(setq arglist (cons
"bbb"
arglist)))
(apply 'call-process "/bin/echo" nil t nil arglist))
(except that the args here are just 80 chars but you should get the point :)
I did not check your patches yet.. buf if you concatenate there you
could try to provide those as separate args...
... then we can experiment how notmuch (and xapian) copes with thosew
IIRC notmuch will concatenate one long string out of those and do
xapian stuff using that (but then we're not limited by command line
arguments restrictions).
>
> Best wishes
>
> Mark
Tomi
>
> [3] id:m2liody7av.fsf at guru.guru-group.fi
> TEST SCRIPT
> (progn
> (setq bigstring
> "")
> (setq n 1310)
> (setq big nil)
> (while (> n 0)
>(setq n (1- n))
> (setq big (concat big (format "%s" n) " " bigstring)))
> (call-process "echo" nil t nil big))
[PATCH 0/3] Outline fix for emacs tagging race
Mark Walters writes: > I can think of several possible solutions (e.g., batch it in my new > stuff, put some batching in notmuch-tag, all notmuch tag to read a > query from stdin). But before any larger more intrusive change do > people like the general approach? Does anyone have a good way to get > round the command line size problem? Jani posted some patches to do batch tagging from stdin. I guess the last full series posted was at id:cover.1334404979.git.jani at nikula.org I believe Jani has slightly updated (or at least rebased) the patches in git. I reworked and re-posted some of the foundational parts of the series at id:1345382314-5330-1-git-send-email-david at tethera.net I want to have some of this infrastucture for a new style of backup/restore, and it's possible that batch tagging would also speed up nmbug. d
[PATCH 3/3] emacs: make emacs use message-ids for tagging
This makes emacs use the new --output=with-ids in search mode and use this for tagging. This fixes the race condition in tagging from search mode so mark the tests fixed. --- emacs/notmuch.el | 22 -- test/emacs |2 -- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/emacs/notmuch.el b/emacs/notmuch.el index f9454d8..49a57b2 100644 --- a/emacs/notmuch.el +++ b/emacs/notmuch.el @@ -492,6 +492,21 @@ If BARE is set then do not prefix with \"thread:\"" "Return a search string for threads for the current region" (mapconcat 'identity (notmuch-search-find-thread-id-region beg end) " or ")) +(defun notmuch-search-find-msg-ids-region (beg end only-matching) + "Return a list of msg-ids for the current region" + (let ((msg-ids)) +(mapc (lambda (msg-id) (setq msg-ids (append msg-id msg-ids))) + (append (notmuch-search-properties-in-region :matching_msg_ids beg end) + (unless only-matching + (notmuch-search-properties-in-region :nonmatching_msg_ids beg end +msg-ids)) + +(defun notmuch-search-find-msg-ids-region-search (beg end only-matching) + "Return a search string for msg-ids in threads for the current region" + (mapconcat (lambda (id) (notmuch-id-to-query id)) +(notmuch-search-find-msg-ids-region beg end only-matching) +" or ")) + (defun notmuch-search-find-authors () "Return the authors for the current thread" (plist-get (notmuch-search-get-result) :authors)) @@ -566,7 +581,7 @@ and will also appear in a buffer named \"*Notmuch errors*\"." (defun notmuch-search-tag-region (beg end tag-changes) "Change tags for threads in the given region." - (let ((search-string (notmuch-search-find-thread-id-region-search beg end))) + (let ((search-string (notmuch-search-find-msg-ids-region-search beg end))) (setq tag-changes (funcall 'notmuch-tag search-string tag-changes)) (notmuch-search-foreach-result beg end (lambda (pos) @@ -842,7 +857,9 @@ non-authors is found, assume that all of the authors match." See `notmuch-tag' for information on the format of TAG-CHANGES." (interactive) - (apply 'notmuch-tag notmuch-search-query-string tag-changes)) + (apply 'notmuch-tag (notmuch-search-find-msg-ids-region-search + (point-min) (point-max) t) +tag-changes)) (defun notmuch-search-buffer-title (query) "Returns the title for a buffer with notmuch search results." @@ -939,6 +956,7 @@ Other optional parameters are used as follows: "notmuch-search" buffer notmuch-command "search" "--format=json" +"--output=with-ids" (if oldest-first "--sort=oldest-first" "--sort=newest-first") diff --git a/test/emacs b/test/emacs index 6246177..4102198 100755 --- a/test/emacs +++ b/test/emacs @@ -123,7 +123,6 @@ output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize) test_expect_equal "$output" "thread:XXX 2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)" test_begin_subtest "Tag all matching messages from search view" -test_subtest_known_broken notmuch tag +test-tag-race from:cworth test_emacs "(notmuch-search \"tag:test-tag-race\") (notmuch-test-wait)" @@ -135,7 +134,6 @@ notmuch tag -test-tag-race '*' notmuch tag -test-tag-race-2 '*' test_begin_subtest "Change tags from search view: another message arriving after thread lookup" -test_subtest_known_broken typsos_id="878we4qdqf.fsf at yoom.home.cworth.org" typsos_thread=$(notmuch search --output=threads id:$typsos_id) test_emacs "(notmuch-search \"$typsos_thread\") -- 1.7.9.1
[PATCH 2/3] cli: all search mode to include msg-ids with JSON output
This adds a --output=with-ids option which gives similar output to the
normal search summary output but with a list of message ids
too. Currently this is not implemented for text format.
---
notmuch-search.c | 40 ++--
1 files changed, 38 insertions(+), 2 deletions(-)
diff --git a/notmuch-search.c b/notmuch-search.c
index 830c4e4..82e168c 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -26,7 +26,8 @@ typedef enum {
OUTPUT_THREADS,
OUTPUT_MESSAGES,
OUTPUT_FILES,
-OUTPUT_TAGS
+OUTPUT_TAGS,
+OUTPUT_SUMMARY_WITH_IDS
} output_t;
static char *
@@ -46,6 +47,23 @@ sanitize_string (const void *ctx, const char *str)
return out;
}
+static void
+output_msg_ids (sprinter_t *format,
+ notmuch_bool_t matching,
+ notmuch_messages_t *messages)
+{
+notmuch_message_t *message;
+for (;
+notmuch_messages_valid (messages);
+notmuch_messages_move_to_next (messages))
+{
+ message = notmuch_messages_get (messages);
+ if (notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ==
matching)
+ format->string (format, notmuch_message_get_message_id (message));
+ output_msg_ids (format, matching, notmuch_message_get_replies
(message));
+}
+}
+
static int
do_search_threads (sprinter_t *format,
notmuch_query_t *query,
@@ -88,7 +106,7 @@ do_search_threads (sprinter_t *format,
format->string (format,
notmuch_thread_get_thread_id (thread));
format->separator (format);
- } else { /* output == OUTPUT_SUMMARY */
+ } else { /* output == OUTPUT_SUMMARY or OUTPUT_SUMMARY_WITH_IDS */
void *ctx_quote = talloc_new (thread);
const char *authors = notmuch_thread_get_authors (thread);
const char *subject = notmuch_thread_get_subject (thread);
@@ -160,6 +178,18 @@ do_search_threads (sprinter_t *format,
printf (")");
format->end (format);
+
+ if (output == OUTPUT_SUMMARY_WITH_IDS) {
+ format->map_key (format, "matching_msg_ids");
+ format->begin_list (format);
+ output_msg_ids (format, TRUE, notmuch_thread_get_toplevel_messages
(thread));
+ format->end (format);
+ format->map_key (format, "nonmatching_msg_ids");
+ format->begin_list (format);
+ output_msg_ids (format, FALSE, notmuch_thread_get_toplevel_messages
(thread));
+ format->end (format);
+ }
+
format->end (format);
format->separator (format);
}
@@ -323,6 +353,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
{ "messages", OUTPUT_MESSAGES },
{ "files", OUTPUT_FILES },
{ "tags", OUTPUT_TAGS },
+ { "with-ids", OUTPUT_SUMMARY_WITH_IDS },
{ 0, 0 } } },
{ NOTMUCH_OPT_KEYWORD, , "exclude", 'x',
(notmuch_keyword_t []){ { "true", EXCLUDE_TRUE },
@@ -398,6 +429,11 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
notmuch_query_set_omit_excluded (query, FALSE);
}
+if (output == OUTPUT_SUMMARY_WITH_IDS && format_sel ==
NOTMUCH_FORMAT_TEXT) {
+ fprintf (stderr, "Warning: --output=with-ids not implemented for text
format.\n");
+ output = OUTPUT_SUMMARY;
+}
+
switch (output) {
default:
case OUTPUT_SUMMARY:
--
1.7.9.1
[PATCH 1/3] test: test for race when tagging from emacs search
When tagging from search view in emacs there is a race condition: it tags all messages in the thread even ones which arrived after the search was made. This can cause dataloss (if, for example, a thread is archived it could archive messages the user has never seen). Mark this test known broken. --- test/emacs | 23 +++ 1 files changed, 23 insertions(+), 0 deletions(-) diff --git a/test/emacs b/test/emacs index 44f641e..6246177 100755 --- a/test/emacs +++ b/test/emacs @@ -122,6 +122,29 @@ test_emacs "(notmuch-search \"$os_x_darwin_thread\") output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize) test_expect_equal "$output" "thread:XXX 2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)" +test_begin_subtest "Tag all matching messages from search view" +test_subtest_known_broken +notmuch tag +test-tag-race from:cworth +test_emacs "(notmuch-search \"tag:test-tag-race\") + (notmuch-test-wait)" +notmuch tag +test-tag-race "id:1258471718-6781-2-git-send-email-dottedmag at dottedmag.net" +test_emacs "(execute-kbd-macro \"*+test-tag-race-2\")" +output=$(notmuch count tag:test-tag-race-2) +test_expect_equal "$output" "12" +notmuch tag -test-tag-race '*' +notmuch tag -test-tag-race-2 '*' + +test_begin_subtest "Change tags from search view: another message arriving after thread lookup" +test_subtest_known_broken +typsos_id="878we4qdqf.fsf at yoom.home.cworth.org" +typsos_thread=$(notmuch search --output=threads id:$typsos_id) +test_emacs "(notmuch-search \"$typsos_thread\") + (notmuch-test-wait)" +add_message "[subject]=\"new-thread-message\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -\"" "[body]=\"new-thread-message\"" "[in-reply-to]=\"<$typsos_id>\"" +test_emacs "(execute-kbd-macro \"+tag-from-search-view -unread\")" +output=$(notmuch search tag:tag-from-search-view | notmuch_search_sanitize) +test_expect_equal "$output" "thread:XXX 2009-11-18 [2/3] Ingmar Vanhassel, Carl Worth| Notmuch Test Suite; [notmuch] [PATCH] Typsos (inbox tag-from-search-view unread)" + test_begin_subtest "Add tag from notmuch-show view" test_emacs "(notmuch-show \"$os_x_darwin_thread\") (execute-kbd-macro \"+tag-from-show-view\")" -- 1.7.9.1
[PATCH 0/3] Outline fix for emacs tagging race
For a long time [1] there have been two related races in tagging from the search buffer. The first is that when tagging (including archiving) a thread message which arrived after the buffer was created may also be tagged. This is because the tagging is done based on the thread-id not on the individual messages. The second is when using the '*' command to tag all messages. This is not quite the same as this command only tags messages matching the query not all messages in all threads that contain a message matching the query. Thus if more messages now match than when the buffer was created (eg some external tagging script has run) then this command can unexpectedly tag these messages too. One solution that was discussed in [2] was for the search output of notmuch to include the message-ids of both matching and non-matching messages. At that time that was difficult to implement as it was unclear how to escape the message ids when using the text format. Since emacs now uses JSON for search mode this problem is solved. This patch series implements the above mentioned solution and seems to work except for one problem. Since emacs now tags each message in a thread in the search buffer it is easy to ask it to tag a lot of messages. This could be done individually which would be ridiculously slow so instead they are all done in one batch. But now it is relatively easy to take notmuch over the threshold for ARG_MAX. In [3] Tomi did some experiments and found on a standard Debian system with getconf ARG_MAX =131072 that command lines with 1 200 byte arguments worked. I am a little puzzled by that as I get the same results and I getconf ARG_MAX gives 2097152 for me. More importantly though, when trying to execute a command from emacs I am finding that 131072 is the limit on the command length in bytes and we can hit this with something around 1500 messages (see end for a very hacky emacs test script). This is probably more than we can expect in a single thread so tagging from show is probably safe but it does cause a problem when tagging from search. I can think of several possible solutions (e.g., batch it in my new stuff, put some batching in notmuch-tag, all notmuch tag to read a query from stdin). But before any larger more intrusive change do people like the general approach? Does anyone have a good way to get round the command line size problem? Best wishes Mark [1] id:87ocmtg9ni.fsf at yoom.home.cworth.org [2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=UN7McECrg at mail.gmail.com [3] id:m2liody7av.fsf at guru.guru-group.fi Mark Walters (3): test: test for race when tagging from emacs search cli: all search mode to include msg-ids with JSON output emacs: make emacs use message-ids for tagging emacs/notmuch.el | 22 -- notmuch-search.c | 40 ++-- test/emacs | 21 + 3 files changed, 79 insertions(+), 4 deletions(-) TEST SCRIPT (progn (setq bigstring "") (setq n 1310) (setq big nil) (while (> n 0) (setq n (1- n)) (setq big (concat big (format "%s" n) " " bigstring))) (call-process "echo" nil t nil big)) -- 1.7.9.1
[PATCH] contrib: pick: bugfix when trying to show a non-message
Quoth Mark Walters on Nov 01 at 7:26 pm: > If the user pressed return on the end result status line it gave a > blank message. Modify the function notmuch-pick-get-message-id to > return nil rather than an empty message-id in this case to fix this. > --- > contrib/notmuch-pick/notmuch-pick.el |4 +++- > 1 files changed, 3 insertions(+), 1 deletions(-) > > diff --git a/contrib/notmuch-pick/notmuch-pick.el > b/contrib/notmuch-pick/notmuch-pick.el > index 15ac5e8..1886469 100644 > --- a/contrib/notmuch-pick/notmuch-pick.el > +++ b/contrib/notmuch-pick/notmuch-pick.el > @@ -241,7 +241,9 @@ Some useful entries are: > > (defun notmuch-pick-get-message-id () >"Return the message id of the current message." > - (concat "id:\"" (notmuch-pick-get-prop :id) "\"")) > + (let ((id (notmuch-pick-get-prop :id))) > +(when id > + (concat "id:\"" id "\"" There's another bug here. The ID query should be constructed with notmuch-id-to-query so it gets properly escaped. > > (defun notmuch-pick-get-match () >"Return whether the current message is a match."
[PATCH] contrib: pick: bugfix when trying to show a non-message
On Fri, Nov 09 2012, David Bremner wrote: > Mark Walters writes: > >> - (concat "id:\"" (notmuch-pick-get-prop :id) "\"")) >> + (let ((id (notmuch-pick-get-prop :id))) >> +(when id >> + (concat "id:\"" id "\"" > > I don't know how other people feel, but I'd rather have an `if' in a > context where I care about the return value. Does (when COND BODY) connotate to the thought that the return value is generally not used ? That expands to (if COND (progn BODY)) so both returns exactly same values. (not that I wouldn't prefer if there :) But instead of this the function in question could use recently introduced (notmuch-id-to-query id) instead. Tomi
Notmuch indexing takes too long
Hi Ondrej, Quoting Ondrej Jombik (2012-11-09 02:58:09) > I am trying to move from mairix to some better solution. mairix has been > working really well for me, but it had some limitations. > > I decided to give a try to notmuch, but I has been suprised with > estimated indexing time: > > Processed 4157 of 822462 files (10h 3m 42s remaining). > > At the beggining when there was 11 hours estimate I simply thought that > those numbers are wrong. But they are not. This is really going to take > that long. I could not believe that you all wait that long to index your > maildirs:) > > Also if I understand correctly the concept, whenever I want to > regenerate database, I have to move old away and create new one, > correct? mairix worked that way that you run it and it updated its own > database. Is it possible with notmuch as well? I simply do not want to > wait 10+ hours for every database regeneration, thought I am willing to > wait that long for initial indexing. Notmuch does incremental idnex updates once the initial index is build. The initial indexing takes a while, depending on the number of mails you have but 10 *hours* seems a little off. For me its ~5mins on the hdd and <2 on a ssd. > My last question is related to notmuch and Alpine. Alpine is > a IMAP/Maildir e-mail client based on old PINE. I get used to that so > much that I cannot move away (to Mutt, for example). Is there anyone who > uses notmuch with Alpine and created some scripts and/or key-bindings? You might want to have a look at notmuchfs: https://github.com/tsto/notmuchfs Also, there are a few other MUAs based on notmuch: Check out http://notmuchmail.org/frontends/ Have fun, /p
[PATCH] contrib: pick: bugfix when trying to show a non-message
Tomi Ollila writes: > Does (when COND BODY) connotate to the thought that the return value > is generally not used ? That expands to (if COND (progn BODY)) so both > returns exactly same values. Err, sorry. Too much scheme. I meant a two branch if (if COND BODY nil)
Notmuch indexing takes too long
I am trying to move from mairix to some better solution. mairix has been working really well for me, but it had some limitations. I decided to give a try to notmuch, but I has been suprised with estimated indexing time: Processed 4157 of 822462 files (10h 3m 42s remaining). At the beggining when there was 11 hours estimate I simply thought that those numbers are wrong. But they are not. This is really going to take that long. I could not believe that you all wait that long to index your maildirs:) Also if I understand correctly the concept, whenever I want to regenerate database, I have to move old away and create new one, correct? mairix worked that way that you run it and it updated its own database. Is it possible with notmuch as well? I simply do not want to wait 10+ hours for every database regeneration, thought I am willing to wait that long for initial indexing. My last question is related to notmuch and Alpine. Alpine is a IMAP/Maildir e-mail client based on old PINE. I get used to that so much that I cannot move away (to Mutt, for example). Is there anyone who uses notmuch with Alpine and created some scripts and/or key-bindings? Of course I am not using nano for editing. I use ViM for e-mail creation and answering. So maybe notmuch-vim is probably better option, I do not know. Thank you for any possible valuable answers and please Cc me in the answer as I am not in the list. Thank you Ondrej -- Ondrej JOMBIK Platon Technologies s.r.o., Hlavna 3, Sala SK-92701 +421 903 PLATON - info at platon.org - http://platon.org My current location: Phoenix, Arizona My current timezone: -0700 UTC (MST) (updated automatically)
Re: [PATCH] contrib: pick: bugfix when trying to show a non-message
On Fri, Nov 09 2012, David Bremner da...@tethera.net wrote: Mark Walters markwalters1...@gmail.com writes: - (concat id:\ (notmuch-pick-get-prop :id) \)) + (let ((id (notmuch-pick-get-prop :id))) +(when id + (concat id:\ id \ I don't know how other people feel, but I'd rather have an `if' in a context where I care about the return value. Does (when COND BODY) connotate to the thought that the return value is generally not used ? That expands to (if COND (progn BODY)) so both returns exactly same values. (not that I wouldn't prefer if there :) But instead of this the function in question could use recently introduced (notmuch-id-to-query id) instead. Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Notmuch indexing takes too long
Hi Ondrej, Quoting Ondrej Jombik (2012-11-09 02:58:09) I am trying to move from mairix to some better solution. mairix has been working really well for me, but it had some limitations. I decided to give a try to notmuch, but I has been suprised with estimated indexing time: Processed 4157 of 822462 files (10h 3m 42s remaining). At the beggining when there was 11 hours estimate I simply thought that those numbers are wrong. But they are not. This is really going to take that long. I could not believe that you all wait that long to index your maildirs:) Also if I understand correctly the concept, whenever I want to regenerate database, I have to move old away and create new one, correct? mairix worked that way that you run it and it updated its own database. Is it possible with notmuch as well? I simply do not want to wait 10+ hours for every database regeneration, thought I am willing to wait that long for initial indexing. Notmuch does incremental idnex updates once the initial index is build. The initial indexing takes a while, depending on the number of mails you have but 10 *hours* seems a little off. For me its ~5mins on the hdd and 2 on a ssd. My last question is related to notmuch and Alpine. Alpine is a IMAP/Maildir e-mail client based on old PINE. I get used to that so much that I cannot move away (to Mutt, for example). Is there anyone who uses notmuch with Alpine and created some scripts and/or key-bindings? You might want to have a look at notmuchfs: https://github.com/tsto/notmuchfs Also, there are a few other MUAs based on notmuch: Check out http://notmuchmail.org/frontends/ Have fun, /p ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] contrib: pick: bugfix when trying to show a non-message
Tomi Ollila tomi.oll...@iki.fi writes: Does (when COND BODY) connotate to the thought that the return value is generally not used ? That expands to (if COND (progn BODY)) so both returns exactly same values. Err, sorry. Too much scheme. I meant a two branch if (if COND BODY nil) ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] contrib: pick: bugfix when trying to show a non-message
Quoth Mark Walters on Nov 01 at 7:26 pm: If the user pressed return on the end result status line it gave a blank message. Modify the function notmuch-pick-get-message-id to return nil rather than an empty message-id in this case to fix this. --- contrib/notmuch-pick/notmuch-pick.el |4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/contrib/notmuch-pick/notmuch-pick.el b/contrib/notmuch-pick/notmuch-pick.el index 15ac5e8..1886469 100644 --- a/contrib/notmuch-pick/notmuch-pick.el +++ b/contrib/notmuch-pick/notmuch-pick.el @@ -241,7 +241,9 @@ Some useful entries are: (defun notmuch-pick-get-message-id () Return the message id of the current message. - (concat id:\ (notmuch-pick-get-prop :id) \)) + (let ((id (notmuch-pick-get-prop :id))) +(when id + (concat id:\ id \ There's another bug here. The ID query should be constructed with notmuch-id-to-query so it gets properly escaped. (defun notmuch-pick-get-match () Return whether the current message is a match. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 1/3] test: test for race when tagging from emacs search
When tagging from search view in emacs there is a race condition: it tags all messages in the thread even ones which arrived after the search was made. This can cause dataloss (if, for example, a thread is archived it could archive messages the user has never seen). Mark this test known broken. --- test/emacs | 23 +++ 1 files changed, 23 insertions(+), 0 deletions(-) diff --git a/test/emacs b/test/emacs index 44f641e..6246177 100755 --- a/test/emacs +++ b/test/emacs @@ -122,6 +122,29 @@ test_emacs (notmuch-search \$os_x_darwin_thread\) output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize) test_expect_equal $output thread:XXX 2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread) +test_begin_subtest Tag all matching messages from search view +test_subtest_known_broken +notmuch tag +test-tag-race from:cworth +test_emacs (notmuch-search \tag:test-tag-race\) + (notmuch-test-wait) +notmuch tag +test-tag-race id:1258471718-6781-2-git-send-email-dotted...@dottedmag.net +test_emacs (execute-kbd-macro \*+test-tag-race-2\) +output=$(notmuch count tag:test-tag-race-2) +test_expect_equal $output 12 +notmuch tag -test-tag-race '*' +notmuch tag -test-tag-race-2 '*' + +test_begin_subtest Change tags from search view: another message arriving after thread lookup +test_subtest_known_broken +typsos_id=878we4qdqf@yoom.home.cworth.org +typsos_thread=$(notmuch search --output=threads id:$typsos_id) +test_emacs (notmuch-search \$typsos_thread\) + (notmuch-test-wait) +add_message [subject]=\new-thread-message\ [date]=\Sat, 01 Jan 2000 12:00:00 -\ [body]=\new-thread-message\ [in-reply-to]=\$typsos_id\ +test_emacs (execute-kbd-macro \+tag-from-search-view -unread\) +output=$(notmuch search tag:tag-from-search-view | notmuch_search_sanitize) +test_expect_equal $output thread:XXX 2009-11-18 [2/3] Ingmar Vanhassel, Carl Worth| Notmuch Test Suite; [notmuch] [PATCH] Typsos (inbox tag-from-search-view unread) + test_begin_subtest Add tag from notmuch-show view test_emacs (notmuch-show \$os_x_darwin_thread\) (execute-kbd-macro \+tag-from-show-view\) -- 1.7.9.1 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 0/3] Outline fix for emacs tagging race
For a long time [1] there have been two related races in tagging from the search buffer. The first is that when tagging (including archiving) a thread message which arrived after the buffer was created may also be tagged. This is because the tagging is done based on the thread-id not on the individual messages. The second is when using the '*' command to tag all messages. This is not quite the same as this command only tags messages matching the query not all messages in all threads that contain a message matching the query. Thus if more messages now match than when the buffer was created (eg some external tagging script has run) then this command can unexpectedly tag these messages too. One solution that was discussed in [2] was for the search output of notmuch to include the message-ids of both matching and non-matching messages. At that time that was difficult to implement as it was unclear how to escape the message ids when using the text format. Since emacs now uses JSON for search mode this problem is solved. This patch series implements the above mentioned solution and seems to work except for one problem. Since emacs now tags each message in a thread in the search buffer it is easy to ask it to tag a lot of messages. This could be done individually which would be ridiculously slow so instead they are all done in one batch. But now it is relatively easy to take notmuch over the threshold for ARG_MAX. In [3] Tomi did some experiments and found on a standard Debian system with getconf ARG_MAX =131072 that command lines with 1 200 byte arguments worked. I am a little puzzled by that as I get the same results and I getconf ARG_MAX gives 2097152 for me. More importantly though, when trying to execute a command from emacs I am finding that 131072 is the limit on the command length in bytes and we can hit this with something around 1500 messages (see end for a very hacky emacs test script). This is probably more than we can expect in a single thread so tagging from show is probably safe but it does cause a problem when tagging from search. I can think of several possible solutions (e.g., batch it in my new stuff, put some batching in notmuch-tag, all notmuch tag to read a query from stdin). But before any larger more intrusive change do people like the general approach? Does anyone have a good way to get round the command line size problem? Best wishes Mark [1] id:87ocmtg9ni@yoom.home.cworth.org [2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=un7mce...@mail.gmail.com [3] id:m2liody7av@guru.guru-group.fi Mark Walters (3): test: test for race when tagging from emacs search cli: all search mode to include msg-ids with JSON output emacs: make emacs use message-ids for tagging emacs/notmuch.el | 22 -- notmuch-search.c | 40 ++-- test/emacs | 21 + 3 files changed, 79 insertions(+), 4 deletions(-) TEST SCRIPT (progn (setq bigstring ) (setq n 1310) (setq big nil) (while ( n 0) (setq n (1- n)) (setq big (concat big (format %s n) bigstring))) (call-process echo nil t nil big)) -- 1.7.9.1 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 3/3] emacs: make emacs use message-ids for tagging
This makes emacs use the new --output=with-ids in search mode and use this for tagging. This fixes the race condition in tagging from search mode so mark the tests fixed. --- emacs/notmuch.el | 22 -- test/emacs |2 -- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/emacs/notmuch.el b/emacs/notmuch.el index f9454d8..49a57b2 100644 --- a/emacs/notmuch.el +++ b/emacs/notmuch.el @@ -492,6 +492,21 @@ If BARE is set then do not prefix with \thread:\ Return a search string for threads for the current region (mapconcat 'identity (notmuch-search-find-thread-id-region beg end) or )) +(defun notmuch-search-find-msg-ids-region (beg end optional only-matching) + Return a list of msg-ids for the current region + (let ((msg-ids)) +(mapc (lambda (msg-id) (setq msg-ids (append msg-id msg-ids))) + (append (notmuch-search-properties-in-region :matching_msg_ids beg end) + (unless only-matching + (notmuch-search-properties-in-region :nonmatching_msg_ids beg end +msg-ids)) + +(defun notmuch-search-find-msg-ids-region-search (beg end optional only-matching) + Return a search string for msg-ids in threads for the current region + (mapconcat (lambda (id) (notmuch-id-to-query id)) +(notmuch-search-find-msg-ids-region beg end only-matching) + or )) + (defun notmuch-search-find-authors () Return the authors for the current thread (plist-get (notmuch-search-get-result) :authors)) @@ -566,7 +581,7 @@ and will also appear in a buffer named \*Notmuch errors*\. (defun notmuch-search-tag-region (beg end optional tag-changes) Change tags for threads in the given region. - (let ((search-string (notmuch-search-find-thread-id-region-search beg end))) + (let ((search-string (notmuch-search-find-msg-ids-region-search beg end))) (setq tag-changes (funcall 'notmuch-tag search-string tag-changes)) (notmuch-search-foreach-result beg end (lambda (pos) @@ -842,7 +857,9 @@ non-authors is found, assume that all of the authors match. See `notmuch-tag' for information on the format of TAG-CHANGES. (interactive) - (apply 'notmuch-tag notmuch-search-query-string tag-changes)) + (apply 'notmuch-tag (notmuch-search-find-msg-ids-region-search + (point-min) (point-max) t) +tag-changes)) (defun notmuch-search-buffer-title (query) Returns the title for a buffer with notmuch search results. @@ -939,6 +956,7 @@ Other optional parameters are used as follows: notmuch-search buffer notmuch-command search --format=json +--output=with-ids (if oldest-first --sort=oldest-first --sort=newest-first) diff --git a/test/emacs b/test/emacs index 6246177..4102198 100755 --- a/test/emacs +++ b/test/emacs @@ -123,7 +123,6 @@ output=$(notmuch search $os_x_darwin_thread | notmuch_search_sanitize) test_expect_equal $output thread:XXX 2009-11-18 [4/4] Jjgod Jiang, Alexander Botero-Lowry; [notmuch] Mac OS X/Darwin compatibility issues (inbox unread) test_begin_subtest Tag all matching messages from search view -test_subtest_known_broken notmuch tag +test-tag-race from:cworth test_emacs (notmuch-search \tag:test-tag-race\) (notmuch-test-wait) @@ -135,7 +134,6 @@ notmuch tag -test-tag-race '*' notmuch tag -test-tag-race-2 '*' test_begin_subtest Change tags from search view: another message arriving after thread lookup -test_subtest_known_broken typsos_id=878we4qdqf@yoom.home.cworth.org typsos_thread=$(notmuch search --output=threads id:$typsos_id) test_emacs (notmuch-search \$typsos_thread\) -- 1.7.9.1 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 0/3] Outline fix for emacs tagging race
On Fri, Nov 09 2012, Mark Walters markwalters1...@gmail.com wrote: This patch series implements the above mentioned solution and seems to work except for one problem. Since emacs now tags each message in a thread in the search buffer it is easy to ask it to tag a lot of messages. This could be done individually which would be ridiculously slow so instead they are all done in one batch. But now it is relatively easy to take notmuch over the threshold for ARG_MAX. In [3] Tomi did some experiments and found on a standard Debian system with getconf ARG_MAX =131072 that command lines with 1 200 byte arguments worked. I am a little puzzled by that as I get the same results and I getconf ARG_MAX gives 2097152 for me. More importantly though, when trying to execute a command from emacs I am finding that 131072 is the limit on the command length in bytes and we can hit this with something around 1500 messages (see end for a very hacky emacs test script). This is probably more than we can expect in a single thread so tagging from show is probably safe but it does cause a problem when tagging from search. That's because your test script below constructs one very lng string (and used just one arg). Like you said, [3] used 1 200-char args. This test script is more like what the perl(1) script did in [3] (progn (setq arglist '(aaa)) (setq n 1) (while ( n 0) (setq n (1- n)) (setq arglist (cons bbb arglist))) (apply 'call-process /bin/echo nil t nil arglist)) (except that the args here are just 80 chars but you should get the point :) I did not check your patches yet.. buf if you concatenate there you could try to provide those as separate args... ... then we can experiment how notmuch (and xapian) copes with thosew IIRC notmuch will concatenate one long string out of those and do xapian stuff using that (but then we're not limited by command line arguments restrictions). Best wishes Mark Tomi [3] id:m2liody7av@guru.guru-group.fi TEST SCRIPT (progn (setq bigstring ) (setq n 1310) (setq big nil) (while ( n 0) (setq n (1- n)) (setq big (concat big (format %s n) bigstring))) (call-process echo nil t nil big)) ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 0/3] Outline fix for emacs tagging race
Mark Walters markwalters1...@gmail.com writes: I can think of several possible solutions (e.g., batch it in my new stuff, put some batching in notmuch-tag, all notmuch tag to read a query from stdin). But before any larger more intrusive change do people like the general approach? Does anyone have a good way to get round the command line size problem? Jani posted some patches to do batch tagging from stdin. I guess the last full series posted was at id:cover.1334404979.git.j...@nikula.org I believe Jani has slightly updated (or at least rebased) the patches in git. I reworked and re-posted some of the foundational parts of the series at id:1345382314-5330-1-git-send-email-da...@tethera.net I want to have some of this infrastucture for a new style of backup/restore, and it's possible that batch tagging would also speed up nmbug. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 0/3] Outline fix for emacs tagging race
Quoth Mark Walters on Nov 09 at 6:58 pm: For a long time [1] there have been two related races in tagging from the search buffer. The first is that when tagging (including archiving) a thread message which arrived after the buffer was created may also be tagged. This is because the tagging is done based on the thread-id not on the individual messages. The second is when using the '*' command to tag all messages. This is not quite the same as this command only tags messages matching the query not all messages in all threads that contain a message matching the query. Thus if more messages now match than when the buffer was created (eg some external tagging script has run) then this command can unexpectedly tag these messages too. One solution that was discussed in [2] was for the search output of notmuch to include the message-ids of both matching and non-matching messages. At that time that was difficult to implement as it was unclear how to escape the message ids when using the text format. Since emacs now uses JSON for search mode this problem is solved. This patch series implements the above mentioned solution and seems to work except for one problem. Since emacs now tags each message in a thread in the search buffer it is easy to ask it to tag a lot of messages. This could be done individually which would be ridiculously slow so instead they are all done in one batch. But now it is relatively easy to take notmuch over the threshold for ARG_MAX. In [3] Tomi did some experiments and found on a standard Debian system with getconf ARG_MAX =131072 that command lines with 1 200 byte arguments worked. I am a little puzzled by that as I get the same results and I getconf ARG_MAX gives 2097152 for me. More importantly though, when trying to execute a command from emacs I am finding that 131072 is the limit on the command length in bytes and we can hit this with something around 1500 messages (see end for a very hacky emacs test script). This is probably more than we can expect in a single thread so tagging from show is probably safe but it does cause a problem when tagging from search. I can think of several possible solutions (e.g., batch it in my new stuff, put some batching in notmuch-tag, all notmuch tag to read a query from stdin). But before any larger more intrusive change do people like the general approach? Does anyone have a good way to get round the command line size problem? Best wishes Mark [1] id:87ocmtg9ni@yoom.home.cworth.org [2] id:CAH-f9WticM4EN8F1_ik_-mcBcBtrXwSpO+Drbtp7=un7mce...@mail.gmail.com [3] id:m2liody7av@guru.guru-group.fi I'm glad to see someone picking up this bug. Besides somehow dealing with long command-lines, when we were last exploring this race, I had found that it was 3-4x more efficient to use Xapian document IDs directly rather than message IDs [1]. It's probably best *not* to do this initially for the sake of simplicity, but I think a simple tweak to your approach would let us seamlessly transition to this in the future. Rather than extending the JSON output with what are explicitly message IDs, instead extend the output with opaque queries that are guaranteed to match the matching/non-matching messages in the thread and are guaranteed to be combinable, but aren't guaranteed to be of any particular form. For now, the CLI can simply output id: queries for these, but in the future we could easily add a special query syntax for docid queries or, if we ever move to a custom query parser, support docids in any query. For the long command line problem, one easy solution is to support, say, '-' as a query syntax that means read the query from stdin. This would be a simple addition to query_string_from_args and would work across the CLI. [1] id:CAH-f9WsPj=1Eu=g3soepjgctbfs6hrldlq18xmenj8az00y...@mail.gmail.com ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
