[PATCH] Place extra_cflags before CONFIGURE_CFLAGS
Moritz Wilhelmywrites: > From: Moritz Wilhelmy > > This ensures that the build will not attempt to use an existing notmuch.h when > an older version of notmuch is already installed elsewhere (e.g. in > /usr/local) > and /usr/local/include is added to CONFIGURE_CFLAGS by one of the libraries > (talloc, in my case) pushed, d
[PATCH] perf: Clean corpus and caches in distclean, not clean
Austin Clements writes: > Previously, we cleaned the downloaded performance corpus and the > cached indexes on 'make clean'. This seems heavy-handed, since these > take a long time to download, unpack, and index. They also aren't > make targets to begin with. Move cleaning these to 'make distclean'. > This isn't exactly the right meaning of "distclean", but it's closer. pushed, d
[PATCH v2] new: Don't scan unchanged directories with no sub-directories
Austin Clements writes: > This can substantially reduce the cost of notmuch new in some > situations, such as when the file system cache is cold or when the > Maildir is on NFS. On my desktop at home (a core i7 950) with spinning rust disks (and lvm on luks) this patch yields about a 7% slowdown in the intial new perf test from Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) Initial notmuch new 579.60 348.86 14.26 217188 5330266/3501272 to Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) Initial notmuch new 620.51 368.62 15.48 217156 5330354/3416456 On an SSD I don't detect a significant different (<0.5% speedup) d
[PATCH v2] new: Don't scan unchanged directories with no sub-directories
On Fri, Oct 25 2013, Austin Clements wrote: > This can substantially reduce the cost of notmuch new in some > situations, such as when the file system cache is cold or when the > Maildir is on NFS. > --- LGTM. The creation and destruction of child directories happens only if there are symlinks to directories in otherwise leaf directories. Tomi > > This should fix the problem with directories containing symlinks to > other directories, but no actual sub-directories. > > notmuch-new.c | 29 + > 1 file changed, 29 insertions(+) > > diff --git a/notmuch-new.c b/notmuch-new.c > index faa33f1..ba05cb4 100644 > --- a/notmuch-new.c > +++ b/notmuch-new.c > @@ -323,6 +323,35 @@ add_files (notmuch_database_t *notmuch, > } > db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0; > > +/* If the directory is unchanged from our last scan and has no > + * sub-directories, then return without scanning it at all. In > + * some situations, skipping the scan can substantially reduce the > + * cost of notmuch new, especially since the huge numbers of files > + * in Maildirs make scans expensive, but all files live in leaf > + * directories. > + * > + * To check for sub-directories, we borrow a trick from find, > + * kpathsea, and many other UNIX tools: since a directory's link > + * count is the number of sub-directories (specifically, their > + * '..' entries) plus 2 (the link from the parent and the link for > + * '.'). This check is safe even on weird file systems, since > + * file systems that can't compute this will return 0 or 1. This > + * is safe even on *really* weird file systems like HFS+ that > + * mistakenly return the total number of directory entries, since > + * that only inflates the count beyond 2. > + */ > +if (directory && fs_mtime == db_mtime && st.st_nlink == 2) { > + /* There's one catch: pass 1 below considers symlinks to > + * directories to be directories, but these don't increase the > + * file system link count. So, only bail early if the > + * database agrees that there are no sub-directories. */ > + db_subdirs = notmuch_directory_get_child_directories (directory); > + if (!notmuch_filenames_valid (db_subdirs)) > + goto DONE; > + notmuch_filenames_destroy (db_subdirs); > + db_subdirs = NULL; > +} > + > /* If the database knows about this directory, then we sort based > * on strcmp to match the database sorting. Otherwise, we can do > * inode-based sorting for faster filesystem operation. */ > -- > 1.8.4.rc3 > > ___ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH v2] new: Don't scan unchanged directories with no sub-directories
Thank you both for your help! -- Vlad
Build broken on OS X 10.9
Looks like notmuch 0.16 won't build on OS X 10.9 due to the switch to libc++. I don't know where to start with fixing this, but thought I'd let y'all know.
[PATCH] emacs: Add a space after completed tag operations
On Fri, Oct 25 2013, Austin Clements wrote: > Previously, when a user fully completed a tag operation, they had to > press space to begin entering another tag operation. This is > different from, say, shell file name completion, which typically > inserts a space after an unambiguous completion under the assumption > that the user will want to enter more input. > > This patch tweaks `notmuch-read-tag-changes' to act more like shell > file name completion: after an unambiguous tag completion, it now > inserts a space, ready and waiting for another tagging operation from > the user. This is backwards-compatible with old habits, since there's > no harm in putting an extra space. > --- LGTM. Tested by hand-editing "(mapcar ...)" part to the code (re-evaluated defun in *scratch* buffer) and it worked fine. Tomi > > Just because of the context, this must be applied on top of the > currently pending tag completion series > id:"1382487721-31776-1-git-send-email-amdragon at mit.edu". The change > itself should be otherwise independent. > > emacs/notmuch-tag.el | 12 ++-- > 1 file changed, 10 insertions(+), 2 deletions(-) > > diff --git a/emacs/notmuch-tag.el b/emacs/notmuch-tag.el > index 7b21006..8b921f1 100644 > --- a/emacs/notmuch-tag.el > +++ b/emacs/notmuch-tag.el > @@ -229,8 +229,16 @@ initial input in the minibuffer." > (set-keymap-parent map crm-local-completion-map) > (define-key map " " 'self-insert-command) > map))) > -(delete "" (completing-read-multiple prompt > - tag-list nil nil initial-input > +(delete "" (completing-read-multiple > + prompt > + ;; Append the separator to each completion so when the > + ;; user completes a tag they can immediately begin > + ;; entering another. `completing-read-multiple' > + ;; ultimately splits the input on crm-separator, so we > + ;; don't need to strip this back off (we just need to > + ;; delete "empty" entries caused by trailing spaces). > + (mapcar (lambda (tag-op) (concat tag-op crm-separator)) > tag-list) > + nil nil initial-input > 'notmuch-read-tag-changes-history > > (defun notmuch-update-tags (tags tag-changes) > -- > 1.8.4.rc3 > > ___ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] emacs: Add a space after completed tag operations
Tested and LGTM +1 Best wishes Mark On Thu, 24 Oct 2013, Austin Clements wrote: > Previously, when a user fully completed a tag operation, they had to > press space to begin entering another tag operation. This is > different from, say, shell file name completion, which typically > inserts a space after an unambiguous completion under the assumption > that the user will want to enter more input. > > This patch tweaks `notmuch-read-tag-changes' to act more like shell > file name completion: after an unambiguous tag completion, it now > inserts a space, ready and waiting for another tagging operation from > the user. This is backwards-compatible with old habits, since there's > no harm in putting an extra space. > --- > > Just because of the context, this must be applied on top of the > currently pending tag completion series > id:"1382487721-31776-1-git-send-email-amdragon at mit.edu". The change > itself should be otherwise independent. > > emacs/notmuch-tag.el | 12 ++-- > 1 file changed, 10 insertions(+), 2 deletions(-) > > diff --git a/emacs/notmuch-tag.el b/emacs/notmuch-tag.el > index 7b21006..8b921f1 100644 > --- a/emacs/notmuch-tag.el > +++ b/emacs/notmuch-tag.el > @@ -229,8 +229,16 @@ initial input in the minibuffer." > (set-keymap-parent map crm-local-completion-map) > (define-key map " " 'self-insert-command) > map))) > -(delete "" (completing-read-multiple prompt > - tag-list nil nil initial-input > +(delete "" (completing-read-multiple > + prompt > + ;; Append the separator to each completion so when the > + ;; user completes a tag they can immediately begin > + ;; entering another. `completing-read-multiple' > + ;; ultimately splits the input on crm-separator, so we > + ;; don't need to strip this back off (we just need to > + ;; delete "empty" entries caused by trailing spaces). > + (mapcar (lambda (tag-op) (concat tag-op crm-separator)) > tag-list) > + nil nil initial-input > 'notmuch-read-tag-changes-history > > (defun notmuch-update-tags (tags tag-changes) > -- > 1.8.4.rc3 > > ___ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: Add a space after completed tag operations
On Fri, Oct 25 2013, Austin Clements amdra...@mit.edu wrote: Previously, when a user fully completed a tag operation, they had to press space to begin entering another tag operation. This is different from, say, shell file name completion, which typically inserts a space after an unambiguous completion under the assumption that the user will want to enter more input. This patch tweaks `notmuch-read-tag-changes' to act more like shell file name completion: after an unambiguous tag completion, it now inserts a space, ready and waiting for another tagging operation from the user. This is backwards-compatible with old habits, since there's no harm in putting an extra space. --- LGTM. Tested by hand-editing (mapcar ...) part to the code (re-evaluated defun in *scratch* buffer) and it worked fine. Tomi Just because of the context, this must be applied on top of the currently pending tag completion series id:1382487721-31776-1-git-send-email-amdra...@mit.edu. The change itself should be otherwise independent. emacs/notmuch-tag.el | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/emacs/notmuch-tag.el b/emacs/notmuch-tag.el index 7b21006..8b921f1 100644 --- a/emacs/notmuch-tag.el +++ b/emacs/notmuch-tag.el @@ -229,8 +229,16 @@ initial input in the minibuffer. (set-keymap-parent map crm-local-completion-map) (define-key map 'self-insert-command) map))) -(delete (completing-read-multiple prompt - tag-list nil nil initial-input +(delete (completing-read-multiple + prompt + ;; Append the separator to each completion so when the + ;; user completes a tag they can immediately begin + ;; entering another. `completing-read-multiple' + ;; ultimately splits the input on crm-separator, so we + ;; don't need to strip this back off (we just need to + ;; delete empty entries caused by trailing spaces). + (mapcar (lambda (tag-op) (concat tag-op crm-separator)) tag-list) + nil nil initial-input 'notmuch-read-tag-changes-history (defun notmuch-update-tags (tags tag-changes) -- 1.8.4.rc3 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] emacs: Add a space after completed tag operations
Tested and LGTM +1 Best wishes Mark On Thu, 24 Oct 2013, Austin Clements amdra...@mit.edu wrote: Previously, when a user fully completed a tag operation, they had to press space to begin entering another tag operation. This is different from, say, shell file name completion, which typically inserts a space after an unambiguous completion under the assumption that the user will want to enter more input. This patch tweaks `notmuch-read-tag-changes' to act more like shell file name completion: after an unambiguous tag completion, it now inserts a space, ready and waiting for another tagging operation from the user. This is backwards-compatible with old habits, since there's no harm in putting an extra space. --- Just because of the context, this must be applied on top of the currently pending tag completion series id:1382487721-31776-1-git-send-email-amdra...@mit.edu. The change itself should be otherwise independent. emacs/notmuch-tag.el | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/emacs/notmuch-tag.el b/emacs/notmuch-tag.el index 7b21006..8b921f1 100644 --- a/emacs/notmuch-tag.el +++ b/emacs/notmuch-tag.el @@ -229,8 +229,16 @@ initial input in the minibuffer. (set-keymap-parent map crm-local-completion-map) (define-key map 'self-insert-command) map))) -(delete (completing-read-multiple prompt - tag-list nil nil initial-input +(delete (completing-read-multiple + prompt + ;; Append the separator to each completion so when the + ;; user completes a tag they can immediately begin + ;; entering another. `completing-read-multiple' + ;; ultimately splits the input on crm-separator, so we + ;; don't need to strip this back off (we just need to + ;; delete empty entries caused by trailing spaces). + (mapcar (lambda (tag-op) (concat tag-op crm-separator)) tag-list) + nil nil initial-input 'notmuch-read-tag-changes-history (defun notmuch-update-tags (tags tag-changes) -- 1.8.4.rc3 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2] new: Don't scan unchanged directories with no sub-directories
On Fri, Oct 25 2013, Austin Clements amdra...@mit.edu wrote: This can substantially reduce the cost of notmuch new in some situations, such as when the file system cache is cold or when the Maildir is on NFS. --- LGTM. The creation and destruction of child directories happens only if there are symlinks to directories in otherwise leaf directories. Tomi This should fix the problem with directories containing symlinks to other directories, but no actual sub-directories. notmuch-new.c | 29 + 1 file changed, 29 insertions(+) diff --git a/notmuch-new.c b/notmuch-new.c index faa33f1..ba05cb4 100644 --- a/notmuch-new.c +++ b/notmuch-new.c @@ -323,6 +323,35 @@ add_files (notmuch_database_t *notmuch, } db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0; +/* If the directory is unchanged from our last scan and has no + * sub-directories, then return without scanning it at all. In + * some situations, skipping the scan can substantially reduce the + * cost of notmuch new, especially since the huge numbers of files + * in Maildirs make scans expensive, but all files live in leaf + * directories. + * + * To check for sub-directories, we borrow a trick from find, + * kpathsea, and many other UNIX tools: since a directory's link + * count is the number of sub-directories (specifically, their + * '..' entries) plus 2 (the link from the parent and the link for + * '.'). This check is safe even on weird file systems, since + * file systems that can't compute this will return 0 or 1. This + * is safe even on *really* weird file systems like HFS+ that + * mistakenly return the total number of directory entries, since + * that only inflates the count beyond 2. + */ +if (directory fs_mtime == db_mtime st.st_nlink == 2) { + /* There's one catch: pass 1 below considers symlinks to + * directories to be directories, but these don't increase the + * file system link count. So, only bail early if the + * database agrees that there are no sub-directories. */ + db_subdirs = notmuch_directory_get_child_directories (directory); + if (!notmuch_filenames_valid (db_subdirs)) + goto DONE; + notmuch_filenames_destroy (db_subdirs); + db_subdirs = NULL; +} + /* If the database knows about this directory, then we sort based * on strcmp to match the database sorting. Otherwise, we can do * inode-based sorting for faster filesystem operation. */ -- 1.8.4.rc3 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Re: [PATCH v2] new: Don't scan unchanged directories with no sub-directories
Thank you both for your help! -- Vlad ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Build broken on OS X 10.9
Looks like notmuch 0.16 won't build on OS X 10.9 due to the switch to libc++. I don't know where to start with fixing this, but thought I'd let y'all know. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2 00/11] Fix search tagging races
On Thu, 24 Oct 2013, Austin Clements amdra...@mit.edu wrote: This is v2 of id:1381185201-25197-1-git-send-email-amdra...@mit.edu. It fixes several comments from Mark and Jani. This has been rebased on top of the tag completion changes, so doing * from a large search buffer will no longer crash. Hence, this series depends on the (currently pending) series in id:1382487721-31776-1-git-send-email-amdra...@mit.edu. This looks good to me +1 Mark This version does not address what happens when you * on a search buffer that's still filling. With this series, it will apply to all messages that have appeared when the user finishes entering tag changes. This isn't ideal, but this seems pretty obscure and I'm not sure what the right answer is, so I'm punting it to the future. Another thing that may be worth changing in a follow-up is what messages * applies to. Currently, * applies *only* to matched messages, not all threads in the search, which I think was an accident of implementation. This series retains that behavior, but opens up the possibility of applying to all threads instead. I think that would be much more consistent and less surprising behavior. An approximate diff from v1 is below. This diff is prior to rebasing, since the post-rebase diff is not useful. diff --git a/emacs/notmuch-tag.el b/emacs/notmuch-tag.el index a4eec14..36937fb 100644 --- a/emacs/notmuch-tag.el +++ b/emacs/notmuch-tag.el @@ -242,7 +242,11 @@ from TAGS if present. (error Changed tag must be of the form `+this_tag' or `-that_tag') (sort result-tags 'string))) -(defconst notmuch-tag-argument-limit 1000) +(defconst notmuch-tag-argument-limit 1000 + Use batch tagging if the tagging query is longer than this. + +This limits the length of arguments passed to the notmuch CLI to +avoid system argument length limits and performance problems.) (defun notmuch-tag (query optional tag-changes) Add/remove tags in TAG-CHANGES to messages matching QUERY. @@ -276,7 +280,6 @@ notmuch-after-tag-hook will be run. ;; Use batch tag mode to avoid argument length limitations (let ((batch-op (concat (mapconcat #'notmuch-hex-encode tag-changes ) -- query))) - (message Batch tagging with %s batch-op) (notmuch-call-notmuch-process :stdin-string batch-op tag --batch))) (run-hooks 'notmuch-after-tag-hook)) ;; in all cases we return tag-changes as a list diff --git a/notmuch-client.h b/notmuch-client.h index 1b14910..8bc1a2a 100644 --- a/notmuch-client.h +++ b/notmuch-client.h @@ -144,7 +144,8 @@ chomp_newline (char *str) #define NOTMUCH_FORMAT_MIN 1 /* The minimum non-deprecated structured output format version. * Requests for format versions below this will print a stern warning. - * Must be = NOTMUCH_FORMAT_MIN and NOTMUCH_FORMAT_CUR. + * Must be between NOTMUCH_FORMAT_MIN and NOTMUCH_FORMAT_CUR, + * inclusive. */ #define NOTMUCH_FORMAT_MIN_ACTIVE 1 diff --git a/notmuch-search.c b/notmuch-search.c index 1d14651..7c973b3 100644 --- a/notmuch-search.c +++ b/notmuch-search.c @@ -53,13 +53,13 @@ sanitize_string (const void *ctx, const char *str) * NULL. */ static int get_thread_query (notmuch_thread_t *thread, - char **matched_out, char **unmached_out) + char **matched_out, char **unmatched_out) { notmuch_messages_t *messages; char *escaped = NULL; size_t escaped_len = 0; -*matched_out = *unmached_out = NULL; +*matched_out = *unmatched_out = NULL; for (messages = notmuch_thread_get_messages (thread); notmuch_messages_valid (messages); @@ -69,17 +69,16 @@ get_thread_query (notmuch_thread_t *thread, const char *mid = notmuch_message_get_message_id (message); /* Determine which query buffer to extend */ char **buf = notmuch_message_get_flag ( - message, NOTMUCH_MESSAGE_FLAG_MATCH) ? matched_out : unmached_out; - /* Allocate the query buffer is this is the first message */ - if (!*buf (*buf = talloc_strdup (thread, )) == NULL) - return -1; + message, NOTMUCH_MESSAGE_FLAG_MATCH) ? matched_out : unmatched_out; /* Add this message's id: query. Since id is an exclusive * prefix, it is implicitly 'or'd together, so we only need to * join queries with a space. */ if (make_boolean_term (thread, id, mid, escaped, escaped_len) 0) return -1; - *buf = talloc_asprintf_append_buffer ( - *buf, %s%s, **buf ? : , escaped); + if (*buf) + *buf = talloc_asprintf_append_buffer (*buf, %s, escaped); + else + *buf = talloc_strdup (thread, escaped); if (!*buf) return -1; } ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2] new: Don't scan unchanged directories with no sub-directories
Austin Clements amdra...@mit.edu writes: This can substantially reduce the cost of notmuch new in some situations, such as when the file system cache is cold or when the Maildir is on NFS. On my desktop at home (a core i7 950) with spinning rust disks (and lvm on luks) this patch yields about a 7% slowdown in the intial new perf test from Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) Initial notmuch new 579.60 348.86 14.26 217188 5330266/3501272 to Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) Initial notmuch new 620.51 368.62 15.48 217156 5330354/3416456 On an SSD I don't detect a significant different (0.5% speedup) d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v2 0/8] Improve tag change completion
Austin Clements amdra...@mit.edu writes: This is v2 of id:1382471457-26056-1-git-send-email-amdra...@mit.edu. It improves some documentation strings, fixes one bug, and elimintes some redundant code. The diff from v1 is below. pushed, d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] perf: Clean corpus and caches in distclean, not clean
Austin Clements amdra...@mit.edu writes: Previously, we cleaned the downloaded performance corpus and the cached indexes on 'make clean'. This seems heavy-handed, since these take a long time to download, unpack, and index. They also aren't make targets to begin with. Move cleaning these to 'make distclean'. This isn't exactly the right meaning of distclean, but it's closer. pushed, d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch