Re: [EMACS] feature proposal (with code): notmuch-rainbow-tags.el

Hi Alexander, Alexander Adolf writes: > > That was the other option. I thought I'd post it here first, to see > whether there would be any momentum for including it in notmuch > itself? That actually sounds like a good idea. Maybe just send it as a patch to the mailinglist? Not sure if there

[PATCH 11/31] util/unicode: allow calling from C++

The omission of the 'extern "C"' machinery seems like an oversight. --- util/unicode-util.h | 7 +++ 1 file changed, 7 insertions(+) diff --git a/util/unicode-util.h b/util/unicode-util.h index 32d1e6ef..1bb9336a 100644 --- a/util/unicode-util.h +++ b/util/unicode-util.h @@ -4,9 +4,16 @@

[PATCH 02/31] lib: split notmuch_query_create

Most of the function will be re-usable when creating a query from an s-expression. --- lib/query.cc | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/lib/query.cc b/lib/query.cc index 792aba21..39b85e91 100644 --- a/lib/query.cc +++ b/lib/query.cc @@ -84,9

[PATCH 06/31] lib/parse-sexp: parse single terms and the empty list.

There is not much of a parser here yet, but it already does some useful error reporting. Most functionality sketched in the documentation is not implemented yet; detailed documentation will follow with the implementation. --- doc/conf.py | 4 ++ doc/index.rst

[PATCH 31/31] lib/parse-sexp: apply macros

Macros implement lazy evaluation and lexical scope. The former is needed to make certain natural constructs work sensibly (e.g. (tag ,param)) but the latter is mainly future-proofing in case the DSL is is extended to allow local bindings. For technical background, see chapters 6 and 17 of [1]

[PATCH 08/31] lib/parse-sexp: stem unquoted atoms

This is somewhat less DWIM than the Xapian query parser, but it has the advantage of simplicity. --- doc/man7/notmuch-sexp-queries.rst | 10 -- lib/parse-sexp.cc | 10 +++--- test/T081-sexpr-search.sh | 5 - 3 files changed, 19 insertions(+), 6

[PATCH 05/31] lib: add new status code for query syntax errors.

This will help provide more meaningful error messages without special casing on the client side. --- bindings/python-cffi/notmuch2/_build.py | 1 + bindings/python-cffi/notmuch2/_errors.py | 3 +++ lib/database.cc | 2 ++ lib/notmuch.h| 4

[PATCH 20/31] lib: generate actual Xapian query for "*" and ""

The previous code had the somewhat bizarre effect that the (notmuch specific) query string was "*" (interpreted as MatchAll) and the allegedly parsed xapian_query was "MatchNothing". This commit also reduces code duplication. --- lib/query.cc | 34 ++ 1 file

[PATCH 24/31] lib/parse-sexp: support infix subqueries

This is necessary so that programs can take infix syntax queries from a user and use the sexp query syntax to construct e.g. a refinement of that query. --- doc/man7/notmuch-sexp-queries.rst | 7 + lib/parse-sexp.cc | 34 test/T081-sexpr-search.sh

[PATCH 09/31] lib/parse-sexp: support and, not, and or.

All operations and (Xapian) fields will eventually have an entry in the prefixes table. The flags field is just a placeholder for now, but will eventually distinguish between various kinds of prefixes. --- doc/man7/notmuch-sexp-queries.rst | 16 --- lib/parse-sexp.cc | 76

[PATCH 16/31] lib/parse-sexp: handle unprefixed terms.

This is equivalent to adding the same field name "" for multiple prefixes in the Xapian query parser, but we have to explicitely construct the resulting query. --- lib/parse-sexp.cc | 36 test/T081-sexpr-search.sh | 31 +++

[PATCH 12/31] lib/parse-sexp: support phrase queries.

Anything that is quoted or not purely word characters is considered a phrase. Phrases are not stemmed, because the stems do not have positional information in the database. It is less efficient to scan the term twice, but it avoids a second pass to add prefixes, so maybe it balances out. In any

[PATCH 29/31] lib/parse-sexp: support saved s-expression queries

It turns out there is not really much code in query-fp.cc useful for supporting the new syntax. The code we could potentially factor out amounts to calling notmuch_database_get_config; both the key construction and the parsing of the results are specific to the query syntax involved. ---

[PATCH 14/31] lib/parse-sexp: 'starts-with' wildcard searches

The many tests potentially overkill, but they could catch typos in the prefixes table. As a simplifying assumption, for now we assume a single argument to the wildcard operator, as this matches the Xapian semantics. The name 'starts-with' is chosen to emphasize the supported case of wildcards in

[PATCH 01/31] configure: optional library sfsexp

The configure part is essentially the same as the other checks using pkg-config. Since the optional inclusion of this feature changes what options are available to the user, include it in the "built_with" pseudo-configuration keys. --- configure| 26 +-

[PATCH 28/31] CLI/config support saving s-expression queries

This commit does not enable using saved s-expression queries, only saving and retrieving them from the config file or the database. Use in queries will be enabled in a following commit. --- doc/man1/notmuch-config.rst | 5 + notmuch-config.c| 1 + test/T081-sexpr-search.sh |

[PATCH 23/31] lib/parse-sexp: expand queries

The code here is just gluing together _notmuch_query_expand with the existing sexp parser infrastructure. --- doc/man7/notmuch-sexp-queries.rst | 20 +++ lib/parse-sexp.cc | 56 +-- test/T081-sexpr-search.sh | 52

[PATCH 21/31] lib/query: factor out _notmuch_query_string_to_xapian_query

When dealing with recursive queries (i.e. thread:{foo}) it turns out to be useful just to deal with the underlying Xapian objects, and not wrap them in notmuch objects. --- lib/database-private.h | 7 ++ lib/query.cc | 51 -- 2 files changed,

[PATCH 27/31] lib/parse-sexp: handle saved queries

This provides functionality analogous to query: in the Xapian QueryParser based parser. Perhaps counterintuitively, the saved queries currently have to be in the original query syntax (i.e. not s-expressions). --- doc/man7/notmuch-sexp-queries.rst | 6 ++ lib/parse-sexp.cc |

[PATCH 07/31] lib: leave stemmer object accessible

This enables using the same stemmer in both query parsers. --- lib/database-private.h | 1 + lib/open.cc| 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/database-private.h b/lib/database-private.h index f206efaf..85d55299 100644 --- a/lib/database-private.h

[PATCH 30/31] lib/parse-sexp: thread environment argument through parser

No functionality change, just an extra argument carried everywhere. --- lib/parse-sexp.cc | 49 +-- 1 file changed, 30 insertions(+), 19 deletions(-) diff --git a/lib/parse-sexp.cc b/lib/parse-sexp.cc index 53168d57..188acffc 100644 ---

[PATCH 19/31] lib/parse-sexp: support regular expressions

At least to the degree that the Xapian QueryParser based parser also supports them. Support short alias 'rx' as it seems to make more complex queries nicer to read. --- doc/man7/notmuch-sexp-queries.rst | 8 lib/parse-sexp.cc | 54 ++-

[PATCH 26/31] lib: factor out expansion of saved queries.

This is intended to allow use outside of the Xapian query parser. --- lib/database-private.h | 5 + lib/query-fp.cc| 22 +++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/lib/database-private.h b/lib/database-private.h index 9ee3b933..8b9d67fe

[PATCH 22/31] lib/thread-fp: factor out query expansion, rewrite in Xapian

It will be convenient not to have to construct a notmuch query object when parsing subqueries, so the commit rewrites the query expansion (currently only used for thread:{} queries) using only Xapian. As a bonus it seems about 15% faster in initial experiments. --- lib/database-private.h | 16

[PATCH 04/31] CLI/search+address: support sexpr queries

Initially support selection of query syntax in two subcommands to enable testing. --- notmuch-search.c | 13 + test/T080-search.sh | 5 + test/T095-address.sh | 5 + 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/notmuch-search.c b/notmuch-search.c

[PATCH 13/31] lib/parse-sexp: add term prefix backed fields

We use "boolean" to describe fields that should generate terms literally without stemming or phrase splitting. This terminology might not be ideal but it is already enshrined in notmuch-search-terms(7). --- doc/man7/notmuch-sexp-queries.rst | 18 +- lib/parse-sexp.cc | 49

[PATCH 03/31] lib: define notmuch_query_create_with_syntax

Set the parsing syntax when the (notmuch) query object is created. Initially the library always returns a trivial query that matches all messages when using s-expression syntax. It seems better to select the syntax at query creation time because the lazy parsing is an implementation detail. ---

[PATCH 10/31] lib/parse-sexp: support subject field

The broken tests are because we do not yet handle phrase searches. --- doc/man7/notmuch-sexp-queries.rst | 62 +-- lib/parse-sexp.cc | 19 +- test/T081-sexpr-search.sh | 57 3 files changed, 133

[PATCH 18/31] lib: factor out query construction from regexp

This will allow re-use of this code outside of the Xapian query parser. --- lib/database-private.h | 5 +++ lib/regexp-fields.cc | 81 +- lib/regexp-fields.h| 6 3 files changed, 68 insertions(+), 24 deletions(-) diff --git

[PATCH 25/31] lib/parse-sexp: parse user headers

One subtle aspect is the replacement of _find_prefix with _notmuch_database_prefix, which understands user headers. Otherwise the code mainly consists of creating a fake prefix record (since the user prefixes are not in the prefix table) and error handling. --- doc/man7/notmuch-sexp-queries.rst |

[PATCH 17/31] lib/query: generalize exclude handling to s-expression queries

In fact most of the code path is in common, only the caching of terms in the query needs to be added for s-expression queries. --- lib/query.cc | 34 ++--- test/T081-sexpr-search.sh | 40 +++ 2 files changed, 63

[PATCH 15/31] lib/parse-sexp: add '*' as syntactic sugar for '(starts-with "")'

Users that insist on using a literal '*' as a tag, can continue to do so by quoting it when searching. --- doc/man7/notmuch-sexp-queries.rst | 19 ++-- lib/parse-sexp.cc | 5 test/T081-sexpr-search.sh | 48 +++ 3 files changed, 70

v4 sexp query parser

Two main user visible changes since v3. 1) I changed the option from --query-syntax=sexp to --query=sexp This is arguably a bit more ambiguous (are we specifying a query "sexp" to search for?) but considerably less painful to type. I previously mentioned `--squery`, but it is a