On Mon, 27 Oct 2014, Michal Sojka <sojkam1 at fel.cvut.cz> wrote: > This adds an algorithm to filter out duplicate addresses from address > outputs (sender, receivers). The algorithm can be configured with > --filter-by command line option. > > The code here is an extended version of a patch from Jani Nikula.
Hi As this is getting into the more controversial bike shedding region I wonder if it would be worth splitting this into 2 patches: the first could do the default dedupe based on name/address and the second could do add the filter-by options. I think the default deduping is obviously worth doing but I am not sure about the rest. In any case I think the default deduping could go in pre-freeze but I would recommend the rest is left until after. > --- > completion/notmuch-completion.bash | 6 ++- > completion/notmuch-completion.zsh | 3 +- > doc/man1/notmuch-search.rst | 38 +++++++++++++++ > notmuch-search.c | 98 > +++++++++++++++++++++++++++++++++++--- > test/T090-search-output.sh | 87 +++++++++++++++++++++++++++++++++ > test/T095-search-filter-by.sh | 64 +++++++++++++++++++++++++ > 6 files changed, 288 insertions(+), 8 deletions(-) > create mode 100755 test/T095-search-filter-by.sh > > diff --git a/completion/notmuch-completion.bash > b/completion/notmuch-completion.bash > index cfbd389..6b6d43a 100644 > --- a/completion/notmuch-completion.bash > +++ b/completion/notmuch-completion.bash > @@ -305,12 +305,16 @@ _notmuch_search() > COMPREPLY=( $( compgen -W "true false flag all" -- "${cur}" ) ) > return > ;; > + --filter-by) > + COMPREPLY=( $( compgen -W "nameaddr name addr addrfold > nameaddrfold" -- "${cur}" ) ) > + return > + ;; > esac > > ! $split && > case "${cur}" in > -*) > - local options="--format= --output= --sort= --offset= --limit= > --exclude= --duplicate=" > + local options="--format= --output= --sort= --offset= --limit= > --exclude= --duplicate= --filter-by=" > compopt -o nospace > COMPREPLY=( $(compgen -W "$options" -- ${cur}) ) > ;; > diff --git a/completion/notmuch-completion.zsh > b/completion/notmuch-completion.zsh > index 3e52a00..3e535df 100644 > --- a/completion/notmuch-completion.zsh > +++ b/completion/notmuch-completion.zsh > @@ -53,7 +53,8 @@ _notmuch_search() > '--max-threads=[display only the first x threads from the search > results]:number of threads to show: ' \ > '--first=[omit the first x threads from the search results]:number of > threads to omit: ' \ > '--sort=[sort results]:sorting:((newest-first\:"reverse chronological > order" oldest-first\:"chronological order"))' \ > - '--output=[select what to output]:output:((summary threads messages > files tags sender recipients))' > + '--output=[select what to output]:output:((summary threads messages > files tags sender recipients))' \ > + '--filter-by=[filter out duplicate > addresses]:filter-by:((nameaddr\:"both name and address part" name\:"name > part" addr\:"address part" addrfold\:"case-insensitive address part" > nameaddrfold\:"name and case-insensitive address part"))' > } > > _notmuch() > diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst > index b6607c9..84af2da 100644 > --- a/doc/man1/notmuch-search.rst > +++ b/doc/man1/notmuch-search.rst > @@ -85,6 +85,9 @@ Supported options for **search** include > (--format=text0), as a JSON array (--format=json), or as > an S-Expression list (--format=sexp). > > + Duplicate addresses are filtered out. Filtering can be > + configured with the --filter-by option. > + > Note: Searching for **sender** should be much faster than > searching for **recipients**, because sender addresses are > cached directly in the database whereas other addresses > @@ -151,6 +154,41 @@ Supported options for **search** include > prefix. The prefix matches messages based on filenames. This > option filters filenames of the matching messages. > > + ``--filter-by=``\ (**nameaddr**\ \|\ **name** \|\ **addr**\ \|\ > **addrfold**\ \|\ **nameaddrfold**\) > + > + Can be used with ``--output=sender`` or > + ``--output=recipients`` to filter out duplicate addresses. The > + filtering algorithm receives a sequence of email addresses and > + outputs the same sequence without the addresses that are > + considered a duplicate of a previously output address. What is > + considered a duplicate depends on how the two addresses are > + compared and this can be controlled with the follwing flags: > + > + **nameaddr** means that both name and address parts are > + compared in case-sensitive manner. Therefore, all same looking > + addresses strings are considered duplicate. This is the > + default. > + > + **name** means that only the name part is compared (in > + case-sensitive manner). For example, the addresses "John Doe > + <me at example.com>" and "John Doe <john at doe.name>" will be > + considered duplicate. > + > + **addr** means that only the address part is compared (in > + case-sensitive manner). For example, the addresses "John Doe > + <john at example.com>" and "Dr. John Doe <john at example.com>" will > + be considered duplicate. > + > + **addrfold** is like **addr**, but comparison is done in > + canse-insensitive manner. For example, the addresses "John Doe > + <john at example.com>" and "Dr. John Doe <JOHN at EXAMPLE.COM>" will > + be considered duplicate. > + > + **nameaddrfold** is like **nameaddr**, but address comparison > + is done in canse-insensitive manner. For example, the > + addresses "John Doe <john at example.com>" and "John Doe > + <JOHN at EXAMPLE.COM>" will be considered duplicate. > + > EXIT STATUS > =========== > > diff --git a/notmuch-search.c b/notmuch-search.c > index ce3bfb2..47aa979 100644 > --- a/notmuch-search.c > +++ b/notmuch-search.c > @@ -34,6 +34,14 @@ typedef enum { > > #define OUTPUT_ADDRESS_FLAGS (OUTPUT_SENDER | OUTPUT_RECIPIENTS) > > +typedef enum { > + FILTER_BY_NAMEADDR = 0, > + FILTER_BY_NAME, > + FILTER_BY_ADDR, > + FILTER_BY_ADDRFOLD, > + FILTER_BY_NAMEADDRFOLD, > +} filter_by_t; > + > typedef struct { > sprinter_t *format; > notmuch_query_t *query; > @@ -42,6 +50,7 @@ typedef struct { > int offset; > int limit; > int dupe; > + filter_by_t filter_by; > } search_options_t; > > typedef struct { > @@ -229,6 +238,52 @@ do_search_threads (search_options_t *opt) > return 0; > } > > +/* Returns TRUE iff name and/or addr is considered duplicite. */ A triviality; duplicite should be duplicate > +static notmuch_bool_t > +check_duplicite (const search_options_t *opt, GHashTable *addrs, const char > *name, const char *addr) I am not sure on style but maybe is_duplicate would be clearer? Best wishes Mark > +{ > + notmuch_bool_t duplicite; > + char *key; > + > + if (opt->filter_by == FILTER_BY_ADDRFOLD || > + opt->filter_by == FILTER_BY_NAMEADDRFOLD) { > + gchar *folded = g_utf8_casefold (addr, -1); > + addr = talloc_strdup (opt->format, folded); > + g_free (folded); > + } > + switch (opt->filter_by) { > + case FILTER_BY_NAMEADDR: > + case FILTER_BY_NAMEADDRFOLD: > + key = talloc_asprintf (opt->format, "%s <%s>", name, addr); > + break; > + case FILTER_BY_NAME: > + key = talloc_strdup (opt->format, name); /* !name results in !key */ > + break; > + case FILTER_BY_ADDR: > + case FILTER_BY_ADDRFOLD: > + key = talloc_strdup (opt->format, addr); > + break; > + default: > + INTERNAL_ERROR("invalid --filter-by flags"); > + } > + > + if (opt->filter_by == FILTER_BY_ADDRFOLD || > + opt->filter_by == FILTER_BY_NAMEADDRFOLD) > + talloc_free ((char*)addr); > + > + if (! key) > + return FALSE; > + > + duplicite = g_hash_table_lookup_extended (addrs, key, NULL, NULL); > + > + if (! duplicite) > + g_hash_table_insert (addrs, key, NULL); > + else > + talloc_free (key); > + > + return duplicite; > +} > + > static void > print_mailbox (const search_options_t *opt, const mailbox_t *mailbox) > { > @@ -263,7 +318,8 @@ print_mailbox (const search_options_t *opt, const > mailbox_t *mailbox) > } > > static void > -process_address_list (const search_options_t *opt, InternetAddressList *list) > +process_address_list (const search_options_t *opt, GHashTable *addrs, > + InternetAddressList *list) > { > InternetAddress *address; > int i; > @@ -279,7 +335,7 @@ process_address_list (const search_options_t *opt, > InternetAddressList *list) > if (group_list == NULL) > continue; > > - process_address_list (opt, group_list); > + process_address_list (opt, addrs, group_list); > } else { > InternetAddressMailbox *mailbox = INTERNET_ADDRESS_MAILBOX > (address); > mailbox_t mbx = { > @@ -287,13 +343,16 @@ process_address_list (const search_options_t *opt, > InternetAddressList *list) > .addr = internet_address_mailbox_get_addr (mailbox), > }; > > + if (check_duplicite (opt, addrs, mbx.name, mbx.addr)) > + continue; > + > print_mailbox (opt, &mbx); > } > } > } > > static void > -process_address_header (const search_options_t *opt, const char *value) > +process_address_header (const search_options_t *opt, GHashTable *addrs, > const char *value) > { > InternetAddressList *list; > > @@ -304,7 +363,13 @@ process_address_header (const search_options_t *opt, > const char *value) > if (list == NULL) > return; > > - process_address_list (opt, list); > + process_address_list (opt, addrs, list); > +} > + > +static void > +_my_talloc_free_for_g_hash (void *ptr) > +{ > + talloc_free (ptr); > } > > static int > @@ -314,8 +379,13 @@ do_search_messages (search_options_t *opt) > notmuch_messages_t *messages; > notmuch_filenames_t *filenames; > sprinter_t *format = opt->format; > + GHashTable *addresses = NULL; > int i; > > + if (opt->output & OUTPUT_ADDRESS_FLAGS) > + addresses = g_hash_table_new_full (g_str_hash, g_str_equal, > + _my_talloc_free_for_g_hash, NULL); > + > if (opt->offset < 0) { > opt->offset += notmuch_query_count_messages (opt->query); > if (opt->offset < 0) > @@ -363,7 +433,7 @@ do_search_messages (search_options_t *opt) > const char *addrs; > > addrs = notmuch_message_get_header (message, "from"); > - process_address_header (opt, addrs); > + process_address_header (opt, addresses, addrs); > } > > if (opt->output & OUTPUT_RECIPIENTS) { > @@ -373,7 +443,7 @@ do_search_messages (search_options_t *opt) > > for (j = 0; j < ARRAY_SIZE (hdrs); j++) { > addrs = notmuch_message_get_header (message, hdrs[j]); > - process_address_header (opt, addrs); > + process_address_header (opt, addresses, addrs); > } > } > } > @@ -381,6 +451,9 @@ do_search_messages (search_options_t *opt) > notmuch_message_destroy (message); > } > > + if (addresses) > + g_hash_table_unref (addresses); > + > notmuch_messages_destroy (messages); > > format->end (format); > @@ -447,6 +520,7 @@ notmuch_search_command (notmuch_config_t *config, int > argc, char *argv[]) > .offset = 0, > .limit = -1, /* unlimited */ > .dupe = -1, > + .filter_by = FILTER_BY_NAMEADDR, > }; > char *query_str; > int opt_index, ret; > @@ -490,6 +564,13 @@ notmuch_search_command (notmuch_config_t *config, int > argc, char *argv[]) > { NOTMUCH_OPT_INT, &opt.offset, "offset", 'O', 0 }, > { NOTMUCH_OPT_INT, &opt.limit, "limit", 'L', 0 }, > { NOTMUCH_OPT_INT, &opt.dupe, "duplicate", 'D', 0 }, > + { NOTMUCH_OPT_KEYWORD, &opt.filter_by, "filter-by", 'b', > + (notmuch_keyword_t []){ { "nameaddr", FILTER_BY_NAMEADDR }, > + { "name", FILTER_BY_NAME }, > + { "addr", FILTER_BY_ADDR }, > + { "addrfold", FILTER_BY_ADDRFOLD }, > + { "nameaddrfold", FILTER_BY_NAMEADDRFOLD }, > + { 0, 0 } } }, > { 0, 0, 0, 0, 0 } > }; > > @@ -500,6 +581,11 @@ notmuch_search_command (notmuch_config_t *config, int > argc, char *argv[]) > if (! opt.output) > opt.output = OUTPUT_SUMMARY; > > + if (opt.filter_by && !(opt.output & OUTPUT_ADDRESS_FLAGS)) { > + fprintf (stderr, "Error: --filter-by can only be used with address > output.\n"); > + return EXIT_FAILURE; > + } > + > switch (format_sel) { > case NOTMUCH_FORMAT_TEXT: > opt.format = sprinter_text_create (config, stdout); > diff --git a/test/T090-search-output.sh b/test/T090-search-output.sh > index 947d572..841a721 100755 > --- a/test/T090-search-output.sh > +++ b/test/T090-search-output.sh > @@ -387,6 +387,93 @@ cat <<EOF >EXPECTED > EOF > test_expect_equal_file OUTPUT EXPECTED > > +test_begin_subtest "--output=sender" > +notmuch search --output=sender '*' >OUTPUT > +cat <<EOF >EXPECTED > +Fran?ois Boulogne <boulogne.f at gmail.com> > +Olivier Berger <olivier.berger at it-sudparis.eu> > +Chris Wilson <chris at chris-wilson.co.uk> > +Carl Worth <cworth at cworth.org> > +Alexander Botero-Lowry <alex.boterolowry at gmail.com> > +Keith Packard <keithp at keithp.com> > +Jjgod Jiang <gzjjgod at gmail.com> > +Rolland Santimano <rollandsantimano at yahoo.com> > +Jan Janak <jan at ryngle.com> > +Stewart Smith <stewart at flamingspork.com> > +Lars Kellogg-Stedman <lars at seas.harvard.edu> > +Alex Botero-Lowry <alex.boterolowry at gmail.com> > +Ingmar Vanhassel <ingmar at exherbo.org> > +Aron Griffis <agriffis at n01se.net> > +Adrian Perez de Castro <aperez at igalia.com> > +Israel Herraiz <isra at herraiz.org> > +Mikhail Gusarov <dottedmag at dottedmag.net> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=sender --format=json" > +notmuch search --output=sender --format=json '*' >OUTPUT > +cat <<EOF >EXPECTED > +[{"name": "Fran?ois Boulogne", "address": "boulogne.f at gmail.com"}, > +{"name": "Olivier Berger", "address": "olivier.berger at it-sudparis.eu"}, > +{"name": "Chris Wilson", "address": "chris at chris-wilson.co.uk"}, > +{"name": "Carl Worth", "address": "cworth at cworth.org"}, > +{"name": "Alexander Botero-Lowry", "address": "alex.boterolowry at > gmail.com"}, > +{"name": "Keith Packard", "address": "keithp at keithp.com"}, > +{"name": "Jjgod Jiang", "address": "gzjjgod at gmail.com"}, > +{"name": "Rolland Santimano", "address": "rollandsantimano at yahoo.com"}, > +{"name": "Jan Janak", "address": "jan at ryngle.com"}, > +{"name": "Stewart Smith", "address": "stewart at flamingspork.com"}, > +{"name": "Lars Kellogg-Stedman", "address": "lars at seas.harvard.edu"}, > +{"name": "Alex Botero-Lowry", "address": "alex.boterolowry at gmail.com"}, > +{"name": "Ingmar Vanhassel", "address": "ingmar at exherbo.org"}, > +{"name": "Aron Griffis", "address": "agriffis at n01se.net"}, > +{"name": "Adrian Perez de Castro", "address": "aperez at igalia.com"}, > +{"name": "Israel Herraiz", "address": "isra at herraiz.org"}, > +{"name": "Mikhail Gusarov", "address": "dottedmag at dottedmag.net"}] > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=recipients" > +notmuch search --output=recipients '*' >OUTPUT > +cat <<EOF >EXPECTED > +Allan McRae <allan at archlinux.org> > +Discussion about the Arch User Repository (AUR) <aur-general at > archlinux.org> > +olivier.berger at it-sudparis.eu > +notmuch at notmuchmail.org > +notmuch <notmuch at notmuchmail.org> > +Keith Packard <keithp at keithp.com> > +Mikhail Gusarov <dottedmag at dottedmag.net> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=sender --output=recipients" > +notmuch search --output=sender --output=recipients '*' >OUTPUT > +cat <<EOF >EXPECTED > +Fran?ois Boulogne <boulogne.f at gmail.com> > +Allan McRae <allan at archlinux.org> > +Discussion about the Arch User Repository (AUR) <aur-general at > archlinux.org> > +Olivier Berger <olivier.berger at it-sudparis.eu> > +olivier.berger at it-sudparis.eu > +Chris Wilson <chris at chris-wilson.co.uk> > +notmuch at notmuchmail.org > +Carl Worth <cworth at cworth.org> > +Alexander Botero-Lowry <alex.boterolowry at gmail.com> > +Keith Packard <keithp at keithp.com> > +Jjgod Jiang <gzjjgod at gmail.com> > +Rolland Santimano <rollandsantimano at yahoo.com> > +Jan Janak <jan at ryngle.com> > +Stewart Smith <stewart at flamingspork.com> > +Lars Kellogg-Stedman <lars at seas.harvard.edu> > +notmuch <notmuch at notmuchmail.org> > +Alex Botero-Lowry <alex.boterolowry at gmail.com> > +Ingmar Vanhassel <ingmar at exherbo.org> > +Aron Griffis <agriffis at n01se.net> > +Adrian Perez de Castro <aperez at igalia.com> > +Israel Herraiz <isra at herraiz.org> > +Mikhail Gusarov <dottedmag at dottedmag.net> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > test_begin_subtest "sanitize output for quoted-printable line-breaks in > author and subject" > add_message "[subject]='two =?ISO-8859-1?Q?line=0A_subject?= > headers'" > diff --git a/test/T095-search-filter-by.sh b/test/T095-search-filter-by.sh > new file mode 100755 > index 0000000..97d9a9b > --- /dev/null > +++ b/test/T095-search-filter-by.sh > @@ -0,0 +1,64 @@ > +#!/usr/bin/env bash > +test_description='duplicite address filtering in "notmuch search > --output=recipients"' > +. ./test-lib.sh > + > +add_message '[to]="Real Name <foo at example.com>, Real Name <bar at > example.com>"' > +add_message '[to]="Nickname <foo at example.com>"' '[cc]="Real Name <Bar at > Example.COM>"' > +add_message '[to]="Nickname <foo at example.com>"' '[bcc]="Real Name <Bar at > Example.COM>"' > + > +test_begin_subtest "--output=recipients" > +notmuch search --output=recipients "*" >OUTPUT > +cat <<EOF >EXPECTED > +Real Name <foo at example.com> > +Real Name <bar at example.com> > +Nickname <foo at example.com> > +Real Name <Bar at Example.COM> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=recipients --filter-by=nameaddr" > +notmuch search --output=recipients --filter-by=nameaddr "*" >OUTPUT > +# The same as above > +cat <<EOF >EXPECTED > +Real Name <foo at example.com> > +Real Name <bar at example.com> > +Nickname <foo at example.com> > +Real Name <Bar at Example.COM> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=recipients --filter-by=name" > +notmuch search --output=recipients --filter-by=name "*" >OUTPUT > +cat <<EOF >EXPECTED > +Real Name <foo at example.com> > +Nickname <foo at example.com> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=recipients --filter-by=addr" > +notmuch search --output=recipients --filter-by=addr "*" >OUTPUT > +cat <<EOF >EXPECTED > +Real Name <foo at example.com> > +Real Name <bar at example.com> > +Real Name <Bar at Example.COM> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=recipients --filter-by=addrfold" > +notmuch search --output=recipients --filter-by=addrfold "*" >OUTPUT > +cat <<EOF >EXPECTED > +Real Name <foo at example.com> > +Real Name <bar at example.com> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_begin_subtest "--output=recipients --filter-by=nameaddrfold" > +notmuch search --output=recipients --filter-by=nameaddrfold "*" >OUTPUT > +cat <<EOF >EXPECTED > +Real Name <foo at example.com> > +Real Name <bar at example.com> > +Nickname <foo at example.com> > +EOF > +test_expect_equal_file OUTPUT EXPECTED > + > +test_done > -- > 2.1.1 > > _______________________________________________ > notmuch mailing list > notmuch at notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch