Wietse Venema writes:
Unless I am mistaken, this implements the same functionality as the
pipemap table. It queries tables in sequence, not in parallel.
Attached is the new patch. Sorry about the confusion.
This one has some documentation changes as well.
Thanks,
Roel
Add support for joinmap tables
This patchs adds support for joinmap tables. A joinmap table is a table
that contains two or more other tables. A lookup in the joinmap tables is
done in all of the tables it contains. The result is concatenated and
returned.
The use case for this type of table is if you have two *different* tables
containing similar information (say an ldap table and a hash table with
virtial aliases) or if you have to do two lookups to get all the information
you want.
diff -ruN a/.indent.pro e/.indent.pro
--- a/.indent.pro 2014-07-18 01:09:15.000000000 +0200
+++ e/.indent.pro 2014-09-11 11:47:57.208236388 +0200
@@ -77,6 +77,7 @@
-TDICT_DEBUG
-TDICT_ENV
-TDICT_FAIL
+-TDICT_JOIN
-TDICT_HT
-TDICT_LDAP
-TDICT_LMDB
diff -ruN a/README_FILES/DATABASE_README e/README_FILES/DATABASE_README
--- a/README_FILES/DATABASE_README 2014-06-30 23:00:51.000000000 +0200
+++ e/README_FILES/DATABASE_README 2014-09-11 15:38:41.238330621 +0200
@@ -215,6 +215,15 @@
iinntteerrnnaall
A non-shared, in-memory hash table. Its content are lost when a process
terminates.
+ jjooiinnmmaapp (read-only)
+ Join the result of lookups in several tables. Example:
"jjooiinnmmaapp::
+ !type1:name1! ... !typen:namen". A "joinmap:" query is given to all
+ tables in the joinmap. Any results that are received are joined and
+ the end result is returned. The query is always given to all tables,
+ even if a result has already been found. Results are not deduplicated.
+ The first ASCII character after "joinmap:" will be used as the
separator
+ between the lookup tables that follow (do not use space, ",", ":" or
+ non-ASCII).
llmmddbb
OpenLDAP LMDB database. This is available only on systems with support
for LMDB databases. Public database files are created with the postmap
diff -ruN a/html/DATABASE_README.html e/html/DATABASE_README.html
--- a/html/DATABASE_README.html 2014-06-30 23:00:50.000000000 +0200
+++ e/html/DATABASE_README.html 2014-09-11 15:54:24.842153050 +0200
@@ -318,6 +318,19 @@
<dd> A non-shared, in-memory hash table. Its content are lost when
a process terminates. </dd>
+<dt> <b>joinmap</b> (read-only) </dt>
+
+<dd> Joins the result of lookups in several tables. Example:
+"<b><a
href="DATABASE_README.html#types">joinmap</a>:</b><i>!type<sub>1</sub>:name<sub>1</sub>!
...
+!type<sub>n</sub>:name<sub>n</sub></i>". A "<a
href="DATABASE_README.html#types">joinmap</a>:" query is
+given to all tables in the joinmap. Any results that are received
+are joined and the end result is returned. The query is always
+given to all tables, even if a result has already been found. Results
+are not deduplicated.
+The first ASCII character after "<a
href="DATABASE_README.html#types">pipemap</a>:"
+will be used as the separator between the lookup tables that follow
+(do not use space, ",", ":" or non-ASCII). </dd>
+
<dt> <b>lmdb</b> </dt>
<dd> OpenLDAP LMDB database. This is available only on systems
diff -ruN a/man/man1/postconf.1 e/man/man1/postconf.1
--- a/man/man1/postconf.1 2014-06-30 19:49:18.000000000 +0200
+++ e/man/man1/postconf.1 2014-09-11 15:43:38.878332662 +0200
@@ -255,6 +255,16 @@
.IP \fBinternal\fR
A non-shared, in-memory hash table. Its content are lost
when a process terminates.
+.IP "\fBjoinmap\fR (read-only)"
+Join the result of lookups in several tables. Example:
+"\fBjoinmap:\fI!type_1:name_1! ... !type_n:name_n\fR".
+A "joinmap:" query is given to all tables in the joinmap.
+Any results that are received are joined and the end
+result is returned. The query is always given to all tables,
+even if a result has already been found. Results are not
+deduplicated. The first ASCII character after "joinmap:"
+will be used as the separator between the lookup tables
+that follow (do not use space, ",", ":" or non-ASCII).
.IP "\fBlmdb\fR"
OpenLDAP LMDB database (a memory-mapped, persistent file).
Available on systems with support for LMDB databases. This
diff -ruN a/mantools/postlink e/mantools/postlink
--- a/mantools/postlink 2014-09-07 18:01:34.000000000 +0200
+++ e/mantools/postlink 2014-09-11 15:45:02.148333229 +0200
@@ -1118,6 +1118,7 @@
s/\b(fail):/<a href="DATABASE_README.html#types">$1<\/a>:/g;
s/\b(hash):/<a href="DATABASE_README.html#types">$1<\/a>:/g;
s/\b(internal):/<a href="DATABASE_README.html#types">$1<\/a>:/g;
+ s/\b(joinmap):/<a href="DATABASE_README.html#types">$1<\/a>:/g;
s/\b(ldap):/<a href="ldap_table.5.html">$1<\/a>:/g;
s/\b(lmdb):/<a href="lmdb_table.5.html">$1<\/a>:/g;
s/\b(memcache):/<a href="memcache_table.5.html">$1<\/a>:/g;
diff -ruN a/proto/DATABASE_README.html e/proto/DATABASE_README.html
--- a/proto/DATABASE_README.html 2014-06-30 22:56:16.000000000 +0200
+++ e/proto/DATABASE_README.html 2014-09-11 15:47:22.272194203 +0200
@@ -318,6 +318,18 @@
<dd> A non-shared, in-memory hash table. Its content are lost when
a process terminates. </dd>
+<dt> <b>joinmap</b> (read-only) </dt>
+
+<dd> Joins the result of lookups in several tables. Example:
+"<b>joinmap:</b><i>!type<sub>1</sub>:name<sub>1</sub>! ...
+!type<sub>n</sub>:name<sub>n</sub></i>". A "joinmap:" query is
+given to all tables in the joinmap. Any results that are received
+are joined and the end result is returned. The query is always
+given to all tables, even if a result has already been found.
+Results are not deduplicated. The first ASCII character after
+"joinmap:" will be used as the separator between the lookup tables
+that follow (do not use space, ",", ":" or non-ASCII).</dd>
+
<dt> <b>lmdb</b> </dt>
<dd> OpenLDAP LMDB database. This is available only on systems
diff -ruN a/src/postconf/postconf.c e/src/postconf/postconf.c
--- a/src/postconf/postconf.c 2014-06-30 19:49:01.000000000 +0200
+++ e/src/postconf/postconf.c 2014-09-11 15:50:07.002151229 +0200
@@ -249,6 +249,16 @@
/* .IP \fBinternal\fR
/* A non-shared, in-memory hash table. Its content are lost
/* when a process terminates.
+/* .IP "\fBjoinmap\fR (read-only)"
+/* Joins the result of lookups in several tables. Example:
+/* "\fBjoinmap:\fI!type_1:name_1! ... !type_n:name_n\fR".
+/* A "joinmap:" query is given to all tables in the joinmap.
+/* Any results that are received are joined and the end result
+/* is returned. The query is always given to all tables, even
+/* if a result has already been found. Results are not
+/* deduplicated. The first ASCII character after "joinmap:"
+/* will be used as the separator between the lookup tables
+/* that follow (do not use space, ",", ":" or non-ASCII).
/* .IP "\fBlmdb\fR"
/* OpenLDAP LMDB database (a memory-mapped, persistent file).
/* Available on systems with support for LMDB databases. This
diff -ruN a/src/util/Makefile.in e/src/util/Makefile.in
--- a/src/util/Makefile.in 2014-07-21 01:21:23.000000000 +0200
+++ e/src/util/Makefile.in 2014-09-11 11:54:25.868238221 +0200
@@ -38,7 +38,7 @@
dict_fail.c msg_rate_delay.c dict_surrogate.c warn_stat.c \
dict_sockmap.c line_number.c recv_pass_attr.c pass_accept.c \
poll_fd.c timecmp.c slmdb.c dict_pipe.c dict_random.c \
- valid_utf8_hostname.c midna.c
+ valid_utf8_hostname.c midna.c dict_join.c
OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \
attr_print64.o attr_print_plain.o attr_scan0.o attr_scan64.o \
attr_scan_plain.o auto_clnt.o base64_code.o basename.o binhash.o \
@@ -78,7 +78,7 @@
dict_fail.o msg_rate_delay.o dict_surrogate.o warn_stat.o \
dict_sockmap.o line_number.o recv_pass_attr.o pass_accept.o \
poll_fd.o timecmp.o $(NON_PLUGIN_MAP_OBJ) dict_pipe.o dict_random.o \
- valid_utf8_hostname.o midna.o
+ valid_utf8_hostname.o midna.o dict_join.o
# MAP_OBJ is for maps that may be dynamically loaded with dynamicmaps.cf.
# When hard-linking these, makedefs sets NON_PLUGIN_MAP_OBJ=$(MAP_OBJ),
# otherwise it sets the PLUGIN_* macros.
@@ -107,7 +107,7 @@
edit_file.h dict_cache.h dict_thash.h ip_match.h nbbio.h base32_code.h \
dict_fail.h warn_stat.h dict_sockmap.h line_number.h timecmp.h \
slmdb.h compat_va_copy.h dict_pipe.h dict_random.h \
- valid_utf8_hostname.h midna.h
+ valid_utf8_hostname.h midna.h dict_join.h
TESTSRC = fifo_open.c fifo_rdwr_bug.c fifo_rdonly_bug.c select_bug.c \
stream_test.c dup2_pass_on_exec.c
DEFS = -I. -D$(SYSTYPE)
@@ -1043,6 +1043,19 @@
dict_ht.o: vbuf.h
dict_ht.o: vstream.h
dict_ht.o: vstring.h
+dict_join.o: argv.h
+dict_join.o: dict.h
+dict_join.o: dict_join.c
+dict_join.o: dict_join.h
+dict_join.o: htable.h
+dict_join.o: msg.h
+dict_join.o: myflock.h
+dict_join.o: mymalloc.h
+dict_join.o: stringops.h
+dict_join.o: sys_defs.h
+dict_join.o: vbuf.h
+dict_join.o: vstream.h
+dict_join.o: vstring.h
dict_lmdb.o: argv.h
dict_lmdb.o: dict.h
dict_lmdb.o: dict_lmdb.c
@@ -1094,6 +1107,7 @@
dict_open.o: dict_env.h
dict_open.o: dict_fail.h
dict_open.o: dict_ht.h
+dict_open.o: dict_join.h
dict_open.o: dict_lmdb.h
dict_open.o: dict_ni.h
dict_open.o: dict_nis.h
diff -ruN a/src/util/dict_join.c e/src/util/dict_join.c
--- a/src/util/dict_join.c 1970-01-01 01:00:00.000000000 +0100
+++ e/src/util/dict_join.c 2014-09-11 14:09:08.548294332 +0200
@@ -0,0 +1,200 @@
+/*++
+/* NAME
+/* dict_join 3
+/* SUMMARY
+/* dictionary manager interface for pipelined tables
+/* SYNOPSIS
+/* #include <dict_join.h>
+/*
+/* DICT *dict_join_open(name, open_flags, dict_flags)
+/* const char *name;
+/* int open_flags;
+/* int dict_flags;
+/* DESCRIPTION
+/* dict_join_open() opens a pipeline of one or more tables.
+/* Example: "\fBpipemap:\fI!type_1:name_1! ... !type_n:name_n\fR".
+/*
+/* Each "pipemap:" query is given to the first table. Each
+/* lookup result becomes the query for the next table in the
+/* pipeline, and the last table produces the final result.
+/* When any table lookup produces no result, the pipeline
+/* produces no result.
+/*
+/* The first ASCII character after "pipemap:" will be used as
+/* the separator between the lookup tables that follow (do not
+/* use space, ",", ":" or non-ASCII).
+/*
+/* The open_flags and dict_flags arguments are passed on to
+/* the underlying dictionaries.
+/* SEE ALSO
+/* dict(3) generic dictionary manager
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Wietse Venema
+/* IBM T.J. Watson Research
+/* P.O. Box 704
+/* Yorktown Heights, NY 10598, USA
+/*--*/
+
+/* System library. */
+
+#include <sys_defs.h>
+#include <string.h>
+
+/* Utility library. */
+
+#include <msg.h>
+#include "mymalloc.h"
+#include "htable.h"
+#include "dict.h"
+#include "dict_join.h"
+#include "stringops.h"
+#include "vstring.h"
+
+/* Application-specific. */
+
+typedef struct {
+ DICT dict; /* generic members */
+ ARGV *map_join; /* pipelined tables */
+ VSTRING *re_buf; /* reply buffer */
+} DICT_JOIN;
+
+#define STR(x) vstring_str(x)
+
+/* dict_join_lookup - search pipelined tables */
+
+static const char *dict_join_lookup(DICT *dict, const char *query)
+{
+ const char myname[] = "dict_join_lookup";
+ DICT_JOIN *dict_join = (DICT_JOIN *) dict;
+ DICT *map;
+ char **cpp;
+ char *dict_type_name;
+ const char *result = 0;
+
+ VSTRING_RESET(dict_join->re_buf);
+ for (cpp = dict_join->map_join->argv; (dict_type_name = *cpp) != 0; cpp++)
{
+ if ((map = dict_handle(dict_type_name)) == 0)
+ msg_panic("%s: dictionary \"%s\" not found", myname,
dict_type_name);
+ if ((result = dict_get(map, query)) == 0)
+ continue;
+ if (VSTRING_LEN(dict_join->re_buf) > 0)
+ VSTRING_ADDCH(dict_join->re_buf, ',');
+ vstring_strcat(dict_join->re_buf, result);
+ }
+ DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE,
+ VSTRING_LEN(dict_join->re_buf) > 0
+ ? STR(dict_join->re_buf)
+ : 0);
+}
+
+/* dict_join_close - disassociate from pipelined tables */
+
+static void dict_join_close(DICT *dict)
+{
+ DICT_JOIN *dict_join = (DICT_JOIN *) dict;
+ char **cpp;
+ char *dict_type_name;
+
+ for (cpp = dict_join->map_join->argv; (dict_type_name = *cpp) != 0; cpp++)
+ dict_unregister(dict_type_name);
+ argv_free(dict_join->map_join);
+ vstring_free(dict_join->re_buf);
+ dict_free(dict);
+}
+
+/* dict_join_open - open pipelined tables */
+
+DICT *dict_join_open(const char *name, int open_flags, int dict_flags)
+{
+ const char myname[] = "dict_join_open";
+ DICT_JOIN *dict_join;
+ char *saved_name = 0;
+ char *dict_type_name;
+ ARGV *argv = 0;
+ char **cpp;
+ DICT *dict;
+ int match_flags = 0;
+ struct DICT_OWNER aggr_owner;
+ char delim[2];
+
+ /*
+ * Clarity first. Let the optimizer worry about redundant code.
+ */
+#define DICT_JOIN_RETURN(x) do { \
+ if (saved_name != 0) \
+ myfree(saved_name); \
+ if (argv != 0) \
+ argv_free(argv); \
+ return (x); \
+ } while (0)
+
+ /*
+ * Sanity checks.
+ */
+ if (open_flags != O_RDONLY)
+ DICT_JOIN_RETURN(dict_surrogate(DICT_TYPE_JOIN, name,
+ open_flags, dict_flags,
+ "%s:%s map requires O_RDONLY access mode",
+ DICT_TYPE_JOIN, name));
+ if (name[0] == ':')
+ DICT_JOIN_RETURN(dict_surrogate(DICT_TYPE_JOIN, name,
+ open_flags, dict_flags,
+ "invalid list delimiter \"%c\" in \"%s:%s\"",
+ name[0], DICT_TYPE_JOIN, name));
+
+ /*
+ * Split the table name on the user-specified delimiter.
+ */
+ delim[0] = name[0]; /* XXX ASCII delimiter
*/
+ delim[1] = 0;
+ saved_name = mystrdup(name + 1); /* XXX ASCII delimiter */
+ if (*saved_name == 0)
+ DICT_JOIN_RETURN(dict_surrogate(DICT_TYPE_JOIN, name,
+ open_flags, dict_flags,
+ "bad syntax: \"%s:%s\"; "
+ "need \"%s:%stype:name%s...\"",
+ DICT_TYPE_JOIN, name,
+ DICT_TYPE_JOIN, delim, delim));
+
+ /*
+ * The least-trusted table in the pipeline determines the over-all trust
+ * level. The first table determines the pattern-matching flags.
+ */
+ DICT_OWNER_AGGREGATE_INIT(aggr_owner);
+ argv = argv_split(saved_name, delim);
+ for (cpp = argv->argv; (dict_type_name = *cpp) != 0; cpp++) {
+ if (msg_verbose)
+ msg_info("%s: %s", myname, dict_type_name);
+ if (strchr(dict_type_name, ':') == 0)
+ DICT_JOIN_RETURN(dict_surrogate(DICT_TYPE_JOIN, name,
+ open_flags, dict_flags,
+ "bad syntax: \"%s:%s\"; "
+ "need \"%s:%stype:name%s...\"",
+ DICT_TYPE_JOIN, name,
+ DICT_TYPE_JOIN, delim, delim));
+ if ((dict = dict_handle(dict_type_name)) == 0)
+ dict = dict_open(dict_type_name, open_flags, dict_flags);
+ dict_register(dict_type_name, dict);
+ DICT_OWNER_AGGREGATE_UPDATE(aggr_owner, dict->owner);
+ if (cpp == argv->argv)
+ match_flags = dict->flags & (DICT_FLAG_FIXED | DICT_FLAG_PATTERN);
+ }
+
+ /*
+ * Bundle up the result.
+ */
+ dict_join =
+ (DICT_JOIN *) dict_alloc(DICT_TYPE_JOIN, name, sizeof(*dict_join));
+ dict_join->dict.lookup = dict_join_lookup;
+ dict_join->dict.close = dict_join_close;
+ dict_join->dict.flags = dict_flags | match_flags;
+ dict_join->dict.owner = aggr_owner;
+ dict_join->re_buf = vstring_alloc(100);
+ dict_join->map_join = argv;
+ argv = 0;
+ DICT_JOIN_RETURN(DICT_DEBUG (&dict_join->dict));
+}
diff -ruN a/src/util/dict_join.h e/src/util/dict_join.h
--- a/src/util/dict_join.h 1970-01-01 01:00:00.000000000 +0100
+++ e/src/util/dict_join.h 2014-09-11 12:03:06.138242563 +0200
@@ -0,0 +1,37 @@
+#ifndef _DICT_JOIN_H_INCLUDED_
+#define _DICT_JOIN_H_INCLUDED_
+
+/*++
+/* NAME
+/* dict_join 3h
+/* SUMMARY
+/* dictionary manager interface for joined tables
+/* SYNOPSIS
+/* #include <dict_join.h>
+/* DESCRIPTION
+/* .nf
+
+ /*
+ * Utility library.
+ */
+#include <dict.h>
+
+ /*
+ * External interface.
+ */
+#define DICT_TYPE_JOIN "joinmap"
+
+extern DICT *dict_join_open(const char *, int, int);
+
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Wietse Venema
+/* IBM T.J. Watson Research
+/* P.O. Box 704
+/* Yorktown Heights, NY 10598, USA
+/*--*/
+
+#endif
diff -ruN a/src/util/dict_open.c e/src/util/dict_open.c
--- a/src/util/dict_open.c 2014-06-30 20:26:57.000000000 +0200
+++ e/src/util/dict_open.c 2014-09-11 12:03:46.898242860 +0200
@@ -300,6 +300,7 @@
#include <dict_fail.h>
#include <dict_pipe.h>
#include <dict_random.h>
+#include <dict_join.h>
#include <stringops.h>
#include <split_at.h>
#include <htable.h>
@@ -343,6 +344,7 @@
DICT_TYPE_SOCKMAP, dict_sockmap_open,
DICT_TYPE_FAIL, dict_fail_open,
DICT_TYPE_PIPE, dict_pipe_open,
+ DICT_TYPE_JOIN, dict_join_open,
#ifdef DICT_TYPE_PIPE_LEGACY
DICT_TYPE_PIPE_LEGACY, dict_pipe_open,
#endif
