Ori.livneh has uploaded a new change for review.
https://gerrit.wikimedia.org/r/109494
Change subject: Add fan-out writer
......................................................................
Add fan-out writer
The fanout writer looks up the schema identifier of each incoming event in an
internal dictionary that maps schemas to lists of callback functions. If the
event's schema has callback functions defined, the fanout writer will call each
of them with the incoming event as argument.
The fanout writer populates its dictionary of callbacks by loading all Python
modules from a file path. Modules in this file path may register callbacks
(known as 'selectors') using the '@selects' decorator. For example:
# /var/lib/eventlogging/selectors/ve_logger.py
import logging
from eventlogging import selects
@selects('VisualEditorActivation', 6424756)
def log_ve_activation(event):
logging.info('VisualEditor activated in %s ms.', event['duration'])
An eventlogging-consumer instance which delegates to this (and other selectors)
may be invoked as follows:
eventlogging-consumer tcp://localhost:8600
fanout:///var/lib/eventlogging/selectors
I think that the fanout writer will make it substantially easier for people to
write one-off EventLogging plugins that report some metric to Graphite or
Ganglia.
The thing I don't like about it is the duplication of concepts. A selector is
basically a writer with a filter, so couldn't writers simply specify a
subscription? And isn't the event stream already subscription-based? And
doesn't the existing plugin architecture allow arbitrary plugins to load from
some file path? (Answer: maybe, but these things don't solve the problem of
having to specify some preprocessing step for an event subscription before it
is passed off to a writer.)
In short, I am not 100% sure on the architecture, but am interested in trying
it out.
Change-Id: Id8596535c7642f22518dc175be4c118c7270ee2b
---
M server/eventlogging/handlers.py
1 file changed, 26 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/EventLogging
refs/changes/94/109494/1
diff --git a/server/eventlogging/handlers.py b/server/eventlogging/handlers.py
index fd24b03..2c9cbeb 100644
--- a/server/eventlogging/handlers.py
+++ b/server/eventlogging/handlers.py
@@ -35,6 +35,10 @@
#: not defined, EventLogging will default to the value specified below.
DEFAULT_PLUGIN_DIR = '/usr/local/lib/eventlogging'
+#: A mapping of schema revision IDs to lists of callback functions
+#: ('selectors') to call when a matching event is encountered.
+selectors = {}
+
def load_plugins(path=None):
"""Load EventLogging plug-ins from `path`. Plug-in module names are mangled
@@ -43,6 +47,17 @@
path = os.environ.get('EVENTLOGGING_PLUGIN_DIR', DEFAULT_PLUGIN_DIR)
for plugin in glob.glob(os.path.join(path, '*.py')):
imp.load_source('__eventlogging_plugin_%x__' % hash(plugin), plugin)
+
+
+def selects(scid):
+ """Decorator that takes a SCID and registeres the decorated function
+ as an event selector of that SCID."""
+ schema_name, revision_id = scid
+ print('registering %s:%s' % scid)
+ def decorator(f):
+ selectors.setdefault(revision_id, []).append(f)
+ return f
+ return decorator
#
@@ -114,6 +129,17 @@
print(json.dumps((yield), **dumps_kwargs))
+@writes('fanout')
+def fanout_writer(path):
+ """Fan-out writer. Registers selectors by loading Python modules
+ from `path`, then fans out incoming events based on SCID."""
+ load_plugins(path)
+ while 1:
+ event = (yield)
+ for selector in selectors.get(event['revision'], ()):
+ selector(event)
+
+
#
# Readers
#
--
To view, visit https://gerrit.wikimedia.org/r/109494
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id8596535c7642f22518dc175be4c118c7270ee2b
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/extensions/EventLogging
Gerrit-Branch: master
Gerrit-Owner: Ori.livneh <o...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
