Martin Kleppmann created SAMZA-423:
--------------------------------------
Summary: Integrate Lucene into Samza
Key: SAMZA-423
URL: https://issues.apache.org/jira/browse/SAMZA-423
Project: Samza
Issue Type: New Feature
Reporter: Martin Kleppmann
At the moment Samza only has a key-value storage engine
(LevelDB/RocksDB/in-memory), but Samza's state abstraction is designed to be
more general than that. In particular, we've discussed integrating Lucene in
order to support full-text indexes.
There are two modes of using Lucene which would make sense in a stream
processing system:
* Treat incoming messages as documents to be added to an index. This would be
akin to the key-value storage model, but with much richer indexing
capabilities. It would enable joining streams not just on a single key, but on
more complex criteria (arbitrary boolean expressions for joins) and stuff like
deduplicating similar documents (which you may want e.g. if implementing a web
crawler).
* Match incoming messages against a set of queries, where the queries are more
or less fixed (perhaps the queries are updated via another input stream). In
this case, the message isn't added to a persistent index, but it's only
analysed and matched against a query as it flows through the stream processor.
This is useful for monitoring an activity stream for events of interest
("notify me whenever a news article mentions my company name"). It's perhaps
comparable to ElasticSearch's
[percolator|http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html].
I was chatting to [~romseygeek] yesterday, a committer on Lucene/Solr. We have
a vague plan to hack on a proof of concept to see what an integration of Lucene
and Samza could look like. This is a placeholder ticket for collecting any
stuff related to that experiment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
