[
https://issues.apache.org/jira/browse/SAMZA-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661769#comment-14661769
]
Robert Žuljević commented on SAMZA-423:
---------------------------------------
I've attached a design document. Since this is my first document of this kind I
would like to ask you to be as brutal as possible and don't hold back on any
suggestions/comments/criticisms that you might have.
Per my suggestion in the document, I would like to rename this ticket to
"Create Document Store" and add two new, related tickets, "Create Lucene
Document Store" and "Create Luwak Document Store". Any objections?
> Integrate Lucene into Samza
> ---------------------------
>
> Key: SAMZA-423
> URL: https://issues.apache.org/jira/browse/SAMZA-423
> Project: Samza
> Issue Type: New Feature
> Reporter: Martin Kleppmann
> Assignee: Robert Žuljević
> Labels: design, project
> Attachments: DESIGN-SAMZA-423-0.pdf
>
>
> At the moment Samza only has a key-value storage engine
> (LevelDB/RocksDB/in-memory), but Samza's state abstraction is designed to be
> more general than that. In particular, we've discussed integrating Lucene in
> order to support full-text indexes.
> There are two modes of using Lucene which would make sense in a stream
> processing system:
> * Treat incoming messages as documents to be added to an index. This would be
> akin to the key-value storage model, but with much richer indexing
> capabilities. It would enable joining streams not just on a single key, but
> on more complex criteria (arbitrary boolean expressions for joins) and stuff
> like deduplicating similar documents (which you may want e.g. if implementing
> a web crawler).
> * Match incoming messages against a set of queries, where the queries are
> more or less fixed (perhaps the queries are updated via another input
> stream). In this case, the message isn't added to a persistent index, but
> it's only analysed and matched against a query as it flows through the stream
> processor. This is useful for monitoring an activity stream for events of
> interest ("notify me whenever a news article mentions my company name"). It's
> perhaps comparable to ElasticSearch's
> [percolator|http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html].
> I was chatting to [~romseygeek] yesterday, a committer on Lucene/Solr. We
> have a vague plan to hack on a proof of concept to see what an integration of
> Lucene and Samza could look like. This is a placeholder ticket for collecting
> any stuff related to that experiment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
