[
https://issues.apache.org/jira/browse/LUCENE-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863062#action_12863062
]
Karthick Sankarachary edited comment on LUCENE-2425 at 5/2/10 3:56 PM:
-----------------------------------------------------------------------
In this comment, we outline all split policies that have been or are in the
process of being implemented. Hopefully, this will serve to not only validate
the framework, but also be a reference point for future work.
The split policies available so far include:
1) LUCENE-2429: A rotating split policy, which is essentially a time-bound
index, where each sub-index denotes a (contiguous) time range, and there's a
cap on the number of sub-indices.
2) LUCENE-2430: An archiving split policy, which builds on the rotating split
policy, where older sub-indexes (that have been rotated out) are kept around
for a while before being removed.
3) LUCENE-2431: A real-time split policy, which overcomes the near-real time
limitation of current indices. It does so by essentially maintaing a cache for
each reader obtained for that index.
4) LUCENE-2432: A caching split policy, which builds on the real-time split
policy, where writes (and other updates) to the index are buffered in-memory
until it is told to flush.
5) LUCENE-2433: A remoting split policy, which is an abstraction where each
sub-directory maps to a (remote) URI.
5) LUCENE-2434: A mirroring split policy, which treats each sub-directory as a
mirror image of the super-directory.
6) LUCENE-2435: A sharding split policy, which treats each sub-directory as a
shard (or slice) or the super-directory.
The split policies under development include:
was (Author: karthick):
In this comment, we outline all split policies that have been or are in the
process of being implemented. Hopefully, this will serve to not only validate
the framework, but also be a reference point for future work.
The split policies currently under development include:
1) A rotating split policy, which is essentially a time-bound index, where each
sub-index denotes a (contiguous) time range, and there's a cap on the number of
sub-indices.
2) An archiving split policy, which builds on the rotating split policy, where
older sub-indexes (that have been rotated out) are kept around for a while
before being removed.
3) A real-time split policy, which overcomes the near-real time limitation of
current indices. It does so by essentially maintaing a cache for each reader
obtained for that index.
4) A caching split policy, which builds on the real-time split policy, where
writes (and other updates) to the index are buffered in-memory until it is told
to flush.
5) A mirroring split policy, which treats each sub-directory as a mirror image
of the super-directory.
6) A sharding split policy, which treats each sub-directory as a shard (or
slice) or the super-directory.
> An Anti-Merging Multi-Directory Indexing Framework
> --------------------------------------------------
>
> Key: LUCENE-2425
> URL: https://issues.apache.org/jira/browse/LUCENE-2425
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/*, Index
> Affects Versions: 3.0.1
> Reporter: Karthick Sankarachary
> Attachments: LUCENE-2425.patch
>
>
> By design, a Lucene index tends to merge documents that span multiple
> segments into fewer segments, in order to optimize its directory structure,
> which in turn leads to better search performance. In particular, it relies on
> a merge policy to specify the set of merge operations that should be
> performed when the index is optimized.
> Often times, there's a need to do the exact opposite, which is to "split" the
> documents. This calls for a mechanism that facilitates sub-division of
> documents based on a certain (ideally, user-defined) algorithm. By way of
> example, one may wish to sub-divide (or partition) documents based on
> parameters such as time, space, real-timeliness, and so on. Herein, we
> describe an indexing framework that builds on the Lucene index writer and
> reader, to address use cases wherein documents need to diverge rather than
> converge.
> In brief, it associates zero or more sub-directories with the index's
> directory, which serve to complement it in some manner. The sub-directories
> (a.k.a. splits) are managed by a split policy, which is notified of all
> changes made to the index directory (a.k.a. super-directory), thus allowing
> it to modify its sub-directories as it sees fit. To make the index reader and
> writer "observable", we extend Lucene's reader and writer with the goal of
> providing hooks into every method that could potentially change the index.
> This allows for propagation of such changes to the split policy, which
> essentially acts as a listener on the index.
> We refer to each sub-directory (or split) and the super-directory as a
> sub-index of the containing index (a.k.a. the split index). Note that the
> sub-directory may not necessarily be co-located with the super-directory.
> Furthermore, the split policy in turn relies on one or more split rules to
> determine when to add or remove sub-directories. This allows for a clear
> separation of the event that triggers a split from the management of those
> splits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
