A Column-Oriented Cassandra-Based Lucene Directory
--------------------------------------------------
Key: LUCENE-2456
URL: https://issues.apache.org/jira/browse/LUCENE-2456
Project: Lucene - Java
Issue Type: New Feature
Components: contrib/*, Store
Affects Versions: 3.0.1
Reporter: Karthick Sankarachary
In brief, the CassandraDirectory maps the concept of a Lucene directory to a
column family that belongs to a certain keyspace located in a given Cassandra
server. Further, it stores each file under this directory as a row in that
column family.
Specifically, its files are broken down into blocks (whose sizes are capped),
where each block (see FileBlock) is stored as the value of a column in the
corresponding row. As per
http://wiki.apache.org/cassandra/CassandraLimitations, this is the recommended
approach for dealing with large objects, which Lucene files tend to be. In
addition, a descriptor of the file (see FileDescriptor) that outlines a map of
blocks therein is stored as one of the columns in that row as well. Think of
this descriptor as an inode for Cassandra-based files.
The exhaustive mapping of a Lucene directory (file) to a Cassandra column
family (row) is captured in the ColumnOrientedDirectory (ColumnOrientedFile)
inner-class. Specifically, it interprets Cassandra's data model in terms of
Lucene's, and vice verca. More importantly, these are the only two
inner-classes that have a foot in both the Lucene and Cassandra camps.
All writes to a file in this directory occur through a CassandraIndexOutput,
which puts the data flushed from a write-behind buffer into the fitting set of
blocks. By the same token, all reads from a file in this directory occur
through a CassandraIndexInput, which gets the data needed by a read-ahead
buffer from the right set of blocks.
The last (but not the least) inner-class, CassandraClient, acts as a facade
over a Thrift-based Cassandra client. In short, it provides operations to
get/put rows/columns in the column family and keyspace associated with this
directory.
Unlike Lucandra, which attempts to bridge the gap between Lucene and Cassandra
at the document-level, the CassandraDirectory is self-sufficient in the sense
that it does not require a re-write of any other component in the Lucene stack.
In other words, one may use the CassandraDirectory in conjunction with the
Lucene IndexWriter and IndexReader, as you would any other kind of Lucene
Directory. Moreover, given the the data unit that is transferred to and from
Cassandra is a large-sized block, one may expect fewer round trips, and hence
better throughputs, from the CassandraDirectory.
In conclusion, this directory attempts to marry the rich search-based query
language of Lucene with the distributed fault-tolerant database that is
Cassandra. By delegating the responsibilities of replication, durability and
elasticity to the directory, we free the layers above from such non-functional
concerns. Our hope is that users will choose to make their large-scale indices
instantly scalable by seamlessly migrating them to this type of directory
(using Directory#copyTo(Directory)).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
