A new approach for main-memory database
---------------------------------------
Key: DERBY-2798
URL: https://issues.apache.org/jira/browse/DERBY-2798
Project: Derby
Issue Type: New Feature
Components: Store
Affects Versions: 10.2.2.0
Environment: all
Reporter: Knut Magne Solem
Fix For: 10.0.2.2
Attachments: Derby-10.2.2.0-memstore.diff
As a part of my Masters degree I have created an extension that allows data to
reside in memory without the need to serialize it to Page-objects. This is a
pretty big chunk of code and is sort of a proof of concept of another way to
make an in-memory storage mode.
I created two new conglomerates, called MemHeap and MemSkiplist. Derby
interfaces them the same way as it does with Heap and BTree. These new
conglomerates use RawStore for transaction support and logging, but not for
storage. Instead it uses a new system service I've called MemStore. This data
store only stores pointers to Slot-objects organized in arrays corresponding to
its container/conglomerate/table. A Slot-object consists mainly of a
DataValueDescriptor[]-object representing a row in a table.
So, instead of just doing dummy-IO in memory where Derby still thinks its doing
real IO and page caching, this new approach bypasses the cache and
page-structure by keeping the DataValueDescriptor[]-objects in memory without
serializing them.
Manipulating operations on data in memory are done via new operation-objects
(ex. MemInsertOperation, MemInsertUndoOperation, MemDeleteOperation...) with
still uses RawStore for transaction control and persistence. Checkpointing is
done by serializing the objects in MemStore fuzzily and completely
unsynchronized to disk. Recovery consists of de-serializing the objects to
MemStore before the existing REDO- and UNDO-phase of Derby recovery in RawStore
will get the data transaction-consistent by replaying or undo the new
operation-objects in the log.
Locking is hard coded as row based with locking degree SERIALIZABLE.
To get Derby to use the new conglomerates I hacked the SQL-layer to create
MemHeap-tables and MemSkiplist-indexes when the table name starts with 'mem_'.
Because this is a major rewrite of the access- and storage-layer there is a lot
of known and unknown bugs and missing functionality. What is working is
essentially select, insert, update and delete on tables with one primary key.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
