I would like to discuss an additional approach, that requires small changes
to current Lucene implementation. Here, the index version (currently in
segments file) is maintained in a separate file, and is used to synchronize
between readers and writers, without requiring readers to create/obtain any
lock files, and without requiring readers to write anything to disk.
- Index version would be maintained in a separate, dedicated Version file -
(say .vsn) - one per index.
- Version file contains two occurrences of the version number - V1 and V2.
- In steady state, V1 == V2, but During update V1 == V2+1.
- Every commit would:
- obtain a write lock (as today), to guarantee single writer at a time.
- increments V1 in that file, using RandomAccessFile API (RAF).
notice: now V1 != V2.
- do the commit work (merge, delete, whatever).
- increments V2 in that file, using RAF.
notice: now, again, V1 == V2.
- release the write lock (as today)
- Every reader would read the version data in opposite order:
(1) read V2 from the version file, using RAF.
(2) read V1 from the version file, using RAF.
(3) if not V1==V2 wait some time, and try again (from step 1), until
V1==V2, or timeout and fail.
(4) initialize reader data (read segment infos, open files).
(5) read again V2 then V1 using RAF.
(6) if not V2==V1 or they changed from steps 1 and 2, try again (from
step 1), or timeout and fail.
A few points to notice:
- Using RAF protects from errors due to IO buffering.
- Only tiny amount of version data is being read/written using RAF, so
performance should not degrade.
- Readers are not writing any data, so they are faster (A reader that does
deleteDoc is a writer in this regard).
- The opposite read/write order of RAF operations, i.e. writing V1 and then
V2 by writer but reading V2 and then V1 by reader, protects from race
conditions between readers and writers that otherwise might have caused
reading corrupt data and concluding wrongly that the data is consistent
while in fact it is not.
- By using RAF and by the order of operations above, this scheme would work
also for NFS (excluding the write lock mechanism which remains an issue in
NFS).
- For backward compatibility with current index structure the code can fall
back to obtain a commit lock file if the .vsn file does not exist, and then
create that .vsn file if it still does not exist.
Regards,
Doron
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
