On Thu, Sep 8, 2011 at 2:54 PM, Yonik Seeley <[email protected]> wrote: > On Thu, Sep 8, 2011 at 5:35 AM, Simon Willnauer > <[email protected]> wrote: >> I don't think this needs to be a core feature at all but I think we need >> to provide the necessary hooks in Lucene core to make this reliable >> and consistent. > > I've thought about it a little - it would be really helpful if a > sequence id were returned from indexwriter operations. > > public void addDocument() -> public long addDocument() > public void deleteDocuments() -> public long deleteDocuments() > public void commit() -> public long commit() > > And of course the id returned by commit would mean that everything > less than that id would be in the commit.
I actually implemented a prototype that does something similar to this. The problem here is some impl detail we have in DWPT with cutting over to a new DelQueue on full flush but I think we can fix that without too much trouble. Another option would be a onDoc(Document, long queueGen, long seqID) / onDelete(...) callback in DW that is passed a doc / delete. This would actually be trivial to implement with what we have currently, yet not ideal. I still wonder how we would resolve the order in the log since you might get seq. ids out of order. Another idea would be writing a log per DWPT which would be continuous in terms of seq. ids. Those logs can be merged sequentially on recovery and we would get concurrent writes for free. Just an idea though. simon > > -Yonik > http://www.lucene-eurocon.com - The Lucene/Solr User Conference > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
