[ http://issues.apache.org/jira/browse/LUCENE-738?page=all ]
Doron Cohen updated LUCENE-738: ------------------------------- Attachment: FileFormatDoc.patch.txt FileFormat document updated to reflect this format change. > read/write .del as d-gaps when the deleted bit vector is sufficiently sparse > ---------------------------------------------------------------------------- > > Key: LUCENE-738 > URL: http://issues.apache.org/jira/browse/LUCENE-738 > Project: Lucene - Java > Issue Type: Improvement > Components: Store > Affects Versions: 2.1 > Reporter: Doron Cohen > Assigned To: Doron Cohen > Attachments: del.dgap.patch.txt, FileFormatDoc.patch.txt > > > .del file of a segment maintains info on deleted documents in that segment. > The file exists only for segments having deleted docs, so it does not exists > for newly created segments (e.g. resulted from merge). Each time closing an > index reader that deleted any document, the .del file is rewritten. In fact, > since the lock-less commits change a new (generation of) .del file is created > in each such occasion. > For small indexes there is no real problem with current situation. But for > very large indexes, each time such an index reader is closed, creating such > new bit-vector seems like unnecessary overhead in cases that the bit vector > is sparse (just a few docs were deleted). For instance, for an index with a > segment of 1M docs, the sequence: {open reader; delete 1 doc from that > segment; close reader;} would write a file of ~128KB. Repeat this sequence 8 > times: 8 new files of total size of 1MB are written to disk. > Whether this is a bottleneck or not depends on the application deletes > pattern, but for the case that deleted docs are sparse, writing just the > d-gaps would save space and time. > I have this (simple) change to BitVector running and currently trying some > performance tests to, yet, convince myself on the worthiness of this. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]