Hi,
I am finding that lucene is slowing down a lot when bigger and bigger
doc/pos files are merged... While it's normally the case, the worrying part
is all my data is in RAM. Version is 4.6.1
Some sample statistics took after instrumenting the SortingAtomicReader
code, as we use a SortingMergePolicy. The times displayed are just for
reading {ex: in.nextDoc(), in.nextPosition()}. It does not include
tim-sorting or new-segment writing times
*337 sec* to merge postings [*281655 docs*] with
*SortingDocsAndPositionEnum-nextPosition()* as [*130sec*] and *Sorting*
*DocsAndPositionEnum-nextDoc()* as [*232sec*] and total-num-terms as [
*2,058,600*]
*482 sec* to merge postings [*475143 docs*] with *Sorting*
*DocsAndPositionEnum-nextPosition()* as [*204sec*] and *Sorting*
*DocsAndPositionEnum-nextDoc()* as [*332sec*] and total-num-terms as [
*3,791,065*]
*898 sec* to merge postings [*890385 docs*] with *Sorting*
*DocsAndPositionEnum-nextPosition()* as [*343sec*] and *Sorting*
*DocsAndPositionEnum-nextDoc()* as [*609sec*] and total-num-terms as [
*5,470,110*]
*1000 sec* to merge postings [*950084 docs*] with *Sorting*
*DocsAndPositionEnum-nextPosition()* as [*361sec*] and *Sorting*
*DocsAndPositionEnum-nextDoc()* as [*686sec*] and total-num-terms as [
*1,108,744*]
I went ahead and did an "mlock" on already mmapped doc/pos files and then
proceeded for merge, to eliminate disk. The numbers shown above come for
iterating all terms/docs/positions sequentially from RAM!!
I understand that there are no bulk-merge of postings currently available,
but given that data is in RAM, doesn't it indicate a slow-down? Is there
some configuration I am missing etc... to speed this up?
--
Ravi
[P.S: I have not verified whether all pages reside in RAM, but "mlock"
doesn't throw any Exceptions and returns success...]
