Re: [jira] Commented: (LUCENE-1483) Change IndexSearcher multisegment searches to search each individual segment using a single HitCollector

Mon, 19 Jan 2009 02:38:24 -0800
I'm also seeing decent gains (~13%) for sort-by-relevance (ie the default sort) term queries w/ large number (~97K and ~386K) of hits on 10 & 36 segment indices. 


So I agree, LUCENE-1483 is not just about speeding up sort-by-field  
queries.  It seems to give good speedups all around, and of course  
warming time for sort-by-field searches goes way way down.


We just gotta wrap it up now!

Mike

Mark Miller wrote:


One more just for a check with much fewer unique terms (20k). Didn't  
catch that I didnt clamp down enough on the uniques last one. Back  
up to 21 segments this time, same wildcard search, 7718 hits, and  
the new method is still approx 20% faster than the old. The last run  
was 16 segments though with way more uniques - this one is 21  
segments and way fewer uniques.



7718

Segments file=segments_l numSegments=21 version=FORMAT_USER_DATA  
[Lucene 2.9]

1 of 21: name=_bbxo docCount=29349
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.92
  docStoreOffset=0
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3875263 terms/docs  
pairs; 4516618 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


2 of 21: name=_bbxp docCount=29459
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.982
  docStoreOffset=29349
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3895590 terms/docs  
pairs; 4540859 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


3 of 21: name=_bbxq docCount=29300
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.97
  docStoreOffset=58808
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3890419 terms/docs  
pairs; 4536052 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


4 of 21: name=_bbxr docCount=29480
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.971
  docStoreOffset=88108
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3894211 terms/docs  
pairs; 4538397 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


5 of 21: name=_bbxs docCount=29470
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.979
  docStoreOffset=117588
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3895226 terms/docs  
pairs; 4540446 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


6 of 21: name=_bbxt docCount=29450
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.98
  docStoreOffset=147058
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3892708 terms/docs  
pairs; 4538338 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


7 of 21: name=_bbxu docCount=29509
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.978
  docStoreOffset=176508
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3894189 terms/docs  
pairs; 4538376 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


8 of 21: name=_bbxv docCount=29401
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.976
  docStoreOffset=206017
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3891986 terms/docs  
pairs; 4538746 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


9 of 21: name=_bbxw docCount=29476
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.988
  docStoreOffset=235418
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3899576 terms/docs  
pairs; 4545210 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


10 of 21: name=_bbxx docCount=29254
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.97
  docStoreOffset=264894
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3888340 terms/docs  
pairs; 4534783 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


11 of 21: name=_bbxy docCount=29537
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.977
  docStoreOffset=294148
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3894706 terms/docs  
pairs; 4538709 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


12 of 21: name=_bbxz docCount=29470
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.982
  docStoreOffset=323685
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3895585 terms/docs  
pairs; 4540100 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


13 of 21: name=_bby0 docCount=29229
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.977
  docStoreOffset=353155
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3891056 terms/docs  
pairs; 4538105 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


14 of 21: name=_bby1 docCount=29259
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.98
  docStoreOffset=382384
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3890415 terms/docs  
pairs; 4537438 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


15 of 21: name=_bby2 docCount=29467
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.981
  docStoreOffset=411643
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3895733 terms/docs  
pairs; 4539617 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


16 of 21: name=_bby3 docCount=29393
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.964
  docStoreOffset=441110
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3888770 terms/docs  
pairs; 4533987 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


17 of 21: name=_bby4 docCount=29342
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.979
  docStoreOffset=470503
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3893658 terms/docs  
pairs; 4540029 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


18 of 21: name=_bby5 docCount=29528
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.973
  docStoreOffset=499845
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3895020 terms/docs  
pairs; 4539289 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


19 of 21: name=_bby6 docCount=29397
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.972
  docStoreOffset=529373
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3893087 terms/docs  
pairs; 4536990 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


20 of 21: name=_bby7 docCount=29339
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=11.979
  docStoreOffset=558770
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 3891393 terms/docs  
pairs; 4537015 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


21 of 21: name=_bby8 docCount=11891
  compound=true
  hasProx=true
  numFiles=2
  size (MB)=4.886
  docStoreOffset=588109
  docStoreSegment=_bbxo
  docStoreIsCompoundFile=true
  no deletions
  test: open reader.........OK
  test: fields, norms.......OK [1 fields]

  test: terms, freq, prox...OK [21840 terms; 1564547 terms/docs  
pairs; 1822780 tokens]
  test: stored fields.......OK [0 total field count; avg 0 fields  
per doc]
  test: term vectors........OK [0 total vector count; avg 0 term/ 
freq vector fields per doc]


No problems were detected with this index.



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
























					Reply via email to