Re: [jira] Created: (LUCENE-1172) Small speedups to DocumentsWriter

Curious... on things like this, is it really worth adding (and maintaining) Lucene's own sort, just to achieve a 1.5 % performance increase. It is almost doubtful that you can even measure an improvement at that level, given all of the variables you can't control. I see a LOT of code in Luc

[jira] Updated: (LUCENE-1172) Small speedups to DocumentsWriter

[ https://issues.apache.org/jira/browse/LUCENE-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1172: --- Attachment: LUCENE-1172.patch > Small speedups to DocumentsWriter >

[jira] Created: (LUCENE-1172) Small speedups to DocumentsWriter

Small speedups to DocumentsWriter - Key: LUCENE-1172 URL: https://issues.apache.org/jira/browse/LUCENE-1172 Project: Lucene - Java Issue Type: Improvement Components: Index Affects Versions: 2.3

[jira] Commented: (LUCENE-1157) Formatable changes log (CHANGES.txt is easy to edit but not so friendly to read by Lucene users)

[ https://issues.apache.org/jira/browse/LUCENE-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567236#action_12567236 ] Steven Rowe commented on LUCENE-1157: - Excellent, the link from the Developer Resource

[jira] Updated: (LUCENE-1171) Make DocumentsWriter more robust on hitting OOM

[ https://issues.apache.org/jira/browse/LUCENE-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1171: --- Attachment: LUCENE-1171.patch Attached patch. > Make DocumentsWriter more robust on

[jira] Created: (LUCENE-1171) Make DocumentsWriter more robust on hitting OOM

Make DocumentsWriter more robust on hitting OOM --- Key: LUCENE-1171 URL: https://issues.apache.org/jira/browse/LUCENE-1171 Project: Lucene - Java Issue Type: Bug Components: Index Af

[jira] Updated: (LUCENE-1044) Behavior on hard power shutdown

[ https://issues.apache.org/jira/browse/LUCENE-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1044: --- Attachment: LUCENE-1044.take8.patch Attached new rev of the patch. Only changes wer

Re: detected corrupted index / performance improvement

Doug Cutting wrote: The linux kernel dynamically increases the readahead window based on the access pattern: the more you read sequentially, the larger the readahead window. Sorry, it appears that's in 2.6.23, which isn't yet broadly used. http://kernelnewbies.org/Linux_2_6_23#head-102af26593

Re: detected corrupted index / performance improvement

robert engels wrote: But that would mean we should be using at least 250k buffers for the IndexInput ? Not the 16k or so that is the default. Is the OS smart enough to figure out that the file is being sequentially read, and adjust its physical read size to 256k, based on the other concurrent

[jira] Commented: (LUCENE-1170) query with AND and OR not retrieving correct results

[ https://issues.apache.org/jira/browse/LUCENE-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567154#action_12567154 ] Daniel Naber commented on LUCENE-1170: -- It's a known problem with QueryParser, see e.

Re: detected corrupted index / performance improvement

But that would mean we should be using at least 250k buffers for the IndexInput ? Not the 16k or so that is the default. Is the OS smart enough to figure out that the file is being sequentially read, and adjust its physical read size to 256k, based on the other concurrent IO operations. See

[jira] Commented: (LUCENE-1169) Search with Filter does not work!

[ https://issues.apache.org/jira/browse/LUCENE-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567115#action_12567115 ] Doug Cutting commented on LUCENE-1169: -- > iterator.skipTo(iterator.doc()) <=> iterato

Re: detected corrupted index / performance improvement

Michael McCandless wrote: Merging is far more IO intensive. With mergeFactor=10, we read from 40 input streams and write to 4 output streams when merging the tii/tis/frq/prx files. If your disk can transfer at 50MB/s, and takes 5ms/seek, then 250kB reads and writes are the break-even point, w

[jira] Created: (LUCENE-1170) query with AND and OR not retrieving correct results

query with AND and OR not retrieving correct results Key: LUCENE-1170 URL: https://issues.apache.org/jira/browse/LUCENE-1170 Project: Lucene - Java Issue Type: Bug Components: Qu

[jira] Resolved: (LUCENE-1164) Improve how ConcurrentMergeScheduler handles too-many-merges case

[ https://issues.apache.org/jira/browse/LUCENE-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-1164. Resolution: Fixed > Improve how ConcurrentMergeScheduler handles too-many-merges c

Re: detected corrupted index / performance improvement

Mike, you're right: all lucene files are written sequentially (flushing or merging). It's just a matter of how many are open at once, and whether we are also reading from source(s) files, which affects IO throughput far less than truly random access writes. Plus, as of LUCENE-843, bytes are wri

[jira] Commented: (LUCENE-1169) Search with Filter does not work!

[ https://issues.apache.org/jira/browse/LUCENE-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12566971#action_12566971 ] Eks Dev commented on LUCENE-1169: - Thank you for fixing it in no time :) But... I am gett

[jira] Commented: (LUCENE-1145) DisjunctionSumScorer small tweak

[ https://issues.apache.org/jira/browse/LUCENE-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12566961#action_12566961 ] Eks Dev commented on LUCENE-1145: - test using Sun 1.4 jvm on the same hardware showed the