[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498652 ] Michael Busch commented on LUCENE-888: -- > At 1 KB (current Lucene trunk) it takes 622 sec to build the index; if

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498696 ] Michael McCandless commented on LUCENE-888: --- > > I'm guessing we should leave BufferedIndexInput's default

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498714 ] Michael Busch commented on LUCENE-888: -- > I wonder if we should just add a ctor to BufferedIndexInput that takes

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498739 ] Michael McCandless commented on LUCENE-888: --- > > I wonder if we should just add a ctor to BufferedIndexInpu

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498757 ] Michael Busch commented on LUCENE-888: -- > Actually, it does have a default public constructor right? Ie if we ad

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498776 ] Michael McCandless commented on LUCENE-888: --- > > I plan to add "private int bufferSize" to BufferedIndexInp

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498780 ] Michael Busch commented on LUCENE-888: -- > OK I will do this. Actually, I think we should also allow making the >

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498784 ] Michael McCandless commented on LUCENE-888: --- > > OK I will do this. Actually, I think we should also allow

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498787 ] Marvin Humphrey commented on LUCENE-888: I would like to know why these gains are appearing, and how specific

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498793 ] Michael Busch commented on LUCENE-888: -- > I'm actually coding it up now. Why don't you commit LUCENE-430 > sooni

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499018 ] Michael McCandless commented on LUCENE-888: --- OK I ran two sets of tests. First is only on Mac OS X to see

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499020 ] Michael McCandless commented on LUCENE-888: --- > I would like to know why these gains are appearing, and how

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499044 ] John Haxby commented on LUCENE-888: --- > Net/net it's between 10-18% performance gain overall. It is > interesting th

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499182 ] Michael Busch commented on LUCENE-888: -- Mike, I tested and reviewed your patch. It looks good and all tests pas

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499199 ] Michael McCandless commented on LUCENE-888: --- > I tested and reviewed your patch. It looks good and all test

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499206 ] Michael Busch commented on LUCENE-888: -- > I'm a little nervous about that: I don't know the impact it will have

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499209 ] Michael McCandless commented on LUCENE-888: --- > > I'm a little nervous about that: I don't know the impact i

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499212 ] Doug Cutting commented on LUCENE-888: - > then we don't save IO by limiting the buffer size to 1 KB I'm confused

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499214 ] robert engels commented on LUCENE-888: -- I think the important consideration is how expensive is the system call.

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499221 ] Michael Busch commented on LUCENE-888: -- > So, back to your statement, a 1k buffer doesn't save > physical i/o,

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499223 ] Michael Busch commented on LUCENE-888: -- Mike, another thing I just noticed is your new testcase doesn't remove

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499224 ] Eks Dev commented on LUCENE-888: we did some time ago a few tests and simply concluded, it boils down to what Doug s

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499225 ] Michael McCandless commented on LUCENE-888: --- > another thing I just noticed is your new testcase doesn't re

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499226 ] Michael McCandless commented on LUCENE-888: --- > we did some time ago a few tests and simply concluded, it b

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499238 ] Michael Busch commented on LUCENE-888: -- Take2 looks good. +1 for committing. > Improve indexing performance by

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499280 ] Marvin Humphrey commented on LUCENE-888: I have some auxiliary data points to report after experimenting with

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499299 ] Michael McCandless commented on LUCENE-888: --- Marvin, it's odd that you see no gains when coming straight fr

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499307 ] Marvin Humphrey commented on LUCENE-888: > Also, how much "merging" is actually done in your test / KS? How

[jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

[ https://issues.apache.org/jira/browse/LUCENE-888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499548 ] Michael McCandless commented on LUCENE-888: --- I re-ran the "second test" above, but this time with compound

Re: [jira] Commented: (LUCENE-888) Improve indexing performance by increasing internal buffer sizes

Sorry, have not noticed LUCENE-893... it was Searching, indexing was slightly above 20% Interesting! Do you remember if your 5-8% gain was for searching or indexing? This issue is focusing on buffer size impact on indexing performance and LUCENE-893 is focusing on search performance.