Good point. I'll set the heap size larger. It will be interesting to measure if there is a noticeable improvement from 2GB RAM buffer size vs say 100MB or 512MB ... I'll report the results here when I have them
But in the meantime, I think, unless it makes absolutely no sense and does not improve anything, that we should not limit to 2GB just because we use an int somewhere. Shai On Mon, Apr 26, 2010 at 5:18 PM, Mark Miller <markrmil...@gmail.com> wrote: > On 4/26/10 10:06 AM, Shai Erera wrote: > >> Hi >> >> Today we limit the RAM buffer size to 2048. I was wondering why, until >> Mark referenced me to that issue >> (https://issues.apache.org/jira/browse/LUCENE-1995) where someone >> complained about it as well, but I didn't find a reason to why it's >> limited. I mean, I understand it's probably an int is used somewhere, >> but all the issue says here >> ( >> https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757 >> < >> https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757 >> >) >> is "Maybe someday Lucene will allow a larger RAM buffer than 2GB". >> >> Why "maybe"? :) >> >> I set up a benchmark run on a new machine I got, 4xQuad, 64GB RAM, and >> turns out I can't really take advantage of all that RAM ... :). >> >> So I don't mind to look at the code, but I guess since Mike and a bunch >> of other smarter people were on that issue, there is probably something >> I need to know before I set to change it? >> >> Appreciate your thoughts on that. Today, more and more machines come w/ >> more than 2GB ... >> >> Shai >> > > Keep in mind that you need a lot more than 2GB of ram to use a really large > RAM buffer: > http://search.lucidimagination.com/search/document/b18fbeaaf53ead6c/io_exception_during_merge_optimize > > 64 GB might cover it though ;) > > -- > - Mark > > http://www.lucidimagination.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
