DONE. > -----Original Message----- > From: Michael McCandless [mailto:[email protected]] > Sent: Monday, July 13, 2009 11:53 AM > To: [email protected] > Subject: Re: OOM with 2.9 > > Ahh good point. I agree it makes sense to make MMapDir's chunking > user-controllable. Can you open an issue? > > Mike > > On Mon, Jul 13, 2009 at 5:49 AM, Uwe Schindler<[email protected]> wrote: > >> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<[email protected]> wrote: > >> > >> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow... > >> maybe for such cases this threshold can become user settable. I will > try > >> to experiment with it (I am talking about MMapDirectory -> private > final > >> int MAX_BBUF = Integer.MAX_VALUE;) > >> > >> This chunking doesn't reduce virtual memory usage, because the buffers > >> are all mapped up front on opening the IndexInput. > >> > >> It's only done because of a limitation of ByteBuffer (it uses int not > >> long to address the values). > > > > The idea of chunking is because of address space fragmentation. If the > Java > > VM allocates or frees memory (e.g. when expanding the heap from Xms to > Xmx), > > the address space of the process gets fragmented. If the index is very > > large, the byte buffer must fit in complete as one fragment into the > address > > space, which may easily fail when you only have 32 bit. > > > > If the file is mapped in different chunks, each chunk can be somewhere > else > > in address space (e.g., not only one 1 Gig junk is needed, there can be > ten > > 100 Meg junks at different places). > > > > Uwe > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected]
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
