Shalin, yes I think it's a case of the Suggester build hitting the index all at once. I'm thinking it's hitting all docs, even the ones without fields relevant to the suggester.
Shawn, I am using ZFS, though I think it's comparable to other setups. mmap() should still be faster, while the ZFS ARC cache may prefer more memory that other OS disk caches. So, it sounds like I enough memory/swap to hold the entire index. When will the memory be released? On a commit? https://lucene.apache.org/core/6_5_0/core/org/apache/lucene/store/MMapDirectory.html talks about a bug on the close(). On 2 May 2017 at 23:07, Shawn Heisey <apa...@elyograg.org> wrote: > On 5/1/2017 10:52 PM, Damien Kamerman wrote: > > I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each > > replica uses about 14GB disk usage. I'm using Solaris 11 and I see the > > 'Page cache' grow by about 7GB for each suggester replica I build. The > > suggester index itself is very small. The 'Page cache' memory is freed > > when the node is stopped. I guess the Suggester component is mmap'ing > > the entire Lucene index into memory and holding it? Is this expected > > behavior? Is there a workaround? > > I found the following. The last comment on the answer, the one about > mmap causing double-buffering with ZFS, is possibly relevant: > > https://serverfault.com/a/270604 > > What filesystem are your indexes on? If it's ZFS, it could completely > explain the behavior. If it's not ZFS, then the only part of it that I > cannot explain is the fact that the page cache is freed when Solr stops. > > If this double-buffering actually means that the memory is allocated > twice, then I think that ZFS is probably the wrong filesystem to run > Solr on, unless you have a LOT of spare memory. You could try changing > the directory factory to one that doesn't use MMAP, but the suggester > index factory probably cannot be easily changed. This is too bad -- > normally MMAP is far more efficient than "standard" filesystem access. > > I could be reaching completely wrong conclusions based on the limited > research I did. > > Thanks, > Shawn > >
