Re: Suggester uses lots of 'Page cache' memory

2017-05-09 Thread Damien Kamerman
Memory/cache aside, the fundamental Solr issue is that the Suggester build operation will read the entire index, even though very few docs have the relevant fields. Is there a way to set a 'fq' on the Suggester build? java.lang.Thread.State: RUNNABLE at

Re: Suggester uses lots of 'Page cache' memory

2017-05-02 Thread Damien Kamerman
Thanks Shawn, I'll have to look closer into this. On 3 May 2017 at 12:10, Shawn Heisey wrote: > On 5/2/2017 6:46 PM, Damien Kamerman wrote: > > Shalin, yes I think it's a case of the Suggester build hitting the index > > all at once. I'm thinking it's hitting all docs, even

Re: Suggester uses lots of 'Page cache' memory

2017-05-02 Thread Shawn Heisey
On 5/2/2017 6:46 PM, Damien Kamerman wrote: > Shalin, yes I think it's a case of the Suggester build hitting the index > all at once. I'm thinking it's hitting all docs, even the ones without > fields relevant to the suggester. > > Shawn, I am using ZFS, though I think it's comparable to other

Re: Suggester uses lots of 'Page cache' memory

2017-05-02 Thread Damien Kamerman
Shalin, yes I think it's a case of the Suggester build hitting the index all at once. I'm thinking it's hitting all docs, even the ones without fields relevant to the suggester. Shawn, I am using ZFS, though I think it's comparable to other setups. mmap() should still be faster, while the ZFS ARC

Re: Suggester uses lots of 'Page cache' memory

2017-05-02 Thread Shawn Heisey
On 5/1/2017 10:52 PM, Damien Kamerman wrote: > I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each > replica uses about 14GB disk usage. I'm using Solaris 11 and I see the > 'Page cache' grow by about 7GB for each suggester replica I build. The > suggester index itself is very

Re: Suggester uses lots of 'Page cache' memory

2017-05-02 Thread Shalin Shekhar Mangar
On Tue, May 2, 2017 at 10:22 AM, Damien Kamerman wrote: > Hi all, > > I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each replica > uses about 14GB disk usage. I'm using Solaris 11 and I see the 'Page cache' > grow by about 7GB for each suggester replica I

Suggester uses lots of 'Page cache' memory

2017-05-01 Thread Damien Kamerman
Hi all, I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each replica uses about 14GB disk usage. I'm using Solaris 11 and I see the 'Page cache' grow by about 7GB for each suggester replica I build. The suggester index itself is very small. The 'Page cache' memory is freed when the