Thanks, Shawn.
> After a look at the code, I found that StandardDirectoryFactory should
> use MMap if the OS and Java version support it. If support isn't there,
> it will use conventional file access methods. As far as I know, all
> 64-bit Java versions and 64-bit operating systems will support MMap.
Considering our JVM is 64-bit, that probably explains why we're
experiencing MMapDirectory like behaviour on our cluster (i.e. high
non-JVM related memory use).
As to NRTCachingDirectoryFactory, when looking up the docs we were in
doubt about what it means to have a "highish reopen rate".
> public class NRTCachingDirectory
> This class is likely only useful in a near-real-time context, where indexing
> rate is lowish but reopen rate is highish, > resulting in many tiny files
> being written.
Can we read "high reopen rate" as "frequent soft commits"? (In our
case, hard commits do not open a searcher. But soft commits do).
Considering it does mean "frequent soft commits", I'd say that it
doesn't fit our setup because we have an index rate of about 10
updates/s and we perform a soft commit at each 15min. So our scenario
is not near real time in that sense. In light of this, do you thing
using NRTCachingDirectory is still convenient?
Best,
Eric
On Wed, Oct 7, 2015 at 12:08 PM, Shawn Heisey <apa...@elyograg.org> wrote:
> On 10/7/2015 8:48 AM, Eric Torti wrote:
>> <directoryFactory
>> class="${solr.directoryFactory:solr.StandardDirectoryFactory}"
>> name="DirectoryFactory"/>
>>
>> I'm just starting to grasp different strategies for Directory
>> implementation. Can I assume that solr.StandardDirectoryFactory is a
>> MMapDirectory as described by Uwe Schindler in this post about the use
>> of virtual memory?
>> [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
>
> After a look at the code, I found that StandardDirectoryFactory should
> use MMap if the OS and Java version support it. If support isn't there,
> it will use conventional file access methods. As far as I know, all
> 64-bit Java versions and 64-bit operating systems will support MMap.
>
> The factory you *should* be using is NRTCachingDirectoryFactory, and you
> should enable the updateLog to ensure data reliability.
>
> Thanks,
> Shawn
>
