On 7/10/2017 2:57 AM, Antonio De Miguel wrote: > I continue deeping inside this problem... high writing rates continues. > > Searching in logs i see this: > > 2017-07-10 08:46:18.888 INFO (commitScheduler-11-thread-1) [c:ads s:shard2 > r:core_node47 x:ads_shard2_replica3] o.a.s.u.LoggingInfoStream > [DWPT][commitScheduler-11-thread-1]: flushed: segment=_mb7 ramUsed=7.531 MB > newFlushedSize=2.472 MB docs/MB=334.132 > 2017-07-10 08:46:29.336 INFO (commitScheduler-11-thread-1) [c:ads s:shard2 > r:core_node47 x:ads_shard2_replica3] o.a.s.u.LoggingInfoStream > [DWPT][commitScheduler-11-thread-1]: flushed: segment=_mba ramUsed=8.079 MB > newFlushedSize=1.784 MB docs/MB=244.978 > > > A flush happens each 10 seconds (my autosoftcommit time is 10 secs and > hardcommit 5 minutes). ¿is the expected behaviour?
If you are indexing continuously, then the auto soft commit time of 10 seconds means that this will be happening every ten seconds. > I thought soft commits does not write into disk... If you are using the correct DirectoryFactory type, a soft commit has the *possibility* of not writing to disk, but the amount of memory reserved is fairly small. Looking into the source code for NRTCachingDirectoryFactory, I see that maxMergeSizeMB defaults to 4, and maxCachedMB defaults to 48. This is a little bit different than what the javadoc states for NRTCachingDirectory (5 and 60): http://lucene.apache.org/core/6_6_0/core/org/apache/lucene/store/NRTCachingDirectory.html The way I read this, assuming the amount of segment data created is small, only the first few soft commits will be entirely handled in memory. After that, older segments must be flushed to disk to make room for new ones. If the indexing rate is high, there's not really much difference between soft commits and hard commits. This also assumes that you have left the directory at the default of NRTCachingDirectoryFactory. If this has been changed, then there is no caching in RAM, and soft commit probably behaves *exactly* the same as hard commit. Thanks, Shawn
