Hi all,
we're using solr in our application and have problem that both searching
and indexing is slowing down over time.
Versions:
---------
Java 1.7
Solr 4.7.2
Lucene 4.1 (luceneMatchVersion param in solrconfig.xml)
App architecture:
-----------------
We don't use solr as standalone application, but we integrated him with
solrj library into our app.
It has 1 CoreContainer with 3 cores (let's call them alpha, beta,
gamma). EmbeddedSolrServer is used as java class to work with each core.
Application does both searching and indexing.
What's going on:
----------------
After Tomcat restart everything is working great for 2-3 days. But after
this time solr (all cores) starts sloooow down until it's unusable and
we have to restart Tomcat. Then it works fine.
For example search time for really complex query is 1,5 s when it works
fine. But then it rises to more than 1 min. Same issue is with indexing
- first fast but then slow.
Searching:
----------
Core apha is used mainly for normal search. But sometimes for faceting too.
Beta and gamma are only for facets.
alpha: 25000 queries/day
beta: 7000 queries/day
gamma: 7000 queries/day
We do lots of query joins, sometimes cross cores.
Indexing:
---------
We commit changes continuously over day. Number of commits is limited to
1 commit/min for all three cores. So we do max 1440 commits daily. One
commit contains between 1 and 100 docs.
Method EmbeddedSolrServer.add(SolrInputDocument) is used and in the end
EmbeddedSolrServer.commit().
Every night we call EmbeddedSolrServer.optimize() on each core.
Index size:
-----------
alpha: 13,5 GB
beta: 300 MB
gamma: 600 MB
Hardware:
---------
Ubuntu 14.04
8 core CPU
java heap space 22 GB RAM
SSD drive with more than 50 GB free space
Solr config (Same configuration is used for all cores):
-------------------------------------------------------
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.MMapDirectoryFactory}"/>
<luceneMatchVersion>LUCENE_41</luceneMatchVersion>
<indexConfig>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<ramBufferSizeMB>32</ramBufferSizeMB>
<maxFieldLength>10000</maxFieldLength>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
<lockType>native</lockType>
<unlockOnStartup>false</unlockOnStartup>
<reopenReaders>true</reopenReaders>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="maxCommitsToKeep">1</str>
<str name="maxOptimizedCommitsToKeep">0</str>
</deletionPolicy>
</indexConfig>
<updateHandler class="solr.DirectUpdateHandler2" />
<filterCache class="solr.FastLRUCache" size="512" initialSize="512"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache" size="512" initialSize="512"
autowarmCount="0"/>
<documentCache class="solr.LRUCache" size="512" initialSize="512"
autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>2</maxWarmingSearchers>
Conclusion:
-----------
Is something wrong in configuration? Or is this some kind of bug? Or...?
Can you give me some advice how to resolve this problem please?
Roman
