Hi, thanks for your response! We use javamelody for some basic statistics about app. Here are some graphs from last 24 hours: http://imgur.com/a/OQxnb
First graph is memory used by application. Second graph shows how seach time rapidly increased. At 13:40 there was neccessary app restart and at 1:30 index optimize job started. Everything has been fine today, no restart needed. Based on these graphs, I don't think problem is tight memory. Can you explain that point with autoCommit? Now we create list of SolrInputDocuments, add them to EmbeddedSolrServer and then call explicit hard commit EmbeddedSolrServer.commit() with both waitFlush and waitSearcher set to true (could be this that problem?). So no autoCommit is used. Main reason for it is that after somebody does some document modifications, we insert document's ID into database. After some time all these documents are reindexed, commited and removed from database. So everything is persistent - modified documents are in database or already commited. Roman 2016-05-18 23:24 GMT+02:00 Joel Bernstein <joels...@gmail.com>: > One thing to investigate is whether your caches are too large and gradually > filling up memory. It does sound like memory is getting tighter over time. > A memory profiler would be helpful in figuring out memory issues. > > Moving to autoCommits would also eliminate any slowness due to overlapping > searchers from committing too frequently. > > > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Wed, May 18, 2016 at 10:39 AM, Roman Slavík <slaav...@gmail.com> wrote: > > > Hi all, > > > > we're using solr in our application and have problem that both searching > > and indexing is slowing down over time. > > > > Versions: > > --------- > > Java 1.7 > > Solr 4.7.2 > > Lucene 4.1 (luceneMatchVersion param in solrconfig.xml) > > > > App architecture: > > ----------------- > > We don't use solr as standalone application, but we integrated him with > > solrj library into our app. > > It has 1 CoreContainer with 3 cores (let's call them alpha, beta, gamma). > > EmbeddedSolrServer is used as java class to work with each core. > > Application does both searching and indexing. > > > > What's going on: > > ---------------- > > After Tomcat restart everything is working great for 2-3 days. But after > > this time solr (all cores) starts sloooow down until it's unusable and we > > have to restart Tomcat. Then it works fine. > > For example search time for really complex query is 1,5 s when it works > > fine. But then it rises to more than 1 min. Same issue is with indexing - > > first fast but then slow. > > > > Searching: > > ---------- > > Core apha is used mainly for normal search. But sometimes for faceting > too. > > Beta and gamma are only for facets. > > alpha: 25000 queries/day > > beta: 7000 queries/day > > gamma: 7000 queries/day > > We do lots of query joins, sometimes cross cores. > > > > Indexing: > > --------- > > We commit changes continuously over day. Number of commits is limited to > 1 > > commit/min for all three cores. So we do max 1440 commits daily. One > commit > > contains between 1 and 100 docs. > > Method EmbeddedSolrServer.add(SolrInputDocument) is used and in the end > > EmbeddedSolrServer.commit(). > > Every night we call EmbeddedSolrServer.optimize() on each core. > > > > Index size: > > ----------- > > alpha: 13,5 GB > > beta: 300 MB > > gamma: 600 MB > > > > Hardware: > > --------- > > Ubuntu 14.04 > > 8 core CPU > > java heap space 22 GB RAM > > SSD drive with more than 50 GB free space > > > > Solr config (Same configuration is used for all cores): > > ------------------------------------------------------- > > <directoryFactory name="DirectoryFactory" > > class="${solr.directoryFactory:solr.MMapDirectoryFactory}"/> > > <luceneMatchVersion>LUCENE_41</luceneMatchVersion> > > <indexConfig> > > <useCompoundFile>false</useCompoundFile> > > <mergeFactor>10</mergeFactor> > > > > <ramBufferSizeMB>32</ramBufferSizeMB> > > <maxFieldLength>10000</maxFieldLength> > > <writeLockTimeout>1000</writeLockTimeout> > > <commitLockTimeout>10000</commitLockTimeout> > > > > <lockType>native</lockType> > > <unlockOnStartup>false</unlockOnStartup> > > <reopenReaders>true</reopenReaders> > > > > <deletionPolicy class="solr.SolrDeletionPolicy"> > > <str name="maxCommitsToKeep">1</str> > > <str name="maxOptimizedCommitsToKeep">0</str> > > </deletionPolicy> > > </indexConfig> > > > > <updateHandler class="solr.DirectUpdateHandler2" /> > > > > <filterCache class="solr.FastLRUCache" size="512" initialSize="512" > > autowarmCount="0"/> > > <queryResultCache class="solr.LRUCache" size="512" initialSize="512" > > autowarmCount="0"/> > > <documentCache class="solr.LRUCache" size="512" initialSize="512" > > autowarmCount="0"/> > > > > <enableLazyFieldLoading>true</enableLazyFieldLoading> > > <useColdSearcher>false</useColdSearcher> > > <maxWarmingSearchers>2</maxWarmingSearchers> > > > > > > Conclusion: > > ----------- > > Is something wrong in configuration? Or is this some kind of bug? Or...? > > Can you give me some advice how to resolve this problem please? > > > > > > Roman > > > > > > > > > > > > > > > > > > > > > > >
