Hi Shawn, Unfortunately your reply of memory may not be valid. Please refer to my explanation below of the strange behaviors (is it much more like a BUG than anything else that is explainable):
Note that we still have 18GB of free unused memory on the server. 1. We indexed the first collection called customers (3.7 millioin records from CSV data), index size is 2.09GB. The search in customers for any keyword is returned within 50ms (QTime) for using highlight (unified highlighter, posting, light term vectors) 2. Then we indexed the second collection called policies (6 million records from CSV data), index size is 2.55GB. The search in policies for any keyword is returned within 50ms (QTime) for using highlight (unified highlighter, posting, light term vectors) 3. But now any search in customers for any keywords (not from cache) takes as high as 1200ms (QTime). But still policies search remains very fast (50ms). 4. So we decided to run the force optimize command on customers collection ( https://localhost:8983/edm/customers/update?optimize=true&numSegments=1&waitFlush=false), surprisingly after optimization the search on customers collection for any keywords become very fast again (less than 50ms). BUT strangely, the search in policies collection become very slow (around 1200ms) without any changes to the policies collection. 5. Based on above result, we decided to run the force optimize command on policies collection ( https://localhost:8983/edm/policies/update?optimize=true&numSegments=1&waitFlush=false). More surprisingly, after optimization the search on policies collection for any keywords become very fast again (less than 50ms). BUT more strangely, the search in customers collection again become very slow (around 1200ms) without any changes to the customers collection. What a strange and unexpected behavior! If this is not a bug, how could you explain the above very strange behavior in Solr 7.5. Could it be a bug? We would appreciate any support or help on our above situation. Thank you. Regards, Edwin On Thu, 24 Jan 2019 at 16:14, Zheng Lin Edwin Yeo <edwinye...@gmail.com> wrote: > Hi Shawn, > > > If the two collections have data on the same server(s), I can see this > > happening. More memory is consumed when there is additional data, and > > when Solr needs more memory, performance might be affected. The > > solution is generally to install more memory in the server. > > I have found that even after we delete the index in collection2, the query > QTime for collection1 still remains slow. It does not goes back to its > previous fast speed before we index collection2. > > Regards, > Edwin > > > On Thu, 24 Jan 2019 at 11:13, Zheng Lin Edwin Yeo <edwinye...@gmail.com> > wrote: > >> Hi Shawn, >> >> Thanks for your reply. >> >> The log only shows a list the following and I don't see any other logs >> besides these. >> >> 2019-01-24 02:47:57.925 INFO (qtp2131952342-1330) [c:collectioin1 >> s:shard1 r:core_node4 x:collection1_shard1_replica_n2] >> o.a.s.u.p.StatelessScriptUpdateProcessorFactory update-script#processAdd: >> id=13245417 >> 2019-01-24 02:47:57.957 INFO (qtp2131952342-1330) [c:collectioin1 >> s:shard1 r:core_node4 x:collection1_shard1_replica_n2] >> o.a.s.u.p.StatelessScriptUpdateProcessorFactory update-script#processAdd: >> id=13245430 >> 2019-01-24 02:47:57.957 INFO (qtp2131952342-1330) [c:collectioin1 >> s:shard1 r:core_node4 x:collection1_shard1_replica_n2] >> o.a.s.u.p.StatelessScriptUpdateProcessorFactory update-script#processAdd: >> id=13245435 >> >> There is no change to the segments info. but the slowdown in the first >> collection is very drastic. >> Before the indexing of collection2, the collection1 query QTime are in >> the range of 4 to 50 ms. However, after indexing collection2, the >> collection1 query QTime increases to more than 1000 ms. The index are done >> in CSV format, and the size of the index is 3GB. >> >> Regards, >> Edwin >> >> >> >> On Thu, 24 Jan 2019 at 01:09, Shawn Heisey <apa...@elyograg.org> wrote: >> >>> On 1/23/2019 10:01 AM, Zheng Lin Edwin Yeo wrote: >>> > I am using Solr 7.5.0, and currently I am facing an issue of when I am >>> > indexing in collection2, the indexing affects the records in >>> collection1. >>> > Although the records are still intact, it seems that the settings of >>> the >>> > termVecotrs get wipe out, and the index size of collection1 reduced >>> from >>> > 3.3GB to 2.1GB after I do the indexing in collection2. >>> >>> This should not be possible. Indexing in one collection should have >>> absolutely no effect on another collection. >>> >>> If logging has been left at its default settings, the solr.log file >>> should have enough info to show what actually happened. >>> >>> > Also, the search in >>> > collection1, which was originall very fast, becomes very slow after the >>> > indexing is done is collection2. >>> >>> If the two collections have data on the same server(s), I can see this >>> happening. More memory is consumed when there is additional data, and >>> when Solr needs more memory, performance might be affected. The >>> solution is generally to install more memory in the server. If the >>> system is working, there should be no need to increase the heap size >>> when the memory size increases ... but there can be situations where the >>> heap is a little bit too small, where you WOULD want to increase the >>> heap size. >>> >>> Thanks, >>> Shawn >>> >>>
