On 6/12/2018 12:06 PM, Chris Troullis wrote: > The issue we are seeing is with 1 collection in particular, after we set up > CDCR, we are getting extremely slow response times when retrieving > documents. Debugging the query shows QTime is almost nothing, but the > overall responseTime is like 5x what it should be. The problem is > exacerbated by larger result sizes. IE retrieving 25 results is almost > normal, but 200 results is way slower than normal. I can run the exact same > query multiple times in a row (so everything should be cached), and I still > see response times way higher than another environment that is not using > CDCR. It doesn't seem to matter if CDCR is enabled or disabled, just that > we are using the CDCRUpdateLog. The problem started happening even before > we enabled CDCR. > > In a lower environment we noticed that the transaction logs were huge > (multiple gigs), so we tried stopping solr and deleting the tlogs then > restarting, and that seemed to fix the performance issue. We tried the same > thing in production the other day but it had no effect, so now I don't know > if it was a coincidence or not.
There is one other cause besides CDCR buffering that I know of for huge transaction logs, and it has nothing to do with CDCR: A lack of hard commits. It is strongly recommended to have autoCommit set to a reasonably short interval (about a minute in my opinion, but 15 seconds is VERY common). Most of the time openSearcher should be set to false in the autoCommit config, and other mechanisms (which might include autoSoftCommit) should be used for change visibility. The example autoCommit settings might seem superfluous because they don't affect what's searchable, but it is actually a very important configuration to keep. Are the docs in this collection really big, by chance? As I went through previous threads you've started on the mailing list, I have noticed that none of your messages provided some details that would be useful for looking into performance problems: * What OS vendor and version Solr is running on. * Total document count on the server (counting all index cores). * Total index size on the server (counting all cores). * What the total of all Solr heaps on the server is. * Whether there is software other than Solr on the server. * How much total memory the server has installed. If you name the OS, I can use that information to help you gather some additional info which will actually show me most of that list. Total document count is something that I cannot get from the info I would help you gather. Something else that can cause performance issues is GC pauses. If you provide a GC log (The script that starts Solr logs this by default), we can analyze it to see if that's a problem. Attachments to messages on the mailing list typically do not make it to the list, so a file sharing website is a better way to share large logfiles. A paste website is good for log data that's smaller. Thanks, Shawn
