[ https://issues.apache.org/jira/browse/SOLR-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871332#comment-17871332 ]
David Smiley commented on SOLR-17391: ------------------------------------- Hello; thanks for reporting in nice detail! I didn't yet look at this carefully but did see that your proposed patch replaces the thread pool with a fixed size (albeit configurable). Can't we have a cached pool so that we don't use any threads if there's nothing to do? Many servers & tests are commonly not doing any of these "expensive" tasks at any one moment. > Optimize Backup/Restore Operations for Large Collections > -------------------------------------------------------- > > Key: SOLR-17391 > URL: https://issues.apache.org/jira/browse/SOLR-17391 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Backup/Restore > Affects Versions: 9.4, 9.5, 9.4.1, 9.6, 9.6.1 > Reporter: Hakan Özler > Priority: Major > > The backup/restore performance issue was first reported on [the users > mailing|https://lists.apache.org/thread/ssmzg5nhhxdhgz4980opn1vzxs81o9pk] > list. > > We're experiencing performance issues in the recent Solr versions — 9.5.0 and > 9.6.1 — regarding backup and restore. In 9.2.1, we could take a backup of > 10TB data in just 1 and a half hours. Currently, as of 9.5.0, taking a backup > of the collection takes 7 hours! We're unable to make use of disaster > recovery effectively and reliably in Solr. Therefore, Solr 9.2.1 still > remains the most effective choice among the other 9.x versions for our use. > It seems that this is the ticket causing this issue: > 1. https://issues.apache.org/jira/browse/SOLR-16879 > Interestingly, we never encountered a throttling problem during operations > when this was introduced to be solved based on this argument on 9.2.1. From a > devops perspective, we have some details and metrics on these tasks to > distinguish the difference between two versions. The overall IOPS was 150MB > on 9.6.1, while IOPS was 500MB on 9.2.1 during the same backup and restore > tasks. In the first below, the peak on the left represents a backup, in > contrast, in the 2nd image, the same backup operation in 9.5.0 uses less > resource. As you may spot, 9.5.0 seems to be using a fifth of the resources > of 9.2.1. > > !https://i.imgur.com/aSrs8OM.png! > Image 1. > !https://i.imgur.com/aSrs8OM.png! > Image 2. > > Apart from that, monitoring some relevant metrics during the operations, I > had some difficulty interpreting the following metrics: > {code:java} > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: > 1,{code} > The pool size was 1 although the pool max size is 5. Shouldn't the pool size > be 5, instead? However, there is always one task running on a single node, > not 5 concurrently, if I'm not mistaken. > I was also wondering if the max thread size, which is currently 5 in 9.4+, > could be configurable with either an environment variable or Java parameter? > The part that needs to be changed seems to be in CoreAdminHandler.java on > line 446 [1] I've made a small adjustment to add a Solr parameter called > `solr.maxExpensiveTaskThreads` for those who want to set a different thread > size for expensive tasks. The number given in this parameter must meet the > criteria of ThreadPoolExecutor, otherwise IllegalArgumentException will > occur. I've generated a patch [2] and I would love to see if someone from the > Solr committers would take on this and apply for the upcoming release. Do you > think our observation is accurate and would this patch be feasible to > implement? > > 1. > [https://github.com/apache/solr/commit/82a847f0f9af18d6eceee18743d636db7a879f3e#diff-5bc3d44ca8b189f44fe9e6f75af8a5510463bdba79ff72a7d0ed190973a32533L446] > 2. [https://gist.github.com/ozlerhakan/e4d11bddae6a2f89d2c212c220f4c965] > > Follow up on this, we managed to backup a data of 3TB in 50 minutes with the > patch using `solr.maxExpensiveTaskThreads=5` : > > !https://i.imgur.com/oeCrhLn.png|width=626,height=239! > > I also answered the questions from @Kevin Liang , > {quote}Was this change tested on a cloud that was also taking active > ingest/query requests as the same time as the backup? > {quote} > The test is completed in a SolrCloud 9.6.1 + the patch cluster managed by the > official Solr operator on Amazon EKS. The backup strategy is not intended to > happen frequently. Instead, we plan to take some backups for a certain period > of time, therefore we won't expect intense search traffic in and out during > backups. > > {quote}This performance is really exciting, but I'm curious how much burden > it puts on CPU and memory. > {quote} > I'd say that Solr was pretty relaxed during the test based on the CPU usage. > It looks like backup and restore are not a CPU intensive task. Each node used > only one core at a time. > !https://i.imgur.com/pEb37nb.png|width=348,height=222! > !https://i.imgur.com/4aFqJVY.png|width=348,height=238! > {quote}Also was this just taking a snapshot backup of the segment files or > did this also include uploading to S3? > {quote} > > We're using the recommended backup functionality, where Solr uploads > everything to S3 [1] During backup and restore ops, the relevant metrics > looked like this: > {code:java} > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 5, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 5, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: > 5,{code} > While, without the patch, It indicated the following behavior: > {code:java} > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: > 1,{code} > > Given that we have the patch, I believe we've returned to the old 9.2.1 > version. Setting the parameter to 1 seems to replicate the current 9.6.1 > version, where the same backup takes 2.5 hours. This is clear, there was one > thread/task running for a shard on every Solr node, as each node has 5 shards > in the cluster for the collection, and there were 4 more tasks in the queue: > {code:java} > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 1, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 1, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: 1, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.active: > 1, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.capacity: > 2147483644, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.completed: > 0, > ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.queued: > 4{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org