Hakan Özler created SOLR-17391:
----------------------------------
Summary: Optimize Backup/Restore Operations for Large Collections
Key: SOLR-17391
URL: https://issues.apache.org/jira/browse/SOLR-17391
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: Backup/Restore
Affects Versions: 9.6.1, 9.6, 9.4.1, 9.5, 9.4
Reporter: Hakan Özler
The backup/restore performance issue was first reported on [the users
mail|https://lists.apache.org/thread/ssmzg5nhhxdhgz4980opn1vzxs81o9pk] list.
We're experiencing performance issues in the recent Solr versions — 9.5.0 and
9.6.1 — regarding backup and restore. In 9.2.1, we could take a backup of 10TB
data in just 1 and a half hours. Currently, as of 9.5.0, taking a backup of the
collection takes 7 hours! We're unable to make use of disaster recovery
effectively and reliably in Solr. Therefore, Solr 9.2.1 still remains the most
effective choice among the other 9.x versions for our use.
It seems that this is the ticket causing this issue:
1. https://issues.apache.org/jira/browse/SOLR-16879
Interestingly, we never encountered a throttling problem during operations when
this was introduced to be solved based on this argument on 9.2.1. From a devops
perspective, we have some details and metrics on these tasks to distinguish the
difference between two versions. The overall IOPS was 150MB on 9.6.1, while
IOPS was 500MB on 9.2.1 during the same backup and restore tasks. In the first
below, the peak on the left represents a backup, in contrast, in the 2nd image,
the same backup operation in 9.5.0 uses less resource. As you may spot, 9.5.0
seems to be using a fifth of the resources of 9.2.1.
!https://i.imgur.com/aSrs8OM.png!
Image 1.
!https://i.imgur.com/aSrs8OM.png!
Image 2.
Apart from that, monitoring some relevant metrics during the operations, I had
some difficulty interpreting the following metrics:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
1,{code}
The pool size was 1 although the pool max size is 5. Shouldn't the pool size be
5, instead? However, there is always one task running on a single node, not 5
concurrently, if I'm not mistaken.
I was also wondering if the max thread size, which is currently 5 in 9.4+,
could be configurable with either an environment variable or Java parameter?
The part that needs to be changed seems to be in CoreAdminHandler.java on line
446 [1] I've made a small adjustment to add a Solr parameter called
`solr.maxExpensiveTaskThreads` for those who want to set a different thread
size for expensive tasks. The number given in this parameter must meet the
criteria of ThreadPoolExecutor, otherwise IllegalArgumentException will occur.
I've generated a patch [2] and I would love to see if someone from the Solr
committers would take on this and apply for the upcoming release. Do you think
our observation is accurate and would this patch be feasible to implement?
1.
[https://github.com/apache/solr/commit/82a847f0f9af18d6eceee18743d636db7a879f3e#diff-5bc3d44ca8b189f44fe9e6f75af8a5510463bdba79ff72a7d0ed190973a32533L446]
2. [https://gist.github.com/ozlerhakan/e4d11bddae6a2f89d2c212c220f4c965]
Follow up on this, we managed to backup a data of 3TB in 50 minutes with the
patch using `solr.maxExpensiveTaskThreads=5` :
!https://i.imgur.com/oeCrhLn.png|width=626,height=239!
I also answered the questions from @Kevin Liang , {quote}Was this change tested
on a cloud that was also taking active ingest/query requests as the same time
as the backup? {quote}
The test is completed in a SolrCloud 9.6.1 + the patch cluster managed by the
official Solr operator on Amazon EKS. The backup strategy is not intended to
happen frequently. Instead, we plan to take some backups for a certain period
of time, therefore we won't expect intense search traffic in and out during
backups.
{quote}This performance is really exciting, but I'm curious how much burden it
puts on CPU and memory.{quote}
I'd say that Solr was pretty relaxed during the test based on the CPU usage. It
looks like backup and restore are not a CPU intensive task. Each node used only
one core at a time.
!https://i.imgur.com/pEb37nb.png|width=348,height=222!
!https://i.imgur.com/4aFqJVY.png|width=348,height=238!
{quote}Also was this just taking a snapshot backup of the segment files or did
this also include uploading to S3?{quote}
We're using the recommended backup functionality, where Solr uploads everything
to S3 [1] During backup and restore ops, the relevant metrics looked like this:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
5,{code}
While, without the patch, It indicated the following behavior:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
1,{code}
Given that we have the patch, I believe we've returned to the old 9.2.1
version. Setting the parameter to 1 seems to replicate the current 9.6.1
version, where the same backup takes
2.5 hours. This is clear, there was one thread/task running for a shard on
every Solr node, as each node has 5 shards in the cluster for the collection,
and there were 4 more tasks in the queue:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.active:
1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.capacity:
2147483644,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.completed:
0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.queued:
4{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
