[
https://issues.apache.org/jira/browse/SOLR-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hakan Özler updated SOLR-17391:
-------------------------------
Description:
The backup/restore performance issue was first reported on [the users
mail|https://lists.apache.org/thread/ssmzg5nhhxdhgz4980opn1vzxs81o9pk] list.
We're experiencing performance issues in the recent Solr versions — 9.5.0 and
9.6.1 — regarding backup and restore. In 9.2.1, we could take a backup of 10TB
data in just 1 and a half hours. Currently, as of 9.5.0, taking a backup of the
collection takes 7 hours! We're unable to make use of disaster recovery
effectively and reliably in Solr. Therefore, Solr 9.2.1 still remains the most
effective choice among the other 9.x versions for our use.
It seems that this is the ticket causing this issue:
1. https://issues.apache.org/jira/browse/SOLR-16879
Interestingly, we never encountered a throttling problem during operations when
this was introduced to be solved based on this argument on 9.2.1. From a devops
perspective, we have some details and metrics on these tasks to distinguish the
difference between two versions. The overall IOPS was 150MB on 9.6.1, while
IOPS was 500MB on 9.2.1 during the same backup and restore tasks. In the first
below, the peak on the left represents a backup, in contrast, in the 2nd image,
the same backup operation in 9.5.0 uses less resource. As you may spot, 9.5.0
seems to be using a fifth of the resources of 9.2.1.
!https://i.imgur.com/aSrs8OM.png!
Image 1.
!https://i.imgur.com/aSrs8OM.png!
Image 2.
Apart from that, monitoring some relevant metrics during the operations, I had
some difficulty interpreting the following metrics:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
1,{code}
The pool size was 1 although the pool max size is 5. Shouldn't the pool size be
5, instead? However, there is always one task running on a single node, not 5
concurrently, if I'm not mistaken.
I was also wondering if the max thread size, which is currently 5 in 9.4+,
could be configurable with either an environment variable or Java parameter?
The part that needs to be changed seems to be in CoreAdminHandler.java on line
446 [1] I've made a small adjustment to add a Solr parameter called
`solr.maxExpensiveTaskThreads` for those who want to set a different thread
size for expensive tasks. The number given in this parameter must meet the
criteria of ThreadPoolExecutor, otherwise IllegalArgumentException will occur.
I've generated a patch [2] and I would love to see if someone from the Solr
committers would take on this and apply for the upcoming release. Do you think
our observation is accurate and would this patch be feasible to implement?
1.
[https://github.com/apache/solr/commit/82a847f0f9af18d6eceee18743d636db7a879f3e#diff-5bc3d44ca8b189f44fe9e6f75af8a5510463bdba79ff72a7d0ed190973a32533L446]
2. [https://gist.github.com/ozlerhakan/e4d11bddae6a2f89d2c212c220f4c965]
Follow up on this, we managed to backup a data of 3TB in 50 minutes with the
patch using `solr.maxExpensiveTaskThreads=5` :
!https://i.imgur.com/oeCrhLn.png|width=626,height=239!
I also answered the questions from @Kevin Liang ,
{quote}Was this change tested on a cloud that was also taking active
ingest/query requests as the same time as the backup?
{quote}
The test is completed in a SolrCloud 9.6.1 + the patch cluster managed by the
official Solr operator on Amazon EKS. The backup strategy is not intended to
happen frequently. Instead, we plan to take some backups for a certain period
of time, therefore we won't expect intense search traffic in and out during
backups.
{quote}This performance is really exciting, but I'm curious how much burden it
puts on CPU and memory.
{quote}
I'd say that Solr was pretty relaxed during the test based on the CPU usage. It
looks like backup and restore are not a CPU intensive task. Each node used only
one core at a time.
!https://i.imgur.com/pEb37nb.png|width=348,height=222!
!https://i.imgur.com/4aFqJVY.png|width=348,height=238!
{quote}Also was this just taking a snapshot backup of the segment files or did
this also include uploading to S3?
{quote}
We're using the recommended backup functionality, where Solr uploads everything
to S3 [1] During backup and restore ops, the relevant metrics looked like this:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
5,{code}
While, without the patch, It indicated the following behavior:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
1,{code}
Given that we have the patch, I believe we've returned to the old 9.2.1
version. Setting the parameter to 1 seems to replicate the current 9.6.1
version, where the same backup takes
2.5 hours. This is clear, there was one thread/task running for a shard on
every Solr node, as each node has 5 shards in the cluster for the collection,
and there were 4 more tasks in the queue:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.active:
1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.capacity:
2147483644,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.completed:
0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.queued:
4{code}
was:
The backup/restore performance issue was first reported on [the users
mail|https://lists.apache.org/thread/ssmzg5nhhxdhgz4980opn1vzxs81o9pk] list.
We're experiencing performance issues in the recent Solr versions — 9.5.0 and
9.6.1 — regarding backup and restore. In 9.2.1, we could take a backup of 10TB
data in just 1 and a half hours. Currently, as of 9.5.0, taking a backup of the
collection takes 7 hours! We're unable to make use of disaster recovery
effectively and reliably in Solr. Therefore, Solr 9.2.1 still remains the most
effective choice among the other 9.x versions for our use.
It seems that this is the ticket causing this issue:
1. https://issues.apache.org/jira/browse/SOLR-16879
Interestingly, we never encountered a throttling problem during operations when
this was introduced to be solved based on this argument on 9.2.1. From a devops
perspective, we have some details and metrics on these tasks to distinguish the
difference between two versions. The overall IOPS was 150MB on 9.6.1, while
IOPS was 500MB on 9.2.1 during the same backup and restore tasks. In the first
below, the peak on the left represents a backup, in contrast, in the 2nd image,
the same backup operation in 9.5.0 uses less resource. As you may spot, 9.5.0
seems to be using a fifth of the resources of 9.2.1.
!https://i.imgur.com/aSrs8OM.png!
Image 1.
!https://i.imgur.com/aSrs8OM.png!
Image 2.
Apart from that, monitoring some relevant metrics during the operations, I had
some difficulty interpreting the following metrics:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
1,{code}
The pool size was 1 although the pool max size is 5. Shouldn't the pool size be
5, instead? However, there is always one task running on a single node, not 5
concurrently, if I'm not mistaken.
I was also wondering if the max thread size, which is currently 5 in 9.4+,
could be configurable with either an environment variable or Java parameter?
The part that needs to be changed seems to be in CoreAdminHandler.java on line
446 [1] I've made a small adjustment to add a Solr parameter called
`solr.maxExpensiveTaskThreads` for those who want to set a different thread
size for expensive tasks. The number given in this parameter must meet the
criteria of ThreadPoolExecutor, otherwise IllegalArgumentException will occur.
I've generated a patch [2] and I would love to see if someone from the Solr
committers would take on this and apply for the upcoming release. Do you think
our observation is accurate and would this patch be feasible to implement?
1.
[https://github.com/apache/solr/commit/82a847f0f9af18d6eceee18743d636db7a879f3e#diff-5bc3d44ca8b189f44fe9e6f75af8a5510463bdba79ff72a7d0ed190973a32533L446]
2. [https://gist.github.com/ozlerhakan/e4d11bddae6a2f89d2c212c220f4c965]
Follow up on this, we managed to backup a data of 3TB in 50 minutes with the
patch using `solr.maxExpensiveTaskThreads=5` :
!https://i.imgur.com/oeCrhLn.png|width=626,height=239!
I also answered the questions from @Kevin Liang , {quote}Was this change tested
on a cloud that was also taking active ingest/query requests as the same time
as the backup? {quote}
The test is completed in a SolrCloud 9.6.1 + the patch cluster managed by the
official Solr operator on Amazon EKS. The backup strategy is not intended to
happen frequently. Instead, we plan to take some backups for a certain period
of time, therefore we won't expect intense search traffic in and out during
backups.
{quote}This performance is really exciting, but I'm curious how much burden it
puts on CPU and memory.{quote}
I'd say that Solr was pretty relaxed during the test based on the CPU usage. It
looks like backup and restore are not a CPU intensive task. Each node used only
one core at a time.
!https://i.imgur.com/pEb37nb.png|width=348,height=222!
!https://i.imgur.com/4aFqJVY.png|width=348,height=238!
{quote}Also was this just taking a snapshot backup of the segment files or did
this also include uploading to S3?{quote}
We're using the recommended backup functionality, where Solr uploads everything
to S3 [1] During backup and restore ops, the relevant metrics looked like this:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
5,{code}
While, without the patch, It indicated the following behavior:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
1,{code}
Given that we have the patch, I believe we've returned to the old 9.2.1
version. Setting the parameter to 1 seems to replicate the current 9.6.1
version, where the same backup takes
2.5 hours. This is clear, there was one thread/task running for a shard on
every Solr node, as each node has 5 shards in the cluster for the collection,
and there were 4 more tasks in the queue:
{code:java}
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: 1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.active:
1,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.capacity:
2147483644,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.completed:
0,
ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.queued:
4{code}
> Optimize Backup/Restore Operations for Large Collections
> --------------------------------------------------------
>
> Key: SOLR-17391
> URL: https://issues.apache.org/jira/browse/SOLR-17391
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
> Affects Versions: 9.4, 9.5, 9.4.1, 9.6, 9.6.1
> Reporter: Hakan Özler
> Priority: Major
>
> The backup/restore performance issue was first reported on [the users
> mail|https://lists.apache.org/thread/ssmzg5nhhxdhgz4980opn1vzxs81o9pk] list.
>
> We're experiencing performance issues in the recent Solr versions — 9.5.0 and
> 9.6.1 — regarding backup and restore. In 9.2.1, we could take a backup of
> 10TB data in just 1 and a half hours. Currently, as of 9.5.0, taking a backup
> of the collection takes 7 hours! We're unable to make use of disaster
> recovery effectively and reliably in Solr. Therefore, Solr 9.2.1 still
> remains the most effective choice among the other 9.x versions for our use.
> It seems that this is the ticket causing this issue:
> 1. https://issues.apache.org/jira/browse/SOLR-16879
> Interestingly, we never encountered a throttling problem during operations
> when this was introduced to be solved based on this argument on 9.2.1. From a
> devops perspective, we have some details and metrics on these tasks to
> distinguish the difference between two versions. The overall IOPS was 150MB
> on 9.6.1, while IOPS was 500MB on 9.2.1 during the same backup and restore
> tasks. In the first below, the peak on the left represents a backup, in
> contrast, in the 2nd image, the same backup operation in 9.5.0 uses less
> resource. As you may spot, 9.5.0 seems to be using a fifth of the resources
> of 9.2.1.
>
> !https://i.imgur.com/aSrs8OM.png!
> Image 1.
> !https://i.imgur.com/aSrs8OM.png!
> Image 2.
>
> Apart from that, monitoring some relevant metrics during the operations, I
> had some difficulty interpreting the following metrics:
> {code:java}
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
> 1,{code}
> The pool size was 1 although the pool max size is 5. Shouldn't the pool size
> be 5, instead? However, there is always one task running on a single node,
> not 5 concurrently, if I'm not mistaken.
> I was also wondering if the max thread size, which is currently 5 in 9.4+,
> could be configurable with either an environment variable or Java parameter?
> The part that needs to be changed seems to be in CoreAdminHandler.java on
> line 446 [1] I've made a small adjustment to add a Solr parameter called
> `solr.maxExpensiveTaskThreads` for those who want to set a different thread
> size for expensive tasks. The number given in this parameter must meet the
> criteria of ThreadPoolExecutor, otherwise IllegalArgumentException will
> occur. I've generated a patch [2] and I would love to see if someone from the
> Solr committers would take on this and apply for the upcoming release. Do you
> think our observation is accurate and would this patch be feasible to
> implement?
>
> 1.
> [https://github.com/apache/solr/commit/82a847f0f9af18d6eceee18743d636db7a879f3e#diff-5bc3d44ca8b189f44fe9e6f75af8a5510463bdba79ff72a7d0ed190973a32533L446]
> 2. [https://gist.github.com/ozlerhakan/e4d11bddae6a2f89d2c212c220f4c965]
>
> Follow up on this, we managed to backup a data of 3TB in 50 minutes with the
> patch using `solr.maxExpensiveTaskThreads=5` :
>
> !https://i.imgur.com/oeCrhLn.png|width=626,height=239!
>
> I also answered the questions from @Kevin Liang ,
> {quote}Was this change tested on a cloud that was also taking active
> ingest/query requests as the same time as the backup?
> {quote}
> The test is completed in a SolrCloud 9.6.1 + the patch cluster managed by the
> official Solr operator on Amazon EKS. The backup strategy is not intended to
> happen frequently. Instead, we plan to take some backups for a certain period
> of time, therefore we won't expect intense search traffic in and out during
> backups.
>
> {quote}This performance is really exciting, but I'm curious how much burden
> it puts on CPU and memory.
> {quote}
> I'd say that Solr was pretty relaxed during the test based on the CPU usage.
> It looks like backup and restore are not a CPU intensive task. Each node used
> only one core at a time.
> !https://i.imgur.com/pEb37nb.png|width=348,height=222!
> !https://i.imgur.com/4aFqJVY.png|width=348,height=238!
> {quote}Also was this just taking a snapshot backup of the segment files or
> did this also include uploading to S3?
> {quote}
>
> We're using the recommended backup functionality, where Solr uploads
> everything to S3 [1] During backup and restore ops, the relevant metrics
> looked like this:
> {code:java}
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 5,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 5,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
> 5,{code}
> While, without the patch, It indicated the following behavior:
> {code:java}
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 0,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 5,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running:
> 1,{code}
>
> Given that we have the patch, I believe we've returned to the old 9.2.1
> version. Setting the parameter to 1 seems to replicate the current 9.6.1
> version, where the same backup takes
> 2.5 hours. This is clear, there was one thread/task running for a shard on
> every Solr node, as each node has 5 shards in the cluster for the collection,
> and there were 4 more tasks in the queue:
> {code:java}
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.core: 1,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.max: 1,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.pool.size: 1,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.running: 1,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.active:
> 1,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.capacity:
> 2147483644,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.completed:
> 0,
> ADMIN./admin/cores.threadPool.parallelCoreExpensiveAdminExecutor.tasks.queued:
> 4{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
