[
https://issues.apache.org/jira/browse/SOLR-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281175#comment-15281175
]
Noble Paul edited comment on SOLR-8744 at 5/12/16 3:59 AM:
-----------------------------------------------------------
good question. Starvation is indeed a problem and we must make the best effort
to run tasks on a first come first served basis
We can address it by making the scheduling itself smarter as follows:
As the scheduler thread starts with trying to acquire a lock, it could build a
tree of its own with nodes marked 'busy' in a local tree for previously
unassignable tasks. The local tree would look as follows.
If the Task {{T1}} needs a lock on Collection {{C1}} and it is unable to
acquire a lock, The local tree would look as follows
{code}
Cluster
/
/
C1
{code}
When task {{T2}} is taken up, which requires a lock on {{Cluster -> C1 -> S1}}
, It would not try to acquire a lock from loct tree because the path {{Cluster
->C1}} is already is marked as 'busy' in the local tree. So {{T2}} would
remain in the work queue till {{T1}} is completed.
was (Author: noble.paul):
good question. Starvation is indeed a problem and we must make the best effort
to run tasks on a first come first served basis
We can address it by making the scheduling itself smarter as follows:
As the scheduler thread starts with trying to acquire a lock, it could build a
tree of its own with nodes marked 'busy' in a local tree for previously
unassaignable tasks. The local tree would look as follows.
If the Task {{T1}} needs a lock on Collection {{C1}} and it is unable to
acquire a lock, The local tree would look as follows
{code}
Cluster
/
/
C1
{code}
When task {{T2}} is taken up, which requires a lock on {{Cluster -> C1 -> S1}}
, It would not try to acquire a lock from loct tree because the path {{Cluster
->C1}} is already is marked as 'busy' in the local tree. So {{T2}} would
remain in the work queue till {{T1}} is completed.
> Overseer operations need more fine grained mutual exclusion
> -----------------------------------------------------------
>
> Key: SOLR-8744
> URL: https://issues.apache.org/jira/browse/SOLR-8744
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Affects Versions: 5.4.1
> Reporter: Scott Blum
> Assignee: Noble Paul
> Labels: sharding, solrcloud
>
> SplitShard creates a mutex over the whole collection, but, in practice, this
> is a big scaling problem. Multiple split shard operations could happen at
> the time time, as long as different shards are being split. In practice,
> those shards often reside on different machines, so there's no I/O bottleneck
> in those cases, just the mutex in Overseer forcing the operations to be done
> serially.
> Given that a single split can take many minutes on a large collection, this
> is a bottleneck at scale.
> Here is the proposed new design
> There are various Collection operations performed at Overseer. They may need
> exclusive access at various levels. Each operation must define the Access
> level at which the access is required. Access level is an enum.
> CLUSTER(0)
> COLLECTION(1)
> SHARD(2)
> REPLICA(3)
> The Overseer node maintains a tree of these locks. The lock tree would look
> as follows. The tree can be created lazily as and when tasks come up.
> {code}
> Legend:
> C1, C2 -> Collections
> S1, S2 -> Shards
> R1,R2,R3,R4 -> Replicas
> Cluster
> / \
> / \
> C1 C2
> / \ / \
> / \ / \
> S1 S2 S1 S2
> R1, R2 R3.R4 R1,R2 R3,R4
> {code}
> When the overseer receives a message, it tries to acquire the appropriate
> lock from the tree. For example, if an operation needs a lock at a Collection
> level and it needs to operate on Collection C1, the node C1 and all child
> nodes of C1 must be free.
> h2.Lock acquiring logic
> Each operation would start from the root of the tree (Level 0 -> Cluster) and
> start moving down depending upon the operation. After it reaches the right
> node, it checks if all the children are free from a lock. If it fails to
> acquire a lock, it remains in the work queue. A scheduler thread waits for
> notification from the current set of tasks . Every task would do a
> {{notify()}} on the monitor of the scheduler thread. The thread would start
> from the head of the queue and check all tasks to see if that task is able to
> acquire the right lock. If yes, it is executed, if not, the task is left in
> the work queue.
> When a new task arrives in the work queue, the schedulerthread wakes and just
> try to schedule that task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
