Boyi Dai created SOLR-18548:
-------------------------------
Summary: Overseer collection lock leaks when marking an async task
as running fails
Key: SOLR-18548
URL: https://issues.apache.org/jira/browse/SOLR-18548
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 10.0
Reporter: Boyi Dai
OverseerTaskProcessor takes the lock for a task (collection, shard or replica)
before storing the task’s async ID in the running map in ZooKeeper. If that
{{runningMap.put}} fails, only {{NodeExistsException}} and
{{InterruptedException}} release the lock. Any other {{KeeperException}} (for
example {{{}OperationTimeoutException{}}}, or a {{ConnectionLossException}}
after retries) or a {{RuntimeException}} goes to the outer loop’s handler with
the lock still held.
No Runner has been created at that point, so nothing ever releases the lock.
Every later task that needs the same lock is blocked until the Overseer
restarts, so a single transient ZooKeeper error during that write can stall all
collection API operations on that collection.
This has the same symptom as SOLR-17754 (a lock left held so tasks never run),
but a different cause: SOLR-17754 fixed a race with the runner thread pool,
while this happens when the ZooKeeper write fails before a Runner exists.
Expected: the lock is released whenever the task is not handed to a Runner, so
the task can be retried and other tasks for the same collection can proceed. A
fix with a test is coming in a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
