psalagnac commented on PR #1761: URL: https://github.com/apache/solr/pull/1761#issuecomment-1671353210
> Any thoughts on how this bug slipped by our notice? I was unable to write a unit that actually do an expensive operation (backup or restore). The low-level queue mechanism is well tested, but I'm not sure how to write an end-to-end test with expensive operations at scale. > read through this, I'm not really familiar with any of this code but the change looks good to me. minor question. why not add another executor (fixed number of threads similar to parallelExecutor) for these expensive operations? all the code that ensures the max number of running threads would no longer be necessary, the finishTask method would also be a bit simpler. Yes, that would be another option. This is at the cost of having many dormant threads most of the time. Also, manually handling a queue for expensive tasks allow eventual future improvements. I don't have any strong opinion. > > another one for my personal understanding: when the status of a task is reported as `RUNNING` it could either be queued or running. this means 'sent to executor', but there is no guarantee this will run immediately. introducing another status as `PENDING` might cause some confusion to what is actually running at a given moment. You're correct, RUNNING status means 'sent to executor'. For most of the tasks, I don't expect the time spend by the task in the executor queue to be significant (otherwise, we would have to mark something else as expensive if it keeps the executor busy for a while). I added the `PENDING` status to make this difference. Expensive tasks can be in the queue for seconds/minutes for big collections, so I think there is some value for the caller to know that at some point. (note that the caller is the overseer only. The top level task with user specified asyncId does not expose this _pending_ status) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
