abderrahim commented on issue #2157: URL: https://github.com/apache/buildstream/issues/2157#issuecomment-5163845005
I've encountered this countless times in the past and tried to investigate / fix it, but it's not really easy. The main issue here is that terminating a job that is blocking does not actually terminate it. Buildstream offers a [blocking_activity](https://docs.buildstream.build/master/buildstream.plugin.html#buildstream.plugin.Plugin.blocking_activity) helper function to run such blocking calls in a subprocess, but doesn't actually use it when talking to buildbox-casd. What's happening is when ask buildstream to terminate a job, it schedules an exception in the thread that is running the job. The exception doesn't actually get thrown until control is back to python. So if the thread is in a blocking call into C/C++, it doesn't actually get the exception until that call returns. The workaround devised for this situation is the `blocking_activity()` helper method mentioned earlier, but it isn't actually used with buildbox-casd. So what happens is that buildstream (main thread) will say that the job is terminating, and schedule the exception. The thread running the job is still stuck waiting for the `FetchMissingBlobs`/`UploadMissingBlobs` grpc call to return. Once the call returns (which might take a long time, or possibly never if you don't have the various timeouts configured), the python thread gets the exception and proceeds to terminate. The workaround for this is, as suggested earlier is to manually kill buildbox-casd (`pkill -9 buildbox-casd`, or probably better to look up the pid if you have another instance of buildbox-casd running). Buildstream detects this situation and just stops everything. There is a race condition where it doesn't and you need to also kill bst, but it's very rare. Now the question is what should we do about it? One thing is somehow avoiding blocking grpc calls, the other one (maybe less urgent) is to cancel the grpc call and teach buildbox-casd to properly react to cancellation. I have a proof of concept for the former, I'll try to clean it up and push it today for discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
