wsalmonct commented on issue #2157: URL: https://github.com/apache/buildstream/issues/2157#issuecomment-5093235337
> > A tip is to kill casd, so pkill -9 buildbox-casd if everything is really stuck. > Good idea, unfortunately that does not actually work! You can kill casd, buildstream just re-prints the "Pull terminating" messages and keeps hanging there forever just like before. I thought I had had susses with this in the past. Apologies for miss remembering. > > is it that a second ctl-c should trigger buildstream to ask buildbox-casd to stop more quickly or is it that buildbox-casd stopped listening to buildstream? > Actually buildstream has the ability to kill casd if it takes too long, however it does not choose to use it very reliably. Its a long time since i looked at that code, but i have a stonge memory of it not asking buildbox-casd to stop in the past. REAPI and GRPC do not have nice ways to trigger a cancellation. At least in the past it simply didnt ask for any new commands which effectively looks like its stopping buildbox-casd. This is normally a very subtle difference, except when the actions being requested are large. My point here was not to define what the code does, but to get some input from the main authors / maintainers of the project as to what a good solutions should be. We can go to the mailing list for high level questions like this but If @tristanvb / @gtristan or @juergbi had a vision for this already then we "simply" need to fix the project, If there is not a clear vision for what good looks like then we probably do need to decide, your issue is highlighting something lots of us have just lived with partly as the "right" solution is not clear. Thank you for highlighting this. > > The issue is that some actions that we ask of casd are not trivial, eg pulling a large artefact over a slow network or doing a fetch from a remote server that is throttling us. > I don't think it is that difficult to figure out that the network is offline and it is not pulling any data anymore or to just drop a half finished download and quit. Ha, you would hope so wouldn't you :) GRPC makes this much harder than it might be. A lot of the API's are setup to favour recovering rather than stopping quickly as soon as there is any blip in the network. But that's not to say we could not improve things. Also the is the extra issue that this information needs passing back from buildbox-casd to buildstream. Often the bugs in this part of buildstream are caused by buildbox and buildstream being perfectly sensible but miss communicating. This is partly due to them talking mostly REAPI which has a limited API, but they also speak some other GRPC messages. ## Also @ada-magicat It would be interesting to know what REAPI cache you are using. Most Bst, AFAICT, are on cooperate networks with co located caches, or involved in FD-SDK which has its cache and runners with very good internet connections. Also most people are using buildbox-casd or buildgrid as there cache server. Knowing a little bit more about your setup might help us diagnose the underlying buildbox-cad issue. Ofcourse bst should be handling these pull failures much better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
