ssssam opened a new issue, #1751: URL: https://github.com/apache/buildstream/issues/1751
BuildStream is a tool to execute full "package" builds, including awkward parts like `configure` scripts, stripping, integration commands, in a repeatable way using a sandbox. It can distribute these builds to build farm workers via the REAPI, at a granularity of one element => one worker. This means, that a large element like WebKit is only compiled on a single worker which can be slow. [Recc](https://gitlab.com/BuildGrid/recc) is a tool to distribute individual C/C++ compile tasks across build farm workers. At present, it cannot be used inside a BuildStream sandbox because it has no way to access the REAPI endpoint, but ... could we allow a controlled opening in the sandbox to enable Recc to be used inside element builds? The ideal scenario would be a large element building as follows: 1. `bst build` runs the element `configure-commands` on a single worker, with `recc` replacing `cc` etc. 2. `bst build` runs the element `build-commands`: a. For each C/C++ compile operation, `recc` hashes the inputs for the given compile operation to produce a cache key, and checks the remote cache via configured REAPI endpoint b. If needed, compile is invoked on a remote worker. Then, the object file is downloaded from the cache. 3. Link operations run on the main worker and still probably take ages to complete. 4. Profit! This is not guaranteed to be faster than building on a single node, a lot depends on the infrastructure being used and the element being built. I think it would be worth exploring as part of a larger project to try and reduce build times of large BuildStream projects like gnome-build-meta. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
