> Are you assuming here that the two machines are the same? Or are they > different?
I am assuming that the two machines are the same with the exception of cpu frequency and threads that are used as a variable to assume the build time from known value. > This approximation would not even be accurate enough for the same machine. > For instance, the test suite of the julia package runs mainly sequential > using one thread... I am aware of this scenario and I adapted the equasion for it, but I recognize that this exponentially increases the inaccuracy with more threads and I don't believe that there is a mathematical way with the provided values to handle that scenario so we would have to adjust the calculation for those packages. Alternative and more robust solution would be to build the package with `--jobs` set on "cpu threads - 1" and then compare the build time to a scenario without this set which would give us two coordinates to be interpreted in an exponential and eventually quadratic or even logarithmic function. As we already have to build the package multiple times to establish reproducibility so this shouldn't add any overhead just slightly slower reproduction build. The exponential function would be less vulnerable to tasks that are unable to utilize all available processing resources, but it's more inaccurate so I assume that we would eventually have to use quadratic/logarithmic definition that uses the previously provided equasion on both coordinates to determine build on different system which to me seems as a robust in my current testing, but I am willing to give it more time if we (me and other GNU Guix devs/contributors) agree that this would be a mergable effort as it would basically require a one value preferably stored in the guix repo that is used to calculate the build time. -- Jacob "Kreyren" Hrbek Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, November 23rd, 2021 at 11:56 AM, zimoun <zimon.touto...@gmail.com> wrote: > Hi, > > On Tue, 23 Nov 2021 at 06:21, Jacob Hrbek krey...@rixotstudio.cz wrote: > > > 1. locally: Storing the value somewhere on the system and adding up to > > > > it each build to provide more accurate average. > > Timing is already stored, see “guix build --log-file”. Give a look at > > ’/var/log/guix/drvs’. For instance, > > --8<---------------cut here---------------start------------->8--- > > $ bzcat > /var/log/guix/drvs/aq/abymi9yk7pv89614dcdfll3hh4i5mc-julia-1.5.3.drv.bz2 | > grep phase | grep seconds > > phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds phase` set-paths' > succeeded after 0.0 seconds > > phase `install-locale' succeeded after 0.0 seconds phase` unpack' succeeded > after 1.1 seconds > > phase `use-system-libwhich' succeeded after 0.0 seconds phase` > disable-documentation' succeeded after 0.0 seconds > > phase `prepare-deps' succeeded after 0.0 seconds phase` bootstrap' succeeded > after 0.0 seconds > > phase `patch-usr-bin-file' succeeded after 0.0 seconds phase` > patch-source-shebangs' succeeded after 0.2 seconds > > phase `patch-generated-file-shebangs' succeeded after 0.0 seconds phase` > fix-include-and-link-paths' succeeded after 0.0 seconds > > phase `replace-default-shell' succeeded after 0.0 seconds phase` > fix-precompile' succeeded after 0.0 seconds > > phase `build' succeeded after 354.3 seconds phase` set-home' succeeded after > 0.0 seconds > > phase `disable-broken-tests' succeeded after 0.0 seconds phase` check' > succeeded after 7428.8 seconds > > phase `install' succeeded after 16.0 seconds phase` make-wrapper' succeeded > after 0.0 seconds > > phase `patch-shebangs' succeeded after 0.0 seconds phase` strip' succeeded > after 0.0 seconds > > phase `validate-runpath' succeeded after 0.0 seconds phase` > validate-documentation-location' succeeded after 0.0 seconds > > phase `delete-info-dir-file' succeeded after 0.0 seconds phase` > patch-dot-desktop-files' succeeded after 0.0 seconds > > phase `install-license-files' succeeded after 0.0 seconds phase` > reset-gzip-timestamps' succeeded after 0.0 seconds > > phase `compress-documentation' succeeded after 0.0 seconds > > --8<---------------cut here---------------end--------------->8--- > > Therefore, you need to extract somehow that information. > > > optionally This local database can be shared across multiple > > > > systems that add values to it like simple listener waiting for POST > > > > requests. > > It should be better to use a content-addressed distribution such as IPFS > > or GNUnet, IMHO. > > > - within the guix repo: Since we are already building the package we > > > > can take the time and then do the provided math in reverse to > > > > calculate the time: > > > > Build took 100 sec on system with 8 threads at 2.4 Ghz max cpu > > frequency: > > > > 100 * (2.4 * 8) = 1920 (build time value per one thread at 1 Ghz) > > > > Building the package on system with 2 threads at 2.4 Ghz max cpu > > frequency: > > > > 1920 / (2 * 2.4) = 400 > > > > We can then assume that the build will take 1920/400=4.8 -> 4.8 > > > > times longer on this system. > > > > Are you assuming here that the two machines are the same? Or are they > > different? > > This approximation would not even be accurate enough for the same > > machine. For instance, the test suite of the julia package runs mainly > > sequential using one thread. If you go back to numbers above, > > build=354.3 seconds and check=7428.8 seconds, so the number of threads > > only tweaks timing of build phase, which will not impact much the > > overall timing. > > Somehow, IIUC your proposal, you would like, based on timings from > > machine A about a set of packages, and timings from machine B about the > > same set of packages, knowing the timing of machine B for package foo, > > then extrapolate timing for machine A of package foo. The maths for > > that are not linear, IMHO, and require “complicated” heuristics. It is > > not that complicated, it “just” require some statistical regression – > > though it is not straightforward either. :-) > > BTW, why not directly substitute package foo from machine B? > > > The math might need to be adjusted, but it seems to be sufficiently > > > > accurate through my testing, fwiw I see that `(max cpu frequency * cpu > > threads)` is an important component of the equasion using the analogy > > > > of a (possibly silly) "pokemon battle" assuming interpreting a > > > > mathematical equasion to define the Health Points of the package and > > > > damage per second of the CPU then simply substracting these two values > > > > to determine how long it will take to build alike package has 500 HP > > > > -> Needs a CPU that deals 100 HP to complete in 5 sec or CPU that > > > > deals 50 HP to finish in 10 sec. > > I will be happy if I am wrong. I guess this back-to-envelope would be > > not accurate enough; for two reasons. As I said elsewhere, to your > > example value of 100 seconds is attached a strong variability, depending > > one on how the package itself scales at build time and more than often > > this scaling is not linear versus the number of threads – from my > > experience; and two on the stressed context where the build happens. > > > About accuracy: I highly doubt that we need to worry about the system > > > > noise as that will be mitigated after enough systems submit average > > > > build time with their max CPU frequency and threads used.. we > > > > shouldn't really bother past that at the current stage and we can > > > > always add additional metadata if needed. > > A average is not meaningful by itself. It provides a first-order > > approximation and generally it is not sufficient; the second-order is > > also required. Especially when drawing a model for prediction. From > > what I remember about stats, and assuming the distribution is Gaussian, > > the mean and standard error are required to capture that information. > > My guess is because standard error, the mean would not provide useful > > prediction shareable across heterogeneous machines. > > I will be happy to be wrong and only numbers can answer to this > > question. If you are interested by building a model or verify your > > assumptions, I am sure it is possible to dump the current Cuirass > > postgres database and then do some analytics. It would be a starting > > point to evaluate if the effort implied by your proposal is worth. > > I am not convinced such model would be doable for practical use across > > heterogeneous machines, but it would help for monitoring CI. > > Cheers, > > simon
publickey - kreyren@rixotstudio.cz - 0x1677DB82.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
