bug#44175: [optimization] Grafting is too slow

[Maxim Cournoyer]

Hello Lars,

Lars-Dominik Braun  writes:

> Hi Maxim,
>
>> Judging from the above, it seems this issue has been resolved.
> grafting is still a performance issue imo. Compare for example:
>
> $ time guix environment --ad-hoc  --search-paths r-learnr
> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 
> 210% cpu 2,844 total
> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 
> 0,08s system 164% cpu 1,277 total

I'm opening a new issue to track optimizing the grafting code, since
it's independent of environments (grafts are applied anytime a
derivation is built, AFAICT).  Grafting is inherently IO-bound, since it
must scan all the built artifacts of a grafted package (from its
dependency that was 'replaced' all the way up), if I understood
correctly.  "info (guix) Security Updates" gives some information.

Perhaps we could benchmark how fast our code currently is for grafting,
and compare it with the fastest sed-like utility out there, as a
starting point, to have an idea of how much there is to gain from
optimization.

Maxim

bug#44175: [optimization] Grafting is too slow

[Ludovic Courtès]

Maxim Cournoyer  skribis:

> Lars-Dominik Braun  writes:
>
>> Hi Maxim,
>>
>>> Judging from the above, it seems this issue has been resolved.
>> grafting is still a performance issue imo. Compare for example:
>>
>> $ time guix environment --ad-hoc  --search-paths r-learnr
>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 
>> 210% cpu 2,844 total
>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 
>> 0,08s system 164% cpu 1,277 total
>
> I'm opening a new issue to track optimizing the grafting code, since
> it's independent of environments (grafts are applied anytime a
> derivation is built, AFAICT).  Grafting is inherently IO-bound,

What is slow above is not grafting itself: it’s determining what to
graft that takes CPU time.

I had reopened the initial bug at ;
should we close this one?

Thanks,
Ludo’.

bug#44175: [optimization] Grafting is too slow

[Maxim Cournoyer]

Hello!

Ludovic Courtès  writes:

> Maxim Cournoyer  skribis:
>
>> Lars-Dominik Braun  writes:
>>
>>> Hi Maxim,
>>>
 Judging from the above, it seems this issue has been resolved.
>>> grafting is still a performance issue imo. Compare for example:
>>>
>>> $ time guix environment --ad-hoc  --search-paths r-learnr
>>> guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 
>>> 210% cpu 2,844 total
>>> $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
>>> guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 
>>> 0,08s system 164% cpu 1,277 total
>>
>> I'm opening a new issue to track optimizing the grafting code, since
>> it's independent of environments (grafts are applied anytime a
>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>
> What is slow above is not grafting itself: it’s determining what to
> graft that takes CPU time.

On my system, grafting seems IO rather than CPU bound, I'm guessing
because of the need to scan all the files for strings to replace in the
graft process.

> I had reopened the initial bug at ;
> should we close this one?

Many optimizations were made in the above issue that were not related to
the grafting process, so to me a fresh entry such as this one is clearer
to follow.  That said, feel free to proceed as you see fit, being the
issue "owner" :-).

Thanks,

Maxim

bug#44175: [optimization] Grafting is too slow

[Ludovic Courtès]

Hi!

Maxim Cournoyer  skribis:

 $ time guix environment --ad-hoc  --search-paths r-learnr
 guix environment --ad-hoc --search-paths r-learnr  5,90s user 0,09s system 
 210% cpu 2,844 total
 $ time guix environment --ad-hoc  --search-paths r-learnr --no-grafts
 guix environment --ad-hoc --search-paths r-learnr --no-grafts  2,03s user 
 0,08s system 164% cpu 1,277 total
>>>
>>> I'm opening a new issue to track optimizing the grafting code, since
>>> it's independent of environments (grafts are applied anytime a
>>> derivation is built, AFAICT).  Grafting is inherently IO-bound,
>>
>> What is slow above is not grafting itself: it’s determining what to
>> graft that takes CPU time.
>
> On my system, grafting seems IO rather than CPU bound, I'm guessing
> because of the need to scan all the files for strings to replace in the
> graft process.

Yes, you’re right of course.  But I think in the example above Lars was
reporting the run time of the ‘guix’ command when the graft itself is
already built.  Just the extra code to compute the graft derivation (not
actually building them) leads to x2 wall-clock time or so.

>> I had reopened the initial bug at ;
>> should we close this one?
>
> Many optimizations were made in the above issue that were not related to
> the grafting process, so to me a fresh entry such as this one is clearer
> to follow.  That said, feel free to proceed as you see fit, being the
> issue "owner" :-).

Alright, thanks!

Ludo’.

bug#44175: [optimization] Grafting is too slow

[Lars-Dominik Braun]

Hi,

> Yes, you’re right of course.  But I think in the example above Lars was
> reporting the run time of the ‘guix’ command when the graft itself is
> already built.  Just the extra code to compute the graft derivation (not
> actually building them) leads to x2 wall-clock time or so.
yes, Ludo is right. The numbers I reported are with grafts already built, warm 
caches and
local socket communication. So we’re only seeing the CPU-bound computation
here. Sorry for the confusion. I should have been more specific in the other
issue.

Cheers,
Lars



signature.asc
Description: PGP signature