Re: Replacing polkit by polkit-duktape on core-updates-frozen ?

[Maxim Cournoyer]

Hi Thiago,

Thiago Jung Bauermann  writes:

[...]

> IIUC it’s a bug to test (%current-system) before
> (%current-target-system). That’s because the latter is only defined
> for cross-builds, while the former is always defined. So in practice
> (%current-target-system) will never be checked.

Oh, good catch!  I've fixed it in the patch I just sent in my earlier
reply to Ludovic.

Thank you,

Maxim

Re: Replacing polkit by polkit-duktape on core-updates-frozen ?

[Maxim Cournoyer]

Hello,

Ludovic Courtès  writes:

> Hi,
>
> Maxim Cournoyer  skribis:
>
>> gnu: polkit: Define polkit package variable based on architecture.
>>
>> * gnu/packages/polkit.scm (polkit): Rename to...
>> (polkit*): ... this.
>> (polkit-duktape): Adjust to inherit from polkit*.
>> (polkit-for-system): New procedure.
>> (polkit): New variable.
>
> LGTM!
>
>> But my "test" fails the same:
>>
>> $ ./pre-inst-env guix build --system=aarch64-linux \
>> -e '(@ (gnu packages polkit) polkit)' -n |& grep polkit
>> /gnu/store/dw11y85xfsb8hcg7w2cw57f1xfs4i74m-polkit-0.120.drv
>> /gnu/store/ric7yf4ra2p14p29fwsh18m1nakciakv-polkit-0.120.tar.xz
>
> That’s expected because here you’re effectively calling
> (%current-system) from the top level, and that’s x86_64.
>
> A good test is to try and build one of its dependents:
>
>   guix build -s aarch64-linux gnome-control-center -n
>
> should list polkit-duktape.

Arf, here's what it throws, after adding the missing polkit export:

--8<---cut here---start->8---
$ ./pre-inst-env guix build  -s aarch64-linux gnome-control-center -n
guix build: error: gnu/packages/gnome.scm:5299:2: package 
`colord-minimal@1.4.5' has an invalid input: ("polkit" #)
--8<---cut here---end--->8---

I don't see how it's different from pkg-config... Ideas?

> Are we done and ready for merging once this patch has been applied to
> ‘core-updates-frozen’?

Mostly, after applying fixes from jpoiret for GDM icons I believe
(thanks!); I'll also have to debug why my GDM-less desktop won't allow
me to login anymore (https://issues.guix.gnu.org/52051).

Thank you!

Maxim

>From 48730390fbfa618216c9e8db65643755ff585175 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer 
Date: Sun, 21 Nov 2021 22:20:35 -0500
Subject: [PATCH] gnu: polkit: Define polkit package variable based on
 architecture.

* gnu/packages/polkit.scm (polkit): Rename to...
(polkit*): ... this.
(polkit-duktape): Adjust to inherit from polkit*.
(polkit-for-system): New procedure.
(polkit): New variable.
---
 gnu/packages/polkit.scm | 76 ++---
 1 file changed, 48 insertions(+), 28 deletions(-)

diff --git a/gnu/packages/polkit.scm b/gnu/packages/polkit.scm
index c9edc53cf5..a1bf9786fd 100644
--- a/gnu/packages/polkit.scm
+++ b/gnu/packages/polkit.scm
@@ -29,6 +29,7 @@ (define-module (gnu packages polkit)
   #:use-module ((guix licenses) #:select (lgpl2.0+))
   #:use-module (guix packages)
   #:use-module (guix download)
+  #:use-module (guix memoization)
   #:use-module (guix utils)
   #:use-module (guix build utils)
   #:use-module (guix build-system cmake)
@@ -46,9 +47,10 @@ (define-module (gnu packages polkit)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages qt)
-  #:use-module (gnu packages xml))
+  #:use-module (gnu packages xml)
+  #:export (polkit))
 
-(define-public polkit
+(define-public polkit*
   (package
 (name "polkit")
 (version "0.120")
@@ -151,32 +153,50 @@ (define-public polkit
 ;;; Variant of polkit built with Duktape, a lighter JavaScript engine compared
 ;;; to mozjs.
 (define-public polkit-duktape
-  (package/inherit polkit
-(name "polkit-duktape")
-(source
- (origin
-   (inherit (package-source polkit))
-   (patches
-(append
-(search-patches "polkit-use-duktape.patch")
-(origin-patches (package-source polkit))
-(arguments
- (substitute-keyword-arguments (package-arguments polkit)
-   ((#:configure-flags flags)
-`(cons "--with-duktape" ,flags))
-   ((#:phases phases)
-`(modify-phases ,phases
-   (add-after 'unpack 'force-gnu-build-system-bootstrap
- (lambda _
-   (delete-file "configure")))
-(native-inputs
- (append `(("autoconf" ,autoconf)
-   ("automake" ,automake)
-   ("libtool" ,libtool)
-   ("pkg-config" ,pkg-config))
- (package-native-inputs polkit)))
-(inputs (alist-replace "mozjs" `(,duktape)
-   (package-inputs polkit)
+  (let ((base polkit*))
+(package/inherit base
+  (name "polkit-duktape")
+  (source
+   (origin
+ (inherit (package-source base))
+ (patches
+  (append
+  (search-patches "polkit-use-duktape.patch")
+  (origin-patches (package-source base))
+  (arguments
+   (substitute-keyword-arguments (package-arguments base)
+ ((#:configure-flags flags)
+  `(cons "--with-duktape" ,flags))
+ ((#:phases phases)
+  `(modify-phases ,phases
+ (add-after 'unpack 'force-gnu-build-system-bootstrap
+   (lambda _
+ (delete-file "configure")))
+  (native-inputs
+   (append `(("autoconf" ,autoconf)
+ ("automake" ,automake)
+ ("libtool" ,libtool)
+ 

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[zimoun]

Hi Denis,

On Wed, 24 Nov 2021 at 00:51, Denis 'GNUtoo' Carikli
 wrote:

> If I understood correctly Florian, the argument here is that it is safe
> to redistribute source code under a GPL incompatible license that links
> to GPL code because it's in source form?

Maybe you could be interested by these explanations:



Cheers,
simon

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[Denis 'GNUtoo' Carikli]

On Wed, 24 Nov 2021 00:50:04 +0100
Denis 'GNUtoo' Carikli  wrote:

> Hi,
> 
> On Tue, 23 Nov 2021 18:29:22 +0100
> Ludovic Courtès  wrote:
> 
> > When consensus cannot be met, maintainers have the last say.
> > 
> > But again, my understanding is that there’s no new decision to be
> > made here.
> If I understood correctly Florian, the argument here is that it is
> safe to redistribute source code under a GPL incompatible license
> that links to GPL code because it's in source form?
After having thought about it, I think I think I found a rationale that
could make sense.

Maybe it's because if the code is in source form, both code are not
combined in the same binary. 

If that's the case then it would also be legal to redistribute binaries
too as long as they are dynamically linked as the linking happens at
runtime.

And that should also not be limited to Linux. 

We could then through dlopen or any similar method reuse code from
one binary in the other because the linking is dynamic as long as both
binaries are not linked together statically?

Or am I on the wrong track here?

Denis.


pgpE4V5jiUsd6.pgp
Description: OpenPGP digital signature

Re: Proposal: Build timers

[Julien Lepiller]

Do we even care that much about accuracy? I don't really care that the build 
takes 30 or 31 seconds, or even 1 minute, but I certainly care whether it takes 
30s or 3h. I think this is also what SBUs give you: a rough estimate of which 
build is longer than the other. I think a simple proportionality relation would 
work well enough in most common cases. It might be quite off on a super 
computer, but who cares, really?

Le 23 novembre 2021 16:35:24 GMT-05:00, Jacob Hrbek  a 
écrit :
>Skimming through the research that lily provided our builds are reproducible 
>so the changes in cpu cycles requirements should be same with any post-build 
>implementation disabled, but i recognize that different CPUs might use 
>different configuration that influences the calculation and it will be a 
>complicated task to account for all variables that influence the build across 
>systems so instead of accurate measurements we should work with a sane 
>tolerance for accuracy.
>
>-- Jacob "Kreyren" Hrbek
>
>Sent with ProtonMail Secure Email.
>
>‐‐‐ Original Message ‐‐‐
>
>On Tuesday, November 23rd, 2021 at 8:09 PM, Liliana Marie Prikler 
> wrote:
>
>> Am Montag, den 22.11.2021, 22:02 + schrieb Jacob Hrbek:
>> 
>
>> > See the proposal in https://git.dotya.ml/guix.next/GUIX.next/issues/5
>> > 
>
>> > -- Jacob "Kreyren" Hrbek
>> > 
>
>> > Sent with ProtonMail Secure Email.
>> 
>
>> Your Pokémon analogy is extremely flawed. The same CPU at a different
>> 
>
>> clockrate does not perform the same task in the same amount of cycles
>> 
>
>> [1, 2].
>> 
>
>> [1] Kotla, Ramakrishna & Devgan, Anirudh & Ghiasi, Soraya & Keller, Tom
>> 
>
>> & Rawson, Freeman. (2004). Characterizing the impact of different
>> 
>
>> memory-intensity levels. 3 - 10. 10.1109/WWC.2004.1437388.
>> 
>
>> [2] Snowdon, David & Sueur, Etienne & Petters, Stefan & Heiser, Gernot.
>> 
>
>> (2009). Koala a platform for OS-level power management. Proceedings of
>> 
>
>> the 4th ACM European Conference on Computer Systems, EuroSys'09. 289-
>> 
>
>> 302. 10.1145/1519065.1519097.

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[Denis 'GNUtoo' Carikli]

Hi,

On Tue, 23 Nov 2021 18:29:22 +0100
Ludovic Courtès  wrote:

> When consensus cannot be met, maintainers have the last say.
> 
> But again, my understanding is that there’s no new decision to be made
> here.
If I understood correctly Florian, the argument here is that it is safe
to redistribute source code under a GPL incompatible license that links
to GPL code because it's in source form?

Denis.


pgptAYKX6Bb82.pgp
Description: OpenPGP digital signature

Re: Proposal: Build timers

[Jacob Hrbek]

Skimming through the research that lily provided our builds are reproducible so 
the changes in cpu cycles requirements should be same with any post-build 
implementation disabled, but i recognize that different CPUs might use 
different configuration that influences the calculation and it will be a 
complicated task to account for all variables that influence the build across 
systems so instead of accurate measurements we should work with a sane 
tolerance for accuracy.

-- Jacob "Kreyren" Hrbek

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐

On Tuesday, November 23rd, 2021 at 8:09 PM, Liliana Marie Prikler 
 wrote:

> Am Montag, den 22.11.2021, 22:02 + schrieb Jacob Hrbek:
> 

> > See the proposal in https://git.dotya.ml/guix.next/GUIX.next/issues/5
> > 

> > -- Jacob "Kreyren" Hrbek
> > 

> > Sent with ProtonMail Secure Email.
> 

> Your Pokémon analogy is extremely flawed. The same CPU at a different
> 

> clockrate does not perform the same task in the same amount of cycles
> 

> [1, 2].
> 

> [1] Kotla, Ramakrishna & Devgan, Anirudh & Ghiasi, Soraya & Keller, Tom
> 

> & Rawson, Freeman. (2004). Characterizing the impact of different
> 

> memory-intensity levels. 3 - 10. 10.1109/WWC.2004.1437388.
> 

> [2] Snowdon, David & Sueur, Etienne & Petters, Stefan & Heiser, Gernot.
> 

> (2009). Koala a platform for OS-level power management. Proceedings of
> 

> the 4th ACM European Conference on Computer Systems, EuroSys'09. 289-
> 

> 302. 10.1145/1519065.1519097.

publickey - kreyren@rixotstudio.cz - 0x1677DB82.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Proposal: Build timers

[Jacob Hrbek]

> Your Pokémon analogy is extremely flawed.  The same CPU at a different 
> clockrate does not perform the same task in the same amount of cycles [1, 2]. 
> -- lily

The theory is that the measurements could be taken after X amount of time to 
adjust the DPS that the CPU does to the package to get the build time, See 
'theory for real time measurements' in 
https://git.dotya.ml/guix.next/GUIX.next/issues/5

using the previous theory using the reproduction build with limited system 
resources to define the package complexity and to provide coordinates for XY 
plane this could each be define per variable that influences the build and then 
define the relation to determine the DPS which so far seem as the most accurate 
in my testing, but I am still experimenting with it

e.g:
Base damage: 1
- CPU at 1 Ghz -> Multiplier 0.24
- CPU at 2 Ghz -> Multiplier 0.31

But the current question should be: Do we want these timers implemented 
assuming that it can be done without overhead and in a cheap way? and if so how 
do we want to store and redistribute the logged data as those have a major 
impact on the calculation and methods used.

-- Jacob "Kreyren" Hrbek

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐

On Tuesday, November 23rd, 2021 at 8:09 PM, Liliana Marie Prikler 
 wrote:

> Am Montag, den 22.11.2021, 22:02 + schrieb Jacob Hrbek:
> 

> > See the proposal in https://git.dotya.ml/guix.next/GUIX.next/issues/5
> > 

> > -- Jacob "Kreyren" Hrbek
> > 

> > Sent with ProtonMail Secure Email.
> 

> Your Pokémon analogy is extremely flawed. The same CPU at a different
> 

> clockrate does not perform the same task in the same amount of cycles
> 

> [1, 2].
> 

> [1] Kotla, Ramakrishna & Devgan, Anirudh & Ghiasi, Soraya & Keller, Tom
> 

> & Rawson, Freeman. (2004). Characterizing the impact of different
> 

> memory-intensity levels. 3 - 10. 10.1109/WWC.2004.1437388.
> 

> [2] Snowdon, David & Sueur, Etienne & Petters, Stefan & Heiser, Gernot.
> 

> (2009). Koala a platform for OS-level power management. Proceedings of
> 

> the 4th ACM European Conference on Computer Systems, EuroSys'09. 289-
> 

> 302. 10.1145/1519065.1519097.

publickey - kreyren@rixotstudio.cz - 0x1677DB82.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Replacing polkit by polkit-duktape on core-updates-frozen ?

[Thiago Jung Bauermann]

Hello,

Em terça-feira, 23 de novembro de 2021, às 14:18:10 -03, Ludovic Courtès 
escreveu:
> > But my "test" fails the same:
> > 
> > $ ./pre-inst-env guix build --system=aarch64-linux \
> > 
> > -e '(@ (gnu packages polkit) polkit)' -n |& grep polkit
> > 
> > /gnu/store/dw11y85xfsb8hcg7w2cw57f1xfs4i74m-polkit-0.120.drv
> > /gnu/store/ric7yf4ra2p14p29fwsh18m1nakciakv-polkit-0.120.tar.xz
> 
> That’s expected because here you’re effectively calling
> (%current-system) from the top level, and that’s x86_64.
> 
> A good test is to try and build one of its dependents:
> 
>   guix build -s aarch64-linux gnome-control-center -n
> 
> should list polkit-duktape.
> 
> HTH!

Sorry, I should have taken a closer look at this patch yesterday. IIUC it’s 
a bug to test (%current-system) before (%current-target-system). That’s 
because the latter is only defined for cross-builds, while the former is 
always defined. So in practice (%current-target-system) will never be 
checked.

This what patch 1 in the series at https://issues.guix.gnu.org/49672 fixes.
Rebasing it and addressing Maxime’s comments is on my todo list.

> Are we done and ready for merging once this patch has been applied to
> ‘core-updates-frozen’?

Hooray! I’m currently investigating a problem with lualatex (issue 51252). 
I don’t know whether it’s serious enough to delay the merge.

-- 
Thanks,
Thiago

Re: Proposal: Build timers

[Liliana Marie Prikler]

Am Montag, den 22.11.2021, 22:02 + schrieb Jacob Hrbek:
> See the proposal in https://git.dotya.ml/guix.next/GUIX.next/issues/5
> 
> -- Jacob "Kreyren" Hrbek
> 
> Sent with ProtonMail Secure Email.
Your Pokémon analogy is extremely flawed.  The same CPU at a different
clockrate does not perform the same task in the same amount of cycles
[1, 2].

[1] Kotla, Ramakrishna & Devgan, Anirudh & Ghiasi, Soraya & Keller, Tom
& Rawson, Freeman. (2004). Characterizing the impact of different
memory-intensity levels. 3 - 10. 10.1109/WWC.2004.1437388. 
[2] Snowdon, David & Sueur, Etienne & Petters, Stefan & Heiser, Gernot.
(2009). Koala a platform for OS-level power management. Proceedings of
the 4th ACM European Conference on Computer Systems, EuroSys'09. 289-
302. 10.1145/1519065.1519097. 

Re: Help with package AppImage support

[Ekaitz Zarraga]

> Could you show the CMake output?

I attach it in this email.
It's a "TARGET NOT FOUND" :S

> Sometimes there’s a -D flag you need to pass to help it find the
> package. (I’m not much of a CMake person though…)

yeah... I tried many, but I didn't find the one.

> Thanks for working on it!

Thank you for the help Ludo

Ekaitz

884g21d0ixc86sbnnzhj8yqyf1cyzj-appimagekit-13.drv.bz2
Description: application/bzip

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[Ludovic Courtès]

Hi,

"pelzflorian (Florian Pelz)"  skribis:

> On Sun, Nov 21, 2021 at 11:54:15AM +0100, pelzflorian (Florian Pelz) wrote:
>> raid5atemyhomework wrote patches to add ZFS to Guix
>> .  I put them in CC.  That there is
>> no decision on ZFS and their patches is bad.  Maybe their patches
>> would be for the RFC model to decide?

The decision to include zfs-on-linux, but not provide substitutes, was
made long ago.

Then there was a discussion in this very thread where I think (but I
don’t remember precisely) there was consensus on not having GNOME depend
on zfs-on-linux, because that’s just inconvenient as everyone would have
to build zfs-on-linux from source.

As for raid5atemyhomework’s patches, it’s just that reviewing such
patches takes time, a number of round trips, and that it’d be better to
get input from zfs-on-linux users.  I don’t think the patches change
anything to the initial decision above.

> Then who decides?

When consensus cannot be met, maintainers have the last say.

But again, my understanding is that there’s no new decision to be made
here.

Rather, there’s review work to be completed—Maxime already did a lot of
that—and commitment from ZFS-savvy people to maintain it going forward.

I hope this clarifies the situation a bit!

Ludo’.

Re: Help with package AppImage support

[Ludovic Courtès]

Hello!

Ekaitz Zarraga  skribis:

> I having issues with AppImageKit (see below)... I'm not able to make it load
> the libsquashfuse I already packaged. CMake refuses to find it.
>
> I could use a hand to make this package work or at least some help with CMake,
> because I'm not used to work with it and I know some of you have dealt with it
> much more than what I did.

Could you show the CMake output?

Sometimes there’s a -D flag you need to pass to help it find the
package.  (I’m not much of a CMake person though…)

Thanks for working on it!

Ludo’.

Re: Replacing polkit by polkit-duktape on core-updates-frozen ?

[Ludovic Courtès]

Hi,

Maxim Cournoyer  skribis:

> gnu: polkit: Define polkit package variable based on architecture.
>
> * gnu/packages/polkit.scm (polkit): Rename to...
> (polkit*): ... this.
> (polkit-duktape): Adjust to inherit from polkit*.
> (polkit-for-system): New procedure.
> (polkit): New variable.

LGTM!

> But my "test" fails the same:
>
> $ ./pre-inst-env guix build --system=aarch64-linux \
> -e '(@ (gnu packages polkit) polkit)' -n |& grep polkit
> /gnu/store/dw11y85xfsb8hcg7w2cw57f1xfs4i74m-polkit-0.120.drv
> /gnu/store/ric7yf4ra2p14p29fwsh18m1nakciakv-polkit-0.120.tar.xz

That’s expected because here you’re effectively calling
(%current-system) from the top level, and that’s x86_64.

A good test is to try and build one of its dependents:

  guix build -s aarch64-linux gnome-control-center -n

should list polkit-duktape.

HTH!

Are we done and ready for merging once this patch has been applied to
‘core-updates-frozen’?

Thanks,
Ludo’.

Re: Using G-Expressions for public keys (substitutes and possibly more)

[Ludovic Courtès]

Liliana Marie Prikler  skribis:

> Am Montag, den 22.11.2021, 14:27 +0100 schrieb Ludovic Courtès:

[...]

> sexp->canonical-sexp is from the comment surrounding it a rather hacky
> beast.  I think (guile-)gcrypt could very well benefit from having a
> better sexp construction API, but again that's outside the scope of
> Guix.
>
> But since you're asking, I do feel there's a lot guile-gcrypt could
> borrow from Guix, just not right now in an efficient manner thanks to
> needing to jump through the sexp->canonical-sexp hoops.  It could have
> semantic constructors and field sanitization à la (guix records) for
> example.  WDYT?

Fundamentally, Guile-Gcrypt is just bindings to libgcrypt.  As such, it
tries hard to remain close to libgcrypt’s API, and I think that’s a safe
approach for bindings in general.

Libgcrypt internally uses canonical sexps for key material,
configuration options, etc. (an illustration of Greenspun’s Tenth Rule
:-)), so that’s what Guile-Gcrypt exposes, and I think it’s fine.

Thanks,
Ludo’.

Re: Flag day for simplified package inputs

[Ludovic Courtès]

Hi,

Jelle Licht  skribis:

> Ludovic Courtès  writes:
>
>> Hi,
>>
>> Jelle Licht  skribis:
>>
>>> When applying this and future bulk changes, could we perhaps list the
>>> specific commits (+ commented shortlog line) in a file. To clarify, if
>>> we were to store these commits in `.git-blame-ignore-revs', later on we
>>> can instruct users to run:
>>>
>>> git config blame.ignoreRevsFile .git-blame-ignore-revs
>>>
>>> to ignore the bulk change for git blame purposes.
>>
>> Sounds like a good idea.  There’s no particular convention regarding the
>> file name of ‘.git-blame-ignore-revs’, right?
>
> As far as I know there is no established convention, although after
> searching around a bit I found that Sarah Morgensen proposed the exact
> same thing in September. It seems the chromium project also follows the
> same naming scheme, so we won't be totally idiosyncratic at least :).

Good.  :-)  Could you (or Sarah?) propose a patch, including a sentence
or two in the manual with the ‘git config’ snippet?

Thanks,
Ludo’.

Re: ZFS part of Guix? RFC? (Re: Effectively force all GNOME users to locally compile ZFS?)

[Denis 'GNUtoo' Carikli]

On Mon, 22 Nov 2021 19:10:48 +0100
"pelzflorian (Florian Pelz)"  wrote:
> I don’t actually care about ZFS myself, but there should be a decision
> because I think current badly supported ZFS makes people here unhappy.
For people that really want the ZFS kernel module, would using an extra
channel for that be a big issue?

As I understand the ZFS module is already built from source so
maybe there is a way to make sure that everything that depends on zfs is
not rebuilt.

Or would there be other concerns for users of the ZFS kernel module if
the module is moved in an unofficial channel?

As long as the module isn't provided, it should also be safe to
maintain the rest (like the services), though I've no idea if doing
that in Guix is desirable or not.

Denis.


pgpE88TrMqbz6.pgp
Description: OpenPGP digital signature

Re: Proposal: Build timers

[zimoun]

Hi,

On Tue, 23 Nov 2021 at 07:05, Julien Lepiller  wrote:

> LFS has a notion of a Standard Build Unit (SBU), that is the build
> time of a particular package on your machine. Each package build time
> is estimated in SBU. However, they do not take threads into account,
> because the relation is not exactly proportional (some parts are
> linear, there is some overhead, …). SBUs change quite often with
> versions, so I don't think averaging on different versions/derivations
> would make a lot of sense… But I suppose this info could help
> determine how long it should take to build the same derivation or a
> similar one.

Thanks for the SBU pointer.  This webpage [1] provides some details.
But I do not find the mentioned table.  I would see more numbers about
averages and standard derivation.

Moreover, I am confused, here [2] GCC@11.2.00 requires 26+56 SBU.  And
there [3] it says 164 SBU.  Why is it so different?


I am not convinced such SBUs makes sense for x86_64 because for instance
my laptop (4 cores+SSD) is different from my desktop (4 cores+HDD) and
really different from my workstation (64 cores+mix).  From my
understanding, this SBU makes sense in the LFS context because machines
are, from my opinion, not so much heterogeneous.

However, other architectures are generally more homogeneous machines.
And because substitutes are less available, maybe such unit could be
worth as a rough indicator.


1: 
2: 
3: 

Cheers,
simon

Re: Proposal: Build timers

[Jacob Hrbek]

> Are you assuming here that the two machines are the same?  Or are they 
> different?

I am assuming that the two machines are the same with the exception of cpu 
frequency and threads that are used as a variable to assume the build time from 
known value.

> This approximation would not even be accurate enough for the same machine.  
> For instance, the test suite of the julia package runs mainly sequential 
> using one thread...

I am aware of this scenario and I adapted the equasion for it, but I recognize 
that this exponentially increases the inaccuracy with more threads and I don't 
believe that there is a mathematical way with the provided values to handle 
that scenario so we would have to adjust the calculation for those packages. 


Alternative and more robust solution would be to build the package with 
`--jobs` set on "cpu threads - 1" and then compare the build time to a scenario 
without this set which would give us two coordinates to be interpreted in an 
exponential and eventually quadratic or even logarithmic function. As we 
already have to build the package multiple times to establish reproducibility 
so this shouldn't add any overhead just slightly slower reproduction build.

The exponential function would be less vulnerable to tasks that are unable to 
utilize all available processing resources, but it's more inaccurate so I 
assume that we would eventually have to use quadratic/logarithmic definition 
that uses the previously provided equasion on both coordinates to determine 
build on different system which to me seems as a robust in my current testing, 
but I am willing to give it more time if we (me and other GNU Guix 
devs/contributors) agree that this would be a mergable effort as it would 
basically require a one value preferably stored in the guix repo that is used 
to calculate the build time.

-- Jacob "Kreyren" Hrbek

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐

On Tuesday, November 23rd, 2021 at 11:56 AM, zimoun  
wrote:

> Hi,
> 

> On Tue, 23 Nov 2021 at 06:21, Jacob Hrbek krey...@rixotstudio.cz wrote:
> 

> > 1.  locally: Storing the value somewhere on the system and adding up to
> > 

> > it each build to provide more accurate average.
> 

> Timing is already stored, see “guix build --log-file”. Give a look at
> 

> ’/var/log/guix/drvs’. For instance,
> 

> --8<---cut here---start->8---
> 

> $ bzcat 
> /var/log/guix/drvs/aq/abymi9yk7pv89614dcdfll3hh4i5mc-julia-1.5.3.drv.bz2 | 
> grep phase | grep seconds
> 

> phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds phase` set-paths' 
> succeeded after 0.0 seconds
> 

> phase `install-locale' succeeded after 0.0 seconds phase` unpack' succeeded 
> after 1.1 seconds
> 

> phase `use-system-libwhich' succeeded after 0.0 seconds phase` 
> disable-documentation' succeeded after 0.0 seconds
> 

> phase `prepare-deps' succeeded after 0.0 seconds phase` bootstrap' succeeded 
> after 0.0 seconds
> 

> phase `patch-usr-bin-file' succeeded after 0.0 seconds phase` 
> patch-source-shebangs' succeeded after 0.2 seconds
> 

> phase `patch-generated-file-shebangs' succeeded after 0.0 seconds phase` 
> fix-include-and-link-paths' succeeded after 0.0 seconds
> 

> phase `replace-default-shell' succeeded after 0.0 seconds phase` 
> fix-precompile' succeeded after 0.0 seconds
> 

> phase `build' succeeded after 354.3 seconds phase` set-home' succeeded after 
> 0.0 seconds
> 

> phase `disable-broken-tests' succeeded after 0.0 seconds phase` check' 
> succeeded after 7428.8 seconds
> 

> phase `install' succeeded after 16.0 seconds phase` make-wrapper' succeeded 
> after 0.0 seconds
> 

> phase `patch-shebangs' succeeded after 0.0 seconds phase` strip' succeeded 
> after 0.0 seconds
> 

> phase `validate-runpath' succeeded after 0.0 seconds phase` 
> validate-documentation-location' succeeded after 0.0 seconds
> 

> phase `delete-info-dir-file' succeeded after 0.0 seconds phase` 
> patch-dot-desktop-files' succeeded after 0.0 seconds
> 

> phase `install-license-files' succeeded after 0.0 seconds phase` 
> reset-gzip-timestamps' succeeded after 0.0 seconds
> 

> phase `compress-documentation' succeeded after 0.0 seconds
> 

> --8<---cut here---end--->8---
> 

> Therefore, you need to extract somehow that information.
> 

> > optionally This local database can be shared across multiple
> > 

> > systems that add values to it like simple listener waiting for POST
> > 

> > requests.
> 

> It should be better to use a content-addressed distribution such as IPFS
> 

> or GNUnet, IMHO.
> 

> > -   within the guix repo: Since we are already building the package we
> > 

> > can take the time and then do the provided math in reverse to
> > 

> > calculate the time:
> > 

> > Build took 100 sec on system with 8 threads at 2.4 Ghz max cpu 
> > frequency:
> > 

> > 100 * (2.4 * 8) = 1920 (build time value per 

Re: Proposal: Build timers

[Julien Lepiller]

Le 23 novembre 2021 01:21:06 GMT-05:00, Jacob Hrbek  a 
écrit :
>I think you are overcomplicating the implementation.. What I am proposing is 
>to store the time value before and after the build and then log the 
>subtraction of these two values per package (or even per package's phase).
>
>For storage it can be either/both:
>1. locally: Storing the value somewhere on the system and adding up to it each 
>build to provide more accurate average.
>
>**optionally** This local database can be shared across multiple systems that 
>add values to it like simple listener waiting for POST requests.

We already log time on cuirass, but we don't use this information at all. If 
you could provide some wip code to show how you would implement the feature 
with this info, that would be great for this discussion.

For jocal logs, there is the store database (in /var/guix) for instance, though 
running the gc will also erase the info along with the store item.

>- within the guix repo: Since we are already building the package we can take 
>the time and then do the provided math in reverse to calculate the time:
>
>Build took 100 sec on system with 8 threads at 2.4 Ghz max cpu frequency:
>
>100 * (2.4 * 8) = 1920 (build time value per one thread at 1 Ghz)
>
>Building the package on system with 2 threads at 2.4 Ghz max cpu frequency:
>
>1920 / (2 * 2.4) = 400
>
>We can then assume that the build will take 1920/400=4.8 -> 4.8 times 
> longer on this system.

LFS has a notion of a Standard Build Unit (SBU), that is the build time of a 
particular package on your machine. Each package build time is estimated in 
SBU. However, they do not take threads into account, because the relation is 
not exactly proportional (some parts are linear, there is some overhead, …). 
SBUs change quite often with versions, so I don't think averaging on different 
versions/derivations would make a lot of sense… But I suppose this info could 
help determine how long it should take to build the same derivation or a 
similar one.

Re: Proposal: Build timers

[zimoun]

Hi,

On Tue, 23 Nov 2021 at 06:21, Jacob Hrbek  wrote:

> 1. locally: Storing the value somewhere on the system and adding up to
> it each build to provide more accurate average.

Timing is already stored, see “guix build --log-file”.  Give a look at
’/var/log/guix/drvs’.  For instance,

--8<---cut here---start->8---
$ bzcat 
/var/log/guix/drvs/aq/abymi9yk7pv89614dcdfll3hh4i5mc-julia-1.5.3.drv.bz2 | grep 
phase | grep seconds
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
phase `set-paths' succeeded after 0.0 seconds
phase `install-locale' succeeded after 0.0 seconds
phase `unpack' succeeded after 1.1 seconds
phase `use-system-libwhich' succeeded after 0.0 seconds
phase `disable-documentation' succeeded after 0.0 seconds
phase `prepare-deps' succeeded after 0.0 seconds
phase `bootstrap' succeeded after 0.0 seconds
phase `patch-usr-bin-file' succeeded after 0.0 seconds
phase `patch-source-shebangs' succeeded after 0.2 seconds
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
phase `fix-include-and-link-paths' succeeded after 0.0 seconds
phase `replace-default-shell' succeeded after 0.0 seconds
phase `fix-precompile' succeeded after 0.0 seconds
phase `build' succeeded after 354.3 seconds
phase `set-home' succeeded after 0.0 seconds
phase `disable-broken-tests' succeeded after 0.0 seconds
phase `check' succeeded after 7428.8 seconds
phase `install' succeeded after 16.0 seconds
phase `make-wrapper' succeeded after 0.0 seconds
phase `patch-shebangs' succeeded after 0.0 seconds
phase `strip' succeeded after 0.0 seconds
phase `validate-runpath' succeeded after 0.0 seconds
phase `validate-documentation-location' succeeded after 0.0 seconds
phase `delete-info-dir-file' succeeded after 0.0 seconds
phase `patch-dot-desktop-files' succeeded after 0.0 seconds
phase `install-license-files' succeeded after 0.0 seconds
phase `reset-gzip-timestamps' succeeded after 0.0 seconds
phase `compress-documentation' succeeded after 0.0 seconds
--8<---cut here---end--->8---

Therefore, you need to extract somehow that information.


> **optionally** This local database can be shared across multiple
> systems that add values to it like simple listener waiting for POST
> requests.

It should be better to use a content-addressed distribution such as IPFS
or GNUnet, IMHO.


> - within the guix repo: Since we are already building the package we
> can take the time and then do the provided math in reverse to
> calculate the time: 
>
> Build took 100 sec on system with 8 threads at 2.4 Ghz max cpu frequency:
>
> 100 * (2.4 * 8) = 1920 (build time value per one thread at 1 Ghz)
>
> Building the package on system with 2 threads at 2.4 Ghz max cpu 
> frequency:
>
> 1920 / (2 * 2.4) = 400
>
> We can then assume that the build will take 1920/400=4.8 -> 4.8
> times longer on this system. 

Are you assuming here that the two machines are the same?  Or are they
different?

This approximation would not even be accurate enough for the same
machine.  For instance, the test suite of the julia package runs mainly
sequential using one thread.  If you go back to numbers above,
build=354.3 seconds and check=7428.8 seconds, so the number of threads
only tweaks timing of build phase, which will not impact much the
overall timing.

Somehow, IIUC your proposal, you would like, based on timings from
machine A about a set of packages, and timings from machine B about the
same set of packages, knowing the timing of machine B for package foo,
then extrapolate timing for machine A of package foo.  The maths for
that are not linear, IMHO, and require “complicated” heuristics.  It is
not that complicated, it “just” require some statistical regression –
though it is not straightforward either. :-)

BTW, why not directly substitute package foo from machine B?


> The math might need to be adjusted, but it seems to be sufficiently
> accurate through my testing, fwiw I see that `(max cpu frequency * cpu
> threads)` is an important component of the equasion using the analogy
> of a (possibly silly) "pokemon battle" assuming interpreting a
> mathematical equasion to define the Health Points of the package and
> damage per second of the CPU then simply substracting these two values
> to determine how long it will take to build alike package has 500 HP
> -> Needs a CPU that deals 100 HP to complete in 5 sec or CPU that
> deals 50 HP to finish in 10 sec.

I will be happy if I am wrong.  I guess this back-to-envelope would be
not accurate enough; for two reasons.  As I said elsewhere, to your
example value of 100 seconds is attached a strong variability, depending
one on how the package itself scales at build time and more than often
this scaling is not linear versus the number of threads – from my
experience; and two on the stressed context where the build happens.


> About accuracy: I highly doubt that we need to worry about the system
> noise as that