Re: Improving the speed of guix time-machine for development environments
Hi Richard, Richard Sent skribis: > If I want to use time-machine as part of entering a development > environment for some channel collection and a new guix commit is pushed, > then the next time I invoke that same time-machine command there will be > a large delay as Guix fetches, authenticates, computes, and builds an > entirely new derivation. With your proposal, I'd only avoid that problem > if I coincidentally happened to run guix pull in-between. > > I don't believe hard-pinning the guix channel is an appropriate solution > in this case since it has several drawbacks as discussed in [1]. I’m not sure I understand the use case. Two different but “common” (?) use cases come to mind. First one is when you want to share the exact same environment within a team of developers, say. In that case, you have a ‘channels.scm’ that pins commits. The first ‘guix time-machine’ is expensive but subsequent invocations are instantaneous (cached). Occasionally, the team updates ‘channels.scm’ to point to a more recent commit of Guix and other channels. Second one is CI-style: you want to build a package against the latest commit of each relevant channel, all the time. In that case, you pay the price on every ‘guix time-machine’ invocation. You seem to be looking for something kinda “in between”, is that right? Ludo’.
Re: CI is not processing jobs
Hi, Ludovic Courtès skribis: > The longer story is that Ricardo noticed that the build backlog had been > growing for a couple of months (see “Pending builds”): > > https://ci.guix.gnu.org/metrics > > We discussed this on guix-sysadmin and found that this was due to the > poor performance of a SQL query at the core of ‘cuirass remote-server’ > that was roughly linear in the number of packages in the database. With > help from Chris Baines, this is now fixed: > > > https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=17338588d4862b04e9e405c1244a2ea703b50d98 This is now deployed on ci.guix.gnu.org. So far all the workers are kept busy. We’ll have to see if it keeps up over time. Ludo’.
Re: Come watch a live stream coding session for the Hurd Video
On Sun, Jun 2, 2024 at 12:22 AM Joshua Branson wrote:
> So we had an awesome time today watching Sergey code a trivial translator (1)
> and do
> some glibc hacking (2). Sergey coded and chatted for 4 and 1/2 hours! Three
> cheers
> for that kind of commitment! Thanks pal!
>
> In the livestream today, Sergey wrote caesar.c, which implements a simple
> caeser
> cipher. It's a toy. A caesar cipher is EASY to break, but it was fun watching
> him code it out!
Hi all,
thanks for attending, and thanks Joshua for organizing it!
Let's do this again sometime? -- hopefully with less technical issues
related to recording/audio/video. I've got plenty of exciting ideas of
things to do. There are projects I started but haven't completed, like
the new bootstrap process (that Josh keeps calling "serverboot v2") or
the new in-tree, Hurd-native libfuse (which is an full translator
framework in its own right, a peer to libdiskfs/libnetfs/libtrivfs,
but mostly API-compatible with the Linux libfuse), or epoll/Wayland
(which is mostly complete, but it needs to be updated / cleaned up,
and published). Or, we could get started on writing that shiny new
translator framework in Rust :)
We ended the stream on a somewhat of a cliffhanger: can we run
caesarfs (see, Joshua misspelled it too, so it's not just me!) on the
aarch64-gnu system? The process was getting created, but then it
crashed before it got a chance to handshake with its parent translator
(root ext2fs), and thus fake-console-run.c was getting EDIED trying to
open the file. Turns out, we need to explicitly null-terminate the
last argv entry too when setting a static translator record from my
GNU/Linux host, so instead of
$ sudo setfattr -n gnu.translator -v
'/hurd/caesar\000/libexec/hello-world.txt'
/mnt/libexec/hello-world.txt.csr
I should have done
$ sudo setfattr -n gnu.translator -v
'/hurd/caesar\000/libexec/hello-world.txt\000'
/mnt/libexec/hello-world.txt.csr
It was crashing inside ld-arrach64.so.1 trying to parse its argv,
since it expected them to be in the argz format, so null-terminated.
(Did I mention how incredibly useful being able to backtrace through
syscall/fault/interrupt boundaries is for debugging?)
With that fixed (no changes to the translator itself, but the
translator record changed as shown above), I do get:
GNU Mach 1.8
Kernel command line: foo=bar
Booting in EL1
vm_page: page table size: 262144 entries (20480k)
vm_page: DMA: pages: 262144 (1024M), free: 221873 (866M)
vm_page: DMA: min:13107 low:15728 high:26214
Model name: linux dummy-virt
module 0: rootfs $(rootfs-device=ramdisk-create)
rd0: 36700160 bytes @83424000
module 1: ld-aarch64.so.1 /hurd/exec $(exec-task=task-create)
module 2: ext2fs --host-priv-port=${host-port}
--device-master-port=${device-port} --exec-server-task=${exec-task}
--multiboot-command-line=${kernel-command-line} -T device
${rootfs-device} $(task-create) $(task-resume)
3 bootstrap modules
task loaded: ld-aarch64.so.1 /hurd/exec
task loaded: ext2fs --host-priv-port=1 --device-master-port=2
--exec-server-task=3 --multiboot-command-line=foo=bar -T device rd0
start ext2fs: Hello world!
fread () -> 5
Uryyb
"Uryyb" is of course "Hello" with ROT13 applied :) So we were very close.
Sergey
Re: Improving the speed of guix time-machine for development environments
Hi Ludo, Ludovic Courtès writes: > I’m not sure I understand the use case. Two different but “common” (?) > use cases come to mind. > > First one is when you want to share the exact same environment within a > team of developers, say. In that case, you have a ‘channels.scm’ that > pins commits. The first ‘guix time-machine’ is expensive but subsequent > invocations are instantaneous (cached). Occasionally, the team updates > ‘channels.scm’ to point to a more recent commit of Guix and other > channels. > > Second one is CI-style: you want to build a package against the latest > commit of each relevant channel, all the time. In that case, you pay > the price on every ‘guix time-machine’ invocation. > > You seem to be looking for something kinda “in between”, is that > right? Yes, that "in-between" use case is what I'm going for. Hard-pinning solves, for example, the "random 5 minute time-machine when invoking Geiser" problem (minus the first use), but introduces others that aren't universally desirable in a short-lived development environment. In other words, it's overkill sometimes. :) In a development environment, I want to achieve two things that are diametrically opposed: 1. Responsiveness/Speed of entry 1. Should be obvious why this is good ;) 2. Continual updates of channels unless explicitly pinned 1. Detect breakages faster without needing a dedicated Cuirass CI server. 2. Less maintainence burden to regularly update pinned commits, particularly as a channel dependency tree grows. I feel that soft-pinning strikes a balance between those two goals that is useful when dependencies on a frequently updated channel are unlikely to break between commits. Elaborating on 2.2, consider a dependency chain that looks like --8<---cut here---start->8--- ___ / \ / v Guix-->A-->B --8<---cut here---end--->8--- If B hard-pins Guix to "foo" and A is updated to Guix "bar", B needs to update their hard-pinned Guix commit in channels.scm to >= "bar" to have a functional development environment again. Alternatively, B could hard-pin A, but if A is a channel the developers of B control they probably want to always use the latest version. Now you're in a catch-22 situation of "Do we hard-pin Guix in B and manually update it every time A's requirements change, or do we not pin Guix at all and deal with regular 5 minute pauses during development?" As channels become more popular and dependency trees grow, this'll become even more annoying. If B is only importing a couple Guix modules that are unlikely to incompatibly change, it's basically wasted effort. This is outside the scope of this topic, but I think it could be interesting if the dependency channels in .guix-channel could specify minimum commits as well. -- Take it easy, Richard Sent Making my computer weirder one commit at a time.
Re: System deployment commands hoarding all RAM
Hi Sergio,
Continuing on from our out-of-band conversation about this topic, the
most likely cause of your RAM issue is that you use recursion
extensively, but not in tail position. This means that Guile cannot
optimize this recursion to re-use stack frames and thus your memory
usage is unbounded. I'll try to explain a bit more what that means in
this email in the hopes that you'll be able to resolve it. Forgive me
if I repeat things you already know; I want to make sure you have all
the information you need to solve your problem.
You know what recursion is: calling a function from inside itself (or
one of the functions it calls). You mentioned you've also heard of tail
calls. I'll go ahead and describe tail calls a couple ways just in case
these different descriptions help something click for you. I know I
needed to hear several different explanations before I really
understood them.
The common definition is that a tail call is a call made as the last
thing a function does. I think of it as calling a function when there's
no more work to do in the current function. Functions called in this
way are said to be "in tail position." Tail recursion is simply making
a recursive call in tail position. Here's a silly example defined first
without tail recursion, then with:
```
(define (add-x n x)
"Add @var{x} to @var{n} one at a time."
(if (= x 0)
n
(+ 1 (add-x n (- x 1)
(define (add-x-tail n x)
(if (= x 0)
n
(add-x-tail (+ n 1) (- x 1
```
An important note: while the recursive call is the last line of text in
the tail-recursive function definition, this isn't necessarily
required. That idea threw me off for quite some time. The following
function is also tail recursive and produces the same output as the
other two definitions:
```
(define (add-x-tail-2 n x)
(if (not (= x 0))
(add-x-tail-2 (+ n 1) (- x 1))
n))
```
Let's return to the first definition for now. You may notice that the
recursive call happens inside of a call to `+`. Because arguments have
to be fully evaluated to values in order to be passed to functions, the
`+` call cannot be fully evaluated without the result of the recursive
call. We therefore have to keep around the outer `add-x` call so that
we can fully evaluate `+` and return its value from `add-x`.
Let's walk through expanding a quick example, skipping some
intermediary evaluation to save space and using ellipses to replace
parts of the function we don't care about anymore but have to keep
around anyway:
```
(add-x 3 2)
-> ;; replace add-x with its body, replacing variable names with values
(if (= 2 0)
3
(+ 1 (add-x 3 1)))
-> ;; repeat the above steps, replacing the inner call to add-x with
its body and its variable names with their values
(... (+ 1 (if (= 1 0)
3
(+ 1 (add-x 3 0)
-> ;; and again
(... (+ 1 (... (+ 1 (if (= 0 0) 3) ;; we skip writing the second
arm because we don't evaluate it
-> ;; now that everything is fully expanded, we can begin evaluating in
ernest
(... (+ 1 (... (+ 1 3
->
(... (+ 1 4))
->
5
```
Compare that to the second function definition. In this definition, the
`+` and `-` calls both happen inside the call to `add-x-tail`. These
are operating on known values and can be fully evaluated before the
call to `add-x-tail`. That is, there is no information in the first
`add-x-tail` call that is required to use the result of the second (or
third, fourth, etc) call to `add-x-tail`. This reveals another way to
think of tail calls. Tail calls are calls for which all arguments are
known and whose result is immediately returned by the calling function.
Here's where we introduce a new term: recursive tail call optimization.
Tail calls and tail recursion just describe situations you find in
code. In and of themselves, they only talk about what code does on a
theoretical level. Tail call optimization is a way to take advantage of
the theoretical aspects I've been discussing. I've been framing tail
calls as calls which do not need any information from the function that
calls them. Recursive tail call optimization allows the compiler to
replace the arguments of the calling function with the new argument
values and then simply re-evaluate that function in-place. If you're
familiar with how computers operate at the level of machine code and
the stack, recursive tail call optimization is usually implemented by
moving the new arguments into the same argument registers as were used
for the first call to a function, then jumping back to the beginning of
the function's stack frame. Guile (and all Scheme implementations
compliant with R5RS or later standards) does something very similar.
We'll step through evaluation of our tail recursive definition with
this understanding in mind, as well as previous rules for making the
steps shorter:
```
(add-x-tail 3 2)
-> ;; replace add-x-tail with its body and replace variable names with
Re: A paper about Plan 9 and Guix
Ludovic Courtès writes: > Hi, > > Edouard Klein skribis: > >> I'll be presenting it not next week end, but the one after (12-14 April >> 2024). > > Yay, congrats! > Thanks :) >> I'd be happy if some of you would be so kind as to read it with their >> extensive knowledge of Guix, in case I've made a mistake somewhere. >> >> https://the-dam.org/docs/explanations/Plan9ListenOnLinux.html > > Interesting read! > > I wonder to what extent the combination of ‘make-inetd-constructor’ and > ‘least-authority-wrapper’ would fit the bill for you? (This is currently > used for the bitlbee, dicod, and rsync services.) It seems to address > the main shortcomings listed in Section 1. > I simply was not aware of the existence of least-authority-wrapper. It does look nicer that passing a slew of options to guix shell --container. It sure would be nice if shepherd could be used to manage those daemons, just to avoid having two concurrent systems doing the same kind of work, but I'd still need a way to monitor the /run/listen directory, and start and stop shepherd services on the fly. It is probably doable, but it is a huge refactor. > Thanks, > Ludo’.
Re: "guix pack -f docker" does too much work
Hello Ricardo, I greatly agree, it would be an awesome QOL improvement. Just want to mention that it might be nice to take inspiration from the Nix dockerTools, since they already have quite a lot of effort put into this. Including for example an option called `streamLayeredImage` [1] which doesn't generate a tarball at all, but rather a script that outputs the layers without assembling them, in a format which Docker or Podman can import without the huge intermediary file. i.e. $(guix pack ...) | docker load [1]: https://ryantm.github.io/nixpkgs/builders/images/dockertools/#ssec-pkgs-dockerTools-streamLayeredImage So that'd allow Guix to skip generating the final tarball altogether, which makes packing very swift. Also seems that Nix's way only quickly imports the changed layers? And Guix's always imports the whole thing, at least I think? Reading through how they do it, it seems that they pass the raw store paths to this python script [2] and it does the rest? Save for figuring out some merging of paths since there's a limit to the number of layers, I don't think this would be too difficult to port (after we find what license the script is under at least, or replicate the behaviour in Guile). [2]: https://github.com/NixOS/nixpkgs/blob/90509d6d66eb1524e2798a2a8627f44ae413f174/pkgs/build-support/docker/stream_layered_image.py What do you think? --- Atlas
Re: Come watch a live stream coding session for the Hurd Video
Hi: I've just sent my recording in other mail https://www.youtube.com/watch?v=lQ7bWzsL7Ps By some reason, the image keeps freezed in some minutes, but most the recording is fine El sáb, 1 jun 2024 a las 23:22, Joshua Branson () escribió: > > Hey bug hurd! > > So we had an awesome time today watching Sergey code a trivial translator > (1) and do > some glibc hacking (2). Sergey coded and chatted for 4 and 1/2 hours! > Three cheers > for that kind of commitment! Thanks pal! > > In the livestream today, Sergey wrote caesar.c, which implements a simple > caeser > cipher. It's a toy. A caesar cipher is EASY to break, but it was fun > watching > him code it out! > > Here is the preliminary video. The audio is pretty bad. Apologies for > that. > > https://video.hardlimit.com/w/aQK46FjVeZ2efMSw1wEtoV > > I can improve my video's audio a little bit, but probably not much... > > Did someone else record video? Please share it with me. Maybe your > recording > is much better than mine! > > Perhaps in the future we can try to use mumble ? Apparently that's used in > games a lot for better quality sound. I could set up such a server. What > other > options do we have for meeting virtually with free software? > > dino perhaps, but this blog post says it's not really great for many calls: > https://dino.im/blog/2022/02/dino-0.3-release/ > > https://jami.net/ I've never tried it. > > Big blue button > > Any other options that I am missing? > > Thanks for attending those that did, and thanks again for Sergey. I can't > believe you straight up coded for 4 and 1/2 hours straight! > > 1) https://paste.debian.net/1318833/ > > 2) Sergey ported the Hurd to AArch64! He is able to run the Hurd on > AArch64 in > qemu on a GNU/Linux host! > https://www.gnu.org/software/hurd/news/2024-q1.html >
Re: Come watch a live stream coding session for the Hurd
Other idea could be a magnet/torrent translator wget magnet:?fl=http://... and downloading the torrent file without a torrent client El vie, 31 may 2024 a las 11:55, escribió: > May 31, 2024 at 5:30 AM, jbra...@dismail.de wrote: > > > > > > > > > Heyo friends! > > > > > > > > Sergey is going to do some live coding on the Hurd this Saturday > > > > > > > > June 1st at 2pm UTC (tomorrow). His goal in the stream is to > > > > > > > > update his glibc patches for running the Hurd on AArch64 (1) AND > > > > > > > > to code a simple trivfs translator live. What cool simple trivfs > > > > > > > > translator do you want to see him code (and run on AArch64)? > > > > > > > > If you want to come, then at 2pm UTC, just point your modern web > > > > > > > > browser at https://jitsi.member.fsf.org/Hurd-code-jam and we'll > > > > > > > > have some fun! If there is a password, it'll be HURD. I can > > > > > > > > probably record the session, but I don't really have the best > > > > > > > > "recording setup". Does anyone else want to volunteer to record as well? > > > > > > > > Thanks, > > > > > > > > Joshua > > > > > > > > 1) Yes, Sergey is porting the Hurd to AArch64. It's a real thing. > > > > > > > > https://www.gnu.org/software/hurd/news/2024-q1.html > > > > > > > > P.S. Please do respond to this email for cool trivfs translators Sergey > could live code: > > > > > > > > https://www.gnu.org/software/hurd/hurd/libtrivfs.html > > > > > > > > example libtrivfs translators are: > > > > > > > > https://darnassus.sceen.net/~hurd-web/hurd/translator/hello/ > > > > > > > > https://darnassus.sceen.net/~hurd-web/hurd/translator/random/ > > > > > > > > https://darnassus.sceen.net/~hurd-web/hurd/translator/mtab/ > > I'll get the ball rolling with some simple translator ideas: > > - translate translator? > > /hurd/translate ? https://github.com/LibreTranslate/LibreTranslate > > They support sending queries to the website (for $29 / month). > If someone wants to spin up a docker server for > libretranslate...that might make it easier to to try to write > such a translator. It also runs offline too apparently. > > - maybe compile translator? > > settrans -ca hello hello.c > > Anytime you update hello.c, hello gets recompiled. > > > - Maybe audio -> text translator > https://github.com/mozilla/DeepSpeech > > > - maybe html -> org translator > markup language -> markup language translator > >
Re: Come watch a live stream coding session for the Hurd
I've upload the recording to youtube. Currently is processing in HD. https://www.youtube.com/watch?v=lQ7bWzsL7Ps There was some problems, and there are some minutes in which the image keeps freezed, but the most important seems be recording fine. Excuse me the inconveniences. El sáb, 1 jun 2024 a las 11:44, Matt () escribió: > On Fri, 31 May 2024 11:30:52 +0200wrote --- > > If you want to come, then at 2pm UTC, just point your modern web > > browser at https://jitsi.member.fsf.org/Hurd-code-jam and we'll > > have some fun! If there is a password, it'll be HURD. I can > > probably record the session, but I don't really have the best > > "recording setup". Does anyone else want to volunteer to record as well? > > I appreciate the effort to record. This is something I'm interested in > and not sure I'll be able to attend. Plus, I think having such real-world > hacking examples lowers the overall barrier to working on Hurd. > >
Daemon deduplication and btrfs compression [was Re: Are 'guix gc' stats exaggerated?]
On Fri, May 31, 2024 at 03:03:47PM -0700, Felix Lechner wrote: > Hi Efraim, > > On Tue, May 28 2024, Efraim Flashner wrote: > > > As your store grows larger the inherent deduplication from the > > guix-daemon approaches a 3:1 file deduplication ratio. > > Thank you for your explanations and your data about btrfs! Btrfs > compression is a well-understood feature, although even its developers > acknowledge that the benefit is hard to quantify. > > It probably makes more sense to focus on the Guix daemon here. I hope > you don't mind a few clarifying questions. > > Why, please, does the benefit of de-duplication approach a fixed ratio > of 3:1? Does the benefit not depend on the number of copies in the > store, which can vary by any number? (It sounds like the answer may > have something to do with store size.) It would seem that this is just my experience and I'm not sure of an actual reason why this is the case. I believe that with the hardlinks only files which are identical would share a link, as opposed to a block based deduplication, where there could be more granular deduplication, so it's quite likely that multiple copies of the same package at the same version would share the majority of their files with the other copies of the package. > Further, why is the removal of hardlinks counted as saving space even > when their inode reference count, which is widely available [1] is > greater than one? I suspect that this part of the code is in the C++ daemon, which no one really wants to hack on. AFAIK Nix turned off deduplication by default years ago to speed up store operations, so I wouldn't be surprised if they also haven't worked on that part of the code. > Finally, barring a better solution should our output numbers be divided > by three to being them closer to the expected result for users? > > [1] https://en.wikipedia.org/wiki/Hard_link#Reference_counting (ins)efraim@3900XT ~$ sudo compsize -x /gnu Processed 39994797 files, 12867013 regular extents (28475611 refs), 20558307 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 56% 437G 776G 2.1T none 100% 275G 275G 723G zstd32% 161G 500G 1.4T It looks like right now my store is physically using 437GB of space. Looking only at the total the Uncompressed -> Referenced ratio being about 2.77:1 and Disk Usage -> Uncompressed being about 1.78:1, I'm netting a total of 4.92:1. Numbers on Berlin are a bit different: (ins)efraim@berlin ~$ time guix shell compsize -- sudo compsize -x /gnu Processed 41030472 files, 14521470 regular extents (37470325 refs), 17429255 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 59% 578G 970G 3.2T none 100% 402G 402G 1.1T zstd31% 176G 567G 2.1T real45m9.762s user1m53.984s sys 24m37.338s Uncompressed -> Referenced: 3.4:1 Disk Usage -> Uncompressed: 1.68:1 Total: 5.67:1 Looking at it another way, the bits that are compressible with zstd together move from 3.79:1 to 12.22:1, with no change (2.8:1) for the uncompressible bits. -- Efraim Flashner רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
