bug#55499: excessively large manifests due to propagation
Hi! I went ahead and pushed it as 4ff12d1de7cd617b791996ee7ca1240660b4c20e. Ludo’.
bug#55499: excessively large manifests due to propagation
Hey Ricardo, Ludovic Courtès skribis: > These patches implement what you suggested on IRC: not repeating > entire manifest entries and their propagated inputs. This has a > dramatic impact on the size of the ‘manifest’ file and on the memory > and processing time to read it for the the use case you gave. > > The second patch goes a tiny bit further by making the ‘search-paths’ > and ‘propagated-inputs’ fields optional, shaving another ~10% on the > size of ‘manifest’ in this example. > > The second patch should be squashed with the first one (so we don’t > bump version formats a second time and duplicate code). It’s kinda > optional because it doesn’t bring much compared to the first patch and > causes a bit of extra complexity, but maybe it’s still worth keeping? > > Could you try this on your larger use cases and tell me how it goes? Did you have a chance to give it a try? Maybe I can double-check that everything’s alright and go ahead. Thanks, Ludo’.
bug#55499: excessively large manifests due to propagation
Ludovic Courtès writes: > Hi! > > Ricardo Wurmus skribis: > >> A profile consisting only of four R packages (r-seurat, r-cistopic, >> r-monocle3, and r-cicero-monocle3) results in a “manifest” file that >> weighs 7.1MB. At the MDC I repeatedly encountered manifest files that >> are exceeding 24MB. > > Commit 93f601d97ca2d9b82c41afeb86879ee37eae39e6 provides a 12% size > reduction on this example, and it’s backward-compatible and cheap. Excellent! This is a great first step. > I’ll try and follow up with changes along the lines you describe. Thank you! -- Ricardo
bug#55499: excessively large manifests due to propagation
Hi! Ricardo Wurmus skribis: > A profile consisting only of four R packages (r-seurat, r-cistopic, > r-monocle3, and r-cicero-monocle3) results in a “manifest” file that > weighs 7.1MB. At the MDC I repeatedly encountered manifest files that > are exceeding 24MB. Commit 93f601d97ca2d9b82c41afeb86879ee37eae39e6 provides a 12% size reduction on this example, and it’s backward-compatible and cheap. I’ll try and follow up with changes along the lines you describe. Ludo’.
bug#55499: excessively large manifests due to propagation
Packages of some languages rely heavily on propagation. R is one example. Since the generated “manifest” file of a Guix profile records entries for all propagated packages, this can get really big really quickly. A profile consisting only of four R packages (r-seurat, r-cistopic, r-monocle3, and r-cicero-monocle3) results in a “manifest” file that weighs 7.1MB. At the MDC I repeatedly encountered manifest files that are exceeding 24MB. Simply reading that big a file with (call-with-input-file "huge-manifest" read) takes several seconds. On the MDC cluster I observed a delay of about 27 seconds. Disabling read positions with (read-disable 'positions) significantly speeds this up (18s vs 27s), but it’s still very slow. We may be able to speed things up by supporting definitions or references in manifest files, so that we don’t need to repeat a sub-tree when generating the file. -- Ricardo
