Supporting *multiple* bootloaders for arm64 on a single install?

[Vagrant Cascadian]

So, I've managed to get a single image that supports booting both the
Pinebook and Pinebook Pro reasonably well! I can pop the microSD card
out of one and put it into the other, and it boots!

Maybe this is not a big deal to those used to x86, but in the ARM world,
it is unfortunately more complicated than one would hope:

  https://archive.fosdem.org/2019/schedule/event/one_image_to_rule_them_all/

The only problem with my single image for multiple pinebooks variants is
it requires manually installing u-boot to different offsets for Pinebook
Pro (e.g. idbloader.img at sector 2112 rather than sector 64), as parts
of the Pinebook bootloader are installed at overlapping offsets.

I don't think it would be terribly hard to create an image that also
supports booting EFI! And as u-boot can provide an EFI implementation,
it would not be totally unreasonable to want to install both u-boot and
EFI in the same system configuration...

But as I understand it, guix only supports a single bootloader entry. To
support all of the above, I would need three separate bootloader
instances... one for Pinebook, one for Pinebook Pro, and lastly a
grub-efi bootloader.

Installing u-boot-pinebook uses:

(define install-allwinner64-u-boot
  #~(lambda (bootloader root-index image)
  (let ((spl (string-append bootloader 
"/libexec/u-boot-sunxi-with-spl.bin"))
(u-boot (string-append bootloader 
"/libexec/u-boot-sunxi-with-spl.fit.itb")))
(write-file-on-device spl (stat:size (stat spl))
  image (* 8 1024))
(write-file-on-device u-boot (stat:size (stat u-boot))
  image (* 40 1024)

Installing u-boot-pinebook-pro-rk3399 uses:

(define install-rockpro64-rk3399-u-boot
  #~(lambda (bootloader root-index image)
  (let ((idb (string-append bootloader "/libexec/idbloader.img"))
(u-boot (string-append bootloader "/libexec/u-boot.itb")))
(write-file-on-device idb (stat:size (stat idb))
  image (* 64 512))
(write-file-on-device u-boot (stat:size (stat u-boot))
  image (* 16384 512)

(define install-pinebook-pro-rk3399-u-boot install-rockpro64-rk3399-u-boot)

;; yes, these could be refactored into a single function!

But now I need to figure out how to pass a non-default offset for the
"idb" part for rockchip platforms.

In a system config.scm, you'd define it like so:

  (bootloader (bootloader-configuration
   (bootloader u-boot-pinebook-pro-rk3399-bootloader)
   (target "/dev/mmcblk0")))

u-boot-pinebook-pro-rk3399-bootloader is defined in
gnu/bootloader/u-boot.scm, which inherits from u-boot-bootloader, which
inherits from extlinux-bootloader in gnu/bootloader/extlinux.scm...

And somewhere along the way I've lost track of how we get to
install-pinebook-pro-rk3399-u-boot...


Is it possible to definte multiple bootloaders currently, or if not,
what would need to change to be able to support that? Where would one
pass non-default offsets for a given platform?


Strictly speaking, the extlinux.conf generation would be optional for an
EFI generated image(as u-boot can load grub-efi), although at the moment
I'd prefer using extlinux.conf if available as it makes for more
consistent matching of device-tree/.dtb file with the running kernel...


A related idea would be to generate an EFI bootable image with
sufficient emtpy space before the first partition (16MB) and then one
could manually install the appropriate u-boot, maybe with some helper
scripts to avoid having to do it completely manually...


Thoughts, advice, help? :)


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Questions regarding Python packaging

[Lars-Dominik Braun]

Hi Tanguy,

(cross-posting this to the issue itself too)

> Sorry if I'm (very) late, but apprently this hasn't made it to master
> yet, so… What the status? Do you still need a willing-but-maybe-not-qualified
> person to review or discuss your patch?
the patch set works, I can build many Python packages, although some
require changes. Still, multiple things need to be done before merging
is possible imo:

1) Validate the general idea of using pypa-build is viable and
   sustainable in the long run – ideally through review by someone else
   than me. We can’t touch python-build-system every week to solve
   structural issues, so it needs to be bullet-proof.
2) Figure out how to run testing code. Currently python-build-system
   just picks pytest, if available – not sure this is the best option we
   have. How do we deal with other test systems? How do we pass options?
3) Determine the fate of Python 2, which is probably broken through this
   patch set. Shall we remove it entirely? Is it worth to keep support?
4) Iron out minor details like including pip in the python package or
   create a new python-toolchain package? What do we include in that
   meta-package? pip? virtualenv? …?
5) Fix my awkward Scheme code, especially regarding unpacking of the
   built wheels. Should we be using Python’s unzip module or can be
   assumed unzip is available in the build environment? (Should we add
   it?)

I’m by no means a Python packaging expert, so any help would be
appreciated, even if it’s just a question or thumbs-up/thumbs-down on my
ideas.

Cheers,
Lars

What is the standard path for documentation?

[Paul A. Patience]

Hi,

What is the standard place to put documentation?
The license files are automatically installed to the
share/doc/NAME-VERSION directory
(see install-licence-files in gnu-build-system.scm),
but it seems like various packages install the actual
documentation to share/doc/NAME.

I ask because I'm fixing the dealii package's documentation
installation step (the documentation installation directories
are not correctly specified).

Also, is it alright for files in the documentation directories
to refer to various other directories in the store?
Deal.II installs the attached files as documentation, but
I thought I saw the source of a Guix package make an attempt
to strip build directory information from one of its output
files.

Thanks,
Paul
###
#
#  deal.II configuration:
#CMAKE_BUILD_TYPE:   DebugRelease
#BUILD_SHARED_LIBS:  ON
#CMAKE_INSTALL_PREFIX:   /gnu/store/rb3pklfxg2d9ls0ag8nsqcaz401kg3hk-dealii-for-lethe-9.3.0-rc1
#CMAKE_SOURCE_DIR:   /tmp/guix-build-dealii-for-lethe-9.3.0-rc1.drv-0/dealii-9.3.0-rc1
#(version 9.3.0-rc1)
#CMAKE_BINARY_DIR:   /tmp/guix-build-dealii-for-lethe-9.3.0-rc1.drv-0/build
#CMAKE_CXX_COMPILER: GNU 7.5.0 on platform Linux x86_64
#/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin/c++
#C++ language standard:  C++14
#
#  Configured Features (DEAL_II_ALLOW_BUNDLED = ON, DEAL_II_ALLOW_AUTODETECTION = ON):
#  ( DEAL_II_WITH_64BIT_INDICES = OFF )
#  ( DEAL_II_WITH_ADOLC = OFF )
#  ( DEAL_II_WITH_ARBORX = OFF )
#DEAL_II_WITH_ARPACK set up with external dependencies
#  ( DEAL_II_WITH_ASSIMP = OFF )
#DEAL_II_WITH_BOOST set up with external dependencies
#DEAL_II_WITH_COMPLEX_VALUES = ON
#  ( DEAL_II_WITH_CUDA = OFF )
#  ( DEAL_II_WITH_GINKGO = OFF )
#  ( DEAL_II_WITH_GMSH = OFF )
#  ( DEAL_II_WITH_GSL = OFF )
#  ( DEAL_II_WITH_HDF5 = OFF )
#DEAL_II_WITH_KOKKOS set up with external dependencies
#DEAL_II_WITH_LAPACK set up with external dependencies
#DEAL_II_WITH_METIS set up with external dependencies
#DEAL_II_WITH_MPI set up with external dependencies
#DEAL_II_WITH_MUPARSER set up with external dependencies
#  ( DEAL_II_WITH_OPENCASCADE = OFF )
#DEAL_II_WITH_P4EST set up with external dependencies
#  ( DEAL_II_WITH_PETSC = OFF )
#DEAL_II_WITH_SCALAPACK set up with external dependencies
#  ( DEAL_II_WITH_SLEPC = OFF )
#  ( DEAL_II_WITH_SUNDIALS = OFF )
#  ( DEAL_II_WITH_SYMENGINE = OFF )
#DEAL_II_WITH_TBB set up with external dependencies
#DEAL_II_WITH_TRILINOS set up with external dependencies
#DEAL_II_WITH_UMFPACK set up with external dependencies
#DEAL_II_WITH_ZLIB set up with external dependencies
#
#  Component configuration:
#DEAL_II_COMPONENT_DOCUMENTATION
#DEAL_II_COMPONENT_EXAMPLES
#  ( DEAL_II_COMPONENT_PACKAGE = OFF )
#  ( DEAL_II_COMPONENT_PYTHON_BINDINGS = OFF )
#
#  Detailed information (compiler flags, feature configuration) can be found in detailed.log
#
#  Run  $ make info  to print a help message with a list of top level targets
#
##
#
#  deal.II configuration:
#CMAKE_BUILD_TYPE:   DebugRelease
#BUILD_SHARED_LIBS:  ON
#CMAKE_INSTALL_PREFIX:   /gnu/store/rb3pklfxg2d9ls0ag8nsqcaz401kg3hk-dealii-for-lethe-9.3.0-rc1
#CMAKE_SOURCE_DIR:   /tmp/guix-build-dealii-for-lethe-9.3.0-rc1.drv-0/dealii-9.3.0-rc1
#(version 9.3.0-rc1)
#CMAKE_BINARY_DIR:   /tmp/guix-build-dealii-for-lethe-9.3.0-rc1.drv-0/build
#CMAKE_CXX_COMPILER: GNU 7.5.0 on platform Linux x86_64
#/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin/c++
#C++ language standard:  C++14
#CMAKE_C_COMPILER:   /gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin/gcc
#CMAKE_GENERATOR:Unix Makefiles
#
#  Base configuration (prior to feature configuration):
#DEAL_II_CXX_FLAGS:-pedantic -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi -fopenmp-simd
#DEAL_II_CXX_FLAGS_RELEASE:-O2 -funroll-loops -funroll-all-loops -fstrict-aliasing -Wno-unused-local-typedefs
#DEAL_II_CXX_FLAGS_DEBUG:  -O0 -ggdb -Wa,--compress-debug-sections
#DEAL_II_LINKER_FLAGS: -Wl,--as-needed -rdynamic -lpthread
#DEAL_II_LINKER_FLAGS_RELEASE: 
#DEAL_II_LINKER_FLAGS_DEBUG:   -ggdb
#DEAL_II_DEFINITIONS:  
#DEAL_II_DEFINITIONS_RELEASE:  
#DEAL_II_DEFINITIONS_DEBUG:DEBUG
#DEAL_II_USER_DEFINITIONS: 
#DEAL_II_USER_DEFINITIONS_REL: 
#DEAL_II_USER_DEFINIT

Re: Questions regarding Python packaging

[Tanguy LE CARROUR]

Hi Lars,


Excerpts from Lars-Dominik Braun's message of May 17, 2021 8:24 am:
> just a quick reminder that an updated version (includes
> python-toolchain) of this proposal is still looking for a code review or
> further discussion. So if you feel confident about touching
> python-build-system, please have a look at
> https://issues.guix.gnu.org/46848#1
> 
> I’d be nice to get this into core-updates before the next merge.
> Otherwise we’ll have to keep adding workarounds (see for example
> python-testpath in master) to Python packages not using setuptools as
> their build system.

Sorry if I'm (very) late, but apprently this hasn't made it to master
yet, so… What the status? Do you still need a willing-but-maybe-not-qualified
person to review or discuss your patch?

Regards,

-- 
Tanguy

Some more rust/cargo insights

[Hartmut Goebel]

Hi.,

these day I had spent some more hours struggling with rust ans cargo, 
trying to get "pre-built" crates.


Summery; Cargo is cruft, no solution found yet.

I tried reusing a crate from the very same place it was built (see 
enclosed script). Anyhow, this does not work since cargo uses a 
different "metadata" value, even if noting changed. Based in the verbose 
output (cargo build -v …) I assume that some parameters of the 
"destination" build get included into this value.


This meets another observation; when building the sequoia suite, several 
crates are build several times - even if all builds are performed in the 
same environment.


Rust's build system is such a cruft - I really would like to throw it 
where it belongs: into the trash.


--
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

(use-modules
 (guix download)
 (guix packages)
 (guix build-system cargo)
 (gnu packages crates-io)
 (srfi srfi-1)
 (srfi srfi-26)
 (gnu packages crates-graphics)
 (gnu packages rust-apps))

(define-public rust-pretty-assertions-0.5
  (package
(inherit rust-pretty-assertions-0.6)
(name "rust-pretty-assertions")
(version "0.5.1")
(source
 (origin
   (method url-fetch)
   (uri (crate-uri "pretty_assertions" version))
   (file-name
(string-append name "-" version ".tar.gz"))
   (sha256
  (base32 "1ins6swkpxmrh8q5h96h8nv0497d3dclsiyx2lyvqi6py0q980is"
(build-system cargo-build-system)
(arguments
 `(#:tests? #f
   #:cargo-inputs
   (("rust-ansi-term" ,rust-ansi-term-0.11)
("rust-difference" ,rust-difference-2))

;; /gnu/store/wknzymkfbfjbxwfd3djrn4hk9zdfgs56-rust-xmlparser-0.13.3 -- original
;; 
;; libxmlparser-f82b201ea4144ed3.rlib

(define-public myrust-xmlparser
  (package
(inherit rust-xmlparser-0.13)
(outputs '("out" "rlib"))
(arguments
 `(#:skip-build? #f
   #:tests? #f
   #:cargo-build-flags (list "--release" "-vv")
   #:phases
   (modify-phases %standard-phases
 (add-after 'install 'install-rlib
   (lambda* (#:key outputs #:allow-other-keys)
 (let* ((rout (assoc-ref outputs "rlib"))
(dest (string-append rout "/rlib")))
   ;;(mkdir dest)
   ;;(for-each (cut install-file <> (string-append rout "/rlib"))
   (for-each (lambda (fn)
   (install-file fn (string-append rout "/rlib")))
 (find-files "target/release/deps" "\\.(rlib|rmeta)$"))
   )))
  ;; (add-after 'install 'fail
  ;;   (lambda _ #f))
)

(define-public myrust-roxmltree
  (package
(inherit rust-roxmltree-0.14)
;;(outputs '("out" "crate"))
(inputs
 `(("rust-xmlparser" ,myrust-xmlparser "rlib")))
(arguments
 `(#:skip-build? #f
   #:tests? #f
   ;;#:vendor-dir "/tmp/src"
   #:cargo-build-flags (list "--release" "-vv")
   #:cargo-inputs
   (("rust-xmlparser:src" ,rust-xmlparser-0.13)
("rust-pretty-assertions" ,rust-pretty-assertions-0.5))
   #:phases
   (modify-phases %standard-phases
 (add-after 'patch-cargo-checksums 'bring-in-rlib
   (lambda* (#:key inputs #:allow-other-keys)
 (let* ((rin (assoc-ref inputs "rust-xmlparser"))
(src (assoc-ref inputs "rust-xmlparser"))
(rlib (string-append rin "/rlib")))
   (mkdir "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/")
   (copy-recursively
   "guix-vendor/rust-xmlparser-0.13.3.tar.gz"
   "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3")
   (rename-file
"guix-vendor/rust-xmlparser-0.13.3.tar.gz"
"../rust-xmlparser-0.13.3.tar.gz")
   (symlink
"/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3"
"guix-vendor/rust-xmlparser-0.13.3")
;;(let ((port (open-file ".cargo/config" "w" #:encoding "utf-8")))
;;  (display "
;; #paths = [\"/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3\"]

;; [source.crates-io]
;; replace-with = 'vendored-sources'

;; #[patch.crates-io]
;; #xmlparser = { path = '/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3' }

;; [source.vendored-sources]
;; directory = '" port)
;;  (display (string-append (getcwd) "/guix-vendor") port)
;;  (display "'
;; " port)
;; (close-port port))
   ;; (substitute* ".cargo/config"
   ;;   (("\\.vendored-sources\\]" _)
   ;;(string-append _ "\n[paths]"))
   (for-each (lambda (fn)
   (install-file fn "target/release/deps"))
 (pk (find-files rlib "\\.(rlib|rmeta)$")))
   #t
 

Re: Question: wrap-program without #:sh can be ok even when cross-compiling?

[Leo Prikler]

Am Sonntag, den 06.06.2021, 12:14 +0200 schrieb Maxime Devos:
> Leo Prikler schreef op zo 06-06-2021 om 09:39 [+0200]:
> > I think we might want to export a utility procedure
> > (patch-shebangs files inputs)
> 
> This procedure already exists, but is undocumented and
> unexported. It is in (guix build gnu-build-system).
> 
> (define* (patch-shebangs #:key inputs outputs (patch-shebangs? #t)
>  #:allow-other-keys)
>   [...])
This patch-shebangs doesn't take a list of files as arguments, it
instead just iterates over all the files.  I don't think this is
particularly helpful in a cross-compiling context, where you want some
files (most notably scripts, that don't get wrapped) to have its
shebang drawn from inputs rather than native-inputs.

You could use the patch-shebang primitive from (guix build utils), but
you'd have to wrap it in something like

(let ((path (search-path-as-list '("bin" "sbin") (map cdr INPUTS
  (for-each (cute patch-shebang <> path) (find-files ...)))

where INPUTS are inputs or native-inputs, whichever you want.  Not very
friendly imo.
> > so that files used during build (e.g. configure, Makefile, etc.)
> > can do
> > (patch-shebangs build-stuff native-inputs) and the rest implicitly
> > gets
> > (patch-shebangs files inputs) during the patch-shebangs
> > phase.  WDYT?
> 
> Looking at %standard-phases, we have
> 
> (define %standard-phases
>   ;; Standard build phases, as a list of symbol/procedure pairs.
>   [...]
> (phases [...] unpack bootstrap patch-usr-bin-file
> patch-source-shebangs configure patch-generated-file-
> shebangs
> build check install
> patch-shebangs [...]))
> 
> Here, patch-source-shebangs calls patch-shebang for every file
> in the source code. As #:path is not set, $PATH is used. Thus,
> when cross-compiling, native-inputs (+ some implicit inputs) is used,
> and when compiling natively, the union of native-inputs and inputs (+
> some implicit inputs) is used (*).
> 
> Thus, the files used during build (configure, Makefile, ...) already
> get a ‘good’ interpreter. (Unless I'm mistaken, I didn't test this.)
I agree, that's the status quo.  We want that to be retained, but we
also want our cross-compilation story to be better.  Hence imo allowing
to pass files, constructing path from inputs and exporting this
procedure might make sense.

> (*) I looked into separating 'native-inputs' and 'inputs' even when
> compiling
> natively but it turned out to be more complicated than first
> expected.
Fair enough, nobody expects you to do everything at once.

Regards,
Leo

Re: Question: wrap-program without #:sh can be ok even when cross-compiling?

[Maxime Devos]

Leo Prikler schreef op zo 06-06-2021 om 09:39 [+0200]:
> I think we might want to export a utility procedure
> (patch-shebangs files inputs)

This procedure already exists, but is undocumented and
unexported. It is in (guix build gnu-build-system).

(define* (patch-shebangs #:key inputs outputs (patch-shebangs? #t)
 #:allow-other-keys)
  [...])

> so that files used during build (e.g. configure, Makefile, etc.) can do
> (patch-shebangs build-stuff native-inputs) and the rest implicitly gets
> (patch-shebangs files inputs) during the patch-shebangs phase.  WDYT?

Looking at %standard-phases, we have

(define %standard-phases
  ;; Standard build phases, as a list of symbol/procedure pairs.
  [...]
(phases [...] unpack bootstrap patch-usr-bin-file
patch-source-shebangs configure patch-generated-file-shebangs
build check install
patch-shebangs [...]))

Here, patch-source-shebangs calls patch-shebang for every file
in the source code. As #:path is not set, $PATH is used. Thus,
when cross-compiling, native-inputs (+ some implicit inputs) is used,
and when compiling natively, the union of native-inputs and inputs (+
some implicit inputs) is used (*).

Thus, the files used during build (configure, Makefile, ...) already
get a ‘good’ interpreter. (Unless I'm mistaken, I didn't test this.)

(*) I looked into separating 'native-inputs' and 'inputs' even when compiling
natively but it turned out to be more complicated than first expected.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: Which kernel series to use in the installer and for installed systems?

[Efraim Flashner]

On Sat, Jun 05, 2021 at 02:09:59PM -0400, Mark H Weaver wrote:
> Hi Efraim,
> 
> Efraim Flashner  writes:
> > I don't really understand why
> > and how a newer kernel would make things stop working,
> 
> Newer kernels _usually_ work fine, but occasionally things break, and
> that's much more likely to happen when switching to a newer kernel
> series.  It happened to me quite recently, when a couple of us found
> that early 5.12.x kernels would lock up a Thinkpad X200 within minutes
> of booting.  See .
> 
> For those who can afford to deal with breakage like this at unscheduled
> times, the 'linux-libre' variable is likely the best choice.  For those
> who prefer stability, I recommend 'linux-libre-X.YY' for some LTS series
> X.YY.
> 
> The 'linux-libre-lts' variable seems to me the worst of both worlds, and
> likely to lead to some people getting burned from unexpected kernel
> upgrades that they might not have been expecting.  If 'linux-libre-lts'
> is included as an option, I recommend adding text that makes it clear to
> users that automatic upgrades to newer LTS series will happen without
> warning, years before such upgrades are needed.
> 
> What do you think?
> 
>  Regards,
>Mark
> 

I know if I were using the zfs kernel module I'd want to use the most
recent lts release. I like the idea of labeling it something like:

* linux-libre  (tracks the latest kernel release)
* linux-libre-lts  (tracks most recent long-term support release, with
upgrades to newer versions automatically)
* linux-libre-5.10 (current long-term support release, supported until
December 2026)

I know upstream uses 'stable' to mean the current release, but I
personally find the terminology confusing.

-- 
Efraim Flashner  אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Question: wrap-program without #:sh can be ok even when cross-compiling?

[Leo Prikler]

I think we might want to export a utility procedure
(patch-shebangs files inputs)
so that files used during build (e.g. configure, Makefile, etc.) can do
(patch-shebangs build-stuff native-inputs) and the rest implicitly gets
(patch-shebangs files inputs) during the patch-shebangs phase.  WDYT?