Hello, Norman. Maybe I had to say this early, but: We, and Maybe someone else, have own repository with packages. We can/would not merge it with main branch of Genode. We have used directory in root Genode source tree, like dde_linux, before. So, how third party repos should look like in new directory layout?
> > Hello, > > now that Genode's new directory structure is in place, let's take the > next step towards a bullet-proof solution for integrating 3rd-party code > with Genode (see [1] for the corresponding issue). The new solution, > which is very much inspired by the fabulous Nix package manager [2] > comes in the form of new tools to be found at 'tool/ports/' on the > staging branch [3]. > > [1] https://github.com/genodelabs/genode/issues/1082 > [2] http://nixos.org/nix/ > [3] https://github.com/genodelabs/genode/commits/staging > > Hereby, I'd like to briefly explain how the new solution works from the > viewpoint of a Genode user, describe the steps needed to add a new port > to Genode, and outline how we will evolutionary move from the old 'make > prepare' mechanism to the new concept. > > Note that even though the port mechanism described herein looks a bit > like "package management", it covers a different problem. The problem > covered here is the integration of existing 3rd-party source code with > the Genode source tree. Packaging, on the other hand, would provide a > means to distribute self-contained portions of the Genode source tree > including their respective 3rd-party counterparts as separate packages. > Package management is not addressed yet. > > > The use of the new mechanism > ---------------------------- > > Genode's source-code repositories used to come with Makefiles in their > respective base directories. Those makefile provided the rules 'prepare' > and 'clean'. The 'prepare' rule allowed for the automated downloading > and installation of 3rd-party code, whereas the 'clean' rule reverted > the installation. Source-code archives were downloaded at > <rep-dir>/download/ whereas the extracted 3rd-party code usually resided > at <rep-dir>/contrib/. In the case of ported libraries, the 'make > prepare' step also used to create a bunch of symlinks within > <rep-dir/include/ that pointed to the respective header files within > <rep-dir>/contrib/. > > The old 'make prepare' approach was implemented for each individual > repository. In contrast, the new solution unifies the procedure across > all the repositories located at <genode-dir>/repos/. To install a port > provided by any repository, just invoke the tool > <genode-dir>/tool/port/prepare_port with the name of the port as > argument. The tool will scan all repositories for the specified port and > install the port to <genode-dir>/contrib/. Each version of an installed > port resides in a dedicated subdirectory within the contrib/ directory. > The port-specific directory is called port directory. It is named > <port-fingerprint>-<port-name>. The <fingerprint> uniquely identifies > the version of the port (it is a SHA1 hash of the ingredients of the > port). If two versions of the same port are installed, each of them will > have a different fingerprint. So they and up in different directories. > > Within the Genode source tree, a port is represented by two files, a > <port-name>.port and a <port-name>.hash file. Both files reside at the > <rep-dir>/ports/ subdirectory of the corresponding repository. The > <port-name>.port file is the port description, which declares the > ingredients of the port, e.g., archives to download, patches to apply. > The <port-name>.hash file contains the fingerprint of the corresponding > port description, thereby uniquely identifying a version of the port. > > To see, which ports are available, look out for <rep-dir>/port/*.port files: > > find <genode-dir> -mindepth 4 -maxdepth 4 -name "*.port" > > So how does Genode's build system find the source codes for the right > port directory to use? If the build system encounters a target that > incorporates ported source code, it looks up the respective > <port-name>.hash file in the repositories as specified in the build > configuration. The fingerprint found in the hash file is used to > construct the path to the port directory under contrib/. > > > Adding a port > ------------- > > The basic steps of adding a new port to Genode are as follows. Let us > assume the 3rd-party source codes comes in the form of a tar.gz archive. > > 1) Create '<rep-dir>/ports/<port-name>.port' file: > > LICENSE := unknown > VERSION := <version> > DOWNLOADS := <archive-name>.archive > URL(<archive-name>) := http://the-url-of-the.archive.tar.gz > > The 'DOWNLOADS' declaration contains a list of items to download. Each > item is suffixed with the type of the download. Supported types are > 'file' (a plain file), 'archive' (an archive of the types tar.gz, > tar.xz, tgz, tar.bz2, or zip), 'git' (a Git repository), or 'svn' (a > Subversion repository). For each item, there have to be a few additional > declarations, in particular the URL where to to download it from. > > 2) Create 'ports/<port-name>.hash' file with the content 'dummy'. > > This one will be used during the porting work and replaced once the port > if finished. Because the hash file contains the string "dummy", the port > directory will be located at <genode-dir>/contrib/dummy-<port-name>/. > > 3) Declaring the hash sum of a downloaded archive > > Try to execute the new port file to download the archive. > > <genode-dir>/tool/ports/prepare_port <port-name> > > This step will create the port directory and download the archive to the > base of this directory. The prepare_port tool tries to validate the > correct version of the downloaded file using an SHA1 hash sum. Because > the port description lack the known-good SHA1 sum, the check will fail: > > Error: Hash sum check for <port-name> failed > > Calculate the SHA1 hash sum of the archive: > > sha1sum <port-dir>/<archive-name>.tar.gz > > Declare the hash sum of the archive in the <port-name>.port file: > > SHA(<archive-name>) := <hash-value> > > When executing the prepare_port step again, the integrity check for the > downloaded archive should succeed. However, we get the following error > message: > > Error: <rep-dir>/ports/<port-name>.port is out of date, expected > <fingerprint> > > We get this message because we had specified the "dummy" hash value in > the <port-name>.hash file. The prepare_port tool computes a fingerprint > of the actual version of the port and compares this fingerprint with the > hash value specified in <port-name>.hash. The computed fingerprint can > be found at <port-dir>/<port-name>.hash. In the final step of the port, > we will replace the dummy fingerprint with the actual fingerprint of the > port. But before finalizing the porting work, it is practical to keep > using the dummy hash and suppress the fingerprint check. This can be > done by adding 'CHECK_HASH=no' as argument to the prepare_port tool: > > <genode-dir>/tool/ports/prepare-port <port-name> CHECK_HASH=no > > 4) Extracting the source code > > When executing the prepare-step now, the tool will present you with the > following message: > > Error: Missing definition of DIR(<port-name>) in <port-file> > > We need to declare where to extract the downloaded archive. E.g., > > DIR(<port-name>) := src/lib/<port-name> > > Each port directory is principally organized like a Genode source-code > repository. So it is good practice to place the extracted code at the > location where it would reside if hosted within a repository. Most > source packages distributed as tar archives contain a directory named > after the package and the version number in the top-level directory of > the archive. By default, the prepare tool will strip this directory when > extracting the archive. So the actual content will be installed at the > directory specified via the 'DIR(<archive-name>)' declaration. You can > override the default extraction argument by specifying a custom > 'TAR_OPT(<archive-name>)' declaration. > > 5) Declaration of the license > > Now that you have downloaded and extracted the 3rd-party source code > within the port directory, revisit the code for its license. Update the > LICENSE declaration in the <port-name>.port file accordingly. > > 6) Assembling the include directory exported by the port > > Define include files to be presented to the Genode build system, e.g.: > > DIRS := include > DIR_CONTENT(include) src/lib/<port-name>/include/*.h > > This declaration tells the prepare_port tool to copy the files found in > the include/ directory of the extracted archive to the > <port-dir>/include/ directory. Using this mechanism, arbitrary directory > structures can be constructed out of the downloaded content. Validate > that the DIRS declarations work as expected by executing the > prepare_port step again and revisiting the content of the port directory. > > 7) Using the ported code from Genode compilation targets > > Now the preparation step is complete. The final piece of the puzzle is > telling the Genode build system to use the port. Within any library > description file, target.mk file, or import-*.mk file, you can use the > function 'select_from_ports' to query a port directory using the port > name as argument. E.g., assuming <port-name> refers to a library, the > corresponding import-<port-name>.mk file may contain the following > declaration: > > INC_DIR += $(call select_from_ports,<port-name>)/include > > This declaration will add the <port-name>/include directory of the port > to the include-search path of every target that uses the library. > Similarly, the library description file may use the 'select_from_ports' > function to define the vpath of the 3rd-party source codes. > > Under the hood, the 'select_from_ports' function looks up the > fingerprint of the specified port by reading the corresponding > <port-name>.hash file. It then uses this hash value to construct the > directory path within the 'contrib/' directory that belongs to the > matching version of the port. If there is no hash file that matches the > port name, or if the port directory does not exist, the build system > will present you with an error message. > > Finally, after having tested that both the preparation-step and the > build of the ported source code works as expected, it is time to > finalize the fingerprint stored in the <rep-dir>/ports/<port-name>.hash > file. This can be done by copying the content of the > <port-dir>/<port-name>.hash file. The next time, you invoke the > prepare_port tool, do not specify the 'CHECK_HASH=no' argument. So the > fingerprint check will validate that the <port-name>.hash file > corresponds to your <port-name>.port file. From now on, the > contrib/dummy-<port-name> directory will no longer be used because the > <port-name>.hash file points to the port directory named after the real > fingerprint. > > > Transition to the new mechanism > ------------------------------- > > In the last couple of days, I have reworked more than 60 ports to use > the new mechanism. The ports not covered so far are: > > * libports; qt5 > * base-codezero > * base-fiasco > * base-pistachio > * base-foc > * dde_rump > * dde_linux > * dde_ipxe > * ports-foc > * ports: gcc > * ports: gdb > * ports: seoul > * ports: virtualbox > > During the transition phase (the next release cycle), we will keep the > original 'make prepare' mechanism as a front end. So the 'make prepare' > instructions as found in many tutorials will still work. But under the > hood, 'make prepare' will just invoke the new _tool/ports/prepare_port_ > tool. > > In the current version at the staging branch, the hash sum check is > disabled for all ports installed via the old 'make prepare' front end. > Nearly all hash files contain "dummy". This is because the prepare_port > tool is still in flux. Each change of the tool would require new > fingerprints for all packages, which is inconvenient while working on > the tool. However, once the development of the tools settles, the dummy > hash sums will be replaced by the real hash sums. > > > I hope that the new mechanism will make Genode more convenient to use. > Hopefully, the errors caused by missing or outdated 'make prepare' steps > will be a thing of the past. However, the current change is just another > step. Once we have fully adopted the prepare_port tool, we can fairly > easily add further utilities to work with ports, e.g., adding dependency > information between ports, garbage collecting stale versions, updating > all installed ports, etc. Also, the new way of how ports are organized > (having a layout similar to Genode repositories) will greatly help us to > implement proper package management for Genode. For the latter, I'd love > to embrace the Nix package manager. > > As immediate steps, we should rework the remaining ports (as mentioned > above) to use the new port mechanism and update the documentation, in > particular the porting guide [4] > > [4] https://github.com/genodelabs/genode/blob/master/doc/porting_guide.txt > > Cheers > Norman > > -- > Dr.-Ing. Norman Feske > Genode Labs > > http://www.genode-labs.com · http://genode.org > > Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden > Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth > > ------------------------------------------------------------------------------ > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE > Instantly run your Selenium tests across 300+ browser/OS combos. > Get unparalleled scalability from the best Selenium testing platform available > Simple to use. Nothing to install. Get started now for free." > http://p.sf.net/sfu/SauceLabs > _______________________________________________ > Genode-main mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/genode-main ------------------------------------------------------------------------------ "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs _______________________________________________ Genode-main mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/genode-main
