Hello all,
I thought it would be a good idea to summarize the system configuration
and initialization feature that has been pushed to develop. This
feature is definitely a work in progress, and this email is meant to
further its development. In other words, there is a lot of room for
improvement, so please don't hold back with any feedback.
Much of this email will be identical to the previous syscfg summary. If
you are already familiar with the feature, you may want to skip to the
list of open questions and unresolved issues at the end.
Proposal:
1. Move package initialization out of apps (main()) and into BSPs
and auto-generated C files.
2. Specify package configuration in the pkg.yml hierarchy.
Motivations:
1. Simplify process of creating an app (no more boilerplate
code).
2. Simplify system configuration and make it more cohesive (don't
specify config within C code).
3. Ability to audit entire system configuration via the "newt
config" command.
### Summary
System initialization happens in two stages:
1. BSP init (bsp_init())
2. sysinit (sysinit())
(1) BSP init sets up everything that is BSP specific:
* Flash map
* Driver initialization: UART, ADC, SPI, etc.
(2) Sysinit sets up all the libraries which aren't BSP or hardware
dependent (os, msys, network stacks, etc). The initialization code is
implemented in the appropriate package. Newt generates C code which
executes each package's initialization function.
### Syscfg header file generation
The system knows what to initialize and how to initialize it based on
the contents of the following header file:
<target-path>/include/syscfg/syscfg.h
For example:
targets/bleprph-nrf52dk/include/syscfg/syscfg.h
This header file is generated by newt during the build process. Newt
arranges for all packages to have access to this header file.
The syscfg.h header file consists of three sections:
* Settings
* Indication of which packages are present
* Indication of which APIs are available
Here is an abbreviated syscfg.h:
/*** Settings */
#ifndef MYNEWT_VAL_CLOCK_FREQ
#define MYNEWT_VAL_CLOCK_FREQ (1000000)
#endif
#ifndef MYNEWT_VAL_MSYS_1_BLOCK_COUNT
#define MYNEWT_VAL_MSYS_1_BLOCK_COUNT (15)
#endif
#ifndef MYNEWT_VAL_MSYS_1_BLOCK_SIZE
#define MYNEWT_VAL_MSYS_1_BLOCK_SIZE (260)
#endif
/*** Packages */
#ifndef MYNEWT_PKG_LIBS_NEWTMGR
#define MYNEWT_PKG_LIBS_NEWTMGR (1)
#endif
#ifndef MYNEWT_PKG_LIBS_NEWTMGR_TRANSPORT_BLE
#define MYNEWT_PKG_LIBS_NEWTMGR_TRANSPORT_BLE (1)
#endif
#ifndef MYNEWT_PKG_SYS_CONFIG
#define MYNEWT_PKG_SYS_CONFIG (1)
#endif
/*** APIs */
#ifndef MYNEWT_API_BLE_DRIVER
#define MYNEWT_API_BLE_DRIVER (1)
#endif
#ifndef MYNEWT_API_BLE_TRANSPORT
#define MYNEWT_API_BLE_TRANSPORT (1)
#endif
#ifndef MYNEWT_API_CONSOLE
#define MYNEWT_API_CONSOLE (1)
#endif
Newt generates the syscfg.h file from information in pkg.yml files.
This proposal adds two items to the pkg.yml structure:
1. pkg.syscfg_defs
2. pkg.syscfg_vals
Both of these items are optional.
(1) pkg.syscfg_defs is an associative array of setting definitions. The
setting name is indicated by the element key. A setting definition has
the following fields:
* description ("description")
* default value ("value")
* type (optional)
Here is an example from net/nimble/host/pkg.yml:
pkg.syscfg_defs:
BLE_SM:
description: 'Security manager legacy pairing.'
value: 1
BLE_SM_SC:
description: 'Security manager secure connections (4.2).'
value: 0
A setting can only be defined once. If a setting with the same name is
defined more than once, in any number of pkg.yml files, newt complains
and aborts the build.
(2) pkg.syscfg_vals is also an associative array keyed by setting name.
However, it only contains setting values. This array is used to
override default setting values.
Here is a hypothetical example:
pkg.syscfg_vals:
MSYS_1_BLOCK_COUNT: 12
MSYS_1_BLOCK_SIZE: 260
CLOCK_FREQ: 1000000
If a package attempts to override an undefined setting, newt warns the
user and ignores the attempt. If several packages override the same
setting, the tie is broken according to package priority. Packages are
prioritized as follows (lowest priority first, highest last):
1. Library
2. BSP
3. App
4. Target
If two packages with the same priority attempt to override the same
setting, newt raises an error and aborts the build. The remedy for this
problem is for a higher-priority package to override the setting as
well.
The expectation is that libraries would define sensible defaults in
their pkg.yml files, and BSPs, apps, and targets would override those
defaults. Library packages may also override settings defined by other
library packages.
Within the syscfg.h file:
* Each setting macro is named as follows:
MYNEWT_VAL_<escaped-setting-name>
* Each package-presence macro is named as follows:
MYNEWT_PKG_<escaped-package-name>
* Each API-presence macro is named as follows:
MYNEWT_API_<escaped-API-name>
A name is escaped by converting it to upper case and replacing all
non-alphanumeric characters with underscores.
To prevent unnecessary rebuilds, newt only overwrites the syscfg.h file
if there are any configuration changes since the previous build.
### Conditional configuration
A pkg.yml file can conditionally specify material depending on the
syscfg state. Items in a pkg.yml file can specify the name of a single
syscfg setting; the item is only processed if the setting has a
non-false value. A setting is "false" if its value is "0" or the empty
string.
That is a pretty cryptic description. Here is an example
(libs/os/pkg.yml):
pkg.deps:
- sys/sysinit
- libs/util
pkg.deps.OS_CLI:
- libs/shell
The first item is the unconditional dependency list. The second item is
a conditional dependency list. If the system enables the OS_CLI
setting, then the os package depends on the shell package, in addition
to the two unconditional dependencies. If the OS_CLI setting is not
enabled, the second dependency list is ignored.
This example uses the pkg.deps item, but other items can also be
dependent on syscfg: cflags, apis, req_apis, init_function, init_stage,
syscfg_defs, syscfg_vals.
Newt raises an error and aborts the build if it detects a circular
dependency among syscfg overrides in a set of packages.
### Setting types
By default, a syscfg setting has the string type. This type is
appropriate for most settings.
Alternatively, a syscfg setting definition may specify one of the
following types:
* task_priority
* interrupt_priority
(1) task_priority
A setting with the task_priority type specifies a Mynewt task's numeric
priority (!). The value of a task_priority setting must be one of the
following forms:
1. A number.
2. "any"
All explicit priorities (form 1) are assigned as specified. All "any"
priorities (form 2) are then iterated alphabetically by setting name;
each setting is assigned a value that is one greater than the current
greatest priority.
Task priorities greater than or equal to 240 are reserved to the system.
Newt raises an error and aborts the build if a task priority setting
resolves to a value >= 240.
Newt raises an error and aborts the build if multiple task priority
settings resolve to the same numeric value.
(2) interrupt_priority
Newt manages interrupt priority settings similarly to task priority
settings. The differences are as follows:
* No maximum value (i.e., no restriction for values >= 240).
* Multiple settings can have the same value.
* Settings specifies as "any" are all assigned the same numeric
value: one greater than the largest explicitly specified value.
### newt target config command
The following command is added to newt:
newt target config <target-name>
This command shows the following information about each setting for the
specified target:
* Setting name
* Setting description
* Actual value and package which specifies it.
* Default value and package which defines it.
* Corresponding C macro name.
* Sequence of setting overrides (i.e., the name of each package
overrides the setting in the current build).
### sysinit pkg.yml items
The following two optional items are added to the pkg.yml structure:
1. pkg.init_function
2. pkg.init_stage
(1) pkg.init_function specifies the name of a C function which
initializes the package.
(2) pkg.init_stage is a numeric value indicating when the package should
get initialized, relative to other packages.
Initialization stages are executed in ascending order. Initialization
of packages within a stage are ordered alphabetically by package name.
### sysinit C code
At build time, newt generates the following target-specific C file:
<target-path>/src/sysinit.c
This file initializes each package whose pkg.yml file specifies a
pkg.init_function item. This file does not depend on any other
packages, and it does not include any headers.
Here is an abbreviated example of how this file might look:
void bootutil_pkg_init(void);
void console_pkg_init(void);
void log_init(void);
void log_reboot_pkg_init(void);
void nffs_pkg_init(void);
void os_pkg_init(void);
void shell_pkg_init(void);
void stats_module_init(void);
void
sysinit(void)
{
/*** Stage 0 */
/* 0.0: libs/os */
os_pkg_init();
/* 0.1: sys/stats */
stats_pkg_init();
/*** Stage 1 */
/* 1.0: sys/log */
log_pkg_init();
/*** Stage 2 */
/* 2.0: fs/nffs */
nffs_pkg_init();
/* 2.1: sys/reboot */
log_reboot_pkg_init();
/*** Stage 5 */
/* 5.0: libs/bootutil */
bootutil_pkg_init();
/* 5.1: libs/console/full */
console_pkg_init();
/* 5.2: libs/shell */
shell_pkg_init();
}
An init function takes no parameters and returns void. The parameter
list is empty because the function is expected to read its configuration
from the syscfg.h file. The void return type is a bit harder to
justify, but here is my thinking. If init functions all returned error
codes, the generated sysinit code would always handle a failure return
code in the same manner: abort the initialization sequence and perform
some emergency behavior (reboot, enter rescure mode, whatever). This
repeated checking of return codes would result in extra code (text) with
no real benefit. It would be better for initialization to call a
panic function on error. The panic function could either assert(0),
or use longjmp to break out of the init sequence. Mynewt would need to
allow the application to assign semantics to the panic function.
### Questions / issues
1. Flash settings. A big piece of configuration that is missing from
syscfg is the flash layout. Ideally, a system could specify its flash
map, and newt would ensure the following:
* Named regions of storage that file systems / FCB can be
initialized with.
* Correct flash addresses in linker scripts
* Correct flash addresses in download and debug scripts
2. Conditional configuration may not be flexible enough. An item in a
pkg.yml file can be dependent on the true/false value of a single syscfg
setting. The following might be preferable.
a. Multiple settings and boolean logic. For example, only depend on
package x if: SETTING_A and (SETTING_B or SETTING_C). This would
be useful for a BSP that needs to depend on the uart driver if
any of its UARTs are enabled. To achieve this today, we need
lots of conditional dependency lists, e.g.,:
pkg.deps.UART_0:
- drivers/uart/uart_hal
pkg.deps.UART_1:
- drivers/uart/uart_hal
pkg.deps.UART_2:
- drivers/uart/uart_hal
b. Depend on the exact value of a setting, rather than its boolean
value. For example,
If LOG_TYPE==CONSOLE, then depend on libs/console/full
If LOG_TYPE==FCB, then depend on sys/fcb
Currently, the only way to achieve this behavior is by defining
lots of boolean settings: LOG_TYPE_CONSOLE, LOG_TYPE_FCB, etc.
c. Depend on the presence of a package or API. Currently, a pkg.yml
item can only depend on the value of a syscfg setting.
3. Setting requirements. Would it be useful if a setting could specify
system requirements in order for it to be set? If a package attempted
to override a setting whose requirements are not met, newt could raise
an error. For example, it doesn't make sense to override an ADC's
interrupt priority setting if the ADC itself is disabled in syscfg.
4. Setting profiles. It might be useful for NimBLE to have a
peripheral-only configuration, for example. In the system I outlined
above, the user would need to override a bunch of BLE settings in his
app or target to achieve this.
5. The init_stage mechanism feels really fragile. My thought was that
there would be defined stages that packages would naturally fit into,
but now I'm not so sure this is practical. What we really want is a way
to express initialization dependencies (e.g., package X can only be
initialized after pacakges Y and Z), but this could lead to a config
system that is incomprehensible.
6. Modular syscfg headers. Currently, there is a singly header:
syscfg/syscfg.h. Lots of files need to include this header to access
the system config, resulting in lots of unnecessary rebuilds when a
single setting is changed.
Instead, we could have a separate header for each pacakge which defines
settings (defines, not overrides). For example,
syscfg/syscfg_libs_os.h
syscfg/syscfg_fs_nffs.h
The syscfg/syscfg.h header would just include all the smaller headers.
7. A "newt target init <target-name>" command: shows the order of
package initialization (probably a poor name for this command).
8. A "newt target deps <target-name>" command: shows the dependency
graph of the build.
Thanks for reading,
Chris
