Newt feature: run custom commands at build time

[Christopher Collins]

Hello all,

I have implemented a feature in newt: the ability to run custom shell
commands at build time.



Is there any extra functionality that you'd like to see here?  All
comments are appreciated.  I am duplicating the PR text here, but the
formatting is a little nicer in the PR.

Thanks,
Chris

---

A package specifies custom commands in its `pkg.yml` file.  There are
two types of commands: 1) pre_cmds (run before the build), and 2)
post_cmds (run after the build). 

### EXAMPLE

Example (apps/blinky/pkg.yml):

pkg.pre_cmds:
scripts/pre_build1.sh: 100
scripts/pre_build2.sh: 200

pkg.post_cmds:
scripts/post_build.sh: 100

For each command, the string on the left specifies the command to run.
The number on the right indicates the command's relative ordering.

When newt builds this example, it performs the following sequence:

scripts/pre_build1.sh
scripts/pre_build2.sh
[compile]
[link]
scripts/post_build.sh

If other packages specify custom commands, those commands would also be
executed during the above sequence.  For example, if another package
specifies a pre command with an ordering of 150, that command would run
immediately after `pre_build1.sh`.  In the case of a tie, the commands
are run in lexicographic order.

All commands are run from the project's base directory.  In the above
example, the `scripts` directory would be a sibling of `targets`.

### CUSTOM BUILD INPUTS

A custom pre-build command can produce files that get fed into the
current build.  A command can generate any of the following:

1) .c files for newt to compile.
2) .a files for newt to link.
3) .h files that any package can include.

.c and .a files should be written to "$MYNEWT_USER_SRC_DIR" (or any
subdirectory within). 

.h files should be written to "$MYNEWT_USER_INCLUDE_DIR".  The
directory structure used here is directly reflected by the includer.
E.g., if a script writes to:

$MYNEWT_USER_INCLUDE_DIR/foo/bar.h

then a source file can include this header with:

#include "foo/bar.h"

### DETAILS

1. Environment variables

In addition to the usual environment variables defined for debug and
download scripts, newt defines the following env vars for custom
commands:

* MYNEWT_USER_SRC_DIR: Path where build inputs get written.
* MYNEWT_USER_INCLUDE_DIR: Path where globally-accessible headers get
   written.
* MYNEWT_PKG_NAME: The full name of the package that specifies
   the command being executed.
* MYNEWT_APP_BIN_DIR:  The path where the current target's binary
   gets written.

These environment variables are defined for each processes that a
custom command runs in.  They are *not* defined in the newt process
itself.  So, the following snippet will not produce the expected
output:

BAD Example (apps/blinky/pkg.yml):

pkg.pre_cmds:
'echo $MYNEWT_USER_SRC_DIR': 100

You can execute `sh` here instead if you need access to the environment
variables, but it is probably saner to just use a script.

2. Detect changes in custom build inputs

To avoid unnecessary rebuilds, newt detects if custom build inputs have
changed since the previous build.  If none of the inputs have changed,
then they do not get rebuilt.  If any of them of them have changed,
they all get rebuilt.

The $MYNEWT_USER_[...] base is actually a temp directory.  After the
pre-build commands have run, newt compares the contents of the temp
directory with those of the actual user directory.  If any differences
are detected, newt replaces the user directory with the temp directory,
triggering a rebuild of its contents.

3. Paths

Custom build inputs get written to the following directories:

* bin/targets//user/src
* bin/targets//user/include

Custom commands should not write to these directories.  They should use
the $MYNEWT_USER_[...] environment variables instead.

Regarding upcoming changes in reboot log

[Manish Varma]

Hello all,


I wanted to give an early heads up on upcoming changes for reboot log. So,
I am working on changing the reboot log from text format to CBOR encoded,
which means, upon receiving the reboot log, the consumer should check the
data type before processing (if not doing already).


A summary of the change is as listed below:


1. CBOR encoded reboot log message rather than text.

2. Over-long diefile strings will be truncated from the left instead of the
right.


Thanks,

Manish

Re: getting rid of ble_hs_dbg.c

[Łukasz Rymanowski]

Hi

On Mon, 23 Sep 2019 at 15:34, Andrzej Kaczmarek <
andrzej.kaczma...@codecoup.pl> wrote:

> Hi,
>
> On Wed, Sep 11, 2019 at 3:40 PM Szymon Janc 
> wrote:
>
> > Hi,
> >
> > Since we have BLE_MONITOR support which is far superior copmaring to
> > ble_hs_dbg.c... how about getting rid of that file?
> >
>
> I'm all for it and also for removing other "decoders" we have - these are
> likely not up to date with new features anyway.
>
>
> > Are there any active users relying on HCI events debugging via logs?
> >
>
> I never used it since it's just hard to follow - this was main reason why
> monitor interface was added to make analyzing HCI stream much easier.
>
>
I agree, however we need a way to grab HCI logs not only on Linux machines.
Maybe we could have some tool for Windows and maybe Macs which would store
logs from monitor interface?
Such logs could be easily read by btmon or wireshark.


>
> > --
> > pozdrawiam
> > Szymon Janc
> >
>
> Best,
> Andrzej
>

Best
Łukasz

Re: getting rid of ble_hs_dbg.c

[Andrzej Kaczmarek]

Hi,

On Wed, Sep 11, 2019 at 3:40 PM Szymon Janc  wrote:

> Hi,
>
> Since we have BLE_MONITOR support which is far superior copmaring to
> ble_hs_dbg.c... how about getting rid of that file?
>

I'm all for it and also for removing other "decoders" we have - these are
likely not up to date with new features anyway.


> Are there any active users relying on HCI events debugging via logs?
>

I never used it since it's just hard to follow - this was main reason why
monitor interface was added to make analyzing HCI stream much easier.


> --
> pozdrawiam
> Szymon Janc
>

Best,
Andrzej