Re: [RFC Patch v3] binman: add support for creating dummy files for external blobs

2022-01-05 Thread Simon Glass
Hi Heiko,

On Wed, 5 Jan 2022 at 09:06, Simon Glass  wrote:
>
> Hi Heiko,
>
> On Wed, 5 Jan 2022 at 05:58, Heiko Thiery  wrote:
> >
> > While converting to binman for an imx8mq board, it has been found that
> > building in the u-boot CI fails. This is because an imx8mq requires an
> > external binary (signed_hdmi_imx8m.bin). If this file cannot be found
> > mkimage fails.
> > To be able to build this board in the u-boot CI a binman option
> > (--fake-ext-blobs) is introduced that can be switched on via the u-boot
> > makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are
> > created.
> >
> > Signed-off-by: Heiko Thiery 
> > ---
> > v3:
> >  - add CheckFakedBlobs() and print a list of faked files at the end
> >  - add unittest
> >
> > v2:
> >  - pass allow_fake_blobs to ProcessImage()
> >  - set AllowAllowFakeBlob() to images/entries
> >  - create fake blob in Entry_blot.ObtainContents() when file is missing and
> >creation is allowed
> >
> >  still missing:
> >   - unittest
> >   - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment
> > variable. With that we could simply set this env variable in the CI
> > (gitlab-ci.yml) with adding support to buildman.
> >
> >  Makefile|  1 +
> >  tools/binman/cmdline.py |  2 ++
> >  tools/binman/control.py | 26 +++---
> >  tools/binman/entry.py   | 23 +++
> >  tools/binman/etype/blob.py  | 18 ++
> >  tools/binman/etype/blob_ext.py  |  8 
> >  tools/binman/etype/mkimage.py   | 20 
> >  tools/binman/etype/section.py   | 20 
> >  tools/binman/ftest.py   | 13 -
> >  tools/binman/test/203_fake_blob.dts | 14 ++
> >  10 files changed, 137 insertions(+), 8 deletions(-)
> >  create mode 100644 tools/binman/test/203_fake_blob.dts
>
> Please check that you keep to 80cols, except where it would split a string.
>
> This is missing a few holes in test coverage. Did you try 'binman test -T' ?
>
> If you are stuck I could fiddle with it a bit.

BTW if you need the external tools you might try u-boot-dm/bin-working
which has a command that might fetch them:

binman tool --fetch missing

It is WIP at present, so has an old version of your patch in there.

Regards,
Simon


Re: [RFC Patch v3] binman: add support for creating dummy files for external blobs

2022-01-05 Thread Simon Glass
Hi Heiko,

On Wed, 5 Jan 2022 at 05:58, Heiko Thiery  wrote:
>
> While converting to binman for an imx8mq board, it has been found that
> building in the u-boot CI fails. This is because an imx8mq requires an
> external binary (signed_hdmi_imx8m.bin). If this file cannot be found
> mkimage fails.
> To be able to build this board in the u-boot CI a binman option
> (--fake-ext-blobs) is introduced that can be switched on via the u-boot
> makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are
> created.
>
> Signed-off-by: Heiko Thiery 
> ---
> v3:
>  - add CheckFakedBlobs() and print a list of faked files at the end
>  - add unittest
>
> v2:
>  - pass allow_fake_blobs to ProcessImage()
>  - set AllowAllowFakeBlob() to images/entries
>  - create fake blob in Entry_blot.ObtainContents() when file is missing and
>creation is allowed
>
>  still missing:
>   - unittest
>   - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment
> variable. With that we could simply set this env variable in the CI
> (gitlab-ci.yml) with adding support to buildman.
>
>  Makefile|  1 +
>  tools/binman/cmdline.py |  2 ++
>  tools/binman/control.py | 26 +++---
>  tools/binman/entry.py   | 23 +++
>  tools/binman/etype/blob.py  | 18 ++
>  tools/binman/etype/blob_ext.py  |  8 
>  tools/binman/etype/mkimage.py   | 20 
>  tools/binman/etype/section.py   | 20 
>  tools/binman/ftest.py   | 13 -
>  tools/binman/test/203_fake_blob.dts | 14 ++
>  10 files changed, 137 insertions(+), 8 deletions(-)
>  create mode 100644 tools/binman/test/203_fake_blob.dts

Please check that you keep to 80cols, except where it would split a string.

This is missing a few holes in test coverage. Did you try 'binman test -T' ?

If you are stuck I could fiddle with it a bit.

Regards,
Simon


[RFC Patch v3] binman: add support for creating dummy files for external blobs

2022-01-05 Thread Heiko Thiery
While converting to binman for an imx8mq board, it has been found that
building in the u-boot CI fails. This is because an imx8mq requires an
external binary (signed_hdmi_imx8m.bin). If this file cannot be found
mkimage fails.
To be able to build this board in the u-boot CI a binman option
(--fake-ext-blobs) is introduced that can be switched on via the u-boot
makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are
created.

Signed-off-by: Heiko Thiery 
---
v3:
 - add CheckFakedBlobs() and print a list of faked files at the end
 - add unittest

v2:
 - pass allow_fake_blobs to ProcessImage()
 - set AllowAllowFakeBlob() to images/entries
 - create fake blob in Entry_blot.ObtainContents() when file is missing and
   creation is allowed

 still missing:
  - unittest
  - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment
variable. With that we could simply set this env variable in the CI
(gitlab-ci.yml) with adding support to buildman.

 Makefile|  1 +
 tools/binman/cmdline.py |  2 ++
 tools/binman/control.py | 26 +++---
 tools/binman/entry.py   | 23 +++
 tools/binman/etype/blob.py  | 18 ++
 tools/binman/etype/blob_ext.py  |  8 
 tools/binman/etype/mkimage.py   | 20 
 tools/binman/etype/section.py   | 20 
 tools/binman/ftest.py   | 13 -
 tools/binman/test/203_fake_blob.dts | 14 ++
 10 files changed, 137 insertions(+), 8 deletions(-)
 create mode 100644 tools/binman/test/203_fake_blob.dts

diff --git a/Makefile b/Makefile
index ae9bfab91a..63d286a4c1 100644
--- a/Makefile
+++ b/Makefile
@@ -1315,6 +1315,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if 
$(BINMAN_DEBUG),-D) \
-a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \
-a spl-dtb=$(CONFIG_SPL_OF_REAL) \
-a tpl-dtb=$(CONFIG_TPL_OF_REAL) \
+   $(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \
$(BINMAN_$(@F))
 
 OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihex
diff --git a/tools/binman/cmdline.py b/tools/binman/cmdline.py
index e73ff78095..700e5a29a5 100644
--- a/tools/binman/cmdline.py
+++ b/tools/binman/cmdline.py
@@ -52,6 +52,8 @@ controlled by a description in the board device tree.'''
 help='Configuration file (.dtb) to use')
 build_parser.add_argument('--fake-dtb', action='store_true',
 help='Use fake device tree contents (for testing only)')
+build_parser.add_argument('--fake-ext-blobs', action='store_true',
+help='Create fake ext blobs with dummy content (for testing only)')
 build_parser.add_argument('-i', '--image', type=str, action='append',
 help='Image filename to build (if not specified, build all)')
 build_parser.add_argument('-I', '--indir', action='append',
diff --git a/tools/binman/control.py b/tools/binman/control.py
index 304fc70f56..ec0905602e 100644
--- a/tools/binman/control.py
+++ b/tools/binman/control.py
@@ -479,7 +479,8 @@ def PrepareImagesAndDtbs(dtb_fname, select_images, 
update_fdt, use_expanded):
 
 
 def ProcessImage(image, update_fdt, write_map, get_contents=True,
- allow_resize=True, allow_missing=False):
+ allow_resize=True, allow_missing=False,
+ allow_fake_blobs=False):
 """Perform all steps for this image, including checking and # writing it.
 
 This means that errors found with a later image will be reported after
@@ -495,12 +496,15 @@ def ProcessImage(image, update_fdt, write_map, 
get_contents=True,
 allow_resize: True to allow entries to change size (this does a re-pack
 of the entries), False to raise an exception
 allow_missing: Allow blob_ext objects to be missing
+allow_fake_blobs: Allow blob_ext objects to be faked with dummy files
 
 Returns:
-True if one or more external blobs are missing, False if all are 
present
+True if one or more external blobs are missing or faked,
+False if all are present
 """
 if get_contents:
 image.SetAllowMissing(allow_missing)
+image.SetAllowFakeBlob(allow_fake_blobs)
 image.GetEntryContents()
 image.GetEntryOffsets()
 
@@ -549,7 +553,13 @@ def ProcessImage(image, update_fdt, write_map, 
get_contents=True,
 tout.Warning("Image '%s' is missing external blobs and is 
non-functional: %s" %
  (image.name, ' '.join([e.name for e in missing_list])))
 _ShowHelpForMissingBlobs(missing_list)
-return bool(missing_list)
+faked_blob_list = []
+image.CheckFakedBlobs(faked_blob_list)
+if faked_blob_list:
+tout.Warning("Image '%s:%s' has faked external blobs and is 
non-functional: %s" %
+ (image.name, image.image_name,
+  ' '.join([