Re: [RFC Patch v2] binman: add support for creating dummy files for external blobs
Hi Simon, Am Mi., 5. Jan. 2022 um 15:05 Uhr schrieb Simon Glass : > > Hi Heiko, > > On Tue, 4 Jan 2022 at 07:22, Heiko Thiery wrote: > > > > Hi Simon, > > > > > > Am So., 2. Jan. 2022 um 18:15 Uhr schrieb Simon Glass : > > > > > > Hi Heiko, > > > > > > On Thu, 2 Dec 2021 at 19:53, Simon Glass wrote: > > > > > > > > Hi Heiko, > > > > > > > > On Mon, 29 Nov 2021 at 02:48, Heiko Thiery > > > > wrote: > > > > > > > > > > While converting to binman for an imx8mq board, it has been found that > > > > > building in the u-boot CI fails. This is because an imx8mq requires an > > > > > external binary (signed_hdmi_imx8m.bin). If this file cannot be found > > > > > mkimage fails. > > > > > To be able to build this board in the u-boot CI a binman option > > > > > (--fake-ext-blobs) is introduced that can be switched on via the > > > > > u-boot > > > > > makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy > > > > > files are > > > > > created. > > > > > > > > > > Signed-off-by: Heiko Thiery > > > > > --- > > > > > v2: > > > > > - pass allow_fake_blobs to ProcessImage() > > > > > - set AllowAllowFakeBlob() to images/entries > > > > > - create fake blob in Entry_blot.ObtainContents() when file is > > > > > missing and > > > > >creation is allowed > > > > > > > > > > still missing: > > > > > - unittest > > > > > - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment > > > > > variable. With that we could simply set this env variable in > > > > > the CI > > > > > (gitlab-ci.yml) with adding support to buildman. > > > > > > > > > > Makefile | 1 + > > > > > tools/binman/cmdline.py| 2 ++ > > > > > tools/binman/control.py| 9 +++-- > > > > > tools/binman/entry.py | 11 +++ > > > > > tools/binman/etype/blob.py | 7 +++ > > > > > tools/binman/etype/blob_ext.py | 8 > > > > > tools/binman/etype/mkimage.py | 9 + > > > > > tools/binman/etype/section.py | 9 + > > > > > 8 files changed, 54 insertions(+), 2 deletions(-) > > > > > > > > This looks good to me! The only thing is that instead of the warning > > > > you should just print a single line at the end saying which blobs were > > > > faked. See missing_list in ProcessImage() for how that could work. You > > > > can set self.fake_blob in your blob.ObtainContents() and then have a > > > > similar thing to CheckMissing() to actually collect the list of > > > > entries which were faked. > > > > > > > > Also, for the real version can you please add a test (so 'binman test > > > > -T' stays at 100% test coverage) and some docs in binman.rst ? You can > > > > use testMissingBlob() as a template. > > > > > > Any word on this? I'd like to get this feature in and take a look at > > > missing vendor tools, too. > > > > I have a new version available with your comments included. But the > > tests are still missing. In the past days I had no motivation to work > > on that. I will try to do this in the next days. > > OK thanks. If you need help with the test-coverage part, let me know. I sent a new version about an hour ago with unittest included. -- Heiko
Re: [RFC Patch v2] binman: add support for creating dummy files for external blobs
Hi Heiko, On Tue, 4 Jan 2022 at 07:22, Heiko Thiery wrote: > > Hi Simon, > > > Am So., 2. Jan. 2022 um 18:15 Uhr schrieb Simon Glass : > > > > Hi Heiko, > > > > On Thu, 2 Dec 2021 at 19:53, Simon Glass wrote: > > > > > > Hi Heiko, > > > > > > On Mon, 29 Nov 2021 at 02:48, Heiko Thiery wrote: > > > > > > > > While converting to binman for an imx8mq board, it has been found that > > > > building in the u-boot CI fails. This is because an imx8mq requires an > > > > external binary (signed_hdmi_imx8m.bin). If this file cannot be found > > > > mkimage fails. > > > > To be able to build this board in the u-boot CI a binman option > > > > (--fake-ext-blobs) is introduced that can be switched on via the u-boot > > > > makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files > > > > are > > > > created. > > > > > > > > Signed-off-by: Heiko Thiery > > > > --- > > > > v2: > > > > - pass allow_fake_blobs to ProcessImage() > > > > - set AllowAllowFakeBlob() to images/entries > > > > - create fake blob in Entry_blot.ObtainContents() when file is missing > > > > and > > > >creation is allowed > > > > > > > > still missing: > > > > - unittest > > > > - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment > > > > variable. With that we could simply set this env variable in > > > > the CI > > > > (gitlab-ci.yml) with adding support to buildman. > > > > > > > > Makefile | 1 + > > > > tools/binman/cmdline.py| 2 ++ > > > > tools/binman/control.py| 9 +++-- > > > > tools/binman/entry.py | 11 +++ > > > > tools/binman/etype/blob.py | 7 +++ > > > > tools/binman/etype/blob_ext.py | 8 > > > > tools/binman/etype/mkimage.py | 9 + > > > > tools/binman/etype/section.py | 9 + > > > > 8 files changed, 54 insertions(+), 2 deletions(-) > > > > > > This looks good to me! The only thing is that instead of the warning > > > you should just print a single line at the end saying which blobs were > > > faked. See missing_list in ProcessImage() for how that could work. You > > > can set self.fake_blob in your blob.ObtainContents() and then have a > > > similar thing to CheckMissing() to actually collect the list of > > > entries which were faked. > > > > > > Also, for the real version can you please add a test (so 'binman test > > > -T' stays at 100% test coverage) and some docs in binman.rst ? You can > > > use testMissingBlob() as a template. > > > > Any word on this? I'd like to get this feature in and take a look at > > missing vendor tools, too. > > I have a new version available with your comments included. But the > tests are still missing. In the past days I had no motivation to work > on that. I will try to do this in the next days. OK thanks. If you need help with the test-coverage part, let me know. > > > Also I think your feature should be on by default. Regards, Simon
Re: [RFC Patch v2] binman: add support for creating dummy files for external blobs
Hi Simon, Am So., 2. Jan. 2022 um 18:15 Uhr schrieb Simon Glass : > > Hi Heiko, > > On Thu, 2 Dec 2021 at 19:53, Simon Glass wrote: > > > > Hi Heiko, > > > > On Mon, 29 Nov 2021 at 02:48, Heiko Thiery wrote: > > > > > > While converting to binman for an imx8mq board, it has been found that > > > building in the u-boot CI fails. This is because an imx8mq requires an > > > external binary (signed_hdmi_imx8m.bin). If this file cannot be found > > > mkimage fails. > > > To be able to build this board in the u-boot CI a binman option > > > (--fake-ext-blobs) is introduced that can be switched on via the u-boot > > > makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files > > > are > > > created. > > > > > > Signed-off-by: Heiko Thiery > > > --- > > > v2: > > > - pass allow_fake_blobs to ProcessImage() > > > - set AllowAllowFakeBlob() to images/entries > > > - create fake blob in Entry_blot.ObtainContents() when file is missing > > > and > > >creation is allowed > > > > > > still missing: > > > - unittest > > > - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment > > > variable. With that we could simply set this env variable in the > > > CI > > > (gitlab-ci.yml) with adding support to buildman. > > > > > > Makefile | 1 + > > > tools/binman/cmdline.py| 2 ++ > > > tools/binman/control.py| 9 +++-- > > > tools/binman/entry.py | 11 +++ > > > tools/binman/etype/blob.py | 7 +++ > > > tools/binman/etype/blob_ext.py | 8 > > > tools/binman/etype/mkimage.py | 9 + > > > tools/binman/etype/section.py | 9 + > > > 8 files changed, 54 insertions(+), 2 deletions(-) > > > > This looks good to me! The only thing is that instead of the warning > > you should just print a single line at the end saying which blobs were > > faked. See missing_list in ProcessImage() for how that could work. You > > can set self.fake_blob in your blob.ObtainContents() and then have a > > similar thing to CheckMissing() to actually collect the list of > > entries which were faked. > > > > Also, for the real version can you please add a test (so 'binman test > > -T' stays at 100% test coverage) and some docs in binman.rst ? You can > > use testMissingBlob() as a template. > > Any word on this? I'd like to get this feature in and take a look at > missing vendor tools, too. I have a new version available with your comments included. But the tests are still missing. In the past days I had no motivation to work on that. I will try to do this in the next days. > Also I think your feature should be on by default. -- Heiko
Re: [RFC Patch v2] binman: add support for creating dummy files for external blobs
Hi Heiko, On Thu, 2 Dec 2021 at 19:53, Simon Glass wrote: > > Hi Heiko, > > On Mon, 29 Nov 2021 at 02:48, Heiko Thiery wrote: > > > > While converting to binman for an imx8mq board, it has been found that > > building in the u-boot CI fails. This is because an imx8mq requires an > > external binary (signed_hdmi_imx8m.bin). If this file cannot be found > > mkimage fails. > > To be able to build this board in the u-boot CI a binman option > > (--fake-ext-blobs) is introduced that can be switched on via the u-boot > > makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are > > created. > > > > Signed-off-by: Heiko Thiery > > --- > > v2: > > - pass allow_fake_blobs to ProcessImage() > > - set AllowAllowFakeBlob() to images/entries > > - create fake blob in Entry_blot.ObtainContents() when file is missing and > >creation is allowed > > > > still missing: > > - unittest > > - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment > > variable. With that we could simply set this env variable in the CI > > (gitlab-ci.yml) with adding support to buildman. > > > > Makefile | 1 + > > tools/binman/cmdline.py| 2 ++ > > tools/binman/control.py| 9 +++-- > > tools/binman/entry.py | 11 +++ > > tools/binman/etype/blob.py | 7 +++ > > tools/binman/etype/blob_ext.py | 8 > > tools/binman/etype/mkimage.py | 9 + > > tools/binman/etype/section.py | 9 + > > 8 files changed, 54 insertions(+), 2 deletions(-) > > This looks good to me! The only thing is that instead of the warning > you should just print a single line at the end saying which blobs were > faked. See missing_list in ProcessImage() for how that could work. You > can set self.fake_blob in your blob.ObtainContents() and then have a > similar thing to CheckMissing() to actually collect the list of > entries which were faked. > > Also, for the real version can you please add a test (so 'binman test > -T' stays at 100% test coverage) and some docs in binman.rst ? You can > use testMissingBlob() as a template. Any word on this? I'd like to get this feature in and take a look at missing vendor tools, too. Also I think your feature should be on by default. Regards, Simon
Re: [RFC Patch v2] binman: add support for creating dummy files for external blobs
Hi Heiko, On Mon, 29 Nov 2021 at 02:48, Heiko Thiery wrote: > > While converting to binman for an imx8mq board, it has been found that > building in the u-boot CI fails. This is because an imx8mq requires an > external binary (signed_hdmi_imx8m.bin). If this file cannot be found > mkimage fails. > To be able to build this board in the u-boot CI a binman option > (--fake-ext-blobs) is introduced that can be switched on via the u-boot > makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are > created. > > Signed-off-by: Heiko Thiery > --- > v2: > - pass allow_fake_blobs to ProcessImage() > - set AllowAllowFakeBlob() to images/entries > - create fake blob in Entry_blot.ObtainContents() when file is missing and >creation is allowed > > still missing: > - unittest > - option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment > variable. With that we could simply set this env variable in the CI > (gitlab-ci.yml) with adding support to buildman. > > Makefile | 1 + > tools/binman/cmdline.py| 2 ++ > tools/binman/control.py| 9 +++-- > tools/binman/entry.py | 11 +++ > tools/binman/etype/blob.py | 7 +++ > tools/binman/etype/blob_ext.py | 8 > tools/binman/etype/mkimage.py | 9 + > tools/binman/etype/section.py | 9 + > 8 files changed, 54 insertions(+), 2 deletions(-) This looks good to me! The only thing is that instead of the warning you should just print a single line at the end saying which blobs were faked. See missing_list in ProcessImage() for how that could work. You can set self.fake_blob in your blob.ObtainContents() and then have a similar thing to CheckMissing() to actually collect the list of entries which were faked. Also, for the real version can you please add a test (so 'binman test -T' stays at 100% test coverage) and some docs in binman.rst ? You can use testMissingBlob() as a template. Regards, Simon
[RFC Patch v2] binman: add support for creating dummy files for external blobs
While converting to binman for an imx8mq board, it has been found that
building in the u-boot CI fails. This is because an imx8mq requires an
external binary (signed_hdmi_imx8m.bin). If this file cannot be found
mkimage fails.
To be able to build this board in the u-boot CI a binman option
(--fake-ext-blobs) is introduced that can be switched on via the u-boot
makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are
created.
Signed-off-by: Heiko Thiery
---
v2:
- pass allow_fake_blobs to ProcessImage()
- set AllowAllowFakeBlob() to images/entries
- create fake blob in Entry_blot.ObtainContents() when file is missing and
creation is allowed
still missing:
- unittest
- option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment
variable. With that we could simply set this env variable in the CI
(gitlab-ci.yml) with adding support to buildman.
Makefile | 1 +
tools/binman/cmdline.py| 2 ++
tools/binman/control.py| 9 +++--
tools/binman/entry.py | 11 +++
tools/binman/etype/blob.py | 7 +++
tools/binman/etype/blob_ext.py | 8
tools/binman/etype/mkimage.py | 9 +
tools/binman/etype/section.py | 9 +
8 files changed, 54 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index f911f70344..1a833a1637 100644
--- a/Makefile
+++ b/Makefile
@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if
$(BINMAN_DEBUG),-D) \
-a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \
-a spl-dtb=$(CONFIG_SPL_OF_REAL) \
-a tpl-dtb=$(CONFIG_SPL_OF_REAL) \
+ $(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \
$(BINMAN_$(@F))
OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihex
diff --git a/tools/binman/cmdline.py b/tools/binman/cmdline.py
index d6156df408..2b29981cb4 100644
--- a/tools/binman/cmdline.py
+++ b/tools/binman/cmdline.py
@@ -52,6 +52,8 @@ controlled by a description in the board device tree.'''
help='Configuration file (.dtb) to use')
build_parser.add_argument('--fake-dtb', action='store_true',
help='Use fake device tree contents (for testing only)')
+build_parser.add_argument('--fake-ext-blobs', action='store_true',
+help='Create fake ext blobs with dummy content (for testing only)')
build_parser.add_argument('-i', '--image', type=str, action='append',
help='Image filename to build (if not specified, build all)')
build_parser.add_argument('-I', '--indir', action='append',
diff --git a/tools/binman/control.py b/tools/binman/control.py
index 0dbcbc28e9..fa034bddef 100644
--- a/tools/binman/control.py
+++ b/tools/binman/control.py
@@ -479,7 +479,8 @@ def PrepareImagesAndDtbs(dtb_fname, select_images,
update_fdt, use_expanded):
def ProcessImage(image, update_fdt, write_map, get_contents=True,
- allow_resize=True, allow_missing=False):
+ allow_resize=True, allow_missing=False,
+ allow_fake_blobs=False):
"""Perform all steps for this image, including checking and # writing it.
This means that errors found with a later image will be reported after
@@ -495,12 +496,14 @@ def ProcessImage(image, update_fdt, write_map,
get_contents=True,
allow_resize: True to allow entries to change size (this does a re-pack
of the entries), False to raise an exception
allow_missing: Allow blob_ext objects to be missing
+allow_fake_blobs: Allow blob_ext objects to be faked with dummy files
Returns:
True if one or more external blobs are missing, False if all are
present
"""
if get_contents:
image.SetAllowMissing(allow_missing)
+image.SetAllowFakeBlob(allow_fake_blobs)
image.GetEntryContents()
image.GetEntryOffsets()
@@ -629,13 +632,15 @@ def Binman(args):
images = PrepareImagesAndDtbs(dtb_fname, args.image,
args.update_fdt, use_expanded)
+
if args.test_section_timeout:
# Set the first image to timeout, used in testThreadTimeout()
images[list(images.keys())[0]].test_section_timeout = True
missing = False
for image in images.values():
missing |= ProcessImage(image, args.update_fdt, args.map,
-allow_missing=args.allow_missing)
+allow_missing=args.allow_missing,
+allow_fake_blobs=args.fake_ext_blobs)
# Write the updated FDTs to our output files
for dtb_item in state.GetAllFdts():
diff --git a/tools/binman/entry.py b/tools/binman/entry.py
index 70222718ea..9aa4359be1 100644
--- a/tools/binman/entry.py
+++ b/tools/binman/entry.py
@@ -70,6 +70,8 @@ class Entry(object):
