Re: [U-Boot] [PATCH v2 33/37] binman: Add a utility library for coreboot CBFS

2019-07-17 Thread sjg
Coreboot uses a simple flash-based filesystem called Coreboot Filesystem
(CBFS) to organise files used during boot. This allows files to be named
and their position in the flash to be set. It has special features for
dealing with x86 devices which typically memory-map their SPI flash to the
top of 32-bit address space and need a 'boot block' ending there.

Create a library to help create and read CBFS files. This includes a
writer class, a reader class and associated other helpers. Only a subset
of features are currently supported.

Signed-off-by: Simon Glass 
---

Changes in v2:
- Deal with travis's old lz4 version by skipping tests as necessary
- Install lzma tool in travis
- Skip use of cbfstool in tests if it is not available

 .travis.yml|   1 +
 tools/binman/binman.py |   4 +-
 tools/binman/cbfs_util.py  | 720 +
 tools/binman/cbfs_util_test.py | 540 +
 4 files changed, 1264 insertions(+), 1 deletion(-)
 create mode 100644 tools/binman/cbfs_util.py
 create mode 100755 tools/binman/cbfs_util_test.py

Applied to u-boot-dm, thanks!
___
U-Boot mailing list
U-Boot@lists.denx.de
https://lists.denx.de/listinfo/u-boot


[U-Boot] [PATCH v2 33/37] binman: Add a utility library for coreboot CBFS

2019-07-08 Thread Simon Glass
Coreboot uses a simple flash-based filesystem called Coreboot Filesystem
(CBFS) to organise files used during boot. This allows files to be named
and their position in the flash to be set. It has special features for
dealing with x86 devices which typically memory-map their SPI flash to the
top of 32-bit address space and need a 'boot block' ending there.

Create a library to help create and read CBFS files. This includes a
writer class, a reader class and associated other helpers. Only a subset
of features are currently supported.

Signed-off-by: Simon Glass 
---

Changes in v2:
- Deal with travis's old lz4 version by skipping tests as necessary
- Install lzma tool in travis
- Skip use of cbfstool in tests if it is not available

 .travis.yml|   1 +
 tools/binman/binman.py |   4 +-
 tools/binman/cbfs_util.py  | 720 +
 tools/binman/cbfs_util_test.py | 540 +
 4 files changed, 1264 insertions(+), 1 deletion(-)
 create mode 100644 tools/binman/cbfs_util.py
 create mode 100755 tools/binman/cbfs_util_test.py

diff --git a/.travis.yml b/.travis.yml
index c883198abbf..70d89d3e233 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,6 +32,7 @@ addons:
 - device-tree-compiler
 - lzop
 - liblz4-tool
+- lzma-alone
 - libisl15
 - clang-7
 - srecord
diff --git a/tools/binman/binman.py b/tools/binman/binman.py
index 52c03f68c6d..613aad5c451 100755
--- a/tools/binman/binman.py
+++ b/tools/binman/binman.py
@@ -62,6 +62,7 @@ def RunTests(debug, verbosity, processes, test_preserve_dirs, 
args, toolpath):
 name to execute (as in 'binman -t testSections', for example)
 toolpath: List of paths to use for tools
 """
+import cbfs_util_test
 import elf_test
 import entry_test
 import fdt_test
@@ -90,7 +91,8 @@ def RunTests(debug, verbosity, processes, test_preserve_dirs, 
args, toolpath):
 suite = unittest.TestSuite()
 loader = unittest.TestLoader()
 for module in (entry_test.TestEntry, ftest.TestFunctional, 
fdt_test.TestFdt,
-   elf_test.TestElf, image_test.TestImage):
+   elf_test.TestElf, image_test.TestImage,
+   cbfs_util_test.TestCbfs):
 # Test the test module about our arguments, if it is interested
 if hasattr(module, 'setup_test_args'):
 setup_test_args = getattr(module, 'setup_test_args')
diff --git a/tools/binman/cbfs_util.py b/tools/binman/cbfs_util.py
new file mode 100644
index 000..197cff89509
--- /dev/null
+++ b/tools/binman/cbfs_util.py
@@ -0,0 +1,720 @@
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright 2019 Google LLC
+# Written by Simon Glass 
+
+"""Support for coreboot's CBFS format
+
+CBFS supports a header followed by a number of files, generally targeted at SPI
+flash.
+
+The format is somewhat defined by documentation in the coreboot tree although
+it is necessary to rely on the C structures and source code (mostly cbfstool)
+to fully understand it.
+
+Currently supported: raw and stage types with compression
+"""
+
+from __future__ import print_function
+
+from collections import OrderedDict
+import io
+import struct
+import sys
+
+import command
+import elf
+import tools
+
+# Set to True to enable printing output while working
+DEBUG = False
+
+# Set to True to enable output from running cbfstool for debugging
+VERBOSE = False
+
+# The master header, at the start of the CBFS
+HEADER_FORMAT  = '>'
+HEADER_LEN = 0x20
+HEADER_MAGIC   = 0x4f524243
+HEADER_VERSION1= 0x31313131
+HEADER_VERSION2= 0x31313132
+
+# The file header, at the start of each file in the CBFS
+FILE_HEADER_FORMAT = b'>8s'
+FILE_HEADER_LEN= 0x18
+FILE_MAGIC = b'LARCHIVE'
+FILENAME_ALIGN = 16  # Filename lengths are aligned to this
+
+# A stage header containing information about 'stage' files
+# Yes this is correct: this header is in litte-endian format
+STAGE_FORMAT   = ' offset:
+raise ValueError('No space for data before offset %#x (current 
offset %#x)' %
+ (offset, fd.tell()))
+fd.write(tools.GetBytes(self._erase_byte, offset - fd.tell()))
+
+def _align_to(self, fd, align):
+"""Write out pad bytes until a given alignment is reached
+
+This only aligns if the resulting output would not reach the end of the
+CBFS, since we want to leave the last 4 bytes for the master-header
+pointer.
+
+Args:
+fd: File objext to write to
+align: Alignment to require (e.g. 4 means pad to next 4-byte
+boundary)
+"""
+offset = align_int(fd.tell(), align)
+if offset < self._size:
+self._skip_to(fd, offset)
+
+def add_file_stage(self, name, data):
+"""Add a new stage file to the CBFS
+
+Args:
+name: String file name to put in CBFS (does not need to correspond
+