[yocto] Automated license incompatibility checks
Hi, I have written a bbclass that does license conflict checking before building images for distribution. I am wondering if that is something you as a project would be interested in integrating, e.g. into poky. I am also looking for feedback to my approach and ideas on how to handle corner cases, such as the OpenSSL exception. To give you a short and rough overview and to avoid wasting your time with a code review before we discussed the general approach, here's the commit message for my local change implementing this: > Implement automatic license conflict checking as QA check. This check > adds a hook to the do_rootfs task, gathers a list of packages to be > installed into the root file system, their licenses and dependency > relations. It then walks this list and removes all licenses or part of > license expressions that are not satisfiable using licenses from a > configurable whitelist. For example, > (GPL-3.0 & LGPL-3.0+) | GPL-2.0 > with a whitelist of "GPL-2.0" will be transformed to > GPL-2.0 > before running license conflict checking. This ensures none of the > licenses you don't want to distribute in your image is required to > fulfill all license constraints. > > Afterwards, each package's license is checked against those of its > dependencies one by one. To detect conflicts, the two license > expressions are converted into conjunctive normal form, concatenated > with all relevant license conflicts read from a configuration file and > handed over to a SAT solver (currently minisat via the satispy python > library). If the solver indicates the expression is satisfiable, there > is no license conflict. If the expression cannot be solved, the > licenses conflict. What are your thoughts on the additional minisat dependency? I assume it would be possible to make it a requirement only if somebody is actually using the bbclass? Most distributions I checked do not currently ship satispy. Would bundling it with poky (or wherever the bbclass + tests would go) be a viable option? -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Reinhard Stolle Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Automated license incompatibility checks
Hi Paul, On Wed, Jan 28, 2015 at 10:25:29AM +, Paul Eggleton wrote: > This sounds like something we should be handling in our > INCOMPATIBLE_LICENSE check code, although it's not clear that we > currently handle where alternatives to an incompatible license are > available for a recipe/package, so that looks like it would be new > functionality. You haven't mentioned INCOMPATIBLE_LICENSE - does your > solution build on that, or replace it? Some of this duplicates what INCOMPATIBLE_LICENSE does, so it should probably replace or use it. However, INCOMPATIBLE_LICENSE is a blacklist, and I'm using a whitelist approach, because in our use case we really only want to distribute under licenses we have explicitly reviewed. I'd like to solve the problem of shipping an image that contains, for example, GPL-2 code that links against BSD-4-Clause code, which are incompatible licenses[1]. Despite its name, the INCOMPATIBLE_LICENSE code does not check for this situation. It is merely a blacklist for licenses that should not be packaged at all, regardless of their dependency context. Our use case includes configuring the license whitelist separately for each image we build, because we have different requirements for development and production images (but that has been possible with INCOMPATIBLE_LICENSE before). Does that shed some light on what I'm trying to achieve? Cheers, Clemens [1] https://www.gnu.org/licenses/license-list.html#OriginalBSD -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Reinhard Stolle Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [PATCH] bitbake: fetch2: Revalidate checksums, YOCTO #5571
[YOCTO #5571] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571
The following workflow (whether accidentally or deliberately) would
previously not result in a checksum error, but would be helpful to do
so:
- Write a recipe with correct checksums
- Fetch the sources for this recipe using bitbake
- Change the checksums
Since the bitbake fetcher writes a done stamp after the initial download
and does not verify the checksums again (even if they are changed in the
recipe afterwards), the change of checksums is silently ignored.
Fix this without the overhead of computing the checksums from scratch on
every do_fetch by storing them in pickled format in the done stamp and
verifying that they still match those in the recipe.
Signed-off-by: Clemens Lang
---
bitbake/lib/bb/fetch2/__init__.py | 106 ++
1 file changed, 96 insertions(+), 10 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/__init__.py
b/bitbake/lib/bb/fetch2/__init__.py
index 599ea8c..c0c8312 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -45,6 +45,13 @@ _checksum_cache = bb.checksum.FileChecksumCache()
logger = logging.getLogger("BitBake.Fetcher")
+try:
+import cPickle as pickle
+except ImportError:
+import pickle
+logger.info("Importing cPickle failed. "
+"Falling back to a very slow implementation.")
+
class BBFetchException(Exception):
"""Class all fetch exceptions inherit from"""
def __init__(self, message):
@@ -525,7 +532,7 @@ def fetcher_compare_revisions(d):
def mirror_from_string(data):
return [ i.split() for i in (data or "").replace('\\n','\n').split('\n')
if i ]
-def verify_checksum(ud, d):
+def verify_checksum(ud, d, precomputed={}):
"""
verify the MD5 and SHA256 checksum for downloaded src
@@ -533,13 +540,28 @@ def verify_checksum(ud, d):
the downloaded file, or if BB_STRICT_CHECKSUM is set and there are no
checksums specified.
+Returns a dict of checksums that can be stored in a done stamp file and
+passed in as precomputed parameter in a later call to avoid re-computing
+the checksums from the file. This allows verifying the checksums of the
+file against those in the recipe each time, rather than only after
+downloading. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571.
"""
+_MD5_KEY = "md5"
+_SHA256_KEY = "sha256"
+
if ud.ignore_checksums or not ud.method.supports_checksum(ud):
-return
+return {}
-md5data = bb.utils.md5_file(ud.localpath)
-sha256data = bb.utils.sha256_file(ud.localpath)
+if _MD5_KEY in precomputed:
+md5data = precomputed[_MD5_KEY]
+else:
+md5data = bb.utils.md5_file(ud.localpath)
+
+if _SHA256_KEY in precomputed:
+sha256data = precomputed[_SHA256_KEY]
+else:
+sha256data = bb.utils.sha256_file(ud.localpath)
if ud.method.recommends_checksum(ud):
# If strict checking enabled and neither sum defined, raise error
@@ -589,6 +611,66 @@ def verify_checksum(ud, d):
if len(msg):
raise ChecksumError('Checksum mismatch!%s' % msg, ud.url, md5data)
+return {
+_MD5_KEY: md5data,
+_SHA256_KEY: sha256data
+}
+
+
+def verify_donestamp(ud, d):
+"""
+Check whether the done stamp file has the right checksums (if the fetch
+method supports them). If it doesn't, delete the done stamp and force
+a re-download.
+
+Returns True, if the donestamp exists and is valid, False otherwise. When
+returning False, any existing done stamps are removed.
+"""
+if not os.path.exists(ud.donestamp):
+return False
+
+if not ud.method.supports_checksum(ud):
+# done stamp exists, checksums not supported; assume the local file is
+# current
+return True
+
+if not os.path.exists(ud.localpath):
+# done stamp exists, but the downloaded file does not; the done stamp
+# must be incorrect, re-trigger the download
+bb.utils.remove(ud.donestamp)
+return False
+
+precomputed_checksums = {}
+try:
+with open(ud.donestamp, "rb") as cachefile:
+pickled = pickle.Unpickler(cachefile)
+precomputed_checksums.update(pickled.load())
+except Exception as e:
+# Avoid the warnings on the upgrade path from emtpy done stamp files to
+# those containing the checksums.
+if not isinstance(e, EOFError):
+# Ignore errors, they aren't fatal
+logger.warn("Couldn't load checksums from donestamp %s: %s "
+"(msg: %s)" % (ud.donestamp, type(e).__name__, str(e)))
+
+try:
+checksums = verify_ch
Re: [yocto] [PATCH] bitbake: fetch2: Revalidate checksums, YOCTO #5571
Hi, On Mon, Feb 23, 2015 at 03:22:43PM +0100, Clemens Lang wrote: > [YOCTO #5571] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571 Can somebody review this? Is there anything I need to do to get this considered for merging? -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Reinhard Stolle Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [PATCH] bitbake: fetch2: Revalidate checksums, YOCTO #5571
Hi Alejandro, On Thu, Feb 26, 2015 at 04:17:23PM -0600, Alejandro Hernandez wrote: > This is the problem I see, I am not sure we want to import extra libraries > here, as you state the use of pickle might be ok since it avoids the > overhead of calculating the checksums each time, but I don't know if it is > actually worth it, since checksums are calculated fairly quick anyway, this > is my personal opinion, you may still submit it for review. I don't think the use of cPickle is an issue here. It's already widely used in other parts of the bitbake codebase. I agree one could just re-compute the checkums every time, but that might take a while for larger downloaded files, especially for SHA-256. > And perhaps the reason it hasn't been reviewed is that I believe this > patch should go to bitbake-de...@lists.openembedded.org instead. I've re-sent the patch to bitbake-devel, thanks for the pointer. > I also had this WIP branch on contrib, its from a couple of months > back its "dirty" but in case you'd like to look around. > > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=hsalejandro/5571&id=2f59e26e534dc6b6d4c5bb8d78c574042e0fe7a7 Thanks, that was helpful in confirming that my approach isn't completely unreasonable. Clemens -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Reinhard Stolle Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] [meta-raspberrypi][PATCH] firmware.inc: Fetch a zip instead of cloning a git repo
Hello, On Fri, Jun 26, 2015 at 09:31:14AM +0100, Burton, Ross wrote: > Github also can and will regenerate these tarballs whenever it feels > like it, so you'll need to periodically update the checksums. > Obviously as existing developers will tend to have the tarballs cached > locally, it can be a while before this failure is reported back. While that does happen from time to time it's pretty rare. I see maybe one case of this every couple of months in MacPorts. Additionally, after a while the checksums generally change back again in almost all cases. So, yes, this brings its own set of problems, but is still a worthwhile improvement over the current situation IMO. Best regards, Clemens -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Reinhard Stolle Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] building on os x
On Mon, Dec 21, 2015 at 11:24:52AM +, Burton, Ross wrote: > Well with El Capitan's improved security apparently crippling > LD_PRELOAD (so I hear, unverified currently), Pseudo won't work, which > means an alternative will need to be researched and implemented. El Capitan has "System Integrity Protection", which is a flag on certain files that now no longer can be changed, even with root privileges. Pretty much all binaries in /usr/bin and /bin have this flag set, so it does affect common things like the shell and compilers. The kernel strips any variables affecting the loader (including DYLD_INSERT_LIBRARIES, OS X' equivalent to LD_PRELOAD) when executables with the SIP flag are started. So yes, this effectively kills library preloading on OS X for anything but your own binaries. I implemented a workaround in MacPorts, where we also use library preloading for sanity checks that works by copying affected executables (thus stripping the flag) and then running the copy instead. Of course, this requires hooking execve(2) and posix_spawn(2), which I don't think pseudo does at the moment. The commit doing most of the grunt work is at http://trac.macports.org/changeset/141420 if somebody wants to give this a shot. HTH, Clemens -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Reinhard Stolle Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] [PATCH] ldconfig-native: Fix ELF flags on 64-bit binaries
Yocto's ldconfig-native was exported from an old version of glibc and
generates an ld.so.cache that is partially ignored by current versions
of glibc when loading libraries. This occurs for 64-bit binaries, where
glibc expects FLAG_ELF_LIBC6, but ldconfig-native only generates the
standard ELF flag. Fix this with an additional patch on top of the patch
for now.
You can verify this by applying the patch below to your target copy of
glibc and running
LD_DEBUG=libs /lib64/ld-linux-x86-64.so.2 --list $anybinary
--- ./elf/dl-cache.c.orig 2016-01-07 11:05:36.823447171 +0100
+++ ./elf/dl-cache.c2016-01-07 11:19:53.925878628 +0100
@@ -106,6 +106,8 @@
if (_dl_cache_check_flags (flags) \
&& _dl_cache_verify_ptr (lib->value)) \
{ \
+ if (__glibc_unlikely (GLRO_dl_debug_mask & DL_DEBUG_LIBS))\
+ _dl_debug_printf (" considering file %s\n", cache_data
+ lib->value); \
if (best == NULL || flags == GLRO(dl_correct_cache_id)) \
{ \
HWCAP_CHECK; \
@@ -117,6 +119,9 @@
searching. */ \
break; \
} \
+ } else {\
+ if (__glibc_unlikely (GLRO_dl_debug_mask &
DL_DEBUG_LIBS)) \
+ _dl_debug_printf (" ignoring file %s due to flags %x,
expected %x\n", cache_data + lib->value, lib->flags,
GLRO(dl_correct_cache_id)); \
} \
} \
while (++middle <= right);\
@@ -265,14 +270,23 @@
/* Only accept hwcap if it's for the right platform. */
#define HWCAP_CHECK \
- if (lib->hwcap & hwcap_exclude)\
+ if (lib->hwcap & hwcap_exclude) {
\
+ if (__glibc_unlikely (GLRO_dl_debug_mask & DL_DEBUG_LIBS))\
+ _dl_debug_printf (" hwcap mismatch %x vs. %x\n", lib->hwcap,
hwcap_exclude); \
continue; \
- if (GLRO(dl_osversion) && lib->osversion > GLRO(dl_osversion)) \
+ }
\
+ if (GLRO(dl_osversion) && lib->osversion > GLRO(dl_osversion)) { \
+ if (__glibc_unlikely (GLRO_dl_debug_mask & DL_DEBUG_LIBS))\
+ _dl_debug_printf (" os version mismatch %x vs. %x\n",
lib->osversion, GLRO(dl_osversion)); \
continue; \
+ }
\
if (_DL_PLATFORMS_COUNT\
&& (lib->hwcap & _DL_HWCAP_PLATFORM) != 0 \
- && (lib->hwcap & _DL_HWCAP_PLATFORM) != platform) \
- continue
+ && (lib->hwcap & _DL_HWCAP_PLATFORM) != platform) { \
+ if (__glibc_unlikely (GLRO_dl_debug_mask & DL_DEBUG_LIBS))\
+ _dl_debug_printf (" platform mismatch %x vs. %x\n", lib->hwcap &
_DL_HWCAP_PLATFORM, platform); \
+ continue; \
+ }
SEARCH_CACHE (cache_new);
}
else
This version of ldconfig-native should really be replaced with a version
matching the glibc source in use on the target platform.
Signed-off-by: Clemens Lang
---
.../add-64-bit-flag-for-ELF64-entries.patch| 27 --
1 file changed, 20 insertions(+), 7 deletions(-)
diff --git
a/meta/recipes-core/glibc/ldconfig-native-2.12.1/add-64-bit-flag-for-ELF64-entries.patch
b/meta/recipes-core/glibc/ldconfig-native-2.12.1/add-64-bit-flag-for-ELF64-entries.patch
index a9af110..f4e38d4 100644
---
a/meta/recipes-core/glibc/ldconfig-native-2.12.1/add-64-bit-flag-for-ELF64-entries.patch
+++
b/meta/recipes-core/glibc/ldconfig-native-2.12.1/add-64-bit-flag-for-ELF64-entries.patch
@@ -64,7 +64,7 @@ index 0bf0de3..6e87afc 100644
#undef check_ptr
#define check_ptr(ptr)
Re: [yocto] linux/limits.h: No such file or directory
Hi, Takashi Matsuzawa wrote: > >configure: error: C preprocessor "x86_64-pokysdk-linux-gcc -E > --sysroot=/mnt/ssd2/>yocto/dev/tmp/x86-wk3/sysroots/x86_64-nativesdk-pokysdk-linux > > " fails sanity check > >| See `config.log' for more details. > > If I look into config.log > (gcc-4.9.2/build.x86_64-pokysdk-linux.x86_64-pokysdk-linux/libgcc/config.log) > > as suggested, I can see the following. > It says linux/limits.h is not found. > > (in config.log) > >usr/include/bits/local_lim.h:38:26: fatal error: linux/limits.h: No such > file or directory This looks a lot like a problem we’ve seen a while ago. See https://lists.yoctoproject.org/pipermail/yocto/2015-July/025856.html which is the mailing list post a colleague sent to the list back then. Looking at our git history, the workaround we did back then was add a dependency from gcc-crosssdk-initial to nativesdk-linux-libc-headers if $TARGET_OS starts with “linux”. HTH, Clemens -- BMW Car IT GmbH Clemens Lang Spezialist Entwicklung Lise-Meitner-Straße 14 89081 Ulm Tel: +49-731-37804182 Mail: clemens.l...@bmw-carit.de Web: http://www.bmw-carit.de BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Alexis Trolin Sitz und Registergericht: München HRB 134810 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] CommonAPI *.so bindings
Hi Pawel,
Pawel Suchy wrote:
> That is causing a problem with binding CommonAPI dynamic libraries via
> commonapi.ini.
> The commonapi.ini file contains sections where dependencies to proxy
> objects could be defined:
> [proxy]
> local:commonapi.examples.Test:commonapi.examples.Test=libTest-DBus.so
>
> But it cannot take link to versioned library because of that code:
> http://docs.projects.genivi.org/ipc.common-api-runtime/3.1.5/Runtime_8cpp_source.html#l00283
>
> Runtime::loadLibrary(const std::string &_library) {
> 321 std::string itsLibrary(_library);
> 322
> 323 // TODO: decide whether this really is a good idea...
> 324 #ifdef WIN32
> 325 if (itsLibrary.rfind(".dll") != itsLibrary.length() - 4) {
> 326 itsLibrary += ".dll";
> 327 }
> 328 #else
> 329 if (itsLibrary.rfind(".so") != itsLibrary.length() - 3) {
> 330 itsLibrary += ".so";
> 331 }
> 332 #endif
>
> Questions:
> 1. How to solve best this?
Our solution to that is to not rely on runtime loading of CommonAPI
libraries and just link the libraries that you need into the binary
directly, in which case this code will not be run at all. If your project
does not need the ability to switch between different communication
backends of CommonAPI, that should also work for you.
Note that you may need -Wl,--push-state,--no-as-needed,-library,--pop-state
depending on what your libraries look like.
HTH,
Clemens
--
BMW Car IT GmbH
Clemens Lang
Spezialist Entwicklung
Lise-Meitner-Straße 14
89081 Ulm
Tel: +49-731-37804182
Mail: clemens.l...@bmw-carit.de
Web: http://www.bmw-carit.de
BMW Car IT GmbH
Geschäftsführer: Michael Würtenberger und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Building on MacOS X
Hi, > On 12 Jan 2017, at 23:59, Mark Hatle wrote: > > As far as I know pseudo and the security introduced in 10.11 that affect > preloading is likely the biggest technical problem... everything else is just > "it's not Linux”. With System Integrity Protection disabled, pseudo should still work as it did before, if that’s an acceptable step for you. If it isn’t, Apple’s new limitations can also be worked around in pseudo by hooking the exec(2) and posix_spawn(2) syscalls, checking if the binary to be executed is under system integrity protection, making a copy without the SIP-bit if it is and transparently running that copy instead. That code would need to be written, though (Let me know if you want to do that, I have the code for a different project.). It’s probably only a matter of time until Apple prevents that from working, too, though, e.g. by making some standard system tools signed binaries that no longer load preloaded libraries. HTH, Clemens -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Alexis Trolin Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Repeteability in image compilation. How to achieve it? How to measure it?
Hi, > On 3 Feb 2017, at 14:11, Daniel. wrote: > > Hi everybody, > > How can I be sure that the same image built by distinct building hosts > are trully identical?? Is it possible to measure that? I can recommend tooling offered by the reproducible builds project. Especially diffoscope can be very helpful in this. See https://reproducible-builds.org/tools/. You may want to start comparing individual packages first, before you tackle image creation, though. HTH, Clemens -- Clemens Lang • Development Specialist BMW Car IT GmbH • Lise-Meitner-Str. 14 • 89081 Ulm • http://bmw-carit.com - BMW Car IT GmbH Geschäftsführer: Michael Würtenberger und Alexis Trolin Sitz und Registergericht: München HRB 134810 - -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
