Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
insights to the problem and seeing if anyone is interested in the feature for
mainline.
The sstate-cache of a recipe is tied closely to its build dependencies, as the
signature generated for a task includes all parent task's signatures as part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.
What this patchset does is propose to add another optional variable to recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
generation it excludes any parent tasks that come from dependencies listed in
it. This is to break the signature change domino effect.
This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks that use
CDEPENDS and allow for deciding to re-execute a task despite being covered by
sstate-cache.
The patchset is based on the jethro branch for the poky repository, as this is
the branch that we are using. If the general idea sounds good, we can consider
porting it to master.
Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility report
containing the file checksums of all files that it packages and which file paths
they are at. Another recipe, cdepends-test2, can then strip this compatibility
report to the minimal files it needs to be consistent and can compare the latest
checksums it used to configure/compile/install with and if they have changed,
trigger a rebuild. If not, the previous version restored from sstate-cache is
used.
We are still experimenting with the usages of this, including the use of having
abi-compliance-checker to compare the ABI of shared libraries as a compatibility
checker during RunQueue and using the results to avoid rebuilding child recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.
We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet
the feature requirements of what this compatibility checker callback is doing,
although maybe when porting to master we could refactor to make better use of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and if
people are interested in making use of it and their use cases.
Kind regards,
Michael Ho
--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm
Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael...@bmw-carit.de
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
From 0afcfc5bde251e96a434c345b4e0e4db895feeae Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:52:12 +0200
Subject: [PATCH 1/9] cache.py: add support for CDEPENDS
To: yocto@yoctoproject.org
Modifies the bitbake recipe cache handling to now parse also the CDEPENDS
variable, the cdepends flag that can be set for tasks, and also the machine
architecture for the recipe target (since compatibility will be tied to the
machine type also).
---
bitbake/lib/bb/cache.py | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index ab09b08..3596242 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -146,6 +146,9 @@ class CoreRecipeInfo(RecipeInfoCommon):
self.fakerootenv = self.getvar('FAKEROOTENV', metadata)
self.fakerootdirs = self.getvar('FAKEROOTDIRS', metadata)
self.fakerootnoenv = self.getvar('FAKEROOTNOENV', metadata)
+ self.cdepends = self.depvar('CDEPENDS', metadata)
+ self.task_cdepends = self.flaglist('cdepends', self.tasks, metadata, True)
+ self.target_sys = self.getvar('MULTIMACH_TARGET_SYS', metadata)
@classmethod
def init_cacheData(cls, cachedata):
@@ -172,6 +175,12 @@ class CoreRecipeInfo(RecipeInfoCommon):
cachedata.rproviders = defaultdict(list)
cachedata.packages_dynamic = defaultdict(list)
+ cachedata.all_cdepends = []
+ cachedata.compatible_deps = defaultdict(list)
+ cachedata.all_compatible_deps = defaultdict(list)
+ cachedata.compatible_task_deps = {}
+ cachedata.target_sys = {}
+
cachedata.rundeps = defaultdict(lambda: defaultdict(list))
cachedata.runrecs = defaultdict(lambda: defaultdict(list))
cachedata.possible_world = []
@@ -197,6 +206,9 @@ class CoreRecipeInfo(RecipeInfoCommon):
cachedata.stamp_extrainfo[fn] = self.stamp_extrainfo
cachedata.file_checksums[fn] = self.file_checksums
+ cachedata.compatible_task_deps[fn] = self.task_cdepends
+ cachedata.target_sys[fn] = self.target_sys
+
provides = [self.pn]
for provide in self.provides:
if provide not in provides:
@@ -214,6 +226,23 @@ class CoreRecipeInfo(RecipeInfoCommon):
if dep not in cachedata.all_depends:
cachedata.all_depends.append(dep)
+ for cdep in self.cdepends:
+ if cdep not in cachedata.deps[fn]:
+ cachedata.deps[fn].append(cdep)
+ if cdep not in cachedata.compatible_deps[fn]:
+ cachedata.compatible_deps[fn].append(cdep)
+ bb.debug(1, "cdepends: Adding %s to compatible_deps for %s", cdep, fn)
+ if cdep not in cachedata.all_compatible_deps[fn]:
+ cachedata.all_compatible_deps[fn].append(cdep)
+ bb.debug(1, "cdepends: Adding %s to all_compatible_deps for %s", cdep, fn)
+ if cdep not in cachedata.all_depends:
+ cachedata.all_depends.append(cdep)
+ cachedata.all_cdepends.append(cdep)
+
+ for cdep in cachedata.all_compatible_deps[fn]:
+ if cdep not in cachedata.compatible_deps[fn]:
+ bb.warn("cdepends: %s removed from CDEPENDS for %s because it is also in DEPENDS" % (cdep, fn))
+
rprovides = self.rprovides
for package in self.packages:
cachedata.packages[package].append(fn)
--
2.7.4
From e2c8ba70a2b723e1fc67215b1ff0ccb105341633 Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:52:52 +0200
Subject: [PATCH 2/9] siggen.py: add support for CDEPENDS
To: yocto@yoctoproject.org
This change modifies the basic signature generator to not add the task
dependencies sstate hash of a dependers task when that task comes from
a provider that is listed in the CDEPENDS variable.
---
bitbake/lib/bb/siggen.py | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 0352e45..e4b8374 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -86,6 +86,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST", True) or "").split())
self.taskwhitelist = None
+ self.cdepends_disabled = data.getVar("BB_CDEPENDS_DISABLED", True)
self.init_rundepcheck(data)
def init_rundepcheck(self, data):
@@ -180,10 +181,20 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.runtaskdeps[k] = []
self.file_checksum_values[k] = {}
recipename = dataCache.pkg_fn[fn]
+
+ def is_cdepends_task(fn, dep, dataCache):
+ for cdep in dataCache.compatible_deps[fn]:
+ for cdep_provider_fn in dataCache.providers[cdep]:
+ if dep.startswith(cdep_provider_fn):
+ return True
+ return False
+
for dep in sorted(deps, key=clean_basepath):
depname = dataCache.pkg_fn[self.pkgnameextract.search(dep).group('fn')]
if not self.rundep_check(fn, recipename, task, dep, depname, dataCache):
continue
+ if not self.cdepends_disabled and is_cdepends_task(fn, dep, dataCache):
+ continue
if dep not in self.taskhash:
bb.fatal("%s is not in taskhash, caller isn't calling in dependency order?", dep)
data = data + self.taskhash[dep]
--
2.7.4
From 229bbc3e3feb0b60a70cb98bc1648ffe09b106ce Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:53:31 +0200
Subject: [PATCH 3/9] runqueue.py: add support for CDEPENDS
To: yocto@yoctoproject.org
This change modifies the runqueue execution to inject a check where
tasks with a cdepends flag run an extra section of code that invokes a
custom sstate hashing check and compatibility checker (defined by the
recipe).
The result of this extra section of code is then used to decide
whether to skip the sstate/stamp coverage checks during the runqueue
execution.
---
bitbake/lib/bb/runqueue.py | 197 ++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 186 insertions(+), 11 deletions(-)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 878028a..3962ee1 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -35,6 +35,7 @@ import bb
from bb import msg, data, event
from bb import monitordisk
import subprocess
+import imp
try:
import cPickle as pickle
@@ -237,6 +238,7 @@ class RunQueueData:
self.runq_depends = []
self.runq_revdeps = []
self.runq_hash = []
+ self.runq_arch = []
def runq_depends_names(self, ids):
import re
@@ -555,6 +557,7 @@ class RunQueueData:
self.runq_depends.append(depends)
self.runq_revdeps.append(set())
self.runq_hash.append("")
+ self.runq_arch.append("")
runq_build.append(0)
@@ -668,6 +671,7 @@ class RunQueueData:
del runq_build[listid-delcount]
del self.runq_revdeps[listid-delcount]
del self.runq_hash[listid-delcount]
+ del self.runq_arch[listid-delcount]
delcount = delcount + 1
maps.append(-1)
@@ -806,6 +810,16 @@ class RunQueueData:
if hasattr(bb.parse.siggen, "tasks_resolved"):
bb.parse.siggen.tasks_resolved(virtmap, virtpnmap, self.dataCache)
+ # Iterate over the fn list and cache the variable MULTIMACH_TARGET_SYS
+ cached_archs = []
+ for i in range(max(self.runq_fnid)+1):
+ cached_archs.append("")
+ dealtwith = set()
+ for i in self.runq_fnid:
+ if i not in dealtwith:
+ dealtwith.add(i)
+ cached_archs[i] = self.dataCache.target_sys[self.taskData.fn_index[i]]
+
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(range(len(self.runq_fnid)))
@@ -818,6 +832,9 @@ class RunQueueData:
for dep in self.runq_depends[task]:
procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
+ self.runq_arch[task] = cached_archs[self.runq_fnid[task]]
+ if self.runq_task[task] == "do_compatibility_report":
+ bb.debug(1, "cdepends: hash for %s:%s is %s" % (self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], self.runq_hash[task]))
return len(self.runq_fnid)
@@ -1281,6 +1298,8 @@ class RunQueueExecute:
self.runq_running = []
self.runq_complete = []
+ self.tainted = []
+
self.build_stamps = {}
self.build_stamps2 = []
self.failed_fnids = []
@@ -1503,6 +1522,18 @@ class RunQueueExecuteTasks(RunQueueExecute):
"""
self.runq_complete[task] = 1
for revdep in self.rqdata.runq_revdeps[task]:
+ # Taint reverse dependency if task is tainted from a cdepends check
+ # in its build tree and if it is not chained by CDEPENDS (we don't
+ # want cdepends tainting to cross the compatibility checking
+ # boundaries)
+ if task in self.tainted:
+ taskfn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
+ taskpn = self.rqdata.dataCache.pkg_fn[taskfn]
+ revdepfn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[revdep]]
+ revdeppn = self.rqdata.dataCache.pkg_fn[revdepfn]
+ # Try to be smart so that tainting stops between CDEPENDS items
+ if revdepfn == taskfn or not taskpn in self.rqdata.dataCache.compatible_deps[revdep]:
+ self.tainted.append(revdep)
if self.runq_running[revdep] == 1:
continue
if self.runq_buildable[revdep] == 1:
@@ -1559,17 +1590,159 @@ class RunQueueExecuteTasks(RunQueueExecute):
fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
taskname = self.rqdata.runq_task[task]
- if task in self.rq.scenequeue_covered:
- logger.debug(2, "Setscene covered task %s (%s)", task,
- self.rqdata.get_user_idstring(task))
- self.task_skip(task, "covered")
- return True
+ # Run compatibility check if task recipe has cdepends flag set and if stamp or setscene covered. Skip if
+ # CDEPENDS is disabled by config however.
+ # TODO: Skip if the cache compatibility task is not covered (maybe we crashed halfway through compile of the
+ # recipe, thus we want to continue with what we have in our workdir and not try to compatibility check -
+ # although we need to check the integrity of the workdir by comparing the sstate hash it starts with and
+ # ends with before we can add this check).
+ compatibility_check_ran = False
+ if (taskname in self.rqdata.dataCache.compatible_task_deps[fn]):
+ if (not self.cfgData.getVar("BB_CDEPENDS_DISABLED", True)) and (task in self.rq.scenequeue_covered or self.rq.check_stamp_task(task, taskname, cache=self.stampcache)):
+ # Parse recipe to extract cdepends variables and run compatibility check for each cdepends
+ tmp_cachedata = bb.cache.Cache.loadDataFull(fn, self.cooker.collection.get_file_appends(fn), self.cooker.data)
+
+ # Loop over all tasks that have the cdepends flag set
+ for cdepend in self.rqdata.dataCache.compatible_task_deps[fn][taskname].split():
+ compatibility_result = "incompatible"
+
+ cdepend_tool = tmp_cachedata.getVar(("CDEPENDS_TOOL_%s" % (cdepend,)), True) or ""
+ cdepend_searchdir = tmp_cachedata.getVar("CDEPENDS_SEARCHDIR", True) or ""
+ cdepend_workdir = tmp_cachedata.getVar("CDEPENDS_WORKDIR", True) or ""
+
+ logger.debug(1, "cdepends: cdepend=%s,cdepend_tool=%s,cdepend_searchdir=%s,cdepend_workdir=%s",
+ cdepend, cdepend_tool, cdepend_searchdir, cdepend_workdir)
+
+ # Determine the provider of the cdepends and get the architecture/hashes
+
+ # XXX: Potential room to improve by fixing this to properly look through indices. However it
+ # may not be possible to do so. We start with PN's and we need to find the actual provider
+ # and the task do_cache_compatibility_reports to calculate the hashes to build against
+ # (using all providers hashes if multiple providers are available)
+ cdepend_providers = self.rqdata.dataCache.providers[cdepend]
+ cdepend_preferred_provider = tmp_cachedata.getVar("PREFERRED_PROVIDER_%s" % (cdepend,)) or cdepend
+ cdepend_hashes = ""
+ cdepend_hashes_provider = []
+ cdepend_arch = ""
+ cdepend_arch_provider = "self"
+
+ if cdepend_preferred_provider in cdepend_providers:
+ for x in range(len(self.rqdata.runq_task)):
+ x_name = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]] + ":" + self.rqdata.runq_task[x]
+ if x_name == cdepend_preferred_provider + ":" + "do_compatibility_report":
+ cdepend_hashes = self.rqdata.runq_hash[x]
+ cdepend_hashes_provider.append(cdepend_preferred_provider)
+ cdepend_arch = self.rqdata.dataCache.target_sys[self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]]]
+ cdepend_arch_provider = cdepend_preferred_provider
+ break
+ else:
+ # if multiple providers available and no preferred provider, then merge all (sorted) hashes for safety
+ found_hashes = dict()
+ for cdepend_provider in cdepend_providers:
+ for x in range(len(self.rqdata.runq_task)):
+ x_name = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]] + ":" + self.rqdata.runq_task[x]
+ if x_name == cdepend_provider + ":" + "do_compatibility_report":
+ found_hashes[x_name] = self.rqdata.runq_hash[x]
+ cdepend_hashes_provider.append(cdepend_provider)
+ cdepend_arch = self.rqdata.dataCache.target_sys[self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]]]
+ cdepend_arch_provider = cdepend_provider
+ sorted_hashes = sorted(found_hashes.values())
+ for cdepend_provider_hash in sorted_hashes:
+ cdepend_hashes += cdepend_provider_hash
+
+ # if cdepends insane skip is set, ignore the sstate hashing checks
+ # (since we won't be able to fetch the required sstate hashes for
+ # dependencies without do_compatibility_report)
+ if not tmp_cachedata.getVar("CDEPENDS_INSANE_SKIP_%s" % (cdepend,), True):
+ if not cdepend_hashes:
+ bb.fatal("cdepends: Cannot find provider task for %s:do_compatibility_report (providers: %s)" % (cdepend, cdepend_providers))
+
+ logger.debug(1, "cdepends: Provider for %s: %s,runq_hash=%s,runq_arch=%s", cdepend, ','.join(cdepend_hashes_provider), cdepend_hashes, cdepend_arch)
+
+ # First just check if the sstate hashes are okay to try to avoid the actual compatibility check
+ cached_sstate_hashes_path = os.path.join(cdepend_workdir, "sstate-hashes")
+ logger.debug(2, "cdepends: Trying to grab sstate hashes from %s.", cached_sstate_hashes_path)
+
+ current_runq_hash = cdepend_hashes
+ cached_runq_hash = ""
+ try:
+ with open(cached_sstate_hashes_path, "r") as hashes_log:
+ hashes_log_lines = hashes_log.readlines()
+ for line in hashes_log_lines:
+ line = line.strip()
+ try:
+ target_name, cached_target_hash = line.split(" ", 1)
+ if target_name == cdepend:
+ cached_runq_hash = cached_target_hash
+ except ValueError:
+ continue
+ except (OSError, IOError) as e:
+ bb.warn("cdepends: Could not open cached sstate hashes for %s:%s: %s" % (fn, taskname, str(e)))
+
+ if not cached_runq_hash == "":
+ logger.debug(2, "cdepends: Comparing sstate hashes: cached: \"%s\" to latest: \"%s\".", cached_runq_hash, current_runq_hash)
+ if current_runq_hash == cached_runq_hash:
+ logger.debug(1, "cdepends: \"%s\" sstate hash \"%s\" still valid for %s:%s.", cdepend, cached_runq_hash, fn, taskname)
+ continue
+
+ # if the sstate hash does not hold, then run the cdepend check
+ logger.debug(1, "cdepends: \"%s\" sstate hash has changed from \"%s\" to \"%s\" for %s:%s.", cdepend, cached_runq_hash, current_runq_hash, fn, taskname)
+ else:
+ logger.debug(1, "cdepends: \"%s\" sstate hash was not found in compatibility cache for %s:%s", cdepend, fn, taskname)
+
+ # Run actual compatibility checkers from here onwards
+ latest_reports_dir = os.path.join(cdepend_searchdir, cdepend_arch, cdepend)
+ cached_reports_dir = os.path.join(cdepend_workdir, cdepend)
+
+ cdepend_module_name = ("cdepends_%s" % (cdepend_tool,))
+ f = None
+ try:
+ logger.debug(2, "cdepends: Searching for python module \"%s\".", cdepend_module_name)
+ f, filename, description = imp.find_module(cdepend_module_name)
+ cdepend_module = imp.load_module(cdepend_module_name, f, filename, description)
+ logger.debug(2, "cdepends: Running compatibility check for %s:%s using python module \"%s\". Passing arguments \"%s\" and \"%s\"",
+ fn, taskname, cdepend_module_name, cached_reports_dir, latest_reports_dir)
+ compatibility_result = cdepend_module.compatibility_check(cached_reports_dir, latest_reports_dir, tmp_cachedata)
+ bb.note("cdepends: Compatibility check for %s:%s using tool (%s) returned \"%s\" for (%s)." % (fn, taskname, cdepend_tool, compatibility_result, cdepend))
+ except ImportError as e:
+ bb.warn("cdepends: Could not find and load/run cdepends module %s" % (cdepend_module_name,))
+ finally:
+ if f is not None:
+ f.close()
+
+ # Set a flag to show we actually ran a compatibility check to help improve our notifications to the UI
+ compatibility_check_ran = True
+
+ if compatibility_result != "compatible":
+ logger.debug(1, "cdepends: Tainting build chain of %s:%s because of %s.", fn, taskname, cdepend)
+ self.tainted.append(task)
+ logger.debug(1, "cdepends: Tainted task %s due to compatibility changes", task)
+
+ if compatibility_result != "incompatible":
+ bb.fatal("cdepends: Fatal compatibility error for %s." % (task,))
+ else:
+ bb.note("cdepends: Skipping cdepends checks for %s (%s)" % (task, self.rqdata.get_user_idstring(task)))
- if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
- logger.debug(2, "Stamp current task %s (%s)", task,
- self.rqdata.get_user_idstring(task))
- self.task_skip(task, "existing")
- return True
+ # Don't believe setscene or stamp if tainted flag set for task
+ if task not in self.tainted:
+ if task in self.rq.scenequeue_covered:
+ logger.debug(2, "Setscene covered task %s (%s)", task,
+ self.rqdata.get_user_idstring(task))
+ self.task_skip(task, "covered")
+ if compatibility_check_ran:
+ bb.note("cdepends: Avoided rebuilding %s" % (fn,))
+ return True
+
+ if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
+ logger.debug(2, "Stamp current task %s (%s)", task,
+ self.rqdata.get_user_idstring(task))
+ self.task_skip(task, "existing")
+ if compatibility_check_ran:
+ bb.note("cdepends: Avoided rebuilding %s" % (fn,))
+ return True
+
+ if compatibility_check_ran and task in self.tainted:
+ bb.note("cdepends: Rebuilding %s" % (fn,))
taskdep = self.rqdata.dataCache.task_deps[fn]
if 'noexec' in taskdep and taskname in taskdep['noexec']:
@@ -1642,7 +1815,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
taskname = self.rqdata.runq_task[revdep]
deps = self.rqdata.runq_depends[revdep]
provides = self.rqdata.dataCache.fn_provides[fn]
- taskdepdata[revdep] = [pn, taskname, fn, deps, provides]
+ taskhash = self.rqdata.runq_hash[revdep]
+ taskarch = self.rqdata.runq_arch[revdep]
+ taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, taskarch]
for revdep2 in deps:
if revdep2 not in taskdepdata:
additional.append(revdep2)
--
2.7.4
From fe6f61e7d29c64d8ffa4943283784339ff2d7411 Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:54:27 +0200
Subject: [PATCH 4/9] taskdata.py: add support for CDEPENDS
To: yocto@yoctoproject.org
This change adds a new variable to the TaskData called cdepids and
contains the task ids of tasks that are CDEPENDS of the task being
called on.
---
bitbake/lib/bb/taskdata.py | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
index 4d12b33..5b9ea8c 100644
--- a/bitbake/lib/bb/taskdata.py
+++ b/bitbake/lib/bb/taskdata.py
@@ -74,6 +74,8 @@ class TaskData:
self.skiplist = skiplist
+ self.cdepids = {}
+
def getbuild_id(self, name):
"""
Return an ID number for the build target name.
@@ -212,6 +214,14 @@ class TaskData:
self.depids[fnid] = dependids.keys()
logger.debug(2, "Added dependencies %s for %s", str(dataCache.deps[fn]), fn)
+ # Work out build compatible dependencies
+ if not fnid in self.cdepids:
+ cdependids = {}
+ for cdepend in dataCache.compatible_deps[fn]:
+ cdependids[self.getbuild_id(cdepend)] = None
+ self.cdepids[fnid] = cdependids.keys()
+ logger.debug(2, "Added compatible dependencies %s for %s", str(dataCache.compatible_deps[fn]), fn)
+
# Work out runtime dependencies
if not fnid in self.rdepids:
rdependids = {}
--
2.7.4
From d0149bcd7fba3d432eebd6218d6199878bdd91f8 Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:54:39 +0200
Subject: [PATCH 5/9] rm_work.bbclass: add exception for
do_cache_compatibility_reports
To: yocto@yoctoproject.org
This change is needed to handle the situation of rm_work clearing
compatibility reports that are actually shared to depender recipes.
This is similar to the workarounds in place for do_package.
---
meta/classes/rm_work.bbclass | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
index 5e9efc1..ce802cd 100644
--- a/meta/classes/rm_work.bbclass
+++ b/meta/classes/rm_work.bbclass
@@ -76,6 +76,11 @@ do_rm_work () {
i=dummy
break
;;
+ *do_cache_compatibility_reports|*do_cache_compatibility_reports.*|*do_cache_compatibility_reports_setscene.*)
+ rm -f $i;
+ i=dummy
+ break
+ ;;
*_setscene*)
i=dummy
break
--
2.7.4
From 77070002f7970ccab89b38eb5eb91bc768120052 Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:55:24 +0200
Subject: [PATCH 6/9] native.bbclass: add support for CDEPENDS
To: yocto@yoctoproject.org
Remap CDEPENDS in the same way DEPENDS is remapped when extending a
recipe.
---
meta/classes/native.bbclass | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/meta/classes/native.bbclass b/meta/classes/native.bbclass
index bcbcd61..1357f1f 100644
--- a/meta/classes/native.bbclass
+++ b/meta/classes/native.bbclass
@@ -163,7 +163,18 @@ python native_virtclass_handler () {
nprovides.append(prov)
e.data.setVar("PROVIDES", ' '.join(nprovides))
-
+ map_dependencies("CDEPENDS", e.data)
+ for cdep in bb.utils.explode_deps(e.data.getVar("CDEPENDS") or ""):
+ cdep_tool = d.getVar("CDEPENDS_TOOL_%s" % (cdep,), True) or ""
+ if cdep == pn:
+ continue
+ elif "-cross-" in cdep:
+ cdep.replace("-cross", "-native")
+ elif not cdep.endswith("-native"):
+ cdep += "-native"
+ else:
+ pass
+ e.data.setVar("CDEPENDS_TOOL_%s" % (cdep,), cdep_tool)
}
addhandler native_virtclass_handler
--
2.7.4
From 2ce44b4ac4ebae7e90e07e95911e8c5be0ff15c6 Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:55:45 +0200
Subject: [PATCH 7/9] nativesdk.bbclass: add support for CDEPENDS
To: yocto@yoctoproject.org
Remap CDEPENDS in the same way DEPENDS is remapped when extending a
recipe. Remaps the CDEPENDS_TOOL_* variable also.
---
meta/classes/nativesdk.bbclass | 3 +++
meta/lib/oe/classextend.py | 5 +++++
2 files changed, 8 insertions(+)
diff --git a/meta/classes/nativesdk.bbclass b/meta/classes/nativesdk.bbclass
index f74da62..6a98732 100644
--- a/meta/classes/nativesdk.bbclass
+++ b/meta/classes/nativesdk.bbclass
@@ -86,6 +86,9 @@ python () {
clsextend.map_packagevars()
clsextend.map_variable("PROVIDES")
clsextend.map_regexp_variable("PACKAGES_DYNAMIC")
+
+ clsextend.rename_cdepends_tools()
+ clsextend.map_depends_variable("CDEPENDS")
}
addhandler nativesdk_virtclass_handler
diff --git a/meta/lib/oe/classextend.py b/meta/lib/oe/classextend.py
index 5107ecd..a19b213 100644
--- a/meta/lib/oe/classextend.py
+++ b/meta/lib/oe/classextend.py
@@ -108,6 +108,11 @@ class ClassExtender(object):
for subs in variables:
self.d.renameVar("%s_%s" % (subs, pkg_mapping[0]), "%s_%s" % (subs, pkg_mapping[1]))
+ def rename_cdepends_tools(self):
+ for cdep in (self.d.getVar("CDEPENDS", True) or "").split():
+ renamed_cdep = self.map_depends(cdep)
+ self.d.renameVar("CDEPENDS_TOOL_%s" % (cdep,), "CDEPENDS_TOOL_%s" % (renamed_cdep,))
+
class NativesdkClassExtender(ClassExtender):
def map_depends(self, dep):
if dep.startswith(self.extname):
--
2.7.4
From 7d9d7e67099ef8ed4c6aa32453bf5cbd8ae53957 Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 06:56:36 +0200
Subject: [PATCH 8/9] classes: add bbclasses compatible-depends and
compatible-depends-report
To: yocto@yoctoproject.org
Adds bbclasses that are used in conjunction with triggering the
CDEPENDS mechanisms in Bitbake.
The compatible-depends bbclass adds the base functionality to cache
compatibility reports from the dependencies.
The compatible-depends-report bbclass adds the base functionality to
generate the compatibility reports for recipes that can be depended
on via CDEPENDS.
---
.../classes/compatible-depends-report.bbclass | 32 +++++
meta-yocto/classes/compatible-depends.bbclass | 144 +++++++++++++++++++++
2 files changed, 176 insertions(+)
create mode 100644 meta-yocto/classes/compatible-depends-report.bbclass
create mode 100644 meta-yocto/classes/compatible-depends.bbclass
diff --git a/meta-yocto/classes/compatible-depends-report.bbclass b/meta-yocto/classes/compatible-depends-report.bbclass
new file mode 100644
index 0000000..ffb9e4e
--- /dev/null
+++ b/meta-yocto/classes/compatible-depends-report.bbclass
@@ -0,0 +1,32 @@
+# Not to be used directly - The concrete bbclass for a tool should inherit this
+
+# Working directory and output directory for reports
+# This needs to stage somewhere that sstate can recover so that when rebuilding
+# sysroots from sstate this is also recovered.
+COMPATIBILITY_REPORT_WORKDIR = "${WORKDIR}/compatibility-info/"
+COMPATIBILITY_REPORT_DIR = "${TMPDIR}/compatibility-info/${MULTIMACH_TARGET_SYS}/${PN}/"
+
+# Dummy task for generating reports - the concrete formats are expected to
+# append to this task to add their actual code for generating reports
+do_compatibility_report[dirs] += "${COMPATIBILITY_REPORT_WORKDIR}"
+do_compatibility_report () {
+ :
+}
+addtask compatibility_report after do_package before do_build
+
+# Make do_compatibility_report task SSTATE compatible
+SSTATETASKS += "do_compatibility_report"
+
+do_compatibility_report[sstate-inputdirs] = "${COMPATIBILITY_REPORT_WORKDIR}"
+do_compatibility_report[sstate-outputdirs] = "${COMPATIBILITY_REPORT_DIR}"
+
+python do_compatibility_report_setscene () {
+ sstate_setscene(d)
+}
+addtask do_compatibility_report_setscene
+
+# Needed for recipes like native XXX: is this better than just adding do_install by default?
+python __anonymous () {
+ if not d.getVarFlag("do_package", "task", True):
+ d.appendVarFlag('do_compatibility_report', 'depends', ' do_install')
+}
diff --git a/meta-yocto/classes/compatible-depends.bbclass b/meta-yocto/classes/compatible-depends.bbclass
new file mode 100644
index 0000000..3e6823d
--- /dev/null
+++ b/meta-yocto/classes/compatible-depends.bbclass
@@ -0,0 +1,144 @@
+# Recipes that use CDEPENDS inherit this bbclass
+
+# Add default CDEPENDS because this variable must be declared
+CDEPENDS ?= ""
+
+# Add CDEPENDS to the sstate hashing signatures so that changes to this variable
+# trigger sstate invalidations and can enforce rebuilds
+do_unpack[vardeps] += "CDEPENDS"
+do_cache_compatibility_reports[vardeps] += "CDEPENDS"
+
+# Delay the compiling tasks of the recipe to wait for the report data of other
+# recipes it CDEPENDS on (if they have a do_compatibility_report task).
+do_unpack[deptask] += "do_compatibility_report"
+
+# Give do_unpack task a cdepends flag, so runQueueExecuteTask can determine if
+# it should run compatibility checks prior to running do_unpack
+do_unpack[cdepends] = "${CDEPENDS}"
+
+# Working directories to get reports from and store them while working on them
+# Extracted by Bitbake during compatibility checking
+# Potential bug here, what if a dependency doesn't fit into these arch targets, how do we find out his architecture?
+# To fix: we now pass architecture through BBTASKDEPS - we need to use this somehow
+CDEPENDS_SEARCHDIR = "${TMPDIR}/compatibility-info/"
+CDEPENDS_STAGEDIR = "${WORKDIR}/dependency-compatibility-info-staging/"
+CDEPENDS_WORKDIR = "${WORKDIR}/dependency-compatibility-info/"
+
+# Task to copy the compatibility reports of its CDEPENDS to the local working
+# directory. These are then cached by sstate and placed into the expected
+# directory for Bitbake to use for future compatibility checks.
+python copy_compatibility_reports () {
+ import errno
+ import os
+ import shutil
+ import subprocess
+
+ cdepends = d.getVar("CDEPENDS", True).strip() or ""
+ cdepends_searchdir = d.getVar("CDEPENDS_SEARCHDIR", True)
+ cdepends_stagedir = d.getVar("CDEPENDS_STAGEDIR", True)
+
+ taskdepdata = d.getVar("BB_TASKDEPDATA", True)
+
+ if cdepends_stagedir:
+ # Need to make the directory ourselves because prefuncs are too early..
+ bb.utils.mkdirhier(cdepends_stagedir)
+
+ for cdepend in cdepends.split():
+ cdepend = cdepend.strip()
+ cdepend_arch = d.getVar("MULTIMACH_TARGET_SYS", True) or ""
+ cdepend_provider = cdepend
+ for task in taskdepdata:
+ if cdepend in taskdepdata[task][4]:
+ if taskdepdata[task][1] == "do_compatibility_report":
+ cdepend_arch = taskdepdata[task][6]
+ cdepend_provider = taskdepdata[task][0]
+
+ shutil.rmtree(os.path.join(cdepends_stagedir, cdepend), ignore_errors=True)
+ try:
+ shutil.copytree(os.path.join(cdepends_searchdir, cdepend_arch, cdepend_provider), os.path.join(cdepends_stagedir, cdepend), symlinks=True)
+ bb.debug(3, "cdepends: %s.copy_compatibility_reports: copied from %s to %s" % (d.getVar("PN", True), os.path.join(cdepends_searchdir, cdepend_arch, cdepend_provider), os.path.join(cdepends_stagedir, cdepend)))
+ except OSError as e:
+ # Sometimes we CDEPENDS on things with no compatibility report
+ # (ie. binary deliveries) - so avoid throwing errors when
+ # CDEPENDS_INSANE_SKIP is set for this dependency
+ insane_skip = d.getVar("CDEPENDS_INSANE_SKIP_%s" % (cdepend,), True)
+ if e.errno == errno.ENOENT and insane_skip:
+ pass
+ else:
+ raise e
+}
+
+do_cache_compatibility_reports () {
+ :
+}
+do_cache_compatibility_reports[prefuncs] += "copy_compatibility_reports"
+addtask cache_compatibility_reports after do_install do_package before do_populate_sysroot do_build
+
+# Make do_cache_compatibility_reports task SSTATE compatible
+SSTATETASKS += "do_cache_compatibility_reports"
+
+do_cache_compatibility_reports[dirs] = "${CDEPENDS_STAGEDIR} ${CDEPENDS_WORKDIR}"
+do_cache_compatibility_reports[sstate-inputdirs] = "${CDEPENDS_STAGEDIR}"
+do_cache_compatibility_reports[sstate-outputdirs] = "${CDEPENDS_WORKDIR}"
+
+python do_cache_compatibility_reports_setscene () {
+ sstate_setscene(d)
+}
+addtask do_cache_compatibility_reports_setscene
+
+python cache_sstate_hashes () {
+ import os
+
+ cdepends = d.getVar("CDEPENDS", True).strip() or ""
+ cdepends_hashes = dict()
+ if cdepends:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS: %s" % (d.getVar("PN", True), cdepends))
+ taskdepdata = d.getVar("BB_TASKDEPDATA", True)
+
+ for cdepend in cdepends.split():
+ cdepend = cdepend.strip()
+ cdepend_preferred_provider = d.getVar("PREFERRED_PROVIDER_%s" % (cdepend), True) or ""
+ cdepend_preferred_provider_hash = ""
+
+ cdepend_providers = ""
+ cdepend_hashes = ""
+ found_hashes = dict()
+ for task in taskdepdata:
+ if cdepend in taskdepdata[task][4]:
+ if taskdepdata[task][1] == "do_compatibility_report":
+ found_hashes[taskdepdata[task][0]] = taskdepdata[task][5]
+ if cdepend_preferred_provider == taskdepdata[task][0]:
+ cdepend_preferred_provider_hash = taskdepdata[task][5]
+
+ # Sort hashes
+ cdepend_providers = ','.join(found_hashes.keys())
+ sorted_hashes = sorted(found_hashes.values())
+ for cdepend_provider_hash in sorted_hashes:
+ cdepend_hashes += cdepend_provider_hash
+
+ if cdepend_preferred_provider_hash:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS_HASH_%s=%s from single provider: %s" % (d.getVar("PN", True), cdepend, cdepend_preferred_provider_hash, cdepend_preferred_provider))
+ cdepends_hashes[cdepend] = cdepend_preferred_provider_hash
+ else:
+ if len(found_hashes.keys()) == 0:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: No sstate hashes found for provider of: %s" % (d.getVar("PN", True), cdepend))
+ cdepends_hashes[cdepend] = ""
+ elif len(found_hashes.keys()) == 1:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS_HASH_%s=%s from single provider: %s" % (d.getVar("PN", True), cdepend, cdepend_hashes, cdepend_providers))
+ cdepends_hashes[cdepend] = cdepend_hashes
+ else:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS_HASH_%s=%s from group of providers: %s" % (d.getVar("PN", True), cdepend, cdepend_hashes, cdepend_providers))
+ cdepends_hashes[cdepend] = cdepend_hashes
+
+ # Write to sstate cached compatibility reports directory
+ try:
+ bb.utils.mkdirhier(d.getVar("CDEPENDS_STAGEDIR", True))
+ sstate_hash_manifest_file_path = os.path.join(d.getVar("CDEPENDS_STAGEDIR", True), "sstate-hashes")
+ with open(sstate_hash_manifest_file_path, "w") as sstate_hash_manifest_file:
+ for cdepend in cdepends_hashes:
+ sstate_hash_manifest_file.write("%s %s\n" % (cdepend, cdepends_hashes[cdepend]))
+ sstate_hash_manifest_file.close()
+ except (OSError, IOError) as e:
+ bb.warn("cdepends: Could not cache sstate hashes for %s: %s" % (d.getVar("PN", True), str(e)))
+}
+do_cache_compatibility_reports[postfuncs] += "cache_sstate_hashes"
--
2.7.4
From ebc8216d7439fbf60c98a326382687f212dbbeb4 Mon Sep 17 00:00:00 2001
From: Michael Ho <michael...@bmw.de>
Date: Fri, 30 Jun 2017 07:07:27 +0200
Subject: [PATCH 9/9] recipes-cdepends-test: example recipe framework for using
CDEPENDS
To: yocto@yoctoproject.org
Adds required classes, python library, and recipes for testing the
CDEPENDS usage of a parent recipe generating text files and another
recipe that depends on the file checksums of a key file from the
parent.
Recipe cdepends-test1 should always rebuild and generate a new text
file. However, it contains a file called motd2.txt that is taken from
the recipe files path. This does not change as often.
Recipe cdepends-test2 depends on the motd2.txt file that is built in
the parent cdepends-test1. It only needs to recompile when this text
file is changed.
What should be found by running these examples is that after the
sstate cache is populated with a first run, the recipe cdepends-test2
should only rebuild when the file motd2.txt is modified in the recipe
cdepends-test1.
---
.../compatible-depends-report-pkgcontents.bbclass | 10 ++++++
meta-yocto/lib/cdepends_pkgcontents.py | 41 ++++++++++++++++++++++
meta-yocto/recipes-cdepends-test | 1 +
3 files changed, 52 insertions(+)
create mode 100644 meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass
create mode 100644 meta-yocto/lib/cdepends_pkgcontents.py
create mode 160000 meta-yocto/recipes-cdepends-test
diff --git a/meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass b/meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass
new file mode 100644
index 0000000..a060a51
--- /dev/null
+++ b/meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass
@@ -0,0 +1,10 @@
+inherit compatible-depends-report
+
+pkgcontents_compatibility_report () {
+ REPORT_TOOL="pkgcontents"
+ mkdir -p "${COMPATIBILITY_REPORT_WORKDIR}"/"${REPORT_TOOL}"
+ find "${WORKDIR}/packages-split/" -type f -exec sha1sum "{}" \; | tee "${COMPATIBILITY_REPORT_WORKDIR}"/"${REPORT_TOOL}"/manifest
+ sed -i -e "s?${WORKDIR}/packages-split/??" "${COMPATIBILITY_REPORT_WORKDIR}"/"${REPORT_TOOL}"/manifest
+}
+
+do_compatibility_report[postfuncs] += "pkgcontents_compatibility_report"
diff --git a/meta-yocto/lib/cdepends_pkgcontents.py b/meta-yocto/lib/cdepends_pkgcontents.py
new file mode 100644
index 0000000..6af4c84
--- /dev/null
+++ b/meta-yocto/lib/cdepends_pkgcontents.py
@@ -0,0 +1,41 @@
+import os.path
+
+import bb
+
+def compatibility_check(cached_report_dir, new_report_dir, bb_cachedata):
+ if not os.path.isdir(cached_report_dir):
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check cannot find %s" % (cached_report_dir,))
+ return "incompatible"
+
+ cached_manifest_path = os.path.join(cached_report_dir, "pkgcontents", "manifest")
+ if not os.path.isfile(cached_manifest_path):
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check cannot find %s" % (cached_manifest_path,))
+ return "incompatible"
+
+ new_manifest_path = os.path.join(new_report_dir, "pkgcontents", "manifest")
+ if not os.path.isfile(new_manifest_path):
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check cannot find %s" % (new_manifest_path,))
+ return "incompatible"
+
+ match = 1
+ try:
+ with open(cached_manifest_path) as f1, open(new_manifest_path) as f2:
+ for line1 in f1:
+ line_match = 0
+ for line2 in f2:
+ if line1 == line2:
+ line_match = 1
+ break
+ if line_match == 0:
+ match = 0
+ break
+ except (IOError, OSError) as e:
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check could not read manifest file/s: %s" % (str(e),))
+ return "incompatible"
+
+ if match == 0:
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check returning incompatible")
+ return "incompatible"
+ else:
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check returning compatible")
+ return "compatible"
diff --git a/meta-yocto/recipes-cdepends-test b/meta-yocto/recipes-cdepends-test
new file mode 160000
index 0000000..013ad65
--- /dev/null
+++ b/meta-yocto/recipes-cdepends-test
@@ -0,0 +1 @@
+Subproject commit 013ad65d7a884931eab619e9f613760ba209e1cf
--
2.7.4
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
