Re: [OE-core] [PATCH V3 03/11] ccache.bbclass: Refactor it to make it more reliable
On 1/16/19 6:40 AM, Richard Purdie wrote: On Tue, 2019-01-15 at 10:11 +0800, Robert Yang wrote: Hi Randy, I'm sorry for the types, and thanks for the correction. I found that it has been merged, I will be more careful. I did tweak the commit message! Thank you very much! // Robert Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH V3 03/11] ccache.bbclass: Refactor it to make it more reliable
On Tue, 2019-01-15 at 10:11 +0800, Robert Yang wrote: > Hi Randy, > > I'm sorry for the types, and thanks for the correction. I found that > it has > been merged, I will be more careful. I did tweak the commit message! Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH V3 03/11] ccache.bbclass: Refactor it to make it more reliable
Hi Randy, I'm sorry for the types, and thanks for the correction. I found that it has been merged, I will be more careful. // Robert On 1/12/19 4:49 AM, Randy MacLeod wrote: A couple of typos and grammar improvements, otherwise a good improvement for iterative development of a package/image where sstate isn't useful. On 12/18/18 2:18 AM, Robert Yang wrote: The previous ccache.bbclass has the following problems: - It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native) - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses * host's ccache ccache-native, this may confuse user. - The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is * and be hard to debug in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native. - Target recipes can't use ccache when no ccahe is installed on the host: * no ccache CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}" After refactored: All types recipes (native, target and others) will use ccache-native except ccache-native itself, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built. And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe * disable ccache ../Randy rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear. Signed-off-by: Robert Yang --- meta/classes/ccache.bbclass | 25 ++--- meta/conf/bitbake.conf | 6 +- meta/conf/layer.conf | 1 - meta/lib/oe/utils.py | 3 +++ 4 files changed, 26 insertions(+), 9 deletions(-) diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass index 9609020..59e1022 100644 --- a/meta/classes/ccache.bbclass +++ b/meta/classes/ccache.bbclass @@ -1,4 +1,14 @@ -CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}" +# +# Usage: +# - Enable ccache +# Add the following line to a conffile such as conf/local.conf: +# INHERIT += "ccache" +# +# - Disable ccache for a recipe +# Add the following line to the recipe if it can't be built with ccache: +# CCACHE_DISABLE = '1' +# + export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}" # We need to stop ccache considering the current directory or the @@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}" # ${PV} or ${PR} change. export CCACHE_NOHASHDIR ?= "1" -DEPENDS_append_class-target = " ccache-native" -DEPENDS[vardepvalueexclude] = " ccache-native" +python() { + """ + Enable ccache for the recipe + """ + pn = d.getVar('PN') + # quilt-native doesn't need ccache since no c files + if not (pn in ('ccache-native', 'quilt-native') or + bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))): + d.appendVar('DEPENDS', ' ccache-native') + d.setVar('CCACHE', 'ccache ') +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 6480062..c88d66c 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -492,7 +492,7 @@ HOSTTOOLS += " \ HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}" # Link to these if present -HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo" +HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo" # Temporary add few more detected in bitbake world HOSTTOOLS_NONFATAL += "join nl size yes zcat" @@ -504,10 +504,6 @@ HOSTTOOLS_NONFATAL += "bzr" HOSTTOOLS_NONFATAL += "scp" CCACHE ??= "" -# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and -# autogen sets HOME=/dev/null so in certain situations builds can fail. -# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host. -export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache" TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}" diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf index c67dec4..4141d6c 100644 --- a/meta/conf/layer.conf +++ b/meta/conf/layer.conf @@ -49,7 +49,6 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \ *->quilt-native \ *->subversion-native \ *->git-native \ -
Re: [OE-core] [PATCH V3 03/11] ccache.bbclass: Refactor it to make it more reliable
A couple of typos and grammar improvements, otherwise a good improvement for iterative development of a package/image where sstate isn't useful. On 12/18/18 2:18 AM, Robert Yang wrote: The previous ccache.bbclass has the following problems: - It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native) - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses * host's ccache ccache-native, this may confuse user. - The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is * and be hard to debug in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native. - Target recipes can't use ccache when no ccahe is installed on the host: * no ccache CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}" After refactored: All types recipes (native, target and others) will use ccache-native except ccache-native itself, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built. And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe * disable ccache ../Randy rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear. Signed-off-by: Robert Yang --- meta/classes/ccache.bbclass | 25 ++--- meta/conf/bitbake.conf | 6 +- meta/conf/layer.conf| 1 - meta/lib/oe/utils.py| 3 +++ 4 files changed, 26 insertions(+), 9 deletions(-) diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass index 9609020..59e1022 100644 --- a/meta/classes/ccache.bbclass +++ b/meta/classes/ccache.bbclass @@ -1,4 +1,14 @@ -CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}" +# +# Usage: +# - Enable ccache +# Add the following line to a conffile such as conf/local.conf: +# INHERIT += "ccache" +# +# - Disable ccache for a recipe +# Add the following line to the recipe if it can't be built with ccache: +# CCACHE_DISABLE = '1' +# + export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}" # We need to stop ccache considering the current directory or the @@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}" # ${PV} or ${PR} change. export CCACHE_NOHASHDIR ?= "1" -DEPENDS_append_class-target = " ccache-native" -DEPENDS[vardepvalueexclude] = " ccache-native" +python() { +""" +Enable ccache for the recipe +""" +pn = d.getVar('PN') +# quilt-native doesn't need ccache since no c files +if not (pn in ('ccache-native', 'quilt-native') or +bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))): +d.appendVar('DEPENDS', ' ccache-native') +d.setVar('CCACHE', 'ccache ') +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 6480062..c88d66c 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -492,7 +492,7 @@ HOSTTOOLS += " \ HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}" # Link to these if present -HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo" +HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo" # Temporary add few more detected in bitbake world HOSTTOOLS_NONFATAL += "join nl size yes zcat" @@ -504,10 +504,6 @@ HOSTTOOLS_NONFATAL += "bzr" HOSTTOOLS_NONFATAL += "scp" CCACHE ??= "" -# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and -# autogen sets HOME=/dev/null so in certain situations builds can fail. -# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host. -export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache" TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}" diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf index c67dec4..4141d6c 100644 --- a/meta/conf/layer.conf +++ b/meta/conf/layer.conf @@ -49,7 +49,6 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \ *->quilt-native \ *->subversion-native \ *->git-native \ - *->ccache-native \ *->icecc-create-env-native \ gcc-cross-${TARGET_ARCH}->virtual/${TARGET_PREFIX}libc-for-gcc \
[OE-core] [PATCH V3 03/11] ccache.bbclass: Refactor it to make it more reliable
The previous ccache.bbclass has the following problems: - It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native) - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses ccache-native, this may confuse user. - The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native. - Target recipes can't use ccache when no ccahe is installed on the host: CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}" After refactored: All types recipes (native, target and others) will use ccache-native except ccache-native itself, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built. And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear. Signed-off-by: Robert Yang --- meta/classes/ccache.bbclass | 25 ++--- meta/conf/bitbake.conf | 6 +- meta/conf/layer.conf| 1 - meta/lib/oe/utils.py| 3 +++ 4 files changed, 26 insertions(+), 9 deletions(-) diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass index 9609020..59e1022 100644 --- a/meta/classes/ccache.bbclass +++ b/meta/classes/ccache.bbclass @@ -1,4 +1,14 @@ -CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}" +# +# Usage: +# - Enable ccache +# Add the following line to a conffile such as conf/local.conf: +# INHERIT += "ccache" +# +# - Disable ccache for a recipe +# Add the following line to the recipe if it can't be built with ccache: +# CCACHE_DISABLE = '1' +# + export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}" # We need to stop ccache considering the current directory or the @@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}" # ${PV} or ${PR} change. export CCACHE_NOHASHDIR ?= "1" -DEPENDS_append_class-target = " ccache-native" -DEPENDS[vardepvalueexclude] = " ccache-native" +python() { +""" +Enable ccache for the recipe +""" +pn = d.getVar('PN') +# quilt-native doesn't need ccache since no c files +if not (pn in ('ccache-native', 'quilt-native') or +bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))): +d.appendVar('DEPENDS', ' ccache-native') +d.setVar('CCACHE', 'ccache ') +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 6480062..c88d66c 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -492,7 +492,7 @@ HOSTTOOLS += " \ HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}" # Link to these if present -HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo" +HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo" # Temporary add few more detected in bitbake world HOSTTOOLS_NONFATAL += "join nl size yes zcat" @@ -504,10 +504,6 @@ HOSTTOOLS_NONFATAL += "bzr" HOSTTOOLS_NONFATAL += "scp" CCACHE ??= "" -# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and -# autogen sets HOME=/dev/null so in certain situations builds can fail. -# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host. -export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache" TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}" diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf index c67dec4..4141d6c 100644 --- a/meta/conf/layer.conf +++ b/meta/conf/layer.conf @@ -49,7 +49,6 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \ *->quilt-native \ *->subversion-native \ *->git-native \ - *->ccache-native \ *->icecc-create-env-native \ gcc-cross-${TARGET_ARCH}->virtual/${TARGET_PREFIX}libc-for-gcc \ gcc-cross-${TARGET_ARCH}->linux-libc-headers \ diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index 29b4115..ee6f0e6 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False): return compiler = d.getVar("BUILD_CC") +# Get rid of ccache since it is