Re: [OE-core] [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* sstate in esdk
Hi Paul, > -Original Message- > From: Paul Eggleton > Sent: Tuesday, September 17, 2019 3:05 PM > To: Jaewon Lee > Cc: openembedded-core@lists.openembedded.org; Alejandro Enedino > Hernandez Samaniego ; Manjukumar Harthikote > Matha ; Bruce Ashfield > Subject: Re: [OE-core] [oe-core][master][PATCH] Introduce mechanism to > keep nativesdk* sstate in esdk > > Hi Jaewon > > Richard was waiting for me to review this - unfortunately another one that > fell between the cracks - sorry about that. [Jaewon] No problem! > > On Friday, 30 August 2019 5:13:39 AM NZST Jaewon Lee wrote: > > > -Original Message- > > > From: Jaewon Lee > > > Sent: Monday, April 1, 2019 5:07 PM > > > To: openembedded-core@lists.openembedded.org; Alejandro Enedino > > > Hernandez Samaniego ; Manjukumar Harthikote > > > Matha ; Bruce Ashfield > > > Cc: Jaewon Lee > > > Subject: [oe-core][master][PATCH] Introduce mechanism to keep > > > nativesdk* sstate in esdk > > > > > > Using SDK_INCLUDE_NATIVESDK flag to toggle inclusion of all > > > nativesdk* sstate into esdk Currently locked-sigs.inc is generated > > > during do_sdk_depends which doesn't pull in nativesdk packages. > > > Generating another locked-sigs.inc in do_populate_sdk_ext and > > > pruning it to only > > > nativesdk* packages by using a modified version of the already > > > existing function prune_locked_sigs and merging it with the current > > > locked-sigs.inc Also adding SDK_INCLUDE_NATIVESDK tasklistfn to the > > > logic surrounding setting tasklist file to not prune esdk sstate > > > during creation > > > > > > Signed-off-by: Jaewon Lee > > The commit message doesn't actually explain why you are adding this > functionality. You explained it elsewhere (bug 13261) but it needs to be in > here. I would also recommend adding a "Fixes [YOCTO #13261]" at the end so > there's a reference back to the bug as well. [Jaewon] Ok I will resend with commit fixed > > > > @@ -414,7 +433,7 @@ python copy_buildsystem () { > > > > > > sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN') == '1') > > > sdk_ext_type = d.getVar('SDK_EXT_TYPE') > > > -if sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative: > > > +if (sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative) > and not sdk_include_nativesdk: > > > # Create the filtered task list used to generate the sstate cache > shipped with the SDK > > > tasklistfn = d.getVar('WORKDIR') + '/tasklist.txt' > > > create_filtered_tasklist(d, baseoutpath, tasklistfn, > > > conf_initpath) @@ - > > This logic change looks a bit odd. Are you sure this is correct? [Jaewon] I kept the original logic and did a "and not" my flag Because if flag sdk_include_nativesdk is set I want tasklistfn to be none so esdksstate is not pruned during creation Thanks, Jaewon -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* sstate in esdk
Hi Jaewon Richard was waiting for me to review this - unfortunately another one that fell between the cracks - sorry about that. On Friday, 30 August 2019 5:13:39 AM NZST Jaewon Lee wrote: > > -Original Message- > > From: Jaewon Lee > > Sent: Monday, April 1, 2019 5:07 PM > > To: openembedded-core@lists.openembedded.org; Alejandro Enedino > > Hernandez Samaniego ; Manjukumar Harthikote > > Matha ; Bruce Ashfield > > Cc: Jaewon Lee > > Subject: [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* > > sstate in esdk > > > > Using SDK_INCLUDE_NATIVESDK flag to toggle inclusion of all nativesdk* > > sstate into esdk Currently locked-sigs.inc is generated during > > do_sdk_depends which doesn't pull in nativesdk packages. Generating > > another locked-sigs.inc in do_populate_sdk_ext and pruning it to only > > nativesdk* packages by using a modified version of the already existing > > function prune_locked_sigs and merging it with the current locked-sigs.inc > > Also adding SDK_INCLUDE_NATIVESDK tasklistfn to the logic surrounding > > setting tasklist file to not prune esdk sstate during creation > > > > Signed-off-by: Jaewon Lee The commit message doesn't actually explain why you are adding this functionality. You explained it elsewhere (bug 13261) but it needs to be in here. I would also recommend adding a "Fixes [YOCTO #13261]" at the end so there's a reference back to the bug as well. > > @@ -414,7 +433,7 @@ python copy_buildsystem () { > > > > sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN') == '1') > > sdk_ext_type = d.getVar('SDK_EXT_TYPE') > > -if sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative: > > +if (sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative) > > and not sdk_include_nativesdk: > > # Create the filtered task list used to generate the sstate cache > > shipped with the SDK > > tasklistfn = d.getVar('WORKDIR') + '/tasklist.txt' > > create_filtered_tasklist(d, baseoutpath, tasklistfn, > > conf_initpath) @@ - This logic change looks a bit odd. Are you sure this is correct? Thanks Paul -- Paul Eggleton Intel Open Source Technology Centre -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* sstate in esdk
Hi Richard, > -Original Message- > From: Jaewon Lee > Sent: Thursday, August 29, 2019 10:14 AM > To: Jaewon Lee ; openembedded- > c...@lists.openembedded.org; Alejandro Enedino Hernandez Samaniego > ; Manjukumar Harthikote Matha > ; Bruce Ashfield > Subject: RE: [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* > sstate in esdk > > ping Can you please review, this is related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=13261 Thanks, Manju > > > -Original Message- > > From: Jaewon Lee > > Sent: Monday, April 1, 2019 5:07 PM > > To: openembedded-core@lists.openembedded.org; Alejandro Enedino > > Hernandez Samaniego ; Manjukumar Harthikote Matha > > ; Bruce Ashfield > > Cc: Jaewon Lee > > Subject: [oe-core][master][PATCH] Introduce mechanism to keep > > nativesdk* sstate in esdk > > > > Using SDK_INCLUDE_NATIVESDK flag to toggle inclusion of all nativesdk* > > sstate into esdk Currently locked-sigs.inc is generated during > > do_sdk_depends which doesn't pull in nativesdk packages. Generating > > another locked-sigs.inc in do_populate_sdk_ext and pruning it to only > > nativesdk* packages by using a modified version of the already > > existing function prune_locked_sigs and merging it with the current > > locked-sigs.inc Also adding SDK_INCLUDE_NATIVESDK tasklistfn to the > > logic surrounding setting tasklist file to not prune esdk sstate > > during creation > > > > Signed-off-by: Jaewon Lee > > --- > > meta/classes/populate_sdk_ext.bbclass | 28 > > +++- > > meta/lib/oe/copy_buildsystem.py | 8 ++-- > > 2 files changed, 33 insertions(+), 3 deletions(-) > > > > diff --git a/meta/classes/populate_sdk_ext.bbclass > > b/meta/classes/populate_sdk_ext.bbclass > > index 40b0375..d98b0e5 100644 > > --- a/meta/classes/populate_sdk_ext.bbclass > > +++ b/meta/classes/populate_sdk_ext.bbclass > > @@ -20,6 +20,7 @@ SDK_EXT_task-populate-sdk-ext = "-ext" > > SDK_EXT_TYPE ?= "full" > > SDK_INCLUDE_PKGDATA ?= "0" > > SDK_INCLUDE_TOOLCHAIN ?= "${@'1' if d.getVar('SDK_EXT_TYPE') == 'full' > > else '0'}" > > +SDK_INCLUDE_NATIVESDK ?= "0" > > > > SDK_RECRDEP_TASKS ?= "" > > > > @@ -401,9 +402,27 @@ python copy_buildsystem () { > > excluded_targets = get_sdk_install_targets(d, images_only=True) > > sigfile = d.getVar('WORKDIR') + '/locked-sigs.inc' > > lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc' > > +#nativesdk-only sigfile to merge into locked-sigs.inc > > +sdk_include_nativesdk = (d.getVar("SDK_INCLUDE_NATIVESDK") == '1') > > +nativesigfile = d.getVar('WORKDIR') + '/locked-sigs_nativesdk.inc' > > +nativesigfile_pruned = d.getVar('WORKDIR') + '/locked- > > sigs_nativesdk_pruned.inc' > > + > > +if sdk_include_nativesdk: > > +oe.copy_buildsystem.prune_lockedsigs([], > > +excluded_targets.split(), > > +nativesigfile, > > + True, > > + nativesigfile_pruned) > > + > > +oe.copy_buildsystem.merge_lockedsigs([], > > +sigfile, > > +nativesigfile_pruned, > > +sigfile) > > + > > oe.copy_buildsystem.prune_lockedsigs([], > > excluded_targets.split(), > > sigfile, > > + False, > > lockedsigs_pruned) > > > > sstate_out = baseoutpath + '/sstate-cache' > > @@ -414,7 +433,7 @@ python copy_buildsystem () { > > > > sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN') == '1') > > sdk_ext_type = d.getVar('SDK_EXT_TYPE') > > -if sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative: > > +if (sdk_ext_type != 'minimal' or sdk_include_toolchain or > > + derivative) and > > not sdk_include_nativesdk: > > # Create the filtered task list used to generate the sstate > > cache shipped with the SDK > > tasklistfn = d.getVar('WORKDIR') + '/tasklist.txt' > > create_filtered_tasklist(d, baseoutpath, tasklistfn, > > conf_initpath) @@ - > >
Re: [OE-core] [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* sstate in esdk
ping > -Original Message- > From: Jaewon Lee > Sent: Monday, April 1, 2019 5:07 PM > To: openembedded-core@lists.openembedded.org; Alejandro Enedino > Hernandez Samaniego ; Manjukumar Harthikote > Matha ; Bruce Ashfield > Cc: Jaewon Lee > Subject: [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* > sstate in esdk > > Using SDK_INCLUDE_NATIVESDK flag to toggle inclusion of all nativesdk* > sstate into esdk Currently locked-sigs.inc is generated during > do_sdk_depends which doesn't pull in nativesdk packages. Generating > another locked-sigs.inc in do_populate_sdk_ext and pruning it to only > nativesdk* packages by using a modified version of the already existing > function prune_locked_sigs and merging it with the current locked-sigs.inc > Also adding SDK_INCLUDE_NATIVESDK tasklistfn to the logic surrounding > setting tasklist file to not prune esdk sstate during creation > > Signed-off-by: Jaewon Lee > --- > meta/classes/populate_sdk_ext.bbclass | 28 > +++- > meta/lib/oe/copy_buildsystem.py | 8 ++-- > 2 files changed, 33 insertions(+), 3 deletions(-) > > diff --git a/meta/classes/populate_sdk_ext.bbclass > b/meta/classes/populate_sdk_ext.bbclass > index 40b0375..d98b0e5 100644 > --- a/meta/classes/populate_sdk_ext.bbclass > +++ b/meta/classes/populate_sdk_ext.bbclass > @@ -20,6 +20,7 @@ SDK_EXT_task-populate-sdk-ext = "-ext" > SDK_EXT_TYPE ?= "full" > SDK_INCLUDE_PKGDATA ?= "0" > SDK_INCLUDE_TOOLCHAIN ?= "${@'1' if d.getVar('SDK_EXT_TYPE') == 'full' > else '0'}" > +SDK_INCLUDE_NATIVESDK ?= "0" > > SDK_RECRDEP_TASKS ?= "" > > @@ -401,9 +402,27 @@ python copy_buildsystem () { > excluded_targets = get_sdk_install_targets(d, images_only=True) > sigfile = d.getVar('WORKDIR') + '/locked-sigs.inc' > lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc' > +#nativesdk-only sigfile to merge into locked-sigs.inc > +sdk_include_nativesdk = (d.getVar("SDK_INCLUDE_NATIVESDK") == '1') > +nativesigfile = d.getVar('WORKDIR') + '/locked-sigs_nativesdk.inc' > +nativesigfile_pruned = d.getVar('WORKDIR') + '/locked- > sigs_nativesdk_pruned.inc' > + > +if sdk_include_nativesdk: > +oe.copy_buildsystem.prune_lockedsigs([], > + excluded_targets.split(), > + nativesigfile, > + True, > + nativesigfile_pruned) > + > +oe.copy_buildsystem.merge_lockedsigs([], > + sigfile, > + nativesigfile_pruned, > + sigfile) > + > oe.copy_buildsystem.prune_lockedsigs([], > excluded_targets.split(), > sigfile, > + False, > lockedsigs_pruned) > > sstate_out = baseoutpath + '/sstate-cache' > @@ -414,7 +433,7 @@ python copy_buildsystem () { > > sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN') == '1') > sdk_ext_type = d.getVar('SDK_EXT_TYPE') > -if sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative: > +if (sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative) and > not sdk_include_nativesdk: > # Create the filtered task list used to generate the sstate cache > shipped > with the SDK > tasklistfn = d.getVar('WORKDIR') + '/tasklist.txt' > create_filtered_tasklist(d, baseoutpath, tasklistfn, conf_initpath) > @@ - > 658,9 +677,16 @@ fakeroot python do_populate_sdk_ext() { > d.setVar('SDKDEPLOYDIR', '${SDKEXTDEPLOYDIR}') > # ESDKs have a libc from the buildtools so ensure we don't ship linguas > twice > d.delVar('SDKIMAGE_LINGUAS') > +if d.getVar("SDK_INCLUDE_NATIVESDK") == '1': > +generate_nativesdk_lockedsigs(d) > populate_sdk_common(d) > } > > +def generate_nativesdk_lockedsigs(d): > +import oe.copy_buildsystem > +sigfile = d.getVar('WORKDIR') + '/locked-sigs_nativesdk.inc' > +oe.copy_buildsystem.generate_locked_sigs(sigfile, d) > + > def get_ext_sdk_depends(d): > # Note: the deps varflag is a list not a string, so we need to specify > expand=False > deps = d.getVarFlag('do_image_complete', 'deps', False) diff --git > a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py > index 7cb784c..5bc728e 100644 > --- a/meta/lib/oe/copy_
[OE-core] [oe-core][master][PATCH] Introduce mechanism to keep nativesdk* sstate in esdk
Using SDK_INCLUDE_NATIVESDK flag to toggle inclusion of all nativesdk* sstate into esdk Currently locked-sigs.inc is generated during do_sdk_depends which doesn't pull in nativesdk packages. Generating another locked-sigs.inc in do_populate_sdk_ext and pruning it to only nativesdk* packages by using a modified version of the already existing function prune_locked_sigs and merging it with the current locked-sigs.inc Also adding SDK_INCLUDE_NATIVESDK tasklistfn to the logic surrounding setting tasklist file to not prune esdk sstate during creation Signed-off-by: Jaewon Lee --- meta/classes/populate_sdk_ext.bbclass | 28 +++- meta/lib/oe/copy_buildsystem.py | 8 ++-- 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass index 40b0375..d98b0e5 100644 --- a/meta/classes/populate_sdk_ext.bbclass +++ b/meta/classes/populate_sdk_ext.bbclass @@ -20,6 +20,7 @@ SDK_EXT_task-populate-sdk-ext = "-ext" SDK_EXT_TYPE ?= "full" SDK_INCLUDE_PKGDATA ?= "0" SDK_INCLUDE_TOOLCHAIN ?= "${@'1' if d.getVar('SDK_EXT_TYPE') == 'full' else '0'}" +SDK_INCLUDE_NATIVESDK ?= "0" SDK_RECRDEP_TASKS ?= "" @@ -401,9 +402,27 @@ python copy_buildsystem () { excluded_targets = get_sdk_install_targets(d, images_only=True) sigfile = d.getVar('WORKDIR') + '/locked-sigs.inc' lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc' +#nativesdk-only sigfile to merge into locked-sigs.inc +sdk_include_nativesdk = (d.getVar("SDK_INCLUDE_NATIVESDK") == '1') +nativesigfile = d.getVar('WORKDIR') + '/locked-sigs_nativesdk.inc' +nativesigfile_pruned = d.getVar('WORKDIR') + '/locked-sigs_nativesdk_pruned.inc' + +if sdk_include_nativesdk: +oe.copy_buildsystem.prune_lockedsigs([], +excluded_targets.split(), +nativesigfile, + True, + nativesigfile_pruned) + +oe.copy_buildsystem.merge_lockedsigs([], +sigfile, +nativesigfile_pruned, +sigfile) + oe.copy_buildsystem.prune_lockedsigs([], excluded_targets.split(), sigfile, + False, lockedsigs_pruned) sstate_out = baseoutpath + '/sstate-cache' @@ -414,7 +433,7 @@ python copy_buildsystem () { sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN') == '1') sdk_ext_type = d.getVar('SDK_EXT_TYPE') -if sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative: +if (sdk_ext_type != 'minimal' or sdk_include_toolchain or derivative) and not sdk_include_nativesdk: # Create the filtered task list used to generate the sstate cache shipped with the SDK tasklistfn = d.getVar('WORKDIR') + '/tasklist.txt' create_filtered_tasklist(d, baseoutpath, tasklistfn, conf_initpath) @@ -658,9 +677,16 @@ fakeroot python do_populate_sdk_ext() { d.setVar('SDKDEPLOYDIR', '${SDKEXTDEPLOYDIR}') # ESDKs have a libc from the buildtools so ensure we don't ship linguas twice d.delVar('SDKIMAGE_LINGUAS') +if d.getVar("SDK_INCLUDE_NATIVESDK") == '1': +generate_nativesdk_lockedsigs(d) populate_sdk_common(d) } +def generate_nativesdk_lockedsigs(d): +import oe.copy_buildsystem +sigfile = d.getVar('WORKDIR') + '/locked-sigs_nativesdk.inc' +oe.copy_buildsystem.generate_locked_sigs(sigfile, d) + def get_ext_sdk_depends(d): # Note: the deps varflag is a list not a string, so we need to specify expand=False deps = d.getVarFlag('do_image_complete', 'deps', False) diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py index 7cb784c..5bc728e 100644 --- a/meta/lib/oe/copy_buildsystem.py +++ b/meta/lib/oe/copy_buildsystem.py @@ -168,7 +168,7 @@ def generate_locked_sigs(sigfile, d): tasks = ['%s.%s' % (v[2], v[1]) for v in depd.values()] bb.parse.siggen.dump_lockedsigs(sigfile, tasks) -def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, pruned_output): +def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, onlynative, pruned_output): with open(lockedsigs, 'r') as infile: bb.utils.mkdirhier(os.path.dirname(pruned_output)) with open(pruned_output, 'w') as f: @@ -178,7 +178,11 @@ def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, pruned_output if line.endswith('\\\n'): splitval = line.strip().split(':') if not splitval[1] in excluded_tasks and not splitval[0] in excluded_targets: -