from:"David Pierret"

Re: [yocto] [yocto-autobuilder-helper][PATCH v3] AUH: Add Openembedded auto-update-helper with list of layer to test

2023-10-26 Thread David Pierret

Hi,

I've made separate steps for setup and run
I didn't yet modify the python script in upgrade-helper to include
Kevin reviews.

follow some questions I have about the autobuilder.

On Thu, Oct 26, 2023 at 3:04 PM David Pierret via
lists.yoctoproject.org 
wrote:
>
> - Modify the setup-auh to use repositories cloned by the auto-builder.
>   Manage the case where meta-openembedded is not used.
>
> - Modify the run-auh to manage the optional additional layer.
>   Only 1 layer can be provided.
>
> - Add auh-meta-openembedded job with 1 step per layer.
>
> Signed-off-by: David Pierret 
> ---
>  config.json   | 102 --
>  scripts/run-auh   |  39 --
>  scripts/setup-auh |  34 ++--
>  3 files changed, 146 insertions(+), 29 deletions(-)
>
> diff --git a/config.json b/config.json
> index 3acb710..a168e48 100644
> --- a/config.json
> +++ b/config.json
> @@ -1416,9 +1416,99 @@
>  "TEMPLATE" : "buildperf"
>  },
>  "auh" : {
> -"EXTRAPLAINCMDS" : [
> -"${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; 
> ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
> -]
> +"NEEDREPOS" : ["poky", "auto-upgrade-helper"],
> +"step1" : {
> +"shortname" : "setup AUH",
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}"
> +]
> +},
> +"step2" : {
> +"shortname" : "run AUH poky",
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
> ${WEBPUBLISH_DIR}/pub/auh/"
> +]
> +}
> +},
> +"auh-meta-openembedded" : {
> +"NEEDREPOS" : ["poky", "meta-openembedded", 
> "auto-upgrade-helper"],
> +"step1" : {
> +"shortname" : "setup AUH",
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}"
> +]
> +},
> +"step2" : {
> +"shortname" : "run AUH for layer meta-oe",
> +
> +"ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-oe"],
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
> ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-oe"
> +]
> +},
> +"step3" : {
> +"shortname" : "run AUH for layer meta-python",
> +"ADDLAYER" : 
> ["${BUILDDIR}/../meta-openembedded/meta-python"],
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
> ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-python"
> +]
> +},
> +"step4" : {
> +"shortname" : "run AUH for layer meta-perl",
> +"ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-perl"],
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
> ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-perl"
> +]
> +},
> +"step5" : {
> +"shortname" : "run AUH for layer meta-networking",
> +"ADDLAYER" : 
> ["${BUILDDIR}/../meta-openembedded/meta-networking"],
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
> ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded 
> meta-networking"
> +]
> +},
> +"step6" : {
> +"shortname" : "run AUH for layer meta-multimedia",
> +"ADDLAYER" : 
> ["${BUILDDIR}/../meta-openembedded/meta-multimedia"],
> +"EXTRAPLAINCMDS" : [
> +"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
> ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded 
> meta-multimedia"
&

[yocto] [yocto-autobuilder-helper][PATCH v3] AUH: Add Openembedded auto-update-helper with list of layer to test

2023-10-26 Thread David Pierret

- Modify the setup-auh to use repositories cloned by the auto-builder.
  Manage the case where meta-openembedded is not used.

- Modify the run-auh to manage the optional additional layer.
  Only 1 layer can be provided.

- Add auh-meta-openembedded job with 1 step per layer.

Signed-off-by: David Pierret 
---
 config.json   | 102 --
 scripts/run-auh   |  39 --
 scripts/setup-auh |  34 ++--
 3 files changed, 146 insertions(+), 29 deletions(-)

diff --git a/config.json b/config.json
index 3acb710..a168e48 100644
--- a/config.json
+++ b/config.json
@@ -1416,9 +1416,99 @@
 "TEMPLATE" : "buildperf"
 },
 "auh" : {
-"EXTRAPLAINCMDS" : [
-"${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; 
${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
-]
+"NEEDREPOS" : ["poky", "auto-upgrade-helper"],
+"step1" : {
+"shortname" : "setup AUH",
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}"
+]
+},
+"step2" : {
+"shortname" : "run AUH poky",
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
${WEBPUBLISH_DIR}/pub/auh/"
+]
+}
+},
+"auh-meta-openembedded" : {
+"NEEDREPOS" : ["poky", "meta-openembedded", "auto-upgrade-helper"],
+"step1" : {
+"shortname" : "setup AUH",
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}"
+]
+},
+"step2" : {
+"shortname" : "run AUH for layer meta-oe",
+
+"ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-oe"],
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-oe"
+]
+},
+"step3" : {
+"shortname" : "run AUH for layer meta-python",
+"ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-python"],
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-python"
+]
+},
+"step4" : {
+"shortname" : "run AUH for layer meta-perl",
+"ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-perl"],
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-perl"
+]
+},
+"step5" : {
+"shortname" : "run AUH for layer meta-networking",
+"ADDLAYER" : 
["${BUILDDIR}/../meta-openembedded/meta-networking"],
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-networking"
+]
+},
+"step6" : {
+"shortname" : "run AUH for layer meta-multimedia",
+"ADDLAYER" : 
["${BUILDDIR}/../meta-openembedded/meta-multimedia"],
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-multimedia"
+]
+},
+"step7" : {
+"shortname" : "run AUH for layer meta-gnome",
+"ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-gnome"],
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} 
${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-gnome"
+]
+},
+"step8" : {
+"shortname" : "run AUH for layer meta-xfce",
+"ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-xfce"],
+"EXTRAPL

Re: [yocto] [yocto-autobuilder-helper][PATCH v2] AUH: Add Openembedded auto-update-helper with list of layer to test

2023-10-24 Thread David Pierret

Hi Richard,

Thanks for the review, please find my feedback in follow.

On Fri, Oct 20, 2023 at 4:57 PM Richard Purdie
 wrote:
>
> Hi David,
>
> Thanks for revising this, it is heading in the right direction.
>
> On Fri, 2023-10-20 at 16:05 +0200, David Pierret wrote:
> > - Add setup and run script openembedded specific
> > - Add job config
> >
> > Signed-off-by: David Pierret 
> > ---
> >  config.json  |  6 ++
> >  scripts/run-auh-oe   | 45 
> >  scripts/setup-auh-oe | 34 +
> >  3 files changed, 85 insertions(+)
> >  create mode 100755 scripts/run-auh-oe
> >  create mode 100755 scripts/setup-auh-oe
> >
> > diff --git a/config.json b/config.json
> > index 3acb710..bbd0aaf 100644
> > --- a/config.json
> > +++ b/config.json
> > @@ -1420,6 +1420,12 @@
> >  "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; 
> > ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
> >  ]
> >  },
> > +"auh-openembedded" : {
>
> This needs to be "auh-meta-oe" since "auh" is for openembedded-core and
> this is otherwise going to be confusing.
>
> > +"NEEDREPOS" : ["poky", "meta-openembedded"],
> > +"EXTRAPLAINCMDS" : [
> > +"${SCRIPTSDIR}/setup-auh-oe ${HELPERBUILDDIR}; 
> > ${SCRIPTSDIR}/run-auh-oe ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ 
> > ${HELPERBUILDDIR}/meta-openembedded 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-oe 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-python 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-perl 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-networking 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-multimedia 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-gnome 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-xfce 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-filesystems 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-initramfs 
> > ${HELPERBUILDDIR}/meta-openembedded/meta-webserver "
> > +]
> > +},
> >  "a-quick" : {
> >  "TEMPLATE" : "trigger-build"
> >  },
>
> Would it be better to have one setup-auh/run-auh script and add
> ${HELPERTARGET}  as a parameter?
>
> This should translate to "auh" or "auh-meta-oe" and the script can then
> have conditionals in to handle both cases rather than duplicating the
> script?

The setup-auh script is cloning its own repository instead of using
the one initialized by CI.
Any reason to not use them ? We could use the NEEDREPOS instead of the
setup script,
and add the auto-upgrade-helper repository in the config.

>
> > diff --git a/scripts/run-auh-oe b/scripts/run-auh-oe
> > new file mode 100755
> > index 000..588755a
> > --- /dev/null
> > +++ b/scripts/run-auh-oe
>
> Similarly, this should be run-auh-meta-oe.
>
> > @@ -0,0 +1,45 @@
> > +#!/bin/bash
> > +#
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# Run Auto Upgrade Helper in a directory set up by setup_auh.
> > +#
> > +# Called with $1 - the directory where the setup was created
> > +
> > +if [ -z $1 ]; then
> > +  echo "Use: $0 [auh_setup_dir] [publish_dir] [meta_dir] [meta_list]"
> > +  exit 1
> > +fi
> > +
> > +full_dir=$(readlink -e $1)
> > +
> > +auh_dir=$full_dir/auto-upgrade-helper
> > +poky_dir=$full_dir/poky
> > +
> > +build_dir=$full_dir/build
> > +sstate_dir=$full_dir/build/sstate-cache
> > +meta_dir=$3
> > +meta_list=${@:4}
> > +machine_list="qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl"
>
> Do we need to vary the machine_list per layer? At present it only seems
> to support one list anyway so I'm not sure this does anything useful?

The list is the same as the one used for poky.

>
> > +pushd $meta_dir || exit 1
> > +
> > +# Base the upgrades on meta_openembedded master
> > +git fetch origin
> > +git checkout -B tmp-auh-upgrades origin/main
> > +
> > +source $poky_dir/oe-init-build-env $build_dir
> > +
> > +# build the layer_names variable to me used in the command line
> > +layer_names=""
> > +for d in $meta_list; do
> > +  layer_names+=" $(basename ${d})"
> > +done
> > +
> > +$auh_dir/upgrade-helper.py -e --layer-dir ${meta_dir} --layer-names 
> > ${layer_names} --layer-machines

[yocto] [yocto-autobuilder-helper][PATCH v2] AUH: Add Openembedded auto-update-helper with list of layer to test

2023-10-20 Thread David Pierret

- Add setup and run script openembedded specific
- Add job config

Signed-off-by: David Pierret 
---
 config.json  |  6 ++
 scripts/run-auh-oe   | 45 
 scripts/setup-auh-oe | 34 +
 3 files changed, 85 insertions(+)
 create mode 100755 scripts/run-auh-oe
 create mode 100755 scripts/setup-auh-oe

diff --git a/config.json b/config.json
index 3acb710..bbd0aaf 100644
--- a/config.json
+++ b/config.json
@@ -1420,6 +1420,12 @@
 "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; 
${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
 ]
 },
+"auh-openembedded" : {
+"NEEDREPOS" : ["poky", "meta-openembedded"],
+"EXTRAPLAINCMDS" : [
+"${SCRIPTSDIR}/setup-auh-oe ${HELPERBUILDDIR}; 
${SCRIPTSDIR}/run-auh-oe ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ 
${HELPERBUILDDIR}/meta-openembedded ${HELPERBUILDDIR}/meta-openembedded/meta-oe 
${HELPERBUILDDIR}/meta-openembedded/meta-python 
${HELPERBUILDDIR}/meta-openembedded/meta-perl 
${HELPERBUILDDIR}/meta-openembedded/meta-networking 
${HELPERBUILDDIR}/meta-openembedded/meta-multimedia 
${HELPERBUILDDIR}/meta-openembedded/meta-gnome 
${HELPERBUILDDIR}/meta-openembedded/meta-xfce 
${HELPERBUILDDIR}/meta-openembedded/meta-filesystems 
${HELPERBUILDDIR}/meta-openembedded/meta-initramfs 
${HELPERBUILDDIR}/meta-openembedded/meta-webserver "
+]
+},
 "a-quick" : {
 "TEMPLATE" : "trigger-build"
 },
diff --git a/scripts/run-auh-oe b/scripts/run-auh-oe
new file mode 100755
index 000..588755a
--- /dev/null
+++ b/scripts/run-auh-oe
@@ -0,0 +1,45 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Run Auto Upgrade Helper in a directory set up by setup_auh.
+#
+# Called with $1 - the directory where the setup was created
+
+if [ -z $1 ]; then
+  echo "Use: $0 [auh_setup_dir] [publish_dir] [meta_dir] [meta_list]"
+  exit 1
+fi
+
+full_dir=$(readlink -e $1)
+
+auh_dir=$full_dir/auto-upgrade-helper
+poky_dir=$full_dir/poky
+
+build_dir=$full_dir/build
+sstate_dir=$full_dir/build/sstate-cache
+meta_dir=$3
+meta_list=${@:4}
+machine_list="qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl"
+
+pushd $meta_dir || exit 1
+
+# Base the upgrades on meta_openembedded master
+git fetch origin
+git checkout -B tmp-auh-upgrades origin/main
+
+source $poky_dir/oe-init-build-env $build_dir
+
+# build the layer_names variable to me used in the command line
+layer_names=""
+for d in $meta_list; do
+  layer_names+=" $(basename ${d})"
+done
+
+$auh_dir/upgrade-helper.py -e --layer-dir ${meta_dir} --layer-names 
${layer_names} --layer-machines ${machine_list} -- all
+
+if [ -n $2 ]; then
+  cp -rf $build_dir/upgrade-helper/* $2
+fi
+
+popd
diff --git a/scripts/setup-auh-oe b/scripts/setup-auh-oe
new file mode 100755
index 000..ba1a7fb
--- /dev/null
+++ b/scripts/setup-auh-oe
@@ -0,0 +1,34 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Initialize Auto Upgrade Helper in a directory.
+#
+# Called with $1 - the directory to place the setup
+CONFIG_DIR=`dirname $0`/auh-config
+
+if [ -z $1 ]; then
+  echo "Use: $0 target_dir"
+  exit 1
+fi
+
+mkdir -p $1
+pushd $1
+
+git clone git://git.yoctoproject.org/poky
+pushd poky
+git config user.email a...@yoctoproject.org
+git config user.name "Auto Upgrade Helper"
+popd
+git clone git://git.openembedded.org/meta-openembedded
+pushd meta-openembedded
+git config user.email a...@yoctoproject.org
+git config user.name "Auto Upgrade Helper"
+popd
+git clone git://git.yoctoproject.org/auto-upgrade-helper
+source poky/oe-init-build-env build
+mkdir -p upgrade-helper
+popd
+
+cp $CONFIG_DIR/upgrade-helper.conf $1/build/upgrade-helper
+cat $CONFIG_DIR/local.conf.append >> $1/build/conf/local.conf
-- 
2.39.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61436): https://lists.yoctoproject.org/g/yocto/message/61436
Mute This Topic: https://lists.yoctoproject.org/mt/102081665/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[yocto] [auh][PATCH v2] upgrade-helper.py: Add layer command line option

2023-10-20 Thread David Pierret

- Layer options can be defined as command line arguments
- multiple layers can be defined in a single command

each layer is analyzed 1 by 1 to check if upgrades are available.

Signed-off-by: David Pierret 
Reviewed-by: Yoann Congal 
---
 upgrade-helper.py | 75 +--
 weeklyjob-oe.sh   | 55 ++
 2 files changed, 101 insertions(+), 29 deletions(-)
 create mode 100755 weeklyjob-oe.sh

diff --git a/upgrade-helper.py b/upgrade-helper.py
index 4f0b6b7..5093ff9 100755
--- a/upgrade-helper.py
+++ b/upgrade-helper.py
@@ -105,6 +105,12 @@ def parse_cmdline():
 help="do not compile, just change the checksums, 
remove PR, and commit")
 parser.add_argument("-c", "--config-file", default=None,
 help="Path to the configuration file. Default is 
$BUILDDIR/upgrade-helper/upgrade-helper.conf")
+parser.add_argument("-l", "--layer-names", nargs='*', action="store", 
default='',
+help="layers to include in the upgrade research")
+parser.add_argument("-L", "--layer-dir", action="store", default='',
+help="the layers root directory")
+parser.add_argument("-m", "--layer-machines", nargs='*', action="store", 
default='',
+help="machine to build for the layers")
 return parser.parse_args()
 
 def parse_config_file(config_file):
@@ -440,8 +446,8 @@ class Updater(object):
 pkgs_ctx = {}
 
 I(" ### The list of recipes to be upgraded #")
-for p, ov, nv, m, r in pkgs_to_upgrade:
-I(" %s, %s, %s, %s, %s" % (p, ov, nv, m, r))
+for l, p, ov, nv, m, r in pkgs_to_upgrade:
+I(" %s, %s, %s, %s, %s, %s" % (l, p, ov, nv, m, r))
 
 pkgs_ctx[p] = {}
 pkgs_ctx[p]['PN'] = p
@@ -471,7 +477,7 @@ class Updater(object):
 succeeded_pkgs_ctx = []
 failed_pkgs_ctx = []
 attempted_pkgs = 0
-for pn, _, _, _, _ in pkgs_to_upgrade:
+for l, pn, _, _, _, _ in pkgs_to_upgrade:
 pkg_ctx = pkgs_ctx[pn]
 pkg_ctx['error'] = None
 
@@ -577,21 +583,24 @@ class UniverseUpdater(Updater):
 if len(args.recipe) == 1 and args.recipe[0] == "all":
 self.recipes = []
 else:
-self.recipes = args.recipe
+self.recipes = ('', args.recipe)
 
 # to filter recipes in upgrade
 if not self.recipes and self.opts['layer_mode'] == 'yes':
 # when layer mode is enabled and no recipes are specified
 # we need to figure out what recipes are provided by the
 # layer to try upgrade
-self.recipes = self._get_recipes_by_layer()
+for layer in self.opts['layer_name'].split(' '):
+recipes_for_layer = self._get_recipes_by_layer(layer)
+I("layer %s => Receipes %s", layer, recipes_for_layer)
+self.recipes.append((layer, recipes_for_layer))
 
 if args.to_version:
 if len(self.recipes) != 1:
 E(" -t is only supported when upgrade one recipe\n")
 exit(1)
 
-def _get_recipes_by_layer(self):
+def _get_recipes_by_layer(self, layer=''):
 recipes = []
 
 recipe_regex = re.compile('^(?P.*):$')
@@ -611,7 +620,7 @@ class UniverseUpdater(Updater):
 if not 'skipped' in line:
 s = layer_regex.search(line)
 if s:
-if s.group('name').strip() == self.opts['layer_name']:
+if s.group('name').strip() == layer:
 recipes.append(name)
 
 return recipes
@@ -662,31 +671,31 @@ class UniverseUpdater(Updater):
 return True
 
 def _get_packages_to_upgrade(self, packages=None):
+pkgs_list = []
+for layer_name, layer_recipes in self.recipes:
+pkgs = oe.recipeutils.get_recipe_upgrade_status(layer_recipes)
 
-pkgs = oe.recipeutils.get_recipe_upgrade_status(self.recipes)
+for pkg in pkgs:
+pn, status, cur_ver, next_ver, maintainer, revision, 
no_upgrade_reason = pkg
 
-pkgs_list = []
-for pkg in pkgs:
-pn, status, cur_ver, next_ver, maintainer, revision, 
no_upgrade_reason = pkg
-
-if self.args.to_version:
- next_ver = self.args.to_version
-
-if status == 'UPDATE' and not no_upgrade_reason:
-# Always do the upgrade if recipes are specified
-if self.recipes and pn in self.recipes:
-pkgs_list.append((pn, cur_ver, next_ver, maintainer, 
revision))
-elif self._pkg_upgradable(p

Re: [yocto] [yocto-autobuilder-helper][PATCH v2] meta-oe-mirror: Use a 2 step job to fetch and verify meta-openembedded mirror.

2023-10-18 Thread David Pierret

Hi,
This patch is v2 for this one :
https://lists.yoctoproject.org/g/yocto/message/61306

Regards
David

On Wed, Oct 18, 2023 at 3:46 PM David Pierret via
lists.yoctoproject.org 
wrote:
>
> Inspired from trigger-build and trigger-build-post-trigger
> The branch must be selected on build configuration.
>
> Signed-off-by: David Pierret 
> Reviewed-by: Yoann Congal 
> ---
>  config.json | 30 ++
>  1 file changed, 30 insertions(+)
>
> diff --git a/config.json b/config.json
> index 3acb710..9e6779f 100644
> --- a/config.json
> +++ b/config.json
> @@ -1420,6 +1420,36 @@
>  "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; 
> ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
>  ]
>  },
> +"meta-oe-mirror" : {
> +"SDKMACHINE" : "x86_64",
> +"MACHINE" : "qemux86-64",
> +"NEEDREPOS" : ["poky", "meta-openembedded"],
> +"ADDLAYER" : [
> +"${BUILDDIR}/../meta-selftest",
> +
> +"${BUILDDIR}/../meta-openembedded/meta-oe",
> +"${BUILDDIR}/../meta-openembedded/meta-python",
> +"${BUILDDIR}/../meta-openembedded/meta-perl",
> +"${BUILDDIR}/../meta-openembedded/meta-networking",
> +"${BUILDDIR}/../meta-openembedded/meta-multimedia",
> +"${BUILDDIR}/../meta-openembedded/meta-gnome",
> +"${BUILDDIR}/../meta-openembedded/meta-xfce",
> +"${BUILDDIR}/../meta-openembedded/meta-filesystems",
> +"${BUILDDIR}/../meta-openembedded/meta-initramfs",
> +"${BUILDDIR}/../meta-openembedded/meta-webserver"
> +],
> +"step1" : {
> +"shortname" : "Sources pre-fetching",
> +"BBTARGETS" : "universe -c fetch -k",
> +"extravars" : [
> +"SOURCE_MIRROR_FETCH = '1'"
> +]
> +},
> +"step2" : {
> +"shortname" : "Source Mirror Selftest",
> +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 
> oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror"],
> +}
> +},
>  "a-quick" : {
>  "TEMPLATE" : "trigger-build"
>  },
> --
> 2.39.2
>
>
> 
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61401): https://lists.yoctoproject.org/g/yocto/message/61401
Mute This Topic: https://lists.yoctoproject.org/mt/102039136/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[yocto] [yocto-autobuilder-helper][PATCH v2] meta-oe-mirror: Use a 2 step job to fetch and verify meta-openembedded mirror.

2023-10-18 Thread David Pierret

Inspired from trigger-build and trigger-build-post-trigger
The branch must be selected on build configuration.

Signed-off-by: David Pierret 
Reviewed-by: Yoann Congal 
---
 config.json | 30 ++
 1 file changed, 30 insertions(+)

diff --git a/config.json b/config.json
index 3acb710..9e6779f 100644
--- a/config.json
+++ b/config.json
@@ -1420,6 +1420,36 @@
 "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; 
${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
 ]
 },
+"meta-oe-mirror" : {
+"SDKMACHINE" : "x86_64",
+"MACHINE" : "qemux86-64",
+"NEEDREPOS" : ["poky", "meta-openembedded"],
+"ADDLAYER" : [
+"${BUILDDIR}/../meta-selftest",
+
+"${BUILDDIR}/../meta-openembedded/meta-oe",
+"${BUILDDIR}/../meta-openembedded/meta-python",
+"${BUILDDIR}/../meta-openembedded/meta-perl",
+"${BUILDDIR}/../meta-openembedded/meta-networking",
+"${BUILDDIR}/../meta-openembedded/meta-multimedia",
+"${BUILDDIR}/../meta-openembedded/meta-gnome",
+"${BUILDDIR}/../meta-openembedded/meta-xfce",
+"${BUILDDIR}/../meta-openembedded/meta-filesystems",
+"${BUILDDIR}/../meta-openembedded/meta-initramfs",
+"${BUILDDIR}/../meta-openembedded/meta-webserver"
+],
+"step1" : {
+"shortname" : "Sources pre-fetching",
+"BBTARGETS" : "universe -c fetch -k",
+"extravars" : [
+"SOURCE_MIRROR_FETCH = '1'"
+]
+},
+"step2" : {
+"shortname" : "Source Mirror Selftest",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 
oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror"],
+}
+},
 "a-quick" : {
 "TEMPLATE" : "trigger-build"
 },
-- 
2.39.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61400): https://lists.yoctoproject.org/g/yocto/message/61400
Mute This Topic: https://lists.yoctoproject.org/mt/102039136/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[yocto] [auh][PATCH] weeklyjob: Add Openembedded update-helper conf

2023-10-10 Thread David Pierret

- Add weeklyjob for openembedded meta
- Add upgrade-helper.in config template

auto-upgrade-helper script accept configurations to specify the layer
to be analyzed.
The .in file is modified by sed command to generate the .conf file
that match the meta to analyze.

We've chosen to use a template configuration file rather than multiple
configuration files (1 per meta). The template file contains tags
between braces that is replaced in the script using simple `sed` calls.
The meta list to be analyzed is defined via the `meta_list` variable.
This variable matches all `meta-*` directories

Signed-off-by: David Pierret 
Reviewed-by: Yoann Congal 
---
 upgrade-helper.in | 124 ++
 weeklyjob-oe.sh   |  48 ++
 2 files changed, 172 insertions(+)
 create mode 100644 upgrade-helper.in
 create mode 100644 weeklyjob-oe.sh

diff --git a/upgrade-helper.in b/upgrade-helper.in
new file mode 100644
index 000..4eedef1
--- /dev/null
+++ b/upgrade-helper.in
@@ -0,0 +1,124 @@
+#
+# This file is your upgrade-helper configuration files and is where all local 
user
+# settings specifically for AUH are placed. The comments in this file provide 
some
+# guidelines to the options a new user to AUH tool might want to change.
+#
+# This file should be placed inside your BUILDDIR/upgrade-helper after 
initializing
+# the OE build environment (this location can be overriden with -c command 
line option).
+#
+# Lines starting with the '#' character are commented out and in some cases the
+# default values are provided as comments to show people example syntax. 
Enabling
+# the option is a question of removing the # character and making any change 
to the
+# variable as required. For boolean settings, anything not 'yes' is treated as 
'no'.
+
+[maintainer_override]
+# e-mail message for recipe upgrades will go to john.doe instead of jane.doe, 
etc
+# see also the global_maintainer_override option
+#jane@doe.com=john@doe.com
+#johhny.br...@bravo.com=john@doe.com
+
+[settings]
+# SMTP server that is used to send e-mails with patches and other information.
+# If you are running AUH locally, you do not need to set this up, as AUH
+# saves everything to BUILDDIR/upgrade-helper/, and does not attempt
+# to send email messages (unless explicitly asked with -e command line option).
+# If no port is specified, port 25 is assumed.
+#smtp=smtp.my-server.com:25
+
+# from whom should the e-mails be sent (mandatory if --send-emails is passed).
+# Also sets the email address of the author of automated commits.
+#from=u...@not.set
+
+# If enabled, emails for all recipe upgrades will go to john.doe,
+# except when recipes are owned by specific maintainer_override entries above.
+#global_maintainer_override=john@doe.com
+
+# who should be CCd with all upgrade emails (optional)
+#cc_recipients=john@doe.com
+
+# who should get the status mail with statistics, at the end (mandatory if
+# --send-emails is passed)
+#status_recipients=john@doe.com
+
+# Only recipes belonging to maintainers in whitelist will be attempted
+# Insert your own e-mail address here together with any other maintainers' 
addresses
+# whose recipes you wish to take care of. This restricts the set of recipes 
that AUH
+# will attempt when it is run with 'all' option.
+#maintainers_whitelist=jane@doe.com john@doe.com johhny.br...@bravo.com
+
+# recipes in blacklist will be skipped (applies only when 'all' or no recipe is
+# passed; does not apply when layer_mode is enabled).
+#blacklist=python glibc gcc
+
+# specify the directory where work (patches) will be saved
+# (optional; default is BUILDDIR/upgrade-helper/)
+#workdir=
+
+# public url with AUH results to include in statistics summary (optional)
+#publish_work_url=http://auh.somehost.com/work
+
+# whether to attach a tarball with AUH artefacts (patches, log, buildhistory, 
etc.)
+# to the summary email
+#summary_includes_tarball=True
+
+# clean sstate directory before upgrading
+# Generally not necessary, as bitbake can handle this automatically.
+#clean_sstate=no
+
+# clean tmp directory before upgrading
+# Generally not necessary as bitbake can handle this automatically.
+#clean_tmp=no
+
+# Machines to test build with.
+# Append _libc-name to test with alternative C library implementations
+# e.g. qemux86_musl.
+#
+# Buildhistory and testimages will be created only for the first
+# machine in the list, as otherwise it adds enormously to AUH run time.
+#
+# AUH has a reasonable default for this, so you do not need to set your own,
+# at least initially.
+#
+# Does not apply when layer_mode is enabled.
+#machines=qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl
+
+# Enables buildhistory feature; this is useful as it produces information
+# about what has changed in the resulting packages, compared to previous 
version
+#
+# Requires 'buildhistory' to be present in INHERIT and BUILDHISTORY_COMMIT to 
be set
+# in your conf/local.conf

[yocto] [yocto-autobuilder-helper][PATCH] oe-mirror: Add job to mirror openembedded sources

2023-10-10 Thread David Pierret

- Add a job that call run-fetchuniverse
- Add script run-fetchuniverse
   - loop over each LTS branches
   - Download all the meta-openembedded layers sources.
   - Call to oe-selftest test_yocto_source_mirror after fetching.
We are not using a second step to keep checkout consistency with potential 
merge.

Signed-off-by: David Pierret 
Reviewed-by: Yoann Congal 
---
 config.json   | 13 +++
 scripts/run-fetchuniverse | 49 +++
 2 files changed, 62 insertions(+)
 create mode 100755 scripts/run-fetchuniverse

diff --git a/config.json b/config.json
index 3acb710..8d59be8 100644
--- a/config.json
+++ b/config.json
@@ -1420,6 +1420,19 @@
 "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; 
${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
 ]
 },
+"oe-mirror" : {
+"SDKMACHINE" : "x86_64",
+"MACHINE" : "qemux86-64",
+"NEEDREPOS" : ["poky", "meta-openembedded"],
+
+"step1" : {
+"shortname" : "Sources pre-fetching",
+"EXTRAPLAINCMDS" : ["${SCRIPTSDIR}/run-fetchuniverse 
${BUILDDIR} ${SCRIPTSDIR}"],
+"extravars" : [
+"SOURCE_MIRROR_FETCH = '1'"
+]
+}
+},
 "a-quick" : {
 "TEMPLATE" : "trigger-build"
 },
diff --git a/scripts/run-fetchuniverse b/scripts/run-fetchuniverse
new file mode 100755
index 000..9ef82de
--- /dev/null
+++ b/scripts/run-fetchuniverse
@@ -0,0 +1,49 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+BUILDDIR=`realpath $1`
+SCRIPTSDIR=`realpath $2`
+ROOTDIR=$BUILDDIR/..
+
+#
+# Fetch mirror for each LTS branch and test integrity
+#
+for branch in master mickledore langdale kirkstone dunfell; do
+  # For each layer
+  for repo in bitbake poky meta-openembedded ; do
+pushd $ROOTDIR/$repo || exit 1
+  # reset branch
+  git reset origin/$branch --hard
+popd || exit 1
+  done
+  # remove config to be re-generated by oe-init-build-env
+  rm conf/local.conf
+  rm conf/bblayers.conf
+  rm -f conf/templateconf.cfg
+  rm tmp/ -rf
+
+  # got back to ROOTDIR
+  pushd $ROOTDIR || exit 1
+  # generate config for current branch
+  . ./poky/oe-init-build-env build
+
+  # We need to add the meta-openembedded layers before fetch
+  bitbake-layers add-layer ${BUILDDIR}/../meta-openembedded/meta-filesystems 
${BUILDDIR}/../meta-openembedded/meta-gnome \
+   ${BUILDDIR}/../meta-openembedded/meta-initramfs 
${BUILDDIR}/../meta-openembedded/meta-multimedia \
+   ${BUILDDIR}/../meta-openembedded/meta-networking 
${BUILDDIR}/../meta-openembedded/meta-oe \
+   ${BUILDDIR}/../meta-openembedded/meta-perl 
${BUILDDIR}/../meta-openembedded/meta-python \
+   ${BUILDDIR}/../meta-openembedded/meta-webserver 
${BUILDDIR}/../meta-openembedded/meta-xfce
+  # call bitbake to fetch universe
+  bitbake universe -c fetch -k
+
+  # call oe-selftest to execute test_yocto_source_mirror
+  # We need the selftest layer
+  bitbake-layers add-layer $BUILDDIR/../meta-selftest
+  # ensure no previous build-st directory exist
+  rm -rf ${BUILDDIR}/../build-st
+  ${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -r 
buildoptions.SourceMirroring.test_yocto_source_mirror || exit 1
+  popd || exit 1
+done
+
+
-- 
2.39.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61306): https://lists.yoctoproject.org/g/yocto/message/61306
Mute This Topic: https://lists.yoctoproject.org/mt/101876565/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[yocto] [yocto-autobuilder-helper][PATCH] AUH: Add Openembedded auto-update-helper conf and run

2023-10-10 Thread David Pierret

- Add setup and run script openembedded specific
- Add upgrade-helper.in config template

auto-upgrade-helper script accept configurations to specify the layer
to be analysed. The .in file is modified by sed command to generate
the .conf file that match the meta to analyze.

Signed-off-by: David Pierret 
Reviewed-by: Yoann Congal 
---
 scripts/auh-config/upgrade-helper-oe.in | 40 +
 scripts/run-auh-oe  | 46 +
 scripts/setup-auh-oe| 34 ++
 3 files changed, 120 insertions(+)
 create mode 100644 scripts/auh-config/upgrade-helper-oe.in
 create mode 100755 scripts/run-auh-oe
 create mode 100755 scripts/setup-auh-oe

diff --git a/scripts/auh-config/upgrade-helper-oe.in 
b/scripts/auh-config/upgrade-helper-oe.in
new file mode 100644
index 000..79467e5
--- /dev/null
+++ b/scripts/auh-config/upgrade-helper-oe.in
@@ -0,0 +1,40 @@
+[maintainer_override]
+# mails for recipe upgrades will go to john.doe instead of jane.doe, etc
+#ross.bur...@intel.com=anibal.li...@linux.intel.com
+
+[settings]
+# recipes in blacklist will be skipped
+blacklist=linux-libc-headers linux-yocto alsa-utils-scripts 
build-appliance-image
+#blacklist=python python3 glibc gcc linux-libc-headers linux-yocto-rt 
linux-yocto linux-yocto-dev linux-yocto-tiny qt4-x11-free qt4-embedded 
qt4-x11-free qt4e-demo-image gnome-common gnome-desktop3 gnome-desktop-testing 
adt-installer build-appliance-image
+# only recipes belonging to maintainers in whitelist will be attempted
+#maintainers_whitelist=anibal.li...@linux.intel.com
+# SMTP server
+smtp=localhost:25
+# from whom should the mails arrive
+from=a...@yoctoproject.org
+# who should get the status mail with statistics, at the end
+status_recipients=openembedded-c...@lists.openembedded.org
+# who should be CCd with upgrade emails
+cc_recipients=openembedded-c...@lists.openembedded.org
+# clean sstate directory before upgrading
+#clean_sstate=yes
+# clean tmp directory before upgrading
+#clean_tmp=yes
+# machines to test build with
+#machines=qemux86 qemux86-64 qemuarm qemumips qemuppc
+#machines=qemux86
+
+buildhistory=yes
+#testimage=yes
+#testimage_name=core-image-minimal
+
+#workdir=/home/auh/work/
+publish_work_url=https://autobuilder.yocto.io/pub/auh
+
+commit_revert_policy=all
+
+layer_mode=yes
+layer_name={meta}
+layer_dir={layerdir}
+layer_machines={machine}
+
diff --git a/scripts/run-auh-oe b/scripts/run-auh-oe
new file mode 100755
index 000..24d3a51
--- /dev/null
+++ b/scripts/run-auh-oe
@@ -0,0 +1,46 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Run Auto Upgrade Helper in a directory set up by setup_auh.
+#
+# Called with $1 - the directory where the setup was created
+
+if [ -z $1 ]; then
+  echo "Use: $0 auh_setup_dir [publish_dir]"
+  exit 1
+fi
+
+full_dir=$(readlink -e $1)
+
+auh_dir=$full_dir/auto-upgrade-helper
+poky_dir=$full_dir/poky
+openembedded_dir=$full_dir/meta-openembedded
+build_dir=$full_dir/build
+sstate_dir=$full_dir/build/sstate-cache
+
+meta_list="${openembedded_dir}/meta-*"
+machine_list="qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl"
+
+pushd $openembedded_dir || exit 1
+
+# Base the upgrades on meta_openembedded master
+git fetch origin
+git checkout -B tmp-auh-upgrades origin/main
+
+source $poky_dir/oe-init-build-env $build_dir
+for d in $meta_list; do
+  meta=$(basename "$d")
+  sed -E "s|\{meta\}|$meta| ; \
+  s|\{layerdir\}|$openembedded_dir/$meta| ; \
+  s|\{machine\}|$machine_list|" \
+  $build_dir/upgrade-helper/upgrade-helper-oe.in \
+  > $build_dir/upgrade-helper/upgrade-helper.conf
+  $auh_dir/upgrade-helper.py -e all
+done
+
+if [ -n $2 ]; then
+  cp -rf $build_dir/upgrade-helper/* $2
+fi
+
+popd
diff --git a/scripts/setup-auh-oe b/scripts/setup-auh-oe
new file mode 100755
index 000..e5cdf9d
--- /dev/null
+++ b/scripts/setup-auh-oe
@@ -0,0 +1,34 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Initialize Auto Upgrade Helper in a directory.
+#
+# Called with $1 - the directory to place the setup
+CONFIG_DIR=`dirname $0`/auh-config
+
+if [ -z $1 ]; then
+  echo "Use: $0 target_dir"
+  exit 1
+fi
+
+mkdir -p $1
+pushd $1
+
+git clone git://git.yoctoproject.org/poky
+pushd poky
+git config user.email a...@yoctoproject.org
+git config user.name "Auto Upgrade Helper"
+popd
+git clone git://git.openembedded.org/meta-openembedded
+pushd meta-openembedded
+git config user.email a...@yoctoproject.org
+git config user.name "Auto Upgrade Helper"
+popd
+git clone git://git.yoctoproject.org/auto-upgrade-helper
+source poky/oe-init-build-env build
+mkdir -p upgrade-helper
+popd
+
+cp $CONFIG_DIR/upgrade-helper-oe.in $1/build/upgrade-helper
+cat $CONFIG_DIR/local.conf.append >> $1/build/conf/local.conf
-- 
2.39.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this gro

Re: [yocto] [yocto-autobuilder-helper][PATCH v3] AUH: Add Openembedded auto-update-helper with list of layer to test

[yocto] [yocto-autobuilder-helper][PATCH v3] AUH: Add Openembedded auto-update-helper with list of layer to test

Re: [yocto] [yocto-autobuilder-helper][PATCH v2] AUH: Add Openembedded auto-update-helper with list of layer to test

[yocto] [yocto-autobuilder-helper][PATCH v2] AUH: Add Openembedded auto-update-helper with list of layer to test

[yocto] [auh][PATCH v2] upgrade-helper.py: Add layer command line option

Re: [yocto] [yocto-autobuilder-helper][PATCH v2] meta-oe-mirror: Use a 2 step job to fetch and verify meta-openembedded mirror.

[yocto] [yocto-autobuilder-helper][PATCH v2] meta-oe-mirror: Use a 2 step job to fetch and verify meta-openembedded mirror.

[yocto] [auh][PATCH] weeklyjob: Add Openembedded update-helper conf

[yocto] [yocto-autobuilder-helper][PATCH] oe-mirror: Add job to mirror openembedded sources

[yocto] [yocto-autobuilder-helper][PATCH] AUH: Add Openembedded auto-update-helper conf and run

10 matches

Site Navigation

Mail list logo

Footer information