[OE-core] [PATCHv2 2/3] libc-package.bbclass: Only extract wanted files in prep_locale_tree()

2019-02-17 Thread Peter Kjellerstedt
prep_locale_tree() predates the usrmerge DISTRO_FEATURE, which meant it
was not prepared for the case when ${base_libdir} == ${libdir}. This
lead to it extracting files and directories where it shouldn't.

Signed-off-by: Peter Kjellerstedt 
---

PATCHv2: Apparently, some versions of tar require the --wildcards 
option to allow wildcard to be used.

 meta/classes/libc-package.bbclass | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index 6a9547..cf0d8901e3 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -67,7 +67,12 @@ do_prep_locale_tree() {
for i in $treedir/${datadir}/i18n/charmaps/*gz; do 
gunzip $i
done
-   tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
$treedir/${base_libdir}
+   # The extract pattern "./l*.so*" is carefully selected so that it will
+   # match ld*.so and lib*.so*, but not any files in the gconv directory
+   # (if it exists). This makes sure we only unpack the files we need.
+   # This is important in case usrmerge is set in DISTRO_FEATURES, which
+   # means ${base_libdir} == ${libdir}.
+   tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
$treedir/${base_libdir} --wildcards './l*.so*'
if [ -f ${STAGING_LIBDIR_NATIVE}/libgcc_s.* ]; then
tar -cf - -C ${STAGING_LIBDIR_NATIVE} -p libgcc_s.* | tar -xf - 
-C $treedir/${base_libdir}
fi
-- 
2.12.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCHv2 3/3] libc-package.bbclass: Add a progress meter for the package task

2019-02-17 Thread Peter Kjellerstedt
The package task for glibc-locale takes a very long time to execute,
especially if using qemu. In that case, a progress meter helps a lot to
show the progress of the task.

Signed-off-by: Peter Kjellerstedt 
---

PATCHv2: Use enumerate().

 meta/classes/libc-package.bbclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index cf0d8901e3..34c9151ae9 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -348,11 +348,14 @@ python package_do_split_gconvs () {
 makefile = oe.path.join(d.getVar("WORKDIR"), "locale-tree", "Makefile")
 m = open(makefile, "w")
 m.write("all: %s\n\n" % " ".join(commands.keys()))
-for cmd in commands:
+total = len(commands)
+for i, cmd in enumerate(commands):
 m.write(cmd + ":\n")
+m.write("\t@echo 'Progress %d/%d'\n" % (i, total))
 m.write("\t" + commands[cmd] + "\n\n")
 m.close()
 d.setVar("EXTRA_OEMAKE", "-C %s ${PARALLEL_MAKE}" % 
(os.path.dirname(makefile)))
+d.setVarFlag("oe_runmake", "progress", "outof:Progress\s(\d+)/(\d+)")
 bb.note("Executing binary locale generation makefile")
 bb.build.exec_func("oe_runmake", d)
 bb.note("collecting binary locales from locale tree")
-- 
2.12.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCHv2 1/3] libc-package.bbclass: Correct a few paths for when usrmerge is used

2019-02-17 Thread Peter Kjellerstedt
Signed-off-by: Peter Kjellerstedt 
---
 meta/classes/libc-package.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index 0b4c666a74..6a9547 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -68,8 +68,8 @@ do_prep_locale_tree() {
gunzip $i
done
tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
$treedir/${base_libdir}
-   if [ -f ${STAGING_DIR_NATIVE}${prefix_native}/lib/libgcc_s.* ]; then
-   tar -cf - -C 
${STAGING_DIR_NATIVE}/${prefix_native}/${base_libdir} -p libgcc_s.* | tar -xf - 
-C $treedir/${base_libdir}
+   if [ -f ${STAGING_LIBDIR_NATIVE}/libgcc_s.* ]; then
+   tar -cf - -C ${STAGING_LIBDIR_NATIVE} -p libgcc_s.* | tar -xf - 
-C $treedir/${base_libdir}
fi
install -m 0755 ${LOCALETREESRC}${bindir}/localedef 
$treedir/${base_bindir}
 }
@@ -278,7 +278,7 @@ python package_do_split_gconvs () {
 qemu_options = d.getVar('QEMU_OPTIONS')
 
 cmd = "PSEUDO_RELOADED=YES PATH=\"%s\" I18NPATH=\"%s\" %s -L %s \
--E LD_LIBRARY_PATH=%s %s %s/bin/localedef %s" % \
+-E LD_LIBRARY_PATH=%s %s %s${base_bindir}/localedef %s" % \
 (path, i18npath, qemu, treedir, ldlibdir, qemu_options, 
treedir, localedef_opts)
 
 commands["%s/%s" % (outputpath, name)] = cmd
-- 
2.12.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH] cairo: fix UPSTREAM_CHECK_REGEX

2019-02-17 Thread Anuj Mittal
Otherwise it matches versions for 'rcairo' as well.

Signed-off-by: Anuj Mittal 
---
 meta/recipes-graphics/cairo/cairo_1.16.0.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-graphics/cairo/cairo_1.16.0.bb 
b/meta/recipes-graphics/cairo/cairo_1.16.0.bb
index 3e176930cc..cdef023198 100644
--- a/meta/recipes-graphics/cairo/cairo_1.16.0.bb
+++ b/meta/recipes-graphics/cairo/cairo_1.16.0.bb
@@ -83,3 +83,5 @@ FILES_${PN}-script-interpreter = 
"${libdir}/libcairo-script-interpreter.so.*"
 FILES_${PN}-perf-utils = "${bindir}/cairo-trace* ${libdir}/cairo/*.la 
${libdir}/cairo/libcairo-trace.so"
 
 BBCLASSEXTEND = "native nativesdk"
+
+UPSTREAM_CHECK_REGEX = "cairo-(?P\d+(\.\d+)+).tar.xz"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH 3/3] libc-package.bbclass: Add a progress meter for the package task

2019-02-17 Thread Peter Kjellerstedt
Nice, then I’ve learnt something new today. :)

//Peter

From: Christopher Larson 
Sent: den 18 februari 2019 02:21
To: Peter Kjellerstedt 
Cc: Patches and discussions about the oe-core layer 

Subject: Re: [OE-core] [PATCH 3/3] libc-package.bbclass: Add a progress meter 
for the package task



On Sun, Feb 17, 2019 at 7:51 AM Peter Kjellerstedt 
mailto:peter.kjellerst...@axis.com>> wrote:
The package task for glibc-locale takes a very long time to execute,
especially if using qemu. In that case, a progress meter helps a lot to
show the progress of the task.

Signed-off-by: Peter Kjellerstedt 
mailto:peter.kjellerst...@axis.com>>
---
 meta/classes/libc-package.bbclass | 5 +
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index ee598905b0..f1c0545c67 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -348,11 +348,16 @@ python package_do_split_gconvs () {
 makefile = oe.path.join(d.getVar("WORKDIR"), "locale-tree", "Makefile")
 m = open(makefile, "w")
 m.write("all: %s\n\n" % " ".join(commands.keys()))
+i = 1
+total = len(commands)
 for cmd in commands:

for i, cmd in enumerate(commands):
--
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt

2019-02-17 Thread Yeoh, Ee Peng
Hi RP,

Thank you very much for providing me your precious advices and I will 
definitely look into them. 

Let me look into all the improvements that you had developed and I will try my 
best to provide further improvement needed. 

Best regards,
Yeoh Ee Peng 

-Original Message-
From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] 
Sent: Monday, February 18, 2019 12:10 AM
To: Yeoh, Ee Peng ; 
openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt

On Thu, 2019-02-14 at 13:50 +0800, Yeoh Ee Peng wrote:
> v1:
>   Face key error from oe-git-archive
>   Undesirable behavior when storing to multiple git branch
> 
> v2: 
>   Include fix for oe-git-archive
>   Include fix for store result to multiple git branch
>   Improve git commit message   
> 
> v3:
>   Enhance fix for oe-git-archive by using exception catch to
>   improve code readability and easy to understand
> 
> v4:
>   Add new features, merge result files & regression analysis 
>   Add selftest to merge, store, report and regression functionalities
>   Revise codebase for pythonic
>   
> v5:
>   Add required files for selftest testing store
>   
> v6:
>   Add regression for directory and git repository
>   Enable regression pairing base set to multiple target sets 
>   Revise selftest testing for regression
>   
> v7: 
>   Optimize regression computation for ptest results
>   Rename entry point script to resulttool
> 
> Mazliana (1):
>   scripts/resulttool: enable manual execution and result creation
> 
> Yeoh Ee Peng (1):
>   resulttool: enable merge, store, report and regression analysis

Hi Ee Peng,

Thanks for working on this, it does get better each iteration. I've been 
struggling a little to explain what we need to do to finish this off. Firstly I 
wanted to give some feedback on some general python
tips:

a) We can't use subprocess.run() as its a python 3.6 feature and we have 
autobuilder workers with 3.5. This lead to failures like: 
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/242
We can use check_call or other functions instead.

b) I'd not recommend using "file" as a variable name in python as its a 
keyword, similarly "dict" (in resultutils.py).

c) get_dict_value() is something we can probably avoid needing if we use the 
.get() methods of dicts (you can specify a value to return if a value isn't 
present).

I started to experiment with the tool to try and get it to follow the workflow 
we need with the autobuilder QA process. Right now I'm heavily focusing on what 
we need it to do to generate reports from the autobuilder, to the extent that 
I'm ignoring most other workflows.

The reason for this is that I want to get it merged and use this to run
2.7 M3 testing on the autobuilder. The other workflows can be added if/as/when 
we find we have need of them.

I ended up making a few changes to alter the tool to do the things I think we 
need it to and to improve its output/usability. I'll send out a separate patch 
with my changes so far. I've tried to summarise some of the reasoning here:

* Rename resultsutils -> resultutils to match the resultstool -> resulttool 
rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code will 
now add into the json configuration data if its not present, based on the 
directory name.

* When we don't have failed test cases, print something saying so instead of an 
empty table

* Tweak the table headers in the report to be more readable (reference "Test 
Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report since 
printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression report, 
also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in the 
regressions report

* Abstracted the "mapping" table concept used to pairing in the regression code 
to general code in resultutils

* Created multiple mappings for results analysis, results storage and 
'flattening' results data in a merge

* Simplify the merge command to take a source and a destination, letting the 
destination be a directory or a file, removing the need for an output directory 
parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings

* Have the store command place the testresults files in a layout from the 
mapping, making commits into the git repo for results storage more useful for 
simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results storage (and 
simplify the commit messages closer to their defaults)


Despite my changes there are things that still need to be done.
Essential things which need to happen before this code merges:

* oe-git-archive is importing using the 

Re: [OE-core] [PATCH 3/3] libc-package.bbclass: Add a progress meter for the package task

2019-02-17 Thread Christopher Larson
On Sun, Feb 17, 2019 at 7:51 AM Peter Kjellerstedt <
peter.kjellerst...@axis.com> wrote:

> The package task for glibc-locale takes a very long time to execute,
> especially if using qemu. In that case, a progress meter helps a lot to
> show the progress of the task.
>
> Signed-off-by: Peter Kjellerstedt 
> ---
>  meta/classes/libc-package.bbclass | 5 +
>  1 file changed, 5 insertions(+)
>
> diff --git a/meta/classes/libc-package.bbclass
> b/meta/classes/libc-package.bbclass
> index ee598905b0..f1c0545c67 100644
> --- a/meta/classes/libc-package.bbclass
> +++ b/meta/classes/libc-package.bbclass
> @@ -348,11 +348,16 @@ python package_do_split_gconvs () {
>  makefile = oe.path.join(d.getVar("WORKDIR"), "locale-tree",
> "Makefile")
>  m = open(makefile, "w")
>  m.write("all: %s\n\n" % " ".join(commands.keys()))
> +i = 1
> +total = len(commands)
>  for cmd in commands:
>

for i, cmd in enumerate(commands):
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH 2/2] kernel-fitimage.bbclass: Introduce a way to provide external dtb

2019-02-17 Thread Philip Balister
With these patches, I made a fitImage that booted on the mini-zed. It
would be nice to include a note on how to enable this. I took at few
stabs at it and ended up with

PREFERRED_PROVIDER_virtual/dtb = "devicetree"

in he machine file to get the code to trigger.

Philip

On 02/14/2019 07:56 PM, Manjukumar Matha wrote:
> Currently kernel-fitimage gets the dtb from KERNEL_DEVICETREE, however
> there are instances when the DTB file can be generated using other
> recipes, provide a way to include the DTB instead of inkernel device
> tree.
> 
> Add support for external device tree during the creation of fitimage.
> If virtual/dtb is set then use the path provided in
> EXTERNAL_KERNEL_DEVICETREE else use KERNEL_DEVICETREE during fitimage
> process
> 
> Signed-off-by: Manjukumar Matha 
> ---
>  meta/classes/kernel-fitimage.bbclass | 20 +++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/kernel-fitimage.bbclass 
> b/meta/classes/kernel-fitimage.bbclass
> index 8ebd3dd..3adb1c9 100644
> --- a/meta/classes/kernel-fitimage.bbclass
> +++ b/meta/classes/kernel-fitimage.bbclass
> @@ -30,6 +30,13 @@ python __anonymous () {
>  if image:
>  d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' 
> ${INITRAMFS_IMAGE}:do_image_complete')
>  
> + #check if there are any dtb providers
> +providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb")
> +if providerdtb:
> +d.appendVarFlag('do_assemble_fitimage', 'depends', ' 
> virtual/dtb:do_populate_sysroot')
> +d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' 
> virtual/dtb:do_populate_sysroot')
> +d.setVar('EXTERNAL_KERNEL_DEVICETREE', 
> "${RECIPE_SYSROOT}/boot/devicetree")
> +
>  # Verified boot will sign the fitImage and append the public key to
>  # U-Boot dtb. We ensure the U-Boot dtb is deployed before assembling
>  # the fitImage:
> @@ -373,7 +380,8 @@ fitimage_assemble() {
>   #
>   # Step 2: Prepare a DTB image section
>   #
> - if [ -n "${KERNEL_DEVICETREE}" ]; then
> +
> + if [ -z "${EXTERNAL_KERNEL_DEVICETREE}" ] && [ -n 
> "${KERNEL_DEVICETREE}" ]; then
>   dtbcount=1
>   for DTB in ${KERNEL_DEVICETREE}; do
>   if echo ${DTB} | grep -q '/dts/'; then
> @@ -391,6 +399,16 @@ fitimage_assemble() {
>   done
>   fi
>  
> + if [ -n "${EXTERNAL_KERNEL_DEVICETREE}" ]; then
> + dtbcount=1
> + for DTBFILE in ${EXTERNAL_KERNEL_DEVICETREE}/*.dtb; do
> + DTB=`basename ${DTBFILE}`
> + DTB=$(echo "${DTB}" | tr '/' '_')
> + DTBS="${DTBS} ${DTB}"
> + fitimage_emit_section_dtb ${1} ${DTB} ${DTBFILE}
> + done
> + fi
> +
>   #
>   # Step 3: Prepare a setup section. (For x86)
>   #
> 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH 1/3] libc-package.bbclass: Correct a few paths for when usrmerge is used

2019-02-17 Thread Richard Purdie
On Sun, 2019-02-17 at 15:51 +0100, Peter Kjellerstedt wrote:
> Signed-off-by: Peter Kjellerstedt 
> ---
>  meta/classes/libc-package.bbclass | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

I included this patchset in a run on the autobuilder and its failing:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/66

I know the resulttool failures are my fault but the glibc-locale ones
aren't. I'll abort the build as there are enough failures to know its
doomed and I need a results set for other testing.

Cheers,

Richard

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt

2019-02-17 Thread Richard Purdie
On Sun, 2019-02-17 at 17:54 +, Richard Purdie wrote:
> > Despite my changes there are things that still need to be done.
> > Essential things which need to happen before this code merges:
> > 
> > * oe-git-archive is importing using the commit/branch of the
> > current 
> >   repo, not the data in the results file.

Also now fixed. I put my patches into master-next too.

With this working, I was able to run something along the lines of:

for D in $1/*; do
resulttool store $D $2 --allow-empty
done

on the autobuilder's recent results which lead to the creation of this
repository:

http://git.yoctoproject.org/cgit.cgi/yocto-testresults/


> > * Revisit and redo the way the git branch handling is happening.
> > We 
> >   really want to model how oe-build-perf-report handles git repos
> > for 
> >   comparisons:
> >   - Its able to query data from git repos without changing the
> > current 
> > working branch, 
> >   - it can search on tag formats to find comparison data

Which means we now need to make the git branch functionality of the
report and regression commands compare with the above repo, so we're a
step closer to getting thie merged.

Ultimately we'll auto-populate the above repo by having the autobuilder
run a "store" command at the end of its runs.

I have a feeling I may have broken the resulttool selftests so that is
something else which will need to be fixed before anything merges. Time
for me to step away from the keyboard for a bit too.

Cheers,

Richard


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH 2/2] glib-2.0: Improve shared-mime-info dependency handling

2019-02-17 Thread Richard Purdie
This cleans up the handling of the dependency and gets rid of a use of
remove which should never be needed in OE-Core.

Signed-off-by: Richard Purdie 
---
 meta/recipes-core/glib-2.0/glib.inc | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/glib-2.0/glib.inc 
b/meta/recipes-core/glib-2.0/glib.inc
index 7e2847d4b22..a186c1d9eb6 100644
--- a/meta/recipes-core/glib-2.0/glib.inc
+++ b/meta/recipes-core/glib-2.0/glib.inc
@@ -77,10 +77,12 @@ FILES_${PN}-codegen = "${datadir}/glib-2.0/codegen/*.py \
${bindir}/gdbus-codegen"
 FILES_${PN}-utils = "${bindir}/*"
 
-RRECOMMENDS_${PN} += "shared-mime-info"
+SHAREDMIMEDEP = "shared-mime-info"
+SHAREDMIMEDEP_class-native = ""
 # When cross compiling for Windows we don't want to include this
-RRECOMMENDS_${PN}_remove_mingw32 = "shared-mime-info"
-RRECOMMENDS_${PN}_class-native = ""
+SHAREDMIMEDEP_mingw32 = ""
+
+RRECOMMENDS_${PN} += "${SHAREDMIMEDEP}"
 
 ARM_INSTRUCTION_SET_armv4 = "arm"
 ARM_INSTRUCTION_SET_armv5 = "arm"
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH 1/2] scripts/oe-git-archive: Separate out functionality to library function

2019-02-17 Thread Richard Purdie
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.

Signed-off-by: Richard Purdie 
---
 meta/lib/oeqa/utils/gitarchive.py | 173 ++
 scripts/oe-git-archive| 166 ++--
 2 files changed, 179 insertions(+), 160 deletions(-)
 create mode 100644 meta/lib/oeqa/utils/gitarchive.py

diff --git a/meta/lib/oeqa/utils/gitarchive.py 
b/meta/lib/oeqa/utils/gitarchive.py
new file mode 100644
index 000..fddd608593a
--- /dev/null
+++ b/meta/lib/oeqa/utils/gitarchive.py
@@ -0,0 +1,173 @@
+#
+# Helper functions for committing data to git and pushing upstream
+#
+# Copyright (c) 2017, Intel Corporation.
+# Copyright (c) 2019, Linux Foundation
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+# more details.
+#
+
+import os
+import re
+import sys
+from oeqa.utils.git import GitRepo, GitError
+
+class ArchiveError(Exception):
+"""Internal error handling of this script"""
+
+def format_str(string, fields):
+"""Format string using the given fields (dict)"""
+try:
+return string.format(**fields)
+except KeyError as err:
+raise ArchiveError("Unable to expand string '{}': unknown field {} "
+   "(valid fields are: {})".format(
+   string, err, ', '.join(sorted(fields.keys()
+
+
+def init_git_repo(path, no_create, bare, log):
+"""Initialize local Git repository"""
+path = os.path.abspath(path)
+if os.path.isfile(path):
+raise ArchiveError("Invalid Git repo at {}: path exists but is not a "
+   "directory".format(path))
+if not os.path.isdir(path) or not os.listdir(path):
+if no_create:
+raise ArchiveError("No git repo at {}, refusing to create "
+   "one".format(path))
+if not os.path.isdir(path):
+try:
+os.mkdir(path)
+except (FileNotFoundError, PermissionError) as err:
+raise ArchiveError("Failed to mkdir {}: {}".format(path, err))
+if not os.listdir(path):
+log.info("Initializing a new Git repo at %s", path)
+repo = GitRepo.init(path, bare)
+try:
+repo = GitRepo(path, is_topdir=True)
+except GitError:
+raise ArchiveError("Non-empty directory that is not a Git repository "
+   "at {}\nPlease specify an existing Git repository, "
+   "an empty directory or a non-existing directory "
+   "path.".format(path))
+return repo
+
+
+def git_commit_data(repo, data_dir, branch, message, exclude, notes, log):
+"""Commit data into a Git repository"""
+log.info("Committing data into to branch %s", branch)
+tmp_index = os.path.join(repo.git_dir, 'index.oe-git-archive')
+try:
+# Create new tree object from the data
+env_update = {'GIT_INDEX_FILE': tmp_index,
+  'GIT_WORK_TREE': os.path.abspath(data_dir)}
+repo.run_cmd('add .', env_update)
+
+# Remove files that are excluded
+if exclude:
+repo.run_cmd(['rm', '--cached'] + [f for f in exclude], env_update)
+
+tree = repo.run_cmd('write-tree', env_update)
+
+# Create new commit object from the tree
+parent = repo.rev_parse(branch)
+git_cmd = ['commit-tree', tree, '-m', message]
+if parent:
+git_cmd += ['-p', parent]
+commit = repo.run_cmd(git_cmd, env_update)
+
+# Create git notes
+for ref, filename in notes:
+ref = ref.format(branch_name=branch)
+repo.run_cmd(['notes', '--ref', ref, 'add',
+  '-F', os.path.abspath(filename), commit])
+
+# Update branch head
+git_cmd = ['update-ref', 'refs/heads/' + branch, commit]
+if parent:
+git_cmd.append(parent)
+repo.run_cmd(git_cmd)
+
+# Update current HEAD, if we're on branch 'branch'
+if not repo.bare and repo.get_current_branch() == branch:
+log.info("Updating %s HEAD to latest commit", repo.top_dir)
+repo.run_cmd('reset --hard')
+
+return commit
+finally:
+if os.path.exists(tmp_index):
+os.unlink(tmp_index)
+
+
+def expand_tag_strings(repo, name_pattern, msg_subj_pattern, msg_body_pattern,
+   keywords):
+  

Re: [OE-core] [PATCH 2/3] libc-package.bbclass: Use install utility to create directories with right permissions

2019-02-17 Thread Khem Raj
On Sun, Feb 17, 2019 at 7:04 AM Peter Kjellerstedt
 wrote:
>
> > -Original Message-
> > From: Khem Raj 
> > Sent: den 14 februari 2019 16:56
> > To: Peter Kjellerstedt 
> > Cc: openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH 2/3] libc-package.bbclass: Use install
> > utility to create directories with right permissions
> >
> > On Wed, Feb 13, 2019 at 10:05 PM Peter Kjellerstedt
> >  wrote:
> > >
> > > > -Original Message-
> > > > From: openembedded-core-boun...@lists.openembedded.org
> >  > > > core-boun...@lists.openembedded.org> On Behalf Of Khem Raj
> > > > Sent: den 12 februari 2019 06:48
> > > > To: openembedded-core@lists.openembedded.org
> > > > Subject: [OE-core] [PATCH 2/3] libc-package.bbclass: Use install
> > > > utility to create directories with right permissions
> > > >
> > > > Could fix issues where dirs with wrong permission is created with mkdir 
> > > > -p
> > > >
> > > > Signed-off-by: Khem Raj 
> > > > ---
> > > >  meta/classes/libc-package.bbclass | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/meta/classes/libc-package.bbclass 
> > > > b/meta/classes/libc-package.bbclass
> > > > index 0b4c666a74..34025635ce 100644
> > > > --- a/meta/classes/libc-package.bbclass
> > > > +++ b/meta/classes/libc-package.bbclass
> > > > @@ -61,7 +61,7 @@ LOCALETREESRC ?= "${PKGD}"
> > > >  do_prep_locale_tree() {
> > > >   treedir=${WORKDIR}/locale-tree
> > > >   rm -rf $treedir
> > > > - mkdir -p $treedir/${base_bindir} $treedir/${base_libdir} 
> > > > $treedir/${datadir} $treedir/${localedir}
> > > > + install -d $treedir/${base_bindir} $treedir/${base_libdir} 
> > > > $treedir/${datadir} $treedir/${localedir}
> > >
> > > This does not change anything (install -d and mkdir -p does the same
> > > thing).
> > >
> > > >   tar -cf - -C ${LOCALETREESRC}${datadir} -p i18n | tar -xf - -C 
> > > > $treedir/${datadir}
> > > >   # unzip to avoid parsing errors
> > > >   for i in $treedir/${datadir}/i18n/charmaps/*gz; do
> > > >   gunzip $i
> > > >   done
> > > >   tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
> > > > $treedir/${base_libdir}
> > >
> > > This is where the problem happens. When the files are copied from
> > > ${LOCALETREESRC}, the ownership of $treedir${localedir} is changed
> > > to that of ${LOCALETREESRC}${localedir}. However, I now realized
> > > that the problem may be due to the fact that I have usrmerge enabled
> > > in DISTRO_FEATURES, which causes the above line to copy things it
> > > was never meant to copy, in this case ${LOCALETREESRC}${localedir}.
> > > I tried to just remove the line altogether and then it built just
> > > fine. But I do not know what the consequences are of removing it...
>
> It turned out that the consequence is that if one actually does use qemu
> to generate the locale files, then it fails without that line. However,
> since qemu is not used by default for this since quite a long time ago,
> it is not needed in the normal case.
>
> > > I guess it is related to setting up an environment for qemu to run
> > > in, but since that seems to work fine without the files copied by
> > > the above line, maybe it is not needed? Or maybe it is just not
> > > needed when having usrmerge in DISTRO_FEATURES? I tried to
> > > backtrack it through the commit logs, but it was introduced with the
> > > do_prep_locale_tree() function, so no help there.
> >
> > I wonder if we should use --preserve option to tar -xf call here for
> > making it explicit.
>
> I have added a pattern to the tar command that unpacks the files so
> only the files that are needed by qemu are unpacked (see the patch
> series I just sent). That should avoid the problem.
>
> However, I wonder if anyone actually sets
> LOCALE_GENERATION_WITH_CROSS-LOCALEDEF to 0 (it is 1 by default), and
> uses qemu to generate the locale files? It takes quite a lot longer
> (four times on my machine, totaling nearly half an hour). Otherwise,
> if we removed that, the code could be simplified and avoid the copying
> of these files in the first place.

I think the option to not use cross-localedef should be kept alive,
even if its slower, its
helpful in cross validating issues when we might have them with cross localedef

>
> > > > --
> > > > 2.20.1
> > >
> > > //Peter
>
> //Peter
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH v3 0/1] busybox: 1.29.3 -> 1.30.1

2019-02-17 Thread Andrej Valek
v0:
 - proposal update

v1:
 - fix failing bc applet
  - add update-alternatives into bc recipe

v2:
 - disabled previously enabled bc applet
 - re-generate defconfig with new busybox
  - no new applet has been enabled

 ...inittab_1.29.3.bb => busybox-inittab_1.30.1.bb} |   0
 ...te-check-CONFIG_DESKTOP-before-using-od-t.patch |  83 -
 ...-testsuite-check-uudecode-before-using-it.patch |   6 +-
 meta/recipes-core/busybox/busybox/defconfig| 191 +++--
 .../{busybox_1.29.3.bb => busybox_1.30.1.bb}   |   5 +-
 5 files changed, 104 insertions(+), 181 deletions(-)
 rename meta/recipes-core/busybox/{busybox-inittab_1.29.3.bb => 
busybox-inittab_1.30.1.bb} (100%)
 delete mode 100644 
meta/recipes-core/busybox/busybox/0001-testsuite-check-CONFIG_DESKTOP-before-using-od-t.patch
 rename meta/recipes-core/busybox/{busybox_1.29.3.bb => busybox_1.30.1.bb} (89%)

Signed-off-by: Andrej Valek 

-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH v3 1/1] busybox: 1.29.3 -> 1.30.1

2019-02-17 Thread Andrej Valek
- update to last stable version 1.30.1
- remove and refresh already merged patches
- re-generate defconfig

Signed-off-by: Andrej Valek 
---
 ...inittab_1.29.3.bb => busybox-inittab_1.30.1.bb} |   0
 ...te-check-CONFIG_DESKTOP-before-using-od-t.patch |  83 -
 ...-testsuite-check-uudecode-before-using-it.patch |   6 +-
 meta/recipes-core/busybox/busybox/defconfig| 191 +++--
 .../{busybox_1.29.3.bb => busybox_1.30.1.bb}   |   5 +-
 5 files changed, 104 insertions(+), 181 deletions(-)
 rename meta/recipes-core/busybox/{busybox-inittab_1.29.3.bb => 
busybox-inittab_1.30.1.bb} (100%)
 delete mode 100644 
meta/recipes-core/busybox/busybox/0001-testsuite-check-CONFIG_DESKTOP-before-using-od-t.patch
 rename meta/recipes-core/busybox/{busybox_1.29.3.bb => busybox_1.30.1.bb} (89%)

diff --git a/meta/recipes-core/busybox/busybox-inittab_1.29.3.bb 
b/meta/recipes-core/busybox/busybox-inittab_1.30.1.bb
similarity index 100%
rename from meta/recipes-core/busybox/busybox-inittab_1.29.3.bb
rename to meta/recipes-core/busybox/busybox-inittab_1.30.1.bb
diff --git 
a/meta/recipes-core/busybox/busybox/0001-testsuite-check-CONFIG_DESKTOP-before-using-od-t.patch
 
b/meta/recipes-core/busybox/busybox/0001-testsuite-check-CONFIG_DESKTOP-before-using-od-t.patch
deleted file mode 100644
index 9b96eeeae5..00
--- 
a/meta/recipes-core/busybox/busybox/0001-testsuite-check-CONFIG_DESKTOP-before-using-od-t.patch
+++ /dev/null
@@ -1,83 +0,0 @@
-From b09db631e77c63ebe733c69b00108f043f1f1d85 Mon Sep 17 00:00:00 2001
-From: Chen Qi 
-Date: Wed, 21 Nov 2018 10:00:29 +0800
-Subject: [PATCH] testsuite: check CONFIG_DESKTOP before using 'od -t'
-
-The '-t' option for od is enabled by CONFIG_DESKTOP. So
-check this config before using 'od -t' in test cases.
-
-Upstream-Status: Submitted 
[http://lists.busybox.net/pipermail/busybox/2018-November/086895.html]
-
-Signed-off-by: Chen Qi 

- testsuite/echo/echo-prints-dash| 1 +
- testsuite/echo/echo-prints-non-opts| 1 +
- testsuite/echo/echo-prints-slash-zero  | 2 +-
- testsuite/echo/echo-prints-slash_00041 | 2 +-
- testsuite/echo/echo-prints-slash_0041  | 2 +-
- testsuite/echo/echo-prints-slash_041   | 2 +-
- testsuite/echo/echo-prints-slash_41| 2 +-
- 7 files changed, 7 insertions(+), 5 deletions(-)
-
-diff --git a/testsuite/echo/echo-prints-dash b/testsuite/echo/echo-prints-dash
-index ddcdbad..f1f31a0 100644
 a/testsuite/echo/echo-prints-dash
-+++ b/testsuite/echo/echo-prints-dash
-@@ -1 +1,2 @@
-+# FEATURE: CONFIG_DESKTOP
- test "`busybox echo - | od -t x1 | head -n 1`" = "000 2d 0a"
-diff --git a/testsuite/echo/echo-prints-non-opts 
b/testsuite/echo/echo-prints-non-opts
-index c7d1e20..4e642a1 100644
 a/testsuite/echo/echo-prints-non-opts
-+++ b/testsuite/echo/echo-prints-non-opts
-@@ -1 +1,2 @@
-+# FEATURE: CONFIG_DESKTOP
- test "`busybox echo -neEZ | od -t x1 | head -n 1`" = "000 2d 6e 65 45 5a 
0a"
-diff --git a/testsuite/echo/echo-prints-slash-zero 
b/testsuite/echo/echo-prints-slash-zero
-index d97ed8e..948f899 100644
 a/testsuite/echo/echo-prints-slash-zero
-+++ b/testsuite/echo/echo-prints-slash-zero
-@@ -1,3 +1,3 @@
--# FEATURE: CONFIG_FEATURE_FANCY_ECHO
-+# FEATURE: CONFIG_FEATURE_FANCY_ECHO CONFIG_DESKTOP
- 
- test "`busybox echo -e -n 'msg\n\0' | od -t x1 | head -n 1`" = "000 6d 73 
67 0a 00"
-diff --git a/testsuite/echo/echo-prints-slash_00041 
b/testsuite/echo/echo-prints-slash_00041
-index 9cffabd..7ea217b 100644
 a/testsuite/echo/echo-prints-slash_00041
-+++ b/testsuite/echo/echo-prints-slash_00041
-@@ -1,3 +1,3 @@
--# FEATURE: CONFIG_FEATURE_FANCY_ECHO
-+# FEATURE: CONFIG_FEATURE_FANCY_ECHO CONFIG_DESKTOP
- 
- test "`busybox echo -ne '\00041z' | od -t x1 | head -n 1`" = "000 04 31 
7a"
-diff --git a/testsuite/echo/echo-prints-slash_0041 
b/testsuite/echo/echo-prints-slash_0041
-index b07429d..941faca 100644
 a/testsuite/echo/echo-prints-slash_0041
-+++ b/testsuite/echo/echo-prints-slash_0041
-@@ -1,3 +1,3 @@
--# FEATURE: CONFIG_FEATURE_FANCY_ECHO
-+# FEATURE: CONFIG_FEATURE_FANCY_ECHO CONFIG_DESKTOP
- 
- test "`busybox echo -ne '\0041z' | od -t x1 | head -n 1`" = "000 21 7a"
-diff --git a/testsuite/echo/echo-prints-slash_041 
b/testsuite/echo/echo-prints-slash_041
-index 1d70cec..60cbfca 100644
 a/testsuite/echo/echo-prints-slash_041
-+++ b/testsuite/echo/echo-prints-slash_041
-@@ -1,3 +1,3 @@
--# FEATURE: CONFIG_FEATURE_FANCY_ECHO
-+# FEATURE: CONFIG_FEATURE_FANCY_ECHO CONFIG_DESKTOP
- 
- test "`busybox echo -ne '\041z' | od -t x1 | head -n 1`" = "000 21 7a"
-diff --git a/testsuite/echo/echo-prints-slash_41 
b/testsuite/echo/echo-prints-slash_41
-index 6d8999b..afd7c36 100644
 a/testsuite/echo/echo-prints-slash_41
-+++ b/testsuite/echo/echo-prints-slash_41
-@@ -1,3 +1,3 @@
--# FEATURE: CONFIG_FEATURE_FANCY_ECHO
-+# FEATURE: CONFIG_FEATURE_FANCY_ECHO CONFIG_DESKTOP
- 
- test "`busybox echo -ne '\41z' | od -t x1 | head -n 1`" = "000 21 7a"
--- 
-2.7.4
-
diff 

Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt

2019-02-17 Thread Richard Purdie
> Despite my changes there are things that still need to be done.
> Essential things which need to happen before this code merges:
> 
> * oe-git-archive is importing using the commit/branch of the current 
>   repo, not the data in the results file.
> 
> * Fix the -t option to merge command

Got rid of this for now, we can add it later if we need it, can become
a "nice to have" for later.

> * Audit the command option help

Done on my branch.

> * Revisit and redo the way the git branch handling is happening. We 
>   really want to model how oe-build-perf-report handles git repos
> for 
>   comparisons:
>   - Its able to query data from git repos without changing the
> current 
> working branch, 
>   - it can search on tag formats to find comparison data
> 
> * Add ptest summary to the report command

Done on my branch.

Cheers,

Richard

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] resulttool: Tweaks on top of Ee Peng's base patchset

2019-02-17 Thread Richard Purdie
A summary of the tweaks I think we need to make on top of the base
patchset. See the separate email for a discussion of the reasoning
behind this and a list of the work remaining.

Signed-off-by: Richard Purdie 
---
 scripts/lib/resulttool/manualexecution.py  |   8 +-
 scripts/lib/resulttool/merge.py|  54 ++---
 scripts/lib/resulttool/regression.py   | 247 -
 scripts/lib/resulttool/report.py   |  86 +++
 scripts/lib/resulttool/resultsutils.py |  67 --
 scripts/lib/resulttool/resultutils.py  | 104 +
 scripts/lib/resulttool/store.py| 109 +++--
 .../resulttool/template/test_report_full_text.txt  |  31 +--
 8 files changed, 294 insertions(+), 412 deletions(-)
 delete mode 100644 scripts/lib/resulttool/resultsutils.py
 create mode 100644 scripts/lib/resulttool/resultutils.py

diff --git a/scripts/lib/resulttool/manualexecution.py 
b/scripts/lib/resulttool/manualexecution.py
index 64ec581a9f..ecdc4e7a7a 100755
--- a/scripts/lib/resulttool/manualexecution.py
+++ b/scripts/lib/resulttool/manualexecution.py
@@ -18,7 +18,11 @@ import sys
 import datetime
 import re
 from oeqa.core.runner import OETestResultJSONHelper
-from resulttool.resultsutils import load_json_file
+
+def load_json_file(file):
+with open(file, "r") as f:
+return json.load(f)
+
 
 class ManualTestRunner(object):
 def __init__(self):
@@ -134,4 +138,4 @@ def register_commands(subparsers):
  description='helper script for 
results populating during manual test execution. You can find manual test case 
JSON file in meta/lib/oeqa/manual/',
  group='manualexecution')
 parser_build.set_defaults(func=manualexecution)
-parser_build.add_argument('file', help='specify path to manual test case 
JSON file.Note: Please use \"\" to encapsulate the file path.')
\ No newline at end of file
+parser_build.add_argument('file', help='specify path to manual test case 
JSON file.Note: Please use \"\" to encapsulate the file path.')
diff --git a/scripts/lib/resulttool/merge.py b/scripts/lib/resulttool/merge.py
index 1d9cfafd41..3a91bd8eac 100644
--- a/scripts/lib/resulttool/merge.py
+++ b/scripts/lib/resulttool/merge.py
@@ -1,4 +1,4 @@
-# test result tool - merge multiple testresults.json files
+# resulttool - merge multiple testresults.json files
 #
 # Copyright (c) 2019, Intel Corporation.
 #
@@ -11,45 +11,23 @@
 # FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
 # more details.
 #
-from resulttool.resultsutils import load_json_file, get_dict_value, 
dump_json_data
 import os
 import json
-
-class ResultsMerge(object):
-
-def get_test_results(self, logger, file, result_id):
-results = load_json_file(file)
-if result_id:
-result = get_dict_value(logger, results, result_id)
-if result:
-return {result_id: result}
-return result
-return results
-
-def merge_results(self, base_results, target_results):
-for k in target_results:
-base_results[k] = target_results[k]
-return base_results
-
-def _get_write_dir(self):
-basepath = os.environ['BUILDDIR']
-return basepath + '/tmp/'
-
-def dump_merged_results(self, results, output_dir):
-file_output_dir = output_dir if output_dir else self._get_write_dir()
-dump_json_data(file_output_dir, 'testresults.json', results)
-print('Successfully merged results to: %s' % 
os.path.join(file_output_dir, 'testresults.json'))
-
-def run(self, logger, base_result_file, target_result_file, 
target_result_id, output_dir):
-base_results = self.get_test_results(logger, base_result_file, '')
-target_results = self.get_test_results(logger, target_result_file, 
target_result_id)
-if base_results and target_results:
-merged_results = self.merge_results(base_results, target_results)
-self.dump_merged_results(merged_results, output_dir)
+import resulttool.resultutils as resultutils
 
 def merge(args, logger):
-merge = ResultsMerge()
-merge.run(logger, args.base_result_file, args.target_result_file, 
args.target_result_id, args.output_dir)
+# FIXME Add -t support for  args.target_result_id
+
+if os.path.isdir(args.target_result_file):
+results = resultutils.load_resultsdata(args.target_result_file, 
configmap=resultutils.store_map)
+resultutils.append_resultsdata(results, args.base_result_file, 
configmap=resultutils.store_map)
+resultutils.save_resultsdata(results, args.target_result_file)
+else:
+results = resultutils.load_resultsdata(args.base_result_file, 
configmap=resultutils.flatten_map)
+if os.path.exists(args.target_result_file):
+resultutils.append_resultsdata(results, args.target_result_file, 

Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt

2019-02-17 Thread Richard Purdie
On Thu, 2019-02-14 at 13:50 +0800, Yeoh Ee Peng wrote:
> v1:
>   Face key error from oe-git-archive
>   Undesirable behavior when storing to multiple git branch
> 
> v2: 
>   Include fix for oe-git-archive
>   Include fix for store result to multiple git branch
>   Improve git commit message   
> 
> v3:
>   Enhance fix for oe-git-archive by using exception catch to
>   improve code readability and easy to understand
> 
> v4:
>   Add new features, merge result files & regression analysis 
>   Add selftest to merge, store, report and regression functionalities
>   Revise codebase for pythonic
>   
> v5:
>   Add required files for selftest testing store
>   
> v6:
>   Add regression for directory and git repository
>   Enable regression pairing base set to multiple target sets 
>   Revise selftest testing for regression
>   
> v7: 
>   Optimize regression computation for ptest results
>   Rename entry point script to resulttool
> 
> Mazliana (1):
>   scripts/resulttool: enable manual execution and result creation
> 
> Yeoh Ee Peng (1):
>   resulttool: enable merge, store, report and regression analysis

Hi Ee Peng,

Thanks for working on this, it does get better each iteration. I've
been struggling a little to explain what we need to do to finish this
off. Firstly I wanted to give some feedback on some general python
tips:

a) We can't use subprocess.run() as its a python 3.6 feature and we
have autobuilder workers with 3.5. This lead to failures like: 
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/242
We can use check_call or other functions instead.

b) I'd not recommend using "file" as a variable name in python as its a
keyword, similarly "dict" (in resultutils.py).

c) get_dict_value() is something we can probably avoid needing if we
use the .get() methods of dicts (you can specify a value to return if a
value isn't present).

I started to experiment with the tool to try and get it to follow the
workflow we need with the autobuilder QA process. Right now I'm heavily
focusing on what we need it to do to generate reports from the
autobuilder, to the extent that I'm ignoring most other workflows.

The reason for this is that I want to get it merged and use this to run
2.7 M3 testing on the autobuilder. The other workflows can be added
if/as/when we find we have need of them.

I ended up making a few changes to alter the tool to do the things I
think we need it to and to improve its output/usability. I'll send out
a separate patch with my changes so far. I've tried to summarise some
of the reasoning here:

* Rename resultsutils -> resultutils to match the resultstool ->
resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
will now add into the json configuration data if its not present, based
on the directory name.

* When we don't have failed test cases, print something saying so
instead of an empty table

* Tweak the table headers in the report to be more readable (reference
"Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
the regressions report

* Abstracted the "mapping" table concept used to pairing in the
regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
letting the destination be a directory or a file, removing the need for
an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
mappings

* Have the store command place the testresults files in a layout from
the mapping, making commits into the git repo for results storage more
useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
storage (and simplify the commit messages closer to their defaults)


Despite my changes there are things that still need to be done.
Essential things which need to happen before this code merges:

* oe-git-archive is importing using the commit/branch of the current 
  repo, not the data in the results file.

* Fix the -t option to merge command

* Audit the command option help

* Revisit and redo the way the git branch handling is happening. We 
  really want to model how oe-build-perf-report handles git repos for 
  comparisons:
  - Its able to query data from git repos without changing the current 
working branch, 
  - it can search on tag formats to find comparison data

* Add ptest summary to the report command


Things which may be "nice to have" which can come 

Re: [OE-core] [PATCH 2/3] libc-package.bbclass: Use install utility to create directories with right permissions

2019-02-17 Thread Peter Kjellerstedt
> -Original Message-
> From: Khem Raj 
> Sent: den 14 februari 2019 16:56
> To: Peter Kjellerstedt 
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 2/3] libc-package.bbclass: Use install
> utility to create directories with right permissions
> 
> On Wed, Feb 13, 2019 at 10:05 PM Peter Kjellerstedt
>  wrote:
> >
> > > -Original Message-
> > > From: openembedded-core-boun...@lists.openembedded.org
>  > > core-boun...@lists.openembedded.org> On Behalf Of Khem Raj
> > > Sent: den 12 februari 2019 06:48
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [PATCH 2/3] libc-package.bbclass: Use install
> > > utility to create directories with right permissions
> > >
> > > Could fix issues where dirs with wrong permission is created with mkdir -p
> > >
> > > Signed-off-by: Khem Raj 
> > > ---
> > >  meta/classes/libc-package.bbclass | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/meta/classes/libc-package.bbclass 
> > > b/meta/classes/libc-package.bbclass
> > > index 0b4c666a74..34025635ce 100644
> > > --- a/meta/classes/libc-package.bbclass
> > > +++ b/meta/classes/libc-package.bbclass
> > > @@ -61,7 +61,7 @@ LOCALETREESRC ?= "${PKGD}"
> > >  do_prep_locale_tree() {
> > >   treedir=${WORKDIR}/locale-tree
> > >   rm -rf $treedir
> > > - mkdir -p $treedir/${base_bindir} $treedir/${base_libdir} 
> > > $treedir/${datadir} $treedir/${localedir}
> > > + install -d $treedir/${base_bindir} $treedir/${base_libdir} 
> > > $treedir/${datadir} $treedir/${localedir}
> >
> > This does not change anything (install -d and mkdir -p does the same
> > thing).
> >
> > >   tar -cf - -C ${LOCALETREESRC}${datadir} -p i18n | tar -xf - -C 
> > > $treedir/${datadir}
> > >   # unzip to avoid parsing errors
> > >   for i in $treedir/${datadir}/i18n/charmaps/*gz; do
> > >   gunzip $i
> > >   done
> > >   tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
> > > $treedir/${base_libdir}
> >
> > This is where the problem happens. When the files are copied from
> > ${LOCALETREESRC}, the ownership of $treedir${localedir} is changed 
> > to that of ${LOCALETREESRC}${localedir}. However, I now realized 
> > that the problem may be due to the fact that I have usrmerge enabled 
> > in DISTRO_FEATURES, which causes the above line to copy things it 
> > was never meant to copy, in this case ${LOCALETREESRC}${localedir}. 
> > I tried to just remove the line altogether and then it built just 
> > fine. But I do not know what the consequences are of removing it...

It turned out that the consequence is that if one actually does use qemu 
to generate the locale files, then it fails without that line. However, 
since qemu is not used by default for this since quite a long time ago, 
it is not needed in the normal case.
 
> > I guess it is related to setting up an environment for qemu to run 
> > in, but since that seems to work fine without the files copied by 
> > the above line, maybe it is not needed? Or maybe it is just not 
> > needed when having usrmerge in DISTRO_FEATURES? I tried to 
> > backtrack it through the commit logs, but it was introduced with the
> > do_prep_locale_tree() function, so no help there.
> 
> I wonder if we should use --preserve option to tar -xf call here for
> making it explicit.

I have added a pattern to the tar command that unpacks the files so 
only the files that are needed by qemu are unpacked (see the patch 
series I just sent). That should avoid the problem.

However, I wonder if anyone actually sets 
LOCALE_GENERATION_WITH_CROSS-LOCALEDEF to 0 (it is 1 by default), and 
uses qemu to generate the locale files? It takes quite a lot longer 
(four times on my machine, totaling nearly half an hour). Otherwise, 
if we removed that, the code could be simplified and avoid the copying 
of these files in the first place.

> > > --
> > > 2.20.1
> >
> > //Peter

//Peter

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH 1/3] libc-package.bbclass: Correct a few paths for when usrmerge is used

2019-02-17 Thread Peter Kjellerstedt
Signed-off-by: Peter Kjellerstedt 
---
 meta/classes/libc-package.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index 0b4c666a74..6a9547 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -68,8 +68,8 @@ do_prep_locale_tree() {
gunzip $i
done
tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
$treedir/${base_libdir}
-   if [ -f ${STAGING_DIR_NATIVE}${prefix_native}/lib/libgcc_s.* ]; then
-   tar -cf - -C 
${STAGING_DIR_NATIVE}/${prefix_native}/${base_libdir} -p libgcc_s.* | tar -xf - 
-C $treedir/${base_libdir}
+   if [ -f ${STAGING_LIBDIR_NATIVE}/libgcc_s.* ]; then
+   tar -cf - -C ${STAGING_LIBDIR_NATIVE} -p libgcc_s.* | tar -xf - 
-C $treedir/${base_libdir}
fi
install -m 0755 ${LOCALETREESRC}${bindir}/localedef 
$treedir/${base_bindir}
 }
@@ -278,7 +278,7 @@ python package_do_split_gconvs () {
 qemu_options = d.getVar('QEMU_OPTIONS')
 
 cmd = "PSEUDO_RELOADED=YES PATH=\"%s\" I18NPATH=\"%s\" %s -L %s \
--E LD_LIBRARY_PATH=%s %s %s/bin/localedef %s" % \
+-E LD_LIBRARY_PATH=%s %s %s${base_bindir}/localedef %s" % \
 (path, i18npath, qemu, treedir, ldlibdir, qemu_options, 
treedir, localedef_opts)
 
 commands["%s/%s" % (outputpath, name)] = cmd
-- 
2.12.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH 3/3] libc-package.bbclass: Add a progress meter for the package task

2019-02-17 Thread Peter Kjellerstedt
The package task for glibc-locale takes a very long time to execute,
especially if using qemu. In that case, a progress meter helps a lot to
show the progress of the task.

Signed-off-by: Peter Kjellerstedt 
---
 meta/classes/libc-package.bbclass | 5 +
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index ee598905b0..f1c0545c67 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -348,11 +348,16 @@ python package_do_split_gconvs () {
 makefile = oe.path.join(d.getVar("WORKDIR"), "locale-tree", "Makefile")
 m = open(makefile, "w")
 m.write("all: %s\n\n" % " ".join(commands.keys()))
+i = 1
+total = len(commands)
 for cmd in commands:
 m.write(cmd + ":\n")
+m.write("\t@echo 'Progress %d/%d'\n" % (i, total))
 m.write("\t" + commands[cmd] + "\n\n")
+i = i + 1
 m.close()
 d.setVar("EXTRA_OEMAKE", "-C %s ${PARALLEL_MAKE}" % 
(os.path.dirname(makefile)))
+d.setVarFlag("oe_runmake", "progress", "outof:Progress\s(\d+)/(\d+)")
 bb.note("Executing binary locale generation makefile")
 bb.build.exec_func("oe_runmake", d)
 bb.note("collecting binary locales from locale tree")
-- 
2.12.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


[OE-core] [PATCH 2/3] libc-package.bbclass: Only extract wanted files in prep_locale_tree()

2019-02-17 Thread Peter Kjellerstedt
prep_locale_tree() predates the usrmerge DISTRO_FEATURE, which meant it
was not prepared for the case when ${base_libdir} == ${libdir}. This
lead to it extracting files and directories where it shouldn't.

Signed-off-by: Peter Kjellerstedt 
---
 meta/classes/libc-package.bbclass | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/meta/classes/libc-package.bbclass 
b/meta/classes/libc-package.bbclass
index 6a9547..ee598905b0 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -67,7 +67,12 @@ do_prep_locale_tree() {
for i in $treedir/${datadir}/i18n/charmaps/*gz; do 
gunzip $i
done
-   tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
$treedir/${base_libdir}
+   # The extract pattern "./l*.so*" is carefully selected so that it will
+   # match ld*.so and lib*.so*, but not any files in the gconv directory
+   # (if it exists). This makes sure we only unpack the files we need.
+   # This is important in case usrmerge is set in DISTRO_FEATURES, which
+   # means ${base_libdir} == ${libdir}.
+   tar -cf - -C ${LOCALETREESRC}${base_libdir} -p . | tar -xf - -C 
$treedir/${base_libdir} './l*.so*'
if [ -f ${STAGING_LIBDIR_NATIVE}/libgcc_s.* ]; then
tar -cf - -C ${STAGING_LIBDIR_NATIVE} -p libgcc_s.* | tar -xf - 
-C $treedir/${base_libdir}
fi
-- 
2.12.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH v2 1/2] busybox: 1.29.3 -> 1.30.1

2019-02-17 Thread Richard Purdie
On Sun, 2019-02-17 at 13:33 +0100, Andrej Valek wrote:
> It was already enabled, so the question was only about differences.

Sorry, yes, it relocated.

Since we've managed without bc, lets leave that disabled. I think the
busybox upgrade patch should refresh the defconfig even with the
version number change and the lines which move since it will help
someone next time we upgrade and make it explicit what we're
enabling/disabling.

Our default would be to match our old configuration unless there is a
pressing reason to do so (e.g. configuration would have been previously
enabled but is now optional).

Cheers,

Richard

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH v2 1/2] busybox: 1.29.3 -> 1.30.1

2019-02-17 Thread Andrej Valek
It was already enabled, so the question was only about differences.


On 2/17/19 1:30 PM, Richard Purdie wrote:
> On Sun, 2019-02-17 at 13:28 +0100, Andrej Valek wrote:
>> I have already made the comparison. Seems to be that relevant
>> differences are:
>>
>> CONFIG_NOLOGIN
>> CONFIG_NOLOGIN_DEPENDENCIES
>>
>> CONFIG_BC
>>
>> CONFIG_FEATURE_SH_EMBEDDED_SCRIPTS
>>
>> We have to check, which feature we really need.
> 
> What about CONFIG_DC ?
> 
> Cheers,
> 
> Richard
> 
> 
> 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH v2 1/2] busybox: 1.29.3 -> 1.30.1

2019-02-17 Thread Richard Purdie
On Sun, 2019-02-17 at 13:28 +0100, Andrej Valek wrote:
> I have already made the comparison. Seems to be that relevant
> differences are:
> 
> CONFIG_NOLOGIN
> CONFIG_NOLOGIN_DEPENDENCIES
> 
> CONFIG_BC
> 
> CONFIG_FEATURE_SH_EMBEDDED_SCRIPTS
> 
> We have to check, which feature we really need.

What about CONFIG_DC ?

Cheers,

Richard



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH v2 1/2] busybox: 1.29.3 -> 1.30.1

2019-02-17 Thread Andrej Valek
I have already made the comparison. Seems to be that relevant
differences are:

CONFIG_NOLOGIN
CONFIG_NOLOGIN_DEPENDENCIES

CONFIG_BC

CONFIG_FEATURE_SH_EMBEDDED_SCRIPTS

We have to check, which feature we really need.

Regards,
Andrej

On 2/17/19 12:19 PM, Richard Purdie wrote:
> On Sat, 2019-02-16 at 21:38 +, Richard Purdie wrote:
>> On Sat, 2019-02-16 at 20:40 +0100, Andrej Valek wrote:
>>> - update to last stable version 1.30.1
>>> - remove and refresh already merged patches
>>>
>>> Signed-off-by: Andrej Valek 
>>> ---
>>>  ...inittab_1.29.3.bb => busybox-inittab_1.30.1.bb} |  0
>>>  ...te-check-CONFIG_DESKTOP-before-using-od-t.patch | 83 --
>>> --
>>> --
>>>  ...-testsuite-check-uudecode-before-using-it.patch |  6 +-
>>>  .../{busybox_1.29.3.bb => busybox_1.30.1.bb}   |  5 +-
>>>  4 files changed, 5 insertions(+), 89 deletions(-)
>>>  rename meta/recipes-core/busybox/{busybox-inittab_1.29.3.bb =>
>>> busybox-inittab_1.30.1.bb} (100%)
>>>  delete mode 100644 meta/recipes-core/busybox/busybox/0001-
>>> testsuite-
>>> check-CONFIG_DESKTOP-before-using-od-t.patch
>>>  rename meta/recipes-core/busybox/{busybox_1.29.3.bb =>
>>> busybox_1.30.1.bb} (89%)
>>
>> I had a closer look at this and I think we need to refresh the
>> defconfig with the new version of busybox and be explicit about the
>> new things we're taking or not taking...
> 
> btw, in case it helps, I'd use meld to bring up a comparison of the
> old/new defconfigs and use that to merge the pieces back we need/want.
> 
> Cheers,
> 
> Richard
> 
--- .config	2019-02-16 20:25:46.0 +0100
+++ /home/projects/yocto/poky/build/tmp/work/armv5e-poky-linux-gnueabi/busybox/1.29.3-r0/busybox-1.29.3/.config	2019-02-17 13:11:31.665142718 +0100
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
-# Busybox version: 1.30.1
-# Sat Feb 16 19:25:02 2019
+# Busybox version: 1.29.3
+# Sun Feb 17 12:10:48 2019
 #
 CONFIG_HAVE_DOT_CONFIG=y
 
@@ -24,7 +24,6 @@
 CONFIG_FEATURE_PIDFILE=y
 CONFIG_PID_FILE_PATH="/var/run"
 CONFIG_BUSYBOX=y
-CONFIG_FEATURE_SHOW_SCRIPT=y
 # CONFIG_FEATURE_INSTALLER is not set
 # CONFIG_INSTALL_NO_USR is not set
 CONFIG_FEATURE_SUID=y
@@ -83,9 +82,7 @@
 # Library Tuning
 #
 # CONFIG_FEATURE_USE_BSS_TAIL is not set
-CONFIG_FLOAT_DURATION=y
 CONFIG_FEATURE_RTMINMAX=y
-CONFIG_FEATURE_RTMINMAX_USE_LIBC_DEFINITIONS=y
 CONFIG_FEATURE_BUFFERS_USE_MALLOC=y
 # CONFIG_FEATURE_BUFFERS_GO_ON_STACK is not set
 # CONFIG_FEATURE_BUFFERS_GO_IN_BSS is not set
@@ -208,7 +205,6 @@
 # CONFIG_COMM is not set
 CONFIG_CP=y
 # CONFIG_FEATURE_CP_LONG_OPTIONS is not set
-# CONFIG_FEATURE_CP_REFLINK is not set
 CONFIG_CUT=y
 CONFIG_DATE=y
 # CONFIG_FEATURE_DATE_ISOFMT is not set
@@ -291,6 +287,7 @@
 CONFIG_SHUF=y
 CONFIG_SLEEP=y
 CONFIG_FEATURE_FANCY_SLEEP=y
+CONFIG_FEATURE_FLOAT_SLEEP=y
 CONFIG_SORT=y
 CONFIG_FEATURE_SORT_BIG=y
 # CONFIG_FEATURE_SORT_OPTIMIZE_MEMORY is not set
@@ -445,7 +442,6 @@
 CONFIG_FEATURE_FIND_MMIN=y
 CONFIG_FEATURE_FIND_PERM=y
 CONFIG_FEATURE_FIND_TYPE=y
-CONFIG_FEATURE_FIND_EXECUTABLE=y
 CONFIG_FEATURE_FIND_XDEV=y
 CONFIG_FEATURE_FIND_MAXDEPTH=y
 CONFIG_FEATURE_FIND_NEWER=y
@@ -459,7 +455,6 @@
 CONFIG_FEATURE_FIND_PAREN=y
 CONFIG_FEATURE_FIND_SIZE=y
 CONFIG_FEATURE_FIND_PRUNE=y
-CONFIG_FEATURE_FIND_QUIT=y
 # CONFIG_FEATURE_FIND_DELETE is not set
 CONFIG_FEATURE_FIND_PATH=y
 CONFIG_FEATURE_FIND_REGEX=y
@@ -662,8 +657,6 @@
 CONFIG_FEATURE_MOUNT_FSTAB=y
 # CONFIG_FEATURE_MOUNT_OTHERTAB is not set
 CONFIG_MOUNTPOINT=y
-CONFIG_NOLOGIN=y
-CONFIG_NOLOGIN_DEPENDENCIES=y
 # CONFIG_NSENTER is not set
 CONFIG_PIVOT_ROOT=y
 CONFIG_RDATE=y
@@ -739,12 +732,6 @@
 # CONFIG_ADJTIMEX is not set
 # CONFIG_BBCONFIG is not set
 # CONFIG_FEATURE_COMPRESS_BBCONFIG is not set
-# CONFIG_BC is not set
-CONFIG_DC=y
-CONFIG_FEATURE_DC_BIG=y
-# CONFIG_FEATURE_DC_LIBM is not set
-CONFIG_FEATURE_BC_INTERACTIVE=y
-CONFIG_FEATURE_BC_LONG_OPTIONS=y
 # CONFIG_BEEP is not set
 CONFIG_FEATURE_BEEP_FREQ=0
 CONFIG_FEATURE_BEEP_LENGTH_MS=0
@@ -763,6 +750,8 @@
 # CONFIG_FEATURE_CROND_SPECIAL_TIMES is not set
 CONFIG_FEATURE_CROND_DIR=""
 # CONFIG_CRONTAB is not set
+CONFIG_DC=y
+# CONFIG_FEATURE_DC_LIBM is not set
 # CONFIG_DEVFSD is not set
 # CONFIG_DEVFSD_MODLOAD is not set
 # CONFIG_DEVFSD_FG_NP is not set
@@ -837,7 +826,6 @@
 # CONFIG_FEATURE_UNIX_LOCAL is not set
 CONFIG_FEATURE_PREFER_IPV4_ADDRESS=y
 # CONFIG_VERBOSE_RESOLUTION_ERRORS is not set
-# CONFIG_FEATURE_TLS_SHA1 is not set
 # CONFIG_ARP is not set
 # CONFIG_ARPING is not set
 # CONFIG_BRCTL is not set
@@ -925,7 +913,6 @@
 # CONFIG_NTPD is not set
 # CONFIG_FEATURE_NTPD_SERVER is not set
 # CONFIG_FEATURE_NTPD_CONF is not set
-# CONFIG_FEATURE_NTP_AUTH is not set
 CONFIG_PING=y
 CONFIG_PING6=y
 CONFIG_FEATURE_FANCY_PING=y
@@ -947,6 +934,10 @@
 CONFIG_TFTP=y
 # CONFIG_FEATURE_TFTP_PROGRESS_BAR is not set
 # CONFIG_TFTPD is not set
+
+#
+# Common options for tftp/tftpd
+#
 CONFIG_FEATURE_TFTP_GET=y
 CONFIG_FEATURE_TFTP_PUT=y
 # CONFIG_FEATURE_TFTP_BLOCKSIZE is not set
@@ 

Re: [OE-core] [PATCH v2 1/2] busybox: 1.29.3 -> 1.30.1

2019-02-17 Thread Richard Purdie
On Sat, 2019-02-16 at 21:38 +, Richard Purdie wrote:
> On Sat, 2019-02-16 at 20:40 +0100, Andrej Valek wrote:
> > - update to last stable version 1.30.1
> > - remove and refresh already merged patches
> > 
> > Signed-off-by: Andrej Valek 
> > ---
> >  ...inittab_1.29.3.bb => busybox-inittab_1.30.1.bb} |  0
> >  ...te-check-CONFIG_DESKTOP-before-using-od-t.patch | 83 --
> > --
> > --
> >  ...-testsuite-check-uudecode-before-using-it.patch |  6 +-
> >  .../{busybox_1.29.3.bb => busybox_1.30.1.bb}   |  5 +-
> >  4 files changed, 5 insertions(+), 89 deletions(-)
> >  rename meta/recipes-core/busybox/{busybox-inittab_1.29.3.bb =>
> > busybox-inittab_1.30.1.bb} (100%)
> >  delete mode 100644 meta/recipes-core/busybox/busybox/0001-
> > testsuite-
> > check-CONFIG_DESKTOP-before-using-od-t.patch
> >  rename meta/recipes-core/busybox/{busybox_1.29.3.bb =>
> > busybox_1.30.1.bb} (89%)
> 
> I had a closer look at this and I think we need to refresh the
> defconfig with the new version of busybox and be explicit about the
> new things we're taking or not taking...

btw, in case it helps, I'd use meld to bring up a comparison of the
old/new defconfigs and use that to merge the pieces back we need/want.

Cheers,

Richard

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core


Re: [OE-core] [PATCH 6/7] native: Enable RDEPENDS handling

2019-02-17 Thread Richard Purdie
On Sat, 2019-02-16 at 21:46 +, Richard Purdie wrote:
> Native recipes don't currently honour their RDEPENDS. In the case of
> some python scripts this has started causing problems since whilst they're
> not needed at build time (DEPENDS), they are needed at runtime.
> 
> We put off making this change due to circular dependency issues. I believe
> the two such problems in OE-Core are now fixed, as is the dependency loop
> identfication code in bitbake so its time to improve this situation.
> 
> [YOCTO #10113]
> 
> Signed-off-by: Richard Purdie 
> ---
>  meta/classes/native.bbclass | 3 +++
>  1 file changed, 3 insertions(+)

Just for interest, the diff for core-image-sato before/after this patch
is included below. Ideally we'd bubble the dependency to
do_prepare_recipe_sysroot rather than do_populate_sysroot but that
isn't the way the system works. In reality this shouldn't make a huge
difference to build performance other than changing task order a
little.

Cheers,

Richard

--- task-depends.dot2019-02-17 10:11:12.216746214 +
+++ task-depends1.dot   2019-02-17 10:10:20.682055039 +
@@ -3055,6 +3055,8 @@
 "autoconf-native.do_populate_lic" -> "autoconf-native.do_patch"
 "autoconf-native.do_populate_sysroot" [label="autoconf-native 
do_populate_sysroot\n:2.69-r11\nvirtual:native:/media/build1/poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb"]
 "autoconf-native.do_populate_sysroot" -> "autoconf-native.do_install"
+"autoconf-native.do_populate_sysroot" -> 
"gnu-config-native.do_populate_sysroot"
+"autoconf-native.do_populate_sysroot" -> "m4-native.do_populate_sysroot"
 "autoconf-native.do_prepare_recipe_sysroot" [label="autoconf-native 
do_prepare_recipe_sysroot\n:2.69-r11\nvirtual:native:/media/build1/poky/meta/recipes-devtools/autoconf/autoconf_2.69.bb"]
 "autoconf-native.do_prepare_recipe_sysroot" -> "autoconf-native.do_fetch"
 "autoconf-native.do_prepare_recipe_sysroot" -> 
"gnu-config-native.do_populate_sysroot"
@@ -3079,6 +3081,7 @@
 "automake-native.do_populate_lic" [label="automake-native 
do_populate_lic\n:1.16.1-r0\nvirtual:native:/media/build1/poky/meta/recipes-devtools/automake/automake_1.16.1.bb"]
 "automake-native.do_populate_lic" -> "automake-native.do_patch"
 "automake-native.do_populate_sysroot" [label="automake-native 
do_populate_sysroot\n:1.16.1-r0\nvirtual:native:/media/build1/poky/meta/recipes-devtools/automake/automake_1.16.1.bb"]
+"automake-native.do_populate_sysroot" -> "autoconf-native.do_populate_sysroot"
 "automake-native.do_populate_sysroot" -> "automake-native.do_install"
 "automake-native.do_prepare_recipe_sysroot" [label="automake-native 
do_prepare_recipe_sysroot\n:1.16.1-r0\nvirtual:native:/media/build1/poky/meta/recipes-devtools/automake/automake_1.16.1.bb"]
 "automake-native.do_prepare_recipe_sysroot" -> 
"autoconf-native.do_populate_sysroot"
@@ -15498,6 +15501,7 @@
 "flex-native.do_populate_lic" -> "flex-native.do_patch"
 "flex-native.do_populate_sysroot" [label="flex-native 
do_populate_sysroot\n:2.6.0-r0\nvirtual:native:/media/build1/poky/meta/recipes-devtools/flex/flex_2.6.0.bb"]
 "flex-native.do_populate_sysroot" -> "flex-native.do_install"
+"flex-native.do_populate_sysroot" -> "m4-native.do_populate_sysroot"
 "flex-native.do_prepare_recipe_sysroot" [label="flex-native 
do_prepare_recipe_sysroot\n:2.6.0-r0\nvirtual:native:/media/build1/poky/meta/recipes-devtools/flex/flex_2.6.0.bb"]
 "flex-native.do_prepare_recipe_sysroot" -> 
"autoconf-native.do_populate_sysroot"
 "flex-native.do_prepare_recipe_sysroot" -> 
"automake-native.do_populate_sysroot"
@@ -17567,6 +17571,7 @@
 "gdk-pixbuf-native.do_populate_lic" -> "gdk-pixbuf-native.do_patch"
 "gdk-pixbuf-native.do_populate_sysroot" [label="gdk-pixbuf-native 
do_populate_sysroot\n:2.36.11-r0\nvirtual:native:/media/build1/poky/meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.36.11.bb"]
 "gdk-pixbuf-native.do_populate_sysroot" -> "gdk-pixbuf-native.do_install"
+"gdk-pixbuf-native.do_populate_sysroot" -> 
"shared-mime-info-native.do_populate_sysroot"
 "gdk-pixbuf-native.do_prepare_recipe_sysroot" [label="gdk-pixbuf-native 
do_prepare_recipe_sysroot\n:2.36.11-r0\nvirtual:native:/media/build1/poky/meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.36.11.bb"]
 "gdk-pixbuf-native.do_prepare_recipe_sysroot" -> 
"autoconf-native.do_populate_sysroot"
 "gdk-pixbuf-native.do_prepare_recipe_sysroot" -> 
"automake-native.do_populate_sysroot"
@@ -19362,6 +19367,7 @@
 "gobject-introspection-native.do_populate_lic" -> 
"gobject-introspection-native.do_patch"
 "gobject-introspection-native.do_populate_sysroot" 
[label="gobject-introspection-native 
do_populate_sysroot\n:1.58.3-r0\nvirtual:native:/media/build1/poky/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.58.3.bb"]
 "gobject-introspection-native.do_populate_sysroot" -> 
"gobject-introspection-native.do_install"
+"gobject-introspection-native.do_populate_sysroot" -> 
"python3-native.do_populate_sysroot"
 "gobject-introspection-native.do_prepare_recipe_sysroot" 

Re: [OE-core] [PATCH 4/7] glib-2.0-native: Break circular dependency on shared-mime-utils

2019-02-17 Thread Richard Purdie
On Sat, 2019-02-16 at 22:45 +, Peter Kjellerstedt wrote:
> > diff --git a/meta/recipes-core/glib-2.0/glib.inc b/meta/recipes-
> > core/glib-2.0/glib.inc
> > index c9093cee866..7e2847d4b22 100644
> > --- a/meta/recipes-core/glib-2.0/glib.inc
> > +++ b/meta/recipes-core/glib-2.0/glib.inc
> > @@ -80,6 +80,7 @@ FILES_${PN}-utils = "${bindir}/*"
> >  RRECOMMENDS_${PN} += "shared-mime-info"
> 
> Wouldn't it be clearer to change this to:
> 
> RRECOMMENDS_${PN}_append_class-target = " shared-mime-info"
> 
> instead of adding the RRECOMMENDS_${PN}_class-native below?

You could argue that both ways, I see the native case as the exception.
Having the mingw32 remove in there is ugly as well and cleaning that up
at the same time might swing it.

Cheers,

Richard

> >  # When cross compiling for Windows we don't want to include this
> >  RRECOMMENDS_${PN}_remove_mingw32 = "shared-mime-info"
> > +RRECOMMENDS_${PN}_class-native = ""
> > 
> >  ARM_INSTRUCTION_SET_armv4 = "arm"
> >  ARM_INSTRUCTION_SET_armv5 = "arm"
> > --
> > 2.20.1
> 
> //Peter
> 
> 

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core