[OE-core] [PATCHv3] devtool-source.bbclass: Only create each patch branch once

[Olof Johansson]

For conditonally applied patches based on SRC_URI overrides, the
devtool-source class would try to create a new branch for each override
assignment as a postfunc to do_patch, but if the same override was used
multiple times, it would try to create the same branch multiple times,
causing errors like

> Exception: bb.process.ExecutionError: Execution of
\   'git checkout f0f0f0f0f0ff0f0f0f0f0f0f0f0f0f0f0ff0f0f0 -b 
devtool-override-foo'
\   failed with exit code 128:
> fatal: A branch named 'devtool-override-foo' already exists.

This change makes sure that the devtool-source bbclass will only create
one branch per override.

Signed-off-by: Olof Johansson 
Reviewed-by: Peter Kjellerstedt 
---
 meta/classes/devtool-source.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
---
v1 -> v2: Peter Kjellerstedt pointed out some issues with v1; this updated
change is simpler. In addition to what Richard said about it causing failures
in oe-selftest. I had a issues running oe-selftest, mostly because of the large
storage requirements of /tmp and issues with company internal git hooks.  I
think I managed to run all the tests with passing results now, but please be
vigilante of testcase failures; sorry in advance!

v2 -> v3: it seems i can manage to use git send-email/format-patch properly
today due to serious pebkac issues, sorry for the spam. Remove change-id and
re-add the patch "changelog".


diff --git a/meta/classes/devtool-source.bbclass 
b/meta/classes/devtool-source.bbclass
index 67cd0bafb20..1372e32c9e5 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -183,14 +183,14 @@ python devtool_post_patch() {
 
 extra_overrides = d.getVar('DEVTOOL_EXTRA_OVERRIDES')
 if extra_overrides:
-extra_override_list = extra_overrides.split(':')
+extra_overrides = set(extra_overrides.split(':'))
 devbranch = d.getVar('DEVTOOL_DEVBRANCH')
 default_overrides = d.getVar('OVERRIDES').split(':')
 no_overrides = []
 # First, we may have some overrides that are referred to in the recipe 
set in
 # our configuration, so we need to make a branch that excludes those
 for override in default_overrides:
-if override not in extra_override_list:
+if override not in extra_overrides:
 no_overrides.append(override)
 if default_overrides != no_overrides:
 # Some overrides are active in the current configuration, so
@@ -208,7 +208,7 @@ python devtool_post_patch() {
 else:
 bb.process.run('git checkout %s -b devtool-no-overrides' % 
devbranch, cwd=srcsubdir)
 
-for override in extra_override_list:
+for override in extra_overrides:
 localdata = bb.data.createCopy(d)
 if override in default_overrides:
 bb.process.run('git branch devtool-override-%s %s' % 
(override, devbranch), cwd=srcsubdir)
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCHv2] devtool-source.bbclass: Only create each patch branch once

[Olof Johansson]

For conditonally applied patches based on SRC_URI overrides, the
devtool-source class would try to create a new branch for each override
assignment as a postfunc to do_patch, but if the same override was used
multiple times, it would try to create the same branch multiple times,
causing errors like

> Exception: bb.process.ExecutionError: Execution of
\   'git checkout f0f0f0f0f0ff0f0f0f0f0f0f0f0f0f0f0ff0f0f0 -b 
devtool-override-foo'
\   failed with exit code 128:
> fatal: A branch named 'devtool-override-foo' already exists.

This change makes sure that the devtool-source bbclass will only create
one branch per override.

Change-Id: I8adca0b54179793ca92478ad5b3b1b6e0448e26b
Signed-off-by: Olof Johansson 
Reviewed-by: Peter Kjellerstedt 
---
 meta/classes/devtool-source.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/devtool-source.bbclass 
b/meta/classes/devtool-source.bbclass
index 67cd0bafb20..1372e32c9e5 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -183,14 +183,14 @@ python devtool_post_patch() {
 
 extra_overrides = d.getVar('DEVTOOL_EXTRA_OVERRIDES')
 if extra_overrides:
-extra_override_list = extra_overrides.split(':')
+extra_overrides = set(extra_overrides.split(':'))
 devbranch = d.getVar('DEVTOOL_DEVBRANCH')
 default_overrides = d.getVar('OVERRIDES').split(':')
 no_overrides = []
 # First, we may have some overrides that are referred to in the recipe 
set in
 # our configuration, so we need to make a branch that excludes those
 for override in default_overrides:
-if override not in extra_override_list:
+if override not in extra_overrides:
 no_overrides.append(override)
 if default_overrides != no_overrides:
 # Some overrides are active in the current configuration, so
@@ -208,7 +208,7 @@ python devtool_post_patch() {
 else:
 bb.process.run('git checkout %s -b devtool-no-overrides' % 
devbranch, cwd=srcsubdir)
 
-for override in extra_override_list:
+for override in extra_overrides:
 localdata = bb.data.createCopy(d)
 if override in default_overrides:
 bb.process.run('git branch devtool-override-%s %s' % 
(override, devbranch), cwd=srcsubdir)
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Only create each patch branch once

[Olof Johansson]

On 18-10-19 18:00 +0200, Olof Johansson wrote:
> For conditonally applied patches based on SRC_URI overrides, the
> devtool-source class would try to create a new branch for each override
> assignment as a postfunc to do_patch, but if the same override was used
> multiple times, it would try to create the same branch multiple times,

Oops, this should have been a v2; should i repost? I'll repost
now, but let me know if this is redundant.

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] devtool-source.bbclass: Only create each patch branch once

[Olof Johansson]

For conditonally applied patches based on SRC_URI overrides, the
devtool-source class would try to create a new branch for each override
assignment as a postfunc to do_patch, but if the same override was used
multiple times, it would try to create the same branch multiple times,
causing errors like

> Exception: bb.process.ExecutionError: Execution of
\   'git checkout f0f0f0f0f0ff0f0f0f0f0f0f0f0f0f0f0ff0f0f0 -b 
devtool-override-foo'
\   failed with exit code 128:
> fatal: A branch named 'devtool-override-foo' already exists.

This change makes sure that the devtool-source bbclass will only create
one branch per override.

Signed-off-by: Olof Johansson 
---
 meta/classes/devtool-source.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
---
Peter Kjellerstedt pointed out some issues with v1; this updated change is
simpler. In addition to what Richard said about it causing failures in
oe-selftest. I had a issues running oe-selftest, mostly because of the large
storage requirements of /tmp and issues with company internal git hooks.  I
think I managed to run all the tests with passing results now, but please be
vigilante of testcase failures; sorry in advance!

diff --git a/meta/classes/devtool-source.bbclass 
b/meta/classes/devtool-source.bbclass
index 67cd0bafb20..1372e32c9e5 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -183,14 +183,14 @@ python devtool_post_patch() {
 
 extra_overrides = d.getVar('DEVTOOL_EXTRA_OVERRIDES')
 if extra_overrides:
-extra_override_list = extra_overrides.split(':')
+extra_overrides = set(extra_overrides.split(':'))
 devbranch = d.getVar('DEVTOOL_DEVBRANCH')
 default_overrides = d.getVar('OVERRIDES').split(':')
 no_overrides = []
 # First, we may have some overrides that are referred to in the recipe 
set in
 # our configuration, so we need to make a branch that excludes those
 for override in default_overrides:
-if override not in extra_override_list:
+if override not in extra_overrides:
 no_overrides.append(override)
 if default_overrides != no_overrides:
 # Some overrides are active in the current configuration, so
@@ -208,7 +208,7 @@ python devtool_post_patch() {
 else:
 bb.process.run('git checkout %s -b devtool-no-overrides' % 
devbranch, cwd=srcsubdir)
 
-for override in extra_override_list:
+for override in extra_overrides:
 localdata = bb.data.createCopy(d)
 if override in default_overrides:
 bb.process.run('git branch devtool-override-%s %s' % 
(override, devbranch), cwd=srcsubdir)
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Only create each patch branch once

[Olof Johansson]

On 18-10-11 10:34 +0100, Richard Purdie wrote:
> This breaks the oe-selftest devtool tests:
> 
> https://autobuilder.yoctoproject.org/typhoon/api/v2/logs/46751/raw
> 
> oe-selftest -r devtool
> 
> should reproduce.

Thanks! Was under the impression that the DEVTOOL_EXTRA_OVERRIDES
was initialized to "", but that's not the case. Should it be? A
simple fix could be to just add DEVTOOL_EXTRA_OVERRIDES ??= "" to
the top of the bbclass. Is that the most appropriate fix? Or should
I make sure to handle None values coming back from getVar?


-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] devtool-source.bbclass: Only create each patch branch once

[Olof Johansson]

On 18-10-10 17:48 +0200, Peter Kjellerstedt wrote:
> > Change-Id: I8adca0b54179793ca92478ad5b3b1b6e0448e26b
> 
> You probably want to remove the Change-Id before applying this...

Sorry! :( I'll probably forget to remove that again in the
future, I should try making git format-patch automatically strip
that. Thanks Peter!

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] devtool-source.bbclass: Only create each patch branch once

[Olof Johansson]

For conditonally applied patches based on SRC_URI overrides, the
devtool-source class would try to create a new branch for each override
assignment as a postfunc to do_patch, but if the same override was used
multiple times, it would try to create the same branch multiple times,
causing errors like

> Exception: bb.process.ExecutionError: Execution of
\   'git checkout f0f0f0f0f0ff0f0f0f0f0f0f0f0f0f0f0ff0f0f0 -b 
devtool-override-foo'
\   failed with exit code 128:
> fatal: A branch named 'devtool-override-foo' already exists.

This change makes sure that the devtool-source bbclass will only create
one branch per override.

Change-Id: I8adca0b54179793ca92478ad5b3b1b6e0448e26b
Signed-off-by: Olof Johansson 
---
 meta/classes/devtool-source.bbclass | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/meta/classes/devtool-source.bbclass 
b/meta/classes/devtool-source.bbclass
index 56882a4..fa4e999 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -169,16 +169,15 @@ python devtool_post_patch() {
 except bb.process.ExecutionError:
 pass
 
-extra_overrides = d.getVar('DEVTOOL_EXTRA_OVERRIDES')
-if extra_overrides:
-extra_override_list = extra_overrides.split(':')
+devtool_overrides = set(d.getVar('DEVTOOL_EXTRA_OVERRIDES').split(':') or 
[])
+if devtool_overrides:
 devbranch = d.getVar('DEVTOOL_DEVBRANCH')
 default_overrides = d.getVar('OVERRIDES').split(':')
 no_overrides = []
 # First, we may have some overrides that are referred to in the recipe 
set in
 # our configuration, so we need to make a branch that excludes those
 for override in default_overrides:
-if override not in extra_override_list:
+if override not in devtool_overrides:
 no_overrides.append(override)
 if default_overrides != no_overrides:
 # Some overrides are active in the current configuration, so
@@ -196,7 +195,7 @@ python devtool_post_patch() {
 else:
 bb.process.run('git checkout %s -b devtool-no-overrides' % 
devbranch, cwd=srcsubdir)
 
-for override in extra_override_list:
+for override in devtool_overrides:
 localdata = bb.data.createCopy(d)
 if override in default_overrides:
 bb.process.run('git branch devtool-override-%s %s' % 
(override, devbranch), cwd=srcsubdir)
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/4] spdx.bbclass: Replace deprecated string.replace with str.replace

[Olof Johansson]

The string.replace function is removed in python3. Instead, the str
method "replace" should be used instead.

Signed-off-by: Olof Johansson 
---
 meta/classes/spdx.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/spdx.bbclass b/meta/classes/spdx.bbclass
index c5f544d2a4f..ab2eaa5c0c1 100644
--- a/meta/classes/spdx.bbclass
+++ b/meta/classes/spdx.bbclass
@@ -226,7 +226,7 @@ def run_fossology(foss_command, full_spdx):
 except subprocess.CalledProcessError as e:
 return None
 
-foss_output = string.replace(foss_output, '\r', '')
+foss_output = foss_output.replace('\r', '')
 
 # Package info
 package_info = {}
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 4/4] spdx.bbclass: Encode strings before passing to hashlib

[Olof Johansson]

In python3, passing a unicode object to hashlib will result in an
exception that encourages you to encode it first.

Signed-off-by: Olof Johansson 
---
 meta/classes/spdx.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/spdx.bbclass b/meta/classes/spdx.bbclass
index a3e22afc33d..fb78e274a8e 100644
--- a/meta/classes/spdx.bbclass
+++ b/meta/classes/spdx.bbclass
@@ -208,7 +208,7 @@ def hash_file(file_name):
 def hash_string(data):
 import hashlib
 sha1 = hashlib.sha1()
-sha1.update(data)
+sha1.update(data.encode('utf-8'))
 return sha1.hexdigest()
 
 def run_fossology(foss_command, full_spdx):
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/4] spdx.bbclass: Fix undefined variable error

[Olof Johansson]

The path variable is used in an error message a few lines later, but was
never defined.

Signed-off-by: Olof Johansson 
---
 meta/classes/spdx.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/spdx.bbclass b/meta/classes/spdx.bbclass
index ab2eaa5c0c1..9e374d70a6c 100644
--- a/meta/classes/spdx.bbclass
+++ b/meta/classes/spdx.bbclass
@@ -289,7 +289,8 @@ def create_spdx_doc(file_info, scanned_files):
 def get_ver_code(dirname):
 chksums = []
 for f_dir, f in list_files(dirname):
-hash = hash_file(os.path.join(dirname, f_dir, f))
+path = os.path.join(dirname, f_dir, f)
+hash = hash_file(path)
 if not hash is None:
 chksums.append(hash)
 else:
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/4] spdx.bbclass: Make use of bb.utils' sha1_file()

[Olof Johansson]

The same functionality already exists within bitbake, so avoid
duplicating.

Signed-off-by: Olof Johansson 
---
 meta/classes/spdx.bbclass | 9 ++---
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/meta/classes/spdx.bbclass b/meta/classes/spdx.bbclass
index 9e374d70a6c..a3e22afc33d 100644
--- a/meta/classes/spdx.bbclass
+++ b/meta/classes/spdx.bbclass
@@ -202,13 +202,8 @@ def list_files(dir):
 return
 
 def hash_file(file_name):
-try:
-with open(file_name, 'rb') as f:
-data_string = f.read()
-sha1 = hash_string(data_string)
-return sha1
-except:
-return None
+from bb.utils import sha1_file
+return sha1_file(file_name)
 
 def hash_string(data):
 import hashlib
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/4] Small spdx.bbclass fixes

[Olof Johansson]

This patch series contain small adaptions, primarily for the spdx bbclasss to
work with Python3. But with that said, I was unable to get it working when it
came to interfacing with Fossology.

Olof Johansson (4):
  spdx.bbclass: Replace deprecated string.replace with str.replace
  spdx.bbclass: Fix undefined variable error
  spdx.bbclass: Make use of bb.utils' sha1_file()
  spdx.bbclass: Encode strings before passing to hashlib

 meta/classes/spdx.bbclass | 16 ++--
 1 file changed, 6 insertions(+), 10 deletions(-)

-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] rpm: Avoid leaking temporary scriplet files

[Olof Johansson]

On 18-06-30 08:37 +0200, Alexander Kanavin wrote:
> Ideally, "/" should've been replaced with rpmtsRootDir(ts), but ts is
> not accessible from the function, or from its parent, or parent
> parent. So yes, I concede this would be too invasive to pass that down
> the call stack. So if you can amend the above patch of mine with your
> change, commit message, and SOB, that should be best. These changes
> really belong together, as a custom fix for the oe build environment.

Thanks for the feedback, I just sent a v2 of the patch. Hope it
is in line with what you asked for.

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] rpm: Avoid leaking temporary scriplet files

[Olof Johansson]

RPM writes each package scriptlet (post-/preinstall) to
/var/tmp/rpm-tmp.XX --- a lot of files potentially gets created.
When debugging is enabled, these temporary scriptlet files aren't
cleaned up at all and after a while this results in the filesystem
resources are eaten up (like running out of available inodes).

Normally, the temporary files would have been written to the tmp
directory of the target sysroot (which we can easily clean up), but in
this tree, you can't necessarily run the scriptlets.

Fixes [YOCTO #12792]

Signed-off-by: Olof Johansson 
---
 ...installing-execute-package-scriptlets-wit.patch | 35 ++
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git 
a/meta/recipes-devtools/rpm/files/0001-When-cross-installing-execute-package-scriptlets-wit.patch
 
b/meta/recipes-devtools/rpm/files/0001-When-cross-installing-execute-package-scriptlets-wit.patch
index 2be3cb5af3c..4020a310926 100644
--- 
a/meta/recipes-devtools/rpm/files/0001-When-cross-installing-execute-package-scriptlets-wit.patch
+++ 
b/meta/recipes-devtools/rpm/files/0001-When-cross-installing-execute-package-scriptlets-wit.patch
@@ -1,4 +1,4 @@
-From a6f269f879221f2777169c5f7291322afe6b661b Mon Sep 17 00:00:00 2001
+From a89daa75ac970d8e247edc762d1181e9a5b0c5d0 Mon Sep 17 00:00:00 2001
 From: Alexander Kanavin 
 Date: Tue, 17 Jan 2017 14:07:17 +0200
 Subject: [PATCH] When cross-installing, execute package scriptlets without
@@ -7,17 +7,42 @@ Subject: [PATCH] When cross-installing, execute package 
scriptlets without
 This is triggered only when RPM_NO_CHROOT_FOR_SCRIPTS environment variable is 
defined.
 Otherwise they will trigger an explosion of failures, obviously.
 
+Amended 2018-07-03 by Olof Johansson :
+
+  Remove leaking temporary scriptlet files
+
+  Since we tell dnf to run rpm with debug output, this will result in rpm not
+  cleaning up written temporary scriptlet files (same flag controls both
+  behaviors). This wouldn't have been a problem since we normally would use the
+  target sysroot also for temporary files, but we need to chroot out to be able
+  to actually run the rpm scriptlets (purpose of this patch), so the temporary
+  files are written to the host's /var/tmp/ directory, causing a gradual
+  resource leakage on the host system for every RPM based do_rootfs task
+  executed.
+
+  Signed-off-by: Olof Johansson 
+
 Upstream-Status: Inappropriate [oe-core specific]
 Signed-off-by: Alexander Kanavin 
 ---
- lib/rpmscript.c | 8 +++-
- 1 file changed, 7 insertions(+), 1 deletion(-)
+ lib/rpmscript.c | 11 ---
+ 1 file changed, 8 insertions(+), 3 deletions(-)
 
 diff --git a/lib/rpmscript.c b/lib/rpmscript.c
-index 98d3f420d..b95b5d606 100644
+index cc98c4885..f8bd3df04 100644
 --- a/lib/rpmscript.c
 +++ b/lib/rpmscript.c
-@@ -467,7 +467,13 @@ rpmRC rpmScriptRun(rpmScript script, int arg1, int arg2, 
FD_t scriptFd,
+@@ -394,8 +394,7 @@ exit:
+   Fclose(out);/* XXX dup'd STDOUT_FILENO */
+ 
+ if (fn) {
+-  if (!rpmIsDebug())
+-  unlink(fn);
++  unlink(fn);
+   free(fn);
+ }
+ free(mline);
+@@ -428,7 +427,13 @@ rpmRC rpmScriptRun(rpmScript script, int arg1, int arg2, 
FD_t scriptFd,
  
  if (rc != RPMRC_FAIL) {
if (script_type & RPMSCRIPTLET_EXEC) {
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] rpm: Avoid leaking temporary scriplet files

[Olof Johansson]

On 18-06-26 12:33 +0200, Alexander Kanavin wrote:
> I believe the culprit is likely this patch:
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/rpm/files/0001-When-cross-installing-execute-package-scriptlets-wit.patch
> 
> As scriptlets are executed outside of rpm's chroot, they are being
> written into system's /var/tmp, not rootfs's. This patch should
> probably be amended to prepend the rootfs path when scriptlets are
> written out, and then we should be sorted.

Is that not a more invasive change than just removing the tmp
files unconditionally (especially in terms of maintainability)?
I'm not even sure how exactly to do that, but I'm obviously
missing something. For instance, is there a way to manipulate the
global rpmMacroContext temporarily?

But your theory seems to be right, no /var/tmp/rpm-tmp.* files
are created if i drop the patch (but some scripts fail, as
expected).

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] rpm: Avoid leaking temporary scriplet files

[Olof Johansson]

On 18-06-25 15:25 +0200, Alexander Kanavin wrote:
> It's better to clean them up in meta/lib/oe/rootfs.py, instead of
> forever carrying a custom patch to rpm.

I have the same hesitations, but we don't know the filenames to
clean up. We can't remove every /var/tmp/rpm-tmp.*, since they
can be created by unrelated rpm executions.

This suggests some (probably[*]) larger, upstreamable work done
in rpm itself, to be able to configure that behavior. I think
this is better than nothing, for now. Perhaps the bug report
shouldn't be closed, and this be merged as a workaround? (It
causes real issues, at least for images with lots of
pre/postinstalls.)

Does anybody see value in what --rpmverbosity=debug brings, as
opposed to the default level of info? Would dropping that from
lib/oe/package_manager.py's RpmPM._invoke_dnf be a better
solution, or workaround?

(Or maybe this is a bug in dnf? At least, this side effect is not
clearly documented, from what I can tell :))

-- 
olofjn

*) i'm not a c developer, so at least for me. :(
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] insane.bbclass: Don't let warnings make previous errors non-fatal

[Olof Johansson]

package_qa_handle_error() returns True on non-fatal issues and False on
fatal issues. But the current usage has been to do

  sane = package_qa_handle_error(...)

which would always reset sanity status to be that of the last issue
identified. This change the assignments to use the &= operator instead:

  sane &= package_qa_handle_error(...)

As far as I can tell, this is not a real problem in practice, because
warnings of different levels (WARN_QA, ERROR_QA) does not seem to have
been mixed in a way that triggered this issue.

Signed-off-by: Olof Johansson 
---
 meta/classes/insane.bbclass | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index bdfdc315aae..713b40eac42 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -600,7 +600,7 @@ python populate_lic_qa_checksum() {
 return
 
 if not lic_files and d.getVar('SRC_URI'):
-sane = package_qa_handle_error("license-checksum", pn + ": Recipe file 
fetches files and does not have license file information (LIC_FILES_CHKSUM)", d)
+sane &= package_qa_handle_error("license-checksum", pn + ": Recipe 
file fetches files and does not have license file information 
(LIC_FILES_CHKSUM)", d)
 
 srcdir = d.getVar('S')
 corebase_licensefile = d.getVar('COREBASE') + "/LICENSE"
@@ -608,11 +608,11 @@ python populate_lic_qa_checksum() {
 try:
 (type, host, path, user, pswd, parm) = bb.fetch.decodeurl(url)
 except bb.fetch.MalformedUrl:
-sane = package_qa_handle_error("license-checksum", pn + ": 
LIC_FILES_CHKSUM contains an invalid URL: " + url, d)
+sane &= package_qa_handle_error("license-checksum", pn + ": 
LIC_FILES_CHKSUM contains an invalid URL: " + url, d)
 continue
 srclicfile = os.path.join(srcdir, path)
 if not os.path.isfile(srclicfile):
-sane = package_qa_handle_error("license-checksum", pn + ": 
LIC_FILES_CHKSUM points to an invalid file: " + srclicfile, d)
+sane &= package_qa_handle_error("license-checksum", pn + ": 
LIC_FILES_CHKSUM points to an invalid file: " + srclicfile, d)
 continue
 
 if (srclicfile == corebase_licensefile):
@@ -696,7 +696,7 @@ python populate_lic_qa_checksum() {
 else:
 msg = pn + ": LIC_FILES_CHKSUM is not specified for " +  url
 msg = msg + "\n" + pn + ": The md5 checksum is " + md5chksum
-sane = package_qa_handle_error("license-checksum", msg, d)
+sane &= package_qa_handle_error("license-checksum", msg, d)
 
 if not sane:
 bb.fatal("Fatal QA errors found, failing task.")
@@ -733,14 +733,14 @@ def package_qa_check_staged(path,d):
 file_content = file_content.replace(recipesysroot, "")
 if workdir in file_content:
 error_msg = "%s failed sanity test (workdir) in path 
%s" % (file,root)
-sane = package_qa_handle_error("la", error_msg, d)
+sane &= package_qa_handle_error("la", error_msg, d)
 elif file.endswith(".pc"):
 with open(path) as f:
 file_content = f.read()
 file_content = file_content.replace(recipesysroot, "")
 if pkgconfigcheck in file_content:
 error_msg = "%s failed sanity test (tmpdir) in path 
%s" % (file,root)
-sane = package_qa_handle_error("pkgconfig", error_msg, 
d)
+sane &= package_qa_handle_error("pkgconfig", 
error_msg, d)
 
 return sane
 
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] insane.bbclass: Make missing license file fatal

[Olof Johansson]

If a license file referenced from LIC_FILES_CHKSUM doesn't exist,
insane.bbclass would output an error message, but would continue the
build. This change makes this error fatal (as I suspect has been the
intention).

Signed-off-by: Olof Johansson 
---
 meta/classes/insane.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index bd7f51956a6..bdfdc315aae 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -612,7 +612,7 @@ python populate_lic_qa_checksum() {
 continue
 srclicfile = os.path.join(srcdir, path)
 if not os.path.isfile(srclicfile):
-package_qa_handle_error("license-checksum", pn + ": 
LIC_FILES_CHKSUM points to an invalid file: " + srclicfile, d)
+sane = package_qa_handle_error("license-checksum", pn + ": 
LIC_FILES_CHKSUM points to an invalid file: " + srclicfile, d)
 continue
 
 if (srclicfile == corebase_licensefile):
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] rpm: Avoid leaking temporary scriplet files

[Olof Johansson]

RPM writes each package scriptlet (post-/preinstall) to /var/tmp/rpm-tmp.XX
 --- a lot of files get created. When debugging is enabled, these temporary
scriptlet files aren't cleaned up at all and after a while this results in the
filesystem resources are eaten up (like running out of available inodes).

This is a quick fix to avoid this. It does degrade functionality for
those working with debugging pre-/postintall scripts.

Fixes [YOCTO #12792]

Signed-off-by: Olof Johansson 
---
 ...ve-written-tmp-scriptlets-even-with-debug.patch | 37 ++
 meta/recipes-devtools/rpm/rpm_4.14.1.bb|  6 
 2 files changed, 43 insertions(+)
 create mode 100644 
meta/recipes-devtools/rpm/files/0001-Always-remove-written-tmp-scriptlets-even-with-debug.patch

diff --git 
a/meta/recipes-devtools/rpm/files/0001-Always-remove-written-tmp-scriptlets-even-with-debug.patch
 
b/meta/recipes-devtools/rpm/files/0001-Always-remove-written-tmp-scriptlets-even-with-debug.patch
new file mode 100644
index 000..fe4b45bfb3b
--- /dev/null
+++ 
b/meta/recipes-devtools/rpm/files/0001-Always-remove-written-tmp-scriptlets-even-with-debug.patch
@@ -0,0 +1,37 @@
+From b74b549ff61ab8a2bdb041febbe135ad10fa303c Mon Sep 17 00:00:00 2001
+From: Olof Johansson 
+Date: Sun, 17 Jun 2018 12:12:36 +0200
+Subject: [PATCH] Always remove written tmp scriptlets, even with debugging
+ enabled
+
+RPM writes each package scriptlet (post-/preinstall) to /var/tmp/rpm-tmp.XX
+ --- a lot of files get created. When debugging is enabled, these temporary
+scriptlet files aren't cleaned up at all and after a while this results in the
+filesystem resources are eaten up (like running out of available inodes).
+
+This is a quick fix to avoid this. It does degrade functionality for
+those working with debugging pre-/postintall scripts.
+
+Upstream-Status: inappropriate [OE specific]
+Signed-off-by: Olof Johansson 
+---
+ lib/rpmscript.c | 3 +--
+ 1 file changed, 1 insertion(+), 2 deletions(-)
+
+diff --git a/lib/rpmscript.c b/lib/rpmscript.c
+index 98d3f420d..6bf23c18c 100644
+--- a/lib/rpmscript.c
 b/lib/rpmscript.c
+@@ -430,8 +430,7 @@ exit:
+   Fclose(out);/* XXX dup'd STDOUT_FILENO */
+ 
+ if (fn) {
+-  if (!rpmIsDebug())
+-  unlink(fn);
++  unlink(fn);
+   free(fn);
+ }
+ free(mline);
+-- 
+2.11.0
+
diff --git a/meta/recipes-devtools/rpm/rpm_4.14.1.bb 
b/meta/recipes-devtools/rpm/rpm_4.14.1.bb
index e5e87d39039..63a00ea1c0b 100644
--- a/meta/recipes-devtools/rpm/rpm_4.14.1.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.14.1.bb
@@ -45,6 +45,12 @@ SRC_URI = 
"git://github.com/rpm-software-management/rpm;branch=rpm-4.14.x \
file://0003-rpmSetCloseOnExec-use-getrlimit.patch \
"
 
+# When we use RPM internally in OpenEmbedded, we enable debug output. This
+# unfortunately makes rpm not remove generated temporary files, which causes
+# incremental leakage for every build. Change behavior in rpm-native to always
+# remove the temporary files.
+SRC_URI_append_class-native = " 
file://0001-Always-remove-written-tmp-scriptlets-even-with-debug.patch"
+
 PE = "1"
 SRCREV = "bfee1410af51c1cc9724791fb8d985260a62102b"
 
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/5] lib/oe/package.py: is_elf: Make it less prone to false positives

[Olof Johansson]

On 17-12-18 12:00 +, Burton, Ross wrote:
> Just pushed ross/elf with this change.  The next obvious step is to turn
> lib/oe/qa.py into a new lib/oe/elf.py which has a better API and includes
> the stripped function.

Great; I've adapted my changes on top of yours and tested it a
bit. I've published what I have on github for now, if you want to
take a look. Is this along the lines of what you want?

 https://github.com/olof/poky-contrib/compare/olof/elf
 https://github.com/olof/poky-contrib/compare/ross/elf...olof/elf

(Let me know if you prefer me to send it as draft patches to the
list instead.)

(Btw, I didn't know about Kaitai before, but I like the idea of
declarative binary parsing a lot :))

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/5] lib/oe/package.py: is_elf: Make it less prone to false positives

[Olof Johansson]

On 17-12-06 21:38 +, Burton, Ross wrote:
> If you are looking at this then you'll want to pull my branch again as I
> just pushed a few fixes to make ARM binaries actually work.
> 
> Also, a prototype "is this binary stripped" function would be:
> 
> elf = oe.elf.Elf.from_file(sys.argv[1])
> for h in elf.header.section_headers:
> if h.type == oe.elf.Elf.ShType.progbits and h.name == ".debug_info":
> return False
> return True
> 
> If you're not working on this yet, please say, as I'll probably have a look
> this week.

I'm so sorry, I missed your email; I am, but maybe not as quickly
as you are able to(?). My is_stripped looks kind of like yours,
but without the ShType.progbits. Thanks for the suggestion! I
were able to hackishly integrate the Elf class it with
staging.bbclass and package.bbclass.

I did have a couple of questions; if I were to add an is_stripped
method, would that fit best in the Elf class itself or in some
wrapper?  Would renaming the elf class to "ElfParser" class make
more sense, and have an ELF class that wraps it? I can publish a
draft somewhere later today.

Thanks your work!
-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/5] lib/oe/package.py: is_elf: Make it less prone to false positives

[Olof Johansson]

On 17-12-04 12:36 +, Burton, Ross wrote:
> You might be interested in some of the patches I've got sitting in
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ross/mutb
> specifically around "Add new ELF parser".  This adds a fully-featured
> Python ELF parser to lib/oe which could be used to inspect the binaries the
> way file does, but without having to call and parse the output of file.

Oh, that's awesome. Please disregard v1 of my patch series, and
I'll try to adapt package.bbclass and lib/oe/package.py to make
use of your changes and see what happens. Thanks!

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/5] lib/oe/package.py: is_elf: Make it less prone to false positives

[Olof Johansson]

On 17-12-01 22:13 +0100, Olof Johansson wrote:
> On 17-12-01 11:43 -0600, Mark Hatle wrote:
> > The original implementation did not include the ',' options because 
> > different
> > versions of file (as well as different architectures) would return different
> > strings.
> > 
> > They all contains the same information, but order and inclusion of ',' 
> > changed
> > regularly.
> > 
> > So I would caution that for this to check out a wide variety of host 
> > systems and
> > architectures would need to be verified.  (It's very possible that all 
> > modern
> > systems now conform to a single standard...)
> 
> I was a little bit worried about this, thanks, this is useful
> feedback. Perhaps dropping the "," while still retaining the
> whitespace delimitation could be enough, and still work with a
> variety of file implementations?

Reading more about this, I was wrong in calling it GNU file, I
don't know where I got that from. It actually seems to be only
one prevalent implementation, one written by Ian Darwin [0]:

> Who's using it?
>
>Most known BSD distributions (FreeBSD, NetBSD, Darwin/Mac OS X, etc)
>Every known Linux distribution

Could you elaborate on the portability issues you've seen?


0: http://www.darwinsys.com/file/

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/5] lib/oe/package.py: Expose is_elf

[Olof Johansson]

On 17-12-01 16:50 +0100, Olof Johansson wrote:
> is_elf/isELF had copies in both staging.bbclass and package.bbclass.
> After recent refactoring in staging.bbclass (involving breaking out the
> isELF function to is_elf in lib/oe/package.py), the implementions
> diverged. It would be beneficial to make everybody use this one
> implementation, so let's expose it here for others to use.
...
> +def is_elf(path, on_error=_is_elf_error):
...
> +:param on_error: callable, gets called when an error occurs.
> + the callback takes a message parameter. A
> + default error handler is provided that prints
> + the message with 'bb.error'.
> +
...
> +"""
...
> +if ret:
> +error_cb('"file %s" failed')

This should be on_error, not error_cb, sorry! Will fix in v2,
together with not assuming "," in file output, as suggested by
Mark.

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/5] lib/oe/package.py: is_elf: Make it less prone to false positives

[Olof Johansson]

On 17-12-01 11:43 -0600, Mark Hatle wrote:
> The original implementation did not include the ',' options because different
> versions of file (as well as different architectures) would return different
> strings.
> 
> They all contains the same information, but order and inclusion of ',' changed
> regularly.
> 
> So I would caution that for this to check out a wide variety of host systems 
> and
> architectures would need to be verified.  (It's very possible that all modern
> systems now conform to a single standard...)
> 
> (The rest of the serious looks like a very good improvement, and I've got no
> further comments on that.)

I was a little bit worried about this, thanks, this is useful
feedback. Perhaps dropping the "," while still retaining the
whitespace delimitation could be enough, and still work with a
variety of file implementations?

And while we're talking portability, is the use of GNU file's
--brief option a problem (as introduced by another patch in this
series)? It isn't specified in posix, I don't know how widespread
it is.

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/5] lib/oe/package.py: Expose is_elf

[Olof Johansson]

is_elf/isELF had copies in both staging.bbclass and package.bbclass.
After recent refactoring in staging.bbclass (involving breaking out the
isELF function to is_elf in lib/oe/package.py), the implementions
diverged. It would be beneficial to make everybody use this one
implementation, so let's expose it here for others to use.

Signed-off-by: Olof Johansson <olo...@axis.com>
---
 meta/lib/oe/package.py | 88 +++---
 1 file changed, 54 insertions(+), 34 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 1e5c3aa8e1..f1f9333e0f 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1,3 +1,6 @@
+import mmap
+import oe.utils
+
 def runstrip(arg):
 # Function to strip a single file, called from split_and_strip_files below
 # A working 'file' (one which works on the target architecture)
@@ -44,6 +47,56 @@ def runstrip(arg):
 
 return
 
+# Detect .ko module by searching for "vermagic=" string
+def is_kernel_module(path):
+with open(path) as f:
+return mmap.mmap(f.fileno(), 0, 
prot=mmap.PROT_READ).find(b"vermagic=") >= 0
+
+def _is_elf_error(msg):
+bb.error('is_elf: %s' % msg)
+
+def is_elf(path, on_error=_is_elf_error):
+"""
+Determine if a given file is an ELF archive (and other attributes),
+using the file utility.
+
+:param path: str, path of potential ELF file
+:param on_error: callable, gets called when an error occurs.
+ the callback takes a message parameter. A
+ default error handler is provided that prints
+ the message with 'bb.error'.
+
+is_elf returns a bitstring of flags, corresponding to various
+properties:
+
+*  1: ELF
+*  2: stripped
+*  4: executable
+*  8: shared library
+* 16: kernel module
+
+A return value of 0 means that the file is not an ELF file.
+"""
+ret, result = oe.utils.getstatusoutput(
+"file \"%s\"" % path.replace("\"", "\\\""))
+
+if ret:
+error_cb('"file %s" failed')
+return
+
+if not "ELF" in result:
+return 0
+
+exec_type = 1
+if "not stripped" not in result:
+exec_type |= 2
+if "executable" in result:
+exec_type |= 4
+if "shared" in result:
+exec_type |= 8
+if "relocatable" in result and is_kernel_module(path):
+exec_type |= 16
+return exec_type
 
 def strip_execs(pn, dstdir, strip_cmd, libdir, base_libdir, 
qa_already_stripped=False):
 """
@@ -56,40 +109,7 @@ def strip_execs(pn, dstdir, strip_cmd, libdir, base_libdir, 
qa_already_stripped=
 :param qa_already_stripped: Set to True if already-stripped' in 
${INSANE_SKIP}
 This is for proper logging and messages only.
 """
-import stat, errno, oe.path, oe.utils, mmap
-
-# Detect .ko module by searching for "vermagic=" string
-def is_kernel_module(path):
-with open(path) as f:
-return mmap.mmap(f.fileno(), 0, 
prot=mmap.PROT_READ).find(b"vermagic=") >= 0
-
-# Return type (bits):
-# 0 - not elf
-# 1 - ELF
-# 2 - stripped
-# 4 - executable
-# 8 - shared library
-# 16 - kernel module
-def is_elf(path):
-exec_type = 0
-ret, result = oe.utils.getstatusoutput(
-"file \"%s\"" % path.replace("\"", "\\\""))
-
-if ret:
-bb.error("split_and_strip_files: 'file %s' failed" % path)
-return exec_type
-
-if "ELF" in result:
-exec_type |= 1
-if "not stripped" not in result:
-exec_type |= 2
-if "executable" in result:
-exec_type |= 4
-if "shared" in result:
-exec_type |= 8
-if "relocatable" in result and is_kernel_module(path):
-exec_type |= 16
-return exec_type
+import stat, errno, oe.path, oe.utils
 
 elffiles = {}
 inodes = {}
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/5] package.bbclass: Make use of common is_elf function

[Olof Johansson]

The isELF and is_elf function share a common ancestry, but have
diverged. Let's use the implementation from oe.package.

Signed-off-by: Olof Johansson <olo...@axis.com>
---
 meta/classes/package.bbclass | 40 +---
 1 file changed, 9 insertions(+), 31 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 2053d46395..f65596126d 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -857,6 +857,12 @@ python fixup_perms () {
 
 python split_and_strip_files () {
 import stat, errno
+import oe.package
+
+def is_elf(path):
+return oe.package.is_elf(
+path, lambda msg: package_qa_handle_error("split-strip", msg, d)
+)
 
 dvar = d.getVar('PKGD')
 pn = d.getVar('PN')
@@ -892,34 +898,6 @@ python split_and_strip_files () {
 sourcefile = d.expand("${WORKDIR}/debugsources.list")
 bb.utils.remove(sourcefile)
 
-# Return type (bits):
-# 0 - not elf
-# 1 - ELF
-# 2 - stripped
-# 4 - executable
-# 8 - shared library
-# 16 - kernel module
-def isELF(path):
-type = 0
-ret, result = oe.utils.getstatusoutput("file \"%s\"" % 
path.replace("\"", "\\\""))
-
-if ret:
-msg = "split_and_strip_files: 'file %s' failed" % path
-package_qa_handle_error("split-strip", msg, d)
-return type
-
-# Not stripped
-if "ELF" in result:
-type |= 1
-if "not stripped" not in result:
-type |= 2
-if "executable" in result:
-type |= 4
-if "shared" in result:
-type |= 8
-return type
-
-
 #
 # First lets figure out all of the files we may have to process ... do 
this only once!
 #
@@ -961,14 +939,14 @@ python split_and_strip_files () {
 # If it's a symlink, and points to an ELF file, we capture 
the readlink target
 if cpath.islink(file):
 target = os.readlink(file)
-if isELF(ltarget):
-#bb.note("Sym: %s (%d)" % (ltarget, 
isELF(ltarget)))
+if is_elf(ltarget):
+#bb.note("Sym: %s (%d)" % (ltarget, 
is_elf(ltarget)))
 symlinks[file] = target
 continue
 
 # It's a file (or hardlink), not a link
 # ...but is it ELF, and is it already stripped?
-elf_file = isELF(file)
+elf_file = is_elf(file)
 if elf_file & 1:
 if elf_file & 2:
 if 'already-stripped' in (d.getVar('INSANE_SKIP_' 
+ pn) or "").split():
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 4/5] lib/oe/package.py: is_elf: Disallow shell specials to be expanded

[Olof Johansson]

By using single quotes instead of double quotes, we don't have to worry
about escaping dangerous characters, other than ' itself.

Signed-off-by: Olof Johansson <olo...@axis.com>
---
 meta/lib/oe/package.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index eab94feb91..976d2ef36c 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -78,7 +78,7 @@ def is_elf(path, on_error=_is_elf_error):
 A return value of 0 means that the file is not an ELF file.
 """
 ret, result = oe.utils.getstatusoutput(
-"file -b \"%s\"" % path.replace("\"", "\\\""))
+"file -b '%s'" % path.replace("'", "\\'"))
 
 if ret:
 error_cb('"file %s" failed')
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 0/5] Improve isELF, gets triggered by ELF anywhere in pathname

[Olof Johansson]

If the substring "ELF" is found anywhere in the pathname, the isELF function
would identify the file as an ELF file. The function could also be used to
execute arbitrary shell commands as the user running bitbake, since the file
execution allows processing of shell meta characters like variable expansion.

The isELF function has been copied and was until this patchset available from
two locations, one in lib/oe/package.py and one in package.bbclass. The two
functions had diverged. This is changed so that one common implementation is
used.

Olof Johansson (5):
  lib/oe/package.py: Expose is_elf
  package.bbclass: Make use of common is_elf function
  lib/oe/package.py: is_elf: Don't let filename influence filetype
  lib/oe/package.py: is_elf: Disallow shell specials to be expanded
  lib/oe/package.py: is_elf: Make it less prone to false positives

 meta/classes/package.bbclass | 40 +---
 meta/lib/oe/package.py   | 88 +++-
 2 files changed, 63 insertions(+), 65 deletions(-)

-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 5/5] lib/oe/package.py: is_elf: Make it less prone to false positives

[Olof Johansson]

Avoid matching substrings that are picked up from paths, for instance.
Do this by anchoring the tokens we look for (e.g "executable" or "not
stripped") with whitespace and punctuation.

Submitted with this patch series is a change that adds the use of
--brief to file. This removes the path prefix to the output, but the
path can still be included in shebang lines (which file will report as
something like "a /foo/bar/baz.py script").

Signed-off-by: Olof Johansson <olo...@axis.com>
---
 meta/lib/oe/package.py | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 976d2ef36c..2bd771cfc5 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -84,17 +84,17 @@ def is_elf(path, on_error=_is_elf_error):
 error_cb('"file %s" failed')
 return
 
-if not "ELF" in result:
+if not result.startswith("ELF "):
 return 0
 
 exec_type = 1
-if "not stripped" not in result:
+if ", not stripped" not in result:
 exec_type |= 2
-if "executable" in result:
+if " executable, " in result:
 exec_type |= 4
-if "shared" in result:
+if " shared object, " in result:
 exec_type |= 8
-if "relocatable" in result and is_kernel_module(path):
+if "relocatable, " in result and is_kernel_module(path):
 exec_type |= 16
 return exec_type
 
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/5] lib/oe/package.py: is_elf: Don't let filename influence filetype

[Olof Johansson]

The is_elf function is simply looking for the substring ELF in the
output from file. But file, by default, prefixes the outut with the
filename. If the filename containts the substring ELF anywhere, is_elf
would identify it as a ELF.

The --brief (or -b) flag of GNU's file utility makes file not prepend
filename to the output.

Signed-off-by: Olof Johansson <olo...@axis.com>
---
 meta/lib/oe/package.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index f1f9333e0f..eab94feb91 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -78,7 +78,7 @@ def is_elf(path, on_error=_is_elf_error):
 A return value of 0 means that the file is not an ELF file.
 """
 ret, result = oe.utils.getstatusoutput(
-"file \"%s\"" % path.replace("\"", "\\\""))
+"file -b \"%s\"" % path.replace("\"", "\\\""))
 
 if ret:
 error_cb('"file %s" failed')
-- 
2.11.0

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] licenses.conf: Fix variable name in comments (FOSS_NO_COPYRIGHT)

[Olof Johansson]

A FOSSology related variable was renamed from FOSS_COPRYIGHT to
FOSS_NO_COPYRIGHT, but the comment block describing the variable
in licenses.conf was missed.

Besides fixing this, this change also removes a redundant comment
about where the variable is defined (it's right there! ;-)).

Signed-off-by: Olof Johansson <olo...@axis.com>
---
 meta/conf/licenses.conf | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/conf/licenses.conf b/meta/conf/licenses.conf
index 9917c40..d210a0e 100644
--- a/meta/conf/licenses.conf
+++ b/meta/conf/licenses.conf
@@ -133,11 +133,10 @@ DATA_LICENSE = "CC0-1.0"
 # You can set option to control if the copyright information will be skipped
 # during the identification process.
 #
-# It is defined as [FOSS_COPYRIGHT] in ./meta/conf/licenses.conf.
-# FOSS_COPYRIGHT = "true"
+# FOSS_NO_COPYRIGHT = "true"
 #   NO copyright will be processed. That means only license information will be
 #   identified and output to SPDX file
-# FOSS_COPYRIGHT = "false"
+# FOSS_NO_COPYRIGHT = "false"
 #   Copyright will be identified and output to SPDX file along with license
 #   information. The process will take more time than not processing copyright
 #   information.
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] how to add new perl modules to stock OE build?

[Olof Johansson]

On 16-10-27 08:46 -0400, Robert P. J. Day wrote:
> 
>   ack ... never mind, i see my fundamental misunderstanding. i thought
> all of the recipes being processed via the "perl-modules" package were
> being (if necessary) downloaded from CPAN, or something like that,
> then turned into installable rpms. i didn't look closely enough into
> the perl source to see that all the modules listed in
> "perl-rdepends_5.22.1.inc" are already in the perl source tree.
> 
>   so i'm back to my original question -- what is the proper way to add
> arbitrary perl modules to an image? if someone has a simple example of
> how it's done, that'd be great. as proof-of-concept, i'd like to add
> the Text::Template module to my "qemuppc" target.
> 
>   thoughts? sorry for all the earlier noise, i thought i had it
> figured out but ... no.

Hi,

Take a look at meta-perl in the meta-openembedded repository. It
contains a lot of recipes for Perl modules (Text::Template isn't
one of them though).

The cpan class in oe-core knows how to build Makefile.PL
(ExtUtils::MakeMaker) based perl modules (like Text::Template),
and cpan_build knows how to handle Build.PL (Module::Build) based
perl modules.

Note also the naming convention of perl-module-foo is used by
core modules (those shipped with perl itself); the convention
used for third party cpan modules is "libfoo-perl", e.g.
libtext-template-perl.

Hope this helps,
-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] sanity.bbclass: Only verify /bin/sh link if it's a link

[Olof Johansson]

If /bin/sh is a regular file (and not a symlink), we assume it's a
reasonable shell and allow it.

Signed-off-by: Olof Johansson <olof.johans...@axis.com>
---
 meta/classes/sanity.bbclass | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 088dd2a..98345ce 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -932,10 +932,11 @@ def check_sanity_everybuild(status, d):
 with open(checkfile, "w") as f:
 f.write(tmpdir)
 
-# Check /bin/sh links to dash or bash
-real_sh = os.path.realpath('/bin/sh')
-if not real_sh.endswith('/dash') and not real_sh.endswith('/bash'):
-status.addresult("Error, /bin/sh links to %s, must be dash or bash\n" 
% real_sh)
+# If /bin/sh is a symlink, check that it points to dash or bash
+if os.path.islink('/bin/sh'):
+real_sh = os.path.realpath('/bin/sh')
+if not real_sh.endswith('/dash') and not real_sh.endswith('/bash'):
+status.addresult("Error, /bin/sh links to %s, must be dash or 
bash\n" % real_sh)
 
 def check_sanity(sanity_data):
 class SanityStatus(object):
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 6/7] gnupg.org-hosted recipes: Change SRC_URI to https site

[Olof Johansson]

On 16-05-25 16:29 +0300, Jussi Kukkonen wrote:
> https version seems more reliable and in an informal test fetching
> all gnupg recipes now takes <20% of the time it used to.
> 
> Define GNUPG_MIRROR in bitbake.conf so future tweaks to this are
> easier. Replace some slower mirrors with the official ftp site
> and another from gnupg.org mirror list.
> 
> Change gnutls to use GNUPG_MIRROR as well: they have their own domain
> but the ftp site is the slow, unreliable gnupg.org one.
> 
> Signed-off-by: Jussi Kukkonen 
> ---
...
> -SRC_URI = "ftp://ftp.gnupg.org/gcrypt/gnupg/gnupg-${PV}.tar.bz2 \
> +SRC_URI = "${GNUPG_MIRROR}/gnupg/gnupg-${PV}.tar.bz2 \

I'm not fond of having variables like these, since it makes the
URL in the recipe harder to work with and interpret. Maybe not a
deal breaker, but we already have PREMIRRORS, which can be used
instead of variable indirection. A similar patch, almost a year
ago, was rejected.

 http://lists.openembedded.org/pipermail/openembedded-core/2015-July/108217.html

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] coreutils: revert upstream commit causing havoc with ls output

[Olof Johansson]

On 16-05-23 14:09 -0400, Paul Gortmaker wrote:
> > On 16-05-20 20:02 -0400, Paul Gortmaker wrote:
> > > Several large distros are voting with their feet and actively
> > > reverting the change, as per what can be seen above for Debian.
> > 
> > To me, it sounds like you're saying that the Debian maintainer is
> > actively taking a stand against upstream, but the changelog makes
> > it sound more like a temporary thing.
> 
> I did write "... it doesn't appear that the coreutils folks are going
> to change the default back to the old way..."

What I lack from the (OE) commit message is that this should be
something temporary, just like in Debian. Instead, you write that
Debian "voted with their feet", when in fact, this is a
maintainer decision out of practical considerations. (As can be
understood from the debian/changelog.)

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] coreutils: revert upstream commit causing havoc with ls output

[Olof Johansson]

On 16-05-20 20:02 -0400, Paul Gortmaker wrote:
> Several large distros are voting with their feet and actively
> reverting the change, as per what can be seen above for Debian.

To me, it sounds like you're saying that the Debian maintainer is
actively taking a stand against upstream, but the changelog makes
it sound more like a temporary thing.

>From the changelog entry:
> * Disable default ls quoting for now to get the rest of 8.25
>   into testing. (Closes: #813164)

Could you add something in the commit message to make this
clearer?

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] os-release: remove double-quotes around VERSION_ID value

[Olof Johansson]

On 16-02-12 14:15 +0200, Dmitry Rozhkov wrote:
> man 5 os-release states that VERSION_ID is
> 
> "
> a lower-case string (mostly numeric, no spaces or other
> characters outside of 0-9, a-z, ".", "_" and "-")
> identifying the operating system version
> "
> 
> This becomes crucial when ClearLinux's software
> update mechanism is integrated into builds, because
> its client side ignores VERSION_ID's value if it
> doesn't conform the definition.

On Debian, the value is quoted:

VERSION_ID="8"

So I think the documentation refers to the value of VERSION_ID,
with the surrounding quotes not being part of the *value*.

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 00/15] Prepare for EXTRA_OEMAKE = "" in bitbake.conf

[Olof Johansson]

On 16-02-08 11:49 +, Burton, Ross wrote:
> On 5 February 2016 at 18:04, Mike Crowe  wrote:
> 
> > This series (sent only to openembedded-core) fixes various recipes
> > that rely on the current default value of
> > EXTRA_OEMAKE = "-e MAKEFLAGS=" to explicitly set that value in the
> > hope that bitbake.conf can set EXTRA_OEMAKE = "" in the future.
> >
> 
> We need to also vet all instances of EXTRA_OEMAKE in recipes.  I did a
> build with the EXTRA_OEMAKE change in bitbake.conf and hdparm failed to
> pass QA because the binary was already stripped, as the recipe does
> EXTRA_OEMAKE += "STRIP=echo".  That assignment isn't getting through to the
> makefiles without -e.

It should though, since all -e does is to make the environment
variables take precedence over make assignments. But in case of
assignments in EXTRA_OEMAKE, they are make assignments, so they
shouldn't be affected:

$ cat Makefile
FOO = bar

all:
echo $(FOO)

.PHONY: all


$ make FOO=qux
echo qux
qux
$ make -e FOO=qux
echo qux
qux
$ FOO=qux make
echo bar
bar
$ FOO=qux make -e
echo qux
qux


-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] bitbake-whatchanged: avoid double do_ task name prefix

[Olof Johansson]

When used with --verbose, the heading for each task looks like

  === The verbose changes of example.do_do_compile:

This should instead be

  === The verbose changes of example.do_compile:

Signed-off-by: Olof Johansson <olof.johans...@axis.com>
---
 scripts/bitbake-whatchanged | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/bitbake-whatchanged b/scripts/bitbake-whatchanged
index 55cfe4b..af54d16 100755
--- a/scripts/bitbake-whatchanged
+++ b/scripts/bitbake-whatchanged
@@ -190,7 +190,7 @@ def print_depchanged(d_new = None, d_old = None, verbose = 
False):
 if sigdata_re.match(full_path_old) and 
sigdata_re.match(full_path_new):
 output = bb.siggen.compare_sigfiles(full_path_old, 
full_path_new)
 if output:
-print("\n=== The verbose changes of %s.do_%s:" % (pn, 
task))
+print("\n=== The verbose changes of %s.%s:" % (pn, task))
 print('\n'.join(output))
 else:
 # Format the output, the format is:
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] insane.bbclass: do_package_qa depends on WARN_QA and ERROR_QA

[Olof Johansson]

On 16-01-21 00:11 -0800, Robert Yang wrote:
> --- a/meta/classes/insane.bbclass
> +++ b/meta/classes/insane.bbclass
> @@ -1098,6 +1098,7 @@ python do_package_qa () {
>  
>  do_package_qa[vardepsexclude] = "BB_TASKDEPDATA"
>  do_package_qa[rdeptask] = "do_packagedata"
> +do_package_qa[vardeps] += "${WARN_QA} ${ERROR_QA}"

Shouldn't this be "WARN_QA ERROR_QA"? Or you'll get the expanded
values of the variables.

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] rpm: remove bashisms: [ x == x ] -> [ x = x ]

[Olof Johansson]

The postinst and postrm of rpm contained a bashism, that could in some
situations potentially cause ldconfig not be triggered when it should
be. If you use dash on host, test would fail because of syntax errors.
But on host, it should fail because of the comparison. On target, you
often use busybox ash, and it supports == as an alias for =. So in
practice, only if you use a shell like dash on target, you'll run into
issues with this.

Signed-off-by: Olof Johansson <olo...@axis.com>
---
 meta/recipes-devtools/rpm/rpm_4.11.2.bb | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/rpm/rpm_4.11.2.bb 
b/meta/recipes-devtools/rpm/rpm_4.11.2.bb
index a48852f..3683971 100644
--- a/meta/recipes-devtools/rpm/rpm_4.11.2.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.11.2.bb
@@ -101,8 +101,7 @@ do_install_append() {
 }
 
 pkg_postinst_${PN}() {
-
-[ "x\$D" == "x" ] && ldconfig
+[ "x\$D" = "x" ] && ldconfig
 test -f ${localstatedir}/lib/rpm/Packages || rpm --initdb
 rm -f ${localstatedir}/lib/rpm/Filemd5s \
   ${localstatedir}/lib/rpm/Filedigests \
@@ -112,7 +111,7 @@ pkg_postinst_${PN}() {
 }
 
 pkg_postrm_${PN}() {
-[ "x\$D" == "x" ] && ldconfig
+[ "x\$D" = "x" ] && ldconfig
 
 }
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] PV from filename, not reflected in siginfo?

[Olof Johansson]

Hi,

I'm currently investigating reports of bitbake not correctly
handling a type of change to a recipe, where the only change is a
filename rename to update the PV.

With bitbake-dumpsigs:

 ...
 Variable PV value is ${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', 
False),d)[1] or '1.0'}
 Variable PN value is ${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', 
False),d)[0] or 'defaultpkgname'}
 ...

... with nothing containing the resolved value of PV. I would
expect the computed value of PV to be a part of the siginfo. The
consquence of this seems to be that bitbake doesn't schedule
dependents to be rebuilt, i.e. if I rename the recipe
foo_1.2.3.bb to foo_1.2.4.bb, the recipe foo is rebuilt, but the
image isn't.

I tried some changes:

 - Changing the PV assignment in bitbake.conf to :=. Didn't work,
   got errors that I suspect are related to not being able to
   include foo-${PV}.inc files. The value of PV was clobbered.
   Also played around with using BB_FILENAME instead, but didn't
   help.

 - Changes within Bitbake that assigned the version part of the
   filename to PV when loading new files ending with .bb. This
   worked but is really ugly.

 - Moving the PV assignment into the recipe (as an explicit
   assignment). This works, makes dependents build as expected
   and bitbake-dumpsigs lists the value, but is also ugly.

I suspect this isn't usually a problem in OE-Core, since most
recipe updates requires updates within the recipe as well
(tarball checksums, for instance), but that's not as true for our
internal recipes.

Is this a known issue? Any ideas on how to solve this? Nicer
workarounds?


Thanks,
-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] PV from filename, not reflected in siginfo?

[Olof Johansson]

On 16-01-18 14:37 +, Richard Purdie wrote:
> On Mon, 2016-01-18 at 13:44 +0100, Olof Johansson wrote:
> > Hi,
> > 
> > I'm currently investigating reports of bitbake not correctly
> > handling a type of change to a recipe, where the only change is a
> > filename rename to update the PV.
...
> > Is this a known issue? Any ideas on how to solve this? Nicer
> > workarounds?
> 
> It isn't known but I can imagine how this could cause a problem.
> 
> I suspect (but am guessing) that:
> 
> PV[vardepvalue] = "${PV}"
> 
> might happen to fix this...

Indeed, that works! Thanks a lot! It's not completely obvious to
me why it works, but I'm guessing that this is just a way to make
sure PV is expanded before siginfo is dumped?

Is something based on this acceptable for integration in oe-core,
or is this just a (much) nicer workaround?

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] bitbake.conf: add GITHUB_GIT

[Olof Johansson]

Excerpts from Otavio Salvador's message of 2015-07-29 18:27:32 +0200:
 It allows for people to add local mirrors for example and even remap
 fetching without touching the recipes. It is a good addition IMO.

Can't you use PREMIRRORS for that?

-- 
olof johansson
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] bitbake.conf: add GITHUB_GIT

[Olof Johansson]

Excerpts from Robert Yang's message of 2015-07-30 09:09:07 +0200:
 PREMIRRORS is for all the recipes/protocols ?

No, PREMIRRORS is quite expressive, for instance:

PREMIRRORS =  \
  https?$://downloads.example.org$/.*   http://mirror.example.com/ \n \


This should only premirror downloads.example.com files, fetched via http
or https. See the bitbake fetcher's uri_replace method in
bitbake/lib/bb/fetch2/__init__.py for the implementation.

-- 
olof johansson
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] base.bbclass: Don't warn about invalid PACKAGECONFIGs by default

[Olof Johansson]

Excerpts from Burton, Ross's message of 2015-07-30 11:53:39 +0200:
 Can you elaborate on what this approach you refer to is?

For some of our internal recipes, we export a list of PACKAGECONFIGs regardless
of their use in the recipe. This is to make sure the list is synchronized over
all the affected recipes.

-- 
olof johansson
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] base.bbclass: Don't warn about invalid PACKAGECONFIGs by default

[Olof Johansson]

The recently added warnings for recipes that try to use undefined
packageconfigs isn't compatible with the PACKAGECONFIG approach of
exporting a common set of configs and letting the recipe look only at
the configs they care about.

This change introduces a STRICT_PACKAGECONFIG variable that one can
enable to get the warnings.

Signed-off-by: Olof Johansson olo...@axis.com
---
 meta/classes/base.bbclass | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index e0f1053..6f652d8 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -340,9 +340,12 @@ python () {
 pkgconfig = (d.getVar('PACKAGECONFIG', True) or ).split()
 pn = d.getVar(PN, True)
 
-for pconfig in pkgconfig:
-if pconfig not in pkgconfigflags:
-bb.warn(%s: invalid PACKAGECONFIG: %s % (pn, pconfig))
+# If STRICT_PACKAGECONFIG is set, warn about recipes that sets 
undefined
+# PACKAGECONFIGs.
+if bb.utils.to_boolean(d.getVar(STRICT_PACKAGECONFIG, True)):
+for pconfig in pkgconfig:
+if pconfig not in pkgconfigflags:
+bb.warn(%s: invalid PACKAGECONFIG: %s % (pn, pconfig))
 
 mlprefix = d.getVar(MLPREFIX, True)
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] base.bbclass: Don't warn about invalid PACKAGECONFIGs by default

[Olof Johansson]

Excerpts from Robert Yang's message of 2015-07-30 12:08:37 +0200:
 I have a new patch for this, it will be moved into insane.bbclass,
 and the recipe can set INSANE_SKIP:
 +
 +# Check invalid PACKAGECONFIG
 +pkgconfig = (d.getVar(PACKAGECONFIG, True) or ).split()
 +if pkgconfig:
 +pkgconfigflags = d.getVarFlags(PACKAGECONFIG) or {}
 +for pconfig in pkgconfig:
 +if pconfig not in pkgconfigflags:
 +pn = d.getVar('PN', True)
 +error_msg = %s: invalid PACKAGECONFIG: %s % (pn, pconfig)
 +package_qa_handle_error(invalid-pkgconfig, error_msg, d)
 
 I'm testing it, and will send it out sooner.

Thanks, such a patch would work nicely with my use case!

-- 
olof johansson
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 0/2] bitbake.conf: add GITHUB_GIT

[Olof Johansson]

Excerpts from Robert Yang's message of 2015-07-29 12:19:06 +0200:
 It is just like what GNOME_GIT, GNU_MIRROR and others did.

Tbh, I don't understand them either, but maybe that's just me. What is
the reason? I think it makes it harder to see what is happening.

-- 
olof johansson
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] bash: define NON_INTERACTIVE_LOGIN_SHELLS

[Olof Johansson]

Excerpts from wenzong@windriver.com's message of 2015-07-07 07:56:48 +0200:
 From: Li Wang li.w...@windriver.com
 
 Allow login shells to read the startup files, even if they are not
 interactive.
 
 The patch comes from:
 * bash-3.2-30.fc10.src.rpm - bash-2.03-profile.patch

How is this different from

commit 3985cef24de2da3b6895e67f4d577d4806c032b0
Author: Chen Qi qi.c...@windriver.com
Date:   Tue Apr 21 17:30:47 2015 +0800

bash: explicitly define NON_INTERACTIVE_LOGIN_SHELLS in CFLAGS

If NON_INTERACTIVE_LOGIN_SHELLS is defined, all login shells read the
startup files, even if they are not interactive.

This is the behaviour of other major distros like Ubuntu and Fedora.
We also need to set it so that when executing `su -l xxx -c env' command,
/etc/profile is parsed.

[YOCTO #5359]
[YOCTO #7137]

?

-- 
olof johansson
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] ^{} what is it good for? (bitbake problem)

[Olof Johansson]

On 14-06-04 10:06 +0200, Steffen Sledz wrote:
 So what is this ^{} which comes from lib/bb/fetch2/git.py good for?
 Or is this a remnant from a typo?

From git-rev-parse(1):

  rev^{}, e.g. v0.99.8^{}
  A suffix ^ followed by an empty brace pair means the object could be a
  tag, and dereference the tag recursively until a non-tag object is
  found.

Without this change, (some?) tag references would end up not
being considered to be on the requested branch by bitbake (by
default, master), as the tag objects themselves are not. The tags
therefore have to be resolved to a reference to a commit.

Given the description in git-rev-parse(1), it surprises me that
it doesn't work for some cases. I think it has to do with the tag
being annotated or not.

-- 
olofjn
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2] sanity.bbclass: support wildcards in SANITY_TESTED_DISTROS

[Olof Johansson]

On 14-03-03 19:17 +0100, Khem Raj wrote:
 On Mon, Mar 3, 2014 at 6:37 AM, Olof Johansson olof.johans...@axis.com 
 wrote:
  With this change, you can use shell like globbing expressions (as
  supported by Python's fnmatch) for entries in SANITY_TESTED_DISTROS.
  This makes it possible to say that, e.g. all Debian 7 Wheezy releases
  are supported with the entry Debian-7.*.
 
 I dont think its a good thing. We should be strict about it as we are.
 otherwise it can end up
 with bigger problems to save few  typing words

I don't really think we are strict about it today. Currently,
Ubuntu LTS (12.04) does not include point release number in its
lsb release name. Debian does (and I noticed a similar bump patch
was sent for CentOS recently). I suspect that in many cases, a
new Ubuntu point release contains about the same changes that the
Debian point release does.

The issue is that new point releases, at least in the case of
Debian, happens about every two months. These bumps are
integrated to master, backported to older branches, but it still
won't reach people working off releases immediately.

Even if Poky chooses not to adopt this, it would still be a nice
thing to have support for doing this internally for our
layer/distro.

-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] sanity.bbclass: support wildcards in SANITY_TESTED_DISTROS

[Olof Johansson]

With this change, you can use shell like globbing expressions (as
supported by Python's fnmatch) for entries in SANITY_TESTED_DISTROS.
This makes it possible to say that, e.g. all Debian 7 Wheezy releases
are supported with the entry Debian-7.*.

[YOCTO #5265]

Signed-off-by: Olof Johansson olof.johans...@axis.com
---
 meta/classes/sanity.bbclass | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index bae010d..d79db8f 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -246,6 +246,8 @@ def check_connectivity(d):
 return retval
 
 def check_supported_distro(sanity_data):
+from fnmatch import fnmatch
+
 tested_distros = sanity_data.getVar('SANITY_TESTED_DISTROS', True)
 if not tested_distros:
 return
@@ -255,12 +257,15 @@ def check_supported_distro(sanity_data):
 except Exception:
 distro = None
 
-if distro:
-if distro not in [x.strip() for x in tested_distros.split('\\n')]:
-bb.warn('Host distribution %s has not been validated with this 
version of the build system; you may possibly experience unexpected failures. 
It is recommended that you use a tested distribution.' % distro)
-else:
+if not distro:
 bb.warn('Host distribution could not be determined; you may possibly 
experience unexpected failures. It is recommended that you use a tested 
distribution.')
 
+for supported in [x.strip() for x in tested_distros.split('\\n')]:
+if fnmatch(distro, supported):
+return
+
+bb.warn('Host distribution %s has not been validated with this version 
of the build system; you may possibly experience unexpected failures. It is 
recommended that you use a tested distribution.' % distro)
+
 # Checks we should only make if MACHINE is set correctly
 def check_sanity_validmachine(sanity_data):
 messages = 
-- 
1.8.5.3

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [RFC] sanity.bbclass: support wildcards in SANITY_TESTED_DISTROS

[Olof Johansson]

On 14-02-28 16:28 +0100, Chris Larson wrote:
 Out of curiosity, why not just use fnmatch for this
 implementation?

No reason. That would probably be far superior. I'll send a
revised patch some time early next week! Thanks!

-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [RFC] sanity.bbclass: support wildcards in SANITY_TESTED_DISTROS

[Olof Johansson]

With this change, you can use wildcards for entries in
SANITY_TESTED_DISTROS. This makes it possible to say that, e.g. all
Debian 7 Wheezy releases are supported with the entry Debian-7.*.

[YOCTO #5265]

Signed-off-by: Olof Johansson olof.johans...@axis.com
---
 meta/classes/sanity.bbclass | 20 
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index bae010d..59f663b 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -245,6 +245,15 @@ def check_connectivity(d):
 
 return retval
 
+def _distro_match_glob(distro, supported):
+if not '*' in supported:
+return distro == supported
+
+bb.debug(2, Glob matching found for %s. Trying to match against %s % (
+supported, distro))
+prefix, sufix = supported.split('*', 1)
+return distro.startswith(prefix) and distro.endswith(sufix)
+
 def check_supported_distro(sanity_data):
 tested_distros = sanity_data.getVar('SANITY_TESTED_DISTROS', True)
 if not tested_distros:
@@ -255,12 +264,15 @@ def check_supported_distro(sanity_data):
 except Exception:
 distro = None
 
-if distro:
-if distro not in [x.strip() for x in tested_distros.split('\\n')]:
-bb.warn('Host distribution %s has not been validated with this 
version of the build system; you may possibly experience unexpected failures. 
It is recommended that you use a tested distribution.' % distro)
-else:
+if not distro:
 bb.warn('Host distribution could not be determined; you may possibly 
experience unexpected failures. It is recommended that you use a tested 
distribution.')
 
+for supported in [x.strip() for x in tested_distros.split('\\n')]:
+if _distro_match_glob(distro, supported):
+return
+
+bb.warn('Host distribution %s has not been validated with this version 
of the build system; you may possibly experience unexpected failures. It is 
recommended that you use a tested distribution.' % distro)
+
 # Checks we should only make if MACHINE is set correctly
 def check_sanity_validmachine(sanity_data):
 messages = 
-- 
1.8.5.3

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 1/1] e2fsprogs/populate-extfs.sh: fix a problem on dash

[Olof Johansson]

On 14-01-24 14:43 +0100, Richard Purdie wrote:
 On Mon, 2014-01-20 at 20:24 +0800, Robert Yang wrote:
  The dash can't handle the or [[ in parameter expansion, for example:

Good to know. Interesting! Noting that escaping the [ would work,
but not applicable in this case.

  
  A=/usr/bin/[[
  B=[[
  C=${A%$B}

Would ${A%/*}/ work?

-DIR=${DIR%$TGT}
+DIR=${DIR%/*}/

It also has the advantage that it doesn't cause a fork (afaict,
the loop is being executed for every file in a rootfs directory
structure, that could cause a lot of forks).

(Disclaimer: I haven't tested this)

-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 4/4] qemu: add bash and python to qemu's RDEPENDS

[Olof Johansson]

On 13-11-22 07:25 +0100, Hongxu Jia wrote:
 Hi Saul,
 
 The script could not be de-bashed,  it was made by create_wrapper
 which is bashism:
 Vim image/usr/bin/qemu-mips
 ...
 #! /bin/bash
 realpath=`readlink -fn $0`
 export QEMU_RESERVED_VA=0x0
 exec -a `dirname $realpath`/qemu-mips `dirname $realpath`/qemu-mips.real $@
 ...
 
 The exec's -a option is bashism, so we need to add bash to RDEPENDS.

1) Is the wrapper still necessary?

The comment above the do_install_append in the recipe:

# The following fragment will create a wrapper for qemu-mips user emulation
# binary in order to work around a segmentation fault issue. Basically, by
# default, the reserved virtual address space for 32-on-64 bit is set to 4GB.
# This will trigger a MMU access fault in the virtual CPU. With this change,
# the qemu-mips works fine.
# IMPORTANT: This piece needs to be removed once the root cause is fixed!

2) More generally, is -a generally necessary when exec'ing like this? Can't
   this be solved in another way?

-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] json-c: do rm -f on config.status before do_configure

[Olof Johansson]

This change adds -f when doing rm on config.status. .config.status is
not always present when doing do_configure, and that would without this
change lead to a fatal error.

Signed-off-by: Olof Johansson olof.johans...@axis.com
---
 meta/recipes-devtools/json-c/json-c_0.11.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/json-c/json-c_0.11.bb 
b/meta/recipes-devtools/json-c/json-c_0.11.bb
index 831f281..d1bf0e9 100644
--- a/meta/recipes-devtools/json-c/json-c_0.11.bb
+++ b/meta/recipes-devtools/json-c/json-c_0.11.bb
@@ -16,5 +16,5 @@ inherit autotools
 
 do_configure_prepend() {
 # Clean up autoconf cruft that should not be in the tarball
-rm ${S}/config.status
+rm -f ${S}/config.status
 }
-- 
1.8.4.rc3

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] Syslinux is compiling in do_install

[Olof Johansson]

Hi,

The syslinux recipe has the following do_compile:

do_compile() {
# Rebuild only the installer; keep precompiled bootloaders
# as per author's request (doc/distrib.txt)
oe_runmake CC=${CC} ${CFLAGS} LDFLAGS=${LDFLAGS} installer
}

i.e, it only builds the syslinux installer, as documented in
doc/distrib.txt. The issue with this though, is that it postpones
building some stuff (like com32, where I had an unrelated issue,
see [1]) to do_install. doc/distrib.txt doesn't say anything
about the install target.

I downloaded the tarball and built it outside of the oe
environment, first doing only `make installer` followed by
`make install`. The install targets results in a lot of building.
After this, in another tree, I simply did `make`, and then `make
install`, and the install target behaves as expected (no
compiling).

As I have very little (none at all, really) experience with
syslinux, I'm not sure if this is a bug in the recipe (should we
expected make install to work after only doing make installer?)
or an upstream bug with the installer target.

Any insights are welcome! :-)


1: https://bugzilla.yoctoproject.org/show_bug.cgi?id=5054
-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] flex: Only use create_wrapper for native and nativesdk

[Olof Johansson]

The create_wrapper functions of utils.bbclass cause implicit
dependencies on bash, which may not be suitable for deployment on
target. Besides, the wrapper doesn't seem to be necessary on target.

Signed-off-by: Olof Johansson olo...@axis.com
---
 meta/recipes-devtools/flex/flex.inc |6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/flex/flex.inc 
b/meta/recipes-devtools/flex/flex.inc
index 517db16..43f1dda 100644
--- a/meta/recipes-devtools/flex/flex.inc
+++ b/meta/recipes-devtools/flex/flex.inc
@@ -13,6 +13,10 @@ inherit autotools gettext
 M4 = ${bindir}/m4
 M4_class-native = ${STAGING_BINDIR_NATIVE}/m4
 
-do_install_append() {
+do_install_append_class-native() {
+   create_wrapper ${D}/${bindir}/flex M4=${M4}
+}
+
+do_install_append_class-nativesdk() {
create_wrapper ${D}/${bindir}/flex M4=${M4}
 }
-- 
1.7.10.4

___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] libdbm-sqlite-perl: add version 1.40

[Olof Johansson]

On 13-08-08 11:01 +0200, Hongxu Jia wrote:
  create mode 100644 meta/recipes-extended/perl/libdbm-sqlite-perl_1.40.bb

The recipe is called libdbm-sqlite-perl, but the perl module is
called DBD::SQLite. Is that intentional, and if so, why?

Regards,
-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] libdbm-sqlite-perl: add version 1.40

[Olof Johansson]

On 13-08-08 11:17 +0200, Hongxu Jia wrote:
 On 08/08/2013 05:06 PM, Olof Johansson wrote:
  The recipe is called libdbm-sqlite-perl, but the perl module is
  called DBD::SQLite. Is that intentional, and if so, why?

 I know what your mean, we should rename the recipe:
 s/libdbm-sqlite-perl_1.40.bb/libdbd-sqlite-perl_1.40.bb/

Yes, exactly. Sorry if I was unclear.

-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/2] libdbm-sqlite-perl: add version 1.40

[Olof Johansson]

On 13-08-08 13:35 +0200, Hongxu Jia wrote:
 Great, how about create a new layer named 'meta-perl', we sould do:
...
 - and than move other perl libraries already existed in oe-core
to that layer, it includes:
* meta/recipes-extended/perl/
* meta/recipes-devtools/perl/

I assume perl module recipes already in oecore are there for a
reason, and we should perhaps avoid moving them, and just let
meta-perl depend on oecore. Right?

But I like the idea of a meta-perl for my personal projects...
Would be happy to help!

-- 
olofjn
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core