Re: [OE-core] [PATCH v3 00/17] NPM refactoring

[Tim Orling]

> On Nov 20, 2019, at 10:27 PM, Tim Orling  
> wrote:
> 
> 
> 
>> On Nov 20, 2019, at 1:33 AM, Jean-Marie LEMETAYER 
>> > > wrote:
>> 
>> The current NPM support have several issues:
>> - The current NPM fetcher downloads the dependency tree but not the other
>>   fetchers. The 'subdir' parameter was used to fix this issue.
>> - They are multiple issues with package names (uppercase, exotic characters,
>>   scoped packages) even if they are inside the dependencies.
>> - The lockdown file generation have issues. When a package depends on
>>   multiple version of the same package (all versions have the same checksum).
>> 
>> This patchset refactors the NPM support in Yocto:
>> - As the NPM algorithm for dependency management is hard to handle, the new
>>   NPM fetcher downloads only the package source (and not the dependencies,
>>   like the other fetchers) (patch submitted in the bitbake-devel list).
>> - The NPM class handles the dependencies using NPM (and not manually).
>> - The NPM recipe creation is simplified to avoid issues.
>> - The lockdown file is no more used as it is no longer relevant compared to 
>> the
>>   latest shrinkwrap file format.
>> 
>> This patchset may remove some features (lockdown file, license management for
>> dependencies) but fixes the majority of the NPM issues. All of these issues
>> from the bugzilla.yoctoproject.org  are 
>> resolved by this patchset:
>> #10237, #10760, #11028, #11728, #11902, #12534
>> 
>> The fetcher and recipetool are now aware of a 'latest' keyword for the 
>> version
>> which allow to build the latest version available on the registry. This 
>> feature
>> fixes the two issues: #10515, #11029
>> 
>> Moreover the issue #13415 should also be fixed but I cannot test it.
>> 
>> I have tested the recipe creation and builds using a self made example to
>> generate build failures:
>> - https://github.com/savoirfairelinux/node-server-example 
>> 
>> - https://npmjs.com/package/@savoirfairelinux/node-server-example 
>> 
>> 
>> The test steps are these ones:
>>  $ source poky/oe-init-build-env
>>  $ bitbake-layers add-layer ../meta-openembedded/meta-oe
>>  $ devtool add 
>> "npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest
>>  
>> ”
> 
> I noticed this before but forgot to report it:
> 
> ERROR: Nothing PROVIDES 'c-ares-native' (but 
> virtual:native:/build/meta-openembedded/meta-oe/recipes-devtools/nodejs/nodejs_10.17.0.bb
>  DEPENDS on or otherwise requires it). ...
> http://layers.openembedded.org/layerindex/recipe/50318/ 
> 
> $ bitbake-layers add-layer ../meta-openembedded/meta-networking
> Which requires
> $ bitbake-layers add-layer ../meta-openembedded/meta-python
> 
> 
> Using tip of bitbake with V3, openembedded-core with V3 and meta-openembedded 
> master, somehow nodejs-native do_compile broke (at least on qemux86-64) since 
> the last time I built it shortly after ELC-E…
> So now I’ll have to git bisect that…more to come…

See https://errors.yoctoproject.org/Errors/Details/274939/ 

The same error I am seeing.

> 
> FWIW I had previously successfully run your instructions and the tests for 
> both bitbake and oe-selftest for V2 (distroless, e.g. not with poky). 
> Something unrelated has broken I suspect.
> 
>>  $ devtool build savoirfairelinux-node-server-example
>>  $ bitbake-layers create-layer ../meta-test
>>  $ bitbake-layers add-layer ../meta-test
>>  $ devtool finish savoirfairelinux-node-server-example ../meta-test
>>  $ echo IMAGE_INSTALL_append = '" savoirfairelinux-node-server-example"' >> 
>> conf/local.conf
>>  $ bitbake core-image-minimal
>> 
>> Also the 'devtool add' url could be one of these:
>> - 
>> npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest
>>  
>> 
>>   This url uses the new npm fetcher and request the latest version available 
>> on
>>   the registry.
>> - 
>> npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0
>>  
>> 
>>   This url uses the new npm fetcher and request a fixed version.
>> - git://github.com/savoirfairelinux/node-server-example.git;protocol=https 
>> 
>>   This url uses the git fetcher. As the dependencies are managed by the NPM
>>   class, any fetcher can be used.
>> 
>> When this patchset will be merged, I have planned to update the NPM wiki:
>>  https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM 
>> 
>> 
>> This patchset is also the base work for the full Angular support in Yocto 
>> that I
>> am preparing. These applications have a huge dependency tree which is ideal 
>> to
>> test the NPM support.
>> 
>> --- V2
>> 
>> - Add the 

[OE-core] [PATCH v2] wic: 'wic cp' to copy from image

[chee . yang . lee]

From: Chee Yang Lee 

currently 'wic cp' only works for copy file from local storage to
wic image.

enhance 'wic cp' to copy file/directory from wic image to local storage.

include selftest and 'wic help' updates.

[YOCTO#12169]

Signed-off-by: Chee Yang Lee 
---
 meta/lib/oeqa/selftest/cases/wic.py | 17 +
 scripts/lib/wic/engine.py   | 35 +++
 scripts/lib/wic/help.py | 29 +++--
 scripts/wic | 16 +---
 4 files changed, 76 insertions(+), 21 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index ea75300..3ed4cab 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -866,6 +866,13 @@ class Wic2(WicTestCase):
 self.assertEqual(8, len(result.output.split('\n')))
 self.assertTrue(os.path.basename(testdir) in result.output)
 
+# copy the file from the partition and check if it success
+dest = '%s-cp' % testfile.name
+runCmd("wic cp %s:1/%s %s -n %s" % (images[0],
+os.path.basename(testfile.name), dest, sysroot))
+self.assertTrue(os.path.exists(dest))
+
+
 def test_wic_rm(self):
 """Test removing files and directories from the the wic image."""
 runCmd("wic create mkefidisk "
@@ -1005,6 +1012,16 @@ class Wic2(WicTestCase):
 newdirs = set(line.split()[-1] for line in 
result.output.split('\n') if line)
 self.assertEqual(newdirs.difference(dirs), 
set([os.path.basename(testfile.name)]))
 
+# check if the file to copy is in the partition
+result = runCmd("wic ls %s:2/etc/ -n %s" % (images[0], sysroot))
+self.assertTrue('fstab' in [line.split()[-1] for line in 
result.output.split('\n') if line])
+
+# copy file from the partition, replace the temporary file content 
with it and
+# check for the file size to validate the copy
+runCmd("wic cp %s:2/etc/fstab %s -n %s" % (images[0], 
testfile.name, sysroot))
+self.assertTrue(os.stat(testfile.name).st_size > 0)
+
+
 def test_wic_rm_ext(self):
 """Test removing files from the ext partition."""
 runCmd("wic create mkefidisk "
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 18776fa..713219b 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -322,16 +322,31 @@ class Disk:
self._get_part_image(pnum),
path))
 
-def copy(self, src, pnum, path):
+def copy(self, src, dest):
 """Copy partition image into wic image."""
+pnum =  dest.part if isinstance(src, str) else src.part
+
 if self.partitions[pnum].fstype.startswith('ext'):
-cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\
-  format(path, src, os.path.basename(src),
+if isinstance(src, str):
+cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\
+  format(os.path.dirname(dest.path), src, 
os.path.basename(src),
  self.debugfs, self._get_part_image(pnum))
+else: # copy from wic
+# run both dump and rdump to support both files and directory
+cmd = "printf 'cd {}\ndump /{} {}\nrdump /{} {}\n' | {} {}".\
+  format(os.path.dirname(src.path), src.path,
+ dest, src.path, dest, self.debugfs,
+ self._get_part_image(pnum))
 else: # fat
-cmd = "{} -i {} -snop {} ::{}".format(self.mcopy,
+if isinstance(src, str):
+cmd = "{} -i {} -snop {} ::{}".format(self.mcopy,
+  self._get_part_image(pnum),
+  src, dest.path)
+else:
+cmd = "{} -i {} -snop ::{} {}".format(self.mcopy,
   self._get_part_image(pnum),
-  src, path)
+  src.path, dest)
+
 exec_cmd(cmd, as_shell=True)
 self._put_part_image(pnum)
 
@@ -523,11 +538,15 @@ def wic_ls(args, native_sysroot):
 
 def wic_cp(args, native_sysroot):
 """
-Copy local file or directory to the vfat partition of
+Copy file or directory to/from the vfat/ext partition of
 partitioned image.
 """
-disk = Disk(args.dest.image, native_sysroot)
-disk.copy(args.src, args.dest.part, args.dest.path)
+if isinstance(args.dest, str):
+disk = Disk(args.src.image, native_sysroot)
+else:
+disk = Disk(args.dest.image, native_sysroot)
+disk.copy(args.src, args.dest)
+
 
 def wic_rm(args, 

Re: [OE-core] [PATCH v3 00/17] NPM refactoring

[Tim Orling]

> On Nov 20, 2019, at 1:33 AM, Jean-Marie LEMETAYER 
>  wrote:
> 
> The current NPM support have several issues:
> - The current NPM fetcher downloads the dependency tree but not the other
>   fetchers. The 'subdir' parameter was used to fix this issue.
> - They are multiple issues with package names (uppercase, exotic characters,
>   scoped packages) even if they are inside the dependencies.
> - The lockdown file generation have issues. When a package depends on
>   multiple version of the same package (all versions have the same checksum).
> 
> This patchset refactors the NPM support in Yocto:
> - As the NPM algorithm for dependency management is hard to handle, the new
>   NPM fetcher downloads only the package source (and not the dependencies,
>   like the other fetchers) (patch submitted in the bitbake-devel list).
> - The NPM class handles the dependencies using NPM (and not manually).
> - The NPM recipe creation is simplified to avoid issues.
> - The lockdown file is no more used as it is no longer relevant compared to 
> the
>   latest shrinkwrap file format.
> 
> This patchset may remove some features (lockdown file, license management for
> dependencies) but fixes the majority of the NPM issues. All of these issues
> from the bugzilla.yoctoproject.org are resolved by this patchset:
> #10237, #10760, #11028, #11728, #11902, #12534
> 
> The fetcher and recipetool are now aware of a 'latest' keyword for the version
> which allow to build the latest version available on the registry. This 
> feature
> fixes the two issues: #10515, #11029
> 
> Moreover the issue #13415 should also be fixed but I cannot test it.
> 
> I have tested the recipe creation and builds using a self made example to
> generate build failures:
> - https://github.com/savoirfairelinux/node-server-example
> - https://npmjs.com/package/@savoirfairelinux/node-server-example
> 
> The test steps are these ones:
>  $ source poky/oe-init-build-env
>  $ bitbake-layers add-layer ../meta-openembedded/meta-oe
>  $ devtool add 
> "npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest”

I noticed this before but forgot to report it:

ERROR: Nothing PROVIDES 'c-ares-native' (but 
virtual:native:/build/meta-openembedded/meta-oe/recipes-devtools/nodejs/nodejs_10.17.0.bb
 DEPENDS on or otherwise requires it). ...
http://layers.openembedded.org/layerindex/recipe/50318/ 

$ bitbake-layers add-layer ../meta-openembedded/meta-networking
Which requires
$ bitbake-layers add-layer ../meta-openembedded/meta-python


Using tip of bitbake with V3, openembedded-core with V3 and meta-openembedded 
master, somehow nodejs-native do_compile broke (at least on qemux86-64) since 
the last time I built it shortly after ELC-E…
So now I’ll have to git bisect that…more to come…

FWIW I had previously successfully run your instructions and the tests for both 
bitbake and oe-selftest for V2 (distroless, e.g. not with poky). Something 
unrelated has broken I suspect.

>  $ devtool build savoirfairelinux-node-server-example
>  $ bitbake-layers create-layer ../meta-test
>  $ bitbake-layers add-layer ../meta-test
>  $ devtool finish savoirfairelinux-node-server-example ../meta-test
>  $ echo IMAGE_INSTALL_append = '" savoirfairelinux-node-server-example"' >> 
> conf/local.conf
>  $ bitbake core-image-minimal
> 
> Also the 'devtool add' url could be one of these:
> - 
> npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest
>   This url uses the new npm fetcher and request the latest version available 
> on
>   the registry.
> - 
> npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0
>   This url uses the new npm fetcher and request a fixed version.
> - git://github.com/savoirfairelinux/node-server-example.git;protocol=https
>   This url uses the git fetcher. As the dependencies are managed by the NPM
>   class, any fetcher can be used.
> 
> When this patchset will be merged, I have planned to update the NPM wiki:
>  https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM
> 
> This patchset is also the base work for the full Angular support in Yocto 
> that I
> am preparing. These applications have a huge dependency tree which is ideal to
> test the NPM support.
> 
> --- V2
> 
> - Add the 'check_network_access' function before each network access to check
>   for 'BB_NO_NETWORK' and 'BB_ALLOWED_NETWORKS' variables.
> 
> - Add a 'recipetool.RecipetoolTests.test_recipetool_create_npm' test case for
>   'oe-selftest' to test the npm recipe creation.
> 
> - Update the 'npm.bbclass' to fetch the dependencies in the 'do_fetch' task.
>   The dependencies are cached in a npm cache stored in '${DL_DIR}/npm_cache'
>   allowing the dependencies to be saved in the download directory and verify
>   on insertion and extraction [1]:
>  "All data that passes through the cache is fully verified
>   for integrity on both insertion and 

[OE-core] [PATCH] wic: 'wic cp' to copy from image

[chee . yang . lee]

From: Chee Yang Lee 

currently 'wic cp' only works for copy file from local storage to
wic image.

enhance 'wic cp' to copy file/directory from wic image to local storage.

include selftest and 'wic help' updates.

Signed-off-by: Chee Yang Lee 
---
 meta/lib/oeqa/selftest/cases/wic.py | 17 +
 scripts/lib/wic/engine.py   | 35 +++
 scripts/lib/wic/help.py | 29 +++--
 scripts/wic | 16 +---
 4 files changed, 76 insertions(+), 21 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py 
b/meta/lib/oeqa/selftest/cases/wic.py
index ea75300..3ed4cab 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -866,6 +866,13 @@ class Wic2(WicTestCase):
 self.assertEqual(8, len(result.output.split('\n')))
 self.assertTrue(os.path.basename(testdir) in result.output)
 
+# copy the file from the partition and check if it success
+dest = '%s-cp' % testfile.name
+runCmd("wic cp %s:1/%s %s -n %s" % (images[0],
+os.path.basename(testfile.name), dest, sysroot))
+self.assertTrue(os.path.exists(dest))
+
+
 def test_wic_rm(self):
 """Test removing files and directories from the the wic image."""
 runCmd("wic create mkefidisk "
@@ -1005,6 +1012,16 @@ class Wic2(WicTestCase):
 newdirs = set(line.split()[-1] for line in 
result.output.split('\n') if line)
 self.assertEqual(newdirs.difference(dirs), 
set([os.path.basename(testfile.name)]))
 
+# check if the file to copy is in the partition
+result = runCmd("wic ls %s:2/etc/ -n %s" % (images[0], sysroot))
+self.assertTrue('fstab' in [line.split()[-1] for line in 
result.output.split('\n') if line])
+
+# copy file from the partition, replace the temporary file content 
with it and
+# check for the file size to validate the copy
+runCmd("wic cp %s:2/etc/fstab %s -n %s" % (images[0], 
testfile.name, sysroot))
+self.assertTrue(os.stat(testfile.name).st_size > 0)
+
+
 def test_wic_rm_ext(self):
 """Test removing files from the ext partition."""
 runCmd("wic create mkefidisk "
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 18776fa..713219b 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -322,16 +322,31 @@ class Disk:
self._get_part_image(pnum),
path))
 
-def copy(self, src, pnum, path):
+def copy(self, src, dest):
 """Copy partition image into wic image."""
+pnum =  dest.part if isinstance(src, str) else src.part
+
 if self.partitions[pnum].fstype.startswith('ext'):
-cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\
-  format(path, src, os.path.basename(src),
+if isinstance(src, str):
+cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\
+  format(os.path.dirname(dest.path), src, 
os.path.basename(src),
  self.debugfs, self._get_part_image(pnum))
+else: # copy from wic
+# run both dump and rdump to support both files and directory
+cmd = "printf 'cd {}\ndump /{} {}\nrdump /{} {}\n' | {} {}".\
+  format(os.path.dirname(src.path), src.path,
+ dest, src.path, dest, self.debugfs,
+ self._get_part_image(pnum))
 else: # fat
-cmd = "{} -i {} -snop {} ::{}".format(self.mcopy,
+if isinstance(src, str):
+cmd = "{} -i {} -snop {} ::{}".format(self.mcopy,
+  self._get_part_image(pnum),
+  src, dest.path)
+else:
+cmd = "{} -i {} -snop ::{} {}".format(self.mcopy,
   self._get_part_image(pnum),
-  src, path)
+  src.path, dest)
+
 exec_cmd(cmd, as_shell=True)
 self._put_part_image(pnum)
 
@@ -523,11 +538,15 @@ def wic_ls(args, native_sysroot):
 
 def wic_cp(args, native_sysroot):
 """
-Copy local file or directory to the vfat partition of
+Copy file or directory to/from the vfat/ext partition of
 partitioned image.
 """
-disk = Disk(args.dest.image, native_sysroot)
-disk.copy(args.src, args.dest.part, args.dest.path)
+if isinstance(args.dest, str):
+disk = Disk(args.src.image, native_sysroot)
+else:
+disk = Disk(args.dest.image, native_sysroot)
+disk.copy(args.src, args.dest)
+
 
 def wic_rm(args, native_sysroot):
 

[OE-core] [PATCH] tiff: Refresh patch

[Zheng Ruoqin]

Refresh CVE-2019-7663.patch as it can't be applyed when using PATCHTOOL = 
"patch".

Signed-off-by: Zheng Ruoqin 
---
 .../libtiff/tiff/CVE-2019-7663.patch   | 71 --
 1 file changed, 26 insertions(+), 45 deletions(-)

diff --git a/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch 
b/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch
index f244fb2..94e4e33 100644
--- a/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch
+++ b/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch
@@ -1,22 +1,37 @@
-CVE: CVE-2019-7663
-Upstream-Status: Backport
-Signed-off-by: Ross Burton 
+CVE: CVE-2019-7663 
+Upstream-Status: Backport 
+Signed-off-by:
+Ross Burton 
 
 From c6fc6c1fa895024c86285c58efd6424cf8078f32 Mon Sep 17 00:00:00 2001
 From: Thomas Bernard 
 Date: Mon, 11 Feb 2019 10:05:33 +0100
 Subject: [PATCH 1/2] check that (Tile Width)*(Samples/Pixel) do no overflow
 
-fixes bug 2833
+From da6454aa80b9bb3154dfab4e8b21637de47531e0 Mon Sep 17 00:00:00 2001
+From: Thomas Bernard 
+Date: Mon, 11 Feb 2019 21:42:03 +0100
+Subject: [PATCH 2/2] tiffcp.c: use INT_MAX
+
+Signed-off-by: Zheng Ruoqin 
+Refresh this patch as it can't be applyed when using PATCHTOOL = "patch".
 ---
- tools/tiffcp.c | 8 +++-
- 1 file changed, 7 insertions(+), 1 deletion(-)
+ tools/tiffcp.c | 9 -
+ 1 file changed, 8 insertions(+), 1 deletion(-)
 
 diff --git a/tools/tiffcp.c b/tools/tiffcp.c
-index 2f406e2d..f0ee2c02 100644
+index 2f406e2..8c81aa4 100644
 --- a/tools/tiffcp.c
 +++ b/tools/tiffcp.c
-@@ -1408,7 +1408,7 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
+@@ -41,6 +41,7 @@
+ #include 
+ #include 
+ #include 
++#include 
+ 
+ #include 
+ 
+@@ -1408,7 +1409,7 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
int status = 1;
uint32 imagew = TIFFRasterScanlineSize(in);
uint32 tilew = TIFFTileRowSize(in);
@@ -25,11 +40,11 @@ index 2f406e2d..f0ee2c02 100644
tsize_t tilesize = TIFFTileSize(in);
tdata_t tilebuf;
uint8* bufp = (uint8*) buf;
-@@ -1416,6 +1416,12 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
+@@ -1416,6 +1417,12 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
uint32 row;
uint16 bps = 0, bytes_per_sample;
  
-+  if (spp > (0x7fff / tilew))
++  if (spp > (INT_MAX / tilew))
 +  {
 +  TIFFError(TIFFFileName(in), "Error, cannot handle that much 
samples per tile row (Tile Width * Samples/Pixel)");
 +  return 0;
@@ -39,39 +54,5 @@ index 2f406e2d..f0ee2c02 100644
if (tilebuf == 0)
return 0;
 -- 
-2.20.1
-
-
-From da6454aa80b9bb3154dfab4e8b21637de47531e0 Mon Sep 17 00:00:00 2001
-From: Thomas Bernard 
-Date: Mon, 11 Feb 2019 21:42:03 +0100
-Subject: [PATCH 2/2] tiffcp.c: use INT_MAX
-

- tools/tiffcp.c | 3 ++-
- 1 file changed, 2 insertions(+), 1 deletion(-)
-
-diff --git a/tools/tiffcp.c b/tools/tiffcp.c
-index f0ee2c02..8c81aa4f 100644
 a/tools/tiffcp.c
-+++ b/tools/tiffcp.c
-@@ -41,6 +41,7 @@
- #include 
- #include 
- #include 
-+#include 
- 
- #include 
- 
-@@ -1416,7 +1417,7 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
-   uint32 row;
-   uint16 bps = 0, bytes_per_sample;
- 
--  if (spp > (0x7fff / tilew))
-+  if (spp > (INT_MAX / tilew))
-   {
-   TIFFError(TIFFFileName(in), "Error, cannot handle that much 
samples per tile row (Tile Width * Samples/Pixel)");
-   return 0;
--- 
-2.20.1
+2.7.4
 
-- 
2.7.4



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [zeus][PATCH] tiff: Refresh patch

[Zheng Ruoqin]

Refresh CVE-2019-7663.patch as it can't be applyed when using PATCHTOOL = 
"patch".

Signed-off-by: Zheng Ruoqin 
---
 .../libtiff/tiff/CVE-2019-7663.patch   | 71 --
 1 file changed, 26 insertions(+), 45 deletions(-)

diff --git a/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch 
b/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch
index f244fb2..94e4e33 100644
--- a/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch
+++ b/meta/recipes-multimedia/libtiff/tiff/CVE-2019-7663.patch
@@ -1,22 +1,37 @@
-CVE: CVE-2019-7663
-Upstream-Status: Backport
-Signed-off-by: Ross Burton 
+CVE: CVE-2019-7663 
+Upstream-Status: Backport 
+Signed-off-by:
+Ross Burton 
 
 From c6fc6c1fa895024c86285c58efd6424cf8078f32 Mon Sep 17 00:00:00 2001
 From: Thomas Bernard 
 Date: Mon, 11 Feb 2019 10:05:33 +0100
 Subject: [PATCH 1/2] check that (Tile Width)*(Samples/Pixel) do no overflow
 
-fixes bug 2833
+From da6454aa80b9bb3154dfab4e8b21637de47531e0 Mon Sep 17 00:00:00 2001
+From: Thomas Bernard 
+Date: Mon, 11 Feb 2019 21:42:03 +0100
+Subject: [PATCH 2/2] tiffcp.c: use INT_MAX
+
+Signed-off-by: Zheng Ruoqin 
+Refresh this patch as it can't be applyed when using PATCHTOOL = "patch".
 ---
- tools/tiffcp.c | 8 +++-
- 1 file changed, 7 insertions(+), 1 deletion(-)
+ tools/tiffcp.c | 9 -
+ 1 file changed, 8 insertions(+), 1 deletion(-)
 
 diff --git a/tools/tiffcp.c b/tools/tiffcp.c
-index 2f406e2d..f0ee2c02 100644
+index 2f406e2..8c81aa4 100644
 --- a/tools/tiffcp.c
 +++ b/tools/tiffcp.c
-@@ -1408,7 +1408,7 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
+@@ -41,6 +41,7 @@
+ #include 
+ #include 
+ #include 
++#include 
+ 
+ #include 
+ 
+@@ -1408,7 +1409,7 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
int status = 1;
uint32 imagew = TIFFRasterScanlineSize(in);
uint32 tilew = TIFFTileRowSize(in);
@@ -25,11 +40,11 @@ index 2f406e2d..f0ee2c02 100644
tsize_t tilesize = TIFFTileSize(in);
tdata_t tilebuf;
uint8* bufp = (uint8*) buf;
-@@ -1416,6 +1416,12 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
+@@ -1416,6 +1417,12 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
uint32 row;
uint16 bps = 0, bytes_per_sample;
  
-+  if (spp > (0x7fff / tilew))
++  if (spp > (INT_MAX / tilew))
 +  {
 +  TIFFError(TIFFFileName(in), "Error, cannot handle that much 
samples per tile row (Tile Width * Samples/Pixel)");
 +  return 0;
@@ -39,39 +54,5 @@ index 2f406e2d..f0ee2c02 100644
if (tilebuf == 0)
return 0;
 -- 
-2.20.1
-
-
-From da6454aa80b9bb3154dfab4e8b21637de47531e0 Mon Sep 17 00:00:00 2001
-From: Thomas Bernard 
-Date: Mon, 11 Feb 2019 21:42:03 +0100
-Subject: [PATCH 2/2] tiffcp.c: use INT_MAX
-

- tools/tiffcp.c | 3 ++-
- 1 file changed, 2 insertions(+), 1 deletion(-)
-
-diff --git a/tools/tiffcp.c b/tools/tiffcp.c
-index f0ee2c02..8c81aa4f 100644
 a/tools/tiffcp.c
-+++ b/tools/tiffcp.c
-@@ -41,6 +41,7 @@
- #include 
- #include 
- #include 
-+#include 
- 
- #include 
- 
-@@ -1416,7 +1417,7 @@ DECLAREreadFunc(readSeparateTilesIntoBuffer)
-   uint32 row;
-   uint16 bps = 0, bytes_per_sample;
- 
--  if (spp > (0x7fff / tilew))
-+  if (spp > (INT_MAX / tilew))
-   {
-   TIFFError(TIFFFileName(in), "Error, cannot handle that much 
samples per tile row (Tile Width * Samples/Pixel)");
-   return 0;
--- 
-2.20.1
+2.7.4
 
-- 
2.7.4



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] ✗ patchtest: failure for busybox:upgrade 1.31.0 -> 1.31.1

[Patchwork]

== Series Details ==

Series: busybox:upgrade 1.31.0 -> 1.31.1
Revision: 1
URL   : https://patchwork.openembedded.org/series/21263/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patchbusybox:upgrade 1.31.0 -> 1.31.1
 Issue Patch is missing Signed-off-by [test_signed_off_by_presence] 
  Suggested fixSign off the patch (either manually or with "git commit 
--amend -s")



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines: 
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] ✗ patchtest: failure for busybox:upgrade 1.31.0 -> 1.31.1 (rev2)

[Patchwork]

== Series Details ==

Series: busybox:upgrade 1.31.0 -> 1.31.1 (rev2)
Revision: 2
URL   : https://patchwork.openembedded.org/series/21263/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patchcmake-native:upgrade 3.15.3 -> 3.15.5
 Issue Patch is missing Signed-off-by [test_signed_off_by_presence] 
  Suggested fixSign off the patch (either manually or with "git commit 
--amend -s")



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines: 
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] cmake-native:upgrade 3.15.3 -> 3.15.5

[Wang Mingyu]

---
 .../cmake/{cmake-native_3.15.3.bb => cmake-native_3.15.5.bb}  | 0
 meta/recipes-devtools/cmake/cmake.inc | 4 ++--
 2 files changed, 2 insertions(+), 2 deletions(-)
 rename meta/recipes-devtools/cmake/{cmake-native_3.15.3.bb => 
cmake-native_3.15.5.bb} (100%)

diff --git a/meta/recipes-devtools/cmake/cmake-native_3.15.3.bb 
b/meta/recipes-devtools/cmake/cmake-native_3.15.5.bb
similarity index 100%
rename from meta/recipes-devtools/cmake/cmake-native_3.15.3.bb
rename to meta/recipes-devtools/cmake/cmake-native_3.15.5.bb
diff --git a/meta/recipes-devtools/cmake/cmake.inc 
b/meta/recipes-devtools/cmake/cmake.inc
index d8a9e2d..24190de 100644
--- a/meta/recipes-devtools/cmake/cmake.inc
+++ b/meta/recipes-devtools/cmake/cmake.inc
@@ -22,7 +22,7 @@ SRC_URI = 
"https://cmake.org/files/v${CMAKE_MAJOR_VERSION}/cmake-${PV}.tar.gz \
file://0004-Fail-silently-if-system-Qt-installation-is-broken.patch 
\
 "
 
-SRC_URI[md5sum] = "bf69e837eeda3dd4e9f52cffcf546a11"
-SRC_URI[sha256sum] = 
"13958243a01365b05652fa01b21d40fa834f70a9e30efa69c02604e64f58b8f5"
+SRC_URI[md5sum] = "5fe3ebca627b4c3dcc2f127fbcfbceba"
+SRC_URI[sha256sum] = 
"fbdd7cef15c0ced06bb13024bfda0ecc0dedbc6b8a5d368c75255243beb4"
 
 UPSTREAM_CHECK_REGEX = "cmake-(?P\d+(\.\d+)+)\.tar"
-- 
2.7.4



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] busybox:upgrade 1.31.0 -> 1.31.1

[Wang Mingyu]

---
 meta/recipes-core/busybox/{busybox_1.31.0.bb => busybox_1.31.1.bb} | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
 rename meta/recipes-core/busybox/{busybox_1.31.0.bb => busybox_1.31.1.bb} (92%)

diff --git a/meta/recipes-core/busybox/busybox_1.31.0.bb 
b/meta/recipes-core/busybox/busybox_1.31.1.bb
similarity index 92%
rename from meta/recipes-core/busybox/busybox_1.31.0.bb
rename to meta/recipes-core/busybox/busybox_1.31.1.bb
index 34b1f2c..11e3aed 100644
--- a/meta/recipes-core/busybox/busybox_1.31.0.bb
+++ b/meta/recipes-core/busybox/busybox_1.31.1.bb
@@ -45,5 +45,5 @@ SRC_URI = 
"http://www.busybox.net/downloads/busybox-${PV}.tar.bz2;name=tarball \
 "
 SRC_URI_append_libc-musl = " file://musl.cfg "
 
-SRC_URI[tarball.md5sum] = "cdba5d4458f944ceec5cdcf7c4914b69"
-SRC_URI[tarball.sha256sum] = 
"0e4925392fd9f3743cc517e031b68b012b24a63b0cf6c1ff03cce7bb3846cc99"
+SRC_URI[tarball.md5sum] = "70913edaf2263a157393af07565c17f0"
+SRC_URI[tarball.sha256sum] = 
"d0f940a72f648943c1f2211e0e3117387c31d765137d92bd8284a3fb9752a998"
-- 
2.7.4



-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Andre McCurdy]

On Wed, Nov 20, 2019 at 2:41 PM Ryan Harkin  wrote:
> On Wed, 20 Nov 2019 at 21:29, Ryan Harkin  wrote:
>>
>> I pulled the whole openssl dir from your repo, added the layer.conf changes 
>> to my layer.conf and rebuilt openssl and my image.
>>
>> Unfortunately, I still have no /usr/bin/openssl in my disk image. So I've 
>> added the RPROVIDES from Andre's in a vain attempt to get it to work:
>>
>> RPROVIDES_${PN} += "openssl-bin"
>>
>> ... although I'm not hopeful it'll do the trick...
>
> It didn't work. Once thing that's puzzling me: where is the package 
> "openssl-bin"? I can only find references to it, but no package.

The "openssl-bin" package is created by the openssl 1.1.x recipe.

Adding "openssl-bin" to RPROVIDES in the openssl 1.0.2 recipe is a
solution for users who are switching from openssl 1.1.x back to 1.0.2
and have an image which is tries to include the new openssl-bin
package. I don't think that's what you are trying to do (?).

If you are using openssl 1.0.2 then the openssl command line tool is
in the openssl package... so to include the openssl command line tool,
add the "openssl" package to your image.

If you are using openssl 1.1.x then the openssl command line tool is
in the openssl-bin package... so to include the openssl command line
tool, add the "openssl-bin" package to your image.

But anyway, in all cases, the way to debug what's going on isn't to
try random recipe changes and then rebuild the final image. Instead
you should build your chosen version of openssl, look in the
packages-split directory to see which package includes the openssl
command line tool and then add that package to your image.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Adrian Bunk]

On Wed, Nov 20, 2019 at 11:56:38PM +0100, Alexander Kanavin wrote:
> On Wed, 20 Nov 2019 at 23:32, Phil Blundell  wrote:
> 
> > However, I think the point still stands that the commit message needs to
> > provide a better description of why the package is being removed.  If you
> > think it represents an ongoing maintenance headache that's already bad and
> > only going to get worse, and this now outweighs its usefulness, let's just
> > say that.  Not all old software is problematic, and not all problematic
> > software is old; the fact that the last release was 20 years ago is an
> > interesting fact but in isolation that doesn't represent a problem.
> > Indeed, compared to some other packages in oe-core, eight patches in
> > total over a 20-year period doesn't seem like that bad of an average.
> >
> 
> Fair enough, I wrote a hasty commit message. Mistakes happen.
> 
> Can I say what my problem is? Here goes: so far, no one in this discussion
> offered actual help with the actual issue. If you need this or that
> functionality from Yocto, please try to place help ahead of complaints and
> criticism.

No, your problem is your way of communication.

It is a very unfriendly way of communication to request help in the form 
patch aiming at immediate removal.

It comes across as if you were just too lazy to even try to fix 
something another patch you are submitting is breaking.

You would have received more constructive answers if you would have 
started with an email describing the non-trivial technical problem
where you need assistance.

> Alex

cu
Adrian
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Adrian Bunk]

On Wed, Nov 20, 2019 at 09:39:51PM +, mikko.rap...@bmw.de wrote:
>...
> I could submit these too if someone wants to setup a communit maintenance 
> branch for sumo.

I would not consider this appropriate for a stable branch. With such 
invasive changes it would no longer be reasonably safe for users to 
follow the branch to receive security updates for other recipes.

In Ubuntu 18.04 security support for OpenSSL 1.0.2 is provided until at 
least April 2023. Similar schedules exist for other LTS distributions.
This provides sources for piggy-backing security support for a few years
after upstream support ends.

> Cheers,
> 
> -Mikko

cu
Adrian
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 23:32, Phil Blundell  wrote:

> However, I think the point still stands that the commit message needs to
> provide a better description of why the package is being removed.  If you
> think it represents an ongoing maintenance headache that's already bad and
> only going to get worse, and this now outweighs its usefulness, let's just
> say that.  Not all old software is problematic, and not all problematic
> software is old; the fact that the last release was 20 years ago is an
> interesting fact but in isolation that doesn't represent a problem.
> Indeed, compared to some other packages in oe-core, eight patches in
> total over a 20-year period doesn't seem like that bad of an average.
>

Fair enough, I wrote a hasty commit message. Mistakes happen.

Can I say what my problem is? Here goes: so far, no one in this discussion
offered actual help with the actual issue. If you need this or that
functionality from Yocto, please try to place help ahead of complaints and
criticism.

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Ryan Harkin]

On Wed, 20 Nov 2019 at 21:29, Ryan Harkin  wrote:

>
>
> On Wed, 20 Nov 2019 at 19:09, Mark Hatle 
> wrote:
>
>>
>>
>> On 11/20/19 1:06 PM, Ryan Harkin wrote:
>> >
>> >
>> > On Wed, 20 Nov 2019 at 18:36, Mark Hatle <
>> mark.ha...@kernel.crashing.org
>> > > wrote:
>> >
>> >
>> >
>> > On 11/20/19 12:18 PM, Ryan Harkin wrote:
>> > > Hi all,
>> > >
>> > > I'm struggling with backporting OpenSSL to my Sumo build [1], so
>> wondered if
>> > > anyone else had done something similar with success.
>> > >
>> > > I copied "meta/recipes-connectivity/openssl" from Poky master
>> branch [2]
>> > into my
>> > > own layer [3]. It didn't pick up, so I discovered I needed to add
>> > > a PREFERRED_VERSION, eg:
>> > >
>> > > +PREFERRED_VERSION_openssl ?= "1.1.%"
>> > > +PREFERRED_VERSION_openssl-native ?= "1.1.%"
>> > > +PREFERRED_VERSION_nativesdk-openssl ?= "1.1.%"
>> > >
>> > > Now it builds fine. However, I no longer have /usr/bin/openssl in
>> my disk
>> > image.
>> > >
>> > > It doesn't appear in FILES_${PN}, and adding it to the recipes
>> doesn't seem to
>> > > make any difference.
>> > >
>> > > What am I missing?
>> > >
>> > > Thanks,
>> > > Ryan.
>> > >
>> > > [1] I'm looking for CVE fixes, 1.0.2p has a lot of CVEs.
>> >
>> > You know that 1.0.2 and 1.1 APIs are not compatible?  So you will
>> need to update
>> > everything that needs OpenSSL to understand the new API.
>> >
>> >
>> > So far, we're only using it in a shell script to sign an image and
>> later verify
>> > the image, so I've assumed, perhaps naively, that the API changes won't
>> matter...
>>
>> Correct, but there may be other components of the system that could be
>> using the
>> API that you are unaware of.  On a system as old as Sumo, you will need
>> to take
>> precautions to ensure that ONLY the 1.1x version is being used.  (There
>> may be
>> an openssl10 for compatibility that will need to be blacklisted.)
>>
>
> Good point. I'll check on it once I get it to work.
>
>
>>
>> >
>> > For CVE fixes, typically you would patch 1.0.2p, or update to the
>> latest
>> > (1.0.2t) as you go.  (If you have an OSV, this should be part of
>> the services
>> > that they offer you.)
>> >
>> >
>> > In my opinion, 1.0.2 will be around for at least another 4-5 years
>> due to the
>> > number of people actively using it in the world.  Until 1.1/3.0
>> (won't be a 2.0
>> > from what I read) exists and has a FIPS-140-2 support available --
>> people will
>> > continue to use 1.0.2 and maintain it as necessary for security.
>> >
>> > As an FYI:
>> http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/
>> >
>> > This version is for thud, warrior, zeus and master.  It is intended
>> to be
>> > maintained until either 1.0.2 is no longer maintainable -- or the
>> FIPS-140-2
>> > needs have been met by OpenSSL.
>> >
>> >
>> > Great, that looks like a better option anyway, assuming it has the
>> latest fixes
>> > I need, and doesn't give me the same build problem.  Thanks for
>> pointing it out.
>> > I'll give it a go.
>>
>> It's better to work with the Sumo version for your needs.  I just posted
>> that as
>> an example of openssl 1.0.2 being needed still by others, even as
>> oe-core/Yocto
>> Project have changed their defaults.
>>
>
> I pulled the whole openssl dir from your repo, added the layer.conf
> changes to my layer.conf and rebuilt openssl and my image.
>
> Unfortunately, I still have no /usr/bin/openssl in my disk image. So I've
> added the RPROVIDES from Andre's in a vain attempt to get it to work:
>
> RPROVIDES_${PN} += "openssl-bin"
>
> ... although I'm not hopeful it'll do the trick...
>

It didn't work. Once thing that's puzzling me: where is the package
"openssl-bin"? I can only find references to it, but no package.


>
>> --Mark
>>
>> > Thanks,
>> > Ryan.
>> >
>> >
>> >
>> > --Mark
>> >
>> > > [2] http://git.yoctoproject.org/git/poky
>> > > I'm at SHA a616ffebdc, so I copied openssl_1.1.1d.bb
>> >  
>> > > and all the other files in the directory.
>> > >
>> > > [3] I have a clone of Linaro's meta-backports. I'm trying to
>> generate a
>> > patch to
>> > > submit for review there.
>> > > https://git.linaro.org/openembedded/meta-backports.git
>> > >
>> >
>>
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Phil Blundell]

On Wed, Nov 20, 2019 at 11:12:37PM +0100, Alexander Kanavin wrote:
> The gettext problem is the immediate reason, but the broader reason is that
> the code is not maintained, and we should try to avoid burdening the
> project with unmaintained code. There's already 8 custom patches fixing
> issues caused by lack of upstream maintenance, and the gettext certainly
> won't be the last required fix. I have no interest in zmodem transfers,
> thus I opted for recipe removal.

I don't in general buy the line of argument that "I have no interest in 
X, and I want to make some change that happens to break X, ergo X should 
be removed", but in this particular case it probably is true that lrzsz 
has few enough users that it wouldn't qualify for inclusion in oe-core 
if someone were trying to add it nowadays.  So if it represents a net
maintenance burden then, although I do still use it myself from time to
time, I would be OK with it being removed from oe-core.

However, I think the point still stands that the commit message needs to
provide a better description of why the package is being removed.  If you
think it represents an ongoing maintenance headache that's already bad and
only going to get worse, and this now outweighs its usefulness, let's just
say that.  Not all old software is problematic, and not all problematic
software is old; the fact that the last release was 20 years ago is an
interesting fact but in isolation that doesn't represent a problem.
Indeed, compared to some other packages in oe-core, eight patches in
total over a 20-year period doesn't seem like that bad of an average.

p.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 22:38, Phil Blundell  wrote:

> If this is the actual reason for removing lrszsz, the commit message should
> say so.  As it stands, this patch provides no meaningful justification for
> why the package should be removed, it just contains a statement about the
> last release having been a long time ago (which may be true, but isn't
> obviously relevant) and some vague sneering about "proper networking"
> which doesn't seem to be helpful either.
>

The gettext problem is the immediate reason, but the broader reason is that
the code is not maintained, and we should try to avoid burdening the
project with unmaintained code. There's already 8 custom patches fixing
issues caused by lack of upstream maintenance, and the gettext certainly
won't be the last required fix. I have no interest in zmodem transfers,
thus I opted for recipe removal.

When the last release was is quite relevant from that perspective.

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Phil Blundell]

On Wed, Nov 20, 2019 at 09:00:28PM +0100, Alexander Kanavin wrote:
> Do you want to maintain it? Protocols are not changing, but the software
> that lrzsz depends on is changing.
> 
> Your first task would be making it build with the latest version of
> gettext, the branch containing that is here:

If this is the actual reason for removing lrszsz, the commit message should
say so.  As it stands, this patch provides no meaningful justification for
why the package should be removed, it just contains a statement about the
last release having been a long time ago (which may be true, but isn't
obviously relevant) and some vague sneering about "proper networking"
which doesn't seem to be helpful either.

p.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Mikko.Rapeli]

On Wed, Nov 20, 2019 at 06:18:05PM +, Ryan Harkin wrote:
> I'm struggling with backporting OpenSSL to my Sumo build [1], so wondered
> if anyone else had done something similar with success.

I've done it by backporting following changes to poky (sorry for subject only):

openssh: upgrade 7.6p1 -> 7.7p1
openssh: drop sshd support for DSA host keys
openssh: stop adding -D__FILE_OFFSET_BITS=64 to CFLAGS
openssh: drop RCONFLICTS for openssh-keygen
openssh: minor indent cleanup for sshd init script
openssh: sync local ssh_config + sshd_config files with upstream 7.7p1
openssh: only create sshd host keys which have been enabled
openssh: update from 7.7p1 to 7.8p1
openssh: upgrade 7.8p1 -> 7.8p1+git to support openssl 1.1.x
openssl-1.1: rework packaging
openssl-1.1: /etc/ssl location compatibility
openssl: minor reformatting to align the 1.0 and 1.1 recipes
openssl: move the libdir openssl.cnf symlink into the openssl package
openssl: fix path in nativesdk environment-setup script
openssl: drop obsolete no-afalgeng workaround for aarch64
openssl: fix hardcoded paths in native for openssl 1.1
openssl: remove dependency on relative_symlinks class
openssl: rename openssl 1.0.x to openssl10 and make openssl 1.1.x the default 
version
openssl: update to 1.1.1
openssl: do not tweak so names, use PRIVATE_LIBS instead
openssl: Handle -conf package file conflicts
openssl: rename PV to 1.1.1~pre9 to avoid future versions from going backwards
openssl_1.1.1: Fix Musl build by disabling async during configure
openssl: update to 1.1.1 final
openssl10: fix compile error for debian-mips64
openssl: skip ptest case `test_symbol_presence'
openssl: use deterministic perl Text::Template module bundled by openssl source
openssl: correct license comment
openssl: fix ptest
openssl: do an out-of-tree build
openssl: fix CVE-2018-0734 for both 1.0.2p and 1.1.1
openssl: fix CVE-2018-0735 for 1.1.1
openssl-1.1.1: remove build path from version info
openssl: update to 1.1.1a
openssl: correct bad path on package preprocess
python3{,-native}: backport openssl 1.1.1 compatibility changes
python3: fix openssl 1.1.1 changes
cryptodev-tests: port to openssl 1.1

Plus a patch to allow overriding openssl version in default-versions.inc,
and one hack to drop perl RDEPENDS from openssl-bin. This is still missing
the latest CVEs and letter releases.

Then meta-openembedded needed at least:

asio: Upgrade to 1.12.1
mailx: support openssl 1.1.x
cyrus-sasl: add UPSTREAM_CHECK_REGEX
cyrus-sasl: CLEANBROKEN = "1"
cyrus-sasl: Update to 2.1.27-rc7
cyrus-sasl: do not set CLEANBROKEN
cyrus-sasl: fix build out of source tree failed while configuring with 
`--enable-ldapdb'
cyrus-sasl: fix parallel build issue

I could submit these too if someone wants to setup a communit maintenance 
branch for sumo.

Cheers,

-Mikko
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Ryan Harkin]

On Wed, 20 Nov 2019 at 19:09, Mark Hatle 
wrote:

>
>
> On 11/20/19 1:06 PM, Ryan Harkin wrote:
> >
> >
> > On Wed, 20 Nov 2019 at 18:36, Mark Hatle  > > wrote:
> >
> >
> >
> > On 11/20/19 12:18 PM, Ryan Harkin wrote:
> > > Hi all,
> > >
> > > I'm struggling with backporting OpenSSL to my Sumo build [1], so
> wondered if
> > > anyone else had done something similar with success.
> > >
> > > I copied "meta/recipes-connectivity/openssl" from Poky master
> branch [2]
> > into my
> > > own layer [3]. It didn't pick up, so I discovered I needed to add
> > > a PREFERRED_VERSION, eg:
> > >
> > > +PREFERRED_VERSION_openssl ?= "1.1.%"
> > > +PREFERRED_VERSION_openssl-native ?= "1.1.%"
> > > +PREFERRED_VERSION_nativesdk-openssl ?= "1.1.%"
> > >
> > > Now it builds fine. However, I no longer have /usr/bin/openssl in
> my disk
> > image.
> > >
> > > It doesn't appear in FILES_${PN}, and adding it to the recipes
> doesn't seem to
> > > make any difference.
> > >
> > > What am I missing?
> > >
> > > Thanks,
> > > Ryan.
> > >
> > > [1] I'm looking for CVE fixes, 1.0.2p has a lot of CVEs.
> >
> > You know that 1.0.2 and 1.1 APIs are not compatible?  So you will
> need to update
> > everything that needs OpenSSL to understand the new API.
> >
> >
> > So far, we're only using it in a shell script to sign an image and later
> verify
> > the image, so I've assumed, perhaps naively, that the API changes won't
> matter...
>
> Correct, but there may be other components of the system that could be
> using the
> API that you are unaware of.  On a system as old as Sumo, you will need to
> take
> precautions to ensure that ONLY the 1.1x version is being used.  (There
> may be
> an openssl10 for compatibility that will need to be blacklisted.)
>

Good point. I'll check on it once I get it to work.


>
> >
> > For CVE fixes, typically you would patch 1.0.2p, or update to the
> latest
> > (1.0.2t) as you go.  (If you have an OSV, this should be part of the
> services
> > that they offer you.)
> >
> >
> > In my opinion, 1.0.2 will be around for at least another 4-5 years
> due to the
> > number of people actively using it in the world.  Until 1.1/3.0
> (won't be a 2.0
> > from what I read) exists and has a FIPS-140-2 support available --
> people will
> > continue to use 1.0.2 and maintain it as necessary for security.
> >
> > As an FYI:
> http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/
> >
> > This version is for thud, warrior, zeus and master.  It is intended
> to be
> > maintained until either 1.0.2 is no longer maintainable -- or the
> FIPS-140-2
> > needs have been met by OpenSSL.
> >
> >
> > Great, that looks like a better option anyway, assuming it has the
> latest fixes
> > I need, and doesn't give me the same build problem.  Thanks for pointing
> it out.
> > I'll give it a go.
>
> It's better to work with the Sumo version for your needs.  I just posted
> that as
> an example of openssl 1.0.2 being needed still by others, even as
> oe-core/Yocto
> Project have changed their defaults.
>

I pulled the whole openssl dir from your repo, added the layer.conf changes
to my layer.conf and rebuilt openssl and my image.

Unfortunately, I still have no /usr/bin/openssl in my disk image. So I've
added the RPROVIDES from Andre's in a vain attempt to get it to work:

RPROVIDES_${PN} += "openssl-bin"

... although I'm not hopeful it'll do the trick...


> --Mark
>
> > Thanks,
> > Ryan.
> >
> >
> >
> > --Mark
> >
> > > [2] http://git.yoctoproject.org/git/poky
> > > I'm at SHA a616ffebdc, so I copied openssl_1.1.1d.bb
> >  
> > > and all the other files in the directory.
> > >
> > > [3] I have a clone of Linaro's meta-backports. I'm trying to
> generate a
> > patch to
> > > submit for review there.
> > > https://git.linaro.org/openembedded/meta-backports.git
> > >
> >
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 21:17, Adrian Bunk  wrote:

> > Do you want to maintain it?
>
> Did you already contact the maintainer whose name your patch removes
> from maintainers.inc?
>

Most entries in oe-core's maintainers.inc do not carry any real
responsibility or commitment; they're there mostly so that monthly emails
from auto upgrade helper end up with someone. Actual maintenance comes from
those who use and need the components.

I can certainly ask Anuj, but I bet he has no interest in lrzsz whatsoever.

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Adrian Bunk]

On Wed, Nov 20, 2019 at 09:00:28PM +0100, Alexander Kanavin wrote:
> On Wed, 20 Nov 2019 at 20:49, Adrian Bunk  wrote:
> 
> > > Using serial lines for console work may have its uses,
> > > but for transferring files please set up proper networking.
> > >...
> >
> > I've had several recent embedded experiences where this was impossible,
> > and lrzsz was extremely useful.
> >
> 
> Do you want to maintain it?
>...

Did you already contact the maintainer whose name your patch removes 
from maintainers.inc?

> Alex

cu
Adrian
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 20:49, Adrian Bunk  wrote:

> > Using serial lines for console work may have its uses,
> > but for transferring files please set up proper networking.
> >...
>
> I've had several recent embedded experiences where this was impossible,
> and lrzsz was extremely useful.
>

Do you want to maintain it? Protocols are not changing, but the software
that lrzsz depends on is changing.

Your first task would be making it build with the latest version of
gettext, the branch containing that is here:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=akanavin/package-version-updates

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Andre McCurdy]

On Wed, Nov 20, 2019 at 11:44 AM Ryan Harkin  wrote:
>
> Hi Andre,
>
> On Wed, 20 Nov 2019 at 19:27, Andre McCurdy  wrote:
>>
>> On Wed, Nov 20, 2019 at 11:09 AM Mark Hatle
>>  wrote:
>> > On 11/20/19 1:06 PM, Ryan Harkin wrote:
>> > > On Wed, 20 Nov 2019 at 18:36, Mark Hatle > > > > wrote:
>> > >
>> > > You know that 1.0.2 and 1.1 APIs are not compatible?  So you will 
>> > > need to update
>> > > everything that needs OpenSSL to understand the new API.
>> > >
>> > >
>> > > So far, we're only using it in a shell script to sign an image and later 
>> > > verify
>> > > the image, so I've assumed, perhaps naively, that the API changes won't 
>> > > matter...
>> >
>> > Correct, but there may be other components of the system that could be 
>> > using the
>> > API that you are unaware of.  On a system as old as Sumo, you will need to 
>> > take
>> > precautions to ensure that ONLY the 1.1x version is being used.  (There 
>> > may be
>> > an openssl10 for compatibility that will need to be blacklisted.)
>> >
>> > > For CVE fixes, typically you would patch 1.0.2p, or update to the 
>> > > latest
>> > > (1.0.2t) as you go.  (If you have an OSV, this should be part of the 
>> > > services
>> > > that they offer you.)
>> > >
>> > >
>> > > In my opinion, 1.0.2 will be around for at least another 4-5 years 
>> > > due to the
>> > > number of people actively using it in the world.  Until 1.1/3.0 
>> > > (won't be a 2.0
>> > > from what I read) exists and has a FIPS-140-2 support available -- 
>> > > people will
>> > > continue to use 1.0.2 and maintain it as necessary for security.
>> > >
>> > > As an FYI:  
>> > > http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/
>> > >
>> > > This version is for thud, warrior, zeus and master.  It is intended 
>> > > to be
>> > > maintained until either 1.0.2 is no longer maintainable -- or the 
>> > > FIPS-140-2
>> > > needs have been met by OpenSSL.
>> > >
>> > >
>> > > Great, that looks like a better option anyway, assuming it has the 
>> > > latest fixes
>> > > I need, and doesn't give me the same build problem.  Thanks for pointing 
>> > > it out.
>> > > I'll give it a go.
>> >
>> > It's better to work with the Sumo version for your needs.  I just posted 
>> > that as
>> > an example of openssl 1.0.2 being needed still by others, even as 
>> > oe-core/Yocto
>> > Project have changed their defaults.
>>
>> If you want an up to date openssl 1.0.2 recipe which is compatible
>> with Sumo, you can find one here:
>>
>>   https://github.com/armcc/meta-plumewifi
>>
>> I'm only actively testing it with OE 1.6 (Daisy) and OE 2.7 (Warrior)
>> but it should work for all versions in between (and if it doesn't I'll
>> accept patches or try to fix it).
>
>
> Thanks! It looks similar to the tree Mark Hatle pointed out to me. Two diffs 
> jump out:
>
> - Your repo adds the RPROVIDES for openssl-bin to "Be compatible with the 
> openssl 1.1.x recipe".

Yes. Makes the transition between 1.0.2 and 1.1.x a little easier.

> - Mark's repo has two extra patches:
>file://0001-Fix-BN_LLONG-breakage.patch \
>file://0001-Fix-DES_LONG-breakage.patch \

Those patches are in my repo too - but only in the master-next branch.
They are not required for Sumo. (Since some might regard those patches
as a little "dubious" I don't pull them in unless they're necessary).

> Regards,
> Ryan.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 08/11] lrzsz: remove the recipe

[Adrian Bunk]

On Wed, Nov 20, 2019 at 02:44:52PM +0100, Alexander Kanavin wrote:
> This is 2019; last release of lrzsz was in 1998.

The protocols are not changing.

> Using serial lines for console work may have its uses,
> but for transferring files please set up proper networking.
>...

I've had several recent embedded experiences where this was impossible, 
and lrzsz was extremely useful.

cu
Adrian
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Ryan Harkin]

Hi Andre,

On Wed, 20 Nov 2019 at 19:27, Andre McCurdy  wrote:

> On Wed, Nov 20, 2019 at 11:09 AM Mark Hatle
>  wrote:
> > On 11/20/19 1:06 PM, Ryan Harkin wrote:
> > > On Wed, 20 Nov 2019 at 18:36, Mark Hatle <
> mark.ha...@kernel.crashing.org
> > > > wrote:
> > >
> > > You know that 1.0.2 and 1.1 APIs are not compatible?  So you will
> need to update
> > > everything that needs OpenSSL to understand the new API.
> > >
> > >
> > > So far, we're only using it in a shell script to sign an image and
> later verify
> > > the image, so I've assumed, perhaps naively, that the API changes
> won't matter...
> >
> > Correct, but there may be other components of the system that could be
> using the
> > API that you are unaware of.  On a system as old as Sumo, you will need
> to take
> > precautions to ensure that ONLY the 1.1x version is being used.  (There
> may be
> > an openssl10 for compatibility that will need to be blacklisted.)
> >
> > > For CVE fixes, typically you would patch 1.0.2p, or update to the
> latest
> > > (1.0.2t) as you go.  (If you have an OSV, this should be part of
> the services
> > > that they offer you.)
> > >
> > >
> > > In my opinion, 1.0.2 will be around for at least another 4-5 years
> due to the
> > > number of people actively using it in the world.  Until 1.1/3.0
> (won't be a 2.0
> > > from what I read) exists and has a FIPS-140-2 support available --
> people will
> > > continue to use 1.0.2 and maintain it as necessary for security.
> > >
> > > As an FYI:
> http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/
> > >
> > > This version is for thud, warrior, zeus and master.  It is
> intended to be
> > > maintained until either 1.0.2 is no longer maintainable -- or the
> FIPS-140-2
> > > needs have been met by OpenSSL.
> > >
> > >
> > > Great, that looks like a better option anyway, assuming it has the
> latest fixes
> > > I need, and doesn't give me the same build problem.  Thanks for
> pointing it out.
> > > I'll give it a go.
> >
> > It's better to work with the Sumo version for your needs.  I just posted
> that as
> > an example of openssl 1.0.2 being needed still by others, even as
> oe-core/Yocto
> > Project have changed their defaults.
>
> If you want an up to date openssl 1.0.2 recipe which is compatible
> with Sumo, you can find one here:
>
>   https://github.com/armcc/meta-plumewifi
>
> I'm only actively testing it with OE 1.6 (Daisy) and OE 2.7 (Warrior)
> but it should work for all versions in between (and if it doesn't I'll
> accept patches or try to fix it).
>

Thanks! It looks similar to the tree Mark Hatle pointed out to me. Two
diffs jump out:

- Your repo adds the RPROVIDES for openssl-bin to "Be compatible with the
openssl 1.1.x recipe".
- Mark's repo has two extra patches:
   file://0001-Fix-BN_LLONG-breakage.patch \
   file://0001-Fix-DES_LONG-breakage.patch \

Regards,
Ryan.
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Andre McCurdy]

On Wed, Nov 20, 2019 at 11:09 AM Mark Hatle
 wrote:
> On 11/20/19 1:06 PM, Ryan Harkin wrote:
> > On Wed, 20 Nov 2019 at 18:36, Mark Hatle  > > wrote:
> >
> > You know that 1.0.2 and 1.1 APIs are not compatible?  So you will need 
> > to update
> > everything that needs OpenSSL to understand the new API.
> >
> >
> > So far, we're only using it in a shell script to sign an image and later 
> > verify
> > the image, so I've assumed, perhaps naively, that the API changes won't 
> > matter...
>
> Correct, but there may be other components of the system that could be using 
> the
> API that you are unaware of.  On a system as old as Sumo, you will need to 
> take
> precautions to ensure that ONLY the 1.1x version is being used.  (There may be
> an openssl10 for compatibility that will need to be blacklisted.)
>
> > For CVE fixes, typically you would patch 1.0.2p, or update to the latest
> > (1.0.2t) as you go.  (If you have an OSV, this should be part of the 
> > services
> > that they offer you.)
> >
> >
> > In my opinion, 1.0.2 will be around for at least another 4-5 years due 
> > to the
> > number of people actively using it in the world.  Until 1.1/3.0 (won't 
> > be a 2.0
> > from what I read) exists and has a FIPS-140-2 support available -- 
> > people will
> > continue to use 1.0.2 and maintain it as necessary for security.
> >
> > As an FYI:  http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/
> >
> > This version is for thud, warrior, zeus and master.  It is intended to 
> > be
> > maintained until either 1.0.2 is no longer maintainable -- or the 
> > FIPS-140-2
> > needs have been met by OpenSSL.
> >
> >
> > Great, that looks like a better option anyway, assuming it has the latest 
> > fixes
> > I need, and doesn't give me the same build problem.  Thanks for pointing it 
> > out.
> > I'll give it a go.
>
> It's better to work with the Sumo version for your needs.  I just posted that 
> as
> an example of openssl 1.0.2 being needed still by others, even as 
> oe-core/Yocto
> Project have changed their defaults.

If you want an up to date openssl 1.0.2 recipe which is compatible
with Sumo, you can find one here:

  https://github.com/armcc/meta-plumewifi

I'm only actively testing it with OE 1.6 (Daisy) and OE 2.7 (Warrior)
but it should work for all versions in between (and if it doesn't I'll
accept patches or try to fix it).
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] u-boot.inc: Fix devtool build u-boot for u-boot without menuconfig

[Tom Hochstein]

For u-boot recipes without menuconfig support, running devtool results
in a do_configure error:

cp: cannot stat 
'/home/r60874/upstream/fsl-xwayland/tmp/work/imx8mmevk-fsl-linux/u-boot-imx/2018.03-r0/u-boot-imx-2018.03//.config':
 No such file or directory

The problem arises because u-boot.inc supports recipes with and without
menuconfig.

Fix the problem by properly setting DEVTOOL_DISABLE_MENUCONFIG so that devtool
can control logic that applies only for menuconfig support.

Signed-off-by: Tom Hochstein 
---
 meta/recipes-bsp/u-boot/u-boot.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-bsp/u-boot/u-boot.inc 
b/meta/recipes-bsp/u-boot/u-boot.inc
index 9a754fd09b..d241347bf7 100644
--- a/meta/recipes-bsp/u-boot/u-boot.inc
+++ b/meta/recipes-bsp/u-boot/u-boot.inc
@@ -87,6 +87,8 @@ do_configure () {
 fi
 merge_config.sh -m .config ${@" ".join(find_cfgs(d))}
 cml1_do_configure
+else
+DEVTOOL_DISABLE_MENUCONFIG=true
 fi
 }
 
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] devtool/standard.py: Allow recipe to disable menuconfig logic

[Tom Hochstein]

u-boot.inc supports u-boot recipes with or without menuconfig [1].
However, running devtool on a u-boot recipe that does not support menuconfig
results in an error:

cp: cannot stat 
'/home/r60874/upstream/fsl-xwayland/tmp/work/imx8mmevk-fsl-linux/u-boot-imx/2018.03-r0/u-boot-imx-2018.03//.config':
 No such file or directory

The problem is the devtool logic assumes that any recipe with a do_menuconfig 
task
will generate a .config in do_configure().

Fix the problem by removing the assumption with a flag that the recipe can 
control,
like this:

do_configure() {
if [ menuconfig-supported ]; then
...
else
DEVTOOL_DISABLE_MENUCONFIG=true
fi
}

[1] 
https://github.com/openembedded/openembedded-core/commit/11278e3b2c75be80645b9841763a97dbb35daadc

Signed-off-by: Tom Hochstein 
---
 scripts/lib/devtool/standard.py | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 8d9c1a3022..66bd1415c3 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -940,8 +940,10 @@ def modify(args, config, basepath, workspace):
 '}\n')
 if rd.getVarFlag('do_menuconfig','task'):
 f.write('\ndo_configure_append() {\n'
-'cp ${B}/.config ${S}/.config.baseline\n'
-'ln -sfT ${B}/.config ${S}/.config.new\n'
+'if [ ! ${DEVTOOL_DISABLE_MENUCONFIG} ]; then\n'
+'cp ${B}/.config ${S}/.config.baseline\n'
+'ln -sfT ${B}/.config ${S}/.config.new\n'
+'fi\n'
 '}\n')
 if initial_rev:
 f.write('\n# initial_rev: %s\n' % initial_rev)
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Mark Hatle]

On 11/20/19 1:06 PM, Ryan Harkin wrote:
> 
> 
> On Wed, 20 Nov 2019 at 18:36, Mark Hatle  > wrote:
> 
> 
> 
> On 11/20/19 12:18 PM, Ryan Harkin wrote:
> > Hi all,
> >
> > I'm struggling with backporting OpenSSL to my Sumo build [1], so 
> wondered if
> > anyone else had done something similar with success.
> >
> > I copied "meta/recipes-connectivity/openssl" from Poky master branch [2]
> into my
> > own layer [3]. It didn't pick up, so I discovered I needed to add
> > a PREFERRED_VERSION, eg:
> >
> > +PREFERRED_VERSION_openssl ?= "1.1.%"
> > +PREFERRED_VERSION_openssl-native ?= "1.1.%"
> > +PREFERRED_VERSION_nativesdk-openssl ?= "1.1.%"
> >
> > Now it builds fine. However, I no longer have /usr/bin/openssl in my 
> disk
> image.
> >
> > It doesn't appear in FILES_${PN}, and adding it to the recipes doesn't 
> seem to
> > make any difference.
> >
> > What am I missing?
> >
> > Thanks,
> > Ryan.
> >
> > [1] I'm looking for CVE fixes, 1.0.2p has a lot of CVEs.
> 
> You know that 1.0.2 and 1.1 APIs are not compatible?  So you will need to 
> update
> everything that needs OpenSSL to understand the new API.
> 
> 
> So far, we're only using it in a shell script to sign an image and later 
> verify
> the image, so I've assumed, perhaps naively, that the API changes won't 
> matter...

Correct, but there may be other components of the system that could be using the
API that you are unaware of.  On a system as old as Sumo, you will need to take
precautions to ensure that ONLY the 1.1x version is being used.  (There may be
an openssl10 for compatibility that will need to be blacklisted.)

> 
> For CVE fixes, typically you would patch 1.0.2p, or update to the latest
> (1.0.2t) as you go.  (If you have an OSV, this should be part of the 
> services
> that they offer you.) 
> 
> 
> In my opinion, 1.0.2 will be around for at least another 4-5 years due to 
> the
> number of people actively using it in the world.  Until 1.1/3.0 (won't be 
> a 2.0
> from what I read) exists and has a FIPS-140-2 support available -- people 
> will
> continue to use 1.0.2 and maintain it as necessary for security.
> 
> As an FYI:  http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/
> 
> This version is for thud, warrior, zeus and master.  It is intended to be
> maintained until either 1.0.2 is no longer maintainable -- or the 
> FIPS-140-2
> needs have been met by OpenSSL.
> 
> 
> Great, that looks like a better option anyway, assuming it has the latest 
> fixes
> I need, and doesn't give me the same build problem.  Thanks for pointing it 
> out.
> I'll give it a go.

It's better to work with the Sumo version for your needs.  I just posted that as
an example of openssl 1.0.2 being needed still by others, even as oe-core/Yocto
Project have changed their defaults.

--Mark

> Thanks,
> Ryan.
>  
> 
> 
> --Mark
> 
> > [2] http://git.yoctoproject.org/git/poky
> > I'm at SHA a616ffebdc, so I copied openssl_1.1.1d.bb
>  
> > and all the other files in the directory.
> >
> > [3] I have a clone of Linaro's meta-backports. I'm trying to generate a
> patch to
> > submit for review there.
> > https://git.linaro.org/openembedded/meta-backports.git
> >
> 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Ryan Harkin]

On Wed, 20 Nov 2019 at 18:36, Mark Hatle 
wrote:

>
>
> On 11/20/19 12:18 PM, Ryan Harkin wrote:
> > Hi all,
> >
> > I'm struggling with backporting OpenSSL to my Sumo build [1], so
> wondered if
> > anyone else had done something similar with success.
> >
> > I copied "meta/recipes-connectivity/openssl" from Poky master branch [2]
> into my
> > own layer [3]. It didn't pick up, so I discovered I needed to add
> > a PREFERRED_VERSION, eg:
> >
> > +PREFERRED_VERSION_openssl ?= "1.1.%"
> > +PREFERRED_VERSION_openssl-native ?= "1.1.%"
> > +PREFERRED_VERSION_nativesdk-openssl ?= "1.1.%"
> >
> > Now it builds fine. However, I no longer have /usr/bin/openssl in my
> disk image.
> >
> > It doesn't appear in FILES_${PN}, and adding it to the recipes doesn't
> seem to
> > make any difference.
> >
> > What am I missing?
> >
> > Thanks,
> > Ryan.
> >
> > [1] I'm looking for CVE fixes, 1.0.2p has a lot of CVEs.
>
> You know that 1.0.2 and 1.1 APIs are not compatible?  So you will need to
> update
> everything that needs OpenSSL to understand the new API.
>

So far, we're only using it in a shell script to sign an image and later
verify the image, so I've assumed, perhaps naively, that the API changes
won't matter...

>
> For CVE fixes, typically you would patch 1.0.2p, or update to the latest
> (1.0.2t) as you go.  (If you have an OSV, this should be part of the
> services
> that they offer you.)


> In my opinion, 1.0.2 will be around for at least another 4-5 years due to
> the
> number of people actively using it in the world.  Until 1.1/3.0 (won't be
> a 2.0
> from what I read) exists and has a FIPS-140-2 support available -- people
> will
> continue to use 1.0.2 and maintain it as necessary for security.
>
> As an FYI:  http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/
>
> This version is for thud, warrior, zeus and master.  It is intended to be
> maintained until either 1.0.2 is no longer maintainable -- or the
> FIPS-140-2
> needs have been met by OpenSSL.
>

Great, that looks like a better option anyway, assuming it has the latest
fixes I need, and doesn't give me the same build problem.  Thanks for
pointing it out. I'll give it a go.

Thanks,
Ryan.


>
> --Mark
>
> > [2] http://git.yoctoproject.org/git/poky
> > I'm at SHA a616ffebdc, so I copied openssl_1.1.1d.bb <
> http://openssl_1.1.1d.bb>
> > and all the other files in the directory.
> >
> > [3] I have a clone of Linaro's meta-backports. I'm trying to generate a
> patch to
> > submit for review there.
> > https://git.linaro.org/openembedded/meta-backports.git
> >
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Ryan Harkin]

On Wed, 20 Nov 2019 at 18:28, Ross Burton  wrote:

> On 20/11/2019 18:18, Ryan Harkin wrote:
> > Now it builds fine. However, I no longer have /usr/bin/openssl in my
> > disk image.
>
> $ oe-pkgdata-util find-path /usr/bin/openssl
> openssl-bin: /usr/bin/openssl
>
> Did you try adding openssl-bin to your image explicitly?
>

No, I hadn't even thought about that, as it was already working before I
updated.

I added openssl-bin to my RDEPENDS for my packagegroup, right after the
openssl entry. I tried to manually bitbake openssl-bin, and it complained:

$ bitbake openssl-bin
[snip]
ERROR: Nothing PROVIDES 'openssl-bin'. Close matches:
  openssl
  openssl-native
  openssl10
  openssl RPROVIDES openssl-bin

So I guess that isn't the problem?


>
> Ross
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 5/5] grub: don't autoreconf twice

[Ross Burton]

do_configure() essentially calls autogen.sh to generate some sources and then
autoreconf, but autogen.sh also calls autoreconf.

Pass a magic variable so that autogen.sh doesn't autoreconf for us.

Signed-off-by: Ross Burton 
---
 meta/recipes-bsp/grub/grub2.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-bsp/grub/grub2.inc b/meta/recipes-bsp/grub/grub2.inc
index 7138e4df872..b3291cb4b80 100644
--- a/meta/recipes-bsp/grub/grub2.inc
+++ b/meta/recipes-bsp/grub/grub2.inc
@@ -60,8 +60,9 @@ BUILD_LDFLAGS = ""
 export PYTHON = "python3"
 
 do_configure_prepend() {
-   ( cd ${S}
-   ${S}/autogen.sh )
+   cd ${S}
+   FROM_BOOTSTRAP=1 ${S}/autogen.sh
+   cd ${B}
 }
 
 RDEPENDS_${PN}_class-native = ""
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/5] packagegroup-core-sdk: remove intltool

[Ross Burton]

Intltool is deprecated these days, as gettext can handle almost everything
intltool could.  Remove it from the SDK packagegroups, if it is needed then the
user can add it explicitly.

Signed-off-by: Ross Burton 
---
 meta/recipes-core/packagegroups/packagegroup-core-sdk.bb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb 
b/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb
index 8854ec4801f..03fa5296c15 100644
--- a/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb
+++ b/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb
@@ -16,7 +16,6 @@ RDEPENDS_packagegroup-core-sdk = "\
 coreutils \
 ccache \
 diffutils \
-intltool \
 perl-module-re \
 perl-module-text-wrap \
 findutils \
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 4/5] grub: inherit pkgconfig

[Ross Burton]

Grub uses pkg-config but is only in the sysroot via dependencies.  Be explicit
and inherit pkgconfig directly.

Signed-off-by: Ross Burton 
---
 meta/recipes-bsp/grub/grub2.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-bsp/grub/grub2.inc b/meta/recipes-bsp/grub/grub2.inc
index c81def2a170..7138e4df872 100644
--- a/meta/recipes-bsp/grub/grub2.inc
+++ b/meta/recipes-bsp/grub/grub2.inc
@@ -33,7 +33,7 @@ GRUBPLATFORM_arm = "efi"
 GRUBPLATFORM_aarch64 = "efi"
 GRUBPLATFORM ??= "pc"
 
-inherit autotools gettext texinfo
+inherit autotools gettext texinfo pkgconfig
 
 EXTRA_OECONF = "--with-platform=${GRUBPLATFORM} \
 --disable-grub-mkfont \
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 3/5] parted: fix whitespace

[Ross Burton]

Signed-off-by: Ross Burton 
---
 meta/recipes-extended/parted/parted_3.2.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-extended/parted/parted_3.2.bb 
b/meta/recipes-extended/parted/parted_3.2.bb
index 21a815345a9..fb5fa33a8ad 100644
--- a/meta/recipes-extended/parted/parted_3.2.bb
+++ b/meta/recipes-extended/parted/parted_3.2.bb
@@ -14,7 +14,7 @@ SRC_URI = "${GNU_MIRROR}/parted/parted-${PV}.tar.xz \
file://0001-Unset-need_charset_alias-when-building-for-musl.patch \

file://0002-libparted_fs_resize-link-against-libuuid-explicitly-.patch \

file://0001-Move-python-helper-scripts-used-only-in-tests-to-Pyt.patch \
-  file://0001-linux-Include-sys-sysmacros.h-for-major-macro.patch \
+   file://0001-linux-Include-sys-sysmacros.h-for-major-macro.patch \
file://run-ptest \
file://Makefile \

file://0001-libparted-Use-read-only-when-probing-devices-on-linu.patch \
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/5] packagegroup-core-self-hosted: remove intltool

[Ross Burton]

Very little software needs intltool to build, and we don't need it on the host
to build Poky.

Signed-off-by: Ross Burton 
---
 meta/recipes-core/packagegroups/packagegroup-self-hosted.bb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb 
b/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
index ee9d0636f2f..1fcc142cacf 100644
--- a/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
+++ b/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
@@ -75,7 +75,6 @@ RDEPENDS_packagegroup-self-hosted-sdk = "\
 g++-symlinks \
 gcc \
 gcc-symlinks \
-intltool \
 ldd \
 less \
 libstdc++ \
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Mark Hatle]

On 11/20/19 12:18 PM, Ryan Harkin wrote:
> Hi all,
> 
> I'm struggling with backporting OpenSSL to my Sumo build [1], so wondered if
> anyone else had done something similar with success.
> 
> I copied "meta/recipes-connectivity/openssl" from Poky master branch [2] into 
> my
> own layer [3]. It didn't pick up, so I discovered I needed to add
> a PREFERRED_VERSION, eg:
> 
> +PREFERRED_VERSION_openssl ?= "1.1.%"
> +PREFERRED_VERSION_openssl-native ?= "1.1.%"
> +PREFERRED_VERSION_nativesdk-openssl ?= "1.1.%"
> 
> Now it builds fine. However, I no longer have /usr/bin/openssl in my disk 
> image.
> 
> It doesn't appear in FILES_${PN}, and adding it to the recipes doesn't seem to
> make any difference.
> 
> What am I missing?
> 
> Thanks,
> Ryan.
> 
> [1] I'm looking for CVE fixes, 1.0.2p has a lot of CVEs.

You know that 1.0.2 and 1.1 APIs are not compatible?  So you will need to update
everything that needs OpenSSL to understand the new API.

For CVE fixes, typically you would patch 1.0.2p, or update to the latest
(1.0.2t) as you go.  (If you have an OSV, this should be part of the services
that they offer you.)

In my opinion, 1.0.2 will be around for at least another 4-5 years due to the
number of people actively using it in the world.  Until 1.1/3.0 (won't be a 2.0
from what I read) exists and has a FIPS-140-2 support available -- people will
continue to use 1.0.2 and maintain it as necessary for security.

As an FYI:  http://git.yoctoproject.org/cgit/cgit.cgi/meta-openssl102/

This version is for thud, warrior, zeus and master.  It is intended to be
maintained until either 1.0.2 is no longer maintainable -- or the FIPS-140-2
needs have been met by OpenSSL.

--Mark

> [2] http://git.yoctoproject.org/git/poky
> I'm at SHA a616ffebdc, so I copied openssl_1.1.1d.bb 
> 
> and all the other files in the directory.
> 
> [3] I have a clone of Linaro's meta-backports. I'm trying to generate a patch 
> to
> submit for review there.
> https://git.linaro.org/openembedded/meta-backports.git
> 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to backport openssl to Sumo

[Ross Burton]

On 20/11/2019 18:18, Ryan Harkin wrote:
Now it builds fine. However, I no longer have /usr/bin/openssl in my 
disk image.


$ oe-pkgdata-util find-path /usr/bin/openssl
openssl-bin: /usr/bin/openssl

Did you try adding openssl-bin to your image explicitly?

Ross
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] rpm: use libgcrypt instead of NSS for cryptography

[Ross Burton]

On 20/11/2019 17:51, Mark Hatle wrote:

Just as an ack.. Please get rid of NSS/NSPR ASAP, and move to libgcrypt.  Long
term it's a much much better solution.


Absolutely.  I saw nss fly past in a build and was surprised we still 
needed it.


This is the last user in oe-core currently, but meta-oe has several. 
Including mozjs, which might end up in core...


Ross

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] How to backport openssl to Sumo

[Ryan Harkin]

Hi all,

I'm struggling with backporting OpenSSL to my Sumo build [1], so wondered
if anyone else had done something similar with success.

I copied "meta/recipes-connectivity/openssl" from Poky master branch [2]
into my own layer [3]. It didn't pick up, so I discovered I needed to add
a PREFERRED_VERSION, eg:

+PREFERRED_VERSION_openssl ?= "1.1.%"
+PREFERRED_VERSION_openssl-native ?= "1.1.%"
+PREFERRED_VERSION_nativesdk-openssl ?= "1.1.%"

Now it builds fine. However, I no longer have /usr/bin/openssl in my disk
image.

It doesn't appear in FILES_${PN}, and adding it to the recipes doesn't seem
to make any difference.

What am I missing?

Thanks,
Ryan.

[1] I'm looking for CVE fixes, 1.0.2p has a lot of CVEs.

[2] http://git.yoctoproject.org/git/poky
I'm at SHA a616ffebdc, so I copied openssl_1.1.1d.bb and all the other
files in the directory.

[3] I have a clone of Linaro's meta-backports. I'm trying to generate a
patch to submit for review there.
https://git.linaro.org/openembedded/meta-backports.git
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] rpm: use libgcrypt instead of NSS for cryptography

[Mark Hatle]

Just as an ack.. Please get rid of NSS/NSPR ASAP, and move to libgcrypt.  Long
term it's a much much better solution.

--Mark

On 11/20/19 10:19 AM, Ross Burton wrote:
> RPM 4.15 by default will use libgcrypt instead of NSS for cryptography
> functions, but as we didn't have libgcrypt in DEPENDS it kept using NSS.
> 
> As RPM is the sole user of NSS/NSPR in oe-core, moving to libgcrypt can make a
> noticable difference to build time.  For example, building rpm (and packaging 
> it
> as RPMs) from scratch is five minutes faster with libgcrypt.
> 
> Signed-off-by: Ross Burton 
> ---
>  .../rpm/files/gcrypt-use-pkgconfig.patch  | 51 +++
>  meta/recipes-devtools/rpm/rpm_4.15.1.bb   |  5 +-
>  2 files changed, 54 insertions(+), 2 deletions(-)
>  create mode 100644 meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch
> 
> diff --git a/meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch 
> b/meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch
> new file mode 100644
> index 000..8c72d2310b6
> --- /dev/null
> +++ b/meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch
> @@ -0,0 +1,51 @@
> +Upstream-Status: Submitted 
> [https://github.com/rpm-software-management/rpm/pull/942]
> +Signed-off-by: Ross Burton 
> +
> +From 3f6cda568853bf7878df704adc75d4a78d75346c Mon Sep 17 00:00:00 2001
> +From: Ross Burton 
> +Date: Wed, 20 Nov 2019 13:06:51 +
> +Subject: [PATCH] configure.ac: prefer pkg-config to find libgcrypt
> +
> +libgcrypt from 1.8.5 provides a pkg-config file as well as the traditional
> +libgcrypt-config script.  As pkg-config is more resiliant in the face of
> +complicated build environments (for example cross-compilation and sysroots)
> +prefer the pkg-config file, falling back to libgcrypt-config if that doesn't
> +exist.
> +---
> + configure.ac | 23 +++
> + 1 file changed, 15 insertions(+), 8 deletions(-)
> +
> +diff --git a/configure.ac b/configure.ac
> +index 0a3a9bbf4..6a3ea3615 100644
> +--- a/configure.ac
>  b/configure.ac
> +@@ -395,14 +395,21 @@ AC_SUBST(WITH_OPENSSL_LIB)
> + WITH_LIBGCRYPT_INCLUDE=
> + WITH_LIBGCRYPT_LIB=
> + if test "$with_crypto" = libgcrypt ; then
> +-AC_PATH_TOOL(LIBGCRYPT_CONFIG, libgcrypt-config, notfound)
> +-if test notfound != "$LIBGCRYPT_CONFIG" ; then
> +-WITH_LIBGCRYPT_INCLUDE=`$LIBGCRYPT_CONFIG --cflags`
> +-WITH_LIBGCRYPT_LIB=`$LIBGCRYPT_CONFIG --libs`
> +-fi
> +-if test -z "$WITH_LIBGCRYPT_LIB" ; then
> +-AC_MSG_ERROR([libgcrypt not found])
> +-fi
> ++  # libgcrypt 1.8.5 onwards ships a pkg-config file so prefer that
> ++  PKG_CHECK_MODULES([LIBGCRYPT], [libgcrypt], [have_libgcrypt=yes], 
> [have_libgcrypt=no])
> ++  if test "$have_libgcrypt" = "yes"; then
> ++WITH_LIBGCRYPT_INCLUDE="$LIBGCRYPT_CFLAGS"
> ++WITH_LIBGCRYPT_LIB="$LIBGCRYPT_LIBS"
> ++  else
> ++AC_PATH_TOOL(LIBGCRYPT_CONFIG, libgcrypt-config, notfound)
> ++  if test notfound != "$LIBGCRYPT_CONFIG" ; then
> ++WITH_LIBGCRYPT_INCLUDE=`$LIBGCRYPT_CONFIG --cflags`
> ++WITH_LIBGCRYPT_LIB=`$LIBGCRYPT_CONFIG --libs`
> ++ fi
> ++ if test -z "$WITH_LIBGCRYPT_LIB" ; then
> ++   AC_MSG_ERROR([libgcrypt not found])
> ++fi
> ++  fi
> + fi
> + 
> + AM_CONDITIONAL([WITH_LIBGCRYPT],[test "$with_crypto" = libgcrypt])
> diff --git a/meta/recipes-devtools/rpm/rpm_4.15.1.bb 
> b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
> index 4fa2d764fb9..f033cf33144 100644
> --- a/meta/recipes-devtools/rpm/rpm_4.15.1.bb
> +++ b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
> @@ -38,6 +38,7 @@ SRC_URI = 
> "git://github.com/rpm-software-management/rpm;branch=rpm-4.15.x \
> 
> file://0001-rpm-rpmio.c-restrict-virtual-memory-usage-if-limit-s.patch \
> 
> file://0016-rpmscript.c-change-logging-level-around-scriptlets-t.patch \
> 
> file://0001-rpmfc.c-do-not-run-file-classification-in-parallel.patch \
> +   file://gcrypt-use-pkgconfig.patch \
> "
>  
>  PE = "1"
> @@ -45,7 +46,7 @@ SRCREV = "ab2179452c5be276a6b96c591afded485c7e58c3"
>  
>  S = "${WORKDIR}/git"
>  
> -DEPENDS = "nss libarchive db file popt xz bzip2 dbus elfutils python3"
> +DEPENDS = "libarchive libgcrypt db file popt xz bzip2 dbus elfutils python3"
>  DEPENDS_append_class-native = " file-replacement-native 
> bzip2-replacement-native"
>  
>  inherit autotools gettext pkgconfig python3native
> @@ -54,7 +55,7 @@ export PYTHON_ABI
>  # OE-core patches autoreconf to additionally run gnu-configize, which fails 
> with this recipe
>  EXTRA_AUTORECONF_append = " --exclude=gnu-configize"
>  
> -EXTRA_OECONF_append = " --without-lua --enable-python"
> +EXTRA_OECONF_append = " --without-lua --enable-python 
> --with-crypto=libgcrypt"
>  EXTRA_OECONF_append_libc-musl = " --disable-nls"
>  
>  # --sysconfdir prevents rpm from attempting to access machine-specific 
> configuration in sysroot/etc; we need to have it in rootfs
> 
-- 
___
Openembedded-core mailing list

[OE-core] ✗ patchtest: failure for rpm: use libgcrypt instead of NSS for cryptography

[Patchwork]

== Series Details ==

Series: rpm: use libgcrypt instead of NSS for cryptography
Revision: 1
URL   : https://patchwork.openembedded.org/series/21254/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue Series does not apply on top of target branch 
[test_series_merge_on_head] 
  Suggested fixRebase your series on top of targeted branch
  Targeted branch  master (currently at dfb3b56641)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines: 
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] How to add build date?

[Joseph Reynolds]

On 11/17/19 3:19 PM, Alexander Kanavin wrote:
I'd write the date into the file at image creation, via 
ROOTFS_POSTPROCESS_COMMAND.


Having it in a recipe means that you either force the recipe to not be 
a part of sstate cache, always rebuilding it and its dependencies (bad 
idea), or accept that the date comes from a previously built cache 
object, which means it will mismatch the actual image creation date.


Got it, thanks. I was unaware of ROOTFS_POSTPROCESS_COMMAND.




A bit of context would help: why would you want that info in 
os-release? Who would use that information and how?


I don't fully understand the use case for adding a build date to the image.  I 
don't have a use for it, but this question keeps coming up in the OpenBMC 
community, so I want to address it.  I am imagining a naive continuous delivery 
process where the build date is used as a substitute for a software version.  
That is easier than keeping track of software versions, but (as I mentioned 
before) does not meet my needs.

I've re-asked the question to the original poster in the OpenBMC email list:
https://lists.ozlabs.org/pipermail/openbmc/2019-November/019493.html

The response so far is: I just want to show the compile time of firmware to the 
user. If  only show the version, it can't correspond to the time. When both are 
displayed at the same time, the information will be clearer.

I'll continue the conversation in that email thread.  Thanks for your help!

 - Joseph




Alex

On Sat, 16 Nov 2019 at 23:14, Joseph Reynolds > wrote:



Is there a best practice for folks who want to put the build date
into
their generated os-release file?  I've seen this question come up at
least 3 times in my brief involvement with OE.  I could perhaps add
comment to the OpenBMC os-release.bbappend file to provide the proper
guidance.  I would like to hear your ideas.

- Joseph


 Forwarded Message 
Subject:        Re: how to solve the error that basehash value
changed from
'xxx' to '' ?
Date:   Sat, 16 Nov 2019 15:50:38 -0600
From:   Joseph Reynolds mailto:j...@linux.ibm.com>>
To:     www mailto:ouyangxua...@163.com>>,
open...@lists.ozlabs.org 
mailto:open...@lists.ozlabs.org>>



Byron,

What modification did you put into the os-release recipe?  It
would help
us help you if you put that in your email.


Are you adding the build date to the generated os-release file? If
so,
be aware that might cause the issues with the shared state cache
you are
experiencing.  Also be aware this practice wrecks reproducible builds.

A better way to get the same effect might be to create a git tag with
the information you want to appear in the generated os-release file.
When you buld with this tag is then picked up by the OpenBMC
meta-phosphor os-release.bbappend (via `git describe`) and
included in
the genersted os-release file as the BUILD_ID. See:

https://github.com/ibm-openbmc/openbmc/blob/d1c59b7a36c10c18838c07af10b18080174cd61d/meta-phosphor/recipes-core/os-release/os-release.bbappend
(But I have not tried that.)

A way to work around the hash change is to add a line to your recipe
like: PR[vardepsxeclude]="DATETIME"
As suggested by:
https://forums.openpli.org/topic/41447-how-to-get-rid-of-taskhash-mismatch/


My apologies for top posting -- my email reader was not quoting the
message properly, so I cut/pasted it below.

- Joseph


On 11/13/19 1:38 AM, www wrote:

When I modify the os-release file in my yocto project, it appear some
error, and how can I solve it ? Who can give me some help or advice?
Thank you！
I execute the recommended command on the console and it didn't work.

ERROR: os-release-1.0-r0 do_compile: Taskhash mismatch
ce133f0458608e03aa55224df28156e523e54903115efbbcd62946f84a867201
versus
7269881f0eb1759ed420a2db4c04fb477cd8c1288bc5f82df5c8161bb926ea1f
for  
/home/temp/xxx/wsp/obmc-source/entity_xxx/meta/recipes-core/os-release/os-release.bb.do_compile
ERROR: Taskhash mismatch
ce133f0458608e03aa55224df28156e523e54903115efbbcd62946f84a867201
versus
7269881f0eb1759ed420a2db4c04fb477cd8c1288bc5f82df5c8161bb926ea1f
for 
/home/temp/xxx/wsp/obmc-source/entity_xxx/meta/recipes-core/os-release/os-release.bb.do_compile
ERROR: When reparsing

/home/temp/xxx/wsp/obmc-source/entity_xxx/meta/recipes-core/os-release/os-release.bb.do_compile,
 the

basehash value changed from
99a42a1a3b1a151de604267b159558ecaf1031a3bec8917df132c81302e729a5 to
4f3288a8763e2e1af78e4b3cdd9c0c0ccb3b0d5c78a3073c188b22200df2a9b0. The
metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake os-release -cdo_compile -Snone
ERROR: Then:
ERROR: $ bitbake os-release -cdo_compile 

[OE-core] [PATCH] rpm: use libgcrypt instead of NSS for cryptography

[Ross Burton]

RPM 4.15 by default will use libgcrypt instead of NSS for cryptography
functions, but as we didn't have libgcrypt in DEPENDS it kept using NSS.

As RPM is the sole user of NSS/NSPR in oe-core, moving to libgcrypt can make a
noticable difference to build time.  For example, building rpm (and packaging it
as RPMs) from scratch is five minutes faster with libgcrypt.

Signed-off-by: Ross Burton 
---
 .../rpm/files/gcrypt-use-pkgconfig.patch  | 51 +++
 meta/recipes-devtools/rpm/rpm_4.15.1.bb   |  5 +-
 2 files changed, 54 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch

diff --git a/meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch 
b/meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch
new file mode 100644
index 000..8c72d2310b6
--- /dev/null
+++ b/meta/recipes-devtools/rpm/files/gcrypt-use-pkgconfig.patch
@@ -0,0 +1,51 @@
+Upstream-Status: Submitted 
[https://github.com/rpm-software-management/rpm/pull/942]
+Signed-off-by: Ross Burton 
+
+From 3f6cda568853bf7878df704adc75d4a78d75346c Mon Sep 17 00:00:00 2001
+From: Ross Burton 
+Date: Wed, 20 Nov 2019 13:06:51 +
+Subject: [PATCH] configure.ac: prefer pkg-config to find libgcrypt
+
+libgcrypt from 1.8.5 provides a pkg-config file as well as the traditional
+libgcrypt-config script.  As pkg-config is more resiliant in the face of
+complicated build environments (for example cross-compilation and sysroots)
+prefer the pkg-config file, falling back to libgcrypt-config if that doesn't
+exist.
+---
+ configure.ac | 23 +++
+ 1 file changed, 15 insertions(+), 8 deletions(-)
+
+diff --git a/configure.ac b/configure.ac
+index 0a3a9bbf4..6a3ea3615 100644
+--- a/configure.ac
 b/configure.ac
+@@ -395,14 +395,21 @@ AC_SUBST(WITH_OPENSSL_LIB)
+ WITH_LIBGCRYPT_INCLUDE=
+ WITH_LIBGCRYPT_LIB=
+ if test "$with_crypto" = libgcrypt ; then
+-AC_PATH_TOOL(LIBGCRYPT_CONFIG, libgcrypt-config, notfound)
+-if test notfound != "$LIBGCRYPT_CONFIG" ; then
+-WITH_LIBGCRYPT_INCLUDE=`$LIBGCRYPT_CONFIG --cflags`
+-WITH_LIBGCRYPT_LIB=`$LIBGCRYPT_CONFIG --libs`
+-fi
+-if test -z "$WITH_LIBGCRYPT_LIB" ; then
+-AC_MSG_ERROR([libgcrypt not found])
+-fi
++  # libgcrypt 1.8.5 onwards ships a pkg-config file so prefer that
++  PKG_CHECK_MODULES([LIBGCRYPT], [libgcrypt], [have_libgcrypt=yes], 
[have_libgcrypt=no])
++  if test "$have_libgcrypt" = "yes"; then
++WITH_LIBGCRYPT_INCLUDE="$LIBGCRYPT_CFLAGS"
++WITH_LIBGCRYPT_LIB="$LIBGCRYPT_LIBS"
++  else
++AC_PATH_TOOL(LIBGCRYPT_CONFIG, libgcrypt-config, notfound)
++  if test notfound != "$LIBGCRYPT_CONFIG" ; then
++WITH_LIBGCRYPT_INCLUDE=`$LIBGCRYPT_CONFIG --cflags`
++WITH_LIBGCRYPT_LIB=`$LIBGCRYPT_CONFIG --libs`
++ fi
++ if test -z "$WITH_LIBGCRYPT_LIB" ; then
++   AC_MSG_ERROR([libgcrypt not found])
++fi
++  fi
+ fi
+ 
+ AM_CONDITIONAL([WITH_LIBGCRYPT],[test "$with_crypto" = libgcrypt])
diff --git a/meta/recipes-devtools/rpm/rpm_4.15.1.bb 
b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
index 4fa2d764fb9..f033cf33144 100644
--- a/meta/recipes-devtools/rpm/rpm_4.15.1.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.15.1.bb
@@ -38,6 +38,7 @@ SRC_URI = 
"git://github.com/rpm-software-management/rpm;branch=rpm-4.15.x \

file://0001-rpm-rpmio.c-restrict-virtual-memory-usage-if-limit-s.patch \

file://0016-rpmscript.c-change-logging-level-around-scriptlets-t.patch \

file://0001-rpmfc.c-do-not-run-file-classification-in-parallel.patch \
+   file://gcrypt-use-pkgconfig.patch \
"
 
 PE = "1"
@@ -45,7 +46,7 @@ SRCREV = "ab2179452c5be276a6b96c591afded485c7e58c3"
 
 S = "${WORKDIR}/git"
 
-DEPENDS = "nss libarchive db file popt xz bzip2 dbus elfutils python3"
+DEPENDS = "libarchive libgcrypt db file popt xz bzip2 dbus elfutils python3"
 DEPENDS_append_class-native = " file-replacement-native 
bzip2-replacement-native"
 
 inherit autotools gettext pkgconfig python3native
@@ -54,7 +55,7 @@ export PYTHON_ABI
 # OE-core patches autoreconf to additionally run gnu-configize, which fails 
with this recipe
 EXTRA_AUTORECONF_append = " --exclude=gnu-configize"
 
-EXTRA_OECONF_append = " --without-lua --enable-python"
+EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=libgcrypt"
 EXTRA_OECONF_append_libc-musl = " --disable-nls"
 
 # --sysconfdir prevents rpm from attempting to access machine-specific 
configuration in sysroot/etc; we need to have it in rootfs
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] bitbake: prserv: fix ResourceWarning due to unclosed socket

[Martin Jansa]

Thanks, but it still starts and terminates PRServer immediately, right?

With your fix, all my builds now start with:

NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 34335, PID: 9429
NOTE: Terminating PRServer...
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3,
IP: 127.0.0.1, PORT: 45753, PID: 9433

before this I was using:
https://github.com/shr-distribution/bitbake/commit/728d9ec3b3448bba91a3cc0d505b804e702aec4b
which doesn't start it just to terminate it, but RP said that my fix
doesn't look correct, so I haven't sent it to ML. BTW: bitbake patches
should go to bitbake-devel ML not oe-core ML.


On Mon, Nov 11, 2019 at 11:15 PM  wrote:

> From: Gavin Li 
>
> With PRSERV_HOST = "localhost:0", this message would occasionally pop up
> during the initial cache read:
>
> WARNING: /home/matic/ambayocto/poky/bitbake/lib/bb/cache.py:446:
> ResourceWarning: unclosed  family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 45655)>
>   value = pickled.load()
>
> The file location stated is irrelevant; it just happens to be wherever
> CPython decides to run the garbage collector. The issue is that after we
> fork off a PRServer, self.socket is also duplicated. The parent side of
> it also needs to be closed.
>
> Signed-off-by: Gavin Li 
> ---
>  bitbake/lib/prserv/serv.py | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index be3acec36a..1d9148b817 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -243,6 +243,7 @@ class PRServer(SimpleXMLRPCServer):
>  try:
>  pid = os.fork()
>  if pid > 0:
> +self.socket.close() # avoid ResourceWarning in parent
>  return pid
>  except OSError as e:
>  raise Exception("%s [%d]" % (e.strerror, e.errno))
> --
> 2.23.0
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] sdk: environment script doesn't set security options in CC and LDFLAGS variables

[Antoine Manache]

When building a SDK from a DISTRO with security flags enabled, options
added to
CC and LDFLAGS are not replicated in the SDK environment script.
This could lead to some situations where an application compiled with
the SDK and having some security weaknesses correctly runs on target but
crashes once integrated to the core image built with the full Yocto
stack.

Signed-off-by: Antoine Manache 
---
 meta/conf/distro/include/security_flags.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/conf/distro/include/security_flags.inc
b/meta/conf/distro/include/security_flags.inc
index 620978a8ed..329482bfa3 100644
--- a/meta/conf/distro/include/security_flags.inc
+++ b/meta/conf/distro/include/security_flags.inc
@@ -56,7 +56,9 @@ SECURITY_STRINGFORMAT_pn-busybox = ""
 SECURITY_STRINGFORMAT_pn-gcc = ""

 TARGET_CC_ARCH_append_class-target = " ${SECURITY_CFLAGS}"
+TARGET_CC_ARCH_append_class-cross-canadian = " ${SECURITY_CFLAGS}"
 TARGET_LDFLAGS_append_class-target = " ${SECURITY_LDFLAGS}"
+TARGET_LDFLAGS_append_class-cross-canadian = " ${SECURITY_LDFLAGS}"

 SECURITY_STACK_PROTECTOR_pn-gcc-runtime = ""
 SECURITY_STACK_PROTECTOR_pn-glibc = ""

--
2.11.0
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 01/13] python: update to 2.7.17

[Tom Rini]

On Wed, Nov 20, 2019 at 04:06:44PM +0100, Alexander Kanavin wrote:
> On Wed, 20 Nov 2019 at 14:50, Tom Rini  wrote:
> 
> > Well, the release is scheduled for January 6th.  I'd really like to see
> > more testing in no-python2 environment.  I believe the Fedora folks have
> > confirmed things are good now but it would be good to see OE do it too,
> > especially on master and not too close to branching out another release.
> >
> 
> I made a few patches to do that, and can confirm that various flavours of
> u-boot recipe build without issues.
> 
> If someone else wants to test a py2-less poky with their builds, here it is:
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/drop-python2

Thanks!

-- 
Tom


signature.asc
Description: PGP signature
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 01/13] python: update to 2.7.17

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 14:50, Tom Rini  wrote:

> Well, the release is scheduled for January 6th.  I'd really like to see
> more testing in no-python2 environment.  I believe the Fedora folks have
> confirmed things are good now but it would be good to see OE do it too,
> especially on master and not too close to branching out another release.
>

I made a few patches to do that, and can confirm that various flavours of
u-boot recipe build without issues.

If someone else wants to test a py2-less poky with their builds, here it is:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/drop-python2

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] ✗ patchtest: failure for SDK environment script doesn't set security options in CC and LDFLAGS variables

[Patchwork]

== Series Details ==

Series: SDK environment script doesn't set security options in CC and LDFLAGS 
variables
Revision: 1
URL   : https://patchwork.openembedded.org/series/21252/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch[yocto] SDK environment script doesn't set security options 
in CC and LDFLAGS variables
 Issue Shortlog does not follow expected format 
[test_shortlog_format] 
  Suggested fixCommit shortlog (first line of commit message) should follow 
the format ": "



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines: 
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [yocto] [PATCH] SDK environment script doesn't set security options in CC and LDFLAGS variables

[Antoine Manache]

When building a SDK from a DISTRO with security flags enabled, options
added to
CC and LDFLAGS are not replicated in the SDK environment script.
This could lead to some situations where an application compiled with
the SDK and having some security weaknesses correctly runs on target but
crashes once integrated to the core image built with the full Yocto
stack.

Signed-off-by: Antoine Manache 
---
 meta/conf/distro/include/security_flags.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/conf/distro/include/security_flags.inc
b/meta/conf/distro/include/security_flags.inc
index 620978a8ed..329482bfa3 100644
--- a/meta/conf/distro/include/security_flags.inc
+++ b/meta/conf/distro/include/security_flags.inc
@@ -56,7 +56,9 @@ SECURITY_STRINGFORMAT_pn-busybox = ""
 SECURITY_STRINGFORMAT_pn-gcc = ""

 TARGET_CC_ARCH_append_class-target = " ${SECURITY_CFLAGS}"
+TARGET_CC_ARCH_append_class-cross-canadian = " ${SECURITY_CFLAGS}"
 TARGET_LDFLAGS_append_class-target = " ${SECURITY_LDFLAGS}"
+TARGET_LDFLAGS_append_class-cross-canadian = " ${SECURITY_LDFLAGS}"

 SECURITY_STACK_PROTECTOR_pn-gcc-runtime = ""
 SECURITY_STACK_PROTECTOR_pn-glibc = ""

--
2.11.0
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 01/13] python: update to 2.7.17

[Tom Rini]

On Wed, Nov 20, 2019 at 02:47:19PM +0100, Alexander Kanavin wrote:
> On Wed, 20 Nov 2019 at 13:56, Tom Rini  wrote:
> 
> > I would _really_ like to see someone move master to using v2020.01-rc3
> > or so for U-Boot as that should not have any Python 2 dependencies left.
> 
> rc3 means the final release is very soon, so we'll gladly update to that.

Well, the release is scheduled for January 6th.  I'd really like to see
more testing in no-python2 environment.  I believe the Fedora folks have
confirmed things are good now but it would be good to see OE do it too,
especially on master and not too close to branching out another release.

-- 
Tom


signature.asc
Description: PGP signature
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 01/13] python: update to 2.7.17

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 13:56, Tom Rini  wrote:

> I would _really_ like to see someone move master to using v2020.01-rc3
> or so for U-Boot as that should not have any Python 2 dependencies left.
>

rc3 means the final release is very soon, so we'll gladly update to that.

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 11/11] systemtap: update to 4.2

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin 
---
 meta/recipes-kernel/systemtap/systemtap_git.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/systemtap/systemtap_git.inc 
b/meta/recipes-kernel/systemtap/systemtap_git.inc
index abb2b3743b4..116e83fe0ff 100644
--- a/meta/recipes-kernel/systemtap/systemtap_git.inc
+++ b/meta/recipes-kernel/systemtap/systemtap_git.inc
@@ -1,7 +1,7 @@
 LICENSE = "GPLv2"
 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
-SRCREV = "57c9aca9f1ff32a6add10e02ecd33b7314fad499"
-PV = "4.1+git${SRCPV}"
+SRCREV = "044a0640985ef007c0b2fb6eaf660d9d51800cda"
+PV = "4.2"
 
 SRC_URI = "git://sourceware.org/git/systemtap.git \

file://0001-Do-not-let-configure-write-a-python-location-into-th.patch \
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 10/11] mc: backport a patch to fix builds with latest gettext

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin 
---
 ...29-configure.ac-drop-bundled-gettext.patch | 110 ++
 meta/recipes-extended/mc/mc_4.8.23.bb |   1 +
 2 files changed, 111 insertions(+)
 create mode 100644 
meta/recipes-extended/mc/files/0001-Ticket-3629-configure.ac-drop-bundled-gettext.patch

diff --git 
a/meta/recipes-extended/mc/files/0001-Ticket-3629-configure.ac-drop-bundled-gettext.patch
 
b/meta/recipes-extended/mc/files/0001-Ticket-3629-configure.ac-drop-bundled-gettext.patch
new file mode 100644
index 000..8f357378d02
--- /dev/null
+++ 
b/meta/recipes-extended/mc/files/0001-Ticket-3629-configure.ac-drop-bundled-gettext.patch
@@ -0,0 +1,110 @@
+From 0d677a014a87b968d79eea2353ac4e342b0fd4ca Mon Sep 17 00:00:00 2001
+From: Sergei Trofimovich 
+Date: Wed, 11 Sep 2019 22:58:18 +0100
+Subject: [PATCH] Ticket #3629: configure.ac: drop bundled gettext
+
+Bundled libintl did not support linking to internal static
+libraries (libmc in our case): directly specified static
+libraries are not pulled by libtool and are not usable for
+dynamic libraries as PIC-related flags are not passed for
+compilation.
+
+This renders bundled libintl library unusable.
+
+The change drops libintl bundling support and always relies
+on external libintl (or falls back to disabled NLS).
+
+On a related note gettext-0.20 drops support for bundling
+or libintl and this change will ease migration to newer version.
+
+The change is tested on x86_64-gentoo-linux-musl: mc builds
+and links all tests successfully. A few tests fail for lack
+of NLS support.
+
+Upstream-Status: Backport 
[https://github.com/MidnightCommander/mc/commit/f30e6ff283f4bc86177e4360de94dad794678395]
+Signed-off-by: Sergei Trofimovich 
+Signed-off-by: Andrew Borodin 
+Signed-off-by: Alexander Kanavin 
+---
+ Makefile.am   | 2 +-
+ configure.ac  | 5 +++--
+ doc/doxygen.cfg   | 2 +-
+ lib/Makefile.am   | 2 +-
+ m4.include/mc-i18n.m4 | 5 -
+ 5 files changed, 6 insertions(+), 10 deletions(-)
+
+diff --git a/Makefile.am b/Makefile.am
+index ac05a83..f86f6ed 100644
+--- a/Makefile.am
 b/Makefile.am
+@@ -1,7 +1,7 @@
+ ## Process this file with automake to create Makefile.in.
+ AUTOMAKE_OPTIONS = 1.5
+ 
+-SUBDIRS = intl po lib src doc contrib misc
++SUBDIRS = po lib src doc contrib misc
+ 
+ if HAVE_TESTS
+ SUBDIRS += tests
+diff --git a/configure.ac b/configure.ac
+index a1948f6..bbc9e71 100644
+--- a/configure.ac
 b/configure.ac
+@@ -272,7 +272,9 @@ dnl 

+ dnl Internationalization
+ dnl 

+ 
+-AM_GNU_GETTEXT([no-libtool], [need-ngettext])
++AC_CHECK_FUNCS([setlocale])
++
++AM_GNU_GETTEXT([external], [need-ngettext])
+ AM_GNU_GETTEXT_VERSION([0.18.1])
+ 
+ mc_I18N
+@@ -680,7 +682,6 @@ doc/hlp/pl/Makefile
+ doc/hlp/ru/Makefile
+ doc/hlp/sr/Makefile
+ 
+-intl/Makefile
+ po/Makefile.in
+ ])
+ 
+diff --git a/doc/doxygen.cfg b/doc/doxygen.cfg
+index 07bc973..1118062 100644
+--- a/doc/doxygen.cfg
 b/doc/doxygen.cfg
+@@ -91,7 +91,7 @@ FILE_PATTERNS  = *.c \
+ RECURSIVE  = YES
+ EXCLUDE=
+ EXCLUDE_SYMLINKS   = NO
+-EXCLUDE_PATTERNS   = */intl/* */tests/* */.git/*
++EXCLUDE_PATTERNS   = */tests/* */.git/*
+ EXCLUDE_SYMBOLS=
+ EXAMPLE_PATH   = $(SRCDIR)
+ EXAMPLE_PATTERNS   =
+diff --git a/lib/Makefile.am b/lib/Makefile.am
+index c448e2d..455f9dd 100644
+--- a/lib/Makefile.am
 b/lib/Makefile.am
+@@ -74,4 +74,4 @@ else
+ libmc_la_LIBADD += $(GLIB_LIBS)
+ endif
+ 
+-libmc_la_LIBADD += $(PCRE_LIBS) $(LIBICONV) $(LIBINTL)
++libmc_la_LIBADD += $(PCRE_LIBS)
+diff --git a/m4.include/mc-i18n.m4 b/m4.include/mc-i18n.m4
+index dd10d00..ec08324 100644
+--- a/m4.include/mc-i18n.m4
 b/m4.include/mc-i18n.m4
+@@ -8,11 +8,6 @@ dnl @license GPL
+ dnl @copyright Free Software Foundation, Inc.
+ 
+ AC_DEFUN([mc_I18N],[
+-
+-if test "x$USE_INCLUDED_LIBINTL" = xyes; then
+-CPPFLAGS="$CPPFLAGS -I\$(top_builddir)/intl -I\$(top_srcdir)/intl"
+-fi
+-
+ dnl User visible support for charset conversion.
+ AC_ARG_ENABLE([charset],
+ AS_HELP_STRING([--enable-charset], [Support for charset selection and 
conversion @<:@yes@:>@]))
diff --git a/meta/recipes-extended/mc/mc_4.8.23.bb 
b/meta/recipes-extended/mc/mc_4.8.23.bb
index 71f61b48485..ac40e561832 100644
--- a/meta/recipes-extended/mc/mc_4.8.23.bb
+++ b/meta/recipes-extended/mc/mc_4.8.23.bb
@@ -9,6 +9,7 @@ RDEPENDS_${PN} = "ncurses-terminfo"
 SRC_URI = "http://www.midnight-commander.org/downloads/${BPN}-${PV}.tar.bz2 \
file://0001-mc-replace-perl-w-with-use-warnings.patch \
file://0001-Add-option-to-control-configure-args.patch \
+   file://0001-Ticket-3629-configure.ac-drop-bundled-gettext.patch \
"
 SRC_URI[md5sum] = "152927ac29cf0e61d7d019f261bb7d89"
 SRC_URI[sha256sum] = 

[OE-core] [PATCH 07/11] console-tools: remove the recipe

[Alexander Kanavin]

The last release of this was in 2002(!), kbd is the modern,
supported alternative.

Signed-off-by: Alexander Kanavin 
---
 meta/conf/distro/include/maintainers.inc  |  1 -
 .../0001-Cover-the-else-with-__GLIBC__.patch  | 51 -
 ...ix-format-security-compilation-error.patch | 57 --
 ...-sys-types.h-for-u_char-and-u_short-.patch | 44 ---
 .../console-tools-0.3.2/Makevars  | 41 --
 .../console-tools-0.3.2/codepage.patch| 17 -
 .../console-tools-0.3.2/compile.patch | 12 ---
 .../console-tools-0.3.2/configure.patch   | 29 ---
 .../fix-libconsole-linking.patch  | 56 --
 .../console-tools-0.3.2/kbdrate.patch | 75 ---
 .../console-tools-0.3.2/lcmessage.m4  | 24 --
 .../console-tools-0.3.2/no-dep-on-libfl.patch | 19 -
 .../console-tools-0.3.2/nodocs.patch  | 22 --
 .../console-tools-0.3.2/uclibc-fileno.patch   | 49 
 .../console-tools/console-tools_0.3.2.bb  | 48 
 15 files changed, 545 deletions(-)
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/0001-Cover-the-else-with-__GLIBC__.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/0001-Fix-format-security-compilation-error.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/0001-kbdtools-Include-sys-types.h-for-u_char-and-u_short-.patch
 delete mode 100644 meta/recipes-core/console-tools/console-tools-0.3.2/Makevars
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/codepage.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/compile.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/configure.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/fix-libconsole-linking.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/kbdrate.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/lcmessage.m4
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/no-dep-on-libfl.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/nodocs.patch
 delete mode 100644 
meta/recipes-core/console-tools/console-tools-0.3.2/uclibc-fileno.patch
 delete mode 100644 meta/recipes-core/console-tools/console-tools_0.3.2.bb

diff --git a/meta/conf/distro/include/maintainers.inc 
b/meta/conf/distro/include/maintainers.inc
index d85e5b697f5..a289008b8c1 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -103,7 +103,6 @@ RECIPE_MAINTAINER_pn-cogl-1.0 = "Ross Burton 
"
 RECIPE_MAINTAINER_pn-connman = "Changhyeok Bae "
 RECIPE_MAINTAINER_pn-connman-conf = "Ross Burton "
 RECIPE_MAINTAINER_pn-connman-gnome = "Ross Burton "
-RECIPE_MAINTAINER_pn-console-tools = "Chen Qi "
 RECIPE_MAINTAINER_pn-consolekit = "Chen Qi "
 RECIPE_MAINTAINER_pn-core-image-base = "Richard Purdie 
"
 RECIPE_MAINTAINER_pn-core-image-minimal = "Richard Purdie 
"
diff --git 
a/meta/recipes-core/console-tools/console-tools-0.3.2/0001-Cover-the-else-with-__GLIBC__.patch
 
b/meta/recipes-core/console-tools/console-tools-0.3.2/0001-Cover-the-else-with-__GLIBC__.patch
deleted file mode 100644
index ace4bf0a76b..000
--- 
a/meta/recipes-core/console-tools/console-tools-0.3.2/0001-Cover-the-else-with-__GLIBC__.patch
+++ /dev/null
@@ -1,51 +0,0 @@
-From b6a59b05f1fa514c6b387c9544bd63b1bfcf2eed Mon Sep 17 00:00:00 2001
-From: Khem Raj 
-Date: Wed, 13 Jan 2016 05:38:29 +
-Subject: [PATCH] Cover the else with __GLIBC__
-
-Fixes build errors on musl/x86
-
-| In file included from
-/home/ubuntu/work/oe/openembedded-core/build/tmp-musl/sysroots/qemux86/usr/include/asm/termios.h:1:0,
-|  from
-/home/ubuntu/work/oe/openembedded-core/build/tmp-musl/sysroots/qemux86/usr/include/linux/termios.h:5,
-|  from
-/home/ubuntu/work/oe/openembedded-core/build/tmp-musl/work/i586-oe-linux-musl/console-tools/0.3.2-r8/console-tools-0.3.2/vttools/resizecons.c:86:
-|
-/home/ubuntu/work/oe/openembedded-core/build/tmp-musl/sysroots/qemux86/usr/include/asm-generic/termios.h:14:8:
-error: redefinition of 'struct winsize'
-|  struct winsize {
-| ^
-| In file included from
-/home/ubuntu/work/oe/openembedded-core/build/tmp-musl/sysroots/qemux86/usr/include/sys/ioctl.h:7:0,
-|  from
-/home/ubuntu/work/oe/openembedded-core/build/tmp-musl/work/i586-oe-linux-musl/console-tools/0.3.2-r8/console-tools-0.3.2/vttools/resizecons.c:81:
-|
-/home/ubuntu/work/oe/openembedded-core/build/tmp-musl/sysroots/qemux86/usr/include/bits/ioctl.h:90:8:
-note: originally defined here
-|  struct winsize {
-| ^
-
-Signed-off-by: Khem Raj 

-Upstream-Status: Pending
-
- vttools/resizecons.c | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/vttools/resizecons.c b/vttools/resizecons.c
-index 

[OE-core] [PATCH 09/11] p11-kit: convert to meson

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin 
---
 .../p11-kit/p11-kit_0.23.18.1.bb  | 26 +++
 1 file changed, 3 insertions(+), 23 deletions(-)

diff --git a/meta/recipes-support/p11-kit/p11-kit_0.23.18.1.bb 
b/meta/recipes-support/p11-kit/p11-kit_0.23.18.1.bb
index 9193a06ea80..dbf8fa2c468 100644
--- a/meta/recipes-support/p11-kit/p11-kit_0.23.18.1.bb
+++ b/meta/recipes-support/p11-kit/p11-kit_0.23.18.1.bb
@@ -2,7 +2,7 @@ SUMMARY = "Provides a way to load and enumerate PKCS#11 modules"
 LICENSE = "BSD-3-Clause"
 LIC_FILES_CHKSUM = "file://COPYING;md5=02933887f609807fbb57aa4237d14a50"
 
-inherit autotools gettext pkgconfig gtk-doc
+inherit meson gettext pkgconfig gtk-doc
 
 DEPENDS = "libtasn1 libffi"
 
@@ -10,30 +10,10 @@ SRC_URI = "git://github.com/p11-glue/p11-kit"
 SRCREV = "b0ebe7555c291808db29377ba79cb8326301f0a6"
 S = "${WORKDIR}/git"
 
-AUTOTOOLS_AUXDIR = "${S}/build/litter"
-
 PACKAGECONFIG ??= ""
-PACKAGECONFIG[trust-paths] = 
"--with-trust-paths=/etc/ssl/certs/ca-certificates.crt,--without-trust-paths,,ca-certificates"
-
-# This recipe does not use the standard gtk-doc m4 macros, and so the 
./configure flags
-# that control gtk-doc build are non-standard
-EXTRA_OECONF_prepend_class-target = "${@bb.utils.contains('GTKDOC_ENABLED', 
'True', '--enable-doc --enable-doc-html --disable-doc-pdf', \
-   
 '--disable-doc', d)} "
-
-# When building native recipes, disable gtkdoc, as it is not necessary,
-# pulls in additional dependencies, and makes build times longer
-EXTRA_OECONF_prepend_class-native = "--disable-doc "
-EXTRA_OECONF_prepend_class-nativesdk = "--disable-doc "
-
-UNKNOWN_CONFIGURE_WHITELIST_append = " --enable-gtk-doc-html 
--disable-gtk-doc-pdf --enable-gtk-doc --disable-gtk-doc"
+PACKAGECONFIG[trust-paths] = 
"-Dtrust_paths=/etc/ssl/certs/ca-certificates.crt,,,ca-certificates"
 
-# p11-kit relies on these two being copied from source tree
-# instead of being regenerated by gtkdoc-scan, but doesn't setup
-# dependencies correctly when there is a parallel build. Let's pre-copy
-# them instead.
-do_compile_prepend () {
-cp ${S}/doc/manual/p11-kit-overrides.txt 
${S}/doc/manual/p11-kit-sections.txt ${B}/doc/manual/
-}
+GTKDOC_MESON_OPTION = 'gtk_doc'
 
 FILES_${PN} += " \
 ${libdir}/p11-kit-proxy.so \
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 02/11] man-pages: update to 5.03

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin 
---
 .../man-pages/{man-pages_5.01.bb => man-pages_5.03.bb}| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
 rename meta/recipes-extended/man-pages/{man-pages_5.01.bb => 
man-pages_5.03.bb} (87%)

diff --git a/meta/recipes-extended/man-pages/man-pages_5.01.bb 
b/meta/recipes-extended/man-pages/man-pages_5.03.bb
similarity index 87%
rename from meta/recipes-extended/man-pages/man-pages_5.01.bb
rename to meta/recipes-extended/man-pages/man-pages_5.03.bb
index ea1249e4396..2381ba4348b 100644
--- a/meta/recipes-extended/man-pages/man-pages_5.01.bb
+++ b/meta/recipes-extended/man-pages/man-pages_5.03.bb
@@ -7,8 +7,8 @@ LICENSE = "GPLv2+"
 LIC_FILES_CHKSUM = "file://README;md5=794f701617cc03fe50c53257660d8ec4"
 SRC_URI = "${KERNELORG_MIRROR}/linux/docs/${BPN}/${BP}.tar.gz"
 
-SRC_URI[md5sum] = "38abead776a506109e128ab96bcbbe58"
-SRC_URI[sha256sum] = 
"070bef794c6826b3fb3965d1a2efdb46c25cb37c06c715987f88a50906cd5b6f"
+SRC_URI[md5sum] = "d1da6babb99cbd269233a35de9d8718e"
+SRC_URI[sha256sum] = 
"e8bab0b10c6fb3a6d69615edb488f8a27020f3625c4cf43b8451ba206acbb1b8"
 
 inherit manpages
 
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 06/11] e2fsprogs: fix build issues with the latest version of gettext

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin 
---
 ...-configure.ac-correct-AM_GNU_GETTEXT.patch | 26 ++
 ...to-use-gettext-defines-that-no-longe.patch | 27 +++
 ...-missing-check-for-permission-denied.patch |  7 ++---
 .../e2fsprogs/e2fsprogs/quiet-debugfs.patch   | 13 +++--
 .../e2fsprogs/e2fsprogs_1.45.4.bb | 12 +
 5 files changed, 73 insertions(+), 12 deletions(-)
 create mode 100644 
meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-configure.ac-correct-AM_GNU_GETTEXT.patch
 create mode 100644 
meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-intl-do-not-try-to-use-gettext-defines-that-no-longe.patch

diff --git 
a/meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-configure.ac-correct-AM_GNU_GETTEXT.patch
 
b/meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-configure.ac-correct-AM_GNU_GETTEXT.patch
new file mode 100644
index 000..2719594c2fa
--- /dev/null
+++ 
b/meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-configure.ac-correct-AM_GNU_GETTEXT.patch
@@ -0,0 +1,26 @@
+From f907fea9ea053d5433b5199ce74008664141f753 Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin 
+Date: Tue, 19 Nov 2019 17:32:33 +0100
+Subject: [PATCH] configure.ac: correct AM_GNU_GETTEXT
+
+New version of gettext require the 'external' argument.
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin 
+---
+ configure.ac | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/configure.ac b/configure.ac
+index 654ff43d..4772b97b 100644
+--- a/configure.ac
 b/configure.ac
+@@ -873,7 +873,7 @@ AC_SUBST(GETTEXT_PACKAGE)
+ AC_SUBST(PACKAGE)
+ AC_SUBST(VERSION)
+ 
+-AM_GNU_GETTEXT
++AM_GNU_GETTEXT([external])
+ dnl @MKDIR_P@ is expanded in AM_GNU_GETTEXT
+ AC_SUBST([mkdir_p],['$(MKDIR_P)'])
+ dnl
diff --git 
a/meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-intl-do-not-try-to-use-gettext-defines-that-no-longe.patch
 
b/meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-intl-do-not-try-to-use-gettext-defines-that-no-longe.patch
new file mode 100644
index 000..31b585fc9a9
--- /dev/null
+++ 
b/meta/recipes-devtools/e2fsprogs/e2fsprogs/0001-intl-do-not-try-to-use-gettext-defines-that-no-longe.patch
@@ -0,0 +1,27 @@
+From d034239f05cfba849f6ad16eb3f1ea37efbe4f95 Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin 
+Date: Tue, 19 Nov 2019 18:34:00 +0100
+Subject: [PATCH] intl: do not try to use gettext defines that no longer exist
+
+Newer version of gettext no longer define this in m4 files,
+so patch it out until e2fsprogs upstream sorts the situation.
+
+Upstream-Status: Inappropriate [hardcodes a choice]
+Signed-off-by: Alexander Kanavin 
+---
+ intl/Makefile.in | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/intl/Makefile.in b/intl/Makefile.in
+index e037e23c..89a876b8 100644
+--- a/intl/Makefile.in
 b/intl/Makefile.in
+@@ -175,7 +175,7 @@ libgnuintl.h.msvc-static libgnuintl.h.msvc-shared 
README.woe32 Makefile.msvc
+ DISTFILES.obsolete = xopen-msg.sed linux-msg.sed po2tbl.sed.in cat-compat.c \
+ COPYING.LIB-2 gettext.h libgettext.h plural-eval.c libgnuintl.h
+ 
+-all: all-@USE_INCLUDED_LIBINTL@
++all:
+ all-yes: libintl.$la libintl.h charset.alias ref-add.sed ref-del.sed
+ all-no: all-no-@BUILD_INCLUDED_LIBINTL@
+ all-no-yes: libgnuintl.$la
diff --git 
a/meta/recipes-devtools/e2fsprogs/e2fsprogs/e2fsprogs-fix-missing-check-for-permission-denied.patch
 
b/meta/recipes-devtools/e2fsprogs/e2fsprogs/e2fsprogs-fix-missing-check-for-permission-denied.patch
index 69c09bc1d76..4d335af4cf6 100644
--- 
a/meta/recipes-devtools/e2fsprogs/e2fsprogs/e2fsprogs-fix-missing-check-for-permission-denied.patch
+++ 
b/meta/recipes-devtools/e2fsprogs/e2fsprogs/e2fsprogs-fix-missing-check-for-permission-denied.patch
@@ -1,4 +1,4 @@
-From c7914309aeee6209ddb6995c084805a911d9cc82 Mon Sep 17 00:00:00 2001
+From e8331a76983e839a3d193446ab8ae9c1b09daa07 Mon Sep 17 00:00:00 2001
 From: Jackie Huang 
 Date: Wed, 10 Aug 2016 11:19:44 +0800
 Subject: [PATCH] Fix missing check for permission denied.
@@ -19,7 +19,7 @@ Signed-off-by: Jackie Huang 
  1 file changed, 1 insertion(+), 1 deletion(-)
 
 diff --git a/lib/support/profile.c b/lib/support/profile.c
-index 9e96673b..32d83002 100644
+index 585ed595..810dd66b 100644
 --- a/lib/support/profile.c
 +++ b/lib/support/profile.c
 @@ -335,7 +335,7 @@ profile_init(const char * const *files, profile_t 
*ret_profile)
@@ -31,6 +31,3 @@ index 9e96673b..32d83002 100644
   strcmp(*fs, default_filename))
goto errout;
  
--- 
-2.16.1
-
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs/quiet-debugfs.patch 
b/meta/recipes-devtools/e2fsprogs/e2fsprogs/quiet-debugfs.patch
index 830e9d57a5c..95e6a7a2d5c 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs/quiet-debugfs.patch
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs/quiet-debugfs.patch
@@ -1,14 +1,23 @@
+From de6d6f0dd010f5b9d917553acb9430278f448f23 Mon Sep 17 00:00:00 2001
+From: Ross Burton 
+Date: Mon, 23 Dec 2013 13:38:34 +

[OE-core] [PATCH 03/11] dummy-sdk-package.inc: do not include files into RREPLACES

[Alexander Kanavin]

rpm 4.15 no longer allows it, which makes sense.

Signed-off-by: Alexander Kanavin 
---
 meta/recipes-core/meta/dummy-sdk-package.inc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc 
b/meta/recipes-core/meta/dummy-sdk-package.inc
index 4d653706b13..58b0482eca5 100644
--- a/meta/recipes-core/meta/dummy-sdk-package.inc
+++ b/meta/recipes-core/meta/dummy-sdk-package.inc
@@ -16,11 +16,12 @@ python() {
 ALLOW_EMPTY_${PN} = "1"
 
 PR[vardeps] += "DUMMYPROVIDES"
+PR[vardeps] += "DUMMYPROVIDES_PACKAGES"
 
 python populate_packages_prepend() {
 p = d.getVar("PN")
 d.appendVar("RPROVIDES_%s" % p, "${DUMMYPROVIDES}")
 d.appendVar("RCONFLICTS_%s" % p, "${DUMMYPROVIDES}")
-d.appendVar("RREPLACES_%s" % p, "${DUMMYPROVIDES}")
+d.appendVar("RREPLACES_%s" % p, "${DUMMYPROVIDES_PACKAGES}")
 }
 
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 05/11] gettext: update to 0.20.1

[Alexander Kanavin]

License-Update: URLs changed from http to https
Signed-off-by: Alexander Kanavin 
---
 .../gettext-0.19.8.1/fix-CVE-2018-18751.patch | 141 
 .../gettext/gettext-0.19.8.1/parallel.patch   |  23 -
 .../gettext-0.19.8.1/use-pkgconfig.patch  | 321 
 .../add-with-bisonlocaledir.patch |   0
 ...nt.c-timsort.h-fix-formatting-issues.patch |  30 +-
 .../gettext/gettext-0.20.1/parallel.patch |  32 +
 .../run-ptest |   0
 .../serial-tests-config.patch |  26 +-
 .../gettext-0.20.1/use-pkgconfig.patch| 713 ++
 ...{gettext_0.19.8.1.bb => gettext_0.20.1.bb} |  23 +-
 10 files changed, 775 insertions(+), 534 deletions(-)
 delete mode 100644 
meta/recipes-core/gettext/gettext-0.19.8.1/fix-CVE-2018-18751.patch
 delete mode 100644 meta/recipes-core/gettext/gettext-0.19.8.1/parallel.patch
 delete mode 100644 
meta/recipes-core/gettext/gettext-0.19.8.1/use-pkgconfig.patch
 rename meta/recipes-core/gettext/{gettext-0.19.8.1 => 
gettext-0.20.1}/add-with-bisonlocaledir.patch (100%)
 rename meta/recipes-core/gettext/{gettext-0.19.8.1 => 
gettext-0.20.1}/cr-statement.c-timsort.h-fix-formatting-issues.patch (68%)
 create mode 100644 meta/recipes-core/gettext/gettext-0.20.1/parallel.patch
 rename meta/recipes-core/gettext/{gettext-0.19.8.1 => 
gettext-0.20.1}/run-ptest (100%)
 rename meta/recipes-core/gettext/{gettext-0.19.8.1 => 
gettext-0.20.1}/serial-tests-config.patch (75%)
 create mode 100644 meta/recipes-core/gettext/gettext-0.20.1/use-pkgconfig.patch
 rename meta/recipes-core/gettext/{gettext_0.19.8.1.bb => gettext_0.20.1.bb} 
(94%)

diff --git 
a/meta/recipes-core/gettext/gettext-0.19.8.1/fix-CVE-2018-18751.patch 
b/meta/recipes-core/gettext/gettext-0.19.8.1/fix-CVE-2018-18751.patch
deleted file mode 100644
index 6dfe200d654..000
--- a/meta/recipes-core/gettext/gettext-0.19.8.1/fix-CVE-2018-18751.patch
+++ /dev/null
@@ -1,141 +0,0 @@
-Upstream-Status: Backport 
[https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=commit;h=dce3a16]
-CVE: CVE-2018-18751
-
-Signed-off-by: Kai Kang 
-
-From dce3a16e5e9368245735e29bf498dcd5e3e474a4 Mon Sep 17 00:00:00 2001
-From: Daiki Ueno 
-Date: Thu, 15 Sep 2016 13:57:24 +0200
-Subject: [PATCH] xgettext: Fix crash with *.po file input
-
-When xgettext was given two *.po files with the same msgid_plural, it
-crashed with double-free.  Problem reported by Davlet Panech in:
-http://lists.gnu.org/archive/html/bug-gettext/2016-09/msg1.html
-* gettext-tools/src/po-gram-gen.y: Don't free msgid_pluralform after
-calling do_callback_message, assuming that it takes ownership.
-* gettext-tools/src/read-catalog.c (default_add_message): Free
-msgid_plural after calling message_alloc.
-* gettext-tools/tests/xgettext-po-2: New file.
-* gettext-tools/tests/Makefile.am (TESTS): Add new test.

- gettext-tools/src/po-gram-gen.y   | 13 -
- gettext-tools/src/read-catalog.c  |  2 ++
- gettext-tools/tests/Makefile.am   |  2 +-
- gettext-tools/tests/xgettext-po-2 | 55 +++
- 4 files changed, 63 insertions(+), 9 deletions(-)
- create mode 100755 gettext-tools/tests/xgettext-po-2
-
-diff --git a/gettext-tools/src/po-gram-gen.y b/gettext-tools/src/po-gram-gen.y
-index becf5e6..4428e77 100644
 a/gettext-tools/src/po-gram-gen.y
-+++ b/gettext-tools/src/po-gram-gen.y
-@@ -221,14 +221,11 @@ message
-   check_obsolete ($1, $3);
-   check_obsolete ($1, $4);
-   if (!$1.obsolete || pass_obsolete_entries)
--{
--  do_callback_message ($1.ctxt, string2, &$1.pos, 
$3.string,
--   $4.rhs.msgstr, $4.rhs.msgstr_len, 
&$4.pos,
--   $1.prev_ctxt,
--   $1.prev_id, $1.prev_id_plural,
--   $1.obsolete);
--  free ($3.string);
--}
-+do_callback_message ($1.ctxt, string2, &$1.pos, $3.string,
-+ $4.rhs.msgstr, $4.rhs.msgstr_len, 
&$4.pos,
-+ $1.prev_ctxt,
-+ $1.prev_id, $1.prev_id_plural,
-+ $1.obsolete);
-   else
- {
-   free_message_intro ($1);
-diff --git a/gettext-tools/src/read-catalog.c 
b/gettext-tools/src/read-catalog.c
-index 571d18e..6af6d20 100644
 a/gettext-tools/src/read-catalog.c
-+++ b/gettext-tools/src/read-catalog.c
-@@ -397,6 +397,8 @@ default_add_message (default_catalog_reader_ty *this,
-  appropriate.  */
-   mp = message_alloc (msgctxt, msgid, msgid_plural, msgstr, msgstr_len,
-   msgstr_pos);
-+  if (msgid_plural != NULL)
-+free (msgid_plural);
-   mp->prev_msgctxt = prev_msgctxt;
- 

[OE-core] [PATCH 01/11] man-pages: correct the SRC_URI

[Alexander Kanavin]

The homepage: https://www.kernel.org/doc/man-pages/download.html
specifies that the latest tarballs are published one level up
from where we currently take them.

Signed-off-by: Alexander Kanavin 
---
 meta/recipes-extended/man-pages/man-pages_5.01.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-extended/man-pages/man-pages_5.01.bb 
b/meta/recipes-extended/man-pages/man-pages_5.01.bb
index 28525f4ba7e..ea1249e4396 100644
--- a/meta/recipes-extended/man-pages/man-pages_5.01.bb
+++ b/meta/recipes-extended/man-pages/man-pages_5.01.bb
@@ -5,7 +5,7 @@ HOMEPAGE = "http://www.kernel.org/pub/linux/docs/man-pages;
 LICENSE = "GPLv2+"
 
 LIC_FILES_CHKSUM = "file://README;md5=794f701617cc03fe50c53257660d8ec4"
-SRC_URI = "${KERNELORG_MIRROR}/linux/docs/${BPN}/Archive/${BP}.tar.gz"
+SRC_URI = "${KERNELORG_MIRROR}/linux/docs/${BPN}/${BP}.tar.gz"
 
 SRC_URI[md5sum] = "38abead776a506109e128ab96bcbbe58"
 SRC_URI[sha256sum] = 
"070bef794c6826b3fb3965d1a2efdb46c25cb37c06c715987f88a50906cd5b6f"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] alsa-lib:upgrade 1.1.9 -> 1.2.1.1

[Tanu Kaskinen]

Hi, I'm still around. Yes, the moved UCM conf files need to be
packaged. (And no doubt the topology files too, although that's an area
I understand less well).

-- Tanu


On Wed, 2019-11-20 at 13:10 +0100, Alexander Kanavin wrote:
> If Tanu is still around, he should probably chime in.
> 
> Alex
> 
> On Wed, 20 Nov 2019 at 12:55, Mittal, Anuj  wrote:
> 
> > This version has moved ucm and topology conf files to separate
> > repositories. We should also probably fetch and package those.
> > 
> > 
> > https://github.com/alsa-project/alsa-lib/commit/1f37ba2a2b1c07c04de1c368cd4943f712cccfa5
> > 
> > 
> > https://github.com/alsa-project/alsa-lib/commit/cdc8aacbb57c82a81e3b398f2ede6440d49334a5
> > 
> > Thanks,
> > 
> > Anuj
> > 
> > On Wed, 2019-11-20 at 13:17 +0800, Zang Ruochen wrote:
> > > The following error occurred during the steps of do-configure:
> > > > Fixing libtool for -rpath problems.
> > > > ../alsa-lib-1.2.1.1/configure: line 16567: libtool: No such file or
> > > > directory
> > > > mv: cannot stat 'libtool-2': No such file or directory
> > > > chmod: cannot access 'libtool': No such file or directory
> > > 
> > > add alsa-lib/0001-Fix-problem-with-libtool-not-found.patch to fix it.
> > > 
> > > Signed-off-by: Zang Ruochen 
> > > ---
> > >  ...1-Fix-problem-with-libtool-not-found.patch | 31
> > > +++
> > >  ...{alsa-lib_1.1.9.bb => alsa-lib_1.2.1.1.bb} |  8 +++--
> > >  2 files changed, 36 insertions(+), 3 deletions(-)
> > >  create mode 100644 meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-
> > > problem-with-libtool-not-found.patch
> > >  rename meta/recipes-multimedia/alsa/{alsa-lib_1.1.9.bb => alsa-
> > > lib_1.2.1.1.bb} (83%)
> > > 
> > > diff --git a/meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-problem-
> > > with-libtool-not-found.patch b/meta/recipes-multimedia/alsa/alsa-
> > > lib/0001-Fix-problem-with-libtool-not-found.patch
> > > new file mode 100644
> > > index 00..5fb84cd6f8
> > > --- /dev/null
> > > +++ b/meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-problem-with-
> > > libtool-not-found.patch
> > > @@ -0,0 +1,31 @@
> > > +From f419276a716daf8d98846b3df50995e5ac030134 Mon Sep 17 00:00:00
> > > 2001
> > > +From: Zang Ruochen 
> > > +Date: Wed, 20 Nov 2019 11:24:17 +0800
> > > +Subject: [PATCH] Fix problem with libtool not found
> > > +
> > > +Upstream-Status: Inappropriate
> > > +Signed-off-by: Zang Ruochen 
> > > +---
> > > + configure.ac | 6 +++---
> > > + 1 file changed, 3 insertions(+), 3 deletions(-)
> > > +
> > > +diff --git a/configure.ac b/configure.ac
> > > +index 35eb315..548b9ea 100644
> > > +--- a/configure.ac
> > >  b/configure.ac
> > > +@@ -769,9 +769,9 @@ dnl Taken from
> > > https://wiki.debian.org/RpathIssue
> > > + case $host in
> > > +*-*-linux-gnu)
> > > +AC_MSG_RESULT([Fixing libtool for -rpath problems.])
> > > +-   sed < libtool > libtool-2 \
> > > ++   sed < "$srcdir"/libtool > "$srcdir"/libtool-2 \
> > > +  's/^hardcode_libdir_flag_spec.*$'/'hardcode_libdir_flag_spec="
> > > -D__LIBTOOL_IS_A_FOOL__ "/'
> > > +-   mv libtool-2 libtool
> > > +-   chmod 755 libtool
> > > ++   mv "$srcdir"/libtool-2 "$srcdir"/libtool
> > > ++   chmod 755 "$srcdir"/libtool
> > > +  ;;
> > > + esac
> > > +--
> > > +2.20.1
> > > +
> > > diff --git a/meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> > > b/meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> > > similarity index 83%
> > > rename from meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> > > rename to meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> > > index 2c63ee7522..56544b0d45 100644
> > > --- a/meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> > > +++ b/meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> > > @@ -7,9 +7,11 @@ LIC_FILES_CHKSUM = "
> > > file://COPYING;md5=a916467b91076e631dd8edb7424769c7 \
> > >  file://src/socket.c;md5=dd1bc7f44872690224d89c1a
> > > 9806e495;beginline=1;endline=26 \
> > >  "
> > > 
> > > -SRC_URI = "https://www.alsa-project.org/files/pub/lib/${BP}.tar.bz2;
> > > -SRC_URI[md5sum] = "e6d429dbdcfaa0f034d907fa6dc3735e"
> > > -SRC_URI[sha256sum] =
> > > "488373aef5396682f3a411a6d064ae0ad196b9c96269d0bb912fbdeec94b994b"
> > > +SRC_URI = "https://www.alsa-project.org/files/pub/lib/${BP}.tar.bz2
> > > \
> > > +   file://0001-Fix-problem-with-libtool-not-found.patch \
> > > +  "
> > > +SRC_URI[md5sum] = "1d428f46d6160f5e4bbdd64a2ff34f45"
> > > +SRC_URI[sha256sum] =
> > > "c95ac63c0aad43a6ac457d960569096b0b2ef72dc4e3737e77e3e2de87022cec"
> > > 
> > >  inherit autotools pkgconfig
> > > 
> > > --
> > > 2.20.1
> > > 
> > > 
> > > 
> > --
> > ___
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org

Re: [OE-core] [PATCH 01/13] python: update to 2.7.17

[Tom Rini]

On Mon, Nov 18, 2019 at 02:07:49PM +0100, Alexander Kanavin wrote:

> Drop backports, rebase a couple of patches.
> 
> This is the second last release of py 2.x; upstream support ends on
> 1 January 2020, there will be one final 2.x afterwards.
> 
> Note that the only thing that still needs python 2.x in oe-core is
> u-boot; when the next u-boot update arrives, we should find out
> where the py3 migration is for that component before merging the
> update.
> 
> Signed-off-by: Alexander Kanavin 

I would _really_ like to see someone move master to using v2020.01-rc3
or so for U-Boot as that should not have any Python 2 dependencies left.

-- 
Tom


signature.asc
Description: PGP signature
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v5 2/2] image_types: add Zstandard conversion support

[Stefan Agner]

From: Stefan Agner 

Add Zstandard (or just Zstd) compression support. This allows to
create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.

Signed-off-by: Stefan Agner 
---
 meta/classes/image_types.bbclass | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 2eeffbb366..de00492932 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
 
+ZSTD_COMPRESSION_LEVEL ?= "-3"
+
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime 
--output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 
${EXTRA_IMAGECMD}"
 
@@ -269,7 +271,7 @@ IMAGE_TYPES = " \
 hddimg \
 squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
 ubi ubifs multiubi \
-tar tar.gz tar.bz2 tar.xz tar.lz4 \
+tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
 cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
 wic wic.gz wic.bz2 wic.lzma \
 container \
@@ -282,7 +284,7 @@ IMAGE_TYPES = " \
 # CONVERSION_CMD/DEPENDS.
 COMPRESSIONTYPES ?= ""
 
-CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum 
sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
${COMPRESSIONTYPES}"
+CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum sha224sum 
sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
${COMPRESSIONTYPES}"
 CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -290,6 +292,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} 
${XZ_DEFAULTS} --check=
 CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
 CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD_zst = "zstd -f -k -T0 -c ${ZSTD_COMPRESSION_LEVEL} 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
 CONVERSION_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
 CONVERSION_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
 CONVERSION_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
@@ -310,6 +313,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
 CONVERSION_DEPENDS_lz4 = "lz4-native"
 CONVERSION_DEPENDS_lzo = "lzop-native"
 CONVERSION_DEPENDS_zip = "zip-native"
+CONVERSION_DEPENDS_zst = "zstd-native"
 CONVERSION_DEPENDS_sum = "mtd-utils-native"
 CONVERSION_DEPENDS_bmap = "bmap-tools-native"
 CONVERSION_DEPENDS_u-boot = "u-boot-tools-native"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v5 1/2] zstd: move recipe to oe-core from meta-oe

[Stefan Agner]

From: Stefan Agner 

Zstd is a very fast compression algorithm and has gained quite some
popularity recently. An upcoming patch allows to compress rootfs
using Zstd. Moving Zstd to oe-core allows this new conversion formats
to be automatically tested by the oe-selftest
imagefeatures.ImageFeatures.test_image_fstypes test.

Furthermore there are three packages in oe-core which allow to enable
Zstd support through packageconfig currently.

Signed-off-by: Stefan Agner 
---
 meta/conf/distro/include/maintainers.inc |  1 +
 meta/recipes-extended/zstd/zstd_1.4.4.bb | 35 
 2 files changed, 36 insertions(+)
 create mode 100644 meta/recipes-extended/zstd/zstd_1.4.4.bb

diff --git a/meta/conf/distro/include/maintainers.inc 
b/meta/conf/distro/include/maintainers.inc
index d85e5b697f..dff7ce28f8 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -774,3 +774,4 @@ RECIPE_MAINTAINER_pn-xwininfo = "Armin Kuster 
"
 RECIPE_MAINTAINER_pn-xz = "Denys Dmytriyenko "
 RECIPE_MAINTAINER_pn-zip = "Denys Dmytriyenko "
 RECIPE_MAINTAINER_pn-zlib = "Denys Dmytriyenko "
+RECIPE_MAINTAINER_pn-zstd = "Alex Kiernan "
diff --git a/meta/recipes-extended/zstd/zstd_1.4.4.bb 
b/meta/recipes-extended/zstd/zstd_1.4.4.bb
new file mode 100644
index 00..eb201f4139
--- /dev/null
+++ b/meta/recipes-extended/zstd/zstd_1.4.4.bb
@@ -0,0 +1,35 @@
+SUMMARY = "Zstandard - Fast real-time compression algorithm"
+DESCRIPTION = "Zstandard is a fast lossless compression algorithm, targeting \
+real-time compression scenarios at zlib-level and better compression ratios. \
+It's backed by a very fast entropy stage, provided by Huff0 and FSE library."
+HOMEPAGE = "http://www.zstd.net/;
+SECTION = "console/utils"
+
+LICENSE = "BSD-3-Clause & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=c7f0b161edbe52f5f345a3d1311d0b32 \
+file://COPYING;md5=39bba7d2cf0ba1036f2a6e2be52fe3f0"
+
+SRC_URI = "git://github.com/facebook/zstd.git;nobranch=1"
+
+SRCREV = "10f0e6993f9d2f682da6d04aa2385b7d53cbb4ee"
+UPSTREAM_CHECK_GITTAGREGEX = "v(?P\d+(\.\d+)+)"
+
+S = "${WORKDIR}/git"
+
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[lz4] = "HAVE_LZ4=1,HAVE_LZ4=0,lz4"
+PACKAGECONFIG[lzma] = "HAVE_LZMA=1,HAVE_LZMA=0,xz"
+PACKAGECONFIG[zlib] = "HAVE_ZLIB=1,HAVE_ZLIB=0,zlib"
+
+# See programs/README.md for how to use this
+ZSTD_LEGACY_SUPPORT ??= "4"
+
+do_compile () {
+oe_runmake ${PACKAGECONFIG_CONFARGS} 
ZSTD_LEGACY_SUPPORT=${ZSTD_LEGACY_SUPPORT}
+}
+
+do_install () {
+oe_runmake install 'DESTDIR=${D}'
+}
+
+BBCLASSEXTEND = "native nativesdk"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4 2/2] image_types: add Zstandard conversion support

[Stefan Agner]

On 2019-11-20 11:49, Alex Kiernan wrote:
> On Wed, Nov 20, 2019 at 9:13 AM Stefan Agner  wrote:
>>
>> From: Stefan Agner 
>>
>> Add Zstandard (or just Zstd) compression support. This allows to
>> create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.
>>
>> Signed-off-by: Stefan Agner 
>> ---
>>  meta/classes/image_types.bbclass | 8 ++--
>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/image_types.bbclass 
>> b/meta/classes/image_types.bbclass
>> index 2eeffbb366..d29b9c5787 100644
>> --- a/meta/classes/image_types.bbclass
>> +++ b/meta/classes/image_types.bbclass
>> @@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
>>
>>  ZIP_COMPRESSION_LEVEL ?= "-9"
>>
>> +ZSTD_COMPRESSION_LEVEL ?= "-3"
>> +
>>  JFFS2_SUM_EXTRA_ARGS ?= ""
>>  IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime 
>> --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 
>> ${EXTRA_IMAGECMD}"
>>
>> @@ -269,7 +271,7 @@ IMAGE_TYPES = " \
>>  hddimg \
>>  squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
>>  ubi ubifs multiubi \
>> -tar tar.gz tar.bz2 tar.xz tar.lz4 \
>> +tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
>>  cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
>>  wic wic.gz wic.bz2 wic.lzma \
>>  container \
>> @@ -282,7 +284,7 @@ IMAGE_TYPES = " \
>>  # CONVERSION_CMD/DEPENDS.
>>  COMPRESSIONTYPES ?= ""
>>
>> -CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum 
>> sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
>> ${COMPRESSIONTYPES}"
>> +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum 
>> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
>> ${COMPRESSIONTYPES}"
>>  CONVERSION_CMD_lzma = "lzma -k -f -7 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>  CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>>  CONVERSION_CMD_bz2 = "pbzip2 -f -k 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>> @@ -290,6 +292,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} 
>> ${XZ_DEFAULTS} --check=
>>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>>  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>>  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>> +CONVERSION_CMD_zst = "zstd -f -k -c ${ZSTD_COMPRESSION_LEVEL} 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
>> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
> 
> Add -T0 so we default to parallel?

Sensible, will send a v5.

--
Stefan
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] alsa-lib:upgrade 1.1.9 -> 1.2.1.1

[Alexander Kanavin]

If Tanu is still around, he should probably chime in.

Alex

On Wed, 20 Nov 2019 at 12:55, Mittal, Anuj  wrote:

> This version has moved ucm and topology conf files to separate
> repositories. We should also probably fetch and package those.
>
>
> https://github.com/alsa-project/alsa-lib/commit/1f37ba2a2b1c07c04de1c368cd4943f712cccfa5
>
>
> https://github.com/alsa-project/alsa-lib/commit/cdc8aacbb57c82a81e3b398f2ede6440d49334a5
>
> Thanks,
>
> Anuj
>
> On Wed, 2019-11-20 at 13:17 +0800, Zang Ruochen wrote:
> > The following error occurred during the steps of do-configure:
> > > Fixing libtool for -rpath problems.
> > > ../alsa-lib-1.2.1.1/configure: line 16567: libtool: No such file or
> > > directory
> > > mv: cannot stat 'libtool-2': No such file or directory
> > > chmod: cannot access 'libtool': No such file or directory
> >
> > add alsa-lib/0001-Fix-problem-with-libtool-not-found.patch to fix it.
> >
> > Signed-off-by: Zang Ruochen 
> > ---
> >  ...1-Fix-problem-with-libtool-not-found.patch | 31
> > +++
> >  ...{alsa-lib_1.1.9.bb => alsa-lib_1.2.1.1.bb} |  8 +++--
> >  2 files changed, 36 insertions(+), 3 deletions(-)
> >  create mode 100644 meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-
> > problem-with-libtool-not-found.patch
> >  rename meta/recipes-multimedia/alsa/{alsa-lib_1.1.9.bb => alsa-
> > lib_1.2.1.1.bb} (83%)
> >
> > diff --git a/meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-problem-
> > with-libtool-not-found.patch b/meta/recipes-multimedia/alsa/alsa-
> > lib/0001-Fix-problem-with-libtool-not-found.patch
> > new file mode 100644
> > index 00..5fb84cd6f8
> > --- /dev/null
> > +++ b/meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-problem-with-
> > libtool-not-found.patch
> > @@ -0,0 +1,31 @@
> > +From f419276a716daf8d98846b3df50995e5ac030134 Mon Sep 17 00:00:00
> > 2001
> > +From: Zang Ruochen 
> > +Date: Wed, 20 Nov 2019 11:24:17 +0800
> > +Subject: [PATCH] Fix problem with libtool not found
> > +
> > +Upstream-Status: Inappropriate
> > +Signed-off-by: Zang Ruochen 
> > +---
> > + configure.ac | 6 +++---
> > + 1 file changed, 3 insertions(+), 3 deletions(-)
> > +
> > +diff --git a/configure.ac b/configure.ac
> > +index 35eb315..548b9ea 100644
> > +--- a/configure.ac
> >  b/configure.ac
> > +@@ -769,9 +769,9 @@ dnl Taken from
> > https://wiki.debian.org/RpathIssue
> > + case $host in
> > +*-*-linux-gnu)
> > +AC_MSG_RESULT([Fixing libtool for -rpath problems.])
> > +-   sed < libtool > libtool-2 \
> > ++   sed < "$srcdir"/libtool > "$srcdir"/libtool-2 \
> > +  's/^hardcode_libdir_flag_spec.*$'/'hardcode_libdir_flag_spec="
> > -D__LIBTOOL_IS_A_FOOL__ "/'
> > +-   mv libtool-2 libtool
> > +-   chmod 755 libtool
> > ++   mv "$srcdir"/libtool-2 "$srcdir"/libtool
> > ++   chmod 755 "$srcdir"/libtool
> > +  ;;
> > + esac
> > +--
> > +2.20.1
> > +
> > diff --git a/meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> > b/meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> > similarity index 83%
> > rename from meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> > rename to meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> > index 2c63ee7522..56544b0d45 100644
> > --- a/meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> > +++ b/meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> > @@ -7,9 +7,11 @@ LIC_FILES_CHKSUM = "
> > file://COPYING;md5=a916467b91076e631dd8edb7424769c7 \
> >  file://src/socket.c;md5=dd1bc7f44872690224d89c1a
> > 9806e495;beginline=1;endline=26 \
> >  "
> >
> > -SRC_URI = "https://www.alsa-project.org/files/pub/lib/${BP}.tar.bz2;
> > -SRC_URI[md5sum] = "e6d429dbdcfaa0f034d907fa6dc3735e"
> > -SRC_URI[sha256sum] =
> > "488373aef5396682f3a411a6d064ae0ad196b9c96269d0bb912fbdeec94b994b"
> > +SRC_URI = "https://www.alsa-project.org/files/pub/lib/${BP}.tar.bz2
> > \
> > +   file://0001-Fix-problem-with-libtool-not-found.patch \
> > +  "
> > +SRC_URI[md5sum] = "1d428f46d6160f5e4bbdd64a2ff34f45"
> > +SRC_URI[sha256sum] =
> > "c95ac63c0aad43a6ac457d960569096b0b2ef72dc4e3737e77e3e2de87022cec"
> >
> >  inherit autotools pkgconfig
> >
> > --
> > 2.20.1
> >
> >
> >
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 01/13] python: update to 2.7.17

[Ross Burton]

On 19/11/2019 16:30, Adrian Bunk wrote:

You want meta-oe to have a LAYERDEPENDS on a new python2 layer that is
not in meta-openembedded?


Would that be a bad thing?  Layers shouldn't care about where they are 
hosted.


My ideal would be that meta-python2 exists with both the python2 recipes 
and all of the py2 parts of meta-python.  Until meta-oe is Py3-only, it 
can either depend on this layer or recipes are blacklisted unless the 
layer is present.


Ross
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] alsa-lib:upgrade 1.1.9 -> 1.2.1.1

[Mittal, Anuj]

This version has moved ucm and topology conf files to separate
repositories. We should also probably fetch and package those.

https://github.com/alsa-project/alsa-lib/commit/1f37ba2a2b1c07c04de1c368cd4943f712cccfa5

https://github.com/alsa-project/alsa-lib/commit/cdc8aacbb57c82a81e3b398f2ede6440d49334a5

Thanks,

Anuj

On Wed, 2019-11-20 at 13:17 +0800, Zang Ruochen wrote:
> The following error occurred during the steps of do-configure:
> > Fixing libtool for -rpath problems.
> > ../alsa-lib-1.2.1.1/configure: line 16567: libtool: No such file or
> > directory
> > mv: cannot stat 'libtool-2': No such file or directory
> > chmod: cannot access 'libtool': No such file or directory
> 
> add alsa-lib/0001-Fix-problem-with-libtool-not-found.patch to fix it.
> 
> Signed-off-by: Zang Ruochen 
> ---
>  ...1-Fix-problem-with-libtool-not-found.patch | 31
> +++
>  ...{alsa-lib_1.1.9.bb => alsa-lib_1.2.1.1.bb} |  8 +++--
>  2 files changed, 36 insertions(+), 3 deletions(-)
>  create mode 100644 meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-
> problem-with-libtool-not-found.patch
>  rename meta/recipes-multimedia/alsa/{alsa-lib_1.1.9.bb => alsa-
> lib_1.2.1.1.bb} (83%)
> 
> diff --git a/meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-problem-
> with-libtool-not-found.patch b/meta/recipes-multimedia/alsa/alsa-
> lib/0001-Fix-problem-with-libtool-not-found.patch
> new file mode 100644
> index 00..5fb84cd6f8
> --- /dev/null
> +++ b/meta/recipes-multimedia/alsa/alsa-lib/0001-Fix-problem-with-
> libtool-not-found.patch
> @@ -0,0 +1,31 @@
> +From f419276a716daf8d98846b3df50995e5ac030134 Mon Sep 17 00:00:00
> 2001
> +From: Zang Ruochen 
> +Date: Wed, 20 Nov 2019 11:24:17 +0800
> +Subject: [PATCH] Fix problem with libtool not found
> +
> +Upstream-Status: Inappropriate
> +Signed-off-by: Zang Ruochen 
> +---
> + configure.ac | 6 +++---
> + 1 file changed, 3 insertions(+), 3 deletions(-)
> +
> +diff --git a/configure.ac b/configure.ac
> +index 35eb315..548b9ea 100644
> +--- a/configure.ac
>  b/configure.ac
> +@@ -769,9 +769,9 @@ dnl Taken from 
> https://wiki.debian.org/RpathIssue
> + case $host in
> +*-*-linux-gnu)
> +AC_MSG_RESULT([Fixing libtool for -rpath problems.])
> +-   sed < libtool > libtool-2 \
> ++   sed < "$srcdir"/libtool > "$srcdir"/libtool-2 \
> +  's/^hardcode_libdir_flag_spec.*$'/'hardcode_libdir_flag_spec="
> -D__LIBTOOL_IS_A_FOOL__ "/'
> +-   mv libtool-2 libtool
> +-   chmod 755 libtool
> ++   mv "$srcdir"/libtool-2 "$srcdir"/libtool
> ++   chmod 755 "$srcdir"/libtool
> +  ;;
> + esac
> +-- 
> +2.20.1
> +
> diff --git a/meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> b/meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> similarity index 83%
> rename from meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> rename to meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> index 2c63ee7522..56544b0d45 100644
> --- a/meta/recipes-multimedia/alsa/alsa-lib_1.1.9.bb
> +++ b/meta/recipes-multimedia/alsa/alsa-lib_1.2.1.1.bb
> @@ -7,9 +7,11 @@ LIC_FILES_CHKSUM = "
> file://COPYING;md5=a916467b91076e631dd8edb7424769c7 \
>  file://src/socket.c;md5=dd1bc7f44872690224d89c1a
> 9806e495;beginline=1;endline=26 \
>  "
>  
> -SRC_URI = "https://www.alsa-project.org/files/pub/lib/${BP}.tar.bz2;
> -SRC_URI[md5sum] = "e6d429dbdcfaa0f034d907fa6dc3735e"
> -SRC_URI[sha256sum] =
> "488373aef5396682f3a411a6d064ae0ad196b9c96269d0bb912fbdeec94b994b"
> +SRC_URI = "https://www.alsa-project.org/files/pub/lib/${BP}.tar.bz2
> \
> +   file://0001-Fix-problem-with-libtool-not-found.patch \
> +  "
> +SRC_URI[md5sum] = "1d428f46d6160f5e4bbdd64a2ff34f45"
> +SRC_URI[sha256sum] =
> "c95ac63c0aad43a6ac457d960569096b0b2ef72dc4e3737e77e3e2de87022cec"
>  
>  inherit autotools pkgconfig
>  
> -- 
> 2.20.1
> 
> 
> 
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] pseudo: Update checksum of the license file

[shohei.maruyama]

On Wed, 2019-11-20 at 11:39 +, shohei.maruy...@cybertrust.co.jp wrote:
> On Wed, 2019-11-20 at 12:23 +0100, Alexander Kanavin wrote:
> > On Wed, 20 Nov 2019 at 07:26,  wrote:
> > > Recently pseudo.inc has been updated to latest master
> > > (d50b9c511e17c6f582e
> > > f39b61eaaa8d4555eabba), but checksum of the license file did not
> > > change.
> > > This commit updates the checksum to latest one.
> > 
> > I am not sure I can make sense of this. When the checksum of the file
> > changes upstream, bitbake will complain and refuse to proceed, which is
> > not happening with the current master branch.
> > What error message are you seeing and what commit are you on?
> > 
> > Alex
> 
> Hi Alex,
> 
> Thank you for replying.
> I'm sorry for I forgot to describe my situation. I'm on
> c9a1a608f5146e5dcfebc8e566cb7f6388a3986f (warrior).
> 
> Regards,
> 
> Shohei Maruyama

Please ignore this patch. The problem is not relating poky, relating other
layer.

Regards,

Shohei Maruyama

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] pseudo: Update checksum of the license file

[shohei.maruyama]

On Wed, 2019-11-20 at 12:23 +0100, Alexander Kanavin wrote:
On Wed, 20 Nov 2019 at 07:26, 
mailto:shohei.maruy...@cybertrust.co.jp>> 
wrote:
Recently pseudo.inc has been updated to latest master (d50b9c511e17c6f582e
f39b61eaaa8d4555eabba), but checksum of the license file did not change.
This commit updates the checksum to latest one.

I am not sure I can make sense of this. When the checksum of the file changes 
upstream, bitbake will complain and refuse to proceed, which is not happening 
with the current master branch.
What error message are you seeing and what commit are you on?

Alex

Hi Alex,

Thank you for replying.
I'm sorry for I forgot to describe my situation. I'm on 
c9a1a608f5146e5dcfebc8e566cb7f6388a3986f (warrior).

Regards,

Shohei Maruyama
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] pseudo: Update checksum of the license file

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 07:26,  wrote:

> Recently pseudo.inc has been updated to latest master (d50b9c511e17c6f582e
> f39b61eaaa8d4555eabba), but checksum of the license file did not change.
> This commit updates the checksum to latest one.
>

I am not sure I can make sense of this. When the checksum of the file
changes upstream, bitbake will complain and refuse to proceed, which is not
happening with the current master branch.
What error message are you seeing and what commit are you on?

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] alsa-lib:upgrade 1.1.9 -> 1.2.1.1

[Alexander Kanavin]

On Wed, 20 Nov 2019 at 06:17, Zang Ruochen 
wrote:

> +From f419276a716daf8d98846b3df50995e5ac030134 Mon Sep 17 00:00:00 2001
> +From: Zang Ruochen 
> +Date: Wed, 20 Nov 2019 11:24:17 +0800
> +Subject: [PATCH] Fix problem with libtool not found
> +
> +Upstream-Status: Inappropriate
>

Why inappropriate? To me this seems like a real problem and should be
submitted upstream.

Alex
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4 2/2] image_types: add Zstandard conversion support

[Alex Kiernan]

On Wed, Nov 20, 2019 at 9:13 AM Stefan Agner  wrote:
>
> From: Stefan Agner 
>
> Add Zstandard (or just Zstd) compression support. This allows to
> create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.
>
> Signed-off-by: Stefan Agner 
> ---
>  meta/classes/image_types.bbclass | 8 ++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/image_types.bbclass 
> b/meta/classes/image_types.bbclass
> index 2eeffbb366..d29b9c5787 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
>
>  ZIP_COMPRESSION_LEVEL ?= "-9"
>
> +ZSTD_COMPRESSION_LEVEL ?= "-3"
> +
>  JFFS2_SUM_EXTRA_ARGS ?= ""
>  IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime 
> --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 
> ${EXTRA_IMAGECMD}"
>
> @@ -269,7 +271,7 @@ IMAGE_TYPES = " \
>  hddimg \
>  squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
>  ubi ubifs multiubi \
> -tar tar.gz tar.bz2 tar.xz tar.lz4 \
> +tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
>  cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
>  wic wic.gz wic.bz2 wic.lzma \
>  container \
> @@ -282,7 +284,7 @@ IMAGE_TYPES = " \
>  # CONVERSION_CMD/DEPENDS.
>  COMPRESSIONTYPES ?= ""
>
> -CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum 
> sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
> ${COMPRESSIONTYPES}"
> +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum 
> sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
> ${COMPRESSIONTYPES}"
>  CONVERSION_CMD_lzma = "lzma -k -f -7 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>  CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> @@ -290,6 +292,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} 
> ${XZ_DEFAULTS} --check=
>  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
>  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> +CONVERSION_CMD_zst = "zstd -f -k -c ${ZSTD_COMPRESSION_LEVEL} 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
> ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"

Add -T0 so we default to parallel?

-- 
Alex Kiernan
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4 2/2] image_types: add Zstandard conversion support

[Alex Kiernan]

On Wed, Nov 20, 2019 at 10:32 AM Adrian Bunk  wrote:
>
> On Wed, Nov 20, 2019 at 09:12:58AM +, Stefan Agner wrote:
> >...
> > --- a/meta/classes/image_types.bbclass
> > +++ b/meta/classes/image_types.bbclass
> > @@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
> >
> >  ZIP_COMPRESSION_LEVEL ?= "-9"
> >
> > +ZSTD_COMPRESSION_LEVEL ?= "-3"
> > +
> >...
>
> Why -3 and not -19 ?
>
> Compression speed is not a real issue, and AFAIR decompression
> tends to be slightly faster with smaller compressed size.
> Plus the benefit that smaller images are faster to load
> from network/storage.
>

I've been carrying similar functionality locally and at some point
pushed to `--ultra -22` and with a large image (~2GiB) the compression
stage then took literally hours. Not sure I tested with `-19` though

-- 
Alex Kiernan
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4 2/2] image_types: add Zstandard conversion support

[Adrian Bunk]

On Wed, Nov 20, 2019 at 09:12:58AM +, Stefan Agner wrote:
>...
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
>  
>  ZIP_COMPRESSION_LEVEL ?= "-9"
>  
> +ZSTD_COMPRESSION_LEVEL ?= "-3"
> +
>...

Why -3 and not -19 ?

Compression speed is not a real issue, and AFAIR decompression
tends to be slightly faster with smaller compressed size.
Plus the benefit that smaller images are faster to load
from network/storage.

cu
Adrian
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 15/17] devtool/standard.py: npm: configure the NPM_CACHE_DIR to be persistent

[Jean-Marie LEMETAYER]

When using devtool the ${WORKDIR} is not persistent as the creation
workflow is using temporary directories and external workspace. For npm
recipes the NPM_CACHE_DIR needs to be persistent to make the link
between the first do_fetch, do_unpack tasks and the do_compile task.

As the only persistent directory is the source directory, it is used to
store the npm cache. And it is also excluded from the git repository to
allow the 'devtool finish' command to succeed.

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/devtool/standard.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 31f0c44b20..1e68d37db9 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -262,6 +262,7 @@ def add(args, config, basepath, workspace):
 f.write('}\n')
 
 if bb.data.inherits_class('npm', rd):
+f.write('NPM_CACHE_DIR = "${S}/.npm_cache"\n')
 f.write('exclude_git() {\n')
 f.write('local exclude="${S}/.git/info/exclude"\n')
 f.write('if [ -f "${exclude}" ] && ! grep -q "${1}" 
"${exclude}" ; then\n')
@@ -269,6 +270,7 @@ def add(args, config, basepath, workspace):
 f.write('fi\n')
 f.write('}\n')
 f.write('do_compile_append() {\n')
+f.write('exclude_git "/.npm_cache"\n')
 f.write('exclude_git "/node_modules"\n')
 f.write('rm -rf ${B}/lib/node_modules/*/.git\n')
 f.write('rm -rf ${B}/lib/node_modules/@*/*/.git\n')
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 17/17] oeqa/selftest/devtool: add npm recipe build test

[Jean-Marie LEMETAYER]

This commit adds a devtool build test for npm recipe:

 - devtool.DevtoolAddTests.test_devtool_add_npm

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/lib/oeqa/selftest/cases/devtool.py | 20 
 1 file changed, 20 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py 
b/meta/lib/oeqa/selftest/cases/devtool.py
index 3a25da2033..d89a33e3c8 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -510,6 +510,26 @@ class DevtoolAddTests(DevtoolBase):
 checkvars['SRC_URI'] = url.replace(testver, '${PV}')
 self._test_recipe_contents(recipefile, checkvars, [])
 
+def test_devtool_add_npm(self):
+pn = 'savoirfairelinux-node-server-example'
+pv = '1.0.0'
+url = 
'npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0'
+# Test devtool add
+self.track_for_cleanup(self.workspacedir)
+self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn)
+self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+result = runCmd('devtool add \'%s\'' % url)
+self.assertExists(os.path.join(self.workspacedir, 'conf', 
'layer.conf'), 'Workspace directory not created')
+self.assertExists(os.path.join(self.workspacedir, 'recipes', pn, 
'%s_%s.bb' % (pn, pv)), 'Recipe not created')
+self.assertExists(os.path.join(self.workspacedir, 'recipes', pn, pn, 
'npm-shrinkwrap.json'), 'Shrinkwrap not created')
+# Test devtool status
+result = runCmd('devtool status')
+self.assertIn(pn, result.output)
+# Clean up anything in the workdir/sysroot/sstate cache (have to do 
this *after* devtool add since the recipe only exists then)
+bitbake('%s -c cleansstate' % pn)
+# Test devtool build
+result = runCmd('devtool build %s' % pn)
+
 class DevtoolModifyTests(DevtoolBase):
 
 def test_devtool_modify(self):
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 13/17] recipetool/create_npm.py: convert the shrinkwrap file

[Jean-Marie LEMETAYER]

A local npm cache pre filled with the dependencies is used to be able to
install the npm package without doing any unauthorized network access.

When it is offline, the npm cache identify a package using its integrity
metadata [1] but only the sha512 algorithm is used (this is the
default cacache algorithm [2]). And to create the shrinkwrap file, npm
use the integrity set in the registry which is not necessarily using
the sha512 algorithm.

The generated shrinkwrap file needs to be converted to use only sha512
integrity to ensure that the packages will be fetched from the cache and
not from the network.

1: https://www.w3.org/TR/SRI/
2: https://www.npmjs.com/package/cacache#optsalgorithms

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/recipetool/create_npm.py | 47 ++--
 1 file changed, 45 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index e6eb002251..dec9b741c5 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -7,15 +7,17 @@
 Recipe creation tool - npm module support plugin
 """
 
+import base64
+import copy
 import json
 import os
 import re
-import shutil
 import sys
 import tempfile
 import bb
 from bb.fetch2 import runfetchcmd
 from bb.fetch2.npm import fetch_dependencies
+from bb.fetch2.npm import fetch_dependency
 from bb.fetch2.npm import foreach_dependencies
 from bb.fetch2.npm import unpack_dependencies
 from recipetool.create import RecipeHandler
@@ -132,6 +134,47 @@ class NpmRecipeHandler(RecipeHandler):
 
 runfetchcmd(cmd, d, workdir=d.getVar("S"))
 
+@staticmethod
+def _convert_shrinkwrap(d, src_shrinkwrap, dst_shrinkwrap):
+"""
+When adding local tarball to the npm cache, only the sha512
+algorithm is used to create the cache metadata. The shrinkwrap file
+must be converted to use only sha512 integrity to be able to
+retrieve dependencies from the npm cache.
+"""
+
+def sha512_integrity(name, version):
+tarball = fetch_dependency(d, name, version)
+sha512 = bb.utils.sha512_file(tarball)
+return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
+
+def convert_deps(src):
+if src is None:
+return None
+dst = copy.deepcopy(src)
+for name in src:
+version = src[name].get("version")
+integrity = src[name].get("integrity")
+if integrity is not None and not 
integrity.startswith("sha512"):
+dst[name]["integrity"] = sha512_integrity(name, version)
+deps = src[name].get("dependencies")
+if deps is not None:
+dst[name]["dependencies"] = convert_deps(deps)
+return dst
+
+def convert(src):
+dst = copy.deepcopy(src)
+deps = src.get("dependencies")
+if deps is not None:
+dst["dependencies"] = convert_deps(deps)
+return dst
+
+with open(src_shrinkwrap, "r") as f:
+src = json.load(f)
+
+with open(dst_shrinkwrap, "w") as f:
+print(json.dumps(convert(src), indent=2), file=f)
+
 def _generate_shrinkwrap(self, d, lines, extravalues, development):
 """
 Check and generate the npm-shrinkwrap.json file if needed.
@@ -151,7 +194,7 @@ class NpmRecipeHandler(RecipeHandler):
 # Convert the shrinkwrap file and save it in a temporary location
 tmpdir = tempfile.mkdtemp(prefix="recipetool-npm")
 tmp_shrinkwrap = os.path.join(tmpdir, "npm-shrinkwrap.json")
-shutil.move(src_shrinkwrap, tmp_shrinkwrap)
+self._convert_shrinkwrap(d, src_shrinkwrap, tmp_shrinkwrap)
 
 # Add the shrinkwrap file as 'extrafiles'
 extravalues.setdefault("extrafiles", {})
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 14/17] npm.bbclass: force to rebuild the prebuild addons

[Jean-Marie LEMETAYER]

This commit forces to rebuild the prebuild addons which are using
node-gyp-build.

  https://www.npmjs.com/package/node-gyp-build

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/classes/npm.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 63b1d9f624..13f6b2b158 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -123,7 +123,10 @@ npm_do_compile() {
 local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} --release"
 local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} 
--nodedir=${RECIPE_SYSROOT_NATIVE}${prefix_native}"
 
+local NPM_INSTALL_GYP_BUILD_ARGS="${NPM_INSTALL_GYP_BUILD_ARGS} 
--build-from-source"
+
 cd ${WORKDIR} && npm install ${NPM_PACK_FILE} \
+${NPM_INSTALL_GYP_BUILD_ARGS} \
 ${NPM_INSTALL_GYP_ARGS} \
 ${NPM_INSTALL_ARGS}
 }
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 16/17] oeqa/selftest/recipetool: add npm recipe creation test

[Jean-Marie LEMETAYER]

This commit adds a recipetool creation test for npm recipe:

 - recipetool.RecipetoolTests.test_recipetool_create_npm

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 21 +
 1 file changed, 21 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py 
b/meta/lib/oeqa/selftest/cases/recipetool.py
index c1562c63b2..fd993ab260 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -421,6 +421,27 @@ class RecipetoolTests(RecipetoolBase):
 inherits = ['cmake']
 self._test_recipe_contents(recipefile, checkvars, inherits)
 
+def test_recipetool_create_npm(self):
+temprecipe = os.path.join(self.tempdir, 'recipe')
+os.makedirs(temprecipe)
+recipefile = os.path.join(temprecipe, 
'savoirfairelinux-node-server-example_1.0.0.bb')
+shrinkwrap = os.path.join(temprecipe, 
'savoirfairelinux-node-server-example', 'npm-shrinkwrap.json')
+srcuri = 
'npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0'
+result = runCmd('recipetool create -o %s \'%s\'' % (temprecipe, 
srcuri))
+self.assertTrue(os.path.isfile(recipefile))
+self.assertTrue(os.path.isfile(shrinkwrap))
+checkvars = {}
+checkvars['SUMMARY'] = 'Node Server Example'
+checkvars['HOMEPAGE'] = 
'https://github.com/savoirfairelinux/node-server-example#readme'
+checkvars['LICENSE'] = set(['MIT', 'ISC', 'Unknown'])
+checkvars['SRC_URI'] = 
'npm://registry.npmjs.org/;name=@savoirfairelinux/node-server-example;version=${PV}'
+checkvars['S'] = '${WORKDIR}/npm'
+checkvars['NPM_SHRINKWRAP'] = '${THISDIR}/${BPN}/npm-shrinkwrap.json'
+checkvars['LICENSE_${PN}-execute'] = 'Unknown'
+checkvars['LICENSE_${PN}-base64'] = 'Unknown'
+inherits = ['npm']
+self._test_recipe_contents(recipefile, checkvars, inherits)
+
 def test_recipetool_create_github(self):
 # Basic test to see if github URL mangling works
 temprecipe = os.path.join(self.tempdir, 'recipe')
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 09/17] devtool/standard.py: npm: exclude the node_modules directory

[Jean-Marie LEMETAYER]

If 'devtool add' was executed without the '--no-git' option a git
repository have been initialized in the source directory in order to
check if the sources stay unchanged after the 'devtool build' command.
The 'devtool finish' command will fail if any modification is found.

As the node_modules directory is added to the source tree in order to
manage the dependency license, it needs to be explicitly excluded from
this git repository.

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/devtool/standard.py | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 2604b79be3..31f0c44b20 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -262,7 +262,14 @@ def add(args, config, basepath, workspace):
 f.write('}\n')
 
 if bb.data.inherits_class('npm', rd):
+f.write('exclude_git() {\n')
+f.write('local exclude="${S}/.git/info/exclude"\n')
+f.write('if [ -f "${exclude}" ] && ! grep -q "${1}" 
"${exclude}" ; then\n')
+f.write('echo "${1}" >> "${exclude}"\n')
+f.write('fi\n')
+f.write('}\n')
 f.write('do_compile_append() {\n')
+f.write('exclude_git "/node_modules"\n')
 f.write('rm -rf ${B}/lib/node_modules/*/.git\n')
 f.write('rm -rf ${B}/lib/node_modules/@*/*/.git\n')
 f.write('rm -f ${B}/lib/node_modules/*/singletask.lock\n')
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 10/17] recipetool/create_npm.py: handle the licenses of the dependencies

[Jean-Marie LEMETAYER]

As usual the 'LICENSE' and the 'LIC_FILES_CHKSUM' values reflects all
the license files discovered in the source tree (including the
dependencies).

For npm recipes the 'LIC_FILES_CHKSUM' value contains also the status of
the 'package.json' file of every packages as it contains license
informations.

Finally each package has a separate 'LICENSE_${PN}-package-name' value
which describes its license.

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/recipetool/create_npm.py | 49 
 1 file changed, 49 insertions(+)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index ea3c3044fc..e6eb002251 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -15,7 +15,12 @@ import sys
 import tempfile
 import bb
 from bb.fetch2 import runfetchcmd
+from bb.fetch2.npm import fetch_dependencies
+from bb.fetch2.npm import foreach_dependencies
+from bb.fetch2.npm import unpack_dependencies
 from recipetool.create import RecipeHandler
+from recipetool.create import guess_license
+from recipetool.create import split_pkg_licenses
 
 tinfoil = None
 
@@ -176,6 +181,43 @@ class NpmRecipeHandler(RecipeHandler):
 name = name.strip("-")
 return name
 
+def _handle_licenses(self, d, shrinkwrap_file, lines, extravalues):
+"""
+This function have 2 goals:
+- Add all the package.json files as license files.
+- Split the license into sub packages.
+"""
+
+lic_files_chksum = []
+packages = {}
+
+def lic_files_chksum_append(licfile):
+licfilepath = os.path.join(d.getVar("S"), licfile)
+licmd5 = bb.utils.md5_file(licfilepath)
+lic_files_chksum.append("file://{};md5={}".format(licfile, licmd5))
+
+# Handle the parent package
+lic_files_chksum_append("package.json")
+packages["${PN}"] = ""
+
+# Handle the dependencies
+def handle_dependency(name, version, deptree):
+prefix = "-".join([self._name_from_npm(x, number=True) for x in 
deptree])
+relpath = os.path.join(*[os.path.join("node_modules", x) for x in 
deptree])
+lic_files_chksum_append(os.path.join(relpath, "package.json"))
+packages["${PN}-" + prefix] = relpath
+
+foreach_dependencies(d, handle_dependency, shrinkwrap_file)
+
+# Add the extra package.json license files. They will be handled by
+# the handle_license_vars() function later.
+extravalues["LIC_FILES_CHKSUM"] = lic_files_chksum
+
+# Split the license into sub packages. The global LICENSE will be
+# processed by the handle_license_vars() function later.
+licvalues = guess_license(d.getVar("S"), d)
+split_pkg_licenses(licvalues, packages, lines, [])
+
 def process(self, srctree, classes, lines_before, lines_after, handled, 
extravalues):
 """
 Handle the npm recipe creation
@@ -209,6 +251,13 @@ class NpmRecipeHandler(RecipeHandler):
 shrinkwrap = self._generate_shrinkwrap(d, lines_before,
extravalues, development)
 
+# Fetch and unpack the dependencies
+fetch_dependencies(d, shrinkwrap)
+unpack_dependencies(d, shrinkwrap)
+
+# Handle the licenses
+self._handle_licenses(d, shrinkwrap, lines_after, extravalues)
+
 extravalues["PN"] = self._name_from_npm(data["name"])
 extravalues["PV"] = data["version"]
 
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 12/17] npm.bbclass: use the local node headers

[Jean-Marie LEMETAYER]

When building addons, the node headers are needed to be able to compile
properly. Usually they are downloaded by npm but network access in the
do_compile task are unauthorized. Hopefully the local node headers are
available in the native sysroot so lets use them.

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/classes/npm.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index b13dca7eb8..63b1d9f624 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -121,6 +121,7 @@ npm_do_compile() {
 local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} --arch=${NPM_ARCH}"
 local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} 
--target_arch=${NPM_ARCH}"
 local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} --release"
+local NPM_INSTALL_GYP_ARGS="${NPM_INSTALL_GYP_ARGS} 
--nodedir=${RECIPE_SYSROOT_NATIVE}${prefix_native}"
 
 cd ${WORKDIR} && npm install ${NPM_PACK_FILE} \
 ${NPM_INSTALL_GYP_ARGS} \
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 11/17] npm.bbclass: restrict the build to be offline

[Jean-Marie LEMETAYER]

After the do_fetch task, every other tasks must not access the network.
In order to ensure this point every npm command must use the '--offline'
argument. In addition, setting an invalid proxy is used as a safety.

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/classes/npm.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index e2a258afe1..b13dca7eb8 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -97,7 +97,7 @@ npm_do_compile() {
 local NPM_SHRINKWRAP_INSTALLED=$(npm_install_shrinkwrap)
 
 # Then create a tarball from a npm package whose sources must be in ${S}.
-local NPM_PACK_FILE=$(cd ${WORKDIR} && npm pack ${S}/)
+local NPM_PACK_FILE=$(cd ${WORKDIR} && npm pack ${S}/ --offline 
--proxy=http://invalid.org)
 
 # Clean source tree.
 rm -f ${NPM_SHRINKWRAP_INSTALLED}
@@ -106,6 +106,8 @@ npm_do_compile() {
 rm -rf ${B}
 
 # Finally install and build the tarball package in ${B}.
+local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --offline"
+local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --proxy=http://invalid.org;
 local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --cache=${NPM_CACHE_DIR}"
 local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --loglevel=silly"
 local NPM_INSTALL_ARGS="${NPM_INSTALL_ARGS} --prefix=${B}"
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 07/17] recipetool/create.py: npm: replace the 'latest' keyword

[Jean-Marie LEMETAYER]

The new npm fetcher allows the 'latest' keyword to be used to download
the latest version on the registry. But the keyword must be replace
as soon as the version is determined to have a stable generated recipe.

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/recipetool/create.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 7754668559..b4ab1117b8 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -831,6 +831,8 @@ def create_recipe(args):
 elif line.startswith('SRC_URI = '):
 if realpv and not pv_srcpv:
 line = line.replace(realpv, '${PV}')
+if scheme == 'npm':
+line = line.replace('version=latest', 'version=${PV}')
 elif line.startswith('PV = '):
 if realpv:
 # Replace the first part of the PV value
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 08/17] npm.bbclass: split the do_compile task

[Jean-Marie LEMETAYER]

The do_compile task must be split in order to fit the yocto workflow.
The dependencies are fetched in the do_fetch task and cached in the
do_unpack task.

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/classes/npm.bbclass | 19 +++
 1 file changed, 19 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index b3f73d9b67..e2a258afe1 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -48,6 +48,25 @@ NPM_CACHE_DIR ?= "${WORKDIR}/npm_cache"
 
 B = "${WORKDIR}/build"
 
+python do_fetch_append() {
+"""
+Fetch all dependencies tarball to DL_DIR.
+"""
+bb.fetch2.npm.fetch_dependencies(d)
+}
+
+python do_unpack_append() {
+"""
+Unpack all dependencies tarball to the 'node_modules' directory and add
+them to the npm cache. The dependencies needs to be unpacked to have
+access to the licenses files checksum feature (LIC_FILES_CHKSUM). They
+also have to be in the cache to properly execute the 'npm install'.
+"""
+bb.fetch2.npm.unpack_dependencies(d)
+}
+
+do_unpack[depends] = "nodejs-native:do_populate_sysroot"
+
 npm_install_shrinkwrap() {
 # This function ensures that there is a shrinkwrap file in the specified
 # directory. A shrinkwrap file is mandatory to have reproducible builds.
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 06/17] recipetool/create.py: npm: remove the 'noverify' url parameter

[Jean-Marie LEMETAYER]

This commit removes the 'noverify' parameter which was added to the url
to fix warnings with the shrinkwrap / lockdown file generation. This is
not needed anymore with the new npm fetcher.

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/recipetool/create.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 932dc3f374..7754668559 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -477,8 +477,6 @@ def create_recipe(args):
 storeTagName = params['tag']
 params['nobranch'] = '1'
 del params['tag']
-if scheme == 'npm':
-params['noverify'] = '1'
 fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, 
params))
 
 tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 02/17] npm.bbclass: refactor the npm class

[Jean-Marie LEMETAYER]

Many issues were related to npm dependencies badly handled: package
names, installation directories, ... In fact npm is using an install
algorithm [1] which is hard to reproduce / anticipate. Moreover some
npm packages use scopes [2] which adds more complexity.

The simplest solution is to let npm do its job. Assuming the fetcher
only get the sources of the package, the class will now run
'npm install' to create a build directory. The build directory is then
copied wisely to the destination.

1: https://docs.npmjs.com/cli/install#algorithm
2: https://docs.npmjs.com/about-scopes

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/classes/npm.bbclass | 199 ++-
 1 file changed, 132 insertions(+), 67 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 4b1f0a39f0..b3f73d9b67 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -1,19 +1,40 @@
+# Copyright (C) 2019 Savoir-Faire Linux
+#
+# This bbclass builds and installs an npm package to the target. The package
+# sources files should be fetched in the calling recipe by using the SRC_URI
+# variable. The ${S} variable should be updated depending of your fetcher.
+#
+# Usage:
+#  SRC_URI = "..."
+#  inherit npm
+#
+# Optional variables:
+#  NPM_SHRINKWRAP:
+#   Provide a shrinkwrap file [1]. If available a shrinkwrap file in the
+#   sources has priority over the one provided. A shrinkwrap file is
+#   mandatory in order to ensure build reproducibility.
+#   1: https://docs.npmjs.com/files/shrinkwrap.json
+#
+#  NPM_INSTALL_DEV:
+#   Set to 1 to also install devDependencies.
+#
+#  NPM_REGISTRY:
+#   Use the specified registry.
+#
+#  NPM_ARCH:
+#   Override the auto generated npm architecture.
+
 DEPENDS_prepend = "nodejs-native "
 RDEPENDS_${PN}_prepend = "nodejs "
-S = "${WORKDIR}/npmpkg"
 
-def node_pkgname(d):
-bpn = d.getVar('BPN')
-if bpn.startswith("node-"):
-return bpn[5:]
-return bpn
+NPM_SHRINKWRAP ?= "${THISDIR}/${BPN}/npm-shrinkwrap.json"
 
-NPMPN ?= "${@node_pkgname(d)}"
+NPM_INSTALL_DEV ?= "0"
 
-NPM_INSTALLDIR = "${libdir}/node_modules/${NPMPN}"
+NPM_REGISTRY ?= "http://registry.npmjs.org;
 
 # function maps arch names to npm arch names
-def npm_oe_arch_map(target_arch, d):
+def npm_oe_arch_map(target_arch):
 import re
 if   re.match('p(pc|owerpc)(|64)', target_arch): return 'ppc'
 elif re.match('i.86$', target_arch): return 'ia32'
@@ -21,74 +42,118 @@ def npm_oe_arch_map(target_arch, d):
 elif re.match('arm64$', target_arch): return 'arm'
 return target_arch
 
-NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH'), d)}"
-NPM_INSTALL_DEV ?= "0"
+NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH'))}"
+
+NPM_CACHE_DIR ?= "${WORKDIR}/npm_cache"
+
+B = "${WORKDIR}/build"
+
+npm_install_shrinkwrap() {
+# This function ensures that there is a shrinkwrap file in the specified
+# directory. A shrinkwrap file is mandatory to have reproducible builds.
+# If the shrinkwrap file is not already included in the sources,
+# the recipe can provide one by using the NPM_SHRINKWRAP option.
+# This function returns the filename of the installed file (if any).
+if [ -f ${S}/npm-shrinkwrap.json ]
+then
+bbnote "Using the npm-shrinkwrap.json provided in the sources"
+elif [ -f ${NPM_SHRINKWRAP} ]
+then
+install -m 644 ${NPM_SHRINKWRAP} ${S}/npm-shrinkwrap.json
+echo ${S}/npm-shrinkwrap.json
+else
+bbfatal "No mandatory NPM_SHRINKWRAP file found"
+fi
+}
 
 npm_do_compile() {
-   # Copy in any additionally fetched modules
-   if [ -d ${WORKDIR}/node_modules ] ; then
-   cp -a ${WORKDIR}/node_modules ${S}/
-   fi
-   # changing the home directory to the working directory, the .npmrc will
-   # be created in this directory
-   export HOME=${WORKDIR}
-   if [  "${NPM_INSTALL_DEV}" = "1" ]; then
-   npm config set dev true
-   else
-   npm config set dev false
-   fi
-   npm set cache ${WORKDIR}/npm_cache
-   # clear cache before every build
-   npm cache clear --force
-   # Install pkg into ${S} without going to the registry
-   if [  "${NPM_INSTALL_DEV}" = "1" ]; then
-   npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --no-registry 
install
-   else
-   npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production 
--no-registry install
-   fi
+# This function executes the 'npm install' command which builds and
+# installs every dependencies needed for the package. All the files are
+# installed in a build directory ${B} without filtering anything. To do so,
+# a combination of 'npm pack' and 'npm install' is used to ensure that the
+# files in ${B} are actual copies instead of symbolic links (which is the
+# default npm behavior).
+
+# First ensure that there is a shrinkwrap file in the 

[OE-core] [PATCH v3 05/17] devtool/standard.py: npm: update the append file

[Jean-Marie LEMETAYER]

When creating a recipe using devtool, a workspace is created to store
the new recipe, the recipe source and some append files. These append
files are used by devtool to build the recipe using externalsrc (to use
the source which are in the workspace). They can also have some
additional actions according to the class of the recipe.

This commit updates the append file for the npm recipes. The
devtool / externalsrc files are removed in the npm build directory
instead of the install directory.

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/devtool/standard.py | 13 +
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 7068a02a01..2604b79be3 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -262,14 +262,11 @@ def add(args, config, basepath, workspace):
 f.write('}\n')
 
 if bb.data.inherits_class('npm', rd):
-f.write('do_install_append() {\n')
-f.write('# Remove files added to source dir by 
devtool/externalsrc\n')
-f.write('rm -f ${NPM_INSTALLDIR}/singletask.lock\n')
-f.write('rm -rf ${NPM_INSTALLDIR}/.git\n')
-f.write('rm -rf ${NPM_INSTALLDIR}/oe-local-files\n')
-f.write('for symlink in ${EXTERNALSRC_SYMLINKS} ; do\n')
-f.write('rm -f ${NPM_INSTALLDIR}/${symlink%%:*}\n')
-f.write('done\n')
+f.write('do_compile_append() {\n')
+f.write('rm -rf ${B}/lib/node_modules/*/.git\n')
+f.write('rm -rf ${B}/lib/node_modules/@*/*/.git\n')
+f.write('rm -f ${B}/lib/node_modules/*/singletask.lock\n')
+f.write('rm -f 
${B}/lib/node_modules/@*/*/singletask.lock\n')
 f.write('}\n')
 
 # Check if the new layer provides recipes whose priorities have been
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 01/17] devtool: npm: update command line options

[Jean-Marie LEMETAYER]

This commit renames the '--fetch-dev' option into '--npm-dev' which is
more easily understandable.

It also adds the '--npm-registry' option to allow creating a npm recipe
with a non default npm registry (e.g. if the SRC_URI is using git://).

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/devtool/standard.py  |  9 ++---
 scripts/lib/recipetool/create.py | 12 +++-
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 8d9c1a3022..7068a02a01 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -145,8 +145,10 @@ def add(args, config, basepath, workspace):
 extracmdopts += ' --src-subdir "%s"' % args.src_subdir
 if args.autorev:
 extracmdopts += ' -a'
-if args.fetch_dev:
-extracmdopts += ' --fetch-dev'
+if args.npm_dev:
+extracmdopts += ' --npm-dev'
+if args.npm_registry:
+extracmdopts += ' --npm-registry "%s"' % args.npm_registry
 if args.mirrors:
 extracmdopts += ' --mirrors'
 if args.srcrev:
@@ -2197,7 +2199,8 @@ def register_commands(subparsers, context):
 group.add_argument('--same-dir', '-s', help='Build in same directory as 
source', action="store_true")
 group.add_argument('--no-same-dir', help='Force build in a separate build 
directory', action="store_true")
 parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and 
extract it to create the source tree (deprecated - pass as positional argument 
instead)', metavar='URI')
-parser_add.add_argument('--fetch-dev', help='For npm, also fetch 
devDependencies', action="store_true")
+parser_add.add_argument('--npm-dev', help='For npm, also fetch 
devDependencies', action="store_true")
+parser_add.add_argument('--npm-registry', help='For npm, use the specified 
registry', type=str)
 parser_add.add_argument('--version', '-V', help='Version to use within 
recipe (PV)')
 parser_add.add_argument('--no-git', '-g', help='If fetching source, do not 
set up source tree as a git repository', action="store_true")
 group = parser_add.add_mutually_exclusive_group()
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 1fb6b55530..932dc3f374 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -716,10 +716,11 @@ def create_recipe(args):
 lines_after.append('INSANE_SKIP_${PN} += "already-stripped"')
 lines_after.append('')
 
-if args.fetch_dev:
-extravalues['fetchdev'] = True
-else:
-extravalues['fetchdev'] = None
+if args.npm_dev:
+extravalues['NPM_INSTALL_DEV'] = 1
+
+if args.npm_registry:
+extravalues['NPM_REGISTRY'] = args.npm_registry
 
 # Find all plugins that want to register handlers
 logger.debug('Loading recipe handlers')
@@ -1315,7 +1316,8 @@ def register_commands(subparsers):
 group.add_argument('-S', '--srcrev', help='Source revision to fetch if 
fetching from an SCM such as git (default latest)')
 parser_create.add_argument('-B', '--srcbranch', help='Branch in source 
repository if fetching from an SCM such as git (default master)')
 parser_create.add_argument('--keep-temp', action="store_true", help='Keep 
temporary directory (for debugging)')
-parser_create.add_argument('--fetch-dev', action="store_true", help='For 
npm, also fetch devDependencies')
+parser_create.add_argument('--npm-dev', action="store_true", help='For 
npm, also fetch devDependencies')
+parser_create.add_argument('--npm-registry', help='For npm, use the 
specified registry', type=str)
 parser_create.add_argument('--devtool', action="store_true", 
help=argparse.SUPPRESS)
 parser_create.add_argument('--mirrors', action="store_true", help='Enable 
PREMIRRORS and MIRRORS for source tree fetching (disabled by default).')
 parser_create.set_defaults(func=create_recipe)
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 04/17] lib/oe/package.py: remove unneeded npm_split_package_dirs function

[Jean-Marie LEMETAYER]

The npm_split_package_dirs function was used by the recipetool when
creating npm recipes. This is not the case anymore.

Signed-off-by: Jean-Marie LEMETAYER 
---
 meta/lib/oe/package.py | 33 -
 1 file changed, 33 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index b8585d4253..dd700cbb0c 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -283,36 +283,3 @@ def read_shlib_providers(d):
 shlib_provider[s[0]] = {}
 shlib_provider[s[0]][s[1]] = (dep_pkg, s[2])
 return shlib_provider
-
-
-def npm_split_package_dirs(pkgdir):
-"""
-Work out the packages fetched and unpacked by BitBake's npm fetcher
-Returns a dict of packagename -> (relpath, package.json) ordered
-such that it is suitable for use in PACKAGES and FILES
-"""
-from collections import OrderedDict
-import json
-packages = {}
-for root, dirs, files in os.walk(pkgdir):
-if os.path.basename(root) == 'node_modules':
-for dn in dirs:
-relpth = os.path.relpath(os.path.join(root, dn), pkgdir)
-pkgitems = ['${PN}']
-for pathitem in relpth.split('/'):
-if pathitem == 'node_modules':
-continue
-pkgitems.append(pathitem)
-pkgname = '-'.join(pkgitems).replace('_', '-')
-pkgname = pkgname.replace('@', '')
-pkgfile = os.path.join(root, dn, 'package.json')
-data = None
-if os.path.exists(pkgfile):
-with open(pkgfile, 'r') as f:
-data = json.loads(f.read())
-packages[pkgname] = (relpth, data)
-# We want the main package for a module sorted *after* its subpackages
-# (so that it doesn't otherwise steal the files for the subpackage), so
-# this is a cheap way to do that whilst still having an otherwise
-# alphabetical sort
-return OrderedDict((key, packages[key]) for key in sorted(packages, 
key=lambda pkg: pkg + '~'))
-- 
2.20.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3 03/17] recipetool/create_npm.py: refactor the npm recipe creation handler

[Jean-Marie LEMETAYER]

This commit refactors the npm recipe creation handler to use the new npm
behavior. The process is kept as simple as possible and only generates
the shrinkwrap file.

To avoid naming issues the recipe name is now extracted from the npm
package name and not directly mapped.

Signed-off-by: Jean-Marie LEMETAYER 
---
 scripts/lib/recipetool/create_npm.py | 475 +++
 1 file changed, 192 insertions(+), 283 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 39429ebad3..ea3c3044fc 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -1,321 +1,230 @@
-# Recipe creation tool - node.js NPM module support plugin
-#
 # Copyright (C) 2016 Intel Corporation
+# Copyright (C) 2019 Savoir-Faire Linux
 #
 # SPDX-License-Identifier: GPL-2.0-only
 #
+"""
+Recipe creation tool - npm module support plugin
+"""
 
+import json
 import os
+import re
+import shutil
 import sys
-import logging
-import subprocess
 import tempfile
-import shutil
-import json
-from recipetool.create import RecipeHandler, split_pkg_licenses, 
handle_license_vars
-
-logger = logging.getLogger('recipetool')
-
+import bb
+from bb.fetch2 import runfetchcmd
+from recipetool.create import RecipeHandler
 
 tinfoil = None
 
 def tinfoil_init(instance):
+"""
+Initialize tinfoil.
+"""
 global tinfoil
 tinfoil = instance
 
-
 class NpmRecipeHandler(RecipeHandler):
-lockdownpath = None
+"""
+Class to handle the npm recipe creation
+"""
+
+@staticmethod
+def _get_registry(extravalues, lines_before):
+"""
+Get the registry value from the '--npm-registry' option
+or the 'npm://registry' url.
+"""
+registry_option = extravalues.get("NPM_REGISTRY")
+
+registry_fetch = None
+
+def handle_metadata(name, value, *unused):
+if name == "SRC_URI":
+for uri in value.split():
+if uri.startswith("npm://"):
+nonlocal registry_fetch
+registry_fetch = re.sub(r"^npm://", "http://;, 
uri.split(";")[0])
+return value, None, 0, True
 
-def _ensure_npm(self, fixed_setup=False):
+bb.utils.edit_metadata(lines_before, ["SRC_URI"], handle_metadata)
+
+if registry_fetch is not None:
+registry = registry_fetch
+
+if registry_option is not None:
+bb.warn("The npm registry is specified multiple times")
+bb.note("Using registry from the fetch url: 
'{}'".format(registry))
+extravalues.pop("NPM_REGISTRY", None)
+
+elif registry_option is not None:
+registry = registry_option
+
+else:
+registry = "http://registry.npmjs.org;
+
+return registry
+
+@staticmethod
+def _ensure_npm(d):
+"""
+Check if the 'npm' command is available in the recipes, then build
+it and add it to the PATH.
+"""
 if not tinfoil.recipes_parsed:
 tinfoil.parse_recipes()
+
 try:
 rd = tinfoil.parse_recipe('nodejs-native')
 except bb.providers.NoProvider:
-if fixed_setup:
-msg = 'nodejs-native is required for npm but is not available 
within this SDK'
-else:
-msg = 'nodejs-native is required for npm but is not available 
- you will likely need to add a layer that provides nodejs'
-logger.error(msg)
-return None
+bb.error("Nothing provides 'nodejs-native' which is required for 
the build")
+bb.note("You will likely need to add a layer that provides nodejs")
+sys.exit(14)
+
 bindir = rd.getVar('STAGING_BINDIR_NATIVE')
 npmpath = os.path.join(bindir, 'npm')
+
 if not os.path.exists(npmpath):
 tinfoil.build_targets('nodejs-native', 'addto_recipe_sysroot')
+
 if not os.path.exists(npmpath):
-logger.error('npm required to process specified source, but 
nodejs-native did not seem to populate it')
-return None
-return bindir
-
-def _handle_license(self, data):
-'''
-Handle the license value from an npm package.json file
-'''
-license = None
-if 'license' in data:
-license = data['license']
-if isinstance(license, dict):
-license = license.get('type', None)
-if license:
-if 'OR' in license:
-license = license.replace('OR', '|')
-license = license.replace('AND', '&')
-license = license.replace(' ', '_')
-if not license[0] == '(':
-license = '(' + license + ')'
-else:
-license = license.replace('AND', '&')
-  

[OE-core] [PATCH v3 00/17] NPM refactoring

[Jean-Marie LEMETAYER]

The current NPM support have several issues:
 - The current NPM fetcher downloads the dependency tree but not the other
   fetchers. The 'subdir' parameter was used to fix this issue.
 - They are multiple issues with package names (uppercase, exotic characters,
   scoped packages) even if they are inside the dependencies.
 - The lockdown file generation have issues. When a package depends on
   multiple version of the same package (all versions have the same checksum).

This patchset refactors the NPM support in Yocto:
 - As the NPM algorithm for dependency management is hard to handle, the new
   NPM fetcher downloads only the package source (and not the dependencies,
   like the other fetchers) (patch submitted in the bitbake-devel list).
 - The NPM class handles the dependencies using NPM (and not manually).
 - The NPM recipe creation is simplified to avoid issues.
 - The lockdown file is no more used as it is no longer relevant compared to the
   latest shrinkwrap file format.

This patchset may remove some features (lockdown file, license management for
dependencies) but fixes the majority of the NPM issues. All of these issues
from the bugzilla.yoctoproject.org are resolved by this patchset:
#10237, #10760, #11028, #11728, #11902, #12534

The fetcher and recipetool are now aware of a 'latest' keyword for the version
which allow to build the latest version available on the registry. This feature
fixes the two issues: #10515, #11029

Moreover the issue #13415 should also be fixed but I cannot test it.

I have tested the recipe creation and builds using a self made example to
generate build failures:
 - https://github.com/savoirfairelinux/node-server-example
 - https://npmjs.com/package/@savoirfairelinux/node-server-example

The test steps are these ones:
  $ source poky/oe-init-build-env
  $ bitbake-layers add-layer ../meta-openembedded/meta-oe
  $ devtool add 
"npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest"
  $ devtool build savoirfairelinux-node-server-example
  $ bitbake-layers create-layer ../meta-test
  $ bitbake-layers add-layer ../meta-test
  $ devtool finish savoirfairelinux-node-server-example ../meta-test
  $ echo IMAGE_INSTALL_append = '" savoirfairelinux-node-server-example"' >> 
conf/local.conf
  $ bitbake core-image-minimal

Also the 'devtool add' url could be one of these:
 - 
npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=latest
   This url uses the new npm fetcher and request the latest version available on
   the registry.
 - 
npm://registry.npmjs.org;name=@savoirfairelinux/node-server-example;version=1.0.0
   This url uses the new npm fetcher and request a fixed version.
 - git://github.com/savoirfairelinux/node-server-example.git;protocol=https
   This url uses the git fetcher. As the dependencies are managed by the NPM
   class, any fetcher can be used.

When this patchset will be merged, I have planned to update the NPM wiki:
  https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

This patchset is also the base work for the full Angular support in Yocto that I
am preparing. These applications have a huge dependency tree which is ideal to
test the NPM support.

--- V2

 - Add the 'check_network_access' function before each network access to check
   for 'BB_NO_NETWORK' and 'BB_ALLOWED_NETWORKS' variables.

 - Add a 'recipetool.RecipetoolTests.test_recipetool_create_npm' test case for
   'oe-selftest' to test the npm recipe creation.

 - Update the 'npm.bbclass' to fetch the dependencies in the 'do_fetch' task.
   The dependencies are cached in a npm cache stored in '${DL_DIR}/npm_cache'
   allowing the dependencies to be saved in the download directory and verify
   on insertion and extraction [1]:
  "All data that passes through the cache is fully verified
   for integrity on both insertion and extraction."

1: https://docs.npmjs.com/cli/cache.html

--- V3

 - Add a test regarding the 'devtool add' and 'devtool build' command:
 - devtool.DevtoolAddTests.test_devtool_add_npm

 - Add the license management for dependencies.

 - Split the npm workflow to use the bitbake default tasks (do_fetch, do_unpack,
   do_compile, do_install). Make sure that only the do_fetch task requires
   network access.

 - Split the commits for better understanding.

 - Force the rebuild of prebuild addons.

 - Use ${nonarch_libdir} instead of ${libdir} in the do_install task.

  - These patches can be found here:
 - 
https://github.com/savoirfairelinux/openembedded-core/tree/npm-refactoring-v3
 - https://github.com/savoirfairelinux/poky/tree/npm-refactoring-v3

Jean-Marie LEMETAYER (17):
  devtool: npm: update command line options
  npm.bbclass: refactor the npm class
  recipetool/create_npm.py: refactor the npm recipe creation handler
  lib/oe/package.py: remove unneeded npm_split_package_dirs function
  devtool/standard.py: npm: update the append file
  recipetool/create.py: npm: remove the 'noverify' url parameter
  

[OE-core] [PATCH v4 1/2] zstd: move recipe to oe-core from meta-oe

[Stefan Agner]

From: Stefan Agner 

Zstd is a very fast compression algorithm and has gained quite some
popularity recently. An upcoming patch allows to compress rootfs
using Zstd. Moving Zstd to oe-core allows this new conversion formats
to be automatically tested by the oe-selftest
imagefeatures.ImageFeatures.test_image_fstypes test.

Furthermore there are three packages in oe-core which allow to enable
Zstd support through packageconfig currently.

Signed-off-by: Stefan Agner 
---
 meta/conf/distro/include/maintainers.inc |  1 +
 meta/recipes-extended/zstd/zstd_1.4.4.bb | 35 
 2 files changed, 36 insertions(+)
 create mode 100644 meta/recipes-extended/zstd/zstd_1.4.4.bb

diff --git a/meta/conf/distro/include/maintainers.inc 
b/meta/conf/distro/include/maintainers.inc
index d85e5b697f..dff7ce28f8 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -774,3 +774,4 @@ RECIPE_MAINTAINER_pn-xwininfo = "Armin Kuster 
"
 RECIPE_MAINTAINER_pn-xz = "Denys Dmytriyenko "
 RECIPE_MAINTAINER_pn-zip = "Denys Dmytriyenko "
 RECIPE_MAINTAINER_pn-zlib = "Denys Dmytriyenko "
+RECIPE_MAINTAINER_pn-zstd = "Alex Kiernan "
diff --git a/meta/recipes-extended/zstd/zstd_1.4.4.bb 
b/meta/recipes-extended/zstd/zstd_1.4.4.bb
new file mode 100644
index 00..eb201f4139
--- /dev/null
+++ b/meta/recipes-extended/zstd/zstd_1.4.4.bb
@@ -0,0 +1,35 @@
+SUMMARY = "Zstandard - Fast real-time compression algorithm"
+DESCRIPTION = "Zstandard is a fast lossless compression algorithm, targeting \
+real-time compression scenarios at zlib-level and better compression ratios. \
+It's backed by a very fast entropy stage, provided by Huff0 and FSE library."
+HOMEPAGE = "http://www.zstd.net/;
+SECTION = "console/utils"
+
+LICENSE = "BSD-3-Clause & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=c7f0b161edbe52f5f345a3d1311d0b32 \
+file://COPYING;md5=39bba7d2cf0ba1036f2a6e2be52fe3f0"
+
+SRC_URI = "git://github.com/facebook/zstd.git;nobranch=1"
+
+SRCREV = "10f0e6993f9d2f682da6d04aa2385b7d53cbb4ee"
+UPSTREAM_CHECK_GITTAGREGEX = "v(?P\d+(\.\d+)+)"
+
+S = "${WORKDIR}/git"
+
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[lz4] = "HAVE_LZ4=1,HAVE_LZ4=0,lz4"
+PACKAGECONFIG[lzma] = "HAVE_LZMA=1,HAVE_LZMA=0,xz"
+PACKAGECONFIG[zlib] = "HAVE_ZLIB=1,HAVE_ZLIB=0,zlib"
+
+# See programs/README.md for how to use this
+ZSTD_LEGACY_SUPPORT ??= "4"
+
+do_compile () {
+oe_runmake ${PACKAGECONFIG_CONFARGS} 
ZSTD_LEGACY_SUPPORT=${ZSTD_LEGACY_SUPPORT}
+}
+
+do_install () {
+oe_runmake install 'DESTDIR=${D}'
+}
+
+BBCLASSEXTEND = "native nativesdk"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v4 2/2] image_types: add Zstandard conversion support

[Stefan Agner]

From: Stefan Agner 

Add Zstandard (or just Zstd) compression support. This allows to
create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.

Signed-off-by: Stefan Agner 
---
 meta/classes/image_types.bbclass | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 2eeffbb366..d29b9c5787 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
 
+ZSTD_COMPRESSION_LEVEL ?= "-3"
+
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime 
--output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 
${EXTRA_IMAGECMD}"
 
@@ -269,7 +271,7 @@ IMAGE_TYPES = " \
 hddimg \
 squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
 ubi ubifs multiubi \
-tar tar.gz tar.bz2 tar.xz tar.lz4 \
+tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
 cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
 wic wic.gz wic.bz2 wic.lzma \
 container \
@@ -282,7 +284,7 @@ IMAGE_TYPES = " \
 # CONVERSION_CMD/DEPENDS.
 COMPRESSIONTYPES ?= ""
 
-CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum 
sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
${COMPRESSIONTYPES}"
+CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum sha224sum 
sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 
${COMPRESSIONTYPES}"
 CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -290,6 +292,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} 
${XZ_DEFAULTS} --check=
 CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
 CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD_zst = "zstd -f -k -c ${ZSTD_COMPRESSION_LEVEL} 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
 CONVERSION_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
 CONVERSION_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
 CONVERSION_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > 
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
@@ -310,6 +313,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
 CONVERSION_DEPENDS_lz4 = "lz4-native"
 CONVERSION_DEPENDS_lzo = "lzop-native"
 CONVERSION_DEPENDS_zip = "zip-native"
+CONVERSION_DEPENDS_zst = "zstd-native"
 CONVERSION_DEPENDS_sum = "mtd-utils-native"
 CONVERSION_DEPENDS_bmap = "bmap-tools-native"
 CONVERSION_DEPENDS_u-boot = "u-boot-tools-native"
-- 
2.17.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v3 1/2] zstd: move recipe to oe-core from meta-oe

[Stefan Agner]

On 2019-11-19 17:18, Alex Kiernan wrote:
> On Mon, Nov 18, 2019 at 6:39 PM Stefan Agner  wrote:
>>
>> [adding Alex, he maintained the recipe in meta-oe so far]
>>
>> On 2019-11-18 13:19, Richard Purdie wrote:
>> > On Thu, 2019-11-14 at 15:47 +0100, Stefan Agner wrote:
>> >> On 2019-11-08 10:14, Stefan Agner wrote:
>> >> > From: Stefan Agner 
>> >> >
>> >> > Zstd is a very fast compression algorithm and has gained quite some
>> >> > popularity recently. An upcoming patch allows to compress rootfs
>> >> > using Zstd. Moving Zstd to oe-core allows this new conversion
>> >> > formats
>> >> > to be automatically tested by the oe-selftest
>> >> > imagefeatures.ImageFeatures.test_image_fstypes test.
>> >> >
>> >> > Furthermore there are three packages in oe-core which allow to
>> >> > enable
>> >> > Zstd support through packageconfig currently.
>> >>
>> >> Any update/thoughts on this?
>> >
>> > I've not made a final decision on whether to merge it fails selftests
>> > due to a lack of a maintainer:
>>
>> Alex, would you be OK with you listed as maintainer?
>>
> 
> I'm fine with that - it's very low overhead.

Thanks Alex, will send a v4 with you as maintainer.

--
Stefan
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core