[OE-core] [PATCH] linux-firmware : Add firmware of RTL8822 serie

2023-07-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

RTL8822 is a serie of wireless modules that need firmwares to function 
correctly.
The linux firmware recipe does not have a package of these firmwares, and this 
commit add them.

Signed-off-by: BELOUARGA Mohamed 
---
 .../linux-firmware/linux-firmware_20230625.bb | 8 
 1 file changed, 8 insertions(+)

diff --git a/meta/recipes-kernel/linux-firmware/linux-firmware_20230625.bb 
b/meta/recipes-kernel/linux-firmware/linux-firmware_20230625.bb
index 329a3e3c9a..3d6645579a 100644
--- a/meta/recipes-kernel/linux-firmware/linux-firmware_20230625.bb
+++ b/meta/recipes-kernel/linux-firmware/linux-firmware_20230625.bb
@@ -241,6 +241,7 @@ PACKAGES =+ "${PN}-ralink-license ${PN}-ralink \
  ${PN}-rtl-license ${PN}-rtl8188 ${PN}-rtl8192cu ${PN}-rtl8192ce 
${PN}-rtl8192su ${PN}-rtl8723 ${PN}-rtl8821 \
  ${PN}-rtl8761 \
  ${PN}-rtl8168 \
+ ${PN}-rtl8822 \
  ${PN}-cypress-license \
  ${PN}-broadcom-license \
  ${PN}-bcm-0bb4-0306 \
@@ -581,6 +582,7 @@ LICENSE:${PN}-rtl8192su = "Firmware-rtlwifi_firmware"
 LICENSE:${PN}-rtl8723 = "Firmware-rtlwifi_firmware"
 LICENSE:${PN}-rtl8761 = "Firmware-rtlwifi_firmware"
 LICENSE:${PN}-rtl8821 = "Firmware-rtlwifi_firmware"
+LICENSE:${PN}-rtl8822 = "Firmware-rtlwifi_firmware"
 LICENSE:${PN}-rtl-license = "Firmware-rtlwifi_firmware"
 LICENSE:${PN}-rtl8168 = "WHENCE"
 
@@ -611,6 +613,11 @@ FILES:${PN}-rtl8761 = " \
 FILES:${PN}-rtl8168 = " \
   ${nonarch_base_libdir}/firmware/rtl_nic/rtl8168*.fw \
 "
+FILES:${PN}-rtl8822 = " \
+  ${nonarch_base_libdir}/firmware/rtl_bt/rtl8822*.bin \
+  ${nonarch_base_libdir}/firmware/rtw88/rtw8822*.bin \
+  ${nonarch_base_libdir}/firmware/rtlwifi/rtl8822*.bin \
+"
 
 RDEPENDS:${PN}-rtl8188 += "${PN}-rtl-license"
 RDEPENDS:${PN}-rtl8192ce += "${PN}-rtl-license"
@@ -619,6 +626,7 @@ RDEPENDS:${PN}-rtl8192su = "${PN}-rtl-license"
 RDEPENDS:${PN}-rtl8723 += "${PN}-rtl-license"
 RDEPENDS:${PN}-rtl8821 += "${PN}-rtl-license"
 RDEPENDS:${PN}-rtl8761 += "${PN}-rtl-license"
+RDEPENDS:${PN}-rtl8822 += "${PN}-rtl-license"
 RDEPENDS:${PN}-rtl8168 += "${PN}-whence-license"
 
 # For ti-connectivity
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184973): 
https://lists.openembedded.org/g/openembedded-core/message/184973
Mute This Topic: https://lists.openembedded.org/mt/100396950/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts

2023-05-31 Thread belouargamohamed 
Hello,

I fixed the problem on master branch.
*For master:*

In order that my changes work, Some patches on bitbake should be accepted:
https://lists.openembedded.org/g/bitbake-devel/message/14815?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99163598
https://lists.openembedded.org/g/bitbake-devel/message/14819?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99230772
https://lists.openembedded.org/g/bitbake-devel/message/14820?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99248868
And:
https://lists.openembedded.org/g/openembedded-core/message/182022?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99248914
And this series of patches:
https://lists.openembedded.org/g/openembedded-core/message/181961?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99230551

*For kirkstone:*
In order that my changes work, Some patches on bitbake should be accepted:
https://lists.openembedded.org/g/bitbake-devel/message/14815?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99163598
https://lists.openembedded.org/g/bitbake-devel/message/14819?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99230772
https://lists.openembedded.org/g/bitbake-devel/message/14820?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99248868

And this series of patches.

Thanks

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182037): 
https://lists.openembedded.org/g/openembedded-core/message/182037
Mute This Topic: https://lists.openembedded.org/mt/99163838/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [master][PATCH 1/6] recipetool: create: npm: Remove duplicate function to not have future conflicts

2023-05-31 Thread belouargamohamed 
Hey Alexander,
Thanks for your answer.

In order that my changes work, Some patches on bitbake should be accepted:
https://lists.openembedded.org/g/bitbake-devel/message/14815?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99163598
https://lists.openembedded.org/g/bitbake-devel/message/14819?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99230772
https://lists.openembedded.org/g/bitbake-devel/message/14820?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99248868

and one patch that I just made on open embedded-core:
https://lists.openembedded.org/g/openembedded-core/message/182022?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Arecentpostdate%2Fsticky%2C%2Cbelouarga%2C20%2C2%2C0%2C99248914

I ran the tests that you asked me to do, you can find the results below.

elf@container:~/workspace$ oe-selftest -r 
recipetool.RecipetoolCreateTests.test_recipetool_create_npm
2023-05-31 17:31:33,552 - oe-selftest - INFO - Changing cwd to 
/home/elf/workspace/build
2023-05-31 17:31:33,552 - oe-selftest - WARNING - meta-selftest layer not found 
in BBLAYERS, adding it
2023-05-31 17:31:39,880 - oe-selftest - INFO - Adding layer libraries:
2023-05-31 17:31:39,880 - oe-selftest - INFO -  
/home/elf/workspace/layers/poky/meta/lib
2023-05-31 17:31:39,880 - oe-selftest - INFO -  
/home/elf/workspace/layers/poky/meta-yocto-bsp/lib
2023-05-31 17:31:39,880 - oe-selftest - INFO -  
/home/elf/workspace/layers/meta-openembedded/meta-oe/lib
2023-05-31 17:31:39,880 - oe-selftest - INFO -  
/home/elf/workspace/layers/poky/meta-selftest/lib
2023-05-31 17:31:39,882 - oe-selftest - INFO - Running bitbake -e to test the 
configuration is valid/parsable
NOTE: Starting bitbake server...
2023-05-31 17:31:49,176 - oe-selftest - INFO - Adding: "include selftest.inc" 
in /home/elf/workspace/build-st/conf/local.conf
2023-05-31 17:31:49,178 - oe-selftest - INFO - Adding: "include bblayers.inc" 
in bblayers.conf
2023-05-31 17:31:49,179 - oe-selftest - INFO - test_recipetool_create_npm 
(recipetool.RecipetoolCreateTests)
2023-05-31 17:33:36,312 - oe-selftest - INFO -  ... ok
2023-05-31 17:33:38,097 - oe-selftest - INFO - 
--
2023-05-31 17:33:38,097 - oe-selftest - INFO - Ran 1 test in 115.852s
2023-05-31 17:33:38,097 - oe-selftest - INFO - OK
2023-05-31 17:33:48,971 - oe-selftest - INFO - RESULTS:
2023-05-31 17:33:48,971 - oe-selftest - INFO - RESULTS - 
recipetool.RecipetoolCreateTests.test_recipetool_create_npm: PASSED (107.13s)
2023-05-31 17:33:49,063 - oe-selftest - INFO - SUMMARY:
2023-05-31 17:33:49,063 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 
115.853s
2023-05-31 17:33:49,064 - oe-selftest - INFO - oe-selftest - OK - All required 
tests passed (successes=1, skipped=0, failures=0, errors=0)
elf@container:~/workspace$ oe-selftest -r 
devtool.DevtoolAddTests.test_devtool_add_npm
2023-05-31 17:34:03,550 - oe-selftest - INFO - Changing cwd to 
/home/elf/workspace/build
2023-05-31 17:34:03,550 - oe-selftest - INFO - Adding layer libraries:
2023-05-31 17:34:03,550 - oe-selftest - INFO -  
/home/elf/workspace/layers/poky/meta/lib
2023-05-31 17:34:03,551 - oe-selftest - INFO -  
/home/elf/workspace/layers/poky/meta-yocto-bsp/lib
2023-05-31 17:34:03,551 - oe-selftest - INFO -  
/home/elf/workspace/layers/meta-openembedded/meta-oe/lib
2023-05-31 17:34:03,551 - oe-selftest - INFO -  
/home/elf/workspace/layers/poky/meta-selftest/lib
2023-05-31 17:34:03,552 - oe-selftest - INFO - Running bitbake -e to test the 
configuration is valid/parsable
NOTE: Starting bitbake server...
2023-05-31 17:34:12,581 - oe-selftest - INFO - Adding: "include selftest.inc" 
in /home/elf/workspace/build-st/conf/local.conf
2023-05-31 17:34:12,582 - oe-selftest - INFO - Adding: "include bblayers.inc" 
in bblayers.conf
2023-05-31 17:34:15,320 - oe-selftest - INFO - test_devtool_add_npm 
(devtool.DevtoolAddTests)
2023-05-31 17:38:02,622 - oe-selftest - INFO -  ... ok
2023-05-31 17:38:04,371 - oe-selftest - INFO - 
--
2023-05-31 17:38:04,371 - oe-selftest - INFO - Ran 1 test in 238.516s
2023-05-31 17:38:04,371 - oe-selftest - INFO - OK
2023-05-31 17:38:21,118 - oe-selftest - INFO - RESULTS:
2023-05-31 17:38:21,118 - oe-selftest - INFO - RESULTS - 
devtool.DevtoolAddTests.test_devtool_add_npm: PASSED (227.30s)
2023-05-31 17:38:21,200 - oe-selftest - INFO - SUMMARY:
2023-05-31 17:38:21,200 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 
238.517s
2023-05-31 17:38:21,201 - oe-selftest - INFO - oe-selftest - OK - All required 
tests passed (successes=1, skipped=0, failures=0, errors=0)

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182033): 
https://lists.openembedded.org/g/openembedded-core/message/182033
Mute This Topic: https://lists.openembedded.org/mt/99230551/21656
Group Owner:

[OE-core] [master][PATCH] meta: lib: oe: npm_registry: Add more safe caracters

2023-05-31 Thread belouargamohamed 
From: BELOUARGA Mohamed 

NPM registry cache should support caracaters like '(' and ')'
Explanation: NPM packages can contains these caracters like : @(._.)/execute

Signed-off-by: BELOUARGA Mohamed 
---
 meta/lib/oe/npm_registry.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/npm_registry.py b/meta/lib/oe/npm_registry.py
index db581e280e..d97ced7cda 100644
--- a/meta/lib/oe/npm_registry.py
+++ b/meta/lib/oe/npm_registry.py
@@ -11,7 +11,7 @@ import subprocess
 _ALWAYS_SAFE = frozenset('ABCDEFGHIJKLMNOPQRSTUVWXYZ'
  'abcdefghijklmnopqrstuvwxyz'
  '0123456789'
- '_.-~')
+ '_.-~()')
 
 MISSING_OK = object()
 
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182022): 
https://lists.openembedded.org/g/openembedded-core/message/182022
Mute This Topic: https://lists.openembedded.org/mt/99248914/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [master][PATCH 5/6] classes: npm: Add support for the new format of the shrinkwrap file

2023-05-30 Thread belouargamohamed 
From: BELOUARGA Mohamed 

1 - Adapt do_configure to the new format of the shrinkwrap

2 - Remove useless function _npmsw_dependency_dict because the dictionnary
is already given by npmsw:foreach_dependencies

3 - Rename arguments of callback functions

Signed-off-by: BELOUARGA Mohamed 
---
 meta/classes-recipe/npm.bbclass | 27 ++-
 1 file changed, 6 insertions(+), 21 deletions(-)

diff --git a/meta/classes-recipe/npm.bbclass b/meta/classes-recipe/npm.bbclass
index 92e59fefce..c1944ca14e 100644
--- a/meta/classes-recipe/npm.bbclass
+++ b/meta/classes-recipe/npm.bbclass
@@ -130,22 +130,6 @@ python npm_do_configure() {
 sha512 = bb.utils.sha512_file(tarball)
 return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
 
-def _npmsw_dependency_dict(orig, deptree):
-"""
-Return the sub dictionary in the 'orig' dictionary corresponding to the
-'deptree' dependency tree. This function follows the shrinkwrap file
-format.
-"""
-ptr = orig
-for dep in deptree:
-if "dependencies" not in ptr:
-ptr["dependencies"] = {}
-ptr = ptr["dependencies"]
-if dep not in ptr:
-ptr[dep] = {}
-ptr = ptr[dep]
-return ptr
-
 # Manage the manifest file and shrinkwrap files
 orig_manifest_file = d.expand("${S}/package.json")
 orig_shrinkwrap_file = d.expand("${S}/npm-shrinkwrap.json")
@@ -177,24 +161,25 @@ python npm_do_configure() {
 progress_total = 1 # also count the main package
 progress_done = 0
 
-def _count_dependency(name, params, deptree):
+def _count_dependency(name, params, destsuffix):
 nonlocal progress_total
 progress_total += 1
 
-def _cache_dependency(name, params, deptree):
-destsubdirs = [os.path.join("node_modules", dep) for dep in deptree]
-destsuffix = os.path.join(*destsubdirs)
+def _cache_dependency(name, params, destsuffix):
 with tempfile.TemporaryDirectory() as tmpdir:
 # Add the dependency to the npm cache
 destdir = os.path.join(d.getVar("S"), destsuffix)
 (tarball, pkg) = npm_pack(env, destdir, tmpdir)
 _npm_cache_add(tarball, pkg)
 # Add its signature to the cached shrinkwrap
-dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
+dep = params
 dep["version"] = pkg['version']
 dep["integrity"] = _npm_integrity(tarball)
 if params.get("dev", False):
 dep["dev"] = True
+if "dependencies" not in cached_shrinkwrap:
+cached_shrinkwrap["dependencies"] = {}
+cached_shrinkwrap["dependencies"][name] = dep
 # Display progress
 nonlocal progress_done
 progress_done += 1
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181965): 
https://lists.openembedded.org/g/openembedded-core/message/181965
Mute This Topic: https://lists.openembedded.org/mt/99230563/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [master][PATCH 6/6] classe-recipes: npm: Add support for dependencies and devDependencies

2023-05-30 Thread belouargamohamed 
From: BELOUARGA Mohamed 

Adapt the npm classe for the shrinkwrap of Nodejs 18, and seperate
dependencies and dev dependencies.

Signed-off-by: BELOUARGA Mohamed 
---
 meta/classes-recipe/npm.bbclass | 21 -
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/meta/classes-recipe/npm.bbclass b/meta/classes-recipe/npm.bbclass
index c1944ca14e..91da3295f2 100644
--- a/meta/classes-recipe/npm.bbclass
+++ b/meta/classes-recipe/npm.bbclass
@@ -153,7 +153,11 @@ python npm_do_configure() {
 
 if has_shrinkwrap_file:
cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
-   cached_shrinkwrap.pop("dependencies", None)
+   for package in orig_shrinkwrap["packages"]:
+if package != "":
+cached_shrinkwrap["packages"].pop(package, None)
+   cached_shrinkwrap["packages"][""].pop("dependencies", None)
+   cached_shrinkwrap["packages"][""].pop("devDependencies", None)
cached_shrinkwrap["packages"][""].pop("peerDependencies", None)
 
 # Manage the dependencies
@@ -177,9 +181,16 @@ python npm_do_configure() {
 dep["integrity"] = _npm_integrity(tarball)
 if params.get("dev", False):
 dep["dev"] = True
-if "dependencies" not in cached_shrinkwrap:
-cached_shrinkwrap["dependencies"] = {}
-cached_shrinkwrap["dependencies"][name] = dep
+if "devDependencies" not in cached_shrinkwrap["packages"][""]:
+cached_shrinkwrap["packages"][""]["devDependencies"] = {}
+cached_shrinkwrap["packages"][""]["devDependencies"][name] = 
pkg['version']
+
+else:
+if "dependencies" not in cached_shrinkwrap["packages"][""]:
+cached_shrinkwrap["packages"][""]["dependencies"] = {}
+cached_shrinkwrap["packages"][""]["dependencies"][name] = 
pkg['version']
+
+cached_shrinkwrap["packages"][destsuffix] = dep
 # Display progress
 nonlocal progress_done
 progress_done += 1
@@ -212,7 +223,7 @@ python npm_do_configure() {
 # Configure the cached manifest file and cached shrinkwrap file
 def _update_manifest(depkey):
 for name in orig_manifest.get(depkey, {}):
-version = cached_shrinkwrap["dependencies"][name]["version"]
+version = cached_shrinkwrap["packages"][""][depkey][name]
 if depkey not in cached_manifest:
 cached_manifest[depkey] = {}
 cached_manifest[depkey][name] = version
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181966): 
https://lists.openembedded.org/g/openembedded-core/message/181966
Mute This Topic: https://lists.openembedded.org/mt/99230564/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [master][PATCH 4/6] recipetool: create: npm: Add support to handle peer dependencies

2023-05-30 Thread belouargamohamed 
From: BELOUARGA Mohamed 

NPM changed its manner to handle peer dependencies over its versions.
Before NPM 3: NPM installs automatically peer dependencies
between NPM 3 and 7: NPM shows a warning about peer dependencies
After NPM 3: NPM reworked its manner how to handle peer dependencies

The shrinkwrap doesn't have the parameters of the peer dependencies, so we 
cannot
fetch them. in the same time peer dependencies are not direct dependencies, 
they should
be installed as run time dependencies.

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 20 
 1 file changed, 20 insertions(+)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 25e7ddb472..113a89f6a6 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -146,6 +146,23 @@ class NpmRecipeHandler(RecipeHandler):
 foreach_dependencies(shrinkwrap, _handle_dependency, dev)
 
 return licfiles, packages
+
+# Handle the peer dependencies   
+def _handle_peer_dependency(self, shrinkwrap_file):
+"""Check if package has peer dependencies and show warning if it is 
the case"""
+with open(shrinkwrap_file, "r") as f:
+shrinkwrap = json.load(f)
+
+packages = shrinkwrap.get("packages", {})
+peer_deps = packages.get("", {}).get("peerDependencies", {})
+
+for peer_dep in peer_deps:
+peer_dep_yocto_name = npm_package(peer_dep)
+bb.warn(peer_dep + " is a peer dependencie of the actual package. 
" + 
+"Please add this peer dependencie to the RDEPENDS variable as %s 
and generate its recipe with devtool"
+% peer_dep_yocto_name)
+
+
 
 def process(self, srctree, classes, lines_before, lines_after, handled, 
extravalues):
 """Handle the npm recipe creation"""
@@ -283,6 +300,9 @@ class NpmRecipeHandler(RecipeHandler):
 classes.append("npm")
 handled.append("buildsystem")
 
+# Check if package has peer dependencies and inform the user
+self._handle_peer_dependency(shrinkwrap_file)
+
 return True
 
 def register_recipe_handlers(handlers):
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181964): 
https://lists.openembedded.org/g/openembedded-core/message/181964
Mute This Topic: https://lists.openembedded.org/mt/99230560/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [master][PATCH 3/6] recipetool: create: npm: Add support for the new format of the shrinkwrap file

2023-05-30 Thread belouargamohamed 
From: BELOUARGA Mohamed 

The shrinkwrap file changed its format, but npm does not version this file. So 
we can use it properly.
The actual changes make the script check if the npm package has dependencies in 
the actual shrinkwrap format.

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index e667a4d19b..25e7ddb472 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -134,11 +134,10 @@ class NpmRecipeHandler(RecipeHandler):
 licfiles.append(os.path.relpath(readme, srctree))
 
 # Handle the dependencies
-def _handle_dependency(name, params, deptree):
+def _handle_dependency(name, params, destdir):
+deptree = destdir.split('node_modules/')
 suffix = "-".join([npm_package(dep) for dep in deptree])
-destdirs = [os.path.join("node_modules", dep) for dep in deptree]
-destdir = os.path.join(*destdirs)
-packages["${PN}-" + suffix] = destdir
+packages["${PN}" + suffix] = destdir
 _licfiles_append_fallback_readme_files(destdir)
 
 with open(shrinkwrap_file, "r") as f:
@@ -234,7 +233,7 @@ class NpmRecipeHandler(RecipeHandler):
 value = origvalue.replace("version=" + data["version"], 
"version=${PV}")
 value = value.replace("version=latest", "version=${PV}")
 values = [line.strip() for line in value.strip('\n').splitlines()]
-if "dependencies" in shrinkwrap:
+if "dependencies" in shrinkwrap.get("packages", {}).get("", {}):
 values.append(url_recipe)
 return values, None, 4, False
 
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181963): 
https://lists.openembedded.org/g/openembedded-core/message/181963
Mute This Topic: https://lists.openembedded.org/mt/99230557/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [master][PATCH 2/6] classes: npm: Handle peer dependencies for npm packages

2023-05-30 Thread belouargamohamed 
From: BELOUARGA Mohamed 

NPM changed its manner to handle peer dependencies over its versions.
Before NPM 3: NPM installs automatically peer dependencies
between NPM 3 and 7: NPM shows a warning about peer dependencies
After NPM 3: NPM reworked its manner how to handle peer dependencies

The shrinkwrap doesn't have the parameters of the peer dependencies, so we 
cannot
fetch them. in the same time peer dependencies are not direct dependencies, 
they should
be installed as run time dependencies.

Signed-off-by: BELOUARGA Mohamed 
---
 meta/classes-recipe/npm.bbclass | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/meta/classes-recipe/npm.bbclass b/meta/classes-recipe/npm.bbclass
index 639f461a3a..92e59fefce 100644
--- a/meta/classes-recipe/npm.bbclass
+++ b/meta/classes-recipe/npm.bbclass
@@ -109,6 +109,7 @@ python npm_do_configure() {
 import tempfile
 from bb.fetch2.npm import NpmEnvironment
 from bb.fetch2.npm import npm_unpack
+from bb.fetch2.npm import npm_package
 from bb.fetch2.npmsw import foreach_dependencies
 from bb.progress import OutOfProgressHandler
 from oe.npm_registry import NpmRegistry
@@ -169,6 +170,7 @@ python npm_do_configure() {
 if has_shrinkwrap_file:
cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
cached_shrinkwrap.pop("dependencies", None)
+   cached_shrinkwrap["packages"][""].pop("peerDependencies", None)
 
 # Manage the dependencies
 progress = OutOfProgressHandler(d, r"^(\d+)/(\d+)$")
@@ -203,6 +205,19 @@ python npm_do_configure() {
 if has_shrinkwrap_file:
 foreach_dependencies(orig_shrinkwrap, _count_dependency, dev)
 foreach_dependencies(orig_shrinkwrap, _cache_dependency, dev)
+
+# Manage Peer Dependencies
+if has_shrinkwrap_file:
+packages = orig_shrinkwrap.get("packages", {})
+peer_deps = packages.get("", {}).get("peerDependencies", {})
+package_runtime_dependencies = d.getVar("RDEPENDS:%s" % d.getVar("PN"))
+
+for peer_dep in peer_deps:
+peer_dep_yocto_name = npm_package(peer_dep)
+if peer_dep_yocto_name not in package_runtime_dependencies:
+bb.warn(peer_dep + " is a peer dependencie that is not in 
RDEPENDS variable. " + 
+"Please add this peer dependencie to the RDEPENDS variable as 
%s and generate its recipe with devtool"
+% peer_dep_yocto_name)
 
 # Configure the main package
 with tempfile.TemporaryDirectory() as tmpdir:
@@ -279,6 +294,9 @@ python npm_do_compile() {
 args.append(("target_arch", d.getVar("NPM_ARCH")))
 args.append(("build-from-source", "true"))
 
+# Don't install peer dependencies as they should be in RDEPENDS 
variable
+args.append(("legacy-peer-deps", "true"))
+
 # Pack and install the main package
 (tarball, _) = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
 cmd = "npm install %s %s" % (shlex.quote(tarball), 
d.getVar("EXTRA_OENPM"))
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181962): 
https://lists.openembedded.org/g/openembedded-core/message/181962
Mute This Topic: https://lists.openembedded.org/mt/99230553/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [master][PATCH 1/6] recipetool: create: npm: Remove duplicate function to not have future conflicts

2023-05-30 Thread belouargamohamed 
From: BELOUARGA Mohamed 

Npm packages do not have yocto friendly names. fore instance we can have names 
like
"@example/npmPackage"

npm fetcher has a function that convert these names to yocto friendly names.
But in recipe tool we have an other function (duplicate).

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 14 +++---
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 3394a89970..e667a4d19b 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -13,6 +13,7 @@ import sys
 import tempfile
 import bb
 from bb.fetch2.npm import NpmEnvironment
+from bb.fetch2.npm import npm_package
 from bb.fetch2.npmsw import foreach_dependencies
 from recipetool.create import RecipeHandler
 from recipetool.create import get_license_md5sums
@@ -30,15 +31,6 @@ def tinfoil_init(instance):
 class NpmRecipeHandler(RecipeHandler):
 """Class to handle the npm recipe creation"""
 
-@staticmethod
-def _npm_name(name):
-"""Generate a Yocto friendly npm name"""
-name = re.sub("/", "-", name)
-name = name.lower()
-name = re.sub(r"[^\-a-z0-9]", "", name)
-name = name.strip("-")
-return name
-
 @staticmethod
 def _get_registry(lines):
 """Get the registry value from the 'npm://registry' url"""
@@ -143,7 +135,7 @@ class NpmRecipeHandler(RecipeHandler):
 
 # Handle the dependencies
 def _handle_dependency(name, params, deptree):
-suffix = "-".join([self._npm_name(dep) for dep in deptree])
+suffix = "-".join([npm_package(dep) for dep in deptree])
 destdirs = [os.path.join("node_modules", dep) for dep in deptree]
 destdir = os.path.join(*destdirs)
 packages["${PN}-" + suffix] = destdir
@@ -173,7 +165,7 @@ class NpmRecipeHandler(RecipeHandler):
 if "name" not in data or "version" not in data:
 return False
 
-extravalues["PN"] = self._npm_name(data["name"])
+extravalues["PN"] = npm_package(data["name"])
 extravalues["PV"] = data["version"]
 
 if "description" in data:
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181961): 
https://lists.openembedded.org/g/openembedded-core/message/181961
Mute This Topic: https://lists.openembedded.org/mt/99230551/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts

2023-05-29 Thread belouargamohamed 
Yes, the same issues are present in master branch.

But this series of patches won't work on master, it needs some adaptation.

Because in kirkstone we have Nodejs 16.19.1 and in master we have Nodejs 18.16, 
and they generate shrinkwrap files that are slightly different.

For Mickledore, I didn't test yet but I think that it will work like in the 
master branch.

Note: these changes work only if my patch for bitbake is accepted: 
https://lists.openembedded.org/g/bitbake-devel/topic/kirkstone_patch_fetch2/99163598?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,99163598,previd%3D1685367798112436212,nextid%3D1683629533018906083=1685367798112436212=1683629533018906083

Best regards,
BELOUARGA Mohamed

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181892): 
https://lists.openembedded.org/g/openembedded-core/message/181892
Mute This Topic: https://lists.openembedded.org/mt/99163838/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 3/5] recipetool: create: npm: Add support for the new format of the shrinkwrap file

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

The shrinkwrap file changed its format, but npm does not version this file. So 
we can use it properly.
The actual changes make the script check if the npm package has dependencies in 
the actual shrinkwrap format.

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index e667a4d19b..25e7ddb472 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -134,11 +134,10 @@ class NpmRecipeHandler(RecipeHandler):
 licfiles.append(os.path.relpath(readme, srctree))
 
 # Handle the dependencies
-def _handle_dependency(name, params, deptree):
+def _handle_dependency(name, params, destdir):
+deptree = destdir.split('node_modules/')
 suffix = "-".join([npm_package(dep) for dep in deptree])
-destdirs = [os.path.join("node_modules", dep) for dep in deptree]
-destdir = os.path.join(*destdirs)
-packages["${PN}-" + suffix] = destdir
+packages["${PN}" + suffix] = destdir
 _licfiles_append_fallback_readme_files(destdir)
 
 with open(shrinkwrap_file, "r") as f:
@@ -234,7 +233,7 @@ class NpmRecipeHandler(RecipeHandler):
 value = origvalue.replace("version=" + data["version"], 
"version=${PV}")
 value = value.replace("version=latest", "version=${PV}")
 values = [line.strip() for line in value.strip('\n').splitlines()]
-if "dependencies" in shrinkwrap:
+if "dependencies" in shrinkwrap.get("packages", {}).get("", {}):
 values.append(url_recipe)
 return values, None, 4, False
 
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181800): 
https://lists.openembedded.org/g/openembedded-core/message/181800
Mute This Topic: https://lists.openembedded.org/mt/99166270/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 5/5] classes: npm: Add support for the new format of the shrinkwrap file

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

1 - Adapt do_configure to the new format of the shrinkwrap

2 - Remove useless function _npmsw_dependency_dict because the dictionnary
is already given by npmsw:foreach_dependencies

3 - Rename arguments of callback functions

Signed-off-by: BELOUARGA Mohamed 
---
 meta/classes/npm.bbclass | 27 ++-
 1 file changed, 6 insertions(+), 21 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index abc4c6f92b..6899a64909 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -129,22 +129,6 @@ python npm_do_configure() {
 sha512 = bb.utils.sha512_file(tarball)
 return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
 
-def _npmsw_dependency_dict(orig, deptree):
-"""
-Return the sub dictionary in the 'orig' dictionary corresponding to the
-'deptree' dependency tree. This function follows the shrinkwrap file
-format.
-"""
-ptr = orig
-for dep in deptree:
-if "dependencies" not in ptr:
-ptr["dependencies"] = {}
-ptr = ptr["dependencies"]
-if dep not in ptr:
-ptr[dep] = {}
-ptr = ptr[dep]
-return ptr
-
 # Manage the manifest file and shrinkwrap files
 orig_manifest_file = d.expand("${S}/package.json")
 orig_shrinkwrap_file = d.expand("${S}/npm-shrinkwrap.json")
@@ -176,24 +160,25 @@ python npm_do_configure() {
 progress_total = 1 # also count the main package
 progress_done = 0
 
-def _count_dependency(name, params, deptree):
+def _count_dependency(name, params, destsuffix):
 nonlocal progress_total
 progress_total += 1
 
-def _cache_dependency(name, params, deptree):
-destsubdirs = [os.path.join("node_modules", dep) for dep in deptree]
-destsuffix = os.path.join(*destsubdirs)
+def _cache_dependency(name, params, destsuffix):
 with tempfile.TemporaryDirectory() as tmpdir:
 # Add the dependency to the npm cache
 destdir = os.path.join(d.getVar("S"), destsuffix)
 (tarball, pkg) = npm_pack(env, destdir, tmpdir)
 _npm_cache_add(tarball, pkg)
 # Add its signature to the cached shrinkwrap
-dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
+dep = params
 dep["version"] = pkg['version']
 dep["integrity"] = _npm_integrity(tarball)
 if params.get("dev", False):
 dep["dev"] = True
+if "dependencies" not in cached_shrinkwrap:
+cached_shrinkwrap["dependencies"] = {}
+cached_shrinkwrap["dependencies"][name] = dep
 # Display progress
 nonlocal progress_done
 progress_done += 1
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181802): 
https://lists.openembedded.org/g/openembedded-core/message/181802
Mute This Topic: https://lists.openembedded.org/mt/99166272/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 4/5] recipetool: create: npm: Add support to handle peer dependencies

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

NPM changed its manner to handle peer dependencies over its versions:

 - Before NPM 3: NPM installs automatically peer dependencies
 - Between NPM 3 and 7: NPM shows a warning about peer dependencies
 - After NPM 3: NPM reworked its manner how to handle peer dependencies

The shrinkwrap doesn't have the parameters of the peer dependencies, so we 
cannot
fetch them. in the same time peer dependencies are not direct dependencies, 
they should
be installed as run time dependencies.

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 20 
 1 file changed, 20 insertions(+)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 25e7ddb472..113a89f6a6 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -146,6 +146,23 @@ class NpmRecipeHandler(RecipeHandler):
 foreach_dependencies(shrinkwrap, _handle_dependency, dev)
 
 return licfiles, packages
+
+# Handle the peer dependencies   
+def _handle_peer_dependency(self, shrinkwrap_file):
+"""Check if package has peer dependencies and show warning if it is 
the case"""
+with open(shrinkwrap_file, "r") as f:
+shrinkwrap = json.load(f)
+
+packages = shrinkwrap.get("packages", {})
+peer_deps = packages.get("", {}).get("peerDependencies", {})
+
+for peer_dep in peer_deps:
+peer_dep_yocto_name = npm_package(peer_dep)
+bb.warn(peer_dep + " is a peer dependencie of the actual package. 
" + 
+"Please add this peer dependencie to the RDEPENDS variable as %s 
and generate its recipe with devtool"
+% peer_dep_yocto_name)
+
+
 
 def process(self, srctree, classes, lines_before, lines_after, handled, 
extravalues):
 """Handle the npm recipe creation"""
@@ -283,6 +300,9 @@ class NpmRecipeHandler(RecipeHandler):
 classes.append("npm")
 handled.append("buildsystem")
 
+# Check if package has peer dependencies and inform the user
+self._handle_peer_dependency(shrinkwrap_file)
+
 return True
 
 def register_recipe_handlers(handlers):
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181801): 
https://lists.openembedded.org/g/openembedded-core/message/181801
Mute This Topic: https://lists.openembedded.org/mt/99166271/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 2/5] classes: npm: Handle peer dependencies for npm packages

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

NPM changed its manner to handle peer dependencies over its versions:
 - Before NPM 3: NPM installs automatically peer dependencies
 - Between NPM 3 and 7: NPM shows a warning about peer dependencies
 - After NPM 3: NPM reworked its manner how to handle peer dependencies

The shrinkwrap doesn't have the parameters of the peer dependencies, so we 
cannot
fetch them. in the same time peer dependencies are not direct dependencies, 
they should
be installed as run time dependencies.

Signed-off-by: BELOUARGA Mohamed 
---
 meta/classes/npm.bbclass | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 8379c7b988..abc4c6f92b 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -108,6 +108,7 @@ python npm_do_configure() {
 import tempfile
 from bb.fetch2.npm import NpmEnvironment
 from bb.fetch2.npm import npm_unpack
+from bb.fetch2.npm import npm_package
 from bb.fetch2.npmsw import foreach_dependencies
 from bb.progress import OutOfProgressHandler
 from oe.npm_registry import NpmRegistry
@@ -168,6 +169,7 @@ python npm_do_configure() {
 if has_shrinkwrap_file:
cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
cached_shrinkwrap.pop("dependencies", None)
+   cached_shrinkwrap["packages"][""].pop("peerDependencies", None)
 
 # Manage the dependencies
 progress = OutOfProgressHandler(d, r"^(\d+)/(\d+)$")
@@ -202,6 +204,19 @@ python npm_do_configure() {
 if has_shrinkwrap_file:
 foreach_dependencies(orig_shrinkwrap, _count_dependency, dev)
 foreach_dependencies(orig_shrinkwrap, _cache_dependency, dev)
+
+# Manage Peer Dependencies
+if has_shrinkwrap_file:
+packages = orig_shrinkwrap.get("packages", {})
+peer_deps = packages.get("", {}).get("peerDependencies", {})
+package_runtime_dependencies = d.getVar("RDEPENDS:%s" % d.getVar("PN"))
+
+for peer_dep in peer_deps:
+peer_dep_yocto_name = npm_package(peer_dep)
+if peer_dep_yocto_name not in package_runtime_dependencies:
+bb.warn(peer_dep + " is a peer dependencie that is not in 
RDEPENDS variable. " + 
+"Please add this peer dependencie to the RDEPENDS variable as 
%s and generate its recipe with devtool"
+% peer_dep_yocto_name)
 
 # Configure the main package
 with tempfile.TemporaryDirectory() as tmpdir:
@@ -277,6 +292,9 @@ python npm_do_compile() {
 args.append(("target_arch", d.getVar("NPM_ARCH")))
 args.append(("build-from-source", "true"))
 
+# Don't install peer dependencies as they should be in RDEPENDS 
variable
+args.append(("legacy-peer-deps", "true"))
+
 # Pack and install the main package
 (tarball, _) = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
 cmd = "npm install %s %s" % (shlex.quote(tarball), 
d.getVar("EXTRA_OENPM"))
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181799): 
https://lists.openembedded.org/g/openembedded-core/message/181799
Mute This Topic: https://lists.openembedded.org/mt/99166269/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

Npm packages do not have yocto friendly names. fore instance we can have names 
like
"@example/npmPackage"

npm fetcher has a function that convert these names to yocto friendly names.
But in recipe tool we have an other function (duplicate).

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 14 +++---
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 3394a89970..e667a4d19b 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -13,6 +13,7 @@ import sys
 import tempfile
 import bb
 from bb.fetch2.npm import NpmEnvironment
+from bb.fetch2.npm import npm_package
 from bb.fetch2.npmsw import foreach_dependencies
 from recipetool.create import RecipeHandler
 from recipetool.create import get_license_md5sums
@@ -30,15 +31,6 @@ def tinfoil_init(instance):
 class NpmRecipeHandler(RecipeHandler):
 """Class to handle the npm recipe creation"""
 
-@staticmethod
-def _npm_name(name):
-"""Generate a Yocto friendly npm name"""
-name = re.sub("/", "-", name)
-name = name.lower()
-name = re.sub(r"[^\-a-z0-9]", "", name)
-name = name.strip("-")
-return name
-
 @staticmethod
 def _get_registry(lines):
 """Get the registry value from the 'npm://registry' url"""
@@ -143,7 +135,7 @@ class NpmRecipeHandler(RecipeHandler):
 
 # Handle the dependencies
 def _handle_dependency(name, params, deptree):
-suffix = "-".join([self._npm_name(dep) for dep in deptree])
+suffix = "-".join([npm_package(dep) for dep in deptree])
 destdirs = [os.path.join("node_modules", dep) for dep in deptree]
 destdir = os.path.join(*destdirs)
 packages["${PN}-" + suffix] = destdir
@@ -173,7 +165,7 @@ class NpmRecipeHandler(RecipeHandler):
 if "name" not in data or "version" not in data:
 return False
 
-extravalues["PN"] = self._npm_name(data["name"])
+extravalues["PN"] = npm_package(data["name"])
 extravalues["PV"] = data["version"]
 
 if "description" in data:
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181798): 
https://lists.openembedded.org/g/openembedded-core/message/181798
Mute This Topic: https://lists.openembedded.org/mt/99163838/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 5/5] classes: npm: Add support for the new format of the shrinkwrap file

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

1 - Adapt do_configure to the new format of the shrinkwrap

2 - Remove useless function _npmsw_dependency_dict because the dictionnary
is already given by npmsw:foreach_dependencies

3 - Rename arguments of callback functions

Signed-off-by: BELOUARGA Mohamed 
---
 meta/classes/npm.bbclass | 27 ++-
 1 file changed, 6 insertions(+), 21 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index abc4c6f92b..6899a64909 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -129,22 +129,6 @@ python npm_do_configure() {
 sha512 = bb.utils.sha512_file(tarball)
 return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
 
-def _npmsw_dependency_dict(orig, deptree):
-"""
-Return the sub dictionary in the 'orig' dictionary corresponding to the
-'deptree' dependency tree. This function follows the shrinkwrap file
-format.
-"""
-ptr = orig
-for dep in deptree:
-if "dependencies" not in ptr:
-ptr["dependencies"] = {}
-ptr = ptr["dependencies"]
-if dep not in ptr:
-ptr[dep] = {}
-ptr = ptr[dep]
-return ptr
-
 # Manage the manifest file and shrinkwrap files
 orig_manifest_file = d.expand("${S}/package.json")
 orig_shrinkwrap_file = d.expand("${S}/npm-shrinkwrap.json")
@@ -176,24 +160,25 @@ python npm_do_configure() {
 progress_total = 1 # also count the main package
 progress_done = 0
 
-def _count_dependency(name, params, deptree):
+def _count_dependency(name, params, destsuffix):
 nonlocal progress_total
 progress_total += 1
 
-def _cache_dependency(name, params, deptree):
-destsubdirs = [os.path.join("node_modules", dep) for dep in deptree]
-destsuffix = os.path.join(*destsubdirs)
+def _cache_dependency(name, params, destsuffix):
 with tempfile.TemporaryDirectory() as tmpdir:
 # Add the dependency to the npm cache
 destdir = os.path.join(d.getVar("S"), destsuffix)
 (tarball, pkg) = npm_pack(env, destdir, tmpdir)
 _npm_cache_add(tarball, pkg)
 # Add its signature to the cached shrinkwrap
-dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
+dep = params
 dep["version"] = pkg['version']
 dep["integrity"] = _npm_integrity(tarball)
 if params.get("dev", False):
 dep["dev"] = True
+if "dependencies" not in cached_shrinkwrap:
+cached_shrinkwrap["dependencies"] = {}
+cached_shrinkwrap["dependencies"][name] = dep
 # Display progress
 nonlocal progress_done
 progress_done += 1
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181795): 
https://lists.openembedded.org/g/openembedded-core/message/181795
Mute This Topic: https://lists.openembedded.org/mt/99163842/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 4/5] recipetool: create: npm: Add support to handle peer dependencies

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

NPM changed its manner to handle peer dependencies over its versions:

 - Before NPM 3: NPM installs automatically peer dependencies
 - Between NPM 3 and 7: NPM shows a warning about peer dependencies
 - After NPM 3: NPM reworked its manner how to handle peer dependencies

The shrinkwrap doesn't have the parameters of the peer dependencies, so we 
cannot
fetch them. in the same time peer dependencies are not direct dependencies, 
they should
be installed as run time dependencies.

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 20 
 1 file changed, 20 insertions(+)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 25e7ddb472..113a89f6a6 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -146,6 +146,23 @@ class NpmRecipeHandler(RecipeHandler):
 foreach_dependencies(shrinkwrap, _handle_dependency, dev)
 
 return licfiles, packages
+
+# Handle the peer dependencies   
+def _handle_peer_dependency(self, shrinkwrap_file):
+"""Check if package has peer dependencies and show warning if it is 
the case"""
+with open(shrinkwrap_file, "r") as f:
+shrinkwrap = json.load(f)
+
+packages = shrinkwrap.get("packages", {})
+peer_deps = packages.get("", {}).get("peerDependencies", {})
+
+for peer_dep in peer_deps:
+peer_dep_yocto_name = npm_package(peer_dep)
+bb.warn(peer_dep + " is a peer dependencie of the actual package. 
" + 
+"Please add this peer dependencie to the RDEPENDS variable as %s 
and generate its recipe with devtool"
+% peer_dep_yocto_name)
+
+
 
 def process(self, srctree, classes, lines_before, lines_after, handled, 
extravalues):
 """Handle the npm recipe creation"""
@@ -283,6 +300,9 @@ class NpmRecipeHandler(RecipeHandler):
 classes.append("npm")
 handled.append("buildsystem")
 
+# Check if package has peer dependencies and inform the user
+self._handle_peer_dependency(shrinkwrap_file)
+
 return True
 
 def register_recipe_handlers(handlers):
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181794): 
https://lists.openembedded.org/g/openembedded-core/message/181794
Mute This Topic: https://lists.openembedded.org/mt/99163841/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 3/5] recipetool: create: npm: Add support for the new format of the shrinkwrap file

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

The shrinkwrap file changed its format, but npm does not version this file. So 
we can use it properly.
The actual changes make the script check if the npm package has dependencies in 
the actual shrinkwrap format.

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index e667a4d19b..25e7ddb472 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -134,11 +134,10 @@ class NpmRecipeHandler(RecipeHandler):
 licfiles.append(os.path.relpath(readme, srctree))
 
 # Handle the dependencies
-def _handle_dependency(name, params, deptree):
+def _handle_dependency(name, params, destdir):
+deptree = destdir.split('node_modules/')
 suffix = "-".join([npm_package(dep) for dep in deptree])
-destdirs = [os.path.join("node_modules", dep) for dep in deptree]
-destdir = os.path.join(*destdirs)
-packages["${PN}-" + suffix] = destdir
+packages["${PN}" + suffix] = destdir
 _licfiles_append_fallback_readme_files(destdir)
 
 with open(shrinkwrap_file, "r") as f:
@@ -234,7 +233,7 @@ class NpmRecipeHandler(RecipeHandler):
 value = origvalue.replace("version=" + data["version"], 
"version=${PV}")
 value = value.replace("version=latest", "version=${PV}")
 values = [line.strip() for line in value.strip('\n').splitlines()]
-if "dependencies" in shrinkwrap:
+if "dependencies" in shrinkwrap.get("packages", {}).get("", {}):
 values.append(url_recipe)
 return values, None, 4, False
 
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181793): 
https://lists.openembedded.org/g/openembedded-core/message/181793
Mute This Topic: https://lists.openembedded.org/mt/99163840/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 2/5] classes: npm: Handle peer dependencies for npm packages

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

NPM changed its manner to handle peer dependencies over its versions:
 - Before NPM 3: NPM installs automatically peer dependencies
 - Between NPM 3 and 7: NPM shows a warning about peer dependencies
 - After NPM 3: NPM reworked its manner how to handle peer dependencies

The shrinkwrap doesn't have the parameters of the peer dependencies, so we 
cannot
fetch them. in the same time peer dependencies are not direct dependencies, 
they should
be installed as run time dependencies.

Signed-off-by: BELOUARGA Mohamed 
---
 meta/classes/npm.bbclass | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 8379c7b988..abc4c6f92b 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -108,6 +108,7 @@ python npm_do_configure() {
 import tempfile
 from bb.fetch2.npm import NpmEnvironment
 from bb.fetch2.npm import npm_unpack
+from bb.fetch2.npm import npm_package
 from bb.fetch2.npmsw import foreach_dependencies
 from bb.progress import OutOfProgressHandler
 from oe.npm_registry import NpmRegistry
@@ -168,6 +169,7 @@ python npm_do_configure() {
 if has_shrinkwrap_file:
cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
cached_shrinkwrap.pop("dependencies", None)
+   cached_shrinkwrap["packages"][""].pop("peerDependencies", None)
 
 # Manage the dependencies
 progress = OutOfProgressHandler(d, r"^(\d+)/(\d+)$")
@@ -202,6 +204,19 @@ python npm_do_configure() {
 if has_shrinkwrap_file:
 foreach_dependencies(orig_shrinkwrap, _count_dependency, dev)
 foreach_dependencies(orig_shrinkwrap, _cache_dependency, dev)
+
+# Manage Peer Dependencies
+if has_shrinkwrap_file:
+packages = orig_shrinkwrap.get("packages", {})
+peer_deps = packages.get("", {}).get("peerDependencies", {})
+package_runtime_dependencies = d.getVar("RDEPENDS:%s" % d.getVar("PN"))
+
+for peer_dep in peer_deps:
+peer_dep_yocto_name = npm_package(peer_dep)
+if peer_dep_yocto_name not in package_runtime_dependencies:
+bb.warn(peer_dep + " is a peer dependencie that is not in 
RDEPENDS variable. " + 
+"Please add this peer dependencie to the RDEPENDS variable as 
%s and generate its recipe with devtool"
+% peer_dep_yocto_name)
 
 # Configure the main package
 with tempfile.TemporaryDirectory() as tmpdir:
@@ -277,6 +292,9 @@ python npm_do_compile() {
 args.append(("target_arch", d.getVar("NPM_ARCH")))
 args.append(("build-from-source", "true"))
 
+# Don't install peer dependencies as they should be in RDEPENDS 
variable
+args.append(("legacy-peer-deps", "true"))
+
 # Pack and install the main package
 (tarball, _) = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
 cmd = "npm install %s %s" % (shlex.quote(tarball), 
d.getVar("EXTRA_OENPM"))
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181792): 
https://lists.openembedded.org/g/openembedded-core/message/181792
Mute This Topic: https://lists.openembedded.org/mt/99163839/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [kirkstone][PATCH 1/5] recipetool: create: npm: Remove duplicate function to not have future conflicts

2023-05-27 Thread belouargamohamed 
From: BELOUARGA Mohamed 

Npm packages do not have yocto friendly names. fore instance we can have names 
like
"@example/npmPackage"

npm fetcher has a function that convert these names to yocto friendly names.
But in recipe tool we have an other function (duplicate).

Signed-off-by: BELOUARGA Mohamed 
---
 scripts/lib/recipetool/create_npm.py | 14 +++---
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 3394a89970..e667a4d19b 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -13,6 +13,7 @@ import sys
 import tempfile
 import bb
 from bb.fetch2.npm import NpmEnvironment
+from bb.fetch2.npm import npm_package
 from bb.fetch2.npmsw import foreach_dependencies
 from recipetool.create import RecipeHandler
 from recipetool.create import get_license_md5sums
@@ -30,15 +31,6 @@ def tinfoil_init(instance):
 class NpmRecipeHandler(RecipeHandler):
 """Class to handle the npm recipe creation"""
 
-@staticmethod
-def _npm_name(name):
-"""Generate a Yocto friendly npm name"""
-name = re.sub("/", "-", name)
-name = name.lower()
-name = re.sub(r"[^\-a-z0-9]", "", name)
-name = name.strip("-")
-return name
-
 @staticmethod
 def _get_registry(lines):
 """Get the registry value from the 'npm://registry' url"""
@@ -143,7 +135,7 @@ class NpmRecipeHandler(RecipeHandler):
 
 # Handle the dependencies
 def _handle_dependency(name, params, deptree):
-suffix = "-".join([self._npm_name(dep) for dep in deptree])
+suffix = "-".join([npm_package(dep) for dep in deptree])
 destdirs = [os.path.join("node_modules", dep) for dep in deptree]
 destdir = os.path.join(*destdirs)
 packages["${PN}-" + suffix] = destdir
@@ -173,7 +165,7 @@ class NpmRecipeHandler(RecipeHandler):
 if "name" not in data or "version" not in data:
 return False
 
-extravalues["PN"] = self._npm_name(data["name"])
+extravalues["PN"] = npm_package(data["name"])
 extravalues["PV"] = data["version"]
 
 if "description" in data:
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#181791): 
https://lists.openembedded.org/g/openembedded-core/message/181791
Mute This Topic: https://lists.openembedded.org/mt/99163838/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-