Hi,
Am 12.02.21 um 11:12 schrieb Josef Holzmayr:
Howdy!
Thanks for sharing, a few comments inline.
Am Fr., 12. Feb. 2021 um 11:03 Uhr schrieb Simon Vogl <si...@voxel.at>:
I have some remarks and questions about the npm/nodejs support in dunfell that
I wanted to share. We are creating nodejs-based IoT edge solutions and
upgrading our build environments to Dunfell one by one. In the course of this,
we are switching to the new npm-implementation and found a few small issues.
Firstly, the do_configure() task takes quite some time to complete. After a
quick analysis, I saw that most of the time is being spent in creating the
npmrc files while packing the dependent packages. I wrote a small workaround to
directly create the file instead of calling 'npm config', which results in a
3x-4x speedup:
Signed-off-by: Simon Vogl <si...@voxel.at>
---
lib/bb/fetch2/npm.py | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 4789850..2720d87 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -97,13 +97,18 @@ class NpmEnvironment(object):
cmd = "NPM_CONFIG_GLOBALCONFIG=%s " % cfgfile + cmd
return runfetchcmd(cmd, d, workdir=workdir)
+ cfg = open(cfgfile, "a")
if self.configs:
for key, value in self.configs:
- _run("npm config set %s %s" % (key, shlex.quote(value)))
+ cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+ #_run("npm config set %s %s" % (key, shlex.quote(value)))
if configs:
for key, value in configs:
- _run("npm config set %s %s" % (key, shlex.quote(value)))
+ cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+ # _run("npm config set %s %s" % (key, shlex.quote(value)))
+
+ cfg.close()
if args:
for key, value in args:
--
2.7.4
Are there any side effects that I did not stumble over yet? And I'd LOVE to
have these calls running in a thread-pool for better performance...
The main side effect is that you're effectively patching poky, which
is bad for maintenance.
I know, that's why I'm asking in the first place. But performance here
is really really improvable.
Secondly, our projects are based on typescript, so a native compile step is
necessary to create a compiled version for packing. We experimented with a
separate release branch to check in compiled versions, but this is not easy to
handle. I played around with npm.bbclass and found a way to extend configure
(!) with a call to our build script before packaging:
diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 068032a1e5..31535098cf 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -170,6 +170,11 @@ python npm_do_configure() {
# Configure the main package
with tempfile.TemporaryDirectory() as tmpdir:
+ # install all (native) build dependencies, overrides npm cache:
+ ret = os.system("npm i")
+ # run build step:
+ env.run("npm run build", args=[], workdir=d.getVar("S"))
+
tarball = npm_pack(env, d.getVar("S"), tmpdir)
npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d)
As we have plain JS packages as well, I put the modified configure() in a
subclass and this works for us, but it does not look like a clean solution to
me. How do you other IoT'ers address this situation?
Again, patching poky is a bad idea. Creating custom bbclasses is much
neater: you could create a base include, and pull that together with
npm.bbclass into two final bbclasses of yours, like npm-js-voxel and
npm-ts-voxel. The former would not have the compilation step. And,
putting the typescript/webpack invocation into configure is also not
exactly how things are meant to work. I know that the dependency
tracking of npm is not easily compatible with bitbake, but the aim
should be to
1) have a recipe that provides typescript-native
2) DEPEND on typescript-native in the recipe which you need to compile
3) add a do_compile stage that does the work.
Agreed, and I have a patched configure in my own subclass without
changing the official codebase -- I just wanted to point where the
modification needs to take place.
I actually tried the approach that you propose by playing around with
configure_append / compile_prepend tasks, but these build steps are
called after the package has already been packed --> the compiled data
is not being installed, I'd have to re-pack things.
Agreed, a typescript-native package would be nice, on the other hand
this is where the npm-version-chaos comes in again: Many packages use
different tsc versions,...
Simon
Greetz
--
VoXel Interaction Design | www.voxel.at
DI Dr.techn. Simon Vogl | si...@voxel.at
Tomaschekweg 46 | +43 650 2323 555
A-4040 Linz - Austria |
Office address: Industriezeile 35, 4020 Linz (2nd floor)
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#52300): https://lists.yoctoproject.org/g/yocto/message/52300
Mute This Topic: https://lists.yoctoproject.org/mt/80579992/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
