[OE-core] Fun with npm shrinkwrap
Hi all,
devtool add / recipetool create do not work properly /wrt shrinkwrap
when it comes to node-red-node-serialport. npm shrinkwrap finds a number
of extraneous packages and refuses to generate an output.
There are various workarounds discussed in [1]. For me this seems to
work:
diff --git a/scripts/lib/recipetool/create_npm.py
b/scripts/lib/recipetool/create_npm.py
index cb8f338b8b..8ce952bb4c 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -68,6 +68,8 @@ class NpmRecipeHandler(RecipeHandler):
def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before,
d):
try:
runenv = dict(os.environ, PATH=d.getVar('PATH'))
+bb.process.run('rm -rf node_modules', cwd=srctree,
stderr=subprocess.STDOUT, env=runenv, shell=True)
+bb.process.run('npm install', cwd=srctree,
stderr=subprocess.STDOUT, env=runenv, shell=True)
bb.process.run('npm shrinkwrap', cwd=srctree,
stderr=subprocess.STDOUT, env=runenv, shell=True)
except bb.process.ExecutionError as e:
logger.warn('npm shrinkwrap failed:\n%s' % e.stdout)
But it may have unwanted side effects. So, what would be the proper way
to address this issue?
Thanks,
Jan
--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Fun with npm shrinkwrap
On 06/06/2017 01:17 PM, Jan Kiszka wrote: devtool add / recipetool create do not work properly /wrt shrinkwrap when it comes to node-red-node-serialport. npm shrinkwrap finds a number of extraneous packages and refuses to generate an output. There are various workarounds discussed in [1]. For me this seems to work: You forgot to include the actual [1] :) But it may have unwanted side effects. So, what would be the proper way to address this issue? A few months ago we had a long, big discussion about how to support these kinds of language environments in a way that is sustainable (I personally do not think the current approach with npm is viable in the long term). Here's the initial couple of emails: http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000481.html Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Fun with npm shrinkwrap
On 2017-06-06 16:12, Alexander Kanavin wrote: > On 06/06/2017 01:17 PM, Jan Kiszka wrote: >> devtool add / recipetool create do not work properly /wrt shrinkwrap >> when it comes to node-red-node-serialport. npm shrinkwrap finds a number >> of extraneous packages and refuses to generate an output. >> >> There are various workarounds discussed in [1]. For me this seems to >> work: > > You forgot to include the actual [1] :) Ah, sorry: https://github.com/npm/npm/issues/4435 > >> But it may have unwanted side effects. So, what would be the proper way >> to address this issue? > > A few months ago we had a long, big discussion about how to support > these kinds of language environments in a way that is sustainable (I > personally do not think the current approach with npm is viable in the > long term). Here's the initial couple of emails: > > http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000480.html > > > http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000481.html > Yeah, good points. Probably this problem is also related to OE reimplementing parts of the npm logic. But I suppose those concepts won't materialize very soon, will they? I'm also looking for a short-term solution, at least some reasonable tactic to generate such recipes in the meantime. Is there something in that thread I should dig into? Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Fun with npm shrinkwrap
On 06/06/2017 06:14 PM, Jan Kiszka wrote: Yeah, good points. Probably this problem is also related to OE reimplementing parts of the npm logic. But I suppose those concepts won't materialize very soon, will they? I'm also looking for a short-term solution, at least some reasonable tactic to generate such recipes in the meantime. Is there something in that thread I should dig into? What I am getting at is that it would be awesome if you convince your management that npm support in Yocto is a problem that needs a better solution and work on it. Yes, this means you can't work on product development for quite some time; this is the price for using open source projects without a commercial support contract :) My apologies if this is not the answer you were seeking. Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Fun with npm shrinkwrap
On 2017-06-07 13:43, Alexander Kanavin wrote: > On 06/06/2017 06:14 PM, Jan Kiszka wrote: > >> Yeah, good points. Probably this problem is also related to OE >> reimplementing parts of the npm logic. >> >> But I suppose those concepts won't materialize very soon, will they? I'm >> also looking for a short-term solution, at least some reasonable tactic >> to generate such recipes in the meantime. Is there something in that >> thread I should dig into? > > What I am getting at is that it would be awesome if you convince your > management that npm support in Yocto is a problem that needs a better > solution and work on it. Yes, this means you can't work on product > development for quite some time; this is the price for using open source > projects without a commercial support contract :) We are such a instance support :) - but we also need to get paid / pay our contractors. > > My apologies if this is not the answer you were seeking. > I'm getting your point, even more now as I ran into a package that does not even install at all (node-red-contrib-opcua). Can't promise anything at this point, but I will carry this forward, in all directions. Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Fun with npm shrinkwrap
If offline builds, license checking, or stable dependency versions are not important to you then you can use the meta-nodejs layer. It just uses the yarn tool directly https://github.com/imyller/meta-nodejs. On a hijacked side note has yarn been considered as a solution to some of our npm issues? It seems to be npm with built in licnese checking and better version determination. Might not cover 100% of our use cases, but then npm only seems to work 50% of the time without tweaking something anyways. -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Jan Kiszka Sent: Wednesday, June 07, 2017 7:04 AM To: Alexander Kanavin; openembedded-core@lists.openembedded.org Cc: Paul Eggleton Subject: Re: [OE-core] Fun with npm shrinkwrap On 2017-06-07 13:43, Alexander Kanavin wrote: > On 06/06/2017 06:14 PM, Jan Kiszka wrote: > >> Yeah, good points. Probably this problem is also related to OE >> reimplementing parts of the npm logic. >> >> But I suppose those concepts won't materialize very soon, will they? I'm >> also looking for a short-term solution, at least some reasonable tactic >> to generate such recipes in the meantime. Is there something in that >> thread I should dig into? > > What I am getting at is that it would be awesome if you convince your > management that npm support in Yocto is a problem that needs a better > solution and work on it. Yes, this means you can't work on product > development for quite some time; this is the price for using open source > projects without a commercial support contract :) We are such a instance support :) - but we also need to get paid / pay our contractors. > > My apologies if this is not the answer you were seeking. > I'm getting your point, even more now as I ran into a package that does not even install at all (node-red-contrib-opcua). Can't promise anything at this point, but I will carry this forward, in all directions. Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Fun with npm shrinkwrap
On 2017-06-07 16:09, Davis, Michael wrote: > If offline builds, license checking, or stable dependency versions are not > important to you then you can use the meta-nodejs layer. It just uses the > yarn tool directly https://github.com/imyller/meta-nodejs. > In fact, meta-nodejs is already in use here, and I've been told to give "inherit npm-base" a try. Is that what you mean? How "unreproducible" will builds become? The license checks are not that critical as I wouldn't trust them right now anyway. Offline builds correlate with reproduciblilty - we need to archive the artifacts and generate images with changes later on, independent of what happened outside meanwhile. Not a killer right now, but not an option on the long run. > > On a hijacked side note has yarn been considered as a solution to some of our > npm issues? > It seems to be npm with built in licnese checking and better version > determination. > Might not cover 100% of our use cases, but then npm only seems to work 50% of > the time without tweaking something anyways. > Can't comment on yarn, but another colleague pointed out that the machinery of arch-linux may also be worth a look, not only for the nodejs/npm galaxy. Thanks, Jan PS: Top-posting is considered evil. -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Fun with npm shrinkwrap
> In fact, meta-nodejs is already in use here, and I've been told to give > "inherit npm-base" a try. Is that what you mean? Yes inherit npm-base should give you what you need. With the before mentioned limitations. > How "unreproducible" will builds become? The license checks are not that > critical as I wouldn't trust them right now anyway. Offline builds > correlate with reproduciblilty - we need to archive the artifacts and > generate images with changes later on, independent of what happened > outside meanwhile. Not a killer right now, but not an option on the long > run. Mostly it just means you may get slightly different versions of dependencies over time. Depending mostly on how locked down your package.json and those of your deps are. https://docs.npmjs.com/misc/semver > PS: Top-posting is considered evil. The sad shame of having to use Outlook. No IMAP / POP and davmail hates me. Bottom-posting is dying a slow death these days. Only see it in Linux mailing lists for the most part. -- Mike -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
