Re: npm (mitigation)
2017-07-24 17:39 GMT+02:00 Ricardo Wurmus : > > Catonano writes: > > >> I am currently working on a guile version of what Sander did for Nix for > >> importing entire npm dependency trees, but this will likely lead to > lots of > >> programmatically > >> defined packages instead of the guix approach of mostly-manually > defining > >> each package. > >> > > > > What ? Who is Sander ans what did the do for Nix ? > > Jelle is referring to the work by Sander van der Burg, who presented > “Deploying NPM packages with the Nix package manager” at FOSDEM 2017. > > Here’s the video: > > https://video.fosdem.org/2017/K.4.601/deploying_npm_ > packages_with_nix.vp8.webm > > Yes, thanks to the both of you Also his blog posts are enlightening but I must confess that I'm not sure I can wrap my mind around the issue It's larger and more nuanced than it looks like I'll read some more about it in the next days Thanks Jelle for the pointers
Re: npm (mitigation)
Catonano writes: >> I am currently working on a guile version of what Sander did for Nix for >> importing entire npm dependency trees, but this will likely lead to lots of >> programmatically >> defined packages instead of the guix approach of mostly-manually defining >> each package. >> > > What ? Who is Sander ans what did the do for Nix ? Jelle is referring to the work by Sander van der Burg, who presented “Deploying NPM packages with the Nix package manager” at FOSDEM 2017. Here’s the video: https://video.fosdem.org/2017/K.4.601/deploying_npm_packages_with_nix.vp8.webm -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: npm (mitigation)
On Mon, Jul 17, 2017 at 11:45:29 +0200, Catonano wrote: > in my idea I would have build a database withh conditions for being non > free forr every npm package. > > So we could have queried the database for questions like: is there any non > free or non buildable package in the dependency tree of, say, the current > Jquery ? Being able to query the graph for non-free dependencies is good, yes. My concern is developing a (reasonably) fool-proof system for detecting those packages that doesn't require manual verification, which would be extremely costly, outside of a reasonable randomly-chosen set. I'm not saying it's impossible; it's just difficult with such wildly varying standards and carelessness with regards to licensing that is prominent in the JS community. But we have to start somewhere, so anything you can come up with would be good. :) > You might remember my post of a few months back about an attempt of mine to > crawl thhe npm registry and storing data found there. I do---I'm sorry if there are details that I missed or should know; I haven't been able to follow this too closely. I can be a bit of a parrot sometimes with certain issues. :x -- Mike Gerwitz Free Software Hacker+Activist | GNU Maintainer & Volunteer GPG: D6E9 B930 028A 6C38 F43B 2388 FEF6 3574 5E6F 6D05 https://mikegerwitz.com signature.asc Description: PGP signature
Re: npm (mitigation)
Mike, 2017-07-15 5:34 GMT+02:00 Mike Gerwitz : > On Fri, Jul 14, 2017 at 13:57:30 +0200, Jelle Licht wrote: > > Regardless, the biggest issue that remains is still that npm-land is > mired > > in cyclical dependencies and a fun-but-not-actually unique dependency > > resolving scheme. > > I still think the largest issue is trying to determine if a given > package and its entire [cyclic cluster] subgraph is Free. That's a lot > of manual verification to be had (to verify any automated > checks). npm's package.json does include a `license' field, but that is > metadata with no legal significance, and afaik _defaults_ to "MIT" > (implying Expat), even if there's actually no license information in the > repository. in my idea I would have build a database withh conditions for being non free forr every npm package. So we could have queried the database for questions like: is there any non free or non buildable package in the dependency tree of, say, the current Jquery ? So we could have focused on such problems before embarking in a demanding packaging process and then get struck by said problem along the way (withh the risk of loosing the work already done) You might remember my post of a few months back about an attempt of mine to crawl thhe npm registry and storing data found there. I used amz3's wrap around Wiredtiger and that was probably not the best choice as I run into some maturity problems (maturity both of the framewrok and my own maturity). And then I slacked a bit I also posted more recently about a research team that published a SPARQL endpoint containing data about the npm packages I thought it would be important but the feedback I collected was not exactly warm So I thought there must be some fundamental flaw in my way of thinking about a more data centric way of dealing with this Now I'm not sure what Jelle is talking about but any approach that cold be shared among at least 2 persons would be a progress, in my opinion. Jelle, please, say something more about whaht you're doing !
Re: npm (mitigation)
2017-07-14 19:11 GMT+02:00 Jan Nieuwenhuizen : > Catonano writes: > > > I read that Jelle and Jan used their own branch in order to have npm > > based software to be installed in their GuixSD environments, as binary > > blobs > > Jelle wrote a nice and clean npm importer, no binary anything. > > Npm packages have the can be source or binary. Depending on how you > want to look at it you can make change this source/binary disctinction > less white/black and turn it into a gray-scale. > > As we are talking about javascript, in some cases source and binary > packages are identical. In other cases, the binary packages come with > preprocessed documentation and lack the sources. Other binary packages > include minimized javascript and even further into the darkness some > binary packages do not include the non-minimized javascript. Then some > binary packages come with pre-compiled binaries and the worst are binary > package that do not come with the C/C++ sources that were used to > compile these binaries. > > In all these cases the binary packages can be built from their source > package. Here is where it starts to get nasty. Building a package from > source can only be done if you have all its dependencies already > installed. In theory that should not be a problem. > > It appears that the npm ecosystem has manouvered itself into a place > where bootstrapping seems impossible: it turns out that any serious > package (notably all npm build system packages) have over 1000 > dependencies, often with cyclic dependencies or even missing packages. > > To break this boostrap loop on Guix I have added a couple of patches > onto Jelle's npm importer branch that implement a --binary flag. This > allows binary npm packages to be installed in Guix and serve as a basis > to build other npm packages from source. > > Apparently have no problem*) adding binary blobs for gcc, haskell (...?) > to Guix. Similarly we could consider adding a/some binary blobs for npm > buildsystem packages to Guix and use those for a basis to build > source-only packages. > > > Can I ask you for instructions about how to do that exactly ? > >git clone https://gitlab.com/janneke/guix.git > > The branch `npm' is rebased on version-0.13.0, have a look at > >guix import npm --help > > and look in gnu/packages/npm.scm for instructions. > > Greetings, > janneke > > *) Actually I do not like this very much and that's why some of us seek >to remove the need for our bootstrap binaries with our stage0 and Mes >projects. > Thanks. Both for the wrap up and for the instructions.
Re: npm (mitigation)
2017-07-14 13:57 GMT+02:00 Jelle Licht : > Hi Catonano, > > I would be be happy to help you with this, but tbh, I am not comfortable > discussing this in-depth on guix-devel, as this seems antithetical to Guix' > goals. > Ok, at least we made this clear. I'll keep the list off the hook, should I need to discuss tis further, in the future. Just for the record, I don't mean to conter thhe Guix aim. I consider this as a temporary mitigation. Were "temporary" is the key word Regardless, the biggest issue that remains is still that npm-land is mired > in cyclical dependencies and a fun-but-not-actually unique dependency > resolving scheme. > I am currently working on a guile version of what Sander did for Nix for > importing entire npm dependency trees, but this will likely lead to lots of > programmatically > defined packages instead of the guix approach of mostly-manually defining > each package. > What ? Who is Sander ans what did the do for Nix ? How is importing "entire npm dependencies trees" different than importing one package at a time and building the dependency tree gradually ? I have no problem with programmatically defined pacages, anyway ;-)
Re: npm (mitigation)
Note that it's not required to include license notices (not to be confused with the license itself) in the header (or as metadata) of each source file. It's a recommendation, such that if someone gets only that specific file, then that person can easily know and have proof that such file is under that license. For this specific/current paragraph, I also want to point out that these are own ways of checking the license notices of works, and I'm not a lawyer. If there is no license notice in a given file, then the sibling or parent README files (or similar in non-English languages) dictate where to find the license notices, or provide the license notices there. Failing this, then one must check for the source file with the same name of the project or some name similar to "main" (assumed to be in English, but this name can also change according to the language). Files similar to COPYING, LICENSE, or [License name, or abbreviation indicating exceptions and if "or later"/"+" applies] files (or similar ones for each of those in non-English languages) are not that useful if there is no match for what was described in the last paragraph. Annoying fact: I just read the package.json npm specification syntax documentation and it says to follow the "SPDX" specification when specifying the licenses. However, the current "SPDX" specification no longer seems to accept licenses with "+" ("or later"), there is no way to tell that there is a proxy involved (such as what can be specified when one adapts a CC BY-SA 4.0 work into another one under GNU GPL 3+, according to [[https://www.gnu.org/licenses/license-list.html#ccbysa]]), there is no exception for website management systems and associated templates (such as what can be found in [[https://www.gnu.org/licenses/gpl-faq.html#WMS]]), and there is exception for source files written in JavaScript ([[https://www.gnu.org/software/librejs/free-your-javascript.html]], in section 3.2.1 of this reference).
Re: npm (mitigation)
Hm... If this is indeed true, then it's indeed a concern. I wonder if we can check that out... -- - [[https://libreplanet.org/wiki/User:Adfeno]] - Palestrante e consultor sobre /software/ livre (não confundir com gratis). - "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard que está no endereço acima aos teus contatos. - Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu aceito, mas não repasso. Entrego apenas em formatos favoráveis ao /software/ livre. Favor entrar em contato em caso de dúvida.
Re: npm (mitigation)
2017-07-15 5:34 GMT+02:00 Mike Gerwitz : > On Fri, Jul 14, 2017 at 13:57:30 +0200, Jelle Licht wrote: > > Regardless, the biggest issue that remains is still that npm-land is > mired > > in cyclical dependencies and a fun-but-not-actually unique dependency > > resolving scheme. > > I still think the largest issue is trying to determine if a given > package and its entire [cyclic cluster] subgraph is Free. That's a lot > of manual verification to be had (to verify any automated > checks). npm's package.json does include a `license' field, but that is > metadata with no legal significance, and afaik _defaults_ to "MIT" > (implying Expat), even if there's actually no license information in the > repository. And that is exactly why this probably won't end up in Guix proper, at least for the foreseeable future. And also the reason that the entire npm situation is so sad. The default MIT/Expat only applies to people who generate their package metadata via npm init by just pressing enter; IANAL, but directly referring to a valid and common SPDX identifier is not that different from including some file under the name of LICENSE/COPYING. It is true that lots of npm projects do not include copyright and/or license headers in each source file, but this is also true for lots of other free software. - Jelle
Re: npm (mitigation)
On Fri, Jul 14, 2017 at 13:57:30 +0200, Jelle Licht wrote: > Regardless, the biggest issue that remains is still that npm-land is mired > in cyclical dependencies and a fun-but-not-actually unique dependency > resolving scheme. I still think the largest issue is trying to determine if a given package and its entire [cyclic cluster] subgraph is Free. That's a lot of manual verification to be had (to verify any automated checks). npm's package.json does include a `license' field, but that is metadata with no legal significance, and afaik _defaults_ to "MIT" (implying Expat), even if there's actually no license information in the repository. -- Mike Gerwitz Free Software Hacker+Activist | GNU Maintainer & Volunteer GPG: D6E9 B930 028A 6C38 F43B 2388 FEF6 3574 5E6F 6D05 https://mikegerwitz.com signature.asc Description: PGP signature
Re: npm (mitigation)
Catonano writes: > I read that Jelle and Jan used their own branch in order to have npm > based software to be installed in their GuixSD environments, as binary > blobs Jelle wrote a nice and clean npm importer, no binary anything. Npm packages have the can be source or binary. Depending on how you want to look at it you can make change this source/binary disctinction less white/black and turn it into a gray-scale. As we are talking about javascript, in some cases source and binary packages are identical. In other cases, the binary packages come with preprocessed documentation and lack the sources. Other binary packages include minimized javascript and even further into the darkness some binary packages do not include the non-minimized javascript. Then some binary packages come with pre-compiled binaries and the worst are binary package that do not come with the C/C++ sources that were used to compile these binaries. In all these cases the binary packages can be built from their source package. Here is where it starts to get nasty. Building a package from source can only be done if you have all its dependencies already installed. In theory that should not be a problem. It appears that the npm ecosystem has manouvered itself into a place where bootstrapping seems impossible: it turns out that any serious package (notably all npm build system packages) have over 1000 dependencies, often with cyclic dependencies or even missing packages. To break this boostrap loop on Guix I have added a couple of patches onto Jelle's npm importer branch that implement a --binary flag. This allows binary npm packages to be installed in Guix and serve as a basis to build other npm packages from source. Apparently have no problem*) adding binary blobs for gcc, haskell (...?) to Guix. Similarly we could consider adding a/some binary blobs for npm buildsystem packages to Guix and use those for a basis to build source-only packages. > Can I ask you for instructions about how to do that exactly ? git clone https://gitlab.com/janneke/guix.git The branch `npm' is rebased on version-0.13.0, have a look at guix import npm --help and look in gnu/packages/npm.scm for instructions. Greetings, janneke *) Actually I do not like this very much and that's why some of us seek to remove the need for our bootstrap binaries with our stage0 and Mes projects. -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
Re: npm (mitigation)
Hi Catonano, I would be be happy to help you with this, but tbh, I am not comfortable discussing this in-depth on guix-devel, as this seems antithetical to Guix' goals. All I will say here is that you need to adapt the npm importer to use the sources from the npm registry instead of resolving to any 'upstream' urls. I believe Jan's importer was already able to do this last time I checked, so you might really only need to checkout their branch and rebase on current master. Regardless, the biggest issue that remains is still that npm-land is mired in cyclical dependencies and a fun-but-not-actually unique dependency resolving scheme. I am currently working on a guile version of what Sander did for Nix for importing entire npm dependency trees, but this will likely lead to lots of programmatically defined packages instead of the guix approach of mostly-manually defining each package. It might therefore be a good candidate for a guix channel, if that is still being worked on. Good luck! - Jelle 2017-07-14 10:52 GMT+02:00 Catonano : > I read that Jelle and Jan used their own branch in order to have npm based > software to be installed in their GuixSD environments, as binary blobs > > Can I ask you for instructions about how to do that exactly ? > > I might need to work a litle on some web siites in the future > > Which branch exactly did you use, and how, exactly ? > > Thanks >