Re: Security concerns with minified javascript code

[Simon Josefsson]

Samuel Thibault  writes:

> I however agree that it seems poor practice to duplicate these build
> modules in every packages. But if the required versions are different,
> there is no real other way.

There is another solution: put several different versions of the same
source code into some Debian meta-package.  Not pleasant, but as long as
different projects have strict version dependencies on the same
libraries, this would work.

That said, I think it is simpler to include the entire dependency chain
into each application package.  It leads to source code duplication
which is a security team concern, but at least that is an understandable
problem.

/Simon


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Neil Williams]

On Fri, 04 Sep 2015 17:54:30 +0200
Vincent Bernat  wrote:

>  ❦  4 septembre 2015 09:32 +0100, Neil Williams
>  :
> 
> [Applying patch to pre-minification source]
> >> Getting the same min.js is not a goal (we don't try to get the same
> >> binaries than upstream from source code in any other language, for
> >> example in Go where upstreams like to provide binaries). Not adding
> >> bugs would be nice, yes, but the situation is improving upstream
> >> (see above). -- 
> >
> > OK. In that case, it might be worth embedding upstreams testing with
> > using uglifyjs on the unminified JS files included upstream and
> > seeing if there are (functional) bugs introduced by using that
> > minifier.
> 
> That would be quite difficult if we are still in the context of
> upstreams embedding their own copy of jQuery. First, some upstreams
> don't have any JS-based tests (notably if JS is only an
> accessory). Second, when they have them, they are unlikely to cover
> jQuery use. Third, I really don't know if they are easy to run.
> Looking at Wordpress, I see they need Grunt, but maybe having only
> qunit would be fine.
> 
> I would be inclined to say "let's see if we get bug reports due to the
> minification". I didn't run into those myself. Pau did, but it was
> quite some time ago.

I was thinking primarily about my own upstream at that point, i.e.
somewhere where there are staging or test services available with real
data and users etc. but which are not used for important stuff. I want
my own upstream to be confident that the javascript in the release is
both tested before release and fixable after release. We don't have JS
specific tests but we do have test boxes with a representative dataset
to test all relevant use cases. That's the thing with embedding
upstreams, the software has to do so many other things, the JS is a
helper.

> > What are the problems likely to be if Debian was to
> > recommend/require that:
> > "minified javascript is not to be regarded as source code suitable
> > for supporting security fixes and needs to be rebuilt from
> > unminified source code or replaced with symlinks to minified
> > javascript provided by other packages, with dependencies added on
> > those packages. When packages need to minify embedded Javascript
> > which is not already packaged in Debian, only  minifier should
> > be used and any minified javascript files included by upstream must
> > be replaced using the output of  minifier."
> 
> I am not an expert enough to say that uglifyjs is really the minifier
> to go. I would say that this is the most popular one, but that's
> really just a hunch. Maybe some input from the Debian Javascript
> Maintainers would help.
> 
> But I am OK with the idea.

I'm no javascript expert either, far from it. I can see how these ideas
can fix the distribution problems but whether the changes would
introduce new bugs which the embedding upstream may be unable to
reproduce without adopting the same process themselves, only time will
tell.

> > This excludes the issues of packaging of libjs* packages themselves
> > (where the same minifier as upstream is likely to be a significant
> > advantage in updating the package) but does provide a solution for
> > the actual security concerns of packaged minified Javascript. A
> > handy tool to do this would be useful as that would allow
> > interested upstreams to test the effects of such a minifier before
> > the upload to Debian.
> >
> > dh_uglify?
> 
> If there is some consensus that pre-minification source code is
> accepted as valid source code by DFSG, I could help to get that.

The manpage for such a thing should be clear that it is meant for
packages which need to embed JS which is not available in Debian, not
for packages which are primarily or solely javascript. Maintainers of
javascript packages could be recommended to use a minifier and build
system which is compatible with the upstream for that package so that
upstream fixes and Debian patches can be integrated smoothly.

It just needs to a) remove the .min.js from the debian/*/ directories
specified b) ensure the specified .js file exists c) uglify it and
write that in place of the .min.js and d) fail if any
unrecognised .min.js are left at the end.

I'm happy with a statement in Policy that *binary* packages must not
include minified javascript which has not been handled by dh_uglify,
once it exists. If there is a suitable lead time, then new source
package uploads could also be prohibited from containing minified
javascript.

I'm still unsure that pre-minification source code should be accepted
in all cases, but I'm happy if pre-minification *with a recognised
minify service* is the accepted way to handle the security concerns
around embedded javascript. To be accepted for everything,
pre-minification would need to be "easily" convertible back to the
format used for modification by the maintainers and/or upstream of the
javascript package in 

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  4 septembre 2015 09:32 +0100, Neil Williams  :

[Applying patch to pre-minification source]
>> This will become more difficult with jQuery 3.x (due to ES6 to ES5
>> transformation). But manual adaptation should stay straightfoward
>> ("transpilation" only does local transformation).
>
> Are there supported tools which could prevent that difficulty becoming
> a problem?

None that I know of. Note that currently, the problem is quite
theoric. I don't know when jQuery 3.x will be released. I don't think
this will happen soon (but I am quite exterior to this community and not
a real jQuery user myself).

>> That's why I consider this pre-minification version as valid source
>> code.
>
> That indeed would support a usable security fix - thanks.

Great!

>> >> I agree. And doing from pre-minification source to minified source
>> >> is possible with the tools in main (uglifyjs or yui-compressor).
>> >
>> > Not in a way that reproduces the same .min.js as the original
>> > source - unless that is from a package already in Debian and the
>> > same minifier is used (with the same options).
>> 
>> Getting the same min.js is not a goal (we don't try to get the same
>> binaries than upstream from source code in any other language, for
>> example in Go where upstreams like to provide binaries). Not adding
>> bugs would be nice, yes, but the situation is improving upstream (see
>> above). -- 
>
> OK. In that case, it might be worth embedding upstreams testing with
> using uglifyjs on the unminified JS files included upstream and seeing
> if there are (functional) bugs introduced by using that minifier.

That would be quite difficult if we are still in the context of
upstreams embedding their own copy of jQuery. First, some upstreams
don't have any JS-based tests (notably if JS is only an
accessory). Second, when they have them, they are unlikely to cover
jQuery use. Third, I really don't know if they are easy to run. Looking
at Wordpress, I see they need Grunt, but maybe having only qunit would
be fine.

I would be inclined to say "let's see if we get bug reports due to the
minification". I didn't run into those myself. Pau did, but it was quite
some time ago.

> What are the problems likely to be if Debian was to recommend/require
> that:
> "minified javascript is not to be regarded as source code suitable for
> supporting security fixes and needs to be rebuilt from unminified source
> code or replaced with symlinks to minified javascript provided by other
> packages, with dependencies added on those packages. When packages need
> to minify embedded Javascript which is not already packaged in Debian,
> only  minifier should be used and any minified javascript files
> included by upstream must be replaced using the output of 
> minifier."

I am not an expert enough to say that uglifyjs is really the minifier to
go. I would say that this is the most popular one, but that's really
just a hunch. Maybe some input from the Debian Javascript Maintainers
would help.

But I am OK with the idea.

> This excludes the issues of packaging of libjs* packages themselves
> (where the same minifier as upstream is likely to be a significant
> advantage in updating the package) but does provide a solution for the
> actual security concerns of packaged minified Javascript. A handy tool
> to do this would be useful as that would allow interested upstreams to
> test the effects of such a minifier before the upload to Debian.
>
> dh_uglify?

If there is some consensus that pre-minification source code is accepted
as valid source code by DFSG, I could help to get that.
-- 
Writing is easy; all you do is sit staring at the blank sheet of paper until
drops of blood form on your forehead.
-- Gene Fowler


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Neil Williams]

On Thu, 03 Sep 2015 22:51:50 +0200
Vincent Bernat  wrote:

>  ❦  3 septembre 2015 13:19 -0700, Nikolaus Rath  :
> 
> > 2. Many javascript packages cannot be build from source with the
> > tools in main.
> 
> This is the one I don't agree. For me, pre-minification source is
> source code. If you show the pre-minification version of jQuery, many
> people will believe this is a valid source code (original comments,
> variable names and indentation are here).

... except that what you are referring to there is the "source" as it
appears as an unminified JS file on the filesystem after the upstream
code is packaged by Debian. The JQuery upstream is a set of directives
to include other snippet files which are then further processed into
both the unminified JS that looks like source code and the minified JS
which tends to get used by applications. The "source code" that
everyone talks about is actually not edited by anyone, least of all
upstream, it not used by applications (which embed the .min.js) and it
is not convertible to the source for modification used by JQuery
upstream, so patches to that are of no use to fixing security bugs in
JQuery as packaged in Debian.

JQuery in Debian *does* build from upstream source using only tools
available in Debian main. What it builds from is not usable to
applications, but if a bug is reported in the minified or unminified
JS, the maintainer (together with the JQuery upstream) can prepare a
patch to the source code that is built in Debian to produce a change in
the minified and unminified *generated files* for other applications to
use.

Patching an unminified JS file as it exists on the filesystem from
packaging doesn't actually help anyone, except possibly in debugging the
security hole. Even then, one of the more serious problems with
minification is that it isn't just a process of removing comments and
whitespace, the minifier can inject bugs which are not present in the
unminified version. Using a different minifier can introduce *different*
*unknown* bugs that only affect the version patched to "improve
security".

We really need to think of minifiers as transformational - these are
compilers in many ways.

For those upstreams who have to embed JS not available in Debian, then
the upstream code often cannot be processed in Debian, neither can the
unminified JS be minified into the same .min.js file as embedded due to
a lack of tools in Debian. In that sense, the current lintian
requirements are a hindrance and a mess because they require the
embedding upstream to include "source code" which cannot be converted
to the actual running code using tools in main in a reproducible
manner. Using a different minifier results in an untested and
unverifiable .min.js with no assurance that new bugs have not been
introduced.

This thread isn't principally about how JS packages already in main
actually get built and patched, this is about how upstreams who need to
embed JS not available in Debian can support security fixes to
javascript. Patches which produce a third form of the code which
produces a diff to the buggy version that is massively larger than the
security fix itself will not find favour with the release team or
security team during a release freeze.

The maintainers and upstreams of such packages *need* Debian to come up
with an answer for how their code needs to handle security fixes within
Debian. A way that produces a minimal diff to the released version to
make it practical to review the change to a package already in stable.
A way that absolutely does NOT introduce new bugs. A way that is
exclusively possible using tools available in Debian main. Can we
*please* talk about that now? Please don't leave this until the release
freeze for Stretch is on the horizon. This thread has gone on for ages
with bike-shedding, tippy-toeing around DFSG sidelines and insane ideas
about non-free but with no useful output for those who are actually
trying to fix the *SECURITY PROBLEM*.

There is a fundamental misconception in a lot of these discussions
which drops embedding upstreams right in the toilet. There is a real
problem here and it has nothing to do with whether packages go into
contrib or non-free. It is whether packages embedding javascript can
have sane security support in any part of Debian. Moving stuff to
non-free is *not* a security fix!

The core problem is *not* about legality or "preferred source for
modification" or the DFSG or a move to contrib/non-free - those are
(large) complicating factors. The core problem is how to fix
undiscovered security holes in packages already in Debian stable which
need to embed JS not available in Debian. This is a technical and
packaging problem, not a licence or legal one. We're taking about the
*security* of free software, not whether it is free or not.

> > 3. Software that cannot be build from source with the tools in main
> >must not go in main but into contrib
> 
> I agree. And 

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  4 septembre 2015 08:29 +0100, Neil Williams  :

>> > 2. Many javascript packages cannot be build from source with the
>> > tools in main.
>> 
>> This is the one I don't agree. For me, pre-minification source is
>> source code. If you show the pre-minification version of jQuery, many
>> people will believe this is a valid source code (original comments,
>> variable names and indentation are here).
>
> ... except that what you are referring to there is the "source" as it
> appears as an unminified JS file on the filesystem after the upstream
> code is packaged by Debian. The JQuery upstream is a set of directives
> to include other snippet files which are then further processed into
> both the unminified JS that looks like source code and the minified JS
> which tends to get used by applications. The "source code" that
> everyone talks about is actually not edited by anyone, least of all
> upstream, it not used by applications (which embed the .min.js) and it
> is not convertible to the source for modification used by JQuery
> upstream, so patches to that are of no use to fixing security bugs in
> JQuery as packaged in Debian.

The pre-minification source code is so close of the original code that
you will have no trouble to apply any security fix to the
pre-minification source code. Let me take a random source file in the
original source:

 https://github.com/jquery/jquery/blob/2.1-stable/src/queue/delay.js

And the pre-minification source code:

 
https://github.com/jquery/jquery-ui/blob/master/external/jquery-2.1.3/jquery.js#L6894-L6906

> JQuery in Debian *does* build from upstream source using only tools
> available in Debian main. What it builds from is not usable to
> applications, but if a bug is reported in the minified or unminified
> JS, the maintainer (together with the JQuery upstream) can prepare a
> patch to the source code that is built in Debian to produce a change in
> the minified and unminified *generated files* for other applications to
> use.

With the only problem that jQuery as shipped by Debian is currently 1.x
while some applications now require 2.x. And jQuery was just an example.

> Patching an unminified JS file as it exists on the filesystem from
> packaging doesn't actually help anyone, except possibly in debugging the
> security hole. Even then, one of the more serious problems with
> minification is that it isn't just a process of removing comments and
> whitespace, the minifier can inject bugs which are not present in the
> unminified version. Using a different minifier can introduce *different*
> *unknown* bugs that only affect the version patched to "improve
> security".

Exactly like a C compiler which can introduce bugs on its own. And
different C compilers have different bugs. Again:
https://www.alchemistowl.org/pocorgtfo/pocorgtfo08.pdf, 8:3.

> We really need to think of minifiers as transformational - these are
> compilers in many ways.

OK with that.

> For those upstreams who have to embed JS not available in Debian, then
> the upstream code often cannot be processed in Debian, neither can the
> unminified JS be minified into the same .min.js file as embedded due
> to a lack of tools in Debian.  In that sense, the current lintian
> requirements are a hindrance and a mess because they require the
> embedding upstream to include "source code" which cannot be converted
> to the actual running code using tools in main in a reproducible
> manner. Using a different minifier results in an untested and
> unverifiable .min.js with no assurance that new bugs have not been
> introduced.

The JS ecosystem is mature enough to know the problems that may arise
with minifiers and work around them. While using yui-compressor is a bit
risky due to its low use, using uglifyjs should be fine. The
pre-minified JS files usually are prepared to be minified (for example,
when using dependency injections, variables are usually converted to
strings to keep their original names during minification, example
available on request).

> This thread isn't principally about how JS packages already in main
> actually get built and patched, this is about how upstreams who need to
> embed JS not available in Debian can support security fixes to
> javascript. Patches which produce a third form of the code which
> produces a diff to the buggy version that is massively larger than the
> security fix itself will not find favour with the release team or
> security team during a release freeze.

Hopefully, this is not the case. An upstream patch can be transformed
into a patch which can be applied to a pre-minification version
"easily". And the result should be of the same size.

There is not a lot of CVE available to use as examples (I think mostly
because bugs get rarely classified as security bugs in this
ecosystem). After a quick search, the most recent one I am able to find
is CVE-2011-4969. The patch is here:

 

Re: Security concerns with minified javascript code

[Neil Williams]

On Fri, 04 Sep 2015 10:07:13 +0200
Vincent Bernat  wrote:

>  ❦  4 septembre 2015 08:29 +0100, Neil Williams
>  :
> 
> > For those upstreams who have to embed JS not available in Debian,
> > then the upstream code often cannot be processed in Debian, neither
> > can the unminified JS be minified into the same .min.js file as
> > embedded due to a lack of tools in Debian.  In that sense, the
> > current lintian requirements are a hindrance and a mess because
> > they require the embedding upstream to include "source code" which
> > cannot be converted to the actual running code using tools in main
> > in a reproducible manner. Using a different minifier results in an
> > untested and unverifiable .min.js with no assurance that new bugs
> > have not been introduced.
> 
> The JS ecosystem is mature enough to know the problems that may arise
> with minifiers and work around them. While using yui-compressor is a
> bit risky due to its low use, using uglifyjs should be fine. The
> pre-minified JS files usually are prepared to be minified (for
> example, when using dependency injections, variables are usually
> converted to strings to keep their original names during
> minification, example available on request).

Actually, that is a great relief. Now this thread is getting somewhere
useful.
:-)
 
> > This thread isn't principally about how JS packages already in main
> > actually get built and patched, this is about how upstreams who
> > need to embed JS not available in Debian can support security fixes
> > to javascript. Patches which produce a third form of the code which
> > produces a diff to the buggy version that is massively larger than
> > the security fix itself will not find favour with the release team
> > or security team during a release freeze.
> 
> Hopefully, this is not the case. An upstream patch can be transformed
> into a patch which can be applied to a pre-minification version
> "easily". And the result should be of the same size.
> 
> There is not a lot of CVE available to use as examples (I think mostly
> because bugs get rarely classified as security bugs in this
> ecosystem). After a quick search, the most recent one I am able to
> find is CVE-2011-4969. The patch is here:
> 
>  
> https://github.com/jquery/jquery/commit/db9e023e62c1ff5d8f21ed9868ab6878da2005e9
> 
> #+begin_src diff
> diff --git a/src/core.js b/src/core.js
> index 694f884..0b99b74 100644
> --- a/src/core.js
> +++ b/src/core.js
> @@ -16,8 +16,8 @@ var jQuery = function( selector, context ) {
>   rootjQuery,
>  
>   // A simple way to check for HTML strings or ID strings
> - // (both of which we optimize for)
> - quickExpr = /^(?:[^<]*(<[\w\W]+>)[^>]*$|#([\w\-]*)$)/,
> + // Prioritize #id over  to avoid XSS via location.hash
> (#9521)
> + quickExpr = /^(?:[^#<]*(<[\w\W]+>)[^>]*$|#([\w\-]*)$)/,
>  
>   // Check if a string has a non-whitespace character in it
>   rnotwhite = /\S/,
> #+end_src
> 
> Let me download a pre-minification version of jQuery 1.6.2:
> 
>  http://code.jquery.com/jquery-1.6.2.js
> 
> Patch applies directly:
> 
> #v+
> $ patch --dry-run jquery-1.6.2.js < cve-2011-4969.patch
> checking file jquery-1.6.2.js
> Hunk #1 succeeded at 37 (offset 21 lines).
> #v-
> 
> This will become more difficult with jQuery 3.x (due to ES6 to ES5
> transformation). But manual adaptation should stay straightfoward
> ("transpilation" only does local transformation).

Are there supported tools which could prevent that difficulty becoming
a problem?

> That's why I consider this pre-minification version as valid source
> code.

That indeed would support a usable security fix - thanks.

> >> > 3. Software that cannot be build from source with the tools in
> >> > main must not go in main but into contrib
> >> 
> >> I agree. And doing from pre-minification source to minified source
> >> is possible with the tools in main (uglifyjs or yui-compressor).
> >
> > Not in a way that reproduces the same .min.js as the original
> > source - unless that is from a package already in Debian and the
> > same minifier is used (with the same options).
> 
> Getting the same min.js is not a goal (we don't try to get the same
> binaries than upstream from source code in any other language, for
> example in Go where upstreams like to provide binaries). Not adding
> bugs would be nice, yes, but the situation is improving upstream (see
> above). -- 

OK. In that case, it might be worth embedding upstreams testing with
using uglifyjs on the unminified JS files included upstream and seeing
if there are (functional) bugs introduced by using that minifier. It
could also be worth asking lintian to change the meaning of the current
warnings so that merely accompanying the minified with the unminified
is not enough, that minification with something like uglifyjs should be
tested and that the maintainer should replace the upstream .min.js
with the output of that known minifier (in much 

Re: Security concerns with minified javascript code

[Bas Wijnen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, Sep 03, 2015 at 08:47:11AM +0200, Vincent Bernat wrote:
> Without minification, we'll just ship packages that people won't
> use. Why would I run a crippled installation of Wordpress that will
> drive of part of my users to another competitor?

Because you know you have the right and the ability to be a part of the free
software community that created the software.  That is why you are running
Debian and don't have contrib or non-free in your sources.list.

- From your mails it is clear that you don't care much about that.  That is 
fine.
I recommend that you do put contrib and non-free in your sources.list.  You'll
get all the non-crippled programs that we ship, but there's no guarantee that
you will be able to get the source, or compile it if you can get it.

To keep all our users happy, not just the ones who want contrib and non-free
enabled, please put things that people like me don't want to see in there, not
in main.  (Of course, fixing the problem is even better, but if you're not
willing to do that work, then at least don't put the software in main.)

> We don't turn C into an interpreted language because it would be easier
> to inspect the resulting binaries.

We do remove non-free content (or things that need non-main content) from
upstream sources and I'm sure some people consider that crippling as well.
There are two options: they are the maintainer and really like the stuff: they
can package it for contrib or non-free.  They are not: they can go use a
distribution that doesn't care as much about freedom as Debian does.

There is also a non-option: they break the rules we have set up for main, with
the excuse that otherwise our users would be unhappy.  That is what you
suggest.  We don't do that.  People who don't like our rules shouldn't be using
Debian.

Thanks,
Bas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBAgAGBQJV6IHpAAoJEJzRfVgHwHE6zHoP/38VNuzOKVDbCeZCpvOBItJh
EFCTkNBxQTBCmTrj7JArMYN1CKUVkkb+qKMLex2xxjfPIdydTeT+bOYHIBIWBNHm
a2ZNEN+0zg5cxte3D8m54YYFBfbaKuFPE9/jqkSWkWAIBjt9p84G663q5S8o6P4X
dqwLTvOjVgPc0BZ6j6Bo7s8BQiFW/UBIyMgsN32HlLhRyYOG7TErkOwdSmFqq6tw
GZEh9GMvRAw/3yTPVoZsJ9RyWKP5TpTRuGZe69CAaTmWDRtuq33O4JSzhmW+hdd7
T8jSAhrhpVe92AGo8qZodF/WLU61J4LbpLg42r9xhmKVu/OkrC7NdHhKK6ypwh6O
MNgxh4jf4MUMysCBuL2F2Fb1kcHb5ezvNq71IzeOIU0ry3EHDSnUlJQGbdwhiqrr
9xTEGeGsSgmUqKR2/twZx6+lk59/sto9WyEpZ21wdKk8zRU+cXaJZC4VREHYky0i
YDORHnt3yYBJnu+C5eYlgNZbXUObOrB2CedjVeLcBIGP9sHwAvOY2A/ZPs1SrseO
y8zc163CjXaZ7ETjcoTT6DGlbT1FUmHwsmOrt34ODERl+GZPX6GbrbiPdBklPsxS
/XTZk61I8SS2c25Bt1QOI+aG/70KKhGqTkY3qd/pT+w7EGEWeE6FqnKAzM/+X1hw
t6yOkF8zT3TQIIfe+918
=JSxd
-END PGP SIGNATURE-

Re: Security concerns with minified javascript code

[Marvin Renich]

* Bas Wijnen  [150902 17:36]:
> > On Wed, 2 Sep 2015 13:33:57 -0400 Marvin Renich  wrote:
> > > No, "A preferred form" is what upstream uses.  The DFSG does not use
> > > the term "THE preferred form", and I believe that was wise.
> 
> The DFSG doesn't define source at all.  There seems to be consensus (you're 
> the
> only one who doesn't seem to agree) that the definition from the GPL is a good
> one, and that does say "the".

Quoting from [1]:
  The process involves human judgement. The DFSG is an attempt to
  articulate our criteria. But the DFSG is not a contract.

People keep trying to use "the preferred form for modification" as a
rule.  This is wrong.  The rest of the paragraph quoted above should
also be read.

I do strongly believe that "preferred form for modification" is a good
test to apply, but it is not an absolute.  I also believe that sometimes
there is more than one form of source that can satisfy the DFSG.  A
simple example is the .xcf/.png/.ico example I gave in a previous
message.  This is why I disagree with using "THE" (implying only one)
instead of "A" (implying one of many).

> > > There can be multiple "preferred forms" for some software, and all are, in
> > > my opinion, acceptable by the DFSG.  The real question is whether it is
> > > reasonable to expect someone who wishes to modify the software to consider
> > > the form "source".
> 
> I disagree partly.  It is possible to copy a generated file and use that as
> source.  IMO that isn't the case until there have actually been made
> modifications to that file, though.  If an upstream (which doesn't need to be
> the original upstream) actually uses a file to make modifications, an argument
> can be made that this format is source.
> 
> At the same time, we should try to convince upstreams that do such a thing to
> stop it; it causes code duplication and a (security) support nightmare.

I'm not sure how that is relevant to what I said.

> "Someone might think they can make modifications to this file" is much too
> broad; for some modifications a hex editor is good enough.  And in some cases
> that is totally reasonable, such as an executable for which you don't have
> source.  That doesn't make binary exectutables source.

That is not at all what I said.  To paraphrase, using a circular
definition, if I can _reasonably_ _expect_ most other people to agree
that it is source, then that is a very good indication that it is
source.

...Marvin

[1] https://people.debian.org/~bap/dfsg-faq.html#testing

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  3 septembre 2015 17:22 GMT, Bas Wijnen  :

> Because you know you have the right and the ability to be a part of the free
> software community that created the software.  That is why you are running
> Debian and don't have contrib or non-free in your sources.list.
>
> From your mails it is clear that you don't care much about that.

Repeating that in each of your email is quite hostile. Please stop
saying that I don't care. I care. I just don't agree with you.
-- 
The first thing we do, let's kill all the lawyers.
-- Wm. Shakespeare, "Henry VI", Part IV


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Marvin Renich]

* Neil Williams  [150902 14:15]:
> On Wed, 2 Sep 2015 13:14:31 -0400 Marvin Renich  wrote:
> > It is presumed that upstream already has what it considers "source";
> > in the case of this thread, that is minified JS.
> 
> Actually, not. Source, for upstream of JQuery at least, is a set of
> directives to include files into a non-minified jquery.js which is then
> minified as part of the build in Debian.

I stand corrected.

> There are three players:
[large snip]
> It is a complete mess - with the embedding upstreams lost in the middle
> with no idea of what they can do to avoid getting in the way.

I completely agree.

...Marvin

Re: Security concerns with minified javascript code

[Gunnar Wolf]

Lars Wirzenius dijo [Wed, Sep 02, 2015 at 09:32:12AM +0300]:
> However, I want to raise the point that upstreams do not always make
> sensible decisions, and if they don't, it's good to raise that with
> them. For example, there was recently an ITP bug for
> node-number-is-nan. Upstream source code is at
> https://github.com/sindresorhus/number-is-nan, and the whole package
> boils down to these four lines of code:
> 
> 'use strict';
> module.exports = Number.isNaN || function (x) {
> return x !== x;
> };
> 
> (https://github.com/sindresorhus/number-is-nan/blob/master/index.js)
> 
> If something or someone needs this, we should package it, and it seems
> Grunt needs it. However, it doesn't seem sensible to have a package
> for every one-liner Javascript function, either in Debian or upstream.
> (...)

I agree, having all the boilerplate and infrastructure that make up a
package makes very little sense in this case. We have several packages
that are made up of bits from different origin, but linked with a
common target. The first example that comes to my mind is devscripts
(which, yes, has important peculiarities, such as being mostly written
by Debian-people as upstreams), but others can be mentioned
(i.e. emacs-goodies, texlive-generic-extra, etc.)

Such a package could be created for all such Javascript snippets which
present a ES6→ES5 "transpiler" (I still don't believe in such a term),
i.e. things such as what's mentioned at:

   https://github.com/addyosmani/es6-equivalents-in-es5

   
http://www.agile-environment.com/blog/fox-on-software/2015-04-06-es6-coffeescript-part-1

That is, one package that would allow using tricks such as Perl
(>=5.10)'s "use feature" and "no feture" tricks that allow for Perl6
idioms.

Re: Security concerns with minified javascript code

[Marvin Renich]

* Neil Williams  [150902 14:33]:
> Minified isn't source for modification... [large snip]

I don't believe I have disagreed with anything you said in the snipped
text.  I certainly did not mean to.  I said that minified JS can only go
in main if both the source and the tools to build it are also in main.
I also said that this is a hard problem to tackle, but Debian should
tackle it (which is what this thread appears to be doing) instead of
ignoring it.

> On Wed, 2 Sep 2015 13:33:57 -0400 Marvin Renich  wrote:
> > This whole thread is about...
> 
> It's not about contrib or main, that is roughly equivalent to thinking
> that every DFSG problem looks like a nail because all you have available
> is a large hammer instead of solving the problem inside main.

First, I will retract the phrase "this whole thread"; I should have said
the points in the thread to which I was responding.

Second, I was not proposing any solution to the problem or agreeing with
anyone's solution.  All of my posts were about exposing incorrect
interpretations of the term "source" in the DFSG.  I think, though I'm
not quite sure, that I replied to posters on both sides of the debate (a
large part of this thread does appear to debate what is required for
minified JS to go in main).

Perhaps my posts were misinterpreted as endorsing or opposing a specific
solution?

...Marvin

Re: Security concerns with minified javascript code

[Russ Allbery]

Paul Wise  writes:
> On Wed, Sep 2, 2015 at 11:47 PM, Russ Allbery wrote:

>> If *no one* has access to anything better than a binary file, then
>> possession of that binary file puts you on an equal footing with
>> everyone else in the world, which I think is all that we can reasonably
>> ask.

> We can of course strongly suggest upstreams not throw away their
> source files and not modify generated files, instead preserving the
> most expressive or information rich formats.

Indeed, and for images this is frequently a problem.  People construct
them using various rich data and then flatten them down to JPEGs or PNGs
and throw away all the other information.

Of course, even if they keep it, it's often Photoshop files, which is only
of marginal utility  :/

-- 
Russ Allbery (r...@debian.org)   

Re: Security concerns with minified javascript code

[Gunnar Wolf]

Vincent Bernat dijo [Wed, Sep 02, 2015 at 09:47:23AM +0200]:
> If you talk about uglifyjs or the like, it is already packaged and
> doesn't solve all the problems we have (see my message to Odyx,
> ).
> 
> If you talk about Grunt, Grunt comes with a lot of plugins (and does
> almost nothing without those) and each upstream will require different
> plugins with different versions (Grunt plugin versions are evolving
> fast). See the tree I posted for jQuery 3.x in
> . All this dependency chain is maintained
> by a variety of upstreams with different release schedules and goals.

This sounds quite similar to the situation we had with Rails (might
still have it, but I cannot say for sure, as I'm not much involved
with it anymore). Rails packages a set of Ruby libraries, each of
which has its schedule and versions.

Rails' developers "curate" such libraries, write some glue between
them (sometimes even take over their whole development), and come up
with "versions". Those versions have a stable set of libraries
presented together.

Of course, that does not (completely) solve the mess we have to deal
with when packaging Ruby, as each developer wants her code to work
with wildly differing versions of the involved "gems", and... and...

Sigh :-) You know what I mean.

But anyway — Grunt can be seen as a whole. If you just see it as a
collection of plugins, packaging them becomes just a pointless
PITA. We just cannot have different versions of hundreds of projects
in Debian and expect to maintain a decent code quality. Bad Things
(i.e. software vulnerabilities) can and will happen, and as Neil
Williams mentioned earlier on this thread, keeping track of all those
embedded code copies becomes an exponentially hard task.

Re: Security concerns with minified javascript code

[Nikolaus Rath]

On Sep 03 2015, Vincent Bernat  wrote:
>  ❦  3 septembre 2015 17:22 GMT, Bas Wijnen  :
>
>> Because you know you have the right and the ability to be a part of the free
>> software community that created the software.  That is why you are running
>> Debian and don't have contrib or non-free in your sources.list.
>>
>> From your mails it is clear that you don't care much about that.
>
> Repeating that in each of your email is quite hostile. Please stop
> saying that I don't care. I care. I just don't agree with you.


As a (mostly) passive observer of this thread, I have to agree with Bas
that what you're saying (and what not saying) in your other emails does
seem to imply that you don't care.

(And why is it hostile to say that someone doesn't care about something?
I don't care about a lot of things, and I wouldn't consider it hostile
for people to point that out).

Can you maybe clarify which of the following statements you don't agree with?

1. Minified javascript isn't source
2. Many javascript packages cannot be build from source with the tools
   in main.
3. Software that cannot be build from source with the tools in main
   must not go in main but into contrib


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  3 septembre 2015 13:19 -0700, Nikolaus Rath  :

>>> Because you know you have the right and the ability to be a part of the free
>>> software community that created the software.  That is why you are running
>>> Debian and don't have contrib or non-free in your sources.list.
>>>
>>> From your mails it is clear that you don't care much about that.
>>
>> Repeating that in each of your email is quite hostile. Please stop
>> saying that I don't care. I care. I just don't agree with you.
>
>
> As a (mostly) passive observer of this thread, I have to agree with Bas
> that what you're saying (and what not saying) in your other emails does
> seem to imply that you don't care.
>
> (And why is it hostile to say that someone doesn't care about something?
> I don't care about a lot of things, and I wouldn't consider it hostile
> for people to point that out).

It's hostile because it depicts as someone that shouldn't be a DD
(because we abided [abode?] to the social contract).

> Can you maybe clarify which of the following statements you don't agree with?
>
> 1. Minified javascript isn't source

I agree, minified JS isn't source.

> 2. Many javascript packages cannot be build from source with the tools
>in main.

This is the one I don't agree. For me, pre-minification source is source
code. If you show the pre-minification version of jQuery, many people
will believe this is a valid source code (original comments, variable
names and indentation are here).

> 3. Software that cannot be build from source with the tools in main
>must not go in main but into contrib

I agree. And doing from pre-minification source to minified source is
possible with the tools in main (uglifyjs or yui-compressor).
-- 
Avoid multiple exits from loops.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Paul Wise]

On Wed, Sep 2, 2015 at 11:47 PM, Russ Allbery wrote:

> I think reading "preferred form of modification" from the perspective of
> upstream is a useful standard because it handles some edge cases like
> that, and because it feels ethically consistent with free software
> principles.  The goal is that everyone with a copy of the software should
> be on equal footing.  The person distributing the software should have no
> special access to sources that those receiving the software don't get.

Thanks for pointing out that the ideas behind Free Software, "source"
and "preferred form for modification" basically boil down to equality
of access.

> If *no one* has access to anything better than a binary file, then
> possession of that binary file puts you on an equal footing with everyone
> else in the world, which I think is all that we can reasonably ask.

We can of course strongly suggest upstreams not throw away their
source files and not modify generated files, instead preserving the
most expressive or information rich formats.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  3 septembre 2015 12:23 +1000, Dmitry Smirnov  :

>> Amazon did a study that showed every ~100ms of page load
>> delay lost them 1% in sales.
>
> It could be that small percentage of Amazon users are impulsive trigger-happy 
> buyers. :)
> However that conclusion is probably wrong due to number of reasons:
[...]

Please, publish your own study. This number is well known and supported
by an entity which is likely to have a population large enough to be
significant.

>> So yes, minifiers matter.
>
> IMHO there is more harm than good. The only case for minification that I can 
> think of is to increase web server capacity a little to cope with flow of new 
> users following some sort of AD campaign. A poor substitute for capacity 
> planning or a case when network link is congested.
>
> Minification makes multiple assumptions such as that web app is perfect and 
> nobody would ever need to open JS console and report errors. Or that nobody 
> would like to learn about web site features from non-minified CSS and JS. Let 
> alone debugging some of us do not like proprietary javascripts running in our 
> browsers -- minification kills opportunity for security peer review etc.
>
> Finally, one may think that maintenance cost of minified JavaScripts in 
> Debian outweighs all the "benefits" by huge margin.

Without minification, we'll just ship packages that people won't
use. Why would I run a crippled installation of Wordpress that will
drive of part of my users to another competitor?

We don't turn C into an interpreted language because it would be easier
to inspect the resulting binaries.
-- 
Don't stop with your first draft.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Dmitry Smirnov]

On Thursday 03 September 2015 08:47:11 Vincent Bernat wrote:
> Please, publish your own study.

I do not need to publish any studies to be sceptical.


> This number is well known and supported
> by an entity which is likely to have a population large enough to be
> significant.

You've mentioned no links to any studies and I found conclusion the way 
you've expressed it to make little sense...


> Without minification, we'll just ship packages that people won't
> use. Why would I run a crippled installation of Wordpress that will
> drive of part of my users to another competitor?

Sorry but that feels like exaggeration. Maybe it is just my perception but  
but it looks like you are trying to justify a bad practice (minification) 
with remotely related arguments of little weight...

Why not use some pluggable minification plug-ins for Wordpress to control 
minification on run-time (not on build time) and being able to opt it out?

-- 
Regards,
 Dmitry Smirnov.



signature.asc
Description: This is a digitally signed message part.

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  3 septembre 2015 21:03 +1000, Dmitry Smirnov  :

>> Without minification, we'll just ship packages that people won't
>> use. Why would I run a crippled installation of Wordpress that will
>> drive of part of my users to another competitor?
>
> Sorry but that feels like exaggeration. Maybe it is just my perception but  
> but it looks like you are trying to justify a bad practice (minification) 
> with remotely related arguments of little weight...

I don't see in which world minification is considered a bad
practice. This is a good practice. All JS are minified to half their
sizes (compared to just gzip) and speed up their evaluation.

> Why not use some pluggable minification plug-ins for Wordpress to control 
> minification on run-time (not on build time) and being able to opt it
> out?

Because nobody want non-minified JS files, like nobody want non-optimized
executables (even if they are easier to debug). You are of course
welcome to implement that into Wordpress if you think this is important.

The security concern at the beginning of this thread also applies fully
to C (https://www.alchemistowl.org/pocorgtfo/pocorgtfo08.pdf, 8:3).
-- 
There's small choice in rotten apples.
-- William Shakespeare, "The Taming of the Shrew"


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  2 septembre 2015 09:32 +0300, Lars Wirzenius  :

> However, I want to raise the point that upstreams do not always make
> sensible decisions, and if they don't, it's good to raise that with
> them. For example, there was recently an ITP bug for
> node-number-is-nan. Upstream source code is at
> https://github.com/sindresorhus/number-is-nan, and the whole package
> boils down to these four lines of code:
>
> 'use strict';
> module.exports = Number.isNaN || function (x) {
> return x !== x;
> };
>
> (https://github.com/sindresorhus/number-is-nan/blob/master/index.js)
>
> If something or someone needs this, we should package it, and it seems
> Grunt needs it. However, it doesn't seem sensible to have a package
> for every one-liner Javascript function, either in Debian or upstream.
> That's going to be a lot of packages, and that alone makes things
> harder to manage for everyone. It'd make sense for the Javascript
> community to produce a more general library to make ES5 look more like
> ES6, which would include a number of such functions.

It exists. For polyfills:

 https://github.com/zloirock/core-js

And for the syntax:

 https://babeljs.io/
-- 
Alas, how love can trifle with itself!
-- William Shakespeare, "The Two Gentlemen of Verona"


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Samuel Thibault]

Vincent Bernat, le Wed 02 Sep 2015 09:47:23 +0200, a écrit :
> If you talk about Grunt,

That's what I'm talking about.

> Grunt comes with a lot of plugins (and does almost nothing without
> those) and each upstream will require different plugins with different
> versions (Grunt plugin versions are evolving fast). See the tree I
> posted for jQuery 3.x in .

That's precisely what I'm talking about.

> All this dependency chain is maintained by a variety of upstreams with
> different release schedules and goals.

Sure, but apparently the set of plugin versions which fit together is
known?  I.e. I guess you didn't write the tree by hand?

Samuel

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  2 septembre 2015 09:54 +0200, Samuel Thibault  :

>> If you talk about Grunt,
>
> That's what I'm talking about.
>
>> Grunt comes with a lot of plugins (and does almost nothing without
>> those) and each upstream will require different plugins with different
>> versions (Grunt plugin versions are evolving fast). See the tree I
>> posted for jQuery 3.x in .
>
> That's precisely what I'm talking about.
>
>> All this dependency chain is maintained by a variety of upstreams with
>> different release schedules and goals.
>
> Sure, but apparently the set of plugin versions which fit together is
> known?  I.e. I guess you didn't write the tree by hand?

Yes, but it's only to compile jQuery. If I take another random project,
I will get a different set of plugins.

Or maybe you propose to just ship the whole "node_modules" directory
(which has all the dependencies) with jQuery sources? This would incur
some work on d/copyright and I don't see like this would be a good
practice. But this would solve some of the problems, yes.
-- 
Why is it that we rejoice at a birth and grieve at a funeral?  It is because we
are not the person involved.
-- Mark Twain, "Pudd'nhead Wilson's Calendar"


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  2 septembre 2015 09:28 +0200, Samuel Thibault  :

>> Healthy language communities have their own metadata systems and
>> standardized build systems that allow Debian packaging to be nearly
>> automated, *provided* that we use the same unit of distribution as
>> upstream.
>
> I understand that using the same unit of distribution helps, but I'd
> tend to think that with not too much work you can achieve automated
> packaging of collections of upstream packages.
>
> Notably, the whole minification toolchain could be uploaded as just one
> package, using on each upload the set of versions that upstream is known
> to be using.

There is no such thing as a "whole minification toolchain".

If you talk about uglifyjs or the like, it is already packaged and
doesn't solve all the problems we have (see my message to Odyx,
).

If you talk about Grunt, Grunt comes with a lot of plugins (and does
almost nothing without those) and each upstream will require different
plugins with different versions (Grunt plugin versions are evolving
fast). See the tree I posted for jQuery 3.x in
. All this dependency chain is maintained
by a variety of upstreams with different release schedules and goals.
-- 
Let the data structure the program.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Samuel Thibault]

Russ Allbery, le Tue 01 Sep 2015 18:05:09 -0700, a écrit :
> Healthy language communities have their own metadata systems and
> standardized build systems that allow Debian packaging to be nearly
> automated, *provided* that we use the same unit of distribution as
> upstream.

I understand that using the same unit of distribution helps, but I'd
tend to think that with not too much work you can achieve automated
packaging of collections of upstream packages.

Notably, the whole minification toolchain could be uploaded as just one
package, using on each upload the set of versions that upstream is known
to be using.

Samuel

Re: Security concerns with minified javascript code

[Samuel Thibault]

Vincent Bernat, le Wed 02 Sep 2015 10:10:55 +0200, a écrit :
>  ❦  2 septembre 2015 09:54 +0200, Samuel Thibault  :
> 
> >> If you talk about Grunt,
> >
> > That's what I'm talking about.
> >
> >> Grunt comes with a lot of plugins (and does almost nothing without
> >> those) and each upstream will require different plugins with different
> >> versions (Grunt plugin versions are evolving fast). See the tree I
> >> posted for jQuery 3.x in .
> >
> > That's precisely what I'm talking about.
> >
> >> All this dependency chain is maintained by a variety of upstreams with
> >> different release schedules and goals.
> >
> > Sure, but apparently the set of plugin versions which fit together is
> > known?  I.e. I guess you didn't write the tree by hand?
> 
> Yes, but it's only to compile jQuery. If I take another random project,
> I will get a different set of plugins.

Ok.

> Or maybe you propose to just ship the whole "node_modules" directory
> (which has all the dependencies) with jQuery sources?

That'd be a lot better than nothing.

> This would incur some work on d/copyright and I don't see like this
> would be a good practice.

Yes, but that work on d/copyright is *needed*: if we don't know for sure
that the compiler itself is really free, then we can't call the result
free and put it in main.

I however agree that it seems poor practice to duplicate these build
modules in every packages. But if the required versions are different,
there is no real other way. If there is a set of modules which are known
to be used widely and with stable versions, then they could be put in a
shared package.

> But this would solve some of the problems, yes.

What problems remain?

Samuel

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  2 septembre 2015 10:18 +0200, Samuel Thibault  :

>> Or maybe you propose to just ship the whole "node_modules" directory
>> (which has all the dependencies) with jQuery sources?
>
> That'd be a lot better than nothing.

OK. Also, node_modules for jQuery is 76M (for 3.x, 70M for 2.x). I still
find using pre-minification jquery.js to be a better alternative due to
the fact is far far simpler. But whatever the consensus we may reach.

>> This would incur some work on d/copyright and I don't see like this
>> would be a good practice.
>
> Yes, but that work on d/copyright is *needed*: if we don't know for sure
> that the compiler itself is really free, then we can't call the result
> free and put it in main.

Yes, but at each release, node_modules will need to be regenerated and
inspected again.

> I however agree that it seems poor practice to duplicate these build
> modules in every packages. But if the required versions are different,
> there is no real other way. If there is a set of modules which are known
> to be used widely and with stable versions, then they could be put in a
> shared package.

I can't say for sure.

>> But this would solve some of the problems, yes.
>
> What problems remain?

Parametrized and custom builds. But we don't have to solve everything at
once.
-- 
Every cloud engenders not a storm.
-- William Shakespeare, "Henry VI"


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Lars Wirzenius]

On Tue, Sep 01, 2015 at 06:05:09PM -0700, Russ Allbery wrote:
> Healthy language communities have their own metadata systems and
> standardized build systems that allow Debian packaging to be nearly
> automated, *provided* that we use the same unit of distribution as
> upstream.  If we want to make any headway on the huge Javascript
> ecosystem, we can't rely on individuals hand-packaging each one of those
> libraries.  We need to be able to use tools to do nearly all the packaging
> work automatically and ask humans only to do sanity checking and bug
> triage and the other parts of the work that we can't automate.  And that's
> way harder if they also have to fight with rebundling upstream releases
> into some other format.

I fully agree with this. Thank you, Russ, for expressing it clearly.

However, I want to raise the point that upstreams do not always make
sensible decisions, and if they don't, it's good to raise that with
them. For example, there was recently an ITP bug for
node-number-is-nan. Upstream source code is at
https://github.com/sindresorhus/number-is-nan, and the whole package
boils down to these four lines of code:

'use strict';
module.exports = Number.isNaN || function (x) {
return x !== x;
};

(https://github.com/sindresorhus/number-is-nan/blob/master/index.js)

If something or someone needs this, we should package it, and it seems
Grunt needs it. However, it doesn't seem sensible to have a package
for every one-liner Javascript function, either in Debian or upstream.
That's going to be a lot of packages, and that alone makes things
harder to manage for everyone. It'd make sense for the Javascript
community to produce a more general library to make ES5 look more like
ES6, which would include a number of such functions.

-- 
Schrödinger's backup hypothesis: the condition of any backup is
undefined until a restore is attempted. -- andrewsh

Re: Security concerns with minified javascript code

[Marvin Renich]

* Ben Hutchings  [150902 10:12]:
> My preferred form is a git repository of code written in C, Python, or
> some other language I know.  That doesn't mean that a tarball of
> Haskell code is non-free!

I can't tell whether you are agreeing or disagreeing with me!

> The preferred form for modification is generally whatever form an
> upstream developer will load into a text editor or other interactive
> editing tool.

No, "A preferred form" is what upstream uses.  The DFSG does not use the
term "THE preferred form", and I believe that was wise.  There can be
multiple "preferred forms" for some software, and all are, in my
opinion, acceptable by the DFSG.  The real question is whether it is
reasonable to expect someone who wishes to modify the software to
consider the form "source".

Again, this is specifically about the DFSG, not the whole Debian Social
Contract.

This whole thread is about how Debian can conform to some points of the
DSC (e.g. points 4 and 2) by providing packages containing minified JS,
and still conform to the DFSG, and whether that means some packages that
are currently in main need to be moved to contrib or non-free.  The
point of contention is not "should we package this software for the
benefit of our users" but whether the packages are allowed in main,
which is strictly based on whether or not they conform to the DFSG.

...Marvin

Re: Security concerns with minified javascript code

[Marvin Renich]

* Neil Williams  [150902 10:22]:
> Upstream is another recipient of code distributed under copyleft.
> Having changes in a format which upstream can use is absolutely a
> sensible and sane criterion for what is regarded as the form of the
> code for modification. To do otherwise is to make the maintenance
> burden untenable.
> 
> Every recipient needs to get the source code and the maintainer changes
> in a format which is suitable for modification and that includes
> the work of modification required to incorporate those changes into the
> next upstream release. To rule out upstream requirements is nonsense.

The whole point of this discussion is what does Debian require of
upstream for upstream to get its software distributed in Debian main.
It is presumed that upstream already has what it considers "source"; in
the case of this thread, that is minified JS.

My point is that if what upstream considers to be "source" is not
acceptable to Debian, and the Debian packager has to grab real source
from other places and use a build process that is different from what
upstream uses in order to make the Debian package satisfy the DFSG, then
upstream's wishes are not relevant to whether the Debian package
conforms to the DFSG.

Furthermore, if the Debian packager does not like upstream's arrangement
of source, even if it would satisfy the DFSG, and wishes to rearrange
it, whether or not the packager's arrangement satisfies the DFSG's
meaning of source should be judged on its own merit, not on whether
upstream is willing to accept patches based on the Debian packager's
arrangement.

I am _not_ saying this this is necessarily a good decision.  The
distinction is between the DFSG, which is one part of the Debian Social
Contract, and the whole DSC.  DSC point 2 requires that the Debian
maintainer give back to upstream.  But that has nothing to do with what
satisfies the DSFG definition of source.

My argument is not that Debian should not use a form that upstream
likes, but that the definition of "source" for purposes of the DFSG is
independent of upstream's definition of source.  If both source forms A
and B satisfy the DFSG, and upstream uses form A, that does not make
form B fail to satisfy the DFSG, even for Debian packages of upstream's
software.

...Marvin

Re: Security concerns with minified javascript code

[Neil Williams]

On Wed, 2 Sep 2015 13:33:57 -0400
Marvin Renich  wrote:

> * Ben Hutchings  [150902 10:12]:
> > My preferred form is a git repository of code written in C, Python,
> > or some other language I know.  That doesn't mean that a tarball of
> > Haskell code is non-free!

> No, "A preferred form" is what upstream uses.  The DFSG does not use
> the term "THE preferred form", and I believe that was wise.  There
> can be multiple "preferred forms" for some software, and all are, in
> my opinion, acceptable by the DFSG.  The real question is whether it
> is reasonable to expect someone who wishes to modify the software to
> consider the form "source".

Minified isn't source for modification, that much is as far as we've
got on consensus in this thread. However, lintian has had checks on
minified without unminified JS available for some time and embedding
upstreams can follow that, so there is source for modification which
the embedding upstream would be able to use but, likely as not, the
original JS upstream would not. Equally, fixing the unminified source
likely also means patching the embedding upstream templates to use
the updated unminified JS as the .min.js cannot be built in the same
way as the original JS upstream. There are other minifiers available
but those could introduce subtle new bugs.

Debian could mandate that embedding upstreams do not include .min.js at
all and I could live with that. It doesn't fix the problem that Debian
lacks a sane way of maintainers keeping javascript packages up to date
with javascript upstreams in ways that embedding upstreams can actually
use.

The result of that is making software in Debian worse by having more
unminified embedded copies at different versions across lots of
packages, exponentially more security issues and more JS packages out of
date relative to the JS upstream. That would not be good.

Embedding software is a bad idea - Debian needs a way of keeping
javascript packages up to date with JS upstreams so that other
upstreams don't need to embed (minified or non-minified) JS in the first
place and everyone has a source for modification in a single place, not
spread across a dozen large packages that simply add in some JS to make
one small (but important) bit of code work.

> This whole thread is about how Debian can conform to some points of
> the DSC (e.g. points 4 and 2) by providing packages containing
> minified JS, and still conform to the DFSG, and whether that means
> some packages that are currently in main need to be moved to contrib
> or non-free.  The point of contention is not "should we package this
> software for the benefit of our users" but whether the packages are
> allowed in main, which is strictly based on whether or not they
> conform to the DFSG.

It's not about contrib or main, that is roughly equivalent to thinking
that every DFSG problem looks like a nail because all you have available
is a large hammer instead of solving the problem inside main.

It is about getting the tools to convert the source for modification
(unminified JS) which is included in packages in main which embed JS
into a minified JS that doesn't make the security problems worse by
introducing it's own bugs by not building the .min.js in the same way
as the JS upstream. It's also about implementing a method which allows
JS maintainers to keep up to date with JS upstreams so that embedding
upstreams don't have to do any of this in the first place.

-- 


Neil Williams
=
http://www.linux.codehelp.co.uk/



pgprkWwZuZgRK.pgp
Description: OpenPGP digital signature

Re: Security concerns with minified javascript code

[Dmitry Smirnov]

On Tuesday 01 September 2015 17:46:30 Josh Triplett wrote:
> Nikolaus Rath wrote:
> > I don't think 28 kB vs 73 kB is a difference that people will notice
> > over the network in *most* situations. Even at just 100 kB/s that's
> > 0.28 vs 0.73 seconds, and only when the page is first loaded.
> 
> That's absolutely a critical difference, even on a faster connection.
> Multiple studies have demonstrated that page load time matters for user
> retention.

Correlation does not imply causation.


> Amazon did a study that showed every ~100ms of page load
> delay lost them 1% in sales.

It could be that small percentage of Amazon users are impulsive trigger-happy 
buyers. :)
However that conclusion is probably wrong due to number of reasons:

 * 1% loss of sales is probably caused not by minification but by other 
means. Remember that minification affects only first page load. I suppose 
most buyers do not just purchase something from the very first page they've 
opened.

 * It is easy to claim loss of sales (and blame it on page load speed) 
without checking how percentage of returns is affected. Impulsive buyers 
never change their minds, right? ;)

 * Most people tend to think harder about more expensive purchases. Therefore 
"loss of revenue" is likely to be much smaller than "loss of sales".


> Google found that half a second slower
> load time for results pages drove off 20% of users.

Who says that loss of this audience matters? Have you ever closed a web site 
just because it is loaded few seconds slower than the other and not due to 
its content?


> Google also prioritizes faster sites in search results.

I doubt that search engine optimisation is important in this context. Google 
have no DFSG concerns and they may prioritise based on other things like SPDY 
support etc. Are you sure Google prioritises smaller web pages over heavy 
ones? (because this is what you're saying as minified content reduces size of 
the web site and not necessary speed of its loading). Besides Google search 
engine is probably just ignores JavaScripts and CSS hence minification should 
not make any difference (unless it is HTML minification).


> So yes, minifiers matter.

IMHO there is more harm than good. The only case for minification that I can 
think of is to increase web server capacity a little to cope with flow of new 
users following some sort of AD campaign. A poor substitute for capacity 
planning or a case when network link is congested.

Minification makes multiple assumptions such as that web app is perfect and 
nobody would ever need to open JS console and report errors. Or that nobody 
would like to learn about web site features from non-minified CSS and JS. Let 
alone debugging some of us do not like proprietary javascripts running in our 
browsers -- minification kills opportunity for security peer review etc.

Finally, one may think that maintenance cost of minified JavaScripts in 
Debian outweighs all the "benefits" by huge margin.

Again, here is my summary why minification is unnecessary:

https://wiki.debian.org/onlyjob/no-minification

I recognise importance of user experience and I know that UI responsiveness 
is important for perception. Actually this is one of my concerns about 
minification: although it negligibly improves speed of downloading of web 
pages for a first time, in theory it may negatively affect JavaScript run-
time performance (e.g. speed of initialisation or UI lag)...

-- 
Regards,
 Dmitry Smirnov.

---

Odious ideas are not entitled to hide from criticism behind the human
shield of their believers' feelings.
-- Richard Stallman


signature.asc
Description: This is a digitally signed message part.

Re: Security concerns with minified javascript code

[Neil Williams]

On Wed, 2 Sep 2015 13:14:31 -0400
Marvin Renich  wrote:

> * Neil Williams  [150902 10:22]:
> > Upstream is another recipient of code distributed under copyleft.
> > Having changes in a format which upstream can use is absolutely a
> > sensible and sane criterion for what is regarded as the form of the
> > code for modification. To do otherwise is to make the maintenance
> > burden untenable.
> > 
> > Every recipient needs to get the source code and the maintainer
> > changes in a format which is suitable for modification and that
> > includes the work of modification required to incorporate those
> > changes into the next upstream release. To rule out upstream
> > requirements is nonsense.
> 
> The whole point of this discussion is what does Debian require of
> upstream for upstream to get its software distributed in Debian main.
> It is presumed that upstream already has what it considers "source";
> in the case of this thread, that is minified JS.

Actually, not. Source, for upstream of JQuery at least, is a set of
directives to include files into a non-minified jquery.js which is then
minified as part of the build in Debian. At that point, we're all good.
It's when other packages have to have other bits of jquery which aren't
packaged or are out of date in Debian and end up pulling those in
as .min.js - there needs to be a way of still using those whilst
allowing for fixes to be applied and used. That is what we lack - the
ability to rebuild embedded .min.js from *any* source for modification.
The problem is that the further that tool diverges from the variety of
tools used by the variety of JS upstreams, the more likely it is that
Debian will introduce further bugs into the .min.js which the JS
upstream will not be able to reproduce.

There are three players:

0: Debian - needs to fix problems in JS wherever we find them.

1: The JS upstream and their non-packaged minifier - may be unaware of
the problem and have their own "source for modification" format. There
may be a maintainer in Debian, likely there is not or the package is
too far out of date. (JS upstreams where the code is up to date must be
dealt with by symlinks into the packages using that JS.)

2: The embedding upstream who use the .min.js but need a way of
patching actual source JS and rebuilding a new .min.js which is,
hopefully, as close to the old one + that one fix as possible without
introducing further bugs.

If there's no embedding upstream, the JS doesn't matter to Debian - we
aren't likely to find bugs in unused libraries of any language, except
FTBFS and someone will probably file an RM RoQA.

If the embedding upstream is writing JS of their own, then they need a
minifier in Debian main too.

None of those players are using or can use the minified JS as source or
for any form of modification other than rebuilding. The current players
are using *different* formats for modification and all lack
packaged or consistent tools to get from one to another.

It is a complete mess - with the embedding upstreams lost in the middle
with no idea of what they can do to avoid getting in the way.

> My point is that if what upstream considers to be "source" is not
> acceptable to Debian, and the Debian packager has to grab real source
> from other places and use a build process that is different from what
> upstream uses in order to make the Debian package satisfy the DFSG,
> then upstream's wishes are not relevant to whether the Debian package
> conforms to the DFSG.

There are different upstreams here. The upstream of the package
embedding the minified JS, yes, that situation needs a fix. The
upstream of the original JS that went into the minified JS has a
different source for modification again - and that one is likely to be
one of the ones which needs a whole gamut of packages to run the
minifier that actually produces the embedded .min.js.

No JS or embedding upstream is going to work with the .min.js as
modifiable source. That's clearly nonsense. What is happening is that
upstreams who need to embed JS are not finding the packages available
and therefore embed and then reference that file in that upstream
as .min.js.

The solution is to have a way of rebuilding that .min.js in Debian so
that fixes can be applied to the available JS for that file - and yet
the upstream for that JS file is likely to have a different build
process from that, so we end up with three possible source formats for
modification - JS upstream, embedded unminified and the one nobody can
actually modify but the one everyone wants to rebuild: minified.

> Furthermore, if the Debian packager does not like upstream's
> arrangement of source, even if it would satisfy the DFSG, and wishes
> to rearrange it, whether or not the packager's arrangement satisfies
> the DFSG's meaning of source should be judged on its own merit, not
> on whether upstream is willing to accept patches based on the Debian
> packager's arrangement.

It's up 

Re: Security concerns with minified javascript code

[Russ Allbery]

The below is very much a tangent from the minified Javascript case, and
not applicable to that case.

Bas Wijnen  writes:

> Here's a rule to limit the selection a bit: a file is certainly not
> source if it was originally generated from a different file, and has not
> been modified.

This makes files for which the source has been irrevocably lost
non-DFSG-free.  I don't think that's a good feature, nor is that the
standard that ftpmaster has used in the past for the archive.

Admittedly, that's something of an edge case, but I've uploaded PostScript
files with that property in one package in the past because they were
still the best available documentation for part of a software package (and
called this out in debian/copyright, and had the package approved by
ftpmaster).  An extensive search had been done for the original source
(which was originally in an internal IBM documentation generation system),
and everyone including IBM was pretty sure that the source was gone
forever and will never be found.

I think reading "preferred form of modification" from the perspective of
upstream is a useful standard because it handles some edge cases like
that, and because it feels ethically consistent with free software
principles.  The goal is that everyone with a copy of the software should
be on equal footing.  The person distributing the software should have no
special access to sources that those receiving the software don't get.

If *no one* has access to anything better than a binary file, then
possession of that binary file puts you on an equal footing with everyone
else in the world, which I think is all that we can reasonably ask.

-- 
Russ Allbery (r...@debian.org)   

Re: Security concerns with minified javascript code

[Jeroen Dekkers]

At Tue, 1 Sep 2015 18:56:45 +0200,
Raphael Hertzog wrote:
> For me, the javascripts bits in wordpress/publican are not part of the
> product, they are external libraries whose preferred form of use is
> by embedding a copy of the library... that sucks but it's the way it is.
> 
> I do not see significant value in extending my packaging to rebuild
> the minified files from source as part of the wordpress/publican source
> package. On the opposite, it has a significant cost:
> - I have to add the sources when upstream does not ship them
>   (which is not a problem for many upstreams since the BSD-ish
>   licences do not require you to provide the sources)
> - ensure the sources are in sync with the minified copy
>   (even when friendly upsreams provide the required sources
>   on our request, they sometimes updates only one the minified file
>   and forget about the sources in some other directory)
> - if the minifier is not the same as upstream, it will create
>   a divergence with upstream and can always be a source of
>   suspicion when we report issues to upstream...

I do see at least one very significant advantage of rebuilding: it's a
lot easier to check that no malicious code is inserted. And if I
understand you correctly you're shipping minified files modified by
upstream for which there might not even be complete up-to-date source
available at all, so how do you know that neither upstream nor someone
who compromised the server used by upstream inserted any backdoor in
the minified file?

What you're saying is like it's fine to have a precompiled static
auxiliary C library that get's linked into the big main program
because making sure that you've got the correct corresponding source
for that small library is hard, the whole world just uses the static
library anyway and in case we do have the correct source then we
probably don't use the exact same compiler as upstream so the object
files would diverge from upstream. Doesn't that sound a bit ridiculous
if we're talking about C? So why would it be okay if we're talking
about javascript?


Kind regards,

Jeroen Dekkers

Re: Security concerns with minified javascript code

[Bas Wijnen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Wed, Sep 02, 2015 at 07:33:10PM +0100, Neil Williams wrote:
> On Wed, 2 Sep 2015 13:33:57 -0400
> Marvin Renich  wrote:
> 
> > * Ben Hutchings  [150902 10:12]:
> > > My preferred form is a git repository of code written in C, Python,
> > > or some other language I know.  That doesn't mean that a tarball of
> > > Haskell code is non-free!
> 
> > No, "A preferred form" is what upstream uses.  The DFSG does not use
> > the term "THE preferred form", and I believe that was wise.

The DFSG doesn't define source at all.  There seems to be consensus (you're the
only one who doesn't seem to agree) that the definition from the GPL is a good
one, and that does say "the".

> > There can be multiple "preferred forms" for some software, and all are, in
> > my opinion, acceptable by the DFSG.  The real question is whether it is
> > reasonable to expect someone who wishes to modify the software to consider
> > the form "source".

I disagree partly.  It is possible to copy a generated file and use that as
source.  IMO that isn't the case until there have actually been made
modifications to that file, though.  If an upstream (which doesn't need to be
the original upstream) actually uses a file to make modifications, an argument
can be made that this format is source.

At the same time, we should try to convince upstreams that do such a thing to
stop it; it causes code duplication and a (security) support nightmare.

"Someone might think they can make modifications to this file" is much too
broad; for some modifications a hex editor is good enough.  And in some cases
that is totally reasonable, such as an executable for which you don't have
source.  That doesn't make binary exectutables source.

A requirement that there is a (serious, not set up to circumvent this rule)
upstream that actually uses this format as their source is still too broad, but
it's a lot better than the overly broad definition you propose.  With your
definition, literally everything is source.

Here's a rule to limit the selection a bit: a file is certainly not source if
it was originally generated from a different file, and has not been modified.

> Minified isn't source for modification, that much is as far as we've
> got on consensus in this thread. However, lintian has had checks on
> minified without unminified JS available for some time and embedding
> upstreams can follow that, so there is source for modification which
> the embedding upstream would be able to use but, likely as not, the
> original JS upstream would not.

If the embedding upstream wants to use updates from the original upstream, they
will not want to make changes to the generated code.  By the sound of it they
haven't figured that out yet, but if they port their changes to new releases of
the original upstream, then those releases are definitely not source,
regardless of whether the embedding upstream is using them for modification.

> Debian could mandate that embedding upstreams do not include .min.js at
> all and I could live with that. It doesn't fix the problem that Debian
> lacks a sane way of maintainers keeping javascript packages up to date
> with javascript upstreams in ways that embedding upstreams can actually
> use.

Why?  If we rebuild everything, the embedding upstreams can copy our build
procedure, or they can take the minified file we generated.

> The result of that is making software in Debian worse by having more
> unminified embedded copies at different versions across lots of
> packages, exponentially more security issues and more JS packages out of
> date relative to the JS upstream. That would not be good.

I thought the minified files were going to be in their own package so there can
be symlinks?  Any other solution involves code duplication which causes those
problems.

> Embedding software is a bad idea - Debian needs a way of keeping
> javascript packages up to date with JS upstreams so that other
> upstreams don't need to embed (minified or non-minified) JS in the first
> place and everyone has a source for modification in a single place, not
> spread across a dozen large packages that simply add in some JS to make
> one small (but important) bit of code work.

Upstreams will always continue embedding though, because even if Debian fixes
it, not everyone will.  And they want their software to run on those other
platforms as well.  So what we want, is to make sure that Debian doesn't need
the embedded copy.  It can still be there, we just shouldn't use it.

> It's not about contrib or main, that is roughly equivalent to thinking
> that every DFSG problem looks like a nail because all you have available
> is a large hammer instead of solving the problem inside main.

But in the end, it is.  If a maintainer refuses to make their package conform
to the DFSG, it cannot be in main.  We all agree that fixing the problem is
better, but people are now claiming that 

Re: Security concerns with minified javascript code

[Thorsten Glaser]

Vincent Bernat  debian.org> writes:

>  2. Upstream may generate the final pre-minification file with complex
> tools, like an AMD loader or an ES6/ES5 transpiler, along with the
> use of non-packaged build tools like Grunt.

> problem. For the second one, a solution would be to consider the
> pre-minification JS code to be perfectly valid source code
> (indentations, comments, variable names, everything is here).

There is (I just had an epiphany) another possible criterium to apply
for to determine what the preferred form of modification is:

Does upstream accept patches for that form?

(The Perl configure script discussion also comes into play, here.)

bye,
//mirabilos

Re: Security concerns with minified javascript code

[Samuel Thibault]

Vincent Bernat, le Wed 02 Sep 2015 11:20:32 +0200, a écrit :
>  ❦  2 septembre 2015 10:18 +0200, Samuel Thibault  :
> >> Or maybe you propose to just ship the whole "node_modules" directory
> >> (which has all the dependencies) with jQuery sources?
> >
> > That'd be a lot better than nothing.
> 
> OK. Also, node_modules for jQuery is 76M (for 3.x, 70M for 2.x).

Is is arch:all?  If so, I guess 76M is completely fine.

> >> This would incur some work on d/copyright and I don't see like this
> >> would be a good practice.
> >
> > Yes, but that work on d/copyright is *needed*: if we don't know for sure
> > that the compiler itself is really free, then we can't call the result
> > free and put it in main.
> 
> Yes, but at each release, node_modules will need to be regenerated and
> inspected again.

Which maps to the fact that we need to continuously make sure that the
software we're basing on is free.

Samuel

Re: Security concerns with minified javascript code

[Marvin Renich]

* Thorsten Glaser  [150902 07:50]:
> There is (I just had an epiphany) another possible criterium to apply
> for to determine what the preferred form of modification is:
   ^ for
  [Okay, so I'm being pedantic, but this is a common mistake.]

> Does upstream accept patches for that form?

I thoroughly and whole-heartedly disagree with this criterion.  As I
stated in an earlier message, the purpose of the source requirement in
the DFSG (and GPL, etc.) is not to protect the rights of the persons
distributing software, but those receiving the software.  There is no
requirement that changes to the software be returned to upstream; such a
requirement would violate the dissident and desert island tests¹.

The source requirement is so that the recipient can make changes if
desired, and if the changes are redistributed (not passed back to
upstream), the second-level recipient may also make changes.

Any test of preferred form for modification must be in terms of how the
recipient is able to use it, not how the distributor would like it.

...Marvin

[1] https://people.debian.org/~bap/dfsg-faq.html#testing

Re: Security concerns with minified javascript code

[Ben Hutchings]

On Wed, 2015-09-02 at 08:59 -0400, Marvin Renich wrote:
> * Thorsten Glaser  [150902 07:50]:
> > There is (I just had an epiphany) another possible criterium to apply
> > for to determine what the preferred form of modification is:
>^ for
>   [Okay, so I'm being pedantic, but this is a common mistake.]
> 
> > Does upstream accept patches for that form?
> 
> I thoroughly and whole-heartedly disagree with this criterion.  As I
> stated in an earlier message, the purpose of the source requirement in
> the DFSG (and GPL, etc.) is not to protect the rights of the persons
> distributing software, but those receiving the software.  There is no
> requirement that changes to the software be returned to upstream; such a
> requirement would violate the dissident and desert island tests¹.
> 
> The source requirement is so that the recipient can make changes if
> desired, and if the changes are redistributed (not passed back to
> upstream), the second-level recipient may also make changes.
> 
> Any test of preferred form for modification must be in terms of how the
> recipient is able to use it, not how the distributor would like it.

My preferred form is a git repository of code written in C, Python, or
some other language I know.  That doesn't mean that a tarball of
Haskell code is non-free!

The preferred form for modification is generally whatever form an
upstream developer will load into a text editor or other interactive
editing tool.

Still, I think there are some exceptions to this.  I used to maintain
the sfc driver in Linux, which has some C header files generated by
script from Verilog or YAML files that aren't published.  In case I
received patches for these headers (usually spelling fixes) I would
make the corresponding change to the unpublished file as well.  I think
that, given the choice, outside developers would still have preferred
editing the C header files, so I was fairly comfortable with this.

Ben.

> [1] https://people.debian.org/~bap/dfsg-faq.html#testing
> 
-- 
Ben Hutchings
Horngren's Observation:
   Among economists, the real world is often a special case.



signature.asc
Description: This is a digitally signed message part

Re: Security concerns with minified javascript code

[Neil Williams]

On Wed, 2 Sep 2015 08:59:11 -0400
Marvin Renich  wrote:

> * Thorsten Glaser  [150902 07:50]:
> > There is (I just had an epiphany) another possible criterium to
> > apply for to determine what the preferred form of modification is:
>^ for
>   [Okay, so I'm being pedantic, but this is a common mistake.]
> 
> > Does upstream accept patches for that form?
> 
> I thoroughly and whole-heartedly disagree with this criterion.  As I
> stated in an earlier message, the purpose of the source requirement in
> the DFSG (and GPL, etc.) is not to protect the rights of the persons
> distributing software, but those receiving the software.  There is no
> requirement that changes to the software be returned to upstream;
> such a requirement would violate the dissident and desert island
> tests¹.

DFSG tests aside, there is copyleft. So where the recipient does
distribute their changes and the code is under a copyleft licence, it
is not just the next recipient who can benefit but also upstream.

There is little point meeting only DFSG requirements if the changes
have to be manually ported from one upstream release to the next simply
because the changes are made using a form which is not used by
upstream. Even without copyleft, the maintainer still has a clear
interest in getting those changes applied upstream or there's little
point in thinking of the software as free - we could easily be accused
of "hoarding" the changes in a way that works against other
distributions by "cutting out the upstream". That is actively unhelpful. 

> The source requirement is so that the recipient can make changes if
> desired, and if the changes are redistributed (not passed back to
> upstream), the second-level recipient may also make changes.
> 
> Any test of preferred form for modification must be in terms of how
> the recipient is able to use it, not how the distributor would like
> it.

Upstream is another recipient of code distributed under copyleft.
Having changes in a format which upstream can use is absolutely a
sensible and sane criterion for what is regarded as the form of the
code for modification. To do otherwise is to make the maintenance
burden untenable.

Every recipient needs to get the source code and the maintainer changes
in a format which is suitable for modification and that includes
the work of modification required to incorporate those changes into the
next upstream release. To rule out upstream requirements is nonsense.

-- 


Neil Williams
=
http://www.linux.codehelp.co.uk/



pgpw8VDlMGOXt.pgp
Description: OpenPGP digital signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  1 septembre 2015 21:10 +0200, Didier 'OdyX' Raboud  :

> I think we should take a strong move there and exercise (as well as 
> justify to the outer world) our free software right to recompile the 
> software that we ship to our users: this could mean to only merge & gzip 
> JS files if minifying isn't realistic [3]. Not doing so _is_ going to 
> hurt our ability to exercise our freedoms in the future, it's also 
> making a disservice to our users.

It seems this thread shed too much tears and is too much focused on
minification. The minification step is usually easy. We have
yui-compressor (that nobody uses upstream, hence the small risk of using
it) and uglifyjs (but a version vulnerable to the attack at the origin
of this thread). What's difficult is to get the code to be modified from
the original source. There are two difficulties:

 1. Upstream may not ship this source but only the minified version
because the JS code is just a dependency and some upstream are used
to just ship the minified source. We can recover the original code
from another source but there is a risk that this is not really the
original code because many JS projects have a modular build (jQuery,
modernizr, ...). This is what Raphael is explaining for Wordpress (I
think).

 2. Upstream may generate the final pre-minification file with complex
tools, like an AMD loader or an ES6/ES5 transpiler, along with the
use of non-packaged build tools like Grunt.

Unfortunately, I don't have an immediate solution for the first
problem. For the second one, a solution would be to consider the
pre-minification JS code to be perfectly valid source code
(indentations, comments, variable names, everything is here).
-- 
Don't compare floating point numbers just for equality.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Marco d'Itri]

On Sep 01, Nikolaus Rath  wrote:

> I don't think 28 kB vs 73 kB is a difference that people will notice
> over the network in *most* situations. Even at just 100 kB/s that's
> 0.28 vs 0.73 seconds, and only when the page is first loaded.
Yes, this is a non trivial difference when loading a web page.
Any tutorial on modern web development would teach this.

-- 
ciao,
Marco


pgpvZb01Jz9YH.pgp
Description: PGP signature

Re: Security concerns with minified javascript code

[Gunnar Wolf]

Vincent Bernat dijo [Fri, Aug 28, 2015 at 10:48:28AM +0200]:
> >> What will happen is that maintainers will fallback to the second less
> >> horrible solution and cripple the package (by using an older version of
> >> the JS lib for example) to allow it to stay in main.
> >
> > Why would they want to stay in main?
> 
> [...]
> 
> > I had the same issue with loadlin: it could only be built on MS-DOS with
> > the proprietary tasm, and thus got #356055. I thus extended the free
> > yasm to recognized the tasm syntax, and patched loadlin a bit to remove
> > some extensions which were hard to implement in yasm but easy to replace
> > in loadlin.
> >
> > Then it could stay in main.
> 
> Here is why.

So, in short, this could be read as "it implies extra work".

But what makes Debian famous for is that we as developers *do* make
that extra work.

It is a great benefit to our users, and it's a core value of the
project. So core, that it is encoded in our foundational documents. 

Re: Security concerns with minified javascript code

[Didier 'OdyX' Raboud]

Le mardi, 1 septembre 2015, 17.50:26 Vincent Bernat a écrit :
>  ❦  1 septembre 2015 08:21 -0700, Nikolaus Rath  :
> >>> Couldn't we just use the non-minified versions in most situations?
> >>> A
> >> 
> >> Not for anything which has actual users over the network.
> > 
> > Why? (Don't forget about gzip encoding).
> 
> See:
>  https://mathiasbynens.be/demo/jquery-size

What this tells us is that the size wins for the latest jquery version 
are the following :

* gzip only -> 29.7%
* zopfli only -> 28.1%
* minifying + gzip -> 11.9%
* minfying + zopfli -> 11.5%

So, taking this jquery example, if minifying is too hard, we can still 
go below 30% of the total size by applying gzip only.

That said…

We should not forget that "minified+gzipped JS" is only a temporary 
state of things. The web ecosystem moved from "ship the full JS source 
to users" (which was way easier in terms of software freedom) to 
"miinified JS". We're at the point where we debate whether this 
"minified JS" is valid source for us (technically, it is valid JS code, 
but not the preferred form of modification).

But the web ecosystem is moving towards WebAssembly [0,1], that will be 
"a portable, size- and load-time-efficient binary format" [2]. Where 
"minified JS" could be seen as source-code, WebAssembly will definitely 
not. If we accept the compromise to not run the compilation step from JS 
to "minified JS", will it be tenable to not run it either for 
_binaries_?

Although I'm concerned by this, I don't doubt much that WebAssembly (or 
any other binary format for the web) _will_ be coming to the web, our 
upstreams, and will become their preferred way to ship frontend code to 
their users. We'll have to deal with that, and we should get ourselves 
ready for that.

The current web compilation stack is arguably ugly, painful to maintain, 
and a big source of frustration, and I can therefore understand how 
maintainers end up not doing the compilation in the Debian package 
build; but really, if we don't do it now, what will happen with binary 
formats?

I think we should take a strong move there and exercise (as well as 
justify to the outer world) our free software right to recompile the 
software that we ship to our users: this could mean to only merge & gzip 
JS files if minifying isn't realistic [3]. Not doing so _is_ going to 
hurt our ability to exercise our freedoms in the future, it's also 
making a disservice to our users.

Cheers,
OdyX

[0] https://blog.mozilla.org/luke/2015/06/17/webassembly/
[1] https://github.com/WebAssembly
[2] https://github.com/WebAssembly/design/blob/master/HighLevelGoals.md
[3] Reducing to 29% instead of 12% might be the price of our freedom
there…

Re: Security concerns with minified javascript code

[Marvin Renich]

* Raphael Hertzog  [150901 12:57]:
> Because we have alternative "compilers" (aka minifier) available to
> recreate another minified file thas should work just as well.

No.  Debian does not allow you to ship a compiled C program that was
compiled elsewhere; the maintainer or a buildd is responsible for taking
the source and creating an executable.  The same applies to minified JS.
I don't see how anyone can argue that minified JS is different.

(And besides, "different but works just as well" is not even close to
the same as "could be built from source, but just wasn't".)

Sometimes doing the right thing is hard.  The GFDL issue is an example;
it took a large effort and a lot of time, but we finally did it.  We did
not remove all GFDL source in one shot.  We decided it really was a
problem, started filing bugs, and started fixing them.  I believe there
was a release in the middle of the purge that ignored (for RC purposes)
those bugs, but we did eventually get all GFDL-licenced material moved
to non-free.

We should do the same for minified JS that is not built using tools in
main.

...Marvin

Re: Security concerns with minified javascript code

[Guido Günther]

On Tue, Sep 01, 2015 at 04:42:15PM +0200, Helmut Grohne wrote:
> On Tue, Sep 01, 2015 at 08:15:19AM +0200, Guido Günther wrote:
> > Couldn't we just use the non-minified versions in most situations? A
> > heavily loaded wordpress site might not be good example but e.g. doxygen
> > documentation probably doesn't suffer much from non minified JS.
> 
> I fail to see what problem that would solve here. The minification
> happens on Debian's buildds using tools from main. What would we gain by
> not doing it?

Iff we have the tools in main to minify there's of course no reason to
ship unminified JS. One can just minify during the build.

> The context of your answer is one of security updates. Why would we want
> to do security updates for the JavaScript shipped with documentation? Do
> you see an attack vector here?
> 
> Even assuming an attack vector, I think the easiest way here would be to
> upload a fixed Doxygen and then binNMU/nochange-NMU all reverse
> dependencies.
> 
> Really, pulling Doxygen in this discussion is a straw man nowadays.
> Please pick better examples or arguments.

There are others. Mozilla extensions, groupware suites, etc. In many
situations going for the unminified version just solves the security
issue without any damage.

Cheers,
 -- Guido

Re: Security concerns with minified javascript code

[Gunnar Wolf]

Vincent Bernat dijo [Fri, Aug 28, 2015 at 11:54:43AM +0200]:
> > I still find it hard to believe that *so* much code is required to
> > minify JS. The excuse that JS is "moving fast" is nonsense. The reality
> > would appear to be that nobody actually *cares* about the mess, they
> > just use it.
> 
> It's a feature. The JS community is quite proud to have so many packages
> for so little tiny tasks. Their packaging system enables to do this
> "easily".

Sounds much like the canonical description of Unix, only throwing
aside tens of years of engineering best practices.

> > Why isn't there a KISS tool to do this? Is it all just special
> > snowflake optimisations for what has to be / should be a simple process
> > of removing whitespace and collapsing the formatting?
> 
> uglifyjs is a KISS tool to minify. Unfortunately, many projects do not
> require only minification. They require transpiling (convert from ES6 to
> ES5 or from CoffeeScript/Typescript/... to vanilla JS) and dependency
> handling (through loaders). And this is becoming more and more common
> because people want to use ES6 or some more modern JS.

...If so, they should be properly labeled and treated as something
different. "Transpiling" effectively means "compiling", and we know
what requirements we have with code in order to accept it compiled: We
need to have the sources as well. Nobody will argue that we don't have
to ship sources for binaries on ARM platforms because ARM has enough
registers so that compiled objects are as good as source.

And, of course, having a tool behave as a compiler when it does not
really understand the mess it is creating... Well, leads to gaping
holes such as what Yan describes here:

https://zyan.scripts.mit.edu/blog/backdooring-js/

A beautiful way of showing how minification hurts.

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  1 septembre 2015 13:45 -0500, Gunnar Wolf  :

>> uglifyjs is a KISS tool to minify. Unfortunately, many projects do not
>> require only minification. They require transpiling (convert from ES6 to
>> ES5 or from CoffeeScript/Typescript/... to vanilla JS) and dependency
>> handling (through loaders). And this is becoming more and more common
>> because people want to use ES6 or some more modern JS.
>
> ...If so, they should be properly labeled and treated as something
> different. "Transpiling" effectively means "compiling", and we know
> what requirements we have with code in order to accept it compiled: We
> need to have the sources as well. Nobody will argue that we don't have
> to ship sources for binaries on ARM platforms because ARM has enough
> registers so that compiled objects are as good as source.

The term transpiling is used because it is mostly a matter of tweaking a
bit the syntax (ES6 has an arrow function syntax not present in ES5) and
inserting polyfills. See for example the future version of jQuery
written with ES6 but transpiled to ES5:

 https://code.jquery.com/jquery-git.js

Unlike ARM binary, this is still Javascript, the indentation and the
comments are still here. The variables have the exact same name. Most
people wouldn't have any problem of believing that this is the base
source code of jQuery because it looks like how jQuery would have been
written in ES5 JS (like it is with jQuery 1.x and 2.x).
-- 
Perilous to all of us are the devices of an art deeper than we ourselves
possess.
-- Gandalf the Grey [J.R.R. Tolkien, "Lord of the
Rings"]


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Don Armstrong]

On Tue, 01 Sep 2015, Vincent Bernat wrote:
>  1. Upstream may not ship this source but only the minified version
>  because the JS code is just a dependency and some upstream are used
>  to just ship the minified source. We can recover the original code
>  from another source but there is a risk that this is not really the
>  original code because many JS projects have a modular build (jQuery,
>  modernizr, ...). This is what Raphael is explaining for Wordpress (I
>  think).

This is precisely why the actual source code is required. This is no
different than a C program shipping a .a file. Perhaps one from a
specific version of a library from which we may have the source code but
which may or may not have been modified in some way which is absolutely
essential for the functioning of the code, but not noted anywhere.

It sucks that this is difficult, but freedom zero is really important,
and requires source code. I mean, we're still working with upstreams who
write C code, and that's been non-controversial for decades.

-- 
Don Armstrong  http://www.donarmstrong.com

As nightfall does not come at once, neither does oppression. In both
instances, there is a twilight when everything remains seemingly
unchanged. And it is in such twilight that we all must be most aware
of change in the air -- however slight -- lest we become unwitting
victims of the darkness.
 -- William O. Douglas

Re: Security concerns with minified javascript code

[Guido Günther]

Hi,
On Mon, Aug 31, 2015 at 09:50:05AM +0200, Raphael Hertzog wrote:
> On Mon, 31 Aug 2015, Simon Josefsson wrote:
> > How would someone rebuild the minified javascript files from the
> > missing-sources files?
> 
> They would not?
> 
> The modified non-minified files are perfectly usable even if they are a
> bit larger than the minified ones.
> 
> >   The included JavaScript file is minified and thus not readily
> >   modifiable, but source code exists in debian/missing-sources/
> >   directory and can be rebuilt into the minified version by doing X.
> 
> ...and can be installed over the minified files if you actually need to make
> changes in those files.

Couldn't we just use the non-minified versions in most situations? A
heavily loaded wordpress site might not be good example but e.g. doxygen
documentation probably doesn't suffer much from non minified JS.

Cheers,
 -- Guido

Re: Security concerns with minified javascript code

[Russ Allbery]

Josh Triplett  writes:

> That said, we absolutely do need to fix this in Debian: it's not OK to
> build packages in main using tools not shipped in Debian, or to ship
> precompiled files.  As a start, it would help if when JavaScript folks
> try to package the packages needed as part of their toolchain, they stop
> getting told that their packages are too small, that they shouldn't be
> packaged at all, or that they should be combined with other packages
> that have different upstream sources and release cycles.

I want to highlight this, because it's an important point that I don't
think had previously been raised in this thread.  There are some
communities that make a practice of releasing very small units of code.  I
understand that our current metadata management and distribution framework
makes this less than ideal for the archive, but I think it would be
worthwhile investing some effort into fixing that instead of pushing
packagers to either not package those components or do a lot more work to
try to create rollup packages that aren't what anyone expects.

Healthy language communities have their own metadata systems and
standardized build systems that allow Debian packaging to be nearly
automated, *provided* that we use the same unit of distribution as
upstream.  If we want to make any headway on the huge Javascript
ecosystem, we can't rely on individuals hand-packaging each one of those
libraries.  We need to be able to use tools to do nearly all the packaging
work automatically and ask humans only to do sanity checking and bug
triage and the other parts of the work that we can't automate.  And that's
way harder if they also have to fight with rebundling upstream releases
into some other format.

I'm not sure how much practical impact this has had, but I know it's come
up a few times, just as it's come up occasionally with Perl modules and
other packages.  If the metadata issues with introducing another ~100
packages in order to model an upstream distribution properly are serious,
that would be a great thing that people in Debian could work on fixing, to
make it much more likely that we can properly package these tools.

-- 
Russ Allbery (r...@debian.org)   

Re: Security concerns with minified javascript code

[Nikolaus Rath]

On Sep 01 2015, Josh Triplett  wrote:
> Nikolaus Rath wrote:
>> I don't think 28 kB vs 73 kB is a difference that people will notice
>> over the network in *most* situations. Even at just 100 kB/s that's
>> 0.28 vs 0.73 seconds, and only when the page is first loaded.
>
> That's absolutely a critical difference, even on a faster connection.
> Multiple studies have demonstrated that page load time matters for user
> retention.  Amazon did a study that showed every ~100ms of page load
> delay lost them 1% in sales.  Google found that half a second slower
> load time for results pages drove off 20% of users.  Google also
> prioritizes faster sites in search results.

Yes, but neither Amazon nor Google (nor any other Web application for
which this matters) is likely to run javascript from a vanilla Debian
package. This is not about minification in general, but about
minification in Debian packages. I don't care enough to actually make a
statistic, but I would not be surprised if most of the javascript that's
carried in Debian ends up in /usr/share/doc.

(end of thread for me)

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Security concerns with minified javascript code

[Josh Triplett]

Nikolaus Rath wrote:
> I don't think 28 kB vs 73 kB is a difference that people will notice
> over the network in *most* situations. Even at just 100 kB/s that's
> 0.28 vs 0.73 seconds, and only when the page is first loaded.

That's absolutely a critical difference, even on a faster connection.
Multiple studies have demonstrated that page load time matters for user
retention.  Amazon did a study that showed every ~100ms of page load
delay lost them 1% in sales.  Google found that half a second slower
load time for results pages drove off 20% of users.  Google also
prioritizes faster sites in search results.

So yes, minifiers matter.  (Hopefully WebAssembly will help in the
future, since even minified JavaScript runs through a text-based
lexing/parsing engine; compiled code could almost certainly beat it on
size.)

That said, we absolutely do need to fix this in Debian: it's not OK to
build packages in main using tools not shipped in Debian, or to ship
precompiled files.  As a start, it would help if when JavaScript folks
try to package the packages needed as part of their toolchain, they stop
getting told that their packages are too small, that they shouldn't be
packaged at all, or that they should be combined with other packages
that have different upstream sources and release cycles.

- Josh Triplett

Re: Security concerns with minified javascript code

[Marco d'Itri]

On Sep 01, Guido Günther  wrote:

> Couldn't we just use the non-minified versions in most situations? A
Not for anything which has actual users over the network.

> heavily loaded wordpress site might not be good example but e.g. doxygen
> documentation probably doesn't suffer much from non minified JS.
doxygen-built documentation is often read online.

-- 
ciao,
Marco


pgptjBHX2z4Py.pgp
Description: PGP signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦ 31 août 2015 11:21 -0400, Marvin Renich  :

>> > However, this is a readable source code that will accomodate any
>> > modification that a end user will deem necessary.
>
> I intentionally did not look at the file referred to above, and have no
> idea whether I would consider it to be a "preferred form" or not.  I
> merely wanted to debunk the "original author's preferred form is the
> only form that can be considered preferred" statement.

Well, I agree with you: there are too many use cases to only have one
preferred form. And this whole thread has a bunch of people who assume
that the unminified jquery.js file is the preferred form of
modification.

The easy actionable step that can be done is to just ensure that
packages are shipped with those pre-minification JS files (considered as
source files) and minify them with either yui-compressor or uglifyjs
that are present in main.
-- 
It usually takes more than three weeks to prepare a good impromptu speech.
-- Mark Twain


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Nikolaus Rath]

On Sep 01 2015, Vincent Bernat  wrote:
>  ❦  1 septembre 2015 08:21 -0700, Nikolaus Rath  :
>
 Couldn't we just use the non-minified versions in most situations? A
>>>
>>> Not for anything which has actual users over the network.
>>
>> Why? (Don't forget about gzip encoding).
>
> See:
>  https://mathiasbynens.be/demo/jquery-size

I don't think 28 kB vs 73 kB is a difference that people will notice
over the network in *most* situations. Even at just 100 kB/s that's
0.28 vs 0.73 seconds, and only when the page is first loaded.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Raphael Hertzog]

Hi,

On Mon, 31 Aug 2015, Bas Wijnen wrote:
> > I certainly do not want to move wordpress or publican to contrib because
> > some of the javascript libraries that it uses can't be rebuilt from main.
> 
> In that case, my question applies to you as well: why do you care for it to be
> in main, if you are unwilling to follow the rules we have set up for it?
>
> > Do you see now how you question is not constructive? The javascript bits
> > are free software
> 
> Which require a compiler that is not in main to build.  That is the definition
> of what contrib is for.  Why shouldn't it go in there?

Because we have alternative "compilers" (aka minifier) available to
recreate another minified file thas should work just as well.

For me, the javascripts bits in wordpress/publican are not part of the
product, they are external libraries whose preferred form of use is
by embedding a copy of the library... that sucks but it's the way it is.

I do not see significant value in extending my packaging to rebuild
the minified files from source as part of the wordpress/publican source
package. On the opposite, it has a significant cost:
- I have to add the sources when upstream does not ship them
  (which is not a problem for many upstreams since the BSD-ish
  licences do not require you to provide the sources)
- ensure the sources are in sync with the minified copy
  (even when friendly upsreams provide the required sources
  on our request, they sometimes updates only one the minified file
  and forget about the sources in some other directory)
- if the minifier is not the same as upstream, it will create
  a divergence with upstream and can always be a source of
  suspicion when we report issues to upstream...

In the end, if the end-user wants to make some changes to the javascript
libraries embedded in wordpress/publican, they can grab the source package
of the corresponding libraries, do their changes and build modified
libraries there. After that they can update it in the publican/wordpress
package by replacing the embedded copy.

> Are you arguing that having tools to go from source to binary available in 
> main
> should not be a requirement for a package to be in main?

No.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Re: Security concerns with minified javascript code

[Helmut Grohne]

On Tue, Sep 01, 2015 at 08:15:19AM +0200, Guido Günther wrote:
> Couldn't we just use the non-minified versions in most situations? A
> heavily loaded wordpress site might not be good example but e.g. doxygen
> documentation probably doesn't suffer much from non minified JS.

I fail to see what problem that would solve here. The minification
happens on Debian's buildds using tools from main. What would we gain by
not doing it?

The context of your answer is one of security updates. Why would we want
to do security updates for the JavaScript shipped with documentation? Do
you see an attack vector here?

Even assuming an attack vector, I think the easiest way here would be to
upload a fixed Doxygen and then binNMU/nochange-NMU all reverse
dependencies.

Really, pulling Doxygen in this discussion is a straw man nowadays.
Please pick better examples or arguments.

Helmut

Re: Security concerns with minified javascript code

[Nikolaus Rath]

On Sep 01 2015, m...@linux.it (Marco d'Itri) wrote:
> On Sep 01, Guido Günther  wrote:
>
>> Couldn't we just use the non-minified versions in most situations? A
>
> Not for anything which has actual users over the network.

Why? (Don't forget about gzip encoding).


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦  1 septembre 2015 08:21 -0700, Nikolaus Rath  :

>>> Couldn't we just use the non-minified versions in most situations? A
>>
>> Not for anything which has actual users over the network.
>
> Why? (Don't forget about gzip encoding).

See:
 https://mathiasbynens.be/demo/jquery-size
-- 
Don't sacrifice clarity for small gains in "efficiency".
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Marvin Renich]

> On Mon, Aug 31, 2015 at 11:21:55AM -0400, Marvin Renich wrote:
> > * Bas Wijnen  [150830 07:53]:
> > > On Sun, Aug 30, 2015 at 10:14:13AM +0200, Vincent Bernat wrote:
> > > > Is that the preferred form of modification? It depends, but from the
> > > > jQuery author point of view, it isn't:
> > > 
> > > Then it isn't.
> > 
> > I take exception to this.
> 
> I agree with your point.  What I meant to say is that what upstream actually
> uses for modifying the work is what we should use as source.  That may change
> if upstream changes, and it may not be a clear definition anyway if upstream
> consists of multiple people and they have different ideas about it.  But most
> of the time this is very clear; if you send a patch and they say "that's not
> the file I use for editing", then it's not the source.

Okay, in general what upstream uses (if it satisfies Debian's definition
of source) is what we should try to use if we don't have a reason to do
otherwise, but not doing so does not violate the DFSG.  That is not the
purpose of the DFSG requiring source.  The purpose is so that
downstreams can fork, using their own repositories and distribution
mechanisms, and perhaps different mandatory coding styles for acceptance
into their repos.  And then a downstream of a first-level downstream can
do the same.

Perhaps one downstream likes to refactor to decrease the total number of
files, and another likes to decrease the average number of lines per
file.  Both are still valid DFSG-compliant source.

> > Also note that the phrase "preferred form of the work for making
> > modifications to it" comes from the GPL, not from the DFSG.
> 
> True, but we don't have a definition ourselves, and there seems to be 
> consensus
> that this is a good one.

Agreed.

...Marvin

Re: Security concerns with minified javascript code

[Nikolaus Rath]

On Sep 01 2015, Helmut Grohne  wrote:
> On Tue, Sep 01, 2015 at 08:15:19AM +0200, Guido Günther wrote:
>> Couldn't we just use the non-minified versions in most situations? A
>> heavily loaded wordpress site might not be good example but e.g. doxygen
>> documentation probably doesn't suffer much from non minified JS.
>
> I fail to see what problem that would solve here. The minification
> happens on Debian's buildds using tools from main. What would we gain by
> not doing it?

We would ensure that the shipped sources actually work. If you ship the
source, but the package is not using it (but the minified version that
was obtained by other means), experience has shown that the shipped
tends to get out-of-sync or otherwise useless.

Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Security concerns with minified javascript code

[Raphael Hertzog]

On Sun, 30 Aug 2015, Bas Wijnen wrote:
> Why do you care that software is in main, if you evidently do not care about
> any of the rules we have for it?

I don't think that implying that Vincent doesn't not care about Free
Software is very constructive.

Can we please stop this now?

If all the energy spent in this thread would have been spent in improving
our javascript ecosystem, it would have been better.

I understand both sides of this discussion and it's a hard problem.

I used to package wordpress with its share of javascript. I maintain
publican which embeds 3 javascript libraries (jquery, jquery.carrousel,
highlight.js).

In both cases, I worked around the problem by shipping the upstream
sources in debian/missing-sources/ but I did not support doing changes
there and did not rebuild the embedded libraries.

In some cases, I do replace the embedded library with a symlink to the
packaged files and I even created dh_linktree to make this easier.

I certainly do not want to move wordpress or publican to contrib because
some of the javascript libraries that it uses can't be rebuilt from main.
I certainly wish we could do it but it's not an itch that I want to
scratch.

Do you see now how you question is not constructive? The javascript bits
are free software and are often a small part of a bigger project that is
free software.

As long as we provide the non-minified javascript files along with all
the embedded copies that we have, we are respecting our social contract.
If some people want to go further, fine, they have all my support.
But now I'd like that people stop to give lessons to their fellow DD who
are actively trying to package parts of the javascript world.

If we want to solve this, we need more such persons, not less.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Re: Security concerns with minified javascript code

[Simon Josefsson]

Raphael Hertzog  writes:

> In both cases, I worked around the problem by shipping the upstream
> sources in debian/missing-sources/ but I did not support doing changes
> there and did not rebuild the embedded libraries.
>
> In some cases, I do replace the embedded library with a symlink to the
> packaged files and I even created dh_linktree to make this easier.

How would someone rebuild the minified javascript files from the
missing-sources files?  I'm looking for something that can become a
template for other packages.  A README-source snippet to document the
process could say something like:

  The included JavaScript file is minified and thus not readily
  modifiable, but source code exists in debian/missing-sources/
  directory and can be rebuilt into the minified version by doing X.

My understanding of the DFSG is still that the compiler must be in main
for the package itself to be in main.  But having a README.source
explaining how to rebuild file files may be a first step.

/Simon


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Helmut Grohne]

On Tue, Aug 25, 2015 at 07:08:06PM +0100, Ian Jackson wrote:
> Not regenerating configure doesn't pose any significant risk that
> we're shipping a configure script that we can't regenerate (or, at
> least, regenerate an equivalent or better one).  I've not heard of
> people (for example) using private autoconf macros not included in
> their build tree.

Even though this has been disputed by Jakub Wilk already, let me give
another example: The configure script included with src:blt cannot be
regenerated anymore. Bug #772590 contains my attempts to fix that. From
my pov src:blt FTBFS.

This is also not the first time where a configure was buggy and
regenerating it would have solved the problem, but regenerating is no
longer possible.

The worst example I ever saw is nothing less than Perl's Configure. In
bug #762638, we agreed that Perl's Configure shell script, which is
generated by metaconfig, is considered preferred form for modification
and thus does not need source code included. But the irony goes on. When
I sent a patch against Configure, it was rejected, because what I wanted
should be done in metaconfig[1], i.e. Perl upstream implicitly
acknowledged that it is not preferred form for modification. The Debian
Perl maintainers have been very helpful in trying to address this
situation, but without more help from upstream, they cannot fix it.

What all of this shows however is that not regenerating configure
scripts during build has a significant risk of causing severe
maintenance problems down the road. When this reaches the point where
the generated scripts is called source, sanity is lost.

Please, let's always build from source. Let's not just apply this to
JavaScript libraries. The only way to provide the freedom to build from
source is to regularly do it.

Helmut

[1] https://rt.perl.org/Public/Bug/Display.html?id=124326#txn-1351869

Re: Security concerns with minified javascript code

[Helmut Grohne]

On Thu, Aug 27, 2015 at 04:14:53PM -0700, Russ Allbery wrote:
> Last time I checked, Doxygen includes minified Javascript in all of its
> generated output.  Would we have to move every piece of Doxygen-generated
> documentation into a separate package so that we could put it in contrib,
> or strip it from our packages?  Maybe someone has fixed this in Doxygen
> somehow?

Indeed, Doxygen used to ship minified JavaScript and the corresponding
source was attached to the Debian packaging, but not regenerated during
build. Since then this has changed and the doxygen package does run the
yui compressor during package build ensuring that the minification is
redoable. The current state is documented at
/usr/share/doc/doxygen/README.jquery. Last time I wondered whether
packages that use Doxygen during build, should also emit a Built-Using
header, it was agreed that Built-Using has a different purpose. So even
though the situation could be improved here, it currently is DFSG
compatible and build from source is ensured. Help with fixing the
embedding issue is welcome of course. When will we have dynamic linking
for JavaScript libraries?

Helmut

Re: Security concerns with minified javascript code

[Marvin Renich]

First, let me make it clear that I am firmly in the camp that believes
minified JS cannot be distributed in main unless the tools to recreate
it are also in main.  It bothers me that there appears to be a
not-insignificant number of people with upload rights who do not believe
this.

This message is not about that, though.

* Bas Wijnen  [150830 07:53]:
> On Sun, Aug 30, 2015 at 10:14:13AM +0200, Vincent Bernat wrote:
> > Is that the preferred form of modification? It depends, but from the
> > jQuery author point of view, it isn't:
> 
> Then it isn't.

I take exception to this.  A number of discussions about "preferred form
for modification" have taken place over the years, and one of the
opinions often set forth is that there is only one "preferred form" for
any given source.  The fact is that different developers have different
preferences.

The fact that the original developer of some software used gimp to
create a simple icon does not mean that the gimp .xcf file must be
considered the preferred form for modification.  Both .png and .ico can
be modified just as easily; in fact there are many more tools in Debian
that can edit those formats than .xcf files.

The basic purpose of the phrase "preferred form for modification" is to
ensure that the right to modify the software is not hindered by only
having an obfuscated version of the source.  I would like to say "any
form of the source that can easily be modified should be allowed," but I
am not sure that I can go quite that far without any qualifications.

Also note that the phrase "preferred form of the work for making
modifications to it" comes from the GPL, not from the DFSG.

> > However, this is a readable source code that will accomodate any
> > modification that a end user will deem necessary.

I intentionally did not look at the file referred to above, and have no
idea whether I would consider it to be a "preferred form" or not.  I
merely wanted to debunk the "original author's preferred form is the
only form that can be considered preferred" statement.

...Marvin

Re: Security concerns with minified javascript code

[Bas Wijnen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, Aug 31, 2015 at 08:49:53AM +0200, Raphael Hertzog wrote:
> On Sun, 30 Aug 2015, Bas Wijnen wrote:
> > Why do you care that software is in main, if you evidently do not care about
> > any of the rules we have for it?
> 
> I don't think that implying that Vincent doesn't not care about Free
> Software is very constructive.

I agree, and if my mail sounded like I was pissed off, that would be correct.
He waves off every criticism that what he's doing is wrong, implying he doesn't
care.  His position seems to be "I have my own definition of what free software
is, and I'll apply that to decide whether software can go into main or not".
That greatly bothers me, because our rules for what software goes into main are
probably the biggest feature of the distribution to me.  His attitude (ignoring
those rules) harms that feature.

I tried to convince him normally first, trying to get him to see why what he
does is wrong.  However, the main question is the one I asked in the text you
quoted from me, it has been asked several times, but never been answered.

> If all the energy spent in this thread would have been spent in improving
> our javascript ecosystem, it would have been better.

If we have people maintaining our packages with the attitude displayed here,
some outsiders trying to patch things for them is not going to help much.  For
all I can see, he'd not even accept the patches because he doesn't consider any
of this a problem.

I agree that working on code is good, but I disagree that talking about
philosophy is wasted time.

> I understand both sides of this discussion and it's a hard problem.

I do understand that packaging minified JS code is hard.  I also understand
that a maintainer may give up on doing it right.  I do not understand that when
that happens, they still insist the package can be in main.

> I certainly do not want to move wordpress or publican to contrib because
> some of the javascript libraries that it uses can't be rebuilt from main.

In that case, my question applies to you as well: why do you care for it to be
in main, if you are unwilling to follow the rules we have set up for it?

> Do you see now how you question is not constructive? The javascript bits
> are free software

Which require a compiler that is not in main to build.  That is the definition
of what contrib is for.  Why shouldn't it go in there?

> and are often a small part of a bigger project that is free software.

If they were separately packaged, they'd need to be in contrib.  The bigger
software, having a Depends on that package, would then also need to be in
contrib.  There is no "unless it's only a little" exception to our requirement
that things must have their compiler in main.

> As long as we provide the non-minified javascript files along with all
> the embedded copies that we have, we are respecting our social contract.

I'll give you that the SC isn't very clear on requiring compilers to be in
main, but policy is (for programs anyway, and javascript certainly is a
program) and I didn't think there was any real discussion about it, really.

Are you arguing that having tools to go from source to binary available in main
should not be a requirement for a package to be in main?

> But now I'd like that people stop to give lessons to their fellow DD who
> are actively trying to package parts of the javascript world.

If people have good intentions (and don't get me wrong, I believe everyone
involved does have good intentions), that doesn't mean I automatically have to
agree that their course of action is acceptable.  And if I feel that what they
do harms Debian, I think it isn't just my right, but it is my duty to say
something about it.  To me, the SC doesn't just mean that my packages will
follow the rules, but also that I will attempt to fix problems that aren't in
my packages (as time permits, of course).  In this case, the main problem that
needs fixing seems to be the interpretation of our rules.  That's a social
problem, which needs to be fixed by talking.

On Mon, Aug 31, 2015 at 11:21:55AM -0400, Marvin Renich wrote:
> * Bas Wijnen  [150830 07:53]:
> > On Sun, Aug 30, 2015 at 10:14:13AM +0200, Vincent Bernat wrote:
> > > Is that the preferred form of modification? It depends, but from the
> > > jQuery author point of view, it isn't:
> > 
> > Then it isn't.
> 
> I take exception to this.

I agree with your point.  What I meant to say is that what upstream actually
uses for modifying the work is what we should use as source.  That may change
if upstream changes, and it may not be a clear definition anyway if upstream
consists of multiple people and they have different ideas about it.  But most
of the time this is very clear; if you send a patch and they say "that's not
the file I use for editing", then it's not the source.

> Also note that the phrase "preferred form of the work for making
> modifications to it" comes from the 

Re: Security concerns with minified javascript code

[Brian May]

On Mon, 31 Aug 2015 at 16:50 Raphael Hertzog  wrote:

> In both cases, I worked around the problem by shipping the upstream
> sources in debian/missing-sources/ but I did not support doing changes
> there and did not rebuild the embedded libraries.
>

I haven't been paying lots of attention to this thread, however in the past
that has been my strategy too.

If I did the minimization myself in the Debian package, or used js from an
existing Debian package, I would be seriously worried about introducing
bugs. There are lots of potential issues here:

* Different versions used.
* Bugs in the minimization process.
* Upstream may have made changes to js code before including it from third
parties.

Some of these packages, the js is only a small part of the full
functionality. Or I just want to package it quickly because it is a
required dependency of the package I really need.

Typically these packages don't have any js testing either, so it is not
always possible to know you broke some obscure feature that requires a
specially crafted input file to enable. Maybe lack of testing is the real
issue that needs to get resolved first?

As an example, my own package - not yet in Debian - I was forced to modify
moment-timezone-with-data.js as downloaded from upstream, because without
this kludged modification, pipeline+slimit combined would completely break
the js to the point that browsers wouldn't touch it.

I reported this against django-pipeline in March, no responses yet.
Actually I don't really understand where the bug is, most likely
django-pipeline or slimit.

https://github.com/cyberdelia/django-pipeline/issues/445

Having said this, yes, I understand the desire to build everything from
source, and I believe this is a good goal.

Re: Security concerns with minified javascript code

[Simon Josefsson]

Raphael Hertzog  writes:

> On Mon, 31 Aug 2015, Simon Josefsson wrote:
>> How would someone rebuild the minified javascript files from the
>> missing-sources files?
>
> They would not?
>
> The modified non-minified files are perfectly usable even if they are a
> bit larger than the minified ones.
>
>>   The included JavaScript file is minified and thus not readily
>>   modifiable, but source code exists in debian/missing-sources/
>>   directory and can be rebuilt into the minified version by doing X.
>
> ...and can be installed over the minified files if you actually need to make
> changes in those files.

Okay, documenting that seems like a good idea -- if there is a security
problem in the minified generated javascript files (the proof-of-concept
that started this thread indicated that is possible), I'm sure we would
want to be able to replace them in a security update.

However I suspect that any security updates will more likely replace the
buggy minified javascript with a fixed minified javascript file (taken
From upstream, generated using some unpackaged compiler), so I'm not
convinced that our shipped source code for this is ever useful if it is
not used to build the minified javascript that we ship.

/Simon


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Raphael Hertzog]

On Mon, 31 Aug 2015, Simon Josefsson wrote:
> How would someone rebuild the minified javascript files from the
> missing-sources files?

They would not?

The modified non-minified files are perfectly usable even if they are a
bit larger than the minified ones.

>   The included JavaScript file is minified and thus not readily
>   modifiable, but source code exists in debian/missing-sources/
>   directory and can be rebuilt into the minified version by doing X.

...and can be installed over the minified files if you actually need to make
changes in those files.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Re: Security concerns with minified javascript code

[Bas Wijnen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Sun, Aug 30, 2015 at 10:14:13AM +0200, Vincent Bernat wrote:
 The build script determines the outcome of what will effectively run on
 our users' machine. I fail to see how this is not an important
 issue.

You are correct, this is important.

 But until the effort to get ppc64el, not regenerating the
 configure script was just a fine option and not considered as DFSG
 violation (all bugs were filed with normal severity). And this existed
 for as long as Debian existed!

We changed our minds on this.  The previous situation _was_ a problem and so we
no longer think this is acceptable.  That's not because autoconf has matured,
but because we changed our priorities.

Your argument seems to be we did this wrong in the past, so we must do it
wrong now.  It surprises me that you think this is a good argument.

 Is that the preferred form of modification? It depends, but from the
 jQuery author point of view, it isn't:

Then it isn't.

 However, this is a readable source code that will accomodate any
 modification that a end user will deem necessary.

That is not the only reason that we want the user to have source.  They are not
some detached customer.  When we make changes to upstream code, we want to
give those changes back to upstream(SC#2).  I expect that our users (other than
ourselves) often want the same thing.  Both for us and for them, it is made
less likely to get changes accepted by upstream if we work on generated files.

 For me, there is no strong problem with DFSG #2 by just using this file as
 the source code.

I wasn't aware that the source was not even provided.  So far, I thought the
problem was that the compiler was not available.

So we have:
- - A problem with SC#2, making it harder to give back to the community.
- - A violation of DFSG#2, because the source isn't in the package.
- - A violation of SC#1/Policy 2.2, because the compiler isn't in main[0].

And you say all of this should be ignored, because otherwise some of our users
might be annoyed by our principles?  I think you're missing the point then.
Those principles are what sets us apart from other distributions.  If users
don't like them, they should use something else.  Users that do like our
principles expect us to live up to them.  Those are the ones I care about.

You wrote before that you don't want to put something in contrib just because
the compiler isn't in main.  But that's exactly what contrib is for.  Why do
you insist on not using it?  Do you feel your software is less free if it's in
there?  Well, our documents agree.  But refusing to do it even if it should be
done doesn't solve that problem.  Packaging the compiler(s) and running it/them
during package build does.

Thanks,
Bas

[0] As was mentioned in this thread before, policy 2.2.2 says:
The contrib archive area contains supplemental packages [...] which
require software outside of the distribution to [...] build [...].
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBAgAGBQJV4u6EAAoJEJzRfVgHwHE67lYP/j2bQnLCOJkurJdSC0/wTCkS
y/m8GMOhr5VdMm8fHAT++YNDJcBUZnFIdYeTXuaD2p15dquCoCOZsnHCo4AO3fC5
NilbtVt301WYGd29svRkhsRDgmZwADKE02NNN7Wm9bhCGoHSFyyfuvjrpwU8aTZ/
7c/VPlYaxbVDPIdSlvuZoYNyMNR6TagjLG9srYAf0WkyMqK18h10vpH4MzTZbnsb
nMU2R4PMUq6j8SXJ0Nb1HphoSchqyvMvMb/pwRJOLsqp4Fxrk8DzPUMcnvwBw/+z
q0XyKEu4jinMsQzjdjQmira5O1x45RGsX4mBzh5rb9vKG2Z7+FhvmIPK1bDUYXDl
qToJCDEwsshyxdJE9HlzB62NGCUInTgx6B5IycSwHoFLfS/dYS+Our4O+L5P6KeD
1QPCeJq6pqn2ip4Lz6QLDC4VVWbmNcrXPS7XmdWa+MwtxJ1nuuCiiQLRC5Loanf3
DCd7NTMLxxrBR2s3vdcArpBwZBzBbfC4xB152pI4ICVsjO6sL/auU47Rrbl9Ud+o
8LdrvPF0VMMTJOUQdIaQQFWQ0hIyRvev6YypYa96HpVk8p6iOg1om0dWDGspZ4Ri
di2Zbav/fMYcsDxu5iF+GgPZYQeueFvD3PHe0e8N1f6NHqVNywmr+v74SbQPqpp/
HN8KCT0/OghVeMeKPC7q
=G8EQ
-END PGP SIGNATURE-

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦ 29 août 2015 19:12 -0700, Steve Langasek vor...@debian.org :

 Yet you try to compare this with autoconf.  Even if we tolerated configure
 scripts today in the archive that we can't rebuild using the software in
 Debian (which by and large we do *not* tolerate - because we've learned our
 lesson), there's a big difference in impact between a build script used once
 at package build time and never shipped in the package to our users, vs.
 swaths of user-facing UI code.

The build script determines the outcome of what will effectively run on
our users' machine. I fail to see how this is not an important
issue. But until the effort to get ppc64el, not regenerating the
configure script was just a fine option and not considered as DFSG
violation (all bugs were filed with normal severity). And this existed
for as long as Debian existed!

 all the code is free software and is provided with the appropriate
 source. A tiny part of it is difficult to rebuild from scratch.

 I don't know if this is true or not; the only package you've used as an
 example in this thread is jquery 3.0.0-pre, which is a) not a package you're
 the maintainer of, and b) not a version of the package that's present in
 Debian.

It seems that I have some difficulties to express myself correctly. I am
sorry for that and let me explain my point more concisely.

Software using jQuery (or another piece of Javascript) usually ship with
a post-processed version of jQuery. For example (I take this example as
Iceweasel doesn't build this component, I don't want to trigger some
witch hunt):
 
https://sources.debian.net/src/iceweasel/38.1.0esr-3/browser/components/loop/content/shared/libs/jquery-2.1.0.js/

Is that the preferred form of modification? It depends, but from the
jQuery author point of view, it isn't:
 https://github.com/jquery/jquery/tree/2.1-stable/src

However, this is a readable source code that will accomodate any
modification that a end user will deem necessary. This is far more
readable than the output of autoconf (again). For me, there is no strong
problem with DFSG #2 by just using this file as the source code.
-- 
Make input easy to proofread.
- The Elements of Programming Style (Kernighan  Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Bas Wijnen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Sun, Aug 30, 2015 at 02:12:43PM +0200, Vincent Bernat wrote:
 This is becoming quite a stretch. At this rate, we will fail to match
 SC#2 because we ship previous versions of software and upstream is
 unlikely to accept a patch against a non-current version.

So you do not care if we (or our users) can send patches upstream in a good
way, you do not care if source is provided, and you do not care if a compiler
is in main.

Why do you care that software is in main, if you evidently do not care about
any of the rules we have for it?

Thanks,
Bas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBAgAGBQJV42k/AAoJEJzRfVgHwHE6YD8P/0GDf8O3gZ5fmnsczeWVRTcf
qWUsWC3/o8tUC2tQ4otxY8TY7/YbkXeV1ZUmu8NlIiPyQPNc+iBDEWElis+cESjg
fPO6MibCL69jHRUCfW1c9p/CNDbdk7qCwuxokTtCBSH2W9j0uBqnFzCbkDqXeR3Q
ZfYDtHdfRW+k7t9pw0uLgiAgRvCzRP0yKehiztOHAm2/lV4pdvMYYzc9CdcGBZr6
J0S9m/R8fdlPBOELXHt+8V1luAbGlTHGqfaEqkHVCrwzdkHnmbI/xLr1wf+wJTM/
Lrotm5CBrVEVGIPX6kUxl+jMLRRRm+tETVEuVHnShD71ZxFmVl4bLcIrtuC/hM9f
GCxICn6KxAHpCPaXZN5U8N8v+JooskGdENwBfLx1xqry1qf4bPruetZ+w6Fn4jjy
rPaJmV6vkhw7dYmGWQHO02mRGLqAyTOsnbQhDOpFyTERU6YO5DrB3DfyGwKimLGE
ff3AUMKBXITq9nn5/lpb28vyHDPCoTq/oxf85V/buSmnTkJr2y8Konv8RATb5ckR
WcqR4QlNRQr+DTSmMuHdeWp6gxUwDq81g2s1ij28Ib5GqtVv2byfl9LB9E8Thdym
90R+ckZ6GU+WmK8NgoF4x4jxgp6L+9BmdnfxePlctSvHNW1SnmTxccffIgWg7cGB
3jf1BmpZQCZs4ddFo8+f
=hkQ+
-END PGP SIGNATURE-

Re: Security concerns with minified javascript code

[Paul Wise]

On Sun, Aug 30, 2015 at 10:14 AM, Vincent Bernat wrote:

 The build script determines the outcome of what will effectively run on
 our users' machine. I fail to see how this is not an important
 issue. But until the effort to get ppc64el, not regenerating the
 configure script was just a fine option and not considered as DFSG
 violation (all bugs were filed with normal severity). And this existed
 for as long as Debian existed!

That we assume that these configure scripts can be regenerated instead
of actually testing that on every build doesn't mean that configure
scripts that can't be regenerated on Debian isn't be a serious bug.

In the long term we probably should talk to autotools upstream about
making it more clear what files are embedded code copies and what
files are generated, so that we can be sure which files to remove when
we want to check that we can regenerate the relevant files.

 However, this is a readable source code that will accomodate any
 modification that a end user will deem necessary. This is far more
 readable than the output of autoconf (again). For me, there is no strong
 problem with DFSG #2 by just using this file as the source code.

That isn't the commonly accepted definition of what source code is AFAIK.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Security concerns with minified javascript code

[IOhannes m zmölnig]

On 08/28/2015 01:14 AM, Russ Allbery wrote:
 Bas Wijnen wij...@debian.org writes:
 
 Last time I checked, Doxygen includes minified Javascript in all of its
 generated output.  Would we have to move every piece of Doxygen-generated
 documentation into a separate package so that we could put it in contrib,
 or strip it from our packages?

how is that related?
doxygen is a compiler (of sorts), and parts of its output is minified js.
the minified js needs to come from somewhere, either it is somehow
generated from the doxy-file or it is copied from a file provided by the
doxygen package.

if doxygen is in main (which might imply that the minified js it
includes in generated output has source-code available and can be
generated using Debian tools), then there is no reason to move a package
into contrib.

  Maybe someone has fixed this in Doxygen
 somehow?

i think the annoying part about doxygen is, that the binary packages
with doxygen generated documentation include a full copy of the js files
(whether minified or not), which is more like *static linking*. urgh.

gfmasrd
IOhannes




signature.asc
Description: OpenPGP digital signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦ 30 août 2015 11:52 GMT, Bas Wijnen wij...@debian.org :

 However, this is a readable source code that will accomodate any
 modification that a end user will deem necessary.

 That is not the only reason that we want the user to have source.
 They are not some detached customer.  When we make changes to
 upstream code, we want to give those changes back to upstream(SC#2).
 I expect that our users (other than ourselves) often want the same
 thing.  Both for us and for them, it is made less likely to get
 changes accepted by upstream if we work on generated files.

This is becoming quite a stretch. At this rate, we will fail to match
SC#2 because we ship previous versions of software and upstream is
unlikely to accept a patch against a non-current version.
-- 
Many pages make a thick book, except for pocket Bibles which are on very
very thin paper.


signature.asc
Description: PGP signature

Re: Pacakging small package (was Re: Security concerns with minified javascript code)

[Samuel Thibault]

Bastien ROUCARIÈS, le Sun 30 Aug 2015 23:15:39 +0200, a écrit :
 What could be done in order to improve the whole system performance in order 
 to package really small package ?

Is it not possible to group them like X.org does?

Samuel

Re: Security concerns with minified javascript code

[Ben Finney]

Steve Langasek vor...@debian.org writes:

 […] Nevertheless, for packages that *are* in Debian, we should expect
 that the source package contains the *full* corresponding source code
 for any minified javascript files. If we can't rebuild it then we
 don't actually have the source, and that's a practical as well as
 philosophical problem for Debian and for our users.

 Yes, packaging a new and fast-moving ecosystem according to Debian's
 standards is a lot of work. Let's figure out the best way to do that
 work, instead of pretending that Debian's standards don't matter.

Thank you Steve, that's a succinct and clear statement of the position
I've been struggling to express so plainly.

My (small, to date) efforts to package works containing JavaScript make
it clear that this is a problem which continues to increase in scale.
It's one that we need the wider community beyond the Debian project to
engage with; but it does at least require the Debian project to be clear
about what is and isn't acceptable.

-- 
 \ “Nature hath given men one tongue but two ears, that we may |
  `\  hear from others twice as much as we speak.” —Epictetus, |
_o__)  _Fragments_ |
Ben Finney

Re: Security concerns with minified javascript code

[Tollef Fog Heen]

]] Vincent Bernat 

  1. package the whole Grunt ecosystem (and maintain it),
  2. cripple their package by substituting some components by a non-working
 version in Debian or,
  3. ship a pre-compiled/minified version of the library with sources.
 
 I know this sucks, but if I have to pick my poison, I'll pick the last
 one. I have tried the second solution in the past, nobody wins (more
 work for the maintainer, more bugs, unhappy users).

In that case, just ship it in contrib until it (or the community) is at
a maturity level where packaging it will work well.

«Free software that requires software outside of Debian to build»

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Re: Security concerns with minified javascript code

[Steve Langasek]

On Fri, Aug 28, 2015 at 07:42:42AM +0200, Vincent Bernat wrote:
  The main effect of this religious and overzealous application of our
  guidelines is that people just stay away of JS stuff in Debian and
  packaging any web-related app is becoming more complex as anyone needs
  to deal with JS stuff in its own package.

  Yes, that is a danger.  I think putting those things in contrib should be a
  good solution if rebuilding is such a big problem.  Because if it is, the 
  code
  really really doesn't belong in main.

 What will happen is that maintainers will fallback to the second less
 horrible solution and cripple the package (by using an older version of
 the JS lib for example) to allow it to stay in main. And here will come
 the angry users and the bad PR.

Sometimes taking a principled stance means that people who don't understand
(or don't agree with) those principles will be angry.  This is not a
justification for abandoning our principles.

I take a dim view of Debian Developers who argue that their packages should
have an exception from the requirements for Debian main because it's hard. 
Sorry, sometimes software is hard.  The constructive thing to do here is to
ask other developers who care about this issue (and this thread shows there
are clearly quite a few) to commit to help with improving Debian's tooling
for JavaScript packages, not to pretend that DFSG#2 doesn't apply to
JavaScript.  The program must include source code, and by any measure,
minified JS is not source code.  And it's not like minified javascript is
some negligible corner case; the browser paradigm has come to so dominate
our user experience over the past few years that this now represents a major
modality of software that users interact with.

Yet you try to compare this with autoconf.  Even if we tolerated configure
scripts today in the archive that we can't rebuild using the software in
Debian (which by and large we do *not* tolerate - because we've learned our
lesson), there's a big difference in impact between a build script used once
at package build time and never shipped in the package to our users, vs.
swaths of user-facing UI code.

 I won't put a package of mine in contrib because of such a technicality:

That's ok, the ftp team is more than capable of taking care of this on your
behalf if you're not going to take responsibility for meeting the
requirements for main.

 all the code is free software and is provided with the appropriate
 source. A tiny part of it is difficult to rebuild from scratch.

I don't know if this is true or not; the only package you've used as an
example in this thread is jquery 3.0.0-pre, which is a) not a package you're
the maintainer of, and b) not a version of the package that's present in
Debian.  Nevertheless, for packages that *are* in Debian, we should expect
that the source package contains the *full* corresponding source code for
any minified javascript files.  If we can't rebuild it then we don't
actually have the source, and that's a practical as well as philosophical
problem for Debian and for our users.

Yes, packaging a new and fast-moving ecosystem according to Debian's
standards is a lot of work.  Let's figure out the best way to do that work,
instead of pretending that Debian's standards don't matter.

Or if there's really no way to do the work that can simultaneously meet
Debian's and upstreams' needs, then let's move the packages to contrib if
that's where they're supposed to be.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Security concerns with minified javascript code

[Philip Hands]

Vincent Bernat ber...@debian.org writes:

 28 août 2015 01:46 GMT, Bas Wijnen wij...@debian.org :

 Or alternatively, by packaging the minifier that is being used with the 
 package
 that needs it.  Yes, that's a horrible idea with lots of code duplication, 
 but
 if I understand the problem, every JS file must be minified with the exact
 version of the minifier that upstream used, so then every package would have
 its own unique package that it depends on, and in that case they can just be
 merged.  But it can't really be that bad, right?

 Here is the dependency graph of jQuery (only to build it!):

 jquery@3.0.0-pre /home/bernat/src/jquery
[ very long list ]
 ├─┬ grunt-contrib-jshint@0.11.2
...
 │ └─┬ jshint@2.8.0


I don't know much about this, but I do know that that is a version that
contains code licensed under the Do No Evil license of JSlint:

  
https://github.com/jshint/jshint/blob/e6611af2d180bd2317d5762e85807a481de99ccb/src/jshint.js#L19

Of course, there is an effort to address this problem, and I've seen
claims that there's a way of getting the same code without that clause
via apache, but that version is not it.

Also of course, JShint is almost certainly not really needed to package
this stuff, so your dependency graph includes optional bits that make
things look much worse than they probably are.

Until some effort is put into taming this beast we'll never know how
unmanageable this really is.  If you're right, and it is impossible,
then who knows what other licensing wrinkles are hiding in this mess.

You seem to be advocating sweeping this under the carpet, but by doing
that you're advocating relying on non-DFSG, unpackaged tools for
building main.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦ 28 août 2015 10:29 +0200, Samuel Thibault sthiba...@debian.org :

 What will happen is that maintainers will fallback to the second less
 horrible solution and cripple the package (by using an older version of
 the JS lib for example) to allow it to stay in main.

 Why would they want to stay in main?

[...]

 I had the same issue with loadlin: it could only be built on MS-DOS with
 the proprietary tasm, and thus got #356055. I thus extended the free
 yasm to recognized the tasm syntax, and patched loadlin a bit to remove
 some extensions which were hard to implement in yasm but easy to replace
 in loadlin.

 Then it could stay in main.

Here is why.
-- 
Choose variable names that won't be confused.
- The Elements of Programming Style (Kernighan  Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Neil Williams]

On Fri, 28 Aug 2015 10:45:16 +0200
Samuel Thibault sthiba...@debian.org wrote:

 Vincent Bernat, le Fri 28 Aug 2015 10:06:17 +0200, a écrit :
  Maybe it can be trimmed a bit more, but that's still 239 unique
  dependencies.
 
 Note that you don't have to make that 239 debian packages, you could
 as well just ship them all in one package, as long as the whole code
 passes NEW, i.e. all their copyrights are fine.

Depends how many upstreams are involved and therefore how many release
schedules. Any more than a handful of differing release schedules and
keeping such a beast up to date is impossible.

I still find it hard to believe that *so* much code is required to
minify JS. The excuse that JS is moving fast is nonsense. The reality
would appear to be that nobody actually *cares* about the mess, they
just use it.

Why isn't there a KISS tool to do this? Is it all just special
snowflake optimisations for what has to be / should be a simple process
of removing whitespace and collapsing the formatting?

Usable software needs usable tools.

As a likely victim of the result(s) of this discussion, I'm going to
say right now that I'm not the one to write such a tool. :-)


-- 


Neil Williams
=
http://www.linux.codehelp.co.uk/



pgpoy_zxYhzlC.pgp
Description: OpenPGP digital signature

Re: Security concerns with minified javascript code

[Dmitry Smirnov]

On Monday 24 August 2015 13:54:21 Simon Josefsson wrote:
 I believe the blog post below has relevance to Debian's stance on
 including minified JavaScript in packages:
 
 https://zyan.scripts.mit.edu/blog/backdooring-js/

Thank you for a nice argument against minification.

During packaging I already had to articulate some reasons against 
minification to several upstreams so I've decided to write a quick summary of 
my reasons against minification in wiki:

https://wiki.debian.org/onlyjob/no-minification

Once improved, I hope this summary might be considered for inclusion into 
UpstreamGuide or to be used as a reference for discouraging such practice.

-- 
Regards,
 Dmitry Smirnov.


signature.asc
Description: This is a digitally signed message part.

Re: Security concerns with minified javascript code

[Samuel Thibault]

Neil Williams, le Fri 28 Aug 2015 10:32:52 +0100, a écrit :
 On Fri, 28 Aug 2015 10:45:16 +0200
 Samuel Thibault sthiba...@debian.org wrote:
 
  Vincent Bernat, le Fri 28 Aug 2015 10:06:17 +0200, a écrit :
   Maybe it can be trimmed a bit more, but that's still 239 unique
   dependencies.
  
  Note that you don't have to make that 239 debian packages, you could
  as well just ship them all in one package, as long as the whole code
  passes NEW, i.e. all their copyrights are fine.
 
 Depends how many upstreams are involved and therefore how many release
 schedules.

As long as we have a set that works, we are fine. Upstream has to have
one anyway, we can just take that one.

 I still find it hard to believe that *so* much code is required to
 minify JS. The excuse that JS is moving fast is nonsense.

I would rather say worrying actually.

 Why isn't there a KISS tool to do this? Is it all just special
 snowflake optimisations for what has to be / should be a simple process
 of removing whitespace and collapsing the formatting?

No, it also modifies the code, e.g. turning !foo  !bar into
!(foo || bar), which is one character less.

I wonder why mere gzip compression is not used. Don't all browsers
support Accept-Compress: gzip?

Samuel

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦ 28 août 2015 10:32 +0100, Neil Williams codeh...@debian.org :

 I still find it hard to believe that *so* much code is required to
 minify JS. The excuse that JS is moving fast is nonsense. The reality
 would appear to be that nobody actually *cares* about the mess, they
 just use it.

It's a feature. The JS community is quite proud to have so many packages
for so little tiny tasks. Their packaging system enables to do this
easily.

 Why isn't there a KISS tool to do this? Is it all just special
 snowflake optimisations for what has to be / should be a simple process
 of removing whitespace and collapsing the formatting?

uglifyjs is a KISS tool to minify. Unfortunately, many projects do not
require only minification. They require transpiling (convert from ES6 to
ES5 or from CoffeeScript/Typescript/... to vanilla JS) and dependency
handling (through loaders). And this is becoming more and more common
because people want to use ES6 or some more modern JS.
-- 
Use the fundamental control flow constructs.
- The Elements of Programming Style (Kernighan  Plauger)


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦ 28 août 2015 12:03 +0200, Samuel Thibault sthiba...@debian.org :

 I wonder why mere gzip compression is not used. Don't all browsers
 support Accept-Compress: gzip?

Minification saves some additional bytes. About 10% (when gzipped).
-- 
If you tell the truth you don't have to remember anything.
-- Mark Twain


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Samuel Thibault]

Vincent Bernat, le Fri 28 Aug 2015 10:06:17 +0200, a écrit :
 Maybe it can be trimmed a bit more, but that's still 239 unique
 dependencies.

Note that you don't have to make that 239 debian packages, you could as
well just ship them all in one package, as long as the whole code passes
NEW, i.e. all their copyrights are fine.

Samuel

Re: Security concerns with minified javascript code

[Samuel Thibault]

Vincent Bernat, le Fri 28 Aug 2015 10:48:28 +0200, a écrit :
  ❦ 28 août 2015 10:29 +0200, Samuel Thibault sthiba...@debian.org :
 
  What will happen is that maintainers will fallback to the second less
  horrible solution and cripple the package (by using an older version of
  the JS lib for example) to allow it to stay in main.
 
  Why would they want to stay in main?

Ergl, I meant to drop that one sentence, and forgot to before sending.

 [...]
 
  I had the same issue with loadlin: it could only be built on MS-DOS with
  the proprietary tasm, and thus got #356055. I thus extended the free
  yasm to recognized the tasm syntax, and patched loadlin a bit to remove
  some extensions which were hard to implement in yasm but easy to replace
  in loadlin.
 
  Then it could stay in main.
 
 Here is why.

And so they should work on making it happen, just like I did.

Samuel

Re: Security concerns with minified javascript code

[Simon Josefsson]

Dmitry Smirnov only...@debian.org writes:

 On Monday 24 August 2015 13:54:21 Simon Josefsson wrote:
 I believe the blog post below has relevance to Debian's stance on
 including minified JavaScript in packages:
 
 https://zyan.scripts.mit.edu/blog/backdooring-js/

 Thank you for a nice argument against minification.

 During packaging I already had to articulate some reasons against 
 minification to several upstreams so I've decided to write a quick summary of 
 my reasons against minification in wiki:

 https://wiki.debian.org/onlyjob/no-minification

 Once improved, I hope this summary might be considered for inclusion into 
 UpstreamGuide or to be used as a reference for discouraging such practice.

Nice writeup, thank you!

/Simon


signature.asc
Description: PGP signature

Re: Security concerns with minified javascript code

[Vincent Bernat]

 ❦ 28 août 2015 08:22 +0100, Philip Hands p...@hands.com :

 Or alternatively, by packaging the minifier that is being used with the 
 package
 that needs it.  Yes, that's a horrible idea with lots of code duplication, 
 but
 if I understand the problem, every JS file must be minified with the exact
 version of the minifier that upstream used, so then every package would have
 its own unique package that it depends on, and in that case they can just be
 merged.  But it can't really be that bad, right?

 Here is the dependency graph of jQuery (only to build it!):

 jquery@3.0.0-pre /home/bernat/src/jquery
 [ very long list ]
 ├─┬ grunt-contrib-jshint@0.11.2
 ...
 │ └─┬ jshint@2.8.0
 

 I don't know much about this, but I do know that that is a version that
 contains code licensed under the Do No Evil license of JSlint:
[...]

It would be quite easy to not use it at all. Here is a curated list
which removes any tool needed only for testing and linting:

jquery@3.0.0-pre /home/bernat/src/jquery
├─┬ commitplease@2.0.0
│ ├─┬ chalk@0.4.0
│ │ ├── ansi-styles@1.0.0
│ │ ├── has-color@0.1.7
│ │ └── strip-ansi@0.1.1
│ ├── mout@0.8.0
│ └── semver@2.2.1
├── core-js@0.9.17
├─┬ grunt@0.4.5
│ ├── async@0.1.22
│ ├── coffee-script@1.3.3
│ ├── colors@0.6.2
│ ├── dateformat@1.0.2-1.2.3
│ ├── eventemitter2@0.4.14
│ ├── exit@0.1.2
│ ├─┬ findup-sync@0.1.3
│ │ ├─┬ glob@3.2.11
│ │ │ ├── inherits@2.0.1
│ │ │ └─┬ minimatch@0.3.0
│ │ │   ├── lru-cache@2.6.5
│ │ │   └── sigmund@1.0.1
│ │ └── lodash@2.4.2
│ ├── getobject@0.1.0
│ ├─┬ glob@3.1.21
│ │ ├── graceful-fs@1.2.3
│ │ └── inherits@1.0.2
│ ├─┬ grunt-legacy-log@0.1.2
│ │ ├── grunt-legacy-log-utils@0.1.1
│ │ ├── lodash@2.4.2
│ │ └── underscore.string@2.3.3
│ ├── grunt-legacy-util@0.2.0
│ ├── hooker@0.2.3
│ ├── iconv-lite@0.2.11
│ ├─┬ js-yaml@2.0.5
│ │ ├─┬ argparse@0.1.16
│ │ │ ├── underscore@1.7.0
│ │ │ └── underscore.string@2.4.0
│ │ └── esprima@1.0.4
│ ├── lodash@0.9.2
│ ├─┬ minimatch@0.2.14
│ │ ├── lru-cache@2.6.5
│ │ └── sigmund@1.0.1
│ ├─┬ nopt@1.0.10
│ │ └── abbrev@1.0.7
│ ├── rimraf@2.2.8
│ ├── underscore.string@2.2.1
│ └── which@1.0.9
├─┬ grunt-babel@5.0.1
│ └─┬ babel-core@5.8.23
│   ├── babel-plugin-constant-folding@1.0.1
│   ├── babel-plugin-dead-code-elimination@1.0.2
│   ├── babel-plugin-eval@1.0.1
│   ├── babel-plugin-inline-environment-variables@1.0.1
│   ├── babel-plugin-jscript@1.0.4
│   ├── babel-plugin-member-expression-literals@1.0.1
│   ├── babel-plugin-property-literals@1.0.1
│   ├── babel-plugin-proto-to-assign@1.0.4
│   ├── babel-plugin-react-constant-elements@1.0.3
│   ├── babel-plugin-react-display-name@1.0.3
│   ├── babel-plugin-remove-console@1.0.1
│   ├── babel-plugin-remove-debugger@1.0.1
│   ├── babel-plugin-runtime@1.0.7
│   ├─┬ babel-plugin-undeclared-variables-check@1.0.2
│   │ └── leven@1.0.2
│   ├── babel-plugin-undefined-to-void@1.1.6
│   ├── babylon@5.8.23
│   ├── bluebird@2.9.34
│   ├─┬ chalk@1.1.1
│   │ ├── ansi-styles@2.1.0
│   │ ├── escape-string-regexp@1.0.3
│   │ ├─┬ has-ansi@2.0.0
│   │ │ └── ansi-regex@2.0.0
│   │ ├─┬ strip-ansi@3.0.0
│   │ │ └── ansi-regex@2.0.0
│   │ └── supports-color@2.0.0
│   ├── convert-source-map@1.1.1
│   ├── core-js@1.1.1
│   ├─┬ debug@2.2.0
│   │ └── ms@0.7.1
│   ├─┬ detect-indent@3.0.1
│   │ ├── get-stdin@4.0.1
│   │ └── minimist@1.2.0
│   ├── esutils@2.0.2
│   ├── fs-readdir-recursive@0.1.2
│   ├── globals@6.4.1
│   ├─┬ home-or-tmp@1.0.0
│   │ ├── os-tmpdir@1.0.1
│   │ └── user-home@1.1.1
│   ├─┬ is-integer@1.0.4
│   │ ├─┬ is-finite@1.0.1
│   │ │ └── number-is-nan@1.0.0
│   │ └─┬ is-nan@1.2.1
│   │   └─┬ define-properties@1.1.1
│   │ ├── foreach@2.0.5
│   │ └── object-keys@1.0.7
│   ├── js-tokens@1.0.1
│   ├── json5@0.4.0
│   ├─┬ line-numbers@0.2.0
│   │ └── left-pad@0.0.3
│   ├── lodash@3.10.1
│   ├─┬ minimatch@2.0.10
│   │ └─┬ brace-expansion@1.1.0
│   │   ├── balanced-match@0.2.0
│   │   └── concat-map@0.0.1
│   ├─┬ output-file-sync@1.1.1
│   │ ├─┬ mkdirp@0.5.1
│   │ │ └── minimist@0.0.8
│   │ └── xtend@4.0.0
│   ├── path-exists@1.0.0
│   ├── path-is-absolute@1.0.0
│   ├── private@0.1.6
│   ├─┬ regenerator@0.8.35
│   │ ├─┬ commoner@0.10.3
│   │ │ ├── commander@2.5.1
│   │ │ ├─┬ glob@4.2.2
│   │ │ │ ├─┬ inflight@1.0.4
│   │ │ │ │ └── wrappy@1.0.1
│   │ │ │ ├── inherits@2.0.1
│   │ │ │ ├─┬ minimatch@1.0.0
│   │ │ │ │ ├── lru-cache@2.6.5
│   │ │ │ │ └── sigmund@1.0.1
│   │ │ │ └─┬ once@1.3.2
│   │ │ │   └── wrappy@1.0.1
│   │ │ ├── graceful-fs@3.0.8
│   │ │ ├── iconv-lite@0.4.11
│   │ │ ├── install@0.1.8
│   │ │ └─┬ mkdirp@0.5.1
│   │ │   └── minimist@0.0.8
│   │ ├─┬ defs@1.1.0
│   │ │ ├─┬ alter@0.2.0
│   │ │ │ └── stable@0.1.5
│   │ │ ├── ast-traverse@0.1.1
│   │ │ ├── breakable@1.0.0
│   │ │ ├── esprima-fb@8001.1001.0-dev-harmony-fb
│   │ │ ├── simple-fmt@0.1.0
│   │ │ ├── simple-is@0.2.0
│   │ │ ├── stringmap@0.2.2
│   │ │ ├── stringset@0.2.1
│   │ │ ├── tryor@0.1.2
│   │ │ └── yargs@1.3.3
│   │ ├── esprima-fb@15001.1.0-dev-harmony-fb
│   │ ├─┬ recast@0.10.24
│   │ │ └── 

Re: Security concerns with minified javascript code

[Samuel Thibault]

Vincent Bernat, le Fri 28 Aug 2015 07:42:42 +0200, a écrit :
  Yes, that is a danger.  I think putting those things in contrib should be a
  good solution if rebuilding is such a big problem.  Because if it is, the 
  code
  really really doesn't belong in main.
 
 What will happen is that maintainers will fallback to the second less
 horrible solution and cripple the package (by using an older version of
 the JS lib for example) to allow it to stay in main.

Why would they want to stay in main?

We could provide an older version of the JS lib in main, and a newer
shiny version of the JS lib in non-free.

 And here will come the angry users and the bad PR.

We already have angry users and bad PR due to DFSG (e.g. firmware blobs,
flash, etc.).  But that's what Debian is.

 The build process can be a bit more complex.
[...]
 I won't put a package of mine in contrib because of such a technicality:
 all the code is free software and is provided with the appropriate
 source. A tiny part of it is difficult to rebuild from scratch.

We still need to package build processes, otherwise we can't call that
free software: the Desert Island test fails if you don't have the build
process.

I had the same issue with loadlin: it could only be built on MS-DOS with
the proprietary tasm, and thus got #356055. I thus extended the free
yasm to recognized the tasm syntax, and patched loadlin a bit to remove
some extensions which were hard to implement in yasm but easy to replace
in loadlin.

Then it could stay in main.

I had the same problem with the pico speech synthesis. The code is
completely free, except that the voices data can not be rebuilt from
source except by using a dedicated compiler, which we happen to not have
the source of. It's not so much a big deal, since we can still run the
existing prebuilt compiler to patch the voices, but we can't fix any
bugs in the compiler or extend it.

And thus the speech synthesizer has to be in non-free until somebody
takes the time to reverse-engineer the compiler.

Samuel

Re: Security concerns with minified javascript code

[Jean-Michel Vourgère]

Vincent Bernat wrote:
 (...)
 It has already been said numerous time in the past, for some Javascript
 code, we don't really have the tools in Debian to easily go from the
 source to the minified version. It's possible, but without the
 appropriate tools, it's painful.

I've been using yui-compressor to get the minified javascript.

I never add any issue this it.

Now if you are talking about generating one big javascript file
containing different fragments in the correct order, that's another
story. But that last issue is not really related to minified js. You can
compress the javascript either before or after yui.

-- 
Nirgal

building jquery without grunt (was Security concerns with minified javascript code)

[Antonio Terceiro]

On Fri, Aug 28, 2015 at 10:06:17AM +0200, Vincent Bernat wrote:
  ❦ 28 août 2015 08:22 +0100, Philip Hands p...@hands.com :
 
  Or alternatively, by packaging the minifier that is being used with the 
  package
  that needs it.  Yes, that's a horrible idea with lots of code 
  duplication, but
  if I understand the problem, every JS file must be minified with the exact
  version of the minifier that upstream used, so then every package would 
  have
  its own unique package that it depends on, and in that case they can just 
  be
  merged.  But it can't really be that bad, right?
 
  Here is the dependency graph of jQuery (only to build it!):
 
  jquery@3.0.0-pre /home/bernat/src/jquery
  [ very long list ]
  ├─┬ grunt-contrib-jshint@0.11.2
  ...
  │ └─┬ jshint@2.8.0
  
 
  I don't know much about this, but I do know that that is a version that
  contains code licensed under the Do No Evil license of JSlint:
 [...]
 
 It would be quite easy to not use it at all. Here is a curated list
 which removes any tool needed only for testing and linting:
 
 jquery@3.0.0-pre /home/bernat/src/jquery
 ├─┬ commitplease@2.0.0
[...]
 └── win-spawn@2.0.0
 
 Maybe it can be trimmed a bit more, but that's still 239 unique
 dependencies. But no jshint is needed (and for building, its usefulness
 is void, so we could just remove the grunt-contrib-jshint package and
 keep every other tests).

At least for jquery 1.x, I just managed to reproduce the upstream build
with ~ 100 lines of code. ~70 of those were basically copied from the
upstream grunt stuff:

https://anonscm.debian.org/cgit/pkg-javascript/jquery.git/tree/debian/build.js?h=debian/1.11.3%2bdfsg-3

the other ~30 lines are make targets in debian/rules:
https://anonscm.debian.org/cgit/pkg-javascript/jquery.git/tree/debian/rules?h=debian/1.11.3%2bdfsg-3

I had to read the source of the upstream grunt build task, understand
that it uses something called requirejs, doing some reading on what
requirejs is and how to use it, then extracting the build logic into a
file that could be used with requirejs (which is already packages as
node-reuqirejs) only -- without grunt -- and there we have it.

-- 
Antonio Terceiro terce...@debian.org


signature.asc
Description: Digital signature

Re: Security concerns with minified javascript code

[Steve McIntyre]

Russ Allbery wrote:
Neil Williams codeh...@debian.org writes:

 Usable software needs usable tools.

The problem is that this *is* usable for nearly all the people who
currently use it, who just run one command to install it and have all
those dependencies pulled from a remote repo for them.  Because the
dependency installation process is so easy, they think no more about
adding new dependencies than we think about installing some application
with apt that happens to require a bunch of shared libraries.

In other words, the people developing and using this tool don't see this
as a problem, and therefore don't care about fixing it.

Depressingly, it seems a lot of the same web typists don't have any
problems with doing the equivalent of curl
http://some.site/install.sh | sudo bash . That doesn't mean we have
to do the same in Debian. If there's no sensible way to do controlled
web development, let's just drop this from Debian *now*.

We can continue having the discussion about how to make things better
and providing clue to clueless upstreams, but in the meantime this is
a massive security breach just waiting to happen.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Further comment on how I feel about IBM will appear once I've worked out
 whether they're being malicious or incompetent. Capital letters are forecast.
 Matthew Garrett, http://www.livejournal.com/users/mjg59/30675.html

Re: Security concerns with minified javascript code

[Pau Garcia i Quiles]

On Fri, Aug 28, 2015 at 4:12 PM, Jean-Michel Vourgère nir...@debian.org
wrote:

Vincent Bernat wrote:
  (...)
  It has already been said numerous time in the past, for some Javascript
  code, we don't really have the tools in Debian to easily go from the
  source to the minified version. It's possible, but without the
  appropriate tools, it's painful.

 I've been using yui-compressor to get the minified javascript.

 I never add any issue this it.

 Now if you are talking about generating one big javascript file
 containing different fragments in the correct order, that's another
 story. But that last issue is not really related to minified js. You can
 compress the javascript either before or after yui.


I have experienced trouble with minifying JavaScript code in my package
witty.

Upstream uses uglifyjs to minify, so do I where it's available.

But on some platforms, uglifyjs is not available due to missing nodejs
(which is in turn due to missing V8 on some platforms). That forces me to
use yui-compressor, which is untested by upstream, and may introduce
hard-to-find bugs:

https://sources.debian.net/src/witty/3.3.4%2Bdfsg-2/debian/rules/

### JavaScript minifier# Use UglifyJS (what upstream uses) where
available,# yui-compressor (what upstream used in the past) where
there is no UglifyJSMINIFIER=$(shell which uglifyjs)ifneq
($(MINIFIER),)  IS_UGLIFY2=$(shell grep -E 'version:
2\.[0-9]+\.[0-9]+' /usr/lib/nodejs/uglify-js/package.json)  ifeq
($(IS_UGLIFY2),)# Legacy: uglifyjs  2.xMINIFIER_FLAGS=-c
--no-seqs -nc  else# uglifyjs = 2.xMINIFIER_FLAGS=-c
sequences=false  endifelse  MINIFIER=/usr/bin/yui-compressor
MINIFIER_FLAGS=--nomungeendif



-- 
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)

Re: Security concerns with minified javascript code

[Russ Allbery]

Neil Williams codeh...@debian.org writes:

 I still find it hard to believe that *so* much code is required to
 minify JS. The excuse that JS is moving fast is nonsense. The reality
 would appear to be that nobody actually *cares* about the mess, they
 just use it.

This is almost certainly correct.

 Usable software needs usable tools.

The problem is that this *is* usable for nearly all the people who
currently use it, who just run one command to install it and have all
those dependencies pulled from a remote repo for them.  Because the
dependency installation process is so easy, they think no more about
adding new dependencies than we think about installing some application
with apt that happens to require a bunch of shared libraries.

In other words, the people developing and using this tool don't see this
as a problem, and therefore don't care about fixing it.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/

Re: Security concerns with minified javascript code

[Pau Garcia i Quiles]

On Fri, Aug 28, 2015 at 4:12 PM, Jean-Michel Vourgère nir...@debian.org
wrote:

Vincent Bernat wrote:
  (...)
  It has already been said numerous time in the past, for some Javascript
  code, we don't really have the tools in Debian to easily go from the
  source to the minified version. It's possible, but without the
  appropriate tools, it's painful.

 I've been using yui-compressor to get the minified javascript.

 I never add any issue this it.

 Now if you are talking about generating one big javascript file
 containing different fragments in the correct order, that's another
 story. But that last issue is not really related to minified js. You can
 compress the javascript either before or after yui.


I have experienced trouble with minifying JavaScript code in my package
witty.

Upstream uses uglifyjs to minify, so do I where it's available.

But on some platforms, uglifyjs is not available due to missing nodejs
(which is in turn due to missing V8 on some platforms). That forces me to
use yui-compressor, which is untested by upstream, and may introduce
hard-to-find bugs:

https://sources.debian.net/src/witty/3.3.4%2Bdfsg-2/debian/rules/

### JavaScript minifier# Use UglifyJS (what upstream uses) where
available,# yui-compressor (what upstream used in the past) where
there is no UglifyJSMINIFIER=$(shell which uglifyjs)ifneq
($(MINIFIER),)  IS_UGLIFY2=$(shell grep -E 'version:
2\.[0-9]+\.[0-9]+' /usr/lib/nodejs/uglify-js/package.json)  ifeq
($(IS_UGLIFY2),)# Legacy: uglifyjs  2.xMINIFIER_FLAGS=-c
--no-seqs -nc  else# uglifyjs = 2.xMINIFIER_FLAGS=-c
sequences=false  endifelse  MINIFIER=/usr/bin/yui-compressor
MINIFIER_FLAGS=--nomungeendif



-- 
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)