Re: How to ensure that your script runs first in a webpage
Le 04/02/2012 01:14, John J Barton a écrit : On Fri, Feb 3, 2012 at 3:14 PM, David Bruant bruan...@gmail.com wrote: Le 03/02/2012 23:26, John J Barton a écrit : On Fri, Feb 3, 2012 at 1:08 PM, David Bruant bruan...@gmail.com wrote: I've been looking at Caja's code recently [1] and noticed that they take (untrusted) code and eval it. Of course, doing a dummy eval would be harmful because eval evaluates with the lexical environment of the calling context. Instead, they've found a way to wrap the eval call and make that when the eval code accesses something, they get to decide what this value is. In a way, they provide the lexical environment of their choice. And I'm guessing, but the objects in that environment need to be special in two ways: the need to have limited API and they need the restrictions like freeze(). It depends on what you call limited. If you mean to limit the number of functions/methods available, then no, you don't need to do this. If you mean to limit the authority of existing API, then yes and that's the best part of it: a trusted code can decide which authority (part of a document, localStorage, XHR, etc.) is available and which is not. Say you want to provide to a given script the ability to send XHRs to a chosen URL, you can reimplement the API to only allow that and fake a network error for any other URL. This would true of all their properties as well, including Object, Function etc. It doesn't have to. It really depends on how much you trust the partialy untrusted code. If you don't care of this code being able to mess with extensible Array.prototype, you can do it (you probably rarerly do, though). Also, forwarding proxies can be useful as they can forward operations that you want and make fail other operations, so you can provide to untrusted code something that looks like a fresh built-in Object, but isn't and if the untrusted code freeze its version of Object.prototype, it doesn't affect your version. The result (which is not unreasonable) is that the API objects have to be low level things, not say dojo widgets or so. Because otherwise the restrictions propagate through out the higher level code. It doesn't have to. The advantage of providing attenuated implementations of existing and standard APIs is that you can easily leverage all code that rely on these APIs. But if you ask untrusted parties to write code according to another API (dojo widget if that's the kind of API you like), then you can do it. A second argument to eval to provide a lexical environment could make things easier. Just to be clear, I am not suggesting a second argument like it was before [2]. The object properties would be the lexical environment of the eval code (and not the lexical environment of the function if what is provided is a function as second argument). It seems to me that such a solution would enable fine granularity attenuated authority at a very small cost: -- // codeInString: localStorage.set('a', 1); var a = localStorage.get('a'); localStorage.set('b', 2); // ... var codeInString = // code above, but as a string var chosenKey; var oneSlotLocalStorage = { get: function(key){ if(typeof chosenKey === 'string') return localStorage.get(chosenKey, val); } set: function(key, val){ if(!chosenKey || chosenKey === key){ chosenKey = key; localStorage.set(chosenKey, val); } else{ throw new Error('You're not allowed to use more than one key!') } } } eval(codeInString, {localStorage:oneSlotLocalStorage}); // 'a' is chosen as the key and an error is thrown when trying to set 'b'. - Of course, it's a dummy example (and the second eval argument doesn't exist), but it seems that this kind of attenuation will be far cheaper (especially when direct proxies will be deployed) than iframes which always recreate an entire environment (ECMAScript built-ins, DOM built-ins, a document instance, etc.) Well I have a soft spot for eval(), but your comparison seems bizarre to me. The codeInString would not have access to ECMAScript built-ins, DOM built-ins, a document instance, etc. So sure the result will have better performance, the only problem is it also can't do anything useful. In my example, no it doesn't, but it would be quite easy to set up a forwarding proxy to my own global object and add restrictions to the APIs I wish to put restrictions on. This way, in a couple of lines, the untrusted code woud have access to my environment minus the APIs I have specifically attenuated (which could range from all of them to none of them, I have the freedom to choose the granularity I want) I'd like to point out that this design also provide a form of security that iframes cannot provide. Same origin iframes have the access to the same localStorage instance(thanks to security based on same origin), but that can be quite a problem when, for instance, a school decides that student addresses will be
Re: How to ensure that your script runs first in a webpage
Le 04/02/2012 04:42, Mark S. Miller a écrit : Considering the 2-argument eval example from above, you could define a 'document' property is the second argument and this document (what the eval code gets when it asks for the document variable) could be an emulation of an HTMLDocument, but forward all (relevant) calls (appendChild...) to a given element you've chosen. It seems it could be easily implemented. Hi David, SES implements essentially your two argument eval, but instead of eval(codeInString, {localStorage:oneSlotLocalStorage}); you'd say var imports = cajaVM.makeImports(); cajaVM.copyToImports(imports, {localStorage:oneSlotLocalStorage}); cajaVM.compileExpr(codeInString)(imports); I had some insight that there was a way, but wasn't sure of how to do it in Caja, thanks. The internalCompileExpr function uses with, how will this code behave in ES6 since it's built on top of ES5 strict? Do you think there would be a value in having the 2-arguments eval as a native construct? Maybe performance? Maybe long-term reliability if there is a test suite in test262 that browsers will be strongly encouraged to comply to? As for your suggestion of providing a virtual document as the binding of document seen by untrusted code, that is precisely how Caja uses SES. The Caja wrapping and virtualization of the browser API (especially the DOM) is known as Domado[3]. [3] http://code.google.com/p/google-caja/source/browse/trunk/src/com/google/caja/plugin/domado.js?r=4758 Thanks for the reference. David ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
On Fri, Feb 3, 2012 at 7:26 PM, Allen Wirfs-Brock al...@wirfs-brock.com wrote: On Feb 3, 2012, at 4:26 PM, Jason Orendorff wrote: On 2/3/12 6:13 PM, Allen Wirfs-Brock wrote: But I also have to validly (and hopefully reasonably) specify exactly what happens for the unrealistic use cases. There is a problem with your desugaring in that the evaluation of INIT isn't scoped correctly relative to V. Hmmm. I don't see the problem yet. I think it's scoped the way I intended it: INIT is evaluated in the enclosing environment; V isn't in scope. Under the scoping rules TC39 has agreed to, the initializer of a let/const is always shadowed by the binding it is initializing[...] That rule doesn't make sense in this context. There should be either one V for the whole loop, or one V per iteration. Having both seems perverse. Again, one can focus on parallels between for(let V=E;;) and let V=E; or on parallels between for(let V=E;;) and for (let V of E). I find the latter more important because experience and evidence suggest that that is what will affect users in practice. -j ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
On Fri, Feb 3, 2012 at 6:13 PM, Allen Wirfs-Brock al...@wirfs-brock.com wrote: On Feb 3, 2012, at 12:44 PM, Jason Orendorff wrote: It might also be useful to look at for(var ...;...;...) and for (...;...;...) loops. Nobody is proposing changing the scoping for those, but they might provide a broader view of how people use the generality of for(;;) loops Out of about 5,500 for loops, I found none with functions in the head. Actually, though, Mozilla's codebase is no better than anyone else's for that purpose. I only picked ours because we have 'let'. I wish we had a bigger corpus to search. This is excluding tests generated by Mozilla's JS fuzzers; the fuzzers often says things like: for(e in this.__defineSetter__(x,function(){})){} ...and much worse. I would actually feel a lot more comfortable with per iteration binding of for(;;) if the for let declaration was limited to a single binding. Of course, that would inconvenience people doing simple stuff that has nothing to do with closures, as in: for (let i = 0, n = a.length; i n; i++) for (let child = element.firstChild, stop = element.lastChild; child !== stop; child = child.nextSibling) My sense is that these uses are even more common than for-loops with escaping closures. -j ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
On Sat, Feb 4, 2012 at 8:02 AM, Jason Orendorff jason.orendo...@gmail.com wrote: On Fri, Feb 3, 2012 at 7:26 PM, Allen Wirfs-Brock al...@wirfs-brock.com wrote: Under the scoping rules TC39 has agreed to, the initializer of a let/const is always shadowed by the binding it is initializing[...] That rule doesn't make sense in this context. There should be either one V for the whole loop, or one V per iteration. Having both seems perverse. I just realized—the loop variables have to be visible in the init-expressions if we want to support this: for (let a = getThings(), i = 0, n = a.length; i n; i++) This is maybe not the best way to write for (thing of getThings()), but people will write it, and so it probably ought to work. I think this is more important than escaping closures. This means that if such loops will have per-iteration bindings, they should have an additional set of bindings just for initialization-time—which seems ugly. Maybe it's not worth it. There is also this: for (let i = 0; i n; ) { setTimeout(...closure using i...); if (shouldAdvance()) i++; } This will not work no matter what semantics we choose. However, per-iteration bindings risk encouraging people to hit this problem. These two issues give me pause. -j ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
On Feb 4, 2012, at 6:02 AM, Jason Orendorff wrote: On Fri, Feb 3, 2012 at 7:26 PM, Allen Wirfs-Brock al...@wirfs-brock.com wrote: On Feb 3, 2012, at 4:26 PM, Jason Orendorff wrote: On 2/3/12 6:13 PM, Allen Wirfs-Brock wrote: But I also have to validly (and hopefully reasonably) specify exactly what happens for the unrealistic use cases. There is a problem with your desugaring in that the evaluation of INIT isn't scoped correctly relative to V. Hmmm. I don't see the problem yet. I think it's scoped the way I intended it: INIT is evaluated in the enclosing environment; V isn't in scope. Under the scoping rules TC39 has agreed to, the initializer of a let/const is always shadowed by the binding it is initializing[...] That rule doesn't make sense in this context. There should be either one V for the whole loop, or one V per iteration. Having both seems perverse. I agree, but having both is exactly what for(let;;) requires in order to satisfy (actually approach satisfying) everybody. If you want the simplicity of the either or alternative then a good approach seems to be that for-in/for-of is one per iteration and for(;;) is one for the whole loop. C# recently when though a similar design change and that is exactly where they ended up. Allen ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
On Feb 4, 2012, at 8:01 AM, Jason Orendorff wrote: On Sat, Feb 4, 2012 at 8:02 AM, Jason Orendorff jason.orendo...@gmail.com wrote: On Fri, Feb 3, 2012 at 7:26 PM, Allen Wirfs-Brock al...@wirfs-brock.com wrote: Under the scoping rules TC39 has agreed to, the initializer of a let/const is always shadowed by the binding it is initializing[...] That rule doesn't make sense in this context. There should be either one V for the whole loop, or one V per iteration. Having both seems perverse. I just realized—the loop variables have to be visible in the init-expressions if we want to support this: for (let a = getThings(), i = 0, n = a.length; i n; i++) This is maybe not the best way to write for (thing of getThings()), but people will write it, and so it probably ought to work. I think this is more important than escaping closures. This means that if such loops will have per-iteration bindings, they should have an additional set of bindings just for initialization-time—which seems ugly. Maybe it's not worth it. That's what the initial let V = INIT I said you needed in your de-sugaring was intended to address There is also this: for (let i = 0; i n; ) { setTimeout(...closure using i...); if (shouldAdvance()) i++; } This will not work no matter what semantics we choose. However, per-iteration bindings risk encouraging people to hit this problem. The } finally { %tmp = V } in your desugaring takes care of making sure that the the V value of the previous iteration is available to the TEST expression These two issues give me pause. -j ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: How to ensure that your script runs first in a webpage
On Sat, Feb 4, 2012 at 3:48 AM, David Bruant bruan...@gmail.com wrote: [...] The internalCompileExpr function uses with, how will this code behave in ES6 since it's built on top of ES5 strict? a) ES6 will still support non-strict code. An indirect ES6 eval (as used here) will still eval non-strict as long as the string being evaluated doesn't start with use strict;. The strictness of the caller of an indirect eval doesn't matter. So the existing SES code should work securely on an ES6 system, as far as we can tell. b) Even the lightweight scanning we're currently doing on ES5 to pick up all potential free variable names will be unnecessary in ES6, since we can just do with (proxy) {. c) The ES6 module loader should make all these with-games unnecessary anyhow, since it gives us a principled approach for controlling the top level scope of untrusted code. Long term, this is the real answer. -- Cheers, --MarkM ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
I want off this merry-go-round! Let's recap: From the January 19th 2012 notes Waldemar took: Discussion about scope of for-bindings. for (var x = ...;;) {...} will, of course, retain ES1 semantics. for (let x = ...;;) {...} Allen: This will behave as in C++: x is bound once in a new scope immediately surrounding just the for statement. DaveH: Strangely enough, this creates a new x binding in Dart at each iteration. There's an alternative semantics that creates an iteration-local second x inside the loop and copies it back and forth. Debate about whether to go to such complexity. Many of us are on the fence. Waldemar: What happens in the forwarding semantics if you capture the x inside a lambda in any of the three expressions in the head? If this happens in the initializer: DaveH's option: The lambda would capture an outer x. Alternative: The lambda captures a hidden second x. Waldemar's option: The lambda would capture the x from the first iteration. The let variable x is bound once through each iteration, just before the test, if for (let x = expr1; expr2;) {...} were: while (true) { let x = first_iteration ? expr1 : value_of_x_from_previous_iteration; if (!expr2) break; ... } MarkM: Just discovered that his desugaring has the same semantics as Waldemar's option. --- end waldemar notes --- Mark's desugaring (https://mail.mozilla.org/pipermail/es-discuss/2008-October/007819.html): You're right. However, the desugaring is more complex than I expected. Thanks for asking me to write it down. for (keyword varName =initExpr;testExpr;updateExpr) {body } desugars to (hygienic renaming aside): breakTarget: { const loop = lambda(iter =initExpr) { keyword varName = iter; if (!testExpr) { break breakTarget; } continueTarget: {body } lambda(iter2 =varName) { keyword varName = iter2; updateExpr; loop(varName); }(); }; loop(); } I believe this meets all your requirements. However, in contradiction to my original claim, one couldn't usefully say const instead of let with a for(;;) loop. --- end mark desugaring --- But then Jon Zeppieri wrote in reply (https://mail.mozilla.org/pipermail/es-discuss/2008-October/007826.html): / I believe this meets all your requirements. / I believe it does. Very cool. It won't handle the fully general for(;;). E.g., for (let fn = lambda(n) { ... fn(...) ... };testExpr;updateExpr) ... Also, for (let i = 0, j = i + 1; ...) ... But the modifications needed to make these work are pretty straightforward. --- end jon citation --- Then Grant Husbands wrote (https://mail.mozilla.org/pipermail/es-discuss/2012-January/019804.html): How about something like this? (given for (letvarName =initExpr;testExpr;updateExpr) {body } ) { letvarName =initExpr; while(true) { if (!testExpr) { break breakTarget; } lettempVar =varName; { // There might be a better way to copy values to/from shadowed variables // (using temporaries seems a bit weak) letvarName =tempVar; continueTarget: {body } tempVar =varName; } varName =tempVar; updateExpr; } } ... Maybe disallowing capture in the for (let ...;...;...) head would be easier. --- end grant citation --- Ok, Brendan here. I agree we want to support multiple declarators for the let or const in the for-head's initialization part. I agree we want to capture the first-iteration bindings in any closures in those declarators' initializers. This requires unrolling the loop once. Let's see how the desugaring from: for (let d1 = e1, ... dN = eN; cond; update) { body; } looks. It doesn't seem terrible: $loopEnd: { let d1 = e1, ... dN = eN; if (cond) { body; update; const $loop = { |d1, ... dN| if (!cond) break $loopEnd; body; update; $loop(d1, ... dN); } $loop(d1, ... dN); } } Notes: * ... is meta-syntax, not rest/spread syntax. * I've left out break and continue in the body. * I'm using a block lambda for fun. Mutations to the first-iteration d1, ... dN bindings in any closures in e1...N propagate to the second iteration. Is this enough to restore the consensus we thought we had at the end of the meeting day on January 19th? /be Jason Orendorff mailto:jason.orendo...@gmail.com February 4, 2012 8:01 AM On Sat, Feb 4, 2012 at 8:02 AM, Jason Orendorff I just realized—the loop variables have to be visible in the init-expressions if we want to support this: for (let a = getThings(), i = 0, n = a.length; i n; i++) This is maybe not the best way to write for (thing of getThings()), but people will write it, and so it probably ought to work. I think this is more important than escaping closures. This means that if such loops will have per-iteration bindings, they should have an additional set of bindings just for initialization-time—which seems ugly. Maybe it's not worth it. There
Re: How to ensure that your script runs first in a webpage
Le 04/02/2012 18:14, Mark S. Miller a écrit : c) The ES6 module loader should make all these with-games unnecessary anyhow, since it gives us a principled approach for controlling the top level scope of untrusted code. Long term, this is the real answer. Ok, good to know. Can you show how eval(string, lexEnv) example would look like with module loaders? The proposal [1] (and the entire web?) lacks a bit of examples. Thanks, David [1] http://wiki.ecmascript.org/doku.php?id=harmony:module_loaders ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
On Feb 4, 2012, at 9:49 AM, Brendan Eich wrote: I want off this merry-go-round! Let's recap: thanks for the recap... ... From the January 19th 2012 notes Waldemar took: Discussion about scope of for-bindings. for (var x = ...;;) {...} will, of course, retain ES1 semantics. for (let x = ...;;) {...} Allen: This will behave as in C++: x is bound once in a new scope immediately surrounding just the for statement. DaveH: Strangely enough, this creates a new x binding in Dart at each iteration. There's an alternative semantics that creates an iteration-local second x inside the loop and copies it back and forth. Debate about whether to go to such complexity. Many of us are on the fence. Waldemar: What happens in the forwarding semantics if you capture the x inside a lambda in any of the three expressions in the head? If this happens in the initializer: DaveH's option: The lambda would capture an outer x. Alternative: The lambda captures a hidden second x. Waldemar's option: The lambda would capture the x from the first iteration. The let variable x is bound once through each iteration, just before the test, if ... --- end grant citation --- Ok, Brendan here. I agree we want to support multiple declarators for the let or const in the for-head's initialization part. check I agree we want to capture the first-iteration bindings in any closures in those declarators' initializers. It isn't clear to me why capture first-iteration is abstractly any better than capture a hidden second x. In both cases, in most iterations of the loop, evaluation of any such captures is going to reference the wrong binding. From a user perspective, the main advantage I see for capture first iteration is that it has a slightly smaller window of wrongness. The captures evaluated in the first iteration will reference the correction binding, while latter iterations reference the wrong binding. From an implementation perspective, it is probably a bit simpler to not have the extra hidden binding for capture. There is another alternative that I haven't seen mentioned. Scope any closures in the initialization expression such that any references to loop declared binding resolve to undeclared bindings, For example (using block lambdas for conciseness) for( let d1={|| d1}, d2={|| d1+d2};;)... would actually be interpreted as if it was for (let d1={|| return {|| d1}; let d1,d2}(), let d2={|| return {|| d1+d2}; let d1,d2}();;)... In words, any references to loop declarations capture uninitialized bindings that never get initialized. If evaluated they will throw as accesses within the TDZ of the binding. Always making such bindings (when used) a error seems preferable I(from a user perspective) than making the first iteration of a loop behave differently from subsequent iterations. ... Mutations to the first-iteration d1, ... dN bindings in any closures in e1...N propagate to the second iteration. Is this enough to restore the consensus we thought we had at the end of the meeting day on January 19th? I really don't like the first iteration is different semantics and think we should think about the above alternative. However, such closure capture is very rare (could use of block lambda based patterns change that??) so it may come down to judgements about implementation costs. Is capture first going to be significantly easier to implement than my alternative scoping? The answer is obvious to me. In either case an implementation is like to special case loops with closure capture in their initializers. Allen ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
Brendan Eich wrote: $loopEnd: { let d1 = e1, ... dN = eN; if (cond) { body; update; const $loop = { |d1, ... dN| if (!cond) break $loopEnd; body; update; $loop(d1, ... dN); } $loop(d1, ... dN); } } Notes: * ... is meta-syntax, not rest/spread syntax. * I've left out break and continue in the body. * I'm using a block lambda for fun. Mutations to the first-iteration d1, ... dN bindings in any closures in e1...N propagate to the second iteration. Another note: * this is a desugaring, so any of d1...N can be destructuring patterns, and with the structuring shorthand for object literals the bindings propagate nicely. Symmetry FTW. /be ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Array extras functionality for iterators
ES5's existing array extras make working with arrays a joy. However, sometimes arrays are not the right tool for the job. Perhaps you want lazy evaluation semantics (generators). Or perhaps you want to communicate that the list is immutable (compare .NET's `IEnumerableT` or Java's `IterableT`). ES Harmony seems to have the answer: iterators! Like `IEnumerableT` or `IterableT`, they are the most basic primitive of iteration. Yay! But, if my `fetchAllProducts()` method returns an iterator, I don't get my array extras. Sad. --- This may be solvable in library-space, but the iterator proposal doesn't seem to have an Iterator.prototype I could extend. So we end up with unfortunate underscore-style wrappers: _(iterator).chain().map(mapper).some(predicate).value() _.some(_.map(iterator, mapper), predicate) I propose adding the array extras to any iterator (in some way), such that we can have syntax similar to the following: iterator.map(mapper).some(predicate) // returns an iterator The methods I would like to see are: * every, filter, forEach, map, reduce, reduceRight, some * Optionally: join, toString * Controversially: sorted, reversed (non-mutating versions of sort and reverse) What do you think? -Domenic ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
Allen Wirfs-Brock wrote: On Feb 4, 2012, at 9:49 AM, Brendan Eich wrote: I agree we want to capture the first-iteration bindings in any closures in those declarators' initializers. It isn't clear to me why capture first-iteration is abstractly any better than capture a hidden second x. In both cases, in most iterations of the loop, evaluation of any such captures is going to reference the wrong binding. The argument is as follows: for (let i = 0, a = some.array, n = a.length; i n; i++) { ... } here we definitely want the a in a.length (n's initializer) to be scoped by the head let -- to be the a declared by the second declarator. Now consider a bit of eta conversion: for (let i = 0, a = some.array, n = (function(){return a})().length; i n; i++) { ... } It would be quite wrong for the a captured by the anonymous function expression to be other than the a binding declared and initialized immediately to the left. It would be bad for the eta conversion to break equivalence (use a block-lambda instead of a function expression for full TCP). From a user perspective, the main advantage I see for capture first iteration is that it has a slightly smaller window of wrongness. The captures evaluated in the first iteration will reference the correction binding, while latter iterations reference the wrong binding. Users expect and even (now that they know, and Dart raises the ante) demand that each iteration gets fresh let bindings. Any who do capture an initial binding in a closure must know, or will learn, that it's just the first one, which fits the model. If this were really a footgun (I don't believe it is without actual evidence from the field) we could try to ban closures capturing the initial bindings. That ad-hoc restriction would be quite a wart. It doesn't seem warranted. From an implementation perspective, it is probably a bit simpler to not have the extra hidden binding for capture. I don't think so. The unrolling I showed was to use a tail-recursive block-lambda helper. But real implementations will do closure analysis and optimization (flat AKA display closures, e.g.) and use branch instructions for loops, jumps for breaks, etc. Having the first binding rib open a bit earlier than subsequent ribs is (I think) a small or zero-cost issue. I really don't like the first iteration is different semantics Different how? Making the first iteration's binding initialization capture guaranteed errors would be different semantics. Capturing the first iteration's bindings from closures in their initializers is not different any more than having initializers is different. The initialization part of the for loop is already special. It's not like the update part. and think we should think about the above alternative. Eta equivalence matters. Given that we want n = a.length to use the a declared to the left in the same for-head declaration, I don't see how we can make closures in a right-ward initializer capture some outer binding. Capturing an error-only binding would need evidence of the footgun not being useful for shooting other things. We don't have such evidence, not by a long shot. However, such closure capture is very rare (could use of block lambda based patterns change that??) I don't think so -- equivalences are stronger, not weaker or different, with block-lambdas vs. functions, due to TCP. so it may come down to judgements about implementation costs. Is capture first going to be significantly easier to implement than my alternative scoping? The answer is obvious to me. Did you mean isn't? In either case an implementation is like to special case loops with closure capture in their initializers. Varying a sketch Jason posted to the SpiderMonkey internals list: for (let V = INIT; TEST; UPD) STMT compiles to enterblock V INIT goto L2 L1: STMT reenterblock UPD L2: TEST iftrue L1 leaveblock This is very close to how SpiderMonkey compiles for(let;;) already -- the only new instruction is reenterblock, which exits the current block and does the equivalent of enterblock V again. V is an immediate operand, in SpiderMonkey a block object literal created by the compiler. It holds all the bindings and their stack offsets, however many declarators with or without destructuring occur after let. I claim implementation is not the driver here. User expectations, esp. savvy users who might make some practical or theoretical (testing) use of eta conversion, matter more. /be ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: Array extras functionality for iterators
Sorry for the resend. Meant to include the list. I like this idea a lot! However, what would be the correct behavior of a method like 'every' on an infinite generator? -Michael A. Smith On Fri, Feb 3, 2012 at 3:54 PM, Domenic Denicola dome...@domenicdenicola.com wrote: ES5's existing array extras make working with arrays a joy. However, sometimes arrays are not the right tool for the job. Perhaps you want lazy evaluation semantics (generators). Or perhaps you want to communicate that the list is immutable (compare .NET's `IEnumerableT` or Java's `IterableT`). ES Harmony seems to have the answer: iterators! Like `IEnumerableT` or `IterableT`, they are the most basic primitive of iteration. Yay! But, if my `fetchAllProducts()` method returns an iterator, I don't get my array extras. Sad. --- This may be solvable in library-space, but the iterator proposal doesn't seem to have an Iterator.prototype I could extend. So we end up with unfortunate underscore-style wrappers: _(iterator).chain().map(mapper).some(predicate).value() _.some(_.map(iterator, mapper), predicate) I propose adding the array extras to any iterator (in some way), such that we can have syntax similar to the following: iterator.map(mapper).some(predicate) // returns an iterator The methods I would like to see are: * every, filter, forEach, map, reduce, reduceRight, some * Optionally: join, toString * Controversially: sorted, reversed (non-mutating versions of sort and reverse) What do you think? -Domenic ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: Array extras functionality for iterators
I'd say every and other methods like that should take another argument that specifies the number if times to test. You could say that every returns: - true if the generator is finite and the function always returned true - false if one of the finite (either because the generator is or because a number was given as argument) number of calls of the function returned false - undefined if the generator is infinite and none of the finite number of calls returned false OR return a generator that tests a new value and returns either true or false every time it is called on the nth call, it tests n values map and filter clearly would be very useful. I'm not sure about the others. On Sat, Feb 4, 2012 at 10:06 PM, Michael A. Smith mich...@smith-li.com wrote: Sorry for the resend. Meant to include the list. I like this idea a lot! However, what would be the correct behavior of a method like 'every' on an infinite generator? -Michael A. Smith On Fri, Feb 3, 2012 at 3:54 PM, Domenic Denicola dome...@domenicdenicola.com wrote: ES5's existing array extras make working with arrays a joy. However, sometimes arrays are not the right tool for the job. Perhaps you want lazy evaluation semantics (generators). Or perhaps you want to communicate that the list is immutable (compare .NET's `IEnumerableT` or Java's `IterableT`). ES Harmony seems to have the answer: iterators! Like `IEnumerableT` or `IterableT`, they are the most basic primitive of iteration. Yay! But, if my `fetchAllProducts()` method returns an iterator, I don't get my array extras. Sad. --- This may be solvable in library-space, but the iterator proposal doesn't seem to have an Iterator.prototype I could extend. So we end up with unfortunate underscore-style wrappers: _(iterator).chain().map(mapper).some(predicate).value() _.some(_.map(iterator, mapper), predicate) I propose adding the array extras to any iterator (in some way), such that we can have syntax similar to the following: iterator.map(mapper).some(predicate) // returns an iterator The methods I would like to see are: * every, filter, forEach, map, reduce, reduceRight, some * Optionally: join, toString * Controversially: sorted, reversed (non-mutating versions of sort and reverse) What do you think? -Domenic ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
It looks to me that all those schemes and desugaring are very complicated. What is bad with scenario of reusing the same variable and isolating it as local only for the async closures? The code: for (let a=..., b=..., otherInit; cond; update) { // use a, b and things from otherInit } Now, if you allow first class refs _in_implementation_, it could be something like this: let a=..., b=..., otherInit; while (cond) { { let _inner_a, _inner_b; // undefined as yet; let ref a :- ref outer a, ref b :- ref outer b, ...; // (1) try { // use a, b and things from otherInit // names a, b and others from for-let are compiled using // refs defined in (1) which point to outer // variables while processing loop body } finally { _inner_a = a; _inner_b = b; ... // assign actual values ref a :- ref _inner_a; ref b :- ref _inner_b; ... // fixing refs at the end of the body to local ones // async closures will use these (since they use the refs // which were rerouted) // the same closures when executed inside loop body would // of course use the outer shared variables, since fixing // only happens after the body // It is always the same ref, but points to different things // in loop and out of the loop } } update; } This way, inside the loop, always the same (outer) variables are used, any inc() things will act upon them, no copying in-out. At the end of every iteration, the refs that pointed to these shared variables are fixed to inner ones, using actual values. Herby ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
On Feb 4, 2012, at 12:55 PM, Brendan Eich wrote: Allen Wirfs-Brock wrote: On Feb 4, 2012, at 9:49 AM, Brendan Eich wrote: I agree we want to capture the first-iteration bindings in any closures in those declarators' initializers. It isn't clear to me why capture first-iteration is abstractly any better than capture a hidden second x. In both cases, in most iterations of the loop, evaluation of any such captures is going to reference the wrong binding. The argument is as follows: for (let i = 0, a = some.array, n = a.length; i n; i++) { ... } here we definitely want the a in a.length (n's initializer) to be scoped by the head let -- to be the a declared by the second declarator. Now consider a bit of eta conversion: for (let i = 0, a = some.array, n = (function(){return a})().length; i n; i++) { ... } It would be quite wrong for the a captured by the anonymous function expression to be other than the a binding declared and initialized immediately to the left. Yes, I support the general principal that that initializers of bindings capture to the left. But the problem here is that conceptually the let is defining multiple bindings (one per iteration). I don't think many people are actually going to understanding the details of the proposed semantics and their implications. Since most uses won't involve closure capture, any of the proposed semantics that have per iteration bindings with forward value propagation are just going to do the right thing. That is good. However, I doubt that someone who actually codes a function in a for(let;;) initializer is going to be thinking, of course, this only captures the first iteration bindings. It would be bad for the eta conversion to break equivalence (use a block-lambda instead of a function expression for full TCP). With the TDZ alternative I proposed, there would still be equivalence for: for(let x=x;;)...; and for(let x={|| x}();;)...; Both throw for accessing an initialized variable. But you're right that equivalence is lost for for(let x=n, y=x;;)...; and for(let x=n, y={|| x}();;)...; Whether this is better or worse than the wrong capture issue complete depends upon the actual programmer intent. From a user perspective, the main advantage I see for capture first iteration is that it has a slightly smaller window of wrongness. The captures evaluated in the first iteration will reference the correction binding, while latter iterations reference the wrong binding. Users expect and even (now that they know, and Dart raises the ante) demand that each iteration gets fresh let bindings. Any who do capture an initial binding in a closure must know, or will learn, that it's just the first one, which fits the model. For the latter, I strongly suspect that they won't know and will be WTF surprised when they encounter it. The saving grace is that this will probably be very rare, although its possible that the introduction of block lambdas might somewhat change that. Just don't know... If this were really a footgun (I don't believe it is without actual evidence from the field) we could try to ban closures capturing the initial bindings. That ad-hoc restriction would be quite a wart. It doesn't seem warranted. My TDZ solution is such a restriction. But I don't see how it is any more ad hoc than any of the other changes we are talking about here in order to give for(;;) per iteration bindings. Its wartiness actually seems small and is restricted to a situation where the programmer probably is actually expecting C=style per loop rather than per iteration binding. From an implementation perspective, it is probably a bit simpler to not have the extra hidden binding for capture. I don't think so. The unrolling I showed was to use a tail-recursive block-lambda helper. But real implementations will do closure analysis and optimization (flat AKA display closures, e.g.) and use branch instructions for loops, jumps for breaks, etc. Having the first binding rib open a bit earlier than subsequent ribs is (I think) a small or zero-cost issue. My comment wasn't about the bind to first unrolling. It was about the extra hidden binding alternative in the first list of alternatives and is probably also applicable to by TDZ alternative. I really don't like the first iteration is different semantics Different how? Different from subsequent iterations... Take Jason's example for (let i = 0; i n; ) { setTimeout(...closure using i...); if (shouldAdvance()) i++; } If somebody decided to abstract the increment: for (let i = 0; advance={|| i++}; i n; ) { setTimeout(...closure using i...); if (shouldAdvance()) advance(); } advance does what is intended if called on the first iteration, but not on subsequent one iterations. I'd soon get an error when I ran this than having
RE: Array extras functionality for iterators
I'd say every and other methods like that should take another argument that specifies the number if times to test. This seems reasonable. Preferably that parameter would be optional, since the majority of the time my iterators will not be derived from infinite generators (and I know this). Otherwise I'd just keep passing `Infinity` as the third argument to my methods. Is it possible to detect infinite generators ahead of time? My intuition says no. map and filter clearly would be very useful. I'm not sure about the others. Losing the others would be a shame. As I mentioned, much of the time I want to expose something with the semantics of an iterator, even if internally to my module or class I am storing the data as an array. So even if infinite generators cause problems for implementing `every` et al., the vast majority of real-world iterators would not be derived from infinite generators. Thus my above desire for the extra argument being optional. Currently in our codebase, we have a number of these methods (like `fetchAllProducts()`) that return `slice()`ed copies of internal arrays, since we don't want to expose the arrays directly. (Iterators would of course be the best solution here.) Over the course of 77K LOC, the only methods we don't use on the products collection are `join`, `reduceRight`, `reverse`, and `toString`. And we do use `join` on some of the simpler collections (`fetchProductTags()`). So they're all useful! -Domenic On Sat, Feb 4, 2012 at 10:06 PM, Michael A. Smith mich...@smith-li.com wrote: Sorry for the resend. Meant to include the list. I like this idea a lot! However, what would be the correct behavior of a method like 'every' on an infinite generator? -Michael A. Smith On Fri, Feb 3, 2012 at 3:54 PM, Domenic Denicola dome...@domenicdenicola.com wrote: ES5's existing array extras make working with arrays a joy. However, sometimes arrays are not the right tool for the job. Perhaps you want lazy evaluation semantics (generators). Or perhaps you want to communicate that the list is immutable (compare .NET's `IEnumerableT` or Java's `IterableT`). ES Harmony seems to have the answer: iterators! Like `IEnumerableT` or `IterableT`, they are the most basic primitive of iteration. Yay! But, if my `fetchAllProducts()` method returns an iterator, I don't get my array extras. Sad. --- This may be solvable in library-space, but the iterator proposal doesn't seem to have an Iterator.prototype I could extend. So we end up with unfortunate underscore-style wrappers: _(iterator).chain().map(mapper).some(predicate).value() _.some(_.map(iterator, mapper), predicate) I propose adding the array extras to any iterator (in some way), such that we can have syntax similar to the following: iterator.map(mapper).some(predicate) // returns an iterator The methods I would like to see are: * every, filter, forEach, map, reduce, reduceRight, some * Optionally: join, toString * Controversially: sorted, reversed (non-mutating versions of sort and reverse) What do you think? -Domenic ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
Herby Vojčík wrote: It looks to me that all those schemes and desugaring are very complicated. And what you showed isn't complicated? LOL. What you're sketching is an optimization, one I believe Chez Scheme and other implementations perform. Indeed any good implementation will optimize let-bound loop variables to registers and avoid any per-iteration overhead in the absence of closures. But what you sketched is nothing like a specification. A normative specification does not describe non-observables, except in informative asides (which should be used sparingly). A specification must simply and clearly state the rules for syntax and semantics. In particular, the ES6 specification governing the for-let loops we're discussing needs specific treatment to avoid the notorious var pigeon-hole problem made observable by closures in the loop body that capture loop variables. The simplest way to do this is being worked out, but it will necessarily entail a scope per iteration. I argue the first iteration's scope must enclose the initialisers for the for(let;;) head's first of three parts. This is an important detail. It's not complicated beyond other parts of the spec; it's just detailed. We have to have consistent for(let;;) semantics, and as the discussion has evolved (since 2008) we've come to agree on the constraints by which we reckon consistent. The constraints require a specific solution, in my view: first iteration's scope covers iniitalizers, and observably-distinct let scopes for each iteration. /be ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: Array extras functionality for iterators
Has anyone checked out or used http://docs.python.org/py3k/library/itertools.html ? Seems worth considering since ES6 generators are derived from Python generators. Indeed, there's no way to detect an infinite generator ahead of time. Some array extras do not make sense even for very large arrays. Again I cite Peter Norvig's constraint-propagating-search-based Sudoku solver, which I ported to JS1.8: https://bug380237.bugzilla.mozilla.org/attachment.cgi?id=266577 It definitely requires generators not arrays in places, to avoid exponential space blow-up. /be ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: Array extras functionality for iterators
On a related note, there are many ways in which some of these methods would overlap with the capabilities of array comprehensions and generator expressions. As far as I can tell, the one thing you cannot do (with, say, map) is take an array as input and process it lazily. For example: someVeryLargeArray.map(someFunction); // guarantees an array result, cannot be lazy vs (someFunction(someVeryLargeArray(i), i, someVeryLargeArray) for (i in someVeryLargeArray)) // Lazy, likely to perform better in some important cases. I think there's a case to be made for direct implementations of such methods. Something like someVeryLargeArray.iMap(someFunction); // Lazy, guaranteed only to be iterable (No apologies to the email protocol.) What do you think? -Michael A. Smith On Fri, Feb 3, 2012 at 3:54 PM, Domenic Denicola dome...@domenicdenicola.com wrote: iterator.map(mapper).some(predicate) // returns an iterator The methods I would like to see are: * every, filter, forEach, map, reduce, reduceRight, some * Optionally: join, toString * Controversially: sorted, reversed (non-mutating versions of sort and reverse) What do you think? -Domenic ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
RE: Array extras functionality for iterators
someVeryLargeArray.iMap(someFunction); // Lazy, guaranteed only to be iterable How about `someVeryLargeArray.asIterator().map(someFunction)`? -Domenic ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
Allen Wirfs-Brock wrote: On Feb 4, 2012, at 12:55 PM, Brendan Eich wrote: The argument is as follows: for (let i = 0, a = some.array, n = a.length; i n; i++) { ... } here we definitely want the a in a.length (n's initializer) to be scoped by the head let -- to be the a declared by the second declarator. Now consider a bit of eta conversion: for (let i = 0, a = some.array, n = (function(){return a})().length; i n; i++) { ... } It would be quite wrong for the a captured by the anonymous function expression to be other than the a binding declared and initialized immediately to the left. Yes, I support the general principal that that initializers of bindings capture to the left. But the problem here is that conceptually the let is defining multiple bindings (one per iteration). I don't think many people are actually going to understanding the details of the proposed semantics and their implications. Since most uses won't involve closure capture, any of the proposed semantics that have per iteration bindings with forward value propagation are just going to do the right thing. That is good. That's all agreed, yes. However, I doubt that someone who actually codes a function in a for(let;;) initializer is going to be thinking, of course, this only captures the first iteration bindings. Why? The initializer runs first. There are two working possibilities: bindings of the same name in an implicit block around the loop (the 0th iteration scope), immediately shadowed on first (if the condition is true) by fresh bindings of the same names with values forwarded; or what I propose, the first iteration's bindings. Why is the 0th iteration scope better than the 1st iteration scope? It may be that some people think of the INIT part in for (INIT; TEST; UPD) STMT as being before the loop starts, but others may see it as part of the first iteration. If we really do have evidence for the 0th iteration scope (hard to imagine), then that is doable but it costs a bit more in the closure-in-initializer case. All else equal I go with lower cost in that rare case. With the TDZ alternative I proposed, there would still be equivalence for: for(let x=x;;)...; and for(let x={|| x}();;)...; Both throw for accessing an initialized variable. Yippee. :-| Making a dynamic error trap for no good reason is a mistake. Where's the evidence that closures in initializers that capture loop control variables are wrong? There isn't any, AFAIK. But you're right that equivalence is lost for for(let x=n, y=x;;)...; and for(let x=n, y={|| x}();;)...; Whether this is better or worse than the wrong capture issue complete depends upon the actual programmer intent. It's worse for the language's consistency of binding rules and equivalences. It's a botch. I think we should not equivocate here. Users expect and even (now that they know, and Dart raises the ante) demand that each iteration gets fresh let bindings. Any who do capture an initial binding in a closure must know, or will learn, that it's just the first one, which fits the model. For the latter, I strongly suspect that they won't know and will be WTF surprised when they encounter it. Why? What binding did the think they captured, given the assumption we share (since you didn't reject it, we do share this, right?) that they know about fresh binding(s) per iteration? Are you implying users will expect the 0th-iteration bindings? Could be, but I rather suspect they'd want 1st. There isn't another choice. They don't want throwing upvars (throwing-up vars :-P). Different how? Different from subsequent iterations... Take Jason's example for (let i = 0; i n; ) { setTimeout(...closure using i...); if (shouldAdvance()) i++; } If somebody decided to abstract the increment: for (let i = 0, advance={|| i++}; i n; ) { setTimeout(...closure using i...); if (shouldAdvance()) advance(); } advance does what is intended if called on the first iteration, but not on subsequent one iterations. I'd soon get an error when I ran this than having it get stuck in a loop. (I s/;/,/ on the for line to fix the typo.) Anyone who writes that must be thinking the advance block-lambda is in the scope of the body, if they understand fresh-binding-per-iteration. But the advance block-lambda is not in the body. It's not before the loop either (wrong [outer] or no i in scope there). This is simply erroneous code by definition -- the assumption of fresh-binding-per-iteration having been learned already is violated by the structure of the code. Rather than a dynamic error (might not test this), anyone writing this might rather get a static error. A warning would be justifed by a high quality implementation, based on analysis of the advance() call possibly coming from later iterations. So here we return to
Re: lexical for-in/for-of loose end
Brendan Eich wrote: I agree we want to capture the first-iteration bindings in any closures in those declarators' initializers. This requires unrolling the loop once. Let's see how the desugaring from: [snip] For what it's worth, I agree with Mark's (2008) and Brendan's desugarings and hereby withdraw my own, as the value-copying covers barely any more use cases. Without some extensive feature like let-aliasing, as Herby suggested, those solutions as good as we're going to get, though I could have missed some of the discussion so far. By the way, if apparent code duplication in Brendan's desugaring is an issue, a small change will fix that, with little cost, like so: $loopEnd: { let $initdone = false; const $loop = { |d1, ... dN| if (!$initdone) { $initdone = true; d1 = e1, ... dN = eN; } if (!cond) break $loopEnd; body; update; $loop(d1, ... dN); } $loop() } As for modifications done by the for-head's closures, I have no strong opinions. In both proposed desugarings, those modifications would be invisible to the loop body fairly quickly. And in all proposals that don't involve some kind of extensive variable aliasing, the for-head's closures cannot observe values set in the loop body (beyond the first iteration). However, I hope I don't delay consensus by raising the possibility of another desugaring using consts: $loopEnd: { const d1 = e1, ... dN = eN; const $loop = { |d1, ... dN| if (!cond) break $loopEnd; body; update; $loop(d1, ... dN); } $loop(d1, ... dN); } That way, the for-head's closures can't easily have hidden unexpected effects, though they'll still observe incorrect iterated values, and it's less of a sledgehammer than disallowing capture altogether. It's late, so I apologise if I've missed something really obvious or typed garbage. Regards, Grant. ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: lexical for-in/for-of loose end
Herby Vojčík wrote: It looks to me that all those schemes and desugaring are very complicated The full problem is complicated, so that's to be expected. What is bad with scenario of reusing the same variable and isolating it as local only for the async closures? Your proposal depends on being able to reassign variable pointers, but they don't necessarily exist. Though I haven't written a JS engine, I believe they are allowed to have variables directly in an activation record (or environment instance) without any (pointer) indirection, so they'd have no mechanism for performing the operation you describe. However, though I haven't read it, I believe that the spec talks a great deal about these activation records and environments, so specifying a mechanism involving those might give you more chance of finding common ground. For what you're talking about, I think this might be an equivalent proposal that's more spec-friendly: 'Note' all closures (dynamically) created in (lexically) the loop initializer. Each time you start an iteration, update all the loop variable activation record pointers within those to point at the current iteration's activation record (which should, with care, have the same shape). Or, here's one that copies the other way (and is probably cleaner): 'Note' all closures (dynamically) created in (lexically, post-desugaring) the loop body. Each time you end an iteration, update all the loop variable activation record pointers to point at a new clone of that activation record. In each case, you require a list of not-necessarily-predictable size to note the closures in. That's not a big problem; it's just something you need to be aware of. Regards, Grant. ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: How to ensure that your script runs first in a webpage
On Sat, Feb 4, 2012 at 1:02 PM, David Bruant bruan...@gmail.com wrote: Le 04/02/2012 18:14, Mark S. Miller a écrit : c) The ES6 module loader should make all these with-games unnecessary anyhow, since it gives us a principled approach for controlling the top level scope of untrusted code. Long term, this is the real answer. Ok, good to know. Can you show how eval(string, lexEnv) example would look like with module loaders? The proposal [1] (and the entire web?) lacks a bit of examples. There's an examples page which includes some module loaders examples (although not this one) [1]. I'm imagining that |lexEnv| is the entire environment that you want available for this evaluation, ie, it shouldn't see any other bindings. For this, we'd do the following: let l = new Loader(system /* the parent loader */, null /* make fresh intrinsics */, lexEnv /* use lexEnv as the global scope */); l.eval(string); Note that this is slightly different than the API currently described on the web page; Dave and I have updates that we need to make there. [1] http://wiki.ecmascript.org/doku.php?id=harmony:modules_examples -- sam th sa...@ccs.neu.edu ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss