Re: Simple Modules: lazy dependency evaluation

2011-01-27 Thread Wes Garland 
On Wed, Jan 26, 2011 at 5:04 PM, Brendan Eich bren...@mozilla.com wrote:

 CommonJS may do that on the server side, assuming fast enough file i/o.
 It's not necessarily a good idea even there (Ryan Dahl has talked about
 this). On the client, it's right out, which is why client-side CommonJS-like
 module systems require a preprocessor or else a callback.


There is work under way at CommonJS to rectify this problem. A few different
proposals have emerged, which all have approximately the same theme:

   -  Explicitly decouple exports-lookup and module loading.
   -  require() remains the exports-lookup interface
   -  Introduce a way to know which modules are needed by a program, such as
   the explicit declaration of dependencies, or static analysis of the source
   code looking for require() statements.
   - Before the main module is executed, load all dependencies (recursively)

Adding a mandatory function wrapper to the module source code also allows
these modules to be loaded and executed directly by DOM script tag injection
(an important technique as XHR has cross-domain restrictions); the function
wrapper also provides a convenient place to hang dependencies.

Here is what a module in one proposal (Modules/2.0-draft7) looks like:

module.declare([*lib/dialog*], function(require, exports, module) {
  /* Any valid Modules/1.1.1 modules goes here */
  require(*lib/dialog*).notify(hello, world);
})

(Oh -- main-modules are modules which are executed automatically by the
CommonJS host environment; e.g. by exec(3) shebang, HTML script tag, or
other mechanism; the mechanism is not part of the specification)

I should also say that all the proposals also have a way to explicitly load
a particular module at run-time, rather than via the dependency graph.  The
interface specifies a module name (or names) and a callback. Once the module
and its dependent modules are loaded, the callback is executed. This lets us
do lazy-loading, like Simple Modules loaders, without breaking
run-to-completion.



On Wed, Jan 26, 2011 at 6:25 PM, Kam Kasravi kamkasr...@yahoo.com wrote:

 Are you guys following modules 2.0 at all that seems to be a parallel
 universe of sorts under co mmonjs?


Full disclosure - I am the principle author of that document. It is one of
the proposals mentioned above. Its current status is pompously-named
document designed to get attention - it is not a standard, and carries only
the weight of the paper it is printed on. FWIW, it discusses much more than
the CommonJS module system -- it also attempts to nail down the execution
environment.  That said, there is no way Modules/2.0 belongs under
consideration by TC39; it is a best-effort with limited tools proposal;
Simple Modules gets to use new tools.



On Wed, Jan 26, 2011 at 5:40 PM, David Herman dher...@mozilla.com wrote:

 Just to flesh this out a bit: simple modules were designed to make it
 possible to import and export variables into lexical scope, and to be
 compatible with checking valid imports and exports statically, as well as
 being able to check for unbound variables statically. Including, for
 example, in the case where you say

import M.*;

 Moreover, they are designed to allow loading modules *without* having to
 use callbacks, in the common case where they can be pre-loaded at
 compile-time.


James' query comes as attempt to understand design decisions made for Simple
Modules with respect to the timing of the evaluation of the module factory
function.  With loading and exports now decoupled, the question becomes --
when does require actually evaluate the module body?  The balancing point
seems to be Is it worth breaking backwards compatibility on existing
platforms in order to try and mimic Simple Modules?.

Breaking backwards compatibility, in this case, means evaluating the
factories eagerly, as they are loaded into the environment as the dependency
tree is satisfied (before the main module runs). Currently, factories are
executed as side-effects of the first require() call (per module - our
modules are singletons).  This timing is important, as factories have
observable side effects (consider the module above).

So, what we're talking about is not just the Simple Modules loader, but the
static declaration form as well.  Static declaration is analogous to loading
a list of modules which is comprised of the main module and its recursive
dependencies and then executing the main module.

FWIW - my take on this is that porting CommonJS to Simple Modules is going
to require a code-audit anyhow; I believe that breaking backwards
compatibility within the CommonJS family is not worth trying to mimic such a
small sliver of Simple Modules semantics. There are much larger mismatches
(singletons, module-keyword, lexical scope, require() to name just a few)
which make it a moot point IMO.

Kris Kowal's query is interesting: is lazy evaluation worth considering for
Simple Modules?

   module M {
   export var foo = 42;
   export

Re: Simple Modules: lazy dependency evaluation

2011-01-27 Thread David Herman 
On Jan 27, 2011, at 8:38 AM, Wes Garland wrote:

 Kris Kowal's query is interesting: is lazy evaluation worth considering for 
 Simple Modules?
 
module M {
export var foo = 42;
export function bar() { return foo; }
alert(hello, world);
}
 
 In the example above, the alert statement would occur when the first import 
 from M statement were executed, rather than when the page containing the 
 SCRIPT type=es-next were loaded.

Believe me, I have considered it. :) We thought for a while about demand-driven 
evaluation of modules. There are a couple reasons why I believe it would be too 
problematic. First, we'd really like to make the act of throwing your code into 
a module as transparent as possible; changing the control flow would make 
modules more heavyweight. But more importantly, since as you mentioned, module 
evaluation can contain arbitrary side effects, evaluating them lazily means 
laziness with side effects. This makes for really hard-to-understand and 
hard-to-debug initialization errors, where you end up having to write 
mysterious top-level imports to force evaluation of modules in particular 
orders. Laziness + side-effects: bad scene, man.

Dave

___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-27 Thread Kris Kowal 
On Thu, Jan 27, 2011 at 7:27 AM, David Herman dher...@mozilla.com wrote:
 We thought for a while about
 demand-driven evaluation of modules. There are a couple reasons why I
 believe it would be too problematic. First, we'd really like to make the act
 of throwing your code into a module as transparent as possible; changing the
 control flow would make modules more heavyweight. But more importantly,
 since as you mentioned, module evaluation can contain arbitrary side
 effects, evaluating them lazily means laziness with side effects. This makes
 for really hard-to-understand and hard-to-debug initialization errors, where
 you end up having to write mysterious top-level imports to force evaluation
 of modules in particular orders. Laziness + side-effects: bad scene, man.

On the opposite side of the argument, I presume that this means that
modules are evaluated when their transitive dependencies are loaded.
This would imply that the order in which the modules are delivered,
possibly over a network using multiple connections, would determine
the execution order, which would in turn be non-deterministic.
Non-determinisim + side-effects is also a bad scene.

Is there an alternate method proposed in Simple Modules for
deterministically linearizing the evaluation order? Non-determinism is
definitely a greater evil than providing developers a means to
explicate the order in which they would like their side-effects to be
wrought.

Kris Kowal
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-27 Thread David Herman 
 On the opposite side of the argument, I presume that this means that
 modules are evaluated when their transitive dependencies are loaded.
 This would imply that the order in which the modules are delivered,
 possibly over a network using multiple connections, would determine
 the execution order, which would in turn be non-deterministic.

No, that's not the case. At compile-time, the compiler may load the *files* 
non-deterministically, but it is required to evaluate them in their declared 
order, deterministically.

 Non-determinisim + side-effects is also a bad scene.

Indeed.

Dave

___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-27 Thread Kris Kowal 
On Thu, Jan 27, 2011 at 9:14 AM, David Herman dher...@mozilla.com wrote:
 …but it is required to evaluate them in their declared order, 
 deterministically.

Would you explain how declaration order is inferred from the contents
of the unordered of files?

It's clear that the order is at least partially knowable through the
order of module declarations within a single file, and that load
directives would be replaced with a nest of modules, which is similar
in effect to loading on demand if the load directive is considered a
point of demand at run-time. And we're guaranteed that there are no
files that would be loaded that are not reachable through transitive
load directives. I suppose I've answered my question, if all my
assumptions are correct.

Kris Kowal
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-27 Thread David Herman 
The easiest way to think about it is to imagine the fully loaded bits of the 
whole program as if they were just declared inline in one big file. Then the 
total order is manifest -- it's just the order they appear in the program. The 
non-deterministic I/O performed by the compiler is just an internal detail of 
the engine's implementation that doesn't leak into the semantics.

Or here's a sort of more operational way to think about it: Start with the 
outermost program. It declares a bunch of modules, some of which are loaded 
from external files. But the declaration order of this first level of 
sub-modules is manifestly ordered in the program. Now maybe the compiler loads 
those files in parallel or out of order, but ultimately it has all the bits. 
Within each loaded file, the order of modules is (recursively now) itself 
totally ordered.

Dave

On Jan 27, 2011, at 11:30 AM, Kris Kowal wrote:

 On Thu, Jan 27, 2011 at 9:14 AM, David Herman dher...@mozilla.com wrote:
 …but it is required to evaluate them in their declared order, 
 deterministically.
 
 Would you explain how declaration order is inferred from the contents
 of the unordered of files?
 
 It's clear that the order is at least partially knowable through the
 order of module declarations within a single file, and that load
 directives would be replaced with a nest of modules, which is similar
 in effect to loading on demand if the load directive is considered a
 point of demand at run-time. And we're guaranteed that there are no
 files that would be loaded that are not reachable through transitive
 load directives. I suppose I've answered my question, if all my
 assumptions are correct.
 
 Kris Kowal

___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-26 Thread Sam Tobin-Hochstadt 
On Wed, Jan 26, 2011 at 2:04 PM, James Burke jrbu...@gmail.com wrote:
 CommonJS Modules 1.1 allows this kind of construct in module code:

 var a;
 if (someCondition) {
    a = require(a1);
 } else {
    a = require(a2);
 }

 and the module a1 is not actually evaluated until execution reaches
 the a = require(a1) call.

 1) Could something like this work in Simple Modules? If so, what would
 be the syntax for it?

You can use module loaders to do exactly this (I believe, based on my
understanding of CommonJS).  It would look like:

var ml = ... the desired module loader ...
var a;
if (someCondition) {
  a = ml.load(a1);
} else {
  a = ml.load(a2);
}

This produces a module instance object, bound to |a|.  It doesn't,
however, allow you to import from |a|, since nothing is statically
known about what the exports of |a| are.

 2) What are the design decisions behind only allowing module and
 use at the top level of a module?

Modules are a statically scoped construct, and the implementation and
the programmer can both statically tell where variables come from.
This prevents modules from being dynamically created, except in the
context of module loaders, as seen above.
-- 
sam th
sa...@ccs.neu.edu
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-26 Thread Kam Kasravi 
I assume you mean m.load(la1).
So what is the behavior if you do new m.load(la1).Foo() if you know Foo is an 
exported object? Is there a module API that allows one to introspect a modules 
content?



On Jan 26, 2011, at 1:46 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote:

 On Wed, Jan 26, 2011 at 2:04 PM, James Burke jrbu...@gmail.com wrote:
 CommonJS Modules 1.1 allows this kind of construct in module code:
 
 var a;
 if (someCondition) {
a = require(a1);
 } else {
a = require(a2);
 }
 
 and the module a1 is not actually evaluated until execution reaches
 the a = require(a1) call.
 
 1) Could something like this work in Simple Modules? If so, what would
 be the syntax for it?
 
 You can use module loaders to do exactly this (I believe, based on my
 understanding of CommonJS).  It would look like:
 
 var ml = ... the desired module loader ...
 var a;
 if (someCondition) {
  a = ml.load(a1);
 } else {
  a = ml.load(a2);
 }
 
 This produces a module instance object, bound to |a|.  It doesn't,
 however, allow you to import from |a|, since nothing is statically
 known about what the exports of |a| are.
 
 2) What are the design decisions behind only allowing module and
 use at the top level of a module?
 
 Modules are a statically scoped construct, and the implementation and
 the programmer can both statically tell where variables come from.
 This prevents modules from being dynamically created, except in the
 context of module loaders, as seen above.
 -- 
 sam th
 sa...@ccs.neu.edu
 ___
 es-discuss mailing list
 es-discuss@mozilla.org
 https://mail.mozilla.org/listinfo/es-discuss
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-26 Thread Brendan Eich 
On Jan 26, 2011, at 1:54 PM, Kam Kasravi wrote:

 I assume you mean m.load(la1).

What is m?

Where did la1 come from?

Confusion.


 So what is the behavior if you do new m.load(la1).Foo() if you know Foo is 
 an exported object?

Sam addressed that directly (nothing is statically known), cited in full 
below.

Also, you can't violate JS's run to completion execution model by nesting an 
event loop and blocking on that load until the module, which could be on a far 
away server or even a too-slow-to-block-on disk filesystem, finishes loading.

CommonJS may do that on the server side, assuming fast enough file i/o. It's 
not necessarily a good idea even there (Ryan Dahl has talked about this). On 
the client, it's right out, which is why client-side CommonJS-like module 
systems require a preprocessor or else a callback.


 Is there a module API that allows one to introspect a modules content?

Are you reading the wiki?

/be

 
 
 
 On Jan 26, 2011, at 1:46 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote:
 
 On Wed, Jan 26, 2011 at 2:04 PM, James Burke jrbu...@gmail.com wrote:
 CommonJS Modules 1.1 allows this kind of construct in module code:
 
 var a;
 if (someCondition) {
   a = require(a1);
 } else {
   a = require(a2);
 }
 
 and the module a1 is not actually evaluated until execution reaches
 the a = require(a1) call.
 
 1) Could something like this work in Simple Modules? If so, what would
 be the syntax for it?
 
 You can use module loaders to do exactly this (I believe, based on my
 understanding of CommonJS).  It would look like:
 
 var ml = ... the desired module loader ...
 var a;
 if (someCondition) {
 a = ml.load(a1);
 } else {
 a = ml.load(a2);
 }
 
 This produces a module instance object, bound to |a|.  It doesn't,
 however, allow you to import from |a|, since nothing is statically
 known about what the exports of |a| are.
 
 2) What are the design decisions behind only allowing module and
 use at the top level of a module?
 
 Modules are a statically scoped construct, and the implementation and
 the programmer can both statically tell where variables come from.
 This prevents modules from being dynamically created, except in the
 context of module loaders, as seen above.
 -- 
 sam th
 sa...@ccs.neu.edu
 ___
 es-discuss mailing list
 es-discuss@mozilla.org
 https://mail.mozilla.org/listinfo/es-discuss
 ___
 es-discuss mailing list
 es-discuss@mozilla.org
 https://mail.mozilla.org/listinfo/es-discuss

___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-26 Thread David Herman 
 You can use module loaders to do exactly this (I believe, based on my
 understanding of CommonJS).  It would look like:
 
 var ml = ... the desired module loader ...
 var a;
 if (someCondition) {
  a = ml.load(a1);
 } else {
  a = ml.load(a2);
 }

Correction: you have to use callbacks for dynamic loading:

var ml = ... the desired module loader ...
if (someCondition) {
ml.load(a1, useA);
} else {
ml.load(a2, useA);
}
function useA(a) {
...
}

Callbacks are the downside of dynamic loading, but that's the nature of 
asynchronous I/O in ES -- it's a fact of life.

However, I would submit that at least in some cases you don't necessarily need 
conditional loading. For example, if you'd like to choose between two different 
exports from two different modules, you can load them both and conditionally 
refer to one or the other:

module a1 = a1;
module a2 = a2;
var x;
if (someCondition) {
x = a1.foo;
} else {
x = a2.bar;
}

Obviously that doesn't cover all situations, but the design of simple modules 
attempts to achieve a balance between the following:

- making it easy to load modules statically
- making modules compatible with static scoping
- making it possible (though admittedly less convenient) to load modules 
dynamically

 2) What are the design decisions behind only allowing module and
 use at the top level of a module?
 
 Modules are a statically scoped construct, and the implementation and
 the programmer can both statically tell where variables come from.
 This prevents modules from being dynamically created, except in the
 context of module loaders, as seen above.

Just to flesh this out a bit: simple modules were designed to make it possible 
to import and export variables into lexical scope, and to be compatible with 
checking valid imports and exports statically, as well as being able to check 
for unbound variables statically. Including, for example, in the case where you 
say

import M.*;

Moreover, they are designed to allow loading modules *without* having to use 
callbacks, in the common case where they can be pre-loaded at compile-time.

Now, we could've gone further by adding a compile-time sub-language for doing 
things like conditional loading, but we wanted to avoid the complexity of 
bifurcating ES into a compile-time language separate from the run-time 
language. If you want more complicated logic, our design requires you to do it 
dynamically. Again, it's a trade-off: we could've made a more expressive static 
module language, but I'm skeptical that its utility would be worth the 
complexity it would introduce.

Dave

___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-26 Thread David Herman 
On Jan 26, 2011, at 4:04 PM, Brendan Eich wrote:

 On Jan 26, 2011, at 1:54 PM, Kam Kasravi wrote:
 
 So what is the behavior if you do new m.load(la1).Foo() if you know Foo is 
 an exported object?
 
 Sam addressed that directly (nothing is statically known), cited in full 
 below.

Oh, I understand this question better. To clarify a bit further: the module 
loading API returns the first-class module instance objects, rather than 
statically known modules. The code would actually look like this:

ml.load(a1, function(m) {
// in here, m is just an ordinary variable
// bound to a first-class object that reflects
// the contents of the module loaded from a1
...
});

 Is there a module API that allows one to introspect a modules content?
 
 Are you reading the wiki?

There are explanations and examples of first-class module instance objects on 
the wiki, but to be fair some of the details are a bit out of date (in 
particular the scoping semantics has changed a bit). Updating the wiki is 
towards the top of my to-do list.

Dave

___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-26 Thread Kam Kasravi 
I understand why callbacks are required for module loaders given they fetch 
asynchronously, though it's too bad this isn't more tightly integrated with an 
eventing model where events could be published when modules are loaded. I 
imagine TC39 has no interest in broaching an event system per se, but IMHO 
there would be rapid adoption and tighter integration if module loading could 
have associated events like DOMReady etc. A different layer and a different 
committee no doubt.

Sent from my iPad

On Jan 26, 2011, at 2:50 PM, David Herman dher...@mozilla.com wrote:

 On Jan 26, 2011, at 4:04 PM, Brendan Eich wrote:
 
 On Jan 26, 2011, at 1:54 PM, Kam Kasravi wrote:
 
 So what is the behavior if you do new m.load(la1).Foo() if you know Foo 
 is an exported object?
 
 Sam addressed that directly (nothing is statically known), cited in full 
 below.
 
 Oh, I understand this question better. To clarify a bit further: the module 
 loading API returns the first-class module instance objects, rather than 
 statically known modules. The code would actually look like this:
 
ml.load(a1, function(m) {
// in here, m is just an ordinary variable
// bound to a first-class object that reflects
// the contents of the module loaded from a1
...
});
 
 Is there a module API that allows one to introspect a modules content?
 
 Are you reading the wiki?
 
 There are explanations and examples of first-class module instance objects on 
 the wiki, but to be fair some of the details are a bit out of date (in 
 particular the scoping semantics has changed a bit). Updating the wiki is 
 towards the top of my to-do list.
 
 Dave
 
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss

Re: Simple Modules: lazy dependency evaluation

2011-01-26 Thread Brendan Eich 
On Jan 26, 2011, at 3:25 PM, Kam Kasravi wrote:

 Are you guys following modules 2.0 at all that seems to be a parallel 
 universe of sorts under co mmonjs? Seems like it should make the strawman in 
 some form  

I'm reading CommonJS  when I have time.

This point has been made before: CommonJS folks have done heroic work building 
something on top of the language as it is. That's not good enough for the 
future, where we can do better by adding special forms that cannot be 
implemented using functions.

And note Kris's reply: a preprocessor to prefetch is required, or else (a la 
RequireJS) require takes a callback. This is all ok and people are using such 
things today. There's no urgent need to standardize this kind of library code 
-- and we are not going to standardize a preprocessor.

Pragmatic JS hackers have to play the game on the field of JS as it is 
implemented today, however tilted and bumpy. Simple modules aim to groom the 
playing field and move the goal posts by improving the core language.

/be

___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss