-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
On 10/24/2011 11:03 PM, Charles Oliver Nutter wrote:
> Nahi planted an interesting seed on Twitter...what if we could
> parallelize parsing of Ruby files when loading a large
> application?
I once stated this idea for CRuby in [ruby-list:48136], as a possible
feature in a discussion for 'require should take multiple features as
an array'. (The proposal is not mine)
require ['csv', 'json', 'yaml']
I said 'it would make require parallelize' but Matz responded at
[ruby-list:48140]
Would you like to take pains over the problem depending on the
loading timing throughout life? Give me a break!
In addition, you might know he hates autoload [ruby-core:39844]. Sigh,
we don't work very well.
Back to the topic.
> At a naive level, parallelizing the parse of an individual file is
> tricky to impossible; the parser state is very much straight-line.
> But perhaps it's possible to parallelize loading of many files?
>
> I started playing with parallelizing calls to the parser, but that
> doesn't really help anything; every call to the parser blocks
> waiting for it to complete, and the contents are not interpreted
> until after that point. That means that "require" lines remain
> totally opaque, preventing us from proactively starting threaded
> parses of additional files. But there lies the opportunity: what if
> load/require requests were done as Futures, require/load lines were
> eagerly interpreted by submitting load/require requests to a thread
> pool, and child requires could be loading and parsing at the same
> time as the parent file...without conflicting.
I might misunderstanding something. But you're discussing only 'load'
(read in source as a stream) and 'parse' (create AST), not evaluating
it, right? Or does it include evaluation phase?
Say:
a.rb
require 'a/b'
require 'a/c'
a/b.rb
module A
sleep 10
module B
FOO = 1
end
end
a/c.rb
module A
module C
BAR = B::FOO
end
end
How does your Futures work?
Ah, a new mail arrived from Jonathan Coveney. I think I'm saying the
same thing.
Changing LoadService as you stated sounds it includes evaluation
phase. But I'm a little bit far from JRuby source code recently, I
might be wrong.
// NaHi
> In order to do this, I think we would need to make the following
> modifications:
>
> * LoadService would need to explose Future-based versions of
> "load" and "require". The initial file loaded as the "main" script
> would be synchronous, but subsequent requires and loads could be
> shunted to a thread pool. * The parser would need to initiate eager
> load+parser of files encountered in require-like and load-like
> lines. This load+parse would encompass filesystem searching plus
> content parsing, so all the heavy lifting of booting a file would
> be pushed into the thread pool. * Somewhere (perhaps in
> LoadService) we would maintain an LRU cache mapping from file paths
> to ASTs. The cache would contain Futures; getting the actual parsed
> library would then simply be a matter of Future.get, allowing many
> of the load+parses to be done asynchronously.
>
> For a system like Rails, where there might be hundreds of files
> loaded, this could definitely improve startup performance.
>
> Thoughts?
>
> - Charlie
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iQEcBAEBAgAGBQJOpkFjAAoJEC7N6P3yLbI2OEAIAMXfaOM666jS40UFm7uHY8/H
2qpkWp8az9cnt8GvBfyx6d1h9aTeo8TQr/r8mXod/Ahq/WyagbbMIGRALr2AA8xA
sDOdw7X5s1QObF4CtlnQ5fRveOtwvPYy1cNgdSizetb2yx6R3xLbdnSeIYbYwsMQ
p5q3GxSnU4o8qY9hm0IzzU9t6qKuWsRXVgknLdJWfIFfccpe8z3HHDCCDScXC08p
nk70hA3px37tmxJM0GNSSNQs/V9D591wegE/xkO/jaM11mAggQPtHCIUxeSy/ikF
vOJ/9N2c79TZHffXsnfq6yEhyBIbWUudhwBt7exIkbmSpGBRVzlwfh2GPj9D5Ss=
=OdJs
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
