On Wed, Feb 23, 2011 at 11:34 AM, William Stein <wst...@gmail.com> wrote: > On Wed, Feb 23, 2011 at 10:57 AM, Jason Grout > <jason-s...@creativetrax.com> wrote: >> On 2/23/11 12:28 PM, William Stein wrote: >>> >>> At lunch yesterday Robert Bradshaw made the interesting suggestion to >>> read the docs for importlib >>> (http://docs.python.org/dev/library/importlib.html) and write a >>> customized import hook, so that every time during Sage startup that a >>> module is imported, the import is done from a single big in-memory zip >>> file instead of done using the filesystem. If this can be made to >>> work, it would be a huge win for slow filesystems. The basic problem >>> is that some filesystems are fast but have huge*latency*. >> >> Is it a big win primarily because the zip file contents can be read in and >> cached by us? I'm just trying to understand it better. > > Which would you rather do on a high latency filesystem: > > (1) Read/stat 20,000 little files, or > (2) Read exactly one 40MB file. > >> Is this the same idea as Jar files in java? > > I don't know.
Yep. In that case the "high latency file system" was a webserver. >> You mean like http://docs.python.org/library/zipimport.html ? > > Cool. Note that this should just involve putting the zip file first in the python path. > I don't know for a fact that Robert Bradshaw's suggestion will be a > big win, since nobody has tried this yet. But I'm optimistic. The > idea would be to make a zip archive of > $SAGE_ROOT/local/lib/python/site-packages (say), and do *all* imports > using that massive zip archive. I'm optimistic too. This would, of course, make more sense for system-wide installs than development versions, but the former are more likely to be on a non-local filesystem anyways. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
