Thanks! The primary work is either quick previewing of or format translation of large numbers of very large files. We want reasonable random access or sequential access speed to what may be terabytes of data. Cleverness with block buffering, i/o, compression, etc. are all games for another part of the system and so I’m trying to just use a terrifically naive mmap strategy. It’s worked great with one file, and now I’d like to see if I can export a 2D array representing the data in lots of files. A few hours of mmap trickery gets me a long way to being able to write boring, understandable code against this data while I wrangle it into a more suitable form. On April 9, 2015 at 4:38:16 AM, Tim Holy (tim.h...@gmail.com) wrote:
https://github.com/tanmaykm/ChainedVectors.jl --Tim On Wednesday, April 08, 2015 09:24:03 PM Kevin Squire wrote: > AFAIK, there's nothing really like that right now, but what do you plan to > do with the data? Most linear algebra code, for example, calls out to > BLAS, which requires data to be contiguous (or at least strided) in > memory. Other code may or may not have this same restriction. > > It should be relatively easy to write an `VCat` type which wraps a variable > sized list of arrays and at least lets you index them as if they were one > array. Making it efficient would probably be more challenging, and making > it an actual AbstractArray might be tedious, but should be doable. > > Cheers! > Kevin > > On Wed, Apr 8, 2015 at 8:56 PM, Sebastian Good < > > sebast...@palladiumconsulting.com> wrote: > > I can use sub to pretend a larger array is a smaller, or differently > > shaped one. Is there functionality to allow me to treat several smaller > > arrays as a larger one without copying them? In effect, mimicking hcat > > and friends, but without copying data.
