Thanks to everyone who's suggested various solutions for my situation--these ideas have been great...and clever!
Devon McCormick wrote: |Rob Hodgkinson wrote: |> But thinking further, the responses you have received so far |> recalculate the entire array of differences upon each new |> column (ie time slice). This becomes inefficient as the array |> grows (eg if 70,000 values in each column and hundreds of |> time slices then the code recalculates the matrix of |> differences again each time slice...) |> |> If your matrix grows by 1 column (and therefore 70,000 |> numbers) on each time slice, then it is likely more efficient |> to calculate and append only the new differences on each time |> slice. If you don¹t need the whole array of values over all |> time slices, then just keep the last column of values (to be |> used to work out the diffs for the next time slice). ... Rob, do I understand you correctly that I would, in such a scenario, operate on *two* older files: the larger "master" file that I would append the new results column to, and the file containing the actually cumulative data in the previous column? In this way, I'd (1) take the column of previous cumulative values and subtract them from the new column of cumulative values, (2) add this column of results to the previous "master" file, and (3) re-use the column of "new" cumulative values as the column of "previous" cumulative values the next time around. Is that a correct understanding of what you suggested? Sounds good to me! |If you go this direction, it might also help to transpose the |matrix, so you're adding new timeslice differences as rows |(for efficiency). That's essentially how I'm adding the new column each time. |Better yet, pre-allocate the matrix to hold the results and |insert rows (in order to move the memory allocation overhead |up front to a single operation). Devon and Rob, although I understand the concept of preallocating a matrix (which you both mentioned), I'm not sure what the nuts-and-bolts code for "filling" it with the current values would look like. Would some form of "amend" be used, or what? Could someone demonstrate with code? Thanks! Harvey ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
