You could create a "phony" 2-dimensional array that computes the distances on the fly… but you won't be able to pass this matrix to, e.g., BLAS.
immutable DistanceMatrix <: AbstractArray{Float64, 2} locs::Array{Float64, 2} # a 2xN or 3xN matrix containing the location coordinates end Base.size(A::DistanceMatrix) = (length(A.locs), length(A.locs)) Base.getindex(A::DistanceMatrix, i::Int, j::Int) = dist(A.locs[:, i], A.locs [:, j]) # could be further optimized (Untested) On Friday, August 14, 2015 at 11:03:06 AM UTC-4, Stefan Karpinski wrote: > > On Friday, August 14, 2015, Charles Novaes de Santana < > charles...@gmail.com <javascript:>> wrote: > >> >> 1) to use only the subset of suitable habitats to build the matrix of >> distances (and then to use sparse matrix as suggested by Stefan) >> > > Distance matrices are not usually sparse – since the farthest apart pairs > of points have large distances and are the most common and least > interesting. However, you could store only distances for close points in a > sparse matrix and use zero to represent the distance between pairs of > points that are not close enough to be of interest. Either that or you > could store 1/d instead of d and then closer points have higher weights and > you can threshold 1/distance so that far apart points have zero entries. > > >> 2) to use a machine with more memory and try to run my models using the >> matrices with all the sites >> > > This is probably the easiest thing to do since your data set is not of a > truly unreasonable size, just largish. However, you may be much happier if > you can make your problem smaller than O(n^2). > > >> 3) to try another language/library that might work better with such big >> amount of data (like python, or R). >> > > This problem isn't going to be fundamentally different no matter what > language you use: you have more data than fits in memory. Spilling memory > to disk is going to be *much* slower than just recomputing distances – > orders of magnitude slower. As John suggested, is there any particular > reason you need to materialize all of these values in a matrix? What > computation are you going to perform over that matrix? >