Thanks Matt! I've just release Vectorz 0.45.0 including your changes. A lot of sparse operations are much faster now!
On Monday, 29 December 2014 21:56:30 UTC+8, Matt Revelle wrote: > > Yes, will do. > > On Dec 28, 2014, at 9:58 PM, Mike Anderson <mike.r.anderson...@gmail.com> > wrote: > > Looks like you have some good changes in your Vectorz branch - any chance > you could tidy up and make a PR? > > I like the idea of specialised getSlices and getColumns in particular - > these should be much faster than getting the slices one-by-one if the data > is very sparse. > > On Monday, 29 December 2014 09:43:54 UTC+8, Matt Revelle wrote: >> >> >> On Dec 28, 2014, at 7:28 PM, Mike Anderson <mike.r.anderson...@gmail.com> >> wrote: >> >> Interesting idea. The challenge is that I'm not sure how to add >> representation specification in an implementation independent way. It's a >> quirk of vectorz that it has both indexed and hashed storage, I probably >> wouldn't expect any other implementations to have that. Likewise row and >> column oriented storage are fairly obvious choices but I still wouldn't >> expect every implementation to support both. >> >> Any idea how you would specify the API? >> >> I guess we could simply pass an optional map argument of options, but >> behaviour would be completely implementation specific. >> >> >> I think the map is the way to go. You’re probably correct about few other >> implementations having as many options, but adding a map of “preferences” >> seems like a good option. Creating a sparse matrix might then look like: >> >> ;; preferences as a separate arg >> (new-sparse-array [100000 100000] :vectorz {:order :row :indexed true}) >> >> ;; an alternative, preferences combined with implementation selection >> (new-sparse-array [100000 100000] {:impl :vectorz :order :row :indexed >> true}) >> >> Implementations should throw an exception if they don’t support (or >> understand) the preferences. >> >> On Monday, 29 December 2014 02:12:05 UTC+8, Matt Revelle wrote: >>> >>> Glad to see the addition of new-sparse-array to core.matrix. It looks >>> like it defaults to SparseRowMatrix for the Vectorz implementation? Should >>> the API provide a way to specify which sparse matrix representation (e.g., >>> row- vs column-based, indexed vs hashed) should be used? I'd suggest a >>> 3-arity new-sparse-array which takes a keyword indicating the >>> representation to use as well as a new function which returns a list of >>> available representations for a specific implementation. >>> >>> I think at this point you incorporated (looks like we have some >>> duplication too, doh) all the changes I had made for sparse matrix support >>> in Vectorz, but will verify. >>> >> >> I definitely haven't covered all the potential code paths - in particular >> a lot of things aren't yet optimised for sparse operations. So any review / >> patches would be appreciated! >> >> >> I did some optimization of sparse ops but the code probably needs to be >> cleaned up before submitting (e.g., generalized and/or moved to the correct >> level in class hierarchy). Those changes were made hastily when I needed to >> quickly get a program running fast. >> >> A branch containing all performance changes based on an older revision >> of the develop branch is available here: >> https://github.com/mattrepl/vectorz/tree/sparse-speed >> >> There is a related sparse-speed branch in my forks of vectorz-clj and >> core.matrix. >> >> We should also look into other sparse array representations for Vectorz >> from: Matlab, MTJ (https://github.com/fommil/matrix-toolkits-java, >> specifically the LinkedSparseMatrix for row and column ops), etc. >> >> -Matt >> >> >> >>> >>> On Saturday, December 27, 2014 4:56:55 AM UTC-5, Mike Anderson wrote: >>>> >>>> Here is a little belated Christmas present for Clojure data aficionados: >>>> >>>> ;; setup >>>> (use 'clojure.core.matrix) >>>> (set-current-implementation :vectorz) >>>> >>>> ;; create a big sparse matrix with a trillion elements (initially zero) >>>> (def A (new-sparse-array [1000000 1000000])) >>>> >>>> ;; we are hopefully smart enough to avoid printing the whole array! >>>> A >>>> => #<SparseRowMatrix Large matrix with shape: [1000000,1000000]> >>>> >>>> ;; mutable setter operations supported so that you can set individual >>>> sparse elements >>>> (dotimes [i 1000] >>>> (mset! A (rand-int 1000000) (rand-int 1000000) (rand-int 100))) >>>> >>>> ;; all standard core.matrix operations supported >>>> (esum A) >>>> => 50479.0 >>>> >>>> ;; efficient addition >>>> (time (add A A)) >>>> => "Elapsed time: 12.849859 msecs" >>>> >>>> ;; matrix multiplication / inner products actually complete in sensible >>>> time >>>> ;; (i.e. much faster than than the usual O(n^3) which might take a few >>>> thousand years) >>>> (time (mmul A (transpose A))) >>>> => "Elapsed time: 2673.085171 msecs" >>>> >>>> >>>> Some nice things to note about the implementation: >>>> - Everything goes through the core.matrix API, so your code won't have >>>> to change to use sparse matrices :-) >>>> - Sparse matrices are 100% interoperable with non-sparse (dense) >>>> matrices >>>> - Sparse arrays are fully mutable. Management of storage / indexing >>>> happens automatically >>>> - It isn't just matrices - you can have sparse vectors, N-dimensional >>>> arrays etc. >>>> - Code is pure JVM - no native dependencies to worry about >>>> >>>> This is all still very much alpha - so any comments / patches / more >>>> rigorous testing much appreciated! >>>> >>>> >>>> >>>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Numerical Clojure" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/numerical-clojure/LLpq4WHx-k8/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> numerical-clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Numerical Clojure" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/numerical-clojure/LLpq4WHx-k8/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > numerical-clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.