Re: math utilities question
Standard way, definitely no. As others have pointed out, Incanter is The Clojure Math Tool, but strongly biased towards statistics and linear algebra, and outside those fields you'll need other tools. Apache Commons Math (http://commons.apache.org/math/) is one of the better self-contained Java math libraries. I think it can do at least 4 (in one dimension), 5, and 7 (e.g. Levenberg-Marquardt). Another tool worth mentioning is jHepWork (http://jwork.org/ jhepwork/). It contains an impressive collection of various numerical Java libraries glued together, using Jython as its scripting language. Probably it wouldn't be too hard to glue Clojure into it, but how idiomatic it would be is another question... Best Regards, Joonas On Dec 6, 1:27 am, Robert McIntyre wrote: > I'm trying to use clojure for scientific data analysis but I keep > running into lacunas of functionality. > > I'd love to hear the community's recommendations and experiences with this: > > Is there a standard way to do things like: > 1. take the convolution of two vectors > 2. work with imaginary numbers, quaternions, octonions, etc > 3. work with matrices of arbitrary dimension > 4. Fourier transform ( in multiple dimensions) > 5. integration / finite difference > 6. symbolic manipulation as in sage > 7. minimizing non-linear functions > 8. finding zeros of non-linear functions > > thank you all in advance for any recommendations you might have. > > --Robert McIntyre -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
(defmacro defchunk [name tps] `(def ~name (quote ~tps))) (defmacro let-chunk [vname name val-vec & body] (let [chunk-def @(resolve name) types (map first chunk-def) part-names (map (comp symbol (partial str vname "!") second) chunk-def)] `(let [~vname ~val-vec ~@(interleave part-names (map #(list %1 (list vname %2)) types (iterate inc 0)))] ~...@body))) user=> (defchunk foo [[int x][double y]]) #'user/foo user=> (let-chunk afoo foo [1 1.7] afoo) [1 1.7] user=> (let-chunk afoo foo [1 1.7] afoo!x) 1 user=> (let-chunk afoo foo [1 1.7] afoo!y) 1.7 Simple enough, and afoo!x and afoo!y are primitive. (defchunk complex [[double real][double imag]]) (defn cx-mult [w z] (let-chunk w complex w (let-chunk z complex z [(- (* w!real z!real) (* w!imag z!imag)) (+ (* w!imag z!real) (* w!real z!imag))]))) (after a few runs to get it all JITted) user=> (time (nth (iterate #(cx-mult % %) [0.0 1.0]) 1)) "Elapsed time: 14.12232 msecs" [1.0 -0.0] Looks like one iteration taking less than 2 microseconds to me. And that's with iterate. Loop gives this: user=> (time (loop [c [0.0 1.0] n 1] (if (= n 0) c (recur (cx-mult c c) (dec n) "Elapsed time: 2.54112 msecs" [1.0 -0.0] Interestingly, using definline to define cx-mult slows this down instead of speeding it up. Another macro could give you a super-defn that let-chunks certain function parameters, so the (let-chunk foo bar foo ...) stuff wrapping the body of cx-mult disappears into a macro. Yet another could give you a loop-chunk. For truly high performance, though, the vector boxing is a problem. JIT doesn't seem to be eliminating it, or the speed would be up to 1000x faster still. Defstruct/defrecord is probably what's needed here; this would mean a) modifying defchunk to emit a suitable defstruct/defrecord in addition to the type-partname pairs structure and b) making the other macros destructure these instead of vectors. But the above shows a working, if crude, first pass at implementing the facility under discussion in this thread. Replacing vectors with structmaps/records in it will probably produce a large speedup when the things are passed across function boundaries. The remaining bugbear will be that the let-chunk and loop-chunk will create a heap allocated structmap/record even if it's unused. Having the macros detect if it's unused might not be easy. I'm unsure, though, that heap allocation is as horrid as everyone here seems to be assuming it to be. It may be horribly slow in FORTRAN or even in C but modern JVMs make heap allocation very cheap. I wouldn't be surprised if using heap-allocated structmaps or records directly will do fine. There are also supposed to be more optimizations for primitive use, including passing across function boundaries, in the forthcoming Clojure 1.3. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On Tue, Dec 7, 2010 at 12:58 AM, Konrad Hinsen wrote: > On 06.12.2010, at 22:35, Ken Wesson wrote: > >>> I'd say what Java needs is not complex numbers as a value type, but a way >>> to define additional value types. Complex numbers are just one >>> applications. Another one is points (2D or 3D) for geometry and graphics. >>> >>> Unfortunately the problem is not just Java, but the JVM, meaning that >>> Clojure inherits the problem. >> >> Clojure does not inherit the problem, if you use macros cleverly. You >> can stack allocate individual primitives, separately, e.g. >> >> (let [x (int 3) y (double 4.2)] >> ...) >> >> And with macros you can wrap that in an abstraction that looks like a >> single object, such as a point or a complex number. Passing it to a >> function would require a bit of magic, though -- say, bundling the > > That's exactly the problem. An abstraction that doesn't pass a function > boundary is of little use in a functional language. Decomposing complex > numbers or points into primitives can be used as a local optimization > technique, but not in a larger-scale design for software systems. First of all, the abstraction does pass the function boundary if the function's a closure and you get it from the lexical scope rather than passing it as an argument. Second, macros can probably be used to make passing them as arguments and returning them transparent to the coder, in a way that boils away to nothing when the -server JIT goes to work on it. I can try to code a rough draft of such a facility and post it later. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On 06.12.2010, at 22:35, Ken Wesson wrote: >> I'd say what Java needs is not complex numbers as a value type, but a way to >> define additional value types. Complex numbers are just one applications. >> Another one is points (2D or 3D) for geometry and graphics. >> >> Unfortunately the problem is not just Java, but the JVM, meaning that >> Clojure inherits the problem. > > Clojure does not inherit the problem, if you use macros cleverly. You > can stack allocate individual primitives, separately, e.g. > > (let [x (int 3) y (double 4.2)] > ...) > > And with macros you can wrap that in an abstraction that looks like a > single object, such as a point or a complex number. Passing it to a > function would require a bit of magic, though -- say, bundling the That's exactly the problem. An abstraction that doesn't pass a function boundary is of little use in a functional language. Decomposing complex numbers or points into primitives can be used as a local optimization technique, but not in a larger-scale design for software systems. Konrad. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On Mon, Dec 6, 2010 at 11:45 AM, Konrad Hinsen wrote: > On 06.12.2010, at 16:02, Johann Hibschman wrote: >> Maybe I'm not reading the right news, but I've not seen all that much on >> using Java for scientific work for a while now. The NIST JavaNumerics >> guys seem to have given up, but if I remember correctly their >> conclusions were that Java really needed complex numbers as a >> value/stack-allocated type. > > I'd say what Java needs is not complex numbers as a value type, but a way to > define additional value types. Complex numbers are just one applications. > Another one is points (2D or 3D) for geometry and graphics. > > Unfortunately the problem is not just Java, but the JVM, meaning that Clojure > inherits the problem. Clojure does not inherit the problem, if you use macros cleverly. You can stack allocate individual primitives, separately, e.g. (let [x (int 3) y (double 4.2)] ...) And with macros you can wrap that in an abstraction that looks like a single object, such as a point or a complex number. Passing it to a function would require a bit of magic, though -- say, bundling the components into a vector and passing that as the fn arg, and unbundling again on the inside. If the call gets inlined the JIT should hopefully be able to optimize away this boxing. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On 06.12.2010, at 16:02, Johann Hibschman wrote: > (Konrad Hinsen had started some work on multiarrays in Clojure, but I've > not been following his progress.) There hasn't been much, unfortunately. I haven't found much time for serious Clojure hacking for a few months. But the project is not abandoned, just slowed down. > I've built the Java interface to HDF5, and I've been using that for > data storage. I would prefer to use a pure-Java solution, but I can't > find anything that's nearly as good. netCDF has a Java library that also reads HDF5, but it's read-only. > Maybe I'm not reading the right news, but I've not seen all that much on > using Java for scientific work for a while now. The NIST JavaNumerics > guys seem to have given up, but if I remember correctly their > conclusions were that Java really needed complex numbers as a > value/stack-allocated type. I'd say what Java needs is not complex numbers as a value type, but a way to define additional value types. Complex numbers are just one applications. Another one is points (2D or 3D) for geometry and graphics. Unfortunately the problem is not just Java, but the JVM, meaning that Clojure inherits the problem. Konrad. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
Robert McIntyre writes: > I'm wondering if people have had experience with java libraries of > that sort and might have some recommendations. > > Anyone use clojure for scientific data analysis? What do you find > helpful to use? I'm still just evaluating clojure for scientific data analysis, but I can share what I've found so far. First of all, Incanter. I like the idea of Incanter, but I don't like its decision to have matrices be the fundamental data object. Matrices are great, but they not the be-all and end-all. Multidimensional arrays are better, like in numpy or APL or J. It's a pet peeve about R that it doesn't distinguish scalars from vectors of length 1. (Konrad Hinsen had started some work on multiarrays in Clojure, but I've not been following his progress.) Also, Incanter seems very tuned to a row-wise view of data sets, while I've spent enough time with R and kdb+/q to prefer a column-wise view of data. (This is just based on reading the Incanter docs quickly; I may be misrepresenting the package.) As far as matrix libraries go, I've settled on EJML, since it seems reasonably fast, and I can understand what it's doing. Bradford Cross blogged a comparison of different libraries at: http://measuringmeasures.com/blog/2010/3/28/matrix-benchmarks-fast-linear-algebra-on-the-jvm.html I can't seem to find a good Java multiarray library, but I have some hope that I could beat EJML into shape, since its representation is just a basic array of doubles. I've built the Java interface to HDF5, and I've been using that for data storage. I would prefer to use a pure-Java solution, but I can't find anything that's nearly as good. Maybe I'm not reading the right news, but I've not seen all that much on using Java for scientific work for a while now. The NIST JavaNumerics guys seem to have given up, but if I remember correctly their conclusions were that Java really needed complex numbers as a value/stack-allocated type. This is a bit of a disjointed ramble, but I'd love to hear what you settle on. Regards, Johann -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
I have looked at incanter and like it very much, but these are all things that incanter can't currently do. --Robert McIntyre On Mon, Dec 6, 2010 at 3:15 AM, Saul Hazledine wrote: > On Dec 6, 12:27 am, Robert McIntyre wrote: >> I'm trying to use clojure for scientific data analysis but I keep >> running into lacunas of functionality. >> >> 6. symbolic manipulation as in sage > > This is something that would be awesome to have in Clojure because, > unlike most non-lisps, you can compile the result and use it. This > makes : > >> 7. minimizing non-linear functions > > much easier. > > Saul > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On Dec 6, 12:27 am, Robert McIntyre wrote: > I'm trying to use clojure for scientific data analysis but I keep > running into lacunas of functionality. > > 6. symbolic manipulation as in sage This is something that would be awesome to have in Clojure because, unlike most non-lisps, you can compile the result and use it. This makes : > 7. minimizing non-linear functions much easier. Saul -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On Mon, Dec 6, 2010 at 2:59 AM, Benny Tsai wrote: > Always nice to see a fellow Neal Stephenson fan! > > On Dec 5, 10:26 pm, Ken Wesson wrote: >> On Mon, Dec 6, 2010 at 12:14 AM, Miki wrote: >> > Have you looked at Incanter? (http://incanter.org/) >> >> Hmm, interesting. Is there a Rhetor too? Wow, that didn't take long. Less than three hours! Obviously it's true what they say about Lisp hackers. :) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
Always nice to see a fellow Neal Stephenson fan! On Dec 5, 10:26 pm, Ken Wesson wrote: > On Mon, Dec 6, 2010 at 12:14 AM, Miki wrote: > > Have you looked at Incanter? (http://incanter.org/) > > Hmm, interesting. Is there a Rhetor too? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On Mon, Dec 6, 2010 at 12:14 AM, Miki wrote: > Have you looked at Incanter? (http://incanter.org/) Hmm, interesting. Is there a Rhetor too? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
Have you looked at Incanter? (http://incanter.org/) On Dec 5, 3:27 pm, Robert McIntyre wrote: > I'm trying to use clojure for scientific data analysis but I keep > running into lacunas of functionality. > > I'd love to hear the community's recommendations and experiences with this: > > Is there a standard way to do things like: > 1. take the convolution of two vectors > 2. work with imaginary numbers, quaternions, octonions, etc > 3. work with matrices of arbitrary dimension > 4. Fourier transform ( in multiple dimensions) > 5. integration / finite difference > 6. symbolic manipulation as in sage > 7. minimizing non-linear functions > 8. finding zeros of non-linear functions > > thank you all in advance for any recommendations you might have. > > --Robert McIntyre -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
Thanks for your input --- I'm hoping that some of this stuff is already written with performance optimizations and the like. I'm wondering if people have had experience with java libraries of that sort and might have some recommendations. Anyone use clojure for scientific data analysis? What do you find helpful to use? --Robert McIntyre On Sun, Dec 5, 2010 at 9:36 PM, Ken Wesson wrote: > On Sun, Dec 5, 2010 at 6:27 PM, Robert McIntyre wrote: >> I'm trying to use clojure for scientific data analysis but I keep >> running into lacunas of functionality. >> >> I'd love to hear the community's recommendations and experiences with this: >> >> Is there a standard way to do things like: >> 1. take the convolution of two vectors >> 2. work with imaginary numbers, quaternions, octonions, etc >> 3. work with matrices of arbitrary dimension >> 4. Fourier transform ( in multiple dimensions) >> 5. integration / finite difference >> 6. symbolic manipulation as in sage >> 7. minimizing non-linear functions >> 8. finding zeros of non-linear functions > > Standard, as in built into Clojure? No. Standard as in algorithmic? Of course. > > There are two options here. First, many of those things are fairly > easily implemented in Clojure, and with a bit more work can be made > very efficient (essentially, native-C efficient). For instance, for > matrices you'd want a contiguous representation in memory so you'd use > a vector, or even a Java array of doubles, of length m*n and functions > that provided a matrix API and used this representation internally. > With definline and macros this can be made efficient, and a Java array > of doubles could have the speed of equivalent C code since it would be > a contiguous block of doubles in RAM just as you'd get in C. > > Second, there are probably lots of existing scientific-computing > libraries for Java out there that do the things you need done, and > Clojure can load and call into Java libraries pretty easily. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: math utilities question
On Sun, Dec 5, 2010 at 6:27 PM, Robert McIntyre wrote: > I'm trying to use clojure for scientific data analysis but I keep > running into lacunas of functionality. > > I'd love to hear the community's recommendations and experiences with this: > > Is there a standard way to do things like: > 1. take the convolution of two vectors > 2. work with imaginary numbers, quaternions, octonions, etc > 3. work with matrices of arbitrary dimension > 4. Fourier transform ( in multiple dimensions) > 5. integration / finite difference > 6. symbolic manipulation as in sage > 7. minimizing non-linear functions > 8. finding zeros of non-linear functions Standard, as in built into Clojure? No. Standard as in algorithmic? Of course. There are two options here. First, many of those things are fairly easily implemented in Clojure, and with a bit more work can be made very efficient (essentially, native-C efficient). For instance, for matrices you'd want a contiguous representation in memory so you'd use a vector, or even a Java array of doubles, of length m*n and functions that provided a matrix API and used this representation internally. With definline and macros this can be made efficient, and a Java array of doubles could have the speed of equivalent C code since it would be a contiguous block of doubles in RAM just as you'd get in C. Second, there are probably lots of existing scientific-computing libraries for Java out there that do the things you need done, and Clojure can load and call into Java libraries pretty easily. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
math utilities question
I'm trying to use clojure for scientific data analysis but I keep running into lacunas of functionality. I'd love to hear the community's recommendations and experiences with this: Is there a standard way to do things like: 1. take the convolution of two vectors 2. work with imaginary numbers, quaternions, octonions, etc 3. work with matrices of arbitrary dimension 4. Fourier transform ( in multiple dimensions) 5. integration / finite difference 6. symbolic manipulation as in sage 7. minimizing non-linear functions 8. finding zeros of non-linear functions thank you all in advance for any recommendations you might have. --Robert McIntyre -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
