Runtime Compilation of Clojure from Android
Remco van't Veer has done a lot of great working porting Clojure to Android, but one thing remains missing, runtime compilation which would allow a fully functional Repl. The problem is that the Android VM doesn't use standard Java class files but rather Android specific ones. There is a host-side tool that converts Java byte codes to Android ones. But that tool doesn't normally run on Android. So, being an Android systems hacker, I was wondering how hard it would be to get runtime compilation working. Turns out, it doesn't seem to be too hard. The trick is basically to run the Java to Android/Dalvik byte code converter through itself, so that the convertor can be run on Android; if you load that converted convertor into Android, you can then invoke it after each class is compiled, essentially like this: if(RT.booleanCast(COMPILE_FILES.deref())) { writeClassFile(internalName, bytecode); String strings[] = {"--dex", "--output=/data/gbj/classes.dex", "/ data/gbj/classes"}; com.android.dx.command.Main.main(strings); } (Note that *compile-path* is set elsewhere to "/data/gbj/classes") Loading the generated class file is also a bit tricky, but not too bad. It isn't too fast, partly because of Android's slow GC and partly because I'm using the filesystem as intermediate storage before loading the classes, (because I haven't figured out how to do it all in memory yet.) I'll send out a full set of patches if there is interest, but for now here is an Android app with the repl built in so you can play with it. (It is version of Remco's simple calculator app with a socket based repl added.) Much thanks to Remco upon whose Android/Clojure work this is all built. App install instructions below. g (Note: I've only tested these install instructions out on a Linux host with this sdk: android-sdk-linux_x86-1.5_r2.zip) Download the prebuilt apk file here: http://georgejahad.com/clojure/calc-debug.apk To configure the emulator: emulator -avd adb install -rcalc-debug.apk adb shell mkdir /data/gbj adb shell mkdir /data/gbj/classes adb shell chmod 777 /data/dalvik-cache adb shell chmod 777 /data/gbj adb shell chmod 777 /data/gbj/classes adb forward tcp:8030 tcp:8030 Then start up the calc app from the emulator gui. Then access the repl like so: telnet localhost 8030 Allow compilation like so: (def *android* true) (def *compile-files* true) Now you can compile from the repl. For example, this adds a simple exponent operator to the calculator: (in-ns 'examples.calc) (defn exp [ b e] (reduce * (repeat b e))) (def calc-opers {\+ #'+ \- #'- \/ #'/ \* #'* \e #'exp}) (def calc-allowed-chars (.toCharArray "e0123456789.")) You can examine Remco's source code to see how the calculator works here: http://github.com/remvee/clj-android/blob/d1e96d33487ddcdc04b403e97f80fcf1b31bb9c2/examples/calc/src/examples/calc.clj --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
always-merge-with
I needed the functionality of merge-with, but need f applied in all cases. This version only works for 2 maps, but you can use reduce for more maps. This lets me incrementally build up maps whose vals can be seqs. (defn always-merge-with [f amap bmap] "Like merge-with, but always applies f" (loop [bkeys (keys bmap) bvals (vals bmap) result amap] (if (nil? bkeys) result (let [bkey (first bkeys) bval (first bvals)] (recur (next bkeys) (next bvals) (merge result {bkey (f (get amap bkey) bval)})) (defn wrap-merge [amap bmap] "If key doesn't exist in amap, wraps val from bmap in []. Useful when incrementally merging a seq of vals into hashmaps when the vals can be seqs themselves." (always-merge-with #(if (nil? %1) [%2] (conj %1 %2)) amap bmap)) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: multiple agents yielding flat performance
> After the change, it runs in 5 seconds on four cores as opposed to > 15 seconds on a single > core. Thank you for taking the time to help me with this. It's been a > learning experience. Great news! Happy to help. This stuff is pretty new to me, too :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: multiple agents yielding flat performance
Nevermind, kindly ignore my last post ;-) You called it. The map inside my decode function returns a lazy seq, and it was being accessed on-demand by the doseq towards the end of the program. To make matters worse, I was consuming the agents in a serial fashion completely eliminating any parallel processing. After the change, it runs in 5 seconds on four cores as opposed to 15 seconds on a single core. Thank you for taking the time to help me with this. It's been a learning experience. Travis On Jun 15, 8:22 pm, tmountain wrote: > Completely omitting work-buckets and spawn-agents, I've replaced with > the following, but the CPU still sits at 100% usage, and the run time > is still ~15 seconds. > > (def work-units (doall (for [x (range 15)] > "88148433eeb5d372c0e352e38ac39aca"))) > (def agents [(agent work-units) > (agent work-units) > (agent work-units) > (agent work-units)]) > > ; send each agent a job to do > (doseq [agent agents] > (send agent decode)) > > ; wait on the agents to complete their jobs > (apply await agents) > > ; see the results > (doseq [agent agents] > (doseq [result @agent] > (println result))) > > ; clean up > (shutdown-agents) > > On Jun 15, 7:06 pm, Richard Newman wrote: > > > Try doall: > > >http://clojure.org/api#toc216 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: multiple agents yielding flat performance
Completely omitting work-buckets and spawn-agents, I've replaced with the following, but the CPU still sits at 100% usage, and the run time is still ~15 seconds. (def work-units (doall (for [x (range 15)] "88148433eeb5d372c0e352e38ac39aca"))) (def agents [(agent work-units) (agent work-units) (agent work-units) (agent work-units)]) ; send each agent a job to do (doseq [agent agents] (send agent decode)) ; wait on the agents to complete their jobs (apply await agents) ; see the results (doseq [agent agents] (doseq [result @agent] (println result))) ; clean up (shutdown-agents) On Jun 15, 7:06 pm, Richard Newman wrote: > Try doall: > > http://clojure.org/api#toc216 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Mnesia like?
On Jun 15, 12:02 pm, Wilson MacGyver wrote: > Does clojure have anything like erlang's Mnesia? or is anyone working on such > project? I know I can fall back to using JDBC+ various RDBMS, but I > was curious if there is something that works like Mnesia. I've dreamed about some kind of "durable Ref" that commits to disk, but never had the time to write any code. At some point, you're basically implementing a database, and that's a lot of work. -Stuart Sierra --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: multiple agents yielding flat performance
Try doall: http://clojure.org/api#toc216 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: multiple agents yielding flat performance
I've tried this in lieu of the way I was generating the work units. (def work-units (for [x (range 100)] "88148433eeb5d372c0e352e38ac39aca")) I know that for is still lazy, so I did this after the binding of work- buckets: (println work-buckets) ; yielded expected result ((88148433eeb5d372c0e352e38ac39aca 88148433eeb5d372c0e352e38ac39aca I'm assuming if I can see work-buckets from the print statement that it's fully materialized. This part all happened in less than a second. >From here, all that's left to do is spawn four agents (assigning each a work bucket), send them each a decode function, and the loop over the result set, and I'm seeing the same results as before. On Jun 15, 6:36 pm, Richard Newman wrote: > > I'm testing on a quad-core box, and I'm seeing virtually identical > > performance numbers switching between one agent and four agents. I've > > also tried larger numbers like 12 agents, but the results are the > > same. The load average also stays the same hovering around 1.0, and > > the execution time is static at approximately 15 seconds. > > It looks to me like your work unit generation is lazy. I also surmise > that it's taking 15 seconds to do 50,000 MD5 computations == 0.3ms per > computation, and that might include the rest of your program, too. > > Is it possible that the lazy computation of the work unit -- which is > not parallelized -- is only running fast enough to supply one agent > with input? > > Have you tried completely materializing your work units before timing > the agent part? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: multiple agents yielding flat performance
> I'm testing on a quad-core box, and I'm seeing virtually identical > performance numbers switching between one agent and four agents. I've > also tried larger numbers like 12 agents, but the results are the > same. The load average also stays the same hovering around 1.0, and > the execution time is static at approximately 15 seconds. It looks to me like your work unit generation is lazy. I also surmise that it's taking 15 seconds to do 50,000 MD5 computations == 0.3ms per computation, and that might include the rest of your program, too. Is it possible that the lazy computation of the work unit -- which is not parallelized -- is only running fast enough to supply one agent with input? Have you tried completely materializing your work units before timing the agent part? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
multiple agents yielding flat performance
Having learned about agents recently, I've created a pretty contrived program, which I believed would easily lend itself to parallel processing. The program takes a list of MD5 sums and then does brute force comparison the find the corresponding four character strings they were generated from. The brute force character generation and MD5'ing are CPU intensive, so I assumed I'd see a big performance improvement by spawning one agent per CPU and then dividing the workload evenly across them. I'm testing on a quad-core box, and I'm seeing virtually identical performance numbers switching between one agent and four agents. I've also tried larger numbers like 12 agents, but the results are the same. The load average also stays the same hovering around 1.0, and the execution time is static at approximately 15 seconds. To make sure Clojure, Java, and agents were at least working, I also tested a very small program to just do an infinite loop per agent, and it instantly maxed out all four CPUs, so I know agents work ;-). (ns agentmd5 (:refer-clojure) (:import (java.security NoSuchAlgorithmException MessageDigest) (java.math BigInteger))) ; computes an md5sum of a string (defn md5-sum "Compute the hex MD5 sum of a string." [#^String str] (let [alg (doto (MessageDigest/getInstance "MD5") (.reset) (.update (.getBytes str)))] (try (.toString (new BigInteger 1 (.digest alg)) 16) (catch NoSuchAlgorithmException e (throw (new RuntimeException e)) ; starts at "" and rotates string based on `n' ; "", "aaab", "aaac", etc... (defn base26 [n] (let [seed-string "" s (new StringBuilder seed-string) a_val (int \a)] (loop [pos 3 x (int n)] (when (pos? x) (let [digit (char (+ a_val (unchecked-remainder x 26)))] (.setCharAt s pos digit) (when (pos? pos) (recur (int (dec pos)) (unchecked-divide x 26)) (.toString s))) ; cycles through character combinations for a given md5 string ; until the matching result is found (defn decode-md5 [md5] (loop [x 0] (let [gen-string (base26 x)] (if (= md5 (md5-sum gen-string)) gen-string (if (< x 456976) ; 26^4 (recur (inc x))) ; decodes a bucket (defn decode [bucket] (map decode-md5 bucket)) ; returns a collection of agents for a given set of work buckets (defn spawn-agents [agents buckets] (if (empty? buckets) agents (recur (conj agents (agent (first buckets))) (rest buckets ; number of tasks to doll out to the agents (def num-work-units 60) ; generate requested number of work units (def work-units (map md5-sum (for [x (range num-work-units)] (base26 5 ; (base26 5) == "cvzc" ; number of agents to spawn (def num-agents 4) ; divide the units into buckets for each agent (def work-buckets (partition (int (/ (count work-units) num-agents)) work- units)) ; create an agent for each bucket (def agents (spawn-agents [] work-buckets)) ; send each agent a job to do (doseq [agent agents] (send agent decode)) ; ensure successful dispatch (apply await agents) ; view the results (doseq [agent agents] (doseq [result @agent] (println result))) ; clean up (shutdown-agents) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Mnesia like?
yea, I thought of couchDB. the nice thing about Mnesia in erlang is that it's completely self-contained. I didn't know if there is such a thing in clojure, or works underway to create one. On Mon, Jun 15, 2009 at 2:20 PM, Phil Hagelberg wrote: > > Daniel Lyons writes: > >> I might throw up a bit for suggesting this, but you might also want to >> look at CouchDB, which uses HTTP to provide a generic JSON "document" >> store based on Mnesia. It would probably be easier to interface with >> than Erlang directly and ought to have most of the same benefits since >> it's built on Mnesia. >> >> http://couchdb.apache.org/ > > CouchDB is definitely awesome, but it's not just a web frontend for > Mnesia. It's a totally different database, it just happens to be > implemented in Erlang. > > The Clojure interface is pretty straightforward, but it works: > > http://github.com/danlarkin/clojure-couchdb/tree/master > > -Phil > > > > -- Omnem crede diem tibi diluxisse supremum. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: matlab like clojure
Hi, I've got a simple implementation of the Jacobi-method for approximating the solution of a linear system of equations, maybe your interested in this. (defn jacobi-x-step [m b x i] "Calculate a part of the solution of a jacobi-method step" ;; 1 ;; xi = --- (bi - ( sum aij * xij) ;; mij(not= i j) (* (/ 1 (get-in m [i i])) (- (b i) (reduce + (map #(* (get-in m [i %]) (x %)) (remove #(= i %) (range (count x (defn jacobi-method-step "Calculate one step of the jacobi-method given a matrix m, a vector b, and a former solution x." ([m b] ;; first step of the jacobi method, init x with nullvector (jacobi-method-step m b (vec (repeat (count b) 0 ([m b x] (vec (map #(jacobi-x-step m b x %) (range (count x)) (defn jacobi-method "Return a lazy sequence of solutions to the equation of matrix m and vector b and x: M*b = x. To obtain a convergent sequence, m should be diagonally dominant." [m b] (iterate (partial jacobi-method-step m b) (jacobi-method-step m b))) ;; example: (last (take 30 (jacobi-method [[3 1 1] [1 3 1] [1 1 3]] [11.0 10.0 8.0] ))) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Mnesia like?
Daniel Lyons writes: > I might throw up a bit for suggesting this, but you might also want to > look at CouchDB, which uses HTTP to provide a generic JSON "document" > store based on Mnesia. It would probably be easier to interface with > than Erlang directly and ought to have most of the same benefits since > it's built on Mnesia. > >http://couchdb.apache.org/ CouchDB is definitely awesome, but it's not just a web frontend for Mnesia. It's a totally different database, it just happens to be implemented in Erlang. The Clojure interface is pretty straightforward, but it works: http://github.com/danlarkin/clojure-couchdb/tree/master -Phil --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Mnesia like?
On Jun 15, 2009, at 10:02 AM, Wilson MacGyver wrote: > > Does clojure have anything like erlang's Mnesia? or is anyone > working on such > project? I know I can fall back to using JDBC+ various RDBMS, but I > was curious if there is something that works like Mnesia. I don't think there is one at the moment. It would be really cool though. If you want to use Mnesia itself, you can always call Erlang from its Java bridge. There is an example of interacting with Erlang in the files section: http://clojure.googlegroups.com/web/erlang+(2).clj If you're interested in Mnesia for scalability, you may be interested in the Java library Terracotta, which is a distributed object persistence system of some kind. I'm curious about it even though it implies some objects and some XML, but I haven't used it yet. http://www.terracotta.org/ I might throw up a bit for suggesting this, but you might also want to look at CouchDB, which uses HTTP to provide a generic JSON "document" store based on Mnesia. It would probably be easier to interface with than Erlang directly and ought to have most of the same benefits since it's built on Mnesia. http://couchdb.apache.org/ Hope that helps, — Daniel Lyons http://www.storytotell.org -- Tell It! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Mnesia like?
There's always using derby or hsqldb as a backend for Hibernate; and if a dumb distributed (disk-backed) cache is good enough for your purposes then you might try Oracle Coherence, which scales great and is incredibly easy to use but sort of expensive. It doesn't feel nearly as language-native as Mnesia, though. --josh On Jun 15, 12:02 pm, Wilson MacGyver wrote: > Does clojure have anything like erlang's Mnesia? or is anyone working on such > project? I know I can fall back to using JDBC+ various RDBMS, but I > was curious if there is something that works like Mnesia. > > Thanks, > Mac > > -- > Omnem crede diem tibi diluxisse supremum. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: dosync ref-set and threads coordination problem
Thanks for your reply, you are right, the server close before all clients completed. (i was suppose that dorun waits for the threads results before returning.) Anyway, i was trying to emulate the real problem with no success here. I 'll try to describe it. The clients connect to an extrernal application via socket with a fixed protocol requesting some information for an object indexed with and wait a confirmation response (not the information) from the external application. The external application connects to my socket-server returns that information, waits for a "received ok" confirmation and then respond to the client with "sending ok". Each client need to call a different function on information data so, in my clients code: put the id and the function to call on data, in my server reference variable connect to external application sending the id. waiting a confirmation response from the external application. in my server code: read the id and information from the external application. respond to the external application with "received ok" lookup that id in reference variable and call the function on data. The problem is that in my testing environment (20 concurent client request repeated 200 times) the server cant find some ids in reference variable.(1 or 2 in 1000). I suspect that i loose those ids because, on high server load, the time to send the id to the external application and receivce the response to server is less that the time needed for the server thread to see the updated value of reference. (is there a maximum time guarantee for content switch of referenced variables?) The solution i gave is a busy loop with a timeout until the server thread finds the id. But the timeout is arbitrary and depends on server load. PS: the flow of information can't be changed because of the fixed communication protocol. On Jun 15, 4:13 pm, sross wrote: > On Jun 14, 12:20 am, vmargioulas wrote: > > > > > In the example below 256 clients threads put a byte (0-255) to server- > > socket reference var "items", > > then connect to server-socket, write the same byte to socket stream > > and close the connection. > > The server connections threads reads a byte from the socket stream and > > remove it from the var "items". > > At the end of the run i expect the var items to be empty but it is > > not. > > It 's contains a random number of bytes. > > Where is the error? > > > (ns > > test-dosync > > (:import (java.net Socket)) > > (:use clojure.contrib.server-socket)) > > > (def server > > (let [items (ref nil) > > server-ref (ref nil) > > server-fn (fn [ins _] > > (let [bt (.read ins)] > > (dosync > > (ref-set items (remove #(= % bt) @items)] > > > {:enq (fn [item] (dosync (alter items conj item))) > > :list-items (fn [] @items) > > :start (fn [port] > > (let [socket-server (create-server port server-fn)] > > (dosync > > (ref-set items nil) > > (ref-set server-ref socket-server > > :stop (fn [] (when @server-ref (close-server @server-ref))) > > })) > > > (defn start-server [port] ((:start server) port)) > > (defn stop-server [] ((:stop server))) > > (defn enq [id] ((:enq server) id)) > > (defn list-items [] ((:list-items server))) > > > (defn client-fn [port #^Byte byte-to-send] > > #(let [socket (Socket. "localhost" port) > > outs (.getOutputStream socket)] > > (try > > (do > > (enq byte-to-send) > > (doto outs (.write byte-to-send) (.flush))) > > (finally > > (.close socket) > > > (defn run-test [] > > (try > > (let [port 10002] > > (start-server port) > > (dorun (apply pcalls (map #(client-fn port %) (range 256) > > (finally (stop-server))) > > (list-items)) > > This is probably a lot simpler than you may think; > > What is happening here is that your run-test function is returning > before the removes have been run since the client functions do not > wait for confirmation from the server before returning. This can > result in ordering like the following > (with 2 client threads) > > Client 1 enqueues byte > Client 1 writes byte and flushes > Client 2 enqueues byte > Client 2 writes byte and flushes > Server Thread for Client1 reads byte > Server Thread for Client2 reads byte > Server gets stopped > run-test returns (0 1) > Server Thread for Client1 removes byte > Server Thread for Client2 removes byte. > > Of course this is only an example of a multitude of ways that these > can combine, you can add a couple of logging > statements to your functions to see when these things are happening, > and I think you may be quite surprised at > the different orderings that can arise, but remember to take them with > a grain of salt as output isn't immediate either ;) > > The crux here is that your are relying on the server functions running > to c
Mnesia like?
Does clojure have anything like erlang's Mnesia? or is anyone working on such project? I know I can fall back to using JDBC+ various RDBMS, but I was curious if there is something that works like Mnesia. Thanks, Mac -- Omnem crede diem tibi diluxisse supremum. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: matlab like clojure
On Jun 15, 2009, at 16:23, Bugs wrote: > I'm a newbie on Clojure and LISP, and I'm interasted in matlab > replacement languages. > It is just my hobby, but I'm trying to implement a matlab like > language extension. > > I think that Clojure has ability enough to be numerical language like > a Matlab or Mathematica. There's nothing wrong with Clojure as a language for such a project, quite on the contrary. However, please keep in mind that both Matlab and Mathematica contain enormous libraries of numerical routines (plus symbolic computation for Mathematica). Creating the equivalent from scratch, in any language, is a very ambitious project. The closest Clojure-based project I know of is Incanter: http://github.com/liebke/incanter/tree/master Its model is R rather than matlab, but there are many characteristics in common. Another projet to look at is http://github.com/aperotte/persistentmatrix/tree/master Its goal is the creation of a N-dimensional array implementation for Clojure. Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: matlab like clojure
On Jun 15, 7:23 pm, Bugs wrote: > Hi, everyone. > > I'm a newbie on Clojure and LISP, and I'm interasted in matlab > replacement languages. > It is just my hobby, but I'm trying to implement a matlab like > language extension. > > I think that Clojure has ability enough to be numerical language like > a Matlab or Mathematica. > > So, I'm finding examples for matrix operation ( + - * .* > transpose ./ .^, and so on) > > Does anyone have the examples? > I don't know much about matlab or mathematica but you might find incanter interesting. http://github.com/liebke/incanter/tree/master Regards, Parth > Regards, --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
matlab like clojure
Hi, everyone. I'm a newbie on Clojure and LISP, and I'm interasted in matlab replacement languages. It is just my hobby, but I'm trying to implement a matlab like language extension. I think that Clojure has ability enough to be numerical language like a Matlab or Mathematica. So, I'm finding examples for matrix operation ( + - * .* transpose ./ .^, and so on) Does anyone have the examples? Regards, --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Universal Date utility
Hey everyone, There have been a couple of threads discussing date utilities in this group. http://groups.google.com/group/clojure/browse_thread/thread/d98e8efd8d5517b2# http://groups.google.com/group/clojure/browse_thread/thread/659503e698ede0b5/9dda25f36f102799?lnk=gst&q=joda#9dda25f36f102799 The unilateral recommendation is to use Joda time to handle date calculations. This makes a lot of sense, because Joda kicks ass at what it does. However, it isn't very abstract. This got me thinking about developing a Clojure wrapper library. I think it should have the following requirements 1. Work on some abstract notion of time, not a specific class. 2. Be able to wrap time-like objects, just like seq wraps seq-like objects 3. Be immuatble 4. Use Joda to do the heavy lifting 5. Be extensible That's what my refined date utility library does. It uses the following multimethod: to-ms-count This mm recursively converts the input down to the number of ms since 1970. In turn, each data type is built upon the Long ms value. This allows date objects to be transformed easily I'm going to keep working on these, and hopefully flesh out the joda- utils more this week I've got some information up on github. The two interesting namespaces are lib.devlinsf.date-utils lib.devlinsf.joda-utils You can see some basic documentation in the docs directory. Specifically, docs/date-utils.markdown docs/joda-utils.markdown PDF showing crude design graph: http://cloud.github.com/downloads/francoisdevlin/devlinsf-clojure-utils/date-utils_design.pdf Sean --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Performance Penalty Converting from Java Code
Thanks for the replies guys. Both solutions outperform the original. The one using unchecked operations is about 60x faster, which is the kind of performance I was hoping for. It's good to see that Clojure has the capacity to do things very quickly. I just need to learn all the tricks to make it do so. I'm going to use this code as part of a small agent demo I'm working on. Basically, it's going to take in work units (MD5'd strings) and run brute force comparisons on them until a match is found. I can fire up an agent per core and compare performance numbers as the workload is parallelized. It's really simple stuff but good for a concurrency beginner like me to explore with. Thanks, Travis On Jun 14, 10:00 am, tmountain wrote: > I've been playing around with rewriting some Java code in Clojure and > did some simple benchmarking in the process. In this case, there's a > huge disparity in the performance numbers between the two languages, > and I'm wondering what the cause may be. The program rotates a string > from "", "aaab", ..., "". The Java version takes 0.77 seconds > to complete while the Clojure version takes 22 seconds. I've tried to > make the scripts relatively isomorphic and have verified that they > produce the same results. I'm pasting the source below. > > tra...@travis-ubuntu:/tmp% time clj base26.clj > clj base26.clj 21.99s user 1.23s system 85% cpu 27.318 total > > tra...@travis-ubuntu:/tmp% time java Base26 > java Base26 0.77s user 0.04s system 78% cpu 1.029 total > > clojure version: > > (defn base26 [n] > (let [seed-string "" > s (new StringBuilder seed-string)] > (loop [pos (- (count seed-string) 1) > x n] > (if (> x 0) > (let [digit (+ (int \a) (mod x 26))] > (. s setCharAt pos (char digit)) > (if (and (> pos 0) (> x 0)) > (recur (- pos 1) (/ x 26)) > (. s toString))) > > (doseq [i (range (Math/pow 26 4))] > (base26 i)) > > java version: > > import java.lang.StringBuilder; > > public class Base26 { > public static void main(String[] args) { > for (int i = 0; i < Math.pow(26, 4); i++) { > Base26.base26(i); > } > } > > public static String base26(int num) { > if (num < 0) { > throw new IllegalArgumentException("Only positive numbers > are supported"); > } > StringBuilder s = new StringBuilder(""); > for (int pos = 3; pos >= 0 && num > 0 ; pos--) { > char digit = (char) ('a' + num % 26); > s.setCharAt(pos, digit); > num = num / 26; > } > return s.toString(); > } > > } > > I've tried warn-on-reflection, and it didn't report anything. > > Thanks, > Travis --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: breaking early from a "tight loop"
On Sun, Jun 14, 2009 at 4:00 AM, Sudish Joseph wrote: > > On Jun 13, 4:17 pm, Laurent PETIT wrote: >> So it really seems to me that the missing abstraction, here, is being >> able to do a reduce over the list of pixels, and being able, from the >> reduction function, to quit the reduction early. > > A lazy right fold[1] allows short-circuiting, so here's one attempt: Here's another way to do a sort of lazy reduce: (use '[clojure.contrib.seq-utils :only [reductions]]) (some #(> % 10) (reductions (fn [counter pixel] (if (< pixel 10) (inc counter) counter)) 0 (iterate dec 15))) --Chouser --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Primitive char Type
Wow, you really got to the bottom of this. Reading your post, it all makes sense, but it leads me to wonder why StringBuilder was designed in such a fashion and why the docs would go so far as to lie about it. Either way, thanks for taking the time to help me out. This community is a big part of what makes working with Clojure so awesome. Travis On Jun 14, 6:58 pm, "Stephen C. Gilardi" wrote: > On Jun 13, 2009, at 10:46 PM, Stephen C. Gilardi wrote: > > > user=> (.setCharAt s 0 \c) > > java.lang.IllegalArgumentException: Can't call public method of non- > > public class: public void > > java.lang.AbstractStringBuilder.setCharAt(int,char) (NO_SOURCE_FILE:0) > > > I'm not sure why. > > StringBuilder extends AbstractStringBuilder (though the JavaDoc docs > lie and say it extends Object). AbstractStringBuilder has default > accessibility (not public, protected, or private) which makes the > class inaccessible to code outside the java.lang package. > > In both Java SE 5 and Java SE 6, StringBuilder does not contain > a .setCharAt method definition. It relies on the inherited public > method in AbstractStringBuilder. (I downloaded the source code for > both versions from Sun to check.) > > In Java SE 5, when Clojure checks whether or not .setCharAt on > StringBuilder is public, it finds that it's a public method of a non- > public base class and throws the exception you saw. (It looks like > you're using a version of Clojure older than 18 May 2009 (Clojure svn > r1371). Versions later than that print the more detailed message I saw.) > > In Java SE 6, Clojure's checks for accessibility of this method > succeed and the method call works. > > I'm not sure whether or not Clojure could be modified to make this > method call work in Java 5. Google searches turn up discussion that > this pattern of using an undocumented abstract superclass with non- > public accessibility is not common in the JDK. > > --Steve > > smime.p7s > 3KViewDownload --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: dosync ref-set and threads coordination problem
On Jun 14, 12:20 am, vmargioulas wrote: > In the example below 256 clients threads put a byte (0-255) to server- > socket reference var "items", > then connect to server-socket, write the same byte to socket stream > and close the connection. > The server connections threads reads a byte from the socket stream and > remove it from the var "items". > At the end of the run i expect the var items to be empty but it is > not. > It 's contains a random number of bytes. > Where is the error? > > (ns > test-dosync > (:import (java.net Socket)) > (:use clojure.contrib.server-socket)) > > (def server > (let [items (ref nil) > server-ref (ref nil) > server-fn (fn [ins _] > (let [bt (.read ins)] > (dosync > (ref-set items (remove #(= % bt) @items)] > > {:enq (fn [item] (dosync (alter items conj item))) > :list-items (fn [] @items) > :start (fn [port] > (let [socket-server (create-server port server-fn)] > (dosync > (ref-set items nil) > (ref-set server-ref socket-server > :stop (fn [] (when @server-ref (close-server @server-ref))) > })) > > (defn start-server [port] ((:start server) port)) > (defn stop-server [] ((:stop server))) > (defn enq [id] ((:enq server) id)) > (defn list-items [] ((:list-items server))) > > (defn client-fn [port #^Byte byte-to-send] > #(let [socket (Socket. "localhost" port) > outs (.getOutputStream socket)] > (try > (do > (enq byte-to-send) > (doto outs (.write byte-to-send) (.flush))) > (finally > (.close socket) > > (defn run-test [] > (try > (let [port 10002] > (start-server port) > (dorun (apply pcalls (map #(client-fn port %) (range 256) > (finally (stop-server))) > (list-items)) This is probably a lot simpler than you may think; What is happening here is that your run-test function is returning before the removes have been run since the client functions do not wait for confirmation from the server before returning. This can result in ordering like the following (with 2 client threads) Client 1 enqueues byte Client 1 writes byte and flushes Client 2 enqueues byte Client 2 writes byte and flushes Server Thread for Client1 reads byte Server Thread for Client2 reads byte Server gets stopped run-test returns (0 1) Server Thread for Client1 removes byte Server Thread for Client2 removes byte. Of course this is only an example of a multitude of ways that these can combine, you can add a couple of logging statements to your functions to see when these things are happening, and I think you may be quite surprised at the different orderings that can arise, but remember to take them with a grain of salt as output isn't immediate either ;) The crux here is that your are relying on the server functions running to completion but you are not waiting on the client side to confirm that this has happened. - sean --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Dynamically accessing static fields
On Jun 15, 7:08 am, James Koppel wrote: > I am trying to write a function to simplify working with GridBagConstraints > -- that is, instead of writing > > (let [c (GridBagConstraints.)] > (set! (.weightx c) 2.0) > (set! (.gridwidth c) GridBagConstraints/REMAINDER) > (let [button (JButton. "Hello, world!")] > (.setConstraints (.getLayout *my-container*) button c) > (.add *my-container* button))) > > I could simply write > > (gridbag-add *my-container* > (JButton. "Hello, world!") > "weightx=2.0;gridwith=GridBagConstraints/REMAINDER") > > A simple combination of regexes and read-string would easily allow me to > extract the symbol 'GridBagConstraints/REMAINDER from the example string, > but I'm having trouble actually converting it into its value. Using resolve > simply returns nil, and getting "." to work dynamically seems to be > fruitless, as even this simple call > > (. (resolve 'GridBagConstraints) REMAINDER) > > throws an exception. > > So, the question is, how do I go dynamically from a string like > "GridBagConstraints/REMAINDER" to the actual value of the static field? > > Of course, eval does the trick, but I'd rather not have to resort to it. One way to do that would be to use a map: user=> (def m {"Math/PI" Math/PI "Math/E" Math/E}) #'user/m user=> (defn foo [n s] [n (get m s :not-found)]) #'user/foo user=> (foo 10 "Math/PI") [10 3.141592653589793] user=> You could also consider writing a function that takes these as parameters and returns the updated container. That way you can avoid the regex. Regards, Parth > > Sorry if the answer is obvious; I'm a Clojure-newbie. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Dynamically accessing static fields
I am trying to write a function to simplify working with GridBagConstraints -- that is, instead of writing (let [c (GridBagConstraints.)] (set! (.weightx c) 2.0) (set! (.gridwidth c) GridBagConstraints/REMAINDER) (let [button (JButton. "Hello, world!")] (.setConstraints (.getLayout *my-container*) button c) (.add *my-container* button))) I could simply write (gridbag-add *my-container* (JButton. "Hello, world!") "weightx=2.0;gridwith=GridBagConstraints/REMAINDER") A simple combination of regexes and read-string would easily allow me to extract the symbol 'GridBagConstraints/REMAINDER from the example string, but I'm having trouble actually converting it into its value. Using resolve simply returns nil, and getting "." to work dynamically seems to be fruitless, as even this simple call (. (resolve 'GridBagConstraints) REMAINDER) throws an exception. So, the question is, how do I go dynamically from a string like "GridBagConstraints/REMAINDER" to the actual value of the static field? Of course, eval does the trick, but I'd rather not have to resort to it. Sorry if the answer is obvious; I'm a Clojure-newbie. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Arraylist faster than primitive array?
I finally see what warn-on-reflection does. It wasn't working in netbeans, it works in emacs. On Jun 14, 2:48 pm, CuppoJava wrote: > It looks like a case of reflection being used. > I would set *warn-on-reflection* to true, and just check which method > is not being resolved. > > My guess is probably the aset. > (aset arlist (int i) (int 1)) would probably fix it. > > -Patrick --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: super-lazy-seq
On Jun 12, 2009, at 7:52 PM, Wrexsoul wrote: > Well, I did it. Made implementing lazy seqs that require stateful > generator functions easy, that is: > > (defn files-and-dirs-recursive [dir] > (super-lazy-seq [stack [(to-file dir)]] >(if-not (empty? stack) > (let [file (first stack) s (rest stack)] >(next-item > file > (if (directory? file) >(concat s (files-and-dirs file)) >s)) Before we order the cake, let's compare to Clojure's own implementation: (defn tree-seq "Returns a lazy sequence of the nodes in a tree, via a depth-first walk. branch? must be a fn of one arg that returns true if passed a node that can have children (but may not). children must be a fn of one arg that returns a sequence of the children. Will only be called on nodes for which branch? returns true. Root is the root node of the tree." [branch? children root] (let [walk (fn walk [node] (lazy-seq (cons node (when (branch? node) (mapcat walk (children node))] (walk root))) (defn file-seq "A tree seq on java.io.Files" [dir] (tree-seq (fn [#^java.io.File f] (. f (isDirectory))) (fn [#^java.io.File d] (seq (. d (listFiles dir)) No macros. No magic names. No explicit state. Easier to explain. Same result. On Jun 13, 2009, at 9:37 PM, Wrexsoul wrote: > When clojure.contrib releases version 1.0, that might be an option. One of the many things I hated about other languages was the maddeningly incipient libraries. With Rails, for example, the documentation was always a blog post telling you to check out the latest version of the source from GitHub, where you would go to find it abandoned and the developed fork of someone's fork of it would have a different, undocumented API, and then you'd find out it doesn't work in your one-minor-release-old version of Rails... etc. But let's face it; we're using a language whose 1.0 release was less than two months ago. Stability is something I would expect to see in Clojure in 3-5 years, not next month. Everyone uses Contrib; it's almost an incubator for future core functionality. I can't imagine using Clojure without duck-streams. On Jun 14, 2009, at 9:58 PM, Wrexsoul wrote: > Of course, super-lazy-seq: > a) does the wrapping AND unwrapping for you > b) manages the rebinding each iteration for you, > transparently and > c) works just as well. I think the amount of debate we've seen about it here is sufficient proof that, though it works just as well, there is a certain funk to it. Personally, I don't like macros whose expansion involves magic names. What happens if you nest them—intentionally or accidentally? Also, this violates another principle of good macro writing, which is that it does not wrap a function call. Generally that means there are situations where you can't get the benefit dynamically. > The loop/recur style of super-lazy-seq use seems "closer to > functional purity" than does an explicitly stateful genfn. True, but tree-seq manages the same task with no explicit state at all. And while we can certainly embark on a debate about whether or not it is more functional to recur or to compose functions, I think we can agree that doing something with a macro that could be done with a regular function is something other than functional. If functional purity is something we are deeply concerned about. > One thing is becoming clear: in clojure, there's generally more than > one way to do it. Whatever "it" is. Agreed! And that all of the options can have their own pragmatic reasoning behind them is great, IMO. I do wonder about the interaction between garbage collection and lazily computed resources. The example code for dealing with databases in the Clojure book seemed to materialize the rows before returning them, which seems like a waste of good laziness, but it probably avoids exactly these kinds of problems. Until I run into actual issues on my own, though, I'm going to let this problem remain theoretical and hope that one of the great minds working on the core can find a solution. For some reason I believe there will probably be a good solution to this, if not generally, then at least in the 80% cases we are likely to run into. — Daniel Lyons http://www.storytotell.org -- Tell It! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: super-lazy-seq
On Jun 15, 4:58 am, Wrexsoul wrote: > Eh. That didn't occur to me. It could be combined with the meta-nil > trick, too: > > (defn custom-lazy-seq [genfn] > (map #(first %) > (take-while (complement nil?) > (repeatedly genfn > > where the genfn returns nil for no-next-item and [item] otherwise. You could also define an explicit end-of-stream object: (def end-of-stream (Object.)) (defn custom-lazy-seq [genfn] (take-while #(not= end-of-stream %) (repeatedly genfn))) > The loop/recur style of super-lazy0-seq use seems "closer to > functional purity" than does an explicitly stateful genfn. Well, coming from a Haskell background I'd consider using higher-level functions to be closer to functional purity, but each to their own. - James --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: super-lazy-seq
On Jun 14, 2009, at 12:53 PM, James Reeves wrote: > > On Jun 14, 6:32 pm, Wrexsoul wrote: >> I wrote super-lazy-seq because repeatedly can't generate a finite >> sequence. It just spat out >> >> (File1 File2 File3 File4 nil nil nil nil nil nil nil ... > > Well, you could wrap it in take-while: > > (defn custom-lazy-seq [stream] >(take-while (complement nil?) > (repeatedly #(next-item-in-seq stream I've done this too. It works great if you're certain the function being called by repeatedly will never return nil until the stream is exhausted. Many streams seem to work like that in practice despite the caveat and I find it nicer looking than an explicit test. — Daniel Lyons http://www.storytotell.org -- Tell It! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---