Re: reducers question
On Wed, Jul 15, 2015 at 12:02 PM, Daniel Higginbotham < nonrecurs...@gmail.com> wrote: > Surely reducers/map should be faster, but it doesn’t seem like I’ve done > something wrong? This is driving me crazy :) It is much faster, and runs in parallel, if you use fold. Try this; (defn fold-into-vec [coll] "Provided a reducer, concatenate into a vector. Note: same as (into [] coll), but parallel." (r/fold (r/monoid into vector) conj coll)) So it should read as; (defn reducer-test [nums] (fold-into-vec (r/map inc nums))) Details here; http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html Hope this helps -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Streaming a big file
I wrote a library ( https://github.com/thebusby/iota ) to handle a very similar issue, which I hope would be of some help to you here. On Wed, May 6, 2015 at 12:18 PM, Sam Raker wrote: > I've got two really big CSV files that I need to compare. Stay tuned for > the semi-inevitable "how do I optimize over this M x N space?" question, > but for now I'm still trying to get the data into a reasonable format--I'm > planning on converting each line into a map, with keys coming from either > the first line of the file, or a separate list I was given. Non-lazy > approaches run into memory limitations; lazy approaches run into "Stream > closed" exceptions while trying to coordinate `with-open` and `line-seq`. > Given that memory is already tight, I'd like to avoid leaving open > files/file descriptors/readers/whatever-the-term-in-clojure-is lying > around. I've tried writing a macro, I've tried transducers, I've tried > passing around the open reader along with the lazy seq, none successfully, > albeit none necessarily particularly well. Any suggestions on streaming > such big files? > > > > Thanks! > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: processing large text files
On Mon, Oct 27, 2014 at 12:52 PM, Brian Craft wrote: > Makes sense, but not an option for this application. What about something > similar to iota, backed with byte arrays, or something? As Patrick pointed out, if you're working directly with byte array's you might want to use mmap which is what Iota does internally. It'll mean handling record delimiters yourself, but depending on the data you may be able to add some optimizations. I tossed together https://github.com/thebusby/clj-mmap for the Clojure Cookbook a while back if that helps. It has a string example, and allows writing back to files if I remember correctly. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: processing large text files
On Mon, Oct 27, 2014 at 7:10 AM, Brian Craft wrote: > I found iota, which looks like a good solution for the read portion of the > problem. However I also need to process the data in the file. If I start > with an iota/vec and need to sort it, something like > > (sort (iota/vec "foo")) > Short disclaimer: I'm the one to blame for Iota. In this situation I've found it easier just to use GNU sort, or an external tool, and then use Iota with the pre-sorted file. Or generate an index with a smaller subset of the data. Examples; $ cat data.tsv | sort -k2 > sorted_data.tsv or (def data (iota/numbered-vec "data.tsv")) (def index (->> data (map (fn [line] (let [[linenum key & _] (clojure.string/split line #"\t" -1)] [key linenum])) (into (sorted-map While Iota does use mmap under the hood for reading/caching, it can't reduce memory consumption if all of the data is converted to a string. For most "Clojurey" operations though, strings are preferred. So the trick is to not realize the entire data set in memory and instead treat it like a stream instead. I'd love to add a mechanism to Iota to solve this problem, but I haven't come upon a good solution yet. I'm all ears though! Hope this helps, TheBusby -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Interop nightmare
On Tue, Sep 9, 2014 at 3:09 AM, Sean Corfield wrote: > > I find that it is quite difficult to use clojure unless one knows Java, > which I believe to be a barrier to new comers. > > I'm surprised every time I hear this. You can write a lot of Clojure > without having to do any interop so you can mostly ignore Java altogether > unless you specifically want to work with a Java library. I don't think any of my Clojure books mention how to convert a string to a numeric, which is missing in Clojure and hence requires Java interop. That's just one example of a fairly basic thing that would be a barrier to new comers. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: fast parallel reduction into hash-set/map
On Fri, Feb 21, 2014 at 8:51 AM, wrote: > One note about your SuperVecs idea though, it seems that using that > approach we could get a deeply nested tree to represent our a vector, and I > think this is the exact thing vectors are trying to avoid.. In general I think this is true for vectors, but in this covers one particular use-case where I've often found generation time to be more important that a faster lookup. If you favor a faster lookup then a normal vector can be used instead of course. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ANN] 美味しいClojure
Hi Dave, Regarding Clojure events in Tokyo, I'm only aware of Tokyo.clj at the moment and it's held in Japanese. It sounds like we have a few expats here in Tokyo using Clojure though, so it might be worth putting something together (in English?). I can likely host something at the Gracenote office here in Shibuya if there is enough interest. If schedules work out Mr. Ambrose may be able to give us a quick intro / Q+A on core.typed as well. Anyone interested? Thanks, Alan Busby On Thu, Oct 3, 2013 at 2:32 PM, Dave Della Costa wrote: > Speaking of, is there anything other than Tokyo.clj (which seems pretty > infrequent based on this: http://atnd.org/event/tokyoclj) going on for > Clojurians in Tokyo? I work remotely, using Clojure, and would love to > get together more frequently with other Clojure developers if possible. > If anyone is interested in doing something like co-working on a more > regular basis, let me know! > > Cheers, > DD > > (2013/10/03 9:44), Alan Busby wrote: > > On Thu, Oct 3, 2013 at 9:30 AM, Marek Kubica > <mailto:ma...@xivilization.net>> wrote: > > > > On Tue, 1 Oct 2013 22:01:02 -0700 (PDT) > > Nicolas Modrzyk mailto:hellon...@gmail.com>> > > wrote: > > > > > The whole text has also been written in English, so I guess there > > > could be a chance to put it out there. > > > But we have had no contacts with US/UK publishers so far. > > > > Did you try the usual suspects like O'Reilly or PragProg? They are > > surprisingly approachable. > > > > > > The O'Reilly Tokyo office is on the ground floor of where Tokyo.clj is > > hosted too! > > > > -- > > -- > > You received this message because you are subscribed to the Google > > Groups "Clojure" group. > > To post to this group, send email to clojure@googlegroups.com > > Note that posts from new members are moderated - please be patient with > > your first post. > > To unsubscribe from this group, send email to > > clojure+unsubscr...@googlegroups.com > > For more options, visit this group at > > http://groups.google.com/group/clojure?hl=en > > --- > > You received this message because you are subscribed to the Google > > Groups "Clojure" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to clojure+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out. > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [ANN] 美味しいClojure
On Thu, Oct 3, 2013 at 9:30 AM, Marek Kubica wrote: > On Tue, 1 Oct 2013 22:01:02 -0700 (PDT) > Nicolas Modrzyk wrote: > > > The whole text has also been written in English, so I guess there > > could be a chance to put it out there. > > But we have had no contacts with US/UK publishers so far. > > Did you try the usual suspects like O'Reilly or PragProg? They are > surprisingly approachable. The O'Reilly Tokyo office is on the ground floor of where Tokyo.clj is hosted too! -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallelising over a lazy sequence - request for help
Sorry to jump in, but I thought it worthwhile to add a couple points; (sorry for being brief) 1. Reducers work fine with data much larger than memory, you just need to mmap() the data you're working with so Clojure thinks everything is in memory when it isn't. Reducer access is fairly sequential, not random, so spinning disks work great here. 2. A 40GB XML file is very often many many smaller XML documents aggregated together. It's often faster to separate each document into it's own line (via various UNIX tools) and parse each line separately. I typically do something like $ zcat bigxml.gz | tr '\n' ' ' | sed 's//\n/' | grep '^' > records.xml . 3. Check out the Iota library, https://github.com/thebusby/iota/ . I often use for reducing over 100's of GB's worth of text data. It does what Jozef suggests, and makes a text file a foldable collection. 4. While pmap is great for advertising the power of Clojure, it's likely safe to say that it should be ignored if you're actually looking for performance. Hope this helps, Alan Busby -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: another game of exploding heap, via clojure.string/split
If you're working with large text files, tsv especially, you may encounter Java's memory overhead for Strings as well. I remember parsing a 1GB TSV file into EDN, and watching it consume 10GB of RAM. I'd recommend taking a look at the Iota library to see if it would be of any help to you, https://github.com/thebusby/iota On Fri, Sep 13, 2013 at 11:47 AM, Cedric Greevey wrote: > On Thu, Sep 12, 2013 at 3:33 PM, Andy Fingerhut > wrote: > >> I have just added some discussion of this on ClojureDocs.org for the >> function clojure.core/subs, and references to that discussion for several >> other Clojure functions that I am pretty sure are affected, e.g. re-find, >> re-seq, re-matches, clojure.string/split, replace, replace-first >> > > We know with certainty that clojure.string/split is affected. Also, the > OP's question about how to use tooling to track down similar leaks in the > future does not appear to have been satisfactorily answered as of yet. > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Handling name collisions with clojure.core
On Thu, Sep 5, 2013 at 2:35 PM, Dave Ray wrote: > could you have a macro that rewrites code to use your ops? > > (require '[clojure.core.matrix :as m]) > (m/with-ops (+ ... (* ...) ...)) I wonder if there is value in a more general (with-ns clojure.core.matrix (foo (bar ...))) macro? I can imagine a couple cases where I could use something like that. -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: I am trying to use lein uberjar to package as jar file that can be executed. but it's not working for another library is not contained.
This isn't the solution you were looking for, but it should work;
1. Add simmetrics to your local maven repo;
$ mvn install:install-file
-Dfile=*simmetrics_jar_v1_6_2_d07_02_07.jar* -DgroupId=simmetrics
-DartifactId=simmetrics -Dversion=1.6.2 -Dpackaging=jar -DgeneratePom=true
2. Refer to it in :dependencies of your project.clj file;
[simmetrics/simmetrics "1.6.2"]
3. Done :)
lein uberjar
On Thu, Sep 5, 2013 at 1:20 PM, Rooney wrote:
> The another library is just jar file what used to use at java.
>
> the following is my project file.
>
> the red part is problem that is not contained from jar file when i use
> lein uberjar command.
>
> (defproject make-sentence "0.1.0-SNAPSHOT"
> :description "FIXME: write description"
> :url "http://example.com/FIXME";
> :license {:name "Eclipse Public License"
> :url "http://www.eclipse.org/legal/epl-v10.html"}
> :dependencies [[org.clojure/clojure "1.5.1"]
> [org.clojure/java.jdbc "0.0.6"]
> [org.xerial/sqlite-jdbc "3.7.2"]
> [seesaw "1.2.2"]]
> :plugins [[lein-swank "1.4.4"]
> [lein-localrepo "0.5.2"]]
> :java-source-paths ["java"]
> * :resource-paths ["resource/simmetrics_jar_v1_6_2_d07_02_07.jar"]*
> :main make-sentence.core
> )
>
>
> How can i solve?
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: core.async - handling nils
On Wed, Aug 28, 2013 at 12:18 PM, guns wrote: > Oh, I was confused; I was thinking about sentinel values in user code. > Yes, I imagine a single private (Object.) would work just fine, with > very little overhead. > First, I'd hope that sentinel values would be handled by the back-end implementation, as we're seeing core.sync implemented on other systems like ZeroMQ already. Second, as (Object.) doesn't play nicely over the wire, a random UUID or similar value would be much preferred. Third, I'd recommend reviewing, http://clojure.com/blog/2013/06/28/clojure-core-async-channels.html to understand why core.async is not just a better queue. Fourth, if you dislike how core.async works, just wrap it in your own library so it works the way you'd like. It looks like "core.async-with-nil" is available on Clojars. ;) With nil, without nil, it's just bike shedding. Clojure gives you the freedom to do it the way you want. -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: lazy seq from reducers with core.async
On Mon, Aug 26, 2013 at 1:37 PM, Timothy Baldridge wrote: > Since reducers use fork-join pools, Reducers use multiple threads and fork-join pools when called with "fold" on vectors (anything else?), not "reduce". By making the single producer thread of the reducer block on writes, per Jozef's code, he's making it act like a lazy-seq. If fold was used, then something similar to the code snippet I provided may be more appropriate. It would really depend on the use-case though. -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: lazy seq from reducers with core.async
Here is something similar pulled from bagotricks "1.5.2", using Java's linked blocking queue; https://github.com/thebusby/bagotricks/blob/master/src/bagotricks.clj#L204-L238 It uses fold instead of reduce to run in parallel, so has a slightly different use case than above. On Mon, Aug 26, 2013 at 10:14 AM, Mikera wrote: > Nice idea Jozef! > > Hmmm this is another example of why nil-as-end-of-channel is a > slightly problematic design decision for core.async: it makes this kind of > code much more fiddly. > > > On Monday, 26 August 2013 01:47:14 UTC+8, Jozef Wagner wrote: >> >> Hi, >> >> A distinctive feature of reducers is that "reducing is a one-shot thing". >> The common understanding is that reducers are fast for cases where you want >> to process whole collection at once, but for infinite and lazy seqs, you >> have to use good old seqs. >> >> With core.async, it is now easy to create a transformation which produces >> lazy seq from any reducible collection. >> >> (defn lazy-seq* >> [reducible] >> (let [c (chan) >> NIL (Object.) >> encode-nil #(if (nil? %) NIL %) >> decode-nil #(if (identical? NIL %) nil %)] >> (thread >>(reduce (fn [r v] (>!! c (encode-nil v))) nil reducible) >>(close! c)) >> (take-while (complement nil?) (repeatedly #(decode-nil (> >> (def s (lazy-seq* (clojure.core.reducers/map inc (range >> >> (first s) >> >> (take 100 s) >> >> This approach can be also extended to produce chunked seqs and chan >> buffer can also be used to further tune the performance. >> >> JW >> >> -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: 'foldcat' slower than 'mapv' even for 10,000 elements
On Wed, Jun 19, 2013 at 4:03 PM, Tassilo Horn wrote: > I might be wrong, but I think reducers are only faster in situations > where you bash many filters/maps/mapcats/etc on top of each other. > Or use fold on a large vector with a multi-core machine. You might try fold-into-vec to see if there is any difference; (defn fold-into-vec [coll] "Provided a reducer, concatenate into a vector. Note: same as (into [] coll), but parallel." (r/fold (r/monoid into vector) conj coll)) -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Parallel let form
Hi All, You may want to take a look at Prismatic's graph library, it does what you've described above in a slightly different way. Link: https://github.com/Prismatic/plumbing On Sun, Apr 28, 2013 at 1:04 AM, Ben Wolfson wrote: > It's not too hard, though, to write a plet form that first does some > analysis on the bindings to see which of them depend on which others. I > believe I mentioned this elsewhere in the group as an example of why a > macro might want to macroexpand its arguments, because doing that is the > simplest way to find out where you've got a binding form. In: > > (plet [a 2 b (my-mystery-macro [a 5] a)] b) > > if my-mystery-macro introduces a binding for 'a', then the expressions can > be evaluated in parallel, since 'a' isn't free in the expression being > bound to 'b'; otherwise, 'a' *is* free in that expression and refers to the > earlier binding, and the two expressions can't be evaluated in parallel. > > > > On Sat, Apr 27, 2013 at 8:59 AM, Ben Wolfson wrote: > >> "guv" is broken if your let form introduces bindings that depend on >> earlier bindings: >> >> user=> (plet [a 2 b a] b) >> CompilerException java.lang.RuntimeException: Unable to resolve symbol: a >> in this context, compiling:(NO_SOURCE_PATH:1) >> >> user=> (clojure.pprint/pprint (macroexpand-1 '(plet [a 2 b a] b))) >> (clojure.core/let >> [G__364 >> (clojure.core/future 2) >> G__365 >> (clojure.core/future a) ;;; oops! >> a >> @G__364 >> b >> @G__365] >> b) >> nil >> user=> >> >> In fact, both of them are broken in this way. >> >> >> >> On Sat, Apr 27, 2013 at 6:55 AM, Glen Mailer wrote: >> >>> Hi All, >>> >>> I was recently looking at how to make better use of parallelisation for >>> simple tasks in my compojure app, I had a construction similar to the >>> following: >>> >>> (views/some-view (api/api-call-1) (api/api-call-2) (api/api-call-3)) >>> >>> It seemed that the easiest way to introduce some parallelism here would >>> be in the style of a let form: >>> >>> (let [result-1 (api/api-call-1) >>> result-2 (api/api-call-2) >>> result-3 (api/api-call-3)] >>> (views/some-view result-1 result-2 result-3) >>> >>> There doesn't appear to be anything in core that does, this - after a >>> brief discussion in the IRC channel, I received the following two >>> suggestions: https://gist.github.com/jcromartie/5459350 and >>> https://gist.github.com/guv/5459364 >>> >>> I ended up going with the approach from "guv", as I understood it better >>> - and I moved the let form inside the view function to cut down on the >>> repetition a bit. >>> >>> Now, to my actual questions: >>> >>> What are the differences between the pmap approach and the futures >>> approach? >>> >>> And would a construction like this be useful in core? If so, how does it >>> potentially get there? >>> >>> >>> Thanks >>> Glen Mailer >>> Budding Clojurer >>> >>> >>> -- >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> >> >> -- >> Ben Wolfson >> "Human kind has used its intelligence to vary the flavour of drinks, >> which may be sweet, aromatic, fermented or spirit-based. ... Family and >> social life also offer numerous other occasions to consume drinks for >> pleasure." [Larousse, "Drink" entry] >> >> > > > -- > Ben Wolfson > "Human kind has used its intelligence to vary the flavour of drinks, which > may be sweet, aromatic, fermented or spirit-based. ... Family and social > life also offer numerous other occasions to consume drinks for pleasure." > [Larousse, "Drink" entry] > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You re
Re: Reducers newbie question
On Sat, Apr 27, 2013 at 10:01 PM, Stanislav Yurin wrote: > By the way, fold function has [n combinef reducef coll] implementation, > where n is number of elements collection is folded by. 512 is just the > default. > Yep I misspoke there, but it is still one of the tricks to be aware of. Lots of people do; (->> (range 100) vec (r/filter even?) (r/fold +)) And wonder why it's not running in parallel. -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Reducers newbie question
On Sat, Apr 27, 2013 at 7:35 PM, Stanislav Yurin wrote: > Actually, what I was trying to do, is to prototype multithreaded i/o > operation via reducers. And then use fold to regulate number of concurrent > operations. > But now something tells me I am doing not very clever thing. > I'm not entirely sure what you mean above, but I've been very happy using reducers with I/O so far. Reducers just have a few tricks you need to be aware of first; 1. You wont get parallel processing unless the input is a vector. 2. Each thread gets ~512 elements each, so reducing a vector of 800 elements will only use two cores. 3. How you aggregate your final result can greatly impact performance. (Ex. (r/fold +) is fast, fold-into-vec is slower, etc) I wrote a library to use reducers over text files (input) and have found it to be invaluable for working with giant TSV files. Link: https://github.com/thebusby/iota/ I often then use fold-into-lazy-seq to write the output back to a file. Link: https://gist.github.com/thebusby/5472980 Hope this helps! ;) -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Reducers newbie question
Some additional pointers here (this is a little old though); http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html On Fri, Apr 26, 2013 at 11:51 PM, László Török wrote: > Hi, > > Not sure what you are trying to do, but xxx is a lazy seq, thus it can > only be consumed sequentially and fold falls back to reduce > > You need a vector. > > Las > > Sent from my phone > On Apr 26, 2013 4:46 PM, "Stanislav Yurin" wrote: > >> I was assuming that following code will fold in parallel, but it is >> reduced sequentially >> >> (require '[clojure.core.reducers :as r]) >> (defn test1 >> [x] >> (Thread/sleep 1000) >> (println (str "Finished:" x)) >> x) >> (def xxx (r/map test1 (range 100))) >> (r/fold + xxx) >> >> What am I doing wrong? >> Thanks. >> >> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Clojure/West 2013 videos?
On Mon, Mar 25, 2013 at 12:24 PM, Rich Morin wrote: > I can wait a bit for the editing; > clean results are more important than saving a month or so. > I wouldn't say anything if it was only a month, it's actually closer to 3-7 months after the conference though. Clojure West 2012 was held March ~17th, the last video was posted on InfoQ October 12th. The bulk of the videos were released over the summer. I'd rather have iPhone videos, or just audio; with a link to the slides on github for $50 than wait 6+ months. -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Clojure/West 2013 videos?
On Sat, Mar 23, 2013 at 2:48 PM, Alex Miller wrote: > The benefit to attendees and non-attendees is that the videos exist at all > - without the InfoQ deal, the cost of recording, editing, and hosting > videos is literally the difference between whether the conference is in the > red or black. For attendees, I do really wish that I could provide talks > sooner just to you and I continue to discuss options for that with InfoQ. Just a thought, but I know a few conferences where the videos are available for sale after the conference (Strataconf?). I know I'd be happy to pay $50-100 for the timely videos instead of waiting 3-12 months after the conference when I can't attend. -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On Tue, Mar 12, 2013 at 11:00 PM, Paul Butcher wrote: > On 12 Mar 2013, at 13:49, Adam Clements wrote: > > How would feeding a line-seq into this compare to iota? And how would that > compare to a version of iota tweaked to work in a slightly less eager > fashion? > > > It'll not suffer from the problem of having to drag the whole file into > memory, but will incur the overhead of turning everything into JVM data > structures that iota avoids. I don't imagine that it would be hard to > modify iota to use a similar approach though. Although I imagine that > Alan's better placed to have an opinion on that :-) > If you're dealing with a stream that you'll only read once, then Paul's foldseq+line-seq should work much more effectively than Iota I'd think. Iota generates an index of line numbers so (nth iota-vec 100) is O(1), which is why it'll read through the entire file on loading. If you only need to access each line once, then that initial indexing step is likely a waste. If Paul wouldn't mind I'd like to add a a similar "seq" function to Iota that would allow for index-less processing like he did in foldable-seq. -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: fold over a sequence
On Mon, Mar 11, 2013 at 9:40 AM, Paul Butcher wrote: > I'm currently working on code that processes XML generated by > clojure.data.xml/parse, and would love to do so in parallel. I can't > immediately see any reason why it wouldn't be possible to create a version > of CollFold that takes a sequence and "chunks" it so that it can be folded > in parallel. Has anyone tried this yet? Is there some gotcha lurking to > catch me out? I've done something like the above here; https://github.com/thebusby/iota/blob/master/src/clj/iota.clj I haven't run in to any gotcha's yet after 6+ months of extensive use. I did have to copy/paste the definitions for pool, fjtask, fjinvoke, fjfork, fjjoin from core/reducers.clj and keep them from being AOT compiled (for Java 1.7 and 1.6 compatibility). I'm not pleased with the way the thread pool is inaccessible in core/reducers.clj because I'm not aware of any mechanism to have fold *NOT* use every available resource. Hope this helps, Alan -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ANN: Iota 1.0.2 release (Reducers for text files)
Hi Bernard, I'm going to see if I can add some performance metrics in the test code to try and determine where various bottlenecks may exist. If that goes well I'll see if I can add a branch that returns Byte[] instead of String for iota/vec. I suspect this will just require modifying iota.FileVector.getChunk() and iota.FileVector.cachedChunk . I'm still not clear on a good way to handle ByteBuffer with offsets though, because Java's mmap() doesn't support reading from files greater that Integer/MAX_VALUE. To work around this I've had to patch the following together; https://github.com/thebusby/iota/blob/master/src/java/iota/Mmap.java I do think I understand what you're asking for though. Unfortunately for this level of tuning I still primarily fall back to C and JNI, so my Java/Clojure is likely a little weak here. Regarding why iota/vec returns nil on empty lines, it's to allow for (->> (iota/vec "filename.tsv") (r/filter identity) ... Hope this helps, Alan On Sat, Mar 9, 2013 at 9:41 AM, bernardH wrote: > Hi Alan, > > > > On Friday, March 8, 2013 4:02:18 PM UTC+1, Alan Busby wrote: >> >> Hi Bernard, >> >> I'd certainly like to add support for binary files, but as I haven't had >> a need for it myself I haven't had a good place to start. >> >> As Java NIO's mmap() doesn't support ranges over 2GB, I've had to paste >> together multiple mmap's to cover files that are larger than 2GB. >> So if a record ended up spanning two mmap()'s, you couldn't return the >> raw data as a single object without copying it into a new buffer first. >> >> Also, if you provide a fixed record size in bytes for "doing the idx >> offset maths", why do you need the end idx for the current line as well? >> For example if you say file.bin is full of records each 100B in size, and >> you ask for the 10th record; don't you already know that the length of the >> record is 100B? >> >> > Indeed, the correlation between txt/binary and char (i.e \n) > delimited/fixed length record is very strong. However in my case I want to > first handle a \n delimited (txt) file as binary for performance reasons. > The context is that I have to consider all the lines of data, but might > not have to do "heavy" processing on all of them, so I want to do as few > work as possible on each line (i.e. not construct any java.lang.String). > This is in no way Clojure specific, I have two implementations in Java of > a small Minimum Spanning Tree program : > - one is constructing Strings from all the lines: > https://www.refheap.com/paste/12312 > - one is using offsets from a raw ByteBuffer : > https://www.refheap.com/paste/12313 > > As most of the lines are not really processed (just sorted according to > the last field), being able to only peek at the relevant bytes instead of > constructing full blown java.lang.Strings is a huge performance boost. > FWIW, as far as performance i concerned, I draw the line not between > Clojure and Java but between objects (constructed by copying some data > somewhere on the heap) and arrays of primitive data types, because > nowadays, cache locality trumps everything (once you got rid of reflection > calls in Clojure, obviously). > > So ideally, maybe 2 x 2 combinations (String / offset in ByteArray) x > (char delimited / fixed length) would be needed to cover all the needs. > > Thanks again for sharing your library ! > > Cheers, > > Bernard > > PS: Is there a rationale for returning nil instead of empty String "" on > empty lines with iota/vec? > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ANN: Iota 1.0.2 release (Reducers for text files)
Hi Bernard, I'd certainly like to add support for binary files, but as I haven't had a need for it myself I haven't had a good place to start. As Java NIO's mmap() doesn't support ranges over 2GB, I've had to paste together multiple mmap's to cover files that are larger than 2GB. So if a record ended up spanning two mmap()'s, you couldn't return the raw data as a single object without copying it into a new buffer first. Also, if you provide a fixed record size in bytes for "doing the idx offset maths", why do you need the end idx for the current line as well? For example if you say file.bin is full of records each 100B in size, and you ask for the 10th record; don't you already know that the length of the record is 100B? Hope I can help, Alan On Fri, Mar 8, 2013 at 1:44 AM, bernardH wrote: > Hi, > > > On Wednesday, March 6, 2013 2:53:26 PM UTC+1, Alan Busby wrote: >> >> >> With the release of Clojure 1.5 and it's new reducers, I figured this >> would be a good time to release a library to help with file IO for >> reducers. As reducers can only operate in parallel over specific >> collections, like vec, it requires some work to use them against files. >> >> Iota wraps a text files using Java NIO's mmap() so you can treat files >> larger than memory like a vector. Clojure's normal collection functions >> (first, last, nth, etc) work, while enabling 1.5's reducer's to operate in >> parallel for fold. >> > > Thanks, I already love your lib ! > However, in order to squeeze the last bits of performance, what I'd really > need would to be directly access the raw data as byte arrays. > The ideal API would provide my function with things looking and quaking > like bytes arrays but doing the idx offset maths to directly access the > maped data (read-only ), the next-to ideal would provide my function the > ref to whole bytes array and give me the start and end idx for current line. > > (I'd like to port a Java code where the most significant performance boost > was avoiding Strings creation when at all possible, which was possible by > only peeking at most of the lines). > > Cheers, > > B. > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
ANN: Iota 1.0.2 release (Reducers for text files)
Hi all, With the release of Clojure 1.5 and it's new reducers, I figured this would be a good time to release a library to help with file IO for reducers. As reducers can only operate in parallel over specific collections, like vec, it requires some work to use them against files. Iota wraps a text files using Java NIO's mmap() so you can treat files larger than memory like a vector. Clojure's normal collection functions (first, last, nth, etc) work, while enabling 1.5's reducer's to operate in parallel for fold. Code is available on Github here: https://github.com/thebusby/iota Library is available on Clojars here: https://clojars.org/iota Hope it helps, Alan Busby -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: reduce, reduce-kv, map, mapv, reducers/map and nil
On Tue, Jan 8, 2013 at 12:18 AM, Wolodja Wentland wrote: > On Thu, Nov 01, 2012 at 22:34 +0900, Alan Busby wrote: > > On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland > wrote: > > > > Oh, fold-into-map and fold-into-map-with would be wonderful and I > tried to > > > implement the former along the lines of fold-into-vec, but the > performance was > > > abysmal. I am now using fold-into-vec + r/map with zipmap which is > better, but > > > I wouldn't consider that optimal. > > > > The bottleneck with fold-into-map is that merge just doesn't scale up > > well and you end up doing a significant amount of merging. > > Currently a merge operation takes elements one at a time from one map, > > and adds them one at a time to another. > > > > This is incredibly wasteful for large maps as you already have two > > "pre-sorted" tries, and can pair each branch together recursively. If > > each branch node stored the number of child nodes, then you can assign > > different threads to work on different branches as well. This would be > > perfect for reducers, but from a quick look it didn't appear that any > > of the key internals were exposed to be taken advantage of. > > > > Unfortunately I haven't discovered a way to facilitate more efficient > > merging without modifying the actual Clojure core source. I guess this > > is my next step. > > Just stumbled over this old thread again and wondered if you found time to > look into this again? > I've done a good bit of work here but haven't finished anything yet unfortunately. In the meantime I've been working on other reducer utility functions/libraries like, 1. fold-into-lazy-seq ( https://gist.github.com/4479877 ) 2. Iota, for treating large text files as input for a reducer ( https://github.com/thebusby/iota ) Unfortunately some changes in reducers between alpha2 and beta1 have slowed performance down a bit, so I've been waiting for 1.5 to be released before spending too much time on it. I guess all of the above is motivation for me to finally send in my Clojure license stuff... -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: reduce-kv incompatible with subvec
Just curious why something like the following wouldn't be part of subvector?
public Object kvreduce(IFn f, Object init){
for(int i=0;i<(end - start);i++){
init = f.invoke(init,i,v.nth(start+i));
return init;
}
On Fri, Dec 28, 2012 at 11:23 PM, Andy Fingerhut
wrote:
> I am not saying that one would *have* to give up O(1) subvec in order to
> support other operations.
>
> I am guessing, without having done a thorough analysis, that an O(log n)
> subvec based on RRB Trees would make most/all operations on subvec's just
> fall out fairly naturally, rather than having to be written as special
> cases.
>
> Andy
>
> On Dec 27, 2012, at 8:54 PM, Alan Busby wrote:
>
> I'm confused why we'd need to give up O(1) just to support something like
> reduce-kv on subvectors.
> Isn't the implementation of subvector just a wrapper around the original
> vector along with a start and end value?
>
> Current source here;
>
> https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/APersistentVector.java
>
> I apologize if I'm missing something key here.
>
>
> On Fri, Dec 28, 2012 at 3:18 AM, Andy Fingerhut
> wrote:
>
>> http://dev.clojure.org/jira/browse/CLJ-1082
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: reduce-kv incompatible with subvec
I'm confused why we'd need to give up O(1) just to support something like reduce-kv on subvectors. Isn't the implementation of subvector just a wrapper around the original vector along with a start and end value? Current source here; https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/APersistentVector.java I apologize if I'm missing something key here. On Fri, Dec 28, 2012 at 3:18 AM, Andy Fingerhut wrote: > http://dev.clojure.org/jira/browse/CLJ-1082 -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: reduce, reduce-kv, map, mapv, reducers/map and nil
On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland wrote: > It seems to me as if we are currently figuring out which (boilerplate?) > functions are missing in reducers.clj and that we will have a nice and > well-integrated library at the end. To be fair, it's in beta and it's open source; so if anyone thinks it needs something they can always volunteer. :) >> > P.S. Would it be possible to have something like fold-into-vec in >> > clojure.reducers? >> >> Don't forget fold-into-map and fold-into-map-with, but both of those >> will likely require a better merge/merge-with function for maps. :( > > Oh, fold-into-map and fold-into-map-with would be wonderful and I tried to > implement the former along the lines of fold-into-vec, but the performance was > abysmal. I am now using fold-into-vec + r/map with zipmap which is better, but > I wouldn't consider that optimal. The bottleneck with fold-into-map is that merge just doesn't scale up well and you end up doing a significant amount of merging. Currently a merge operation takes elements one at a time from one map, and adds them one at a time to another. This is incredibly wasteful for large maps as you already have two "pre-sorted" tries, and can pair each branch together recursively. If each branch node stored the number of child nodes, then you can assign different threads to work on different branches as well. This would be perfect for reducers, but from a quick look it didn't appear that any of the key internals were exposed to be taken advantage of. Unfortunately I haven't discovered a way to facilitate more efficient merging without modifying the actual Clojure core source. I guess this is my next step. If you're interesting in using reducers over text files, I've uploaded a library to facilitate that here; http://github.com/thebusby/iota I'll be cleaning up the code with more documentation and examples at some point in the near future, and will send out an announcement then. In the mean time though feel free to use it. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: reduce, reduce-kv, map, mapv, reducers/map and nil
On Mon, Oct 29, 2012 at 10:00 PM, Wolodja Wentland wrote: > I find this behaviour quite unfortunate because I now have to explicitly test > for nil? and ensure consistent behaviour. This inconsistency violates the > principle of least-surprise and I am not sure if the current behaviour is > intentional or merely an unfortunate implementation detail. fold wont work in parallel on list/seqs/etc, so if you're trying to get the improved threading performance out of fold/fold-into-vec you'll have to supply a vector. (vec nil) => [] Then; (->> data vec (r/map inc) fold-into-vec) Unfortunately if you "vec" a vector, it'll actually do the work so you may want; (defn ensure-vec [xs] "Ensure's that the value provided is a vector" (if (= (type xs) clojure.lang.PersistentVector) xs (vec xs))) > P.S. Would it be possible to have something like fold-into-vec in > clojure.reducers? Don't forget fold-into-map and fold-into-map-with, but both of those will likely require a better merge/merge-with function for maps. :( P.S. As soon as I can find a moment, I can provide a JVM friendly library for reducing over mmap'ed text files as well. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [emacs over ssh limitations]
On Thu, Aug 30, 2012 at 1:13 AM, Phil Hagelberg wrote: > Alan Busby writes: >> To support what Tim said, after killing an afternoon I got iTerm2 from >> OSX to play nice with an Emacs in gnu screen on a remote Linux host. >> >> All keys and combos were working by the end. I don't have my laptop >> near me to check now, but I think making *EVERY* setting along the >> path as "xterm" is what finally did the trick. > > Doesn't setting TERM=xterm restrict you to 8 colors? I have to use > TERM=xterm-256color. Evidently it does. I guess that's a trade off then, but I'm curious what you're doing that requires all those colors in a terminal? If it's not pink text with a purple background, I think pics would be beneficial. ;) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [emacs over ssh limitations]
On Tue, Aug 28, 2012 at 10:29 PM, Stuart Sierra wrote: > SSH in iTerm 2 from an OS X machine to a Linux server. $TERM is > "xterm-256color" at both ends. We use this for pair-programming, so X and > tramp are not helpful. To support what Tim said, after killing an afternoon I got iTerm2 from OSX to play nice with an Emacs in gnu screen on a remote Linux host. All keys and combos were working by the end. I don't have my laptop near me to check now, but I think making *EVERY* setting along the path as "xterm" is what finally did the trick. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Let's see how fast we can make this
On Sat, Dec 25, 2010 at 11:28 AM, David Nolen wrote: > On Fri, Dec 24, 2010 at 8:19 PM, David Nolen wrote: > >> On OS X at least the following program shows identical performance to the >> JVM using 64 bit integers, ~2000 nanoseconds on my machine. So Clojure is >> not too far behind. >> > > w/o any GCC optimizations of course. With O2, the C is twice as fast on > this particularly micro microbenchmark. > On my "newish" Macbook I compiled and tested your code. I changed the thousand loop to a million loop and your code went from ~1000ns to ~600ns w/ O2 optimization. > In anycase, Clojure is no slouch :) > I agree 100%, but I've been surprised to find such significant performance improvements in my own code by incorporating JNI for certain hot spots. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Let's see how fast we can make this
On Fri, Dec 24, 2010 at 7:10 PM, Devrim Baris Acar wrote: > I guess it would be a better guess if you could includethe cpu type/speed > for a rough reference... It was on an Intel Xeon E5410 (2.33GHz), though like others have already said there are a number of factors that affect performance. I was simply curious of the relative speeds of the various code snippets on a production server. Also, I tried the C version using unsigned long long ints (64bit) and it was roughly the same speed. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Let's see how fast we can make this
Hi All,
On Fri, Dec 24, 2010 at 4:32 PM, Meikel Brandmeyer wrote:
> Most interesting is also the relation between the different versions on the
> given machine. Just the numbers of one algorithm aren't really comparable, I
> guess. (different machine, different load, different phase of moon, who
> knows...)
>
Did the below just for my own amusement on a 64bit linux box, and thought it
might be interesting to others. The fastest implementation was using areduce
with 1.3.0-alpha and unchecked ops for 2800ns. Unoptimized C was doing 700ns
though, and with -O2 it was 400ns. Reminds me why JNI can be so valuable
sometimes.
/* -O0 low of 700ns
-02 low of 400ns */
int char_count(char* ptr)
{
int count = 0;
int i;
for(i=0; ptr[i]; ptr++)
if(ptr[i]!=32)
count++;
return count;
}
;; 1.2.0 low of 65000ns
;; 1.3.0-alpha3 low of 46000ns
;;
(defn count-num-chars-v1 [^String s]
(loop [s s acc 0]
(if (seq s)
(recur (rest s) (if (= (first s) \space) acc (inc acc)))
acc)))
;; 1.2.0 low of 6200ns
;; 1.3.0-alpha3 low of 4800ns
;;
(defn count-num-chars-v2 [^String s]
(let [len (.length s)
space (int 32)]
(loop [i (int 0), c (int 0)]
(if (< i len)
(recur
(inc i)
(if (== (.codePointAt s i) space)
c
(unchecked-inc c)))
c
;; 1.2.0 low of 12500ns
;; 1.3.0-alpha3 low of 4450ns
;; 1.3.0-alpha3 with *unchecked-math* enabled low of 2800ns
;;
(defn count-num-chars-v3 [^String s]
(let [as (.toCharArray s)]
(areduce as idx acc 0 (if (= (int (aget as idx)) (int \space)) acc
(inc acc)
;; 1.2.0 untested
;; 1.3.0-alpha3 low of 4500ns
;;
(defn count-num-chars-v4 ^long [^String s]
(let [l (.length s)
c \space]
(loop [i 0 acc 0]
(if (< i l)
(recur (inc i)
(if (identical? (.charAt s i) c) acc
(inc acc)))
acc
Hope this helps,
Alan
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: removing parethesis
On Tue, Apr 13, 2010 at 11:54 AM, Douglas Philips wrote: > On 2010 Apr 12, at 10:48 PM, Glen Rubin wrote: > >> I am working with a collection of integer sequences ...(e.g. >> coll: >> ((3 7 3 5 9 2 0 8 4 0 1 2323 4 11...) (243 4 664 478 3948...) (6 3 7 4 >> 3335 2 4 5 7 6...)...) >> >> I want to create an average sequence such that all of the first >> elements are averaged, all of the second elements, etc >> >> However, I am unable to manipulate the collection. If I try something >> simple like: >> >> (map + coll) >> >> I get java.lang.ClassCastException >> > > Take a look at apply, it sounds as if you want something akin to: > (apply map + coll) > > --Doug Or maybe? Further help lurking in Clojure IRC. ;) user=> (use '[clojure.contrib.seq-utils :only (flatten)]) nil user=> (map + (flatten '((1 2 3) (4 5 6 (1 2 3 4 5 6) -- TheBusby -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe, reply using "remove me" as the subject.
Re: Funding Clojure 2010
I'd just like to second the request for selling a CD with Clojure 1.0 on it. No support, no additional features; just a CD with the Clojure jar file or something. I'd even go a step further and have multiple versions that would be identical except for the disc label, Gold, $1000 Silver, $500 Bronze, $100 Rich could order a big box of CDs with 1.0 on them, and then it would be easy for people to purchase them for the office and expense them like any other software. Many companies wouldn't bat an eyelash at paying $$$ for great software that improves productivity, but getting a large organization to donate to something can be *very* difficult. Just my $0.02, Alan -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: clojure vs scala
On Wed, Aug 26, 2009 at 5:43 AM, npowell wrote: > > I mean, I didn't think the article was terribly in depth, but a real, > evenhanded comparison would be enlightening. Reducing it further, I'd be interested just to hear more about the contrast of static typing versus macros. Which is more beneficial for different situations and why? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Convincing others about Clojure
Make them watch the following video. http://ocaml.janestreet.com/?q=node/61 Although the video isn't about Clojure, I think most of the points regarding ML are true of Clojure as well. On Thu, Jun 25, 2009 at 3:50 PM, Timothy Pratley wrote: > > > 1. How do you get Clojure programmers? Lisp is not for the faint hearted. > > Clojure is extremely easy to learn compared to 'Lisp': > You get collection based functions that apply other functions map/ > reduce/filter > You get simple powerful data structures hashmaps sets vectors lists > You get atomic, thread safe mutables > You can write Java without all the crud. > > > > 2. What about the performance of Clojure? Is it fast? > > Its not the fastest. But I've written much slower C code because > searching an array linearly is easier than 'doing it right' and I have > a deadline. I'd say with a sensible language you actually have time to > focus on performance. With the 'fastest' language sometimes I'm just > fighting to make things work. > > > > 3. People who want to use this are more academically inclined and are > > not practical. This will make the whole project fail. > > Rich is not an academic > http://en.wikipedia.org/wiki/Rich_Hickey > He wrote Clojure as a better way of developing, and it is! > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
