On Wed, Jul 15, 2015 at 12:02 PM, Daniel Higginbotham <
nonrecurs...@gmail.com> wrote:
> Surely reducers/map should be faster, but it doesn’t seem like I’ve done
> something wrong? This is driving me crazy :)
It is much faster, and runs in parallel, if you use fold.
Try this;
(defn fold-into-v
I wrote a library ( https://github.com/thebusby/iota ) to handle a very
similar issue, which I hope would be of some help to you here.
On Wed, May 6, 2015 at 12:18 PM, Sam Raker wrote:
> I've got two really big CSV files that I need to compare. Stay tuned for
> the semi-inevitable "how do I opti
On Mon, Oct 27, 2014 at 12:52 PM, Brian Craft wrote:
> Makes sense, but not an option for this application. What about something
> similar to iota, backed with byte arrays, or something?
As Patrick pointed out, if you're working directly with byte array's you
might want to use mmap which is wha
On Mon, Oct 27, 2014 at 7:10 AM, Brian Craft wrote:
> I found iota, which looks like a good solution for the read portion of the
> problem. However I also need to process the data in the file. If I start
> with an iota/vec and need to sort it, something like
>
> (sort (iota/vec "foo"))
>
Short d
On Tue, Sep 9, 2014 at 3:09 AM, Sean Corfield wrote:
> > I find that it is quite difficult to use clojure unless one knows Java,
> which I believe to be a barrier to new comers.
>
> I'm surprised every time I hear this. You can write a lot of Clojure
> without having to do any interop so you can
On Fri, Feb 21, 2014 at 8:51 AM, wrote:
> One note about your SuperVecs idea though, it seems that using that
> approach we could get a deeply nested tree to represent our a vector, and I
> think this is the exact thing vectors are trying to avoid..
In general I think this is true for vectors,
Gracenote office here in Shibuya if
there is enough interest. If schedules work out Mr. Ambrose may be able to
give us a quick intro / Q+A on core.typed as well.
Anyone interested?
Thanks,
Alan Busby
On Thu, Oct 3, 2013 at 2:32 PM, Dave Della Costa wrote:
> Speaking of, is there anythi
On Thu, Oct 3, 2013 at 9:30 AM, Marek Kubica wrote:
> On Tue, 1 Oct 2013 22:01:02 -0700 (PDT)
> Nicolas Modrzyk wrote:
>
> > The whole text has also been written in English, so I guess there
> > could be a chance to put it out there.
> > But we have had no contacts with US/UK publishers so far.
and makes a text file a foldable collection.
4. While pmap is great for advertising the power of Clojure, it's likely
safe to say that it should be ignored if you're actually looking for
performance.
Hope this helps,
Alan Busby
--
--
You received this message because you are subscribed
If you're working with large text files, tsv especially, you may encounter
Java's memory overhead for Strings as well.
I remember parsing a 1GB TSV file into EDN, and watching it consume 10GB of
RAM.
I'd recommend taking a look at the Iota library to see if it would be of
any help to you,
https://
On Thu, Sep 5, 2013 at 2:35 PM, Dave Ray wrote:
> could you have a macro that rewrites code to use your ops?
>
> (require '[clojure.core.matrix :as m])
> (m/with-ops (+ ... (* ...) ...))
I wonder if there is value in a more general (with-ns clojure.core.matrix
(foo (bar ...))) macro?
I can
This isn't the solution you were looking for, but it should work;
1. Add simmetrics to your local maven repo;
$ mvn install:install-file
-Dfile=*simmetrics_jar_v1_6_2_d07_02_07.jar* -DgroupId=simmetrics
-DartifactId=simmetrics -Dversion=1.6.2 -Dpackaging=jar -DgeneratePom=true
2. Refer to it in :
On Wed, Aug 28, 2013 at 12:18 PM, guns wrote:
> Oh, I was confused; I was thinking about sentinel values in user code.
> Yes, I imagine a single private (Object.) would work just fine, with
> very little overhead.
>
First, I'd hope that sentinel values would be handled by the back-end
implementa
On Mon, Aug 26, 2013 at 1:37 PM, Timothy Baldridge wrote:
> Since reducers use fork-join pools,
Reducers use multiple threads and fork-join pools when called with "fold"
on vectors (anything else?), not "reduce".
By making the single producer thread of the reducer block on writes, per
Jozef's co
Here is something similar pulled from bagotricks "1.5.2", using Java's
linked blocking queue;
https://github.com/thebusby/bagotricks/blob/master/src/bagotricks.clj#L204-L238
It uses fold instead of reduce to run in parallel, so has a slightly
different use case than above.
On Mon, Aug 26, 2013 a
On Wed, Jun 19, 2013 at 4:03 PM, Tassilo Horn wrote:
> I might be wrong, but I think reducers are only faster in situations
> where you bash many filters/maps/mapcats/etc on top of each other.
>
Or use fold on a large vector with a multi-core machine.
You might try fold-into-vec to see if there
Hi All,
You may want to take a look at Prismatic's graph library, it does what
you've described above in a slightly different way.
Link: https://github.com/Prismatic/plumbing
On Sun, Apr 28, 2013 at 1:04 AM, Ben Wolfson wrote:
> It's not too hard, though, to write a plet form that first does s
On Sat, Apr 27, 2013 at 10:01 PM, Stanislav Yurin wrote:
> By the way, fold function has [n combinef reducef coll] implementation,
> where n is number of elements collection is folded by. 512 is just the
> default.
>
Yep I misspoke there, but it is still one of the tricks to be aware of.
Lots o
On Sat, Apr 27, 2013 at 7:35 PM, Stanislav Yurin wrote:
> Actually, what I was trying to do, is to prototype multithreaded i/o
> operation via reducers. And then use fold to regulate number of concurrent
> operations.
> But now something tells me I am doing not very clever thing.
>
I'm not entir
Some additional pointers here (this is a little old though);
http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html
On Fri, Apr 26, 2013 at 11:51 PM, László Török wrote:
> Hi,
>
> Not sure what you are trying to do, but xxx is a lazy seq, thus it can
> only be consumed sequentia
On Mon, Mar 25, 2013 at 12:24 PM, Rich Morin wrote:
> I can wait a bit for the editing;
> clean results are more important than saving a month or so.
>
I wouldn't say anything if it was only a month, it's actually closer to 3-7
months after the conference though.
Clojure West 2012 was held Mar
On Sat, Mar 23, 2013 at 2:48 PM, Alex Miller wrote:
> The benefit to attendees and non-attendees is that the videos exist at all
> - without the InfoQ deal, the cost of recording, editing, and hosting
> videos is literally the difference between whether the conference is in the
> red or black. Fo
On Tue, Mar 12, 2013 at 11:00 PM, Paul Butcher wrote:
> On 12 Mar 2013, at 13:49, Adam Clements wrote:
>
> How would feeding a line-seq into this compare to iota? And how would that
> compare to a version of iota tweaked to work in a slightly less eager
> fashion?
>
>
> It'll not suffer from the
On Mon, Mar 11, 2013 at 9:40 AM, Paul Butcher wrote:
> I'm currently working on code that processes XML generated by
> clojure.data.xml/parse, and would love to do so in parallel. I can't
> immediately see any reason why it wouldn't be possible to create a version
> of CollFold that takes a seque
returns nil on empty lines, it's to allow for
(->> (iota/vec "filename.tsv")
(r/filter identity)
...
Hope this helps,
Alan
On Sat, Mar 9, 2013 at 9:41 AM, bernardH wrote:
> Hi Alan,
>
>
>
> On Friday, March 8, 2013 4:02:18 PM UTC+1, Alan Bus
dx for the current line as well?
For example if you say file.bin is full of records each 100B in size, and
you ask for the 10th record; don't you already know that the length of the
record is 100B?
Hope I can help,
Alan
On Fri, Mar 8, 2013 at 1:44 AM, bernardH wrote:
> Hi,
>
com/thebusby/iota
Library is available on Clojars here: https://clojars.org/iota
Hope it helps,
Alan Busby
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that p
On Tue, Jan 8, 2013 at 12:18 AM, Wolodja Wentland wrote:
> On Thu, Nov 01, 2012 at 22:34 +0900, Alan Busby wrote:
> > On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland
> wrote:
>
> > > Oh, fold-into-map and fold-into-map-with would be wonderful and I
> tried to
>
rally, rather than having to be written as special
> cases.
>
> Andy
>
> On Dec 27, 2012, at 8:54 PM, Alan Busby wrote:
>
> I'm confused why we'd need to give up O(1) just to support something like
> reduce-kv on subvectors.
> Isn't the implementation of sub
I'm confused why we'd need to give up O(1) just to support something like
reduce-kv on subvectors.
Isn't the implementation of subvector just a wrapper around the original
vector along with a start and end value?
Current source here;
https://github.com/clojure/clojure/blob/master/src/jvm/clojure/l
On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland wrote:
> It seems to me as if we are currently figuring out which (boilerplate?)
> functions are missing in reducers.clj and that we will have a nice and
> well-integrated library at the end.
To be fair, it's in beta and it's open source; so if any
On Mon, Oct 29, 2012 at 10:00 PM, Wolodja Wentland wrote:
> I find this behaviour quite unfortunate because I now have to explicitly test
> for nil? and ensure consistent behaviour. This inconsistency violates the
> principle of least-surprise and I am not sure if the current behaviour is
> intent
On Thu, Aug 30, 2012 at 1:13 AM, Phil Hagelberg wrote:
> Alan Busby writes:
>> To support what Tim said, after killing an afternoon I got iTerm2 from
>> OSX to play nice with an Emacs in gnu screen on a remote Linux host.
>>
>> All keys and combos were working by the
On Tue, Aug 28, 2012 at 10:29 PM, Stuart Sierra
wrote:
> SSH in iTerm 2 from an OS X machine to a Linux server. $TERM is
> "xterm-256color" at both ends. We use this for pair-programming, so X and
> tramp are not helpful.
To support what Tim said, after killing an afternoon I got iTerm2 from
OSX
On Sat, Dec 25, 2010 at 11:28 AM, David Nolen wrote:
> On Fri, Dec 24, 2010 at 8:19 PM, David Nolen wrote:
>
>> On OS X at least the following program shows identical performance to the
>> JVM using 64 bit integers, ~2000 nanoseconds on my machine. So Clojure is
>> not too far behind.
>>
>
> w/o a
On Fri, Dec 24, 2010 at 7:10 PM, Devrim Baris Acar wrote:
> I guess it would be a better guess if you could includethe cpu type/speed
> for a rough reference...
It was on an Intel Xeon E5410 (2.33GHz), though like others have already
said there are a number of factors that affect performance. I
Hi All,
On Fri, Dec 24, 2010 at 4:32 PM, Meikel Brandmeyer wrote:
> Most interesting is also the relation between the different versions on the
> given machine. Just the numbers of one algorithm aren't really comparable, I
> guess. (different machine, different load, different phase of moon, wh
On Tue, Apr 13, 2010 at 11:54 AM, Douglas Philips wrote:
> On 2010 Apr 12, at 10:48 PM, Glen Rubin wrote:
>
>> I am working with a collection of integer sequences ...(e.g.
>> coll:
>> ((3 7 3 5 9 2 0 8 4 0 1 2323 4 11...) (243 4 664 478 3948...) (6 3 7 4
>> 3335 2 4 5 7 6...)...)
>>
>> I want to
I'd just like to second the request for selling a CD with Clojure 1.0 on it.
No support, no additional features; just a CD with the Clojure jar file or
something.
I'd even go a step further and have multiple versions that would be
identical except for the disc label,
Gold, $1000
Silver, $500
Bronz
On Wed, Aug 26, 2009 at 5:43 AM, npowell wrote:
>
> I mean, I didn't think the article was terribly in depth, but a real,
> evenhanded comparison would be enlightening.
Reducing it further, I'd be interested just to hear more about the contrast
of static typing versus macros. Which is more benef
Make them watch the following video.
http://ocaml.janestreet.com/?q=node/61
Although the video isn't about Clojure, I think most of the points regarding
ML are true of Clojure as well.
On Thu, Jun 25, 2009 at 3:50 PM, Timothy Pratley
wrote:
>
> > 1. How do you get Clojure programmers? Lisp is
41 matches
Mail list logo
