Re: Weird performance issue with reduce
@Alex, I am using 1.6.0. Transducers is something i wanna try out. @Andy, I wasn't aware of YourKit. I have been using JProfiler and VisualVM. I liked it, seems smoother than the other two. Alex On Monday, November 17, 2014 8:31:25 PM UTC+2, Alex Miller wrote: > > What version of Clojure are you using? > > This seems like a use case where transducers could help significantly in > avoiding lazy effects and intermediate objects. > > On Monday, November 17, 2014 4:28:12 AM UTC-6, Alexander L. wrote: >> >> Hi all, >> >> I understand that the following question is a long shot without any >> proper proof/tests from my side but it's a little bit difficult to make a >> test case from the specific part of my app so I will just ask anyway in >> case anyone knows anything. >> >> The situation is like this: >> >>- I have a hashmap with *3386* items that I pass through few >>functions in order to append new keys or update existing ones to each >>hashmap entry. >>- Each hashmap item has 20 keys with various data types (mostly >>strings) >>- All my transformation functions use `reduce`. >> >> The problem: >> >> I have a top level function which I inside it I call 7 other functions >> (all written by me) and for some reason I haven't discovered is that it >> needs around 2 seconds to return a result even though the items aren't many. >> Now, i used `time` to benchmark each function and when I found which one >> is taking a lot of time to return, after I removed it, I discovered that >> the problem still existed but now moved on to a different function. >> >> I did a bunch of tests with those 3386 on the REPL and reduce but I >> didn't notice anything weird/slow so it must be a combination of things. >> Also, i doubt that this is a RAM problem, i have allocated 4GB for the JVM. >> >> So, my question is, has anyone every seen a situation like this with a >> bunch of `reduce` calls? Is there anything at all that I should check and >> maybe missed it? >> >> Regards >> >> Alex >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Weird performance issue with reduce
Thanks for the input Jozef. I just want to clarify that I am not doing any nested reduce calls within my code. Alex On Monday, November 17, 2014 1:00:43 PM UTC+2, Jozef Wagner wrote: > > I would look for unncessary object creation inside reduction function (use > transients) and lazy seq realization (realize only what you really need). > Calling reduce inside reduce (inside reduce...) may easily make the > complexity of your code exponential. Consider caching of intermediate > results if that's the case. > > You can also consider switching to reducers/transducers, which may clean > your design. In that case, the call to reduce should be as high in > hierarchy as possible, ideally in your top level function, with other > functions dealing with the transformation of 'step' values. > > Jozef > > On Mon, Nov 17, 2014 at 11:28 AM, Alexander L. > wrote: > >> Hi all, >> >> I understand that the following question is a long shot without any >> proper proof/tests from my side but it's a little bit difficult to make a >> test case from the specific part of my app so I will just ask anyway in >> case anyone knows anything. >> >> The situation is like this: >> >>- I have a hashmap with *3386* items that I pass through few >>functions in order to append new keys or update existing ones to each >>hashmap entry. >>- Each hashmap item has 20 keys with various data types (mostly >>strings) >>- All my transformation functions use `reduce`. >> >> The problem: >> >> I have a top level function which I inside it I call 7 other functions >> (all written by me) and for some reason I haven't discovered is that it >> needs around 2 seconds to return a result even though the items aren't many. >> Now, i used `time` to benchmark each function and when I found which one >> is taking a lot of time to return, after I removed it, I discovered that >> the problem still existed but now moved on to a different function. >> >> I did a bunch of tests with those 3386 on the REPL and reduce but I >> didn't notice anything weird/slow so it must be a combination of things. >> Also, i doubt that this is a RAM problem, i have allocated 4GB for the JVM. >> >> So, my question is, has anyone every seen a situation like this with a >> bunch of `reduce` calls? Is there anything at all that I should check and >> maybe missed it? >> >> Regards >> >> Alex >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Weird performance issue with reduce
Hi all, I understand that the following question is a long shot without any proper proof/tests from my side but it's a little bit difficult to make a test case from the specific part of my app so I will just ask anyway in case anyone knows anything. The situation is like this: - I have a hashmap with *3386* items that I pass through few functions in order to append new keys or update existing ones to each hashmap entry. - Each hashmap item has 20 keys with various data types (mostly strings) - All my transformation functions use `reduce`. The problem: I have a top level function which I inside it I call 7 other functions (all written by me) and for some reason I haven't discovered is that it needs around 2 seconds to return a result even though the items aren't many. Now, i used `time` to benchmark each function and when I found which one is taking a lot of time to return, after I removed it, I discovered that the problem still existed but now moved on to a different function. I did a bunch of tests with those 3386 on the REPL and reduce but I didn't notice anything weird/slow so it must be a combination of things. Also, i doubt that this is a RAM problem, i have allocated 4GB for the JVM. So, my question is, has anyone every seen a situation like this with a bunch of `reduce` calls? Is there anything at all that I should check and maybe missed it? Regards Alex -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: core.async and channel flushing
Sorry for bringing this back up, but I was wondering if anyone figured out something better... On Saturday, September 14, 2013 10:49:08 PM UTC+3, Alexander L. wrote: > > > I am developing an application and I use core.async to push data from > multiple threads within an infinite > > (go (while true >> (let [data (> (do processing here...))) > > > in order to be processed. As you probably already figured out, I store my > channel inside an atom defined at the start of my app. > > If an exception occurs in one of the threads, then I am calling a global > exception handler which will reset the atom that holds the channel in order > to flush it and make sure the processing of the data it contains will stop. > > Now, my question is, does anyone know a better way to flush a core.async > channel or the only way is to re-create the channel? > > Thank you for your time. > > Regards, > > Alexander > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
core.async and channel flushing
I am developing an application and I use core.async to push data from multiple threads within an infinite (go (while true > (let [data ( (do processing here...))) in order to be processed. As you probably already figured out, I store my channel inside an atom defined at the start of my app. If an exception occurs in one of the threads, then I am calling a global exception handler which will reset the atom that holds the channel in order to flush it and make sure the processing of the data it contains will stop. Now, my question is, does anyone know a better way to flush a core.async channel or the only way is to re-create the channel? Thank you for your time. Regards, Alexander -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
core.async and channel flushing
Hello all, I am developing an application and I use core.async to push data from multiple threads within an infinite (go (while true (let [data (http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Why is using (not (empty? coll)) not idiomatic?
I really liked your take on this *Brian*. You kinda convinced me to use (if (not-empty? foo)) from now on :) Alexander On Monday, May 27, 2013 2:58:38 AM UTC+3, Brian Marick wrote: > > > On May 26, 2013, at 5:47 AM, "Alex L." > > wrote: > > First, the use of seq as a > > terminating condition is the idiomatic way to test whether a sequence is > empty. > > In natural languages, idioms change. Sometimes it's to the despair of > purists: for example, I've had to accept that "hopefully" at the beginning > of a sentence doesn't act as an adverb: > > Hopefully, he will ascend to a higher plane. > Having consumed the HOPE1 drug, he will hopefully ascend to a higher > plane. > > The same is true of programming languages. When I was programming on the > PDP-11, it was idiomatic to use pre-increment instead of post-increment > when either would do: > > for (i=0; i for (i=0; i > That was because the former compiled into one machine language > instruction, but the latter required two. PDP-11s were slow, so it could > matter. > > There exists in 2013 a person who, in a code review, insists that every > post-increment be changed to a pre-increment, even though (1) compilers are > way smarter than they were in 1981, (2) computers are way faster too, and > (3) insisting on a stylistic point only relevant in the distant past is the > sign of a mind past its sell-by date. I've met that person. > > There is no one who understands `(if (seq thing)` who wouldn't understand > `(if (not (empty? thing))` or, better, `(if (not-empty? thing)`. The > converse is not true. That suggests that the latter should be the idiom, > given that the difference between them is as consequential as the > difference between `++i` and `i++`. > > It's fun to make use of esoterica like `seq`'s behavior with an empty > list. Back in the early days, it was necessary. Witness Guy Steele's > StrangeLoop talk that began with the need to get a program to fit onto a > single punched card. And language implementors still need to care about > those things. > > But, for the rest of us, the necessity has drained out of that kind of > esoterica. It's now more of a shibboleth, a way to identify yourself as one > of the tribe. That's actually tolerable human behavior, but those who > indulge in it shouldn't feel *smug*. Rather, the opposite. > > > Latest book: /Functional Programming for the Object-Oriented Programmer/ > https://leanpub.com/fp-oo > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
