Re: Performance optimizations dealing with java collections
That's it! Thanks so much for the help! In what cases is a function
turned into a RestFn? variadic clojure or vararg java? I was confused
about how clojure is turned into java, but things are a little clearer now.
(make-array) uses java.lang.reflect.Array.newInstance, which is bad as
well. I'm going to try and push this interpolation down into the java API.
On Tuesday, June 18, 2013 3:00:23 PM UTC-7, John Hume wrote:
>
> Offhand it looks like the only RestFn you call from filter-link is
> clojure.core/format. Have you tried replacing that with something like this?
>
> (String/format (.get link 1) (doto (make-array String 1) (aset 0 (.get
> link 2)))
>
> I'm not suggesting that's idiomatic, but if it addresses the issue then
> you can focus on the difference and look for a happy medium.
>
>
> On Tue, Jun 18, 2013 at 4:10 PM, Tim Jones
> > wrote:
>
>>
>> (defn- filter-link
>> "Format lang and cc into the link returned from the product."
>> [^List link]
>> [(.get link 0) (.get link 1) (.get link 2) (.get link 3)
>>(-> (.get link 4)
>>(URLDecoder/decode "UTF-8")
>>(format (.get link 1) (.get link 2)))])
>>
>> (defn link-info
>> "Retrieve all link info from a product as a lazy-seq of vectors of
>> String. The function
>>handles parameter substitution within the URL"
>> [^Product p]
>> (map filter-link
>>(.getAllLinkInfo p @lib {"h_lang" "%1$s" "h_cc" "%2$s"})))
>>
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Performance optimizations dealing with java collections
Offhand it looks like the only RestFn you call from filter-link is
clojure.core/format. Have you tried replacing that with something like this?
(String/format (.get link 1) (doto (make-array String 1) (aset 0 (.get link
2)))
I'm not suggesting that's idiomatic, but if it addresses the issue then you
can focus on the difference and look for a happy medium.
On Tue, Jun 18, 2013 at 4:10 PM, Tim Jones wrote:
>
> (defn- filter-link
> "Format lang and cc into the link returned from the product."
> [^List link]
> [(.get link 0) (.get link 1) (.get link 2) (.get link 3)
>(-> (.get link 4)
>(URLDecoder/decode "UTF-8")
>(format (.get link 1) (.get link 2)))])
>
> (defn link-info
> "Retrieve all link info from a product as a lazy-seq of vectors of
> String. The function
>handles parameter substitution within the URL"
> [^Product p]
> (map filter-link
>(.getAllLinkInfo p @lib {"h_lang" "%1$s" "h_cc" "%2$s"})))
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Performance optimizations dealing with java collections
On Tuesday, June 18, 2013 9:58:20 AM UTC-7, Michael Klishin wrote:
>
> 2013/6/18 Tim Jones >
>
>> How do I get to near-java performance?
>
>
> Start by providing a snippet of your code and profiling.
>
> Great. Here's the context: iterate through a list of Product, and for
each Product, get a List> which is URL info for that product.
Convert each List to a vector and do a little processing on it.
Here is the code:
(defn- filter-link
"Format lang and cc into the link returned from the product."
[^List link]
[(.get link 0) (.get link 1) (.get link 2) (.get link 3)
(-> (.get link 4)
(URLDecoder/decode "UTF-8")
(format (.get link 1) (.get link 2)))])
(defn link-info
"Retrieve all link info from a product as a lazy-seq of vectors of
String. The function
handles parameter substitution within the URL"
[^Product p]
(map filter-link
(.getAllLinkInfo p @lib {"h_lang" "%1$s" "h_cc" "%2$s"})))
Here's a VisualVM screenshot. 500 products takes ~45s, with 18s spent in
one call:
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Performance optimizations dealing with java collections
Let's try this again. Maybe the question is what is the most performant way to consume a native java collection (List, array, or other) in clojure? What I've tried so far hits clojure-imposed performance issues. How do I get to near-java performance? I'm trying to create a polemic for clojure and "the java version of this program is 30% faster" isn't where I want to start. Am I missing something? On Monday, June 17, 2013 3:34:07 PM UTC-7, Tim Jones wrote: > > I'm working on a small clojure program which pulls data from a custom > memory-backed data store via a java api. When looking at performance, > there is a hotspot at the point of interaction with this API. One of the > fields of each record being exported contains a list of lists of strings > (List>) with about 60-150 items in the outer list and 5 items > in the inner list. The problem is that at the point of assimilating this > java data type into clojure, there are a couple calls taking ~30-40% of the > execution time. Here are the senarios; I'm just starting to get familiar > with clojure internals, so forgive obvious blunders in this. > > Processing it as: > > (map parse-field (.getAllFields datastore)) > > with parse-field defined like: > > (defn parse-field [^List field] ...) > > or (if I change the API call) > > (defn parse-field [^"[Ljava.lang.String;" field] ...) > > it calls clojure.lang.RestFn.invoke() which wants to build an ArraySeq, > which calls java.lang.Class.getComponentType, which is a killer (20 of 50 > seconds on 500 records). > > With parse-field using destructuring, on either the List or String array, > clojure.lang.RT.nthFrom() is invoked, which calls java.lang.Class.isArray ( > https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L773), > > which is a killer as well. > > So I inlined parse-field and used loop/recur (instead of map) and .get on > the list to get the next item, but this still results in the > RestFn.invoke()->ArraysSeq->Class.getComponentType path. > > This is related to http://dev.clojure.org/jira/browse/CLJ-1200, although > called via a different path. > > Does anyone have any ideas on how I could avoid these calls? > > Thanks, > > Tim > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Performance optimizations dealing with java collections
2013/6/18 Tim Jones > How do I get to near-java performance? Start by providing a snippet of your code and profiling. Most likely you are hitting either boxing or extra method calls that are not obvious from the code. But nobody can tell if that's really the case, esp. without seeing the actual code. Profiling will give you cold hard data. -- MK http://github.com/michaelklishin http://twitter.com/michaelklishin -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Performance optimizations dealing with java collections
I'm working on a small clojure program which pulls data from a custom memory-backed data store via a java api. When looking at performance, there is a hotspot at the point of interaction with this API. One of the fields of each record being exported contains a list of lists of strings (List>) with about 60-150 items in the outer list and 5 items in the inner list. The problem is that at the point of assimilating this java data type into clojure, there are a couple calls taking ~30-40% of the execution time. Here are the senarios; I'm just starting to get familiar with clojure internals, so forgive obvious blunders in this. Processing it as: (map parse-field (.getAllFields datastore)) with parse-field defined like: (defn parse-field [^List field] ...) or (if I change the API call) (defn parse-field [^"[Ljava.lang.String;" field] ...) it calls clojure.lang.RestFn.invoke() which wants to build an ArraySeq, which calls java.lang.Class.getComponentType, which is a killer (20 of 50 seconds on 500 records). With parse-field using destructuring, on either the List or String array, clojure.lang.RT.nthFrom() is invoked, which calls java.lang.Class.isArray (https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L773), which is a killer as well. So I inlined parse-field and used loop/recur (instead of map) and .get on the list to get the next item, but this still results in the RestFn.invoke()->ArraysSeq->Class.getComponentType path. This is related to http://dev.clojure.org/jira/browse/CLJ-1200, although called via a different path. Does anyone have any ideas on how I could avoid these calls? Thanks, Tim -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
