On Friday, March 2, 2018 at 9:58:35 PM UTC+5:30, Rob Nikander wrote:
>
>
>
> On Friday, March 2, 2018 at 12:48:28 AM UTC-5, Daniel wrote:
>>
>> How do you know this code is causing the error? Unless this is all your
>> code does, descriptions of the error suggest the memory leak might be
>> coming from anywhere. This tight loop might trigger too many successive GCs
>> though: have you observed if the error only occurs under load, or under
>> large time intervals? Some combination?
>>
>
> Okay, thanks for looking it over. This is a single-threaded, short lived
> script, so this is the only thing happening. But it's possible there is an
> unexpected large result set. I'll look into that. It may be smarter to use
> `limit N` in my SQL, rather than the time range.
>
Rob,
While I could see the intent of the original code, I found it hard to
reason about where the problem might lie, given the loops in loops in loops.
Assuming the code is causing an error, I feel it will help to separate out
the db query, and factor the rest of the logic into a pipeline.
This way one could:
- profile the DB query and fix it separately
- isolate and check each part of the records transformation pipeline
- make more obvious what's going on (aid visual audit)
May I suggest something like this (If I've understood what the original
code is trying to achieve):
(defn query-recs
[db min-time max-time]
(jdbc/query db
["select id, a, b, c from sometable where t > ? and t <= ?
order by t"
min-time max-time]))
(define intervals
[min-time max-time]
(->> (iterate (partial plus-minutes min-t) 30)
(take-while #(before? max-time %))
(partition 2 1)))
(defn find-db-records [db min-time max-time]
(let [group-rows
(fn [min-t max-t]
(->> (query-recs db min-t max-t)
(partition-all 200)))
extract-relevant-records
(fn [[rows foo]]
(filter #(relevant-record? foo (:a %) (:b %))
rows))
proc-relevant-records
(fn [rows]
(map #(assoc % :d (computed-d (:a %) (:b %) (:c %)))
rows))]
(reduce (fn [rxs [min-t max-t]]
(->> (group-rows min-t max-t)
(map (juxt identity make-foo))
(map extract-relevant-records)
(mapcat proc-relevant-records)
(into rxs)))
[]
(intervals min-time max-time))))
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
