Thank Ken, your suggestion solved my problem with the OOM exception.
I tried your suggestion to run it in parallel but I didn't see much
difference. Instead I called future on the let call and that helped
the performance.
On Dec 17, 2:55 pm, Ken Wesson wrote:
> On Fri, Dec 17, 2010 at 5:39 PM, c
On Fri, Dec 17, 2010 at 5:39 PM, clj123 wrote:
> (defn persist-rows
> [headers rows id]
> (let [mrows (transform-rows rows id)]
> (with-db *db* (try
> (apply insert-into-table
> :my-table
> [:col1 :col2 :col3]
> mrows)))
> nil ))
>
> (d
(defn persist-rows
[headers rows id]
(let [mrows (transform-rows rows id)]
(with-db *db* (try
(apply insert-into-table
:my-table
[:col1 :col2 :col3]
mrows)))
nil ))
(defn filter-data
[rows item-id header id]
(persist-rows
On Thu, Dec 16, 2010 at 09:19, clj123 wrote:
> Hello,
>
> I'm trying to insert in a database large number of records, however
> it's not scaling correctly. For 100 records it takes 10 seconds, for
> 100 records it takes 2 min to save. But for 250 records it
> throws Java Heap out of memor
You might be coming to near OOM with using in-memory processing but
don't know it, and the batched (lazy) version is probably holding onto
data creating the mem leak. Would you be able to post the relevant
source?
--
You received this message because you are subscribed to the Google
Groups "Cloj
Hello,
I'm trying to insert in a database large number of records, however
it's not scaling correctly. For 100 records it takes 10 seconds, for
100 records it takes 2 min to save. But for 250 records it
throws Java Heap out of memory exception.
I've tried separting the records processing
