Thank Ken, your suggestion solved my problem with the OOM exception. I tried your suggestion to run it in parallel but I didn't see much difference. Instead I called future on the let call and that helped the performance.
On Dec 17, 2:55 pm, Ken Wesson <kwess...@gmail.com> wrote: > On Fri, Dec 17, 2010 at 5:39 PM, clj123 <ariela2...@gmail.com> wrote: > > (defn persist-rows > > [headers rows id] > > (let [mrows (transform-rows rows id)] > > (with-db *db* (try > > (apply insert-into-table > > :my-table > > [:col1 :col2 :col3] > > mrows))) > > nil )) > > > (defn filter-data > > [rows item-id header id] > > (persist-rows > > header > > (filter #(= (:item_id %) item-id) rows) > > id)) > > > (dorun (pmap #(filter-data rows %1 header %2) > > items id )) > > Rows gets traversed repeatedly, for each item/id pair. So the head > gets held onto. > > To make this scale you're going to need to regenerate the rows seq for > each traversal; you need something like > > (doseq [[item-id id] (map vector items ids)] > (let [rows (generate-rows-however)] > (filter-data rows item-id header id))) > > That's not parallel, but I don't know that parallel buys you much when > the task is I/O bound and takes however long it takes the database > server to process that number of requests. > > If parallelism really does buy you something here you could replace > (doseq ...) with (dorun (pmap identity (for ...))) and that might > parallelize the realization of the items, and thus all of the nested > operations.- Hide quoted text - > > - Show quoted text - -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
