I tend to think of pull operations in these situations. So perhaps wrap a lazy sequence around a chunked form of the read + transform operation, perhaps parallelizing the transform, and do the insert in a doseq on the lazy sequence?
On Tue, Oct 22, 2013 at 8:03 PM, Brian Craft <craft.br...@gmail.com> wrote: > I'm doing a load to db, which looks roughly like "read, transform, insert, > repeat". > > Blocking on the inserts leaves the cores sitting cold while they could be > doing the next read/transform. It's tempting to try an agent for the > inserts. If I understand them correctly, that would queue the inserts as > they are ready, allowing the read/transform to continue. I'm concerned that > this would just fill memory, as the inserts can't keep up with the reads. > > Is there some other obvious way to structure this? The only relevant hits > I've found on this list are about the async lib, which I gather can handle > this by keeping a bounded queue that can block the producer. I was hoping > to put off learning the async stuff until later. ;) > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
