Hello, I'm using clojure to create a database from various files. Because the database in batch insert mode is single threaded I am using an agent to handle communications with the database. Creating a node in the database doesn't have any dependencies, and I return a promise and deliver it in the function that the agent executes (using send-off). I also create various relationships between these nodes, but to do this the promises need to be fulfilled.
What I have been doing is simply appending onto that atom a vector of the promises for the start node, end node, relationship type, and any additional data. If anyone is interested this is Neo4j in batch insertion mode. My questions: 1) Will the die out if the relationship queue gets up to 100's of millions of vectors? Is there something better? 2) What is a good way to periodically process it checking if both promises are realized? and doing a send-off to the database for one and re-queuing for the others? Should I just append to another atom and once complete swap! out the old one for the new (probably shorter) one and periodically call this in the code? I envision running this before the final shutdown as well. Right now I run this after all nodes have been created, so all promises have been delivered, but I'll be handling more files at once and want to periodically add them out of order to limit how many millions the vector of vectors gets, although it can still hit 100 million from 1 file at times. Up until now I've been processing each file individually, but there is a lot of time for database shutdown/start-ups, but it means the vector of vectors does not get past 1 million at the most. But now I am in a re-write to bring up all the database interacting code to this methodology (still learning clojure, so each file type goes from processing sequentially, to pmap, some with reducers, etc. and this method with the promises is the best so far for me). Thanks, sorry for any confusion, it's been a long day but before I get all this re-written into hard code I'd like to make sure I'm doing it in decent way. Best, --Joseph -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
