Re: Keeping a ref and a DB in sync
On Apr 3, 12:42 pm, rzeze...@gmail.com rzeze...@gmail.com wrote: Brian, I imagine you are asking this in relation to your blog engine? Yep. Thanks everyone for the help and ideas. Unfortunately I do have a ton of data in a DB already so Terracotta would be a lot of work. I rewrote everything to do ref updates inside agents, and it worked (almost), but when I fired up 1000 concurrent threads to do simultaneous updates and set a certain percentage of them to throw an exception mid-transaction, I still ended up with the DB and the in- memory copy out of sync. Cleaning up when something failed was a nightmare to try to coordinate. I ended up storing all of my data in a single ref that holds the state of the universe and running a background thread in an agent that grabs a snapshot of the ref and syncs it with the DB every so often. I guess it's probably a little more Clojure-ish that way anyways, since the DB will always see a consistent snapshot. If this doesn't work I'll have to try watches I guess. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Keeping a ref and a DB in sync
Perhaps you could do the db update as an agent action and then the ref update within the agent action if it is successful - see http://groups.google.co.za/group/clojure/browse_thread/thread/d645d77a8b51f01/667e833c1ea381d7 Regards, Adrian. On Fri, Apr 3, 2009 at 11:02 AM, Brian Carper briancar...@gmail.com wrote: Is there a safe way to keep the data in a Clojure ref and the data in a table in an external (e.g. mysql) database in sync, given concurrent creates/updates/deletes from within Clojure? I can't do a DB update from within a dosync because of retries. If I send-off an agent for the DB update from within a dosync, it won't happen until after the dosync is done, and then if the DB update throws an exception, it'll be too late to rollback the ref and they'll be out of sync. If I do the DB update and ref update separately, there's the potential for race conditions if things happen in between. Is manual locking the only way? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Keeping a ref and a DB in sync
In case you haven't seen this thread: http://groups.google.com/group/clojure/browse_frm/thread/aa22a709501a64ac/79b1c858c6d50497?lnk=gstq=transaction+database#79b1c858c6d50497 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Keeping a ref and a DB in sync
I'll just throw this out there. It may not be exactly what you're looking for, but you could use a Terracotta cluster. Terracotta will persist the cluster to disk. If you have an existing database that you are working with, then this may not help, but if you are starting from scratch you may not need a database if you use Terracotta. Paul On Fri, Apr 3, 2009 at 5:02 AM, Brian Carper briancar...@gmail.com wrote: Is there a safe way to keep the data in a Clojure ref and the data in a table in an external (e.g. mysql) database in sync, given concurrent creates/updates/deletes from within Clojure? I can't do a DB update from within a dosync because of retries. If I send-off an agent for the DB update from within a dosync, it won't happen until after the dosync is done, and then if the DB update throws an exception, it'll be too late to rollback the ref and they'll be out of sync. If I do the DB update and ref update separately, there's the potential for race conditions if things happen in between. Is manual locking the only way? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Keeping a ref and a DB in sync
Brian, I imagine you are asking this in relation to your blog engine? I came up with solution, that is, if you don't mind writing the persistent data fresh every time. http://paste.lisp.org/display/77987 Basically, I added a watch to the *comment* ref, which set the *db* ref to the new state of the *comment* ref. I wrote a test that hits it with 10 current threads (that each add 10 comments) and then validate that both refs are the same at the end (I borrowed the code from Rich's example on the concurrency page). This is my first real dive into the concurrency aspect of Clojure, and just wanted to play around, but maybe this could give you other ideas. Maybe you could write to a file, and then every so often commit the contents of the file to the db? Like a buffer of commits or something that is persisted to disk. Meh, I'm just throwing up ideas, good luck to ya. On Apr 3, 5:02 am, Brian Carper briancar...@gmail.com wrote: Is there a safe way to keep the data in a Clojure ref and the data in a table in an external (e.g. mysql) database in sync, given concurrent creates/updates/deletes from within Clojure? I can't do a DB update from within a dosync because of retries. If I send-off an agent for the DB update from within a dosync, it won't happen until after the dosync is done, and then if the DB update throws an exception, it'll be too late to rollback the ref and they'll be out of sync. If I do the DB update and ref update separately, there's the potential for race conditions if things happen in between. Is manual locking the only way? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
