On Aug 13, 2010, at 7:55 AM, Simon Woodhead wrote: > Hi folks > > As we explore more ways to migrate to CouchDB we're exploring alternatives to > transactions. The books shows a bank where transfers are stored as a single > document with the balance being the result of a map/reduce function. That > makes sense. > > For us, our scale is tipped differently in that we have hundreds of millions > of tiny transactions affecting relatively few balances. In MySQL (ironically) > we denormalise this and hold balances in their own table but then can insert > transactions and update balances within a single transaction. > > Looking at moving this to CouchDB would mean getting rid of the balances > table and just using a map/reduce function. I recognise that a given document > will only be handled once and that this is therefore more efficient than it > may seem to a SQL jock like me but I wanted to ask about whether it truly > scales at volume. > > I guess I'm asking whether whatever index the view creates contains a > reference to every document (and thus gets bigger with more documents) or > just contains the output and the _id of the last document processed. I can > see the first one running into issues quickly whilst the second would seem to > scale indefinitely. > > FWIW it takes over a day to compute the sum against the table in MySQL as the > table is being constantly appended - hence why we denormalised it! It > therefore feels really strange to normalise more when moving to a document > store so any advice is welcome!
The index in the banking example in the book will indeed grow as transactions happen. If you have a very high rate of transactions, you might want to do like banks do, and "close" each month, by moving new transactions to a new database file at the end of the month. Then you can do a transaction for each account to carry forward the closing balance at the end of the last month, to the current month. After X months, you can throw away the old transaction databases, probably storing the monthly rollups somewhere for posterity. If you rate is high enough, substitute days or hours for months. > > Thanks > Simon-- > ***** Email confidentiality notice ***** > > This message is private and confidential. If you have received this message > in error, please notify us and remove it from your system. > > > Simwood eSMS Limited is a limited company registered in England and Wales. > Registered number: 03379831. Registered office: c/o HW Chartered Accountants, > Keepers Lane, The Wergs, Wolverhampton, WV6 8UA. Trading address: Falcon > Drive, Cardiff Bay, Cardiff, CF10 4RU. > >
