[akka-user] Event Sourcing with eventual consistency guarantees

[Stefan Schmidt]

Hi guys,

I am currently prototyping a new app which involves transferring money 
between various accounts. On a very high level I have a system account 
(which is used to collect fees), one account for each member in the 
platform (many of them), and group accounts. Money needs to be moved on a 
frequent basis from member accounts (M) to group accounts (G) and the 
system account (S). 

Traditionally a transaction like this would be accomplished atomically:

tx start
  - read M account to check for sufficient funds
  - deduct money from M account
  - add money to S account
  - add money to G account
tx end

I know already that the system account S will be involved in most of these 
transactions and eventually become a bottleneck in the platform. Another 
requirement in the platform is to have all money movements fully auditable, 
which is a very common requirement. 

So using event sourcing and CQRS comes to mind to solve this problem. 
Initially my thinking is to have a single persistent actor for each member 
account (M), a persistent actor for the system account (S) and a single 
persistent actor for each group account (G). Each will store events related 
to their respective accounts and offer different views (to keep the 
balance, monitor fraudulent behaviour, statistics, etc).

In addition I would like to have a persistent actor to persist the 
overarching transaction events (lets call it TX actor), mostly for 
bookkeeping & statistics via its views. The idea is that a 'transaction' 
starts with this TX actor which then issues money transfer commands to all 
account actors involved (M, G, S), monitors their responses and either 
persists his own event of the successful transfer or issues a compensation 
commands in case something goes wrong).

Because there will be a large number of members in the platform I would 
like to use Akka clustering where the persistent actors may live on 
different nodes. I have played with hash based routing and cluster sharding 
to address the single writers per account. 

My problem at the moment is to figure out how each transaction can become 
eventually consistent (say within a few seconds) in a clustered environment 
like this where there are multiple points of failure. Obviously I need to 
ensure that a transaction cannot leave the system in an inconsistent state 
and potential manual compensations are also subject to their own errors.

I guess my question is if anyone has used Akka persistence / event sourcing 
/ CQRS for handling financial transactions in a clustered environment? If 
so can you share some experiences or ideas, especially around ensuring 
(eventual) consistency? 

It seems like event sourcing is a good solution to overcome some of the 
bottlenecks which a SQL database will create (especially where there is one 
very contentious resource (account S)) but there is are not many reports 
out there where people have used ES specifically for money handling 
(perhaps for good reason ;)).

Thanks in advance.

-Stefan

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Event Sourcing with eventual consistency guarantees

[Konrad 'ktoso' Malawski]

Hi Stefan,
Systems like you describe definitely *are* implemented in terms of persisting 
events.

I would highly recommend these following videos / papers as a "holiday-read”:
* Eric Evans (Dad of DDD) about modeling to such constraints WITH thinking of 
time and WITHOUT transactions http://www.ustream.tv/recorded/46744749
-- 
Konrad 'ktoso' Malawski
hAkker @ typesafe
http://akka.io

On 25 December 2014 at 21:39:24, Stefan Schmidt (stsme...@gmail.com) wrote:

Hi guys,

I am currently prototyping a new app which involves transferring money between 
various accounts. On a very high level I have a system account (which is used 
to collect fees), one account for each member in the platform (many of them), 
and group accounts. Money needs to be moved on a frequent basis from member 
accounts (M) to group accounts (G) and the system account (S). 

Traditionally a transaction like this would be accomplished atomically:

tx start
  - read M account to check for sufficient funds
  - deduct money from M account
  - add money to S account
  - add money to G account
tx end

I know already that the system account S will be involved in most of these 
transactions and eventually become a bottleneck in the platform. Another 
requirement in the platform is to have all money movements fully auditable, 
which is a very common requirement. 

So using event sourcing and CQRS comes to mind to solve this problem. Initially 
my thinking is to have a single persistent actor for each member account (M), a 
persistent actor for the system account (S) and a single persistent actor for 
each group account (G). Each will store events related to their respective 
accounts and offer different views (to keep the balance, monitor fraudulent 
behaviour, statistics, etc).

In addition I would like to have a persistent actor to persist the overarching 
transaction events (lets call it TX actor), mostly for bookkeeping & statistics 
via its views. The idea is that a 'transaction' starts with this TX actor which 
then issues money transfer commands to all account actors involved (M, G, S), 
monitors their responses and either persists his own event of the successful 
transfer or issues a compensation commands in case something goes wrong).

Because there will be a large number of members in the platform I would like to 
use Akka clustering where the persistent actors may live on different nodes. I 
have played with hash based routing and cluster sharding to address the single 
writers per account. 

My problem at the moment is to figure out how each transaction can become 
eventually consistent (say within a few seconds) in a clustered environment 
like this where there are multiple points of failure. Obviously I need to 
ensure that a transaction cannot leave the system in an inconsistent state and 
potential manual compensations are also subject to their own errors.

I guess my question is if anyone has used Akka persistence / event sourcing / 
CQRS for handling financial transactions in a clustered environment? If so can 
you share some experiences or ideas, especially around ensuring (eventual) 
consistency?

It seems like event sourcing is a good solution to overcome some of the 
bottlenecks which a SQL database will create (especially where there is one 
very contentious resource (account S)) but there is are not many reports out 
there where people have used ES specifically for money handling (perhaps for 
good reason ;)).

Thanks in advance.

-Stefan
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Event Sourcing with eventual consistency guarantees

[Konrad 'ktoso' Malawski]

Hit command-enter too soon by accident before finishing my email - sorry! 
Here’s the rest.

To read:
And also Pet Helland’s (at that time at Amazon) legendary "Life Beyond 
Distributed Transactions”

In general though. What you described as a transaction is in fact a series of 
events _plus_ a so called “Process Manager”.
This is DDD terminology, you’ll find more about thise in Books around Domain 
Driven Design or some of Vaughn’s talks ( I did not fully watch this one, but 
topic wise seems to be just what you’re asking https://vimeo.com/104021785 ).
So in essence, there is a persistent entity, which “takes care to drive the 
operations to their end”, for example it sees that this and that “transaction” 
(or simply “process”) did not succeed, maybe some message was lost, or maybe 
some server was down and we couldn’t proceed etc. So for furher reading on 
process managers I’ll refer to The “CQRS Journey” which I think you should go 
over while thinking about your business, esp. this chapter on “Sagas" 
http://msdn.microsoft.com/en-us/library/jj591569

While we’re discussing this, please remember that money transactions don’t 
always offer full guarantees anyway.
Typical examples here being ATMs which can be not connected at all times, and 
*may* give out 50 bucks without checking if you really can because it’s *low 
risk* and they can block your account afterwards (hello eventual consistency!) 
anyway if you did in fact overdraw. However the same logic does not apply if 
you’re trying to get a few k out of an ATM, that won’t be as “low risk” of 
course, so the checks will have to be run before handing out the monies.

So instead of transactions, we can (just like real people) keep a process 
running and react to things happening around it,
and if we failed - take compensating actions - by always appending data + 
making sure *global* invariants are perserved (check Eric Evan’s talk I liked 
for awesame examples of this).

I hope this helps!
Merry x-mas hakking!

-- 
Konrad 'ktoso' Malawski
hAkker @ typesafe
http://akka.io

On 25 December 2014 at 21:39:24, Stefan Schmidt (stsme...@gmail.com) wrote:

Hi guys,

I am currently prototyping a new app which involves transferring money between 
various accounts. On a very high level I have a system account (which is used 
to collect fees), one account for each member in the platform (many of them), 
and group accounts. Money needs to be moved on a frequent basis from member 
accounts (M) to group accounts (G) and the system account (S). 

Traditionally a transaction like this would be accomplished atomically:

tx start
  - read M account to check for sufficient funds
  - deduct money from M account
  - add money to S account
  - add money to G account
tx end

I know already that the system account S will be involved in most of these 
transactions and eventually become a bottleneck in the platform. Another 
requirement in the platform is to have all money movements fully auditable, 
which is a very common requirement. 

So using event sourcing and CQRS comes to mind to solve this problem. Initially 
my thinking is to have a single persistent actor for each member account (M), a 
persistent actor for the system account (S) and a single persistent actor for 
each group account (G). Each will store events related to their respective 
accounts and offer different views (to keep the balance, monitor fraudulent 
behaviour, statistics, etc).

In addition I would like to have a persistent actor to persist the overarching 
transaction events (lets call it TX actor), mostly for bookkeeping & statistics 
via its views. The idea is that a 'transaction' starts with this TX actor which 
then issues money transfer commands to all account actors involved (M, G, S), 
monitors their responses and either persists his own event of the successful 
transfer or issues a compensation commands in case something goes wrong).

Because there will be a large number of members in the platform I would like to 
use Akka clustering where the persistent actors may live on different nodes. I 
have played with hash based routing and cluster sharding to address the single 
writers per account. 

My problem at the moment is to figure out how each transaction can become 
eventually consistent (say within a few seconds) in a clustered environment 
like this where there are multiple points of failure. Obviously I need to 
ensure that a transaction cannot leave the system in an inconsistent state and 
potential manual compensations are also subject to their own errors.

I guess my question is if anyone has used Akka persistence / event sourcing / 
CQRS for handling financial transactions in a clustered environment? If so can 
you share some experiences or ideas, especially around ensuring (eventual) 
consistency?

It seems like event sourcing is a good solution to overcome some of the 
bottlenecks which a SQL database will create (especially where there is one 
very contentious resource (account S)) but

Re: [akka-user] Event Sourcing with eventual consistency guarantees

[Stefan Schmidt]

Hi Konrad,

Merry Christmas to you too!

Thanks for your suggestions. 

As it happens I have read / watched many of the materials you mentioned 
after reading through this 
thread https://groups.google.com/forum/#!topic/akka-user/BRh3YNjP0kY. The 
link to the saga chapter is quite useful though. 

Its good to know I'm on the right track.

Cheers,
Stefan

On Friday, December 26, 2014 10:31:06 AM UTC+11, Konrad Malawski wrote:
>
> Hit command-enter too soon by accident before finishing my email - sorry! 
> Here’s the rest.
>
> To read:
> And also Pet Helland’s (at that time at Amazon) legendary "Life Beyond 
> Distributed Transactions 
> ”
> 
>
> In general though. What you described as a transaction is in fact a series 
> of events _plus_ a so called “Process Manager”.
> This is DDD terminology, you’ll find more about thise in Books around 
> Domain Driven Design or some of Vaughn’s talks ( I did not fully watch this 
> one, but topic wise seems to be just what you’re asking 
> https://vimeo.com/104021785 ).
> So in essence, there is a persistent entity, which “takes care to drive 
> the operations to their end”, for example it sees that this and that 
> “transaction” (or simply “process”) did not succeed, maybe some message was 
> lost, or maybe some server was down and we couldn’t proceed etc. So for 
> furher reading on process managers I’ll refer to The “CQRS Journey” which I 
> think you should go over while thinking about your business, esp. this 
> chapter on “Sagas" http://msdn.microsoft.com/en-us/library/jj591569
>
> While we’re discussing this, please remember that money transactions don’t 
> always offer full guarantees anyway.
> Typical examples here being ATMs which can be not connected at all times, 
> and *may* give out 50 bucks without checking if you really can because it’s 
> *low risk* and they can block your account afterwards (hello eventual 
> consistency!) anyway if you did in fact overdraw. However the same logic 
> does not apply if you’re trying to get a few k out of an ATM, that won’t be 
> as “low risk” of course, so the checks will have to be run before handing 
> out the monies.
>
> So instead of transactions, we can (just like real people) keep a process 
> running and react to things happening around it,
> and if we failed - take compensating actions - by always appending data + 
> making sure *global* invariants are perserved (check Eric Evan’s talk I 
> liked for awesame examples of this).
>
> I hope this helps!
> Merry x-mas hakking!
>
> 
> -- 
> Konrad 'ktoso' Malawski
> hAkker @ typesafe
> http://akka.io
>
> On 25 December 2014 at 21:39:24, Stefan Schmidt (stsm...@gmail.com 
> ) wrote:
>  Hi guys, 
>
> I am currently prototyping a new app which involves transferring money 
> between various accounts. On a very high level I have a system account 
> (which is used to collect fees), one account for each member in the 
> platform (many of them), and group accounts. Money needs to be moved on a 
> frequent basis from member accounts (M) to group accounts (G) and the 
> system account (S). 
>
> Traditionally a transaction like this would be accomplished atomically:
>
> tx start
>   - read M account to check for sufficient funds
>   - deduct money from M account
>   - add money to S account
>   - add money to G account
> tx end
>
> I know already that the system account S will be involved in most of these 
> transactions and eventually become a bottleneck in the platform. Another 
> requirement in the platform is to have all money movements fully auditable, 
> which is a very common requirement. 
>
> So using event sourcing and CQRS comes to mind to solve this problem. 
> Initially my thinking is to have a single persistent actor for each member 
> account (M), a persistent actor for the system account (S) and a single 
> persistent actor for each group account (G). Each will store events related 
> to their respective accounts and offer different views (to keep the 
> balance, monitor fraudulent behaviour, statistics, etc).
>
> In addition I would like to have a persistent actor to persist the 
> overarching transaction events (lets call it TX actor), mostly for 
> bookkeeping & statistics via its views. The idea is that a 'transaction' 
> starts with this TX actor which then issues money transfer commands to all 
> account actors involved (M, G, S), monitors their responses and either 
> persists his own event of the successful transfer or issues a compensation 
> commands in case something goes wrong).
>
> Because there will be a large number of members in the platform I would 
> like to use Akka clustering where the persistent actors may live on 
> different nodes. I have played with hash based routing and cluster sharding 
> to address the single writers per account. 
>
> My problem at t