For example, there are very good reasons for some read sides having the read model stored in an SQL database, or a graph database, or a NOSQL database, or some or all of these at the same time. As you mentioned, this needs to be idempotent to handle at least once delivery.
If a projection can store its checkpoint atomically with it's read model it simulates only-once messaging providing there is an ordering assurance on the stream it's listening to On Monday, August 18, 2014, Ashley Aitken <amait...@gmail.com> wrote: > > Hi Roland (and everyone), > > Welcome back Roland - I hope you had a great vacation. > > Thank you for your post. > > > Here’s my response summary: > > I believe Akka needs to allow actors to: > > (i) persist events with as much information as efficiently possible on the > write side to allow the store to facilitate the read side extracting them > according to what criteria is needed, > > (ii) persist events that don’t relate to a change in state of the actor > per se, which I assume is already achievable since an actor can just ignore > them on replay, > > (iii) read from (and replay) streams of events on the read and write side > according to a range of criteria supported and defined within the store or > via the store API (e.g. using a DSL), and > > (iv) reliably (at least once) deliver information to other read side > store(s) and systems above and beyond the store used for persisting the > events. > > I believe each of these is readily achievable with Akka but: > > (i) doesn’t mean explicitly persisting the events to specific topics as > you suggest in your (1) (although this may be how some stores implement the > required functionality on the read side). Instead it means transparently > including information like the actorId, event type, actor type, probably > the time and possibly information to help with causal ordering (see my next > post). > > (iii) with (i) would enable the read side (if the store supports it) to > read all events from a particular actor(s), of particular event types, to > read events from a particular type(s) of actors, and to read all events. > It would also need to allow the read side to read from where it last > finished reading, from now, and from the start again. (iv) is necessary > for projections. > > > If you are interested, here’s my detailed explanation: > > I think some of the confusion surrounding these issues is caused by the > fact that we seem to be discussing and, if I may suggest, Akka appears to > be trying to implement three quite different (but also somewhat related) > pieces of functionality within this domain. These are: > > A. Actor Persistence > > The ability to persist actor state changes incrementally (or wholly) and > reconstruct that state at a later time, which we know as event sourcing. I > think Akka provides a great distributed and scalable mechanism for doing > this with the current akka.persistence. > > B. Publish/Subscribe to Persistent Queues/Topics > > This functionality would allow actors to write data/events/messages to one > (or more) topics and to subscribe to receive similar from one or more > topics. These differ from normal publish/subscribe queues in that they are > persistent and the consumer can reread from the topic. > > This is what I think of as the LogProducer and LogConsumer, of which > PersistentActor and PersistentView can be thought of as specialisations, > i.e. a single topic for each actor. The current and popular example of a > store for this sort of functionality, as you know, is Kafka. > > C. CQRS with Event Sourcing > > And finally, there is CQRS with Event Sourcing, which I believe is much > more that (A) and (B) and particularly doesn’t necessarily require (B.) for > all event stores. So if Akka were to implement (B), which I think would be > very useful for other reasons, it would not specifically be for CQRS. > > Please consider this diagram overviewing CQRS with Event Sourcing: > > <https://www.dropbox.com/s/z2iu0xi4ki42sl7/annotated_cqrs_architecture.jpg > > > > adapted from > > <http://www.gridshore.nl/wp-content/uploads/cqrs_architecture.jpg> > > As I understand it, CQRS separates the write model and store from one or > *more* read models and stores, with each model and store being optimised > for their particular role. CQRS says nothing specific about the types of > store (e.g. SQL or NOSQL, event sourced or not) and how consistency is > achieved. > > As you know, when using event sourcing the changes to the write model > entities (e.g. Aggregate Roots) are stored as events and the write model is > reconstructed by replaying those events. This is (A) above and what > akka.persistence has achieved very well in a distributed and scalable way. > > This is the dashed area labelled [1] in the diagram. > > Further, CQRS uses commands to initiate changes to the write model and > signals theses changes with events (whether the events are used for event > sourcing or not). These events are what allows sagas and other systems to > track changes and respond to changes in the write model. > > This is the dashed area labelled [2] in the diagram. > > For example, a saga could be waiting for an event indicating funds had > been withdrawn from a bank account after it had issued a command requesting > that be done. The saga could subscribe to events from the bank account > before issuing the command and watch for a specific event from that time on. > > This event notification system is conceptually independent of how the read > store(s) will eventually become consistent (i.e. there are other means of > achieving this consistency without events). However, it just so happens > that these events are also a great way to notify the read store(s) of > changes. > > So conceptually, the read side receives all the events from the write side > independent of the write side event store. I believe we should logically > think of this as one big stream of events since the write side doesn’t (and > cannot) know what the read store(s) needs from these events in advance. > > This is the dashed area labelled [3] in the diagram. > > Akka seems to distribute the event store used for persistence of ARs [1] > on the write side to the read side, which is an interesting idea. But I > don’t believe this is enough for CQRS. One event store cannot provide all > the required read models. > > Because the read side *must* support a number of different stores (not > just the one used for actor persistence and to distribute the events to the > read side) it is also necessary for there to be some reliable way to get > event information into the other read store(s). > > For example, there are very good reasons for some read sides having the > read model stored in an SQL database, or a graph database, or a NOSQL > database, or some or all of these at the same time. As you mentioned, this > needs to be idempotent to handle at least once delivery. > > > I hope that Akka can really fully support CQRS with Event Sourcing to > become the premiere platform for its use in highly scalable, reliable and > efficient systems. I also hope this explanation can help that happen > sooner rather than later since I think it is a great opportunity for Akka. > > Please anyone, correct me if I am misunderstanding things or have left > anything out. > > Cheers, > Ashley. > > > > On 16 Aug 2014, at 1:39 am, Roland Kuhn <goo...@rkuhn.info > <javascript:_e(%7B%7D,'cvml','goo...@rkuhn.info');>> wrote: > > Dear hakkers, > > unfortunately it took me a long time to catch up with akka-user to this > point after the vacation, but on the other hand this made for a very > interesting and stimulating read, thanks for this thread! > > If I may, here’s what I have understood so far: > > 1. In order to support not only actor persistence but also full CQRS > we need to adjust our terminology: events are published to topics, where > each persistenceId is one such topic but others are also allowed. > 2. Common use-cases of building projections or denormalized views > require the ability to query the union of a possibly large number of topics > in such a fashion that no events are lost. This union can be viewed as a > synthetic or logical topic, but issues arise in that true topics provide > total ordering while these synthetic ones have difficulties doing so. > 3. Constructing Sagas is hard. > > > AFAICS 3. is not related to the other two, the mentions in this thread > have only alluded to the problems so I assume that the difficulty is > primarily to design a process that has the right eventual consistency > properties (i.e. rollbacks, retries, …). This is an interesting topic but > let’s concentrate on the original question first. > > The first point is a rather simple one, we just need to expose the > necessary API for writing to a given topic instead of the local Actor’s > persistenceId; I’d opt for adding variants of the persist() methods that > take an additional String argument. Using the resulting event log is then > done as for the others (i.e. Views and potentially queries should just > work). The only concern is that the Journal needs to be prepared to receive > events concurrently from multiple sources instead of just the same Actor, > but since each topic needs to be totally ordered this will not be an > additional hassle beyond just routing to the same replica, just like for > persistenceIds. > > The second point is the contentious one, since a feature request > (consistent iteration over a query) clashes with a design choice > (scalability). First it is important to note that this clash is genuine: > scalability means that we do not want to limit the size of a topic to > always fit one unit of consistency, our default assumption is that > everything should be prepared for distribution. We all know that in a > distributed system linearizability is not generally achievable, meaning > that a distributed (synthetic) topic that receives events from concurrent > sources will not be able to provide a global ordering. A non-distributed > Journal, OTOH, is a single point of failure which is not desirable for many > applications (i.e. your business will go down while the Journal has > issues—true replication requires the ability to fail independently and > hence is distributed in the CAP sense). > > As I see it, a query (like “all events of this type” etc.) should be > configured for the given Journal and should then be available as a > (synthetic) topic for normal consumption—but not for being written to. The > Journal is then free to implement this in any way it sees fit, but barring > fundamental advances in CS or errors on my part this will always require > that the synthetic topic is not scalable in the way we usually define that > (i.e. distributable). As Vaughn points out this may not be an issue at all, > actual benchmarks would help settle this point. Journal backends that > already implement a global order can make use of that, for others the > synthetic topic would work just like any other non-PersistentActor topic > with manual duplication of those events that match the query (akin to (a) > in the first post of this thread); this duplication does not necessarily > need to double the memory consumption, it could also only persist the > events by reference (depending on the storage engine). > > When it comes to providing queries in a way that does not have a global > ordering, my current opinion is that we should not do this because it would > be quite pointless (a.k.a. unusable). A compromise would be to provide > eventually linearizable queries based on the premise that the application > of events should be idempotent in any case and overlapping replay (i.e. > where necessary from the last known-linear point instead of the requested > one) must be tolerated. AFAIK this is the topic of ongoing research, > though, so I’d place that lower on the priority list. > > Does this sound like a fair summary? Please let me know in case I > misrepresent or misunderstand something, once we reach consensus on what we > need we’ll ticket and solve it, as usual ;-) > > Regards, > > Roland > > 12 aug 2014 kl. 18:10 skrev Ashley Aitken <amait...@gmail.com > <javascript:_e(%7B%7D,'cvml','amait...@gmail.com');>>: > > > Thanks for your post Vaughn. > > On Monday, 11 August 2014 05:57:05 UTC+8, Vaughn Vernon wrote: >> >> None of this stuff is easy to do, and even harder to do right. >> > > I am the first to agree with that. > > >> Your post gives away the main problem with getting this to work >> correctly, because Actor Model and akka-persistence currently supports the >> first half of A, but not the second half. In other words, to make the >> interface rich we not only need a new set of abstractions, we also need to >> overcome the direct messaging nature of actors because it can be limiting >> in some use cases. >> > With the messaging library I am building, currently named Number 9, which >> includes both persistence and a process manager, this problem is handled as >> follows. Any actor that sends a message may: >> > >> 1. send a persistent message to another actor >> 2. send a persistent message to a topic >> 3. send a persistent message primarily to another actor, but also to a >> topic >> > > That is very interesting. > > It seems to me that CQRS commands should be sent as messages (persistent > or not) - your (1.) and changes of state (AR or application) should be > published as events (to topics or more generally) - your (2.) but I can't > see a need for (3.)? > > Further, a process manager for a bank account transfer could be > implemented with a command to the source account (withdrawForTransfer) that > would be acknowledged by a published persistent event > (WithdrawnForTransfer). Similar for deposit into target account. > > Pawel Kaczor in his DDD-Leaven-Akka series (Lesson 3) includes projections > from aggregated streams of events and a process manager / saga using Akka > Persistence by having the ARs persisting their events and also publishing > their events. > > > http://pkaczor.blogspot.com.au/2014/06/reactive-ddd-with-akka-projections.html > > https://github.com/pawelkaczor/ddd-leaven-akka > > The only shortcomings (not his fault or a criticism) seem to be: 1) the > use of two event infrastructures (one for persistence and one for pub/sub), > 2) the limited ability for complex projections (like Greg mentioned and > available in Event Store), and 3) lack of persistence for pub/sub events. > > The latter makes reconstruction of a read model or construction of a new > read model after the events have been published more difficult. > > >> If you have watched any of my presentations on this subject you have >> heard this before. I am presenting most of this to the DDD Denver meetup >> this Monday night. The title of the talk is "Building a Reactive Process >> Manager, Twice". The twice part is because I will demonstrate this working >> both in Scala with Akka and also in C# with Dotsero: >> > > Thank you I will look out for that (please share the video link if it is > recorded and put on the Web). I have seen (but not watched) some of your > videos because I am unsure as to who is leading here and the videos I saw > seemed to be from a few years ago. > > I've just got your book so I will get on with reading that (for DDD and > CQRS enlightenment). > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com > <javascript:_e(%7B%7D,'cvml','akka-user%2bunsubscr...@googlegroups.com');> > . > To post to this group, send email to akka-user@googlegroups.com > <javascript:_e(%7B%7D,'cvml','akka-user@googlegroups.com');>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > > > *Dr. Roland Kuhn* > *Akka Tech Lead* > Typesafe <http://typesafe.com/> – Reactive apps on the JVM. > twitter: @rolandkuhn > <http://twitter.com/#!/rolandkuhn> > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to a topic in the > Google Groups "Akka User List" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/akka-user/SL5vEVW7aTo/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > akka-user+unsubscr...@googlegroups.com > <javascript:_e(%7B%7D,'cvml','akka-user%2bunsubscr...@googlegroups.com');> > . > To post to this group, send email to akka-user@googlegroups.com > <javascript:_e(%7B%7D,'cvml','akka-user@googlegroups.com');>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to a topic in the > Google Groups "Akka User List" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/akka-user/SL5vEVW7aTo/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > akka-user+unsubscr...@googlegroups.com > <javascript:_e(%7B%7D,'cvml','akka-user%2bunsubscr...@googlegroups.com');> > . > To post to this group, send email to akka-user@googlegroups.com > <javascript:_e(%7B%7D,'cvml','akka-user@googlegroups.com');>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Studying for the Turing test -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
