Re: [akka-user] Event Stores for Akka Persistence for CQRS?

Martin Krasser Tue, 19 Aug 2014 05:58:13 -0700


On 19.08.14 14:37, Roland Kuhn wrote:

19 aug 2014 kl. 13:49 skrev Patrik Nordwall <patrik.nordw...@gmail.com<mailto:patrik.nordw...@gmail.com>>:

On Tue, Aug 19, 2014 at 1:46 PM, Martin Krasser<krass...@googlemail.com <mailto:krass...@googlemail.com>> wrote:



    On 19.08.14 13:40, Patrik Nordwall wrote:




    On Tue, Aug 19, 2014 at 1:35 PM, Martin Krasser
    <krass...@googlemail.com <mailto:krass...@googlemail.com>> wrote:


        On 19.08.14 13:00, Roland Kuhn wrote:


        19 aug 2014 kl. 11:28 skrev Ashley Aitken
        <amait...@gmail.com <mailto:amait...@gmail.com>>:



        On Tuesday, 19 August 2014 16:53:46 UTC+8, Martin Krasser
        wrote:


            journal -> akka actor(s) -> read model datastore

            when I can do this much more efficiently via

            journal -> spark -> read model datastore


            directly, for example

        I am confused, are you suggesting that spark is talking to
        the journal data store directly, without any involvement
        of Akka / Akka Persistence?  If so, it sounds like a great
        solution but why would that require an extension to the
        Akka Persistence design/API?


        Well, another comment is that spark uses Akka actors in its
        implementation, so I don’t see why it would magically be
        “much more efficient”. I think we are mixing up two
        concerns here, will reply later when I can type properly again.


        This is a misunderstanding. As mentioned in my previous
        message, scaling reads through a single journal actor
        doesn't work, it's not about that I see a general
        performance issue with Akka actors.



    I think the integration "akka persistence -> kafka -> spark ->
    whatever" looks great, but not everybody has that
    infrastructure, and therefore we provide PersistentView as a
    simple way to replicate events to the read side, and then a
    de-normalized representation can be stored in whatever makes
    sense for the queries.


    Of course, that should be possible too, and as I already said,
    backend store providers can choose to use the very same backend
    for both plugins. There is absolutely no need that applications
    must use Spark as part of their infrastructure infrastructure.
    But if it is needed in large-scale applications, a seconds plugin
    API for on the read side would make things much more flexible.
    For users, who just want to have a single backend store, they
    just have to configure one additional line (plugin) in their
    application conf.


    Martin, what do you suggest? Removing PersistentView altogether?


    No, not at all, with an additional plugin, PersistentViews should
    have the option to read transformed/joined/... streams from a
    backend store that is optimized for that.


ah, now I understand what you mean. That makes sense.

I’m not completely there yet: in which way does this require changesto Akka Persistence? The only thing we need is to support multipleJournals in the same ActorSystem, and a way for PersistentView andPersistentActor to select between them, is this what you mean?

This would go into the right direction, except that I wouldn't call theplugin that serves PersistentViews a "journal" because it only providesan interface for reading. Furthermore, this plugin could additionallyoffer an API for passing backend-specific query statements forjoining/transforming/... streams on the fly (if supported/wanted).

Or do you mean that the read-side would be a new kind of plugin?


Yes, see above

OTOH this would not solve the read-side concerns raised by Greg:building a View on top of an incoming event stream is precisely notwhat he wants, unless I got that wrong. The idea behind CQRS/ES isthat the events from the write-side drive updates of the read-sidewhich is then queried (i.e. actively asked instead of passivelyupdating) in whatever way is appropriate (e.g. graph searches).

I cannot see how my proposal is in contradiction with that. Can youplease explain?

Regards,

Roland

/Patrik


    /Patrik


        Regards,

        Roland