Re: [akka-user] Akka Persistence: LogProducer and LogConsumer?
Hi Ashley, most of my response is already in my recent post, what you are describing are topics that are not coupled to a persistenceId and the ability to consume them in other actors as a stream of events since a given position in the log. One thing below resonates with me, and that is that the ability to express queries should be defined by the Journal (backend) implementation and not by introducing up-front categorization using tags. One of the very powerful notions of event sourcing is that you just persist what defines your domain without prior knowledge of the ways you are going to query it later, and tags go against that notion. The event store should provide appropriate filtering options, which in collaboration with the right serializer settings can enable deep inspection of the persisted content (for document-oriented stores). Regards, Roland 14 aug 2014 kl. 18:03 skrev Ashley Aitken amait...@gmail.com: Hi Patrik, Thanks for your post. On 14 Aug 2014, at 10:31 pm, Patrik Nordwall wrote: I think this might fit in rather well with the tag based PersistentView. The Event Store journal plugin could interpret the tag as a projection identifier. Yes, I think that could easily even work now with the journal plugins. What would these plugins do if a PersistentView was created with a persistenceId that didn’t correspond to any current PersistentActor but did correspond to a topic (like the “events” topic in Kafka or a projection in the Event Store say)? However, that said, I have some other concerns with the tag-based approach. IMHO, the projections should not require “a priori” event tagging. Developers should be able to construct just about any projection for the read side even after the events have been written. Developers shouldn’t have to think in advance of what tags will be needed later or change code on the write side. As a result of that, it seems to me Akka Persistence should just automatically tag all events with the type of the actor, the type of the event, and the actorId (not presuming that all actors will be using the log for persisting their state). How this is implemented would be up to the store (e.g. duplicate events vs links). PersistentViews for my use-case of denormalisation could just be based on the actor type, Greg’s example from the other thread could be based on a combination of event types, and views for truly PersistentActors could just use the actorId. But there may be some other use-cases people have? As you suggest, PersistentViews could just use a label as a projection identifier. These projections could be rich combinations of possibly multiple actor types, multiple event types, and multiple actorIds. They could be specified in the store but it would be nice to eventually be able to specify them in Akka as well. If so, textual tags in each event for these types would be relatively easy but I don’t think it would be the “Typesafe way” ;-) It would be great to eventually have a typesafe way of specifying projections that could type check the components of a projection (and take into account subtyping etc., e.g. all Customer subtypes). And please don’t forget to consider having some way of configuring a PersistentActor not to receive all events previously persisted each time it is reactivated. If a PersistentActor is really a “LogProducer” creating a large log of events, it doesn’t want to have to trundle through all of those events each time it is reactivated. Sorry for such long posts :-( Cheers, Ashley. -- Dr Ashley Aitken Running Code Productions LinkedIn/Twitter/Skype: AshleyAitken -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. Dr. Roland Kuhn Akka Tech Lead Typesafe – Reactive apps on the JVM. twitter: @rolandkuhn -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit
Re: [akka-user] Akka Persistence: LogProducer and LogConsumer?
Hello there, as Patrik redirected – we’ve talked a lot about this and are aiming to provide better views. There already exists integration with Kafka https://github.com/krasserm/akka-persistence-kafka#journal-topics developed by Martin Krasser, it adresses some of the points raised actually. Similarily there already exists an EventStore akka-persistence plugin: https://github.com/EventStore/EventStore.Akka.Persistence. The big thing is that we would like to update the plugin API to such that would allow plugins to “do more” on the read side of things (Patrik linked to the “improved Views” proposal already). Thanks for the interest and ideas! -- Konrad 'ktoso' Malawski hAkker @ typesafe http://akka.io -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] Akka Persistence: LogProducer and LogConsumer?
Hi Patrik, Thanks for your post. On 14 Aug 2014, at 10:31 pm, Patrik Nordwall wrote: I think this might fit in rather well with the tag based PersistentView. The Event Store journal plugin could interpret the tag as a projection identifier. Yes, I think that could easily even work now with the journal plugins. What would these plugins do if a PersistentView was created with a persistenceId that didn’t correspond to any current PersistentActor but did correspond to a topic (like the “events” topic in Kafka or a projection in the Event Store say)? However, that said, I have some other concerns with the tag-based approach. IMHO, the projections should not require “a priori” event tagging. Developers should be able to construct just about any projection for the read side even after the events have been written. Developers shouldn’t have to think in advance of what tags will be needed later or change code on the write side. As a result of that, it seems to me Akka Persistence should just automatically tag all events with the type of the actor, the type of the event, and the actorId (not presuming that all actors will be using the log for persisting their state). How this is implemented would be up to the store (e.g. duplicate events vs links). PersistentViews for my use-case of denormalisation could just be based on the actor type, Greg’s example from the other thread could be based on a combination of event types, and views for truly PersistentActors could just use the actorId. But there may be some other use-cases people have? As you suggest, PersistentViews could just use a label as a projection identifier. These projections could be rich combinations of possibly multiple actor types, multiple event types, and multiple actorIds. They could be specified in the store but it would be nice to eventually be able to specify them in Akka as well. If so, textual tags in each event for these types would be relatively easy but I don’t think it would be the “Typesafe way” ;-) It would be great to eventually have a typesafe way of specifying projections that could type check the components of a projection (and take into account subtyping etc., e.g. all Customer subtypes). And please don’t forget to consider having some way of configuring a PersistentActor not to receive all events previously persisted each time it is reactivated. If a PersistentActor is really a “LogProducer” creating a large log of events, it doesn’t want to have to trundle through all of those events each time it is reactivated. Sorry for such long posts :-( Cheers, Ashley. -- Dr Ashley Aitken Running Code Productions LinkedIn/Twitter/Skype: AshleyAitken -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] Akka Persistence: LogProducer and LogConsumer?
On Thu, Aug 14, 2014 at 6:03 PM, Ashley Aitken amait...@gmail.com wrote: Hi Patrik, Thanks for your post. On 14 Aug 2014, at 10:31 pm, Patrik Nordwall wrote: I think this might fit in rather well with the tag based PersistentView. The Event Store journal plugin could interpret the tag as a projection identifier. Yes, I think that could easily even work now with the journal plugins. What would these plugins do if a PersistentView was created with a persistenceId that didn’t correspond to any current PersistentActor but did correspond to a topic (like the “events” topic in Kafka or a projection in the Event Store say)? That is true, but not a real solution. Right now PersistentView uses the ordinary Recover meachanism, so the journal doesn't really know if it is a view or a persistent actor that is requesting the events. However, that said, I have some other concerns with the tag-based approach. IMHO, the projections should not require “a priori” event tagging. Developers should be able to construct just about any projection for the read side even after the events have been written. Developers shouldn’t have to think in advance of what tags will be needed later or change code on the write side. As a result of that, it seems to me Akka Persistence should just automatically tag all events with the type of the actor, the type of the event, and the actorId (not presuming that all actors will be using the log for persisting their state). I think you have a good point. That is 3 pre-defined (fixed) tags, and that might be enough. I was thinking that it would be more flexible to be able to define tags, but that might be overkill. How this is implemented would be up to the store (e.g. duplicate events vs links). I agree PersistentViews for my use-case of denormalisation could just be based on the actor type, Greg’s example from the other thread could be based on a combination of event types, and views for truly PersistentActors could just use the actorId. But there may be some other use-cases people have? As you suggest, PersistentViews could just use a label as a projection identifier. These projections could be rich combinations of possibly multiple actor types, multiple event types, and multiple actorIds. They could be specified in the store but it would be nice to eventually be able to specify them in Akka as well. They must execute in the store to be efficient. Inventing an event query language in Akka that journal plugins must implement is currently out of scope. Some kind of projections can be implemented with Akka streams on top of PersistentViews, and that will be useful, but never as efficient as in running in the store, since all events must be retrieved, deserialized, and so on. If so, textual tags in each event for these types would be relatively easy but I don’t think it would be the “Typesafe way” ;-) It would be great to eventually have a typesafe way of specifying projections that could type check the components of a projection (and take into account subtyping etc., e.g. all Customer subtypes). And please don’t forget to consider having some way of configuring a PersistentActor not to receive all events previously persisted each time it is reactivated. If a PersistentActor is really a “LogProducer” creating a large log of events, it doesn’t want to have to trundle through all of those events each time it is reactivated. Sure. Shouldn't be difficult. Might even be possible today with a custom recovery start message. Please open a specific ticket https://github.com/akka/akka/issues about this, so that it is not forgotten. Sorry for such long posts :-( Thanks for the feedback. I will link to this tread from the PersistentView ticket. /Patrik Cheers, Ashley. -- Dr Ashley Aitken Running Code Productions LinkedIn/Twitter/Skype: AshleyAitken -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Patrik Nordwall Typesafe http://typesafe.com/ - Reactive apps on the JVM Twitter: @patriknw -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to
Re: [akka-user] Akka Persistence: LogProducer and LogConsumer?
On 14.08.14 18:17, Ashley Aitken wrote: Hi Konrad, On Thursday, 14 August 2014 22:39:04 UTC+8, Konrad Malawski wrote: There already exists integration with Kafka https://github.com/krasserm/akka-persistence-kafka#journal-topics https://github.com/krasserm/akka-persistence-kafka#journal-topics developed by Martin Krasser, it adresses some of the points raised actually. Yes, I am aware of that (and thanks to Martin!) but Kafka (please correct me if I am wrong) doesn't have the ability to create projections. Perhaps for now we can just manually post events to extra topics each time we persist actor events in the Kafka store? Yes, this is already possible with user-defined topics https://github.com/krasserm/akka-persistence-kafka#user-defined-topics. At the moment this is a plugin-specific feature but can later be generalized to support akka-persistence tags. Similarily there already exists an EventStore akka-persistence plugin: https://github.com/EventStore/EventStore.Akka.Persistence https://github.com/EventStore/EventStore.Akka.Persistence. Yes, I am aware of that (and thanks to ES folks!) but it is version 0.0.3 and it seems quite different from the other stores. And in both of these cases I am also not sure if PersistentViews can specify other topics / projections as their persistentIds. I wonder if this is something controlled by Akka Persistence or the Event Store? The big thing is that we would like to update the plugin API to such that would allow plugins to “do more” on the read side of things (Patrik linked to the “improved Views” proposal already). Yes, it seems there are many people very interested in this - I wish I could help more but I am new to all these technologies. Cheers, Ashley. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com mailto:akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com mailto:akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Martin Krasser blog:http://krasserm.blogspot.com code:http://github.com/krasserm twitter: http://twitter.com/mrt1nz -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
[akka-user] Akka Persistence: LogProducer and LogConsumer?
I've been thinking about how Akka Persistence could possibly be improved for CQRS and *persistent* stream or log processing more generally. Here's my stream (no pun intended) of thought: Currently, PersistentActor is focussed on persisting state changes for an Actor (persistenceId) but why couldn't that be extended to just persisting a stream of events (or more generally messages) with the option to skip the reconstruction of state, i.e. PersistentStreamProducer. Similarly, PersistentView could be extended to consume a persistent stream of messages (keeping track of where it was up to, with the option to start again if needed), i.e. PersistentStreamConsumer. It should be possible to label these persistent streams with something other than a persistenceId (if appropriate), i.e. something like a topic. And finally, the most important for CQRS, PersistentStreamConsumer shouldn't need to be tied to a real PersistentStreamProducer. It should be possible to create (perhaps with a DSL in Scala or separately in the store itself, or elsewhere) virtual streams (projections) based on PersistentStreamProducer(s), PSP types, message types, etc. Greg Young's Event Store would seem to be a great store for facilitating most (if not all) of this, with its projections functionality. Kafka would seem to be an even more scalable solution but (AFAIK) currently lacks the ability to create virtual streams (i.e. virtual logs). I think this approach is somewhat different from the current Akka Streams, as far as I understand them, which seem to be run-time only streams. Perhaps these actors would be better named PersistentLogProducer and PersistentLogConsumer (or just LogProducer and LogConsumer). Or they could just be traits. In that case PersistentActor would be Actor with LogProducer and PersistentView would be Actor with LogConsumer. Perhaps this can (almost) already be done with PersistentActor and PersistentView? I'd be very interested to know if the current PersistentViews could somehow use virtual persistenceIds resulting from projections created in Event Store. I assume the current journals create topics for each PersistentActor persistenceId and the PersistenceViews just look them up. So if it was possible to create a projection in Event Store with a synthetic persistenceId then perhaps a PersistenceView could just find it, believing it to be a journal for a real PersistentActor. Your thoughts? I note there is already: 1. An Akka client for Event Store (EventStore.JVM) 2. An early Event Store plugin for Akka Persistence 3. An Akka consumer for Kafka (akka-kafka) -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
