Hi Martin, Had to think about it a little, hereby my follow up. (hope you don't mind the continues discussion, it helps me a lot in defining the right approach, thanks for that)
On Saturday, April 19, 2014 7:11:23 AM UTC+2, Martin Krasser wrote: > > Hi Olger, > > installing 10k views/producers won't scale, at least not with the current > implementation. Here are some alternatives: > Intresting, what would need to change to have is scaling ? (Idea is to have the eventsourcedprocessors reflect a DDD style Aggregate Root instance and have those distributed using cluster sharding) > > - Maybe a custom journal plugin is what you need: a plugin that delegates > all write/read requests to the actual journal actor and that additionally > updates a database with the events to be written. This essentially installs > a single "listener" per ActorSystem (this is to some extend comparable to a > database trigger that executes additonal commands. If the backend datastore > supports that directly, I recommend implementing the trigger there, if > possible). > I am not sure, if I understand it.. the basic idea is to have the 'events' stored via the eventsourcedprocessor being published to 'n' views. The actual number of view that need to listen to these events are not known up front (people can add their own views... at system startup, it will be clear) As every eventsourced actor is actually an AggregateRoot (in DDD terms) and thereby something of an instance with it's own state, the changes in these states need to be aggregated (that can be done with the streaming as you mention) and published to the views that are interested (subscribed). Doing this by hand in the aggregate root actor is not a problem, thereafter write your own listener actor and that will populate a view data store. Still I have the feeling that the actual 'View' (or ViewProducer) could be implemented in such a way that it's done by the view. > > - Instead of having thousands of processors, what speaks against combining > them into a single processor (or only a few) per node? > This would mean that I'll have all my aggregate root instances running in 1 processor meaning that I need to reconstruct state per aggregate root instance in some way. Using EventsourcedProcessor, I'd expect that I need to replay everything for all instances and pick the one that I need for processing at that moment. (this can of course be optimized with snapshots and something like memcached). This appears to be a performance hit as I feel it. > > Further comments inline ... > > On 18.04.14 16:10, Olger Warnier wrote: > > Hi Martin, > > > I'm currently working on view composition using the brand new >> akka-stream module. Basic idea is to make views stream producers and to use >> the akka-stream DSL to merge message/event streams from several producers >> into whatever you need. See also >> https://twitter.com/mrt1nz/status/457120534111981569 for a first running >> example. >> >> WDYT? >> > > First of all Nice stuff !, I think this is useful for the system at my > hands (real-time patient monitoring based on medical data) > I've seen the streams announcements but did not dive into that yet. > Looking at your code StreamExample.scala it more or less 'clicks' in > concept. (and hopefully in the right way) > > From a 'View' perspective as currently is available in akka-persistence, > every producing actor needs a view attached to it in order to push the > events to the streams producer, right ? (when I look at the > ViewProducer.scala code, this is what is done.) > > PersistentFlow.fromProcessor("p1").toProducer(materializer) > Now, I have a sharding cluster with an EventsourcedProcessor (expect > 10.000ths of these EventsourcedProcessor actor instances) , so I'll need to > create a line like this for every EventsourcedProcessor in order to get the > stream of events together. Thereafter, I need to merge them together to get > a single stream of events. (at least that is one of the features of using > the streams) > > > Every processor instance itself could create such a producer during start > and send it to another actor that merges received producers. > That would not allow me to implement 'View' (as is known in the persistence package) in order to listen to events within my cluster of aggregate root instances, I'll need to build something additional for that (as View is more used for the collection of those events and thereafter will push them through) At this moment, I use an akka extension (gives more or less a singleton) that is used directly in the EventSourcedProcessor after storage (persist) of the event. Thereafter I have listeners that get these events and transform them into data that needs storage for a certain type of view (CQRS style) (this is where I expected the 'View' to be used) > > > My goal is to have 'Listeners' (that is my interpretation of a 'View' > due to historic reasons...) that will for instance update a data store, > this will probably happen on on just a few nodes (maybe 1 and some failover > stuff). These 'Listeners' need to attach to the sharded Eventsourced system > and ask to get all event sourced events forwarded. (publish subscribe more > or less). > > I wonder if the current View (or ViewProducer) fits this situation due > to the fact you need to create as many views as eventsourcedprocessors are > created. > With the merged streams thereafter, it seems a possibility to have just > one thing per node (I assume actor) that will do the writing to a data > store (not being the eventstore). > What would be the way to get these Views 'automagically' attached to the > proper procesors ? > > > See above. > > > And, do you have a pointer how this issue is solved with it's own > eventstore ? In a sharding cluster, you more or less have the same issue. > (would streams change your approach there ?) > > > There's one journal actor per ActorSystem where n journal actors in a > cluster update a replicated > journal<http://akka.io/community/#plugins_to_akka_persistence> > . > Thanks, thats not done as an akka extension right ? (why not ?) > > Hope that helps. > Certainly, really appreciate your patience. > > Cheers, > Martin > Kind regards, Olger -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
