Hi Akka hackers, I wrote a first version of a application using Akka Actors with only Actors fetching data from Kafka. This version works well and now I preparing a new version using Akka Cluster to scale it.
In my application scope, process means : 1. Read data from kafka 2. Temporary write them to Redis (using this lib https://github.com/etaty/rediscala) 3. Push some jobs to a Master (a singleton actor in the cluster) 4. Master gives jobs to workers 5. Worker get job, read from redis, write metadata in mongo and data to S3 (using reactivemongo) In a first attempt, I used a Cluster Aware Routers using a pool because my workers are stateless so you can give a job to any of them. Also, I liked this kind of router because I can add new nodes and then have more workers processing data. Things got harder for me when I discovered (it explains the newbie word in the title) every message between Actors and props fields must be serializable using Akka Cluster : a naive approch was to give a redis client and a mongo client to each worker like. So it's created once and share with workers like below : case class Worker(mongoClient : MongoClient, redisClient : RedisClientMasterSlaves) extends Actor with ActorLogging { // mongo client wrap reactivemongo MongoDriver, MongoConnection ... } But this didn't worker : 1. The Redis Client requires an Actor System so it's not serializable 2. MongoDriver, MongoConnection, JSONCollection are not serializable. A quick and dirty fix is to create in each worker to create a new mongo and redis client : case class Worker(masterRedis : RedisServer, slaves : Seq[RedisServer], mongoURI : String) extends Actor { // create redis and mongo clients here } This approch works but it's against Akka principes : a redis and mongo client create a new ActorSystem so if I have 1000 workers, it will create 1000 ActorSystems. From what I learnt with Akka, it's not recommanded. I search some similar post in this group to seach a solution but I also have the impression that the code didn't respect an another Akka principe :* share nothing*. I wanna share one instance of an object in the cluster with other actors. I also read this part of documentation : http://doc.akka.io/docs/akka/2.4/scala/howto.html but I cannot found a use case close to the mind. In my developer mind, I think that I'm not the first to have this kind of problem so if you have any advices or remarks about my current design, it will really be helpful for me. Thank you hackers, Alifirat Kilic. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
