Hi, so the problem is that you don't need all the actors for all nodes at all times, but rather on demand, correct?
You mention that you check the count of routees for each router. Are those pool routers? If yes, do you really need many routees for each check or could you reduce the amount there? With that many "node actors" running concurrently, if each of them has its own set of child routers, it would seem that you'd exhaust the resources of your machine quite soon anyway so trying to increase throughput by adding more routees wouldn't help very much. Of the top of my mind: - are the checks stateful? I.e. do they need to be specialized / persisted for each agent? Or could they be stateless, the state being carried by messages, and hence be shared across nodes? - in order to throttle things, you could look into using Akka Streams. That is, you'd need your producing side to support backpressure - but perhaps you do control the execution of checks on the nodes, in which case you could make sure to not overflow your own system. See e.g. Colin's series on integrating actors and streams: http://blog.colinbreck.com/integrating-akka-streams-and-akka-actors-part-i/ Manuel On 29 January 2018 at 18:42, Yaron Shani <yaron.sh...@gmail.com> wrote: > Hey, > I am writing a POC for developing an large scale application. > My application is constructed from agents which can run some checks on the > node (a computer). The agents talk with the application using gRpc. > Each check is getting back to my application, which decides by the > response to run other checks - leading something resembling a graph in the > end. > I made partially successful POC with an real network of an 1000 nodes, but > I have some questions. > I created an actor for each mutable state (for each check). Each node can > run thousands of different check requests - leading to millions of actors > or more. In my POC, at the start of the application, all the actors are > being created, and my server almost runs out of resources from the amount > of actors and states being saved in the memory. > I am trying to find way to throttle the "boot" resource usage to the > amount my server can handle it even when I grow to 50,000 nodes or 500,000 > nodes. > One of the ways I tried to do it is by limit the amount of active agents > for each check for a certain amount as I dont mind the process taking > longer or not being super responsive as it does not interacts with users. > I am trying to that by telling the router of each check to count its > children and stash\unstash the messages when its getting to the limit. The > children actors are stopping themself using setReceiveTimeout. > The problem with that is that in Akka I can only unstashAll, causing all > the actor queue to drain\fill every time the limit goes below the > threshold, causing a lot of CPU being wasted. > Implementing my own stash\unstash using Queue is felling to me like I am > doing something really wrong. > Also killing actors and creating them constantly from snapshots\events > feels like abusing the Akka Persistence module. > How would solve the the ability to run millions and ten of millions of > requests, persisting their state, without exhausting the server resources, > while using them as much as possible, without really worrying about being > super responsive as those processes can take hours in some time? > Thanks! > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
