On Tue, Apr 21, 2015 at 1:25 PM, Anders Båtstrand <ander...@gmail.com> wrote:
> Have you tried running the master actor and the workers with different > dispatchers? > > http://doc.akka.io/docs/akka/snapshot/scala/dispatchers.html > > You could, just for test, assign a PinnedDispatcher to the master actor, > that way you will reserve a thread for that actor. > > Regards, > > Anders > > > tirsdag 21. april 2015 12.25.29 UTC+2 skrev Todd Leo følgende: >> >> Hi akka users, >> >> I'm new to scala/akka/JVM platform, and frankly I had a hard time since I >> tried to implement a simple web crawler using actor model with akka. In >> detail, this crawler reads a list of URLs from a local file, dispatch each >> URL to workers from master actor, and finally return webpage content to >> master. Seems to be a very simple project. >> >> At first I ran my crawler with 3 workers and feed them with dozens of >> URLs and the crawlers runs fine except some 403 errors. Then I applied the >> crawler to a 5k-lines-of-url with 1000 workers, the master actor can't >> print "acknowledge" in its receive function as well in 3-worker situation. >> Is this because of the master actor had been flooded? And how to solve this? >> > That can be the problem. If the workers are faster than the master the master will become a bottleneck and the mailbox of the master actor will grow, until you get out-of-memory. If that is the problem you need to add some kind of flow control. See for example this blog post: http://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/ /Patrik > >> Regards, >> Todd >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
