On Mon, Nov 24, 2014 at 1:32 AM, Nidhi <nidhi.ashwani.aggar...@gmail.com> wrote:
> Hello all, we are working on a course project to simulate twitter server > and twitter users and test the server for the load it can handle. We have > users on one system (with one client master and around 10,000 user actors) > and server on another machine(one master and 1000 worker actors) to resolve > the queries it gets from the user actors remotely. We are sending an > average of 6000 queries/sec to the server. We have one TCP connection > between the 2 systems and we get the following warning > > [TwitterServerSystem-akka.remote.default-remote-dispatcher-6] [akka.tcp:// > TwitterServerSystem@122.122.122.122:2552/system/endpointManager/ > reliableEndpointWriter-akka.tcp%3A%2F%2FTwitterClientSystem%40127.0. > 0.1%3A56833-0/endpointWriter > <http://www.google.com/url?q=http%3A%2F%2FTwitterServerSystem%4010.136.43.158%3A52335%2Fsystem%2FendpointManager%2FreliableEndpointWriter-akka.tcp%253A%252F%252FTwitterClientSystem%2540127.0.0.1%253A56833-0%2FendpointWriter&sa=D&sntz=1&usg=AFQjCNFv8cFbjN-nkft3bH--gLuNWbbdVQ>] > [65138] buffered messages in EndpointWriter for [akka.tcp:// > TwitterClientSystem@127.0.0.1:56833 > <http://www.google.com/url?q=http%3A%2F%2FTwitterClientSystem%40127.0.0.1%3A56833&sa=D&sntz=1&usg=AFQjCNHDGGNE6uGCLY9WRq5G-oSxo0FaGQ>]. > You should probably implement flow control to avoid flooding the remote > connection. > This warning indicates that many remote messages have been queued on the sender side. It is configured by: # Log warning if the number of messages in the backoff buffer in the endpoint # writer exceeds this limit. It can be disabled by setting the value to off. akka.remote.log-buffer-size-exceeding = 50000 The recommendation about flow control means that you should add some application level protocol (messages) between sender and receiver that controls how much the sender is allowed to produce before it stops sending more. Without that you will get out of memory if the sender continues to produce faster than what can be consumed. > Just came across this thread and thought it is relevant to our problem. We > know since we have one tcp connection(can we increase the number of > connections?), it might be a bottleneck. Both the server and client > machines buffer messages for each other. > > How do we go about using this new fix ? > The improvements for sending bursts of remote messages have been included in Akka 2.3.x since 2.3.3. Regards, Patrik > > Thank you. > > Regards, > Nidhi > > On Friday, April 25, 2014 9:48:55 AM UTC-4, Patrik Nordwall wrote: >> >> Boris, you should try the timestamped snapshot 2.3-20140425-151510 that >> is published to repo http://repo.akka.io/snapshots/ >> >> It is supposed to handle bursts of many messages without (much) degraded >> throughput or false failure detection. More details here: >> https://groups.google.com/d/msg/akka-dev/mFvz_d737t4/pZSmbFRLAV8J >> >> Regards, >> Patrik >> >> On Mon, Mar 24, 2014 at 5:00 PM, √iktor Ҡlang <viktor...@gmail.com> >> wrote: >> >>> Nice! >>> >>> >>> On Mon, Mar 24, 2014 at 4:55 PM, Boris Capitanu <bor...@gmail.com> >>> wrote: >>> >>>> Anyway, one thing to try is to set "akka.remote.backoff-interval" to a >>>>> larger value while setting the send-buffer-size to 1024000b. I would >>>>> try with backoffs 0.5s and 1s. While 1s is not a very good setting, it is >>>>> a >>>>> good way to test our hypothesis. >>>>> >>>> >>>> I've used the backoff-interval = 0.5s and send-buffer-size=1024000b and >>>> I do see the timings becoming more consistent (albeit worse). >>>> The standard deviation of the timings observed is much lower. >>>> >>>> Well - I think we narrowed down the issue. I'll wait for a fix... >>>> I'll be glad to test any nightly builds that include a fix if it would be >>>> helpful. >>>> >>>> -Boris >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>>> current/additional/faq.html >>>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>>> group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to akka-user+...@googlegroups.com. >>>> To post to this group, send email to akka...@googlegroups.com. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Cheers, >>> √ >>> >>> * ——————— **Viktor Klang* >>> *Chief Architect - **Typesafe <http://www.typesafe.com/>* >>> >>> Twitter: @viktorklang >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to akka-user+...@googlegroups.com. >>> To post to this group, send email to akka...@googlegroups.com. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> >> Patrik Nordwall >> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >> Twitter: @patriknw >> JOIN US. REGISTER TODAY! <http://www.scaladays.org/> >> Scala <http://www.scaladays.org/> >> Days <http://www.scaladays.org/> >> June 16th-18th, <http://www.scaladays.org/> >> Berlin <http://www.scaladays.org/> >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
