I have not tried to investigate why akka remote is slow, however I wrote a out of band netty transport(not in framework of akka remote) as simple as possible which 1) do batching send and receive 2) remove sender and receiver address overhead 3) simplified wire format. 4) try to avoid to dispatch to single intermediate layer queue(mailbox). Now we are able to reduce the mean latency from 300ms to 30ms, the throughput about is 3x speedup.
However, it cannot be implemented with akka remote transport interface, as it has many assumptions(like only support one way message sending, and the sender and recevier address is not in format of ActorRef). so it is purely application level optimization. On Wednesday, August 13, 2014 4:14:27 PM UTC+8, Patrik Nordwall wrote: > > > > > On Wed, Aug 13, 2014 at 3:18 AM, Sean Zhong <cloc...@gmail.com > <javascript:>> wrote: > >> Thanks >> >> with extrem high latency 727ms, >>>> If I remember the test correctly, that is not the latency of one >>>> message, but the total latency of all 10000 messages in one burst. >>> >>> >> >> If little's law applies here, the estimated latency is about 0.3s. >> (Assume pattern: burst -> wait for ack back -> burst -> wait for ack >> back....) >> The latency is consistent with my test. >> >> It was tested on GCE, with two machines in same zone. >> >> My test shows there is no difference between the two cases, two machine, >> or two jvm on same machine. It should not be a network bandwidth problem. >> >> After some performance profile, the bottleneck seems to be caused by >> netty transport, and it is not caused by the backoff logic. So, it will be >> interesting why netty performance here is so bad. >> > > Thank you for investigating. Keep us updated. > /Patrik > > >> >> >> On Tuesday, August 12, 2014 11:36:03 PM UTC+8, Patrik Nordwall wrote: >> >>> >>> >>> >>> On Tue, Aug 12, 2014 at 3:36 PM, Sean Zhong <cloc...@gmail.com> wrote: >>> >>>> Hi Patrik, >>>> >>>> In your report, looks like only be able to send 3.8MB/s data >>>> >>> >>> Note that payload size (100 bytes) is not equal to message size. >>> Additional overhead of the sender and receiver actor refs are also part of >>> the full message size, and that is probably larger than the payload size in >>> this case. The GCE console reported 23 MBytes/s as max during the test. >>> >>> >>>> with extrem high latency 727ms, >>>> >>> >>> If I remember the test correctly, that is not the latency of one >>> message, but the total latency of all 10000 messages in one burst. >>> >>> >>>> did you test this with sender and receiver on same machine or on >>>> different machine? >>>> >>> >>> It was tested on GCE, with two machines in same zone. >>> Ping rtt min/avg/max/mdev = 0.516/0.683/0.895/0.102 ms >>> >>> >>>> The performance is terrible. >>>> >>> >>> The test was focused on testing something else than looking for maximum >>> throughput, so I have no comment on that. >>> >>> >>>> >>>> (payload size) (burst size) (new: msg/s) (old: msg/s) (new latency ms) >>>> (old latency ms) >>>> 100 10000 42668 38058 727 744 >>>> >>>> >>>> >>>> >>>> On Monday, August 11, 2014 9:41:17 PM UTC+8, Patrik Nordwall wrote: >>>> >>>>> Hi, >>>>> >>>>> Unfortunately, the only test I'm aware of is: >>>>> https://github.com/akka/akka/tree/v2.3.4/akka-samples/akka-s >>>>> ample-remote-scala/src/main/scala/sample/remote/benchmark >>>>> >>>>> We have only used it for add-hoc testing. You can find some results of >>>>> running it here: https://groups.google.com/d/msg/akka-dev/mFvz_d737t4/ >>>>> pZSmbFRLAV8J >>>>> >>>>> Regards, >>>>> Patrik >>>>> >>>>> >>>>> On Sat, Aug 9, 2014 at 4:08 PM, Sean Zhong <cloc...@gmail.com> wrote: >>>>> >>>>>> For Akka 2.3.4, 2 machines, GbE network, one machine contains a >>>>>> sender actor, the other contains a reciver actor. Sender actor will >>>>>> continuously send message (100 bytes each, pre-created in memory) to >>>>>> receiver actor with simple flow control based on sliding window and >>>>>> acking(one ack per 100 messages). >>>>>> >>>>>> I can only get throughput about 60K messages/second , which is much >>>>>> slower than my expectation. The CPU usage is almost idle(3%). Network >>>>>> usage >>>>>> is about 20% of the bandwidth. I use kryo serialization, and >>>>>> singleConsumerUnboundedMailBox, 4 local dispatcher, 4 remote >>>>>> dispatcher, default setting for netty. >>>>>> >>>>>> Is there performance baseline number for akka remoting in your smoke >>>>>> test? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>> urrent/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>> p/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to akka-user+...@googlegroups.com. >>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>> >>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Patrik Nordwall >>>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>>>> Twitter: @patriknw >>>>> >>>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>>> current/additional/faq.html >>>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>>> group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to akka-user+...@googlegroups.com. >>>> To post to this group, send email to akka...@googlegroups.com. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> >>> Patrik Nordwall >>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>> Twitter: @patriknw >>> >>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+...@googlegroups.com <javascript:>. >> To post to this group, send email to akka...@googlegroups.com >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
